A sketch of a "design papers/pearls" category in academic conferences

Recent LtU discussions have mentioned (eg. here) that it is hard to get programming language "design work" to get recognized academically. Good design work that is *also* accompanied by solid benchmarks or strong correctness argument will get recognized for the benchmarks or formal guarantees, but design work on domains that do not lend themselves to easy performance metrics, or done by people that do not use formalization as a validation tool will be hard to get recognized. A recent blog post by Jonathan Edwards echoes the same frustration (note that it also complains about the worse-is-better philosophy prevalent in industry not being good at fostering radical changes either).

(I would like to apologize to the part of the LtU community that does not care about academia. I try to keep my usual submissions/topics independent of professional occupation, and this one is the exception. Feel free to just skip it.)

There are some academic spaces where design work can stand of its own. I know of the Future of Programming workshop, the Onward! conference, and maybe (it's probably too soon to tell) the new SNAPL conference. However, at least the first two events are considered somewhat second-class in the academic community (which has an unfortunate tendancy to foster competitivity through metrics and will keep core of "top tier" conferences, creating strong incentives to publish only at the most selective/established places). ICFP has a "functional pearl" category (see Call For Papers (CFP), section "Special categories of papers") that counts as a first-class ICFP publication, ITP has a "rough diamond" category (CFP, but-last paragraph), Oleg just mentioned FLOPS "system descriptions" (it also has "declarative pearls", CFP, "Scope"), what would a category of "design papers" look like? (Yes, the idea is have a niche to protect endangered species that the free market would tend to root out; I think we can win big by dedicating a reasonable proportion of our scarce resources to fostering intellectual and methodological diversity.)

Here is a proposal for how it would be defined, or rather, how it would be reviewed (which I think is the subtle question):

Each proposed language/feature design must come with a working implementation. The author should highlight the proper domain area in the submission, and maybe make a few suggestion of potential programs to implement. Reviewers are then asked to actually implement some programs using this language/feature, and rate the system based on whether they enjoyed this implementation attempt.

(I don't know where such a category would fit; PLDI has a D for "Design" in the name, but then I don't know much about this conference.)

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Communities are

Communities are conversations. The PL community is on a science kick right now and just isn't collectively interested in design. There is a changing of the guard right now, communities like OOPSLA (and...especially OOPSLA) are losing their quirky/fearless design crowd, who are being replaced by more and more conservative academics who play publish/perish games that non-academics are not really interested in. You can see it in conference programs, in LtU activity, in keynotes, and especially the movement of hot PL topics away to other places. The Internet is a great equalizer...it is super easy to bypass community gates and connect with exactly who you need to. This mirrors the trouble that all fields are having, not to mention the ACM...what is their roll to the world beyond the game?

But getting back to DESIGN, there are actually a couple of directions to consider. First, there is the incremental/iterative design that is going on with languages like Rust, Clojure, Go, Dart, ...these are evolving to be industry friendly but still have a lot of interesting details to work out. Then you have what is probably more "invention" than "design": Subtext, Eve and LightTable (what it was supposed to be, anyways), Elm, Bret Victor's design work, Lambdu. These systems are less practical (from the industry and academic perspectives), but are trying to set up a vision for the NEXT 20 years of PL work. This is not design, actually, this is just old-fashioned PL research via crazy experimentation in virgin territory. It is not science, it is invention.


Subtext is an example the kind of work that I would like to see in such a category (but then a nice property of the proposed evaluation is that it cannot really be gamed: papers that pass this test, even if they are about more academia-mainstream ideas, are interesting and valuable papers). Would you have a better name than "design paper" that captures this wider variety?

design vs. mad science

My wife is a designer. The work designers do is basically to apply thinking to solve user problems, it is very similar to engineering with a different paradigm (it is not even HCI, which is an acedemic discipline loosely related to design). The crazier more inventive designs are cream that designers don't get to do very often, design is very much evolutionary and not revolutionary in standard practice.

Also, there is no such thing as UI design, only user experience design - UXD (emphasis on users). Likewise, there is no such thing as programming language design, only programmer experience design (our users happen to be programmers). The field of programmer experience design is still kind of hazy, but it puts the core priorities in place. But again, most PXD is grounded and not very crazy, it isn't supposed to be visionary but solve problems for programmers. Most people think of concepts when they think of design, since that is sexier....

Invention and visionizing are attempts to explore what could be tomorrow and probably be wrong about it. Most concept cars never make it in production, but the process is necessary to feed innovation. They could lack implementation, in which case it is more like science fiction, or there could be prototypes barely held together by duck tape, that kind of work but without the niceties of correctness proofs and/or lots of mature engineering. They might be oriented torward improving programmer experiences, but they could just as well be crazy type systems or programming models. Picture a mad scientist working alone in their garage.

There can be no formal communities for mad scientists, who just look like crackpots to more conventional researchers (who are also right 80% of the time with that label). Also, they are so out there working on different directions, that the "me too"-ism and common ground that allows communities to be formed at all is just lacking. Communities are conversations, but if everyone is talking about different things...you can see where the crackpot accusations come from (if you do your own thing, peer review as a check just doesn't work very well, so we cannot easily separate out the loons from the geniuses. Get two mad scientists in the same room, they will accuse each other of being well..mad). The problem then with the PL community isn't really fake science, just a poor selection of topics with multiple researchers working on them.

So, it is very worthwhile to pursue programmer experience design as a topic, and the acedemic communities will probably do so once they understand it better. However, I think you (and everyone else) are fishing for mad science, not design. Design just gets mixed up with that since design is supposed to be edgy (most of it isn't, just what makes the news), and is poorly understood by computer scientists. The design papers of the past have also been mostly in the mad science category so....


Thanks for the additional comments. I am not sure I agree about the "fishing for mad science" point. I wouldn't consider Subtext to be "mad science" (in the sense that it is close enough from what I already know for me to relate to it), or the recent Mezzo paper probably wouldn't qualify either -- and I feel it could be tuned to fit this proposed category. Maybe the discussion could be refined by proposing examples of papers that would be suitable for such a category -- but then the point is more the papers that never got written.

Whenever I have personally thought "ah, I'm doing design right now" I think it corresponded to more incremental changes. An example would be in the work on Full reduction in the face of absurdity (pdf). The general idea is to have a language with full reduction (reducing even under lambda-abstractions, which corresponds to a symbolic evaluation of terms with free variables), and only provide a kind of (assert P in t) construction that blocks reduction in (t), because it depends on an logical assumption P (and could be unsound to reduce if P doesn't actually hold).

We played a bit with what we could express with the system, and decided to add a dual construction, morally (hide P in t), to locally un-block sub-subterms which do not depend on the assumption. This was motivated by looking at various assumption-blocking pattern, wondering about whether/how the programmer could express where subterms could and coult not be reduced. Without (hide), you sometimes have to perform O(N) code reordering to get the desired reducibility properties, which come done to a local O(1) modification if (hide) is available. Strictly more convenient to use.

Then we found out that (hide) was also essential in establishing a meta-theoretical property (confluence), and we marketed it as such, with "it is more convenient for the hypothetical programmer" presented as a side-benefit.

Labels are inaccurate.

Mostly we just do what we think is cool and useful. It is not always easy to categorize it: am I doing science, engineering, design? Most of us are not even trained in design thinking, and we are probably using more creative engineering to solve problems rather than design, but does it really matter?

Where we don't do that it is lack of better ideas or, more likely, a pressure to fit in. It is high school: what clique you are in is important, no one wants to be a loner (we also have notable bullies in the PL community, so the analogy is very apt))! That is my point about mad science: it lacks the community conformance pressure.

Adding and removing constraints is a common technique in lateral thinking. So take some constraint that everyone thinks is necessary (e.g, type inference must generalize types, glitch freedom is necessary) and nuke it. Then see if the resulting system is viable, or what needs to be added to make it viable. This is even more mad science because you are a heretic to what the community thinks is holy.


please keep up the seriously good work.

Also note that your proposal

Also note that your proposal excludes lots of programmer experience design work. My work in particular is just as much about co-designed IDEs as it is about the language. Languages are not just static textual communication anymore.

It is a good idea overall, whatever you decide to call it. It probably isn't so much about design as it is about new languages and new language concepts (concept cars).

Nice idea. Hopefully this

Nice idea. Hopefully this can even let the work that can be accompanied by a correctness proof or benchmark be recognized for its true design contribution. I don't think anybody seriously believes that the correctness proof is the main contribution as if the paper were a mathematics paper (most correctness proofs are uninteresting from a mathematical viewpoint), but that's what people pretend anyway. Programming language design is neither math nor science. A working implementation and a few demonstration programs that convey the main benefits is far more convincing, imo.