Lambda the Ultimate

Erik started an interesting post, but since not everyone can see it on Facebook, I've reproduced it here. I apologize for any errors in transcription.

Erik Meijer: Oh no, Wirth, Liskov, Allen are pimping DSLs. Please elders of the PL community, don't teach our grasshoppers the wrong ideas ...

Mark Hamburg: Link?

Erik Meijer: panel at ACM Turing celebration (#ACMTuring100)

Rastislav Bodik: Look who is speaking, Mr. Rx. :-)

Damien Guard: DSLs are bad?

Erik Meijer: I fully ascribe to Hudak-style embedded DSL, which really just are well-designed APIs. External DSLs on the other hand are like puppies, they all start out cute and happy, but without exception turn into vicious beasts as the grow up (make, XSLT, regular expressions, ...).

Michael Maximilien: Like all (most) puppies, it takes constant care and feeding over long (especially from start) to end up with a well behaved, integrated dog... Unless you are the dog whisperer ;)

Rastislav Bodik: ‎I can agree with your comment, mostly. While technically correct, calling Rx an API does not do it full justice, I think. It distracts from the fact that Rx introduces abstractions that are composable, analyzable, and --- in an alternative implementation --- also compilable. And this is the point that I was hoping the Turing elders emphasized: regardless of the implementation (API or external), DSLs provide systematic building blocks for today's complex software.

Rastislav Bodik: On the question of cancerous growth of external DSLs, I also mostly agree, but I came to believe that making a language internal helps only so much. Three other factors for a good DSL are the existence of a theory (observable-iterable symmetry in Rx; attribute grammars in parser genrators); whether you need to compile the DSLs (having to do so restricts reliance on the host language); and the experience of the designer (I can share with you some beast internal DSLs from cs164). But this is already too much info for facebook...

Will Cook: I disagree. I think the problem is that we have never really taken seriously the kind of tooling (and theory) required to define and integrate families of external DSLs. But language workbenches are starting to make progress. My Ensō project (with Tijs van der Storm) is based on integrating external DSLs with interpreters (not compilers/translators) . There are significant problems with internal DSLs as well.

Rastislav Bodik: Let collect these comments into a workshop proposal!

Erik Meijer: The cost of building a great language experience is extremely high, language workbenches may help some but I learned that a large part of the fit and finish required to make a truly great experience is hard to automate/generalize. Also we should not underestimate the (informal) ecosystem of documentation, books, samples, blogs, courses, ... All that leads me to believe that "worse is better" and we should do the best job you can by defining a great API/internal DSL.

Rastislav Bodik: ==> Sounds brilliant an timely, as this thread shows.

Will Cook: What about:
International Working Conference on Domain-Specific Languages (DSL 2011) (not sure about status of next one)
International Conference on Domain-Specific Languages (DSL 99) [Oddly, this conference is unrelated to the one above]
Agile Development with Domain Specific Languages

Will Cook: I understand that there are costs, but some of these costs are subject to better tools. I am focusing on creating and integrating a few DSLs for widely used domains, the kind that can be widely implemented and shared (SQL DDL, SQL, BNF, Attribute, Authorization, GUI, WebUI, Datalog, etc). For smaller domains and more narrow languages, embedded is definitely the way to go.

Yannis Smaragdakis: Ok, I'm late to the party, but it's a topic I care about a lot.
On extensible langs/workbenches: I'm afraid that there are "inevitable" DSLs (the "deep domain theory" kind, e.g., SQL, yacc), which will be successful regardless of how they are implemented and "trivial" DSLs that could well be APIs or lexical macros. There is little room between the two for large impact through good "workbench" infrastructure.

On lang design: I have to agree with Erik, language design is far from modular. It's more "integrated experience" and less "mix-and-match".

On whether external DSLs are ugly: I think they are, but so are death and taxes. There will always be a need for external DSLs. E.g., computation in different domains is just too diverse for a single control-flow pattern, even with higher-order functions or other clever constructions.

Will Cook: Yannis, you make some great points. I hope there is room in the middle. But don't give up on external language modularity yet :)

As a datapoint compare these approaches to internal and external bidirectional parsing/pretty printing on the dimensions of simplicity, usability, modularity, flexibility and elegance:

Invertible syntax descriptions: Unifying parsing and pretty printing

Object Grammars: Compositional & Bidirectional Mapping Between Text and Graphs

There certainly are advantages and disadvantages on both sides.

Yannis Smaragdakis: ‎Will, I am also hopeful there is some room in the middle for workbenches. But I'm pessimistic about external language modularity. It may be just lack of imagination: I recall several instances of language features that I thought were very nice and orthogonal, elegantly translated away, for which I was surprised to find out ugly interactions with seemingly unconnected features.
BTW, thanks for the pointers! They'll definitely be part of my summer reading! The parsing/unparsing problem is fascinating and I'm eager to see the state of the art.

Will Cook: I think part of the problem is the pervasive assumption that external DSLs must be translated/compiled. My hunch (research hypothesis) is that compilers don't compose well but interpreters can be composed. You are right that the composition must sometimes deal with interactions between features. Can list list the examples that you are thinking of?

Will Cook: Also WG 2.11 is now actively planning for another DSL conference. Current question is where it should be colocated. Here are the proceedings of the last one:
EPTCS 66: Domain-Specific Languages
If you are interested in helping to organize, please let me know.
There is also Code Generation
(Also, does anyone mind if I move this discussion to a more public place? LtU? Even though Erik knows everyone, I don't think FB realizes this)

Yannis Smaragdakis: Will, It's not easy to precisely recall or list examples of complex lang. features' interactions, exactly because they are weird and complex. But I remember several instances in my experience at the Intentional Programming project (at MS in the 90s). Things like Java-style exception handling, which seemed orthogonal to the base language, turned out to interact with other features in strange ways.

What I often bring up as a toy example (not representative of deep complexity, but easy to remember and explain) is the implementation of a "for" loop. It's easy to claim that "for" can be a high-level macro: for (init; cond; incr) body is short for: init; while(cond) { body; incr; } Only it's not. The body may contain a "continue" which makes the translation invalid. You try a different translation and "break" breaks. There is probably no valid translation without labels and gotos. I can't see this being a unique problem to compilation. It would be the same with interpretation. You'd need to expose lower-level control flow, or you can't express the new language feature.