Lambda the Ultimate

From the Part 5, after the author added Ruby-style "last closure parameter is a syntactically separate code block", with clever extensions for several "named" closures and a special case of variable binding of loop parameters:

There are many possibilities here, as this opens up the doors to syntax experimentation without actually having to modify the language grammar itself. This is very similar to macros in languages like Lisp and Nemerle. On one hand, it’s more constrained to ensure that each extensibility point always conforms to a common set of structural principles, but the variety of structural extensibility points make it extremely versatile.

This is a theme that I see coming back a lot, such as for example in mixfix operators (which have been discussed here) extended in Agda with binders, and by shap with control structures (in a strict language).
On a related note, some languages try to make the existing syntax more flexible : Haskell do-notation, F# computation expressions, or Scala "for loop is a message".

The idea is to isolate "syntax extension patterns" that are used in numerous situations, and to devise a specific extension method for each.
I see those as counterparts to the Lisp/Scheme macro tradition, where the goal is to have one syntax extension facility to rule them all. With a good, general and robust enough definition of macros, all kinds of syntax extension become possible. Of course, this is not contradictory to pattern recognition: "extension methods" for specific patterns may be provided as libraries implemented with the general macro facilities, but providing a simpler specialized interface. But this impose the burden of coming with a good macro framework (hygiene, error messages...), while a ad-hoc extension pattern may be much simpler to implement directly, with less failure points.

Syntax extension is just the "fancy" side of the iceberg : to be useful it should be coupled with powerful semantics facilities in the language. Lambda-abstractions are enough for binders (and even a bit "too much", because of "exotic" terms) and thunks/lazyness for control structure, even if continuations are sometimes handful there. I'm specifically interested in the syntactic part.

What are the important patterns that language designers should try to take into account when adding syntax extension facilities in their language, or evaluating the power of their syntax extension framework ? I have collected a few of my own:

- quotations as a generic way to safely embed a completely different syntax inside delimiters, that get expanded at preprocessing-time into the host language AST -- which make antiquotations possible. This is present in Camlp4 and Template Haskell for example, and does not require to make the syntax of the host language extensible (only to have a hardcoded quotation syntax which is parametrizable enough)

- binder structures implemented as syntaxic sugar over lambda-abstractions -- makes it possible to infer the intended role and scope of the binder structure from the proposed expansion, and report meaningful error messages in term of the sugar.

- the more general monad syntax, extended with various common patterns (MonadPlus, MonadError...)

- annotations as a generic, non-intrusive way to add comments, control the language tools ("deprecated", or "do not raise a warning here"...), or preprocessing meta-programming tools ("please generate an accessor function here")