Lambda the Ultimate

But it might be better from an architectural point of view to avoid macro definitions everywhere in the application code and factor them out into a preprocessor framework like pivot instead, where they are ( hopefully ) well separated together with the description of the extension / DSL. Writing all application code in a style such that a certain transformer component can be written most easily strikes me a somewhat strange language design philosophy. So the advancement may simply be a well factored solution without the arbitrary constraint of writing source code in the form of parse trees?

i agree that this is an advancement (some lisp/scheme aficionados might not ;). still, the concept is the same: a code analysis and generation phase for a machine processable representation of the source code. the paper mainly talks about the high costs for dsl support in tool chains and the like, which is a non-issue for scheme.

besides, this is not new in the domain of non-machine-processable languages either. see A Metaobject Protocol for C++ for a compile time programming framework quite similar to the pivot in stroustrup (though it lacks the explicitness in regards to design conventions and associated costs). openc++ is a mature implementation of that protocol. similarly, gccxml provides machine processable intermediate xml output. edg earns a living by licensing their front ends for that kind of work.

it's a good idea, but not a new one...

Could you be a bit more precise about what you mean with "Meta-programming"? Macros are meta-programming, meta-object protocols are. Sure, extensible programming language are incredibly powerful. But, speaking of (Lisp-like) macros, I wonder if there will ever be a macro system for languages "with syntax" that will be as powerful and as easy to use as the ones for Lisp. Trying to use Template Haskell just showed me that there's still a long way to go and I'm not even sure this is necessary...

(In fact Lisp code is even much easier to write once you have structural editing support--which is very easy to implement due to the simple syntax--and helps you avoid stupid "the compiler interprets it differently than you"-bugs.

Taking languages like Java, Visual Basic, C# and C++ into consideration, meta-programming means to execute some code at compile-time instead of run-time. The compile-time code would be given full access to the environment of the compiler, thus being able to transform any program into any other program, as well as prepare the necessary infrastructure for the program to run.

Exactly this is what Template Haskell does. It even integrates with the type checker: The transformer function is type checked to return the correct type (basically a syntax tree) and the resulting code is of course type-checked, too.

The big problem is: Without a nice quoting facility this becomes incredibly unusable and ugly. The following code:

 (InstanceD [] (AppT (ConT (mkName "Compos")) (ConT name))
          [FunD (mkName "compos")
             [Clause 
               [VarP (mkName "return"),
                VarP (mkName "ap"),
                VarP (mkName "f"),
                VarP (mkName "t")]
               (NormalB
                  (CaseE (VarE (mkName "t"))
                    (matches ++
                     [(Match WildP 
                        (NormalB (AppE (VarE (mkName "return"))
                                       (VarE (mkName "t"))))
                        [])])))
               []]])

generates this simple snippet:

instance Compos Main.SExp where 
  compos return ap f t = 
    case t of
     ... other matches ...
     _ -> return t

Furthermore, it's not clean either, since it might interfere with names in the context. There is a simple quoting facility, where you can write your "code template" like [| let x = foo b in ... |] but you can no longer use it once you do little more complicated stuff. The description of the syntax tree datatype also takes multiple pages.

So if you want to design a language with a usable meta-programming scheme in this sense, then you have to design your syntax appropriately. Otherwise, it can quickly get a big PITA.

You don't need any special syntax except for one keyword. This keyword is responsible for identifying the parts of code to be executed at compile-time.

A compiler should just be smart and figure out which parts should be partly evaluated at compile-time. What you do need is 2 environments, so that you can either write CompileTimeSystem.loadFile or RunTimeSystem.loadFile. For websites, you could have 3 environments, compile-time, server-side run-time and client-side run-time.

I don't think your argument is valid. The constructive approach of creating nodes as shown in the TH example is just a basic but essential step on writing a more user friendly macro facility on its top. Once you have established it you can reverse the creation process and start with the target of the transformation and insert the necessary nodes found in the source.

I've done all this recently for Python using the following ingredients:

1) A grammar rule description of source code to be transformed into the target language e.g.

repeat_stmt ::= 'repeat' ':' suite 'until' ':' test 

2) A function that extracts nodes like suite and test from the parse tree, which is trivial.

3) The description of the transformation target with slots / node variables being interspersed:

while True:
   <suite>
   if <test>:
      break

4) A function that keeps the target, passes the nodes into the appropriate slots and calls the transformer to spit out the resulting tree. I simply called it expand().

This can be enhanced by introducing an operator that executes arbitrary code during transformation/expansion within the target which is mostly used for conditional expansion with an unknown number of branches ( think about a switch statement that shall be transformed in a long sequence of else-if's for example ) or in cases where e.g. <test> represents a list of nodes and one has to select exactly one using a subscript e.g. <test>[0].

The implementation of the machinery and the particular evaluation operator is neither very extensive ( ~400 LOC ) nor trivial but it doesn't matter because it is hidden from the user.

As long as all of the bugs really are worked out ;-)

Some bugs must stay. Otherwise people have no fun in finding them. Where should the self-affirmation come from in an age where the best development methodology is to use Google codesearch and paste found code snippets together?

The most severe problem with a powerfull macro system is that it conflicts with the intention of the language designer to grow the language himself. He decides about the default syntax and semantics of all language constructs. So either the language is still immature and has a way to go. Then using macros is premature design and syntactical decisions made by a framework/library programmer are likely duplicated or broken as the language advances. Or the language is mature and cares for everything in its huge ecosystem then macros are superflous and confuse programmers who are not familiar with custom syntax.