Lambda the Ultimate

(BTW, I'm not sure how much my objections have to do with my being a theoretician; I added that bit because Olin Shivers took a few friendly jabs at us in his message.)

from http://www.ai.mit.edu/%7Egregs/ll1-discuss-archive-html/msg00361.html

> In this case, having a few useful builtin/standard datatypes or
> collections helps me write a small program quickly.

The implicit claim here is that having many built-in datatypes is good. I think it is bad. When I see a language like this, I think, "This language addresses the symptoms, not the problem."

The need to build in datatypes reveals a deficiency in the expressiveness of a programming languages. More expressive languages allow user-defined datatypes to be behave more like built-in (first-class?) types.

You can recognize this very clearly if you are familiar with a few typed functional calculi. For example, in early versions of (LCF) ML, there was no `datatype' declaration; you could only define datatypes via recursive type equations, and so the coproduct and product type operators had to be built in. In Standard ML, coproducts can be defined using `datatype', i.e., they got subsumed by a more general construction, but product is still primitive. In a language like Charity, there is a declaration form for `codatatypes' which lets you define not only products, but also the function space type. In polymorphic lambda-calculus, you can do away with recursive type definitions altogether, since they are definable using impredicative polymorphism. You still need the function space operator, though; in pure type systems, even that disappears, and all you have is the dependent product binder.

Now, to get really abstract: This pattern of things at the object level being subsumed by constructions at the meta-level is more than a coincidence. It is an instance of a free construction across semantic dimensions which can actually be formalized and, indeed, it is one of the things which forms the foundations for the programming language I am designing.

I admit that this idea is a little bit different from what the poster had in mind, since he was talking about a dynamically typed language. But, see my rant on languages vs. libraries below.

from http://www.ai.mit.edu/%7Egregs/ll1-discuss-archive-html/msg00430.html

Shriram Krishnamurthi gave a great rebuttal to this too but, darnit, I have axes to grind! ;)

> Or , to put it as simply and briefly as possible, in answer to "Why
> is Python considered lightweight and Lisp is not" , there are three
> reasons.
>
> 1. Syntax.
> 
> 2. Syntax.
> 
> and
> 
> 3. Syntax.
>
> Syntax matters. It matters a lot. Academic language designers seem
> to be most concerned with semantics, completeness, doing the right
> thing, etc; but not as much concerned with syntax.

Boy, if I had a nickel for every time... grumble, moan. Is syntax the last refuge of linguistic luddites?

This poster claims syntax matters a lot. It sure seems to matter a lot to the poster. I suspect the reason for that is that the only major differences between the languages he has used are syntactic differences.

Needless to say, I don't think syntax matters much. As with anything, if you take it to extremes you will succeed in making it a burden (as the designers of XSL must have intended!), but for most people this issue was settled long ago, once they figured out how to read and write BNF grammars. Once you can do that, you know that you can write many parser frontends for the same language, each using any number of different syntaxes.

One might even go so far as to say that syntax is not part of a `language' at all: if you change the syntax but not the semantics, there is a strong sense that the `language' has not changed; on the other hand, if you change the semantics but not the syntax, then it seems more like a different `language'. I guess even Perl people, who are certainly among the staunchest syntax evangelists in existence, must recognize this fact, since they plan to allow Perl 6 to have several different concrete syntaxes, though they still call it Perl 6.

There is no perfect syntax. A good syntax for a language is one which is optimized for the language semantics. Even among optimal solutions, there are usually several representations which are equally good, or make balanced trade-offs. (Example: cons and snoc lists.)

I haven't yet met a language whose syntax issues overwhelmed its semantic issues. I have used Scheme, C, ML, Haskell, Prolog, Perl and many other languages, and I never found the syntax to be even a quarter as important as the semantics. Maybe XSL is an exception, but it is such an extreme I don't even count it---though I feel compelled to keep mentioning it! Anyway, if XSL had much(, much, MUCH) more to offer me on the semantic side, I would probably grin and bear it. If I had access to an Emacs, that is.

(The first person who now mentions Intercal, TECO or any of that other crap will get a complimentary bitchslap.)

To put it pedantically, the relative importance of syntax and semantics is analagous to the relative importance of manual optimization and algorithmic optimization. Translating a fragment of your program into assembly for speed will may buy you some constant factors (but, on today's machines, only if you are especially clever), but improving your algorithms can get you exponential increases in performance. I think it is the same with syntax and semantics.

If the syntax is so terrible it gets in your way, you can use a programmable editor or a macro system or something. (That might even make XSL usable...!) If that is still not enough, and you need some sort of metaprogramming device, then something semantic is lacking in the language. After all, a compiler is a metaprogramming device: it's a device which transforms a semantic specification into a syntactic entity which is directly executable, i.e., without further analysis.

Now, you can argue that for short and dirty one-off programs like the sort Perl seems to be designed for, the tight-loop assembly optimization argument implies that syntax becomes relatively more important. I can agree that that is true to an extent, but again I think this is addressing the symptom, not the problem. The problem is, why are you writing so many short and dirty one-off programs? Programming is supposed to be about automation and reuse, isn't it? Short, trivial programs like this tend to solve trivial problems; we should not be writing trivial programs each time we run into a trivial problem. We should be solving them once and for all, with comprehensive and uncompromising solutions.

Hacks breed hacks. (Yes, you may quote me. :)

BTW, I should say that, while I think syntax is not interesting, metasyntax issues (parsers, for example) are. (In general, it seems meta-things are usually more interesting than the things themselves.)

Well, I could ramble on about this for ages, for example about why I don't think LISP/Scheme's syntax gives any particular advantages for metaprogramming, but there are other points to cover...

from http://www.ai.mit.edu/%7Egregs/ll1-discuss-archive-html/msg00311.html

> There will be no concrete definition of
> "lightweight language" but let's look at the creators intent. Did the
> creator optimize common micro operations like string interopolation,
> regular expression matching, basic arithmetic, networking libraries,
> graphic libraries and list processing? Or did they spend more effort on
> in-the-large features like exception handling, type declarations,
> interfaces, polymorphism and templates? If the former, it is a
> lightweight language. If the latter, it is a scalable language. If
> *both*, it is the language you want to choose.

Actually, I rather like this message (the lightweight vs. scalable perspective is interesting), but what bugs is me is that the poster regards "string interopolation, regular expression matching, basic arithmetic, networking libraries, graphic libraries and list processing" as language issues. They are not language issues; they are library issues. You can add these capabilities to any reasonable language by writing a suitable library.

If you're crazy about Perl regexps, use the Perl regexp library. It's not written in Perl; it's written in C. (What does that tell you about Perl?)

Are you wild about graphics programming? Hey, look no further: there's MSFC, wxWindows, Tk, Gnome, KDE, Qt, Mesa, ... What do they have in common? They are all libraries.

Need basic arithmetic? No problem; every modern processor comes equipped with instructions which access its ALU and FPU. You don't even need a library here.

Want bignums? Get one of the exact real arithmetic packages.

Where are the language isssues? These packages (basic arithmetic excepted) are all written in C or C++, not in high-level languages. There aren't any language issues here, except inasmuch as the language influences how much abstraction you can do in the libraries, and even there the library design issues tend to dominate, because the library designers are restricted by the semantics of the underlying C/C++ implementations.

If your approach to designing your language starts with one of these topics, you are solving a non-problem.

from http://www.ai.mit.edu/%7Egregs/ll1-discuss-archive-html/msg00274.html

Here is one of the few places I disagree with Shriram. While trying to formulate a definition of "lightweight language", he comes up with this criterion:

3. I should be allowed to submit, but not run, buggy programs.  That
   is, for rapid prototyping, I should be able to write a program with 
   3 functions, A, B, and C; C can be buggy, but so long as I use A
   and B only, I shouldn't be prevented from running the code.  ML
   fails at this.  The ML-like type systems for Scheme succeed.

First, the claim I disagree with: by his definition, ML does not, in fact, fail this criterion, since ML does type-checking, not `bug-checking'.

That's a minor quibble, though. What he is trying to get at here is that ML type-checking disallows unbound variables, so you cannot leave a function `unimplemented', even if it never gets called at runtime.

In Scheme (Shriram is a Schemer), on the contrary, you are allowed to do this. For example, you can write

(define (f x) (+ y 1))

even if y is not in scope. If f ever gets called, you will get a runtime error saying y is unbound.

This is one of the things I don't really like about Scheme, actually. It is pretty easy, for example, for typos to give you runtime errors. In my example, I could easily have meant to have written x, but have typed y by mistake.

In an ML-like language, perhaps you might reasonably assign the type 'false', or a fully polymorphic type to an unbound variable, but forcing the programmer to bind it to _something_ explicitly ensures that he doesn't completely forget about it later.

Forcing him to bind it doesn't mean he has to write a whole implementation for it, though. In Haskell, you can bind a variable to the bottom value (e.g., to 'error'). In ML, you could bind a value to something that raises an exception. In both cases, the variable will be assigned a fully polymorphic type (unless you constrain it with a type annotation).

(Of course, Shriram is trying to define the notion of "lightweight language", not "good language", so I suppose it is not impossible that he might agree with me here.)

Well, that's enough for tonight. Apologies for the typos, etc., which undoubtedly exist; it's now 3am and I'm tired.

Except that in the real world, most people don't know how to read and write BNF grammars.

> Except that in the real world, most people don't know how to read and write BNF grammars.

Not only that, but people who can, don't spend their time writing new frontends for existing languages.

I do think that the importance of syntax is often overrated, but I have to say I think it is important at some level, if only because of the "ergonomic" factors of programming: you have to type programs, and you have to read them. Like many professional programmers, I have some slight wrist/hand aches, so I tend to prefer syntax that doesn't involve many shift keys. My hands are noticably more sore after 8 hours of Java than after 8 hours of Python. So the prize for most comfortable language to type goes to Forth, but personally I find Forth hard to read. Reading is the other important part. Brevity is important, too (though this can be taken to cryptic extremes, which is not what I'm talking about).

As long as I've been programming, I've hated S-expression syntax. But I've always admired the simplicity, clarity, and power of Scheme. Recently I decided to just start writing in it, and never mind the syntax. Not too surprisingly, S-exp syntax has grown on me. I'm not a "convert", but I don't mind it anymore. So there's a point in favor of the syntax-doesn't-matter argument. I freely admit you can learn a new syntax faster than you can learn a new semantics. I'm just saying syntax is still important.

Another important aspect of syntax is that it's important for the user community. If I invent a new s-exp or Pythonic syntax for Java, I can't submit articles that use that syntax to trade journals, and (more importantly) I probably can't use that syntax at work, because it's not standard, and people will be attached to the standard.

Even if you make inventing new syntax easy, it doesn't happen in a useful way. Take Objective Caml for instance. This is the only language I'm aware of off the top of my head which ships with a preprocessor that allows you to completely redefine the syntax. Now, Ocaml has a somewhat (IMHO) warty syntax. I much prefer Haskell's. A lot of people dislike OCaml's syntax, including (I assume) the author of the preprocessor. He wrote a "Revised" syntax for OCaml, which is somewhat cleaner and more consistent. Most people don't use it. Examples don't use it. He also wrote an s-expr syntax for OCaml with the same tool. Number of examples of s-expr OCaml I have seen outside of that preprocessor grammar itself: zero.

So I think syntax does matter. I'm much more worried about semantics, but I think syntax has a great deal of impact on the "feel" of the language, the user community, and on reading and writing of programs. I don't think we're at the point yet where we can just say "syntax doesn't matter, it's simple to roll your own".

Because it's basically a parse tree, and you can manipulate the parse tree for any syntax, or is there some other reason you have in mind?

> Are you wild about graphics programming? Hey, look no further: there's MSFC, wxWindows, Tk, Gnome, KDE, Qt, Mesa, ... What do they have in common? They are all libraries.

I'll assume you were just tired for this one. :-) You're bashing Perl for using C libraries in one sentence, and then saying you should re-use all these great existing C/C++ libraries in the next sentence! I think you're right though that library and language (where language = semantics, or = semantics + syntax) often get confused. At least in Perl's case, that's because the syntax makes special provisions for using, e.g., regexps. The open question is: is this justifiable? I'm guessing you'd say no, and so would I. I'd really like to be able to define syntax that helps me use a particular library as macros, though, as one might do in Scheme, Dylan, or what have you.

Can you tell us more about this? Sounds interesting.

Unless you have great flexibility over the syntax, this is false (see below)

If you're crazy about Perl regexps, use the Perl regexp library.

Alas this is not so simple. In many languages, "\d" and "d" are the same. The result is:

• either you write "\\d", loosing the readability of regexps (which can be hard to read already)
• either you preprocess your language

Library without sugar can be sour!

(i've already trolled about this on Ltu)

We got off on a bad start, so I'm going to try to choose my words carefully. (I really do want to have objective, civilized discussions, but I tend to take a few too many liberties here because it's a weblog.)

> for most people this issue was settled long ago, once they
> figured out how to read and write BNF grammars


Except that in the real world, most people don't know how to read and write BNF grammars.

First, BNF was published in 1960, and was used to describe the syntax of Algol 60. That was over 40 years ago. Is it unreasonable to expect a programming professional to know BNF?

Maybe Python does not target professional programmers. I have seen the CP4E (Computer Programming For Everyone) document, and I think it would be great if we could get more people programming, but let's try to structure their experiences in a reasonable way. You can't even write a program, unless you know what is a well-formed program, and what isn't, and the best way to do that is BNF.

Second, XML is not so far from BNF, and it seems to be quite popular.

Third, what is "the real world"? Is it the set of all programmers in industry (many, dare I say `most', of whom have no formal education in CS)? The set of all programmers, everywhere? Or the set of all people in the world?

I think "the real world" is a loaded phrase. (Thank you for not capitalizing it, BTW. :) It means what you want it to mean.

Fourth, BNF is for describing syntax, but the underlying principles are much more broadly applicable. For example, initial algebras are almost identical to BNF grammars, and an intuitive understanding of these is, I believe, a prerequisite for building non-trivial programs. By "intuitive" I mean that, though you may not know what an initial algebra is, someone is incapable of understanding initial algebras will never be able to write non-trivial programs. (OO programmers will be more comfortable with final coalgebras, but they are just dual to initial algebras, so it's no biggy.)

> > Except that in the real world, most people don't know how to
> > read and write BNF grammars.
[..]
> I do think that the importance of syntax is often overrated, but
> I have to say I think it is important at some level, if only
> because of the "ergonomic" factors of programming: you have to
> type programs, and you have to read them. Like many professional
> programmers, I have some slight wrist/hand aches, so I tend to
> prefer syntax that doesn't involve many shift keys. My hands are
> noticably more sore after 8 hours of Java than after 8 hours of
> Python.

Java is long-winded. No argument here! (I teach Java, BTW.) But is that because of the syntax? I don't agree there.

Operationally, there is not much difference between Java and Python. Both are garbage-collected, object-oriented, etc. The reason Java is long-winded is that it forces you to write over and over things which, at a more general level of abstraction, are instances of each other. If Java supported type inference and parametric polymorphism, I am sure that you would find it less verbose and more attractive. These are semantic features.

> Another important aspect of syntax is that it's important for the
> user community. If I invent a new s-exp or Pythonic syntax for
> Java, I can't submit articles that use that syntax to trade
> journals, and (more importantly) I probably can't use that syntax
> at work, because it's not standard, and people will be attached
> to the standard.

Sure, but I am talking about the relative and practical importance of syntax and semantics. I don't deny that people are attached to syntax; if I thought most people were comfortable with switching syntaxes on a dime, I would not have written my rant. What I am saying is that, in the end, 1) semantics is the more important factor, and 2) the domain in which it is more important is the readability and writeability of new, better programs.

> Even if you make inventing new syntax easy, it doesn't happen in
> a useful way. Take Objective Caml for instance. This is the only
> language I'm aware of off the top of my head which ships with a
> preprocessor that allows you to completely redefine the syntax.

Yes, I'm very familiar with Ocaml.

> Now, Ocaml has a somewhat (IMHO) warty syntax. I much prefer
> Haskell's. A lot of people dislike OCaml's syntax [..] Most people
> don't use [the preprocessor].

So you are claiming that, though it is widely acknowledged that Ocaml's syntax sucks, people still use it, despite the fact that it's easy to change it with the preprocessor?

To me, that sounds like an argument that syntax isn't very important. If syntax is so important, and Ocaml syntax sucks, why keep using it?

> So I think syntax does matter. I'm much more worried about
> semantics, but I think syntax has a great deal of impact on
> the "feel" of the language, the user community, and on reading
> and writing of programs.

Impact on the feel: I disagree. Scheme syntax is totally different from ML syntax, but I feel quite certain that the reason Scheme feels different to me from ML is not the parentheses, but rather the lack of static typing, algebraic datatypes, and other semantic components. I feel certain about this, because every time I start programming in Scheme now, the first thing I want to program with the macro system is a static type checker, and not a parser for an ML-like grammr.

Impact on the user community: As far as attracting people to the user community, I agree that it has a big effect, since most programmers seem to hate s-exps. But that is more prejudice and culture than an inherent fault of the syntax. So do you think that quantity of users is absolutely more important than quality of users?

Impact on readability/writeability: Only in extreme cases, as I mentioned in my original message.

> I don't think we're at the point yet where we can just
> say "syntax doesn't matter, it's simple to roll your own". 

We're not, but we should be. I feel the biggest hurdle is that we should have tools which can deal easily and effectively with any (at least, LALR) syntax, but we don't. Text files are, unfortunately, still the common denominator. (I think the reason for this is that we are afraid that if our tools break, we won't be able to access data in structured files. XML is the compromise we made to address this: it's structured, but it's still text, so it has to be parsed every time you read it. If we had reliable programming languages which built reliable programs, we wouldn't have to store structured data as text, since our tools would be reliable.)

Well, I could ramble on about this for ages, for example about why I don't think LISP/Scheme's syntax gives any particular advantages for metaprogramming, but there are other points to cover...

Because it's basically a parse tree, and you can manipulate the parse tree for any syntax, or is there some other reason you have in mind?

A bit more than that. A (functional) language like ML, which has a more structured syntax, typically has algebraic datatypes, which are just enough to handle the additional richness of the syntax. The analogy I often think of is similar triangles: ML has as much semantic richness to handle its own syntax (which is typed, in a way), as LISP/Scheme have to handle their own syntax, which is untyped in a way.

Java is long-winded. No argument here! (I teach Java, BTW.) But is that because of the syntax? I don't agree there.

[..] The reason Java is long-winded is that it forces you to write over and over things which, at a more general level of abstraction, are instances of each other. If Java supported type inference and parametric polymorphism, I am sure that you would find it less verbose and more attractive. These are semantic features.

I'm stunned. I've been thinking of Java's wordiness as syntactic obtuseness, but that's just sloppy thinking - you're absolutely right, it's a lack of semantic features that's biting me, not syntax. Hrmph.

[..]What I am saying is that, in the end, 1) semantics is the more important factor, and 2) the domain in which it is more important is the readability and writeability of new, better programs.

Yes! Just so long as we don't dismiss (2) out of hand.

To me, that sounds like an argument that syntax isn't very important. If syntax is so important, and Ocaml syntax sucks, why keep using it?

Well, yes and no. People will still use it. But would they enjoy their work more if they didn't have to fight with the syntax? I think so. People will subject themselves to all sorts of abuse from computers, and programmers have a much higher tolerance for this pain than normal folks. But I think we owe it to ourselves to improve our lot.

Impact on the feel: I disagree. Scheme syntax is totally different from ML syntax, but I feel quite certain that the reason Scheme feels different to me from ML is not the parentheses, but rather the lack of static typing, algebraic datatypes, and other semantic components. I feel certain about this, because every time I start programming in Scheme now, the first thing I want to program with the macro system is a static type checker, and not a parser for an ML-like grammr.

Mmm, not quite the feel I'm talking about. I mean feel in the sense, "how do you feel after getting an eyeful of source code in this language"? If just looking at the code fatigues you, this is an important user interface problem. For instance, Dylan is an interesting language to me, it gets a lot of things right. But Dylan source code for some reason tends to have lines that are rather too long (mostly due to long naming conventions and especially to long type information), so my eye (at least) slides right over it without absorbing much. It takes a lot of concentration, because it doesn't scan well. Python, on the other hand, has very nice block indentation which describes the structure of the program visually, and it scans well because the lines tend to stay fairly short. I maintain that readability and writability is an important factor, and too-often overlooked.

> I don't think we're at the point yet where we can just > say "syntax doesn't matter, it's simple to roll your own".

We're not, but we should be.

Yup.

So you are claiming that, though it is widely acknowledged that Ocaml's syntax sucks, people still use it, despite the fact that it's easy to change it with the preprocessor?

To me, that sounds like an argument that syntax isn't very important. If syntax is so important, and Ocaml syntax sucks, why keep using it?

In my experience, syntax is even more important to "average" programmers than it is to me and other good programmers. And yes, it's "unreasonable to expect a programming professional to know BNF", at least in today's world, and assuming you define a programming professional as someone who's paid to write programs. I'm not defending that state of affairs - I love meta-things as much as anyone, and enjoy writing little languages using parser generators driven by formal grammars - but the sorry truth is that there are a lot of "professional programmers" out there who have picked up everything they know as they go along, starting e.g. with something like Perl in high school, to use an example of someone I worked with recently.

Also, rolling your own syntax is a long way from being viable, even if it's easy to do technically. A big reason that a language like Java succeeds, despite its many and painfully obvious flaws, is that all of the tools that are available for it, such as editing environments and documenting tools. Custom syntaxes would have to be handled by such tools. A big part of the reason that "mainstream" languages are so primitive today, is that it makes it easier on the tools, and doesn't require an enormous standardization effort in order to be able to support custom everything.

So while the pointy-headed theorizing is wonderful, and I'm a big fan of it myself, some of what's being said here is classic "ivory tower" - it simply dismisses what's really going on down on the ground, where the people live in mud huts, and those who can work ivory are prohibitively expensive. Besides, the ivory-workers really don't want to be working amongst the mud huts (unless perhaps an IPO is in the offing), and would rather be in that cozy tower with the awe-inspiring 1000-foot view (and tenure).

The trickle-down effect in the language space is incredibly slow, with a timeline of decades for a given set of features (perhaps accelerating in the Internet era). But it's only been 40-odd years since the first "high level" language was invented, and there's plenty of stuff that still has to trickle. Incontrovertible proof of this can be found in the fact that the entire concept of functional programming has virtually *zero* presence in the commercial world, some few special-case exceptions like Erlang notwithstanding.

While I'm rambling, I'll add one more thing: languages like Perl have something to add to the mix here, in terms of languages that work the way non-mathematical humans expect. Functional programming and languages derived from formalisms such as the lambda calculus are a huge leap forward, compared to what came before; but they only address one side of the equation. The other side won't be addressed by mathematicians or mathematical computer scientists, especially not if they think "syntax doesn't matter". They'll be addressed by people who are thinking about other, less uncompromising things, in the domain of what humans are comfortable with, rather than what theory dictates.

I consider Perl's success evidence of my last claim - a language that superficially eschews many traditional and formal approaches, yet provides features integrated into the syntax of the language in untraditional ways, that ordinary users find useful and usable. But this human factor is still largely unexplored, and I think it's quite possible that the future may very well be custom syntax layers built on top of a functional, intermediate-language core. (The JVM and .NET come to mind as early precursors of this approach) We still seem to be a long way from that, though.

If you're crazy about Perl regexps, use the Perl regexp library. It's not written in Perl; it's written in C. (What does that tell you about Perl?)

Are you wild about graphics programming? Hey, look no further: there's MSFC, wxWindows, Tk, Gnome, KDE, Qt, Mesa, ... What do they have in common? They are all libraries.

I'll assume you were just tired for this one. :-) You're bashing Perl for using C libraries in one sentence, and then saying you should re-use all these great existing C/C++ libraries in the next sentence!

You are generous. :) Yes, I guess I was being unfair to Perl.

Part of what was bugging me was that I think useful programming languages ought to be able to implement their own runtimes. Perl can't do this, but neither can ML. (Some peeople disagree that you can't, for example, write a garbage collector in ML, though.)

I think you're right though that library and language (where language = semantics, or = semantics + syntax) often get confused. At least in Perl's case, that's because the syntax makes special provisions for using, e.g., regexps. The open question is: is this justifiable? I'm guessing you'd say no, and so would I.

What I don't like is that languages like Perl make specific provisions in the language for specific external libraries and applications. Though it certainly helps when you want to use that library, or build an application of that type, it makes the language bigger, more complex, more implementation-dependent and no more useful for applications outside that narrow area. This is a bad tradeoff.

I'd really like to be able to define syntax that helps me use a particular library as macros, though, as one might do in Scheme, Dylan, or what have you.

The general case of this is domain-specific languages and language embedding. For example, theorem-provers and proof assistants need to do this. For those sorts of things, a good syntax extension mechanism certainly has its place. But it really takes a lot of work to make it clean. Scheme's macros are better than most in this respect because they are hygienic, but I think they are still quite limited.

Anyway, I have found that combinator libraries in Haskell give you most of the benefits that syntax extensions would, plus many more because you can still use the facilities of Haskell itself to do abstraction. A good example is parser combinators: they are far more powerful than regular expressions, plus you can write new combinators, give them a name and pretend they are primitive; you can even create parsers dynamically. Here at Utrecht, we have two parser combinator libraries (Parsec and UU_Parsing), and they are both quite efficient and fast.

Now, to get really abstract: This pattern of things at the object level being subsumed by constructions at the meta-level is more than a coincidence. It is an instance of a free construction across semantic dimensions which can actually be formalized and, indeed, it is one of the things which forms the foundations for the programming language I am designing.

Can you tell us more about this? Sounds interesting.

Happy to. I started writing a post about it, but it got long and involved. Give me a few days and I will either post something here, or on my web page and post the link.

So while the pointy-headed theorizing is wonderful, and I'm a big fan of it myself, some of what's being said here is classic "ivory tower" - it simply dismisses what's really going on down on the ground, where the people live in mud huts, and those who can work ivory are prohibitively expensive. Besides, the ivory-workers really don't want to be working amongst the mud huts (unless perhaps an IPO is in the offing), and would rather be in that cozy tower with the awe-inspiring 1000-foot view (and tenure).

Hey, thanks for pigeonholing me so quickly.

In my experience, syntax is even more important to "average" programmers than it is to me and other good programmers.

No argument there! The point is not whether it is more important to them, but whether that fact is an obstacle to becoming "above-average" programmers.

And yes, it's "unreasonable to expect a programming professional to know BNF", at least in today's world, and assuming you define a programming professional as someone who's paid to write programs.

No, I was thinking of a `professional' as a person who is competent and in command of his field, not just any bozo who knows how to use Microsoft FrontPage.

I know that there are lots of people paid to be programmers who were never educated as programmers. I worked in industry for five years, at a company (not a dot com IPO, BTW) with about 20 programmers, and of them I was the only person who had a degree in computer science, rather than, say, electrical engineering. I was shocked when I discovered that no one knew what I was talking about when I mentioned `big O' notation or algorithmic complexity.

If we want to turn programming into a discipline analagous to engineering, something which produces reliable products in a regular fashion, and not just by shooting in the dark, then we have to create a professional workforce. The most effective way to create professionals is to educate and train them in an academic setting.

Maybe you find that elitist and unfair to all the people whose God-given right it is to grab a piece of the IT dream but, hey, let's not barricade ourselves in our ivory towers and ignore "The Real World", eh?

If you wanted to build a skyscraper, would you hire some guy just because he knows how to use the latest jackhammer model, or a would you hire a structural engineer, who justifies his designs with detailed diagrams, keeps up on the materials science literature, can prove scientifically that the structure is safe and reliable, and has a degree to show he is qualified?

I'm not defending that state of affairs

You're not trying to change it either, though, are you?

No doubt some of my goals will prove unrealistic, but I will find out which ones they are sooner or later, and perhaps I will make a difference before I die. But it won't be by telling other people what the status quo is.

Also, rolling your own syntax is a long way from being viable, even if it's easy to do technically.

I don't advocate rolling your own syntax. Why would I do that? I've been saying that syntax is largely irrelevant.

On the other hand, if syntax is largely irrelevant, then factoring it out of an application, allowing it to vary, and making it easy to change makes sense. That current tools don't support this is a problem, admittedly, and is one of the things that would need to change to make that feasible. Isn't something like this happening already, with XML?

But it's only been 40-odd years since the first "high level" language was invented, and there's plenty of stuff that still has to trickle. Incontrovertible proof of this can be found in the fact that the entire concept of functional programming has virtually *zero* presence in the commercial world, some few special-case exceptions like Erlang notwithstanding.

A couple of years ago I would have agreed with this, but not anymore. Several archetypal features of functional programming are becoming more and more widespread. Both garbage collection and closures (or some semblance thereof) are available in Java, Perl, Python and Ruby. (I think C# must have GC too.) Python's latest version has adopted lexical scoping. Java will probably be retrofitted with parametric polymorphism soon. Programs in these languages still don't look like FP programs, but you can hardly say there has been no effect, and with this trend, who knows what will happen in the next few years?

Ocaml seems to be making substantial inroads in the commercial community. I notice one of the regular posters to the caml-list now is a fairly well-known games programmer using Ocaml for his current project.

Scheme is being taught very successfully to high school students in the Rice PLT program.

While I'm rambling, I'll add one more thing: languages like Perl have something to add to the mix here, in terms of languages that work the way non-mathematical humans expect.

Not that old saw again. The principle of least astonishment? Hasn't every language user community cornered the market on this one by now? I seem to get conflicting reports. Here is my response:

Perl works the way Perl users expect it, because they only expect what it does.

You need to be surprised at least once to get extra expressive power from a language. This is pretty obvious if you think about it. If something doesn't surprise you, then you probably knew it already.

Probably people who had only ever used C and then learned about Perl/Python/Ruby were surprised the first time they learned that you could execute scripts interactively, that there is such a thing as garbage collection, what a regular expression is, etc. But I strongly suspect that the rest of Perl/Python/Ruby is just abbreviations for stuff they already knew.

BTW, since I'm on the subject, I think that the most significant difference between Perl, Python and Ruby is the syntax. (I think I would like to retract the following two sentences.) Do you want proof? The Parrot project is planning to use the same bytecode machine to run all of them. The reason there is so much mobility between the user communities of these languages is that the underlying semantics, divorced from the syntax, is basically the same.

I think that notations are tools for thought. Designing good notations can be extremely difficult. Part of language design is syntax design, even though this part is less exciting than working on the semantics.

Most programmers want the whole package: useful syntax and "correct" semantics.

I agree that in principle the domain specific syntax could be built using some syntax extension menchanism (i.e., macros), just as domain specific semantics can come from library routines.

In practice, however, most libraries for commonly used languages don't come with syntax abstractions (mostly because language like C don't provide the required mechanisms), so programmers looking for the right syntax for a given job must learn an appropriate language.

Sad but true.

There's another side to this. Outside the scope of Lisp-like language. syntax extension tends to cause more harm than good. Different syntaxes don't compose well, and software readability in general deteriorate.

This, of course, speaks loudly for Lisp-like syntax. Ufortunately, most programmers - even educated ones - find it rather hard to use.

Hey, thanks for pigeonholing me so quickly.

What I didn't make clear in my message, but should have, is that I agreed strongly with most of what you said - about e.g. built-in datatypes, solving problems once and for all, hacks breeding hacks, etc., because it mirrors my experience. Meta-things are more interesting than the things themselves, because once you've solved the meta-things, the things themselves become trivial instantiations. But in the commercial world, money is often made and jobs kept secure by solving the same problem over and over without meta-tools. It was my agreement with most of your points that led me to respond to the rest.

I'll respond to the rest of your message later...

What I didn't make clear in my message, but should have, is that I agreed strongly with most of what you said [..]

OK, glad to hear it, truly.

But in the commercial world, money is often made and jobs kept secure by solving the same problem over and over without meta-tools.

Which is part of the reason I got out of that place! Bryn put it pretty well. We are a bunch of money-mongering masochists:

People will subject themselves to all sorts of abuse from computers, and programmers have a much higher tolerance for this pain than normal folks. But I think we owe it to ourselves to improve our lot.

Amen, brother. :)

One of the selling points MS is using in trying to push .NET over Java is multi-language support. The argument generally goes something like: "they have multi-platform support but we have multi-language support (and anyway we also have multi-platform support: Windows XP, Windows 2000, ... :-)

I've always had a bit of a problem with this multi-lang support issue. All the .NET langs that MS provides compile to the same byte-code, use the same data types, use the same library funcs, etc. Indeed, it has always seemed to me that all these languages are the same language (C#). Put in the context of this thread: they differ on syntax but have the same semantics

MS seems, therefore, to think that syntax is as important as semantics, at least from a marketting perspective. I disagree, and so apperently do many "professional" programmers: see the uproar about VB.NET. While preserving most of VB's syntax, it changed the semantics, resulting in a language that was just too different for the average Joe VB programmer.

The only scenarios where different langs on .NET seem to have any type of advantage over simply using C# for everything is when they break out of the .NET common-type mold, e.g., unmanaged C++, interpretted JavaScript. I'm still waiting to see how the .NET versions of Scheme, Haskell and Smalltalk come out. My bet, however, is that 3 years from now everybody using .NET will be using C#

[Ehud writes:] I agree with you on almost all points, but I think my perspective is a bit different. I separate the issue of language expressiveness from the daya to day usability.

Since I have probably already demonstrated that I am a hardliner on these issues, let me go the last mile and tentatively suggest a really heretical idea: maybe we shouldn't be using programming languages on a day-to-day basis at all! Maybe we should be sitting at a desk, thinking hard and planning out (using the semantics!) what we are going to do, and then, at the end of the week, when we are comfortable with our plans, translate all of that into the syntax of the programming language when we sit down in front of a keyboard.

Is this different from waterfall software engineering, or one of the other SE methodologies? Maybe not, but what I'm imagining now is a language with a formal semantics, and a programmer as someone who spends 90% of his time doing mathematics using that semantics, and only 10% coding. To me this is distinct from SE methodologies, where you are just mucking around with diagrams that have only a tenuous connection with any specific language's semantics.

There's another side to this. Outside the scope of Lisp-like language. syntax extension tends to cause more harm than good. Different syntaxes don't compose well, and software readability in general deteriorate.

"Different syntaxes don't compose well" popped out to me. At first, I thought I agreed strongly with this, but then I realized it's more accurate to say: "syntax recognizers (parsers) don't compose well." (Maybe you didn't mean that, though.) But even that isn't accurate: parser combinators compose well, by definition. Perhaps this can be exploited for syntax extensions? Here in my group, Doaitse Swierstra and Arthur Baars have been experimenting with using the UU_Parsing library to implement syntax macros.

This, of course, speaks loudly for Lisp-like syntax.

Does it? Again, I agreed with this at first, but a moment's consideration makes me doubt it. LISP-like syntax is just a subset of ML-like syntax, since I can always fully parenthesize an ML program. So you could have a LISP-like macro system in language with ML-like syntax, if you are willing to put up with that. The problem is how to extend that sort of system to allow not just LL(1) grammars, but LR(1) (or at least LALR) grammars, but that does not obviate my point.

Frankly, I think the limited syntax extensions possible in ML and Haskell are too ad hoc. For example, because he must assign a numeric precedence and fixity to each infix operator, the designer of an Haskell module either has to assume that no one will mix his operators with the operators defined in other modules, or he has to carefully pick the precedences and fixities to work well with every other possible module, which is impossible. So, user-defined syntax in ML-like languages sort of requires global knowledge, which is clearly bad. You need a way of defining composable parsers to get around the closed-world assumption.

[Pixel wrote:]

If you're crazy about Perl regexps, use the Perl regexp library.

Alas this is not so simple. In many languages, "\d" and "d" are the same. The result is:

• either you write "\\d", loosing the readability of regexps (which can be hard to read already)
• either you preprocess your language

There are two other options.

• You can map the backslash to a non-backslash character, dynamically.
• You can represent a regular expression as an abstract datatype, and use constructors or combinators to build it up as an abstract syntax tree.

IIRC, the latter is what Olin Shivers did for his Scheme regexp package. It has the advantages of making regular expressions more readable (though, of course, significantly longer), and disallowing at least some of ill-formed regular expressions (forgetting a closing parenthesis, for example)---though, since Perl's regexps have a lot of strange extensions, I'm not sure if it's feasible to try to eliminate all the possibilities for ill-formedness.

A similar idea has been used by others to handle printf format strings in an abstract, and even extensible, way in typed languages.

Sorry Frank, but I don't think I can have a civilized discussion with someone who cannot refrain from showing off his academic arrogance in every single post (referring to people as bozos, whining about your ex-colleagues not having the right academic degrees, that old skyscraper and "we, the academics, have a God-given right to control people" nonsense, etc).

So I'll stay out of this discussion; shouldn't have entered in the first place.

But before I leave, just let me mention that for me, being able and allowed to read and write computer programs is a basic human right. To quote Bob Frankston, "Just as there weren't going to be enough phone operators, there aren't enough programmers to add all the little bits of intelligent behavior we are going to expect of the infrastructure.". Our society cannot afford to keep programming away from the masses. It's all about empowering people. Every single one of them.

Over and out /F

I don't want LtU to become a second comp.lang.misc. We want informed opinions, and this site is based on linking interesting work, esp. research work. But let's try not to hurt each other's feelings. This doesn't promote fruitful discussion.

Well wonder why i prefer <tt>"c[ad]+r"</tt> over <tt>(: "c" (+ ("ad")) "r")</tt> ! (example from Olin Shivers) Even if people are not very used to BNF grammars, having "+" and "*" prefix does not ease understanding.

And don't get me wrong, i like static checking, but not the unneeded syntax cost. I really like Caml's printf with has static checks, together with expressivity and smooth transition from C (try to i18n without printf and you'll see!)

* You can map the backslash to a non-backslash character, dynamically.

uh? i don't grasp this. What i meant is:

guile> "\d"
"d"

Now i'm not saying perl's regexps are the ultimate choice. But they have proven their expressivity and power.

But before I leave, just let me mention that for me, being able and allowed to read and write computer programs is a basic human right.

Oh yeah, "life, liberty and the pursuit of computer programming". How could I have forgotten that one?

Besides, certainly no one is planning to take away your freedom to read or write programs.

To quote Bob Frankston, "Just as there weren't going to be enough phone operators, there aren't enough programmers to add all the little bits of intelligent behavior we are going to expect of the infrastructure.".

Didn't Frederick Brooks already put to rest the idea that throwing programmers at burgeoning software problems is not a viable solution?

Our society cannot afford to keep programming away from the masses. It's all about empowering people. Every single one of them.

I guess if you believe this, I can see why you react so strongly to what I've said. I see zero connection between political empowerment and programming language design. For me it is about technical empowerment, the ability to write better programs more quickly and easily, not about turning everybody on earth into a programmer. (I should say, a "programming professional," by my earlier definition.)

I do want to increase the transfer of information from academia to industry (and vice versa, but admittedly it's a fairly one-sided proposition for me); that's part of why I post this stuff here, and part of why I teach.

On the other hand, if something has been in the literature for 40 years or so, I don't want to spend my time explaining that, when people are perfectly capable of picking up a book and reading a much better presentation written by someone who was completely focused on that topic.

What is needed is probably to motivate programmers better. Many programmers don't see any value in computer science, because they never see it applied. Peer pressure is also a factor, but I'm a believer in The-Right-Thing winning out, not Worse-Is-Better. :)

Well wonder why i prefer <tt>"c[ad]+r"</tt> over <tt>(: "c" (+ ("ad")) "r")</tt> ! (example from Olin Shivers) Even if people are not very used to BNF grammars, having "+" and "*" prefix does not ease understanding.

To reply to your second remark first, if you are already a Scheme programmer, you are not going to have trouble with prefix syntax.

As for brevity, if your program is a big one, the extra verbosity is not going to matter much. If your big program has lots and lots of regexps, than the readability will certainly be improved. Also, you will be able to name and combine regexps, which is a win. Only if your program is really small will the sexp form increase the size of it by a non-negligible percentage.

You can map the backslash to a non-backslash character, dynamically.

uh? i don't grasp this.

Pick a character like, say, %, and write things like "^%(%s%)$" instead of "^\\(\\s\\)$". Your interface to the regexp library hides the functions from the regexp library, takes strings of the first form, maps % to a backslash, and passes it to the regexp library functions.

Snide remarks are really not helpful.

Though I have an academic bent, I think many (even in academia) have similar views. We may not all be interested in them, but they are worth discussing.

For example, we had quite a few discussions on end-user programming. Python fans have CP4E.

Obviously the issues here are not the venerable concepts of syntax and semantics, but rather more cognitive science issues. See work on PBD, for examples.

Indeed, one should first ask whether end-user programming, or CP4E (as a goal) are of interest. And each is entitled to his own views (I am with Fredrik on this one..)

But from a PL research point of view, I think these questions are well worth our research efforts.

It is simply wrong to dismiss the cognitive aspects of language. One may find these less interesting than formal models, or not, but they exist none the less.

referring to people as bozos,

(BTW, you may not realize it, since I suspect you are not a native English speaker, but "bozo" is not a very harsh insult.)

Do you deny that there are many programmers working in industry who overvalue their expertise? I see stories about prima donna programmers all the time. (I, uh, cough, probably fit that description myself when I was industry, but I no longer consider myself an exceptional programmer.)

It is interesting that you call me arrogant, because I detect an unusually large amount of arrogance in the programming community. You just have to read Slashdot to encounter this phenomenon. There is a huge amount of peer pressure.

I became infected with this myself, and now I find it hinders me; as a scientist, I have to actually back up my claims with proof, but I often find myself sliding into faux-hacker styles of speech which obscure the truth. For example, my supervisor once caught me out when I said, "The solution to this problem is of course...", where I should have said "*A* solution to this..."

Programmer culture seems to be suffused with this sort of thing. I see all sorts of loaded phrases bandied about and accepted with little criticism. For example, the phrase "Information wants to be free": it's totally meaningless, but people use it all the time as if it warranted their opinions. What gets communicated by that phrase is not a fact about the nature of information, but rather a political opinion---one I agree with, by the way, but one that I think is presented dishonestly in this manner.

I once thought about writing a paper on this phenomenon, because I got so sick of it. The idea was to compare hacker culture with the American myth of the rugged pioneer, or the lone gunfighter.

whining about your ex-colleagues not having the right academic degrees,

It was not a complaint, it was an earnest expression of surprise. When I was in college, before I became employed, I really thought that to get a programming job you had to have a degree in computer science, just like you need an engineering degree to get a job as an engineer, and that the stories you hear about whiz kids making it big in industry because they spent their childhood programming at home was just something the media exaggerated to make stories interesting. Well, I think those are exaggerations, but it is not an exaggeration to say that in today's world anybody with a modicum of computer expertise can get a job as an entry-level programmer.

that old skyscraper

How would you approach the skyscraper problem? Or do you not see any problem there?

and "we, the academics, have a God-given right to control people" nonsense, etc

Now you're putting words in my mouth.

Though I have an academic bent, I think many (even in academia) have similar views. We may not all be interested in them, but they are worth discussing.

I also hold political views about programming, but insisting that "the right to program" is an inalienable right is just too absurd for me. There are a thousand things I would put first: life, liberty, the pursuit of happiness, food, access to health care, housing, companionship, etc. To jump straight to something so specific and narrow suggests a grievous misperception of what 50% or more of the world really needs and wants, e.g., food.

As for CP4E, I am all in favor of it! I'm just not in favor of teaching people Python. I already mentioned the Rice PLT courses in Scheme; I think they are fantastic, and I would love to see a similar sort of effort with an ML-like language.

I was thinking the other day, which language would I choose these days, since I like to revisit this question every couply of months. I really don't like the ML environments that I saw, but I know I must look at Ocaml one of these days. However, ML functors are really cool...

Haskell is pretty, but I think lazyness is problematic. So are monads.

So should we stick with Scheme?

My SE students complain that knowing Ada isn't going to land them a job. Now convince them to learn ML...

Monads are great, though their treatment in Haskell is marred by complications in the type class system. What is it you don't like about monads?

Scheme is a pretty good choice, I think. There is something to be said for teaching an untyped language to beginners first, not least because it lets them experience firsthand the problems which typed languages address (and don't address, though of course I think static typing is a big win overall).

It is not so much that I am against this in principle, I am just sure this isn't going to work (at least not in general). Complexity is going to come and bite you in the a.. Testing and playing with a working system really helps, in my experience.

I wasn't thinking about composing parsers, though it is part of the problem. I was thinking in terms of defining coorect semantics, and in terms of readability. When you think about constructs that cross experssion boundaries (things like new block strcutres etc.) and constructs that allow nesting, things can get pretty subtle. That's why they are usualyl left to language desginers (who may be using syntax extension mechanisms, of course, but who are much more sophisticated in dealing with such issues).

You need a way of defining composable parsers to get around the closed-world assumption

And composable syntaxes. So do you have any suggestions for tackling this?

It is not so much that I am against this in principle, I am just sure this isn't going to work (at least not in general). Complexity is going to come and bite you in the a.. Testing and playing with a working system really helps, in my experience.

There is that. Perhaps what I am trying to say is that we just need to spend less time coding, take a step back, and spend more time thinking about what we need to code, what we don't, and why or why not. There is something unnatural about spending the entire day in front of your computer, and nature is showing many of us nowadays (via wrist, arm, back and eye problems) that we just aren't built for it.

Well, nothing too profound there, I guess, and the part about nature will improve as user interfaces improve, and ubiquitous computing gains momentum.

And composable syntaxes.

Well, composable parsers induce composable syntaxes, don't they? Otherwise, I don't understand.

So do you have any suggestions for tackling this?

Not at the moment. I still have to experiment more with Doaitse's UU parsers and his attribute grammar system. But my experience with them is that, once you write the grammar, it is pretty easy to see what it recognizes, and it ensures that you can't write ambiguous grammars. I suspect the upshot is that the parsers encourage writing grammars which are fairly easy to reason about, for reasons analagous to how referential transparency makes it easy to reason about functional programs in general. It's a relative thing, though; the UU parsers still take some getting used to.

On the other hand, if syntax is largely irrelevant, then factoring it out of an application, allowing it to vary, and making it easy to change makes sense. That current tools don't support this is a problem, admittedly, and is one of the things that would need to change to make that feasible.

Agreed.

But it's only been 40-odd years since the first "high level" language was invented, and there's plenty of stuff that still has to trickle. Incontrovertible proof of this can be found in the fact that the entire concept of functional programming has virtually *zero* presence in the commercial world, some few special-case exceptions like Erlang notwithstanding.

A couple of years ago I would have agreed with this, but not anymore. Several archetypal features of functional programming are becoming more and more widespread. Both garbage collection and closures (or some semblance thereof) are available in Java, Perl, Python and Ruby. (I think C# must have GC too.) Python's latest version has adopted lexical scoping. Java will probably be retrofitted with parametric polymorphism soon. Programs in these languages still don't look like FP programs, but you can hardly say there has been no effect, and with this trend, who knows what will happen in the next few years?

Ah, but you have to look at how the features you mention got to where they are now. Garbage collection and closures were in Smalltalk, but it pretty much adopted those both from Scheme/Lisp. But it was Smalltalk that had the greatest influence on the mainstream languages, because the mainstream picked up on the idea of object-orientation, rather than functional programming, which was still in its infancy at the time. Smalltalk had these features in the early seventies - Java was only released in 1995 or so. So we have a twenty-odd year trickle down for garbage collection. Java still doesn't have proper closures, although if you're a masochist you can pretend that inner classes are closures if the variable scoping happens to work in your favor for what you want to do. The other languages that really do support closures are all the "lightweight languages" that tend to be fairly nimble about adopting new features. Of course, that was the point of LL1; these languages have a chance of becoming a model for the adoption of advanced academic features, which other languages will be forced to follow. So I think that this trend, combined with "Internet time", may increase the speed of propagation in language features in future; but there's no obvious example of that having happened to a great degree yet. All of the features you mentioned, that are actually in a major language today, have taken over twenty years to reach those languages.

Ocaml seems to be making substantial inroads in the commercial community. I notice one of the regular posters to the caml-list now is a fairly well-known games programmer using Ocaml for his current project.

Scheme is being taught very successfully to high school students in the Rice PLT program.

These are all positive things, but if you ask the question what percentage of commercial programming is done in functional languages, then my characterization of "virtually *zero*" is accurate.

While I'm rambling, I'll add one more thing: languages like Perl have something to add to the mix here, in terms of languages that work the way non-mathematical humans expect.

Not that old saw again. The principle of least astonishment? Hasn't every language user community cornered the market on this one by now?

Actually, I didn't mean least astonishment, although that's how I said it. What I mean is this: most humans do not think mathematically. At all. In fact, mathematics is an incredibly foreign discipline, to most humans. Those of us who can think mathematically have effectively trained our messy neural networks like a topiary bush - all neatly arranged, trimmed, and coaxed into particular shapes. But most people simply have a tangle of brambles, and that tangle seems to be more comfortable dealing with languages like Perl than languages like ML. Not because of the principle of least surprise, but perhaps because of the principle of least exposed mathematics. It reminds of Stephen Hawking's publisher's advice about how the inclusion of formulae in his book would drive down sales. ML doesn't "sell" to the general programming public because it has "too many formulae": as simple as that. I predict this will never change.

Before you react to the above claim, read this: we're talking about two vastly different communities here. On the one hand, the subject of professional engineers has been raised. Professional engineers should certainly be trained, know BNF, and a boatload of other stuff that most software engineers today do not know. Many of these future engineers may very well use and enjoy functional languages. I'm sure that'll start happening as soon as universities figure out what they should be teaching such engineers; right now, it's a hodge-podge, and whether software engineering even exists at the moment, as an academic discipline taught to students, is debatable. But still, trained engineers is a worthy goal, and ours is a complex discipline, so it's going to take some time, and we should expect that.

Then there's the other community: the self-trained or minimally trained community of people who just want to get a job done, or just enjoy playing around with software, and who may not even be aware of the vast storehouse of academic information out there (perhaps they were put off by academics when someone called them a "bozo" ;) This community will always exist, and will always be bigger than the community of trained engineers.

The problem is this: today, the two communities I've described overlap to such a degree that they are barely distinguishable, and there are many more of the untrained type than otherwise. Examining academic credentials isn't good enough: I know people with bachelor's and master's degrees in computer science who, while certainly better than average, and good programmers, don't live up to the standards which I think "software engineers" should. This isn't their fault, but rather a function of the fact that as I mentioned, software engineering as a discipline is hardly being taught, yet.

So until a vast corps of competent and well-trained software engineers arises (and I'm not holding my breath), what those of us out here in the trenches have to work with is people who do not think mathematically, for whom social and economic considerations are far more important when choosing a language than technical considerations, etc.

It is these people that I'm mainly thinking of (although not only them), when I say that there's a human side of programming languages which is currently under-addressed, that I think languages like Perl (and I'm just picking Perl as a canonical example) have perhaps begun to address, in a perhaps limited and precursory way. Languages that make it easy to deal with basic data types like text, for example, interspersing and embedding it in a program, and building it into the language's syntax. Some functional languages have a list comprehension feature; Perl has a limited sort of text comprehension.

I'm not being very concrete, partly because I'm trying to identify something that is currently missing, and I don't know its exact shape. But a good start would be better support for complex datatypes like text, better support for integration of multiple languages - including natural language - in the same "source" file, better support for polymorphism of operators on types, etc. etc. You can find some of these features scattered around in various languages, but they don't exist in any single accessible language today.

BTW, since I'm on the subject, I think that the most significant difference between Perl, Python and Ruby is the syntax. (I think I would like to retract the following two sentences.) Do you want proof? The Parrot project is planning to use the same bytecode machine to run all of them. The reason there is so much mobility between the user communities of these languages is that the underlying semantics, divorced from the syntax, is basically the same.

I think what you're describing is what will make us both happy: mainstream languages will get increasingly sophisticated cores with a great deal in common: runtimes like JVM, .NET, Parrot etc. These will become more functionally-oriented over time (because, after all, lambda is the ultimate!) On top of these cores, a variety of languages will continue to bloom and cross-pollinate, some of them oriented towards the inexperienced, and others more rigorous. There can still be semantic differences between these languages: you only have to look at all the languages implemented on top of the JVM for proof of this. However, they are more likely to be able to interoperate with other, semantically different languages, if they share a common, high-level core, as opposed to just being compiled to the same machine code.

The UU parsers are available from the UU Software Technology group software page. They were developed by Doaitse Swierstra.

But it was Smalltalk that had the greatest influence on the mainstream languages,

Eh? Despite the facts that both C and C++ have neither garbage collection nor closures, Stroustrup claims Simula was his greatest influence, and Java was principally designed by a well-known LISPer?

Smalltalk has had a big influence on object-oriented languages in research circles, I will surely grant you, but I don't see much of an effect in industry.

because the mainstream picked up on the idea of object-orientation, rather than functional programming, which was still in its infancy at the time.

The mainstream picked up on OO. No argument.

But I think FP was more mature than OO was. LISP was born in 1959. (In fact, as a topic of formal research, OO is still far less mature.)

Those of us who can think mathematically have effectively trained our messy neural networks like a topiary bush - all neatly arranged, trimmed, and coaxed into particular shapes. But most people simply have a tangle of brambles, and that tangle seems to be more comfortable dealing with languages like Perl than languages like ML.

I still don't buy this argument, and I will give you not one, but six reasons for my skepticism.

First, we are not born thinking in Perl anymore than we are born thinking in ML. I really do not buy this argument that Perl is somehow magically better suited to non-mathematicians than ML is. I think the converse is true (mathematically-trained people will have an easier time with ML), but the average person probably has at least as much difficulty grasping assignment, imperative variables and for-loops as he has grasping lets and recursion. They are both initially alien, and this is why your argument re: Stephen Hawking does not hold water. (I teach Java, remember? I haven't yet taught FP to a group of non-programmers so I can't compare, but I know that imperative features and iteration are a real sticking point for my Java students, none of whom are CS majors, BTW.)

Second, maybe you mean that the oft-touted similarity of Perl syntax to "natural language" syntax, and not imperative programming in general, makes it easier for people to learn it. My responses are: a) I have yet to see any objective evidence of this from, say, a linguist who actually conducted a scientific study on the matter. All I ever hear are vague, unconvincing claims about context-sensitivity. b) A virtue of natural language is that it is ambiguous. But programmers, even beginning programmers, probably do not want their programs to interpret their instructions ambiguously. It seems this would only make it harder to produce working programs.

Third, comparing ML to Perl is like comparing a grand-size T-bone steak to a quarter pounder at McDonald's. Of course it will take you longer to eat the first: there is more to eat! ML is simply more sophisticated than Perl. There is more information in an ML program than there is in an equivalent Perl program. There are more structures to learn, because there is more power to be had. There are more concepts to learn, because more is possible. If you restricted an ML-like language to some subset which was roughly equivalent to Perl, and you added to it a library of functions which equate with what Perl has built-in, I am fairly confident that it would be easier to teach and use than Perl.

Fourth, you present mathematics as though it were some forbidden mystery which is only whispered in low tones at secret meetings of the illuminati. Of course that is far from the truth. Everybody who graduates high school has learned arithmetic, algebra, some functional analysis and probably some differential and integral calculus. In college, they may learn more. The ideas which ML is based on, are very closed related to the ones they learn in algebra. Even the syntax is in many cases similar (modulo the limitations of ASCII). Perl, I think, is the more alien for non-programmers.

For experienced programmers, OTOH, I agree Perl is far more familiar.

Fifth, the success of the TeachScheme! Project (which I incorrectly referred to as the "Rice/PLT Scheme teaching project" more than once above) suggests that beginners can learn FP, and from what I've seen, they get more out of it than they do being taught C++ or Pascal. (And they have the standardized test scores to prove it.) That doesn't say anything against Perl but, thankfully, Perl is not being taught in American schools yet. :)

Sixth, just on the merits of the language, FP must be easier to teach because you can formalize exactly what the reduction rules are, show how the program changes during evaluation, etc. Perl, OTOH, is an implementation-dependent mess, full of poorly specified, ad hoc, irregular behavior. I just can't believe that Perl would be the more easily grasped.

Before you react to the above claim, read this: we're talking about two vastly different communities here. On the one hand, the subject of professional engineers has been raised. [..] Then there's the other community: the self-trained or minimally trained community of people who just want to get a job done, or just enjoy playing around with software, and who may not even be aware of the vast storehouse of academic information out there (perhaps they were put off by academics when someone called them a "bozo" ;)

OK, OK, I retract "bozo".

To all the bozos who were offended by my remark: I apologize. Anton is right: you aren't bozos, you're hobbyists. ;)

Seriously though, your model appeals to me.

The problem is this: today, the two communities I've described overlap to such a degree that they are barely distinguishable, and there are many more of the untrained type than otherwise.

Agreed, I think.

It is these people that I'm mainly thinking of (although not only them), when I say that there's a human side of programming languages which is currently under-addressed, that I think languages like Perl (and I'm just picking Perl as a canonical example) have perhaps begun to address, in a perhaps limited and precursory way.

Hm, that is an interesting perspective, which evokes some sympathy in me. However, I am not sure I understand the consequences of this.

So are you saying it is an apples-and-oranges thing, and I am picking fights with the wrong people? Or are you saying that academics should dumb down their research to make it more accessible? Or are you saying perhaps that for each "abstract" programming language, there should be one elegant, streamlined, general version for the engineers, and one redundant, sugarified, specialized version for the hobbyists ("programmers?" what is a good name?)---sort of a domain-specific language, if you will.

BTW, as a postscript, let me add that the reason I take liberties with my remarks about "bozos" on this weblog, is that I feel confident that there are none here who fit that description. (No, not even you, Fredrik.)

Fourth, you present mathematics as though it were some forbidden mystery which is only whispered in low tones at secret meetings of the illuminati.

Aha - therein lies the rub! Perhaps you've come across references to "Matlab de Illuminati", a Latin phrase referring to the copy of Matlab used by the Illuminati, to model their secret world-controlling activities? You may be surprised to find that "Matlab de Illuminati" is an exact anagram for "Lambda Ultimate in IL", a clear reference to Ehud's weblog, right down to the country code of his email address!!! Faced with such evidence, do you still foolishly cling to the belief that mere everyday bozos will ever be allowed to learn real mathematics? Why do you think most people forget anything but the most basic algebra when they leave high school? It's the mind rays, of course, beamed from orbiting platforms sent up by the CIA, which, of course, is under Illuminati control!

My more serious response will have to fnord wait until "tomorrow", defined as a function of some timezone in which it isn't already tomorrow right now...

Stroustrup claims Simula was his greatest influence, and Java was principally designed by a well-known LISPer? Smalltalk has had a big influence on object-oriented languages in research circles, I will surely grant you, but I don't see much of an effect in industry.

I do think it fairly safe to say that Smalltalk was very influential in the development of user interfaces, but it's effect on programming language design was not nearly as dramatic.

This is kind of fair given the interests of the Smalltalk developers (esp. Kay).

Wow.

Frank, how about a simple experiment?

For the next few days, assume that every single person you talk to is at least as smart and rational as you are. Don't limit yourself to this site; apply this to everyone you meet.

If they say something you don't agree with, try to figure out what they know that you don't. If they say something that sounds ludicrous, try to figure out if there's another way to interpret what they said that makes more sense. If they appear to act irrationally, try to figure out what piece of information they have access to that you don't.

(and with "figure out", I mean "think, don't ask")

When you've tried this technique on other bozos for a few days, I'm pretty sure you can figure out what I was talking about all by yourself.

I spent a lot of time trying to run this site for the benefit of all of us. Please don't ruin it.

I have gotten a lot more out of Anton's posts than yours. About all I get from you is that you feel the same contempt for me that you claim I hold for you.

Ironic, eh?

Let me just add the standard disclaimer, then.

It seems that I have ended up dominating this discussion, and, unfortunately because of this whole academia vs. hackers thing, it may appear that I am somehow representing the opinions of PL researchers as a whole. This is certainly false, not least because I am still a novice at research work myself. Ehud is here, and I know there are some other people lurking who evidently possess