Lambda the Ultimate

Maybe a programming language should be backed up by a) a big corporation, b) a shiny development environment, c) an operating system, in order to be widely used.

Well, of course Lisp had all that and it didn't work out (well, I don't know if the Lisp Machine companies were actually big corporations, but in any case Lisp definitely had commercial backing). And on the other hand, Python and Ruby and all those didn't have any of that when they were gaining popularity, and still don't really have a lot of it, and yet they are better off (in terms of number of users) than Lisp.

I really wonder about what it is that makes "the market" go to one language over another. It seems almost random and only very slightly related to the technical merits of a given language.

On the other hand, it may not matter all that much for some purposes if a great language "catches on". I get the feeling (I may be completely wrong...) that it is being considered more acceptable to use whatever the heck language you want. Whereas for a while, everything just had to be in C or C++ and later Java, etc (by the way, did you know that the GNU coding standards, right now, in 2006, recommend that people write their programs in C? sigh...).

Right now OCaml and Common Lisp and Scheme and Standard ML and Haskell may not be extremely popular compared to Java, but that just doesn't matter that much for a project that wants to use one of those languages - they all have good, solid, free implementations that aren't disappearing any time soon.

Well, of course Lisp had all that and it didn't work out

I studied CS around 1993 and I never heard of LISP until I embarqued on a journey to find what other programming languages exist. Maybe you are referring to an older era where LISP machines and the companies that produced them were known.

My co-workers have no idea about LISP; they have heard the name, but they think it is a custom language for AI research.

I really wonder about what it is that makes "the market" go to one language over another. It seems almost random and only very slightly related to the technical merits of a given language.

I wouldn't say it is random. I think it is marketing. That's where the big corporation/operating systems comes in. Microsoft has a proprietary O/S that dominates the desktop which is built on C; it promotes C/C++ as the primary development tools and .NET in the last few years. Sun also makes a proprietary O/S which is based on a "standard", promotes Java and has a whole army of programmers dedicated to bringing the "best SDK ever". LISP, Ruby, Python etc have none of the above.

The Chicken & Egg problem is at work to make certain languages dominate while others remain the dark: people write code for the mainstream programming languages, which in turn makes more people prefer those languages.

On the other hand, it may not matter all that much for some purposes if a great language "catches on". I get the feeling (I may be completely wrong...) that it is being considered more acceptable to use whatever the heck language you want.

Right now OCaml and Common Lisp and Scheme and Standard ML and Haskell may not be extremely popular compared to Java, but that just doesn't matter that much for a project that wants to use one of those languages - they all have good, solid, free implementations that aren't disappearing any time soon.

That's true for personal projects. But how do we persuade managers to use a non-mainstream programming language? who is going to risk his career by proposing LISP for a big web application to their boss, instead of Java? and how do you persuade your fellow programmers that it is better to use a certain language?

There is also another problem: there is a category of applications that are very important for everyone of us and almost everybody uses. An Office suite, for example. But almost all of these programs have many bugs, and those bugs are because the programming languages used are not good PLs. If good programming languages were used, people would not waste their time with apps crashing, losing work, deleting temp files, have security problems, etc.

So I do not agree with the attitude "it does not matter if good languages are not widely adopted". I think it matters.

I studied CS around 1993 and I never heard of LISP until I embarqued on a journey to find what other programming languages exist. Maybe you are referring to an older era where LISP machines and the companies that produced them were known.

Well, yes, I'm talking about a few years earlier than that. And that's my point. Lisp had commercial backing, fancy development environments, and an OS supporting it... and then a few years later you studied CS and didn't even hear about Lisp. So Lisp had the 3 things you said were necessary, and failed anyway.

But how do we persuade managers to use a non-mainstream programming language? who is going to risk his career by proposing LISP for a big web application to their boss, instead of Java? and how do you persuade your fellow programmers that it is better to use a certain language?

Maybe this is my lack of industry experience, but I don't really see the difference between Java and Lisp here. Now, yes. Java is established. Lisp is not. Fine. But 10+ years ago, Java was being adopted by people who hadn't used it previously. People were choosing to use a non-mainstream language. So why were they choosing a technically inferior language? Just marketing? Are people really that gullible?

And more to the point, why did the people who chose those languages to be the ones to promote choose those languages? As far as I can tell James Gosling is technically competent and has experience with lots of languages... so why did he create Java of all things? And then why did Sun decide, hey, let's promote this as the great language for everybody to use? There are so many steps in the process even before it gets to the level of being marketed where it's hard for me to see what people were thinking and why they made the decisions they did.

But almost all of these programs have many bugs, and those bugs are because the programming languages used are not good PLs. If good programming languages were used, people would not waste their time with apps crashing, losing work, deleting temp files, have security problems, etc.

I certainly think using a good language is important and that many products would be better if people didn't make astoundingly bad decisions in this department, but... I wouldn't say they are the solution to all bugs.

So I do not agree with the attitude "it does not matter if good languages are not widely adopted". I think it matters.

Well. What I said was, "it may not matter all that much for some purposes."

Java was initially being marketed at problems that the existing big languages (C, C++ and to a lesser extent things like VB) couldn't tackle at all. That's a bit different to marketing a language without an existing user base at an existing niche, or for general purpose use.

Well, yes, I'm talking about a few years earlier than that. And that's my point. Lisp had commercial backing, fancy development environments, and an OS supporting it... and then a few years later you studied CS and didn't even hear about Lisp. So Lisp had the 3 things you said were necessary, and failed anyway.

LISP did not have the 3 things I said.

There was not a commercial O/S running on PCs and written in LISP (as far as I know). There were expensive LISP machines though. If there was something like Windows 3.0 but made with LISP, the situation would be different.

What companies were there to support LISP with applications and IDEs? C had Microsoft and the Unix world. Very big companies were behind C, as very big companies are behind Java (Sun, IBM).

For the fancy development environments, I have no idea.

So it is not that LISP had all 3 but failed. LISP maybe had 1 (the fancy dev envs) but it seriously lacked in the other two departments.

So why were they choosing a technically inferior language? Just marketing? Are people really that gullible?

Yes, it's all about marketing. In the early 90s, the buzz was around C++. When C++ failed to deliver, Java came in and promised to solve the problems C++ had. And people adopted Java as a better C++!

As far as I can tell James Gosling is technically competent and has experience with lots of languages... so why did he create Java of all things?

Maybe because he wanted his own creation to be the one that is the most popular and/or saw that people would like a C++ without the problems. Plus LISP was divided and in a not very good state so he did not want to participate in a barrage of politics. There can be many reasons.

And then why did Sun decide, hey, let's promote this as the great language for everybody to use?

That's a decision based on economics. Basically for the same reason Microsoft did not improve C++ but instead chose to promote Visual Basic.

Lisp, Lisp machines and Lisp environments went down because of PCs rise. They were a separate environment, a different world while the rest of the commercial world shifted from mainframes to PCs.

There were top IDEs for Lisp back then that were completely integrated to the whole system environment like even Eclipse or NetBeans can't really lay claim. There were commercial companies with Lisp-based products -- even Autodesk with AutoCAD's AutoLisp -- and even Lisp-based machines. This is far more than even java today.

And btw, back when Lisp was king, Microsoft was just a small company trying its luck in the market. We're talking pre-PC era here and about mid 1980's Lisp market was a small niche...

LISP, Ruby, Python etc have none of the above.

So why is it that Ruby and Python caught on? Or PHP, Perl, TCL, etc..? What makes those better than Lisp and Scheme? They were not commercially backed; they sure didn't have good IDEs; their implementations have always been rather slowish (when compared to good Lisps or Schemes). So what, are they really tecnically better somehow? - and at what, exactly?

Well,

1. Python only caught on quite recently really. It had been around for over 10 years before it got a lot of traction on Linux and kind of spread from there (I remember using it a few years ago on Windows for embedded scripting but that was exotic for the time)
2. Ruby basically didn't catch on until RoR came along. It was thought of as "nice but we have Python for that...." then the 'killer framework' came and off it went
3. PHP was designed to solve a specific problem and has lots of domain specific stuff in there specifically for creating web apps.

Why these languages rather than a Lisp? Familiar syntax helps, excellent tutorials, open communities, readily identifiable leaders, great standard libraries, domain specific features etc ... Lisp has a lot of cute stuff but if I have to handle HTTP all the macros in the world won't be faster than a pre-written documented HTTP library.

Familiar syntax helps

IMHO, Python and Ruby aren't exactly "familiar". At least no more familiar than Lisp. As for the rest:

• excellent tutorials
• open communities
• readily identifiable leaders
• great standard libraries
• domain specific features

Lisp has a lot of cute stuff but if I have to handle HTTP all the macros in the world won't be faster than a pre-written documented HTTP library

Did you mean something like this, or like this?

Disclaimer: I'm not a Lisp user. In fact, I've never written a line of Common Lisp in my life (the closest I've come is a few small Scheme programs in undergrad). The links above were collected in a few minutes worth of Google trawling. I just don't like seeing the lack of Lisp popularity being attributed to things that are in many cases myths. I'd rather understand what really prevented Lisp from taking off.

Python & Ruby are certainly much more familiar to the average programmer than throwing a great big mass of cons-chain s-exps at them and telling them it's a program. It may all be a parse-tree under the hood, but syntax means you don't have to evaluate most of the program in your head just to comprehend its structure.

I hear (well, read) some folk's saying "if you had a good enough editor", like Emacs, for example, then that wouldn't be so hard, but as a rabid convert to functional languages (Haskell, especially) outside of my day-job, I for one would far rather code in C than wade through a morass of parenthetic expressions to find what syntactic structure would show me instantly and clearly. Just as "Assembler may be wicked-fast, but in the end it's still assembler", Lisp may be wicked powerful, but in the end it's still Lisp. With their lack of macroscopic structure, both are light-years from what anyone weaned on Oh, Pascal would consider remotely familiar.

I hear (well, read) some folk's saying "if you had a good enough editor", like Emacs, for example, then that wouldn't be so hard

For writing Lisp, the only thing that really makes a difference is parentheses matching. Hopefully your editor can do that. For reading, normal syntax highlighting should make it easier, but it's not that hard. More advanced features like code-browsing things etc. are not needed due to the syntax; at that level the superficial syntax is irrelevant.

but as a rabid convert to functional languages (Haskell, especially) outside of my day-job, I for one would far rather code in C than wade through a morass of parenthetic expressions to find what syntactic structure would show me instantly and clearly.

What does being a "convert to functional languages" have to do with this? Haskell is my current favorite language, and has been for a couple years now and I have no trouble with Lisp's syntax (and definitely would prefer to deal with Lisp than C).

[reordered]

but syntax means you don't have to evaluate most of the program in your head just to comprehend its structure.

With their lack of macroscopic structure, both are light-years from what anyone weaned on Oh, Pascal would consider remotely familiar.

What are you talking about (w.r.t. Lisp)?

What are you talking about (w.r.t. Lisp)?

... is that with Lisp, everything LOOKS the same. Great big masses of parenthetic expressions. I'm sure for many people that's fine, but to say that a language made of words, numbers and parenthesis is as familiar to most programmers as a language whose syntax is structured doesn't make sense to me. The text of Ruby & Python (which was the point I was replying to), much like C, Algol &c., even if not formally block-structured, have visual structure to them that makes them easy to read & find one's way through it. Lisp (like assembler) code all LOOKS the same. It's a strictly visual concern, but enough of one that every time I think of wading into Lisp programming, I invariably go "blehh" and return to a language that has textual structure, even if the particulars of the structure aren't what I'm accostomed to.

What does being a "convert to functional languages" have to do with this? Haskell is my current favorite language, and has been for a couple years now and I have no trouble with Lisp's syntax (and definitely would prefer to deal with Lisp than C).

What it has to do with is that I love functional languages, but would rather deal in low-level imperative coding than program in a shapeless language. That'd be a loss to me, but less of one than dealing with Lisp. Not a statement against Lisp's semantics, power, or anything else, just against the way that I like to read & write code, which's strongly informed by my familiarity with syntax. For me (YMMV), syntax is a feature, not a bug.

Can somebody make a Lisp that does something like: use () when things are not quoted, and then use [] instead of '() when things are quoted? I think that kind of syntax improvement would make things more readable to me...

There'd be no trouble doing that within Common Lisp itself via reader macros. My impression is this is not the area most people complain about, but as you said, it might help for you...

I guess this may really be just purely a matter of taste, end of story.

For me, Lisp's simple syntax was not a barrier in the slightest, and eventually became a huge advantage. I really think Lisp macros (especially Scheme's hygienic macros, although I don't think they are quite perfect) are a HUGE win. I am using OCaml right now, and I really like it, but... It actually has syntax extension facilities in the form of Camlp4, but I haven't learned how to use it (I skimmed the manual on it but I don't quite get it yet), and it's just not the same.

Is this:

(if (some-predicate a)
(do-something-1 b)
(do-something-2 c))

Really hard to read for you (other than the fact that I don't know how to make this preserve my indentation)? I dunno. It's fine for me. The parens really do become a non-issue. I like the simplicity and consistency of syntax. I also HATE keywords. They seem so silly. In OCaml I periodically try to use the word "begin" or "end" as an identifier or something. I also hate infix operators. If they are really just functions (and they are), what is the advantage of giving them a different syntax from all other function calls? To me, there is no advantage.

I'll tell you one thing that does slightly grate on me about all Lisps I know... Function calls and syntax are expressed in exactly the same way. I would like a function call to look different. ie, you would say

(apply some-function arg-1 arg-2 arg-3)

And then there would be a Lisp reader extension so that you could sugar that by instead saying

[some-function arg-1 arg-2 arg-3]


Some folks love the ultra-regularity of the syntax; I prefer keywords and punctuation (actually, I prefer lots of operators, like APL, but with standard keyboards you reach a point where you run out of practical operators and fall back on a few keywords). Totally just a matter of taste. The reason I replied way back up top as I did is that people who're enthralled with the undeniable expressive power of Lisp don't seem to be bothered by the shapelessness of the language -- they "get it", which's fine, but what they don't often get is that one of the things holding Lisp back is that for lots of folks the shapelessness of the language is distasteful. They're the ones making up acronyms involving Lisp and parentheses. Like in your example code, sure, it's an "if", and it's interpretable, but, the if doesn't look any different than anything else -- until you read the whole thing through, how do you know it isn't a "let" or a "cond", &c. Vis-a-vis the aegis of this article, it's also the same reason why I think hand-editing of XML really sucks. What I want is a great language that compiles to Lisp but doesn't lose its expressive power, much the same way that I want my document editor to use XML under the hood. Give me the language, not the assembly language it's implemented in. :-)

Camlp4 can be tricky for a Schemer to get a grip on, because it's not just a macro system. It's also an extensible quotation system, and it lets you define new forms of quotation and anti-quotation. Since you have to understand that, too, the hurdle to getting Camlp44 is a little higher.

Most of this long post is about Lisp syntax and some hypothetical options for variation not available any time soon.

The title "What a gyp!" is a joke since gyp is a codename -- my rough synonym from a thesausus for Scheme -- for a kind of Lisp I'm designing to meld with Smalltalk, XML, Python, etc. (But pessimistically assume I won't get beyond design, and all this hypothetical; maybe lowercase gyp is a theory, and capitalized Gyp is an actual future implementation.) So when I say gyp, just think Lisp or Scheme with some changes, especially changes for multi-language integration. Similarly, codename gab means a Smalltalk with similar sorts of changes, such as using gyp parse trees.

raould: Can somebody make a Lisp that does something like: use () when things are not quoted, and then use [] instead of '() when things are quoted?

jimdescu: Lisp (like assembler) code all LOOKS the same.

mike: And then there would be a Lisp reader extension so that you could sugar that by instead saying[some-function arg-1 arg-2 arg-3]

Such syntax is both doable and a good idea with some provisos: once you use a useful bit of unused syntax for something, you can't use it for something else too (unless ambiguity doesn't bother you, and it bothers me). Since there are so few delimitors to go around, they must be reserved for more important semantic indications when a language has them.

Strengths and weaknesses go hand in hand, so every good thing is bad for something else. The uniformity of Lisp can be a weakness when it comes to human perception that likes more variety in patterns for chunking. Lisp code does all look the same -- it's good and bad depending on context.

But Lisp syntax being all the same is one of its strengths, one that most clearly unifies the view code and data as interchangeable. It's instrumental in making macros simple, for example, since macros can be expressed as simple editing of code as data before it gets parsed as code. In most Lisps, 'x is considered a reader macro for (quote x). If you allowed other ways to quote data, you'd want them all to map to the same (quote x) for consistency, and so a view of code as an initial parse tree remains true.

In gyp syntax, I might use square brackets [] to enclose gab syntax, the same way Objective-C uses square brackets to enclose Smalltalk style syntax. But in gab syntax I could not use parens () to revert to enclosing gyp syntax, since Smalltalk uses parens to group expressions. I'd need to use something else to go from gab syntax to embedded gyp syntax.

But neither Lisp nor Smalltalk traditional syntaxes use braces {} for anything, so they could be used in both syntaxes to mean something special and consistent. For example, {[ ]} might mean gab syntax and {( )} might mean gyp syntax. I bet that's had to read at first; but one might learn to see { and say, "Oh, I'm changing from one syntax to another -- what's the next character?" So braces might appear in every type of cross-syntax markup.

(Larry Tesler once said a Smalltalk at Apple used braces for binding multiple lhs variables to multiple value returns from method calls. So this is an example of how individual language implementations have
added non-standard features using otherwise unused syntax in the original Smalltalk. Both gab and gyp could use the same syntax for variable binding of multiple value returns since {} in this context would not contain the other delimitor characters [ or ( meaning a syntax style change. Would that be good?)

Both Lisp and Smalltalk have profoundly simple syntax and grammar, which add to their beauty. Last time I wrote a Smalltalk parser, it still required only a single token lookahead. I don't think many folks appreciate this for what it is: a very unambiguous grammar. Once you see the next token, you know which way you are going in the grammar (or what error you need to generate).

What if you could write code in either Lisp or Smalltalk syntax, using either interchangeably as suitable for a particular place? A Lisp syntax is best when you want to express parse trees more directly (for example, in macros), and Smalltalk syntax might be better for message passing style code. Why not choose strengths and weaknesses of one syntax or another depending on context? Then you can get uniformity when you want it, and avoid it when you don't.

If an environment was willing to translated any syntax to another, you could see the same code from multiple views, and you could translate code for export to another system that only understands one coding style. For example, if you had to deploy in a Python only context, you'd want to be able to convert all your Lisp and Smalltalk style code into that, at least for that deployment.

I hope someday to be in a position that I can generate C++ (and occasionally assembler) from gyp and gab, rather than writing C++ in the traditional and agonizingly slow way. I'd even expect the C++ to be more coherent and manageable than usual. Of course, generating Java would be easier, in comparison. But the need to deploy in a particular language doesn't require that you write all the code (or even most of the code) in the deployment language.

Anyway, gyp would clearly be some kind of Lisp. The question is, would gab be considered some kind of Lisp? It would depend on whether you meant the syntax or the runtime, since the runtime would be the same one used by gyp, and you'd get the same results in either syntax. Runtime differences between Lisp and Smalltalk are mostly unnecessary.

(In the early 90's most of the C++ coders I knew loathed Lisp for the parens alone; the syntax seemed to fill them with revulsion, in a most visceral reaction. Only one guy at Taligent used an interpreter of mine, but since he'd had Scheme at MIT, he considered it no big deal. I'd have pursued my interest in Lisp and Smalltalk hybrids more at the time, but the Dylan folks had the territory all covered, as far as I could see. I was waiting to get my hands on the Dylan they were developing then.)

Thomas Sutton: I though that you were using "gyp" in a pejorative sense (either swindling, or as an offensive noun for gypsy).

I'm glad it's interesting, but it's just an idea. I do spend about all my play-thinking time on this though, and it's the only thing I'm interested in doing for fun these days, so it will probably pick up some reality. I guess I don't hope to reform Lisp's reception by folks who avoid it now.

I think most folks who want to make a language do so in order to please themselves, and I like using Lisp enough to ensure it's a first priority in the mix. I find it easier to think about editing trees in Lisp than in other languages. Hmm, there are probably at least a couple things I'm hoping to get. One is leverage from a scale of flexibility that gives Lisp a reputation for being able to do anything without too much pain.

When anyone asks "Can X be done?" the answer is always, of course. And the demonstration isn't any more horrible than trying to explain continuations. (Yeah, that's a joke.) You can walk someone through a proof by demonstration in Lisp in far less code that in C style languages.

The other important goal is complete replaceability with another syntax, for several reasons, each of which is fairly important, at least to me. I can't order them easily, so I'll start with the least offensive, so I can be politically incorrect last, in the more memorable position. So first, it's important be able to hand an end result to someone in the syntax they are willing to accept. Not being able to do so is limiting, and the idea was to remove some limitations.

Now let me dance around the politically incorrect reason. There are words one might use (but I won't here, just to be amusing) to describe inability in other folks to understand concepts, or inability to be flexible in thinking, or inability to listen without first pre-filtering to narrow standards. Let's not use the word stupid -- that's not nearly subtle enough to cover the complexities involved.

I find this quality in folks to be by far the major barrier to progress in most of my work, that coming up with right answers is not nearly as important to answers being accepted as is being agreeable to psychology and current popular authority (gets a thumbs up from a conservative chap with +1 standard deviation intelligence). In short, good ideas are rejected for random knee jerk reasons, which are highly resistant to education. Actually, education usually incurs blackmarks and will lose you the ear of someone who wants to avoid any more. To a person of near average mind, the difference between interesting and crazy is so thin it's moot. Of course I don't mean you; I mean the guy at work driving me nuts.

Anyway, when someone objects to a syntax, it's simplest to just run the code through a translator and ask, "Well how about this one then?" Syntax is a battle not worth the effort. You lose as soon as you have a struggle with someone. Better to avoid and dodge the early simple rejections and save energy for later more complex things.

The name gyp is self-deprecating since it has a more pejorative sense than anything else in the dictionary. Naming a programming language 'gyp' implies maybe you aren't offering a lot, which sounds safe to me. And how could I turn down a crisp one-syllable three letter word when I found it? Especially one with a 'y' in the middle? Obviously it would make sense to refer to a gyp coder as a gypsy; sounds cool to me. I'd love to hear managers complain about gypsy code.

I realize this is getting away from the whole point of Lisp, but is there some way to have the AST be nice raw Lisp, and then have "views" on top of it? You'd still be able to do things directly to the "pure"ly formatted AST. So, I want to not be easily confused by '() and would rather see [] I can do that (I know, there is a domino effect becuase then you have to figure out what to use for whatever [] used to be used for, etc). Sounds like the old "underneath, our programming languages should be stored as XML" shtick.

That was the original plan for Lisp. S-expressions were supposed to be used for abstract syntax, while M-expressions would be used for actual coding. It just never happened. Apparently because Lisp programmers found S-expressions much more convenient in the long run.

...M-expressions aren't nearly as macro friendly - which would account for why the Lisp community never went that direction.

is there any papers anywhere on how this M-expressions where suposed to lock?

The LISP I Programmers Manual contains many examples of M-expressions and their translation to S-expressions. Pages 56-59 include examples of sequential M-expression programs that include variable assignments, labels, and GOTOs.

Basically I want to use Smalltalk as M-expressions in gab, with S-expressions in gyp. (I'd extend the meaning of S-expr syntax in gyp to allow message-sending style dispatch.) Python would be M-expressions in another parser, etc.

Let Tm mean M-expression parse tree, and Ts mean S-expression parse tree. You want Tm and Ts to be different so debug info is good for source debugging of the actual source. But you want to be able to translate Tm to Ts and have the new parse tree generate the same code when compiled or interpreted.

You also want transducers that can achieve paths like:

gab => Tm => gab /*round trip*/
gab => Tm => Ts => gyp /*translation*/
gab => Tm => Ts => gyp => Ts => Tm => gab /*round trip*/

IMHO, Python and Ruby aren't exactly "familiar". At least no more familiar than Lisp.

There are some ways in which Python, Ruby, etc. are more familiar than Lisp. The two most critical are:

1) Common control structures like "if/else" do not look like function calls. In particular there is no "else" keyword in Lisp.

2) Infix operators for logic and math. Everything is prefix in Lisp.

The above two points make Lisp code hard to read for people like me. With all the parenthesis everything has a sameness to it.

I'd rather understand what really prevented Lisp from taking off.

I don't know about "really", but I can offer my own experience. I was exposed to Lisp as an undergrad in two courses. The first one's purpose was just to teach the language basics. The second was an AI class. I came away with the understanding that Lisp was an "AI language", though I didn't really understand what made it so suitable for AI. In general, I did not enjoy using the language and did not persue it.

Now compare this experience to the one I had with Perl. I learned Perl when everybody was using it to do "CGI" web programming. The syntax was definitely quirky, but once you learned it Perl was really productive for putting together simple scripts. Stuff like built-in syntax for regular expressions, hashtables, and arrays. Growable lists. A convenient syntax for looping over a file. The foreach loop. Etc.

So Perl served a very useful niche (scripting) and did it well with syntax.

I posit that there is a scientific basis for the popular preference for more texturally varied syntaxes, namely preattentive visual processing.

Alignment is preattentive, thus indentation as a way to structure Lisp code.
IDEs that do syntax coloring are exploiting preattentive qualities.

It took me years to get over my aversion to reading Lisp. Now it is ok for me. But I know a number of very smart folks who find it objectionable.

someone once said something like: it's almost pseudocode.

I find it very familiar, and very easy to write useful things in it without having to really get into the syntax, and what's more important leaving my apps and application alone for weeks, coming back, and taking back up where I left off.

I'm not saying it's my favorite language, but I think the ease of its syntax is a definite plus.

I find it very familiar

Sure. And as I said, I find it no more familiar than Lisp. That's the point. "Familiarity" is a subjective property that depends on one's background. I'm not saying that Python is "unfamiliar". Simply that I don't find it more familiar than Lisp.

OTOH, I was briefly exposed to Scheme for a few months about 13 years ago. Maybe that makes a difference. Then again, it used to be the case that many undergrad CS programs did some introductory work in Scheme. Which would presumably mean that there are a lot of other people out there for whom Lisp syntax isn't "unfamiliar". Which leads me back to my original point: I am skeptical that "syntactic familiarity" is the reason that Python and Ruby have prospered while Lisp has failed to achieve the same level of popularity.

Since this is a historical question, you should approach it as such.

When did the things you linked to arrive? Was it perhaps too late, after niches filled? After Lisp got a reputation for being a bad choice (beacuse it was considered slow, alien, just for AI, or some other reason)?

One should also consider who promoted the language and where (compare the success of C++ and Simula).

A proper historical analysis of programming languages is quite difficult, and as far as I am aware has never been done. One should be careful not to assume the issue is "which language is better" from a universal point of view, and keep in mind the specific context in which the language had to fight for survival.

This issue isn't just of theoretical importance, I think many of the ideas evolving in the Haskell community (not just the language itself), are being subject to the same pressures.

You make a couple of very good points. Unfortunately, I'm in no position to really carry out such an analysis, because my knowledge of Lisp history is pretty limited.

I suppose that one thing worth keeping in mind (following your advice about context) is that the things that made languages like Python and Ruby popular may simply not have been relevant in the days when Lisp was a rising star. And once Lisp got a reputation as a language "on the way down" it wouldn't necessarily matter what features it supported - people tend to resist using a "dying" language.

The links above were collected in a few minutes worth of Google trawling

Well, exactly, they were just plucked from the ether. I could do that too but I wouldn't have much confidence in the results. For instance, you assert Kent Pitman is the readily identifiable leader of Lisp. But the guys website seems mostly devoid of Lisp - he did a Slashdot interview on it, great - so I'm not sure he is what I was thinking of. For instance, Python has Guido van Rossum, Ruby has Matz, C++ has Stroustroup, C# has Microsoft etc.

Lisp has ... Paul Graham? That's the name that comes to mind for me. But Paul Graham doesn't do much to dispel the image of Lisp as being an elitist language, which is offputting ... his writings are full of references to Lisp being for "real hackers" and so on.

If I look at Python I can go to python.org, read the documentation, see the discussion lists etc and have some reasonable degree of confidence that what I'm seeing represents what Python is. No equivalent for Lisp. This might *seem* trivial but at least to me it matters.

Of course we see a lot of stuff on the internet about these languages, but I do not think they are widely adopted on a worldwide basis. Please correct me if I am wrong, but from where I stand (in Europe) I see VB/Java everywhere. Maybe in USA things are different?

Yeah obviously VB/VBScript, Java and C++ stomp all over Python and Ruby in terms of raw numbers, I guess what I really meant was that if you wrote a program in a commercial software shop using Python it wouldn't really raise eyebrows. People would probably have heard of it at least. Whereas using Lisp would definitely would.

I'm sure you're right there's a great deal of randomness, though I think there's a number of patterns we can divine.

* Unix explained its success in Kernighan/Mashey paper. A minimum bar of technical ability is necessary, but afterwards other factors come in to play. They attributed unix success to cheap licensing and small footprint, simple to port. Over time orphaned machines ran unix and the virus spread. Then, many Lisp Machines were presumably costly (Symbolics, Connection Machine) -- one thing about a market is cost is a major factor, not just quality of some product.

* Successful languages languages are increasingly "lisp-like" -- take "MatzLisp" and Guy Steele's claim that Java dragged C++ programmers halfway to Lisp. Strong pattern of the mainstream moving towards lisp, which makes sense if we consider that shifting preferences is a fairly gradual process.

Apologists for the duller languages often protest that non-professionals, especially those in the arts, want languages in which they can "just get the job done" - in which sophisticated semantics are irrelevant or unhelpful compared to the pleasant sweetness of syntactic icing. I think this unlikely. I'm an artist, without any training in mathematics or science since the age of fourteen, and lisp is the only language attractive enough for me to want to use. Its conceptual beauty is so extraordinary as to be obvious to lots of people who've never had any desire to program a computer. Several friends, who a few years ago would merrily lave their programming classmates' heads in school lavatories, have been delighted with the described idea of lisp; and even girls agree, picking scheme over C, says Matthias Felleisen, in the ratio 4:1. So perhaps lisp could, on merely aesthetic grounds, find a niche amongst those for whom, as Keats pointed out, beauty is t.

Scheme vs C is hardly fair play though. Most of the time I'd rather Haskell than either.

True, it's unfair.

Lisp will probably outlive all other PLs ever invented. That's not to say that it will ever reach the level of being considered popular. And this particular argument about why lisp hasn't caught on may still be taking place a hundred years from now.

I suppose popularity can be an interesting discussion, but most of the time it seems to boil down to the participants having an economic stake (in both time spent learning/applying a language, and the ability to get others to pay you while doing the same). At the end of the day, one should admire Lisp for McCarthy's shear brilliance, whatever attributes and motive one wishes to assign to the elusive "average" programmer that wields the hammer of mass acceptance.

(Disclaimer - I personally prefer to ply my trade in PLs with static typing. But that is not to say that I'm blind).

Had lisp (and the PC) been invented in the 1860s, or, for different reasons, in the 1890s, it would have become mainstream. But our moral language is fragmented; our contemporaries reject the Kantian hunch that choosing those things most admirable and plausible as ends in themselves is the best practice; autonomous sources of the good are everywhere brown and broken. Thus we have PHP.

You've really "drank the kool-aid"! :o)

Perhaps you are joking, but if not, I would say that it seems quite unlikely that Lisp would have caught on in the late 1800s. Perhaps if you assumed that computers even remotely as capable as we have today would have been around back then, but that would have required such a radically different history that it would boggle the mind to imagine. Remember that it's not just the mechanics to computers that would have had to have been around, but also the theory, and that was nowhere even close to having been developed thoroughly at that point.

Frege's Begriffsschrift (which if anything of the sort could have been considered as virtuous, then surely this development would have been), which might be taken to be the first serious step toward making Lisp possible, was largely ignored at first. In fact, as I understand it, much of his work in general was. It wasn't until his first formulations of this new logic were later developed and refined, and mathematical logic proper became a mature subject, at which point the lambda calculus was developed (simultaneously with the rest of the necessary foundations for CS), that we can see the beginnings of the possibility for Lisp.

Just skipping from Frege's early work to the point at which "mathematical logic proper became a mature subject" passes over so many important details that it seems trivial (when it really isn't) that we should ever have gotten Lisp in the first place. But I digress...

Lisp is beautiful in a way, sure. But if you want to really talk about something even more profound and beautiful, consider the (typed) lambda calculus in general, particularly with respect to its deep relationships with so many fundamental subjects like logical deduction (via Curry-Howard Isomorphism) and category theory (via Cartesian closed categories).

Of course you're right that the histories of logic, mathematics, physics and a few other details would have to be rejigged rather substantially for lisp to have been possible in the 1860s or 1890s. I was thinking instead of the approaches to practices, in MacIntyre's sense, that characterise those decades. High Victorians would have loved the Right Thing, Decadents would have loved macros. And both would have preferred what R6RS looks like it'll be to CL.

Lisp will probably outlive all other PLs ever invented.

Except for Fortran. And considering that it's hard for a language to "die", then "outliving" doesn't really mean much.

Languages may not die in the physical sense, but many do become antiquated, disused and/or irrelevant. Lisp pretty well defines the dynamic end of the spectrum for PL's, so it will always survive at some level, even if it's purely a research and exploration vehicle.

On another somewhat related note, there's a discussion in the Ruby community about whether becoming mainstream would be a good thing. As we've discussed in the past, it can be better for a PL that has a motivated community with critical mass, but not to become so popular that the demands drive it over the brink - trying to please everyone and thus pleasing no one in particular.

Lisp pretty well defines the dynamic end of the spectrum for PL's, so it will always survive at some level, even if it's purely a research and exploration vehicle.

And I'm sure Fortran will still be used in engineering at some level for just as long. I'll guess that 10x more (if not more) production code is written in Fortran than Lisp today.

What I see in current research today is more geared at the ML family and not Lisp.

On another somewhat related note, there's a discussion in the Ruby community about whether becoming mainstream would be a good thing. As we've discussed in the past, it can be better for a PL that has a motivated community with critical mass, but not to become so popular that the demands drive it over the brink - trying to please everyone and thus pleasing no one in particular.

I read the blog entry a couple weeks ago, and came away with a different interpretation - as in, "Mainstream ain't cool and I sure don't want Ruby to be attacked like Java is today".

Languages may not die in the physical sense, but many do become antiquated, disused and/or irrelevant.

Since it seems to be my job to bring a social science perspecitve into this discussion, let me just mention that metaphors are imporant (since, as we all know, language influences thought).

The lifecycle of programming languages is more complicated than "born-exist-die". Programming languages can be overlooked and later resurface, they can influence other languages and later be influenced by the very languages they influenced, they can be merged together (into "multiparagimatic languages") and so on.

By using metaphors like "die" we risk neglecting these possibilites. We shouldn't assume a linear lifecylce, since the interesting stuff surfaces when you look into the complex historical details.

A book I've read recently aimed at mathematics but readily applicable to computer science is Where Mathematics Comes From. I'm willing to bet almost all LtU readers would find it interesting even if they don't agree with the authors. (I do have to say that their perspective does coincide fairly well with my personal philosophy of mathematics, albeit they have a much more refined and scientific approach.)

I've seen a few negative comments about the tutorial. Most of them are not related to the ideas within, but to the writing style. They range from constructive criticisms like yours (I'll try to keep this in mind for the next tutorial, will probably be on FP concepts) to outright saying that I am full of myself and just love to listen to myself talk.

What I wanted to do is write an extremely accessible Lisp tutorial that will help non-Lisp programmers to understand the elegance of Lisp. The side effect of that is being verbose and cutesy. I'll consider limiting the rhetoric for the next tutorial but the feedback I've received so far is that verbosity and cheesy humor helps people get through it. Hopefully my writing will improve with time, as I am not a writer and have never considered writing to be one of my strengths.

Concerning your macro example. Remember, macros are just there to transform code. Your macro can accept closures as arguments that the resulting code can take advantage of. A macro helps you design your own domain specific language. It's up to you how flexible you want to make it.

I quite enjoyed the tutorial, and found the style very engaging. Nice work!

I found the "cutesy" style appropriate, and I was most impressed by the Ant analogy. Regarding verbosity: it's hard to strike a balance between over- and under-explaining, and one can never please everyone.

FWIW, there were moments when I thought coffeemug might have let me do some of my own brainwork. For instance, here's how I would have made the introductory paragraphs to Lisp Macros a bit tighter:

So far we've looked at metaprogramming with a simple templating engine similar to JSP, and we've generated our code via simple string manipulations. But we can do much better than that.

Imagine writing a tool that looks at a source code directory and automagically generates a suitable Ant script. We could just use string manipulation, but that would be dumb. We have XML libraries to do all that drudge work for us, leaving us free to concentrate on higher-level abstractions.

But why stop there? Our XML libraries don't just write XML; they also read and manipulate it. By choosing XML as our source code format, we get a code transformation library for free! Heck, we could even follow XSLT's lead by building the XML libraries into Ant itself.

Or we could just use Lisp.

I'm sure I'll be pointing this article out to a few friends in future.

Keep up the good work! :-)

Or something like that. The tutorial is one gazillion times better than not having it, thank you for taking the time to write it. I really did learn ideas and that is terrific!

As an aside, I think the verboseness/cutsiness could be really engaging in a live presentation. And I'm guessing Wittgenstein said something about how there is no control anybody can really impose on teaching and learning. It just has to happen with some back-and-forth?

I can't resist responding to my own post. Lisp is very acceptable from both points of view when the "lambda gap" is successfully bridged. A language that did this was MuLisp produced by Soft Warehouse. MuLisp was developed in assembly language with 8086 machine codes in order to run on early microprocessors. A more familiar application using MuLisp was Derive, a computer algebra program. I recently discovered that both products were bought out by Texas Instruments and now run on "all?" TI calculators. Hope I am right about the details? This tells me that Lisp is alive and well.

Another problem I've found with Common Lisp is that unlike Ruby, Python, Perl and other languages, there doesn't seem to be a really good free, crossplatform implementation. Allegro and LispWorks are expensive and the trial editions are limited. SBCL seems to be the one high-performance implementation that will eventually get there with threads and being crossplatform, but it's not there yet.

"I never could figure out why the obverse is not equally as important - will I gain some advantage or benefit if I do use this language. It seems that businessmen want an advantage, but are loathe to actually grab one when it's there. Maybe they only want to take advantages that they themselves think of, and not one that comes from the 'techies'."

That's actually been studied. There is a broad class of technologies that managers tend to stifle, for clear reasons having to do with internal forces in the company.