"Popular vs. Good" in Programming Languages

Here is part of a discussion from the "Language Smiths" discussion group that I thought might be of interest to some here:


From: http://groups.yahoo.com/group/langsmiths/message/2281




--- In langsmiths@yahoogroups.com, Daniel Ehrenberg

wrote:

> What are your goals with this language? Are you trying

> to be popular, or are you trying to make a good

> language?

My goals are probably the same as just about every other language
designer: to create a better language for me to create the types of
programs that I typically need to create. What is perhaps slightly
different for me is the types of programs that I typically need to
create. I write large enterprise class applications for: financial
institutions, the telecom industry, huge inventory systems, CRM and
e-commerce. This is probably the sort of thing that about 90% of the
world's programmers work on I'm guessing. (In short: the types of
programs that nobody would every write for fun.) The types of
programs that I write a far too large for me to write on my own so I
need to make my language acceptable to the masses if I want to be able
to use it. As a result, I want to make a language which is useable by
90% of the worlds' programmers (and some percentage of the world's
non-programmers). Language experts design languages for their own
needs, and because they're always writing languages, we end up with
many languages which are excellent for language experts to write
languages in. Languages for their own needs and for their own kind.
Unfortunately, the offerings for non-language designers to create
non-languages in, is drastically under-served.

Computer languages are actually for people, not computers. The
computer is happy with machine code. If you make something which is
for people but then most of the people don't like it, then you've
failed. Could you imagine a Chef saying "do you want me to make good
food, or food that people like?" It sounds like a ridiculous thing
to say in the context of food (or clothing, or anything else
supposedly designed for people) and yet it seems to be the common
belief in the design of programming languages.

I wouldn't say that being "good" and being "popular" are mutually
exclusive. A popular language is more likely to have better compilers
and tools, be available for any particular platform, and more likely
to have some desired functionality already implemented by someone
else. Pragmatically, a language which lets me *not* write something
because it has already been written by someone else, is better than a
language which lets me write it any number of times faster. I'm more
likely to be able to buy a book about a popular language, or to write
a book about a popular language and get paid for it. I'm more likely
to be able to find a job or conversely to hire developers for a
popular language. Popularity is a feature, and in many cases, the
most important one.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Quality vs. Popularity

Could you imagine a Chef saying "do you want me to make good food, or food that people like?"

Well, yes. A lot of people like McDonald's food, way more than like French cuisine, but if I go to Chez Antoine for dinner, I don't want a Big Mac.

Pragmatically, a language which lets me *not* write something because it has already been written by someone else, is better than a language which lets me write it any number of times faster.

This assumes that the problem solved is exactly like yours and that the solution has the same hierachy of values and trade-offs you would make.

I think we routinely over-estimate the degree to which all business problems are the same in our hurry to find the next one-size-fits-all solution.

Intentions vs. reality

I doubt that many language designers make an actual explicit decision about whether they're designing a "popular" or "good" language.

My guess is that most designers of languages, popular or not, base their design on their (usually extremely incomplete) knowledge of the current state of the art plus their personal biases.

Most of the time, the end result is something that is most likely to appeal to people with a background similar to the original designer.

Drawing conclusions as to what'll end up being most popular is left as an exercise to the reader.

Correlation? Maybe not.

I think there's very little correlation between "popular" and "good" in any field. All four combinations exist in abundance: popular and good, unpopular and good, popular and bad, unpopular and bad. Of course, you don't hear much about that last group, since a bad, unpopular language doesn't attract enthusiastic advocates to keep reminding us of what we're missing out on.

On the other hand...

Unpopular, bad languages are a great source of horror stories. :) Try googling for "Bancstar".

One difference between "good" and "popular"

is that "popular" can be measured with some accuracy. Good, OTOH, frequently is in the eye of the beholder. Many of us think that languages such as Haskell, Smalltalk, Scheme, AspectJ, etc. are "good"--all of these allow skilled programmers to specify certain classes of algorithms more succinctly than many of the industrial languages out there.

Many programmers (and more importantly their bosses!), OTOH, look at these academic and limited-use languages (with limited tool-support, minimal documentation, small user/developer communities), etc.--and wonder what the fuss is all about. Languages which to us are state-of-the-art are to them, backwards. (And from where I sit--I am firmly in both camps; at work I code almost exculsively in C++). A "good" language, to many, is one with an extensive support infrastructure, rather than an extensive (and elegant) feature set.

Many folks more eloquent than I--Richard Gabriel, Paul Graham--have written about this phenomenon. And of course, as alluded to by a prior poster, it occurs in all fields of endeavor--every major city newspaper, it seems, has at least one movie critic whose job it is to write about how terrible Hollywood movies and are (and to accuse the general populace of being simple-minded fools for going to see them). In some ways, the critic might be right--Hollywood movies are usually made for commercial rather than artistic purposes. But in other ways, he's missing the point--movies are made to entertain the general public, not to entertain movie critics.

In short, "good" is frequently in the eye of the beholder. While it would be a mistake to equate goodness with popularity--it's also a mistake to declare that there is a single metric for goodness that we all must agree on.

In Business, Good means less time or less money.

Good in a business sense means less time or less money to fulfill a customer request.

If programmers were paid by the contract rather than by the hour, I believe a large number would switch to Haskell and other 'good' languages.

I'm self-employed. I'm putting my money where my mouth is and investing my time and effort into Haskell. I'd like to switch my business from Python-based Zope to a Haskell-based alternative.

--Shae Erisson - ScannedInAvian.com

Re: In business, good means less time or money

Shae wrote:

Good in a business sense means less time or less
> money to fulfill a customer request.

Good also means a) maintainable, b) maintainable by someone else besides the intial programmer, and a lot of other things to (some of them which might be irrational...but it's the customer's dime).

If programmers were paid by the contract rather than
> by the hour, I believe a large number would switch
> to Haskell and other 'good' languages.

Hmm. Reading between the lines, one detects a subtle claim that many programmers actively choose "lesser" languages (all the ones with curly braces :) ) in order to feather their beds and guarantee themselves a more reliable income stream.

I have yet to meet a single programmer with that attitude. I have met numerous who have little clue about the good-but-not-popular languages we are discussing, and many who consider them irrelevant to their jobs. I have met many, including myself, who aren't at liberty to choose the language we develop in.

But the suggestion that C++, Java, etc. exist so that developers can accure more billable hours by deliberately using underpowered tools is questionable if not outright wrong. Most of us don't get paid by the hour; we get paid a straight salary and have to work overtime if the project slips.

I'm self-employed. I'm putting my money where my
> mouth is and investing my time and effort into
> Haskell. I'd like to switch my business from
> Python-based Zope to a Haskell-based alternative.

Good luck... I hope it works out well. OTOH, if there is a place where there really is no substitute for dynamic type judgements, it occurs when you have to read in arbitrary data from who-knows-where, a common occurence in all sorts of Web development. There was on WardsWiki a while back a thread on the type-safety of Haskell's deserialization libraries, and the conclusion that was reached is that they don't work terribly well if fed a serialized object of an unexpected type. Of course, my memory of the discussion could be a bit hazy... but certainly no static type inference system can deal with data which is completely unknown at compile-time.

Go home early? and Dynamic Loading

I don't mean to imply that anyone actively chooses a 'lesser' language, but every salary job I've seen wants you there forty hours a week. Do you get to go home early when all your work is done?

I think if everyone got to go home early when their work was done, programmers would look harder for better or faster methods. What do you think?



As for dynamic loading in Haskell, Don Stewart's hs-plugins and Template Haskell cover almost everything.

I don't think they cover mobile values of arbitrary types, I would like to find a solution for that.

--Shae Erisson - ScannedInAvian.com

Arbitrary types

I guess the question is, how arbitrary? If the answer is totally arbitrary then even very dynamic code will be left squinting at exotic values and scratching its head in confusion. If you're meant to be able to find out what to do through reflection, querying of tag metadata and things like that, then you have the beginnings of a very polymorphic type - like the basic interfaces (IUnkown etc.) implemented by every COM object.

Take a look at how Hugs.Net deals with the .Net framework. The key to it is the type information attached to .Net assemblies and classes, which is what makes dynamic assembly loading, reflection, late binding and dynamic method invocation possible. Mobile values never travel without their passports.

At the Border Between Static and Dynamic

Scott Johnson: OTOH, if there is a place where there really is no substitute for dynamic type judgements, it occurs when you have to read in arbitrary data from who-knows-where, a common occurence in all sorts of Web development. There was on WardsWiki a while back a thread on the type-safety of Haskell's deserialization libraries, and the conclusion that was reached is that they don't work terribly well if fed a serialized object of an unexpected type. Of course, my memory of the discussion could be a bit hazy... but certainly no static type inference system can deal with data which is completely unknown at compile-time.

This is indeed where life gets interesting. Earlier here on LtU, someone linked to Acute, which represents Peter Sewell's and his colleagues' current research into type-safe distributed programming. It's rather complex when you try to take versioning and rebinding into account! Simpler systems have been developed for existing languages, e.g. boost::serialization or XParam for C++.

But it's hard to see what the issue is in practice: the fact that you have encoding/decoding work to do "at the border," if you will, in no way negates the value of inferred static typing throughout the rest of the application, and a latently-typed language has exactly the same issue, so there isn't a win there.

Their all the same thing

Good also means a) maintainable, b) maintainable by someone else besides the intial programmer, and a lot of other things to (some of them which might be irrational...but it's the customer's dime).

Those all come down to "less money". It could be less in the intial creation, or less for long term maintance, but it's less money either way.

In any case, maintance is more about the quality of programmers than choice of language, excepting some extreme examples (Brainf*ck).

Good vs popular

In the context of the above cited "...large enterprise class applications...that about 90% of the world's programmers work on... (which I also work on)

There are beautiful (good) languages (Lisp, ML, Oz, ...) that cannot easily be used for these applications. We are forced to use "popular" languages. The demands are driven by user expectations of GUIs and Reporting,... and by enterprise infrastructure elements such as Database, Security, Message Queuing,... that these beautiful language implementors can never get around to supporting.

My solution so far is to use Lisp to generate C#, getting at least some of the benfits of "good" languages.

Beauty in the Eye of the Beholder?

Alex Peake: There are beautiful (good) languages (Lisp, ML, Oz, ...) that cannot easily be used for these applications. We are forced to use "popular" languages. The demands are driven by user expectations of GUIs and Reporting,... and by enterprise infrastructure elements such as Database, Security, Message Queuing,... that these beautiful language implementors can never get around to supporting.

In Lisp:

First, read Rebel With a Cause and follow the links. Then investigate the following:

CL-HTTP
Araneida
UnCommon Web
CLSQL
CLORB

When this isn't enough, there's always an FFI.

In Objective Caml:

Too numerous to count; see The Humps and follow the "Communications and Networking," "Database Interface," and "CORBA and COM" links.

When this isn't enough, there's always its FFI.

In Oz:

It's hard to know what to say here; there's an ODBC interface for Oz. Oz makes it easier to be secure than any other language I know except E, and its support for distributed computing, whether via messaging or other concepts, is unparalleled.

When this isn't enough, there's always its FFI.

It's funny: every single time someone says "I can't do enterprise development in [Lisp|ML|Oz]," I reflect on having done J2EE development for some six years or so, and how much easier and more pleasant it is to accomplish the same tasks, and more, in [Lisp|ML|Oz]. I have to offer my honest opinion that those who say it can't be done haven't tried. At all. Not even once.

It's nowhere near enough!

Thank you Paul, but I am well aware of these, and more. This is a fairly standard answer. And I have tried - with PLT Scheme, Allegro Lisp, Oz and OCaml.

What is the answer to the Infragistics Grid, the FarPoint Input Controls, to Windows Authentication, to Message Queuing, to Reporting Tools (print a packing slip, for example)? And have you ever used raw ODBC? Your answer suggests "...with these wonderful languages it is really easy to write your own, or write your own interfaces with FFI...".

Trouble is I have applications to deliver. It is foolish to spend the time to write stuff that already exists (in .NET (for me), Java or whatever).

This is why I use Lisp to write C#, so that I can use all this stuff that already exists. See Component Source. Write this stuff yourself?

Monoculture

Alex Peake: What is the answer to the Infragistics Grid, the FarPoint Input Controls, to Windows Authentication, to Message Queuing, to Reporting Tools (print a packing slip, for example)? And have you ever used raw ODBC? Your answer suggests "...with these wonderful languages it is really easy to write your own, or write your own interfaces with FFI...".

OK, now we're getting somewhere useful. There isn't much, commercial or open source, in "beautiful" languages or "popular" ones, that can compete with the Microsoft monoculture. So your point wasn't really about languages or even libraries that might exist, but rather one economic observation, which is that the bandwagon effect surrounding Windows has led to the components you mention. OK, granted.

Suppose now that I came to you and said "You have to develop application X. It has to be cost-effective to deploy, so expect to run a server farm on OpenBSD on several inexpensive Intel boxen. There's an Oracle database on a Solaris box that you'll need to talk to, a Siebel 6 app on a Windows NT box over there, and some Lawson accounting packages that need integrating. Oh, we have an MQ Series license or two floating around you should take advantage of. We have desk jockeys who'll need Windows and Mac OS X clients, and the road warriors will need a web browser client." Now what?

My proposition is that once your requirements include "can't just run on Windows," the notion that the "popular" languages have any infrastructure availability benefit over the more "beautiful" languages crumbles to dust. In fairness, Java still has some on the server side—I certainly haven't seen anything as good as Spring, Hibernate, or ActiveMQ on any other platform—but the moment you actually need a comprehensive end-to-end system, Java isn't the answer either.

Alex: Trouble is I have applications to deliver. It is foolish to spend the time to write stuff that already exists (in .NET (for me), Java or whatever).

You're preaching to the choir, although availability of some components in a popular language can lead one down the primrose path of "Well, we can write the non-commodity portion of the system in this popular language that's totally ill-suited for the job" and eat up all of your cost savings and then some.

Alex: This is why I use Lisp to write C#, so that I can use all this stuff that already exists. See Component Source. Write this stuff yourself?

I have to, if I want to be anywhere other than Windows. Better yet, if I'm targeting Windows, I can take advantage of CamlIDL, OCam'OLE, and/or ODLL to do the hard parts in O'Caml while taking advantage of graphing, reporting, widgets, etc. etc. via COM/ActiveX/OLE. This isn't entirely dissimilar in spirit to your approach of using Lisp to generate C#, although I have to wonder why you don't just write in Corman Common Lisp, which can be accessed as a COM Server or generate DLLs or applications, and be done with it.

Not Exactly Monoculture

Paul:

There are two dominant cultures - Microsoft and Java. And as you mention, Java (an equally "popular" language) has much infrastructure.

Also the "mixed" environment you mention (Oracle, Siebel, Lawson, ...) does not preclude Microsoft, though some elements of your scenario may (and in the case of the Mac - do).

Note also that int the context of the initial post ('...90% of the world's programmers work on..."), we do (like it or not) work in a world of 93% Windows Clients and 50% Windows Servers (according to the last statistics I saw).

You state "...So your point wasn't really about languages..." but it is in the sense that a language definition in and of itself is not useful. A language must include connections to the outside world to have any utility, and those must not be left to the user! In the world of "...large enterprise class applications..." (the context of the original post) then Database, Security, Message Queueing, GUI... are just as fundamental as File Access! (And no self respecting language implementor would leave File Access as an exercise for the user?)

As for Corman Lisp - it is a very limited implentation and not particularly useful in this world (no ActiveX support, an unsupported, too simple ODBC solutions for database access, etc.) OCaml is also (IMHO) too limited.

I am looking at some other potentially "beautiful" solutions such as F# and Scala. Too early to say yet. Above all I am in favor of generative development, which my current solution supports.

Ideal solution? Allegro on .NET? RDNZL may help here. Oz on .NET?

The Tomaytoe-Tomahtoe Thing

Alex Peake: You state "...So your point wasn't really about languages..." but it is in the sense that a language definition in and of itself is not useful. A language must include connections to the outside world to have any utility, and those must not be left to the user! In the world of "...large enterprise class applications..." (the context of the original post) then Database, Security, Message Queueing, GUI... are just as fundamental as File Access! (And no self respecting language implementor would leave File Access as an exercise for the user?)

My point was precisely that you rattled off a bunch of MS technologies that, the moment you have a single non-Windows box to develop for or, in some cases, talk to, you can't use. At that point, then, the focus shifts from isn't-the-extent-of-MS-technology-base-great to what-will-allow-me-to-glue-this-stuff-most-effectively, and that's a vastly harder question to answer, and admits of different selection criteria. The situation that I mentioned, with the platforms and pre-packaged software and requirements, wasn't contrived at all, and so all of this business of pre-existing components in COM/ActiveX/OLE etc. is completely irrelevant. A 50% Windows server base means that it's a coin toss as to whether I can take advantage of that market. OpenBSD (vs. Linux) means Java's out, too. I repeat: now what?

Alex: As for Corman Lisp - it is a very limited implentation and not particularly useful in this world (no ActiveX support, an unsupported, too simple ODBC solutions for database access, etc.) OCaml is also (IMHO) too limited.

My concern is that your standard is the Windows market, and that your claim is that, because everything else is "too limited" relative to what's available in that market, your proposed solution is to switch to Windows.

Alex: I am looking at some other potentially "beautiful" solutions such as F# and Scala. Too early to say yet. Above all I am in favor of generative development, which my current solution supports.

F# looks like it made good progress getting to 1.0, but it's an unsupported past research project. Scala and Nice both look... well... nice if you're looking for modern OO-functional languages that target the JVM, for some reason (again, I like the idea, but imagine the screaming if I checked Scala or Nice code into version control on a Java team project...) But if they get you where you want to go and don't cause you political/social nightmares, mazel tov.

Alex: Ideal solution? Allegro on .NET? RDNZL may help here. Oz on .NET?

Allegro certainly is a mature system with lots of support. Oz? No way, not if you don't find O'Caml adequate. And I'm a huge Oz fan.

The demands are driven by use

The demands are driven by user expectations of GUIs and Reporting,... and by enterprise infrastructure elements such as Database, Security, Message Queuing,... that these beautiful language implementors can never get around to supporting.

The discussion has drifted toward the usual "none of the good languages are practical" debate. I'd rather complain about the original message's espousal of "Forget about desiging a good language and just make it popular," with a slight toward the expert language designers for meeting their own needs and not the needs of most programmers.

Many important aspects of language design can be separated from Reporting, Security, Database, etc. Given that these enterprise needs are the business of large, proprietary vendors, and that competition keeps the technology changing, it's no surprise that expert language designers are not providing a platform with interfaces to the current hot technologies. (It could be done if a big corporation got behind it, and to a substantial extent Java brought experts into the process.) A language is not helped toward success by ignoring theory and aiming at incorporating popular features—the enterprise hurdles remain.

I follow the beautiful languages because it helps iron out the mental kinks that accrue from coding in C++. If a "good" language should become popular, I'll be ready.

Databases are a language design issue...

A lot people have mentioned databases and reporting as one of those industry needs current language research has snubbed. But even if you think the DBDebunk crowd gets a little obnoxious, they mostly have their points: Database are the languages that solve these problems, SQL sucks, everyone knows it, and there are theoretical ideas which have worked for a long time, and are totally ignored by industry.

One day we'll have a real relational database language, and, as Java did for garbage collection, it will make everyone laugh about how stupid the past was when we ignored this really good idea, even though it will be mostly the same people who got all stupid about it in the past who are saying this :).

I guess it seems to me that there are languages(SQL, QUEL, Tutorial D, etc.) which have been designed to meet exactly those enterprise needs, and while they don't get a lot of press time here, the better ideas seem just as neglected there as everywhere else.

Tutorial D?

There is an implementation of Tutorial D called D4 from Alphora.

I spent some time working with it and (once again) it is not on the language issues that it falls short. For example, you may only build a GUI their way.

As for reporting, getting the data is not particularly challenging (most of the time), but getting a user acceptable layout is the challenge, and if you cannot hook into Java, .NET or COM, then you have a problem here.

As for other attempts at "real relational database languages" HaskellDB and COmega are, I think, reasonable attempts.

We are, as I think you suggest, limited by the implementations of the SQL Database vendors, who have failed to implement the full Relational model (no transitive closures, domains too simple). See "The Third Manifesto" Date & Darwin.

See also Rel

http://dbappbuilder.sourceforge.net/Rel.html

"Good" and Popular

By "Good" I am assuming the academic view.

I have felt this issue during my career as a Developer: trying to balance my resume needs with my intelectual needs as a Programmer.

The solution that I found was to bounce between typical "industrial" clients (read Java, C, VB) and one academic client (read Prolog).

I am now trying to find a language that is:
1. "Good" from an academic point of view
2. "Good" from an industrial point of view
3. With a strong community behind

At most what I get is "Acceptable" on the 3 issues. I am currently considering Erlang, Python and (may be) Ruby.

I have discarded (because of lack of "industrial" and/or community): Prolog, Haskell, Mozart, OCaml.

Honestly I see no language that can really balance both notions of good. And that is frustrating

Requirements?

Tiago Antao: I have discarded (because of lack of "industrial" and/or community): Prolog, Haskell, Mozart, OCaml.

I wonder if perhaps you'd be willing to expand upon what your requirements are, because my experience with O'Caml has been that with its combination of interactive development, bytecode compilation with time-travel debugging, native compilation with profiling, support for imperative programming, functional programming, and object-oriented programming, broad-and-sometimes-deep library support, foreign-function interface, and extremely active community, it's an easily-justifiable competitor to the more popular languages.

I think Oz is getting there thanks to CTM, and wins handily on expressive power, but has a ways to go before the library support and community are as expansive as some of the other options.

Requirements

In my view OCaml fails by not having (correct me if I am wrong) usage in industrial settings. As an opposite we have Erlang, with a niche environment in an industrial setting.

A requirement for me is having potential costumers, even if only on a niche market.

Wiki for O'Caml industrial users

It has a little.

From lurking on the OCaml mailing list:

Alex Baretta uses it in realtime applications at www.barettadeit.com.
Markus Mottl is working on logging high volume, realtime financial trade data with it.
Richard Jones has written some web programs and libraries in it. (try www.merjis.com and www.annexia.org. I can't check them now; the websites seem to be down.)

A Compromise

There is something I don't understand in this discussion.

It seems there are a lot of people here looking for a compromise between a good language and standard development platforms in the industry. Notice the use of the term "platform". There is a (recent?) trend to provide languages that fulfill this need: F# and Nice most notably. Of course there are other possibilities, you can access every J2EE API from Jython, you can use Eiffel on .NET...

But when there is a post on this topic on LtU it generates very little interest. It's a mystery to me.

Really?

But when there is a post on this topic on LtU it generates very little interest. It's a mystery to me.

Really? I haven't noticed this.

In fact I think Nice was discussed here many times. But I agree that this trend deserves some attention.

Fish/Fowl

Alex Drahon: It seems there are a lot of people here looking for a compromise between a good language and standard development platforms in the industry. Notice the use of the term "platform". There is a (recent?) trend to provide languages that fulfill this need: F# and Nice most notably. Of course there are other possibilities, you can access every J2EE API from Jython, you can use Eiffel on .NET...

But when there is a post on this topic on LtU it generates very little interest. It's a mystery to me.

Speaking for myself, the issue with languages that generate bytecode for popular VMs such as .NET or the JVM is twofold: first, as implementations of whatever language, they often suffer from a serious impedance mismatch (F# is a poor O'Caml; any Scheme on JVM is a poor Scheme at least from a performance perspective, etc.) Second, if the objective is to play nicely with the other children, no one is going to be happy when I check my F# or Nice code into version control if the rest of the team is using C# or Java. So for me, tools like F# and Nice only work out in situations where I'm working by myself and really feel that I must take advantage of some library in another language that sits atop the same VM and using this language that targets that VM is the only way to do that. That's so rare as to never happen, to a first approximation.

Interoperability

Of course, you don't have to target the VM directly if all you need is access to the platform so you can get inside the MS monoculture and get some work done. I'm not certain of the particulars, but I believe that .Net interop can be done in a similar fashion to COM interop - see e.g. Hugs.Net for an example of this in practice.

F# is a poor O'Caml Could

F# is a poor O'Caml

Could you elaborate on this, I haven't found any in-depth analysis of F#'s limitations. The OCaml people seem to like it.

F# Critique?

Alex Drahon: Could you elaborate on this, I haven't found any in-depth analysis of F#'s limitations. The OCaml people seem to like it.

Embarrassingly, as of this writing, I can't seem to find what I'd seen before, and in the meantime, F# has gone 1.0 and acquired a "subset" of the O'Caml 3.06 standard libraries. So without downloading the release and really banging on it, I'm going to have to withdraw my observation.

My recollection of the issues is that they mostly revolved around what you'd expect them to: O'Caml's type system is structural whereas the CLR's is nominal; O'Caml modules and CLR packages don't really map, that sort of thing. My intuitive reaction to the observations was that one could probably write a great deal of code that didn't encounter the issues, but once you really started to take advantage of O'Caml's modules, objects, and some aspects of parametric polymorphism, things would get weird, and/or you'd take vicious performance hits as the mapping couldn't possibly be efficient. Note that similar thoughts apply to the Scheme -> JVM mapping: at some point you're just up against a very different machine model than you'd like to be. Note that this doesn't mean that, e.g. SISC isn't an astonishing accomplishment and well worth considering if you need a JVM-based Scheme, or that F# isn't a great tool if your goal is to target the CLR from a very nice ML dialect. It's really just the trivially obvious claim that doing so involves some compromises.

No retreat, baby, no surrender

I've been using other languages on the JVM for years, but I prefer to use relatively portable languages like Javascript and Scheme, since they allow code to be moved between environments.

A new advanced language which only targets a single imperative/OO VM isn't very attractive to me. In addition to the downsides of using a brand-new language with a small user community is the fact that you're restricted to a single rather odd "platform". It's even worse if the language design was affected by the limitations of its host VM.

Tools vs Toolbox

There is one interesting dichotomy between many industrial programmers and many in the PL community, that hasn't been explored much--but it is, I think, key to understanding and appreciating the points of view of both sides of this debate. For lack of a better name, I call it the "tool vs toolbox" metaphor.

This metaphor is rather weak, and many of the observations I make, it should be noted, are anecdotal in nature. So take everything with a couple grains of salt.

Many industrial programmers--as well as many non-academic programers in the open source community, view programming languages as tools. Such programmers will have many tools in their toolbox--several different languages for different applications (C/C++, Python/Perl, Java, SQL, etc.), a few off-the-shelf domain-specific languages (lex/yacc for scanner/parser generation, etc.), a robust IDE or refactoring browser, build configuration tools, version control. They mix and match them when they need to. Some such programmers use an underpowered set of tools, requiring additional work on their part to do certain tasks that their tools aren't well suited to--i.e. writing text processing code in C. Others have a wide array of tools, and often these are very efficient programmers.

Many folks here--the users of "good" languages--tend to view languages as a complete toolbox. There is frequently the view that the language of choice should be capable of doing 90+% of what they need, without having to resort to something else. Custom-built DSLs and other forms of metaprogramming seem to be popular, and many of the languages (Common Lisp, Smalltalk, etc.) are geared towards maximum mutability and extensibility. The lack of such features in many industrial languages is seen as a weakness. Whereas a C programmer would pull out a separate tool (yacc) if he needs a LALR(1) parser, a Lisp programmer might find such a utility--written in Lisp--and integrate it into his codebase. Or else write it himself. To some, it seems that the use of external tools in software development is proof positive of a weakness in the primary development language. When other languages are used, it's either through a FFI to access some library for talking to the real world, or else for the occasional performance-critical or systems programming task.

In short, many industrial programmers--even those of us who are well-aware of what Haskell, Erlang, Lisp, etc. have to offer--still find the curly-brace languages acceptable for many day-to-day tasks. Why? Because we have no qualms about using a different tool--even a different HLL--when it's appropriate. (And "appropriateness" is context-dependent in that a programmer who is a C++ expert and a Haskell novice will probably be more productive, for most tasks, in C++ than in Haskell). Add in all the well-documented network effects and other externalities which prop up the current set of industrial languages, and it's no surprise that there is little incentive to switch en masse. Worse is better and all of that.

Again, this is a rather weak analogy, and a rather weak claim on my part. It isn't intended as a justification of the poor state of affairs of many industrial languages; however in many cases the state of affairs isn't as poor as some might think.

Tools vs one integrated language

In short, many industrial programmers--even those of us who are well-aware of what Haskell, Erlang, Lisp, etc. have to offer--still find the curly-brace languages acceptable for many day-to-day tasks. Why? Because we have no qualms about using a different tool--even a different HLL--when it's appropriate.

This makes some sense. However you still lose all the niceties of having one high-level (and staticly-typed if you wish) language. It works to replace this with a mess of C, Perl and some Yacc (or whatever), but the result is much less maintainable I would guess.

Then again most industrial programming seems so be geared more towards the next deadline and just-getting-the-job-done than towards maintainability. Language choice is dictated mostly by the project manager (who probably knows all that J2EE much better than even any non-mainstream language) and by the availability of skilled college graduates and what not. What strikes me most sad is that more and more colleges actually went to teach only Java and nothing else to their students. What happened to the use of books like SICP (which I was allowed to enjoy some years ago) in teaching?

Strange

I think it is almost exactly the other way around. PL fans often know a dozen or so languages, tend to choose the one that seems most suited to the task at hand, and refer to mix languages instead of using languages for tasks they aren't suited for.

Industrial programmers know a couple of languages (upto 5 normally, from my experience) and prefer to choose one and concentrate their efforts on mastering it. This leads them to choose the language with the widest range of uses and applicability, i.e., C++/Java etc.

Programmers of this kind often know at least one "scripting" language for quick and dirty jobs, and that's an improvement because in the past many didn't master more than one language (and they were damn proud of this!)

But maybe both are viewpoints are valid, since it really depends on waht exactly falls under the description of "seen as a toolbox".

Language As Toolbox

I think the advent of the modern-day scripting language has changed the definition of a "toolbox". In Unix-land in days of yore, the toolbox was the Unix environment, and the tools were the little languages in that environment: awk, sed, grep, etc. But then Perl came along, and it became the toolbox. The tools were the same, but they were incorporated into a single language. Lisp and Smalltalk had been doing this for years, but Perl brought the language-as-toolbox meme to Unix. (Unfortunately, Perl didn't bring along the conceptual elegance and simple syntax of Lisp or Smalltalk, but that's another story.)

not(Good vs. Popular)

As Scott Turner says: “The discussion has drifted toward the usual ‘none of the good languages are practical’ debate.”

This was not my point.

The title shouldn’t have been “Popular vs. Good” but instead “not(Popular vs. Good)”. What caused me to write the post was someone asking whether I wanted to create a good language *OR* a popular one. I didn’t see that the two were mutually exclusive. It occurred to me that any definition of “good” which excluded or precluded being “popular” was somehow backwards. If anything, being popular is a feature.

Computer languages designers and user interface designers have in common that they both create things for use by real people. In user interface design however they actually care about and study how users use what they’ve created in practice. They perform extensive user studies to verify the practical usability of what they’ve created and they’re very often surprised by what users actually do or want to do. If this weren’t the case then they wouldn’t bother with the studies. The idea of having purely theoretical user interface design which isn’t actually concerned about the usability by real users would be absurd, and yet with programming language design, it seems completely acceptable.

The PL theory community has marginalized itself by failing to address issues of relevance to the majority of programmers. The PL theory community often laments the deficiencies of popular languages, but they only have themselves to blame. They’ve failed to create good populist languages and so the task has been left to those from outside of the PL theory community instead.

When designing the next “good” language, please try to add “practicality” and “suitability” components to your “good” function.

Significant assumption

You're assuming that popular languages are popular due to their having been more practical or suitable to begin with.

I'd say that it's more likely that they were popular first (due more to historical accidents, timing, similarity to existing languages, or even marketing than merit), and became more practical as a result.

Demarginalization...

I think the purpose of the Lightweight Languages workshops is to address exactly that point(I may be wrong here...), and it certainly has introduced me to a whole new side of programming languages.

The other part of your criticism I think stems from the fact that research into programming languages is typically coming from a mathematics background where being logically/formally correct are paramount to anything else.

I think it is important to make sure that a programming language is suitable for human consumption, and I would wager most of the language designers on LtU feel very similar. The theory is often more usable than people think, Garbage Collection was decidedly non-mainstream and academic for a long time, but with Java, people now think its the best thing since sliced bread, sometimes theory and usability seem at odds when they're not. We're in a renaissance of language design right now it seems, and forums like LtU and the Lightweight Languages workshops are bringing together both sides(sometimes with some friction).

This is just my perspective. I was pretty ignorant about these subjects 4 years ago, but reading about the first Lightweight Languages workshop in Dr. Dobbs Journal(decidedly mainstream/hobbyist magazine) got me interested, and trying to make sense of the discussions on the LL1-discuss mailing list and then eventually here has really helped educate me about this stuff. I suspect I'm not the only one.

Whoa, my name's up there in this article's contents

Of course "good" should be defined including "practical" and "suitable". In my mind, those are the biggest components of "goodness". Those, to me, necessitate simplicity, the ability to succinctly encode algorithms, and possibly elegance in the language design, but the goal is usability. All I was saying when I asked "are you trying to make this language good or popular" was that if your goal is to make your language popular (you keep talking about "populist languages"), no offence, but you don't have a very good chance of suceeding since you don't have a corporate or even university backing, AFAIK. Languages don't become popular because they're practical or suitable (at least they're not at first, necessarily); they become popular because of marketing and similarity to existing languages. Take Java. AFAICT, Java doesn't have many (any?) advantages over other languages, and I've never seen an algorithm that wasn't implemented more succinctly and easily in another language. But Sun agressively marketed Java and there were many libraries written for it, so now Java is fairly popular, especially for programs using certain libraries written in Java. Languages like Common Lisp are already as "populist" as they can be; programmers just don't use them much since they're very different from the languages they know and they aren't backed by as large corporations. So if you want to be popular by being good by being practical and suitable (which would be good), I'm not sure you'll succeed


Daniel Ehrenberg

Ideas on how to make a language good AND popular.

You don't need to have a multi-million dollar marketing department, and Python and Perl have shown. But... in the spirit of "Being Popular", my two cents.

* The average programmer doesn't know, or care much, about type theory, the lambda calculus, and numerous other things from theoretical computer science and the related mathematical disciplines. Most don't have advanced degrees, some don't even have BS degrees (and thus wouldn't have been exposed to this material as part of their education), and many who did take a class in this have forgotten about all of it. References to mathematics are not going to sell the language. Instead, focus on the concrete features, and how they will help the average programmer get their job done. If a PhD is required to understand the language, only PhDs will use it.

* In this age of the Internet, a single implementation (ala Python or Perl) is probably to be preferred than a standardized language with multiple implementations, each with their own somewhat-incompatible extensions (ala Scheme). That implemention should be freely available, at least for most applications.

* Most computing tasks are IO-centric, not computational centric or algorithmically difficult. Most of the "advanced" languages, it seems, focuses on more succinct ways of describing/implementing algorithms rather than on implementing communication between systems. In short, they are solving the wrong set of problem(s) for the industrial community. A truly great language, it goes without same, ought to be as powerful as Common Lisp (or nearly so), as elegant as Scheme, and as promiscuous as PHP.

* The community at large--rightly or wrongly--resents Pinky and the Brain languages. To be popular, your language must play nice with the rest of the world. Any language that requires massive amounts of infrastructure be redesigned in order for it to be used, won't be used. No matter how good it is.

* The community at large--rightly or wrongly--also resents Bondage and Discipline languages. Many Haskell fans view the elimination of mutable variables as a great thing, because it gives Haskell compilers much greater ability to reason about the behavior of Haskell programs. Most industrial programmers think it's a pain in the a**. In the same vein, languages with exceedingly verbose syntax--despite advantages in readability and maintainability--are not well-received.

* A language needs to find its niche--a problem that is solves well, and lots of people are interested in a solution for. For Perl and Python it was numerous scripting tasks and CGIs. For Java it was originally applets, now its primarily custom business software. (One must be careful here, lest the language be typecast, and considered suitable only for its niche)

* The language, in this day and age, probably ought be developed in a bazaar and not a cathedral. Python and Perl are both excellent examples. Haskell too, within the academic community. OTOH, languages such as REBOL, Concurrent Clean, and Eiffel--with limited availability and limited ways for the community to affect development--haven't taken off.

* Don't be a twit. Quite a few language communities have repuations--rightly or wrongly--for loudly and persistently proclaiming their favorite language the One True Language, and suggesting that users of other languages must have some deficiency in order for them to not drop their lesser language and immediately switch. This doesn't help. Would Perl have been a success if Larry Wall had the temperment of E*** N*****?

* Good documentation is important. If we can't figure out how to use it, we won't. Most programmers are lazy, and unwilling to invest significant amounts of time to learn something new unless there is a good reason.

* And, to rephrase Richard Gabriel's widsom in "Worse is better", it's better to have a good language that's available and usable, than a perfect language that's vaporware, buggy, or otherwise unsuitable for actual use.

* One final point. Hang in there. Just because you put up a webpage on SourceForge, don't expect the world to beat down your door and download your stuff. How long did Perl and Python languish in obscurity before building up credibility among the programming community?

The language, in this day and

The language, in this day and age, probably ought be developed in a bazaar and not a cathedral.

Actually, I'd say the popular languages out there weren't truly developed in bazaars either. One of the big wins of Erlang and Python and Perl and so on is that they have a small group pulling all the strings. People write libraries, but the language itself is tightly controlled. This is still better than being close source controlled by a business that may or may not be around next year.

Most computing tasks are IO-centric,

Scott, I think this is the single most limiting factor on the adoption of Oz, OCaml, Scheme, Lisp and the rest. In the world of "...large enterprise class applications ...where about 90% of the world's programmers work..." this is why these languages are not sufficiently practical. You would lose all their productivity advantages implementing this needed functionality.

What functionality?

What functionality do you mean?

xml-rpc? asn.1? A foreign function interface that can bind to C programs? What IO are you talking about?

--Shae Erisson - ScannedInAvian.com

What Functionality?

Certainly not "...A foreign function interface that can bind to C programs..." since the majority of infrastructre is built in Java and .NET these days (in this thread context).

IO, as mentioned elswhere in this thread, is about (good** versions of) GUI, Relational Database, Authentication, Message Queueing, Report Design, ...

**good means -> meets user requirements, productive development and productive maintenance and evolution.

Good vs Productive languages

Unfortunately the whole discussion is full of unproven assumptions about "good languages" (which most people seem to equate with the usual functional languages like Lisp and Haskell). If "good languages" are so great and programmers only write in curly-brace languages because they are either uneducated or forced to, where are the amazing programs written in the "good languages"?

There are many million lines of open-source code written in curly-brace languages. Most of these programs have been started by people in their spare time, so there was no pressure to use a specific language. But is there only one major open source program written in a "good language", beside the 20 year old Emacs? If "good languages" make you so incredibly productive, why don't you convince the world by writing a web browser or a datavase server that is better than the curly-brace software? Hey, you claim to be 5 times more productive in your "good language", so why don't you put your code where your mouth is?

Right now, there are just completely unfounded claims. And until you prove them, you may convince people that your language may look good, or that it is "pure" and "good" from a theoretical point of view. With which I may even agree. But they are just beautiful toy languages, and not much more.

Paul Graham

Beating the Averages

Except:

Sometimes, in desperation, competitors would try to introduce features that we didn't have. But with Lisp our development cycle was so fast that we could sometimes duplicate a new feature within a day or two of a competitor announcing it in a press release. By the time journalists covering the press release got round to calling us, we would have the new feature too.

Paul Graham's Yahoo code

Yes, Paul Graham's Viaweb/Yahoo code is No 2 of the most frequently cited examples of how productive Lisp is. Unfortunately almost no one ever saw it; and the ones who did apparantly reimplemented it in curly-brace languages. Any other example of great "good language" code, preferably examples that are still actively maintained?

Founding the claims

There are plenty of amazing programs written in the "good languages", but they tend to be in different problem domains than the ones you're thinking of. There are a few more mainstream-oriented programs, e.g. the Unison file synchronizer, implemented in OCaml. There are also many very powerful open source web and application servers written in languages like Lisp, Scheme, ML, OCaml, and Haskell, but to fully realize their benefits, you need to use their language, which is unattractive to programmers who're fully invested in mainstream languages.

The playing field is heavily tilted against languages other than mainstream ones. The mainstream languages are all essentially alike, mostly being some variation on imperative and object-oriented. They're mostly implemented in C or similar, which is no problem for the languages being implemented, because those languages all tend to share C's limitations. C programs integrate well with operating systems also implemented in C.

One problem for functional languages is that it's often a challenge to implement them efficiently in C. That's an indictment of C, not functional languages. There's nothing fundamental about functional languages that is difficult to implement, it's only that the mainstream computing infrastructure has limitations that really don't have any justification other than an historical one which boils down to "they didn't know any better at the time". Unfortunately, some of these limitations are difficult or costly to work around, and this tends to be a handicap for functional languages.

Then there are the network effects which work against these languages. Providing powerful features often requires providing an infrastructure, and that can be a problem for acceptance - languages like Java only overcame this with huge amounts of marketing and (originally) a deal with Microsoft.

But they are just beautiful toy languages, and not much more.

That's a wild overstatement. It's a bit like claiming that because you've never been able to take a ride in an F-16 fighter jet, and you don't see them flying around much, that they're just beautiful toy planes, and not much more.

Some of these languages - Lisp in particular - are used in very demanding, real-world, commercial applications, like airline scheduling, financial applications, chip manufacture, and scientific applications. However, the nature of these applications is such that you're not likely to find many of them in open-source form, or even advertised for sale in a shrink-wrap box. There are some exceptions, though, like SIMSYNCH, a simulator for digital electronics, implemented in Scheme.

"Good languages" in practice

There are also many very powerful open source web and application servers written in languages like Lisp, Scheme, ML, OCaml, and Haskell, but to fully realize their benefits, you need to use their language, which is unattractive to programmers who're fully invested in mainstream languages.

So are there any public sites that use these web servers? I know that mainstream programmers can not be expected to write code in "good languages". But I would expect that their proponents are writing lots of code, and I wonder where it hides.

One problem for functional languages is that it's often a challenge to implement them efficiently in C. That's an indictment of C, not functional languages.

Today there should be a sufficient supply of stable functional language implementations that can be used to bootstrap more efficient implementations. That's what I would do if I write a language, the first real task is to implement the language in itself. Why should they be implemented in C? Or do you want to imply that "good languages" are not general purpose languages, but can only be used for a few selected tasks?

Some of these languages - Lisp in particular - are used in very demanding, real-world, commercial applications, like airline scheduling, financial applications, chip manufacture, and scientific applications.

I don't doubt that, someone must have bought all those Lisp machines. But they competed with Cobol, Fortran and Pascal; and with this competition, I probably would have preferred Lisp as well. If I had the choice between Java and Lisp today, I'd rather take Java, even if I only got the java.lang package and had to implement all other libraries myself. And believe me, i do hate Java.

Re: "Good languages" in practice

So are there any public sites that use these web servers?

Sure. A few off the top of my head are CLiki, implemented in Common Lisp; Community Scheme Wiki runs on WiLiKi, implemented in Gauche Scheme; the OCaml Alliance Network runs on an OCaml wiki; there's Seaside for Smalltalk; and many others. As an example of something more directly commercial, here's a short paper on Scheme in the Real World, a case study of SISC Scheme being used with Tomcat to implement a commercial web application.

There's a huge amount of stuff out there, but it's not going to jump out at you from mainstream sources of info. The problem with mainstream info is that it's all biased to making you think that the mainstream is all there is. When articles appear about something slightly unusual, it's usually written by someone who doesn't know the subject very well and as often as not, just perpetuates the stereotypes and tells people what they were expecting to hear.

Today there should be a sufficient supply of stable functional language implementations that can be used to bootstrap more efficient implementations. That's what I would do if I write a language, the first real task is to implement the language in itself. Why should they be implemented in C? Or do you want to imply that "good languages" are not general purpose languages, but can only be used for a few selected tasks?

First, I should note that the PHP compiler Roadsend is implemented in Bigloo Scheme, which provides a kind of answer to your question. Also, many of the functional languages are indeed bootstrapped. However, there are a few assumptions you seem to be making that are questionable. First of all, the goal of languages like ML and Haskell is not to compete with C as a systems language. They provide much higher-level abstractions, and in that sense compete more directly with other higher-level languages. As it happens, one of the benefits of functional languages is that they have good static properties, so can be compiled quite efficiently. This means that many of them perform better than their mainstream competition. But that still doesn't necessarily make them good systems languages. In many respects, a systems language and a very high-level language should not be the same language - there are some fairly fundamental reasons why that's the case.

One of the major reasons C is important is in the need to interface to operating systems and the like - the binary interfaces of C and even (in some cases) C++ are embedded at the lowest levels of most systems. Even if you don't implement a language in C, to interoperate you still need to emulate C's calling conventions and deal with some of its limitations (or the limitations of the operating system, which pretty much amount to the same thing) in order to integrate with C-based systems. This has a cost that's higher the more unlike C a language is.

There are systems languages like C-- and PreScheme, and more recently, BitC, which address the issues of providing a systems language that are good for implementing advanced languages in. That doesn't necessarily resolve the issues of integrating with the C world, though — some of those issues are just fundamentally difficult.

Also, bootstrapping a language only takes you so far: if you're going to go all the way and generate efficient native code, you need a different native code generator for every platform, and there are a lot of desirable target platforms out there. The people building advanced languages don't have the kind of resources to compete with the decades of commercial work and enormous manpower that have gone into producing efficient C code generators, so leveraging C is just a natural thing to do. Everyone else does it — Python, Perl, Ruby etc. are implemented in C, for example. It's just that the functional languages are more handicapped by doing this. The very features the functional languages offer that distinguish them from the mainstream are the ones that don't translate well into mainstream-compatible implementations. It's easy to make a functional language as a closed world; it's much harder to make it play well with its lower-level cousins that don't share its basic design parameters.

In short, there's a huge array of tightly interconnected challenges in integrating with the mainstream. One could take the position that hey, that's the way it is, deal with it. Despite all the challenges, functional language implementors have been doing just that, and there are good, fast implementations of all the usual suspects. The problem is that if we want to move forward to more advanced, reliable, expressive systems, we will ultimately have to stop building systems on top of languages like C. That's the reality that we have to deal with, but we probably have many more decades of denial to live through while we slowly inch our way towards a more robust foundation. It reminds me of the old joke about a farmer giving directions, who says "you can't get there from here" — that's the situation we're in right now.

I don't doubt that, someone must have bought all those Lisp machines. But they competed with Cobol, Fortran and Pascal; and with this competition, I probably would have preferred Lisp as well. If I had the choice between Java and Lisp today, I'd rather take Java, even if I only got the java.lang package and had to implement all other libraries myself. And believe me, i do hate Java.

People are still choosing Lisp, Scheme, OCaml, Erlang etc. for new projects today, I wasn't talking about a "legacy" thing. Ignoring Erlang, which targets its own specialized domain, the reason organizations today choose these languages for a project is because of advantages in expressive power of the language, where that's an important factor. Java is fine if you want to churn out database-backed web applications, and scale them up across server farms, but do you really want to implement seriously complex algorithms in it? I could turn this discussion around and ask you to show me the Java applications that implement something really advanced. I work with Java myself, and most of the applications I see being developed are ultimately pretty trivial, regardless of the sometimes overwhelming level of mind-numbing detail that they can involve. In my experience, when people start needing to do something algorithmically complex, they do tend to look further afield for languages. Sometimes that happens because projects in languages like C++ and Java will hit a complexity wall which forces people to move either to domain-specific languages or to more expressive languages.

Problems with C as an intermediate language

WardsWiki has a bit on the use of C as an intermediate language, including some rather annoying disadvantages--many of them resolved by C--.

For many HLLs, a key is little support for flow control mechanisms that C doesn't natively support--that includes exceptions, backtracking, continuations, coroutines, threads, and quite a few others. The closest C comes (while still being portable) is setjmp()/longjmp()--these are barely adequate for exception handling, and not suitable at all for general-purpose continuations. C works better an an implementation language for an interpreter/VM, where you can simulate these things at the VM level. C can suffice as a portable assembler, but it doesn't really excel at it.

No support for high level features is better than incomplete one

I've written a compiler which targets C, and I think the result is better than if I targeted C++, JVM or OCaml VM.

C provides numbers, data pointers, function pointers, global data, global functions, and not much else.

There are several reasons for implementing the stack by hand:

  • garbage collection
  • tail calls
  • exceptions
  • lightweight threads
  • resizing the stack on stack overflow
  • printing a stack trace after unlandled exception

(My language doesn't support continuations.)

This required some work, but a single design provides all these features. What if I used a more advanced language as the target?

C++ provides exceptions, but it doesn't provide anything else, so its exceptions would be useless, because I would still have to use a custom stack.

JVM provides GC, exceptions, heavy threads, and Java stack traces. Unfortunately TCO in Java is a pain in the ass. I would also have to give up lightweight threads, deep recursion would fail, and it's not clear whether it's possible to extract the necessary information from a Java stack trace on unhandled exception (this depends on the mapping of calling conventions). OTOH if I didn't want to lose these features and wanted to implement my own stack, Java support for exceptions, threads and stack traces would be wasted, and perhaps the only feature I would actually use is the GC. Provided that I remember to always clear memory below the emulated stack pointer.

While OCaml calling convention seems fine for another language (except that it's hard to handle statically unknown arities), it would constrain object representation: there would be more indirections (I can't attach a variable number of fields to a tagged object, I must make a separate array), all internal function pointers would become closures even if I knew that they are always global, dynamic types would require adding a new word to each object in addition to OCaml's internal header, while my current C design has only one header word, etc.

C as a target language requires much work, but at least it doesn't get in the way.

Could you elaborate?

One problem for functional languages is that it's often a challenge to implement them efficiently in C.

I'm not sure I understand why this is a problem. Could you expand on it a bit?

Certainly it's a much smaller issue than the already mentioned inertia behind the mainstream languages, and I'd have to guess that it's less of an issue than the lack of exposure given to non-(imperative/OO) languages during one's education. How many universities out there really give their students any experience with non-mainstream languages?

E.g. at my university, they have just two required classes that use languages other than C/Java. One is a scheme class taken after an introductory programming course in Java, usually in the second semester of the first year, and the other is a survey of programming paradigms, usually taken during the last semester before graduating, where the semester is generally split into fourths with one fourth going to Prolog, one fourth going to ML, and the other half of the semester given to the procedural and OO paradigms the students spent the rest of their education with already (though it should be noted that at least one of the professors used this time to introduce the students to CLOS). For the rest of their coursework, they will use nothing but C and Java in their required courses. Even in the data structures and algorithms courses which can easily be taught using just about any language (unlambda, etc. need not apply).

And so we have these non-mainstream languages being treated in a rather second-class way. They are initially introduced as an alternative approach to programming, but then are utterly dropped for all but a couple of weeks at the end of the program. Is it any surprise that the students leave thinking that these languages can't be good for doing "real work"? After all, if they were useful at all, they would have shown up sometime, right?

As it is, we seem to have more students leaving schools saying things like (from a thread on slashdot a few months back):

Iterative programming is simply more natural. A five year old can count down from ten. But he doesn't do it by by constructing a list of ten elements, mentally reversing it, then applying a function that strips off and prints the first element of the list, then recursively apply the same printing function to the rest of the list until it is empty.

because

the example I gave was straight from the (minimal) introduction to functional programming I got in school. Which re-enforces my point that functional programming isn't so simple, if I got it wrong.

Elaborating, and then some.

One problem for functional languages is that it's often a challenge to implement them efficiently in C.
I'm not sure I understand why this is a problem. Could you expand on it a bit?

The canonical example of a problem is tail call optimization (TCO). Any fully functional language runs into this issue. C doesn't support TCO in general, even though some compilers have limited support for it. Workarounds include trampolines, which are inefficient; or less commonly, the Cheney on the MTA technique used e.g. by Chicken Scheme; or compiling programs to one huge function and using labels and gotos; and so on. These techniques all work, but they all have costs and drawbacks.

TCO is a fairly simple issue, though, compared to e.g. first-class continuations. For a random example of some issues in integrating continuations into a mainstream environment, see this description of implementation issues in Elk Scheme. Integrating continuations with operating system threads and libraries presents a challenge - not purely C-specific, but still symptomatic of features prevalent in the C-based world, such as the shared-memory concurrency paradigm. Ultimately, some of the more advanced capabilities will not fully come into their own until we've succeeded in completely replacing the current mainstream foundations, which is a long-term project to say the least.

Simon-Peyton Jones' paper Tackling the Awkward Squad covers issues that Haskell faces in dealing with its own restrictions on side effects, but also covers issues with concurrency and foreign-language calls which relate to the above.

Certainly it's a much smaller issue than the already mentioned inertia behind the mainstream languages, and I'd have to guess that it's less of an issue than the lack of exposure given to non-(imperative/OO) languages during one's education.
I don't agree that it's such a small issue. The issues I've been describing conspire to make it more difficult to just fire up a functional language and churn out a program that does all the same kinds of things that the average commercial program needs to do. It's not that it can't be done, but there are many more hurdles to deal with for advanced languages right now. Even if programmers have the luxury to choose a non-mainstream language in the first place, those who just need to get stuff done usually can't afford to be pioneers. The impedance mismatch between advanced languages and the C hegemony is a significant one — it's an omnipresent impeding factor which is easy to overlook because it's so ubiquitous and taken for granted.

Functional Language Education

How many universities out there really give their students any experience with non-mainstream languages? Is it any surprise that the students leave thinking that these languages can't be good for doing "real work"?

Oh no, I'm not surprised at all. I went through the same sort of schooling. Overall, I see us as being at the beginning of a process of emerging from a kind of pre-theoretic dark ages, comparable to the days when we built bridges that could shake apart due to resonance because we didn't fully understand the physics. The process of propagation of education and change that needs to happen on a wide scale will probably take generations.

BUT! That doesn't mean that smart and aware people right now can't take advantage of the enormous advances in understanding of programming that haven't yet made it into the mainstream. As Paul Graham points out, it can be a competitive advantage, and he's only one of the most prominent examples of that, not the only one by far. A similarly inspiring description can be found in Dan Friedman's The Role of the Study of Programming Languages in the Education of a Programmer, e.g. the description of the "Online Anywhere" service — also acquired by Yahoo, as it happens.

The Slashdot quote given above ends with "Which re-enforces my point that functional programming isn't so simple, if I got it wrong." I think the author is right on a number of levels, but wrong in his interpration of the implications. Part of the problem with teaching functional programming is that it's usually associated with teaching more fundamental concepts — really, teaching powerful ways to think about problems, ways to abstract, ways to capture and describe and think about problems formally.

Some of this should really be taught in mathematics and elsewhere, e.g. logic, but the entire curriculum surrounding formal thinking is pretty weak and disconnected. In part, that's because the recognition of the deep connections between logic, mathematical logic, mathematics, programming, and just plain thinking haven't really sunk in on a large scale. (Besides, schools in the U.S. at least are too busy backlashing against the "Social Math" backlash against the old "New Math" which replaced the old Math...)

Teaching these concepts is more difficult than simply teaching someone how to use a canned iteration construct, and I'm sure it's not being done as well as it could be right now. Nevertheless, it's worth trying, we'll improve, and the result is more capable programmers and thinkers. The Slashdot quote shows that the wrong lesson was taken from the teaching of fundamental concepts — ask the author of that quote to compare an iterative version of quicksort to the recursive version, for example. Of course, if schools did a better job of teaching logic to all students, they'd know that they shouldn't extrapolate from individual examples to reach a conclusion like "Iterative programming is simply more natural."

BTW, while I'm monologuing, I may as well add that I'm not one of those who thinks everyone should just switch to a functional language and be done with it. To me, functional languages aren't the point, they're just one of the shining examples of languages which have had a lot of formal thought put into them, that have solved a lot of difficult problems, and offer ways to abstract problems that can't easily be duplicated in more ad-hoc languages. There's still plenty of cross-fertilization that needs to happen in both directions. The point, though, is that individual smart programmers in the 21st century owe it to themselves to educate themselves on these concepts and techniques so that they can use the languages we have more effectively, propagate that knowledge, and put pressure on mainstream communities, languages and tools to move in directions that will benefit us all.

</climb-down-from-podium>

Much appreciated

Thanks for the exposition (both here and in your Re: "Good languages" in practice response up above), as they brought up a few points I hadn't considered, e.g. interacting cleanly with the OS. (I'm still fairly new to the PL game, slowly evolving from a controls engineer to a cryptographer, and now drifting into PLT and doing my damndest at the moment to learn about/understand the pi and spi calculi...) I suppose I can chalk this up into the "good things to have learned this week" column.

And drifting back to the education bit (Sorry for going so OT Kevin):

Teaching these concepts is more difficult than simply teaching someone how to use a canned iteration construct, and I'm sure it's not being done as well as it could be right now.

Absolutely. Which is part of the reason I'm a bit.. well, surprised isn't quite the right word... that these concepts, and the languages that are well suited for them are ignored for three out of four years. Why aren't these concepts that take some time to sink in and age given repeated exposure in many different classes? Why aren't they brought back in the data structures or foundations of computer science classes, a la Teaching recursion as a problem-solving tool using standard ML? Why not introduce a project with a sub-problem that is cleanly handled in, say, scheme or Oz in the typical design of large programs classes so the students can learn about interfacing with other languages when they can make a job easier?

I'm not saying to switch over completely to an FP based curriculum --that would be no better-- but I would like to see these things kept alive in the students' minds throughout their education. It's that whole "How do you get to Carnegie Hall?" idea... But I suppose I'm preaching to the choir here.

http://franz.com/success/all_

http://franz.com/success/all_customer_apps.lhtml

Perspective?

If "good languages" are so great and programmers only write in curly-brace languages because they are either uneducated or forced to, where are the amazing programs written in the "good languages"?

That's a good question. There really aren't that many programs written in LtU Approved (tm) languages which are easily accessible and have wide appeal. I really hope that this improves, but I do think programs like Darcs, Unison, Regex Coach, ejabberd, and Wings 3D are a good start.

On the other hand, much more of the software that I'm interested in is written in these languages than in Java, C#, C++, etc. (Most is written in C, which is another kettle of fish.) Probably the kind of work you like doing and the kind of programs you like using are the biggest factor in how relevant various programming languages seem to be.

Of course my answer is probably not very satisfying to you. I'd much prefer to be able to point out a large body of the kind of software that you like. Sadly this is only possible for people having certain acquired tastes (e.g. Emacs, Squeak).

The debugging problem

One more thing that need to be solved. (This is especially true of declarative and lazy-functional languages).

That thing is the debugging problem. Using a debugger to debug an imperative program is straightforward. The downside of imperative programming, of course, is that the programmer is responsible for specifying the explicit steps of the algorithm, rather than what he/she wants. The upside is that when things go wrong, there is a great deal of transparency into the inner workings of a machine. Even if imperative code is run on a VM, most VMs are sufficiently transparent in their operation that the programmer need only consider his/her source code.

Compare that situation to debugging Haskell, or Prolog. (Or worse, consider a lazy functional language, or lazy construct in an otherwise imperative language, that allows side effects). Often times, the problem is discovered miles away from where it occurs--often expressed as a unification failure or similar--and debugging the problem requires knowledge of how the underlying inference engine or graph reducer operates. (On the plus side for these languages, there is generally less that can go wrong).

Actually, many users of industrial languages DO run into this issue, and don't like the results one bit. Regular expressions (the programming construct in Perl, grep, etc; not the mathematical construct) are notoriously difficult to debug--especially if complicated--and even if correct, are occasionally known to have hyper-exponential running time. Regex's are, of course, a largely declarative programming tool--when they work, they work great. When they don't work, they're a pain.

Another example, of course, is C++ templates. Simple templates aren't a problem, but the C++ template system is essentially a Turing-complete lazy functional language, with very little transparency in its operation (assuming you're not one of the few who have swallowed the ANSI C++ standard). Any C++ programmer knows the sheer pain of the incomprehensible gobbledygook produced by the compiler when STL (or Boost, or any non-trivial template metaprogramming) is not done correctly. The stream of error messages emmitted by the compiler--simultaneously verbose, but telling you nothing--is a nightmare. And it seems to be the concensus of the C++ compiler community that there's little to be done about this state of affairs.

As a final example, most programmers (in shops that still use make) loathe supporting Makefiles. Make (particularly Gnu make), of course, is one part declarative language (the rules and dependencies), one part functional language (the macros and variables--its almost a simple Lisp dialect with dollar signs), and one part imperative language (the underlying shell, usually the Bourne shell or a derivative thereof). Debugging a makefile is a decidedly unpleasant experience. I've been tasked with this chore numerous times.

Debugabbility--even considering the use of other correctness-verification techniques like assertions, unit tests, design by contract, typechecking and the like--is a paramount concern to programmers everywhere.

What does this mean?

For a language to become mainstream, a reasonable (not necessarily excellent, but reasonable) source-level debugger needs to exist. Can be an instrumented VM or interpreter, a standalone program, etc. Running gdb on the interpreter is not an acceptable solution, especially if the workings of the interpreter aren't well-known. For a declarative or constraint system, it would be nice to know which facts and/or rules are found to be in conflict by the constraint engine or inference engine. And so on. It's a common trap to assume that a) once people see how cool my language is, they'll race to develop a debugger mor me, or b) my language is so elegant it doesn't NEED a debugger. Wrong on both counts.

O'Caml and Oz...

... both have excellent debugging tools, and neither is mainstream in any sense. Oz I can understand; it's still pretty new, still has some deployment issues, and is VM-without-JIT based: it's hard to make claims of high performance for it.

As I've written before, though, O'Caml's relative obscurity outside the intersection of functional/ML programmers and "I like C/C++-competitive performance" programmers (talk about a small set!) continues to confuse me: there are scads of good libraries, Tuareg mode is excellent for us EMACS types and Cameleon is excellent if you can run GTK+. There's a time-travel (!) source-level debugger for the bytecode compiler and a profiler for the native-code compiler. There are machine-native integer and floating-point types, and special attention has been paid to ensuring that arrays of these types play nicely with C code. There are several FFIs, including the particularly-nice Forklift, which keeps its annotations separately from the C headers so the mapping can evolve as the header, which you often don't control, evolves. There are bindings to a gazillion things, including a type-safe binding to OpenGL and to the afore-mentioned GTK+. O'Caml has a very sophisticated preprocessor that lets you do anything from add things to the syntax to replacing the syntax, with complete type safety. People have written Quake-level games in O'Caml, Y2K remediation tools, DNA pattern-matchers, OpenPGP key servers, process group communication systems, C-code verification and transformation tools, hardware design languages and, of course, theorem provers. :-) People have cross-compiled O'Caml code to run on the Sharp Zaurus. O'Caml is the language of choice for exposition in the TAPL series. O'Caml frequently wins the ICFP contest and is generally in the top three in the Great Computer Language Shootout and I don't think has ever failed to be in the top five. O'Caml is multiparadigm, covering imperative, functional, and object-oriented programming. O'Caml has an extremely sophisticated module language and true separate compilation. O'Caml can load new primitives from DLLs as well as dynamically load bytecode modules, making "plug-in" architectures a snap.

O'Caml really does deserve to be C++ successor, IMHO, and yes, I do believe that most of O'Caml's challenges in that regard have to do with advocacy and education rather than any particular issues with O'Caml itself.

How does O'Caml do it?

Something I've been wondering for a while is how O'Caml performs so well compared to other functional languages.

I mean, there are some pretty smart people working on compilers for SML or Haskell, so why would O'Caml be beating them consistently.

I have always suspected that there would have to be some compromise or gotcha behind this...

Anyone know anything about this?

Is strictness an advantage?

One of the big divergences between theory and practice concerns eager vs lazy evaluation in functional languages.

In theory, a lazy functional language (with memoization, so that any given term is evaluated at most once) ought to be faster than a strict one--plus have the other advantage of call-by-name; namely not diverging when unneed computations diverge (or terminate with an error).

In practice, all of the tricks needed to implement lazy evaluation (thunking, graph reduction, etc.) seem to be, in most cases, sufficiently expensive that lazy languages are invariably slower than eager ones--excluding cases where the function in question can be inlined into the caller. O'Caml (like all ML dialects) is of course strict, so its calling sequences can be as fast as C. As O'Caml is statically typed, the runtime typechecks and tag-bits commonly found in most everything-is-an-object languages can also be optimized away. Of course, this doesn't explain why O'Caml is (alledgedly) faster than Standard ML, etc. There, the difference might simply be compiler maturity.

O'Caml Speed

Marc Hamann: Something I've been wondering for a while is how O'Caml performs so well compared to other functional languages.

AFAICT:

  • They don't fetishistically abstract away from the machine: primitive ints overflow, etc. If you want bignums, that's a distinct module.
  • Relatedly, arrays of floats are unboxed and contiguous as part of a deliberate policy to support numerical/scientific computing efficiently.
  • A fair amount of effort has been put into codegen, especially care with respect to alignment issues and so forth. The compiler also does a good job of eliding dead code and inlining trivial functions.
  • Having said that, there are surprising limitations on the compiler's ability to infer when inlining is safe.
There's a good explanation of some of O'Caml's compilation process, with x86 assembly examination, in Chapter 11 of this O'Caml tutorial. The bottom line seems to be: take advantage of the things O'Caml has explicitly done to be fast when you can, and the rest of the time, watch out for inadvertently using polymorphic functions when a simple type annotation will allow inlining, e.g. (<) is polymorphic, so if you know you're comparing integers or floats, say so.

Implementation vs. Specification

I think I see now.

To paraphrase essentially, O'Caml encourages the programmer to worry about the implementation details instead of just the specification for the language.

This reminds me of one of the bugs (or features) of C programming: you had to constantly be thinking of not only what a piece of code did, but how it did it if you were concerned about performance.

(And of course you were, because if performance wasn't an issue, why would you be using C? ;-) )

OCaml performance

OCaml does perform well in practice, and in benchmarks. However, compilers such as MLton and even SML/NJ are far more "advanced" than the OCaml native code compiler, and generate better code for many cases.

The performance of OCaml seems to be more due to design than code generation. There are several reasons why OCaml performs well, some of them (based in part on educated guesses) are:

  • Very efficient garbage collector, allocation is mostly inlined. This is probably the most significant thing affecting the kind of code usually written in OCaml.
  • Extremely lightweight exception mechanism.
  • Inlining works across compilation units.
  • Constructs for writing things in imperative style (for and while loops). This is (sadly) probably a significant factor in benchmark performance.
  • Traditional stack based execution (no continuations). I kind of dislike this sometimes, as I'd really like to have extremely lightweight threads (like CML and Oz).

An interesting thing is that there are simple changes that could improve OCaml code generation significantly for many cases. Currently, it's fairly eager to perform closure conversions (it never does lambda lifting), and doesn't even support inlining functions that are trivially self-tail-recursive and could be transformed into loops.

Part of the reason is because the OCaml compiler doesn't really use conventional compilation techniques and it combines many typical passes together (e.g. in the single perhaps most significant pass, it does closure conversion and "value approximation", which is a kind of combination of constant propagation and inlining). This makes the compiler itself fast but harder to modify incrementally to add optimizations.

Even so, simply by generating explicit loops before inlining and making inlining slightly more aggressive, things like List.iter (fun x -> ...) should be transformable into imperative-equivalent code.

The OCaml compiler also forgets about types very early. Doing this later would allow further optimizations.

Replace C++? That's unpossible!

I agree with most of what you're saying about OCaml (although I'm not a big fan of OCaml's syntax!) However, I don't think the lack of mainstream interest in it is any mystery. The reason is precisely that the languages that it would most directly compete with are C, C++, and maybe Java, not Perl or Python etc.

The former group has much higher barriers to entry, for a multitude of reasons: operating systems are implemented in C, programmers are interested in understanding and controlling the mapping between their programs and the machine, and these kinds of reasons plus other requirements of that development space are such that people are much more skeptical of possible alternatives. Even though not all C/C++ applications have these requirements, and many could reasonably be implemented in OCaml, you have to consider that most C/C++ programmers see the world in terms of the sort of requirements I mentioned, and even if you could get them to consider an alternative language, they aren't easily going to embrace a language that appears more abstract. It's no surprise that C++ only succeeded by being an extension of C. For better or (mainly?) for worse, C is the foundation of the mainstream computing universe, and short of a revolutionary paradigm shift, the foundations are the most resistant to change.

It's a very different situation for languages like Perl and Python, which are mostly used for different kinds of applications in which performance is not as critical, the details of the mapping of program to the machine is not often particularly relevant, etc. New languages can gain a foothold in this space much more easily - Ruby, for example.

Because of this, the first "advanced" language that succeeds in going mainstream will likely resemble a language like Python much more than it resembles C/C++/Java. Lisp-like languages would probably have made it long before now, if it weren't for the mainstream aversion to their syntax. It's probably no coincidence that Python is one of the most Lisp-like mainstream languages out there (or at least, Peter Norvig thought so; I'm not as easy to please).

C and C++ are much more likely to be succeeded by a language that has already gained a strong foothold in the same universe.

out-of-date information

OCaml... top three in the Great Computer Language Shootout and I don't think has ever failed to be in the top five

OCaml is a fine implementation, and for some of the old toy benchmarks it's pushed out of the top 5 by Clean or Bigloo Scheme or Intel C or MLton...

The new toy benchmarks are extremely incomplete but seeing the how much higher Java appears in the rankings when the task takes 1.8s rather than 0.2s should give pause for thought.

The Company You Keep

Isaac Gouy: OCaml is a fine implementation, and for some of the old toy benchmarks it's pushed out of the top 5 by Clean or Bigloo Scheme or Intel C or MLton...

I can live with O'Caml being within ε in CPU performance of Intel C, Concurrent Clean, Bigloo Scheme, and MLton! Those are extremely aggressive compilers! When I run the new benchmarks with the default settings, O'Caml hangs in there at #5. If I revise the top-level weights to be 1 CPU, 1 Memory, and 1 LOC, O'Caml moves up to #4. I stand by the general claim that O'Caml native code is generally competitive with C/C++ code.

Isaac: The new toy benchmarks are extremely incomplete but seeing the how much higher Java appears in the rankings when the task takes 1.8s rather than 0.2s should give pause for thought.

Java is clearly benefiting from some of the same laser focus that's benefitted C/C++ over the years: the JRockit VM, for example, kicks holy butt as a server-side deployment environment.

CRAPS!

O'Caml hangs in there at #5

I never imagined you were taking CRAPS seriously! (Look at the individual toy benchmark measurements.)

Java is clearly benefiting...

Maybe the toy benchmarks are just really bad at showing the performance of long running VMs.

Snake Eyes

Isaac Gouy: I never imagined you were taking CRAPS seriously! (Look at the individual toy benchmark measurements.)

I don't take it that seriously, and in fact I don't take it seriously at all without fiddling with the weights and doing a kind of intuitive multi-dimensional analysis of those results, because, you know, it's all relative.

My personal, anecdotal experience with O'Caml is very, very pleasant. But that just begs the question as to why it isn't more popular. It's one thing to observe that, e.g. CMU CL or SBCL have all of the power of Lisp's 30+ year history, do some level of type inference, and generally kick butt; they still can't touch C/C++ without a lot of hand-tweaking of code, type annotations, etc. O'Caml routinely gets within spitting distance of C/C++ without jumping through any hoops other than maybe preferring for-loops and references to mapping a function over a list. That's what I think would draw more C/C++ programmers if they were aware of it.

Accidents & Ecology

I don't take it seriously at all without

The CRAPS! scores are accidental - more programs have been written in A than B, so A has the opportunity for a higher score.

just begs the question as to why it isn't more popular

When we say A is so much better than B, why doesn't everyone use A; maybe we're correct (A is so much better than B), and they're also correct (the cost and risk of switching to A is too high).

A must be so much better than B, that the returns will obviously cover the cost of displacing B.



(It doesn't matter that A is better suited to a niche in some ideal sense if that niche is already occupied. What matters is that A can displace the current occupier, and then itself resist displacement.)

I guess that's a kind of sad

I guess that's a kind of sad comment, but everytime I turn to learn some more O'Caml, I get annoyed by its superficial unelegantness and unaestheticness (like +. and ;;), and I go back to Haskell (realizing that those languages are quite different, though).

I think this week-end I'll give it another try, just to prove myself wrong ;)

Focus

Michael Walter: I guess that's a kind of sad comment, but everytime I turn to learn some more O'Caml, I get annoyed by its superficial unelegantness and unaestheticness (like +. and ;;), and I go back to Haskell (realizing that those languages are quite different, though).

Interesting; when I go back for another shot at Haskell, I just can't get excited about monadic state and I/O... :-) And then there's its performance, which seems disappointingly slow to me (at least for GHC 6.2.2).

Seriously, O'Caml and Haskell aren't that different. You're observing that Haskell has type classes and therefore can overload + and so forth. That's true, and I'd like to see G'Caml progress, too. OK, Haskell's lazy by default. I'm not so sure that's a win, frankly. Other than that, the languages really seem like kissing cousins to me.

O'Caml revised syntax

Maybe the revised syntax is more to your liking?

Hmm..

Apparent vs Inherent Usability

Just means you're human. People perceive more-aesthetic designs as easier to use than less-aesthetic designs - whether they are or not.

Apparent Usability vs. Inherent Usability Experimental analysis on the determinants of the apparent usability

Aesthetics Are Easy, Right?

What I don't get is why PL designers don't pay more attention to surface aesthetics, since it's such low-hanging fruit. Surely it's much easier to design a pleasant syntax than to write a great compiler. Why, then, does O'Caml have the latter but not the former?

Form vs. Function

Matthew Morgan: What I don't get is why PL designers don't pay more attention to surface aesthetics, since it's such low-hanging fruit. Surely it's much easier to design a pleasant syntax than to write a great compiler. Why, then, does O'Caml have the latter but not the former?

My own take on this is that programming language design is one of the clearest domains in which form really does follow function: your surface syntax is going to be constrained by your abstract syntax, which is going to be constrained by your evaluation regime, which is going to be constrained by your type system, which is going to be constrained by your semantics, and so forth. That's not to say that a heroic effort to overcome this isn't warranted, and I think we're at the precipice of a very exciting time in PL design, where people like Connor McBride and Tim Sweeney are making such attempts. I'm particularly excited by Tim's work because I know that he has what amounts to a captive audience for it: future licensees of the Unreal technology and mod developers for the titles that are built around it. So a certain level of success is effectively guaranteed. What's less clear is whether Tim will make his language efforts available outside the context of the Unreal technology and/or Epic Games—a business decision that is obviously left to him. If you haven't read Tim's comments here about his work, just type "site:lambda-the-ultimate.org tim sweeney" into Google.

Surely it's much easier to de

Surely it's much easier to design a pleasant syntax than to write a great compiler. Why, then, does O'Caml have the latter but not the former?

One possible answer is that the compiler can be incrementally improved and even replaced without disrupting the users. When it comes to syntax, you basically have to get it perfect the very first time, which is notoriously difficult in any domain...

Oops, The Concrete Dried...

Good point. I'm wondering too if designer-implementors often just whip up a syntax that will be easy to parse so they can get to the fun stuff, i.e. coding that great compiler. And then, before they know it, they have users, and they can't fix up the syntax even if they want to.

i was in your position for ye

i was in your position for years. when i really needed speed (and didn't have the patience to work out how to get it with haskell) i went to mlton rather than ocaml, because ocaml seemed such a mess (despite paul's advice). but mlton's compile times became too long as the code grew, and i was tempted by some of the libraries, so i eventually switched to ocaml, use a (standard ml) subset of the language (which was paul's advice too, iirc...) and, really, it's just fine.

not only is the debugger impressive (i don't normally use debuggers either, but got stuck/confused, tried it out, and was surprised), but the profiling tools are pretty solid too (i think they're pretty much "unix standard").

i doubt it will ever feel "really beautiful", but you do get used to it...

I've never needed a debugger?

Call me unrealistic if you like, but I found WardsWiki before I got around to learning how to use a debugger. I tried Extreme Programming early on and I've been hooked ever since. TestDrivenDevelopment lets me completely escape the need for a debugger. Projects with comprehensive unit tests don't need a debugger.

As for software testing in Haskell, QuickCheck integrates elements of unit testing, design by contract, and obviously generative testing.

I think that 'mainstream' languages are missing out because most of them don't have a generative testing tool like QuickCheck.

On a vaguely related note, I hacked up a test-first version of QuickCheck that reruns failed test cases next time. Now I don't need HUnit either.

--Shae Erisson - ScannedInAvian.com

Maintenance mode

What if the bug is in some substantial piece of code (such as a library) that you didn't write?

Never needed a debugger...

While Forget the debugger is a popular sentiment these days, especially among the XP crowd, it's always useful to have one handy. I won't argue with the notion that other ways of testing/constraining your code (and I listed several) are quite valuable--and probably should be the tool of first choice (a good UnitTest will help anyone else who tests the code; a debugger session will only help you).

That said, there are times where the code is behaving sufficiently strangely that white-box inspection of the machine state is the most useful way to identify and resolve a problem. (In many cases, the discovery of a problem while debugging ought to be instantly turned into a test case or assertion, to make sure it doesn't happen again). The best toolbox is the one with the most tools.

I've run into too many languages/tools, however, where use of an external debugger is the only way to see what caused something to occur. Quite a few languages (C/C++) don't provide portable ways to get backtraces, for instance.

I'd definitely put the debugger on the back shelf, and prefer the numerous techniques for preventing defects rather than detecting their presense, as the first line of defense. But I wouldn't throw it away altogether; unless you're sure that your unit test suite is sufficiently comprehensive that the number of defects remaining in your code is zero.

Debugger use case

As Ronny pointed out, one big reason debuggers are important in commercial development is for debugging code you didn't write. The reason a debugger is useful in that situation is because it can compensate for not having a very complete mental model of the program.

That's also the argument against debuggers: a tool that reduces the need to have a good mental model of the program is not necessarily a good thing. Linus' rant against debuggers is essentially based on this intuition: "It's that you have to look at the level _above_ sources. At the meaning of things. Without a debugger, you basically have to go the next step: understand what the program does. Not just that particular line."