New Paul Graham thing...

The Python Paradox


I mostly agree with the thrust of this piece, I think, but here's the most interesting bit (to me, at least):

Both languages [Python and perl] are of course moving targets. But they share, along with Ruby (and Icon, and Joy, and J, and Lisp, and Smalltalk) the fact that they're created by, and used by, people who really care about programming. And those tend to be the ones who do it well.
It's interesting that all of those languages are dynamic and thus favored by Paul Graham. Does he really think that, e.g., Haskell and Ocaml are being created by people who don't "really care about programming." Or is this just a cheap shot? Or are those languages really just completely off his radar?

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Self-Selection

It's true that Paul Graham's chosen examples are dynamic languages, but what he actually says is:
Hence what, for lack of a better name, I'll call the Python paradox: if a company chooses to write its software in a comparatively esoteric language, they'll be able to hire better programmers, because they'll attract only those who cared enough to learn it. And for programmers the paradox is even more pronounced: the language to learn, if you want to get a good job, is a language that people don't learn merely to get a job.
He explicitly acknowledges that "Python paradox" isn't a very good name, and the phenomenon he describes is parametrically polymorphic: if the big job-requirement language were to evolve to be Python, then the language to learn, according to Graham, would be anything-but-Python, and his argument doesn't change at all.

Now, having said all of this, I've read a number of Paul Graham's comments on statically-typed languages on the LL1 list, and I do have to conclude that he's one of the most ignorant DL apologists I've ever read. Oh well. He wouldn't be the first person to specialize heavily in a particular subject (not just the Lisp family, but specifically Common Lisp), accurately identify its strong points (first-class functions and metalinguistic abstraction via macros), and decide that he was going to exploit them for all they're worth, giving a tremendous amount back to the community in the process (his books are priceless). But this has all come at the cost of anything like an accurate understanding of the alternatives. It's generally not a problem—he doesn't seem to make the mistake of talking about ML/Haskell/etc. much. But when he does, *ouch*.

Ouch

It's generally not a problem—he doesn't seem to make the mistake of talking about ML/Haskell/etc. much. But when he does, *ouch*.

Ain't that the truth. :-) I'm glad to report that I recently read his new book and I enjoyed it immensely. There's nothing to fear in this department.

I wonder if his claims about other languages are strategic? I'm betting that Arc will benefit from the most critical feedback of any language in history. :-)

Well...

I'll call the Python paradox: if a company chooses to write its software in a comparatively esoteric language, they'll be able to hire better programmers, because they'll attract only those who cared enough to learn it.

I like this idea. I would have called it the erlang paradox or the haskell paradox...

Designers who don't care about programming?

they're created by, and used by, people who really care about programming. And those tend to be the ones who do it well.

And BCPL, and Wirth's languages, and...

Wait a minute! Have that many languages been designed by people who don't care about programming?

Wait a minute! Have that ma

Wait a minute! Have that many languages been designed by people who don't care about programming?
Exactly. Does he really want to argue that Kernighan, Richie, Gosling, and so on really don't care at all about programming? What about Guy Steele? He worked on Java, too, remember... Just seems like kind of a dumb thing to say...


But yes, I generally agree with the "paradox." When looking at resumes, I always want to see evidence of interest in and knowledge of non-mainstream buzzword topics. I'll always choose to interview someone who both (a) knows a language they don't think they'll get paid to use, and (b) puts it on the resume anyway. So, yes, apart from the one gripe, I totally agree with this article. I hope I didn't imply otherwise.

Caring About Programming

I'm going to defend Graham a little, having hit him pretty hard earlier on an unrelated topic. :-)

Graham comes from the Lisp tradition; he knows he comes from the Lisp tradition, and most people who read or hear him know he comes from the Lisp tradition. So when Graham talks about "caring about programming," it's pretty safe to assume that he draws his set of priorities with respect to defining his terms from that community. Given that, I tend to transcribe "care about programming" to "care about abstraction" in my head, and that seems to work pretty well: his example languages all allow the programmer to build abstractions pretty well one way or another, whereas the obvious unmentioned ones (Java, C, C++) do not.

Yes, I know about C++ and template metaprogramming and Boost's preprocessor library and... C++ makes a lot of things possible, in some sense, but in a profoundly hostile way relative to the alternatives.

Not inconsistent

If I remember correctly Paul Graham says nice things about K&R. I don't know much about Gosling, but Guy Steele's work on Java seems to fit well with Paul Graham's world view. On ll1-discuss GLS said:

    And you're right: we were not out to win over the Lisp programmers; we were after the C++ programmers. We managed to drag a lot of them about halfway to Lisp. Aren't you happy?

I have a feeling he is thinki

I have a feeling he is thinking about languages like PL/I, Ada and COBOL not about C and ilk.

Individuals and committees

These persons may individually care about programming, but the committee they are part of only cares about consensus. In general, languages designed by one person are clean and focussed, while languages designed by committee are messy and convoluted.

There are committees and committees

The obvious counter example is Haskell; but perhaps Haskell's was a different sort of committee to Java's?

Certainly I don't think that collaboration necessarily dilutes the commitment and "vision" of language designers. Rather like Eric Raymond, Graham tends to lean rather hard on an individualistic mythos - the lone hacker, expressing his genius - which I find seductive and questionable in about equal measure. It's clear that Graham approves of renegades, but that doesn't preclude the possibility of successful partnerships - Butch and Sundance, say.

The difference

The obvious counter example is Haskell; but perhaps Haskell's was a different sort of committee to Java's?

The difference being that designing the ideal theoretical language is more suited for design by committee than practical languages? Certainly when for practical languages a design requirement might be that 80% of all programmers employed by the BigCos have to understand it.

What do you mean by understand? ;-)

Certainly when for practical languages a design requirement might be that 80% of all programmers employed by the BigCos have to understand it.

I don't think that the languages used by these programmers are simple to understand, or that they understand them. For example trying to come up with a correct copy constructor in C++ is really tricky (or so C++ programmers tell me), after all we have to deal with memory management, exceptions, identity, equality, object slicing, etc., and these issues aren't simple. Let's take a look at Java: we have issues with synchronized keyword, proper implementation of equals and hashCode, many library warts, final, finally, exceptions, the subtle issues created by the language and VM specs, etc. I can point to many articles, papers, anedocts, etc. on these "language for the masses" that make them don't be simple. IME as a Java programmer usually other Java programmers don't understand these issues and just follow expert advice as commandments.


I think the problem isn't with understanding but with the approach used to teach the language. My ideas are in this old LtU thread.

knowledge or smarts

For Java and C, there's a lot you need to know. But most people can learn (memorize) all these facts.

Haskell otoh requires you to grasp some high level concepts. There is actually not that much to learn. The trouble is to understand it all.

Psychology of Computer Programming

In The Psychology of Computer Programming, Weinberg reports a study that showed that the more special cases and rules a programming language has, the LESS of those cases programmers actually use. You certainly see that in the Java community, where Java bloggers frequently post about discovering language features that are well described in the language spec. It will be interesting to see whether this effect is reproduced as people migrate to Java 1.5, sorry 5.0. The number of wierd interactions between autoboxing, coercion, inheritance and overloading might result in most Java code being written to a very restricted subset of the language.

Java and C# seem to be evolving by the designers adding features to simplify the job of programmers. If Weinberg is correct, this approach makes the job of programmers much harder. Perhaps a better approach would be to refactor the language: add features by designing a simpler, more expressive core language that can express existing and new language features as standard libraries.

The obvious counter example

The obvious counter example is Haskell; but perhaps Haskell's was a different sort of committee to Java's?

Also, wasn't Haskell originally quite closely based on Miranda, which was designed by one individual?

Python's David Mertz has done

Python's David Mertz has done a good job popularizing Haskell (and functional languages in general). His writings on IBM developerWorks and elsewhere probably helped drive me to reading up on it.

Further, I once wrote up how a lisper created a little type system to check certain consistencies statically. I would think the lisp community, which increasingly is interested in a multiparadigm view, is more welcoming to Haskell/Ocaml than mainstream languages. The perceived disjointness of dynamic and static typing (if that's what you're referring to) is a stumbling block, and I would like to see it eliminated.

I used to think all languages Pythonic

It's interesting that Haskell has been described as "the most Pythonic of all languages that have nothing in common with Python" (list comprehensions and indentation-based syntax being among the more obvious reasons why). The general flow of ideas I think is somewhat in the other direction.

the first that came to mind

I just listed the first languages that came to mind. But it is probably no accident that those came to mind first. Haskell, for example, seems to me a language that's designed more to write papers about than to hack in. I get the impression that using it would feel like reading a novel written by a literary critic.

See http://paulgraham.com/desres.html

Rhetorically compelling does not equal true

I get the impression that using it would feel like reading a novel written by a literary critic.

First, let me say that, like many people, I enjoy your essays and find them well-written and thought-provoking.

In line with most of your writing, the quoted sentence is quite rhetorically compelling: you evoke almost poetically the wisdom of the common man in the face of a self-absorbed intellectual elite. Many of us know that feeling; it is one of cornerstone tropes of Western culture, and anyone who has seen more than a few Hollywood movies will certainly recognize it.

The catch is that you are using this feeling to defend a self-absorbed intellectual elite: the mythical hacker who is a smarter and better programmer than the plebs who use other languages, say Java, C, C++, etc.

So how does the hacker get to be "just right" on the scale between mouth-breather and pretentious twit? If he can use the "overly-academic" card to neutralize an elite he fears may be smarter than him (Haskell users), why can't a Java pleb do him in with the same ploy?

various elites

There are various kinds of elites. The kind of hacker I wrote about, though smart, is not (usually) a professor.

So there are (at least) two continuums: one related to writing research papers, and one related to writing software. And "just right", at least when it comes to solving demanding problems, is as far along the latter as you can get.

Academic = Hacker?

There are various kinds of elites. The kind of hacker I wrote about, though smart, is not (usually) a professor.

I'm not sure the two are as far apart as you suggest.

A key attribute of both the academic and the hacker (as you have presented him) is that both tend to choose their projects based on the inherent interest of the problem rather than the demands of the marketplace.

Haskell is almost exclusively used to produce software to solve new and interesting computational problems. Perl and Python are widely used to solve already solved problems (e.g. yet another e-commerce app), and sometimes very badly.

Which is more likely to have a concentrated group of hard-core problem solvers using it?

Perl and Python...

Haskell is almost exclusively used to produce software to solve new and interesting computational problems. Perl and Python are widely used to solve already solved problems...

Which is more likely to have a concentrated group of hard-core problem solvers using it?

Well... from where I'm sitting, Perl and Python are used by quite a few academics in computational biology, because they are the quickest way to create the software supporting their main research, software that can end up being surprisingly complex. In machine learning, it seems like most research tends to get done in either Matlab (for speed of development) or C (for speed of execution). Haskell may have a higher concentration of hard-core problem solvers, but unless "hard-core" is defined exceptionally narrowly, I would guess that most hard-core problems are solved with other, "dirty" languages. And even when the problems solved are mundane, the resulting software, and the languages used, benefit from being developed by hard-core problem solvers in other domains.

Perl/Python vs. Haskell for biology

Perl and Python are used by quite a few academics in computational biology, because they are the quickest way to create the software supporting their main research

Who's to say one could not do the job as well, or better, in Haskell?

I'm sure that Perl and Python have more libraries available for such tasks, but surely there was a time, back in the jurassic period perhaps, when Perl and Python had no such libraries, and one would have said the same about them and C as you now say about Haskell and them.

I don't see that there is anything intrinsic about Perl or Python, as programming languages, that makes them better suited for that domain than Haskell. (Not that I know anything about that domain, of course. :)

Unbearable absence of libraries

Who's to say one could not do the job as well, or better, in Haskell?

I'm sure that Perl and Python have more libraries available for such tasks, but surely there was a time, back in the jurassic period perhaps, when Perl and Python had no such libraries

Perl didn't spring from Wall's head fully-formed, but it did come with enough libraries (and/or builtins) to make it an almost unambigously better alternative to shell script, and enough examples to whet the appetite (see the Perl 1 distribution). I've been impressed by how well Haskell is doing in the library department, but perhaps Perl's biggest advantage is in having libraries for the most obscure tasks. The only way to get these libraries in many domains is to have an enormous user base, since their pool of potential developers is so small.

As for intrinsic advantage, I think Haskell has at least one obvious handicap -- monadic I/O. Every minute I spend trying to understand how to program with it is a minute I am solving a problem I don't care about.

Perhaps the worst problem for a newcomer is that it makes printf-style debugging, a sort of lowest common denominator for program understanding, impossible. Say I write a beautiful quicksort in Haskell:

qsort [] = []
qsort (x:xs) = [y | y <- xs, y < x] ++ 
                  [x] ++ [y | y <- xs, y >= x]
but it is too slow. I suspect it might be because my data is ordered such that I choose bad partitions, and decide to print out the partition sizes at each step. Wanting to do so is evidence of a misunderstanding of the program I've just written, but what would apparently be a simple one-line change in another language becomes a frustrating trek through Monad-land. I gather unsafePerformIO, Hood and/or Hat may improve this, but the first sounds, well, "unsafe", and the last two aren't standard issue, and therefore probably aren't found with many installations.

A similar situation occurs with profiling, another common task that is, if not harder, at least completely different in Haskell. Perl and Python are easy to pick up because there is very little startup cost for solving simple problems. I'm not sure about eventual suitability for the domain, but the large conceptual overhead for solving simple problems makes Haskell IMHO a poor choice for someone whose primary interest is the application domain, not the software itself. Even Lisp, which lacks this conceptual overhead, has famously suffered even for its syntactic overhead; this doesn't make me optimistic about Haskell.

Bah, humbug!

perhaps Perl's biggest advantage is in having libraries for the most obscure tasks.

Yes, that is my point. It's biggest advantage has nothing to do with the language itself, but rather the social/historical context. And that context could as easily have been/be established for Haskell, if people (had) want(ed) it to be that way.

(I'm sorry but I really need to get this off my chest, again:) Scripting languages thrive on hype because they have no other advantage; their advantage is the hype, and the popularity which accompanies it. Sure, Perl may be an improvement on sh (frankly, I am not so sure :) but it is hard to find a strawer man.

(Man, what the hell kind of marketing campaign is that? "Perl: Better than sh!"??? How about, "Bush: Better than Osama!" or "Spam: Better than starving!" Then again, spam is considered a delicacy in certain countries...)

Consider Scheme: Scheme has been—for, what, fifteen years now?—superior to extant scripting languages in nearly every way as a programming language, and continues to be so, even though it has changed only a little, but no one wants to use it, simply because it isn't popular. Scheme was a great language, even when Perl was still a glimmer in Larry Wall's eye. But, no, we can't use Scheme for scripting tasks, right, because it was already invented, and moreover by academics, and it has far too many parentheses. Instead, we need a new language like Perl, which is based on sound principles like DWIM, TMTOWTDI and Mr. Wall's personal theory of natural linguistics. Brilliant.

Most programmers don't want better tools as much as they want self-validation and a veneer of novelty. Inasmuch as library size is a measure of popularity, it doesn't reflect on the usefulness of the language itself.

Perhaps the worst problem for a newcomer is that it makes printf-style debugging, a sort of lowest common denominator for program understanding, impossible.

Here is the standard retort: It is a problem for newcomers who come from procedural languages, because they were miseducated, but it ought not to be a problem for non-programmers, which is who we expect domain experts to be. Haskell's computational model is far closer to the mathematics that we learn in high school than that of procedural languages, and it lets you reuse those principles. In arithmetic, there is no printf. Nor is there an assignment statement, any form of imperative loop, or array, malloc, free, pointers, setjmp or a hundred other of the things one needs to learn to use a procedural language. In arithmetic, referential transparency holds and a variable is just a name for a value and it doesn't matter what order you evaluate an expression in.

Anyway, we all know the problems that arise from relying on execution order, especially in the presence of state. It bites programmers every single day, but in procedural languages this problem is only dealt with haphazardly, one a case by case basis, or by OO-style objects, which are really only a convention for encapsulation of state. The need in procedural languages to always deal with state is a symptom of a poor separation of concerns.

But, I don't want to get into this debate...

Trying to avoid "this debate", but...

I don't want to reopen the age-old debate, either (I'll try to steer clear of "human factors";), but just a couple of points:

Man, what the hell kind of marketing campaign is that? "Perl: Better than sh!"???

Actually, a pretty good one, with slightly different wording -- "Are you currently using X to perform some tasks? Here is Y, and here is how it can better perform them." It is addressed to someone who, like most people, has already found a tolerable way to perform various tasks, and who is therefore open to suggestions for improvement, but skeptical of suggestions that he completely discard that solution and try something radically different. This isn't "hype". Instead it is catering ("slavishly", I hear you saying) to users' expressed needs.

Most programmers don't want better tools as much as they want self-validation and a veneer of novelty

And I assume category theorists are above this.

Haskell's computational model is far closer to the mathematics that we learn in high school than that of procedural languages, and it lets you reuse those principles.

Agreed, but a lot of programming addresses problems that are not mathematically formalized, though they may be formalizable. If the problem description is informal, like a sequence of steps that one typically performs by hand, but wishes to automate, high-school math may not be the most "natural" (by which I mean most likely to occur first to someone with a typical non-programming education) way to express it. I know this is close to the dreaded "HF", and I don't have empirical studies to back it up, but it seems reasonable. When you decide to script something you have been doing by hand, do you (a) formalize it first, or do you (b) go brashly forth and write out the ugly, temporal, stateful steps, then debug the result? (note: if you answer "a", then maybe category theorists really are allowed to condescend to us mere humans)

Cookbooks do differ

When you decide to script something you have been doing by hand, do you (a) formalize it first, or do you (b) go brashly forth and write out the ugly, temporal, stateful steps, then debug the result?

I do not see (a) and (b) as either exclusive or exhaustive. Formalization can be ugly, or temporal, or stateful, or have any combination of these properties.
Also, I don't agree that sequential description is more natural (you are right, it's HF after all). When cooking, I use constraints and futures, and not sequences :) Then again, that can be unique to my cuisine...

The newbies shall inherit the earth

Scripting languages thrive on hype because they have no other advantage; their advantage is the hype, and the popularity which accompanies it.

...

Most programmers don't want better tools as much as they want self-validation and a veneer of novelty.

I think that one point is missed here: the popularity provides support for peers who operate roughly at each other's level, e.g. providing explanations in terms the recipient can understand, even if those explanations would make an academic or a competent professional developer grind his teeth.

A language like Haskell acts as a selection filter for such communities: either sufficient people of a certain level just avoid the language, or features that are considered barriers would have to be modified to make the language more comfortable for the community. At the very least, in Haskell's case, you'd end up with a strict, mutating language instead of a lazy, pure one, not to mention the slew of imperative control structures that would be added. ;)

An argument one sees occasionally, e.g. on comp.lang.lisp, is that it's not a good idea to pitch language features to newbies, because soon they're not newbies and don't need them, and it's really the more advanced, general, powerful features that are important and help you do the hard stuff more easily. That all makes perfect sense, in the LFSP context, but if popularity is the goal, then this argument doesn't take the previous paragraph's point into account.

A significant proportion of the userbase of some of these languages are perma-newbies, who either don't write a lot of code or just aren't very good at it; plus, there's a constant stream of fresh newbies. If a language doesn't pitch features to newbies, that decision selects out a huge potential market, not just of the newbies themselves but the entire community that goes along with them - people who enjoy helping newbies, people who don't want to feel as though everyone else using the language is smarter than them, etc.

So, the lesson for language designers who want their languages to be popular is, concentrate on the marketing and feel-good messages - DWIM, TMTOWTDI, principle of least surprise - that appeal to newbies, because newbies are the plankton of the popular language ecosystem.

Then Haskell is a blue whale...

because newbies are the plankton of the popular language ecosystem

Setting aside the question of catering to plankton as a language design strategy ;-), I think this is beside the point in the overall context of the thread.

Paul Graham's original contention was that certain languages appeal to the best programmers. If they are really just following the lamest programmers, because that is where the libraries and support community are, then we have shown nothing that distinguishes good from bad programmers. (It would be a bit like saying good programmers all share the characteristic of breathing.)

If language choice is an indicator of a good programmer at all (and I think most of us here would dispute this in one way or another), we would have to show that such a "filter" language possessed some feature that only a good programmer could use effectively, or that had appeal to a good programmer but not to a bad one.

Unfortunately, the only obvious candidate for such a feature has already been discussed to death in another thread... ;-)

Me, offtopic?! :)

I think this is beside the point in the overall context of the thread.

Paul Graham's original contention was that certain languages appeal to the best programmers. If they are really just following the lamest programmers, because that is where the libraries and support community are,

Let me first restate my conjecture more generally, since I think it loses something of what I intended to convey when summarized as "following the lamest programmers" (although the way I stated it, that certainly seems to follow).

The general principle I was getting at, in response to Frank's post, is that barriers to adoption have a compounding effect on the growth of communities, and the relative unpopularity of some languages can be at least partly explained in those terms.

Based on this blander restatement, please now imagine my newbie-centric post rewritten in more politically correct terms. ;)

then we have shown nothing that distinguishes good from bad programmers.
You're right. When it comes to that, I think the point Paul mentioned about Google looking for Python knowledge when hiring Java programmers is more significant for the greater breadth of knowledge it implies, than for the specific other languages that are known. The choice of a specific language like Python seems more likely to have to do with corporate culture than anything else. I mean, wouldn't a Java programmer who loves Scheme be acceptable? (Maybe not, if they're concerned with a creating a kind of corporate monoculture.)

Newbies are people too (just barely... ;-))

Based on this blander restatement, please now imagine my newbie-centric post rewritten in more politically correct terms

Political correctness really isn't my issue. ;-)

I think the point Paul mentioned about Google looking for Python knowledge when hiring Java programmers is more significant for the greater breadth of knowledge it implies

You may think that, but that isn't the position I saw in the essays.

On the contrary, the argument seems to be:

1) There is some minority of programmers that are just way better than others.
2) These programmers gravitate to language X
3) Therefore, if you want to hire the good programmers, look for skill in language X.

As your newbie argument shows, the statement that "good programmers like X" is in no way equivalent to "all (or even most) X users are good programmers", which is what you would need to make the argument work.

If we accept premise 1, one winning strategy if you want your language to be popular is to cater to newbies and/or lamers, since there are more of them to draw from.

This still does nothing to explain what language the superior programmers prefer, if such a language exists. (Or even if these superior programmers exist. ;-) )

State and execution order

Here is the standard retort: It is a problem for newcomers who come from procedural languages, because they were miseducated, but it ought not to be a problem for non-programmers, which is who we expect domain experts to be.

So Haskell is only for domain experts?

Haskell's computational model is far closer to the mathematics that we learn in high school than that of procedural languages, and it lets you reuse those principles.

But a procedural model is closer to what I do in real life. We do things sequentially, and sometimes (when using a scripting language) I am programming the computer to do what I would do sequentially and by hand.

In arithmetic, there is no printf. Nor is there an assignment statement, any form of imperative loop, or array, malloc, free, pointers, setjmp or a hundred other of the things one needs to learn to use a procedural language. In arithmetic, referential transparency holds and a variable is just a name for a value and it doesn't matter what order you evaluate an expression in.

That's fine, but there's a tool problem in Haskell without printf. There is no debugger. Oh yes, there's Hood, and Buddha, and hat, but none of them support the whole range of extensions GHC supports. You could argue that we shouldn't use those extensions, but then you have to defend a Haskell without them, which gets much trickier.

Now, I would be happy with a good Haskell debugger. Even so, it's not very inviting to ask people to learn a language when they have to learn an additional tool (as opposed to printf) before being able to debug.

Anyway, we all know the problems that arise from relying on general recursion, especially in the absence of printf. It bites programmers every single day, but in purely functional languages this problem is only dealt with haphazardly, on a case by case basis. The need for purely functional languages to always deal with termination drops serious mathematical quandries in the lap of Suzy Q. Coder

Yes, I'm reaching. No, I don't actually think that's true, but I do think recursion and state are both valuable features. I think I could have taken that paragraph and substituded monads, or laziness, or just about anything; my point was that it contains claims, but no warrants or data. So, I would ask for some explanation of why the benefits of the simplicity of state do not outweight the disadvantages, but:

But, I don't want to get into this debate...

Another time, then.

Roly-poly fish eyes

Just for my amusement, and since I am in a flippant mood today, let me address this old canard:
But a procedural model is closer to what I do in real life. We do things sequentially, and sometimes (when using a scripting language) I am programming the computer to do what I would do sequentially and by hand.
OK, let's see, how many ways can I attack this?
  • The point of programming is not to replicate how you have to do things in real life; the point of programming is to make things easier to do. In "real life", to get to my office, I take a certain path to work every day. In a program, I would rather not specify a particular path, but just specify the endpoints. In some cases, the particular path is relevant, and I can fix it if I please; but in most it is not, and being able to postpone that choice increases separation of concerns and decreases cognitive burdens.
  • Sequential processes are a subset of parallel ones.
  • Wrong, "real life" (whatever that is) is not sequential—it's massively parallel. Your life might be sequential, but there are other people in this world, and other agents, processes, particles, all interacting, and there is no linear order on them.
  • Almost every form of I/O is non-sequential; as soon as you talk about networks, other processes and threads, even terminal I/O (which only presents parallel processes as sequential ones, by interleaving their output), you see parallelism. If you've ever tried programming a robot, or writing some kind of VR environment (say a first-person shooter), too, which certainly comes about as close to "modeling real life" as we can get, there is lots of parallelism, and trying to express it sequentially is hard. Ultimately, the reason is that there are many ways to sequentialize a parallel process, and all of them lose information and decrease locality.
  • Well, let's look at how more knowledgeable people than us have tried to characterize this "real world" you speak of. We have physics, chemistry, mechanics, engineering. All of these are based on what we regard as standard mathematics, which is in form much closer to the formalism of Haskell than any of: Perl, Python, C, C++, Java, Pascal or even ML. If you are really serious about talking about the real world, it would behoove you to reuse that theory using a language which makes it as easy as possible to do so.

For the rest:

So Haskell is only for domain experts?
Yes, precisely! In fact the Haskell specification takes great care to note that any programmers found using Haskell are subject to summary execution. Haskell was absolutely not designed for programming and is completely unsuitable as a language for writing programs of any sort!
That's fine, but there's a tool problem in Haskell without printf. There is no debugger. Oh yes, there's Hood, and Buddha, and hat, but none of them support the whole range of extensions GHC supports. You could argue that we shouldn't use those extensions, but then you have to defend a Haskell without them, which gets much trickier.
I don't have to defend that. I only have to point out that the lack of debuggers for full GHC Haskell is a defect in the implementation status, and not the language itself.
Yes, I'm reaching. No, I don't actually think that's true, but I do think recursion and state are both valuable features. I think I could have taken that paragraph and substituded monads, or laziness, or just about anything

Haskell has state, (proper tail) recursion, referential transparency, monads and laziness. Pick almost any language X: it has a proper subset of these features. (BTW, monads are not really a feature; they are just part of the library, with some syntax support.)

But, as it happens, and since you bring it up, I think general recursion should be controlled, same as any other side effect, and similarly I don't like laziness-by-default.

my point was that it contains claims, but no warrants or data. So, I would ask for some explanation of why the benefits of the simplicity of state do not outweight the disadvantages

This debate is so old that the supporting arguments are not worth repeating. Just look at Wadler's paper, How to declare an imperative, for example.

Scheme

Consider Scheme: Scheme has been—for, what, fifteen years now?—superior to extant scripting languages in nearly every way as a programming language, and continues to be so, even though it has changed only a little, but no one wants to use it, simply because it isn't popular.

I for one have tried using Scheme (Scsh) for my shell scripting, but to my surprise it turned out that I prefer using bourne shell.

There's a huge difference between theory and practice. I think it's your loss not to appreciate this.

Turtles all the way down

There's a huge difference between theory and practice. I think it's your loss not to appreciate this.

If the theory doesn't fit the practice, you're using the wrong theory, or you're using the right theory wrongly. I think this is what you don't appreciate.

Insisting, like you, as a maxim on some ubiquitous gap between theory and practice isn't constructive. In the absence of theory there is no practice, only happy coincidence, from which grows superstition and mysticism.

I might be wrong in suggesting that Scheme is well-suited for scripting; if so, and I discover the reasons for it, I will improve or revise my theories. But someone who follows your philosophy will never even look for the reasons, putting it all down to the gap between theory and practice.

Paradigm vs. Language

Luke Gorrie: I for one have tried using Scheme (Scsh) for my shell scripting, but to my surprise it turned out that I prefer using bourne shell.

FWIW, I'm an old Schemer from Indiana (literally, born and raised there) and I prefer bash to scsh also. I take that to mean that familiarity counts, rather than that Scheme is bad for scripting or even that scsh is a bad implementation of scripting-in-Scheme. Besides, I also have DrScheme as a counterexample: it makes a very nice programming language implementation/web services/debugging/etc. scripting environment using Scheme.

So maybe it would be worth trying some other functional shells and seeing whether the issue really is about the "Scheme nature" or about accidents of the language that might be more about personal experience and taste than anything else. I found Es: A shell with higher-order functions, fsh—a functional UNIX command interpreter, and of course scsh itself. es looks pretty interesting to me.

Luke: There's a huge difference between theory and practice. I think it's your loss not to appreciate this.

As usual, I'll just ditto Frank's response to this. The assertion suggests replacing or refining the theory; positing that there's necessarily a huge difference between theory and practice is anti-scientific nonsense.

Okay

Sure. Let's just agree that theories about how to write programs shouldn't be considered correct-until-proven-otherwise.

I've found that it's an interesting experience to mistake SICP for a book on software engineering, for example.

Esther

Also see:

Arjen van Weelden, Rinus Plasmeijer. A Functional Shell that Dynamically Combines Compiled Code. Proceedings 15th International Workshop on the Implementation of Functional Languages, IFL 2003, Selected Papers, Edinburgh, Scotland, September 8-10, 2003.

which is from the Concurrent Clean group.

Haskell can be quite Unix-y

inasmuch as function composition can work in a similar way to Unix pipes. Consider the following rather dull example, which prints a list of all the alphabet characters in an input string:

writeLine x = do putStr x; putStr "\n"
makeSentence x = x : " is a letter in the alphabet"

conv = mapM writeLine . map makeSentence . filter isAlpha

main = do chars <- getLine; conv chars

The function conv is defined by stringing a few other functions together: filter to select the inputs we want, map to transform them, and mapM to perform an action on each of them. It seems to me that quite a lot of "scripting language" programming - programming to automate some batch task or other - runs along similar lines.

Monadic I/O and printf

It's been a while since I tried to do anything much in Haskell, but I would like to have a go at contesting the meme that monadic I/O is difficult to understand (or, at any rate, that it's difficult to do). Here's a short Haskell program that allows a user to build a list of words by inputting them one at a time. Observe the liberal use of printf statements.
main =
	do	wordList <- loopWhile continueq getNext []
		printf (show wordList)

loopWhile cond next init =
	let loop = loopWhile cond next in
	do	new <- next init
		continue <- cond
		if not continue then return new else loop new

continueq =
	do	continue <- input "Continue?"
		return (continue=="y")

getNext wordList =
	do	nextWord <- input "Enter a word:"
		let newWordList = nextWord : wordList
		printf ((show (length newWordList)) ++ " words collected")
		return newWordList

input str = do putStr str; putStr " "; getLine
printf val = do putStr val; putStr "\n"

Observe also the number of type annotations in the code (in actual fact, I put them in while I was writing it and then took them out again once I'd finished).

Admittedly this is a very trivial program. But it gets you over the main hurdle for writing interactive programs: it accepts inputs, processes them and writes out results. Many more complex programs can be built around the same (or a similar) skeleton.

Different problem.

The problem with monadic I/O w.r.t. debugging isn't so much writing programs at all, but modifying programs to do I/O. It's a sort of tar baby, in that you find yourself warping your entire program to add I/O to a leaf routine that didn't have it before, because the I/O changes the function's type. For example, say you have a complex comparison routine that may contain a bug, and the bug shows up when you sort a large pile of data. Before, we have:

Lisp:
    (sort things #'myfunc)
Haskell:
    sort myfunc things

After:

Lisp:
(sort things
      (lambda (a b)
        (let ((res (myfunc a b)))
          (if (not (member res '(t nil)))
              (format t "(cmp ~s ~s) ==> ~s~%"
                      a b res))
          res)))
Haskell:
    -- ???

You've just changed the type of your comparison function in Haskell, so you need to change sort, all its callers, etc., and that's just ugly.

Re: Perl/Python vs. Haskell for biology

There's at least one person using Haskell for bioinformatics.

At least one person using Haskell for biology

Amanda Clare is (or was) one more. Her paper makes for interesting reading even if bioinformatics is not your thing.

But bioinformatics is a very bad choice of example in a discussion of "true hacking" because most writers of bioinformatics programs are motivated by the biology, not the programming (in fact they are often biologists) and do not care how ugly the code is as long as it gives them some results, so they gravitate towards whatever is popular (e.g., perl), as it is only a tool to them. Thus IMO bioinformatics, on the whole, falls into the ugly web app category, really.

This is not to belittle bioinformatics (or web apps, for that matter) but I am getting tired of hearing programmers from outside the field reflexively go "Bionformatics? Oooh, that must be so exciting..." Well, not necessarily.

Unwanted laziness

From the paper of Amanda Clare:

The main disadvantage that was faced was dealing with unwanted laziness.
In a data mining environment, where all data needs to be read and counted,
and all calculations will be used, laziness provides few advantages, and usually
takes up huge amounts of heap space while delaying all computation until the
last minute. At every step of the coding process it was found that code was easy
to get right, but then difficult to get working in practice without running out of
memory.

So, in practice, it might be beneficial for a lazy PL to not only provide forcing primitives, but probably also support regions of default evaluation strategy switched to eager (though I do not know how to limit these regions).

Re: Unwanted laziness

In any case, didn't SPJ himself concede recently that laziness is overrated and that he was no longer sure it should be given such pride of place in Haskell?

There's laziness and there's laziness

If we compare Haskell and Clean we have two different strategies of dealing with laziness. In Clean you can add strictness annotations to function's arguments while in Haskell you use strict datatypes or sequencing operations (e.g. seq, deepSeq). AFAICT Clean programs are much simpler to optimize for strictness.

Standard implementation

Who's to say one could not do the job as well, or better, in Haskell?

I think Haskell's lack of a standard, simple to use, cross-platform implementation hurts quite a bit here. As an aside, I'm always surprised at how many caveats are involved in this page. (And as another aside, I'd love to hear from anyone doing actual software development with Haskell under Windows.)

Usable Haskell Systems

James Hague: I think Haskell's lack of a standard, simple to use, cross-platform implementation hurts quite a bit here. As an aside, I'm always surprised at how many caveats are involved in this page.

From that page:

The Glasgow Haskell compiler is a full implementation of Haskell, also offering many language extensions... It is available for a few Unix platforms (including Linux, Solaris, *BSD, and MacOS-X) and also for Windows.
Yes, the ellipsis means that I left something out. Some of it is about additional features, but some of it notes that GHC is slow and wants a lot of memory. Hmmm. Relative to what, I wonder? g++ 3.3 on my 256M Mac OS X box won't compile Crypto++ 5.2 because it runs out of virtual memory after running overnight. GHC 6.2, OTOH, compiles every release of darcs so far in about half an hour. YMMV.

Perhaps you're referring to IDE support. I was able to find hIDE, KDevelop, Vital, Haskell Support for Eclipse, Haskell support for JCreator, and of course Haskell modes for the popular extensible text editors, generally complete with interaction with either Hugs or GHCi.

Whatever Haskell's limitations, full implementation and platform availability doesn't seem to be among them.

Hugs

Again, not really a discussion I want to pursue but, you cannot fairly compare the compiled implementations of Haskell with the interpreted implementations of scripting languages, which are far easier to port. Better to compare with Hugs, which, as the page says, indeed runs on just about any machine.

Really?

I am not sure I agree. It's like saying that no one should learn lisp because there are no job ads for lisp programmers in your local paper. It sounds convincing, but it's not really a valid argument.

A tool (i.e., a programming language) can be useful both for writing papers and for writing software (emacs is :-). The fact that a language inspires many researchers shouldn't be held against it. You should examine the language and see how and when it can be useful.

Moreover, if the papers we are talking about are concerned with software development activities, than hopefully these papers -- if they are any good -- tell us something about real life programming.

I think the examples I gave of Haskell DSL design are a case in point. If one thinks DSLs can be useful in practice, one should look for good ways to design and build DSLs. I think the Haskell examples are inspiring.

Most important of all, I think curiosity is the key trait you are looking for. And curious folks learn esoteric languages. Why? Because they are curious to know what it feels like to program in them, of course...

Depends on the domain

For instance, Haskell is a very nice language to write papers *in* - it comes far closer than python to "executable pseudocode".

Teeter-Tottering on Languages

Paul Graham: I just listed the first languages that came to mind. But it is probably no accident that those came to mind first. Haskell, for example, seems to me a language that's designed more to write papers about than to hack in. I get the impression that using it would feel like reading a novel written by a literary critic.

I have to agree: I've never been able to be down with Haskell, despite a general appreciation for functional programming, type inference, and lazy evaluation. Unfortunately for those of us who want to be able to associate a joyful, revelatory experience with adopting a new language—a feeling I most definitely associate with the Lisp family, the ML family, Smalltalk, and Oz—the Haskell experience for me just felt bewildering. Needing to learn about monads just to do I/O and have mutation was all but a non-starter. Given how successful others have been at implementing/explaining monads via Scheme or O'Caml or other languages, I have to say that Haskell is the only functional language I've ever used that left a taste in my mouth suggesting a need for more sugar. :-)

Having said that, it's also true that Haskell was deliberately designed for purity and as a grand testbed for future ideas in functional languages. I doubt anyone can reasonably argue that it's failed to meet those goals when you look at Haskell's evolutionary history and the gap between the Haskell 98 spec and the features of a top-flight implementation like GHC. "The awkward squad" have earned their name, but a level of mastery of them can't help but meet the Alan Perlis standard of changing the way you think about programming. But what I find myself doing is learning concepts from Haskell, and using them in O'Caml.

Hacking Haskell

As far as personal taste goes I am more of a Scheme man myself, but I feel I have to stand up for Haskell. True, it can take getting used to - but there are nice examples of Haskell hacking out there. Quite a few "algebra of programming" papers mentioned here in the past fall in this category.

Which brings me to ask LtU readers to nominate cool and/or accessible examples of Haskell hacking. Links can point to code or to papers and articles. Extra credit will be awarded for pointing to previous LtU discussions.

QuickCheck

To me, QuickCheck (yay, extra credit! :) ) is the best example of an extremely cool and useful Haskell "hack" (in the best possible sense of the word) out there. The source code is sufficiently small to understand in one sitting, but it is still sufficiently clever, specially when it comes to generating random functions of arbitrary types. Everytime I look at QuickCheck, I think: "What a cool idea". Most of the advanced stuff in Hudak's "Haskell School of Expression" also left me with similar, if not as strong, feelings.

Functional Unparsing and Sorting Morphisms are two other papers that while not directly related to Haskell, would fall into this category, because of the interplay between research (in type theory, etc) and applications they show. The "Functional Unparsing" paper, specially, gives a quite cool way to alleviate the need for dependent types in some situations.

QuickCheck

I'd like to read about some more uses of QuickCheck. The idea is great but I think there's a knack for knowing when it's applicable.

I was excited about QuickCheck when John Hughes presented it at last year's Erlang conference. Unfortunately I couldn't figure out how to apply it to the problem I had in mind, which was a small in-memory database. I had trouble formulating properties that would be easy to check -- they always seemed to require the checker to be a reimplementation of the checkee in order to know the right answer.

On reimplementing the checkee as a checker

I've experienced this too.

I agree with you that it is sometimes hard to think of what properties you want to test, but I think this is something we should be doing anyways, as part of a "real" software engineering discipline: since doing proofs is hard, and arguably boring, we are helped by a systematic way of trying to find negative proofs.

With properties, the intent of the program becomes clearer: properties are excellent documentation. It's like having blueprints for a mechanical part, a formal description of the expected behavior of the part in terms of physical quantities (ie, a formal description), _and_ a way of testing whether those expectations hold (modulo test quality issues).

To me, this is the really cool part about QuickCheck: it brings "coding by specification" so much closer to practice, which by itself is of tremendous value. It feels closer to doing algebraic specifications than anything else I've used.

QuickCheck can also be used as a smart assert: instead of trying to prove stronger properties of the program, you can use it for sanity checks of data structures, etc. that don't incur in run-time overhead (it may feel weird to talk about performance in Haskell, but some of the checks I wrote are pretty slow :).

My most "advanced" quickcheck uses were probably when I was hacking some geometrical data structures in Haskell, and I needed all sorts of sanity checks. For example: when implementing a triangulator for polygons, I checked whether the sum of the areas was equal to the area of the polygon, (yes, there are floating point issues), whether all triangles are disjoint, and whether they cover the whole polygon. My BSP tree tester would check the code to generate the polygon of a leaf node, so that it didn't overlap with any other leaf polygon in the tree; that the sum the the areas of two children have to be equal to the area of the parent; that a parent polygon contains both children; that all of the polygons were convex, etc.

Another trick that may help to avoid the reimplementation issues is to have the tests use different parts of the same program. For example, you have two procedures, A and B, and then state the properties of A with regards to B and the properties of B with regards to A. If the checks fail, you won't know which procedure is wrong, but usually the counterexample QuickCheck gives helps.

I have very little experience implementing databases, but I'd try to figure out the properties from relational algebra (was it a relational DB?). For example:

- the size of 'A join B' is at most |A||B|;
- if you do a 'A join B' and then select back all the keys and columns from A, you should get exactly A;
- A union A = A;
- for every 'A union B', a select of a single item from either A or B should return exactly one row;

etc. I'm probably showing my DB ineptitude, but I hope you get the idea: although these don't give you everything you need to describe a relational DB, they are all valid properties that the DB operations should hold. The properties don't have to be sufficient to describe the program algebrically. If you can pull it off, great, but this doesn't stop you from using QuickCheck.

QuickCheck is not quite there when it comes to dealing with state, though, and that is probably the largest setback right now. (shameless plug: This is the reason my Scheme port of QuickCheck is little more than an exercise in getting typeclasses in Scheme: pure code can be easily tested, but stateful code is a whole different story which I honestly don't know how to deal with. Obviously the same problem would happen if you want to test a program inside the IO monad, but since programmers are discouraged to do anything not strictly necessary in it, this is not as big of a deal)

Thanks for the examples

Thanks for the examples. I can probably come up with some properties with more thought.

I'm not sure this little database has such nice properties as relational algebra though. It's designed to exactly match the semantics of the command-line configuration interface on an existing line of ethernet switches.

Revision Control in Haskell

I think DARCS, a revision control system implemented in Haskell, qualifies as a cool hack (and gets me my extra credit).

I think Haskell is a very cool language - anyone with the slightest respect for the lambda calculus must see that. But, not having the ability to resort to a little mutation if you feel the need, is a big restriction to live with for real projects. That's the reason I tend to prefer ML, which also has the benefit of being a lot like a typed Scheme.

Darcs

I'd nominate Darcs as the program of the year. It's much more sophisticated than CVS and yet much simpler to use and comprehend. I love it.

Darcs, Unison, and the ICFP contest winners are great ambassadors for Haskell and ML. Because of them (and The Evolution of a Haskell Programmer) I actually do think that those languages are important and I do write a small program in them every so often in the hope of enlightenment. (So far it just feels like writing Erlang with a slightly different syntax. I suspect I'm not writing the right sort of programs.)

I'm also pleased that although Darcs does work very well it also has bugs (see the mailing list), and they're exactly the sort of bugs that I expect real programs to have. Somehow it's reassuring to see Haskell connect with reality in this way.

My list

It seems that I have a different taste in hacks. I prefer cool coding examples, even if they lack direct practical use.

Saying that, here's my list:

Many examples of DSL design. On the top of the cool scale I'd put Pan/Fran and Haskore. So easy, yet so much fun.

I am also a fan of the origami school, and would recommend the maximum segment sum paper (from O(n^3) to O(n) by program transformation) and the radix sort paper. Both by Gibbons.

"The Fun of Programming" is a good choice, if you are looking for a book on Haskell hacks.

printf debugging

Since the focus of the debugging effort is myfunc, perhaps we don't need to be passing it to sort in order to test it (after all, all sort does is pass it pairs of values to compare). Why not mapM (printResult . myFunc) [some list of tuples of test values]?

Admittedly, this is a different debugging method to inserting printfs, so I am effectively admitting that it's difficult to do that in this case. But in the Haskell world, changing myfunc to a function that performs I/O really does change the meaning of myfunc and everything that uses it. It's difficult to do precisely because it's a significant change. It isn't so much that Haskell is unduly fussy about such things, but that programs written in Haskell are programs in which such things really do matter.

Here's a question: given a "pure" Haskell program of type a, would it be possible to transform it automatically into a program of type Debug a in which stack traces could be recorded, breakpoints set, and assertions and printfs inserted at arbitrary points in the code? I think a manual transformation would certainly be possible in all the cases we've discussed so far - are there any impossible or undecidable cases?

Just an example

This was just an example, but there are some reasons you might want not to pull out myfunc and test it -- the sort does a particular n log n comparisons, one of which hits the bug, but finding which would require n^2 comparisons. While I realize that in one sense it's a significant change -- adding a side effect to the comparison -- in another sense it's utterly insignificant. The function still takes two comparable objects and returns the same boolean result. If the order of in which debug messages are printed does not matter, the change is even less important.

The problem is that something that looks superficially similar to what you would do in another language means something completely different in Haskell. When you run into the difference in a case like this, the solution to what appears to be a "local" problem is to either globally and significantly change the way you reason about and write your programs, or thread a "world" argument through the whole thing for a single printf. Unless there is an automatic transformation like you describe, this just looks ugly.

Trace?

I'm sure you already know this, but for the record, Haskell does provide the "trace" function, which is not in the IO monad, and is intended for just the same sort of debugging that printf is used for.

Trace??

I was about to be somewhat ashamed -- I'm not a Haskell programmer, and I have never heard of trace. But are you sure it's standard?

Prelude> :t trace

Variable not in scope: `trace'
Prelude> Leaving GHCi
rh220-50:src% ghci --version
The Glorious Glasgow Haskell Compilation System, version 6.0.1

Debug.Trace

Not sure if it's standard, but it's well supported. It's in a different module, though, so you need to import it.

Prelude> :module + Debug.Trace

Prelude Debug.Trace> :t trace

trace :: forall a. String -> a -> a

A better example?

I'm pretty sure there must be better ways of locating a bug in a comparison function than relying on a sort algorithm to come up with the particular test case for which it fails. I can see that it would be frustrating, in the event that you had detected a failure that way, if you couldn't then find out what particular value the function had failed for.

I mentioned the possibility of "out-of-band" side-effects in another discussion a while back: side effects that take place in what from the program's point of view is a parallel universe (a separate value of "world"). The programmer, as overseer of all possible worlds, gets to observe these side-effects, but referential transparency is preserved for the program in its own world (since leakage between dimensions is one-way). Maybe you could do this in Haskell using various shady operators that pretended to be id, but actually funnelled information into the parallel world. Of course, you'd have to teach compilers not to optimise the fake ids away (in debug mode, at least). The fact that I know very little about compilers is probably the reason why I don't instantly see why this is a stupid idea.

Not a new idea

See http://www-users.cs.york.ac.uk/~olaf/PUBLICATIONS/transHaskellTrace.pdf for a paper on Hat, which performs more or less the kind of program transformation I was describing.

(edited) - actually slightly different, as it uses side-effecting FFI calls outside of the I/O monad rather than threading the state of a debugger through the program.

Java Defence

This doesn't seem to cover anything really useful. Maybe that in itself says something. From /. (so currently very slow).