Lambda the Ultimate

I agree. Some of the talks weren't even vaguely on-topic. I was expecting each presenter to propose some general structure to abstract over the huge mess that is the current state of web development. Even the ones that were related to web programming seemed to focus on the relatively unimportant surface problems (like XML).

One possible reason is that these guys might not have much web development experience. People who have developed complicated, real-world web applications might be more aware of all the wrinkles that need to be overcome.

Too bad that Christian Queinnec wasn't there. I think he surprised us all at the Oslo Lisp meeting by excitedly talking at length about all the PHP hacking he was doing. I for one was impressed :-)

An often overlooked element of these discussions is the fact that whereas it is possible to do many things in many languages, it is often not practical.

Much of programming today is business systems consisting mostly of GUI and RDBMS activities. Most of the theoretically and elegantly good languages (Haskell, ML, Lisp, Oz, ...) have the capability of accessing RDBMSs and building GUIs. The issue is that in the business environment they are impractical.

For database access, usually a primitive, partially implemented ODBC interface is offered (as demonstration). Or else access to databases that are interesting, but not used much in the business world (MySQL, PostgreSQL, SQLite ... vs. Oracle, DB2, MS SQL Server...).

For GUIs, similarly primitive offerings based on tcl/tk, WxWindows, GTK... vs. the business world expected native Windows look offered by hundreds of off-the-shelf components.

So let us say that some great new language comes out of this, or a discovery that Oz (or pick your own) is actaully the great new language. To most programmers - the business community - it will not matter. They cannot practically do the 80% of what they need to do. (And remember that in the business world, 90% of desktops are Windows - like it or not)

For GUIs, similarly primitive offerings based on tcl/tk, ...

Work is being done on improving Tk's default look and feel, and the Tile project is working on a new theming engine for Tk, with support for native themes on Win XP and Mac OS X. There is also a theme which ties in to QT somewhere. See the wiki for some shots of actual commercial apps using Tile already. You can do a lot, even without Tile, to improve the appearance of a Tk app, by adjusting the options of widgets, and spending a bit of time on polish (time you've probably already saved by not using a low-level toolkit), e.g. as the Coccinella demonstrates. Tk 8.5 should be a major jump forward towards getting the "native Windows look". A lot of languages have GUI toolkits based on Tcl/Tk (e.g. QTk in Oz) and so will get a boost from this too.

It seems a shame that Tk is so much maligned for it's supposed poor look, as it is a much higher-level toolkit than most of the alternatives. Platform-specific APIs are just about reaching the level of abstraction that Tk has been at for a decade (e.g. Cocoa, XAML), and other cross-platform projects, such as Mozilla XUL, are inventing a very similar wheel.

Paul, it appears to me that it's painfully obvious that you've spent no time having to develop real business applications otherwise you would understand the difference between possible (as you mention, and as I pointedly mention in my post) vs. practical.

And contrary to your claim I have spent extensive time in many of these other languages. For example, with Lispworks ODBC a simple select of 8000 address records takes 10 times longer than in more practical environments - possible, not practical. In Oz, ODBC barely functions. In Erlang it is very good. (I could list many others, but I do not think this is the place.)

I only claim that a language is not complete until it is practical to the user - and the majority of language users develop business applications.

I spend time here to learn, discuss (civily would be nice) and perhaps influence the academic to understand the world of most corporate programmers.

Alex Peake: Paul, it appears to me that it's painfully obvious that you've spent no time having to develop real business applications otherwise you would understand the difference between possible (as you mention, and as I pointedly mention in my post) vs. practical.

I suppose I had that coming, even though you're wrong. But my point was precisely that your claims regarding practicality are off. The fact that you can name an instance of impracticality doesn't serve as proof that none of the environments that we discuss here are servicable in an enterprise software development context.

Alex: And contrary to your claim I have spent extensive time in many of these other languages. For example, with Lispworks ODBC a simple select of 8000 address records takes 10 times longer than in more practical environments - possible, not practical.

It's good to have a concrete example. Do you have any data on where that difference arose? My guess would be that it arose in the marshalling of the C-level ODBC data types to whatever their moral equivalents were on the Lisp side. My follow-up question would be whether that was inherent to Lispworks' ODBC support (i.e. that code was closed-source and you couldn't futz with the marshalling) or whether there was more that could be done to address the issue.

This goes directly to the point that many of us who appreciate some of these alternative environments are making, which is: there's value in being able to quickly develop a correct system that might not be as performant or GUI-sexy or whatever as it ultimately needs to be in order to succeed in its market, whether that market is internal or shrink-wrapped or what have you. Then, when you have your working code, you revise it until it performs and/or is sexy. With respect to this example, I have a tough time believing that the Lispworks product is simply broken, and none of their customers have ever used the ODBC support and noticed the fact. It strikes me as quite a bit more likely that the folks who would buy Lispworks and use the ODBC support would look at those results and think, hmmm, we'd better figure out why we're seeing that 8x difference and what we can do about it.

Alex: In Oz, ODBC barely functions.

That appears to be true today.

Alex: In Erlang it is very good. (I could list many others, but I do not think this is the place.)

On the contrary. This is the place that you complained that you can't "practically" write software using the tools we tend to talk about here, so this is exactly the place that we should explore why that may or may not hold for specific examples. I chose to mention CLSQL precisely because it's not just a low-level wrapper around the access libraries for the database it supports, but offers a higher-level functional API as well as CLOS metaobject support for persistence. It's very practical: it lets you save your CLOS objects (which you're going to want to use to model your domain) in a plain ol' relational database (which is what people have in the real world).

Alex: I only claim that a language is not complete until it is practical to the user - and the majority of language users develop business applications.

This is simply unfair on at least a couple of levels: a language that's useful to game developers or streaming media developers might not be useful to business developers, but it's still useful to its users. More importantly, though, "practical" lies in the eye of the beholder. It was "practical" for Paul Graham to develop ViaWeb in Lisp, and that was years ago. How much more practical would it be now that we have several continuation-based web application frameworks, in Lisp, Scheme, Smalltalk...? ITA found it "practical" to develop the software that finds optimal flight schedules that powers Orbitz in Lisp. Many hospitals and telecom organizations find M "practical" for their business applications. And so on.

It just continues to seem to me that you outrageously self-select to make your case: because Word, Excel, Powerpoint, etc. for Windows use COM and ActiveX, only languages suited to the COM and ActiveX model are practical. QED.

Alex: I spend time here to learn, discuss (civily would be nice) and perhaps influence the academic to understand the world of most corporate programmers.

Alex, I think I've seen you post maybe six times total, and twice now it's been the blanket generalization that, basically, systems that I and many others have developed and continue to develop don't exist ("practical business applications written in something other than Microsoft technologies"). Both times I've had to press you for specifics; both times you've cherry-picked examples and ignored counterexamples. This time, the Lispworks example makes me suspect that you didn't investigate much further than to run a query, not see C-like performance, and throw your hands up. If that's how you measure "practicality," then yes, I suggest that you're going to continue to be disappointed, and yes, I find it hard to respond to such an approach with civility, so I'll stop here.

Paul, you clearly do not understand the world of corporate business application development. With your attitude you would never make any money. We get paid for delivering business apps that meet requirements, in a timely manner. We cannot spend the time to go diving in to someone else's low level implementation of some ODBC driver. It is a productivity trade off.

If you are a language implementor, then please learn to listen to users. Yelling at them will not help you, nor them.

Take a typical scenario, and you tell me how you would do this. Get selection criteria for some (product) order from the user. Select the appropriate data from an RDBMS and return in timely fashion. Display the results in grid. Allow users to sort and filter the data in the grid on any suitable criteria. Select and edit a few orders. Save the data back in a transactionally safe way without locking other users. Do this in a modern multi-tiered architechture, supporting hundreds of users. And oh, by the way, the web app needs to use the same functionality that the desktop uses. (This is our world.)

Now show me the user acceptable look and performance you come up with and tell me how long that took you. (It takes me less than 4 hours.)

Funny, I actually did quite well with consulting, thanks.

Alex: We get paid for delivering business apps that meet requirements, in a timely manner. We cannot spend the time to go diving in to someone else's low level implementation of some ODBC driver. It is a productivity trade off.

I'm not suggesting that you have to, and this is where you continue to misunderstand: I often use technologies that let me get something very finely tuned to the requirements working vastly faster than you can customize an assembly of off-the-shelf components to meet the requirements, and do so sufficiently quickly that I have plenty of time left over for profiling and taking care of the performance and spit-polish. The overwhelming majority of the time, I already know what to look for from prior experience. To take the Lispworks ODBC example again, since it's using the same ODBC driver and network as it would be if it were written in C, that suggests data type marshalling overhead. OK, I've been writing Lisp for >20 years; I'm pretty familiar with how to tackle that issue. Assuming that I have source code access above the FFI to ODBC, I can solve that problem, probably within an hour.

So much of what works in a context is about familiarity. That's why these arguments are tiresome: you can always argue from familiarity. That's not helpful or enlightening; it does nothing to help us "academics" (I haven't set foot inside an academic institution as a student in 20 years, and never as a teacher) address the "real world." It just exposes your prejudices.

Alex: ake a typical scenario, and you tell me how you would do this. Get selection criteria for some (product) order from the user. Select the appropriate data from an RDBMS and return in timely fashion. Display the results in grid. Allow users to sort and filter the data in the grid on any suitable criteria. Select and edit a few orders. Save the data back in a transactionally safe way without locking other users. Do this in a modern multi-tiered architechture, supporting hundreds of users. And oh, by the way, the web app needs to use the same functionality that the desktop uses. (This is our world.)

Once again, your assumption that I'm not familiar with the domain is simply dead wrong.

Assuming Windows, I'd probably go for either Corman Common Lisp and Chris Double's Mixwin and IDL for the UI work, CLSQL for the database integration, and UnCommon Web for the web app; or else O'Caml and Osiris, OCaml-Win32, OCam'OLE, and/or CamlIDL for the UI work and probably OcamlODBC for the DB integration, and XCaml on the web.

Alex: Now show me the user acceptable look and performance you come up with and tell me how long that took you. (It takes me less than 4 hours.)

Assuming familiarity with the components you want to use, e.g. for the grid, I don't see a problem here at all. In fact, if you're using asp.net for your web app, I'd be willing to bet that my web app will correctly handle, e.g. the user spawning new windows in the middle of a chain of forms and changing values in them, using the back button and editing, etc. and yours won't.

Again, this teaches us nothing about fitness to purpose (you can develop a continuation-based web app framework for ASP, I'm quite sure, and once someone does that will stop being a competitive advantage for Lisp/Smalltalk frameworks) and everything about familiarity. It also just shows that saying "do this" without being very clear about the requirements (is it OK if your web app breaks when the user uses the back button?) is a good way to do demoware, but no way to actually work in the marketplace.

At this point, can we just stipulate that we can both build enterprise apps successfully because we've both done it? The only thing that I'm arguing is that you've chosen to say "none of these tools are practical," which falls into the category of "extraordinary claims that require extraordinary evidence." Unfortunately for you, it's also the negation of an existential quantifier, so I only need a single example in order to refute you. The quality of this conversation would be totally different even if you were just to say something like "Hey, it looks like there's a lot of variation in the maturity levels of these tools for GUI/DB work; given that I'm a Windows developer in a corporate setting, where should I focus and what caveats would you offer? By the way, I tried Lispworks and its ODBC support and got an 8X performance hit relative to __________. Is there a <1 hour solution to that, or is their implementation just horribly broken?" My reaction to something like that would have been something like "Well, Oz is an amazing language, but it's pretty new, and it doesn't seem that enterprise integration has been a focus for the user community yet. Oz is also bytecode-compiled without even a JIT, so you won't be replacing C# with it. O'Caml has a pretty pragmatic focus, including some nice COM/ActiveX/ODBC integration, a great time-travel debugger, and a profiler for its native-code compiler. But Lisp is the most mature of all of the alternative languages, and if you need GUI toolkits, database access including O/R mapping, CORBA for interop with other systems, etc. it's the one to look at. Weird stuff about Lispworks ODBC; I don't own Lispworks, but let me see if their Personal Edition or maybe a 30-day eval version will let me try that out and see if I can help. In the meantime, maybe one of the good open-source Lisps like SBCL, coupled with CLSQL, would be a better choice."

Anyway, my point is that we're just talking past each other, and it doesn't have to be that way. But obviously I've contributed to the problem, and I have to apologize for that.

Assuming that I have source code access above the FFI to ODBC, I can solve that problem, probably within an hour.

Interesting. I presented the same problem to Franz on their ODBC (AODBC). They had John Foderero on the job - pretty experienced! Anyway, in days rather than hours work (I met with Franz) they had reduced 16 seconds to 8 seconds. A further suggestion that it was not usable and more days and JF got it to 2.5 seconds. The underlying performance possibilities (as I demonstrated in Erlang and others) are about 0.5 seconds. So perhaps you are more magic than John Federero, or perhaps it is a more difficult problem than you think?

... for the UI work...

I will take a look. Previous explorations have demonstrated fairly primitive grids and/or lots of "you can easily implement that yourself". Take a look at the Infragistics Grid, or the ComponentOne Grid.

I'd be willing to bet that my web app will correctly handle, e.g. the user spawning new windows in the middle of a chain of forms and changing values in them, using the back button and editing, etc. and yours won't.

That is a bet you would lose in short order - I have been doing this for many years. And as for Continuation Based Web Servers, there is an all too often missed aspect (you should ask business app developers). There are often database transactions in the middle of all these pages, so there is no going back. The order has gone to the production floor. Too late.

Of course it depends upon the nature of the dataset, whether it really is marshalling that's thrashing or whether it's GC or some combination of the above, whether there's an opportunity to manage storage manually, take advantage of simple arrays of unboxed values, etc. etc. etc. My point, and it's really a small one, is just that once you have a certain level of experience in infrastructure (language, libraries, culture) X, you can probably be "productive enough" in it such that just walking away when your query takes 8x the time you expect is no longer necessarily the most economical response. Implicit in this is the observation that sometimes it is the most economical response. Experience is what makes it possible to make that judgment call effectively, and perhaps I'm a little touchy because, like many Lispers, I've seen too many people just walk away and carry their misconceptions about the possibilities with them.

I certainly don't think I'm magic, although I did support MCL while at Apple and did tech review "Paradigms of Artificial Intelligence Programming: Case Studies in Common Lisp," Peter Norvig's first text. It thrills me no end to see work like "Practical Common Lisp" and "Successful Lisp" appear, and for Graham's "On Lisp" to be available online.

Alex: I will take a look. Previous explorations have demonstrated fairly primitive grids and/or lots of "you can easily implement that yourself". Take a look at the Infragistics Grid, or the ComponentOne Grid.

Right. Given your challenge, I took as a given that I'd need COM access, and the more support for IDL, IUnknown, and ActiveX, the better. I'm really trying not to be one of those language wonks who insists that integration with C/C++ and a good GUI framework don't matter. On the contrary, one of my I-wish-I-had-more-time-for-it projects is to extend the Forklift FFI for O'Caml to cover Objective-C, work with another young programmer who's begun an O'Caml <-> Objective-C bridge, and work towards 100% coverage of Apple's Cocoa APIs for O'Caml. I wholeheartedly agree with the point that in a business context, a non-native interface is a losing proposition. Actually, I find that true in all but certain heavily specialized domains: Maya and Blender can get away with having their own UIs...

Alex: That is a bet you would lose in short order - I have been doing this for many years. And as for Continuation Based Web Servers, there is an all too often missed aspect (you should ask business app developers). There are often database transactions in the middle of all these pages, so there is no going back. The order has gone to the production floor. Too late.

Once again, you presume that I didn't spend about eight years writing business web apps. You personally may have made sure that page-spawning and the back button work, but I've seen plenty enough ASP apps to feel quite confident in saying that the framework isn't helping make that happen. The point of a continuation-based framework is precisely that you can say:

(begin-transaction *connection*
  (do-this)
  (do-that)
  (do-the-other)
  (commit))

Where do-this, do-that, etc. result in navigation among different pages that collectively contribute to the transaction. That is, it simplifies scope management issues in developing stateful (database-based or otherwise) web apps. Again, I didn't mean to imply that it's impossible to do that with, e.g. asp.net out of the box, only that the folks who plow through the wizards and bang out their FrontPage pages and say "See? A whole web app in a day!' (Orkut, I'm looking at you) tend to overlook such fine points, and it's important to talk about them if the question is "how do your tool choices make you more productive in developing this type of app?" It's easy to claim high productivity if you (or your toolset) cut corners. That's all.

Paul, don't feed the troll.

The line between "possible" and "practical", which is the distinction Alex made, depends on the observer. While most of us here might not have a problem using these tools, it's often very much a kind of early-adopter experience. The problem that many of these tools face is simply that they haven't been used in as many situations as equivalent mainstream tools, and as a result any individual user is much more likely to run into the kind of corner case which would normally send them running to the vendor's support line. This is definitely a barrier to adoption.

But the interesting phenomenon is another one. Some languages manage to become popular in spite of not having a good implementation (e.g., a compiler producing fast code). Why? What makes them succeed where other, possibly better languages fail (yes, "better" is subjective, I know).

Two examples (not implying the languages mentioned are 100% bad, only that they didn't have great implementations when they started to become popular):

Java - Even today I hear many complaints about Java's speed (even with new VMs). Yet the same people who explained that GC andVMs are bad due to the speed penalty when discussing Scheme, seem to live with it just fine (or to put a fact on it) when it comes to Java.

Python - The success wasn't because the implementation of CPython is great or very fast. APIs were important, but I don't think SQL via Python is so much faster than via Scheme. (Never measured, just guessing).

To sum up, let me just mention that "practical" is an umbrella term. It means having many documented APIs, cookbooks, example code, efficient implementations, tools etc. It might be a good idea to try and see which of these makes the biggest difference.

Speed is mostly about "fast enough". Obviously a user waiting 20 ms vs. 200 ms is not noticeable. A user waiting 500 ms vs. 5 s is quite noticeable and crosses the "not fast enough" barrier.

Java, Python, Microsoft Stuff, ... have convenient access to components (commercial and other) that save one from developing significant functionality oneself (a productivity win).

Books on paper or the web have a big influence. If you are having trouble following something, access to two or three different approaches is highly beneficial. Java, Python, MS, ... have lots of books (on lots of domains - and for me business applications).

Good IDEs are indeed a great productivity aid.

Does it not come down to practical means productive (in one's domain)?

Alex Peake: Does it not come down to practical means productive (in one's domain)?

Yes, this is an excellent launching-off point for a more productive discussion than we've had so far, precisely because productivity involves so many factors apart from whatever abstract notions of "quality" we may attach to any given tool, even given a relatively fixed domain. The interesting exercise would be to actually have a [Lisp|O'Caml|Haskell|Oz] expert and a [C|C++|Java|VB|C#] expert develop your business app from the other thread, on the same platform, at the same time. My claim is that we'll end up measuring each developer's familiarity with the toolset rather than anything meaningful about the toolset, assuming a well-understood domain. If the goal is to learn about the relative costs/benefits of the toolsets, I think we'll have to find a different exercise. But if the point is to measure relative productivity, I'm convinced that the exercise will end in a draw—which is as it should be.

I am almost in agreement with you on this. Experience with the toolset can often dominate the productivity, as can being a great programmer (>5:1 differences between median and great on productivity).

We run into trouble when what we need is some pre-existing commercial component (see Commercial Components).

So if I need to give a user a professional looking, integrated with the app, reporting tool - what now (without these components)? Or graphing tools or Outlook controls or modern Grids (like Infragistics or ComponentOne)...(such as our users demand)?

And then take a look at the sheer depth of the System.Data namespace in .NET - try matching the DataSet and friends.

I guess we come back to "what problem are you trying to solve". If these are non-issues in your world, then you are absolutely correct. In my world, you are missing major productivity gains.

Now if we could get Lisp running well on .NET, or Oz. I am currently working on F# and SML.NET which give me (roughly) Caml and SML. Hard to beat Lisp or Scheme though.

I tend to agree with you that in the real world integration matters a lot (including integration with so-called components). My interest to PLT is motivated (partially) by a quest to understand how and where the current component architectures are falling short (and they are, big time), and what can be done about that. This quest lead me to learn (or start learning) lambda and pi calculi, type theory, basics of category theory, formal semantics, and much more. While some of the current implementations are not of industrial strength today, the ideas behind them will help to make the industrial-strength implementations of tommorow (and the X-sharps of the world will be not industrial-strength by that time). That's just a personal opinion, of course.

Now if we could get Lisp running well on .NET, or Oz. I am currently working on F# and SML.NET which give me (roughly) Caml and SML.

I would say it's a barrier to being good software!

It really takes a tremendous amount of work and experience and re-work before programs get really good. Languages designers must hate this because it makes it very hard to put a new language into the hands of working programmers, but it's a fact of life and doing this work is really what programming is all about!

After all, there are no programming languages, only programs!

This is all true, but once you've created an enterprise level development environment for Haskell, there's nothing stopping you from including an easter egg that STABS STUPID PROGRAMMERS IN THE FACE. That way everyone wins.

I believe there is a way - macros (real lisp macros). A key to significant business app development productivity is automation. Too much code written in business apps is code of similar patterns.

So the tool of the "top programmer(s)" is a tool to write code that writes code (macros). Then the average programmer glues the pieces together. I know this works because I have proven the concept, albeit in a rather crude way, using Lisp to generate C#. 400 lines of a Lisp DSL generate 25,000 lines of working C# app.

So if we focus on a different aspect of language development, where the .NET or Java VMs are merely the intermediate language, we can have a good high level language, and full intereoperability with components. It is also easier to introduce new languages and concepts to the top programmer in an organization, than to the programming community at large.