Lambda the Ultimate

That link goes back to LtU, but I think I know the page you mean. I am looking for concrete example implementations of a set of common specifications, so a big ol' table of features is not that helpful.

Here are a couple of presentations that compare
features among Java web frameworks and design of some MVC web frameworks. A couple of years ago I was trying to compare various frameworks by implementing a blog engine but that didn't go far (too many frameworks and too little time).

Thanks, though I found similar presentations quickly enough in my initial Google search. I have a hard requirement of concrete, runnable code, with the same specification implemented in multiple languages/frameworks.

I don't know of anything beyond the somewhat frivolous Arc challenge (essentially, whether there are continuations or not).

For typical web apps, as ground zero, I'd want a framework that supports:

* Scaling of data actions (DB-style benchmarks) -- static code doesn't matter
* Dynamic scaling (eg., good use of EC2)
* Discretionary & mandatory access control
* Controlled XSS
* Disconnected client functionality
* Logging/tracing/analytics
* UI reactivity
* RESTful interfaces & evolving service generation
* Persistent data analysis
* Query integration
* Search engine optimization
* An application component framework
* A visual component framework

There are many more domain-specific tasks (eg., charting support) as well -- a niche seems useful in this general space.

I was working on formulating a qualitative benchmark suite last year of prototypical future applications to drive some of our research, if you'd be interested in it.

In the web domain, I think the biggest challenge is hemorrhaging out the code to beginwith, and then updating and scaling it. Libraries etc. are critical to get functionality and security (much of which is at the specification layer). This is likely somewhat true of most non-embedded, NASA, and banking systems.

note: I'm biased based on my industry and research work in this space

back to work...

edit: interoperability with other systems (gui frameworks, backend data stuff, etc)

If you have implementations of important specifications in several of the most visible frameworks of today, then I'd definitely appreciate access to the code.

Nah, that was part of the point -- it's fairly obvious many of these things are not really supported. Current frameworks are largely oriented towards serverside CMSs and CDNs, pure GUI scripting (eg., ajax frameworks), or pure app clients with services (eg., Flex).

Microbenchmarks permeate this field -- like the Arc example -- while full benchmark apps are lacking. I'd say this is true of many modern proposed PLs. For example, doing n-body makes sense for HPC apps, but doing an adder is too contrived for a web app. Something like what the Flex guys demonstrated with http://coenraets.org/blog/2007/02/building-the-back-end-of-the-tour-of-california-%E2%80%9Ctour-tracker%E2%80%9D-using-flex-data-services/ is more interesting.

I'd be pretty surprised if you find anything useful. It's a complex set of criteria and many of them are fairly subjective. It's also a big domain, so many "competing" web frameworks really have very different goals.

There's a reason that the PL shootout focuses on performance, rather than "overall solution goodness" or something.

Maybe the best you can do is look for a somewhat standard "hello world" app, like the Java Pet Store, that people may have implemented in lots of languages/frameworks (for instance, I know that there is a .NET Pet Store, there are probably others). But this is a pretty poor answer. :(

Maybe the best you can do is look for a somewhat standard "hello world" app, like the Java Pet Store, that people may have implemented in lots of languages/frameworks (for instance, I know that there is a .NET Pet Store, there are probably others). But this is a pretty poor answer. :(

I was going to suggest a blog or a wiki, since most frameworks nowadays seem to have "create a blog in 20 minutes" webcasts on their sites.

There's a reason that the PL shootout focuses on performance, rather than "overall solution goodness" or something.

I can think of several:

• rhetorical AAA performs better than BBB squabbles have been commonplace - it obviously is a matter of interest and an opportunity for education
• rather than complain about the rhetorical AAA performs better than BBB squabbles, we can be curious and ask does this AAA program perform better than that BBB program?
• rather than complain about the skilled AAA programmers who compare the performance of their well implemented AAA programs against their ineptly implemented BBB programs, we can ask both the AAA programmers and BBB programmers to do their best given the situation
• before we show whether this AAA program performs better than that BBB program, we show that this AAA program and that BBB program actually build and produce the expected result - I suspect that sidesteps a multitude of bogus claims

What did you have in mind?

That seems more like a sequence of reasoning than several distinct reasons, but anyway it's related to what I had in mind: performance is quantifiable. It's (more or less) possible to get past the rhetorical squabbles in that arena. I'm not sure the same thing is true with web frameworks.

An existence proof is one step. Hence our proposal of prototypical web applications: if you're framework can't express and run them, it's not complete. That is coarse and hard to argue against, but it's something. As a language and framework designer, it seems distracting, and microbenchmarks are easier (I include no-frills-attached blogs here). Either you get a user to do it or you do it yourself, but I'm not very convinced without it. I don't think we've gotten very far from System R etc. in terms of demonstrating these things.

Again, I'm all for counter proposals! They would certainly make my life easier.

what I had in mind: performance is quantifiable

Many things can be made quantifiable.

I think the difficulty is getting to grips with all the things "how each tool enables component reuse" might mean, and which of them we should care about.

When it comes to program "performance" we already have some clichés to work with.

To me, this is quantifiable by asking to implement several similar applications, and then measuring complexity as a function of the total amount of code needed.

If the question is "seeing how each tool enables component reuse" then I'd naïvely guess we need to - say what we think a component is for each tool, identify components in the source, identify reuse of those components in the source.

Is that the question or is it a somewhat abstract proxy for some notion of maintainability, some notion of cheaper/sooner? In which case, showing better component use would be interesting as a possible explanation of cheaper/sooner but we'd still need to show cheaper/sooner.

I'm hoping you did not interpret me as suggesting that the way to compare languages through implementations of the same program is through line count. As I stated earlier, it only shows that, likely with somewhat comparable effort, either can initially get the job done: it's an existence proof. Software engineering is a process, so I'd want to judge the language in terms of how building a web program with it fits into that process for web development. The loc metric does not seem to apply to that process: we'd all be writing web apps in perl6. I think there are continuations, it's easy to prevent sql injection through a library, they have a great vm, etc.

If you do want to use lines of code as a metric, perhaps an approach closer to it would be to measure how many places code most change to add, remove, or change a set of common features. I'm still shying away from actual lines of code -- I'd hope future systems would do write most of it for me (like through Dave Mandelin's Prospector tool). When writing academic code, I have to roll a lot of my own, but, when doing normal web code, reuse is more of the norm, modulo the UI layer.

I haven't read any papers about how to go about a quantitative analysis of languages -- still hoping somebody will bite on this point :)

I wasn't interpreting anyone's suggestion. I believe that a metric like size of gzipped code is very useful, because it's objective and has a clear connection to maintainability of code. The connection can be tenuous and misleading, but I know of no better alternative that enables citing statistics in research papers. I don't have the resources to run timed user studies right now.

Much worse, even. If you think that a metric has a "tenuous and misleading" connection to what you want to measure, why would you include it? It's better not to have statistics at all.

I don't have the resources to run timed user studies right now.

That's completely understandable but leaves you stuck in the same place as everyone else who claims we can do stuff cheaper/sooner with tool X without actually showing that we can do stuff cheaper/sooner with tool X.

We've heard that so many times that it would be greeted with skepticism even if there was a direct cheaper/sooner demonstration.

Thanks. I had seen ocsigen before, but not Kaya. All the same, I think I need to be even more explicit about what I'm looking for in starting this thread. :-)

All I'm looking for is concrete problems in web application development implemented in several languages/frameworks, with the code available. The end goal for me is a table for quantitative comparison of the alternatives. Suggestions of which features matter could be useful if I have to decide that no such benchmark project exists already, but they are "off-topic" w.r.t. my current goal.

Even though the introductory remarks have no relevance to the discussion of web frameworks, seeing something that seems wrong in those incidental remarks still lessens confidence in the rest of the discussion:

Seaside
Alan Kay famously stated, "I invented the term Object Oriented, and I can tell you I did not have C++ in mind." What, then, did he have in mind? Smalltalk, which he developed over various iterations...

Apparently not: "Actually I made up the term object-oriented and I can tell you I did not have C++ in mind. The important thing here is that I have many of the same feelings about Smalltalk ..."

Alan Kay: The Computer Revolution hasn't happened yet. Keynote OOPSLA 1997 Google Video

Thanks for the link! That is exactly the rough kind of comparison I was looking for. Now if only it had used a realistic example instead of using so many obviously clunky frameworks....