You can see slides from the Links meeting here and commentary and pictures here. (Thanks to Ethan Aubin for already starting a thread under the former, and to Ehud Lamm for inviting me to guest blog.)

Ethan Aubin writes:

So why do we need a new language? What cannot be accomplished with existing frameworks? There is a slide following this asking why can't you do this in Haskell or ML, but I don't know why they (or even java/php/etc) aren't enough.
Let me try to answer this. Links is aimed at doing certain specific things.
• (a) Generate optimized SQL and XQuery directly from the source code -- you don't need to learn another language, or work out how to partition your program across the three tiers. This idea is stolen from Kleisli. You need to build it into the compiler, so Haskell, Ocaml, or PLT Scheme won't do.
• (b) Generate Javascript to run in the browser directly from the source code -- you don't need to learn another language, or work out how to partition your program across the three tiers. We're hoping to provide a better way to write AJAX style programs. Again, you need compiler support -- vanilla Haskell, Ocaml, or PLT Scheme can't do this.
• (c) Generate scalable web interfaces -- where scalable means all state is kept in the client (in hidden fields), not in the server. PLT Scheme and WASH provide the right interface, but they are not scalable in this sense, precisely because making them scalable involves fiddling with the compiler. (Felleisen and company have pointed out the right way to do this, applying the well-known CPS and defunctionalization transformations.)
So that's my argument for a new language.

Is it a good enough argument? Is this enough of an advantage to get folk to move from PHP, Perl, Python? Not clear. I suspect if it is good enough, a major motivating factor is not going to be anything deep, but simply the fact that being able to write everything down in one language instead of three or four will make people's brains hurt less.

Ethan Aubin also writes:

Wadler goes into the FP success stories, Kleisli, Xduce, PLT Scheme (Continuations on the Web), Erlang. If you take the befenits of these individually, you've got a language which solves the 3-tier problem better than what we have now, but I don't think it meet the criteria of "permitting its users to do something that cannot be done in any other way". So, I'd like to ask the all the perl/php/asp/pythonistas on LtU, what it is the killer-app that that your language cannot handle?
I'd love to see answers to this question!

## Comment viewing options

### Language vs. Compiler?

You need to build it into the compiler, so Haskell, Ocaml, or PLT Scheme won't do.

It's a proof that specific implementations (compilers) are not good. This does not mean trivially that these languages are not good (for this "extraction" of SQL/JS out of the Haskell, Ocaml, Scheme code).

I believe there are some arguments for this conclusion, I just don't see them on the surface (of the presentation). Are type systems of these languages bad for this extraction to be done efficiently? Is their semantics bad for that? Their abstract syntax? Concrete syntax?

User base? :-)

It would be very interesting to learn your opinion on this!

### Compiler support?

You need to build it into the compiler, so Haskell, Ocaml, or PLT Scheme won't do.
It's a proof that specific implementations (compilers) are not good. This does not mean trivially that these languages are not good (for this "extraction" of SQL/JS out of the Haskell, Ocaml, Scheme code).

Also, a nit: the emphasis on needing compiler support for these things seems to be predicated on not having macros. In fact, Scheme and Lisp can do at least (a) and (b) above in an integrated fashion. SchemeQL is a good example of this.

Although I'm fairly sure it wasn't his intent, Philip is in fact making a great argument for the necessity of macros. :) This is particularly true if there's any interest in delegating significant tool-creation capability beyond the designers of a language.

Of course, the argument can be made that macros make Todd mad, and that most programmers want more syntactic sugar than a regular syntax offers, which means that compiler support is in fact wanted to satisfy this perception of market requirements.

[P.S. I'm not arguing against the potential value of a language like Links.]

### Language Problems?

While I was initially enthusiastic about the Links project, I'm starting to have my doubts.

In particular, I don't think that some of the things you are trying to solve are really language issues.

Generate optimized SQL and XQuery directly from the source code
Generate Javascript to run in the browser directly from the source code

The big problems with both of these things is heterogeneity of environments. Depending on what specific RDBMS or browser you are running, "optimized" means something slightly different.

In particular, Javascript tends to run slightly differently on every browser/platform combo. I think the hype about AJAX has glossed that over. I suspect that Google just has enough bodies to support Javascript libraries for all the combos they want to support.

I'm not sure that making Links into a big multi-platform code-generator will really solve the complexities here.

where scalable means all state is kept in the client

One word here: security. If the client has the definitive view of the universe, the server is vulnerable to manipulation by the client. There will always have to be SOME state at the server side.

Maybe you can off-load a lot of "state" to the client, but you still may open up performance issues, privacy issues, client-side security issues, etc.

Again, I'm not sure that these are language issues per se, but architecture issues inherent in web development.

### SQL Generation

HaskellDB doesn't yet, but could and probably will one day, generate optimized SQL for different environments. It abstracts away from SQL by acting as a compiler for an embedded relational query language. I imagine that something similar could be done with XQuery and Javascript - make the different platforms different targets of a retargetable compiler.

### generate optimized SQL for different environments

From my experience, you will always need a way to get at the raw SQL generated and tweak that. Databases are strange beasts, and optimizers change query path constantly. What used to be an optimized query suddenly runs slowly. Even as table sizes change, the optimizer changes the query path. I have found just re-arranging the order of statement elements radically (order of magnitude) changes performance (and of course it should not - but optimizers are written by mere mortals).

### To Say Nothing...

...of how a Links is supposed to know when it would be appropriate to build some indices, and on what. I suppose that, in the limit, it could periodically do an "EXPLAIN" and attempt to do some inference. That would be very interesting to see.

### Wrong level problem

it could periodically do an "EXPLAIN" and attempt to do some inference

I think this underlines my qualms with the idea: in order to make the SPECIFICATION (language design) work, we are already pretty deep into IMPLEMENTATION issues.

This suggests the problems don't live at the language design level.

### Yes and No...

...No, because a language that has the explicit goal of interoperating productively with existing "relational" databases has no choice but to finesse the interaction with SQL somehow.

Yes, because it's not clear that this "finessing" can't consist of having the usual kinds of support for extensible control-flow coupled with a good syntactic extension mechanism. Ehud recently reminded us of SchemeQL, which seems like an excellent example of the latter. I'm also reminded of Oleg's work on database interfacing in Scheme.

### Permeable abstractions

From my experience, you will always need a way to get at the raw SQL generated and tweak that

It's sometimes necessary to get at the output of a C compiler and tweak that, too. Sometimes, but rarely. I think a relational query language compiler targeting SQL could get it right enough of the time to be useful. But I agree that it's important to be able to access the lower-level representation from time to time.

### but rarely

Obviously my exepriences are not "statistically significant". I have, however, been using Oracle, SQLServer and others for over 15 years. Rarely is not the term from my experiences. Regularly might be the term I would use.

Why, for example, do databases offer "hints" as a SQL facility? Because they know the issue is significant.

### Why databases offer "hints"

Because they're committed to supporting a crude language in which trivia such as order of operands are significant, and are used to determine performance. A better approach would be to take the existing core engine and mate it to a more sophisticated language, one whose interpreter would do performance analysis on the fly to respond to the changing conditions in the core.

I've written a compiler targeting SQL, and getting the generated code to perform was indeed the hard part. I had serious DB experts advising me on how to improve the output, but the problem was generalizing that advice. The improvement added to make customer A's code perform later turned out to hurt customer B; and the information to distinguish the two cases just wasn't available at compile time. I had a design laid out to let runtime performance data feed back to compile time, but never got a chance to build it before the company imploded.

### Costs/benefits model?

Why databases offer "hints"
IMO, because neither databases nor computer systems in general are aware of real-life costs. They might optimize against costs model that is obvious to them - better response time, better throughput, better memory usage. Unless told by humans, they do not know, whether the enterprise gets more bucks out of insert or select operations. This knowlegde is not obtainable by any degree of introspection.

Thus it has to be specified externally.

One problem with current state of affairs is that this information must be provided in wrong form by wrong people at wrong time (SQL hints by developers at time when a statement is prepared). One cannot easily decouple this costs/benefits information from SQL statements in both space and time.

Probably, Links will provide a separate life cycle for costs/benefits specifications.

### Database Optimizers that Learn

Perhaps what we need are database query optimizers that learn?

Optimizers often produce query paths that are nonesensical, taking one or more orders of magnitude to do the query than is necessary. The current solution is either hints or humans re-arrange the queries.

If we had query optimizers that learned, we could either:
* Tell the optimizer to try again (offline - while tuning)
* Have the optimizer use idle time to experiment with the last periods queries to find better paths, and remember

### And are not dogmatic

I've been having problems on the stored proc side where the compiler will figure out the plan based on the current inputs. Once compiled, the proc doesn't need to be recompiled until the conditions change or a recompile is needed.

Problem is, that the parameters used in the first call may or may not be representative of the data as a whole. Means that you can get a very small dataset which might perform a microsecond better doing one path, but that same path can take minutes to complete for the larger sets.

Haven't really figured a good way around the problem other than dropping hints and trying to force a compile on the worst case datasets.

### Nobody is teaching them

Database Optimizers that Learn
To learn, they must know about utility, and that's something they don't know.

Even if we forget about economical values: DBMS can optimize locally - in the scope of the database, but not in the scope of the whole system, because nobody tells them about outcomes at the level of the whole system.

### optimize locally

Even a "local" optimization is better than none?

A query that should take under a second with a good plan often will take several minutes. This one requires little knowledge of context.

### Partially agree

Even a "local" optimization is better than none?
Yes, if it is a win-win. No, if it is a trade-off (as optimizations quite often are).
A query that should take under a second with a good plan often will take several minutes. This one requires little knowledge of context.
Now imagine that this query is executed by night, when nobody cares about its response time. Additionally, imagine that to ensure this query runs under a second we have to degrade performance of some other query by 10%. If this other query happens to have less "elastic demand", doing this optimization might be a mistake.

Granted, "with a good plan" does not necessarily means harming "utility" for other queries - it may be a win-win. But a classical DB dilemma - to index or not to index - is an example of a trade-off decision, which cannot be meaningfully taken locally.

### State in Client

For security, you could encrypt the state held in the client?

### The only secure way for the c

The only secure way for the client to modify the state then would be to have it pass to the server for modification. Hardly a performance win.

### Proof-Carrying Execution

The client could always modify state locally and pass the a trace of execution back to the server for verification. If the server detects an something is off the client would have to roll-back the changes made.

### Security blanket

If the server detects an something is off the client would have to roll-back the changes made.

What if the change being made is to upgrade the security access of the client?

### This does not require the sta

This does not require the state to be stored in encrypted format on the client; in fact it requires the state to available in clear.

While less egregious, this is still egregious. What benefit does this scheme provide over having the master copy of the state on the server?

### Scalability

Scalability. If the session state lives on the server, then you have to either store it in the DB or tie the client's session to a single Web server. If it lives in the client, then it comes back from the client on each request, and any Web server can handle the request--no application server needed. (Plus, the Back button doesn't break the app, since going Back unwinds the session state.)

### For proprietary frameworks?

why then didn't the javascript server/client combination become popular?
I witnessed a system based on this combination to die before going to production. JS was lacking infrastructure (in those days, not sure about today), so the developers were forced to implement infrastructure in C++ (I mean networking, clustering, metadata, database access, multitasking etc.).

They still had a benefit of the same language for business logic on both client and server, and that was indeed good. OTOH, a boundary between business logic and infrastructure became socially inpenetrable - BTW, this is not different from EJBs, where a developer of a bean cannot go and hack the container. For a proprietary frameworks, though, this property was not economically attractive (the whole point of keeping a proprietary framework is flexibility).

So this framework was replaced by Java-based one.

### Some people work very hard / but still they never get it right

Well, I'm starting to see where this web development stuff is going. It's funny, because I've known about client-side scripting using hidden IFrames since my internship at Microsoft in 1998 (DHTML had just come out in IE4 and we were working on a product that made heavy use of this trick), but I didn't have the foresight to realize that this would eventually become a standard platform. At the time it just seemed like a massive hack to fake a crippled UI inside an application that was never meant for UI's.

Given enough market demand, though, people will find a way turn the most hostile environment into a home. Even as hostile as HTTP and HTML. Applications like this are convincing me. (As are all the amazing things coming out of Google these days.)

Incidentally, I also consider this a big win for high-level languages that Java completely lost to Javascript as a client-side web application development platform.

### v. smart!

Philip, with the rise of Ajax, it will become increasingly obvious that you are on the right track.

### Ajax

But isn't Links about XML, SQL and other server-side stuff, while Ajax is about heavily Javascripted client-side interfaces (plus GUI, so lots of CSS and JS)?

Maybe the killer technology would be a Common Lisp in JavaScript and a declarative layout language that compiles to CSS :)

(yes, I've tried size-independent layout with CSS; it just doesn't work; sometimes 10px might be understood, but 1% isn't and stupid stuff like that)