Lambda the Ultimate

It was very interesting. As you might expect from a varied language there were people from varied backgrounds. It was an "unconference" so the bulk of the schedule was determined at the beginning in part based on comments and questions from the audience. One comment that sticks out in my mind was something like "I just want a simple explanation of co- and contra- variance" - unfortunately nobody gave a type theory 101 presentation so I don't think that person was satisfied. Paul Snively commented to me that he was both heartened and disheartened to hear people asking about such important concepts.

Another attendee asked about "rebranding Scala as a 'blue collar' language" which got no small amount of groaning from the audience plus some sympathetic noises. There was a fair amount of discussion throughout the day about "marketing" Scala and how to introduce it: "Java 3.0, only better" or "Statically Typed Ruby". For some reason, nobody suggested something like "A More Integrated OCaml" :-). In fact, second only to marketing the most popular discussion (complaint) was about syntax and the use of underscore as a wildcard at both the term and type level plus as a way to coerce a method into a first class function.

There was at least one discussion on governance and intellectual property related to the project. The main take aways were that Martin Odersky wanted to follow a more-or-less Python model of having himself as a central benevolent dictator plus something like the "PEP" system to get input from the community. He also said he foresaw having a research fork separate from the more stable, usable fork meant for a wider audience and that the stable fork would have some form of compatibility policies. Finally, he admits to not knowing much about nor being very interested in managing IP issues like trademarks so he needs some help on that front (any LtUers who know about that stuff please contact the Scala mailing list).

At the other extreme, Lucius Gregory Meredith gave a theory driven presentation on automatically generating a query language over a computational calculus like lambda. Some NLP people in the crowd started asking questions which lead down the path of trying to extract probabilistic meaning from natural languages using Bayesian and non-Bayesian techniques and whether there could be any tie in to the presentation.

Along the way, Paul Snively had a discussion on creating an embedded DSL in Scala for general purpose GPU programming. He was specifically interested in building a SAT solver.

There was a small coding scrum where a team built an implementation of JSR 223 which will allow Scala to be used as a scripting language from within another program running on the JVM.

For people looking at how best to use Scala and lift, Martin Odersky showed a small GUI spreadsheet program that's going to be part of the upcoming book. In another presentation, David Pollak took the core engine of Odersky's spreadsheet and wrapped it in the lift framework to create an real-time collaberative online spreadsheet.

As the links above show there were quite a few prominent LtUers in attendance. In addition to those I mentioned Matt Hellige made a brief appearance just like the movie star he is.

For me personally it was good to meet so many faces behind the LtU names and it was also good to see a language drawing both the PLT types and the "I just want to get a web app up with less pain" types.

Thanks! I am sorry I missed this.

I was at another unconference this weekend (since I didn't know about the Scala one, this does not reflect my priorities...) Amusingly enough Scala came up twice during the weekend. The first time was when I was asked over dinner if Scala is indeed the best practical choice out there at the moment (My answer: yes it is). The second was when the facilitator from the Scala camp dropped by, and mentioned she came from there (I was quite surprised that this was the first I heard of the meeting. I guess that was on purpose...)

As James Iry recently put it on the Scala mailing list, Scala has attracted a lot of people who can see a "diamond in the rough." At this stage I think the practicality of Scala is extremely dependent on how much roughness you are willing to deal with. As more and more people use Scala I have no doubt that the roughness will be smoothed out, but as it stands today there's a fair likelihood that a non-trivial project will encounter a compiler crash, standard library bug, and/or incompatibility when upgrading. The standard library also has some glaring holes (although they don't glare that much because you can always call into Java).

I think Scala is probably at least a year away from being ready for production usage at my day job, maybe more, but I use it almost exclusively for personal projects. But obviously this is a situation dependent decision. David Pollak seems like he's had considerable success with Scala for production development.

So, I'm curiuous: What makes a language practical for production use?

[Update: I should qualify that my "day job" is at a very large and conservative corporation where "a year" is about the equivalent of "a day" for those of you in the Web 2.0 world, so please scale readiness and timeframes accordingly. I don't think there's anything wrong with Scala that can't be readily worked around with all the productivity savings you'd get over Java.]

Practical" really isn't the word I should have used. I did not think about these aspects at all, but rather about the location of the langaueg in the "design-space" of programming languages.

In fact, second only to marketing the most popular discussion (complaint) was about syntax and the use of underscore as a wildcard at both the term and type level plus as a way to coerce a method into a first class function.

Heh, I was heard :)

For some reason, nobody suggested something like "A More Integrated OCaml" :-).

How about "Haskell for Dummies?" ;)

Movie star indeed! Actually I really wish I could've been there for the whole day. I totally agree that it's really nice to see a language that draws a crowd with such diverse interests. It was a great group, and this is exactly what I like about Scala people.

But the main excitement for me was getting to meet some LtU and Scala folks face-to-face, a rare opportunity here in Chicago. :)

...particularly of a technical nature. I volunteered to present two sessions, both of which actually happened, which in a way is a shame: I would have liked to attend some of the others!

First and foremost, it was wonderful to meet so many fascinating people. Dave Pollak, lead developer of the Lift framework, can be considered the prime mover of the unconference. Dave is an easygoing, gregarious, funny guy whose demeanor belies a serious approach to real-world issues in building highly-scalable collaborative web applications, which he has done using a range of technologies. I'm firmly convinced that Scala has no better champion, because Dave is no fad-chaser. He will tell you why he makes the choices that he does, what's good about them, and, crucially, what's bad about them. I have a great deal to learn from Dave.

The unconference was facilitated by Kaliya Hamlin. Kaliya very deftly created the container, or context, in which the unconference could self-organize, ensuring that sessions emerged, that there was clarity as to the shape of the space, and that everyone involved was vested in participating fully, whether by volunteering to lead sessions or by making themselves heard in the sessions. Kaliya is so good at what she does that, in classic great facilitator/manager style, she effectively disappeared most of the time, reappearing only when some collective transition was to be made. That's a very poor description of the fluidity that she brought to the process—I'll try to come up with a better one.

Dave and Kaliya, with the help of the sponsors, ensured that we had bagels, fruit, chips, soda, coffee, and lunch. I've seen events with hungry geeks. It ain't pretty. This wasn't one of them.

We were honored to have the keynote address, "A Scalable Language," delivered by Dr. Martin Odersky. I always find it helpful to hear from a language's designer what the language's philosophy is. Martin's keynote explicated Scala's focus on the design and implementation of component-based software via constructs that scale both up and down. I continue to be pleasantly surprised at how many things that seem like syntax in Scala code are actually introduced by libraries—and then by how much conceptual power can be brought to bear upon the development of those libraries. Scala is beginning to take some criticism in some quarters for being a "kitchen sink" language; Martin's keynote reminds us of how many of the things we see in Scala really are not intrinsic, and are in fact built from a relatively small number of nicely orthogonal fundamental constructs. Finally, Martin himself, like Dave Pollak, is an extremely modest, charming, low-key person. People like Dave and Martin serve as an extremely effective reminder that the truly great don't need to waste energy proclaiming their greatness.

The unconference's Platinum Sponsor, ZeroTurnaround, made an exciting announcement: all liftoff attendees received a free license for JavaRebel, their technology for hot-deployment of Java class files into a JVM, making Java (and Scala) development much more closely approximate the fluidity of working with the immediate-turnaround environments such as Ruby on Rails or Pylons. In addition, ZeroTurnaround announced that a free license will be extended to the lift project for inclusion in the lift distribution, for a minimum of one year—the point being made that it's easier to convince management to review a free license deal every year than to convince them to make one free in perpetuity. :-)

Our own James Iry delivered a well-received presentation on monads and why you should care. Even a one-hour presentation from someone as engaging and skilled at making such abstract concepts approachable as James isn't completely sufficient. James did a masterful job at opening the door, the follow-up activity being to read any of the better monad tutorials available online, such as James' own Monads Are Elephants. After the presentation, there was a lively discussion about just how far down this rabbit hole goes. Greg Meredith (Lucius Gregory Meredith, apparently, who I regrettably failed to recognize and who looks even more like a rock star than Matt Hellige does) enlivened the discussion by introducing concepts from category theory that related to his own presentation on "Programs as data elements," including intriguing thoughts about automatically generating queries against programs.

I then offered two discussions, one revolving around the idea of leveraging Scala's approach to what can be made to look like syntactic extension ("target typing") to embed a stream-processing language more-or-less like the BrookGPU dialect, for GPGPU programming by way of translation to GLSL by an underlying combinator library. There was some discussion as to motivation, in which I acknowledged that my personal itch is implementing a SAT solver on the GPU in order to efficiently implement the Event Calculus as described in Commonsense Reasoning. Lucius Gregory Meredith provided some valuable push-back about the use of SAT solving for the domain; I have to admit that I only consider it because it seems to be the tool of choice from the book. It's quite possible that pursuing the Event Calculus is barking up the wrong tree, and that something akin to FLORA-2 would be a better goal. Meredith also asked why Brook and not, e.g. Esterel. My answer, again, is mere familiarity: Brook already has at least one GPU implementation with which I'm familiar. I'm not familiar at all with Esterel.

My next discussion was about planet-scale applications with lift, somewhat misleadinging titled "Lift in a Box." This really consisted merely of observing that Scala and lift are becoming quite mature, that Amazon has recently announced persistent storage for their EC2 service, and that vendors such as EnterpriseDB are releasing "Cloud Edition" versions of their product, so wouldn't it be nice if we could assemble a lift distribution that included the downloadable versions of these products and allowed for one-click deployment to Amazon's EC2? The point being that this could be a game-changer for the current generation of web startups. "No garage required" is the tagline I came up with in the discussion. This conversation was, as you might imagine, quite lively, as there's a fair amount of healthy skepticism in the atmosphere about scaling any SQL database to, e.g. Facebook levels, countered by an equally fair amount of healthy skepticism that, e.g. S3 or SimpleDB are adequate substitutes. With all of that said, several people seemed quite interested both in a basic "live lift" distro that could, e.g. run from a thumb drive, and in the larger notion of easy deployment to Amazon's EC2.

The last session that I attended was nominally about ORM in Scala, but the conversation made quite clear that there are at least two interested communities: one seems to want a persistence API that might talk to a SQL database but might also not; the other seems to want something much more akin to Hibernate but specifically for Scala. My takeaway from this is that satisfying everyone's needs remains a tremendous burden, that many people acknowledge that lift's ORM solution is probably not appropriate for applications with a certain level of complexity, and that "simply" adopting Hibernate (more generally, the JPA) by virtue of Scala's support for calling out to existing Java code isn't terribly satisfactory for other reasons. If you have to build a lift webapp today, though, it's probably the way to go.

The overwhelming majority of us retired to a local Hunan restaurant for dinner. I had the great pleasure of sitting between Martin Odersky and James Iry, so there was certainly no lack of fascinating conversation over dinner. Unfortunately, perhaps thanks to having flown into town for the event late the night before, my recollection of the contents of those conversations is quite vague. Perhaps James can fill in here. :-)

Finally, Steve Yen of Accel Partners very generously drove me back to the airport. Along the drive we discussed where some value propositions might arise from Scala and lift, and the challenges inherent in identifying commercial value based on open-source frameworks.

I apologize, of course, for the nearly technical-content-free nature of this post, but I do feel that Scala is an important language, and events such as the first liftoff unconference reflect the growing community and industrial support around the language and the lift framework. I should also point out that Programming in Scala is in pre-print and available as a PDF e-book with updates, as well as in paper form upon completion.

Oh. Variance. You should have joined the "WTF is Monad" talk, hosted by Robin. We could have explained it to you in plain functorial terms.

Let me use this opportunity to remind everyone that while anonymity is not disallowed, we much prefer people identifying themselves (see the policies document and related threads for the extensive discussion of this issue).

Sorry, I didn't see a place to put my full name (and going to the user profile page, I still don't--it claims my full name is private and not shown). I'm Chris Doherty.

The guidelines recommend using your name as your user name.

Personally I'm not really disheartened. I'm more like mildly irritated that this aspect of Scala's type system should cause hang ups. I once saw a blog post where people had decided that Scala's type system wasn't for ordinary mortals because it had co- and contra-variance.

The thing is, the concept of variance is not new with Scala by any means. It's a fundamental aspect of subtyping which means it's particularly relevant to the kind of OO languages that have dominated the landscape for decades. Even in a dynamically typed language it behooves an API designer or user to understand variance issues.

As an example of how long variance has been or should have been in the mainstream it plays an important role in Meyer's "Object-Oriented Software Construction" first published in 1988.

Given that most people use C++ or Java, it's not very surprising that they don't know about co/contra-variance.

Ah, OOSC a *great* book.. I remember that I was quite puzzled by these 'object' things until I read it (I never used Eiffel though, hopefully Scala will have more success).

Given that most people use C++ or Java, it's not very surprising that they don't know about co/contra-variance.

Why is that? It certainly applies to C++ and Java. I remember the paper Contravariance for the rest of us being discussed at a programmer's conference in the early '90s. It's a pretty accessible paper, not highly theoretical, which uses C++ for its examples. (It also quite nicely lays out the connection between OO and higher-order functions.)

Given that most people use C++ or Java, it's not very surprising that they don't know about co/contra-variance.

Anton gave a good example of relevance in C++.

As for Java, we can go straight to the horse's mouth. The Java Language Specification, Third Edition uses the word "variance" 7 times (all in this context), "covariant" 6 times, and "contravariant" 1 time.

I suspect that if you approach Scala like that, you're likely to be disappointed, and maybe a little confused. Scala definitely approaches the OO/FP hybrid from the OO side, and is strongly rooted in an OO foundation rather than, for example, a lambda calculus or similar. For that reason and others, you're probably better off thinking of it as a 'usable' variant of Java. To see this, you might contrast Scala with F#, which seems to be a lot more like what you describe.

That said, knowing SML will certainly help, and will probably give you a head start in certain areas that some Scala programmers consider "advanced". On the other hand, the most novel and advanced parts of Scala are innovations on the OO side (self types, variance annotations, mixin composition, etc.), and SML will probably not help much with those.

Anyway, I really think Scala is the best thing going on the JVM, so I do encourage you to take a look.

When I want to give a metaphorical explanation of Scala I say it's as if Java and SML had an illicit love child midwifed by Haskell. It definitely favors the Java side over SML and the Haskell influence is relatively small (implicits as a poor man's way to do something like type classes, a slightly richer man's monadic sugar in the form of "for", and support for optional laziness at definition sites).

The reason I say that SML is a big influence is that there seems to be a direct transliteration of many distinctly SML constructs into Scala, and unlike the Haskell comparisons, there doesn't appear to need to be as many a "poor man's version" caveats other than perhaps a bit of verbosity.

let x = ... -> val x = ...

let x = ref ... -> var x = ...

signature -> trait

structure -> object (not class! something declared as a Scala "object" is a "singleton" or a "module")

type member/abstract type -> type member/abstract type

algebraic data type -> abstract class plus case classes and/or case objects

etc.

With all that said, "writing SML in Scala" can feel unnatural and a bit against the grain as compared to say "writing Java in Scala." For one thing, Scala's type system has too much going on for full blown HM type inference; type inference is a much more local affair in Scala. For another, ignoring Scala's OO nature is difficult since most of the libraries are written in a hybrid OO/functional form. I'm also not certain on whether there's a natural transliteration for all of SML's constructs like opaque vs transparent type members. That's my fault; my SML is pretty limited. Finally, things like pattern matching and first class functions are all implemented as syntactic sugar on top of OO constructs and that sugaring is not hidden from the user: in fact, some of Scala's most powerful idioms can arise by treating things as both OO objects and functions at the same time. If you just approach Scala as a "useful" SML variant you'll likely miss out on that aspect plus some of the modular constructs that Matt alluded to. At the same time you'll likely be disappointed by the relative verbosity of the more SML-ish constructs.

It is true that ML has had many influences on Scala, and there are many parallels. Regarding the module system, Scala mainly inherited type members, especially abstract ones. OTOH, there are fundamental differences with ML modules, the three most important ones (from the ML perspective) being that (1) ML signatures are structural types with structural subtyping while Scala only has nominal types (the signature/trait correspondence thus is not quite faithful), (2) Scala does not provide hierarchical, after-the-fact abstraction via sealing (aka opaque signature ascription), and (3) functors only have a limited counterpart in classes (especially higher-order functors).

Of course, Scala offers a whole lot of other things instead, most prominently, mixin composition, and the fact that objects are first-class.

I did warn that my SML is sadly weak. ;-) I wasn't aware that SML signatures are structural types. For the sake of completeness, I will mention that Scala has some support for structural typing, but does not support it in this context. I.e. in Scala you can declare a method with an argument that must comply with a structural type, but you can't declare that a class or object guarantees that it will implement a structural type.

I'm curious why Scala classes with mixin composition and parametric polymorphism don't give an adequate encoding of SML functors. Again it's my ignorance. I just don't understand enough of the subtleties of SML functors to even guess what the limitations of such an encoding are.

Geoffrey Washburn has weighed in on the subject and come up with an alternative way to encode SML style structural signatures and structures that I hadn't thought of.

Edit: another post talking about encoding SML functors using some experimental extensions to Scala. Not quite ready for prime time is the conclusion.

It certainly is obvious that ML's "where type" is a special case of refinement. But I'm not sure I really understand the semantics of annotating something with a structural object type, as in Geoff's example. This seems to be a relatively new feature, is that right? Does it really hide away the representation of nat.T? If so, how does Scala avoid running into the "double vision" problem for recursive definitions?

Also, even if this works, I think it is still fair to say that Scala cannot encode sealing in the general case. In particular, sealing in ML is really a module expression, so you don't need to bind the result. More importantly, it is hierarchical: it applies to nested structures. For example, I don't think there is a Scala equivalent for something like

signature S = 
sig
   structure A : sig type t; val x : t end
   val f : A.t -> int
end

structure M =
struct
   structure A = struct type t = int; val x = 666 end
   fun f x = x
end

structure N = M :> S

But I would be interested in being proven wrong...

Re the functor example: I'm afraid I don't understand its intent. Why perform sealing with a signature that actually contains no abstract type?