Lambda the Ultimate

so maybe what this rejection of Java's new features shows is that there is no chance in which theorethical achievements can easily translate to common developers.

There are a number of reasons why this isn't supported by what has happened.

First, Guy Steele's talk doesn't necessarily indicate that generics were planned from early on in Java's design - if they had been, you would think that some support would have been built in at the VM level.

Second, the fact that support for generics doesn't exist at the VM level, and the desire for backward compatibility, were constraints on the design of Java generics, and are part of the reason people are having problems with them now. So this tells us that grafting features onto languages after their initial design can be problematic, which we already knew.

Third, although there were some very smart minds behind generics in Java, they were still limited by the basic features of Java's type system. They couldn't simply throw that out and switch to a more functionally-oriented polymorphic type system, for example.

I think what this all really tells us is that it's very difficult to get from a mainstream language to theoretical nirvana in a series of evolutionary steps. It's much more likely to require a revolution — new languages which satisfy the needs and at least some of the expectations of mainstream developers, and is thus attractive to them, but which still have more desirable theoretical properties and capabilities than current mainstream languages.

Such languages may not even be that radically different from existing languages, but they'll need certain theoretical design features built in at a low level from the beginning, no matter what interface is exposed to programmers at the highest levels.

I expect it isn't controversial to say that I don't think any of the existing functional languages are likely to fit that bill — not that they couldn't handle the job technically, but that they don't have enough appeal to mainstream developers.

I think that the story of Java's generics is a bit unfortunate. I strongly agree with Anton's assessment that the team that put together the Generic Java (GJ) project and prototypes and handed these to Sun were brilliant people. Unfortunately, Sun made some poor, unpublicized back-room decisions that greatly affected what was released.

One of the primary design constraints was that code written using generics must run on any JVM. (This was "insisted on by Sun" according to Philip Wadler.) This led to the approach of using type erasure to remove the meta-information about the generic types so that no changes would have to be made to the JVM. The prototypes by the GJ team and Sun's earlier prototypes worked wonderfully this way.

I feel that the design constraint was reasonable--it allowed "write-once-run-anywhere" to be meaningful, and it allowed programmers to write the best code they knew how, getting code reuse and type safety with collection classes, and letting it run on any JVM.

This was approved by the companies and individuals that contributed to the JSR-14 proposal over the course of several years.

However, late in the game, Sun made a closed-door decision to break this backward compatibility (or "forward compatibility" as Neal Gafter called it.) By his admission, Neal was Sun's entire compiler team for Java. On 2003-02-20, I got an e-mail from Neal that said:

You reported a problem with the last generics prototype that it could not be used with generics enabled (-source 1.5) to generate code to run on an older VM (-target 1.1, for example). We debated this issue at great length yesterday, and our decision is that the product (Tiger, 1.5) will not support that kind of forward compatibility. Consequently, I will not add support for that into the generics prototype.

I should clarify that Sun didn't need to "add support" for this--it had worked perfectly for years. (In later releases, you could still patch one line of Sun's compiler and make it work again.) Sun never publicized this decision. They did not even inform nor request input from the GJ team that made the original working prototype. Philip Wadler expressed surprise to me when I informed him of Sun's decision:

Thank you for your note and your efforts. This decision is news to me. I've written to my contacts at Sun to ask what is going on. The GJ design was predicated on the constraint (insisted upon by Sun) that GJ had to work without changing the Java vm. So I'm quite surprised if this has been casually reversed.

Sun even went so far as to censor any discussion of these issues from their forums! They performed a huge selective deletion of almost every thread that mentioned these changes. (I do have copies of some of the threads from Google's cache if anyone's interested.) They refused to discuss the technical reasons, either, except for some thin reasons that were easily dismissed (e.g. "we had to add some new exceptions to support this.")

I tried really hard to get Sun to at least discuss this with their users, and come to a better solution given the new constraints, or at least not break something that already worked, but they didn't. They "won" by being silent about it, and censoring the issue. I wrote to the representatives of almost every company that participated in the JSR-014 spec. Later e-mails and forum postings from their compiler team noted that Sun simply didn't have the resources to test generic features on old JVMs, so it was largely a cost-cutting measure.

Given the original design constraint, type erasure was one of the few ways to make code run on old JVMs. However, when Sun decided to abandon this constraint, and forced an entirely new VM to run generic programs, I think most people here would agree that type erasure became one of the least effective methods for implementing generics, and introduced shortcomings that are simply there because of a design constraint that no longer existed.

I should clarify that Sun didn't need to "add support" for this--it had worked perfectly for years. (In later releases, you could still patch one line of Sun's compiler and make it work again.)

I have been playing around with getting around code compiled under a 1.5 javac to run on a 1.4 JVM. The result is that it's trivial. It's a single bit which indicates the version of the class file and once that's been twiddled the class files will happily run on 1.4 JVMs. Of course the Java libraries have changed so you could well run into problems but it at least shows that the desire for backwards compatibility was realised, only to be covered up.

The result is that it's trivial.

Making programs compiled for the 1.5 VM work under a 1.4 VM, is definitely not flipping a single bit. There were several adjustments in the class file format, like the changes to the LDC instruction and the changes to access specifiers. Changes to the language, like extended for loops, autoboxing, and enums, all require new runtime support. I wrote a free tool over a year ago, Retroweaver, that takes care of these basic issues.

If you're looking for some historical background on generics support, here's a thread where Neal sheds some light on the issue.

Edit: Oh, and a version number change is hardly a "cover up". What the version numbers are, what they stand for, and how they are used is all documented in the specification. Damn that sneaky Sun.

Haven't looked closely at the 1.5 JVM. Have any new instructions been added? Or are we just talking about a tweak in the class file layout? Personally, I think the VM needs to be subject to change just like any other part of the library or language spec.

At any rate, I'm not sure I understand the nature of the problem. Was Sun faithful to the GJ implementation? Do generics work as GJ designed (making the assumption you upgrade the VM)?

So is the complaint (a) Generics shouldn't require an upgrade to the JVM? or (b) Generics could've been designed better had they not had backward compatibility as a constraint?

So is the complaint (a) Generics shouldn't require an upgrade to the JVM? or (b) Generics could've been designed better had they not had backward compatibility as a constraint?

Both-ish.

• It seems that Sun wanted a generics implementation which would be backwards compatible with older JVMs which is what they got but then decided against the backwards compatibility for seemingly strange reasons.
• Many people think that using erasure for generics is the wrong way to do generics as you loose information when you compile.
  

      Set<Number> set = new HashSet<Number>();
      set.add(new Integer(1));

  I may be wrong, but I've looked and can't seemingly solve it: at run time, how do you find out that the type parameter for Set has been assigned to Number? Erasure makes sure that at compile time all the type parameters are correctly assigned and all is well type-wise, but then ends up inserting casts into the Java bytecode. So you loose the information about what types have been assigned to the type parameters.

If set is a field, then the Java compiler records its generic type in the class file format with a Signature attribute. You can get to that static type information with the updated reflection API in 1.5.

The updated JVMS (Java Virtual Machine Specification) is the appropriate source for obtaining this kind of information.

-- Java.Next

I believe the gripe is that if there didn't need to be such strict backward compatibility, then the compromise to make the generics system fit the 1.4 VM wasn't necessary. The generics handling could have then been much more elegant and easier to use.

Such issues of migration compatibility didn't play much role when the collections were last overhauled (1.1 to 1.2 IIRC). This non-migration compatibility was compromised even though changes to the language and underlying VM were not needed for the upgrade - (owing mostly to the original collection classes being badly designed such that a clean set of API's were in order).