More on the "generics are evil" meme

Bruce Eckel, famous author of programming books Thinking in Java and Thinking in C++, writes some commentary about Arnold's article on generics (discussed here on LtU):

Ken Arnold just posted a blog entry describing how difficult he found generics when working on the upcoming edition of The Java Programming Language book.

The fact that it's taken this book so long to get published is just one indicator of the complexity of the language. I've also noticed that most other Java books have either taken a long time, or just briefly glossed over generics, and often both.

[...]

One commenter to Ken's blog points out that it's not the idea of generics that is the problem, but the implementation, to which I agree, and I also think this is what Ken actually meant.

Here Eckel seems to agree with many opinions expressed in the previous discussion: that's not a problem with generics in general, but with the Java implementation. And he argues that it would be better if the language was designed with generics in mind in the first place, which I don't think anyone would disagree with.

But the question is, how can we change the course of a moving train? It seems to be never easy. Can PL theory help the evolution of programming systems and not only their design?

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

When Generics Fail - NOT

Bruce Eckel, famous author of programming books...

Perhaps it's worth knowing that he also boldly posted an Artima article titled When Generics Fail

Here's an example of a situation where I needed Java generics to work -- and where C++ templates would have worked -- but they failed. I think this not only shows what parameterized types are really about (writing code that works with a broader range of types) but also why latent typing is so important.

The article was deleted after someone demonstrated a simple solution using Java Generics.


See both solutions here: Generics vs. Dynamic Checking

Java Generics and PL theory

I think that it should be worth to think about Guy Steele's "Growing a Language". IIRC the two key points he hoped to see in the future of Java were Generics and operator overload.
So it seem that this is some kind of designed evolution, and this could mean something for people smarter than me.

About the generics problem.. Is'nt this a perfect example of PL theory that helps evolution of a programmins system?

The people behind the GenericJava/Pizza projects were great minds, so maybe what this rejection of Java's new features shows is that there is no chance in which theorethical achievements can easily translate to common developers.

Growing a mainstream language is tough

so maybe what this rejection of Java's new features shows is that there is no chance in which theorethical achievements can easily translate to common developers.

There are a number of reasons why this isn't supported by what has happened.

First, Guy Steele's talk doesn't necessarily indicate that generics were planned from early on in Java's design - if they had been, you would think that some support would have been built in at the VM level.

Second, the fact that support for generics doesn't exist at the VM level, and the desire for backward compatibility, were constraints on the design of Java generics, and are part of the reason people are having problems with them now. So this tells us that grafting features onto languages after their initial design can be problematic, which we already knew.

Third, although there were some very smart minds behind generics in Java, they were still limited by the basic features of Java's type system. They couldn't simply throw that out and switch to a more functionally-oriented polymorphic type system, for example.

I think what this all really tells us is that it's very difficult to get from a mainstream language to theoretical nirvana in a series of evolutionary steps. It's much more likely to require a revolution — new languages which satisfy the needs and at least some of the expectations of mainstream developers, and is thus attractive to them, but which still have more desirable theoretical properties and capabilities than current mainstream languages.

Such languages may not even be that radically different from existing languages, but they'll need certain theoretical design features built in at a low level from the beginning, no matter what interface is exposed to programmers at the highest levels.

I expect it isn't controversial to say that I don't think any of the existing functional languages are likely to fit that bill — not that they couldn't handle the job technically, but that they don't have enough appeal to mainstream developers.

Java's Generics, behind the scenes

I think that the story of Java's generics is a bit unfortunate. I strongly agree with Anton's assessment that the team that put together the Generic Java (GJ) project and prototypes and handed these to Sun were brilliant people. Unfortunately, Sun made some poor, unpublicized back-room decisions that greatly affected what was released.

One of the primary design constraints was that code written using generics must run on any JVM. (This was "insisted on by Sun" according to Philip Wadler.) This led to the approach of using type erasure to remove the meta-information about the generic types so that no changes would have to be made to the JVM. The prototypes by the GJ team and Sun's earlier prototypes worked wonderfully this way.

I feel that the design constraint was reasonable--it allowed "write-once-run-anywhere" to be meaningful, and it allowed programmers to write the best code they knew how, getting code reuse and type safety with collection classes, and letting it run on any JVM.

This was approved by the companies and individuals that contributed to the JSR-14 proposal over the course of several years.

However, late in the game, Sun made a closed-door decision to break this backward compatibility (or "forward compatibility" as Neal Gafter called it.) By his admission, Neal was Sun's entire compiler team for Java. On 2003-02-20, I got an e-mail from Neal that said:

You reported a problem with the last generics prototype that it could not be used with generics enabled (-source 1.5) to generate code to run on an older VM (-target 1.1, for example). We debated this issue at great length yesterday, and our decision is that the product (Tiger, 1.5) will not support that kind of forward compatibility. Consequently, I will not add support for that into the generics prototype.

I should clarify that Sun didn't need to "add support" for this--it had worked perfectly for years. (In later releases, you could still patch one line of Sun's compiler and make it work again.) Sun never publicized this decision. They did not even inform nor request input from the GJ team that made the original working prototype. Philip Wadler expressed surprise to me when I informed him of Sun's decision:

Thank you for your note and your efforts. This decision is news to me. I've written to my contacts at Sun to ask what is going on. The GJ design was predicated on the constraint (insisted upon by Sun) that GJ had to work without changing the Java vm. So I'm quite surprised if this has been casually reversed.

Sun even went so far as to censor any discussion of these issues from their forums! They performed a huge selective deletion of almost every thread that mentioned these changes. (I do have copies of some of the threads from Google's cache if anyone's interested.) They refused to discuss the technical reasons, either, except for some thin reasons that were easily dismissed (e.g. "we had to add some new exceptions to support this.")

I tried really hard to get Sun to at least discuss this with their users, and come to a better solution given the new constraints, or at least not break something that already worked, but they didn't. They "won" by being silent about it, and censoring the issue. I wrote to the representatives of almost every company that participated in the JSR-014 spec. Later e-mails and forum postings from their compiler team noted that Sun simply didn't have the resources to test generic features on old JVMs, so it was largely a cost-cutting measure.

Given the original design constraint, type erasure was one of the few ways to make code run on old JVMs. However, when Sun decided to abandon this constraint, and forced an entirely new VM to run generic programs, I think most people here would agree that type erasure became one of the least effective methods for implementing generics, and introduced shortcomings that are simply there because of a design constraint that no longer existed.

Re: Java's Generics, behind the scenes

I should clarify that Sun didn't need to "add support" for this--it had worked perfectly for years. (In later releases, you could still patch one line of Sun's compiler and make it work again.)

I have been playing around with getting around code compiled under a 1.5 javac to run on a 1.4 JVM. The result is that it's trivial. It's a single bit which indicates the version of the class file and once that's been twiddled the class files will happily run on 1.4 JVMs. Of course the Java libraries have changed so you could well run into problems but it at least shows that the desire for backwards compatibility was realised, only to be covered up.

Not That Trivial

The result is that it's trivial.

Making programs compiled for the 1.5 VM work under a 1.4 VM, is definitely not flipping a single bit. There were several adjustments in the class file format, like the changes to the LDC instruction and the changes to access specifiers. Changes to the language, like extended for loops, autoboxing, and enums, all require new runtime support. I wrote a free tool over a year ago, Retroweaver, that takes care of these basic issues.

If you're looking for some historical background on generics support, here's a thread where Neal sheds some light on the issue.

Edit: Oh, and a version number change is hardly a "cover up". What the version numbers are, what they stand for, and how they are used is all documented in the specification. Damn that sneaky Sun.

-- Java.Next

Not sure I Understand

Haven't looked closely at the 1.5 JVM. Have any new instructions been added? Or are we just talking about a tweak in the class file layout? Personally, I think the VM needs to be subject to change just like any other part of the library or language spec.

At any rate, I'm not sure I understand the nature of the problem. Was Sun faithful to the GJ implementation? Do generics work as GJ designed (making the assumption you upgrade the VM)?

So is the complaint (a) Generics shouldn't require an upgrade to the JVM? or (b) Generics could've been designed better had they not had backward compatibility as a constraint?

Re: Not sure I Understand

So is the complaint (a) Generics shouldn't require an upgrade to the JVM? or (b) Generics could've been designed better had they not had backward compatibility as a constraint?

Both-ish.

  • It seems that Sun wanted a generics implementation which would be backwards compatible with older JVMs which is what they got but then decided against the backwards compatibility for seemingly strange reasons.
  • Many people think that using erasure for generics is the wrong way to do generics as you loose information when you compile.
        Set<Number> set = new HashSet<Number>();
        set.add(new Integer(1));

    I may be wrong, but I've looked and can't seemingly solve it: at run time, how do you find out that the type parameter for Set has been assigned to Number? Erasure makes sure that at compile time all the type parameters are correctly assigned and all is well type-wise, but then ends up inserting casts into the Java bytecode. So you loose the information about what types have been assigned to the type parameters.

If set is a field, then the J

If set is a field, then the Java compiler records its generic type in the class file format with a Signature attribute. You can get to that static type information with the updated reflection API in 1.5.

The updated JVMS (Java Virtual Machine Specification) is the appropriate source for obtaining this kind of information.

-- Java.Next

downcasts, covariant arrays

Java's generics problems are mostly caused by two holes in the type system: downcasts and covariant arrays. Runtime checks are really just a security feature. From the perspective of the static type system, those features are unsafe. The addition of generics simply exposed and amplified existing problems (and, as a result, has become the scapegoat).

If Bruce Eckel would stop screaming "latent typing" he might realize that what he really wants is structural typing (along with either static type inference or dynamic typing).

A much misunderstood issue

I've never shied away from explaining what was going on with Generics, and it seems misinformation abounds. I'll try to keep it short and sweet.

Sun tries to keep Java to as close to one language as possible. It is not a whole family of languages with slight variations. Up-versioning the language from one release to the next necessarily compromises this ideal, but the goal is furthered by not allowing implementors to pick and choose which language features they implement. In the case of JDK5, you either get all the new features - including Generics, Autoboxing, and Enums - by using "-source 1.5" or you use an earlier version of the language by using "-souce 1.4" or something else.

Some of these features require core library changes or API changes. Generics come with a set of reflection libraries that are not implemented in any 1.4 VM, and could not be so implemented without violating the Java conformance guidelines. Autoboxing requires new methods in Integer and elsewhere. Enums require new platform classes and changes in java.lang.Class. The for-each loop requires interface changes in java.lang and java.util. If you enable the new language features, then, you had better have a new VM to run the code on. Thats why javac requires -target 1.5 when you use -source 1.5. Javac could easily be rigged to generate 1.4-format class files, but many of the language constructs would simply not work under a 1.4 VM. To be considered a Java implementation, the language features must work.

In addition, there are indeed some changes in the generated code under -target 1.5. Specifically, a class literal is compiled to a single instruction in JDK 1.5. String concatenation uses new APIs that are not present in 1.4. When you compile against a 1.5 JDK, you're running overload resolution against all available methods, not just the old ones, so you might end up generating code to call new methods, even if the source hasn't changed. You can't just flip a version bit and expect it to work.

I should not mention the javac flag "-target jsr14", which causes javac to generate maximally-1.4-compatible code with the new language features enabled. That flag is useful for bootstrapping and testing javac, but you should not depend on it. If it doesn't work as you want and you report a bug about it, you can expect the flag to be removed.

Please feel free to implement a Java compiler that supports the new language features - ALL of them - and yet generates code that runs on 1.4. Many people will be happy to see it. I don't know why this is such a bigger deal now than it was in the JDK 1.4 days, when you had to use a 1.4 VM if you enabled assertions.

Then why hobble the generics?

I believe the gripe is that if there didn't need to be such strict backward compatibility, then the compromise to make the generics system fit the 1.4 VM wasn't necessary. The generics handling could have then been much more elegant and easier to use.

Re: Then why hobble the generics?

Nobody has figured out how to support backward compatibility (the use of existing byte and source code in the new VM and language, which is fully supported) and migration compatibility (see http://gafter.blogspot.com/2004/09/puzzling-through-erasure-answer.html) any other way.

If I remember my Java history.

Such issues of migration compatibility didn't play much role when the collections were last overhauled (1.1 to 1.2 IIRC). This non-migration compatibility was compromised even though changes to the language and underlying VM were not needed for the upgrade - (owing mostly to the original collection classes being badly designed such that a clean set of API's were in order).