Lambda the Ultimate

Good to see Maxine getting some interest, and on the front page of LtU too.

The C1X compiler was the last thing I did before I left Sun, and I think we did a pretty good job separating the runtime system from the compiler and made some pretty big improvements to the performance of Maxine. There are some details in our VEE 2010 paper Improving compiler-runtime separation with XIR.

I noticed that Tony Hosking opened a Codehaus project to port C1X to JikesRVM. We talked at length about this a few times, and I hope he gets funding and volunteers to do it. MmTk (and perhaps the Harmony libraries) aside, there has been precious little code reuse in VM projects and I hope this idea may get some traction.

Non-ACM access to Improving Compiler-Runtime Separation with XIR paper.

As the developer of the Maxine Inspector since its infancy (though not from birth) I can add some background on the choices mentioned here: Java and Swing.
First Java. The early Inspector was co-designed with the VM, and was in fact essential: no other tools would support the VM's development. An (inspired, IMHO) early architectural decision made by the original development team (before I joined) was to expedite the ability of the Inspector to analyze and model VM state by sharing code with the VM: not only some of the core VM code, but also the specific implementations of various VM functions that might be bundled into a particular VM build. This, combined with Java reflection, allows the Inspector to "read the bits" from the VM process and pretty much "disassemble" the entire VM state. The part of the Inspector that does this, which I have gradually been disentangling from the presentation layer, can only be Java. Furthermore, the ability to load both the VM and Inspector code into the IDE of my choice, and to browse, refactor, and build them both in place, has greatly amplified my productivity while working on Maxine.
Now Swing. The original choice of Swing was the natural default in that time and place, and I chose not to change it for a variety of reasons. Yes, programming in Swing is an incredible PITA: one of the worst parts of the job. I dislike many things about it, and I know more than a few of the guilty parties. Enough said about that. On the other hand, it wasn't a bad choice. For one thing, the Inspector has never been about aesthetics, but rather about expediency: helping VM developers with the incredibly challenging task of diagnosing failure by looking at the bits of a broken VM state. A developer needs access to a lot of state, must be able to think about it simultaneously at many levels of abstraction (bits, words, memory addresses, data structures, language constructs, objects, etc.), and must do so in the presence of likely breakage. I submit that this is mainly a numeric/textual exercise (as is programming itself), for which I've exploited the ability of Swing to create complex, fine-grained display and interactive behavior for otherwise ordinary text displays. Once that's available, the real power of the Inspector derives from the sophistication of the analysis/modeling code. I'm not enthusiastic about rich graphics for this sort of thing, a topic I'll expand upon in a separate comment.

Text is really just one visual representation: one way to View Source for an object.

In a fully object-oriented display system, you should be able to sink/hoist visual representations. For example, Henry Baker once pointed out that the ideal way to read a linear object / linearly typed resource is graphic in nature, because text itself doesn't really communicate the linearity of the resource well. The same logic applies to graph grammars, or grammars in general, such as web grammars. There is more to languages than manipulating text.

Something I've been toying with: In a dynamically federated, dynamically distributed system, like the Internet, you can can control representation of resources. This means you can recursively model various visualizations of resources as simple transformations. For example, minification tends to obfuscate code but can produce a tighter wire format while preserving the original semantics of the code. So what you are fundamentally doing here is manipulating a shadow of a resource, not the resource itself. If you can provide a schematic mapping for your obfuscator, then prove that schematic mapping is complete and sound, then you now have a degree of freedom is how you think about text-agents. A hot button here is that most mash-up tools for the Web demonstrate a complete non-understanding of this principle and how powerful it can be, and that this problem, in my humble opinion, is greatly exacerbated by the design of modern Web browsers.

Ergo, I think it is not enough to have a meta-circular VM. You want a recursive virtual machine -- a virtual machine that is objects all the way down. Actually, representations of objects. Then all your bookkeeping becomes agents that manage those representations.

I am interested in you expanding upon your position on not being enthusiastic about rich graphics, in the separate comment you've promised.

There have been some enthusiastic discussions lately about the problems of black box technology creating an impediment to effective autodidactism in budding programmers. In my own anecdotal experience, the greatest system builders I've worked with all got their start piecing together environments from the barest metal on up.

These two things -- autodidactism and bottom-up system building experience -- are not equivalent.

I am self-taught, but I tend to learn top-down and breadth-first, owing to being highly associative. I only appear to have bottom-up knowledge, because I've simply put in so much time learning and doing.

Anyway, I'm with you, one-by-one Java is finally getting everything Smalltalk had given the world in 1980. It only took thousands of Java developers ~15 years to do what it took a small team of Xerox PARC researchers ~8 years. (Side note: Code Bubbles, previously discussed on LtU, uses an Eclipse backend that communicates via Java XML-RPC to a Windows Presentation Foundation frontend.)

Java is a very hard language to bootstrap. It is even harder to bootstrap with "official" JDK and set of required VM features. Most of this is due to the lazy initialization and dynamic class loading model but a large part is also the poorly factored design of the JDK classes.

There have been half a dozen or so really serious tries at this, but unfortunately Java was never really designed for this and seems to fight you every step of the way. We had to work around this in Maxine in several ways, with some shortcuts.

That said, I think the Maxine inspector is certainly more feature rich and robust than anything that was ever available for Smalltalk, or for any other meta-circular VM project. Michael Van DeVanter put a huge amount of time and energy into making the Inspector not just a tool for controlling and using a "working" VM, but a tool for debugging a broken VM. And there have been a number of innovations there, some of which aren't published yet, including automatic relocatable watchpoints and lots of subtle tricks for continuing to provide debugging service through GC periods, etc. While the UI is ugly and full of machine-level grit, it is also thoroughly laced with both VM level and object-level information and hyperlinks (in a sense) that are automatically derived and verified robustly by the inspector itself.

Awhile ago I read his work on language-based editing environments and so I don't doubt his attention to detail.

I think the problem with Java visualization is more systematic than a personal responsibility of one person; awt was implemented in one month and Swing made the poor decision to be backward compatible with awt, to get everyone psyched for JavaOne. Swing was also designed before HLSL was standardized, back when everyone was still writing games at a very low-level without support like OpenGL and DirectX, and its lack of retained mode design reflects it. While JavaFX has promise, it remains to be seen what Oracle will do with it; the 2010 Winter Olympics will be an interesting test (TIOBE Index never even listed it as an up and coming programming language discused on the Web, even after JavaOne, and even in April 2010 they say it is only "now approaching the top 20", whatever that means as a statement from a Java vendor), and some of NetBeans' add-on libraries for it are (IMHO) poorly designed.

This really just reflects my frustration with trying to build a new language with visualizable code fragments. When I get a cool idea for something, the amount of effort to express that idea can be daunting and I get headaches trying to implement it. As inspiring as something like Code Bubbles is, it is inherently not meta-circular. So you can really just add awt and Swing to the list of problems with building a meta-circular JVM.