Lambda the Ultimate

Note to Ehud: server response time is getting so bad that posting news items is just too painful.

This white paper on Mark Hammond's efforts to port Python to .NET provides an interesting perspective on this question.

The underlying mechanisms for the CLR as well as for the JVM *are* dynamic. The problems for making these systems universal are in the necessity to use nearly the entire language model of C# or Java, rspectively, in order to achieve any kind of integration.

A better approach, IMHO, would be to make integration more abstract and more loosely coupled than the "shared object model" of the CLR or JVM.

Components written in various languages should be integrated via a simpler, asynchronous, shared-nothing, message passing model. Then those components could be instantiated in the same OS process, a different OS process, or on a distinctly different host on the network, without having to alter the mechanism for integration.

A virtual machine in this model would offer a reusable implementation of garbage collection, component instantiation, time-sharing, channel discovery, message formatting, optimized "intra-OS-process" message passing, and an instruction set for sequential processing. A universal virtual machine would not require a shared-memory, monitor-based, object model.

I would choose Scheme as the "instruction set" of my virtual machine, and not make language implementors deal with low level CPU models.

Apparently, the main issues are portability and the support of polymorphic scalars, closures, co-routines, and continuations.

We have to run on any of a half-zillion platforms, and .NET is windows only. Mono makes that somewhat better, but still... got Mono for a Cray system, a VMS system, or a Palm? Probably not. I certainly don't.

...

To make perl work means completely rewriting the system allocation scheme, and using our own custom polymorphic object type. In JVM/.NET bytecode. Doable? Sure. Fast? No way in hell.

And continuations. Yow. To do continuations is non-trivial, and I don't think it's possible to do in the JVM or .NET without treating them as glorified CPUs and use none of their control and stack features.

Besides, continuations are really cool. While I didn't know about them back when the mail you reference was written, I do now and, well, I can do all sorts of nifty and profoundly evil things if I have them at hand, and what better reason to include them? (Well, OK, there's the fact that I need them if Parrot's to properly run both Ruby and Scheme, but...)

Gregory Williams wrote: "...and I don't think it's possible to do in the JVM or .NET without treating them as glorified CPUs and use none of their control and stack features."

Regarding .NET and continuations, David Simmons (Smalltalk VM guru and prime developer of SmallScript/S#) has support for continuations and much, much more.

He has been working on enabling SmallScript/S# (do not be fooled by the name, it is probably one of the most capable dynamic languages in existence today) on the .NET platform. Highly recommended reading at:

http://www.smallscript.org/Downloads/SSharp_NET_Notes.asp

So in short - yes, it can be done - no, it isn't easy.

Having talked with Dave Simmons and seen his presentations on what Evil Things he's done to .NET, my statement stands. I should've perhaps gone one step further and added "Or rewrite their firmware, in ways akin to the Alpha's PAL code, by changing their behaviour with add-on C code."

Which is what Dave's done--to get the features he needed, he wrote add-on native code in some spots. To get features equivalent to what you get with his very nice AOS platform on .NET you have to run as untrusted code because of what he's had to do.

This isn't to belittle his acoomplishments in any way, but he has had to change the base platform to get what he needs done.

Yes, I agree that what David has had to do should probably be classified as "cheating". :-) By the way, Squeak has now gotten support for real closures and continuations - work done by Anthony Hannan (not yet included in a released version though):

http://minnow.cc.gatech.edu/squeak/2981

He has reworked Compiler, Debugger and the VM in a pretty impressive way and we are right now planning on how to integrate all that work into version 3.6 or later. And cool enough the new VM is plenty faster too.

Who needs CLR! ;-)

I've read your blog entry regarding Parrot vs. JVM and CLR. In it you write:

Perl 5 has two big features that make using the JVM or .NET problematic--closures and polymorphic scalars ... To do closures means capturing and maintaining persistent lexical state. Neither .NET nor the JVM have support for this, as they use a simpler stack-based allocation of lexical variables. To handle lexicals the way perl needs them means we'd have to basically ignore the system variable allocation system and do it ourselves.

And also

In this case mostly closures and continuations, which are the biggies. (weakly, runtime typed variables are less of a problem, though still a problem) The answer is yes, but they'd be stupid to do so. As I said, those features have unavoidable overhead. Running perl faster at the cost of running C# slower is not, at least in my estimation, a good tradeoff.

Emphasis is mine. You are the VM expert, not I, but it seems to me that at least for closures you can implement them in a way so that you only pay for them if you use them. IMO it should be pretty straightforward to determine at compile-time if a scope is being used as a closure. If its not then the standard stack-based allocation of lexical variables can be used. Only if a scope is being used as a closure would you allocate it as an object on the free-store (and place a ref to it on the stack).

Using this technique would mean that C# and VB.NET would not have to pay the price. Even Perl would benefit as I assume that most scopes aren't used as closures.

BTW, JScript.NET does implement closures, for example the following sample (based on Paul Graham's Accumulator Generator pattern):

function accumulator(n : int) {
  return function(i : int) : int {
    return n += i;
  };
}

var acc = accumulator(3);
print(acc(2));
print(acc(3));

compiles just fine using jsc /fast. I don't know how MS implemented this, nor how efficient their implementation is.

Still, I think I may have been a bit over-absolutist--if you don't mind things being really expensive in the case where you use closures, the non-closure case can still be cheap. I sort of alluded to this, but I wasn't clear and I think I got a bit enthusiastic for rhetorical reasons.

Bad Dan, no cookie for me! :)

As I have previously stated I think that delegate support in the .NET VM could be leveraged to implement functional languages on that framework. Yes, the price you pay for closures is indeed a bit high, but you only pay for the closures you actually use.

But this type of payment is common in environments such is the JVM or .NET, at least when compared to a PL such as C++. The very fact that you use a VM, coupled with extensive heap usage and the GC, etc.

Closures and lambda's are some of the things I miss most when working in C++. Getting them in C# makes the switch tempting. Regardless, I think I'll wait until .NET becomes much more prevalent and C# gets generics.

I think the Metrowerks approach is better (putting aside compatibility issues) because exceptions are, after all, exceptional.

This, of course, is not the same as closures. I, for one, would use closures quit a bit if I had access to the feature. But I believe that the .NET team assumes that closures usage in C# would be limited, and I think their assumption is correct in this case. Being able to add this feature to C# without having to modify the VM or degrade the performance of existing code is therefore quit a win for them.

I not sure what Sam Ruby is referring to when he states that "The danger is that 20% of the time you will get a result that you didn't expect".

But obviously if using closures has a large overhead, you guarantee that their usage would be limited. It's a vicious circle.

Basically there are reference types, like objects, that live in the heap, and value types, like integers and structs, that live on the stack. If you look at Miguel's example, you can see where the problem lies. If you're closing over a reference type, everything works since you're just passing around a reference to a heap variable. If, however, you're closing over a value type you're not copying in a reference to the variable but rather the value of the variable. That means that if multiple closures are made over the same variables, they won't interact as they should for value types, but will for reference types.

There are probably other issues, but that's the big one that springs to mind right from the start.

People use high-level PL features all the time that have significant runtime overhead because they make the development process easier/faster. You might even say that this is true for PLs themselves.

As I have pointed out MS made numerous design decisions in this vain with the move from native win32 programming to .NET. The same can be said about Sun and the JVM and the various scripting languages out there.

I think that unless the overhead is exceptionally high (in which case articles will appear admonishing developers not to use this feature) this feature will be used. If developers see samples on MSDN showing how this can make their code short and cool, they will copy these samples.

Having support for closures where non existed before is IMO a GOOD THING. If lots of software will leverage this capability I'm guessing that MS will invest effort in improving performance.

Now if MS only adds tail recursion ...