Lambda the Ultimate

(none)

The DLR provides a much nicer interface to write a dynamic language than is to write straight to the JVM or CLR.

The CLR supports proper tail call elimination, which helps a lot implementing erlang style actors.

Aditionally, if you use mono as your CLR VM, you get support for delimited continuations, which helps even further to implement actors.

DLR is very nice for generating code; its very fast and the Linq.Expression API is easy to use as well as expressive (since DLR/.NET 4.0). You don't have to bother with GC (if you don't want to), while you can take advantage of certain CLR libraries so your language can be used to actually do something rather quickly.

This would apply to the JVM also once Da Vinci is implemented, but right now DLR is a convenient choice.

AFAIU, vs. e.g. JVM or CLR or LLVM or other not-dead-yet things.

For a honors thesis you shouldn't wish for anything thing more than some DSL or a shallow embedding of that in a known language.

Of course it depends on what you are doing, where your experience is, how much you want to learn, and so on...

DLR is accessible if you are already doing F/C# programming, getting ramped up on another tool could take time (you'll still need a custom parser BTW, if you are doing your own syntax). Many DSLs are done in Scheme or Haskell or whatever, and you could always go straight to your own run-time.

I'm wondering if modeling languages are a tangent: they are great if your purpose is formal exploration of a system, but if you are looking at design, usability, utility, and ergonomics, then they don't add anything.

if you are looking at design, usability, utility, and ergonomics, then they don't add anything

What does? How do we validate a requirement or verify a claim about design, usability, utility, or ergonomics of a programming language?

I can't even imagine how to efficiently and humanely perform comparative user studies with a new language to measure 'usability' and 'ergonomic' properties between distinct languages.

1. How do you control for such variables as quality of documentation and interactive tutorials affecting the usability of the language (independently of the language's intrinsic properties)?
2. How do we produce a set of sample problems that will be unbiased across language paradigms? (e.g. A "Hello World" of reactive programming is to draw and continuously maintain an alarm clock. A "Hello World" of mobile computing puts the words on someone else's console.)
3. Which is most reasonable: do we control for publicly available 'libraries' (i.e. limit to 'standard' libs) or are those considered among the language's features?
4. How do we fairly compare maintainability given the wide variability of modularization within a language?
5. How do we afford to test enough developers for enough problems that sample sizes reduce the enormous variability among programmers?
6. How do we control for population biases, i.e. towards familiar syntaxes (such as algol-derived syntax, or English-based keywords) that may be independent of intrinsic usability and ergonomic properties?

I'm assuming an acceptable operational definition of 'usability' and 'ergonomics' with regards to languages, but I'm not even convinced that such definitions exist.

Assertions about usability and ergonomics are more likely anecdotal hand-waving than controlled empirical observation. (To be more forgiving, I could say that it's more an 'art' than a 'science'.) We can't practically study these properties scientifically across languages, at least not in any manner that allows us to assert which variable made the difference. We could, perhaps, achieve incremental improvements to an existing language (i.e. small tweaks) because that would allow us to control more variables.

But that really leaves studying these properties formally, but indirectly. We don't have a great model for how local reasoning (about performance, safety, progress), coupling, live programming, number of distinct language concepts, and so on affect 'usability'. But we can describe those properties and study them formally. If that is your approach, then a formal model of the language (including its syntactic aspects) might well add something useful.

A lot of language design IS art more than science: you need to prototype your language and see how it works for yourself and other real people. There are no modelling techniques that can be used to approximate this, and you are right, we are stuck with that reality. It doesn't make the task any less worthwhile.

Most languages start real and then, if appropriate, you can abstract out a small version of the language to act as a model for reasoning about its formal properties; i.e., Featherweight Java rather than actual Java (Java came first, of course). Alternatively, if your goals are completely formal, then by all means start with COQ or the equivalent. If you are designing a language for Boeing or NASA, then you'll have to wind up there sooner rather than later.

A lot of language design IS art more than science

Reminded me of Einstein. One of his personal believes was that a theory should just look 'right,' i.e. be concise and elegant, in that sense, I would call science art.

Except for the 'little detail' that Alan Aspect's experiments has shown that the theory which looked right for Einstein didn't describe correctly the reality (EPR paradox).
Now I admit that we still don't have a theory which merge QM and GR so the final answer is still unknown, still Einstein would have been very surprised by the results of Alan Aspect's experiments..

Yeah, OT I know, still interesting in the elegance vs usefulness 'debate'.

To the extend that we can, largely, just devise our own 'easthetically' pleasing theories and whatnot where the only limiting laws I can think of is computability and the human factor. If anything, I would assume art, or probably easthethics, would play a greater role in CS than in any other science (with the exception of math).

What is the role of 'art' in a thesis describing and justifying a language? If a thesis were instead about the process of implementing a language, it would be wiser to choose an existing language or model... i.e. such as Orc, CSP, Pi-calculus. So I'm assuming that the thesis mentioned in the OP is about the new language.

Using a formal model is to support 'describing and justifying' a language. It allows you to say justifiable things about the language, and to say why they are justified. It would help you write a thesis.

You misrepresent those methods as heavyweight by mention of Boeing and NASA. It is true that you can use these formal methods in heavyweight manners. But there is a reason I suggested Maude for rapid prototyping.

Modeling a language in Maude is no more difficult than implementing the language in any other general purpose, concurrent, Object-oriented programming language. The total LOC is likely the same or less. You get much more bang for your line of code: you can ask abstract, meta-level questions; you can describe classes of programs rather than ask questions about individual programs; you can readily generate many variations on the language, validate optimizations, verify consistency properties; and so on. There are downsides (resulting time/space performance of the implementation, integration with FFI, licensing) but those aren't likely to be relevant for the thesis work.

Anyhow, I'll agree that most languages start without a formal model in mind (much less on paper). I don't believe that's a good thing. One of the first questions I ask upon seeing any new language is why might I use this language? Too often, I'm answered with excitable hand-waving and unjustified lists of effectively unverifiable (or even unfalsifiable) features. That makes for a blog, not an honors thesis.

One of the first questions I ask upon seeing any new language is why might I use this language?

Modeling a language get's you absolutely nowhere in answering this question! I'm sick and tired of PL papers that present a novel language idea and then justify it with...a formal model...huh? This is a logical fallacy of the shaggy dog variety. I know why they do it, because they agree with you, they know that they cannot rigidly evaluate their idea's usefulness. Accordingly, they create a diversion by filling their eval section with either formalisms or performance benchmarks so the program committee will be able to check that off on their criteria list. But these evaluation sections are completely pointless unless their is some sort of formal or performance question that is key to the paper.

If you go with a formal model, make sure the formal model is relevant and not just window dressing for your honor's thesis. If you are instead trying to evaluate a new language idea, you'll have to be much fuzzier: prototype it and use it in a lot of examples, maybe even a larger case study. You can try to be quantitative by measuring and comparing complexity, but this won't be very robust, this is the route I went in my dissertation (summarized in ECOOP '06). But ultimately, only usage will justify a design, either it continues to work or fizzles out because better solutions exist.

Modeling a language get's you absolutely nowhere in answering this question! I'm sick and tired of PL papers that present a novel language idea and then justify it with...a formal model...huh?

Of course a formal model doesn't validate the language by itself. But it does allow you to verify the statements you make about the language.

evaluation sections are completely pointless unless their is some sort of formal or performance question that is key to the paper

The critical part that your missing: developing a new language is also nigh completely pointless unless there is something useful and interesting you can say about it. (Excepting didactic development so you can learn models.)

If you are instead trying to evaluate a new language idea, you'll have to be much fuzzier: prototype it and use it in a lot of examples, maybe even a larger case study. You can try to be quantitative by measuring and comparing complexity, but this won't be very robust, this is the route I went in my dissertation (summarized in ECOOP '06). But ultimately, only usage will justify a design, either it continues to work or fizzles out because better solutions exist.

I disagree. Usage doesn't justify a design. Valid technical design seems to be a relatively minor factor in market success.

The critical part that your missing: developing a new language is also nigh completely pointless unless there is something useful and interesting you can say about it. (Excepting didactic modeling so you can learn models.)

I disagree completely: we design new languages to solve problems, not to say interesting and useful things about them. The only metric of a language's success is its efficiency in solving the intended problem, and that's it. If there are interesting problems in the language or class of languages, then that's an orthogonal topic of separate value. It may be that interesting techniques are needed to make the language successful in solving problems, and indeed much language research revolves around these interesting techniques. However, if your goal is to design a language, you shouldn't go looking for extra unnecessary complexity.

we design new languages to solve problems, not to say interesting and useful things about them

My words were: "unless there is something useful and interesting you can [justifiably] say". You misinterpret them as "we develop languages so that we may talk about why we personally find them interesting", which has a vastly different meaning.

Anyhow, you go on to say that: a language was designed to solve a problem. Presumably this was not a problem that could readily be solved by other languages, nor even by library. Thus, if your language solves that problem more readily or with nicer properties than would more established languages, that is something useful and interesting you can say about your language.

But you can't justifiably claim that until you fairly measure against your competition (or, instead, leverage a model and logic to argue the point). Justification seems often to be a missing step, not least of all because it is difficult to 'fairly' measure anything of size if you need to implement it twice: the second implementation has a vast lessons-learned advantage over the first.

The reason somebody would use a language is that that person can use the language to solve a given problem more quickly/reliably/maintain-ably. To formally answer this question you'd have to formally model a human being. This is currently completely impossible, and probably will stay that way while the human race needs programming languages.

Yet in many programming language/type system papers there will be lots of formal proofs without connecting these results to why these properties make the language useful. An experiment where you give a couple of people are a problem to solve with your language and you study the outcomes will be a much better predictor of the usefulness of your language than proving any sort of formal properties ever will.

To give a simple and exaggerated example take the language Java which is reasonably useful and now change it so that newlines are not allowed in programs. Every program has to be on 1 line. This is a trivial syntactic change and you lose none of your formal proofs yet it makes the language completely useless.

So why is this field saturated with formality and user studies pretty much nonexistent?

Only an academic part of language development is 'saturated with formality', those other languages such as D,Ruby,Python,etc are built more from users discussions/feedback than from formal proof!

Issues such as usability, utility and ergonomics are aspects of quality, ie fitness for purpose.

That implies that there is a stated purpose for a language, but I'm not aware of any way to formally state many of the sorts of requirements listed here. What is the target problem domain, programmer audience, etc.

Without such a definition how would you compare Coq to Basic? Both have utility in some situations, but without knowing the target is is fruitless to try to compare them.

Appropriately choosing and defining your target could be a method of limiting your project scope appropriately whilst still being able to say something useful and possibly partly quantitative.

You can only measure against requirements

That is not true at all. Even if there is no stated requirement for performance (efficiency, utilization, latency), I can measure performance. Similarly, I can measure variability of performance (and perhaps utilize that as a rough measure of its predictability).

Conversely, having a stated requirement doesn't mean you can measure it. I still don't know what 'ergonomics' means (with respect to PL).

Having 'requirements' means having a model of the world and a system's role within it. One of the biggest challenges in software engineering is getting the model and, hence, the requirements correct.(Validation: Do we state the right requirements? Verification: Did we meet the stated requirements?)

It would be correct to say we can only verify a system against requirements.

Yes you can always measure "something" about performance, but without knowing what the language is required to perform you don't know what to measure, for example comparing matrix manipulations is useless when I want a text manipulation language. In general comparing irrelevant benchmarks is irrelevant. And anyway how do you attribute the result to the implementation or the inherent features of the language? Comparing an interpretor and a compiler is likely to be much more about implementation than language features.

To slightly modify your conclusion, you can only compare two languages against requirements. My point is that there is no way of saying that a PL is "better" than another because it has/does not have certain features without knowing what the PLs were meant to achieve.

As for PL 'ergonomics' I agree. I am not aware of any formal method of specifying such requirements, let alone measuring them.

Whilst PL "science" has many formalized areas I submit that its on shaky foundations without any means of formally specifying and measuring just what a PL is meant to achieve. And that includes such currently hand wavy topics as "ease of use".

As you say getting the requirements right is one of the biggest challenges, but I haven't seen much work that applies formal requirements processes to PL requirements (happy to be pointed to anything I might have missed).

Just because ergonomics and ease of use are hard to measure doesn't mean they are not important and not key in PL design. These are not hand wavy topics, they are just not understood from a formal point. Ignore them at your own peril.

As I understand it, most attempts to achieve 'ease-of-use' and 'ergonomics' involve guidance by reasoned principles... a sort of Feng Shui for HCI (which holistically includes PL). Ka-Ping Yee's Secure Interaction Design has guided my work on language design.

I don't mind resorting to principled design. But it remains difficult to claim success - that your principles actually achieved the desired usability characteristics.

Anyhow, can you point to useful and well-tested models of PL 'ergonomics' and ease-of-use that would support a claim that these aren't hand-wavy topics?

To get 'hard' numbers out of a designed/implemented system, user/test groups which rate the system can give you numbers and all that.

So, the requirements are not hard at all for ergonomics. 'A group of one hundred volunteers will rate the system with at least a 9 on a 1 to 10 scale on usability given that they follow use-case X-Z'.

It may not be hard math, but you can quantify it.

This assumes that the "goodness" of a tool corresponds to it's appeal to novices. It's an utter fallacy. Programming languages are tools that require significant investment of time to learn to use well. To a large degree, their value is measured in terms of how valuable they are to the people who know them best.

This is exactly how we end up with 10,000 poor copies of Algol.

This assumes that the "goodness" of a tool corresponds to it's appeal to novices. It's an utter fallacy.

Indeed, the most highly used tools in every industry are steeped in esoterica confusing to novices. For example, ProTools for audio/video composition, Blender for rendering and animation, AutoCAD for design, etc. Each of these tools rely on concepts that are likely utterly alien to most users, but must be learned in order to effectively express our ideas, and experts in these systems are highly productive.

I think this applies equally to programming languages, and demonstrates the point that has been made in this thread a few times: fitness of a tool for solving problems is the only metric that matters.

I've never appreciated the toolbox approach to PLs (this language for that problem). In my experience, a problem solved well by a dedicated language is rarely the only problem needing a solution in the software product. To make different languages work together is often painful - especially if they make different assumptions regarding concurrency, confinement, progress, delay, memory management, atomicity, persistence. [I suppose I should add: exceptions, asynchronous signaling, etc.]

A PL or computation model's ability to solve one class of problems might be interesting and useful, but does not necessarily make that language fit or practical. For a general purpose PL, what matters is achieving a solution for a large variety of coupled problems all at once. And that doesn't seem something for which metrics exist.

I do appreciate a well-made DSL, but I've also seen too many DSLs creep towards general-purpose to cover the corner cases. I believe we should measure general-purpose PLs by their ability to efficiently express effective solutions to wide varieties of problems by library, and ability to compose these solutions into cohesive gestalt.

To make different languages work together is often painful - especially if they make different assumptions regarding concurrency, confinement, progress, delay, memory management, atomicity, persistence.

Isn't the lesson of OSes and the internet that using different languages is quite practical, if you hide their differences behind interfaces (e.g. filesystems, databases, protocols, shared memory, message queues)?

Regarding "global" features, such as cross-language/runtime atomicity and persistence for example, I think you'd first have to show that these are actually desirable - most people are doing quite fine without them, and in fact increasingly moving even further away from them (for example, VoltDB, by database R&D veterans is talking about "extending its shared-nothing foundation to the per-core level".)

I find your insistence on "global" solutions very interesting and thought-provoking, but it seems to me that we're actually moving in the opposite direction. Is it painful? Maybe. Is it practical? Yes.

Isn't the lesson of OSes and the internet that using different languages is quite practical, if you hide their differences behind interfaces (e.g. filesystems, databases, protocols, shared memory, message queues)?

No. Study of Operating Systems and the Internet taught me those lessons I just stated: that DSLs (such as HTML and SQL) tend to get corrupted into mediocre GPPLs pretty darn quickly (e.g. by adding DOM and JavaScript and XMLHttpRequest and <object>, or Stored Procedures for SQL), that it takes ad-hoc standards and layers upon layers of ad-hoc complexity to make different systems work together. VMs and browsers are basically reinventions of operating systems. Ad-hoc XML languages are often counter-productive attempts to fight this tower of babel. (How often have you seen an XML language slowly add 'conditionals' and basic general-purpose scripting structures? I've seen it happen three independent times.)

The end result becomes calcified by inertia, inefficient due to layers and translations, complex, organic, difficult to garbage collect, difficult to reason about, difficult to maintain or upgrade, difficult to secure.

And this happens not just for the Internet as a whole, but also for large projects of any sort. I don't think this a problem inherent to large projects, but rather to the languages we use to describe and compose them. It is when we discover our GPPL is inadequate for a particular aspect of the solution (such as distribution, concurrency, security, runtime code or data tweaks (i.e. live programming and debugging), AI, reactive rules (observers on complex expressions), etc.) that our projects tend to splinter into greater mess of languages.

Regarding "global" features, such as cross-language/runtime atomicity and persistence for example, I think you'd first have to show that these are actually desirable

I claimed above only that it is more painful than usual to combine languages that make different assumptions about such properties. I don't believe this claim to be especially controversial. I never said that 'cross-language atomicity and persistence' is desirable. (I'd prefer to avoid cross-language anything within a project, perhaps excepting a straightforward interpretation of a DSL or monad by object or module.)

As to whether atomicity and persistence and other properties (confinement, progress, safety, concurrency, indeterminism, distribution, robust error handling with simple failure modes, etc.) within a language are desirable properties, I'll save that argument for a forum discussing languages that pursue those properties.

I will note that your reference to VoltDB with its atomicity, concurrency, consistency, persistence, and distribution... hardly seems a valid example that people are "doing quite fine" without global features, much less "moving even further away from them".

[sorry for continuing this OT thread...]

The examples you present for DSLs turning into GPPLs are persuasive at first (HTML->general UI platform, SQL->SQL with stored procs, XML format->PL).

But are they really representative? To be specific [and extremely subjective!], HTML as a GUI platform seems to have found its mainstream pinnacle with GMail, and many people seem to have given up on this particular (ab)use of HTML, and moved to (iPhone, Android, ...) "apps". Complex SQL seems to be increasingly disused in favor of dead-simple, reduced usage of the RDB as nothing but a primitive storage + indexing solution (How FriendFeed uses MySQL to store schema-less data). XML has been on its last legs for years, with many people looking back in ridicule at the excesses of the past.

All of this is from my limited free-software-web-developer perspective, but at least in this area of computing, we seem to be moving away from shoehorning general-purpose paradigms into DSLs.

So, right now, I'm using a plain HTML textarea in my C++-written browser to write this, which gets transmitted via the HTTP interface (which, with ReST, has usable theory behind it), to be processed by a PHP script, which will spit out a plain HTML page in response to a simple SQL query. [Edit: I want to stress that neither the HTML nor, I suppose, the SQL in this particular application are stretched beyond their intended uses as hypertext markup and query language, respectively.]

All in all I'd say, yay for multiple languages connected by standardized interfaces! ;)

This assumes that the "goodness" of a tool corresponds to it's appeal to novices. It's an utter fallacy. Programming languages are tools that require significant investment of time to learn to use well. To a large degree, their value is measured in terms of how valuable they are to the people who know them best.

This is exactly how we end up with 10,000 poor copies of Algol.

Measurements are valuable and meaningful even without the requirement. The reason? Because our requirements are, themselves, rarely valid. Requirements are based on sketchy models of the world and sketchy models of our system's role within it. These models must evolve, and for them to evolve we need... observations, and measurements. QED.

comparing matrix manipulations is useless when I want a text manipulation language

A vector is a 1D matrix. Complex pattern-matching, pattern-mining, and clustering techniques often involve matrices, even when applied upon text. You'd be surprised what those Matlab fellows do with matrices in support of text manipulation (among everything else).

How do you attribute the result to the implementation or the inherent features of the language?

A language only offers a glass-ceiling for performance, i.e. by affecting which sorts of static and runtime analyses and optimization and parallelization are feasible or valid. But there is no limit to how slow you can make a language implementation. (Give me any language implementation, and I can make it slower. :-)

So you can only attribute performance to the language to the extent that the language leverages optimization or parallelization that would be infeasible in another language. (And that's the sort of thing you can prove readily via a formal model.)

But I was using performance only as an example of something that can be readily measured without a requirement, in contradiction to your assertion. Indeed, if there is truth to the notion that "we optimize what we measure", then there's a reason performance gets a lot more attention than other language and implementation properties.

there is no way of saying that a PL is "better" than another because it has/does not have certain features without knowing what the PLs were meant to achieve

The goal is often 'general purpose programming' unless otherwise stated. A good example of software that must cover almost everything is development of a massive-multiplayer video-game. This covers: persistence, data management, HCI, multimedia, portability, concurrency, distribution, planning and AI, plan refinement, rules and physics, continuous system maintenance and upgrade, security concerns, robustness and reliability concerns, performance and performance-predictability concerns, etc.. About the only things missing are embedded programming and mutating FPGAs. So, that would a very effective scenario for testing languages (and computation models) that claim or aim to be 'general purpose'. ;-)

There are other systems with similar properties, such as military command and control.

These are scenarios I keep in mind, and have led me towards a new model for general purpose programming.

You state a very different criterion than I was told when I was pursuing my degree. A senior honor's thesis was a dedicated 3 hour course for a semester. "Interesting and fun" was something to seek, of course, but that came from having interest in the material.

Something more was needed for a decent grade.

I would agree with the earlier notes that CLR with DLR is probably your best choice, at least from among the three you mention. You've stated dynamic typing, but JVM will give you headaches there. BEAM lacks the reverse engineering tools to help you debug compilation errors.

CLR would also give you tail-recursion, which would be useful if you pursue event-loop concurrency (single-threaded concurrency). CLR and JVM both offer atomic and lockless operations if you need them, but parallelism is a challenge that you might not want to deal with in your limited time. (Single-threaded event-loop concurrency would also be deterministic up to ordering of inputs from outside the VM, which would greatly simplify testing.)

A few related LtU discussions:

• Erlang green threads and the CLR
• Actors that Unify Threads and Events
• Event-Based Programming without Inversion of Control
• A Language Based Approach to Unifying Events and Threads
• Writing an interpreter, targeting a VM or writing from scratch?
• VM's... What's the best?

A few VMs named in the latter discussions, but not until now mentioned here, include NekoVM and Parrot.

You might consider targeting LLVM or C-- if you're interested in native code compilation.

I am going to try to avoid expressing value judgments on these things, but some sentiments may leak through.

You'll probably want to figure out whether you want to have continuations in your language, and, if so, how powerful they should be. Marshall's technique is the most widely used I know of, but it is not call/cc. And on the JVM there is RIFE/Continuations and Swarm for insight and inspiration. Da Vinci Machine Project has a continuations sub-project I've never looked at. For .NET, no language has actually tried to implement continuations with the DLR that I am aware of, so performance could suck. For example, IronRuby in theory should have support for call/cc since Ruby has a continuations library, but the language team has explicitly chosen to stay away from it since none of the core test suites and gems people tend to use actually leverage the continuations library. And Jim Hugunin has basically said he doesn't think the DLR needs to support continuations, which probably pisses off a ton of PL researchers w/ a practical software engineering background.

Under the Mono implementation of the CLR, there are intrinsics for coroutines which greatly simplify some implementation. When Rodrigo mentions simplicity of .NET, he is probably partially biased towards the Mono project. The disadvantage with choosing Mono is that it has yet developed a precise GC that can handle the wide scenarios of MS's, so you could end up with memory leaks in your implementation if you don't understand what these corner cases are. The upside is that the Mono mailing list is friendly and helpful and the source code is open.

For .NET, you should look at the Phoenix and Rotor homepage at MS Research. It has a lot of materials about learning experiences people had prior to the introduction of the DLR. To understand the design motivations behind the DLR, you should read Jim Hugunin's blog, starting in early to mid 2007. That way you'll see how the DLR evolved into the way it is and give you some insights on how you might add primtitive flow control constructs to your language.

There are still some thorny issues with dealing with CIL (Common Intermediate Language), such as ensuring module loading is run before you try opening the module and using it. This comes up in the Compact Framework and Silverlight moreso than mainstream .NET VM profile. In Silverlight, the design for loading assemblies remotely is a bit weird and you have to use obscure encoding tricks to get it to load correctly without pissing off the assembly language type system.

Support for concurrency is pretty good on both platforms, JVM and .NET, especially when you look at .NET 4, but the JVM has way better support for software transactional memory via multiversion concurrency control (Clojure, Multiverse, Akka), as far as I'm aware.

I really don't know anything about BEAM internals, just Erlang. Sorry.

I would also make sure whatever environment you use that you are comfortable with the profilers you can afford or acquire via academic project donation.

I would also use an automated testing tool like Pex (.NET) or some equivalent in Java. If you use Pex, and use Visual Studio, you may want to have a look at the AutoPex plug-in, which executes Pex tests in the background.

[edit: I may add more hyperlinks to the vague references I provide, later on, but for now that is it, brain dumped.]

Clojure has been ported to CLR. I suspect that means bringing STM along with it. (I'm not sure of its maintenance status, though.)

Of course I'm biased. Much more for my deep familiarity with mono/CIL than for an agenda of pushing its usage. Oh, mono's new GC, that's coming out soon (couple of weeks) alleviates the retention issue a lot.

BEAM is really a bad choice as documentation for the format is non existant. All you can find are docs for R6 which is very old and has a lot of deprecated opcodes.

I have no doubt that once Java 7 is out a DLR like project will emerge and put the JVM on similar ground to the CLR. But until then, one has to deal with JVM bytecode.

One advantage of the JVM is that is has a much simpler typesystem, so doing interop to java libraries is much simples. Same apply to writting a CPS transformation.

OTOH, the richer CLR typesystem enables a lot of interesting things. So in the end it's a tradeoff one has to agree with.

You don't really need to have experience with VM's these days to do a language, its just a matter of translating your language into another VM format: don't worry about GC or low-level details unless you want to make that a part of your project.

In my experience, the first step to building a language is to prototype it in an existing language as a library/embedded DSL. It won't look pretty at all, but you can iron out lots of problems early this way. .NET 4.0 supports dynamics, it might be useful for this reason. Any dynamic language would also work if you already have a favorite. After you are comfortable with your prototype, move to a custom parser with either an interpreter or compiler on the backend.

You can always do a simple interpreter. I've implemented a couple of OO concurrent languages with interpreters on the JVM and haven't been disappointed. Depending on how far you want to go, Scheme/Clojure/Racket might be better for writing an interpreters and you could also play around with decent macros.

If you want to compile, then a good bet is the CLR/DLR: compiling is painless and just one step more complicated than doing an interpreter in that you have to manipulate expression trees.

my goal is to have something that can be compared to mainstream languages/libraries. Hence my desire to compile to an existing VM.

Using an existing VM may make your effort comparable to other languages that run on top of VMs, but why restrict yourself to those? A huge part of any software stack today does not run in a VM. VM-based languages are a strict subset of all languages.

concurrency and object orientation are the main pillars of my language

Using a VM will limit you to that VM's concurrency and OO facilities, as opposed to C or assembler, which give you much more freedom in both areas. I think that by not restricting yourself to existing VMs, you'll gain a much broader basis for comparing your specific approaches to concurrency and object orientation to existing languages.

VM is a simplification: its easier to build a language on top of a VM then on top of C, assembler, or LLVM. The run-time is easier, you don't have to worry about the underlying platform, you get access to a lot of pre-existing libraries without worrying about linking formats.

A huge part of any software stack does not run in a VM, but the diversity in that area is decreasing (programs that are run a lot vs. the long tail of programs that are not run often but are very diverse). There are definitely projects where going old-school/low level is useful, but you'd really have to be at the point where rolling your own v-tables or doing your own automatic memory management is relevant to your project (we can role our own v-tables on the JVM via an interpreter, of course).

So many diverse opinions on this topic. Its like "Ask Slashdot"

AFAIU, the main problem with using more than "dozens" of (NPTL) pthreads is the fixed stack size. Fortunately, there appears to be a project underway, by gold and gccgo's Ian Taylor, to allow split stacks, i.e. the expanding and contracting of individual thread stacks.

Other than Erlang, Haskell is also a good competitor for large amounts of concurrency (ten million threads? no problem).

Please avoid conflating concurrency and parallelism.

My writing is shorter using imprecise terminology. Informally, we often conflate concurrency and parallelism because nearly the same race conditions must be handled. In some contexts we have multiple processors, and elsewhere we have async activities scheduled serially in single-threaded contexts which can overlap when scheduled in ways looking just like concurrency in the sense critical sections interfere with one another. So parallel is shorthand for "must handle race conditions" when conflict can occur. When talking about specific hardware concurrency contexts, we normally talk about CMP or SMP. (Getting read/write ordering right in shared memory is hard.)

Conflating concurrency with parallelism is inaccurate, not merely imprecise. Reducing precision to avoid verbosity is acceptable, so long as you're accurate. (Reducing precision tends to increase accuracy.)

But I'm detecting confusion about the terminology rather than a conscious decision for imprecision.

Parallelism describes many things physically happening at the same instant. Concurrency describes many things logically happening at the same instant. We might conflate them because concurrency offers obvious opportunity for parallelism. Whether we can leverage that opportunity efficiently depends upon our hardware and the ratio of work to synchronization and parallelization overhead.

But there exist many opportunities for parallelism that do not depend upon concurrency (and that do not introduce race-conditions). According to the Coconut video, placing a task on each of 8 cores of the Cell processor would account for only 2% of the processor's potential parallelism. To get the rest, said video claims you need 4x SIMD, 6x Loop Unrolling, and double-buffering. Modern GPUs perform heavy data-parallelism, achieving simultaneous computations for hundreds of pixels at a given physical instant - assuming the software utilizes this potential.

Regarding the 'race conditions', it is difficult to talk about 'race conditions' as logical entities. Which concurrency-related indeterminacies qualify as race-conditions depends subjectively upon one's expectations.

There are several forms of deterministic concurrency (Kahn process networks, restricted forms of promise pipelining and dataflow programming, restricted concurrent constraint programming) that cannot exhibit race-conditions. Deterministic concurrency is logical because you must understand many things happening at once to reason about progress (or other critical properties) of the computation - even if implemented without parallelism. I'll grant that many interesting and popular forms of concurrency are indeterministic (necessary if concurrent elements are to perform IO with other systems) and thus may exhibit race-conditions.

dmbarbour: Conflating concurrency with parallelism is inaccurate, not merely imprecise.

If I knew you better I'd tease you about having an Aristotelian bent with a focus on prescription over description. But humor seldom works well in writing. So let me just note I described how we actually talk at work: it's how we talk in practice. Since all of us have been out of college for twenty or more years, our terminology might not match what folks say in academic contexts.

Usually we speak about logically happening at the same time, with physically as a special case indicated by throwing in remarks about SMP-safe or CMP-safe when that matters. So unfortunately we say parallel to mean logically overlapping; sorry, that's just how we talk. We treat serial but interleaved as just as big a problem in a logical sense as physical conflicts, just a little easier to manage.

In short, we treat async, parallel, and concurrent as synonyms emphasizing different issues, depending on what we're worried about. The phrase "race condition" is usually what we're worried about, and that's also what we say, sometimes shortened to race; it never seems confusing. It characterizes a form of misbehavior where interleaved but logically separate activities manage to interfere with one another in a way ruining an intended outcome.

I appreciate your comments as educational, but I need to move on now.

I did consider adding an emoticon at the end of the sentence you quoted. If you want the short version: parallelism about performance, concurrency about communication. That's what I tell people who get lost in a precise definition. ;-)

I'll grant that I've been studying every darn form of concurrency I could find for eight years now (aiming to discover a better way). That makes me sensitive to the distinctions. Also, the communities that discuss technologies and methods for parallelism overlap only slightly with those that discuss methods and models for concurrency. You've got the high-performance computing gurus talking about scientific computing, collision-detection, pixel shaders, and GPGPUs on one side. On the other you have PL guys discussing rendezvous, orchestration, synchronization, workflows, branching time, reactive systems, message-passing, shared memory, locks, semaphores, STM, and various ramifications of these approaches on disruption tolerance, graceful degradation, garbage collection, and security. The college courses are, similarly, quite distinct.

This bifurcation within the communities is among the reasons I recommend getting the terminology right. It isn't that hard: Parallelism about performance. Concurrency about communication.

And of course 'race-condition' will rarely be confusing in context. That context includes your 'intended outcome' that allows you to categorize an indeterministic behavior as a 'misbehavior'. ;-)

In effect I've hand-rolled something like a concurrent green thread library for thousands of parallel network processing tasks. But it's all in naked C and it's slightly mind-bending. [...] If you go that route, you need a scheduler, trampoline, continuations, and a way to provide mutex for activities that would otherwise conflict with one another.

Sounds like Protothreads is what you're after. It provides much of this framework for you. The "threads" are really a generator-like abstraction in C exploiting Duff's device.

I was just trying to offer useful advice, but that might be something I can use if I do something similar again. I'm too many months invested in my current scheme. Protothreads might work if it can be non-invasive to the extent a host environment cannot detect presence of Protothreads in a single-threaded environment with soft real-time requirements. Thanks for the pointer.

Just about the nicest thing I've used from Java. Don't have much more to say...