Lambda the Ultimate

I read through this, but I'm not sure what I was supposed to learn. Perhaps these essays are just giving us a historical perspective?

Mathematica is a symbolic language that focuses on term rewriting. So its quite different from a language where expressions are instead evaluated. I'm still not sure what epiphany they've had in making this a general purpose paradigm (via the Wolfram language).

The author didn't go near APL, which is kind of strange. It was the first interactive calculation language, I think. They even had it running on type writers...interactively.

You can do symbolic math with something like SymPy but this is still far from ideal. I think another critical feature of Mathematica is interactive graphs. A lot of progress in IPython land on this in recent months.

Mathematica is a symbolic language that focuses on term rewriting. So its quite different from a language where expressions are instead evaluated.

Clarifying a bit: Mathematica's primary rewriting strategy is essentially classic "normal order" evaluation with two twists: 1) Evaluation iterates until terms stop changing and 2) There are various operators reminiscent of "quote" that prevent some or all forms of evaluations from proceeding, and interpret their arguments "symbolically".

The best overviews are here:
The Standard Evaluation Procedure
and here: Non-standard Evaluation

In practice, interactive programming is no different than any functional language, other than increased usage of explicit substitution and rule application. See Applying Transformation Rules

I'm still not sure what epiphany they've had in making this a general purpose paradigm (via the Wolfram language).

Absolutely no epiphany. They've always known that it's somewhere between workable and pretty damn good for programming, but the name "Mathematica" conveys a very limited purpose. Make no mistake, the name "The Wolfram Language" is simply a rebranding; one that appears to be quite successful in expanding the scope of those paying attention to it.

I highly recommend learning Mathematica to in-the-trenches programmers and PLT folks alike. It's heads and shoulders above any other live programming environment out there, but it's far from perfect. There's quite a bit of two-way learning that can happen.

That is not the author's assertion, but Ousterhout's. And even then it is a bit biased coming from the designer of TCL.

I had the same reaction. There are several interesting topics hinted at, but they are mostly dropped very quickly.

I did not find the discussion of the dawn of computing very helpful. In an article about programming languages, let's just assume that we plan to have an interpreter or compiler, and go from there.

It's a good point about symbolic versus numeric computing. My impression is that numerical systems are much more widely applicable than symbolic ones. If you are working outside of pure mathematics, then you don't need a tidy closed form for whatever you are working on; you need a way to compute numeric data that are of interest for some application.

There's a rich design question hidden in the simplification down to the "Ousterhout Dichotomy". How do you fit high-performance subroutines into a general-purpose language? Talking about "C speed" is very misleading here: much of the reason to embed a high-performance sublayer is that if you write something in C, much less a higher-level language, it might not make good use of special hardware like vector processing. Also, saying that Lisp abandons the Ousterhout Dichotomy seems misleading, because Lisp is a language designed for making new languages. The way I usually think about Lisp, it feels like Tcl+C, except instead of two layers you have 10-20 layers.

It's somewhat sad to see no mention of writing *correct* scientific code. Units checking can prevent a lot of errors. As well, using an existing numeric pack, rather than open coding a new computation, has the advantage of leading to better general data-handling practices, whenever there's an existing routine to do the sort of computation you are thinking about.

All in all, many interesting topics being broached in these articles. I entirely agree that scientific computing is an interesting and highly relevant area that can be somewhat opaque given the symbolic backgrounds PL geeks often come from.

. If you are working outside of pure mathematics, then you don't need a tidy closed form for whatever you are working on

That's over-stating it. You may want to derive analytic formulas (I know I do), not just produce numeric results. But I'd agree that this might be mostly of interest to people doing scientific theory, not applied numerical computing (e.g., data science).

I think a basic distinction should be made between exploratory programming of various kinds, and justifying final results (e.g., deriving, proving). Most blog posts etc. on scientific computing focus on the former, even when reproducability and so on is being championed. So you point on correctness is well taken.

How do you fit high-performance subroutines into a general-purpose language?

This is a good point. I think one of the attractions of Python is that its "change everything from anywhere" approach enables really nicely done libraries. They are of course hacks, but when done right these libraries combine very nicely with the language. Think Pandas dataframes, NumPy arrays. This also allows such libraries to interface to foreign language routines that can do the heavy lifting and provide great speed advantages.

I don't see how "change everything from anywhere" leads to interfacing foreign languages more smoothly.

Suppose you want library code to be able to change the semantics of (some) assignment statements...

Is the "change" part necessary? Or just the ability to participate in the dispatch of virtually all core operations? Julia, for instance, is almost entirely built on generic multi-method dispatch (docs here). In some sense, you're mutating the function dispatch tables, but it's mutation in a much narrower sense than altering the metaclasses for the top type. Seems like it offers the same (or better) extensibility than the Python/Ruby/etc models.

Good point, and I don't really know the answer. Maybe someone else does.

Respectfully, you appear to have read neither article fully. While Hoare does appreciate the facilities of IPython et al, he does not advocate the split between scripting and speedy languages.

You also appear to have an outdated understanding of the state of programming language implementation. It is perfectly possible to produce fast language implementations with a squirrely mutable runtime: see Java (with dynamic classloading), LuaJIT, and all of the JavaScript implementations, not to mention the more historical example of Self and the example (from the second article) of Julia. They all use adaptive optimization. Python's dynamicity is not dissimilar to Lua's.

Well, to be fair, these relatively fast implementations quickly become ludicrous complexity monsters, with 100+ man years easily sunken e.g. into a competitive JavaScript VM -- 80 of which could probably be avoided by saner language design and less ridiculous amounts of mutability. And they are still nowhere close to native performance, let alone its predictability.

as an end-user, it also freaks me out that i'm always depending on ever-growing inscrutable code bases. i have *no freaking idea* what 99.9% of hotspot is doing, i assume. so "a sufficiently smart {compiler,runtime,whatever}" is very much a double-edged banana.

HotSpot is pretty complicated, to be sure, but the lion's share of that is in the server compiler and the GCs, IMO. V8's complexity is more heavily centered in the runtime and the complexity of the object model, ICs, and compile-only architecture. JavaScript is just flatout an insane tangle of complexity of interactions--mutability and indiretion requiring speculation and backout and shortcuts on a level undreamed of in the Java world.

Who you calling a ludicrous complexity monster? :)

Actually in recent times I've started to think that V8's complexity is at least 75% incidental. I'd say why but I don't want to be shot.

If you mean driven by incidental picks of random benchmarks that bear no relation to reality but are what everybody is gaming, then yes, I agree. Although I still think your 75% number is far too high. Maybe half of that.

You're back in Europe now, nobody will be shooting. :)

Lancet has gotten me excited about the possibility that you're right: contemporary JIT implementations are mostly incidental complexity. I'd love to see more work in the direction of Surgical Precision JITs.

Well, this is nice in that it gives the user a way to control particular optimisations. But by itself, it doesn't really simplify much if the goal still is to be fast on common code without requiring user intervention. (In fact, the heavy reliance on partial evaluation, and the meta-circularity aspect that goes with it, can easily make a runtime even more complicated, though perhaps less ad-hoc.)

While it is possible to produce fast implementations of languages with a "squirrely mutable runtime", it is not easy. It's been done for Javascript because it was worth the trouble to devote huge resources to the problem. (Javascript also lacks shared data concurrency, which eliminates many of the painful cases.) Julia's model of concurrency is remote procedure call, not shared data between threads. Much simpler from a locking standpoint.

Google's project to speed up Python, "Unladen Swallow", was a total failure. The PyPy project has struggled for a decade, is five years behind schedule, and still isn't either comprehensive enough for widespread use or fast enough to eliminate the need to do array math in C.

It's not fundamentally impossible, but it's demonstratably very difficult.

It would be quite easy to enforce a non-GC subset of the language inside low-level components, preferably using the type system. The key is the language must be able to express concepts like memory layout, allocation and deallocation, so that you can use them where GC is not allowed. Conversely you might want to restrict those features to inside low-level I components. Decisions as to the precise nature of restrictions can be deferred providing both the language and type system are capable of expressing the required high and low level concepts in the same language.

The issue isn't really technical but human/organizational if you have a language which allow the use of a GC, developers will use it: so in the end you have a standard library which use the GC which makes not using the GC unpleasant..
As writing a standard library takes many man year and is a very 'distributed' effort, it's difficult to force developers to not use the GC..
Sure tools and type systems helps discover code which use the GC where it shouldn't, but the main issue is that not using the GC is important only for real-time development which is a minority of case, so it's difficult to require everyone to take care of issues which doesn't affect them directly..

To address the points, if developers don't need high performance they are welcome to use the GC, this is not a problem.

A non-GC component would look like a GC component from the outside. So for example lets say I write a high performance matrix library that manages its own low level memory, from the outside I would use it just like a matrix library that uses GC, except it would be much faster. It may be that you ship with a slow high-level matrix library, and replace it with a low-level high performance one later. None of the code using the library need change providing they use the same API definition.

There is no using-GC where you should not. The type system can rigorously enforce it, so if you declare a component is 'low-level' you will not be allowed to use GC inside the component definition.

There are plenty of applications that are not real-time, where performance is an issue. For example compiling a program faster makes a developer more effective. Improving performance of libraries can reduce required hardware costs, or even make some application practical, that was previously too slow to be any use.

Doing that in a safe way, checked by a type system, is known to be feasible but maybe not as easy as you make it seem. To my knowledge ATS is the only language doing that -- Cyclone has used regions to support some, but not all, of the usual C memory ownership patterns. Chlipala's Bedrock verification framework also knows how to deal with allocation, but it's a heavier approach. None of those count as "quite easy".

One way to look at this is that it is similar to the Android NDK, communicating between a garbage collected language (Java) and C/C++. The marshalling is complicated on the C/C++ side because these languages have no awareness of the Java stack and heap formats.

A simple solution would be to start with a common basic syntax (operators, functions, blocks etc... having two just does not make sense), and add garbage collection on one side, and memory primitives on the other (like Java and C++ respectively), except they would share a common object and stack memory layout, and each would be aware of the marshalling requirements for the other, so the programmer would not need to explicitly code this.

This would not even need the type system to control as to separate sub-languages exist, but I would prefer something enforced by the type system.

Maybe Cython/RPython is the closer thing?

I think is necessary that 2 different things look different, so is clear in what universe I'm. If my "low-level" and "high-level" read equally, then Is hard to switch context.

If developing a new language from scratch would you really want to include two distinct syntaxes? Surely the differing use of operators would be confusing? Say one used () for functions and [] for arrays, but these are swapped in the other. Would this really help you write better code?