Lambda the Ultimate

The whole point of the post is the personal journey. If you just want to read the code, it is here: https://github.com/breuleux/earl-grey

Also LtU is not the target audience, so "we've all been there" is not necessarily true.

I get that. For this audience, I would have liked to have seen more code excerpts to go with your exposition...it would have been useful in better understanding your journey. And no, I don't mean let the code talk completely by itself :) A grand language essay benefits from both.

For the intended audience, it might be hard for them to connect with the content anyways.

FYI, that's not the author. :) If Olivier Breuleux had come to us with this blog post to communicate about Earl Grey, I guess a request for more code samples would make sense as part of an active communication process. Right now it's more like unsolicited public criticism of their writing style.

If you expected an essay that concisely advocated for the Earl Grey language, try the Earl Grey website. The website has a sleek logo and plenty of syntax-highlighted code samples right on the front page. It even has an interactive tutorial. This is a well-polished operation, and I think it indicates that the blog post would have had some code samples if it were really meant to introduce the Earl Grey language. The blog post's goal was probably something else.

I liked it pretty well as a report of a personal experience. I relate to a lot of the same challenges, such as the isolating effect where I knowing how to take full advantage of my own languages better than I know how to take full advantage of mainstream languages. Sometimes it's just nice to hear these things made explicit so that I know that I'm not alone. (And where the experiences are a bit different from mine, it's nice to compare notes.)

I think the blog post is probably even more valuable to the people who actively follow the Earl Grey project. It clarifies why Olivier Breuleux will be devoting attention to new language designs, it clarifies that they'll still be maintaining Earl Grey and using it as a platform for new projects, and it explains why the Earl Grey language came to be the way it is, warts and all.

I spent a while thinking about this and now consider it to be a solved problem, thanks to a simple precedence rule: operators surrounded by spaces have lower precedence than operators not surrounded by spaces. This reads really cleanly:

(n * n-1)

(The drawback is obvious from the above example: you can't use dashes in your variable names. I don't want to deemphasize that; I love dashes too. However, there are no other drawbacks if you aren't tied to backwards-compatibility.)

More details.

Precedence is an ugly user unfriendly feature even for those of us with lots of experience in math. The problem can be solved simply by getting rid of it, making it a feature of the input editor that puts in braces automatically. Of course, brace rendering in most program,ing environments sucks, but that can be solved also.

out of operators, then precedence rules might have more uses than just mathematical convention.

Getting rid of braces in the common use cases is worth a lot in terms or readability.

You're probably forgetting that if THE WHOLE syntax is defined as a precedence grammar then define function is an operator, assignment is an operator, conditionals are operator, message sends are operators, not just add and subtract and divide.

So for instance you don't want every assignment to need parens. a = (1 + 2)

If you have to fully parenthesize your program, then you're back to s-expressions.

So if (a la prolog) define clause is an infix operation then you want it to have the lowest possible precedence so that each clause doesn't need delimeters.

Since operators really act in relation to each other's precedence, having an editor that keeps all operators in a list, sorted by precedence would make adding operators much less painful. In fact one doesn't want numbers so much as a position in a list relative to the other operators that are legal to appear in the same context.

But under it, the information is still no different that what they had in prolog, where a define might look like:
:- op( 1200, xfx, [ :-, --> ]).
:- op( 1200, fx, [ :-, ?- ]).
:- op( 1100, xfy, [ ; ]).
:- op( 1000, xfy, [ ’,’ ]).
:- op( 700, xfx, [ =, is, =.., ==, \==,
=:=, =\=, , == ]).
:- op( 500, yfx, [ +, -]).
etc.

If your parents have light weight graphical representations, then you can have them everywhere without adversely effecting readability, so why not? I think a lot of bugs occur because of subtle precedence relationships between even common operators, it is redicullous that we aren't looking for better solutions. More precedence is definitely not the answer. Especially if those precedence relations are even more subtle.

I agree that we should have fewer precedence relations, and that ambiguity of operator order should be an error, but I don't think lightweight parens are the best way to indicate precedence. I think spacing works better. You can also use operator font (size) in the same way that you'd use bigger parens for larger expressions.

totally impossible for them not to adversely affect readability when applied to the WHOLE grammar, not just expressions. And frankly I think they ruin expressions too. I even prefer method calls to avoid unecessary parens like in smalltalk. Instead of object.method(x expression,y expression) in smalltlk it might be "object methodX: xexpression y: yexpression."

But you do need mixfix for such operations as if - then - else

Nonsense. If you are just thinking about plain text, you deserve your 1970s programming experience. Otherwise, syntax is much more than a BNF in code presentation and even code input.

harm readability, dim or not,even if aligned.

Who cares about unadorned text on a 1970s teletype? There are much better solutions to these problems using modern typography.

I guess you want graphical delimiters but not custom text delimiters.

Seems arbitrarily elitist.

Why aren't they equivalent? Or combinable?

In the real world, the most available cross platform programmer's text editting widget is scite in wx.

Custom text delimiters don't work, at least without a good amount of typographical hacking. It isn't arbitrarily elitist to shun solutions that simply don't work.

The real programming world is still living in the 70s and we are suffering the consequences as a result. Type writer text is all we can afford, so let's have programmers interpret a bunch of subtle precedences in their heads since anything else is unreadable! God forbid if we actually change the first premise.

else doesn't work?
repeat until doesn't work?
function endfunction doesn't work?
for by next doesn't work?

I think they work just fine.

I'll admit that C doesn't work well
if () {statement1; statement2;}
vs
if () statement1; /*oops*/statement2;
is a failure.
The need for generic delimeters () and {} is a failure, they don't supply enough context to the eye and make you look back and forth. And if they're optional, changing the meaning by position, my god.

So the problem isn't CUSTOM delimeters, it's bad grammars with generic delimeters. And the worst offender is Lisp

The point is to have a program that your eye can scan without moving back and forth and counting, and without wasting screen real estate.

I think custom delimeters and operations are fine.

Quite the opposite, I think structural program editors fail.
If you have trouble entering in an incomplete structure and finishing it. If you have trouble moving and copying part of a construct... If you have to click around for constructs instead of typing, you have a slow workflow

If you think you have a much superior writing and editing workflow than typing, you're going to have demonstrate that.

Typing is fine. It's the expectation that what is typed is vanilla unembellished text that isn't. I've already made posts in this direction (e.g. highlighting precedence in the editor). Also, making braces/parens typographically less annoying. Still a lot of work needs to be done, and I'm not a professional typographer. But I can totally see the day when we type/edit text as usual, but what is presented on the screen is much more rich and useful.

Colored and sized parens are still worse than symbols that don't need context. Even colored, sized parens aren't self explanitory and make you move your eye back and forth to see what construct they belong to.

"Endif" explains much more without moving your eye.

And in a lot of cases or just the common case, you can get away with no paren. Say calls with one parameter don't need parens, in that case those elided parens don't confuse the eye when you're trying to match parens that ARE necessary.

Making code understandable locally, no need to move your eyes for context, no question of what matches what, no ambiguous symbols is principle.

You can't get around the fact that the grammar is important to that goal, no matter what the editor does.

Again, no typography is better than crappy typography. Color and size aren't good enough, we have to etch the background itself with contrinuous shapes and layouts. We have to create a real graphical syntax for our languages.

We have more to work with than just grammar/syntax. Bitmapped displays are quite mature now.

I imagine you could come up with a very dense display that only children would be flexible enough to learn and appreciate.

Maybe.

Highlighting/animating parts on mouseover would be interesting.

As long as we're pie in the sky I wonder if 3d or real 3d would be useful. I remember that it's useful in visualizing data, if not programs.

Personally, I'm holding off until I understand newer system needs more. There is a lot to improve in standard code presentation, but I've also discovered the usefulness of the *gasp* mouse in manipulating code in a live programming context (e.g. Giving real estate to the unknown with holey terms to enable very quick local refactoring). If we are going to embellish code presentation, then we might as well provide some affordances for fluid non-text based editing (e.g. Scrubbing).

Hololens programming environment ala something like the ironman UI is much farther off. But it is a nice direction to move in.

unless you have links.

I'm interested.

See my latest demos. I've been lazy about aggregating them, but Yow! or even Strangeloop will have some of this.

Grammar is important, and there is grammatical difference between

(if T C A)

and

if T then
C
else
A
endif

but all you're arguing is syntax, which is utterly irrelevant.

speed of understanding, effort for a human to see and compare, obviousness of errors?

What can I say but "wrong"?

There is a whole field of typography that is used to reason about this kind of thing. Typographers are also quite adept at her ally judging readability and such. Computer scientist have uninformed opinions about typography like designers can have uninformed opinions about computer science.

I hope someday we can have more informed debates about this, until then we are probably just devaluing the professionalism of another very important field.

if the syntax is designed so that meaning is as local to a symbol as possible, where ambiguities are reduced by not having multiple contexts that could apply to that symbol - so that in the context of a typical expression the meaning is local - the code can be read faster and more accurately.

There's an analogous point that CODE whose meaning is dependent on OTHER CODE is harder to understand. Well that principle applies ALL THE WAY DOWN IN SCALE like a fractal. If a symbol can be understood locally that's better than if the meaning is dependent on further away symbols.

The meaning of ')' depends on where the matching '(' is.
But the meaning of 'endfunction' does not require matching, typically, one will simply remember where the latest lambda was.

Move your eyes this way or that say is typography, where do you think narrow newspaper and paper columns came from?

Expecting humans to run push down automatas in their heads is a bit masachistic, which is why I think nested structures should be more explicitly chunked visually. And there is always a limitation on how much nesting is viable before heads explode, which is quite independent of locality. Endfunction wins because it doesn't nest easily, or nests vertically rather than horizontally. If you have a 4-level deep endfunction nesting, however, well...it will devolve into the unusability of parens very quickly.

Note that this moves from typography to straight usability. Why it is hard has more to do with it being hard than ugly.

Speed and position on the screen
...
On some current processors they'll slow down as typed text implies a need to restructure the program

And if the processor was infinitely fast, it's a problem that the eye isn't. If code jumps around the screen to accommodate the updating structure.

And (as in some structure editors) you can't just type, you have to choose, well there's an interestingly different workflow. Will it be too slow? Who knows. There are structure editors that don't allow you to make a grammatical error. Is that a loss? Most current programmers would say yes.

I've never built a structured editor before, so I have no idea. Fortunately, parsing and good error recovery is easy if done by hand, and graphical structure can be easily projected around the parse tree.

No idea about old processors. I get that supporting programmers using 8086s on old teletype terminals is still a thing to some PL markets, I personally find it completely uninteresting.

IDEA is too slow for 4 year old atom processors, as is Visual Studio 2010

Jetbrains makes a structure editor for DSLS that doesn't let you make grammatical errors. It's probably too slow for large files on current processors. And WAY too slow to use on atoms.

Hell, Racket's IDE is too slow on Atom processors.

I don't know. On the one hand, computing resources are getting better, but on the other hand, IDEs like eclipse and visual studio continue to unimpress. I haven't found these features to be particularly computationally intensive yet. Simply putting more pixels on the screen is definitely rather cheap, even on 220 PPI displays.

JetBrains MPS is a projectional editor, it's probably doing a lot more than my systems are. I don't see why it should be that slow on large files however, maybe they aren't virtualising enough? Or maybe they are being incremental enough? It's hard to say.

but jetbrains products are based on IDEA which are hard to use on slower processors and this one should only be slower than the others.

And they take a gigabyte of RAM! Oh god.

I just loaded it once without using it, seemed at least as heavy weight as their other products.

This is why I mentioned scite, it takes no ram and no processing power.

I love ides built on it!

We complain about optimizing programmer time, yet are still worried about...a gigabyte of memory? Is hardware still so expensive? Ya, if I want to run on an iPad pro, I need to be careful...4 GB of RAM in 2016? Really?

A fundamental problem with extensible syntax is that for usability a programmer needs to be able to look an an expression and know how to parse it without first studying the unbounded context in which it occurs, which could include thousands or millions or whatever lines of library code but in any case too much. A theoretical form of that practical observation showed up in my formal study of extensible syntax: in order for a parser to be well-behaved, as pinned down by formal definitions in the paper, it has to exhibit unambiguous behavior across varying sets of syntax rules, and that unambiguity includes fixing all operator precedence beforehand. ALGOL 68 had really general facilities for programmer specification of operator-precedence games and frankly that feature was a blot on the language design.

I came out of that investigation still thinking in terms of flexible syntax, but for basic usability I reckon the flexibility has to be in the nature of selecting amongst possibilities within an unambiguous space. Standard Lisp syntax provides one particularly simple way of doing that; most languages, while their basic syntax is more complicated than Lisp's, limit their syntactic extensibility to defining more identifiers within an unambiguous parsing structure. It wouldn't be too hard to provide an unambiguous parsing structure with much more complexly structured operators, my favorite being "simple" alternation between keywords and subexpressions. But afaics variable operator precedence is just as bad as more overtly insane forms of ambiguity.

... and that unambiguity includes fixing all operator precedence beforehand

I think I've been through your formal report earlier, but I guess I didn't retain enough to quickly re-learn the setup. But I don't agree with the conclusion here, so I must not agree with one of your hypotheses. I think operator precedence is important for readability in some domains and is not something that should be baked in (libraries should be able to specify new operators with new precedence).

I think standard operators like +, * etc should have pre-defined fixed precedence, and should also be restricted by usage axioms. For example the common usage of '+' to concatenate two strings is bad in my opinion.

I think there should be a central repository for operator definitions, and new operators should be registered along with their precedence and axioms.

This way you can rely on the algebraic properties of addition, even when used in other unfamiliar problem domains.

If you don't want to publish a new operator to the repository, you should be using a function with clear precedence instead.

This gives the flexibility to define new operators, but makes sure the reading of code containimg operators is clear and unambiguous.

I routinely fully parenthesize expressions, even things like ((x * y) + (z * w)), in languages that don't require me to. My suspicion is that the single biggest problem with attempts to "improve" Lisp syntax by reducing the number of parentheses is that they focus on reducing parenthesis at the small scale, by introducing operator precedence, whereas the parentheses one should be eliminating are the ones used on a large scale.

if then elseif else endif
Infinitely better than
(cond (()())(()())(t()))
or
(if () ()(if () () ()))

Anyone who disagrees... Well I'd probably be breaking the rules of this website to finish that sentence.

As I said, the parentheses one should be eliminating are the ones at a large scale rather than at a small scale. if constructions are not usually small-scale.

that large scale parens are a problem.

Medium too

I prefer custom delimiters on method calls

for instance smalltalkish, mostly
aVisibleObject moveToX: max(min(aVisibleObject atX,window maxX) window minX) y: window top.

You seem to be finding it difficult to argue your point. Generally speaking, that is always good feedback that one has reached some kind of cognitive blindspot. Sure- you have a very strong subjective opinion on whether one form is better than another, but you are also unable to see where your subjective opinions are not universal.

I would say that your third form is probably the sweetspot. I wouldn't make claims about this being some sort of objective fact, or about people who don't see it that way. You are discussing the perception of an unnatural, trained ability. Nobody was born with an innate ability to read code in any form, we would all be somewhat more comfortable with whichever of those forms matched the majority of our training.

In particular - the third form makes a lot of sense to anyone who has spent much time debugging parse-trees being generated from a grammar. They would spot instantly that it does not make your first form, because the parentheses have made one structural attribute (the number of children of each node) more visible, at the expense of making another structural attribute (the role of each child) less visible.

From this point of view we should all be programming in an XML like syntax, however I personally find this difficult to work with.

It's a shame that XML is so verbose otherwise it could be useful. It's a good idea hidden inside a terrible implementation.

If we are favouring named brackets, XML only has a 2 character overhead for start, and 3 for end. You would give arguments between function brackets like this:

<add> <multiply> 3 2 </multiply> 4 </add>

Of course this is not strictly XML and just borrowing its tag concept for named brackets, without the rest.

I don't see the problem with fully parenthesising expressions. I don't see the magical difference between an operator and any other arbitrary function either.

As such, precedence beyond "there is no precedence" seems futile. You'll always have to parenthesise at some point. Without written mathematics' typographical wealth (in this case, superscripts) to fall back on, is 2 ^ n + 1 "raise 2 to the power of n + 1" or "raise 2 to the power of n and then add 1"?

Explicitly parenthesising is an explicit definition of a dependency tree. Making parts of that tree implicit invites error.

In APLish languages the rule is simple: it has only unary and binary operators and everything to the right of an operator is the right side argument! So 2^n+1 is 2^(n+1) and not (2^n)+1. And it is also how you usually say it: two raised to the negative of en plus one. Contrast that with 2^(-n)+1, which you'd say as "two raised to minus en plus one". For more complex parenthesized expressions you'd have to say open paren ... close paren which soon gets unwieldy. You may even get in the habit of using commutativity to reduce parentheses! So 1+2^n instead (2^n)+1.

In practice this gets you quite far without needing too many parentheses. You can even do nifty things like a@<a:x^2 which, given a list x of numbers, returns a list of squares of these numbers in ascending order. And you never have to look up precedence rules (unlike in pedestrian languages like C, C++ etc.)!

I also like Scheme for the same reason (among many).

My choice is that we type '+' to select the concatenation operator, but that's a different symbol than the addition operator and the two can be visually distinguished in an IDE. Overloading is just used to solve the problem of inputting the symbols easily with a keyboard. Readability is addressed by making distinct symbols visually distinct in the IDE.

Regarding axioms, I think every symbol (not just operators) should have a defining theory. I can't get my head around a central repository being a good idea.

I think too many symbols is a problem too, they are hard to remember, and obscure what without a key. The only reason I am prepared to include common symbols is because their behaviour is generic.

Edit: the problem I am trying to solve with a central repository is that if you let people assign random semantics to operators it becomes difficult to read code. On the other hand you want to be able to define operators over new types, such as vector addition, and you may want to define new operators that are not in the language by default. The only way I can think of meeting both these requirements is that the algebraic properties of operators get defined in a central repository, so you can define new operators and publish their properties, but be sure when reading code what the properties of the operators are, and not find the same custom operator used in two different incompatible ways in two different libraries.

for compact languages. But then you have to speak Chinese.

FYI, when I used the term 'symbol', I didn't mean non-letter glyph, but something more like symbols of Lisp that include ordinary names.

Traditionally operators are non letter symbols. Of course Haskell let's you infix any two parameter function with backquotes.

There is also left and right associativity to consider as well as precedence.

I think part of the problem is that keyboards are not good at typing symbols. I find it annoying to have to slow down to work out how to input unusual symbols. I think the best option is to use ascii symbols for input, but you can certainly pretty print ascii using unicode.

I would also add "legacy tools" to the reasons to stick to an ascii encoding. I still have to figure out how to add reasoning about quotient types, but with Construct as a quotient type, you can have multiple encodings (say, ascii and structural) which are identical as values of Construct type. I think this is strictly better than object encapsulation, with which you'd lose the ability to "unseal" the construct and view the underlying encoding.

I am playing with language design, and I wouldn't release a language without trying to release an IDE too.

It's getting easy enough. I hope.

I always tend to plan to make unicode versions, but it's not easy.

For one thing it takes a HUGE piece of data and code to normalize unicode so that all unicode inputs that look the same get converted to a canonical form and interpreted the same by the compiler.

I have written an implementation of that in lua (actually translated a C library).

But I won't do that for a first pass. It's a lot of work.

And I suspect that the result will be that people in other countries will start using variable names that I can't duplicate on my keyboard.

And it will make me a little sad to look at asian language programs and have trouble distinguishing words...

But it will happen if it hasn't already.

Some languages support Unicode input.

For reasons others don't support Unicode: legacy, no really satisfying input manners, consumes lots of memory, and you need a handwritten somewhat complex lexer.

Unicode has categories.

It already knows which characters are letters which are punctuation which are numbers which are whitespace.

You can use the standard data sets to make your lexer. And you can use free, unencumbered libraries to interpret it.

Read: Hand written as in you cannot really use the standard lexer analysis tools since Unicode tends to blow up the state space of the generated lexers.

Your lexer generator is using the wrong algorithm.

Using an algorithm from the 50's or 60's on a processor from this decade is a bad impedance match. Just use one that reads the specification directly and doesn't generate combinatorial tables.

If it takes 10 times as long as some Chomsky DFA and 10 times the ram, so what?

Just noting most lexer tools don't handle Unicode well. That's more or less a fact, and someday might -or will- change.

very optimized for the language it was written in - 300 lines.

Is hand written so very bad?

It depends on the language you're working in, the complexity of the regular expressions you want to recognize, and the performance you're aiming for.