Earl Grey; the story of a new programming language

"Screw it, I'll make my own!"

If you ask me why I made a programming language, I could justify it in a lot of ways, point out its strengths, what I think it does better than the others, and so on. But I don't think that's really the driving force. As I see it, that driving force is, basically, a kind of conceit. A typical programmer will learn one or several well-established languages, depending on what they aim to achieve. They will adapt their way of thinking to fit these tools as best they can. But perhaps you don't want to adapt. You don't like any of the tools you can find because they are never exactly the way you want them, it's like they don't fit your brain at the moment. If you won't adapt to the language, the only alternative is to adapt the language to you. And if you are like me it becomes a bit of an obsession, an itch you just have to scratch...

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

The unreasonable man

"The reasonable man adapts himself to the world: the unreasonable one persists in trying to adapt the world to himself. Therefore all progress depends on the unreasonable man." —George Bernard Shaw, "Rules for Revolutionaries" (1903)

Let the code talk more for

Let the code talk more for itself, so much talking about the personal journey! We've all been there, it's cool but frankly you can't really expect people to get much of it without taking the journey themselves.

The whole point of the post

The whole point of the post is the personal journey. If you just want to read the code, it is here: https://github.com/breuleux/earl-grey

Also LtU is not the target audience, so "we've all been there" is not necessarily true.

I get that. For this

I get that. For this audience, I would have liked to have seen more code excerpts to go with your exposition...it would have been useful in better understanding your journey. And no, I don't mean let the code talk completely by itself :) A grand language essay benefits from both.

For the intended audience, it might be hard for them to connect with the content anyways.

It's a personal journey and a status report

FYI, that's not the author. :) If Olivier Breuleux had come to us with this blog post to communicate about Earl Grey, I guess a request for more code samples would make sense as part of an active communication process. Right now it's more like unsolicited public criticism of their writing style.

If you expected an essay that concisely advocated for the Earl Grey language, try the Earl Grey website. The website has a sleek logo and plenty of syntax-highlighted code samples right on the front page. It even has an interactive tutorial. This is a well-polished operation, and I think it indicates that the blog post would have had some code samples if it were really meant to introduce the Earl Grey language. The blog post's goal was probably something else.

I liked it pretty well as a report of a personal experience. I relate to a lot of the same challenges, such as the isolating effect where I knowing how to take full advantage of my own languages better than I know how to take full advantage of mainstream languages. Sometimes it's just nice to hear these things made explicit so that I know that I'm not alone. (And where the experiences are a bit different from mine, it's nice to compare notes.)

I think the blog post is probably even more valuable to the people who actively follow the Earl Grey project. It clarifies why Olivier Breuleux will be devoting attention to new language designs, it clarifies that they'll still be maintaining Earl Grey and using it as a platform for new projects, and it explains why the Earl Grey language came to be the way it is, warts and all.

Never been happy with

alternate syntaxes for s-expressions.

I tried one for scheme, it was buggy as hell.

I'm not sure what to make of combining the same operator next to itself always becomes a single function call. Yes you do want that a lot of the time, but not always.

I like what prolog does better than sexpression alternatives. Having an operator parser and allowing operators with set precedence and direction and arity makes a language you can compose and you can match on directly. You could play with extending that to mixfix if you like.

extensible infix operators with homoiconic precedence

I spent a while thinking about this and now consider it to be a solved problem, thanks to a simple precedence rule: operators surrounded by spaces have lower precedence than operators not surrounded by spaces. This reads really cleanly:

(n * n-1)

(The drawback is obvious from the above example: you can't use dashes in your variable names. I don't want to deemphasize that; I love dashes too. However, there are no other drawbacks if you aren't tied to backwards-compatibility.)

More details.

Precedence is an ugly user

Precedence is an ugly user unfriendly feature even for those of us with lots of experience in math. The problem can be solved simply by getting rid of it, making it a feature of the input editor that puts in braces automatically. Of course, brace rendering in most program,ing environments sucks, but that can be solved also.

If you build your syntax

out of operators, then precedence rules might have more uses than just mathematical convention.

Getting rid of braces in the common use cases is worth a lot in terms or readability.

You're probably forgetting that if THE WHOLE syntax is defined as a precedence grammar then define function is an operator, assignment is an operator, conditionals are operator, message sends are operators, not just add and subtract and divide.

So for instance you don't want every assignment to need parens. a = (1 + 2)

If you have to fully parenthesize your program, then you're back to s-expressions.

So if (a la prolog) define clause is an infix operation then you want it to have the lowest possible precedence so that each clause doesn't need delimeters.

Since operators really act in relation to each other's precedence, having an editor that keeps all operators in a list, sorted by precedence would make adding operators much less painful. In fact one doesn't want numbers so much as a position in a list relative to the other operators that are legal to appear in the same context.

But under it, the information is still no different that what they had in prolog, where a define might look like:
:- op( 1200, xfx, [ :-, --> ]).
:- op( 1200, fx, [ :-, ?- ]).
:- op( 1100, xfy, [ ; ]).
:- op( 1000, xfy, [ ’,’ ]).
:- op( 700, xfx, [ =, is, =.., ==, \==,
=:=, =\=, , == ]).
:- op( 500, yfx, [ +, -]).
etc.

If your parents have light

If your parents have light weight graphical representations, then you can have them everywhere without adversely effecting readability, so why not? I think a lot of bugs occur because of subtle precedence relationships between even common operators, it is redicullous that we aren't looking for better solutions. More precedence is definitely not the answer. Especially if those precedence relations are even more subtle.

Minor differences

I agree that we should have fewer precedence relations, and that ambiguity of operator order should be an error, but I don't think lightweight parens are the best way to indicate precedence. I think spacing works better. You can also use operator font (size) in the same way that you'd use bigger parens for larger expressions.

absolute nonsense

totally impossible for them not to adversely affect readability when applied to the WHOLE grammar, not just expressions. And frankly I think they ruin expressions too. I even prefer method calls to avoid unecessary parens like in smalltalk. Instead of object.method(x expression,y expression) in smalltlk it might be "object methodX: xexpression y: yexpression."

But you do need mixfix for such operations as if - then - else

Nonsense. If you are just

Nonsense. If you are just thinking about plain text, you deserve your 1970s programming experience. Otherwise, syntax is much more than a BNF in code presentation and even code input.

Unnecessary symbols and line breaks

harm readability, dim or not,even if aligned.

Who cares about unadorned

Who cares about unadorned text on a 1970s teletype? There are much better solutions to these problems using modern typography.

I guess you want graphical delimiters but not custom text

I guess you want graphical delimiters but not custom text delimiters.

Seems arbitrarily elitist.

Why aren't they equivalent? Or combinable?

In the real world, the most available cross platform programmer's text editting widget is scite in wx.

Custom text delimiters don't

Custom text delimiters don't work, at least without a good amount of typographical hacking. It isn't arbitrarily elitist to shun solutions that simply don't work.

The real programming world is still living in the 70s and we are suffering the consequences as a result. Type writer text is all we can afford, so let's have programmers interpret a bunch of subtle precedences in their heads since anything else is unreadable! God forbid if we actually change the first premise.

i have no idea what you're claiming

else doesn't work?
repeat until doesn't work?
function endfunction doesn't work?
for by next doesn't work?

I think they work just fine.

I'll admit that C doesn't work well
if () {statement1; statement2;}
vs
if () statement1; /*oops*/statement2;
is a failure.
The need for generic delimeters () and {} is a failure, they don't supply enough context to the eye and make you look back and forth. And if they're optional, changing the meaning by position, my god.

So the problem isn't CUSTOM delimeters, it's bad grammars with generic delimeters. And the worst offender is Lisp

The point is to have a program that your eye can scan without moving back and forth and counting, and without wasting screen real estate.

I think custom delimeters and operations are fine.

Quite the opposite, I think structural program editors fail.
If you have trouble entering in an incomplete structure and finishing it. If you have trouble moving and copying part of a construct... If you have to click around for constructs instead of typing, you have a slow workflow

Extraordinary claims require extraordinary evidence

If you think you have a much superior writing and editing workflow than typing, you're going to have demonstrate that.

Typing is fine. It's the

Typing is fine. It's the expectation that what is typed is vanilla unembellished text that isn't. I've already made posts in this direction (e.g. highlighting precedence in the editor). Also, making braces/parens typographically less annoying. Still a lot of work needs to be done, and I'm not a professional typographer. But I can totally see the day when we type/edit text as usual, but what is presented on the screen is much more rich and useful.

Colored&sized parens worse than symbols that don't need context

Colored and sized parens are still worse than symbols that don't need context. Even colored, sized parens aren't self explanitory and make you move your eye back and forth to see what construct they belong to.

"Endif" explains much more without moving your eye.

And in a lot of cases or just the common case, you can get away with no paren. Say calls with one parameter don't need parens, in that case those elided parens don't confuse the eye when you're trying to match parens that ARE necessary.

Making code understandable locally, no need to move your eyes for context, no question of what matches what, no ambiguous symbols is principle.

You can't get around the fact that the grammar is important to that goal, no matter what the editor does.

Again, no typography is

Again, no typography is better than crappy typography. Color and size aren't good enough, we have to etch the background itself with contrinuous shapes and layouts. We have to create a real graphical syntax for our languages.

We have more to work with than just grammar/syntax. Bitmapped displays are quite mature now.

Dense and unfamiliar

I imagine you could come up with a very dense display that only children would be flexible enough to learn and appreciate.

Maybe.

Highlighting/animating parts on mouseover would be interesting.

As long as we're pie in the sky I wonder if 3d or real 3d would be useful. I remember that it's useful in visualizing data, if not programs.

Personally, I'm holding off

Personally, I'm holding off until I understand newer system needs more. There is a lot to improve in standard code presentation, but I've also discovered the usefulness of the *gasp* mouse in manipulating code in a live programming context (e.g. Giving real estate to the unknown with holey terms to enable very quick local refactoring). If we are going to embellish code presentation, then we might as well provide some affordances for fluid non-text based editing (e.g. Scrubbing).

Hololens programming environment ala something like the ironman UI is much farther off. But it is a nice direction to move in.

I'll google all of those

unless you have links.

I'm interested.

See my latest demos. I've

See my latest demos. I've been lazy about aggregating them, but Yow! or even Strangeloop will have some of this.

Grammar is important, and

Grammar is important, and there is grammatical difference between

(if T C A)

and

if T then
C
else
A
endif

but all you're arguing is syntax, which is utterly irrelevant.

Irrelevant to readability

speed of understanding, effort for a human to see and compare, obviousness of errors?

What can I say but "wrong"?

Sigh

There is a whole field of typography that is used to reason about this kind of thing. Typographers are also quite adept at her ally judging readability and such. Computer scientist have uninformed opinions about typography like designers can have uninformed opinions about computer science.

I hope someday we can have more informed debates about this, until then we are probably just devaluing the professionalism of another very important field.

locality is my point not typography

if the syntax is designed so that meaning is as local to a symbol as possible, where ambiguities are reduced by not having multiple contexts that could apply to that symbol - so that in the context of a typical expression the meaning is local - the code can be read faster and more accurately.

There's an analogous point that CODE whose meaning is dependent on OTHER CODE is harder to understand. Well that principle applies ALL THE WAY DOWN IN SCALE like a fractal. If a symbol can be understood locally that's better than if the meaning is dependent on further away symbols.

The meaning of ')' depends on where the matching '(' is.
But the meaning of 'endfunction' does not require matching, typically, one will simply remember where the latest lambda was.

Move your eyes this way or

Move your eyes this way or that say is typography, where do you think narrow newspaper and paper columns came from?

Expecting humans to run push down automatas in their heads is a bit masachistic, which is why I think nested structures should be more explicitly chunked visually. And there is always a limitation on how much nesting is viable before heads explode, which is quite independent of locality. Endfunction wins because it doesn't nest easily, or nests vertically rather than horizontally. If you have a 4-level deep endfunction nesting, however, well...it will devolve into the unusability of parens very quickly.

Note that this moves from typography to straight usability. Why it is hard has more to do with it being hard than ugly.

Also, structure editors

Speed and position on the screen
...
On some current processors they'll slow down as typed text implies a need to restructure the program

And if the processor was infinitely fast, it's a problem that the eye isn't. If code jumps around the screen to accommodate the updating structure.

And (as in some structure editors) you can't just type, you have to choose, well there's an interestingly different workflow. Will it be too slow? Who knows. There are structure editors that don't allow you to make a grammatical error. Is that a loss? Most current programmers would say yes.

I've never built a

I've never built a structured editor before, so I have no idea. Fortunately, parsing and good error recovery is easy if done by hand, and graphical structure can be easily projected around the parse tree.

No idea about old processors. I get that supporting programmers using 8086s on old teletype terminals is still a thing to some PL markets, I personally find it completely uninteresting.

Not that old

IDEA is too slow for 4 year old atom processors, as is Visual Studio 2010

Jetbrains makes a structure editor for DSLS that doesn't let you make grammatical errors. It's probably too slow for large files on current processors. And WAY too slow to use on atoms.

Hell, Racket's IDE is too slow on Atom processors.

I don't know. On the one

I don't know. On the one hand, computing resources are getting better, but on the other hand, IDEs like eclipse and visual studio continue to unimpress. I haven't found these features to be particularly computationally intensive yet. Simply putting more pixels on the screen is definitely rather cheap, even on 220 PPI displays.

JetBrains MPS is a projectional editor, it's probably doing a lot more than my systems are. I don't see why it should be that slow on large files however, maybe they aren't virtualising enough? Or maybe they are being incremental enough? It's hard to say.

Haven't tried to actually edit with it

but jetbrains products are based on IDEA which are hard to use on slower processors and this one should only be slower than the others.

And they take a gigabyte of RAM! Oh god.

I just loaded it once without using it, seemed at least as heavy weight as their other products.

This is why I mentioned scite, it takes no ram and no processing power.

I love ides built on it!

We complain about optimizing

We complain about optimizing programmer time, yet are still worried about...a gigabyte of memory? Is hardware still so expensive? Ya, if I want to run on an iPad pro, I need to be careful...4 GB of RAM in 2016? Really?

Operator precedence and extensibility

A fundamental problem with extensible syntax is that for usability a programmer needs to be able to look an an expression and know how to parse it without first studying the unbounded context in which it occurs, which could include thousands or millions or whatever lines of library code but in any case too much. A theoretical form of that practical observation showed up in my formal study of extensible syntax: in order for a parser to be well-behaved, as pinned down by formal definitions in the paper, it has to exhibit unambiguous behavior across varying sets of syntax rules, and that unambiguity includes fixing all operator precedence beforehand. ALGOL 68 had really general facilities for programmer specification of operator-precedence games and frankly that feature was a blot on the language design.

I came out of that investigation still thinking in terms of flexible syntax, but for basic usability I reckon the flexibility has to be in the nature of selecting amongst possibilities within an unambiguous space. Standard Lisp syntax provides one particularly simple way of doing that; most languages, while their basic syntax is more complicated than Lisp's, limit their syntactic extensibility to defining more identifiers within an unambiguous parsing structure. It wouldn't be too hard to provide an unambiguous parsing structure with much more complexly structured operators, my favorite being "simple" alternation between keywords and subexpressions. But afaics variable operator precedence is just as bad as more overtly insane forms of ambiguity.

... and that unambiguity

... and that unambiguity includes fixing all operator precedence beforehand

I think I've been through your formal report earlier, but I guess I didn't retain enough to quickly re-learn the setup. But I don't agree with the conclusion here, so I must not agree with one of your hypotheses. I think operator precedence is important for readability in some domains and is not something that should be baked in (libraries should be able to specify new operators with new precedence).

Standardised Precedence

I think standard operators like +, * etc should have pre-defined fixed precedence, and should also be restricted by usage axioms. For example the common usage of '+' to concatenate two strings is bad in my opinion.

I think there should be a central repository for operator definitions, and new operators should be registered along with their precedence and axioms.

This way you can rely on the algebraic properties of addition, even when used in other unfamiliar problem domains.

If you don't want to publish a new operator to the repository, you should be using a function with clear precedence instead.

This gives the flexibility to define new operators, but makes sure the reading of code containimg operators is clear and unambiguous.

parentheses

I routinely fully parenthesize expressions, even things like ((x * y) + (z * w)), in languages that don't require me to. My suspicion is that the single biggest problem with attempts to "improve" Lisp syntax by reducing the number of parentheses is that they focus on reducing parenthesis at the small scale, by introducing operator precedence, whereas the parentheses one should be eliminating are the ones used on a large scale.

custom delimeters are better than parens

if then elseif else endif
Infinitely better than
(cond (()())(()())(t()))
or
(if () ()(if () () ()))

Anyone who disagrees... Well I'd probably be breaking the rules of this website to finish that sentence.

if then else endif

As I said, the parentheses one should be eliminating are the ones at a large scale rather than at a small scale. if constructions are not usually small-scale.

I was agreeing

that large scale parens are a problem.

Medium too

how about medium scale?

I prefer custom delimiters on method calls

for instance smalltalkish, mostly
aVisibleObject moveToX: max(min(aVisibleObject atX,window maxX) window minX) y: window top.

Difficult to argue.

You seem to be finding it difficult to argue your point. Generally speaking, that is always good feedback that one has reached some kind of cognitive blindspot. Sure- you have a very strong subjective opinion on whether one form is better than another, but you are also unable to see where your subjective opinions are not universal.

I would say that your third form is probably the sweetspot. I wouldn't make claims about this being some sort of objective fact, or about people who don't see it that way. You are discussing the perception of an unnatural, trained ability. Nobody was born with an innate ability to read code in any form, we would all be somewhat more comfortable with whichever of those forms matched the majority of our training.

In particular - the third form makes a lot of sense to anyone who has spent much time debugging parse-trees being generated from a grammar. They would spot instantly that it does not make your first form, because the parentheses have made one structural attribute (the number of children of each node) more visible, at the expense of making another structural attribute (the role of each child) less visible.

XML

From this point of view we should all be programming in an XML like syntax, however I personally find this difficult to work with.

It's a shame that XML is so

It's a shame that XML is so verbose otherwise it could be useful. It's a good idea hidden inside a terrible implementation.

Named brackets are necessarily verbose

If we are favouring named brackets, XML only has a 2 character overhead for start, and 3 for end. You would give arguments between function brackets like this:

<add> <multiply> 3 2 </multiply> 4 </add>

Of course this is not strictly XML and just borrowing its tag concept for named brackets, without the rest.

To hell with precedence

I don't see the problem with fully parenthesising expressions. I don't see the magical difference between an operator and any other arbitrary function either.

As such, precedence beyond "there is no precedence" seems futile. You'll always have to parenthesise at some point. Without written mathematics' typographical wealth (in this case, superscripts) to fall back on, is 2 ^ n + 1 "raise 2 to the power of n + 1" or "raise 2 to the power of n and then add 1"?

Explicitly parenthesising is an explicit definition of a dependency tree. Making parts of that tree implicit invites error.

In APLish languages the rule is simple

In APLish languages the rule is simple: it has only unary and binary operators and everything to the right of an operator is the right side argument! So 2^n+1 is 2^(n+1) and not (2^n)+1. And it is also how you usually say it: two raised to the negative of en plus one. Contrast that with 2^(-n)+1, which you'd say as "two raised to minus en plus one". For more complex parenthesized expressions you'd have to say open paren ... close paren which soon gets unwieldy. You may even get in the habit of using commutativity to reduce parentheses! So 1+2^n instead (2^n)+1.

In practice this gets you quite far without needing too many parentheses. You can even do nifty things like a@<a:x^2 which, given a list x of numbers, returns a list of squares of these numbers in ascending order. And you never have to look up precedence rules (unlike in pedestrian languages like C, C++ etc.)!

I also like Scheme for the same reason (among many).

Keep symbols visually distinct

My choice is that we type '+' to select the concatenation operator, but that's a different symbol than the addition operator and the two can be visually distinguished in an IDE. Overloading is just used to solve the problem of inputting the symbols easily with a keyboard. Readability is addressed by making distinct symbols visually distinct in the IDE.

Regarding axioms, I think every symbol (not just operators) should have a defining theory. I can't get my head around a central repository being a good idea.

Too many symbols

I think too many symbols is a problem too, they are hard to remember, and obscure what without a key. The only reason I am prepared to include common symbols is because their behaviour is generic.

Edit: the problem I am trying to solve with a central repository is that if you let people assign random semantics to operators it becomes difficult to read code. On the other hand you want to be able to define operators over new types, such as vector addition, and you may want to define new operators that are not in the language by default. The only way I can think of meeting both these requirements is that the algebraic properties of operators get defined in a central repository, so you can define new operators and publish their properties, but be sure when reading code what the properties of the operators are, and not find the same custom operator used in two different incompatible ways in two different libraries.

Chinese might have an advantage

for compact languages. But then you have to speak Chinese.

Symbol a la Lisp

FYI, when I used the term 'symbol', I didn't mean non-letter glyph, but something more like symbols of Lisp that include ordinary names.

Operator symbols

Traditionally operators are non letter symbols. Of course Haskell let's you infix any two parameter function with backquotes.

There is also left and right associativity to consider as well as precedence.

Uhm.... Unicode?

It is difficult to understand why we persist in making our programs in a 7-bit subset of Unicode, when at this point we have all of Unicode available. We could at the very least use all of Latin-1, Latin-2, Greek, and Mathematical Symbols.

So, for starters, we could have syntactic keywords render in italic and imported definitions from libraries in bold. We could use different kinds of delimiters to delimit different syntactic and control constructs. We could use Greek characters for variable names to match the Greek characters in mathematical formulas we're implementing. And why would anyone overload the addition operator for concatenation with the actual concatenation symbol available, or for set union with the actual union operator available? Why would anyone overload the subtraction operator for set difference, with the set-difference operator directly available?

We could use the actual correct mathematical symbols for other operators too, like 'ForAll' and 'IsAMemberOf' and 'IsContainedIn' and 'Union' and 'Intersection' and 'Integral' and 'Derivative' and so on. And if you define APL functions, there's no reason not to use APL symbols for them.

And why aren't we letting programmers who use those languages name their variables in Cyrillic, Arabic, and Hebrew characters?

I mean, I'm no typographer, but I don't have to be. I'm still talking about plain text here. Program files ought IMO to be plain text, and at this point Unicode is plain text. So why is nobody doing that?

Keyboards

I think part of the problem is that keyboards are not good at typing symbols. I find it annoying to have to slow down to work out how to input unusual symbols. I think the best option is to use ascii symbols for input, but you can certainly pretty print ascii using unicode.

Agreed

I would also add "legacy tools" to the reasons to stick to an ascii encoding. I still have to figure out how to add reasoning about quotient types, but with Construct as a quotient type, you can have multiple encodings (say, ascii and structural) which are identical as values of Construct type. I think this is strictly better than object encapsulation, with which you'd lose the ability to "unseal" the construct and view the underlying encoding.

change is in the air...

I am playing with language design, and I wouldn't release a language without trying to release an IDE too.

It's getting easy enough. I hope.

I always tend to plan to make unicode versions, but it's not easy.

For one thing it takes a HUGE piece of data and code to normalize unicode so that all unicode inputs that look the same get converted to a canonical form and interpreted the same by the compiler.

I have written an implementation of that in lua (actually translated a C library).

But I won't do that for a first pass. It's a lot of work.

And I suspect that the result will be that people in other countries will start using variable names that I can't duplicate on my keyboard.

And it will make me a little sad to look at asian language programs and have trouble distinguishing words...

But it will happen if it hasn't already.

So why is nobody doing that?

Some languages support Unicode input.

For reasons others don't support Unicode: legacy, no really satisfying input manners, consumes lots of memory, and you need a handwritten somewhat complex lexer.

Not hand written.

Unicode has categories.

It already knows which characters are letters which are punctuation which are numbers which are whitespace.

You can use the standard data sets to make your lexer. And you can use free, unencumbered libraries to interpret it.

Falls into my category of hand written

Read: Hand written as in you cannot really use the standard lexer analysis tools since Unicode tends to blow up the state space of the generated lexers.

Still not a problem

Your lexer generator is using the wrong algorithm.

Using an algorithm from the 50's or 60's on a processor from this decade is a bad impedance match. Just use one that reads the specification directly and doesn't generate combinatorial tables.

If it takes 10 times as long as some Chomsky DFA and 10 times the ram, so what?

It's not my impendence match

Just noting most lexer tools don't handle Unicode well. That's more or less a fact, and someday might -or will- change.

My last hand written lexer

very optimized for the language it was written in - 300 lines.

Is hand written so very bad?

Well. Depends

It depends on the language you're working in, the complexity of the regular expressions you want to recognize, and the performance you're aiming for.