Eve: the development diary of a programming environment aimed at non-programmers

In spring 2012 Chris Granger successfully completed a Kickstarter fundraising and got $300K (instead of the requested $200K) to work on a live-feedback IDE inspired by Bret Victor "Inventing on principle" talk. The IDE project was called Light Table. It initially supported Clojure (the team's favourite language) only, but eventually added support for Javascript and Python. In January 2014, Light Table was open sourced, and in October 2014 the Light Table development team announced that they decided to create a new language, Eve, that would be a better fit for their vision of programming experience.

There is little public about Eve so far, no precise design documents, but the development team has a public monthly Development Diary that I found fairly interesting. It displays an interesting form of research culture, with in particular recurrent reference to academic works that are coming from outside the programming-language-research community: database queries, Datalog evaluation, distributed systems, version-control systems. This diary might be a good opportunity to have a look at the internals of a language design process (or really programming environment design) that is neither academic nor really industrial in nature. It sounds more representative (I hope!) of the well-educated parts of startup culture.

Eve is a functional-relational language. Every input to an Eve program is stored in one of a few insert-only tables. The program itself consists of a series of views written in a relational query language. Some of these views represent internal state. Others represent IO that needs to be performed. Either way there is no hidden or forgotten state - the contents of these views can always be calculated from the input tables.

Eve is designed for live programming. As the user makes changes, the compiler is constantly re-compiling code and incrementally updating the views. The compiler is designed to be resilient and will compile and run as much of the code as possible in the face of errors. The structural editor restricts partially edited code to small sections, rather than rendering entire files unparseable. The pointer-free relational data model and the timeless views make it feasible to incrementally compute the state of the program, rather than starting from scratch on each edit.

The public/target for the language is described as "non-programmers", but in fact it looks like their control group has some previous experience of Excel. (I would guess that experimenting with children with no experience of programming at all, including no Excel work, could have resulted in very different results.)

Posts so far, by Jamie Brandon:

Some random quotes.

Retrospective:

Excited, we presented our prototype to a small number of non-programmers and sat back to watch the magic. To our horror, not a single one of them could figure out what the simple example program did or how it worked, nor could they produce any useful programs themselves. The sticking points were lexical scope and data structures. Every single person we talked to just wanted to put data in an Excel-like grid and drag direct references. Abstraction via symbol binding was not an intuitive or well-liked idea.

[...]

Our main data-structure was now a tree of tables. Rather than one big top-level function, we switched to a pipeline of functions. Each function pulled data out of the global store using a datalog query, ran some computation and wrote data back. Having less nesting reduced the impact of lexical scope and cursor passing. Using datalog allowed normalising the data store, avoiding all the issues that came from hierarchical models.

At this point we realised we weren't building a functional language anymore. Most of the programs were just datalog queries on normalised tables with a little scalar computation in the middle. We were familiar with Bloom and realised that it fit our needs much better than the functional pidgin we had built so far - no lexical scoping, no data-structures, no explicit ordering. In late March we began work on a Bloom interpreter.

October:

Where most languages express state as a series of changes ('when I click this button add 1 to the counter'), Eve is built around views over input logs ('the value of the counter is the number of button clicks in the log'). Thinking in terms of views makes the current language simple and powerful. It removes the need for explicit control flow, since views can be calculated in any order that is consistent with the dependency graph, and allows arbitrary composition of data without requiring the cooperation of the component that owns that data.

Whenever we have tried to introduce explicit change we immediately run into problems with ordering and composing those changes and we lose the ability to directly explain the state of the program without reference to data that no longer exists.

[...]

In a traditional imperative language, [context] is provided by access to dynamic scoping (or global variables - the poor mans dynamic scope) or by function parameters. In purely functional languages it can only be provided by function parameters, which is a problem when a deeply buried function wants to access some high up data and it has to be manually threaded through the entire callstack.

December:

Eve processes can now spawn subprocesses and inject code into them. Together with the new communication API this allowed much of the IDE architecture to be lifted into Eve. When running in the browser only the UI manager lives on the main thread - the editor, the compiler and the user's program all live in separate web-workers. The editor uses the process API to spawn both the compiler and the user's program and then subscribes to the views it needs for the debugging interface. Both the editor and the user's program send graphics data to the UI manager and receiving UI events in return.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

The act of programming as user interaction

This reminds me of the principles/slogans that went along with the original development of graphic user interfaces and smalltalk.
The main idea was that people shouldn't have to remember commands and modes. Instead of of a command line interface, people would be offered a visually navigable set of choices.

Modes are implicit context, and that's confusing.

And choosing from a concrete list is easier than remembering choices per modes.

I remember a tee shirt that said "don't mode me in."

So I guess if you want programming to be an exercise for naive users then you have to automate the engineering part and hide any math - let the computer keep track of the options and offer concrete lists to choose from.

Suddenly programming has to be follow the rule that the user doesn't have to remember any entities or principles, and that navigating is all.

If that sounds impossibly limiting, it occurs to me that abstractions usually start from lists of specific items that those abstractions apply to. So if you want to make an abstraction such as a type clear to a user, you can just list all of the entities that satisfy that abstraction - turn the abstract into concrete.

And this is why code

And this is why code completion has become so important. It turns programming into a multiple choice affair.

I have another idea for advance programming

I think more choosing and less remembering could help us DESIMPLIFY our languages.

Say there's a construct with a plethora of possible options.

Current programming languages pick a small number of cases they hope is general enough, but there are lots of possibilities they leave out.

What if programming constructs come with pop-up options pages, check boxes, radio-buttons, fields and property sheets so that you can find and pick between a large bunch of options quickly, then those are displayed in the code as some compact textual description. But you can bring the sheet back up at any time.

And part of designing your own apis could involve generating your own option sheets for your own objects or constructs.

Bad user interfaces.

Options pages bristling with check-boxes and more buttons than TV remote controls are really bad UI design. Simple generic operations that are applicable in more cases are better. Its all about finding the right abstraction, that results in a simpler, more generally applicable solution, not proliferation of hidden state and weird modes.

there are use cases

These are tools to help people pick between complex options.

Are you saying that APIs and other constructs NEVER have complex options?

Unless our programming languages have this built in, then we're denying engineers a tool. A tool that they are familiar with and supply to all of their users because they already know how useful it is.

Ideal situations vs real ones.

Another way to put it is that while "simple" is an ideal, not all of reality can be fit into the ideal. Some problems "suck" and we'd do better to "embrace the suck" and admit that it's better to have tools that make it more barable. Saying "we refuse to help you with a class of ugly problems - avoid those problems, we'll make sure you suffer more when you have them" Is not how you make engineers' life better.

And as for more basic things than apis, languages, it can take decades to find the perfect abstraction - managing the job quickly may be better. And maybe our abstractions aren't as good as we think, we're just stuck in a mindset that abstraction is better.

yes

Anybody here who is grossed out by e.g. "programmer art" or "programmer ui/ux" has a knee-jerk reaction. You sound like you know what you are doing vs. the rest of the world, so sure, you can make a scary ui if you believe that is our current local maxima. :-)

What if programming

What if programming constructs come with pop-up options pages, check boxes, radio-buttons, fields and property sheets so that you can find and pick between a large bunch of options quickly, then those are displayed in the code as some compact textual description.

Quartz Composer is popular with the design crowd. I like the idea of a meta user-interface for object configuration, but realizing that is a challenge. Code completion is simple and universal.

Bret Victor has the right ideas in his Learnable Programming essay, but try generalizing that to non-visual code...more work needs to be done.

dialog boxes for language dialects

Actually my original idea wasn't language constructs configured with dialog boxes, it was language dialects, both semantic and syntactic.

So you could have a very configurable language where you could make complex choices about which features of the language you want, but the granularity wouldn't have to be a whole project. One idea was that different parts of the same program could even have features that are usually considered incompatible to have in the same system, and have the system robustly glue them together.

Examples of incompatible features could grammar differences, could be safe, statically type checked code, vs. dynamic typed, vs, unsafe with casting could be abis, such as spaghetti stacks for continuations, vs. normal stacks, could be C style numeric tower vs. LISP style numeric tower, vs. objects all the way down, could be impure logic search strategies vs. pure, breadth first, iterative deepening etc. default lazy evaluation vs. default strict, could be allocates and garbage collects memory vs. uses preallocated memory vs. manual allocation, could be compiled ahead of time vs. trace compiled vs. jit with static types but staging, could be multiprocessor optimized vs. uses optimizations that forbid threads and state.

Robustly glue them together

Robustly gluing together program fragments whose semantics are given by disjoint (or even worse, incompatible) features is an open problem. Gradual typing (mixing static and dynamic parts in a single program) is subtle enough that there was a paper published this year about what its "correct" definition is, and which results do we have about it (not a lot), Mixing strict and lazy code has a very subtle semantics that we don't really know how to reason about well right now. Reasoning on a program fragment written without control-effects (setjmp/longjmp for example) but that may call other parts that do use control-effects is also a challenge.

Changed my post

I had forgotten part of my original idea while posting about "my original idea". I was a bit distracted and tired, sorry. So we cross posted.

So grammar changes were something important I left out.

:) well we'll learn a lot by trying

Gluing together bits from different languages is currently harder than whatever problem you're probably trying to solve, so having tools with dialect level differences designed to work with each other would be novel.

Other kinds of engineering don't have the "you can only use one tool per job" problem that software engineering has. I'd like to see that restriction disappear.

Agreed

In fact I'm planning to work on that. For an ambitious programme in this direction, you may be interested in the recent Verified Compilers for a Multi-Language World article by Amal Ahmed.

What I remember from the Victor Brent talk

(which I saw a long time ago mind you) was better debugging and tests in a very simple way.

One was data tagged with what code produced it. His example was pixels on a screen, but it could just as easily be any data in any variable or array. That's simple. Expensive, but debugging is expensive anyway. - That looked extremely useful, as would be variations on it. I could imagine putting more complex tags on data. Anyway I don't know of any tools that have this. Will EVE?

The other was "live test cases for each line of code (in some cases)," where he had tests coupled with the code they tested and while he edited code, you could see the test case results changing.

That's a bit harder and needs some IDE support, but nothing really hard there...

Debugging in datalog

Debugging in datalog languages is interesting. You can run rules backwards to figure out which inputs were used to derive the data you are looking at, and follow that cause and effect graph all the way back to the beginning of time. Similar in effect to http://www.cs.cmu.edu/~NatProg/whyline.html but without any runtime overhead.

We already execute code live at it is edited, so live test cases are easy. I want to try automatic fuzzing too ie given the integrity constraints specified by the user we can try to generate inputs that break them.