Lambda the Ultimate

There are many things which I believe really, really, suck when done in 2D ASCII (or, worse, Unicode) files. State machines, for example. So I would at least wish that the state of the industry was that we all had visual tools for subsets of things, and that they all behaved really well together. Magically. E.g. round-tripping.

I don't think that we're 'programmed' by default, to process visual complexity hierarchically. Grammar - that is, expression relating one clause to another in particular ways and with particular degrees - is something we do linguistically, and unless you're someone who's deaf and have learned sign language, you haven't usually learned to process vision (ie, graphics) with the linguistic parts of the brain.

But we've all learned to read written language with the linguistic parts ofthe brain, and we can bring our full powers of grammatical structure to bear on expressions in written form. This allows us to express subtle differences.

Point-and-click interfaces, and most visual languages, are sort of like pointing at things and menu items in a foreign restaurant where you don't speak the language; you can get certain things done, but you'll never be able to express any particular subtle variations on the standard recipe that you want, because you don't have the means of expressing that relationship between ideas without grammar. Or maybe you can, but it's laborious to try to get the staff to understand you.

Weaknesses of visual programming environments:

1. Typically first-order nature: we have boxes and wires, but we cannot easily abstract the wiring.
2. Low density of information: text can easily encode a lot of meaningful data per region compared to most visual representations. If we're not careful, text can encode too much (e.g. APL). Potentially we could find better visual representations, but a viable alternative is to just spread out.
3. High UI contention and overheads: screen space for menus or toolboxes risk covering the content we want to modify, drag and drop or navigation costs rise if we avoid covering the content.
4. Low UI bandwidth: compared to keyboards, a lot of relevant user input for visual programming is lost on modern desktop systems (e.g. second hand, individual fingers, head motion, eye focus).

I believe that a VR or AR environments will eventually address the latter two concerns by providing essentially unlimited screen space and a more natural workspace - e.g. the ability to move your head a little to see behind things (enabling greater density), or to pick something up with each hand. Accepting a little audio input would also be useful, e.g. so we can grab (or filter) tools or components by naming them rather than searching for them visually.

The first order nature demands some variation of visual macros or DSLs, or minimally an easy way to use textual components together with visual components.

With VR and AR, the loss of easy access to a keyboard will likely provide some new pressures to support visual programming. Though, it might be interesting to just whip out a notepad and a pen to use with AR.

While I think there's a lot of disappointments in how current languages use context and allow metacode and modifiers to code, at least we have technology for that.

The only way I can see to write metaprograms in a visual language is to convert it to a textual language or tree representation, modify the alternate representation and find some way to translate back to visual for display.

All visual languages and tools that I have worked with had several big problems:

1. Switch from keyboard to mouse and back was required, so intput and navigation was tricky. This slow down the development process big deal. For rarely modified DSL, it might be ok, but for other places, it is a big problem.
2. All these languages were very hostile to version control tools. In practice merge was not supported at all. The locking was the only way out, but for cooperative development it is not acceptable.
3. There was a lack of ability to work with bad source code and to transition code throw invalid states. For example, when migrating from one library to another, it is often possible to turn off dependency and do search and replace to new names and then fix code basing compilation errors and unit tests. For visiual languages distruptive changes is a big deal.
4. The other big issue is that visual representation is usually projectional, and hides some key information (if not, the visual part becomes almost unberable due to detail overload and formatting problems), one has to drill down throw dialogs and menus to figure out some part that becomes important, but is hidden from representation chosen by tool author.
5. It is just impossible in most tools to quickly examine the entire source code. This makes catching the patterns difficult.

These issues currently make the visual languages acceptable only for niche applications in commercial software development. Small DSL part that is managed by single person is ok. But for bigger applications, the development process does not scale well. BTW demos usually look very nice, particularly to managers.

In flow-based programming, which is a style of coarse-grained dataflow programming similar in level to Unix pipelines, the components are written in textual languages but connected using a visual editor.

For interoperation with standard tools, the graphs are represented by a declarative node-and-edge notation whose details depend on the framework. In JavaFBP/C#FBP it is just a stylized Java/C# class with calls on parent methods to create nodes and edges; in NoFlo it is JSON.

It would be perfectly practical to use a FP-style notation (with keyword arguments) to represent FBP graphs, but this suggestion has had no takeup in the community so far: people are apparently very happy with designing and editing them visually.

What I'd really like is not a visual general purpose language, but rather a lot of specific languages for different domains where visualization and visual editing could help a lot. I'd like a visual language in the form of a tile editor when building levels for a 2D game. I'd like a visual language in the form of Conal Elliott's tangible programming when modeling image transforms. I'd like a visual language in the form of dance notation when developing animations for CGI. I'd like a visual language in the form of conventional graphical user interfaces to unify GUI with live programming.

Today, to have many different visual models requires a lot of painful interfacing between applications, typically via weakly typed filesystems and interpreters, or difficult to maintain code generation.

How can we create a language that can easily host many different visual DSLs, and allow developers to add their own?

I have a few ideas, but they aren't tested. Thomas Lord's older ideas for gesture based programming are inspiring. The 2014 paper on editing functional programs without breaking them (by Amsden, Newton, and Siek) seems potentially relevant.