Lambda the Ultimate

The main result: overall, graphics was _significantly_ slower than text, both for novice and experts. Novices were especially confused by visual notation: they don't know where to start looking and how to continue looking.

Perhaps the principal characteristic of visual notation is that it permits more of a _secondary notation_: "the use of layout and perceptual cues (elements such as adjacency, clustering, white space, labeling, and so on) to clarify information (such as structure, function, or relationships) or to give hints to the reader."

Unlike primary notation, the secondary notation is not formalized: you cannot apply it "mechanically". It takes time to learn and to recognize. You can always read text serially. Reading a diagram on the other hand requires an inspection strategy. Reading a diagram requires an ability to recognize abstractions, to understand that shape and placement cue the underlying structure, to distinguish important pictorial elements from inessential. The secondary notation, these _informally_ defined cues, let the expert see what others can't.

"It is worth noting that expertise cannot compensate for everything. Graphics was uniformly slower than text; it is apparent that even the expert reader of graphical notations is doing a hard job. ... One of the distinctions between expert and novice behavior was that the experts made better use of their fingers, occasionally using all ten to mark points along a circuit."

"When comparable graphical and textual representations were presented side-by-side, experienced readers nearly always used the text to guide their reading of the graphics."

Because the secondary notation is not formal, it cannot be understood and applied mechanically. Although it's possible to enumerate "design layout rules," they are often in conflict. It takes a skillful designer to use heuristics and his understanding of the task in order to use, or not to use, a rule. This "feeling for" the rules, an ability to apply or break them is what makes an expert expert.

It's often said that graphical notation makes it easy to "grasp" the overall picture. "Yet in the reading comprehension experiments, programmers did not recognize structural similarities among the graphical representations and often found it difficult to compare two graphical programs. In contrast, those programmers who did notice structural similarities among the programs presented were all looking at the textual representations."

The only positive aspect of visual notation for the beginners is that it _seems_ easier, and it is certainly more fun to play with. But for experts this playing around and finding the most expressive layout with the most secondary cues is an added effort. "'I quite often spend an hour or two just moving boxes and wires around, with no change in functionality, to make it that much more comprehensible when I come back to it.'"

It's paramount to realize that visual notation is just as abstract as the textual one. Both use symbols (and their shape, placement, color, font, etc.) to express or hint at abstract concepts. Visual notation has often bigger alphabet and the bigger set of hints. A symbol for an 'OR' gate is not a metaphor for a chip; The 'OR' diagram is just as abstract as the word 'OR' in the textual description of the corresponding Boolean function. A circuit diagram is most certainly _not_ a drawing of a circuit board.

In conclusion, people indeed see largely what they've learned to look for. Graphical notation _is_ richer -- it gives more information (cues) to the expert. The secondary notation is also easy to misuse and thus confuse the expert readers. The richness of graphical notation has more capacity to 'go wrong'. In any case, the secondary notation is far from being obvious, requires considerable skill to use and to _read_. Novices are precisely the wrong group to benefit from the visual notation as they lack the readership skills (as well as the domain knowledge needed to build abstractions). Novices will benefit from a more structural, constrained, linearized _textual_ notation.

-- Oleg

My views exactly.

It may well be that extant visual programming languages are more difficult to use than textual ones; that has certainly been my experience.

But I still think that graphical languages have the potential to replace textual ones. The summary makes the point that the graphical cues are difficult to interpret because they don't constitute a formal language: this is precisely the problem.

In fact, there are "graphical" languages which are formal, e.g., proof nets in linear logic, Hasse diagrams for posets, commutative diagrams in category theory, term graphs. Each of them makes good use of additional dimensions in a way which textual languages can't. I don't think their notations scale to the sorts of large projects which programmers must deal with, but the spark of a graphical language is there.

An example I always think of in this context is term graphs, which are almost directed trees representing the subexpression relation, but you have a back arrow going from a variable to its binder. They demonstrate how we are forced to replace what is intuitively a one-step process (just follow the arrow) to a complex multiple-step process (read the variable name, then ascend the subexpression tree until you find the variable name in a binding position) because of the inexpressiveness of textual notation.

The problem with the formal graphical notations I mentioned above is that they are all still discrete, and the mapping we use to a computer display (regarded as a continuous 2D quarter plane) is too sparse, so you get a very inefficient use of precious screen real estate. What is needed is a notation which emphasizes the important parts of a program, and whose image looks mostly continuous. Perhaps the domain models of languages are good place to look for this, but I think that we tend to regard the termination properties of programs as less structurally important than their partial correctness, so I think this would be emphasizing the wrong thing.

It seems you're speaking about a primary notation -- which is indeed formal. Which arrows go to which vertices is a primary notation. Let's take an example of a diagram with three objects, A, B and C and two arrows A->B and A->C. We can draw the diagram as follows:

 A --> C         A
 |		/ \
 V             /   \
 B            B     C


Fig. 1         Fig. 2

The secondary notation is present even in textual languages. For example, the following two snippets are equivalent

	while( (c=getc()) != EOF && isspace(c) );

	while( (c=getc()) != EOF && isspace(c) )
	  ;

yet the latter is more comprehensible. To a human reader, not to the compiler. A compiler won't mind if we name all identifiers as i1, i2, i3, ... Well-chosen names however give a human reader a deeper insight into a piece of code. It takes some practice in a programming language to typeset it lucidly. BTW, people often post source code snippets on comp.lang.scheme and similar groups. You can almost instantly spot code posted by a novice -- by the layout of the code. Of course the are "rules" of layout, which are mostly the rules of thumb. Sometimes it's necessary to break them for one reason or another. It takes practice to grasp these reasons. In visual languages, the layout is far reacher, and is far more difficult to formalize. It takes longer to learn and to master.