Computer music: a bastion of interactive visual dataflow languages

The area of computer music has been designing systems for aiding music composition and performance for over 40 years now. The developers have not been much influenced by mainstream programming fads, but have been driven mainly by the needs of composers and performers. Current systems keep the principal original metaphor, the patchboard: a collection of live modules ("patches") connected by cables and easily inspected and modified on the fly. This metaphor has full GUI support and is extended with interactive visual tools for abstraction and exploration, such as the maquette. The language semantics are based on deterministic concurrency, with controlled use of nondeterminism and state.

Current systems are full-fledged programming platforms with visual dataflow languages. Two examples among many are the Max/MSP platform for controlling and generating music performances in real time and the OpenMusic design studio for composers. Max/MSP has two visual dataflow languages: one for control and one for real-time waveform generation. OpenMusic has a dataflow language controlled interactively with many tools for composers.

These systems are actually general-purpose programming platforms. They show that visual dataflow can be made both practical and scalable. In my view, this is one promising direction for the future of programming.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Good as DSLs, not in general

These systems are actually general-purpose programming platforms. They show that visual dataflow can be made both practical and scalable. In my view, this is one promising direction for the future of programming.

They are general purpose in a Turing completeness sense, and they can be nice for what they've been designed to do, but if these tools aren't much different from the visual dataflow tools I've used, there will be a large subset of problems for which it will be painful to use them. I agree that the idea is more general than just music and might be usefully deployed as a general tool, but not as the only tool.

What is a general tool?

Replace "visual dataflow" by "object-oriented" or just about any other paradigm and your statement is still true. What other visual dataflow paradigms have you used? Do they handle state and nondeterminism well? Are they higher-order?

What is a paradigm?

I haven't used higher-order or stateful visual tools, but I can imagine how you'd go about making those additions. I still don't think that doing everything "visually" is a good idea - encoding something as simple as "(-b + s*sqrt(b^2-4ac))/ 2a" becomes quite annoying with boxes and wires. And I don't think doing everything with "dataflow" is a good idea. Actually, this makes it sound like you agree:

Replace "visual dataflow" by "object-oriented" or just about any other paradigm and your statement is still true.

The way I see it, most of the "paradigms" that are baked into languages can and should instead be modeled functionally in a single core language. This meta-paradigm of first crafting an architecture that defines a custom "paradigm" against which the rest of the code is written seems more promising as a general approach than a language that presents a small handful of paradigms to present to the programmer.

Dataflow is just one paradigm

Yes, I agree with you. A language like Max/MSP or OpenMusic just changes the default paradigm (the one the others are built on). It is a good default for music, since the GUI and the program are almost identical (you can see immediately what control knobs do and change it if it's not right). Equations like "(-b + s*sqrt(b^2-4ac))/ 2a" aren't programmed in dataflow, they are defined as text in their own boxes. The behavior of Max/MSP is similar to functional reactive programming, except that glitches are avoided by programmer convention and not by the system (this makes the implementation *much* simpler). OpenMusic is even more multiparadigm: it is built on top of Common Lisp and any Lisp function can be used in the dataflow layer.

Why does that make the

Why does that make the implementation much simpler?

The lack of strong semantics, such as synchrony / glitch prevention, probably leads to *more* complex implementations once you want to start optimizing things. In the case of Max/MSP, they already hit this wall. E.g., to add parallelism, they're adding an explicit parallel polyphony component, instead of taking advantage of the overall data flow model.

optimistic about frp's future pay-back

i continue to try to grok the evolution of frp (pre-arrows, arrows, post-arrows, etc.) and trust that some day it will have sort of come full circle in something of a zen like way, such that we can all use it to do seemingly imperative things in really nice clean safer ways, and we can all thumb our noses at plain old unconstrained imperative badness. (of course, by then maybe the Max people will have retired on their own private island with all the money they've been making in the meanwhile. d'oh.)

Max/MSP does not enforce determinism

In the case of Max/MSP, they already hit this wall.

Yes, you're right. If the implementation would enforce a stronger semantics, programs would be simpler and less prone to strange behavior. I recently showed a number of expert Max/MSP programmers at IRCAM how easy it was to create glitches in Max/MSP, and their reaction was "Oh, but your program has an error! No Max programmer would do things that way!". The Max/MSP programmer community has assimilated its limitations so they do not feel them as limitations. But Max/MSP programs occasionally have strange behavior that remains unexplained.

It's an impressively product

It's an impressively product environment, but they are cognizant of oddities (and subtleties of use) like this. They even practice (manual) boilerplate approaches to get around it.

Wall?

To hit a wall with respect to optimization, optimization would have to be the goal. Parallelism is managed with an explicit polyphony component because they're typically pairing off note/gate events to instances of a monophonic synthesizer. There's no way that "taking advantage of the overall data flow model" would help in this respect since explicit instantiation is exactly what they're after.

The hope was that they could

The hope was that they could get the parallelism from the runtime. They hit a complexity wall -- perhaps it was due to legacy issues, and not the language (but the ad-hoc form of determinism in the language mentioned earlier makes me skeptical of this). Not being able to extract parallelism from computationally intensive dataflow music language suggests something went wrong, however.

functor in vvvv

I agree that the biggest problem with the visual dataflow languages is lack of higher-order function support.

I can't speak for Max, but I have used one called vvvv for video. It was interesting in that it allowed lists to pass through wires wherever single values could and then with each function it would do a map and interpolate (if you sent a function multiple parameters with different list lengths).

This simple use of a functor along with the convenience of real-time response to changes (best thing about these visual dataflow languages) gave the program a significant amount of artistic power.

This is the relevant documentation for this feature (called "spreads"):
http://vvvv.org/tiki-index.php?page=Tutorial+Spreads
http://vvvv.org/tiki-index.php?page=Tutorial+SpreadsII

Already possible

It is already possible to do higher order programming in Max. "Modules" can take arguments and those arguments can be names of other modules. Using this, you can write a module that, given a name, instantiates the module within itself and connects it to its inputs and outputs. You'll likely want one version of this "instantiator" module for every reasonable combination of input and output port numbers. It's not pretty, but it works.

Higher-order in liquidsoap

Let me share our experiment with liquidsoap (savonet.sf.net), a language which targets audio, but at a higher level (it's about net radios more than sound synthesis).

We started with a textual, purely applicative language. At this stage it was only about building a DAG, and could have been done through a GUI. However, we found text convenient (this might be related to the fact that in liquidsoap, you write a conf and run it for days or months, whereas max & others let you interactively tweak your conf).

At some point we realized that we wanted transitions, e.g., between two tracks, fade out, insert a jingle, fade in. A general and elegant way to do it was to add functions to our language: our cross() operator takes a (source,source)->source function (source is like stream). At this point the graphical intuition doesn't work anymore.

Lots of choices

This is a really vibrant space right now, there are tons of options to explore:

Pure Data was a rewrite of Max/MSP in 1996 made by the original author of MSP (Miller S. Puckette) that is still being actively developed and supported by a community of thousands: http://puredata.info/ It is free and cross platform.

ChucK is another free dataflow media language developed at Princeton: http://chuck.cs.princeton.edu/

Supercollider was originally a Mac OS 9 only commercial written language that bears very strong similarity to a visual dataflow paradigm, while being strictly a written language. It was open sourced and is now community supported, free and cross platform: http://supercollider.sourceforge.net/

On a higher level (less general purpose) you also have music environments such as http://beast.gtk.org/ and http://buzztard.org

general programming with visual dataflow

I don't know if LabVIEW counts, but the folks at http://www.lilyapp.org/ are trying to use visual data flow for the web, which I would claim covers a lot of general programs. Instead of speculating, perhaps try it out :)

It doesn't (currently?) fit for a lot of things. The folks on the Ptolemy project have explored a lot of domains -- browsing through their demos should provide an intuition towards both hits and misses.

Ptolemy II (at Berkley) has take this very far theoretically

See this:

http://ptolemy.eecs.berkeley.edu/ptolemyII/summary.htm

A good overview:

http://www.eecs.berkeley.edu/Pubs/TechRpts/2008/EECS-2008-28.pdf

I believe it is has a strong future in concurrent programming.
Ptolemy II is very mature research tool.

Re: mature

The Ptolemy project is so tantelizing. I want it to somehow magically transition from a mature research tool to something of industrial robustness (whatever that subjective might mean), if only so more people could become aware of it.

Ptolemy II misrepresentation of Esterel model?

Ptolemy II does look impressive at first glance. But I am puzzled because they seem to misrepresent Esterel's synchronous model. In the Ptolemy II documentation, the Synchronous/Reactive model (SR) is identified with Esterel. They say (on page 23):

Moreover, in most realizations, modularity is compromised by the need to seek a global fixed point at each clock tick.

This is not true of Esterel: there is no one global clock signal, and only local fixed points are needed among the signals involved in each transition. Esterel is completely modular.

Yes and No

I agree completely that there are some interesting programming-language angles on digital art right now, but it's probably not centered on data-flow languages.

There's both less and more than meets the eye there. "Less" mainly because these data-flow environments are all pretty compromised by the field's need to have tools that are quickly teachable to non-programmers with little math. Ultimately Max/MSP and Pure Data end up embodying all the limits of the pure-dataflow, pure-GUI approach to programming. I actually disagree that these things show that data-flow can scale well — the longevity here shows that it remains easy to teach and that's very important in a country where most artists teach. Putting objects rather than audio signals through the edges of the graph — typed or not, you can have your choice with various extensions — doesn't seem to change as much as you'd initially expect. You are still in a language/environment where both iteration and recursion are fundamentally hard to express.

For all of the vibrant energy you'd expect in digital art, this is actually quite a static place — you can see the debate "raging" in the Computer Music Journal in '93, between LISP hackers, C hackers and at least one Forth hacker over what Max is doing / going to do to their world. Last I heard that MacArthur award-winning Forth hacker, George Lewis, was teaching Max/MSP based course at Columbia.

But there's also more than meets the eye, because most people who are using these tools know these things, so the tools themselves are constantly trying to hybridize with other things to compensate. Thus Max/MSP has MXJ~ — embedded Java in a ClassLoader sandbox — and thus various JVM language implementations through that. It's now a data-flow graph with Python, for example, at the vertices. And OpenMusic is after all just a slightly terrifying CLOS GUI — press the right button and you really are writing LISP.

The reason that there's at least a little more lambda-the-ultimate-ness in digital art now is that people have been shifting back towards the text-based — ChucK, Java (via Processing), ActionScript and even C++ and so on. But these haven't yet been accompanied by particularly interesting environments yet. In my group's work we've started in this corner — a text based language in a text editor — and walked back towards the middle. We now have a (recently open sourced) environment for writing code in various JVM-supported languages that both executes-to-make-art and executes-to-make-the-environment — so if you want a data-flow GUI, you just get on and make one in your "editor".

I'd like to think that we'll see interesting programming language / programming environment pairings emerge in the next few years because of pressure from the digital art community, but I don't any rapid changes.

not sure i follow

You are still in a language/environment where both iteration and recursion are fundamentally hard to express.

i witnessed a demo of LabVIEW once where iirc (my memory is fuzzy and I don't use the product myself) iteration/recursion were easily done by having a type of box connector that went from output back to input on the same box.

There are a few variations

There are a few variations of getting around this in visually-compatible ways.

You can get recursive forms with Haskell-style FRP programs. Ptolemy has some languages that go further, letting you specify stuff like differential equations. On the other end, a lot of systems provide a visual macro language and/or lazy component instantiation.

quartz composer

I'd like to think that we'll see interesting programming language / programming environment pairings emerge in the next few years because of pressure from the digital art community, but I don't any rapid changes.

There is some progress already. Apple's Quartz Composer has support for a box in which you can key in a JavaScript function. It also has another box for keying in image processing kernels in C-like code. You can connect output pins to input pins to create feedback loops (essentially recursion). Macro patches can iterate other patches.

Btw this also came up in a different thread context.

I actually disagree somewhat

I actually disagree somewhat with the "easy to teach" comment. In most cases, I find that non-programmers have an easier time with Processing than they do Max. The problem with Max is that some easy things are easy, but other easy things are quite difficult due to the dataflow approach. Furthermore, many aspects of the language are downright silly. For example, simply moving a module on the 2D plane can result in your patch no longer working properly.

Max position conventions

simply moving a module on the 2D plane can result in your patch no longer working properly

There's a reason for that. It's because of the Max conventions on execution order, which are important to ensure deterministic behavior. So if you want to dismiss this as "silly", you have to address the behavioral issue too!

See the layout rules as Max "syntax"

In order to get a program to fail or not even compile in haskell, all you have to do is to simply move a function or even just change the indentation of an expression. This problem with textual syntax in logo is one of the reasons behind creating scratch as "the next logo".

It's "silly" simply because

It's "silly" simply because one output should not be allowed to be connected to two inputs in the first place. It would be simple to enforce that an appropriate trigger object be used instead negating the whole issue.

I'm not as convinced on the

I'm not as convinced on the one output / two input issue.

Not being able to safely move a component around is a usability problem in a tangible, visual language. Tying scheduler semantics to this was an odd choice. Other schemes, in the presence of cycles, may also be odd, but I refuse to believe this was the right choice (e.g., Esterel gets around this).

I'm not sure I follow.

I'm not sure I follow. Requiring a trigger object would eliminate the problem related to moving components around. If programs should be dependent on the order of operations in the first place is another question.

real-time feedback

It's often harder to do something if you already know what you want to do in these visual programming environments.

But it's way easier to experiment, due to the real-time feedback, which is why I think they're popular (and why electronic musicians still love the plugging in wires paradigm).

vvvv, visual languages, and concurrency

why the lucky stiff wrote a bit about this in Ctrl+B For Concurrency:

"One profound change of mind for me is this: I haven’t given visual languages a fair shake. These languages (such as Max/MSP and vvvv) aren’t just good languages. To watch someone code fluently on a canvas just looks completely natural. And, particularly in the case of vvvv, you have a language which moves effortlessly between visual and textual."

on fads ..

The developers have not been much influenced by mainstream programming fads, but have been driven mainly by the needs of composers and performers.

The fads may not be mainstream prog langs, but the field has its own fad history :) Chuck, for example, seems to be a recent entrant.

.. and I think many tool creations have been based on the need or urge of (more or less) only one composer or performer who is also savvy with programming .. at which point the creator goes through a transition. One of my favourites is Dave Zicarelli's (one of Max creators) comment that he lost interest in composing once he started developing a tool for the kinds of compositions he wanted to do. (can't find the page ref now).

you've excluded a whole class of computer music environments

"Current systems are full-fledged programming platforms with visual dataflow languages."
This statement excludes quite a number of systems used by computer music composers, including: Nyquist, SuperCollider, Common Music, Chuck, Siren.

All of these embody data flow in some way since that is the nature of signal processing and control, but they are not visual environments. Nyquist and SuperCollider build data flow graphs functionally which is IMHO a more powerful way of working than wiring together concrete boxes.

SuperCollider is very much influenced by certain trends in computer science, incorporating both object oriented and functional features.

HCI: Reactable

I didn't read all the posts in this thread.

You need a good interface to make life real-time music. I really liked this: http://www.reactable.com/. Check out the life performance.

Visual Dataflow is Commonplace in Graphics

ICE, Shake, Houdini, Cineon, Khoros and Nuke for example. Not just for commercially available tools. For many graphics related companies visual dataflow is practically the default design for in-house proprietary tools. In many cases there is convertbility with a text based language, eg. Shake's scripting language. In some cases the underlying software uses a dataflow model even though the default user interface doesn't involve dataflow, eg. Maya.

Impromptu

What do you think of Impromptu? It is based on tinyScheme, I think it looks very cool.

Another way to control signal flow

My impression is that composers and musicians use graphical programming tools
like Max because it is easy to learn, and that it's a fast way to see the signal flow.

In text-based music programming languages, I would say that:

1. It's inconvenient to define the signal flow. You have to write code to create
audio buses and also define which sound objects sends to the audio buses, and
which sound objects recieves from the audio buses.

2. It's hard to see, by looking at the code, the signal flow. You don't immediately
see a graphical line connecting sound objects.

In my music programming system called snd-rt, I have tried to make these two
problems less of a problem by using dynamic scoping to automatically
take care audio buses plus connecting sound objects to the buses.

For example, the following code implements a minimal polyphonic midi synthesizer:

(define-stalin (midi-synth)
 (while #t
   (wait-midi :command note-on
     (define phase 0.0)
     (define phase-inc (hz->radians (midi->hz (midi-note))))
     (define tone (sound
                    (out (* (midi-vol) (sin phase)))
                    (inc! phase phase-inc)))
     (spawn
       (wait-midi :command note-off :note (midi-note)
         (stop tone))))))

And the following code implements a volume controller:

(define-stalin (volume sound vol)
  (* sound vol)))

The following code connects the output of the "midi-synth" sound object to
the input of the "volume" sound object, and makes sound:

(<rt-stalin>
 (sound
   (out (volume (in (midi-synth))
                 0.5))))) ;; 0.5 is the volume.

The "in" operator in the above code created a bus which was connected to the
output of the "out" operator in the "midi-synth" function. The output of the "out"
operator in block above was not called from inside an "in" operator, and therefore
this sound was sent to the soundcard.

Structure and Behavior

I haven't worked with the music languages under discussion at all. However, my experience with tools like Simulink and Ptolemy has been that structural properties (connections between components) are more easily expressed and understood using diagrams, while behavioral properties (what a component does) are often more easily expressed in text. Certainly you can express structure in text -- that's what everything from your language, through the various FRP systems, to nesC do. But in my opinion it's a lot easier to work with structure by manipulating lines and blocks. Similarly, you can express complex behaviors visually, but it's often easier to do so in text. I can recall several Simulink modelling experiences where I started trying to build a complex component behavior by diagrammatically composing simpler parts in complicated ways. The resulting diagram was difficult to understand and hard to debug. I eventually broke down and rewrote the whole thing as an S-function block. Ideally, I'd like a language to allow me to form new components using either text descriptions of behavior or as a structural composition of other components -- whichever works best for the task at hand.

What is the meaning of

What is the meaning of "structure" here? "Connections between components" doesn't seem clearly distinguishable from behaviour to me, unless all the components in question conform to some sort of standard interface (like logic gates).

Perhaps a graphical interface to monads would make them easy for the masses to use? :-)

Components and structure

I don't think it's necessary to have a single logic-gate-like standard interface (Ptolemy II is an excellent example of heterogeneous component composition). I suppose that what defines a "component" in the sense I meant above is something that has input and output ports, and specifies some transformation from inputs to outputs (the behavior of the component). We are, after all, speaking of dataflow languages. Or, in the case of Simulink, signal-flow languages.

Of course, one can create a complex behavior by building a structure composed of simpler behaviors. So in that sense you're correct that the concepts are not "clearly distinguishable". But sometimes it's esier to decribe the behavior you want as an textually described algorithm instead of as a visual composition of primitives. That's why I'd like to have a language that allows complex behavior to be built both ways.

And yes, a graphical interface to monads might help... although my impression is that the problem most people have with monads isn't so much using them, as figuring out when and how to construct them.

I find it to be domain

I find it to be domain dependent. For example, textual descriptions of animations are hard to deal with. Seeing a path, destination, etc. are useful. Not to say graphical representations are necessarily easy to deal with ;-)

Sometimes, I want the leaves of computations to be textual -- e.g., arithmetic -- and higher level manipulations to be graphical -- e.g., the music circuit. Othertimes, I want the leaf to be graphical -- a teen -- and the composed orchestration textual. Sometimes, I want to be able to slip in and out of both modes. There isn't a system that lets me comfortably do both -- I suspect a more restricted and library aware version of FRP is the best next bet -- but I'll keep searching :)

Computer music languages

Reality check: one of the most widely used computer music languages is Csound, which is a simple assembler like language without any features of interest to computer scientists. Csound also happens to be arguably the most powerful of these systems because it has a longer history and has built up a larger library of unit generators.

Comment on data flow languages: one reason for these languages is that the close fit between the data flow paradigm and the problem domain means that simple implementations can run efficiently. This is as true of Csound or the SuperCollider synthesizer as it is of visual languages like Max/MSP. This efficiency has been and continues to be important.

It's probably worth

It's probably worth mentioning Michael is credited on the Csound wiki page.

1. I suspect Max has more patches. E.g., look at, http://www.maxobjects.com/?request=a&operateur=AND&id_format=0&id_plateforme=0&Submit=OK . 261 pages of lists of patches with the letter 'a', with each paging showing 15 objects.

2. I agree about performance. Interestingly, and I've written this here before, the dataflow model is very appealing performance-wise because of multi/manycore opportunities, which are finally starting to be explored.

2. I agree about

2. I agree about performance. Interestingly, and I've written this here before, the dataflow model is very appealing performance-wise because of multi/manycore opportunities, which are finally starting to be explored.

Any recent work to point us at? I've seen claims about how data-flow will scale better across many cores, but I haven't seen any language that follows through on the claim. Still, I'd love to see it, but I won't let the claim by without evidence.

Claims are always made about how functional programming languages/actors/... will scale to many cores, but then there is
no fast many core version of Haskell/Erlang/Scala to show off... Many/multi-core itself with unrestricted memory models might be a dead end in of itself, and GPU/restricted memory model processing (e.g., Larabee) might be more promising. Here data-flow shows promise as it can match how textures are transformed and referenced on the GPU.

I agree about

I agree about Erlang/Haskell/Scala. Bryan Sullivan had a wonderful Bay FP talk about Realworld / Parallel Haskell saying essentially the same thing.

For parallel dataflow.. what about Labview? I haven't played with it, but they seem to be getting success with parallelization. Caveat: the demos I have seen seem very write-once, though my FRP background may bias me.

Manycore seems biased towards simpler cores, at least from the architecture guys I've talk to (as opposed to, say, most speculative architectures) -- Larabee is an example of that. Furthermore, a lot of the proposed 'manycore' OSs focus on partitioning more than just CPU (e.g., network or bus), which seems in line with dataflow methodologies. My concern is how it scales from controlling machines (labview) to applications (Flapjax) -- I've been focusing on core libraries this year and will start to poke at it next year once those are ready :)

Btw, I'll be at MSR Redmond this summer with Ben Livshits and browser or security folks -- if you ever visit, I'd like to meet you :)

Why do they conflate Visual and Dataflow?

Labview and Pure Data really, really bug me, because:

1. On the one hand, they're doing dataflow, and I think a pure-dataflow language has some huge advantages for modelling interactive, real-time systems

2. On the other hand, they go out of their way to cripple the expressiveness of their language and confuse the issue by making their language *visual* rather than textual. Drawing boxes and lines in a GUI.

It's not just limited to these two languages. Almost everywhere I look, when I look for 'dataflow' languages what it seems people think that what I mean by that is 'visual', and usually once they think 'visual' they go on to decide that this implies 'made for non-programmers' and then remove all the interesting/useful language features which would make such a language worthwhile for real work.

But I don't want a visual language, at all, and I don't understand why anyone else would.

Visual means I have to assume a whole pile of GUI cruft which is intensely platform-specific, before I can even start writing code. Visual means no cleanly-defined 7-bit clean serialisation. Visual means being dependent on a manufacturer's proprietary editing tool or a non-portable desktop environment. Visual often means the designer thinks they have to write a custom widget set, from scratch.

This, to me, is the opposite of how a language should be designed. You start with the semantics, then you build a clean serialisation syntax, then you write an interpreter/compiler/REPL, then you write an environment, then you write an editor mode, then.... if you really must, and only if you're doing GUI work that would require it... you write a GUI development shell on top of all the other layers.

So where can I find the real, *textual*, Turing-complete dataflow languages?

I don't mean 'a functional reactive toolkit for Common Lisp or Javascript' (like Cells or Flapjax). That's a good start, but it's not a language, it's a library, and suffers from the 'not written with this in mind' problem.

Any takers? Because I'm tinkering with writing my own concatenative-dataflow language, and while it's fun, I'd much rather use someone else's work if I can.

Its been done. Still, you

Its been done. Still, you put yourself in a straight jacket with dataflow, even when text based. Text excels at abstraction, so the trick is to throw some abstraction in the mix.

Lucid

Have you looked at Lucid?