Expressive lisp ...

As a LtU junkie, I've learnt a lot from the discussions that LtU editors bring up and I've been hacking away trying to implement some "fairly modern" concepts in a scheme-based open-source scripting language muSE. The latest hacks are in the processes branch and include the following -

  • Erlang-style message passing processes
  • Concurrency management using (atomic ...) blocks
  • Resumable exception mechanism
  • Exception handler choice using argument pattern matching
  • Support for guards in patterns

Some basic description of this stuff is available from the muSE blog page. This is public code that's there for anyone interested in exploring lisp-based expressions of such functionality. Comments, suggestions, questions, brick bats, rotten eggs, all are welcome.

Notes: You'll need MS VC++ Express edition in order to build a muSE interpreter. "Solution" files are available from the above mentioned branch. Currently only MS Windows+Intel combination is supported [.. ducking to avoid laser beams! ..]

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Release binaries added.

For those who can't or won't build from source, I've added release binaries to the SVN repository. (62.6kB)
The "with diagnostics" build that provides fairly descriptive messages on error conditions. (42.8kB)
The "release" build with fewer diagnostics.

Release binary for MacOSX PPC

muse-v0.2cp-darwin-ppc.tar.gz (50kB)
Just updated the processes branch with porting instructions for MacOSX PPC. So added a release build for MacOSX PPC users.

Google code has added "Downloads" and "Wiki" facilities!

I've moved the above downloads to the new Google code Downloads page.


You might want to take a look at Termite which is another Scheme based Lisp dialect with Erlang style processes. But allows continuations to be called in a diffrent process than the one that created it. It also alows you to link process together for faulttollerance just like in Erlang. muSE seems to lack this as far as I can tell.

I had the option of

I had the option of implementing continuations to be invokable across process boundaries. In fact, its hardly a line of C code or two to enable that. The semantics weren't very clear to me though and I hope to study it a bit more. I also think that message passing between processes satisfies most of the situations for which cross-process continuations might be useful. So there's an alternative there from a design perspective.

As for linking processes for fault tolerance, I'm considering that but haven't implemented it 'cos its easy to add support for it at the scheme level using closures and the exception mechanism. For example,

(define with-supervisor
  (fn (sup thunk)
    (fn ()
      (try (while (not (thunk))) T) ; Run the thunk till it evaluates to T
      (sup 'SlaveDied)  ; Notify the sup that we died.

Transforms a thunk that you need to evaluate in a separate process into a thunk that will send a message to a given supervisor upon death. You can then spawn a thunk with a supervisor like this -

(spawn (with-supervisor sup thunk))

Anyway, the current implementation in muSE is more like a playground for me to explore these concepts and I thought others might benefit from it as well. As is noted on Termite's post, it takes very little code to implement these things :)

To me invoking a

To me invoking a continuation from process A in process B would mean that B would become a copy of A at the moment the continuation was captured. Except perhaps that it still returns its value to the same point. And I'm not sure if it should copy A's pid or continue using the old pid.

Yes, unless your going into distrubution there it could be a hardvare error in the new process, which wherefor couldn't be handled in that process.

Othervise I must say muSE seems like an nice lisp dialect. Especially using { and } for read time evaluation.

On cross-process

On cross-process continuations, I went through the same thought process and decided that I wasn't clear enough. For example, if you have the continuation invocation complete in the calling process, then the calling process needs to distinguish between a cross process continuation and an in-process continuation invocation. That wasn't a consistent enough model for me. So I've (sort of artificially) forbidden cross-process continuations.

On supervisors, I agree that what I wrote in my post is no substitute for direct exception propagation (as in Erlang). muSE doesn't check unix signals yet as well, to trap hardware errors. So its a bit too early and I make no claim to total fault tolerance.

Thanks for the note on the {} notation. It came about almost by accident, but it has proven to solve many representation and optimization problems rather elegantly.

I don't see why you have to

I don't see why you have to distinguish between the two. Neighter of them would change the return point. You could think of them as delimited continuations, delimited by the process thunk.

Also, you have first class macros. How about other special-form functions? are they first class to?

You might be right there. I

You might be right there. I was stuck at the problem of having to decide which pid to use. The captured continuation might have a reference to the original pid whereas the invoking process will have another. I couldn't figure out (at the time) how to make them all consistent (A or B) when B starts executing like A. Hmmm .. will revisit the topic this weekend. Thanks for the tip.

Are you referring to define, try, case, let, etc? Yes they are all first class as well. Its valid in muSE to do -

(define using let)
(using ((a 10) (b 20))
   (write (+ a b)))

Basically, there are no "reserved symbols".

However, I usually refer to them as "sort of first-class". Though you can play with the symbols as you like (in the manner above), there is a restriction on the constructs that introduce bindings within sub-expressions - the only 3 of them being fn, let and case. When they are used inside closures, they must be known at closure capture time. For example -

(define a 5)
(define b 6)
(define joke
  (fn (mylet)
    (mylet ((a 100) (b 200))
      (write (+ a b)))))
(write joke)

will print a definition of joke that you probably didn't intend -

(fn (mylet)
    (mylet ((5 100)
            (6 200))
           (<prim:18c3> (<prim:d3b> 5 6))))

If you do want to plugin different definitions of let into the body of joke, you could use fn: instead of fn in its definition. That causes joke to be defined using dynamic scoping. If you do that, you get what you want -

(joke let)

will then print.


Yeah thats tricky to. I have

Yeah thats tricky to. I have been thinking about seperating mailboxes from processes, but im not sure of that.

A mailbox is essentially a

A mailbox is essentially a FIFO queue of messages. So there is no difficulty there. I guess your doubt is whether a separate queue implementation is useful (design-wise) when each process comes with its own queue. Am I right?

dont forget patternmatching

Erlang mailboxes have patternmatching to which makes them a bit diffrent from queues. And yeah I implemented a simillar system some time ago so Im pretty clear on the implementation issus. And by seperating mailboxes from process I meant that the processes shouldn't come with there own mailboxes. A recive operation would have to explicitly specify which mailbox it would be listening on. (of course you could still use macros to create processes with an implicit mailbox.)

My reason for seperation, is mostly the erlang pattern of tagging messages by a symbol as in {set, Key, Value} or {get, Key} and patternmatching like:

dictionary(...) ->
        {set, Key, Value} -> ...
        {get, Key}        -> ...

to simulate methods. I thought that this is somewhat weak typewise. Its also not so nice then reificating the mailbox as a closuer, as you do in muSE, as you would have to call it like (dictionary (list 'get key)) instead of (get-dictionery key). but on the other hand you might lose the temporal ordering between set and get instructions.

Pattern matching does make

Pattern matching does make the queue different. I'm not very experienced in this matter, but how useful is the facility to dispatch messages out of order in general? I chose to experiment with FIFO dispatch for the moment in muSE because I wanted to control concurrency using atomic blocks. With out-of-order dispatch, when I do

(atomic (procA 'take 10)
        (procB 'give 10))

there is no guarantee that some other process won't interfere with procA and procB's mailboxes after exiting the atomic block before those messages get processed. With FIFO dispatch, this guarantee is fairly easy to provide.

I didn't understand your comment about "reificating the mailbox as a closure". Can you please explain?

If you want to use processes

If you want to use processes as active objects as in Erlang I think its rather necessary as the process might be in a state there its not ready for a certain kind of message, but expects annother kind of message, and if it would then get a message of the first type it would stall forever, if it has to take care of them in order. And you could still access the messages in order by not patternmatching on them.

But i dont know, you might be using a slightly diffrent concurrency modell so it might not be a problem. Have you tried using muSE for any substansiall concurrency programing?

Not really, but will soon

Not really, but will soon be. It is basically the same concurrency model of another system that I'll be moving over to muSE. The processes I need to support have the property that each can accept any of a set of message types at any time. There is no "sub-protocol" with these processes as they largely work in streaming mode.

Time to do more research I guess.

I checked the meaning of

I checked the meaning of "reify" and now I understand what you mean :)

In muSE, I've generally chosen to avoid passive identifiers whose sole purpose is to, well, identify an object. For many types of objects, there is an important function they serve that makes it worth presenting them as a closure. For example, vectors are functions from indices to objects, hashtables are functions from keys to values, ports are functions whose sole side effect is to transport their argument message across, pids are functions which send their arguments to the process' mailbox as a message. It makes the language feel cleaner. Though there is no attempt to unify ports and processes, the fact that pids and ports are both functions that send messages lets you treat them without worrying about whether you're dealing with in-process communication or across the network.

It makes higher order

It makes higher order functions more powerfull to. I really think more languages should do this.