Lambda the Ultimate

The Common Lisp reader has a feature known as The Readtable which can be used to configure an ad hoc but interesting assignment, to individual characters, of a syntax type or reader macro.

I'm interested in historic and current experience with these features. I'm especially interested in experience that does not make use of the reader macro feature but that uses the mutability of the "syntax type" of characters in interesting ways.

How good or bad an approach to lexical analysis is this? How general purpose has this approach proved to be in a practical rather than theoretical sense? Has Common Lisp's list of character syntax types been compelling extended?

The more general question I'm contemplating asks what good alternatives there are to regular expressions for specifying lexical syntax. Common Lisp syntax tables have some appeal to me because the character syntax types reflect how a person might describe a lexical syntax to another person: "These characters can begin an identifier. This other, overlapping set of characters can be constituents of an identifier. This is the single-escape character for identifiers."... and so forth.

I'm wondering about the possibility of a system for defining lexers in those kinds of familiar terms.

I've written a short piece, illustrating the use of coroutines to build up UIs compositionally, at my blog.

I'd be curious if anyone here knows of prior research in this direction. I think there is some similarity between this approach, and immediate-mode GUI, discussed previously here. I believe that immediate-mode GUI could be seen as a special case of coroutine UI.