Lambda the Ultimate

I'm a long-time lurker on LtU and just noticed this comment.

I happen to be a cofounder at StreamBase, and one of the people working on StreamSQL design/implementation/standardization. I'd actually be really interested in your experience with this kind of a project. The looping constructs (and of course tables for storage) in StreamSQL make it certainly possible, but how you choose to approach it would be interesting to me. If you go anywhere with it, I'd appreciate if you could drop me a note, tibbetts at innocuous dot org.

You can get a free developer edition of streambase at
http://www.streambase.com/developers/ </shameless-plug>

Incidentally, I am using one stream programming product (not StreamBase, let's say X) in my current project, so I am really curious to hear what LtU people think of this "paradigm". Is stream programming (to you) a paradigm like functional reactive programming (FRP) or concurrent constraint programming (CCP), or is it a special case of FRP or CCP, or is it just a marketing name to any DSL that deals with streams, or the previous narrowed to DSLs with SQL-like syntax? Or am I misguided, and stream programming is unrelated to event stream processing?

When I started using product X, initially it looked as a very nice tool. However, the more I tried to get from it, the more I understood the limitations. For business users, it inherits all the warts of SQL and magnifies them, as the domain is more complex than RDBMS; so only very simple cases are doable declaratively, without resorting to imperative coding. For programmers, it's not powerful enough to code the whole application, so it needs to be integrated into some general-purpose host language. And here the idea to use text-based interface (X-SQL statements) does not work well. I would choose a combinator library over text representation any day (especially if the host language is friendly to embedded DSLs, like Haskell). And even in Java, libraries like Hibernate provide both textual query language and a "query by criteria", which is a simple combinator library.

So the question to people behind stream programming software would be - why these products try to stick to SQL? Is it just a marketing move? Or is SQL perceived as a solid base for building DSLs? Even in this case, why not provide an alternative, programmer-friendly way to access the functionality? (I didn't use StreamBase, so it may well provide this way)

Another suggestion would be to make the approach of the specific software either completely declarative (so the programmer does not have to ensure the query respects causality, for example) or completely low-level imperative (so it can be used as an efficient assembly language by value-adding tools).

BTW, Derek, nice to see you posting again!

I guess my question to Derek was whether he meant Event Stream Processing or Stream Processing (or something completely different) when he mentioned stream programming.

What I meant by stream processing was the concept briefly described in SICP and used and expanded upon in various places, especially in the Haskell world. I didn't realize it was ambiguous.

My original plan was to use lazy lists as streams in Haskell to implement a LC interpreter in a non-trivial way. Just thinking of ways that might work is a bit of a challenge.

And a random thought that passed through my head is applying the AntiObject idea to stream implementation. 'have no idea where that would lead.

Hi Andris,

As the founder of Coral8 (http://www.coral8.com), a company that markets an SQL-based event stream processing engine, I find your post very interesting. I would love to know more about your experience with applying an SQL-based language to your problems, and why you found this experince less than satisfactory.

You write, for example, than only very simple cases can be done declaratively. Could you elaborate, and provide an example of a use case that was hard to implement? Perhaps I can suggest a way to address the problem?

The question of why base event stream processing on SQL comes up frequently. SQL is one of the few universal standards when it gets to programming language. People use SQL to query static data inside an RDBMS, so, the logic goes, it makes sense to use an SQL-based language to process real-time streaming data. In my experience, this approach works reasonably well, and most programmers pick up the new approach pretty easily. This is NOT true for business users, but let's face it, business users don't really use SQL either.

Finally, I'm really intrigued at how one might provide "an alternative, programmer-friendly way to access the functionality". We provide, for example, dynamic interfaces to submit queries via Java, .NET, and C/C++, but I think you may have something else in mind. Again, any idea would be welcome.

Regards,

Mark Tsimelzon
Coral8, Inc.

I would love to know more about your experience with applying an SQL-based language to your problems, and why you found this experince less than satisfactory.

I must warn first that all my posts here represent only my personal opinions.

Having said that, I would separate the issues with SQL-like DSL in two groups: one is expressivity of the language, another is usability of text-based API to DSL.

So, expressivity.
One can write volumes of analysis of problems with SQL for querying relational data. Here I will only mention non-orthogonality, verbosity, and lack of support for hierarchical data (attempts to achieve this by adding XML to SQL notwithstanding). More detailed analysis and an example of the query language that to me seems better designed than SQL can be found here. Note that this research was conducted in early 90's, so there may be even better results available elsewhere.

The main advantage of SQL - its standartization - does not look very important when creating a new product for a new domain - event stream processing. Compatibility-wise it does not give anything, as the language will deviate from SQL anyway, training-wise it may hurt more than it helps as people will think they know what they are doing when in fact they don't, and the only real reason to base a language on SQL is its marketing appeal.

Now turning to text-based aspects.

We provide, for example, dynamic interfaces to submit queries via Java, .NET, and C/C++, but I think you may have something else in mind.

You are correct, I am not talking here about a function in my language of choice that takes a query as a string. Queries as strings work wonderfully for casual users, or very loose integration, or when the queries are few and hardcoded, or when the queries are written by end users. However, there are programs that would like to use event processing engine for heavy lifting, at the same time retaining a lot of control on the queries - generating them on the fly, for example, or distributing them, or splitting and merging them from different sources, or doing all of the above. In formal talk, these programs are doing metaprogramming as they manipulate other programs - queries. It is a well established fact that string-based representation of programs is really underoptimal for metaprogramming for a number of reasons (again, I will not enumerate them in this post).
One representation that I have in mind is similar to internal one used in RDBMS by plan optimizers. Query would be a graph of objects, each of them being a simple operation on streams - filter, window, aggregation, map, cartesian product, source, sink, etc. These nodes specifically are not nodes of AST of parsed SQL, for example.
I understand that real-life software products need to support bindings for multiple languages, so I see the appeal of the textual interface. Also, different companies have different business goals and target customers, so nothing I said is critisism of any product - here I wear my computer scientist's hat, not my software engineer's one, and present theoretical ideas.

I could also elaborate on the causality and referential transparency issues, but I already feel we abused this thread too much. If you feel this discussion is fruitful and relevant to theory of programming languages, we can start a new thread. Alternatively you may contact me by email on abirkmanis at yahoo dot com.

I find this discussion interesting. If you continue the discussion please do it on LtU.

Your points about SQL being a not-very-good query language are well taken. Unfortunately, it is the best one that most people know, and so it is going to be the default choice for anyone who wants to extend a static data query language to a streaming data query langauge.

You might ask why extend a static data query language at all. The reason is that most applications are not purely streaming oriented, they combine streaming and historical data. Stream processing, beyond enabling streaming applications, is a way to make any database that is changing over time an active part of the application, rather than a passive query executor.

One representation that I have in mind is similar to internal one used in RDBMS by plan optimizers. Query would be a graph of objects, each of them being a simple operation on streams - filter, window, aggregation, map, cartesian product, source, sink, etc. These nodes specifically are not nodes of AST of parsed SQL, for example.

Alternative ways to express queries are interesting. StreamBase has two, StreamSQL and EventFlow. EventFlow is similar to the representation you describe, it is a graph of operators, input and output streams (sources and sinks) and data constructs. It is primarily manipulated graphically, but programs could certainly generate it (the intermediate representation is XML).

Other alternative query representations, or other ways of exposing the functionality of our engine, are something we are continuously exploring.

One representation that I have in mind is similar to internal one used in RDBMS by plan optimizers. Query would be a graph of objects, each of them being a simple operation on streams - filter, window, aggregation, map, cartesian product, source, sink, etc. These nodes specifically are not nodes of AST of parsed SQL, for example.

It is interesting that you mention this. I designed such a system as a database abstraction layer for PHP a few years ago (unfortunately not free software). The basic idea was that you wired together these query objects, then executed the query, which would optimize the objects by pushing as much functionality towards the database as possible (hopefully generating a single SQL query, or some operations on a CSV file), then return a result set.

I wonder how to incorporate its concepts in a general-purpose, object-oriented (!) language.

I understood that AntiObjects is an approach to design, not a programming paradigm, so it should be compatible with OOP.

AntiObjects is based on an approach to design (put the intelligence in the environment instead of the agents) but it translates in a specific programming style (evaluate the environment of agents by collaborative diffusion to determine their action).

In the context of an OOP language, the question is how to make this programming style a full citizen of the language and help the programmer with deciding which style to use. Can I call the "programming style" a programming paradigm?

The examples in the paper show a spreadsheet-like structure (the environment) where the agents evolve, diffusing properties ("scent") in the environment. A more general structure would look like interconnected "diffusion cells" with each agent providing diffusion functions to one or more cells.

The paper tries to attack the object paradigm on a quite subtle level: OOP encourages a "syntonic" approach to problem solving. The programmer tries to invent an action model that centers around major intelligent agents ( actors in UseCase diagrams ) while a more promising approach in some cases might be exploiting the actor/environment distinction and put computational power into the environment i.e. making it visible/tactile/tastable to the actors.

But in the end the author objectifies the environment by separating it into objects/agents as well. AntiObjects can be considered again as objects that control their own state and hold references to other AntiObjects and we can ask ourselves: "what would I do if I were a an instance of a cell grid agent". Answer: adapt myself according to other environmental agents that are receptive to the goal agents. So the object paradigm seems not to be replaced but enriched by a further layer reified as foreground/background objects. I would be curious if layers or foreground/background issues could be reified as aspects and whether this adds any benefit?

On a cursory reading of the paper what struck me is the similarity in focus to the concept of field in physics. They're modelling the 'field' rather than the particles, although the particles can be inferred.

I agree with other posts that defines this as a design principle rather than a programming paradigm per se. They've really just made a fat 'maze' class.

The main thing I liked about the paper is the particular solution he came up with rather than the concept they were espousing. That said, he is pointing out a perspective that most programmers would not immediately think of.

Ironically though, it is naturally arrived at via normal OOD and the typical tactic of looking at real-life for inspiration. As soon as you think of the ghosts as "scenting" Pacman, the entire solution, including active tiles, follows from normal object-oriented design. I think the author is partially thrown by how simple the behaviour of the ghost needs to be, but this is not surprising. This is the part that I find most interesting; complex behaviour emerging from many simple local interactions. I, however, do agree that some OOD methodologies would never lead there.

Where can I find discussions and papers on the subject?