Lambda the Ultimate

Experienced C programmers might scoff at Python as a serious protocol development tool. However low-level work is possible in Python via its struct module, which gives precise bytewise control over data layout, complete with endianness control, and with full platform independence. (The hexlify function from the binascii module helps debugging.) C-to-Python interfacing is also possible, but I found Python's struct module adequate to translate between binary packets and Python-friendly data structures. Twisted also sports low-level access to Internet headers, if you need them.

Of course, I advocate text protocols, not binary ones. Twisted easily supports them. A common category is that of line-oriented command handling (LineReceiver).

Why?

And aslong as we keep textencoding out of the picture, text protocols are system independant, no endian problems for example.

Of course, I advocate text protocols, not binary ones.

Why?

http://www.faqs.org/docs/artu/ch05s01.html

One nasty (but amusing) side-effect is that by running Ethereal on your computer, you typically capture all packets received by your network card. This is bad from a security perspective, because you are running yet another C server program that is processing packets that might otherwise be ignored (dropped by kernel for being malformed or rejected by firewalling). There have been cases of buffer-overruns in ethereal which open up the risk of having your machine cracked by packets that would otherwise be dropped by your kernel.

The moral seems to be the usual one: don't write those programs in C.

An analogous problem is that adding new networking code to your kernel is very easy nowadays. It takes an afternoon to learn enough iptables to do a new form of packet filtering/mangling. Many people are doing this, and unfortunately the standard C programming mistakes they (well, we) make then cause kernel panics instead of a "mere" "Segmentation fault (core dumped)".

The moral I draw from this is that putting networking code in the kernel is usually a premature optimization. Unfortunately, it is much easier to do it this way, at least at present on Linux. And as far as I know the performance implications of doing this in user-space on Linux have not been thoroughly analysed (?).

Hi Frank, the answer to the why question is experience. Text has proven itself superior from almost every point of view, particularly debugging, which is where protocol projects often bog down. Ethereal and similar hex dumpers leave unsolved the problem of human readability. The typical solution is to pretty-print on an ad-hoc, custom basis (per Daniel). So ultimately, one deals with text anyway, and protocols might as well begin as text. Usually the opposition to text emanates from engineers uncomfortable writing parsers or, less frequently, misguided about the nature and merits of data compression.

LtU is a language board, not a protocol board. What do folks think about the Perspective Broker for quote-unquote spreadable computing, and Deferreds? Are we in Oz territory here?

Well, the Erlang people certainly disagree with this. There is an extension for Erlang, UBF, to support binary properly in the language (e.g., with binary pattern matching), and they are quite happy with it. See Appendix C in Joe Armstrong's thesis which explains how to do protocols with UBF.

PVR - I'd rather not dwell on the binary/text debate. It was not the theme of my post. Armstrong's thesis echoes tired old arguments - ease of parsing and text verbosity. The only text alternative he examines is XML, justifying such limited scope with a breezy assertion, "Out of this mess a single universal panacea [XML] has emerged." Well, if the solution space is the set of pre-existing universal panaceas, then UBF has no standing, either. We might be stuck with, say, HDF, the last hurrah of binary do-it-all formats.

Having said all that, I respect Armstrong's effort for what it is, and have known about it for some time. Sadly, I have seen UBF virtually duplicated in half a dozen industrial settings (in the present context, see the Twisted package). Conversely, a few of my most successful projects have entailed simple parsers to facilitate textual transmission of data that was previously binary. It just makes the work go much, much faster, and no one will ever convince me otherwise. I've been there.

I've had a bit of a play with the Twisted Matrix web package. I was really impressed at how easy it was to construct a simple web application. At first I used the framework that they provided, then I had a go at making my own, which was also surprisingly easy.

The component architecture seems very nice. While, I had been familiar with the Adapter pattern, I hadn't been exposed to anything like Twisted's AdapterRegistry before.

The biggest thing I will have to get used to is the fact that the framework is designed to run in a single thread (which I am not used to). However they do seem to provide a lot of support for this style of programming, so I should be ok once I am used to it.