Lambda the Ultimate

inactiveTopic The Tragedy of the API
started 5/18/2003; 9:47:07 AM - last post 6/22/2003; 8:33:32 AM
Dan Shappir - The Tragedy of the API  blueArrow
5/18/2003; 9:47:07 AM (reads: 2330, responses: 16)
The Tragedy of the API
I perhaps now understand the need for standards bodies more than I ever have before-even though the term gives me willies.

This post is not really about PLs, rather it's about the conflicts surrounding the design (if you can call it that) and evolution of the blogging API (or should I say APIs). I believe, however, that API design is very much related to PL design (one might say that PLs are a type of API), thus this post is educational (or at the very least entertaining).
Posted to general by Dan Shappir on 5/18/03; 9:48:18 AM

Ehud Lamm - Re: The Tragedy of the API  blueArrow
5/18/2003; 10:27:32 AM (reads: 1323, responses: 0)
one might say that PLs are a type of API

I think I know what you have in mind, but to me it makes more sense to say that most APIs are PL based, and thus are dependent on PLs.

Indeed, I think the interesting angle here is how languages influence API design. An obvious example is "inteface" in OOP, but issues like versioning, and concurrency support are also quite well known.

Interestingly enough, in this context, is that the Blogging APIs are supposed to be corss-platform and cross-language tools. This means that they rely on techniques that exist in the lowest common denominator, and that have sensible wire protocols.

Examples include error codes (vs. exceptions) and structs (vs. objects).

One issue we seldom discuss is the introduction of tradtional language features into operating systems, network stacks etc. [We often talk about the oter direction: "language as OS"]

SOAP and XML-RPC can be seen as simple examples of this trend.

Dan Shappir - Re: The Tragedy of the API  blueArrow
5/18/2003; 1:55:13 PM (reads: 1277, responses: 3)
When developing code it often a good idea to develop just what you need and extend functionality via refactoring when the need changes. When developing APIs this approach can be very problematic. Once an API becomes public domain it's pretty much "set in stone" and immune to refactoring. Sure, an API can be designed to be extensible, but that generally means that new functionality can be tacked on, the original functionality must still be supported "as is", and deprecation can be a very long process.

Techniques such as polymorphism, and the FP equivalents, can be applied to squeeze new behavior out of an existing API, but in my experience, there are significant limitations to the changes you can achieve in this way. And in any event, such techniques are often not applicable to wire protocols such as SOAP and XML-RPC.

The same type of limitations, I believe, also apply to programming languages. The initial PL design, the very first architectural decisions made by its creator, are often the most important and can seldom be changed once the PL becomes public domain. And yet, at that stage the PL's creator may have a very incomplete grasp of how and when the PL will be used, and what the requirements from it will be.

Obviously some PLs are more malleable than others (which is why, though I doubt every programmer will use Lisp in a hundred years, I can certainly accept that there will be more Lisp programmers than Java programmers by that time), but still there are certain semantic and syntactic constraints which can't (and shouldn't) be overcome.

Chris - Re: The Tragedy of the API  blueArrow
5/18/2003; 3:07:28 PM (reads: 1286, responses: 2)
re API extension, the Win32 API commonly uses three methods for extending APIs.

1. Copy the old API name, add the needed new parameters, and add an -Ex suffix. Example: CoInitialize() and CoInitializeEx().

2. When creating a new API, include an used reserved parameter for later use. Example: CoInitialize(void *pvReserved). This poses the question. Why was CoInitializeEx(void *pvReserved, DWORD dwCoInit) created when CoInitialize()'s reserved parameter could have been "cashed in" and used for DWORD dwCoInit??

3. When creating a new API, use a struct parameter and include the size of the struct as the first struct member (commonly 'cbSize'). The size of the struct allows API versioning because newer versions of the API will presumably require a struct parameter with additional parameters.

There is a fourth, but undocumented, method used to extend the Win32 API. I used to work at Microsoft on the Windows NT QA team. I know of at least one instance where an existing API was extended for internal Microsoft use. To maintain backwards compatibility, the function signature obviously could not change. One of the API's parameters was a pointer to something. The extended version of the API would interpret the pointer as a two-item array if a certain flag parameter was set, allowing an addition secret parameter for secret new, Microsoft-only API behavior. I don't remember the exact API name, but it was one of the Crypto APIs in CRYPT32.DLL.

Patrick Logan - Re: The Tragedy of the API  blueArrow
5/18/2003; 4:40:36 PM (reads: 1263, responses: 1)
Sam Ruby summarizes in Coping With Change...

Two things required to make a protocol which can withstand the test of time: namespaces and named parameter associations.

Then the API has the ability to evolve, support multiple versions of clients, and not require a client to upgrade to the latest API.

Property lists in Lisp, dictionaries in Python, or XML documents in general fit this bill. Sam predicts more interlanguage APIs will support XML documents explicitly as arguments, just for this reason.

Dan Shappir - Re: The Tragedy of the API  blueArrow
5/19/2003; 12:46:26 AM (reads: 1238, responses: 0)
Nice post. Using XML documents arguments is something I've done in the past, and have found to be something of a double-edged sword. This is because constructing interfaces in this way you risk loosing any contractual information they might contain. Taken to extreme one might create the following generalized COM interface:

interface IDoSomething {
  HRESULT Process([in]BSTR sInXML, [out,retval]BSTR* psOutXML);
};

basically using COM (or DCOM) as a transport for XML (a la HTTP for SOAP). Because this COM interface can be used to activate any functionality, it can no longer be considered a real interface. Open property lists or dictionaries are just different ways to do that same thing.

This is sort of related to the everlasting ST vs. DT debates. In a DT environment a caller can pass whatever he wishes to the callee (this is especially true for extreme DT environments such as JavaScript where the arguments are not a part of the function signature). Because the interface provides so little contractual information its up to the callee to be able to parse and handle most anything that might be thrown its way. This results in a high degree of flexibility but can also be error prone.

A good ST environment OTOH makes easy to properly specify the contract between caller and callee in the type system. At the same time it may result a rigidity that makes extensibility very difficult.

Dan Shappir - Re: The Tragedy of the API  blueArrow
5/19/2003; 12:54:35 AM (reads: 1225, responses: 0)
Copy the old API name, add the needed new parameters, and add an -Ex suffix. Example: CoInitialize() and CoInitializeEx().

That's exactly the problem I was referring to. Making it possible to add new functions is usually simple. The problem is that you must continue to support the original functions.

When creating a new API, include an used reserved parameter for later use.

As an aside, sometimes the propose of reserved parameters is not future extensibility. Instead, the called function pokes values into the stack frame and reuses it to invoke some other function. Sort of a poor-man's continuation.

When creating a new API, use a struct parameter and include the size of the struct as the first struct member

Though I can understand the need for this technique in C, I still hate it. It's so error prone.

Toby Reyelts - Re: The Tragedy of the API  blueArrow
5/19/2003; 8:37:08 AM (reads: 1114, responses: 2)
Ehud:

to me it makes more sense to say that most APIs are PL based, and thus are dependent on PLs.

I don't want to put words in Dan's mouth, but I believe he was trying to express something more along the lines of this wonderful quote from Neel Krishnaswami:

A language is an interface, just like every other API, and you can and should manage it the same way.

I find this quote interesting because it was one of the arguments I once made for using a language to expose a certain set of functionality in our system as opposed to an API.

Dan Shappir - Re: The Tragedy of the API  blueArrow
5/19/2003; 8:51:52 AM (reads: 1145, responses: 1)
Thanks for the quote. This is what I ment.

Patrick Logan - Re: The Tragedy of the API  blueArrow
5/19/2003; 10:12:46 AM (reads: 1093, responses: 0)
...using COM (or DCOM) as a transport for XML (a la HTTP for SOAP). Because this COM interface can be used to activate any functionality, it can no longer be considered a real interface. Open property lists or dictionaries are just different ways to do that same thing.

Yes, this implies there is no "meaning" in the protocol itself. All the meaning is in the contents of the message.

Not a bad thing, per se. Every API is a set of name/value pairs along with a specification of behavior, perhaps with side effects. Specifications (possibly checked statically) describe the meaning in either case. Documents just provide a richer (e.g. nested) name/value mechanism than a simple argument list.

Beyond that, I kind of like the "blackboard" model, esp. that defined by the Linda language. Cooperating processes pass documents back and forth via shared, asynchronous spaces. You can put a document in the space, take a document (that matches a pattern) from the space, or peek to see which documents in the space match a pattern.

Ehud Lamm - Re: The Tragedy of the API  blueArrow
5/19/2003; 11:33:40 AM (reads: 1129, responses: 0)
Yeah, I got that.

I just meant to say that you can look at the same issue for a different perspective. The two points of view aren't contradictory.

Dominic Fox - Re: The Tragedy of the API  blueArrow
5/20/2003; 4:41:35 AM (reads: 939, responses: 0)
In the context of SOAP, the API is partly definable through XML Schema. A valid document fragment, embedded in a SOAP message, can be seen as a single instance of a complex type. It's also possible to extend an API defined in this way through XML Schema's type extension mechanism - or you could define multiple distinct APIs in separate namespaces.

andrew cooke - Capabilities in compilers Was: The Tragedy of the API  blueArrow
5/20/2003; 5:27:24 AM (reads: 966, responses: 2)
So languages should be designed, rather than evolving? But then you'd end up with Ada! (I'm sorry, this is probably only funny if you've been following the tedious "debate" on the ll1 list over the last few days http://www.ai.mit.edu/~gregs/ll1-discuss-archive-html/threads.html).

A more serious question, although possibly completely half-baked. I've just started thinking about capabilities (incidentally, I believe that they (teh suggested implementation in the "oz book") are broken in Oz - http://www.mozart-oz.org/lists/oz-users/4420.html). Could you write a language in which capabilities allowed the system to make certain optimizations? I'm not sure how this would work in detail, but maybe entities within the code would need capabilities to do certain things like modify source (macros, introspection) or change persistent state. Then the runtime system could make appropriate optimizations - you might have the same code compiler several different ways, to be used differently by entities with different capabilites.

This might seem like pointless complexity, but I'm also wondering how far you can take the idea of an extensible language. Is Lisp as good as it gets? Why can't I (perhaps I can) modify how the language binds variables, for example (maybe I want to use declarative variables)? Or maybe I want to change how namespaces are handled. Why can't I get some part of the runtime system and ask how it is implemented, then modify it (Smalltalk goes some way towards this, I think)? If I could do all that, though, how on earth could it be compiled efficiently? Hence my wondering about capabilities...

andrew cooke - Re: The Tragedy of the API  blueArrow
5/20/2003; 5:42:20 AM (reads: 965, responses: 0)
3. When creating a new API, use a struct parameter and include the size of the struct as the first struct member

In the case of an OO language with (minimal) RTTI this is pretty much equivalent to asking for a certain interface and then accepting subclasses for additional functionality (an obvious point, but no-one's mentioned it afaik as another way of allowing interfaces to adapt).

Noel Welsh - Re: Capabilities in compilers Was: The Tragedy of the API  blueArrow
5/20/2003; 3:47:15 PM (reads: 935, responses: 1)
Do you have any references on capabilities, and are they related to the program annotations .Net / Java 1.5 allow?

You might be interested in what's being done in PLT Scheme. It allows you to define you own languages by compiling to a few basic forms (application and definition I believe; I haven't yet looked at it closely). Languages for Algol 60, Python, Java and O'Caml have been defined. You don't have to be this radical - you could tinker with just variable binding as you suggest above.

As for reflecting the runtime you are correct that most (all?) Smalltalk systems do this. In Squeak you can quite happily crash the VM by tinkering with it ;-)

For compilation to be efficient I think you need dynamic recompilation ala Self and probably powerful partial evaluation if you're going to allow the language semantics or runtime to be altered by the programmer. I think all the technology is there but its a mammoth task to bring it all together into a working system.

andrew cooke - Re: Capabilities in compilers Was: The Tragedy of the API  blueArrow
5/20/2003; 6:47:59 PM (reads: 945, responses: 0)
Do you have any references on capabilities, and are they related to the program annotations .Net / Java 1.5 allow?

I don't know anythng about .Net/Java annotations. The little I know about capabilites is that you restrict access to resources. Owning a reference to a resource is equivalent to owning permission to use it. There's a language called E that is based on this idea (but again, I know little about it).

I have no idea how you get from this to considering mutable variables as a resource and so simplifying strictness analysis...

You might be interested in what's being done in PLT Scheme. It allows you to define your own languages by compiling to a few basic forms (application and definition I believe; I haven't yet looked at it closely).

Thanks. I just subscribed to the PLT mail list a day or two ago. I've not looked at the details, but it does seem very interesting. They expose some kind of annotated parse tree, for example.

For compilation to be efficient I think you need dynamic recompilation ala Self

Something else I know nothing about :o)

and probably powerful partial evaluation if you're going to allow the language semantics or runtime to be altered by the programmer. I think all the technology is there but its a mammoth task to bring it all together into a working system.

Yes (at least, as a dynamically typed language). My first hope for Oz was that it did this, since their book talks about extending a kernel language to support different paradigms. However, it turns out that you need to use a separate parser.

So much to learn, and only one lifetime :o)

andrew cooke - Re: The Tragedy of the API  blueArrow
6/22/2003; 8:33:32 AM (reads: 711, responses: 0)
more info on oz and e and capabilities at http://www.eros-os.org/pipermail/e-lang/2003-June/008843.html

there's also these, from a while ago, that maybe never got posted back here: http://www.mozart-oz.org/lists/oz-users/4513.html http://www.mozart-oz.org/lists/oz-users/4512.html http://www.mozart-oz.org/lists/oz-users/4578.html