Lambda the Ultimate

I think I know what you have in mind, but to me it makes more sense to say that most APIs are PL based, and thus are dependent on PLs.

Indeed, I think the interesting angle here is how languages influence API design. An obvious example is "inteface" in OOP, but issues like versioning, and concurrency support are also quite well known.

Interestingly enough, in this context, is that the Blogging APIs are supposed to be corss-platform and cross-language tools. This means that they rely on techniques that exist in the lowest common denominator, and that have sensible wire protocols.

Examples include error codes (vs. exceptions) and structs (vs. objects).

One issue we seldom discuss is the introduction of tradtional language features into operating systems, network stacks etc. [We often talk about the oter direction: "language as OS"]

SOAP and XML-RPC can be seen as simple examples of this trend.

Techniques such as polymorphism, and the FP equivalents, can be applied to squeeze new behavior out of an existing API, but in my experience, there are significant limitations to the changes you can achieve in this way. And in any event, such techniques are often not applicable to wire protocols such as SOAP and XML-RPC.

The same type of limitations, I believe, also apply to programming languages. The initial PL design, the very first architectural decisions made by its creator, are often the most important and can seldom be changed once the PL becomes public domain. And yet, at that stage the PL's creator may have a very incomplete grasp of how and when the PL will be used, and what the requirements from it will be.

Obviously some PLs are more malleable than others (which is why, though I doubt every programmer will use Lisp in a hundred years, I can certainly accept that there will be more Lisp programmers than Java programmers by that time), but still there are certain semantic and syntactic constraints which can't (and shouldn't) be overcome.

1. Copy the old API name, add the needed new parameters, and add an -Ex suffix. Example: CoInitialize() and CoInitializeEx().

2. When creating a new API, include an used reserved parameter for later use. Example: CoInitialize(void *pvReserved). This poses the question. Why was CoInitializeEx(void *pvReserved, DWORD dwCoInit) created when CoInitialize()'s reserved parameter could have been "cashed in" and used for DWORD dwCoInit??

3. When creating a new API, use a struct parameter and include the size of the struct as the first struct member (commonly 'cbSize'). The size of the struct allows API versioning because newer versions of the API will presumably require a struct parameter with additional parameters.

There is a fourth, but undocumented, method used to extend the Win32 API. I used to work at Microsoft on the Windows NT QA team. I know of at least one instance where an existing API was extended for internal Microsoft use. To maintain backwards compatibility, the function signature obviously could not change. One of the API's parameters was a pointer to something. The extended version of the API would interpret the pointer as a two-item array if a certain flag parameter was set, allowing an addition secret parameter for secret new, Microsoft-only API behavior. I don't remember the exact API name, but it was one of the Crypto APIs in CRYPT32.DLL.

Two things required to make a protocol which can withstand the test of time: namespaces and named parameter associations.

Then the API has the ability to evolve, support multiple versions of clients, and not require a client to upgrade to the latest API.

Property lists in Lisp, dictionaries in Python, or XML documents in general fit this bill. Sam predicts more interlanguage APIs will support XML documents explicitly as arguments, just for this reason.

interface IDoSomething {
  HRESULT Process([in]BSTR sInXML, [out,retval]BSTR* psOutXML);
};

basically using COM (or DCOM) as a transport for XML (a la HTTP for SOAP). Because this COM interface can be used to activate any functionality, it can no longer be considered a real interface. Open property lists or dictionaries are just different ways to do that same thing.

This is sort of related to the everlasting ST vs. DT debates. In a DT environment a caller can pass whatever he wishes to the callee (this is especially true for extreme DT environments such as JavaScript where the arguments are not a part of the function signature). Because the interface provides so little contractual information its up to the callee to be able to parse and handle most anything that might be thrown its way. This results in a high degree of flexibility but can also be error prone.

A good ST environment OTOH makes easy to properly specify the contract between caller and callee in the type system. At the same time it may result a rigidity that makes extensibility very difficult.

That's exactly the problem I was referring to. Making it possible to add new functions is usually simple. The problem is that you must continue to support the original functions.

When creating a new API, include an used reserved parameter for later use.

As an aside, sometimes the propose of reserved parameters is not future extensibility. Instead, the called function pokes values into the stack frame and reuses it to invoke some other function. Sort of a poor-man's continuation.

When creating a new API, use a struct parameter and include the size of the struct as the first struct member

Though I can understand the need for this technique in C, I still hate it. It's so error prone.

to me it makes more sense to say that most APIs are PL based, and thus are dependent on PLs.

I don't want to put words in Dan's mouth, but I believe he was trying to express something more along the lines of this wonderful quote from Neel Krishnaswami:

A language is an interface, just like every other API, and you can and should manage it the same way.

I find this quote interesting because it was one of the arguments I once made for using a language to expose a certain set of functionality in our system as opposed to an API.

Yes, this implies there is no "meaning" in the protocol itself. All the meaning is in the contents of the message.

Not a bad thing, per se. Every API is a set of name/value pairs along with a specification of behavior, perhaps with side effects. Specifications (possibly checked statically) describe the meaning in either case. Documents just provide a richer (e.g. nested) name/value mechanism than a simple argument list.

Beyond that, I kind of like the "blackboard" model, esp. that defined by the Linda language. Cooperating processes pass documents back and forth via shared, asynchronous spaces. You can put a document in the space, take a document (that matches a pattern) from the space, or peek to see which documents in the space match a pattern.

I just meant to say that you can look at the same issue for a different perspective. The two points of view aren't contradictory.

A more serious question, although possibly completely half-baked. I've just started thinking about capabilities (incidentally, I believe that they (teh suggested implementation in the "oz book") are broken in Oz - http://www.mozart-oz.org/lists/oz-users/4420.html). Could you write a language in which capabilities allowed the system to make certain optimizations? I'm not sure how this would work in detail, but maybe entities within the code would need capabilities to do certain things like modify source (macros, introspection) or change persistent state. Then the runtime system could make appropriate optimizations - you might have the same code compiler several different ways, to be used differently by entities with different capabilites.

This might seem like pointless complexity, but I'm also wondering how far you can take the idea of an extensible language. Is Lisp as good as it gets? Why can't I (perhaps I can) modify how the language binds variables, for example (maybe I want to use declarative variables)? Or maybe I want to change how namespaces are handled. Why can't I get some part of the runtime system and ask how it is implemented, then modify it (Smalltalk goes some way towards this, I think)? If I could do all that, though, how on earth could it be compiled efficiently? Hence my wondering about capabilities...

In the case of an OO language with (minimal) RTTI this is pretty much equivalent to asking for a certain interface and then accepting subclasses for additional functionality (an obvious point, but no-one's mentioned it afaik as another way of allowing interfaces to adapt).

You might be interested in what's being done in PLT Scheme. It allows you to define you own languages by compiling to a few basic forms (application and definition I believe; I haven't yet looked at it closely). Languages for Algol 60, Python, Java and O'Caml have been defined. You don't have to be this radical - you could tinker with just variable binding as you suggest above.

As for reflecting the runtime you are correct that most (all?) Smalltalk systems do this. In Squeak you can quite happily crash the VM by tinkering with it ;-)

For compilation to be efficient I think you need dynamic recompilation ala Self and probably powerful partial evaluation if you're going to allow the language semantics or runtime to be altered by the programmer. I think all the technology is there but its a mammoth task to bring it all together into a working system.

I don't know anythng about .Net/Java annotations. The little I know about capabilites is that you restrict access to resources. Owning a reference to a resource is equivalent to owning permission to use it. There's a language called E that is based on this idea (but again, I know little about it).

I have no idea how you get from this to considering mutable variables as a resource and so simplifying strictness analysis...

You might be interested in what's being done in PLT Scheme. It allows you to define your own languages by compiling to a few basic forms (application and definition I believe; I haven't yet looked at it closely).

Thanks. I just subscribed to the PLT mail list a day or two ago. I've not looked at the details, but it does seem very interesting. They expose some kind of annotated parse tree, for example.

For compilation to be efficient I think you need dynamic recompilation ala Self

Something else I know nothing about :o)

and probably powerful partial evaluation if you're going to allow the language semantics or runtime to be altered by the programmer. I think all the technology is there but its a mammoth task to bring it all together into a working system.

Yes (at least, as a dynamically typed language). My first hope for Oz was that it did this, since their book talks about extending a kernel language to support different paradigms. However, it turns out that you need to use a separate parser.

So much to learn, and only one lifetime :o)

there's also these, from a while ago, that maybe never got posted back here: http://www.mozart-oz.org/lists/oz-users/4513.html http://www.mozart-oz.org/lists/oz-users/4512.html http://www.mozart-oz.org/lists/oz-users/4578.html