## Languages ready for API Evolution

When I was describing my API design adventures in the Practical API Design book (btw. if there is anyone who would like to review the book for LtU, I am ready to provide a copy), I could not realize that many of these problems would not even appear if we had better languages, or systems more suitable for the DistributedDevelopment.

I may be wrong, but I do not know about any language that would support modularity. And here I mean not compilation piece by piece, but also modularity in time. Because that is the kind of modularity that is needed quite often during development of APIs for our libraries.

At one point of time we release a version of a library and distribute it to others by publishing it on the website. Now people unknown to us, distributed all around the world download it and use it. As a result, the amount of applications built on top of such library is increasing, which implies also that the amount of bugs and feature requests is growing too. After a while the time to release new version of the library comes. However what if the first version used to have a class:

public abstract class View {
public abstract String getDisplayName();
}

What if one of the feature requests demands to add an HTML title to each view. Can we change the view class to following form:

public abstract class View {
public abstract String getDisplayName();
public abstract String getHTMLTitle();
}

Indeed, this cannot be done. Existing subclasses of View would no longer compile, because the change is not source compatible. Also, even if someone links already compiled subclass with the new version of the library, the Java virtual machine will complain and throw some kind of linkage error, as definition of an abstract super class method is missing.

I would love to see a language or system that would fail the compilation whenever I want to modify my already released classes in a style that could prevent some of their users to continue to use them in previously working way. This would be the real modularity, which is ready for changes in time. So far it seems to me that the current languages do not deal with changes in time really well. Just like Andrei described in his Enums in APIs blog, it seems that our languages do not think about the process of API evolution much and include constructs that one needs to avoid to make DistributedDevelopment possible.

Releasing new version of libraries with modified APIs is really common and our software engineering practices cannot live without it, yet it seems that there is no language in the world that would make this easy and clueless. Or am I wrong?

## Comment viewing options

### Do you mean versioning?

Do you mean versioning?

### Library versioning

With library versioning, you can make sure that the API client application is using a consistent set of classes and interfaces. Is that not one feature of the Microsoft CLR? (maybe not, a quick search did not give me the expected results)

You could go one step further and design a compatible API extension: in your example, it does not really matter if there is one more method in View, as long as either

• The method has a default implementation
• The method is not used with the class which was developed before the API change

The second will be true in a self-contained application, it would likely be false in a library or reusable component.

### Versioning seems to be necessary part of evolution

Annotating various library revisions with version number seems really necessary for purposes of distributed development. Also when using a library, one shall not specify just the library itself, but also its version or version range. This is somehow present in existing modular and packaging systems (NetBeans, OSGi, RPM, etc.), however language designers seem to be reluctant to include this into any language as first class citizen. All I can see is import javax.swing.JTree statements, and not import javax.swing.JTree version 1.3 or greater.

Java VM as well as CLR have their rules for compatibility on binary level. And there are external tools that can verify them (apitest, for example). However no language I know comes with such support for evolution itself.

### Assembly versioning

Is that not one feature of the Microsoft CLR? (maybe not, a quick search did not give me the expected results)

Microsoft calls it "assembly versioning".

### scoping problem not language problem?

Coincidentally I wrote about this topic tonight in a page I'm posting later. (I'd include a link here, but it contains profanity, and the section on server/language versioning isn't large enough to warrant reading the majority of the page.)

I think you want to show a consistent view of code to be compiled together. If your users can download and then mix and match inconsistent versions, I'm not sure how you can solve that distributed development problem without giving them tools they can run locally to duplicate the desired distributed consistent view.

I'm basically planning to address that problem inside the scope of a single server site, so you can connect to the same site as other users and yet see a different self-consistent version of the server software because the "mount path" controlling symbol visibility would be different in your view. (This model allows you to debug updates to a server live without subjecting other users to your changes.)

But what I'm describing here is design only. I don't know anyone else who talks about this. I've been tossing it at folks for a couple years now.

I don't think the language involved matters a great deal. But it would help to have a tree-shaped module system for symbols to permit tree-patching and copy-on-write versioning for alternative simultaneous views. I'm aiming for Lisp right now, but I expect it to work equally well for any other language built on a similar infrastructure.

I can't point you at working code, so there's no proof of concept this will work. It might be too much painful detail to achieve in practice even if it sounds clear in theory.

It looks to me, that you want to achieve reproducibility. Ability to compile and get the same bits as someone who compiled a year ago, or ability to configure the runtime with exactly the same environment as done before. I believe that for example Maven with its on-line repositories is system that can guarantee that. You are right that for this kind of snapshotting there is no need for any language changes.

However I am more interested in upgrades. From time to time I want to upgrade one library used on my server. Either to fix bugs or get new features. At this point of time, I'd like to be sure that the system does not destabilize, that the new library version is at least as good as the old version used to be. That there are no missing symbols, the whole application that I compiled previously can still be compiled, run, and so on.

This can, with a bit of carefulness, be done for Java, without any language features, however as I confess in the Practical API Design, it requires a lot of self control, awareness of too much compatibility rules, bunch of support tools, etc. My feeling is that with languages designed with the API evolution in mind, this could be task achievable with much more cluelessness - task approachable to wider audience of developers.

I believe this kind language feature is needed and I am still hoping some readers of LtU will just point me to a language or system and say: Hey, look, this has already been done here...

### sharing, granular, multiplicity

Jaroslav Tulach: However I am more interested in upgrades. From time to time I want to upgrade one library used on my server.

That's one of the use cases I find interesting. The basic question of interest is this one: "Which version of the software is running on the server(s) now?" I want one possible answer to be: all of them.

I want this answer for multiple reasons. First, I want a debug version running all the time — or able to run on demand — so problems can be studied in as near to real time context as possible. That would improve reasoning about actual problem causes, and open the door to correct problems with live update, on the fly.

Second, once peaceful co-existence of more than one version of a system is feasible, you can do staged version switching just by editing your namespace (if you have privilege). This would let you partition users on different versions, so a smaller group can be exposed to new risks at one time. And it would allow you to back out flawed updates simply by reverting a path binding that revealed the version you now want to redact.

However, servers have limited resources (I sometimes write code using more than half available memory) so running multiple live versions at once on a "real" system creates a resource problem unless some layering is involved, where some parts share common parts that don't change when one version changes (because that part remains the same in the new version).

The last paragraph is where programming language factors might enter the picture. Just aiming for multiple simultaenous versions isn't really on topic for LtU. But using a language like Erlang where high granularity lightweight process design might be on topic. I don't mean Erlang specifically, just one like it in terms of isolated dependencies in lightweight processes. Then you ask questions about your language like this: "Does my language allow me to upgrade this lightweight process without affecting other processes written in this language?"

I can expand this indefinitely by Socratic questions, so I'll stop here.

### Fighting for sparse resources

All! That is interesting, yet reasonable. However memory is not the only resource that you need to share. If your system deals with database, you do not want the unstable libraries to modify data of the production system[1]. Sharing files is bad as well. I slightly talk about coexistance of similar APIs in Chapter 15, where I also talk about another sparse resource: providers. If you have an encryption provider written against old API version and another against new API version, you probably want to use them both at once...

This is not easy, but with a few bridges it is possible. It just takes time to write the bridge. It would be even better if the bridge could be generated by the compiler automatically. Anyone knows a language suitable for generating API bridges?

[1] Recently I let my development version of mediawiki server talk to my production database and I could not believe my eyes! I knew I should modify no page, but still the production system got influenced! Mediawiki stores cached version of pages in database and suddenly my intranet IP address started to leak to public pages served by the production server. Frameworks are full of surprises.

### A corrolary problem

There is an interesting corollary problem. Assume, for a moment, that we could wave a magic wand and make library versioning a solved problem. To some extent I think this is even possible. If we had a workable library versioning solution, is it actually safe to use it?

For just a moment, take the perspective of the application vendor. From this perspective, transparent dynamic library updates are a nightmare. The problem is that you cannot test an interface, only an implementation. In consequence, every application vendor is at the mercy of the software process and testing regime of every project that manages every upgradeable dynamic library on which they depend. In such an environment, offering any sort of concrete warranty would be an act of complete idiocy.

The practical answer adopted in Linux seems to be mixed. Some applications link against specific library versions, while others link only against "version 2.x" (for some value of two). The former practice seems more prevalent in C++, where almost any change to anything in a library induces incompatible structure changes in the API.

### Right on

Years ago when I first thought about these issues, I thought of some form of dynamic linking, possibly with version numbering of the sort suggested above. In discussion with others I was quickly disillusioned with this approach, since things are way more complicated. You can try to design a language for expressing dependencies (X can work with Y providing both X and Y use ver. n of Z, for example), or rely on common sense and static linking...

### API is more than signatures

I share your concerns. We need more than just to version our libraries, we need to version them properly. It takes a bit of self control to do it right. Also it requires trust from others in one's ability to do it right. Moreover it is necessary to realize that APIs do not end at level of function or method signatures, they span much more and include also runtime aspects of the libraries. Proper versioning needs to apply to all these kinds of API.

We have two versioning styles in NetBeans. One is used to identify that a library absolutely compatible, just throw the new version in, you do not have to worry. There will be no incompatibilities what so ever. We use this style for patches and minor bug fix releases. The other versioning says something like: You will not have troubles to update, but you need to test. Here we do not guarantee absolute, bug to bug compatibility, just reasonable amount of it.

Btw. I agree with your last paragraph. C++ is not really language suitable for distributed development.

### One is used to identify that

One is used to identify that a library absolutely compatible

But this is just a programmer declaration, right? Not something that is mechanically verified.

### With a bit of tooling...

Right now my project relies on programmer's declaration, however if there was a bit of support from tools, the absolute compatibility could even be verified automatically. It is not impossible to verify that all code which can be executed by usage of the old API remains unchanged, while the new code additions can only be invoked with at least one use of the new API. If this check passes, then it implies 100% backward compatibility. As Matt M notes bellow, this is really draconical and probably only suitable for patch releases. At least this is my experience from creating a desktop framework. However, it all depends on the need of quality. If my library was used in heart of plane control system, I would rather accept this check for all releases.

### Draconian?

Wouldn't such an automatic verification of *zero* change preclude the use of this flag for "patches and minor bug fix releases" - your stated use case?

### Draconian enough?

I am actually more concerned that this is not draconian enough. Suppose you have two routine get_date which return the initialization date of an object. Since no routine allows the user to change the date, you are ensured that the date remains and initialized. Assume no routine is changed, but an additional set_date routine is supplied. Suddenly the previous invariant is no longer relevant.

This is a toy example, but if you think of external effects, most notably changes to databases, the general issue should become apparent.

### I think it is enough!

Yesterday I was a bit sad after reading Ehud's comment, but at the morning I realized that this kind of "draconian" check is enough to guarantee safe upgradability.

If you add new API, like the setter for changing the date, then someone has to call it. But if the only change in your whole system is just the library itself (and it passes the "no change to previously existing code" test), then there is nobody who could call the new setter method. Your whole system knows nothing about it, as it was compiled against version of the library which did not have it. So we are safe, the behaviour has to stay the same.

Now imagine, that you change code of your system to call the new setter. Well, invariants can no longer hold, and things can break, but I'd say that it is OK - programmers are used to destabilize the system when they change their source code ;-)

Also notice that if you upgrade more libraries, and all of them pass the "no change to previously existing code" test, the whole set of them will pass the test as well. This property of the libraries seems to be transitive. Which is good, I still see it as a possible, yet draconian test for safe upgradability.

### there is nobody who could call the new setter

If you add new API, like the setter for changing the date, then someone has to call it. But if the only change in your whole system is just the library itself (and it passes the "no change to previously existing code" test), then there is nobody who could call the new setter method.

I suppose that the test include a check that no pre-existing code in the library happens to call set_date?

### For some reason I was

For some reason I was thinking that a library is something that can have users outside your control. Am I missing something?

### Library versioning versus application versioning

Sure, but here we are discussing about using an upgraded library as "drop-in replacement"-- supposedly the library changes, not the application. So the application should make no calls to the new API.

### This is similar how binary

This is similar how binary compatibility worked from Java 1.0.2 to 1.1. Java 1.1 added new abstract APIs to java.awt.Graphics. Previous implementations of Graphics were broken at the source level in 1.1 because they didn't implement the new APIs. However, at the binary level everything worked fine for two reasons:

* The JVM doesn't enforce that concrete classes implement all their abstract methods. Instead, this is a run-time check (on first use).

* The unimplemented APIs wouldn't be called as long as no 1.1 APIs were called.

You would get into trouble, of course, if you passed the graphics object into a library that did use 1.1 APIs. In general, this was a very fragile API changed and Sun hasn't seem to have done this since.

### dependencies

shap: From this perspective, transparent dynamic library updates are a nightmare.

I've been looking at it from the server at the end of the food chain, not in terms of library replacement (except by a server using a library). I'm not looking at it from the perspective of a library vendor. I'm not interested in defending their position in an ecology.

If component B depends on A, then I would expect B to be replaced/versioned too when A is versioned. The safe thing to do is version everything downstream from a change.

Swapping a piece in the middle of a dependency chain wouldn't be safe in general, as you noted. Transparent dynamic library update should be strongly discouraged. Generally, transparent anything tends to have problems.

I currently work in a market segment where all the software in a box is under control of the development team putting the entire package together. There's no such thing as replacing a library in this context without versioning the product. However, in this sort of context, I'm interested in live update, provided the same care is taken as any other release (ie extreme caution and paranoia about errors).

Any kind of ecology that mixes and matches random libraries without a coordinating oversight controller seems doomed to chaos.

### Nice if you have it.

I currently work in a market segment where all the software in a box is under control of the development team putting the entire package together.

From the standpoint of the software industry, that position is such an outlier that it might as well not exist. Certainly it is not a position that any sensible designer should design to. In the rest of the world, software is an aggregate containing components from multiple parties, and upgrades don't happen in lockstep.

But I suspect that your assertion is just flatly mistaken. If you do indeed control everything in your package, then I imagine you must be in the compiler business. Also the operating system business. You obviously don't use SSL or a web server... In the modern software ecology it simply isn't humanly possible for your assertion to hold up to inspection. And given that this is the case, it's absolutely imperative to defend the position of the component vendor, because without those vendors you won't ship a darned thing.

shap: But I suspect that your assertion is just flatly mistaken.

My assertion was oversimplified. I usually can't manage to say anything I feel is unqualified truth in less than several times as much as anyone is willing to read. I cut it close sometimes, but I hope for less distortion than reality. Your doubts are both interesting and well said (these don't go together very often :-).

I need to be careful about what I say. (Just enough people know where I work that I can't say anything interesting without it being used as potential spin. That I don't publicize where I work makes spin that much more effective because third parties assume I think I'm speaking anonymously; however, I'm not that naive.) Caution might make extra detail I add come out slightly flat.

Let's say they try to control everything, including things you mention (interesting list by the way :-), and this throttling keeps us behind the curve a little (very irritating) and is expensive to qualify. Control is a fuzzy word with a lot of things under the umbrella, so I don't feel responsible for your interpretation that it means only things one writes oneself.

They look at things I'd never have time to even begin trying to categorize because I lack the requisite background — I'm just a software builder and not a networking wonk. But I read a lot into status details passing through channels. Folks have a good idea what's happening at the byte level in comm channels and in memory. I'm mostly in the memory business, representing and implementing context, lifecycle, resources, and use of a main software feature.

I'm the person they expect to answer questions about effect of dependencies on a central feature. Not that there's much to it. Sorry I said so much. It's hard not to rise to such a well prosecuted accusation that I'm either dishonest or incompetent (which I don't mind -- it was just interesting). It seems more than half my time is consumed by effects of environment dependencies. It's a pain.

If without those vendors we didn't ship a darned thing, then I'd do something else. They have to defend themselves. Not my job.

### Having suffered my 15

Having suffered my 15 minutes on the front page of the Wall Street Journal at one point, I sympathize. :-)

### big media blues

Our direct competitors know me personally (nice folks but bright and aggressive) so I can't safely make substantive remarks of any kind, good or bad, without worrying what unexpected effect it might have. I try to cultivate a low profile.

### Book

I haven't read Jaroslav' book but it seems to touch on issues that are important for PLT, and that quite a few LtU members work on, especially those interested in system programming issues and those actually working on module systems.

It would definitely be interesting if a LtU contributor interested in these issues would like to write a short review of the book for us.

### You might want to look at an

You might want to look at an earlier discussion about NixOS, which is a new style of package management system. The problem of package mangement is closely related to library management, so there might be ideas worth adapting.

I have collected a few papers on program upgrades over the years, in particular live upgrade. I recently posted my brief list on cap-talk.

### Re: there is nobody who could call the new setter

Right. The test would verify that no previously existing line of code reachable with old APIs has been changed. This implies that the pre-existing code cannot call set_date. Neither in old version (the method did not existed), neither in new version (the test would found a modification).

### Only if there are only two parties involved

Once you have two customers with access to the same object, one might upgrade to call set_date with the other failing to update or even recompile.
The real point about Ehud's comment is that when you are committing to backwards compatibility, you have to have some understanding between the parties as to what constitutes compatibility worth maintaining.

If the get_date function had a comment stating "Returns the date at which the object was created" then you have to update that comment -- and that semantic change to get_date is the break.

It is hard to think of a semantic meaning for a get_date function that covers the setting of an arbitrary date, so let's imagine that we want to add "refresh_date" and that the original documentation of get_date said "Returns some date between the creation of the object and the calling of the function", then adding refresh_date is OK.

If get_date is undocumented or ambiguously documented, you need some process to resolve this. At Symbian we have a board of beard-strokers (of whom I am an occaisional member) who deliberate on the difficult cases and inform the customers of our decisions, and get feedback on certain issues. This process is considered integral to our backwards compatibility promise.

It is tempting to dream of a world where a programming language or supporting infrastructure would free us of the compatibility burden, turning it into just another language feature. However, it is in the mucky world of semantics, where interfaces meet the world of human use and abuse where this stuff really matters. You can break compatibility if you know it won't matter (we once changed our formerly weird floating-point implementation to match IEEE with no-one even noticing), but you can't make some compatible changes (changing priorities of processes that have no stated dependency on each other is a classic).

Having said all that, there is much work to be done on merely keeping interfaces stable. You can get quite far just by assuming that the source interface (your class and function signatures in your .h files in C++) matches the semantics, and picking up incompatible changes to that over time (removal of functions for example) and alterations of the binary interface in relation to the source interface (adding a virtual function for example).

Of course, occaisionally you must allow customers to override common sense; I'm sure many readers here are familiar with the story of Microsoft Windows' memory manager running in a different mode if it detected Sim City running -- this is always a risk and you have to be aware that you might be forced to retract changes in the face of politics.

I should probably stop now, although really I'm just getting started!

Jaroslav, I'd be happy to review your book.

### The type system is what we

The type system is what we know about the structure of a program. Flow analysis augments our knowledge to include behavior.

If type and flow information are placed into a rule system, rules can annotate the type system with behavioral dependencies on the inputs. Richer type systems imply richer behavioral annotations. We can capture the augmented type system hierarchy as a tree of "versions". The "label" for a version is the set of inputs and outputs, where each of inputs and outputs carries behavioral annotations.

Many techniques might be used to capture the differences between two versions. Further, a series of rules can capture well-known situations and attempt to automatically bridge differences, or ignore them where appropriate.

Scala has a path-dependent type system. An _intensional_ type system could capture typing information in a context of arbitrary dimensions. Guarantees of termination are unlikely, but rule systems can thrive in these environments.

Some day we might be able to have a session like this:

>superlang *.slang
>superlang --mark
superlang: Marked as V1
>... edit files ...
>superlang *.slang
lint-behavior: 'hello(who)'/V1->* is sensitive to null in parameter who
>superlang --mark
sense-error: hello(who)/V1->* has 'null sensitivity'; unable to mark
>superlang --inject-safe
inject: 'hello(who)'/V1->*: Injected 'null-check' to satisfy 'sensitive to null'
>superlang --mark
info: injection 'null-check' used to satisfy 'null sensitivity'
superlang: Marked as V2



### Are tools ready for API evolution? Are tools ready for people?

On 5 September 2008, JaroslavTulach also wrote:
"yet it seems that there is no language in the world that would make this easy"

Is this really a language problem? I think it's a tool and communication problem. You want a tool that communicates to you that the change you're about to make will result in incompatibilities with your previous consumers.

We can even imagine that they could automatically be notified that you are doing this, which might permit them in turn to notify their own consumers in some way.

The gnarliest problem here is illustrated in this scenario- imagine you used this tool and it notified the distributed developers and happily enough they all got the memo. How are they then supposed to communicate anything so technical as "you can't download the latest version of such and such library because that program that you bought from us (remember us?) on such and such a date could stop working properly. Thanks."

A lot of interesting programming problems aren't really programming problems, they're user interface problems. The bulk of Tulach's brilliant API book is an attempt to deal with limitations on our ability to communicate intent and process information, i.e. we're not able to communicate to other programmers what not to do and we have a hard time with big balls of mud.

The reason that's germane is because a lot of problems that have our information processing limitations as their root cause, as opposed to say limitations in our ability to measure the position of a particle or limitations in the strength of materials, can be dealt with through the magic of user interface engineering.

For example, one thing mentioned in the book is that people complain their brains are being all "twisted up" with " new rules" for API writing. Why not then just have the tool automatically present to the programmer what the potential downsides, from an API perspective, are to the code they just typed?

Generally, the idea that code can be self documenting is all wrong in my opinion, or at least it's one of those things whose effectiveness is greatly exaggerated. The reality is, no matter how simple the code is, or how familiar the idioms are or meaningful the variable names are or clever the intent-revealing method names are, programming is still a pretty abbreviated and arcane expression of someone's larger intent and is designed to be compilable first and communicative to humans second.

I don't know why literate programming never took off the way it should have, but there's a rich body of thought there that deserves more respect than it's getting. There's no reason I should have to look at code itself- code is not its own best representation.

Instead, I should be looking at native language representations of a programmer's intent. If I don't believe a method or algorithm does what it says it does, or I want to see how the magic is done, or I smell a rat, then I should be able to reveal the code underlying the explanations, but that's not what I want to see for the most part.

The UML goes a ways to dealing with this, but it's cumbersome and anyways it's too isomorphic to the code; it's too slavishly bound to the programming language itself, (it needs to be automatically generatable). Like programming languages, it's trying to serve two gods- one is human comprehension and the other is something we call a compiler.

What we need in a nutshell is semantic drill down, where we go from high level concepts all the way down to the code itself, with each successive representation becoming more and more isomorphic to the code.

My editor, all editors, are totally stupid; they don't allow me to express anything other than code, except in comments, which are limited in where they can appear, what they can look like etc. etc.

Editors fix my mind forever at the level of classes and "class file" which correlates to exactly nothing in the mind I use to problem solve and reason and talk with.

Program editors should permit me to take advantage of every trick of story telling, every rhetorical device, all typographic, and diagrammatic know-how, all cinematography tricks and techniques and any other form of communication we have available.

Programming is supposedly the most complicated activity humankind engages in; programs are supposedly the most complicated machines made by people. Programs have many more possible states than, say, AMD's latest processor. So why am I forced to communicate how this machine works to other people using only an expressive ghetto?

### Let me be very clear in my

Let me be very clear in my opinion that this particular example given by Jaroslav is a poor argument for versioning components. In my applications, I never, ever put an ABI for the user interface, because it is so volatile. Instead, I push things into a schema authoring tool for dependency injection.