Lambda the Ultimate

With library versioning, you can make sure that the API client application is using a consistent set of classes and interfaces. Is that not one feature of the Microsoft CLR? (maybe not, a quick search did not give me the expected results)

You could go one step further and design a compatible API extension: in your example, it does not really matter if there is one more method in View, as long as either

• The method has a default implementation
• The method is not used with the class which was developed before the API change

The second will be true in a self-contained application, it would likely be false in a library or reusable component.

Annotating various library revisions with version number seems really necessary for purposes of distributed development. Also when using a library, one shall not specify just the library itself, but also its version or version range. This is somehow present in existing modular and packaging systems (NetBeans, OSGi, RPM, etc.), however language designers seem to be reluctant to include this into any language as first class citizen. All I can see is import javax.swing.JTree statements, and not import javax.swing.JTree version 1.3 or greater.

Java VM as well as CLR have their rules for compatibility on binary level. And there are external tools that can verify them (apitest, for example). However no language I know comes with such support for evolution itself.

Is that not one feature of the Microsoft CLR? (maybe not, a quick search did not give me the expected results)

Microsoft calls it "assembly versioning".

It looks to me, that you want to achieve reproducibility. Ability to compile and get the same bits as someone who compiled a year ago, or ability to configure the runtime with exactly the same environment as done before. I believe that for example Maven with its on-line repositories is system that can guarantee that. You are right that for this kind of snapshotting there is no need for any language changes.

However I am more interested in upgrades. From time to time I want to upgrade one library used on my server. Either to fix bugs or get new features. At this point of time, I'd like to be sure that the system does not destabilize, that the new library version is at least as good as the old version used to be. That there are no missing symbols, the whole application that I compiled previously can still be compiled, run, and so on.

This can, with a bit of carefulness, be done for Java, without any language features, however as I confess in the Practical API Design, it requires a lot of self control, awareness of too much compatibility rules, bunch of support tools, etc. My feeling is that with languages designed with the API evolution in mind, this could be task achievable with much more cluelessness - task approachable to wider audience of developers.

I believe this kind language feature is needed and I am still hoping some readers of LtU will just point me to a language or system and say: Hey, look, this has already been done here...

Jaroslav Tulach: However I am more interested in upgrades. From time to time I want to upgrade one library used on my server.

That's one of the use cases I find interesting. The basic question of interest is this one: "Which version of the software is running on the server(s) now?" I want one possible answer to be: all of them.

I want this answer for multiple reasons. First, I want a debug version running all the time — or able to run on demand — so problems can be studied in as near to real time context as possible. That would improve reasoning about actual problem causes, and open the door to correct problems with live update, on the fly.

Second, once peaceful co-existence of more than one version of a system is feasible, you can do staged version switching just by editing your namespace (if you have privilege). This would let you partition users on different versions, so a smaller group can be exposed to new risks at one time. And it would allow you to back out flawed updates simply by reverting a path binding that revealed the version you now want to redact.

However, servers have limited resources (I sometimes write code using more than half available memory) so running multiple live versions at once on a "real" system creates a resource problem unless some layering is involved, where some parts share common parts that don't change when one version changes (because that part remains the same in the new version).

The last paragraph is where programming language factors might enter the picture. Just aiming for multiple simultaenous versions isn't really on topic for LtU. But using a language like Erlang where high granularity lightweight process design might be on topic. I don't mean Erlang specifically, just one like it in terms of isolated dependencies in lightweight processes. Then you ask questions about your language like this: "Does my language allow me to upgrade this lightweight process without affecting other processes written in this language?"

I can expand this indefinitely by Socratic questions, so I'll stop here.

All! That is interesting, yet reasonable. However memory is not the only resource that you need to share. If your system deals with database, you do not want the unstable libraries to modify data of the production system[1]. Sharing files is bad as well. I slightly talk about coexistance of similar APIs in Chapter 15, where I also talk about another sparse resource: providers. If you have an encryption provider written against old API version and another against new API version, you probably want to use them both at once...

This is not easy, but with a few bridges it is possible. It just takes time to write the bridge. It would be even better if the bridge could be generated by the compiler automatically. Anyone knows a language suitable for generating API bridges?

[1] Recently I let my development version of mediawiki server talk to my production database and I could not believe my eyes! I knew I should modify no page, but still the production system got influenced! Mediawiki stores cached version of pages in database and suddenly my intranet IP address started to leak to public pages served by the production server. Frameworks are full of surprises.

There is an interesting corollary problem. Assume, for a moment, that we could wave a magic wand and make library versioning a solved problem. To some extent I think this is even possible. If we had a workable library versioning solution, is it actually safe to use it?

For just a moment, take the perspective of the application vendor. From this perspective, transparent dynamic library updates are a nightmare. The problem is that you cannot test an interface, only an implementation. In consequence, every application vendor is at the mercy of the software process and testing regime of every project that manages every upgradeable dynamic library on which they depend. In such an environment, offering any sort of concrete warranty would be an act of complete idiocy.

The practical answer adopted in Linux seems to be mixed. Some applications link against specific library versions, while others link only against "version 2.x" (for some value of two). The former practice seems more prevalent in C++, where almost any change to anything in a library induces incompatible structure changes in the API.

Years ago when I first thought about these issues, I thought of some form of dynamic linking, possibly with version numbering of the sort suggested above. In discussion with others I was quickly disillusioned with this approach, since things are way more complicated. You can try to design a language for expressing dependencies (X can work with Y providing both X and Y use ver. n of Z, for example), or rely on common sense and static linking...

I share your concerns. We need more than just to version our libraries, we need to version them properly. It takes a bit of self control to do it right. Also it requires trust from others in one's ability to do it right. Moreover it is necessary to realize that APIs do not end at level of function or method signatures, they span much more and include also runtime aspects of the libraries. Proper versioning needs to apply to all these kinds of API.

We have two versioning styles in NetBeans. One is used to identify that a library absolutely compatible, just throw the new version in, you do not have to worry. There will be no incompatibilities what so ever. We use this style for patches and minor bug fix releases. The other versioning says something like: You will not have troubles to update, but you need to test. Here we do not guarantee absolute, bug to bug compatibility, just reasonable amount of it.

Btw. I agree with your last paragraph. C++ is not really language suitable for distributed development.

One is used to identify that a library absolutely compatible

But this is just a programmer declaration, right? Not something that is mechanically verified.

Right now my project relies on programmer's declaration, however if there was a bit of support from tools, the absolute compatibility could even be verified automatically. It is not impossible to verify that all code which can be executed by usage of the old API remains unchanged, while the new code additions can only be invoked with at least one use of the new API. If this check passes, then it implies 100% backward compatibility. As Matt M notes bellow, this is really draconical and probably only suitable for patch releases. At least this is my experience from creating a desktop framework. However, it all depends on the need of quality. If my library was used in heart of plane control system, I would rather accept this check for all releases.

Wouldn't such an automatic verification of *zero* change preclude the use of this flag for "patches and minor bug fix releases" - your stated use case?

I am actually more concerned that this is not draconian enough. Suppose you have two routine get_date which return the initialization date of an object. Since no routine allows the user to change the date, you are ensured that the date remains and initialized. Assume no routine is changed, but an additional set_date routine is supplied. Suddenly the previous invariant is no longer relevant.

This is a toy example, but if you think of external effects, most notably changes to databases, the general issue should become apparent.

Yesterday I was a bit sad after reading Ehud's comment, but at the morning I realized that this kind of "draconian" check is enough to guarantee safe upgradability.

If you add new API, like the setter for changing the date, then someone has to call it. But if the only change in your whole system is just the library itself (and it passes the "no change to previously existing code" test), then there is nobody who could call the new setter method. Your whole system knows nothing about it, as it was compiled against version of the library which did not have it. So we are safe, the behaviour has to stay the same.

Now imagine, that you change code of your system to call the new setter. Well, invariants can no longer hold, and things can break, but I'd say that it is OK - programmers are used to destabilize the system when they change their source code ;-)

Also notice that if you upgrade more libraries, and all of them pass the "no change to previously existing code" test, the whole set of them will pass the test as well. This property of the libraries seems to be transitive. Which is good, I still see it as a possible, yet draconian test for safe upgradability.

If you add new API, like the setter for changing the date, then someone has to call it. But if the only change in your whole system is just the library itself (and it passes the "no change to previously existing code" test), then there is nobody who could call the new setter method.

I suppose that the test include a check that no pre-existing code in the library happens to call set_date?

For some reason I was thinking that a library is something that can have users outside your control. Am I missing something?

Sure, but here we are discussing about using an upgraded library as "drop-in replacement"-- supposedly the library changes, not the application. So the application should make no calls to the new API.

This is similar how binary compatibility worked from Java 1.0.2 to 1.1. Java 1.1 added new abstract APIs to java.awt.Graphics. Previous implementations of Graphics were broken at the source level in 1.1 because they didn't implement the new APIs. However, at the binary level everything worked fine for two reasons:

* The JVM doesn't enforce that concrete classes implement all their abstract methods. Instead, this is a run-time check (on first use).

* The unimplemented APIs wouldn't be called as long as no 1.1 APIs were called.

You would get into trouble, of course, if you passed the graphics object into a library that did use 1.1 APIs. In general, this was a very fragile API changed and Sun hasn't seem to have done this since.

shap: From this perspective, transparent dynamic library updates are a nightmare.

I've been looking at it from the server at the end of the food chain, not in terms of library replacement (except by a server using a library). I'm not looking at it from the perspective of a library vendor. I'm not interested in defending their position in an ecology.

If component B depends on A, then I would expect B to be replaced/versioned too when A is versioned. The safe thing to do is version everything downstream from a change.

Swapping a piece in the middle of a dependency chain wouldn't be safe in general, as you noted. Transparent dynamic library update should be strongly discouraged. Generally, transparent anything tends to have problems.

I currently work in a market segment where all the software in a box is under control of the development team putting the entire package together. There's no such thing as replacing a library in this context without versioning the product. However, in this sort of context, I'm interested in live update, provided the same care is taken as any other release (ie extreme caution and paranoia about errors).

Any kind of ecology that mixes and matches random libraries without a coordinating oversight controller seems doomed to chaos.

I currently work in a market segment where all the software in a box is under control of the development team putting the entire package together.

From the standpoint of the software industry, that position is such an outlier that it might as well not exist. Certainly it is not a position that any sensible designer should design to. In the rest of the world, software is an aggregate containing components from multiple parties, and upgrades don't happen in lockstep.

But I suspect that your assertion is just flatly mistaken. If you do indeed control everything in your package, then I imagine you must be in the compiler business. Also the operating system business. You obviously don't use SSL or a web server... In the modern software ecology it simply isn't humanly possible for your assertion to hold up to inspection. And given that this is the case, it's absolutely imperative to defend the position of the component vendor, because without those vendors you won't ship a darned thing.

shap: But I suspect that your assertion is just flatly mistaken.

My assertion was oversimplified. I usually can't manage to say anything I feel is unqualified truth in less than several times as much as anyone is willing to read. I cut it close sometimes, but I hope for less distortion than reality. Your doubts are both interesting and well said (these don't go together very often :-).

I need to be careful about what I say. (Just enough people know where I work that I can't say anything interesting without it being used as potential spin. That I don't publicize where I work makes spin that much more effective because third parties assume I think I'm speaking anonymously; however, I'm not that naive.) Caution might make extra detail I add come out slightly flat.

Let's say they try to control everything, including things you mention (interesting list by the way :-), and this throttling keeps us behind the curve a little (very irritating) and is expensive to qualify. Control is a fuzzy word with a lot of things under the umbrella, so I don't feel responsible for your interpretation that it means only things one writes oneself.

They look at things I'd never have time to even begin trying to categorize because I lack the requisite background — I'm just a software builder and not a networking wonk. But I read a lot into status details passing through channels. Folks have a good idea what's happening at the byte level in comm channels and in memory. I'm mostly in the memory business, representing and implementing context, lifecycle, resources, and use of a main software feature.

I'm the person they expect to answer questions about effect of dependencies on a central feature. Not that there's much to it. Sorry I said so much. It's hard not to rise to such a well prosecuted accusation that I'm either dishonest or incompetent (which I don't mind -- it was just interesting). It seems more than half my time is consumed by effects of environment dependencies. It's a pain.

If without those vendors we didn't ship a darned thing, then I'd do something else. They have to defend themselves. Not my job.

Having suffered my 15 minutes on the front page of the Wall Street Journal at one point, I sympathize. :-)

Our direct competitors know me personally (nice folks but bright and aggressive) so I can't safely make substantive remarks of any kind, good or bad, without worrying what unexpected effect it might have. I try to cultivate a low profile.

Once you have two customers with access to the same object, one might upgrade to call set_date with the other failing to update or even recompile.
The real point about Ehud's comment is that when you are committing to backwards compatibility, you have to have some understanding between the parties as to what constitutes compatibility worth maintaining.

If the get_date function had a comment stating "Returns the date at which the object was created" then you have to update that comment -- and that semantic change to get_date is the break.

It is hard to think of a semantic meaning for a get_date function that covers the setting of an arbitrary date, so let's imagine that we want to add "refresh_date" and that the original documentation of get_date said "Returns some date between the creation of the object and the calling of the function", then adding refresh_date is OK.

If get_date is undocumented or ambiguously documented, you need some process to resolve this. At Symbian we have a board of beard-strokers (of whom I am an occaisional member) who deliberate on the difficult cases and inform the customers of our decisions, and get feedback on certain issues. This process is considered integral to our backwards compatibility promise.

It is tempting to dream of a world where a programming language or supporting infrastructure would free us of the compatibility burden, turning it into just another language feature. However, it is in the mucky world of semantics, where interfaces meet the world of human use and abuse where this stuff really matters. You can break compatibility if you know it won't matter (we once changed our formerly weird floating-point implementation to match IEEE with no-one even noticing), but you can't make some compatible changes (changing priorities of processes that have no stated dependency on each other is a classic).

Having said all that, there is much work to be done on merely keeping interfaces stable. You can get quite far just by assuming that the source interface (your class and function signatures in your .h files in C++) matches the semantics, and picking up incompatible changes to that over time (removal of functions for example) and alterations of the binary interface in relation to the source interface (adding a virtual function for example).

Of course, occaisionally you must allow customers to override common sense; I'm sure many readers here are familiar with the story of Microsoft Windows' memory manager running in a different mode if it detected Sim City running -- this is always a risk and you have to be aware that you might be forced to retract changes in the face of politics.

I should probably stop now, although really I'm just getting started!

Jaroslav, I'd be happy to review your book.

Perfect, please send me your email and snail address to team@apidesign.org.