Why API Design Matters

Michi Henning, Why API Design Matters, Communications of the ACM, May 2009.

After more than 25 years as a software engineer, I still find myself underestimating the time it takes to complete a particular programming task. Sometimes, the resulting schedule slip is caused by my own shortcomings: as I dig into a problem, I simply discover it is a lot more difficult than I initially thought, so the problem takes longer to solve—such is life as a programmer. Just as often I know exactly what I want to achieve and how to achieve it, but it still takes far longer than anticipated. When that happens, it is usually because I am struggling with an application programming interface (API) that seems to do its level best to throw rocks in my path and make my life difficult. What I find telling is that, even after 25 years of progress in software engineering, this still happens. Worse, recent APIs implemented in modern programming languages make the same mistakes as their 20-year-old counterparts written in C. There seems to be something elusive about API design that, despite years of progress, we have yet to master.

This is a rather accessible look at the consequences of bad API design. Interestingly enough, the main example revolves around the inappropriate use of side effects. The last section concludes with cultural changes the author feels is necessary to improve the situation.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

When starting by saying

When starting by saying after 25 years API design is still hard to get right (agreed), maybe one shouldn't end by contemplating "criminal penalties" for bad API. I think that actually gets to the main problem though, no one agrees what is good and bad as it depends on user preference and context. There are some great examples in there of perennial problems, maybe some of that is could be a language's responsibility.

Most beautiful API's tend to describe problems that lend themselves to code. Not to take away from the great work done, but sometimes bad API's reflect messy moving problems.

Criminal Penalties

His proposal is actually even worse than that: he's not proposing criminal penalties for bad APIs, he's proposing them for changing any APIs to anything deemed 'critical'; to quote: "It is irresponsible to allow a single company (let alone a single developer) to make changes to such critical APIs [he gives OpenSSL and the C library as examples] without external controls."

This is not just absurd, but also completely counter-productive. He himself notes that backwards compatibility concerns have prevented many APIs from being made rational: "This is also an example of how concerns about backward compatibility erode APIs over time: the API accumulates crud that, eventually, does more damage than the good it ever did by remaining backward compatible." And I can't imagine a worse incentive to backwards compatibility than putting someone in jail for changing an API! And I tend to think really good APIs only come via iteration - to use is .NET Select() example, it could easily be extended to support the cases he wants, now that the problems with the current one have been identified. It's very hard to create an API that is great from the start, especially in a new problem domain.

(I'm also baffled by his example of OpenSSL; should someone tell him that it's an open source project with almost all of the work done by about a dozen people who regularly replace or remove APIs?)

Re: Criminal Penalties

Well, in all fairness to the author, he puts in a big disclaimer that he isn't advocating criminal penalties, merely that he thinks they might be inevitable. My summary oversimplified the author's message quite a bit.

Of course, trying to compare construction with software engineering is a bit of a tortured analogy...

Re: Criminal Penalties

I've never heard anybody ever even bring up the concept of criminalizing "mission-critical" API changes. But not only is his construction analogy a huge stretch, but he also said:

"It is irresponsible to allow a single company (let alone a single developer) to make changes to such critical APIs without external controls."

External controls? Luckily the vast majority of developers would probably laugh at these ideas, but unfortunately there's probably a few fascist-like politicians that wouldn't mind enacting some kind of new government agency to oversee that nonsense.

We are not finished yet: the

We are not finished yet: the preceding code still contains comments in place of copying the input parameters and copying the results back into those parameters. One would think that this is easy: simply call a Clone() method, as one would do in Java. Unlike Java, however, .NET's type Object (which is the ultimate base type of all types) does not provide a Clone method; instead, for a type to be cloneable, it must explicitly derive from an ICloneable interface.

There is another subtlety here not touched upon by the author.

Both Java's Clone and .NET's Clone are stupid and ugly. The person who initially created .NET's Ecma standardized base classes even goes so far as to say Do Not Implement ICloneable and asking customers if it should simply be obsoleted (note: not simply deprecated, but obsoleted).

So even in a journal like Communications of the ACM, a very important API design point -- that is even published in Brad Abram and Kryzstof Cwalina's .NET Framework Design Guidelines: Conventions, Idioms, and Patterns for Reusable .NET Libraries -- is overlooked.

Edit: I would conclude by saying that, culturally, it has been my experience that people only care about API Design when the current API pisses them off.

One would think that this is easy

I take particular exception to Henning's comment:

One would think that this is easy: simply call a Clone() method...

This is just rhetorical handwaving. "Simply call a Clone() method"? What are the semantics of that magical clone method (beyond the obvious "do what I need right now, exactly the way I want it done at the moment")? The reason for the "ICloneable considered harmful" sentiments in the blog post you cite is that there are no universally-appropriate semantics to something like Clone(); therefore any implementation will be subtly wrong in some (likely common) set of circumstances.

When critiquing an API design as being inelegant, citing the lack of a semantic monstrosity like Clone() doesn't bolster one's case: "this API makes it difficult to do something I want to do; a better design would assume I always want to do it this other way (and never differently)".

much like equals()

funny how there's no universally-appropriate semantics for equal in java/c# OOs as well?

I think his point is more

I think his point is more that IClonable is practically worthless for writing code other programmers link to. Since it isn't very well defined, ICloneable doesn't add much to the API. Something like IDeepCloneable and IShallowCloneable would be better defined and clean up some of the inherent ambiguity of ICloneable.

While I am very sympathetic

While I am very sympathetic to the problems the author wishes to raise, I find several little mistakes and oversights troubling:

"We are not finished yet: the preceding code still contains comments in place of copying the input parameters and copying the results back into those parameters. One would think that this is easy... Again, the only way to achieve this is to iterate and copy the elements one at a time."

It is easy, without getting into the mess that a universal clone method creates. Just use "ArrayList copy = new ArrayList(original)" or "targetList.AddRange(original)". Am I missing something? Surely this is easy stuff?

"The time-out is provided as a Timespan (a type provided by .NET) that has resolution down to 100 nanoseconds, a range of more than 10 million days, and can be negative (or null) to cover the "wait forever" case."

Timespan is a struct, a value type. It cannot be null.

"The Unix kernel also is not without blemish and passes the buck occasionally: many a programmer has cursed the decision to allow some system calls to be interrupted, forcing programmers to deal explicitly with EINTR and restart interrupted system calls manually, instead of having the kernel do this transparently."

This is certainly a pain, but is it really that simple? If the system calls restarted transparently, how would users write programs that respond promptly to signals while still making use of such system calls? You could add another argument to specify what to do in the event of a signal, but the author has already pointed out that this makes for bloated, awkward APIs.

I think it's important to consider APIs first from a user's point of view, but I don't think that overall it makes the vast difference the author suggests. For one thing, the API may have multiple users with very different requirements, especially a low level operating system one, so what seems like an obvious improvement to one user could be a crippling defect to another. I also find that if I write an API before it's implementation, there's a good chance I'll overlook some subtle but important piece of information that makes the implementation impossible. I'm not saying it's not an improvement, but it's not without its own pitfalls.

"The time-out is provided as

"The time-out is provided as a Timespan (a type provided by .NET) that has resolution down to 100 nanoseconds, a range of more than 10 million days, and can be negative (or null) to cover the "wait forever" case."

Timespan is a struct, a value type. It cannot be null.

Not just that, but it also abuses TimeSpan by co-opting its semantics to represent Forever. Really, the solution here is to use a union structure, TimeSpan | Forever. This should mean the range of 24 days or literally forever. Alternatively, Forever can be: (1) Implementation defined (2) Defined as being at least as great as TimeSpan.MaxValue's value in a specific version of the SDK, etc. The sin being committed here is relying on an implementation detail, rather than an interface.

This article feels more like

This article feels more like a rant that doesn't really explore what API design is, rather, it just presents some of the author's experiences about what is good and bad.

To me, API design is very similar to language design, just at a different level: you don't get to declare new syntax when doing API design while the semantics of an API abstraction are often drastically more complicated than a language abstraction. API design is also very much "design" in that it is as much as an artistic and social experience as it is an engineering experience.

Studying API design is then much like studying architecture: you study the designs of existing APIs, good and bad, and practice by writing a lot of APIs. Invariably, your early APIs will suck even though people will probably use them (if you are learning on the job), so perhaps an apprenticeship system would be more appropriate like they have in architecture (good idea for programming in general).

The really good API designers are probably the ones who wrote the ugly C APIs of the 70s or the increasingly better APIs of the 80s, but they've learned a lot from practice and are doing really beautiful APIs now (probably at places like Google). Unfortunately, there isn't enough of those guys to go around, so here we are with sucky APIs again...starting the cycle all over again.

The really good API

The really good API designers are probably the ones who wrote the ugly C APIs of the 70s or the increasingly better APIs of the 80s, but they've learned a lot from practice and are doing really beautiful APIs now (probably at places like Google).

Not necessarily. What you are saying is that great designers teach themselves. As Ben Franklin once said, "he who teaches himself hath a fool for a master."

When paying attention to who is being hired where, you should pay attention to where the top technical wizards go. For example, Bruce Blumberg, one of the ingenius minds behind NeXT's Interface Builder and GUI API, now works at MIT doing research on character synthesis.

Another example: Contrary to the default belief of some I've encountered (and then corrected), Gavin King has never programmed in C++. He was a Smalltalk guy, and when he invented Hibernate ORM, he did so because he knew from the Smalltalk community that this was the right way to manage the Data Access Layer of extremely complex enterprise systems.

Likewise, people who build good component-oriented OO systems seem to have a consistent background in Delphi. It seems like Microsoft now has a lot of ex-Borland employees or ex-Smalltalk programmers working for them on their IDE products.

Unfortunately, there isn't enough of those guys to go around, so here we are with sucky APIs again...starting the cycle all over again.

<HotButton>
I've offered to help Microsoft design and/or improve an API in the past, and they wanted me to sign a non-disclosure agreement. This is just not my way. I'll use proprietary products and even build them, but I will never do pro bono design-level work for a company whose business model is proprietary software... as long as they think that private conversation about design among a few people is superior to public input. Creativity is draining, and NDAs just compound that drain. My policy is "look, I'll do this for free, but make the conversation open."
</HotButton>

Hibernate API

Another example: Contrary to the default belief of some I've
> encountered (and then corrected), Gavin King has never programmed
> in C++. He was a Smalltalk guy, and when he invented Hibernate ORM,
> he did so because he knew from the Smalltalk community that this
> was the right way to manage the Data Access Layer of extremely
> complex enterprise systems.

Actually, he never programmed Smalltalk either. He was hired just as we were (sob!) converting from Smalltalk to Java, and spent all of his time doing Java+Corba until we hit the wall of bankruptcy.

I never had much to do with our DB layer, but I do recall other ex co-workers saying the design of Hibernate was very much borrowed from our in-house Smalltalk DB layer, so you're right about that.

Quoting Gavin King: And FYI,

Quoting Gavin King:

And FYI, I learned OO from SmallTalk. I have never been a C or C++ developer.

It's possible he borrowed design ideas from elsewhere. The spirit of my post was that great designers rarely learn from their own mistakes inasmuch as they learn from other great designers, so I'm not bothered to hear he "very much borrowed from [an] in-house Smalltalk DB layer [somebody else designed]". It's a catch-22, and many here might want to refute this point. However, it is very consistent with Fred Brooks' Mythical Man Month advice: highlight design talent early (when they're young and still very much impressionable to good habits), and give them an apprenticeship with your current top master designers.

J/w... are you Alex "Teapot" Belits?

Gavin King + Hibernate

And FYI, I learned OO from SmallTalk. I have never been a C or C++ developer.

Which is true - but sadly, Smalltalk was going out the door as he came in. I don't think that's the only reason the company died, but I can't think that deciding to rewrite the whole code base in Java did wonders for us. I don't recall him working on any Smalltalk stuff, but it was a good place to soak up some knowledge.

"very much borrowed (the design)

Doesn't bother me either - good designs are there to be (re)used.

J/w... are you Alex "Teapot" Belits?

Afraid not.

Intrinsic vs. extrinsic design

I have made the experience that a lot of bad API designs are - at least in part - caused by not abstracting from the underlying implementation in the right way. By "in the right way" I do not mean the level of abstraction but simply the way the functionality is exposed. Oftentimes I find the interfaces to be informed by the structure of the implementation rather by how one should write good client code.

Thus, when designing an API, one might want to contemplate a hypothetical client application first in order to observe what is really needed in order to write good client code for the future API. In other words APIs should be designed more top-down rather than bottom-up. Of course, this requires that one has a realistic idea of what clients will want to do with the API.

A nice side effect of doing it this way is that you will already have example code to go with your documentation.

Basic software engineering?

If you read any decently modern textbook on software engineering (i.e. one which spends sufficient time on requirements analysis), you'll see a huge emphasis on 'use cases' as one of the primary artifacts at the requirements stage [sometimes different names are used than 'use case', like 'scenario', but they are all the same, the different names are an artifact of publish-or-perish and from different segments of CS developing different terminology in parallel]. Since API design is part of software design, it is only natural that coming up with use cases will lead to better APIs - for exactly the reasons you outline: they will be designed to follow good usage patterns rather than bottom-up.

I am still constantly astonished by how little of that seems to be taught, even in programs specializing in software engineering.

These names do mean different things

[sometimes different names are used than 'use case', like 'scenario', but they are all the same, the different names are an artifact of publish-or-perish and from different segments of CS developing different terminology in parallel]

While I don't disagree that publish-or-perish can cause a proliferation of terminology, it is not the case here. All of this terminology came first from the UML community, and later on more terminology came from the Agile community in response to the UML community's heavier ceremony. These communities aren't different segments of CS, but rather people who prefer different methodologies for different projects.

In a nutshell:

Use case is heavyweight, and used to precisely specify things. Scenarios break down a use case into certain situations a use case might address. [It does not help much that .NET Framework Design Guidelines broadly refers to this as "Scenario-Driven Design".]

User story is lightweight. It states a very broad intention of what the code should do, not how to do it. The 'how' part is worked out through experimentation and prototyping and consistent feedback with the customer. It is a reaction to Big Design Up Front-style specifications that are written in stone. Basically, the best way to prevent "written on stone tablet" specs is to make sure the spec can constantly change. In XP, this means every two weeks you re-visit the spec.

Finally,

The point behind use case, scenario and user story is always the same: To test how maintainable ('decoupled') the code is and how well it can achieve everything the user has asked for. It tests how well the programmer understands the problem domain expert's needs. If you have a use case and only one type or class is being used to complete that use case, then chances are you are not building a modular solution. Thus, you can generally see a maintainable solution by looking at the use cases and the number of classes that collaborate to achieve it. In this sense, we seek less to model the 'real world' and seek more to model 'plug-and-play' modularity that allows us to swap components for one another.

Much older

Go read the software engineering literature from the mid-70s to mid-80s and you'll see use cases, scenarios and so on. They'll be called something different, but this is not new. The OO people just gave it a fresh coat of paint in the mid 90s, thinking they'd discovered something. The basic problem is that there are very few 'real' software engineers around; by this I don't mean self-taught software developers who have figured out a lot of this for themselves, but rather people really trained in 'software engineering'. Especially paradigm-independent software engineers - those are really rare.

And the point of use cases, scenarios and user stories is exactly the opposite of what you claim: they are there solely to capture user requirements. That they can be used to generate tests is rather nice, but that's not their purpose at all.

I refactor my code all the time, I do some test-driven development, I modify my specs (and APIs) when I need to -- and I haven't written a line of OO code in 10 years, and never have written a single UML diagram [and hope I never have to]. Properly done use cases, scenarios and user stories can just as easily be used for driving the design of Haskell code as they can for any other language/paradigm.

Jackson Development

Sorry for implying you need to know UML to do this stuff. I know plenty of situations where UML is uncalled for.

As for the 70s: Yes! I own a lot of software engineering texts from this era. Michael Jackson called these "problem frames". I believe CMU still uses a book he wrote in the 1970s, but am not positive about this (however, it was definitely used as recently as the last decade, exemplifying the staying power of the text). There's also other classical methodologies, like Gane&Sarsen. Yourdon and Constantine had their own definitions, but they focused very much on the plug-and-play stuff.

What the OO people did was take methodology and deeply integrate it into programming implementation. This resulted in the 3GL / 4GL schism between UML and all OOPLs, and as a result it seems that many people learn OO from the perspective of the capabilities of an OO language, as opposed to general systems thinking about plug-and-play design.

As an aside, there's this essay by Grady Booch where he disects three major approaches to gathering requirements for the design of an OO system. It's a pretty neat essay, where he argues certain people are predisposed to want to explain and think about designs in certain ways. I'm trying to dig it up, because I thought about it when the end of Henning's essay finished with cultural speculation... the essay I have in mind is published in Best of Booch.

The basic problem is that there are very few 'real' software engineers around; by this I don't mean self-taught software developers who have figured out a lot of this for themselves, but rather people really trained in 'software engineering'. Especially paradigm-independent software engineers - those are really rare.

What software engineering programs do you think turn out 'real' software engineers? The only Masters program I reviewed that impressed me was McGill (Montreal, Canada). I am effectively self-trained. My undergrad CS diploma is pretty watered-down. I taught myself via reading ACM Portal papers... I started during an boring Operating Systems class with a professor that sounded like she didn't know anything about OSes. But she said Dijkstra was a badass, so I started by reading his papers. From there I basically just bought books to teach myself, since our CS library was pretty pathetic. It's pretty expensive to teach oneself via books, though.

Booch's essay The Macro Process

Approaches To Process
Alas, I cannot make the same positive statement with regard to process: there exists considerable diversity in thought over how the object-oriented project should conduct its business. To that end, I roughly divide the object-oriented world into five somewhat overlapping schools of philosophy regarding process:

  • The anarchists, who largely ignore process and prefer to be left alone with their computers.
  • The behaviorists, whose focus is upon identifying and then evolving roles and responsibilities.
  • The storyboarders, who see the world in terms of scenarios.
  • The information modelers, who focus on data and treat behavior as secondary.
  • The architects, who focus upon frameworks (both building and using them).

I've found this to rather accurately describe the 'societies' within large software organizations.

The social realities of APIs

Since API design is part of software design, it is only natural that coming up with use cases will lead to better APIs - for exactly the reasons you outline: they will be designed to follow good usage patterns rather than bottom-up.

You seem to be assuming that the only culprit for poor APIs is lack of experience, knowledge and skill on the part of the programmers, but I would argue that those factors account for at most half the problem.

Many of the problems of software design in general and API development in particular are sociological.

Some examples:

  • APIs are often released "before their time" for business or organizational reasons
  • once an API is released, changing it becomes very difficult, because some important groups are now "depending" on the broken features you want to fix
  • the people whose use cases were the basis of the design are not the people who are now using the end result
  • there were two or more camps of users with conflicting goals contributing to the use cases
  • APIs are often an afterthought, viewed as an "advanced" feature to be used by masochists, er, I mean programmers. Most development effort goes into the "user-friendly" components, i.e. those intended for less technical people.

This is just a sampling; I could go on and on. Requirements analysis is as much political art as technical discipline.

It is interesting to contrast this situation with PL design, where there is often an identifiable designer or group of designers with explicitly technical goals that can be assessed directly on technical merits: no wonder everyone wants to develop their own language. ;-)

I agree

I over-simplified things - thank you for your thoughtful post. I agree that all of the factors you mention are extremely important forces in the development of APIs. I have witnessed every single one of them 'at work' when I was doing professional software development.

To bring a PL angle: think how much worse API design is in 'dynamic languages' with strong introspection facilities. These languages (like Ruby, Python, but also Maple) have their data-structures completely exposed, as there are no proper information hiding mechanisms that are guaranteed to be opaque. [I guess I should throw in Java bytecode and .NET's CIL in here too]. Mistakes in API design and data-structure design live on forever, because of powerful pressures to maintain 100% backward compatibility.

My experience was that, if the developers had ensured that they had sufficient use cases (scenarios, user stories, etc) in hand, they would have designed a better API. They were certainly competent enough to do so, they just didn't have the right 'raw data' to do a good job. The APIs were frequently designed with some abstract elegance goal in mind, but never tested for proper composability and/or usefulness for a variety of tasks. But they sure were elegant.

Collaboration and out-of-band introspection of unencapsulated...

...details

To bring a PL angle: think how much worse API design is in 'dynamic languages' with strong introspection facilities. These languages (like Ruby, Python, but also Maple) have their data-structures completely exposed, as there are no proper information hiding mechanisms that are guaranteed to be opaque.

Another way to state this is that in a language like ActionScript you are not forced to expose an interface that programmer's can see on a diagram, such as a UML Class Diagram or UML Collaboration Diagram. ActionScript literally allows you to use the object dictionary model to circumvent 'private' keyword; even though ActionScript supports the private keyword there is no real notion of a private variable. This has the effect of allowing the API designer to have out-of-band class collaboration information that client API users do not.

Maple is a joke, so I won't even comment on it in detail. They completely broke their graphical input system at one major release of the language!

To bring a PL angle: think

To bring a PL angle: think how much worse API design is in 'dynamic languages' with strong introspection facilities. These languages (like Ruby, Python, but also Maple) have their data-structures completely exposed, as there are no proper information hiding mechanisms that are guaranteed to be opaque.

Not sure if this is of any interest but my opinion about this is almost the exact opposite. Can't speak of Maple but data-structures and APIs in Python and Ruby exhibit growth pattern, which is compensated in languages like Java using XML extensively. By no means does it imply that this leads to elegant designs by virtue of the languages, but hardly ones that are in the programmers way and lead to misery because it is too rigid and changes can't be applied without breaking lots of dependencies. Moreover types aren't all that important. If some function expects a file one can pass something which is just file-like in the relevant aspects.

So maybe API design doesn't matter that much after all when we can live also with weaker designs.

The relevant aspects

If some function expects a file one can pass something which is just file-like in the relevant aspects.

The problem is that you often have no way to tell from the outside what the "relevant aspects" might be, especially when the documentation is vague or non-existent.

One of my peeves when I was doing the Python Challenge was that different libraries were somewhat inconsistent about, for example, their containers, so that a traversal that worked for one container would fail for another.

Generally the docs weren't specific enough to avoid this problem, and there was no help from the compiler, so basically you had to try something, find that it blew up, and then trial and error to work around the problem.

While many people are perfectly happy with this style of development, I can't see that anyone would be able to convincingly argue that this is "good API design".

So maybe API design doesn't

So maybe API design doesn't matter that much after all when we can live also with weaker designs.

I've never been happy with weak designs.

I've said already, people only care about bad APIs when it pisses them off. I should add that they also need to feel pain. It has been my experience that Java GUI programmers do not know what pain is, even after being kicked in the stomach by a Chuck Norris roundhouse kick. Swing simply attracts programmers who naturally secrete morphine instead of gastrointestinal muscosal; they have stomach ulcers from coding with such bad APIs, but the pain killers make them totally oblivious to this awful health problem.

I just happen to be the sort of developer who is incensed by bad design, in every way imaginable. I don't even like living in an ugly place. So just about every API pisses me off and every API is an opportunity to make developer lives everywhere easier!

When I talk about weak

When I talk about weak design I mean something like Rubys non-orthogonal libraries. Python was already mentioned by Marc. My first programming language was Mathematicas scripting language. API design? Flat kernel library with powerful functions. A more extreme example of the same non-design approach to APIs is PHP. I never used PHP and heard lots of horror stories about it. However, apparently mediocre programmers were able to create most of the web applications which matter for me using PHP. How could that happen?

Java Swing ranges at the opposite and is strong design.

Java Swing ranges at the

Java Swing ranges at the opposite and is strong design.

I have to disagree. In fact, the Program Managers on the Swing project, such as Dean Iverson, tried to improve Swing to correct many of its accidental complexity flaws. This is part of what morphed into the "Swing Application Framework".

I could go into more detail as to how Swing is an abomination, if you'd like. Dean has only limited examples on his blog, such as how tedious it is to set-up a Hello World Swing app. Swing does not follow a basic OO practice: Hide the sequence. Instead, everyone has to write code to manually hide a sequence that Swing's programmers didn't bother to hide for you. Finally, if it is any indication, all of the public-facing talent on Swing's team is now gone to Adobe and Google. The guy they brought in for JavaFX came from a separate division of the company, because they had to start over. Sun's Desktop division has been consistently underfunded for years. They couldn't even get proper antialiased text into Java until JDK 6.

Further, Swing does not have any way to declaratively define layout constraints. This led to Karsten Lentzsch creating a bunch of layouts for developers so that Swing UI's weren't unbearably ugly. However, if you wanted to refactor from one layout to another, you had to do a lot of moving of code mountains. Even refactoring IDEs aren't prepared to reshuffle thousands of layout constraints.

Also, binding in Swing is awful. It's not even binding. Model definition is ugly and it forces you to write wrapper types everywhere. Likewise, the interface for event listening typically requires a similar listener adapter wrapped around it so that you don't write excessive boilerplate.

Finally, extending Java Text editing is ridiculous. Is it any wonder Eclipse is implemented in SWT? At least it is marginally better.

John, let's continue this

John, let's continue this discussion another time. My thoughts are not cleaned up and seeking to troll, just because I find the extreme reduction of complexity in qualifications such as "good design" and "bad design" hilarious, isn't adequate either.

Just to close from my side with a little background. Right now I'm writing some sort of IDE for DLR languages in Silverlight using C#. It is obvious to ask how to improve accessibility of the Silverlight API for Ruby/Python users who might want to avoid using Xaml and program small dialogs and applications directly in interactive style - not much unlike they'd do in an environment such as Mathematica. Finding a model for "programming in the small" is something I'm interested in here rather than SE / "programming in the large", which is approached more adequately using different tools.

Previous Discussions

Previous Discussions on API Design when reviewing Practical API Design by Jaroslav Tulach:

  1. Languages ready for API Evolution
  2. Review of Practical API Design by Jaroslav Tulach

Paradoxes of API Design

Thanks for sharing the pointers to discussions that I was involved at.

Recently I managed to make one of my presentations video taped. In case you have one and half hour of spare time, feel free to give my Paradoxes of API Design video a chance.

Comments and publicity is welcomed!

It's interesting that

It's interesting that Microsoft makes such API usability studies and ends up providing schizoid APIs as like in WPF. In that particular case Xaml users are allowed to set a color value to "Red" or "#FF000000" whereas the code behind programmer must write

txtbox.Background = new SolidColorBrush(Colors.Red);

or

txtbox.Background = new SolidColorBrush(Color.FromArgb(0xFF, 0x00, 0x00, 0x00));

without any chance that WPF does the dirty work of coercing the plain value on assignment. Moving from Xaml to C# or other .NET languages breaks almost every single expectation about setting property values. Since WPF defines a GridLength object for grid lengths, a Thickness object for border thicknesses and so one must wonder that the height of a control is determined by a double and not by a Height object.

The code behind design is surely not a "mistake" or lacks abstraction and extensibility. On the contrary. This is good style in all manners of configuration over convention and with Visual Studios super-smart type aware intellisense which often hints to the right constructor while typing even clean design becomes bearable.

To be fair,

What the C# language lacks is support for compile-time literals via a CompilerService. C# 3.0 was supposed to have XML literal support. However, what you really want is XAML literal support. Why? Because hierarchical syntax like XML is the most natural way to define a GoF Builder Pattern. Somebody on the Apache Jetty project did analysis of this and it turns out XML is just a very succinct DSL for this sort of stuff.

Anyway, this usability problem you point out is only somewhat true. In my books, the Original Sins in WPF are:

1) No "Syntax for hashes" for XAML... why create another .NET language without "syntax for hashes"... we already have VB.NET and C#
2) Not creating a lot of the dependency analysis subsystem as a CLR intrinsic
3) and then not providing C# and VB.NET syntax for this intrinsic - in short, both C# and VB.NET could arguably support some data flow syntax.

On the other hand, when WPF was designed, we knew a lot less about PLT and even software engineering... for example, FRP was only a few years old.

If you read what I've just said carefully, you'll note that Scala supports everything I just said. You should ponder how good Sean McDirmid's Bling library would be if he had Scala.NET at his disposal instead of C#. -- Your examples would be trivial in a Scala version of Bling.

What did we DEFINITELY know about software engineering that WPF blatantly ignored? Complexity cycles. A basic structured programming and OO metric. WPF violates dependency structure everywhere. Ask the Ximian guys who built Moonlight (after first trying to implement WPF) how much of a pain this is. You simply CANNOT rebuild WPF. It is too stupid and ugly for anyone to re-implement. Silverlight, on the other hand, is much nicer. However, people at Microsoft, in the name of "usability", are doing their best to turn Silverlight back into WPF again. Why, I don't know. It's sad. I predicted everything that would happen as a fallout of .NET 3.0. I knew WF would be re-written from the ground up and that XAML would hack in support for WCF Dependency subsystem to talk to the WPF Dependency subsystem. It's just too complex and stupid. Anyway, end of rant.

To be fair, the WPF

To be fair, the WPF designers probably had many concerns to balance and they had a mandate to build a designer-friendly UI toolkit with much more aggressive goals than past toolkits. We will see how this pans out, though WPF does seem to be pretty popular with designers (because of Blend, which is enabled by XAML, which drives a lot of the complexity in WPF's library design).

If you read what I've just said carefully, you'll note that Scala supports everything I just said. You should ponder how good Sean McDirmid's Bling library would be if he had Scala.NET at his disposal instead of C#. -- Your examples would be trivial in a Scala version of Bling.

If there was a decent version of Scala.NET I would be back in the Scala community tomorrow, if they would take me back that is! That being said C# isn't that bad, I can see it getting just a few features (case classes, traits, a bit more aggressive type inference, more expressive implicit conversions) and my desire to use Scala would melt away.

You simply CANNOT rebuild WPF. It is too stupid and ugly for anyone to re-implement. Silverlight, on the other hand, is much nicer.

I prefer WPF to Silverlight because Silverlight is much more XAML biased than WPF--they are really pushing the designer story more I guess. But...I think I'm more at harmony with WPF because I only use 20% of it because of Bling.

It's interesting that

It's interesting that Microsoft makes such API usability studies and ends up providing schizoid APIs as like in WPF. In that particular case Xaml users are allowed to set a color value to "Red" or "#FF000000" whereas the code behind programmer must write...

Users of Bling (a C#-based wrapper around Bling) can write just:

txtbox.Background = Colors.Red;

Because there is an implicit conversion from Color to BrushBl. The only reason the code behind story sucks in WPF is because it just wasn't given much attention! (Bling fixes that).

Since WPF defines a GridLength object for grid lengths, a Thickness object for border thicknesses and so one must wonder that the height of a control is determined by a double and not by a Height object.

Again, solved by an implicit conversion from double or DoubleBl to ThicknessBl (where d is defined to convert to (d,d,d,d)).

The code behind design is surely not a "mistake" or lacks abstraction and extensibility. On the contrary. This is good style in all manners of configuration over convention and with Visual Studios super-smart type aware intellisense which often hints to the right constructor while typing even clean design becomes bearable.

I would disagree with this. Code behind in WPF is just not very elegant, they should have pushed more functionality into declarative land or added more meta-programming style sugar to code behind so that it was easier to deal with.

I know about Bling. Right

I know about Bling. Right now I'm using Silverlight which shares the particular API I was talking about with WPF. So there was no reason to make a distinction in my post.

Anyway, we still talk about workarounds and wrappers. I can create a wrapper for each single DLR language which enables overwriting attribute settings on classes generically and this wouldn't be hard to program but it is still work used to correct an API which is designed correctly but experienced as cumbersome.

I know MS had labor division between designers and programmers in mind just like big software engineering and provides excellent tools in support but since I'm in DLR territory and work out some sort of web based "Python/Ruby programming service" for casual users who might be neither graphic designers nor professional .NET programmers, a steep learning curve for a complex toolkit doesn't feel appropriate.

Twisty maze of classes

I know MS had labor division between designers and programmers in mind just like big software engineering and provides excellent tools in support but since I'm in DLR territory and work out some sort of web based "Python/Ruby programming service" for casual users who might be neither graphic designers nor professional .NET programmers, a steep learning curve for a complex toolkit doesn't feel appropriate.

The problem, more aptly coined by Jonathan Edwards as a "Twisty maze of classes," is maybe that frameworks are too big for people to take in very easily, that you would feel overwhelmed the first time you encounter a library like say WPF with lots of chunky abstractions. The solution, I think, is to go with more simple and general abstractions; e.g., rather than have panels to do layout, people could just specify their layouts directly using canvas + data binding. Same with storyboards and styling and...whatever, that is what Bling is about...providing a more simple way of programming WPF. Then with fewer abstractions to learn, the library should be quicker to learn and more appropriate for casual usage.

The solution, I think, is to

The solution, I think, is to go with more simple and general abstractions; e.g., rather than have panels to do layout, people could just specify their layouts directly using canvas + data binding.

Yes, that's also why Flash, even without data binding or Bling style constraints, is much easier to use than Silverlight. It is an animation toolkit with GUI elements being an afterthought. At the core it is about a bunch of moving and shape changing rectangles and geometrical and topological relationships between them which can be always queried. One knows which rectangles are under a certain point, there are enclosing rectangles and so on. No matter how everything is layered one has access to the fundamentals and a simple cognitive model. Layouts are then used to express or constrain relationships between those rectangles. Advancing this in the direction of physics engines and eventually non-linear relationships seems plausible to me.

I'm not too concerned when data binding is completely expressed in callback style instead of declaratively which is somewhat more powerful. Data binding seems to be a natural extension of the concept of user events. So one binds to changes of arbitrary properties of some type ( "dependency properties" ) instead of just mouse clicks or key strokes.

I'll see what's doable for DLR languages in case of Silverlight in the next couple of weeks.

This reminds me of how we

This reminds me of how we did animation in Java2D via immediate mode rendering: just put the objects in a different place on each time step.

Its amazing that doing things the "right way" via dedicated abstractions (storyboards) is more difficult to learn than doing something the direct way without dedicated abstraction. Surely storyboards provides more headway to do more complicated stuff (synchronize scenes with actions, framerate independence), but they actually make doing simple animations harder.

API designers should keep that in mind.

I'm not too concerned when data binding is completely expressed in callback style instead of declaratively which is somewhat more powerful. Data binding seems to be a natural extension of the concept of user events. So one binds to changes of arbitrary properties of some type ( "dependency properties" ) instead of just mouse clicks or key strokes.

I don't see databinding like this. An event stream is just one stream of events, a dependency property can be said to have a value + a stream of changed events. But if the property was just that, there would be no data binding involved. Instead, data binding is a continuous relationship, even if, as an encapsulated detail, the data binding is implemented by subscribing to the discrete stream of changed events. I think treating databinding as just another form of event stream is likely to remove its elegance, making it as ugly/complicated as raw event handling.

This reminds me of how we

This reminds me of how we did animation in Java2D via immediate mode rendering: just put the objects in a different place on each time step.
Its amazing that doing things the "right way" via dedicated abstractions (storyboards) is more difficult to learn than doing something the direct way without dedicated abstraction. Surely storyboards provides more headway to do more complicated stuff (synchronize scenes with actions, framerate independence), but they actually make doing simple animations harder.
API designers should keep that in mind.

Processing.org has the good old immediate mode animation approach. You set your animation parameters in a few global variables. You update them at each time step and draw. This may not be the best way to do complex animations but remember that humans are smart. We can do complicated animations easily even in this simplistic rendering model. It takes less time to do the animation you want this way than to learn a complicated API to do the animation the "right" way. For example I was able to write simple electromechanics simulations that display the field/potential very easily. Would a more sophisticated animation model have helped? Perhaps, but the time to learn the API would be far far longer than the time to code it up in Processing.

<rant>

I've been coding a structured editor with WPF and as much as I want to like WPF it is a huge pain. The effort required is comparable to painting the pixels myself like in Processing. This is partly due to WPF being far too complicated which means that I'm constantly looking at (poor quality and incomplet) documentation. Getting all the nodes in the tree to display properly is difficult. Part of the problem is C#. For example getting the right display settings to every node in the AST requires either polluting the code by passing objects around the entire program OR hard coding them which means that tuning the display settings is very time consuming because every change requires a long recompilation (and one setting is duplicated in multiple places).

Is the precooked widgets-with-tons-of-properties based approach that GUI libraries take really the right one for programs that aren't run-of-the-mill business apps?

On NOT creating "twisty maze of classes"

I found Jonathan's series of blog posts on WPF, Swing, and Flex to be some of the best cross-comparison blog posts by anyone on the Internet, and I've bookmarked them. I have also used WPF, Swing, SWT, and Curl, so I can probabl shed some additional light.

The problem, more aptly coined by Jonathan Edwards as a "Twisty maze of classes," is maybe that frameworks are too big for people to take in very easily, that you would feel overwhelmed the first time you encounter a library like say WPF with lots of chunky abstractions.

Sorry, but this was not Jonathan's point. In his blog post, You are in a maze of twisty classes, all alike, he criticizes the following flaws of WPF:

  • "[...] incredibly over-generalized and over-engineered [...]"
  • steep learning curve

There is nothing unique to frameworks here.

In his next blog post, Staying the course, Jonathan then goes on to say:

WPF reminds me of the early “component frameworks” in the 90’s like OLE, OpenDoc, and COM. These frameworks popularized the ideas of properties, events, reflection, and dynamic loading. Since the mainstream languages of the day (C and C++) could not handle these concepts directly, they had to be emulated in libraries with massively complex API’s. Eventually languages evolved to incorporate these features directly, making component programming much easier, as with JavaBeans and .NET. WPF is the same story all over again. It simulates a whole new layer of semantics involving declarative bindings between dynamically attached properties, a “style sheet” semantics for property overriding, “routed events”, a sub-language of time-varying functions, and a limited code-data isomorphism via XAML. The resulting complexity is really a plea for new programming language semantics.

I think we're also starting to see PLT come up with some solutions here, too. For example, Alexey Radul's Ph.D. thesis under Gerald Jay Sussman, Propagation Networks: A Flexible and Expressive Substrate for Computation is one example of looking at foundations for how to implement the automatic change propagation networks in WPF applications.

Finally, in his Beachhead Dispatch blog post, Jonathan provides further commentary on WPF, where he does a good job assessing WPF 3.0's major strengths and weaknesses. Weaknesses Jonathan observes center around XAML:

The biggest problems with WPF are due to its ambitious goals to make UI design more declarative and less code-centric. I applaud the intentions, but the results are decidedly mixed. One example of these problems is the styling and themeing system. UI controls have no hard-wired look and feel – it is completely specified in an XML dialect called XAML. This is extremely powerful and customizable. And unusable, because it is almost completely undocumented. For example, you can’t change the color of the border around a text box. Sure you can change the color – until the mouse travels over it, when it reverts to the themed color.

What Jonathan is describing is a serious architectural limitation of WPF 3.0. WPF 3.5 SP1 shipped with support for the VisualStateManager that solves this problem (more or less). Unfortunately, this attached property for WPF controls was posted on Expression Blend architect John Gossman's web page, and not really widely publicized by Microsoft. This move reminds me of the SwingWorker background thread pattern Sun had in their documentation as a necessary feature to have realistic performance, but not shipped as part of the SDK - and so a lot of lazy developers were never acquainted with it.

Note that in Jonathan's blog post, he mentions how the Microsoft Expression Blend team, led by John Gossman, has more or less solved all of his problems and that a major sticking point is that none of these solutions propagated back into mainstream WPF documentation.

What a fiasco. A UI framework that ships with a set of controls whose behavior is undocumented and not incrementally customizable. And this was done all in the name of easier customization. Microsoft’s Blend design app solves this problem by providing a complete new set of “simple controls”, defined by 1100 lines of XAML which replace all the undocumented XAML of the standard controls. The Blend docs explain how all these styles work, and the set of global resources they depend upon, so you can customize them as needed. I have adopted Blend’s styles for my UI, so I am satisfied that I can do what I need. But the WPF team should be embarrassed by the necessity of this giant workaround.

John is incidentally the person Microsoft put in charge of "subsetting" the WPF API so that Silverlight would not repeat the same mistakes as WPF. He has more or less been doing a phenomenal job, and the velocity of Silverlight releases -- Version 4.0 is coming out now -- is simply unreal. In fact, I think it is too fast, and you see some of WPF's bad design slowly creeping into Silverlight starting at the edges (NotificationWindow and NavigationWindow are both breathlessly bad designs from WPF that are more or less being copied in Silverlight).

However, despite these "fixes", WPF still suffers from one other major problem, which I am working on a blog post to elegantly explain. In short, they strongly violate proper structured programming practices by allowing dynamic injections to create a context between the caller and callee (Routed UI Commands are a good example, as they route business logic through the user interface; this is a strong violation of Clemens Szyperski's domaincasting principle for component-oriented systems, as you are sending a message through the wrong layer of the system). In my experience with visual programming, this is bad for rapid application development frameworks like Blend and lively kernel frameworks like Smalltalk IDEs. There is no way for the design tools to do static analysis and help the visual designer and programmer complement each other in their roles developing the application. Where I work, we do have a designer/developer user story, even though this role is filled by the same person 80% of the time. The reason for this schizophrenic separation is good design.

Bottom line: Frameworks are not too big for people to take in easily. Crazy dependencies are too big for people to take in easily. The first thing I did as an architect of a Silverlight/WPF cross-compiled application was not rely on WPF's internal dependency system, and instead replace as much of it as possible. The skeleton structure of our app thus behaves in much the way you explain Bling to be superior to what I call "The WPF Architect's Kitchen SInk".

Sorry, but this was not

Sorry, but this was not Jonathan's point. In his blog post, You are in a maze of twisty classes, all alike, he criticizes the following flaws of WPF:...

Sorry, maybe we are disagreeing on what "big" means. To me, big just means mentally complex, I didn't mean it in a physical size context.

There is nothing unique to frameworks here.

I would disagree. WPF is fairly representative of a contemporary framework. My next example would be Eclipse, which generally follows the same pattern (steep learning curve, over generalized and over engineered, if a bit better documented and engineered). I'm making some observations about why that is so, Jonathan has his own explanation, which I pretty much agree with, and I don't think my own spin on this problem is incompatible with what was said.

Bottom line: Frameworks are not too big for people to take in easily. Crazy dependencies are too big for people to take in easily. The first thing I did as an architect of a Silverlight/WPF cross-compiled application was not rely on WPF's internal dependency system, and instead replace as much of it as possible. The skeleton structure of our app thus behaves in much the way you explain Bling to be superior to what I call "The WPF Architect's Kitchen SInk".

All frameworks have crazy dependencies, if you don't agree with me, name one that doesn't! Crazy dependencies result from chunky abstractions, and you can't avoid them unless you instead design a bunch of small generic orthogonal abstractions. But in the latter case, you have something more like a language than a framework.

Everyone (including competing product groups) seems to believe that their own chunky abstractions are better than someone else's. This is BS, you just wind up replacing something you aren't familiar with with something that you are familiar with because you wrote it yourself, which makes a different set of trade offs that caters specifically to your needs. There are of course better designs, but the intrinsic complexity remains no matter how good the design is. Bling tries to break this cycle by just reskinning some of WPF's more generic abstractions (dependency properties, events, data binding) to be more usable so that they can replace the more hairy ones.

Please don't get me wrong, I'm interested in your explanation as I'm working on my own. I think you have specific recommendations on how to improve UI framework design, but can you generalize these recommendations? Even if we fix WPF, what about the next big framework? What about the hulking monstrosity known as Eclipse (beyond SWT) or even visual studio 2010 (whose APIs look similar)? What is fundamental to "better" API design?

Great points

As for generalizing these recommendations, yes... it's possible. Unfortunately, WPF cannot be saved in terms of "removing" complexity, just as Swing cannot be saved in terms of "removing" complexity. The best you can do is "guide" developers away from complexity and just outright shield them.

I've actually got three blog posts I wrote 7 months ago stored in the kitty, and haven't published them mainly because they are currently littered with too many comments like the ones I made to Kay about Chuck Norris roundhouse kicks. I try really hard to balance "dumbing things down" through use of hyperbole and correct use of design principles. -- This series of blog posts is aimed at establishing several key points, including why Silverlight is superior to WPF, both from a MS perspective and a Novell perspective. It breaks down the decisions the Silverlight team has had to make, guesses why they made them, and shows you why a lot of the features in WPF could be implemented using more fundamental abstractions. The second part of the series is supposed to be dedicated to looking at the frameworks shipped with .NET 3.0 as a whole. In my books, the only one MS "got right" on the first try was CardSpace, but unfortunately no one is using it and instead using inferior specs like OpenID.

What about the hulking monstrosity known as Eclipse (beyond SWT) or even visual studio 2010 (whose APIs look similar)? What is fundamental to "better" API design?

This is tricky, and something myself and a few others occasionally discuss. "Better" for some of these people means "4th generation languages", or at least "3rd generation languages" that solve all the problems with existing "3rd generation languages". Gilad Bracha's Newspeak is a good example of solving 3GL coupling concerns, and from Gilad's perspective, it is easy to understand why Erik Meijer says Gilad Is Right. The problem that Model-Driven Engineering enthusiasts have been working on for about 30 years is building robust 4GLs (that perform well!) and thus try to skip past this problem, but such a skip maneauver is tricky to pull off.

.NET itself is actually transitioning away from being solely about type systems. With Code Contracts, PeX and the new security transparency system in .NET 4, there are way more ways to specify a system than ever before. Now, for a hot button: I am not too happy with the design of the DLR. I personally don't understand the interface design choices made here. Then again, I don't also 100% like the interface design choices of the Object class in the core type system (but the CLR guys are quite proud of the Object class... there is a page on MSDN bragging about how this class has never had to change). For example, I think any design of the DLR (and re-design of WPF) should begin by redesigning the System.Uri class (easily one of the worst designed classes in all of .NET, and its monolithic design contributes to why it has the largest class overhead in the installation of Silverlight!). I don't know why you would build a dynamic language runtime without making every object in the system addressable via a Uri. (I still think .NET is vastly superior to Java on many levels, despite my heavy criticisms. -- My criticisms are a result of not taking crap from my tools or taking crap from tool vendors.)

Finally, looking at the past as a way to predict the future, Microsoft's UI innovations are the direct result of acquiring RenderMorphics Reality Lab in February 1995 (owned by three British students), which ended up helping shape DirectX 6... eventually, MS released DirectX 8, which was the first version of DirectX competitive with OpenGL. Fun fact: Kate Seekings, the designer of the API that led to Direct3D, flunked her masters course in computer graphics due to not strictly obeying the design specs the professor laid out. Since you are working on the next-gen MS stuff, a fun book for you to read could be Renegades of the Empire: How Three Software Warriors Started a Revolution Behind the Walls of Fortress. This covers the history of DirectX upto 1999.

James Waletzky

There's also this short article by James Waletzky of Microsoft: Perfect API Design

My main problem with advice like this is too much emphasis is placed on discoverability, rather than correct logical design. When you focus too much on discoverability, you forget about math and logic.

[Edit: James did not work on WPF, and cannot be blamed for the following API atrocity.]

WPF Example:

DependencyObject.SetValue and DependencyObject.GetValue are not symmetric and there is no reason why they should be asymmetric. I would argue that this asymmetry creates poor discoverability; if I set a value X, and then attempt to get that value back as X', then X should equal X'. However, this is NOT how WPF works! Why?

Two reasons.

1) DependencyProperty.UnsetValue is incorrectly designed. It returns a singleton that ultimately maps to a "new Object()" instruction. This is wrong - it's just as bad as writing C or C++ code that allows for a Dangling Pointer! The value returned by this field should map to an explicit singleton type: DependencyPropertyUnsetValue. To be clear, the API designer saw the problem with using "null" to represent UnsetValue, but rather than program to an interface, he/she programmed to an implementation: the universal type, in effect Any implementation imaginable. This makes it impossible to really program logic, such as triggers, to detect when a value equals UnsetValue, because GetValue can't return UnsetValue's type.

2) The non-exposed GetLocalValue method will return UnsetValue, but GetValue "pretends" objects always have a set value, even if it is the default. LocalValue behavior is only observable via LocalValueEnumerator, which is exposed via DependencyObject.GetLocalValueEnumerator

So is this API really discoverable? I'm told the reasons it was designed this way are forgotten, but that it was most likely in the name of "discoverability".

Language design vs library design

One of the epigraphs in Stroustrup's C++ book is a pair of quotes: "Library design is language design" -- ".. and vice versa." -- (both from 70s Bell Labs people.)

The boundary between language design and API design is fuzzy. I think the only real distinction between them lies in pervasiveness and permanence. To illustrate: many people write C++ programs; fewer people write C++ libraries; vastly fewer people write the *standard* C++ libraries; and only a vanishingly small percentage write the language itself. Yet all of it is notation.

So, I'm not sure that the thesis of this paper is very helpful. Does anybody enjoy poor API design? Of course not. But is poor API design any worse than poor language design? Is it a moral evil to build something which is less than perfect? Of course not. Let him cast the first stone .. (CORBA, anyone?)

It's good that he is encouraging people to do better, and even pointing out particular kinds of deficiencies -- that's very helpful.

I think that if he means to address a general problem, it confuses the issue to focus only on 'good API design': maybe it would be better to think about the science and craft of programming generally. I don't know why it's necessary to zero in on this somewhat arbitrary division of it, except to address specific practical problems.

It's certainly an important thing to study, as it is in any other art.

(His comments about estimation are very interesting, and a little depressing. Still, it's not necessarily a good habit to blame one's tools. I know a man who can accurately estimate software projects. I have never met anybody else with quite his ability, but I know it exists; so it's not impossible. I wish I could do it. For many people, that's the really hard problem!)