Lambda the Ultimate

His proposal is actually even worse than that: he's not proposing criminal penalties for bad APIs, he's proposing them for changing any APIs to anything deemed 'critical'; to quote: "It is irresponsible to allow a single company (let alone a single developer) to make changes to such critical APIs [he gives OpenSSL and the C library as examples] without external controls."

This is not just absurd, but also completely counter-productive. He himself notes that backwards compatibility concerns have prevented many APIs from being made rational: "This is also an example of how concerns about backward compatibility erode APIs over time: the API accumulates crud that, eventually, does more damage than the good it ever did by remaining backward compatible." And I can't imagine a worse incentive to backwards compatibility than putting someone in jail for changing an API! And I tend to think really good APIs only come via iteration - to use is .NET Select() example, it could easily be extended to support the cases he wants, now that the problems with the current one have been identified. It's very hard to create an API that is great from the start, especially in a new problem domain.

(I'm also baffled by his example of OpenSSL; should someone tell him that it's an open source project with almost all of the work done by about a dozen people who regularly replace or remove APIs?)

Well, in all fairness to the author, he puts in a big disclaimer that he isn't advocating criminal penalties, merely that he thinks they might be inevitable. My summary oversimplified the author's message quite a bit.

Of course, trying to compare construction with software engineering is a bit of a tortured analogy...

I've never heard anybody ever even bring up the concept of criminalizing "mission-critical" API changes. But not only is his construction analogy a huge stretch, but he also said:

"It is irresponsible to allow a single company (let alone a single developer) to make changes to such critical APIs without external controls."

External controls? Luckily the vast majority of developers would probably laugh at these ideas, but unfortunately there's probably a few fascist-like politicians that wouldn't mind enacting some kind of new government agency to oversee that nonsense.

I take particular exception to Henning's comment:

One would think that this is easy: simply call a Clone() method...

This is just rhetorical handwaving. "Simply call a Clone() method"? What are the semantics of that magical clone method (beyond the obvious "do what I need right now, exactly the way I want it done at the moment")? The reason for the "ICloneable considered harmful" sentiments in the blog post you cite is that there are no universally-appropriate semantics to something like Clone(); therefore any implementation will be subtly wrong in some (likely common) set of circumstances.

When critiquing an API design as being inelegant, citing the lack of a semantic monstrosity like Clone() doesn't bolster one's case: "this API makes it difficult to do something I want to do; a better design would assume I always want to do it this other way (and never differently)".

funny how there's no universally-appropriate semantics for equal in java/c# OOs as well?

I think his point is more that IClonable is practically worthless for writing code other programmers link to. Since it isn't very well defined, ICloneable doesn't add much to the API. Something like IDeepCloneable and IShallowCloneable would be better defined and clean up some of the inherent ambiguity of ICloneable.

"The time-out is provided as a Timespan (a type provided by .NET) that has resolution down to 100 nanoseconds, a range of more than 10 million days, and can be negative (or null) to cover the "wait forever" case."

Timespan is a struct, a value type. It cannot be null.

Not just that, but it also abuses TimeSpan by co-opting its semantics to represent Forever. Really, the solution here is to use a union structure, TimeSpan | Forever. This should mean the range of 24 days or literally forever. Alternatively, Forever can be: (1) Implementation defined (2) Defined as being at least as great as TimeSpan.MaxValue's value in a specific version of the SDK, etc. The sin being committed here is relying on an implementation detail, rather than an interface.

The really good API designers are probably the ones who wrote the ugly C APIs of the 70s or the increasingly better APIs of the 80s, but they've learned a lot from practice and are doing really beautiful APIs now (probably at places like Google).

Not necessarily. What you are saying is that great designers teach themselves. As Ben Franklin once said, "he who teaches himself hath a fool for a master."

When paying attention to who is being hired where, you should pay attention to where the top technical wizards go. For example, Bruce Blumberg, one of the ingenius minds behind NeXT's Interface Builder and GUI API, now works at MIT doing research on character synthesis.

Another example: Contrary to the default belief of some I've encountered (and then corrected), Gavin King has never programmed in C++. He was a Smalltalk guy, and when he invented Hibernate ORM, he did so because he knew from the Smalltalk community that this was the right way to manage the Data Access Layer of extremely complex enterprise systems.

Likewise, people who build good component-oriented OO systems seem to have a consistent background in Delphi. It seems like Microsoft now has a lot of ex-Borland employees or ex-Smalltalk programmers working for them on their IDE products.

Unfortunately, there isn't enough of those guys to go around, so here we are with sucky APIs again...starting the cycle all over again.

<HotButton>
I've offered to help Microsoft design and/or improve an API in the past, and they wanted me to sign a non-disclosure agreement. This is just not my way. I'll use proprietary products and even build them, but I will never do pro bono design-level work for a company whose business model is proprietary software... as long as they think that private conversation about design among a few people is superior to public input. Creativity is draining, and NDAs just compound that drain. My policy is "look, I'll do this for free, but make the conversation open."
</HotButton>

Actually, he never programmed Smalltalk either. He was hired just as we were (sob!) converting from Smalltalk to Java, and spent all of his time doing Java+Corba until we hit the wall of bankruptcy.

I never had much to do with our DB layer, but I do recall other ex co-workers saying the design of Hibernate was very much borrowed from our in-house Smalltalk DB layer, so you're right about that.

Quoting Gavin King:

And FYI, I learned OO from SmallTalk. I have never been a C or C++ developer.

It's possible he borrowed design ideas from elsewhere. The spirit of my post was that great designers rarely learn from their own mistakes inasmuch as they learn from other great designers, so I'm not bothered to hear he "very much borrowed from [an] in-house Smalltalk DB layer [somebody else designed]". It's a catch-22, and many here might want to refute this point. However, it is very consistent with Fred Brooks' Mythical Man Month advice: highlight design talent early (when they're young and still very much impressionable to good habits), and give them an apprenticeship with your current top master designers.

J/w... are you Alex "Teapot" Belits?

Which is true - but sadly, Smalltalk was going out the door as he came in. I don't think that's the only reason the company died, but I can't think that deciding to rewrite the whole code base in Java did wonders for us. I don't recall him working on any Smalltalk stuff, but it was a good place to soak up some knowledge.

Doesn't bother me either - good designs are there to be (re)used.

Afraid not.

If you read any decently modern textbook on software engineering (i.e. one which spends sufficient time on requirements analysis), you'll see a huge emphasis on 'use cases' as one of the primary artifacts at the requirements stage [sometimes different names are used than 'use case', like 'scenario', but they are all the same, the different names are an artifact of publish-or-perish and from different segments of CS developing different terminology in parallel]. Since API design is part of software design, it is only natural that coming up with use cases will lead to better APIs - for exactly the reasons you outline: they will be designed to follow good usage patterns rather than bottom-up.

I am still constantly astonished by how little of that seems to be taught, even in programs specializing in software engineering.

[sometimes different names are used than 'use case', like 'scenario', but they are all the same, the different names are an artifact of publish-or-perish and from different segments of CS developing different terminology in parallel]

While I don't disagree that publish-or-perish can cause a proliferation of terminology, it is not the case here. All of this terminology came first from the UML community, and later on more terminology came from the Agile community in response to the UML community's heavier ceremony. These communities aren't different segments of CS, but rather people who prefer different methodologies for different projects.

In a nutshell:

Use case is heavyweight, and used to precisely specify things. Scenarios break down a use case into certain situations a use case might address. [It does not help much that .NET Framework Design Guidelines broadly refers to this as "Scenario-Driven Design".]

User story is lightweight. It states a very broad intention of what the code should do, not how to do it. The 'how' part is worked out through experimentation and prototyping and consistent feedback with the customer. It is a reaction to Big Design Up Front-style specifications that are written in stone. Basically, the best way to prevent "written on stone tablet" specs is to make sure the spec can constantly change. In XP, this means every two weeks you re-visit the spec.

Finally,

The point behind use case, scenario and user story is always the same: To test how maintainable ('decoupled') the code is and how well it can achieve everything the user has asked for. It tests how well the programmer understands the problem domain expert's needs. If you have a use case and only one type or class is being used to complete that use case, then chances are you are not building a modular solution. Thus, you can generally see a maintainable solution by looking at the use cases and the number of classes that collaborate to achieve it. In this sense, we seek less to model the 'real world' and seek more to model 'plug-and-play' modularity that allows us to swap components for one another.

Go read the software engineering literature from the mid-70s to mid-80s and you'll see use cases, scenarios and so on. They'll be called something different, but this is not new. The OO people just gave it a fresh coat of paint in the mid 90s, thinking they'd discovered something. The basic problem is that there are very few 'real' software engineers around; by this I don't mean self-taught software developers who have figured out a lot of this for themselves, but rather people really trained in 'software engineering'. Especially paradigm-independent software engineers - those are really rare.

And the point of use cases, scenarios and user stories is exactly the opposite of what you claim: they are there solely to capture user requirements. That they can be used to generate tests is rather nice, but that's not their purpose at all.

I refactor my code all the time, I do some test-driven development, I modify my specs (and APIs) when I need to -- and I haven't written a line of OO code in 10 years, and never have written a single UML diagram [and hope I never have to]. Properly done use cases, scenarios and user stories can just as easily be used for driving the design of Haskell code as they can for any other language/paradigm.

Sorry for implying you need to know UML to do this stuff. I know plenty of situations where UML is uncalled for.

As for the 70s: Yes! I own a lot of software engineering texts from this era. Michael Jackson called these "problem frames". I believe CMU still uses a book he wrote in the 1970s, but am not positive about this (however, it was definitely used as recently as the last decade, exemplifying the staying power of the text). There's also other classical methodologies, like Gane&Sarsen. Yourdon and Constantine had their own definitions, but they focused very much on the plug-and-play stuff.

What the OO people did was take methodology and deeply integrate it into programming implementation. This resulted in the 3GL / 4GL schism between UML and all OOPLs, and as a result it seems that many people learn OO from the perspective of the capabilities of an OO language, as opposed to general systems thinking about plug-and-play design.

As an aside, there's this essay by Grady Booch where he disects three major approaches to gathering requirements for the design of an OO system. It's a pretty neat essay, where he argues certain people are predisposed to want to explain and think about designs in certain ways. I'm trying to dig it up, because I thought about it when the end of Henning's essay finished with cultural speculation... the essay I have in mind is published in Best of Booch.

The basic problem is that there are very few 'real' software engineers around; by this I don't mean self-taught software developers who have figured out a lot of this for themselves, but rather people really trained in 'software engineering'. Especially paradigm-independent software engineers - those are really rare.

What software engineering programs do you think turn out 'real' software engineers? The only Masters program I reviewed that impressed me was McGill (Montreal, Canada). I am effectively self-trained. My undergrad CS diploma is pretty watered-down. I taught myself via reading ACM Portal papers... I started during an boring Operating Systems class with a professor that sounded like she didn't know anything about OSes. But she said Dijkstra was a badass, so I started by reading his papers. From there I basically just bought books to teach myself, since our CS library was pretty pathetic. It's pretty expensive to teach oneself via books, though.

Approaches To Process
Alas, I cannot make the same positive statement with regard to process: there exists considerable diversity in thought over how the object-oriented project should conduct its business. To that end, I roughly divide the object-oriented world into five somewhat overlapping schools of philosophy regarding process:

• The anarchists, who largely ignore process and prefer to be left alone with their computers.
• The behaviorists, whose focus is upon identifying and then evolving roles and responsibilities.
• The storyboarders, who see the world in terms of scenarios.
• The information modelers, who focus on data and treat behavior as secondary.
• The architects, who focus upon frameworks (both building and using them).

I've found this to rather accurately describe the 'societies' within large software organizations.

Since API design is part of software design, it is only natural that coming up with use cases will lead to better APIs - for exactly the reasons you outline: they will be designed to follow good usage patterns rather than bottom-up.

You seem to be assuming that the only culprit for poor APIs is lack of experience, knowledge and skill on the part of the programmers, but I would argue that those factors account for at most half the problem.

Many of the problems of software design in general and API development in particular are sociological.

Some examples:

• APIs are often released "before their time" for business or organizational reasons
• once an API is released, changing it becomes very difficult, because some important groups are now "depending" on the broken features you want to fix
• the people whose use cases were the basis of the design are not the people who are now using the end result
• there were two or more camps of users with conflicting goals contributing to the use cases
• APIs are often an afterthought, viewed as an "advanced" feature to be used by masochists, er, I mean programmers. Most development effort goes into the "user-friendly" components, i.e. those intended for less technical people.

This is just a sampling; I could go on and on. Requirements analysis is as much political art as technical discipline.

It is interesting to contrast this situation with PL design, where there is often an identifiable designer or group of designers with explicitly technical goals that can be assessed directly on technical merits: no wonder everyone wants to develop their own language. ;-)

I over-simplified things - thank you for your thoughtful post. I agree that all of the factors you mention are extremely important forces in the development of APIs. I have witnessed every single one of them 'at work' when I was doing professional software development.

To bring a PL angle: think how much worse API design is in 'dynamic languages' with strong introspection facilities. These languages (like Ruby, Python, but also Maple) have their data-structures completely exposed, as there are no proper information hiding mechanisms that are guaranteed to be opaque. [I guess I should throw in Java bytecode and .NET's CIL in here too]. Mistakes in API design and data-structure design live on forever, because of powerful pressures to maintain 100% backward compatibility.

My experience was that, if the developers had ensured that they had sufficient use cases (scenarios, user stories, etc) in hand, they would have designed a better API. They were certainly competent enough to do so, they just didn't have the right 'raw data' to do a good job. The APIs were frequently designed with some abstract elegance goal in mind, but never tested for proper composability and/or usefulness for a variety of tasks. But they sure were elegant.

...details

To bring a PL angle: think how much worse API design is in 'dynamic languages' with strong introspection facilities. These languages (like Ruby, Python, but also Maple) have their data-structures completely exposed, as there are no proper information hiding mechanisms that are guaranteed to be opaque.

Another way to state this is that in a language like ActionScript you are not forced to expose an interface that programmer's can see on a diagram, such as a UML Class Diagram or UML Collaboration Diagram. ActionScript literally allows you to use the object dictionary model to circumvent 'private' keyword; even though ActionScript supports the private keyword there is no real notion of a private variable. This has the effect of allowing the API designer to have out-of-band class collaboration information that client API users do not.

Maple is a joke, so I won't even comment on it in detail. They completely broke their graphical input system at one major release of the language!

To bring a PL angle: think how much worse API design is in 'dynamic languages' with strong introspection facilities. These languages (like Ruby, Python, but also Maple) have their data-structures completely exposed, as there are no proper information hiding mechanisms that are guaranteed to be opaque.

Not sure if this is of any interest but my opinion about this is almost the exact opposite. Can't speak of Maple but data-structures and APIs in Python and Ruby exhibit growth pattern, which is compensated in languages like Java using XML extensively. By no means does it imply that this leads to elegant designs by virtue of the languages, but hardly ones that are in the programmers way and lead to misery because it is too rigid and changes can't be applied without breaking lots of dependencies. Moreover types aren't all that important. If some function expects a file one can pass something which is just file-like in the relevant aspects.

So maybe API design doesn't matter that much after all when we can live also with weaker designs.

If some function expects a file one can pass something which is just file-like in the relevant aspects.

The problem is that you often have no way to tell from the outside what the "relevant aspects" might be, especially when the documentation is vague or non-existent.

One of my peeves when I was doing the Python Challenge was that different libraries were somewhat inconsistent about, for example, their containers, so that a traversal that worked for one container would fail for another.

Generally the docs weren't specific enough to avoid this problem, and there was no help from the compiler, so basically you had to try something, find that it blew up, and then trial and error to work around the problem.

While many people are perfectly happy with this style of development, I can't see that anyone would be able to convincingly argue that this is "good API design".

So maybe API design doesn't matter that much after all when we can live also with weaker designs.

I've never been happy with weak designs.

I've said already, people only care about bad APIs when it pisses them off. I should add that they also need to feel pain. It has been my experience that Java GUI programmers do not know what pain is, even after being kicked in the stomach by a Chuck Norris roundhouse kick. Swing simply attracts programmers who naturally secrete morphine instead of gastrointestinal muscosal; they have stomach ulcers from coding with such bad APIs, but the pain killers make them totally oblivious to this awful health problem.

I just happen to be the sort of developer who is incensed by bad design, in every way imaginable. I don't even like living in an ugly place. So just about every API pisses me off and every API is an opportunity to make developer lives everywhere easier!

When I talk about weak design I mean something like Rubys non-orthogonal libraries. Python was already mentioned by Marc. My first programming language was Mathematicas scripting language. API design? Flat kernel library with powerful functions. A more extreme example of the same non-design approach to APIs is PHP. I never used PHP and heard lots of horror stories about it. However, apparently mediocre programmers were able to create most of the web applications which matter for me using PHP. How could that happen?

Java Swing ranges at the opposite and is strong design.

Java Swing ranges at the opposite and is strong design.

I have to disagree. In fact, the Program Managers on the Swing project, such as Dean Iverson, tried to improve Swing to correct many of its accidental complexity flaws. This is part of what morphed into the "Swing Application Framework".

I could go into more detail as to how Swing is an abomination, if you'd like. Dean has only limited examples on his blog, such as how tedious it is to set-up a Hello World Swing app. Swing does not follow a basic OO practice: Hide the sequence. Instead, everyone has to write code to manually hide a sequence that Swing's programmers didn't bother to hide for you. Finally, if it is any indication, all of the public-facing talent on Swing's team is now gone to Adobe and Google. The guy they brought in for JavaFX came from a separate division of the company, because they had to start over. Sun's Desktop division has been consistently underfunded for years. They couldn't even get proper antialiased text into Java until JDK 6.

Further, Swing does not have any way to declaratively define layout constraints. This led to Karsten Lentzsch creating a bunch of layouts for developers so that Swing UI's weren't unbearably ugly. However, if you wanted to refactor from one layout to another, you had to do a lot of moving of code mountains. Even refactoring IDEs aren't prepared to reshuffle thousands of layout constraints.

Also, binding in Swing is awful. It's not even binding. Model definition is ugly and it forces you to write wrapper types everywhere. Likewise, the interface for event listening typically requires a similar listener adapter wrapped around it so that you don't write excessive boilerplate.

Finally, extending Java Text editing is ridiculous. Is it any wonder Eclipse is implemented in SWT? At least it is marginally better.

John, let's continue this discussion another time. My thoughts are not cleaned up and seeking to troll, just because I find the extreme reduction of complexity in qualifications such as "good design" and "bad design" hilarious, isn't adequate either.

Just to close from my side with a little background. Right now I'm writing some sort of IDE for DLR languages in Silverlight using C#. It is obvious to ask how to improve accessibility of the Silverlight API for Ruby/Python users who might want to avoid using Xaml and program small dialogs and applications directly in interactive style - not much unlike they'd do in an environment such as Mathematica. Finding a model for "programming in the small" is something I'm interested in here rather than SE / "programming in the large", which is approached more adequately using different tools.

Thanks for sharing the pointers to discussions that I was involved at.

Recently I managed to make one of my presentations video taped. In case you have one and half hour of spare time, feel free to give my Paradoxes of API Design video a chance.

Comments and publicity is welcomed!

It's interesting that Microsoft makes such API usability studies and ends up providing schizoid APIs as like in WPF. In that particular case Xaml users are allowed to set a color value to "Red" or "#FF000000" whereas the code behind programmer must write

txtbox.Background = new SolidColorBrush(Colors.Red);

or

txtbox.Background = new SolidColorBrush(Color.FromArgb(0xFF, 0x00, 0x00, 0x00));

without any chance that WPF does the dirty work of coercing the plain value on assignment. Moving from Xaml to C# or other .NET languages breaks almost every single expectation about setting property values. Since WPF defines a GridLength object for grid lengths, a Thickness object for border thicknesses and so one must wonder that the height of a control is determined by a double and not by a Height object.

The code behind design is surely not a "mistake" or lacks abstraction and extensibility. On the contrary. This is good style in all manners of configuration over convention and with Visual Studios super-smart type aware intellisense which often hints to the right constructor while typing even clean design becomes bearable.

What the C# language lacks is support for compile-time literals via a CompilerService. C# 3.0 was supposed to have XML literal support. However, what you really want is XAML literal support. Why? Because hierarchical syntax like XML is the most natural way to define a GoF Builder Pattern. Somebody on the Apache Jetty project did analysis of this and it turns out XML is just a very succinct DSL for this sort of stuff.

Anyway, this usability problem you point out is only somewhat true. In my books, the Original Sins in WPF are:

1) No "Syntax for hashes" for XAML... why create another .NET language without "syntax for hashes"... we already have VB.NET and C#
2) Not creating a lot of the dependency analysis subsystem as a CLR intrinsic
3) and then not providing C# and VB.NET syntax for this intrinsic - in short, both C# and VB.NET could arguably support some data flow syntax.

On the other hand, when WPF was designed, we knew a lot less about PLT and even software engineering... for example, FRP was only a few years old.

If you read what I've just said carefully, you'll note that Scala supports everything I just said. You should ponder how good Sean McDirmid's Bling library would be if he had Scala.NET at his disposal instead of C#. -- Your examples would be trivial in a Scala version of Bling.

What did we DEFINITELY know about software engineering that WPF blatantly ignored? Complexity cycles. A basic structured programming and OO metric. WPF violates dependency structure everywhere. Ask the Ximian guys who built Moonlight (after first trying to implement WPF) how much of a pain this is. You simply CANNOT rebuild WPF. It is too stupid and ugly for anyone to re-implement. Silverlight, on the other hand, is much nicer. However, people at Microsoft, in the name of "usability", are doing their best to turn Silverlight back into WPF again. Why, I don't know. It's sad. I predicted everything that would happen as a fallout of .NET 3.0. I knew WF would be re-written from the ground up and that XAML would hack in support for WCF Dependency subsystem to talk to the WPF Dependency subsystem. It's just too complex and stupid. Anyway, end of rant.

To be fair, the WPF designers probably had many concerns to balance and they had a mandate to build a designer-friendly UI toolkit with much more aggressive goals than past toolkits. We will see how this pans out, though WPF does seem to be pretty popular with designers (because of Blend, which is enabled by XAML, which drives a lot of the complexity in WPF's library design).

If you read what I've just said carefully, you'll note that Scala supports everything I just said. You should ponder how good Sean McDirmid's Bling library would be if he had Scala.NET at his disposal instead of C#. -- Your examples would be trivial in a Scala version of Bling.

If there was a decent version of Scala.NET I would be back in the Scala community tomorrow, if they would take me back that is! That being said C# isn't that bad, I can see it getting just a few features (case classes, traits, a bit more aggressive type inference, more expressive implicit conversions) and my desire to use Scala would melt away.

You simply CANNOT rebuild WPF. It is too stupid and ugly for anyone to re-implement. Silverlight, on the other hand, is much nicer.

I prefer WPF to Silverlight because Silverlight is much more XAML biased than WPF--they are really pushing the designer story more I guess. But...I think I'm more at harmony with WPF because I only use 20% of it because of Bling.

It's interesting that Microsoft makes such API usability studies and ends up providing schizoid APIs as like in WPF. In that particular case Xaml users are allowed to set a color value to "Red" or "#FF000000" whereas the code behind programmer must write...

Users of Bling (a C#-based wrapper around Bling) can write just:

txtbox.Background = Colors.Red;

Because there is an implicit conversion from Color to BrushBl. The only reason the code behind story sucks in WPF is because it just wasn't given much attention! (Bling fixes that).

Since WPF defines a GridLength object for grid lengths, a Thickness object for border thicknesses and so one must wonder that the height of a control is determined by a double and not by a Height object.

Again, solved by an implicit conversion from double or DoubleBl to ThicknessBl (where d is defined to convert to (d,d,d,d)).

The code behind design is surely not a "mistake" or lacks abstraction and extensibility. On the contrary. This is good style in all manners of configuration over convention and with Visual Studios super-smart type aware intellisense which often hints to the right constructor while typing even clean design becomes bearable.

I would disagree with this. Code behind in WPF is just not very elegant, they should have pushed more functionality into declarative land or added more meta-programming style sugar to code behind so that it was easier to deal with.

I know about Bling. Right now I'm using Silverlight which shares the particular API I was talking about with WPF. So there was no reason to make a distinction in my post.

Anyway, we still talk about workarounds and wrappers. I can create a wrapper for each single DLR language which enables overwriting attribute settings on classes generically and this wouldn't be hard to program but it is still work used to correct an API which is designed correctly but experienced as cumbersome.

I know MS had labor division between designers and programmers in mind just like big software engineering and provides excellent tools in support but since I'm in DLR territory and work out some sort of web based "Python/Ruby programming service" for casual users who might be neither graphic designers nor professional .NET programmers, a steep learning curve for a complex toolkit doesn't feel appropriate.

I know MS had labor division between designers and programmers in mind just like big software engineering and provides excellent tools in support but since I'm in DLR territory and work out some sort of web based "Python/Ruby programming service" for casual users who might be neither graphic designers nor professional .NET programmers, a steep learning curve for a complex toolkit doesn't feel appropriate.

The problem, more aptly coined by Jonathan Edwards as a "Twisty maze of classes," is maybe that frameworks are too big for people to take in very easily, that you would feel overwhelmed the first time you encounter a library like say WPF with lots of chunky abstractions. The solution, I think, is to go with more simple and general abstractions; e.g., rather than have panels to do layout, people could just specify their layouts directly using canvas + data binding. Same with storyboards and styling and...whatever, that is what Bling is about...providing a more simple way of programming WPF. Then with fewer abstractions to learn, the library should be quicker to learn and more appropriate for casual usage.

The solution, I think, is to go with more simple and general abstractions; e.g., rather than have panels to do layout, people could just specify their layouts directly using canvas + data binding.

Yes, that's also why Flash, even without data binding or Bling style constraints, is much easier to use than Silverlight. It is an animation toolkit with GUI elements being an afterthought. At the core it is about a bunch of moving and shape changing rectangles and geometrical and topological relationships between them which can be always queried. One knows which rectangles are under a certain point, there are enclosing rectangles and so on. No matter how everything is layered one has access to the fundamentals and a simple cognitive model. Layouts are then used to express or constrain relationships between those rectangles. Advancing this in the direction of physics engines and eventually non-linear relationships seems plausible to me.

I'm not too concerned when data binding is completely expressed in callback style instead of declaratively which is somewhat more powerful. Data binding seems to be a natural extension of the concept of user events. So one binds to changes of arbitrary properties of some type ( "dependency properties" ) instead of just mouse clicks or key strokes.

I'll see what's doable for DLR languages in case of Silverlight in the next couple of weeks.

This reminds me of how we did animation in Java2D via immediate mode rendering: just put the objects in a different place on each time step.

Its amazing that doing things the "right way" via dedicated abstractions (storyboards) is more difficult to learn than doing something the direct way without dedicated abstraction. Surely storyboards provides more headway to do more complicated stuff (synchronize scenes with actions, framerate independence), but they actually make doing simple animations harder.

API designers should keep that in mind.

I'm not too concerned when data binding is completely expressed in callback style instead of declaratively which is somewhat more powerful. Data binding seems to be a natural extension of the concept of user events. So one binds to changes of arbitrary properties of some type ( "dependency properties" ) instead of just mouse clicks or key strokes.

I don't see databinding like this. An event stream is just one stream of events, a dependency property can be said to have a value + a stream of changed events. But if the property was just that, there would be no data binding involved. Instead, data binding is a continuous relationship, even if, as an encapsulated detail, the data binding is implemented by subscribing to the discrete stream of changed events. I think treating databinding as just another form of event stream is likely to remove its elegance, making it as ugly/complicated as raw event handling.

This reminds me of how we did animation in Java2D via immediate mode rendering: just put the objects in a different place on each time step.
Its amazing that doing things the "right way" via dedicated abstractions (storyboards) is more difficult to learn than doing something the direct way without dedicated abstraction. Surely storyboards provides more headway to do more complicated stuff (synchronize scenes with actions, framerate independence), but they actually make doing simple animations harder.
API designers should keep that in mind.

Processing.org has the good old immediate mode animation approach. You set your animation parameters in a few global variables. You update them at each time step and draw. This may not be the best way to do complex animations but remember that humans are smart. We can do complicated animations easily even in this simplistic rendering model. It takes less time to do the animation you want this way than to learn a complicated API to do the animation the "right" way. For example I was able to write simple electromechanics simulations that display the field/potential very easily. Would a more sophisticated animation model have helped? Perhaps, but the time to learn the API would be far far longer than the time to code it up in Processing.

<rant>

I've been coding a structured editor with WPF and as much as I want to like WPF it is a huge pain. The effort required is comparable to painting the pixels myself like in Processing. This is partly due to WPF being far too complicated which means that I'm constantly looking at (poor quality and incomplet) documentation. Getting all the nodes in the tree to display properly is difficult. Part of the problem is C#. For example getting the right display settings to every node in the AST requires either polluting the code by passing objects around the entire program OR hard coding them which means that tuning the display settings is very time consuming because every change requires a long recompilation (and one setting is duplicated in multiple places).

Is the precooked widgets-with-tons-of-properties based approach that GUI libraries take really the right one for programs that aren't run-of-the-mill business apps?

I found Jonathan's series of blog posts on WPF, Swing, and Flex to be some of the best cross-comparison blog posts by anyone on the Internet, and I've bookmarked them. I have also used WPF, Swing, SWT, and Curl, so I can probabl shed some additional light.

The problem, more aptly coined by Jonathan Edwards as a "Twisty maze of classes," is maybe that frameworks are too big for people to take in very easily, that you would feel overwhelmed the first time you encounter a library like say WPF with lots of chunky abstractions.

Sorry, but this was not Jonathan's point. In his blog post, You are in a maze of twisty classes, all alike, he criticizes the following flaws of WPF:

• "[...] incredibly over-generalized and over-engineered [...]"
• steep learning curve

There is nothing unique to frameworks here.

In his next blog post, Staying the course, Jonathan then goes on to say:

WPF reminds me of the early “component frameworks” in the 90’s like OLE, OpenDoc, and COM. These frameworks popularized the ideas of properties, events, reflection, and dynamic loading. Since the mainstream languages of the day (C and C++) could not handle these concepts directly, they had to be emulated in libraries with massively complex API’s. Eventually languages evolved to incorporate these features directly, making component programming much easier, as with JavaBeans and .NET. WPF is the same story all over again. It simulates a whole new layer of semantics involving declarative bindings between dynamically attached properties, a “style sheet” semantics for property overriding, “routed events”, a sub-language of time-varying functions, and a limited code-data isomorphism via XAML. The resulting complexity is really a plea for new programming language semantics.

I think we're also starting to see PLT come up with some solutions here, too. For example, Alexey Radul's Ph.D. thesis under Gerald Jay Sussman, Propagation Networks: A Flexible and Expressive Substrate for Computation is one example of looking at foundations for how to implement the automatic change propagation networks in WPF applications.

Finally, in his Beachhead Dispatch blog post, Jonathan provides further commentary on WPF, where he does a good job assessing WPF 3.0's major strengths and weaknesses. Weaknesses Jonathan observes center around XAML:

The biggest problems with WPF are due to its ambitious goals to make UI design more declarative and less code-centric. I applaud the intentions, but the results are decidedly mixed. One example of these problems is the styling and themeing system. UI controls have no hard-wired look and feel – it is completely specified in an XML dialect called XAML. This is extremely powerful and customizable. And unusable, because it is almost completely undocumented. For example, you can’t change the color of the border around a text box. Sure you can change the color – until the mouse travels over it, when it reverts to the themed color.

What Jonathan is describing is a serious architectural limitation of WPF 3.0. WPF 3.5 SP1 shipped with support for the VisualStateManager that solves this problem (more or less). Unfortunately, this attached property for WPF controls was posted on Expression Blend architect John Gossman's web page, and not really widely publicized by Microsoft. This move reminds me of the SwingWorker background thread pattern Sun had in their documentation as a necessary feature to have realistic performance, but not shipped as part of the SDK - and so a lot of lazy developers were never acquainted with it.

Note that in Jonathan's blog post, he mentions how the Microsoft Expression Blend team, led by John Gossman, has more or less solved all of his problems and that a major sticking point is that none of these solutions propagated back into mainstream WPF documentation.

What a fiasco. A UI framework that ships with a set of controls whose behavior is undocumented and not incrementally customizable. And this was done all in the name of easier customization. Microsoft’s Blend design app solves this problem by providing a complete new set of “simple controls”, defined by 1100 lines of XAML which replace all the undocumented XAML of the standard controls. The Blend docs explain how all these styles work, and the set of global resources they depend upon, so you can customize them as needed. I have adopted Blend’s styles for my UI, so I am satisfied that I can do what I need. But the WPF team should be embarrassed by the necessity of this giant workaround.

John is incidentally the person Microsoft put in charge of "subsetting" the WPF API so that Silverlight would not repeat the same mistakes as WPF. He has more or less been doing a phenomenal job, and the velocity of Silverlight releases -- Version 4.0 is coming out now -- is simply unreal. In fact, I think it is too fast, and you see some of WPF's bad design slowly creeping into Silverlight starting at the edges (NotificationWindow and NavigationWindow are both breathlessly bad designs from WPF that are more or less being copied in Silverlight).

However, despite these "fixes", WPF still suffers from one other major problem, which I am working on a blog post to elegantly explain. In short, they strongly violate proper structured programming practices by allowing dynamic injections to create a context between the caller and callee (Routed UI Commands are a good example, as they route business logic through the user interface; this is a strong violation of Clemens Szyperski's domaincasting principle for component-oriented systems, as you are sending a message through the wrong layer of the system). In my experience with visual programming, this is bad for rapid application development frameworks like Blend and lively kernel frameworks like Smalltalk IDEs. There is no way for the design tools to do static analysis and help the visual designer and programmer complement each other in their roles developing the application. Where I work, we do have a designer/developer user story, even though this role is filled by the same person 80% of the time. The reason for this schizophrenic separation is good design.

Bottom line: Frameworks are not too big for people to take in easily. Crazy dependencies are too big for people to take in easily. The first thing I did as an architect of a Silverlight/WPF cross-compiled application was not rely on WPF's internal dependency system, and instead replace as much of it as possible. The skeleton structure of our app thus behaves in much the way you explain Bling to be superior to what I call "The WPF Architect's Kitchen SInk".

Sorry, but this was not Jonathan's point. In his blog post, You are in a maze of twisty classes, all alike, he criticizes the following flaws of WPF:...

Sorry, maybe we are disagreeing on what "big" means. To me, big just means mentally complex, I didn't mean it in a physical size context.

There is nothing unique to frameworks here.

I would disagree. WPF is fairly representative of a contemporary framework. My next example would be Eclipse, which generally follows the same pattern (steep learning curve, over generalized and over engineered, if a bit better documented and engineered). I'm making some observations about why that is so, Jonathan has his own explanation, which I pretty much agree with, and I don't think my own spin on this problem is incompatible with what was said.

Bottom line: Frameworks are not too big for people to take in easily. Crazy dependencies are too big for people to take in easily. The first thing I did as an architect of a Silverlight/WPF cross-compiled application was not rely on WPF's internal dependency system, and instead replace as much of it as possible. The skeleton structure of our app thus behaves in much the way you explain Bling to be superior to what I call "The WPF Architect's Kitchen SInk".

All frameworks have crazy dependencies, if you don't agree with me, name one that doesn't! Crazy dependencies result from chunky abstractions, and you can't avoid them unless you instead design a bunch of small generic orthogonal abstractions. But in the latter case, you have something more like a language than a framework.

Everyone (including competing product groups) seems to believe that their own chunky abstractions are better than someone else's. This is BS, you just wind up replacing something you aren't familiar with with something that you are familiar with because you wrote it yourself, which makes a different set of trade offs that caters specifically to your needs. There are of course better designs, but the intrinsic complexity remains no matter how good the design is. Bling tries to break this cycle by just reskinning some of WPF's more generic abstractions (dependency properties, events, data binding) to be more usable so that they can replace the more hairy ones.

Please don't get me wrong, I'm interested in your explanation as I'm working on my own. I think you have specific recommendations on how to improve UI framework design, but can you generalize these recommendations? Even if we fix WPF, what about the next big framework? What about the hulking monstrosity known as Eclipse (beyond SWT) or even visual studio 2010 (whose APIs look similar)? What is fundamental to "better" API design?

As for generalizing these recommendations, yes... it's possible. Unfortunately, WPF cannot be saved in terms of "removing" complexity, just as Swing cannot be saved in terms of "removing" complexity. The best you can do is "guide" developers away from complexity and just outright shield them.

I've actually got three blog posts I wrote 7 months ago stored in the kitty, and haven't published them mainly because they are currently littered with too many comments like the ones I made to Kay about Chuck Norris roundhouse kicks. I try really hard to balance "dumbing things down" through use of hyperbole and correct use of design principles. -- This series of blog posts is aimed at establishing several key points, including why Silverlight is superior to WPF, both from a MS perspective and a Novell perspective. It breaks down the decisions the Silverlight team has had to make, guesses why they made them, and shows you why a lot of the features in WPF could be implemented using more fundamental abstractions. The second part of the series is supposed to be dedicated to looking at the frameworks shipped with .NET 3.0 as a whole. In my books, the only one MS "got right" on the first try was CardSpace, but unfortunately no one is using it and instead using inferior specs like OpenID.

What about the hulking monstrosity known as Eclipse (beyond SWT) or even visual studio 2010 (whose APIs look similar)? What is fundamental to "better" API design?

This is tricky, and something myself and a few others occasionally discuss. "Better" for some of these people means "4th generation languages", or at least "3rd generation languages" that solve all the problems with existing "3rd generation languages". Gilad Bracha's Newspeak is a good example of solving 3GL coupling concerns, and from Gilad's perspective, it is easy to understand why Erik Meijer says Gilad Is Right. The problem that Model-Driven Engineering enthusiasts have been working on for about 30 years is building robust 4GLs (that perform well!) and thus try to skip past this problem, but such a skip maneauver is tricky to pull off.

.NET itself is actually transitioning away from being solely about type systems. With Code Contracts, PeX and the new security transparency system in .NET 4, there are way more ways to specify a system than ever before. Now, for a hot button: I am not too happy with the design of the DLR. I personally don't understand the interface design choices made here. Then again, I don't also 100% like the interface design choices of the Object class in the core type system (but the CLR guys are quite proud of the Object class... there is a page on MSDN bragging about how this class has never had to change). For example, I think any design of the DLR (and re-design of WPF) should begin by redesigning the System.Uri class (easily one of the worst designed classes in all of .NET, and its monolithic design contributes to why it has the largest class overhead in the installation of Silverlight!). I don't know why you would build a dynamic language runtime without making every object in the system addressable via a Uri. (I still think .NET is vastly superior to Java on many levels, despite my heavy criticisms. -- My criticisms are a result of not taking crap from my tools or taking crap from tool vendors.)

Finally, looking at the past as a way to predict the future, Microsoft's UI innovations are the direct result of acquiring RenderMorphics Reality Lab in February 1995 (owned by three British students), which ended up helping shape DirectX 6... eventually, MS released DirectX 8, which was the first version of DirectX competitive with OpenGL. Fun fact: Kate Seekings, the designer of the API that led to Direct3D, flunked her masters course in computer graphics due to not strictly obeying the design specs the professor laid out. Since you are working on the next-gen MS stuff, a fun book for you to read could be Renegades of the Empire: How Three Software Warriors Started a Revolution Behind the Walls of Fortress. This covers the history of DirectX upto 1999.

There's also this short article by James Waletzky of Microsoft: Perfect API Design

My main problem with advice like this is too much emphasis is placed on discoverability, rather than correct logical design. When you focus too much on discoverability, you forget about math and logic.

[Edit: James did not work on WPF, and cannot be blamed for the following API atrocity.]

WPF Example:

DependencyObject.SetValue and DependencyObject.GetValue are not symmetric and there is no reason why they should be asymmetric. I would argue that this asymmetry creates poor discoverability; if I set a value X, and then attempt to get that value back as X', then X should equal X'. However, this is NOT how WPF works! Why?

Two reasons.

1) DependencyProperty.UnsetValue is incorrectly designed. It returns a singleton that ultimately maps to a "new Object()" instruction. This is wrong - it's just as bad as writing C or C++ code that allows for a Dangling Pointer! The value returned by this field should map to an explicit singleton type: DependencyPropertyUnsetValue. To be clear, the API designer saw the problem with using "null" to represent UnsetValue, but rather than program to an interface, he/she programmed to an implementation: the universal type, in effect Any implementation imaginable. This makes it impossible to really program logic, such as triggers, to detect when a value equals UnsetValue, because GetValue can't return UnsetValue's type.

2) The non-exposed GetLocalValue method will return UnsetValue, but GetValue "pretends" objects always have a set value, even if it is the default. LocalValue behavior is only observable via LocalValueEnumerator, which is exposed via DependencyObject.GetLocalValueEnumerator

So is this API really discoverable? I'm told the reasons it was designed this way are forgotten, but that it was most likely in the name of "discoverability".