Is API evolution of exceptions special?

Casper Bang, an early reader of the Practical API Design book, asked following question after reading it:

I was curious as to know how come, in a book strictly about API design in Java, you do not mention exceptions (particular checked exceptions) and the role they play in documenting assertions vs. hampering versionability. Did you simply think this to be too controversial an issue I wonder?

My initial reaction was that there is not much special related to exceptions, but later I decided to provide more detailed answer. I'd be interested to here LtU members thoughts on this topic.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

The problem with checked exceptions

(For convenience I'll limit myself exclusively to talking about checked exceptions unless explicitly noted.)

Dislaimer: My practical experience is limited to Java 1.4.x (the horror!) -- so things may have changed.

The fundamental problem with checked exceptions is that the declared exceptions of a method are contravariant while inheritance is covariant. (Or the other way round, I always forget which is which ;)). By the way, this is also an issue for the design of closures for Java, as you can probably tell from the example below.

The issue arises frequently in API design, let's say you want a functional-ish API for collections:

    interface Filter {
              public boolean transform(Object x);
    }

    interface Collection {
        public void filter(Filter t);
    }

    class List implements Collection {
          public List filter(Filter t) {
             // Just imagine a trivial implementation here.
          }
    }

If you now try to write a filter

    class FilterByDatabaseLookup {
          public boolean transform(Object x_) {
                 Long x = (Long) x_;       // Generics (Java 1.5) at least fix this ;)

                 // Obtain DB connection
                 ...
                 // Look up x in table XS
                 ...
                 // If x exists in table XS, return true, otherwise return false
                 ...
          }
    }

The problem with this is that it's highly unlikely to compile, because almost all DB access methods throw JDBCException (or whatever). As far as I know the only workaround is this:

    class FilterByDatabaseLookup {
          public boolean transform(Object x_) {
                 Long x = (Long) x_;       // Generics (Java 1.5) at least fix this ;)

                 try {
                     // Obtain DB connection
                     ...
                     // Look up x in table XS
                     ...
                     // If x exists in table XS, return true, otherwise return false
                     ...
                 } catch (JDBCException e) {
                     // We can't really do anything at this point, so we need to punt.
                     throw new RuntimeException(e);
                 }
          }
    }

The workaround also means that you really can't know what's being thrown in any particular block because you're sometimes disguising JDBCExceptions as RuntimeExceptions. Also, if you really want to catch that JDBCException in some outer code, you're now forced to use the rather non-idiomatic

        try {
           // Do whatever with a List.filter using a FilterByDatabaseLookup
        } catch (RuntimeException e) {
          if (e.getCause() instanceof JDBCException) {
             // Do whatever.
          } else {
             throw e; // Re-throw
          }
        }

I know this little example may look trivial, but in a codebase of any respectable size this particular issue with exceptions is a huge pain in the rear. The more different APIs you have to combine to solve your problem the worse it gets.

In some of the projects I'm working on, there are huge swaths of code just to deal with exception "conversion" such as above. Code which is worse than useless since it provides no function, but can lead to errors if you're not careful.

The only real solution I've seen for this issue is to do away with checked exceptions completely. The only examples I've seen which attempt to justify the existence of checked exceptions are trivial ones.

(Above may contain serious errors of thinking since I'm sick and have a massive headache.)

If you like generics, use them

The following code typechecks without warnings in Java 5:

interface Filter<T, E extends Throwable> {
    boolean transform(T t) throws E;
}

interface Collection<T> {
    <E extends Throwable> Collection filter(Filter<T, E> f) throws E;
}

class List<T> implements Collection<T> {
    T head;

    public <E extends Throwable> List<T> filter(Filter<T, E> f) throws E {
        f.transform(head);        // use the filter somehow
        return null;              // return something
    }
}

class FilterString implements Filter<String, RuntimeException> {
    public boolean transform(String s) {
        return false;             // return something
    }
}

class FilterExceptionalString implements Filter<String, Exception> {
    public boolean transform(String s) throws Exception {
        throw new Exception();    // throw the checked exception
    }
}

public class Main {
    List<String> list = new List<String>();

    public static void main(String... args) throws Throwable {
        Main main = new Main();
        main.testFilterString();
        main.testFilterExceptionalString();
    }

    void testFilterString() {
        list.filter(new FilterString());
    }

    void testFilterExceptionalString() throws Exception {
        list.filter(new FilterExceptionalString());
    }
}

The only thing I don't like about this is forcing non exceptional uses to add RuntimeException, but that can be hidden by replacing Filter with:

interface Filter<T> extends ExceptionalFilter<T, RuntimeException> {}

interface ExceptionalFilter<T, E extends Throwable> {
    boolean transform(T t) throws E;
}

No unions

As I mention in this comment, without unions that's not so effective. There's no easy way to declare a filter that might throw any of several exceptions.

Interfaces?

Unless I'm mistaken, in Java, unions are called interfaces. Or, at least, the mechanism for implementing union is the interface. You don't get nice functional-style type-safe unions, of course, but you get a reasonable approximation. Of course, by comparison with Functional Programming, it's easier to add new elements to a union post-facto, which may or may not come into the way of safety.

Not exactly

With union types "FooEx" is a subtype of "FooEx or BarEx" and also of "FooEx or BazEx", all of which are subtypes of "FooEx or BarEx or BazEx". Java has a limited form of union subtyping in its rules for overriding methods with "throws" declarations as in "foo() throws FooEx, BarEx, BazEx" can be overriden by "foo() throws FooEx,BarEx". But there would have to be a more general mechanism for polymorphically declared checked exceptions to be as useful as they could be.

Union types are a bit different from Java style interfaces. Java interfaces force a subtyping relation to be declared at the time of defining a class; you can't take a collection of existing types and express a supertype for all of them.

And union types are a bit different from the ADTs found in ML/Haskell style functional languages. ADTs don't create subtypes, they create "tagged" constructors for a single type. Something like union types can be faked using Either and variants, but they create an ordering issue. "Either A B" is a distinct type from "Either B A," but with union typing "A or B" is the same type as "B or A". You've also got a subtyping issue in that you have to manually convert from "Either A B" to "OneOfThree A B C" - the language won't do it.

Another perspective

I believe, instead, that the point of checked exceptions is to urge the programmer to deal with problems in a controlled and task-appropriate manner, rather than sticking their heads in the sand.

Applying this to your example, I suggest that the key (no pun intended) is that your example FilterByDatabaseLookup#transform must be viewed as a (potentially) partial function; it is the programmer's responsibility to consider as part of the design what it means to be unable to answer true or false for a particular argument. More precisely, FilterByDatabaseLookup#transform actually wraps a partial function (the database lookup), and the determination of what to do if that lookup returns bottom (throws an exception) is a context-sensitive design issue.

To extend the example, if I'm filtering a list of membership IDs based on whether the membership is still current (not yet due for renewal), I have three cases:

  1. Assume true: If I'm filtering a list of members to decide which ones should get an invitation to an organizational event, perhaps I want to give the benefit of the doubt in case of inability to determine current status, and include that the member for invitation. Here I'm assuming that the consequences of an incorrect assumption (erroneous inclusion of a member) are small.
  2. Assume false: If I'm filtering the list to see which ones should be sent a renewal reminder, perhaps I want to avoid wasting postage (or raising concern) by not dunning members unless the d/b confirms that they are due for renewal. Here again, I assume other processes (e.g. repeat renewal notices, etc.) that make the consequences of an error acceptable.
  3. Assume bottom: If I'm filtering the list for some security-critical purpose where the entire result must be correct with high reliability (make up your own case, ranging from secret ballots to a query from Homeland Security ;-), I may want to treat a database exception on a single ID as invalidating the entire resulting list, by throwing a FilterPredicateFailedException to the layer of code that is expecting the filtered list. In this case, the consequences of a single-element error are unacceptable.

I believe that the only real solution for this issue is to consider what failure to be able to answer at the element level means about the usability of the entire resulting list, and design accordingly. Assume inclusion (risking a false positive), assume exclusion (risking a false negative), or fail the entire filtering operation -- each of these is reasonable for some scenarios.

One of my collegues is fond of saying that 80% of all production code should be about error handling. We would certainly be dissatisfied with a compiler that treated all errors by simply responding "You messed up somewhere" and crashing. We expect precise identification of the location and nature of the error, and really like it when the compiler is robust enough to make a reasonable adjustment and keep working. (How many of us have yelled at a compiler, "If you knew I needed a comma there, why didn't you put it in for me and keep working?")

Likewise, I believe that production-quality software should be concerned with precision and context-appropriate response to errors that occur in its own domain.

Footnote

I didn't emphasize this detail (and I apologize if I'm belaboring a point), but it's obviously impossible to simply allow the JDBC exception to propagate; it's nearly as obviously incorrect to turn it into an uninformative RuntimeException, once you think about the situation from the perspective of the caller of Collection#filter.

Filtering a collection inherently has nothing to do with databases, so a database-specific exception has no business at that level. On the other hand, filtering a collection is all about whether the predicate applied to an individual element succeeded. Telling the caller of Collection#filter that, "I failed to give you a resulting collection because the filter failed on an element" is quite meaningful at that level, especially because it was that caller that provided the filtering function to be used for the task at hand.

I don't believe it is fair to the user of the API to throw away information by turning that opportunity for precise troubleshooting into, "Something went wrong somewhere. Good luck figuring it out!"

Solutions to checked exceptions

The only real solution I've seen for this issue is to do away with checked exceptions completely.

Checked exceptions aren't inherently evil, just poorly implemented in Java. If another language ever takes up the idea then the designer would be well advised to look at the latest research in other effects systems. A few ideas

1) Make exception declarations polymorphic ("generic" in Java terms). That neatly solves the problem above. In my example I stick to the overly verbose way of creating higher order functions in Java, and use an in-place imperative version of filter instead of a functional one so the examples can be compared side by side. Also I use square brackets so I don't have to mess with HTML entities.

    interface Filter[T,E] {
              public boolean transform(T x) throws E;
    }

    interface Collection {
        public void filter[T,E](Filter[T,E] f) throws E;
    }

So now xs.filter(f) throws whatever exception f throws

    class FilterByDatabaseLookup extends Filter[Long, JDBCException] {
          public boolean transform(Long x) throws JDBCException {
             // do whatever, the database exception is declared
          }
    }

Of course, there needs to be a way to create a union type of exceptions for that to work.

2) Infer the throws clause, at least for implementations. I hate repeating myself. Make the tools do the leg work, not me. (While we're at it, infer everything else, and figure out a way to recognize the forAll on type parameters so I don't have to declare them - see ML and Haskell).

3) With exception inference, add a wontThrow clause. If I declare that a certain method won't throw IOException and the inferencer can't prove it won't then I have to handle it somehow.

As you rightly point out

(WARNING: Rambling ahead, I'm still sick. :()

As you rightly point out, this doesn't work if you now want to extend FilterByDatabaseLookup() and also need to throw, say, NumberFormatException. Mind you, the original example is also just one example off the top of my head, I'm sure there are better ones where the problems are harder to solve than by allowing unions of exceptions.

Short of an effect system or exception inference, which let's face it is not exactly likely to happen any time soon in Java's case, I don't see any obvious solutions.

As a practical matter (and in my opinion) the issue with doing away with checked exceptions isn't nearly as bad as many people think.

My opinion is that almost all exceptions in Java that are currently checked are in fact exceptions which are almost impossible to deal with anyway. My favorite example comes from EJB where almost every method can throw NamingException, CreateException, FinderException, etc. (all checked). These are basically almost always impossible to deal with in any sane way, and so are propagated to the main server thread loop, where they print a nice stack trace and wait for a human to analyze them to see if they're actual real problems or just intermittent failures. And this is the thing: Most Java application servers already have a catch-all, so it's not exactly likely that you'll crash your server from an uncaught (undeclared) exception. Meanwhile, we need lots of boilerplate code saying:

      try {
        ...
      } catch (NamingException e) {
        throw new NewExceptionOurFrameworkHasDeclared(blahblah, e);
      } catch (FinderException e) {
        throw new NewExceptionOurFrameworkHasDeclared(blahblah, e);
      }

(Can you tell I'm bitter? ;))

The only checked exception that I can think of off-hand which often can be dealt with would be NumberFormatException, but in that case maybe we'd be better off with a simple option type, a la:

type Maybe a = Nothing | Just a

and simple pattern matching.

That's essentially my gripe with checked exceptions: In all the cases I can think of where they're useful to force you to deal with something and where there are sane ways to recover, the better solution would have been a sum type + pattern matching on that. In all other cases, checked exceptions just add noise.

Since checked exceptions are overused to an extreme degree in various APIs, and since there are so few cases where you actually want to catch exceptions (from our own code, I'd guesstimate <5%), APIs with checked exceptions essentially force "the wrong default", i.e. 95% of catch statements are pure boilerplate. The amount of boilerplate can also make it harder to spot cases where some genuinely interesting exception handling is occurring in your code.

Rant over. :)

Encapsulation

While there is a good bit of generic code that would benefit from being able to treat exceptions generically, I don't think that's the solution to code like the snippet you posted above. If I understand your example correctly, I think you really do want some mechanism that converts EJB exceptions to your framework declared exception, as a matter of encapsulation. What's needed is a way to declare the forwarding policy in a single place.

Laziness

What's needed is a way to declare the forwarding policy in a single place.

It's easy to do with call-by-name and/or call-by-value parameters. I'll borrow Scala here, though Scala does not have checked exceptions.

Here "a" is a call-by-name value of type A. Call-by-name means the expression passed to handleException isn't evaluated before the call, it's wrapped up in a thunk to be evaluated inside handleException.

@throws MyCoolException
def handleException[A](a: => A) : A = {
   try {
     a
   } catch {
     case e : MyCoolException => throw e
     // other mappings can be here, or just one generic one
     case e => throw new MyCoolException(e)     
   }
}

@throws MyCoolException
def doSomething(x:Int) : Foo = {
  handleException {
    // insert exception throwing code here - no need to handle the exception, it's already taken care of
  } 
}

The policy could be more interesting - it could involve logging or even retrying, for instance.

Something like that is doable in Java now with an interface and anonymous classes, it's just pretty awkward. When/if Java gets closures, laziness can also be faked using functions from some dummy value (unit?) to the result.

Type of handleException is wrong

If you forward several types of exceptions on to corresponding types of exceptions in your framework, it looks like handleException will be typed to throw all of your framework exceptions, regardless of what the parameter threw.

type level mapping

The lazy solution with only a single exception type does match Bárður Árantsson's original problem description. But I admit it doesn't deal with more specific type mapping, i.e. you want to map from function that may throw A1 to a function that may throw B1, a function that may throw A2 to one that may throw B2, one that may throw A3 or A4 to one that might throw B3 or B4, etc. For that, laziness and/or higher order functions aren't enough alone.

If you add polymorphic exception types, union types, and type aliases, you should be able to get pretty close.

Ah, well

I was about to mention that. You just saved me from the embarrassment of self-reference.

Beauty or cluelessness

Thank you all for your comments. I am slightly surprised all reactions touch only checked vs. unchecked exception topic. I'd expect there is more to say about exceptions, but still your takes on checked exceptions are really interesting, so I guess this is OK.

I really like the explanation of partial function. I have to agree that in case an API deals with partial functions and the partial function result can influence the return values, it seems more appropriate to use checked exceptions. However this leads to my favorite topic of "correctness vs. empiristic programming" - it is correct, right and elegant to define partial functions with checked exceptions, however if we define them with unchecked ones, they will be easier to use, at least in most common cases. The use will not be fully correct, but in the world where the most important thing is time-to-market, this does not that important to majority of users.

I seem to like the idea of generification of exceptions too. Is not this another example of co-variation and contra-variation? In case one is offering an API for providers, e.g. interfaces that others can implement it is good idea to give them wider choice of which exception (if any) they want to throw. In case one is offering a client API (which others can call), it is desirable to throw as little exceptions as possible. However, if one provides both types of API (like in the Filter example above), the best variety of API user choices is obtained, if the exception remains parametrizable until used, e.g.: E extends Throwable.

I also understand the pain when converting one checked exception to another and the desire to have Maybe a type. On the other hand it seems to me that this would trivially be satisfied if we all decided that the right checked exception is IOException and subclasses. Methods returning Maybe a values could then throw IOException while those returning a would throw nothing. This seems to have at least the same expressiveness as the Maybe a case, does it not?

Thanks for sharing your thoughts with me. I'll convert them to a topic on apidesign.org wiki, when I find some time.

however if we define them

however if we define them with unchecked ones, they will be easier to use, at least in most common cases. The use will not be fully correct, but in the world where the most important thing is time-to-market, this does not that important to majority of users.

I think this is only true only when checked exceptions aren't allowed to propagate, but must be handled and rethrown in the same scope in which the function is called. So how an exception propagates is another design choice for checked exceptions.