## Grady Booch: Software Engineering Grand Challenges

A couple of weeks ago, I posed the following in my blog: what are the grand challenges of software and software engineering? What do we know we don't know?

I had about a dozen people rise to the question and a lively discussion among many of them unfolded via email. Following are some highlights of that correspondence.

Even though most of the responses weren't about coding and programming language issues, I think this discussion is worth a look.

It might be intereting to discuss if and to what extent programming languages can help improve the state of the art in SE.

I'll go first and argue that a large part of the answer to How to keep significantly raising the level of abstraction in face of ever growing software complexity? lies in better programming languages. Especially as regards module systems, expressive type systems and better concurrency and distributed computing idioms.

## Comment viewing options

### The complexity is Irreducible.

Sigh! Fred Brooks spotted the basic problem way back in 1975. The complexity is irreducible. Raising the level of abstraction does not make the complexity go away. It merely creates increasing fuzziness and uncertainty, resulting in lowered ability to reason accurately about what we are doing.

I will agree with you we need better programming languages, but not in the direction we have been going for the last 30 years. We need more analysable languages.

We are building systems that are too complex for us to understand. We need better tools and techniques to help us understand and manipulate the systems we build.

C++ is appalling from this point of view. It's like perl of which Larry Wall said, "The only thing that can parse Perl is perl." The only thing that can parse C++ is a full blown C++ compiler. And having done so, it cannot analyse it and certainly not manipulate it.

We have been on a 40 year long digression away from analysable languages such as Lisp / Scheme.

UML will not save us either. It merely increases the fuzz without increasing the analysability.

We need tools that will show us _exactly_ what we have, and allow us to manipulate large systems, at the following levels.

• Compilation Structure.
• Control Flow structure
• Data Flow structure.
• Component (100-1000 .c file level) Structure.
• Persistent Storage Structure.

### Also...

• dependency structure, with provisions for explicitly creating rules for prohibitted and allowed dependencies
• ownership, aggregation, and association relationships between components
• resource usage patterns and limits
• concurrency and timing constraints
• binding of components and communcations paths to processors and transport layers
• Once a language gets clean enough that most of the complexity of a given implementation is irreducible instead of accidental, the next step is to have the language support explication of design intent and analysis of design and implementation quality.

Actually, there is now a reasonable argument to be made that both Smalltalk and Java have reached the level of analyzability you described. Commercial refactoring IDEs, interactive static analyzers and code rewrite systems are available for both of them. Those languages seem trickier than the various Lisps due to the base complexity of the syntax, but are actually easier in that their syntax is not extensible. Preprocessors, code generators, dynamic scoping and macro systems may make code easier to write and more terse, but complicate the work of code analyzers immensely. I get the feeling that time will eventually show them to be a bad path.

### macroless lisp

macro systems may make code easier to write and more terse, but complicate the work of code analyzers immensely

Are there major wins that could be gained in this area from a Lisp without macros? If everything was a primitive or a function, what interesting thing would be possible immediately? (I'm not familiar with what Java and Smalltalk analyzers do, so a point or two from their book might be the major win that applied?)

### Codewalkers

randallsquared: Are there major wins that could be gained in this area from a Lisp without macros? If everything was a primitive or a function, what interesting thing would be possible immediately? (I'm not familiar with what Java and Smalltalk analyzers do, so a point or two from their book might be the major win that applied?)

Absolutely. If you want to see a real horror story, find a copy of the Xerox Portable CommonLoops source code and find their codewalker, which is 90% hair around portability issues in macroexpansion. Take away macros, take away macroexpansion, take away 90% of PCL's codewalker, which then reduces to just... knowing what the standard Common Lisp special forms are, and using normal evaluation rules for the rest. Simple.

It's much worse in languages where "macros" are done with a preprocessor, especially if it's just plain ol' textual substitution. Parsing standard C isn't hard at all. Parsing C as it's found in the wild, including preprocessor use, is nearly impossible, and in fact a good many C analysis tools require that you run any "real-world code" through the preprocessor before running it through the analyzer in question.

### Paul Snively: It's much worse

Paul Snively: It's much worse in languages where "macros" are done with a preprocessor[...]

Funny, I was just about to say, "Wouldn't it solve this to use a preprocessor?" If you had macros that weren't really part of the language, but were expanded by a preprocessor that used the language, then you could have tools that did analysis do said analysis on the expanded functions-and-primitives-only code. Further, if the external macro system expanded macros in such a way as to leave a signature, you could have editors and other tools for human viewing of source to code recompact the macros into their written forms, so it seems like you could get all the benefits of macros to people, and not have them in the language itself, so that codewalkers and other tools wouldn't have to have any handling for them.

Maybe there's some reason you can't do this?

### Input's not so bad, it's output...

One problem is that the whole point of analyzable languages is to present results back to the developer, in a way that they can understand them. The system you describe can do analyses, but the analyses can't report back the actual locations of any of the issues found. That information was destroyed within the preprocessor and recompact phases. Even if you can get back location information, the resulting analyses will be more difficult to interpret, as they will require implicit knowlege of any macros used. "The analyzer says I've got a possible memory leak here, but it's buried somewhere in this 500 line macro which is expanded by this 200 line macro." As analysis interpretation gets more difficult, it is is less likely that analyses will actually be used.

None of this is technically impossible, but from a practical point of view it's extremely challenging to do so well enough that busy programmers will actually use your code-walkers.

### The point

I see. I'd imagined that the point of analysis (as in the PCL example) was to do some sort of transformation on the code.

### The point...

is to show me what I've written, or what my coworkers have written, and how it is flawed or can be improved. Show me how it's mis-structured. Show me how it's abstractions leak. Show me how I'm misusing my dependency APIs. Show me the deadlocks, races, and stale data issues. Show me the invariants. Show me the parts of my code that aren't being tested, even if they are being executed by the tests, and show me what tests I should write. Show me what needs to be documented, and why. Show me the requirements I've missed, and the requirements I've hit. Ideally, I should be shown all of this as early as possible, and in as insightful and compelling a way as possible, so that I can claim to actually understand the engineering tradeoffs I'm paid to make.

Market my code to me, in the best possible sense of the term.

Preprocessors and pluggable compilation paths are ten-a-penny, and execution is just about the least interesting story you can tell about code. "Analyzable" means much more than that.

### Macroless Lisp

A truly macroless Lisp isn't a posibility because Lisp uses eager evaluation. This is why "if" has to be a macro; if it were a function, "(if (x = y) (do-this) (do-that)) would evaluate the condition and then both consequences before it passed the results of all of these to "if".

Smalltalk avoids the problem with lazy evaluation. When you say "(x = y) ifTrue: [ self.doThis ] ifFalse: [ self.doThat ] you are passing two as-yet-unevaluated blocks of code to the boolean object resulting from (x = y), and that boolean knows whether it's true or false, and thus knows which block to execute. (I find this amazingly elegant, by the way.)

The beauty of Lisp, well, moreso Scheme, is that it's a very simple language (so you don't get caught up in trying to find the odd way to do this or that, a la Ruby) and you can essentially write new languages in it.

### Oh, and the last bit I left o

Oh, and the last bit I left off of the previous post:

The beauty of Lisp being that you can use the macros essentially to write new languages, well, it's a good thing in many ways, all of us agree. But I've met Smalltalkers who claim to be able to do pretty much the same thing in Smalltalk without macros, so perhaps macros aren't really necessary, but are more a crutch to get around the eager evaluation issue.

### ...and Tcl too

You can do a similar thing to Smalltalk from Tcl too: if you use [] brackets then the argument is evaluated eagerly, but if you use {} then it is passed in as-is (as a string). For instance, just yesterday I wrote this little DSL for computer game AI on the comp.lang.tcl newsgroup:

plan Attack {
when {$health < 40} do { goal Retreat } when {$nearest < $range} do { fire_at$target }
otherwise { move_to $target } } plan Retreat { when {$health > 60} do { goal Attack }
when {![at $base]} do { move_to$base }
otherwise { repair }
}
goal Attack


The action parts of rules are only evaluated when their conditions are met (and when no previous rule in the plan has fired — a form of teleoreactive program). The actual code for the DSL is quite tiny:

proc goal {name} {
global GOAL
set GOAL $name } proc plan {goal body} { global GOAL ACTIONS ELSE set GOAL$goal
set ACTIONS($GOAL) [list] set ELSE($GOAL) ""
uplevel #0 $body } proc when {cond _do_ action} { global ACTIONS GOAL lappend ACTIONS($GOAL) $cond$action
}
proc otherwise {action} {
global ELSE GOAL
set ELSE($GOAL)$action
}
# Typically, "run" would be incorporated into your game main-loop
proc run {} {
global ACTIONS GOAL ELSE
while 1 {
set fired 0
foreach {cond action} $ACTIONS($GOAL) {
if {[uplevel #0 [list expr $cond]]} { uplevel #0$action
set fired 1
break; # Only first matching action fires
}
}
if {!$fired} { uplevel #0$ELSE(\$GOAL)
}
}
}


Hope you'll excuse the use of global vars: it was just a quick demo. I wonder what a Lisp macro version would look like?

### "if"

Actually, "if" is a special operator in Common Lisp (and in pretty much every other language), exactly because it is non-strict in its second and third argument. But that does not automatically make a language not "truly macroless", of course.

(p.s.: I don't like no double negations. :))

### Is that really what's meant by "lazy evaluation"?

Seems like this is just the same short circuit evaluation we see in C, Pascal... maybe I'm just used to seeing lazy evaluation mean delayed evaluation.

### smalltalk isn't any lazier than lisp

the only difference is that smalltalk has a lightweight notation for thunks.

(if (= x y) (lambda () (do-this)) (lambda () (do-that)))

is equivalent to what you wrote (assuming full eager evaluation). I believe goo & arc have a more lightweight notation.

### wrong

those blocks that smalltalk uses for control flow are simply thunks (lambdas). So your contention regarding smalltalk is incorrect.

### Complexity

It's worthwhile to distinguish between unnecessary and inherent complexity. Programming languages can help reduce the former and help manage the latter.

### True, but...

Even in the world of the perfect language, there will still be a very large measure of inherent complexity.

More than we can really cope with.

The Dilbert Principle says it well. "People Are Stupid." You, me, the author of Dilbert, Einstein, we are all too plain way too stupid to really understand the enormous programs the software industry is creating.

This is where we need a _lot_ of help.

### Even in the world of the perf

Even in the world of the perfect language, there will still be a very large measure of inherent complexity.

Obviously. I never implied otherwise.

### Abstraction Stratum?

How can we cope with complexity? How much of this complexity is inherent vs accidental? (Thanks, Fred!) Given:
• Joel Spolsky's Law of Leaky Abstractions
• Abstractions = page faults
isn't the solution to reduce the number of abstractions we use? I don't mean we should return to writing assembly code or even C, but that maybe we need to revisit our foundational abstractions. Thin abstractions are good; abstraction stratum are bad.

For example, the Java hype (purposely) confused the advantages of garbage collection and virtual machines. J2ME is the worst example. If you have a mobile phone with a slow processor, why do you want to write software for a JVM bytecode interpretor? Write native code, please. And whether garbage collection is a good or bad idea for mobile phone software is a separate question. :)

### Good points

We need more analysable languages.

I agree. But to focus the issue, let's make itclear that the difficulty of building a parser isn't the real issue. The issue is having clear, formal and useful static semantics. Hence, for example, the advantages of having to think carefully about state in pure languages (note: this is an example, I don't think pure languages and monads should become mainstream, the way they currently are).

Preprocessors, code generators, dynamic scoping and macro systems may make code easier to write and more terse, but complicate the work of code analyzers immensely. I get the feeling that time will eventually show them to be a bad path.

Two points: (a) there is work on making meta-programming more manageable/analyzable (e.g, type systems for macros, multi-stage programming languages etc.) We discussed some of this work in the past. (b) A separate yet related concern is that code generation techinuqes can potentially make code harder to follow and understand (by adding complexity to the solution that isn't part of the problem domain). My position is that in enough cases the opposite is what happens: meta-programs are easier to understand and hide unimportant details by raising the level of abstraction (essentially creating a DSL). Yet, deciding whether the impact on readability and maintainability is positive or negative is, and will remain, an engineering decision which will involve careful analysis of the tradeoffs.

### difference between software engineering and computer science

I believe it was highlighted by this comment:

"How do we bridge the communication gap between business (and government, etc.) people who want all the benefits of IT - NOW, cheaply, and without taking much trouble - and the technical experts who know (or can find out) how to build software that works?"

### Lack of communication = problem not explicitely defined

I think the problem of software engineering is the same problem in our everyday lifes, including programming languages: when we describe a problem, we don't describe it completely, but we leave out various details that we think they either are insignificant or implicit. In other words, we do not define a problem explicitely.

A large part, maybe the biggest one, is spent in a project for communication purposes. Software engineers and system analysts (most often the same persons) spend a lot of great effort to understand the customer's requirements. If the customer is organized properly, then the requirements are easily formulated and occasionally formally expressed. But that is not the usual case; the most usual case is almost chaotic: every person knows his/her job in a very fuzzy way, usually disconnected from the process it belongs. And most people can't fully and explicitely describe the requirements of their job, because they know their job by heart.

So if people can't express themselves in a mathematical way, the burden is on tools: the tools must flexible enough so as that changes can be quickly done, without too much breaking of the stuff that already runs. Unfortunately, all mainstream programming languages are anything but flexible.

The problems with software development, in my opinion, are:

• the nature of software development: programs need to be 'compiled' instead of just 'executed'; after all, a compiled program is just an optimized version of the original. This makes software development slow.
• object orientation: it may come as a suprise, but I think the 'cast-in-stone' coupling of data and code of object-oriented programming languages is evil. It certainly makes changes difficult to incorporate. Classes should be 'extendable' both horizontally and vertically. What is needed is pattern matching, in the way functional programming languages do it: the piece of code to invoke depends on the type and content of values that the piece of code is invoked with, decided at run-time. Again, the selection of method to call at compile time is just an optimization. Classes are not just different structures of values, but values themselves can be different classes (after all, a type system is a system of categorizing values according to their properties).
• the concept of 'application': just like object-orientation, applications are huge monolithic systems that can't be modified unless great work is inserted into them. More loose coupling is needed; the ideal would be if data exist in some store independently of code that processes those data. Code that processes those data in a meaningful way should be written in small independent modules, so it can be easily changed, thrown out etc.
• The distance between databases and programs: database datatypes do not map with types of programming languages; a great deal is spent in trying to interface with databases within a programming language. Database constructs do not map directly into programming languages. Take Java, for example: a whole lot of classes must be created for each project that are mapped to a database's tables and views. These classes are totally useless, because their only purpose is to be an interface between the programming language and the database.
• The non-distributed nature of programming languages: 'modern' programming languages are not transparently network-enabled. For example, there is no concept of Model-View-Controller across an intranet; I just can't write a simple module that attaches itself to some event in the intranet. Therefore writing distributed applications is very hard, and various technologies are often vastly incompatible.
• Lack of standards for description of information: there is no unified away for describing the nature of data across different programming languages, operating systems and architectures. Ideally, a number is just a number. In reality, there are multiple definitions: each programming language and database defines its own datatypes, usually incompatible in some foundamental way with all the others, usually because of the selection of trade-offs.

### Programming Systems

I'd agree that we need languages with better modules, type systems and distributed computing support. I also agree with the above comment regarding program analysis.

The biggest front on which I think productivity can be improved in SE is development environments. Having a language with the above wishlist items (as well as some form of meta-programming IMPO) that was also designed with a source controled, refactoring IDE in mind would be a boon.

That and reducing the no. of design by commitee standards.

### Things to improve

...we need languages with better modules, type systems...

What might be more useful is developing a better understanding of how to work with the module facilities and type systems we already have: What constitutes a "good module"? Or a "good type scheme"? What metrics can we use to help compare two different module or type schemes, and pick the best one for the problem at hand? How can we determine how well a particular set of module definitions, or a given type scheme, fits a problem domain?

IMHO part of the problem with modern "software engineering" is that there's no way to judge a good design. As far as I am aware, we have no way of judging which of two (or more) possible abstractions is the "better" one. In a lot of cases we don't even know what criteria we should use to judge "better" (aside from vague concepts like "coupling" and "cohesion"). Since the job of software engineers is essentially to design "good" abstractions, the lack of tools with which to judge "good" seems like a particularly serious problem to me.

Of course, these problems are only made worse by the fact that the customer often is not able to fully communicate the key features of the problem domain. How can you design an "appropriate" (let alone "good") abstraction if you don't have a clear understanding of what it is you're trying to abstract? The result is that developers demand tools that allow them to rapidly reconfigure their abstractions as they gain a better understanding of the problem domain (that's possibly one of the reasons that "dynamic" languages are so popular - and why "refactoring" IDEs are in such demand). So perhaps it would be a good idea to develop better techniques for understanding problem domains as well.

### Re: Good Points

The issue is having clear, formal and useful static semantics

You've said well. To my mind, this is the central issue.

A separate yet related concern is that code generation techinuqes can potentially make code harder to follow and understand (by adding complexity to the solution that isn't part of the problem domain). My position is that in enough cases the opposite is what happens...

I also agree. Code generation is a tool that, like any tool, should be carefully chosen to suit the problem at hand. When used this way, I've found that the complexity of the generated code is strongly correlated with the complexity of the problem. This is especially true of (explicit) state machine compilers.

I would argue that the readability/maintainability argument against code generators is specious since generated code should not be expected to be readable/maintainable by anything other than the generator itself.

I'd learned early that one should never revise generated code by hand, but refine the specification until the generated code behaved as expected. This wisdom is captured in the conventional code/debug cycle.

If, on the other hand, the specification cannot be refined to obtain the desired behavior, then the tool is lacking power, which is a separate issue. In sum, readability/maintainability can be considered by the producers of the code generator, but shouldn't matter so much to the rest of us.

### code generation

I would argue that the readability/maintainability argument against code generators is specious since generated code should not be expected to be readable/maintainable by anything other than the generator itself.

I agree. But I triedto debate another argument as well. According to this argument, the problem isn't the the generated code is unreadable (they'll grant that that this isn't really an important concern), butthat code generation makes the software architecture more complex. There are cases where this is true, but the generalized assertion isn't.

### Code generation can be...

...a sign that the programming language isn't expressive enough to capture the abstraction pattern in question. Instead of kludging together a code generator in the mismatched PL, sometimes it would just make more sense to move the problem to a more capable language where the pattern can be expressed sans code generation.

### What counts as generation?

In many languages you can simply embed a DSL in the same language; Scheme and Lisp are the obvious ones, with the macros. Haskellers often make combinator libraries, which are nothing more than a DSL. Even C++ can be used to embed many things, for example Boost.Spirit, which is an LL1 parser that allows BNF to be written (bastardized) in native C++. There are even people trying to create DSLs in Java (there was an earlier thread on that). Do these count as "generating" code? I've personally used C++ templates to create optimized sparse vector libraries, which have their multiplication routines generated from a multiplication table.

### Re: code generation

code generation makes the software architecture more complex

In what way? Do you mean topologically? I'm supposing that generated code introduces obscurity into an endeavor that, as semanticists, we greatly desire be transparent.

### Both

I meant both of these. Just so I am clear: I am not saying this is the case, I am saying that the argument is often made.

Semantically, if done well code generation doesn't have to be a problem. The specification language can have sound semantics, the code generator verfied, and the resulting code can be analyzed as well. It would be great to have traceability of semantic information between all levels. This would mean better tools/multi-stage languages.

### Re: Code generation can be...

...a sign that the programming language isn't expressive enough to capture the abstraction pattern in question

I can't disagree with you...though I really want to :^).

Do I divine from your comment, "kludging together a code generator..." that you distrust code generation? If so, why?

I guess I'm not a purist at heart since I believe that code generation has its place. I think it's prodigal to move every problem for which the abstraction pattern cannot be easily expressed in an extant language to a new one. What, then, would motivate abstraction?

I'm guessing that you're an exponent of DSL's then? They, too, have their place, but it seems to me that DSL's and code generators (human code generators included) so potentiate each other that to champion one is to champion the other.

I guess I just can't penetrate the reasoning behind trumpeting one at the expense of the other.