Lambda the Ultimate

Looking over the description, it looks like there's a recognition by the organizers that there's a lot of work on the purely software engineering side, otherwise language-design for library-building is gonna be more or less a shot in the dark.

Considering that libraries have been standard practice in software engineering since almost the beginning of the computer era, there seems to be remarkably little work done on what makes a good library and what makes a bad one. Even the folklore is sketchy. This could be because 9X% of developers never write a library, but even criteria for evaluating the quality of third-party libraries are not well known or used.

At the language design level, it seems like most work gets done at the data-object (ADT/class/whatever..) and control-flow levels, with design support for modules and libraries being decidedly an afterthought.

Surely "9X% of developers never write a library" (Dave) is false if you allow yourself to equate the concept of library with that of any factored out entity or group thereof. And to this day I still fail to see any description of "library" that infirms this equation. "Factored out entity" meaning simply an entity e.g. a procedure that is used e.g. called (typically more than once, typically at more than one point) by another entity and eventually by the main program.

Surely "9X% of developers never write a library" (Dave) is false if you allow yourself to equate the concept of library with that of any factored out entity or group thereof.

If you use that definition, you're probably right. But then if you use that definition, you're talking about something completely different from what most software engineers talk about when they use the word "library". What you're talking about is usually called a "module" or "package", and is both vastly more common and vastly easier to build than a library.

If I had to give a first shot at the differences, I'd say that a library is a module which is independently packaged, used in multiple applications, possibly used in multiple application domains, used by programmers who do not have personal access to the library developer, and is used by programmers who either don't have source access or don't use source access. It also needs to be of high enough quality that developers will use it, rather than reinventing it.

Factoring out functionality is easy. Factoring out functionality in forms that will actually get reused by harried developers is surprisingly hard.

It's good we shoot at the differences.

I should have said *fully* factored out. That takes care of "independently packaged" and "multiple applications" instantly.

Further assuming *well written* entities (including documented non-obvious behavior), we have:
"access to the library developer" => not required
"access to the source code" => not required
"high enough quality" => assumed

Also, "package" is equally underdefined. You yourself use the term with two different senses.

It should be clear that I'm interested in a 'positivist' approach to this problem, meaning using (finding) rock solid, grounded, (quasi)formal concepts. I would not mind at all coauthoring a workshop paper on this. Yesterday I jotted down the following:

Library development driven by program development

This paper brings attention to the fact that the concept of software library currently subsumes that of fully factored out entities. A "fully factored out entity" being any entity e.g. a procedure

- whose definition does not depend on the user context e.g. of 'global' entities visible at the call point¹

- but it is used on that context e.g. called, typically more than once, at more than than one point, with different arguments on each occasion.

Given this state of affairs, the anedoctal, but often assured (LtU), claim that most programmers never write a library is utterly false.

In fact, the structure represented by factored out entities, fully or not, and their use, is an ubiquous programming idiom, as a look at any moderately sized program will show.

Factored out entities are a result of either good old top-down design (Wirth), or refactoring (XP). Or, as usually is the case, both processes.

Notes

¹ In most languages. Exceptions include Algol 68, where a procedure may use names defined only at the point of call.

To add to the discussion of what a library is, it often seems to me that libraries and DSLs are not so different. You (can) have libraries to talk in a certain DSLs, and DSLs to use libraries, but isn't all this just wrapping things around? Where should one draw the line, if there is any to draw?

The LtU archive contains many discussions related to this issue.

Most DSLs are just a syntax for some library. A library proper has the advantage of perfect integration with the 'main' language. Using a DSL as a library falls into the multilingual programming problem. We could make a brief comment on this on the paper, but I don't think it is essential. I'd like to focus on the conceptual problem of distinguishing between library and fully factored out entities.

There are also many poplular languages that focus on gluing existing library components together; e.g., VB, Python, and in some respects even Java. Abstractions needed to implement a library (mostly algorithmic) are not necessarily the best abstractions to reuse the library (focus on ease-of-use, complexity hiding).

I was able to download it, anyway.

BTW, everyone should know about scholar.google.com for this this sort of thing. Searching for a paper will give you not just it's location at Springer or portal.acm.org, but also any other locations it can be found (something plain google is loathe to do). Since papers often show up on research groups' or author's personal web sites, no Springer tax. I'd estimate you can find about eight papers out of ten, that way.

I should try clicking before I start speaking :)

I've noticed that the newer python libraries are suffering from this extendability-itis. To do any work at all, one has to subclass an existing implementation.

Gone are the days of smtplib when sending email was a three liner.