Lambda the Ultimate

The traditional decomposition into the two questions of "Is there a correct specification?" and "Does the program implement the given specification?" have some merit. In most engineering, including civil/structural, it is almost taken for granted that reliable constructions need to be specified with degrees of error tolerance and fail safe backups. One version of applying that to programming to check if a program implemented a spec would mean that functions were first specified and then handed off to different independent teams of programmers to implement independently. CASE tools could then check whether the independent implementations gave identical behavior for all code paths. Independent tests of specification correctness would also be helpful, but harder to automate if the specification isn't formalized in some logically complete language. But perhaps "test oriented specification making" would allow good test coverage of where a spec is approximately correct, and the idea of cross validating spec work from independent teams would again be helpful.

Cost (time, money, and run-time efficiency) is usually a hidden issue in all these discussion; producing large, error filled programs is very expensive, and producing much more reliable programs would be more expensive on one or more cost dimensions. The builder/programmer comparisons tend to be unfair to the programmer in a bunch of ways, starting with the fact that the large program is much, much, bigger than the large building in an information-theoretic sense - i.e. bigger in terms of how many bits are required to specify what matters about that particular program vs. that particular building. Buildings and their construction are more similar from one to the next.

... the so-called free market looks more like an abstraction than an actual functioning system, never mind anything fit to model good engineering on.

I've always said that if an engineer designed a system whose behavior was as unstable that exhibited by market economies, he'd never again find work as an engineer.

Is that unstable markets continue to function and correct themselves, whereas unstable programs (at least the way we make 'em now) just crash and die.

Most software does not need any of this: if you examine Martin Rinard's work, you will see experiments to demonstrate that your premise is wrong:

A single bit going stray usually crashes a program, or causes it to give ridiculously wrong answers.

There are many caveats to this -- say hardware becoming more unreliable as it continues to shrink -- but that does not necessarily require a change to how we specify software. A lot of the ideas you mentioned are interesting (e.g., lottery scheduling is a Good Thing and fits into this model), but I suggest focusing more on the problem before picking your solution.

For very large scale programming the use of service markets makes sense. Certainly, achieving this as common in the large scale will resist Denial of Service attacks, Viruses, Worms, and other forms of malice that will suddenly become 'expensive' to create and maintain.

Automating the market can help make it much cheaper and more competitive than it is today, where the need for human intervention has a cost (and effectively causes vendor lock-in). That is, people will be able to compete by offering their own resources and services on the market, build reputations for quality, etc.

That said, I doubt you'll want "threads" to be the elements paid. At the low level, it'd be clouds and independent hosts. At the high level, it'd be services, which could be running any number of threads. The services would pay other services and pay clouds and independent hosts.

How, in a programming language, would you handle a marketplace of software services, where threads advertise what they do and offer rates for which they'll do it (and the schedules on which they will do it, where appropriate)? Can that be abstracted below programmer notice, or not?

Is there any reason to think this is a programming language issue?

To build large-scale robust systems you need two things: redundancy and feedback. I don't understand why you limit your inspiration to ecological and market-based systems. A more direct inspiration is (parts of) biological organisms, like the human respiratory system.

In computing, probably the platform that lets you do the most in this regard is Erlang. It supports redundancy (each process or "agent" is independent) and feedback (each internal node in a supervisory tree is a feedback loop guarding part of the system). We have extended these ideas to loosely coupled systems: see The Limits of Network Transparency in a Distributed Programming Language.

How, in a programming language, would you handle a marketplace of software services, where threads advertise what they do and offer rates for which they'll do it (and the schedules on which they will do it, where appropriate)? Can that be abstracted below programmer notice, or not?

I'm fairly skeptical about the general applicability of these ideas.

The reason is that mathematical models of markets (and ecologies) are essentially feedback systems whose semantics are given as the implicit solution to a fixed point equation. E.g., the proof of the existence of Nash equilibria in game theory works by an appeal to Brouwer's fixed point theorem. However, the computational cost of calculating these fixed points is, in general, quite high -- finding Nash equilibria is the archetypal PPAD-complete problem. So this means we face the traditional obstacle to declarative programming: if we view specifying a program as a market-optimization problem as a kind of declarative program, then we face the problem that generic strategies for solving these problems can be very slow in practice.

Now, there are subclasses where efficient solutions are possible: Daskalakis and Papadimitriou have shown that polynomial time approximations are possible for anoynmous games. This may make a reasonable basis for a DSL.

Your points about the complexity of, eg, finding equilibria are well-taken, but why would any specific part of the system need to do that? Equilibria in markets and biomes are emergent properties. A trader is never thinking about the market as a whole; he is making a decision about the current deal. A woodchuck, similarly, doesn't work out exactly how much oxygen it needs to consume and how much CO2 it needs to produce in order to have the optimum regulatory net effect on the local biome. It just breathes as much as it needs to, thrives and raises young if the biome favors its needs, or has limited success of the biome does not, and the equilibrium in the local biome emerges over woodchuck generations.

Just a note, but the word "equilibrium" itself is something of a misnomer; what markets and biomes find are more properly attractors which affect a dynamic (chaotic) current configuration.

"Just a note, but the word "equilibrium" itself is something of a misnomer; what markets and biomes find are more properly attractors which affect a dynamic (chaotic) current configuration."

You want to apply chaotic processes modeled after those that occur naturally to what problems? Flying the Space Shuttle? Keeping your credit card balance accurate over time? How is that desirable anyway? Isn't there a field that researches this already? Distributed agents? Network protocols?

This is all just non language related bloviating in my opinion.

See "Combining Agoric and Genetic Methods in Stochastic Design" at either http://autogeny.org/chsmith.html

or at it's Google-cached copy

http://74.125.47.132/search?q=cache:rdB35R8s8ocJ:autogeny.org/chsmith.html+autogeny.ORG/chsmith.html&cd=3&hl=en&ct=clnk&gl=us

In particular, scroll down to the section titled "Charles Smith: an Evolutionary Economic Framework".

Even more relevant may be Eric B. Baum's Research at http://www.whatisthought.com/eric.html

Scroll down to "artificial economies of agents that reinforcement learn". Start with Baum's paper "Evolution of Cooperative Problem-Solving in an Artificial Economy" which describes his program Hayek.