Lambda the Ultimate

Gerald P. Weinberg's second law says: "If builders built houses the way programmers build programs, the first woodpecker that came along would destroy civilization." We've all heard that one, right?

Thousands of reams of paper have been spent (wasted?) on "what can we do about it?" And, to date, we don't have really good answers. A single bit going stray usually crashes a program, or causes it to give ridiculously wrong answers. Every program has to have everything it depends on arranged just so, or it ignominiously fails or bails out with an error. And entire industries are built around putting together suites of software that don't have conflicting requirements that prevent them from working together, or babysitting complicated applications that will die horribly if there is so much as a bobble in any of a score of systems that provide it with a score of different things it needs.

What we've seldom done is to look at the relatively robust very large systems that we *have* studied and try to abstract principles that can be applied to programming. Two such systems are ecologies and free markets. Both are characterized by: many relatively independent agents with limited lifespans; agents in proximity, even if previously unknown to each other, are able to perceive each other and interact; competition and cooperation between independent agents; a small number of outliers, atavisms, recessives and contrarians constantly explore non-mainstream strategies for many generations, even when those strategies have failed or had limited success in the past; survival/reproduction advantages given to more efficient (or more correct) agents; diminished returns for doing something when there are many agents which do that same thing, and, crucially, dynamic reconfiguration of dependencies in that when some agent fails (dies or goes out of business) other agents that formerly depended on its output can switch to depending on different agents rather than dying or going out of business themselves.

Can these lessons be taken to heart in a software system or programming paradigm? Should threads "pay" other threads for services, in tokens redeemable for compute time or memory space (and, in finding a "going rate", eliminate inefficient providers?) How would we simulate "ecology" or "marketplace" of services that different threads can provide, to one another and ultimately to end-user programs? Should threads reproduce or "copy successful business strategies" automatically?

I think you'd need to keep track of state pretty carefully. If a subcontractor makes a hash of something, or dies due to an error, you need the power to detect the problem, revert your data, and hire a different subcontractor. So each thread would need data versioning, or something like it; the power to remember the current state of its own local environment, and return to that state if needed. You'd also need to automatically check subcontractors' work against each other from time to time, to detect errors/cheaters.

I think you'd need a "DoOneOf" statement, for trying to do something in each (or all) of many different ways (or via subcontracting to many different threads or spawning many different threads) and proceeding if any one of them is successful.

I think you'd need to automatically make occasional "configuration tests" (maybe whenever launching new threads of the relevant type) even when things were going fine, to find, for example, databases at new locations, new network interfaces, etc. This might be handled via a "DoOneOf" mentioned above - and would also automate load-balancing, since the query to the least-used database or interface would tend to return first and thereby get "paid" for the completion of the task.

I think you'd need capabilities (revocable or failable service hooks via other threads) rather than a simple linked-call interface.

How, in a programming language, would you handle a marketplace of software services, where threads advertise what they do and offer rates for which they'll do it (and the schedules on which they will do it, where appropriate)? Can that be abstracted below programmer notice, or not?

Ray