Lambda the Ultimate

This gets me thinking further.

I think there's an implicit dichotomy in "Is it better to let IT systems become a mess and periodically just replace them by brute force (and start a new mess) -- or is it more profitable to make an IT infrastructure that will last for ages and hum like a hummingbird?"

I wonder if that's a false dichotomy. Could there be some sort of balancing act? And if so, how would we know when we've reached it?

Should it be proposed that profit / loss could inform those questions as well?

Should it be proposed that profit / loss could inform those questions as well?

Because of competition among the major shareholders, the firms will tend to consider nothing but the profit of dominant shareholders. Nothing you can propose will change that.

At the risk of over-generalizing, you'll find that it's very hard to make much of a business case for "better" business IT systems (where "better" is from the non-money perspective of "cleaner code" and things like that).

(Back in the 1980s and early 1990s when "software engineering" was to become a hot new discipline, academics used to flatter themselves to believe that software quality really mattered much.)

The problem you face is that for most firms with an IT budget, the IT budget is tiny. There is no business case for making IT a little bit better because the impact on the overall budget will still be tiny. Its easiest just to stick to hiring cheap, commodity coders and a few wranglers to keep them going in an acceptable direction.

It might be worth noting that this approach to IT has created a kind of environmental disaster. You can see evidence of that in things like all the recent massive credit card data thefts. You can see evidence of that in things like the Heartbleed bug which occurred (I heard on the radio today) partly because even though millions of firms use the OpenSSL code, and even though firms like Red Hat and Ubuntu "support" the code, no firm has ever thought it a fitting use of their money to pay for even so much as a security audit of OpenSSL.

The environmental disaster doesn't really change the decision to keep hiring cheap programmers and keep using messy code. The costs when things go wrong get spread around far and wide. The loss to each firm is typically not very large.

Should it be proposed that profit / loss could inform those questions as well?

Because of competition among the major shareholders, the firms will tend to consider nothing but the profit of dominant shareholders. Nothing you can propose will change that.

At the risk of over-generalizing, you'll find that it's very hard to make much of a business case for "better" business IT systems (where "better" is from the non-money perspective of "cleaner code" and things like that).

The problem you face is that for most firms with an IT budget, the IT budget is tiny. There is no business case for making IT a little bit better because the impact on the overall budget will still be tiny. Its easiest just to stick to hiring cheap, commodity coders and a few wranglers to keep them going in an acceptable direction.

It might be worth noting that this approach to IT has created a kind of environmental disaster. You can see evidence of that in things like all the recent massive credit card data thefts. You can see evidence of that in things like the Heartbleed bug which occurred (I heard on the radio today) partly because even though millions of firms use the OpenSSL code, and even though firms like Red Hat and Ubuntu "support" the code, no firm has ever thought it a fitting use of their money to pay for even so much as a security audit of OpenSSL.

The environmental disaster doesn't really change the decision to keep hiring cheap programmers and keep using messy code. The costs when things go wrong get spread around far and wide. The loss to each firm is typically not very large.

(Back in the 1980s and early 1990s when "software engineering" was to become a hot new discipline, academics used to flatter themselves to believe that software quality really mattered much.)

If you get one that wants to make programmer life easier by investing a little to "clean things up", it's often cheaper to ditch that person and keep hiring cheap programmers that will just work long hours maintaining junk.

That can't possibly be true. The cost of training new employees and the lost productivity while they learn enough of the code to be useful is very high if code is junk. It would almost certainly exceed the cost of letting the trained and knowledgable employee spend some time cleaning things up "a little".

If we're talking about "a lot" of refactoring, then the picture changes somewhat, but the intersection point where time spent refactoring is no longer worth the cost can't be t=0 in any program except write-once throwaways. If growth is high, then the cost of refactoring would become even more attractive and would pay for itself in lowering the entry cost incurred when adding new labour.

And to bring it back to programming languages, any language that makes it harder to write crappy code, or for which any code is more understandable and thus which reduces all of these costs automatically, should be economically preferred. The problem then is having a sufficiently large labour supply with knowledge of this language to make the business case.

That can't possibly be true. The cost of training new employees and the lost productivity while they learn enough of the code to be useful is very high if code is junk.

The market is flooded with cheap programmers who won't rock the boat trying to clean things up in any serious and lasting way. The issue of having to train replacements for those troublemakers isn't a large issue. Mainly nobody hires people like that in the first place, anymore.

And to bring it back to programming languages, any language that makes it harder to write crappy code, or for which any code is more understandable and thus which reduces all of these costs automatically, should be economically preferred.

Now there's where training expenses really do constrain the market.

The problem then is having a sufficiently large labour supply with knowledge of this language to make the business case.

The critical mass of new-language adopters you'll need is a mass large enough to contain lots of cheap programmers who will work stupid hard rather than clean up messes.

The market is flooded with cheap programmers who won't rock the boat trying to clean things up in any serious and lasting way. The issue of having to train replacements for those troublemakers isn't a large issue. Mainly nobody hires people like that in the first place, anymore.

This just lifts the cost to the business level: businesses that don't refactor some quantifiably "efficient" amount X aren't competitive against those that do because they have to throw increasingly more labour at every problem. Doesn't matter how cheap this labour is, it adds up. And if refactoring is correlated to lower bug counts, as seems to be the case on average, then the business that refactors will always have a technical advantage as well.

I understand you're making the case that when some difficulty threshold is reached, it might be cheaper for businesses to simply throw it away and rewrite it than making incremental maintainability improvements, because the industry will have changed so much in the interim anyway, ie. changing languages, platforms, standards, etc. I think this will be the case for some businesses, but I'm skeptical that it's nearly universal as you imply.

For instance, consider all the internet companies like Amazon, Google, etc. that would face millions in lost sales if some small change prevented customers from paying for their purchases. You're suggesting that Amazon uses unskilled programmers to add features without breaking checkout, and it's sinking progressively more and more into empirically checking that customers can successfully pay for their purchases, instead of using skilled programmers that refactor code to minimize the chance of a change preventing a customer from checking out. At some point, adding a feature will become infeasible, and the entire Amazon codebase will be rewritten, and this will cost less in the end than refactoring incrementally. Is that an accurate characterization of your position?

For instance, consider all the internet companies like Amazon, Google, etc. that would face millions in lost sales if some small change prevented customers from paying for their purchases.

For one thing I'm pretty sure small changes break things regularly at such companies -- just during small test deployments.

For another, those companies don't count, in a way. A very tiny number of programmers (much fewer than the # of programmers they employ) handle their most critical systems.

Their gross profit per dollar of labor to those core programmers is astronomically high. For those few exceptions it makes sense to do very high quality systems programming. But we are talking about a very small number of jobs out of a much larger "programmer" industry.

For another, those companies don't count, in a way. A very tiny number of programmers (much fewer than the # of programmers they employ) handle their most critical systems.

Right, so you're essentially saying that when the softare has direct implications for the bottom line, refactoring is valuable, and when the software is not so important to the bottom line, ie. practically a throw-away, refactoring is less valuable. I don't think this is contentious.

The only remaining contentious part is the claimed prevalence of software that has no direct impact on the bottom line. In my experience, software that isn't important to the bottom line is a, possibly customized, off-the-shelf commodity software package. No one employs programmers to write this stuff except the commodity software distributors. Software developers are generally only hired to write software that impacts the bottom line. There would be no advantage in hiring them otherwise.

No doubt there are some developers that live on some fringe where software has small enough impact on the bottom line but isn't available as a commodity and yet is still sufficiently necessary to justify hiring someone to write it, but you seem to be claiming that most of the software industry lives there. This doesn't sound likely.

I think I'm saying something a bit more nuanced.

Let's say, as an approximation, firms can spend more on more highly skilled software development labor, doing more to make well built systems. Or on the other hand they can spend less on less skilled labor, building systems that tend towards being a mess.

There's two margins to compare: the marginal gain of "better" software vs. the marginal cost increase of the "better" labor.

Looking around at the market, at who is working, at what jobs are offered, at what IT systems you can more or less see on the surface -- I conclude that the marginal cost of the "better" software is, approximately, never worth it.

Software developers are generally only hired to write software that impacts the bottom line.

Again, it's the comparison of those margins that matters.

Looking around at the market [..] the marginal cost of the "better" software is, approximately, never worth it.

I'm sure there's no bias in that estimate whatsoever. ;)

I find it difficult, looking at a body of software, to estimate how many hours and dollars were spent cleaning it up (e.g. across previous development cycles). I also find it difficult to estimate how much worse code would be if it were not cleaned up, or how much better it could be if it were cleaned up.

And if I were to look around the market today, I think it would also be difficult to account for the software companies that are not on the market, e.g. because they failed to survive. We would miss the data from their relative expenditure on maintainable software, and what it cost them. For someone who is arguing towards balance, such data might be useful.

I believe we need a more rigorous approach to measuring or estimating these margins before we can reasonably confirm or deny any hypotheses.

So that we are on the same page: I'm sure that something called "code cleanup" happens very often and that overall the result is still systems that are a mess.

I believe we need a more rigorous approach to measuring or estimating these margins before we can reasonably confirm or deny any hypotheses.

I'm all in favor of greater rigor. As things stand, I draw my conclusions based on qualitative data such as experience with popular platforms and their libraries and documentation, reading the communications of working programmers, hearing anecdotes about the job hunting experiences of programmers with various degrees of skill, looking at job listings, looking at the conduct of critical free software projects, looking at agendas for professionally-oriented conferences, reading informed accounts of high-impact bugs, looking at how the institutional forms of commercial software development have changed over the years,.... that kind of thing.

Many of your arguments are based on a premise that the market is a rational actor - i.e. that X is 'worth it' if the market says so. But now you're saying "something called 'code cleanup' happens very often. Presumably, this costs time and money. Does this not conflict with your conclusion, that a dollar spent on "'better' software is, approximately, never worth it"?

Successful market systems are messy. They could be messier. If we spend dollars cleaning up code from 'terrible mess' to 'moderate mess', it seems reasonable to say those dollars were spent on 'better' software. Better is relative, not absolute. Do you disagree with any of these sentences?

I suppose you might be meaning "better than the 'average' software system" for some vague and fuzzy average. That was not how I originally interpreted your words. However, I'd be more inclined to agree if that is your intended meaning - assuming the toolsets are fixed (which they effectively are in the market, with everyone using roughly the same amalgalm of popular web, scripting, and systems languages).

OTOH, even then we might benefit from 'better' software, but simply be limited by our languages and tools - i.e. it costs too much to improve and maintain C++ code beyond a certain soft limit, but different languages may have different limits. Can we design languages and toolsets that give us more bang for the buck towards 'better' software? This seems a feasible possibility for real improvements in the quality of market software.

Many of your arguments are based on a premise that the market is a rational actor

To the best of my knowledge, none of my arguments are based on that premise.

- i.e. that X is 'worth it' if the market says so.

I think I have assumed that what people pay for various products is the price of those products. (Yes, that's tautological.)

I have also made assumptions like "Firms maintain balance sheets" and "Revenue from selling output minus expenditures producing it is profit." (Those are also tautologies.)

I haven't used any abstract definition of "worth it" that I'm aware of.

I really can't quite understand you because you say "the market is a rational actor". That phrase doesn't mean anything clear to me.

Some economic models assume that buyers and sellers are "rational actors" in some technical sense. Whether or how that is the case isn't really important to anything I've said here.

Some economists make the unsubstantiated (and apparently false) claim that "markets are efficient" (in a technical sense). Since I (in another comment) raised the issue of cascade failures then I guess that hints strongly that I do not believe that markets are efficient.

Yes, I said that "something called code cleanup" happens often but I meant that to sound denigrating. Overall, I think most IT systems in use are a mess.

Why would a firm pay for even that much "code cleanup"? Whatever the answer it isn't important to anything I've said. We only know, and only need to know for this discussion, that it's not a lot of money relative to everything else.

Can we design languages and toolsets that give us more bang for the buck towards 'better' software?

The option to significantly reduce risk doesn't seem to be worth very much in the industry. The option to increase "productivity" as employers measure it might have some value but since productivity in this area is already very, very high it is hard to imagine it getting much better.

(Indeed, a lot of what some programmers do these days is increase efficiency in other areas of production, so that wages can be lowered and/or fewer people employed.)

I think most IT systems in use are a mess.

I believe the same.

We only know, and only need to know for this discussion, that it's not a lot of money relative to everything else.

I'm unable to follow the apparent leap in logic from "IT systems are a mess" to "relatively little money is spent on cleanup during an IT system's lifetime".

Can we design languages and toolsets that give us more bang for the buck towards 'better' software?

The option to significantly reduce risk doesn't seem to be worth very much in the industry.

Due to circumstances, price might be higher than value (thus 'not worth it'). But that doesn't imply the value is low, nor that the high price is essential.

Technologies, such as programming languages, have potential to reduce costs and prices. When a price is reduced to become commensurate with its value, that's a soft threshold. New market opportunities arise. At this threshold, a feature that was 'not worth it' before may become 'worth it'. You seem to speak of worthiness - e.g. of software quality - as though it were immutable, independent of circumstance and tooling. I understand worthiness to be a more dynamic relationship; what is worthwhile to the industry can change year to year.

It is reasonable to question the value (actual and perceived) of 'better' code in industry. Unfortunately, prices and expenditures in market is a very poor indicator of value or potential worthiness.

The option to increase "productivity" as employers measure it might have some value but since productivity in this area is already very, very high it is hard to imagine it getting much better.

Woah. How did you come to a conclusion that 'productivity is already very, very high'? What are you comparing it to? This assertion certainly doesn't match my experience or impressions.

I really can't quite understand you because you say "the market is a rational actor". That phrase doesn't mean anything clear to me.

Many of your economic arguments seem to implicitly assume that price is commensurate (strongly correlated) with value. I.e. you tend to argue that the value and worth of something can be approximated by the expenditure of money upon it.

This assumption, however, could only be true in a market with rational actors and excellent foresight and knowledge... where nobody undervalues their own labor or clean air, and where nobody overvalues convention, cronies, prestige, luxury. Of course, such a market is fragile - even a few irrational actors could mess it up for everyone else. I doubt a rational market would survive unless stabilized by something inhuman, like AI.

In real markets, a more accurate assumption is the opposite: that prices and values are very weakly correlated. Many valuable things are free or cheap. Many expensive things have very little value. Many valuable things are unnecessarily expensive. There is very little relationship between the value of a person's labor and the price of it. Waste, fraud, and inefficiency are rampant.

You say (and I agree) that inefficiency results in-the-large from greedy local decisions. However, I would further say that inefficiency is prevalent even in those local decisions. The actors making those greedy decisions are short-sighted, biased, poorly informed, or at least must take into account the irrationality of other actors. We also tend to favor convention or precedent over thinking (best practices, tech fads, buzzwords, cargo cult, etc.).

An argument regarding 'worth' based on observing market tendencies is dubious at its very premise. :/

I doubt a rational market would survive unless stabilized by something inhuman, like AI.

Seems likely a less-than-full AI would be subject to the whims of its operators, while a full AI would be subject to its own whims.

People have many false expectations for AI based on movies or games (why have a boring idiot-savant AI when we could have a sexy and personable Rommie or Cortana?). I think it more likely that most AIs will be highly specialized to specific duties, like idiot savants. Even a 'full AI' might not interact directly with humans, but instead simulate a bunch of avatar personalities like a game master controls characters in a desktop RPG.

Anyhow, even a full AI - one able to socially interact with humans and address ad-hoc problems - would be subject to whims of its operators, and potential biases from its design. The trick is to use more than one AI, such that a single group of corrupt operators cannot game the system.

It's quite possible that people who don't really know anything about AI have opinions formed, to the extent they have opinions at all on the matter, based on movies or games. Indeed, a great deal of ignorance about science is propagated by science fiction written by the scientifically illiterate. However, contributors to LtU are far less likely than the general population to belong to that set. It's more likely that the scientifically illiterate science fiction was based on its author imitating what they thought was being said by someone quite scientifically literate, and the version you encouter here is the informed original rather than the laughably distored imitation.

I see that my use of the word "worth" is tripping you up:

An argument regarding 'worth' based on observing market tendencies is dubious at its very premise.

Think of me as using the word "worth" as a technical term in a formal game called "the capitalist system of production".

An expenditure is "worth it" to a business in this game if making the expenditure benefits the health of their balance sheet, "not worth it" if it leaves the balance sheet in poorer shape.

This is not the same sense of "worth" your spiritual adviser might use, for example. It isn't meant to be.

I'm unable to follow the apparent leap in logic from "IT systems are a mess" to "relatively little money is spent on cleanup during an IT system's lifetime".

I conclude that relatively little is spent on cleanup because relatively little is spent on IT. IT staffs tend to be very small compared to the size of their host firms. Expenditures tend to be small parts of corporate budgets. They are treated mostly as pure cost centers: places to cut; places to not expand expenditures without a very compelling reason.

Little is spent on clean-ups because little is spent on IT, period.

Woah. How did you come to a conclusion that 'productivity is already very, very high'? What are you comparing it to?

"Productivity" here is a ratio. The denominator is, approximately, the wages of IT staff engaged in a particular project. The numerator depends on circumstance. I'll give two examples:

Suppose that an insurance company decides it is worth $X per customer if customers will start using a new mobile app related to some insurance company service. The work of slapping an app onto the side of infrastructure can be done (often is) by a very small number of programmers working for a short amount of time. Many thousands of customers times $X divided by a tiny FTE expenditure in IT is high productivity.

You can compare the productivity of the app developers to that of, say, in-the-field insurance adjusters. The work per claim filed tends to be labor intensive. In the absence of alternatives the work is necessary, but the ratio of return to cost of these in-the-field workers is apt to be much smaller than that of the app developers. Indeed, one incentive for the app development might be to reduce the need for in-the-field adjusters, replacing them with a much smaller number of higher productivity workers.

Another example: a state university system might hire a few IT staffers to replace a variety of self-hosted systems (such as system-wide email, say) with local support for "cloud services" like "free" email from Google. If the net result is a large system-wide reduction in IT staff expenses, that can be considered the output of the very few, high productivity IT staffers that brought it about.

Many of your economic arguments seem to implicitly assume that price is commensurate (strongly correlated) with value.

They really do not. As I've tried to explain to you, I'm not relying on any abstract notion of "value". Firms have definite, quantifiable expenditures and returns. We usually don't need to look beyond those to understand whether a firm will spend more money on this or that.

You have in your mind some abstract sense of "value". That's apparent when you say something like "Many valuable things are free. Many expensive things have very little value."

Whatever vision you have of "value", in sentences like that, the banks don't accept deposits of that kind of value; suppliers don't accept it as payment.

You might wonder why that which customers pay for goods and services seems sometimes so out of whack with your personal sense of value. Maybe you have thought "If people knew more they wouldn't pay so much for this but that they'd pay a lot more for that."

Adam Smith, David Ricardo, Karl Marx, and many others have theorized about how the prices we observe in markets come to be. I'd refer you to literature like that if the topic is of interest to you.

The question isn't important here, though. We can get very far in understanding the markets for software and for PLT advances without having to provide a general theory of prices. We can get very far knowing only that prices are real, observable quantities and that businesses have balance sheets that reflect them.

If you want to somehow step outside the existing determinations of prices and change how some abstract notion of "value" relates to prices, that's a whole separate field of inquiry.

Little is spent on clean-ups because little is spent on IT, period.

How are you measuring expenditure on IT?

We could add up the purchase costs of software and hardware. Or we could add these to the the overhead labor costs of time spent installing, waiting on, recovering after failure, administrating, and maintaining these things. I expect there are many other methods.

Some IT expenditure metrics will be opaque with respect to software quality. Others will offer an effective perspective for judging how software quality affects a business's bottom line. ... And some will give much higher numbers than what you seem to be assuming.

We can get very far knowing only that prices are real, observable quantities and that businesses have balance sheets that reflect them.

We cannot get very far if those balance sheets are opaque with respect to the dimension under study - e.g. the impact of software quality. And, relevantly, neither can business men making decisions (who thus cannot make rational judgements about what is 'worth it').

You have in your mind some abstract sense of "value" [..] banks don't accept deposits of that kind of value

I'm sure banks would embrace the value of 'air' right quick if you withheld it from them. They might even accept deposits. :)

My understanding of value, framed in an economic context, is close to: what is the maximum amount one should reasonably be willing to expend on this resource or feature if it were otherwise withheld? Often, we can pay less than that amount (the price is less than the value), and thus the difference is an implicit profit (in a 'penny saved is penny earned' way). However, paying less isn't always a good thing: it can hurt the providers or be unsustainable. Conversely, the price is sometimes higher than the value.

Rationally, the value of better software is proportional to the lifetime savings using the improved software. The expenditure is the price, not the value. OTOH, for software qualities or features beyond various technology-dependent soft limits, the cost tends to quickly become exorbitantly high. And price tends to follow cost.

If you believe that you have some new accounting methods that will lead to more being spent on software development because you can show the businesses how they'll get a good return on the extra expense, then you should consider becoming a business consultant and getting paid handsomely to present your ideas to corporate officers everywhere.

There are ways to precisely measure the impact of software quality, e.g. one can study UI quality aspects by actually measuring user behaviors. But there isn't very good tooling for it today. Good metrics tend to be expensive without good tools.

In any case, I'm not at all interested in helping companies better understand their actual IT expenditures. I believe companies and markets are irrational and wouldn't listen anyway, unless someone much more charismatic than myself pushed the point. I'm only explaining why I am skeptical of your (many) economic based arguments.

There's two margins to compare: the marginal gain of "better" software vs. the marginal cost increase of the "better" labor.

Except these aren't always at odds. For instance, you ask above why anyone would pay for "code cleanups". One answer that has gained some traction in the recent years is that code cleanups are done in order to open source software in the possible hope of benefitting from free labour.

In other words, the marginal cost of one better developer and his time spent cleaning up code can attract a non-marginal product improvement due to contributions from free labour.

You are right that it's not a lot of money relative to other investment in software, but I don't see why it ever would or should be. If you're spending that much time cleaning up, perhaps you need more competent developers to get a better design from the get-go.

I guess I'm a little confused here. I'm responding to the topic post with my take on why business software is pretty consistently a shoddy mess.

Are you arguing that it isn't typically a shoddy mess? Or that it is but businesses would perform much better if they did things differently? Or, what?

My impression is that it's mostly because businesses don't know how to spend a little more money and get software that isn't a shoddy mess.

I'm responding to the topic post with my take on why business software is pretty consistently a shoddy mess.

If we agree that it's a shoddy mess, then I disagree that it's a shoddy mess for the economic reasons you've stated here. Certainly more investment in refactoring could slightly improve the mess, but not much more than a small constant factor IMO.

IMO, any shoddy mess is a largely result of:

1. many people working concurrently on a single system each with incomplete understanding of it and the problem at any given time (consider a group writing a fiction book, each believing they're writing about a slightly different topic)
2. little industry consistency on best practices, best tools, best language, etc. (although there might exist a rough consensus in academia, this doesn't seem to translate)
3. what sparse consensus exists on the best tools and languages isn't justified by any rigour (why still so much C and C++? why so little Ada, ML, etc. despite the empirical data supporting their adoption?)
4. no regulations for quality control

If any one of these factors were eliminated, many messes would be considerably ameliorated. I'd say their impact would be more significant than investment in refactoring would accomplish.

I'd like to point out that you say you don't buy my economic argument but that all four reasons you give actually support my argument

1. many people working concurrently on a single system each with incomplete understanding of it and the problem at any given time (consider a group writing a fiction book, each believing they're writing about a slightly different topic)

The expense of coordinating these groups isn't worth it to firms that exhibit that behavior.

2. little industry consistency on best practices, best tools, best language, etc. (although there might exist a rough consensus in academia, this doesn't seem to translate)

3. what sparse consensus exists on the best tools and languages isn't justified by any rigour (why still so much C and C++? why so little Ada, ML, etc. despite the empirical data supporting their adoption?)

This is precisely where it would be lawful for firms to form consortia to promote consistent best practices but they don't find it worth the expense (for the most part).

4. no regulations for quality control

Yes, the state could externally and artificially impose higher costs on firms whose IT exhibited bugs, particular bugs whose costs were naturally externalized.

That would, if the regulations were stiff enough, create incentive for businesses to spend more on software quality.

This idea had some currency in the academic software engineering world of the 1980s/early-1990s as I recall. Some argued for, for example, mandatory licensing for programmers.

I don't think it's correct that software shops don't care about clean code. The places I've worked have always wanted cleaner code, but I think there's a question of how much benefit is really going to come from your request to go "refactor" for a week or two. After all, your boss remembers how great you thought this current design was going to be last month and now you're asking for more time to go fix things. Maybe the grass isn't really that much greener. What we have now mostly works and refactoring might look like the greater risk.

If we're talking about blue sky projects, then your question seems loaded:

Does spending more to reduce complexity increase profit or decrease profit?

Shouldn't the point be to push languages / practices that result in lower complexity and therefore lower cost?

Shouldn't the point be to push languages / practices that result in lower complexity and therefore lower cost?

I don't see any reason to accept that as a normative prescription. Seems problematic due to cui bono issues.

I don't have any to offer in its place, either.

Commercial software usually gets more complex over time, as a function of entropy, choice of what to value, team social interaction and evolution, and games. (There is strong incentive to declare victory — to count more victories — whether or not code is successful or finished.)

It can be a large subject. Complexity is under-reported, because everything bad is under-reported, when somebody looks bad and shoot-the-messenger is a viable strategy for PR management. Speaking ill of results after-the-fact is not a good personal strategy for anyone I know. Note before is okay: proving a plan cannot possibly work, before anyone wastes time on it, is highly regarded. Proof is whatever an audience takes as conclusive, but best is explanation making outcomes seem obvious.

There's a lot of anecdotal data on how software gets worse that cannot be told because, socially, it's telling tales out of school and therefore a good way to lose. So I have no stories to offer unless cast in hypothetical terms. For example, being able to deal with mess pays really well, especially if you make some of it go away while addressing assigned tasks.

Refactoring is hard to approve without an objective that requires it. In a commercial situation, there's no cleaning up code unless that's how you get from A to B, and B is now the goal. Irv: so can you do it? Wil: no, the code is really crazy. Irv: what if you fix it first? Wil: (clears throat), maybe, but ... let me think ... is it worth doubling the schedule? Irv: yes. Wil: (rubs face and looks miserable).

There's more than one programming language angle to it. A language harder to make messes in would be better. But nothing prevents mess per se, as long as you have goals to hit with incentives to cut corners, and you pile on layer after layer of system interaction with tiny semantic mis-alignments. You can also take an existing language and write new tools to look at things from a higher level, imposing a more stringent view than a programmer is required to apply in legal code. Using meta-language, you can write redundant specs for things you want to be true, then automate finding whatever disagrees.

Not to mention physical systems. Buildings, for example, are built "well enough," last a certain unpredictable amount of time, and are then completely demolished and replaced. People tend to maintain them just well enough to be serviceable, particularly when everybody has the sense that they're nearing the end of their life.

My parents, for instance, live in a house that they suspect will be torn down and replaced when they leave, and so there are lots of relatively basic repairs that they're quite rightly unwilling to pay for. Of course, this makes the teardown a self-fulfilling prophecy, but so what?

And lots of builders and contractors, at the end of a job, might look at the project and say, well, that piece of trim isn't quite straight or, well, I think it would've worked out better to put the duct and soffit on the other side of the hallway, or whatever, but nobody's ever going to actually go back and fix all that stuff. Or maybe they'll do some of it, eventually, when they happen to be fixing something else anyway, but who knows when? (When they finally want to sell the building, most likely.) And maybe you never got around to getting the screen for that window in the stairs, but you don't really want to open it very often anyway. And you skipped the framing inspection because you think requiring steel studs is stupid and you know the county's not seriously gonna make you tear down all the drywall to see if you really did it right.

Things are just like that. Why should software be different? I think that our engineering envy makes us tend to really wildly overestimate the percentage of construction projects that are managed like skyscrapers. Those are more visible and maybe more enviable and they're certainly more exciting, but honestly I think most building projects aren't like that at all. You do a little bit of drawing, a little bit of engineering, try to make sure you get the footings poured in the right places, and then you just basically hack at it until it's pretty much done. Things are just like that...

Things are just like that...

Right. But... software presents a problem that isn't found in all fields: Cascade Failures.

A hypothetical (but not farfetched) example of a software-induced cascade failure would be if (a) the IT systems of more and more firms that handle payments get horked; (b) the resulting glut of identity theft is sufficiently large to disable most of the payments system across the entire consumer economy.

Each individual firm may be making the same kind of rational decision your parents make with respect to limits on how much they'll spend fixing a house as it nears end of life. Collectively, in software, those decisions can add up to disaster.

(The vulnerability to cascade failures isn't unique to software, of course. I just mean it isn't universal and it is worth noting here.)

I agree, although I would like to underscore your point that this really isn't unique to software. I hinted at it with my offhand comment about "skipping the framing inspection." When that kind of inattention to best practices and circumvention of regulatory oversight becomes widespread, cascade failures really are much more likely and, for example, urban areas become much more prone to widespread fire damage. And this is true even though rationally it makes sense for each individual builder to cut that corner.

But yeah, basically I agree that software infrastructure is particularly prone to this, and I agree that it's a serious issue. I guess good old-fashioned regulations, audits and building codes remain our best tools to address these problems, even though people sure do love to complain about them...

Anyway, I'd also like to say that I generally broadly agree with your points in this thread. A bit cynical maybe, but insightful as always.

These are good guidelines for homepage items. The forum is not that strict. But discussions should be tied to programming languages.