Home » forums » LtU Forum

Toward a better programming

I wrote a blog post for my talk from Strange Loop 2013 about the framework I've come up with for describing what's wrong with programming and how we might go about fixing it. Here's an excerpt:

The other day, I came to the conclusion that the act of writing software is actually antagonistic all on its own. Arcane languages, cryptic errors, mostly missing (or at best, scattered) documentation - it's like someone is deliberately trying to screw with you, sitting in some Truman Show-like control room pointing and laughing behind the scenes. At some level, it's masochistic, but we do it because it gives us an incredible opportunity to shape our world. With a definition for what programming is and a concrete understanding of what's wrong with it, we have a framework for capturing that opportunity and removing the rest. But in order to do so, we can't just bolt it on to what we have now. Another layer of abstraction won't be enough. Instead, we have to address these fundamental problems at the very foundation of our solution. No more teacups and no more compromises.

Read the full post here

By ibdknox at 2014-03-28 17:45 | LtU Forum | previous forum topic | next forum topic | other blogs | 9345 reads

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

It didn't deliver us from

It didn't deliver us from incredible frustration that is writing and debugging software.

You'll forgive me for saying so I hope, but I think you're working from an incorrect premise.

In a good environment, with the right language, I find that very few frustrations come from my language or my IDE - they come from people. Someone gives me shoddy requirements. One of my peers wrote code when they were drunk. My boss wants the world by Thursday.

And I feel that some of the symbolic programming concepts you touch on are just translating one frustration for another.

By RM Schellhas at Fri, 2014-03-28 20:03 | login or register to post comments

Patience?

In a good environment, with the right language, I find that very few frustrations come from my language or my IDE

It may just be you have a lot more patience than I do haha :) But I've seen these frustrations at every place I've ever worked. Ridiculous error messages, impossible to reproduce bugs, git screwed up a merge, a misspelled key ... I've never seen people swear at an inanimate object more than a computer.

Personally I find the number of hoops I have to jump through to get what I want very frustrating. I want tools that allow me to more directly express my intent so that I don't have to go through some crazy translation process that is going to be full of issues. If I'm doing something simple I should be able to get it nearly immediately and that's so far from the truth right now that I can't help but be disappointed.

By ibdknox at Fri, 2014-03-28 20:43 | login or register to post comments

word

Both of you are right. The subjective reaction each person has often leans either towards seeing either the people, or the tools as the bigger of the two evils. But some see the truth that anything can be screwed up, anywhere along the stack of life. So then you gotta figure out your priorities. I'd hire both of you if I were hiring ;-)

By raould at Sat, 2014-03-29 15:31 | login or register to post comments

And boy are my arms tired.

I don't think there's a PL (programming language) solution to what makes programming hard these days, which is due more to wondering what in the heck other folks were thinking when they whipped up interfaces you have to use in libraries forced on you by requirements (established by the politics of project management).

I agree with RM Schellhas:

I find that very few frustrations come from my language or my IDE - they come from people.

When it's just you, programming isn't that hard. If all code is written by you, not much time is lost researching constraints in each code context. But given an operating system, that's never more than superficially true, since even if all code in a process is written entirely by one person, it must still make system calls. Usually those are documented, sort of. (Go read epoll docs again, and tell me if every little edge condition is really described.) In practice, everyone uses libraries written by others.

If you work at a company employing hundreds of developers, revising products over ten or twenty years, you get to work on components written by several people now lost in the mists of time, who leave behind helpful comments like /* ### What does this do? */, so you know it's not their fault it makes no sense now. (In this case, when the satype field is constant 1 in a pfkey message for IPsec controls, it turns out RFC 2367 Appendix A reserves constant one to enable promiscuous message reception. What, you didn't read Appendix A of that RFC?)

Anyway, software is often written to follow RFCs from the 90's, updated with subsequent drafts that never hit a final form, based on prototypes written by researchers in Japan (and thus sparsely commented in English), then revised by several waves of local devs who added three new processes to manage security policy databases in daemons, whose only documentation is the code, in C++ and C, which send binary messages in structs whose fields are in host platform endianess — except those that obviously must remain network byte order like spi and socket port. (Are those sockaddr's appended to that message aligned to 64-bit sizes or not? A header comment says no, but this sample code writing a message says yes. Maybe it won't matter if all the sockaddr definitions are already aligned to 64-bit sizes.)

As a handicap, figuring out existing old code slows you down by a factor of ten to fifty. If code is really beautiful, maybe it's not that bad. But nice looking code is rare. Usually, when I look at code, I find bugs that disagree with each possible interpretation I consider, which drags out a weighing and measuring phase, before fixing the old while adding the new. A complete absence of debug and test infrastructure is common. Design documents are usually circles for things that must exist, connected by lines denoting something that must happen. Prose is sparse, such as "A talks to B." (About what? And when?)

In Jean-Paul Sartre's 1944 No Exit, which is about software development (yes I'm kidding), he explains "Hell is other people." This makes perfect sense in software when you must sleep in the bed made by others.

If you read some of the material above in the correct tone of voice, some parts might be funny. A dozen years ago I used to complain about the state of software to a peer (the guy handling socket api for the reverse proxy server we were writing). And he used to laugh and laugh, like I was doing George Carlin performing a routine named Why Does Software Suck So Much? Since at first I thought I was actually complaining, the laughter puzzled me a bit. But then I started seeing why it was so amusing.

By Rys McCusker at Sat, 2014-03-29 02:46 | login or register to post comments

It's plenty hard enough for me

When it's just you, programming isn't that hard.

It's plenty hard enough for me. Here is an extract from a talk I propsed recently:

How to make my laptop beep whenever I get a bug report email, in 322 easy steps. Based on a true story.

Step 1. Make a new folder
Step 2. Create a text file
Step 3. Open my favourite editor
Step 4. Look at the blank screen for a while
Step 5. Google how to connect to email
Step 6. Copy example
Step 7. Look up gmail IMAP details
Step 8. Edit example
Step 9. Run the program
Step 10. Error!
Step 11. Google the error
Step 12. Open command line and install ssl headers
Step 13. Run program
Step 14. Error! What went wrong this time? Dunno, it threw away all the useful information and exited the program.
Step 15. I...what?
Step 16. Ok, attach a debugger
Step 17. Run the program again
Step 18. Aha, I made a typo
Step ... wait, I typed in gibberish code and not only did it let me, it then ran the gibberish program, reported a useless error, threw away all the useful information and exited.
Step 19. I...
Step 20. Fix the typo
Step 21. Run the program.
Step 22. Run the program again. This time print the email data-structure to see what we can do with it.
Step 23.
Step 24. Google how to get the subject of an email
Step 25. Google how regular expressions work in this language
Step 26. Google how to make sounds
Step 27. Google how to install jack on ubuntu
...several hours later...
Step 317. Success! It works!
Step 318. ...most of the time. Every few hours it silently ignores an email. Cannot reproduce with the debugger attached.
Step 319. Add lots of print statements.
Step 320. Try to reconstruct the execution of a concurrent program by reading a list of println statements.
Step 321. Stare at the wall for a while. Cry inside.
Step 322. Give up and just refresh gmail every five minutes.

There are plenty of useful, simple automations that *should* be easy for an end-user to whip up. With our current tools the activation energy is so high that even as a professional programmer I tend to just give up.

By jamii at Sat, 2014-03-29 06:05 | login or register to post comments

Using someone else's server is not "just you" as I intended.

(Wait, IMAP is an example of something that's supposed to be simple?)

You're using an IMAP service provided by Google, reading docs they wrote, and possibly other docs about IMAP in general. So that's exactly my point: using software written by others is hard, especially when semantics are complex and associated docs are (I'm guessing) less than stellar.

By Rys McCusker at Sat, 2014-03-29 07:12 | login or register to post comments

Would reading my email be

Would reading my email be easier if I started by writing my own operating system in assembly? Of course everything we do relies on software written by others, and that software is imperfect. If communicating with other programmers is the problem then we should look at the languages and tools we use to communicate and colloborate.

But before we even get that far - most of the pain in my example was down to difficulty in observing state, unhelpful error handling (especially when the runtime discards context) and confusing concurrency constructs. Even if I somehow managed to be programming entirely alone those would still be severe problems.

By jamii at Sat, 2014-03-29 08:58 | login or register to post comments

i feel your pain

Do I cry on your shoulder, or do you cry on mine? Is it some kind of recursive looping cring upon shoulders?

I 110% have the same mental journal diary script in my head every day. And then when my spouse asks me to teach programming I just want to give up the ghost and also prevent them from going into that not so good night.

BUT we've got to figure out: what is the sloppyness caused by at the root? Multi factored in reality. Economically not worth fixing by those with big resources. Still if we could elucidate the causes we could have do gooders try to make solutions. Eg if the docs suck and thats a problem, how do we stop needing docs? as a random example of the weird thinking one must maybe do. Everything sucks, we gotta enumerate and brainstorm?

By raould at Sat, 2014-03-29 16:06 | login or register to post comments

Red tape

The dominant problem with programming, these days, is red tape. The red tape is there because (well, partly because) the main problems used to be lack of safeguards and lack of clear organization, and we gradually introduced the red tape to try to fix those things; not that we necessariy succeeded, but the red tape has gotten prominent enough to distract us from the other stuff.

I'm concerned that the approach taken here may be, in effect, looking for a new kind of red tape. Getting rid of the red tape is a tricky proposition, because of the things that motivated the red tape in the first place.

God had a deadline, so he wrote it all in Lisp. — Bob kanefsky

By John Shutt at Sat, 2014-03-29 12:56 | login or register to post comments

According to xkcd, the

According to xkcd, the universe was obstensively done in Lisp, but was actually hacked up in Perl.

By Sean McDirmid at Sat, 2014-03-29 13:08 | login or register to post comments

these are your f

When I type into google's search box "these are your f", the first completion it suggests is "these are your father's parentheses".

By John Shutt at Sat, 2014-03-29 13:35 | login or register to post comments

I've never seen a clearer

I've never seen a clearer demonstration of Google's ability to track the individual it's interacting with than that!

By Jason Grossman at Sun, 2014-03-30 09:02 | login or register to post comments

For perspective, at the time

For perspective, at the time there were no cookies in my browser that Google could have accessed; so as best I can figure they only had my IP address to work with. (On the Internet, nobody knows your'e a dog except Google and the NSA.)

By John Shutt at Sun, 2014-03-30 11:27 | login or register to post comments

XKCD is just really popular

on a school computer "these are your f" suggests "these are your father's parentheses

By Anders Horn at Sun, 2014-03-30 18:52 | login or register to post comments

how about #99ff0000 tape?

I want it to be easy to assert anything about anything. I dont want things locked down so much as observed and warned. Like the Lean Startup thing where you can push anything live any time and if it is crap it will get diasable automatically in a few minutes. I want to have tools that can answer any crazy question I can come up with. Throw a bunch of them into the Trust But Verify pipeline ecosystem. Combine that with maybe some tazering when people ignore the warnings. Yes theres lots of Ui Ux nuanes here, like how they gotta make fighter jet warning systems usable useful when all things are exploding pear shaped simultaneously, but I wish we were working on that path, rather than on the dumb bog standard hard coded opaque red tape approaches we have mostly.

By raould at Sat, 2014-03-29 15:43 | login or register to post comments

I think there are two things

I think there are two things worth distinguishing. One is that simple things are often hard in rich general purpose languages. This is a well known issue, of course, with various standard explanations: reliance on libraries, which means that you need to learn diverse and often inconsistent libraries; lack of domain specific abstractions; tension between ease of writing code and readability (e.g., verbosity), etc.
The second issue is about programmability: are simple, quick and dirty tasks becoming more difficult? I think so. I propose two non-mutually exclusive reasons. First is that as the technological ecosystem becomes more complex there is an increase in the inherent complexity of many such automation tasks (e.g., retrieving email from the net is more complex, and has more failure scenarios, than reading a local mailbox file). Second, there is a trend away from programmability (think iPad versus unix). One particular issue is that powerful abstractions are perhaps not provided, and do not seem to be in the works (e.g., why shouldn't reading mail from the server look exactly like opening and reading a local file?)

By Ehud Lamm at Sat, 2014-03-29 15:42 | login or register to post comments

bipolar lisp programmer

If we were all using the same language, maybe we'd have the networking email handling library by now for all to use. Or three or four. Humans can't agree on squat let alone programmers.

Ok so then in this particular email case I guess I want an abstract state machine definition in a standard spec system, so I can either hand or better auto generate the email engine, and then compose in my Ui or whatever. :-)

I mean, what would be some kind of abstraction that maybe we could agree on? I know lots of people hate model driven stuff, but I suspect that today's bad situation there doesn't have to be a law of the universe.

We sure can't agree on programming language.

I want a modeling system that allows us to specify what can be concurrent, parallel, and then systems can choose to reify that however they want, can. In the end it will become of course its own general purpose programming language! Thereby reducing it an earlier nihilist joke.

By raould at Sat, 2014-03-29 15:58 | login or register to post comments

Bipolar minds and binary satisfaction

I mean, what would be some kind of abstraction that maybe we could agree on?

Cannot answer that question in general but in the area I've been working for long, telecommunication and smart cards, the answer might be binary message formats. It is something I've never heard anyone complaining about. So binary messages are the ontological domain which provide consensus whereas everything else seems to be optional and eternally disputed.

The desire for insight porn seems to be actually satisfied by binaries and those who deal with them seem to have little longing for the lost object. Of course they often like to see the binaries as something else but this can be easily accomplished and doesn't change their minds about the subject.

So if the low-level programmers are the happy ones, then maybe there is something we have missed in the stormy uptake of high level languages, programming environments and systems which exceed by far the capabilities of its engineers-becoming-plumbers.

By Kay Schluehr at Sun, 2014-03-30 09:28 | login or register to post comments

Assemblyis nice, but where

Assemblyis nice, but where you really want to be is microcode.

By Ehud Lamm at Sun, 2014-03-30 10:10 | login or register to post comments

A microcode view is usually

A microcode view is usually not available in emulators / simulators and it is not missed. So it doesn't really count as a lost object which causes "plumbing" instead of "problem solving", all sorts of incidental and accidental complexity and the prevention of insight porn. The gap between the programmer and the electrical engineer falls on the side of the engineer who maintains the invisibility of the gap. This is our innocence and we are finally out of work on Level 1.

By Kay Schluehr at Sun, 2014-03-30 11:02 | login or register to post comments

I suddenly realized that plumbing is needed to solve problems caused by leacky abstractions.

By Ehud Lamm at Sun, 2014-03-30 11:18 | login or register to post comments

Ruined by plumbers

I hope you your home is alright. A colleague of mine regularly complains about the craftsmen who are currently tinkering at his new house. They would be out of schedule and budget and steadily discover new leaks. He once weirdly laughed when someone mentioned "planning" in the same sentence with craftsmen. He is one of those few consistently careful and reliable programmers who doesn't deserve such a bad fate.

By Kay Schluehr at Sun, 2014-03-30 12:16 | login or register to post comments

That was kind of my point...

That was kind of my point...

By Ehud Lamm at Sun, 2014-03-30 12:28 | login or register to post comments

free epiphaies on ltu

and here i've never connected so directly the fact that i am always seeking insight porn for programming, and that any time i do any plumbing at home i am just utterly infurated by how screwed up it all is. wow.

By raould at Mon, 2014-03-31 17:24 | login or register to post comments

i'm sorry zaphod i don't believe a word of it

or at least i think the reason that is the way it is is because somebody with more power and money than *you* or anybody else has *dictated* what the binary format *shall be* if you want anything to work at all. "we're cisco! you can eff off if you don't like it!" or some such. some things somehow have enough weight or dictatorial control that they seem to work :-) but in reality if those external heavyweight forces weren't involved, you'd be in the hell that most web programmers are in of spending time fighting over xml vs. binary xml vs. yaml (oh god why doesn't anybody respect asn.1?) or markdown vs. html etc. etc.

By raould at Mon, 2014-03-31 17:27 | login or register to post comments

The victorious, bloodthirsty

The victorious, bloodthirsty dictators and villains who claimed land for the bourgeoisie become culture heroes when the concrete memories of their atrocities is fading. It doesn't mean everything is being preserved and likewise respected. Who uses TTCN-3 or CHILL? The heavyweight forces didn't always succeed and there are snides about committee designed languages or aristocracies in general everywhere but they have also designed everything else which goes undisputed.

In the past year I've created an elusive data structure on top of simple BER encoded TLVs. BER has an ASN.1 ancestry but hardly anyone cares about the universal class bits in the tag except for the long-form tag encoding. So there is basically no overdesign left. The data structure I mentioned can seen as a crossing between a formal grammar, a parse tree, a spread sheet and a C-struct. It is extensible, can be replicated with modifications, it can parse hex strings and can be unparsed into it. Although it sounds like being a monstrous beast from a mesozoic swamp of the computing age, its public interface isn't much more complicated than that of a list or a hash table. I started its design in the context of a domain specific language with full syntactical freedom. Later I gave a Python implementation which is the one we currently have in use. I could imagine to improve it using experiences from other implementations in particular those of complementary languages like C and Haskell. I haven't yet found a catchy name and gave little attention to its career path.

By Kay Schluehr at Tue, 2014-04-01 05:35 | login or register to post comments

A search for foundations

This was a fun read, and I especially liked the way that you ended it. I also think that everybody will be "programmers" in the future (or most likely that the job description "programmer" will go away), and I agree with you that this won't happen with everybody jumping on C++ or Javascript or ...

By presenting the effort to construct this future as a search for foundations, I think you've got identified the right direction and maybe the right area where this work is already being done.

Do you have any thoughts on homotopy type theory as a common foundation? I think that characterizing type theories as "constraints on data" is not quite fair. TT has a much more ambitious program going than that.

By Kalani at Mon, 2014-03-31 17:11 | login or register to post comments

I don't think it will ever be easy.

Programming is the act of specification in a formal language. An informal language is always open to ambiguous interpretation, so is no good for specifying things. The problem is not the computer, it is the human. Consider something humans consider simple like making a nice hot cup of tea. How would you describe this to someone? You might say, put the tea-bag in the cup and pour in the hot water. Even with a human this is no guarantee of a good cup of tea. You might add some details, like use the yellow mug, and to stop filling with water when its almost to the top of the mug. Good tea? Nope, the water must be piping hot, so make sure the kettle is boiling before you pour. etc...

Further this requires understanding of a lot of concepts like what a kettle is, what water is, etc. To produce something that understands linguistic commands like a human requires solving hard-AI (producing something that thinks like a human).

Programming is also managing complexity. As such each abstraction layer needs to be kept as simple as possible, otherwise the complexity quickly balloons out of control. In many cases this actually works against code reuse, and a simple minimal implementation just for that one application results in less complexity than reusing a generic component.

Programming language design is an act of compromise, as one must choose between mutually exclusive features to include in the language, it turns into an engineering task of matching the choices agains the language goals and choosing accordingly. Almost every decision will make some problems more complex to solve at the same time as making other s easier. For example 'no gotos' makes formal reasoning about code easier, but makes it harder to escape from a deeply nested loop, or function.

In summary, complex problems need complex specifications, and often what appear to be simple problems are not as simple as they first appear. This extends to programming language design, which is by definition a series of compromises (teacups optional).

By Keean Schupke at Mon, 2014-03-31 18:49 | login or register to post comments

but but but

one problem is that there's so much accidental incidental gratuitous stupid dumb extra complexity that we manage to heap on top of the essential stuff. :-)

By raould at Mon, 2014-03-31 21:58 | login or register to post comments

See above

This is what I was referring to in the paragraph about managing complexity, and the necessity of Keeping each layer of abstraction as simple as possible. Thats just going to make it less hard, not easy.

I forget where I heard this, but debugging is an order of magnitude harder than writing code. So you had better keep your code simple, because if you write the cleverest code you can, you will be too stupid to debug it.

By Keean Schupke at Mon, 2014-03-31 22:24 | login or register to post comments

Abstraction, Modularity, Composition

I had some thoughts in reaction to this that I decided to put up as a blog post instead.

In summary, I think these Big Three Principles have simply not fully made it into the design of our programming languages (or libraries or other tools) yet, and they're so important to writing programs that I'm not sure "what's wrong with programming" has to go any deeper than the need for all our tools to actually support them.

Which is far easier said than done, of course.

By Ted Kaminski at Mon, 2014-03-31 19:29 | login or register to post comments

Useful Coupling of Concerns

We can create a PL where:

  • every component is a function
  • the unit of modularity is a function
  • a function is the unit of abstraction

Of course, we could substitute 'function' with lens, procedure, process, pipeline or similar, so long as it supports composition (in the algebraic sense).

The main benefit of coupling these units is that doing so eliminates nearly all accidental entanglements between subprograms, and greatly simplifies sharing or distribution of code.

By dmbarbour at Mon, 2014-03-31 20:12 | login or register to post comments

Indeed

Yes, this is one of the things I was looking at for my language project. By allowing functions to have associated types and principal typings (simplifying things a bit) you get composable functions that are also modules. As modules can be ADTs, then functions can be the abstraction as well. I'm still a long way from anything usable though, as I am still working on the type system.

By Keean Schupke at Mon, 2014-03-31 22:37 | login or register to post comments