Lambda the Ultimate

In 1960Mumble I was using Algol 60 and Lisp 1.5 now ("I'm retired and can use whatever I want) and I use Scheme and Ada.

Is this almost the same or very different??

Web

For most users voice has less reliable bandwidth in total than the fingers, FWIW. There're experiments with mouse-based systems based on a predictive text-style scheme which I'm told can go pretty fast when you're practiced - I imagine a variation using a thumbpad or the like'd work too.

It would be easy for me to blame everything on Sun and Java, but I know that most businesses aren't interested in developing software for the sake of developing software.

I'd be interested in specific business cases that you used in your argument to your colleagues for using something like Eiffel or Haskell.

Academics need to do a better job of presenting compelling business cases for the switch to "better languages".

I think we're starting to see a thaw in Java-inspired "staganation" though - maybe not something "better" like Haskell or Eiffel, but to languages like Ruby and possibly (probably not though) to an old timer like Smalltalk (if Croquet gets some traction with its 1.0 release), and maybe to something like Arc, if Paul Graham pulls a rabbit out of the hat.

Academics need to do a better job of presenting compelling business cases for the switch to "better languages".

That sounds backwards to me. After all, it's the technology companies that claim to be good at making money out of existing technologies. If academics, who claim to be good at primarily discovering new knowledge, are also supposed to give business advice, then something's wrong somewhere.

That sounds backwards to me. After all, it's the technology companies that claim to be good at making money out of existing technologies. If academics, who claim to be good at primarily discovering new knowledge, are also supposed to give business advice, then something's wrong somewhere.

You have three general entities here. You have the vast majority of businesses for which software development is just a means to an end. Then you have the middle men like Sun, Microsoft, IBM...who take the research they've done and which academics have done and produce something sanitized from all of that for "the masses". And then you have the academics who are feeding the Microsofts and Suns.

So the problem that many people (around here at least) is that the general business community is just buying up whatever Sun and Microsoft package up for them (.NET/Java). So "the goal" would be to do an end-run around these middle men and bring primarily academic (or at least something different and more productive) to these general business organization.

But let's try a couple examples. One is Ruby on Rails, which is bringing Ruby into the mainstream now. The key was that RoR was a business case that is tangible. Just handwaving about how productive Ruby (and dynamic languages in general) was not enough.

Another example is Linux. It did an end run around and destroyed proprietary Unix (for the most part).

So the goal for something like Haskell or whatever academic language you want to throw out, is for the academics or non-academics that use it to come up with tangible business cases for its use. The middle men (Sun and Microsoft) want you to buy their packaged up solutions.

Of course there are shades of gray in all of this, but if some academic is living in a bubble and doesn't have any business sense or doesn't make an effort to engage the business community or even the more business-oriented MIS department then it'll just have to wait unti someone else packages it up.

Does Haskell become another Lisp?

Does Haskell become another Lisp?

I wonder about this comparison. At least Lisp was somewhat successfull in the business world in the 80s with AI- and ( expensive ) Lisp-machines. The Common Lisp standard was driven by company requirements for a large part.

I don't see yet a large market segment for Haskell or a Haskell successor. Maybe security critical applications where rigour and proof techniques are prescribed by customers. Since we are doing unbiased prognostics here I would expect that the small domain of "specification languages" will be merged with pure functional languages. Allthough I do not expect that this drags the masses into Haskell business we might experience a surprise? But are there any indications? No reliable IDE's, no "Learning Haskell", "The Haskell Cookbook" or "Haskell in a Nutshell" in O'Reilly bookstores. No RoR new-age hipsters. Just a couple of CS-students and PhDs talking about type theory and monads. A good indication would also be a "Haskell practitioners conference" where the category theory guys would have to stay at home.

My favorite revision control system, at least for personal to medium-sized projects, is Darcs, which is written in Haskell. And then there is, of course, Pugs. The latter page has a link to a Learning Haskell, a 254-slide presentation by the inimitable Audrey Tang, complete with comparisons between Perl and Haskell.

I don't think Haskell's adoption curve outside of academia will be anything like that of any other language we've ever seen, but I think the above examples are signs that it has a broader role to play, that it's not playing yet.

admirable and instructive set of slides about Haskell in Mozilla's XUL format! :)

gotta love that lady...

This may paint me as a bit out of touch, but... how in the hell am I supposed to view that slide presentation?

Any recent enough mozilla-based browser should be able to handle Audrey's presentation. It looks great in Firefox.

So I have to download yet another browser to view these slides? Whatever. Bandwidth and disk space are more or less cheap these days.

* trundles off to get firefox *

OK, so I tried to open the file I had downloaded earlier in firefox and was greeted with a whole lot of nothing... some navigation looking bar at the top of the window and no slides. I guess that not only do I have to download a new browser to go through the slides, but I also have to be online to view them? No thanks.

I know I've gone on here and elsewhere about publishing for the web vs. publishing for print (and this was touched on a bit in the discussion about The Problem With Parsing), but this seems to be an even more egregious sin. It's publishing for a very small subset of the web -- those with mozilla based browsers. A horrible publishing decision - and that's without even touching on the clear readability issues with these slides - at least if one wants this information to be widely read.

at least, you are now able to view the web with a more up-to-date rendering engine rather than through some 2001 antique. :)

why don't you try to simply click on the link rather than opening the file you downloaded? i don't know why it didn't work, but following the link should be fine.

yes, she could just go with some clever html+css presentation alright, but i wouldn't expect that from a perl/haskell fan... :)

why don't you try to simply click on the link rather than opening the file you downloaded?

Two reasons: When I clicked on the link the first time, it downloaded the xul file. Why should I bother to revisit the link when I supposedly have a local copy of the content? Also, it's occasionally nice to view a document when you are, say, on an airplane, and the 35,000 foot cruising altitude puts your wireless card a wee bit out of range of the hotspots on the ground.

yes, she could just go with some clever html+css presentation

Oh, I was thinking something more along the lines of a pdf -- far more portable, actually printable (unlike these slides), and available for offline reading on a number of devices with a number of different readers. And preferably it would be a pdf generated with readability in mind, and not the unfortunate typesetting stew that made up those slides.

And since I'm already dangerously off-topic, I'm using Safari as a browser, and its rendering engine is as "up-to-date" as gecko is.

Ok, man, i downloaded the xul file just to see what was going on. It simply turns out you just download the xul when you click on the link on a xul-unaware browser, but then, the xul only works with the associated javascript file:

http://www.pugscode.org/osdc/takahashi.js

It was just a matter of some source-code inspection. Download it too and save to the same directory as the previously saved xul. You'll also like to have the associated css file as well:

http://www.pugscode.org/osdc/euroscon.css

though it's by no means necessary...

She could have just packaged the javascript and css in the very xul file.

"Why should I bother to revisit the link when I supposedly have a local copy of the content?"

because it's so easy to visit a link?

"Oh, I was thinking something more along the lines of a pdf -- far more portable, actually printable (unlike these slides), and available for offline reading on a number of devices with a number of different readers."

I'm not sure a pdf is as portable as html. And, with the right css, you can have pretty good printing (page breaks as well) and nice fonts. But, then again, she choose xul, not html+css... xul is for GUI building AFAIK...

"And since I'm already dangerously off-topic, I'm using Safari as a browser, and its rendering engine is as 'up-to-date' as gecko is."

great then!

As we have gotten so far afield, this will be my last post on the subject, unless it actually gets steered back to the realm of programming languages...

It was just a matter of some source-code inspection.

And you really think that this is a reasonable way to distribute a document? It

• Requires the reader to use a specific piece of software that probably isn't on their system, while an alternative program is. So you're requiring the user to install redundant software.
• As it is, the slides are not readable offline. Unless, of course, you are willing and knowledgeable enough to do "some source-code inspection."
• The slides are not printable.
• The slides have no coherent sense of typesetting, and would actually require a conscious effort to make them less readable.

About the only thing this distribution method has going for it is the relatively small size. However, we could do something similar by distributing a *TeX file, which would require about as much work to view by the end user as these slides. Of course, there are obvious reasons why people stopped distributing their papers and slides as tex files, many of which are shared by this XUL format.

because it's so easy to visit a link?

Granted, it's not hard to visit a link. However, launching firefox, typing in lambda-the-ultimate.org, waiting for the page to load, clicking on the discussions link, clicking on the link for this thread, searching for Audrey, then clicking on the link... well it is more involved than double-clicking the icon in my inbox directory.

I'm not sure a pdf is as portable as html.

Well, readers for both formats are nearly ubiquitous. However, PDFs have the advantage of being a single, self contained file, perfect for offline reading (and easy annotation with modern readers). In order to work with an HTML file offline, one needs to download all of the associated images, all of the pages that are linked to in the case of multipage documents, and so on -- and then there's the matter of things like the css and javascript files that need to come along as well...

And, with the right css, you can have pretty good printing (page breaks as well) and nice fonts.

I suppose. In theory. But I have yet to see a page with a print css fie that results in anything more than a passable result. Sure, it's great that we actually get things like page breaks and links printing in black, not light grey. But there's a lot more to good typography than that, and you just don't get more than that -- if you get anything at all...

Of course, I'd be willing to forgive the sins of the distribution if the content is good enough to overcome the problems of getting to it. And these slides just don't seem to be. They're not bad, per se, just not particularly instructive. They certainly aren't a very good method of "Learning Haskell" -- one would be far better off here with something like Two Dozen Short Lessons In Haskell (which, sadly, hasn't been updated in almost 9 years), or something else along the lines of Jason Hickey's Introduction to OCaml (PDF) (only for Haskell) which offer exposition as well as examples.

"And you really think that this is a reasonable way to distribute a document?"

I didn't agree with it being in xul, but didn't have a problem with that. I understand html+css would do a pretty good job, while being portable and readily available. Html documents are composable documents and can have other formats embedded, like css. So, if i was to distribute one such document, i would package everything altogether in a single file.

Beats me why she decided to restringe herself to xul, which is used to render GUIs on the fly via an xml description, like the user interface of Firefox itself.

"* As it is, the slides are not readable offline. Unless, of course, you are willing and knowledgeable enough to do 'some source-code inspection.'"

which is not that difficult, since it's just a plain text file with some hrefs...

"* The slides have no coherent sense of typesetting, and would actually require a conscious effort to make them less readable."

really? Perhaps i'm too used to a text-only command shell, cause I didn't feel it half as bad... but then again, Mac users tend too be a lot touchy about design...

"a *TeX file, which would require about as much work to view by the end user as these slides."

oh yeah, sure. Because viewing a TeX file is just as difficult as downloading Firefox and clicking on a link...

"Granted, it's not hard to visit a link. However, launching firefox, typing in lambda-the-ultimate.org, waiting for the page to load, clicking on the discussions link, clicking on the link for this thread, searching for Audrey, then clicking on the link..."

See that little integrated Google search box in the right-upper corner in Firefox? Type "learning haskell" + ENTER, then look the 7th link down... or "learning haskell audrey" one better...

"They're not bad, per se, just not particularly instructive. They certainly aren't a very good method..."

They are very instructive for programmer with a long imperative programming past, with no idea whatsoever of functional programming...

I have done exactly the kind of thing Audrey did with these slides - namely, experiment with a format which seemed promising, and use it in a presentation. They are slides, after all, i.e. they were presumably associated with a live presentation. I don't think there's any moral imperative to create slides in the most widely distributable possible format.

As for the content, I mentioned those slides partly because they were targeted at people already familiar with Perl, and I was pointing out not that those slides provided the perfect introduction to Haskell, but rather as an example of a non-academic context in which Haskell is being introduced to programmers. Whether you like XUL, or these particular slides, isn't really relevant. People like Audrey push the envelope in all sorts of ways, and that should be valued, not subjected to endless trivial criticism.

and maybe to something like Arc, if Paul Graham pulls a rabbit out of the hat

I'm surprised this is the first I heard of Arc. It's nice to see a change in focus, instead of blaming Lisp's failure on "Blub" and looking more at the language itself. Even so, Arc looks like a dead project, which would explain why I never heard of it :)

Paul Graham always says that Arc isn't dead. He says he's working on it, but just hasn't released anything yet (and might not release anything for several years). It's like a seed: alive inside, but dead-looking. And it's just sitting there frustratingly at the moment.

I believe a millionaire with a deep passion for Lisp and lots of time in his hands can eventually produce some significant contribution. Graham is the author of "Hackers and Painters" and i believe he thinks of himself as some kind of Michelangelo, working on the Sistine Chapel... a true work of art, not the kind of thing to be rushed.

His not competing with Microsoft or Sun for the hearts and minds of the young generation of client programmers... its just his take on Lisp.

He was asked, "When will Arc be finished?" at the Startup School in November. His response was, "Next question?", followed by much laughter from the audience. So I think it's fairly safe to assume that Arc is dead, at least for the forseeable future.

And more info, as of last night:

http://reddit.com/info?id=38vr.

So yes, basically dead, but he may start working on it again this summer.

Our company develops two kinds of products: a) business applications (web, desktop) in the cellphone billing domain, and b) realtime defense applications (radar consoles etc). The business department uses VB/Java, and our departnment uses C++/Java.

As computing devices shrink, I think some sort of projection-style keyboards could catch on, once a few more usability experiments are done. Since the patterns projected as fake key labels are all software, this also neatly solves the key labeling problem for non-ASCII characters, like those used in Fortress [aha! an LtU-related bit :-)]. Also see this related half-bakery discussion.

I'm a keyboard man myself, but you guys seem to be forgetting that in the near future, either people will be talking to computers or exercising some aerobics in virtual visual interfaces like the one Tom Cruise interacts to in Minority Report...

Programmers and analysts in the future? What for?, when you have more advanced intelligence available than our own... more likely, we'll serve as batteries...

I wonder what side-effect free programming will be like in the light of quantum computing...

I always figure that making computers simpler to use amounts to adding more complexity on the software construction side. And even if you're not in on the core development side, there's going to be lot's of DSL's to master for those that are intermediaries.

Well, rmalafias point seems to be that you, if you become a computer yourself, won't need particular stuff anymore for programming them; just like nobody is employed to connect the synapses of your brain.

This is a rather depressing perspective. Either mankind will make progress in the domain of autonomous synthetic bioware or we will continue programming in C also in 50 or 100 yrs.

but we have lots of people who program our minds (teachers, psychologists, managers?). they just do it in a fuzzy high-level language that noone understands realy well. But you could see this occupations merging with that of the programer. And perhaps the programing would be more on a direct neural interface.

This is an interesting topic with a very wide scope. I remember also Michel Foucault inisting on techniques of body discipline that where invented in 19-th century schools, prisons, factories and armys. Neural programming on that level may be simpler than the convenient SF themes like para-telepathic communication and re-wiring of the neural cortex. It didn't even have to be invasive.

While I consider the "we'll serve as batteries" part to be utterly silly (are pilots "just batteries" now that autopilots have been around for ages? Does a wrench render an engineer unnecessary? I doubt I'll live to see the 'more advanced intelligence' in form of a computer, unless of course people soon figure out that immortality thing which is just around the corner since centuries...), check out this if you're interested in innovative user interfaces, it's really cool: http://mrl.nyu.edu/~jhan/ftirtouch/

I don't know about you guys, but I'm ordering an Optimus Keyboard...

I know I'm taking something off-topic further afield, but hey, its a cool looking keyboard :).

Seriously, I think this is targetted at gamers, but its aesthetically pleasing and it does what you're describing. Its been a bit of vapourware, but they are intending to market it, so I'm hopeful.

Wait til' it's beyond the preorder stage. If that device ever sees the market, I'll eat my hat.

Is a doctored photo of an early 1960s-era US nuclear submarine operation console set. It is a total forgery; I can't believe anyone would think that the TV set and teletype of the era were actually present in that scene, especially given the differences in lighting. It's a really bad cut and paste job on photos and captions - I guess people are fooled because you can't find other pictures of that online. I trained on one of those consoles a decade ago (yeah, they don't throw much out) and can name everything on all of those panels and how to use them. It is in no way useful as an actual example.

Is a doctored photo of an early 1960s-era US nuclear submarine operation console set. It is a total forgery; I can't believe anyone would think that the TV set and teletype of the era were actually present in that scene, especially given the differences in lighting. It's a really bad cut and paste job on photos and captions - I guess people are fooled because you can't find other pictures of that online.

What a shame, I have been using that picture as my desktop wallpaper for the last couple of days. ;-)

Still, made me think. I wonder how long it will take before we can by a "personal computer on a chip" solution, i.e. some Intel chip with some gigabytes of ram, some gigabytes of flash, and wifi/bluetooth build in. How long before such a chip would drop into to the $20 range?

When it comes to buying stuff I am thinking about buying a solid state hard drive, or IDE/compact flash, in the coming months for my laptop. (Bit too costly though, and I am also not really in the mood of configuring linux to run on solid state).

I trained on one of those consoles a decade ago (yeah, they don't throw much out) and can name everything on all of those panels and how to use them.

Say what! Now we also have people here who know how to handle nuclear submarines? Only on LtU.... ;-)

Or another term "ubiquitous computing" (and just to toss it out, "amorphous computing"). See Paintable Computing.

Responding to a fairly particular part of the above reply: what we'll see more of is a lot of very cheap simple computers that are everywhere (the resulting computing capacity would be enormous and even enormous per dollar). Already today computers are all over the place and in many places people don't expect. In the future it will be this way to an even more ridiculous degree. (Computers in the fabric of your clothing, in your bloodstream, in the paint on your walls and cars, in the structural materials of your buildings and vehicles, sprinkled about on your lawn and in parks, ...) This is the pre-nanotechnology technology.

To talk about the interface side (of the general discussion): no one seems to have mentioned one of the obvious interfaces (albeit this may require more technological advancement than one can expect in the given timeframe); namely, thought. Certainly one can think faster than one can type, and such an interface is private and intuitive. There is some science (I don't have any handy links though) that suggests that we may be able to do a fairly crude form of this in the not too distant future. We won't (unfortunately) be able to just "think" what we want, but we may be able to associate certain thoughts or actions with certain inputs. It wouldn't work this way (as this wouldn't be dense enough), but like associating the character 'a' with the thought of an apple.

As a completely different aside, eyetracking technology and a system like Dasher (which I originally heard about from shapr), might be a viable interface with a simple HUD glasses combination*. Providing reasonable speed (better than many people can type) and privacy, while being more portable. This however isn't really futuristic technology. Adoption of this would be more a cultural thing.

* Incidentally, this should be very highly effective for programming languages as it would be very easy to accurately predict the continuation of some chunk of code (syntax wise).

If you had a touch screen, you could easily program it to display a visual of a keyboard that you could type on. You wouldn't get any tactile feedback, but you also wouldn't have any problem reprogramming it for different layouts and character sets, like the projection keyboard above.

Personally, I want an IPod-sized nanocomputer with a roll-out touch-sensitive E-ink display. The technology for that should exist within the next couple years; I've seen demos on the web for all the individual technologies, so somebody just needs to put them together. Any enterprising hardware engineers in the crowd?

I don’t know about the automobile industry, but other industries are already regulated by government requirements. A friend of mine develops software for navigation systems in airplanes. According to him, there are all sorts of government guidelines regulating the software they produce. Every time a piece of code changes (regardless of the type or size of change), a lot of documentation must be written (along with the obvious stringent testing), just to satisfy government regulations.

I don’t know about the automobile industry, but the "liability-limiting software license” only exists for shrink-wrapped consumer software (a la Microsoft Word). Companies buying specialized software just wouldn't go for that. Ford is still liable for the behavior of the automatic transmissions, so they aren't going to buy a product from a company that doesn't indemnify their software. A large financial firm has a lot to lose if the custom software regulating financial transactions is hacked. Big companies care about buying software that is indemnified, or else they are legally up a creek when something breaks.

Consumer level products don’t need the same sort of stringent requirements that these “life critical” systems require. Dynamic languages will find a place in the less critical markets of consumer and business software (the sort of software that doesn’t control critical systems in planes, trains, or automobiles, or financial transactions). If the software breaks, it may be annoying, but the risk of breakage will generally be worth the benefits gained by using a more dynamic language and agile development methods.

What would a world where all software is liable for failures look like? Software on the whole is much more robust; however, the price of robustness is not small. Small software firms, because they can no longer afford to stay in business, either go out of business or, if they are lucky, are bought out by larger firms. The legal fees along with the increased cost of a longer more thorough development process drive the price of consumer software way up. As a result, the number of pirated copies of software increases, while legal sales of software decrease, placing further burdens on software corporations who are now unable to pay developers the wages they use to earn. Soon, all software developers are required to get expensive liability insurance to protect them in case they ever make a mistake and are sued, just like a doctor (and despite all efforts to the contrary, mistakes are still made; they’re just rarer than they use to be). The number of software developers decreases dramatically, due to the decreased net pay and increased stress. Open-source and free software quickly dies out due to the wave of lawsuits that break out over damage claimed to be caused by said software, regardless of the actual fault of said damage.

No, such rash decisions are just as bad, if not worse, than the original problem. (This brings up another point; all decisions made in the name of the public good should be eyed at very suspiciously, because it usually means some group of people will get screwed, the dictatorship of the majority, and all that. There’s a win-win decision somewhere, we just need to find it.) Should developers of software of life-critical systems be liable for errors in their software? Yes, just the same as a civil engineer is liable for errors in a bridge design. However, that liability need not, and should not, extend to the developer of a non critical application, just like an electrical engineer is not liable for an electronic toy or game breaking.

Those safety regulations do exist for security critical software as well. Some companies already live from those obligations. A nice portrait of the britain company "Praxis" is freely available from EEE Spectrum.

I wasn't recommending that state of affairs, just offering a somewhat whimsical prediction.

Usually in discussions like these, people focus on language ideas coming out of academia (or currently popular with early adopters), rather that thinking about exogenous financial forces that historically have more impact on the mainstream programming language landscape. Just trying to inject something into this discussion beyond navel-gazing, really.

Usually in discussions like these, people focus on language ideas coming out of academia (or currently popular with early adopters), rather that thinking about exogenous financial forces that historically have more impact on the mainstream programming language landscape.

Yes, I think this is true. For us language people, looking at economic forces helps us understand why things are the way they are, when there are languages out there that are clearly better than the mainstream.

Since failing to follow "industry-standard best practices" could easily end you up in front of a unfriendly judge, dynamically typed languages quickly fell from favor, while design-by-contract, effects and aliasing annotations, and similar safety-oriented practices became much more common. Sales of static and dynamic code analysis tools soared, as development houses furiously attempted to hold down the costs of insurance-mandated code reviews.

An interesting prediction, but IMHO I rather see verification instead of testing as the future. In fact, verification tools are already widely applied in life critical systems. See for example some recent applications of the SPIN model checker.

I somehow have the feeling that a big part of the programming community has no idea about the state of verification today, and therefore keep pushing languages like C which are harder to verify than e.g. side-effects free languages.

Furthermore, language usage often seems to be an emotional affair instead of a matter of rational analysis. Just have a look at the flamewars people start on Slashdot if you should make the mistake of mentioning Java somewhere (though I do not want to imply here that Slashdot represents a significant amount of professional/academic programmers).

When I was at IBM Watson one summer back in '84 or '85, Tony Hoare (Sir Tony?) visited to talk about developing correct code from specs. In responding criticisms that these techniques were too hard for programmers to learn, he related how he had come across something in the Oxford archives to the effect that (historically) division was considered too difficult for undergraduates. Of course now (and even back in the mid-80's :) ) we can teach the long division algorithm to kids in 3rd grade, basically due to the highly structured combination of algorithm and standard layout used.

If anyone knows of a written version of this anecdote, I'd be pleased to hear about it (I couldn't get email through to Sir Tony himself: maybe Oxford filters stuff from the Colonies).

(And I'll pick up my library's copy of Capiatalism & Arithmetic in case it appears there.)

Also relevant is the science fiction story by Isaac Asimov "The Feeling of Power", available free on-line at http://www.themathlab.com/writings/short%20stories/feeling.htm.

(Sir) Tony Hoare has retired from Oxford, and is now with Microsoft Research's Cambridge Lab. You might try contacting him through there.

Yes, I knew about the move: I looked all over their website (both then and just now), and never found a way to contact him directly. It has the feel of a deliberate attempt to shield them from unsolicited contacts, which makes sense. In the end I tried a couple of variations based on Simon PJ's address, as well as one of the old Oxford ones from a paper, but never heard back. (That doesn't mean the mail didn't go through, which is one reason I suppose I haven't tried harder.)

There are already commercial products that will search for duplicate code, and help you abstract it away with automated refactorings. They are currently fairly clunky, but still eminently servicable for production work. What you are describing is basically raising the game a bit for these tools.

I guess I want more than just raising the game of clunky tools. I want to not even think about needing those tools.

At the moment, my IDE (IntelliJ) can discover duplication and refactor it away into shared methods. Great. But I still have to decide to remove duplication.

Would it be possible for the tool to detect duplication, manage copy-and-paste operations, keep track of differences as I edit pasted code?

What would the effect be on design? Removing duplication is a design decision. It would be hard to make a tool that worked in a copy-and-paste-and-edit-a-bit style, removed duplication to produce well designed code under the hood and then somehow let the programmer learn the new design to get a better understanding of the domain they are working in.

50, 100 years might be enough time to invent something like that...

Right now, IntelliJ will let you search for duplicated code. It's fairly bright about it, letting you set parameters for how small are large of duplicate chunks you are looking for, and what differences between reported duplicates you wish to ignore, so that you can find "duplicates modulo abstraction". It does this by creating a database of semantic hashes, and only doing structural comparisons when the hashes match. Right now, it only checks for duplicate expressions or self-contained statement sequences (which can be abstracted as methods), but it would be fairly straightforward to make it work for finding duplicate methods or even entire classes.

Unfortunately, there's a usability issue. IntelliJ only exposes all of this as a batch process. You request a duplicate report, and it gives you one. What if instead, it maintained the semantic hash database in sync with the project's code base, searched for duplicates against this database in real time as you were editing, highlighted potential duplicates in your edit window, and offered you appropriate "quickfixes" to abstract them out. IntelliJ is already interactively highlighting several hundred other coding errors or design weaknesses and offering automated fixes. You could think of this as just one more.

So my question is: Would that fulfill the use case you're envisioning? I don't really see the value of maintaining information about cut-and-paste "lineage", as opposed to finding abstractable duplications. I don't much care where my design flaws came from, all I need to know is where they are.

Nothing here seems like rocket surgery, give the current state of the art. Probably the IntelliJ guys could code it up in a month, if they thought that enough of their market had the high-end quad-processor development boxes it would take to make this a pleasurable user experience.

I'm sure that the implementation technology is getting there, although the refactorings that IntelliJ can suggest are currently quite limited. It cannot pull out classes and interfaces, work out where to introduce delegation and so forth.

However, the user interface is not right. The most successful programming environment in the world, Excel, is designed around copying code. What it doesn't do is find and introduce abstractions. But people don't find thinking about abstraction very easy. It takes programmers decades to become good at it, and warps their thinking about other things in the process. It's not that non-programmers aren't good at abstraction, it's that they actively don't like it. They want to work with the concrete.

Now, we all know as programmers that working with the concrete becomes increasingly tedious and error prone until the code is unmaintainable and has to be discarded.

So what I'm envisaging is a programming environment based around copy-and-paste that then introduces new abstractions to the programmer/user when they find that they need them. I'm thinking of the programming tool really being an iterative learning tool, a way of exploring and understanding the domain that you're writing programs for without lots of up-front analysis and design that delays delivery of actual functionality.

The system might work in a question-and-answer style. If the tool finds it hard to keep track of all the duplicated logic you could explore the abstractions that it has found and it would then ask you to name them and name the relationships between them. You could then select abstractions that you feel comfortable working with and the user interface to your program would morph to allow you to work with those abstractions. While working, you could morph between different levels of abstraction and the tool would reconfigure abstractions to reflect changes you've made at different levels and illustrate those reconfigurations with some whizzy graphical goodness. You could have different abstractions on the go at once, giving you different viewpoints on the system, and could morph between them.

I know I'm being vague but I have trouble foretelling tomorrow morning, let alone 50 years from now.

I'm pretty sure that what you are getting at maps to a subset of Generative Programming/Programming By Intention. Worth checking out if you haven't, yet.

I'm sure that the implementation technology is getting there, although the refactorings that IntelliJ can suggest are currently quite limited. It cannot pull out classes and interfaces, work out where to introduce delegation and so forth.

Funny you should mention that. I'm the implementer of most of IntelliJ's code inspections, and many of it's small refactorings. Now that JetBrains has opened up some more API, I'll be adding global analyses to suggest classes and interfaces that need split or merged, methods and fields that need moved to a subclass or merged to a superclass, methods that should be inlined, delegation that should be introduced or removed, packages that need merged or split, and a bunch of other goodies. Exciting stuff. Should ship in late summer, I'm guessing.

That sounds fantastic! You guys really know how to produce a good development tool.

Will any of this end up in Resharper? How do you share technology between the two products?

But I should note that I don't actually work for JetBrains. My work was done as an open-source plugin to IDEA, which was merged into the product a couple of years ago.

Will any of this end up in Resharper?

I can't say anything about that, other than that it would certainly be cool, and I'd love to do it.

I swapped a few emails with Fowler years ago on the subject of code duplication. Everyone I know does it. It's a very common operation in real world refactoring of code. And yet he didn't want to add it to his canonical list of refactorings. A typical scenario: a class appears to be serving two separate purposes. You duplicate it and then remove the parts from each that no longer serve a purpose. You could, instead, write two new classes from scratch - but that is more error-prone. It's far easier to test at every stage when you duplicate the class and delete functionality that has become redundant as a result.

Everyone I know does it.

I (generally) don't do it. On the contrary, I aggressively look for duplicated snippets of code and remove the duplication.

You duplicate it and then remove the parts from each that no longer serve a purpose.

That is unlike the cases of duplication that I see dozens of times a day. What I see is that code is duplicated and then some parts of the duplicate are changed. Often it is just partial alpha conversion to account for differently named local variables. I've never seen a case of (what I would call) duplication, where parts of the original were deleted after the copying.

What you seem to be describing, I would call "splitting". The smell you are describing is a violation of the "Single Responsibility Principle" or the "Interface Segregation Principle" as described by Robert Martin.

Frankly, I view code duplication as a strong indicator of incompetence (inability to form abstractions) or immaturity (does not understand the maintenance burden) or both.

...on the way to "splitting". Often you can't make a clean split because some piece is still shared between the two 'children' (which may or may not itself be refactorable later) so you can't literally 'split' the code. Instead you duplicate and remove redundancy. I agree that duplication as a final step is likely to be a sign of incompetence (or simply not yet having finished the job).

Look at subtext (subtextual.org)
Read the 3 papers (including the recent one published by the author) and the discussion in ltu about it.

I have seen that presentation before (perhaps an older version). I like the idea of not having to work with text-source-code (again, only because existing methods of 'coding' are so good, whatever replaces it should be very exciting). Unfortunately, I don't like the solution very much. It looks like coding in subtextual will take longer and be more difficult to understand than just typing out code in a nice editor.

Well, it's very early days yet and Jonathon Edwards admits that he's working on the implementation and semantics and that the UI is work to be done.

Why bother with artificial minds? If they are going to be minds of human ability we might as well just use humans.

we'll be too busy in a perpetual vacation in dreamland... let the machines think for (and crop) us! ;)

IMO there'll be only a few moments where a artificial mind will be as able as a human mind. After that it'll evolve itself and be smarter. A couple of days later it'll solve the dynamic vs. static debate and all programmers (and LtU) will become obsolete ;)

People have been saying this for years. Back in the late 80's it was the CASE tool folks. Now it's the MDA folks. In a few years it will be the graphical DSL folks. However, they all run into the same problem: if the model is detailed enough to be turned into a program the modelling language is a programming language that just happens to have a graphical representation and you're not modelling any more, you're programming.

However, I think the line between modelling and programming is arbitrary, ultimately the complexity of the model is irrelevant, as you would continously be modelling ever-finer sub elements of parent elements without it being caught up in any sort of syntax or side effect. The graphical representation can hide the implementation details of the sub elements, and modifying the sub-elements does not affect the overall architectural flow of the model.

Hmm. I just realised that I've been talking about FP in a graphical way :eek:

Of course, you have to put in more work to build an *attractive* phone UI that is fully Web 2.0 buzzword compliant.