Lambda the Ultimate

I know TeX is not much loved as a programming language, but it (oops, Charles Stewart correctly points out that I mean LaTeX2e) is still very much the markup language of choice in many professional disciplines, with my own (mathematics) perhaps the most insistent on this choice. I think that means that this:

Lout would be a more sensible choice these days, if you didn't care about Unicode.

at least deserves some explanation. I'm not saying it's untrue—I've no experience with Lout (and quite a bit with TeX), so no basis on which to speak—just that I don't think that it's self evident.

The markup language usually chosen is not Tex but Latex 2e: while the reference implementation of Latex 2e is Tex 3, there are two main reasons to count Latex as not just being a purely macro-realised Tex dialect:

1. Source-to-source transformations: publishers typically want to rerepresent Latex code so that it conforms to their house style, to gather together several Latex articles into one Latex book, &c. Unrestricted use of Tex can subvert some assumptions about the structure of the document.
2. Translations of Latex into other formats, e.g., SGML, for typesetting: these translations typically don't support all of the Tex language. E.g., the Handbook for the Digital Library of Mathematical Functions uses a Perl-based Latex-to-Xml converter, LaTeXML, which simulates the execution of a subset of the Tex language.

You make a very good point, that one should not conflate LaTeX and TeX, any more than one should call Perl C because the Perl interpreter is written in C. Thanks!

I hadn't heard of LaTeXML before; I only knew about JadeTeX, which goes the other way (and therefore supports TeX more or less perfectly, and XML less perfectly). It does remind me of my one gripe about TeX/LaTeX, that, with all the bizarre powers it offers—like manipulating catcodes—that make it a less-than-ideal programming language, it doesn't offer what seems like the natural power to swap out the back-end. That is, it wants to output DVIs, or at least PDF or PS or some other sort of final-product document.

I have wished many many times to be able to have access to a TeX document just after it was compiled down to ‘primitives’ (by which I mean something a little less primitive than the actual primitives—I don't directly care about \kern, for example) and just before it started lumping the text into boxes to be laid out on a page; that is, to be able to use TeX as something like an m4-style macro expander. It seems to me that being able to do this would make both of the applications you mention, to which TeX is exactly not suited and even LaTeX is not perfectly suited, essentially trivial.

Axiom is a large (about 1M things of code) computer algebra program
written in common lisp. It is being rewritten into a literate style.
(http://axiom-developer.org/axiom-website/documentation.html)

I was one of the original authors at IBM Research. The code was sold
to another company and was a commercial product for years. It was
withdrawn from the market and given to me to open source it.

I discovered that I was unable to understand programs I wrote 15 years
ago. I knew WHAT it did but I did not know WHY it did it or WHY it did
it that way.

The issue of literate programming is an issue of writing a program
that LIVES rather than writing a program that WORKS. In a commercial
setting you pay to train new people on programs but in an open source
setting there is no training. Beyond a certain point of complexity
you can only understand a program by talking to the author about why
they wrote it the way they did.

Axiom, for instance, contains algorithms that are backed by research
papers. However, the algorithm gets changed over time so it no longer
matches the research due to optimizations or other rewrites. These
changes can be important but very obscure. No amount of clever naming
and javadoc-API commenting will help.

So I searched for a long term solution where hundreds of authors can
communicate over dozens of years. Literate programming is that
solution.

Literate programming is not a good idea for programs that just need
to work rather than live. If the program is going to be rewritten or
thrown away in less than 5 years then don't bother with literate
programming. But if your program needs to live forever then you
really need literate code.

Good to know somebody else feels my pain.

I've been encouraged to write about this. See:

My Re-thinking of Literate Programming
Flexible Order of Elaboration
Planned Obsolesence

Warning: this is very much brainstorming personal diary style writing.

Feedback, here or via e-mail, appreciated.

One thing to note is I don't typically share your pain about why I coded something some way. Instead, my concerns are data schema related. I want to be able to ask, "If I wiggle this in the front, what wiggles in the back?" For me, design and optimization tend to be orthogonal.

I plan to add a recommended reading list soon, as well as an assessment of current tools.

It seems to me that a better answer to your kind of WHY question would be a well-maintained revision history of the code base, with properly explanatory commit messages and a toolset that makes it easy to browse the code's history.

What rcs system lets you input searchable rich hypermedia or searchable typeset commit messages?

Moreover, this is like comparing ACM Portal Search for "tree pattern matching" and "tree acceptance" and "tree parsing" to Loek Cleophas compilation of tree algorithms into a coherent taxonomy with FireWOOD and ForestFIRE. Yeah, you could do things the hard way, but over the next 200 years, your way will have much more bit rot.

Axiom, for instance, contains algorithms that are backed by research
papers. However, the algorithm gets changed over time so it no longer
matches the research due to optimizations or other rewrites. These
changes can be important but very obscure. No amount of clever naming
and javadoc-API commenting will help.

In my experience code repository histories with good commit messages often do a decent job of documenting the WHY behind small-to-medium sized code changes.

Didn't see boston lisp group had this as a discussion... I'll probably stop by if I can finish early today.

Literate Haskell is not traditional literate programming in some respects...

I planned to mention the Haskell case, and ask why it is so much more successful. Thanks for saving me the need to raise the question. The traditional explanation (== the multiplication of two small probabilities) does seem to apply to the Haskell case, as well.

Haskell is more successful, IMHO, b/c of the tools MODEL. The compilation model for Haskell fits the natural ebb and flow of a diary.

In short, Haskell allows very rich flexible order of elaboration. Still not nirvana, though.

Don't forget that besides LP abstraction (aka chunking) you can also extend chunks (multiple definitions are flattened in sequence). This is a powerful tool to support separation of concerns, at least when features can be composed linearly.

Any system that involves running some sort of tangle & weave operations as separate tasks so that the compiler sees something different than the programmer is ultimately doomed.

More generally, any composition-based approach that uses Web/Tangle/Weave is ultimately doomed (a slight alteration of my original position; see the LtU thread What is the best literate programming tool/environment/research work? and in particular this comment). It is trying to solve problems at the wrong levels of abstraction, at every level of abstraction, thus creating Abstraction Inversions everywhere that are downright counterproductive and unnecessary.

Again, Haskell does well mainly because its programming model is one of the best for flexible order of elaboration.

The Great Wheel of Reincarnation must be turned, again - THIS TIME for programming environments. (A point not mentioned by Barbara Liskov at OOPSLA when she was dissing aspect-oriented and giving the audience a CS History 101 lesson).

I think you can consider any real LP system (e.g., CWEB or NOWEB) as an command-line alternative to a graphical development environment: it trades obscurity of control commands for obscurity of toolkit options. Both give a visually very much subjectively pleasant way of looking at programs, in the sense that you can include any figures, formulas in the text and structure it in any way you like. Doesn't this help capture some architectural details?

Also, inversion of code and docs containment helps to abstract away some incidental syntactic clutter that you get with specific languages such as ML or Haskell or alleviate some idiosyncrasies of Scheme. That is, you can start with the document/program structure (using named chunks) and refine it all in the appendix with the actual code.

But I have to say that I do miss the power of a Turing-complete language at the LP chunking level...

I always thought that LP was a great idea much before its time. With the Internet, HTTP/HTML/XML, blogging, and the long tail of context on the WWW, we *now* have sufficient technology to host our code in rich content, maintaining context to make it properly "live". Even code which is capital equipment, destined to be used up and thrown away, could be (and sometimes is) "opened" onto wikis and blog/tutorials which intermix natural language explication with the code itself. This can and (rarely) does include multi-media explication.

We just don't develop and deliver code that way, even open-source code which has a purpose to attract users and support is hidebound with concepts of the installer separate from the support web pages and documentation. Partly this is because coders are just as rarely good movie makers as they are superb writers, but also this is because no one has taken the time to re-design LP into a modern context:

I want a "CodePress" which supports LP in multiple languages across distributed teams of developers in a way analogous to the way that Drupal and "WordPress" supports the interchange what we are engaged in this minute!

Have you been paying attention to new open-source project hosting companies/websites like Gitorious and GitHub?

I'll probably write a blog post about this in the future as part of my LP diary, but the bottom line is that new DVCS hosted websites are completely changing the way software is done, and it will have dramatic effects throughout industry!

Here is one example: Microsoft just launched CodePlex Foundation and put Sam Ramji as its lead. CPF is now independent of MS and Ramji thus no longer works there as an open source evangelist. He has clearly done his job in that capacity, and now it is time to move on to something bigger. Yet CPF is behind the times in terms of how to charter a massive online collaborative code outsourcing system.

Punditry: I think this spells the death of the concept of a SDK as we know it, and permanently changes what qualities we use as an industry to define 'platform'.

I think discussions of the wonders of literate programming tend to vastly underestimate how hard it is to practice in your typical industrial project environment, even if you are a decent writer and coder and value clear code.

Just as a new idea late in an essay or blog post can call for a substantial rewrite of earlier material, relatively small changes made to the code can require a pretty major rewrite of your literate text.

When you are making dozens of changes daily (bug fixes and features), the text that made perfect sense this morning could be lying gibberish by quitting time.

On top of that, unless your literate code is a valued deliverable as it can be in an academic context, the cost of the upkeep isn't going to be worth it.

This is why agile approaches to documentation focus on clarity and comprehensibility of the code itself and of its supporting tests: the pay-offs are similar but more closely aligned with the deliverables.

On a happy PL note, this is actually a problem that PL expressiveness features can help you with: offering better abstractions, such as rich type systems, helps you to make what the code means and what the code says come closer together, without having to resort to separate literature.

Software development processes need to be more than just a neat homoiconicity trick.

Data has no value in and of itself; put it into a context, and you get information. The programmer should be a knowledge worker, not a punch card operator.

Big idea in software architecture right now: Put your codebase to work to help you solve problems, rather than just a developer taking swags at solving problems in your codebase.

I'm not sure if the content of your message was intended to respond to the content of mine or not, but the sentiment expressed by your title

Code is just Data

strikes me as an odd one for a PL/PLT enthusiast.

A big part of the appeal of this subject area for me is exactly because code is a very elegant language that at its best is rigorous enough to be understood by a machine, but clear enough to be understood by a person.

A competent writer needs to master the structure of their ideas to be able to express them clearly, and also needs to master the structures and idioms of their language to express those ideas effectively.

I think of a competent programmer as just a competent writer in a specialized domain using a specialized type of language.

Relying on non-code prose to express the clarity of your programming is like relying on being able to wave your hands to clarify things when you giving an oral presentation: it may help on occasion, but the effort might be better spent making the main content stronger.

For some reason I am reminded of the discussion of pseudo-code...

I am reminded of the discussion of pseudo-code...

That was lurking in the back of my mind too... ;-)

I should've said:

I agree that automated testing is important, and strongly believe in doing BDD or TDD or contracts-first where it makes sense.

At the Boston Lisp meeting, somebody actually asked CLWEB's author, Alex Plotnick, about including tests as part of the literate program's WEB. Alex said he chose not to do this directly, but there was a separate output stream for testing the code.

Another good point Alex made was that he found he didn't write long, monolithic subroutines like Knuth did, because he felt Common Lisp provided him with more code-oriented ways to chunk his program. Alex then directly compared Common Lisp's modularization facilities, and noted he used fewer WEB named sections than Knuth, because his programming language eradicated many uses Knuth had for named sections. In turn, Alex said he was also then able to use named sections in different ways than Knuth had.

Relying on non-code prose to express the clarity of your programming is like relying on being able to wave your hands to clarify things when you giving an oral presentation: it may help on occasion, but the effort might be better spent making the main content stronger.

And relying on humans to read your code, when a machine can do it for you as you program, is premature quality gate optimization. Just as a code review doesn't really need a human element, literate programs don't necessarily need an uninterpretable English prose element. Natural language researchers have been studying the Bible and other classical works forever, discovering new forms of how phrases in one verse of prose can make an avert reference back to another verse of prose. Sometimes, these references aren't even in the same "book" or "chapter". For this reason, linked commenting is very useful.

Automated tests, especially done in a BDD-style, tend to serve as a form of linked comments. Except the linking is done in build.xml or hoisted into a Continuous Integration server. (Before Alex's talks, Daniel Herring gave a talk that basically said, among other things, yeah, great build systems are still a research problem.)

Now, you might argue that in most cases this is overkill. You're right. Ted Nelson was simply wrong about transclusions being that necessary, as the design of the world wide web has proven. On the other hand, the failure of the semantic web suggests things like transclusions are very useful, as when you only have one-directional links you often end up inventing isolated subsystems like RDF and OWL to relate documents to one another.

Relying on non-code prose [...] may help on occasion, but the effort might be better spent making the main content stronger.

I agree. David hit a major pain point of most programmers being afraid to use good abstractions due to trying to address deep composition of orthogonal concerns in very non-orthogonal, tightly coupled ways.

Going back to premature quality gate optimization...

Code is just Data

strikes me as an odd one for a PL/PLT enthusiast.

Brooks said you can't make a baby 1 month with 9 women. He may not be a medical doctor, but I believe him. He also said you should eliminate communication between people as much as possible, and suggested you make a team of 7 members. One of these members he called a Language Lawyer, responsible for peer review of code and other duties. Well, if your Language Lawyer is a computer with a ferociously acute antennae for detail as Herb Sutter or Andrei Alexandrescu, then you can pair program with this Language Lawyer and also have very fine control over when you ask for advice and when you listen to advice. You can't get this with humans. Moreover, a computer can paint a flourish on your 30" wide screen computer screen, and bring up associated documentation for you on your 20" vertically positioned side monitor. Herb Sutter would instead maybe point a bulbous finger at some line of code and say something. Moreover, his message might not be the same thing every time, so you've got way more to interpet. Finally, because computers' brains can be copied easily, you can share Herb's Laws with every programmer in your office.

I am a usability freak first and foremost, then a theory enthusiast.

_

Code can't explain who should be contacted if a bug is found, or that this code is due to be superseded in 2010Q2, or point out how it needs to be adapted if moved to another environment, or tell you the practical limitations of the algorithm it implements.

Yes, you could do all these things in annotations, but they'd have to be an open-ended set, just like natural language.

Code can't explain who should be contacted if a bug is found, or that this code is due to be superseded in 2010Q2

Just use Intentional Programming Studio and your weirdest code snippet becomes transparent to the reader - maybe even to yourself.

I used Leo for organizing a major rewrite (9? years ago). Literate programming was 'ok', but the major revelation for me was the usefulness of outline editors.

Since then I've been complaining about having to put code (and documentation) in "files".

Mono's approach is more easily extensible to the "Cards" metaphor.

Only up until recently was the Javadoc/NDoc/Sandcastle approach was more practical. IDEs are now good enough where they should be able to easily load associated "Cards" when looking at a particular piece of source code.

Most peer code review tools do something like "Cards" anyway, so the Mono approach just supports superior integration of concerns.

Things MSDN does, like displaying the language syntax in the help file, can also be done with "cards" metaphor by just writing a "Card" that takes a CodeDOM Walker and Language as parameters, since for interfaces the mappings are usually very simple.

{edit: forgot to mention that you do want code and comments integrated (but not necessarily tightly coupled to the same file). Somebody did a research study around the time javadoc was built, showing that the locality of documentation with respect to code was a MAJOR factor in detecting avoidable flaws. Unfortunately, I can't find this research paper.]

Is-it still litterate programming?

If you're using an IDE and you can bring up the right pieces of information, then it is still jointly connected and should be browsable by any hypermedia-aware system -- such as the IDE's browser. [Edit: Knuth also prophesized this would happen!]

Mono's tool designs are very unusual to most programmers. Mono just places a really high investment in tools compared to most open source projects. This is mainly due, I think, to how C++ caused the Ximian folks to mentally meltdown at times, and they basically said, "Never again." Also, great tools (and licenses) make it easier to scale up a project to thousands of contributors.

What I call the "Knuth Model of Literate programming" views things differently. First, Knuth does things very monolithically. He thinks your prose should all go in one file with your code, and you do things like write alternating weave and tangle escape sequences to make the LP environment "Web/Tangle/Weave"-aware.

The real downside to Mono, when I looked into it, was you had no way to certify or whatever a set of documentation really matched a set of source code. In other words, you had no way to say, "this documentation is English prose for this set of design philosophies, unit tests and integration tests".

The real downside to Mono, when I looked into it, was you had no way to certify or whatever a set of documentation really matched a set of source code.

I thought that the usual LP programming methodology also offered no such certification. Have I misunderstood it?

You are correct - I know of no component or module packaging solution that carries with the component or module a resource that describes all the QA measures used with the code. Proof assistants and automated theorem provers have a concept of "proof-carrying code", but there is no tight integration of concerns for non-heavyweight formal methods like program proofs. I'd say "quality gates" are the best metaphor.

However, that doesn't change the principle movement of the early 2000s of "opinionated software" that code should be test-driven, which later morphed into ever more "opinionated software" called Behavior-Driven Development that tests should be literate specifications that describe use cases and scenarios as "user stories".

A lot of the points I've mentioned (and David Barbour also mentioned this) kind of take a step back and look at what people actually want - as well as what would attract the non-traditional non-stereotype programmer who might prefer a language like Field or Subtext or Processing.

My views are very much a hybrid of Ted Nelson and Donald Knuth, and I try to capture a lot of Ted Nelson's forward-thinking about technology that he did in Computer Lib/Dream Machines.

What do all these activities, if practiced, have in common? They help communicate what (should) happens when code executes.

I hope this doesn't sound ideological. The whole point is to say multiple communities exist, each with a piece to the puzzle.

Seems to me, we simply need a type class-like abstraction for comments. This way we can overload comments for each language to which the comments need to be translated, and the compiler can know exactly which comments are available or missing, and a smart code editor can use that information to provide traditional code views with embedded comments.

Also, the formatting of this paper is insane. The long code examples would best be pushed into an appendix -- OOPSLA's required two column layout is not friendly to LP demonstration and is more concerned with tight page count!

I wouldn't blame OOPSLA (and I actually much prefer 2 column papers). Perhaps this is a weakness of LP systems? 2-column papers are just one viewing medium: as a developer, I use many when looking at code.

Maybe it is a shortcoming of LP systems.

Systems do have weaknesses. Some systems are based on merely good ideas. Some systems are based on great ideas.

Would you clarify those closing thoughts?

Another feature that seems essential is the visually polished nature of the woven output. TEX seems like an overkill, yet something as lame as ReST is not good enough.

Are you referring to Representational State Transfer? If so, then how could you compare this to TEX? It is an architectural style, and not even close to TEX. However, the idea of separating resources from their representation is a very good idea.

What makes sense is to compare the world wide web to Project Xanadu or Chimera or something else, and then compare TEX (a very static, monolithic hypermedia type) to a new hypermedia type.

ReST == ReStructured Text.

By your own admission, Literate Haskell is anything but.

What I wrote was that "Literate Haskell is not traditional literate programming in some respects, but that may actually make it more practical." My own experience working with literate tools[*] is that Literate Haskell is one of the more practical systems, and that it more easily produces more useful and maintainable results than WEB-style tools, partly because it has a more streamlined workflow and a more direct relationship between executable code and literate code.

I'm not sure then what to make of your intimation that GHC furnished a counterexample to my assertion that LP was unlikely to work for programming in the large, trivial counterexamples be damned.

I think we're talking at cross-purposes to some extent, and need to distinguish clearly between the vision of LP, as opposed to what most existing, rather inflexible tools actually achieve.

The idea of being able to associate rich documentation with a codebase and transcend the linear limitations imposed by file-based program source representations is an admirable goal, and I look forward to systems that can do this credibly. Repository-based languages like Smalltalk have demonstrated the possibilities. However, outside of single-language sandboxes, there are also many practical barriers to achieving this goal. Largely as a result of those barriers, existing LP tools tend to be rather flawed.

Re the objections to Literate Haskell:

Literate Haskell goes against the "essence of WEB". (A quick account of its transgressions can be found in Glenn's paper.)

Two of the three objections in Glenn's paper are trivial and have long since been resolved: the lack of "automatic cross reference of functions and types" and "no prettyprinting facilities," both of which are now ably handled by tools like Haddock, HsColour, etc. These were never serious conceptual objections. Some things in this area could still be improved today, but it doesn't require any conceptual breakthroughs.

The other objection of Glenn's is the central one whose status we disagree about: Literate Haskell has "no explicit naming of code chunks." But I consider that its main advantage over traditional LP tools, since it allows it to do away with the tangle & weave layer, and removes one of the main sources of disconnect between the literate program and the underlying source. However, I'm making this criticism in the context of current literate tools, and in the context of programming languages that have sufficient abstraction power to not require extralinguistic aid in this area. In this context, the Literate Haskell approach is preferable to allowing the LP tool to dabble inexpertly in the creation of abstractions.

You gave the following manifesto for the underlying requirement:

Stated briefly, if I feel like moving a chunk code out of a where clause into a literate section of its own, I should be able to do so. This is my inviolable right as a Literate Programmer. I, the author, dictate the order of exposition — not the syntactic restrictions of my programming language.

However, nothing in a functional language stops you from naming a chunk of code in a 'where' clause (or its equivalent) and moving it somewhere else. There are two scenarios, though. The simplest one is where the chunk has no implicit dependencies on its context, i.e. doesn't close over any variables defined in enclosing functions. In this case, the chunk can simply be named and extracted. I don't see any good justification for using an LP tool, rather than the language, to do this. There's no "syntactic restriction" here.

The other scenario is where the chunk does close over variables from its context. In this case, to move the chunk, you either need to manually lambda-lift it, adding the necessary arguments to eliminate the implicit dependencies, or else (particularly in Haskell) use an abstraction like monads or arrows to provide a carrier for the implicit context.

The traditional LP approach to this just doesn't care about context. I don't think I need to expound on the problems with this, since as you point out: "It's abundandly clear that PEWS is not a good feature for any sane programming language." You suggest that because of this, it makes sense to introduce the feature extralinguistically. But to me, that just compounds the insanity you rightly refer to.

Every time you introduce a PEWS that has implicit dependencies on the context it's been extracted from, you're effectively writing a function which, for reasons that are unclear to me, isn't being expressed in the underlying program as a function. This creates a situation in which the reader of the literate program is likely to be left wondering about the meaning of the free variables that are present in that chunk of code, not to mention the potential for error this creates in the ongoing maintenance of the program, particularly in the invoking function where the presence of the dependency is completely invisible. In contrast, turning the dependencies into explicit arguments allows them to be individually documented with an API documentation tool (like Haddock), improving the literateness of the program, and eliminating the semantic danger posed by invisible dependencies.

It seems to me that what we're really dealing with here is a legacy of the time Before Steele (B.S.) when the expensive procedure call "myth" prevailed. (For many languages of the time, it wasn't a myth, of course.) For languages where writing long functions was de rigeur, it's easy to see benefits in being able to break up and rearrange functions for documentation purposes. But in modern functional languages, functions tend to be quite short - often just one or two lines, and when longer, often broken up into natural parts e.g. as pattern match cases which each look like a separate function. Compilers are very good at inlining and otherwise removing overhead introduced by this functional abstraction. I think you'd be hard-pressed to find examples of well-written Haskell, ML, or Scheme code that can't be well documented, literately, without needing to resort to LP quasi-functions. Code that's been written in the FORTRAN style might need some refactoring, but that's not a bad thing, and the last thing you really want is to waste effort on refactoring your program at the level of documentation while leaving the underlying program as a pile of un-abstracted spaghetti.

In conclusion, if you want practical, literate programming today, the Literate Haskell approach is a serious contender. However, I'm not claiming that it's a step forward towards the vision of transcending linearity, it's just that it's more practical than the traditional LP approach to trying to fake that transcendence.

I agree that things like wiki technology, hypertextual approaches in general, integration with API documentation, and exploitation of (distributed) revision control are all promising directions for real ultimate literate programming power. In that context, there's a strong case for being able to click on some name or icon in a piece of code and have it expand in situ into the code represented by the abstraction. But short of that, trying to fake such rich relationships with a non-integrated multiphase toolchain manipulating static bodies of text is an idea whose time doesn't deserve to come.

[*] I have some public examples of the R5RS semantics and the SPJ/Eber/Seward financial contracts language. I've also used LP on numerous occasions to document algorithmic code for the benefit of domain experts who aren't primarily programmers. I'm hoping to put some small examples of that up on the web in the next few months(/years.)

Re: comment-52052:

Alex Plotnick's CLWEB turned out to be really, really nice for what it is.

A new kid on the literate Lisp block: LP/Lisp by Roy Turner.

On an entirely unrelated note, here's an amusing speculation from http://c2.com/cgi/wiki?LiterateProgramming

One speculation for the reason behind Knuth's pushing of LP is that according to Stanford's intellectual property policy, Stanford would have owned all of Knuth's code, but not his published writing. So the answer's simple: make the code part of the document, and while you're at it, be sure to minimize the appearance and importance of the code itself.

The work is still in progress, as you might expect.

Learning is happening and several interesting system-wide changes have
been added. As with any large project, you eventually discover certain
interesting changes to organization.

First, we've added a general bibliography document to collect all of
the references over the 20 volumes.

Second, there is a slow-moving change to remove noweb and add latex
style chunk names. Thus
<<definition of chunk>>=
becomes
\begin{chunk}{definition of chunk}

and the use of the chunk becomes
\getchunk{definition of chunk}

This makes the whole document just pure latex.
You no longer need a weave function.
The tangle function can now be done at (read) time so lisp can
treat the literate document as lisp source code.
Thus, noweb is begin phased out as the rewrite happens.

It is trivial in lisp to read and store document chunks in a hash
table so multi-pass noweb extractions become single pass.

As more of the system is embedded into literate books, more of the
"make" functionality is being written in lisp, the language of the
implementation. Thus, not only is noweb going away, so is "make".

The system organization is much cleaner. Literate programming forces
you to "place" your code into the overall organization in a logical
way. You don't just add a file in some random directory hierarchy,
you add it "where it belongs". The index gives hyperlinked cross
reference information that includes not only where variables and
functions are defined but also where they are used. The combined
table of contents gives an automatically generated overview of the
whole system structure.

Literate programming also forces certain documentation standards.
Help files and test files are kept with the code and automatically
extracted. Eventually it will be an error if these do not exist.

Literate programming is a "large system" discipline where "large"
is either many lines of code or intended to live a very long time.
Individual programmers will only find it a burden since they don't
have the need to communicate with other humans. However, big systems
and long-lived systems like Firefox, Linux, MySQL, could gain a lot
by having the code laid out in book form. New people joining the
project can understand the organization, the reasoning, and the
low-level hacks and performance tweaks. Imagine all of linux in
literate form where you could read about device drivers, their
timing constraints, their data structures, their API, etc. A lot of
that wisdom sits in the heads of those who have done it. I'd rather
just grab Volume 42: Linux Device Drivers and read how to do it.

I think that big commercial vendors of large pieces of software
(e.g. Mathematica and Maple in computer algebra) would be wise to
transition to literate programming. IBM, Google and Microsoft could
use the technology. Imagine the grief that must be happening at Oracle
because they now have to decode the Sun software without the original
authors. I'd bet they would love to be able to give the "Sun books"
to the Oracle employees.

In fact, literate programming opens up a whole new discipline.
There is a need for "Editor-in-chief" on the software teams, that is,
someone who knows how to enforce "readable prose" rather than jargon.

Ultimately, the key issue is quality. Literate programming is a tool
to improve the quality of long-lived software.

Note to self: Comment-62437 was apparently in response to the the question I asked in comment-52052:

Re: comment #51878

I'm looking forward to finding out how your rewrite goes. Overhauling “1M code things” is quite a bit of work. Would you mind posting a progress update on this page in a year or so?

It's been a year or so. I'm duly impressed with Tim Daly for keeping track of such things.

Interesting perspective you provide to the LP debate. I actually didn't get to stick around for the end of Alex's talk (and came late to Dan's talk due to having a hard time figuring out where Room 34-XXXX was). I wish I could come to more Boston Lisp meetings (this was my first since moving here in April 2008), but they are usually on nights I do community service.

At some point I need to write an explanation of how various IDEs tackle basic issues that form a foundation for writing code well. For example, good continuous error messages. Where do you place the handlers in the overall system? I think this basically echoes your point about productizing LP, although I'm not sold on all of your conclusions. But there really is no guide for people who want to productize programming languages and their associated environments. There is no open explanation of what problems to expect, and ways to solve them. Most state-of-the-art uses a fair amount of heuristics.

I am guessing that you feel Lout is better than the G-expressions in the paper I posted above?

_

I'm a little puzzled by your questions. It seems you are thinking
of literate programming as a form of "documentation" (which it is)
rather than a form of "communication" (which it is). If you are
writing a literate program, you are trying to communicate to another
human separated in space and time.

Many people make the mistake of thinking that literate programming
is just "better documentation". This mindset is easy to spot, just
look for suggestions about "better variable names", "refactoring",
"javadoc", "self-documenting code". Clearly this misses the point.
This view sees literate programming as another programming "tool"
or another programming "style".

Step away from the machine. Literate programming has nothing to do
with tools or style. It has very little to do with programming.

One of the hard transitions to literate programming is "literate
thinking". Suppose you were writing a novel. One of the primary things
you need to do is make sure that everything that happens in the book
is "motivated", that is, it is introduced because it it needed.

Imagine explaining your software by motivating the need for it, then
motivating the design issues, then motivating the organization, then
motivating the implementation details. Natural divisions are organized
book-by-book and chapter-by-chapter.

The documentation "evolves". If you were writing a novel where the
character had to do something uncommon (recite shakespeare from
memory) then you need to go adjust the prior chapters to hint why
the character might be able to do that. Thus, you tend to "bounce
around", trying to connect the parts into a coherent whole.

Parts that are "invalidated" need to be rewritten. In fact, that is
a good way to know that the underlying software needs to be rewritten.
Other parts of the book (and the underlying software) that use the
rewritten parts need to be reworked. Often this doesn't happen because
the programmer has no idea that someone else depends on his work,
especially in a large project.

Large programs and long-lived programs (IRS tax codes, MS Windows,
Mathematica, Space Shuttle software) written by many people who are
spread out in space and time really need this level of documentation.

Writing your diary can take any form. Writing for a large audience
requires much more discipline. It is not about the tools (pencils?
javadoc?) or the style (first-person? good variable names?), it is
about the audience.