Lambda the Ultimate

This paper fits into the discussion about being more scientific about programming language design, or in this case editor design. I particularly liked figure 6.

The impact of syntax colouring on program comprehension

Abstract:

We present an empirical study investigating the effect of syntax highlighting on program
comprehension and its interaction with programming experience. Quantitative data was captured
from 10 human subjects using an eye tracker during a controlled, randomised, within-subjects
study. We observe that syntax highlighting significantly improves task completion time, and that
this effect becomes weaker with an increase in programming experience.

We've mentioned some empirical studies of programming languages a few times, but I haven't seen a comprehensive list we can use as a reference.

Fortunately, I just came across this pretty decent overview of existing literature on how types impact development. Agree or disagree with Dan Luu's position, the comprehensive list warrants a front-page post in my opinion.

One point worth noting is that all the studies used relatively inexpressive languages with bland type systems, like C and Java, and compared those against typed equivalents. A future study ought to compare a more expressive language, like OCaml, Haskell or F#, which should I think would yield more pertinent data to this age-old debate.

Part of the benefits of types allegedly surround documentation to help refactoring without violating invariants. So another future study I'd like to see is one where participants develop a program meeting certain requirements in their language of choice. They will have as much time as needed to satisfy a correctness test suite. They should then be asked many months later to add a new feature to the program they developed. I expect that the maintenance effort required of a language is more important than the effort required of initial development, because programs change more often than they are written from scratch.

This could be a good thread on how to test the various beliefs surrounding statically typed and dynamically languages. If you have any studies that aren't mentioned above, or some ideas on what would make a good study, let's hear it!