Lambda the Ultimate

inactiveTopic Evaluating a new programming language
started 3/18/2002; 1:43:39 AM - last post 3/20/2002; 1:18:12 AM
Ehud Lamm - Evaluating a new programming language  blueArrow
3/18/2002; 1:43:39 AM (reads: 1505, responses: 5)
Evaluating a new programming language
We describe our efforts using the Cognitive Dimensions framework to evaluate a new programming language. We used a questionnaire approach to gather framework data related to the new language and combined this with a traditional lab-based evaluation. With this approach, we were able to validate the findings from the cognitive dimensions questionnaire by correlating them with the findings from the lab-based study. From this, we were able to determine if it is feasible to perform such an evaluation with usability participants who have no prior experience of the language being evaluated. Our findings have implications for anyone intending to use the cognitive dimensions to evaluate new notations or programming languages.

The language studied is C#.

I am far from certain that this type of study is the 'right' way to check language usability. But at least it is a bit more scientific than introspection...

Comments?


Posted to critiques by Ehud Lamm on 3/18/02; 2:04:53 AM

John Lawter - Re: Evaluating a new programming language  blueArrow
3/18/2002; 8:40:42 AM (reads: 1023, responses: 0)
The sample group they used (5 people) seems too small to me to form reliable conclusions.

(I wasn't familiar with Cognitive Dimensions before reading this paper. A quick Google search produced this link which has some more useful information and bibliography.)

Jo Totland - Re: Evaluating a new programming language  blueArrow
3/18/2002; 12:22:24 PM (reads: 988, responses: 0)
Summary

We had five more or less experienced C++ programmers do something in C# using Visual Studio as the programming environment. Then we had them do a questionnary. Some of them were a bit confused about parts of the language, or in using Visual Studio. We believe one must be critical when looking at the data we collected, and that more research in this area is needed.

Is that really all? Well, some of the comments they collected were actually interesting, especially the one about IntelliSense. I've never thought about that as something that encourages bottom-up design, but it probably is (although I still consider it a good thing, and would have, even if I weren't partly a bottom-up-er myself).

The idea of cognitive dimensions seems like an interesting viewpoint, but like all theories or models in social science, it's a bit fluffy, can be applied to anything, and the important thing isn't which one you use (although it needs to be fashionable), but how good you are at pointing out interesting stuff in light of it.

Noel Welsh - Re: Evaluating a new programming language  blueArrow
3/18/2002; 1:42:46 PM (reads: 996, responses: 0)
Interesting research. It will certainly but the old `my language is better than yours' bunfight on firmer grounds. (Actually it'll probably just fuel more arguments).

As others have said the sample size is too small to draw any solid conclusion. My take home conclusions:

- There seemed to be a few problems that were environment not prog. lang. related. Give 'em Emacs, I say!

- Gosh C# is complex! Inherits/overrides/default super classes etc! If there weren't so many concepts and special cases I doubt there'd be so many problems. I'd like so see programmers going from Common Lisp, say, to Scheme or maybe SML to Haskell (something where the difference in syntax is comparable to the jump from C++ to C#).

Steve Merrick - Re: Evaluating a new programming language  blueArrow
3/20/2002; 1:02:16 AM (reads: 946, responses: 1)
"But at least it is a bit more scientific than introspection..."

You say this as though it's automatically a Good Thing...

Ehud Lamm - Re: Evaluating a new programming language  blueArrow
3/20/2002; 1:18:12 AM (reads: 993, responses: 0)
Actually, that's not what I think in this particular case.