Testing first year programming students

Saeed Dehnadi and Richard Bornat describe a test for programming aptitude:

Learning to program is notoriously difficult. A substantial minority of students fails in every introductory programming course in every UK university. Despite heroic academic effort, the proportion has increased rather than decreased over the years. Despite a great deal of research into teaching methods and student responses, we have no idea of the cause.

It has long been suspected that some people have a natural aptitude for programming, but until now there has been no psychological test which could detect it. Programming ability is not known to be correlated with age, with sex, or with educational attainment; nor has it been found to be correlated with any of the aptitudes measured in conventional intelligence or problem-solving-ability tests.

We have found a test for programming aptitude, of which we give details. We can predict success or failure even before students have had any contact with any programming language with very high accuracy, and by testing with the same instrument after a few weeks of exposure, with extreme accuracy. We present experimental evidence to support our claim. We point out that programming teaching is useless for those who are bound to fail and pointless for those who are certain to succeed.

Based on their experience, there are essentially three groups in an introductory programming class, those who want to learn more faster, those who manage to pass and those who still have no idea what programming is all about once the course is completed. It seems it is not the teachers fault either,or as they put it:

The cause isn't to be found in inappropriate teaching materials or methods either. Essentially, the computer science community has tried everything (see section 2) and nothing works. Graphics, artificial intelligence, logic programming languages, OOP, C, C++, PROLOG, Miranda: you name it, we've tried it. We've tried conventional teaching, lab-based learning by discovery and remedial classes. We've tried enthusiasm and cold-eyed logical clarity. Nothing makes a difference. Even the UK's attempt in the 1980s to teach the whole UK population to program on the BBC Micro ran into the sand.

Food for thought.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Semantic Barriers

For what it is worth I have long associated programming with skills like code cracking, puzzle solving, pattern recognition (as in one part of an IQ test), and Sherloch Holms type mystery solving (ie abduction). This would seem to be consistent with the notion that code is meaningless and the job of the programmer is to find meaning or to put meaning into the code. It is a game that many people refuse to play, no amount of explanation or encouragement will work.

To add yet one more unproven theory to the soup, I would argue that "people" are not naturally logical in the sense of formal logic. Normal thinking is activity oriented . It is better described as synthetic logic with "works", "doesn't work" semantics. This is strongly related also to american pragmatism, and the forgotten subject of Cybernetics. Thinking of programs this way is more natural and should be easier to learn.
Edit: see also Scandinavian Activity Theory , A competing article.

Notions of intelligence

Notions of intelligence change with cultural context. From well before the Classical period to well after the Renaissance, for example, memory was much more highly emphasized than it is now -- but the emphasis changed gradually, from correct memorization of stories to thorough memory of the canon, to be drawn from at need.

Will our intelligence be consistency? Memory lost out long ago. Cleverness comes under ever more fire, year after year. Wit, alas, does not even cut it in politics any more.

This study is drivel

Besides the obvious statistical problems like woefully inadequate sample size, the test reflects more on its makers than the students. I answered this once before when it was posted on reddit months ago so I'll quote myself here.

"What this paper does do is pinpoint a specific area of misconception that many students coming into programming have. But the authors then go on to propose that it's a litmus test for telling those who 'can' from those who 'can't'. They mention in passing that they 'might' get better results by, surprise, trying to correct this in student's minds.

"THAT'S THE WHOLE POINT. That shouldn't be some minor sentence burried deep in the paper. The paper spends paragraphs talking about how all attempts at improved programming education have failed. It also points out this precise thing that if students have wrong hurts them. But the authors completely fail to realize then that the reason all previous attempts have failed is because they haven't targeted this precise thing, the mental model for assignment/sequence, not because of some inherent property of students."

This might be particularly interesting to LtU readers, because they are basically faulting students for not intuitively grasping the imperative programming model. What if they used a functional test?

"I have seen countless numbers of teachers tell students to just "think a little more" about a problem when it is quite clear they've given up on the student being able to solve a problem. They convince themselves that the student is incapable of understanding because of their own inability to explain. The authors state that teaching the consistent group is "easier." Well, yeah, if you take the group that already has the correct mental model coming into the course, yes, things are "easier." Easier in the sense that you are no longer functioning as a teacher but instead as a text-to-speech reader for a Java tutorial."

I think the paper is not only wrong but pedagogically irresponsible. One of the things students could potentially learn from programming is the notion of being able to deduce behavior if one assumes that symbols have a consistent meaning (the development of this expectation is one of the reasons I've found picking up new programming languages easier as I've learned more of them). Just taking the students who have already had this revelation defeats the purpose of education.

Anecdotal evidence

Having taught a few first year programming classes, I had believed that
some students had "it", and some didn't. But then I had one student (about 20 years ago) that obviously didn't have "it", who spent a few days really trying to do her assembly language programming assignment, and eventually got "it". After that, she seemed to me to belong to the "have it" group. That experience indicated to me that it was possible to learn this so called natural aptitude for programming.


I had the exact same experience. One of my students just couldn't get his head around the concept of a loop. In an assembly language programming assignment, I overheard him suddenly telling his fellow student in the group "..., but that jump is part of the loop!" He had obviously finally understood it, and he passed my class with a very decent grade.

This experience made it clear to me that anyone can learn "it", and that it is essential to teach some machine-code for first-year students, even if only in a very simplified way.

What is a computer?

The paper seems to say that everything has been tried to teach programming and nothing works. I wonder if this is true. For some reason I get the feeling that many "students" do not "get" or understand the basic idea of a computer independent of the language plus machine system we now use. Suppose we first introduce students to computers and computing in general and work into programming gradually in small steps.

There are many ways to do this. Here are a few suggestions. Create slide rule or graphic computers on paper. Try computing with logarithms. Try paper spreadsheets and introduce computer spreadsheets. Introduce spreadsheet macros and simple DSL's that illustrate basic ideas such as loops, substitution, logic, etc. but without coding. Introduce systems analysis drawings for simple algorithms. I can't believe this has not been tried. What is wrong with this approach?



Slight disagreement

Even the UK's attempt in the 1980s to teach the whole UK population to program on the BBC Micro ran into the sand.

That's strange. I see the fruits of the BBC's efforts everywhere. I find British people well represented in computing related companies around the world and many of these people got their boost from learning on the BBC Micro. The BBC failed to get *everyone* programming, but it helped produce a large number of highly employable people. What's more, from a programming language perspective it raised the bar on home micros, introducing people to 'structured programming', relatively painless recursion (compared to GOSUBs), typed pointers and an integrated assembler that made it easy to implement things like compilers.

We can only hope the UK will

We can only hope the UK will attempt to teach the whole UK population to write Haskell...

This paper has been

This paper has been discussed here previously.


I forgot it showed up on LtU around the same time as the reddit article and I had already posted my rant here :)


The paper has been retracted.