Home » forums » LtU Forum

Predicting programming course success from a single entrance exam

An simple test that is a good predictor of whether a new student with no programming background will pass a progamming course. The test categorises students into three bins - Students that:

  • are consistent (those that answer questions according to an internal mental model)
  • are inconsistent
  • answer no questions (blank)

There is a good correlation between the consistent answerers and their final exam pass rate:

Consistent(A1)Inconsistent/Blank (A1)
Pass218
Fail626

The paper The camel has two humps and the homepage of one of the authors have more information. Links found from Hack the planet.

The paper (and the reasoning) seem wishy-washy to me - but the correlation is impressive.

If their premise is correct, I wonder if similar tests could be constructed to help sieve for functional programmers, engineers, or any other logical tasks?

By Morris Johns at 2006-07-15 06:30 | LtU Forum | previous forum topic | next forum topic | other blogs | 12221 reads

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

I think the conclusions are bunk

This was my response from when the article showed up on reddit.com:

I think the author's of this paper and several of the posters here completely miss the boat.

What this paper does do is pinpoint a specific area of misconception that many students coming into programming have. But the authors then go on to propose that it's a litmus test for telling those who 'can' from those who 'can't'. They mention in passing that they 'might' get better results by, surprise, trying to correct this in student's minds.

THAT'S THE WHOLE POINT. That shouldn't be some minor sentence burried deep in the paper. The paper spends paragraphs talking about how all attempts at improved programming education have failed. It also points out this precise thing that if students have wrong hurts them. But the authors completely fail to realize then that the reason all previous attempts have failed is because they haven't targetted this precise thing, the mental model for assignment/sequence, not because of some inherent property of students.

Commenters above posit that that engineering and comp sci schools are just for weeding people out. They are mostly correct. That's not because the students suck. That's because the type of people who succeed in these areas tend to have the communicative skills of a gerbil. Even when they have great command of the English language (which is a small subset in itself), engineering types are often at a total loss when it comes to seeing a problem from a non-hyper-logical perspective and pinpointing what's wrong with a student's reasoning, i.e., empathizing, demonstrating EQ not just IQ.

I have seen countless numbers of teachers tell students to just "think a little more" about a problem when it is quite clear they've given up on the student being able to solve a problem. They convince themselves that the student is incapable of understanding because of their own inability to explain. The authors state that teaching the consistent group is "easier." Well, yeah, if you take the group that already has the correct mental model coming into the course, yes, things are "easier." Easier in the sense that you are no longer functioning as a teacher but instead as a text-to-speech reader for a Java tutorial.

Overall, I'm glad the paper pinpointed a way of testing for a common source of errors coming into the course. But on the other hand I think their conclusions are flat out wrong.

I am entirely self taught. I code C++ now as a part time job, in and of itself an incredibly obfuscated language, but I also make use of things most C++ programmers find too esoteric, the boost libraries, heavy template usage, etc. I harbor considerably more skill than the hordes of UML diagram drawing software engineers out there. I am also fairly certain that I would have failed the test when I started, probably being in the 'inconsistent' group. I got over the hurdles by thinking about them over and over over long periods of time. A teacher who would have told me that "binary is just another way of writing the same numbers" or that "computers follow everything literally and always the same way" and demonstrated such probably would have cut 2 years off my learning time.

By dataangel at Sat, 2006-07-15 07:55 | login or register to post comments

"Correct" mental model?

Well, yeah, if you take the group that already has the correct mental model coming into the course, yes, things are "easier."

The consistent set needn't have had the correct mental model. Students who answered all questions using the same model -- '=' is a comparison, '=' is left-to-right assignment, etc. -- did markedly better on the final than those who where inconsistent, using different meanings for the same operator.

Edit: I see now that what you might mean by "correct mental model" is the idea that computers are naive machines. This was said on the first day of my introduction course (C211 at IU, taught by George Springer); around half the class failed nonetheless.

I would guess that those students who remained inconsistent heard the comment, but failed to understand its meaning. You are correct, however, that the authors are mum about whether or not machine- and language-models were part of the course. The latter was in C211.

By Michael Greenberg at Sat, 2006-07-15 23:53 | login or register to post comments

Well, thinking to have a

Well, thinking to have a consistent model is in itself a mental model. Guessing that the rules governing the symbols between examples probably stay consistent and then deducing that you probably have the best chance of getting everything right if you stay consistent is in itself an approach some students came in with a priori. People who have more of a 'voodoo' feeling about mathematics and computers won't expect things to stay consistent in any way.

Mentioning that computers are "naive machines" and moving on with the rest of the lecture is not enough. I could definitely see a teacher stating that, and still having students trying to give their variables more descriptive names to convey intent. That they are naive has to be properly illustrated in order for it to mean anything to students. I think a better word than naive would be "stupid."

If I were teaching an introductory programming course I'd have students write down instructions on how to perform some common task, like make an omellete or walk to the door, and then follow their instructions absolutely literally (cracking two eggs onto the floor or plowing through desks and chairs). Then closer to real practice examples could be introduced, "Is it possible to divide by zero?" "No," and then run a program that asks for user input, have the student enter 0, and watch the floating point exception occur.

And I think it'd be best to use a learning language with no syntactic sugar to start with, to hit the point home. When I was learning programming I'd often expect consistency and have it hurt me -- try deducing the meaning of the * operator in C. It has 3 different meanings. (declare pointer, dereference, multiply)

By dataangel at Sun, 2006-07-16 02:39 | login or register to post comments

Different meanings...

Maybe you have the correct mental picture but your wording here still smacks of the wrong model in that the * "operator" does not have 3 different meanings: rather, there are are 3 different operators sharing the same glyph. Or, to put it differently, * is not an operator, merely a glyph, and as such yes it is overloaded.

By Olivier Lefevre at Tue, 2006-07-18 21:36 | login or register to post comments

people are different

You honestly believe everybody is equally able to become a programmer? Or an engineer, a physician, a historian, an archaeologist, a carpenter, a butcher, ...?

I find this an inhuman view. People are different. We should search for ways to quickly find out what will benefit them and society the most instead of devising ways to shoehorn all people into the same template. The former is the premise of this paper, and I commend the authors for following through with it, despite it being considered politically incorrect.

By Udo Stenzel at Sun, 2006-07-16 11:35 | login or register to post comments

Uh...

That would be an interesting point if it had anything to do with what I said.

I didn't say that I thought all human beings were of equal ability in all tasks. I'm saying that what these professors would like to attribute to a talent gap what is actually a product of pedagoligical shortcoming. A big part of the role of a teacher is supposed to be identifying misconceptions that students have so that they can be corrected. In the paper, they have attempted to do the identification part. But then they neglect the correction part, simply preferring to take the easier route of only teaching students with a correct conception coming into the class.

And as far as natural talent gaps go, frankly I am much more pessimistic about computer scientist's ability to empathize than I am with the average person's ability to code.

By dataangel at Sun, 2006-07-16 19:41 | login or register to post comments

61 Subjects?

61 Subjects seems like a depressingly small sample size. It seems arrogant to assume a class of people is fundamentally incapable of learning a given discipline.

The paper also suggested that the language used to teach has no bearing on the student's learning, and yet the test they propose intentionally relies on Java's conflation of assignment with the mathematical symbol for equality. How might the same test have gone with Pascal's ':=' assignment operator or the completely foreign Scheme notation?

By Fred LeMaster at Sat, 2006-07-15 22:13 | login or register to post comments

Actually it seems the

Actually it seems the confusion between assignment and equality wasn't planned, but rather an accident of the poorly planned experiment. It still turned out to be a remarkably good predictor.

Forget any conclusions, they are ill-supported with that small a sample and without more studies. The high correlation between "being consistent" and "passing the course" still means that these guys are on to something.

Forget the conclusions anyway. I still want more papers written in this style! Gave me a couple of laughs, never mind if it's correct.

By Udo Stenzel at Sun, 2006-07-16 11:28 | login or register to post comments

What about later?

The research is interesting and corroberation would be nice but most CS programs have 2-3 cullings and intro is only the first one. While the notion of being able to take an aptitude test rather then sit through a survey for exploration seems attractive if there is no correlation at the later hurdles the value would be no more than saving some college students some money.

By noah healy at Sun, 2006-07-16 01:50 | login or register to post comments

Consistent group needs subdividing

They pose questions along the lines: 

int a = 10;
int b = 20;

a = b

What are the changed values of a and b?
Some-one whose mental model is that a = b is a left to right move and empty is represented by zero answers
a = 0, b = 10
If they follow that through consistently they are in the consistent group.

Look at the company they keep. They are lumped in with those who correctly guess what JAVA does:
a = 20, b = 20
The tests at the end of the course don't seem to be an awful lot harder. It would be appalling if students who guessed the right answer consistently, whose intuitions were aligned with the language they were about to be taught, were messed up by the course and ended up failing.

The paper emphasises the current absense of effect aptitude tests. This is presumably a comment on the absense of double sided tests able to predict both success and failure. It seems obvious that testing students to see if they belong to the consistently correct subgroup of the consistent group is going to be a one-sded test for success. Pass and you will succeed. Fail and the test does not predict. It also seems obvious that such an aptitude test makes a mockery of a university level course - we can claim little credit as educators if we only admit those who can guess much of what is to be taught before they have attended any lectures.

The interest of the observations lies in the success of the consistent but incorrect group constrasted with the failure of the inconsistent group. The suggestion seems to be that those with an incorrect mental model fix it within the several-week timescale of the course, but those without a mental model do not build one within that timescale.

The authors have made a potentially very interesting observation, but until they split up the consistent group into correct and incorrect none of us know the quantitative detail.

By Alan Crowe at Mon, 2006-07-17 16:08 | login or register to post comments

Easy to reproduce

I suspect it would be fairly easy to reproduce these results---give students the final exam coming in the door at the beginning of the semester and again going back out at the end, and don't teach them anything in the middle.

Aside from a sad attempt at humor, why do I think that is relevant? Statements like the following:

There was an attempt to administer the test to another 30 or so students on a foundation (predegree) programming course at Middlesex University. That administration failed, because the students – rightly, in our opinion – were incensed at the conduct of their teaching and the arrangements for their study, and simply refused to do anything that wasn’t directly beneficial to themselves.

and

Another group had very much lower than average A-level scores, and had been admitted to a maths and computing course largely to boost numbers in the mathematics department: we call this group the low achievers.

In short, I suspect there's something wrong in Middlesex.

Further, if the author's goals were to avoid multiple humps in the their grade graphs, I would have to suggest that they have not succeeded; see Figure 16, Figure 13, and Figure 10. They may be on to something with the consistency criteria, but I would be very hesitant to associate that with "natural aptitude".

By Tommy McGuire at Mon, 2006-07-17 20:53 | login or register to post comments