A member of the CAS group wrote:

> For the last three years I have used the 'Mental Models' programming
> aptitude test, giving the test in the first or second lessons of a new
> AS level course.
>
> Rather to my surprise I have found it extremely accurate in
> identifying students who "won't make it" - students who, despite the
> right motivation and masses of personality, can't handle the demands
> of programming, even at the novice level.

When I first learnt about the test from one of the authors, Richard Bornat,
several years ago, it rang bells because I had previously found that there was a
subset of students whom I seemed to be unable to teach programming, even though
they were highly motivated, worked very hard, and even sat with me in one to one
sessions trying to get over their hurdles.

Examining their problems closely led me to the hypothesis that a subset of
students lack a particular class of short term memory mechanisms required for
advanced programming, even if they are very good at reading books and papers in
other disciplines, writing essays and getting very good marks and in some cases
doing very simple programming tasks.

There are some types of programming (and I suspect also certain branches of
mathematics, and also musical composition, and poetry composition) that require
the ability to construct quite complex interlocking structures in short term
memory, save them, create new structures and then perform operations on the old
and new structures involving changing relationships.

An example might be building two parse trees then noticing the kind of
transformation that will convert the first to the second. I have deliberately
chosen a fairly extreme case, to illustrate the type of thing. There are
probably more complex cases and also simpler cases.

I have no idea whether the students who seemed to be unable to do that sort of
thing (even with the help of diagrams and prompts from me) had an innate
cognitive deficiency or whether then had not been given the opportunity to
develop because of some aspect of their environment, or whether they had the
potential but required highly specialised teaching or training of a type that I
could not offer them.

Anyhow, it seems clear that that's something like what the test is probing, by
requiring students to invent a syntax and a semantics for some inherently
ambiguous symbolic structures then use what they have invented to solve a
problem. It's intrinsic to the test that there is no single correct solution.
And it is totally different from the task of interpreting some correct code in a
conventional programming language.

Anyone who looks at it as if it were a simple programming task or program
description task will probably completely misunderstand the point of the test,
so I suspect that it cannot be used for students who have already learnt some
programming -- e.g. because whatever instructions are given, they will
misunderstand the test and try to map the structures onto some programming
language they already know that it doesn't fit and so fail the test, even if in
fact they have the abilities being tested for.

Sometimes misleading prior knowledge can interfere with finding a solution to a
problem.

> As with all such tests, there's a danger that the result becomes self-
> fulfilling - leading a teacher to concentrate effort on those who do
> well and neglect those who do poorly, but with that health warning, it
> is certainly worth a look.

If what I've understood about it is correct, then a prior test may be required
for detecting students who already have knowledge that will prevent them doing
the test well even if they have the ability being tested for. Or in some cases
they may be able blindly to apply some prior knowledge that happens to work, but
not because they have worked out a solution themselves. In either case the test
will give a misleading result -- false negative or false positive.

> The test with marking guide is here:
>
> http://www.eis.mdx.ac.uk/research/PhDArea/saeed/
>
> It needs to be given early, before you have formally covered assignment.

But even if students have not been taught assignment they may have come
across similar ideas that interfere with the functioning of the test.

I recommend anyone interested to read the paper before forming any
opinions about the test. It's a long time since I read it, so what I've
written may be misleading in some details.
Long term memory can also be faulty!

Aaron
http://www.cs.bham.ac.uk/~axs/

Maintained by Aaron Sloman
School of Computer Science
The University of Birmingham