Aaron Sloman is Honorary Professor of Artificial Intelligence and Cognitive Science at the University of Birmingham. Here he reflects on what is known about learning, and on the difficulties that exist in understanding it. These reflections were prompted by watching remotely part of the panel presentations at the 23 May launch of the Ufi Charitable Trust.

Aaron had participated as an interviewee for the trust’s study Scaling up – achieving a breakthrough in adult learning with technology.

Aaron Sloman will be an invited speaker at ALT-C 2012.
(See the video: http://www.youtube.com/watch?v=QXAFz3L2Qpo)

Is education research a form of alchemy?

Alchemists did masses of data collection, seeking correlations. In the process they learnt a great many useful facts – but lacked deep explanations. Searching for correlations can produce results of limited significance when studying processes with an underlying basis of mechanisms with astronomical generative power. But this correlation-seeking approach characterises much educational research.

Accelerated progress in chemistry came from developing a deep explanatory theory about the hidden structure of matter and the processes such structure could support (atoms, subatomic particles, valence, constraints on chemical reactions, etc.). Thus deep research requires (among other things) the ability to invent powerful explanatory mechanisms, often referring to unobservables.

My experience of researchers in education, psychology, social science and similar fields is that the vast majority of the ones I have encountered have had no experience of building, testing, and debugging, deep explanatory models of any working system. So their education does not equip them for a scientific study of education, a process that depends crucially on the operations of the most sophisticated information processing engines on the planet, many important features of which are still unknown.

Learning and teaching are a bit like life. As every gardener knows, almost every generalisation about life is false, because there are so many forms of life, showing enormous variation. There may not be quite so many forms of learning (and failure to learn) but the underlying information processing systems have great internal complexity and diversity (even within one individual, between infancy and grand-parenthood) and they can interact in very different ways with learning materials, the everyday environment, laboratory equipment, peers, teachers, books, computers, the internet, family, physical disability, etc.

I suspect that as a result, most generalisations about learning (e.g. what works, and what doesn’t) are false, unless very heavily qualified. For example, if research shows that a particular method of teaching children to read works much better than all other known methods for 80% of the under-seven population, that does not tell us why, or what to do about the other 20%, or whether for some or all of the 80% there might be an even better way. A deep understanding of the computational mechanisms of both reading and learning to read might provide a basis for teaching methods that are more sensitive to the diversity of child minds, home environments, brain development, and potential.[*] An implication is that we still need deep research into mechanisms of learning (of which our current knowledge is still limited — even in the most advanced AI/robotics and neuroscience laboratories) and research into the diversity of processes that can occur when those mechanisms interact with a wide range of learning environments, tasks, external and internal motivations, and social contexts. This also needs to take account of the fact that the mechanisms themselves can be very different in different individuals, since each learner is a product of a complex developmental process with many genetic and other influences, not a mass-produced instance of a common human genome.

[*] Note added 30 Jan 2014
This is closely related to criticisms of research that has been used to support rigid teaching and assessment of phonics-based reading. See
Andrew Davis
A Monstrous Regimen of Synthetic Phonics: Fantasies of Research-Based Teaching `Methods' Versus Real Teaching
in Journal of Philosophy of Education, Vol. 46, No. 4, 2012, pp 560--573.
https://www.dur.ac.uk/education/staff/profile/?mode=pdetail&id=617&sid=617&pdetail=82425

I also take up this critique of bad use of research about reading in a presentation on "What is computational thinking? Who needs it? Why? How can it be learnt? (Can it be taught?), here:
http://www.cs.bham.ac.uk/research/projects/cogaff/talks/#talk105

We need a powerful generative theory to provide the context for data collection. (Compare research in chemistry.) Data collection without a powerful theory can occasionally produce gold, but mostly it’s like shooting in the dark, and any correlations found will have unknown scope. Moreover, they will tell you nothing about what might have been achieved by trying a different technique you haven’t thought of.

When I was teaching (first at Sussex University, then at Birmingham), along with a bunch of high-powered and highly creative colleagues with diverse talents, we discovered that for a range of students, learning about programming, artificial intelligence (AI), cognitive science, logic, and linguistics, it was possible to develop interactive learning environments that allowed different students to drive the systems in different directions at different speeds with lots of readily available support (documentation, libraries, language extensions, suggested problems) for a wide range of learning trajectories.

So, as in kindergarten, the students learnt different things, but many of them also learnt how to learn.

They learnt at very different speeds — some with an initial long plateau before taking off, others flying from the start, and a small subset for whom the approach did not work at all, even with hours of individual tuition. I think some kinds of learning (not yet specified precisely enough) require short-term memory mechanisms that, for unknown reasons, are either lacking or seriously underdeveloped in a small subset of the population [i].

Some of those students developed strategies for getting high marks in other courses on the basis of very shallow understanding. When we expanded to teaching a MSc conversion course, we found highly intelligent people whose brains had been seriously handicapped by learning bad things. For example some of the people who had become expert Fortran programmers found it harder than young children to learn logic programming.

There’s still a huge amount to be done to:

1. transplant those techniques onto modern technology,
2. expand the content to fit a wider range of subjects,
3. educate teachers to understand enough about the mechanisms, and about the diversity of learners, to be able build the extensions required to support the kind of content they want to teach, and to
4. find and repair gaps and inadequacies in the theories and the mechanisms developed.

Our teaching/learning environments were based on a partial theory about the processes of learning — partly inspired by progress and problems in AI. The theory presupposed that the process of learning is in some cases partly like the process of programming, including similar stages of implementing ideas, testing them, debugging them, and extending them, most of which learners do unconsciously. This is very different from a theory that learners develop networks of associations, for example [ii].

A much deeper theory is required. No doubt there are different approaches to teaching programming that can produce even better results, using the same cognitive mechanisms in learners: humans evolved learning capabilities that could be made to work in a huge variety of environments, as shown by toddlers learning to use things nowadays that none of their ancestors had ever encountered. (Of course, humans don’t all start the same.)

The teachers also need help acquiring a deep understanding of the diversity of information-processing systems found in different learners and how to accommodate that diversity in their teaching.

A major challenge is that this is all totally alien to a world with fixed syllabus structures, numerical marking schemes, and league tables.

What next?

These are not the theories of a specialist researcher in education or educational technology. My own work over the last 50 years or so has been concerned with attempting to combine ideas and methods from research in philosophy (especially philosophy of mathematics), artificial intelligence, robotics, psychology, neuroscience, evolution, animal cognition, software engineering and computer science. Some ideas derived from that work are presented in a disorganised collection of materials still under development here http://www.cs.bham.ac.uk/research/projects/cogaff/misc/AREADME.html.

I think the theories sketched in Annette Karmiloff-Smith’s 1992 book "Beyond Modularity: A Developmental Perspective on Cognitive Science" will prove crucial for educational theory, though they require further development in order to be expressed computationally.

Progress has been slow because the products of millions of years of biological evolution interacting with very varied and very complex physical, biological, and social environments are too complex to be fully understood using current theories and research methods.

But the new wave of enthusiasm to teach our children to think computationally, starting in primary school with new powerful learning environments, may, over the next few decades, produce a new wave of scientists and teachers whose insight into multifarious processes of learning and teaching, will be deeper than anything we have at present, especially if they learn to do “thinky” programming, in the sense explained in http://www.cs.bham.ac.uk/research/projects/poplog/examples/

It may turn out that human minds are too complex to be understood by human minds. But we should at least try.

[i] Some research that can be interpreted in this light is: http://www.eis.mdx.ac.uk/research/PhDArea/saeed/

[ii] Some of these ideas were presented long ago in Gerald J. Sussman, A computational model of skill acquisition, 1975, American Elsevier, San Francisco, CA, http://dspace.mit.edu/handle/1721.1/6894

NOTES (Added 3 Aug 2014)
This is a slightly modified version. The original version of this article was published here:
https://newsletter.alt.ac.uk/2012/06/is-education-research-a-form-of-alchemy/

Also relevant:
http://www.cs.bham.ac.uk/research/projects/cogaff/misc/black-box-tests.html
A paper showing that IF a system has at least the power of a Turing machine (as brains do in the
sense described by Noam Chomsky as "infinite competence" contrasted with "finite performance
limitations"), THEN no finite amount of behavioural testing (examining relations between inputs
and outputs, can be a basis for statistical reasoning that a law about the system has been identified.