School of Computer Science THE UNIVERSITY OF BIRMINGHAM Ghost Machine

THIS DOCUMENT IS A PLACE-HOLDER - TO BE EXPANDED LATER

The relevance of explanations of possibilities
to assessing competences
(DRAFT: Liable to change)

Place-holder text

Aaron Sloman
School of Computer Science, University of Birmingham
(Philosopher in a Computer Science department)

Installed: 18 Oct 2015
Last updated: 19 Oct 2015

Can cognitive competences be assessed reliably?
A partial answer based on explanations of possibilities

Place-holder text

Karl Popper is famous for, among many other things, proposing falsifiability as a criterion for a theory to have scientific content. On that criterion a theory claiming only that certain types of objects, processes, states of affairs are possible, without providing a basis for predicting when such possibilities would be realised, could not be falsified, and would therefore not be a scientific theory Popper (1934). So Darwin's theory of natural selection would not be a scientific theory.

It is not widely known by admirers or critics of Popper that he later adopted a more flexible approach to the falsifiability requirement and came to express great enthusiasm for Darwin's ideas Popper (1978), Sonleitner(1986).

In Chapter 2 of my 1978 book The Computer Revolution in Philosophy I claimed that an important type of advance in science could be a theory explaining "How X is possible" for some X, even if the theory did not provide a basis for predicting when instances of X would occur, and was not empirically falsifiable. One of the examples I gave was

"The theory of natural selection explained how it was possible for undirected ('random') mutations to lead to apparently purposive or goal-directed changes in biological species. The theory of genes explained how it was possible for offspring to inherit some but not all of the characteristics of each parent, and for different siblings to inherit different combinations. (Section 2.5.2)

The importance of theories that explain "How X is possible" for some X is further defended in
http://www.cs.bham.ac.uk/research/projects/cogaff/misc/explaining-possibility.html, a paper responding to some criticisms of Chapter 2.

As a result of a recent email exchange with Jose Hernandez-Orallo I realised that the notion of explanations of possibilities could have important applications in designs for methods of assessing competences. This is because the criteria for evaluating theories (or research programmes in the sense of Lakatos(1980)) have some important implications.

Section 2.5.8 of Sloman (1978) states:

.... If unrefutable theories are to be dubbed 'metaphysical', then what I am saying is that even important scientific theories have a metaphysical component, and that the precision, generality, fine structure, non-circularity, rigour, plausibility, economy and heuristic power are among the objective criteria by which scientific and metaphysical theories are in fact often assessed (and should be assessed)....
This implies that although theories explaining how various things are possible cannot be falsified empirically they can nevertheless be compared objectively e.g. in (These features have to be assessed over time. The history of physics shows that a research programme that seems to be progressing turns out to be degenerating and another that appeared to be degenerating starts making more progress.)

Such an explanatory theory (e.g. a theory of how human mathematical competences are possible) does not yield predictions regarding who will acquire or demonstrate those competences and under what conditions. Moreover observations of such competences at work does not prove the theory correct. The fact that someone is observed to have made a discovery in geometry proves that making such discoveries is possible, and therefore that there is something to be explained by a theory of how such discoveries are possible. But such a theory may merely demonstrate that if various types of information-processing mechanism and forms of information are assembled then that will make possible certain sorts of mathematical discoveries. But if not enough is known about how to predict when such a mechanism will produce results, then it may be impossible to falsify the theory.

Nevertheless such a theory could have practical utility insofar as it provides the basis of a mode of testing individuals for possession of such competences, as follows:

Suppose individuals P1, P2, P3, ... in some group of people (or animals) appear to have a collection of competences C1, C2, C3, .... as demonstrated by their exercise of those competences, and other individuals in the group show no evidence of possessing all those competences when tested, and there is no explanation of their non-performance, such as illness, lack of interest, dislike of the tester, etc.

Suppose T is the best current theory about what makes those competences C1, C2, etc. possible, where T presents a collection of mechanisms (e.g. information processing mechanisms, including forms of representation used, ontologies used, methods of inference, forms of perception, forms of problem-solving, grammars used, etc.)

If we have no other good rival theory explaining the competences demonstrated by individuals P1, P2, ... then we can say that T is the best currently available explanation of how those individuals do what they do.

But if T is an interesting theory with generative power it will typically explain the possibility of far more competences than those demonstrated in the tests, i.e. it can be shown also to explain the possibility of other, related, competences C4,C5,C6...

E.g. the best available explanation of how individuals P1, P2, solve various number-theoretic problems unaided could be that they have something like a grasp of Peano's axioms, including the induction axiom, and various practical skills in the use of those axioms. In that case, we may be able to show that those skills and conceptual achievements can be used to solve many more problems in number theory. This does not imply that everyone who has that combination of abilities will necessarily be able to solve those problems, since there can be many performance errors and lapses, as Chomsky emphasised in the context of linguistic competences).

In that sort of situation, a clever tester will be able to use the explanatory theory to generate sets of tests such that the ability to pass them cannot be explained by any other known mechanism associated with a good (progressing) research programme. (Use of successful random guessing is always a possible alternative explanation for success, but that explanation is inferior according to the criteria listed above, e.g. it gives no basis for specifying which test behaviours will and which will not occur. Another possible alternative explanation is that the candidate has previously encountered the tests along with answers and has memorised tests and answers. If the competence in question is rich and deep examiners will be able to generate tests that are unlikely to have been generated previously, making it unlikely that the candidate has memorised the answers. Of course, various sorts of cheating, e.g. using miniaturized radio equipment to communicate with an external expert, cannot be conclusively eliminated.)

I suspect that good designers of tests (especially designers of mathematical examinations for advanced students) always have in mind an explicit or implicit theory of what sort of mechanism makes it possible to succeed in those tests. A really good, deep, broad, theory can be used systematically to generate diverse tests that help to rule out alternative explanations of success.

If this is right, then one of the corollaries seems to be that instead of such tests merely producing numerical grades they can produce summaries of mechanisms/competences exhibited by those tested: a potentially far more useful outcome than a grade.

Of course, there are many academic subjects whose practitioners know nothing about cognitive mechanisms or computation, and they may be unable to use the strategy described here. However, there is good reason to believe that gifted teachers have good "implicit" theories related to what they are teaching, and therefore may be able to devise good tests even though they don't have deep theories concerning what they are doing.

NOTE:
Performance in such tests can have positive implications, but cannot have negative implications. The fact that someone tested displays a collection of abilities of the sorts tested for gives strong evidence that that individual has acquired the required mechanisms, but non-performance (e.g. non-answers, or incorrect answers) always leaves open the possibility that something other than lack of competence is to blame. Imaginative readers with a deep understanding of how human minds can vary, or how states of mind in an individual can vary, will be able to think up varied alternative explanations.

TO BE CONTINUED

This paper is
http://www.cs.bham.ac.uk/research/projects/cogaff/misc/assessing-competences.html
A PDF version may be added later.

A partial index of discussion notes is in
http://www.cs.bham.ac.uk/research/projects/cogaff/misc/AREADME.html

REFERENCES AND LINKS

Maintained by Aaron Sloman
School of Computer Science
The University of Birmingham