Impossible Objects

Key points:
Mathematical cognition is primarily about understanding necessities and impossibilities, recognized through understanding structural relationships
NOT
regularities and probabilities derived by collecting statistical evidence.
So it cannot be based on use of neural nets.

E.g. if spatial volume V1 contains V2, and V2 contains V3,
then V1 must contain V3.
I.e. V1 contains V2, and V2 contains V3 and V1 does not contain V3 is impossible.
Similarly 3+5 must equal 8. It cannot equal 9.
(Immanuel Kant pointed out this feature of mathematics in Kant(1781).)

Surprisingly many neuroscientists, psychologists, philosophers and AI researchers are blind to this limitation of neural nets, including some who are much admired for research on mathematical cognition! Perhaps they and their admirers are simply blind to key features of mathematical cognition pointed out by Kant.

Euclid's Elements includes many well known examples but misguided 20th century educational decisions deprived hordes of intelligent learners of personal experience of discovering geometrical constructions, proofs and refutations.

Current AI vision systems and reasoning systems lack the sorts of mathematical qualia experienced by Euclid, Archimedes, and many of their contemporaries, predecessors and successors! This document is part of a project that aims to understand requirements for removing that gap in AI, inspired in part by my mathematics teacher (Mr Adendorff?) at school in Cape Town many years ago, where I first studied Euclidean geometry and, like many others at that time, and in preceding centuries, learnt to find geometric constructions and proofs (before going to university in 1954). Later inspiration came from (among others) Immanuel Kant, James Gibson, Oscar Reutersvard, whose triangle made of cubes floating in space is depicted below, and especially Max Clowes (1933-1981), who introduced me to AI work on scene analysis around 1969, including attempts to model perception of impossibility. See Appendix 2 of this memorial tribute. Roger Penrose has thought and written about such matters including the well known Penrose impossible triangle also discussed below. Our analyses are very different, however! At present I don't think anyone knows how ancient brains represented and detected impossibility, or how that ability evolved. I have also been inspired by the art of Escher.

Note added 9 Mar 2021
Thinking about what must happen inside an egg to produce a hatchling (baby chicken, duckling, crocodile, etc.) has led me to the conclusion that sub-neural biochemical mechanisms (e.g. mechanisms producing a chick that hatches with important perceptual and other cognitive competences available immediately) must be an important part of the answer. Mechanisms using interconnected neurons (neural networks) may provide additional functionality some time after hatching.

Note: this paper has grown too long and messy. I hope some day to reorganise the material. Perhaps some kind person will do it first!

NOTE: This is work in progress
This is part of the Turing-inspired Meta-Morphogenesis project, concerned with identifying and explaining the many transitions in types of information-processing in the course of biological evolution on Earth: http://www.cs.bham.ac.uk/research/projects/cogaff/misc/meta-morphogenesis.html

This document has been through several major reorganisations, which may have led to internal inconsistencies, duplications that need to be removed, and poor formatting, to be fixed later. Some of the ideas go back to my 1962 DPhil thesis defending Kant's philosophy mathematics, written before I had learnt about AI or computers.

The Meta-Morphogenesis project:
An introduction to the Turing-inspired Meta-Morphogenesis project can be found here:
http://www.cs.bham.ac.uk/research/projects/cogaff/misc/meta-morphogenesis.html
It includes a large, growing and messy collection of draft papers on evolution of biological information processing mechanisms, partly inspired by the work of Alan Turing. Recurring themes in this work include the role of implicit mathematical discoveries made by biological evolution (natural selection as a "Blind Theorem-Prover") Around November 2014 the project began to emphasise 'construction kits' of many sorts, including the fundamental construction kit (FCK) provided by physics and chemistry, and increasingly complex and more specialised derived construction kits (DCKs).
http://www.cs.bham.ac.uk/research/projects/cogaff/misc/construction-kits.html

Background
[Updated: 8 Nov 2015; 28 Dec 2015; 18 Jan 2016; 3 Feb 2016; 29 Nov 2016; 16 Oct 2017; 5 Jan 2018; 29 Oct 2018; 20 Jul 2019; 16 Nov 2020;]

The original version of this document was intended for a presentation on vision at Bristol University, in October 2015. After the event I kept on adding examples, and attempting to clarify their theoretical significance.

The nature of the subject matter, including the variety of examples that kept turning up, forced the document to grow considerably beyond the first draft. At some future date it may become a collection of separate documents. This is part of the Turing-inspired Meta-Morphogenesis project, begun in 2012:
http://www.cs.bham.ac.uk/research/projects/cogaff/misc/meta-morphogenesis.html

My original interest in the topics presented here was sparked around 1958 years, when I was studying mathematics as a graduate student in Oxford and learnt about Immanuel Kant, whose views about mathematical discovery in his Critique of Pure Reason Kant(1781), seemed to me to be correct, whereas all my philosophical friends thought his views had been refuted by developments in 20th century physics, especially Einstein's theory of general relativity, which implied that physical space was not Euclidean and Eddington's observation of the solar eclipse in 1919 which supported Einstein's theory. So I switched to philosophy and wrote a thesis DPhil thesis(1962), in which I attempted to clarify, defend and extend Kant's ideas.

Calling some mathematical discoveries "synthetic" was Kant's way of saying that they were not merely examples of definitions, or purely logical deductions from definitional truths.

Kant pointed out that there is a third distinction, between statements that are necessarily true or false, or contingently true or false. A statement P is necessarily true if it expresses some fact about how reality is constrained: some fact makes its falsehood impossible. For example, analytically true statements are necessarily true. But he claimed that there are also synthetic statements that are necessarily true, because something makes it impossible for them to be false, but their truth cannot be derived simply by specifying the definitions of the terms involved and then performing logical deductions. He thought there were many truths of mathematics that are necessarily true, but not analytic, because their truth cannot be established by using definitions of the terms and performing purely logical deductions. One of Kant's examples was "Three plus five equals eight" (and all other similar truths of arithmetic) and many truths of geometry, such as "Two straight lines cannot completely enclose a finite portion of a planar surface" "Three planar surfaces cannot completely enclose a finite volume of 3-D space", and many theorems of Euclidean geometry, e.g. Pythagoras' theorem about right angled triangles. Later sections of this paper present many more examples.

Defending Kant requires, among other things, analysing requirements for perceptual systems, especially spatial perception in animals and future intelligent machines. At that time, I knew nothing about computers or AI. I had heard of Turing and Turing machines, but did not study any details, despite the fact that, for a short time my supervisor was Hao Wang, who was then working on a logical theorem prover for IBM.
https://en.wikipedia.org/wiki/Hao_Wang_(academic)

In the rest of this paper I'll present a wide range of examples of types of discovery including many that I think Kant would have classified as being discoveries that are synthetic, i.e. not provable merely using definitions and logic, non-empirical (= "apriori") because we can discover their truth value without searching around the universe for counter examples, and also examples of necessary truth or falsity, because it is impossible for anything to exist that would make them false.

The remarks so far are intended merely to motivate the rest of this document, which provides many different sorts of examples, which I invite readers to think about. It was originally intended to be a much simpler document.

A few more remarks about the the notions of possibility, impossibility and necessity follow. I have assumed, and will give a variety of examples below, that humans do have abilities to perceive, think about, and reason about not only what currently exists or is happening in the environment but also what could, could not, or must, exist or happen. These can be described as modal features of the environment, in contrast with categorical features, that are restricted to what is the case and statistical features that summarise regularities, in the form of countable or measurable ratios of frequencies of occurrence of various types of object, state, or process.

However, it is not at all clear how human brains achieve such results, and what kind of AI mechanism might enable a robot to make similar discoveries. In part that's because of the variety of different types of discovery of necessity or impossibility, illustrated in the examples below.

The ability to perceive and reason about modal features can be described as a modal competence. As Immanuel Kant recognised, Kant (1781), these competences are essential to ancient mathematical discoveries, which are concerned with what is possible, impossible, or necessarily the case.

Also, as Kant pointed out, they are not all derivable from explicit definitions using nothing but logical reasoning -- i.e. they are synthetic, not analytic, in his terminology.

The competences involved in discovering modal features (e.g. impossibility, necessary truth) are totally different from competences concerned with perception of and reasoning about what occurs more or less frequently, or what is more or less probable.

This means that statistical evidence combined with probabilistic reasoning cannot explain the competences: statistical evidence cannot prove that something is impossible (e.g. a regular polyhedron with seven faces, i.e. a polyhedron with seven faces, all of which are exactly the same size and shape) or that something is necessarily the case (e.g. a regular polyhedron has faces with no more than 5 edges -- Why?).

It follows that statistics-based neural nets, however deep, cannot make such ancient mathematical discoveries, despite the huge, and steadily growing, collection of partial successes (many of which are deceptive).

I have the impression that many researchers in psychology, neuroscience, and neural-net inspired AI (e.g. Deep Learning mechanisms) have never noticed that key features of mathematical discovery, closely related to important kinds of practical spatial reasoning and perception of positive and negative affordances, involve discovering facts about impossibilities and necessary connections. Such modal facts cannot be expressed in terms of probabilities.

As far as I know there is nothing in current psychology, neuroscience, or AI that gives any indication of how animal brains can discover or represent impossibilities or necessary connections, a requirement for explaining human mathematical abilities. Piaget, who studied Kant, is one of the few psychologists I've encountered who understood the need for such explanations. But he lacked conceptual tools capable of formulating explanations. It is not clear that current AI provides such tools, except for limited classes of logic-based reasoning and discovery. For more on this see the Meta-Morphogenesis project mentioned above.

Researchers who have never previously noticed or learnt about this (Kantian, modal) feature of mathematical knowledge, seem to find it hard to understand at first. (Or perhaps I am bad at explaining...)

This prevents them grasping the deep limitations of statistics-based intelligence, i.e. intelligence based on abilities to acquire, reason about and use statistical information and probabilities. (Impossibility and necessity are not the same concepts as 0% and 100% probability.)

These scientific/philosophical failures of observation (or analysis) can lead to very shallow seriously mistaken descriptions and explanations of mathematical competences, when researchers don't realise that what they have explained is something much simpler/shallower than mathematical competences. (A study of Kant is not normally part of a developmental psychology degree, or training in AI or robotics, unfortunately.)

The perceptual abilities I'll draw attention to can be thought of as extending James Gibson's ideas, summarised below. He regarded the function of perception as being primarily to provide perceivers with information about affordances -- that is, information about actions that perceivers can or cannot perform in their current situation, among the actions that might be relevant to their needs or interests Gibson(1979).

This also includes information about how to vary actions, e.g. when to decelerate while approaching a target or obstacle.

Mechanisms for perception of possible actions between which the perceiver can choose must have evolved later than evolution of the much simpler reflex actions triggered without any consideration of alternatives.

Evolutionary developments also provided abilities for some organisms to acquire and use information about things that are impossible, or which could obstruct, or fail to be be useful for a particular goal, i.e. negative affordances. (This can save a lot of time when confronted with difficult tasks.)

These abilities to detect and use positive and negative affordances can be seen as constituting a subset of the phenomena Immanuel Kant thought about in connection with the nature of mathematical knowledge. Some of the connections between Kant and Gibson were pointed out in [Mace,2005], though not the connection with mathematical discovery.

Gibson seems not to have noticed that the perceived affordances he discussed are a subset of a broader collection of modal perceptual competences. An example is the ability to identify possible and impossible structures and processes in the environment, and the ability to think about necessary consequences of possible events or actions that need have nothing to do with the perceiver's current needs or interests. E.g. seeing that what someone else is trying to do is impossible. I suspect that even some pre-school children will realise not only that it is possible to remove a shoelace from a shoe by pulling one end of the lace, and impossible to remove it, without breaking anything, by pulling both ends at the same time.

It is not hard to grasp that it is possible to add exactly three new buttons to a collection of exactly five buttons, without changing anything else: e.g. one can try it out, or imagine trying it out. However, it not easy to explain how such a process will necessarily produce a collection of eight buttons. Moreover, this is not a fact specifically about buttons, but about any collection of distinguishable enduring countable objects.

Other examples include abilities to discover necessary truths in topology and geometry, for example that containment is transitive. Examples of such necessary connections can be relevant to a perceiver's actions but not all need be. For example, if an event happens on Mars, and Mars is part of the solar system, then the event happens within the solar system. Noticing the necessity has nothing to do with finding practical uses or limitations of the practical use of such information.

Some Kantian examples
(Added 29 Oct 2018)

An example given by Kant is that it is impossible for two straight lines in the same plane to completely enclose a finite portion of the plane. A more complex geometric insight is that is impossible for three plane surfaces to completely enclose a finite portion of 3D space. Those impossibilities could be relevant as negative affordances if someone wished to fence off part of a field by using only two straight fences and no other pre-existing barriers, or wished to create a completely closed container for tools or for an animal that might attempt to escape, made of exactly three flat pieces of material. What brain mechanisms allow humans to recognize such impossibilities?

These examples are closely related to well known facts of Euclidean geometry, though one does not need to have studied mathematics to recognise the claimed necessities and impossibilities:

There are many more illustrations of the fact that, in humans (and possibly several other intelligent types of animal), the functions of vision include perception of modalities (i.e. what is possible, impossible, or necessarily the case). This has nothing to do with discovering probabilities or combining sensory modalities (touch, sound, sight, etc.), though it can use any or all of those sensory modalities.

These discoveries can be about exosomatic information, for example discoveries about what is or is not possible in the environment -- i.e. outside the skin. That contrasts with learning sensory-motor and other somatic relationships (correlations inside the organism's skin). Evolution made use of many such implicit mathematical discoveries long before there were human mathematicians, but that's another sub-topic.

I recently learnt that in 1938 Alan Turing had noticed a distinction between mathematical intuition and mathematical ingenuity, claiming that only the latter could be implemented in computers. Most, if not all, of the examples in this document, seem to be illustrations of powers of human mathematical intuition, especially spatial intuition providing mathematical knowledge of geometry and topology. Turing's ideas are summarised and discussed in
http://www.cs.bham.ac.uk/research/projects/cogaff/misc/turing-intuition.html
also (Pdf) (Still work in progress.)

As far as I know nobody in AI knows how to replicate these abilities (involving intuition, or insight) in machines, and no psychologists or neuroscientists can explain how brains make such discoveries possible. I hope new answers will eventually emerge from the Meta-Morphogenesis (M-M) project, summarised in:
http://www.cs.bham.ac.uk/research/projects/cogaff/misc/meta-morphogenesis.html

The claims about human abilities to perceive possibilities, impossibilities and necessities are illustrated below using modified versions of a picture drawn by Oscar Reutersvard in 1934, as a key example. However, many other examples are presented. In particular I'll offer examples related to proto-mathematical discoveries made by pre-verbal human toddlers presented and discussed in this (also very messy) document on "Toddler theorems": http://www.cs.bham.ac.uk/research/projects/cogaff/misc/toddler-theorems.html

Personal history

I first attempted to explicate the modal concepts used here in chapter 7 of my 1962 DPhil thesis defending Kant's philosophy of mathematics against attacks by philosophers who had no personal experience of discovering and proving geometrical truths. Kant's distinctions were also summarised very briefly in Sloman(1965). A digitised, searchable, version of the 1962 thesis was made freely available online in 2016:
http://www.cs.bham.ac.uk/research/projects/cogaff/sloman-1962/) (HTML and PDF)

Some readers may wish to skip the preliminary remarks below and jump straight to examples, e.g. the section on Representing possible processes.

What are the functions of perception?

Many researchers assume that the function of perception is to find out what IS the case, what WAS the case and what WILL BE the case in the environment -- including the immediately perceived environment and the extended environment. Examples include perceiving objects as space-filling solids, even when only their external surfaces are perceived. Sometimes, for various reasons, the inferences from sensory and other information are not totally reliable. Some researchers therefore regard the functions of vision (or more generally perception) as including finding out what is PROBABLY the case.

However, there are important functions of vision that are not included in these obvious functions. Vision also provides information about possibilities, impossibilities, and necessary consequences. That can also include acquiring conditional information, e.g. about what WOULD HAVE BEEN the case if ..., or what WOULD BE or WILL BE the case if... . E.g. perceiving an apple hanging in a tree as supported by its stalk provides information about what would happen if the stalk were to break. As in the previous cases the information available in some cases is incomplete or unreliable or in some other way less than perfect. What is derived is then thought to require inferences about probabilities of various alternatives. However this paper is not concerned with those cases, but cases where a change will make something possible, or impossible, or necessarily the case. E.g. it is possible for me to move closer to an open doorway to another room, and if I do I shall necessarily see more of the room if information travels in straight lines.

These un-noticed or inadequately understood functions of vision are concerned with obtaining information about what is POSSIBLE or IMPOSSIBLE, or NECESSARILY or CONTINGENTLY the case in the environment. (What is contingent is possibly true and possibly false and neither necessarily true nor necessarily false.) For more on these "alethic" modal concepts see https://en.wikipedia.org/wiki/Modal_logic#Alethic_logic.

The spatial perceptual functions described in terms of modal concepts of possibility, impossibility, necessity and contingency, have nothing to do with PROBABILITIES (although probabilities presuppose possibilities). In particular, the concepts "possible", "impossible", and "necessarily true" are totally different from notions of a non-zero, zero, or 100% probability. Probabilities are essentially ratios of numbers produced by counting or measuring.

The functions of vision related to perception of possibilities and impossibilities (constraints on possibilities) seem rarely to be noticed by vision researchers, although some researchers interested in perception have investigated at least some of them, including Immanuel Kant Kant (1781) and, more recently, James Gibson Gibson (1979)) and his followers. But Gibson and most psychologists, unlike Kant, typically fail to address relationships between mathematical competences and these spatial competences, the main topic of this document. Piaget was an exception, especially in his last book, and his 1952 book e.g. on gradual development of understanding of 1-1 correlations and cardinality.

Probability concepts presuppose concepts of possibility, since probabilities are comparisons among sets of possibilities. These are often partial orderings, sometimes with numerical comparisons added. However, the (alethic) modal concepts of possibility, impossibility, and necessity used here do not presuppose probabilities. In particular, they are totally different from numerical probability concepts.

There are deep unanswered questions about whether and how the alethic modal concepts, "possible", "impossible", "necessary", and "contingent" (= neither impossible nor necessary) are used by other animals, and about how they can be represented in information-processing systems (e.g. in minds of animals or robots). The roles these concepts play in intelligence tend to be mis-described, or ignored by perception researchers, especially their connection with mathematical knowledge.

I shall use a variety of examples to illustrate some of the ways these modal concepts work, why they are important for intelligent animals or machines, how the functions of vision (and more generally perception) involve them, and how they are connected with mathematical discoveries.

One of the important functions of vision is to obtain information about how the environment relates to abilities, risks, needs, or intentions of other agents -- i.e. "vicarious" affordances. Gibson ((1966) and (1979)) discussed some special cases of this, but I don't think he saw all the important implications or pre-requisites of being able to see what is relevant to the desires, intentions, preferences, beliefs, of other agents, or oneself. These are topics that need full discussion on another occasion, though they will be briefly mentioned below. (See also Sloman (2009a).)

We'll see that information about what is or is not possible is relevant both to the immediate practical uses of vision and also to the roles of vision, and meta-cognition, in some types of mathematical discovery. I'll try to indicate, in very crude outline, how the earliest mathematical discoveries might have been concerned with meta-theories about possibilities for action. The need for theories about possibilities arises naturally for intelligent agents choosing and acting in a structured environment. The need for meta-theories arises if those agents have the ability to detect and reflect on their own theories.

Meta-meta theories are required for reflecting on or discussing the properties of those meta-theories, and how they can be found to be true. The evolutionary changes making that possible also made possible the kinds of mathematical presentation found in Euclid's work.

Later forms of mathematics (based on formal systems developed in the last two centuries) have other functions, which will not be discussed here. So I'll ignore most mathematics based on the developments in logic, set-theory and formal systems since around 1900. The view of some mathematicians that what happened earlier was not really mathematical discovery and reasoning is just false, unless the label "mathematics" is re-defined to make it true.

There can be enormous variations between the spatial capabilities and performances of individual humans, individual monkeys, individual crows, etc. For the abilities to use vision to acquire modal information are not all innate: they develop under the influence of the environment. Individuals in different environments will therefore develop different spatial competences. (There may also be genetic differences.)

Learning to perceive different sets of possibilities and constraints can be compared with differences between learning to read French and learning to read Chinese. In both cases there are considerable individual and cultural differences. It follows that requiring experiments on vision to provide reliable repeatable data about humans will rule out many experiments that provide information about important visual/spatial capabilities, since what is true of one person may not be true of another. Yet facts about individuals, even unusual or unique individuals, are part of what science needs to explain, e.g. by explaining how the individuals process information, including how and why they differ.

A good theory of vision should explain how the individual competences work (preferably demonstrated in working AI models) AND how a (mostly) common genetic heritage can produce differences in competences of individuals that share the heritage. That requires a model of individual development, some features of which are sketched in the section on Evo-Devo Issues.

My complaints about wide-spread neglect of important functions of vision apply both to theories of human vision and theories of animal vision, and to statistics-based models and theories of vision that have been successfully applied in special purpose robots and other machines with useful, but very limited functionality. The main theme here is the need for a good theory of vision to be part of a good theory explaining important types of mathematical discovery.

This point can be generalised: a good theory of mind, or of evolution of minds, or of development of minds needs to explain the abilities of at least one sort of mind to discover and use mathematical truths about what is and is not possible. Such information is very different from abilities to acquire and use information about probabilities that many current AI systems focus on. In particular, a good theory of what minds are and how they evolved needs to explain what made it possible for Euclid and other ancient mathematicians to make the discoveries reported in Euclid's Elements. Additional examples are presented in these papers and papers they reference:
http://www.cs.bham.ac.uk/research/projects/cogaff/misc/triangle-sum.html
http://www.cs.bham.ac.uk/research/projects/cogaff/misc/trisect.html

Although no detailed explanation (or working model) exists, we can discuss requirements for future candidates. Some incomplete conjectures are presented below.

Introduction: Gibson's notion of affordance

Some of the earliest AI vision work (and some research in psychology) focused on perception and recognition of 2-D patterns in Images. But it was soon realised that human and animal vision goes beyond that, e.g. because the visible part of the three-dimensional world is projected into a two-dimensional image by the sensing apparatus. So a visual system needs to "reverse" this process: the original three-dimensional reality must be inferred from the two-dimensional image (plus some background knowledge, where necessary). This (obviously) does not involve building a new 3-D structure inside the brain. It requires building an information structure that provides spatial information about the 3-D structure in the environment. Often some or all of the same spatial information can be acquired through other sensory/motor subsystems, including tactile, haptic and vestibular (semi-circular canal) mechanisms.

This claim that seeing is "reconstructing" is often attributed to David Marr (1982), though it was taken for granted by AI researchers much earlier, e.g. Roberts, 1965 and others surveyed in Ballard and Brown, 1983, though they proposed different theories about the details.

On that view, the main functions of vision should be the same across all species, though Marr acknowledged that for some species, e.g. insect species, the functions might be different, and of course some human visual capabilities, such as reading text or musical scores, understanding sign languages, and interpreting maps and engineering drawings, are unique to humans.

In opposition to these views, James Gibson ((1966) and (1979)) criticised researchers who thought the function of vision in animals was simply to produce some sort of representation (or collection of representations) of the objects visible in the environment, including information about distances to their surfaces, orientations of visible surfaces, illumination, and a variety of other geometrical relationships.

He pointed out that there is a completely different function for perception in general, and vision in particular, which he called perception of affordances. In that sort of perception, the information acquired is not about the actual contents of the environment in a form that is independent of the perceiver's capabilities and interests, but is information relevant to potential actions of the perceiver: e.g. what actions are possible for the perceiver in the current environment, given the perceiver's physical capabilities and current needs (or goals, preferences, etc.). Actions produce changes, so the perceived information is about possible changes the perceiver can bring about. We'll generalise that below.

A more extreme version of Gibson's view, treats the information content derived from visual sensory information as heavily dependent on the viewer's anatomy and physiology, and current or possible needs, preferences, dislikes, etc.

An even more extreme theory could claim that there is no explicitly describable spatial content, only an unintelligible mass of conditional causal connections between sensor neurones and motor neurones, modulated by signals from internal sensors concerned with the organism's current needs. I think some researchers who emphasise embodied cognition and who deny the use of representations, are implicitly committed to such an extreme position. But that view will be ignored here. (I expect any experienced engineer will easily see its flaws.) Most of this paper presents and analyses cases of perception of what is and is not possible. Some parts discuss implications for mathematical cognition.

Beyond Gibson: Possibilities, impossibilities, necessities

Most models and theories of perception seem to restrict the functions of perception, including spatial perception, to detecting and recording how things are (or probably are) in the environment, classifying them and predicting what will happen or what will be the case (with or without probability estimates), and in some cases retrodicting or diagnosing causes of, or predecessors of, what is the case.

For more intelligent species, perception, and especially vision, can also be used for acquiring information about what is and is not possible: i.e. modal information. So theories, models, and robotic implementations of vision systems that ignore perception of and reasoning about possibilities, impossibilities and necessities are seriously impoverished. That criticism of standard theories of vision is closely related to Gibson's criticisms, but he does not go far enough in the direction proposed below (elaborating on Sloman (2009a)).

So, vision (and to a lesser extent other modes of perception) can be used not only to gain information about what is the case in the environment, but also information about possibilities, and relations between possibilities. This generalises Gibson's ideas about perception of positive and negative "affordances" for the agent. In particular possibilities and constraints on possibilities that are perceived visually need not concern actions or needs or preferences of the perceiver; the visually acquired information can go not only beyond spatial structures, and immediately useful information about possible actions for the perceiver, but can also include future possibilities, explanations of previously realised possibilities or failures, and also discovering possible or impossible events or processes that have nothing to do with the perceiver's intentions, plans, or actions.

The need to generalise Gibson's ideas
Most "Gibsonian" theories of perception (especially visual perception) that I am aware of fail to do justice to the variety of functions of vision, the variety of types of contents of visual experience, and consequently the variety of requirements for explanatory mechanisms, or mechanisms needed to give robots human-like (or even squirrel-like, crow-like, etc.) visual or more generally spatial perceptual capabilities. This is also true of theories of intelligence or cognition that (over-)emphasise embodiment. Focusing on too few examples of what needs to be explained leads to bad theories in both science and philosophy. It can also lead to impoverished engineering.

In particular, theorists emphasising embodiment often ignore the distinction between "online" and "offline" uses of visual information, discussed below, and the more subtle division between different "offline" uses of perception of what is the case, including perception of what is and is not possible and how those possibilities and impossibilities can change if some current possibility is realised. Processes of predicting planning, designing and explaining may all use chains of alterations in what is and is not possible.
Note added 22 Feb 2020
I previously failed to make clear that the possibilities and impossibilities mentioned here are not possible and impossible sensory or perceptual contents (in the minds of perceivers) but possible and impossible spatial contents in the physical environment --- including some that are relevant to choices in engineering or architectural design, for example.

Moreover, some of the perceptually available kinds of information about possibilities and impossibilities illustrated below are also essential to some ancient mathematical discoveries, for example in geometry and topology, recorded in Euclid's Elements, which I contrasted with logical discoveries in http://www.cs.bham.ac.uk/research/projects/cogaff/misc/ijcai-2017-cog.html

I also include below some examples that were not part of ancient mathematics, but require similar abilities to discover spatial possibilities and impossibilities. One example that as far as I know has not previously been discussed is the necessary connection between two aspects of a process of triangle deformation:
http://www.cs.bham.ac.uk/research/projects/cogaff/misc/deform-triangle.html

The connection between the functions of visual perception in humans and other animals, and mathematical discoveries made by Euclid and his predecessors is the main topic driving the construction of this paper, but the connections involve somewhat long and tortuous links.

In other documents I'll focus on some visual capabilities of mathematicians: not modern mathematicians reading logical and algebraic formulae and proofs (requiring a related, but different, set of competences), but the ancient mathematicians whose discoveries I suspect led, eventually, to Euclid's Elements. As far as I know, these abilities are still unexplained and have not been replicated in AI systems.

A full investigation will require understanding how evolution of biological functions of human vision, including visual competences shared with other species, led to capabilities that were able to support mathematical discoveries, even though that was not why they evolved.

Some of those evolutionary changes seem to be recapitulated in child development, as described in the Meta-Configured genome theory. XXXX Understanding the details may be essential for high quality mathematics teaching, but that's a topic that will not be addressed here. (See the sections on Toddler Topology, and Toddler Theorems below, and the epigenetic schema in the section on Evo-Devo Issues.)

In my youth it was still customary to teach geometric mathematical competences at school, but most current youngsters (at least most young students and researchers I meet in Universities), seem to be deprived of that privilege.

Many potential readers of this document will therefore unfortunately have no prior experience of some of the phenomena under discussion. Links are provided to web pages presenting various more or less elementary fragments of Euclidean geometry and a subset of topology concerned with continuous deformations in space. I shall try to present examples that are intelligible to non-mathematicians, all of whom have mathematical competences, whether recognized or not.

The work presented here implicitly presents requirements for some of the construction kits that build human visual systems. We need open minds as to whether well-known forms of computation and physical assembly suffice.

Online and offline uses of visual information

A crucial first step in understanding the connections between vision and mathematics, and the roles of visual mechanisms (in contrast with separate cognitive mechanisms) in making mathematical discoveries, is to distinguish uses of perceptual information in (a) online intelligence and (b) offline intelligence:

This is not necessarily a sharp dichotomy: there may be processes/activities that use both online and offline functions of vision, sometimes in succession and sometimes in combination. However, in the extreme cases the types of information-processing mechanism required are very different, even if intermediate cases arise from use of both types of mechanism in combined tasks.

I suspect some of the enthusiasm for "embodied cognition" and "extended mind" theories is based partly on recognition of the importance of online intelligence coupled with blindness concerning offline intelligence, and partly on ill-founded anti-computational prejudices in some cases. But I shall not pursue those points here.

Requirements for the online and offline functions of vision

What sorts of visual mechanisms give organisms online and offline forms of intelligence?

(a) Online use of visual information requires fast-acting information stores (memory mechanisms) whose contents constantly influence forms of behaviour, and which are constantly overwritten as new information comes in, so that any use of the information has to be fast.

In many (most?) cases the mechanisms using such information are fast-acting (i.e reflexes) and either innate or produced by extended learning or training, e.g. in many sporting activities, musical competences, linguistic competences, and others. Some may use evolutionarily very old mechanisms (e.g. blinking), others newer, more sophisticated, mechanisms (e.g. musical sight-reading).

(b) Offline use of visual information requires longer term forms of storage, so that information acquired at a particular time can be used at different times, for multiple purposes, usually in combination with other forms of information, new and old, often on the basis of temporarily assembled structures -- using what are often referred to as "deliberative" mechanisms, discussed in more detail in Sloman (delib).

Biological examples of offline use of vision

An example of offline use is an animal seeing some fruit in a tree and being motivated to climb the tree to get to the fruit. The fruit need not, and typically will not, remain visible to the animal during the process. A more complex use would be an animal recording information about the location of the fruit and not using the information until later, when it is hungry. Stored information in combination with the need for food can trigger a process of planning a route back to the tree, and use of the route to get to where the fruit was. This can happen even if the fruit is no longer there, because it has fallen to the ground, or been consumed by something else.

The ability to do use information offline, in forming and executing multi-step plans is often thought to be restricted to a small subset of vertebrates, but there is evidence of such abilities in other species, including the Portia Spider.

Planning and deliberation by portia spiders
The portia spider works out a route to its prey then follows it even when it can no longer see the prey, making detours if necessary and avoiding branches that would not lead to the prey.

Another kind of offline use involves passing information to another agent: e.g. pointing at where the fruit is, or telling someone where it is, or explaining how to get to it. (These are three different cases.) It is easy to think of other cases of offline use of perceptual information: left as an exercise for readers.

Proto-affordances
Gibson's idea that the main function of vision is to provide information about affordances, can be further generalised to include the role of vision not only in acquiring information about possible actions of the perceiver (used in either online or offline intelligence), but also information about possible changes in the environment, and constraints on those changes, irrespective of whether the changes are produced by the perceiver, and irrespective of whether the changes are known to be relevant to the current or future needs or interests of the perceiver. An example would be noticing the possibility of the fruit falling and hitting a branch below it. Every physical configuration of objects has multiple possibilities of change that can be understood by perceivers who have no interest in whether the changes occur or not. I call those "proto-affordances". See (e.g. Hartson (2003), Sloman (2008) and Siegel (2014)). Examples of scenes with multiple proto-affordances are presented below.

Representing possible processes

Information about actual or possible processes can take many forms. At one extreme all processes are represented in terms of rates of change of measurable quantities, or vectors (usually represented by coordinates). If the information available is not sufficiently precise to provide numerical values an alternative is to encode possibilities either in terms of enclosing intervals (perhaps with fuzzy boundaries) or by probability distributions over possible values, or by ... (need to enlarge list of options).

A quite different approach to representing possibilities is to switch to topological, or more generally relational and structural descriptions. Such descriptions can specify parts, and relationships between parts, possibly parametrised relationships; e.g. A and B meet at an angle that is smaller than the angle between C and D, or the distance between A and B is less than the distance between C and D, or A, B, C, and D are parallel with gaps of increasing size along the sequence, or the vertex at which A and B meet lies on C, and many more.

The availability of such non-numerical representations of structure can make it inappropriate (and wasteful) to use the most powerful mathematical methods for representing processes, e.g. using differential and integral calculus. It can be especially wasteful and inappropriate if the precision of those methods is overkill for the information needs of an animal (e.g. answering "Am I getting closer to my prey?"). Use of qualitative, topological, comparative, imprecise descriptions may allow far greater generality at the cost of some ad-hocery: e.g. learning about special important cases (structures) and describing others in terms of them.

The use of grammars and parse trees (or nets) in linguistics and in compiler design illustrates the power and versatility of non-numerical forms of representation: for some purposes. In the 1960s, Clowes (see Clowes Tribute) and others proposed similar techniques for visual systems, including claiming that, like sentences, pictures/images could have (two-dimensional) syntactic structures representing (in many cases three-dimensional) contents, though finding appropriate generalisations of the notions of "grammar" and "semantic content" was not easy Kaneff(1970). Clowes and others pointed out that pictures, like sentences, could have inconsistent semantics.

Chapter 9 of Sloman (1978) provided a demonstration of multi-layer semantics, as a proof of principle, showing how images, like spoken, written and signed sentences, can be interpreted as having several distinct levels of structured (syntactic/semantic) content, detected in messy images by the POPEYE program. (At that time AI theories of vision as requiring a mixture of bottom up and top down -- and middle out-- concurrent processing were unfashionable. An application for funds to continue the research was refused, and a paper reporting the work proved unpublishable, partly because, as the main reviewer pointed out, it was inconsistent with David Marr's ideas regarding vision, which I thought then, and still think, were oversimplified. (His admirers were more narrow minded than he was!)

Such multi-layered semantic content is related to the use of multi-layered genomes, in which complex structures (usually developed later in evolution) are expressed by the genome at later stages of individual development, but in ways that make use of the structures/competences produced at earlier stages, which can vary according to physical and cultural environments. (This is how a particular genome has the ability to produce a very wide variety of types of mind, using information acquired from the environment during development. This is totally unlike any standard learning mechanism fixed at an early stage, and producing increasingly complex products, by repeatedly using statistical regularities found earlier.)

Possibility perception

To recapitulate: besides the processes actually occurring in any situation, other processes are possible, and if they were to occur the results would include perception of processes that were not perceived previously. One thing that should be obvious, but apparently is not obvious to many vision researchers is that perception of processes is not a matter of finding depictions of moving boxes in an image and tracking them. Watching a ballet performance in which there are not only groups of dancers performing on the stage, but also other animals, furniture, perhaps books and toys, provides opportunities for very rich visual processing, in some ways (but not all ways) more complex than videos of varied vegetable matter.

Could challenges for online intelligence lead to a new kind of offline intelligence

Is it possible that mechanisms that originally evolved to serve fast-acting behavioural reflexes, may later have been modified to serve fast-acting mechanisms for building new temporary internal information structures triggered by the contents of fast-acting sensory buffers developed for online intelligence (type (a) above)?

During speech understanding these extended online control mechanisms would construct intermediate information structures representing phonemes, morphemes, words, phrases, clauses, sentences and other linguistic entities.

But before that, evolution seems to have have produced older visual mechanisms shared across more species: mechanisms that that rapidly construct a variety of information structures about aspects of the environment, including parts of familiar objects, combinations of familiar objects, possible action trajectories, possible consequences of such actions, some of which are necessary consequences, constraints on such possibilities (i.e. impossibilities), possible non-action processes, and combinations of all the above to form information structures about a complex situation, structure or process.

All of these modal features (types of possibility and impossibility) perceived, are distinct from, though dependent on, perception of actual structures and processes.

Many contents of such perceptions are not used at the time for any practical purposes, but some may be used as a source of inferences for decision making later on, e.g. perceiving a person eating a complex object, might be a source of information relevant to the perceiver assembling ingredients for making a sandwich, locations where ingredients might be stored, and various partially assembled sandwich states of changing complexity. For some of the entities thus recorded at high speed, e.g. various objects, spaces, spatial relations, and motions, the relevance to possible future moves by the viewer may also be derived and recorded. That could be the birth of certain kinds of affordance perception discussed by Gibson.

I suggest that many such perceived situations and processes involve unavoidable constraints some of which were noticed and thought about by long dead ancestors of humans, using ancient possibility/impossibility perception mechanisms -- laying foundations for later evolution and development of sophisticated mathematical discovery and reasoning mechanisms.

If perception of a static scene can trigger rapid construction on varying spatial scales and temporal scales, with varying combinations of concreteness and abstractness, then perception of a complex moving scene, or a complex static scene perceived by a moving viewer will require mechanisms that can rapidly modify the information structures, driven by information about changes in receptor information contents in combination with other information, including information about the viewer's actions, and additional background knowledge about the type of environment.

If all that apparatus for motion perception is already available to deal with a wide variety of types of motion, whether motion of the viewer, or motion of perceived objects or both, then perhaps the same apparatus can also play a role in a new kind of perception of static scenes, by implicitly representing widely varying possibilities that cover things that could happen in such a situation.

If the mechanisms for abstraction are available for dealing economically with actual processes they may also allow representation and reasoning about possible processes.

These generalisations of Gibson's ideas seem to be crucial for understanding mathematical cognition in humans, other animals and possible future robots. That's because key forms of mathematical discovery are concerned with what is possible and what is impossible, and how the set of possibilities and impossibilities relevant to a situation can change if some of the possibilities are realised. Your possibilities for action and perception outside a doorway are different from the possibilities just inside the doorway.

Possibilities vs Probabilities

The possibilities and impossibilities mentioned above have nothing to do with information about probabilities. Possibilities are more fundamental than probabilities: probabilities can exist only in relation to sets of possibilities, but sets of possibilities, e.g. possible actions that an animal can perform in some situation, or possible ways in which three coins could be arranged on a table, have nothing to do with probabilities. The possibilities can be noticed, reasoned about, and in some cases made actual, without any consideration of probabilities. If you know what coins are and you see four coins on the table there are many possibilities for changing the configuration on the table including adding more coins, removing coins turning coins over, rotating the coins without turning them over, and moving them into new locations.

Temporally chained possibilities

A great deal of attention has given to probabilities of various things happening when one or more coins are flipped, or one coin is flipped several times. Here's one of many online tutorials:
http://gwydir.demon.co.uk/jo/probability/info.htm

What has not been noticed is that there are also many impossibilities associated with coins -- not when they are flipped but when they turned over in a controlled way. Repeatedly turning over only one coin gives a very boring predictable result. If there are N coins, with N > 1, and only one coin is turned over at a time, then the resulting sequences of patterns can have very different structures. E.g. here are five sequences of patterns formed by three coins, using H or T to show whether each coin is head up or tail up.

Four of the sequences shown are impossible. One of the sequences is possible. Which?
(Correction 6 Jan 2019: Previously the description erroneously said four are possible and one impossible. My thanks to Marcel Kvassay for pointing out the slip.)

Now consider what happens if you combine a pattern of controlled coin turning while counting, using only a single coin?

If you think about a flat coin showing only either H or T, and you keep counting as you turn a coin, are any other combinations possible?

Now suppose you have two coins, and as you count you turn only one coin over, but sometimes you turn over the first coin and sometimes the second: i.e. there is not a regular alternation. Here are sequences combining counting (numbers) and pairs of coins

The number of possible sequences is far greater than if you have only one coin and turn it over for each number counted. If you have more than one coin, only one of which is turned over at a time, then you cannot work out in advance what coin pattern will be reached at the 11th step, since it will depend on the earlier sequence of choices.

Question: If you turn over exactly one coin at a time as you count, with no constraints on which coin, are there any constraints on the resulting possible sequences, e.g. any possible patterns of count number and states of the two coins that cannot occur? What about combinations of types of number, e.g. odd numbers, even numbers, multiples of 3, powers of 3, with coin-face combinations?

Some of the possibilities, if realised, will necessarily have certain consequences concerning what can happen next. For example if there are several coins on the table then each one will either be H up or T up. Then turning any of them over will switch to the other state. If you are counting, then each number will have certain mathematical properties, e.g. which other numbers divide into it with no remainder (i.e. what its factors are), and when you move to the next number the properties (the list of factors) will change. Can you find some combinations of coin states nd count numbers that cannot occur in such a sequence?

Apologies to number theorists who find this too easy. My point is not to display deep new mathematical results, but to raise questions about what sort of mind a machine, e.g. a future robot, needs to have in order to make discoveries about which static combinations are impossible, or which sequences of combinations are impossible.

More importantly, are the same kinds of mechanisms required for discovering the coin-plus-counting impossibilities as the other possibilities presented in this document, e.g. geometrical impossibilities and topological impossibilities. In other words, could there have been a stage in our evolutionary history (or in the development of an individual human) before which none of these impossibilities could be discovered followed by evolutionary change to a new state that allowed our ancestors to discover all of these impossibilities?

Or were different evolutionary transitions in information processing required to enable the different impossibility discoveries to be made?

One of the simplest examples is the fact that if in the two coin example, two turning-over processes occur in sequence: involving either only one coin turned twice or first one coin turned then the other, then if they initially had the same state they will end up with the same state, and if they initially had different states they will end up with different states -- after two such turns. Why must one of those be the result? How can we design machines that are able to answer these questions? (What sorts of machines could think of these questions without being tested by humans? What makes you think of new questions? What mechanisms allow you to respond to such triggers?)

Can these discoveries be made using statistical/probabilistic reasoning?
My experience of talking to many researchers about examples presented in this document is that often they assume that a robot could make these discoveries by experimenting with lots of coin turning sequences and making statistical inferences. And some psychologists or neuroscientists will assume that brains make the discoveries by using such reasoning.

But it is impossible to use statistical evidence to discover that something is impossible, or that something is necessarily the case. Our ancestors may have used empirical evidence to conclude that it is impossible for a person in Paris and a person in Toronto to have a normal conversation, at the same speed as a face to face conversation. But we now know that that conclusion would have been false.

To make the mathematical discoveries about possibilities, impossibilities and necessary connections discussed in this document (and many others) no amount of empirical evidence will suffice, though it may be suggestive.

As Immanuel Kant pointed out over two centuries ago, the discovery of necessity or impossibility requires something different: something that mathematicians have been doing for millennia, and I believe other humans and many other intelligent animals who were not explicitly doing mathematics were also able to make and use such discoveries, but without the meta-cognitive abilities to notice and think about their reasoning.

Of course, discovering examples can provide knowledge of probabilities, and that can in some cases trigger investigation leading to a deeper insight into what is impossible or necessarily the case, in addition to discoveries about probabilities.

For example if you have a collection of coins and a chess board, and place one coin in each square on the board randomly or systematically placing them with head or tail up, you can do experiments to find out the probability of all four corner locations being occupied by coins in the same state, e.g. all H or all T. However, using mathematical, not observational reasoning, we can find the probability without sampling the space of possible configurations, which is just as well because getting a fair subset of the space would take a long time, since there are

possibilities. With that size of space, no human could ever collect adequate statistical evidence to settle the probability of all for corners having the same colour.

Isn't it fortunate that evolution produced brain mechanisms that are far more powerful than probabilistic learners? What mechanisms? When will robots have them? Do some AI machines already have them? Does anyone know how brains do such things?

Perceiving vs understanding

It is possible to perceive convincing evidence of a regularity without understanding why the regularity holds. For example, Morley's amazing theorem states "The three points of intersection of the adjacent trisectors of the angles of any triangle form an equilateral triangle." This wonderful web site http://www.cut-the-knot.org/triangle/Morley/ includes a Javascript applet that allows the corners of a triangle to be moved arbitrarily. When that happens, the applet automatically adjusts the trisectors of the angles and the points of intersection of adjacent trisectors. Playing with this demonstration may convince someone that theorem is true, or that it has a very high probability, but that's not the same as understanding why it must be true, or seeing that there cannot be any counter examples. Without that insight you cannot be sure that Morley's claim isn't merely a very close approximation to the truth: e.g. if the angles of the central triangle remain very close to 60° while changing imperceptibly. A person who has not grasped a proof lacks the "mathematical qualia" for that theorem.

Contrast Mary Pardoe's proof of the Triangle Sum Theorem, in the form: the interior angles of any planar triangle must sum to a half rotation (180°):

The above example illustrates ways in which possibilities and necessities or impossibilities can be closely related: realisation of some possibilities may necessarily have certain consequences. What they are and why they are inevitable differs from case to case. The ability to notice possibilities and impossibilities (necessities) and the consequences of realising some possibilities in a situation is an important aspect of human development, as Piaget noticed. His last two books (1981-1983) discuss many examples.

However, although Piaget realised that these are important aspects of human cognition, and some of ways of probing children's minds are based on deep insights about varieties of cognitive function, I am not sure that his theories regarding the cognitive mechanisms (which I found hard to follow) were sufficiently well developed to be useful, e.g. in explaining mathematical cognition, or in designing intelligent machines with human-like powers of mathematical discovery. Piaget's work on cardinality is mentioned below below..

One of the problems of discussing such issues is that there are so many different types of case, and we need to understand the variety in order to come up with good theories about what's going on. In particular there are some cases where the cognitive competences involved are purely logical reasoning capabilities, whereas in other cases more varied mathematical abilities are required, e.g. concerned with reasoning about spatial structures and processes, as in topology and Euclidean geometry.

Pictures of possible and impossible object configurations

In order to illustrate some of these points I shall discuss some examples of pictures of impossible objects, for which the mathematical physicist Roger Penrose and the artist Maurits Cornelis Escher are famous. Less well known is the Swedish artist Oscar Reutersvard, whose pictures of impossible objects came earlier (1934), and have some interesting features more directly relevant to mathematical reasoning about possibilities and impossibilities, because the pictures invite changes of various sorts, or, in other words, display multiple affordances. The pictures also help to shed light on requirements for artificial vision systems, and theories of vision. Indirectly they also challenge theories about what brains can do and how they work. (There are also some well known paintings of impossible scenes by much older artists.)

Instead of jumping straight to impossible objects, it is illuminating to consider pictures of scenes containing items in different arrangements, some possible, and some not and try to understand what exactly changes, both in the scenes and in the cognitive/perceptual processes.

What can you do with a collection of eight similar cubes?

Alternative configurations of the blocks are presented below. You can probably imagine a series of individual block-trajectories that would transform the above figure into the one below, and similarly for (most of) the later examples.

AI vision systems in the early 1970s could already (very, very slowly, because of computer speeds at the time) interpret a variety of 2D line drawings as representing 3D polyhedral objects, including deciding which picture lines corresponded to concave, convex, or occluding edges, e.g. Clowes (1971). A short time later shadows and cracks were added, and then the ability to cope (to some extent) with noisy images containing spurious line fragments, missing line fragments or lines with gaps due to image noise (for example). (Ref G.Grape, G. Hinton). A tutorial presentation on some of those techniques is referred to below.

However such programs did not have the ability to suggest, or reason about, alternative configurations of blocks: they saw only what existed. Moreover they were not able to assign precise lengths or angles in all the images in which they could perceive structure. And although they could in some cases detect that one visible surface must be further from the viewer, they did not reason about whether the whole scene depicted was consistent. So they could not detect circular "further than" relationships, though I suspect that could have been added. But that's just a special case of detecting impossibilities.

Thinking about, or imagining possible variations in a scene is a crucial ability for many intelligent animals, including nest-builders, hunters, and animals that care for their young.

Humans can do this not only for real scenes containing physical objects but also for depicted scenes: where the pictures specify physical objects in physical relationships, including relationships like adjacency, co-linearity, being above, being between, or supporting.

They presumably cannot do all this at birth. Why not? What mechanisms do they lack? How do they acquire the mechanisms that provide the new abilities later on? Is it merely a process of learning to use mechanisms they already have from birth? Or from before birth -- e.g. from month X of foetal development?), or do new brain mechanisms grow during years of physical growth (and thereafter)?

What if you had nine similar cubes?

Try testing your own brain mechanisms for imagining configurations in which there are not only eight, but nine cubes arranged in space? Here are some sample configurations. Each could be transformed into any of the others by moving individual blocks around.

Note: Piaget asked children that sort of question using a few objects on a flat surface (1981). Not all realised that there is no fixed finite set of possibilities. Some answered after a while that no more arrangements were possible.

More possible moves

Seeing possibilities extends beyond seeing possibilities for re-arranging objects in the scene. In this case it also includes seeing locations where you might put a flat, or nearly flat object (e.g. your hand opened out) into the scene depicted. E.g. for any pair of adjacent cubes you could move your hand between them, provided that you rotated the hand into the right orientation.

Other things you can do in the perceived configurations include swapping pairs of blocks: move one onto the table, move another block to the newly emptied location, then move the first block into the new space. Or move both simultaneously using two hands.

Several more configurations of nine blocks are depicted below. You may or may not find one of them anomalous.

As before you can visualise ways of rearranging the blocks or moving your hand between the blocks, as described above in Section Possible moves. Look closely at the differences between the last two configurations. The left and middle pictures (J and K) depict perfectly possible 3-D configurations of cubes (though in a normal gravitational field something would be required to hold in place the cubes that are not resting on the table or on other cubes).

But there are subtle 2-D features of the rightmost picture L that indicate that the 3-D configuration that it represents, if interpreted as a picture of 9 cubes, involves a collection of pair-wise relationships between the cubes that are all possible in isolation, but not all possible in the same 3-D configuration. This impossibility does not arise out of any mis-use of pictorial conventions. The image uses only examples of image fragments that occur in other pictures of configurations that are perfectly possible.

The fact that the scene is experienced as impossible only if all the blocks are included challenges theories about limitations of numbers of objects that can be attended to simultaneously.

Examining the image L you should be able to imagine ways of removing one block that would leave the object depicted impossible, and also ways of removing other individual blocks that would render the scene perfectly possible.

When a 3-D scene depicted is geometrically impossible there need be nothing impossible about the configuration of lines in the picture. The impossibility concerns which 3-D structure, if any, the picture depicts if all the parts are interpreted normally as depictions of 3-D structures and relationships.

More challenges to psychology, neuroscience and AI

Compare the above two pictures. A complex picture made of parts representing possible 3-D configurations may have N parts such that if a certain part X is added (e.g. a picture of an extra block that is simultaneously co-linear with two other linear groups, as in the above figure on the right), then it becomes anomalous and cannot represent a 3-D configuration using the same rules of interpretation (based roughly on reversing projections from 3-D to 2-D). Notice that in this case, the addition of X required changes to the (2-D) depictions of blocks A and H that preserved the 3-D relationships between A and B, and between G and H, but altered the 3-D relationships A and H, depicting A as occluding H. That produces a contradiction even if block X is not depicted. If X were removed, more of H would be visible, but the impossibility would remain.

In other words the original N parts have a joint interpretation that entails that the situation depicted by adding the part X cannot exist, but not because of the addition of the new block, but because of a subtle change in the relationships between pre-existing blocks. If the blocks were depicted spread out more in space, so that they are not overlapping, this change would not be necessary, but various relationships would become more ambiguous.

This is partly analogous to logical reasoning where N consistent propositions entail that an additional proposition X is false. So its negation can be inferred to be true.

This picture cannot be handled by the Huffman-Clowes line-labelling mechanism described in Clowes(1971) as it requires a richer grasp of geometry than the line-labelling provides (3-D relationships between adjoining or connected portions of an interpreted image). It requires an ontology of opaque 3-D objects with relationships between whole objects, not merely an ontology of edges, vertices and faces, with 2-D and 3-D relationships between them.

Humans can reason that the "obvious" interpretation of the 2D picture on the right represents a configuration that is impossible without knowing any of the actual distances or sizes, whereas I don't believe any current AI vision system can do that, though it may not be very difficult to implement one to deal with the special case of opaque rectangular blocks in static scenes.

The detailed requirements for the richer ontology, if extended beyond 3-D objects bounded by plane surfaces, and beyond rigid objects (e.g. to include objects made of different "kinds of stuff"), and beyond static configurations, will vary for different species of animal, and for different developmental stages in the same species. As far as I know very little of this is in any current AI systems (or psychology, or neuroscience).

I see no reason to believe these capabilities could be acquired by any of the forms of learning currently fashionable in AI/Robotics. Much deeper epigenetic mechanisms are required e.g. as speculated in connection with Figure Evo-Devo, below.

This requires researchers themselves to develop deeper (meta-cognitive) ideas about forms of geometrical and topological perception and reasoning.

Moreover I don't think neuroscientists have any idea how brains can support this kind of reasoning. (Please let me know if I am wrong!)

The above example is partly comparable to a collection of sentences, each of which describes a perfectly possible state of affairs, though their conjunction does not, e.g. "Tom is older than Dick", "Dick is older than Harry" and "Harry is older than Tom". Older than is a transitive relation, which means that
"X is older than Y" and "Y is older than Z" implies "X is older than Z"

So the first two conjuncts above imply "Tom is older than Harry" but that contradicts the the third one because it is not possible for Tom to be older than Harry while Harry is older than Tom. ("Older than" is an anti-symmetric relation.) Why it is impossible, and how it is possible for an individual (human, other animal, or intelligent machine) to know that a relation is transitive and anti-symmetric, will not be discussed here.

The situations depicted in the pictures of blocks are more complicated than the linguistic example because there are several different relationships, including "further from the viewer", "higher than" or (further above the surface of the table), and "further along" in various directions in the scene, all of which are transitive and antisymmetric relations. It is left as an exercise for the reader to work out which 3-D relationships between blocks or between groups of blocks are depicted in the various pictures, and which combinations are inconsistent.

The pictures of impossible objects and sentences describing impossibilities both illustrate a deep and important point: if a form of representation (pictorial or linguistic) is to be suitable for expressing information about a rich domain of possible structures and processes, there may be no syntactic constraint on pictorial or verbal modes of composition, that is GUARANTEED to prevent self-contradiction AND provides the desired expressive power. In general discovery of features like transitivity and self contradiction can't simply be based on definitions or logical forms. Other modes of discovery -- using topological or geometric insight -- are needed. As far as I know, there are no psychological, neural, or AI theories explaining such capabilities. I suspect we'll need to understand sub-neural chemical information processing, with its mixture of discrete and continuous processes, in order to fully understand how ancient brains discovered many examples of mathematical impossibility and necessity.

3 + 5 < 6

The ability to notice impossibilities is an aspect of intelligence. For example, an animal thinking about possible ways to move a physical object to get it from one location to another location should be able to detect, at least in some cases, that moving the object through a particular gap is impossible, because the object is too wide. The ability of a 2-D structure to depict an impossible 3-D structure, and the ability of (some) humans to detect the impossibility have been much studied. But I think many of the examples have subtle clues about functions and mechanisms of vision that have not been appreciated. This is particularly evident in the 1934 picture produced by Reutersvard, presented below.

In the above examples, I have tried to show how to build up to Reutersvard's example gradually, in order to get an accurate account of the phenomenon, by locating the discovery of impossibility in a space of possible actions using vision along with a visualisation of a target configuration.

Although James Gibson did not, as far as I know, ever discuss such examples (or the others mentioned below), I hope it is clear that the discovery of such impossibilities could arise in the context of perceiving affordances in the environment, i.e. possibilities for change of spatial relationships, and using those perceived affordances to construct something. The formation of an affordance-based intention generally leads to either a plan or an exploratory process that eventually culminates in construction of the intended spatial configuration. But in special cases it is possible to discover that the intention cannot be fulfilled because of negative, obstructive, affordances, which in some cases cannot be overcome using greater strength, new materials, collaboration with helpers, etc. I suspect that such discoveries could, over hundreds or thousands of years, have led our ancestors to formulate mathematical theories about spatial structures including discovering proofs of the sort presented by Euclid (Elements).

But even if these historical conjectures are correct, that still leaves us with the problem of explaining how the impossibilities are understood: what sorts of cognitive mechanisms allow proofs of impossibility to be constructed? What forms did those proofs take? How are they related to the processes in the mind of a child, or a squirrel, or a crow, who not merely tries and fails in a task, but comes to understand the failure? At present I don't believe there is anything in psychology, philosophy, neuroscience, or AI that provides a rigorous explanation. Part of my reason for collecting a large and varied set of examples is to build up requirements for explanatory mechanisms. Combined with evolutionary investigations and exploration of designs for intelligent robots, we may be able to come up with a good theory that can be implemented and tested. I don't think anyone has such a theory at present.

Reutersvard 1934

The above examples were inspired by a picture originally drawn by Swedish artist Oscar Reutersvard in 1934.

An important feature of this picture is the number and variety of possibilities for change implicitly depicted: e.g. all the places where you could put your hand between two of the cubes, all the cubes that could be removed leaving a gap, producing a possible 3-D configuration, all the pairs of cubes that could be swapped, etc. So we have a very rich collection of imagined structures, relationships and possibilities for change, including cases where what is imagined is geometrically impossible. Compare the following description of a collection of numbers.

Reutersvard went on to produce many variants of his idea and a selection of his pictures were used in Swedish stamps. More pictures by Reutersvard are available here:
http://www.step-hen.dk/reutersvard.htm

It is arguable that Escher's pictures of impossible objects or scenes were more subtle and creative, with their rich blends of geometric and biological forms, but that feature is not relevant to our current discussion:
http://www.mcescher.com/gallery/impossible-constructions/

Pre-20th Century pictures of impossible scenes

Although 20th Century examples are often treated as original, much older artists had discovered the possibility of producing drawings or paintings of impossible scenes, for example Hogarth's Satire on False Perspective (1754) shown here:

Mis-use of the word "illusion"

Pictures of impossible objects are sometimes erroneously described as "illusions". They are illusions for viewers who accept all the 3-D structures depicted and do not detect the global impossibility (e.g. young children, or adults looking at very complex examples of such pictures). But not everyone is deceived, which raises the question: what visual/cognitive mechanisms enable non-mathematicians to realise that a picture is of something impossible, so that they are not deceived and suffer no illusion, though they may be entertained, or even amazed by the pictures? However special cases can be illusions, like the next Figure.

In contrast, the example above is illusory, because the 3-D printed triangle gives the appearance of having an impossible 3-D structure that it does not have. It really is a 3-D object, seen from a special viewpoint, but it is not a triangle. It uses a technique first demonstrated by Richard Gregory, many years ago. For more information on the origin of this triangle see:

Tangled bodies
Added: 15 Oct 2017

In 1971 Max Clowes gave a lecture entitled 'Man the creative machine: A perspective from Artificial Intelligence research' at the Institute of Contemporary Arts (ICA), in London. A chapter based on the lecture was later included in a book of ICA lectures Clowes(1972). One of his examples (below) used a "Still" picture from the movie Sunday Bloody Sunday, showing tangled male and female bodies. Finding which visible parts in the picture belong to which person is non trivial, illustrating one of his favourite slogans "Perception is controlled hallucination". (E.g. how many people are in the scene? Which hands belong to the man?) Below I have made two sketches of the scene, one of which corresponds to the movie "still" used in the lecture. The other has a part misplaced to an impossible location. Which picture depicts the impossible scene, (a) or (b) and why? Which hands belong to whom? How many noses can you see? Answering the above questions in relating to the (possible) scene depicted requires detecting impossibilities in erroneous interpretations -- filtering interpretations.

Figure Bodies

Consider the differences between the two figures.
Everyone depicted is only partly visible
How many people are visible in Figure (a)?
How many in Figure (b)? How do you know?

What brain mechanisms support the information processing that allows you to generate interpretations of fragments of the image, and also to detect and rule out combinations of fragments that represent impossible configurations? Do you have to be trained on many examples of tangled arms, legs and bodies? Or can generic spatial reasoning abilities be combined with structural knowledge about human bodies.

I suggest that the mechanisms involved in rejecting wrong interpretations of ambiguous views of spatial structures in novel scenes are the same sorts of mechanism as were used by ancient mathematicians, in making geometrical and topological discoveries. At this stage I suspect nobody knows how biological brains can do that sort of thing. I'll turn now to a set of different, though partly related, examples.

Impossible transitions on a grid

Consider a grid on which it is possible to slide square tiles horizontally or vertically on the surface of the grid, like the one below. Here is a problem that is likely to be found insultingly easy for anyone reading this, but is a gateway into some much deeper problems, not all of which will be presented here.

In the above configuration, is it possible to move the red tiles on the right, one by one, by sliding horizontally and vertically to the locations of the grey tiles on the left, so that each occupied square has exactly one grey tile and one red tile? If not why not? Would it make any difference if those red tiles had started in a different configuration, or if the grey tiles had been in a different configuration? Is it necessary to try all possible routes through which the tiles can be slid (subject to the horizontal/vertical constraint) in order to discover that the goal cannot be achieved? Remember this is supposed to be an easy question.

A slightly harder question: what cognitive capabilities would a child, or a robot require in order to answer the question.

An even harder question: what kinds of evolutionary transitions might have produced the capabilities required for this task? (Later we shall contrast physically constrained and rule-constrained versions of the same problem.) What sorts of naturally occurring situations might have this sort of structure? What kinds of brain mechanisms might have been required in our earliest ancestors capable of answering this sort of question?

Some brain mechanisms would allow the question to be answered only for small groups of tiles, e.g. two or three of each colour. Do your brain mechanisms have such a limit, and if not why not?

Could a robot acquire the required abilities by being trained on lots of examples? If not, how could it be given those abilities -- i.e. what sort of artificial brain could deal with such problems, without being restricted to particular sizes of grid and particular numbers or configurations of the red and grey tiles?

The next example is partly similar, and partly different. Similar questions can be asked about brain mechanisms required, and their evolutionary origins.

Figure Accessibility

(a) Only horizontal/vertical moves allowed (b) Only diagonal moves allowed

Some readers will recognize this as mathematically related to a very well known puzzle that at first looks totally unrelated. If you don't recognize it follow [*]this link. (There may be better explanations on the internet.)

What kind of information processing system can allow an animal or robot to think about what transformations between configurations are and are not possible?

How is this related to mathematical discoveries in geometry, topology and arithmetic?

By what means could those discovery abilities possibly have evolved? What sorts of transitions in functions of vision, or more generally functions of perception, might have led up to such competences?

Can these mechanisms and processes be implemented using current tools and ideas for programming intelligent robots?

Clearly newborn human infants cannot answer these questions about objects sliding around on grids, or similar questions using different spatial configurations. Why not? In what ways would their brains have to change, or be changed, in order to enable them to think about such problems and not only work out the answers but understand that the answers are necessarily correct, and could not be different for grids and tiles made of different materials, or located at different altitudes above sea level or even on another planet?

How might Gibson's theory of perception of affordances have to be revised to cope with these questions?

While thinking about the above problems did you consider the possibility of altering the grid of squares so that they have two colours, like a chess board, with diagonally adjacent squares the same colour and horizontally or vertically adjacent squares different colours. That transformation makes the answers to the second set of questions, with two types of grid, trivial to answer. See this link[*] (same link as above).

What kind of brain mechanism in a human or a robot allows that sort of solution to be discovered and used in a proof? Curiously, many highly intelligent humans who already have the required brain mechanisms and knowledge of chess boards don't notice the relevance -- perhaps because they have too much potentially relevant knowledge.

For example, in my experience some expert mathematicians immediately notice that assigning a coordinate frame to the grid gives each square two coordinates and their sum (or difference) is either odd or even. So they try to find answers in terms of parity-preserving operations, using their knowledge of arithmetic and algebra. This leads to a mathematically acceptable solution, but non-mathematicians who merely notice the consequences of having two colours alternating horizontal and vertically, as on a chess board, can also find a mathematically acceptable proof without mentioning coordinates of squares or division by 2, etc. They are simply using a different sort of mathematics, closer to the reasoning in Euclid's Elements

What sort of brain mechanism is required to enable a person presented with a problem to notice the relevance of apparently unrelated knowledge? [*]

Impossible number relationships

Returning to our blocks on a table scenario, we can consider ways of forming neat rectangular arrays of blocks using some or all of the blocks. The figure below shows what can be done with all nine blocks, and what happens if you remove first one block then another. How do you know that after removing the second block the remaining blocks cannot be arranged in a regular NxM 2-D array?

If we put all nine movable cubes depicted above on the table we can produce different configurations on the surface of the table. In (A) there's a regular NxM array, with N and M both = 3. If we remove one cube, as in (B) we can arrange the remaining cubes in a regular NxM array, e.g. N=4 and M=2. If we remove yet another cube, as in (C), can we still produce a regular array using the remaining cubes? In which situation is it possible to arrange the remaining blocks in a 3-D configuration of NxMxO blocks, where N, M and O are all > 1?

Note that unlike the previous examples, in Case (C) we can't draw the impossible configuration under discussion, namely a regular NxM array made up of exactly seven blocks. Case (C) would be dealt with easily by someone who has already learnt about prime numbers, and understands the unique factorisation theorem. But perhaps playing with the cube rearrangement task could lead a bright child to notice the impossibility, and eventually prove that the problem is not one of a failure to explore enough configurations, even without previously having learnt about prime numbers. Exercise for the reader: what would have to go on in the child's mind for this to happen?

One of the features of mathematics is the variety of interconnections between different problems. Sometimes essentially the same mathematical problem is discovered in quite different contexts. I think even human toddlers and intelligent non-human animals can make such discoveries and use them, but without being aware that they have done so, and consequently not being able to ask or even think about the question: "How do I know that no exception will be discovered on a high mountain, or at a freezing temperature, or while travelling in space?" They are incapable of noticing the epistemological features of the mathematical discoveries they use. And if they grow up to be philosophers with no understanding of computation, they may misdescribe what they have learnt.

Discovering primeness geometrically

A child given a set of wooden cube-shaped blocks can do all sorts of experiments -- exploring the space of processes involving the blocks.

A child may notice that in certain cases, attempts to rearrange a configuration into a rectangle always fail: What kind of experimentation can that provoke, and what sorts of discoveries can be made?

How could one be sure that there is NO way of arranging the last collection into a rectangular array, apart from the straight line shown? Could a child playing with such blocks (or discs, or other movable objects) discover the concept of a prime number? I suspect it could be done using an ability to "carve up" a spatial region in a systematic way and then confirm by exhaustive analysis that no possible distribution of the blocks produces a rectangle, other than the co-linear arrangement.

Could the child rearranging blocks discover and articulate the fundamental theorem of arithmetic? (The unique factorization theorem.)

Physical constraints vs rule-based constraints

The sliding tiles puzzles above were introduced as if the routes along which the tiles could slide were physically constrained. Related problems can occur naturally when the task is to find a route from some location to another location satisfying some constraints.

However the physical world leaves open the possibility of lifting the tiles and moving them to new squares, a process that would be unconstrained in both varieties of the puzzle presented earlier. Someone wishing to achieve the specified end state could therefore ignore the constraints and make more direct moves.

But for some reason there could be a preference for moving the tiles along the constrained paths, requiring routes to be found, when possible. In that case, the preference could arise even if there were no physical constraints: one can "playfully" explore what is possible (a) when only vertical and horizontal moves are considered and (b) when only diagonal moves are considered. The same sort of mathematical reasoning would be relevant both to the situation with physical constraints and to the situation with non-physical constraints, only freely adopted rules.

One consequence of this is that an engineer who discovers "in the abstract" that certain sorts of constrained moves would have useful consequences in some situation can then construct mechanisms in which physical structures impose those constraints, so that all the permitted changes in those physical structures necessarily have the desired properties. Examples include designing channels for flow of water or other liquids, designing grooves along which balls can roll, designing rails to control motion of trucks, designing linkages (e.g. to produce bi-stable car boot (trunk)) lids, designing gears that control the relative speeds of rotation of two axles (and things attached to them), and many more. (The use of tools, a focus of much research in psychology, is a special case of this phenomenon: tools are aids to controlled matter manipulation. Some of them transform forces in addition to constraining motion, e.g. levers, gear-wheels, screw-drivers, pincers, etc.)

The fact that mathematical investigations can be addressed in contexts where structural constraints are "freely adopted" could lead (and I think has led) some philosophers to the mistaken conclusion that mathematics is a human creation, and contains only freely created constructs, with no absolute necessities. Wittgenstein famously wrote: "For mathematics is after all an anthropological phenomenon" (in Remarks on the Foundations of Mathematics). But clearly the consequences of freely adopting a precisely defined constraint are not themselves freely adopted, any more than the consequences of a strong physical constraint with the same structure are freely adopted.

Moreover, it is not only humans who discover and use mathematical structures and their properties: other intelligent animals do also. E.g. weaver birds make use of mathematical properties of knots. Moreover evolution (natural selection) has discovered and made use of many mathematical facts, e.g. that certain sorts of control systems (using negative feedback to achieve homeostasis) will produce stable temperatures or pressures or orientations. Many more subtle mathematical facts must have been used to allow control systems in developing brains to control various kinds of motion by changing their details (their parameters) during growth of an organism, a process in which absolute and relative sizes of body parts change, and their weights, moments of inertia, strength and other features change, but without requiring growth of new brains (or sub-brains) to control the physical configurations with all their new properties and relationships.

Such control mechanisms can be thought of as using primitive "grammars" for processes. This may be relevant to unanswered questions about evolution of languages for internal use and for communication. Sloman[Vis-Lang].

This discussion raises interesting biological questions. What differences are there between brains that can solve problems involving satisfying constraints only when the constraints are externally imposed, e.g. by physical structures, and brains that can also solve similar constraint satisfaction problems where the constraints are not freely chosen, but adopted for some practical reason, e.g. using rules as constraints in reasoning about a diagram representing a physical structure with corresponding physical constraints, like an architect deriving consequences of some design decisions by reasoning with architectural drawings.

This is part of the evidence that evolution can be usefully considered to be a "blind mathematician", a view discussed in several of the Meta-Morphogenesis project papers.

In the context of the discussion of evolved construction-kits in a separate document construction-kits.html a distinction is made between physical, abstract and hybrid construction kits. A game in which players are constrained by physical structures is a kind of physical construction kit. If some of the physical constraints are removed, like the constraints restricting the trajectories of tiles on a surface, but the effects of the constraints are adopted as constraints on solutions to problems (i.e. rules restricting possible actions), then the result is a "hybrid" construction kit. Many games played by humans, such as soccer, cricket, tennis, and others are based on such hybrid construction kits.

In some cases, such as chess, or draughts (checkers) or GO it is even possible to play the game without making any use of physical pieces or a board, by simulating their effects. Such a game is then a purely abstract construction kit. A great deal of mathematics is concerned with investigation of properties of such abstract construction kits. But many of the original mathematical discoveries were based on concrete versions of those construction kits, where constraints (or rules) came from perceived physical structures not from intentions of players or social agreements. These developmental steps are nothing like the repeated use of statistical evidence in neural-net based models of learning, e.g. in "deep learning" mechanisms.

How and why and when and how many times, did evolution produce new brain mechanisms making novel problem solving (and problem recognition) capabilities possible? Which were the earliest cases in evolutionary trajectories leading to modern humans? Producing such mechanisms is itself a partly mathematical problem, of finding structures and rules that support the discovery processes. The ability to solve problems arising in the use of abstract construction kits, benefited from earlier evolutionary transitions that solved useful practical problems. But the new mechanisms constructed often had additional powers beyond the powers required to deal with the specific problems encountered.

Added 10 Nov 2016
Some of these ideas are taken a bit further in a paper (written October/November 2016) discussing the nature of mathematics and the argument that mathematical discoveries were made and used by natural selection long before there were human mathematicians.
http://www.cs.bham.ac.uk/research/projects/cogaff/misc/what-is-maths.html
(That was followed by later papers on mathematical discovery on the same web site.)

Domestic possibilities and impossibilities (furniture)

Domestic life is full of possibilities and impossibilities that you may or may not care about. Many people have discovered impossibilities involving doors and chairs illustrated in the example below. If you wished to slide the chair out of the room through the doorway, you might find it impossible because the chair is too wide. It is possible to rotate such a chair about a vertical axis (e.g. rotating it with all four legs in contact with the floor), but that does not allow the chair to pass through the door, if the width of the seat and also its depth (front to back) both exceed the width of the door.

However, in such a case it may be possible to rotate the chair in a different way, about a horizontal axis, so that it ends up on its side, with its feet pointing horizontally instead of downwards. In that orientation the chair may be able to pass through the doorway if it is slid until two feet are projecting around the door frame then rotated so that the other two feet can move through the doorway, after which the chair can be rotated again into the original vertical orientation.

If height of back of chair is less than width of doorway, the chair on its side can pass through doorway if the back feet are pushed through first, then the chair slid to the right, then the chair rotated around a vertical axis to get the back past the door frame after which the seat and front legs can be pushed through.

Can you see an alternative possible arrangement that allows the chair to go through the doorway (without being dismantled, folded, etc.)?

The chair example illustrates the fact that realising a new possibility (rotation) can remove some old possibilities (e.g. sitting on the chair) and introduce some new possibilities (e.g. getting through the doorway). Furthermore such removal and creation of possibilities can be chained to form a sequence of steps to bring about a new situation that could not be achieved by simpler direct motion of an object from its initial position to a desired new position.

Of course the possibility of getting the chair through the door existed from the start, but it was temporarily blocked by the orientation of the chair and the width of the door. Changing the chair's orientation produced a new direct possibility.

Many action plans created by humans and other intelligent animals depend on the ability to recognize and reason about sets of possibilities and impossibilities, and ways in which they can be chained usefully.

Constructive, supportive impossibilities

Sometimes the impossibilities are obstructive, like the impossibility of moving the chair through the doorway in its original orientation. But other impossibilities provide essential constructive support for new possibilities. For example if you cannot reach something on a high shelf, you may be aware that there is a rigid box that you can place on the floor below the desired object. The rigidity of the box may remove the possibility of your being moved downwards by gravity if you stand on the box. In that case standing on the box creates a new possibility for you, reaching the object on the high shelf.

This is a common feature of biological processes: some new possibility once realised removes previous possibilities (moving downwards) and thereby enables further new possibilities (reaching to a new height). This depends on the rigidity and strength of the material used. The ability of plants to grow upwards (e.g. to get more light) depended on evolution of mechanisms making possible the growth of materials that reduced or removed possibilities of bending, i.e. rigid materials. Without such changes to the materials constructed during development, plants could not have evolved on dry land as we know them. Giant redwood trees are an extreme example. Roses or sunflowers held up by stalks would also have been impossible.

Terrence Deacon (2011) seems to me to be confused about achievements of biological evolution because he emphasises the negative aspects (constraints) in new developments (e.g. production of rigid materials) without noticing the positive aspects (e.g. enabling new possibilities, such as supporting heavy structures). Anyone who has played with construction kits, such as Meccano, will have made use of the fact that separate parts can be locked together (e.g. forming a hinge) thereby both constraining their independent motion and enabling new possibilities, such as building a structure with a part that can move relative to another part, such as the jib of a crane. Related points about molecular level construction kits were made in What is life?, by [Schrödinger 1944]. It seems to me that in that little book he came very close to recognising the practical importance to many organisms of abilities to recognise examples of impossibility and necessity. but did not quite get there.

So new developments may be both negative (constraining) and thereby also positive (enabling) at the same time. This is part of the intrinsic nature of possibility and necessity in spatial structures, relationships and processes, with rich implications in structures with non-rigid spatial relationships between rigid parts. See also Sloman (1996).

Being aware of such relationships and their implications is an important feature of (well developed) human perceptual consciousness, and apparently also consciousness of some other intelligent species.

Impossibilities concerned with cardinality

The following is a subset of the discussion on "What needs to be explained" added recently as a new Note to Chapter 8 on "Learning about numbers" in Sloman (1978).

Piaget, unlike many researchers into number-cognition, had read deeply in philosophy of mathematics (including work by Gottlob Frege and Bertrand Russell) and knew that understanding one-one correspondences is central to understanding the natural numbers (as cardinals). Consequently, whereas many researchers assume that being able to recognize and name the numerosity of small collections of objects is evidence for possession of a concept of number, Piaget realised that far more is required: in particular a grasp of features of one-one correspondences that children acquire only gradually. In some cases they are not understood fully until close to age 6 years (Piaget, 1952). [The precise ages are not important: the possible forms of partial understanding are.]

Chapter 8 of Sloman (1978) attempted to illustrate some of the algorithmic and architectural requirements of a learner developing information about number names and how to use them in various practical tasks, all of which depend on the use of one or more one-one correspondences, including correspondences between objects or events and an initial sequence of number names. Some of the implicit themes were made explicit in the Note to Chapter 8, added in 2016. This section summarises a subset of that note.

In his (1952) Piaget used a variety of experiments to probe the ability of children to recognize and make use of one-one correspondences, and their abilities to reason about those correspondences, e.g. answering questions about whether and why the correspondences are or are not preserved by various actions. Many researchers attempted to replicate, or modify his experiments, but often labelled what they were studying as something like "understanding conservation", without any theory of what made such understanding possible. A useful summary of some of this work by Saul McLeod with videos can be found here: http://www.simplypsychology.org/concrete-operational.html

Piaget famously discovered the apparently staged development of the ability to understand that it is impossible for a one-one correspondence to be destroyed by a mere re-arrangement of the objects involved. His postulated stages of development need not concern us now.

Understanding that sort of "invariance" is essential to understanding the cardinal numbers. But before children reach full understanding many of them seem to regard a rearrangement that stretches out or compresses a collection of objects as altering numerical equality between that collection and another collection. So even if they have seen and accepted the original correspondence and have also seen objects being moved to form a perceptibly longer collection, but without addition or removal of any objects (i.e. simply creating a new one-one correspondence between initial and final elements of the groups) some of the children apparently think that this rearrangement changes the one-one correspondence between the objects as they were and the objects in the new configuration. But the questions asked do not normally explicitly refer to one-one correspondence. The experimenter may ask whether there are more objects than before the rearrangement. When children mistakenly say there are more, they may be answering the wrong question, or they may not understand the invariance. Figure Transitive below, illustrates a special case of the problem.

Figure Transitive

As far as I know, Piaget was not able to explain how a child (or adult) can see in both Case 1 and Case 2, that if the first two correspondences exist the third must also exist. My impression is that many psychologists who have read or heard about this work by Piaget do not understand the deep implications of the computational requirement to represent and reason about 1-1 correspondences. So the label "conservation" is used to sum up what the children have or have not understood when they succeed or fail in Piaget's tests. And, as Annette Karmiloff-Smith once remarked, they try to vary the tests to find out whether children can pass some variant at an earlier age, but without providing any analysis of requirements for passing or for mechanisms that can meet those requirements.

Frege and Russell essentially tried to show that this is merely a case of (rather complicated) logical deduction that could be expressed in the symbolism of modern logic. However the ancient Greeks and many others had already discovered and used such properties of numbers long before the invention of modern logic.

Understanding the concept of cardinal number includes understanding why a one-one correspondence between two collections of discrete items is preserved no matter how the items are re-arranged, as long as no objects are removed, merged or separated into two or more parts. Piaget's work showed that this understanding does not come automatically with being able to count or being able to answer questions correctly in special cases. I suspect that neither Piaget nor anyone else knows how brains represent information about particular one-one mappings or acquire abstract non-empirical knowledge about general properties of transformations that involve one-one mappings. The concept of a one-one correspondence between two arbitrarily large collections of objects of any type (concrete, abstract, physical, mental, etc.) is not one that fits any mechanism I have ever heard a neuroscientist describe. Without that, our concept of cardinal number cannot be understood. I suspect animal brains, and especially human brains, use important mechanisms that have not yet been identified by neuroscientists.

Frege 1950 attempted to show that such mathematical knowledge is purely logical, but it is clear that mathematicians understood these properties of cardinality before the logical apparatus used by Frege and others had been discovered. I think the suggestion that humans had been using that logical apparatus subconsciously for hundreds (or thousands) of years before Euclid wildly implausible, but will not discuss that here. Since December 2020 I have been arguing that competences of newly-hatched chicks, ducklings, etc. provide evidence that biochemical mechanisms can provide the required competences -- which may then be extended by brains and interactions with the environment, after hatching. See the video presentation at Sussex University on 16 Feb 2021 reported here: http://www.cs.bham.ac.uk/research/projects/cogaff/misc/sloman-chemneuro-sussex.html

In order to understand how an AI system can understand the natural numbers as they were understood before the rise of logic, we shall have to explain how it can reason visually (e.g. using a diagram, or imagining a possible change in some collection of objects) and thereby discover that one-one correspondence is transitive (among many other properties). The figure illustrates what is discovered but does not explain how. I don't think anyone knows how human brains make such discoveries, how the discovered information is stored, how the brain mechanisms allow future inferences to be made, and how all this knowledge is acquired in a form that is independent of how many objects are involved, how big or small they are, what shape they have and whether they are physical objects or locations, or places, or abstractions such as number names, or how the information that there are infinitely many possible cardinalities is represented in brains. These cannot be statistical generalisations from perceived examples, since when understood the generalisations are known to have no exceptions. Moreover they can be understood as applying not only to collections previously encountered but to arbitrarily large collections of objects or events or names, etc. How can biological brains support such competences and discoveries? So far I don't think these abilities have been replicated in AI systems, though I suspect that may simply be because we have not yet discovered the right forms of representation and the required information processing architectures.
[To be added: refer to work of Doug Lenat, Simon Colton, Alison Pease and others who have attempted to model mathematical cognition.]

Non-human reasoners about possibilities and impossibilities

A close study of many intelligent non-human animals reveals wide-spread abilities to identify and use chains of possibility and impossibility in complex actions that lead to achievements of goals. A striking example was Betty -- a New Caledonian crow studied in Oxford -- who in 2002 astounded not only the researchers but also many reporters and members of the public, when she managed to use a straight piece of wire to make a hook that she then used to fish a bucket of food out of a vertical glass tube, as reported in Weir, Chappell and Kacelnik (2002) and demonstrated in this Youtube video:
https://www.youtube.com/watch?v=UDg0AKfM8EY.
There are several more videos showing Betty's behaviour here:
http://users.ox.ac.uk/~kgroup/tools/movies.shtml.

It was not reported at the time, but is clear from the videos on the project web site (a) that she makes the hooks in several different ways, with (approximately) functionally equivalent results (i.e. using a crack in the plastic tray to grip the end of the wire, using the tape at the base of the tube to grip the wire, using her foot on a horizontal rail to grip the wire, and using a hole in the wall next to a small perch to grip the wire, (b) that there appears to be no random trial and error in her hook-making or hook-using behaviour, (c) that each of the episodes of hook-making and use involves several different steps in which some possibility is identified, and then new possibilities and impossibilities are achieved on the basis of the previously realised possibilities. There is no evidence that she had had experience of bending pieces of wire, or similar materials before these experiments, although when she first bent a straight piece of wire she had previously used a bent piece of wire provided by the experimenters.

A detail that is easily overlooked is that when she inserts the hook into the tube she also places one foot on the rim of the tube. That provides two widely separated support points, presumably allowing more precise control of the wire when moving the end under the handle of the bucket. Later, she uses the foot on the rim to achieve sufficient height to pull the hook and the bucket out of the tube. Examining the last few seconds of the video suggests to me that without the foot on the tube she would not have been able to achieve the height required. How much of that Betty understood is not clear, but in the video of trial 7 she did not first try without grasping the rim.

Toddler Topology

Piaget seems to have been aware that for young children, understanding and using topological relationships (relationships between structures that involve continuity, orderings and "betweenness", but without using metrical properties, e.g. length, area, volume, exact orientation, degree of curvature, etc.) played an important role. An example is understanding the advantage of pushing a drawer shut with a flat palm rather than using a hand curved over the top edge of the drawer, as explained here:
http://www.cs.bham.ac.uk/research/projects/cogaff/misc/toddler-theorems.html#drawer

Another sort of example is understanding some of the things that can be done with holes, an example being Betty's ability to pass the end of a hook through the "hole" provided by the bucket and its handle. A video of a pre-verbal toddler apparently using a pencil to explore different routes through a hole in a sheet of paper is here:
http://www.cs.bham.ac.uk/research/projects/cogaff/misc/toddler-theorems.html#holes
She seems to be testing a hypothesis about 3-D topology (the existence of continuous deformations between two configurations) though she could not, at that age, have formulated such a hypothesis in words.
As far as I know there is no current robot that can become aware of such a possibility and thereby acquire the motive to make the possibility actual, which seems to happen a great deal with human children (illustrating conjectures about "Architecture-based motivation" Sloman (2009b)).

The Side Stretch Theorem

A separate discussion document concerned with changing areas of triangles Sloman (2012) introduced the Side Stretch Theorem (SST), illustrated below

This illustrates a deceptively simple example of a mathematical relationship between a type of spatial structure, types of process that can occur, and types of change necessarily associated with those processes. The simplicity is deceptive because although you are likely to find the claimed mathematical relationship obviously true, it is not at all clear what sorts of visual information processing mechanism provides the ability to think about the possibility of change when it is not actually occurring and find the relationships obvious. For example, the formulation in terms of an area increasing or decreasing presupposes the existence of a shape independent measure of area, and that any such measure exists is not obvious. Clarifying that concept for an arbitrary area was a major mathematical achievement, closely related to the discovery of integral calculus. However, there is a simpler mathematical discovery that does not require the notion of a measure, only the notion of inclusion or containment, a part-whole relationship.

The Side Containment Theorem (SCT)

It might be fruitful for the reader to pause here and try to formulate and prove a Side Containment Theorem (SCT) expressed in terms of which of two triangles contains the other, without assuming any measure of area or length, illustrated in Figure SCT:

For now, I'll leave open the question whether the Side Stretch Theorem (SST-in/out), or the Side Containment Theorem can be derived from something more basic and obvious, requiring biologically simpler, evolutionarily older, forms of information processing.

Note, however, that the notion of the vertex being "moved along" a line "away from" or "towards" another point on the line implicitly makes use of a metrical or semi-metrical (ordering of lengths without a numerical measure) notion of length, which increases or decreases as the vertex moves. The concept of motion between two locations on a line also implicitly assumes the existence of intermediate locations between those locations.

Properties of triangles and other abstract shapes

The thought experiments discussed there could all have been performed by ancient thinkers long before the development of modern logic and formal methods of proof. Such pre-logical mathematical thinking was the only kind available to Euclid and his predecessors, since modern logic was developed only in the last few hundred years.

Conjecture: the ability to make those mathematical discoveries, and others in Euclid's Elements were dependent on abilities to notice and reason about possibilities, of the sorts discussed above, along with other social abilities necessary for publishing or teaching the materials.

Work to be done still includes identifying the precise visual functions needed and precise specifications for the mechanisms providing those functions. We may then be able to build artificial systems (e.g. robots) whose visual and mathematical capabilities are far more like those of humans than any existing robots.

Until we know how to do that, our robots and other forms of AI will all be severely limited and capable only of very simple forms of learning. They will not be capable of the forms of learning and discovery that drove mathematical discoveries in humans.

Strings

There are many ways flexible strings can be moved around. In particular, a string can be threaded through one or more holes in a piece of leather (as in a shoe). Suppose it goes through only two holes: how many different ways are there of removing the string from the holes? How can you be sure that you have counted them all? [Assume two removal processes are the same if the ends of the string go through the same holes in the same direction.]

You can remove the string by pulling one end, or by pulling the other end. Why can't you remove it even faster by pulling both ends? What needs to be added to current robots to enable them to (a) discover such impossibilities, (b) understand why they are not possible?

If you pull both ends at the same time, there is a configuration that can be achieved faster: what configuration? The ability to answer that might be based on searching through a mass of data concerning previous pulling episodes. But that isn't required. What sort of ability would enable a robot to answer the question without resorting to experiments with strings and holes, and without searching through stored records of previous such experiments? How do you answer the question?

Strings, pulleys and levers

In Sloman (1971) and in Chapter 7 of Sloman (1978) a comparison was made between reasoning processes using Fregean forms of representation (I.e. representations based on application of functions to arguments, e.g. logical and algebraic notations) and reasoning using "Analogical" forms of representation (often confused with representations based on isomorphism between thing represented and thing representing -- which is merely a special case).

Such diagrams of mechanisms can be interpreted subject to constraints, such as that levers are rigid and can move only by rotating around their pivot points, and strings are flexible but unstretchable. Using such assumed constraints it is possible to reason that if end a of the left lever moves up then end f of the right lever will necessarily also move up, so that both levers will rotate clockwise about their pivot points (indicated here by small triangles). Whether actual strings, levers, pulleys, etc. have the postulated properties is an empirical question, but the consequences of having them can be derived by non-empirical reasoning, using the diagram.

It was argued that such reasoning could be as useful, and as reliable, as reasoning based on logical and algebraic forms of representation. In both cases the reasoner has to make assumptions about how the form of representation works, i.e. what various structures and transformations represent, and on the basis of such general assumptions draw conclusions by reasoning about particular cases. These forms of reasoning, familiar in uses of maps, in uses of diagrams in physics and electronics, and in architectural drawings are all examples of valid reasoning that is not based on statistical evidence or inferred probabilities.

In simple cases such reasoning can make use of imagined transformations of an imagined diagram: there is no need for a physical diagram to be used. (A similar comment applies to reasoning using a logical notation, or natural language: the reasoning can be done "in one's head".)

I suspect that pre-verbal children and some highly intelligent non-human animals (e.g. squirrels and crows) are to some extent capable of such valid reasoning using non-Fregean forms of representation internally, but unlike adult humans they are incapable of reflecting on what they are doing, communicating it to others and defending or criticising the validity of the reasoning. That requires meta-cognitive extensions to the information-processing architecture.

Topological additions October 2017

Superpositions
There are several examples of impossibility of two object views being views of the same object after translation, rotation or viewing from a different direction. This includes:

There has been considerable psychological research on human "mental rotation" abilities, initiated by Shepard and Metzler, summarised in https://en.wikipedia.org/wiki/Mental_rotation, comparing difficulty of tasks, times required to detect rotation, etc.

The following figure (from Wikipedia) includes typical 2D and 3D examples of mental rotation challenges.

For a machine vision system to perform this task the fact that the structures compared decompose into straight segments, or segments composed of cubes or rectangular blocks meeting at right angles makes it possible for a relatively simple algorithm to check whether superposition is possible in a finite number of steps (left as an exercise for the reader).

But as far as I know nobody has investigated brain mechanisms that could enable brains to detect impossibility of superposition by rotation and translation. I suspect individuals use a large collection of learnt heuristics that vary according to culture, age and individual.

Things get far more computationally explosive if more general shapes are used, as illustrated here using 2D shapes as examples. (Compare Minsky and Papert on Perceptrons. [REF])

Figure 2D Congruence
Which pairs of shapes are congruent if translation and rotation in the
plane are allowed, but not flipping over or reflection across a line?

How would the answers change if reflection across a line, or flipping a shape over in 3D, were permitted?

For different pairs of images different heuristic methods can lead fairly quickly to answers, e.g. counting the number of 'ends' in each image (some have four some have two), testing whether a shape can be coloured in more than one colour without any two colours merging (a test for connectedness), imagining a feature of one image being superimposed on the other and following round both images looking for a location of mis-match, and finding what happens if attempts are made to imagine pairs of ends superimposed then checking how far the superposition extends. (That reveals that two of the non-symmetrical images that are reflections of each other about a line in the plane cannot be superimposed.)

By now some readers will have noticed that far from being purely mathematical challenges unrelated to anything else, these challenges are closely related to a common type of picture puzzle for children, where the task is to look at a picture and find one pictures of more more familiar objects forming undifferentiated parts of the picture. The concept of impossibility is relevant to this task in order to rule out answers based on finding a structure that could be part of something (e.g. a bicycle) but whose continuation could not be.

In that sense impossibility detection is part of common recognition tasks. The same can be said of the task of identifying the grammatical structure of sentences (including "garden path sentences") that have a part with a possible interpretation made impossible by another part of the sentence, as in this old example

Changing containment
Detecting impossibility of changing containment relationships:
E.g. consider two closed non-self-crossing curves in a plane. If one curve is inside the other then a continuous deformations can get them to coincide without it ever being the case that one curve crosses the other (i.e. part of curve 1 is inside curve 2 and another part of curve 1 is outside curve 2 during the process of merging).

However, if neither curve is initially inside the other, no such continuous deformation in the plane can get them to coincide.

If the two closed curves are on the surface of a sphere there is no such impossibility. If the two closed curves are on the surface of a torus it may or may not be impossible, depending on their initial locations. See
http://www.cs.bham.ac.uk/research/projects/cogaff/misc/torus.html
The example generalises to two closed continuous surfaces in 3D space: if one is contained in the other then smooth deformations can get them to coincide without there ever being a place where one surface crosses the other (i.e. they never intersect in a line in 3D space). However if neither surface contains the other, no continuous deformations can get them co coincide without the surfaces ever crossing. (What would correspond to the case of two curves on the surface of a sphere?)

What brain mechanisms can detect such impossibilities? Can this be done by any robot vision system so far built?

Other examples

There are more examples in this online PDF slide presentation:
http://www.cs.bham.ac.uk/research/projects/cogaff/talks/#talk79
Talk 79: If learning maths requires a teacher, where did the first teachers come from?
Why (and how) did biological evolution produce mathematicians?
Presented at Symposium on Mathematical Practice and Cognition 2010
AISB 2010 Convention, Leicester, March 29-30 2010.

Evolution: the Blind Mathematician

Biological evolution had to find ways of representing processes in order to be able to specify organisms that grow. Many people have been fascinated by the phenomena of shape change in biological development, e.g. Darcy-Thompson. This is problematic because as organisms get bigger various size relationships tend to change (e.g. head size in humans starts relatively large at birth), weight and strength relationships change, types of opportunity for locomotion, climbing, etc. change, and with these developing physical features the relevant information processing (e.g. control) mechanisms also have to change. Something similar happens after evolutionary changes: old control mechanisms have to be modified. If they were previously parametrised the modifications may be much easier to produce by natural selection. (Compare advantages of GP over GA for solving some search problems.)

The discovery of such useful re-usable patterns essentially involves mathematical abstraction: though in these cases the abstraction is not done consciously or deliberately: Evolution is a "Blind mathematician", implicitly discovering and using mathematical facts long before there were human mathematicians. It can be argued that this provides one of several types of "foundation" for mathematics, as discussed in
http://www.cs.bham.ac.uk/research/projects/cogaff/misc/maths-multiple-foundations.html

Visual process perception is a particularly difficult problem, since, as we'll see perceived scenes can have very complex structures and can change in ways that are not well represented by the kinds of mathematics currently used for dynamical systems e.g. using collections of parameters and their derivatives (of various orders). Such a mode of representation would not, for example, be well suited for describing changes during construction of a meccano model such as the model crane illustrated in this document:
http://www.cs.bham.ac.uk/research/projects/cosy/photos/crane/

Mechanisms of process-perception

Varieties of process-perception involve acquisition and use of many different kinds of perception relevant to ongoing changes.

In any perceived portion of the physical world there will be collections of objects with various properties and various relationships and various changes going on, so that perceptual information includes not only static scene information, but information about processes. It is arguable that for biological vision process perception is the main achievement, and is far more demanding than perception of static scenes, or scenes that are almost static except for the motion of the viewer. For every possible static configuration of objects there are many different possible processes that either start or end with that configuration or transiently include that configuration. So the combinatorics of process perception vastly outstrip the combinatorics of static scene recognition and description.

One of my reasons for selecting a video taken by a camera moved around in a garden is that plant scenes (below) can show a great deal of structure on different levels of scale with a wide range of variation within each type of structure, and with structures that change radically from one location to another and from one view of a particular location to another. So videos, or live views, from changing viewpoints in natural or cultivated environments can provide processes in which the complexity and speed of changes, and the variety of types of change, provide extreme challenges to natural or artificial vision systems.

Those challenges seem to be exacerbated for viewers with two eyes moving at once and constantly having to recompute the information required for perception of 3-D distances and structures. I suspect that very fast and accurate, but low-resolution stereo vision uses large scale corresponding object features (e.g. vertical or approximately vertical edges or portions of edges, of doorways, tables, cups, tree-trunks, etc.) in results of monocularly processed images.

Those results can guide the search for more fine-grained very low level correspondences for precise, close up, stereo vision using small scale image features. That guidance is not available in random dot stereograms, so stereo merging is then much more complex and much slower, though it can give higher resolution when successful. This aspect of stereo vision may have evolved later.
https://en.wikipedia.org/wiki/Random_dot_stereogram

Although useful some of the time this mechanism for very low level stereo vision is not essential for normal vision, and there are significant numbers of people who cannot see random dot stereograms, although their 3-D vision appears to be normal most of the time.

Moreover, whereas an agent moving voluntarily may be able constantly to provide information about the (intended, or actual) changing viewpoint for use by the visual system in integrating information across time and between eyes, viewing a previously produced video does not make that extra motor-control-based information available. Yet for many decades humans have been happily sitting in fixed seats viewing cinema shows and television shows in which there's a great deal of motion, including changes of viewpoint. The fact that intentional or kinaesthetic or vestibular (semi-circular canal) information about viewer motion is not available does not seem to be a great handicap. It's possible that that is true only for animals that spend a great deal of their early life being carried by a parent, so that evolution was under pressure to develop motion perception systems that work without the support of motor control information available to a voluntarily moving agent.

Garden scenes/Gardens seen
Added 18 Oct 2015

Variety of functions of vision

The online functions of vision can include using what's seen at a particular time to help control what's seen immediately after and possibly in using information about what was seen a moment earlier. These influences/constraints on interpretation can spread in all directions across an image, perceived 3-D structure, or process Clowes (1971). (These ideas, developed independently by Max Clowes and David Huffman, were later generalized by others including David Waltz, Geoffrey Hinton, Alan Mackworth, and many more.

Massive constraint satisfaction/propagation challenges

The garden videos presented above illustrate two challenges that I believe are not addressed by current artificial systems described as capable of motion-perception.

One is the complexity and speed of change that can be perceived by the human visual system (even if we don't see as much as we think we do!). This seems to require concurrently changing structural relations of different sorts across both scales and ontologies (also illustrated for static scene perception in Chapter 9 of Sloman 1978). This is the phenomenon Clowes labelled perception involving different domains.

The second challenge is the diversity of ontologies required for perception of different scenes, and different aspects of the same scene, including not only the many different parts of a typical horticultural center, but also changes across types of things seen including maps, handwritten text, printed text, music, computer programs, mathematical notations, and the different written/printed forms some bi-lingual or multi-lingual readers can cope with. This sort of ontological diversity is illustrated in the garden videos referenced above.

It is not obvious to me that current forms of artificial computation or known forms of neural computation can handle such varieties of complexity at the sorts of speeds human visual systems (and presumably also visual systems of other fast moving animals) can deal with.

Question/Conjecture: A biological role for quantum computation?
Could this processing challenge prove to be the first significant and non-artificial challenge for quantum computation -- a much more urgent problem than getting computers to understand Gödel's theorem.

Any such biological solution will almost certainly require the use of sub-neuronal information processing mechanisms (e.g. possibly microtubules suggested by Hameroff and Penrose as solutions for far less compelling and demanding challenges?). [REF needed]

An important point in this context is that the key features of quantum mechanisms required do not include the randomness that is so often emphasized. Instead, what is important in many biological phenomena is the opposite: quantum mechanisms allow structures like chemical bonds to form that are highly resistant to perturbation: a structural feature that is essential for preservation of genetic material from one generation to another, as pointed out by [Schrödinger 1944].

Evolution has clearly used the mechanism of chemistry (and therefore quantum physics) to create enduring structures that persist despite many surrounding changes. If that feature is also used for information-processing, but with much faster locking and unlocking of states, combined with superpositions of sets of alternative possibilities (another important feature of quantum mechanics) then perhaps the use of quantum mechanisms for solving constraint propagation problems that pervade visual processes may turn out to be essential for recently evolved brains.

Is there a need for a Super-Turing "Membrane" machine?

Any discussion of information processing (e.g. acquisition, derivation, construction, manipulation, storage, retrieval, use, evaluation, etc. of information) must implicitly or explicitly take account of possible information "vehicles", i.e. entities that in some way make the information available for use, and mechanisms for using the information (which may be different in different contexts -- e.g. checking the correctness of the information or deriving consequences of the information, or using the information to take control decisions.

Development of digital computers and their applications has enormously expanded the means by which humans can deal with information, including creation of new machines that have begun to do kinds of information processing previously only done by humans and other organisms.

But the spectacular successes do not imply that those digital forms of encoding suffice for all the known forms of information processing. There have been applications that use continuously varying physical properties instead of, or in addition to, collections of binary switches, e.g. analog computing devices.

However, in 1969 McCarthy and Hayes produced an important manifesto claiming that a particular subclass of digital forms of information processing, based on propositional/predicate logic with some additional operators to represent time and actions, would be sufficient in three ways for all intelligent functions, which I'll loosely summarise as:

The third claim (heuristic adequacy) was challenged in Sloman (1971) and Chapter 7ff of Sloman (1978), but without challenging the assumption that whatever forms of representation and reasoning were needed could be implemented (e.g. as virtual machinery) in digital computers.

Recently, however, investigation of mechanisms required for detecting geometrical and topological necessities and impossibilities of the kinds that were used in ancient mathematical discoveries made by Archimedes, Euclid, Pythagoras, Zeno and many others, including many of the examples in this document, has led to a (tentative) conjecture that biological evolution produced a sequence of types of information processing mechanism based not on a linear "tape" with discrete locations, discrete contents of those locations, and discrete operations on the contents, nor based solely on some other kind of discrete information processing mechanism, but a different kind of data-structure that allows shapes of various sorts to be added or removed, and to be altered continuously, e.g. by translations, rotations, stretching, and other kinds of deformation, including mechanisms allowing such processes to be superimposed and the results observed and generalised. (This suggestion was partly inspired by the visual information-processing architecture proposed in Trehub(1991).)

It seems that different "layers" of such machinery, representing structures, relationships, and processes with additional types of manipulation and inspection machinery were added in later stages of evolution, supporting meta-cognitive representations of spatial structures and processes. Eventually the added mechanisms turned out to be adequate not only for many aspects of animal intelligence but also the reflective extension of such intelligence that led to ancient discoveries in topology and geometry, including the kinds of discovery of spatial impossibility and necessity illustrated in this document.

A conjecture about the discovery of cardinals and ordinals

Discovery and use of number concepts and their properties would be a special case of the above proposal. I suggest that before there were number (cardinal or ordinal) concepts as we know them now, with an unlimited supply of numbers, some of our ancestors discovered the practical usefulness of one-to-one correlations, e.g. checking that enough seats had been provided, or enough fish caught for all the members of one's family.

Later, some genius may have noticed the (necessary) transitivity of one-to-one correlations and thereby inferred the usefulness of a collection of arbitrary objects that could be used as intermediaries in checking the existence of one-to-one correlations, e.g. a physical token for each member of the family instead of the members themselves. Later still, the same or some other genius may have realised that the tokens do not need to be physical: e.g. they can be arbitrary sound patterns (e.g. ancient ancestors of our "one", "two", "three", etc.), and the transitivity will still always necessarily hold. (This fairy-tale was presented as an illustrative hypothesis in Sloman(2016).) (A very interesting question is how many times these discoveries have been made and put to use.)

The abilities involved also made possible, using currently unknown mechanisms, a host of further discoveries about the necessary properties of collections of equivalence classes and their token-names. This sort of development is totally different from the often proposed innate perceptual ability to classify collections of objects according to their numeracy, shared with some non-human species. Among other deficiencies, that mechanism would not provide the reflective ability to discover that it is impossible for a largest number to exist, a crucial feature of the concept of cardinality. E.g. see Chapter 8 of Sloman(1978) and Sloman(2016),

Some preliminary requirements for such machinery, based on analysis of some cognitive processes involving imagined distortion of spatial structures and relationships, can be found in these documents, though the ideas will be extended later:
http://www.cs.bham.ac.uk/research/projects/cogaff/misc/super-turing-geom.html
There is still a great deal of work to be done, including generating proposals for new empirical research on brain mechanisms that might be capable of supporting the proposed mechanisms. Current neural net mechanisms do not have the ability to represent, discover or reason about necessities and impossibilities.

The need for layers of meta-cognition

For some organisms this feature may be repeated at meta-cognitive levels, allowing introspective mechanisms to access complex ongoing processes to produce "summary" descriptions of what's going on, and also to re-direct those processes -- a mixture of bottom-up and top-down processing that has increasingly pervaded computer systems engineering in the last 60-70 years.

Such forms of meta-cognition may not be required for organisms to make use of mathematical structures in their perceptual processing and problem solving, but they are essential if information-processing systems are to have useful knowledge of what they are doing, so that they can control it according to specific goals, learn from it (e.g. debug features that lead to failures) and produce new kinds of functionality: as computer scientists and engineers have been doing for decades -- but for systems much less complex and sophisticated than brains.

These mechanisms will require specific extensions to the collection of construction-kits produced and used by biological evolution, discussed in
http://www.cs.bham.ac.uk/research/projects/cogaff/misc/construction-kits.html
[NOTE]
I don't know enough about quantum mechanics or neuroscience to take this possibility further. If anyone who has the required expertise would like to discuss this please get in touch with me.

The key implication for us is that a system whose capabilities include detection and representation of possibilities for change, even when those changes are not occurring, potentially has massive requirements beyond what is needed for scene perception. One way to deal with this is to find ways of representing or categorising, and reasoning about, classes of possibilities that are relevant to an organism, rather than handling all possibilities separately. That can be achieved by using appropriate levels of abstraction, ignoring details that vary within a class of cases. I suspect evolution discovered and used far more examples of that strategy than have been identified: in that sense evolution is more a blind mathematician than a blind watchmaker.

NOTE on Change-blindness

People who have had no experience of designing information-processing systems are surprised by the phenomenon of change blindness, and ask "Why don't we see the changes?", instead of asking "Why do we see changes?" and "What mechanisms make it possible to see changes?" First you need to explain how it is possible to see anything at all. Seeing changes adds significant extra complexity, that may not be obvious to people who have never tried to design a working vision system.

A video recorder recording a complex scene with constant changes does not see any changes, because it does not see. It does not know the difference between recording a constant scene and a changing scene. It has no idea what information is contained in the recording. All it does is acquire information that we (or our programs) can access about a long sequence of very short episodes where information about very short-lived states of a collection of photo-receptors is copied into some form of enduring record.

For specific changes to be detected more is required than for changing information to be recorded. What more?

Is it trivial to give a computer based vision system an ability to detect change in a scene? What if it is not fixating a particular location but continually centering on different parts of the scene, while only one small part of the scene changes? What would be required for that change to be detected? The machine would need a kind of enduring memory for perceived states of the parts of the environment being monitored and continual comparisons between newly acquired information about those states and the recorded state information. However, correspondences between locations in the environment and locations in the records would keep changing between frames (snapshots).

Many biological control mechanisms (homeostatic mechanisms in particular) perform functions based on change detection: some information about a useful or desired state (e.g. body temperature, or fluid pressure) may be stored somewhere as a target state. Then potentially constantly changing actual sensed states are compared with that target state: if there is a discrepancy then some compensatory mechanism to reduce the difference may be activated, or its level of activation modified.

More sophisticated versions compare current and recent recorded states to see whether the gap between target state and actual state is increasing or decreasing -- using mechanisms that regularly replicate and retain current sensed state, long enough to compare it with the target state.

If the gap/discrepancy exists and is increasing, the strength of the gap-reducing influence may be increased. But if the gap is already reducing, then it may be useful to reduce the gap-reducing influence -- to avoided over-shooting the target state. These considerations will be very familiar to any control engineer accustomed to designing control systems were all relevant states can be represented by measures (including measures of change and measures of rate of change, etc.).

Visual information has a far more complex structure than a particular measure such as pressure or temperature at a particular location, or a collection of measures. So the mechanisms for making use of the visual information need to be far more complex -- and the types of information obtained may be far richer.

There may be over a million sensors in an eye, with varying receptive fields, and different policies for connecting the sensor states to other parts of the nervous system.

Moreover saccades, head movements, body movements and movements of objects in the environment can all produce substantial and in some cases very fast changes in the patterns of neural activation. With all those high speed constantly changing patterns of activation, with constantly changing relationships to the light sources and light-reflecting surfaces, and in some cases constantly changing goals (e.g. while running across uneven, rocky terrain, where feet have to be carefully placed on new surfaces at high speed) no simple and obvious ideas about how the information is processed are likely either to explain the phenomena or to provide a basis for designing robots with visual capabilities of humans or other highly mobile animals.

For all these reasons, the detailed ways in which visual information about the environment is actually used must be far more complex than the cases considered by Gibson, where relatively simple mathematical transformations can extract information from changing retinal stimulation (e.g. the "rate of expansion" in an optical flow pattern, where flow radiates outward from a target location).

In a more complex visual scene different things can be changing in different directions at the same time. Mechanisms for detecting and using information about all those changes will need to be very complicated, except in cases where multiple changes are understood as different aspects of one fairly simple change: e.g. where textured items on a floor move in a complex but regular way because of horizontal motion of the viewer (as noted by Gibson and others), or the less obvious case where many cogs on a wheel are perceived to be moving in different directions, because the wheel is rotating about its axis. Gibson drew attention to the ways in which such patterns of stimulation can provide useful information about speed and direction of relative motion to an animal with eyes.

But those scenes based on large regular structures and processes are relatively rare outside human-designed environments, and are mostly products of relatively recent human construction capabilities. In contrast, the changes in patterns of retinal stimulation of an animal moving in its natural environment may be far more complex, and carry information that is far more complex than any of Gibson's examples, and require far more complex mechanisms than he seems to have considered.

The garden videos mentioned below recorded using a camera moved around near flower beds, shrubs, bushes, and trees. provide examples showing how when a camera or eye moves around in a densely foliated space, instead of a mathematically regular structure producing a time-varying retinal image that changes in a mathematically regular way, there are very many different spatial structures moving to some extent independently (e.g. when stirred by a breeze). Yet you can probably keep up with a high level overview of what's happening to the changing viewpoint and a lot of information about what is in the field of view, and where things are moving in and out of the field of view, in real time.

I don't know at what age a child is able to do this sort of thing, and I don't know how many other species can do it, but it seems likely that many birds, tree-climbers, hunting mammals and others will have at least partly similar visual capabilities.

The videos show large numbers of partly coordinated, partly uncoordinated, changes. How a seeing machine can be made to see all the changes evident to a normal (adult?) human in those videos is a hard, unsolved research problem. However, the hardest problem may not be specifying the design, but specifying the requirements to be satisfied by a good design. By what criteria could be evaluate a machine vision system's performance in interpreting such videos as evidence for a good model (or simulation) of human vision in that sort of context? It is not at all clear what the requirements are. Part of the reason is that so much more happens in human (and presumably animal) visual processing than could be characterised by results of behavioural tests.

These capabilities cannot all be products of a single, general-purpose, learning mechanism acting on the organism's sensory and motor signals during the early years of life, if they are based in part on genetic information built up many years ago by ancestors in different lineages. In particular, a totally general learning mechanism would be much too slow for precocial species where the offspring need to move around pecking for food unaided (ducklings, chicks) or need to be able to run with the herd soon after birth (e.g. wildebeest foals only a few hours old, without having had time to learn).

Social influences in learning

Are some forms of mathematical discovery impossible without a social environment?

Don't assume a teacher with prior knowledge of the theorems has to be involved: someone must have made some of these discoveries without being told them by a teacher.

NOTE 1 This kind of discovery of primeness by a computer program was discussed in Pease et al. (2010). But their program seems to have been given a systematic way of carving up spatial possibilities by the designers. It did not understand enough to work that out.

NOTE 2 One of the fundamental requirements for mathematical thinking is being able to organise collections of possibilities and making sure that you have checked them all. If you can't do that you don't have a mathematical result, only a guess. How can you know that you have checked all possibilities? The history of mathematics shows that even brilliant mathematicians can make mistakes Lakatos (1976). This means that the traditional emphasis on the role of "certainty" in mathematics may be misguided: certainty, or its absence, like infallibility or its absence, is a matter of the psychology of mathematicians, not the subject matter they investigate, which is something richer and deeper: a feature of the universe that was playing a role in evolution (the "Blind Mathematician") long before human mathematicians existed. Computers, like drawings in sand, slates and chalk, pen and paper, 3-D models made of wires and beads, and other aids to thinking and communication, have expanded what human mathematicians can do, but not changed the nature of the subject matter. Some are tempted to conclude that mathematics is essentially a social phenomenon. That's true only for relatively weak-minded mathematicians (e.g. human mathematicians). See also Wolfram (2007).

Further discussion: Functions of visual perception

The ideas presented above imply criticisms of all models and theories of perception that restrict the functions of perception to detecting and recording how things are (or probably are) in the environment, classifying them and predicting what will happen or what will be the case (with or without probability estimates), and retrodicting or diagnosing causes of or predecessors of what is the case.

Although some researchers may regard that list is comprehensive, it leaves out some important functions of vision that are rarely noticed by vision researchers (although there are researchers who do investigate some of them). For example vision is often used for obtaining information about what is possible or impossible in the environment and obtaining information about how the environment relates to abilities, risks, needs, or intentions of other agents. Information about what is or is not possible is relevant both to the immediate practical uses of vision, but also to mathematical discoveries, as we'll see. This generalises Gibson's claim that the function of vision is to provide information about affordances for the perceiver in the environment Gibson (1979).

In particular, my complaint about omission of important functions of vision (and more generally perception) applies to statistics-based models and theories of vision that have been very successfully applied to special purpose robots and other machines with limited(!) functionality.

A core symptom of the inadequacy of those models and theories is that their proponents (in my experience) neither acknowledge, nor attempt to explain, the roles of vision in mathematical discovery, such as discovery of theorems and proofs in Euclidean geometry (including topology). Those discoveries are concerned with what is possible (e.g. it is possible, using compasses, to produce circles of any radius around a specified point, in a specified plane), what is impossible (e.g. it is impossible for three finite planar surfaces to completely enclose a 3-D space, though three lines can completely enclose a 2-D area in a plane), and what is necessarily the case, e.g. if A is longer than B and B is longer than C then A must be longer than C.

Another triangular impossibility
(Added 29 Nov 2019)
A less familiar example is this: Can a planar triangle have one side longer than the combined lengths of the other two sides? I've found that non-mathematicians can think about this and fairly quickly give an answer. They can recognise impossibilities and necessary consequences of spatial configurations even if they cannot easily articulate their reasoning. They are neither merely summarising statistical evidence collected in the past, nor deriving the impossibility logically, by reasoning about consequences of Euclid's axioms. In other documents I have begun to explore the possibility of entirely new forms of non-discrete computation that might be implemented in sub-neural chemistry (e.g. Super-Turing machines mentioned above: but the problems are very difficult, and there's still a long way to go.

I'll offer some conjectures about the evolutionary history of visual functions and mechanisms in humans and other species, including abilities to see what is and is not possible in a situation: a very different type of function from merely recording sensory states or environmental states, responding to those states or predicting future states. This is related to a contrast between online and offline intelligence. Offline intelligence is central to science and mathematics, and also to some forms of perception of affordances.

Mathematical Consciousness

Another theme, not developed much in this document, is that understanding varieties of human consciousness and the underlying mechanisms, requires understanding kinds of mathematical consciousness that lead to mathematical discoveries: e.g. as a result of noticing that something seems to be possible, impossible or necessarily the case, and then asking "How do I know?" This relates some kinds of mathematical discovery to forms of introspection, which in turn are related to forms of information-processing architecture produced by biological evolution that support the required varieties of introspection: architectures that include meta-cognitive mechanisms.

I suspect some other species, and pre-verbal children, can make use of mathematical discoveries, but lack the information processing (meta-cognitive) architectures required to notice what they are doing and understand why it works, or why it never works!

The questions and tentative answers presented here are inspired in part by Immanuel Kant's philosophy of mathematics in Kant (1781), in part by an extension of James Gibson's ideas about the functions of vision (Gibson (1966) and (1979)), and in part by work by Max Clowes in the 1960s and 1970s concerning vision's connection with multiple "domains", and the possibilities of ambiguity, paraphrase, and anomaly (impossibility) in visual contents (see Clowes Tribute). I make particular use of the first picture of an "impossible triangle", or, more precisely, an impossible configuration of cubes produced by Oscar Reutersvard, in 1934, introduced piecemeal above. It is richer, in interesting ways, than many better known pictures of impossible objects.

The functions of vision that relate to mathematical discoveries also require meta-cognitive mechanisms -- using more than one layer of meta-cognition -- closely integrated with vision. (In blind mathematicians the visual sensory apparatus may not work, but that does not prevent their brains using and extending evolved mechanisms of spatial reasoning that underly much mathematical cognition.) This mechanisms must not be tied to vision, since they are relevant to haptic and tactile sensing, auditory sensing, and also to control of various kinds of motion in space.

In particular, mathematical discovery often requires a form of introspection, first generating discoveries similar to "This seems to me to be impossible", then "Why am I sure this is impossible?", followed later by attempts to influence the introspections of other mathematicians (or learners). But that claim is still highly programmatic: there are still important gaps in the work, including gaps regarding architectures for cognition and gaps regarding the forms of representation required. For example, logical and algebraic forms do not seem to be adequate. (I am not claiming that these mathematical discovery processes cannot be implemented in computers or robots, merely pointing to unsolved problems, continuing work begun in Sloman (1971). Contrast the impossibility claims in Penrose (1994).

This paper is also related to work on virtual machinery and consciousness, including the causal powers and varieties of "privacy" (of qualia) in virtual machines discussed in Sloman&Chrisley (2003), and Sloman (2013).

Above all, this paper continues a decades long attempt (begun in my DPhil thesis(1962)) to understand the nature of mathematics and the biological mechanisms that made it possible for Euclid and his predecessors to achieve so much, long before the development of modern logic, algebra, formal methods, and proof theory.

The mechanisms of mathematical discovery by humans have deep roots in biological mechanisms for information processing, but the mathematical facts discovered are not products of biology or human culture, contrary to the claim often made that mathematics is a human creation. The mathematical facts discovered by human mathematicians should mostly be discoverable by any sufficiently intelligent animal or machine. Some of them were "implicitly discovered" by natural selection mechanisms as explained here. There are some exceptions, including mathematical facts about how human languages work, e.g. the kinds of structures and structure-mappings they use.

Some of the mechanisms required may be incapable of being implemented in digital computers as we know them. Some half-baked speculations exist about possibly relevant quantum mechanisms (e.g. for constraint propagation). There may, or may not, be a requirement for specific forms of quantum computation in some particularly demanding forms of perception, e.g. of contents of an open-air horticultural centre, while the wind is blowing and the perceiver is moving (illustrated by videos above).

This has nothing to do with the arguments of Penrose and Hameroff, as far as I can tell. That topic is still left open here.

Experiences of mathematical discovery are among the richest, deepest, and most useful experiences of which humans are capable. Many of them have deep connections with biological functions of animal vision. That's because evolution produced visual mechanisms whose main functions include discovery and use of affordances of many kinds, and affordances, like the contents of mathematical discoveries are not merely concerned what is the case, but also with what is and is not possible, or necessarily the case. (This has nothing to do with probabilities or statistical information.)

The overlap between common animal functions of visual perception and the peculiarly human ability to make mathematical discoveries using vision illustrates the slogan of Max Clowes: "Perception is controlled hallucination", vaguely echoing von Helmholtz: "Perception is unconscious inference", and Kant's suggestion that all our empirical knowledge is made up of both "what we receive through impressions" and of what "our own faculty of knowledge supplies from itself". Kant, however, did not, as far as I know, link the powers of our faculties to their evolutionary history or to related capabilities in other intelligent species, as I am attempting to do. Neither did he have the opportunity I have had to use computers to work on problems in Artificial Intelligence. I suspect he would have appreciated the deep philosophical importance of AI instantly.

NOTE: AI vision mechanisms based on so-called deep learning are incapable of producing or explaining the mathematical competences discussed here, insofar as they merely deduce probabilistic relationships from statistical evidence. That cannot support conclusions about impossibilities and necessary connections -- the stuff of mathematics.

I have also argued elsewhere that the ancient mathematical discoveries by Aristotle, Euclid, Zeno and others could not have been based on learning capabilities provided by neural nets (e.g. Deep Learning). In particular those statistics-based AI systems are incapable of representing, let alone discovering or proving the fact that something is impossible, or necessarily true. Logic based AI reasoners can make such discoveries, but (as Kant seems to have noticed) use of pure logic could not suffice to make the mathematical discoveries about spatial structures and processes discussed here and in other documents on this web site Sloman(1962), which I conjecture are products of very sophisticated evolutionary and developmental mechanisms, based on "Meta-Configured Genomes", summarised here:
http://www.cs.bham.ac.uk/research/projects/cogaff/movies/meta-config

Note on Wittgenstein's Tractatus
(Added: 4 Oct 2015)

It is not clear what 3.032 is claiming about language, since far from being impossible it is easy to express in words things that are logically inconsistent, e.g. if I were to claim that that is both possible and impossible. Wittgenstein must have known this.

3.0321 is more interesting for us: there are interesting and important counter-examples in the form of 2-D pictures of geometrically impossible 3-D objects, which we'll come to later. Some of them had not been discovered at the time Wittgenstein was writing.

In a philosophy graduate seminar around 1960 I offered a line drawn on a blackboard as a counter-example to 3.0321, like this:

I don't remember what sort of reaction it produced, but I don't recall anyone being impressed. More interesting examples, some discussed above, had previously been produced by Oscar Reutersvard, Roger and Lionel Penrose, and Maurits Escher, though I did not learn about them till later. The Wikipedia entry for the Tractatus also draws attention to the conflict between the possibility of the Penrose triangle and Wittgenstein's claim in Tractatus 3.0321, quoted near the top of this paper.

A more complicated, more interesting, example from Reutersvard was presented above..

Evo-Devo Issues (Evolution and Development)

A separate paper on construction-kits points out that biological evolution constructs layer upon layer of new construction kits of ever increasing complexity and power (in branching sequences). Some of the construction kits are physical, others abstract and some are hybrids. Some of the evolved construction kits of some species need to be built during individual development, so that they are tailored to features of the environment and needs of members of the species, and this may be true of construction kits used for sophisticated perceptual systems. This is a special case of a general phenomenon also illustrated fairly clearly by construction-kits for language development. Those kits are to some extent specified in the human genome, but deployment in different cultures leads to very varied products and differences in later language learning capabilities.

This pattern of variation in expression of a common genome was described in terms of a distinction between "pre-configured" and "meta-configured" competences in Chappell and Sloman 2007, which presented an epigenetic theory summarised in Figure Evo-Devo, below, also summarised in a video presentation here http://www.cs.bham.ac.uk/research/projects/cogaff/movies/meta-config (link added 18 Oct 2019):

In at least some animals ("precocial" species) the specification for detailed spatial competences is clearly in the genome. But not in all animals. It seems that more sophisticated, more abstract, competences, capable of being instantiated in more varied ways, are provided in a more generic form in genomes for more sophisticated species (often described as "altricial" species), such as humans, squirrels, and crows, whose offspring do not show (or need) such competences until long after birth. The competences develop in a context sensitive manner on the basis of complex interactions between the genome and the environment, as depicted in Figure Evo-Devo, above.

In humans, this sort of developmental diversity is most evident in language development, which produces very different linguistic competences in different cultures -- as shown both by how children in different cultures express themselves (what they can say) and also differences in what they understand. The differences between sign-languages and spoken languages are particularly striking. (Elsewhere I have argued that sign-languages must have evolved first, and that structured internal languages evolved even earlier, and in more species Sloman[Vis-Lang].)

The kind of diversity spawned in this way is also evident in other intelligent competences demonstrated in games, music making, art forms, playing with construction kits, and environmentally related skills, e.g. dealing with clothing or food, swimming, climbing trees, climbing rocks, building structures out of sand, etc.

Those complex combined products of evolution and environment that take time to develop and can differ between members of the same species, both across cultures and within cultures are well known. I am suggesting that abilities to perceive structures and structural changes may be equally complex, slow to develop and liable to differ between species and also between members of the same species.

I hope that all this will help to make readers understand that asking why we do not see certain changes is the wrong question if we don't yet know how changes are seen, and what conditions need to be satisfied, both in the environment, and in cognitive/perceptual mechanisms, for changes of various sorts to be detected. If mechanisms of change detection, instead of being simple and universal are subject to the kind of developmental diversity indicated in Figure Evo-Devo then before asking why environmental changes are not seen in some situations, we need to develop good theories as to how they are seen. Detecting Some changes may require far more complex perceptual systems than those that are used in simple forms of "online" intelligence.

The point is that much that appears to be "learnt anew" may in fact be new instantiations of previously evolved highly generic competences, some of which capture mathematical features of the environments in which ancestors evolved.

The results of such evolution can produce "meta-configured" generic genomes that are instantiated in different ways in different developmental contexts by using (sometimes very complex) parameters acquired from the environment in ways that can vary both across species and between members of the same species (e.g. young humans developing in cultures at very different levels of technical competence).

However, this kind of "parameter substitution" will be far more complex than the simple forms of parameter substitutions found in mathematics or common programming languages. Compare "parametric polymorphism" in programming languages, where the effect of a parameter may depend not only on its type but also the types of additional parameters supplied in the context. Forks have a generic type of use, but garden forks, pitchforks and table forks have different uses requiring different skills and knowledge, even though they share a common abstract schema for structure and function.

Chappell and Sloman (2007) offer a schematic model, crudely depicted in Figure Evo-Devo, of how genetic specifications at different levels of abstraction, activated at different stages of development, can combine with information acquired directly or indirectly (via earlier instantiations) from the environment at different stages of development. (This generalises Waddington's concept of an "epigenetic landscape". Our schematic theory is closer to the concept of "Representational Redescription" presented in Karmiloff-Smith (1992), but there are important differences as described in Sloman (in press))

Theories of learning based on completely general statistical mechanisms interacting with sensory data will not accurately describe learning based partly on abstract genetic information about the environment acquired previously by natural selection, in altricial species, such as humans and crows, where the genetic mechanisms make sophisticated use of parametric polymorphism, for example.

Being able to detect actual changes in the environment is one thing (though it may have surprising complexity, as discussed above) but being able to consider, and reason about, possible changes that are not occurring, and are not actions of the perceiver, is very different. Compare the discussion of perception of possible, and impossible, changes above. That is a kind of affordance perception that as far as I know Gibson did not notice, a very serious omission in theories of animal intelligence and functions of human vision. Most other researchers also seem not to notice. It is also not usually discussed in publications on the nature of mathematical discovery.

Note on sliced polyhedron example
The problem: what happens to numbers of vertices, edges and faces if you slice a single vertex off a solid convex polyhedron was posed above. There is a standard (obvious) interpretation of the problem and also a non-standard interpretation. On the standard interpretation the cut that removes a vertex goes through each of the edges connected to that vertex, cutting each edge into two parts, one part removed by the slice and one part left as part of the remaining polyhedron.

It is possible also to consider "non-standard" interpretations, where the plane slicing off the chosen vertex goes through one or two of the remaining vertices. Working out what happens in those cases is left as an exercise for the reader. (A non-trivial exercise!)

REFERENCES

Dana Ballard and Chris Brown, Computer Vision 1982,
Prentice Hall, Inc., Englewood Cliffs, New Jersey 07632, USA,
Freely available online:
http://homepages.inf.ed.ac.uk/rbf/BOOKS/BANDB/bandb.htm
A.A.S. Weir, J. Chappell and A, Kacelnik, Shaping of hooks in New Caledonian crows, Science, 297, 9 August 2002 p. 981,
https://www.youtube.com/watch?v=UDg0AKfM8EY
J. Chappell and A. Kacelnik, New Caledonian crows manufacture tools with a suitable diameter for a novel task, Animal Cognition, 7, 121--127, 2004
Jackie Chappell and Aaron Sloman, 2007a, Natural and artificial meta-configured altricial information-processing systems, International Journal of Unconventional Computing 3, 3, pp. 211--239,
http://www.cs.bham.ac.uk/research/projects/cogaff/07.html#717
Jackie Chappell and Aaron Sloman, 2007b,
Two ways of understanding causation: Humean and Kantian,
Two Presentations at WONAC: International Workshop on Natural and Artificial Cognition, Pembroke College, Oxford, June 25-26, 2007
http://www.cs.bham.ac.uk/research/projects/cogaff/talks/wonac
Andy Clark (2013)
Whatever next? Predictive brains, situated agents, and the future of cognitive science, in Behavioral and Brain Sciences, 36, 3, pp. 1--24,
M.B. Clowes, 1971, On seeing things, in Artificial Intelligence 2, 1, pp. 79--116, http://dx.doi.org/10.1016/0004-3702(71)90005-1
Clowes, M. (1972). Man the creative machine: A perspective from Artificial Intelligence research. In The Limits of Human Nature, Ed Jonathan Benthall Allen Lane.
Max Clowes. Life and work of Max Clowes: an annotated bibliography.
http://www.cs.bham.ac.uk/research/projects/cogaff/sloman-clowestribute.html#bio
(Maintained by Aaron Sloman. Contributions welcome. Needs a Wikipedia page?)
Terrence Deacon (2011). Incomplete Nature: How Mind Emerged from Matter, W.W. Norton & Company.
Euclid, Elements
http://aleph0.clarku.edu/~djoyce/java/elements/toc.html
http://www.gutenberg.org/ebooks/21076
Gottlob Frege, 1950, The Foundations of Arithmetic: a logico-mathematical enquiry into the concept of number, Oxford, B.H. Blackwell, (Tr. J.L. Austin. Original 1884),
C Freksa, Barkowsky, Falomir and van de Ven, (2019), Geometric problem solving with strings and pins, Spatial Cognition & Computation,
https://doi.org/10.1080/13875868.2018.1531415
J. J. Gibson, 1966, The Senses Considered as Perceptual Systems, Houghton Mifflin, Boston, MA.
J. J. Gibson, 1979, The Ecological Approach to Visual Perception, Houghton Mifflin, Boston, MA,
Immanuel Kant Critique of Pure Reason (1781)
This has relevant ideas and questions, but he lacked our present understanding of information processing (which is still too limited)
http://archive.org/details/immanuelkantscri032379mbp
Rex Hartson (2003)
Cognitive, physical, sensory, and functional affordances in interaction design, in Behaviour and Information Technology, 22, 5, pp. 315--338,
http://www.tandfonline.com/doi/pdf/10.1080/01449290310001592587
S. Kaneff, Ed. Picture language machines, 1970, Academic Press, New York,
Annette Karmiloff-Smith (1992)
Beyond Modularity,
A Developmental Perspective on Cognitive Science,
MIT Press --Informally reviewed in
http://www.cs.bham.ac.uk/research/projects/cogaff/misc/beyond-modularity.html
Annette Karmiloff-Smith, (1994). Precis of Beyond modularity: A developmental perspective on cognitive science, in Behavioral and Brain Sciences
17 (4): 693-745.
Imre Lakatos, Proofs and Refutations,
Cambridge University Press, 1976,
John McCarthy and Patrick J. Hayes, 1969, "Some philosophical problems from the standpoint of AI", Machine Intelligence 4, Eds. B. Meltzer and D. Michie, pp. 463--502, Edinburgh University Press,
http://www-formal.stanford.edu/jmc/mcchay69/mcchay69.html
John McCarthy, 2008, The Well-Designed Child, in Artificial Intelligence 172, 18, pp.2003--2014,
http://www-formal.stanford.edu/jmc/child.html
William M. Mace, 2005, James J. Gibson's ecological approach: perceiving what exists, in Ethics and the Environment, vol 10, no 2, Indiana University Press,
http://www.trincoll.edu/~wmace/publications/enviro_chapt.pdf
Macpherson, F. (2010) "Impossible Figures", in The Sage Encyclopedia of Perception, Ed. E. Bruce Goldstein, Thousand Oaks, California: Sage Publications, Inc. http://www.gla.ac.uk/media/media_127562_en.pdf
David Marr, 1982, Vision, W.H.Freeman, San Francisco,
Roger B. Nelsen, (1993) Proofs without words: Exercises in Visual Thinking, Mathematical Association of America, Washington DC. (His name is often mis-spelled "Nelson")
Alison Pease, Simon Colton, Ramin Ramezani, Alan Smaill and Markus Guhe, (2010), Using Analogical Representations for Mathematical Concept Formation, in Model-Based Reasoning in Science & Technology, Eds. L. Magnani et al, Springer-Verlag, pp. 301-314, 2010,
http://homepages.inf.ed.ac.uk/apease/papers/pease_mbr09.pdf
Roger Penrose, 1994, Shadows of the mind: A Search for the Missing Science of Consciousness, OUP, Oxford,
Roger Penrose, (1992), On the Cohomology of Impossible Figures, Leonardo, Visual Mathematics: Special Double Issue 25.3/4, pp. 245-247. The MIT Press, http://www.jstor.org/stable/1575844,
Jean Piaget, 1952, The Child's Conception of Number, Routledge & Kegan Paul, London,
Jean Piaget (1981,1982)
Much of Jean Piaget's work is relevant, especially his last two (closely related) books written with his collaborators:
Possibility and Necessity (1981/1983)

Vol 1. The role of possibility in cognitive development (1981)
Vol 2. The role of necessity in cognitive development (1983)
Tr. by Helga Feider from French in 1987

Also much of his earlier work, e.g. Piaget (1952) above.
Like Kant, Piaget had deep observations and intuitions about what needed to be explained, but lacked an understanding of information processing mechanisms, required for explanatory theories, especially explanations of possibilities. (Sloman, 2014)
L. G. Roberts, 1965, Machine perception of three dimensional solids, in
Optical and Electro-optical Information Processing Eds. J.P. Tippett et al., MIT Press, Cambridge, MA,
Jean Sauvy and Simonne Sauvy, with an introduction by Bill Brookes
The Child's Discovery of Space: From hopscotch to mazes -- an introduction to intuitive topology,
Penguin Education, Harmondsworth, 1974. Translated from the French by Pam Wells,
Schrödinger E (1944). What is life? CUP, Cambridge
Extracts from this book, with some added comments, and a note added in the 1955 edition, can be found here:
http://www.cs.bham.ac.uk/research/projects/cogaff/misc/schrodinger-life.html
Shultz, T. R. and Vogel, A., (2004), "A connectionist model of the development of transitivity", Proc of 26th Annual Conference of the Cognitive Science Society, pp. 1243--1248, Mahwah NJ, Erlbaum, https://beta.escholarship.org/uc/item/2n1411n7
Susanna Siegel, (2014),
Affordances and the Contents of Perception, in Does Perception Have Content? Ed. B. Brogaard, OUP, pp. 39--76,
http://philpapers.org/rec/SIEAAT
A. Sloman, (1962 -- transcribed/searchable version 2016), Knowing and Understanding: Relations between meaning and truth, meaning and necessary truth, meaning and synthetic necessary truth, DPhil Thesis, Oxford University,
http://www.cs.bham.ac.uk/research/projects/cogaff/62-80.html#1962-01
A. Sloman, 1965, "Necessary", "A Priori" and "Analytic", Analysis, Vol 26, No 1, pp. 12--16.
http://www.cs.bham.ac.uk/research/projects/cogaff/62-80.html#1965-02
A. Sloman (1971), Interactions between philosophy and AI: The role of intuition and non-logical reasoning in intelligence, in Proceedings 2nd IJCAI, 1971, pp. 209--226, London, William Kaufmann, Reprinted in Artificial Intelligence, vol 2, 3-4, pp 209-225, 1971. http://www.cs.bham.ac.uk/research/cogaff/62-80.html#1971-02
This was the basis of Chapter 7 of Sloman (1978).
A. Sloman, 1971b, Tarski, Frege and the Liar Paradox, Philosophy, 46, 176, pp. 133--147,
http://www.cs.bham.ac.uk/research/projects/cogaff/62-80.html#1971-03
A. Sloman (1978), The Computer Revolution in Philosophy: Philosophy, Science and Models of Mind, Harvester Press (and Humanities Press), Hassocks, Sussex, (Updated, digitised version, with additional links.)
http://www.cs.bham.ac.uk/research/cogaff/62-80.html#crp
A. Sloman, (1983), Image interpretation: The way ahead?, in Physical and Biological Processing of Images (Proceedings of an international symposium organised by The Rank Prize Funds, London, 1982.), Ed. O.J. Braddick and A.C. Sleigh., pp. 380--401, Springer-Verlag, Berlin,
http://www.cs.bham.ac.uk/research/projects/cogaff/81-95.html#57
A. Sloman, (1996) Actual Possibilities, in Principles of Knowledge Representation and Reasoning: Proc. 5th Int. Conf. (KR `96), Eds. L.C. Aiello and S.C. Shapiro, Morgan Kaufmann Publishers, Boston, MA, 1996, pp. 627--638,
http://www.cs.bham.ac.uk/research/cogaff/96-99.html#15
A. Sloman and R.L. Chrisley, (2003), Virtual machines and consciousness, in Journal of Consciousness Studies 10, 4-5, pp. 113--172,
http://www.cs.bham.ac.uk/research/projects/cogaff/03.html#200302
A. Sloman (2007-2014)
Unpublished discussion Paper: Predicting Affordance Changes: Steps towards knowledge-based visual servoing. (Including videos).
http://www.cs.bham.ac.uk/research/projects/cogaff/misc/changing-affordances.html
A. Sloman, (2008)
Architectural and Representational Requirements for Seeing Processes, Proto-affordances and Affordances, in Logic and Probability for Scene Interpretation, Eds. A.G. Cohn, D.C. Hogg, Ralf Moeller & Bernd Neumann, Dagstuhl Seminar Proceedings, No 08091, Schloss Dagstuhl - Leibniz-Zentrum fuer Informatik, Germany, 2008,
http://www.cs.bham.ac.uk/research/projects/cogaff/08.html#806
(And several other closely related papers.)
A. Sloman, (2006,--) Requirements for a Fully Deliberative Architecture (Or component of an architecture), Originally: CoSy Project Research Note, COSY-DP-0604, University of Birmingham, UK,
http://www.cs.bham.ac.uk/research/projects/cogaff/misc/fully-deliberative.html
Aaron Sloman, Jackie Chappell and the CoSy PlayMate team, 2006 (and after)
Orthogonal Recombinable Competences Acquired by Altricial Species (Blankets, string, and plywood),
http://www.cs.bham.ac.uk/research/projects/cogaff/misc/orthogonal-competences/
School of Computer Science, University of Birmingham, Research Note, COSY-DP-0601, Birmingham, UK,
Jackie Chappell and Aaron Sloman, 2007,
Natural and artificial meta-configured altricial information-processing systems, International Journal of Unconventional Computing, 3, 3, 2007, pp. 211--239. (An early version of the meta-configured genome theory.)
http://www.cs.bham.ac.uk/research/projects/cogaff/07.html#717
A. Sloman, (2009a). Some Requirements for Human-like Robots: Why the recent over-emphasis on embodiment has held up progress. In B. Sendhoff, E. Koerner, O. Sporns, H. Ritter, and K. Doya, editors, Creating Brain-like Intelligence, pp 248-277. Springer-Verlag, Berlin
http://www.cs.bham.ac.uk/research/projects/cogaff/09.html#912
A. Sloman, (2009b) Architecture-Based Motivation vs Reward-Based Motivation, in Newsletter on Philosophy and Computers, American Philosophical Association, 09, 1, pp. 10--13, Newark, USA.
http://www.cs.bham.ac.uk/research/projects/cogaff/misc/architecture-based-motivation.html
A. Sloman (2012), Hidden Depths of Triangle Qualia: The Median Stretch, Side Stretch, and Triangle Area Theorems (Discussion note).
http://www.cs.bham.ac.uk/research/projects/cogaff/misc/triangle-theorem.html
A. Sloman, 2013, Virtual Machine Functionalism (The only form of functionalism worth taking seriously in Philosophy of Mind and theories of Consciousness), Research Note,
School of Computer Science, The University of Birmingham,
http://www.cs.bham.ac.uk/research/projects/cogaff/misc/vm-functionalism.html
A. Sloman (2014)
Using construction kits to explain possibilities
http://www.cs.bham.ac.uk/research/projects/cogaff/misc/explaining-possibility.html
A. Sloman, 2015 (and earlier) What are the functions of vision? How did human language evolve? (Online tutorial presentation, University of Birmingham.)
http://www.cs.bham.ac.uk/research/projects/cogaff/talks/#talk111
A. Sloman (2016), Natural Vision and Mathematics: Seeing Impossibilities, Second Workshop on: Bridging the Gap between Human and Automated Reasoning, IJCAI 2016, pp.86--101, Eds. Ulrich Furbach and Claudia Schon, New York,
http://www.cs.bham.ac.uk/research/projects/cogaff/Sloman-bridging-gap-2016.pdf
http://ceur-ws.org/Vol-1651/
Aaron Sloman and Jackie Chappell, (2017ff) The meta-configured genome (Multi-layered, multi-stage, parametrised, epigenesis): online, on-going, discussion paper begun 2017, extending Chappell/Sloman 2007.
http://www.cs.bham.ac.uk/research/projects/cogaff/misc/meta-configured-genome.html
http://www.cs.bham.ac.uk/research/projects/cogaff/misc/meta-configured-genome.pdf
Extended 2019, with a video lecture and discussion of implications and presuppositions http://www.cs.bham.ac.uk/research/projects/cogaff/movies/meta-config/ and a draft over-simplified tutorial summary of some of the chemical processes involved in gene-expression, illustrating their potential use in the enabling meta-configured genomes: http://www.cs.bham.ac.uk/research/projects/cogaff/misc/dna-uses.html (also pdf).
A. Sloman (in Press, expected 2021), Biological evolution's use of representational redescription, in a volume in memory of Annette Karmiloff-Smith Eds. Victoria Knowland, Michael Thomas, Denis Mareschal, Taylor and Francis. (Preprint available at http://www.cs.bham.ac.uk/research/projects/cogaff/misc/AKS.pdf)
D'Arcy Wentworth Thompson On Growth and Form (1917/1992)
The Complete Revised Edition (Dover Books on Biology) Originally published 1917. http://www.amazon.com/On-Growth-Form-Complete-Revised/dp/0486671356/ref=cm_cr_pr_orig_subj
Arnold Trehub, 1991, The Cognitive Brain, MIT Press, Cambridge, MA,
http://people.umass.edu/trehub/
Emre Ugur, 2010, A Developmental Framework for Learning Affordances, (PhD thesis)
The Graduate School of Natural and Applied Sciences, Middle East Technical University, Ankara, Turkey,
http://www.cns.atr.jp/~emre/papers/PhDThesis.pdf
Stephen Wolfram (2007), Mathematics, Mathematica and Certainty Wolfram Blog December 8, 2007
http://blog.wolfram.com/2007/12/08/mathematics-mathematica-and-certainty/

History of this document
This is a seriously incomplete document, yet is much too long for anyone to read. I hope someone gets new ideas as a result of browsing here. This paper started life as a set of notes for a talk to the Bristol Visual Information Laboratory, on 2nd October 2015, with draft abstract here:
http://www.cs.bham.ac.uk/research/projects/cogaff/misc/bvi-talk-2015.html

Installed: 28 Sep 2015;
Last updated: 12 Sep 2020 (Corrected many formatting errors, that led to gaps in the PDF version.)
29 Nov 2019 More on Super Turing machinery.
7 Jan 2019 (fixed error in coins example); Tidied a bit 15 Jul 2019;
3 Jan 2018 (added super-Turing idea); 26 Jul 2018; 29 Oct 2018 (more Kantian impossibilities);
8 Sep 2017; 16-19Oct 2017(New links, partial reorganisation); 29 Oct 2017
29 Jul 2016 (expanded discussion of Reutersvard picture); 2 Nov 2016; 31 Jan 2017
29 Mar 2016; 9 May 2016; 6 Jun 2016; 22 Jul 2016
18 Jan 2016 ; 31 Jan 2016; 3 Feb 2016; 16 Feb 2016; 22 Feb 2016;
29 Sep 2015;30 Sep 2015; 3 Oct 2015; 16 Oct 2015; 28 Dec 2015; 13 Jan 2016;

Some (Possibly) New Considerations Regarding Impossible Objects

Alternative titles: Impossibility: the dark face of necessity. What don't we know about spatial perception/cognition? especially the perceptual abilities of ancient mathematicians? Aspects of mathematical consciousness, Possible impossible contents of consciousness!

CONTENTS

Background [Updated: 8 Nov 2015; 28 Dec 2015; 18 Jan 2016; 3 Feb 2016; 29 Nov 2016; 16 Oct 2017; 5 Jan 2018; 29 Oct 2018; 20 Jul 2019; 16 Nov 2020;]

Some Kantian examples (Added 29 Oct 2018)

Personal history

What are the functions of perception?

Introduction: Gibson's notion of affordance

Beyond Gibson: Possibilities, impossibilities, necessities

Online and offline uses of visual information

Requirements for the online and offline functions of vision

Biological examples of offline use of vision

Representing possible processes

Possibility perception

Possibilities vs Probabilities

Temporally chained possibilities

Perceiving vs understanding

Pictures of possible and impossible object configurations

What can you do with a collection of eight similar cubes?

What if you had nine similar cubes?

More possible moves

More challenges to psychology, neuroscience and AI

3 + 5 < 6

Reutersvard 1934

Pre-20th Century pictures of impossible scenes

Mis-use of the word "illusion"

Tangled bodies Added: 15 Oct 2017

Impossible transitions on a grid

Impossible number relationships

Discovering primeness geometrically

Physical constraints vs rule-based constraints

Domestic possibilities and impossibilities (furniture)

Constructive, supportive impossibilities

Impossibilities concerned with cardinality

Figure Transitive

Non-human reasoners about possibilities and impossibilities

Toddler Topology

The Side Stretch Theorem

The Side Containment Theorem (SCT)

Properties of triangles and other abstract shapes

Strings

Strings, pulleys and levers

Topological additions October 2017

Other examples

Evolution: the Blind Mathematician

Mechanisms of process-perception

Garden scenes/Gardens seen Added 18 Oct 2015

Variety of functions of vision

Massive constraint satisfaction/propagation challenges

Is there a need for a Super-Turing "Membrane" machine?

A conjecture about the discovery of cardinals and ordinals

The need for layers of meta-cognition

NOTE on Change-blindness

Social influences in learning

Further discussion: Functions of visual perception

Mathematical Consciousness

Note on Wittgenstein's Tractatus (Added: 4 Oct 2015)

Evo-Devo Issues (Evolution and Development)

REFERENCES

Some (Possibly) New Considerations
Regarding Impossible Objects

Alternative titles:
Impossibility: the dark face of necessity.
What don't we know about spatial perception/cognition?
especially the perceptual abilities of ancient mathematicians?
Aspects of mathematical consciousness,
Possible impossible contents of consciousness!

Background
[Updated: 8 Nov 2015; 28 Dec 2015; 18 Jan 2016; 3 Feb 2016; 29 Nov 2016; 16 Oct 2017; 5 Jan 2018; 29 Oct 2018; 20 Jul 2019; 16 Nov 2020;]

Some Kantian examples
(Added 29 Oct 2018)

Tangled bodies
Added: 15 Oct 2017

Garden scenes/Gardens seen
Added 18 Oct 2015

Note on Wittgenstein's Tractatus
(Added: 4 Oct 2015)