Domains that occur naturally include configurations of water, sand, mud, stones, trees, grass, other plants, and many kinds of animals. Some of the objects naturally move autonomously while others move only when manipulated by something else or blown by the wind, etc.

Non-natural domains include things like string, string plus other materials, paper, cloth, cotton-wool, plasticene, toy blocks, dolls and their clothing, toy vehicles, cutlery, crockery, and many kinds of construction kits, e.g. Lego, Meccano, Tinker toys, Fischer tecknik, etc. (See Michael Adler's Meccano web site: http://www.meccanotec.com/ and the International society of Meccanomen: http://www.internationalmeccanomen.org.uk/ -- why only men???)

Combining domains provides new richer domains with novel capabilities that can be explored and used. E.g. combining string and paper produces possibilities like parcels that can be pushed, thrown, etc. without falling apart. Combining cutlery, crockery and various kinds of solid and liquid edible substances produces many new possible ways of consuming food and liquid.

There are also abstract domains, such as arithmetic, many kinds of games, words, phrases, sentences, poems, limericks, and many branches of mathematics. In the last few decades new domains have been developed that young children learn to interact with using computing devices even though their ancestors never encountered anything similar. Some primates can also learn to interact with them.

Our domain of polyflaps was invented partly to produce shapes and processes that might be easier for a robot to see than some of the more common shapes (e.g. avoiding curved surfaces and curved edges), while being cheap to make, and lightweight, and indefinitely extendable.

The pictures shown below are examples of 'polyflaps'. Each polyflap is made from a single sheet of paper cut in a polygonal shape (convex or concave, regular or irregular) and then folded once, but not folded flat, so as to form a 3-D structure. (Future examples may be made of cardboard, or some other material).

Any collection of polyflaps can clearly be extended indefinitely, forming more and more complex domains of perception and action, e.g.

• by folding an already used polygon shape in a new way
• by adding a new polygon shape with the same number of sides as used previously (e.g. varying relative line-lengths or angles)
• by adding a new polygon shape with more sides
• by providing larger numbers of polyflaps making more complex configurations possible, including new 3-D structures built from polyflaps
• by combining polyflaps with other physical objects of varying complexity
• by having two or more sources of light producing complex shadows
• by allowing more varieties of actions like pushing, grasping, lifting, rotating, placing on something else, inserting into a space, etc.

The set of possible polyflaps is very large (a) because there is a very large set of possible polygons, taking into account different numbers of sides, different angles between sides, and different lengths of sides, and (b) because for each such polygon a large number of more or less different polyflaps can be produced from one fold of between 0 and 180 degrees, to produce a 3-D object.

Different polyflaps will have different multi-stable possibilities depending on the shape of the original polygon and how it is folded.

So there is a large set of possible configurations of a single polyflap. Each configuration can be viewed from multiple viewpoints, adding to the complexity.

ADDED 22 Nov 2017: Note on triangle deformation
I now realise that videos of polyflaps, either taken with a moving camera while the polyflaps are stationary, or taken with a stationary camera while polyflaps are moving, or taken with moving camera while polyflaps are moving, will include rich examples of processes with mathematical structures that enable causal relations within the processes to be understood, even on a first viewing. I suspect this will turn out to be closely related to mechanisms required to explain the mathematical discovery process illustrated in this document on triangle deformation:
http://www.cs.bham.ac.uk/research/projects/cogaff/misc/deform-triangle.html

Also relevant is this discussion of the role of perception of partial orderings and topological changes, in reducing or eliminating the need for numerical estimation in perception of affordances and how they change:
http://www.cs.bham.ac.uk/research/projects/cogaff/misc/changing-affordances.html

• the biologists' notion of a species whose members are born or hatched underdeveloped and incompetent
  and also to refer to

• a pattern of cognitive development found in many such species that leads to unusual breadth and depth of cognitive competence most of which is not genetically determined but a result of learning about the environment.

In contrast, species described as 'precocial' by biologists are born or hatched relatively advanced physiologically, and behaviourally and cognitively competent (e.g. deer that run with the herd soon after birth, and chicks that peck for food soon after hatching). Their cognitive competence must be largely genetically determined, since there is no time to learn about the environment between birth and demonstrating the competence.

In fact, all species involve a mixture of both kinds of features as explained in [2]. While continuing to use the labels 'precocial' and 'altricial' to refer to types of species, we now propose to use different labels for the two main kinds of capabilities found in those species.

• Preconfigured capabilities (which may involve mechanisms, forms of representation, architectures, and behavioural competences based on them) are mainly determined genetically, though they need not be manifested at birth and there may be some calibration in use or re-shaping through reinforcement learning, usually a slow process.

• Non-preconfigured capabilities are discovered, created, constructed, developed by the individual during exploration and play, sometimes as a result of observing what others do sometimes as a side-effect of performing some old task in a new context where a surprise is encountered, or a feature of the task is noticed for the first time. Often, though not always, this is very fast learning, leading immediately to a new, lasting competence involving a structural change in behaviour, and possibly also a new understanding of structures and processes in the environment.

Note that the non-preconfigured capabilities depend on the operation of "higher level" pre-configured (i.e. genetically determined) bootstrapping capabilities. Adult members of altricial species seem to have a higher proportion of non-preconfigured competences than adult members of precocial species (born or hatched physically developed and behaviourally competent).

We are interested in explaining kinds of competence in the second category, competences that are not genetically determined but arise from play, exploration, observation and imitation using powerful genetically determined 'meta-level' learning mechanisms. This document is about some possible domains in which such mechanisms might operate in robots and animals.

-- NB 1: Not necessarily the best or only domain

-- NB 2: Little or no role for object recognition initially

When the lighting produces shadows or highlights or merely variations of intensity the complexities increase. Likewise structure in the background.

In addition to the variation in the possible scenes and the possible images that can be produced in those scenes, there are multiple possible actions that can be performed by a robot with a two-fingered gripper on each polyflap in each of its stable configurations. Once the polyflap is grasped and lifted there are large numbers of additional configurations many not included in the stable configurations when the polyflap is resting.

Moreover from each configuration there is a large space of possible types of movement (combinations of translation and rotation) that can lead to many other configurations.

Two views of some unfolded polyflaps lying on the carpet
(which the robot will not necessarily encounter -- e.g. because they are too hard to pick up with a typical gripper, though at a later stage the robot might learn how to use the corner of a polyflap to lift an unfolded polyflap):

Fig 1: flat1.jpg

Fig 2: flat2.jpg

Each polyflap is made by folding one of those polygonal shapes once and leaving the fold 'open' to produce a 3-D shape. Several polyflaps can be placed close together, in some cases with one resting on two or more others, to produce 3-D polyflap configurations. Some examples are in the next lot of pictures.

3-D Polyflap configurations derived from the above polygons:

Can you tell which polyflap comes from which polygon?

Fig 3: pol01.jpg

Fig 4: pol02.jpg

Fig 5: pol03.jpg

Fig 6: pol04.jpg

Fig 7: pol05.jpg

Fig 8: pol06.jpg

Fig 9: pol07.jpg

Fig 10: pol08.jpg

Fig 11: pol09.jpg

Fig 12: pol10.jpg

Fig 13: pol11.jpg

Fig 14: pol12.jpg

Fig 15: pol13.jpg

Fig 16: pol14.jpg

Fig 17: pol15.jpg

Fig 18: pol16.jpg

The pictures shown here all include some hastily made polyflaps, and are merely illustrative. The polyflaps were made by cutting an old brown paper envelope into polygons, then folding each one once.

For the purposes of the research (on requirements for an altricial robot learning how to perceive, understand and use spatial affordances) I would expect to move to more carefully made polyflaps which are not as flexible as paper. Plastic polyflaps may be used by Jackie Chappell in experiments with parakeets.

No flat polygons are being used because robot hands likely to be available to us would not easily be able to lift a completely flat object. (Likewise a very young child?)

We would start with simpler configurations than those shown, e.g. providing one polyflap at a time at first.

Sub-projects
An interesting sub-project would aim to produce a robot that when faced with a previously constructed configuration of polyflaps can dismantle it one piece at a time without knocking anything over in the process.

Another interesting sub-project would be to get a robot to develop a mathematical theory of polyflaps, e.g. specifying conditions under which they are stable, and relationships between initial polygonal shapes and the 3-D structures that can be obtained from them.

Note added 25 Sep 2017
An extension of this would include understanding proto-affordances, e.g. ways of reasoning about possible variations in polyflap configurations, and considering questions such as which are stable, which would be useful intermediary stages in constructing new more complex structures (given a particular collection of primitive polyflaps).

These would require development of topological and geometric forms of reasoning, not learning statistical regularities (which seems to have been the only use of polyflaps in working AI systems since this project was first proposed). E.g.

Marek Kopicki, Sebastian Zurek, Rustam Stolkin, Thomas Moerwald, Jeremy Wyatt, 'Learning modular and transferable forward models of the motions of push manipulated objects'
Autonomous Robots,, June, 2017, 01, vol 41, issue 5, pp. 1061--1082, Springer, US.
https://doi.org/10.1007/s10514-016-9571-3

So polyflap competence requires structural-generative competence, probably using forms of representation with compositional semantics. Thus we have a domain that seems to have some of the key features often attributed to linguistic competence but not involving linguistic competence.

The suggestion is that this is closely related to the kinds of domains in which altricial animals gain competence without using language, and that the abilities that evolved to support development of such competence were important precursors to evolution of linguistic capabilities.

Like Galileo's and Newton's thought experiments, the polyflap world abstracts away from a huge amount of complex and messy detail of the real world (including some of the detail) involved in the ability to produce fast fluent behaviours.

We conjecture that what remains includes a substantial challenge that can lead to a major advance in our understanding of a variety of issues regarding perception of affordances, learning, reasoning, forms of representation, and architectures that build themselves while acting in the world, as seems to occur in the so-called 'altricial' species (unlike 'precocial' species whose individuals start with a much richer collection of genetically determined capabilities whose subsequent development seems to be far more circumscribed).

It is also conjectured that despite the complexity of the polyflap domain there is an interesting subset of it that can be fruitfully explored in the near future in research on cognitive robotics -- both in CoSy and in other related projects by our collaborators.

Moreover, this is only one of a collection of domains that can be investigated in similar fashion within a larger interdisciplinary collaborative research programme including cognitive robotics, linguistics, developmental psychology, neuroscience, biology, and philosophy (among others). In parallel with the polyflap domain we are also investigating simpler domains, e.g. objects being pushed around on 2-D surfaces.

Our physical world allows a huge variety of more or less distinct domains to be learnt about by children, animals, and robots, and the polyflap domain is merely one of those selected because it seems to suit some of the constraints of what can be done with a rather limited one armed robot with a gripper. We also expect to investigate what happens when birds with manipulative capabilities meet polyflaps. Perhaps parallel research could introduce children to polyflaps.

The polyflap domain is just one among many other domains associated with classes of toys and types of materials, e.g. blocks, sand, stones, water, mud, string, paper, pencils, crayons, paint, plasticine, lego, meccano, tinker-toys, dolls of many kinds and their accessories, electronic-circuit building toys, drawing and painting kits, and of course other animals and/or other humans.

We hope that collaborators in other projects will do parallel investigations, so that we can share our tools and our discoveries in a process of joint exploration.

Of course, one of the important questions is how learning about different domains supports creative new combinations, e.g. combining string with other things, in actions like wrapping up, pulling, swinging, tying down, or assembling structures from things that don't intrinsically stick together, e.g. making things out of sticks.

One conjecture underlying this work is that the notion of learnt sensorimotor contingencies will not prove rich enough to capture what a child or robot has to learn in dealing with a domain like this. The issue is discussed further in http://www.cs.bham.ac.uk/research/projects/cosy/papers/#dp0603 (COSY-DP-0603: Sensorimotor vs objective contingencies).

So we have deliberately chosen tasks where the recognition of 'whole' objects is not an intermediate goal (though it may be a side-effect), and where most of the visual perception that is produced does not depend on recognition of objects, though it may depend on learning to recognise various 3-D structure fragments and their relationships (described below) and also fragments of motion of 3-D structure fragments.

So our aim is to find ways of perceiving and understanding what is seen that does not presuppose recognition of previously encountered objects nor any linguistic competence.

If we can explain the nature of that spatial understanding we may hope to shed light on a number of central features of human intelligence:

(a) generative learning capability which involves the *unending* ability to extend the ontology used for thinking about, perceiving and acting in the world

(b) the fact that thinkers of many kinds have attested to the power of human-spatial forms of representation and reasoning, and the use of spatio-temporal metaphors in thinking about and dealing with problems in many domains that are not spatial, including such things as family relationships, search strategies in abstract problem solving, and many projects of mathematics.

-- Static fragments

-- Larger fragments and combinations of fragments

-- Seeing fragments of processes

Clearly there are various kinds of surprise (Matthias Scheutz and I have an incomplete partial conceptual analysis of surprise in a larger incomplete paper. We should make this available.)

• Passive surprise: something novel

  There's noticing something that has not been seen before (lump of yogurt on the carpet in the video below[4]) which can trigger various new actions, e.g. pick it up, put it in spoon, put it in mouth. But then having done that, a new goal can be triggered: put another lump of yogurt on carpet.

• Active surprise 1: failed prediction
• surprise 2: failed action

• Other surprises: unexpected (side-)effects of actions,
  o interesting ones
  o good ones
  o bad ones

• sketch some other domains
• identify initial architecture and patterns of growth of architecture
• identify initial ontology and patterns of growth of ontology (both in the robot and also in *us* as we study these phenomena!)
• identify initial forms of representation and patterns of growth of forms of representation
• identify initial chunking mechanisms for isolating fragments that should be stored, 'labelled' and made re-usable in various ways (what ways?)
• identify initial forms of 'syntactic' or 'combinatorial' competence for recombining chunks in perception and action. (For a very simple example in a very simple world with a moving finger and a counter, originally invented by Oliver Selfridge see http://www.cs.bham.ac.uk/research/poplog/teach/finger )
• identify forms of motivation and affective responses that will help to drive the robot's exploration and learning.
• some of this may include mathematical properties of patterns of sensory input or patterns of motor output that trigger some 'store this as a chunk' mechanism. (Compare Gardenfors, among others.)
• once the robot has started learning about chunks it can start learning generalisations associating those chunks.
• Having learnt to construct representations of what is perceived, or done, it could also (with an architectural extension -- towards deliberative mechanisms) start 'playfully' constructing representations of possible percepts, possible actions, not currently perceived or done.
• With an appropriate architectural change some of those representations could begin to function as goals. (How would they be chosen? more on motivation.)
• Using the deliberative capability (representing what does not exist), stored associations will trigger predictions of various sorts (forward chaining) but can also trigger action selection processes, then later on planning processes (backward chaining), and still later explanation processes.
• Once predictions are made, predictions can be checked (another architectural development), and the difference between successful and unsuccessful predictions learnt. (Or does that distinction have to be innate -- like Kant's presumption of an 'objective' external reality underlying what is perceived??).
• and then various ways of dealing with failed predictions can be explored and learnt, including changing associations, debugging prediction and planning strategies, etc. (Sussman's HACKER, Push Singh's PhD thesis and other refs.)

More on affect: perceived internal and external states and processes, and various kinds of actions need to be evaluated. It's clear from watching very young children that a lot of the motivation and evaluation has nothing to do with satisfying biological physical needs (getting food, reducing various kinds of physical discomfort, generating 'physical pleasures') or even social needs. There are deeper cognitive needs.
(See also this paper on "Architecture-based motivation" contrasted with "Reward-based motivation":
http://www.cs.bham.ac.uk/research/projects/cogaff/misc/architecture-based-motivation.html

One of the interesting questions to be addressed in the long run on the basis of research like this could be: what are the differences in learning capabilities of human children born with or without arms, e.g. those affected by the thalidomide tragedy in the 1960s. E.g. See

Other investigations of a related type could investigate learning capabilities of blind children, using robots with only arms, and no vision.

If we can identify different sorts of 'initial' architectures etc. we may be able to come up with a new way of classifying types of animal species, going far beyond the altricial/precocial distinction.

This can also lead to new kinds of research into evolutionary trajectories and the kinds of environments that trigger, facilitate or inhibit them.

Various Polyflap-based domains could provide useful environments in which to test the ideas about pre-configured and meta-configured competences in Ref [2.b], especially the ideas about cascaded interactions between genome and environment summarised in this diagram:

Diagram based on ideas in Ref 2b.

Three multidisciplinary research proposals to investigate these ideas, by relating
experiments on animal cognition and work on robot cognition were turned down.

The possibilities for expanding the robot work in different kinds of directions and also the interdisciplinary collaborations are endless.

[2.a] http://www.cs.bham.ac.uk/research/cogaff/05.html#200502
The altricial precocial spectrum for robots
Paper co-authored with Jackie Chappell, for IJCAI'05.
The next paper was a sequel, expanding these ideas.

[2.b] http://www.cs.bham.ac.uk/research/projects/cogaff/07.html#717
Jackie Chappell and Aaron Sloman, (2007)
Natural and artificial meta-configured altricial information-processing systems, invited paper for International Journal of Unconventional Computing, 3, 3, pp. 211--239,

[3] http://www.cs.bham.ac.uk/research/cogaff/challenge.pdf
Challenge to AI vision researchers and others, regarding perception of shape-based affordances.

[4] Baby and yogurt video

3.6MB Mpeg

19MB Mpeg

[5] http://www.cs.bham.ac.uk/research/cogaff/
http://www.cs.bham.ac.uk/research/cogaff/talks/
Papers and talks in the Cognition and Affect project directory.

[6] The UKCRC 'Architecture of Brain and Mind' research grand challenge.
http://www.cs.bham.ac.uk/research/cogaff/gc/
http://www.ukcrc.org.uk/

[7] On scenario driven research:

Overview of methodology
Draft Scenario template

[8] Provisional work on ontologies for a pre-linguistic agent

ontology.pdf
ontology-for-a-manipulator.txt
evolving-ontologies-for-minds.txt (newsgroup posting 1999)

This web site is still under construction.

Updates:

• REORGANISED 6 Jun 2006

• NOTE ADDED: 16 Jan 2006:
  My thoughts have moved on a lot since I wrote these notes between May and October 2005. In particular
  o See the discussion of Orthogonal recombinable competences.
  The ideas presented there suggest that a project of the sort envisaged should also explore learning about different kinds of physical stuff.

  The ideas are expanded in

  o A presentation on A (possibly) new theory of vision as concurrent simulation of processes at different levels of abstraction in partial registration with the optic array,
  o A partly overlapping presentation on Two views of child as scientist: Humean and Kantian

  Part of the background to this paper is a claim that AI vision research (as opposed to pattern recognition, tracking, and learning research has made very little progress in the last 30 years, for reasons indicated in http://www.cs.bham.ac.uk/research/cogaff/challenge.pdf
• 10 Aug 2005
  Introduction expanded, terminology clarified (following IJCAI presentation with Jackie Chappell).
  A new, related web page concerned with the hypothesis of Primacy of dynamical systems (PDS) provides some of the context for the polyflap discussion.

• Tue Jul 26 11:45:08 BST 2005
  (Converted from plain text to html)
• Wed May 25 12:10:18 BST 2005
  (Extended descriptions of what the robot will learn, and elaborated on the context)
  (New pictures from stiffer, brown paper)