Grounded Models as a Basis for Intuitive Reasoning

1 Introduction

The concept of grounded model introduced in this paper is based on some ideas and mechanisms described in The Talking Heads Experiment [Steels, 1999]. The experiment involves a set of robotic "Talking Heads" playing language games with each other about scenes perceived through their cameras on a white board in front of them. In particular, we focus on the conceptualization part of a language game, called the discrimination game, which generates the meaning of the verbal hint transmitted by the speaker to the hearer in a language game.

The discrimination game [Steels, 1996] is played by a single agent, and consists of the following steps. First, the agent perceives an image on a white board through his camera, segments the image into coherent units, and computes various sensory characteristics about each image segment, such as its color, horizontal or vertical position. Then, the agent chooses an image segment as topic, and tries to find a combination of categories that distinguishes the topic from the other objects in the image. The game succeeds if the agent finds a combination of categories that is true for the topic, but it isn't true for the other objects in the image. If the game fails, the agent adapts its internal structures to become more successful in future games.

The rest of this section describes, in some detail, the different processes and cognitive structures involved in the discrimination game.

1.1 Perception

The agent captures an image on a white board in front of him. Geometric figures of various sizes, shapes and colors can be pasted on the white board. Then, the agent segments the image into coherent units. Next low level visual processes gather information about each segment, such as its color, horizontal or vertical position. Each process outputs its information on a sensory channel. We assume that there are only three primitive sensory channels.

•  HPOS(obj) contains the x-midposition of object obj.
•  VPOS(obj) contains the y-midposition of object obj.
•  GREY(obj) contains the average gray-scale of obj.

The values on the sensory channels HPOS, VPOS and GREY are scaled so that its range is the interval (0.0 1.0). Consider the three objects numbered in figure 1, object 1 has the values HPOS(Obj1)=0.5, VPOS(Obj1)=0.5, GREY(Obj1)=0.5, object 2 the values HPOS(Obj2)=0.5, VPOS(Obj2)=0.2, GREY(Obj2)=0.1, and object 3 the values HPOS(Obj3)=0.8, VPOS(Obj3)=0.2, GREY(Obj3)=0.3.

In addition to three primitive sensory channels, there are some sensory channels constructed from them.

•  HPOS-DIFF(obj1,obj2) contains the difference of the x-midpositions of objects obj1 and obj2, i.e. HPOSDIFF(obj1,obj2)=HPOS(obj1) - HPOS(obj2).
•  VPOS-DIFF(obj1,obj2) contains the difference of the ymidpositions of segmented objects obj1 and obj2.

•  GREY-DIFF(obj1,obj2) contains the difference of the average gray-scale of objects obj1 and obj2.
•  EQUAL(obj1,obj2) is defined as a predicate, i.e. a functionwhich takes the value 1 (i.e. true) if the values of the primitive sensory channels HPOS and VPOS are equal for obj1 and obj2, and 0 (i.e. false) otherwise.

For example, the sensory channel VPOS-DIFF has the value 0.3 when it is applied to the pair of objects formed by object 1 and object 2, i.e. VPOS-DIFF(Obj1,Obj2)=0.3. The range of the sensory channels HPOS-DIFF, VPOS-DIFF and GREY-DIFF is (-1.0 1.0). The range of the sensory channel EQUAL is the discrete set of Boolean values f0;; 1g.

1.2 Perceptually Grounded Categories

The data on sensory channels are values from a continuous domain (except for sensory channel EQUAL). To be the basis of natural language communication, these values must be transformed into a discrete domain. One means of categorization is to divide up each domain of values on a particular sensory channel into regions and assign a category to each region. For example, the range of the HPOS channel can be cut in two halves leading to a distinction between [LEFT] (0.0 < HPOS < 0.5) and [RIGHT] (0.5 < HPOS < 1.0). Object 3 in figure 1 has the value HPOS=0.8 and would therefore be characterized as [RIGHT]. Similarly, the VPOS-DIFF channel can be cut in two halves as well leading to a distinction between [ABOVE] (0.0 < VPOS-DIFF < 1.0) and [BELOW] (-1.0 < VPOS-DIFF < 0.0).

It is always possible to refine a distinction by dividing its region. Thus an agent could divide the bottom region of the HPOS channel (categorized as [LEFT]) in two subregions [TOTALLY-LEFT] (0.0 < HPOS < 0.25), and [MID-LEFT] (0.25 < HPOS < 0.5). The categorization networks resulting from these consecutive binary divisions form discrimination trees.

We label categories using the sensory channel fromwhich they operate, followed by the upper and lower bound of the region they carve out. Thus [TOTALLY-LEFT] is labeled as [HPOS 0.0 0.25], because it is true for a region between 0.0 and 0.25 on the HPOS channel. We assume that perceptually grounded categories correspond to n-ary predicates of first order logic. For example, the category [HPOS 0.0 0.25] corresponds to the unary predicate [HPOS 0.0 0.25](obj), and the categories [VPOS-DIFF 0.0 1.0] and [EQUAL] to the binary predicates [VPOS-DIFF 0.0 1.0](obj1,obj2) and [EQUAL](obj1,obj2).

There are other ways to move from the continuous domain of sensory channels to the discrete domain of categories. We could introduce focal values and associate a category with each of them. In this case, the categorization process consists in identifying the focal point that is closest to an object's value.

1.3 Concepts

Perceptually grounded categories can be combined to construct concepts. We use a set of concepts which can be defined by induction as follows.

1. If p is an n-ary category and x 1 ;; : : : ;; x n are variables, then p(x 1 ;; : : : ;; x n ) is a concept.
2. If c is a concept, then the negation of c (written :c) is a concept.
3. If c 1 and c 2 are two concepts, then the disjunction of c 1 and c 2 (written c 1 _ c 2 ) is a concept.

The symbols ^, ! and $ are introduced as abbreviations: (1) c 1 ^ c 2 is an abbreviation of :(:c 1 _ :c 2 ); (2) c 1 ! c 2 is an abbreviation of :c 1 _ c 2 ; (3) c 1 $ c 2 is an abbreviation of (c 1 ! c 2 ) ^ (c 2 ! c 1 ). Notice that the set of possible concepts is the set of free-quantifier formulas that can be constructed from the predicates associated with perceptually grounded categories.

We distinguish three types of concepts: intuitive truths, intuitive falsehoods, and regular concepts. Some concepts have the property of being true for every possible tuple of segmented objects. We will call them intuitive truths or just truths. For example, the concept [VPOS-DIFF 0:0 1:0](obj1;; obj2) ! :[VPOS-DIFF 0:0 1:0](obj2;; obj1) is an intuitive truth.

Other concepts have the property of being false for every possible tuple of segmented objects. We will call them intuitive falsehoods. For example, the concept [VPOS-DIFF 0:0 1:0](obj1;; obj1) is an intuitive falsehood.

The rest of the concepts, called regular concepts or simply concepts, can be used to discriminate those tuples of objects that satisfy them from those tuples of objects which do not make them true. For example, the concept :[VPOS-DIFF 0:0 1:0](obj1;; obj2) ^ :[VPOS-DIFF 0:0 1:0](obj2;; obj1), which is true for a pair of segmented objects obj1 and obj2 if neither obj1 is above obj2, nor obj2 is above obj1, is a regular concept.

2 GroundedModels

2.1 Categorizers

A categorizer is a cognitive procedure capable of determining whether a category applies or not. For example, the behavior of the categorizer for category [VPOS 0.0 0.5](obj) can be described by a function that takes the value 1 if 0.0 < VPOS(obj) < 0.5, and 0 otherwise.

The categorizers of nonatomic concepts are compositions of the categorizers of categories. For example, the behavior of the categorizer for concept [VPOS-DIFF 0:0 1:0](obj1;; obj2)g ^

:[VPOS-DIFF 0:0 1:0](obj2;; obj1) can be described by a function that takes the value 1 if 0:0 < [VPOS-DIFF 0:0 1:0](obj1;; obj2) < 1:0, and [VPOS-DIFF 0:0 1:0](obj2;; obj1)  0:0.

Categorizers establish explicit connections between symbols (categories and concepts) and reality (perceptual input as processed by sensory channels). These connections are learnt through language games [Wittgenstein, 1953], and allow agents to check whether a concept is true or not for a given tuple of segmented objects. Most importantly, they provide information on the perceptual and cognitive processes an agent must go through in order to understand or evaluate a given concept.

2.2 Grounded Models

A grounded model is the set of concepts and categorizers constructed by an agent at a given time in its development history. These concepts and categorizers are organized as follows.

1. A discrimination tree for every sensory channel containing categories and categorizers which use the values on that sensory channel.
2. A set of intuitive truths containing concepts which are true for every possible tuple of segmented objects.
3. A set of defined concepts containing nonatomic concepts which are neither intuitive truths nor intuitive falsehoods.

A grounded model reflects the conceptualization of the world constructed by an agent at a given time in its development history. Grounded models are not static, but evolve and adapt as the agent is confronted with new experiences.

Consider the grounded model G of an agent that has constructed top-level categories and categorizers for each sensory channel, but does not have any defined concept or intuitive truth yet. The categorizers of this grounded model are cognitive procedures whose behavior can be described by linear constraints. We associate a linear constraint describing the behavior of each categorizer with a predicate symbol that corresponds to the category it is capable of recognizing.

[VPOS 0:0 0:5](x)  0:0<VPOS(x) ^ VPOS(x)<0:5 [VPOS 0:5 1:0](x)  0:5<VPOS(x)^VPOS(x)<1:0 . . . [GREY-DIFF -1:0 0:0](x;; y)  -1:0 <GREY(x)-GREY(y) ^ GREY(x)-GREY(y) < 0:0 [GREY-DIFF 0:0 1:0](x;; y)  0:0<GREY(x)-GREY(y) ^ GREY(x)-GREY(y)<1:0

Categorizers are probably implemented by neural networks in natural agents. We only use linear constraints to model their behavior. We are not assuming therefore that agents do learn such constraints, but that they build them in their perceptual systems.

2.3 Grounded Models and Axiomatic Theories

A grounded model looks like an axiomatization or a theory of first order logic, but differs from it in some aspects. Like a logical theory, it has a set of basic concepts (which correspond to perceptually grounded categories), and a syntax that allows constructing defined concepts from basic concepts 1 .

One of the most important differences between grounded models and logical theories is the form in which basic concepts are defined. In a grounded model, concepts are defined by cognitive procedures which given a tuple of objects return one of the Boolean values f0;; 1g. For basic concepts, these cognitive procedures are the categorizers associated with perceptually grounded categories. For nonatomic concepts, the cognitive procedures are logical compositions of the cognitive procedures associated with basic concepts. Basic concepts have, therefore, precise and explicit meanings in grounded models, which determine the set of concepts that can constitute the intuitive truths of a grounded model. This can be contrasted with logical theories, in which concepts are defined by simple symbols whose meaning is only constrained by the relationships the axioms in the theory postulate about them.

1 In logical theories, basic concepts are specified by the nonlogical symbols of the language, and defined concepts by mathematical abbreviations or definitions.

It should be observed as well, that the meanings of basic concepts in grounded models are not arbitrary functions from the set of possible tuples of segmented objects on the Boolean values f0;; 1g, as it happens with the meanings of predicate symbols in model theory semantics. They are cognitive procedures which should make intuitively plausible distinctions (in general, relations) using data extracted by realistic sensory channels operating on real world environments. This fact has important consequences on the shape and inferential power of grounded models. First, the meanings of basic concepts are highly constrained by physical aspects of the environment and the sensory-motor apparatus of the agent. Because, in order to qualify as a possible meaning, a cognitive procedure has to be implementable as a relatively easy computation on the range of values produced by realistic sensory channels. Second, the set of intuitive truths is constrained as well by mathematical relationships holding among the meanings of basic concepts. For example, the following relationship holds among the meanings of the concepts [VPOS 0:0 0:5](obj1), :[VPOS 0:0 0:5](obj2) and [VPOS-DIFF 0:0 1:0](obj1;; obj2).

[VPOS 0:0 0:5](obj1) ^ :[VPOS 0:0 0:5](obj2) ! :[VPOS-DIFF 0:0 1:0](obj1;; obj2)

Therefore, the following concept can never be an intuitive truth of a grounded model containing the basic concepts [VPOS-DIFF 0:0 1:0](obj1;; obj2) and [VPOS 0:0 0:5](obj1).

This contrasts with axiomatic theories in which any well formed formula can be stated as an axiom.

The mathematical relationships holding among the meanings of basic concepts are one of the most important features of a grounded model. As we will see later on, they allow agents to reason about their environment without having an explicit axiomatization of it. This form of intuitive reasoning is commonly used by people to reason about everyday problems, and it is also the basis of formal reasoning in the sense of mathematics. This is so, because grounded models provide an explanation for the fact that we accept certain axioms as intuitively true when we build axiomatic theories in mathematics.

We can observe a dual character between grounded models and logical theories. Grounded models are constructed around the notion of concept, i.e. the meanings of concepts determine the set of intuitive truths of a grounded model. Logical theories are instead constructed around the notion of axiom, i.e. the set of axioms determine the set of theorems of a theory, and to some extent the meanings of concepts. By looking at our example of grounded model G, one could say that grounded models are logical theories whose axiom set consists only of definitions of concepts. But this is not true, because the meanings of concepts in grounded models are cognitive procedures rather than logical formulas. Sometimes, the behavior of these procedures can be described by mathematical functions, as in the case of groundedmodel G, but other times it cannot be easily described this way, as in the case of categorizers for approximate concepts. In the first case, intuitive reasoning can be done by exact mathematical methods, as we will see later on, but in the second case different techniques (e.g. simulation) must be applied.

3 ConstructingGroundedModels

Grounded models are constructed as a side effect of agents' activity. In particular, the grounded models studied in the paper are constructed by agents as they play discrimination games. A discrimination game [Steels, 1996] is played by a single agent. The agent perceives a scene and chooses a topic from the possible tuples of segmented objects in the scene. He then uses his current grounded model to come up with a category or nonatomic concept that is valid for the topic, but not for any other tuple of objects in the context. The game succeeds if the agent can find such a category or concept. If the game succeeds, the use and success counters of the categorizers involved go up 2 . If the game fails, the use counters of the categorizers involved go up, and a repair process in which a new category or concept is generated takes place.

Initially, the agent constructs top-level categorizers for each sensory channel that has contained distinctive data in the recent past. If a channel has the same data for every segment it is not going to be possible to find a distinctive category from it. Afterwards, the agent extends his discrimination trees or constructs new concepts from existing categories or concepts.

A categorizer for a new category is constructed by taking a categorizer node in a discrimination tree and dividing its range into two new subranges. For example, if we take the categorizer for [HPOS 0.0 0.5](x), which is true when the object is in the left most half of a scene, two new categorizers are created by dividing [0.0 0.5] into two halves, one for the range [0.0 0.25] ([HPOS 0.0 0.25](x) or totally left), and one for the range [0.25 0.5] ([HPOS 0.25 0.5](x) or mid left). A new categorizer is added to the tree for each of these halves.

A categorizer for a new concept is constructed by composition of the categorizers of existing categories or concepts. We use three composition operations: negation, conjunction and substitution. These operations allow constructing every free-quantifier formula of the first order language defined by the categories of the grounded model. In general, new concepts are constructed from categories or concepts that have been useful in previous games, i.e. which have a high rate.

2 The use and success counters of a categorizer are used for different purposes, such as computing the categorizer rate (which is the result of dividing success by use), or choosing which categories or concepts should be used to create new concepts.

Two logical compositions are preferred to relate existing categories and concepts: conjunction and implication. Conjunctions tend to be more discriminating than their components, increasing the chances of success in future games. Implications, on the other hand, are commonly used to describe causality, and world facts are often expressed as causal laws.

The final step of the discrimination game, called assimilation, is as follows. Whenever the agent constructs a new concept, he checks whether it is an intuitive truth or a falsehood. If the new concept is an intuitive truth, the concept is added to the set of intuitive truths of his grounded model. If it is an intuitive truth and it is already in his grounded model, or if it is a falsehood, the concept is ignored and it is not included in the grounded model. Finally, if the concept is a regular concept, and it is not already in his current groundedmodel, it is added to the set of defined concepts of the grounded model.

4 IntuitiveReasoning

The process by which an agent tries to determine whether a concept is an intuitive truth, a regular concept or a falsehood of a grounded model is called, in this paper, intuitive reasoning. An intuitive truth of a grounded model is a concept whose meaning is true for every possible tuple of segmented objects. A regular concept is a concept whose meaning is true for some tuples of objects, but false for others. And a falsehood is a concept whose meaning is false for every possible tuple of segmented objects.

We hypothesize that intuitive reasoning happens by a process of simulation in natural agents. First, they construct the categorizer of the concept they want to check combining the categorizers of basic concepts of their grounded models. Then, they try to find combinations of values of their sensory channels that can satisfy the categorizer of the concept. This process can be seen as a form of constraint satisfaction by search, in which the agents generate possible combinations of values for sensory channels, and test them using the concept's meaning. Of course, the search cannot be exhaustive, because sensory channels take values on a continuous domain. The search is performed at the level of subregions in which the categorizers involved take different values. Each subregion is represented by a single value of a sensory channel, and the combinations of subregions for sensory channels by tuples of values. As soon as a value is shown incompatible with the concept's meaning or sufficient for satisfying it, all the combinations containing that value can be eliminated from the search space. This allows reducing the complexity of the search process. Sometimes, however, the agents cannot explore the entire space of possibilities, because it is too complex, and they make errors when they try to approximate the result of the search.

If, after the search process, the agents cannot find any combination of values which does not satisfy the concept's meaning, the concept is seen as an intuitive truth. If they find some combinations that satisfy it, and others which do not, the concept is seen as a regular concept; and, otherwise, as an intuitive falsehood.

5 Spatial Reasoning

To clarify the ideas discussed in previous sections, we compare a first order theory T S , which can be used for reasoning about spatial relations among objects on the plane, with the grounded model G described in section 2. The language of T S consists of two binary predicates A(x;; y) and R(x;; y). Its axiom set is as follows.

In the classical approach to AI, if one wants to build an agent capable of reasoning about spatial relations, such as Above or Right-of, one must construct a first order theory like T S and apply automatic theorem proving methods. Theories like this one are constructed by agent designers, not by the agents themselves, and they must be extended or updated by designers as well when the agents are faced with new challenges. These theories are normally used to determine whether a fact follows or not from the axioms of the theory, i.e. whether it is a theorem, or to extract answers in the form of sets of tuples of objects that satisfy a particular property expressed as a logic formula.

We are going to see how a simple grounded model, such as the one shown in section 2, can be used for the same tasks as T S . The difference is that grounded models are constructed and updated by agents rather than agent designers, and that inference is done by intuitive reasoning.

First, we see that every theorem of T S is an intuitive truth of the grounded model G. In order to do that, we need to prove that every axiom of T S is an intuitive truth of G, and that the intuitive truths of G are closed under the inference rule of resolution.

When the behavior of the categorizers for basic concepts can be described by linear constraints, intuitive reasoning can be seen as a process of linear constraint satisfaction. In particular, in the grounded model G, the behavior of the categorizer of every concept can be described by a disjunction of linear constraints which can be computed by replacing every category by a linear constraint describing the behavior of its categorizer in the concept expression, and computing the disjunctive normal form of the result. Proving that the meaning of a concept is true for every possible tuple of segmented objects requires proving that the constraint system associated with the concept is true for every value of every sensory channel. This is equivalent to proving that the constraint system associated with the negation of the concept's meaning is unsatisfiable.

For example, it can be shown that axiom 2 (which states that the relation above - A(x,y) - is transitive) is an intuitive truth of the grounded model G by checking that the following disjunction of constraint systems is unsatisfiable for every value of x, y and z in the interval (0.0 1.0). This formula has been obtained by: (1) replacing every category by a linear constraint describing the behavior of its categorizer in the concept associated with the negation of axiom 2 in the grounded model G; (2) replacing every instance of VPOS(x), VPOS(y) and VPOS(z) by x, y and z in the expression resulting from step 1; and (3) computing the disjunctive normal form of the result of step 2.

It is easy to check that this constraint system is unsatisfiable. A disjunction of constraint systems is unsatisfiable if each disjunct is unsatisfiable. Each disjunct is a linear arithmetic constraint that can be checked by a linear constraint solver, such as the one implemented in Sicstus Prolog.

The rest of the axioms of T S can be shown to be intuitive truths of the grounded model G by intuitive reasoning as well. Grounded model G provides, therefore, an explanation for the fact that we accept the axioms of T S as intuitively true, and consequently use them to build logical theories for spatial reasoning.

It can also be proved that intuitive reasoning in grounded models in which the behavior of the categorizers for basic concepts can be described by linear constraints is closed under resolution. That is, if two concepts are intuitive truths of a grounded model, its resolvent is an intuitive truth of the groundedmodel as well. Therefore, every theorem of T S can be shown to be an intuitive truth of the grounded model G by intuitive reasoning. Intuitive reasoning is not only much simpler than theorem proving for this domain, but it can also be automated using constraint satisfaction techniques as we have explained above. In fact, the grounded model G can be used to derive many more intuitive truths than the theorems of T S , because it contains categorizers for a broader set of basic concepts including the standard notions of up(x), down(x), right(x), left(x), dark(x), light(x), darker(x,y) and lighter(x,y).

Grounded models can be used to extract answers as well. For example, we can obtain all the triples of objects in the scene of figure 1 such that A(x;; y) ^ A(y;; z) by applying the categorizer of this concept to every triple of segmented objects in the scene, and picking up those triples that satisfy it.

Notice that, as soon as the agents have constructed categorizers for the basic concepts above and right-of, they are capable of deriving every theorem of T S by intuitive reasoning. This means that an agent capable of linguistic competence [Steels, 1998] at the level of interpreting and generating first order logic formulas does not need an axiomatization to reach sound conclusions about its environment by intuitive reasoning in this domain. It only needs to understand the basic concepts involved, and the grammar rules by which concept expressions are translated into concept meanings. We have made some recent progress in this aspect, which is illustrated by the following experiment. Figure 2 shows the results of an experiment in which a couple of agents learn both logical categories, which correspond to the usual connectives of propositional logic (:, ^, _, !, $), and a shared vocabulary to refer to them. The particular language game used to learn logical connectives is a variation of the language games described in [Steels, 1999]. The experiment shows that agents can indeed acquire linguistic competence at the level of interpreting and generating free-quantifier first order formulas.

Finally, grounded models can be used for a task that logical theories do not support, namely, concept generation. As a side effect of their activity playing discrimination games, the agents construct new concepts which they store and can use afterwards for different purposes, such as classification, discrimination or communication. Some of these new concepts are intuitive truths. These are stored in the set of intuitive truths of a grounded model, and allow the agents to learn general facts about their environment 3 . Others are regular concepts, and are stored in the set of defined concepts of a grounded model. Axiomatizations and conceptualizations get constructed then as a side effect of agents' activity, and do not have to be built into them.

6 Conclusions

We have introduced grounded models and compared them to axiomatic models of mathematics. Grounded models differ from axiomatic theories in establishing explicit connections between language and reality that are learnt through language games. They are constructed and updated by autonomous agents connected to their environment through sensors and actuators using some conceptualization mechanisms and language games described in [Steels, 1999]. They are based on conceptualization and support a form of intuitive reasoning, which can be done sometimes by constraint satisfaction and it is argued to be the basis of some axiomatizations. This has been illustrated with a simple example of spatial reasoning.

Acknowledgments

The author would like to thank Luc Steels for many interesting conversations on the topics of the origins of language and grounded representations.

3 The set of intuitive truths of a grounded model plays an analogous role to that of the axiom set of a logical theory.