Learning Forward Models for Robots

1 Introduction

The purpose of any robot is to interact with its environment. There is, however a separation between the effects a robot can produce in the environment and its mechanisms for producing them, its motor commands. To overcome this, inverse and forward models can be used [Jordan and Rumelhart, 1992]. In robotics, forward models enable a robot to predict the consequences of its actions: given the current state of the robot they predict how a set of motor commands will in�uence the state of the robot or its environment, or how the observations will be perceived by the robot's sensors, as shown in �gure 1. Conversely, inverse models are used to output the necessary motor commands to achieve or maintain a desired goal. Both these internal models have been hypothethised to be used in human motor control [Wolpert et al., 1998]. The ability to predict the consequence of actions has several uses in robotics, such as allowing mental rehearsal of possible actions or imitating the actions of humans or other robots [Demiris, 2002].

Practically any environment a robot works in will change, or have properties which cannot be modelled beforehand. Even if the environment is assumed to be completely predictable, endowing the robot with this knowledge may be beyond the abilities or desires of its programmer. A truly autonomous robot, therefore, needs to be able to learn and adapt its own forward models. Forward models of dynamic and uncertain environments, and how they can be learnt, is the topic which is investigated in this paper. Although forward models are used in a different context, there are similarities here to work on calibrating action-sensor models in robotics e.g. [Roy and Thrun, 1999]. A robot learning about itself and its environment is in the position of being an active part of the system it is trying to learn about; this situation draws interesting parallels with learning in human infants, with mechanisms such as babbling (issuing random motor commands) used by infants to learn how to control their own motor system [Meltzoff and Moore, 1997].

A system is presented here that enables a robot to autonomously learn a forward model with no prior knowledge of its motor system or the external environment. Information about the effects of the robot's actions are captured via a vision system that clusters low-level image features to automatically �nd and track moving objects in a scene. The robot sends out randommotor commands to its motor system and receives information back from the vision system. This set of evidence is used to learn the structure and parameters of a Bayesian network, which represents the forward model. This model can then be used to enable the robot to predict the effects of its actions. By observing the gestures of a human with its own vision system, the robot can imitate the human gesture using the forward model it has learnt of its own motor system.

2 Representing forward models with Bayesian networks

Bayesian networks [Pearl, 1988] are an ideal method for representing forward models that can be learnt. They offer a way of representing the causal nature of a robot's control system within a rigorous probabilistic framework. The system here is aimed at learning and representing the causal associations between the robot's motor commands, the robot's state, and the observations of the robot's state that are received from the vision system. The motor commands and state of the robot are represented as random variables in the Bayesian network, and the causal relationships between them are shown with arcs. The Bayesian network represents a learnt probability distribution across the previous N motor commands, M1:N[t-d], the P current states S1:P [t], and the P observations of the state, O1:P [t]. The variable d represents the delay between a motor command being issued and robot's state changing; in real robotic systems it cannot be assumed that the effect of a motor command will occur after just one time-step, so this is a parameter that must be modelled and learnt. Figure 2 shows this structure.

The N motor random variables represent speci�c low-level motor commands of the robot. They could be discrete, e.g. 'open' or 'close' robot's gripper, or continuous, e.g. to set the height of the gripper. When the robot is issuing these commands there is no uncertainty as to their value, so they can be modelled as observed nodes. Alternatively, when the robot is observing the actions of another robot, or a human, whilst using its own forward model, they will be hidden nodes because the Bayesian network will receive no direct evidence of their value.

The P observations of the state-space O[t] are represented with square nodes descended from S[t] on �gure 2. The model treats O[t] as noisy observations of the actual state space so that noise from the vision system and other uncertainty can be modelled. Evidence about the actual state of the robot therefore enters the system here. How the state space, S[t], of the robot will be represented depends on the information received from the computer vision system. The state variables are all hidden nodes, because the robot's knowledge of itself only comes from noisy observations of the state, O[t].

3 How a robot can learn and use forward models

The structure for a forward model described in section 2 require several aspects to be learnt. Firstly, a representation of the state of the robot or the environment is needed. The richest source of this information comes from the robot's own vision system. However, this is also the most challenging source: information comes in the low-level format of pixels. Furthermore, if the robot is seeking to learn its forward models it has no prior information about what to track, because the environment is unknown. To extract information from a scene, a method is needed of automatically initialising and tracking the objects in it. The system presented here works by clustering tracked low-level features in the image according to their position and dynamics. This allows the position of objects in the scene to be tracked without any knowledge of the objects having to be supplied.

The low-level point features used are tracked using the Lucas-Kanade optical �ow algorithm [Lucas and Kanade, 1981]. The tracked points are clustered using the k-means algorithm according to their position and velocity. One drawback of this method of clustering is that the number of clusters must be known beforehand. This is overcome by �rst excluding stationary points and extracting increasing numbers of clusters. The error of a particular number of clusters is then taken to be the average squared distance of the optical �ow points from the centre of the cluster they are part of. The cluster number used is the one at which the curvature of this error with respect to the number of clusters is minimised [Hoppner et al., 1999]. The positions, velocities and sizes of each of these clusters can now be used as the observations, O[t], of objects for the forward model learning system. The actual state of the motor system, S[t], is taken to be a hidden discrete node for each object in the scene found by the vision system, and this will be connected to a continuous observation. With all the possible nodes in the Bayesian network speci�ed, the �nal task is to learn how the motor commands interact with the state, and train the parameters of the Bayesian network so that its predictions are as accurate as possible. To train a Bayesian network, a set of evidence is required: a data set containing actual observed values of the observed nodes in the network. This is a set of N motor commands executed at each time step, M1:N [t - d], together with the P observations of the state O1:P [t], the size, position or velocity of an object. The problem now is how to use this data to learn the structure and parameters of the Bayesian network. An inspiration for solving the problem of how to learn this Bayesian network can be taken from developmental psychology. Gopnik and Meltzoff compare the mechanism an infant uses to learn to that of a scientist: scientists form theories, make observations, perform experiments, make generalisations and revise existing theories [Gopnik and Meltzoff, 1998]. The notion of forming theories about the world is already dealt with here: a forward model can be seen as a theory of the robot's own world. The process a robot uses to obtain its data and infer a forward model from it is its experiment. An outline of what a robot's experiment involves is as follows, and is highlighted in �gure 4:

1. A particular motor command, M1[t] is chosen;
2. A value for this motor command is sent to the robot's motor system;
3. The vision system returns a set of observations of the scene, O1:P [t+d], for example, moving forward v ms-1 in direction x;
4. This training set of data, M1[t] and O1:P [t+d], is used to train the Bayesian network to develop a forward model;
5. The whole process is repeated, either with a different value for the current motor command, or a different motor command altogether.

The open issues here relate to the last two stages. Firstly, if the structure of the Bayesian network is not completely known, how should the data be used to learn the structure and parameters of the network? Most methods for learning structure from data are based on performing a search through the space of possible structures, with the goal of �nding the one that maximises the log-likelihood of the model given the data [Friedman, 1997; Neapolitan, 2004]; the higher the log likelihood, the more accurate a particular network is at predicting the data. This is subject to a constraint on the complexity of the model, as the structure with the highest likelihood will be the one with connections between every node. In this paper the network structure is learnt as part of the experiment by performing a search through the set of possible structures and choosing the one which maximises the log-likelihood. This search is performed online by training and evaluating simultaneously a set of candidate models. The search space is the set of models with different delays in motor commands and different observation nodes for each possible object.

The second issue is how particular motor commands and their values should be chosen so as to learn the most accurate forward model. Unlike most machine learning situations, with a forward model the robot is an active part of the Bayesian network structure it is attempting to learn. This situation, referred to as active learning, has received relatively little attention in Bayesian networks literature. Active learning for Bayesian networks has been discussed for unknown parameters and unknown structure [Tong and Koller, 2000; 2001]. The method of choosing motor commands in the experiment in this paper was inspired by the idea of babbling in infant motor skill learning [Meltzoff and Moore, 1997]; the motor commands were chosen from a random walk through a Markov model.

4 Experiments to learn a forward model

An experiment was carried out to show how forward models can be autonomously learnt and used. The robot used was an ActivMedia PeopleBot, as shown in �gure 5. The task was to learn the forward model for the movement of the robot's grippers; the robot needed to learn to predict the effects of the motor commands. No prior information about the appearance of the grippers or the nature of the motor commands was speci�ed. The robot was to perform an experiment: it babbled with its motor commands, observed what happened, and then learnt the relationship between the motor command and the observation using a Bayesian network; this process was repeated until an accurate model had been learnt.

The motor command in this case was limited to just one degree of freedom: the robot's grippers, which could be either opened, closed or halted. The motor commands were decided at random using a random walk through the Markov model shown in �gure 6. Motor commands were sent according to the current state of the Markov model, and the state would change each time-step according to the transition probabilities shown on the �gure. The parameters of the Markov model were chosen so that the motor system stays in one babbling state just long enough to get enough information to train its forward model before moving to the next state.

When the motor commands were sent, the grippers moved; the vision system correctly calculated and tracked the positions of the moving grippers in the scene: �gure 7 shows the two grippers being located and tracked. The tracking system worked well in this situation: black objects are automatically being tracked even against the black background of the robot's base.

The vision system provided the random variables for the states S[t] and observations O[t] in the Bayesian network for the forward model. In this situation the two objects, the grippers, were automatically added to the Bayesian network. Each gripper was represented as a discrete node for the state, and a set of continuous nodes for the observation, both of which were represented as a Gaussian distribution. The set of possible observations, O[t], was the velocity, VEL[t], the position, POS[t], or size, SIZE[t] of the tracked objects. The basic template for the structure of the forward models to be learnt is shown by the Bayesian network in �gure 8.

Even though the forward model of the robot's gripper was relatively simple, this example highlighted how much the robot needed to learn: all the parameters and two important properties of the structure needed to be learnt. Firstly, it is not known how long the delay is between the motor command being issued and the gripper changing state. This is shown in �gure 8 as the multiple possible motor commands connected to each state; only the four previous commands are shown to keep the diagram simple. Secondly, it was not known which observation from the vision system is the best one to use as part of the forward model: the velocity, position or size. The task of the structure learning algorithm here is therefore to maximise the log-likelihood of the data given the model, which in this case is log (P (M, O | G)), where M is the set of motor commands over the experiment, O the set of corresponding observations and G is the directed acyclic graph representing the structure of the model, together with the parameters of the model. This value is calculated as part of the parameter learning process, as a particular model is trained to the data.

For each model, the parameters of the Bayesian network were learnt using the expectation maximisation (EM) algorithm, with the inference stage being performed with the junction-tree algorithm [Pearl, 1988]. In this experiment, the search space is small enough for the robot to consider each possible candidate structure simultaneously. If the previous 20 motor commands are considered and 6 possible combinations of observation variables (VEL[t], POS[t] and SIZE[t], for each gripper), the search space contains 120 possible models. Figure 9 shows how the log-likelihoods for the Bayesian networks with varying motor delays and the velocity observation change over time. As the �gure shows, the motor delay which maximise the log-likelihood, d, was 11 time-steps, which is about 550 ms. Following the search, the learnt forward model for the grippers is shown in �gure 10. The motor command was learnt to be M[t-11], and the best observation was learnt to be the velocity of the grippers. This was correct since the motor commands do control the grippers' velocities.

5 Using the Bayesian network as a forwardmodel

Once the Bayesian network in �gure 10 has been learnt, it can be used as a forward model to give a prediction of the consequences of the motor command. For the forward models of the grippers, the predicted output is the velocity of the grippers, calculated from the conditional probability distribution: P(VEL[t]|M[t-4]=m). Because the observations here are modelled as Gaussians, the most likely values will be the means of the observations, VEL1[t] and VEL2[t]. The inference algorithm used to calculate this marginal distribution was the junction-tree algorithm [Pearl, 1988]. To evaluate the quality of the learnt forward model the grippers were babbled in a similar way as before, using a different Markov model. Figure 11 shows how the most likely predicted observation compares with the actual observation. Only the x-coordinate predictions are plotted because the grippers' movements are predominantly in this plane. The prediction is shown to be very accurate.

The learnt Bayesian network can also be used as an inverse model by calculating the most likely motor command given the current velocity observation for each time-step, as shown in �gure 12. One use of this inverse model is for imitation: by replacing the robot's observations of its own movement with its observations of a human's hand movements in the inverse model, the robot is able to reproduce the motor commands that would be most likely to recreate that human movement in its own gripper system. The same principle is used by Demiris to switch from imitation to recognition, albeit in a multiple paired inverse and forward model system [Demiris, 2002]. Figure 13 shows the results of the imitation experiment: the robot is able to imitate a simple hand waving motion. The human's hand movements are tracked and recognised using the vision system, which is able to correctly track the two hands. The observations are used as evidence for the Bayesian network already learnt by the robot. This network is then used as an inverse model to predict the most likely motor commands that would produce this observation. These motor commands are sent to the robot's motor system enabling it to perform the action that best replicates the observed movement.

6 Conclusion

Forward models relate a robot's motor commands to their effect on its motor system and the robot's environment. The system presented here allows a robot to autonomously learn forward models of its motor system, using feedback from its vision system, by babbling with its motor commands. By representing the forward model with a Bayesian network, the uncertainty in its prediction can also be represented. An experiment was implemented showing how a robot can learn a forward model of its gripper system. The forward model was subsequently used to allow the robot to imitate a simple 'waving hand' gesture of a human. This imitation occurred without any prior information having to be supplied by a human: the robot learns how to control its motor system, then uses this information to imitate a human gesture. Future work will involve extending the system to learn forward models with more degrees of freedom and interaction with objects.

7 Acknowledgements

This work has been supported through a Doctoral Training Award from the UK's Engineering and Physical Sciences Research Council (EPSRC), and through a bursary from the BBC Research & Development Department. The authors wish to thank Gavin Simmons, Bassam Khadhouri, Matthew Johnson and Paschalis Veskos of the Biologically inspired Autonomous Robots Team (BioART) at Imperial College.