Learning to Walk through Imitation

1 Introduction

Imitation is an important learning mechanism in many biological systems including humans [Rao and Meltzoff, 2003]. It is easy to recover kinematic information from human motion using, for example, motion capture, but imitating the motion with stable robot dynamics is a challenging research problem. Traditional model-based approaches based on zero-moment point (ZMP) [Vukobratovic and Borovac, 2004], [Kajita and Tani, 1996] or the inverted pendulum model [Yamaguchi et al., 1996] require a highly accurate model of robot dynamics and the environment in order to achieve a stable walking gait. Learning approaches such as reinforcement learning [Sutton and Barto, 1998] are more flexible and can adapt to environmental change but such methods are typically not directly applicable to humanoid robots due to the curse of dimensionality problem engendered by the high dimensionality of the full-body joint space of the robot.

In this paper, we propose a model-free approach to achieving stable gait acquisition in humanoid robots via imitation. The framework for our method is shown in Figure 1. First, a motion capture system transforms Cartesian position of markers attached to the human body to joint angles based on kinematic relationships between the human and robot bodies. Then, we employ dimensionality reduction to represent posture information in a compact low-dimensional subspace. Optimization of whole-body robot dynamics to match human motion is performed in the low dimensional space. In particular, sensory feedback data are recorded from the robot during motion and a causal relationship between actions in the low dimensional posture space and the expected sensory feedback is learned. This learned sensorymotor mapping allows humanoid motion dynamics to be optimized. An inverse mapping from the reduced space back to the original joint space is then used to generate optimized motion on the robot. We present results demonstrating that the proposed approach allows a humanoid robot to learn to walk based solely on human motion capture without the need for a detailed physical model of the robot.

2 Human Motion Capture and Kinematic Mapping

In this paper, we manually map the joint angle data from a motion capture system to a kinematic model for a Fujitsu HOAP-2 humanoid robot. To generate the desired motion sequence for the robot, we capture example motions from a human subject and map these to the joint settings of the robot. Initially, a set of markers is attached to the human subject and the 3-D positions of these markers are recorded for each pose during motion. We use a Vicon optical system running at 120Hz and a set of 41 reflective markers. These recorded marker positions provide a set of Cartesian points in the 3D capture volume for each pose. To obtain the final subject poses, the marker positions are then assigned as positional constraints on a character skeleton to derive the joint angles using standard inverse kinematics (IK) routines.

As depicted in Figure 2, in order to generate robot joint angles, we simply replace the human subject's skeleton with a robot skeleton of the same dimensions. For example, the shoulders were replaced with three distinct 1-dimensional rotating joints rather than one 3-dimensional ball joint. The IK routine then directly generates the desired joint angles on the robot for each pose. There are limitations to such a technique (e.g, there may be motions where the robot's joints cannot approximate the human pose in a reasonable way), but since we are interested only in classes of human motion that the robot can handle, this method proved to be a very efficient way to generate large sets of human motion data for robotic imitation.

3 Sensory-Motor Representations

3.1 Low-Dimensional Representation of Postures

Particular classes of motion such as walking, kicking, or reaching for an object are intrinsically low-dimensional. We apply the well known method of principal components analysis (PCA) to parameterize the low-dimensional motion subspaceX . Although nonlinear dimensionality reduction methods could be used (e.g., [MacDorman et al., 2004], [Grochow et al., 2004]), we found the standard linear PCA method to be sufficient for the classes of motion studied in this paper.

The result of linear PCA can be thought of as two linear operatorsC and -1 C which map from high to low and low to high dimensional spaces respectively. The low dimensional representation of the joint angle space of the HOAP-2 robot executing a walking gait (in the absence of gravity) is shown in Figure 3.

We use a straightforward, standard linear PCA method to map between the low and high-dimensional posture spaces. Vectors in the high-dimensional space are mapped to the low-dimensional space by multiplication with the transformation matrixC . The rows of C consist of the eigenvectors, computed via singular value decomposition (SVD), of the motion covariance matrix. SVD produces transformed vectors whose components are uncorrelated and ordered according to the magnitude of their variance.

For example, let 21 1 = × q vector of joint angles (the high-dimensional space) and 3 1 = × p vector in 3D space. We can calculate p in 3D space by usingr = Cq , where r is a 21 1 × vector of all principal component coefficients of q and C is the 21 21 × transformation matrix. We then pick the first three elements of r (corresponding to the first three principal components) to be p . The inverse mapping can be computed likewise using the pseudo inverse of C .

3.2 Action Subspace Embedding

High-level control of the humanoid robot reduces to selecting a desired angle for each joint servo motor. As discussed previously, operations in the full space of all robot joint angles tend to be intractable. We leverage the redundancy of the full posture space and use the reduced dimensional subspaceX to constrain target postures. Any desired posture (also referred to as an action) can be represented by a point ∈ a X .

A periodic movement such as walking is represented by a loop inXas shown in Figure 4. In the general case, we consider a non-linear manifold representing the action space ⊆ A X . Non-linear parameterization of the action space allows further reduction in dimensionality. We embed a one dimensional representation of the original motion in the three dimensional posture space and use it for constructing a constrained search space for optimization as discussed in Section 5. Using the feature representation of the set of initial training examples i i x = C z , we first convert each point to its representation in a cylindrical coordinate frame. This is done by establishing a coordinate frame with three basis directions θ θ θ x , y , z in the feature space. The zero point of the coordinate frame is the empirical mean μ of the data points in the reduced space. We recenter the data around this new zero point and denote the resulting data ˆ i x .

We then compute the principal axis of rotation θ z :

Next, θ x is chosen to align with the maximal variance of i x in a plane orthogonal to θ z . Finally, θ y is specified as orthogonal to θ x and θ z . The final embedded training data is obtained by cylindrical conversion to φ ( ,r,h)where r is the radial distance, h the height above the - θ θ x y plane, and φ the angle in the - θ θ x y plane. The angle φ can also be interpreted as the phase angle of the motion.

Given the loop topology of the latent training points, one can parameterize r and h as a function of φ . The embedded action space is represented by a learned approximation of the function:

where 0 φ 2π ≤ ≤ . Approximation of this function is performed by using a radial basis function (RBF) network.

4 Learning to Predict Sensory Consequences of Actions

A central component of our proposed framework is learning to predict future sensory inputs based on actions and using this learned predictive model for action selection. The goal is to predict the future sensory state of the robot, denoted by t+1 s . In general, the state space Ρ S = × Z is the Cartesian product of the high-dimensional joint space Z and the space of other perceptsΡ . Other percepts could include, for example, a torso gyroscope, an accelerometer, and foot pressure sensors as well as information from camera images. The goal then is to learn a function : × F S A S arrowbarright that maps the current state and action to the next state. For this paper, we assume that F is deterministic.

Often the perceptual state t s is not sufficient for predicting future states. In such cases, one may learn a higher order mapping based on a history of perceptual states and actions, as given by an -th n order Markovian function:

We use a radial basis function (RBF) approximator to learn F from sensory-motor experience. In particular, the RBF network approximates F by learning a function : α β F' arrowbarright :

where K represents the number of kernels, μk and 1 kΣ are the mean and inverse covariance of the -th k kernel respectively. The output weight vector k w scales the output of each kernel appropriately, and the input and output are [ ] α t t -1 t-n-1 t t -1 t -n-1 = s , s ,...,s ,a ,a ,...,a andβ = t+1 s respectively. For convenience, one can instead view the RBF as a time delay network [Lang et al., 1990] for which the input simplifies to [ ] α = t t s ,a . The previous state and action inputs are implicitly remembered by the network using recurrent feedback connections.

In this paper, we use a second-order (n = 2) RBF network with the state vector equal to the three-dimensional gyroscope signal ( ) t t ≡ s ω . As discussed in the previous section, an action represents the phase angle, radius, and height of the data in latent posture space ( ) t χ ≡ ∈ a X .

5 Motion Optimization using the Learned Predictive Model

The algorithm we present in this section utilizes optimization and sensory prediction to select optimal actions and control the humanoid robot in a closed-loop feedback scenario. Figure 5 illustrates the optimization process.

One may express the desired sensory states that the robot should attain through an objective function ( ) Γ s . Our algorithm then selects actions * * ,..., t T a a such that the predicted future states ,..., t T s s will be optimal with respect to ( ) Γ s :

The objective function used in this paper is a measure of torso stability as defined by the following function of gyroscope signals:

where x y z ω ,ω ,ω refer to gyroscope signals in the x, y, z axes respectively. The constants x y z λ , λ ,λ allow one to weight rotation in each axis differently. The objective function (6) provides a measure of stability of the posture during motion. For our second-order predictive function F, the optimization problem becomes one of searching for optimal stable actions given by:

To allow for efficient optimization, we restrict the search space to a local region in the action subspace as given by:

The phase motion command search range φs begins after the position of the phase motion command at the previous time step -1 φt . The radius search range s r begins from a point in the action subspace embedding A that is defined by (12) in both positive and negative directions from a r along r for the distance ε 0 r > . The search range s h is defined in the same manner as s r according to a h and εh . In the experiments, the parameters φ ε , εr and εh were chosen to ensure efficiency while at the same time allowing a reasonable range for searching for stable postures. An example of the search space for a walking motion is shown in Figure 6.

Selected actions will only truly be optimal if the sensory-motor predictor is accurate. We therefore periodically re-train the prediction model based on the new posture commands generated by the optimization algorithm and the sensory feedback obtained from executing these commands. After three iterations of sensory-motor prediction learning, an improved dynamically balanced walking gait is obtained. The trajectory of the optimized walking gait in the low dimensional subspace is shown in Figure 6.

We summarize below the entire optimization and action selection process:

1. Use PCA to represent in a reduced 3D space the initial walking gait data from human motion capture.
2. Employ the non-linear embedding algorithm for parameterization of the gait.
3. Start the learning process by projecting actions back to the original joint space and executing the corresponding sequence of servo motor commands in the Webots HOAP-2 robot simulator [Webots, 2004].
4. Use the sensory and motor inputs from the previous step to update the sensory-motor predictor as described in Section 4 where the state vector is given by the gyroscope signal of each axis and the action variables are φ,r and h in the low-dimensional subspace.
5. Use the learned model to estimate actions according to the model predictive controller framework described above (Figure 5).
6. Execute computed actions and record sensory (gyroscope) feedback.
7. Repeat steps 4 through 6 until a satisfactory gait is obtained.

6 Experimental Results

-3

Figure 7. Motion pattern scaling. The target motion pattern is scaled down until it can produce a stable motion to start the motion optimization process.

This section explains how the optimization methodology in the previous section is used in conjunction with the mocap data. From our study of the motion pattern in the reduced subspace, we found that we can scale up and down the motion pattern and get similar humanoid motion patterns except for changes in the magnitude of motion. When we scale down the pattern in the reduced subspace, it produces a smaller movement of the humanoid robot, resulting in smaller changes in dynamics during motion. Our strategy is to scale down the pattern until we find a dynamically stable motion and start learning at that point. We apply the motion optimization method in Section 5 to the scaled-down pattern until its dynamic performance reaches an optimal point; then we scale up the trajectory of the optimization result toward the target motion pattern. In our experiments, we found that a scaling down of 0.3 of the original motion pattern is typically stable enough to start the learning process. Our final optimization result obtained using this procedure is shown as a trajectory of red circles in Figure 7. It corresponds to about 80% of the full scale motion.

Our simulation and experimental results are shown in Figure 8. We performed the learning process in the simulator [Webots, 2004] and tested the resulting motion on the real robot. The walking gait on the real robot is not as stable as the results in the simulator because of differences in frictional forces between the simulator and the floor. We expect an improvement in performance when learning is performed directly on the real robot. We note that the learned motion is indeed dynamic and not quasi-static motion because there are only two postures in our walking gait that can be considered statically stable, namely, the two postures in the walking cycle when the two feet of the robot contact the ground. The remaining postures in the walking gait are not statically stable as the gait has a fairly fast walking speed.

7 Conclusion

Our results demonstrate that a humanoid robot can learn to walk by combining a learned sensory-motor model with imitation of a human gait. Our approach does away with the need for detailed, hard-to-obtain, and often fragile physicsbased models that previous methods have relied on for stable gait generation in humanoids. Our approach builds on several previous approaches to humanoid motion generation and imitation. Okada, Tatani and Nakamura [Okada et al., 2002] first applied non-linear principal components analysis (NLPCA) [Kirby and Miranda, 1996] to human and humanoid robot motion data. The idea of using imitation to train robots has been explored by a number of researchers [Hayes and Demiris, 1994], [Billard, 2001]. In [Ijspeert et al., 2001], a nonlinear dynamical system was carefully designed to produce imitative behaviors. The mimesis theory of [Inamura et al., 2003] is based on action acquisition and action symbol generation but does not address dynamics compensation for real-time biped locomotion. The motion segmentation framework in [Jenkins and Mataric, 2003] uses dimensionality reduction and segmentation of motion data in the reduced dimensional space but without dynamics compensation.

The framework described in this paper has several practical applications. One scenario we are currently investigating is a general navigation task involving our humanoid robot. A modular architecture could be used to switch between a set of learned modules, such as walking-straight, turningleft, turning-right and stepping-backward. Using visual information as feedback, we could control robot direction by switching behaviors (actions) of the robot. We have also found that translation in the x-y plane in the reduced 3D latent space can be used for changing direction. Because our method does not depend on a model of the world, it can be applied to the problems of learning to walk up and down stairs, and walking on constant slopes. To learn actions other than walking straight, we can modify our optimization function in Eq. (6) or introduce additional sensory variables such as velocity or foot-contact pressure. We are currently investigating these lines of research.

Clearly, the present framework cannot be applied directly to the problem of navigation on uneven terrain. To effectively navigate on uneven terrain, we may need a higher degree of compliance control in the leg and foot actuators. However, a hybrid active-passive actuator will likely produce even more complex dynamics than typical actuators used in current humanoid robots. Since our approach does not require a physics-based dynamics model for learning, it lends itself naturally to tackling this problem. We hope to investigate this important research direction in the near future.