Control of Robotic Systems for Safe Interaction with Human Operators

1 Motivation and Problem Statement

In recent years, the problem of how to manipulate industrial robots that interact with human operators has attracted a lot of attention in artificial intelligence. This interest comes from the insight that the integration of human operators into robot based systems may increase productivity by combining the abilities of machines with those of humans. In such a Human Robot Interaction (HRI) scenario, the challenge is to manipulate the robots both safely and efficiently.

This problem can be formally described as: Given an initial configuration, the motion of driving the end effector of a robotic manipulator into a specified goal set is to be determined for optimizing a given performance criterion. All relevant kinematic and dynamic constraints including particularly the avoidance of collision with obstacles, like humans, operating in the same space as the robot have to be satisfied.

Additionally, study of human behavior is indispensable when robots interact with humans. This requires a better understanding and modeling of the human motion. The prediction of human behavior with a certain probability can be used for improving the performance of the robot motion and of the human robot collaboration. The collisions can also be avoided within a model predictive control framework.

This paper describes the following research topics of mine. We propose an online optimization-based motion planning approach for robotic manipulators with collision avoidance using a predictive control scheme in Sec. 2.1. A method on efficient generation of safety relevant regions that are possibly occupied by humans is proposed in Sec. 2.2. We finally describe an HRI demonstration platform for realization of our proposed methods.

2 Achieved Solutions and Current Research

2.1 Motion planning for robotic manipulators

The planning problem for robotic manipulators with obstacle avoidance is complex due to its nonconvexity, system nonlinearity and coupling, dynamic and time varying constraints, etc. The task leads to global, nonconvex optimization problems, which may be formulated and solved in the workspace (Cartesian space), in the configuration/joint space, or in combinations thereof. Note that velocity limits of joints are normally prescribed regarding to the obstacle movement, which can be easily dealt with in the workspace. Thus, the planning strategy of the robotic manipulators that we are mainly considering is directly performed in the workspace. For appropriately describing the planning task, the geometric constraints, namely the kinematics, have to be considered. Selected points of the robot geometry (particles) are used for this purpose to formulate obstacle avoidance constraints. The overall global and nonconvex optimization problem is approximated by a Mixed Integer Program (MIP) [Ding et al., 2009]. With respect to the optimal criterion specifying the average kinetic energy, the optimized paths are shown in Fig. 1(a) and 1(b) with varying constraints in different regions (hybrid optimal control) and with a moving obstacle, respectively.

We have then presented a proved geometric result in R2 in [Ding et al., 2011a] whose application drastically reduces the number of binary decision variables in the MIPs, compared to previous results. In addition, the number of simplex iterations, which is a good measure for the computational effort, is reduced by a factor of roughly 200 over previous methods. The result is shown in Fig. 1(c). We have also demonstrated the application of our method to an industrial robot. The computation is further accelerated by using the moving horizon scheme. Specifically, for 4 particles per link and prediction horizon 7, the solution was determined within 0.13 seconds, on a single thread of an i5-CPU PC with 2.67 GHz clock rate. The approach has been further extended to R3 in [Ding et al., 2011b].

2.2 Humanmotion modeling and prediction

One challenge for human motion modeling and prediction is that human operators tend to follow different motion patterns, depending on several factors such as intentions and the structure of the environment. Instead of explicitly modeling such factors, the probabilistic framework of Hidden Markov Models (HMMs) is chosen [Vasquez et al., 2009]. This choice appears to be reasonable because uncertainties are intrinsically involved in predicting complex factors of human behavior such as perception of the environment and intention. In addition to a probabilistic description of specific motion patterns, HMMs also adequately describe stochastic transitions between different patterns.

Based on HMMs, a method has been proposed in [Ding et al., 2011c] to predict the safety relevant region that is possibly occupied by the human within a prediction horizon. In contrast to previous approaches which derive predictions based on probability density functions in the form of most likely or expected human positions, the proposed method computes safety relevant subsets of the workspace as a region with a probability not less than a predefined threshold. The safety relevant region, in which the human is possibly positioned at future time instances, can subsequently serve as safety constraints when the robot motion is planned by the optimizer.

The practicability of our method is demonstrated by successfully and accurately predicting the motion of a human arm in two scenarios involving multiple motion patterns. F(t +H) in Fig. 2(a) indicates the predicted safety relevant region of a human hand with a certain probability (so called 'forbidden regions'). In another HRI scenario, the HMM of a human hand is generated based on the experimental data, shown in Fig. 2(b). According to the model and the current observation (red marker), the prediction of the human hand and the intended goal (green circles) are computed, shown in 2(c). The results indicate that the selected probabilistic model is appropriate for human motion modeling and prediction. The approach is capable of predicting the future human behavior online based on a probability density function.

The approach has been extended for online adaptation of motion patterns and of safety relevant regions, as well as been enhanced by introducing reachability analysis [Ding et al., 2011d] with respect to motion dynamics and limits of humans in order to increase the prediction accuracy. The approach is additionally being applied to and examined for different scenarios.

2.3 An HRI demonstration platform

We have developed an HRI demonstration platform in Fig. 3, on which the proposed methods have been realized. It consists of: 1) a real time framework comprising the industrial robot and an external robot controller, which guarantees online adaptation of robot motion, 2) an open source motion planning software platform for prototyping and simulating planning algorithms, 3) a human (arm) motion detection system using a stereo camera system with a robust marker detection algorithm, and 4) data communication via Ethernet.