Non-Invasive Brain-Actuated Control of a Mobile Robot

1 Introduction

There is a growing interest in the use of physiological signals for communication and operation of devices for physically-disabled as well as able-bodied people. Over the last years evidence has accumulated to show the pos­ sibility to analyze brainwaves on-line in order to deter­ mine the subjects' mental state that is then mapped into actions such as selecting a letter from a virtual keyboard or moving a robotics device [Birbaumer et a!., 1999; Kennedy et al., 2000; Millan, 2002; Millan et al., 2002; Pfurtscheller and Neuper, 2001; Roberts and Penny, 2000; Serruya et al., 2002; Taylor et al., 2002; Wolpaw and McFarland, 1994; Wolpaw eta!., 2002]. This alterna­ tive communication and control channel, which does not require the user to perform any physical action, is called a brain-computer interface (BCI).

A BCI may monitor a variety of brainwave phenom­ ena. Most BCis use electroencephalogram (EEG) signals;

i.e., the brain electrical activity recorded from electrodes placed onto the scalp. The main source of the EEG is the synchronous activity of thousands of cortical neurons. Measuring the EEG is a simple noninvasive way to moni­ tor brain electrical activity, but it does not provide de­ tailed information on the activity of single neurons (or small clusters of neurons) that could be recorded from microelectrodes surgically implanted in the cortex.

Some groups exploit evoked potentials-the automatic responses of the brain to external stimuli-recorded from either scalp or intracranial electrodes (for a review, see [Wolpaw et al., 2002]). Evoked potentials are, in princi­ ple, easy to pick up but constrain the subject to synchro­ nize themselves to the external machinery. A more natu­ ral and suitable alternative for controlling devices is to analyze components associated with spontaneous mental activity. Thus, some researchers measure slow cortical potentials-whose negative amplitudes are related to the overall preparatory excitation level of a given cortical network-over the top of the scalp [Birbaumer et al., 1999]. Other groups look at local variations of EEG rhythms. The most used of such rhythms are related to the imagination of body movements and are recorded from the central region of the scalp overlying the sen­ sorimotor cortex [Pfurtscheller and N euper, 2001; W ol­ paw and McFarland, 1994]. But, in addition to motor­ related rhythms, other cognitive mental tasks are being explored [Millan et al., 2002; Roberts and Penny, 2000] as a number of neurocognitive studies have found that different mental tasks-such as imagination of move­ ments, arithmetic operations, or language-activate local cortical areas at different extents. In this case, rather than looking for predefined EEG phenomena as when using slow cortical potentials or movement rhythms, the ap­ proach aims at discovering mental-specific EEG patterns embedded in the continuous EEG signals. Yet, another kind of spontaneous signals is the direct activity of neu­ rons in the motor cortex measured with implanted elec­ trodes [Kennedy et al., 2000; Serruya et al., 2002; Taylor et al., 2002; Wessberg et al., 2000].

Recent experiments have shown the near possibility to use the brain electrical activity to directly control the movement of robotics or prosthetic devices. In these ex­ periments, several monkeys have been implanted with microelectrodes recording the activity of single neurons (their spiking rate) in the motor and premotor areas of the cortex. Then, the monkey's hand trajectory was predicted (and replicated online with a robot arm) from the activity of the neural populations [Wessberg et al., 2000] . Also, after appropriate training, monkeys were able to move a computer cursor to desired targets using only their brain activity [Serruya eta!. , 2002; Taylor et al., 2002]. Until now, brain-actuated control of robots has being only tried with this kind of invasive approaches-requiring surgical implantation of electrodes-since EEG-based systems have been considered too slow for controlling rapid and complex sequences of movements. In this paper we show that two human subjects could, within a few days, learn to master a portable EEG-based brain-computer interface that recogni zed three mental states. Subjects successfully moved a robot between several rooms by mental control only. Furthermore, mental control was only marginally worse than manual control on the same task.

2 Brain Interface Protocol

EEG-based brain-computer interfaces are limited by a low channel capacity. Most of the current systems have a channel capacity below 0.5 bits/second [Wolpaw et al., 2002] . One of the main reasons for such a low bandwidth is that they are based on synchronous protocols where EEG is time-locked to externally paced cues repeated every 4-10 s and the response of the BCI is the average decision over this period [Birbaumer et al., 1999; Pfurtscheller and Neuper, 2001 ; Wolpaw and McFarland, 1994]. In contrast, our approach uses an asynchronous protocol that analyzes the ongoing EEG to determine the subject's mental state, which they can voluntarily change at any moment. The rapid responses of the BCI, together with its performance (see Section 3), give a theoretical channel capacity in between 1 and 1.5 bits/second.

Two volunteer healthy subjects "A" and "B" wore a commercial EEG cap with integrated electrodes (white spots in Figure 1). EEG potentials were recorded at the 8 standard fronto-centro-parietal locations F3, F4, C3, Cz, C4, P3 , Pz, and P4. The sampling rate was 128 Hz. The raw EEG potentials were first transformed by means of a surface Laplacian (SL) computed globally by means of a spherical spline of order 2 [Perrin et a!., 1989, 1990]. This spatial filtering yields new potentials that should represent better the cortical activity due only to local sources below the electrodes . Then, we used the Welch periodogram algorithm to estimate the power spectrum of each channel over the last second. We averaged 3 seg­ ments of 0.5 second with 50% overlap, what yields a fre­ quency resolution of 2 Hz. The values in the frequency band 8-30 Hz were normalized according to the total en­ ergy in that band. Thus an EEG sample has 96 features (8 channels times 12 components each). EEG samples were computed every 62.5 ms (i.e., 16 times per second).

During an initial training period of a few days, the two subjects learned to control 3 mental tasks of their choice. The subjects tried the following mental tasks : "relax", imagination of "left" and "right" hand (or arm) move­ ments, "cube rotation", "subtraction", and "word associa­ tion". The tasks consisted of getting relaxed, imagining repetitive self-paced movements of the limb, visualizing a spinning cube, performing successive elementary sub­ tractions by a fixed number (e.g., 64- 3=61, 61- 3=58, etc.), and concatenating related words 1 · After a short evaluation, the experimental subjects "A" and "B" chose to work with the combination of 3 tasks relax-left-cube and relax-left-right, respectively. In the sequel, we will refer to these mental tasks as #1, #2 and #3 (i .e., relax is #1, left is #2, and cube or right is #3). Neither subject had previous experience with BCis or mental training.

Each day, subjects participated in four consecutive training sessions of about 5 min, separated by breaks of 5-l 0 min. During each training session subjects switched randomly every 10-15 s between the three tasks. Subjects received feedback online through three colored buttons on a computer screen. Each button is associated to one of the mental tasks to be recognized. A button flashed when an EEG sample is classified as belonging to the corre­ sponding mental task. After each training session the sta­ tistical classifier was optimized offline. After this initial training, subjects learned to control mentally the mobile robot for 2 days. The results reported here were obtained at the end of the second day of work with the robot. Dur­ ing this training period, the user and the BCI engaged in a mutual learning process where they were coupled and adapted to each other.

3 Statistical Classifier

The mental tasks (or classes) are recognized by a Gaus­ sian classifier trained to classify EEG samples as state # 1, #2, #3 or "unknown". In this statistical classifier, every unit represents a prototype of one of the classes to be recognized. Its output gives an estimation of the poste­ rior class probability distribution for an EEG sample. The challenge is to find the appropriate position, and recep­ tive field, of the prototypes in the high-dimensional input space described above to differentiate the desired classes.

Although Gaussian classifiers are well known, our im­ plementation differs from classical ones in a few re­ spects. We assume that the class-conditional density function of class Ck is a superposition of Nk Gaussians (or prototypes) and that classes have equal prior probabili­ ties. In our case, all the classes have the same number of prototypes, namely 4. In addition, we assume that all four prototypes have an equal weight of 1/4. Then, dropping constant terms, the posterior probability Yk of class Ck for sample xis

1 Relax is done with eyes closed, whereas the other tasks are performed with eyes opened. But the recognition of the task "re­ lax" is not based on the detection of eye movements.

where Nc is the number of classes and ai is the activa­ tion level of the ith prototype of the class ck

where ;..t~ corresponds to the center of the ith prototype of class Ck, L:k is the covariance matrix of class Ck, and [Ikl is the determinant of that matrix. In our case, L:k is diagonal and common to all the prototypes of the class. In this way, we reduce the number of parameters and pool data to increase the accuracy of their estimation.

The response of the network for sample x is the class ck with the highest posterior probability provided that is greater than a given probability threshold of 0.85; other­ wise the response is "unknown." This rejection criterion keeps the number of errors (false positives) low, because recovering from erroneous actions (e.g., robot turning in the wrong direction) has a high cost. The choice of this probability threshold was guided by a previous ROC study where different subjects only carried out the initial training described before [Hauser et a!., 2002], and the actual value was selected based on the performance of the two subjects during the initial period of training.

To initialize the center of the prototypes and the co­ variance matrix of the class Ck we run a clustering algo­ rithm (typically, a self-organizing map [Kohonen, 1997]) to compute the position of the desired number of proto­ types. Then, the covariance matrix is

where sk denotes the number of training samples belonging to the class Ck and f..ik is the nearest prototype of this class to the sample xn.

We then improve these initial estimations iteratively by stochastic gradient descent so as to minimize the mean square error. For every sample x in the training set, the update rule for all the prototypes of all the classes is

where a is the learning rate, tk is the kth component of the target vector in the form 1-ofc and A is the total ac­ tivity of the network-i.e., the denominator in (I). Intui­ tively, during training, units are pulled towards the EEG samples of the mental task they represent and are pushed away from EEG samples of other tasks.

Finally, after every iteration over the training set, we estimate again the new value of L:k using expression (3).

It is possible to estimate the covariance matrices in more elaborated ways, including through gradient descent in order to minimize their contribution to the error function.

The brain-computer interface responds every 0.5 s. Firstly, it computes the class-conditioned probability for each class- i.e., the mixture of Gaussians in the numera­ tor of Eq. (1). Secondly, it averages the class-conditioned probabilities over 8 consecutive samples. Thirdly, it es­ timates the posterior probability based on the average class-conditioned probability of each class using Bayes' formula ; cf Eq. (I). Finally, it compares the posterior probability with a threshold value of 0.85. At the end of training, errors and "unknown" responses are below 5% and 30%, respectively. The theoretical channel capacity of the interface is hence above I bit/second (operation mode I) . In addition, the interface could also operate in another mode (operation mode II) where classification errors are further reduced by requiring that two consecu­ tive periods of 0.5 s give the same classification re­ sponse. In this mode II errors and "unknown" responses are below 2% and 40%, respectively, and the theoretical channel capacity is about I bi t/second.

4 Robot Setup and Control

The task was to drive the robot through different rooms in a house-like environment (Figure 1). The robot was a small Khepera (5 .7 em diameter) that closely mimics a motorized wheelchair. The robot moved at a constant speed of one third of its diameter per second, simi lar to the speed of a wheelchair in an office building.

To make the robot move along a desired trajectory it is necessary to determine the speed of the motors control­ ling the wheels at each time step. Obviously, this is im­ possible by means of just three mental commands. A key idea is that the user's mental states are associated to high-level commands (e.g., "turn right at the next occa­ sion") that the robot executes autonomously using the readings of its on-board sensors. Another critical aspect for the continuous control of the robot is that subjects can issue high-level commands at any moment. This is possi­ ble because the operation of the BCI is asynchronous and does not require waiting for external cues, unlike syn­ chronous approaches. The robot will continue executing a high-level command until the next is received.

The robot relies on a behavior-based controller [Arkin, 1998] to implement the high-level commands that guar­ antees obstacle avoidance and smooth turns. In this kind of controller, on-board sensors are read constantly and determine the next action to take. The mapping from the user's mental states (or commands) to the robot's behav­ iors is not simply one-to-one, but, in order to achieve a more flexible control of the robot, the mental states are just one of the inputs for a finite state automaton with 6 states (or behaviors). The transitions between behaviors are determined by the 3 mental states (#I, #2, #3), 6 per­ ceptual states of the environment (as described by the robot's sensory readings: left wall, right wall, wall or obstacle in front, left obstacle, right obstacle, and free space) and a few internal memory variables. Figure 2 shows a simplified version of the finite state automaton. The memory variables were required to implement cor­ rectly the different behaviors. Thus, if the robot is per­ forming the behavior "forward" and perceives a wall to the left, it switches automatically to the behavior "follow left wall". The actual transitions between the behaviors "forward" and "follow left/right wall" are not exactly as indicated in the figure, otherwise the robot would stay following walls forever. The transition to the behavior "forward" is necessary, for example, in the case the robot is approaching an open door and the user wants the robot not to enter into the room. On the other hand, the robot "stops" whenever it perceived an obstacle in front to avoid collisions (not all the transitions to the behavior "stop" appear in the figure for the sake of simplicity). Briefly, the interpretation of a mental state depends on the perceptual state of the robot. Thus, in an open space the mental state #2 means "left turn" while the same mental state is interpreted as "follow left wall" if a wall is detected on the left-hand side. Similarly, mental state #3 means "right turn" or "follow right wall"; mental state #I always implied "move forward". Altogether experi­ mental subjects felt that our control schema was simple and intuitive to use.

The Khepera robot is a two-wheeled vehicle. It has 8 infrared sensors around its diameter to detect obstacles. The sensors have a limited perception range, what makes difficult the recognition of the different perceptual states from the raw readings. To overcome this limitation, the robot uses a multilayer perceptron that maps the 8 raw infrared sensory readings into the current perceptual state.

A final element is the use of an appropriate feedback indicating the current mental state recognized by the em­ bedded classifier. This is done by means of three lights on top of the robot, with the same colors as the buttons used during the training phase. The front light is green and is on when the robot receives the mental command # 1. The left light is blue and is associated to the mental command #2, whereas the right light is red and is associ­ ated to the mental command #3. Thus, if the robot is fol­ lowing the left wall and is approaching an open door, a blue feedback indicates that the robot will turn left to continue following the left wall (and, so, it will enter into the room). On the contrary, a green feedback indicates that robot will move forward along the corridor when facing the doorway and will not enter into the room. This simple feedback allows users to correct rapidly the ro­ bot's trajectory in case of errors in the recognition of the mental states or errors in the execution of the desired behavior (due to the limitations of the roboes sensors).

5 Experimental Results

After 5 and 3 days of initial training with the interface operating in mode I, subjects "A" and "B", respectively, achieved a satisfactory level of performance (correct rec­ ognition was above 60% while errors were below 5%). At this moment, subjects started to learn to control mentally the robot with the interface operating in mode II. During this second period of training subjects had to drive the robot mentally from a starting position to a first target room; once the robot arrived, a second target room was selected and so on. The starting position and the target rooms were drawn at random.

Figure 3 shows a trajectory generated by subject "A" after two days of training. The robot had to visit 3 differ­ ent rooms, drawn randomly, starting from location "S".

Although the figure does not show the details of the tra­ jectory inside the rooms, the robot made a short explora­ tion in each of them. During the experiment, the subject was driving the robot for about 10 minutes continuously. Although the subject brought the robot to the desired room each time, there were a few occasions where the robot did not follow the optimal trajectory. This was mainly because the brain interface took a longer time than usual to recognize the subject's mental state. For instance, in one case the robot missed a turn because the brain interface did not recognize the appropriate mental state until the robot had passed the doorway of the de­ sired room, and so the subject needed to maneuver men­ tally the robot to bring it back. In other situations, the robot's sensors perceived a wall or corner too close, thus making the robot stop automatically to avoid collisions. In these situations, the subject needed to turn (by mental control) the robot away from the phantom obstacle and then resume the trajectory.

Qualitatively, the trajectory is rather good as the robot visited the 4 rooms in the desired order and it was never necessary to make significant corrections to the robot's active behaviors. But in order to evaluate quantitatively the performance of the brain-actuated robot, subjects "A" and "B" also carried out a second set of experiments in a slightly different arrangement of the rooms that were now located along the two sides of a corridor (Figure 4).

In a given trial, the robot must travel from a starting room to a target room as well as also visiting an interme­ diate room. The rooms and their order were selected at random. First, the subject made the robot visit the desired sequence of rooms by mental control. In a later session, the subject drove the robot along the same sequence of rooms by manual control. In this case, the subject used the same controller described above but, instead of send­ ing mental commands to the robot, he simply pressed one of three keys. This procedure allowed us to compare mental and manual control for a system that is identical in all other aspects. In addition, the manual trajectory should be quite close to the optimal path that can be gen­ erated with the current controller. It is worth noting that the reason why the subject controls the robot mentally first and only afterwards manually is to avoid any learn­ ing process that could facilitate mental control.

Table 1 gives the time in seconds necessary to generate the desired trajectory for three different trials for the two subjects. For each trial, the table indicates the time re­ quired for mental control and manual control. Surpris­ ingly, we can see that mental control was only marginally worse than manual control. On average, brain-actuated control of the robot is only 35% longer than manual con­ trol for both subjects.

6 Discussion

In this paper we have reported first results of a brain­ actuated mobile robot by means of a portable non­ invasive BCI. Although the quality and resolution of the brain signals measured with our EEG system are not comparable to those recorded by implanted electrodes, they are sufficient to operate robots in indoor environ­ ments. This is possible because of the combination of advanced robotics, an asynchronous protocol for the analysis of online EEG signal, and machine learning techniques.

The work described in this paper suggests that it could be possible for human subjects to mentally operate a wheelchair. But porting the current results to the wheel­ chair is not straightforward for, at least, two reasons. First, the performance of the BCI will suffer once the subject is seated on a mobile platform. This will require longer training times for the subject. Second, the current finite state automaton only allows for simple control ac­ tions, and so the resulting wheelchair could be too con­ strained for practical use in cluttered environments. In this respect, recent progress in EEG analysis [Michel et a!., 200 I] suggests that a sufficient number of mental states can be recognized to control robotics and pros­ thetic devices and in a more natural and flexible way. In this approach we will transform scalp poten­ tials-recorded with a sufficiently high number of elec­ trodes (32, 64 or more)-to brain maps to get detailed information on the activity of small cortical areas. The Gaussian classifier embedded in the BCI would work upon selected parts of these brain maps instead of using EEG features .

Acknowledgments

JRM is supported by the Swiss National Science Founda­ tion through the National Centre of Competence in Re­ search on "Interactive Multimodal Information Manage­ ment (1M2)."