Conditional Visuomotor Learning and Viability Theory

Introduction

Studies of arbitrary visuomotor learning in laboratory situations generally use several stimuli taken from the same category (colors, pictures, positions etc.) and an equivalent number of motor responses (hand postures, lever displacements, etc.). Subjects are required to learn by trial and error, conditional rules such as 'if green go right, if red go left, if blue, go up', etc. This requires on-line monitoring of the associations already tried and of their outcome (correct or incorrect). With large sets of items and actions to associate, this monitoring represents a high memory load, which can be decreased by using specific learning strategies.

Conditional associative learning: behavioral and neuronal data

The striatum, the larger structure composing the basal ganglia, constitutes a key neural substrate for arbitrary visuo-motor learning (see e.g. Hadj-Bouziane et al., 2003 for review). Hence, in a recent study, we recorded single cell activity in the striatum of two rhesus monkeys during learning of a conditional visuomotor task that required to associate complex visual cues with joystick movements (Hadj-Bouziane & Boussaoud, 2003). The animals searched, by trial and error, for the correct joystick movement associated with each of four visual cues. The four associations comprising a set were presented concurrently, that is, each appeared once, in randomized order, within each block of four consecutive trials.

Behavior. Although the four associations were presented concurrently within each block of 4 trials, monkeys learned them sequentially, focusing on one association at a time in order to decrease the memory load. The use of strategies to organize learning was also observed in humans in a separate study (Hadj-Bouziane et al., 2004).

Neuronal activity: Neuronal activity in the striatum showed two main types of modulations during learning. One was transient, consisting in a time-limited increase in activity during early learning stages. The other was long-lasting, consisting in a gradual increment of activity over the entire course of the learning session. The former could underlie the development and use of learning strategies, whereas the latter could be responsible for long term storage of the individual associations.

Mathematical model: the striatum and viability theory

Several not mutually exclusive computational models of basal ganglia have been proposed in the literature (e.g. Houk et al., 1995; Barto, 1995). Many of these models use the reinforcement learning (RL) algorithm. For learning associations by RL algorithms one would need to build a cost function adapted to this particular learning task and then to find a way to maximize this function. However, as described above, learning arbitrary visuomotor associations is not based on random selection of a behavioral response, but rather implies strategies. Mathematically the learning of associations leads to the so called target problem of control theory and there are different tools (cf. viability theory and capture basin algorithms, time optimal control, adaptive control) that allow to find policies theoretically and construct solutions numerically. The learning of policies leading to a target can be derived from viability theory (Aubin 1991). Instead of a repetitive improvement of the cumulative reward to make it as close as possible to a maximum (as it is done in RL algorithms), the viability kernel algorithms are based on the systematic exploration of boundaries of sets. Indeed when we investigate the boundary of a set, instead of the set itself, we reduce the dimension of the problem (Cardaliaguet et al., 1999). We propose that the use of strategies such as the sequential strategy evoked above correspond to investigation of boundaries. From the mathematical view point the viability and capture basin algorithms are based on elimination: taking out points from which it is impossible to get to a target and in the same time constructing the winning strategies for Capture Basin (e.g. Frankowska & Quincampoix, 1991). This elimination is based on a local analysis of constraints. Instead of applying exploration type arguments and trying different controls from the same initial condition, the viability kernel algorithms consider only critical points, that is the boundary of the set of constraints and check which points of the boundary are not viable.

Summary and conclusions

Learning arbitrary visuo-motor associations implies to learn the correct stimulus-movement associations and then to store them in long term memory. One has to learn by trial and error to select the appropriate movement according to the stimulus presented. During early stages of learning, processes such as discrimination, recognition and memorization of the stimuli are required, but also monitoring of events and behavioral outcome. Learning time can be reduced by the use of particular strategies such as focusing on a single cue at a time or repeating a motor response until correct.

We propose a computational model based on 1) these behavioral data showing the use learning strategies, 2) the possible correlates found in the dynamics of neuronal changes in the striatum, and 3) viability theory. This model complements existing ones, by taking into account the strategies, viewed as a way to determine the boundaries of the learning space. In our model the learning strategies artificially reduce the dimension of learning space (number of actions or number of images), a process that simplifies the task. At the neuronal level, we suggest that the strategy choice may be carried out by the transient activities, whereas the boundary encodings may be accomplished using the rules proposed in Bar-Gad et al. (2003), and local analysis done using the reinforcement signal. We thus propose an additional role of the basal ganglia, that of reducing the dimension of learning space.

Acknowledgements

This work was supported by the French Ministry of Research (ACI Cognition et traitement de l'information), and by the CNRS (PI TCAN), and a Bettencourt-Schuller foundation fellowship for FHB.