Thoughts About a Visually Guided Grasp Reflex

Introduction

The structure of animal nervous systems was first analyzed in terms of reflexes. A reflex is sometimes thought of as a response which involves only a few synaptic delays and which is independent of higher centers of the brain. Thus the doctor's hammer strik­ ing (and stretching) the tendon just below the kneecap results in a reflexive kick no matter what we may be thinking about at the time. However, more current notions hold that reflexes are to be thought of as part of a hierarchical control scheme, functioning to carry out incompletely specified commands from above. Thus, we find that the familiar knee jerk reflex does not occur when the knee's owner is in free fall.9 · ~le t~is particular reflex is adequate and appro­ prlate 1n some cases (especially locomotion), it is under the control of other regions of the brain. And so it goes: if we look at the nervous system hier­ archi~ally, we can discover layer upon layer of relat1vely autonomous subsystems controlling and controlled by others. ·

The visual orienting reflex has received a sub­ stantial amount of attention in a wide range of animals. Attempts to model it have led me -through a ~umber of concepts which I thought would be interest­ lng to apply to the robot problem. The main notion is that this system is a rapid, parallel device which typically needs to know remarkably little about an object to respond appropriately to it. Thus, it seems to me, a robot grasp reflex could be constructed which could appropriately grasp an object in the visual scene quickly and cheaply upon a very crude command (i.e. without being told the object's exact location, orientation or shape).

The next section of this paper provides a brief review of midbrain mechanisms which mediate the vis­ ual orienting response. The last section describes a r~ther brief and superficial effort at applying not1ons from the nervous system to the problem of using a serial computer to control a robot hand in a s~mple environmen~ .. Finally there are some sugges­ tlons for general1z1ng to a more realistic case.

Author's address as of September, 1973 :

R. L. Di dday Information and Computer Science Applied Sciences Bu~lding University of California Santa Cruz, California 95060

THOUGHTS ABO~T A VISUALLY GUiDED GRASP REFLEX

Richard L. Didday Department of ~1athemati cs and Computer Science Colorado State University

Fort Collins, Colorado

Animal visual orienting reflexes

The optic tectum (or suPerior colliculus as this midbrain structure is called in mammals) is common to vertebrates and is implicated in visual localizing responses. In the frog, it mediates visual approach responses (snapping and orienting), while in humans it produces reflex eye movements. Let us look at its structure in the frog.

The optic tectum stands between sensory and motor sy~tems. There is an orderly, somatotopic mapping of po1nts on the retina onto the tectum and from tectum to motor response. This is illustrated by microelec­ trode recordings and by electrical stimulation of the tectum--such experiments show that a particular region of the tectum can produce a motor response which is "aimed" at the same region of visual space which pro­ jects to that spot on the tectum.

fln early paper by Pitts and McC'ulloch 8 Provides a provocative framework for describing the operation of the frog tectum. Picture the retina and its pro­ cesses as sending a spatially arrayed somatotopically mappe? representation of the visual scene through the gangl1on cell axons to the tectum. Regions of this array wh~ch are highly excited correspond to regions of the v1sual scene containing visual details which "fit" the feature detecting processes of the retina. We might then imagine that each locus of excitation on the tectum triggers a motor command aimed at the corresponding point in space so that the summated effect of all the muscle commands yields a response to the "average" location--the "feature center of gr~vity" (see Didday for a more detailed discussion) . Th1s scheme would predict proper behavior when faced by one "prey object", but would oredict responses to the average spatial location of a number of visual stimul~. Surpris~ngly enough, this does sometimes occur 1n the frog , but usually response .is made to only one of those objects present. The basic Pitts­ McCulloch scheme does not predict this, usually more appropriate, type of resnonse .

. The notion that the· image of an object can directly lead to the proper combination of muscle contrac­ tions required to deal with it is very appealing. It offers a rapid way to deal with the oroblem of orient­ ing the body with respect to objects ' in the environ­ ment. One possible way to allow for responses to only one of several stimuli present while maintaining the feature of allowin.9 the stimulus to shape the response directly is to alter the Pitts-McCulloch scheme by div~ding the inc?m~ng information into overlapping reg1ons and prov1d1no a mechanism to select one re­ gion. The "super-poiition" effect which uses an object's image to tailor the response then works with­ in the chosen region. The more rare responses "between" two stimuli would then res present failure of the selection mechanism.

. Clearly, there are a great number of ways to 1mplem~nt such a function, and three different imple­ mentatlons are contrasted in Didday4. One implemen­ tation utilizes cells which function much like the "newnesf and "sameness" cells described by Lettvin, et. a 1 ..

Arbib and Diddayl describe how this basic frame­ work might be used as a localizing reflex which acts in conjunction with recognition mechanisms to form a perceiving system capable of using memnry to interact

with a complex environment.

The basic organization provides a means of using visual features to control motor behavior which must be accurately tailored to local details of an object in the visual scene. Behavior which depends on de­ tailed analysis and knowledge of more subtle aspects of the objects is carried out by other regions (cortical visual centers in mammals). The optic tec­ tum or superior colliculus thus is seen as performing a reflexive localizing role.

A grasp reflex for robots

This section describes a first attemot to define a system which is organized like the visual mechan­ isms depicted in the previous section. The motor behavior which the system is to control is arm and hand movement.

I have made a number of assumptions in what follows, many of them with an eye to simplifying the problem. Many of them could be relaxed with present knowledge and further effort. Some thoughts on such expansions of the power of the reflex , are described later. The most serious simplifying assumption is that the world may be treated as two-dimensional.

One of the principal notions put forth by Arbib and Didday was that the goal of perception is in determining appropriate motor behavior. This led to the idea that the task of perception is not to con­ struct a delicately detailed internal reconstruction of the input scene but rather to help choose possible motor responses. Thus we find sensory feature detec­ tors elegantly matched to possible motor response (or perhaps vice versa). The feature detectors soon to be described owe their details to the hand they are to help control.

The hand is assumed to consist of two "fingers" on a palm as shown in Fig. 1. Details of the hand which are reflected in the desiqn of the feature detectors are the length and width of the fingers and the maximum opening distance of the fingers. As seems to be the case in animal nervous systems, motor programs (are assumed to) exist which, when given the appropriate parameters, can direct the hand so that the palm lies over point (x, y) at an angle e and with the fingers w units apart. It is then the task of the grasp reflex to analyze the visual scene and produce values of these four parameters which are appropriate for graspina some object in the scene. Four parameters suffice because the system to be described functions in two dimensions only.

Further simplifications arise from the assump­ tion that the two-dimensional visual input is ob­ tained from "above" the real world scene and in such a way that all objects are seen from the same dis­ tance. Thus an object's size is immediately recov­ erable from the size of its retinal image. Finally I have assumed that all objects are darker than the surround.

Figure 2 provides an outline of the conceptual organization of the system. The implementation is for a serial computer, thus where layers of parallel operating cell types would exist in animals, this implementation simply maintains lists of the loca­ tion, orientation and level of excitation of active "cells". The terminology from nervous systems is used throughout, however.

There are five classes of "cells" which process the visual scene to produce a specific motor command. All are "directional". The first two are sensory feature detectors which receive inputs from local regions of the retina and detect "tongues" (class a) and "inward boundedness" (class b). These somewhat obscure descriptions are ellucidated in Fig. 3a and b. There is already a large amount of detail of the nature of the hand included at this level. The max­ imum opening width of the hand determines the width of the excitatory region of the class a cel .ls. The length of the class b cells corresponds to the length of the hand, and the width of the inhibitory cell row in the class o cells corresponds to the widths of the fingers. All these would chanae if a different hand were to be used. - In a nervous system implementation, there would be some number of overlappina cells of both classes for each point on the retina, and the outputs of these feature detectors would lie in some topographically mapped region. In the current, serial implementation, a list is created with an element for each class a cell which has a non-zero level of excitation, and similarly for class b.

Class d cells are excited by the concurrence of a class a cell at angle e and a class b cell at the same anqle in a reaion "to the left of" the class a cell. Thus a class d cell signals the joint existence of a "tongue" and a reqion appropriate for the left finger to occupy. Class e cells do the same but for the right finaer.

Class c cells are excited by concurrent class d and class e excitation which is centered on the same class a cell. Thus class c excitation sianals an appropriate angle and position for the nalm of the hand to occupy.

The selection mechanism operates on the totality of class c excitation, and selects that spatial reaion which has maximal class c excitation, just as the selection mechanism in the froq model selects that region which has maximal "foodness" potency. Since an object which is small enough to fit entirely within the opened hand could be grasped equally well from any direction, there will in general be many class c cells excited. To aid the selection mechanism in such cases, a bias is applied to those class c firing levels which result in arm positions which are less "awkward" being favored. That is, hand orientations that call for the least deviation from the hand/arm starting position are preferred.

Once a region has been chosen, the class c, class d and class e excitation levels in that reqion are used to derive the motor command (the positional par­ ameters for the hand). The hand position and orien­ tation command is obtained by computing the sum of the x and y coordinates and angle for each class e cell in the region weighted by the firing rate of each cell. It is this weighting process which allows the determination of position and angle to a greater precision than the spacing of the cells. The command for finger width is computed in a similar manner from class d and class e cells within the chosen region.

Figs. 4-6 show three visual scenes on which the simulated mechanism has been tested. The input scene is a 50 x 50 array of squares. The actual inout to the program is an array which for each square has a number corresponding to the proportion of that sauare which is dark. Fig. 4 illustrates a blob shaped ob­ ject which has a lobe to the right which {given the feature detectors used) is good for grasping. This region produced maximal class c excitation, so the parameters are tailored to grasp the object there. The "hand" is sketched in the chosen orientation.

Fig. Sa illustrates an object which is squared off at one end and pointed at the other and which is at an angle with respect to the coordinates of the input grid. Fig. Sb illustrates the regions corre­ sponding to those class a cells which were excited. The response is made up of weighted sums of this information, and so is appropriate to the position of the object even though no cell is oriented colinearly wi.th any side of it.

Fiq. 6 illustrates the ability of the mechanism to choose one of several objects present . Given the forms of the class a and class b cells, a squared off end is more appropriate for graspina than is a blob, so the upper object is chosen. The approach from the right was chosen due to the bias toward smaller arm movement.

Conclusion

In its current, preliminary form, the grasp re­ flex carries out the operation "go get the best look­ ing thing to grasp" . What is "best" is determined by the two classes of feature detection units operating on the input scene. Another layer of feature detec­ tors which process the scene before it gets to the grasp reflex could bias what is "best". This new layer (which would be under program control) could serve to enrich the power of the grasp reflex by allowing different sorts of things to be "best" at different times. If, for example, "handles" have some particular feature characteristics, appropriate settings of this new layer could cause the reflex to pick an object up by the handle (for using it) one time, and pick it up by some other appropriate part (for g)ving the object to someone else) the next.

L1ke any reflex, the mechanism has a fairly narrow range in which it can function appropriately. If, for example, the reflex chooses to grasp an ob­ j~ct which turns out to be too heavy to move, some h1gher level function which makes a more sophisticated analysis of the situation will have to be alerted. In cases when the grasp reflex is adequate, it can serve to perform the task of orienting the hand very rapidly at very little computational effort.

The scheme must be further enlarged to incor­ porate "negative" sorts of information, e.g. excita­ tion caused by objects not in the chosen region could be used to prevent those motor responses which would bump into other objects. Also, negative information could be used to allow picking up an object w.ith the !in~ertip~ when it is.shaped so that a full hand grip 1s 1mposs1ble. More 1mportantly, negative information is needed when the object is so small that it can be picked up from any orientation. In this case even weighting in favor of minimal arm movements will not preclude the possibility that the average location of excitation lies within the boundaries of the object-­ thus producing a command which would tend to place t~e.h~nd on top of the chosen object. One other pos­ s1b1l1ty that shows promise here is to use information fro~ the object to alter the size and shape of the reg1ons used by the selection mechanism.

The most important alteration would be to have the system function in a three dimensional world. Feature detectors responsive to visual features such as those Guzman5 utilized might prove helpful here . Most of Guzman's features are local in nature--if com­ plex global features prove to be necessary then the reflex organization becomes unattractive--the reflex is of value only if it is extremely fast.

It is conceivable that the organization described here is really only appropriate to the parallel nature o! the brain. Much of the efforts in the analysis of Dldday2,4 were concerned with the problems of global versu~ local information flow. In the brain, global funct1ons are very costly (require many interconnec­ tions)--in a serial computer they are somewhat more expensive than local ones due to the combinatorics involved. It remains to be seen whether taking ideas of organization from the brain is of use to robots. Perhaps the organization into hierarchically con­ trolled reflexes will provide less direct insights than those suggested here, but it seems to have been so universally employed in brains as to be of some basic value.