A Parallel Computation that Assigns Canonical Object-Based Frames of Reference

A PARALLF.L COMPUTATION THAT ASSIGNS CANONICAL OBJECT-PASED FRAMFS OF RF.FFRFNCE

Geoffrey F. Hinton

MRC Applied Psychology Unit Cambridge, Fngland

shape description. An upside down table, for example, is seen as just that, because by seeing it as upside down we can see it as a familiar shape. In this case, there is clearly nothing in the image to indicate that an upside-down frame should be assigned.

This paper describes a way of using parallel hardware to implement a cooperative computation in which the process of choosing an object-based frame and the generation of a description relative to that frame occur simultaneously, with each influencing the other.

II FRAMES, FEATURES, AND MAPPINGS

Following Marr f4~, I shell assume that early visual processing yields a representation of a scene in terms of 3-D features relative to a . 3-n frame of reference defined by the retina (or camera). These "retina-based" features cannot be used directly for object recognition because they depend not only on the shape of the object but also on the spatial relationship between the retina and the object. Choosing an object-based frame of reference ia equivalent to choosing a mapping from 3-D retina-based features to 3-n object-baaed features. This mapping compensates for the relationship between the retina and the object and thus yields features that are independent of this relationship. These features constitute a shape description that can be used for object recognition.

If we think in terms of a parallel system in which each feature corresponds to a specific hardware unit which is active when the feature is present, then a particular choice of an object­ based frame must be capable of pairing each retina-based unit with a corresponding object­ baaed unit, eo that activity in the one can cause activity in the other. Fig. 1 shows such an arrangement for a eimpl ified 2-D domain. There are many "channels" emanating from each retina­ based unit. A choice of an object-based frame corresponds to activation of a particular "mapping" unit. An active mapping unit opens one channel from each retina-based unit to the ob,i ect­ baeed unit that is appropriate given that mapping. Thus, once the mapping has been selected, a pattern of active retina-based unite

ABSTRACT

A. vievpoint-independent description of the shape of en object can be generated by imposing a canonical frame of reference on the object and describing the spatial dispositions of the parte relative to this object-based frame. When a familiar object is in an unusual orientation, the deciding factor in the choice of the canonical object-based frame 11ay be the fact that relative to this frame the object has a familiar shape description. This may suggest that we first hypothesise an object-based frame and then test the resultant shape description for familiarity. However, it is possible to organise the interactions between units in a parallel network so that the pattern of activity in the network simultaneously converges on a representation of the shape and a representation of the object­ based frame of reference. The connections in the network are determined by the constraints inherent in the image formation process.

I INTRODUCTION

People can recognise a familiar spatial structure from a novel viewpoint. There is considerable evidence that they do this by imposing a canonical, object-based frame of reference and generating a description of the spatial structure relative to the assigned frame f1 2 3]. A tilted square, for example, can still be seen as a square because relative to the til ted frame that is imposed on it, it still has two vertical edges on either side and horizontal edges at top and bottom.

The central problem in using object-based frames is to devise a way of assigning the appropriate frame to a perceived object. This is not an easy task, even if sources of information like stereo, shape-from-shading, or optical flow hav"e yielded the precise 3-D structure of the object relative to the viewer-centered frame of reference. Heuristics like planes of bilateral symmetry, gross elongation, and the gravitational or contextual vertical can help to suggest candidate object-based frames, but the final choice between the alternatives often depends on which object-based frame gives rise to a familiar will cause e pattern of active object-based units, end this later pattern will constitute e description of the object's shape.

Figure 1 · This shove how each mapping unit (large box-on right) controls a set of channels from the retina-based units (bottom) to the object-based units (top). Only e few of the mapping units end channels ere shown. The topmost units respond to combinations of active object­ based units. The meaning of the triangular symbols is explained in the text.

JJI TOP-DOWN SELECTION OF A MAPPING

The meaning of the triangular symbol in fig. 1 is quite complex. It stands for two rules:

1. Multiply the activity level in the retina­ based unit by the activity level in the mapping unit end send the product to the object-based unit.

2. Multiply the activity level in the retinA­ based unit by the activity level in the object­ based unit end send the product to the mapping unit.

The first rule is whet causes the appropriate mapping when e single mapping unit is fully active (activity varies from 0 to 1 ). The second rule selects e particular mapping when the activity of the object-based units is determined top-down. Fech possible pairillf'; of en object­ based feature vi th e retina-based feature of the same general type will send activity to 8 particular mapping. If the object is present in the input, the "correct" pairings will ell agree on the same mapping, whereas the "cross-pairings" will not agree with one another. ~o the best vey of instantiating the object in the scene will be indicated by the mapping with the greatest total input. This vsy of using en explicit representation of mapping space to eccumul e te evidence for 8 particular mapping is also used by Ballard (5].

The structure of the connectivity in this network captures the mathematical structure of e three-dimensional spatial relationship. There ere constraints on the veys in which features relative to one frame csn be paired vi th features relative to another frame if the pairings ere to correspond to s single 3-D relationship between the two frames. Only certain sets of pairings ere mathematically consistent, end each of these sets corresponds to e set of channels that ere ell connected to the same mapping unit.

IV SIMULTANEOUS SELECTION 01 SHAPE AND MAPPING

I shell assume that the retina-based representation is rich enough to allow e roughly correct segmentation of e "figure" end a] eo to. allow e number of plausible object-based frames to be selected for the candidate figure using bottom-up heuristics. The evidence in favour of en object-based frame provides enough input to the relevant mapping unit to give it e low initial level of activity. The pattern of activity in the retina-based units is then mapped through ell the candidate mappings simultaneously, but because the mapping unite are not fully active, each mapping is attenuated (see rule 1 above) end so there ere many slightly active object-based units. Among these object­ based features there will be subsets which can be recognised as partial descriptions of familiar shapes.

The precise details of how shape descriptions are recognised as familiar is not central to the model. In Fig. 1 this mechanism is grossly simplified end shown es e set of shape­ recognition units. All ve need to assume ie some kind of positive feedback mechanism which provides top-down support for familiar combinations of object-based features. So, among the many slightly active object-based units, certain subsets will receive more top-down support than others. Once this happens, there is a runaway process. The pairings of above-average object-based features with retina-based features cause above-average support for the relevant mapping. This, in turn, causes less attenuation in the mapping from retina-based to object-based features. The more active a mapping unit gets, the more it contributes to its corresponding shape description, and vice versa. Jn a few iterations the system converges on s particular mapping and a particular shape description, and the inhibition between the mapping units (and also, optionally, between the shape units) causes the unsuccessful candidate mappings to be completely suppressed. Other things being equal, canonical frames are preferred to non-canonical ones, because of the top-down support for familiar combinations of object-based features.

V THE N~ PROBLEM

If there are N retina-based units each of which can activate any of N object-based units, Nl channels are required. This number can be reduced in various ways. First, N itself can be dramatically reduced before the mapping by distributed encoding of the individual features and mappings as patterns of activity in many different hardware units. Each unit represents a connected region in the space of possible features. A specific feature is coded by activity in all the units whose regions contain it. The number of effectively different features can then greatly exceed the number of units. This type of encoding is especially effective when the number of features present at any one time is much smaller than the number of possible discriminable features. There is not space here to include a formal treatment of the efficiency and limitations of this type of encoding.

Second, if the mapping is performed inK sequentially linked stages, the fan-out from a unit in one layer to the unit in the layer above need only be ~ , and since there are K layers only K.N . .ljlr connections are needed. However, iterations take K times as long and different mappings inevitably become confused during the settling phase when many mappings are partially active. If, for example, there is one mapping to handle translation followed by another to handle rotation, it is impossible to allow just the mappings~ R 1 and TlR~ to occur simultaneously, where T and R denote the translational and rotational constituents of a full mapping. The retina-based features will be mapped through both T 1 and T.l and the resulting intermediate features will be mapped through both R1 and R~, and so the full mappings T1 Rl and T.:~R1 will also have occurred.

VI CONCLUSION

I have outlined a way of using parallel hardware to implement a flexible mapping system that can perform a parallel search for a canonical, object-based reference frame. This is an important advance from the pandemonium or perceptron type of model because it handles the effect of viewpoint on the image in a principled way. There are, of course, other sources of variation in the image of an object, such as non­ rigid transformations of the object itself, but

there is no reason to lump these together with the effects of viewpoint. They are separate problems with their own separate structure. Finally, J should point out a major and controversial implicit assumption of the model. Object-based frames must be assigned one at a time, so that they can all share the same mapping apparatus.

RFFFRFNCFS

1. Rock, J. Orientation and form. New York: Academic Press, 1973.---
2. [2) Marr, r. a Nishihara, H. K. Representation and recognition of the spatial organisation of three-dimensional shapes. ~ ~ Soc. Series!!. 1978, 200, 269-294.
3. Hinton, G. F. Some demonstrations of the effects of structural descriptions in mental imsgery. Cognitive Science, 1979, l· 2~1-250.
4. [4) Marr, r. Representing visual information. Jn A. R. Hanson a F. M. Riseman (Eds.), Computer~ Systems. New York: Academic Press, 1978.
5. [5) Ballard, r. H. Generalising the Hough transform to detect arbitrary shapes. Computer Science Department TR-55, 1979, University of Rochester. ---