The Use of Geons for Generic 3D Object Recognition

1 Introduction (Chair)

In 1985, Biederman introduced to the computer vision community a theory of human visual object recognition called Recognition by Components (RBC) [BIE85]. The main tenets of RBC, supported by psychophysical exper­ iments [BIE91; BIE93], are that : (i) the basic-level repre­ sentation recovered from an image during primal access is metrically invariant, i.e., invariant to scale, transla­ tion, or orientation in depth, and (ii) the basic-level rep­ resentation is composed of part.s and their interrelations. In addressing these two principles, RBC also suggests that from nonaccidental contour relations in the image, a set of contrastive dichotomous (e.g., straight vs. curved axis) and trichotomous (e.g., constant vs. tapering vs. expanding/ contracting cross-sectional sweep) 3-D vol­ umetric primitive properties can be determined. The Cartesian product of the values of these properties gives rise to a set ofvolumetric primitives, or parts, which Bie­ derman called geons. The notion of quickly recovering a set of object-centered, qualitative parts (geons) from an

Irving Biederman

Dept. of Psychology University of Southern California Los Angeles, CA USA 90089-2520

Alex Pentland

Vision and Modeling Group Media Laboratory Massachusetts Institute of Technology Cambridge, MA USA 021:39

image, both without a priori knowledge of the scene con­ tents and without expending any effort on depth recov­ ery, quickly attracted the attention of the object recog­ nition community. Since then , a number of researchers from around the world have engaged in building compu­ tational models based on Biederman's RBC theory.

Beginning in 1988, Bergevin and Levine [BER88; BER89a] introduced their PARVO system, represent­ ing the first significant effort to build a vision system based on geons. From manually segmented line draw­ ings (and from range data in 1991 [BER91]), they were able to recover and recognize a variety of multi-part ob­ jects. Then in 1989, Dickinson, Pentland, and Rosenfeld [DIC89] introduced a hybrid object representation in­ tegrating object-centered (volumes) and viewer-centered (aspects) models, with the object-centered models com­ posed of parts closely resembling geons. This was to become the backbone of the OPTICA system which rec­ ognizes 3-D objects from real images [DIC92a; DIC92b] .

A long-time advocate of recognition-by-parts, Pent­ land was the first to introduce parametric part repre­ sentations to computer vision, using deformable implicit functions such as superquadrics [PEN86] . Similar to geons but with continuous rather than qualitative prop­ erties, this type of representation has been successfully used for recognition in segmented data [PEN91], and in modeling complex natural objects [SCL91] .

In 1991, M unck-Fairwood investigated a logic pro­ gramming approach to the recovery of geons from a manually segmented line drawing [MUN91a] . Re­ c.ently, he has integrated causal probabilistic networks into his approach, assigning confidence measures to Biederman's dichotomous and trichotomous properties

[MUN9lb]. Diverging from the analysis of labeled con­ tours, Raja and Jain explored geon recovery from range data [RAJ92] . Their technique, which recovers geons from superquadrics fitted to range data, has been suc­ cessfully tested on range images of real geon-based ob­ jects with straight axes.

Eklundh and his colleagues are building an active vi­ sion system whose goal is to recognize geon-based ob­ jects from images acquired by their stereo head [EKL90] . In recent work, Eklundh and Olofsson were the first to study the problem of geon-based object maintenance over time [EKL92], a critical problem facing the design and construction of geon-based active vision systems.

During these efforts by the vision community to build geon-based recognition systems, Biederman was explor­ ing his own computational models for geon recovery. In 1988, Hummel, Biederman, Gerhardstein, and Hilton be­ gan exploring connectionist models for geon recovery, while in recent work [HUM92], Hummel and Biederman report a neural net model for shape recognition, using dynamic binding, that can recognize simple geon-based objects from line drawings.

Despite the growing interest in geons in the vision community, the above systems are still being applied to very constrained domains in which segmentation is of­ ten performed manually, lighting is tightly controlled, or the objects themselves are unrealistically simple. Never­ theless, the motivation for building generic recognition systems is strong. If a robotic vision system is to move about its environment, quickly recognizing new object exemplars from a large database of generic, articulated object models , as well as learning new object descrip­ tions, it must have the capability to recover and match image representations which capture the coarse shape of an object in terms of its parts. Geons have emerged as the most popular shape representation for generic ob­ ject recognition. However, before successful geon-based recognition systems can be realized, a number of impor­ tant issues must be addressed, including:

• Are geons an appropriate modeling vocabulary for 3-D generic object recognition systems? If so, for what classes of objects are they best suited? If not, can the representation be altered or enhanced to make them more usable?
• What are the major obstacles in recovering geons from real imagery? What image features should be extracted in order to recover geons?
• What kinds of architectures and control strategies are appropriate for geon-based recognition?
• Finally, with the growing interest in active vision, what issues face the design and construction of geon­ based active recognition systems? And what role can geon-based recognition play in a larger context where object manipulation may be required?

In the following sections, six researchers, representing broad experience in the design and construction of geon­ based object recognition systems, address the above is­ sues.

2 Testing the RBC Theory: An Approach Respecting the General Principles and the Computational Hypotheses

Robert Bergevin

Laval University

We see the original RBC proposal by Biederman as made up of two loosely connected sets of predictions: (i) the general principles, and (ii) the computational hypothe­ ses. The general principles state that:

1. A line drawing which represents the discontinuities in an image is an effic.ient description of the. image and is sufficient for the primal access task.
2. Objects are better represented and analyzed by decomposing them into their natural components (parts) .
3. :3 . A qualitative description of the components is nec­ essary and sufficient to permit fast access to a large database of object models.
4. Non-accidental instances of viewpoint-invariant fea­ tures in the 2D line drawing are sufficient to permit fast access to the qualitative model of a 3D object .
5. Primal access for visual object recognition is ob­ tained by matching a description of the spatial structure of components making up an object to an indexed database of models in a similar representa­ tion.

On the other hand, the computational hypotheses state that :

1. Five specific classes of 2D line groupings are suffi­ cient to access the parts representation.
2. Segmentation should happen at concavities in the outline of an object.
3. The geons, a subset of the generalized cylinders , form an efficient qualitative shape representation for the parts which is suitable for the primal access task .
4. The symbolic descriptions for objects and models should include geon labels, aspect ratios, and rela­ tive sizes of parts.

While the psychological results directly support the general principles, the computational hypotheses are more like extrapolations or predictions regarding an ac­ tual implementation. This was meant to fill the gap be­ tween the levels of study in the psychological and com­ putational perception fields (the "what" and the "how" of the problem, respectively) .

The main assumption of RBC is the suitability of sin­ gle view line drawings for primal access. This, together with the small set of volumetric primitives making up the geons, represent, in our opinion, the atomic kernel of the theory without which the goal of rapidly recogniz­ ing arbitrary unexpected objects cannot be attained. Of course, it could make sense to diverge from these prin­ ciples and, say, analyze multiple 2D views of an object , recover the object from range data, reconstruct quanti­ tative surfaces, volumes, or exact models, etc. But the result would neither represent primal access nor even generic recognition, which is simply primal access fol­ lowed by sophisticated high-level knowledge-based rea­ sonmg.

The difficulty to extract a very good line drawing in an uncontrolled imaging situation is well acknowledged in the computer vision field . Two approaches have been taken around this problem. Most groups assumed the (usually ideal) line drawing as given. Others decided to extract line drawings from real data (usually from range images) but had to restrict themselves to ideally shaped custom made objects. From the point of view of low level vision processes, the real question to be answered is how one goes from imperfect data from a non-ideally shaped object to the much simplified and compressed description in terms of an ideal line drawing. No one has yet a satisfactory answer to this question. Even worse, no one knows what would be such an ideal line drawing suitable for primal access and eventually generic recognition .

In our work with the PARYO system, two main goals were pursued [BER89a; BER92c]. First, we aimed to determine what are the underlying assumptions about the geometric properties of objects and their line draw­ ings that make them acceptable to the kind of processing proposed in RBC. That is, what objects may be recog­ nized with such an approach, what is their complexity, etc. Second, given such acceptable line drawings (ideal or noisy) , we wanted to develop methods to actually per­ form the feature extraction, part segmentation, geon la­ beling, object description, and model matching to show that complex object recognition is actually possible in realistic conditions. Our main c.ondusions are that:

1. Primal access geon-based recognition is possible for a large number of man-made objects.
2. Recovery of geon-based descriptions from natural objects is likely to be very difficult .
3. :3. To get ac.ceptable line drawings will also be a quite difficult task especially if some of the object parts and connections do not respect the implicit assump­ tions in space or in the projected view.
4. Explicit part segmentation is needed and may still be possible even if some elements of the object or line drawing do not respect the implicit assump­ tions.
5. Face extraction and labeling is necessary for proper geon labeling.
6. Errors arising from blind labeling of the parts (with­ out explicit verification of the hypotheses) is likely to destroy the indexing power of the geons.
7. Explicit verification is also necessary to make the recognition system robust against noisy (missing or spurious) data.
8. Parallel architectures and algorithms are necessary to cope with the combinatorial complexity arising from manipulating sets of features.

3 Recognition-by-Geons: Current Progress and Current Challenges

Irving Biederman

Peter C. Gerhardsteiu Eric E. Cooper J ozsef Fiser University of Southern California John E. Hummel

University of California, Los Angeles (UCLA)

In a fraction of a second humans are able to r.ompre­ hend novel images of objects and scenes. Recognition­ by-Components ([BIE87]) offered an account of this phe­ nomenon characterized by four general assumptions: a) Objects are represented as an arrangement of simple con­ vex or singly concave parts (geons), b) The geons can be distinguished by binary contrasts (differences) in view­ point invariant properties (YIPs), such as straight vs. curved, rather than metric properties such as aspect ra­ tio, c) The relations among the geons are explicit, such as PERPENDICULAR-TO or TOP-OF, as part of a struc­ tural description, rather than implicit in a coordinate space, and d) A relatively small number of geons is suf­ ficient.

Results from priming experiments [BIE91; BIE9:3], in which the speed and accuracy of perceiving a briefly pre­ sented picture is facilitated by a prior presentation of a related picture, provide support for a simple, parts-based representation that is intermediate between a) local fea­ tures such as filter outputs or image edges, and b) a global model (shape or concept) of the object. More­ over, variations in YIPs, such as changing a geon class from a cylinder to a brick, have a more significant effect on an object's real-time classification than variations in the geon 's aspect ratio. A recent study of a success­ ful face recognition system, which assumes a coordinate space and no intermediate primitives, showed fairly good overall accuracy in the rer.ognition of images but its per­ formance was strikingly different from that shown by human observers.

A neural net implementation of RBC ([HUM92]) takes a line drawing of an object as input and recruits a unit that is invariant over translation, size, and rotation­ in-depth. The unit represents a structural description (geons + relations) of the object . The model's ca­ pacity for structural description derives from its origi­ nal solution to the binding problem of neural networks. The input layer consists of a lattice of orientation-tuned cells. Fast Enabling Links (FELs) between cells that are collinear, coterminating, or closely parallel cause those cells that are currently activated by one geon to fire in synchrony but out of phase with the cells activated by another geon. The synchronous activity in a single time slice is used to bind independent units representing an object's geons and its attributes (e.g., vertical) and re­ lations to other geons (e.g., above). For example, given an image of a vertical cylinder above a horizontal brick, the units for cylinder, vertical, and above will be induced to fire in synchrony but out of phase with the units for brir.k, below, and horizontal. Limitations in the temporal bandwidth for establishing phase differences results in accidental synchrony causing a large number of ele­ ments to be perceived as texture unless irrelevant activ­ ity is suppressed by attention.

Geon representations are not suitable for representing objects that do not readily decompose into simple parts. Differences among such objects, however, tend not to be incorporated as boundaries between basic level objects nor do they evidence the viewpoint invariance shown by objects that are readily described as an arrangement of geons [BIE93] . A possible benefit from employing a neu­ ral net model is that it can offer some capacity for achiev­ ing recognition (through partial activation) in the face of the ambiguity often inherent in images of such objects.

Challeuges

Are there a small number of visual primitives? We are in­ vestigating this issue in the context of aspect graphs. Are there a limited number of part types that would be dis­ tinguishable from all other part types over large regions of the viewing sphere generating the aspect graph'? For example, a brick can be identified as such from almost any viewpoint. If the areas of the viewing sphere corre­ sponding to each interpretation are converted to proba­ bilities, then geons have the property of low uncertainty in the Shannon sense. Is this a good criterion for identi­ fying part types for human recognition? One attractive feature of this conceptualization is that one can define geons to be the set of volumes that have minimal view­ point uncertainty. Low viewpoint unr.ertainty, moreover, may yield optimal conditions for a self-organizing net­ work to develop hidden units that function as geon de­ tectors.

Can a theory such as RBC be fast enough? Can the mechanisms of contour binding and geon activation be completed quickly enough to account for 100 msec recog­ nition times'?

Ignoring small variations. We can often determine, without a great cost in processing difficulty, volumetric structure despite the presence of small variations in dec­ oration, flanges , etc. How?

4 Geons: A Viable Approach for Computer Recognition of Generic Objects?

Jau-Olof Ekluudh

The Royal Institute of Technology

Visual object recognition is a wide and unwieldy sub­ ject on which computer vision has made but limited progress . One reason for this may be that problems range from recognizing known objects to general catego­ rization, and that a common framework therefore may be elusive. Considering recognition of generic objects using volumetric primitives, like geons, can in this perspec­ tive form an attractive middle ground. Although it does not encompass all thinkable types of objects, since some natural objects like bushes may not even be character­ ized by their shapes, it does provide a rich enough world to be of interest. Moreover, it is also known that geon representations allow indexing and hence rapid recogni­ tion . Finally, even though categorization problems in their most general form can hardly be addressed if only shape is used, recognition by shape primitives does apply to some of them. Notwithstanding important advances in the past few years, there are several critical issues to be addressed before we know how far the geon~based ap­ proach will take us. I will discuss some of these and how I think they can be treated at least in a general sense.

The first issue concerns how geons can be recovered from imagery. Most high level work has in this field, like in other recognition work, been based on perfect or near-perfect line-drawings. This is certainly not a realis­ tic way to go. However, geons are qualitative in nature, and could be derived by some shape abstraction prin­ ciple, whir.h fits or associates primitive shapes to data. Several schemes in this vein have been suggested in the literature. However, a crucial requirement is that the typical geon properties should be made explicit, that is properties like being flat or curved, growing or taper­ ing etc. This is not generally the case for approaches based on fitting methods, and may require non-linear techniques or cause uniqueness problems. Another im­ portant aspect is the treatment of structures at different scales. Generally speaking I don't think we know how to handle these problems, but I do believe that there exist promising methods.

The second issue is about the use of depth informa­ tion . In his work on human recognition, Biederman is using line-drawings only. I think one must be careful in drawing conclusions about machine vision approaches from that . The display situations are such that the sub­ ject sees the object clearly separated from other objects in the fore- and background. Unless qualitative depth is provided either from accommodation cues or from binoc­ ular or other cues, a machine vision system using im­ ages with a large depth of field is faced with difficulties. Truly, Lowe and others have shown that recognition can be done in static monocular images. However, I think that then the possibilities of performing rather complex types of reasoning are needed, which goes against the general advantages of geon-based recognition . In prin­ ciple, I feel that the problem should and can be posed in the presence of qualitative depth information, which can be used for segmentation and disambiguation. On the other hand I think that having dense depth informa­ tion from stereo reconstructions or range sensing defines a completely different problem, and also a problem that is mainly of a technical character.

The third issue concerns whether the full geons need to be recovered or if observed subparts can have enough indexing power. Biederman has convincingly demon­ strated that partial information is sufficient. However, this is again in a context where you don't see a large scene collapsed in depth. If those conditions are valid, partial information may not constraining enough and can lead to combinatorial problems. On the other hand, if it seems as if there is some other process selecting a re­ gion of interest, or generally guiding the attention, then there are viable computational techniques which could be based on typical partial information that we can obtain from real images.

Generally I think that geon-based recognition of generic objects can be a powerful approach, even though it has limitations. However, I also think that the tech­ nique must be applied in a context where the issues I've listed, and also other problems, can be addressed appro­ priately.

5 Geons and the 'What'-function

Roger Munck-Fairwood Li Du

University of Surrey

Generic object recognition (GOR) may be regarded as the attempt to fulfill the 'what'-function of machine vi­ sion. We have identified a fundamental question which must be answered ifGOR (as it is currently understood) is to be posed as a scientific pursuit. The question is: How can the 'what'-function be defined for both human and machine vision?

We further propose a system architecture which ad­ dresses the first three of the issues listed in the introduc­ tion. We also discuss the fourth issue.

5.1 'What'-function and the Purpose Concept

In our definition, the 'what' function puts a 3D entity into a predefined bin of classification on the basis of the recovered geometrical properties. Let us assume that W is the vocabulary for describing the geometry of 3D entities (as deliverable by a collection of lower level vision processes) , C is the set of predefined dassification bins and M is the classifications process. A single task of recognition can then be formally expressed as c = M ( w) where c E C and wE W.

In the light of this formalism, the current talk about generic object recognition systems which should work 'reliably' implies the assumption that there exists a uni­ versal (all-purpose) C. Clearly, this is a false assump­ tion, because even given a perfect w (geometrical de­ scription of an object), the way the bins are designed strongly depends on either engineering criteria or cul­ tural customs. Regarding W, one can always resort to object geometry, in mathematical terms, as an ultimate unambiguous object description (but not classification), although the lower level processes might not yet be able to provide this from an image. Whereas for C, we do not even know how many overlapping C's exist in our culture.

Another apparent flaw is that in the absence of a con­ straint on C, the indexing power of a GOR system can be grossly deceptive. Given a W (e.g. geon clusters), it is a trivial task to imagine an M which results in a C with hundreds of thousands of bins but many or most of these may be practically meaningless.

In conclusion, this formalism pinpoints two fundamen­ tal aspects of GOR: (i) The bins of C which must satisfy a purpose or a set of purposes and (ii) an architecture of geometry recovery processes which delivers the best collection of W. The benefit of this formalism is that the generality (or 'generic-ness') of any 'generic object recognition system' can be measured- not by the num­ ber of bins in a C - but by the richness of the set of purposes P which its C satisfies and the versatility of its W which can be delivered by the recovery sub-system. So, we no longer regard GOR as universal but bounded by purposes on the top end and richness of geometric description on the lower end .

5.2 A Framework for Recovery Sub-system

We have outlined [M U N93] a recovery sub-system design to provide a versatile repertoire of image labeling tools as a servant to an object dassification system. The basic principles behind the system design include the follow­ ing. (a) It is a servant to a classification sub-system hav­ ing a high degree of transparency in the sense of making process parameters (scale, thresholds, etc.) explicit. (b) The architecture has the potential to incorporate any existing approaches of 3D geometry recovery. (c) The image labeling vocabulary is divided both into levels of abstraction and levels of cost. (d) Reasoning occurs in three directions: bottom-up (b-u), top-down (t-d) and horizontally. When b-u and t-d are in conflict, either b-u can dominate (corresponding to visual illusion in human vision) or t-d can dominate (corresponding to subjective contours) .

We illustrate the system using 2D edge-based features, but surface features can also be incorporated as an alter­ native channel to the recovery of primitives (geons being the best so far) .

5.3 Object Classification Sub-system

This sub-system has two-way communication with the W-sub-system. It not only takes b-u input from the latter but gives cognitive (predictive) information to it when possible. Its basic function is to take the recovered geometry at the geon-cluster level of the W -system and put each cluster into the appropriate bin of C.

In a given engineering task, the human operator may define the bins of C, thus simplifying the task of this sub-system. But in general, the sub-system should au­ tomatically c.reate the bins of C according to a given purpose or set of purposes P.

5.4 About Active Vision

Our definition of GOR shares the 'purpose' concept with that used in the study of active vision, but from an en­ tirely different point of view. Currently in active vision research, the emphasis is shifted towards sensor-motor interaction and the 'purpose' concept is used to circum­ vent the need for the 'what'-function (e.g. by gazing on a bright spot or a color pattern). In contrast, we em­ ploy the 'purpose' concept to directly address the issue of object recognition by linking P with C. We consider object recognition as a key component for a general ac­ tive vision system, where often the actions can only be taken when 'what' is answered.

6 Parts Representations

Alex Pentland

Massachusetts Institute of Technology The representation of objects by their parts has a long tradition in computer-aided design, simulation, and cog­ nitive psychology. Indeed, in these areas it is the domi­ nant strategy for representing complex 3-D objects. It is absolutely clear, therefore, that part representations are excellent for many computational and cognitive tasks. What is not so clear is how they might be useful in com­ puter vision.

The first parts representation was suggested by Bin­ ford 1971 ; this is the idea of generalized cylinders (G.C.). Because of the relatively unconstrained nature of this representation, the recovery process seems to require elaborate line grouping and reasoning. Consequently there are relatively few reports of recovering suc.h de­ scriptions from real imagery. Moreover, because such descriptions are often not unique, it is unclear how they aid in object recognition.

About 1985, these problems with G.C. representations began to be taken seriously, and two different proposals arose to deal with them. One variation on the G.C. idea is due to Biederman [BIE85], who suggested using the Cartesian product of qualitative properties such as tapering, c.ross-section , etc., in order to create a quali­ tative taxonomy of G.C. One advantage of this type of representation, called geons, is that the properties can be chosen to be ones that are more easily recovered from imagery. Another is that it provides a way to define qual­ itative shape classes, an important problem in general­ purpose VlSJOn.

To date, however , only Dickinson , Pentland, and Rosenfeld [DIC92b] have successfully used this ap­ proach to recognize objects in real imagery, although Bergevin and Levine [BER89a], Hummel and Bieder­ man [HUM92], and Munck-Fairwood [MUN91b] have re­ ported good success in interpreting line drawings. The principal problems with this approach seem to be the difficulty in extracting sufficiently good line drawings, and the idealized nature of the geon representation .

A different variation of the G .C. idea was suggested by Pentland [PEN86], who proposed a parametric ver­ sion of G.C. based on deformable implicit functions. The original version of this proposal used superquadric functions with multiplicative deformations ; later ver­ sions used spheres with additive ("Modal") deforma­ tions. Use of a parameterized implicit function (de­ formed superquadrics or spheres) converts the problem of recovering a description into a relatively simple nu­ merical optimization. Moreover, if the parameterization is orthogonal then the description is unique, making the recognition problem much easier.

In recent years, several authors have reported success at recovering this type of description and using it for recognition, e.g., Pentland and Sclaroff [PEN91] were able to recognize people's head shape with 96% accu­ racy. The major limitations of this approach seem to be the requirement for pre-existing part segmentation, and an initial estimate of orientation.

It is interesting to compare the geon and parametric approaches. Both describe a similar range of shapes; perhaps the major difference is that one is qualitative while the other is quantitative. This similarity has made it possible to generate geon statistics using parametric models [DIC92al, and allowed Raja and Jain [RAJ92] to use superquadrics to recover geons.

It seems to me that the qualitative geon representa­ tion is best suited to understanding intersections and groupings of parts, while the parametric representation is best for classifying individual part shapes. I speculate that the most successful systems will employ first a qual­ itative pruning/grouping stage to perform segmentation, and then a parametric fitting stage to extract detailed part shape. This approach was recently successfully em­ ployed in [DIC92c] .

A final question is representational adequacy. It seems clear that to model natural objects the G.C. represen­ tation must be augmented by a displacement map as in [SCL91]. Moreover, the G.C. representations cur­ rently used in vision are much simpler than those used in CAD. Consequently, I predict that the more expres­ sive G.C. representations [PEN9l][SCL9l], will become increasingly popular.

7 The Obstacles Facing Practical Geon-Based Recognition

Sven Dickinson

University of Toronto

I see three major obstacles facing the success of geon­ based object recognition systems: (i) the recovery of geons from real imagery; (ii) the difficultly in explicitly modeling real objects using geons; and (iii) the lack of representational power, e.g., depth and fine shape, pro­ vided by geons for the task of interacting with the world.

7.1 Recovery

In our work on geon-based recognition, we have at­ tempted to address the issues in building geon-based systems that work on real imagery [DIC92a; DIC92b] . We chose an approach to volumetric primitive recovery based on part-based aspect matching which, in fact , can be applied not only to geons, but to other classes of volumetric primitives as well. From our analysis, we discovered that image regions, not lines or line groups, were the most appropriate features to extract from an image. Image regions not only provide a natural group­ ing of image contours, i.e., the bounding contours of a region , but region grouping provides a powerful infer­ encing mechanism for volume recovery. Our aspect hier­ archy, a multi-level, viewer-centered representation of a set of volumetric parts, allows us to recover geons in the presence of noise, occlusion, and low probability views.

Despite the utility of regions in recovering geons , we are still faced with the problem of quickly extracting regions from an image. ln addition, how do we deal with the inevitable under- and over-segmentation of im­ age regions? And finally, at what stage in the recovery of geons, do we attempt to rectify such segmentation prob­ lems: immediately following segmentation? at the geon verification stage? or, at the object verification stage? ln working with real images, it becomes very clear that the segmentation cannot be decoupled from the recognition. I believe that a dynamic feedback scheme between the various stages in the recognition process must be ex­ plored before a geon-based object recognition system can be successfully applied to real images. We are currently exploring feedback schemes applied to segmentation and recognition.

7.2 Modeling

One of the most challenging tasks in building a geon­ based recognition system was finding geon-based objects to test it on! In far.t, we ended up constructing our own geons in order to build our own objer.ts. That is not to say that there are not geon-based objects in the world . However, what we found was that although many of the real objects we did look at could be coarsely described in terms of geons, there was a plethora of detail in the ob­ jects that did not contribute to their coarse shape. Even ignoring surface markings and texture, there would be a lip on a cup, a small ridge around the edge of a table, or other fine object structure whose projected contours in the image were quite salient. Although Biederman sug­ gested the use of nonaccidental relations between con­ tours in recovering geons, neither he nor the line-drawing based geon-recovery systems which he inspired addressed the issue of structural scale. I believe that geons are an appropriate indexing primitive for recognition . However, we must be able to recover abstract geon descriptions of objects with structural detail.

7.3 Representational Power

For a vision system to interact with the world requires that it not only identify or recognize the objects that it wishes to interact with , but that is sufficiently localize (in depth) and characterize (in dimension) the objects in order to manipulate them. Although geon-based ob­ ject recognition techniques may prove suitable for the identification task , they fall short in localizing and char­ acterizing the object. Although geons were not intended for this task , we are studying ways in which geon-based recognition can be used within a larger framework.

At the University of Toronto, we are building a vision­ based mobile robot whose goal is to allow handicapped children to interact with their environment through the use of a robotic manipulator . The child interacts with the system by instructing it to find and manipulate toys on a table; hence, we need both recognition and local­ ization of objects. In the spirit of decoupling the "what" and "where" tasks, which are typically tightly coupled in verification-based object recognition approaches, we can maintain the speed and efficiency of a qualitative front-end recognition module, and invoke a localization module only when necessary. We have built such a sys­ tem, using qualitative shape to constrain the fitting of physics-based deformable models to the image [DIC92c].