Experiments In Knowledge-Driven Interpretation of Natural Scenes

Introduction

The general goal of processes operating on visual data in the VISIONS system is the construction of a symbolic representation of the objects, and of the relations among those objects, from images of a given natural outdoor scene. Recent experiments have been designed to determine the effectiveness of the use of world knowledge in the interpretation task and to explore representations of this knowledge [9]. A portion of the knowledge used in the system consists of descriptions of specific objects (e.g. house, tree, grass, etc.) and the representation of three-dimensional spatial relationships among those objects [3]. This paper discusses one way in which such geaDetric knowledge can be used to control image processing. The use of geaDetric knowledge to guide image processing has been used with some success by [ 1 ],[2], and [5].

In the experiments described here, . a three-dimensional scene model which represents the relative location and size of objects is projected to the two-dimensional image to form a plan. This plan can be used to restrict image analysis to certain portions of the image and to trigger the application of procedures for the recognition of specific objects during image interpretation. The results will be illustrated using the house scene shown in Figure 1.

A Spatial Plan from ! Geometric Scene Hodel

Knowledge about approximate object location and size was used to investigate the effect of geometric constraints on a region-labeling process. The goal of the experiment was to partition a

·This research was National Science MCS79-1 8209. supported in Foundation part by the under Grant

EXPERIMENTS IN KNOWLEDGE-DRIVEN INTERPRETATION OF NATURAL SCENES·

Terry E. Weymouth

Computer and Information Science Department University of Massachusetts Amherst, Massachusetts 01003

region segmentation into non-disjoint subsets of regions, each containing the projection of an object fran the scene. The data used is an initial segmentation of the image into regions of similar color and texture [7]. An image region does not necessarily correspond to a single object, nor is it the case that a given object will unfailingly project to a single region . Scene knowledge can be used to partially overcaDe these difficulties through the formation of a plan which predicts the location of the image projection of actual scene objects. The subimage ·produced by a matching process which uses this plan can then be further processed to better extract the image location of the object. One possibility is resegmentation, which can be performed with greater accuracy using the local image context [7]. Spec i fic knowledge of the surface features of the object can also be used to constrain the subimage even further.

If all the objects in a scene were of known size, shape, and at a known location and orientation, and if the scene were viewed with a known camera fran a known camera position and orientation, then the scene model would be easily projected to the image using the standard techniques of computer graphics: rigid object transformation, perspective projection, and hidden surface removal [8). In general, however, the scene model must capture uncertainties as to object location and permit projection methods that are tolerant to errors in expectations of camera position and orientation. Consequently, several less constraining assumptions were made .

The three-dimensional geometry of an object in a scene can be characterized by: location, orientation, scale (size), and shape. Of these, shape is obviously the most canplex. Easing the constraint on shape by generalization to a sphere permits a rough description of location, location uncertainty, and size. The location of an object, expressed in the coordinate system of the scene model, is approximated by the center of a "location sphere." The radius of this sphere expresses uncertainty as to location of the object center. The object size is also represented by a sphere, centered at the object location. The radius of this "size sphere" is related to the size of the object. Because these spheres are embedded in a scene coordinate system, the addition of a camera model permits the transformation of the scene model (by translation and rotation) to a specific viewpoint and the projection of that model to the image.

Projection ~ ~ ~ ~ Image

There are two problems in the projection of this three-dimensional model to the image ~ the projection of the location sphere and the projection of the size sphere. The projection of the location sphere is straightforward. Using a perspective transformation (from a camera model) the center and radius of the sphere are projected to the image as a circle. Figure 2 illustrates the way in which this three-dimensional object model can be projected onto any image (using a camera model) to form an image-specific plan for that object.

The projection of the size sphere is handled in a slightly different manner. Due to uncertainty of the exact location, just the radius of the size model is projected to the imqe. The projected radius is added to the plan as a "size filter;" no region greater than that size is expected as a projection of that object. In .the experiments shown here this size radius was added to the location radius to form a composite plan, shown as the outer circle in Figure 3. The other two circles shown are: the projection of the location sphere (the smaller of the two inner circles) and the projection of the size sphere (the next larger circle) arbitrarily positioned at the center of the projection of the location sphere. Note that, although these illustrations show tightly constrained locations (small location spheres), it is possible to represent small objects and/or larger uncertainties in location.

Once the model has been projected to the imqe, the resulting plan can be used in a matchins process · . Regions in the image can be matched against the projection circles to form a subimage. All regions that are contained within the composite circle are taken to be candidates for inclusion in the subimage, Further, all regions for which the ratio of region area within the circle to the total reston area is above an object-·odel-specific threshold are accepted as candidates for inclusion. These candidates are then filtered by size, and any region which is too large is rejected. (Such rejected candidates could be split by resegmentation; this was not done here.) The union of the remaintns candidates forms the subimage.

The two spherical representations whose projections are shown in Figure 3 are for "tree-crown" and "house-body." Figure 4 shows the regions selected when those projections are used. Figure 5 shows the way in which geometric matching can be used to limit other recognition procedures. A feature matching process for house-body, based on a color and texture prototype from the object model, was run on an image segmentation to select the set of regions shown in Figure 5a. The same segmentation, tosether with the plan frc. the projection of the house-body spherical model, produced the subimase shown in Fisure 4c. Usins the seometrically based subimage to restrict the feature matching i·proves the results. Compare the regions from feature matching alone (Figure 5a) with those selected when the feature matching was L-----------------------~ constrained by the geometric plan (Figure 5b). The Figure 2. The geometric model of location is a sphere in space (A) which is projected onto the image as a circle (B). Note different viewpoints. matching usins the plan is i·proved by the elimination of those restons which are inconsistent with size and location' information.

Figure 3. The circles of projection of two different object model spheres in three views of the house scene. For each projection: the inner circle is from the size sphere, the middel circle is from the size radius, and the outer circle is from the sum of the two radii.

Future Work -----

In addition to controlling region grouping procedures, knowledge of the geometric constraints implied by object subpart relations can be used to guide the extraction of image features [10]. The three-dimensional object geometry, when combined with information about the location of object features in the image, can be used to build a three-dimensional model of the objects in the image [4]. This model, in turn, can be projected to the image to provide more speci fie information about the location of object projections in the image. Such specific information will eventually be used to control the invocation of both feature extraction procedures and procedures for the recognition of objects from two-dimensional image characteristics [6]. Work is currently underway to examine how such control can best be implemented.

Acknowledgments

Edward H. Riseman , Allen R. Hanson, Clif McCormick, and Leonard P. Wesley helped by critiquing the general concepts and specific implementations discussed in this

Figure 5. Images showing use of the geometric plan (refer to Figure 4c) on a feature matchina process: feature matching without aeometric constraints (5a), . and with geometric constraints (5b).