Experiments in Map-Guided Photo Interpretation

Introduction

A central scientific goal of image understanding research is to investigate ways of bringing diverse sources of knowledge to bear on the problem of interpreting images. An important source of knowledge used by humans in the interpretation of aerial photographs is the topographic map. This paper describes some initial experiments in the use of map knowledge to automate an important class of photo-interpretation tasks involving monitoring and change detection. This work is part of a larger image understanding research program aimed at developing collaborative aids for photo interpreters and cartographers [Barrow 1977a].

Photo interpreters spend a significant proportion of their time in routine monitoring tasks such as counting box cars stored in railyards or keeping track of the ships berthed in a harbor. A major difficulty in automating such tasks is locating the monitoring sites in sensed imagery, which may be taken from arbitrary viewpoints. Once the location is known many monitoring tasks are reduced to simpler detection problems that can be solved by straightforward techniques such as template matching.

Map-Image Correspondence

To locate monitoring sites it is necessary to put the sensed image into correspondence with reference imagery or a map. Image matching (by techniques such as cross-correlation) is a computationally expensive process and works only when the reference and sensed images are obtained under similar viewing conditions. To overcome these limitations, we have developed a new approach, parametric correspondence, for matching images directly to a three-dimensional, symbolic reference map.

The map contains a compact three-dimensional representation of the shape of major landmarks, such as coastlines, buildings, and roads. An analytic camera model is used to predict the location and appearance of landmarks in the image, generating a projection for an assumed viewpoint. Correspondence is achieved by adjusting the parameters of the camera model until the predicted appearances of the landmarks optimally match a symbolic description extracted from the image. A complete description of parametric correspondence is given in a companion paper [Barrow 1977b].

Map-Guided Monitoring

Having placed the image into parametric correspondence with the 3-D map, it is possible to predict the image coordinates of any feature in the map, and conversely, to predict the map features corresponding to any point in the image. Given this capability, a monitoring task can be performed as follows: determine the image coordinates for a reference structure (such as, a railroad track, ship berth, or road) and apply a special-purpose operator to detect objects of interest (such as, box cars, ships, or cars). For example, we have implemented a box car counting routine that flies statistical operators along a track to hypothesize possible ends of box cars. The hypotheses are combined with a knowledge of standard box car lengths and the characteristics of an empty track to locate the gaps between box cars. The program then reports the number of cars, classified by length [Barrow 1976). For ship monitoring, the berth locations from the map are projected onto the image and It is important to keep in mind that a map is only an approximation to reality: it may be incomplete, be out of date, suppress details, or contain errors. Therefore, it is sometimes necessary to locate the image coordinates of objects more precisely than can be predicted using the map and a calibration. Routines are needed that can take predictions and verify them in the image. As a first step in that direction we developed a guided line tracing routine that accepts a rough approximation to the path of a linear feature, such as a river or a road, and extracts a better estimate for its path in the image. The routine applies a specially developed line detector in the vicinity of the approximate path, and then finds a globally optimal path based on the local feature values [Barrow 1976).

Discussion

Given precise estimates for the image coordinates of key objects, many previously intractable monitoring tasks become feasible, and in some cases, even easy. For example, knowing the layout of railroad tracks transforms the box car counting task into essentially a one-dimensional template matching problem. We have implemented three representative demonstrations of this approach and believe that many others are possible, especially in remote sensing applications such as pollution monitoring and forest fire detection [Tenenbaum 1976].

The demonstrations, to date, have emphasized one major source of knowledge; the map. The primitive descriptions employed for object recognition are adequate only for straightforward cases and can easily be fooled by the unexpected. Work remains at both the developmental and research le.vels to achieve the long range goal of completely automated photo interpretation. The demonstrations do, however, show the potential of bringing image understanding and artificial intelligence approaches to bear on problems in photo interpretation.

Acknowledgments

This work was supported by ARPA under contract DAAG29-76-C-0012. Additional support was provided by

NASA under contract NASW-2865.