Structural Analysis of Complex Aerial Photographs

1. INTRODUCTION

Recently several systems for the analysis of aerial photographs have been developed to locate objects on the ground surface[l],[2],[3]. When we are going to get a description of the structure on the ground surface by analysing an aerial photograph, we find several difficulties which are not encountered in other image analysis areas:

1. Tht· size of a picture is very large.
2. As it is impossible to control photographing conditions, the quality of a picture is apt to change.
3. The sizes and the textural properties of objects vary quite widely.
4. Variation of objects in a scene entails calcu­ lation of many different fea tures and the diverse knowledge of the world.
5. There are too many situations on the ground surface to build a general model which can represent all possible spatial arrangements of objects.

In ordt·r to overcome thPse diffi culties and to realize nn efficient and reliable analysis, we havl" developed a nl'w system based on the ' production systcm'[4]. ThP system consists of a group of ··hject-detect ion subsystems which individually search for specific objects by communicating with each other via a common 'black­ board' [ 5], [ 6].

The analysis process of this system is divided into the following steps.

1. Segmentation: After noise removal, an aerial photograph is segmented into element­ ary regions according to the multispectral property.
2. Global survey of the whole scene: Regions with characteristic properties are extracted. These are used to confine the spatial domains of objects where object-detection subsystems work.
3. Detailed analysis of focussed local areas: Each object-detection subsystem focuses its attention on a specific local area and checks the existence of a specific object.
4. Communication among object-detec tion sub­ systems: All the information about the properties of r egions and recognized objects is stored in the blackboard. Object-detect­ ion subsystems interface with it in a uni­ form way. The system controls the overall flow of the analysis by managing the infor­ mation in the blackboard. It solves conflict among object-detection subsystems, and corrects Prrors by backtracking.

This paper mainly describes the control struc­ tur e of the system, and demonstrates its per­ formance with experimental results. The detailed algorithms of picture processing routines are described in [7].

2. FOCUS OF ATTENTION IN THE ANALYSIS OF AERIAL PHOTOGRAPHS

Aerial photographs we want to analyse are 4 band multispectral images of complex suburban areas taken from the low altitude(Fig. 1).

As the size of a picture is very large, the analysis should be done as efficiently as possi­ ble. If each object-detection subsystem, which is very sophisticated and time-consuming, were applied to the whole picture area, it would take a prohibitive .time to complete the analysis. In order to solve this problem, we adopted the focussing mechanism which human being seems to use when he interprets a complex scene. When he sees a scene, at first he globally surveys it to find the characteristic areas which attract his interest. Then, he goes into the detailed examination of some local area to find out objects using his knowledge of the world. The more intensively he focuses, the more specialized knowledge he comes to use.

In our system, several kinds of regions with prominent features(characteristic regions) are extracted by simple picture processing programs. Then, object-detection subsystems apply sophis­ ticated programs only in the local areas where specific objects are supposed to exist. As the time-consuming processings are applied only in small local areas, the total processing time can be reduced very much. Fig.2 shows the sche­ matic drawing of this focussing process.

In order to specify the spatial domains of objects as correctly as possible, we utilize such features as size, shape, brightness, multi­ spectral properties, · texture and spatial relations among regions. We extract seven types of char­ acteristic regions, i.e. large homogeneous regions, elongated regions, shadow regions, shadow-making regions, vegetation regions, water regions and high contrast texture regions.

All the features used here are stable against the change of photographing conditions, and all parameters are automatically and adaptively determined from the picture data under analysis. Therefore extracted characteristic regions can be thF reliable basis for the subsequent detail­ ed analysis.

Fig.3 shows the characteri~~~c regions extracted from the picture of Fig.l.

3. OBJECT-DETECTION SUBSYSTEM

An aerial photograph contains a variety of objects such as crop fields, forest areas, grass lands, roads, rivers, houses and so on. The diverse knowledge should be incorporated to describe the structure on the ground surface. In addition, these objects,especially in a small country as Japan, are intricately arranged with­ out definite spatial relationships.

Taking these conditions into account, it seems to be natural to divide the system into a group of object-detection subsystems. Each of them is designed to find out specific objects using the knowledge of their intrinsic properties and the environments in which they are embedded. The diverse knowledge of the world is distributed in object-detection subsystems. This facilitates the implementation of the system.

The type of object-detection subsystems can be classified into two categories according to the information they use in selecting the candidate regions of objects.

(1) Picture data-driven subsystem

r~arse Description Detailed Analysis

Fig.2 Focussing mechanism of the system

The subsystems of this type check the ex­ istence of the local areas with specified properties by combining characteristic regions. If there are such local areas, they examine them in detail by activating special purpose feature extraction programs.

(2) Model-driven subsystem

The subsystems of this type pick up the candidate regions by using the spectral and spatial relationships with already recog­ nized objects. For example, it is very diffi­ cult to detect cars by a simple knowledge that they are rectangular unless we impose the condition that they are on the roads or the parking lots. Therefore the subsystem for the detection of cars entails the recog­ nition of roads or parking lots.

The present system has thirteen object-detection subsystems for eight kinds of objects: crop field , bare soil(crop field without plantation), forest, grass land, road, river, car and house. Fig.4 shows intermediate results of detection of these objects.

4. THE STRUCTURE OF THE BLACKBOARD

Each object-detection subsystem interfaces with the common data base, the blackboard, in order to test conditions for activation and to write

(a)

in the result of the analysis. All the communi­ cations among object-detection subsystems are made via this blackboard. Fig.S shows the sche­ matic drawing of the structure of the blackboard.

The blackboard contains global parameters as well as the properties of regions and objects. They denote the global properties of the picture data, that is, photographing conditions of an aerial photograph such as the direction of the sun, threshold values for checking the similari­ ty of multispectral propreties and so on. From these global parameters, each object-detection subsystem gets the information of the quality of the picture data under analysis. Therefore, it can successfully find out objects in spite of unstable photographing conditions of aerial photographs.

In this system, elementary regions, which are segmented according to the multispectral proper­ ty in segmentation, are considered as the basic units fnr all hi~her level processings. Their fundamental properties( the average gray level in each spectral band, size, location and some basic shape features) are stored in the prop_erty table in the blackboard(Fig.S).

We store LABEL PICTURE in the blackboard in order to denote the spatial relationships among elementary regions. It is the symbolic picture where each pixel in an elementary region is labeled with the same unique region number( Fig.6). The reason for this is:

The spatial relationships among regions used by object-detection subsystems are very different and depend on the properties of objects they want to find out. Therefore it is not economi­ cal, and is even sometimes impossible to calcu­ lat~ all spatial relationships in advance. For example, when we want to estimate the location of a house by using the regurality of the spatial arrangement of already recognized ones, it will take a long time to find a candidate region of a house if we do not use the two­ dimendional image.

As each pixel in LABEL PICTURE contains an unique region number,we can easily get the properties of the region to which it belongs. On the other hand, the location of a region in LABEL PICTURE iq denoted by the two coordinate pairs,(BX,BY) and (EX,EY), in the property table(Fig.7). When one wants to calculate a new feature of a region, one only has to scan within the rectangle area specified by these ~oordinate pairs(Minimum Bounding Rectangle). This is very useful to save the processing time of picture processing programs.

Each characteristic region in the blackboard is represented by a characteristic region node which denotes a group of elementary regions. Two-di­ mensional images of characte~istic regions can also be constructed from LABEL PICTURE. These are very useful in loca ting clusters of ele­ mentary regions such as forest and residential areas.

When each object-detection subsystem recognizes an object, it generates an object node in the blackboard, and the object node and its consti­ tuent elementary regions are connected by part/ whole relations. If the recognilion of an objec t depends on the properties of already recognized ones, the node of a new recognized object is ·

number.

linked with those of old ones(Fig.S). By storing the history of the recognition process, the system c,·,· give back the state of the blackboard to the correct one when it finds out an error in it.

5. CONTROL MECHANISM OF THE SYSTEM

As each objec t-detection subsystem recognizes objects independenLly of the others, the system incorporates the mechanism to solve contlicts between these subsystems.

The property table has the field where each object-detection subsystem returns one of the following recognition statuses: "unanalysed", "recognized", "irregular shaped" and "rejected". the system checks this field, and if some ele­ mentary region is recognized as different objects by multiple subsystems, it evaluates the relia­ bility value of each object to which that region belongs. Then the objects nodes except the most reliable one are deleted, and the corresponding recognition statuses of the region are changed from "recognized" to "rejected".

If there are other objects which have been recog­ nized in connection with rejected objects, the system also deletes those object nodes by tra­ versing the dependency links from rejected objects. In this case, the recogniL ion status!~S of their constituent regions are given back to "unanalysed", tor they 111ight be recognized by using the properties of the other objects of the same kind.

Errors in segmentation are also repaired by the system. Object-detection subsystems checks vari­ uus properties of regions to recognize objects. They returns the recognition status, "irregular shaped", if the shape of a region is not suitable for an object while all the other properties are satisfactory. Then, the system re-analyses that region by applying the split/merge program. If the region has the bottle-neck as shown in Fig. 8, it is divided into small regions. If the boundary of the region is very rough and irregular (Fig. 9), neighboring small regions with the similar color are merged with it. These newly generated regions are added in the blackboard as temporary regions and analysed by object-detection subsystems. If a temporary region is successfully recognized, it is registered as a new elementary region in connection with its related object, and the original region is deleted from the blackboard. If the result of the recognition of the new region contradicts with that of the original one, the system deletes one of them depending on the reli­ ability. When the temporary region is not recog­ nized, it is removed from the blackboard, and the corresponding recognition status of the origina l region is changed to "rejected".

In the case of Fig. 8, the bottle-neck results from the error of mismerging two adjacent houses. When this region is splitted into two regions, the house detection subsystem comes to recognize them successfully. The region in Fig. 9 corre­ sponds a crop field, but it is not recognized because of the irregularity of the boundary. After the merging process, it comes to take the smooth boundary, and is recognized as a crop field.

By the benefits of the above-mentioned control mechanisms of the system, each object-detection subsystem can use the properties of already rec­ ognized objects without considering the results of recognition by the others. The system stops the analysis when no new objects are recognized.

Fig. 10 shows the final result of the analysis. Although white regions with .no marks are left unanalysed, we can see that almost all objects which belong to clear semantic categories are successfully recognized. (As we do not have the ground truth data, the evaluation of the result is done by visual inspection.)

6. CONCLUSIONS

The system for the structural analysis of com­ plex aerial photographs has been presented. The major conclusions on this system are the follow­ ing:

1. The focussing mechanism realizes the ef­ ficient analysis and successfully isolates objects. Table 1 shows the efficiency of this focussing mechanism.
2. The production system architecture gives us the following benefits.

1. Modularity : Diverse knowledge required to describe the structure of the ground surface can be individually implemented in object­ detection subsystems.
2. Model-driven analysis: We can use the proper­ ties of already recognized objects in order to analyse unrecognized areas and to find out contextual-dependent objects.
3. General control mechanism: The system takes the full responsibility for the maintenance of the blackboard. It solves the conflicts among object-detection subsystems and corrects the errors in segmentation. If necessary, it undoes the content of the blackboard to remove the effects of errors. These mechanisms of the system integrate mutually independent object-detection sub­ systems.

This system is implemented on a large computer, FACOM M-190. The total CPU time was about 150 sec. in the picture of Fig.l. Experiments on several different aerial photographs have shown that this system works fairly well.

Time denotes the processing time for the crop field detection subsystem to calculate straightness of region boundaries. As we focus only on large homogeneous regions, we can save process­ ing time very much.

• MBR denotes a minimun bounding rectangle(see Fig.7)

~.r~~~~~~--~~~~~--~~· White regions without mark are un­

Fig.lO Final result of the analysis:

recognized(20% of the whole picture area)

BS: bare soil

(crop field without plantation)

CF: crop field

G : grass land

F : forest

RD: road

R roof of a house

C car

r misrecognized roof

(these regions have similar multi­ spectral properties to that of true roof)