Extraction of Moving Object Images Through Change Detection

1. INTRODUCTION

Computer vision systems are usually composed of subsystems for segmentation and interpreta­ tion. Although motion has been suggested as a powerful cue for segmentation and has attracted increasing attention from researchers [6,7], little research has been concerned with the use of motion for segmenting images. Potter [8], and Fennema and Thompson [2], proposed schemes for segmentation using the velocity estimated at individual pixels. Jain et al.[3], and Jain and Nagel [4] showed that the image of a moving object may be extracted by using interframe differencing operations. Their scheme [4], however, has limited applicability because images can be extracted only when they have been displaced completely from their position in a specified reference frame, which is usually the first frame of the sequence.

We have deve 1 oped an approach for recovering the images of moving objects, which exploits the image transformations due to motion. This approach has been implemented on a DEC-10 in Pascal. The input to the program is a sequence of registered frames of a dynamic scene. The output is in the form of binary images, i.e., two-dimensional masks, of the moving objects, or greylevel images obtained from the input frame using the mask. This paper describes our approach. The concepts and details of the method are given in [5].

This research was supported in part by the Air Force Office of Scientific Research under grant AFOSR 77-3190 and in part by the National

Science Foundation under grant ENG 74-04986.

2. OVERVIEW

On receiving each raw image, a condensed frame is prepared [4,5]. Starting with the second frame, a difference picture, denoted DP, is prepared for each frame by comparing it with the preceding frame. Regions are found in the difference picture, and then the previous frame edge picture and current frame edge pictures are prepared. As has been shown by Jain and Nagel [4], it is possible in many cases to determine whether a DP region is due to the covering of the background, due to uncovering of the background, or due to both the covering and uncovering of the background. Let us denote these three types of DP regions as type 0, B, and X, respectively. For a given DP region, the type can be found by computing a ratio called CURPRE, which is defined as CURPRE = CC/CP

where CC (CP) is the number of points which are both extreme points of a DP region and edge points in the current (previous) frame. The extreme points of a region are the leftmost and rightmost 1 for a row, and the topmost and bottommost 1 for a column. As has been shown in [4], this ratio is much less than 1 for type B regions, much higher than 1 for type 0 regions, and near 1 for type X regions.

After the classification of the DP regions, our approach comprises two main phases which are applied to each pair of consecutive frames in the input image sequence. In the first phase, depending on the class of the DP region, a process is applied to obtain a region in the previous (or current) frame which is an estimate of the mask of the moving object image. This phase is called the recovery process. The second phase modifies this estimate of the mask by testing whether or not the motion of an object having the image corresponding to the mask could result in the transformations indicated by the difference picture. This phqse is called the refinement process .

3. RECOVERY PROCESSES

We use region igrowing for 0 and B type regions to obtain ,masks in previous and current frames. This met~od is applied to the DP regions which are either 0 or B type, since a region which is classified as one of these two types is considered to be a core mask. A region is grown by taking each nonregion pixel which has a horiz6ntal neighbor within the region and compar~ng its greylevel to that of the adjacent regipn pixel. If the pixels are similar then the ponregion pixel is added to the region. The greylevels are taken from the previous frame if the DP region is type B and from the current frame if it is type 0. A similar process ik applied to nonregion pixels which have vertical neighbors within theregion. Both processes are then iterated until no new pixels are added ~o the region. The region thus obtained represents the previous frame object mask when ~rown from a type B region and the current frame , object mask when grown from a type 0 region. l

4. REFINEMENT PRPCESSES

In this section we consider various refinement proces~es, which are used to improve the masks obtained by the recovery process. The details of these processes may be found in [5]. I

SAME OBJECT. After recovering masks in the previous and currbnt frames, for B and 0 type regions, respectively, the following method of refining masks is ! used. In this refinement, masks are modified by using the fact that a moving object has lmasks in the previous as well as current frames l Some parts of the masks may not be recovered ~ue to noise or coincidences. The same object r~finement modifies the masks to account for such lost parts.

Same object refinement is also important in processing DP regions formed by occluding objects. If region K is type 0 while regions I and J are type B and the masks of regions I and J in the previous frame have some overlap with the mask of region K in the current frame, then objects whose images do not occlude in the previous frame have occlusion in the current frame. Similarly, occlusion of objects in the previous frame may also be found.

Type X regions become important for occlud­ ing object analysis in the following case. If region K is type X while regions I and J are B type and if the masks of regions I and J in previous frames touch or overlap region K in the difference picture, then objects whose images do not have occlusion in the previous frame have occlusion in the current frame. The region K is the result of occlusion of the objects. This region is grown in the current frame to obtain a mask of the composite object in the current frame. In such a situation, if the regions I and J are type 0, then region K is grown in the previous frame.

TERMINATION REFINEMENT . Normally for a rigid object moving completely in the field of view, difference entries result at both its leading and trailing ends. This fact may be utilized for refining those regions which have been incorrectly grown by including non-object points {those points which do not belong to the object image) in the object mask during the region growing process.

GAP FILLING. For a variety of reasons the object masks resulting from the recovery processes may have holes or significant concavities. \~e use "gap filling" to check to see if such areas are representative of the object image or just side effects of the recovery processes when applied to the particular object shape and motion.

SAME FRAME REFINEMENT. When analyzing a sequence of frames, say frames F1,F2, ... Fn, a difference picture is prepared for each successive pair of frames {i.e., F1 and F2, F2 and F3, F3 and F4, . . . , Fn-1 and Fn) . The object masks are extracted in each frame. For each frame, F2 through Fn-1, object masks are extracted twice. For a moving object, the two masks extracted at different times but from a given frame should be the same. Due to noise and inexact recovery processes, this may not be the case in most practical situations . However, the fact that the masks are of the same object in the same frame is exploited for refining the mask.

5. RESULTS

Due to space limitation, we present the results for only one input sequence. The sequence shows the motion of two blocks in a scene containing many other objects. In Figs . 1, 2 and 3 we show three frames from the sequence taken in our laboratory. The difference picture for the first two frames i s shown in Fig. 4, while the region types and CURPRE values are given in Table 1. The object masks recovered by the program are shown in Figs. 5 and 6 for the previous and current frames, respectively. After processing the third frame of the sequence, the program can apply the same frame refinement to yield the masks of the moving objects for the second frame. The moving objects start occluding in this frame; thus a single mask is formed for the composite object, as shown in Fig. 7. The image of the objects is obtained from this mask as presented in Fig. 8. Note that all other objects have been neglected as they are stationary.

6. CONCLUSION

The present version of our program was implemented to support our belief that the transformations in frames of a scene due to motion may help in the recovery of the images of moving objects. The results of the current evolutionary stage of the program indicate that it is possible to recover the images of moving objects using only low level knowledge, even from the frames of nontrivial scenes.

The processes implemented in the current version are based on the knowledge derived from simple idealized scenes as presented in [4,5]. It is clear that a deeper understanding of the image transformations due to motion will be useful in developing better recovery and refinement processes.