A Spatiotemporal/Spatiotemporal-Frequency Interpretation of Apparent Motion Reversal

1 Introduction

In temporal aliasing, components of the image sequence with high temporal frequency appear at lower frequen­ cies due to an insufficient temporal sampling rate (com­ monly referred to as the frame rate). These components affect the visual quality of the sequence in two respects. ~he ?rst, most noticeable at low frame rates, is a disruJ>­ twn m the smoothness of perceived motion (a breakdown of the apparent motion illusion underlying all motion pi~ture display methods). Although predicting the sam­ pling rate at which this effect will become visible is not trivial, the effect itself is intuitive - the motion appears nonsmooth because too few samples are presented. The second, which may be seen at comparatively high frame rates, is a distortion of the direction and speed of the perceived motion. This distortion is common in current film and video, and manifests in connection with ob­ jects that exhibit both significant high-spatial-frequency components (e.g., the spokes of a wheel) and that move wit? relatively high speed (the wheel is turning relatively rapidly). In the classical manifestation of this effect a rotating wheel is seen to reverse its direction of rotation as the rate of rotation increases - the "wagon wheel ef­ fect." This effect is not so intuitive, and to the author's knowledge the connection between aliasing and apparent motion reversal, while observed, has not been explained in the literature.

The sel~tion of an appropriate temporal sampling rate for a grven application is of significant practical im­ portance. A broad range of rates are in use, spanning from _the very low rates (10 frames per second or less) used m current visual communications systems, through the moderate 24 - 30 frame per second rates used in an­ imation, film, and standard television, to 60 frames per s~ond in high definition television. Much higher sam­ plmg rates are used in scientific applications (hundreds and .eve~ tho~ands of frames per second), and are be­ commg mcreasmgly common and economically viable as technology improves. For sequences involving high de­ grees of motion (particularly when detailed spatial struc­ ture ~ust also be represented), temporal aliasing is the most Important phenomenon in determining the tempo­ ral sampling rate 1 , providing an additional motivation to understand it fully.

Motion is most intuitively a spatiotemporal phe­ nomenon. However, it has been shown that it can also be characterized in the frequency domain via Fourier analysis. [Watson and Ahumada, 1983] have used this approach to investigate the relationship between frame rate, the .bandwidth of the human visual system, and the perceived smoothness of motion for a moving line stimulus. Their results clearly demonstrate the value of frequency domain analysis for understanding aspects of visual motion perception.

In this paper, we consider the manner in which aliasing affects sequences at comparatively high frame rates, where motion is generally perceived as smooth yet distortions of speed and direction may be visi~ ble. A method is demonstrated by which motion esti­ m~tes c~nsistent with those perceived visually are ob­ tamed, m cas~ ?oth with and without aliasing, for sequences consistmg of regions with different motions (a task that cannot be undertaken with conventional ~ourier analysis). To do so, we utilize a generaliza­ tl~n. of fre~uency domain motion analysis, based on a JOint spatwtemporal/spatiotemporal-frequency representation (the 3-D Gabor transform).

1 Note that we distinguish here between sampling and screen ~pdate r~tes, the later being driven primarily by the pe~ception of w1de area flicker. Screen updates need not be umque samples, a fact reflected, e.g., in current film to video conversion practice.

2 Motion and Aliasing in the Frequency Domain

The frequency domain characterization of motion has been studied for some time, and for certain situations is well understood. In the case of an initially static image undergoing translation with constant velocity, motion is particularly straightforward to describe in the frequency domain. The spectrum of the sequence lies in an oblique plane, the slope of which indicates the velocity of the motion.

Consider a simple 1-D continuous sinusoidal "im­ age" with spatial frequency Uo, moving with velocity r: f(x, t) = sin(uo(x - rt)). When r = 0 (the image is static), the Fourier transform of the sequence consists of two components at the spatial frequencies ±uo. As r increases, the temporal frequency coordinates of the two components change from zero to =fru0 , respectively (see Figure 1). The slope of the line connecting the two components is -r.

As is well known from elementary sampling theory, if the sinusoid is spatially and temporally sampled, the spectrum described above is scaled and replicated on centers determined by the sampling rates. We will as­ sume the spatial sampling rate is sufficient, and focus on the temporal sampling. As illustrated in Figure 2, for a given temporal sampling rate W8 , the sequence can be either oversampled, critically sampled, or undersampled depending on whether r is less than, equal to, or greater than w 8 j2u0 . Temporal aliasing occurs when the compo­ nents of the replicated spectra occur at frequencies less than or equal to w8 j2 (the later two cases). Again this is well known, but sampling theory does not predict the visual impact of this aliasing.

Consider these two cases from the standpoint of the apparent motion represented (the slopes of the line or lines connecting the components of the spectra with tem­ poral frequency less than W 8 j2), however. In the criti­ cally sampled case there are in fact two lines, with slopes of ±r, indicating simultaneously motions of the same speed but opposite directions. In this case, one would ex­ pect the sequence to appear essentially static (although a "jitter" or "flashing" might be seen). In the undersam­ pled case there is only a single line, with slope depending on r and the degree of undersampling. For moderate un­ dersampling the sign of the slope is positive indicating motion in the direction opposite to that of the original sequence. This is the basis of the ''wagon wheel" illusion.

The above simple example might lead one to ex­ pect that temporal aliasing and its effects on perceived motion can be completely characterized using Fourier techniques. However, in sequences of practical interest (which may include multiple objects in independent mo­ tion) this is not the case. Aliasing is a local phenomenon, and the artifacts associated with aliasing are restricted to regions of the sequence which are insufficiently sam­ pled. In practice, these regions typically correspond to objects or surfaces exhibiting high spatial frequencies, that are also moving at relatively high velocities. Al­ though aliasing is certainly reflected in the Fourier trans­ forms of sequences of this type, the connection between the aliased spectral components (or indeed any of the spectral components) and the spatiotemporal locations of the associated regions or objects cannot in general be made. For this reason, the Fourier transform cannot be used for the analysis of motion in complex sequences.

To identify the locations and motions of objects, fre­ quency analysis localized to the neighborhoods of the objects is required. Windowed Fourier analysis has been proposed for such cases [Gafni and Zeevi, 1979]. However, the accuracy of a motion analysis method of this type is highly dependent on the resolution of the underlying transform, in both the spatiotemporal and spatiotemporal-frequency domains. It is known that the windowed Fourier transform does not perform particu­ larly well in this regard. Filterbank-based approaches to this problem have also been proposed, e.g. [Fleet and Jepson, 1990], [Heeger, 1987]. A shortcoming of these approaches is the lossy nature of the proposed filter­ banks, which can introduce a bias in the motion esti­ mates.

There are a variety of alternative methods for local frequency analysis beyond the windowed Fourier trans­ form. Examples include the Wigner distribution (a bilin­ ear local frequency representation) and the Gabor trans­ form (which is linear). Because they can provide a large degree of spatiotemporal locality and spatiotemporal­ frequency resolution simultaneously (within the bounds of uncertainty), the use of these techniques is particu­ larly promising for frequency-based motion analysis with multiple motions. The use of the Wigner distribution for this task was examined in [Jacobson and Wechsler, 1987]. However, the bilinear nature of the Wigner dis­ tribution (and the attendant cross terms produced by multicomponent signals) can make motion analysis dif­ ficult, in practice. A motion analysis technique based on the Gabor transform has recently been demonstrated [Reed, 1997]. We will briefly described this approach in the following section, and use it to investigate apparent motion reversal in Section 4.

3 Motion Analysis using the 3-D Gabor Transform

The Gabor representation was first introduced for time­ frequency analysis [Gabor, 1946]. In this representation, the signal of interest is expressed as a weighted sum of basis functions formed by the products of shifted ( usu­ ally Gaussian) windows and complex exponentials. The relative popularity of this representation has been due in part to the good spatial and spectral localization prop­ erties of the Gabor functions. It has also been demon­ strated [Mareelja, 1980], [Daugman, 1985], [Webster and De Valois, 1985], [Field and Tolhurst, 1986], [Jones and Palmer, 1987] that the 2-D Gabor functions agree rea­ sonably well with receptive-field profiles measured for simple cells in the cat striate cortex.

An image sequence can be considered a 3cdimensional (spatiotemporal) volume of data. Extending the Gabor representation to 3-D, this volume can be represented as the weighted sum of 3-D Gabor functions of the form

where

is a 3-D Gaussian function, (Jx, (JY and (Jt determine the scale of the Gaussian along the respective axes, (xo, Yo, to) is the center of the function in the spatiotem­ poral domain, and (Uo, vo, wo) is the center of support in the spatiotemporal-frequency domain.

A complete basis can be formed using the Gabor func­ tions, resulting in an invertible transform (the Gabor transform). In the discrete case, for an image sequence with spatial dimensions N by M and P frames in length, N · M · P basis functions are required. The sequence can then be expressed at each discrete point (xm, Yn, tp) as

where J · K · L · P · Q · R = N · M · P for completeness, 9xq,yr,t.,u;,vk,w1 (xm, Yn, tp) denotes the Gabor basis func­ tion with spatiotemporal and spatiotemporal-frequency centers of (xq, Yr, t 8 ) and (uj, Vk, Wt) respectively, and Cxq,Yr,t.,u;,vk,w1 is the associated coefficient, which is gen­ erally complex. There is substantial freedom in selecting the locations of these basis functions, while maintaining completeness. Largely for computational reasons, in this work the functions are centered on a regular cubic grid.

Because the Gabor functions are not orthogonal, the Gabor transform coefficients cannot be calculated by simply computing the inner products of the basis func­ tions and the signal to be transformed (or, equivalently, by convolving with the basis functions and subsampling).

Due to substantial interest over the past several years in the computation of this transform, there are a number of alternative methods available. The method used in this work is a 3-D extension of the algorithm first reported in [Ebrahimi et al., 1990].

From the 3-D Gabor transform of an image sequence, the motion parameters can be estimated at each spa­ tiotemporal location by fitting the surface representing the spectral signature of the motion to the local spec­ trum. In the case of uniform translational motion, the slope of the planar spectrum is sought, yielding the mo­ tion vector r. There are a number of ways in which this can be done.

A straightforward approach to estimating the slope of the local spectra, used in the examples which follow, is to form vectors of the u, v, and w coordinates of the basis functions that have significant energy (magnitudes exceeding a threshold) for each point in the sequence at which basis functions are centered. The motion vector and the coordinate vectors u, v, and w at each point are related as

where A= (ulv). An LMS estimate of the motion vector at a given point can then be found using the pseudoin­ verse of A:

4 Results

To examine the effects of temporal aliasing, we will first use a simple test sequence in which each frame includes two fields, consisting of both horiwntal and vertical si­ nusoids with frequencies of rr /4 and rr /2 radians/pixel respectively, one field above the other. The sinusoids are scaled and offset so that all pixel intensities fall in the range 0 to 255. The sequence is 24 frames in length, with 256-by-256 pixels per frame. As the sequence pro­ gresses, the top and bottom fields move to the right at the same rate. We will consider three rates of transla­ tion: 1, 2, and 3 pixels/frame. The first three frames from the 3 pixel/frame sequence are shown in Figure 3. The top field is oversampled for all three velocities. The bottom field is oversampled, critically sampled, and un­ dersampled, respectively.

We next compute the Gabor transform of the se­ quences, using a complete basis on 8 pixel centers in spac~time, with a 3.5 pixel offset in each dimension, spaced rr /4 apart in spatiotemporal- frequency, and with u., = Uy = <Tt = 1/8. Computing the slope of the plane which best fits the local spectra for each sequence, the motion estimates shown in Figure 4 result for the point in time between frames 12 and 13 of the sequences. Note that the estimates are located between frames because the basis functions used in the transform are centered between frames. The arrows in the figure are scaled to the maximum velocity for each case. Similar results are obtained between frames 4 and 5, and frames 20 and 21.

As shown to the left in Figure 4, for a translation of 1 pixel/frame where both fields are oversampled, the mo­ tion estimates for the two fields are correct and ident ical (with the exception of some edge effects). For 2 pix­ els/frame (Figure 4, center), the estimate for the upper field (which is still oversampled) remains correct, while the motion estimate for the lower field is zero. In this case, the lower field is critically sampled. Visually, this field appears to "flash" or "jitter", but not to translate, which is consistent with the motion estimate. In the third sequence (Figure 4, right) , the motion estimate for the (oversampled) upper field is correct (3 pixels/frame to the right). The estimate for the lower field, which is now undersampled, is 1 pixel/frame in the reverse direc­ tion, exhibiting the apparent motion reversal discussed above. Visually, the lower field appears to move to the left, consistent with the estimate.

We next consider the sequence shown in Figure 5, 24 frames in length, with 256-by-256 pixels per frame. The background is static, consisting of a sinusoidal "plaid" field with horizontal and vertical frequencies of rr /4 ra­ dians/pixel. Superimposed on this background are two blocks, 80 pixels square, starting in the upper and lower left of the frame. The first has spatial frequency com­ ponents identical to the background, while the second has horizontal and vertical frequency components of rr /2. The sinusoidal plaids are each scaled and offset, so that all pixel intensities fall in the range 0 to 255. As the sequence progresses, the two blocks move to the right at 3 pixels/frame.

The 3-D Gabor transform of the sequence was com­ puted as in the previous example. Computing the LMS estimate described in equation 4 the motion estimates shown in Figure 6 result, for the points in time between frames 4 and 5, 12 and 13, and 20 and 21,respectively.

This example illustrates two points. Viewed as ob­ jects, the surface characteristics of the blocks lead to correct motion estimates for the interior of the upper block (consistent with the object motion), but incorrect (reversed in direction and reduced in speed) for the in­ terior of the lower block. This is exactly as found for the two fields moving at 3 pixels/frame in the previous example. However, as is clear by examining Figure 6 from left to right, the motion of the blocks themselves remains correctly represented. They both move from left to right, retaining the same relative position. This is the same behavior observed in the classical ''wagon wheel" illusion, where the wheel appears to reverse its direction of rotation and to rotate at a lower rate, but as a whole continues to move in the proper direction at the proper speed.

5 Conclusions

In this paper we have investigated the mechanism un­ derlying apparent motion reversal in image sequences which exhibit temporal aliasing. We have demon-

................................

·····················~·········

. ............................. .

········~······················

-··········

···················~· .. ·······~·

Figure 5: From left to right: frames 5, 9, 13 and 21 (of 24) from the second test sequence.

strated analytically and experimentally that this phe­ nomenon can be both understood and predicted us­ ing spatiotemporal/ spatiotemporal-frequency represen­ tations. Using a motion estimation procedure based on one such representation, the 3-D Gabor transform, mo­ tion estimates consistent with those perceived visually were obtained in cases with and without aliasing. Fi­ nally, it was shown that overall object motion informa­ tion is preserved using a procedure of this type, even when the surface properties of the object induce appar­ ent motion reversal over the object surface. This is just as observed in the classical "wagon wheel" illusion.

Acknowledgments

This work was supported in part by the Computer Vi­ sion Laboratory, Department of Electrical Engineering, Linkoping University, Sweden.