Spectral Continuity and Eye Vergence Movement

1. Introduction

The structure of the physical world is constrained by physical laws, which in turn constrain images depicting it. The brain exploits these underlying constraints when it builds its world model from the images it receives from the eyes. An interesting conjec­ ture is that it is able to do this with little or no a priori knowledge of semantic content. In this sense, early visual processing is a con­ structive process of description and is independent of recognition [1] . Perhaps the best example of the success of this approach is the Marr-Poggio model of human stereopsis, and its subsequent com­ puter implementation by E.L. Grimson [2,3]. Marr and Poggio pro­ pose that matehing is conducted on zero crossings of the \l 2 G con­ volution of the left and right eye iiJlage. The results of matehing zero crossings from low spatial frequency tuned channels are used to guide vergence movements which bring zero crossings from high spatial frequency tuned channels into their smaller range of correspondence. The vergence control mechanism is local, based on the success or failure of matehing in local neighborhoods of each image.

The principal motivation behind this study is the desire to model the control of eye vergence movement in a biologically plau­ sible manner. Grimson's implementation of the Marr-Poggio theory uses the results of matehing within local areas of a channel to decide whether or not vergence movement is necessary to bring zero crossings within those local areas into correspondence. Local areas with less than ·70% matches are declared out of range, and require vergence movement. The cross channel coupling is through the 21!! -D sketch, which stores the results of matehing from the low fre­ quency channels. When a local area is out of range, these results are consulted to determine whether a convergent or divergent movement should be initiated.

However, there is an inherent difficulty with this control stra­ tegy, since vergence movement is a resource of limited access. Ver­ gence movement is a physical process; the eyes are unable to make both convergent and divergent movements simultaneously. The single access resource of eye movement must however, satisfy the conßicting requirements of hundreds of different local areas within a single channel. The problern is further complicated because requests for eye movement can come from any local area of any between this proposal and Mayhew and Frisby's proposal that "the patterns of between-ehannel eorrespondenee ... also help disambigu­ ate within-ehannel fusions." Their program, FRECKLES, uses a eross ehannel deseription as an enriehed matehing primitive, allow­ ing disambiguation of false targets to oeeur over a !arger fusional area. The disambiguating power aetually seems to stem from a form of compatibility constraint; matehes are forbidden between eross-ehannel deseriptions that eould not have arisen from the same physieal phenomenon [1].

In eontrast, this paper proposes that a requirement of eon­ tinuity of disparity in the frequeney domain is used to seleet the optimal eross ehannel eombination of matehes obtained indepen­ dently within eaeh spatial frequeney tuned ehannel. The intra­ frequeney matehing is eondueted in essentially the same manner as in the Marr-Poggio model, and like the Marr-Poggio model, the ehief disambiguating power lies in the faet that the size of fusional area is restrieted to the small interval in whieh false targets ean be resolved by loeal neighborhood support [1,4]. Global disambigua­ tion depends on the seleetion of the optimal eross ehannel eombina­ tion during the eourse of vergenee movement.

That there is an optimal eross ehannel eombination of within ehannel matehes follows from the fact that for eaeh spatially loeal­ ized area of the left or right eye image there will be a speeifie ver­ genee fixation whieh will preserve speetral eontinuity optimally (Fig. 1). Sinee these matehes are assoeiated with a partieular pri­ ma! sketeh token, the raw prima! sketeh provides a frame of refer­ enee for the maintenanee of disparity values aeross vergenee move­ ments. The solution to the problern of updating the 2'a -D sketeh without explieit knowledge of eye position is thus implieit in the solution of the problern of eorrespondenee in time between tokens of the prima! sketeh, a problern examined extensively by Ullman [9].

This vergence strategy also has the advantage of eliminating the problern Grimson's program had solving images with periodie features [2,3]. The problern arose because the criterion for deter­ mining whether or not a vergenee movement would be initiated was of a purely local nature. Any area of a channel with more than 70% matehes would not initiate a vergence movement, irrespective of the results of matehing within spatially coincident areas of lower frequency channels. Thus, different initial vergence positions could produce different matehing results. By requiring that disparity remain relatively eonstant for a given feature through the frequency domain, the more local matches of higher frequency channels are forced to agree with the more global definition of disparity provided by lower frequency channels. As long as the lowest frequency chan­ nel is of lower spatial frequency than the period of the repeating pattern (windows on an office building in the case of Grimson's pro­ gram) correct disparity values will be assigned.

3. lmplementation

The program proceeds by first extraeting zero erossings for both eyes at each spatial frequeney . This is accomplished by traversing each horizontal raster looking for sign ehanges in the \lG profile. The eurrent implementation uses two spatial frequency tuned ehannels. Each channel is processed indepen­ dently, the low frequency channel first (in the human system, intra-frequeney matehing would be eonducted concurrently within all channels). Vergence movement is simulated by incrementally inereasing the displacement between the left and right images by an amount equal to half the fusional area of the high frequency channel (the actual amount is not critical). This has the effect of moving the plane of fixation through the disparity range deter­ mined by the matehes of the low frequeney ehannel. Matehing is repeated at each fixation, only matehes yielding disparities eon­ sistent with those of the lower frequeney being aeeepted. For the two ehannel implementation, minimal spectral continuity is also optimal spectral continuity.

Matehing proeeeds within a frequency by first forming "com­ petition" matrices from the cross produet of like signed zero cross­ ings from corresponding rasters of the left and right images (Fig. 2). Zero erossings from the right eye form rows while zero erossings from the left form eolumns. All potential matches are explicitly represented by a unique position in the competition matrix and associated with eaeh row and eolumn is the x-eoordinate corresponding to the location of the zero crossing along the raster of the appropriate image. A disparity is assigned to eaeh potential mateh by subtraeting the X-coordinates of all rows and columns. Those potential matehes having disparity smaller than 20' for the channel being processed are marked as targets.

Targets are ambiguous when more than one target is present in any given row or column. Two targets in the same column represent ambiguity to the left eye, while two targets in the same row represent ambiguity to the right. Every raster is first processed for unambiguous matches and the results stored in a buffer so that the pulling effect, as in the Marr-Poggio model, may use these results to resolve ambiguous targets. Before a mateh from a high frequency ehannel is aceepted, the majority disparity within a circular region surrounding the spatially coincident area of the low ~requ~nc;y channel is calculated. Only those matches having dispar1ty w1thm a small f (equal to u Cor the channel) of the majority disparity are accepted as correct. The entire matehing process is then repeated at the next vergence fixation.

4. Conclusion

A computer implementation has been tested on a random dot stereogram (Fig. 3,4) and a stereo pair of a natural scene (Fig. 5,6) with good qualitative results. The vergence strategy was very sim­ ple and consisted of moving the plane of fixation through the entire disparity range in a single uniform movement. Global disambigua­ tion was effected by requiring that matches from spatially coin­ cedent areas of adjacent spatial frequencies possess similar dispar­ ity. This constraint is a binocular extension of the spectral con­ tinuity property of raw prima! sketch tokens. Thus it has been shown that there is . no need for local control of the vergence mechanism and its associated problems. Additionally, it has been suggested that the raw prima! sketch provides a frame of reference for the maintenance of disparity values across vergence movements.

ACKNOWLEDGMENTS

Special thanks to Bill Sakoda. Thanks also to Tony Maida, Gordon Shulman, John Sacha, and Jon Crouse. This wo~k was supported in part through a grant from the General Electnc Cor­ poration.

HEFERENCES

1. Marr, D. 1982. Vision. San Francisco: W.H. Freeman and Com­ pany.
2. Grimson, W.E.L. 1980. A computer implementation of a theory of human stereo vision. MIT A.l. Lab. Memo 565. "Phi!. Trans. Soc. Lond. B292" ,217-253.
3. Grimson, W.E.L. 1981. From Images to Surfaces. Cambridge, Mass.: MIT Press.
4. Marr, D., and T. Poggio. 1979. A computational theory of human stereo vision. "Proc. R. Soc. Lond. B204" ,301-328.
5. Kidd, A.L., J.P. Frisby, and J.E.W. Mayhew. 1979. Texture con­ t,ours can facilitate stereopsis by initiating appropriate vergence eye movements. "Nature 280" ,82S-832.
6. Marr, D. 1976. Early processing of visual information. "Phi!. Trans. R. Soc. Lond. B275" ,483-524.
7. Marr, D., and E. Hildreth. 1980. Theory of edge detection. "Proc. R. Soc. Lond. B207" ,187-217.
8. Mayhew, J.E.W., and J.P. Frisby. 1981. Psychophysical and computational studies toward a theory of human stereopsis. "Artificial Intelligence 17", 34S-385.
9. Ullman, S. 1979. The Interpretation of Visual Motion. Garn­ bridge, Mass.: MIT Press.