--- title: "Sensor Motion and Relative Depth From Difference Fields of Optic Flows" authors: - "Joachim H. Rieger" - "Daryl T. Lawton" doi: "" url: "https://ijcai.org/proceedings/1983/237" pdf: "https://ijcai.org/proceedings/1983/0237.pdf" venue: "Proceedings of the Eighth International Joint Conference on Artificial Intelligence" year: 1983 pages: "1027-1030" track: "Vision" keywords: --- # Sensor Motion and Relative Depth From Difference Fields of Optic Flows Joachim H. Rieger, Daryl T. Lawton *Proceedings of the Eighth International Joint Conference on Artificial Intelligence*, pages 1027-1030. ## Abstract This paper develops a simple and robust procedure for recovering sensor motion parameters from image sequences induced by unconstrained sensor motion relative to a stationary environment. Difference vectors of optic flows approximate the orientations of the translational field lines in image areas in which there is depth variance between the corresponding environmental points and sufficient angular separation from the translational axis. This is developed into a procedure consisting of four steps: 1) locally computing difference vectors from an optic flow field; 2) thresholding the difference vectors; 3) minimizing the angles between the difference vector field and a set of radial field lines which correspond to a particular translational axis; and 4) extracting the translational and rotational component fields given the translational axis. This procedure does not require a priori knowledge_ about sensor motion or structure of the scene. It depends critically on sufficient variation in depth along some visual directions to endow the flow field with discontinuities. We present results of applying the procedure to sparse and low resolution displacement fields. ## Full text ## Introduction The motion of an observer/sensor is in general composed of a translation and a rotation. It generates an optic flow field in the image plane of the sensor due to changes of visual directions of details in the environment over time (Gibson et. al. 1955). The translation of · the sensor induces a radial flow in the image with the intersection of the translational axis and image plane as its center. Sensor rotatation induces a rotational field in the image that is purely direction dependent (that is, a function of image position only). There are significant difficulties in applying this observation to actual image sequences. Flow fields computed from actual image sequences are not arbitrarily dense and are in fact generally sparse so there will not be two distinct flow vectors positioned at the same image point. Thus it is necessary to perform the computation using difference vectors determined from image displacement vectors which are spatially separated. From images formed at discrete, successive instants we obtain image displacements and not instantaneous optic velocities. Thus the computation must be expressed in terms of discrete sensor motions. Also, real flow fields are noisy and errorful, especially near occlusion boundaries because of the changes in image structure that occur there. Thus the procedure must be robust to such distortions in the determined difference vectors. These problems are addressed in this paper. ## Difference Vectors from Spatially Separated Flow Vectors Here we present results on the effects of using spatially separated image velocity vectors to determine difference vectors. A difference vector formed from spatially separated image velocity vectors can be decomposed into a signal component oriented along the correct translational field line and a noise component. We find that the signal component increases for difference vectors formed at image locations where large depth changes occur in the corresponding environmental positions. It also increases with increasing distance between the difference vector and the intersection of the translational axis with the image plane. To the extent that these conditions are satisfied for an optic flow field, its difference vector field will approach the corresponding set of correct translational field lines. The computation of difference vectors over the image does not require initially determining the location of occlusion boundaries or of image areas corrsponding to large visual slant. Consider a sensor 0 moving relative to a static environment. As in figure 1 the point P at the image position r = (i,Y) = (xfz,y/z) corresponds to the environmental point P at the location r = (x,y,z). We obtain the image velocity u at P by differentiating wrt time Letting v = (vx,vy,vz) and w = (Wx·Wy·wz) denote the translational and rotational velocities of 0 the relative motion of P becomes Figure 1 ![Image](https://ijcai.org/proceedings/1983/fulltext/237/image_000000_3d1b2c878b12b69317e74ab599446fec50970fbc798993696ea0c6f36ec4d314.png) Eliminating i, y, and i between the above we can write the translational and components of image velocity u separately equations rotational Two image points. PI and P2 that are separated by r2-ri = (rlx.rly) differ in their rotational flow vectors by If rT = (vxfvz,vyvz) denotes the intersection of the translational axis with the image plane we can rewrite the translational flow vector as uT = vz(r-rT)/z. Then the difference vector of two translational flow vectors - - at separated image positions PI and P2 becomes where is the depth separation of the environmental details PI and P2 that correspond to PI and P2 in the image. Now we can rewrite ~u as consisting of a component along a translational fieldline and a noise component ~u = ~uT + ~uR = For difference vectors with sufficient angular separation from the translatory axis and separation in depth ~0Signal > >~0Noise· ## Recovery of Motion Parameters and Depth In order to compute difference vectors from image displacement fields formed over discrete time intervals (as opposed to continuous intantaneous image velocity fields), we have to be careful to describe all quantities with respect to the same reference system. Suppose two environmental points lie along the same ray of projection in an image at time t. Translating and rotating the sensor will displace the projections of these points to new positions in the image at time t + 1. In the image at time t + t, the image points will be separated due to the translational component of the sensor motion (unless they are located on the translational axis). The separated image points and the intersection of the translational axis with the image plane will be collinear at time t + t. This is the discrete analog of the fact that difference vectors at discontinuities of an instantaneous optic velocity field are oriented along translational field lines. . Thus, given image displacements Dt and D2 at positions Pt and P2, the difference vector between points t and 2 is obtained by subtracting D2 from Dt and positioning the resulting vector at Pt + Dl. Two thresholds are used in evaluating difference· vectors. The separation threslwld determines the maximal allowable distance between displacement vectors in determining difference vectors. The neighborhood of a given displacement vector contains all other displacement vectors which lie within a distance determined by the separation threshold. The length threslwld determines the minimal allowable length for a difference vector. For a given difference vector and a set of radial field lines, the error angle is the angle between the difference vector and the fieldline at that position. We have found that reducing the number of difference vectors by increasing the length threshold and decreasing the separation threshold improves the fit of the difference vector field to the set of correct field lines up to a certain degree. This is because short difference vectors (compared to the local average magnitude) are more likely to deviate from the correct field lines and computing difference vectors over larger neighborhoods increases the noise components. If, however, thresholding eliminates too many difference vectors the fit detonates, since the signal of the difference vector field becomes less distinguished for a decreasing number of vectors. For each image displacement vector a set of difference vectors of sufficient length is determined over its neighborhood. For the resulting field of difference vectors, processing involves finding a translational axis and the corresponding set of radial field lines which minimizes the sum of the magnitude of the error angles. The procedure used is basically that used in Lawton (t982, t983) to determine the translational axis from noisy displacement fields induced by rectilinear sensor motion. The error measure is defined on a half sphere, where points on the half sphere are possible candidates for the translational axis. The advantage of using a sphere as a domain is that it allows for a uniform, global sampling of the error function. The search process itself consists of a global sampling of the error measure to determine its rough shape using a generalized Hough transform (Ballard t980, 0' Rourke t98t) followed by a local search to find a minimum. The computation of the sensor rotation (scaled by focal length) from the original flow field and the radial (translational) fieldlines is straightforward. Note that the components of the flow perpendicular to the radial fieldlines are induced by sensor rotation. Introducing, for convenience, a polar coordinate system (r,9) in the image plane centered at PT we have a system of overconstrained linear equations of the type UR . ea = u . ea in the three unknowns (I)X' Wy· and wz. Knowing the rotational parameters yields the translational and rotational component fields of the orginal flow field. The translational component is directly related to the relative depth of a scene (i.e. the depth scaled by the sensor displacement in depth 8z) by the relation z/8z = I i - iT I /1 uT I , where uT is a translational flow vector in the image. If the frame rate is known the relative depth of an environmental point corresponds to its temporal separation from the sensor (under constant approach velocity). Biological systems seem to exploit this optical relation for a variety of navigational tasks (Lee t980, Wagner 1982). ## Experiments The flow field in figure 2a shows image displacements at pixel positions having coordinates which are multiples of 8 from a 128 x 128 pixel field. The components of the displacement vectors were stored as 8 bit integers. The environment consisted of a spherical surface patch at depth of 10 units along the z axis and a background spherical &urface patch at a depth of 30 units along the z axis. The obvious discontinuities in the flow field in figure 2a indicate the boundary of the nearer surface. The sensor motion consisted of an intial rotation of 0.1 radians about the (1,1,1) axis followed by a translation of 2 units along (0,0,1). The separation threshold was set to 1 pixel and the length threshold was set to 3 pixels. Figure 2b shows the average difference vectors which exceeded the length threshold. Note their occurrance along the occlusion boundary and their strong radial character. The resulting error function is shown in figure 2c (Darker in the figure corresponds to less error; also recall that this is a plot of a hemisphere in polar coordinates and not the image plane). As can be seen, it is strongly unimodal. The minimum in the global histogram corresponded to the image position (60, 60). The local search determined the minimum to be at (63, 63). The correct, subpixel, position was (635, 635). The determined rotational and translational components are shown in figures 2d and 2e respectively. The relative depth map determined from the translational component field is shown in figure 2f encoded by intensity (darker means closer to the observer). Figure 2a ![Image](https://ijcai.org/proceedings/1983/fulltext/237/image_000001_e82277196a11d93872fee2a8e145b06291071aefa37e989b401d39b8f10b3470.png) Figure 2b ![Image](https://ijcai.org/proceedings/1983/fulltext/237/image_000002_fa8672e725efab63386205427bb4b4fe46c31b38fa4a42d6ef016648eed3f808.png) Figure 2c ![Image](https://ijcai.org/proceedings/1983/fulltext/237/image_000003_9e00ca14b556fb89368773c7f53bab2ce43c8de74d2473ff0c9d46d346ce1b23.png) Figure 2d ![Image](https://ijcai.org/proceedings/1983/fulltext/237/image_000004_83f03a24221c6a2bdce19c3e9118337a54d82c08ce62d8e4fbfda849a5b6b52b.png) .. .. .. ! ' 7 Figure 2e ![Image](https://ijcai.org/proceedings/1983/fulltext/237/image_000005_ab625b8db4900111249cdf8b1a7130e767417b10c0c565ba2bc0c9c3c8bac0ca.png) · · .. · 1 \ \ "\ "" ""'""' Several experiments with simulated displacement fields have confirmed the expected effects of environmental depth variance and displacement vector density and demonstrated robust performance with respect to various kinds of noise. The procedure has also s4ccessfully determined the translational axis from displacement fields obtained from low resolution image sequences from a solid state camera. For such image sequences with large environmental depth variances the translational axis has been determined within a few, degrees of visual angle (Rieger and Lawton 1983). We thank Frank Glazer for pointing out a bug in this paper. This research was supported by DARPA grant N0000014-82-K-04064 to the MOTICN group at UMASS. ## References 1. Ballard, D. H., "Parameter Networks: Towards a Theory of Low-Level Vision", Proc. of 7th IJCAI, Vancouver, British Columbia, pp. 1068-1078, 1981. 2. Gibson, J. J., Olum, P., and Rosenblatt, F., "Parallax and Perspective During Aircraft Landings", Am. Journ. Psycho/ ., vol. 68, pp. 372-385, 1955. 3. J . Rieger and D. Lawton 1031 Figure 2f 4. Koenderink, J . J., and van Doom, A. J., "How an Ambulant Observer can Construct a Model of the Environment from the Geometrical Structure of the Visual Inflow", Kybernetik 1977, G. Hauske and E. Butenandt, editors, R. Oldenbourg Verlag, Munich, 1978. 5. Lawton, D. T., "Motion Analysis via Local Translational Processing", JEEE Workshop on Computer Vision: Representation and Control, pp. 59-72, 1982. 6. Lawton, D. T., ''Processing Translational Motion Sequences", Computer Graphics and ]mage Processing, in press, 1983. 7. Lee, D. N., 'The Optic Flow Field: the Foundation of Vision", Phil. Trans. R. Soc. Lond. B., vol 290, pp. 169-179, 1980. 8. Longuet-Higgins, H. C. and Prazdny, K., "The Interpretation of a Moving Image", Proc. R. Soc. Lond. B., vol 208, pp. 385-397, 1980. 9. O'Rourke, J., "Motion Detection Using Hough Techniques", Proceedings of PJUP. pp. 82-87, 1981. 10. Prazdny, K., "Egomotion and Relative Depth Map from Optical Flows", Bioi. Cybernet ., vol. 36, pp. 87-102, 1980. 11. Rieger, J. H., "'nformation in Optical Flows Induced by Curved Paths of Observation", J. Opt. Soc. Am., vol. 73, pp. 339-344, 1983. 12. Rieger, J. H., and Lawton, D. T., "Determining the Axis of Translation from Optic Flow Generated by Arbitrary Sensor Motion", COINS Technical Report, 83-01,1983. 13. Wagner, H., "Flow-field Variables Trigger Landing in Flies", Nature, vol. 297, pp. 147-148, 1982.