Fitting Models to Distributed Representations of Vision
Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence. Action and Perception. Pages 3-8.
© 1995 International Joint Conferences on Artificial Intelligence
Abstract
Many representations in early VISion can be constructed by performing orientation analysis along several sampling dimensions. Texture is often oriented in space, motion is oriented in space-time, and stereo is oriented in spacedisparity . In these modalities, we can construct distributed representations with oriented energy measures used in models of biological vision. Surface models of orientation, velocity, and disparity can easily be fit to distributed re presentations of texture, motion, and stereo by combining tools of orientation analysis and regularization. We describe base representation construction and model fitting processes in these domains.
- [Adelson and Bergen, 1985] E. H. Adelson and J. R. Bergen. Spatiotemporal energy models for the perception of mo- tion. JOSA A, 2:284-299, 1985.
- [Adelson and Bergen, 1991] E.H. Adelson and J .R. Bergen. The plenoptic function and the elements of early vision. In M. Landy and J. Movshon, editors, Computational Models of Visual Processing. MIT Press, 1991.
- [Bergen and Landy, 1991] J .R. Bergen and M. Landy. Com- putational models of visual texture segregation. In M. Landy and J . Movshon, editors, Computational Mod- els of Visual Processing. MIT Press, 1991.
- [Bergen et al., 1992] J. R. Bergen, P. J. Burt, K. Hanna, R . Hingorani, and S. Peleg. A three-frame algorithm for estimating two-component image motion. IEEE PAM!, 14:886-896, 1992.
- [Black and Anandan, 1993] M. J. Black and P. Anandan. A framework for the robust estimation of optical flow. In !CCV, pages 231-236, Berlin, Germany, 1993.
- [Finkel and Sajda, 1992] L. H. Finkel and P. Sajda. Object discrimination based on depth-from-occlusion. Neural Computation, 4:901-921, 1992.
- [Fleet and Jepson, 1990] D. Fleet and A. Jepson. Measure- ment of image velocity by phase information. Interna- tional Journal of Computer Vision, 5(1):77, 1990.
- [Freeman and Adelson, 1991] W . T. Freeman and E. H. Adelson. The design and use of steerable filters. IEEE PAM!, 13(9):891-906, 1991.
- [Grzywacz and Yuille, 1990] N. M. Grzywacz and A. L. Yuille. A model for the estimate of local image velocity by cells in the visual cortex. Proceedings Royal Society London B, 239:129-61, 1990.
- [Heeger, 1987] D. Heeger. Model for the extraction of image flow . JOSA A, 4:1455-1471, 1987.
- [Heeger, 1992] D . Heeger. Normalization of cell responses in cat striate cortex. Visual Neuroscience, 9:181-197, 1992.
- [Heitger and von der Heydt, 1993] F . Heitger and R. von der Heydt. A computational model of neural contour pro- cessing: Figure-ground segregation and illusory con- tours. In !CCV, pages 32-40, Berlin, Germany, 1993.
- [Heitger, 1992] F. et al. Heitger. Simulation of neural contour mechanisms: From simple to end-stopped cells. Vision Research, 32(5):963-981, 1992.
- [Kass and Witkin, 1987] M. Kass and A. Witkin. Analyz- ing oriented patterns. Computer Vision, Graphics and Image Processing, 37:362-385, 1987. IJCAI
- [Kass et al., 1987] M. Kass, A. Witkin, and D. Terzopoulos. Snakes: Active contour models. International Journal of Computer Vision, 1(1):1, 1987.
- [Koenderink, 1992] J. Koenderink. Generic neighborhood operators. IEEE PAM!, 14(6):597-605, 1992.
- [Niyogi, 1995a] S. Niyogi. Detecting kinetic occlusion. In International Conference on Computer Vision, 1995.
- [Niyogi, 1995b] S. Niyogi. Kinetic occlusion. Technical Re- port 319, MIT Media Lab, 1995.
- [Olshausen et al., 1993] B. Olshausen, C . Anderson, and D. C. van Essen. A neurobiological model of visual at- tention and invariant pattern recognition based of dy- namic routing of information. Journal of Neuroscience, 13(11):4700, 1993.
- [Perona, 1992] P. Perona. Steerable-scalable kernels for edge detection and junction analysis. In Proceedings of the European Conference on Computer Vision, pages 3-18, 1992.
- [Poggio et al., 1985] T . Poggio, V. Torre, and C. Koch. Com- putational vision and regularization theory. Nature, 317(26):314-319, 1985.
- [Qian, 1994] N. Qian. Computing stereo disparity and mo- tion with known binocular cell properties. Neural Com- putation, 6(3):390, 1994.
- [Sanger, 1988] T. D. Sanger. Stereo disparity computation using gabor filters . Biological Cybernetics, 59:405-418, 1988.
- [Shizawa and Mase, 1991] M. Shizawa and K. Mase. A uni- fied computational theory for motion transparency and motion boundaries based on eigenenergy . analysis. In IEEE CVPR, 1991.
- [Simoncelli, 1993] E. Simoncelli. Distributed Representations of Motion. PhD thesis, MIT EECS, January 1993.
- [Terzopoulos, 1988] D. Terzopoulos. The computation of vis- ible surface representations. IEEE Pattern Analysis and Machine Intelligence, 10( 4):417-438, 1988.
References were extracted automatically from the PDF and may contain errors.
Full text extracted automatically from the PDF; figures, tables and equations may differ. The PDF is the version of record.
1 Distributed Representations
Conventional approaches to early vision have focussed on recovering explicit models of local structure and sub sequently fitting models to local structure. When the assumptions made in modeling local structure hold, the local structure is successfully recovered. When model assumptions are violated, however, local structure is in correctly recovered in unstructured ways. For example, conventional motion estimation algorithms assume that there is a single motion, that the intensity is constant in space time, and that the flow field is smooth. These assumptions are violated, for example, at motion bound aries. At motion boundaries, two motions are present, intensity is not conserved, and the flow field is discontin uous. Consequently, conventional approaches of recover ing flow fields fail at occlusion boundaries. Various tech niques have been proposed to counter these assumptions: regularizing with line processes [Poggio et al., 1985], gen eralizing local models [Shizawa and Mase, 1991], using global parametric models [Bergen et al., 1992], and using robust estimation [Black and Anandan, 1993].
Alternatively, we can use distributed representations of vision, where a population of responses implicitly represents local structure. This strategy is used in vi sual cortex, where neurons do not represent parame ters explicitly, but instead are tuned to stimulus pa rameters of .spatiotemporal frequency, horizontal dis parity, velocity, global motion patterns, and so forth.
Several have produced models of tuning properties of cells in the visual cortex, in parallel to conventional ap proaches of modeling and recovering local structure in computer vision: spatially oriented filters model orienta tion selectivity of simple cells [Bergen and Landy, 1991]; spatiotemporally oriented filters model directional se lectivity [Adelson and Bergen, 1985]; stereo spatial fil ters out of phase model disparity tuning [Qian, 1994; Sanger, 1988]; and combining the outputs spatiotempo rally oriented filters models velocity tuning [Grzywacz and Yuille, 1990; Heeger, 1987; Simoncelli, 1993].
T.he construction of distributed representations in dif ferent modalities becomes remarkably similar once ob serving that most local structure in visual stimuli is ori ented along several sampling dimensions [Adelson and Bergen, 1991]:
- Space. Spatial textures and edges are oriented in space, where x = (x, y).
- Space-time. Motion is orientation in space-time, where x = (x, t) or x = (x, y, t) in one and two di mensional motion (see [Adelson and Bergen, 1985]).
- Space-disparity. Stereo is orientation in space disparity, where x = (x, Vx) or x = (x, y, Vx) for one and two-dimensional stereo analysis . Typically, two samples of Vx form a stereo pair.
Figure 1 illustrates oriented structures in each of these dimensions. In the above modalities, constructing dis tributed representations of orientation is possible by filtering a stimulus I(x) with multiple oriented filters tuned to different orientations in space, space-time, or space-disparity. Each spatiotemporally oriented filter signals the presence of a given orientation; by having banks of oriented filters tuned to different orientations , a population of responses can be obtained to represent local structure. Figure 2(a,b) shows an (x, t) stimulus with occluding and transparent motions, and a slice of the base representation Eu ( x, t, B) we can construct from the stimulus using oriented filters . Where there is trans parency, a bimodal distribution of energy is present at each point x, and where there is occlusion, a bimodal dis tribution of energy is present to both sides of x. While it is unclear how properties such as color and shading could be encoded in distributed representations, we can take a unified approach for the domains of texture, motion and stereo.
Rather than fitting models to explicit representations of orientation, such as range data or optical flow, we fit models to implicit, distributed representations of vi sion. Distributed representations are advantageous be cause they: (1) represent complex naturally occurring stimuli, such as transparency and occlusion; (2) repre sent uncertainty implicitly; (3) degrade gracefully with noise; ( 4) do not commit to an (often wrong) answer, unlike explicit representations; (5) are also used in bio logical visual systems.
The problem remains, however, of how to utilize distributed representations of vision to recover infor mation about global surfaces. Most of the past ap proaches used distributed representations to estimate local structure, in the domains of texture [Kass and Witkin, 1987], stereo [Sanger , 1988; Qian, 1994] and mo tion [Fleet and Jepson, 1990; Grzywacz and Yuille, 1990; Heeger, 1987]. In regions containing little or no texture, filter responses are negligible, so approaches which at tempt to use distributed representations to estimate lo cal structure will fail. Thus the surface interpolation problem remains.
The surface interpolation problem in explicit repre sentations is typically solved by taking sparse represen tations of explicit local measurements, and smoothly in terpolating between them. Canonical approaches to sur face interpolation (e.g. [Terzopoulos, 1988]) use both a fidelity criteria, describing the fit of the surface to sparse measurements, and a smoothness criteria so as to constrain regions without measurements. Typically, weighted combinations of the square of the first and second derivatives of surfaces are used as a measure of smoothness. By combining tools of orientation analysis with simple modifications of standard regularization pro cesses, we show that it is straightforward to fit models to distributed representations.
Rather than recovering explicit local structure, and then fitting surface models to local structure, we fit mod els to distributed representations of vision directly. The generic approach is contrasted with our approach in Fig ure 3( a, b). We describe base representation construction and our adaptation of regularization in the domains of space, space-time, and space-disparity, and demonstrate preliminary success on simple imagery.
2 Fitting Models
Our task is to fit models to time varying visual stim uli I(x) = I(x, y, t, Vx) with sheet models u(u, t); a list of the domains we consider is shown in Figure 4. For one-dimensional problems, u = u tessellates spatial vari able x in x when x = (x, t) or x = (x, Vx)· For two dimensional problems, u = ( u, v) tessellates the spatial variables (x, y) in x when x = (x, y), x = (x, y, t), or x = (x, y, Vx)·
For sheet model fitting problems, the surface is mod eled with an array of nodes, where at each node, the spatial coordinates and an orientation e is stored in the combined model u(u, t). The goal is to fit a sur face 8(u, t) to a base representation Eu(x, 8) indicat ing the presence of the orientation e at point X. For two-dimensional texture, the orientation e = 0 is local spatial orientation; for one-dimensional and two dimensional stereo, the orientation e = Vx is horizontal disparity; for one-dimensional motion, the orientation e = Vx is horizontal motion; for two-dimensional mo tion, the orientation e = ( Vx, Vy) is a two dimensional vector.



The model u(u, t) indexes into the base representa tion Eu(x, e), so that a total "energy" of a surface can be obtained. We seek to fit a model where the popula tion response in the base representation E(x, e) across the surface is maximized, and where the magnitudes of the derivatives in the orientation surface are minimal, as in standard regularization processes ( c.f. [Terzopou los, 1988]). Since multiple surfaces can be fit, no answer is correct; the only satisfactory answer will require the observer to shift attention between portions of the base representation.
Models are fit over multiple scales (J' by constructing a scale-space:
where Gu(x) represents a Gaussian convolution ker nel of width (J'. Scale-space is approximated with a Gaussian pyramid at discrete scales of (J'. An N dimensional interpolatable base distributed representa tion Eu(x, e) is constructed from Iu(x) through orien tation analysis. Steerable [Freeman and Adelson, 1991 ; Koenderink, 1992; Perona, 1992] filters allow for continu ous interpolation to any e . The stimulus at a given scale is used to compute n steerable "basis images" J!(x) via convolution with a set of basis filters g;(x):
where e represents spatial orientation, spatiotemporal orientation, velocity, or disparity, depending on the sam pling dimensions being considered. We will describe the details of base representation construction in subsequent sections.
Models can be fit these base representations by modi fying standard regtilarization procedures ( c.f. [Terzopou los, 1988]) to use this base representation to dynamically estimate the state of a model u(u, t):
where assumptions of smoothness in the nth order spa tial derivatives of orientation are embedded within the stiffness matrix K , and base representations E17 (x, e) are used to apply observation "forces" f(u, t) to the sur face by integrating forces through discrete summation over several scales:
Here, H indexes the orientation surface component e(u, t) of the model u(u, t). We can easily modify the above to introduce additional "forces" to shift attention from certain portions of base representation to others, as in the deformable contours [Kass et a/., 1987].
Thus we employ the following procedure to fit models u to stimuli I(x) :
| Stimulus J(x) , Model u(u) |
| Space (2-d) lu(X, y) Model u(u,t) = [x(u,v,t),y(u,v,t),B(u,v,t)f |
| Space-time (1-d)_ 117(x, t) Model u(u,t) = [x(u,t) ,vx(u,t)f |
| Space-disparity \1-d) 117 (x, Vx) Model u(u, t) = [x(u , t), Vx(u, t)f |
| Space-time (2-d) lu(x, y, t) Model u( u, v, t) = [x( u, v, t), y( u, v, t), vx( u, v, t), vy( u, v, t)f |
| Space-disparity (2-d)J.,.(x, y, Vx) Model u( u, v, t) = [x( u, v, t), y( u, v, t), Vx( u, v, t)f |
- Approximate scale-space Iu(x) through construc tion of a Gaussian pyramid of I(x).
- Over all scales (J', construct base representation E17 (x, e) by convolving Iu(x) with n basis filters g;(x) to compute J!(x), where i = 1, ... , n.
- Initialize model u.
- Continually subject model u to data forces f(u, t) derived from base representation Eu(x, e).
Below we describe base representation construction, and methods to apply observation forces derived from base representations in the domains of texture, motion, and stereo. We illustrate model fitting processes on canonical examples in our preliminary experiments here.
2.1 Two-dimensional space
For two-dimensional spatial texture, X= (x, y), and e = 8, where 8 represents the local orientation of the texture. A single 2-d sheet model is superimposed on the data:
Oriented energy[Bergen and Landy, 1991] measures are used to construct the base representation Eu(x, y, 8) via:
where the spatial stimulus Iu(x, y) at multiple scales (J' is used to compute a set of steerable "basis" images J!(x, y) with:
Filters g; ( x, y) form the steerable separable basis for the quadrature pair of filters Ge(x, y) and He(x, y), which are interpolated to any orientation 8 with interpolation functions /;(8); refer to [Freeman and Adelson, 1991] for filter taps and interpolation !unctions for all of our experiments; we use a second derivative of Gaussian as a basis in our examples except where noted. An example stimulus, basis images, and the base representation are shown in Figure 5.
Forces are applied to the model via the gradient of Eu(x , y,8) integrated over all scales:
5


" "
Figure 6: Two-dimensional texture. (a) A simple ori ented texture; (b) Orientation map depiction of the fit ted model.
here H = [OIOII] indexes into the surface B(x, y) compo nent of u(u, t). To avoid singularities due to the discon tinuous nature of angles, a vector fte = [cos( B), sin(B)]T is used to represent the state of the model, and is contin uously normalized to one (lnel = 1) . We illustrate the analysis of oriented texture patterns containing simple deformations in texture orientation. Figure 6(a) shows one of several sample oriented patterns we have experi mented with. Figure 6(b) shows an orientation map re covered using our approach. With multiple orientations in the stimulus, either multiple models must be fit, or an attentional mechanism to shift the surface is neces sary. With detected orientation stopping ( c.f. [Heitger, 1992]), smoothing processes should be halted.
2.2 One-dimensional space-time
For one-dimensional space-time, X= (x, t) and e = Vx, where Vx represents local 1-d motion. A time-varying 1-d sheet model is imposed on the data: .
For spatiotemporal stimuli I(x, t), motion is orientation in space-time lAdelson and Bergen, 1985]; the motion Vx is related to the spatiotemporal orientation B via Vx = u tan( B).
.lust as in ( x, y), oriented energy measures are used for base representation construction in ( x, t):

2 -

where J~(x, t) are steerable basis images:
An example stimulus, basis images, and its base repre sentation are shown in Figure 7.
Forces f( u, t) are applied to the model via the gradient of Eu(x, t, B) integrated over all scales:
Here, H =[Oil] indexes into the surface vx(x, t) compo nent of the model u(u, t). Figure 8(a) shows a simple stimulus where one surface is moving rightward occlud ing a background moving leftward. Figure 8(b,c) shows the sheet model fit via our approach for the stimulus for a given instant in time.
2.3 Two-dimensional space-time
In two dimensional space-time, X = (x, y, t) and e = (vx,vy), where v= [vx,vyf represents local2-d motion. A sheet model is imposed on the data:
Now simple orientation analysis is inadequate; a two stage calculation is necessary [Heeger, 1987]. The first stage involves filtering an image sequence with spa tiotemporally oriented filters. The second stage pools the squared responses of these filters to construct units tuned to velocity. Stimuli translating with velocity v = [vx , vy]T, when viewed in the frequency domain, contain energy along the plane WxVx + WyVy + Wt = 0.
Constructing E"(x, y, t, Vx, vy) is possible through lin ear filtering of the stimulus I(x, y, t) with banks of spa tiotemporal filters centered on this plane [Heeger, 1987; Simoncelli, 1993].
We utilize the method developed in [Simoncelli, 1993] to extract energy along these planes in the frequency do main, using tools of steerable filtering. We summarize the method below. Convolution of an image sequence I(x) with a set of basis filters g;(x) allows for steer able filtering to any portion of the frequency domain Wj = ±[wt, w2, waf, where /w/ = 1, with interpolation functions fn ( w) :
Steering functions are summarized in Figure 9(a) for Nth order derivative of Gaussians (N = {1, 2} ). Filter coef ficients used in these (x, y, t) experiments can be found in [Niyogi, 1995b] . To tile the plane n ,;n · w = 0, energy is extracted with N + 1 weightings centered at locations Wj, as summarized in Figure 9( a). Figure 9(b) depicts energy extraction for a first derivative of a 3-d Gaussian (N=1); two separate weightings are required to tile the plane. Choosing a different scale of Gaussian extracts a different sized ring centered around the plane. By filter ing the image sequence I(x) with filters with frequency selectivities centered at locations Wj, and subsequently applying squaring operations, we extract energy E,;n (x) along a ring on the plane via:
High frequency components introduced by squaring are diminished through subsequent convolution with a 3-d Gaussian G0 (x). An example stimulus , two of the basis image sequences used to construct the base representa tion, and the base representation sampled at one partic ular velocity is shown in Figure 10.
Time-varying forces f( u, v, t) are applied to the model vta:
We illustrate this approach on a simple example. Fig ure ll(a) shows an (x, y, t) cube representation of a sim ple stimulus where two moving surfaces occlude one an other over a static background. Figure 11(b) shows re covered velocity flow field for the stimulus; the motion is extracted for the rightward and leftward moving bar cor rectly, although oversmoothing is present at the bound aries as in conventional approaches. We have designed methods to detect motion boundaries defined by kinetic occlusion ( c.f. [Niyogi , 1995a]) using distributed repre sentations of motion; detected motion boundaries should constrain smoothing processes.

2.4 Two-dimensional space-disparity
In two-dimensional space-disparity, x = (x, y, Vx) and e = Vx, where where Vx represents horizontal dispar ity. A stereo pair I" ( x, y, Vx) is sampled at two posi tions of Vx to yield two signals I~(x, y) and I~(x, y). We construct a two-dimensional I(x, y, Vx) window from the stereo pair with:
At each scale (T, the disparity Vx is related to the lo cal orientation B in the ( x, Vx) direction extracted from this window by Vx = (T tan( B). A 2-d surface model is imposed on the data:
Oriented energy measures are used to construct the base representation E"(x, y, B):
where the space-disparity stimulus I" ( x, y, Vx) at multi ple scales (T is used to compute a set of steerable "basis" images J~(x, y):




Now, 9i(x, y, Vx) = c5(y) * 9i(x, Vx) forms an approxi mation to Ge(x) and He(x), but only the center slice (Vx = 0) of the basis is used to represent the space disparity orientation ( x, Vx) at each point in space ( x, y) . An example stimulus, basis signals used to simulate the base representation, and a constructed base representa tion are shown in Figure 12. A slice at a given height y of the above signals is shown in Figure 13; fitting models to distributed representations of one-dimensional space disparity signals would operate on these signals.
Forces f( u, v, t) are applied to the model via the gra dient of E17 (x, y, B) integrated over all scales CT:
Figure 14(a) shows a simple stimulus where the observer is fixated on a plane, with a bar on the left and right closer and further away from the observer than the plane. Figure 14(b) shows the sheet model fit for stimulus.
3 Discussion
In addition to sheet models, which are directly attached to image data, other models can be fit. Surface models, attached only to a portion of the visual field, would be required to recover the parameters of an object. We have not dealt with the problem of model initialization. To do so, we require a model of attention; extensions to surface models are straightforward once a model of attention is adopted (see, for example, [Olshausen et al., 1993]). Models of controlling attention shifts between surfaces in these distributed spatial representations must be designed.
Seeking enhanced base representations improves re sults greatly. For example, some degree of normalization
[Heeger, 1992] in the base representation may reduce the dependence on local contrast, although the conve nience of steerability is lost . Some operations, such as occlusion detection [Heitger, 1992; Niyogi, 1995a], may be based on the distributed representations and/or the surfaces fit to them; using these representations to con strain model-fitting processes is desired. Linking our model-fitting processes to solutions to surface segmenta tion mechanisms using distributed representations [Heit ger and von der Heydt, 1993; Finkel and Sajda, 1992] is clearly desirable.
The majority of computer vision efforts in early vi sion have been directed towards estimating explicit rep resentations of local structure, and subsequently fitting models to these representations. These explicit repre sentations are prone to failure where model assumptions break down. Implicit representations do not have these problems: Problems in modeling local structure, object model recovery, handling occlusion, surface segregation, attention, etc. can and should be attempted using dis tributed representations of vision. A parallel effort using distributed representations of vision is likely to be more fruitful than relentlessly improving methods of recover ing explicit local structure. We have presented a first step at fitting models to distributed representations of vision here.
@inproceedings{ijcai1995p1,
title = {Fitting Models to Distributed Representations of Vision},
author = {Niyogi, Sourabh A.},
booktitle = {Proceedings of the Fourteenth International Joint Conference on
Artificial Intelligence, {IJCAI-95}},
publisher = {International Joint Conferences on Artificial Intelligence},
editor = {Chris S. Mellish},
pages = {3--8},
year = {1995},
note = {Action and Perception},
url = {https://ijcai.org/proceedings/1995/1},
}
TY - CPAPER TI - Fitting Models to Distributed Representations of Vision AU - Niyogi, Sourabh A. T2 - Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence, IJCAI-95 PB - International Joint Conferences on Artificial Intelligence PY - 1995 SP - 3 EP - 8 UR - https://ijcai.org/proceedings/1995/1 ER -Copied
Machine-readable versions of this paper, for tools and AI agents: