IRV: Learning to Integrate Visual Information Across Camera Movements

IRV: Learning to Integrate Visual Information Across Camera Movements

Peter N. Prokopowicz

Artificial Intelligence Laboratory Department of Computer Science University of Chicago 1100 East 58th Street Chicago, Illinois 60637

Our eyes see well only what is directly in front of them; they must continually scan the faces, words, and ob­ jects around us. Perceptual integration is the process of combining the resulting jumpy, nonuniform images (Fig­ ure 1) into our stable, comprehensive perception of the world. Visual robots that scan their environment with moving cameras must also integrate visual information. We describe IRV, a visual robot that sees across cam­ era movements [Prokopowicz, 1995]. Furthermore, IRV learn& to integrate from experience, which consists of a series of random movements of a camera mounted on a motorized pan-tilt platform, observing the day-to-day activity in a laboratory. The learning procedure makes minimal assumptions, is robust, and scales. That is, learning proceeds without a prior analytic model, exter­ nal calibration references, or a contrived environment, and can compensate for arbitrary imaging distortions, including lens aberrations, rotation of the camera about its viewing axis, and spatially-varying or even random sampling patterns.

IRV develops an accurate model of its own visual­ motor geometry by learning to predict the sampled im­ ages that follow each random, but precise, camera move­ ment (Figure 2). The model describes the relationship between any relative camera movement vector and the subsequent apparent motion of each pixel in the im­ age. This relationship between corresponding pairs of visual points and camera movement vectors is stored in a representation we call the vi&ual motor calibration map. The map is filled over time from natural observations during development . Such table-based techniques for perceptual-motor development have been used to learn hand-eye coordination [Mel, 1990] and dynamic arm con­ trol policies [Atkeson, 1990]. For example, Mel's MUR­ PHY memorized the relationship between the visual po­ sition of key points on its arm and the joint angles of the arm in that position. The individual experiences that IRV uses to fill its table are the visual shifts of pixels be­ tween successive images. This fundamentally ambiguous correspondence problem can not be determined from any single example. IRV overcomes this ambiguity by accu­ mulating evidence from every repetition of each possi­ ble camera movement. Effectively, every apparent pixel correspondence (there are typically hundreds for every pixel) vote& for the existence of an actual correspon­ dence under the camera movement that just occurred.

Paul R. Cooper

Intelligent Perception and Action Lab Dept . of Elec. Eng. and Computer Science Northwestern University 1890 Maple Avenue Evanston, Illinois 60201

Figure 1: The problem: A sequence of three overlap­ ping views taken by a foveal , or spatially-varying, cam­ era. Slight changes in viewpoint emphasize completely different details. Any understanding of the whole scene (lower right) demands integration of information across eye movements, both for human and foveal computer vi­ siOn.

Eventually, enough votes accumulate to determine the true geometric relationship between pixels in successive images for the entire repertoire of movements, and the predictions become more accurate.

The calibration map is implemented as a connection­ ist visual memory that, during each movement , trans­ forms visual information from the previous fixation into a reference frame centered on the new viewing direction. The visual shift of a single pixel for a particular move­ ment is embodied as a three-way connection between units representing the camera movement, the visual lo­ cation of a feature before the movement , and the location of the same feature after the movement . These connec­ tions are not present initially but develop during early experience by a hypothesize-and-test process. Over re­ peated practice movements, each visual unit notes any correlation between its inputs after the movement, and those of other units before the movement. The com­ binatorics of the problem make it impossible to record the frequency of every conceivable correspondence be­ tween pixels. Instead, only a small random set of pos­ sible correspondences is evaluated at any time. This re­ duces the number of connections needed for learning to O(Npiul· Nmo1Jement. ), which is feasible both for artifi­ cial and biological systems [Prokopowicz, 1995].

The learned relationship between corresponding pairs of visual features and relative camera movements de­ fines a motor-based metric that IRV uses to interpret an arbitrary non-uniform visual representation in terms of known movement angles. This interpretation assigns a true visual angle for each pixel, regardless of optical or sampling distortions. The calibration works by con­ straining the angles assigned to pairs of corresponding pixels so that they are separated by the size and direc­ tion of the movement angle for which they have been found to correspond. As the accuracy of learned visual­ motor correspondences improves, so does the accuracy of the constrained assignment of visual angles to pixels (Figure 3).

IRV can see over a field of view wider than that observ­ able from the camera in a single position (Figure 1). IRV learns to visualize internally the location of peripheral image details that can no longer be resolved. The con­ nectionist visual memory continually accumulates visual features near the fovea, and. integrates them over time and eye movements by imagining where they would ap­ pear from the present viewing direction. We have repli­ cated human psychophysical experiments which show that IRV can perceive and make accurate judgements about simple forms too large to fit in a single view.

The connectionist computational architecture and the experimental environment approximate the conditions of biological perceptual development; the learning algo­ rithm is neurophysiologically plausible. Learning and mature performance both manifest time and space com­ plexities commensurate with human abilities and re­ sources. In other words, the learning algorithm scales completely. The results confirm the practicality of vi­ sual robots that learn to perceive the stability of the world despite eye movements, learn to integrate geomet­ ric features across fixations, and, in general, develop and calibrate accurate models of their own perceptual-motor systems.