Boundary Conditions in Multiple Intrinsic Images

I. Introduction

One important purpose of vision is to build useful descriptions of the environment and intrinsic images are a first step in the description -building process. Intrinsic images are so called because they represent physical (intrinsic) properties of surfaces in a viewer-centered image format.

Over the past few years, enormous progress has been made in specifying the computation of these images by parallel-iterative methods. Such methods have been applied to disparity [Marr and Poggio , 1976; Baker and Rlicher, 1982; Grimson, 1981], optic flow [Horn and Sch unck, 1981; Ullman, 1979; Barnard and Thompson, 1979], and shape from shading [Ikeuchi and Horn, 1981; Brown et al., 1982]. While such methods constitute an important advance, they only work under most or all of the following conditions: (1) the variation in the image intensity can be entirely explained by the intrinsic parameter being computed; (2) any large discrete variations in the intrinsic parameter are pre-specified; (3) certain free parameters (such as light source dLrection and reflectance function , or equivalently the reflectance map, in shape-from-shading calculations) must also be known a priori ; anJ (4) the image resolution must be chosen appropriately.

Barrow and Tenenbaum [1978] pointed out that intrinsic images are logically computed simultaneously. They also realized the importance of boundary conditions in defining the intrinsic: image computations. However, their edge catalog (page 12 in above reference) is restricted to variations in the illumination of the scene. Subsequent work in shape from shading [Bruss, 1981] and optic flow [Horn and Schunck, 1981] has shown that the estimation of boundaries within a given intrinsic image is

The approximation ( 1.2) does not hold at boundaries between different objects (or at houndaries arising from :t self-occ luding object) , but may be expected to hold tor large patches of the image if the variations L'lx. L'ly. and I'll are appropriately small. Now the p;tf'Lial deriVatives are all measurable quantities, and L'.x/ I'll and L'.y/ I'll are estimates of the optic t1ow (u ,v). Thus

where vf is the spatial gradient of the image and u = (u,v) the optic tlow.

There are several ways ( 1.2) could be satisfied. r:or example, (1.2) holds if t(x ,t) is proportional to the matte reflectance of the surface: f( x, t) ::::: r(x,t). However, if the surface had a spa tially varying reflecumce fUiteliOII , then t1x ,t) ~ r(x ,t) and this constraint would be in va lid. Another way to satisfy ( 1.2) would be to have a bumpy, shiny surface (e.g. , orange skins). In this case the specular reflections would move with surface mol!OitS. These observations point out that although one can keep the notion of intrinsic image, the following further distinctions are helpful:

• a surface properly (e.g., curvature, ref1ectance) IS truly intrinsic. Such a property may or may not satisfy ( 1.2).
• a viewer-cenlt'red properly (e.g., relative depth, p-q space surface orientation, optic flow) .
• -- a correspondence property is one used for matching. lly definition, it is any property satisfying (1.2). Viewer-centered properties are usually not correspondence properties.

One would like surface properties and viewer-centered properties to have the correspOitdence property but this is not guaranteed, and each case must be tested individually.

Sometimes an image intensity function that does not sa tisfy the correspondence property can be fi xed. For example, is there any way to sa tisfy (1.4) given a ~ urt~tce of varying albedo'~ One sample way would be to filter out slow variations in the intensity to get back to lightness [Hom, 1975], but this requires knowing the spatial resolution of the lightness variation compared to spatial variations in surface reflectance and orientation. Less delicate methods involve multiple intrinsic images. For example, Stuth [J9g2l uses edge density for highly­ textured surfaces, and Wohn et al. [19821 have applied the df/ dt == U using parameters other than intensity. Another proposal is to use the color image f(x ,t) (r(x,t),g(x,t),b(x,t)) . Transtorming this image into an intensity-saturation-hue basis tends to isolate the non­ intrinsic achromatic component from the more intrinsic chromatic components. To see that this would work consider the definition of intensity, saturatioi1, and hue by [Tenenbaum and Weyl, 1975). Where r, g, and b are pnmary color measurements,

If the intensities are suhject to variJtion in s url~t cc albedo, then the imaging process will measure (r/x, g/x. b/x) tor some x that affects only the intensity measurement. Specular variations are (approx imately) lll<tlliksted as r+x, g+x, b+x for some x and this affects intensity and sa turation . Thus hue is a true intrinsic image. Of course, Equation (1.4) must also be sa tisfied nontri vially tor this to be helpful.

Ill. Interpolation

In any particular intrinsic image, the points that contain measures of the physical property can be sparse. One such example is the disparity image; the system must interpolate disparity measures between the data points. Such interpolation does not add information to the data but an explicit representation of the surface is helpful i~ such tasks as tactile manipulation.

Much work has been done on defining good interpolation methods [Grimson, 1981; Terzopoulos, 1983], but as Baker and Blicher [l982] pointed out, these methods usually treat all points uniformly . They do not automatically acknowledge the , llCitural , s~a rp disconLinuities that arise in the scene (lor example, !rom occlusion). Thus, while interpolation is Jesirahle, the key problem is to limit the process to an appropnate set of points. Figure 2 shows this problem.

Another problem raised by interpolation across boundaries is that it raises erroneous expectallons as to how the scene woulJ look if viewed from a ditTen.:nt angle. If the interpolation col!IJ be piecewise through appropriate points, then those data w?uld sui! be valid from di flerent view angles, once transformed ( 1-igure 2). Recent work by Teuopolllos [1 983]takes break points intu account and even Jiscovers them automaucally within a single i'mage. Multiple intrinsic images may be more helpful in the interpolation process as they can serve to isolate bre<:tkpoints in the interpolation gnd more sunply, and the breakpoints can be based on physical rather than mathematical features . As an example, constder the use of color image together with the depth image.

We assume that a depth edge (a sharp change in the disparity map) in conjunction with a color edge indicates a breakpoint. Depth edges that occur independently of color changes are not used to signify breakpoints. The idea is that discontinuities may be expressed as conjunciions of spaLially-aligned intrinsic image gradients. The color edge is directional and defines two half spaces. Owing to occlusion, the edge may be usee! with local points in the nearest half space but not with local poinLs in the furthest half space (Figure 2). The idea here is that edge informaLion is asymmetrical. Color and depth gradients are only one example of these general principles.

IV. Dynamic Roundaries

Consider the optic: flow example meJltionecl earlier. In the case where the sphere does not translate the fl ow is stationary (does not change from frame to frame) , even though the intensities change. This allows the flow to be set to zero in places where t~:::: 0 and not propagated Into regions of zero flow. However, when the sphere translates across the field of view, the image intensities at the houndaries are time-varying, (ft"'O) but at the sam~: time the intrinsic property is violated (dfldt"'O). The constraint equations do not hold at the boundary. This can prove disastrous when the erroneous estimates propagate to neighboring points via relaxat ion methods.

One solution to this problem uses the time-vary ing disparity image. Large differences in tempond disparity change are likely to be occlusion boundaries. These large differences can be used to delimit areas in the tlow image where the continuity constraint df/dt =O is unlikely to hold.

Experiments in computing flow [Stuth, 1982] have shown that if the surface reflectance is varying (e .g., the surface is textured), then estimates for llow can be obtained from very local measurements. Consequently, the segmentation based on depth discontinuities can he very conservative (there can be more regions than necessary) without disturbing the t1ow computations.

V. Houndary Conditions Based on Global Criteria

A well known method for delimiting prominent color image boundaries is to use a global color space. l:or a smvey of this method, see [Shafer and Kanade, 1982]. The basic idea is to construct a histogram in a color space h(r,g,b) where h(r 0 ,g 0 ,b 0 ) is the number of ptxels 1n the image with color (r 0 ,g 0 ,b 0 ). The histogram may be processed into a set of local maxima h1 ... h 11 . Then each pixel in the image is assigned a label k where the color denoted by hk is the maximum nearest the pixel 's color. With regard to boundary conditions, there are two key points. First, a global criterion-- nearness to prominent color features in color space--is used to define boundaries. Second, the process of delimiting the boundaries may he seen as a two-way transformation: the transformation to color space defines color features and the transformaLion back to the color image defines boundaries. This process is fami liar from histogram-based segmentation schemes (e.g., [Shafer, 1980]).

The central idea of fmding boundary conditions via global transformations that is captured by the color example has been extended to several other intrinsic images [Ballard, 1983]. These applications involve several technical refinements of the color technique. The two most important are: (1) instead of a single glohal feature space there may be several that must be coniputed sequenLially; and (2) the boundary must be computed sufficiently fast to help the intrinsic image relaxation computations. Our intent here is to suppress these details but at the same time describe the basic interactions that allow boundary conditions to be computed.

A. Exam11le 1: Shape from Shading

Shape from shading has been studied extensively by Horn [ 1975] and a relaxation algorithm for surface orientation has been described by lkeuchi and Hom [1981]. In this algorithm, boundary conditions a11d reflectance map were specified. Both of these conditions can be relaxed if additional information is specified (J-ig. 3). First, oriented intensity di scontinuities (edges) constrain normals at boundaries of corresponding surfaces. 1-forever, there may be many edges that are not on the boundary of surfaces. A second constraint uses a global feature space of possible light source orientations to constrain smooth surface boundaries further. Boundaries that are inconsistent with a global light source directi on are disallowed. 'rhis algorithm has been extensively tested [Brown et al. , ln2]. The essential steps are as follows:

1. Ldges in the body give possible space curves in three dimensions. (In the experiments by Brown et al. [1n2], the three-space curve was assumed known , but the surface normal direction in the plane orthogonal to the curve was not. Methods of estimating surface Lilt from boundary have been developed (e.g., [Brady and Yuille, 19~3] .
2. An initial estimate for the illuminant, together with the constraint that normals at the boundary must be perpendicular to the space curve, determines normal estimates.
3. These estimates refine the estimate for the illuminant, and smoothness and irradiance constraints allow more normals to be determined and refined.

B. Example 2: Rigid Hody Motion

Consider the problem raised by a di sk sliding over a planar surface. Here there are only small depth changes. One approach to the 2-D problem is to invoke u rigid body constraint ~nd determine global mollon parameters by a transfonnation technique [Ballard and Kimball, 1983). Briefly, the method works as follows in 3-D (Figure 4):

1. The origin of the body, xb, is assumed to he initially positioned at the viewer.
2. Lstimates for !low and depth are used to compute estimates for three-dimensional velocities v(x,t).
3. Three successive frames of three - dimensional velocities can determine the direction of rotation w.
4. This direction, together with the 3-d velocities, can determine the magnitude of rotation.
5. Knowing lrJI. w, and vT allows xb(x,t) to be updated.
6. The parameters xb, lrJI, w, and vT, can segmen t the three-dimensional velocities (and therefore tlow) into a region where u is consistent with the parameters and a region where u is inconsistent with the parameters.

VI. Conclusions

The original concept of a boundary in the computation of multiple intrinsic images was a luminan..:e boundary [Barrow and Tenenbaum, 1978]. Since then in the process of computing intrinsic images, the general problem of having appropriate boundary conditions has become more apparent. However, intervening work has concentrated on boundary conditions in single intrinsic images. Our work shows that the collaborative computation of several intrinsic images allows the possibility of: (1) boundary conditions being cletined as conjunctions of multiple intrinsic properties; (2) a well ­ defined boundary in one image helping the computation of another; and (3) boundaries being determined by global featmes.