Color Algorithms for a General Vision System

1. Introduction

A general vision system must be capable of generating meaningful descriptions of a scene in any of the diverse environments for which human vision is useful. Nearly all existing artificial vision systems are special purpose. They rely heavily on domain-specific constraints and special-case engineering. Although significant progress has been made in machine vision, the best artificial vision systems still fall far short of achieving the capabilities associated with general vision.

In this paper, we analyze the role and use of color in a general vision system. We have previously examined the physics of reflection to develop general models which de­ scribe the formation of color images (4] . Using these models, we isolate the invariant properties of objects with respect

to color. Once these invariant properties are understood, we derive general procedures which reliably extract these invariant properties from images.

2. The Physics of Reflection

There are several ways an object can modify incident light . An object can change light spatially by reflecting it into a small or large angle. Incident light can be modified spectrally. Also, the energy of incident light can be reduced where the specular component of the reflected light be­ comes significant. The most conspicuous feature of spec­ ular reflection is that it is invariably associated with image irradiance values which are much larger than those in neigh­ boring image regions. Moreover, the expected magnitude of the difference in irradiance is directly related to proper­ ties of the imaging system which are often known. Another quasi-invariant property of specular features is that they are typically small, especially for curved surfaces. Thus unless a specularity occurs at the edge of a surface, it will be surrounded on all sides by diffusely reflected light of ap­ proximately the same color and power. Finally, we observe that most specular discontinuities occur in places where the reflecting surface is continuous. It has been shown that im­ age irradiance for diffusely reflected light can be used to compute local descriptions of a surface to at least second order [6]. It has also been shown that image irradiance for specularly reflected light can be used to compute similar local surface descriptions [3]. At specular image irradiance discontinuities, we usually expect the first few derivatives of the surface to be continuous. Thus by comparing the local surface descriptions generated by [6] and [3] we have another way to verify the presence of a specular image ir­ radiance discontinuity.

4. Generic Classification of Materials

In this section and the next, we describe procedures for using color to extract distinctive invariant properties of objects from images. In this section, we show how color can be used to recover a symbolic description of the material an object is made of. In section 5, we describe a method for recovering an object's surface spectral reflectance.

Classifying objects according to material is important because material is an invariant property of an object. It is very valuable, for example, to be able to decide that an object is metal rather than plastic or painted wood rather than dyed cloth.

An important property of a material is whether it is optically homogeneous or optically inhomogeneous. By ex­ amining the physics of reflection, we have derived a pro­ cedure for classifying a material as either homogeneous or inhomogeneous.

Homogeneous materials reflect light only from the sur­ face. The color of this reflected light is determined by the Fresnel equations from the complex index of refraction of · the material as a function of wavelength. For a single color of illumination L(A), the color of light reflected from a ho­ mogeneous material will be nearly constant with only slight variations due to changing gem·~etry.

Inhomogeneous materials both reflect light from the surface and scatter light from the body of the material. The color of the light reflected from the surface is determined by the index of refraction of the vehicle. The color of the light scattered from the body is determined by the selective absorption properties of the colorant particles embedded in the vehicle. In general, the color of the surface reflected light will be different from the color of the body scattered light . Therefore, given a single color of illumination L(A) , there will be two distinct colors of light reflected from an inhomogeneous material.

Using the technique discussed in section 3, we are able to find image irradiance discontinuities corresponding to the places where the power of the specularly reflected light becomes significant. Using our metric for color space [4], we can examine whether these image irradiance discontinu­ ities coincide with discontinuities in color space. If a color discontinuity is not detected, there is strong evidence for a homogeneous material. If we do detect a color discontinu­ ity, then the material is probably optically inhomogeneous.

We see that in most situations, it is possible to distin­ guish homogeneous materials from inhomogeneous materi­ als using techniques in color space. Once the homogeneous­ inhomogeneous classification has been made, it is possible to use color to distinguish different homogeneous materials, e.g. aluminum and copper, and to distinguish different in­ homogeneous materials, e.g. white plastic and red plastic. Our method to achieve this additional level of classifica­ tion is based on recovering· surface spectral reflectance. We describe our method in the next section.

5. Recovering Surface Spectral Reflectance

Another invariant property of an object which is valu­ able for recognition is the object's surface spectral re­ flectance. Unfortunately, surface spectral reflectance is not determined by the spectral distribution of the light reflected by a surface. It is this reflected light which is directly sensed by a vision system. The light which is reflected by a surface is the product of the spectral distribution of the incident light and the spectral reflectance of the surface. To recover the spectral reflectance of a surface, some mechanism must be available to factor out the effects of the incident light. Many experiments have shown that the human vision sys­ tem is capable of making this computation. This ability of humans to see objects as having a constant color despite varying illumination conditions is called color constancy.

Many theories have been advanced to explain color constancy [1]. Our approach is based on the physics of reflection. Another class of approaches views the task as an underconstrained mathematical problem [8], [12] . These approaches identify the assumptions about incident illumi­ nation and surface spectral reflectance which are required to make color constancy possible from a purely computational point of view. In other work, psychologists have suggested that the eye selectively adapts to the color of the ambient light. Experiments have shown that this selective adapta­ tion might be partly responsible for human color constancy [5].

Our method for recovering surface spectral reflectance is applicable to instances of surfaces which are illuminated by the same spectral distribution of light as that which illuminates an inhomogeneous object in the scene. This condition is quite general , and is almost always satisfied in real situations where a small number of different illuminants contribute to the image forming process.

For inhomogeneous materials the complex index of refraction is nearly constant across the visible spectrum. From the Fresnel equations, the specular reflectance of an inhomogeneous material is a constant function of wave­ length for fixed geometry. Moreover, the Fresnel equations tell us that the specular reflectance for fixed wavelength is constant with respect to geometry for almost all incidence angles. Therefore, for inhomogeneous objects the Fresnel

component of the reflectance can be regarded as constant with respect to both geometry and wavelength.

Given the physical segmentation technique (section 3) and our technique for classifying inhomogeneous materials (section 4), we can recover surface spectral reflectance up to a multiplicative constant. We use the previously described procedures to locate a specular-body reflection boundary on an inhomogeneous object. From (2.1) and (2.2), the measured function /(~) is given by

On the body reflection side of the boundary, Rs(~) = 0 giving

where Rs(~) can be considered to be the same as in (5.1) since Re(~) changes slowly with respect to geometry. We can solve for Rs(~)L(~) using

But Rs(~) is constant with respect to~ and geometry. Let Rs(~) = k. Therefore, using (5.3) we can compute L(~) up to the constant k by

From (5.2), Rs(~) can now be computed up to the constant 1/k by

For a homogeneous material in the scope of L(~), we have

because Rs(~) = 0. Thus we can computeRs(~) to within 1/k by

Therefore, using (5.5) and (5. 7) we can compute the surface spectral reflectance for any surface illuminated by L(~) up to a multiplicative constant. Examples of the performance of this method on real images ~ill be given in section 6.

It should not be surprising that there is a fundamental ambiguity in the computed spectral reflectance correspond­ ing to the constant k. From (2.2)

We see that an arbitrary constant t can be introduced into both L(~) and R(~) such that the resultant /(~) will be indistinguishable from /(~) in (5.8):

Thus without using additional assumptions, we cannot ex­ pect to determine R(~) better than to within a multiplica­ tive constant. We note that our notion of normalized phys­ ical color [4] is independent of these scaling constants.

6. Experimental Results

A simple laboratory setup has been used to test our color methods. We digitize color images using a solid-state camera and four gelatin filters. The camera is equipped with an infrared cutoff filter.

Two different light sources have been used for exper­ iments. One is a tungsten halogen lamp of color temper­ ature 3400°K which is typical of indoor illumination. The other lamp has color temperature 4800° and is intended to simulate daylight.

Several simple objects have been used to test our color methods. Our objects include plastic cups, metal cylinders, and painted wooden blocks.

In Figure 1, we show the performance of our algorithms on color images of plastic cups illuminated by the 4800° K color temperature lamp. Since the image irradiance corre­ sponding to the specular reflection is markedly larger than

the image irradiance corresponding to the diffuse reflection, the physical segmentation process of section 3 easily locates the specular-diffuse boundaries. The method of [4] is then used to recover the function /(~) for both the specularly reflected light and the diffusely reflected light. Figure 1(a) is/(~) for the Fresnel reflection from a blue cup. It agrees well with the actual color of the light source. Figure 1(b) is /(~) for the diffuse reflection from the blue cup. From the large color difference between Figure 1(a) and Figure 1(b), the algorithm of section 10 easily is able to identify the plas­ tic cup as being made of an inhomogeneous material. We remark that it would be very difficult to infer that the cup is blue by simply inspecting the color of the reflected light /(~) in Figure 1(b). In fact, the largest amount of power is in the red part of the visible spectrum (near 700 nm) . To

I

I

Figure 1(c). Computed Reflectance for blue cup

determine the color of the cup (as distinct from the color of the light reflected from the cup), we must compute the sur­ face spectral reflectance. This is done using the method of section 5. Figure 1(c) shows the spectral reflectance com­ puted for the blue cup. From Figure 1(c), we can tell that the cup is blue.

Acknowledgements

This work has been supported by an NSF graduate fellowship, AFOSR contract F33615-85-C-5106, and ARPA contract N000-39-84-C-0211.