From [R, G, B] to Surface Reflectance: Computing Color Constant Descriptors in Images

1 Introduction

One of the most interesting perceptual abilities of the human visual system is to assign the same colors to objects under dif­ ferent lighting conditions. In other words, a human observer looking at a certain scene perceives the colors of surfaces in a consistent way, although the spectral distribution of the illu­ minant may vary considerably. This degree of independence of perceived object color on the illuminating spectrum is called color constancy.

In this paper, we describe an algorithm for generating color constant descriptors of surfaces. The idea is to transform an input image representing the intensities reflected from objects into a color constant image which is a representation of re­ flectances, therefore independent of illumination effects. When put formally, the problem is to find a conversion matrix A that transforms the original [R,G,B] values of the image into color constant descriptors C. If the original [R,G,B] values of the image are defined as PR, Po, and PB, denoted asP, then the following relation can be formulated:

The main goal of the work described in this paper is to find the values of the vector C, which for each pixel, is a set of (three) numbers representing the color descriptors of the material, re­ gardless of the illumination.

The approach we take is similar to the one forwarded by Buchsbaum [1]. Thus, it is a two-step process: estimating the illuminant, and using the estimated illuminant to obtain the descriptors. The method makes use of a finite-dimensional linear model which represents light sources and reflectances. Using this model, we show how an estimate to the illuminant is obtained, and consequently, color constant descriptors. In the data leading them to suggest a linear model of light with three degrees of freedom. Therefore our choice of three basis functions describing the luminance will be the mean and first two characteristic vectors, which will be denoted as I,(> .. ) where I = 1, 2, 3 and .>. E (380, 770).

· Also, Canadian Institute for Advanced Research.

To combine the basis functions into the linear model, we ta.ke the product of the three functions. This is due to the fact that reflection is a product of illumination and reflectance, and to measure the reflection one needs sensitivity curves to filter the reflected intensities. Therefore a three-dimensional tensor T;,t,j is defined as:

It is important to note that the tensor is a fixed system of constants, and does not depend at all on the materials being viewed.

3 Estimating the Illuminant

Buchsbaum (1) proposed a partial solution for the recovery of the illuminant: he assumed that the average reflected intensi­ ties (a 3 x 3 x 3 system along the lines of T;,t,;) corresponds to the one obtained from the actual illuminance acting on a standard homogeneous field, internally embedded in the model. This standard internal reflectance was assumed to approximate the entire field average reflectance. There are two assumptions Buchsbaum made which wea.ken the solution considerably: (1) the tensor T;,t,; wa.s picked in an ad hoc manner, without any relevance to naturally-occurring reflectances and illuminants; (2) a fixed internal reflectance vector for the overall actual field average wa.s built into the system. This vector was chosen to be equal reflectance at all wavelengths, implying that the overall mean pixel values of all images are grey. Moreover, there is no indication to whether this vector should be updated if the scene is known to be very different from normal rich scenes (e.g., a forest full of green colors). -

The way we suggest to estimate the illuminant is to match a system representing a general model of material reflectances with an average of reflected intensities exhibited by the mate­ rials within a given image. This poses two problems which we address in the following sections:

1. What can be considered as a "general" set of materials which includes, with high probability, most of the mate­ rials found in scenes?
2. How should the image average reflectance be generated such that it will exhibit dependency on the illuminant and represent all materials equally?

3.1 "Ideal" Material Space

In order to be able to treat most of the naturally occurring ma­ terials, one has to find a large ensemble of materials and record their spectral properties. One such study was conducted by Krinov (5), who measured and documented the spectral se~­ sitivity curves of 370 materials of various types, such as sotl, vegetation, and water. This sample is "ideal" since the more materials it contains, the closer its average will be to the .aver­ age of the sample being examined (the image). As ment10ned in Section 2, the characteristic vectors suggested by Cohen [2) represent a good approximation of reflectances. Therefore we suggest to represent this ideal material space with th~ ~hree characteristic vectors and to compute the average of this tdeal space in terms of the' characteristic vectors. Thus, if K ~o( .>.) is the k-th material in Krinov's sample set, then its representation using Cohen's basis functions is:

where Q is a 3 x 3 matrix defined as Qa,b = L:r~380 Ra(>.)Rb(>.). Thus the average of all materials 0 is a vector computed as:

yielding a triplet of numbers which are the average of the ideal material space a.s represented by three characteristic vectors.

3.2 Average Image Reflectance

There are two ma.in requirements being imposed on the average image reflectance. The first requirement is that it should de­ pend on the illumination and its effects on the scene. The mo­ tivation behind this requirement is that different light sources will affect the scene differently, resulting in different reflected intensities. It is therefore desirable to generate an average which will be sensitive to these changes and represent each light source differently. The second requirement is that all ma­ terials should be represented equally, no matter how large a regio~ they occupy in the image. This is due to the fact that in the ideal material space presented in the previous section, each material is represented only once. Since our technique tries to find correspondence between the ideal space and the image space, it is desirable that the image space be sampled in the same manner.

To satisfy both requirements, we chose a method which seg­ ments the image into regions which differ from each other in their chromatic appearance. The averaging of the image re­ flectance is computed by segmenting the image into regions which differ from each other according to some criterion of ho­ mogeneity (set by the segmentation scheme). Each meaningful region is then averaged, and this average is added up to the total average of the image reflectance. The total average is computed for all three dimensions (i.e., [R,G,B)) and will be denoted as V. Details regarding the segmentation are given in [3).

4 The Algorithm

We now turn to describe the actual algorithm for computing color descriptors in detail. As mentioned before, the proposed technique consists of two steps: estimating the illuminant vec­ tor £, and finding the transformation matrix A, which will en­ able the conversion from [R,G,B)-space to color constant space, which will be referred to a.s C-space. The two steps will be dis­ cussed in deta.il below.

4.1 Step 1 - Computation of €

1. Compute the average image reflectance V.
2. (a) Segment the image into regions using some homo­ geneity criterion.
3. (b) For each region, compute the averag: (R,G,B) values and add to the average reflectance V.
4. Compute l.
5. (a) Define the 3 x 3 matrix B( U) to be a representation of the re:llectances in the image space in terms of

the linear model Ti,l,j· Thus each element of B is defined as:

• (b) Find the correspondence between the average image reflectance (V) and the average of the ideal material space (iJ). The underlying assumption is tha.t there are enough materials in the image to justify such a. correspondence. Formally:

The estimated illuminant is therefore:

4.2 Step 2 - Computation of A

1. Define the 3 x 3 matrix A to be the fundamental linear model T;,l,j as updated by illumination vector l. Thus A has the form:

1. To obtain the specific C-values of the image, reorganize Equation 1 into:

The output of this step is an "image" of C-values; tha.t is, each pixel value in the image is transformed from (R,G,B) to a. new set of three numbers: C1o C2, and Ca.

4.3 Discussion

One of the assumptions ma.de, used in Step 1 of the algorithm, is tha.t the image reflectance spa.ce has enough materials to allow the transformation to the ideal material space. The mo­ tivation behind this assumption was the fact tha.t the ideal material space contains a. rich ensemble of materials, and thus its average is likely to be similar to averages of many scenes. The implication of this assumption is tha.t the accuracy of the algorithm depends on the richness in materials of the input image. Clearly, the situation can be simplified if there exists a priori knowledge a.bout the domain from which the images are ta.ken, since the ideal material space can be reduced to in­ clude materials which are ekpected to be in the input images. For example, if it is known in a.dva.nce tha.t forests will be the subject of the images, then only materials which ma.y appear in forests, such as leaves and wood, should be included in the ideal material space, thus increasing the accuracy of the algorithm.

The C-values computed by the algorithm provide a. mea.ns of recovering the actual reflectance of the material, as repre­ sented by the reflectance basis functions. Since the C-values are color descriptors which are constant regardless of the illu­ minant, and since they were computed using the linear model, it is a. straightforward computation tha.t allows the recovery of the reflectances. H we define the recovered reflectance as RR(~), then this computation is simply:

5 Experiments

In order to evaluate the effectiveness of the algorithm, a. few experiments were designed and carried out on simulated and real images. Although each experiment was designed with a. different goal in mind, the overall goal was to demonstrate the robustness and accuracy of the algorithm.

5.1 Accuracy as a Function of Knowledge About Materials

The first goal was to evaluate the accuracy of the algorithm and the claim ma.de earlier tha.t a priori knowledge a.bout the materials being viewed increases the accuracy of the results. Therefore the experiment was intended to compare the real C­ values of materials to the ones output by the algorithm given different illuminants. The real C-values were computed in the following wa.y. Seven materials were chosen a.t random from the da.ta. collected by Krinov [5) and their real C-values were computed according to Equation 3. In addition, knowing tha.t only the seven materials chosen were going to be used in the experiment, the ideal space was reduced to include only those materials, thus resulting in an updated set of averages of this space.

As pointed out, the real C-values were to be compared to the C-values computed by the algorithm. The input to the algo­ rithm was computed in the following way. The materials were combined into a. Mondrian. Since the only information about the materials was their spectral reflectance curves, there was a. need to create an (R,G,B) image out ofthese reflectances, using some random illuminantsl, and assuming some [R,G,B) filters. Therefore the [R,G,B) values of the patches were computed as:

where S;( ~) represents the sensitivity curves of the filters, and I,(~) represents the characteristic vectors describing light source~ The [R,G,B) image was used as input to the algorithm. As are­ sult of Step 1, an estimate of the illuminant was computed. The comparison between the illuminant we used in the experiment I, and the estimated illuminant as computed by the algorithm fl are depicted in Figure 1. In Step 2 of the algorithm, the

C-values were computed for ea.ch ~naterial. The absolute error in the C-values compared to the true C-values are given in Ta­ ble 1. As can be observed from the figure, given the a priori knowledge a.bout the materials, the results of the algorithm are accurate.

5.2 Constancy Under Different Illuminants

In this experiment the goal was to verify that the C-values generated by the algorithm were indeed constant under differ­ ent illumination conditions. Therefore a simple method was used: the same scene was photographed and digitized under two different illuminants; the first was illuminated with white

I

The numbers in each box, from top to bottom, represent C1, C2 , and Ca respectively. All numbers were multiplied by a factor of 100.

incandescent light (termed a.s Image #1) a.nd the second with the sa.me light source with a. yellow filter in front of it (Image #2). 1 For ea.ch image, the a.lgorithm wa.s run and C-va.lues of the sa.me region were recorded. The results a.ppea.r in the upper portion of Figure 2. We a.lso plotted the recovered il­ luminants a.s computed by the a.lgorithm (Figure 2 - center portion), and the estimated reflecta.nces in ea.ch a.rea. in both images (Figure 2 - bottom portion). It is clear from the ta­ ble and the plots tha.t the C-va.lues generated a.re very close, a.s a.re the estimated reflecta.nces. In addition, the estimated illurnina.nts do exhibit curves which a.re close to approximating white a.nd yellow illurnina.nts, respectively. Note tha.t the es­ timates of the reflecta.nces, a.s well a.s the illumina.nts, a.re not a.ccura.te due to the fa.ct tha.t the sample of "ma.teria.ls" is not rich enough (there is only one materia.!!). Nevertheless, the color descriptors a.re fairly constant, indicating the robustness of this technique.

6 Summary

This paper dea.lt with the problem of color constancy, or the fa.ct tha.t the perceived color of surfaces tends to remain con­ stant despite changes in illumination tha.t a.lter the intensities reflected off the surfaces. We ha.ve presented an a.lgorithm which generates color constant descriptors for different surfaces. There a.re two steps in the a.lgorithm: the first estimates the il­ lumina.nt present in the scene, a.nd the second step makes use of the estimated illumina.nt to generate the color descripto-rs. We ha.ve shown tha.t by computing a.n image a.vera.ge reflectance, the a.lgorithm ca.n estimate the illumina.nt present in the scene a.nd with it, produce color descriptors for ea.ch of the surfaces in the image. Although the sa.me ma.teria.ls ma.y exhibit differ­ ent reflected intensities under different illumination conditions, the a.lgorithm produces the sa.me descriptors in ea.ch ca.se. The main result of the a.lgorithm is tha.t it allows the treatment of surfaces independent of the spectra.! power of the light source affecting them.