Natural Basis Functions and Topographic Memory for Face Recognition

1 Introduction

The problem of object recognition has been a central subject in the field of computer vision. An especially interesting albeit difficult subproblem is that of recog­ nizing human faces . In addition to the difficulties posed by changing viewing conditions, computational methods for face recognition have had to confront the fact that faces are complex non-rigid stimuli that defy easy ge­ ometric characterizations and form a dense cluster in the multidimensional space of input images. One of the most important issues in face recognition has therefore been the representation of faces. Early schemes for face recognition utilized geometrical representations; promi­ nent features such as eyes, nose, mouth, and chin were detected and geometrical models of faces given by fea­ ture vectors whose dimensions, for instance, denoted the relative positions of the facial features were used for the purposes of recognition [Bledsoe, 1966; Kanade, 1973]. Recently, researchers have reported successful results us­ ing photometric representations i.e. representations that are computed directly from· the intensity values of the input image. Some prominent examples include face representations based on biologically-motivated Gabor filter "jets" [Buhmann et al., 1990], randomly placed zeroth-order Gaussian kernels [Edelman et al., 1992], isodensity maps [Nakamura et al., 1991], and principal component analysis (PCA) [Turk and Pentland, 1991; Pentland et al., 1994].

*This work is supported by NSF research grant no. CDA8822724, NIH/PHS research grant no. 1 R24 RR06853, and a grant from the Human Science Frontiers Program.

This paper explores the use of an iconic representa­ tion of human faces that exploits the dimensionality­ reducing properties of PCA. However, unlike previous approaches employing PCA for recognition [Turk and Pentland, 1991; Murase and Nayar, 1995], our appro~ch uses a fixed set of basis functions that are learned dunng an initial "development" phase; the costly and time con­ suming step of having to recompute basis functions when new faces (or other objects) are encountered is thereby avoided. In addition, the basis functions used to generate face representations are based on PCA of localized natu­ ral image patches at multiple scales rather than PCA of entire face images at a single scale; the localized nature of the representation helps to make it tolerant to mi­ nor changes in facial expressions and partial occlusions while the multiscale structure allows strategies for scale in variance.

The iconic face representations are formed from n­ dimensional photometric feature vectors comprised of the responses of m derivative of Gaussian basis functions at a range of orientations, each at k scales (n = mk); for the experiments in the paper, nine basis filters at five scales were used generating a forty-five element re­ sponse vector characterizing the local image regio~ at .a point of interest. We have previously shown these tcomc feature vectors to be useful for active vision [Rao and Ballard, 1995a], visuomotor learning [Rao and Ballard, 1995b], and general object indexing [Rao and Ballard,

1995c]. Here, we show that such a representation may be used for the difficult problem of expression-invariant face recognition as well.

A face is represented by a collection of iconic fea­ ture vectors automatically extracted from specific loca­ tions in the input image (Section 3) . A topographically­ organized sparse distributed memory is used to learn the association between the appearance of a face as given by its feature vectors and the identity of the face (Section 4). Implementation of the recognition scheme is achieved using a Datacube MV200 pipeline image processor for both real-time visual preprocessing as well as indexing into the face database, utilizing frame-rate convolutions for distance computations (Section 5) . We present pre­ liminary results on the performance of the method on a face database of 140 images from 20 different persons ex­ hibiting a range of facial expressions; a recognition rate of 93.3% was achieved by the method when a set of 33 points were used for characterizing a face.

2 Nat ural Basis Functions

Images of natural scenes, unlike random collections of pixels, are characterized by a high degree of statistical regularity. For instance, pixel values in a given neigh­ borhood tend to be highly correlated owing to the mor­ phological consistency of objects. Thus, a pixel-wise rep­ resentation of objects obtained from a camera is highly redundant and some form of redundancy reduction is desirable.

2.1 Redundancy Reduction via Principal Component Analysis

The optimal linear method (in the least mean squared error sense) for reducing redundancy is the Karhunen­ Loeve transform or eigenvector expansion via Princi­ pal Component Analysis (PCA) . PCA generates a set of orthogonal axes of projections known as eigenvectors or principal components of the input data distribution in the order of decreasing variance. The eigenvectors form a set of orthogonal basis functions for represent­ ing the input. In particular, consider a set of input im­ ages of size N x N represented as N 2 -dimensional vec­ tors ,h, ... , f.. and the corresponding mean-centered set of vectors I;_, ... , fn obtained by subtracting each pixel value from the mean value for that pixel over all input images:

PCA achieves decorrelation by extracting n N 2 - dimensional eigenvectors ei such that the variance of projections along the direction ei

is maximized subject to the orthonormality condition

In principle, all n :$ N 2 eigenvectors are needed in or­ der to completely represent the input image set but due to the statistics of natural images, it is usually the case that only a small number m of eigenvectors ( m < < n) account for almost all of the variance in the input data. Thus, by using only the first m dominant eigenvectors as basis functions (or orthogonal axes) for projecting new inputs, considerable computational savings can be achieved.

Turk and Pentland [1991] used PCA to synthesize the eigenvectors ( "eigenfaces") of a training set of face images; they achieve recognition by using a template­ matching strategy with the vectors obtained by project­ ing new face images alon~ a small number of eigenfaces. Murase and Nayar [1995j applied PCA to the problem of object recognition and pose estimation; they repre­ sent objects as manifolds in the low-dimensional sub­ space ("eigenspace") formed by the dominant eigenvec­ tors of a set of training images and achieve recognition by finding the manifold that is closest to the projection of an input image in the eigenspace formed by all objects.

2.2 Unsupervised Learning of Basis Functions

The methods of [Turk and Pentland, 1991] and [Murase and Nayar, 1995] both require recomputation of the eigenvectors when new faces/objects are encountered. It is therefore natural to ask what the results of PCA would be if one were to take the above process to its limit i.e. to perform PCA on a set J;., .. . , f.. of arbitrary natural images containing a wide variety of natural and man­ made stimuli. Recently, Hancock et al. [1992] used a neural network introduced by Sanger [1989] to extract the first few eigenvectors of an ensemble of natural im­ ages. They discovered that the eigenvectors were very close approximations of different oriented derivative-of­ Gaussian operators.

We employed Sanger's PCA network to ascertain whether the results of Hancock et al. remained true for collections of images containing equal proportions of natural and man-made stimuli. In particular, we used 32 x 32 Gaussian-windowed image patches obtained by scanning across a number of arbitrary images of natu­ ral scenes (Figure 1 (a)). Suppose l represents an in­ put mean-centered image patch and Wj represents the weight vector from the input layer (which in our case consists of 1024 units) to the output unit j . Sanger's PCA network uses linear output units i.e.

When presented with input f,· the network adapts its weight vectors according to a generalized Hebbian Learn­ ing rule:

where "Y > 0 is a gain term whose value is decreased gradually as training progresses. Figure 1 (b) shows the eigenvectors to which the weight vectors Wj converged

(c)

Figure 1: (a) Twelve of the 20 images that we used for train­ ing Sanger's PCA network. (b) First nine dominant eigen­ vectors that the weights of the network converged to, shown here for different scales (a) of the Gaussian window (intensity is proportional to magnitude). (c) The Gaussian derivative basis functions of up to the third-order used in our iconic representations. The first few dominant eigenvectors of nat­ ural images shown in (b) closely resemble these analytically derived function profiles.

after 12000 presentations. These nine eigenvectors alone account for as much as 83% of the input variance. It is clear that regardless of the scale of analysis, the eigen­ vectors closely approximate different oriented derivative­ of-Gaussian operators.

In summary, the derivative-of-Gaussian filters are well-suited for use as natural basis functions for general­ purpose visual recognition because: (a) they are ob­ tained as result of applying PCA to arbitrary collec­ tions of images containing diverse elementary features from natural as well as man-made structures rather than just the images of particular objects or faces. In ad­ dition, correlation filters generated via PCA have been shown to maximize signal-to-noise ratio and yield much sharper correlation peaks than traditional raw image cross-correlation techniques [Kumar et al., 1g82]; (b~ they form the class of real-valued functions that simul­ taneously minimize the product of the standard de­ viation of the spatial position sensitivity and spatial frequency sensitivity ([Gabor, 1g46] p. 441) 1 ; and (c) they are endorsed by neurobiological studies [Young, 1g35] which show that the different order derivative-of­ Gaussian functions provide the best fit to primate corti­ cal receptive field profiles among the different functions suggested in the literature.

1 The class of complex-valued functions that minimize this conjoint localization metric are the well-known Gabor ele­ mentary functions [Gabor, 1946] .

3 Iconic Representations of Faces

The iconic representations used in our recognition scheme are based on the natural basis functions men­ tioned in the previous section. The exact number and type of Gaussian derivative basis functions used is moti­ vated by the need to make the representations invariant to rotations in the image plane. This invariance can be achieved by exploiting the property of steerability [Free­ man and Adelson, 1gg1] and using a minimal basis set of two first-order directional derivatives at 0° and goo, three second-order derivatives at 0°, 60° and 120°, and four third-order derivatives oriented at 0°, 45°, goo, and 135° (Figure 1 (c)). We omit the zeroth order to reduce illumination dependence and do not use higher orders since variance of the higher-order filters can be expected to approach that of image noise as suggested by the re­ sults of PCA. The use of the non-orthogonal oriented filters also obviates the use of mixed derivatives (some of which were obtained in Figure 1 (b)) since the other oriented filters yield a complete basis.

3.1 Representing Image Regions

The current implementation uses nine Gaussian direc­ tional derivatives denoted by

where n denotes the order of the filter and On refers to the preferred orientation of the filter. The response of an image patch I centered at (x0 , y0 ) to a particular basis filter �~' can be obtained by convolving the image patch with the filter :

The iconic representation for the local image patch centered at (xo, Yo) is formed by combining into a single high-dimensional vector the responses from the nine ba­ sis filters, each (in the current implementation) at five different scales:

where i = 1, 2, 3 denotes the order of the filter, j = 1, . .. , i + 1 denotes the different filters per order, and s = Smin, . . . , Smax denotes the different scales (as given by the levels of a low-pass filtered image pyramid) . The use of multiple scales increases the perspicuity of the rep­ resentation and allows interpolation strategies for scale invariance (see [Rao and Ballard, 1ggsa] for more de­ tails). In addition, the high-dimensionality of the vec­ tors makes them remarkably robust to noise due to the orthogonality inherent in high-dimensional spaces: given any vector, most of the other vectors in the space tend to be relatively uncorrelated with the given vector [Rao and Ballard, 1gg5c].

The iconic representations can be made invariant to rotations in the image plane (for a fixed scale) by ex­ ploiting the property of steerability [Freeman and Adel­ son, 1gg1]. The current orientation is computed using the first-order responses as:

(a)

and the entire set of filter responses is rotated to a canon­ ical orientation (in this case, horizontal) by using linear combinations of the original responses:

where i = 1,2,3;j = 1, ... ,i + 1; 8 = Smin 1 ··· ,Smax and kj'i are interpolation functions given by :

and

with j' = 1, ... , 4. Note that this normalization proce­ dure does not apply to rotations in 3D. Modest rotations in depth can be handled as noise by the representation but larger 3D rotations require the use of responses from multiple views as we have shown in [Rao and Ballard, 1995c]. This view-based approach is similar in spirit to the one used by Beymer [1993] (see also [Pentland et al., 1994]).

3.2 Representing Faces

The response vectors described in the previous section serve as iconic descriptions of individual image regions. In order to represent a given model face with a set of such vectors, the problem of selecting suitable points within a face fro~ which model response vectors can be extracted must be addressed. In our current implementation, we apply the following simple strategy after the approxi­ mate boundary of the face is determined (by using, for instance, stereo and a technique such as zero disparity filtering [Coombs, 1992]):

A face is represented by response vectors from the centroid of the face and each of the points lying on the intersections of radial lines with concentric circles of exponentially increas­ ing radii centered on the centroid.

This strategy is illustrated in Figure 2. Only points ly­ ing within the approximate face boundary are used. This method ensures a dense representation of the "foveal" re­ gion near the centroid while at the same time including information from the "parafoveal" regions as well. Note that this strategy is a form of representation by parts. Also, due to the tendency of nearby points to be highly correlated, the above method does not require the cen­ troid and the other points of an input face to be precisely registered with the corresponding points from a model face; an approximate registration suffices in most cases.

4 Topographic Sparse Distributed Memory

For the task of face recognition using the iconic repre­ sentations discussed above, two requirements need to be met: (a) a mode of long-term storage for the represen­ tations of model faces, and (b) a method for learning the association between the representation of a face and the identity of the face. Both these requirements are met by using a modified form of sparse distributed mem­ ory (SDM), first proposed by Kanerva [1988; 1993] as a model of human long-term memory.

4.1 Operation of the SDM

The organization of sparse distributed memory (Fig­ ure 3) is similar to that of conventional random-access memory. There exists an array of data storage lo­ cations, each identified by a number (the address of the location). However, the address vectors are usu­ ally high-dimensional and hence only a sparse subset of the address space is used for identifying data loca­ tions. Suppose the SDM contains M storage locations where the address vectors are p-ary and n-dimensional, and the data vectors are derived from the set { -1, 1}k. For our current purposes, the feature vectors describing faces correspond to the addresses of the SDM while the data being associated with the feature vectors represent the identities of the faces, each person being defined by a small interval of possible values. Specific values of the data vector may be interpreted as corresponding to a particular facial expression of a person.

Organization of the Address Space

Kanerva suggests randomly picking M unique vectors Ai from the pn possible address vectors for addressing each of the M data storage locations of the memory. However, in our case, the set of response vectors will be clustered in many correlated groups distributed over a large portion of the response vector space. Therefore, if addresses are picked randomly, a large number of loca­ tions will never be activated while a number of locations will be selected so often that their contents will resemble noise. The solution is to. pick addresses according to the distribution of the data [Keeler, 1988]. In our case, we simply use an initial subset of the training response vec­ tors as the addresses. When all address locations have subsequently been filled, the address space can be al­ lowed to self-organize using the well-known competitive Hebbian learning rule: given a new input vector r, the closest addresses Ak are adapted according to

where 17 > 0 is a gain term and g is a radial basis func­ tion [Poggio and Girosi, 1990] that weights the second summand according to the distance between r and Ak . However, for the experiments in this paper, we did not employ the above self-organization rule.

Activation of Data Locations

The distance between response vectors r 1 and f2 is de­ fined to be their normalized dot product (or correlation):

Given a response vector r for indexing into the mem­ ory, all storage locations whose addresses lie within a distance of D from rare selected (the selected locations are indicated by ones in the vector s in Figure 3). Note that in general, an arbitrary radial basis function such as a Gaussian may be used to obtain the components of s instead of a strict binary threshold function. This allows for smoother interpolation between stored data vectors especially when they are used to indicate facial expression in addition to identity.

Hebbian Learning of Identity

During the training phase, the input response vector r is used to find s and the associated identity vector dis added to the previous contents of each of the selected storage locations. This corresponds to a form of gener­ alized Hebbian learning as pointed out in [Keeler, 1988]. Note that this is different from a conventional memory where addresses are required to exactly match for se­ lection and previous contents are overwritten with new data.

Retrieval of Identity

After training, the memory can be used to yield hypothe­ ses for the identity of the object given a response vector r. First, the locations selected for rare found as above and the values of these selected locations are added in parallel (vector addition) to yield a sum vector § con­ taining the k sums. These k sums are thresholded at 0 to obtain the data vector J i.e. d~ = 1 if Si > 0 and d~ = -1 otherwise.

The statistically reconstructed data vector should closely resemble the original data vector (or some linear combination of the stored vectors in the case of inter­ polation between stored facial expressions) provided the capacity of the SDM [Kanerva, 1993] has not been ex­ ceeded. The intuitive reason for this is as follows: When storing d using r, each of the selected locations receives one copy of the data. During retrieval with an address close to r, say f", most of the locations that were se­ lected with r are also selected with f". Thus, the sum vector contains most of the copies of J, plus copies of other different words; however, due to the orthogonality of the address space for large n, these extraneous copies are much fewer than the number of copies of J. This biases the sum vector in the direction of d and hence, dis output with a high probability (see [Kanerva, 1993] for a more rigorous argument based on a signal-to-noise ratio analysis).

4.2 Topographical Organization of Memory

Topology of the model points, as given by the concentric circular template (Figure 2), is preserved by using sep­ arate SDMs for storing vectors from each of the sparse number of locations on a face. The final output of the memory is obtained from the cumulative sum vector over the SDMs for the different facial locations. This arrange­ ment offers at least two advantages over using a single SDM for storing vectors as proposed earlier in [Rao and Ballard, 1995c]: (a) the crosstalk between response vec­ tors from different locations on a face is eliminated, and (b) a given response vector from a facial location needs to compared to only the model vectors in the SDM for that location, thereby speeding up the recognition pro­ cess.

5 Implementation

The face recognition scheme described above has been implemented using an active vision system comprised of the University of Rochester (UR) binocular head which (Figure 4) provides input to a Datacube Max Video™ MV200 pipeline image-processing system capable of per­ forming convolutions at frame-rate (30/ sec). The pitch of the two-eye platform is controlled by a single servo motor while separate motors control each camera's pan angle, thereby providing independent vergence control. This allows strategies for figure-ground segmentation of faces using, for instance, zero disparity filtering [Coombs, 1992]; once a face has been approximately segmented from the background, the concentric circular template of points can be centered on the centroid of the face. Given an input face image, the MV200 executes nine convolutions with the different 8 x 8 Gaussian deriva­ tive kernels on a low-pass filtered five-level pyramid of the input image; filter responses are then extracted from each of the sparse number of points whose coordinates are given by the concentric circular template.

Traditionally, the most time-consuming step during object indexing has been linearly accessing the large number of object representations in memory. However, our implementation greatly optimizes this step by imple­ menting memory directly within the pipeline image pro­ cessing system itself and using convolutions for distance computations. During indexing, an input response vec­ tor is loaded into the 8 x 8 convolution kernel and con­ volved with a "memory surface" containing the stored model vectors; the closest vectors can be selected by sim­ ply thresholding the results of the convolution.

6 Experimental Results

The first experiment (Figure 5) illustrates the discrim­ ination ability of the feature vectors. The vector ex­ tracted from a point near the approximate centroid of a given face was compared with those for five other faces. It is clear that the five vectors are relatively uncorrelated with the vector for the original face, the closest vector having a correlation of 0.43. For the SDMs, thresholds D in the range 0.80-0.95 were found to yield satisfactory re­ sults. Further discriminability is obtained by using more than one vector per face from different facial locations as previously discussed.

The next experiment examines the effect of varying facial expressions on the iconic feature vectors. Figure 6 shows a set of face images of a person exhibiting a range of facial expressions. The correlation between the vec­ tors for two different points on the neutral face image and the corresponding vectors for the other images is plotted below. The graph indicates that as expected, vectors for some facial points change much more than the others though the correlation still remains relatively high (above 0.45) . This motivates the need for using a small number of images of a person under varying facial expressions for training the memory in order to achieve expression-invariant recognition. Due to the interpola­ tion inherent in the output of the SDM, the output of the memory can then be interpreted as an indication of facial expression in addition to the identity of the person.

The third experiment tests vulnerability to occlusions. Figure 7 shows a sequence of face images with increasing facial occlusions; a plot below shows the correlations between the iconic vectors for two different points on the first face image and the corresponding vectors for the other images. The results seem to indicate that modest occlusions (as in images 2, 3, and 4) can be handled but larger occlusions (such as in 5) may require other strate­ gies such as the one suggested in [Ballard and Rao, 1994]. The results also motivate the need for using more than one point per face in order to be able to compensate for partial occlusions near specific facial locations.

Finally, we tested the recognition performance of the method by training the memory on a face database con­ sisting of images of 20 persons (Figure 8 (a)) exhibiting 6 different facial expressions (as in Figure 6 (a)). All images were of size 128 x 128, greyscale, 8 bit quantized, and taken under normal (overhead) illumination condi­ tions with the face only approximately centered in the image frame. For testing the method, we used face im­ ages of the persons exhibiting facial expressions which were not used in the training set. Figure 8 (b) and (c) give examples of success and failure of the method re­ spectively. Figure 8 (d) shows the recognition rate (the fraction of 60 test faces correctly recognized) plotted as a function of the number of points used per training face. A peak performance of 93.3% was achieved when 33 fa­ cial points were used; only 4 of the 60 test faces were incorrectly classified, with the· correct identity finishing second in 3 of these 4 cases.

7 Conclusions and Future Work

A new approach to the problem of face recognition was proposed which uses iconic representations of faces as input to a topographically-organized sparse distributed memory. The iconic feature vectors are attractive as representations of faces because:

• They simultaneously achieve the dual goals of dimensionality-reduction and orthogonality.
• They are tolerant to modest changes in facial fea­ tures or expressions due to the large number of mea­ surements incorporated in the representations.
• They allow simple strategies for rotation normaliza­ tion and scale invariance 2 .
• They can be computed efficiently on pipeline image processors such as the Datacube MV200.
• They facilitate real-time indexing of large face databases by allowing strategies such as using con­ volutions for distance computations.

The sparse distributed memory model of Kanerva was used for learning the association between feature vec­ tors of a face and its identity. This model enjoys several favorable properties such as the ability to interpolate be­ tween stored facial expressions/views of a person, theo­ retically constant indexing time (due to the fixed number M of storage locations), possibly greater storage capac­ ity over conventional linear memory, and anthropomor­ phic learning behavior in addition to the favorable prop­ erties (such as fault tolerance) that are known to accrue to distributed representations. In addition, recognition memory was topographically organized, thereby reduc­ ing crosstalk and speeding up the indexing process.

The method is clearly computation intensive; however, the recent availability of pipeline image processors sig­ nificantly ameliorates this drawback since the ability of these processors to perform convolutions at frame rate (30/sec) can be effectively exploited. For example, the feature vectors for a face can be extracted after only 13 convolutions (four for generating the low-pass filtered five-level pyramid and nine for the basis filter kernels) i.e. in approximately half a second. Indexing into the SDMs using convolutions for distance computations fur­ ther optimizes the recognition process. Storing upto 33

2 See [Rao and Ballard, 1995a] for more details.

vectors for a face may seem extravagant but note that this choice still results in considerable savings over the alternative of pixel-wise storage of images (33 x 45 ver­ sus 128 x 128). An interestihg question is whether the method will fail when extremely large model bases of faces are used. However, the use of more than one vec­ tor per face potentially allows an extremely large num­ ber of persons to be handled. Kanerva [1993] estimates the capacity of the SDM to be about 5% of the number of storage locations; thus, even with only 1000 storage locations for each SDM, the number of potentially distin­ guishable items is still 50 33 which is an extremely large number even after ruling out a significant proportion of the possible combinations as being unlikely to be en­ countered in practice. The accuracy of the above naive estimate clearly depends on the extent to which the filter response vectors are shared between the different stored model faces; while there exists some sharing in general due to the similarity of certain facial features across per­ sons, we believe that the possible use of self-organization within the address space of the SDMs will greatly help in further extending the capacity of the memory.

As described in Section 6, preliminary results us­ ing the proposed method have been encouraging. Fu­ ture work will involve augmentation of the filter re­ sponses with color information (using, for instance, a va­ riety of color-opponent Gaussian center-surround mecha­ nisms derived from unsupervised learning along the RGB planes), motion-based segmentation and recognition of persons, and further testing of the method on large face databases.