Regularized Latent Least Square Regression for Cross Pose Face Recognition

1 Introduction

In the past several decades, face recognition has received a great deal of attentions from the scientific and industrial communities, due to its wide range of applications including access control, security and surveillance, and so on. Although many face recognition approaches [Zhao et al., 2003; Wright et al., 2009] have reported satisfactory performance, they require accurate alignment and feature correspondence between the face images to be compared. In many real scenarios, the face images maybe captured under various poses, illuminations, or occlusions, which may cause the misalignment problem. According to the survey of face recognition techniques [Zhao et al., 2003], the pose variation is identified as one of the most challenging factors.

The difficulty for cross-pose face recognition is that the pose varies in 3D space, while the image captures only 2D appearances. When the pose changes, some visible parts on face may become self-occluded, while some invisible parts may appear. It leads to the special phenomenon that the distance between two different persons with similar pose is smaller than the distance between the same person under different poses. The performance of many popular face recognition methods, such as eigen-face or fisher-face [Belhumeur et al., 1997], decreases significantly when confronted with large pose variations.

Many approaches have been proposed to deal with the pose problem [Zhang et al., 2009]. A natural approach to solve this problem is to generate the virtual frontal pose images. [Blanz et al., 2003] proposed the 3D morphable model (3DMM). For a given 2D face image, they constructed its corresponding 3D model, and did matching in the 3D shape and texture space. However this process is computationally expensive, and relies on manual initialization of several facial points. [Chai et al., 2007] proposed the local linear regression method to synthesize a virtual pose image directly in the 2D domain. They assumed that a facial image could be faithfully represented as a linear combination of a set of face images with the same pose, and the coefficients of the linear combination remained roughly constant across poses. [Li et al., 2012] reported significantly improved performance by using Ridge regression to estimate the coefficients. [Ying et al., 2010] proposed the associate-predict model. They divided the face image into patches, then each patch was associated with a similar patch from a reference set under the same pose. In the prediction step, the associated patch's corresponding patch under the gallery pose was used as proxy for the original patch.

Besides explicitly synthesizing the virtual images, many researchers employ some statistical machine learning methods to find pose-dependent projections, which could project the images into a pose-independent latent subspace. [Li et al., 2009] proposed to use Canonical Correlation Analysis (CCA) to learn a pair of projections which make the projected intra-individual features to be maximally correlated in the latent space. [Sharma et al., 2011] employed the partial least square regression (PLS) to project samples from two poses to a common latent space. However, these methods considered only two poses at one time, and could not get a common space across all poses. So [Rupnik et al., 2010] proposed an extension of CCA for the multi-view problems. [Kan et al., 2012] proposed the Multi-view Discriminative Analysis (MvDA) to learn view-specific transforms by maximizing the between-class variation while minimizing the within class variation from all views in a common discriminant space. [Prince et al., 2008] proposed a generative model, named Tied Factor Analysis (TFA), for cross pose face recognition. They assumed that the observed images could be generated by a pose-specific linear transformation of identity variables in the presence of Gaussian noise. They used the EM algorithms to learn the model parameters from a set of training images in different known poses. In recognition step, they computed the probability that the probe and gallery share the same identity vectors.

The common goal of these existing approaches is to build a bridge between the observed image space and the pose free representation space. Since the images of one person are captured under different poses of the same face, they are highly related to each other. Similar to [Prince et al., 2008], it is reasonable to assume that the images taken under different poses can be viewed as pose-specific transforms of a single ideal object. In real applications, there exist some noises, such as illumination, expression et al., so this assumption may not hold exactly. We relax this hard assumption, and try to find a set of pose-specific transforms, which make the projected images of the same person as close as possible to the single ideal object. We treat the observed images as regressor, the latent ideal objects as response, and then formulate this relaxed assumption in a compact least square regression framework. Compared with previous methods, the advantages of our method are that: 1) we formulate the problem of cross multi-poses face recognition in a compact framework; 2) two regularization terms can be naturally incorporated into the regression based approach. The first is the smoothness regularization, which enforces the transforms for nearby poses to be close to each other. The second is the local consistency regularization, which enforces that the distribution of the latent ideal objects should preserve the geometric structure of the observed image space; 3) the final optimization problem can be solved efficiently by an alternating algorithm.

The remainder of this paper is organized as follows. In Section 2, we give a brief review of some related works. In Section 3, we present the formulation of our proposed method, and develop an alternating algorithm to solve the optimization problem efficiently. In Section 4, we evaluate the proposed method on the Multi-PIE dataset [Gross et al., 2007], followed by the conclusions.

2 Related Works

2.1 Canonical Correlation Analysis and Partial Least Square Regression

Canonical Correlation Analysis (CCA) and Partial Least Square Regression (PLS) are two classic statistical techniques to find linear relations between two multidimensional variables. Formally, let X� = [x�� , x�� , … , x�� ] and X� = [x�� , x�� , … , x�� ] represent the sets of facial images from two different poses respectively, where x�� � R�, x�� � R� . And the samples in X� and X� are coupled by identities. The goal of both CCA and PLS is to find two linear transformations W� � R�×� ,W� � R�×� for X� and X� respectively, so that the projected images of the same person are similar to each other in the projected latent subspace. The definition of similarity varies with the methods, so they are different in their objective functions. CCA tries to maximize the correlation between the projected images of the same person:

PLS tries to maximize the covariance between the projected images of the same person:

CCA can be naturally extended to deal with multi-view problems, which is called Multi-view CCA [Rupnik et al,.2010]. The goal of MCCA is to find a set of linear transformations {W�}(i=1,…, p), one for each pose, so that the sum of all pair-wise correlations is maximized:

2.2 Tied Factor Analysis

Tied Factor Analysis (TFA) [Prince et al., 2008] is a generative model to describe pose variations of facial images. They introduce two spaces: the observed space and the identity space. The observed space means the raw pixel feature or a transformed space after simple pose-independent transformations. In this space, the locations of pose variant and identity variant facial images are mixed altogether, which makes the separation of identity from pose variations infeasible. To effectively separate the identity from pose variations, an ideal identity space is introduced. The underlying assumption is that the face images of a person across different poses can be generated from a common latent variable in the identity space. The relationship between these two spaces can be described as:

where x�� � representes the jth image of individual i under pose k. h� is the latent identity vector for individual i. F� and M� are the linear transformation and offset respectively for pose k. E�� � is a zero-mean Gaussian noise with unknown diagonal covariance matrix ��. More formally, the model can be written in terms of conditional probabilities:

From a set of training images in different known poses, the identity vectors and the pose-specific transformations can be estimated by maximizing the joint likelihood of the observed image data x and its associated identity h.

where � = {F�,..,�,M�,…,�, ��,…,�} is the model parameters. They use the EM algorithm [Dempster et al., 1977] to learn the model parameters, which is prone to local minima and computationally intensive. In the recognition phase, they compute the probability that the probe face and the gallery face are generated by exactly the same value of the identity vector under the linear transformation scheme, and the final decision is made through maximum a posteriori mechanism by choosing the gallery image which corresponds to the maximum probability.

3 Regularized Latent Least Square Regression

3.1 Problem Formulation and Learning Model

Formally, we assume the pose can be discretized into p different bins. Let {X�}, k=1,…,p denote a set of training images, where X� = �x�� � �i = 1,… , �, � = 1, … , n�� } denotes the samples in pose k, x�� � � R� is the jth image of individual i from pose k in d dimensions, n�� is the number of images for individual i under pose k, and c is the number of individuals.

Since the images are captured from different poses of the same person, it is reasonable to assume that there are some common features across all poses. Both of PLS [Sharma et al., 2011] and CCA [Li et al., 2009] approaches try to find a set of projections, which make the projected images of the same person from different poses similar to each other in the latent pose-free space. Similar to the TFA [Prince et al., 2008], we assume the idealized version, which means that the images of the same person from different poses can be projected to the same identity vector in the latent space. So we reformulate the Eq(5) as:

(9) can be written in a generalized form:

where h� � R� is the idealized identity vector of individual i; f�w� , x�� � � is the mapping function for pose k, and w� � R�� is the parameter. m� � R�, and ��� � are the offset and noise term respectively for pose k. For simplicity, we use the linear mapping function:

Our goal is to obtain the parameters {w�,m�}��� � of a set of pose-dependent (but identity-independent) mapping functions, so that in the recognition phase, we can get the identity vectors h� and h� of the gallery xg and probe x�. But as the noise term � exists, we just could get the estimated identity vectors h� � and h� �:

For the training set, the estimated identity vector should be as close as possible to the idealized identity vector (as shown in Fig 1). We formulate the above motivation in the least square sense as:

The formulation (14) can be interpreted as a least square regression problem, if we treat the observed images as regressor, the parameters of mapping functions as regression matrixes, and the identity vectors as response. But the idealized identity vectors cannot be observed directly, so we call the formulation (14) as the Latent Least Square Regression.

Let X�� = [x�� � , … , x���� � ] � R�×��� collect the images of individual i under pose k , and X� = [X�� , X�� ,… , X�� ] � R�×�� collect all the individuals' images under pose k.

Then the objective function (14) can be rewritten as:

where �. �� stands for the Frobenius norm of matrix.

However, using the least square approach in (19) is not enough for finding a faithful mapping, because the least square approach may lead to the problem of over fitting and thus has poor generalization ability. In order to deal with this problem, we impose some prior knowledge as regularizations into the objective function to limit the solution space and improve the generalization performance.

1) Smoothness regularization. Though we want to project the facial images in distinct poses by different mapping functions, the facial images under nearby poses don't differ too much. Hence, the mapping functions should vary as smoothly as possible when the pose varies. This smoothness regularization can also tolerate small pose estimation error, which will be validated in the experimental part. The smoothness regularization can be written as:

2) Local consistency regularization. As revealed in many previous studies [Roweis et al., 2000; He et al., 2005], locality information is an important clue in manifold learning. In most manifold learning methods, researchers try to find a subspace that best preserves the local structure of the observed data. In this work, we could regard the idealized identity vectors as the low-dimensional embeddings of the observed facial images. So the distribution of the identity vectors in the identity space should preserve the local geometric structure of the observed image space, which means if two persons are similar in the observed image space, their identity vectors should be close to each other. We achieve this objective by minimizing the following energy function:

R����������� = � � �h� �h���S�� ���� ���� = tr(HBH�) (21) where h�, h� is the identity vectors for individual i and j; S�� is the similarity between individual i and j, S = {S��}, B = � �� is the Laplacian matrix, and D is a diagonal matrix with D�� = � S�� � . We select one image x� of each person under the same pose (e.g., frontal view), and define S�� as:

where�� (x�� ) denotes the set of k nearst-neighbors of x��. �� is the distance from x� i to its qth nearest neighbor, and q is a local scale factor. This way of defining similarity matrix is called local scaling [Manor et al., 2005], and it controls the decreasing speed of S�� with the distance between x�� and x�j. The local parameter k and q influence the relative contribution of local structure preserving in the observed space.

In addition, in order to control the complexity of mapping function, an extra normalization term � �w� ��� ���� needs to be included. To sum up, the proposed Regularized Latent Least Square Regression model is formulated as follows:

and �, �, � are nonnegative trade-off parameters.

3.2 Solving the Optimization Model

Here we develop an alternating method to solve the optimization problem (23). We decompose the original problem into two key subproblems as following:

Subproblem 1: Given the identity vectors {h�}��� � , the mapping functions' parameters {w�,m�}��� � can be obtained by solving the following optimization problem:

where

A straightforward way to minimize (25) is the gradient descent approach. According to the matrix theory, the derivative of g�{w�,…,�,m�,…,�}� respect to {w�,m� }(k = 1, … , p) can be computed as:

where �(x) is the indicator function, which returns 1 if x is true, and 0 otherwise. After obtaining the gradient, the parameters {w�,m� }(k = 1, … , p) can be updated by (29) until convergence

w� = w� ���g �w� � m� = m� ���g �m� � (29) Subproblem 2: Given the mapping functions' parameters {w�, m�}��� � , the identity vectors {h�}��� � can be obtained from the following optimization problem:

Naturally, we consider the derivative of g(h�,…, h� ) with respect to H, and take it to be zero. This gives us the analytical solution as follows:

�g �H

� =0

On the basis of above two subproblems, we develop an alternating method to solve the original optimization problem (23). Algorithm 1 lists the steps of RLLSR algorithm.

4 Experiments

Like any other learning based methods [Li et al., 2009; Kan et al., 2012], we require training data to learn the model par- Algorithm 1: Regularized Latent Least Square Regression Input: Training set {X� }, k=1,…,p, where X� = �x�� � �i = 1, … , c, j = 1, … , n��} are the samples in pose k, and x�� � is the jth image of individual i under pose k; the positive tradeoff parameters �, �, �; maximum number of iterations T; step size �.

Output: The parameters {w�,m�}��� � of pose-dependent mapping functions and the identity vectors H={h�}��� � for the training individuals.

1. Select one frontal pose image of each person from the training set, and construct the similarity matrix S according to (22).
2. Randomly initialize {h�}��� � , {w�,m�}��� � , and normalize �h���� = 1 (i=1,…,c); Iter =1.
3. while Iter<T do
4. Compute the objective value L of (24)
5. t=1;
6. while (t<T) do

7.

for k=1:p

1. Compute the gradient of �g �w� � (27), �g �m� � (28)
2. Compute w�� and m�� (k=1,…,p) by
3. w�� = w� ���g �w� � m�� =m� ���g �m� �
4. end for

1. w� = w��, m� = m��.(k=1,…,p); t=t+1;
2. if (delta<1e-5)
3. Break;
4. end if
5. end while
6. Compute H according to (32), and normalize �h���� = 1
7. Compute the new objective value L��� of (24)
8. if(|L��� � L|<1e-5)
9. Objective function converges. Break.
10. end if
11. Iter= Iter+1;
12. end while

ameters. We assume access to a training dataset that has multiple images of each person under different known poses. In the training phase, we obtain the pose-dependent mapping functions. At testing time, we assume the pose of the gallery and probe images are known, and use the corresponding mapping functions to estimate the identity vectors; then we adopt the cosine distance to measure the similarity of the estimated identity vectors, and the nearest neighbor classifier is chosen to do the recognition task.

4.1 Dataset and Experimental Settings

We conducted experiments on CMU Multi-PIE [Gross et al., 2007] dataset. It contains 337 subjects, recorded during four sessions under various poses, illumination and facial expressions. The interval pose angle is about 15� . Following [Kan et al., 2012], we select 6 images with neutral expression and no flush illuminations of each subject under seven poses (�45o , �30o , �15o , 0o , 15o , 30o , 45o ), so a subset about (6 ×7 ×337)14154 images is selected as the evaluation dataset. For the experiments, this dataset is split into a training set, which is used to obtain the pose-dependent mapping functions' parameters {w�,m� }(k = 1, … , p) and a testing set, which is used to perform the closed-set identification. The first 231 subjects are used for training, and the next 106 subjects for testing. We used the publicly available labeled locations of facial points provided by [Sharma et al., 2011] to crop the face regions, and then normalize to32×32.

Images are turned into gray-scale and direct intensity values mapped between 0 and 1 are used as features. Thus the length of the pixel feature is 1024. Some normalized examples are shown in Fig 2.

Parameters selection is a key issue. The trade-off parameters {�, �, �} of our model are chosen from {1e-4, 1e-3,…, 1,…,1e3,1e4} and the dimension m of identity vector is chosen from {50,100,150,…,1000} by performing cross validation on the training data. In our experiment, we finally set � = 0.1, � = 0.01,�=0.01, and m=200. We implement the proposed method in MATLAB, and the cropped images and source code are available upon request.

4.2 Effectiveness of the Regularization Terms

We evaluate the performance of the proposed model with and without the regularization terms, which are the two objective functions (23) and (33) respectively:

where

Similar to other methods [Li et al., 2009; Prince et al., 2008; Kan et al., 2012], we use the images from one pose as gallery, images from another pose as probe, and then perform closed-set identification. This yields a correct recognition rate for all pairs of poses. The results are shown in Table 1&2 respectively. The last row and column of these two tables denote the average recognition rate.

The comparison between Table 1&2 clearly highlights the improvement offered by using the smoothness and local consistency regularizations. We especially observe a significant improvement (near or over 10%) for gallery and probe with large pose difference (the result in bold). The average improvement of 42 pairs of gallery and probe setting is 6.9%. The reason for the improvement can be interpreted from the Eq(27). Without the smoothness regularization, we just use the samples from the kth pose to compute the gradient �g �w� � . While with the smoothness regularization, we consider the relationship between the kth pose and its nearby ((k-1)th and (k+1)th) poses, thus the samples from nearby poses can also contribute to computing the mapping function for pose k. And the local consistency constraint works as a term to utilize the structure of the observed data, which helps to improve the generalization.

4.3 Experiment with Inaccurate Probe Pose

In section 4.2, we assume that the pose angles of gallery and probe are known. In order to evaluate the robustness to inaccurate pose estimation, we conduct experiments with unknown probe pose. We assume that the gallery poses are known, while the probe poses are estimated. We evaluate the performance when the difference between the estimated pose and the real pose are ±15o , ±30o , ±45o . For example, if the real pose is �, and the estimated pose is �, we use the mapping function for pose � to map the probe images into the identity space followed by nearest neighbor matching.

Fig 3 shows the results when we set the gallery pose to frontal. It can be seen that when the error of pose estimation is within ±15o , the performance declination is small. But as the pose estimation error increases, the performance decreases significantly. So it is difficult to recognize faces with big error in pose estimation. In Table 3, we shown the results for all gallery and probe poses when the pose error is within ±15o . By comparison with Table 1, it indicates that besides the difference between the real pose and estimated pose, the gap between the gallery and real probe pose is another factor that influences the performance. Table 4 shows the mean accuracy of 42 pairs of gallery and probe pose settings under different pose estimation errors. When the pose estimation error is ±15o , the performance decreases 5.5%, but it is still much better than the average performance shown in Table 2. So in some sense, it also demonstrates the effectiveness of the regularization terms.

4.4 Comparison of Different Approaches

We compare the proposed method (RLLSR) with following algorithms:

• ♥ Fisherface model (FLDA) [Belhumeur et al.,1997]. It is a baseline method, and it is directly applied on the training set regardless of the pose variations.
• ♥ Tied factor Analysis (TFA) [Prince et al., 2008]. We use the code provided by the author to evaluate its performance on the Multi-PIE dataset.

• ♥ Partial Least Square Regression (PLS) [Sharma et al., 2011] and Canonical Correlation Analysis (CCA) [Li et al., 2009]. Both of them are pairwise method, which means they just consider two poses at the training and testing phases.
• ♥ Multi-view CCA [Rupnik et al., 2010] and Multi-view Discriminative Analysis (MvDA) [Kan et al., 2012]. Both of them learn the pose-dependent projections for multi-poses simultaneously. We re-implement these two methods, and get similar results as reported in their original papers.

Fig 4 illustrates the performance comparisons of different approaches under frontal gallery faces. Besides that we also evaluate the rank-one recognition rate under all possible pairs of gallery and probe pose settings (as shown in Table 5~7). And these results are averaged as mean accuracy (as shown in Table 8). From these results, we can observe that:

• (1)The performance of TFA, MCCA, MvDA and RLLSR is much better than the pair-wise methods (PLS,CCA), which demonstrates the effectiveness of considering multi-poses simultaneously.
• (2)The FLDA uses the multi-poses images, but it learns just one projection. It achieves lower accuracy than the method using different projections for different pose images (e.g TFA, MCCA, MvDA and RLLSR). So it is feasible and effective to project different pose images respectively into a common space.

(3)All methods achieve comparable accuracy, when the difference between the gallery pose and probe pose is in ±15o . However as the pose difference increases, the performance of PLS, CCA, TFA and MCCA degrade significantly. Compared with PLS, TFA, and MvDA, our proposed RLLSR gains 24%, 11% and 2.6% improvement respectively when the pose difference between gallery and probe is larger than 45o .

(4)The proposed RLLSR achieves a mean accuracy of 96.8%, which is significantly better than other methods. The MvDA approach employs the fisher criterion to find a discriminant common space, while the proposed RLLSR doesn't apply any discriminative information. It demonstrates that the proposed generative model with effective regularization terms can obtain better relationships between the observed image space and the latent common space, than MvDA and other methods.

5 Conclusions and Future Works

In this paper, we have developed an approach, named as Regularized Latent Least Square Regression, to deal with the cross pose face recognition problem. We assume that the images of one person captured under different poses could be mapped into a single ideal point in the latent pose-free space. The RLLSR framework provides such a way to formulate this assumption that it can integrate some regularization technology to improve the generalization performance efficiently. Comparative experiments indicated that the proposed method results in high accuracy and robustness for the cross pose face recognition problem. Our future work involves the extension to nonlinear mapping functions, and the exploration of using different holistic or local features at fiducial points.

Acknowledgements

This work was supported by the National Natural Science Foundation of China under grants No.61172103, No.61271429, No.60933010, and Tencent Research Institute of Beijing.