Mirror Representation for Modeling View-Specific Transform in Person Re-Identification

1 Introduction

In recent years, the close-circuit television (CCTV) has been widely deployed in public area such as hospitals, railway stations, office buildings, etc. Because of economical or privacy issue, there are always non-overlapping regions between camera views. Inevitably, re-identifying a pedestrian who disappears from a camera view and re-appears in another disjoint camera view is indispensable for some cross-camera analysis like tracking or activity prediction, and such a problem is called the person re-identification.

The non-overlapping cross-view re-identification differs conventional recognition in the same view due to the great changes of illumination, viewpoint or camera features, so that appearance of a pedestrian changes dramatically across camera views. Such environmental condition inconsistency results in view-specific feature distortion, i.e., the pedestrian's appearance itself from view a and view b is transformed to T a (x) and T b (x) respectively, where T a and T b are unknown distortion functions. Consequently, the features distributions of the two views are not well-aligned. Since T a and T b are usually of complex nonlinearity, it is the fact that P (T a (x)) ̸= P (T b (x)) and P (y|T a (x)) ̸= P (y|T b (x)).

∗Corresponding Author

In the field of person re-identification, a lot of methods have been developed including descriptor-based methods[Zhao et al., 2014; Yang et al., 2014; Kviatkovsky et al., 2013] and subspace/metric learning methods[Weinberger et al., 2006; Mignon and Jurie, 2012; Davis et al., 2007; Zheng et al., 2013; Li et al., 2013; Pedagadi et al., 2013; Xiong et al., 2014]. Descriptor-based methods seek for more reliable descriptors across views. However, it is extremely difficult to extract robust descriptors against large appearance changes. Subspace learning seeks for discriminative combinations of features and enhances the performance, and metric learning aims to learn a variation insensitive and discriminant similarity measure between samples. However, these approaches all have an underlying assumption that all samples are drawn from the same distribution, which becomes invalid for matching person across disjoint camera views.

Previous approaches assume there is a uniform transformation to alleviate the feature distortions in any camera view. In this work, we relax this assumption by learning view-specific transform for each camera view. To derive our approach, we start from an intuitive feature transformation strategy called zero-padding, which augments the original features with zeros in view-specific entries. In this way, the view-specific knowledge is embedded in the augmented features and traditional subspace/metric learning can learn view-specific feature combinations. This idea is similar to [Daum´e III, 2009], which aims at domain adaptation. However, as shown in Sec. 2.1, zero-padding may extract irrelevant feature transformations of different camera views and the discrepancy between view-specific feature transformations is not explicitly controlled. This ignores the fact that albeit captured from disjoint camera views, the appearances of two cross-view person images are still relevant.

To control the discrepancy between view-specific transformations explicitly, we first generalize the zero-padding strategy and introduce a novel augmented representation of a pedestrian's feature representation for a pair of camera views. This achieves the discrepancy modeling on featurelevel. Secondly, we further introduce a transformation-level discrepancy by explicitly measuring the difference between the transforms of two disjoint views. We call this the discrepancy regularization. By transforming the augmented feature to a new space, this regularization is involved in the new representation. We call this transformed augmented representation the Mirror Representation, since it is generated by both the original feature and the phantom feature. With the mirror representation, we demonstrate that existing metric learning methods can be greatly enhanced on person re-identification across two disjoint camera views.

Our work is related to domain adaptation in machine learning. Some domain adaptation methods like [Pan et al., 2011; Si et al., 2010; Geng et al., 2011; Xiao and Guo, 2014; Li et al., 2014] aim at diminishing the gap between source and target domains. However, these domain adaptation methods assume existence of the overlap between the classes of training set and testing set so that a classifier/metric learned from training set can be adapted to the testing one, while for person re-identification the training and testing classes are nonoverlapping. Our work is also different from the transfer reidentification methods [Zheng et al., 2012; Ma et al., 2013]. We do not address the problem of incorporating any other source data for transferring knowledge to target task. Alternatively, we address a very different problem against these re-identification work by alleviating the mismatch of the feature distributions of different camera views.

In summary, our approach is efficient but not sophisticated, and it is low cost and easy-to-use. Our contributions are listed as follows:

• We propose a mirror representation to address the view-specific feature distortion problem in person reidentification, and then further extend it to the kernel version;
• Based on the mirror representation, we are able to significantly enhance existing metric learning/subspace models, and we demonstrate that such a generalization ob-

tains significant improvement on three popular datasets.

2 Approach

In this section, we propose two types of discrepancy modeling to form the mirror representation. Before introducing our approach, we first present the analysis on the zeropadding which motivates our modeling.

2.1 Zero-Padding Augmentation

To deal with the view-specific distribution mismatch problem, learning view-specific mapping is desirable for a subspace model. An intuitive idea is to apply zero-padding for each view, i.e., transforming Xa to [Xa ; 0] and Xb to [0;Xb ], where 0 denotes zero matrix whose size is d × na or d × nb . Applying subspace/metric learning methods (like MFA [Yan et al., 2007]) over such augmented features results in a 2d × c projection matrix U = [U1; U2]. Projecting the augmented features into U results in Y a = U T 1 Xa and Y b = U T 2 Xb . Since U1 and U2 are different, they are free to adjust according to the feature distortion. The zero-padding strategy can be visualized in Figure 1. Note that in this example we use the linearity of distortion functions for demonstration; for more complex functions, non-linear mapping is needed, which can be achieved by the kernel trick.

However, by using the zero-padding, one loses direct control of the relation between U1 and U2 on data of both views, since U1 is always on Xa and U2 is always on Xb as shown above. In person re-identification, the view-specific distortion consists of two types, namely the environment-specific and person-specific feature distortions (see Figure 2). The environment-specific feature distortion concerns environment change such as change of lighting, view point and camera features, and the pedestrian-specific feature distortion concerns the appearance's change of a person himself/herself. No matter which type of distortion, there is relation between the same distortions across camera views. Without explicitly embedding the relation between U1 and U2, the extracted view-specific transforms of two disjoint views may be irrelevant, and thus an inferior re-identification performance will be gained as shown in our experiment.

2.2 A Feature-level DiscrepancyModeling

Formally, the zero-padding augmentation could be rewritten as:

where I is an identity matrix and 0 is a zero matrix.

Note that the feature transformation matrices [I, 0] and [0, I] are totally orthogonal, since [I, 0]T [0, I] = 0, and thus the mapping functions are irrelevant. To impose restriction on the discrepancy of two mapping functions, we generalize (1) to:

Thus the mapping functions are also generalized to:

Note that in this generalized version, the discrepancy of mapping functions is determined by either the discrepancy R and M or U1 and U2. Note that the feature transformation matrices [R,M] and [M,R] do not need to be orthogonal. We can define R and M so that the discrepancy between fa(·) and fb(·) can be measured. In this way, the discrepancy of mapping functions is explicitly controlled/parameterized on the feature-level. Let r be the parameter that controls the discrepancy of [R,M] and [M,R]: when r = 0, the discrepancy is minimized and [R,M] is identical with [M,R]; when r = 1, the discrepancy is maximized and (2) is reduced to the zero-padding; when 0 < r < 1, the discrepancy increases as r increases. Therefore, we design the R and M as follows:

where z is the normalization term.

The choice of z affects the relation between r and the discrepancy of feature transformations when 0 < r < 1. Empirically, we find that choosing z as the L2 normalization, i.e., z = √(1- r)2 + (1+ r)2 , resulting in satisfying performance. Theorem 1 shows the relation between r and the discrepancy of the feature transformations, which is measured by the principle angles as detailed later. This theorem shows that the principle angle is a monotonic function of r.

Theorem 1. Under the definition of (4) and let z = √(1- r)2 + (1+ r)2 , all the principle angles of [R,M]T and [M,R]T are

proof. Let uk ∈ span([R,M]T ) and vk ∈ span([M,R]T ), then the k.th principle angle is defined as:

Note that uk and vk can be represented as uk = [R,M]T hk and vk = [M,R]T lk where hk and lk are d×1 unit vectors, uT v can be induced as follows:

Since hk and lk are unit vectors, then maxhTk lk = 1 for k = 1, 2, . . . , d.

2.3 A Transformation-level DiscrepancyModeling

In the last section, we measure the discrepancy of mapping functions in the feature level. From (3), we find that the discrepancy can be further controlled by restricting the difference between U1 and U2, which can be achieved by forming a regularization term for minimization. Such a regularization is complementary to the feature-level augmentation introduced in the last section, since it is on the transformation level when applied our approach to existing models.

More specifically, we form ||U1 - U2||2 as a regularization term for minimization, which can be further represented as U T BU where B = [I,-I;-I, I]. By further combining with the ridge regularization λ′U T U, i.e. U T BU + λ′U T U, the proposed regularization term can be re-expressed λU T CU where C = [I,-βI;-βI, I] and β = 1 1+λ′ < 1. We call this regularization term the View Discrepancy Regularization.

Now we show that one can perform a matrix transform to integrate this regularization, so that the subspace/metric learning methods need not to be modified except adding a simple ridge regularization. Note that with the view discrepancy regularization, many subspace/metric learning methods ([Pedagadi et al., 2013; Xiong et al., 2014; Mignon and Jurie, 2012] can be cast as:

where f is the objective function and gi are the constraints.

Since β < 1, C is full-rank and can be decomposed into C = PΛP T , and thus P T CP = Λ. Let U = PΛ- 12H, and so U T CU =HT H. (8) is equivalent to:

If we transform the augmented features Xaug to Λ- 1 2P T Xaug, H could be solved using standard subspace/metric learning with the normal ridge regression, and then U can be obtained by U = PΛ- 12 H.

In this work, the transformed feature, Λ- 1 2P T Xaug, is called Mirror Representation, since the original feature as well as its phantom are jointly transformed into a more robust space for better representation of the view-specific knowledge.

2.4 KernelizedMirror Representation

The Mirror Representation discussed above provides linear view-specific mappings. However, the distortion functions and the intrinsic data distribution are usually non-linear. To overcome this problem, we borrow the idea of kernel modeling in machine learning to develop a kernel mirror representation.

In order to extend the feature augmentation approach to the kernel version, the anchor points are enriched to include both training data points and their phantoms, and the location of phantoms are view-specific. In our modeling, the anchor points are data points from training set. Hence the mapping functions are defined as below:

where R and M are n × n matrices defined in (4), and thus X[R,M] and X[M,R] are anchor points of view a and view b respectively. k(A, ·) is defined as [k(A1, ·), k(A2, ·), · · · , k(An, ·)] where k(·, ·) is the kernel function and A is the anchor points.

Compared to the linear case, for the kernel case, U1 and U2 are replaced with Xα1 and Xα2, respectively. Hence the view discrepancy regularization U T CU is replaced with αT1 XT CXα2 = αT1 C′α2, respectively, where C′ = [K, -βK, -βK, K].

3 A Principle Angle based Parameter Estimation Strategy

The Mirror Representation consists of feature-level discrepancy and transformation-level discrepancy. For those modeling, there are two parameters r and β to jointly control discrepancy.

Intuitively, when feature distributions of different views are largely mismatched, the optimal view-specific mapping functions are more discrepant; otherwise, samples of different views consist of more shared features and the mapping functions are likely to be similar. In this work, we propose to measure the degree of distribution mismatch by the principle angles of different views. Let Ga and Gb be the PCAsubspace of samples of view a and view b. The principle angles can be computed by the Singular Value Decomposition of GTa Gb [Hamm and Lee, 2008]:

where cos(Θ) = diag(cos(θ1), cos(θ2), · · · cos(θd)) and θi are principle angles.

With the principle angles, r and β are set as:

4 Experiment

4.1 Datasets and Settings

Datasets: Our experiments were conducted on three publicably available datasets: PRID450S [Roth et al., 2014], VIPeR [Gray et al., 2007] and CUHK01 [Li et al., 2012]. All three datasets are with significant feature distortion. PRID450S contains 450 image pairs recorded from two different but static surveillance cameras. VIPeR contains 632 pedestrian image pairs captured outdoor with varying viewpoints and illumination conditions. CUHK01 contains 971 pedestrians from two disjoint camera views. Each pedestrian has two samples per camera view.

Features: We equally partitioned each image into 18 horizontal stripes, and RGB, HSV, YCbCr, Lab, YIQ and 16 Gabor texture features were extracted for each stripe. For each feature channel, a 16D histogram was extracted and then normalized by L1-norm. All histograms were concatenated together to form a single vector. Since the PRID450S provides automatically generated foreground mask, our features were extracted from the foreground area in this dataset. For CUHK01 and VIPeR, our features were extracted on the whole image.

Experimental Protocol: Our experiments follow the same single-shot protocol: each time half of the pedestrians were selected randomly to form the training set, and the remaining pedestrian images were used to form the gallery set and testing set. For CUHK01, each pedestrian has 2 images for each view; we randomly selected one of them to form the gallery. The cumulative matching characteristic (CMC) curve is used to measure the performance of each method on each dataset. A rank k matching rate indicates the percentage of the probe image with correct matches found in the top k rank against the p gallery images. In practice, a high rank-1 matching rate is critical and the top k matching rank matching rate with a small k value is also important since the top matching images can be verified by human [Zheng et al., 2013]. To obtain statistically significant results, we repeated the procedure 10 times and reported the average results.

Methods for Comparison: PCCA, KPCCA[Mignon and Jurie, 2012], MFA and KMFA [Yan et al., 2007] are selected to evaluate the improvement of using our Mirror Representation. The kernels used in KPCCA and KMFA are χ2 and Rχ2 (RBF-χ2 ). The Zero-Padding representation is also evaluated for comparison. Besides, Mirror-KMFA is also compared with various person re-identification methods [Zhao et al., 2014; 2013a; Li et al., 2013; Pedagadi et al., 2013; Zheng et al., 2013; Yang et al., 2014; Kostinger et al., 2012; Hirzer et al., 2012; Weinberger et al., 2006; Davis et al., 2007; Zhao et al., 2013b], domain adaptation methods [Daum´e III, 2009; Pan et al., 2011; Si et al., 2010] as well as the baselines a regularized CCA (rCCA) and L1-norm method. In order to balance the scale of the objective function when applying our mirror representation Eq. 9), λ is set as 10-6 , 10-2 , 10-2 and 0.5 for KPCCA, KMFA, PCCA and MFA respectively.

4.2 Effectiveness of Mirror Representation

Table 1 shows the comparison among the top rank matching performance of MFA, KMFA, PCCA, and KPCCA across 3 representations: theMirror Representation, the original feature and the zero-padding. It is clear that the Mirror Representation enhances the performance of all the methods we tested, while the zero-padding does not. As we discussed in Sec. 2.1, zero-padding is likely to loses the control of the discrepancy of view-specific transformations, probably leading to a serious overfitting to training data and therefore generalizing not very well on unseen testing data, since in person re-identification the people for training will not appear in the testing stage.

Our Mirror Representation enhances the performance in both linear case and non-linear case. It is not surprising that using the kernel trick boosts the performance, since the distortion functions are non-linear and non-linear mappings are more suitable. Note that a more complex kernel Rχ2 usually achieves better results than the simpler one χ2 , which may also indicates the complexity of the feature distortion.

4.3 Comparison to RelatedMethods

From tables 2-4, we find that the KMFA(Rχ2) with Mirror Representation achieves promising results on the three datsets. We compare this method with the other stateof-the-art methods. Tables 2-4 and Figure 3 show that our method significantly outperforms other methods except MLF+LADF [Zhao et al., 2014] at rank-1 on VIPeR. However, MLF+LADF combines MLF [Zhao et al., 2014] and LADF [Li et al., 2013], and neither of the these two methods outperforms ours. We also compare another feature augmentation method [Daum´e III, 2009] with exactly the same KMFA setting denoted as Daum´e-KMFA(Rχ2). Since it also does not control the discrepancy of view-specific mappings, the performance is unsatisfactory.

We additionally compare two popular domain adaptation methods TCA[Pan et al., 2011] and TFLDA[Si et al., 2010]. As shown, they do not perform well. This is probably because both TCA and TFLDA assume P (Ys|Xs) = P (Yt|Xt) or P(Xs) = P(Xt), which is not suitable for person reidentification, since the people to train are the ones to test.

4.4 Parameter Sensitivity Analysis

The Mirror Representation has two parameters: r and β. Using either the feature-level discrepancy or the transformation-level discrepancy could enhance the performance. Due to space limit, we take KMFA on CUHK01 for illustration. To demonstrate the effectiveness of the featurelevel discrepancy, we disable the transformation-level discrepancy by setting β = 0 and evaluate the parameter sensitivity of r. Figure 4 (a) shows that turning r can significantly enhance the performance by nearly 5%. When r = 1 and β = 0, the performance is unsatisfactory, since the two learned view-specific transformation may be irrelevant. The demonstration of the transformation-level discrepancy is similar: we disable the feature-level discrepancy by setting r = 1 and evaluate the sensitivity of β. Figure 4 (b) shows that turning β can also boost the performance significantly. Note that the best result achieved by turning r or β exhaustively is 40.90% rank-1 matching rate, while in our experiments we turn r and β automatically and achieve 40.40%. This demonstrates the effectiveness of our parameter estimation strategy. Figure 4 (c) shows the parameter sensitivity of λ, which is the ridge regularizer in (9). As shown, there is a clear trend that the performance will go down as λ increase.

5 Conclusion

In this paper, we have proposed the Mirror Representation to alleviate the view-specific feature distortion problem for person re-identification. The Mirror Representation consists of two complementary components: feature augmentation and view discrepancy regularization, designed for featurelevel discrepancy and transformation-level discrepancy respectively. Principles angles of different camera views are used to estimate the parameters. With our Mirror Representation, the kernel marginal fisher discriminant analysis significantly outperforms the state-of-the-art on all three popular datasets.

6 Acknowledgements

This project was partially supported by National Science & Technology Pillar Program (No. 2012BAK16B06), Natural Science Foundation Of China (No. 61472456), Guangzhou Pearl River Science and Technology Rising Star Project under Grant 2013J2200068, the Guangdong Natural Science Funds for Distinguished Young Scholar under Grant S2013050014265, Guangdong Provincial Government of China through the Computational Science Innovative Research Team Program, and GuangZhou Program (2014J4100114).