Disentangled Feature Learning Network for Vehicle Re-Identification

1 Introduction

Vehicle Re-Identification (ReID) aims to retrieve all the images of a given query vehicle identity, from a large image database. Deep learning techniques have greatly promoted the development of vehicle ReID in the past few years. Many previous works [Bulan et al., 2017] conduct vehicle ReID as a license plate recognition procedure. However, license plate recognition requires high-resolution images captured with front or rear views. Moreover, in some extreme cases, the license plates may be deliberately removed, occluded, or even faked. Recent works [Liu et al., 2016b; Liu et al., 2016c] start to focus on visual feature-based ReID, where ReID is performed as feature matching between the query and reference vehicle images.

Orientation is a crucial factor in vehicle ReID. Given two vehicle images captured from the same orientation, we expect the extracted features are capable of encoding vehicle subtle details, such as the tissue boxes or air inlet of engine, as illustrated in Fig.1. These details are orientation specific information. The works in [He et al., 2019; Pirazh et al., 2019] use local part and keypoint information to learn orientation specific features and show the efficacy of such features for vehicle ReID. In contrast, when given two vehicle images with different orientations, the learned features are desired to capture orientation-invariant common characteristics of the vehicles, such as colors and vehicle design styles. The methods using orientation invariant feature embedding [Wang et al., 2017] and viewpoint-aware metrics [Chu et al., 2019] show the benefits of such orientation invariant information for cross-view ReID. Though orientation specific and orientation common information are both useful for vehicle ReID, the previous methods often focus on learning a single representation which is however difficult to simultaneously capture these two types of powerful information.

∗Corresponding Author

Actually, such two characteristics are hard to be simultaneously obtained with a single feature representation learned under unified supervision. Moreover, using a single feature representation, it becomes difficult to flexibly choose the proper information for vehicle matching under variant conditions. For example, the vehicle details such as decorations in vehicle front-view are important for retrieving front-view vehicles, but may even degrade the performances of retrieving the rear-view ones, as shown in Fig. 1. This means when comparing vehicle pairs under the same orientation, orientation specific features are often useful, while the orientation common features are important to recall the same vehicles with huge orientation variations.

This motivates us to disentangle the learning of these two types of features from a single embedding model and design an adaptive matching method for ReID. Concretely, instead of learning a single feature representation, we propose a Disentangled Feature Learning Network (DFLNet) to explicitly learn the orientation specific and orientation common features concurrently, i.e., the learning of the two types of information is disentangled yet within a joint network.

Moreover, to learn powerful common features for ReID under variant orientations, we propose a novel "Odd-One-Out" adversarial scheme that distills the orientation invariant information shared by all samples of the same vehicles to obtain strong orientation independent ability. Besides, to learn orientation specific features, we also design an attention scheme to mine and focus on the useful detailed information.

To effectively utilize these two types of features, in this paper, a hybrid ranking strategy with feature metric alignment is also designed for adaptive matching in the ReID procedure. Overall, our contributions can be summarized as follows:

• A novel DFLNet is proposed to explicitly learn orientation common and specific features jointly for Vehicle ReID.
• A novel "Odd-One-Out" adversarial scheme is proposed to learn representative common features. An attention module is also designed for specific feature learning.
• A hybrid ranking strategy is designed to take advantages of specific and common features in the ReID procedure.
• Our DFLNet achieves state-of-the-art performances on all the evaluated benchmarks. It brings 12%mAP gains on cross-view ReID compared to the baseline model.

2 RelatedWork

Vehicle ReID. Recently, vehicle ReID has attracted much research focus [Liu et al., 2016a; He et al., 2019]. Some methods [He et al., 2019] focus on improving the discriminative capability of the models to distinguish specific subtle details of similar vehicles. Liu et al. [Liu et al., 2016a] introduce a mixed difference network in which both the vehicle model and ID information are used as supervisions for learning an embedding model. He et al. [He et al., 2019] propose to use vehicle part information to regularize the global feature learning. In [Pirazh et al., 2019], Pirazh et al, propose a dualpath model with adaptive attention model, which is able to get orientation conditioned keypoints to extract local features.

Some other methods focus on learning orientation invariant features. In [Wang et al., 2017], Wang et al. propose to use vehicle keypoint localization to align and generate orientation invariant features for vehicle ReID. In [Zhou and Shao, 2018], Zhou et al. propose a multi-view feature inference scheme, which uses a single-view input vehicle to generate the multiview features. In [Tang et al., 2019], Tang et al. propose a pose-aware multi-task learning method using synthetic data to learn viewpoint invariant features.

A few methods have considered using these two types of information simultaneously. In [Chu et al., 2019], Chu et al. use two triplets constraints for the same and different viewpoints in two feature spaces. Bai et al. [Bai et al., 2018] propose a group sensitive triplet embedding model to build up a type of "similar attribute, closer distance" feature embedding by a two-level margin constraint. However, they both learn and represent these two types of information under the same optimization objectives, and the differences between these two types of features have not been explicitly investigated.

Different from all the aforementioned works, we propose a disentangled feature learning network to learn these two types of features concurrently. The common features are the invariant and consistent representations shared by all samples of the same vehicles, and the specific features are the representations exploiting the subtle difference cues.

DisentangledRepresentation. Disentangled schemes have been used in image generation [Ma et al., 2017] and poseinvariant representation learning [Tran et al., 2017]. Tran et al. [Tran et al., 2017] propose explicit disentangled representations based on face variations through pose codes. Zhao et al. [Zhao et al., 2019] propose an attribute-driven method to disentangle several sub-features corresponding to semantic attribute groups for video-based person ReID. However, these methods disentangle sub-features based on each separate attribute, and common representations are ignored.

3 ProposedMethod

The architecture of our proposed DFLNet is illustrated in Fig. 2. In training stage, the orientation common features are learned by adversarial learning with an "Odd-One-Out" scheme. The orientation specific features are learned by an attention scheme in a triplet embedding design. During testing, we design a hybrid ranking strategy with a feature metric alignment scheme. We use common features to get initial recall list and use specific features to compare recall samples with the same orientation as the query for re-ranking.

3.1 Orientation Common Feature Learning by Odd-One-Out Adversary

To improve feature robustness under variant orientations, the ideal orientation common features are expected to encode the information shared by all samples of the same vehicle and ignore the orientation specific information, as the orientation specific information is often unuseful for cross-view ReID and may even degrade the performance. This means the common feature representation needs to be orientation independent. However, features learned with general embedding network designs [Schroff et al., 2015a] often contain orientation specific information. To get ideal common features, here we disentangle the common features from the base embedding features and distill the orientation invariant information.

Specifically, we design a novel "Odd-One-Out" adversarial learning scheme to generate orientation common features. A unit consisting of some vehicle images captured from the same orientation, together with one image captured from another orientation is constructed, and the "Odd-One" here means the sample has a different orientation compared to other samples in this unit. Concretely, we build a unit <xi , xj , xk> composed of three image samples of the same vehicle ID. O(xi ) = O(xj ) and O(xi ) ̸= O(xk ), where O(xi ) denotes the orientation of the sample xi . In this case, sample xk is the odd-one sample.

In our "Odd-One-Out" adversarial learning scheme, the discriminator D is pushed to learn and recognize which one is the odd-one sample, while the feature generator G aims to generate the common features that cannot be recognized by D. With the adversarial training going on, G will gain the ability of generating orientation independent features, i.e., the common features.

Common Feature Discriminator. When training the discriminator D, the generator G serves as an orientation common feature extractor. Given a vehicle sample unit <xi , xj , xk> from the same ID, we extract and concatenate their common features G(.) as the input of D, i.e., [G(xi );G(xj );G(xk )]. D learns to predict which sample in the unit is the odd-one, and the position of the odd-one sample in the unit is used as the label for prediction. For example, for the unit <xi , xj , xk>, if O(xi ) = A, O(xj ) = A, and O(xk ) = B, then the label of this unit is 3. This odd-one prediction process can be easily implemented as a classification task by using the cross entropy loss for supervision.

Common Feature Generator. When training the common feature generator G, the parameters in D are fixed. The ideal common features are independent to the variant orientations. Thus, we design a novel uniform loss Luniform, which constrains G to generate common features that will make D produce a uniform probability distribution on the odd-one prediction (classification). Such a constraint drives G to exploit the orientation independent information and try to eliminate the specific information in the generated features.

The Luniform is inspired by label smoothing regularization [Szegedy et al., 2016] that assigns small values to the non-groundtruth classes in cross entropy loss.

The standard cross entropy loss is formulated as:

where yk is the ground-truth class label vector in one-hot distribution. ˆ yk is the prediction probability of the input belonging to class k. For Luniform loss, to be constant over all classes, the ground-truth class label distribution yk in Eq. 1 is defined as yk = 1K . Thus Luniform loss is formulated as:

where ˆ yk is derived from the softmax function, and ezk is the output of the last fully connected layer in D.

In this manner, the adversarial relation between G (with Luniform) and D (with Lentropy) can be constructed. G attempts to fool D to get a wrong odd-one prediction. Conversely, D tries to exploit the orientation specific information to predict which one is odd. As the adversarial learning goes on, in order to cheat D, G will try to eliminate the orientation specific information in the common features. Finally, only the orientation invariant information is reserved in the common features.

Note that since the common representation still needs to be discriminative for ReID, the cross entropy loss (for identity classification) and triplet loss (for distance metric learning) that will be introduced in Sec. 3.2 are both used during training the common feature learning modules.

3.2 Orientation Specific Feature Learning with an Attention Scheme

The orientation specific features need to exploit and encode subtle details of the vehicle images. It is beneficial to selectively focus on the informative regions that are useful for ReID, as shown by the yellow boxes in Fig. 1.

This selectively focusing scheme is also termed as attention that has been demonstrated to be effective in various areas, such as machine translation [Bahdanau et al., 2014] and image caption generation [Xu et al., 2015]. Therefore, we adopt the visual attention scheme to enable our network to learn and find the crucial details that need to be focused on.

In our method, the attention module computes the importance scores for each patch in the feature maps. Let x denotes the input and f(x) denotes the representation obtained after network mapping. Meanwhile, the attention score s(x) is generated by the attention module, which serves as gates for the base branch b(x), as follows:

where (i, j) indicates the patch position over the feature maps. This element-wise product can promote the responses on interested regions.

For learning orientation specific features, the triplet loss [Schroff et al., 2015b] and cross-entropy loss are used for metric learning as below:

where ω is the weight in optimization. Using this scheme with the attention module, our network is thus able to learn discriminative orientation specific features.

3.3 Hybrid Ranking Strategy

To effectively take advantages of both the orientation common features and specific features, we propose a hybrid ranking strategy for vehicle ReID, in which the common and specific features are adaptively used for distance computation. Concretely, given a query image, we use common features to get initial recall list, then use specific features to compare samples with the same orientation as the query. Then, the hybrid ranking procedure can be formulated as:

where dc and ds are the common and specific feature distances, respectively. The sample's orientation O(·) is obtained by training an orientation classification model. As orientation classification is a very simple task, we empirically observe the orientation can be recognized very accurately, e.g., using MobileNet V2 [Sandler et al., 2018] can achieve 99.5% classification accuracy on VehicleID dataset.

Note falign in Eq. 5 is the feature distance mapping from specific feature to common feature, as we can not simply compare these two feature distances, since they are learned in different feature spaces. The feature distributions are shown in Fig. 3. The different distributions indicate the evaluation metrics in two feature spaces are different. The distances can only be compared under the same evaluation metric. Thus we design an effective feature metric alignment scheme falign before ranking, as follows.

Feature Distance Metric Mapping. The feature distance metric can be obtained from distance distribution. If two distributions are the same, consistent metric values can be obtained under any metrics. Therefore, we first analyze the feature distribution statistics in training set. We sample a large number of image pairs and calculate pair distances in two feature spaces. As shown in Fig. 3(a)(b), The median of the distances for common features is 13.67, while for the specific features is 9.85, i.e., half of the distances are less than 13.67 and 9.85 in two feature space respectively. Therefore, the distance 13.67 for common features and 9.85 for specific features are the same evaluation metric scale in two feature spaces. The median is the special case of kth smallest. More generally, the kth smallest distance values can be selected to calculate mapping dictionary. By dense sampling, we obtain the mapping dictionary ( key-value pairs) between common and specific feature distances.

Chebyshev Polynomials. The obtained mapping dictionary is discrete, but we hope to obtain a continuous mapping function to fit it. Thus we take a further step to fit it by using Chebyshev polynomials[Rivlin, 1974] that can provide an effective near-optimal approximation under the maximum norm. The Chebyshev polynomials of the first kind are defined by the recurrence relation:

Given the mapping dictionary, the Chebyshev approximation is to find the coefficients {c0, ..., cn} to represent the mapping relation,

By truncating the function with n terms, we have the approximation function ˜ f(θ) = ∑ni=0 ciTi(θ) to fit the mapping dictionary. In our work, when the order n is set to 4, the mapping can be well fitted.

The mapped common feature distance distribution is illustrated in Fig. 3(c), which well fits the real common distance distribution. The KLD (Kullback-Leibler divergence) calculated by these two distributions is 0.007. It shows the proposed feature metric alignment is reasonable and precise.

3.4 Implementation Details

The backbone of DFLNet is ResNet50 [He et al., 2016]. DFLNet has two branches after "pool5" layer. One branch is two fully-connected (fc) layers with Lemb in Eq. 4 for specific features. The attention module in specific branch consists of two 1×1 convolution layers (channel 1st layer: 2048→512, 2nd layer: 512 → 1). The other branch is an embedding layer with adversarial learning for common features. The dimensions of the common and specific features are both 128. We adopt a hard example mining mechanism to obtain a strong baseline. The discriminator for common feature learning is a small network with three fc layers followed by a classifier. The number of classifier output is the size of odd-one unit, and we set it to 3. The input channels for each fc layer are 384 (128x3), 128 and 128. The DFLNet is optimized by SGD algorithm. Regarding parameters, we set ω as 0.5 and triplet margin as 0.6 in metric learning following [Lou et al., 2019b], and λ = 0.5 in hybrid ranking. The models are trained for 50 epochs. Learning rate starts from 0.003. The size of the input image is 256 × 256.

4 Experiments

4.1 Experiment Setting

Dataset. We conduct experiments on VehicleID [Liu et al., 2016a], VeRI-776 [Liu et al., 2016c] and VERI-Wild [Lou et al., 2019b] datasets, which are widely used vehicle ReID benchmarks. VehicleID consists of 26,267 vehicle IDs, and most of the vehicles only have two views: front-view and rear-view. VeRI-776 is a small-scale vehicle dataset containing 776 vehicle IDs, which are captured by 20 cameras in unconstrained traffic scenarios. VERI-Wild is a large-scale vehicle dataset, which contains in total 416,314 images of

40,671 IDs captured by 176 surveillance cameras in the wild. The images in VehicleID dataset have 2 (front, back) orientations, and images in VeRI-776 and VERI-Wild datasets contain 5 (front, front-side, side, back-side, back) orientations.

Evaluation Metrics. We use mean Average Precision (mAP) and Cumulative Match Curve (CMC) in experiments.

4.2 Experimental Results Analysis

Evaluation on VehicleID Dataset

The results on VehicleID dataset are shown in Table 1. Using specific features only, we can achieve a strong baseline with the attention module and the hard example mining scheme. Our DFLNet outperforms EALN method by 5.33% (mAP). EALN uses a hard negative generation scheme for discriminative feature learning and a cross-view generation scheme to improve cross-view vehicle ReID. In Top 5 recall, DFLNet can achieve 9% mAP advantages over EALN. Besides, HDC+Contrastive cascades a set of GoogleNet which is a more complex method than a single network in our model, which further demonstrates the effectiveness of DFLNet. VAMI aims to infer viewpoint aware attentive regions for multi-view feature representation and AAVER proposes an adaptive attention mechanism to capture discriminative features. Compared with these methods, DFLNet achieves better performance with a disentangled learning scheme.

Evaluation on VeRI-776 Dataset

The results on VeRI-776 dataset are shown in Table 1. Compared to using specific features, the proposed DFLNet can significantly improve the performance from 67.72% to 73.29%mAP via introducing another common feature. Such gains demonstrate that by disentangling specific and common features, more adaptive feature representation and matching can be achieved. Our DFLNet significantly outperforms the state-of-the-art method PAMTRI, and it is worthy noting that PAMTRI uses DenseNet201 and the dimension of feature is 1024, while we use a simple ResNet50 network and two 128dimension features in DFLNet. Such observation verifies that DFLNet can better promote feature representation.

Evaluation on VERI-Wild Dataset

The results on VERI-Wild dataset are illustrated in Table 2. Compared with FDA-Net that uses GAN to generate hard negative samples, we get better performance under same network backbone. Moreover, we show the comparisons between hybrid ranking and simple combination of common and specific features. Simple combination directly weighted two distances of common and specific features, i.e. λdc + (1 - λ)ds. As shown in Tables 1 and 2, the simple combination can also bring performance gains, but is lower than hybrid ranking. We vary the value of λ in ranking scheme as shown in Fig. 4(a). The hybrid ranking can get consistent performance superiority over different λ. Since specific features focus on subtle differences, using it for cross-view ReID may degrade the cross-view ReID performance. We also provide the results of hybrid ranking without alignment, the performance drops significantly. This is mainly due to the feature distance distribution differences between these two features.

Ablation Study of Common and Specific Features

In Table 3, we present the ablation results of DFLNet. EALN [Lou et al., 2019a] is the first work that reports the same-view and cross-view performances. Specifically, given a query vehicle, for cross-view ReID, we treat the reference vehicles belonging to this query but with the same view as the junk samples, which are not involved in mAP computation. Simply comparing specific and common features, it can be observed that specific features are good at finding the same view vehicles while the common features are good at finding crossview vehicles. Such results show the effectiveness of our disentangled feature learning scheme.

In Fig. 4(c), we visualize the retrieval results. During testing, we use common features to get initial recall list (second row), then use specific features to further compare samples with same orientation as query to get hybrid ranking results (third row). The wrong samples with the same orientation as query can be further filtered by specific features. Therefore, the performance of hybrid ranking is superior than only using specific or common features in both same and cross view.

The "Odd-One-Out" Adversarial Learning Analysis

We design a toy experiment to verify whether orientation specific features learned by embedding network contain enough orientation specific information. For VehicleID and VeRI776 dataset, the "Odd-One-Out" classification accuracy is up to 93.28% and 87.69%, respectively. That means for welltrained embedding networks, the specific information is implicitly encoded in the features and the discriminator has the ability to recognize the odd one in given specific supervision.

For adversarial learning, as the training continues, the classification accuracy for odd-one prediction gradually decreases, as shown in Fig. 4(b). Thus, the expected orientation common features can be learned.

5 Conclusion

In this paper, we focus on exploiting the common and specific features to improve vehicle ReID performance. A novel disentangled feature learning network is proposed to jointly learn these two features. Such a unified end-to-end solution lays the groundwork for the subsequent improvements of vehicle ReID in terms of discriminative and invariant feature learning. Experiments demonstrate the effectiveness of the proposed method.

Acknowledgements

This work was supported by the National Natural Science Foundation of China under Grant U1611461, and was partly supported by the SUTD SRG project (T1SRIS20153).