A Part Power Set Model for Scale-Free Person Retrieval

1 Introduction

Person retrieval, a.k.a., person re-identification (re-ID), aims at retrieving pedestrians across non-overlapping camera views distributed at distinct locations. To calculate the similarities between person images, visual features play a central role. In the past decade, deep features have shown their superiority in person re-ID [Li et al., 2014; Chen et al., 2017a; He et al., 2018]. Despite the remarkable progress, person re-ID is still suffering from large variations on persons such as pose, occlusion, clothes, background clutters, and detection failures. To handle such variations, recent advances have advocated the use of part-level features to offer fine-grained description [Sun et al., 2018; Suh et al., 2018].

∗Corresponding author.

A common practice for part-based models is to partition the intermediate feature maps of an input image into multiple horizontal parts uniformly [Sun et al., 2018]. Subsequently, each partial region is used for identity classification independently. To that effect, various architectures for partitioning person images are exploited recently [Wang et al., 2018a; Fu et al., 2019]. Such a pipeline has three key drawbacks: Firstly, the overall performance seriously depends on how well the pedestrian parts are aligned. Therefore, in challenging scenes, most existing methods suffer from part misalignment due to inaccurate detection, pose variation, occlusion, etc. Secondly, most methods attempt to combine features extracted from different manually selected partial regions. Such handcrafted designs require extra human efforts and may lead to suboptimal solutions. Thirdly, most works only exploit the coarsest feature map extracted from the top of the backbone network. Thus, spatial details in finer-resolution feature maps are lost. In fact, finer-resolution feature maps contain meaningful low-level information, such as colour and texture, which are crucial to distinguish similar pedestrians.

In order to overcome the above limitations, we propose an end-to-end Part Power Set (PPS) Model, which enables robust and scale-free part-based person re-ID, as shown in Fig. 1. Firstly, to deal with misalignments of parts, our method enumerates k-combinations for all k of n body parts coming from a uniform partition, result in ∑nk=1 C k n combinations in total. As illustrate in Fig. 2, the PPS model is able to align discriminative parts in pedestrian images and exploit global-to-partial information elegantly. Enumerating alignment of body parts with different scales and ratios would have provided an almost infinite candidate set of combinations. However, PPS efficiently filter the candidates in accordance with a power set of 2n - 1 combinations. Secondly, we introduce a combination ranking module (CRM) to guide the model training with all combinations of body parts. It considers inter-class relations by constructing two subnets to perform pedestrian identification and combination ranking in parallel. Compared with manually-designed part-based models [Wang et al., 2018a; Fu et al., 2019], this pivotal module relieves the requirement of human efforts in designing specific part-based model, and thus has potential to achieve global optimal. Thirdly, we merge features extracted from different scales of CNNs with a feasible amount of extra cost in terms of memory and time, while gaining stronger abstract semantic as well as higher resolution features. In particular, before feeding features at different scales of the backbone network to PPS, we take advantage of the feature maps from the top layers to enhance the abstract semantics for the bottom layers. It derives from our observation that, feature maps in the top layers have strong abstract semantic but are coarse due to loss of details, while maps in the bottom layers have fine resolutions but suffer from lacking of abstract semantics.

In sum, the main contributions of this paper are three-fold:

• We propose an end-to-end part power set model by enumerating k-combinations for all k of n body parts, which exploits, in an all-inclusive manner, from the global to various levels of the partial information to learn discriminative features.
• Our method introduces a combination ranking module to guide the optimization to converge rapidly and stably, which performs combination identification and ranking in parallel. It replaces handcrafted designs and selections that require extra human efforts and may lead to suboptimal solutions.
• We exploit multi-scale features of the backbone network with single stream and image, to further boost the performance. It is trained end-to-end by combining all scales to enhance feature representation in a deeply supervised fashion.

We perform extensive quantitative evaluation on mainstream datasets, including Market-1501, DukeMTMC-reID and CUHK03, with comparisons to cutting-edge methods. Results show our method achieves the state-of-the-art performance. Code and models will be made publicly available.

2 RelatedWork

Person re-ID. Deep learning methods have dominated person re-ID in the last decade. Deep neural network is first employed by Li et al. [Li et al., 2014] to determine if a pair of input images belong to the same ID. Different loss functions have been designed for network optimization, e.g., the siamese loss [Radenovi´c et al., 2016], triplet loss [Hermans et al., 2017], and quadruplet loss [Chen et al., 2017a]. Many efforts focus on reducing the impact of the misalignment and occlusion [Su et al., 2017; He et al., 2018].

Part-basedModel. Several recent works propose to generate deep representation from body parts as fine-grained discriminative features of pedestrians. Such part-based models can be divided into three groups. The first one leverages external cues to partition pedestrian parts, e.g., assistance from the latest progress on human pose estimation [Qian et al., 2018; Suh et al., 2018; Su et al., 2017; Xu et al., 2018]. The second group utilizes attention-based methods to handle the misaligned matching in re-ID [Li et al., 2018; Xu et al., 2018]. The third group crops the intermediate feature maps into pre-defined patches (i.e., patches) [Sun et al., 2018; Wang et al., 2018a; Fu et al., 2019]. However, most methods usually require extra human efforts to examine and select the combinations of parts, and thus may lead to suboptimal solutions. To solve all these problems, we propose to factorize the visual appearance by enumerating k-combinations for all k of n body parts.

Multi-scale FeatureLearning. Most existingmethods typically consider only one resolution of person appearance by a standard scale normalisation process. It discards the potentially useful information of other different scales. To mine complementary information across different scales, one of the earliest endeavors is [Li et al., 2015], which jointly trains multi-scale images. Recent works [Liu et al., 2016; Chen et al., 2017b] use an image pyramid to build multi-scale features by designing multi-scale streams. Instead, our method works in a single streamwith top-down pathway, i.e., taking a single-scale image of an arbitrary size as input, which is more feasible in termed of time and memory than [Liu et al., 2016; Chen et al., 2017b]. Qian et al. [Qian et al., 2017] designs a deep architecture, which has multiple streams with different sizes of receptive fields. Our method is distinct from [Qian et al., 2017] from the following two aspects. Firstly, our method is independent of the backbone architecture. Secondly, the output feature maps of our method are with proportional sizes and strong semantics at multiple levels, in a fully convolutional fashion.

3 The Method

3.1 Overview

Our method is designed from the following three aspects: First, part power set model relaxes the requirement of wellaligned body parts, and smoothly incorporates global-to-local information by enumerating k-combinations for all k of n body parts. Secondly, a combination ranking module is introduced to guide the training process to converge rapidly and stably. Thirdly, we exploit multi-scale features with a feasible amount of extra cost in terms of memory and time.

3.2 Part Power Set Model

Our model is built on a feature map M extracted from the backbone network. M is a 3-dimensional tensor of the size C × H × W, where C is the number of the channels, W and H are the spatial width and height, respectively. The feature map M is divided into n horizontal parts uniformly with a fixed size of C × (H/n) ×W. Then we enumerate kcombinations for all k of n body parts, where 1 ≤ k ≤ n and n is a hyper-parameter, as illustrate in the middle of Fig. 3. It can be viewed as a specific global-to-local architecture, in which each component captures the discriminative information at different spatial scales. For example, n is set to 4 and thus 15 combinations, i.e., ∑4k=1 (4k), are enumerated in total, which is also illustrated in Fig. 1. For each combination, the feature map(s) of corresponding body part(s) is cropped and concatenated. Then, global max and average pooling layers are applied to the feature maps of each combination, followed by element-wise sum. A convolutional layer followed by a batch normalization and a ReLU activation layer further reduces the dimension and produces the final feature vector for the re-ID task.

However, due to the time and memory constraints, it is infeasible to directly crop and concatenate feature maps of corresponding body parts from feature maps of the backbone network. The reason is clear: as the number of k-combinations (1 ≤ k ≤ n) is the coefficient of the xk term in the polynomial expansion of the binomial power (1 + x)n , the total number of available combinations is ∑nk=1 (nk) = 2n - 1, where (n0 ) is abandoned. As a result, the size of intermediate feature maps for all combinations before two pooling operations is C(H/n)W ∑nk=1 k(nk), which is also the number of elements that need to be stored in memory and be accessed by each pooling operation.

To address the above limitations, we propose a novel paradigm by reusing the pre-computed pooling values of each part, as illustrate in Fig. 3. More specially, a group of global max and average pooling layers are applied to the feature maps of each body part to get two sets of vectors: u = {ui|1 ≤ i ≤ n, ui ∈ ℜC } and v = {vi|1 ≤ i ≤ n, vi ∈ ℜC }, respectively. Then for each combination, the pooled vectors of the corresponding body parts in u and v are picked and reduced by another group of global max and average pooling layers, respectively. Finally, an element-wise sum is applied to the output vectors. As the size of intermediate feature maps after first two pooling operations, is 2nC. This paradigm significantly reduces the size of intermediate feature maps for all combinations from C(H/n)W ∑nk=1 k(nk) to 2C ∑nk=1 k(nk). And it minimizes the number of elements that the four pooling operations need to access to 2HWC + 2C ∑nk=1 k(nk). Thus, this paradigm is a tensor decomposition to decouple the hyper-parameter n and spatial size of feature maps HW.

To empower our model to be sufficiently discriminative, a softmax cross-entropy identification loss is introduced in the fully-connected layer, which has the feature vector of combinations as the input:

where Nim and Nid are the number of images and person identities, respectively. And yi, W c yi , and f c θ (Ii) denote the identity of the i-th image, the weight matrix of the fullyconnected layer for the yi-th identity, and the feature vector of the i-th image in the c-th combination, respectively. On the other hand, a triplet loss based metric learning is imposed to the feature maps:

where Ii and I p i have the same identity, Ii and I n i are the images of different identities, d(·) is the normalized Euclidean distance, [·]+ is the hinge loss, and δ denotes the margin hyper-parameter to control the distance differences.

The extra computations are mainly on fully connected layers, which classify each combination. The extra FLOPs [Molchanov et al., 2017] are computed as: (2n -1)(2I-1)O, where I = 128 is the input feature size and O = 752 is the number of identity in Market1501. The extra FLOPs are 6 × 106 and 0.2 × 109 for n = 5 and n = 10, respectively, while the original ResNet50 has 3.8×109 FLOPs. Therefore, the increased computational complexity is marginal.

3.3 Combination RankingModule

The proposed PPS model is imposed to classify each combination in the power set independently. However, the discriminative powers of different partial combinations are not the same. We except that the model highlights discriminative combinations during learning. To this end, we alternate between ranking all combinations and estimating an appearance model using the weighted combinations. More concretely, we construct two subnets to perform such simultaneous identity classification and combination ranking, as illustrate in Fig. 4(a). The first subnet outputs an identity score for each combination individually. In particular, we get the j-th identity prediction probability for c-th combinations by applying a fully-connected layer to the feature vector, followed by a softmax operator, defined as: sid cj =

the fully-connected layer. The second subnet has a similar structure with the first one. However, in order to rank combinations, we use a different axis for normalization in the soft-

is the weight matrix of the fully-connected layer in the second subnet. The ranking matrix sra is then used to weight the identity matrix sid by taking the element-wise (Hadamard) product: srid = sra ⊙ sid . It's noted that in CRM, all combinations share the same parameters, i.e., W id and W ra . Finally, a cross-entropy loss function can be applied to srid . However, the computed loss may fail to converge and oscillates model parameters continuously, since each combination is competed to each other. Therefore, a sum pooling layer is further applied: ssum j = ∑2n-1 c=1 srid cj , which unifies the identity prediction vector to a single scalar for each identity. Then we define the loss function LCRM as:

where yi is the identity of the i-th image.

3.4 Multi-Scale Feature Learning

To overcome the major limitation of multi-scale discriminative feature learn as we analyzed in Section 1, inspired by [Honari et al., 2016], we introduce a multi-scale deep feature architecture to capture the determinative feature of pedestrians from coarse to fine, as illustrated in Fig. 4(b). Specifically, for ResNets, we use the feature maps output by the last residual block of the last 4 stages to construct multi-scale feature maps by the following steps: Firstly, starting from the coarsest feature map, we use a 1 × 1 convolutional layer to reduce the channel dimensions, and upsample the spatial resolution by a factor of 2. The reduced channel dimensions and upsampled spatial resolution are determined by the corresponding bottom feature maps. Secondly, the upsampled maps are merged with the corresponding bottom feature maps by element-wise addition with skip connection. We iterate the above two steps until all feature maps from the backbone network are traversed. Finally, the output feature maps are fed to PPS model for re-ID. To enforce direct and early supervision for both the output layer and the new layers, instead of using only the last feature map, we impose re-ID to all the new feature maps including the coarsest map from the backbone network with shared parameters.

4 Experimental Evaluation

4.1 Dataset and Evaluation Protocol

Market-1501. Market-1501 [Zheng et al., 2015] contains bounding boxes from a person detector, which have been selected based on their intersection-over-union overlap with annotated bounding boxes. It has 1, 501 persons and is split into training/test sets of 12, 936/19, 732 images.

DukeMTMC-reID. DukeMTMC-reID [Ristani et al., 2016; Zheng et al., 2017] is a subset of Duke-MTMC for person re-ID. It contains 36, 411 annotated images of 1, 812 different identities captured by eight high-resolution cameras. A total of 1, 404 identities are observed by at least two cameras, and the remaining 408 identities are distractors. The training set contains 16, 522 images of 702 identities and the test set contains the other 702 identities.

CUHK03. CUHK03 [Li et al., 2014] consists of 14, 097 cropped images from 1, 467 identities. For each identity, images are captured from two cameras, and there are about 5 images for each view. Two ways are used to produce the cropped images, i.e., human annotation and detection DPM. We follow the new training/test protocol, which has 767 identities for training and 700 identities for testing. Datasets named as labelled and detected are both used for training and testing.

Evaluation metrics. We adopt the Cumulative Matching Characteristics (CMC) at Rank-1, Rank-5, and Rank-10, and mean Average Precision (mAP). It is worth noting that all our results are obtained in a single-query setting. For the proposed method, no re-ranking is used in all the experiments.

4.2 Implementation Details

Our experiments are implemented based on the Caffe2 framework. For the backbone network, we use Resnet50 initialized with the weights pretrained on ImageNet. We remove the last fully-connected layer and the average pooling layer and set the stride of last resent conv5 1 from 2 to 1. The feature maps are reduced to a 128-dimensional vector using a 1 × 1 convolutoinal layer, followed by Batch Normalization and ReLU layers. We also apply dropout with a ratio of 0.2 on the output feature vector before the fully-connected layer. All images are resized into a resolution of 384 × 128 by following [Sun et al., 2018]. The training images are augmented with horizontal flipping. We use a step strategy with minibatch Stochastic Gradient Descent (SGD) to train the neural networks on a Tesla V100 GPU. Parameters of the maximum number of epochs, batch size, momentum, weight decay factor and base learning rate are set as 120, 64, 0.9, 0.0005 and 0.01, respectively. The base learning rate is dropped by a half every 10 epochs from epoch 60 to epoch 90. The learning rate for all new layer is set to 10× the base learning rate. The margin in the triplet loss is 1.4 in all our experiments. Multiloss dynamic training [Zheng et al., 2019] is also used. We use the normalized feature for retrieval evaluation.

4.3 Comparison with State-of-the-Art Methods

Market-1501. Tab. 1 shows the performance comparison between our method and the state of the arts on Market-1501. It can be observed that, being facilitated by the proposed method, our performance surpasses all the state-of-the-art approaches in all the evaluation metrics. We have improved the baseline method namely the PCB model [Sun et al., 2018] by 7.92% and 2.04% in terms of mAP and Rank-1, respectively. It shows that both our PPS model play a crucial role beyond the backbone architecture. Fig. 5 shows the top-10 ranking results for some exemplar queries. It demonstrates that the proposed model is more robust to pose variation, blur, and occlusion than the baseline.

DukeMTMC-reID. Tab. 2 shows the performance comparison between our approach and the state of the arts on DukeMTMC-reID. Our method consistently achieves competitive performance on all metrics. In particularly, we achieve absolute improvement of 9.84% and 6.40% for mAP and Rank-1, respectively, compared with the baseline PCB model. It demonstrates the effectiveness of the proposed PPS model on person re-ID.

CUHK03. Performance comparison on CUHK03 is shown in Tab. 3. We report the results with two types of boxes: man- PPS

PCB

ually labeled and detected. We achieve an mAP of 72.66% for manually labeled boxes, outperforming the previous best mAP reported in [Wang et al., 2018a; Yang et al., 2018] by 5.26% and 18.86%, respectively. It is noted that CUHK03 is the most challenging benchmark among the three datasets. And the result indicates that our proposed PPS model is more robust than all other methods.

4.4 Ablation Study

Influence of the body part number. As showed in Tab. 4, with fewer body partitions n ≤ 3, the performance drops dramatically, due to that local feature is not fully exploited. With an increasing n (n ≤ 6), the performance consistently boosts and converges to about 83% in terms of mAP. It is excepted that the more local feature is exploited, the more PPS model is robust to the misalignment of body parts. When n increases to 10, which enumerates 1, 023 (210 - 1) combination and extracts 1, 023 different local feature, it is observed that the performance drops about 5% to 11% in terms of mAP. We argue that imposing too many combinations to identify person may cause the optimization being hard to converge.

Impact of combination ranking module. As shown in Tab. 4, CRM is able to improve the performance especially with a large n. Results also demonstrate that CRM refines the model to focus on discriminative combinations. Our method achieves the best performance when n is set to 5.

Multi-scale feature versus single-scale feature. Finally, we explore the multi-scale features to help re-ID. Tab. 4 shows that the proposed multi-scale architecture improves the results by 0.29% to 0.81% in terms of mAP.

5 Conclusion

In this paper, we propose an end-to-end part power set model for part-based person retrieval, which is robust to misalignment of body parts. In particular, a combination ranking module is introduced to perform identify classification and combination ranking in parallel. We further exploit how to extract scale-free features with deep supervision, which has received little attention so far. Extensive experiments demonstrate that our method substantially outperforms the state of the arts.

Acknowledgements

This work is supported by the National Key R&D Program (No.2017YFC0113000, and No.2016YFB1001503), Nature Science Foundation of China (No.U1705262, No.61772443, and No.61572410), Post Doctoral Innovative Talent Support Program under Grant BX201600094, China Post-Doctoral Science Foundation under Grant 2017M612134, Scientific Research Project of National Language Committee of China (Grant No. YB135-49), and Nature Science Foundation of Fujian Province, China (No. 2017J01125 and No. 2018J01106).