Incremental Few-Shot Learning for Pedestrian Attribute Recognition

1 Introduction

Pedestrian attribute recognition, aiming at recognizing human part attributes such as age, gender and clothing, is in growing demand for its wide applications in visual surveillance such as person re-identification, social behavioral analysis, and face verification. In the deep learning era, various methods based on either CNN or RNN have been proposed, and have achieved great success in this area. However, in realworld visual surveillance applications, the demand is usually dynamic and evolving, that is, the set of human part attributes that need to be recognized can be changed or enlarged over time.

Consider the example in Figure 1. During phase one, a model is trained (which we call base model) to recognize the existence of several attributes such as 'Customer', 'Suitup', and 'Long Trousers'. Then during phase two, a few-shot dataset annotated with some new attributes is provided. Our goal is to develop an efficient algorithm that can utilize both the few-shot data and the already well-trained base model, and produce an extended model that can not only successfully recognize the new attributes such as 'hat' and 'Single Shoulder Bag', but also retain the capability of classifying old attributes such as 'Customer' and 'Suit-up'. For clarity we denote the previous well-recognized attributes as base attributes, and the new coming attributes as novel attributes.

∗Corresponding authors

Given the scenarios described above, two straight-forward solutions would be finetuning and re-training. However, finetuning, i.e. adding randomly initialized output layers to the previous model and tuning the whole network on the few-shot examples, is prone to overfitting on novel attributes, and may even lead to catastrophic forgetting on base attributes[Goodfellow et al., 2013]. Re-training, i.e. discarding the previous well-trained model and training from scratch using both base and novel data, not only requires to store the previous largescale training data, but is also time-consuming. Thus it is infeasible to re-train in most applications where the time and memory are limited. Thus, we need to find a solution, that given the previously well-trained model and without storing the previous training data, can efficiently extend the previous model so that it is able to recognize both the base and novel attributes. It is worth to note that this task is challenging, since it involves incremental learning, few-shot learning and multi-label learning. Simply applying or combining any technique from only one of these areas is insufficient to solve this complex problem.

In order to address the challenges described above, we propose to extend the base model with an extra module called Attribute Prototype Generator Module (APGM). The APGM can be regarded as a higher-level meta-learner that extracts the multiple-attribute information from the feature embedding, and produces discriminative Attribute Prototype Embedding (APE). Then the classification weights for the novel attributes are acquired based on APE. Further, if the base attributes of the few-shot dataset are also annotated, we put forward another module called Attribute Relationship Module (ARM) that transfers knowledge from the base classification weights to the novel weights based on attributes cooccurrence, to further improve the performances of the generated novel classification weights.

To verify that our proposed method is capable of performing incremental few-shot learning for pedestrian attribute recognition, we conduct experiments on two benchmark datasets for pedestrian attribute recognition: PETA and RAP. We first transform these datasets into incremental fewshot case, and show by experiments that our method performs superior to the baselines with less time and memory consumption.

Besides, we also discuss the internal relationships between our method and other two contemporary meta learning methods. We explain intuitively the reason why our method is more suitable for the multi-label few-shot case, and demonstrate experimentally that simply extending the multiclass state-of-the-art few-shot meta learning algorithms into their multi-label counterparts will result in poor performances under this context.

In summary, we make the following contributions in this paper:

• We raise a practical issue termed as incremental fewshot (multi-label) attribute learning which commonly exists in real-word applications.
• We put forward a meta learning based method that achieves competitive performances.
• Our method demonstrates the capability of fast adaptation with low annotation and memory requirements which is desirable in low-resource on-device scenarios.

2 RelatedWork

2.1 Pedestrian Attribute Recognition

Pedestrian attributes have been used as soft biometric traits, and their recognition has been intensively studied due to its wide applications in video surveillance. Traditional methods usually involve extracting hand-crafted features and optimizing classifiers such as Support Vector Machine (SVM) [Prosser et al., 2010]. However, due to the limited representation capability of these hand-designed features, these traditional methods' performances still need to be improved.

With the development of automatic representation learning, deep architectures based on CNN, CNN-RNN and ConvLSTM have been proposed. In particular, [Sudowe et al., 2015] first adopts CNN that is trained end-to-end with independent layers for each single attribute. [Li et al., 2015] proposes DeepMAR with a weighted sigmoid cross-entropy loss to utilize the attribute distribution prior information to handle the class-imbalance problem. Later, [Wang et al., 2017] proposes a CNN-RNN based encoder-decoder framework, and [Zhao et al., 2018] further improves recurrent learning via attribute grouping. Apart from designing novel network architectures, [Li et al., 2017] exploits a Spatial Transformer Networks (STN) to extract multi-scale context-aware discriminative features, where [Zhu et al., 2017] divides the human body into several parts for more accurate attribute recognition. Efforts have also been made to model the local or hierarchical relationships where techniques such as attention mechanism and automatic gating [Liu et al., 2017; Jin et al., 2018] are exploited.

2.2 Few-Shot Learning andMeta Learning

Meta learning, also known as learning to learn [Thrun and Pratt, 2012], has achieved remarkable success in few-shot learning and other applications [Ravi and Larochelle, 2016; Finn et al., 2017; Snell et al., 2017; Xiang et al., 2019].

The key idea of meta learning is to learn a higher-level parameterized network (which we call meta-learner), to guide an ordinary network (which we call learner) to learn. This is often achieved by the so-called episodic training, which enables the meta-network to generalize to few examples by consistently mimicking the few-shot scenarios. Among these methods, [Finn et al., 2017] learns an initialization of a neural network capable of fast adaptation, where [Ravi and Larochelle, 2016] utilizes an LSTM as meta-learner to guide the learner to update parameters using the gradient information. [Snell et al., 2017] and [Sung et al., 2018] approach few-shot learning via metric learning, where the former learns an embedding function followed by KNN, and the latter learns the metric function as well. Dynamical weight methods including [Bertinetto et al., 2016; Qi et al., 2018; Gidaris and Komodakis, 2018] propose to train a metanetwork to produce parameters of a base learner, so that the produced parameters (fast weights) can quickly adapt to few examples.

However, most existing meta learning methods are targeted for multi-class few-shot classification, and is not applicable to our multi-label setting where complex attribute relationships and co-occurrence exist.

2.3 Incremental Learning

The problem we address here is also related to (but distinct from) incremental learning, also known as lifelong learning or continual learning, where the model is given a sequence of tasks to learn. The main issue of incremental learning is to overcome catastrophic forgetting [Goodfellow et al., 2013], a phenomenon where the model performs poorly on the previous tasks after learning the new tasks.

Two representative works of overcoming catastrophic forgetting are LwF [Li and Hoiem, 2018] and EWC [Kirkpatrick et al., 2017]. LwF aims to alleviate the catastrophic forgetting by exploiting knowledge distillation on the old output nodes of the model. EWC, on the other hand, evaluates each neurons importance for previous tasks using fisher information matrix and keep the important neurons close to their original values. The incremental few-shot problem we are facing here differs from the standard incremental setting where the learning of the novel classes is harder and catastrophic forgetting is less severe.

3 Problem Setup

Incremental few-shot attribute learning is essentially a multilabel few-shot learning problem. We assume that in phase one we are given a large-scale pedestrian attribute dataset Dbase = {(x1, y1), ..., (xKbase , yKbase )}, yi ∈ {0, 1}Nbase , with attributes Abase = {a1, ..., aNbase }, together with a model Mbase which is well-trained on it.

Then in phase two, we are provided with a new dataset Dnovel, in which novel attributes Anovel = {a1, ..., aNnovel } appear. The Dnovel containsKnovel examples annotated with Nnovel novel attributes where Knovel is usually small, and the entire learning process is termed as N-way K-shot learning.

Our goal is to extend the Mbase to Mext in the second phase so that it is able to recognize both Abase and Anovel. It is worth noting that Anovel can be either absent or not annotated in Dbase. Unlike the normal multiclass incremental few-shot learning setting which assumes Dbase ∩Dnovel = φ , our setting only requires Abase ∩ Anovel = φ and Anovel will not appear in Dbase, in which case the same example may appear both in Dbase and Dnovel, but are annotated with Abase and Anovel respectively. In other words, it is more of class incremental rather than the data incremental setting.

4 ProposedMethod

4.1 Overview

Our overall approach for incremental few-shot attribute learning is illustrated in Figure 2(a). We first start with a welltrained base model Mbase (the blue part) with conceptually two parts: feature extractor F and classifier W. We then introduce the APGM module which has no interference with the base model so that the performances on Abase will not be hampered. The APGM takes the feature embedding that contains the encoded information from multiple attributes as inputs, and produces the separated APE embedding for each attribute. Then the classification weights for novel attributes are produced using the APEs. Moreover, when Dnovel is also annotated with base attributes, we propose the ARM, which takes the base attributes' classification weights as inputs, and works in a similar way as APGM to produce the novel classification weights. In order to facilitate few-shot learning, we train these modules with episodic training on Dbase, which will be discussed in the next section.

4.2 Episodic Training

The essence of meta learning involves two levels of learning. During the low-level training, tasks {Tt} (such as an N-way K-shot classification) are sampled from Dbase as support examples Dsupport, on which the learner is guided by the meta-learner to learn. Then during the high-level training, the learner is evaluated on Dquery (which is also fromDbase and held out from Dsupport) with some meta-objective to find out how well the meta-learner is capable of guiding the learner to generalize to new tasks, and then the meta-learner is optimized using the meta-objective (which is cross-entropy loss in our case). This whole process is called an episode [Vinyals et al., 2016]. By repetitively episodic training, which is essentially mimicking the expected few-shot setting, the metalearner will gradually acquire the ability to generalize on fewshot task T ′ fromDnovel held-out from {Tt}.

Following the idea of episodic training, we first sample N-way K-shot tasks from Dbase as Dsupport, and treat the sampled attributes from Abase as fake novel attributes Afake novel. We then produce the classification weights ˆWfake novel for Afake novel under the guidance of APGM (and ARM). Then we evaluate the classification performances of ˆWfake novel on Dquery, and let the gradient backpropagated to update the APGM and ARM.

The whole episodic training procedure on Dbase is shown in Algorithm 1.

4.3 Attribute Prototype Generator Module

The basic pipeline of APGM is shown in Figure 2(b). Note that different from the multiclass classification case, where each feature embedding only belongs to a single class, here each feature embedding contains fused information of multiple attributes, and the goal of APGM is to make efficient transformation to separate the fused information so that the generated APE is discriminative enough for classification.

The APGMmainly consists of two components: a pool and an MLP. The pool contains Nnovel slots, and each slot stores all the relevant feature embedding for one novel attribute. The

Algorithm 1: N-way K-shot Episodic Training

Input : Base dataset Dbase, Base modelMbase, Base attributes set Abase, Number of sampled fake novel attributes Nfake novel, Number of support examples Ksupport, Number of episodes E. Output: Mbase with trained APGM,ARM modules while episode ≤ E do Afake novel = Sample(Abase, Nfake novel) Dfake novel,Dfake base = Split(Dbase) // where Dfake novel contains at least one // attribute fromAfake novel Dsupport = Sample(Dfake novel, Ksupport) Dquery = Dbase \ Dsupport for (xi, yi) in Dsupport do zi = FeatureExtract(xi) Zsupport.append(zi) end ˆWAPE = APGM(Zsupport, Dsupport) ˆWARM = ARM(Wfake base, Dsupport) ˆWfake novel = ˆWAPE + ˆWARM ˆW = concat(Wfake base, ˆWfake novel) // low-level learning for (xj, yj) in Dquery do score = FeedForward(xj, [F, ˆW]) update AGPM and ARM using meta-objective L(score, yj ) // high-level update end end return APGM, ARM module

MLP serves as the parameterized meta-learner, which is optimized during the high-level learning. A well-trained APGM works as follows. Suppose that we have Nnovel novel attributes Anovel, where in Figure 2(b) Nnovel = 3. First, the ith example (xi, yi) ∈ Dnovel are fed into the feature extractor to obtain the feature embedding zi. With its annotated novel attributes {ai1, ..., aili } ⊆ Anovel, we push zi into the slots corresponding to ai1, ..., aili , where in the figure, zi is pushed into the first and the third slots. After forwarding the whole few-shot dataset Dnovel, we aggregate features over each slot, and feed the aggregated embedding for each novel attribute into the MLP to produce the APE. The aggregation operation can have multiple choices, and we choose summation in our experiment. The whole forward process of APGM module during meta training is shown in Algorithm 2.

4.4 Attribute RelationshipModule

During the generation of attribute prototype generation, only the novel attributes Anovel in Dnovel is needed. If we are further provided with the base attributes Abase on Dnovel, then we can exploit the well-trained base classification weights (the output node in blue) and attributes relationships for further improvements.

The ARM module shares a similar structure with APGM module with a disjoint set of parameters where the input to

Algorithm 2: APGM Module

Input : Support set Dsupport = {(xi, yi)}, support features Zsupport = {zi}, fake novel attributes Afake novel. Output: Attribute Prototype Embedding ˆWAPE for zi in Zsupport do Get xi's attributes [ai1, ai2, ..., aili ] for aij in [ai1, ai2, ..., aili ] do slot[aij ].append(zi) end end for ak in Afake novel do ˆWAPE[ak] = MLP(Aggregate(slot[ak])) end return ˆWAPE

the ARM is the base classification weights Wbase (the output node in blue). When forwarding an example (xi, yi), whose attributes are denoted as {ai1, ..., ail}, for each co-occurred base/novel attribute pair (aij, aik), aij ∈ Abase and aik ∈ Anovel, we push waij ∈ Wbase into aik's corresponding slots. Then all the weights in the slots are aggregated and feed into the MLP. This process is in essence a weighted attention over Wbase determined by how often two attributes will co-appear on the same instance.

Finally, the output classification weights for novel attributes are the addition of APGM and ARM's outputs. During practical implementation, we further normalize the classification weights to be unit vector following [Qi et al., 2018], in which case the softmax classifier is equivalent to nearest neighbour with cosine distance, which will help stabilizing the meta-training.

4.5 Inference

During the second phase when the actual novel attributes come, we fix the meta-learner, and feed forward the whole Dnovel through APGM (and ARM if possible) to produce the classification weights Wnovel for Anovel. Then a complete model Mext is obtained by concatenating Wbase and Wnovel, where the feature extractor F remains unchanged. It is worth noting that unlike re-training or finetuning which entails several steps of backpropagation and optimization, our method only requires forwarding Dnovel once to generate the Mext, which could be valuable for applications where models need to update on-device.

4.6 Discussions: Relationships with Other Meta Learning Methods

In this section, we discuss the relationships among our method and two state-of-the-art meta learning based methods [Snell et al., 2017; Gidaris and Komodakis, 2018]. First we analyze by decoupling the whole network M into feature extractor F and classifier W. In [Snell et al., 2017] the Prototypical Network adopts the entire feature embedding encoder F as meta-learner, where the classifier W is the non-parametric KNN, which indicates that the prototype is the feature embedding itself. Both [Gidaris and Komodakis,

2018] and our method use an extra parameterized weight generator as meta-learner, which outputs the classifier weights W. However, the meta-learner in [Gidaris and Komodakis, 2018] is a Hadamard production with learnable parameters that transforms the feature embedding into W, where our meta-learner is an Multi-Layer Perceptron (MLP).

As we will illustrate in the experimental section, that simply modifying [Snell et al., 2017; Gidaris and Komodakis, 2018] to their multi-label versions perform poorly in our setting. For Prototypical Network, we argue that choosing F as the meta-learner in multi-label setting will not be able to learn class prototypes that generalize and transfer as they do in the multiclass case. This is because the same prototype can be attributed to multiple labels, thus it is almost impossible to represent prototype information frommultiple labels within a single class prototype. Moreover, updating the F during episodic training will cause catastrophic forgetting on the base attributes. By comparison, our method and [Gidaris and Komodakis, 2018] avoid this issue by decoupling the meta-learner and the feature extractor F, so that F can remain unchanged during the second phase, thus keeping the performances on the base attributes. However, [Gidaris and Komodakis, 2018] uses Hadamard production with learnable parameters as meta network, which indicates that the same set of coefficients is used for all attribute prototypes generation and we argue that it is not discriminative enough to disentangle the fused information and produce the classification weights. Our meta network, however, is an MLP that provides different sets of coefficients for each class prototype generation, thus leading to more discriminative transformation and superior performances.

5 Experimental Results

5.1 Data Preparation

We evaluate on two pedestrian attribute benchmark datasets PETA and RAP. We first modify these two datasets into incremental few-shot setting. To be specific, for N-way Kshot incremental few-shot learning, we first sample N attribute groups (see [Zhao et al., 2018]), then randomly choose one attribute from each of the selected N groups to form Anovel, and treat the remaining attributes as Abase. We follow [Li et al., 2015] and [Li et al., 2016], and divide the PETA and RAP into train/val/test set with 5 random partitions. All the reported results are the average results of these 5 partitions. We then further divide each the train/val/test set into two parts: base and novel. An example is included in the novel sets if it contains at least one novel attribute, and is included in the base sets following the same rule. We then randomly keep K examples in the trainnovel set and discard the rest of the examples. During phase one, the model is trained on trainbase/valbase, and test with testbase. During the second phase, the model is provided with the K-shot trainnovel/valnovel, and test on the whole test set containing both base and novel attributes.

5.2 Baselines

Finetune with Different Learning Rate is the most straight-forward method in dealing with few-shot settings.

We fix the learning rate of classifier W to 0.001, and without specification, the lr refers to the learning rate of the feature extractor F.

Prototypical Network [Snell et al., 2017] is a simple yet effective meta learning method. During the experiment, we change the metric from Euclidean distance to cosine similarity to cater for the multi-label setting.

DFWF [Gidaris and Komodakis, 2018] is a recent state-ofthe-art meta learning method. During the experiment, we also modify the original DFWF for the multi-label setting.

5.3 EvaluationMetric

In the experiment, we mainly focus on four metrics: (1) Class-centric mA: the mean average of each attribute's accuracy of positive and negative example. Specifically, we mainly focus on mAbase and mAnovel as they indicate how well the model can adapt to the novel attributes as well as remembering the old attributes in our class-incremental setting. (2) Instance-centric: the Precision, Recall and F1 calculated on each instance.

5.4 Implemetation Details

Base Network. Since we focus on the learning algorithm which is agnostic to the specific model, we adopt DeepMAR[Li et al., 2015] with ResNet50 as backbone as our base model Mbase for its competitive performances and simplicity.

Hyperparameters. In order to train our meta network, we adopt SGD and a small learning rate of 10-5 , with a momentum of 0.9 and weight decay of 0.0005. We choose the batch size to be 32, train for 800 episodes and choose the sampled Nfake novel to be twice the expected Nnovel. During the meta training, we re-sample the support set if any afake novel is absent in the Dsupport.

Alleviating Overfitting. In order to alleviate overfitting to a certain sampled task, we apply early stopping when the loss stops to decrease within each episode. During inference, we choose the model with the smallest training loss at the end of each episode as our final extended model.

5.5 Results on 5-way 50-shot Learning

We first experiment on PETA and RAP with 5-way 50-shot learning with two sets of randomly selected novel attributes, and the result is shown in Table 1. Here only novel attributes are annotated on Dnovel. It is also worth to mention that although the total number of examples is 50, these attributes are highly imbalanced, where some attributes only contain less than 5 positive examples. The attribute distribution of the first five selected novel attributes is shown in Figure 3. Without specification, the rest of the results are reported on these five novel attributes. The results from Table 1 show that compared to finetuning (Ft) with different learning rates, our method outperforms significantly in mAnovel on both datasets, and holds comparable F1 on RAP while suffers a decrease in F1 on PETA. The drop in F1 mainly comes from the drop in Precision, and further regularization might be considered in future work. Besides, finetuning comes at a cost of forgetting on the base attributes where our method is able to retain the base model's performances. From the class incremental perspective, the class-centric metric mAbase/mAnovel demonstrates our method's superiority in learning novel attributes on such few-shot learning setting. Moreover, finetuning involves several backpropagation and updating while our method requires much less time consumption. Finally, the results also accord with the discussions in Section 4.6, that the current meta learning method is not applicable in our setting where the ProtoNet acts almost like random guess, and DFWF also yields poor performances.

5.6 Effect of ARMModule

If we have access to the base attributes of the novel data, then we can apply ARM for further improvements. We experiment on PETA with 5-way 50-shot, and RAP with 8-way 80shot where ARM can bring improvements in mAnovel from 67.16%/71.90% to 68.05%/72.88%.

5.7 Effect of Number of Ways and Shots

We compare our method against finetuning (lr=0.0001) with different number of shots and ways. The results in Figure 4 show that our method is able to consistently outperform the baseline on mAnovel. With different number of ways, our method is still able to generalize well to the novel attributes. However, the gap gets smaller as the number of shots increases, which implies that finetuning becomes a stronger baseline as the number of examples increases, and our method's superiority lies mostly on the few-shot scenarios.

During the experiments, we also empirically find that sampling more fake novel attributes than the expected number of ways, will yield a mAnovel gain of ∼ 1%. This training strategy is firstly adopted in [Snell et al., 2017], which can be explained intuitively that making the sampled task more difficult (higher ways) during meta training is beneficial to the future generalization in the actual test case.

6 Conclusion and FutureWork

In this paper, we raise the practical issue of incremental few-shot learning for pedestrian attribute recognition, a commonly existing problem which has gained little attention by the academic community. We tackle this challenging task by proposing a meta learning based framework, which only requires annotations on the novel attributes and one feed forward time to produce the adapted model. We evaluate our method against finetuning and other state-of-the-art meta learning methods, and argue that our method is able to rapidly adapt to the novel few-shot data with low resource consumption and yield competitive performances. We believe that this task is far from fully solved, and future work may involve designing meta-learner with stronger representation capability such as graph neural networks.

Acknowledgements

The work is supported by National Key R&D Program of China (2018YFC0806900), National Natural Science Foundation of China (61571269) and National Basic Research Program of China (2015CB352300).