Multi-scale and Discriminative Part Detectors Based Features for Multi-label Image Classification

1 Introduction

Multi-label image classification has attracted particular attention recently driven by its broad applications [Geng and Luo, 2014; George and Floerkemeier, 2014; Gong et al., 2013; Jing et al., 2015; Li et al., 2016a; Li et al., 2016b; Li et al., 2017; Murthy et al., 2016; Tan et al., 2015; Wang et al., 2016; Wei et al., 2014; Wei et al., 2016; Xie et al., 2017b; Yeh et al., 2017; Zhu et al., 2017]. The task of multi-label image classification is to predict the presence or absence of multiple specific object categories in an image. Compared with single-label image classification which has been actively studied in recent years [Herranz et al., 2016; Krizhevsky et al., 2012; Simon et al., 2014; Simonyan and Zisserman, 2015; Szegedy et al., 2015], multi-label image classification is a more practical problem because most of the real-world images usually contain multiple objects from different categories. Besides, as shown in Figure 1, each object class in real-world multi-label images often has large intra-class variations caused by occlusion, scale, viewpoint, illumination, etc., and the composition and interaction between object categories also increase the complexity of the problem, which make the task of multi-label image classification more challenging.

* Corresponding author.

During the past few years, various deep learning methods especially convolutional neural networks (CNNs) have shown their promise as a universal representation and have dominated most of the recent works on image classification task. However, most research efforts made on image classification mainly focus on addressing the task of single-label image classification. Although several recent works [Oquab et al., 2014; Sharif Razavian et al., 2014; Simonyan and Zisserman, 2015] have demonstrated that a pre-trained CNN model can also be straightforwardly transferred to multi-label image classification, they do not perform well for recognizing complex object layouts and scenes in multi-label images. This mainly because global CNN features still lack geometric invariance for addressing the problem of intra-class variations and so are not optimal for multi-label image classification.

In this paper, we propose a novel and effective framework built upon CNNs to learn multi-scale and discriminative part detectors (MsDPD)-based feature representations for the task of multi-label image classification, as shown in Figure 2. Specifically, at each scale, we first present an object-proposal-free and entropy-rank based scheme to generate and select a number of discriminative part detectors (DPD). Then, we obtain a set of DPD-based feature maps with each feature map representing the occurrence probability of a particular part detector, and learn the pooled DPD-based features by using a task-driven pooling scheme. We formulate the two steps into a unified optimization framework, which trains part detectors incrementally and integrates the learning of feature representations into the classification task, as shown in Figure 3. Finally, the features from different scale levels are aggregated to produce the ultimate multi-label predictions. In the experiments, we evaluate the proposed framework on the PASCAL

VOC 2007 and VOC 2012 datasets [Everingham et al., 2015] and achieve state-of-the-art results when compared with the existing multi-label image classification methods.

To sum up, our main contributions are as follows. First, we propose a unified framework by leveraging the highly expressive CNNs to learn a kind of discriminative part detectors-based feature representation, termed MsDPD, to address the problems of intra-class variations faced for multi-label image classification. The proposed approach formulates the training of part detectors and the learning of feature representations into a unified optimization framework by developing a new objective function. Second, we present an entropy-rank based scheme to evaluate the distinctiveness of part detectors and then train part detectors incrementally by mining reliable instances iteratively. Third, we propose a task-driven pooling technique to integrate the learning of feature representation into classification task to improve its generality.

Fourth, different from previous region proposal-based image classification methods [Wei et al., 2016; Wu et al., 2015; Yang et al., 2016], our method does not need ground-truth bounding boxes or object proposals, making the proposed method more efficient and practical. We have confirmed through experiments that the feature representation obtained by using the proposed method is capable of delivering state-of-the-art results on two popular multi-label classification benchmarks including PASCAL VOC 2007 and VOC 2012 datasets.

2 Methodology

While many CNN-based methods have achieved successful results on image classification, most of them are developed for single-label image classification by extracting global CNN features. Inspired by the fact that each object class in multi-label images generally exhibits dramatically different appearances, shapes, occlusions and interactions, we propose to extract discriminative part detectors-based features, a kind of local CNN-based features, to handle the problem of intra-class variations. Here, part detectors are used to capture generalized objects and their parts that are discriminative (being different enough from each other) and representative (occuring frequently enough). As shown in Figure 2, the core task of this proposed method is to learn Multi-scale and Discriminative Part Detectors (MsDPD)-based features for multi-label image classification. Figure 3 illustrates how to learn DPD-based features from each scale of convolutional features denoted by gray blocks. Specifically, we first present an entropy-rank based scheme to generate a number of discriminative part detectors. Then, we obtain part detectors-based convolutional feature maps and generate the pooled feature representations by using a task-driven feature pooling scheme. For ease of optimization we integrate the two steps into a unified framework by developing a new objective function to jointly train part detectors and learn the feature representations. The final multi-label prediction results are obtained by fusing the features from different scale levels.

2.1 Model Architecture

Figure 2 illustrates the overall architecture of our MsDPD framework. The basic configuration of our model is similar to that of Single Shot MultiBox Detector (SSD) [Liu et al., 2016]. The early layers (Conv1 to Conv7) are transferred from the pre-trained VGGNet-16 [Simonyan and Zisserman, 2015], where the convolutional layers Conv6 and Conv7 are converted from the fully-connected layers FC6 and FC7 by using a scheme that is similar to SSD as follows: subsample parameters from FC6 and FC7, change pool5 from 2×2 - s2 to 3×3 - s1, and use the `a trous algorithm to fill the "holes". The fully-connected layers are converted to convolutional ones to cope with the uncertainty for the localization of object parts. These layers are followed by some extra convolutional layers (Conv8 to Conv12) to extract much deeper features and even bigger object parts. The last convolutional layer (Conv12_2) is used to fine-tune the network by using multi-label images. The detailed model parameters can be found in Figure 2.

In this work, DPD-based feature representations are extracted based on four scales of convolutional layers including Conv7, Conv8_2, Conv9_2, and Conv10_2 (denoted by gray blocks in Figure 2), which decrease in size progressively to allow the detections of object parts at multiple scales. Specifically, for a convolutional layer of size m×m with n channels, we convolve it with K 1×1×n part detectors to produce a number of part detectors-based convolutional feature maps of size m×m with K channels, where each feature map represents the occurrence probability of a particular part detector, followed by a task-driven pooling step to produce the final K-dimensional DPD-based feature representation. For ease of reference, we index the four DPD-based feature layers and the last convolutional layer by using scale 1 through scale 5. Next we describe how to train part detectors and learn part detectors-based feature representations.

2.2 Initializing Discriminative Part Detectors (DPD)

To initialize the candidate part detectors that are shared across all image categories, we randomly sample a large number of (about one hundred thousands) pixels from each scale of the convolutional feature maps of all training images. Each pixel from the feature maps can be considered as a "local" CNN feature which has a very large receptive field in the original image, and the length of the pixel equals the channel number of the convolutional features. Then we perform k-means clustering over these sampled pixels and only retain sufficiently large clusters to ensure the representativeness, where each cluster corresponds to a to-be-learned candidate part detector.

We consider an object part is discriminative if it only appears frequently in some specific image categories rather than almost all classes. For example, "wheel" will occur in the object classes of "bus" and "car", so the entropy would be low. In contrast, a non-discriminative "sky" could occur uniformly in almost any of the classes with higher entropy. To select discriminative part detectors, we introduce an entropy-rank based scheme to measure the discrimination of each part detector by computing their entropies across all image categories. Specifically, the entropy   E D for a part detector D is computed by

where C is the number of image classes and   c p D is the fraction of the members of part detector D that are from the images of the c-th class. Then, we take the entropy as a measure of discrimination of a part detector to select K detectors with low entropy values.

2.3 Training DPD and Learning DPD-based Feature

Let { | 1,2, , } i i N   X   be the set of training images and { | 1,2, , } i i N   Y   be the set of image labels of  , where { | 1,2, , } n i ij j M    X x   is represented by the entries from its CNN convolutional layer of size m×m with n channels and 2 M m  , N is the total number of training images, C i  Y  denotes the ground truth label vector of sample i X with at least one element being 1, and C is the number of image classes. For a given training image i X , let

    { | 1,2, , } K i ij j M    O X O x   be its DPD-based feature maps,   K i  X   be its pooled DPD-based feature,   i X P be its multi-label prediction result,   1 1 , W b be the parameters of the to-be-learned DPD, and   2 2 , W b be the parameters of C multi-label classifiers, where 1 n K   W  , 1 K  b  , 2 K C   W  , and 2 C  b  . Thus,   ij O x and   i X P can be computed by

where 1 ( ) exp( )/ || exp( ) || S  x x x and   1 ( ) 1 exp( )      x x are the softmax and sigmoid non-linear activation functions, which are used for predicting the occurrence probabilities of DPD and the multi-label classification results, respectively.

As shown in Figure 3, to leverage CNNs to learn effective feature representation, we formulate the training of DPD and the learning of DPD-based feature representation into a unified framework, which jointly trains part detectors incrementally and integrates DPD-based feature learning into classification. To this end, we develop a new objective function as follows, which contains three terms including an image-level classification loss term, a generalized max pooling regularization term, and an object part-level classification loss term:

where 1  and 2  are two trade-off parameters that control the relative importance of these three terms.

1) Image-level classification loss term. This term is defined as sigmoid cross-entropy loss function for multi-label image classification. It aims to minimize the classification error for the given training images and is computed by

where   i c Y and   i c     X P denote the c-th entries of i Y and   i X P , respectively.

2) Generalized max pooling regularization term. This term is used to learn the pooled DPD-based feature   i X  from the input   i O X by enforcing the pooled representation to be close to each column of the input   i O X , which is computed by using the following formula

Similar to [Murray and Perronnin, 2014; Xie et al., 2015], by using this pooling regularization term, the learned DPD-based feature could enforce the dot product similarity between   ij O x and the pooled feature   i X  to be a constant one. By integrating feature learning into classification, we can use more information from   i O X to get a task-driven feature representation which is more suitable for classification than traditional pooling strategies such as max/average-pooling.

1. Object part-level classification loss term. This term is defined as softmax cross-entropy loss function for object part classification. It aims to minimize the classification error for the mined object part instances and is computed by

where K l  y  stands for object part label vector of object part instance l x with only one element being 1, K is the number of DPD selected in subsection 2.2, and t is the number of high-confident object parts we select in each iteration used for updating each part detector. In such way, we can mine reliable instances iteratively and train the DPD incrementally.

2.4 Optimization

To solve the optimization problem of Eq. (4), we present a simple EM-like iterative minimization method to update   1 1 , W b ,   2 2 , W b and  alternatively via stochastic gradient descent method (SGD) [Williams and Hinton, 1986].

1. Initialization. Given K selected part detectors, we initialize the parameters   1 1 , W b by using Eq. (7). The DPD-based feature representation  for all images are initialized by using (6) and generalized max pooling method [Murray and Perronnin, 2014]. The parameters   2 2 , W b are initialized by using Eq. (5).

2) Updating   1 1 , W b and   2 2 , W b by fixing  . The gradients of the objective function J with respect to the parameters   1 1 , W b and   2 2 , W b can be computed by

where ij g and ij z are defined as follows

with the operation  denoting element-wise multiplication.

Thus, the parameters   1 1 , W b and   2 2 , W b can be updated by using gradient descent method as follows

where  is the learning rate.

3) Updating  by fixing   1 1 , W b and   2 2 , W b . The gradients of the objective function J with respect to  can be computed by

Thus, the parameter  can be updated as follows

2.5 Multi-label Image Classification

After optimizing Eq. (4), the pooled DPD-based features for all training samples are learned at the same time. However, the feature representations of the test images still need to be learned. Since the part detectors make the distributions of training and test data consistent, we can obtain DPD-based features of test images by optimizing Eq. (6). For the ultimate predictions, we concatenate the features from different scale levels to train a set of sigmoid classifiers for prediction.

3 Experiments

In the experiments, we evaluate our method on PASCAL VOC 2007 and VOC 2012 datasets [Everingham et al., 2015], which have been widely used for multi-label image classification by predicting whether the object is present/absent in the image. The performance is measured by using the average precision (AP) and the mean AP over all object classes.

3.1 Parameter Settings

We train the proposed model as shown in Figure 2 by using SGD with initial learning rate of 10 -4 for the early layers (Conv1 to Conv7), initial learning rate of 10 -3 for the latter layers (Conv8 to Conv12), momentum of 0.9, weight decay of 0.0005, and batch size of 32. The learning rate decays by 0.1 after 60k iterations and is fixed for the rest 20k iterations. For the training of DPD and DPD-based features, the parameters in Eq. (4) are set to 1 =1  and 2 =0.01  , and the learning rate  in Eqs. (14), (15), (17) is set to 0.01.

For the initialization of DPD, we run k-means clustering on the convolutional feature maps of scales 1 to 4 with the cluster numbers being set to 2000, 1000, 400, and 200, and then take the entropy-rank based scheme as a measure to select 700, 400, 300, and 200 detectors, respectively.

3.2 Experimental Results

Comparison of features from different scales. We first give the results obtained by using different scales on PASCAL VOC 2007 dataset. Table 1 reports the detailed results.

As shown in Table 1, some object classes, such as "bird" and "bottle", fire on small scales and some object categories, such as "person" and "train" fire on big scales. This is because that our MsDPD feature layers are decreased in size progressively to allow the predictions of objects and their parts at multiple scales, thereby for better capture of object variations caused by viewpoint, scale, occlusion, etc. The best results are obtained by fusing the features of different scales.

State-of-the-art CNN-based methods. The following CNN-based methods are used for comparison: VGG-16-SVM and VGG-19-SVM [Simonyan and Zisserman, 2015], ResNet-101-Sigmoid [He et al., 2016], SDE [Xie et al., 2017a], HCP [Wei et al., 2016], CNN-RNN [Wang et al., 2016], and FeV+LV-20-VD [Yang et al., 2016]. The work of [Simonyan and Zisserman, 2015] densely extracts 4096-D CNN features across five image scales {256,384,512,640,748} of the given image with VGG-16 and VGG-19, performs global average pooling on the resulting CNN features, and finally classifies the image with linear SVM classifiers. ResNet-101-Sigmoid trains a multi-label classification system using a pre-trained ResNet-101 model [He et al., 2016] with a sigmoid cross entropy loss function, densely computes sigmoid outputs across five image scales {256,384,512,640,748} of the given image, and finally performs classification by max-pooling the resulting sigmoid outputs as HCP [Wei et al., 2016]. SDE [Xie et al., 2017a] presented a feature learning framework by optimizing the features with the aim of learning selective, discriminative and equalizing representations. HCP [Wei et al., 2016] proposed to address the multi-label classification by extracting object proposals from the given images and the final image-level scores are obtained by max-pooling the scores of the proposals. CNN-RNN [Wang et al., 2016] combined RNNs with CNNs in a unified framework to learn a joint image-label embedding. FeV+LV-20-VD [Yang et al., 2016] proposed a multi-view multi-instance framework to utilize both weak and strong labels (bounding box).

Comparison with state-of-the-art methods on PASCAL VOC 2007 dataset. Table 2 summarizes the results of our MsDPD method and the aforementioned seven state-of-the-art methods on PASCAL VOC 2007 dataset. As shown in Table 2: (i) Compared with global CNN-based approaches such as VGG-16-SVM and VGG-19-SVM [Simonyan and Zisserman, 2015], ResNet-101-Sigmoid [He et al., 2016], and CNN-RNN [Wang et al., 2016], our proposed method obtains significant performance gains of 4.2%, 3.3% and 9.5% in terms of mAP. This shows the superiority of our local CNN-based method. (ii) Compared with other methods, such as HCP [Wei et al., 2016], SDE [Xie et al., 2017a], and FeV+LV-20-VD [Yang et al., 2016], which can be regarded as a kind of local feature based methods, our method still outperforms them with a big margin measured in terms of mAP (at least 2.3%).

Comparison with state-of-the-art methods on PASCAL VOC 2012 dataset. We report our experimental results in Table 3 and compare it with six state-of-the-art CNN-based methods on VOC 2012 dataset. The results are consistent with those on the VOC 2007 dataset. To be specific, we achieve state-of-the-art results for 16 out of 20 object categories. Especially for the difficult categories such as "chair", "cow",

"table", "plant", and "sofa", our method shows good performance. This significant performance gain shows the effectiveness of our DPD-based feature representation.

Ablation experiments. To analyze the importance of each component of our method (part detectors and task-driven pooling), we conducted ablation experiments on the PASCAL VOC 2007 dataset. Table 4 shows the results obtained with part detectors and without part detectors (pool features from Conv7/Conv8_2/Conv9_2/Conv10_2 layers) by using different pooling strategies measured in terms of mAP. As shown in Table 4, task-driven pooling obtains the highest mAP than max-pooling and average-pooling. More importantly, by using our proposed part detectors could obtain big accuracy gains compared with that obtained by directly pooling features from the original convolutional layers.

4 Conclusion

In this paper, we proposed to build upon CNNs to learn part detectors-based features for multi-label image classification. To this end, we first present an entropy-rank based scheme to obtain a set of discriminative part detectors. Then, we generate part detectors-based convolutional feature maps and learn part detectors-based features with a task-driven pooling scheme. For optimization, the aforementioned two steps are formulated into a unified framework by developing a new objective function, which incrementally trains part detectors and integrates the learning of feature representations into the classification task. However, by using the proposed objective function it is difficult to train the whole network end-to-end. Our future work will address this issue.

Acknowledgments

This work was supported in part by the NSFC under Grants 61772425 and 61473231, in part by the Natural Science Basic Research Plan in Shaanxi Province of China under Grant 2017JM6044, and in part by the Fundamental Research Funds for the Central Universities under Grant 3102018zy023.