Saliency Guided End-to-End Learning for Weakly Supervised Object Detection

1 Introduction

With the prevalence of deep architectures, significant progress has been made recently in object detection. However, most state-of-the-art object detectors [Girshick, 2015; Ren et al., 2015; Redmon et al., 2016; Dai et al., 2016] require instance-level bounding boxes and their associated class labels for training. Such full supervision is expensive to obtain, preventing these methods from large-scale practical applications. In order to release the burden of annotation, researchers opted to learn object detectors under weak supervision, in which only image-level labels are used. Image-level annotations are readily available in large amounts because they can be obtained by inexpensive human annotation or collected from the web. Thus, the weakly supervised setting is attractive, especially for the data-hungry models of today.

Although weakly supervised object detection (WSOD) sidesteps the labor-intensive annotating process, the lack of location supervision makes it an extremely challenging problem. In recent years, increased efforts have been made in WSOD. Early attempts are often based on the multiple instance learning (MIL) framework [Song et al., 2014; Bilen et al., 2014; 2015; Shi and Ferrari, 2016; Cinbis et al., 2017], and recently more on deep convolutional neural networks (CNN) [Oquab et al., 2015; Zhou et al., 2016; Bency et al., 2016; Bilen and Vedaldi, 2016; Kantorov et al., 2016]. Most of these methods learn object localizers, together with classifiers, in either an alternative or a parallel way. Object size [Shi and Ferrari, 2016], context [Kantorov et al., 2016], class activation maps [Diba et al., 2016; Zhou et al., 2016] and other cues [Bilen et al., 2014; 2015] are exploited to infer locations better.

∗The corresponding author

In this paper, we propose a novel method that integrates saliency information into an end-to-end architecture to perform WSOD. Our work is motivated by two research findings. (1) The end-to-end training is one of the key ingredients making CNNs remarkably successful in fully supervised vision tasks. Therefore, it is reasonable to expect good performance when applying to WSOD. (2) The knowledge learned in CNNs has been gradually explored. For instance, [Simonyan et al., 2014; Shimoda and Yanai, 2016] have developed a way to infer class-specific saliency maps from the CNNs pre-trained on the large-scale image-level classification tasks [Russakovsky et al., 2015]. We believe such information can be effectively used in the WSOD task.

In order to utilize saliency information derived from the pre-trained CNNs for WSOD, we make contributions in the following aspects:

• With the guidance of class-specific saliency maps, we propose a context-aware approach that selects classspecific proposals (referred to as seeds). The seeds are of highly confident location and semantic information.
• We propose two types of strategies to integrate saliency into an end-to-end architecture. One is using the location, semantic, and saliency information of seeds to explicitly regularize the network. The other is embedding a saliency sub-network into the architecture to predict the saliency of each proposal. The saliency prediction is multiplied to each proposal to indicate the possibility of localizing an object.
• Extensive experiments on the PASCAL VOC 2007 and 2012 datasets demonstrate signification improvements over the baseline.

2 RelatedWork

Existing approaches for WSOD can be roughly categorized into MIL-based and CNN-based research lines. Therefore, we make a brief review along these two lines. The work related to the guidance of saliency is also introduced.

2.1 MIL-basedWSOD

A conventional way to learn object detectors under weak supervision formulates the task as a MIL problem. It treats each image as a bag of object proposals generated by certain methods [van de Sande et al., 2011; Zitnick and Dollr, 2014]. When an image has a positive class label, the image must contain at least one proposal of that class. Negative images only contain negative instances. The MIL strategy alternatively learns object classifiers and uses the classifiers to select the most likely object proposals in positive images.

However, the formulated MIL problems are non-convex and prone to stuck in local optima. To address this issue, different strategies are developed, which either help to make good initializations [Kumar et al., 2010; Song et al., 2014; Shi and Ferrari, 2016; Cinbis et al., 2017], or regularize the models with extra cues [Bilen et al., 2014; 2015; Shi and Ferrari, 2016]. These methods have demonstrated their effectiveness, especially when CNN features are used for representing object proposals [Shi and Ferrari, 2016; Cinbis et al., 2017]. Thus, a current trend along this research line is to integrate the MIL strategy with deep networks, as the exemplar work done in [Diba et al., 2016].

2.2 CNN-basedWSOD

Recent efforts have been devoted more on CNN-based WSOD. The methods either sequentially or parallelly learn object classifiers, together with localizers that select the best candidates also from an initial object proposal set. In these work, CNNs are exploited mainly for the purpose of knowledge transfer or end-to-end learning.

Knowledge transfer aims to utilize the knowledge learned with CNNs on other vision tasks to help the learning of object detector under weak supervision. For instance, the CNN pretrained on the large-scale image-level classification task [Russakovsky et al., 2015] not only is able to extract discriminative features but also has the localization ability. Therefore, [Shi and Ferrari, 2016; Cinbis et al., 2017] use CNNs to represent the features of object proposals. [Oquab et al., 2015; Zhou et al., 2016; Li et al., 2016; Bency et al., 2016] explore the encoded semantic and spatial information in convolutional layers to predict approximate locations of objects. Most of these methods treat classification and localization as separate procedures. Thus, their localization performance is limited.

End-to-end learning is an important ingredient to make CNN remarkable in fully supervised vision tasks. Intrigued by its success, very recent efforts are often made to construct end-to-end architectures for WSOD. For instance, WSDDN [Bilen and Vedaldi, 2016] proposes a two-stream network to parallelly learn classifiers and localizers in an endto-end manner. [Kantorov et al., 2016] incorporates context information into the two-stream network. WCCN [Diba et al., 2016] integrates the MIL strategy into an end-to-end deep network. Benefited from the end-to-end training, these methods achieve state-of-the-art performance. Our work is along this research line. Different from these approaches, we explore highly confident information from class-specific saliency maps and integrate it into the network to supervise the end-to-end training.

2.3 The Guidance of Saliency

Saliency detection [Zhu et al., 2014; Simonyan et al., 2014] can automatically highlight image regions containing objects of interest. Thus, the generated saliency maps provide approximate information of object locations. Based on this observation, saliency has been used as a prior in different weakly supervised vision tasks [Lai and Gong, 2016; Zhang et al., 2016; Shimoda and Yanai, 2016]. In WSOD, category-free saliency detection [Zhang et al., 2016] is exploited via a selfpaced curriculum learning strategy. [Teh et al., 2016] designs an attention network for WSOD. [Simonyan et al., 2014; Shimoda and Yanai, 2016] propose a convenient way to get class-specific saliency maps from the CNN pre-trained on ILSVRC [Russakovsky et al., 2015], which provide both location and semantic information. Our work is motivated by their findings. We aim to incorporate such class-specific saliency maps into an end-to-end framework to boost the performance of detection.

3 The ProposedMethod

In this section, we introduce the constructed network and the training procedure. Assume that a collection of images is given. It has NI training images and C object classes. Each image is labeled with a vector y = [y1, ..., yC]T ∈ {1,-1}C , which indicates the presence/absence of object categories. For each image, an initial set of region proposals is first generated by COB [Maninis et al., 2016] and the corresponding bounding boxes (object proposals) are fed into the network. Meanwhile, class-specific saliency maps {M1, ...,MC} are obtained by [Shimoda and Yanai, 2016; Simonyan et al., 2014] to guide the training of the entire network.

Our architecture is built based on the weakly supervised deep detection network (WSDDN) [Bilen and Vedaldi, 2016], which consists of a pre-trained CNN model and a classification-localization sub-network optimized with respect to an image-level loss function. In contrast to WSDDN, we make the following modifications: (1) We construct a saliency sub-network, composed of two fully-connected layers and one sigmoid layer, to predict the saliency score for each object proposal. The score is then multiplied with the feature for weighting the proposal. (2) Highly confident object proposals are selected as seeds and used to guide the training of both sub-networks. Particularly, a seed's saliency loss and a seed's classification loss are designed, respectively, to regularize the saliency sub-network and the classificationlocalization sub-network. Figure 1 presents the overview of our architecture. Note our two modifications are not coupled with WSDDN, so they can also be embed into other frameworks.

3.1 Context-aware Seed Selection

This part aims to select a highly confident object proposal for each labeled class under the guidance of saliency. The selected proposals are referred to as seeds, which will be further used to supervise both the saliency sub-network and the classification-localization sub-network.

Specifically, let us denote the region proposals of an image as R = {R1, ...,RNR }, where NR is the total number. Since COB [Maninis et al., 2016] produces the proposals by first segmenting the image into superpixels and then grouping them. We also denote the superpixel set as S = {S1, ..., SNS }, where NS is the corresponding total number. For a region proposal Ri, we compute its saliency score specific to a class c by averaging its pixels' saliency values. That is,

where Area() counts the pixel number within a region; p is a pixel; and Mc(p) gets p's saliency value in the class-specific saliency map. The saliency score for Ri's neighborhood is computed by

where N(Ri)) represents the set of superpixels adjacent to Ri. The class-specific saliency contrast is now defined as:

which is large when the region proposal is salient while its local context is not. This contrast also takes into account the region proposal's area to avoid the selection of tiny regions.

For each labeled class, the proposal which has the highest class-specific saliency contrast is chosen as the seed. Note that although the proposed approach is similar to the maskout strategy [Li et al., 2016], our superpixel-wise manner can better localize object boundaries so that better object proposals are selected. In addition, in contrast to the methods [Diba et al., 2016; Zhou et al., 2016] that select candidates by directly thresholding the class-specific saliency maps then computing the mininum enclosing rectangles, our approach can deal with complex scenarios better. Figure 2 demonstrates some examples, in which green boxes are the bounding boxes of our seeds while red boxes are from the thresholding method [Diba et al., 2016; Zhou et al., 2016]. The results show that our method can select good seeds even if two objects are connected, or an object is broken, or an object is diffused to the background in the saliency maps.

3.2 Seed's Classification Loss

The class-specific seeds selected above have both semantic and spatial information with high confidence. Thus, we use them to guide the classification-localization sub-network. A loss is designed to encourage the seeds to be classified into their corresponding categories. In specific, let us denote Φ ∈ RC×NR as the output of this sub-network, in which each column indicates the classification scores of a proposal. The loss is then defined as:

Here, Λc is the index of the seed for labeled class c; Φ(c, Λc) is the entry in the c-th row and Λc-th column of Φ; and T = {i|yi = 1} indicates the categories presented in the image.

3.3 Seed's Saliency Loss

The seeds are also used to guide the training of the saliency sub-network. In this sub-network, when the feature of the i-th object proposal is input, the Sigmoid layer outputs a categoryfree saliency score Pi ∈ [0, 1]. We take the seed proposals as salient samples and choose the same number of negative proposals having the lowest values in the class-specific saliency maps. The "ground-truth" saliency scores for these positive and negative samples are, respectively, assigned as 1 and 0. Denote Λs as the indexes of the both positive and negative proposals, and Q as their "ground-truth" scores. Then, the saliency loss is defined by

3.4 Image-level Classification Loss

The classification-localization sub-network outputs class prediction scores for each proposal. The image-level scores are then obtained by summation over proposals, which is

Then the image-level classification loss is defined by

which is a binary-log-loss.

3.5 Network Training

We now summarize the entire loss function of our network. Note that the losses defined above are all functions of the network's parameters w, which were dropped for notational convenience. The losses are also defined with respect a single image. When considering the entire collection, we get the loss of our network as follows:

where λ1, λ2 and λ3 are the weighting factors. The L2 regularization term is added to avoid overfitting. This function is optimized by stochastic gradient descent with momentum.

4 Experimental Results

In this section, we present a series of experiments to thoroughly investigate the performance of our approach.

4.1 Datasets and EvaluationMetrics

The experiments are conducted on the PASCAL VOC 2007 and 2012 datasets [Everingham et al., 2010], which are the benchmark most widely used in WSOD. The VOC 2007 dataset contains 2501 training, 2510 validation, and 4952 test images. VOC 2012 has 5717 training, 5823 validation, and 10991 test images. Both datasets have 20 object categories. In all the experiments we follow the standard training/validation/test splits.

We investigate the performance of both object detection and classification. For detection, two metrics are adopted, which are average precision (AP) and correct localization (CorLoc) [Deselaers et al., 2012]. AP is the standard metric for PASCAL VOC that is measured at 50% intersection-overunion of the detected boxes with ground truth. CorLoc is the percentage of positive images that have at least one object correctly localized. As previous methods, AP is evaluated on the test sets and CorLoc is reported on the training and validation sets. For classification, the standard average precision is measured on the test sets.

4.2 Experimental Setup

Our architecture is constructed on the baseline network WSDDN [Bilen and Vedaldi, 2016], which transforms a pretrained CNN by replacing the last pooling layer with a spatial pyramid pooling (SPP) layer and connect it to the classification-localization sub-network. Based on WSDDN, we embed in a saliency sub-network composed of two fully connected layers, respectively, of 512×NR and 1×NR outputs, together with a Sigmoid layer outputing a 1×NR vector. In addition, two loss functions regarding to the selected proposals are also imposed. The entire network is trained in an end-to-end manner and errors are back propagated through the green lines in Figure 1. For testing an image, the imagelevel labels and class-specific saliency maps are unavailable. The output of the classification-localization sub-network is taken as the detection score for each proposal.

Our approach is implemented using the MatConvNet toolbox [Vedaldi and Lenc, 2015]. For training, we run 20 epochs, in which the first 10 epochs take a learning rate of 10-5 and the second 10 epochs take 10-6 . Each image is randomly flipped and scaled to have maximal width or height of {480, 576, 688, 864, 1200} with respect to the original aspect ratio. In test, each image is resized to the five scales and the detection scores over all scales are averaged. The hyper parameters in our network are set empirically as σ = 103 , λ1 = 0.1, λ2 = 1 and λ3 = 5 × 10-4 .

4.3 Experimental Results

Performance of different components. We first conduct experiments to investigate the effectiveness of each proposed component. Therefore, three configurations are tested: (1) the full model, referred to as SGWSOD; (2) the model removing the saliency sub-network and the saliency loss, denoted as SGWSOD-SAL; (3) the model removing all proposed modifications, denoted as SGWSOD-SAL-SC, which in essence is the baseline network WSDDN [Bilen and Vedaldi, 2016]. All these models are built on the pre-trained VGG-VD16 [Simonyan and Zisserman, 2015]. Table 1 reports the detection and classification APs evaluated on the PASCAL VOC 2007

test set. It shows that both the saliency sub-network and the additional losses contribute substantial improvement.

Comparison with the state-of-the-arts. We compare our full model with the state-of-the-arts. Table 2, 3, and 4 report the metrics evaluated on VOC 2007, including the scores for each class and the mean score of all classes. The comparators include one MIL-based [Cinbis et al., 2017] and three CNN-based [Teh et al., 2016; Diba et al., 2016; Bilen and Vedaldi, 2016] methods. [Teh et al., 2016] includes an attention network as well, but with different design. WCCN [Diba et al., 2016] is the most recent end-toend method. WSDDN [Bilen and Vedaldi, 2016] is the baseline of our model, whose variants corresponding to three pretrained CNNs, including VGG-CNN-F, VGG-CNN-M-1024, and VGG-VD16 [Simonyan and Zisserman, 2015], are investigated. The ensemble results that average over three variants are also provided. Correspondingly, our three variants and the ensemble results are also presented. The results show that the proposed method achieves significant improvement over the baseline with respect to all three metrics. Our SGWSOD VGG16 and SGWSOD ensemble outperform all the state-of-thearts, except the CorLoc of [Teh et al., 2016] whose network is designed particularly for localization.

In addition, two phenomena can be observed from these results. (1) In contrast to WSDDN, our approach achieves significant improvements especially on 'bike', 'car', 'cat', 'table', 'dog', and 'person' classes. (2) WSDDN VGG16 achieves the detection performance similar to or even worse than its two variants (34.8 vs. 34.5 and 34.9 in detection AP, and 53.5 vs. 54.2 and 56.1 in CorLoc). In contrast, our SGWSOD VGG16 performs better than the corresponding variants (43.5 vs. 38.9 and 39.4 in detection AP, and 62.9

Table 6: CorLoc (%) on the PASCAL VOC 2012 trainval set.

Table 7: Classification AP (%) on the PASCAL VOC 2012 test set.

vs. 60.8 and 59.6 in CorLoc). The detection performance of three pre-trained CNNs in our network is consistent with the classification performance, and also consistent with fully supervised detection methods [Girshick, 2015]. These observations show that, with the additional regularization losses, our model can avoid overfitting better than the baseline.

Table 5, 6, and 7 report the results evaluated on VOC 2012. Due to the space limit, only the VGG16 and ensemble results of our model are included. Since most of the compared methods have not published their evaluations on this dataset in detail, we include all the results publicly available. Compared to the most recent method WCCN [Diba et al., 2016], our approach gains about 3 point improvement on detection.

Figure 3 illustrates typical detection results. The correctly detected objects are marked as green bounding boxes and the failed cases are on red. The results show that our approach can successfully deal with the following cases: (1) an image containing multiple objects which are from either the same or different categories, (2) an image is occupied by one large object. However, although better thanWSDDN, our approach is still prone to group objects together when they are occluded by each other and focus on object parts. When an object is of low contrast to its background, the approach fails also.

5 Conclusion

In this paper, we have presented an approach integrating class-specific saliency maps into an end-to-end architecture to performWSOD. It exploits saliency information thoroughly to boost the performance of both detection and classification. Experiments on PASCAL VOC show that the proposed method achieves significant improvement over the baseline and performs better than existing weakly supervised object detection methods.

Acknowledgments

This work was supported in part by the State High-Tech Development Plan (863 Program) of China under Grant 2014AA09A510 and in part by the Natural Science Foundation of Zhejiang Province, China under Grant LY17F010007.