Low Shot Box Correction for Weakly Supervised Object Detection

1 Introduction

Weakly Supervised Object Detection (WSOD) has been widely studied because annotating a large-scale dataset with bounding boxes is expensive and time-consuming. For WSOD methods, datasets with only image-level annotations are required to do the task of detecting objects. Although a lot of great works have been made in this area, the accuracy of state-of-art methods remains far lower than their strongly supervised baselines (0.47 vs 0.67 mAP by Fast-RCNN on PASCAL VOC 2007).

We think one of the major reasons for this huge gap might be the incomplete box detection. Here, we take a simple example to illustrate this problem. Given an image with label person like the first image in Figure 1, the trained detector only knows that there's a person in this image but has no idea of where she/he is. Most of previous methods are structured on deep classification networks and tend to recognize the most discriminative parts (red boxes) instead of complete object bounding boxes (green boxes). In the above example, these models are tended to recognize the person's head instead of the full body because heads are similar and easy to distinguish while the full bodies are much more diverse and harder to distinguish due to different clothes or postures. For some object labels like person, their most discriminative parts are much smaller than complete bounding boxes and thus will result in a very low detection accuracy. Through the observation, we think incomplete box detection is the main reason why previous methods perform badly on these object classes.

∗Corresponding Author

Figure 1: Images in first row show the incomplete box detections (red, small) predicted by state-of-art WSOD method and manually annotated complete bounding boxes (green, big). In general, our method is trying to learn a transfer function T that can predict strong boxes from weak boxes. θ is the model parameter.

To address the incomplete box detection problem, it is necessary to introduce auxiliary information about the true complete bounding boxes. Inspired by few-shot and semisupervised learning methods, we introduce a low-shot weakly supervised object detection task in this paper and propose a novel box correction method to address it. The definition of low-shot weakly supervised object detection task is: given a large dataset with image-level classification labels and only a

௦ ௪

Box Correction Function

small subset of it has bounding box annotations, the model is expected to detect the complete objects, rather than a part of them. The term low-shot here includes both semi-supervised and few-shot settings so that we use this term for brevity.

For this low-shot weakly supervised object detection task, we propose a novel box transfer method to correct the incomplete box predictions. Our method starts with generating weak boxes and strong boxes pairs. Here, weak boxes are those incomplete boxes predicted by common weakly supervised methods which are trained only by image-level annotations, while strong boxes are the ground truth complete bounding boxes annotated manually. In the low-shot task, all images have weak boxes while only a small subset (e.g. 10%) of images have both weak and strong boxes. For images with ground truth boxes, we use a greedy method to select possible weak-to-strong box pairs and design a box transfer function to learn a transformation from weak boxes to strong boxes. For images without ground truth boxes, we apply the learned box transfer function on them to generate pseudo complete bounding boxes. These generated pseudo boxes are trained together to maintain the model's diversity and avoid overfitting into the small subset.

We explore our method in two different settings. 1) Semisupervised learning: Only a small proportion of images have box annotations, irrespective of these images' class labels. 2) Few-shot learning: Only a few shots of images have box annotations for each class while different classes have the same number of images. We compare our method with most available methods under these two settings on PASCAL VOC dataset. Our method achieves state-of-art accuracy in most experiments.

2 RelatedWork

2.1 Weakly Supervised Object Detection.

WSOD has attracted much attention in the last decade. In recent years, due to the breakthrough brought by deep learning, a large amount of deep models are proposed for WSOD. Most of these models follow a similar pipeline: box proposal initialization, proposal classification, proposal refinement and retraining. We classify these models into three categories according to their mainly contributed pipeline steps.

For proposal initialization, Selective Search [Uijlings et al., 2013] and Edge Boxes [Zitnick and Doll´ar, 2014] are the most commonly used proposal methods. Zhu et al. [2017] propose to train an integrated weakly supervised object proposal network, based on the class activation maps. Different from [Zhu et al., 2017], Tang et al. [2018] propose to use a multi-layer fused attention map to refine the initial proposals.

For proposal classification, Bilen and Vedaldi [2016] employ a two-branch network, called WSDDN, to simultaneously perform region selection and classification. However, WSDDN only uses an image-level classification loss which makes it tend to recognize the most discriminative parts. To address the problem, [Diba et al., 2017] and [Lai and Gong, 2017] introduce salient segmentation attention maps to regularize the training. The introduced segmentation regularization is helpful but not enough to address the huge gap between those discriminative parts and the complete bounding boxes. Our method explicitly formulates the transfer function to correct the proposals instead of adding regularization. This makes our method more effective than previous models.

For proposal refinement and retraining, most of these methods are heuristic. Tang et al. [2017] employ WSDDN as the basic network and refine the prediction with several Online Instance Classifier Refinement (OICR) branches. Ge et al. [2018] propose a multi-task ensemble framework to refine the weakly detection results by some other tasks. After proposal refinement, the refined proposals are commonly used to train a strongly supervised methods. Zhang et al. [2018] heuristically employ an easy to hard training strategy to discover reliable proposals. Most of these refinement and retraining strategies heavily rely on the initialization. If the initial model is trapped in false incomplete detection, these strategies can not efficiently correct it.

2.2 Few Shot Object Detection

Dong et al. [2018] propose to generate trustworthy training samples for few-shot object detection task. They iteratively train model and select high-confidence samples for retraining. Compared with them, our proposed task has more annotations because we have image-level labels for all images. Through experiments, the image-level labels are essential for accuracy improvement. We argue that our task is reasonable since attaining image-level labels is much easier than bounding boxes.

2.3 Semi Supervised Object Detection

Yan et al. [2017] have designed an EM model for semi supervised object detection task. They combine the ground truth boxes with image-level labels to iteratively retrain the EM model. The biggest difference between their model and ours is that we explicitly separate the weak boxes and strong boxes while Yan et al. just put them together. As we shown above, these two boxes may have large difference in size and putting them together may be contradictory in training. Wang et al. [2018] design a similar model with [Yan et al., 2017] which employs Faster RCNN as a base network. Experiments confirm that our method outperforms both of Yan and Wang's model in various semi supervised settings.

2.4 Partially Supervised Learning

Hu et al. [2018] propose a new partially supervised training paradigm that enables training instance segmentation models on a large data set which has box annotations, but only a small fraction of which have mask annotations. The proposed paradigm is quite similar to our task. The major difference is that we have a large data set with only image-level annotations while they have box annotations. Since the state-of-arts WSOD accuracy is still unsatisfactory while the weakly semantic segmentation methods [Khoreva et al., 2017] are approaching their supervised counterparts, we suppose the task from image-level annotation to box is harder than the task from box to segmentation mask.

3 Method

Let I be all the images having image-level annotations. We split I into two parts, I = A ∪ B, where images in A have both image-level and box annotations while images in B are only image-level annotated. Note that under the low-shot settings, A is a small part of I (i.e. 10% ) and A ∩ B = ∅.

Our method starts from obtaining weak and strong boxes. We pretrain a weakly supervised object detection model and use it to generate pseudo bounding boxes for all images in I. These generated pseudo bounding boxes are denoted as their weak boxes, bw. For images in A, their manually labeled bounding boxes are denoted as strong boxes, bs. Each box is defined by a four-tuple (x, y, h, w) that specifies its center coordinate (x, y) and its height and width (h, w). Weak boxes are (bxw , byw , bww , bhw ) and strong boxes are (bxs , bys , bws , bhs ).

In general, our method is trying to learn a transfer function T that can predict strong boxes from weak boxes,

Before learning the transfer function between weak and strong boxes, we need to generate one-to-one, weak-to-strong box pairs. Only images in A can be used to generate the oneto-one pairs. We design a greedy algorithm to accomplish this. Given an image with its box bw and bs, we firstly calculate the overlaps between bw and bs. Then, each weak box is assigned a corresponding strong box which owns the largest overlap with the same label as bw. If the largest overlap is smaller than a threshold, this pair will be removed.

Having generated the weak boxes for B and weak-to-strong boxes for A, we'll utilize both images in A and B to train a box correction network for the low-shot weakly supervised object detection task. The overall network structure is shown in Figure 2. Details will be introduced respectively in next subsections.

3.1 Box Correction

Given the weak and strong box pairs for A, instead of directly predicting strong boxes bs, we formulate the transfer function to predict the offset between bw and bs. The offset targets are defined as t = (tx , ty , tw , th ), in which

We can easily attain bs by t and bw.

This formulation is similar to the regression in FastRCNN [Girshick, 2015] while we employ it to transfer different boxes instead of refining location. For clarity, we denote this offset prediction function as F and our general transfer function T is replaced by F ,

The offset prediction function F can be implemented as a fully connected neural network. In training, we apply a smoothL1 loss function to learn F,

in which v is the coordinates and height/width offset predicted by F .

Notice that the box-annotated data set A is a very small part of I. If we only train the transfer function T with the weakto-strong box pairs, the number of training set may be too small such that the model may easily overfit into these limited samples. In order to enrich the training samples, we integrate the transfer function T together with roi proposal classification. In Fast-RCNN, some roi proposals are pre-calculated by Selective Search [Uijlings et al., 2013]. These proposals are utilized to jointly learn object classification and bounding box regression. The original loss function of Fast-RCNN is

where p is the roi class predictions and t is the predicted offset between rois and targets. u is the class label and v is the target offset. Only rois with foreground label will contribute to the regression loss Lreg.

We adapt this loss function to introduce the T function as

Here, uw is the class label calculated from weak boxes bw, and vs is the offset target between rois and strong boxes bs. Only rois with foreground label in uw will contribute to the regression loss LT .

Comparing with the original loss function, the regression part in the adapted function is no longer used for refining location. In our loss function, the classification loss attempt to select rois who have a large overlap with weak boxes bw while the regression loss is used to predict the offset between these selected rois and paired strong boxes bs. This way ensures the transfer function T can be trained by all possible proposals, thus enabling a better transfer between weak and strong boxes.

3.2 Balanced Training

For images in A, the loss function is shown as equation 6. For images in B, since they only have weak boxes, as a result, we can only train the classification part of loss function

Note that the size of A is much smaller than B in our proposed low-shot tasks. In order to fully exploit all the images in B, we use the trained box correction function to generate pseudo corrected boxes for images in B and use all of the transferred boxes in A ∪ B to learn a new refinement branch of proposal classification and regression.

For each training iteration, one image za from A and one image zb from B are trained together to make balanced training samples. za and zb are firstly fed into a box correction branch with the following loss function,

The outputs of the first box correction branch are transferred boxes, denoted as b′a w and b′b w . These boxes are in turn used to train a second refinement branch. We suppose the second branch is functionally the same as Fast-RCNN whose regression loss is used to refine location so that a classification loss by bas is utilized to jointly train this branch as shown in Figure 2. The entire loss of the second branch is

where uas is the class label calculated from strong boxes for za . In Figure 2, we replace u, v with bs or bw to show it more clearly that which boxes are used in particular loss.

3.3 Multi-Stage Correction

During experiments, we find that the learned transfer function T is able to transfer the weak boxes towards the strong ones. However, in some cases, if the strong boxes are much bigger than the weak boxes, T can only enlarge the boxes a little but the enlarged ones are still much smaller than the strong boxes. We think the result is caused by the smoothL1 loss function,

If the strong boxes are much bigger than the weak boxes, |x| may be very large while training. Though smoothL1 has the same gradient when |x| ≥ 1, it can not effectively give enough loss penalty on these samples to cover the huge gap between the strong and weak boxes.

To address this problem, we introduce a simple but effective technique, the multi-stage correction. The intuition is straightforward. If a single transfer function can only make a limited step from the weak boxes towards the strong boxes, why not continuously make several steps? In the proposed multi-stage correction, a later stage uses the transferred boxes produced by its previous stage as the weak boxes and learns a new transfer function T ′ for current stage. Only the last stage has the classification loss by the strong boxes which is Lcls(p′ , bs) in Figure 2 because we treat it as a correct box detection branch rather than a weak-to-strong transfer branch.

3.4 Image-level Regularization

In the multi-stage model, our method actually follows an iterative training strategy, training a function to transfer boxes and then using the transferred boxes for retraining. One concern over this strategy is that it may amplify the noise from the first stage. To avoid this dilemma, we introduce an imagelevel regularization function in the final stage since all the images have image-level annotations.

For an image, let pic be the ith roi prediction for class c. We use the maximum pic across rois to represent the image-level prediction for class c, that is qc = maxi(pic). The imagelevel regularization is defined as the binary cross entropy with qc and the image-level label yc,

4 Experiments

4.1 Experiment Settings

We evaluate our method on the PASCAL VOC 2007 data set which contains 5011 training images and 4952 images for test. The evaluation metric is mean Average Precision (mAP) which is commonly used in object detection. In experiments, we explore our method in two different settings, semisupervised and few-shot. For semi-supervised settings, only a limited number of images have box annotations, irrespective of these images' labels. For few-shot settings, different labels have a same number of images with box annotations.

4.2 Implementation Details

Network backbone. We mainly use the VGG16 as base network for our experiments because most of the previous WSOD methods are structured on it. For fair comparison with [Yan et al., 2017], we also report results based on AlexNet. In addition, we give the results on ResNet for reference.

Pretraining. We use OICR [Tang et al., 2017] to generate the weak boxes. In experiments, we find that only using ImageNet pre-trained weight as initialization performs poorly for small networks, e.g. AlexNet, or with very small number of box annotations, e.g. 4 shots. We suppose this is caused by the limited generalization ability in these experiments. Instead of using ImageNet pre-trained weight, we employ a Fast-RCNN which is trained by weak boxes to initialize our model in these experiment.

Training and inference. For fair comparison with the prior arts, we use Edge Boxes as our roi proposal method. SGD is used to optimize the models. In experiments, models are finetuned with 60 epochs. The learning rate is 0.001 in the first 40 epochs and will be reduced to 0.0001 in the last 20 epochs. All other hyper-parameters follow those in Fast-RCNN. We resize the image minimum side into {400, 600, 750} as data augmentation. To jointly train the model with images from A and B, we randomly crop the input images by a fixed 600 × 600 window. In most cases, two-stage box correction branchs are used for multi-stage correction.

Low shot selection. One concern over the experiments is the sensitivity of our method to different selection of box annotated images. To figure out whether different selection of box annotated images can greatly influence the results, we have conducted several experiments and find that for experiments with more than 10% of all or 10 shots box annotations, different selections have very little effect on final accuracy. For experiment with very small number of box annotations, an average result is reported instead.

Reproducibility. We implement our method on Pytorch and Code is available at https://github.com/ptx9363/BCNet.

4.3 Comparison with State-of-the-arts

We evaluate our model on PASCAL VOC 2007 test benchmark and compare it with most available state-of-art methods under semi-supervised or few-shot settings. The results are shown in Table 1.

For methods with image labels and semi-supervised boxes, our method outperforms all of previous methods under the equal semi-supervised settings. Both [Yan et al., 2017] and [Wang et al., 2018] treat the weak boxes and the strong boxes equally while our method explicitly separate them. The comparison results confirm the necessity and effectiveness of this separation technique.

For methods with image labels and few-shot boxes, our method also performs better than these methods in equal settings. Dong et al. [Dong et al., 2018] have proposed to use image-level labels as an object filter in their paper. Compared with them, we have fully utilized the images in B and used balanced box transfer to get a better prediction.

By comparing our BCNet with common WSOD methods, we find that introducing the box correction transfer network can bring huge improvement on these methods. We would like to highlight that even if only 4 images with box annotations are provided, our method could greatly improve the prediction accuracy of OICR (our base WSOD model) for some classes like person (13.6% to 54.3%), cat (34.4% to 62.7%), dog (31.3% to 49.6%). These huge improvements show that our BCNet is robustly effective for helping some weakly supervised object detection methods which may have serious incomplete detection problem.

By comparing our BCNet with previous few-shot methods, we find that the image-level labels are essential for accuracy improvement (52.0% vs 41.7%). Since attaining image-level labels are much easier than box annotations, e.g. from search engine, we think our model is more practical in real world applications.

4.4 Ablation Study

Effectiveness of components. Firstly we conducted experiments to understand the different contribution of submodules. Tabel 2 shows the ablation results. We can see that the balanced training and box transfer are essential for our method. Without balanced training, we treat images equally from A and B while the size of A is much smaller than B so that the model can not fit the transfer function T well enough. Without box transfer, we directly finetune the features from WSDO with few shot annotated boxes and the results confirm that transferring boxes is more effective than transferring features.

Amounts of box annotations. We also conducted experiments by varying box annotations proportions in semisupervised task and number of shots in few-shot task. The results are summarized in Figure 3 and Table 3. As illustrated in Figure 3, our method robustly performs better than the state-of-art method [Yan et al., 2017]. When 40% boxes are annotated, our method achieves 56.0% mAP which is only 1.1 lower than fully supervised upper bound. We compare our method with [Wang et al., 2018] by varying number of annotation shots in Table 3. It's important to note that Wang employs a mixed model which contains several different base networks while our method is structured on a single base network. Therefore, their model owns a much higher upper bound (74.8%) than ours (68.9%) as shown in the table and we think it explains why their model performs better in 20-shot experiments. Even if only be structured on single ResNet101, our method achieves better results in 4-shot and 10-shot experiments. It confirms the effectiveness of our method on few-shot tasks.

5 Conclusion and FutureWork

In this paper, we are trying to solve the incomplete detection problem that exists in most previous weakly supervised object detection methods. We introduce a low-shot weakly supervised object detection task and propose a novel Box Correction Network (BCNet) to address it. We explicitly separate the incomplete boxes and complete boxes and our BCNet attempts to learn a transfer function to correct those incomplete boxes into complete ones. Experiment results show that BCNet is more effective than previous models.

Exploring more forms of transfer function will be our future works. In addition, how to learn a better transfer function by using advanced semi-supervised method or few shot learning method (e.g. vat [Miyato et al., 2018] or maml [Finn et al., 2017]) is another problem that we are interested in.

Acknowledgements

This work was supported by the NSFC under Grants 61772301, 61672307 and 61571269.