Deep-dense Conditional Random Fields for Object Co-segmentation

1 Introduction

We consider the task of object co-segmentation. The goal is to segment common objects in a set of images. Specifically, given a set of images, we try to answer two questions: which objects occur in different images? and where are they in each image? Object co-segmentation has multiple potential applications in AI agents, e.g., we input a query word to any image search engine and it can help summarise a large amount of returned images by picking out common objects inside them. Compared to single foreground segmentation, object co-segmentation can make use of shared information across images to assist final segmentation, which is also the most crucial step for co-segmentation.

Object co-segmentation has been widely researched in recent years [Vicente et al., 2011; Rubinstein et al., 2013; Wang et al., 2013; Zhang et al., 2015; Lee et al., 2015; Quan et al., 2016; Wu et al., 2016]. These methods are mainly different in how to include shared information into image segmentation. For example, [Rubinstein et al., 2013]

∗This work is partially supported by the Natural Science Foundation of China under Grant No. 61672273, No. 61272218 and No. 61321491, and the Science Foundation for Distinguished Young Scholars of Jiangsu under Grant No. BK20160021. Correspondence should be addressed to Tong Lu.

use SIFT Flow to match pixels across images, [Vicente et al., 2011] match regions, [Zhang et al., 2015] compare object proposals, etc. However, sharing information remains to be a challenge for several reasons. On one hand, low-level matching is noisy and hard to include high-level similarity information. On the other hand, robustly measuring similarity between regions is difficult due to possible illumination changes, viewpoint variations and so on. Another challenge is the diversity of image sets. Other uncommon objects may also appear somewhere. Co-segmentation system must robustly address different scenarios.

In order to overcome these challenges, inspired by [Hayder et al., 2016], we propose an object co-segmentation system that learns to share common information for segmentation. Specifically, given a set of images, we first generate multiple object proposals for each of them and every proposal is associated with a label to represent if it is a common object instance or not. All proposals across images are linked in a deep-dense conditional random field (DDCRF), where unary potentials measure how likely an object proposal belongs to a potential cooccurring object and pairwise terms model the compactness between neighbour proposals under different label combinations. The DDCRF is an extension of dense CRF [Kr¨ahenb¨uhl and Koltun, 2012] to deep neural network. In a word, its unary and pairwise potential are modelled using deep neural network. By this way, the probability of each proposal to be common corresponds to the marginal distribution of each node and can be easily calculated by performing standard inference on CRF. In addition, we augment DDCRF with a branch to estimate segmentation masks of proposals.

With the common probabilities and segmentation masks of all proposals, we develop a cooccurrence map to summerize them for the following segmentation. The coocurrence map measures how likely an image pixel belongs to potential common objects and is calculated by accumulating evidences from all proposals. After getting cooccurrence maps of all images, object co-segmentation is naturally converted into image foreground segmentation and can be done independently for each image.

In summary, our contributions are twofold: 1) a cooccurrence map is proposed to convert object co-segmentation into independent image segmentation, on which tons of systems exist. Every image has a cooccurrence map to encode its shared information across images, which can be used as extra information for foreground segmentation; 2) a deepdense conditional random field is introduced to automatically discover common information between object proposals and then calculate cooccurrence maps. The entire network is learned in an end-to-end way. To our best knowledge, our system is the first to introduce deep neural network to object co-segmentation. We evaluate our system on two challenging datasets and achieve competitive performance.

2 RelatedWork

Image co-segmentation aims to segment multiple images in an unified way. The topic has attracted a lot of attentions in recent years because it has proved capable of improving unsupervised foreground segmentation in case that no priors exist for inputing images. Following [Yuan et al., 2014], we roughly categorize co-segmentation into two branches based on how many common classes are considered: object cosegmentation and multi-class image co-segmentation.

Object Co-segmentation. Object co-segmentation aims to segment common objects in multiple images. Most object co-segmentation systems assume that only one single object cooccurs in an input image set. To this end, the popular attempt is to construct saliency maps [Rubinstein et al., 2013; Jerripothula et al., 2016], objectness maps [Vicente et al., 2011] and other feature maps [Mukherjee et al., 2011; Vicente et al., 2011; Wang et al., 2013; Dai et al., 2013] to distinguish foreground objects. Then segmentation masks are propagated across images based on the assumption that matched regions should have similar masks. Images are connected at pixel level [Rubinstein et al., 2013], region level [Rubio et al., 2012; Faktor and Irani, 2013; Lee et al., 2015], object proposal level [Zhang et al., 2015] and even featurespace level [Wang et al., 2013] . Then the problem can be naturally converted into a two-label optimization problem and can be easily solved. Although much better performance is obtained compared to baselines, there still exist a lot of challenges. The key limitation is that manually designed features can not be robust to complex scenarios. [Wang et al., 2013] propose function mapping to link regions but still need a carefully designed features. Recently, [Zhang et al., 2015] first propose to combine Restricted Boltzmann Machines [Bengio, 2009] and Convolutional Neural Network to embed object proposals into a high-dimensional space so that regions can be robustly matched in the space. However, their cosaliency model is still pre-defined instead of automatic learning. [Quan et al., 2016] adopt a manifold ranking algorithm to optimize their constructed superpixel graph. Their method aims to reduce the common strict assumption that cooccurring regions are objects. Similarly, low-level linking is not robust to practical internet images.

Multi-class Object Co-segmentation. Multi-class segmentation systems [Joulin et al., 2012; Kim and Xing, 2012; Ma and Latecki, 2013; Tsai et al., 2016] remove the assumption that one single common object appear in images and aims to segment all common classes. For example, [Kim and Xing, 2012] propose multi-object co-segmentation to segment multiple objects from background using a combinatorial auction optimization framework. [Wu et al., 2016] add human locations as extra priors to segment common foreground objects around a person. [Joulin et al., 2012; Ma and Latecki, 2013; Tsai et al., 2016] even extend it to multi-class image cosegmentation without differentiating objects and background. They iteratively perform discriminative clustering over oversegmented image regions and then multiple class are distinguished from each other. [Yuan et al., 2014] introduce a topic-level random-walk framework to give high votes to cooccurring regions. However, the majority of them still construct across-image relations using predefined features.

3 Our Model

We address the problem of object co-segmentation. To simplify the task and make "common" more clear, we assume that there is one single common object in a given image set. That means other objects will not appear or appear much less. Our system first generates object proposals for every image and our goal is to make use of them to generate the cooccurrence map of each image. Deep-dense conditional random field is adopted to encode shared information across images.

3.1 Deep-dense Conditional RandomField

Given a set of n images {I1, I2, . . . , In}, we first generate a pool of object proposals O = {o11 , o12 , . . . , o1g1 , o21 , . . . , ongn } for all images, where g represents the number of proposals in each image. To simplify the denotation, we omit the image superior n and rewrites O = {o1, o2, . . . , oG}. G is the total number of object proposals. Each object proposal oi has two variables ci ∈ {0, 1} and mi. ci represents if oi is a common object. ci = 0 means that the proposal belongs to background or an uncommon object. The segmentation mask mi is to show pixel locations of potential objects in oi. A deep-dense conditional random field is adopted to model the joint distribution P(C,M|O) of C = {c1, c2, . . . , cG} and M= {m1,m2, . . . ,mG} given O. Formally,

where φ(·) and ψ(·) represent the unary term and pairwise potential, respectively.

Unary Potential. φ(ci,mi|oi) models how likely the object proposal oi is a common object and has the segmentation mask mi. Following the Bayes rule, φ(ci,mi|oi) can be further decomposed into

However, it is impossible to measure the common characteristic by one single object proposal. Instead, we use objectness vi as its proxy, which measures the likelihood of oi as an object. Finally, the final unary potential φ(ci,mi|oi) is further simplified into φ(vi|oi)+φ(mi|oi) because mi can be mainly determined by appearance.

Pairwise Potential. To simply formulation, we ignore the influence of masks in pairwise potential. This is reasonable because a segmentation mask is highly related to appearance [Pinheiro et al., 2015] and does not rely much on predictions from others. Assume that we have extracted feature fi for any oi, then the pairwise potential is reformulated as

where µ measures the compatibility between labels. We use the typical Potts model µ(ci, ci) = [[ci ̸= cj]]. κ(·, ·) is a Gaussion kernel

Deep Network to Represent Potentials. We use deep convolutional neural network to model unary potentials and extract high-level features of object proposals . VGG16 [Simonyan and Zisserman, 2014] is adopted as our backbone network but the final two fully connected (fc) layers are removed. We add three branches above the final pooling layer (pool5) to represent φ(ci), φ(mi) and fi respectively. The first two branches share fc layers (fc6 and fc7) at the beginning and then divide into two ways to encode φ(ci) and fi respectively with each following a fc layer. mi is estimated in another branch by following one convolution layer and two fully connected layers. In order to make outputs having fixed dimension, a v × v(v = 40) mask is used. In testing stage, the mask will be resized into its original size. In addition, we replace pool5 with ROI pooling [Ren et al., 2015] to share computations of object proposals in the same image. The detailed architecture is shown in Fig. 1.

3.2 Generate Cooccurrence Maps

For each image I, we use object proposals inside it to generate its cooccurrence map R. The element of R represents how confident its corresponding pixel appears in most of other images. In order to generate these maps, we first maximize P(C,M|O) and get MAP estimates for object proposals. Note that P(C,M|O) can be further decomposed after our simplification for both unary and pairwise potentials

Therefore, M can be inferred independently for each object proposal by only performing forward pass. In term of C, the mean field variational inference [Kr¨ahenb¨uhl and Koltun, 2013] is adopted to get MAP estimates. Specifically, P (C|O) is approximated by Q(C) = ∏i qi(ci). The marginal distribution qi(ci) of each oi can be calculated in an iterative way

The iteration proceeds until convergence or the iteration step exceeds a threshold. Although the update can be boosted by high-dimensional filtering techniques, we use the standard convolution operator because convolution is fast enough on current GPUs. By this way, we get qi(ci) for any object proposal to encode its probability to appear in other images.

With the marginal distributions and mask predictions of all object proposals inside an image I, it's cooccurrence map is calculated by max pooling over them

where (x, y) corresponds to the relative coordinate of (x, y) in oi.

Object Co-segmentation. Given cooccurrence maps of all images, object co-segmentation can be converted into simple foreground segmentation for separate images. We adopt dense CRF in [Kr¨ahenb¨uhl and Koltun, 2012] as the foreground segmentation method. Specifically, given an image, we use its cooccurrence map as the single unary term and a bilateral filter as pairwise potential. By this way, we obtain object segmentations for all images.

4 End-to-end Training

In this section, we will learn the parameters of our deep neural network to make estimated segmentation masks consistent with groundtruth, embedding features capable of comparing proposals with Euclidean metric and finally inferred objects common in training dataset.

4.1 Gradients Computation

Specifically, given the groundtruth labels (ˆ ci, ˆmi)of any oi, we define the loss function as the summation of two parts

where ˆM and ˆC correspond to groundtruth mask labels and common objects labels of all proposals, respectively.

In order to define L( ˆM,M) elegantly, we assume that groundtruth ˆmi ∈ {-1, 1}v×v with 1 corresponding to object and -1 background. Then L( ˆM,M) is defined as

where the multiplying factor means only segmentation masks of common objects are considered. ˆw(x, y) is the weight of the (x, y)-th pixel to balance foreground and background

Note that minimizing L( ˆM,M) will increase estimated scores for object pixels but reduce them for background. The gradients of all parameters can be calculated easily by backpropagation.

In terms of L( ˆC,C), we use the standard cross-entropy, namely, L( ˆC, C) = ∑i - log(qi(ˆ ci). However, the computation of gradients is not trivial due to the pairwise potential. We derive mean-field gradients with respect to network parameters ω as following

where f = (fT1 , fT1 , . . . , fTG)T and φ = (φT1 , φT1 , . . . , φTG)T . Note that ∂φT ∂ω and ∂f T ∂ω can be easily calculated by backpropagation. Now we derive ∂L( ˆC,C) ∂φ and ∂L( ˆC,C) ∂f .

We rewrite L( ˆC, C) as L(q) for brevity, where q is a concatenated vector (qT1 , qT1 , . . . , qTG)T of marginal probabilities. Similarly, let Φ = K ⊗ µ where K is a kernel matrix [kij]G∗Gwith kij = κ(fi, fj). Following [Kr¨ahenb¨uhl and Koltun, 2013], we can compute them iteratively

where h = {1, 2, . . . ,H} represents the iteration step and qhj corresponds to the marginal probabilities in the h step of inference. Simultaneously,

where A(h) is a block diagonal matrix with each block A(h) i = q(h) i q(h)T i - diag(qhi ). Therefore, both two meanfield gradients can be iteratively refined until convergence. Then backpropagation is performed to update parameters of all layers. The iteration is very fast on GPU because L(q) ∂f can be computed as convolution as shown in Equation 13.

4.2 Training Strategy

The entire network is fine-tuned on a pretrained model for ImageNet classification except new layers. We randomly initialize all new layers by drawing weights from a zero-mean Gaussian distribution with the standard deviation 0.01. In order to share computations among object proposals, we follow the same sampling strategy as [Ren et al., 2015], namely, 2 images are sampled per iteration and 64 object proposals per image. Therefore, positives and negatives account for half respectively in each mini-batch with 128 proposals totally.

In order to make optimization faster and converge to a good optima, a two-stage training is adopted to train the network. Firstly, we learn parameters without including pairwise term. In this case, the training reduces to classify proposals and regress masks similar as in [Pinheiro et al., 2015]. Then we finetune the network by adding pairwise term with customized mini-batches, where a proposal is considered as positive if its overlap with any groundtruth is larger than a threshold (we fixed it 0.7) and its object class is common. In contrast, negatives are composed of a mixture of proposals with maximal overlaps between [0.1, 0.5] and those belonging to other uncommon classes. Uncommon objects are enforced to have less than num+/6 examples so that positives are saliently common. Instead of selecting background regions, we use proposals with the limited overlap range as negatives to make training robust and fast. It has the same effect as hard mining. The two-stage training strategy is necessary because training deep-dense CRF needs stable unary estimation. Direct optimization will mostly cause divergence in our early experiments.

5 Experiments and Discussion

We train DDCRF network on PASCAL VOC 2012 dataset [Everingham et al., ] and test the performance on the widely used benchmark iCoseg [Batra et al., 2010] and one more challenging Internet dataset [Rubinstein et al., 2013].

Datasets. We learn parameters on PASCAL VOC 2012 dataset, in which 11540 images have groundtruth detection boxes and 2913 images have segmentation masks. The iCoseg dataset is an object co-segmentation benchmark widely used to evaluate co-segmentation systems, which contains a total of 643 images for 38 object classes. All images are segmented well at pixel level. In contrast, Internet dataset is proposed to evaluate co-segmentation methods on natural images. It contains 3 classes (airplane, car and house) with each class having thousands of downloaded internet images. Compared to iCoseg, images in Internet are mixed with other uncommon objects and multiple background images.

Training Details. We adopt selective search [van de Sande et al., 2011] to generate multiple object proposals in training images. Then mini-batches are sampled from the combination of ground truth bounding boxes and these potentials in the same way as Sec. 4.2. Since there are relatively less images for segmentation masks, joint training will cause severe overfitting. Therefore, we first finetune the network on images with annotated boxes without considering segmentation masks. The training follows two-stage learning strategy as shown in Sec. 4.2 with the initial learning rate 0.001. Stochastic Gradient Descent (SGD) is used as optimizer and the learning rate decreases to 0.0001 after 80K iterations. After finishing the first two-stage training, we add the segmentation branch. In order to prevent overfitting, we fix parameters of the first four convolutional layers and give different learning rates for remaining layers. Specifically, the learning rates of layers in segmentation branch start with 0.001 and others keep training with 0.0001. Additionally, we add dropout layers for all three branches. In the segmentation branch, fc8 is trained with the dropout rate 0.5. Two dropout layers with dropout rate 0.8 are followed after fc6 and fc7, respectively.

Evaluation Metrics. Two standard metrics are used: Precision P and Jaccard index J. P represents the percentage of correctly labeled pixels including both foreground and background, while J focuses on the overlap (Intersection of Union) between estimated object segmentation and foreground.

5.1 Evaluation on ICoseg

iCoseg consists of 38 classes and we evaluate for each class independently. Firstly, selective search [van de Sande et al., 2011] is used to generate a lot of object proposals. Every proposal is forwarded into our network to get its feature f and unary score φ. All proposals are then connected into a dense CRF. Iterative inference is performed to get their marginal probabilities and then further cooccurrence maps. Finally, segmentation masks are obtained separately for each image. We show some segmented examples in Fig. 2. We can see our systems generate promising segmentations for listed classes, which indicates our system is effective to co-segment common objects in this dataset.

Comparison with Existing Systems. Qualitatively, we report both P and J metric in Table. 1. Several comparisons with recent object co-segmentation systems are also included. Besides, we report the performance on sub-iCoseg dataset (Table. 2) to make more comparisons because they only report their performance in this split. sub-iCoseg has 122 images of 16 classes selected from iCoseg. We get the state-of-theart performance on both datasets with respect to both metrics . This indicates our deep system can encode shared information effectively and transfer them into segmentation successfully. We analyse that main reasons are twofold: 1) the deep network can learn robust similarity metric compared to other existing systems, which will help co-segmentation a lot with dense CRF; 2) segmentation masks are learned to rectify object proposals so that each object proposal has accurate segmentation mask compared to regard original segments in proposals as final masks.

Ablation Study. We also did ablation study to show the contribution of each component in our full system. Firstly, we remove pairwise potential in DDCRF as our baseline. The baseline reduces to simple image segmentation problem without sharing information among images. In w/o seg, we do not estimate segmentation masks but use accompanying masks of object proposals to calculate cooccurrence maps instead. Dropping each of these components will hurt the final performance. In particular, sharing information among images can help final segmentation of each image, which is consistent with the observations of prior works.

In summary, all scaffoldings of our system are necessary and our full system gives promising performance on iCoseg.

5.2 Evaluation on Internet

Internet images are collected by downloading returned images after querying by key words of three classes, i.e., car, horse and airplane. We use its widely used subset for evaluation, in which each class has 100 images. Similarly, for each class, we generate their object proposals using selective search, then all proposals are connected using dense CRF based on their unary potentials, segmentation masks and embedding features. In Fig. 3, we give some results produced by our full system. Despite large intra-class variations, our system can obtain good segmentations indicating the learned neural network can help object co-segmentation in images.

Comparison with Existing Systems. Similarly, we compare with several existing methods on both metrics and results are reported in Table. 3. Our full system outperforms all other methods on all three methods, which indicates our method can effectively discover shared information by robustly comparing object proposals. It is worthy noting that the performance increase by a larger margin on Internet compared to iCoseg. The main reason is that each of the classes in Internet has more images and intra-class variations are relatively bigger than iCoseg. Furthermore, there exist several noisy examples including background images and other uncommon objects. Therefore, existing low level systems will be influenced a lot by linking ambiguous regions. Our system has learned robust similarity metric between object proposals and thus robust to complex scenarios.

Ablation Study. In order to evaluate the impact of each component on Internet dataset, we also report the performance of baseline and w/o seg in Table. 3. The performance is consistent with those on iCoseg, indicating every component is crucial to our system.

6 Discussion and Limitations

Although the proposed method gets promising results on both datasets, there still are some inherent disadvantages. Firstly, our system relies on selective search to generate proposals. If an image is complex enough and no object proposals cover object instances, our system will fail. In this case, the region proposal network can be used as an alternative in future to reduce failures. Secondly, the estimation of segmentation masks is still not accurate enough for complex objects with large occlusion or slim structures. The main reason is that training images with segmentation groundtruth in PASCAL is not enough to generalize very well. Furthermore, the image segmentation method dense-CRF that we used also performs badly for these objects, which will magnify negative effects. Our future work lies on training DDCRF on larger datasets to further improve segmentation quality.

7 Conclusion

In this paper, we propose deep-dense conditional random fields for object co-segmentation in images. A cooccurrence map is introduced to summarise shared information after performing inference for constructed DDCRF. Based on cooccurrence maps, object co-segmentation is converted into single foreground segmentation, where several excellent systems exist. Our system is trained in an end-to-end way with a twostage training strategy. We evaluate DDCRF on the benchmark iCoseg and a challenging Internet dataset. Competitive performance on both datasets indicates that our method is effective for object co-segmentation.