Weakly Supervised RBM for Semantic Segmentation

1 Introduction

Semantic image segmentation is a fundamentally challenging problem, aiming at assigning semantic labels to image regions [Xie et al., 2014]. Compared with traditional image segmentation, it provides higher-level understanding about image contents. While compared with typical image classification, it provides more fine-grained semantic understanding of images. Therefore, image semantic segmentation provides a virtual solution to bridge the semantic gap, and becomes one of the core problems in computer vision [Shotton et al., 2009][Zhang et al., 2012].

In the past years, semantic image segmentation has attracted a lot of attention, and significant progress has been achieved [Shotton et al., 2006; 2008; Liu et al., 2009a; Vezhnevets et al., 2012; Farabet et al., 2013]. Although these methods have shown promising results, they rely on a training set of images with pixel-level labels. However, the high cost on the ground truth acquisition restricts the wide usage of such work. Fortunately, with the rapid spread of online photo sharing websites (e.g., Flicker), large numbers of images with image-level labels become available. These image-level labels can be further exploited to make semantic segmentation [Liu et al., 2009b; Vezhnevets et al., 2011; Xie et al., 2014]. In contrast to fully supervised setting with pixel-level labels, it is more challenging to make weakly supervised semantic segmentation with image-level labels.

Recently, a few methods have been proposed to address the weakly supervised segmentation problem [Liu et al., 2009b; 2013; Zhang et al., 2013; Xie et al., 2014]. In general, the core task of weakly supervised semantic segmentation is to learn the mapping between image label and low-level feature of local regions. [Liu et al., 2009b] attempted to capture the cooccurrence by a bi-layer sparse coding model. [Zhang et al., 2013] developed a way to learn the mapping by evaluating the classifier, in which a good classifier will have good reconstruction basis for positive samples and large reconstruction error for the negative samples. Meanwhile, Graph propagation based methods are proposed to restrict visually similar regions to have similar labels [Liu et al., 2012; 2013; Xie et al., 2014]. Generally, the existing methods initialize the label of superpixels with image-level labels explicitly and refine the model consequently to get the final label for each superpixel. Such an explicit importation of image-level labels as supervision may be crude a little, since the limited discriminative ability of superpixels tends to make the performance of label refinement unsatisfied to some extent. Thus, how to import the image-level labels as weak supervision is a challenging but valuable problem for semantic segmentation.

In this paper, we propose a Weakly supervised Restricted Boltzmann Machines (WRBM) to deal with the problem of semantic segmentation by exploring the image-level labels as shown in Figure 1. The images are first over segmented into superpixels, and the extracted feature of each superpixel is set as the input to the visible nodes of WRBM. The hidden nodes of WRBM are divided into several blocks, where each block corresponds to a specific label. Under such setting, the superpixel label can be decided by the hidden-node block with the maximum response. The proposed WRBM is an extended version of the standard RBM by introducing two additional terms in its objective function. The first term is a non-image-level label suppression one to implicitly import the image-level labels as weak supervision. Such implicit importation works on the intuition that the label of a superpixel is impossible to be the ones outside its parent image-level labels. Thus, we employ the term to suppress the response of blocks corresponding to those impossible labels. The second term is a semantic graph propagation one to make sure that visually similar superpixels sharing common image-level label have similar hidden response. Besides, we modify the model to deal with the problems of diverse backgrounds and label imbalance in the training dataset. For the former, we add multiple background blocks to the hidden layer, and assume each image corresponds to one background block. To deal with the label imbalance, we design the size of the block corresponding to each label to be an inversely correlated number with the label frequency. That is, we expect to amplify the response of an infrequent label by designing a larger block size. Finally, extensive experiments on two real-world datasets, i.e., PASCAL [Everingham et al., 2010] and LabelMe [Russell et al., 2008], demonstrate the effectiveness of our approach compared with some state-of-the-art methods. Generally, the main contributions are summarized as follows.

• We propose a RBM-based learning framework for the task of semantic segmentation. The correspondence between labels and hidden responses of WRBM gives a direct solution to the prediction of superpixel label.
• The non-image-level label suppression and the semantic graph propagation are employed together to make full use of the image-level labels and alleviate the effect of

the noisy labels.

• The changeable block size and block number of the hidden layer are designed to handle the problems of label imbalance and diverse backgrounds of training data.

2 RelatedWork

Semantic segmentation has attracted wide interests due to its importance in bridging high-level concepts to low-level features of local regions. Most semantic segmentation approaches suppose that a training dataset with pixel-level labels is given [Vezhnevets et al., 2012] [Shotton et al., 2006] [Farabet et al., 2013] [Liu et al., 2009a] [Farabet et al., 2013]. A typical way to model the problem with pixel-level label is based on Conditional Random Field (CRF) [Shotton et al., 2006], The basic formulation is defined on pixel values with various potential functions, including shape, texture, color, location and edge cues. Lots of extensions are proposed to modify the CRF with high-order potentials [Ladicky et al., 2009], hierarchical features [Farabet et al., 2013], label cooccurrence [Ladicky et al., 2010]. Another direction is to develop non-parametric methods to transfer labels from training images to the query image [Liu et al., 2009a] [Liu et al., 2011]. [Myeong and Lee, 2013] further explores high-order semantic relation with label transfer. All of these approaches require pixel-level labels for training, which are expensive to obtain in practise.

Weakly-supervised methods have emerged and attracted more attention due to the weak requirement of supervision [Vezhnevets et al., 2011; 2012; Vezhnevets and Buhmann, 2010; Liu et al., 2013; 2012; Zhang et al., 2013; 2014;

Xie et al., 2014]. [Vezhnevets and Buhmann, 2010] cast the semantic segmentation task as a multiple instance learning problem. They adopted semantic texton forest as the basic framework and extended it for the MIL setting. [Vezhnevets et al., 2011] extended [Vezhnevets and Buhmann, 2010] with a multi-image model, in which smoothness between adjacent and similar superpixels is encouraged. [Vezhnevets et al., 2012] exploited multiple visual cues in this weakly supervised setting with a parametric family of structured models. Meanwhile, [Liu et al., 2009b] proposed a bi-layer sparse coding method, in which an image region is sparsely constructed with the regions of the same image-level label. What is more, [Liu et al., 2012; 2013] developed a weakly supervised graph propagation model by considering superpixel consistency and weak supervision information simultaneously. [Xie et al., 2014] further verified the importance of semantic graph construction in the graph propagation model.

Generally, the existing weakly supervised methods leverage the image-level labels explicitly, they initialize the label of superpixel with image-level labels and refine the model consequently to get the final label for each superpixel. Such methods are easily affected by the noise labels, which usually exist in real applications. Furthermore, different label concepts occur quite differently in the real case. Treating different labels equally without considering label imbalance will limit the performance of segmentation methods. Leveraging the weak image-level labels properly and addressing the problems above become the focus of this paper.

3 The Proposed Approach

We propose a weakly supervised RBM based method for semantic segmentation via non-image-level label suppression. RBM is an undirected graphical model, which consists of a visible layer and a hidden layer. It can model the data in a transformed subspace with the unsupervised setting. To address the semantic segmentation problem, units of the hidden layer are divided into multiple blocks, where each block corresponds to a specific label concept. Mapping associations between low-level features and hidden label blocks can be achieved via non-image-level label suppression and graph propagation.

Before the detailed discussion of each component, we first summarize some notations. Given a set of images τ, each image i is oversegmented into Ni superpixels [Achanta et al., 2012]. Feature is extracted on each superpixel with the bag-of-words model using SIFT [Lowe, 2004] descriptor and color feature. xij denotes the feature of the j-th superpixel in image i. In addition, the image-level label set for the i-th image is denoted as Si.

3.1 Non-image-level Label Suppression

Generally, the semantic segmentation problem is to learn the mapping from low-level features of local regions to high-level concepts. Unlike the fully supervised setting [Shotton et al., 2006] [Farabet et al., 2013], the problem under weakly supervised setting becomes very challenging because of the absence of pixel-level labels. For a specific image i, any label among the image-level label set Si may be the truth for the superpixels. We can not learn a direct mapping from low-level features to high-level labels under the weakly supervised setting. However, the image-level label provides the important cue that there will be no mapping from the superpixels to the none-image-level image labels ( labels not in Si ). To incorporate such idea into the standard RBM framework, we import a regularization term to suppress blocks corresponding to the none-image-level labels.

Standard RBM has a single layer of hidden units that are not connected to each other and have undirected, symmetrical connections to a layer of visible units [Hinton and Salakhutdinov, 2006; Bengio, 2009]. A joint configuration of the visible units v and hidden units h has an energy function as follows,

where W is the weight matrix between visible units and hidden units, b and c are the offsets of visible units v and hidden units h. The joint probability distribution of all the units is defined as follows,

where Z is the partition function.

Given visible units v, the probability for the binary hidden unit hm to be 1 can be obtained as follows,

whereWm,: is them-th rowof thematrixW. Similarly, given hidden units h, the probability for the binary visible unit vn to be 1 is as follows,

where W:,n is the n-th column of the matrix W.

To import semantic information, the hidden layer units of RBM are divided into multiple blocks, and block Bk corresponds to the k-th label concept Lk. The response function for the units in block Bk is defined as follows,

where m is the index for the hidden unit in block Bk. For a specific image i with image-level label set Si, response of the blocks corresponding to the non-image-level labels should be small. We capture this property by importing non-image-level suppression term as follows,

As a result, mapping to the image-level labels will be encouraged for each superpixel with the suppression term.

Adaptive Block Size

The hidden units of RBM are divided into multiple blocks according to the label concept. A natural division method for the blocks is of the equal size. However, frequencies of different labels are usually highly imbalanced as shown in Figure 2. The background label 'sky' occurs in 85% images of the dataset, while the specific object 'moon' occurs in no more than 1% images. If the block is of the same size, images of high frequency labels will dominate the non-imagelevel suppression term of Equation (6). To deal with the label imbalance and emphasize the effect of low frequency labels, we increase the block size of the labels with low frequency directly. Empirically, the block size QBk is modeled with respect to the label frequency as follows,

where M is a hyper-parameter to control the scale of the block size, C is a smooth constant, δ is a free parameter to control the decay rate. fk is the label frequency calculated from the training dataset as follows, fk = nk ∑k′ nk′ , and nk is the number of images containing label k.

Diverse Backgrounds

To deal with the diverse backgrounds, a intuitive idea is to add a block in the hidden layer corresponding to the class "background". However, the added block will have maximum response to the most commonly occurred visually similar superpixels among the dataset instead of the background regions for each image. Therefore, multiple blocks are preferred to deal with the diverse backgrounds. The background for a specific image belongs to one possible background block. A random background label is added to each image. With nm multiple background blocks, the probability of ni different images to have the same background label is ( 1 nm )ni-1 , which is usually very small.

3.2 Semantic Graph Propagation

Label propagation plays an important role for weakly supervised semantic segmentation [Liu et al., 2012; Zhang et al., 2014; Xie et al., 2014]. Semantic graph propagation term is proposed to make sure that similar superpixels sharing common image-level label have similar hidden response. If two similar local regions from different images share common label, then it is natural to tag these regions with the common label. [Xie et al., 2014] validates the importance of affinity graph with kinds of construction methods. Some labels usually occur together, like "grass" and "sheep". It is difficult to label superpixels in such images. In order to embed more discriminative information into the affinity graph, we propose a affinity graph construction method with semantic exclusion property. For a specific image i with label set Si, a label Li in Si is denoted as Li ∈ Si, and its complementary set is denoted as Lcs i with respect to Si. For the superpixels in image i, we seek to find the K nearest neighbors for each label Li separately. The K-NN superpixels are selected in the images, which contain label Li, but do not contain the labels in Lcs i . Figure 3 illustrates such process intuitively. Finally, the KNN superpixels for a superpixel j in image i is decided with maximum semantic similarity,

where ALi ij,l = exp(- ‖xij-xl‖2 t ) is the similarity measure for the superpixel pair with label Li, and t is a free parameter to control the decay rate. The advantage of such semantic graph provides more discriminative information about the true label for each superpixel.

With the semantic affinity graph, the graph propagation term can be formulated as follows,

where xij is the extracted feature of the j-th superpixel in image i, and K(ij) denotes the index set of the K nearest neighbors. h(x) is the hidden response vector corresponding to the visible input x.

3.3 Objective Function of WRBM

By incorporating the non-image-level label suppression term and semantic graph propagation term into the basic RBM framework, we can get the final energy function as follows,

where α and β are the tradeoff parameters of the proposed two terms.

3.4 Semantic Segmentation withWRBM

Given that the proposed model WRBM is well learnt, the semantic segmentation process can be performed by finding the block with maximum response. Specifically, for any given image i with label set Si, The pixel-level label Tij of the superpixel xij can be assigned by maximizing the block response as follows,

where EBk is the response of block k given input feature xij , which is defined in Equation (5).

4 Optimization

For the basic problem of RBM, parameters can be estimated by minimizing the negative log-likelihood - ∑h log P(v, h) via Contrastive Divergence [Hinton, 2002]. Contrastive Divergence is an approximation of the log-likelihood gradient that has been found to be a successful update rule for training RBM [Bengio, 2009]. Optimization algorithm of the proposed WRBM can be achieved by modifying Contrastive Divergence directly, since the non-image-level suppression term and graph propagation term are both convex and differentiable. The Contrastive Divergence algorithm with Weak Supervision (CDWS) is shown in Algorithm 1, where ⊙ denotes element-wise product of two vectors.

Algorithm 1 CDWS

Input: Extracted feature xij, image-level label set Si, parameter weights α and β, learning rate ε, maximum epoch number Ech,

• similarity matrix Aij,l 1: Initialize: W, b, c 2: While ite < Ech do 3: P(h1 = 1|xij) = 1 1+exp(c+W xij ) 4: sample h1 ∈ {0, 1} fromP(h|v) 5: P(xij = 1|h) = 1 1+exp(b+WT h1) 6: sample x2ij ∈ 0, 1 fromP(x2ij|h2 ) 7: P(h2 = 1|x2ij) = 1 1+exp(c+W x2ij ) 8: calculate gradient of Equ. 6 w.r.t h as db 9: calculate gradient of Equ. 9 w.r.t h as ds 10: dc = (αdb +βds) ⊙P(h1 = 1|xij) ⊙ (1-P(h1 = 1|xij)) 11: Dw = dcxTij 12: Update parameters 13: W ← W + ε(h1 xTij - P (h2 = 1|x2ij )(x2ij )T - Dw) 14: b ← b+ ε(x1ij - x2ij) 15: c ← c + ε(h1 -P(h2 = 1|x2ij) -dc) 16: EndWhile

Output: W, b, c

5 Experiments and Results

5.1 Experimental Setup

We evaluate our algorithm on two real world datasets, PASCAL VOC 2007 dataset (PASCAL for short) [Everingham et al., 2010] and LabelMe dataset [Russell et al., 2008; Liu et al., 2009a]. Extensive comparisons are presented with several related work, including state-of-the-art methods [Xie et al., 2014], [Zhang et al., 2014], [Zhang et al., 2013], [Liu et al., 2012], [Liu et al., 2013], [Vezhnevets et al., 2012], [Vezhnevets et al., 2011], [Vezhnevets and Buhmann, 2010], [Liu et al., 2009b], [Verbeek and Triggs, 2007]. For fair comparison, we directly cite released results of comparison methods. Methods are typically compared using the average per-class accuracy. For a given class, the accuracy is calculated by the percentage of correctly classified pixels.

To further validate the robustness of the proposed method, experiments are performed under the setting of different levels of label noise and different numbers of noise images.

5.2 Results on PASCAL Dataset

The PASCAL VOC 2007 dataset was used for the PASCAL visual object category segmentation contest with 20 object classes [Everingham et al., 2010]. We conduct experiments on the segmentation set with the "train-val" split including 422 training-validation images and 210 test images. For the segmentation dataset, only obvious object labels are provided for each image. As a result, the image-level labels are incomplete to describe the image regions, and large numbers of superpixels are deemed as "background", which leads to diverse backgrounds. It is very challenging to make semantic segmentation on this dataset due to high intra-class variations and diverse backgrounds.

To deal with large numbers of background regions, objectness of each image is calculated to help build semantic graph of object regions [Alexe et al., 2010]. Grabcut [Rother et al., 2004] is adopted to refine the boundary of object regions.

The experimental results of our method compared with other related work are presented in Table 1. The last column shows the average accuracy, and our approach with adaptive block size achieves the best performance with 14.3% relative improvement to state of the art [Xie et al., 2014]. For individual concepts, our approach with adaptive block size and equal block size outperforms the comparison methods on 5 classes and 6 classes respectively. Compared with state of the art [Xie et al., 2014], our approach with adaptive block size is more robust to deal with different kinds of object classes. Although SGC [Xie et al., 2014] achieves best performance in a few classes ( e.g., plane, bird and cow), it fails in several classes (e.g., horse, motorbike and dog).

By comparing results of our approach with equal block size and adaptive block size, we can find that the model with adaptive block size is more robust to class changes and achieves better performance by taking data distribution into consideration. The model with equal block size tends to prefer labels with high frequency (e.g. person and chair), while the model with adaptive block size can handle most classes well at the cost of some classes with high label frequency.

5.3 Results on LabelMe LMODataset

This dataset is a subset of LabelMe dataset [Russell et al., 2008] provided by [Liu et al., 2009a]. It contains 2688 fully annotated images of 33 object categories, most of which are outdoor scenes including sky, street, buildings, mountain. For the supervised methods [Liu et al., 2009a; Myeong and Lee, 2013; Tighe and Lazebnik, 2010; Shotton et al., 2006], there are 2488 randomly selected images for training and 200 for testing. The occurrence frequency for different labels is highly imbalanced as shown in Figure 2. The label 'sky' occurs in more than 85% images, while the label 'moon' occurs in less than 1% images. Such a dataset is very challenging to make semantic segmentation due to high label imbalance. To deal with the high label imbalance, the block size is adaptively tuned with Equation (7) according to the label frequency.

Comparisons with fully supervised methods and weakly supervised methods are given in Table 2. Our approach achieves even better performance than the fully supervised methods [Myeong and Lee, 2013] by taking label imbalance into consideration. It outperforms [Liu et al., 2013] by 15% improvement and [Myeong and Lee, 2013] by 9% improvement. It can handle most classes well at the cost of some classes with high label frequency like 'sky'. Moreover, For such scene datasets, context information will be helpful to make prediction for the position-fixed classes like 'sky' and 'sea', which will be exploited in future.

Furthermore, to validate the robustness of our model to label noise, we conduct extensive experiments with different numbers of noise labels and noise images. Specifically, we randomly select p ∈ {10, 20, · · · , 90, 100} percents of images and add r ∈ {1, 2, 3} randomly selected labels. We repeat each experiment 5 times and report the average accuracy. Detailed results can be found in Figure 4. With fixed number of noise label, we find that the average class accuracy decays linearly with the increase of noise images. Our approach achieves 30% accuracy even when every sample is with a noise label. Moreover, the label accuracy drops little with the increase of noise labels (from 1 to 3) when the noise samples are less than 30 percents, since the non-image-level suppression term and semantic graph propagation term help to alleviate the effect of noisy labels. As the number of noise images increases, many cooccurrence of labels may appear, which may have a larger effect on the final result.

6 Conclusion

In this paper, we propose a weakly supervised semantic segmentation method via non-image-level suppression. Hidden units in the WRBM are divided into multiple blocks, and each block corresponds to a specific label. A non-image-level suppression term is imported to suppress the response of blocks with impossible labels, while a semantic graph propagation term is imported to regularize similar features to have similar hidden response. Extensive experiments on two real world challenging datasets demonstrate the good performance of our approach.

7 Acknowledgments

This work was supported by 973 Program (2012CB316304) and National Natural Science Foundation of China (61332016, 61272329 and 61472422).