MEnet: A Metric Expression Network for Salient Object Segmentation

1 Introduction

Image saliency detection and segmentation is of significant interest in the fields of computer vision and pattern recognition. Recent saliency detection studies can be divided into two categories: those based on hand-crafted features and learning-based approaches. In previous literature, the majority of saliency detection methods have used hand-crafted features. Traditional low-level features for such saliency detection models mainly consist of color, intensity, texture and structure [Yang et al., 2013; Cheng et al., 2015; Borji and Itti, 2012]. Though hand-crafted features with heuristic priors perform well in simple scenes, they are not robust to more challenging cases, such as when salient regions have similar color to background.

Learning-based methods, in particular using convolutional neural networks (CNNs) [LeCun et al., 1998] have been proposed to address the shortcomings of using hand-crafted features for saliency detection. For example, [Wang et al., 2017] uses a multi-stage refinement mechanism to effectively combine high-level object semantics with low-level image features to produce high-resolution saliency maps, while [Luo et al., 2017; Liu and Han, 2016; Zhang et al., 2017a] exploit multi-level and multi-scale convolutional features for object segmentation. But even though they obtain good performance, CNN-based approaches also have room for improvement in their robustness to distorted scenes and to other common distortions such as noise [Chen et al., 2017].

∗The co-first authors contributed equally.

†Corresponding author: dlzeng@scut.edu.cn

Metric learning is an area receiving much attention in computer vision, such for image segmentation [Fathi et al., 2017], face recognition [Hu et al., 2014] and human identification [Yi et al., 2014], as a way for measuring similarity between objects. Inspired by the metric learning framework, we propose a deep metric learning architecture for image saliency segmentation that is also robust to potential distortions within an image. Our goal is to learn a metric space containing semantic features using a deep CNN such that two homogeneous sections of this space are learned for the salient and non-salient regions of the image space.

These features are learned at the pixel level and allow for distinguishing between salient regions and background using a distance measure. Simultaneously, we introduce a metric loss function based on metric learning and cross entropy. We also use multi-level information for feature extraction, similar to other approaches, such as Hypercolumns [Hariharan et al., 2015] and U-net [Ronneberger et al., 2015].

We experiment on several benchmark data sets and show how our proposed approach achieves results at state-of-art level. Moreover, we show how the proposed model is robust to distortions within an image.

2 AMetric Expression Network (MEnet)

We illustrate our proposed model architecture MEnet in Figure 1. As shown there, an encoder-decoder CNN first generates feature maps at different scales (blocks), which through convolution and up-sampling gives a feature vector for each pixel of an image according to how it maps through the layers. These extracted features are then used in a combined metric loss and cross entropy function to learn the salient regions as described below. We first discuss the encoder-decoder CNN

followed by our loss function and finally our semantic distance measure.

2.1 Encoder-Decoder CNN for Feature Extraction

Saliency segmentation usually requires global information about the image [Wang et al., 2015], and thus multi-scale information is beneficial for more precise image segmentation. To learn this global information with deep learning, we use convolutions and pooling layers to increase the receptive field of the model and compress all feature information into feature maps whose size are 1 × 1, as shown as the white box in Figure 1. For multi-scale information, in previous approaches such as SegNet [Badrinarayanan et al., 2017] and U-net, the encoder-decoder is used to extract multi-scale features. Here we use a similar structure. Through the decoder module, we up-sample these feature maps and view the feature map at each scale as representing information at a certain semantic level. We propose a symmetric encoder-decoder CNN architecture to extract global and multi-scale feature maps.

The encoder-decoder network of Figure 1 uses a deep symmetric CNN architecture with skip connections as indicated by black arrows. It consists of an encoder half and a decoder half, each block of which contains one of the two basic blocks shown in Figure 2. For encoding, at each down-sampling step we double the number of feature channels using a convolution with stride 2. For decoding, each step in the decoder path consists of an up-sampling of the feature map by a deconvolution after concatenating the input with the skip connection, also with stride 2. This part is similar to U-Net, but the difference is that U-net is designed for image segmentation, which is objective and works well even with cropped feature maps. For saliency segmentation, it is subjective and easily affected in different scenarios. Thus, global information is of significant importance to salient object segmentation. We maintain the size of the feature map to make full use of all the information in the larger receptive field. Our goal in using a symmetric CNN is to generate different scales of feature maps, which are concatenated to obtain feature vectors for each corresponding pixel in the input image that contain multi-scale information across the channel dimension. For instance, previous work in this direction showed that deep CNNs can learn such a feature representation that captures local and global context information for saliency segmentation [Zhao et al., 2015].

We ultimately want to distinguish salient objects from background and so want to map image pixels into a feature space where that distance across salient and background regions is large, but within regions is small. Therefore, as shown in Figure 1, we can convert the 13 different scales of the encoder-decoder network into a set of feature vectors as indicated by the green dashed lines. That is, in the feature extraction part, each scale generates one output feature map of the same size via a single convolution and up-sampling; while the first "feature map" is simply obtained from convolving the original image across its RGB channels. Though the proposed algorithm is similar to the Hypercolumns model, one difference is that when training, the Hypercolumns model predicts heatmaps from feature maps of different scales by stacking additional convolutional layers. Hypercolumns is more like DHSNet [Liu and Han, 2016] which uses multiscale saliency labels for segmentation. Instead, MEnet upsamples each scale of feature map to the same size during training. Another difference is that, where Hypercolumns classifies each category at separate layers, MEnet integrates the multi-scale feature maps for these tasks. As these 13 features may have unequal information value, learn this with another convolutional filter of these 13 feature maps. After concatenating the feature maps at each level we further use convolutional operations with 16 kernels to generate the final feature maps. We incorporate cross entropy to help with this task, as described in the following section. In this case, the final feature vector is in R16 .

2.2 Loss Function

Many previous works on saliency detection based on deep learning uses cross entropy (CE) to optimize the network [Li and Yu, 2016; Luo et al., 2017]. This loss function is written as follows:

where θ1 is the set of learnable parameters of network, Ω is the pixel domain of the image, LCE denotes the loss of the entire training set, N is the number of training data, 1{·} is the indicator function, and y ∈ {0, 1}, where y = 1 denotes the salient pixel and y = 0 denotes the non-salient pixel. P(l(n) i = y|θ1) is the label probability of the i-th pixel predicted by network. InMEnet, we generate P(l(n) i = y|θ1) via a convolution with 2 kernels from feature extraction part as shown in Figure 1.

Metric learning has been widely used in computer vision tasks. For instance, in [Fathi et al., 2017; Harley et al., 2017], the idea of metric learning is applied to segmentation. However, in [Fathi et al., 2017], only one scale of the input corresponds to one corresponding size feature map, while we propose to use more feature maps from different scales to generate the final saliency map. In [Harley et al., 2017], local attention masks are constructed by pairwise distance computations from a neighborhood around each pixel, which may not be suitable for saliency segmentation. Therefore, we instead use the triplet loss to compute the global information. Our metric loss function (ML) is defined as in Equation 2. In our network, the input is an RGB image whose size is H ×W ×3, and all the images are resized to 224×224×3, hence H = W = 224 here. The output is a feature metric space which is generated by 16 kernel convolutions in Figure 1, and the size is H ×W × C (we set C = 16). Each pixel in the H ×W image corresponds to a C-dimension vector in the salient feature map. The metric loss function is defined as following:

where θ2 is the set of learnable parameters of network and f (n) i denotes the feature vectors corresponding to the pixel in the n-th image of the training set. We denote k ∈ set+ (or k ∈ set-), with Ω = set+ ∪ set-, meaning that f (n) k is the positive or negative feature vector of f (n) i , respectively. That is, either f (n) i and f (n) k are from the same region (salient or non-salient), otherwise, f (n) k is from a different region from f (n) i . We use Euclidean distance to calculate the distance between two feature vectors.

This loss function in (2) encourages an encoder-decoder network that enlarges the distance between any pair of feature vectors having different saliency, and reduces the distance for those with the same saliency. This is equivalent to

where we average all f (n) k in Equation 3 to get ¯ f+(n) and ¯ f-(n) . That is ¯ f+(n) is the mean of all positive pixels from a single image, while ¯ f-(n) corresponds to all negative pixels. Intuitively, Equation 3 enforces that the feature vectors extracted from the same region be close to the center of that region while keeping away from the center of the other region in salient feature space. In this case, we can obtain a more robust distance evaluation between the salient object and background. We also add a second cross entropy loss function as a constraint which shares the same network architecture with the objective function and empirically we have observed that the combined results were significantly better than only using either the metric loss or the cross entropy loss alone. Therefore, our final loss function is defined as below:

where θ = θ1 ∪ θ2 and λ is set to 1 in our experiments.

2.3 Semantic Distance Expression

If we train the proposed MEnet to minimize the loss function LMEnet(·), we will obtain a network Tθ∗, where θ∗ is converged value of θ. Given an observed input image for testing, where the pixel domain is Ω, we usually describe pixel i ∈ Ω by its intensities Ii across the channels. But it is difficult to define the semantic distance by dIΩ ij = d(Ii, Ij), e.g., by Euclidean distance dij = ‖Ii - Ij‖2. However, through transformation of Tθ∗, we will obtain the corresponding feature vectors {fi}i∈Ω to represent the input. Then the distance can be expressed as d′ij = dTθ∗ (IΩ) ij = ‖fi - fj‖2, and finally the saliency map S for saliency segmentation is obtained by:

where PB(·) is the probability distribution of the feature vector fj ∈ ΩB, and Ω = ΩB ∪ ΩS, where ΩB and ΩS denote the background region and salient region only computed from the component of LCE in the loss function (4) within the converged network Tθ∗ . We note that, ΩB and ΩS are not accurate segmentations and they are to be further investigated in the experimental section. To conclude, by network transformation we can express dIΩ ij as dTθ∗(IΩ) ij . As illustrated in Figure 3, we anticipate that through this space transformation, the intra-class distance will be smaller than the inter-class distance.

3 Experiments

We test our proposed MEnet on several public saliency datasets and distorted images, comparing with state-of-theart saliency detection methods. We use the Caffe software package to train our model [Jia et al., 2014].

3.1 Datasets

The datasets we consider are: MSRA10K [Cheng et al., 2015], DUT-OMRON (DUT-O) [Yang et al., 2013], HKUIS [Li and Yu, 2015], ECSSD [Yan et al., 2013], MSRA1K [Liu et al., 2011] and SOD [Martin et al., 2001]. MSRA10K contains 10000 images. It is the largest dataset and covers a large variety of content. HKU-IS contains 4447 images, most images containing two salient objects or multiple objects. ECSSD dataset contains 1000 images. DUT-OMRON contains 5168 images, which was originally designed for image segmentation. This dataset is very challenging since most of the images contain complex scenes. Existing saliency detection models have yet to achieve high accuracy on this dataset. MSRA1K including 1000 images, all belongs to the MSRA10K. SOD contains 300 images.

3.2 Training

We use stochastic gradient descent (SGD) for optimization, and the MSRA10K and HKU-IS are selected for training. For MSRA10K, 8500 images for training, 500 images for validation and the MSRA1K for testing; HKU-IS was divided into approximately 80/5/15 training-validation-testing splits. To prevent overfitting, all of our models use cropping and flipping images randomly as data augmentation. We use batch normalization [Ioffe and Szegedy, 2015] to speed up convergence.

All experiments are performed on a PC with Intel(R) Xeon(R) CPU I7-6900k, 96GB RAM and GTX TITAN X Pascal (12G). We use a 4 convolutional layer block in the upsample and downsample operations. Therefore the depth of our MEnet is 52 layers. The parameter sizes are shown in Figure 1 and Figure 2. We set the learning rate to 0.1 with weight decay of 10-8 , a momentum of 0.9 and a mini-batch size of 5. We train for 110,000 iterations. Since salient pixels and non-salient pixels are very imbalanced, network convergence to a good local optimum is challenging. Inspired by object detection methods such as SSD [Liu et al., 2016], we adopt hard negative mining to address this problem. This sampling scheme ensures salient and non-salient sample ratio equal to 1, eliminating label bias.

3.3 Performance Comparison

We compare MEnet with 10 state-of-the-art models for saliency detection: MC [Zhao et al., 2015], ELD [Wang et al., 2015], DCL [Li and Yu, 2016], DHSNet [Liu and Han, 2016], DS [Li et al., 2016], UCF [Zhang et al., 2017b], Amulet [Zhang et al., 2017a], SRM [Wang et al., 2017], NLDF [Luo et al., 2017], MSRNet [Li et al., 2017] and 2 traditional metric learning methods: AML [Li et al., 2015] and Lu's method [You et al., 2016].

A visual comparison is shown in Figure 4 along with other state-of-the-art methods. MEnet performs better in these challenging scenes, e.g., when the salient region is similar to background. In addition, F-measure scores and MAE are shown in Table 1. We note that the better models (e.g., DHSNet, NLDF, Amulet, SRM, MSRNet and etc.) need pre-training and the conditional random field (CRF) method [Kr¨ahenb¨uhl and Koltun, 2011] is used as post-processing in DCL and MSRNet. MEnet is trained from scratch and does not require pre/post-processing. It is still competitive with state-of-the-art models, particularly on the challenging datasets DUT-O and HKU-IS.

Table 2 shows the running times of the compared methods. For fair evaluation, the time efficiency of all models are performed on the same PC described above. It takes 86ms for our model to generate each saliency map with a GPU. Though our model is deeper, our test time is comparable with the fast models.

Evaluation on Distorted Images

We also test the models on distorted images. We note that MEnet does not train on distorted images for this case, as similar to previous works. During testing, the trained models are then directly tested on distorted images. To show the robustness of MEnet in this setting, we work with public datasets corrupted by Additive White Gaussian Noise (AWGN) and JPEG compression (with random strengths). For AWGN, we let the variance vary from 0.07 to 0.29, while for JPEG compression, we vary the quality factor from 3 to 6. We compare F-measure scores in Table 3. We can see that MEnet clearly outperforms other methods. Additionally, we show PR curves of our approach in Figure 5. Since the saliency maps generated by metric loss prediction tend to be binary, it is difficult to draw PR curves which need continuous salient values. Therefore, we select saliency maps generated by CE prediction to draw PR curves. In Figure 5, we observe that the performance of the proposed method is a little better than others on distorted datasets. As shown in Figure 7, the performance of other methods degrade rapidly with increasing noise, while MEnet still achieves robust performance. We believe reason for the robustness of MEnet owes to the fact that multi-scale features and metric loss are integrated into this structure, where features from either low or high levels can be fully utilized. In particular, we can see some evidence in [Du et al., 2017] for denoising which uses an auto-encoder (similar to our Encoder-Decoder module) to obtain more robust features. A similar metric loss idea was shown to be robust to lighting conditions, deformation, and angle for human re-identification [Yi et al., 2014], all of which can be regarded as "noise." Also, for vehicle re-identification, the distance similarity has been shown to provide vital information for robustly estimating the similarity among objects [Shen et al., 2017]. In real-world scenes, images are easily impacted by noise and compression. Therefore, we consider the proposed work to be a more robust model.

Advantages of MEnet

In previous work, multi-scale features have been applied to produce saliency maps [Liu and Han, 2016; Zhang et al., 2017a]. Although this is similar to our approach, there exist some differences in that these mentioned works predict saliency maps at each scale and so feature maps from the last layer of each scale may be similar. We propose to integrate multi-scale feature maps for classification and distance prediction. We only concatenate the feature maps to generate the final saliency maps.

To intuitively illustrate the advantage of MEnet, we select several feature maps for visualization. As we move to deeper layers, the receptive field of each neuron becomes larger. As shown in Figure 6, we observe that each convolutional layer contains different semantic information, and moving deeper allows the models to capture richer structures. Within the decoding part, Scale2-de, 3-de, 4-de are sensitive to the salient region, while Scale1-de has higher response against the background region. Other layers like Scale0-de can distinguish the boundary of salient objects.

Also, in [Liu and Han, 2016; Zhang et al., 2017a], a convolutional layer with 1 × 1 kernels is used to fuse multi-scale features, which may lead to the receptive field being restricted. Instead of using 1 × 1 convolutions in the last layer, we instead use an n × n convolution in the last layer, containing more units to capture information from its neighborhood.

To show the effectiveness of our proposed multi-scale feature extraction and loss function, we use different strategies for semantic saliency detection/segmentation as shown in Table 4. CE-only uses the cross entropy as its loss function, while CE-plain omits the feature extraction part and metric loss layer, and the loss layer is added directly to the decoder module in the framework. Therefore, the difference between CE-only and CE-plain is that CE-plain does not use multiscale information which will lead to performance degradation. We also note that the performance of MEnet improves after introducing the metric loss. The multi-scale framework (encoder-decoder) and metric loss help make it feasible to distinguish saliency from background during training.

We compare MEnet with two other traditional metric learning methods for saliency segmentation, AML [Li et al., 2015] and Lu [You et al., 2016]. The results in Table 5 demonstrates the potential superiority of deep metric learning over traditional metric learning for semantic saliency segmentation.

4 Conclusion

In this paper, we present an end-to-end deep metric learning architecture called MEnet for salient object segmentation. We use multi-scale features extraction to obtain semantic information and combine with deep metric learning for mapping pixels into a "saliency space" where Euclidean distances can be used. The resulting mapping distinguishes salient image elements (pixels) from background efficiently. The proposed model is trained from scratch and does not require pre/post-processing. Experiments on benchmark datasets clearly demonstrate the effectiveness of our model, and robustness when handling distorted images.

Acknowledgments

This work was supported in part by grants from National Science Foundation of China (6151005, 61571382, 61103121, 81671766), the China Scholarship Council (201806155037), Guangdong Natural Science Foundation (2015A030313007, 2015A030313589), and the Science and Technology Research Program of Guangzhou, China (201804010429).