Salient Object Detection by Lossless Feature Reflection

1 Introduction

As a fundamental yet challenging task in computer vision, salient object detection (SOD) aims to identify and locate distinctive objects or regions which attract human attention in natural images. In general, SOD is regarded as a prerequisite step to narrow down subsequent object-related vision tasks. For example, it can be used in image retrieval, sematic segmentation, visual tracking and person re-identification, etc.

In the past two decades, a large number of SOD methods have been proposed. Most of them have been well summarized in [Borji et al., 2015]. According to that work, conventional SOD methods focus on extracting discriminative local and global handcrafted features from pixels or regions to represent their visual properties. With several heuristic priors, these methods predict salient scores according to the extracted features for saliency detection. Although great success has been made, there still exist many important problems which need to be solved. For example, the low-level handcrafted features suffer from limited representation capability, and are difficult to capture the semantic and structural information of objects in images, which is very important for more accurate SOD. What's more, to further extract powerful and robust visual features manually is a tough mission.

With the recent prevalence of deep architectures, many remarkable progresses have been achieved in a wide range of computer vision tasks, e.g., image classification [Simonyan and Zisserman, 2014] and semantic segmentation [Long et al., 2015]. Thus, many researchers start to make their great efforts to utilize deep convolutional neural networks (CNNs) for SOD and have achieved favourable performance, since CNNs have strong ability to automatically extract high-level feature representations, successfully avoiding the drawbacks of handcrafted features. However, most of state-of-theart SOD methods still require large-scale pre-trained CNNs, which usually employ the strided convolution and pooling operations. These downsampling methods increase the receptive field of CNNs, helping to extract high-level semantic features, nevertheless they inevitably drop the location information and fine details of objects, leading to unclear boundary predictions. Furthermore, the lack of structural supervision also makes SOD an extremely challenging problem in complex image scenes.

In order to utilize the semantic and structural information derived from deep pre-trained CNNs, we propose to solve both tasks of complementary feature extraction and saliency region classification with an unified framework which is learned in the end-to-end manner. Specifically, we design a symmetrical fully convolutional network (SFCN) architecture which consists of two sibling branches and one fusing branch, as illustrated in Fig. 1. The two sibling branches take reciprocal image pairs as inputs and share weights for learning complementary visual features under the guidance of lossless feature reflection. The fusing branch integrates the multi-level complementary features in a hierarchical manner for SOD. More importantly, to effectively train our network, we propose a novel loss function which incorporates structural information and supervises the three branches during the training process. In this manner, our proposed model sufficiently captures the boundaries and spatial contexts of salient objects, hence significantly boosts the performance of SOD.

In summary, our contributions are three folds:

• We present a novel network architecture, i.e., SFCN, which is symmetrically designed to learn complementary visual features and predict accurate saliency maps under the guidance of lossless feature reflection.
• We propose a new structural loss function to learn clear object boundaries and spatially consistent saliency. This loss function is able to utilize the location, contextual and semantic information of salient objects to supervise the proposed SFCN for performance improvements.
• Extensive experiments on seven large-scale saliency benchmarks demonstrate that the proposed approach achieves superior performance and outperforms the very recent state-of-the-art methods by a large margin.

2 RelatedWork

Salient Object Detection. Recent years, deep learning based methods have achieved solid performance improvements in SOD. For example, [Wang et al., 2015] integrate both local pixel estimation and global proposal search for SOD by training two deep neural networks. [Zhao et al., 2015] propose a multi-context deep CNN framework to benefit from the local context and global context of salient objects. [Li and Yu, 2015] employ multiple deep CNNs to extract multi-scale features for saliency prediction. Then they propose a deep contrast network to combine a pixel-level stream and segmentwise stream for saliency estimation [Li and Yu, 2016]. Inspired by the great success of fully convolutional networks (FCNs) [Long et al., 2015], [Wang et al., 2016] develop a recurrent FCN to incorporate saliency priors for more accurate saliencymap inference. [Liu and Han, 2016] also design a deep hierarchical network to learn a coarse global estimation and then refine the saliency map hierarchically and progressively. Then, [Hou et al., 2017] introduce dense short connections to the skip-layers within the holistically-nested edge detection (HED) architecture [Xie and Tu, 2015] to get rich multi-scale features for SOD. [Zhang et al., 2017a] propose a bidirectional learning framework to aggregate multilevel convolutional features for SOD. And they also develop a novel dropout to learn the deep uncertain convolutional features to enhance the robustness and accuracy of saliency detection [Zhang et al., 2017b]. [Wang et al., 2017b] provide a stage-wise refinement framework to gradually get accurate saliency detection results. Despite these approaches employ powerful CNNs and make remarkable success in SOD, there still exist some obvious problems. For example, the strategies of multiple-stage training reduce the efficiency. And the explicit pixel-wise loss functions used by these methods for model training cannot well reflect the structural information of salient objects. Hence, there is still a large space for performance improvements.

Image Intrinsic Reflection. Image intrinsic reflection is a classical topic in computer vision field. It aims to separate a color image into two intrinsic reflection images: an image of just the highlights, and the original image with the highlights removed. It can be used to segment and analyze surfaces with image color variations. Most of existing methods are based on the Retinex model [Land and McCann, 1971]. Recent years, researchers have augmented the basic Retinex model with non-local texture cues and sparsity priors. Sophisticated techniques that recover reflectance and shading along with a shape estimate have also been proposed. Inspired by these works, we construct a reciprocal image pair based on the input image (see Section 3.1). However, there are three obvious differences between our method and previous intrinsic reflection methods: 1) the objective is different. The aim of previous methods is explaining an input RGB image by estimating albedo and shading fields. Our aim is to learn complementary visual features for SOD. 2) the resulting image pair is different. Image intrinsic reflection methods usually factory an input image into a reflectance image and a shading image, while our method builds a reciprocal image pair for each image as the input of deep networks. 3) the source of reflection is different. The source of previous intrinsic reflection methods is the albedo of depicted surfaces, while our reflection is originated from deep features in CNNs. Therefore, our reflection is feature-level not image-level.

3 The ProposedMethod

Fig. 1 illustrates the semantic overview of our method. In the following subsections, we elaborate the proposed SFCN architecture and the weighted structural loss.

3.1 Symmetrical FCN

The proposed SFCN is an end-to-end fully convolutional network. It consists of three main branches with a paired reciprocal image input to achieve lossless feature reflection learning. Reciprocal Image Input. To capture complementary image information, we first convert the given RGB image X ∈ RW×H×3 to a reciprocal image pair by the following reflection function,

where k is a hyperparameter to control the reflection scale and M ∈ RW×H×3 is the mean of an image or image dataset. From above equations, one can see that the converted image pair, i.e., XO and XkR, is reciprocal with a reflection plane. In detail, the reflection scheme is a pixel-wise negation operator, allowing the given images to be reflected in both positive and negative directions while maintaining the same content of images. In the proposed reflection, we use the multiplicative operator to measure the reflection scale, but it is not the only feasible method. For example, this reflection can be combined with other non-linear operators, such as quadratic form, to add more diversity. To reduce the computation burden, we use k = 1 and the well-known mean of the ImageNet dataset.

Sibling Branches with AdaBN. Based on the reciprocal image pair, we propose two sibling branches to extract complementary reflection features. More specifically, we build each sibling branch, following the VGG-16 model [Simonyan and Zisserman, 2014]. Each sibling branch has 13 convolutional layers (kernel size = 3 × 3, stride size = 1) and 4 max pooling layers (pooling size = 2 × 2, stride = 2). To achieve the lossless reflection features, the two sibling branches are designed to share weights in convolutional layers, but with adaptive batch normalization (AdaBN). In other words, we keep the weights of corresponding convolutional layers of the two sibling branches the same, while use different learnable BN between the convolution and ReLU operators [Zhang et al., 2017a]. The main reason of this design is that after the reflection transform, the reciprocal images have different image domains. Domain related knowledge heavily affects the statistics of BN layers. In order to learn domain invariant features, it's beneficial for each domain to keep its own BN statistics in each layers.

Hierarchical Feature Fusion. After extracting multi-level reflection features, we adhere an additional fusing branch to integrate them for the saliency prediction. In order to preserve the spatial structure and enhance the contextual information, we integrate the multi-level reflection features in a hierarchical manner. Formally, the fusing function is defined by

where h denotes the integration operator, which is a 1 × 1 convolutional layer followed by a deconvolutional layer to ensure the same resolution. [·] is the concatenation operator in channel-wise. gl and g∗ l are the reflection features of the l-th convolutional layer in the two sibling branches, respectively.

In the end, we add a convolutional layer with two filters for the saliency map prediction. The numbers in Fig. 1 illustrate the detailed filter setting in each convolutional layer.

3.2 Weighted Structural Loss

Given the SOD training dataset S = {(Xn, Yn)}Nn=1 with N training pairs, where Xn = {xni , i = 1, ..., T} and Yn = {yn i , i = 1, ..., T} are the input image and the binary groundtruth image with T pixels, respectively. yn i = 1 denotes the foreground pixel and yn i = 0 denotes the background pixel. For notional simplicity, we subsequently drop the subscript n and consider each image independently. In most of existing SOD methods, the loss function used to train the network is the standard pixel-wise binary cross-entropy (BCE) loss:

where θ is the parameter of the network. Pr(yi = 1|X; θ) ∈ [0, 1] is the confidence score of the network prediction that measures how likely the pixel belong to the foreground.

However, for a typical natural image, the class distribution of salient/non-salient pixels is heavily imbalanced: most of the pixels in the ground truth are non-salient. To automatically balance the loss between positive/negative classes, we introduce a class-balancing weight β on a per-pixel term basis, following [Xie and Tu, 2015]. Specifically, we define the following weighted cross-entropy loss function,

The loss weight β = |Y+|/|Y |, and |Y+| and |Y-| denote the foreground and background pixel number, respectively.

For saliency detection, it is also crucial to preserve the overall spatial structure and semantic content. Thus, we also minimize the differences between their multi-level features by a deep convolutional network [Johnson et al., 2016]. The main intuition behind this operator is that minimizing the difference between multi-level features, which encode low-level fine details and high-level coarse semantics, helps to retain the spatial structure and semantic content of predictions. Formally, let φl denotes the output of the l-th convolutional layer in a CNN, our semantic content (SC) loss is defined as

where ˆ Y is the overall prediction, w is the parameter of a pretrained CNN and λl is the trade-off parameter, controlling the influence of the loss in the l-th layer. In our case, we use the light CNN-9 model [Wu et al., 2015] to calculate the above loss between the ground-truth and the prediction.

To overcome the blurry boundary problem [Li et al., 2016], we also introduce the smooth L1 loss which encourages to keep the details of boundaries of salient objects. Specifically, the smooth L1 loss function is defined as

where D = Y - ˆ Y and ϵ is a predefined threshold. Following the practice in [Xiao et al., 2018], we set ϵ = 0.5. This training loss also helps to minimize pixel-level differences between the overall prediction and the ground-truth.

By taking all above loss functions together, we define our final loss function as

where µ and γ are hyperparameters to balance the specific terms. All the above losses are continuously differentiable, so we can use the standard stochastic gradient descent (SGD) method to obtain the optimal parameters. In addition, we use λl = 1, µ = 0.01 and γ = 20 to optimize the final loss function for our experiments without further tuning.

4 Experimental Results

4.1 Datasets and EvaluationMetrics

To train our model, we adopt the MSRA10K [Borji et al., 2015] dataset, which has 10,000 training images with high quality pixel-wise saliency annotations. To combat overfitting, we augment this dataset by random cropping and mirror reflection, producing 120,000 training images totally.

For the performance evaluation, we adopt seven public saliency detection datasets as follows: DUT-OMRON [Yang et al., 2013] dataset has 5,168 high quality natural images. Each image in this dataset has one or more objects with relatively complex image background. DUTS-TE dataset is the test set of currently largest saliency detection benchmark (DUTS) [Wang et al., 2017a]. It contains 5,019 images with high quality pixel-wise annotations. ECSSD [Shi et al., 2016] dataset contains 1,000 natural images, in which many semantically meaningful and complex structures are included. HKU-IS-TE [Li and Yu, 2015] dataset has 1,447 images with pixel-wise annotations. Images of this dataset are well chosen to include multiple disconnected objects or objects touching the image boundary. PASCAL-S [Li et al., 2014] dataset is generated from the PASCAL VOC [Everingham et al., 2010] dataset and contains 850 natural images with segmentation-based masks. SED [Borji et al., 2015] dataset has two non-overlapped subsets, i.e., SED1 and SED2. SED1 has 100 images each containing only one salient object, while SED2 has 100 images each containing two salient objects. SOD [Jiang et al., 2013] dataset has 300 images, in which many images contain multiple objects either with low contrast or touching the image boundary.

To evaluate the performance of varied SOD algorithms, we adopt four metrics, including the widely used precision-recall (PR) curves, F-measure, mean absolute error (MAE) [Borji et al., 2015] and recently proposed S-measure [Fan et al., 2017]. The PR curve of a specific dataset exhibits the mean precision and recall of saliency maps at different thresholds. The Fmeasure is a weighted mean of average precision and average recall, calculated by

We set η2 to be 0.3 to weigh precision more than recall as suggested in [Borji et al., 2015].

For fair comparison on non-salient regions, we also calculate the mean absolute error (MAE) by

where W and H are the width and height of the input image. S(x, y) and G(x, y) are the pixel values of the saliency map and the binary ground truth at (x, y), respectively.

To evaluate the spatial structure similarities of saliency maps, we also calculate the S-measure, defined as

where λ ∈ [0, 1] is the balance parameter. So and Sr are the object-aware and region-aware structural similarity, respectively. We set λ = 0.5 as suggested in [Fan et al., 2017].

4.2 Implementation Details

We implement our model based on the Caffe toolbox [Jia et al., 2014] with the MATLAB 2016 platform. We train and test our method with an NVIDIA Titan 1070 GPU (8G memory) and an i5-6600 CPU. Following [Zhang et al., 2017a; 2017b], we do not use validation set and train the model until its training loss converges. The input image is uniformly resized into 384 × 384 × 3 pixels and subtracted the ImageNet mean [Deng et al., 2009]. The weights of sibling branches are initialized from the VGG-16 model. For the fusing branch, we initialize the weights by the "msra" method. During the training, we use standard SGD method with batch size 12, momentum 0.9 and weight decay 0.0005. We set the base learning rate to 1e-8 and decrease the learning rate by 10% when training loss reaches a flat. The training process converges after 150k iterations. When testing, our proposed SOD algorithm runs at about 12 fps. The source code is publicly available at http://ice.dlut.edu.cn/lu/.

4.3 Comparison with the State-of-the-arts

To fully evaluate the detection performance, we compare our proposed method with other 14 state-of-the-art ones, including 10 deep learning based algorithms (Amulet [Zhang et al., 2017a], DCL [Li and Yu, 2016], DHS [Liu and Han, 2016], DS [Li et al., 2016], ELD [Lee et al., 2016], LEGS [Wang et al., 2015], MCDL [Zhao et al., 2015], MDF [Li and Yu, 2015], RFCN [Wang et al., 2016], UCF [Zhang et al., 2017b]) and 4 conventional algorithms (BL [Tong et al.,

2015], BSCA [Qin et al., 2015], DRFI [Jiang et al., 2013], DSR [Li et al., 2013]). For fair comparison, we use either the implementations with recommended parameter settings or the saliency maps provided by the authors.

Quantitative Evaluation. As illustrated in Tab. 1, Tab. 2 and Fig. 2, our method outperforms other competing ones across all datasets in terms of near all evaluation metrics. Due to the limitation of space, we only present the PR curves on the DUTS-OMRON, DUTS-TE, ECSSD and HKU-IS. From these results, we have other notable observations: (1) deep learning based methods consistently outperform traditional methods with a large margin, which further proves the superiority of deep features for SOD. (2) our method achieves higher S-measure than other methods, especially on complex structure datasets, e.g., the DUT-OMRON, SED and SOD datasets. We attribute this result to our structural loss. (3) without segmentation pre-training, our method only finetuned from the image classification model still achieves better results than the DCL and RFCN, especially on the HKU-IS and SED datasets. (4) compared to the DHS and Amulet, our method is inferior on the DUTS-TE and PASCAL-S datasets. However, our method still ranks in the second place.

Qualitative Evaluation. Fig. 3 provides several visual examples in various challenging cases, where our method outperforms other compared methods. For example, the images in the first two rows are of very low contrast, where most of the compared methods fail to capture the salient objects, while our method successfully highlights them with sharper edges preserved. The images in the 3-4 rows are challenging with complex structures or salient objects near the image boundary, and most of the compared methods can not predict the whole objects, while our method captures the whole salient regions with preserved structures.

Ablation Analysis. We also evaluate the main components in our model. Tab.3 shows the experimental results with different model settings. All models are trained on the augmented MSRA10K dataset and share the same hyper-parameters described in subsection 4.2. Due to the limitation of space, we only show the results on the ECSSD dataset. Other datasets have the similar performance trend. From the results, we can see that the SFCN only using the channel concatenation operator without hierarchical fusion (model (a)) has achieved comparable performance to most deep learning methods. This confirms the effectiveness of reflection features. With the hierarchical fusion, the resulting SFCN (model (b)) improves the performance by a large margin. The main reason is that the fusion method introduces more contextual information from high layers to low layers, which helps to locate the salient objects. In addition, it's no wonder that training with the Lwbce loss achieves better results than Lbce. With other two losses Lsc and Ls1, the model achieves better performance in terms of MAE and S-measure. These results demonstrate that individual components in our model complement each other. When taking them together, the overall model, i.e., SFCN+Lwbce+Lse+Ls1, achieves best results under all evaluation metrics.

5 Conclusion

In this work, we propose a novel end-to-end feature learning framework for SOD. Our method uses a symmetrical FCN to learn complementary visual features under the guidance of lossless feature reflection. For training, we also propose a new weighted structural loss that integrates the location, semantic and contextual information of salient objects to boost the detection performance. Extensive experiments on seven large-scale saliency datasets demonstrate that the proposed method achieves significant improvement over the baseline and performs better than other state-of-the-art methods.