Salient Object Detection with Semantic Priors

1 Introduction

Salient object detection aims to determine the salient objects which draw the attention of humans on the input image. It has been successfully adopted in many practical scenarios, including image resizing [Goferman et al., 2010], attention retargeting [Nguyen et al., 2013a], dynamic captioning [Nguyen et al., 2013b] and video classification [Nguyen et al., 2015b]. The existing methods can be classified into biologically-inspired and learning-based approaches.

The early biologically-inspired approaches [Itti et al., 1998; Koch and Ullman, 1985] focused on the contrast of low-level features such as orientation of edges, or direction of movement. Since human vision is sensitive to color, different approaches use local or global analysis of (color-) contrast. Local methods estimate the saliency of a particular image region based on immediate image neighborhoods, e.g., based on histogram analysis [Cheng et al., 2011]. While such approaches are able to produce less blurry saliency maps, they are agnostic of global relations and structures, and they may also be more sensitive to high frequency content like image edges and noise. In a global manner, [Achanta et

Luoqi Liu

Department of ECE National University of Singapore liuluoqi@u.nus.edu

al., 2009] achieves globally consistent results by computing color dissimilarities to the mean image color. There also exist various patch-based methods which estimate dissimilarity between image patches [Goferman et al., 2010; Perazzi et al., 2012]. While these algorithms are more consistent in terms of global image structures, they suffer from the involved combinatorial complexity, hence they are applicable only to relatively low resolution images, or they need to operate in spaces of reduced image dimensionality [Bruce and Tsotsos, 2005], resulting in loss of salient details and highlighting edges.

For the learning-based approaches, [Jiang et al., 2013] trained a model to learn the mapping between regional features and saliency values. Meanwhile, [Kim et al., 2014] separated the salient regions from the background by finding an optimal linear combination of color coefficients in the highdimensional color space. However, the resulting saliency maps tend to also highlight adjacent regions of salient object(s). Additionally, there exist many efforts to study visual saliency with different cues, i.e., depth matters [Lang et al., 2012], audio source [Chen et al., 2014], touch behavior [Ni et al., 2014], and object proposals [Nguyen and Sepulveda, 2015].

Along with the advancements in the field, a new challenging question is arisen "why an object is more salient than others". This emerging question appears along with the rapid evolvement of the research field. The early datasets, i.e., MSRA1000 [Achanta et al., 2009], only contain images with one single object and simple background. The challenge is getting more serious when more complicated saliency datasets, ECSSD [Yan et al., 2013] and HKUIS [Li and Yu, 2015] are introduced with one or multiple objects in an image with complex background. This drives us to the difference between the human fixation collection procedure and the salient object labeling process. In the former procedure, the human fixation is captured when a viewer is displayed an image for 2-5 seconds under free-viewing settings. Within such a short period of time, the viewer only fixates to some image locations that immediately attract his/her attention. For the latter process, a labeler is given a longer time to mark the pixels belonging to the salient object(s). In case of multiple objects appearing in the image, the labeler naturally identifies the semantic label of each object and then decides which object is salient. This bridges the problem of salient object detection into the semantic segmentation research. In the latter semantic segmentation problem, the semantic label of each single pixel is decided based on a trained model which maps the features of the region containing the pixel and a particular semantic class label [Liu et al., 2011]. There are many improvements in this task by handling the adaptive inference [Nguyen et al., 2015a], adding object detectors [Tighe and Lazebnik, 2013], or adopting deep superpixel's features [Nguyen et al., 2016]. There emerges a deep learning method, fully connected network (FCN) [Long et al., 2015], which modifies the popular Convolutional Neural Networks (CNN) [Krizhevsky et al., 2012] to a new deep model mapping the input pixel directly to a semantic class label. There are many works improving FCN by considering more factors such as probabilistic graphical models [Zheng et al., 2015].

Recently, along with the advancement of deep learning in semantic segmentation, deep networks, such as CNN, or even FCN, have been exploited to obtain more robust features than handcrafted ones for salient object detection. In particular, deep networks [Wang et al., 2015; Li and Yu, 2015; Li et al., 2016] achieve substantially better results than previous state of the art. However, these works only focus on switching the training data (with output from semantic classes to binary classes for salient object detection problem), or adding more network layers. In fact, the impact of the semantic information is not explicitly studied in the previous deep network-based saliency models. Therefore, in this work, we investigate the application of semantic information into the problem of salient object detection. In particular, we propose the semantic priors to form the explicit and implicit semantic saliency maps in order to produce a high quality salient object detector. The main contributions of this work can be summarized as follows.

• We conduct the comprehensive study on how the semantic priors affect the salient object detection.
• We propose the explicit saliency map and the implicit saliency map, derived from the semantic priors, which can discover the saliency object(s).
• We extensively evaluate our proposed method on two challenging datasets in order to know the impact of our work in different settings.

2 ProposedMethod

In this section, we describe the details of our proposed semantic priors (SP) based saliency detection, and we show how to integrate the semantic priors as well as the saliency fusion can be efficiently computed. Figure 1 illustrates the overview of our processing steps.

2.1 Semantic Extraction

Saliency detection and semantic segmentation are highly correlated but essentially different in that saliency detection aims at separating generic salient objects from background, whereas semantic segmentation focuses on distinguishing objects of different categories. As aforementioned, fully connected network (FCN) [Long et al., 2015] and its variant, i.e., [Zheng et al., 2015] are currently the state-of-the-art methods in the semantic segmentation task. Therefore, in this paper, we consider the end-to-end deep fully connected networks into our framework. Here, "end-to-end" means that the deep networks only need to be run on the input image once to produce a complete semantic map C with the same pixel resolution as the input image. We combine outputs from the final layer and the pool4 layer, at stride 16 and pool3, at stride 8. In particular, we obtain the confidence score Cx,y for each single pixel (x, y) as below.

where C 1 x,y , C 2 x,y , · · · ,C nc x,y indicate the likelihood that the pixel (x, y) belongs to the listed nc semantic classes. Given an input image with size h × w, the dimension of C is h × w × nc.

2.2 Explicit SaliencyMap

The objective of the explicit saliency map is to capture the preference of humans over different semantic classes. In other words, we aim to investigate which class is favoured by humans if there exist two or more classes in the input image. The class label Lx,y of each single pixel (x, y) can be obtained as below:

Next, given a groundtruth map G, the density of each semantic class k in the input image is defined by:

where (Lx,y = k) is a boolean expression which verifies whether Lx,y is equivalent to class k. Note that the size of the groundtruth map is also h×w. Given the training dataset, we extract the co-occurrence saliency pairwise of one class and other nc - 1 classes. The pairwise value θg,t of two semantic classes g and t is computed as below.

We define the explicit semantic priors as the accumulated co-occurrence saliency pairwise of all classes. The explicit semantic priors of two classes g and t is calculated as below.

where nt is the number of images in the training set, and ϵ is inserted to avoid the division by zero. For the testing phase, given a test image, the explicit saliency value of each single pixel (x, y) is computed as:

2.3 Implicit SaliencyMap

Obviously the explicit saliency map performs well in case of the detected objects are in the listed class labels. However, the explicit saliency map fails in case of the salient objects are not in the nc class labels. Therefore, we propose the implicit saliency map which can uncover the salient objects not in the listed semantic classes. To this end, we oversegment the input image into non-overlapping regions. Then we extract features from each image region. Different from other methods which simply learn the mapping between the locally regional features with the saliency values, here, we take the semantic information into consideration. In particular, we are interested in studying the relationship between the regional features with the saliency values under the impact of semantic-driven features. Therefore, besides the off-the-shelf region features, we add two new features for each image region, namely, global semantic and local semantic features. The local semantic feature of each image region q is defined as:

where r(x, y) returns the region index of pixel (x, y). Meanwhile, the global semantic feature of the image region q is defined as:

The semantic features spImplicit = {sp1, sp2} are finally combined with other regional features. We consider the semantic features here as the implicit semantic priors since they implicitly affect the mapping of the regional features and saliency scores. All of regional features are listed in Table 1. Then, we learn a regressor rf which maps the extracted features to the saliency values. In this work, we adopt the random forest regressor in [Jiang et al., 2013] which demonstrates a good performance. The training examples include a set of nr regions {{r1, spImplicit 1 }, {r2, spImplicit 2 }, · · · , {rnr , spImplicit nr }} and the corresponding saliency scores {s1, s2, · · · , snr }, which are collected from the oversegmentation across a set of images with the ground truth annotation of the salient objects. The saliency value of each training image region is set as follows: if the number of the pixels (in the region) belonging to the salient object or the background exceeds 80% of the number of the pixels in the region, its saliency value is set as 1 or 0, respectively.

For the testing phase, given the input image, the implicit saliency value of each image region q is computed by feeding the extracted features into the trained regressor rf:

2.4 Saliency Fusion

Given an input image with a size h × w, the saliency maps from both aforementioned saliency maps are fused at the end. In particular, we scale the implicit saliency map SImplicit , explicit saliency map SExplicit , to the range [0..1]. Then we combine these maps as follows to compute a saliency value S for each pixel:

where the weights α is adaptively set as ∑x,y SImplicit x,y h×w . Actually α measures how large the semantic pixels occupied in the image. Meanwhile, γ is set as 1-α. The resulting pixel-level saliency map may have an arbitrary scale. Therefore, in the final step, we rescale the saliency map to the range [0..1] or to contain at least 10% saliency pixels. Fig. 2 demonstrates that the two individual saliency maps, i.e., explicit and implicit ones, complement each other in order to yield the good result.

2.5 Implementation Settings

For the implementation, we adopt the extension of FCN, namely CRF-FCN [Zheng et al., 2015], to perform the semantic segmentation for the input image. In particular, we utilize the CRF-FCN model trained from the PASCAL VOC 2007 dataset [Everingham et al., 2010] with 20 semantic classes 1 . Therefore, the regional feature's dimensionality is 79. We trained our SP framework on HKUIS dataset [Li and Yu, 2015] (training part) which contains 4, 000 pairs of images and groundtruth maps. For the image over-segmentation, we adopt the method of [Achanta et al., 2012]. We set the number of regions as 200 as a trade-off between the fine over-segmentation and the processing time.

1 There are 20 semantic classes in the PASCAL VOC 2007 ('aeroplane', 'bicycle', 'bird', 'boat', 'bottle', 'bus', 'car', 'cat', 'chair', 'cow', 'diningtable', 'dog', 'horse', 'motorbike', 'person', 'pottedplant', 'sheep', 'sofa', 'train', 'tvmonitor'); and an extra 'others' class label.

3 Evaluation

3.1 Datasets and EvaluationMetrics

We evaluate and compare the performances of our algorithm against previous baseline algorithms on two challenging benchmark datasets: ECSSD [Yan et al., 2013] and HKUIS [Li and Yu, 2015] (testing part). The ECSSD dataset contains 1,000 images with the complex and cluttered background. Meanwhile, the HKUIS contains 1,447 images. Note that each image in both datasets contains single or multiple salient objects.

The first evaluation compares the precision and recall rates. In the first setting, we compare binary masks for every threshold in the range [0..255]. In the second setting, we use the image dependent adaptive threshold proposed by [Achanta et al., 2009], defined as twice the mean value of the saliency map S. In addition to precision and recall we compute their weighted harmonic mean measure or F -measure, which is defined as:

As in previous methods [Achanta et al., 2009; Perazzi et al., 2012], we use β2 = 0.3.

For the second evaluation, we follow [Perazzi et al., 2012] to evaluate the mean absolute error (MAE) between a continuous saliency map S and the binary ground truth G for all image pixels (x, y), defined as:

3.2 Performance on ECSSD dataset

For the evaluation, we compare our work with 17 state-ofthe-art methods by running the approaches' publicly available source code: attention based on information maximization (AIM [Bruce and Tsotsos, 2005]), boolean map saliency (BMS [Zhang and Sclaroff, 2016]), context-aware saliency (CA [Goferman et al., 2010]), discriminative regional feature integration (DRFI [Jiang et al., 2013]), frequencytuned saliency (FT [Achanta et al., 2009]),global contrast saliency (HC and RC [Cheng et al., 2011]), high-dimensional color transform (HDCT [Kim et al., 2014]), hierarchical saliency (HS [Yan et al., 2013]), spatial temporal cues (LC [Zhai and Shah, 2006]), local estimation and global search (LEGS [Wang et al., 2015]), multiscale deep features (MDF [Li and Yu, 2015]), multi-task deep saliency (MTDS [Li et al., 2016]), principal component analysis (PCA [Margolin et al., 2013]), saliency filters (SF [Perazzi et al., 2012]), induction model (SIM [Murray et al., 2011]), saliency using natural statistics (SUN [Zhang et al., 2008]. Note that LEGS, MDF, and MTDS are deep learning based methods. The visual comparison of saliency maps generated from our method and different baselines are demonstrated in Figure 3. Our results are close to ground truth and focus on the main salient objects. As shown in Figure 4a,b , our work reaches the highest precision/recall rate over all baselines. As a result, our method also obtains the best performance in terms of F-measure.

As discussed in [Perazzi et al., 2012], neither the precision nor recall measure considers the true negative counts. These measures favor methods which successfully assign saliency to salient pixels but fail to detect non-salient regions over methods that successfully do the opposite. Thus they suggested that MAE is a better metric than precision recall analysis for this problem. As shown in Figure 4c, our work outperforms the state-of-the-art performance [Li and Yu, 2015] by 10%.

3.3 Performance on HKUIS dataset

Since HKUIS is a relatively new dataset, we only have 15 baselines. We first evaluate our methods using a precision/recall curve which is shown in Figure 5a, b. Our method outperforms all other baselines in both two settings, namely fixed threshold and adaptive threshold. As shown in Figure 5c, our method achieves the best performance in terms of MAE. In particular, our work outperforms other methods by a large margin, 25%.

3.4 Effectiveness of Explicit and Implicit Saliency Maps

We also evaluate the individual components in our system, namely, the explicit saliency map (EX), and the implicit saliency map (IM), in both ECSSD and HKUIS. As shown in Fig. 4 and Fig. 5, the two components generally achieve the acceptable performance (in terms of precision, recall, Fmeasure and MAE) which is comparable to other baselines. EX outperforms IM in terms of MAE, whereas IM achieves a better performance in terms of F-measure. When adaptively fusing them together, our unified framework achieves the state-of-the-art performance in every single evaluation metric. That demonstrates that these individual components complement each other in order to yield the good result.

3.5 Computational Efficiency

It is also worth investigating the computational efficiency of different methods. In Table 2, we compare the average running time for a typical 300 × 400 image of our approach to other methods. The average time is taken on a PC with Intel i7 2.6 GHz CPU and 8GB RAM with our unoptimized Matlab code. Performance of all the methods compared in this table are based on implementations in C++ and Matlab. Basically, C++ implementation runs faster than the Matlab based code. The CA method [Goferman et al., 2010] is the slowest one because it requires an exhaustive nearest-neighbor search among patches. Meanwhile, our method is able to run faster than other Matlab based implementations. Our procedure spends most of the computation time on semantic segmentation and extracting regional features.

4 Conclusion and FutureWork

In this paper, we have presented a novel method, semantic priors (SP), which adopts the semantic segmentation in order to detect salient objects. To this end, two maps are derived from semantic priors: the explicit saliency map and the implicit saliency map. These two maps are fused together to give a saliency map of the salient objects with sharp boundaries. Experimental results on two challenging datasets demonstrate that our salient object detection results are 10% - 25% better than the previous best results (compared against 15+ methods in two different datasets), in terms of mean absolute error.

For future work, we aim to investigate other sophisticated techniques for semantic segmentation with a larger number of semantic classes. Also, we would like to study the reverse impact of salient object detection into the semantic segmentation process.

Acknowledgements

Thanh Duc Ngo, Khang Nguyen from Multimedia Communications Laboratory, University of Information Technology are additional authors. We gratefully acknowledge the support of NVIDIA Corporation with the GPU donation.