Saliency Detection with a Deeper Investigation of Light Field

1 Introduction

Saliency detection, aiming at identifying the most salient regions or objects that most attract the viewers' visual attention in a scene, has become a popular area in computer vision. It plays an important role in recognition [Rutishauser et al., 2004; Han and Vasconcelos, 2014], image segmentation [Goferman et al., 2012], and visual tracking [Mahadevan and Vasconcelos, 2012].

Depending on how data are used, saliency detection can be classified into three main categories: 2D, 3D and light field saliency. The most common approach is to apply 2D salient cues such as intensity and color in contrast-based datadriven bottom-up saliency frameworks [Itti and Koch, 2001; Cheng et al., 2011; Perazzi et al., 2012; Jiang et al., 2013] owing to the observation that human vision is either particularly sensitive to high-contrast stimuli [Reynolds and Desimone, 2003], or incorporated with object/background priors in context-dependent top-down mechanisms [Wei et al., 2012; Yang et al., 2013; Zhu et al., 2014]. A comprehensive study on existing 2D saliency detection approaches is conducted by Borji et al. [2012]. Although these 2D models are based on visual attention mechanisms especially rooted in an early visual process in primary visual cortex (area V1) [Koch and Ullman, 1985; Itti and Koch, 2001; Li, 2002], most methods ignore important aspects of eye movements such as attention shifting across depth planes [Jansen et al., 2009].

With the availability of commercial 3D cameras such as Kinect [Zhang, 2012], another approach to saliency detection involves the saliency computation of RGBD images [Zhang et al., 2010; Lang et al., 2012; Desingh et al., 2013; Ciptadi et al., 2013; Peng et al., 2014; Ju et al., 2014]. In these methods, depth, as one of the feature dimensions, is more directly bound to objects thus beneficial for saliency analysis. However, most current methods require high-quality depth maps and ignore the relations between depth and appearance cues [Peng et al., 2014].

In recent years, the light field has opened up a new research area with the development of digital photography. Using consumer light field cameras such as Lytro [Ng et al., 2005] and Raytrix [Lumsdaine and Georgiev, 2009], we can simultaneously capture the total amount of light intensity and the direction of each ray from incoming light in a single exposure. Therefore, a light field can be represented as a 4D function of light rays in terms of their (2D spatial) positions and (2D angular) directions [Adelson and Wang, 1992; Gortler et al., 1996]. These information items can be converted into various interesting 2D images (e.g., focal slices, depth maps and all-focus images) through rendering and refocusing techniques [Ng et al., 2005; Tao et al., 2013]. For example, a light field image can generate a set of focal slices focusing at different depth levels [Ng et al., 2005] (see Figure 1(a) and 1(b) for examples), which suggests that one can determine background regions by selecting the slice focused on the background. Then the corresponding focusness maps can be computed to separate in-focus and our-of-focus regions so as to identify salient regions. An all-focus image (Figure 1(c)), containing a series of images captured at different focal planes, provides the sharpest pixels and can be approximately recovered by depth invariant blurs from the focal stack images [Nagahara et al., 2008] or the depth of field (DOF)-dependent rendering algorithm [Zhang et al., 2014]. In addition, the depth of each ray of light recorded in the sensor can be estimated by measuring pixels in the focus [Tao et al., 2013], as shown in Figure 1(d). Regions at closer depth ranges tend to be salient, while far in the distance mostly belong to backgrounds.

Li et al. [2014] pioneered the idea of light field for resolving traditionally challenging problems in saliency detection. They proposed a new saliency detection method tailored for light field by combining foreground and background cues generated from focal slices. In addition, they collected the first challenging dataset of 100 light fields using Lytro camera, i.e., the Light Field Saliency Dataset (LFSD). They demonstrated that the light field can greatly improve the accuracy of saliency detection within challenging scenarios. Although their work aims to explore the role of light field in saliency detection by using focusness to facilitate the saliency estimation, it is still at the initial stage of exploration and has some limitations: (1) the focusness and objectness are calculated to select foreground saliency candidates, which inevitably ignores the explicit use of depth data associated with salient regions/objects, and (2) the performance of existing 2D/3D saliency detection approaches and their extended versions with light field cues on the LFSD dataset is not well explored.

In this paper, we make an attempt to address saliency detection by the use of light field properties in the following aspects: (1) we generate focusness maps from focal slices at different depth levels via invariant blurs [Shi et al., 2014] and introduce a light field depth cue into saliency contrast computation within a L2-norm metric, (2) to facilitate the saliency estimation, we compute the background prior on the focusness map of a selected focal slice and incorporate it with the location prior, (3) we extend the 2D/3D state-of-the-art saliency detection approaches with our proposed light field depth contrast and focusness-based background priors, and show the effectiveness and superiority of light field properties, and (4) comparing with [Li et al., 2014], our approach increases the performance by 4%-7% on the LFSD dataset in terms of different quantitative measurements.

2 RelatedWork

2.1 2D Saliency

Results from many existing visual saliency approaches [Itti and Koch, 2001; Cheng et al., 2011; Perazzi et al., 2012; Jiang et al., 2013] indicate that the contrast is the most influential factor in the bottom-up visual saliency. For examples, Itti et al. [1998] proposed a saliency model that computes the local contrast from color, intensity and orientation. Cheng et al. [2011] defined the contrast by computing dissimilarities among color histogram bins of all image regions. To take global spatial relationships into account, Perazzi et al. [2012] considered saliency estimation as two Gaussian filters performing on region uniqueness and spatial distribution respectively. Other global methods such as appearance reconstruction [Li et al., 2013] and fully connected MRF [Jia and Han, 2013] are recently proposed to uniformly identify salient objects. Recently, background priors are incorporated into proposed methods to reduce the distraction of salient regions from backgrounds [Yang et al., 2013; Wei et al., 2012; Zhu et al., 2014]. Such approaches are generally based on some specific assumptions that all the image patches heavily connected to backgrounds and image boundaries belong to backgrounds [Yang et al., 2013; Zhu et al., 2014], are very large and homogeneous [Wei et al., 2012]. Additionally, Jiang et al. [2013] showed that the focus blur in 2D images can improve fixation predictions. Despite many recent improvements, the generations of accurate saliency maps are still very difficult in some challenging scenes, such as cluttered backgrounds, textured objects, and similar colors between salient objects and their surroundings, etc (see Figure 7 for some examples).

2.2 3D Saliency

Besides 2D information, several studies have exploited the depth cue in saliency analysis [Zhang et al., 2010; Lang et al., 2012; Desingh et al., 2013; Peng et al., 2014; Ciptadi et al., 2013; Ju et al., 2014]. Specifically, Zhang et al. [2010] designed a stereoscopic visual attention algorithm for 3D video based on multiple perceptual stimuli. Desingh et al. [2013] fused appearance and depth cues by using non-linear support vector regression. Ciptadi et al. [2013] demonstrated the effectiveness of 3D layout and shape features from depth images for calculating more informative saliency map. Ju et al. [2014] proposed a depth saliency method based on anisotropic center-surround difference, and used depth and location priors to refine the saliency map. Recently, a largescale RGBD salient object detection benchmark is built up with unified evaluation metrics and a multi-stage saliency estimation algorithm is proposed to combine depth and appearance cues [Peng et al., 2014]. The above approaches demonstrate the effectiveness of the depth in saliency detection, while their performance is highly dependent on the quality of depth estimation. All these methods may fail when salient objects cannot be distinguished at the depth level.

2.3 Light Field Saliency

To the best of our knowledge, Li et al.'s work [2014] is the first saliency detection method by using light field data that shows that light field can greatly improve the accuracy of saliency detection. They used focusness priors to extract background information and computed the contrastbased saliency between background and non-background regions. In addition, the objectness is computed as the weight for combining contrast-/focusness-based saliency candidates to generate the final saliency map.

3 Approach

Figure 2 shows the pipeline of our approach, and the details are described in the following sections.

3.1 Light Field Contrast-based Saliency

We build the contrast-based saliency based on the light field depth and color from the all-focus image. We employ Simple Linear Iterative Clustering (SLIC) algorithm [Achanta et al., 2012] to segment an all-focus image into a set of nearly regular super-pixels, which can preserve edge consistency and yield compact super-pixels. We define the contrast saliency S(pi) for super-pixel pi as:

where N is the total number of super-pixels and we found that 300 super-pixels are enough to obtain high performance for saliency detection. Ufea(pi) and Ufea(pj) are average feature (depth or color) values of super-pixels pi and pj. Wpos(pi, pj) is the spatial weight factor for controlling the pair-wise distance of super-pixels, that is, closer regions or similar colors would have higher contribution to the saliency. ||U ∗ pos (pi) - U ∗ pos (pj)|| defines the L2-norm distance of normalized average coordinates between pairs of super-pixels pi and pj and measures spatial relationships of super-pixels, σw is specified as 0.67 throughout our experiments.

Based on the above definitions, we denote the depthinduced contrast saliency from the light field as SD(pi), which is useful for separating the salient foreground from the similar or cluttered background (e.g., the 1st example in Figure 4(c)). However, saliency detection with the depth contrast may fail when the background is close to the foreground object or the salient object is placed in the background, as shown in the 2nd example of Figure 4(c). We solve this issue by considering the color contrast as a complementary prior. Therefore, we compute the color contrast saliency SC(pi) in CIE LAB color space on the all-focus image.

Then, we combine the depth saliency and color saliency as:

where α and β are two weight parameters for leveraging depth and color cues with β = 1 - α. We empirically set them as α = 0.3 and β = 0.7.

3.2 Background Priors Encoded by Focusness

Similar with [Li et al., 2014], we select the background slice Ibg through analyzing focusness distributions of different focal slices Ik, k = 1, ..., K. More specifically, we compute the focusness map Fk for each focal slice using focusness detection technique [Shi et al., 2014].

We compute the background likelihood score Bk for each slice Ik along Fk(x) and Fk(y) by U-shaped filtering, and choose the slice with the highest Bk as the background slice Ibg ,

filtering function along the x axis, and η = 28 controls the bandwidth.

To enhance the saliency contrast, we compute the background probability Pbbg on the focusness map Fbg through:

where σbg = 1, Ubg(pi) is the average value of super-pixel pi on the focusness map Fbg. ||C -U ∗ pos (pi)|| measures the spatial information of super-pixels related to the image center C, here, U ∗ pos (pi) defines normalized average coordinates of super-pixel pi. Therefore, regions that belong to the background have higher background probability Pbbg on the focusness map.

3.3 BackgroundWeighted Saliency Computation

We incorporate the background probability into the contrast saliency as follows:

It can be seen that the saliency value of a foreground region is increased by multiplying a high Pbbg from background regions. On the contrary, the saliency value of background regions is reduced by multiplying a small P bbg from foreground regions.

In order to obtain cleaner foreground objects, we applied saliency optimization algorithm [Zhu et al., 2014] onto the above saliency map (Eq. 6). We found that the addition of this optimization procedure consistently increases the performance of the proposed approach by about 2% for MAE, 5% for F-measure, and 6% for AUC.

4 Experiments

We conduct extensive experiments to demonstrate the effectiveness and superiority of our proposed approach.

4.1 Experimental Setup

Dataset

Light Field Saliency Dataset (LFSD) 1 is the only reported dataset captured by Lytro camera for saliency analysis, which contains 100 light fields acquired in 60 indoor and 40 outdoor scenes. The light field data of a scene are composed of 5 types of image data - the raw light field data, a focal slice, an all-focus image derived from the focal stack, a rough depth map, and the corresponding 2D binary ground truth (GT).

Evaluation Measures

We use standard precision recall (PR) and receiver operating characteristic (ROC) curves for evaluations. When computing the overall quality on the whole dataset, we consider three metrics for determining the accuracy of saliency detection: Fmeasure, area under curve (AUC), and mean absolute error (MAE).

Given a saliency map, a PR curve is obtained by generating binary masks with a threshold t ∈ [0, 255] and comparing these masks against the GT to obtain precision and recall rates. The PR curves are then averaged over the dataset. We compute F-measure as Fβ = (1+β2 )·P·R β2 ·P+R , and set β2 = 0.3 to highlight precision [Achanta et al., 2009]. The ROC curve can also be generated based on true and false positives obtained during the calculation of PR curve. MAE measures the average per-pixel difference between the binary GT and the saliency map, which is found complementary to PR curves [Perazzi et al., 2012; Cheng et al., 2013].

4.2 Results

Evaluating the Different Light Field Properties

To assess the different light field properties in the proposed approach, we show comparisons of the accuracy using different model components. Here we focus on different light field properties themselves regardless of saliency optimization [Zhu et al., 2014], .

Figure 3 shows PR and ROC curves of saliency detection. It can be seen that light field properties complement each other and none of them alone suffices to achieve good results. When linearly combining the depth and color contrast saliency, our approach outperforms the versions with individual ones, suggesting its ability to leverage diversified light field cues. Furthermore, the performance is significantly improved by computing the background probability from the focusness which is additional light field support of our approach.

1 http://www.eecis.udel.edu/∼nianyi/LFSD.htm

Table 1 shows F-measure, ROC and MAE results for comparisons. There is a consistent improvement in performance for all the metrics, which further validates the effectiveness of our approach for light field saliency detection.

Figure 4 visually compares the performance of different light field properties. We observe that each of cues has its unique advantage to saliency detection in different ways. Depth cue is exploited to detect foreground salient objects. However, it may fail when the depth contrast is low or salient object is placed in the background, e.g., the 2nd example. In this example, the color cue from the all-focus image can be used to distinguish salient and non-salient colors in the entire scene. Further, the focusness at different depth levels is beneficial for efficient foreground and background separation. The 3rd and 4th examples show that background priors encoded by light field focusness are helpful to eliminate the background distraction and enhance salient foreground objects.

Extending 2D Saliency Models with Light Field Depth-induced Saliency

To validate the benefit of light field depth, we extend 8 stateof-the-art 2D saliency approaches by fusing 2D saliency maps with light field depth contrast saliency maps into final ones through the standard pixel-wise summation. These methods include Tavakoli [Tavakoli et al., 2011], CNTX [Goferman et al., 2012], GS [Wei et al., 2012], SF [Perazzi et al., 2012], TD [Scharfenberger et al., 2013], CovSal [Erdem and Erdem, 2013], GBMR [Yang et al., 2013], and wCtr* [Zhu et al., 2014]. We set the models all with default parameters in their original implementations.

Figure 5 presents the PR and ROC curves of our results. The comparisons of F-measure, ROC and MAE are given in Table 2. Here the postfix ' D' denotes depth-extended saliency methods. We can see that our approach is superior to all the state-of-the-art 2D models even combined with the light field depth-induced saliency. The accuracy from all the 2D saliency methods are improved by incorporating the light field depth saliency by about 1-5% and 3-6% for F-measure and MAE, respectively. It is worth to note that we obtain the significant improvement for CNTX by 10% in the AUC metric.

Figure 6(c)-6(j) show qualitative comparisons of all the 2D saliency methods (Top) and their depth-extended versions (Bottom) for two examples. It is obvious that most approaches fail when the object has the similar appearance as the background or the background is cluttered. However, the inclusion of the light field depth contrast helps to capture homogeneous color elements and subtle textures within the object so as to identify foreground salient objects.

We also visually compare our approach (Figure 7(n)) with all the depth-extended approaches (Figure 7(c)-7(j)). Benefiting from the combination of color and depth contrast with background priors, our approach still efficiently works when the background is not distant enough or the salient object is not distinct at the depth level.

Extending 3D Saliency Models with Light Field Focusness-induced Background Priors

In order to show the role of light field focusness, we incorporate background priors (Eq. 5) computed from focusness maps into 2 state-of-the-art 3D saliency models: DVS [Ciptadi et al., 2013] and ACSD [Ju et al., 2014].

Similarly, the quantitative results are shown in Figure 8 and Table 2. The postfix ' Bg' indicates the methods extended with background priors from the light field focusness. Overall, the background prior encoded by the focusness cue improves original 3D saliency detection, which can also be seen in Figure 6(k) and (l) for visual comparisons. Apparently,

(a)

(b)

(c)

(d)

(e)

(f )

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

Figure 8: Quantitative results of our approach, LFS, state-ofthe-art 3D approaches and their focusness-extended versions. (a) PR curves; (b) ROC curves.

DVS pays more attention on object contours while ignores inner salient regions. ACSD is the second best approach in our comparisons because they also consider the depth contrast and location cue which is very beneficial for saliency detection. Additionally, Figure 7 visually shows that our approach performs better than these extended approaches (Figure 7(k) and (l)).

Comparison with the LFS Method

LFS [Li et al., 2014] is so far the only reported approach for saliency detection on the light field data. As can be seen from Figure 8, our approach achieves much better performance than LFS at the higher precision rates in PR curves and the lower false positive rates in ROC curves. Figure 7(m) and (n) show the qualitative comparisons of LFS and our approach. Our method locates foreground objects more accurately. This is mainly because the depth contrast saliency provides us strong salient cue in cluttered background or the background having the similar colors with the foreground. Besides, background regions are falsely detected as salient objects by LFS in some cases while ours not (see the 5th, 7th and 11th examples). The possible reason could be that our approach makes an appropriate use of focal slices to introduce the background probability on the focusness map, which is beneficial for saliency detection when salient objects cannot be distinguished at the depth level.

5 Conclusions

The light field camera allows one to capture the total amount of light intensity and the direction of each ray from incoming light in a single exposure simultaneously, which provides not only intensity information, but also depth and focusness information. These interesting properties of the light field motivated us to investigate its capabilities for visual saliency. In this paper, we proposed a new saliency detection approach using light field focusness, depth and all-focus cues. Our approach produced state-of-the-art saliency maps on the LFSD dataset. Through extensive evaluations, we showed that various 2D approaches supported by our light field depth-induced saliency improved their accuracy of saliency detection, and by considering different focus areas from the light field, backgrounds are easily separated from foregrounds, which is beneficial for 3D saliency detection. Compared with [Li et al., 2014], our approach achieved substantial gains in accuracy. However, the depth reliefs of the light field are limited in the current cameras. In future work, we are interested in making use of light field depth and focusness to predict gaze shifts in real 3D scenes and estimating more accurate depth maps from light field cameras with large depth reliefs to improve visual saliency applications.

Acknowledgments

This work was supported by the Natural Science Foundation of China (61272393, 61229301, 61322201, 61403116), the National 973 Program of China (2013CB329604, 2014CB34760), the Program for Changjiang Scholars and Innovative Research Team in University (PCSIRT) of the Ministry of Education, China (IRT13059), the Program for New Century Excellent Talents in University (NCET-12-0836), the China Postdoctoral Science Foundation (2014M560507), and the Fundamental Research Funds for the Central Universities (2013HGBH0045).