Detecting Robust Co-Saliency with Recurrent Co-Attention Neural Network

1 Introduction

In practice, the frequently occurring patterns or the prime foregrounds can be utilized to represent the main content of the image group. The co-saliency task is derived with the goal of discovering the common and salient objects in a group of related images. Unlike traditional saliency detection [Li et al., 2019; Hou et al., 2019; Wang et al., 2018; Zhang et al., 2017b], co-saliency explore more synergetic information from multiple images. It endows many high-level computer vision systems with the capability to fixate their attention on the most valuable information in the image group, such as image co-segmentation [Chang et al., 2011], object co-localization [Xue et al., 2013], and video foreground extraction [Fu et al., 2015].

∗Correspondence should be addressed to Zhengxing Sun and Jinlong Shi

As a fundamental but challenging research topic, to detect co-saliency accurately, there are two key issues must be concerned: i) extract and learn effective feature representations of images in the group; ii) model the synergetic relationship among the co-salient regions at the group level to generate the final co-saliency maps. Conventional approaches [Fu et al., 2013; Liu et al., 2014] utilize handcrafted features, such as color, texture and SIFT descriptors etc., and these methods rely on researcher's prior knowledge to model the interaction between the group images, like inter-image saliency. However, low-level features and fixed hand-designed interaction models are too subjective to face the multiple challenges including background clutter, appearance variance of co-salient object across images, and similarity between co-object and non-common object, etc. Deep learning has recently emerged and demonstrated success in many computer vision applications. Recent researches use deep visual features to improve co-saliency detection and they also try to learn more robust co-salient properties among images in a data driven manner. However, as feature extraction and co-saliency detection are treated as separate steps in these approaches [Zhang et al., 2016; Zhang et al., 2017a; Li et al., 2017; Zheng et al., 2018], they can't customize features for better inferring co-salient regions, leading to suboptimal performance. For robust co-saliency detection, the feature representation should not only express the individual properties of each image, but also contain the relevance and interaction among group images. Recently, [Wei et al., 2017] integrated feature learning and detection as a whole process and proposed an end-to-end deep learning co-saliency detection method. The designed group-wise representation can capture the interaction among group images and improve the results, which demonstrates its importance for co-saliency detection.

As the first end-to-end co-saliency network, [Wei et al., 2017] had made a remarkable success, however in real-world robust co-saliency detection this kind of network architecture suffers from several drawbacks. i) Their end-to-end network architecture requires constant input data, so they construct the network based on the assumption that there are a constant number of images in each group, which are set to be 5 in their work. However, the size of each group is not fixed in real-world scenarios as well as the experimental cosaliency dataset. When encountering the group with various sizes, they either can't make the detection or only can capture incomplete synergetic relationships with partial group-wise features, limiting the robustness of the model. ii) As there is unpredictable appearance and location variance of the cosalient object across images, their group-wise representation which is based on the direct concatenation of individual image features contains much noise information of un-salient or non-common regions, leading to unsatisfying results.

In this paper, in order to detect robust co-saliency, we propose a novel deep learning approach with recurrent coattention network. Our aim is to make use of all available information including individual image properties as well as the group synergetic information to create a robust and effective co-saliency network. Specifically, our network will first extract the semantic features of all images, then will be processed by two branches. The single image representation branch processes each image individually to learn the unique properties. Since there are an unfixed number of images in each group and the appearance as well as the location of co-salient object varies across images, we specially design a novel Recurrent Co-Attention Unit (RCAU) to learn the group information. By using the spatial and channel coattention between images, the RCAU recurrently explores all images in the group to gradually learn the robust group representation and meanwhile suppresses noisy information. The group feature is then broadcasted to each individual image, which allows the network to leverage the synergetic information and unique properties between the images. So the complementarity and interaction of group and single representation are sufficiently exploited to facilitate the robust cosaliency reasoning. Moreover, to make full use of the interactive relationships of whole images in the training group, inspired by [Hsu et al., 2018], we further propose a novel coperceptual loss as the additional supervision in our end-to-end training process. As far as we know, this is the first attempt to introduce the recurrent architecture into deep co-saliency detection, which allows the proposed method to make full use of all images information in various size groups without retraining the model or introducing extra parameters. Extensive experimental results demonstrate the superiority of our approach in robust co-saliency detection as compared to the state-of-the-art methods.

2 Proposed Approach

2.1 ProblemFormulation

Co-saliency detection aims at discovering the common and salient objects in a group of N relevant images I = {In}Nn=1 . The co-saliency maps M = {Mn}Nn=1 are produced by a cosaliency detection model:M = F (I; Θ), (1)

where F() is the model function that takes an image group as input and outputs a group of co-saliency maps simultaneously. Θ represents model parameters which are optimized by an end-to-end learning scheme in this work. The core idea of this work is trying to make full use of all available information to learn the effective feature representations which can not only express the image's individual properties, but also reflect the interaction among group images for robust co-saliency referring. The overall architecture of the proposed approach is illustrated in Figure 1. For an input image group with an arbitrary size, our network will first extract the semantic features of all images. Then the single image representation (SIR) branch processes each image individually to learn the unique properties. Meanwhile the Recurrent Co-Attention Unit (RCAU) recurrently explores all images in the group to learn the robust group representation. The two branches are later merged for the final co-saliency detection.

2.2 Single Image Representation

As a basic rule in co-saliency, in most cases, the co-salient regions should be salient with respect to the background in each image. So, it is important to learn the unique properties of each image to capture the saliency of the potential co-salient objects in the individual image. In the proposed approach, for each image In in the input group I we first use a pre-trained convolutional neural network (CNN) to extract its semantic features Xn ∈ RH×W×C . Then we construct an SIR block with 3 convolutional layers to encode the individual properties for each image S = {Sn}Nn=1 , which is defined as follows:

where ΘS are the parameters learned from the convolutional process fS.

2.3 Group Representation with RCAU

As images within a co-saliency group are contextually associated with each other in different ways such as common objects, similar categories, and related scenes, learning a robust group representation which contains the relevance and interaction between group images is extremely important for co-saliency referring. In this work, we proposed to use a recurrent architecture to learn the group representation GN for an arbitrary size group I. It is defined as follows:

where ΘG are the parameters learned from the recurrent convolutional process fG. Since there is much noise information of non-salient or non-common regions in the group and the appearance as well as the location of co-salient object varies across images, we specifically design a novel recurrent unit RCAU to gradually explore all the synergetic relationships between images for the group representation. As illustrated in Fig. 2(a), for step n, RCAU takes two inputs: the current image feature map Xn and the group representation Gn-1 of all the images just been explored. In the first step, G0 is initialized with X1. We construct two gates in our RCAU, the reset gate gd is used for denoising current image and the update gate gz is used to decide how to update current group representation Gn.

Since all images in the group share the common objects, we want to use the synergetic relationships between the explored images and the current image to suppress the noise data in current image, like the non-salient background and non-common regions. The reset gate D is defined as:

(4)

Then the denoised feature map ˜Xn is computed as

As the appearance and the location of co-salient object varies across images, we want to fully explore the spatialchannel-wise variation of the co-salient object with coattention mechanism to determine what group information should be retained in Gn-1 and what new information should

be updated from ˜Xn. So Zn = (Gn-1 -Xn ) is used as an input of gz to model the cross-images variation in each step. The update gate Z is defined as follows,

Fig.2(b) illustrates the structure of update gate model gz.

For spatial attention model, we first use a global crosschannel average pooling layer to get the overall response in each spatial position.Then two FC layers (the first layer is followed by ReLU) are applied to generate the spatial attention maps Zs ∈RH×W×1 . It is formulated as

where Z H,W n ∈ RH,W is the result of Zn after cross-channel average pooling.

For channel attention model, we first introduce a global spatial space average pooling layer to get overall response of each channel. Then a FC layer is applied to get the channel attention maps Zc ∈ R1×1×C , which is formulated as

where Z C n ∈ RC is the result of Zn after global spatial space average pooling.

The overall attention maps of current input feature are the product of spatial attention maps and channel attention maps.

After a sigmoid operation, the overall attention maps are normalized into the range between 0 and 1, formulated as

where Z ∈ RH×W×C . Then the RCAU updates Gn by

The value of each position in Z denotes the probability for the activation value in corresponding position of Gn-1 to be reserved and position of ˜Xn to be updated. Higher probability value indicates that the update gate model gz considers that last group feature Gn-1 in this location has high quality group information and should be reserved. While locations with lower probability means there is new useful synergetic information in ˜Xn should be updated to Gn. Unlike the conventional recurrent units (GRU or LSTM), our RCAU can use co-attention to recurrently learn robust group representation for co-saliency referring meanwhile reducing the noise data. We will provide justification on this issue in the later experiments section.

2.4 Detect Co-saliency with Robust Representation

As described previously, the group feature is then broadcasted to each individual image, which allows the network to leverage the synergetic information and unique properties between the images. So the complementarity and interaction of group and single representation are sufficiently exploited to facilitate the robust co-saliency reasoning. We concatenate each Si with GN to get final fused features, and to alleviate the aliasing effect of upsampling, we add a 3 × 3 convolutional layer after merging operations. So the co-saliency maps M can be obtained with the final fused features by applying 3 × 3 convolutional layers and deconvolutional layers followed with a sigmoid activation function. It is formulated as:

3 Loss Function

Let I = {In}Nn=1 and their groundtruth {G T n }Nn=1 denote a collection of training samples where N is the number of images. After co-saliency detection, co-saliency maps are {Mn}Nn=1 . We use the sigmoid cross-entropy loss as the individual supervision for each image In:

To fully explore the group information, inspired by the wildly used perceptual loss in Neural Style Transfer (NST) [Gatys et al., 2016] works, we proposed a novel coperceptual loss to model the synergetic relationships between the co-salient objects in training images as an additional supervision. As defined in NST, the core idea of feature perceptual loss is to seek the consistency of two images between the hidden representations in a pre-trained CNN φ [Simonyan and Zisserman, 2014]. In this work, for a image In, we can generate two masked images with its co-saliency map Mn and groudtruth G T n

where ⊗ denotes element-wise multiplication. The masked image I on means our detected co-salient regions of In while image I G n means the real co-salient regions of In. Then we apply the extractor φ to all masked images {I on , I G n }Nn=1 and obtain their corresponding hidden representations {φ(I on ) ∈ RV , φ(I G n ) ∈ RV }Nn=1 , where V is the feature dimension. So, in the training process, we construct the co-perceptual loss for image In by measuring the similarity between its detected region I on and the real co-salient regions of all the rest images in the group I Gm ( Im ∈ I and m ̸= n ). The coperceptual loss function corresponds to the squared Euclidean loss term and is defined by

Note that all parts of our network are trained jointly, and the over all loss function is given as

where λ is the tradeoff parameter and Θ = {ΘS, ΘG, ΘC} is the all learnable parameters set of our network.

4 Experiments

4.1 Experimental Setup

Datasets

We evaluated the proposed approach on three public benchmark datasets: iCoseg [Batra et al., 2010], MSRC [Winn et al., 2005] and Cosal2015 [Zhang et al., 2016]. iCoseg consists of 38 groups of total 643 images which are challenging for co-saliency detection task because of the complex background and multiple co-salient objects, and the group size varies from 4 to 42. MSRC contains 7 groups of total 240 images, and each group has 30 ∼ 53 images. Cosal2015 is a more recent dataset and it has 50 groups and a total of 2015 images, each group containing 26 ∼ 52 images. Compared with iCoseg, MSRC and Cosal2015 have relative larger group size and are more challenging with various co-salient objects poses and sizes, greater appearance variations and even more complex backgrounds.

Implementation Details

We select the widely used pre-trained VGG 16-layer net [Simonyan and Zisserman, 2014] (over the MS COCO dataset [Lin et al., 2014]) as the backbone network to extract the semantic features Xn for each image. The deconvolutional layers are initialized with simple bilinear interpolation parameters. For the sake of fair comparison, we following the same setting in [Wei et al., 2017; Zheng et al., 2018], the training groups are randomly selected from a subset of COCO dataset (which has 9213 images with pixelwised ground-truth) using the global similarity (Gist and Lab color histogram features), and then fine-tune the model by randomly selecting 50%-50% training-test images for three datasets. All images and groundtruth maps are resized to 224 × 224. The proposed models are optimized by standard SGD in which the momentum parameter is chosen as 0.99, the learning rate is set to 1e-5, and the weight decay is 0.0005. We need about 50000 training iterations for convergence. And for co-perceptual loss, we follow the setting in Neural Style Transfer and use the activated layers Relu3 1, Relu4 1, Relu5 1 of VGG as the hidden representation. And the loss tradeoff parameter λ is set to be 0.1 in our work.

Evaluation Metrics

To evaluate the performance of the proposed method, four widely-used metrics are adopted: (1) Precision-Recall (PR) curve, which shows the tradeoff between precision and recall for different threshold (ranging from 0 to 255). (2)

F-measure (Fβ) denotes the harmonic mean of the precision and recall values obtained by a self-adaptive threshold T = µ + σ (µ and σ are the mean value and standard deviation of co-saliency map.) The Fβ is computed by Fβ = (1+β2 )×Precision×Recall β2×Precision+Recall , where β2 is typically set to 0.3 as suggested in [Han et al., 2018; Yang et al., 2013]. (3) Mean Absolute Error (MAE), which characterize the average 1-norm distance between ground truth maps and predictions. (4) Structure Measure (Sα) is adopted to evaluate the spatial structure similarities of saliency maps based on both region-aware structural similarity Sr and object-aware structural similarity So, defined as Sα = α ∗ Sr + (1 - α) ∗ So, where α = 0.5 [Fan et al., 2017].

4.2 Comparison to the State-of-the-Arts

In order to evaluate the effectiveness of the proposed method, we compare it against 10 state-of-the-art algorithms. CSHS and CBCS are two conventional co-saliency approaches based on hand-crafted features that are widely compared in literatures. CoDW, SPMI, FASS and UCSG are 4 co-saliency methods using the deep learning features. GW is an end-toend deep learning method for co-saliency detection. We also compare our method with 3 end-to-end single saliency methods which are DCL, Amulet and DSS. The experimental results are shown in Table 1. For fair comparison, we use either the implementations with recommended parameter settings or the saliency maps provided by the authors. As shown in Table 1, our approach outperforms all the state-of-the-art methods significantly in terms of all the metrics on three datasets. For example, for dataset Cosal2015, our method improves upon the second best algorithm FASS by about 8%, 4.1% and 30.3% in terms of F-measure, Structure Measure and MAE respectively. Note that although GW also had leveraged group feature in their work, our approach still exceeds GW by 13.6%, 12.1% and 46.8% in these three metrics. The same conclusion can be obtained from PR curves in Figure 4, the proposed method outperforms the state-of-the-arts by a large margin.

Figure 5 shows some sample co-saliency maps produced by the proposed approach and the state-of-the-art methods. From the results on Cosal2015 dataset, it can be observed that traditional method CSHS can hardly find common salient areas in complex cases of background clutter and crossimages variations. While the deep learning based singleimage saliency method DSS and co-saliency methods UCSG and CoDW can better find the salient regions. But because of lacking group representation learning process, they all can't well handle the similarity between co-objects and noncommon objects, resulting in undesired regions in co-saliency maps, like the white soccer players in iCoseg. The end-to-end deep learning method GW performs significantly better than others with less noise regions. However, our method produces the best saliency maps both in terms of the accuracy of contours and discrimination of different objects since we fully explore the single image properties and the robust group representation by RCAU and co-perceptual loss. Moreover, our results are more assertive than those of the others, which helps to easily select a binarization threshold to segment out the foregrounds given a co-saliency map.

4.3 Ablation Studies

In this section, we conduct evaluation on MSRC dataset to investigate the effectiveness of various components of the proposed model. The MSRC dataset has relative larger operable group size and is more challenging with various co-salient objects poses and sizes, greater appearance variations and complex backgrounds. The results are shown in Table 2. We set the baseline approach by only using single feature learning branch and training it with the cross-entropy loss Ls alone.

As can be seen, the baseline approach can't well handle the co-saliency task. After applying the RCAU, with the group representation, the performance of baseline approach is improved by 7.6%, 7.7% and 36.5% in terms of F-measure, Structure Measure and MAE respectively. This shows that the group representation is essentially crucial for co-saliency detection. When we replace our RCAU with the conventional recurrent units GRU, the performance drops a lot. That means our RCAU is more effective to capture the synergetic information in group compared with conventional recurrent units. However, the GRU still improves the performance of baseline, which means the recurrent architecture is naturally suitable for co-saliency detection task. We then add the coperceptual loss Lc to form the completed version of our approach. The co-perceptual loss helps to further improve the performance comparing with RCAU version by 2.5%, 1.1% and 5% in terms of F-measure, Structure Measure and MAE. This indicates that the co-perceptual supervision can better model the synergetic relationships between the co-salient objects and help the network to learn more effective group representation, which in turn boost the co-saliency detection.

We evaluated the effects of RCAU with different component settings. As shown in Table 2, for update gate gz, the combination of Zs and Zc achieves better performances than using them alone. When we remove the reset gate gd from RCAU, the performance declines on three metrics especially on MAE. This indicates the reset gate gd is able to suppress the noise information in the group. In order to verify that our recurrent architecture can handle different size image groups, we construct new testing groups by randomly selecting a sub-group with different size 5, 10 and 15 from the original groups. As reported in Table 2, our approach achieves good performance on different size groups and still consistently outperforms all the state-of-the-art methods. And the performance raises along with the group size, which emphasizes the importance of the group information completeness to robust co-saliency detection. To further justify the denoising ability of our RCAU, we add a noise image which is randomly selected from COCO dataset to the testing groups in size 5 and 10. As shown in results, although the noise data damages our performance a little, we still outperforms all the state-of-the-art methods, which demonstrate the robustness of the proposed method. As for other methods like GW, their performance declines badly under the influence of noise data.

5 Conclusion

In this paper, we propose a novel end-to-end deep learning approach with the recurrent co-attention network for robust co-saliency detection. The proposed approach explores single image properties and robust group representation simultaneously, which are essentially crucial for co-saliency detection. By using the spatial and channel co-attention between images, the special designed RCAU recurrently explores all images in the group to learn the robust group representation and meanwhile suppresses noisy information. The group feature is then broadcasted to each individual image for boosting the inferring of co-saliency. Moreover, we propose a novel coperceptual loss to make full use of interactive relationships of co-salient objects in the training group as the additional supervision. The whole modules are collaboratively optimized in an end-to-end manner, further improving the robustness of the approach. Extensive experimental results demonstrate the superiority of our approach.

Acknowledgments

This work was supported by National High Technology Re-search and Development Program of China (No. 2007AA01Z334), National Natural Science Foundation of China (Nos. 61321491 and 61272219), National Key Research and Development Program of China (Nos.2018YFC0309100, 2018YFC0309104), Program for New Century Excellent Talents in University of China (NCET-04-04605), The China Postdoctoral Science Foundation (Grant No. 2017M621700), National key research and development program of China (Nos.2018YFC0309100, 2018YFC0309104), and Innovation Fund of State Key Laboratory for Novel Software Technology (Nos. ZZKT2013A12ZZKT2016A11 and ZZKT2018A09).