{"type":"ScholarlyArticle","title":"Detecting Robust Co-Saliency with Recurrent Co-Attention Neural Network","authors":[{"name":"Bo Li","slug":"bo-li","given":"Bo","family":"Li","affiliation":"State Key Lab for Novel Software Technology , Nanjing University , Nanjing , China"},{"name":"Zhengxing Sun","slug":"zhengxing-sun","given":"Zhengxing","family":"Sun","affiliation":"State Key Lab for Novel Software Technology , Nanjing University , Nanjing , China"},{"name":"Lv Tang","slug":"lv-tang","given":"Lv","family":"Tang","affiliation":"State Key Lab for Novel Software Technology , Nanjing University , Nanjing , China"},{"name":"Yunhan Sun","slug":"yunhan-sun","given":"Yunhan","family":"Sun","affiliation":"Jiangsu University of Science and Technology Zhenjiang , China"},{"name":"Jinlong Shi","slug":"jinlong-shi","given":"Jinlong","family":"Shi","affiliation":"Jiangsu University of Science and Technology Zhenjiang , China"}],"abstract":"Effective feature representations which should not only express the images individual properties, but also reflect the interaction among group images are essentially crucial for robust co-saliency detection. This paper proposes a novel deep learning co-saliency detection approach which simultaneously learns single image properties and robust group feature in a recurrent manner. Specifically, our network first extracts the semantic features of each image. Then, a specially designed Recurrent Co-Attention Unit (RCAU) will explore all images in the group recurrently to generate the final group representation using the co-attention between images, and meanwhile suppresses noisy information. The group feature which contains complementary synergetic information is later merged with the single image features which express the unique properties to infer robust co-saliency. We also propose a novel co-perceptual loss to make full use of interactive relationships of whole images in the training group as the supervision in our end-to-end training process. Extensive experimental results demonstrate the superiority of our approach in comparison with the state-of-the-art methods.","doi":"10.24963/ijcai.2019/115","url":"https://ijcai.org/proceedings/2019/115","pdf":"https://ijcai.org/proceedings/2019/0115.pdf","markdown":"https://ijcai.org/proceedings/2019/115.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/115","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Computer Vision","track":"Main track","number":115,"pages":{"first":818,"last":825},"keywords":["Computer Vision: Perception","Computer Vision: Computer Vision","Computer Vision: Recognition: Detection, Categorization, Indexing, Matching, Retrieval, Semantic Interpretation"],"videos":[],"references":[{"position":1,"raw":"[Batra et al., 2010] Dhruv Batra, Adarsh Kowdle, Devi Parikh, Jiebo Luo, and Tsuhan Chen. icoseg: Interac- tive co-segmentation with intelligent scribble guidance. In CVPR, pages 3169-3176, 2010.","title":"icoseg: Interactive co-segmentation with intelligent scribble guidance","authors":"Dhruv Batra, Adarsh Kowdle, Devi Parikh, Jiebo Luo, Tsuhan Chen","year":"2010","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chang et al., 2011] Kai-Yueh Chang, Tyng-Luh Liu, and Shang-Hong Lai. From co-saliency to co-segmentation: An efficient and fully unsupervised energy minimization model. In cvpr, pages 2129-2136, 2011.","title":"From co-saliency to co-segmentation: An efficient and fully unsupervised energy minimization model","authors":"Kai-Yueh Chang, Tyng-Luh Liu, Shang-Hong Lai","year":"2011","venue":"cvpr","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Fan et al., 2017] Deng-Ping Fan, Ming-Ming Cheng, Yun Liu, Tao Li, and Ali Borji. Structure-measure: A new way to evaluate foreground maps. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pages 4558-4567, 2017.","title":"Structure-measure: A new way to evaluate foreground maps","authors":"Fan","year":"2017","venue":"IEEE International Conference on Computer Vision, ICCV 2017","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Fu et al., 2013] Huazhu Fu, Xiaochun Cao, and Zhuowen Tu. Cluster-based co-saliency detection. IEEE Trans. Im- age Processing, 22(10):3766-3778, 2013.","title":"Cluster-based co-saliency detection","authors":"Fu","year":"2013","venue":"IEEE Trans. Image Processing","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Fu et al., 2015] Huazhu Fu, Dong Xu, Bao Zhang, Stephen Lin, and Rabab K. Ward. Object-based multiple fore- ground video co-segmentation via multi-state selection graph. IEEE Trans. Image Processing, 24(11):3415-3424, 2015.","title":"Object-based multiple foreground video co-segmentation via multi-state selection graph","authors":"Huazhu Fu, Dong Xu, Bao Zhang, Stephen Lin, Rabab K Ward","year":"2015","venue":"IEEE Trans. Image Processing","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gatys et al., 2016] Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In CVPR, pages pages 2414-2423, 2016.","title":"Image style transfer using convolutional neural networks","authors":"Leon A Gatys, Alexander S Ecker, Matthias Bethge","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Han et al., 2018] Junwei Han, Gong Cheng, Zhenpeng Li, and Dingwen Zhang. A unified metric learning-based framework for co-saliency detection. IEEE Trans. Circuits Syst. Video Techn., 28(10):2473-2483, 2018.","title":"A unified metric learning-based framework for co-saliency detection","authors":"Han","year":"2018","venue":"IEEE Trans. Circuits Syst. Video Techn","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hou et al., 2019] Qibin Hou, Ming-Ming Cheng, Xiaowei Hu, Ali Borji, Zhuowen Tu, and Philip Torr. Deeply su- pervised salient object detection with short connections. IEEE TPAMI, 2019.","title":"Deeply supervised salient object detection with short connections","authors":"Hou","year":"2019","venue":"IEEE TPAMI","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hsu et al., 2018] Kuang-Jui Hsu, Chung-Chi Tsai, Yen-Yu Lin, Xiaoning Qian, and Yung-Yu Chuang. Unsupervised cnn-based co-saliency detection with graphical optimiza- tion. In Computer Vision -ECCV 2018 -15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part V, pages 502-518, 2018.","title":"Unsupervised cnn-based co-saliency detection with graphical optimization","authors":"Kuang-Jui Hsu, Chung-Chi Tsai, Yen-Yu Lin, Xiaoning Qian, Yung-Yu Chuang","year":"2018","venue":"Computer Vision -ECCV 2018 -15th European Conference","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li and Yu, 2016] Guanbin Li and Yizhou Yu. Deep contrast learning for salient object detection. In 2016 IEEE Confer- ence on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 478- 487, 2016.","title":"Deep contrast learning for salient object detection","authors":"Yu ; ] Guanbin Li, Yizhou Li, Yu","year":"2016","venue":"IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2017] Bo Li, Zhengxing Sun, Jiagao Hu, and Jun- feng Xu. Co-saliency detection via sparse reconstruction and co-salient object discovery. In Advances in Multime- dia Information Processing -PCM 2017 -18th Pacific-Rim Conference on Multimedia, Harbin, China, September 28- 29, 2017, Revised Selected Papers, Part II, pages 222-232, 2017.","title":"Co-saliency detection via sparse reconstruction and co-salient object discovery","authors":"Bo Li, Zhengxing Sun, Jiagao Hu, Junfeng Xu","year":"2017","venue":"Advances in Multimedia Information Processing -PCM 2017 -18th Pacific-Rim Conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2019] Bo Li, Zhengxing Sun, and Yuqi Guo. Su- pervae: Superpixelwise variational autoencoder for salient object detection. In Proceedings of the Thirty-Three AAAI Conference on Artificial Intelligence, (AAAI-19), the 31th innovative Applications of Artificial Intelligence (IAAI- 19), and the 9th AAAI Symposium on Educational Ad- vances in Artificial Intelligence (EAAI-19), Hawaii, Hon- olulu, USA, February, 2019.","title":"Supervae: Superpixelwise variational autoencoder for salient object detection","authors":"Bo Li, Zhengxing Sun, Yuqi Guo","year":"2019","venue":"AAAI-19), the 31th innovative Applications of Artificial Intelligence (IAAI-19), and the 9th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-19)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lin et al., 2014] Tsung-Yi Lin, Michael Maire, Serge J. Be- longie, James Hays, Pietro Perona, Deva Ramanan, Pi- otr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In Computer Vision -ECCV 2014 -13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V, pages 740- 755, 2014.","title":"Microsoft COCO: common objects in context","authors":"Tsung-Yi Lin, Michael Maire, Serge J Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, C Lawrence Zitnick","year":"2014","venue":"Computer Vision -ECCV 2014 -13th European Conference","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu et al., 2014] Zhi Liu, Wenbin Zou, Lina Li, Liquan Shen, and Olivier Le Meur. Co-saliency detection based on hierarchical segmentation. IEEE Signal Processing Let- ters, 21(1):88-92, 2014.","title":"Co-saliency detection based on hierarchical segmentation","authors":"Zhi Liu, Wenbin Zou, Lina Li, Liquan Shen, Olivier Le Meur","year":"2014","venue":"IEEE Signal Processing Letters","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR, abs/1409.1556, 2014.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Wang et al., 2018] Tiantian Wang, Lihe Zhang, Shuo Wang, Huchuan Lu, Gang Yang, Xiang Ruan, and Ali Borji. De- tect globally, refine locally: A novel approach to saliency detection. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 3127-3135, 2018.","title":"Detect globally, refine locally: A novel approach to saliency detection","authors":"Tiantian Wang, Lihe Zhang, Shuo Wang, Huchuan Lu, Gang Yang, Xiang Ruan, Ali Borji","year":"2018","venue":"IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Wei et al., 2017] Lina Wei, Shanshan Zhao, Omar El Farouk Bourahla, Xi Li, and Fei Wu. Group- wise deep co-saliency detection. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI 2017, Melbourne, Australia, August 19-25, 2017, pages 3041-3047, 2017.","title":"Groupwise deep co-saliency detection","authors":"Lina Wei, Shanshan Zhao, Omar El, Farouk Bourahla, Xi Li, Fei Wu","year":"2017","venue":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI 2017","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Winn et al., 2005] John M. Winn, Antonio Criminisi, and Thomas P. Minka. Object categorization by learned uni- versal visual dictionary. In ICCV, pages 1800-1807, 2005.","title":"Object categorization by learned universal visual dictionary","authors":"John M Winn, Antonio Criminisi, Thomas P Minka","year":"2005","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Xue et al., 2013] Jianru Xue, Le Wang, Nanning Zheng, and Gang Hua. Automatic salient object extraction with contextual cue and its applications to recognition and al- pha matting. Pattern Recognition, 46(11):2874-2889, nov 2013.","title":"Automatic salient object extraction with contextual cue and its applications to recognition and alpha matting","authors":"Xue","year":"2013","venue":"Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Yang et al., 2013] Chuan Yang, Lihe Zhang, Huchuan Lu, Xiang Ruan, and Ming-Hsuan Yang. Saliency detection via graph-based manifold ranking. In 2013 IEEE Confer- ence on Computer Vision and Pattern Recognition, Port- land, OR, USA, June 23-28, 2013, pages 3166-3173, 2013.","title":"Saliency detection via graph-based manifold ranking","authors":"Chuan Yang, Lihe Zhang, Huchuan Lu, Xiang Ruan, Ming-Hsuan Yang","year":"2013","venue":"2013 IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Zhang et al., 2016] Dingwen Zhang, Junwei Han, Chao Li, Jingdong Wang, and Xuelong Li. Detection of co-salient objects by looking deep and wide. International Journal of Computer Vision, 120(2):215-232, 2016.","title":"Detection of co-salient objects by looking deep and wide","authors":"Zhang","year":"2016","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zhang et al., 2017a] Dingwen Zhang, Deyu Meng, and Jun- wei Han. Co-saliency detection via a self-paced multiple- instance learning framework. IEEE Trans. Pattern Anal. Mach. Intell., 39(5):865-878, 2017.","title":"Co-saliency detection via a self-paced multipleinstance learning framework","authors":"Zhang","year":"2017","venue":"IEEE Trans. Pattern Anal. Mach. Intell","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhang et al., 2017b] Pingping Zhang, Dong Wang, Huchuan Lu, Hongyu Wang, and Xiang Ruan. Amulet: Aggregating multi-level convolutional features for salient object detection. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pages 202-211, 2017.","title":"Amulet: Aggregating multi-level convolutional features for salient object detection","authors":"Zhang","year":"2017","venue":"IEEE International Conference on Computer Vision, ICCV 2017","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zheng et al., 2018] Xiaoju Zheng, Zheng-Jun Zha, and Liansheng Zhuang. A feature-adaptive semi-supervised framework for co-saliency detection. In 2018 ACM Multi- media Conference on Multimedia Conference, MM 2018, Seoul, Republic of Korea, October 22-26, 2018, pages 959-966, 2018.","title":"A feature-adaptive semi-supervised framework for co-saliency detection","authors":"Zheng","year":"2018","venue":"ACM Multimedia Conference on Multimedia Conference, MM 2018","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}