{"type":"ScholarlyArticle","title":"Deep Light-field-driven Saliency Detection from a Single View","authors":[{"name":"Yongri Piao","slug":"yongri-piao","given":"Yongri","family":"Piao","affiliation":"School of Information and Communication Engineering , Dalian University of Technology , Dalian , China"},{"name":"Zhengkun Rong","slug":"zhengkun-rong","given":"Zhengkun","family":"Rong","affiliation":"School of Information and Communication Engineering , Dalian University of Technology , Dalian , China"},{"name":"Miao Zhang","slug":"miao-zhang","given":"Miao","family":"Zhang","affiliation":"Key Laboratory for Ubiquitous Network and Service Software of Liaoning Province , Dalian University of Technology , Dalian , China; DUT-RU International School of Information and Software Engineering , Dalian University of Technology , Dalian , China"},{"name":"Xiao Li","slug":"xiao-li","given":"Xiao","family":"Li","affiliation":"School of Information and Communication Engineering , Dalian University of Technology , Dalian , China"},{"name":"Huchuan Lu","slug":"huchuan-lu","given":"Huchuan","family":"Lu","affiliation":"School of Information and Communication Engineering , Dalian University of Technology , Dalian , China"}],"abstract":"Previous 2D saliency detection methods extract salient cues from a single view and directly predict the expected results. Both traditional and deep-learning-based 2D methods do not consider geometric information of 3D scenes. Therefore the relationship between scene understanding and salient objects cannot be effectively established. This limits the performance of 2D saliency detection in challenging scenes. In this paper, we show for the first time that saliency detection problem can be reformulated as two sub-problems: light field synthesis from a single view and light-field-driven saliency detection. We propose a high-quality light field synthesis network to produce reliable 4D light field information. Then we propose a novel light-field-driven saliency detection network with two purposes, that is, i) richer saliency features can be produced for effective saliency detection; ii) geometric information can be considered for integration of multi-view saliency maps in a view-wise attention fashion. The whole pipeline can be trained in an end-to-end fashion. For training our network, we introduce the largest light field dataset for saliency detection, containing 1580 light fields that cover a wide variety of challenging scenes. With this new formulation, our method is able to achieve state-of-the-art performance.","doi":"10.24963/ijcai.2019/127","url":"https://ijcai.org/proceedings/2019/127","pdf":"https://ijcai.org/proceedings/2019/0127.pdf","markdown":"https://ijcai.org/proceedings/2019/127.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/127","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Computer Vision","track":"Main track","number":127,"pages":{"first":904,"last":911},"keywords":["Computer Vision: Recognition: Detection, Categorization, Indexing, Matching, Retrieval, Semantic Interpretation","Computer Vision: Computer Vision"],"videos":[],"references":[{"position":1,"raw":"[Buehler et al., 2001] Chris Buehler, Michael Bosse, Leonard McMillan, Steven J. Gortler, and Michael F. Cohen. Unstructured lumigraph rendering. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques, pages 425-432, 2001.","title":"Unstructured lumigraph rendering","authors":"Chris Buehler, Michael Bosse, Leonard Mcmillan, Steven J Gortler, Michael F Cohen","year":"2001","venue":"Proceedings of the 28th annual conference on Computer graphics and interactive techniques","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Deng et al., 2018] Zijun Deng, Xiaowei Hu, Lei Zhu, X- uemiao Xu, Jing Qin, Guoqiang Han, and Pheng-Ann Heng. R 3 net: Recurrent residual refinement network for saliency detection. In IJCAI, pages 684-690, 2018.","title":"R 3 net: Recurrent residual refinement network for saliency detection","authors":"Zijun Deng, Xiaowei Hu, Lei Zhu, Xuemiao Xu, Jing Qin, Guoqiang Han, Pheng-Ann Heng","year":"2018","venue":"IJCAI","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Fan et al., 2017] Deng-Ping Fan, Ming-Ming Cheng, Yun Liu, Tao Li, and Ali Borji. Structure-measure: A new way to evaluate foreground maps. In ICCV, pages 4558-4567, 2017.","title":"Structure-measure: A new way to evaluate foreground maps","authors":"Deng-Ping Fan, Ming-Ming Cheng, Yun Liu, Tao Li, Ali Borji","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Fan et al., 2018] Deng-Ping Fan, Cheng Gong, Yang Cao, Bo Ren, Ming-Ming Cheng, and Ali Borji. Enhanced- alignment measure for binary foreground map evaluation. In IJCAI, pages 698-704, 2018.","title":"Enhancedalignment measure for binary foreground map evaluation","authors":"Deng-Ping Fan, Cheng Gong, Yang Cao, Bo Ren, Ming-Ming Cheng, Ali Borji","year":"2018","venue":"IJCAI","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Gortler et al., 1996] Steven J. Gortler, Radek Grzeszczuk, Richard Szeliski, and Michael F. Cohen. The lumigraph. In Proceedings of the 23rd annual conference on Comput- er graphics and interactive techniques, pages 43-54, 1996.","title":"The lumigraph","authors":"Steven J Gortler, Radek Grzeszczuk, Richard Szeliski, Michael F Cohen","year":"1996","venue":"Proceedings of the 23rd annual conference on Computer graphics and interactive techniques","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Guo et al., 2017] Xinqing Guo, Zhang Chen, Siyuan Li, Yang Yang, and Jingyi Yu. Deep depth inference using binocular and monocular cues. arXiv: Computer Vision and Pattern Recognition, 2017.","title":"Deep depth inference using binocular and monocular cues","authors":"Xinqing Guo, Zhang Chen, Siyuan Li, Yang Yang, Jingyi Yu","year":"2017","venue":"arXiv: Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Han et al., 2017] Junwei Han, Hao Chen, Nian Liu, Cheng- gang Yan, and Xuelong Li. Cnns-based rgb-d saliency de- tection via cross-view transfer and multiview fusion. IEEE transactions on cybernetics, (99):1-13, 2017.","title":"Cnns-based rgb-d saliency detection via cross-view transfer and multiview fusion","authors":"Junwei Han, Hao Chen, Nian Liu, Chenggang Yan, Xuelong Li","year":"2017","venue":"IEEE transactions on cybernetics","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hou et al., 2017] Qibin Hou, Ming-Ming Cheng, Xiaowei Hu, Ali Borji, Zhuowen Tu, and Philip H.S. Torr. Deeply supervised salient object detection with short connections. In CVPR, pages 5300-5309, 2017.","title":"Deeply supervised salient object detection with short connections","authors":"Hou","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Ju et al., 2015] Ran Ju, Yang Liu, Tongwei Ren, Ling Ge, and Gangshan Wu. Depth-aware salient object detec- tion using anisotropic center-surround difference. Signal Processing-image Communication, 38:115-126, 2015.","title":"Depth-aware salient object detection using anisotropic center-surround difference","authors":"Ran Ju, Yang Liu, Tongwei Ren, Ling Ge, Gangshan Wu","year":"2015","venue":"Signal Processing-image Communication","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lee et al., 2015] Chen-Yu Lee, Saining Xie, Patrick W. Gal- lagher, Zhengyou Zhang, and Zhuowen Tu. Deeply- supervised nets. international conference on artificial in- telligence and statistics, pages 562-570, 2015.","title":"Deeplysupervised nets. international conference on artificial intelligence and statistics","authors":"Chen-Yu Lee, Saining Xie, Patrick W Gallagher, Zhengyou Zhang, Zhuowen Tu","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Levoy and Hanrahan, 1996] Marc Levoy and Pat Hanrahan. Light field rendering. In Proceedings of the 23rd annu- al conference on Computer graphics and interactive tech- niques, pages 31-42, 1996.","title":"Light field rendering","authors":"Hanrahan Levoy, Marc Levoy, Pat Hanrahan","year":"1996","venue":"Proceedings of the 23rd annual conference on Computer graphics and interactive techniques","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li and Yu, 2015] Guanbin Li and Yizhou Yu. Visual salien- cy based on multiscale deep features. In CVPR, pages 5455-5463, 2015.","title":"Visual saliency based on multiscale deep features","authors":"Yu ; Li, Guanbin Li, Yizhou Yu","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li and Yu, 2016] Guanbin Li and Yizhou Yu. Deep contrast learning for salient object detection. In CVPR, pages 478- 487, 2016.","title":"Deep contrast learning for salient object detection","authors":"Yu ; ] Guanbin Li, Yizhou Li, Yu","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li et al., 2013] Xiaohui Li, Huchuan Lu, Lihe Zhang, Xi- ang Ruan, and Ming-Hsuan Yang. Saliency detection via dense and sparse reconstruction. In ICCV, pages 2976- 2983, 2013.","title":"Saliency detection via dense and sparse reconstruction","authors":"Xiaohui Li, Huchuan Lu, Lihe Zhang, Xiang Ruan, Ming-Hsuan Yang","year":"2013","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2014] Nianyi Li, Jinwei Ye, Yu Ji, Haibin Ling, and Jingyi Yu. Saliency detection on light field. In CVPR, pages 2806-2813, 2014.","title":"Saliency detection on light field","authors":"Li","year":"2014","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Li et al., 2015] Nianyi Li, Bilin Sun, and Jingyi Yu. A weighted sparse coding framework for saliency detection. In CVPR, pages 5216-5223, 2015.","title":"A weighted sparse coding framework for saliency detection","authors":"Li","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liu and Han, 2016] Nian Liu and Junwei Han. Dhsnet: Deep hierarchical saliency network for salient object de- tection. In CVPR, pages 678-686, 2016.","title":"Dhsnet: Deep hierarchical saliency network for salient object detection","authors":"Nian Liu, Junwei Han","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Qin et al., 2015] Yao Qin, Huchuan Lu, Yiqun Xu, and He Wang. Saliency detection via cellular automata. In CVPR, pages 110-119, 2015.","title":"Saliency detection via cellular automata","authors":"Huchuan Yao Qin, Yiqun Lu, He Xu, Wang","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Qu et al., 2017] Liangqiong Qu, Shengfeng He, Jiawei Zhang, Jiandong Tian, Yandong Tang, and Qingxiong Yang. Rgbd salient object detection via deep fusion. IEEE Transactions on Image Processing, 26(5):2274- 2285, 2017.","title":"Rgbd salient object detection via deep fusion","authors":"Qu","year":"2017","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Song and Lee, 2018] Gwangmo Song and Kyoung Mu Lee. Depth estimation network for dual defocused images with different depth-of-field. In 2018 25th IEEE Internation- al Conference on Image Processing (ICIP), pages 1563- 1567, 2018.","title":"Gwangmo Song and Kyoung Mu Lee. Depth estimation network for dual defocused images with different depth-of-field","authors":"Lee Song","year":"2018","venue":"25th IEEE International Conference on Image Processing (ICIP)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Srinivasan et al., 2017] Pratul P. Srinivasan, Tongzhou Wang, Ashwin Sreelal, Ravi Ramamoorthi, and Ren Ng. Learning to synthesize a 4d rgbd light field from a single image. In ICCV, pages 2262-2270, 2017.","title":"Learning to synthesize a 4d rgbd light field from a single image","authors":"P Pratul, Tongzhou Srinivasan, Ashwin Wang, Ravi Sreelal, Ren Ramamoorthi, Ng","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Tu et al., 2016] Wei-Chih Tu, Shengfeng He, Qingxiong Yang, and Shao-Yi Chien. Real-time salient object de- tection with a minimum spanning tree. In CVPR, pages 2334-2342, 2016.","title":"Real-time salient object detection with a minimum spanning tree","authors":"Wei-Chih Tu, Shengfeng He, Qingxiong Yang, Shao-Yi Chien","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wang and Gong, 2019] Ningning Wang and Xiaojin Gong. Adaptive fusion for rgb-d salient object detection. arXiv preprint arXiv:1901.01369, 2019.","title":"Adaptive fusion for rgb-d salient object detection","authors":"Ningning Wang, Xiaojin Gong","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang et al., 2016a] Linzhao Wang, Lijun Wang, Huchuan Lu, Pingping Zhang, and Xiang Ruan. Saliency detec- tion with recurrent fully convolutional networks. In ECCV, pages 825-841, 2016.","title":"Saliency detection with recurrent fully convolutional networks","authors":"Wang","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wang et al., 2016b] Ting-Chun Wang, Jun-Yan Zhu, Ebi Hiroaki, Manmohan Chandraker, Alexei A. Efros, and Ravi Ramamoorthi. A 4d light-field dataset and cnn ar- chitectures for material recognition. european conference on computer vision, pages 121-138, 2016.","title":"A 4d light-field dataset and cnn architectures for material recognition","authors":"Wang","year":"2016","venue":"european conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yeung et al., 2019] Henry Wing Fung Yeung, Junhui Hou, Xiaoming Chen, Jie Chen, Zhibo Chen, and Yuk Ying Chung. Light field spatial super-resolution using deep ef- ficient spatial-angular separable convolution. IEEE Trans- actions on Image Processing, 28(5):2319-2330, 2019.","title":"Light field spatial super-resolution using deep efficient spatial-angular separable convolution","authors":"Henry Wing, Fung Yeung, Junhui Hou, Xiaoming Chen, Jie Chen, Zhibo Chen, Yuk Ying, Chung","year":"2019","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang et al., 2015] Jun Zhang, Meng Wang, Jun Gao, Y- i Wang, Xudong Zhang, and Xindong Wu. Saliency de- tection with a deeper investigation of light field. In IJCAI, pages 2212-2218, 2015.","title":"Saliency detection with a deeper investigation of light field","authors":"Jun Zhang, Meng Wang, Jun Gao, Yi Wang, Xudong Zhang, Xindong Wu","year":"2015","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2015/313"},{"position":28,"raw":"[Zhang et al., 2017a] Jun Zhang, Meng Wang, Liang Lin, X- un Yang, Jun Gao, and Yong Rui. Saliency detection on light field: A multi-cue approach. ACM Transactions on Multimedia Computing, Communications, and Appli- cations, 13(3):32, 2017.","title":"Saliency detection on light field: A multi-cue approach","authors":"Zhang","year":"2017","venue":"ACM Transactions on Multimedia Computing, Communications, and Applications","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhang et al., 2017b] Pingping Zhang, Dong Wang, Huchuan Lu, Hongyu Wang, and Xiang Ruan. Amulet: Aggregating multi-level convolutional features for salient object detection. In ICCV, pages 202-211, 2017.","title":"Amulet: Aggregating multi-level convolutional features for salient object detection","authors":"Zhang","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhang et al., 2017c] Pingping Zhang, Dong Wang, Huchuan Lu, Hongyu Wang, and Baocai Yin. Learning uncertain convolutional features for accurate saliency detection. In ICCV, pages 212-221, 2017.","title":"Learning uncertain convolutional features for accurate saliency detection","authors":"Zhang","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhou et al., 2018] Wenhui Zhou, Linkai Liang, Hua Zhang, Andrew Lumsdaine, and Lili Lin. Scale and orientation aware epi-patch learning for light field depth estimation. In 2018 24th International Conference on Pattern Recog- nition (ICPR), pages 2362-2367, 2018.","title":"Scale and orientation aware epi-patch learning for light field depth estimation","authors":"Wenhui Zhou, Linkai Liang, Hua Zhang, Andrew Lumsdaine, Lili Lin","year":"2018","venue":"24th International Conference on Pattern Recognition (ICPR)","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhu et al., 2018] Chunbiao Zhu, Xing Cai, Kan Huang, Thomas H Li, and Ge Li. Pdnet: Prior-model guided depth-enhanced network for salient object detection. arXiv preprint arXiv:1803.08636, 2018.","title":"Pdnet: Prior-model guided depth-enhanced network for salient object detection","authors":"Chunbiao Zhu, Xing Cai, Kan Huang, Thomas H Li, Ge Li","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhu et al., 2019] Hao Zhu, Mantang Guo, Hongdong Li, Qing Wang, and Antonio Robles-Kelly. Breaking the spatio-angular trade-off for light field super-resolution via lstm modelling on epipolar plane images. arXiv preprint arXiv:1902.05672, 2019.","title":"Breaking the spatio-angular trade-off for light field super-resolution via lstm modelling on epipolar plane images","authors":"Mantang Hao Zhu, Hongdong Guo, Qing Li, Antonio Wang, Robles-Kelly","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}