{"type":"ScholarlyArticle","title":"Beyond Universal Saliency: Personalized Saliency Prediction with Multi-task CNN","authors":[{"name":"Yanyu Xu","slug":"yanyu-xu","given":"Yanyu","family":"Xu","affiliation":"ShanghaiTech University , Shanghai , China."},{"name":"Nianyi Li","slug":"nianyi-li","given":"Nianyi","family":"Li","affiliation":"University of Delaware , Newark , DE , USA.; Plex-VR digital technology Co. ,"},{"name":"Junru Wu","slug":"junru-wu","given":"Junru","family":"Wu","affiliation":"ShanghaiTech University , Shanghai , China."},{"name":"Jingyi Yu","slug":"jingyi-yu","given":"Jingyi","family":"Yu","affiliation":"ShanghaiTech University , Shanghai , China.; Plex-VR digital technology Co. ,"},{"name":"Shenghua Gao","slug":"shenghua-gao","given":"Shenghua","family":"Gao","affiliation":"ShanghaiTech University , Shanghai , China."}],"abstract":"Saliency detection is a long standing problem in computer vision. Tremendous efforts have been focused on exploring a universal saliency model across users despite their differences in gender, race, age, etc. Yet recent psychology studies suggest that saliency is highly specific than universal: individuals exhibit heterogeneous gaze patterns when viewing an identical scene containing multiple salient objects. \r\n\r\nIn this paper, we first show that such heterogeneity is common and critical for reliable saliency prediction. Our study also produces the first database of personalized saliency maps (PSMs). We model PSM based on universal saliency map (USM) shared by different participants and adopt a multi-task CNN framework to estimate the discrepancy between PSM and USM. Comprehensive experiments demonstrate that our new PSM model and prediction scheme are effective and reliable.","doi":"10.24963/ijcai.2017/543","url":"https://ijcai.org/proceedings/2017/543","pdf":"https://ijcai.org/proceedings/2017/0543.pdf","markdown":"https://ijcai.org/proceedings/2017/543.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/543","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Multidisciplinary Topics and Applications","track":"Main track","number":543,"pages":{"first":3887,"last":3893},"keywords":["Robotics and Vision: Vision and Perception","Multidisciplinary Topics and Applications: Personalization and User Modeling"],"videos":[],"references":[{"position":1,"raw":"[Borji et al., 2014] Ali Borji, Ming Ming Cheng, Huaizu Jiang, and Jia Li. Salient object detection: A survey. Eprint Arxiv, 16(7):3118, 2014.","title":"Salient object detection: A survey","authors":"Ali Borji, Ming, Ming Cheng, Huaizu Jiang, Jia Li","year":"2014","venue":"Eprint Arxiv","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"Bylinskii et al., ] Zoya Bylinskii, Tilke Judd, Ali Borji, Laurent Itti, Frédo Durand, Aude Oliva, and Antonio Tor- ralba. Mit saliency benchmark.","title":"Mit saliency benchmark","authors":"Bylinskii","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Carreira et al., 2015] Joao Carreira, Pulkit Agrawal, Kate- rina Fragkiadaki, and Jitendra Malik. Human pose esti- mation with iterative error feedback. arXiv preprint arX- iv:1507.06550, 2015.","title":"Human pose estimation with iterative error feedback","authors":"Joao Carreira, Pulkit Agrawal, Katerina Fragkiadaki, Jitendra Malik","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chang et al., 2016] Miko May Lee Chang, Soh Khim Ong, and Andrew Yeh Ching Nee. Automatic information posi- tioning scheme in ar-assisted maintenance based on visu- al saliency. In SAIENTO AVR, pages 453-462. Springer, 2016.","title":"Automatic information positioning scheme in ar-assisted maintenance based on visual saliency","authors":"Chang","year":"2016","venue":"SAIENTO AVR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cornia et al., 2016] Marcella Cornia, Lorenzo Baraldi, Giuseppe Serra, and Rita Cucchiara. A deep multi-level network for saliency prediction. arXiv preprint arX- iv:1609.01064, 2016.","title":"A deep multi-level network for saliency prediction","authors":"Marcella Cornia, Lorenzo Baraldi, Giuseppe Serra, Rita Cucchiara","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fang et al., 2016] S. Fang, J. Li, Y. Tian, T. Huang, and X. Chen. Learning discriminative subspaces on random contrasts for image saliency analysis. TNNLS, 2016.","title":"Learning discriminative subspaces on random contrasts for image saliency analysis","authors":"S Fang, J Li, Y Tian, T Huang, X Chen","year":"2016","venue":"TNNLS","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gygli et al., 2013] Michael Gygli, Helmut Grabner, Hayko Riemenschneider, Fabian Nater, and Luc Van Gool. The interestingness of images. In ICCV, pages 1633-1640, 2013.","title":"The interestingness of images","authors":"Michael Gygli, Helmut Grabner, Hayko Riemenschneider, Fabian Nater, Luc Van Gool","year":"2013","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Huang et al., 2015] Xun Huang, Chengyao Shen, Xavier Boix, and Qi Zhao. Salicon: Reducing the semantic gap in saliency prediction by adapting deep neural networks. In ICCV, pages 262-270, 2015.","title":"Salicon: Reducing the semantic gap in saliency prediction by adapting deep neural networks","authors":"Xun Huang, Chengyao Shen, Xavier Boix, Qi Zhao","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Itti, 2004] Laurent Itti. Automatic foveation for video com- pression using a neurobiological model of visual attention. IEEE TIP, 13(10):1304-1318, 2004.","title":"Automatic foveation for video compression using a neurobiological model of visual attention","authors":"Laurent Itti","year":"2004","venue":"IEEE TIP","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jia et al., 2014] Yangqing Jia, Evan Shelhamer, Jeff Don- ahue, Sergey Karayev, Jonathan Long, Ross Girshick, Ser- gio Guadarrama, and Trevor Darrell. Caffe: Convolution- al architecture for fast feature embedding. arXiv preprint arXiv:1408.5093, 2014.","title":"Caffe: Convolutional architecture for fast feature embedding","authors":"Jia","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Jiang et al., 2015] Ming Jiang, Shengsheng Huang, Juany- ong Duan, and Qi Zhao. Salicon: Saliency in context. In CVPR, pages 1072-1080, 2015.","title":"Salicon: Saliency in context","authors":"Ming Jiang, Shengsheng Huang, Juanyong Duan, Qi Zhao","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Judd et al., 2009] Tilke Judd, Krista Ehinger, Frédo Durand, and Antonio Torralba. Learning to predict where humans look. In ICCV, pages 2106-2113, 2009.","title":"Learning to predict where humans look","authors":"Tilke Judd, Krista Ehinger, Frédo Durand, Antonio Torralba","year":"2009","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Judd et al., 2012] Tilke Judd, Frédo Durand, and Antonio Torralba. A benchmark of computational models of salien- cy to predict human fixations. In MIT Technical Report, 2012.","title":"A benchmark of computational models of saliency to predict human fixations","authors":"Tilke Judd, Frédo Durand, Antonio Torralba","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kruthiventi et al., 2015] Srinivas SS Kruthiventi, Kumar Ayush, and R Venkatesh Babu. Deepfix: A fully convolu- tional neural network for predicting human eye fixations. arXiv preprint arXiv:1510.02927, 2015.","title":"Deepfix: A fully convolutional neural network for predicting human eye fixations","authors":"S S Srinivas, Kumar Kruthiventi, R Ayush, Babu Venkatesh","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kruthiventi et al., 2016] Srinivas S. S. Kruthiventi, Vennela Gudisa, Jaley H. Dholakiya, and R. Venkatesh Babu. Saliency unified: A deep architecture for simultaneous eye fixation prediction and salient object segmentation. In CVPR, pages 5781-5790, 2016.","title":"Saliency unified: A deep architecture for simultaneous eye fixation prediction and salient object segmentation","authors":"S S Srinivas, Vennela Kruthiventi, Gudisa, H Jaley, R Dholakiya, Venkatesh, Babu","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lee et al., 2014] Chen Yu Lee, Saining Xie, Patrick Gal- lagher, Zhengyou Zhang, and Zhuowen Tu. Deeply- supervised nets. Arxiv, pages 562-570, 2014.","title":"Deeplysupervised nets","authors":"Chen Yu, Lee, Saining Xie, Patrick Gallagher, Zhengyou Zhang, Zhuowen Tu","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Li et al., 2014] Yin Li, Xiaodi Hou, Christof Koch, James M Rehg, and Alan L Yuille. The secrets of salient object segmentation. In CVPR, pages 280-287, 2014.","title":"The secrets of salient object segmentation","authors":"Yin Li, Xiaodi Hou, Christof Koch, James M Rehg, Alan L Yuille","year":"2014","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Li et al., 2016] X. Li, L. Zhao, L. Wei, M. H. Yang, F. Wu, Y. Zhuang, H. Ling, and J. Wang. Deepsaliency: Multi- task deep neural network model for salient object detec- tion. TIP, 25(8):3919-3930, 2016.","title":"Deepsaliency: Multitask deep neural network model for salient object detection","authors":"Li","year":"2016","venue":"TIP","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Liu et al., 2015] Nian Liu, Junwei Han, Dingwen Zhang, Shifeng Wen, and Tianming Liu. Predicting eye fixation- s using convolutional neural networks. In CVPR, pages 362-370, 2015.","title":"Predicting eye fixations using convolutional neural networks","authors":"Nian Liu, Junwei Han, Dingwen Zhang, Shifeng Wen, Tianming Liu","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Pan et al., 2016] Junting Pan, Elisa Sayrol, Xavier Giroini- eto, Kevin Mcguinness, and Noel E. Oconnor. Shallow and deep convolutional networks for saliency prediction. In CVPR, pages 598-606, 2016.","title":"Shallow and deep convolutional networks for saliency prediction","authors":"Pan","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Russakovsky et al., 2015] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. Ima- geNet Large Scale Visual Recognition Challenge. IJCV, 115(3):211-252, 2015.","title":"Ima-geNet Large Scale Visual Recognition Challenge","authors":"Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C Berg, Li Fei-Fei","year":"2015","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Russell et al., 2008] Bryan C Russell, Antonio Torralba, Kevin P Murphy, and William T Freeman. Labelme: a database and web-based tool for image annotation. IJCV, 77(1-3):157-173, 2008.","title":"Labelme: a database and web-based tool for image annotation","authors":"Antonio Bryan C Russell, Kevin P Torralba, William T Murphy, Freeman","year":"2008","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Setlur et al., 2005] Vidya Setlur, Saeko Takagi, Ramesh Raskar, Michael Gleicher, and Bruce Gooch. Automatic image retargeting. In MUM, pages 59-68, 2005.","title":"Automatic image retargeting","authors":"Vidya Setlur, Saeko Takagi, Ramesh Raskar, Michael Gleicher, Bruce Gooch","year":"2005","venue":"MUM","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Xu et al., 2014] Juan Xu, Ming Jiang, Shuo Wang, Mohan S Kankanhalli, and Qi Zhao. Predicting human gaze beyond pixels. Journal of vision, 14(1):28-28, 2014.","title":"Predicting human gaze beyond pixels","authors":"Juan Xu, Ming Jiang, Shuo Wang, Mohan S Kankanhalli, Qi Zhao","year":"2014","venue":"Journal of vision","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Xu et al., 2015] Pingmei Xu, Krista A Ehinger, Yinda Zhang, Adam Finkelstein, Sanjeev R Kulkarni, and Jianx- iong Xiao. Turkergaze: crowdsourcing saliency with web- cam based eye tracking. arXiv preprint arXiv:1504.06755, 2015.","title":"Turkergaze: crowdsourcing saliency with webcam based eye tracking","authors":"Xu","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhang and Sclaroff, 2013] Jianming Zhang and Stan Sclaroff. Saliency detection: A boolean map approach. In ICCV, pages 153-160, 2013.","title":"Saliency detection: A boolean map approach","authors":"Jianming Zhang, Stan Sclaroff","year":"2013","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang et al., 2014] Zhanpeng Zhang, Ping Luo, Change Loy Chen, and Xiaoou Tang. Facial land- mark detection by deep multi-task learning. In ECCV, pages 94-108, 2014.","title":"Facial landmark detection by deep multi-task learning","authors":"Zhanpeng Zhang, Ping Luo, Change Loy Chen, Xiaoou Tang","year":"2014","venue":"ECCV","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}