{"type":"ScholarlyArticle","title":"Scanpath Prediction for Visual Attention using IOR-ROI LSTM","authors":[{"name":"Zhenzhong Chen","slug":"zhenzhong-chen","given":"Zhenzhong","family":"Chen","affiliation":"School of Remote Sensing and Information Engineering , Wuhan University , Wuhan , P.R. China"},{"name":"Wanjie Sun","slug":"wanjie-sun","given":"Wanjie","family":"Sun","affiliation":"School of Remote Sensing and Information Engineering , Wuhan University , Wuhan , P.R. China"}],"abstract":"Predicting scanpath when a certain stimulus is presented plays an important role in modeling visual attention and search. This paper presents a model that integrates convolutional neural network and long short-term memory (LSTM) to generate realistic scanpaths. The core part of the proposed model is a dual LSTM unit, i.e., an inhibition of return LSTM (IOR-LSTM) and a region of interest LSTM (ROI-LSTM), capturing IOR dynamics and gaze shift behavior simultaneously. IOR-LSTM simulates the visual working memory to adaptively integrate and forget scene information. ROI-LSTM is responsible for predicting the next ROI given the inhibited image features. Experimental results indicate that the proposed architecture can achieve superior performance in predicting scanpaths.","doi":"10.24963/ijcai.2018/89","url":"https://ijcai.org/proceedings/2018/89","pdf":"https://ijcai.org/proceedings/2018/0089.pdf","markdown":"https://ijcai.org/proceedings/2018/89.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/89","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Computer Vision","track":"Main track","number":89,"pages":{"first":642,"last":648},"keywords":["Computer Vision: Perception","Computer Vision: Computer Vision"],"videos":[],"references":[{"position":1,"raw":"[Anderson et al., 2015] Nicola C. Anderson, Fraser Ander- son, Alan Kingstone, and Walter F. Bischof. A comparison of scanpath comparison methods. Behav. Res. Methods, 47(4):1377-1392, 2015.","title":"A comparison of scanpath comparison methods","authors":"Nicola C Anderson, Fraser Anderson, Alan Kingstone, Walter F Bischof","year":"2015","venue":"Behav. Res. Methods","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Assens Reina et al., 2017] Marc Assens Reina, Xavier Giró-i Nieto, Kevin McGuinness, and Noel E. O'Connor. Saltinet: Scan-path prediction on 360 degree images using saliency volumes. In ICCV Workshop on EPIC, 2017.","title":"Saltinet: Scan-path prediction on 360 degree images using saliency volumes","authors":"Assens Reina","year":"2017","venue":"ICCV Workshop on EPIC","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bays and Husain, 2012] Paul M. Bays and Masud Husain. Active inhibition and memory promote exploration and search of natural scenes. J. Vis., 12(8):8, 2012.","title":"Active inhibition and memory promote exploration and search of natural scenes","authors":"Paul M Bays, Masud Husain","year":"2012","venue":"J. Vis","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Boccignone, 2016] Giuseppe Boccignone. A probabilistic tour of visual attention and gaze shift computational mod- els. arXiv preprint arXiv:1607.01232, 2016.","title":"A probabilistic tour of visual attention and gaze shift computational models","authors":"Giuseppe Boccignone","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cornia et al., 2017] Marcella Cornia, Lorenzo Baraldi, Giuseppe Serra, and Rita Cucchiara. Predicting human eye fixations via an lstm-based saliency attentive model. In ICVSS, 2017.","title":"Predicting human eye fixations via an lstm-based saliency attentive model","authors":"Marcella Cornia, Lorenzo Baraldi, Giuseppe Serra, Rita Cucchiara","year":"2017","venue":"ICVSS","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Cristino et al., 2010] Filipe Cristino, Sebastiaan Mathôt, Jan Theeuwes, and Iain D. Gilchrist. Scanmatch: A novel method for comparing fixation sequences. Behav. Res. Methods, 42(3):692-700, 2010.","title":"Scanmatch: A novel method for comparing fixation sequences","authors":"Filipe Cristino, Sebastiaan Mathôt, Jan Theeuwes, Iain D Gilchrist","year":"2010","venue":"Behav. Res. Methods","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dewhurst et al., 2012] Richard Dewhurst, Marcus Nyström, Halszka Jarodzka, Tom Foulsham, Roger Johansson, and Kenneth Holmqvist. It depends on how you look at it: Scanpath comparison in multiple dimensions with mul- timatch, a vector-based approach. Behav.Res. Methods, 44(4):1079-1100, 2012.","title":"It depends on how you look at it: Scanpath comparison in multiple dimensions with multimatch, a vector-based approach","authors":"Richard Dewhurst, Marcus Nyström, Halszka Jarodzka, Tom Foulsham, Roger Johansson, Kenneth Holmqvist","year":"2012","venue":"Behav.Res. Methods","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Comput., 9(8):1735-1780, 1997.","title":"Long short-term memory","authors":"Schmidhuber Hochreiter, Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural Comput","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hotta et al., 2010] Shinji Hotta, Shigeyuki Oba, and Shin Ishii. Visual attention model involving feature-based inhi- bition of return. Artif. Life Robotics, 15(2):129-132, 2010.","title":"Visual attention model involving feature-based inhibition of return","authors":"Shinji Hotta, Shigeyuki Oba, Shin Ishii","year":"2010","venue":"Artif. Life Robotics","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Itti et al., 1998] Laurent Itti, Christof Koch, and Ernst Niebur. A model of saliency-based visual attention for rapid scene analysis. IEEE Trans. Pattern Anal. Mach. Intell., 20(11):1254-1259, 1998.","title":"A model of saliency-based visual attention for rapid scene analysis","authors":"Laurent Itti, Christof Koch, Ernst Niebur","year":"1998","venue":"IEEE Trans. Pattern Anal. Mach. Intell","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Jiang et al., 2016] Ming Jiang, Xavier Boix, Gemma Roig, Juan Xu, Luc Van Gool, and Qi Zhao. Learning to predict sequences of human visual fixations. IEEE Trans. Neural Netw. Learn. Syst., 27(6):1241-1252, 2016.","title":"Learning to predict sequences of human visual fixations","authors":"Ming Jiang, Xavier Boix, Gemma Roig, Juan Xu, Luc Van Gool, Qi Zhao","year":"2016","venue":"IEEE Trans. Neural Netw. Learn. Syst","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Judd et al., 2009] Tilke Judd, Krista Ehinger, Frédo Durand, and Antonio Torralba. Learning to predict where humans look. In ICCV, 2009.","title":"Learning to predict where humans look","authors":"Tilke Judd, Krista Ehinger, Frédo Durand, Antonio Torralba","year":"2009","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kümmerer et al., 2017] Matthias Kümmerer, Thomas S. A. Wallis, Leon A. Gatys, and Matthias Bethge. Understand- ing low-and high-level contributions to fixation predic- tion. In ICCV, 2017.","title":"Understanding low-and high-level contributions to fixation prediction","authors":"Matthias Kümmerer, S A Thomas, Leon A Wallis, Matthias Gatys, Bethge","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Le Meur and Liu, 2015] Olivier Le Meur and Zhi Liu. Sac- cadic model of eye movements for free-viewing condition. Vision Res., 116:152-164, 2015.","title":"Saccadic model of eye movements for free-viewing condition","authors":"Le Meur, ; Liu, Le Olivier, Zhi Meur, Liu","year":"2015","venue":"Vision Res","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Le Meur et al., 2017] Olivier Le Meur, Antoine Coutrot, Zhi Liu, Pia Rämä, Adrien Le Roch, and Andrea Helo. Vi- sual attention saccadic models learn to emulate gaze pat- terns from childhood to adulthood. IEEE Trans. Image Processing, 26(10):4777-4789, 2017.","title":"Visual attention saccadic models learn to emulate gaze patterns from childhood to adulthood","authors":"Le Meur","year":"2017","venue":"IEEE Trans. Image Processing","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Ngo and Manjunath, 2017] Thuyen Ngo and B.S. Manju- nath. Saccade gaze prediction using a recurrent neural net- work. In ICIP, 2017.","title":"Saccade gaze prediction using a recurrent neural network","authors":"Thuyen Ngo, B S Manjunath","year":"2017","venue":"ICIP","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Samuel and Kat, 2003] Arthur G. Samuel and Donna Kat. Inhibition of return: A graphical meta-analysis of its time course and an empirical test of its temporal and spatial properties. Psychon Bull. Rev., 10(4):897-906, 2003.","title":"Inhibition of return: A graphical meta-analysis of its time course and an empirical test of its temporal and spatial properties","authors":"Arthur G Samuel, Donna Kat","year":"2003","venue":"Psychon Bull. Rev","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Shi et al., 2015] Xingjian Shi, Zhourong Chen, Hao Wang, Dit-Yan Yeung, Wai-kin Wong, and Wang-chun Woo. Convolutional lstm network: A machine learning approach for precipitation nowcasting. In NIPS, 2015.","title":"Convolutional lstm network: A machine learning approach for precipitation nowcasting","authors":"Shi","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Sun et al., 2012] Xiaoshuai Sun, Hongxun Yao, and Ron- grong Ji. What are we looking for: Towards statistical modeling of saccadic eye movements and visual saliency. In CVPR, 2012.","title":"What are we looking for: Towards statistical modeling of saccadic eye movements and visual saliency","authors":"Xiaoshuai Sun, Hongxun Yao, Rongrong Ji","year":"2012","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wandell, 1995] Brian A. Wandell. Foundations of Vision. Sinauer Associates, 1995.","title":"Foundations of Vision","authors":"Brian A Wandell","year":"1995","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wang and Klein, 2010] Zhiguo Wang and Raymond M. Klein. Searching for inhibition of return in visual search: A review. Vision Res., 50(2):220-228, 2010.","title":"Searching for inhibition of return in visual search: A review","authors":"Wang, Klein ; ] Zhiguo Wang, Raymond M Klein","year":"2010","venue":"Vision Res","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang et al., 2011] Wei Wang, Cheng Chen, Yizhou Wang, Tingting Jiang, Fang Fang, and Yuan Yao. Simulating human saccadic scanpaths on natural images. In CVPR, 2011.","title":"Simulating human saccadic scanpaths on natural images","authors":"Wei Wang, Cheng Chen, Yizhou Wang, Tingting Jiang, Fang Fang, Yuan Yao","year":"2011","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wang et al., 2017] Yixiu Wang, Bin Wang, Xiaofeng Wu, and Liming Zhang. Scanpath estimation based on foveated image saliency. Cogn. Process., 18(1):87-95, 2017.","title":"Scanpath estimation based on foveated image saliency","authors":"Wang","year":"2017","venue":"Cogn. Process","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Xu et al., 2014] Juan Xu, Ming Jiang, Shuo Wang, Mo- han S. Kankanhalli, and Qi Zhao. Predicting human gaze beyond pixels. J. Vis., 14(1):28, 2014.","title":"Predicting human gaze beyond pixels","authors":"Juan Xu, Ming Jiang, Shuo Wang, Mohan S Kankanhalli, Qi Zhao","year":"2014","venue":"J. Vis","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Yu and Koltun, 2016] Fisher Yu and Vladlen Koltun. Multi- scale context aggregation by dilated convolutions. In ICLR, 2016.","title":"Multiscale context aggregation by dilated convolutions","authors":"Koltun ; Fisher Yu, Vladlen Yu, Koltun","year":"2016","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhang and Lin, 2013] Liming Zhang and Weisi Lin. Selec- tive visual attention: computational models and applica- tions. John Wiley & Sons, 2013.","title":"Selective visual attention: computational models and applications","authors":"Liming Zhang, Weisi Lin","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}