{"type":"ScholarlyArticle","title":"Video Question Answering via Hierarchical Spatio-Temporal Attention Networks","authors":[{"name":"Zhou Zhao","slug":"zhou-zhao","given":"Zhou","family":"Zhao","affiliation":"College of Computer Science , Zhejiang University"},{"name":"Qifan Yang","slug":"qifan-yang","given":"Qifan","family":"Yang","affiliation":"College of Computer Science , Zhejiang University"},{"name":"Deng Cai","slug":"deng-cai","given":"Deng","family":"Cai","affiliation":"State Key Lab of CAD&CG , Zhejiang University"},{"name":"Xiaofei He","slug":"xiaofei-he","given":"Xiaofei","family":"He","affiliation":"State Key Lab of CAD&CG , Zhejiang University"},{"name":"Yueting Zhuang","slug":"yueting-zhuang","given":"Yueting","family":"Zhuang","affiliation":"College of Computer Science , Zhejiang University"}],"abstract":"Open-ended video question answering is a challenging problem in visual information retrieval, which automatically generates the natural language answer from the referenced video content according to the question. However, the existing visual question answering works only focus on the static image, which may be ineffectively applied to video question answering due to the temporal dynamics of video contents. In this paper, we consider the problem of open-ended video question answering from the viewpoint of spatio-temporal attentional encoder-decoder learning framework. We propose the hierarchical spatio-temporal attention network for learning the joint representation of the dynamic video contents according to the given question. We then develop the encoder-decoder learning method with reasoning recurrent neural networks for open-ended video question answering. We construct a large-scale video question answering dataset. The extensive experiments show the effectiveness of our method.","doi":"10.24963/ijcai.2017/492","url":"https://ijcai.org/proceedings/2017/492","pdf":"https://ijcai.org/proceedings/2017/0492.pdf","markdown":"https://ijcai.org/proceedings/2017/492.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/492","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning S-Z","track":"Main track","number":492,"pages":{"first":3518,"last":3524},"keywords":["Machine Learning: Neural Networks","Natural Language Processing: Information Retrieval","Machine Learning: Data Mining"],"videos":[],"references":[{"position":1,"raw":"[Antol et al., 2015] Stanislaw Antol, Aishwarya Agrawal, Ji- asen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zit- nick, and Devi Parikh. Vqa: Visual question answering. In ICCV, pages 2425-2433, 2015.","title":"Vqa: Visual question answering","authors":"Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, Devi Parikh","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chung et al., 2014] Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evalua- tion of gated recurrent neural networks on sequence mod- eling. arXiv preprint arXiv:1412.3555, 2014.","title":"Empirical evaluation of gated recurrent neural networks on sequence modeling","authors":"Chung","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Fellbaum, 1998] Christiane Fellbaum. WordNet. Wiley On- line Library, 1998.","title":"WordNet","authors":null,"year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Fernando and Gould, 2016] Basura Fernando and Stephen Gould. Learning end-to-end video classification with rank- pooling. In ICML), 2016.","title":"Learning end-to-end video classification with rankpooling","authors":"Basura Fernando, Stephen Gould","year":"2016","venue":"ICML)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Heilman and Smith, 2010] Michael Heilman and Noah A Smith. Good question! statistical ranking for question gen- eration. In Human Language Technologies: The 2010 An- nual Conference of the North American Chapter of the As- sociation for Computational Linguistics, pages 609-617. ACL, 2010.","title":"Good question! statistical ranking for question generation","authors":"Smith ; Heilman, Michael Heilman, Noah A Smith","year":"2010","venue":"Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Kim et al., 2016] Jin-Hwa Kim, Sang-Woo Lee, Dong- Hyun Kwak, Min-Oh Heo, Jeonghee Kim, Jung-Woo Ha, and Byoung-Tak Zhang. Multimodal residual learning for visual qa. NIPS, 2016.","title":"Multimodal residual learning for visual qa","authors":"Jin-Hwa Kim, Sang-Woo Lee, Dong-Hyun Kwak, Min-Oh Heo, Jeonghee Kim, Jung-Woo Ha, Byoung-Tak Zhang","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Kumar and Irsoy, 2015] Ankit Kumar and Ozan Irsoy. Ask me anything: Dynamic memory networks for natural lan- guage processing. 2015.","title":"Ask me anything: Dynamic memory networks for natural language processing","authors":"Ankit Kumar, Ozan Irsoy","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Li and Jia, 2016] Ruiyu Li and Jiaya Jia. Visual question answering with question representation update (qru). In NIPS, pages 4655-4663, 2016.","title":"Visual question answering with question representation update (qru)","authors":"Jia ; Li, Ruiyu Li, Jiaya Jia","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Li et al., 2016] Yuncheng Li, Yale Song, Liangliang Cao, Joel Tetreault, Larry Goldberg, Alejandro Jaimes, and Jiebo Luo. Tgif: A new dataset and benchmark on ani- mated gif description. In CVPR, pages 4641-4650, 2016.","title":"Tgif: A new dataset and benchmark on animated gif description","authors":"Yuncheng Li, Yale Song, Liangliang Cao, Joel Tetreault, Larry Goldberg, Alejandro Jaimes, Jiebo Luo","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lu et al., 2016] Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh. Hierarchical question-image co-attention for visual question answering. In NIPS, pages 289-297, 2016.","title":"Hierarchical question-image co-attention for visual question answering","authors":"Lu","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Malinowski and Fritz, 2014] Mateusz Malinowski and Mario Fritz. A multi-world approach to question answer- ing about real-world scenes based on uncertain input. In NIPS, pages 1682-1690, 2014.","title":"A multi-world approach to question answering about real-world scenes based on uncertain input","authors":"Mateusz Malinowski, Mario Fritz","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Mazaheri et al., 2016] Amir Mazaheri, Dong Zhang, and Mubarak Shah. Video fill in the blank with merging lstms. arXiv preprint arXiv:1610.04062, 2016.","title":"Video fill in the blank with merging lstms","authors":"Amir Mazaheri, Dong Zhang, Mubarak Shah","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Mikolov et al., 2013] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013.","title":"Efficient estimation of word representations in vector space","authors":"Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Shih et al., 2016] Kevin J Shih, Saurabh Singh, and Derek Hoiem. Where to look: Focus regions for visual question answering. In CVPR, pages 4613-4621, 2016.","title":"Where to look: Focus regions for visual question answering","authors":"Kevin J Shih, Saurabh Singh, Derek Hoiem","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Sukhbaatar et al., 2015] Sainbayar Sukhbaatar, Jason We- ston, Rob Fergus, et al. End-to-end memory networks. In NIPS, pages 2440-2448, 2015.","title":"End-to-end memory networks","authors":"Sukhbaatar","year":"2015","venue":"Sainbayar Sukhbaatar","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Tapaswi et al., 2016] Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. Movieqa: Understanding stories in movies through question-answering. In CVPR, pages 4631-4640, 2016.","title":"Movieqa: Understanding stories in movies through question-answering","authors":"Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, Sanja Fidler","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Tu et al., 2014] Kewei Tu, Meng Meng, Mun Wai Lee, Tae Eun Choe, and Song-Chun Zhu. Joint video and text parsing for understanding events and answering queries. IEEE MultiMedia, 21(2):42-70, 2014.","title":"Joint video and text parsing for understanding events and answering queries","authors":"Kewei Tu, Meng Meng, Mun Wai Lee, Tae, Eun Choe, Song-Chun Zhu","year":"2014","venue":"IEEE MultiMedia","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wu and Palmer, 1994] Zhibiao Wu and Martha Palmer. Verbs semantics and lexical selection. In Proceedings of the 32nd annual meeting on Association for Computa- tional Linguistics, pages 133-138. ACL, 1994.","title":"Verbs semantics and lexical selection","authors":"Palmer ; Wu, Zhibiao Wu, Martha Palmer","year":"1994","venue":"Proceedings of the 32nd annual meeting on Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wu et al., 2016] Qi Wu, Damien Teney, Peng Wang, Chun- hua Shen, Anthony Dick, and Anton van den Hengel. Visual question answering: A survey of methods and datasets. arXiv preprint arXiv:1607.05910, 2016.","title":"Visual question answering: A survey of methods and datasets","authors":"Qi Wu, Damien Teney, Peng Wang, Chunhua Shen, Anthony Dick, Anton Van Den, Hengel","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Xiong et al., 2016] Caiming Xiong, Stephen Merity, and Richard Socher. Dynamic memory networks for visual and textual question answering. ICML, 1603, 2016.","title":"Dynamic memory networks for visual and textual question answering","authors":"Xiong","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Xu et al., 2015] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C Courville, Ruslan Salakhutdi- nov, Richard S Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual at- tention. In ICML, volume 14, pages 77-81, 2015.","title":"Show, attend and tell: Neural image caption generation with visual attention","authors":"Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C Courville, Ruslan Salakhutdinov, Richard S Zemel, Yoshua Bengio","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yang et al., 2016] Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alex Smola. Stacked attention net- works for image question answering. In CVPR, pages 21- 29, 2016.","title":"Stacked attention networks for image question answering","authors":"Yang","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yao et al., 2015] Li Yao, Atousa Torabi, Kyunghyun Cho, Nicolas Ballas, Christopher Pal, Hugo Larochelle, and Aaron Courville. Describing videos by exploiting tempo- ral structure. In ICCV, pages 4507-4515, 2015.","title":"Describing videos by exploiting temporal structure","authors":"Li Yao, Atousa Torabi, Kyunghyun Cho, Nicolas Ballas, Christopher Pal, Hugo Larochelle, Aaron Courville","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yu et al., 2015] Licheng Yu, Eunbyung Park, Alexander C Berg, and Tamara L Berg. Visual madlibs: Fill in the blank description generation and question answering. In ICCV, pages 2461-2469, 2015.","title":"Visual madlibs: Fill in the blank description generation and question answering","authors":"Licheng Yu, Eunbyung Park, Alexander C Berg, Tamara L Berg","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhao et al., 2015] Zhou Zhao, Lijun Zhang, Xiaofei He, and Wilfred Ng. Expert finding for question answering via graph regularized matrix completion. TKDE, 27(4):993- 1004, 2015.","title":"Expert finding for question answering via graph regularized matrix completion","authors":"Zhou Zhao, Lijun Zhang, Xiaofei He, Wilfred Ng","year":"2015","venue":"TKDE","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhao et al., 2016] Zhou Zhao, Qifan Yang, Deng Cai, Xi- aofei He, and Yueting Zhuang. Expert finding for community-based question answering via ranking metric network learning. In IJCAI, pages 3000-3006. AAAI Press, 2016.","title":"Expert finding for community-based question answering via ranking metric network learning","authors":"Zhou Zhao, Qifan Yang, Deng Cai, Xiaofei He, Yueting Zhuang","year":"2016","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2016/426"},{"position":28,"raw":"[Zhu et al., 2015] Linchao Zhu, Zhongwen Xu, Yi Yang, and Alexander G Hauptmann. Uncovering temporal con- text for video question and answering. arXiv preprint arXiv:1511.04670, 2015.","title":"Uncovering temporal context for video question and answering","authors":"Zhu","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}