{"type":"ScholarlyArticle","title":"Multi-Turn Video Question Answering via Multi-Stream Hierarchical Attention Context Network","authors":[{"name":"Zhou Zhao","slug":"zhou-zhao","given":"Zhou","family":"Zhao","affiliation":"College of Computer Science , Zhejiang University"},{"name":"Xinghua Jiang","slug":"xinghua-jiang","given":"Xinghua","family":"Jiang","affiliation":"College of Computer Science , Zhejiang University"},{"name":"Deng Cai","slug":"deng-cai","given":"Deng","family":"Cai","affiliation":"State Key Lab of CAD&CG , Zhejiang University"},{"name":"Jun Xiao","slug":"jun-xiao","given":"Jun","family":"Xiao","affiliation":"College of Computer Science , Zhejiang University"},{"name":"Xiaofei He","slug":"xiaofei-he","given":"Xiaofei","family":"He","affiliation":"State Key Lab of CAD&CG , Zhejiang University"},{"name":"Shiliang Pu","slug":"shiliang-pu","given":"Shiliang","family":"Pu","affiliation":"Hikvision Research Institute"}],"abstract":"Conversational video question answering is a challenging task in visual information retrieval, which generates the accurate answer from the referenced video contents according to the visual conversation context and given question. However, the existing visual question answering methods mainly tackle the problem of single-turn video question answering, which may be ineffectively applied for multi-turn video question answering directly, due to the insufficiency of modeling the sequential conversation context. In this paper, we study the problem of multi-turn video question answering from the viewpoint of multi-step hierarchical attention context network learning. We first propose the hierarchical attention context network for context-aware question understanding by modeling the hierarchically sequential conversation context structure. We then develop the multi-stream spatio-temporal attention network for learning the joint representation of the dynamic video contents and context-aware question embedding. We next devise the hierarchical attention context network learning method with multi-step reasoning process for multi-turn video question answering. We construct two large-scale multi-turn video question answering datasets. The extensive experiments show the effectiveness of our method.","doi":"10.24963/ijcai.2018/513","url":"https://ijcai.org/proceedings/2018/513","pdf":"https://ijcai.org/proceedings/2018/0513.pdf","markdown":"https://ijcai.org/proceedings/2018/513.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/513","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Machine Learning Applications","track":"Main track","number":513,"pages":{"first":3690,"last":3696},"keywords":["Machine Learning: Data Mining","Machine Learning Applications: Applications of Supervised Learning"],"videos":[],"references":[{"position":1,"raw":"[Antol et al., 2015] Stanislaw Antol, Aishwarya Agrawal, Ji- asen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zit- nick, and Devi Parikh. Vqa: Visual question answering. In ICCV, pages 2425-2433, 2015.","title":"Vqa: Visual question answering","authors":"Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, Devi Parikh","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen and Dolan, 2011] David L Chen and William B Dolan. Collecting highly parallel data for paraphrase eval- uation. In ACL: Human Language Technologies-Volume 1, pages 190-200. ACL, 2011.","title":"Collecting highly parallel data for paraphrase evaluation","authors":"L David, William B Chen, Dolan","year":"2011","venue":"ACL: Human Language Technologies","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Das et al., 2017] Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José MF Moura, Devi Parikh, and Dhruv Batra. Visual dialog. CVPR, 2017.","title":"Visual dialog","authors":"Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, M F José, Devi Moura, Dhruv Parikh, Batra","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735-1780, 1997.","title":"Long short-term memory","authors":"Schmidhuber Hochreiter, Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Jang et al., 2017] Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. Tgif-qa: Toward spatio- temporal reasoning in visual question answering. CVPR, 2017.","title":"Tgif-qa: Toward spatiotemporal reasoning in visual question answering","authors":"Jang","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"Kim et al., ] Jin-Hwa Kim, Sang-Woo Lee, Donghyun Kwak, Min-Oh Heo, Jeonghee Kim, Jung-Woo Ha, and Byoung-Tak Zhang. Multimodal residual learning for vi- sual qa. In NIPS.","title":"Multimodal residual learning for visual qa","authors":"Kim","year":null,"venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"Krizhevsky et al., ] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep con- volutional neural networks. In NIPS.","title":"Imagenet classification with deep convolutional neural networks","authors":"Krizhevsky","year":null,"venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"Li and Jia, ] Ruiyu Li and Jiaya Jia. Visual question answer- ing with question representation update (qru). In NIPS.","title":"Visual question answering with question representation update (qru)","authors":"Jia Li, ] Ruiyu Li, Jiaya Jia","year":null,"venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Mazaheri et al., 2016] Amir Mazaheri, Dong Zhang, and Mubarak Shah. Video fill in the blank with merging lstms. arXiv preprint arXiv:1610.04062, 2016.","title":"Video fill in the blank with merging lstms","authors":"Amir Mazaheri, Dong Zhang, Mubarak Shah","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Mei et al., 2017] Hongyuan Mei, Mohit Bansal, and Matthew R Walter. Coherent dialogue with attention- based language models. In AAAI, pages 3252-3258, 2017.","title":"Coherent dialogue with attentionbased language models","authors":"Hongyuan Mei, Mohit Bansal, Matthew R Walter","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Mikolov et al., 2013] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013.","title":"Efficient estimation of word representations in vector space","authors":"Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Orr and Müller, 2003] Genevieve B Orr and Klaus-Robert Müller. Neural networks: tricks of the trade. Springer, 2003.","title":"Neural networks: tricks of the trade","authors":"Genevieve B Orr, Klaus-Robert Müller","year":"2003","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"Pennington et al., ] Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In EMNLP.","title":"Glove: Global vectors for word representation","authors":"Pennington","year":null,"venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"Rohrbach et al., ] Anna Rohrbach, Marcus Rohrbach, Wei Qiu, Annemarie Friedrich, Manfred Pinkal, and Bernt Schiele. Coherent multi-sentence video description with variable level of detail. In German conference on pattern recognition.","title":"Coherent multi-sentence video description with variable level of detail","authors":"Rohrbach","year":null,"venue":"German conference on pattern recognition","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Serban et al., 2016] Iulian Vlad Serban, Alessandro Sor- doni, Yoshua Bengio, Aaron C Courville, and Joelle Pineau. Building end-to-end dialogue systems using gen- erative hierarchical neural network models. In AAAI, pages 3776-3784, 2016.","title":"Building end-to-end dialogue systems using generative hierarchical neural network models","authors":"Iulian Vlad Serban, Alessandro Sordoni, Yoshua Bengio, Aaron C Courville, Joelle Pineau","year":"2016","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Serban et al., 2017a] Iulian Vlad Serban, Tim Klinger, Ger- ald Tesauro, Kartik Talamadupula, Bowen Zhou, Yoshua Bengio, and Aaron C Courville. Multiresolution recurrent neural networks: An application to dialogue response gen- eration. In AAAI, pages 3288-3294, 2017.","title":"Multiresolution recurrent neural networks: An application to dialogue response generation","authors":"Serban","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Serban et al., 2017b] Iulian Vlad Serban, Alessandro Sor- doni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron C Courville, and Yoshua Bengio. A hierarchical latent vari- able encoder-decoder model for generating dialogues. In AAAI, pages 3295-3301, 2017.","title":"A hierarchical latent variable encoder-decoder model for generating dialogues","authors":"Serban","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Shih et al., 2016] Kevin J Shih, Saurabh Singh, and Derek Hoiem. Where to look: Focus regions for visual question answering. In CVPR, pages 4613-4621, 2016.","title":"Where to look: Focus regions for visual question answering","authors":"Kevin J Shih, Saurabh Singh, Derek Hoiem","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"Sukhbaatar et al., ] Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al. End-to-end memory networks. In NIPS.","title":"End-to-end memory networks","authors":"Sukhbaatar","year":null,"venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Tapaswi et al., 2016] Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. Movieqa: Understanding stories in movies through question-answering. In CVPR, pages 4631-4640, 2016.","title":"Movieqa: Understanding stories in movies through question-answering","authors":"Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, Sanja Fidler","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Tran et al., 2015] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spa- tiotemporal features with 3d convolutional networks. In ICCV, pages 4489-4497, 2015.","title":"Learning spatiotemporal features with 3d convolutional networks","authors":"Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, Manohar Paluri","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Weston, 2016] Jason E Weston. Dialog-based language learning. In NIPS, pages 829-837, 2016.","title":"Dialog-based language learning","authors":"Jason E Weston","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wu et al., 2016] Qi Wu, Damien Teney, Peng Wang, Chun- hua Shen, Anthony Dick, and Anton van den Hengel. Visual question answering: A survey of methods and datasets. arXiv preprint arXiv:1607.05910, 2016.","title":"Visual question answering: A survey of methods and datasets","authors":"Qi Wu, Damien Teney, Peng Wang, Chunhua Shen, Anthony Dick, Anton Van Den, Hengel","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yang et al., 2016] Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alex Smola. Stacked attention net- works for image question answering. In CVPR, pages 21- 29, 2016.","title":"Stacked attention networks for image question answering","authors":"Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Smola","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zeng et al., 2017] Kuo-Hao Zeng, Tseng-Hung Chen, Ching-Yao Chuang, Yuan-Hong Liao, Juan Carlos Niebles, and Min Sun. Leveraging video descriptions to learn video question answering. In AAAI, pages 4334-4340, 2017.","title":"Leveraging video descriptions to learn video question answering","authors":"Kuo-Hao Zeng, Tseng-Hung Chen, Ching-Yao Chuang, Yuan-Hong Liao, Juan Carlos Niebles, Min Sun","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhao et al., 2017] Zhou Zhao, Qifan Yang, Deng Cai, Xi- aofei He, and Yueting Zhuang. Video question answering via hierarchical spatio-temporal attention networks. In IJ- CAI, volume 2, 2017.","title":"Video question answering via hierarchical spatio-temporal attention networks","authors":"Zhou Zhao, Qifan Yang, Deng Cai, Xiaofei He, Yueting Zhuang","year":"2017","venue":"IJ-CAI","doi":null,"url":null,"internal_url":"/proceedings/2017/492"},{"position":28,"raw":"[Zhou et al., 2015] Xiaoqiang Zhou, Baotian Hu, Qingcai Chen, Buzhou Tang, and Xiaolong Wang. Answer se- quence learning with neural networks for answer selection in community question answering. ACL, 2015.","title":"Answer sequence learning with neural networks for answer selection in community question answering","authors":"Xiaoqiang Zhou, Baotian Hu, Qingcai Chen, Buzhou Tang, Xiaolong Wang","year":"2015","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhu et al., 2015] Linchao Zhu, Zhongwen Xu, Yi Yang, and Alexander G Hauptmann. Uncovering temporal context for video question and answering. IJCV, 2015.","title":"Uncovering temporal context for video question and answering","authors":"Linchao Zhu, Zhongwen Xu, Yi Yang, Alexander G Hauptmann","year":"2015","venue":"IJCV","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}