{"type":"ScholarlyArticle","title":"Open-Ended Long-form Video Question Answering via Adaptive Hierarchical Reinforced Networks","authors":[{"name":"Zhou Zhao","slug":"zhou-zhao","given":"Zhou","family":"Zhao","affiliation":"College of Computer Science , Zhejiang University"},{"name":"Zhu Zhang","slug":"zhu-zhang","given":"Zhu","family":"Zhang","affiliation":"College of Computer Science , Zhejiang University"},{"name":"Shuwen Xiao","slug":"shuwen-xiao","given":"Shuwen","family":"Xiao","affiliation":"College of Computer Science , Zhejiang University"},{"name":"Zhou Yu","slug":"zhou-yu","given":"Zhou","family":"Yu","affiliation":"College of Computer Science , Hangzhou Dianzi University"},{"name":"Jun Yu","slug":"jun-yu","given":"Jun","family":"Yu","affiliation":"College of Computer Science , Hangzhou Dianzi University"},{"name":"Deng Cai","slug":"deng-cai","given":"Deng","family":"Cai","affiliation":"State Key Lab of CAD&CG , Zhejiang University"},{"name":"Fei Wu","slug":"fei-wu","given":"Fei","family":"Wu","affiliation":"College of Computer Science , Zhejiang University"},{"name":"Yueting Zhuang","slug":"yueting-zhuang","given":"Yueting","family":"Zhuang","affiliation":"College of Computer Science , Zhejiang University"}],"abstract":"Open-ended long-form video question answering is challenging problem in visual information retrieval, which automatically generates the natural language answer from the referenced long-form video content according to the question. However, the existing video question answering works mainly focus on the short-form video question answering, due to the lack of modeling the semantic representation of long-form video contents. In this paper, we consider the problem of long-form video question answering from the viewpoint of adaptive hierarchical reinforced encoder-decoder network learning. We propose the adaptive hierarchical encoder network to learn the joint representation of the long-form video contents according to the question with adaptive video segmentation. we then develop the reinforced decoder network to generate the natural language answer for open-ended video question answering. We construct a large-scale long-form video question answering dataset. The extensive experiments show the effectiveness of our method.","doi":"10.24963/ijcai.2018/512","url":"https://ijcai.org/proceedings/2018/512","pdf":"https://ijcai.org/proceedings/2018/0512.pdf","markdown":"https://ijcai.org/proceedings/2018/512.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/512","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Machine Learning Applications","track":"Main track","number":512,"pages":{"first":3683,"last":3689},"keywords":["Machine Learning Applications: Applications of Supervised Learning","Machine Learning: Data Mining"],"videos":[],"references":[{"position":1,"raw":"[Antol et al., 2015] Stanislaw Antol, Aishwarya Agrawal, Ji- asen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zit- nick, and Devi Parikh. Vqa: Visual question answering. In ICCV, pages 2425-2433, 2015.","title":"Vqa: Visual question answering","authors":"Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, Devi Parikh","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bahdanau et al., 2016] Dzmitry Bahdanau, Philemon Brakel, Kelvin Xu, Anirudh Goyal, Ryan Lowe, Joelle Pineau, Aaron Courville, and Yoshua Bengio. An actor- critic algorithm for sequence prediction. arXiv preprint arXiv:1607.07086, 2016.","title":"An actorcritic algorithm for sequence prediction","authors":"Dzmitry Bahdanau, Philemon Brakel, Kelvin Xu, Anirudh Goyal, Ryan Lowe, Joelle Pineau, Aaron Courville, Yoshua Bengio","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bengio et al., 2013] Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432, 2013.","title":"Estimating or propagating gradients through stochastic neurons for conditional computation","authors":"Yoshua Bengio, Nicholas Léonard, Aaron Courville","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Fellbaum, 1998] Christiane Fellbaum. WordNet. Wiley On- line Library, 1998.","title":"WordNet","authors":null,"year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Heilbron et al., 2015] Fabian Caba Heilbron, Victor Escor- cia, Bernard Ghanem, and Juan Carlos Niebles. Activi- tynet: A large-scale video benchmark for human activity understanding. In CVPR, pages 961-970. IEEE, 2015.","title":"Activitynet: A large-scale video benchmark for human activity understanding","authors":"Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, Juan Carlos Niebles","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Heilman and Smith, 2010] Michael Heilman and Noah A Smith. Good question! statistical ranking for question gen- eration. In Human Language Technologies: The 2010 An- nual Conference of the North American Chapter of the As- sociation for Computational Linguistics, pages 609-617. ACL, 2010.","title":"Good question! statistical ranking for question generation","authors":"Smith ; Heilman, Michael Heilman, Noah A Smith","year":"2010","venue":"Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735-1780, 1997.","title":"Long short-term memory","authors":"Schmidhuber Hochreiter, Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Jang et al., 2017] Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. Tgif-qa: Toward spatio- temporal reasoning in visual question answering. In CVPR 2017, pages 2680-8, 2017.","title":"Tgif-qa: Toward spatiotemporal reasoning in visual question answering","authors":"Jang","year":"2017","venue":"CVPR 2017","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Krishnan and Sitaraman, 2013] S Shunmuga Krishnan and Ramesh K Sitaraman. Understanding the effectiveness of video ads: a measurement study. In Proceedings of the 2013 conference on Internet measurement conference, pages 149-162. ACM, 2013.","title":"Understanding the effectiveness of video ads: a measurement study","authors":"Sitaraman ; S Shunmuga Krishnan, Ramesh K Krishnan, Sitaraman","year":"2013","venue":"Proceedings of the 2013 conference on Internet measurement conference","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lezama et al., 2011] José Lezama, Karteek Alahari, Josef Sivic, and Ivan Laptev. Track to the future: Spatio- temporal video segmentation with long-range motion cues. In CVPR, pages 3369-3376. IEEE, 2011.","title":"Track to the future: Spatiotemporal video segmentation with long-range motion cues","authors":"Lezama","year":"2011","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li and Jia, 2016] Ruiyu Li and Jiaya Jia. Visual question answering with question representation update (qru). In NIPS, pages 4655-4663, 2016.","title":"Visual question answering with question representation update (qru)","authors":"Jia ; Li, Ruiyu Li, Jiaya Jia","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lu et al., 2016] Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh. Hierarchical question-image co-attention for visual question answering. In NIPS, pages 289-297, 2016.","title":"Hierarchical question-image co-attention for visual question answering","authors":"Lu","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Malinowski and Fritz, 2014] Mateusz Malinowski and Mario Fritz. A multi-world approach to question answer- ing about real-world scenes based on uncertain input. In NIPS, pages 1682-1690, 2014.","title":"A multi-world approach to question answering about real-world scenes based on uncertain input","authors":"Mateusz Malinowski, Mario Fritz","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mazaheri et al., 2016] Amir Mazaheri, Dong Zhang, and Mubarak Shah. Video fill in the blank with merging lstms. arXiv preprint arXiv:1610.04062, 2016.","title":"Video fill in the blank with merging lstms","authors":"Amir Mazaheri, Dong Zhang, Mubarak Shah","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Pan et al., 2016] Pingbo Pan, Zhongwen Xu, Yi Yang, Fei Wu, and Yueting Zhuang. Hierarchical recurrent neural encoder for video representation with application to cap- tioning. In CVPR, pages 1029-1038, 2016.","title":"Hierarchical recurrent neural encoder for video representation with application to captioning","authors":"Pan","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Ren et al., 2017] Zhou Ren, Xiaoyu Wang, Ning Zhang, Xutao Lv, and Li-Jia Li. Deep reinforcement learning- based image captioning with embedding reward. CVPR, 2017.","title":"Deep reinforcement learningbased image captioning with embedding reward","authors":"Xiaoyu Zhou Ren, Ning Wang, Xutao Zhang, Li-Jia Lv, Li","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Shih et al., 2016] Kevin J Shih, Saurabh Singh, and Derek Hoiem. Where to look: Focus regions for visual question answering. In CVPR, pages 4613-4621, 2016.","title":"Where to look: Focus regions for visual question answering","authors":"Kevin J Shih, Saurabh Singh, Derek Hoiem","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Tapaswi et al., 2016] Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. Movieqa: Understanding stories in movies through question-answering. In CVPR, pages 4631-4640, 2016.","title":"Movieqa: Understanding stories in movies through question-answering","authors":"Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, Sanja Fidler","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wu and Palmer, 1994] Zhibiao Wu and Martha Palmer. Verbs semantics and lexical selection. In Proceedings of the 32nd annual meeting on Association for Computa- tional Linguistics, pages 133-138. ACL, 1994.","title":"Verbs semantics and lexical selection","authors":"Palmer ; Wu, Zhibiao Wu, Martha Palmer","year":"1994","venue":"Proceedings of the 32nd annual meeting on Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wu et al., 2016] Qi Wu, Damien Teney, Peng Wang, Chun- hua Shen, Anthony Dick, and Anton van den Hengel. Visual question answering: A survey of methods and datasets. arXiv preprint arXiv:1607.05910, 2016.","title":"Visual question answering: A survey of methods and datasets","authors":"Qi Wu, Damien Teney, Peng Wang, Chunhua Shen, Anthony Dick, Anton Van Den, Hengel","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Yang et al., 2016] Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alex Smola. Stacked attention net- works for image question answering. In CVPR, pages 21- 29, 2016.","title":"Stacked attention networks for image question answering","authors":"Yang","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yao et al., 2015] Li Yao, Atousa Torabi, Kyunghyun Cho, Nicolas Ballas, Christopher Pal, Hugo Larochelle, and Aaron Courville. Describing videos by exploiting tempo- ral structure. In ICCV, pages 4507-4515, 2015.","title":"Describing videos by exploiting temporal structure","authors":"Li Yao, Atousa Torabi, Kyunghyun Cho, Nicolas Ballas, Christopher Pal, Hugo Larochelle, Aaron Courville","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zeng et al., 2016] Kuo-Hao Zeng, Tseng-Hung Chen, Ching-Yao Chuang, Yuan-Hong Liao, Juan Carlos Niebles, and Min Sun. Leveraging video descriptions to learn video question answering. arXiv preprint arXiv:1611.04021, 2016.","title":"Leveraging video descriptions to learn video question answering","authors":"Kuo-Hao Zeng, Tseng-Hung Chen, Ching-Yao Chuang, Yuan-Hong Liao, Juan Carlos Niebles, Min Sun","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zeng et al., 2017] Kuo-Hao Zeng, Tseng-Hung Chen, Ching-Yao Chuang, Yuan-Hong Liao, Juan Carlos Niebles, and Min Sun. Leveraging video descriptions to learn video question answering. In AAAI, pages 4334-4340, 2017.","title":"Leveraging video descriptions to learn video question answering","authors":"Kuo-Hao Zeng, Tseng-Hung Chen, Ching-Yao Chuang, Yuan-Hong Liao, Juan Carlos Niebles, Min Sun","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhao et al., 2017] Zhou Zhao, Qifan Yang, Deng Cai, Xi- aofei He, and Yueting Zhuang. Video question answering via hierarchical spatio-temporal attention networks. In In- ternational Joint Conference on Artificial Intelligence (IJ- CAI), volume 2, 2017.","title":"Video question answering via hierarchical spatio-temporal attention networks","authors":"Zhou Zhao, Qifan Yang, Deng Cai, Xiaofei He, Yueting Zhuang","year":"2017","venue":"ternational Joint Conference on Artificial Intelligence (IJ-CAI)","doi":null,"url":null,"internal_url":"/proceedings/2017/492"},{"position":27,"raw":"[Zhu et al., 2015] Linchao Zhu, Zhongwen Xu, Yi Yang, and Alexander G Hauptmann. Uncovering temporal con- text for video question and answering. arXiv preprint arXiv:1511.04670, 2015.","title":"Uncovering temporal context for video question and answering","authors":"Zhu","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}