{"type":"ScholarlyArticle","title":"Open-Ended Long-Form Video Question Answering via Hierarchical Convolutional Self-Attention Networks","authors":[{"name":"Zhu Zhang","slug":"zhu-zhang","given":"Zhu","family":"Zhang","affiliation":"College of Computer Science , Zhejiang University , China"},{"name":"Zhou Zhao","slug":"zhou-zhao","given":"Zhou","family":"Zhao","affiliation":"College of Computer Science , Zhejiang University , China"},{"name":"Zhijie Lin","slug":"zhijie-lin","given":"Zhijie","family":"Lin","affiliation":"College of Computer Science , Zhejiang University , China"},{"name":"Jingkuan Song","slug":"jingkuan-song","given":"Jingkuan","family":"Song","affiliation":"University of Electronic Science and Technology of China , China"},{"name":"Xiaofei He","slug":"xiaofei-he","given":"Xiaofei","family":"He","affiliation":"State Key Lab of CAD&CG , Zhejiang University , China"}],"abstract":"Open-ended video question answering aims to automatically generate the natural-language answer from referenced video contents according to the given question. Currently, most existing approaches focus on short-form video question answering with multi-modal recurrent encoder-decoder networks. Although these works have achieved promising performance, they may still be ineffectively applied to long-form video question answering due to the lack of long-range dependency modeling and the suffering from the heavy computational cost. To tackle these problems, we propose a fast hierarchical convolutional self-attention encoder-decoder network. Concretely, we first develop a hierarchical convolutional self-attention encoder to efficiently model long-form video contents, which builds the hierarchical structure for video sequences and captures question-aware long-range dependencies from video context. We then devise a multi-scale attentive decoder to incorporate multi-layer video representations for answer generation, which avoids the information missing of the top encoder layer. The extensive experiments show the effectiveness and efficiency of our method.","doi":"10.24963/ijcai.2019/609","url":"https://ijcai.org/proceedings/2019/609","pdf":"https://ijcai.org/proceedings/2019/0609.pdf","markdown":"https://ijcai.org/proceedings/2019/609.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/609","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning M-Z","track":"Main track","number":609,"pages":{"first":4383,"last":4389},"keywords":["Machine Learning: Deep Learning","Computer Vision: Language and Vision"],"videos":[],"references":[{"position":1,"raw":"[Anderson et al., 2018] Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for im- age captioning and visual question answering. In CVPR, pages 6077-6086, 2018.","title":"Bottom-up and top-down attention for image captioning and visual question answering","authors":"Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, Lei Zhang","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Antol et al., 2015] Stanislaw Antol, Aishwarya Agrawal, Ji- asen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zit- nick, and Devi Parikh. Vqa: Visual question answering. In ICCV, pages 2425-2433, 2015.","title":"Vqa: Visual question answering","authors":"Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, Devi Parikh","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bahdanau et al., 2014] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014.","title":"Neural machine translation by jointly learning to align and translate","authors":"Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Fellbaum, 1998] Christiane Fellbaum. WordNet. Wiley On- line Library, 1998.","title":"WordNet","authors":null,"year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Gao and Han, 2017] Kun Gao and Yahong Han. Spatio- temporal context networks for video question answering. In Pacific Rim Conference on Multimedia, pages 108-118. Springer, 2017.","title":"Spatiotemporal context networks for video question answering","authors":"Han ; ] Kun Gao, Yahong Gao, Han","year":"2017","venue":"Pacific Rim Conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gao et al., 2018] Jiyang Gao, Runzhou Ge, Kan Chen, and Ram Nevatia. Motion-appearance co-memory networks for video question answering. CVPR, 2018.","title":"Motion-appearance co-memory networks for video question answering","authors":"Jiyang Gao, Runzhou Ge, Kan Chen, Ram Nevatia","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gehring et al., 2017] Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. Convo- lutional sequence to sequence learning. arXiv preprint arXiv:1705.03122, 2017.","title":"Convolutional sequence to sequence learning","authors":"Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, Yann N Dauphin","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Jang et al., 2017] Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. Tgif-qa: Toward spatio- temporal reasoning in visual question answering. In CVPR, pages 2680-8, 2017.","title":"Tgif-qa: Toward spatiotemporal reasoning in visual question answering","authors":"Jang","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Krishna et al., 2017] Ranjay Krishna, Kenji Hata, Fred- eric Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense- captioning events in videos. In ICCV, pages 706-715, 2017.","title":"Densecaptioning events in videos","authors":"Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, Juan Carlos Niebles","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li and Jia, 2016] Ruiyu Li and Jiaya Jia. Visual question answering with question representation update (qru). In NIPS, pages 4655-4663, 2016.","title":"Visual question answering with question representation update (qru)","authors":"Jia ; Li, Ruiyu Li, Jiaya Jia","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lu et al., 2016] Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh. Hierarchical question-image co-attention for visual question answering. In NIPS, pages 289-297, 2016.","title":"Hierarchical question-image co-attention for visual question answering","authors":"Lu","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Maharaj et al., 2017] Tegan Maharaj, Nicolas Ballas, Anna Rohrbach, Aaron C Courville, and Christopher Joseph Pal. A dataset and exploration of models for understanding video data through fill-in-the-blank question-answering. In CVPR, pages 7359-7368, 2017.","title":"A dataset and exploration of models for understanding video data through fill-in-the-blank question-answering","authors":"Tegan Maharaj, Nicolas Ballas, Anna Rohrbach, Aaron C Courville, Christopher Joseph Pal","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Malinowski and Fritz, 2014] Mateusz Malinowski and Mario Fritz. A multi-world approach to question answer- ing about real-world scenes based on uncertain input. In NIPS, pages 1682-1690, 2014.","title":"A multi-world approach to question answering about real-world scenes based on uncertain input","authors":"Mateusz Malinowski, Mario Fritz","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mazaheri et al., 2016] Amir Mazaheri, Dong Zhang, and Mubarak Shah. Video fill in the blank with merging lstms. arXiv preprint arXiv:1610.04062, 2016.","title":"Video fill in the blank with merging lstms","authors":"Amir Mazaheri, Dong Zhang, Mubarak Shah","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mikolov et al., 2013] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013.","title":"Efficient estimation of word representations in vector space","authors":"Mikolov","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Tapaswi et al., 2016] Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. Movieqa: Understanding stories in movies through question-answering. In CVPR, pages 4631-4640, 2016.","title":"Movieqa: Understanding stories in movies through question-answering","authors":"Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, Sanja Fidler","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Tran et al., 2015] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spa- tiotemporal features with 3d convolutional networks. In ICCV, pages 4489-4497, 2015.","title":"Learning spatiotemporal features with 3d convolutional networks","authors":"Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, Manohar Paluri","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, pages 5998-6008, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wu and Palmer, 1994] Zhibiao Wu and Martha Palmer. Verbs semantics and lexical selection. In Proceedings of the 32nd annual meeting on Association for Computa- tional Linguistics, pages 133-138. ACL, 1994.","title":"Verbs semantics and lexical selection","authors":"Palmer ; Wu, Zhibiao Wu, Martha Palmer","year":"1994","venue":"Proceedings of the 32nd annual meeting on Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wu et al., 2017] Qi Wu, Damien Teney, Peng Wang, Chun- hua Shen, Anthony Dick, and Anton van den Hengel. Visual question answering: A survey of methods and datasets. Computer Vision and Image Understanding, 163:21-40, 2017.","title":"Visual question answering: A survey of methods and datasets","authors":"Qi Wu, Damien Teney, Peng Wang, Chunhua Shen, Anthony Dick, Anton Van Den, Hengel","year":"2017","venue":"Computer Vision and Image Understanding","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Xue et al., 2017] Hongyang Xue, Zhou Zhao, and Deng Cai. Unifying the video and question attentions for open- ended video question answering. IEEE Transactions on Image Processing, 26(12):5656-5666, 2017.","title":"Unifying the video and question attentions for openended video question answering","authors":"Hongyang Xue, Zhou Zhao, Deng Cai","year":"2017","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zeng et al., 2017] Kuo-Hao Zeng, Tseng-Hung Chen, Ching-Yao Chuang, Yuan-Hong Liao, Juan Carlos Niebles, and Min Sun. Leveraging video descriptions to learn video question answering. In AAAI, pages 4334-4340, 2017.","title":"Leveraging video descriptions to learn video question answering","authors":"Kuo-Hao Zeng, Tseng-Hung Chen, Ching-Yao Chuang, Yuan-Hong Liao, Juan Carlos Niebles, Min Sun","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhao et al., 2018] Zhou Zhao, Zhu Zhang, Shuwen Xiao, Zhou Yu, Jun Yu, Deng Cai, Fei Wu, and Yueting Zhuang. Open-ended long-form video question answering via adaptive hierarchical reinforced networks. In IJCAI, pages 3683-3689, 2018.","title":"Open-ended long-form video question answering via adaptive hierarchical reinforced networks","authors":"Zhou Zhao, Zhu Zhang, Shuwen Xiao, Zhou Yu, Jun Yu, Deng Cai, Fei Wu, Yueting Zhuang","year":"2018","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2018/512"},{"position":24,"raw":"[Zhao et al., 2019] Zhou Zhao, Zhu Zhang, Xinghua Jiang, and Deng Cai. Multi-turn video question answering via hierarchical attention context reinforced networks. IEEE Transactions on Image Processing, 2019.","title":"Multi-turn video question answering via hierarchical attention context reinforced networks","authors":"Zhou Zhao, Zhu Zhang, Xinghua Jiang, Deng Cai","year":"2019","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhu et al., 2017] Linchao Zhu, Zhongwen Xu, Yi Yang, and Alexander G Hauptmann. Uncovering the temporal con- text for video question answering. IJCV, 124(3):409-421, 2017.","title":"Uncovering the temporal context for video question answering","authors":"Zhu","year":"2017","venue":"IJCV","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}