{"type":"ScholarlyArticle","title":"Multilevel Hierarchical Network with Multiscale Sampling for Video Question Answering","authors":[{"name":"Min Peng","slug":"min-peng","given":"Min","family":"Peng","affiliation":"University of Chinese Academy of Sciences; Chongqing Institute of Green and Intelligent Technology , Chinese Academy of Sciences"},{"name":"Chongyang Wang","slug":"chongyang-wang","given":"Chongyang","family":"Wang","affiliation":"University College London"},{"name":"Yuan Gao","slug":"yuan-gao","given":"Yuan","family":"Gao","affiliation":"Shenzhen Institute of Artificial Intelligence and Robotics for Society"},{"name":"Yu Shi","slug":"yu-shi","given":"Yu","family":"Shi","affiliation":"Chongqing Institute of Green and Intelligent Technology , Chinese Academy of Sciences"},{"name":"Xiang-Dong Zhou","slug":"xiang-dong-zhou","given":"Xiang-Dong","family":"Zhou","affiliation":"Chongqing Institute of Green and Intelligent Technology , Chinese Academy of Sciences"}],"abstract":"Video question answering (VideoQA) is challenging given its multimodal combination of visual understanding and natural language processing. While most existing approaches ignore the visual appearance-motion information at different temporal scales, it is unknown how to incorporate the multilevel processing capacity of a deep learning model with such multiscale information. Targeting these issues, this paper proposes a novel Multilevel Hierarchical Network (MHN) with multiscale sampling for VideoQA. MHN comprises two modules, namely Recurrent Multimodal Interaction (RMI) and Parallel Visual Reasoning (PVR). With a multiscale sampling, RMI iterates the interaction of appearance-motion information at each scale and the question embeddings to build the multilevel question-guided visual representations. Thereon, with a shared transformer encoder, PVR infers the visual cues at each level in parallel to fit with answering different question types that may rely on the visual information at relevant levels. Through extensive experiments on three VideoQA datasets, we demonstrate improved performances than previous state-of-the-arts and justify the effectiveness of each part of our method.","doi":"10.24963/ijcai.2022/178","url":"https://ijcai.org/proceedings/2022/178","pdf":"https://ijcai.org/proceedings/2022/0178.pdf","markdown":"https://ijcai.org/proceedings/2022/178.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/178","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":178,"pages":{"first":1276,"last":1282},"keywords":["Natural Language Processing: Question Answering","Computer Vision: Video analysis and understanding","Computer Vision: Vision and language","Computer Vision: Scene analysis and understanding","Machine Learning: Multi-modal learning"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986603","length_sec":85,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984746","length_sec":354,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Cai et al., 2020] Jiayin Cai, Chun Yuan, Cheng Shi, Lei Li, Yangyang Cheng, and Ying Shan. Feature augmented memory with global attention network for videoqa. In IJCAI'20, pages 998-1004, 7 2020. Main track.","title":"Feature augmented memory with global attention network for videoqa","authors":"Jiayin Cai, Chun Yuan, Cheng Shi, Lei Li, Yangyang Cheng, Ying Shan","year":"2020","venue":"IJCAI'20","doi":null,"url":null,"internal_url":"/proceedings/2020/139"},{"position":2,"raw":"[Chowdhury et al., 2018] Muhammad Iqbal Hasan Chowdhury, Kien Nguyen, Sridha Sridharan, and Clinton Fookes. Hierar- chical relational attention for video question answering. In 2018 25th IEEE International Conference on Image Processing (ICIP), pages 599-603. IEEE, 2018.","title":"Hierarchical relational attention for video question answering","authors":"Muhammad Iqbal, Hasan Chowdhury, Kien Nguyen, Sridha Sridharan, Clinton Fookes","year":"2018","venue":"25th IEEE International Conference on Image Processing (ICIP)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Dai et al., 2021] Zhigang Dai, Bolun Cai, Yugeng Lin, and Juny- ing Chen. Up-detr: Unsupervised pre-training for object detec- tion with transformers. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR), pages 1601-1610, June 2021.","title":"Up-detr: Unsupervised pre-training for object detection with transformers","authors":"Zhigang Dai, Bolun Cai, Yugeng Lin, Junying Chen","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dang et al., 2021] Long Hoang Dang, Thao Minh Le, Vuong Le, and Truyen Tran. Hierarchical object-oriented spatio-temporal reasoning for video question answering. In IJCAI'21, pages 636- 642, 8 2021.","title":"Hierarchical object-oriented spatio-temporal reasoning for video question answering","authors":"Long Hoang Dang, Thao, Minh Le, Vuong Le, Truyen Tran","year":"2021","venue":"IJCAI'21","doi":null,"url":null,"internal_url":"/proceedings/2021/88"},{"position":5,"raw":"[Fan et al., 2019] Chenyou Fan, Xiaofan Zhang, Shu Zhang, Wen- sheng Wang, Chi Zhang, and Heng Huang. Heterogeneous mem- ory enhanced multimodal attention model for video question an- swering. In CVPR'19, pages 1999-2007, 2019.","title":"Heterogeneous memory enhanced multimodal attention model for video question answering","authors":"Chenyou Fan, Xiaofan Zhang, Shu Zhang, Wensheng Wang, Chi Zhang, Heng Huang","year":"2019","venue":"CVPR'19","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gao et al., 2018] Jiyang Gao, Runzhou Ge, Kan Chen, and Ram Nevatia. Motion-appearance co-memory networks for video question answering. In CVPR'18, pages 6576-6585, 2018.","title":"Motion-appearance co-memory networks for video question answering","authors":"Jiyang Gao, Runzhou Ge, Kan Chen, Ram Nevatia","year":"2018","venue":"CVPR'18","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gao et al., 2021] Chenyu Gao, Qi Zhu, Peng Wang, and Qi Wu. Chop chop bert: Visual question answering by chopping visual- bert's heads. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI'21, pages 664-670, 8 2021. Main Track.","title":"Chop chop bert: Visual question answering by chopping visualbert's heads","authors":"Gao","year":"2021","venue":"Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI'21","doi":null,"url":null,"internal_url":"/proceedings/2021/92"},{"position":8,"raw":"[Gentile and Warmuth, 1998] Claudio Gentile and Manfred K. Warmuth. Linear hinge loss and average margin. In Proceed- ings of the 11th International Conference on Neural Information Processing Systems, page 225-231, 1998.","title":"Linear hinge loss and average margin","authors":"Claudio Gentile, Manfred K Warmuth","year":"1998","venue":"Proceedings of the 11th International Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Girdhar et al., 2019] Rohit Girdhar, Joao Carreira, Carl Doersch, and Andrew Zisserman. Video action transformer network. In CVPR'19, pages 244-253, 2019.","title":"Video action transformer network","authors":"Rohit Girdhar, Joao Carreira, Carl Doersch, Andrew Zisserman","year":"2019","venue":"CVPR'19","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hara et al., 2018] Kensho Hara, Hirokatsu Kataoka, and Yutaka Satoh. Can spatiotemporal 3d cnns retrace the history of 2d cnns and imagenet? In CVPR'18, pages 6546-6555, 2018.","title":"Can spatiotemporal 3d cnns retrace the history of 2d cnns and imagenet?","authors":"Hara","year":"2018","venue":"CVPR'18","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR'16, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR'16","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Huang et al., 2020] Deng Huang, Peihao Chen, Runhao Zeng, Qing Du, Mingkui Tan, and Chuang Gan. Location-aware graph convolutional networks for video question answering. In Pro- ceedings of the AAAI Conference on Artificial Intelligence, vol- ume 34, pages 11021-11028, 2020.","title":"Location-aware graph convolutional networks for video question answering","authors":"Deng Huang, Peihao Chen, Runhao Zeng, Qing Du, Mingkui Tan, Chuang Gan","year":"2020","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Jang et al., 2017] Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. Tgif-qa: Toward spatio- temporal reasoning in visual question answering. In CVPR'17, pages 2758-2766, 2017.","title":"Tgif-qa: Toward spatiotemporal reasoning in visual question answering","authors":"Jang","year":"2017","venue":"CVPR'17","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Jiang and Han, 2020] Pin Jiang and Yahong Han. Reasoning with heterogeneous graph alignment for video question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 11109-11116, 2020.","title":"Reasoning with heterogeneous graph alignment for video question answering","authors":"Han ; Jiang, Pin Jiang, Yahong Han","year":"2020","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kim et al., 2019] Junyeong Kim, Minuk Ma, Kyungsu Kim, Sungjin Kim, and Chang D Yoo. Progressive attention memory network for movie story question answering. In CVPR'19, pages 8337-8346, 2019.","title":"Progressive attention memory network for movie story question answering","authors":"Kim","year":"2019","venue":"CVPR'19","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kingma and Ba, 2015] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR (Poster), 2015.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2015","venue":"ICLR (Poster)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Ge- offrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing sys- tems, 25:1097-1105, 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton","year":"2012","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Le et al., 2020] Thao Minh Le, Vuong Le, Svetha Venkatesh, and Truyen Tran. Hierarchical conditional relation networks for video question answering. In CVPR'20, pages 9972-9981, 2020.","title":"Hierarchical conditional relation networks for video question answering","authors":"Thao Minh, Le, Vuong Le, Svetha Venkatesh, Truyen Tran","year":"2020","venue":"CVPR'20","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Li et al., 2019] Xiangpeng Li, Jingkuan Song, Lianli Gao, Xiang- long Liu, Wenbing Huang, Xiangnan He, and Chuang Gan. Be- yond rnns: Positional self-attention with co-attention for video question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 8658-8665, 2019.","title":"Beyond rnns: Positional self-attention with co-attention for video question answering","authors":"Li","year":"2019","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Liang et al., 2020] Justin Liang, Namdar Homayounfar, Wei-Chiu Ma, Yuwen Xiong, Rui Hu, and Raquel Urtasun. Polytrans- form: Deep polygon transformer for instance segmentation. In CVPR'20, pages 9131-9140, 2020.","title":"Polytransform: Deep polygon transformer for instance segmentation","authors":"Justin Liang, Namdar Homayounfar, Wei-Chiu Ma, Yuwen Xiong, Rui Hu, Raquel Urtasun","year":"2020","venue":"CVPR'20","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Liu et al., 2021] Fei Liu, Jing Liu, Weining Wang, and Hanqing Lu. Hair: Hierarchical visual-semantic relational reasoning for video question answering. In ICCV'21, pages 1698-1707, 2021.","title":"Hair: Hierarchical visual-semantic relational reasoning for video question answering","authors":"Fei Liu, Jing Liu, Weining Wang, Hanqing Lu","year":"2021","venue":"ICCV'21","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Park et al., 2021] Jungin Park, Jiyoung Lee, and Kwanghoon Sohn. Bridge to answer: Structure-aware graph interaction net- work for video question answering. In CVPR'21, pages 15526- 15535, June 2021.","title":"Bridge to answer: Structure-aware graph interaction network for video question answering","authors":"Park","year":"2021","venue":"CVPR'21","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Pennington et al., 2014] Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word repre- sentation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532- 1543, 2014.","title":"Glove: Global vectors for word representation","authors":"Jeffrey Pennington, Richard Socher, Christopher D Manning","year":"2014","venue":"Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Par- mar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, pages 6000-6010, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, Illia Polosukhin","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wolf et al., 2020] Thomas Wolf, Julien Chaumond, Lysandre De- but, Victor Sanh, Clement Delangue, Anthony Moi, Pierric Cis- tac, Morgan Funtowicz, Joe Davison, Sam Shleifer, et al. Trans- formers: State-of-the-art natural language processing. In Pro- ceedings of the 2020 Conference on Empirical Methods in Natu- ral Language Processing: System Demonstrations, pages 38-45, 2020.","title":"Transformers: State-of-the-art natural language processing","authors":"Thomas Wolf, Julien Chaumond, Lysandre Debut, Victor Sanh, Clement Delangue, Anthony Moi, Pierric Cistac, Morgan Funtowicz, Joe Davison, Sam Shleifer","year":"2020","venue":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Xiong et al., 2020] Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. On layer normalization in the trans- former architecture. In International Conference on Machine Learning, pages 10524-10533. PMLR, 2020.","title":"On layer normalization in the transformer architecture","authors":"Xiong","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Xu et al., 2017] Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Han- wang Zhang, Xiangnan He, and Yueting Zhuang. Video ques- tion answering via gradually refined attention over appearance and motion. In Proceedings of the 25th ACM International Con- ference on Multimedia, page 1645-1653, 2017.","title":"Video question answering via gradually refined attention over appearance and motion","authors":"Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, Yueting Zhuang","year":"2017","venue":"Proceedings of the 25th ACM International Conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Yang et al., 2016] Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alex Smola. Stacked attention networks for im- age question answering. In CVPR'16, pages 21-29, 2016.","title":"Stacked attention networks for image question answering","authors":"Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Smola","year":"2016","venue":"CVPR'16","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zha et al., 2019] Zheng-Jun Zha, Jiawei Liu, Tianhao Yang, and Yongdong Zhang. Spatiotemporal-textual co-attention network for video question answering. ACM Trans. Multimedia Comput. Commun. Appl., 15(2s), July 2019.","title":"Spatiotemporal-textual co-attention network for video question answering","authors":"Zheng-Jun Zha, Jiawei Liu, Tianhao Yang, Yongdong Zhang","year":"2019","venue":"ACM Trans. Multimedia Comput. Commun. Appl","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}