{"type":"ScholarlyArticle","title":"Feature Augmented Memory with Global Attention Network for VideoQA","authors":[{"name":"Jiayin Cai","slug":"jiayin-cai","given":"Jiayin","family":"Cai","affiliation":"Department of Computer Science and Technology , Tsinghua University"},{"name":"Chun Yuan","slug":"chun-yuan","given":"Chun","family":"Yuan","affiliation":"Tsinghua Shenzhen International Graduate School 3 ARC , Tencent PCG"},{"name":"Cheng Shi","slug":"cheng-shi","given":"Cheng","family":"Shi","affiliation":"Department of Computer Science and Technology , Tsinghua University"},{"name":"Lei Li","slug":"lei-li","given":"Lei","family":"Li","affiliation":"Department of Computer Science and Technology , Tsinghua University"},{"name":"Yangyang Cheng","slug":"yangyang-cheng","given":"Yangyang","family":"Cheng","affiliation":"Department of Computer Science and Technology , Tsinghua University"},{"name":"Ying Shan","slug":"ying-shan","given":"Ying","family":"Shan","affiliation":""}],"abstract":"Recently, Recurrent Neural Network (RNN) based methods and Self-Attention (SA) based methods have achieved promising performance in Video Question Answering (VideoQA). Despite the success of these works, RNN-based methods tend to forget the global semantic contents due to the inherent drawbacks of the recurrent units themselves, while SA-based methods cannot precisely capture the dependencies of the local neighborhood, leading to insufficient modeling for temporal order. To tackle these problems, we propose a novel VideoQA framework which progressively refines the representations of videos and questions from fine to coarse grain in a sequence-sensitive manner. Specifically, our model improves the feature representations via the following two steps: (1) introducing two fine-grained feature-augmented memories to strengthen the information augmentation of video and text which can improve memory capacity by memorizing more relevant and targeted information. (2) appending the self-attention and co-attention module to the memory output thus the module is able to capture global interaction between high-level semantic informations. Experimental results show that our approach achieves state-of-the-art performance on VideoQA benchmark datasets.","doi":"10.24963/ijcai.2020/139","url":"https://ijcai.org/proceedings/2020/139","pdf":"https://ijcai.org/proceedings/2020/0139.pdf","markdown":"https://ijcai.org/proceedings/2020/139.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/139","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Computer Vision","track":"Main track","number":139,"pages":{"first":998,"last":1004},"keywords":["Computer Vision: Language and Vision","Machine Learning: Deep Learning","Computer Vision: Video: Events, Activities and Surveillance"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25884/file_25884.m3u8","length_sec":295,"provider":"hls"}],"references":[{"position":1,"raw":"[Agrawal et al., 2015] Aishwarya Agrawal, Jiasen Lu, Stanislaw Antol, Margaret Mitchell, C. Lawrence Zitnick, Dhruv Batra, and Devi Parikh. Vqa: Visual question answering. In International Conference on Computer Vision, 2015.","title":"Vqa: Visual question answering","authors":"Agrawal","year":"2015","venue":"International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Ashish et al., 2017] Vaswani Ashish, Shazeer Noam, Par- mar Niki, and Uszkoreit. Attention is all you need. In Advances in neural information processing systems, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Shazeer Noam, Niki Parmar, Uszkoreit","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Fan et al., 2019] Chenyou Fan, Xiaofan Zhang, Shu Zhang, Wensheng Wang, Chi Zhang, and Heng Huang. Heteroge- neous memory enhanced multimodal attention model for video question answering. In Computer Vision and Pat- tern Recognition, 2019.","title":"Heterogeneous memory enhanced multimodal attention model for video question answering","authors":"Chenyou Fan, Xiaofan Zhang, Shu Zhang, Wensheng Wang, Chi Zhang, Heng Huang","year":"2019","venue":"Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Gao et al., 2018] Jiyang Gao, Runzhou Ge, Kan Chen, and Ram Nevatia. Motion-appearance co-memory networks for video question answering. In Computer Vision and Pat- tern Recognition, 2018.","title":"Motion-appearance co-memory networks for video question answering","authors":"Jiyang Gao, Runzhou Ge, Kan Chen, Ram Nevatia","year":"2018","venue":"Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Gao et al., 2019] Lianli Gao, Pengpeng Zeng, Jingkuan Song, YuanFang Li, Wu Liu, Tao Mei, and Heng Tao Shen. Structured two-stream attention network for video ques- tion answering. In Association for the Advancement of Ar- tificial Intelligence, 2019.","title":"Structured two-stream attention network for video question answering","authors":"Gao","year":"2019","venue":"Association for the Advancement of Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Computer Vision and Pattern Recognition, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Jang et al., 2017] Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. Tgif-qa: Toward spatio- temporal reasoning in visual question answering. In Com- puter Vision and Pattern Recognition, 2017.","title":"Tgif-qa: Toward spatiotemporal reasoning in visual question answering","authors":"Jang","year":"2017","venue":"Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[L and B, 2011] Chen David L and Dolan William B. Col- lecting highly parallel data for paraphrase evaluation. In Proceedings of the 49th Annual Meeting of the Associ- ation for Computational Linguistics: Human Language Technologies-Volume 1, pages 190-200, 2011.","title":"Collecting highly parallel data for paraphrase evaluation","authors":"B ; ] Chen David, L, Dolan William, B","year":"2011","venue":"Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Li et al., 2016] Yuncheng Li, Yale Song, Liangliang Cao, Joel Tetreault, Larry Goldberg, Alejandro Jaimes, and Jiebo Luo. Tgif: A new dataset and benchmark on an- imated gif description. In Computer Vision and Pattern Recognition, 2016.","title":"Tgif: A new dataset and benchmark on animated gif description","authors":"Yuncheng Li, Yale Song, Liangliang Cao, Joel Tetreault, Larry Goldberg, Alejandro Jaimes, Jiebo Luo","year":"2016","venue":"Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li et al., 2019] Xiangpeng Li, Jingkuan Song, Lianli Gao, Xianglong Liu, Wenbing Huang, Xiangnan He, and Chuang Gan. Beyond rnns: Positional self-attention with co-attention for video question answering. In Association for the Advancement of Artificial Intelligence, 2019.","title":"Beyond rnns: Positional self-attention with co-attention for video question answering","authors":"Li","year":"2019","venue":"Association for the Advancement of Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Simonyan and Zisserman, 2015] K. Simonyan and A. Zis- serman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learn- ing Representations, 2015.","title":"Very deep convolutional networks for large-scale image recognition","authors":"K Simonyan, A Zisserman","year":"2015","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[T et al., 2010] Mikolov T, Karafiat Martin, Burget L, Ver- nocky Jan, and Khudanpur Sanjeev. Recurrent neural net- work based language model. In Eleventh annual confer- ence of the international speech communication associa- tion, 2010.","title":"Recurrent neural network based language model","authors":"T","year":"2010","venue":"Eleventh annual conference of the international speech communication association","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Tran et al., 2015] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spa- tiotemporal features with 3d convolutional networks. In International Conference on Computer Vision, 2015.","title":"Learning spatiotemporal features with 3d convolutional networks","authors":"Tran","year":"2015","venue":"International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Wang et al., 2018] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Computer Vision and Pattern Recognition, 2018.","title":"Non-local neural networks","authors":"Xiaolong Wang, Ross Girshick, Abhinav Gupta, Kaiming He","year":"2018","venue":"Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Xu et al., 2015] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In International conference on machine learning, pages 2048-2057, 2015.","title":"Show, attend and tell: Neural image caption generation with visual attention","authors":"Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, Yoshua Bengio","year":"2015","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Xu et al., 2016] Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Computer Vision and Pattern Recognition, 2016.","title":"Msr-vtt: A large video description dataset for bridging video and language","authors":"Jun Xu, Tao Mei, Ting Yao, Yong Rui","year":"2016","venue":"Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Xu et al., 2017] Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, and Xiangnan He. Video question an- swering via gradually refined attention over appearance and motion. In Proceedings of the 25th ACM international conference on Multimedia, 2017.","title":"Video question answering via gradually refined attention over appearance and motion","authors":"Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He","year":"2017","venue":"Proceedings of the 25th ACM international conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Zhao et al., 2018] Zhou Zhao, Zhu Zhang, Shuwen Xiao, Zhou Yu, Jun Yu, Deng Cai, Fei Wu, and Yueting Zhuang. Open-ended long-form video question answering via adaptive hierarchical reinforced networks. In Proceed- ings of the 27th International Joint Conference on Artifi- cial Intelligence, 2018.","title":"Open-ended long-form video question answering via adaptive hierarchical reinforced networks","authors":"Zhou Zhao, Zhu Zhang, Shuwen Xiao, Zhou Yu, Jun Yu, Deng Cai, Fei Wu, Yueting Zhuang","year":"2018","venue":"Proceedings of the 27th International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2018/512"}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}