{"type":"ScholarlyArticle","title":"Video Question Answering on Screencast Tutorials","authors":[{"name":"Wentian Zhao","slug":"wentian-zhao","given":"Wentian","family":"Zhao","affiliation":"Adobe Research"},{"name":"Seokhwan Kim","slug":"seokhwan-kim","given":"Seokhwan","family":"Kim","affiliation":"Amazon Alexa AI"},{"name":"Ning Xu","slug":"ning-xu","given":"Ning","family":"Xu","affiliation":"Adobe Research"},{"name":"Hailin Jin","slug":"hailin-jin","given":"Hailin","family":"Jin","affiliation":"Adobe Research"}],"abstract":"This paper presents a new video question answering task on screencast tutorials. We introduce a dataset including question, answer and context triples from the tutorial videos for a software. Unlike other video question answering works, all the answers in our dataset are grounded to the domain knowledge base. An one-shot recognition algorithm is designed to extract the visual cues, which helps enhance the performance of video question answering. We also propose several baseline neural network architectures based on various aspects of video contexts from the dataset. The experimental results demonstrate that our proposed models significantly improve the question answering performances by incorporating multi-modal contexts and domain knowledge.","doi":"10.24963/ijcai.2020/148","url":"https://ijcai.org/proceedings/2020/148","pdf":"https://ijcai.org/proceedings/2020/0148.pdf","markdown":"https://ijcai.org/proceedings/2020/148.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/148","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Computer Vision","track":"Main track","number":148,"pages":{"first":1061,"last":1068},"keywords":["Natural Language Processing: Question Answering","Computer Vision: Language and Vision","Machine Learning: Deep Learning"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25303/file_25303.m3u8","length_sec":299,"provider":"hls"}],"references":[{"position":1,"raw":"[Huang et al., 2017] De-An Huang, Joseph J Lim, Li Fei-Fei, and Juan Carlos Niebles. Unsupervised visual-linguistic reference resolution in instructional videos. In Proceedings of the IEEE CVPR, pages 2183-2192, 2017.","title":"Unsupervised visual-linguistic reference resolution in instructional videos","authors":"De-An, Joseph J Huang, Li Lim, Juan Carlos Fei-Fei, Niebles","year":"2017","venue":"Proceedings of the IEEE CVPR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Huang et al., 2018] De-An Huang, Shyamal Buch, Lucio Dery, Animesh Garg, Li Fei-Fei, and Juan Carlos Niebles. Finding \"it\": Weakly-supervised reference-aware visual grounding in instructional videos. In The IEEE CVPR, pages 5948-5957, 2018.","title":"Finding \"it\": Weakly-supervised reference-aware visual grounding in instructional videos","authors":"De-An, Shyamal Huang, Lucio Buch, Animesh Dery, Li Garg, Juan Carlos Fei-Fei, Niebles","year":"2018","venue":"The IEEE CVPR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Jang et al., 2017] Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. Tgif-qa: Toward spatio- temporal reasoning in visual question answering. In Pro- ceedings of the IEEE CVPR, pages 2758-2766, 2017.","title":"Tgif-qa: Toward spatiotemporal reasoning in visual question answering","authors":"Jang","year":"2017","venue":"Proceedings of the IEEE CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Jin et al., 2019] Weike Jin, Zhou Zhao, Mao Gu, Jun Yu, Jun Xiao, and Yueting Zhuang. Multi-interaction network with object relation for video question answering. In Lau- rent Amsaleg, Benoit Huet, Martha A. Larson, Guillaume Gravier, Hayley Hung, Chong-Wah Ngo, and Wei Tsang Ooi, editors, Proceedings of the 27th ACM International Conference on Multimedia, MM 2019, Nice, France, Octo- ber 21-25, 2019, pages 1193-1201. ACM, 2019.","title":"Multi-interaction network with object relation for video question answering","authors":"Weike Jin, Zhou Zhao, Mao Gu, Jun Yu, Jun Xiao, Yueting Zhuang","year":"2019","venue":"Proceedings of the 27th ACM International Conference on Multimedia, MM 2019","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Kang et al., 2019] Gi-Cheon Kang, Jaeseo Lim, and Byoung-Tak Zhang. Dual attention networks for visual reference resolution in visual dialog. arXiv preprint arXiv:1902.09368, 2019.","title":"Dual attention networks for visual reference resolution in visual dialog","authors":"Gi-Cheon Kang, Jaeseo Lim, Byoung-Tak Zhang","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Kim et al., 2017] Kyung-Min Kim, Min-Oh Heo, Seong-Ho Choi, and Byoung-Tak Zhang. Deepstory: video story qa by deep embedded memory networks. In Proceedings of the IJCAI, pages 2016-2022, 2017.","title":"Deepstory: video story qa by deep embedded memory networks","authors":"Kyung-Min Kim, Min-Oh Heo, Seong-Ho Choi, Byoung-Tak Zhang","year":"2017","venue":"Proceedings of the IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2017/280"},{"position":7,"raw":"[Kim, 2014] Yoon Kim. Convolutional neural networks for sentence classification. In Proceedings of the EMNLP, pages 1746-1751, 2014.","title":"Convolutional neural networks for sentence classification","authors":"Yoon Kim","year":"2014","venue":"Proceedings of the EMNLP","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kingma and Ba, 2014] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Mikolov et al., 2013] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed represen- tations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111-3119, 2013.","title":"Distributed representations of words and phrases and their compositionality","authors":"Mikolov","year":"2013","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Perozzi et al., 2014] Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. Deepwalk: Online learning of social rep- resentations. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 701-710, 2014.","title":"Deepwalk: Online learning of social representations","authors":"Bryan Perozzi, Rami Al-Rfou, Steven Skiena","year":"2014","venue":"Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Povey et al., 2011] Daniel Povey, Arnab Ghoshal, and Gilles Boulianne. The kaldi speech recognition toolkit. In IEEE 2011 Workshop on Automatic Speech Recognition and Un- derstanding (ASRU), 2011.","title":"The kaldi speech recognition toolkit","authors":"Daniel Povey, Arnab Ghoshal, Gilles Boulianne","year":"2011","venue":"IEEE 2011 Workshop on Automatic Speech Recognition and Understanding (ASRU)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Rajpurkar et al., 2016] Pranav Rajpurkar, Jian Zhang, Kon- stantin Lopyrev, and Percy Liang. Squad: 100,000+ ques- tions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392, 2016.","title":"Squad: 100,000+ questions for machine comprehension of text","authors":"Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, Percy Liang","year":"2016","venue":"Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Redmon et al., 2016] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Uni- fied, real-time object detection. In Proceedings of the IEEE CVPR, pages 779-788, 2016.","title":"You only look once: Unified, real-time object detection","authors":"Joseph Redmon, Santosh Divvala, Ross Girshick, Ali Farhadi","year":"2016","venue":"Proceedings of the IEEE CVPR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Singh et al., 2019] Gursimran Singh, Leonid Sigal, and James J. Little. Spatio-temporal relational reasoning for video question answering. In 30th British Machine Vision Conference 2019, BMVC 2019, Cardiff, UK, September 9-12, 2019, page 295. BMVA Press, 2019.","title":"Spatio-temporal relational reasoning for video question answering","authors":"Singh","year":"2019","venue":"30th British Machine Vision Conference 2019, BMVC 2019","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Song et al., 2018] Xiaomeng Song, Yucheng Shi, Xin Chen, and Yahong Han. Explore multi-step reasoning in video question answering. In Susanne Boll, Kyoung Mu Lee, Jiebo Luo, Wenwu Zhu, Hyeran Byun, Chang Wen Chen, Rainer Lienhart, and Tao Mei, editors, 2018 ACM Multi- media Conference on Multimedia Conference, MM 2018, Seoul, Republic of Korea, October 22-26, 2018, pages 239- 247. ACM, 2018.","title":"Explore multi-step reasoning in video question answering","authors":"Xiaomeng Song, Yucheng Shi, Xin Chen, Yahong Han","year":"2018","venue":"ACM Multimedia Conference on Multimedia Conference, MM 2018","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Tapaswi et al., 2016] Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. Movieqa: Understanding stories in movies through question-answering. In Proceedings of the IEEE CVPR, pages 4631-4640, 2016.","title":"Movieqa: Understanding stories in movies through question-answering","authors":"Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, Sanja Fidler","year":"2016","venue":"Proceedings of the IEEE CVPR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Wang et al., 2007] Mengqiu Wang, Noah A Smith, and Teruko Mitamura. What is the jeopardy model? a quasi- synchronous grammar for qa. In Proceedings of the EMNLP-CoNLL, pages 22-32, 2007.","title":"What is the jeopardy model? a quasisynchronous grammar for qa","authors":"Wang","year":"2007","venue":"Proceedings of the EMNLP-CoNLL","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Xue et al., 2017] Hongyang Xue, Zhou Zhao, and Deng Cai. Unifying the video and question attentions for open-ended video question answering. IEEE Trans. Image Processing, 26(12):5656-5666, 2017.","title":"Unifying the video and question attentions for open-ended video question answering","authors":"Hongyang Xue, Zhou Zhao, Deng Cai","year":"2017","venue":"IEEE Trans. Image Processing","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Yang et al., 2015] Yi Yang, Wen-tau Yih, and Christopher Meek. Wikiqa: A challenge dataset for open-domain ques- tion answering. In Proceedings of the EMNLP, pages 2013- 2018, 2015.","title":"Wikiqa: A challenge dataset for open-domain question answering","authors":"Yi Yang, Wen-Tau Yih, Christopher Meek","year":"2015","venue":"Proceedings of the EMNLP","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Ye et al., 2017] Yunan Ye, Zhou Zhao, Yimeng Li, Long Chen, Jun Xiao, and Yueting Zhuang. Video question answering via attribute-augmented attention network learn- ing. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 829-832, 2017.","title":"Video question answering via attribute-augmented attention network learning","authors":"Yunan Ye, Zhou Zhao, Yimeng Li, Long Chen, Jun Xiao, Yueting Zhuang","year":"2017","venue":"Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Yu et al., 2014] Shoou-I Yu, Lu Jiang, and Alexander Haupt- mann. Instructional videos for unsupervised harvesting and learning of action examples. In Proceedings of the 22nd ACM international conference on Multimedia, pages 825-828, 2014.","title":"Instructional videos for unsupervised harvesting and learning of action examples","authors":"Yu","year":"2014","venue":"Proceedings of the 22nd ACM international conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zhao et al., 2017] Zhou Zhao, Qifan Yang, Deng Cai, Xi- aofei He, and Yueting Zhuang. Video question answering via hierarchical spatio-temporal attention networks. In Proceedings of the Twenty-Sixth International Joint Con- ference on Artificial Intelligence, IJCAI 2017, Melbourne, Australia, August 19-25, 2017, pages 3518-3524, 2017.","title":"Video question answering via hierarchical spatio-temporal attention networks","authors":"Zhou Zhao, Qifan Yang, Deng Cai, Xiaofei He, Yueting Zhuang","year":"2017","venue":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI 2017","doi":null,"url":null,"internal_url":"/proceedings/2017/492"},{"position":23,"raw":"[Zhou et al., 2018] Luowei Zhou, Chenliang Xu, and Jason J. Corso. Towards automatic learning of procedures from web instructional videos. In Proceedings of AAAI, New Orleans, Louisiana, USA, February 2-7, 2018, pages 7590-7598, 2018.","title":"Towards automatic learning of procedures from web instructional videos","authors":"Zhou","year":"2018","venue":"Proceedings of AAAI","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}