{"type":"ScholarlyArticle","title":"DeepStory: Video Story QA by Deep Embedded Memory Networks","authors":[{"name":"Kyung-Min Kim","slug":"kyung-min-kim","given":"Kyung-Min","family":"Kim","affiliation":""},{"name":"Min-Oh Heo","slug":"min-oh-heo","given":"Min-Oh","family":"Heo","affiliation":""},{"name":"Seong-Ho Choi","slug":"seong-ho-choi","given":"Seong-Ho","family":"Choi","affiliation":""},{"name":"Byoung-Tak Zhang","slug":"byoung-tak-zhang","given":"Byoung-Tak","family":"Zhang","affiliation":""}],"abstract":"Question-answering (QA) on video contents is a significant challenge for achieving human-level intelligence as it involves both vision and language in real-world settings. Here we demonstrate the possibility of an AI agent performing video story QA by learning from a large amount of cartoon videos. We develop a video-story learning model, i.e. Deep Embedded Memory Networks (DEMN), to reconstruct stories from a joint scene-dialogue video stream using a latent embedding space of observed data. The video stories are stored in a long-term memory component. For a given question, an LSTM-based attention model uses the long-term memory to recall the best question-story-answer triplet by focusing on specific words containing key information. We trained the DEMN on a novel QA dataset of children’s cartoon video series, Pororo. The dataset contains 16,066 scene-dialogue pairs of 20.5-hour videos, 27,328 fine-grained sentences for scene description, and 8,913 story-related QA pairs. Our experimental results show that the DEMN outperforms other QA models. This is mainly due to 1) the reconstruction of video stories in a scene-dialogue combined form that utilize the latent embedding and 2) attention. DEMN also achieved state-of-the-art results on the MovieQA benchmark.","doi":"10.24963/ijcai.2017/280","url":"https://ijcai.org/proceedings/2017/280","pdf":"https://ijcai.org/proceedings/2017/0280.pdf","markdown":"https://ijcai.org/proceedings/2017/280.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/280","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning A-R","track":"Main track","number":280,"pages":{"first":2016,"last":2022},"keywords":["Natural Language Processing: Question Answering","Machine Learning: Deep Learning","Machine Learning: Neural Networks"],"videos":[],"references":[{"position":1,"raw":"[Abney et al., 2000] Steven Abney, Michael Collins, and Amit Singhal. Answer Extraction. In Proceedings of the 6th Applied Natural Language Processing Conference, 296-301, 2000.","title":"Answer Extraction","authors":"Steven Abney, Michael Collins, Amit Singhal","year":"2000","venue":"Proceedings of the 6th Applied Natural Language Processing Conference","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Agrawal et al., 2015] Aishwarya Agrawal, Jiasen Lu, Stanislaw Antol, Margaret Mitchell, C. Lawrence Zitnick, Dhruv Batra, and Devi Parikh. VQA: Visual Question Answering. In Pro- ceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2015.","title":"VQA: Visual Question Answering","authors":"Aishwarya Agrawal, Jiasen Lu, Stanislaw Antol, Margaret Mitchell, C Lawrence Zitnick, Dhruv Batra, Devi Parikh","year":"2015","venue":"Proceedings of IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bengio et al., 2009] Yoshua Bengio, J´erˆome Louradour, Ronan Collobert, and Jason Weston, Curriculum Learning, In Pro- ceedings of International Conference on Machine Learning, 2009.","title":"Curriculum Learning","authors":"Yoshua Bengio, J´erˆome Louradour, Ronan Collobert, Jason Weston","year":"2009","venue":"Proceedings of International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Frome et al., 2013] Andrea Frome, Greg S. Corrado, Jonathon Shlens, Samy Bengio, Jeffrey Dean, Marc' Aurelio Ranzato, and Tomas Mikolov. DeViSE: A Deep Visual-Semantic Em- bedding Model. In Proceedings of Advances in Neural Infor- mation Processing Systems, 2121-2129, 2013.","title":"DeViSE: A Deep Visual-Semantic Embedding Model","authors":"Andrea Frome, Greg S Corrado, Jonathon Shlens, Samy Bengio, Jeffrey Dean, Marc' Aurelio Ranzato, Tomas Mikolov","year":"2013","venue":"Proceedings of Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Fukui et al., 2016] Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darrell, and Marcus Rohrbach. Mul- timodal Compact Bilinear Pooling for Visual Question An- swering and Visual Grounding. In Proceedings of Conference on Empirical Methods on Natural Language Processing, 2016.","title":"Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding","authors":"Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darrell, Marcus Rohrbach","year":"2016","venue":"Proceedings of Conference on Empirical Methods on Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Ha et al., 2015] Jung-Woo Ha, Kyung-Min Kim, and Byoung-Tak Zhang. Automated Construction of Visual-Linguistic Knowledge via Concept Learning from Cartoon Videos. In Proceedings of AAAI Conference on Artificial Intelligence, 2015.","title":"Automated Construction of Visual-Linguistic Knowledge via Concept Learning from Cartoon Videos","authors":"Jung-Woo Ha, Kyung-Min Kim, Byoung-Tak Zhang","year":"2015","venue":"Proceedings of AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hochreiter et al., 1997] Sepp Hochreiter, and Jürgen Schmidhuber. Long short-term memory, Neural Computation, 9(8): 1735-1780, 1997.","title":"Long short-term memory","authors":"Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hovy et al., 2001] Eduard Hovy, Laurie Gerber, Ulf Hermjakob, Chin-Yew Lin, and Deepak Ravichandran. Toward Seman- tics-based Answer Pinpointing. In Proceedings of Human Language Technology Conference, 339-345, 2001.","title":"Toward Semantics-based Answer Pinpointing","authors":"Eduard Hovy, Laurie Gerber, Ulf Hermjakob, Chin-Yew Lin, Deepak Ravichandran","year":"2001","venue":"Proceedings of Human Language Technology Conference","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kaiming et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2016.","title":"Deep Residual Learning for Image Recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"Proceedings of IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kim et al., 2015] Kyung-Min Kim, Chang-Jun Nan, Jung-Woo Ha, Yu-Jung Heo, and Byoung-Tak Zhang. Pororobot: A Deep Learning Robot that Plays Video Q&A Games. In Proceedings of AAAI Fall Symposium on AI for HRI, 2015.","title":"Pororobot: A Deep Learning Robot that Plays Video Q&A Games","authors":"Kyung-Min Kim, Chang-Jun Nan, Jung-Woo Ha, Yu-Jung Heo, Byoung-Tak Zhang","year":"2015","venue":"Proceedings of AAAI Fall Symposium on AI for HRI","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kim et al., 2016] Jin-Hwa Kim, Sang-Woo Lee, Dong-Hyun Kwak, Min-Oh Heo, Jeonghee Kim, Jung-Woo Ha, and Byoung-Tak Zhang. Multimodal residual learning for visual QA. In Pro- ceedings of NIPS, 2016.","title":"Multimodal residual learning for visual QA","authors":"Jin-Hwa Kim, Sang-Woo Lee, Dong-Hyun Kwak, Min-Oh Heo, Jeonghee Kim, Jung-Woo Ha, Byoung-Tak Zhang","year":"2016","venue":"Proceedings of NIPS","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kiros et al., 2015a] Ryan Kiros, Ruslan Salakhutdinov, and Rich- ard S. Zemel. Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models, In Proceedings of Transactions of the Association for Computational Linguistics, 2015.","title":"Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models","authors":"Kiros","year":"2015","venue":"Proceedings of Transactions of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kiros et al., 2015b] Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, and Richard S. Zemel, Antonio Torralba, Raquel Urtasun, Sanja Fidler. Skip-Thought Vectors. In Proceedings of Advances in Neural Information Processing Systems, 2015.","title":"Skip-Thought Vectors","authors":"Kiros","year":"2015","venue":"Proceedings of Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li et al., 2016] Yuncheng Li, Yale Song, Liangliang Cao, Joel Tetreault, Larry Goldberg, Alejandro Jaimes, and Jiebo Luo. TGIF: A New Dataset and Benchmark on Animated GIF De- scription. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2016.","title":"TGIF: A New Dataset and Benchmark on Animated GIF De-scription","authors":"Yuncheng Li, Yale Song, Liangliang Cao, Joel Tetreault, Larry Goldberg, Alejandro Jaimes, Jiebo Luo","year":"2016","venue":"Proceedings of IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Rohrbach et al., 2014] Anna Rohrbach, Marcus Rohrbach, Wei Qiu, Annemarie Friedrich, Sikandar Amin, Mykhaylo Andriluka, Manfred Pinkal, and Bernt Schiele. Coherent multi-sentence video description with variable level of detail. In Proceedings of the German Confeence on Pattern Recognition, 2014.","title":"Coherent multi-sentence video description with variable level of detail","authors":"Anna Rohrbach, Marcus Rohrbach, Wei Qiu, Annemarie Friedrich, Sikandar Amin, Mykhaylo Andriluka, Manfred Pinkal, Bernt Schiele","year":"2014","venue":"Proceedings of the German Confeence on Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Rohrbach et al., 2015] Anna Rohrbach, Marcus Rohrbach, Niket Tandon, and Bernt Schiele. A dataset for movie description. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 3202-3212, 2015.","title":"A dataset for movie description","authors":"Anna Rohrbach, Marcus Rohrbach, Niket Tandon, Bernt Schiele","year":"2015","venue":"Proceedings of IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Schuster et al., 1997] Mike Schuster, and Kuldip K. Paliwal. Bi- directional Recurrent Neural Networks, IEEE Transactions on Signal Processing, 45(11):2673-2681, 1997.","title":"Bidirectional Recurrent Neural Networks","authors":"Mike Schuster, Kuldip K Paliwal","year":"1997","venue":"IEEE Transactions on Signal Processing","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Sukhbaatar et al., 2015] Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, and Rob Fergus. End-To-End Memory Networks. In Proceedings of Advances in Neural Information Processing Systems, 2015.","title":"End-To-End Memory Networks","authors":"Sukhbaatar","year":"2015","venue":"Proceedings of Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Tan et al., 2016] Ming Tan, Cicero dos Santos, Bing Xiang, and Bowen Zhou. LSTM-based Deep Learning Models for Non-factoid Answer Selection. International Conference on Learning Representations Workshop, 2016.","title":"LSTM-based Deep Learning Models for Non-factoid Answer Selection","authors":"Ming Tan, Bing Cicero Dos Santos, Bowen Xiang, Zhou","year":"2016","venue":"International Conference on Learning Representations Workshop","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Tapaswi et al., 2016] Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. MovieQA: Understanding Stories in Movies through Question-Answering. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2016.","title":"MovieQA: Understanding Stories in Movies through Question-Answering","authors":"Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, Sanja Fidler","year":"2016","venue":"Proceedings of IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Torabi et al., 2015] Atousa Torabi, Christopher Pal, Hugo Laro- chelle, Aaron Courville. Using descriptive video services to create a large data source for video annotation research. arXiv preprint arXiv:1503.01070v1, 2015.","title":"Using descriptive video services to create a large data source for video annotation research","authors":"Torabi","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang et al., 2015] Di Wang, and Eric Nyberg. A Long Short-Term Memory Model for Answer Sentence Selection in Question Answering, In Proceedings of Association for Computational Linguistics, 2015.","title":"A Long Short-Term Memory Model for Answer Sentence Selection in Question Answering","authors":"Di Wang, Eric Nyberg","year":"2015","venue":"Proceedings of Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Weston et al., 2010] Jason Weston, Samy Bengio, and Nicolas Usunier. Large Scale Image Annotation: Learning to Rank with Joint Word-image Embeddings. Machine Learning. 81(1):21-35, 2010.","title":"Large Scale Image Annotation: Learning to Rank with Joint Word-image Embeddings","authors":"Jason Weston, Samy Bengio, Nicolas Usunier","year":"2010","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Weston et al., 2015] Jason Weston, Sumit Chopra, and Antoine Bordes. Memory Networks. In Proceedings of International Conference of Learning representations, 2015.","title":"Memory Networks","authors":"Jason Weston, Sumit Chopra, Antoine Bordes","year":"2015","venue":"Proceedings of International Conference of Learning representations","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Xu et al., 2016] Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2016.","title":"Msr-vtt: A large video description dataset for bridging video and language","authors":"Jun Xu, Tao Mei, Ting Yao, Yong Rui","year":"2016","venue":"Proceedings of IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yu et al., 2014] Lei Yu, Karl Moritz Hermann, Phil Blunsom, and Stephen Pulman. Deep Learning for Answer Sentence Selection, arXiv preprint arXiv:1412.1632, 2014.","title":"Deep Learning for Answer Sentence Selection","authors":"Lei Yu, Karl Moritz Hermann, Phil Blunsom, Stephen Pulman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zitnick et al., 2013] C. Lawrence Zitnick, and Devi Parikh. Bringing Semantics Into Focus Using Visual Abstraction. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2013.","title":"Bringing Semantics Into Focus Using Visual Abstraction","authors":"C Lawrence Zitnick, Devi Parikh","year":"2013","venue":"Proceedings of IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}