{"type":"ScholarlyArticle","title":"MAM-RNN: Multi-level Attention Model Based RNN for Video Captioning","authors":[{"name":"Xuelong Li","slug":"xuelong-li","given":"Xuelong","family":"Li","affiliation":"Xian Institute of Optics and Precision Mechanics , Chinese Academy of Sciences , Xian 710119 , P. R. China"},{"name":"Bin Zhao","slug":"bin-zhao","given":"Bin","family":"Zhao","affiliation":"School of Computer Science and Center for OPTical IMagery Analysis and Learning (OPTIMAL) , Northwestern Polytechnical University , Xian 710072 , P. R. China"},{"name":"Xiaoqiang Lu","slug":"xiaoqiang-lu","given":"Xiaoqiang","family":"Lu","affiliation":"Xian Institute of Optics and Precision Mechanics , Chinese Academy of Sciences , Xian 710119 , P. R. China"}],"abstract":"Visual information is quite important for the task of video captioning. However, in the video, there are a lot of uncorrelated content, which may cause interference to generate a correct caption. Based on this point, we attempt to exploit the visual features which are most correlated to the caption. In this paper, a Multi-level Attention Model based Recurrent Neural Network (MAM-RNN) is proposed, where MAM is utilized to encode the visual feature and RNN works as the decoder to generate the video caption. During generation, the proposed approach is able to adaptively attend to the salient regions in the frame and the frames correlated to the caption. Practically, the experimental results on two benchmark datasets, i.e., MSVD and Charades, have shown the excellent performance of the proposed approach.","doi":"10.24963/ijcai.2017/307","url":"https://ijcai.org/proceedings/2017/307","pdf":"https://ijcai.org/proceedings/2017/0307.pdf","markdown":"https://ijcai.org/proceedings/2017/307.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/307","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning A-R","track":"Main track","number":307,"pages":{"first":2208,"last":2214},"keywords":["Machine Learning: Data Mining","Machine Learning: Feature Selection/Construction","Machine Learning: Machine Learning","Machine Learning: Classification"],"videos":[],"references":[{"position":1,"raw":"[Bengio et al., 1994] Yoshua Bengio, Patrice Y. Simard, and Paolo Frasconi. Learning long-term dependencies with gradient descent is difficult. IEEE Trans. Neural Networks, 5(2):157-166, 1994.","title":"Learning long-term dependencies with gradient descent is difficult","authors":"Yoshua Bengio, Patrice Y Simard, Paolo Frasconi","year":"1994","venue":"IEEE Trans. Neural Networks","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Denkowski and Lavie, 2014] Michael J. Denkowski and Alon Lavie. Meteor universal: Language specific trans- lation evaluation for any target language. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 376-380, 2014.","title":"Meteor universal: Language specific trans-lation evaluation for any target language","authors":"J Michael, Alon Denkowski, Lavie","year":"2014","venue":"Proceedings of the Ninth Workshop on Statistical Machine Translation","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Fakoor et al., 2016] Rasool Fakoor, Abdel-rahman Mo- hamed, Margaret Mitchell, Sing Bing Kang, and Push- meet Kohli. Memory-augmented attention modelling for videos. CoRR, abs/1611.02261, 2016.","title":"Memory-augmented attention modelling for videos","authors":"Rasool Fakoor, Abdel-Rahman Mohamed, Margaret Mitchell, Sing, Bing Kang, Pushmeet Kohli","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Guadarrama et al., 2013] Sergio Guadarrama, Niveda Kr- ishnamoorthy, Girish Malkarnenkar, Subhashini Venu- gopalan, Raymond J. Mooney, Trevor Darrell, and Kate Saenko. Youtube2text: Recognizing and describing ar- bitrary activities using semantic hierarchies and zero-shot recognition. In IEEE International Conference on Com- puter Vision, pages 2712-2719, 2013.","title":"Youtube2text: Recognizing and describing arbitrary activities using semantic hierarchies and zero-shot recognition","authors":"Sergio Guadarrama, Niveda Krishnamoorthy, Girish Malkarnenkar, Subhashini Venugopalan, Raymond J Mooney, Trevor Darrell, Kate Saenko","year":"2013","venue":"IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735-1780, 1997.","title":"Long short-term memory","authors":"Schmidhuber Hochreiter, Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Krishnamoorthy et al., 2013] Niveda Krishnamoorthy, Girish Malkarnenkar, Raymond J. Mooney, Kate Saenko, and Sergio Guadarrama. Generating natural-language video descriptions using text-mined knowledge. In Proceedings of the Twenty-Seventh AAAI Conference on Artificial Intelligence, 2013.","title":"Generating natural-language video descriptions using text-mined knowledge","authors":"Krishnamoorthy","year":"2013","venue":"Proceedings of the Twenty-Seventh AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in Neural In- formation Processing Systems, pages 1106-1114, 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton","year":"2012","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Lin and Och, 2004] Chin-Yew Lin and Franz Josef Och. Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics. In Proceedings of the 42nd Annual Meeting of the Associa- tion for Computational Linguistics, pages 605-612, 2004.","title":"Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics","authors":"Lin, Och ; ] Chin-Yew Lin, Franz Josef, Och","year":"2004","venue":"Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Pan et al., 2016] Yingwei Pan, Tao Mei, Ting Yao, Houqiang Li, and Yong Rui. Jointly modeling embed- ding and translation to bridge video and language. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, pages 4594-4602, 2016.","title":"Jointly modeling embedding and translation to bridge video and language","authors":"Pan","year":"2016","venue":"2016 IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Papineni et al., 2002] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for au- tomatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics, pages 311-318, 2002.","title":"Bleu: a method for automatic evaluation of machine translation","authors":"Kishore Papineni, Salim Roukos, Todd Ward, Wei-Jing Zhu","year":"2002","venue":"Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Pennington et al., 2014] Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vec- tors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pages 1532-1543, 2014.","title":"Glove: Global vectors for word representation","authors":"Jeffrey Pennington, Richard Socher, Christopher D Manning","year":"2014","venue":"Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Shetty and Laaksonen, 2015] Rakshith Shetty and Jorma Laaksonen. Video captioning with recurrent networks based on frame-and video-level features and visual con- tent classification. CoRR, abs/1512.02949, 2015.","title":"Rakshith Shetty and Jorma Laaksonen. Video captioning with recurrent networks based on frame-and video-level features and visual content classification","authors":"Laaksonen Shetty","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Sigurdsson et al., 2016] Gunnar A. Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta. Hollywood in homes: Crowdsourcing data collec- tion for activity understanding. In European Conference on Computer Vision, pages 510-526, 2016.","title":"Hollywood in homes: Crowdsourcing data collection for activity understanding","authors":"A Gunnar, Gül Sigurdsson, Xiaolong Varol, Ali Wang, Ivan Farhadi, Abhinav Laptev, Gupta","year":"2016","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR, abs/1409.1556, 2014.","title":"Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition","authors":"Zisserman Simonyan","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Szegedy et al., 2015] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott E. Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabi- novich. Going deeper with convolutions. In IEEE Confer- ence on Computer Vision and Pattern Recognition, pages 1-9, 2015.","title":"Going deeper with convolutions","authors":"Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott E Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, Andrew Rabinovich","year":"2015","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Thomason et al., 2014] Jesse Thomason, Subhashini Venu- gopalan, Sergio Guadarrama, Kate Saenko, and Ray- mond J. Mooney. Integrating language and vision to gen- erate natural language descriptions of videos in the wild. In International Conference on Computational Linguistic- s, pages 1218-1227, 2014.","title":"Integrating language and vision to generate natural language descriptions of videos in the wild","authors":"Jesse Thomason, Subhashini Venugopalan, Sergio Guadarrama, Kate Saenko, Raymond J Mooney","year":"2014","venue":"International Conference on Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Vedantam et al., 2015] Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. Cider: Consensus- based image description evaluation. In IEEE Conference on Computer Vision and Pattern Recognition, pages 4566-4575, 2015.","title":"Cider: Consensusbased image description evaluation","authors":"C Lawrence Ramakrishna Vedantam, Devi Zitnick, Parikh","year":"2015","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Venugopalan et al., 2015a] Subhashini Venugopalan, Mar- cus Rohrbach, Jeffrey Donahue, Raymond J. Mooney, Trevor Darrell, and Kate Saenko. Sequence to sequence -video to text. In IEEE International Conference on Com- puter Vision, pages 4534-4542, 2015.","title":"Sequence to sequence -video to text","authors":"Venugopalan","year":"2015","venue":"IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Venugopalan et al., 2015b] Subhashini Venugopalan, Hui- juan Xu, Jeff Donahue, Marcus Rohrbach, Raymond J. Mooney, and Kate Saenko. Translating videos to natu- ral language using deep recurrent neural networks. In The 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, pages 1494-1504, 2015.","title":"Translating videos to natural language using deep recurrent neural networks","authors":"Venugopalan","year":"2015","venue":"The 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Yao et al., 2015] Li Yao, Atousa Torabi, Kyunghyun Cho, Nicolas Ballas, Christopher J. Pal, Hugo Larochelle, and Aaron C. Courville. Describing videos by exploiting tem- poral structure. In IEEE International Conference on Computer Vision, pages 4507-4515, 2015.","title":"Describing videos by exploiting temporal structure","authors":"Li Yao, Atousa Torabi, Kyunghyun Cho, Nicolas Ballas, Christopher J Pal, Hugo Larochelle, Aaron C Courville","year":"2015","venue":"IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Yu et al., 2016] Haonan Yu, Jiang Wang, Zhiheng Huang, Yi Yang, and Wei Xu. Video paragraph captioning us- ing hierarchical recurrent neural networks. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, pages 4584-4593, 2016.","title":"Video paragraph captioning using hierarchical recurrent neural networks","authors":"Haonan Yu, Jiang Wang, Zhiheng Huang, Yi Yang, Wei Xu","year":"2016","venue":"2016 IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zeng et al., 2016] Kuo-Hao Zeng, Tseng-Hung Chen, Juan Carlos Niebles, and Min Sun. Title generation for user generated videos. In European Conference on Computer Vision, pages 609-625, 2016.","title":"Title generation for user generated videos","authors":"Kuo-Hao Zeng, Tseng-Hung Chen, Juan Carlos Niebles, Min Sun","year":"2016","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}