{"type":"ScholarlyArticle","title":"Hierarchical LSTM with Adjusted Temporal Attention for Video Captioning","authors":[{"name":"Jingkuan Song","slug":"jingkuan-song","given":"Jingkuan","family":"Song","affiliation":"Center for Future Media and School of Computer Science and Engineering , University of Electronic Science and Technology of China , Chengdu 611731 , China."},{"name":"Lianli Gao","slug":"lianli-gao","given":"Lianli","family":"Gao","affiliation":"Center for Future Media and School of Computer Science and Engineering , University of Electronic Science and Technology of China , Chengdu 611731 , China."},{"name":"Zhao Guo","slug":"zhao-guo","given":"Zhao","family":"Guo","affiliation":"Center for Future Media and School of Computer Science and Engineering , University of Electronic Science and Technology of China , Chengdu 611731 , China."},{"name":"Wu Liu","slug":"wu-liu","given":"Wu","family":"Liu","affiliation":"Beijing University of Posts and Telecommunications , Beijing 100876 , China."},{"name":"Dongxiang Zhang","slug":"dongxiang-zhang","given":"Dongxiang","family":"Zhang","affiliation":"Center for Future Media and School of Computer Science and Engineering , University of Electronic Science and Technology of China , Chengdu 611731 , China."},{"name":"Heng Tao Shen","slug":"heng-tao-shen","given":"Heng Tao","family":"Shen","affiliation":"Center for Future Media and School of Computer Science and Engineering , University of Electronic Science and Technology of China , Chengdu 611731 , China."}],"abstract":"Recent progress has been made in using attention based encoder-decoder framework for video captioning. However, most existing decoders apply the attention mechanism to every generated words including both visual words (e.g., “gun” and \"shooting“) and non-visual words (e.g. \"the“, \"a”).However, these non-visual words can be easily predicted using natural language model without considering visual signals or attention.Imposing attention mechanism on non-visual words could mislead and decrease the overall performance of video captioning.To address this issue, we propose a hierarchical LSTM with adjusted temporal attention (hLSTMat) approach for video captioning. Specifically, the proposed framework utilizes the temporal attention for selecting specific frames to predict related words, while the adjusted temporal attention is for deciding whether to depend on the visual information or the language context information. Also, a hierarchical LSTMs is designed to simultaneously consider both low-level visual information and deep semantic information to support the video caption generation. To demonstrate the effectiveness of our proposed framework, we test our method on two prevalent datasets: MSVD and MSR-VTT, and experimental results show that our approach outperforms the state-of-the-art methods on both two datasets.","doi":"10.24963/ijcai.2017/381","url":"https://ijcai.org/proceedings/2017/381","pdf":"https://ijcai.org/proceedings/2017/0381.pdf","markdown":"https://ijcai.org/proceedings/2017/381.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/381","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning S-Z","track":"Main track","number":381,"pages":{"first":2737,"last":2743},"keywords":["Natural Language Processing: Natural Language Generation","Machine Learning: Neural Networks"],"videos":[],"references":[{"position":1,"raw":"[Banerjee and Lavie, 2005] Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for ma- chine translation and/or summarization, pages 65-72, 2005.","title":"Meteor: An automatic metric for mt evaluation with improved correlation with human judgments","authors":"Lavie Banerjee, Satanjeev Banerjee, Alon Lavie","year":"2005","venue":"Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bengio et al., 1994] Yoshua Bengio, Patrice Simard, and Paolo Frasconi. Learning long-term dependencies with gradient descent is difficult. IEEE transactions on neural networks, 5(2):157-166, 1994.","title":"Learning long-term dependencies with gradient descent is difficult","authors":"Yoshua Bengio, Patrice Simard, Paolo Frasconi","year":"1994","venue":"IEEE transactions on neural networks","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen and Dolan, 2011] David L Chen and William B Dolan. Col- lecting highly parallel data for paraphrase evaluation. In ACL, pages 190-200, 2011.","title":"Collecting highly parallel data for paraphrase evaluation","authors":"L David, William B Chen, Dolan","year":"2011","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen and Zitnick, 2014] Xinlei Chen and C Lawrence Zitnick. Learning a recurrent visual representation for image caption gen- eration. arXiv preprint arXiv:1411.5654, 2014.","title":"Learning a recurrent visual representation for image caption generation","authors":"Zitnick Chen","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2016] Long Chen, Hanwang Zhang, Jun Xiao, Liqiang Nie, Jian Shao, and Tat-Seng Chua. Sca-cnn: Spatial and channel-wise attention in convolutional networks for image cap- tioning. arXiv preprint arXiv:1611.05594, 2016.","title":"Sca-cnn: Spatial and channel-wise attention in convolutional networks for image captioning","authors":"Long Chen, Hanwang Zhang, Jun Xiao, Liqiang Nie, Jian Shao, Tat-Seng Chua","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Cho et al., 2015] Kyunghyun Cho, Aaron Courville, and Yoshua Bengio. Describing multimedia content using attention-based encoder-decoder networks. Multimedia, IEEE Transactions on, 17(11):1875-1886, 2015.","title":"Describing multimedia content using attention-based encoder-decoder networks","authors":"Cho","year":"2015","venue":"Multimedia, IEEE Transactions on","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Fang et al., 2015] Hao Fang, Saurabh Gupta, Forrest Iandola, Ru- pesh K Srivastava, Li Deng, Piotr Dollár, Jianfeng Gao, Xi- aodong He, Margaret Mitchell, John C Platt, et al. From captions to visual concepts and back. In CVPR, pages 1473-1482, 2015.","title":"From captions to visual concepts and back","authors":"Saurabh Hao Fang, Forrest Gupta, Iandola, K Rupesh, Li Srivastava, Piotr Deng, Jianfeng Dollár, Xiaodong Gao, Margaret He, John C Mitchell, Platt","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gan et al., 2016a] Chuang Gan, Chen Sun, Lixin Duan, and Bo- qing Gong. Webly-supervised video recognition by mutually vot- ing for relevant web images and web video frames. In ECCV, 2016.","title":"Webly-supervised video recognition by mutually voting for relevant web images and web video frames","authors":"Gan","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gan et al., 2016b] Zhe Gan, Chuang Gan, Xiaodong He, Yunchen Pu, Kenneth Tran, Jianfeng Gao, Lawrence Carin, and Li Deng. Semantic compositional networks for visual captioning. CoRR, abs/1611.08002, 2016.","title":"Semantic compositional networks for visual captioning","authors":"Gan","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gao et al., 2017] Lianli Gao, Peng Wang, Jingkuan Song, Zi Huang, Jie Shao, and Heng Tao Shen. Event video mashup: From hundreds of videos to minutes of skeleton. In AAAI, pages 1323-1330, 2017.","title":"Event video mashup: From hundreds of videos to minutes of skeleton","authors":"Gao","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Gu et al., 2016] Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O. K. Li. Incorporating copying mechanism in sequence-to- sequence learning. In ACL, 2016.","title":"Incorporating copying mechanism in sequence-tosequence learning","authors":"Jiatao Gu, Zhengdong Lu, Hang Li, O K Victor, Li","year":"2016","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Karpathy et al., 2014] Andrej Karpathy, Armand Joulin, and Fei Fei F Li. Deep fragment embeddings for bidirectional image sen- tence mapping. In NIPS, pages 1889-1897, 2014.","title":"Deep fragment embeddings for bidirectional image sentence mapping","authors":"Andrej Karpathy, Armand Joulin, Fei Fei, F Li","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li et al., 2015] Guang Li, Shubo Ma, and Yahong Han. Summarization-based video caption via deep neural networks. In ACM Multimedia, pages 1191-1194, 2015.","title":"Summarization-based video caption via deep neural networks","authors":"Guang Li, Shubo Ma, Yahong Han","year":"2015","venue":"ACM Multimedia","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lu et al., 2016] Jiasen Lu, Caiming Xiong, Devi Parikh, and Richard Socher. Knowing when to look: Adaptive attention via a visual sentinel for image captioning. arXiv preprint arXiv:1612.01887, 2016.","title":"Knowing when to look: Adaptive attention via a visual sentinel for image captioning","authors":"Lu","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Luong et al., 2015] Minh-Thang Luong, Hieu Pham, and Christo- pher D Manning. Effective approaches to attention-based neural machine translation. arXiv preprint arXiv:1508.04025, 2015.","title":"Effective approaches to attention-based neural machine translation","authors":"Minh-Thang Luong, Hieu Pham, Christopher D Manning","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Pan et al., 2015] Yingwei Pan, Tao Mei, Ting Yao, Houqiang Li, and Yong Rui. Jointly modeling embedding and translation to bridge video and language. arXiv preprint arXiv:1505.01861, 2015.","title":"Jointly modeling embedding and translation to bridge video and language","authors":"Pan","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Pan et al., 2016] Pingbo Pan, Zhongwen Xu, Yi Yang, Fei Wu, and Yueting Zhuang. Hierarchical recurrent neural encoder for video representation with application to captioning. In CVPR, pages 1029-1038, 2016.","title":"Hierarchical recurrent neural encoder for video representation with application to captioning","authors":"Pan","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Papineni et al., 2002] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evalu- ation of machine translation. In ACL, pages 311-318, 2002.","title":"Bleu: a method for automatic evaluation of machine translation","authors":"Kishore Papineni, Salim Roukos, Todd Ward, Wei-Jing Zhu","year":"2002","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Ren et al., 2015] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NIPS, pages 91-99, 2015.","title":"Faster r-cnn: Towards real-time object detection with region proposal networks","authors":"Ren","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Song et al., 2016] Jingkuan Song, Lianli Gao, Feiping Nie, Heng Tao Shen, Yan Yan, and Nicu Sebe. Optimized graph learning using partial tags and multiple features for image and video annotation. IEEE Transactions on Image Processing, 25(11):4999-5011, 2016.","title":"Optimized graph learning using partial tags and multiple features for image and video annotation","authors":"Song","year":"2016","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Song et al., 2017] Jingkuan Song, Lianli Gao, Li Liu, Xiaofeng Zhu, and Nicu Sebe. Quantization-based hashing: a general framework for scalable image and video retrieval. Pattern Recog- nition, 2017.","title":"Quantization-based hashing: a general framework for scalable image and video retrieval","authors":"Song","year":"2017","venue":"Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Venugopalan et al., 2014] Subhashini Venugopalan, Huijuan Xu, Jeff Donahue, Marcus Rohrbach, Raymond Mooney, and Kate Saenko. Translating videos to natural language using deep recur- rent neural networks. arXiv preprint arXiv:1412.4729, 2014.","title":"Translating videos to natural language using deep recurrent neural networks","authors":"Subhashini Venugopalan, Huijuan Xu, Jeff Donahue, Marcus Rohrbach, Raymond Mooney, Kate Saenko","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Venugopalan et al., 2015] Subhashini Venugopalan, Marcus Rohrbach, Jeffrey Donahue, Raymond Mooney, Trevor Darrell, and Kate Saenko. Sequence to sequence-video to text. In ICCV, pages 4534-4542, 2015.","title":"Sequence to sequence-video to text","authors":"Subhashini Venugopalan, Marcus Rohrbach, Jeffrey Donahue, Raymond Mooney, Trevor Darrell, Kate Saenko","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Vinyals et al., 2015] Oriol Vinyals, Alexander Toshev, Samy Ben- gio, and Dumitru Erhan. Show and tell: A neural image caption generator. In CVPR, pages 3156-3164, 2015.","title":"Show and tell: A neural image caption generator","authors":"Oriol Vinyals, Alexander Toshev, Samy Bengio, Dumitru Erhan","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wang et al., 2017] Jingdong Wang, Ting Zhang, Jingkuan Song, Nicu Sebe, Heng Tao Shen, et al. A survey on learning to hash. IEEE Transactions on Pattern Analysis and Machine In- telligence, 2017.","title":"A survey on learning to hash","authors":"Jingdong Wang, Ting Zhang, Jingkuan Song, Nicu Sebe, Heng Tao Shen","year":"2017","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Xu et al., 2015] Kelvin Xu, Jimmy Ba, Ryan Kiros, Aaron Courville, Ruslan Salakhutdinov, Richard Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. arXiv preprint arXiv:1502.03044, 2015.","title":"Show, attend and tell: Neural image caption generation with visual attention","authors":"Kelvin Xu, Jimmy Ba, Ryan Kiros, Aaron Courville, Ruslan Salakhutdinov, Richard Zemel, Yoshua Bengio","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Xu et al., 2016] Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr- vtt: A large video description dataset for bridging video and lan- guage. In CVPR, 2016.","title":"Msrvtt: A large video description dataset for bridging video and language","authors":"Jun Xu, Tao Mei, Ting Yao, Yong Rui","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Yang et al., 2016] Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alex Smola. Stacked attention networks for im- age question answering. In CVPR, 2016.","title":"Stacked attention networks for image question answering","authors":"Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Smola","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Yao et al., 2015] Li Yao, Atousa Torabi, Kyunghyun Cho, Nicolas Ballas, Christopher Pal, Hugo Larochelle, and Aaron Courville. Describing videos by exploiting temporal structure. In ICCV, 2015.","title":"Describing videos by exploiting temporal structure","authors":"Li Yao, Atousa Torabi, Kyunghyun Cho, Nicolas Ballas, Christopher Pal, Hugo Larochelle, Aaron Courville","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Yu et al., 2016] Haonan Yu, Jiang Wang, Zhiheng Huang, Yi Yang, and Wei Xu. Video paragraph captioning using hierarchical re- current neural networks. In CVPR, 2016.","title":"Video paragraph captioning using hierarchical recurrent neural networks","authors":"Haonan Yu, Jiang Wang, Zhiheng Huang, Yi Yang, Wei Xu","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zeiler, 2012] Matthew D Zeiler. Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.5701, 2012.","title":"Adadelta: an adaptive learning rate method","authors":"D Matthew, Zeiler","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}