{"type":"ScholarlyArticle","title":"Video Captioning with Tube Features","authors":[{"name":"Bin Zhao","slug":"bin-zhao","given":"Bin","family":"Zhao","affiliation":"School of Computer Science and Center for OPTical IMagery Analysis and Learning (OPTIMAL) , Northwestern Polytechnical University , Xi'an 710072 , P. R. China"},{"name":"Xuelong Li","slug":"xuelong-li","given":"Xuelong","family":"Li","affiliation":"Xi'an Institute of Optics and Precision Mechanics , Chinese Academy of Sciences , Xi'an 710119 , P. R. China"},{"name":"Xiaoqiang Lu","slug":"xiaoqiang-lu","given":"Xiaoqiang","family":"Lu","affiliation":"Xi'an Institute of Optics and Precision Mechanics , Chinese Academy of Sciences , Xi'an 710119 , P. R. China"}],"abstract":"Visual feature plays an important role in the video captioning task. Considering that the video content is mainly composed of the activities of salient objects, it has restricted the caption quality of current approaches which just focus on global frame features while paying less attention to the salient objects. To tackle this problem, in this paper, we design an object-aware feature for video captioning, denoted as tube feature. Firstly, Faster-RCNN is employed to extract object regions in frames, and a tube generation method is developed to connect the regions from different frames but belonging to the same object. After that, an encoder-decoder architecture\r\nis constructed for video caption generation. Specifically, the encoder is a bi-directional LSTM, which is utilized to capture the dynamic information of each tube. The decoder is a single LSTM extended with an attention model, which enables our approach to adaptively attend to the most correlated tubes when generating the caption. We evaluate our approach on two benchmark datasets: MSVD and Charades. The experimental results have demonstrated the effectiveness of tube feature in the video captioning task.","doi":"10.24963/ijcai.2018/164","url":"https://ijcai.org/proceedings/2018/164","pdf":"https://ijcai.org/proceedings/2018/0164.pdf","markdown":"https://ijcai.org/proceedings/2018/164.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/164","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Computer Vision","track":"Main track","number":164,"pages":{"first":1177,"last":1183},"keywords":["Computer Vision: Video: Events, Activities and Surveillance","Computer Vision: Recognition: Detection, Categorization, Indexing, Matching, Retrieval, Semantic Interpretation"],"videos":[],"references":[{"position":1,"raw":"[Baraldi et al., 2017] Lorenzo Baraldi, Costantino Grana, and Rita Cucchiara. Hierarchical boundary-aware neural encoder for video captioning. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, pages 3185- 3194, 2017.","title":"Hierarchical boundary-aware neural encoder for video captioning","authors":"Lorenzo Baraldi, Costantino Grana, Rita Cucchiara","year":"2017","venue":"2017 IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Denkowski and Lavie, 2014] Michael J. Denkowski and Alon Lavie. Meteor universal: Language specific trans- lation evaluation for any target language. In Workshop on Statistical Machine Translation, 2014.","title":"Meteor universal: Language specific translation evaluation for any target language","authors":"J Michael, Alon Denkowski, Lavie","year":"2014","venue":"Workshop on Statistical Machine Translation","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Fakoor et al., 2016] Rasool Fakoor, Abdel-rahman Mo- hamed, Margaret Mitchell, Sing Bing Kang, and Push- meet Kohli. Memory-augmented attention modelling for videos. CoRR, abs/1611.02261, 2016.","title":"Memory-augmented attention modelling for videos","authors":"Rasool Fakoor, Abdel-Rahman Mohamed, Margaret Mitchell, Sing, Bing Kang, Pushmeet Kohli","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Gkioxari and Malik, 2015] Georgia Gkioxari and Jitendra Malik. Finding action tubes. In IEEE Conference on Com- puter Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, pages 759-768, 2015.","title":"Georgia Gkioxari and Jitendra Malik. Finding action tubes","authors":"Malik Gkioxari","year":"2015","venue":"IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Guadarrama et al., 2013] Sergio Guadarrama, Niveda Kr- ishnamoorthy, Girish Malkarnenkar, Subhashini Venu- gopalan, Raymond J. Mooney, Trevor Darrell, and Kate Saenko. Youtube2text: Recognizing and describing ar- bitrary activities using semantic hierarchies and zero-shot recognition. In ICCV, 2013.","title":"Youtube2text: Recognizing and describing arbitrary activities using semantic hierarchies and zero-shot recognition","authors":"Sergio Guadarrama, Niveda Krishnamoorthy, Girish Malkarnenkar, Subhashini Venugopalan, Raymond J Mooney, Trevor Darrell, Kate Saenko","year":"2013","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8), 1997.","title":"Long short-term memory","authors":"Schmidhuber Hochreiter, Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hori et al., 2017] Chiori Hori, Takaaki Hori, Teng-Yok Lee, Ziming Zhang, Bret Harsham, John R. Hershey, Tim K. Marks, and Kazuhiko Sumi. Attention-based multimodal fusion for video description. In ICCV, pages 4203-4212, 2017.","title":"Attention-based multimodal fusion for video description","authors":"Chiori Hori, Takaaki Hori, Teng-Yok Lee, Ziming Zhang, Bret Harsham, John R Hershey, Tim K Marks, Kazuhiko Sumi","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kalchbrenner and Blunsom, 2013] Nal Kalchbrenner and Phil Blunsom. Recurrent continuous translation models. In EMNLP, 2013.","title":"Recurrent continuous translation models","authors":"Nal Kalchbrenner, Phil Blunsom","year":"2013","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li et al., 2017] Xuelong Li, Bin Zhao, and Xiaoqiang Lu. MAM-RNN: multi-level attention model based RNN for video captioning. In IJCAI, 2017.","title":"MAM-RNN: multi-level attention model based RNN for video captioning","authors":"Xuelong Li, Bin Zhao, Xiaoqiang Lu","year":"2017","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2017/307"},{"position":11,"raw":"[Lin and Och, 2004] Chin-Yew Lin and Franz Josef Och. Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics. In ACL, 2004.","title":"Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics","authors":"Lin, Och ; Chin-Yew Lin, Franz Josef, Och","year":"2004","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Long et al., 2016] Xiang Long, Chuang Gan, and Gerard de Melo. Video captioning with multi-faceted attention. CoRR, abs/1612.00234, 2016.","title":"Video captioning with multi-faceted attention","authors":"Xiang Long, Chuang Gan, Gerard De, Melo","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Luong et al., 2015] Thang Luong, Hieu Pham, and Christo- pher D. Manning. Effective approaches to attention-based neural machine translation. In EMNLP, 2015.","title":"Effective approaches to attention-based neural machine translation","authors":"Thang Luong, Hieu Pham, Christopher D Manning","year":"2015","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Pan et al., 2016a] Pingbo Pan, Zhongwen Xu, Yi Yang, Fei Wu, and Yueting Zhuang. Hierarchical recurrent neural encoder for video representation with application to cap- tioning. In CVPR, 2016.","title":"Hierarchical recurrent neural encoder for video representation with application to captioning","authors":"Pan","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Pan et al., 2016b] Yingwei Pan, Tao Mei, Ting Yao, Houqiang Li, and Yong Rui. Jointly modeling embedding and translation to bridge video and language. In CVPR, 2016.","title":"Jointly modeling embedding and translation to bridge video and language","authors":"Pan","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Papineni et al., 2002] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for au- tomatic evaluation of machine translation. In ACL, 2002.","title":"Bleu: a method for automatic evaluation of machine translation","authors":"Kishore Papineni, Salim Roukos, Todd Ward, Wei-Jing Zhu","year":"2002","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Pennington et al., 2014] Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectors for word representation. In EMNLP, 2014.","title":"Glove: Global vectors for word representation","authors":"Jeffrey Pennington, Richard Socher, Christopher D Manning","year":"2014","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Ren et al., 2015] Shaoqing Ren, Kaiming He, Ross Gir- shick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NIPS, 2015.","title":"Faster r-cnn: Towards real-time object detection with region proposal networks","authors":"Ren","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sigurdsson et al., 2016] Gunnar A. Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta. Hollywood in homes: Crowdsourcing data collec- tion for activity understanding. In ECCV, 2016.","title":"Hollywood in homes: Crowdsourcing data collection for activity understanding","authors":"A Gunnar, Gül Sigurdsson, Xiaolong Varol, Ali Wang, Ivan Farhadi, Abhinav Laptev, Gupta","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR, abs/1409.1556, 2014.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Szegedy et al., 2015] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott E. Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabi- novich. Going deeper with convolutions. In CVPR, 2015.","title":"Going deeper with convolutions","authors":"Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott E Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, Andrew Rabinovich","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Thomason et al., 2014] Jesse Thomason, Subhashini Venu- gopalan, Sergio Guadarrama, Kate Saenko, and Ray- mond J. Mooney. Integrating language and vision to gen- erate natural language descriptions of videos in the wild. In ICCL, 2014.","title":"Integrating language and vision to generate natural language descriptions of videos in the wild","authors":"Jesse Thomason, Subhashini Venugopalan, Sergio Guadarrama, Kate Saenko, Raymond J Mooney","year":"2014","venue":"ICCL","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Tran et al., 2015] Du Tran, Lubomir D. Bourdev, Rob Fer- gus, Lorenzo Torresani, and Manohar Paluri. Learning s- patiotemporal features with 3d convolutional networks. In ICCV, 2015.","title":"Learning spatiotemporal features with 3d convolutional networks","authors":"Du Tran, Lubomir D Bourdev, Rob Fergus, Lorenzo Torresani, Manohar Paluri","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Vedantam et al., 2015] Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. Cider: Consensus- based image description evaluation. In CVPR, 2015.","title":"Cider: Consensusbased image description evaluation","authors":"C Lawrence Ramakrishna Vedantam, Devi Zitnick, Parikh","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Venugopalan et al., 2015a] Subhashini Venugopalan, Mar- cus Rohrbach, Jeffrey Donahue, Raymond J. Mooney, Trevor Darrell, and Kate Saenko. Sequence to sequence -video to text. In ICCV, 2015.","title":"Sequence to sequence -video to text","authors":"Venugopalan","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Venugopalan et al., 2015b] Subhashini Venugopalan, Hui- juan Xu, Jeff Donahue, Marcus Rohrbach, Raymond J. Mooney, and Kate Saenko. Translating videos to natural language using deep recurrent neural networks. In ACL, 2015.","title":"Translating videos to natural language using deep recurrent neural networks","authors":"Venugopalan","year":"2015","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Yao et al., 2015] Li Yao, Atousa Torabi, Kyunghyun Cho, Nicolas Ballas, Christopher J. Pal, Hugo Larochelle, and Aaron C. Courville. Describing videos by exploiting tem- poral structure. In ICCV, 2015.","title":"Describing videos by exploiting temporal structure","authors":"Li Yao, Atousa Torabi, Kyunghyun Cho, Nicolas Ballas, Christopher J Pal, Hugo Larochelle, Aaron C Courville","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Yu et al., 2016] Haonan Yu, Jiang Wang, Zhiheng Huang, Yi Yang, and Wei Xu. Video paragraph captioning using hierarchical recurrent neural networks. In CVPR, 2016.","title":"Video paragraph captioning using hierarchical recurrent neural networks","authors":"Haonan Yu, Jiang Wang, Zhiheng Huang, Yi Yang, Wei Xu","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}