{"type":"ScholarlyArticle","title":"Deep Learning for Video Captioning: A Review","authors":[{"name":"Shaoxiang Chen","slug":"shaoxiang-chen","given":"Shaoxiang","family":"Chen","affiliation":"Shanghai Key Lab of Intelligent Info. Processing , School of Computer Science , Fudan University , China"},{"name":"Ting Yao","slug":"ting-yao","given":"Ting","family":"Yao","affiliation":"JD AI Research , China"},{"name":"Yu-Gang Jiang","slug":"yu-gang-jiang","given":"Yu-Gang","family":"Jiang","affiliation":"Shanghai Key Lab of Intelligent Info. Processing , School of Computer Science , Fudan University , China; Jilian Technology Group (Video++) , Shanghai , China"}],"abstract":"Deep learning has achieved great successes in solving specific artificial intelligence problems recently.\r\nSubstantial progresses are made on Computer Vision (CV) and Natural Language Processing (NLP).\r\nAs a connection between the two worlds of vision and language, video captioning is the task of producing a natural-language utterance (usually a sentence) that describes the visual content of a video. The task is naturally decomposed into two sub-tasks. \r\nOne is to encode a video via a thorough understanding and learn visual representation. The other is caption generation, which decodes the learned representation into a sequential sentence, word by word.\r\nIn this survey, we first formulate the problem of video captioning, then review state-of-the-art methods categorized by their emphasis on vision or language, and followed by a summary of standard datasets and representative approaches.\r\nFinally, we highlight the challenges which are not yet fully understood in this task and present future research directions.","doi":"10.24963/ijcai.2019/877","url":"https://ijcai.org/proceedings/2019/877","pdf":"https://ijcai.org/proceedings/2019/0877.pdf","markdown":"https://ijcai.org/proceedings/2019/877.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/877","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Survey track","track":"Survey track","number":877,"pages":{"first":6283,"last":6290},"keywords":["Machine Learning: Deep Learning","Computer Vision: Video: Events, Activities and Surveillance","Computer Vision: Language and Vision","Natural Language Processing: Natural Language Generation"],"videos":[],"references":[{"position":1,"raw":"[Aafaq et al., 2018] Nayyer Aafaq, Syed Zulqarnain Gilani, Wei Liu, and Ajmal Mian. Video description: A survey of methods, datasets and evaluation metrics. arXiv preprint arXiv:1806.00186, 2018.","title":"Video description: A survey of methods, datasets and evaluation metrics","authors":"Nayyer Aafaq, Wei Syed Zulqarnain Gilani, Ajmal Liu, Mian","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Anderson et al., 2016] Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. SPICE: semantic propositional image caption evaluation. In ECCV, pages 382-398, 2016.","title":"SPICE: semantic propositional image caption evaluation","authors":"Anderson","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Buch et al., 2017] Shyamal Buch, Victor Escorcia, Chuanqi Shen, Bernard Ghanem, and Juan Carlos Niebles. SST: single-stream temporal action proposals. In CVPR, pages 6373-6382, 2017.","title":"SST: single-stream temporal action proposals","authors":"Shyamal Buch, Victor Escorcia, Chuanqi Shen, Bernard Ghanem, Juan Carlos Niebles","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen and Dolan, 2011] David Chen and William B. Dolan. Collecting highly parallel data for paraphrase evaluation. In ACL-HLT, pages 190-200, 2011.","title":"Collecting highly parallel data for paraphrase evaluation","authors":"David Chen, William B Dolan","year":"2011","venue":"ACL-HLT","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen and Jiang, 2019a] Shaoxiang Chen and Yu-Gang Jiang. Motion guided spatial attention for video caption- ing. In AAAI, 2019.","title":"Motion guided spatial attention for video captioning","authors":"Chen, Jiang, Shaoxiang Chen, Yu-Gang Jiang","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen and Jiang, 2019b] Shaoxiang Chen and Yu-Gang Jiang. Semantic proposal for activity localization in videos via sentence query. In AAAI, 2019.","title":"Semantic proposal for activity localization in videos via sentence query","authors":"Chen, Jiang, Shaoxiang Chen, Yu-Gang Jiang","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chen et al., 2017] Shizhe Chen, Jia Chen, Qin Jin, and Alexander G. Hauptmann. Video captioning with guidance of multimodal latent topics. In ACM MM, pages 1838- 1846, 2017.","title":"Video captioning with guidance of multimodal latent topics","authors":"Shizhe Chen, Jia Chen, Qin Jin, Alexander G Hauptmann","year":"2017","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chen et al., 2018a] Hongge Chen, Huan Zhang, Pin-Yu Chen, Jinfeng Yi, and Cho-Jui Hsieh. Attacking visual lan- guage grounding with adversarial examples: A case study on neural image captioning. In ACL, pages 2587-2597, 2018.","title":"Attacking visual language grounding with adversarial examples: A case study on neural image captioning","authors":"Chen","year":"2018","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Chen et al., 2018b] Jingyuan Chen, Xinpeng Chen, Lin Ma, Zequn Jie, and Tat-Seng Chua. Temporally grounding nat- ural sentence in video. In EMNLP, pages 162-171, 2018.","title":"Temporally grounding natural sentence in video","authors":"Chen","year":"2018","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Chen et al., 2019] Jingwen Chen, Yingwei Pan, Yehao Li, Ting Yao, Hongyang Chao, and Tao Mei. Temporal deformable convolutional encoder-decoder networks for video captioning. In AAAI, 2019.","title":"Temporal deformable convolutional encoder-decoder networks for video captioning","authors":"Jingwen Chen, Yingwei Pan, Yehao Li, Ting Yao, Hongyang Chao, Tao Mei","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Das et al., 2013] Pradipto Das, Chenliang Xu, Richard F. Doell, and Jason J. Corso. A thousand frames in just a few words: Lingual description of videos through latent topics and sparse object stitching. In CVPR, pages 2634-2641, 2013.","title":"A thousand frames in just a few words: Lingual description of videos through latent topics and sparse object stitching","authors":"Das","year":"2013","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Duan et al., 2018] Xuguang Duan, Wen-bing Huang, Chuang Gan, Jingdong Wang, Wenwu Zhu, and Junzhou Huang. Weakly supervised dense event captioning in videos. In NeurIPS, pages 3063-3073, 2018.","title":"Weakly supervised dense event captioning in videos","authors":"Xuguang Duan, Wen-Bing Huang, Chuang Gan, Jingdong Wang, Wenwu Zhu, Junzhou Huang","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Escorcia et al., 2016] Victor Escorcia, Fabian Caba Heil- bron, Juan Carlos Niebles, and Bernard Ghanem. Daps: Deep action proposals for action understanding. In ECCV, pages 768-784, 2016.","title":"Daps: Deep action proposals for action understanding","authors":"Victor Escorcia, Fabian Caba Heilbron, Juan Carlos Niebles, Bernard Ghanem","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Fang et al., 2015] Hao Fang, Saurabh Gupta, Forrest N. Ian- dola, Rupesh Kumar Srivastava, Li Deng, Piotr Dollár, Jianfeng Gao, Xiaodong He, Margaret Mitchell, John C. Platt, C. Lawrence Zitnick, and Geoffrey Zweig. From captions to visual concepts and back. In CVPR, pages 1473-1482, 2015.","title":"From captions to visual concepts and back","authors":"Saurabh Hao Fang, Forrest N Gupta, Rupesh Kumar Iandola, Li Srivastava, Piotr Deng, Jianfeng Dollár, Xiaodong Gao, Margaret He, John C Mitchell, C Lawrence Platt, Geoffrey Zitnick, Zweig","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Gao et al., 2017a] Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia. TALL: temporal activity localization via language query. In ICCV, pages 5277-5285, 2017.","title":"TALL: temporal activity localization via language query","authors":"Gao","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Gao et al., 2017b] Lianli Gao, Zhao Guo, Hanwang Zhang, Xing Xu, and Heng Tao Shen. Video captioning with attention-based LSTM and semantic consistency. IEEE Trans. Multimedia, 19(9):2045-2055, 2017.","title":"Video captioning with attention-based LSTM and semantic consistency","authors":"Gao","year":"2017","venue":"IEEE Trans. Multimedia","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Gehring et al., 2017] Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. Convo- lutional sequence to sequence learning. In ICML, pages 1243-1252, 2017.","title":"Convolutional sequence to sequence learning","authors":"Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, Yann N Dauphin","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Ghanem et al., 2017] Bernard Ghanem, Juan Carlos Niebles, Cees Snoek, Fabian Caba Heilbron, Humam Alwassel, Ranjay Krishna, Victor Escorcia, Kenji Hata, and Shyamal Buch. Activitynet challenge 2017 summary. arXiv preprint arXiv:1710.08011, 2017.","title":"Activitynet challenge 2017 summary","authors":"Bernard Ghanem, Juan Carlos Niebles, Cees Snoek, Fabian Caba Heilbron, Humam Alwassel, Ranjay Krishna, Victor Escorcia, Kenji Hata, Shyamal Buch","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Ghanem et al., 2018] Bernard Ghanem, Juan Carlos Niebles, Cees Snoek, Fabian Caba Heilbron, Humam Alwassel, Victor Escorcia, Ranjay Krishna, Shyamal Buch, and Cuong Duc Dao. The activitynet large-scale activity recognition challenge 2018 summary. arXiv preprint arXiv:1808.03766, 2018.","title":"The activitynet large-scale activity recognition challenge 2018 summary","authors":"Bernard Ghanem, Juan Carlos Niebles, Cees Snoek, Fabian Caba Heilbron, Humam Alwassel, Victor Escorcia, Ranjay Krishna, Shyamal Buch, Cuong Duc Dao","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Guadarrama et al., 2013] Sergio Guadarrama, Niveda Kr- ishnamoorthy, Girish Malkarnenkar, Subhashini Venu- gopalan, Raymond J. Mooney, Trevor Darrell, and Kate Saenko. Youtube2text: Recognizing and describing ar- bitrary activities using semantic hierarchies and zero-shot recognition. In ICCV, pages 2712-2719, 2013.","title":"Youtube2text: Recognizing and describing arbitrary activities using semantic hierarchies and zero-shot recognition","authors":"Sergio Guadarrama, Niveda Krishnamoorthy, Girish Malkarnenkar, Subhashini Venugopalan, Raymond J Mooney, Trevor Darrell, Kate Saenko","year":"2013","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Hendricks et al., 2017] Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. Localizing moments in video with natural lan- guage. In ICCV, pages 5804-5813, 2017.","title":"Localizing moments in video with natural language","authors":"Anne Lisa, Oliver Hendricks, Eli Wang, Josef Shechtman, Trevor Sivic, Bryan Darrell, Russell","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Hershey et al., 2017] Shawn Hershey, Sourish Chaudhuri, Daniel P. W. Ellis, Jort F. Gemmeke, Aren Jansen, R. Channing Moore, Manoj Plakal, Devin Platt, Rif A. Saurous, Bryan Seybold, Malcolm Slaney, Ron J. Weiss, and Kevin W. Wilson. CNN architectures for large-scale audio classification. In ICASSP, pages 131-135, 2017.","title":"CNN architectures for large-scale audio classification","authors":"Shawn Hershey, Sourish Chaudhuri, P W Daniel, Jort F Ellis, Aren Gemmeke, R Channing Jansen, Manoj Moore, Devin Plakal, Rif A Platt, Bryan Saurous, Malcolm Seybold, Ron J Slaney, Kevin W Weiss, Wilson","year":"2017","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8):1735-1780, 1997.","title":"Long short-term memory","authors":"Schmidhuber Hochreiter, Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Hori et al., 2017] Chiori Hori, Takaaki Hori, Teng-Yok Lee, Ziming Zhang, Bret Harsham, John R. Hershey, Tim K. Marks, and Kazuhiko Sumi. Attention-based multimodal fusion for video description. In ICCV, pages 4203-4212, 2017.","title":"Attention-based multimodal fusion for video description","authors":"Chiori Hori, Takaaki Hori, Teng-Yok Lee, Ziming Zhang, Bret Harsham, John R Hershey, Tim K Marks, Kazuhiko Sumi","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Kojima et al., 2002] Atsuhiro Kojima, Takeshi Tamura, and Kunio Fukunaga. Natural language description of hu- man activities from video images based on concept hierar- chy of actions. International Journal of Computer Vision, 50(2):171-184, 2002.","title":"Natural language description of human activities from video images based on concept hierarchy of actions","authors":"Atsuhiro Kojima, Takeshi Tamura, Kunio Fukunaga","year":"2002","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Krishna et al., 2017] Ranjay Krishna, Kenji Hata, Fred- eric Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense- captioning events in videos. In ICCV, pages 706-715, 2017.","title":"Densecaptioning events in videos","authors":"Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, Juan Carlos Niebles","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Lavie and Agarwal, 2007] Alon Lavie and Abhaya Agar- wal. METEOR: an automatic metric for MT evaluation with high levels of correlation with human judgments. In WMT@ACL, pages 228-231, 2007.","title":"METEOR: an automatic metric for MT evaluation with high levels of correlation with human judgments","authors":"Alon Lavie, Abhaya Agarwal","year":"2007","venue":"WMT@ACL","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Li et al., 2016] Yuncheng Li, Yale Song, Liangliang Cao, Joel R. Tetreault, Larry Goldberg, Alejandro Jaimes, and Jiebo Luo. TGIF: A new dataset and benchmark on ani- mated GIF description. In CVPR, pages 4641-4650, 2016.","title":"TGIF: A new dataset and benchmark on animated GIF description","authors":"Yuncheng Li, Yale Song, Liangliang Cao, Joel R Tetreault, Larry Goldberg, Alejandro Jaimes, Jiebo Luo","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Li et al., 2017] Xuelong Li, Bin Zhao, and Xiaoqiang Lu. MAM-RNN: multi-level attention model based RNN for video captioning. In IJCAI, pages 2208-2214, 2017.","title":"MAM-RNN: multi-level attention model based RNN for video captioning","authors":"Li","year":"2017","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2017/307"},{"position":31,"raw":"[Li et al., 2018] Yehao Li, Ting Yao, Yingwei Pan, Hongyang Chao, and Tao Mei. Jointly localizing and describing events for dense video captioning. In CVPR, pages 7492-7500, 2018.","title":"Jointly localizing and describing events for dense video captioning","authors":"Li","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Lin et al., 2017] Tianwei Lin, Xu Zhao, and Zheng Shou. Single shot temporal action detection. In ACM MM, pages 988-996, 2017.","title":"Single shot temporal action detection","authors":"Tianwei Lin, Xu Zhao, Zheng Shou","year":"2017","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Liu et al., 2017] Siqi Liu, Zhenhai Zhu, Ning Ye, Sergio Guadarrama, and Kevin Murphy. Improved image caption- ing via policy gradient optimization of spider. In ICCV, pages 873-881, 2017.","title":"Improved image captioning via policy gradient optimization of spider","authors":"Siqi Liu, Zhenhai Zhu, Ning Ye, Sergio Guadarrama, Kevin Murphy","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Pan et al., 2016] Yingwei Pan, Tao Mei, Ting Yao, Houqiang Li, and Yong Rui. Jointly modeling embedding and translation to bridge video and language. In CVPR, pages 4594-4602, 2016.","title":"Jointly modeling embedding and translation to bridge video and language","authors":"Pan","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Pan et al., 2017] Yingwei Pan, Ting Yao, Houqiang Li, and Tao Mei. Video captioning with transferred semantic at- tributes. In CVPR, pages 984-992, 2017.","title":"Video captioning with transferred semantic attributes","authors":"Pan","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Pancoast and Akbacak, 2014] Stephanie Pancoast and Mu- rat Akbacak. Softening quantization in bag-of-audio- words. In ICASSP, pages 1370-1374, 2014.","title":"Softening quantization in bag-of-audiowords","authors":"Stephanie Pancoast, Murat Akbacak","year":"2014","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Papineni et al., 2002] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for au- tomatic evaluation of machine translation. In ACL, pages 311-318, 2002.","title":"Bleu: a method for automatic evaluation of machine translation","authors":"Kishore Papineni, Salim Roukos, Todd Ward, Wei-Jing Zhu","year":"2002","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Pasunuru and Bansal, 2017] Ramakanth Pasunuru and Mo- hit Bansal. Reinforced video captioning with entailment rewards. In EMNLP, pages 979-985, 2017.","title":"Ramakanth Pasunuru and Mohit Bansal. Reinforced video captioning with entailment rewards","authors":"Bansal Pasunuru","year":"2017","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Ramanishka et al., 2016] Vasili Ramanishka, Abir Das, Dong Huk Park, Subhashini Venugopalan, Lisa Anne Hen- dricks, Marcus Rohrbach, and Kate Saenko. Multimodal video description. In ACM MM, pages 1092-1096, 2016.","title":"Multimodal video description","authors":"Vasili Ramanishka, Abir Das, Dong Huk Park, Subhashini Venugopalan, Lisa Anne Hendricks, Marcus Rohrbach, Kate Saenko","year":"2016","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Ranzato et al., 2016] Marc'Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. Sequence level training with recurrent neural networks. In ICLR, 2016.","title":"Sequence level training with recurrent neural networks","authors":"Ranzato","year":"2016","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Regneri et al., 2013] Michaela Regneri, Marcus Rohrbach, Dominikus Wetzel, Stefan Thater, Bernt Schiele, and Man- fred Pinkal. Grounding action descriptions in videos. Transactions of the Association for Computational Lin- guistics, 1:25-36, 2013.","title":"Grounding action descriptions in videos","authors":"Michaela Regneri, Marcus Rohrbach, Dominikus Wetzel, Stefan Thater, Bernt Schiele, Manfred Pinkal","year":"2013","venue":"Transactions of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Rennie et al., 2017] Steven J. Rennie, Etienne Marcheret, Youssef Mroueh, Jarret Ross, and Vaibhava Goel. Self- critical sequence training for image captioning. In CVPR, pages 1179-1195, 2017.","title":"Selfcritical sequence training for image captioning","authors":"Steven J Rennie, Etienne Marcheret, Youssef Mroueh, Jarret Ross, Vaibhava Goel","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Rohrbach et al., 2014] Anna Rohrbach, Marcus Rohrbach, Wei Qiu, Annemarie Friedrich, Manfred Pinkal, and Bernt Schiele. Coherent multi-sentence video description with variable level of detail. In GCPR, pages 184-195, 2014.","title":"Coherent multi-sentence video description with variable level of detail","authors":"Anna Rohrbach, Marcus Rohrbach, Wei Qiu, Annemarie Friedrich, Manfred Pinkal, Bernt Schiele","year":"2014","venue":"GCPR","doi":null,"url":null,"internal_url":null},{"position":44,"raw":"[Rohrbach et al., 2015] Anna Rohrbach, Marcus Rohrbach, Niket Tandon, and Bernt Schiele. A dataset for movie de- scription. In CVPR, pages 3202-3212, 2015.","title":"A dataset for movie description","authors":"Anna Rohrbach, Marcus Rohrbach, Niket Tandon, Bernt Schiele","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":45,"raw":"[Shen et al., 2017] Zhiqiang Shen, Jianguo Li, Zhou Su, Minjun Li, Yurong Chen, Yu-Gang Jiang, and Xiangyang Xue. Weakly supervised dense video captioning. In CVPR, pages 5159-5167, 2017.","title":"Weakly supervised dense video captioning","authors":"Zhiqiang Shen, Jianguo Li, Zhou Su, Minjun Li, Yurong Chen, Yu-Gang Jiang, Xiangyang Xue","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":46,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":47,"raw":"[Song et al., 2017] Jingkuan Song, Lianli Gao, Zhao Guo, Wu Liu, Dongxiang Zhang, and Heng Tao Shen. Hier- archical LSTM with adjusted temporal attention for video captioning. In IJCAI, pages 2737-2743, 2017.","title":"Hierarchical LSTM with adjusted temporal attention for video captioning","authors":"Song","year":"2017","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2017/381"},{"position":48,"raw":"[Szegedy et al., 2016] Christian Szegedy, Vincent Van- houcke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In CVPR, pages 2818-2826, 2016.","title":"Rethinking the inception architecture for computer vision","authors":"Szegedy","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":49,"raw":"[Torabi et al., 2015] Atousa Torabi, Christopher J. Pal, Hugo Larochelle, and Aaron C. Courville. Using descriptive video services to create a large data source for video anno- tation research. arXiv preprint arXiv:1503.01070, 2015.","title":"Using descriptive video services to create a large data source for video annotation research","authors":"Torabi","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":50,"raw":"[Tran et al., 2015] Du Tran, Lubomir D. Bourdev, Rob Fer- gus, Lorenzo Torresani, and Manohar Paluri. Learning spatiotemporal features with 3d convolutional networks. In ICCV, pages 4489-4497, 2015.","title":"Learning spatiotemporal features with 3d convolutional networks","authors":"Du Tran, Lubomir D Bourdev, Rob Fergus, Lorenzo Torresani, Manohar Paluri","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":51,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, pages 6000-6010, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, Illia Polosukhin","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":52,"raw":"[Vedantam et al., 2015] Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. Cider: Consensus- based image description evaluation. In CVPR, pages 4566-4575, 2015.","title":"Cider: Consensusbased image description evaluation","authors":"C Lawrence Ramakrishna Vedantam, Devi Zitnick, Parikh","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":53,"raw":"[Venugopalan et al., 2015] Subhashini Venugopalan, Mar- cus Rohrbach, Jeffrey Donahue, Raymond J. Mooney, Trevor Darrell, and Kate Saenko. Sequence to sequence -video to text. In ICCV, pages 4534-4542, 2015.","title":"Sequence to sequence -video to text","authors":"Subhashini Venugopalan, Marcus Rohrbach, Jeffrey Donahue, Raymond J Mooney, Trevor Darrell, Kate Saenko","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":54,"raw":"[Wang et al., 2018a] Huiyun Wang, Youjiang Xu, and Ya- hong Han. Spotting and aggregating salient regions for video captioning. In ACM MM, pages 1519-1526, 2018.","title":"Spotting and aggregating salient regions for video captioning","authors":"Wang","year":"2018","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":55,"raw":"[Wang et al., 2018b] Jingwen Wang, Wenhao Jiang, Lin Ma, Wei Liu, and Yong Xu. Bidirectional attentive fusion with context gating for dense video captioning. In CVPR, pages 7190-7198, 2018.","title":"Bidirectional attentive fusion with context gating for dense video captioning","authors":"Wang","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":56,"raw":"[Wang et al., 2018c] Xin Wang, Yuan-Fang Wang, and William Yang Wang. Watch, listen, and describe: Glob- ally and locally aligned cross-modal attentions for video captioning. In NAACL-HLT, pages 795-801, 2018.","title":"Watch, listen, and describe: Globally and locally aligned cross-modal attentions for video captioning","authors":"Wang","year":"2018","venue":"NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":57,"raw":"[Williams, 1992] Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforce- ment learning. Machine Learning, 8:229-256, 1992.","title":"Simple statistical gradient-following algorithms for connectionist reinforcement learning","authors":"Ronald J Williams","year":"1992","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":58,"raw":"[Xu et al., 2016] Jun Xu, Tao Mei, Ting Yao, and Yong Rui. MSR-VTT: A large video description dataset for bridging video and language. In CVPR, pages 5288-5296, 2016.","title":"MSR-VTT: A large video description dataset for bridging video and language","authors":"Jun Xu, Tao Mei, Ting Yao, Yong Rui","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":59,"raw":"[Xu et al., 2017] Jun Xu, Ting Yao, Yongdong Zhang, and Tao Mei. Learning multimodal attention LSTM networks for video captioning. In ACM MM, pages 537-545, 2017.","title":"Learning multimodal attention LSTM networks for video captioning","authors":"Xu","year":"2017","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":60,"raw":"[Yao et al., 2015] Li Yao, Atousa Torabi, Kyunghyun Cho, Nicolas Ballas, Christopher J. Pal, Hugo Larochelle, and Aaron C. Courville. Describing videos by exploiting tem- poral structure. In ICCV, pages 4507-4515, 2015.","title":"Describing videos by exploiting temporal structure","authors":"Li Yao, Atousa Torabi, Kyunghyun Cho, Nicolas Ballas, Christopher J Pal, Hugo Larochelle, Aaron C Courville","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":61,"raw":"[Yao et al., 2017] Ting Yao, Yingwei Pan, Yehao Li, and Tao Mei. Incorporating copying mechanism in image caption- ing for learning novel objects. In CVPR, pages 5263-5271, 2017.","title":"Incorporating copying mechanism in image captioning for learning novel objects","authors":"Ting Yao, Yingwei Pan, Yehao Li, Tao Mei","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":62,"raw":"[Yao et al., 2018] Ting Yao, Yingwei Pan, Yehao Li, and Tao Mei. Exploring visual relationship for image captioning. In ECCV, pages 684-699, 2018.","title":"Exploring visual relationship for image captioning","authors":"Ting Yao, Yingwei Pan, Yehao Li, Tao Mei","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":63,"raw":"[You et al., 2016] Quanzeng You, Hailin Jin, Zhaowen Wang, Chen Fang, and Jiebo Luo. Image captioning with semantic attention. In CVPR, pages 4651-4659, 2016.","title":"Image captioning with semantic attention","authors":"You","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":64,"raw":"[Yu et al., 2016] Haonan Yu, Jiang Wang, Zhiheng Huang, Yi Yang, and Wei Xu. Video paragraph captioning using hierarchical recurrent neural networks. In CVPR, pages 4584-4593, 2016.","title":"Video paragraph captioning using hierarchical recurrent neural networks","authors":"Haonan Yu, Jiang Wang, Zhiheng Huang, Yi Yang, Wei Xu","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":65,"raw":"[Zhou et al., 2018] Luowei Zhou, Yannis Kalantidis, Xinlei Chen, Jason J. Corso, and Marcus Rohrbach. Grounded video description. arXiv preprint arXiv:1812.06587, 2018.","title":"Grounded video description","authors":"Zhou","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}