{"type":"ScholarlyArticle","title":"Video Interactive Captioning with Human Prompts","authors":[{"name":"Aming Wu","slug":"aming-wu","given":"Aming","family":"Wu","affiliation":"College of Intelligence and Computing , Tianjin University , Tianjin , China"},{"name":"Yahong Han","slug":"yahong-han","given":"Yahong","family":"Han","affiliation":"College of Intelligence and Computing , Tianjin University , Tianjin , China"},{"name":"Yi Yang","slug":"yi-yang","given":"Yi","family":"Yang","affiliation":"School of Computer Science , University of Technology Sydney , Australia; Baidu Research"}],"abstract":"Video captioning aims at generating a proper sentence to describe the video content. As a video often includes rich visual content and semantic details, different people may be interested in different views. Thus the generated sentence always fails to meet the ad hoc expectations. In this paper, we make a new attempt that, we launch a round of interaction between a human and a captioning agent. After generating an initial caption, the agent asks for a short prompt from the human as a clue of his expectation. Then, based on the prompt, the agent could generate a more accurate caption. We name this process a new task of video interactive captioning (ViCap). Taking a video and an initial caption as input, we devise the ViCap agent which consists of a video encoder, an initial caption encoder, and a refined caption generator. We show that the ViCap can be trained via a full supervision (with ground-truth) way or a weak supervision (with only prompts) way. For the evaluation of ViCap, we first extend the MSRVTT with interaction ground-truth. Experimental results not only show the prompts can help generate more accurate captions, but also demonstrate the good performance of the proposed method.","doi":"10.24963/ijcai.2019/135","url":"https://ijcai.org/proceedings/2019/135","pdf":"https://ijcai.org/proceedings/2019/0135.pdf","markdown":"https://ijcai.org/proceedings/2019/135.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/135","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Computer Vision","track":"Main track","number":135,"pages":{"first":961,"last":967},"keywords":["Computer Vision: Video: Events, Activities and Surveillance","Computer Vision: Language and Vision"],"videos":[],"references":[{"position":1,"raw":"[Baraldi et al., 2017] Lorenzo Baraldi, Costantino Grana, and Rita Cucchiara. Hierarchical boundary-aware neural encoder for video captioning. CVPR, 2017.","title":"Hierarchical boundary-aware neural encoder for video captioning","authors":"Lorenzo Baraldi, Costantino Grana, Rita Cucchiara","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bengio et al., 2012] Yoshua Bengio, Aaron C Courville, and Pascal Vincent. Unsupervised feature learning and deep learning: A review and new perspectives. CoRR, abs/1206.5538, 1:2012, 2012.","title":"Unsupervised feature learning and deep learning: A review and new perspectives","authors":"Yoshua Bengio, Aaron C Courville, Pascal Vincent","year":"2012","venue":"CoRR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2018] Yangyu Chen, Shuhui Wang, Weigang Zhang, and Qingming Huang. Less is more: Picking informative frames for video captioning. arXiv preprint arXiv:1803.01457, 2018.","title":"Less is more: Picking informative frames for video captioning","authors":"Yangyu Chen, Shuhui Wang, Weigang Zhang, Qingming Huang","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cho et al., 2014] Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078, 2014.","title":"Learning phrase representations using rnn encoder-decoder for statistical machine translation","authors":"Cho","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Denkowski and Lavie, 2014] Michael Denkowski and Alon Lavie. Meteor universal: Language specific translation evaluation for any target language. In Proceedings of the ninth workshop on statistical machine translation, pages 376-380, 2014.","title":"Meteor universal: Language specific translation evaluation for any target language","authors":"Lavie Denkowski, Michael Denkowski, Alon Lavie","year":"2014","venue":"Proceedings of the ninth workshop on statistical machine translation","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gehring et al., 2017] Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. Convo- lutional sequence to sequence learning. arXiv preprint arXiv:1705.03122, 2017.","title":"Convolutional sequence to sequence learning","authors":"Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, Yann N Dauphin","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Keneshloo et al., 2018] Yaser Keneshloo, Tian Shi, Chan- dan K Reddy, and Naren Ramakrishnan. Deep reinforce- ment learning for sequence to sequence models. arXiv preprint arXiv:1805.09461, 2018.","title":"Deep reinforcement learning for sequence to sequence models","authors":"Yaser Keneshloo, Tian Shi, Naren Chandan K Reddy, Ramakrishnan","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kordopatis-Zilos et al., 2018] Giorgos Kordopatis-Zilos, Symeon Papadopoulos, Ioannis Patras, and Ioannis Kompatsiaris. Fivr: Fine-grained incident video retrieval. arXiv preprint arXiv:1809.04094, 2018.","title":"Fivr: Fine-grained incident video retrieval","authors":"Giorgos Kordopatis-Zilos, Symeon Papadopoulos, Ioannis Patras, Ioannis Kompatsiaris","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Lamb et al., 2016] Alex M Lamb, Anirudh Goyal ALIAS PARTH GOYAL, Ying Zhang, Saizheng Zhang, Aaron C Courville, and Yoshua Bengio. Professor forcing: A new algorithm for training recurrent networks. In NIPS, pages 4601-4609, 2016.","title":"Professor forcing: A new algorithm for training recurrent networks","authors":"Alex M Lamb, Anirudh Goyal, Alias Parth Goyal, Ying Zhang, Saizheng Zhang, Aaron C Courville, Yoshua Bengio","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li et al., 2017] Xuelong Li, Bin Zhao, and Xiaoqiang Lu. Mam-rnn: multi-level attention model based rnn for video captioning. In IJCAI, 2017.","title":"Mam-rnn: multi-level attention model based rnn for video captioning","authors":"Xuelong Li, Bin Zhao, Xiaoqiang Lu","year":"2017","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2017/307"},{"position":11,"raw":"[Oord et al., 2016] Aaron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. arXiv preprint arXiv:1601.06759, 2016.","title":"Pixel recurrent neural networks","authors":"Aaron Van Den Oord, Nal Kalchbrenner, Koray Kavukcuoglu","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Pan et al., 2016] Pingbo Pan, Zhongwen Xu, Yi Yang, Fei Wu, and Yueting Zhuang. Hierarchical recurrent neural encoder for video representation with application to cap- tioning. In CVPR, pages 1029-1038, 2016.","title":"Hierarchical recurrent neural encoder for video representation with application to captioning","authors":"Pan","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Pan et al., 2017] Yingwei Pan, Ting Yao, Houqiang Li, and Tao Mei. Video captioning with transferred semantic at- tributes. In CVPR, 2017.","title":"Video captioning with transferred semantic attributes","authors":"Pan","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Papineni et al., 2002] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for au- tomatic evaluation of machine translation. In ACL, pages 311-318. ACL, 2002.","title":"Bleu: a method for automatic evaluation of machine translation","authors":"Kishore Papineni, Salim Roukos, Todd Ward, Wei-Jing Zhu","year":"2002","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Pasunuru and Bansal, 2018] Ramakanth Pasunuru and Mo- hit Bansal. Game-based video-context dialogue. arXiv preprint arXiv:1809.04560, 2018.","title":"Ramakanth Pasunuru and Mohit Bansal. Game-based video-context dialogue","authors":"Bansal Pasunuru","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Phan et al., 2017] Sang Phan, Gustav Eje Henter, Yusuke Miyao, and Shin'ichi Satoh. Consensus-based se- quence training for video captioning. arXiv preprint arXiv:1712.09532, 2017.","title":"Consensus-based sequence training for video captioning","authors":"Sang Phan, Gustav Eje Henter, Yusuke Miyao, Shin'ichi Satoh","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Ramakrishna et al., 2015] Vedantam Ramakrishna, Parikh Devi, and C Lawrence Zitnick. Cider: Consensus-based image description evaluation. In CVPR, pages 4566-4575, 2015.","title":"Cider: Consensus-based image description evaluation","authors":"Vedantam Ramakrishna, Parikh Devi, C Lawrence, Zitnick","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Rocchio, 1971] Joseph John Rocchio. Relevance feedback in information retrieval. The SMART retrieval system: ex- periments in automatic document processing, pages 313- 323, 1971.","title":"Relevance feedback in information retrieval. The SMART retrieval system: experiments in automatic document processing","authors":"Joseph John, Rocchio","year":"1971","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"Szegedy et al., ] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, and Dragomir Anguelov. Go- ing deeper with convolutions. In CVPR, pages 1-9.","title":"Going deeper with convolutions","authors":"Szegedy","year":null,"venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Venugopalan et al., 2015] Subhashini Venugopalan, Mar- cus Rohrbach, Jeffrey Donahue, Raymond Mooney, Trevor Darrell, and Kate Saenko. Sequence to sequence- video to text. In ICCV, pages 4534-4542, 2015.","title":"Sequence to sequencevideo to text","authors":"Subhashini Venugopalan, Marcus Rohrbach, Jeffrey Donahue, Raymond Mooney, Trevor Darrell, Kate Saenko","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang et al., 2016] Junbo Wang, Wei Wang, Yan Huang, Liang Wang, and Tieniu Tan. Multimodal mem- ory modelling for video captioning. arXiv preprint arXiv:1611.05592, 2016.","title":"Multimodal memory modelling for video captioning","authors":"Junbo Wang, Wei Wang, Yan Huang, Liang Wang, Tieniu Tan","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang et al., 2018] Xin Wang, Wenhu Chen, Jiawei Wu, Yuan-Fang Wang, and William Yang Wang. Video cap- tioning via hierarchical reinforcement learning. In CVPR, pages 4213-4222, 2018.","title":"Video captioning via hierarchical reinforcement learning","authors":"Xin Wang, Wenhu Chen, Jiawei Wu, Yuan-Fang Wang, William Yang, Wang","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wu and Han, 2018] Aming Wu and Yahong Han. Multi- modal circulant fusion for video-to-language and back- ward. In IJCAI, volume 3, page 8, 2018.","title":"Multimodal circulant fusion for video-to-language and backward","authors":"Han ; Wu, Aming Wu, Yahong Han","year":"2018","venue":"IJCAI","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Xu et al., 2016] Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In CVPR, pages 5288-5296, 2016.","title":"Msr-vtt: A large video description dataset for bridging video and language","authors":"Jun Xu, Tao Mei, Ting Yao, Yong Rui","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yao et al., 2015] Li Yao, Atousa Torabi, Kyunghyun Cho, Nicolas Ballas, Christopher Pal, Hugo Larochelle, and Aaron Courville. Describing videos by exploiting tempo- ral structure. In ICCV, pages 4507-4515, 2015.","title":"Describing videos by exploiting temporal structure","authors":"Li Yao, Atousa Torabi, Kyunghyun Cho, Nicolas Ballas, Christopher Pal, Hugo Larochelle, Aaron Courville","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yu and Koltun, 2015] Fisher Yu and Vladlen Koltun. Multi- scale context aggregation by dilated convolutions. arXiv preprint arXiv:1511.07122, 2015.","title":"Multiscale context aggregation by dilated convolutions","authors":"Koltun ; Fisher Yu, Vladlen Yu, Koltun","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang et al., 2016] Yuting Zhang, Kibok Lee, and Honglak Lee. Augmenting supervised neural networks with unsu- pervised objectives for large-scale image classification. In ICML, pages 612-621, 2016.","title":"Augmenting supervised neural networks with unsupervised objectives for large-scale image classification","authors":"Yuting Zhang, Kibok Lee, Honglak Lee","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}