{"type":"ScholarlyArticle","title":"Knowledgeable Storyteller: A Commonsense-Driven Generative Model for Visual Storytelling","authors":[{"name":"Pengcheng Yang","slug":"pengcheng-yang","given":"Pengcheng","family":"Yang","affiliation":"Deep Learning Lab , Beijing Institute of Big Data Research , Peking University; MOE Key Lab of Computational Linguistics , School of EECS , Peking University"},{"name":"Fuli Luo","slug":"fuli-luo","given":"Fuli","family":"Luo","affiliation":"MOE Key Lab of Computational Linguistics , School of EECS , Peking University"},{"name":"Peng Chen","slug":"peng-chen","given":"Peng","family":"Chen","affiliation":"MOE Key Lab of Computational Linguistics , School of EECS , Peking University"},{"name":"Lei Li","slug":"lei-li","given":"Lei","family":"Li","affiliation":"MOE Key Lab of Computational Linguistics , School of EECS , Peking University"},{"name":"Zhiyi Yin","slug":"zhiyi-yin","given":"Zhiyi","family":"Yin","affiliation":"MOE Key Lab of Computational Linguistics , School of EECS , Peking University"},{"name":"Xiaodong He","slug":"xiaodong-he","given":"Xiaodong","family":"He","affiliation":"JD AI Research , China"},{"name":"Xu Sun","slug":"xu-sun","given":"Xu","family":"Sun","affiliation":"Deep Learning Lab , Beijing Institute of Big Data Research , Peking University; MOE Key Lab of Computational Linguistics , School of EECS , Peking University"}],"abstract":"The visual storytelling (VST) task aims at generating a reasonable and coherent paragraph-level story with the image stream as input. Different from caption that is a direct and literal description of image content, the story in the VST task tends to contain plenty of imaginary concepts that do not appear in the image. This requires the AI agent to reason and associate with the imaginary concepts based on implicit commonsense knowledge to generate a reasonable story describing the image stream. Therefore, in this work, we present a commonsense-driven generative model, which aims to introduce crucial commonsense from the external knowledge base for visual storytelling. Our approach first extracts a set of candidate knowledge graphs from the knowledge base. Then, an elaborately designed vision-aware directional encoding schema is adopted to effectively integrate the most informative commonsense. Besides, we strive to maximize the semantic similarity within the output during decoding to enhance the coherence of the generated text. Results show that our approach can outperform the state-of-the-art systems by a large margin, which achieves a 29\\% relative improvement of CIDEr score. With additional commonsense and semantic-relevance based objective, the generated stories are more diverse and coherent.","doi":"10.24963/ijcai.2019/744","url":"https://ijcai.org/proceedings/2019/744","pdf":"https://ijcai.org/proceedings/2019/0744.pdf","markdown":"https://ijcai.org/proceedings/2019/744.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/744","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Natural Language Processing","track":"Main track","number":744,"pages":{"first":5356,"last":5362},"keywords":["Natural Language Processing: Natural Language Processing","Natural Language Processing: Natural Language Generation"],"videos":[],"references":[{"position":1,"raw":"[Bahdanau et al., 2014] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014.","title":"Neural machine translation by jointly learning to align and translate","authors":"Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2017] Zhiqian Chen, Xuchao Zhang, Arnold P. Boedihardjo, Jing Dai, and Chang-Tien Lu. Multimodal storytelling via generative adversarial imitation learning. In IJCAI, pages 3967-3973, 2017.","title":"Multimodal storytelling via generative adversarial imitation learning","authors":"Chen","year":"2017","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2017/554"},{"position":3,"raw":"[Cho et al., 2014] Kyunghyun Cho, Bart van Merrienboer, C ¸aglar Gülc ¸ehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In EMNLP, pages 1724-1734, 2014.","title":"Learning phrase representations using RNN encoder-decoder for statistical machine translation","authors":"Cho","year":"2014","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Gonzalez-Rico, 2018] Diana Gonzalez-Rico. Contextual- ize, show and tell: A neural visual storyteller. arXiv preprint arXiv:1806.00738, 2018.","title":"Contextualize, show and tell: A neural visual storyteller","authors":"Diana Gonzalez-Rico","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Guan et al., 2018] Jian Guan, Yansen Wang, and Minlie Huang. Story ending generation with incremental en- coding and commonsense knowledge. arXiv preprint arXiv:1808.10113, 2018.","title":"Story ending generation with incremental encoding and commonsense knowledge","authors":"Jian Guan, Yansen Wang, Minlie Huang","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hsu et al., 2018] Chao-Chun Hsu, Szu-Min Chen, Ming- Hsun Hsieh, and Lun-Wei Ku. Using inter-sentence di- verse beam search to reduce redundancy in visual story- telling. arXiv preprint arXiv:1805.11867, 2018.","title":"Using inter-sentence diverse beam search to reduce redundancy in visual storytelling","authors":"Chao-Chun, Szu-Min Hsu, Ming-Hsun Chen, Lun-Wei Hsieh, Ku","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Huang et al., 2016] Ting-Hao (Kenneth) Huang, Francis Ferraro, Nasrin Mostafazadeh, Ishan Misra, Aishwarya Agrawal, Jacob Devlin, Ross B. Girshick, Xiaodong He, Pushmeet Kohli, Dhruv Batra, C. Lawrence Zitnick, Devi Parikh, Lucy Vanderwende, Michel Galley, and Margaret Mitchell. Visual storytelling. In NAACL-HIT, pages 1233- 1239, 2016.","title":"Visual storytelling","authors":"Ting-Hao ( Kenneth, ) Huang, Francis Ferraro, Nasrin Mostafazadeh, Ishan Misra, Aishwarya Agrawal, Jacob Devlin, Ross B Girshick, Xiaodong He, Pushmeet Kohli, Dhruv Batra, C Lawrence Zitnick, Devi Parikh, Lucy Vanderwende, Michel Galley, Margaret Mitchell","year":"2016","venue":"NAACL-HIT","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Huang et al., 2018] Qiuyuan Huang, Zhe Gan, Asli Celiky- ilmaz, Dapeng Wu, Jianfeng Wang, and Xiaodong He. Hierarchically structured reinforcement learning for top- ically coherent visual story generation. arXiv preprint arXiv:1805.08191, 2018.","title":"Hierarchically structured reinforcement learning for topically coherent visual story generation","authors":"Qiuyuan Huang, Zhe Gan, Asli Celikyilmaz, Dapeng Wu, Jianfeng Wang, Xiaodong He","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jain et al., 2017] Parag Jain, Priyanka Agrawal, Abhijit Mishra, Mohak Sukhwani, Anirban Laha, and Karthik Sankaranarayanan. Story generation from sequence of independent short descriptions. arXiv preprint arXiv:1707.05501, 2017.","title":"Story generation from sequence of independent short descriptions","authors":"Parag Jain, Priyanka Agrawal, Abhijit Mishra, Mohak Sukhwani, Anirban Laha, Karthik Sankaranarayanan","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kim et al., 2017] Yoon Kim, Carl Denton, Luong Hoang, and Alexander M Rush. Structured attention networks. arXiv preprint arXiv:1702.00887, 2017.","title":"Structured attention networks","authors":"Yoon Kim, Carl Denton, Luong Hoang, Alexander M Rush","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kim et al., 2018] Taehyeong Kim, Min-Oh Heo, Seonil Son, Kyoung-Wha Park, and Byoung-Tak Zhang. Glac net: Glocal attention cascading networks for multi-image cued story generation. arXiv preprint arXiv:1805.10973, 2018.","title":"Glac net: Glocal attention cascading networks for multi-image cued story generation","authors":"Kim","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kingma and Ba, 2014] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lewis et al., 2018] Mike Lewis, Yann Dauphin, and Angela Fan. Hierarchical neural story generation. In ACL, pages 889-898, 2018.","title":"Hierarchical neural story generation","authors":"Mike Lewis, Yann Dauphin, Angela Fan","year":"2018","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ma et al., 2017] Shuming Ma, Xu Sun, Jingjing Xu, Houfeng Wang, Wenjie Li, and Qi Su. Improving seman- tic relevance for sequence-to-sequence learning of chinese social media text summarization. In ACL, pages 635-640, 2017.","title":"Improving semantic relevance for sequence-to-sequence learning of chinese social media text summarization","authors":"Shuming Ma, Xu Sun, Jingjing Xu, Houfeng Wang, Wenjie Li, Qi Su","year":"2017","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Mostafazadeh et al., 2016] Nasrin Mostafazadeh, Ishan Misra, Jacob Devlin, Margaret Mitchell, Xiaodong He, and Lucy Vanderwende. Generating natural questions about an image. In ACL, 2016.","title":"Generating natural questions about an image","authors":"Nasrin Mostafazadeh, Ishan Misra, Jacob Devlin, Margaret Mitchell, Xiaodong He, Lucy Vanderwende","year":"2016","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Speer and Havasi, 2012] Robert Speer and Catherine Havasi. Representing general relational knowledge in conceptnet 5. In LREC, pages 3679-3686, 2012.","title":"Representing general relational knowledge in conceptnet 5","authors":"Havasi Speer, Robert Speer, Catherine Havasi","year":"2012","venue":"LREC","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Sutskever et al., 2014] Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to sequence learning with neural networks. In NIPS, pages 3104-3112, 2014.","title":"Le. Sequence to sequence learning with neural networks","authors":"Sutskever","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, pages 6000-6010, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, Illia Polosukhin","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wang et al., 2018a] Jing Wang, Jianlong Fu, Jinhui Tang, Zechao Li, and Tao Mei. Show, reward and tell: Auto- matic generation of narrative paragraph from photo stream by adversarial training. In AAAI, pages 7396-7403, 2018.","title":"Show, reward and tell: Automatic generation of narrative paragraph from photo stream by adversarial training","authors":"Wang","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang et al., 2018b] Xin Wang, Wenhu Chen, Yuan-Fang Wang, and William Yang Wang. No metrics are perfect: Adversarial reward learning for visual storytelling. In ACL, pages 899-909, 2018.","title":"No metrics are perfect: Adversarial reward learning for visual storytelling","authors":"Wang","year":"2018","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Xu et al., 2018] Jingjing Xu, Xuancheng Ren, Yi Zhang, Qi Zeng, Xiaoyan Cai, and Xu Sun. A skeleton-based model for promoting coherence among sentences in narra- tive story generation. In EMNLP, pages 4306-4315, 2018.","title":"A skeleton-based model for promoting coherence among sentences in narrative story generation","authors":"Jingjing Xu, Xuancheng Ren, Yi Zhang, Qi Zeng, Xiaoyan Cai, Xu Sun","year":"2018","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yang et al., 2018] Pengcheng Yang, Xu Sun, Wei Li, Shum- ing Ma, Wei Wu, and Houfeng Wang. SGM: sequence generation model for multi-label classification. In COL- ING, pages 3915-3926, 2018.","title":"SGM: sequence generation model for multi-label classification","authors":"Pengcheng Yang, Xu Sun, Wei Li, Shuming Ma, Wei Wu, Houfeng Wang","year":"2018","venue":"COL-ING","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yao et al., 2018] Lili Yao, Nanyun Peng, Weischedel Ralph, Kevin Knight, Dongyan Zhao, and Rui Yan. Plan- and-write: Towards better automatic storytelling. arXiv preprint arXiv:1811.05701, 2018.","title":"Planand-write: Towards better automatic storytelling","authors":"Lili Yao, Nanyun Peng, Weischedel Ralph, Kevin Knight, Dongyan Zhao, Rui Yan","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yu et al., 2017] Licheng Yu, Mohit Bansal, and Tamara L. Berg. Hierarchically-attentive RNN for album summariza- tion and storytelling. In EMNLP, pages 966-971, 2017.","title":"Hierarchically-attentive RNN for album summarization and storytelling","authors":"Licheng Yu, Mohit Bansal, Tamara L Berg","year":"2017","venue":"EMNLP","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}