{"type":"ScholarlyArticle","title":"Controllable Neural Story Plot Generation via Reward Shaping","authors":[{"name":"Pradyumna Tambwekar","slug":"pradyumna-tambwekar","given":"Pradyumna","family":"Tambwekar","affiliation":"School of Interactive Computing , Georgia Institute of Technology"},{"name":"Murtaza Dhuliawala","slug":"murtaza-dhuliawala","given":"Murtaza","family":"Dhuliawala","affiliation":"School of Interactive Computing , Georgia Institute of Technology"},{"name":"Lara J. Martin","slug":"lara-j-martin","given":"Lara J.","family":"Martin","affiliation":"School of Interactive Computing , Georgia Institute of Technology"},{"name":"Animesh Mehta","slug":"animesh-mehta","given":"Animesh","family":"Mehta","affiliation":"School of Interactive Computing , Georgia Institute of Technology"},{"name":"Brent Harrison","slug":"brent-harrison","given":"Brent","family":"Harrison","affiliation":"Department of Computer Science , University of Kentucky {ptambwekar3 ,"},{"name":"Mark O. Riedl","slug":"mark-o-riedl","given":"Mark O.","family":"Riedl","affiliation":"School of Interactive Computing , Georgia Institute of Technology"}],"abstract":"Language-modeling--based approaches to story plot generation attempt to construct a plot by sampling from a language model (LM) to predict the next character, word, or sentence to add to the story. LM techniques lack the ability to receive guidance from the user to achieve a specific goal, resulting in stories that don't have a clear sense of progression and lack coherence. We present a reward-shaping technique that analyzes a story corpus and produces intermediate rewards that are backpropagated into a pre-trained LM in order to guide the model toward a given goal. Automated evaluations show our technique can create a model that generates story plots which consistently achieve a specified goal. Human-subject studies show that the generated stories have more plausible event ordering than baseline plot generation techniques.","doi":"10.24963/ijcai.2019/829","url":"https://ijcai.org/proceedings/2019/829","pdf":"https://ijcai.org/proceedings/2019/0829.pdf","markdown":"https://ijcai.org/proceedings/2019/829.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/829","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"AI for Improving Human Well-being","track":"AI for Improving Human Well-being","number":829,"pages":{"first":5982,"last":5988},"keywords":["Special Track on AI for Improving Human-Well Being: Entertainment;games applications (Special Track on AI and Human Wellbeing)"],"videos":[],"references":[{"position":1,"raw":"[Bamman et al., 2013] David Bamman, Brendan O'Connor, and Noah A. Smith. Learning Latent Personas of Film Characters. In ACL, pages 352-361, 2013.","title":"Learning Latent Personas of Film Characters","authors":"David Bamman, O' Brendan, Noah A Connor, Smith","year":"2013","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cavazza et al., 2002] Marc Cavazza, Fred Charles, and Steven J. Mead. Planning characters' behaviour in inter- active storytelling. Journal of Visualization and Computer Animation, 13:121-131, 2002.","title":"Planning characters' behaviour in interactive storytelling","authors":"Marc Cavazza, Fred Charles, Steven J Mead","year":"2002","venue":"Journal of Visualization and Computer Animation","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Fan et al., 2018] Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical Neural Story Generation. In ACL, pages 889-898, 2018.","title":"Hierarchical Neural Story Generation","authors":"Angela Fan, Mike Lewis, Yann Dauphin","year":"2018","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Gehring et al., 2017] Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. Convolu- tional Sequence to Sequence Learning. In ICML, 2017.","title":"Convolutional Sequence to Sequence Learning","authors":"Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, Yann N Dauphin","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Gervás et al., 2005] Pablo Gervás, Belen Díaz-Agudo, Fed- erico Peinado, and Raquel Hervás. Story plot generation based on CBR. Journal of Knowledge-Based Systems, 18(4-5):235-242, 2005.","title":"Story plot generation based on CBR","authors":"Gervás","year":"2005","venue":"Journal of Knowledge-Based Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Jenks and Caspall, 1971] George F. Jenks and Fred C Cas- pall. Error on choroplethic maps: definition, measurement, reduction. Annals of the Association of American Geogra- phers, 61(2):217-244, 1971.","title":"Error on choroplethic maps: definition, measurement, reduction","authors":"George F Jenks, Fred C Caspall","year":"1971","venue":"Annals of the Association of American Geographers","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Khalifa et al., 2017] Ahmed Khalifa, Gabriella AB Barros, and Julian Togelius. Deeptingle. In ICCC, 2017.","title":"Deeptingle","authors":"Ahmed Khalifa, Gabriella Ab Barros, Julian Togelius","year":"2017","venue":"ICCC","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Khandelwal et al., 2018] Urvashi Khandelwal, He He, Peng Qi, and Dan Jurafsky. Sharp nearby, fuzzy far away: How neural language models use context. In ACL, 2018.","title":"Sharp nearby, fuzzy far away: How neural language models use context","authors":"Urvashi Khandelwal, He He, Peng Qi, Dan Jurafsky","year":"2018","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Lebowitz, 1987] Michael Lebowitz. Planning stories. In CogSci, pages 234-242, 1987.","title":"Planning stories","authors":"Michael Lebowitz","year":"1987","venue":"CogSci","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li et al., 2013] Boyang Li, Stephen Lee-Urban, George Johnston, and Mark O. Riedl. Story generation with crowdsourced plot graphs. In AAAI, July 2013.","title":"Story generation with crowdsourced plot graphs","authors":"Boyang Li, Stephen Lee-Urban, George Johnston, Mark O Riedl","year":"2013","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2016] Jiwei Li, Will Monroe, Alan Ritter, Michel Galley, Jianfeng Gao, and Dan Jurafsky. Deep reinforce- ment learning for dialogue generation. In EMNLP, pages 1192-1202, 2016.","title":"Deep reinforcement learning for dialogue generation","authors":"Jiwei Li, Will Monroe, Alan Ritter, Michel Galley, Jianfeng Gao, Dan Jurafsky","year":"2016","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Martin et al., 2018] Lara J. Martin, Prithviraj Am- manabrolu, Xinyu Wang, William Hancock, Shruti Singh, Brent Harrison, and Mark O. Riedl. Event Rep- resentations for Automated Story Generation with Deep Neural Nets. In AAAI, pages 868-875, 2018.","title":"Event Representations for Automated Story Generation with Deep Neural Nets","authors":"Lara J Martin, Prithviraj Ammanabrolu, Xinyu Wang, William Hancock, Shruti Singh, Brent Harrison, Mark O Riedl","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Meehan, 1977] James R. Meehan. TALE-SPIN: An inter- active program that writes stories. In IJCAI, pages 91-98, 1977.","title":"TALE-SPIN: An interactive program that writes stories","authors":"James R Meehan","year":"1977","venue":"IJCAI","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Miller, 1995] George A. Miller. Wordnet: a lexical database for english. Communications of the ACM, 38(11):39-41, 1995.","title":"Wordnet: a lexical database for english","authors":"George A Miller","year":"1995","venue":"Communications of the ACM","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mostafazadeh et al., 2016] Nasrin Mostafazadeh, Lucy Vanderwende, Wen-tau Yih, Pushmeet Kohli, and James Allen. Story Cloze Evaluator: Vector Space Representa- tion Evaluation by Predicting What Happens Next. In 1st Workshop on Evaluating Vector Space Representations for NLP, pages 24-29, 2016.","title":"Story Cloze Evaluator: Vector Space Representation Evaluation by Predicting What Happens Next","authors":"Nasrin Mostafazadeh, Lucy Vanderwende, Wen-Tau Yih, Pushmeet Kohli, James Allen","year":"2016","venue":"1st Workshop on Evaluating Vector Space Representations for NLP","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Ng et al., 1999] Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In ICML, vol- ume 99, pages 278-287, 1999.","title":"Policy invariance under reward transformations: Theory and application to reward shaping","authors":"Daishi Andrew Y Ng, Stuart Harada, Russell","year":"1999","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Pérez y Pérez and Sharples, 2001] Rafael Pérez y Pérez and Mike Sharples. MEXICA: A computer model of a cogni- tive account of creative writing. Journal of Experimental and Theoretical Artificial Intelligence, 13:119-139, 2001.","title":"MEXICA: A computer model of a cognitive account of creative writing","authors":"Sharples Pérez Y Pérez, Rafael Pérez Y Pérez, Mike Sharples","year":"2001","venue":"Journal of Experimental and Theoretical Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Porteous and Cavazza, 2009] Julie Porteous and Marc Cavazza. Controlling narrative generation with planning trajectories: the role of constraints. In ICIDS, pages 234-245. Springer, 2009.","title":"Controlling narrative generation with planning trajectories: the role of constraints","authors":"Cavazza ; Porteous, Julie Porteous, Marc Cavazza","year":"2009","venue":"ICIDS","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Purdy et al., 2018] Christopher Purdy, Xinyu Wang, Larry He, and Mark Riedl. Predicting story quality with quanti- tative measures. In AIIDE, 2018.","title":"Predicting story quality with quantitative measures","authors":"Christopher Purdy, Xinyu Wang, Larry He, Mark Riedl","year":"2018","venue":"AIIDE","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Radford et al., 2019] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language Models are Unsupervised Multitask Learners. Technical report, OpenAI, 2019.","title":"Language Models are Unsupervised Multitask Learners","authors":"Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Riedl and Young, 2010] Mark O. Riedl and R. Michael Young. Narrative planning: Balancing plot and charac- ter. Journal of Artificial Intelligence Research, 39:217- 268, 2010.","title":"Narrative planning: Balancing plot and character","authors":"Mark O Riedl, R Michael Young","year":"2010","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Roemmele and Gordon, 2015] Melissa Roemmele and An- drew S. Gordon. Creative help: A story writing assistant. In ICIDS, pages 81-92, 2015.","title":"Creative help: A story writing assistant","authors":"Melissa Roemmele, Andrew S Gordon","year":"2015","venue":"ICIDS","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Schuler and Kipper-Schuler, 2005] Karin Kipper Schuler and Karen Kipper-Schuler. VerbNet: A Broad-Coverage, Comprehensive Verb Lexicon. PhD thesis, University of Pennsylvania, 2005.","title":"VerbNet: A Broad-Coverage, Comprehensive Verb Lexicon","authors":"Kipper-Schuler ; Schuler, Kipper Karin, Karen Schuler, Kipper-Schuler","year":"2005","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Sutskever et al., 2014] Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In NeurIPS, pages 3104-3112, 2014.","title":"Quoc V Le. Sequence to sequence learning with neural networks","authors":"Sutskever","year":"2014","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Sutton and Barto, 1998] Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. MIT press, 1998.","title":"Reinforcement Learning: An Introduction","authors":"Richard S Sutton, Andrew G Barto","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Swanson and Gordon, 2012] Reid Swanson and Andrew Gordon. Say Anything: Using textual case-based reason- ing to enable open-domain interactive storytelling. ACM TiiS, 2(3):16:1-16:35, 2012.","title":"Say Anything: Using textual case-based reasoning to enable open-domain interactive storytelling","authors":"Gordon ; ] Reid Swanson, Andrew Swanson, Gordon","year":"2012","venue":"ACM TiiS","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Ware and Young, 2011] Stephen Ware and R. Michael Young. CPOCL: A Narrative Planner Supporting Conflict. In AIIDE, pages 97-102, 2011.","title":"CPOCL: A Narrative Planner Supporting Conflict","authors":"Stephen Ware, R Michael Young","year":"2011","venue":"AIIDE","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Williams, 1992] Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforce- ment learning. Machine Learning, 8(3):229-256, May 1992.","title":"Simple statistical gradient-following algorithms for connectionist reinforcement learning","authors":"Ronald J Williams","year":"1992","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Yao et al., 2019] Lili Yao, Nanyun Peng, Weischedel Ralph, Kevin Knight, Dongyan Zhao, and Rui Yan. Plan-and- write: Towards better automatic storytelling. In AAAI, 2019.","title":"Plan-andwrite: Towards better automatic storytelling","authors":"Lili Yao, Nanyun Peng, Weischedel Ralph, Kevin Knight, Dongyan Zhao, Rui Yan","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}