{"type":"ScholarlyArticle","title":"Interactive Narrative Personalization with Deep Reinforcement Learning","authors":[{"name":"Pengcheng Wang","slug":"pengcheng-wang","given":"Pengcheng","family":"Wang","affiliation":"Department of Computer Science , North Carolina State University , Raleigh , NC 27695 , USA"},{"name":"Jonathan Rowe","slug":"jonathan-rowe","given":"Jonathan","family":"Rowe","affiliation":"Department of Computer Science , North Carolina State University , Raleigh , NC 27695 , USA"},{"name":"Wookhee Min","slug":"wookhee-min","given":"Wookhee","family":"Min","affiliation":"Department of Computer Science , North Carolina State University , Raleigh , NC 27695 , USA"},{"name":"Bradford Mott","slug":"bradford-mott","given":"Bradford","family":"Mott","affiliation":"Department of Computer Science , North Carolina State University , Raleigh , NC 27695 , USA"},{"name":"James Lester","slug":"james-lester","given":"James","family":"Lester","affiliation":"Department of Computer Science , North Carolina State University , Raleigh , NC 27695 , USA"}],"abstract":"Data-driven techniques for interactive narrative generation are the subject of growing interest. Reinforcement learning (RL) offers significant potential for devising data-driven interactive narrative generators that tailor players’ story experiences by inducing policies from player interaction logs. A key open question in RL-based interactive narrative generation is how to model complex player interaction patterns to learn effective policies. In this paper we present a deep RL-based interactive narrative generation framework that leverages synthetic data produced by a bipartite simulated player model. Specifically, the framework involves training a set of Q-networks to control adaptable narrative event sequences with long short-term memory network-based simulated players. We investigate the deep RL framework’s performance with an educational interactive narrative, Crystal Island. Results suggest that the deep RL-based narrative generation framework yields effective personalized interactive narratives.","doi":"10.24963/ijcai.2017/538","url":"https://ijcai.org/proceedings/2017/538","pdf":"https://ijcai.org/proceedings/2017/0538.pdf","markdown":"https://ijcai.org/proceedings/2017/538.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/538","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Multidisciplinary Topics and Applications","track":"Main track","number":538,"pages":{"first":3852,"last":3858},"keywords":["Multidisciplinary Topics and Applications: Computer Games","Multidisciplinary Topics and Applications: Interactive Entertainment"],"videos":[],"references":[{"position":1,"raw":"[Harmon and Baird, 1995] M. Harmon, L. Baird, and A. Klopf. Advantage updating applied to a differential game. In Proceeding of Advances in Neural Information Processing Systems, pages 353-360, 1995.","title":"Advantage updating applied to a differential game","authors":"M Harmon, L Baird, A Klopf","year":"1995","venue":"Proceeding of Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Hausknecht and Stone, 2015] M. Hausknecht, and P. Stone. Deep Recurrent Q-Learning for Partially Observable MDPs. In AAAI Fall Symposium Series, 2015.","title":"Deep Recurrent Q-Learning for Partially Observable MDPs","authors":"M Hausknecht, P Stone","year":"2015","venue":"AAAI Fall Symposium Series","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Henderson et al., 2005] J. Henderson, O. Lemon, and K. Georgila. Hybrid reinforcement/supervised learning for dialogue policies from communicator data. In IJCAI Workshop on Knowledge And Reasoning in Practical Dialogue Systems, pages 68-75, 2005.","title":"Hybrid reinforcement/supervised learning for dialogue policies from communicator data","authors":"J Henderson, O Lemon, K Georgila","year":"2005","venue":"IJCAI Workshop on Knowledge And Reasoning in Practical Dialogue Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Hochreiter and Schmidhuber, 1997] S. Hochreiter, J. Schmidhuber. Long Short-Term Memory. Neural Computation, 9(8): 1735-1780, 1997.","title":"Long Short-Term Memory","authors":"S Hochreiter, J Schmidhuber","year":"1997","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Kingma and Ba, 2015] D. Kingma, and J. Ba. Adam: A Method for Stochastic Optimization. In International Conference for Learning Representations, 2015.","title":"A Method for Stochastic Optimization","authors":"D Kingma, J Ba, Adam","year":"2015","venue":"International Conference for Learning Representations","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Lamstein and Mateas, 2004] A. Lamstein, and M. Mateas. Search-based drama management. In 2004 AAAI Workshop on Challenges in Game Artificial Intelligence, pages 103-107, 2004.","title":"Search-based drama management","authors":"A Lamstein, M Mateas","year":"2004","venue":"AAAI Workshop on Challenges in Game Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Lee et al., 2014] S. Lee, J. Rowe, B. Mott, and J. Lester. A supervised learning framework for modeling director agent strategies in educational interactive narrative. IEEE Transactions on Computational Intelligence and AI in Games, 6: 203-215, 2014.","title":"A supervised learning framework for modeling director agent strategies in educational interactive narrative","authors":"S Lee, J Rowe, B Mott, J Lester","year":"2014","venue":"IEEE Transactions on Computational Intelligence and AI in Games","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Li et al., 2013] B. Li, S. Lee-Urban, G. Johnston, and M. Riedl. Story Generation with Crowdsourced Plot Graphs. In Proceedings of the 27th AAAI Conference on Artificial Intelligence, pages 598-604, 2013.","title":"Story Generation with Crowdsourced Plot Graphs","authors":"B Li, S Lee-Urban, G Johnston, M Riedl","year":"2013","venue":"Proceedings of the 27th AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Lin, 1993] L.J. Lin. Reinforcement learning for robots using neural networks. Technical Report, DTIC Document, 1993.","title":"Reinforcement learning for robots using neural networks","authors":"L J Lin","year":"1993","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Mnih et al., 2015] V.Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis. Human-level control through deep reinforcement learning. Nature, 518 (7540): 529-533, 2015.","title":"Human-level control through deep reinforcement learning","authors":"V Mnih, K Kavukcuoglu, D Silver, A A Rusu, J Veness, M G Bellemare, A Graves, M Riedmiller, A K Fidjeland, G Ostrovski, S Petersen, C Beattie, A Sadik, I Antonoglou, H King, D Kumaran, D Wierstra, S Legg, D Hassabis","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Mnih et al., 2016] V. Mnih, A. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Proceedings of 33 rd International Conference on Machine Learning. Pages 1928-1937, 2016.","title":"Asynchronous methods for deep reinforcement learning","authors":"V Mnih, A Badia, M Mirza, A Graves, T Lillicrap, T Harley, D Silver, K Kavukcuoglu","year":"2016","venue":"Proceedings of 33 rd International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Nelson et al., 2006] M. Nelson, D. Roberts, C. Isbell, and M. Mateas. Reinforcement learning for declarative optimization-based drama management. In Proceedings of the 5th International Conference on Autonomous Agent and Multi-Agent Systems, pages 775-782, 2006.","title":"Reinforcement learning for declarative optimization-based drama management","authors":"M Nelson, D Roberts, C Isbell, M Mateas","year":"2006","venue":"Proceedings of the 5th International Conference on Autonomous Agent and Multi-Agent Systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Porteous et al., 2015] J. Porteous, A. Lindsay, J. Read, M. Truran, M. Cavazza. Automated Extension of Narrative Planning Domains with Antonymic Operators. In Proceedings of the 14th International Conference on Autonomous Agents and Multiagent Systems, pages 1574- 1555, 2015.","title":"Automated Extension of Narrative Planning Domains with Antonymic Operators","authors":"Porteous","year":"2015","venue":"Proceedings of the 14th International Conference on Autonomous Agents and Multiagent Systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Riedl and Bulitko, 2013] M. Riedl, and V. Bulitko. Interactive Narrative: An Intelligent Systems Approach. AI Magazine, 34(1): 67-77, 2012.","title":"Interactive Narrative: An Intelligent Systems Approach","authors":"M Riedl, V Bulitko","year":"2012","venue":"AI Magazine","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Roberts et al., 2006] D. Roberts, M. Nelson, C. Isbell, M. Mateas, and M. Littman. Targeting specific distributions of trajectories in MDPs. In Proceedings of the 21st AAAI Conference on Artificial Intelligence, pages 1213-1218, 2006.","title":"Targeting specific distributions of trajectories in MDPs","authors":"D Roberts, M Nelson, C Isbell, M Mateas, M Littman","year":"2006","venue":"Proceedings of the 21st AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Robertson and Young, 2015] J. Robertson, and R. Young. Automated Gameplay Generation from Declarative World Representations. In Proceedings of the 11th AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, pages 72-78, 2015.","title":"Automated Gameplay Generation from Declarative World Representations","authors":"J Robertson, R Young","year":"2015","venue":"Proceedings of the 11th AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Rowe et al., 2014] J. Rowe, B. Mott, and J. Lester. Optimizing Player Experience in Interactive Narrative Planning: A Modular Reinforcement Learning Approach. In Proceedings of the 10th AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, pages 160-166, 2014.","title":"Optimizing Player Experience in Interactive Narrative Planning: A Modular Reinforcement Learning Approach","authors":"J Rowe, B Mott, J Lester","year":"2014","venue":"Proceedings of the 10th AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Schatzmann et al., 2006] J. Schatzmann, K. Weilhammer, M. Stuttle, and S. Young. A survey of statistical user simulation techniques for reinforcement-learning of dialogue management strategies. The Knowledge Engineering Review, 21(2): 97-126, 2006.","title":"A survey of statistical user simulation techniques for reinforcement-learning of dialogue management strategies","authors":"J Schatzmann, K Weilhammer, M Stuttle, S Young","year":"2006","venue":"The Knowledge Engineering Review","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Tsitsiklis and Roy, 1997] Tsitsiklis, J. and Roy, B. V. An Analysis of Temporal-Difference Learning with Function Approximation. IEEE Trans. Automat. Contr. 42, 674- 690, 1997.","title":"An Analysis of Temporal-Difference Learning with Function Approximation","authors":"J Tsitsiklis, B V Roy","year":"1997","venue":"IEEE Trans. Automat. Contr","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wang et al., 2016a] P. Wang, J. Rowe, B. Mott and J. Lester. Decomposing Drama Management in Educational Interactive Narrative: A Modular Reinforcement Learning Approach. In Proceedings of the 9th International Conference on Interactive Digital Storytelling, pages 270-282, 2016.","title":"Decomposing Drama Management in Educational Interactive Narrative: A Modular Reinforcement Learning Approach","authors":"Wang","year":"2016","venue":"Proceedings of the 9th International Conference on Interactive Digital Storytelling","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang et al., 2016b] Z. Wang, T. Schaul, M. Hessel, H. Hasselt, M. Lanctot, and N. Freitas. Dueling Network Architectures for Deep Reinforcement Learning. In Proceedings of The 33rd International Conference on Machine Learning, pages 1995-2003, 2016.","title":"Dueling Network Architectures for Deep Reinforcement Learning","authors":"Wang","year":"2016","venue":"Proceedings of The 33rd International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Young et al., 2013] S. Young, M. Gašić, B. Thomson, and J. Williams. Pomdp-based Statistical Spoken Dialog Systems: A Review. Proceedings of IEEE, 101(5): 1160- 1179, 2013.","title":"Pomdp-based Statistical Spoken Dialog Systems: A Review","authors":"S Young, M Gašić, B Thomson, J Williams","year":"2013","venue":"Proceedings of IEEE","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yu and Riedl, 2014] H. Yu, and M. Riedl. Personalized Interactive Narratives via Sequential Recommendation of Plot Points. IEEE Transactions on Computational Intelligence and AI in Games, 6(2): 174-187, 2014.","title":"Personalized Interactive Narratives via Sequential Recommendation of Plot Points","authors":"H Yu, M Riedl","year":"2014","venue":"IEEE Transactions on Computational Intelligence and AI in Games","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}