{"type":"ScholarlyArticle","title":"High-Fidelity Simulated Players for Interactive Narrative Planning","authors":[{"name":"Pengcheng Wang","slug":"pengcheng-wang","given":"Pengcheng","family":"Wang","affiliation":"Department of Computer Science , North Carolina State University , Raleigh , NC 27695 , USA"},{"name":"Jonathan Rowe","slug":"jonathan-rowe","given":"Jonathan","family":"Rowe","affiliation":"Department of Computer Science , North Carolina State University , Raleigh , NC 27695 , USA"},{"name":"Wookhee Min","slug":"wookhee-min","given":"Wookhee","family":"Min","affiliation":"Department of Computer Science , North Carolina State University , Raleigh , NC 27695 , USA"},{"name":"Bradford Mott","slug":"bradford-mott","given":"Bradford","family":"Mott","affiliation":"Department of Computer Science , North Carolina State University , Raleigh , NC 27695 , USA"},{"name":"James Lester","slug":"james-lester","given":"James","family":"Lester","affiliation":"Department of Computer Science , North Carolina State University , Raleigh , NC 27695 , USA"}],"abstract":"Interactive narrative planning offers significant potential for creating adaptive gameplay experiences. While data-driven techniques have been devised that utilize player interaction data to induce policies for interactive narrative planners, they require enormously large gameplay datasets. A promising approach to addressing this challenge is creating simulated players whose behaviors closely approximate those of human players. In this paper, we propose a novel approach to generating high-fidelity simulated players based on deep recurrent highway networks and deep convolutional networks. Empirical results demonstrate that the proposed models significantly outperform the prior state-of-the-art in generating high-fidelity simulated player models that accurately imitate human players’ narrative interactions. Using the high-fidelity simulated player models, we show the advantage of more exploratory reinforcement learning methods for deriving generalizable narrative adaptation policies.","doi":"10.24963/ijcai.2018/540","url":"https://ijcai.org/proceedings/2018/540","pdf":"https://ijcai.org/proceedings/2018/0540.pdf","markdown":"https://ijcai.org/proceedings/2018/540.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/540","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Multidisciplinary Topics and Applications","track":"Main track","number":540,"pages":{"first":3884,"last":3890},"keywords":["Multidisciplinary Topics and Applications: Interactive Entertainment","Multidisciplinary Topics and Applications: Computer Games"],"videos":[],"references":[{"position":1,"raw":"[Bengio, 2009] Y. Bengio. Learning Deep Architectures for AI. Foundations and Trends in Machine Learning, 2(1): 1-127, 2009.","title":"Learning Deep Architectures for AI","authors":"Y Bengio","year":"2009","venue":"Foundations and Trends in Machine Learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Gal and Ghahramani, 2016] Y. Gal, and Z. Ghahramani. A Theoretically Grounded Application of Dropout in Recurrent Neural Networks. In NIPS, pages 1019-1027, 2016.","title":"A Theoretically Grounded Application of Dropout in Recurrent Neural Networks","authors":"Y Gal, Z Ghahramani","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Gehring et al., 2017] J. Gehring, M. Auli, D. Grangier, D. Yarats, and Y. Dauphin. Convolutional Sequence to Sequence Learning. arXiv:1705.03122, 2017.","title":"Convolutional Sequence to Sequence Learning","authors":"J Gehring, M Auli, D Grangier, D Yarats, Y Dauphin","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Harrison and Riedl, 2016] B. Harrison, and M. Riedl. Learning from Stories: Using Crowdsourced Narratives to Train Virtual Agents. In AIIDE, pages 183-189, 2016.","title":"Learning from Stories: Using Crowdsourced Narratives to Train Virtual Agents","authors":"B Harrison, M Riedl","year":"2016","venue":"AIIDE","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[He et al., 2016] K. He, X. Zhang, S. Ren, and J. Sun. Deep Residual Learning for Image Recognition. In CVPR, pages 770-778, 2016.","title":"Deep Residual Learning for Image Recognition","authors":"K He, X Zhang, S Ren, J Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hochreiter and Schmidhuber, 1997] S. Hochreiter, J. Schmidhuber. Long Short-Term Memory. Neural Computation, 9(8): 1735-1780, 1997.","title":"Long Short-Term Memory","authors":"S Hochreiter, J Schmidhuber","year":"1997","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Kalchbrenner et al., 2016] N. Kalchbrenner, I. Danihelka, and A. Graves. Grid Long Short-term Memory. In ICLR, 2016.","title":"Grid Long Short-term Memory","authors":"N Kalchbrenner, I Danihelka, A Graves","year":"2016","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kingma and Ba, 2015] D. Kingma, and J. Ba. Adam: A Method for Stochastic Optimization. In ICLR, 2015.","title":"Adam: A Method for Stochastic Optimization","authors":"D Kingma, J Ba","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Lamstein and Mateas, 2004] A. Lamstein, and M. Mateas. Search-Based Drama Management. In AAAI Workshop on Challenges in Game AI, pages 103-107, 2004.","title":"Search-Based Drama Management","authors":"A Lamstein, M Mateas","year":"2004","venue":"AAAI Workshop on Challenges in Game AI","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[LeCun et al., 1989] Y. LeCun, B. Boser, J. Denker, D. Henderson, R. Howard, W. Hubbard, and L. Jackel. Backpropagation Applied to Handwritten Zip Code Recognition. Neural Computation, 1(4): 541-551. 1989.","title":"Backpropagation Applied to Handwritten Zip Code Recognition","authors":"Y Lecun, B Boser, J Denker, D Henderson, R Howard, W Hubbard, L Jackel","year":"1989","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lee et al., 2014] S. Lee, J. Rowe, B. Mott, and J. Lester. A Supervised Learning Framework for Modeling Director Agent Strategies in Educational Interactive Narrative. T- CIAIG, 6(2): 203-215, 2014.","title":"A Supervised Learning Framework for Modeling Director Agent Strategies in Educational Interactive Narrative","authors":"S Lee, J Rowe, B Mott, J Lester","year":"2014","venue":"T-CIAIG","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2013] B. Li, S. Lee-Urban, G. Johnston, and M. Riedl. Story Generation with Crowdsourced Plot Graphs. In AAAI, pages 598-604, 2013.","title":"Story Generation with Crowdsourced Plot Graphs","authors":"B Li, S Lee-Urban, G Johnston, M Riedl","year":"2013","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Marx and Cummings, 2007] J. Marx, and K. Cummings. Normalized Change. American Journal of Physics, 75(1): 87-91, 2007.","title":"Normalized Change","authors":"J Marx, K Cummings","year":"2007","venue":"American Journal of Physics","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Masoudnia and Ebrahimpour, 2014] S. Masoudnia, and R. Ebrahimpour. Mixture of Experts: A Literature Survey. Artificial Intelligence Review, 42(2): 275-293, 2014.","title":"Mixture of Experts: A Literature Survey","authors":"S Masoudnia, R Ebrahimpour","year":"2014","venue":"Artificial Intelligence Review","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mnih et al., 2016] V. Mnih, A. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu. Asynchronous Methods for Deep Reinforcement Learning. In ICML, pages 1928-1937, 2016.","title":"Asynchronous Methods for Deep Reinforcement Learning","authors":"V Mnih, A Badia, M Mirza, A Graves, T Lillicrap, T Harley, D Silver, K Kavukcuoglu","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Nelson et al., 2006] M. Nelson, D. Roberts, C. Isbell, and M. Mateas. Reinforcement Learning for Declarative Optimization-Based Drama Management. In AAMAS, pages 775-782, 2006.","title":"Reinforcement Learning for Declarative Optimization-Based Drama Management","authors":"M Nelson, D Roberts, C Isbell, M Mateas","year":"2006","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Porteous et al., 2015] J. Porteous, A. Lindsay, J. Read, M. Truran, M. Cavazza. Automated Extension of Narrative Planning Domains with Antonymic Operators. In AAMAS, pages 1574-1555, 2015.","title":"Automated Extension of Narrative Planning Domains with Antonymic Operators","authors":"J Porteous, A Lindsay, J Read, M Truran, M Cavazza","year":"2015","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Riedl and Bulitko, 2013] M. Riedl, and V. Bulitko. Interactive Narrative: An Intelligent Systems Approach. AI Magazine, 34(1): 67-77, 2012.","title":"Interactive Narrative: An Intelligent Systems Approach","authors":"M Riedl, V Bulitko","year":"2012","venue":"AI Magazine","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Roberts et al., 2006] D. Roberts, M. Nelson, C. Isbell, M. Mateas, and M. Littman. Targeting Specific Distributions of Trajectories in MDPs. In AAAI, pages 1213-1218, 2006.","title":"Targeting Specific Distributions of Trajectories in MDPs","authors":"D Roberts, M Nelson, C Isbell, M Mateas, M Littman","year":"2006","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Robertson and Young, 2015] J. Robertson, and R. Young. Automated Gameplay Generation from Declarative World Representations. In AIIDE, pages 72-78, 2015.","title":"Automated Gameplay Generation from Declarative World Representations","authors":"J Robertson, R Young","year":"2015","venue":"AIIDE","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Rowe et al., 2014] J. Rowe, B. Mott, and J. Lester. Optimizing Player Experience in Interactive Narrative Planning: A Modular Reinforcement Learning Approach. In AIIDE, pages 160-166, 2014.","title":"Optimizing Player Experience in Interactive Narrative Planning: A Modular Reinforcement Learning Approach","authors":"J Rowe, B Mott, J Lester","year":"2014","venue":"AIIDE","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Schulman et al., 2015] J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz. Trust Region Policy Optimization. In ICML, pages 1889-1897, 2015.","title":"Trust Region Policy Optimization","authors":"J Schulman, S Levine, P Abbeel, M Jordan, P Moritz","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Schulman et al., 2017] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal Policy Optimization Algorithms. arXiv:1707.06347, 2017.","title":"Proximal Policy Optimization Algorithms","authors":"J Schulman, F Wolski, P Dhariwal, A Radford, O Klimov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Srivastava et al., 2015] R. K. Srivastava, K. Greff, and J. Schmidhuber. Training Very Deep Networks. In NIPS, pages 2377-2385, 2015.","title":"Training Very Deep Networks","authors":"R K Srivastava, K Greff, J Schmidhuber","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Thue et al., 2007] D. Thue, V. Bulitko, M. Spetch, and E. Wasylishen. Interactive Storytelling: A Player Modelling Approach. In AIIDE, pages 43-48, 2007.","title":"Interactive Storytelling: A Player Modelling Approach","authors":"D Thue, V Bulitko, M Spetch, E Wasylishen","year":"2007","venue":"AIIDE","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wang et al., 2016] P. Wang, J. Rowe, B. Mott and J. Lester. Decomposing Drama Management in Educational Interactive Narrative: A Modular Reinforcement Learning Approach. In ICIDS, pages 270-282, 2016.","title":"Decomposing Drama Management in Educational Interactive Narrative: A Modular Reinforcement Learning Approach","authors":"P Wang, J Rowe, B Mott, J Lester","year":"2016","venue":"ICIDS","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wang et al., 2017a] P. Wang, J. Rowe, W. Min, B. Mott and J. Lester. Interactive Narrative Personalization with Deep Reinforcement Learning. In IJCAI, pages 3852-3858, 2017.","title":"Interactive Narrative Personalization with Deep Reinforcement Learning","authors":"Wang","year":"2017","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2017/538"},{"position":28,"raw":"[Wang et al., 2017b] P. Wang, J. Rowe, W. Min, B. Mott and J. Lester. Simulating Player Behavior for Data-Driven Interactive Narrative Personalization. In AIIDE, pages 255-261, 2017.","title":"Simulating Player Behavior for Data-Driven Interactive Narrative Personalization","authors":"Wang","year":"2017","venue":"AIIDE","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Yu and Riedl, 2014] H. Yu, and M. Riedl. Personalized Interactive Narratives via Sequential Recommendation of Plot Points. T-CIAIG, 6(2): 174-187, 2014.","title":"Personalized Interactive Narratives via Sequential Recommendation of Plot Points","authors":"H Yu, M Riedl","year":"2014","venue":"T-CIAIG","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zilly et al., 2017] J. Zilly, R. Srivastava, J. Koutník, and J. Schmidhuber. Recurrent Highway Networks. In ICML, pages 4189-4198, 2017.","title":"Recurrent Highway Networks","authors":"J Zilly, R Srivastava, J Koutník, J Schmidhuber","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zook et al., 2015] A. Zook, B. Harrison, and M. Riedl. Monte-Carlo Tree Search for Simulation-based Strategy Analysis. In FDG, 2015.","title":"Monte-Carlo Tree Search for Simulation-based Strategy Analysis","authors":"A Zook, B Harrison, M Riedl","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}