{"type":"ScholarlyArticle","title":"Learning to Design Games: Strategic Environments in Reinforcement Learning","authors":[{"name":"Haifeng Zhang","slug":"haifeng-zhang","given":"Haifeng","family":"Zhang","affiliation":"Peking University"},{"name":"Jun Wang","slug":"jun-wang","given":"Jun","family":"Wang","affiliation":"University College London"},{"name":"Zhiming Zhou","slug":"zhiming-zhou","given":"Zhiming","family":"Zhou","affiliation":"Shanghai Jiao Tong University"},{"name":"Weinan Zhang","slug":"weinan-zhang","given":"Weinan","family":"Zhang","affiliation":"Shanghai Jiao Tong University"},{"name":"Yin Wen","slug":"yin-wen","given":"Yin","family":"Wen","affiliation":"University College London"},{"name":"Yong Yu","slug":"yong-yu","given":"Yong","family":"Yu","affiliation":"Shanghai Jiao Tong University"},{"name":"Wenxin Li","slug":"wenxin-li","given":"Wenxin","family":"Li","affiliation":"Peking University"}],"abstract":"In typical reinforcement learning (RL), the environment is assumed given and the goal of the learning is to identify an optimal policy for the agent taking actions through its interactions with the environment.  In this paper, we extend this setting by considering the environment is not given, but controllable and learnable through its interaction with the agent at the same time. This extension is motivated by environment design scenarios in the real-world, including game design, shopping space design and traffic signal design. Theoretically, we find a dual Markov decision process (MDP) w.r.t. the environment to that w.r.t. the agent, and derive a policy gradient solution to optimizing the parametrized environment. Furthermore, discontinuous environments are addressed by a proposed general generative framework. Our experiments on a Maze game design task show the effectiveness of the proposed algorithms in generating diverse and challenging Mazes against various agent settings.","doi":"10.24963/ijcai.2018/426","url":"https://ijcai.org/proceedings/2018/426","pdf":"https://ijcai.org/proceedings/2018/0426.pdf","markdown":"https://ijcai.org/proceedings/2018/426.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/426","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Machine Learning","track":"Main track","number":426,"pages":{"first":3068,"last":3074},"keywords":["Machine Learning: Machine Learning","Machine Learning: Reinforcement Learning"],"videos":[],"references":[{"position":1,"raw":"[Abdulhai et al., 2003] Baher Abdulhai, Rob Pringle, and Grigoris J Karakoulas. Reinforcement learning for true adaptive traffic signal control. Journal of Transportation Engineering, 2003.","title":"Reinforcement learning for true adaptive traffic signal control","authors":"Baher Abdulhai, Rob Pringle, Grigoris J Karakoulas","year":"2003","venue":"Journal of Transportation Engineering","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"Busoniu and De Schutter, ] Lucian Busoniu and Bart De Schutter. A comprehensive survey of multiagent reinforcement learning. [","title":"A comprehensive survey of multiagent reinforcement learning","authors":"De Busoniu, Lucian Schutter, Bart De Busoniu, Schutter","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cai et al., 2017] Han Cai, Kan Ren, Weinan Zhang, Klean- this Malialis, Jun Wang, Yong Yu, and Defeng Guo. Real- time bidding by reinforcement learning in display advertis- ing. In Proceedings of the Tenth ACM International Con- ference on Web Search and Data Mining, pages 661-670. ACM, 2017.","title":"Realtime bidding by reinforcement learning in display advertising","authors":"Han Cai, Kan Ren, Weinan Zhang, Kleanthis Malialis, Jun Wang, Yong Yu, Defeng Guo","year":"2017","venue":"Proceedings of the Tenth ACM International Conference on Web Search and Data Mining","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Ceylan and Bell, 2004] Halim Ceylan and Michael GH Bell. Traffic signal timing optimisation based on genetic algorithm approach, including drivers' routing. Trans- portation Research Part B: Methodological, 2004.","title":"Traffic signal timing optimisation based on genetic algorithm approach, including drivers' routing","authors":"Halim Ceylan, G H Michael, Bell","year":"2004","venue":"Transportation Research Part B: Methodological","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Garcıa and Fernández, 2015] Javier Garcıa and Fernando Fernández. A comprehensive survey on safe reinforcement learning. JMLR, 2015.","title":"Javier Garcıa and Fernando Fernández. A comprehensive survey on safe reinforcement learning","authors":"Fernández Garcıa","year":"2015","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gmytrasiewicz and Doshi, 2005] Piotr J Gmytrasiewicz and Prashant Doshi. A framework for sequential planning in multi-agent settings. JAIR, 2005.","title":"A framework for sequential planning in multi-agent settings","authors":"J Piotr, Prashant Gmytrasiewicz, Doshi","year":"2005","venue":"JAIR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Goodfellow et al., 2014] Ian Goodfellow, Jean Pouget- Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Gen- erative adversarial nets. In NIPS, pages 2672-2680, 2014.","title":"Generative adversarial nets","authors":"Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hom and Marks, 2007] Vincent Hom and Joe Marks. Auto- matic design of balanced board games. In Proceedings of the AAAI Conference on Artificial Intelligence and Inter- active Digital Entertainment (AIIDE), 2007.","title":"Automatic design of balanced board games","authors":"Vincent Hom, Joe Marks","year":"2007","venue":"Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment (AIIDE)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hu and Wellman, 2003] Junling Hu and Michael P Well- man. Nash q-learning for general-sum stochastic games. Journal of Machine learning research, 2003.","title":"Junling Hu and Michael P Wellman. Nash q-learning for general-sum stochastic games","authors":"Wellman Hu","year":"2003","venue":"Journal of Machine learning research","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kaelbling et al., 1996] Leslie Pack Kaelbling, Michael L Littman, and Andrew W Moore. Reinforcement learning: A survey. Journal of artificial intelligence research, 1996.","title":"Reinforcement learning: A survey","authors":"Leslie Pack, Kaelbling Michael L Littman, Andrew W Moore","year":"1996","venue":"Journal of artificial intelligence research","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[LeCun et al., 2015] Yann LeCun, Yoshua Bengio, and Ge- offrey Hinton. Deep learning. Nature, 2015.","title":"Deep learning","authors":"Yann Lecun, Yoshua Bengio, Geoffrey Hinton","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Littman, 1994] Michael L Littman. Markov games as a framework for multi-agent reinforcement learning. In Pro- ceedings of the eleventh international conference on ma- chine learning, 1994.","title":"Markov games as a framework for multi-agent reinforcement learning","authors":"Michael L Littman","year":"1994","venue":"Proceedings of the eleventh international conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Mnih et al., 2013] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.","title":"Playing atari with deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, Martin Riedmiller","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidje- land, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Mnih","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Moore and Atkeson, 1993] Andrew W Moore and Christo- pher G Atkeson. Prioritized sweeping: Reinforcement learning with less data and less time. Machine learning, 1993.","title":"Prioritized sweeping: Reinforcement learning with less data and less time","authors":"Andrew W, Moore, Christopher G Atkeson","year":"1993","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Morimoto and Doya, 2005] Jun Morimoto and Kenji Doya. Robust reinforcement learning. Neural computation, 2005.","title":"Robust reinforcement learning","authors":"Doya Morimoto, Jun Morimoto, Kenji Doya","year":"2005","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Nisan and Ronen, 2001] Noam Nisan and Amir Ronen. Al- gorithmic mechanism design. Games and Economic Be- havior, 35(1-2):166-196, 2001.","title":"Noam Nisan and Amir Ronen. Algorithmic mechanism design","authors":"Ronen Nisan","year":"2001","venue":"Games and Economic Behavior","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Penn, 2005] Alan Penn. The complexity of the elementary interface: shopping space. In Proceedings to the 5th In- ternational Space Syntax Symposium. Akkelies van Nes, 2005.","title":"The complexity of the elementary interface: shopping space","authors":"Alan Penn","year":"2005","venue":"Proceedings to the 5th International Space Syntax Symposium","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Plappert, 2016] Matthias Plappert. keras-rl. https:// github.com/keras-rl/keras-rl, 2016.","title":"keras-rl","authors":"Matthias Plappert","year":null,"venue":null,"doi":null,"url":"https://github.com/keras-rl/keras-rl,2016","internal_url":null},{"position":20,"raw":"[Silver et al., 2016] David Silver, Aja Huang, Chris J Maddi- son, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Veda Pan- neershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. Nature, 2016.","title":"Mastering the game of go with deep neural networks and tree search","authors":"David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den, Julian Driessche, Ioannis Schrittwieser, Veda Antonoglou, Marc Panneershelvam, Lanctot","year":"2016","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Sorg et al., 2010] Jonathan Sorg, Richard L Lewis, and Satinder P Singh. Reward design via online gradient as- cent. In NIPS, 2010.","title":"Reward design via online gradient ascent","authors":"Jonathan Sorg, Richard L Lewis, P Satinder, Singh","year":"2010","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sutton and Barto, 1998] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press Cambridge, 1998.","title":"Reinforcement learning: An introduction","authors":"S Richard, Andrew G Sutton, Barto","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Sutton et al., 1999] Richard S Sutton, David A McAllester, Satinder P Singh, Yishay Mansour, et al. Policy gradient methods for reinforcement learning with function approx- imation. In NIPS, 1999.","title":"Policy gradient methods for reinforcement learning with function approximation","authors":"David A Richard S Sutton, Mcallester, P Satinder, Yishay Singh, Mansour","year":"1999","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Sutton, 1990] Richard S Sutton. Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In ICML, 1990.","title":"Integrated architectures for learning, planning, and reacting based on approximating dynamic programming","authors":"Richard S Sutton","year":"1990","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Togelius and Schmidhuber, 2008] Julian Togelius and Jur- gen Schmidhuber. An experiment in automatic game de- sign. In Computational Intelligence and Games, 2008. CIG'08. IEEE Symposium On. IEEE, 2008.","title":"Julian Togelius and Jurgen Schmidhuber. An experiment in automatic game design","authors":"Schmidhuber Togelius","year":"2008","venue":"Computational Intelligence and Games","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Togelius et al., 2011] Julian Togelius, Georgios N Yan- nakakis, Kenneth O Stanley, and Cameron Browne. Search-based procedural content generation: A taxonomy and survey. IEEE Transactions on Computational Intelli- gence and AI in Games, 3(3):172-186, 2011.","title":"Search-based procedural content generation: A taxonomy and survey","authors":"Julian Togelius, N Georgios, Kenneth O Yannakakis, Cameron Stanley, Browne","year":"2011","venue":"IEEE Transactions on Computational Intelligence and AI in Games","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Vezhnevets et al., 2017] Alexander Sasha Vezhnevets, Si- mon Osindero, Tom Schaul, Nicolas Heess, Max Jader- berg, David Silver, and Koray Kavukcuoglu. Feudal networks for hierarchical reinforcement learning. arXiv preprint arXiv:1703.01161, 2017.","title":"Feudal networks for hierarchical reinforcement learning","authors":"Sasha Alexander, Simon Vezhnevets, Tom Osindero, Nicolas Schaul, Max Heess, David Jaderberg, Koray Silver, Kavukcuoglu","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Williams, 1992] Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforce- ment learning. Machine learning, 1992.","title":"Simple statistical gradient-following algorithms for connectionist reinforcement learning","authors":"Williams Ronald","year":"1992","venue":"Machine learning","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}