{"type":"ScholarlyArticle","title":"Generating Behavior-Diverse Game AIs with Evolutionary Multi-Objective Deep Reinforcement Learning","authors":[{"name":"Ruimin Shen","slug":"ruimin-shen","given":"Ruimin","family":"Shen","affiliation":""},{"name":"Yan Zheng","slug":"yan-zheng","given":"Yan","family":"Zheng","affiliation":"College of Intelligence and Computing , Tianjin University; Nanyang Technological University , Singapore"},{"name":"Jianye Hao","slug":"jianye-hao","given":"Jianye","family":"Hao","affiliation":"College of Intelligence and Computing , Tianjin University; Noah's Ark Lab , Huawei; Tianjin Key Lab of Machine Learning"},{"name":"Zhaopeng Meng","slug":"zhaopeng-meng","given":"Zhaopeng","family":"Meng","affiliation":"College of Intelligence and Computing , Tianjin University"},{"name":"Yingfeng Chen","slug":"yingfeng-chen","given":"Yingfeng","family":"Chen","affiliation":""},{"name":"Changjie Fan","slug":"changjie-fan","given":"Changjie","family":"Fan","affiliation":""},{"name":"Yang Liu","slug":"yang-liu","given":"Yang","family":"Liu","affiliation":"Nanyang Technological University , Singapore; Institute of Computing Innovation , Zhejiang University"}],"abstract":"Generating diverse behaviors for game artificial intelligence (Game AI) has been long recognized as a challenging task in the game industry. Designing a Game AI with a satisfying behavioral characteristic (style) heavily depends on the domain knowledge and is hard to achieve manually. Deep reinforcement learning sheds light on advancing the automatic Game AI design. However, most of them focus on creating a superhuman Game AI, ignoring the importance of behavioral diversity in games. To bridge the gap, we introduce a new framework, named EMOGI, which can automatically generate desirable styles with almost no domain knowledge. More importantly, EMOGI succeeds in creating a range of diverse styles, providing behavior-diverse Game AIs. Evaluations on the Atari and real commercial games indicate that, compared to existing algorithms, EMOGI performs better in generating diverse behaviors and significantly improves the efficiency of Game AI design.","doi":"10.24963/ijcai.2020/466","url":"https://ijcai.org/proceedings/2020/466","pdf":"https://ijcai.org/proceedings/2020/0466.pdf","markdown":"https://ijcai.org/proceedings/2020/466.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/466","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning Applications","track":"Main track","number":466,"pages":{"first":3371,"last":3377},"keywords":["Machine Learning Applications: Game Playing","Heuristic Search and Game Playing: Game Playing and Machine Learning","Machine Learning Applications: Applications of Reinforcement Learning"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24772/file_24772.m3u8","length_sec":277,"provider":"hls"}],"references":[{"position":1,"raw":"[Agapitos et al., 2008] Alexandros Agapitos, Julian To- gelius, Simon M. Lucas, Jürgen Schmidhuber, and An- dreas Konstantinidis. Generating diverse opponents with multiobjective evolution. In IEEE Symposium On Compu- tational Intelligence and Games. IEEE, 2008.","title":"Generating diverse opponents with multiobjective evolution","authors":"Alexandros Agapitos, Julian Togelius, Simon M Lucas, Jürgen Schmidhuber, Andreas Konstantinidis","year":"2008","venue":"IEEE Symposium On Computational Intelligence and Games","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Alt, 2004] Greg Alt. The suffering: A game ai case study. In Challenges in Game AI workshop, Nineteenth na- tional conference on Artificial Intelligence, pages 134- 138, 2004.","title":"The suffering: A game ai case study","authors":"Greg Alt","year":"2004","venue":"Nineteenth national conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Deb and Agrawal, 1994] Kalyanmoy Deb and Ram Bhu- san Agrawal. Simulated binary crossover for continuous search space. Complex Systems, 9(2):115-148, 1994.","title":"Kalyanmoy Deb and Ram Bhusan Agrawal. Simulated binary crossover for continuous search space","authors":"Deb, Agrawal","year":"1994","venue":"Complex Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Deb et al., 2000] Kalyanmoy Deb, Samir Agrawal, Amrit Pratap, and T Meyarivan. A Fast Elitist Non-dominated Sorting Genetic Algorithm for Multi-objective Optimisa- tion -NSGA-II. PPSN, 2000.","title":"A Fast Elitist Non-dominated Sorting Genetic Algorithm for Multi-objective Optimisation -NSGA-II","authors":"Kalyanmoy Deb, Samir Agrawal, Amrit Pratap, Meyarivan","year":"2000","venue":"PPSN","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Drachen et al., 2009] Anders Drachen, Alessandro Canossa, and Georgios N Yannakakis. Player mod- eling using self-organization in tomb raider: Underworld. In 2009 IEEE symposium on computational intelligence and games, pages 1-8. IEEE, 2009.","title":"Player modeling using self-organization in tomb raider: Underworld","authors":"Anders Drachen, Alessandro Canossa, Georgios N Yannakakis","year":"2009","venue":"IEEE symposium on computational intelligence and games","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Holmgård et al., 2014] Christoffer Holmgård, Antonios Li- apis, Julian Togelius, and Georgios N Yannakakis. Per- sonas versus clones for player decision modeling. In Inter- national Conference on Entertainment Computing, pages 159-166. Springer, 2014.","title":"Personas versus clones for player decision modeling","authors":"Christoffer Holmgård, Antonios Liapis, Julian Togelius, Georgios N Yannakakis","year":"2014","venue":"International Conference on Entertainment Computing","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Isla, 2008] Damian Isla. Halo 3-building a better battle. In Game Developers Conference, GDC, 2008.","title":"Damian Isla. Halo 3-building a better battle","authors":"Isla","year":"2008","venue":"Game Developers Conference","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Jaderberg et al., 2017] Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojciech M Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, Iain Dunning, Karen Simonyan, et al. Population based training of neural networks. arXiv preprint arXiv:1711.09846, 2017.","title":"Population based training of neural networks","authors":"Jaderberg","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Lehman and Stanley, 2011] Joel Lehman and Kenneth O. Stanley. Evolving a diversity of creatures through novelty search and local competition. In Genetic and Evolutionary Computation Conference, pages 211-218, July 2011.","title":"Evolving a diversity of creatures through novelty search and local competition","authors":"Stanley ; Lehman, Joel Lehman, Kenneth O Stanley","year":"2011","venue":"Genetic and Evolutionary Computation Conference","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li et al., 2019] Ang Li, Ola Spyra, Sagi Perel, Valentin Dal- ibard, Max Jaderberg, Chenjie Gu, David Budden, Tim Harley, and Pramod Gupta. A Generalized Framework for Population Based Training. CoRR, 2019.","title":"A Generalized Framework for Population Based Training","authors":"Ang Li, Ola Spyra, Sagi Perel, Valentin Dalibard, Max Jaderberg, Chenjie Gu, David Budden, Tim Harley, Pramod Gupta","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Millington, 2019] Ian Millington. AI for Games, Third Edi- tion. CRC Press, 2019.","title":"AI for Games, Third Edition","authors":"Ian Millington","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fid- jeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Has- sabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, Demis Hassabis","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Mnih et al., 2016] Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P Lilli- crap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous Methods for Deep Reinforcement Learn- ing. ICML, 2016.","title":"Asynchronous Methods for Deep Reinforcement Learning","authors":"Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P Lillicrap, Tim Harley, David Silver, Koray Kavukcuoglu","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mouret and Clune, 2015] Jean-Baptiste Mouret and Jeff Clune. Illuminating search spaces by mapping elites. arXiv:1504.04909, 2015.","title":"Illuminating search spaces by mapping elites","authors":"Clune ; Jean-Baptiste Mouret, Jeff Mouret, Clune","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Newzoo, 2019] Newzoo. Global games market report. https://newzoo.com/solutions/standard/market-forecasts/ global-games-market-report, 2019.","title":"Newzoo. Global games market report","authors":"Newzoo","year":"2019","venue":null,"doi":null,"url":"https://newzoo.com/solutions/standard/market-forecasts/global-games-market-report,2019","internal_url":null},{"position":16,"raw":"[Oh et al., 2019] Inseok Oh, Seungeun Rho, Sangbin Moon, Seongho Son, Hyoil Lee, and Jinyun Chung. Creating Pro- Level AI for a Real-Time Fighting Game Using Deep Re- inforcement Learning. arXiv.org, April 2019.","title":"Creating Pro-Level AI for a Real-Time Fighting Game Using Deep Reinforcement Learning","authors":"Oh","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Ortega et al., 2013] Juan Ortega, Noor Shaker, Julian To- gelius, and Georgios N Yannakakis. Imitating human play- ing styles in super mario bros. Entertainment Computing, 4(2):93-104, 2013.","title":"Juan Ortega, Noor Shaker, Julian Togelius, and Georgios N Yannakakis. Imitating human playing styles in super mario bros","authors":"Ortega","year":"2013","venue":"Entertainment Computing","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Rockstar Games, 2018] Rockstar Games. Red dead redemption 2. https://www.rockstargames.com/ reddeadredemption2/, 2018.","title":"Rockstar Games. Red dead redemption 2","authors":null,"year":"2018","venue":null,"doi":null,"url":"https://www.rockstargames.com/reddeadredemption2/,2018","internal_url":null},{"position":19,"raw":"[Suay et al., 2016] Halit Bener Suay, Tim Brys, Matthew E Taylor, and Sonia Chernova. Learning from Demonstra- tion for Shaping through Inverse Reinforcement Learning. AAMAS, 2016.","title":"Learning from Demonstration for Shaping through Inverse Reinforcement Learning","authors":"Halit Bener Suay, Tim Brys, Matthew E Taylor, Sonia Chernova","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Sun et al., 2020] Jianwen Sun, Tianwei Zhang, Xiaofei Xie, Lei Ma, Yan Zheng, Kangjie Chen, and Yang Liu. Stealthy and efficient adversarial attacks against deep reinforce- ment learning. In Proceedings of the AAAI Conference on Artificial Intelligence, 2020.","title":"Stealthy and efficient adversarial attacks against deep reinforcement learning","authors":"Jianwen Sun, Tianwei Zhang, Xiaofei Xie, Lei Ma, Yan Zheng, Kangjie Chen, Yang Liu","year":"2020","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Sutton and Barto, 2018] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.","title":"Reinforcement learning: An introduction","authors":"S Richard, Andrew G Sutton, Barto","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Szita et al., 2009] Istvan Szita, Marc Ponsen, and Pieter Spronck. Effective and diverse adaptive game AI. IEEE Transactions on Computational Intelligence and AI in Games, 1(1):16-27, 2009.","title":"Effective and diverse adaptive game AI","authors":"Szita","year":"2009","venue":"IEEE Transactions on Computational Intelligence and AI in Games","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zheng et al., 2018] Yan Zheng, Zhaopeng Meng, Jianye Hao, Zongzhang Zhang, Tianpei Yang, and Changjie Fan. A deep bayesian policy reuse approach against non- stationary agents. In Advances in Neural Information Pro- cessing Systems, pages 954-964, 2018.","title":"A deep bayesian policy reuse approach against nonstationary agents","authors":"Yan Zheng, Zhaopeng Meng, Jianye Hao, Zongzhang Zhang, Tianpei Yang, Changjie Fan","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zheng et al., 2019] Yan Zheng, Xiaofei Xie, Ting Su, Lei Ma, Jianye Hao, Zhaopeng Meng, Yang Liu, Ruimin Shen, Yingfeng Chen, and Changjie Fan. Wuji: Automatic on- line combat game testing using evolutionary deep rein- forcement learning. In International Conference on Auto- mated Software Engineering (ASE), pages 772-784, 2019.","title":"Wuji: Automatic online combat game testing using evolutionary deep reinforcement learning","authors":"Yan Zheng, Xiaofei Xie, Ting Su, Lei Ma, Jianye Hao, Zhaopeng Meng, Yang Liu, Ruimin Shen, Yingfeng Chen, Changjie Fan","year":"2019","venue":"International Conference on Automated Software Engineering (ASE)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}