{"type":"ScholarlyArticle","title":"Playing FPS Games With Environment-Aware Hierarchical Reinforcement Learning","authors":[{"name":"Shihong Song","slug":"shihong-song","given":"Shihong","family":"Song","affiliation":"Department of Computer Science and Technology , Tsinghua University Beijing National Research"},{"name":"Jiayi Weng","slug":"jiayi-weng","given":"Jiayi","family":"Weng","affiliation":"Department of Computer Science and Technology , Tsinghua University Beijing National Research"},{"name":"Hang Su","slug":"hang-su","given":"Hang","family":"Su","affiliation":"Institute for AI , Tsinghua University"},{"name":"Dong Yan","slug":"dong-yan","given":"Dong","family":"Yan","affiliation":"Institute for AI , Tsinghua University"},{"name":"Haosheng Zou","slug":"haosheng-zou","given":"Haosheng","family":"Zou","affiliation":"Center for Information Science and Technology Tsinghua Laboratory of Brain and Intelligence Lab Center for Intelligent Connected Vehicles and Transportation , Tsinghua University"},{"name":"Jun Zhu","slug":"jun-zhu","given":"Jun","family":"Zhu","affiliation":"Center for Information Science and Technology Tsinghua Laboratory of Brain and Intelligence Lab Center for Intelligent Connected Vehicles and Transportation , Tsinghua University"}],"abstract":"Learning rational behaviors in First-person-shooter (FPS) games is a challenging task for Reinforcement Learning (RL) with the primary difficulties of huge action space and insufficient exploration. To address this, we propose a hierarchical agent based on combined options with intrinsic rewards to drive exploration. Specifically, we present a hierarchical model that works in a manager-worker fashion over two levels of hierarchy. The high-level manager learns a policy over options, and the low-level workers, motivated by intrinsic reward, learn to execute the options. Performance is further improved with environmental signals appropriately harnessed. Extensive experiments demonstrate that our trained bot significantly outperforms the alternative RL-based models on FPS games requiring maze solving and combat skills, etc.\r\nNotably, we achieved first place in VDAIC 2018 Track(1).","doi":"10.24963/ijcai.2019/482","url":"https://ijcai.org/proceedings/2019/482","pdf":"https://ijcai.org/proceedings/2019/0482.pdf","markdown":"https://ijcai.org/proceedings/2019/482.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/482","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning M-Z","track":"Main track","number":482,"pages":{"first":3475,"last":3482},"keywords":["Machine Learning: Reinforcement Learning","Machine Learning Applications: Game Playing","Heuristic Search and Game Playing: General Game Playing and General Video Game Playing","Planning and Scheduling: Markov Decisions Processes"],"videos":[],"references":[{"position":1,"raw":"[Alvernaz and Togelius, 2017] Samuel Alvernaz and Julian Togelius. Autoencoder-augmented neuroevolution for vi- sual doom playing. In IEEE Conference on Computational Intelligence and Games, 2017.","title":"Samuel Alvernaz and Julian Togelius. Autoencoder-augmented neuroevolution for visual doom playing","authors":"Togelius Alvernaz","year":"2017","venue":"IEEE Conference on Computational Intelligence and Games","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bacon et al., 2017] Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In AAAI, 2017.","title":"The option-critic architecture","authors":"Bacon","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bhatti et al., 2016] Shehroze Bhatti, Alban Desmaison, On- drej Miksik, Nantas Nardelli, N Siddharth, and Philip HS Torr. Playing doom with slam-augmented deep reinforce- ment learning. arXiv preprint arXiv:1612.00380, 2016.","title":"Shehroze Bhatti, Alban Desmaison, Ondrej Miksik","authors":"Bhatti","year":"2016","venue":"Playing doom with slam-augmented deep reinforcement learning","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cook and Colton, 2011] Michael Cook and Simon Colton. Multi-faceted evolution of simple arcade games. In IEEE Conference on Computational Intelligence and Games, 2011.","title":"Multi-faceted evolution of simple arcade games","authors":"Michael Cook, Simon Colton","year":"2011","venue":"IEEE Conference on Computational Intelligence and Games","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dayan and Hinton, 1993] Peter Dayan and Geoffrey E Hin- ton. Feudal reinforcement learning. In NeurIPS, 1993. 5 https://www.crowdai.org/challenges/ visual-doom-ai-competition-2018-track-1 [","title":"Feudal reinforcement learning","authors":"Peter Dayan, Geoffrey E Hinton","year":"1993","venue":"NeurIPS","doi":null,"url":"https://www.crowdai.org/challenges/visual-doom-ai-competition-2018-track-1","internal_url":null},{"position":6,"raw":"[Dhariwal et al., 2017] Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plap- pert, Alec Radford, John Schulman, Szymon Sidor, Yuhuai Wu, and Peter Zhokhov. Openai baselines. https://github.com/openai/baselines, 2017.","title":"Openai baselines","authors":"Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, Yuhuai Wu, Peter Zhokhov","year":"2017","venue":null,"doi":null,"url":"https://github.com/openai/baselines","internal_url":null},{"position":7,"raw":"[Dietterich, 1998] Thomas G Dietterich. The maxq method for hierarchical reinforcement learning. In ICML, 1998.","title":"The maxq method for hierarchical reinforcement learning","authors":"G Thomas, Dietterich","year":"1998","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dosovitskiy and Koltun, 2017] Alexey Dosovitskiy and Vladlen Koltun. Learning to act by predicting the future. In ICLR, 2017.","title":"Alexey Dosovitskiy and Vladlen Koltun. Learning to act by predicting the future","authors":"Koltun Dosovitskiy","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Huang et al., 2019] Shiyu Huang, Hang Su, Jun Zhu, and Ting Chen. Combo-action: Training agent for fps game with auxiliary tasks. In AAAI, 2019.","title":"Combo-action: Training agent for fps game with auxiliary tasks","authors":"Shiyu Huang, Hang Su, Jun Zhu, Ting Chen","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kempka et al., 2016] Michał Kempka, Marek Wydmuch, Grzegorz Runc, Jakub Toczek, and Wojciech Jaśkowski. ViZDoom: A Doom-based AI research platform for visual reinforcement learning. In IEEE Conference on Computa- tional Intelligence and Games, 2016.","title":"ViZDoom: A Doom-based AI research platform for visual reinforcement learning","authors":"Michał Kempka, Marek Wydmuch, Grzegorz Runc, Jakub Toczek, Wojciech Jaśkowski","year":"2016","venue":"IEEE Conference on Computational Intelligence and Games","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kolishchak, 2018] Andrey Kolishchak. Doomnet. https:// github.com/akolishchak/doom-net-pytorch, 2018.","title":"Doomnet","authors":"Andrey Kolishchak","year":null,"venue":null,"doi":null,"url":"https://github.com/akolishchak/doom-net-pytorch,2018","internal_url":null},{"position":12,"raw":"[Kulkarni et al., 2016a] Tejas D. Kulkarni, Karthik R. Narasimhan, Ardavan Saeedi, and Joshua B. Tenenbaum. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. In NeurIPS, 2016.","title":"Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation","authors":"Kulkarni","year":"2016","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kulkarni et al., 2016b] Tejas D Kulkarni, Ardavan Saeedi, Simanta Gautam, and Samuel J Gershman. Deep successor reinforcement learning. arXiv preprint arXiv:1606.02396, 2016.","title":"Deep successor reinforcement learning","authors":"Kulkarni","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lample and Chaplot, 2017] Guillaume Lample and Deven- dra Singh Chaplot. Playing fps games with deep reinforce- ment learning. In AAAI, 2017.","title":"Guillaume Lample and Devendra Singh Chaplot. Playing fps games with deep reinforcement learning","authors":"Chaplot Lample","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li, 2017] Yuxi Li. Deep reinforcement learning: An overview. arXiv preprint arXiv:1701.07274, 2017.","title":"Deep reinforcement learning: An overview","authors":"Li ;, Yuxi Li","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Mishra et al., 2018] Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta-learner. In ICLR, 2018.","title":"A simple neural attentive meta-learner","authors":"Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, Pieter Abbeel","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Mnih et al., 2016] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asyn- chronous methods for deep reinforcement learning. In ICML, 2016.","title":"Asynchronous methods for deep reinforcement learning","authors":"Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, Koray Kavukcuoglu","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Pathak et al., 2017] Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In ICML, 2017.","title":"Curiosity-driven exploration by self-supervised prediction","authors":"Deepak Pathak, Pulkit Agrawal, Alexei A Efros, Trevor Darrell","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Redmon and Farhadi, 2018] Joseph Redmon and Ali Farhadi. Yolov3: An incremental improvement. arXiv preprint arXiv:1804.02767, 2018.","title":"Yolov3: An incremental improvement","authors":"Joseph Redmon, Ali Farhadi","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Schulman et al., 2017] John Schulman, Filip Wolski, Pra- fulla Dhariwal, Alec Radford, and Oleg Klimov. Prox- imal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.","title":"Proximal policy optimization algorithms","authors":"John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Sutton et al., 1999] Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A frame- work for temporal abstraction in reinforcement learning. Artificial intelligence, 1999.","title":"Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning","authors":"Doina Richard S Sutton, Satinder Precup, Singh","year":"1999","venue":"Artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Van Hoorn et al., 2009] Niels Van Hoorn, Julian Togelius, and Jurgen Schmidhuber. Hierarchical controller learning in a first-person shooter. In IEEE Conference on Compu- tational Intelligence and Games, 2009.","title":"Hierarchical controller learning in a first-person shooter","authors":"Van Hoorn","year":"2009","venue":"IEEE Conference on Computational Intelligence and Games","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Vezhnevets et al., 2017] Alexander Sasha Vezhnevets, Si- mon Osindero, Tom Schaul, Nicolas Heess, Max Jader- berg, David Silver, and Koray Kavukcuoglu. Feudal net- works for hierarchical reinforcement learning. In ICML, 2017.","title":"Feudal networks for hierarchical reinforcement learning","authors":"Sasha Alexander, Simon Vezhnevets, Tom Osindero, Nicolas Schaul, Max Heess, David Jaderberg, Koray Silver, Kavukcuoglu","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wu and Tian, 2016] Yuxin Wu and Yuandong Tian. Train- ing agent for first-person shooter game with actor-critic curriculum learning. In ICLR, 2016.","title":"Training agent for first-person shooter game with actor-critic curriculum learning","authors":"Tian ; ] Yuxin Wu, Yuandong Wu, Tian","year":"2016","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yan et al., 2019] Dong Yan, Shiyu Huang, Hang Su, and Jun Zhu. Learning to assign credit in reinforcement learn- ing by incorporating abstract relations. In AAAI Workshop on Reinforcement Learning in Games, 2019.","title":"Learning to assign credit in reinforcement learning by incorporating abstract relations","authors":"Dong Yan, Shiyu Huang, Hang Su, Jun Zhu","year":"2019","venue":"AAAI Workshop on Reinforcement Learning in Games","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}