{"type":"ScholarlyArticle","title":"Obstacle Tower: A Generalization Challenge in Vision, Control, and Planning","authors":[{"name":"Arthur Juliani","slug":"arthur-juliani","given":"Arthur","family":"Juliani","affiliation":"Unity Technologies"},{"name":"Ahmed Khalifa","slug":"ahmed-khalifa","given":"Ahmed","family":"Khalifa","affiliation":"New York University {arthurj ,"},{"name":"Vincent-Pierre Berges","slug":"vincent-pierre-berges","given":"Vincent-Pierre","family":"Berges","affiliation":"Unity Technologies"},{"name":"Jonathan Harper","slug":"jonathan-harper","given":"Jonathan","family":"Harper","affiliation":"Unity Technologies"},{"name":"Ervin Teng","slug":"ervin-teng","given":"Ervin","family":"Teng","affiliation":"Unity Technologies"},{"name":"Hunter Henry","slug":"hunter-henry","given":"Hunter","family":"Henry","affiliation":"Unity Technologies"},{"name":"Adam Crespi","slug":"adam-crespi","given":"Adam","family":"Crespi","affiliation":"Unity Technologies"},{"name":"Julian Togelius","slug":"julian-togelius","given":"Julian","family":"Togelius","affiliation":"New York University {arthurj ,"},{"name":"Danny Lange","slug":"danny-lange","given":"Danny","family":"Lange","affiliation":"Unity Technologies"}],"abstract":"The rapid pace of recent research in AI has been driven in part by the presence of fast and challenging simulation environments. These environments often take the form of games; with tasks ranging from simple board games, to competitive video games. We propose a new benchmark - Obstacle Tower: a high fidelity, 3D, 3rd person, procedurally generated environment. An agent in Obstacle Tower must learn to solve both low-level control and high-level planning problems in tandem while learning from pixels and a sparse reward signal.\r\nUnlike other benchmarks such as the Arcade Learning Environment, evaluation of agent performance in Obstacle Tower is based on an agent's ability to perform well on unseen instances of the environment. In this paper we outline the environment and provide a set of baseline results produced by current state-of-the-art Deep RL methods as well as human players. These algorithms fail to produce agents capable of performing near human level.","doi":"10.24963/ijcai.2019/373","url":"https://ijcai.org/proceedings/2019/373","pdf":"https://ijcai.org/proceedings/2019/0373.pdf","markdown":"https://ijcai.org/proceedings/2019/373.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/373","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning A-L","track":"Main track","number":373,"pages":{"first":2684,"last":2691},"keywords":["Machine Learning: Reinforcement Learning","Agent-based and Multi-agent Systems: Agent-Based Simulation and Emergence","Heuristic Search and Game Playing: Game Playing and Machine Learning","Heuristic Search and Game Playing: General Game Playing and General Video Game Playing"],"videos":[],"references":[{"position":1,"raw":"[Bellemare et al., 2013] Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, 2013.","title":"The arcade learning environment: An evaluation platform for general agents","authors":"Yavar Marc G Bellemare, Joel Naddaf, Michael Veness, Bowling","year":"2013","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Brockman et al., 2016] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv:1606.01540, 2016.","title":"Openai gym","authors":"Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, Wojciech Zaremba","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Castro et al., 2018] Pablo Samuel Castro, Subhodeep Moitra, Carles Gelada, Saurabh Kumar, and Marc G Bellemare. Dopamine: A research framework for deep reinforcement learning. arXiv:1812.06110, 2018.","title":"Dopamine: A research framework for deep reinforcement learning","authors":"Castro","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cobbe et al., 2018] Karl Cobbe, Oleg Klimov, Chris Hesse, Taehoon Kim, and John Schulman. Quantifying gener- alization in reinforcement learning. arXiv:1812.02341, 2018.","title":"Quantifying generalization in reinforcement learning","authors":"Karl Cobbe, Oleg Klimov, Chris Hesse, Taehoon Kim, John Schulman","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dhariwal et al., 2017] Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plap- pert, Alec Radford, John Schulman, Szymon Sidor, Yuhuai Wu, and Peter Zhokhov. Openai baselines. https://github.com/openai/baselines, 2017.","title":"Openai baselines","authors":"Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, Yuhuai Wu, Peter Zhokhov","year":"2017","venue":null,"doi":null,"url":"https://github.com/openai/baselines","internal_url":null},{"position":6,"raw":"[Dormans, 2010] Joris Dormans. Adventures in level de- sign: Generating missions and spaces for action adventure games. In Workshop on Procedural Content Generation in Games. ACM, 2010.","title":"Adventures in level design: Generating missions and spaces for action adventure games","authors":"Joris Dormans, Dormans","year":"2010","venue":"Workshop on Procedural Content Generation in Games","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Ecoffet et al., 2018] Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth Stanley, and Jeff Clune. Montezuma's revenge solved by go-explore, a new algorithm for hard- exploration problems (sets records on pitfall too), 2018.","title":"Montezuma's revenge solved by go-explore, a new algorithm for hardexploration problems","authors":"Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth Stanley, Jeff Clune","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Henderson et al., 2017] Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. arXiv:1709.06560, 2017.","title":"Deep reinforcement learning that matters","authors":"Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, David Meger","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hessel et al., 2018] Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Sil- ver. Rainbow: Combining improvements in deep rein- forcement learning. In AAAI Conference on Artificial In- telligence, 2018.","title":"Rainbow: Combining improvements in deep reinforcement learning","authors":"Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, David Silver","year":"2018","venue":"AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Huang et al., 2017] Sandy Huang, Nicolas Papernot, Ian Goodfellow, Yan Duan, and Pieter Abbeel. Adversarial at- tacks on neural network policies. arXiv:1702.02284, 2017.","title":"Adversarial attacks on neural network policies","authors":"Sandy Huang, Nicolas Papernot, Ian Goodfellow, Yan Duan, Pieter Abbeel","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Johnson et al., 2016] Matthew Johnson, Katja Hofmann, Tim Hutton, and David Bignell. The malmo platform for artificial intelligence experimentation. In IJCAI, pages 4246-4247, 2016.","title":"The malmo platform for artificial intelligence experimentation","authors":"Matthew Johnson, Katja Hofmann, Tim Hutton, David Bignell","year":"2016","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2016/643"},{"position":12,"raw":"[Juliani et al., 2018] Arthur Juliani, Vincent-Pierre Berges, Esh Vckay, Yuan Gao, Hunter Henry, Marwan Mattar, and Danny Lange. Unity: A general platform for intelligent agents. arXiv:1809.02627, 2018.","title":"Unity: A general platform for intelligent agents","authors":"Arthur Juliani, Vincent-Pierre Berges, Esh Vckay, Yuan Gao, Hunter Henry, Marwan Mattar, Danny Lange","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Justesen et al., 2018] Niels Justesen, Ruben Rodriguez Tor- rado, Philip Bontrager, Ahmed Khalifa, Julian Togelius, and Sebastian Risi. Illuminating generalization in deep re- inforcement learning through procedural level generation. In NeurIPS Workshop on Deep RL, 2018.","title":"Illuminating generalization in deep reinforcement learning through procedural level generation","authors":"Niels Justesen, Ruben Rodriguez Torrado, Philip Bontrager, Ahmed Khalifa, Julian Togelius, Sebastian Risi","year":"2018","venue":"NeurIPS Workshop on Deep RL","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Karakovskiy and Togelius, 2012] Sergey Karakovskiy and Julian Togelius. The mario ai benchmark and competi- tions. IEEE Transactions on Computational Intelligence and AI in Games, 4(1):55-67, 2012.","title":"The mario ai benchmark and competitions","authors":"Sergey Karakovskiy, Julian Togelius","year":"2012","venue":"IEEE Transactions on Computational Intelligence and AI in Games","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kempka et al., 2016] Michał Kempka, Marek Wydmuch, Grzegorz Runc, Jakub Toczek, and Wojciech Jaśkowski. Vizdoom: A doom-based ai research platform for visual reinforcement learning. In IEEE Conference on Computa- tional Intelligence and Games (CIG), 2016.","title":"Vizdoom: A doom-based ai research platform for visual reinforcement learning","authors":"Michał Kempka, Marek Wydmuch, Grzegorz Runc, Jakub Toczek, Wojciech Jaśkowski","year":"2016","venue":"IEEE Conference on Computational Intelligence and Games (CIG)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Machado et al., 2017] Marlos C Machado, Marc G Belle- mare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environ- ment: Evaluation protocols and open problems for general agents. arXiv:1709.06009, 2017.","title":"Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents","authors":"C Marlos, Marc G Machado, Erik Bellemare, Joel Talvitie, Matthew Veness, Michael Hausknecht, Bowling","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidje- land, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Mnih et al., 2016] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asyn- chronous methods for deep reinforcement learning. In In- ternational conference on machine learning, pages 1928- 1937, 2016.","title":"Asynchronous methods for deep reinforcement learning","authors":"Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, Koray Kavukcuoglu","year":"2016","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Nichol et al., 2018] Alex Nichol, Vicki Pfau, Christopher Hesse, Oleg Klimov, and John Schulman. Gotta learn fast: A new benchmark for generalization in rl. arXiv:1804.03720, 2018.","title":"Gotta learn fast: A new benchmark for generalization in rl","authors":"Alex Nichol, Vicki Pfau, Christopher Hesse, Oleg Klimov, John Schulman","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Perez-Liebana et al., 2016] Diego Perez-Liebana, Spyridon Samothrakis, Julian Togelius, Simon M Lucas, and Tom Schaul. General video game ai: Competition, challenges and opportunities. In Thirtieth AAAI Conference on Artifi- cial Intelligence, pages 4335-4337, 2016.","title":"General video game ai: Competition, challenges and opportunities","authors":"Perez-Liebana","year":"2016","venue":"Thirtieth AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Resnick et al., 2018] Cinjon Resnick, Wes Eldridge, David Ha, Denny Britz, Jakob Foerster, Julian Togelius, Kyunghyun Cho, and Joan Bruna. Pommerman: A multi- agent playground. arXiv:1809.07124, 2018.","title":"Pommerman: A multiagent playground","authors":"Resnick","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Rohlfshagen et al., 2018] Philipp Rohlfshagen, Jialin Liu, Diego Perez-Liebana, and Simon M Lucas. Pac-man con- quers academia: Two decades of research using a classic arcade game. IEEE Transactions on Games, 10(3):233- 256, 2018.","title":"Pac-man conquers academia: Two decades of research using a classic arcade game","authors":"Jialin Philipp Rohlfshagen, Diego Liu, Simon M Perez-Liebana, Lucas","year":"2018","venue":"IEEE Transactions on Games","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Samuel, 1959] Arthur L Samuel. Some studies in machine learning using the game of checkers. IBM Journal of re- search and development, 3(3):210-229, 1959.","title":"Some studies in machine learning using the game of checkers","authors":"L Arthur, Samuel","year":"1959","venue":"IBM Journal of research and development","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Schulman et al., 2017] John Schulman, Filip Wolski, Pra- fulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv:1707.06347, 2017.","title":"Proximal policy optimization algorithms","authors":"John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Stiny and Gips, 1971] George Stiny and James Gips. Shape grammars and the generative specification of painting and sculpture. In IFIP Congress (2), volume 2, 1971.","title":"George Stiny and James Gips. Shape grammars and the generative specification of painting and sculpture","authors":"Gips Stiny","year":"1971","venue":"IFIP Congress (2)","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhang et al., 2018] Chiyuan Zhang, Oriol Vinyals, Remi Munos, and Samy Bengio. A study on overfitting in deep reinforcement learning. arXiv:1804.06893, 2018.","title":"A study on overfitting in deep reinforcement learning","authors":"Zhang","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}