{"type":"ScholarlyArticle","title":"An Atari Model Zoo for Analyzing, Visualizing, and Comparing Deep Reinforcement Learning Agents","authors":[{"name":"Felipe Petroski Such","slug":"felipe-petroski-such","given":"Felipe","family":"Petroski Such","affiliation":""},{"name":"Vashisht Madhavan","slug":"vashisht-madhavan","given":"Vashisht","family":"Madhavan","affiliation":""},{"name":"Rosanne Liu","slug":"rosanne-liu","given":"Rosanne","family":"Liu","affiliation":""},{"name":"Rui Wang","slug":"rui-wang","given":"Rui","family":"Wang","affiliation":""},{"name":"Pablo Samuel Castro","slug":"pablo-samuel-castro","given":"Pablo Samuel","family":"Castro","affiliation":"Google Brain"},{"name":"Yulun Li","slug":"yulun-li","given":"Yulun","family":"Li","affiliation":""},{"name":"Jiale Zhi","slug":"jiale-zhi","given":"Jiale","family":"Zhi","affiliation":""},{"name":"Ludwig Schubert","slug":"ludwig-schubert","given":"Ludwig","family":"Schubert","affiliation":"OpenAI"},{"name":"Marc G. Bellemare","slug":"marc-g-bellemare","given":"Marc G.","family":"Bellemare","affiliation":"Google Brain"},{"name":"Jeff Clune","slug":"jeff-clune","given":"Jeff","family":"Clune","affiliation":""},{"name":"Joel Lehman","slug":"joel-lehman","given":"Joel","family":"Lehman","affiliation":""}],"abstract":"Much human and computational effort has aimed to improve how deep reinforcement learning (DRL) algorithms perform on benchmarks such as the Atari Learning Environment. Comparatively less effort has focused on understanding what has been learned by such methods, and investigating and comparing the representations learned by different families of DRL algorithms. Sources of friction include the onerous computational requirements, and general logistical and architectural complications for running DRL algorithms at scale. We lessen this friction, by (1) training several algorithms at scale and releasing trained models, (2) integrating with a previous DRL model release, and (3) releasing code that makes it easy for anyone to load, visualize, and analyze such models. This paper introduces the Atari Zoo framework, which contains models trained across benchmark Atari games, in an easy-to-use format, as well as code that implements common modes of analysis and connects such models to a popular neural network visualization library. Further, to demonstrate the potential of this dataset and software package, we show initial quantitative and qualitative comparisons between the performance and representations of several DRL algorithms, highlighting interesting and previously unknown distinctions between them.","doi":"10.24963/ijcai.2019/452","url":"https://ijcai.org/proceedings/2019/452","pdf":"https://ijcai.org/proceedings/2019/0452.pdf","markdown":"https://ijcai.org/proceedings/2019/452.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/452","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning M-Z","track":"Main track","number":452,"pages":{"first":3260,"last":3267},"keywords":["Machine Learning: Deep Learning","Machine Learning: Reinforcement Learning","Heuristic Search and Game Playing: Game Playing and Machine Learning","Machine Learning: Interpretability"],"videos":[],"references":[{"position":1,"raw":"[Annasamy and Sycara, 2018] Raghuram Mandyam An- nasamy and Katia Sycara. Towards better interpretability in deep q-networks. arXiv preprint arXiv:1809.05630, 2018.","title":"Towards better interpretability in deep q-networks","authors":"Sycara Annasamy","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bellemare et al., 2013] Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, 2013.","title":"The arcade learning environment: An evaluation platform for general agents","authors":"Yavar Marc G Bellemare, Joel Naddaf, Michael Veness, Bowling","year":"2013","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bellemare et al., 2017] Marc G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on re- inforcement learning. arXiv preprint arXiv:1707.06887, 2017.","title":"A distributional perspective on reinforcement learning","authors":"Will Marc G Bellemare, Rémi Dabney, Munos","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Castro et al., 2018] Pablo Samuel Castro, Subhodeep Moitra, Carles Gelada, Saurabh Kumar, and Marc G Bellemare. Dopamine: A research framework for deep reinforcement learning. arXiv preprint arXiv:1812.06110, 2018.","title":"Dopamine: A research framework for deep reinforcement learning","authors":"Castro","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Conti et al., 2017] Edoardo Conti, Vashisht Madhavan, Fe- lipe Petroski Such, Joel Lehman, Kenneth O Stanley, and Jeff Clune. Improving exploration in evolution strate- gies for deep reinforcement learning via a population of novelty-seeking agents. arXiv preprint arXiv:1712.06560, 2017.","title":"Improving exploration in evolution strategies for deep reinforcement learning via a population of novelty-seeking agents","authors":"Edoardo Conti, Vashisht Madhavan, Felipe Petroski Such, Joel Lehman, Kenneth O Stanley, Jeff Clune","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dann et al., 2016] Christoph Dann, Katja Hofmann, and Se- bastian Nowozin. Memory lens: How much memory does an agent use? arXiv preprint arXiv:1611.06928, 2016.","title":"Memory lens: How much memory does an agent use?","authors":"Dann","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Erhan et al., 2009] Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. Visualizing higher-layer features of a deep network. University of Montreal, 1341(3):1, 2009.","title":"Visualizing higher-layer features of a deep network","authors":"Yoshua Dumitru Erhan, Aaron Bengio, Pascal Courville, Vincent","year":"2009","venue":"University of Montreal","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Espeholt et al., 2018] Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymir Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, et al. Impala: Scalable distributed deep-rl with impor- tance weighted actor-learner architectures. arXiv preprint arXiv:1802.01561, 2018.","title":"Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures","authors":"Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymir Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Finn et al., 2017] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. arXiv preprint arXiv:1703.03400, 2017.","title":"Model-agnostic meta-learning for fast adaptation of deep networks","authors":"Finn","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Greydanus et al., 2017] Sam Greydanus, Anurag Koul, Jonathan Dodge, and Alan Fern. Visualizing and under- standing atari agents. arXiv preprint arXiv:1711.00138, 2017.","title":"Visualizing and understanding atari agents","authors":"Sam Greydanus, Anurag Koul, Jonathan Dodge, Alan Fern","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hessel et al., 2017] Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dab- ney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep re- inforcement learning. arXiv preprint arXiv:1710.02298, 2017.","title":"Rainbow: Combining improvements in deep reinforcement learning","authors":"Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, David Silver","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Horgan et al., 2018] Dan Horgan, John Quan, David Bud- den, Gabriel Barth-Maron, Matteo Hessel, Hado Van Has- selt, and David Silver. Distributed prioritized experience replay. arXiv preprint arXiv:1803.00933, 2018.","title":"Distributed prioritized experience replay","authors":"Dan Horgan, John Quan, David Budden, Gabriel Barth-Maron, Matteo Hessel, Hado Van Hasselt, David Silver","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Jaderberg et al., 2016] Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. arXiv preprint arXiv:1611.05397, 2016.","title":"Reinforcement learning with unsupervised auxiliary tasks","authors":"Jaderberg","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mahendran and Vedaldi, 2016] Aravindh Mahendran and Andrea Vedaldi. Visualizing deep convolutional neural networks using natural pre-images. International Journal of Computer Vision, 120(3):233-255, 2016.","title":"Visualizing deep convolutional neural networks using natural pre-images","authors":"Vedaldi Mahendran, Aravindh Mahendran, Andrea Vedaldi","year":"2016","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidje- land, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Mnih et al., 2016] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asyn- chronous methods for deep reinforcement learning. In In- ternational conference on machine learning, pages 1928- 1937, 2016.","title":"Asynchronous methods for deep reinforcement learning","authors":"Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, Koray Kavukcuoglu","year":"2016","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Nguyen et al., 2017] Anh Nguyen, Jeff Clune, Yoshua Ben- gio, Alexey Dosovitskiy, and Jason Yosinski. Plug & play generative networks: Conditional iterative generation of images in latent space. In CVPR, volume 2, page 7, 2017.","title":"Plug & play generative networks: Conditional iterative generation of images in latent space","authors":"Anh Nguyen, Jeff Clune, Yoshua Bengio, Alexey Dosovitskiy, Jason Yosinski","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Olah et al., 2018] Chris Olah, Arvind Satyanarayan, Ian Johnson, Shan Carter, Ludwig Schubert, Katherine Ye, and Alexander Mordvintsev. The building blocks of in- terpretability. Distill, 3(3):e10, 2018.","title":"The building blocks of interpretability","authors":"Chris Olah, Arvind Satyanarayan, Ian Johnson, Shan Carter, Ludwig Schubert, Katherine Ye, Alexander Mordvintsev","year":"2018","venue":"Distill","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Pathak et al., 2017] Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International Conference on Machine Learning (ICML), volume 2017, 2017.","title":"Curiosity-driven exploration by self-supervised prediction","authors":"Deepak Pathak, Pulkit Agrawal, Alexei A Efros, Trevor Darrell","year":"2017","venue":"International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Salimans et al., 2017] Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. Evolution strategies as a scalable alternative to reinforcement learning. arXiv preprint arXiv:1703.03864, 2017.","title":"Evolution strategies as a scalable alternative to reinforcement learning","authors":"Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, Ilya Sutskever","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Schulman et al., 2017] John Schulman, Filip Wolski, Pra- fulla Dhariwal, Alec Radford, and Oleg Klimov. Prox- imal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.","title":"Proximal policy optimization algorithms","authors":"John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Simonyan et al., 2013] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034, 2013.","title":"Deep inside convolutional networks: Visualising image classification models and saliency maps","authors":"Karen Simonyan, Andrea Vedaldi, Andrew Zisserman","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Such et al., 2017] Felipe Petroski Such, Vashisht Madha- van, Edoardo Conti, Joel Lehman, Kenneth O Stanley, and Jeff Clune. Deep neuroevolution: genetic algo- rithms are a competitive alternative for training deep neu- ral networks for reinforcement learning. arXiv preprint arXiv:1712.06567, 2017.","title":"Deep neuroevolution: genetic algorithms are a competitive alternative for training deep neural networks for reinforcement learning","authors":"Felipe Petroski Such, Vashisht Madhavan, Edoardo Conti, Joel Lehman, Kenneth O Stanley, Jeff Clune","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang et al., 2015] Ziyu Wang, Tom Schaul, Matteo Hes- sel, Hado Van Hasselt, Marc Lanctot, and Nando De Fre- itas. Dueling network architectures for deep reinforcement learning. arXiv preprint arXiv:1511.06581, 2015.","title":"Dueling network architectures for deep reinforcement learning","authors":"Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Van Hasselt, Marc Lanctot, Nando De Freitas","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yosinski et al., 2015] Jason Yosinski, Jeff Clune, Anh Nguyen, Thomas Fuchs, and Hod Lipson. Understanding neural networks through deep visualization. arXiv preprint arXiv:1506.06579, 2015.","title":"Understanding neural networks through deep visualization","authors":"Jason Yosinski, Jeff Clune, Anh Nguyen, Thomas Fuchs, Hod Lipson","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zahavy et al., 2016] Tom Zahavy, Nir Ben-Zrihem, and Shie Mannor. Graying the black box: Understanding dqns. In International Conference on Machine Learning, pages 1899-1908, 2016.","title":"Graying the black box: Understanding dqns","authors":"Zahavy","year":"2016","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zeiler and Fergus, 2014] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pages 818-833, 2014.","title":"Visualizing and understanding convolutional networks","authors":"D Matthew, Rob Zeiler, Fergus","year":"2014","venue":"European conference on computer vision","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}