{"type":"ScholarlyArticle","title":"Back to Basics: Benchmarking Canonical Evolution Strategies for Playing Atari","authors":[{"name":"Patryk Chrabąszcz","slug":"patryk-chrabaszcz","given":"Patryk","family":"Chrabąszcz","affiliation":"University of Freiburg , Freiburg , Germany"},{"name":"Ilya Loshchilov","slug":"ilya-loshchilov","given":"Ilya","family":"Loshchilov","affiliation":"University of Freiburg , Freiburg , Germany"},{"name":"Frank Hutter","slug":"frank-hutter","given":"Frank","family":"Hutter","affiliation":"University of Freiburg , Freiburg , Germany"}],"abstract":"Evolution Strategies (ES) have recently been demonstrated to be a viable alternative to reinforcement learning (RL) algorithms on a set of challenging deep learning problems, including Atari games and MuJoCo humanoid locomotion benchmarks. While the ES algorithms in that work belonged to the specialized class of natural evolution strategies (which resemble approximate gradient RL algorithms, such as REINFORCE), we demonstrate that even a very basic canonical ES algorithm can achieve the same or even better performance. This success of a basic ES algorithm suggests that the state-of-the-art can be advanced further by integrating the many advances made in the field of ES in the last decades.We also demonstrate that ES algorithms have very different performance characteristics than traditional RL algorithms: on some games, they learn to exploit the environment and perform much better while on others they can get stuck in suboptimal local minima. Combining their strengths and weaknesses with those of traditional RL algorithms is therefore likely to lead to new advances in the state-of-the-art for solving RL problems.","doi":"10.24963/ijcai.2018/197","url":"https://ijcai.org/proceedings/2018/197","pdf":"https://ijcai.org/proceedings/2018/0197.pdf","markdown":"https://ijcai.org/proceedings/2018/197.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/197","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Heuristic Search and Game Playing","track":"Main track","number":197,"pages":{"first":1419,"last":1426},"keywords":["Heuristic Search and Game Playing: Game Playing and Machine Learning","Machine Learning: Reinforcement Learning","Heuristic Search and Game Playing: Evaluation and Analysis"],"videos":[],"references":[{"position":1,"raw":"[Brockhoff et al., 2010] Dimo Brockhoff, Anne Auger, Niko- laus Hansen, Dirk V Arnold, and Tim Hohm. Mirrored sampling and sequential selection for evolution strategies. In PPSN, pages 11-21. Springer, 2010.","title":"Mirrored sampling and sequential selection for evolution strategies","authors":"Dimo Brockhoff, Anne Auger, Nikolaus Hansen, Dirk V Arnold, Tim Hohm","year":"2010","venue":"PPSN","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Brockman et al., 2016] Greg Brockman, Vicki Cheung, Lud- wig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540, 2016.","title":"Openai gym","authors":"Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, Wojciech Zaremba","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Clevert et al., 2015] Djork-Arné Clevert, Thomas Un- terthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv:1511.07289, 2015.","title":"Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units","authors":"Clevert","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Conti et al., 2017] Edoardo Conti, Vashisht Madhavan, Fe- lipe Petroski Such, Joel Lehman, Kenneth O Stanley, and Jeff Clune. Improving exploration in evolution strategies for deep reinforcement learning via a population of novelty- seeking agents. arXiv preprint arXiv:1712.06560, 2017.","title":"Improving exploration in evolution strategies for deep reinforcement learning via a population of noveltyseeking agents","authors":"Edoardo Conti, Vashisht Madhavan, Felipe Petroski Such, Joel Lehman, Kenneth O Stanley, Jeff Clune","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dhariwal et al., 2017] Prafulla Dhariwal, Christopher Hesse, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, and Yuhuai Wu. Openai baselines. https://github. com/openai/baselines, 2017.","title":"Openai baselines","authors":"Prafulla Dhariwal, Christopher Hesse, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, Yuhuai Wu","year":"2017","venue":null,"doi":null,"url":"https://github.com/openai/baselines","internal_url":null},{"position":6,"raw":"[Duan et al., 2016] Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In ICML, pages 1329-1338, 2016.","title":"Benchmarking deep reinforcement learning for continuous control","authors":"Yan Duan, Xi Chen, Rein Houthooft, John Schulman, Pieter Abbeel","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Glasmachers and Igel, 2008] Tobias Glasmachers and Chris- tian Igel. Uncertainty handling in model selection for sup- port vector machines. In PPSN, pages 185-194, 2008.","title":"Tobias Glasmachers and Christian Igel. Uncertainty handling in model selection for support vector machines","authors":"Igel Glasmachers","year":"2008","venue":"PPSN","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gomez et al., 2008] Faustino Gomez, Jürgen Schmidhuber, and Risto Miikkulainen. Accelerated neural evolution through cooperatively coevolved synapses. Journal of Ma- chine Learning Research, 9(May):937-965, 2008.","title":"Accelerated neural evolution through cooperatively coevolved synapses","authors":"Faustino Gomez, Jürgen Schmidhuber, Risto Miikkulainen","year":"2008","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hansen and Ostermeier, 1996] Nikolaus Hansen and An- dreas Ostermeier. Adapting arbitrary normal mutation dis- tributions in evolution strategies: The covariance matrix adaptation. In CEC 1996, pages 312-317. IEEE, 1996.","title":"Adapting arbitrary normal mutation distributions in evolution strategies: The covariance matrix adaptation","authors":"Ostermeier Hansen","year":"1996","venue":"CEC 1996","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hansen et al., 2003] Nikolaus Hansen, Sibylle D Müller, and Petros Koumoutsakos. Reducing the time complexity of the derandomized evolution strategy with covariance matrix adaptation (CMA-ES). Evolutionary computation, 11(1):1- 18, 2003.","title":"Reducing the time complexity of the derandomized evolution strategy with covariance matrix adaptation (CMA-ES)","authors":"Nikolaus Hansen, Sibylle D Müller, Petros Koumoutsakos","year":"2003","venue":"Evolutionary computation","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Heidrich-Meisner and Igel, 2009] Verena Heidrich-Meisner and Christian Igel. Hoeffding and Bernstein races for se- lecting policies in evolutionary direct policy search. In ICML, pages 401-408. ACM, 2009.","title":"Hoeffding and Bernstein races for selecting policies in evolutionary direct policy search","authors":"Igel Heidrich-Meisner, Verena Heidrich-Meisner, Christian Igel","year":"2009","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Igel, 2011] Christian Igel. Evolutionary kernel learning. In Encyclopedia of Machine Learning, pages 369-373. Springer, 2011.","title":"Christian Igel. Evolutionary kernel learning","authors":"Igel","year":"2011","venue":"Encyclopedia of Machine Learning","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Ioffe and Szegedy, 2015] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, pages 448-456, 2015.","title":"Batch normalization: Accelerating deep network training by reducing internal covariate shift","authors":"Szegedy Ioffe, Sergey Ioffe, Christian Szegedy","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kingma and Ba, 2014] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lehman et al., 2017] Joel Lehman, Jay Chen, Jeff Clune, and Kenneth O Stanley. ES is more than just a tra- ditional finite-difference approximator. arXiv preprint arXiv:1712.06568, 2017.","title":"ES is more than just a traditional finite-difference approximator","authors":"Joel Lehman, Jay Chen, Jeff Clune, Kenneth O Stanley","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Li, 2017] Yuxi Li. Deep reinforcement learning: An overview. arXiv preprint arXiv:1701.07274, 2017.","title":"Deep reinforcement learning: An overview","authors":"Li ;, Yuxi Li","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Loshchilov and Hutter, 2016] Ilya Loshchilov and Frank Hutter. CMA-ES for Hyperparameter Optimization of Deep Neural Networks. arXiv preprint arXiv:1604.07269, 2016.","title":"CMA-ES for Hyperparameter Optimization of Deep Neural Networks","authors":"Hutter Loshchilov, Ilya Loshchilov, Frank Hutter","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Loshchilov et al., 2017] Ilya Loshchilov, Tobias Glasmach- ers, and Hans-Georg Beyer. Limited-memory matrix adap- tation for large scale black-box optimization. arXiv preprint arXiv:1705.06693, 2017.","title":"Limited-memory matrix adaptation for large scale black-box optimization","authors":"Loshchilov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Mnih and others, 2015] Volodymyr Mnih et al. Human- level control through deep reinforcement learning. Nature, 518(7540):529-533, 2015.","title":"Volodymyr Mnih et al. Humanlevel control through deep reinforcement learning","authors":"Others Mnih","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Mnih et al., 2013] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep rein- forcement learning. arXiv preprint arXiv:1312.5602, 2013.","title":"Playing atari with deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, Martin Riedmiller","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Mnih et al., 2016] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asyn- chronous methods for deep reinforcement learning. In ICML, pages 1928-1937, 2016.","title":"Asynchronous methods for deep reinforcement learning","authors":"Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, Koray Kavukcuoglu","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Plappert et al., 2017] Matthias Plappert, Rein Houthooft, Prafulla Dhariwal, Szymon Sidor, Richard Y Chen, Xi Chen, Tamim Asfour, Pieter Abbeel, and Marcin Andrychowicz. Parameter space noise for exploration. arXiv preprint arXiv:1706.01905, 2017.","title":"Parameter space noise for exploration","authors":"Matthias Plappert, Rein Houthooft, Prafulla Dhariwal, Szymon Sidor, Xi Richard Y Chen, Tamim Chen, Pieter Asfour, Marcin Abbeel, Andrychowicz","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Rechenberg, 1973] Ingo Rechenberg. Evolutionsstrategie- optimierung technisher systeme nach prinzipien der biolo- gischen evolution. 1973.","title":"Evolutionsstrategieoptimierung technisher systeme nach prinzipien der biologischen evolution","authors":"Ingo Rechenberg","year":"1973","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Rudolph, 1997] Günter Rudolph. Convergence properties of evolutionary algorithms. Kovac, 1997.","title":"Convergence properties of evolutionary algorithms","authors":"Rudolph ;, Günter Rudolph","year":"1997","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Salimans et al., 2016] Tim Salimans, Ian Goodfellow, Woj- ciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In NIPS, pages 2234-2242, 2016.","title":"Improved techniques for training gans","authors":"Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, Xi Chen","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Salimans et al., 2017] Tim Salimans, Jonathan Ho, Xi Chen, and Ilya Sutskever. Evolution strategies as a scalable alternative to reinforcement learning. arXiv preprint arXiv:1703.03864, 2017.","title":"Evolution strategies as a scalable alternative to reinforcement learning","authors":"Tim Salimans, Jonathan Ho, Xi Chen, Ilya Sutskever","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Stulp and Sigaud, 2013] Freek Stulp and Olivier Sigaud. Robot skill learning: From reinforcement learning to evolu- tion strategies. Paladyn, Journal of Behavioral Robotics, 4(1):49-61, 2013.","title":"Robot skill learning: From reinforcement learning to evolution strategies","authors":"Sigaud Stulp, Freek Stulp, Olivier Sigaud","year":"2013","venue":"Paladyn, Journal of Behavioral Robotics","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Such et al., 2017] Felipe Petroski Such, Vashisht Madhavan, Edoardo Conti, Joel Lehman, Kenneth O Stanley, and Jeff Clune. Deep neuroevolution: Genetic algorithms are a competitive alternative for training deep neural networks for reinforcement learning. arXiv preprint arXiv:1712.06567, 2017.","title":"Deep neuroevolution: Genetic algorithms are a competitive alternative for training deep neural networks for reinforcement learning","authors":"Felipe Petroski Such, Vashisht Madhavan, Edoardo Conti, Joel Lehman, Kenneth O Stanley, Jeff Clune","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Todorov et al., 2012] Emanuel Todorov, Tom Erez, and Yu- val Tassa. Mujoco: A physics engine for model-based control. In IROS, pages 5026-5033. IEEE, 2012.","title":"Mujoco: A physics engine for model-based control","authors":"Emanuel Todorov, Tom Erez, Yuval Tassa","year":"2012","venue":"IROS","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wierstra et al., 2014] Daan Wierstra, Tom Schaul, Tobias Glasmachers, Yi Sun, Jan Peters, and Jürgen Schmidhuber. Natural evolution strategies. Journal of Machine Learning Research, 15(1):949-980, 2014.","title":"Natural evolution strategies","authors":"Daan Wierstra, Tom Schaul, Tobias Glasmachers, Yi Sun, Jan Peters, Jürgen Schmidhuber","year":"2014","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Williams, 1992] Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforce- ment learning. In Reinforcement Learning, pages 5-32. Springer, 1992.","title":"Simple statistical gradient-following algorithms for connectionist reinforcement learning","authors":"Williams Ronald","year":"1992","venue":"Reinforcement Learning","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}