{"type":"ScholarlyArticle","title":"A Restart-based Rank-1 Evolution Strategy for Reinforcement Learning","authors":[{"name":"Zefeng Chen","slug":"zefeng-chen","given":"Zefeng","family":"Chen","affiliation":"School of Data and Computer Science , Sun Yat-sen University"},{"name":"Yuren Zhou","slug":"yuren-zhou","given":"Yuren","family":"Zhou","affiliation":"School of Data and Computer Science , Sun Yat-sen University; Engineering Research Institute , Guangzhou College of South China University of Technology"},{"name":"Xiao-yu He","slug":"xiao-yu-he","given":"Xiao-yu","family":"He","affiliation":"School of Data and Computer Science , Sun Yat-sen University"},{"name":"Siyu Jiang","slug":"siyu-jiang","given":"Siyu","family":"Jiang","affiliation":"School of Software Engineering , South China University of Technology"}],"abstract":"Evolution strategies have been demonstrated to have the strong ability to roughly train deep neural networks and well accomplish reinforcement learning tasks. However, existing evolution strategies designed specially for deep reinforcement learning only involve the plain variants which can not realize the adaptation of mutation strength or other advanced techniques. The research of applying advanced and effective evolution strategies to reinforcement learning in an efficient way is still a gap. To this end, this paper proposes a restart-based rank-1 evolution strategy for reinforcement learning. When training the neural network, it adapts the mutation strength and updates the principal search direction in a way similar to the momentum method, which is an ameliorated version of stochastic gradient ascent. Besides, two mechanisms, i.e., the adaptation of the number of elitists and the restart procedure, are integrated to deal with the issue of local optima. Experimental results on classic control problems and Atari games show that the proposed algorithm is superior to or competitive with state-of-the-art algorithms for reinforcement learning, demonstrating the effectiveness of the proposed algorithm.","doi":"10.24963/ijcai.2019/295","url":"https://ijcai.org/proceedings/2019/295","pdf":"https://ijcai.org/proceedings/2019/0295.pdf","markdown":"https://ijcai.org/proceedings/2019/295.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/295","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning A-L","track":"Main track","number":295,"pages":{"first":2130,"last":2136},"keywords":["Machine Learning: Reinforcement Learning","Heuristic Search and Game Playing: Heuristic Search and Machine Learning","Machine Learning: Deep Learning","Machine Learning Applications: Applications of Reinforcement Learning"],"videos":[],"references":[{"position":1,"raw":"[Beyer, 1993] Hans-Georg Beyer. Toward a theory of evolu- tion strategies: Some asymptotical results from the (1,+λ)- theory. Evolutionary Computation, 1(2):165-188, June 1993.","title":"Toward a theory of evolution strategies: Some asymptotical results from the (1,+λ)theory","authors":"Hans-Georg Beyer, Beyer","year":"1993","venue":"Evolutionary Computation","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Brockman et al., 2016] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. CoRR, ab- s/1606.01540, 2016.","title":"Openai gym","authors":"Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, Wojciech Zaremba","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chrabaszcz et al., 2018] Patryk Chrabaszcz, Ilya Loshchilov, and Frank Hutter. Back to basics: Bench- marking canonical evolution strategies for playing atari. CoRR, abs/1802.08842, 2018.","title":"Back to basics: Benchmarking canonical evolution strategies for playing atari","authors":"Patryk Chrabaszcz, Ilya Loshchilov, Frank Hutter","year":"2018","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2018/197"},{"position":4,"raw":"[Conti et al., 2017] Edoardo Conti, Vashisht Madhavan, Fe- lipe Petroski Such, Joel Lehman, Kenneth O. Stanley, and Jeff Clune. Improving exploration in evolution strate- gies for deep reinforcement learning via a population of novelty-seeking agents. CoRR, abs/1712.06560, 2017.","title":"Improving exploration in evolution strategies for deep reinforcement learning via a population of novelty-seeking agents","authors":"Edoardo Conti, Vashisht Madhavan, Felipe Petroski Such, Joel Lehman, Kenneth O Stanley, Jeff Clune","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Hansen and Kern, 2004] Nikolaus Hansen and Stefan Kern. Evaluating the cma evolution strategy on multimodal test functions. In Xin Yao, Edmund K. Burke, José A. Lozano, Jim Smith, Juan Julián Merelo-Guervós, John A. Bullinar- ia, Jonathan E. Rowe, Peter Tiňo, Ata Kabán, and Hans- Paul Schwefel, editors, Parallel Problem Solving from Nature -PPSN VIII, pages 282-291, Berlin, Heidelberg, 2004. Springer Berlin Heidelberg.","title":"Evaluating the cma evolution strategy on multimodal test functions","authors":"Kern Hansen, Nikolaus Hansen, Stefan Kern","year":"2004","venue":"Parallel Problem Solving from Nature -PPSN VIII","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hansen and Ostermeier, 2014] Nikolaus Hansen and An- dreas Ostermeier. Completely derandomized self- adaptation in evolution strategies. Evolutionary Compu- tation, 9(2):159-195, 2014.","title":"Completely derandomized selfadaptation in evolution strategies","authors":"Ostermeier Hansen","year":"2014","venue":"Evolutionary Computation","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hausknecht et al., 2014] Matthew Hausknecht, Joel Lehman, Risto Miikkulainen, and Peter Stone. A neu- roevolution approach to general atari game playing. IEEE Transactions on Computational Intelligence and AI in Games, 6(4):355-366, 2014.","title":"A neuroevolution approach to general atari game playing","authors":"Matthew Hausknecht, Joel Lehman, Risto Miikkulainen, Peter Stone","year":"2014","venue":"IEEE Transactions on Computational Intelligence and AI in Games","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Koutnik et al., 2010] Jan Koutnik, Faustino Gomez, and Jürgen Schmidhuber. Evolving neural networks in com- pressed weight space. In Proceedings of the 12th Annu- al Conference on Genetic and Evolutionary Computation, GECCO '10, pages 619-626, New York, NY, USA, 2010. ACM.","title":"Evolving neural networks in compressed weight space","authors":"Koutnik","year":"2010","venue":"Proceedings of the 12th Annual Conference on Genetic and Evolutionary Computation, GECCO '10","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Koutník et al., 2013] Jan Koutník, Giuseppe Cuccu, Jürgen Schmidhuber, and Faustino Gomez. Evolving large-scale neural networks for vision-based reinforcement learning. In Proceedings of the 15th Annual Conference on Genetic and Evolutionary Computation, GECCO '13, pages 1061- 1068, New York, NY, USA, 2013. ACM.","title":"Evolving large-scale neural networks for vision-based reinforcement learning","authors":"Koutník","year":"2013","venue":"Proceedings of the 15th Annual Conference on Genetic and Evolutionary Computation, GECCO '13","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lehman and Stanley, 2011] Joel Lehman and Kenneth O. S- tanley. Novelty Search and the Problem with Objectives. Springer New York, 2011.","title":"Novelty Search and the Problem with Objectives","authors":"Stanley ; Lehman, Joel Lehman, Kenneth O Stanley","year":"2011","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lehman et al., 2017] Joel Lehman, Jay Chen, Jeff Clune, and Kenneth O. Stanley. ES is more than just a tradition- al finite-difference approximator. CoRR, abs/1712.06568, 2017.","title":"ES is more than just a traditional finite-difference approximator","authors":"Joel Lehman, Jay Chen, Jeff Clune, Kenneth O Stanley","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li and Zhang, 2017] Zhenhua Li and Qingfu Zhang. A sim- ple yet efficient evolution strategy for large scale black- box optimization. IEEE Transactions on Evolutionary Computation, pages 1-1, 2017.","title":"A simple yet efficient evolution strategy for large scale blackbox optimization","authors":"Li, Zhang, Zhenhua Li, Qingfu Zhang","year":"2017","venue":"IEEE Transactions on Evolutionary Computation","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fidje- land, and Georg Ostrovski. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Pugh et al., 2016] Justin K Pugh, Lisa B Soros, and Ken- neth O Stanley. Quality diversity: A new frontier for evo- lutionary computation. Frontiers in Robotics and AI, 3:40, 2016.","title":"Quality diversity: A new frontier for evolutionary computation","authors":"Justin K Pugh, Lisa B Soros, Kenneth O Stanley","year":"2016","venue":"Frontiers in Robotics and AI","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Qian, 1999] Ning Qian. On the momentum term in gradient descent learning algorithms. Neural Networks, 12(1):145 -151, 1999.","title":"On the momentum term in gradient descent learning algorithms","authors":"Ning Qian, Qian","year":"1999","venue":"Neural Networks","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Risi and Togelius, 2017] Sebastian Risi and Julian Togelius. Neuroevolution in games: State of the art and open chal- lenges. IEEE Transactions on Computational Intelligence and AI in Games, 9(1):25-41, 2017.","title":"Neuroevolution in games: State of the art and open challenges","authors":"Togelius Risi, Sebastian Risi, Julian Togelius","year":"2017","venue":"IEEE Transactions on Computational Intelligence and AI in Games","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Ruder, 2016] Sebastian Ruder. An overview of gradien- t descent optimization algorithms. arXiv preprint arX- iv:1609.04747, 2016.","title":"An overview of gradient descent optimization algorithms","authors":"Sebastian Ruder","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Salimans et al., 2016] Tim Salimans, Ian J. Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and X- i Chen. Improved techniques for training gans. CoRR, abs/1606.03498, 2016.","title":"Improved techniques for training gans","authors":"Tim Salimans, Ian J Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, Xi Chen","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Salimans et al., 2017] Tim Salimans, Jonathan Ho, X- i Chen, and Ilya Sutskever. Evolution strategies as a s- calable alternative to reinforcement learning. CoRR, ab- s/1703.03864, 2017.","title":"Evolution strategies as a scalable alternative to reinforcement learning","authors":"Tim Salimans, Jonathan Ho, Xi Chen, Ilya Sutskever","year":"2017","venue":"CoRR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Schwefel, 1981] Hans Paul Schwefel. Numerical optimiza- tion of computer models. Journal of the Operational Re- search Society, 33(12):1166-1166, 1981.","title":"Numerical optimization of computer models","authors":"Hans Paul Schwefel, Schwefel","year":"1981","venue":"Journal of the Operational Research Society","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Sehnke et al., 2010] Frank Sehnke, Christian Osendorfer, Thomas Rückstieß, Alex Graves, Jan Peters, and Jürgen Schmidhuber. Parameter-exploring policy gradients. Neu- ral Networks the Official Journal of the International Neu- ral Network Society, 23(4):551-559, 2010.","title":"Parameter-exploring policy gradients","authors":"Frank Sehnke, Christian Osendorfer, Thomas Rückstieß, Alex Graves, Jan Peters, Jürgen Schmidhuber","year":"2010","venue":"Neural Networks the Official Journal of the International Neural Network Society","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Srivastava et al., 2012] Rupesh Kumar Srivastava, Jürgen Schmidhuber, and Faustino Gomez. Generalized com- pressed network search. In Conference Companion on Genetic and Evolutionary Computation, pages 647-648, 2012.","title":"Generalized compressed network search","authors":"Rupesh Kumar Srivastava, Jürgen Schmidhuber, Faustino Gomez","year":"2012","venue":"Conference Companion on Genetic and Evolutionary Computation","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Sutton and Barto, 1998] Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. MIT Press, 1998.","title":"Reinforcement Learning: An Introduction","authors":"Richard S Sutton, Andrew G Barto","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Varelas et al., 2018] Konstantinos Varelas, Anne Auger, Di- mo Brockhoff, Nikolaus Hansen, Ouassim Ait ElHara, Yann Semet, Rami Kassab, and Frédéric Barbaresco. A comparative study of large-scale variants of cma-es. In","title":"A comparative study of large-scale variants of cma-es","authors":"Konstantinos Varelas, Anne Auger, Dimo Brockhoff, Nikolaus Hansen, Ouassim Ait Elhara, Yann Semet, Rami Kassab, Frédéric Barbaresco","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"Anne Auger, Carlos M. Fonseca, Nuno Lourenc ¸o, Pe- nousal Machado, Luís Paquete, and Darrell Whitley, ed- itors, Parallel Problem Solving from Nature -PPSN XV, pages 3-15, Cham, 2018. Springer International Publish- ing.","title":"itors","authors":"Anne Auger, Carlos M Fonseca, Nuno Lourenc ¸o, Penousal Machado, Luís Paquete, Darrell Whitley","year":"2018","venue":"Parallel Problem Solving from Nature -PPSN XV","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wierstra et al., 2008] Daan Wierstra, Tom Schaul, Jan Pe- ters, and Juergen Schmidhuber. Natural evolution strate- gies. In 2008 IEEE Congress on Evolutionary Compu- tation (IEEE World Congress on Computational Intelli- gence), pages 3381-3387, June 2008.","title":"Natural evolution strategies","authors":"Daan Wierstra, Tom Schaul, Jan Peters, Juergen Schmidhuber","year":"2008","venue":"IEEE Congress on Evolutionary Computation (IEEE World Congress on Computational Intelligence)","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Williams, 1992] Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforce- ment learning. Machine Learning, 8(3-4):229-256, 1992.","title":"Simple statistical gradient-following algorithms for connectionist reinforcement learning","authors":"Ronald J Williams","year":"1992","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhang et al., 2017] Xingwen Zhang, Jeff Clune, and Ken- neth O. Stanley. On the relationship between the openai evolution strategy and stochastic gradient descent. CoRR, abs/1712.06564, 2017.","title":"On the relationship between the openai evolution strategy and stochastic gradient descent","authors":"Zhang","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}