{"type":"ScholarlyArticle","title":"Self-Guided Evolution Strategies with Historical Estimated Gradients","authors":[{"name":"Fei-Yu Liu","slug":"fei-yu-liu","given":"Fei-Yu","family":"Liu","affiliation":"National Key Laboratory for Novel Software Technology , Nanjing University , Nanjing 210023 , China; University of Science and Technology of China , Hefei 230027 , China"},{"name":"Zi-Niu Li","slug":"zi-niu-li","given":"Zi-Niu","family":"Li","affiliation":"Polixir , Nanjing 210038 , China"},{"name":"Chao Qian","slug":"chao-qian","given":"Chao","family":"Qian","affiliation":"National Key Laboratory for Novel Software Technology , Nanjing University , Nanjing 210023 , China"}],"abstract":"Evolution Strategies (ES) are a class of black-box optimization algorithms and have been widely applied to solve problems, e.g., in reinforcement learning (RL), where the true gradient is unavailable. ES estimate the gradient of an objective function with respect to the parameters by randomly sampling search directions and evaluating parameter perturbations in these directions. However, the gradient estimator of ES tends to have a high variance for high-dimensional optimization, thus requiring a large number of samples and making ES inefficient. In this paper, we propose a new ES algorithm SGES, which utilizes historical estimated gradients to construct a low-dimensional subspace for sampling search directions, and adjusts the importance of this subspace adaptively. We prove that the variance of the gradient estimator of SGES can be much smaller than that of Vanilla ES; meanwhile, its bias can be well bounded. Empirical results on benchmark black-box functions and a set of popular RL tasks exhibit the superior performance of SGES over state-of-the-art ES algorithms.","doi":"10.24963/ijcai.2020/205","url":"https://ijcai.org/proceedings/2020/205","pdf":"https://ijcai.org/proceedings/2020/0205.pdf","markdown":"https://ijcai.org/proceedings/2020/205.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/205","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Heuristic Search and Game Playing","track":"Main track","number":205,"pages":{"first":1474,"last":1480},"keywords":["Machine Learning: Reinforcement Learning","Heuristic Search and Game Playing: Heuristic Search and Machine Learning","Heuristic Search and Game Playing: Heuristic Search"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24704/file_24704.m3u8","length_sec":291,"provider":"hls"}],"references":[{"position":1,"raw":"[Auger and Hansen, 2012] A. Auger and N. Hansen. Tutorial CMA-ES: Evolution strategies and covariance matrix adaptation. In Proceedings of the 14th ACM Conference on Genetic and Evo- lutionary Computation (GECCO), pages 827-848, Philadelphia, PA, 2012.","title":"Tutorial CMA-ES: Evolution strategies and covariance matrix adaptation","authors":"A Auger, N Hansen","year":"2012","venue":"Proceedings of the 14th ACM Conference on Genetic and Evolutionary Computation (GECCO)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Brockman et al., 2016] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. OpenAI Gym. CoRR abs/1606.01540, 2016.","title":"OpenAI Gym","authors":"G Brockman, V Cheung, L Pettersson, J Schneider, J Schulman, J Tang, W Zaremba","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Choromanski et al., 2018] K. Choromanski, M. Rowland, V. Sind- hwani, R. E. Turner, and A. Weller. Structured evolution with compact architectures for scalable policy optimization. In Pro- ceedings of the 35th International Conference on Machine Learn- ing (ICML), pages 969-977, Stockholm, Sweden, 2018.","title":"Structured evolution with compact architectures for scalable policy optimization","authors":"K Choromanski, M Rowland, V Sindhwani, R E Turner, A Weller","year":"2018","venue":"Proceedings of the 35th International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Choromanski et al., 2019] K. Choromanski, A. Pacchiano, J. Parker-Holder, Y.-H. Tang, and V. Sindhwani. From complex- ity to simplicity: Adaptive ES-active subspaces for blackbox optimization. In Advances in Neural Information Processing Systems 32 (NeurIPS), pages 10299-10309, Vancouver, Canada, 2019.","title":"From complexity to simplicity: Adaptive ES-active subspaces for blackbox optimization","authors":"K Choromanski, A Pacchiano, J Parker-Holder, Y.-H Tang, V Sindhwani","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Colas et al., 2018] C. Colas, O. Sigaud, and P. Y. Oudeyer. GEP- PG: Decoupling exploration and exploitation in deep reinforce- ment learning algorithms. In Proceedings of the 35th Interna- tional Conference on Machine Learning (ICML), pages 1038- 1047, Stockholm, Sweden, 2018.","title":"GEP-PG: Decoupling exploration and exploitation in deep reinforcement learning algorithms","authors":"C Colas, O Sigaud, P Y Oudeyer","year":"2018","venue":"Proceedings of the 35th International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Conti et al., 2018] E. Conti, V. Madhavan, F. P. Such, J. Lehman, K. O. Stanley, and J. Clune. Improving exploration in evolu- tion strategies for deep reinforcement learning via a population of novelty-seeking agents. In Advances in Neural Information Processing Systems 31 (NeurIPS), pages 5032-5043, Montréal, Canada, 2018.","title":"Improving exploration in evolution strategies for deep reinforcement learning via a population of novelty-seeking agents","authors":"E Conti, V Madhavan, F P Such, J Lehman, K O Stanley, J Clune","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Friedrichs and Igel, 2005] F. Friedrichs and C. Igel. Evolutionary tuning of multiple SVM parameters. Neurocomputing, 64:107- 117, 2005.","title":"Evolutionary tuning of multiple SVM parameters","authors":"F Friedrichs, C Igel","year":"2005","venue":"Neurocomputing","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hansen, 2016] N. Hansen. The CMA evolution strategy: A tuto- rial. CoRR abs/1604.00772, 2016.","title":"The CMA evolution strategy: A tutorial","authors":"N Hansen","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Khadka and Tumer, 2018] S. Khadka and K. Tumer. Evolution- guided policy gradient in reinforcement learning. In Advances in Neural Information Processing Systems 31 (NeurIPS), pages 1196-1208, Montréal, Canada, 2018.","title":"Evolutionguided policy gradient in reinforcement learning","authors":"S Khadka, K Tumer","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kingma and Ba, 2015] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In Proceedings of the 3rd Inter- national Conference on Learning Representations (ICLR), San Diego, CA, 2015.","title":"Adam: A method for stochastic optimization","authors":"D P Kingma, J Ba","year":"2015","venue":"Proceedings of the 3rd International Conference on Learning Representations (ICLR)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lehman et al., 2018a] J. Lehman, J. Chen, J. Clune, and K. O. Stanley. ES is more than just a traditional finite-difference ap- proximator. In Proceedings of the 20th ACM Conference on Ge- netic and Evolutionary Computation (GECCO), pages 450-457, Kyoto, Japan, 2018.","title":"ES is more than just a traditional finite-difference approximator","authors":"Lehman","year":"2018","venue":"Proceedings of the 20th ACM Conference on Genetic and Evolutionary Computation (GECCO)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lehman et al., 2018b] J. Lehman, J. Chen, J. Clune, and K. O. Stanley. Safe mutations for deep and recurrent neural networks through output gradients. In Proceedings of the 20th ACM Con- ference on Genetic and Evolutionary Computation (GECCO), pages 117-124, Kyoto, Japan, 2018.","title":"Safe mutations for deep and recurrent neural networks through output gradients","authors":"Lehman","year":"2018","venue":"Proceedings of the 20th ACM Conference on Genetic and Evolutionary Computation (GECCO)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lillicrap et al., 2016] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. Contin- uous control with deep reinforcement learning. In Proceedings of the 4th International Conference on Learning Representations (ICLR), San Juan, Puerto Rico, 2016.","title":"Continuous control with deep reinforcement learning","authors":"T P Lillicrap, J J Hunt, A Pritzel, N Heess, T Erez, Y Tassa, D Silver, D Wierstra","year":"2016","venue":"Proceedings of the 4th International Conference on Learning Representations (ICLR)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Maheswaranathan et al., 2019] N. Maheswaranathan, L. Metz, G. Tucker, D. Choi, and J. Sohl-Dickstein. Guided evolutionary strategies: Augmenting random search with surrogate gradients. In Proceedings of the 36th International Conference on Machine Learning (ICML), pages 4264-4273, Long Beach, CA, 2019.","title":"Guided evolutionary strategies: Augmenting random search with surrogate gradients","authors":"N Maheswaranathan, L Metz, G Tucker, D Choi, J Sohl-Dickstein","year":"2019","venue":"Proceedings of the 36th International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mania et al., 2018] H. Mania, A. Guy, and B. Recht. Simple ran- dom search of static linear policies is competitive for reinforce- ment learning. In Advances in Neural Information Process- ing Systems 31 (NeurIPS), pages 1805-1814, Montréal, Canada, 2018.","title":"Simple random search of static linear policies is competitive for reinforcement learning","authors":"H Mania, A Guy, B Recht","year":"2018","venue":"Advances in Neural Information Processing Systems 31 (NeurIPS)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Nesterov and Spokoiny, 2017] Y. E. Nesterov and V. G. Spokoiny. Random gradient-free minimization of convex functions. Foun- dations of Computational Mathematics, 17(2):527-566, 2017.","title":"Random gradient-free minimization of convex functions","authors":"Y E Nesterov, V G Spokoiny","year":"2017","venue":"Foundations of Computational Mathematics","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Oord et al., 2017] A. Oord, O. Vinyals, and K. Kavukcuoglu. Neu- ral discrete representation learning. In Advances in Neural Infor- mation Processing Systems 30 (NIPS), pages 6306-6315, Long Beach, CA, 2017.","title":"Neural discrete representation learning","authors":"A Oord, O Vinyals, K Kavukcuoglu","year":"2017","venue":"Advances in Neural Information Processing Systems 30 (NIPS)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Plappert et al., 2018] M. Plappert, R. Houthooft, P. Dhariwal, S. Sidor, R. Y. Chen, X. Chen, T. Asfour, P. Abbeel, and M. Andrychowicz. Parameter space noise for exploration. In Pro- ceedings of the 6th International Conference on Learning Repre- sentations (ICLR), Vancouver, BC, 2018.","title":"Parameter space noise for exploration","authors":"M Plappert, R Houthooft, P Dhariwal, S Sidor, R Y Chen, X Chen, T Asfour, P Abbeel, M Andrychowicz","year":"2018","venue":"Proceedings of the 6th International Conference on Learning Representations (ICLR)","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Pourchot and Sigaud, 2019] A. Pourchot and O. Sigaud. CEM- RL: Combining evolutionary and gradient-based methods for policy search. In Proceedings of the 7th International Conference on Learning Representations (ICLR), New Orleans, LA, 2019.","title":"CEM-RL: Combining evolutionary and gradient-based methods for policy search","authors":"A Pourchot, O Sigaud","year":"2019","venue":"Proceedings of the 7th International Conference on Learning Representations (ICLR)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Rapin and Teytaud, 2018] J. Rapin and O. Teytaud. Nevergrad -A gradient-free optimization platform. http://GitHub.com/ FacebookResearch/Nevergrad, 2018.","title":"Nevergrad -A gradient-free optimization platform","authors":"J Rapin, O Teytaud","year":"2018","venue":null,"doi":null,"url":"http://GitHub.com/FacebookResearch/Nevergrad","internal_url":null},{"position":21,"raw":"[Salimans et al., 2017] T. Salimans, J. Ho, X. Chen, and I. Sutskever. Evolution strategies as a scalable alternative to rein- forcement learning. CoRR abs/1703.03864, 2017.","title":"Evolution strategies as a scalable alternative to reinforcement learning","authors":"T Salimans, J Ho, X Chen, I Sutskever","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Schulman et al., 2015] J. Schulman, S. Levine, P. Abbeel, M. I. Jordan, and P. Moritz. Trust region policy optimization. In Pro- ceedings of the 32th International Conference on Machine Learn- ing (ICML), pages 1889-1897, Lille, France, 2015.","title":"Trust region policy optimization","authors":"J Schulman, S Levine, P Abbeel, M I Jordan, P Moritz","year":"2015","venue":"Proceedings of the 32th International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Schulman et al., 2017] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization al- gorithms. CoRR abs/1707.06347, 2017.","title":"Proximal policy optimization algorithms","authors":"J Schulman, F Wolski, P Dhariwal, A Radford, O Klimov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Schwefel, 1984] H. P. Schwefel. Evolution strategies: A family of non-linear optimization techniques based on imitating some principles of organic evolution. Annals of Operations Research, 1(2):165-167, 1984.","title":"Evolution strategies: A family of non-linear optimization techniques based on imitating some principles of organic evolution","authors":"H P Schwefel","year":"1984","venue":"Annals of Operations Research","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Song et al., 2019] X.-Y. Song, W.-B. Gao, Y.-X. Yang, K. Choro- manski, A. Pacchiano, and Y.-H. Tang. ES-MAML: Simple Hessian-free meta learning. CoRR abs/1910.01215, 2019.","title":"ES-MAML: Simple Hessian-free meta learning","authors":"X.-Y Song, W.-B Gao, Y.-X Yang, K Choromanski, A Pacchiano, Y.-H Tang","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Sutskever et al., 2013] I. Sutskever, J. Martens, G. E. Dahl, and G. E. Hinton. On the importance of initialization and momen- tum in deep learning. In Proceedings of the 30th International Conference on Machine Learning (ICML), pages 1139-1147, At- lanta, GA, 2013.","title":"On the importance of initialization and momentum in deep learning","authors":"Sutskever","year":"2013","venue":"Proceedings of the 30th International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Vidnerova and Neruda, 2017] P. Vidnerova and R. Neruda. Evolu- tion strategies for deep neural network models design. In Pro- ceedings of the 17th Conference on Information Technologies - Applications and Theory (ITAT), pages 159-166, Martinské hole, Slovakia, 2017.","title":"Evolution strategies for deep neural network models design","authors":"P Vidnerova, R Neruda","year":"2017","venue":"Proceedings of the 17th Conference on Information Technologies -Applications and Theory (ITAT)","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wierstra et al., 2014] D. Wierstra, T. Schaul, T. Glasmachers, Y. Sun, J. Peters, and J. Schmidhuber. Natural evolution strate- gies. Journal of Machine Learning Research, 15(1):949-980, 2014.","title":"Natural evolution strategies","authors":"D Wierstra, T Schaul, T Glasmachers, Y Sun, J Peters, J Schmidhuber","year":"2014","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}