{"type":"ScholarlyArticle","title":"Hard-Thresholding Meets Evolution Strategies in Reinforcement Learning","authors":[{"name":"Chengqian Gao","slug":"chengqian-gao","given":"Chengqian","family":"Gao","affiliation":"Mohamed bin Zayed University of Artificial Intelligence , UAE"},{"name":"William de Vazelhes","slug":"william-de-vazelhes","given":"William","family":"de Vazelhes","affiliation":"Mohamed bin Zayed University of Artificial Intelligence , UAE"},{"name":"Hualin Zhang","slug":"hualin-zhang","given":"Hualin","family":"Zhang","affiliation":"Mohamed bin Zayed University of Artificial Intelligence , UAE"},{"name":"Bin Gu","slug":"bin-gu","given":"Bin","family":"Gu","affiliation":"Mohamed bin Zayed University of Artificial Intelligence , UAE; School of Artificial Intelligence , Jilin University , China"},{"name":"Zhiqiang Xu","slug":"zhiqiang-xu","given":"Zhiqiang","family":"Xu","affiliation":"Mohamed bin Zayed University of Artificial Intelligence , UAE"}],"abstract":"Evolution Strategies (ES) have emerged as a competitive alternative for model-free reinforcement learning, showcasing exemplary performance in tasks like Mujoco and Atari. Notably, they shine in scenarios with imperfect reward functions, making them invaluable for real-world applications where dense reward signals may be elusive. Yet, an inherent assumption in ES—that all input features are task-relevant—poses challenges, especially when confronted with irrelevant features common in real-world problems. This work scrutinizes this limitation, particularly focusing on the Natural Evolution Strategies (NES) variant. We propose NESHT, a novel approach that integrates Hard-Thresholding (HT) with NES to champion sparsity, ensuring only pertinent features are employed. Backed by rigorous analysis and empirical tests, NESHT demonstrates its promise in mitigating the pitfalls of irrelevant features and shines in complex decision-making problems like noisy Mujoco and Atari tasks. Our code is available at https://github.com/cangcn/NES-HT.","doi":"10.24963/ijcai.2024/441","url":"https://ijcai.org/proceedings/2024/441","pdf":"https://ijcai.org/proceedings/2024/0441.pdf","markdown":"https://ijcai.org/proceedings/2024/441.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/441","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Machine Learning","track":"Main Track","number":441,"pages":{"first":3989,"last":3997},"keywords":["Machine Learning: ML: Feature extraction, selection and dimensionality reduction","Machine Learning: ML: Learning sparse models","Machine Learning: ML: Optimization","Machine Learning: ML: Evolutionary learning"],"videos":[],"references":[{"position":1,"raw":"[Bellemare et al., 2013] Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, 2013.","title":"The arcade learning environment: An evaluation platform for general agents","authors":"Yavar Marc G Bellemare, Joel Naddaf, Michael Veness, Bowling","year":"2013","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Blalock et al., 2020] Davis W. Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John V. Guttag. What is the state of neural network pruning? In Proceedings of Machine Learning and Systems 2020, MLSys 2020, Austin, TX, USA, March 2-4, 2020. mlsys.org, 2020.","title":"What is the state of neural network pruning?","authors":"Davis W Blalock, Jose, Javier Gonzalez Ortiz, Jonathan Frankle, John V Guttag","year":"2020","venue":"Proceedings of Machine Learning and Systems 2020, MLSys 2020","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Blumensath and Davies, 2009] Thomas Blumensath and Mike E Davies. Iterative hard thresholding for compressed sensing. Applied and computational harmonic analysis, 27(3):265-274, 2009.","title":"Iterative hard thresholding for compressed sensing","authors":"Thomas Blumensath, Mike E Davies","year":"2009","venue":"Applied and computational harmonic analysis","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2023] Jou-An Chen, Wei Niu, Bin Ren, Yanzhi Wang, and Xipeng Shen. Survey: Exploiting data redun- dancy for optimization of deep learning. ACM Comput. Surv., 55(10):212:1-212:38, 2023.","title":"Survey: Exploiting data redundancy for optimization of deep learning","authors":"Jou-An Chen, Wei Niu, Bin Ren, Yanzhi Wang, Xipeng Shen","year":"2023","venue":"ACM Comput. Surv","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Deleu and Bengio, 2021] Tristan Deleu and Yoshua Bengio. Structured sparsity inducing adaptive optimizers for deep learning. arXiv preprint arXiv:2102.03869, 2021.","title":"Structured sparsity inducing adaptive optimizers for deep learning","authors":"Bengio ; Deleu, Tristan Deleu, Yoshua Bengio","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Ermoliev and Norkin, 1995] Yuri M Ermoliev and Vladimir Ivanovich Norkin. On nonsmooth problems of stochastic systems optimization. 1995. [Franc ¸ois-Lavet et al., 2015]","title":"On nonsmooth problems of stochastic systems optimization","authors":"M Yuri, Vladimir Ermoliev, Ivanovich Norkin","year":"1995","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"Vincent Franc ¸ois-Lavet, Raphaël Fonteneau, and Damien Ernst. How to discount deep reinforcement learning: Towards new dynamic strategies. CoRR, abs/1512.02011, 2015.","title":"How to discount deep reinforcement learning: Towards new dynamic strategies","authors":"Vincent Franc ¸ois-Lavet, Raphaël Fonteneau, Damien Ernst","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Fujimoto et al., 2018] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pages 1587-1596. PMLR, 2018.","title":"Addressing function approximation error in actor-critic methods","authors":"Scott Fujimoto, Herke Hoof, David Meger","year":"2018","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Garg and Khandekar, 2009] Rahul Garg and Rohit Khan- dekar. Gradient descent with sparsification: an iterative algorithm for sparse recovery with restricted isometry prop- erty. In Proceedings of the 26th Annual International Con- ference on Machine Learning, ICML 2009, Montreal, Que- bec, Canada, June 14-18, 2009, volume 382 of ACM In- ternational Conference Proceeding Series, pages 337-344. ACM, 2009.","title":"Gradient descent with sparsification: an iterative algorithm for sparse recovery with restricted isometry property","authors":"Rahul Garg, Rohit Khandekar","year":"2009","venue":"Proceedings of the 26th Annual International Conference on Machine Learning, ICML 2009","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Grooten et al., 2023a] Bram Grooten, Ghada Sokar, Shib- hansh Dohare, Elena Mocanu, Matthew E. Taylor, Mykola Pechenizkiy, and Decebal Constantin Mocanu. Automatic noise filtering with dynamic sparse training in deep rein- forcement learning. In Proceedings of the 2023 Interna- tional Conference on Autonomous Agents and Multiagent Systems, AAMAS 2023, London, United Kingdom, 29 May 2023 -2 June 2023, pages 1932-1941. ACM, 2023.","title":"Automatic noise filtering with dynamic sparse training in deep reinforcement learning","authors":"Grooten","year":"2023","venue":"Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2023","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Grooten et al., 2023b] Bram Grooten, Ghada Sokar, Shib- hansh Dohare, Elena Mocanu, Matthew E. Taylor, Mykola Pechenizkiy, and Decebal Constantin Mocanu. Automatic noise filtering with dynamic sparse training in deep rein- forcement learning. The 22nd International Conference on Autonomous Agents and Multiagent Systems (AAMAS), 2023. URL: https://arxiv.org/abs/2302.06548.","title":"Automatic noise filtering with dynamic sparse training in deep reinforcement learning","authors":"Grooten","year":"2023","venue":"The 22nd International Conference on Autonomous Agents and Multiagent Systems (AAMAS)","doi":null,"url":"https://arxiv.org/abs/2302.06548","internal_url":null},{"position":12,"raw":"[Hubara et al., 2017] Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Quan- tized neural networks: Training neural networks with low precision weights and activations. J. Mach. Learn. Res., 18:187:1-187:30, 2017.","title":"Quantized neural networks: Training neural networks with low precision weights and activations","authors":"Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, Yoshua Bengio","year":"2017","venue":"J. Mach. Learn. Res","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li and Tang, 2015] Zechao Li and Jinhui Tang. Unsuper- vised feature selection via nonnegative spectral analysis and redundancy control. IEEE Trans. Image Process., 24(12):5343-5355, 2015.","title":"Unsupervised feature selection via nonnegative spectral analysis and redundancy control","authors":"Li, Tang, Zechao Li, Jinhui Tang","year":"2015","venue":"IEEE Trans. Image Process","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lillicrap et al., 2015] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous con- trol with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015.","title":"Continuous control with deep reinforcement learning","authors":"Jonathan J Timothy P Lillicrap, Alexander Hunt, Nicolas Pritzel, Tom Heess, Yuval Erez, David Tassa, Daan Silver, Wierstra","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Majid et al., 2021] Amjad Yousef Majid, Serge Saaybi, Tomas van Rietbergen, Vincent Franc ¸ois-Lavet, R. Venkate- sha Prasad, and Chris J. M. Verhoeven. Deep reinforcement learning versus evolution strategies: A comparative survey. CoRR, abs/2110.01411, 2021.","title":"Deep reinforcement learning versus evolution strategies: A comparative survey","authors":"Amjad Yousef Majid, Serge Saaybi, Tomas Van Rietbergen, Vincent Franc ¸ois-Lavet, R Venkatesha, Chris J M Prasad, Verhoeven","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Mania et al., 2018] Horia Mania, Aurelia Guy, and Ben- jamin Recht. Simple random search of static linear policies is competitive for reinforcement learning. In Advances in Neural Information Processing Systems 31: Annual Con- ference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pages 1805-1814, 2018.","title":"Simple random search of static linear policies is competitive for reinforcement learning","authors":"Horia Mania, Aurelia Guy, Benjamin Recht","year":"2018","venue":"Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Metel, 2023] Michael R Metel. Sparse training with lips- chitz continuous loss functions and a weighted group l 0 - norm constraint. Journal of Machine Learning Research, 24(103):1-44, 2023.","title":"Sparse training with lipschitz continuous loss functions and a weighted group l 0norm constraint","authors":"Michael R Metel","year":"2023","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Belle- mare, Alex Graves, Martin A. Riedmiller, Andreas Fid- jeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nat., 518(7540):529-533, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin A Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, Demis Hassabis","year":"2015","venue":"Nat","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Nguyen et al., 2017] Nam Nguyen, Deanna Needell, and Tina Woolf. Linear convergence of stochastic iterative greedy algorithms with sparse constraints. IEEE Trans. Inf. Theory, 63(11):6869-6895, 2017.","title":"Linear convergence of stochastic iterative greedy algorithms with sparse constraints","authors":"Nguyen","year":"2017","venue":"IEEE Trans. Inf. Theory","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Qian and Yu, 2021] Hong Qian and Yang Yu. Derivative- free reinforcement learning: a review. Frontiers Comput. Sci., 15(6):156336, 2021.","title":"Derivativefree reinforcement learning: a review","authors":"Yu ; ] Qian, Yang Hong Qian, Yu","year":"2021","venue":"Frontiers Comput. Sci","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Rockafellar, 1976] R Tyrrell Rockafellar. Monotone opera- tors and the proximal point algorithm. SIAM journal on control and optimization, 14(5):877-898, 1976.","title":"Monotone operators and the proximal point algorithm","authors":"Rockafellar Tyrrell","year":"1976","venue":"SIAM journal on control and optimization","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Salimans et al., 2017] Tim Salimans, Jonathan Ho, Xi Chen, and Ilya Sutskever. Evolution strategies as a scalable alter- native to reinforcement learning. CoRR, abs/1703.03864, 2017.","title":"Evolution strategies as a scalable alternative to reinforcement learning","authors":"Tim Salimans, Jonathan Ho, Xi Chen, Ilya Sutskever","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Schulman et al., 2015] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pages 1889-1897. PMLR, 2015.","title":"Trust region policy optimization","authors":"John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, Philipp Moritz","year":"2015","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Sokar et al., 2022] Ghada Sokar, Elena Mocanu, Dece- bal Constantin Mocanu, Mykola Pechenizkiy, and Peter Stone. Dynamic sparse training for deep reinforcement learning. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI 2022, Vi- enna, Austria, 23-29 July 2022, pages 3437-3443. ijcai.org, 2022.","title":"Dynamic sparse training for deep reinforcement learning","authors":"Ghada Sokar, Elena Mocanu, Constantin Mocanu, Mykola Pechenizkiy, Peter Stone","year":"2022","venue":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI 2022","doi":null,"url":null,"internal_url":"/proceedings/2022/477"},{"position":25,"raw":"[Such et al., 2017] Felipe Petroski Such, Vashisht Madhavan, Edoardo Conti, Joel Lehman, Kenneth O. Stanley, and Jeff Clune. Deep neuroevolution: Genetic algorithms are a competitive alternative for training deep neural networks for reinforcement learning. CoRR, abs/1712.06567, 2017.","title":"Deep neuroevolution: Genetic algorithms are a competitive alternative for training deep neural networks for reinforcement learning","authors":"Felipe Petroski Such, Vashisht Madhavan, Edoardo Conti, Joel Lehman, Kenneth O Stanley, Jeff Clune","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Todorov et al., 2012] Emanuel Todorov, Tom Erez, and Yu- val Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pages 5026-5033. IEEE, 2012.","title":"Mujoco: A physics engine for model-based control","authors":"Emanuel Todorov, Tom Erez, Yuval Tassa","year":"2012","venue":"IEEE/RSJ international conference on intelligent robots and systems","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wierstra et al., 2014] Daan Wierstra, Tom Schaul, Tobias Glasmachers, Yi Sun, Jan Peters, and Jürgen Schmidhu- ber. Natural evolution strategies. J. Mach. Learn. Res., 15(1):949-980, 2014.","title":"Natural evolution strategies","authors":"Daan Wierstra, Tom Schaul, Tobias Glasmachers, Yi Sun, Jan Peters, Jürgen Schmidhuber","year":"2014","venue":"J. Mach. Learn. Res","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Xu et al., 2019] Yi Xu, Rong Jin, and Tianbao Yang. Non- asymptotic analysis of stochastic methods for non-smooth non-convex regularized problems. Advances in Neural Information Processing Systems, 32, 2019.","title":"Nonasymptotic analysis of stochastic methods for non-smooth non-convex regularized problems","authors":"Yi Xu, Rong Jin, Tianbao Yang","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Yu et al., 1992] Ermoliev Yu, V Norkin, and R Wets. The minimization of discontinuous functions: Mollifier sub- gradients. Technical report, Working Paper, International Institute for Applied Systems Analysis . . . , 1992.","title":"The minimization of discontinuous functions: Mollifier subgradients","authors":"Yu","year":"1992","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}