{"type":"ScholarlyArticle","title":"Dynamic Sparse Training for Deep Reinforcement Learning","authors":[{"name":"Ghada Sokar","slug":"ghada-sokar","given":"Ghada","family":"Sokar","affiliation":"Eindhoven University of Technology , The Netherlands"},{"name":"Elena Mocanu","slug":"elena-mocanu","given":"Elena","family":"Mocanu","affiliation":"University of Twente , The Netherlands"},{"name":"Decebal Constantin Mocanu","slug":"decebal-constantin-mocanu","given":"Decebal Constantin","family":"Mocanu","affiliation":"Eindhoven University of Technology , The Netherlands; University of Twente , The Netherlands"},{"name":"Mykola Pechenizkiy","slug":"mykola-pechenizkiy","given":"Mykola","family":"Pechenizkiy","affiliation":"Eindhoven University of Technology , The Netherlands"},{"name":"Peter Stone","slug":"peter-stone","given":"Peter","family":"Stone","affiliation":"The University of Texas at Austin , Sony AI , United States"}],"abstract":"Deep reinforcement learning (DRL) agents are trained through trial-and-error interactions with the environment. This leads to a long training time for dense neural networks to achieve good performance. Hence, prohibitive computation and memory resources are consumed. Recently, learning efficient DRL agents has received increasing attention. Yet, current methods focus on accelerating inference time. In this paper, we introduce for the first time a dynamic sparse training approach for deep reinforcement learning to accelerate the training process. The proposed approach trains a sparse neural network from scratch and dynamically adapts its topology to the changing data distribution during training. Experiments on continuous control tasks show that our dynamic sparse agents achieve higher performance than the equivalent dense methods, reduce the parameter count and floating-point operations (FLOPs) by 50%, and have a faster learning speed that enables reaching the performance of dense agents with 40−50% reduction in the training steps.","doi":"10.24963/ijcai.2022/477","url":"https://ijcai.org/proceedings/2022/477","pdf":"https://ijcai.org/proceedings/2022/0477.pdf","markdown":"https://ijcai.org/proceedings/2022/477.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/477","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Machine Learning","track":"Main Track","number":477,"pages":{"first":3437,"last":3443},"keywords":["Machine Learning: Representation learning","Machine Learning: Deep Reinforcement Learning","Machine Learning: Learning Sparse Models"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986774","length_sec":81,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984917","length_sec":361,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Chaudhry et al., 2018] Arslan Chaudhry, Marc'Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem. In International Conference on Learning Representations, 2018.","title":"Efficient lifelong learning with a-gem","authors":"Arslan Chaudhry, Marc'aurelio Ranzato, Marcus Rohrbach, Mohamed Elhoseiny","year":"2018","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Denil et al., 2013] Misha Denil, Babak Shakibi, Laurent Dinh, Marc'Aurelio Ranzato, and Nando de Freitas. Pre- dicting parameters in deep learning. In Proceedings of the 26th International Conference on Neural Information Pro- cessing Systems-Volume 2, pages 2148-2156, 2013.","title":"Predicting parameters in deep learning","authors":"Misha Denil, Babak Shakibi, Laurent Dinh, Marc'aurelio Ranzato, Nando De Freitas","year":"2013","venue":"Proceedings of the 26th International Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Erdos et al., 1960] Paul Erdos, Alfréd Rényi, et al. On the evolution of random graphs. Publ. Math. Inst. Hung. Acad. Sci, 5(1):17-60, 1960.","title":"On the evolution of random graphs","authors":"Paul Erdos, Alfréd Rényi","year":"1960","venue":"Publ. Math. Inst. Hung. Acad. Sci","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Evci et al., 2020] Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In International Conference on Machine Learning, pages 2943-2952. PMLR, 2020.","title":"Rigging the lottery: Making all tickets winners","authors":"Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, Erich Elsen","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Frankle and Carbin, 2018] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2018.","title":"The lottery ticket hypothesis: Finding sparse, trainable neural networks","authors":"Carbin Frankle, Jonathan Frankle, Michael Carbin","year":"2018","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fujimoto et al., 2018] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International Conference on Machine Learning, pages 1587-1596. PMLR, 2018.","title":"Addressing function approximation error in actor-critic methods","authors":"Scott Fujimoto, Herke Hoof, David Meger","year":"2018","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Haarnoja et al., 2018a] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning, pages 1861-1870. PMLR, 2018.","title":"Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor","authors":"Haarnoja","year":"2018","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Haarnoja et al., 2018b] Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905, 2018.","title":"Soft actor-critic algorithms and applications","authors":"Haarnoja","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Heidrich-Meisner and Igel, 2009] Verena Heidrich-Meisner and Christian Igel. Neuroevolution strategies for episodic reinforcement learning. Journal of Algorithms, 64(4):152- 168, 2009. Special Issue: Reinforcement Learning.","title":"Neuroevolution strategies for episodic reinforcement learning","authors":"Igel Heidrich-Meisner, Verena Heidrich-Meisner, Christian Igel","year":"2009","venue":"Journal of Algorithms","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hoefler et al., 2021] Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in deep learning: Pruning and growth for efficient infer- ence and training in neural networks. Journal of Machine Learning Research, 22(241):1-124, 2021.","title":"Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks","authors":"Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, Alexandra Peste","year":"2021","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Jayakumar et al., 2020] Siddhant Jayakumar, Razvan Pas- canu, Jack Rae, Simon Osindero, and Erich Elsen. Top- kast: Top-k always sparse training. Advances in Neural Information Processing Systems, 33:20744-20754, 2020.","title":"Topkast: Top-k always sparse training","authors":"Siddhant Jayakumar, Razvan Pascanu, Jack Rae, Simon Osindero, Erich Elsen","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Jouppi et al., 2017] Norman P Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, et al. In-datacenter performance analysis of a tensor processing unit. In Proceedings of the 44th annual international sym- posium on computer architecture, pages 1-12, 2017.","title":"In-datacenter performance analysis of a tensor processing unit","authors":"Cliff Norman P Jouppi, Nishant Young, David Patil, Gaurav Patterson, Raminder Agrawal, Sarah Bajwa, Suresh Bates, Nan Bhatia, Al Boden, Borchers","year":"2017","venue":"Proceedings of the 44th annual international symposium on computer architecture","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lee et al., 2021] Juhyoung Lee, Sangyeob Kim, Sangjin Kim, Wooyoung Jo, and Hoi-Jun Yoo. Gst: Group- sparse training for accelerating deep reinforcement learn- ing. arXiv preprint arXiv:2101.09650, 2021.","title":"Gst: Groupsparse training for accelerating deep reinforcement learning","authors":"Lee","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu et al., 2020] Shiwei Liu, Decebal Constantin Mocanu, Amarsagar Reddy Ramapuram Matavalam, Yulong Pei, and Mykola Pechenizkiy. Sparse evolutionary deep learn- ing with over one million artificial neurons on com- modity hardware. Neural Computing and Applications, 33:2589-2604, 2020.","title":"Sparse evolutionary deep learning with over one million artificial neurons on commodity hardware","authors":"Shiwei Liu, Constantin Mocanu, Amarsagar Reddy Ramapuram, Yulong Matavalam, Mykola Pei, Pechenizkiy","year":"2020","venue":"Neural Computing and Applications","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu et al., 2021] Shiwei Liu, Decebal Constantin Mocanu, Yulong Pei, and Mykola Pechenizkiy. Selfish sparse rnn training. In Marina Meila and Tong Zhang, editors, Pro- ceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learn- ing Research, pages 6893-6904. PMLR, 18-24 Jul 2021.","title":"Selfish sparse rnn training","authors":"Shiwei Liu, Constantin Mocanu, Yulong Pei, Mykola Pechenizkiy","year":"2021","venue":"Proceedings of the 38th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Livne and Cohen, 2020] Dor Livne and Kobi Cohen. Pops: Policy pruning and shrinking for deep reinforcement learn- ing. IEEE Journal of Selected Topics in Signal Processing, 14(4):789-801, 2020.","title":"Pops: Policy pruning and shrinking for deep reinforcement learning","authors":"Cohen ; ] Dor Livne, Kobi Livne, Cohen","year":"2020","venue":"IEEE Journal of Selected Topics in Signal Processing","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Mocanu et al., 2018] Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature communications, 9(1):1-12, 2018.","title":"Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science","authors":"Mocanu","year":"2018","venue":"Nature communications","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Mocanu et al., 2021] Decebal Constantin Mocanu, Elena Mocanu, Tiago Pinto, Selima Curci, Phuong H Nguyen, Madeleine Gibescu, Damien Ernst, and Zita A Vale. Sparse training theory for scalable and efficient agents. In Proceedings of the 20th International Conference on Au- tonomous Agents and MultiAgent Systems, pages 34-38, 2021.","title":"Sparse training theory for scalable and efficient agents","authors":"Mocanu","year":"2021","venue":"Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Mostafa and Wang, 2019] Hesham Mostafa and Xin Wang. Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization. In Inter- national Conference on Machine Learning, pages 4646- 4655. PMLR, 2019.","title":"Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization","authors":"Hesham Mostafa, Xin Wang","year":"2019","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Shi et al., 2020] Qian Shi, Hak-Keung Lam, Chengbin Xuan, and Ming Chen. Adaptive neuro-fuzzy pid con- troller based on twin delayed deep deterministic policy gradient algorithm. Neurocomputing, 402:183-194, 2020.","title":"Adaptive neuro-fuzzy pid controller based on twin delayed deep deterministic policy gradient algorithm","authors":"Qian Shi, Hak-Keung Lam, Chengbin Xuan, Ming Chen","year":"2020","venue":"Neurocomputing","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Sokar et al., 2021] Ghada Sokar, Decebal Constantin Mo- canu, and Mykola Pechenizkiy. Spacenet: Make free space for continual learning. Neurocomputing, 439:1-11, 2021.","title":"Spacenet: Make free space for continual learning","authors":"Sokar","year":"2021","venue":"Neurocomputing","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Stanley and Miikkulainen, 2002] Kenneth O Stanley and Risto Miikkulainen. Evolving neural networks through augmenting topologies. Evolutionary computation, 10(2):99-127, 2002.","title":"Evolving neural networks through augmenting topologies","authors":"O Kenneth, Risto Stanley, Miikkulainen","year":"2002","venue":"Evolutionary computation","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Stanley, 2003] Kenneth O. Stanley. Evolving adaptive neu- ral networks with and without adaptive synapses. In In Proceeedings of the 2003 Congress on Evolutionary Com- putation (CEC 2003, pages 2557-2564. Press, 2003.","title":"Evolving adaptive neural networks with and without adaptive synapses","authors":"Kenneth O Stanley","year":"2003","venue":"Proceeedings of the 2003 Congress on Evolutionary Computation (CEC 2003","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Vinyals et al., 2019] Oriol Vinyals, Igor Babuschkin, Woj- ciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in star- craft ii using multi-agent reinforcement learning. Nature, 575(7782):350-354, 2019.","title":"Grandmaster level in starcraft ii using multi-agent reinforcement learning","authors":"Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev","year":"2019","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Vischer et al., 2021] Marc Aurel Vischer, Robert Tjarko Lange, and Henning Sprekeler. On lottery tickets and min- imal task representations in deep reinforcement learning. arXiv preprint arXiv:2105.01648, 2021.","title":"On lottery tickets and minimal task representations in deep reinforcement learning","authors":"Vischer","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wang et al., 2020] Hao-nan Wang, Ning Liu, Yi-yun Zhang, Da-wei Feng, Feng Huang, Dong-sheng Li, and Yi-ming Zhang. Deep reinforcement learning: a survey. Frontiers of Information Technology & Electronic Engi- neering, pages 1-19, 2020.","title":"Deep reinforcement learning: a survey","authors":"Ning Hao-Nan Wang, Yi-Yun Liu, Da-Wei Zhang, Feng Feng, Dong-Sheng Huang, Yi-Ming Li, Zhang","year":"2020","venue":"Frontiers of Information Technology & Electronic Engineering","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Whiteson and Stone, 2006] Shimon Whiteson and Peter Stone. Evolutionary function approximation for reinforce- ment learning. Journal of Machine Learning Research, 7:877-917, May 2006.","title":"Evolutionary function approximation for reinforcement learning","authors":"Stone Whiteson, Shimon Whiteson, Peter Stone","year":"2006","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Woo et al., 2020] Jong Ha Woo, Lei Wu, Jong-Bae Park, and Jae Hyung Roh. Real-time optimal power flow using twin delayed deep deterministic policy gradient algorithm. IEEE Access, 8:213611-213618, 2020.","title":"Real-time optimal power flow using twin delayed deep deterministic policy gradient algorithm","authors":"Jong Ha Woo, Lei Wu, Jong-Bae Park, Jae Hyung, Roh","year":"2020","venue":"IEEE Access","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Yu et al., 2019] Haonan Yu, Sergey Edunov, Yuandong Tian, and Ari S Morcos. Playing the lottery with rewards and multiple languages: lottery tickets in rl and nlp. arXiv preprint arXiv:1906.02768, 2019.","title":"Playing the lottery with rewards and multiple languages: lottery tickets in rl and nlp","authors":"Yu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhang et al., 2019] Hongjie Zhang, Zhuocheng He, and Jing Li. Accelerating the deep reinforcement learning with neural network compression. In 2019 International Joint Conference on Neural Networks (IJCNN), pages 1- 8. IEEE, 2019.","title":"Accelerating the deep reinforcement learning with neural network compression","authors":"Hongjie Zhang, Zhuocheng He, Jing Li","year":"2019","venue":"2019 International Joint Conference on Neural Networks (IJCNN)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}