{"type":"ScholarlyArticle","title":"A Low Latency Adaptive Coding Spike Framework for Deep Reinforcement Learning","authors":[{"name":"Lang Qin","slug":"lang-qin","given":"Lang","family":"Qin","affiliation":"College of Computer Science and Technology , Zhejiang University , Hangzhou , China"},{"name":"Rui Yan","slug":"rui-yan","given":"Rui","family":"Yan","affiliation":"College of Computer Science , Zhejiang University of Technology , Hangzhou , China"},{"name":"Huajin Tang","slug":"huajin-tang","given":"Huajin","family":"Tang","affiliation":"College of Computer Science and Technology , Zhejiang University , Hangzhou , China; Zhejiang Lab , Hangzhou , China"}],"abstract":"In recent years, spiking neural networks (SNNs) have been used in reinforcement learning (RL) due to their low power consumption and event-driven features. However, spiking reinforcement learning (SRL), which suffers from fixed coding methods, still faces the problems of high latency and poor versatility. In this paper, we use learnable matrix multiplication to encode and decode spikes, improving the flexibility of the coders and thus reducing latency. Meanwhile, we train the SNNs using the direct training method and use two different structures for online and offline RL algorithms, which gives our model a wider range of applications. Extensive experiments have revealed that our method achieves optimal performance with ultra-low latency (as low as 0.8% of other SRL methods) and excellent energy efficiency (up to 5X the DNNs) in different algorithms and different environments.","doi":"10.24963/ijcai.2023/340","url":"https://ijcai.org/proceedings/2023/340","pdf":"https://ijcai.org/proceedings/2023/0340.pdf","markdown":"https://ijcai.org/proceedings/2023/340.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/340","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Humans and AI","track":"Main Track","number":340,"pages":{"first":3049,"last":3057},"keywords":["Humans and AI: HAI: Cognitive modeling","Robotics: ROB: Cognitive robotics","Machine Learning: ML: Deep reinforcement learning"],"videos":[],"references":[{"position":1,"raw":"[Arulkumaran et al., 2017] Kai Arulkumaran, Marc Peter Deisenroth, Miles Brundage, and Anil Anthony Bharath. A brief survey of deep reinforcement learning. CoRR, abs/1708.05866, 2017.","title":"A brief survey of deep reinforcement learning","authors":"Kai Arulkumaran, Marc Peter Deisenroth, Miles Brundage, Anil Anthony Bharath","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bellec et al., 2020] Guillaume Bellec, Franz Scherr, Anand Subramoney, Elias Hajek, Darjan Salaj, Robert Legen- stein, and Wolfgang Maass. A solution to the learning dilemma for recurrent networks of spiking neurons. Na- ture Communications, 11:1-15, 2020.","title":"A solution to the learning dilemma for recurrent networks of spiking neurons","authors":"Guillaume Bellec, Franz Scherr, Anand Subramoney, Elias Hajek, Darjan Salaj, Robert Legenstein, Wolfgang Maass","year":"2020","venue":"Nature Communications","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bellemare et al., 2013] Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. In Proceedings of the 24th International Joint Conference on Artificial Intelligence, pages 253-279, 2013.","title":"The arcade learning environment: An evaluation platform for general agents","authors":"Yavar Marc G Bellemare, Joel Naddaf, Michael Veness, Bowling","year":"2013","venue":"Proceedings of the 24th International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bing et al., 2018] Zhenshan Bing, Claus Meschede, Kai Huang, Guang Chen, Florian Rohrbein, Mahmoud Akl, and Alois Knoll. End to end learning of spiking neural network based on R-STDP for a lane keeping vehicle. In 2018 IEEE International Conference on Robotics and Automation, pages 1-8, 2018.","title":"End to end learning of spiking neural network based on R-STDP for a lane keeping vehicle","authors":"Bing","year":"2018","venue":"IEEE International Conference on Robotics and Automation","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Brockman et al., 2016] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. CoRR, abs/1606.01540, 2016.","title":"Openai gym","authors":"Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, Wojciech Zaremba","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen et al., 2022] Ding Chen, Peixi Peng, Tiejun Huang, and Yonghong Tian. Deep reinforcement learning with spiking q-learning. CoRR, abs/2201.09754, 2022.","title":"Deep reinforcement learning with spiking q-learning","authors":"Ding Chen, Peixi Peng, Tiejun Huang, Yonghong Tian","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Duan et al., 2016] Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In Pro- ceedings of the 33nd International Conference on Machine Learning, pages 1329-1338, 2016.","title":"Benchmarking deep reinforcement learning for continuous control","authors":"Yan Duan, Xi Chen, Rein Houthooft, John Schulman, Pieter Abbeel","year":"2016","venue":"Proceedings of the 33nd International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Frémaux et al., 2013] Nicolas Frémaux, Henning Sprekeler, and Wulfram Gerstner. Reinforcement learning using a continuous time actor-critic framework with spiking neu- rons. PLoS Computational Biology, 9:e1003024, 2013.","title":"Reinforcement learning using a continuous time actor-critic framework with spiking neurons","authors":"Frémaux","year":"2013","venue":"PLoS Computational Biology","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Fujimoto et al., 2018] Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In Proceedings of the 35th Inter- national Conference on Machine Learning, pages 1582- 1591, 2018.","title":"Addressing function approximation error in actor-critic methods","authors":"Scott Fujimoto, Herke Van Hoof, David Meger","year":"2018","venue":"Proceedings of the 35th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Fujimoto et al., 2019] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In Proceedings of the 36th Inter- national Conference on Machine Learning, pages 2052- 2062, 2019.","title":"Off-policy deep reinforcement learning without exploration","authors":"Scott Fujimoto, David Meger, Doina Precup","year":"2019","venue":"Proceedings of the 36th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Girshick, 2015] Ross Girshick. Fast R-CNN. In Pro- ceedings of the 2015 IEEE International Conference on Computer Vision, pages 1440-1448, 2015.","title":"Fast R-CNN","authors":"Ross Girshick","year":"2015","venue":"Proceedings of the 2015 IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hagenaars et al., 2021] Jesse J. Hagenaars, Federico Paredes-Vallés, and Guido de Croon. Self-supervised learning of event-based optical flow with spiking neural networks. In Advances in Neural Information Processing Systems 34, pages 7167-7179, 2021.","title":"Self-supervised learning of event-based optical flow with spiking neural networks","authors":"Jesse J Hagenaars, Federico Paredes-Vallés, Guido De Croon","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hu et al., 2021] Siyi Hu, Fengda Zhu, Xiaojun Chang, and Xiaodan Liang. Updet: Universal multi-agent RL via policy decoupling with transformers. In Proceedings of the 9th International Conference on Learning Representa- tions, 2021.","title":"Updet: Universal multi-agent RL via policy decoupling with transformers","authors":"Siyi Hu, Fengda Zhu, Xiaojun Chang, Xiaodan Liang","year":"2021","venue":"Proceedings of the 9th International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kingma and Welling, 2014] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In Proceedings of the 2nd International Conference on Learning Repre- sentations, 2014.","title":"Auto-encoding variational bayes","authors":"P Diederik, Max Kingma, Welling","year":"2014","venue":"Proceedings of the 2nd International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lahijanian et al., 2018] Morteza Lahijanian, Maria Svoren- ova, Akshay A Morye, Brian Yeomans, Dushyant Rao, Ingmar Posner, Paul Newman, Hadas Kress-Gazit, and Marta Kwiatkowska. Resource-performance tradeoff anal- ysis for mobile robots. IEEE Robotics and Automation Letters, 3:1840-1847, 2018.","title":"Resource-performance tradeoff analysis for mobile robots","authors":"Morteza Lahijanian, Maria Svorenova, A Akshay, Brian Morye, Dushyant Yeomans, Ingmar Rao, Paul Posner, Hadas Newman, Marta Kress-Gazit, Kwiatkowska","year":"2018","venue":"IEEE Robotics and Automation Letters","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Legenstein et al., 2008] Robert Legenstein, Dejan Pecevski, and Wolfgang Maass. A learning theory for reward- modulated spike-timing-dependent plasticity with appli- cation to biofeedback. PLoS Computational Biology, 4:e1000180, 2008.","title":"A learning theory for rewardmodulated spike-timing-dependent plasticity with application to biofeedback","authors":"Robert Legenstein, Dejan Pecevski, Wolfgang Maass","year":"2008","venue":"PLoS Computational Biology","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Li et al., 2021] Yuhang Li, Yufei Guo, Shanghang Zhang, Shikuang Deng, Yongqing Hai, and Shi Gu. Differentiable spike: Rethinking gradient-descent for training spiking neural networks. In Advances in Neural Information Processing Systems 34, pages 23426-23439, 2021.","title":"Differentiable spike: Rethinking gradient-descent for training spiking neural networks","authors":"Yuhang Li, Yufei Guo, Shanghang Zhang, Shikuang Deng, Yongqing Hai, Shi Gu","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Li et al., 2022] Wenshuo Li, Hanting Chen, Jianyuan Guo, Ziyang Zhang, and Yunhe Wang. Brain-inspired multi- layer perceptron with spiking neurons. In Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 783-793, 2022.","title":"Brain-inspired multilayer perceptron with spiking neurons","authors":"Li","year":"2022","venue":"Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Lillicrap et al., 2016] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In Proceedings of the 4th International Conference on Learning Representations, 2016.","title":"Continuous control with deep reinforcement learning","authors":"Jonathan J Timothy P Lillicrap, Alexander Hunt, Nicolas Pritzel, Tom Heess, Yuval Erez, David Tassa, Daan Silver, Wierstra","year":"2016","venue":"Proceedings of the 4th International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Liu et al., 2022] Guisong Liu, Wenjie Deng, Xiurui Xie, Li Huang, and Huajin Tang. Human-level control through directly trained deep spiking q -networks. IEEE Transac- tions on Cybernetics, pages 1-12, 2022.","title":"Human-level control through directly trained deep spiking q -networks","authors":"Guisong Liu, Wenjie Deng, Xiurui Xie, Li Huang, Huajin Tang","year":"2022","venue":"IEEE Transactions on Cybernetics","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Maass, 1997] Wolfgang Maass. Networks of spiking neu- rons: The third generation of neural network models. Neural Networks, 10:1659-1671, 1997.","title":"Networks of spiking neurons: The third generation of neural network models","authors":"Wolfgang Maass, Maass","year":"1997","venue":"Neural Networks","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidje- land, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518:529-533, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Neftci and Averbeck, 2019] Emre O. Neftci and Bruno B. Averbeck. Reinforcement learning in artificial and bio- logical systems. Nature Machine Intelligence, 1:133-143, 2019.","title":"Reinforcement learning in artificial and biological systems","authors":"O Emre, Bruno B Neftci, Averbeck","year":"2019","venue":"Nature Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Niroui et al., 2019] Farzad Niroui, Kaicheng Zhang, Zendai Kashino, and Goldie Nejat. Deep reinforcement learning robot for search and rescue applications: Exploration in unknown cluttered environments. IEEE Robotics and Automation Letters, 4:610-617, 2019.","title":"Deep reinforcement learning robot for search and rescue applications: Exploration in unknown cluttered environments","authors":"Farzad Niroui, Kaicheng Zhang, Zendai Kashino, Goldie Nejat","year":"2019","venue":"IEEE Robotics and Automation Letters","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Patel et al., 2019] Devdhar Patel, Hananel Hazan, Daniel J Saunders, Hava T Siegelmann, and Robert Kozma. Im- proved robustness of reinforcement learning policies upon conversion to spiking neuronal network platforms applied to atari breakout game. Neural Networks, 120:108-115, 2019.","title":"Improved robustness of reinforcement learning policies upon conversion to spiking neuronal network platforms applied to atari breakout game","authors":"Patel","year":"2019","venue":"Neural Networks","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Shrestha and Orchard, 2018] Sumit Bam Shrestha and Gar- rick Orchard. SLAYER: spike layer error reassignment in time. In Advances in Neural Information Processing Systems 31, pages 1419-1428, 2018.","title":"SLAYER: spike layer error reassignment in time","authors":"Sumit Bam, Shrestha, Garrick Orchard","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Silver et al., 2016] David Silver, Aja Huang, Chris J. Mad- dison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Ve- davyas Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy P. Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mas- tering the game of go with deep neural networks and tree search. Nature, 529:484-489, 2016.","title":"Mastering the game of go with deep neural networks and tree search","authors":"David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Vedavyas Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy P Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, Demis Hassabis","year":"2016","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Sutton et al., 1999] Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approx- imation. In Advances in Neural Information Processing Systems 12, pages 1057-1063, 1999.","title":"Policy gradient methods for reinforcement learning with function approximation","authors":"David Richard S Sutton, Satinder Mcallester, Yishay Singh, Mansour","year":"1999","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Tan et al., 2021] Weihao Tan, Devdhar Patel, and Robert Kozma. Strategy and benchmark for converting deep q- networks to event-driven spiking neural networks. In Proceedings of the 35th AAAI conference on artificial intelligence, pages 9816-9824, 2021.","title":"Strategy and benchmark for converting deep qnetworks to event-driven spiking neural networks","authors":"Tan","year":"2021","venue":"Proceedings of the 35th AAAI conference on artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Tang et al., 2020] Guangzhi Tang, Neelesh Kumar, and Konstantinos P Michmizos. Reinforcement co-learning of deep and spiking neural networks for energy-efficient mapless navigation with neuromorphic hardware. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 6090-6097, 2020.","title":"Reinforcement co-learning of deep and spiking neural networks for energy-efficient mapless navigation with neuromorphic hardware","authors":"Tang","year":"2020","venue":"IEEE/RSJ International Conference on Intelligent Robots and Systems","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Todorov et al., 2012] Emanuel Todorov, Tom Erez, and Yu- val Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026-5033, 2012.","title":"Mujoco: A physics engine for model-based control","authors":"Emanuel Todorov, Tom Erez, Yuval Tassa","year":"2012","venue":"IEEE/RSJ International Conference on Intelligent Robots and Systems","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Wu et al., 2018] Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, and Luping Shi. Spatio-temporal backpropagation for training high-performance spiking neural networks. Fron- tiers in Neuroscience, 12:331, 2018.","title":"Spatio-temporal backpropagation for training high-performance spiking neural networks","authors":"Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, Luping Shi","year":"2018","venue":"Frontiers in Neuroscience","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Wu et al., 2021] Jibin Wu, Yansong Chua, Malu Zhang, Guoqi Li, Haizhou Li, and Kay Chen Tan. A tandem learn- ing rule for effective training and rapid inference of deep spiking neural networks. IEEE Transactions on Neural Networks and Learning Systems, pages 1-15, 2021.","title":"A tandem learning rule for effective training and rapid inference of deep spiking neural networks","authors":"Jibin Wu, Yansong Chua, Malu Zhang, Guoqi Li, Haizhou Li, Kay Chen, Tan","year":"2021","venue":"IEEE Transactions on Neural Networks and Learning Systems","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Yuan et al., 2019] Mengwen Yuan, Xi Wu, Rui Yan, and Huajin Tang. Reinforcement learning in spiking neural networks with stochastic and deterministic synapses. Neu- ral Computation, 31:2368-2389, 2019.","title":"Reinforcement learning in spiking neural networks with stochastic and deterministic synapses","authors":"Mengwen Yuan, Xi Wu, Rui Yan, Huajin Tang","year":"2019","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Zhang et al., 2022] Duzhen Zhang, Tielin Zhang, Shuncheng Jia, and Bo Xu. Multiscale dynamic coding improved spiking actor network for reinforcement learning. In Proceedings of the 36th AAAI conference on artificial intelligence, pages 59-67, 2022.","title":"Multiscale dynamic coding improved spiking actor network for reinforcement learning","authors":"Zhang","year":"2022","venue":"Proceedings of the 36th AAAI conference on artificial intelligence","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}