{"type":"ScholarlyArticle","title":"Multi-Level Firing with Spiking DS-ResNet: Enabling Better and Deeper Directly-Trained Spiking Neural Networks","authors":[{"name":"Lang Feng","slug":"lang-feng","given":"Lang","family":"Feng","affiliation":"College of Computer Science and Technology , Zhejiang University , Hangzhou , China"},{"name":"Qianhui Liu","slug":"qianhui-liu","given":"Qianhui","family":"Liu","affiliation":"College of Computer Science and Technology , Zhejiang University , Hangzhou , China"},{"name":"Huajin Tang","slug":"huajin-tang","given":"Huajin","family":"Tang","affiliation":"College of Computer Science and Technology , Zhejiang University , Hangzhou , China; Zhejiang Lab , Hangzhou , China"},{"name":"De Ma","slug":"de-ma","given":"De","family":"Ma","affiliation":"College of Computer Science and Technology , Zhejiang University , Hangzhou , China"},{"name":"Gang Pan","slug":"gang-pan","given":"Gang","family":"Pan","affiliation":"College of Computer Science and Technology , Zhejiang University , Hangzhou , China; Zhejiang Lab , Hangzhou , China"}],"abstract":"Spiking neural networks (SNNs) are bio-inspired neural networks with asynchronous discrete and sparse characteristics, which have increasingly manifested their superiority in low energy consumption. Recent research is devoted to utilizing spatio-temporal information to directly train SNNs by backpropagation. However, the binary and non-differentiable properties of spike activities force directly trained SNNs to suffer from serious gradient vanishing and network degradation, which greatly limits the performance of directly trained SNNs and prevents them from going deeper. In this paper, we propose a multi-level firing (MLF) method based on the existing spatio-temporal back propagation (STBP) method, and spiking dormant-suppressed residual network (spiking DS-ResNet). MLF enables more efficient gradient propagation and the incremental expression ability of the neurons. Spiking DS-ResNet can efficiently perform identity mapping of discrete spikes, as well as provide a more suitable connection for gradient propagation in deep SNNs. With the proposed method, our model achieves superior performances on a non-neuromorphic dataset and two neuromorphic datasets with much fewer trainable parameters and demonstrates the great ability to combat the gradient vanishing and degradation problem in deep SNNs.","doi":"10.24963/ijcai.2022/343","url":"https://ijcai.org/proceedings/2022/343","pdf":"https://ijcai.org/proceedings/2022/0343.pdf","markdown":"https://ijcai.org/proceedings/2022/343.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/343","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Humans and AI","track":"Main Track","number":343,"pages":{"first":2471,"last":2477},"keywords":["Humans and AI: Cognitive Modeling","Humans and AI: Cognitive Systems"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986597","length_sec":85,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984740","length_sec":360,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Amir et al., 2017] Arnon Amir, Brian Taba, David Berg, Timothy Melano, Jeffrey McKinstry, Carmelo Di Nolfo, Tapan Nayak, Alexander Andreopoulos, Guillaume Gar- reau, Marcela Mendoza, Jeff Kusnitz, Michael Debole, Steve Esser, Tobi Delbruck, Myron Flickner, and Dhar- mendra Modha. A low power, fully event-based gesture recognition system. In CVPR, pages 7243-7252, 2017.","title":"A low power, fully event-based gesture recognition system","authors":"Arnon Amir, Brian Taba, David Berg, Timothy Melano, Jeffrey Mckinstry, Carmelo Di Nolfo, Tapan Nayak, Alexander Andreopoulos, Guillaume Garreau, Marcela Mendoza, Jeff Kusnitz, Michael Debole, Steve Esser, Tobi Delbruck, Myron Flickner, Dharmendra Modha","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Diehl and Cook, 2015] Peter U. Diehl and Matthew Cook. Unsupervised learning of digit recognition using spike- timing-dependent plasticity. Frontiers in Computational Neuroscience, 9:99, 2015.","title":"Unsupervised learning of digit recognition using spiketiming-dependent plasticity","authors":"U Peter, Matthew Diehl, Cook","year":"2015","venue":"Frontiers in Computational Neuroscience","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Fang et al., 2020] Haowen Fang, Amar Shrestha, Ziyi Zhao, and Qinru Qiu. Exploiting neuron and synapse filter dy- namics in spatial temporal learning of deep spiking neural network. In IJCAI, pages 2799-2806, 2020.","title":"Exploiting neuron and synapse filter dynamics in spatial temporal learning of deep spiking neural network","authors":"Fang","year":"2020","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2020/388"},{"position":4,"raw":"[Gu et al., 2019] Pengjie Gu, Rong Xiao, Gang Pan, and Huajin Tang. STCA: Spatio-temporal credit assignment with delayed feedback in deep spiking neural networks. In IJCAI, pages 1366-1372, 2019.","title":"STCA: Spatio-temporal credit assignment with delayed feedback in deep spiking neural networks","authors":"Gu","year":"2019","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2019/189"},{"position":5,"raw":"[Han et al., 2020] Bing Han, Gopalakrishnan Srinivasan, and Kaushik Roy. RMP-SNN: Residual membrane po- tential neuron for enabling deeper high-accuracy and low- latency spiking neural network. In CVPR, volume 1, pages 13555-13564, 2020.","title":"RMP-SNN: Residual membrane potential neuron for enabling deeper high-accuracy and lowlatency spiking neural network","authors":"Han","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[He et al., 2020] Weihua He, YuJie Wu, Lei Deng, Guoqi Li, Haoyu Wang, Yang Tian, Wei Ding, Wenhui Wang, and Yuan Xie. Comparing SNNs and RNNs on neuromorphic vision datasets: similarities and differences. Neural Net- works, 132:108-120, 2020.","title":"Comparing SNNs and RNNs on neuromorphic vision datasets: similarities and differences","authors":"Weihua He, Yujie Wu, Lei Deng, Guoqi Li, Haoyu Wang, Yang Tian, Wei Ding, Wenhui Wang, Yuan Xie","year":"2020","venue":"Neural Networks","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hu et al., 2021] Yangfan Hu, Huajin Tang, and Gang Pan. Spiking deep residual networks. IEEE Transactions on Neural Networks and Learning Systems, pages 1-6, 2021.","title":"Spiking deep residual networks","authors":"Hu","year":"2021","venue":"IEEE Transactions on Neural Networks and Learning Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Ioffe and Szegedy, 2015] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep net- work training by reducing internal covariate shift. In ICML, pages 448-456, 2015.","title":"Batch normalization: Accelerating deep network training by reducing internal covariate shift","authors":"Szegedy Ioffe, Sergey Ioffe, Christian Szegedy","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kugele et al., 2020] Alexander Kugele, Thomas Pfeil, Michael Pfeiffer, and Elisabetta Chicca. Efficient process- ing of spatio-temporal data streams with spiking neural networks. Frontiers in Neuroscience, 14:439, 2020.","title":"Efficient processing of spatio-temporal data streams with spiking neural networks","authors":"Alexander Kugele, Thomas Pfeil, Michael Pfeiffer, Elisabetta Chicca","year":"2020","venue":"Frontiers in Neuroscience","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lee et al., 2016] Jun Haeng Lee, Tobi Delbruck, and Michael Pfeiffer. Training deep spiking neural networks using backpropagation. Frontiers in Neuroscience, 10:508, 2016.","title":"Training deep spiking neural networks using backpropagation","authors":"Lee","year":"2016","venue":"Frontiers in Neuroscience","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lee et al., 2020] Chankyu Lee, Syed Shakib Sarwar, Priyadarshini Panda, Gopalakrishnan Srinivasan, and Kaushik Roy. Enabling spike-based backpropagation for training deep neural network architectures. Frontiers in Neuroscience, 14:119, 2020.","title":"Enabling spike-based backpropagation for training deep neural network architectures","authors":"Lee","year":"2020","venue":"Frontiers in Neuroscience","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2017] Hongmin Li, Hanchao Liu, Xiangyang Ji, Guoqi Li, and Luping Shi. CIFAR10-DVS: An event- stream dataset for object classification. Frontiers in Neu- roscience, 11:309, 2017.","title":"CIFAR10-DVS: An eventstream dataset for object classification","authors":"Li","year":"2017","venue":"Frontiers in Neuroscience","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu et al., 2020] Qianhui Liu, Gang Pan, Haibo Ruan, Dong Xing, Qi Xu, and Huajin Tang. Unsupervised aer objec- t recognition based on multiscale spatio-temporal features and spiking neurons. IEEE Transactions on Neural Net- works and Learning Systems, 31(12):5300-5311, 2020.","title":"Unsupervised aer object recognition based on multiscale spatio-temporal features and spiking neurons","authors":"Qianhui Liu, Gang Pan, Haibo Ruan, Dong Xing, Qi Xu, Huajin Tang","year":"2020","venue":"IEEE Transactions on Neural Networks and Learning Systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu et al., 2022] Qianhui Liu, Dong Xing, Lang Feng, Hua- jin Tang, and Gang Pan. Event-based multimodal spik- ing neural network with attention mechanism. In ICASSP, pages 8922-8926, 2022.","title":"Event-based multimodal spiking neural network with attention mechanism","authors":"Qianhui Liu, Dong Xing, Lang Feng, Huajin Tang, Gang Pan","year":"2022","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Maass, 1997] Wolfgang Maass. Networks of spiking neu- rons: the third generation of neural network models. Neu- ral networks, 10(9):1659-1671, 1997.","title":"Networks of spiking neurons: the third generation of neural network models","authors":"Wolfgang Maass, Maass","year":"1997","venue":"Neural networks","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Neftci et al., 2019] Emre O. Neftci, Hesham Mostafa, and Friedemann Zenke. Surrogate gradient learning in spiking neural networks: Bringing the power of gradient-based op- timization to spiking neural networks. IEEE Signal Pro- cessing Magazine, 36(6):51-63, 2019.","title":"Surrogate gradient learning in spiking neural networks: Bringing the power of gradient-based optimization to spiking neural networks","authors":"O Emre, Hesham Neftci, Friedemann Mostafa, Zenke","year":"2019","venue":"IEEE Signal Processing Magazine","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Qi et al., 2018] Yu Qi, Jiangrong Shen, Yueming Wang, Huajin Tang, Hang Yu, Zhaohui Wu, and Gang Pan. Joint- ly learning network connections and link weights in spik- ing neural networks. In IJCAI, pages 1597-1603, 2018.","title":"Jointly learning network connections and link weights in spiking neural networks","authors":"Yu Qi, Jiangrong Shen, Yueming Wang, Huajin Tang, Hang Yu, Zhaohui Wu, Gang Pan","year":"2018","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2018/221"},{"position":19,"raw":"[Sengupta et al., 2019] Abhronil Sengupta, Yuting Ye, Robert Wang, Chiao Liu, and Kaushik Roy. Going deeper in spiking neural networks: VGG and residual architectures. Frontiers in Neuroscience, 13:95, 2019.","title":"Going deeper in spiking neural networks: VGG and residual architectures","authors":"Sengupta","year":"2019","venue":"Frontiers in Neuroscience","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wu et al., 2018] Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, and Luping Shi. Spatio-temporal backpropagation for training high-performance spiking neural networks. Fron- tiers in Neuroscience, 12:331, 2018.","title":"Spatio-temporal backpropagation for training high-performance spiking neural networks","authors":"Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, Luping Shi","year":"2018","venue":"Frontiers in Neuroscience","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wu et al., 2019] Yujie Wu, Lei Deng, Lei Deng, Jun Zhu, Yuan Xie, and Yuan Xie. Direct training for spiking neural networks: Faster, larger, better. In AAAI, volume 33, pages 1311-1318, 2019.","title":"Direct training for spiking neural networks: Faster, larger, better","authors":"Yujie Wu, Lei Deng, Lei Deng, Jun Zhu, Yuan Xie, Yuan Xie","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wu et al., 2021] Hao Wu, Yueyi Zhang, Wenming Weng, Yongting Zhang, Zhiwei Xiong, Zheng-Jun Zha, Xiaoy- an Sun, and Feng Wu. Training spiking neural networks with accumulated spiking flow. In AAAI, volume 35, pages 10320-10328, 2021.","title":"Training spiking neural networks with accumulated spiking flow","authors":"Hao Wu, Yueyi Zhang, Wenming Weng, Yongting Zhang, Zhiwei Xiong, Zheng-Jun Zha, Xiaoyan Sun, Feng Wu","year":"2021","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yan et al., 2021] Zhanglu Yan, Jun Zhou, and Weng-Fai Wong. Near lossless transfer learning for spiking neural networks. In AAAI, volume 35, pages 10577-10584, 2021.","title":"Near lossless transfer learning for spiking neural networks","authors":"Yan","year":"2021","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zheng et al., 2021] Hanle Zheng, Yujie Wu, Lei Deng, Yi- fan Hu, and Guoqi Li. Going deeper with directly-trained larger spiking neural networks. In AAAI, volume 35, pages 11062-11070, 2021.","title":"Going deeper with directly-trained larger spiking neural networks","authors":"Hanle Zheng, Yujie Wu, Lei Deng, Yifan Hu, Guoqi Li","year":"2021","venue":"AAAI","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}