{"type":"ScholarlyArticle","title":"Learnable Surrogate Gradient for Direct Training Spiking Neural Networks","authors":[{"name":"Shuang Lian","slug":"shuang-lian","given":"Shuang","family":"Lian","affiliation":"College of Computer Science and Technology , Zhejiang University"},{"name":"Jiangrong Shen","slug":"jiangrong-shen","given":"Jiangrong","family":"Shen","affiliation":"College of Computer Science and Technology , Zhejiang University"},{"name":"Qianhui Liu","slug":"qianhui-liu","given":"Qianhui","family":"Liu","affiliation":"Department of Electrical and Computer Engineering , National University of Singapore"},{"name":"Ziming Wang","slug":"ziming-wang","given":"Ziming","family":"Wang","affiliation":"College of Computer Science and Technology , Zhejiang University"},{"name":"Rui Yan","slug":"rui-yan","given":"Rui","family":"Yan","affiliation":"College of Computer Science and Technology , Zhejiang University of Technology"},{"name":"Huajin Tang","slug":"huajin-tang","given":"Huajin","family":"Tang","affiliation":"College of Computer Science and Technology , Zhejiang University; Zhejiang Lab"}],"abstract":"Spiking neural networks (SNNs) have increasingly drawn massive research attention due to biological interpretability and efficient computation. Recent achievements are devoted to utilizing the surrogate gradient (SG) method to avoid the dilemma of non-differentiability of spiking activity to directly train SNNs by backpropagation. However, the fixed width of the SG leads to gradient vanishing and mismatch problems, thus limiting the performance of directly trained SNNs. In this work, we propose a novel perspective to unlock the width limitation of SG, called the learnable surrogate gradient (LSG) method. The LSG method modulates the width of SG according to the change of the distribution of the membrane potentials, which is identified to be related to the decay factors based on our theoretical analysis. Then we introduce the trainable decay factors to implement the LSG method, which can optimize the width of SG automatically during training to avoid the gradient vanishing and mismatch problems caused by the limited width of SG. We evaluate the proposed LSG method on both image and neuromorphic datasets. Experimental results show that the LSG method can effectively alleviate the blocking of gradient propagation caused by the limited width of SG when training deep SNNs directly. Meanwhile, the LSG method can help SNNs achieve competitive performance on both latency and accuracy.","doi":"10.24963/ijcai.2023/335","url":"https://ijcai.org/proceedings/2023/335","pdf":"https://ijcai.org/proceedings/2023/0335.pdf","markdown":"https://ijcai.org/proceedings/2023/335.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/335","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Humans and AI","track":"Main Track","number":335,"pages":{"first":3002,"last":3010},"keywords":["Humans and AI: HAI: Brain sciences","Humans and AI: HAI: Cognitive modeling","Humans and AI: HAI: Cognitive systems"],"videos":[],"references":[{"position":1,"raw":"[Davies et al., 2018] Mike Davies, Narayan Srinivasa, Tsung-Han Lin, Gautham Chinya, Yongqiang Cao, Sri Harsha Choday, Georgios Dimou, Prasad Joshi, Nabil Imam, Shweta Jain, Yuyun Liao, Chit-Kwan Lin, Andrew Lines, Ruokun Liu, Deepak Mathaikutty, Steven McCoy, Arnab Paul, Jonathan Tse, Guruguhanathan Venkatara- manan, Yi-Hsin Weng, Andreas Wild, Yoonseok Yang, and Hong Wang. Loihi: A neuromorphic manycore pro- cessor with on-chip learning. IEEE Micro, 38(1):82-99, 2018.","title":"Loihi: A neuromorphic manycore processor with on-chip learning","authors":"Mike Davies, Narayan Srinivasa, Tsung-Han Lin, Gautham Chinya, Yongqiang Cao, Sri Harsha Choday, Georgios Dimou, Prasad Joshi, Nabil Imam, Shweta Jain, Yuyun Liao, Chit-Kwan Lin, Andrew Lines, Ruokun Liu, Deepak Mathaikutty, Steven Mccoy, Arnab Paul, Jonathan Tse, Guruguhanathan Venkataramanan, Yi-Hsin Weng, Andreas Wild, Yoonseok Yang, Hong Wang","year":"2018","venue":"IEEE Micro","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Deng et al., 2022] Shikuang Deng, Yuhang Li, Shanghang Zhang, and Shi Gu. Temporal efficient training of spiking neural network via gradient re-weighting. In International Conference on Learning Representations, 2022.","title":"Temporal efficient training of spiking neural network via gradient re-weighting","authors":"Deng","year":"2022","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Duan et al., 2022] Chaoteng Duan, Jianhao Ding, Shiyan Chen, Zhaofei Yu, and Tiejun Huang. Temporal effective batch normalization in spiking neural networks. In Ad- vances in Neural Information Processing Systems, 2022.","title":"Temporal effective batch normalization in spiking neural networks","authors":"Duan","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Fang et al., 2021a] Wei Fang, Zhaofei Yu, Yanqi Chen, Tiejun Huang, Timothée Masquelier, and Yonghong Tian. Deep residual learning in spiking neural networks. In Thirty-Fifth Conference on Neural Information Processing Systems, 2021.","title":"Deep residual learning in spiking neural networks","authors":"Fang","year":"2021","venue":"Thirty-Fifth Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Fang et al., 2021b] Wei Fang, Zhaofei Yu, Yanqi Chen, Timothée Masquelier, Tiejun Huang, and Yonghong Tian. Incorporating learnable membrane time constant to en- hance learning of spiking neural networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2661-2671, 2021.","title":"Incorporating learnable membrane time constant to enhance learning of spiking neural networks","authors":"Fang","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Feng et al., 2022] Lang Feng, Qianhui Liu, Huajin Tang, De Ma, and Gang Pan. Multi-level firing with spiking ds- resnet: Enabling better and deeper directly-trained spik- ing neural networks. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22, pages 2471-2477, 2022.","title":"Multi-level firing with spiking dsresnet: Enabling better and deeper directly-trained spiking neural networks","authors":"Lang Feng, Qianhui Liu, Huajin Tang, De Ma, Gang Pan","year":"2022","venue":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Garg et al., 2020] Isha Garg, Sayeed Shafayet Chowdhury, and Kaushik Roy. Dct-snn: Using dct to distribute spa- tial information over time for learning low-latency spiking neural networks. arXiv preprint arXiv:2010.01795, 2020.","title":"Dct-snn: Using dct to distribute spatial information over time for learning low-latency spiking neural networks","authors":"Isha Garg, Sayeed Shafayet Chowdhury, Kaushik Roy","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gu et al., 2019] Pengjie Gu, Rong Xiao, Gang Pan, and Huajin Tang. STCA: Spatio-Temporal Credit Assign- ment with Delayed Feedback in Deep Spiking Neural Net- works. In Proceedings of the Twenty-Eighth Interna- tional Joint Conference on Artificial Intelligence, IJCAI- 19, pages 1366-1372, 7 2019.","title":"STCA: Spatio-Temporal Credit Assignment with Delayed Feedback in Deep Spiking Neural Networks","authors":"Gu","year":"2019","venue":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19","doi":null,"url":null,"internal_url":"/proceedings/2019/189"},{"position":9,"raw":"[Guo et al., 2022a] Yufei Guo, Yuanpei Chen, Liwen Zhang, Xiaode Liu, Yinglei Wang, Xuhui Huang, and Zhe Ma. IM-loss: Information maximization loss for spiking neural networks. In Advances in Neural Information Processing Systems, 2022.","title":"IM-loss: Information maximization loss for spiking neural networks","authors":"Guo","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Guo et al., 2022b] Yufei Guo, Xinyi Tong, Yuanpei Chen, Liwen Zhang, Xiaode Liu, Zhe Ma, and Xuhui Huang. Recdis-snn: Rectifying membrane potential distribution for directly training spiking neural networks. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 326-335, 2022.","title":"Recdis-snn: Rectifying membrane potential distribution for directly training spiking neural networks","authors":"Guo","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Haeng et al., 2016] Lee Jun Haeng, Delbruck Tobi, and Pfeiffer Michael. Training deep spiking neural networks using backpropagation. Frontiers in Neuroscience, 10, 2016.","title":"Training deep spiking neural networks using backpropagation","authors":"Lee Jun Haeng, Delbruck Tobi, Pfeiffer Michael","year":"2016","venue":"Frontiers in Neuroscience","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hagenaars et al., 2021] Jesse Hagenaars, Federico Paredes- Vallés, and Guido De Croon. Self-supervised learn- ing of event-based optical flow with spiking neural net- works. Advances in Neural Information Processing Sys- tems, 34:7167-7179, 2021.","title":"Self-supervised learning of event-based optical flow with spiking neural networks","authors":"Jesse Hagenaars, Federico Paredes-Vallés, Guido De, Croon","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Han et al., 2020] Bing Han, Gopalakrishnan Srinivasan, and Kaushik Roy. Rmp-snn: Residual membrane potential neuron for enabling deeper high-accuracy and low-latency spiking neural network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13558-13567, 2020.","title":"Rmp-snn: Residual membrane potential neuron for enabling deeper high-accuracy and low-latency spiking neural network","authors":"Han","year":"2020","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hu et al., 2021] Yangfan Hu, Huajin Tang, and Gang Pan. Spiking deep residual networks. IEEE Transactions on Neural Networks and Learning Systems, pages 1-6, 2021.","title":"Spiking deep residual networks","authors":"Hu","year":"2021","venue":"IEEE Transactions on Neural Networks and Learning Systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ioffe and Szegedy, 2015] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep net- work training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.","title":"Batch normalization: Accelerating deep network training by reducing internal covariate shift","authors":"Szegedy Ioffe, Sergey Ioffe, Christian Szegedy","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Ju et al., 2020] Xiping Ju, Biao Fang, Rui Yan, Xiaoliang Xu, and Huajin Tang. An fpga implementation of deep spiking neural networks for low-power and fast classifica- tion. Neural computation, 32(1):182-204, 2020.","title":"An fpga implementation of deep spiking neural networks for low-power and fast classification","authors":"Xiping Ju, Biao Fang, Rui Yan, Xiaoliang Xu, Huajin Tang","year":"2020","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Kim et al., 2020] Seijoon Kim, Seongsik Park, Byunggook Na, and Sungroh Yoon. Spiking-yolo: Spiking neural net- work for energy-efficient object detection. In Proceed- ings of the AAAI Conference on Artificial Intelligence, vol- ume 34, pages 11270-11277, 2020.","title":"Spiking-yolo: Spiking neural network for energy-efficient object detection","authors":"Kim","year":"2020","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Kundu et al., 2021] Souvik Kundu, Gourav Datta, Massoud Pedram, and Peter A Beerel. Spike-thrift: Towards energy- efficient deep spiking neural networks by limiting spiking activity via attention-guided compression. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3953-3962, 2021.","title":"Spike-thrift: Towards energyefficient deep spiking neural networks by limiting spiking activity via attention-guided compression","authors":"Souvik Kundu, Gourav Datta, Massoud Pedram, Peter A Beerel","year":"2021","venue":"Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Leng et al., 2022] Luziwei Leng, Kaiwei Che, Kaixuan Zhang, Jianguo Zhang, Qinghu Meng, Jie Cheng, Qing- hai Guo, and Jianxing Liao. Differentiable hierarchical and surrogate gradient search for spiking neural networks. In Advances in Neural Information Processing Systems, 2022.","title":"Differentiable hierarchical and surrogate gradient search for spiking neural networks","authors":"Leng","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Li et al., 2017] Hongmin Li, Hanchao Liu, Xiangyang Ji, Guoqi Li, and Luping Shi. CIFAR10-DVS: An event- stream dataset for object classification. Frontiers in Neu- roscience, 11:309, 2017.","title":"CIFAR10-DVS: An eventstream dataset for object classification","authors":"Li","year":"2017","venue":"Frontiers in Neuroscience","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Li et al., 2021a] Yuhang Li, Yufei Guo, Shanghang Zhang, Shikuang Deng, Yongqing Hai, and Shi Gu. Differen- tiable spike: Rethinking gradient-descent for training spik- ing neural networks. Advances in Neural Information Pro- cessing Systems, 34:23426-23439, 2021.","title":"Differentiable spike: Rethinking gradient-descent for training spiking neural networks","authors":"Li","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Li et al., 2021b] Yuhang Li, Yufei Guo, Shanghang Zhang, Shikuang Deng, Yongqing Hai, and Shi Gu. Differentiable spike: Rethinking gradient-descent for training spiking neural networks. In Advances in Neural Information Pro- cessing Systems, volume 34, pages 23426-23439, 2021.","title":"Differentiable spike: Rethinking gradient-descent for training spiking neural networks","authors":"Li","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Neftci et al., 2019] Emre O Neftci, Hesham Mostafa, and Friedemann Zenke. Surrogate gradient learning in spik- ing neural networks. IEEE Signal Processing Magazine, 36:61-63, 2019.","title":"Surrogate gradient learning in spiking neural networks","authors":"Hesham Emre O Neftci, Friedemann Mostafa, Zenke","year":"2019","venue":"IEEE Signal Processing Magazine","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Rathi and Roy, 2021] Nitin Rathi and Kaushik Roy. Diet- snn: A low-latency spiking neural network with direct input encoding and leakage and threshold optimization. IEEE Transactions on Neural Networks and Learning Sys- tems, pages 1-9, 2021.","title":"Dietsnn: A low-latency spiking neural network with direct input encoding and leakage and threshold optimization","authors":"Nitin Rathi, Kaushik Roy","year":"2021","venue":"IEEE Transactions on Neural Networks and Learning Systems","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Sengupta et al., 2019] Abhronil Sengupta, Yuting Ye, Robert Wang, Chiao Liu, and Kaushik Roy. Going deeper in spiking neural networks: Vgg and residual architectures. Frontiers in Neuroscience, 13:95, 2019.","title":"Going deeper in spiking neural networks: Vgg and residual architectures","authors":"Sengupta","year":"2019","venue":"Frontiers in Neuroscience","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Shrestha and Orchard, 2018] Sumit Bam Shrestha and Gar- rick Orchard. SLAYER: Spike layer error reassignment in time. In Advances in Neural Information Processing Sys- tems, pages 1412-1421, 2018.","title":"SLAYER: Spike layer error reassignment in time","authors":"Sumit Bam, Shrestha, Garrick Orchard","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wang et al., 2022a] Siqi Wang, Tee Hiang Cheng, and Meng-Hiot Lim. LTMD: Learning improvement of spik- ing neural networks with learnable thresholding neurons and moderate dropout. In Advances in Neural Information Processing Systems, 2022.","title":"LTMD: Learning improvement of spiking neural networks with learnable thresholding neurons and moderate dropout","authors":"Wang","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wang et al., 2022b] Ziming Wang, Shuang Lian, Yuhao Zhang, Xiaoxin Cui, Rui Yan, and Huajin Tang. To- wards lossless ann-snn conversion under ultra-low la- tency with dual-phase optimization. arXiv preprint arXiv:2205.07473, 2022.","title":"Towards lossless ann-snn conversion under ultra-low latency with dual-phase optimization","authors":"Wang","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wu et al., 2018] Yujie Wu, Deng Lei, Guoqi Li, Jun Zhu, and Luping Shi. Spatio-temporal backpropagation for training high-performance spiking neural networks. Fron- tiers in Neuroscience, 12:331, 2018.","title":"Spatio-temporal backpropagation for training high-performance spiking neural networks","authors":"Yujie Wu, Deng Lei, Guoqi Li, Jun Zhu, Luping Shi","year":"2018","venue":"Frontiers in Neuroscience","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wu et al., 2019] Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, Yuan Xie, and Luping Shi. Direct training for spiking neural networks: Faster, larger, better. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 1311-1318, 2019.","title":"Direct training for spiking neural networks: Faster, larger, better","authors":"Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, Yuan Xie, Luping Shi","year":"2019","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Wu et al., 2021] Jibin Wu, Yansong Chua, Malu Zhang, Guoqi Li, Haizhou Li, and Kay Chen Tan. A tandem learn- ing rule for effective training and rapid inference of deep spiking neural networks. IEEE Transactions on Neural Networks and Learning Systems, 2021.","title":"A tandem learning rule for effective training and rapid inference of deep spiking neural networks","authors":"Jibin Wu, Yansong Chua, Malu Zhang, Guoqi Li, Haizhou Li, Kay Chen, Tan","year":"2021","venue":"IEEE Transactions on Neural Networks and Learning Systems","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Yu et al., 2021] Qiang Yu, Chenxiang Ma, Shiming Song, Gaoyan Zhang, Jianwu Dang, and Kay Chen Tan. Con- structing accurate and efficient deep spiking neural net- works with double-threshold and augmented schemes. IEEE Transactions on Neural Networks and Learning Sys- tems, 33(4):1714-1726, 2021.","title":"Constructing accurate and efficient deep spiking neural networks with double-threshold and augmented schemes","authors":"Qiang Yu, Chenxiang Ma, Shiming Song, Gaoyan Zhang, Jianwu Dang, Kay Chen, Tan","year":"2021","venue":"IEEE Transactions on Neural Networks and Learning Systems","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zenke and Vogels, 2021] Friedemann Zenke and Tim P Vo- gels. The remarkable robustness of surrogate gradient learning for instilling complex function in spiking neural networks. Neural computation, 33(4):899-925, 2021.","title":"The remarkable robustness of surrogate gradient learning for instilling complex function in spiking neural networks","authors":"Friedemann Zenke, Tim P Vogels","year":"2021","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zhang and Li, 2020] Wenrui Zhang and Peng Li. Tem- poral spike sequence learning via backpropagation for deep spiking neural networks. arXiv preprint arXiv:2002.10085, 2020.","title":"Temporal spike sequence learning via backpropagation for deep spiking neural networks","authors":"Li ; ] Wenrui Zhang, Peng Zhang, Li","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Zheng et al., 2021] Hanle Zheng, Yujie Wu, Lei Deng, Yi- fan Hu, and Guoqi Li. Going deeper with directly-trained larger spiking neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 11062-11070, 2021.","title":"Going deeper with directly-trained larger spiking neural networks","authors":"Hanle Zheng, Yujie Wu, Lei Deng, Yifan Hu, Guoqi Li","year":"2021","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}