{"type":"ScholarlyArticle","title":"Pruning of Deep Spiking Neural Networks through Gradient Rewiring","authors":[{"name":"Yanqi Chen","slug":"yanqi-chen","given":"Yanqi","family":"Chen","affiliation":""},{"name":"Zhaofei Yu","slug":"zhaofei-yu","given":"Zhaofei","family":"Yu","affiliation":""},{"name":"Wei Fang","slug":"wei-fang","given":"Wei","family":"Fang","affiliation":""},{"name":"Tiejun Huang","slug":"tiejun-huang","given":"Tiejun","family":"Huang","affiliation":""},{"name":"Yonghong Tian","slug":"yonghong-tian","given":"Yonghong","family":"Tian","affiliation":""}],"abstract":"Spiking Neural Networks (SNNs) have been attached great importance due to their biological plausibility and high energy-efficiency on neuromorphic chips. As these chips are usually resource-constrained, the compression of SNNs is thus crucial along the road of practical use of SNNs. Most existing methods directly apply pruning approaches in artificial neural networks (ANNs) to SNNs, which ignore the difference between ANNs and SNNs, thus limiting the performance of the pruned SNNs. Besides, these methods are only suitable for shallow SNNs. In this paper, inspired by synaptogenesis and synapse elimination in the neural system, we propose gradient rewiring (Grad R), a joint learning algorithm of connectivity and weight for SNNs, that enables us to seamlessly optimize network structure without retraining. Our key innovation is to redefine the gradient to a new synaptic parameter, allowing better exploration of network structures by taking full advantage of the competition between pruning and regrowth of connections. The experimental results show that the proposed method achieves minimal loss of SNNs' performance on MNIST and CIFAR-10 datasets so far. Moreover, it reaches a ~3.5% accuracy loss under unprecedented 0.73% connectivity, which reveals remarkable structure refining capability in SNNs. Our work suggests that there exists extremely high redundancy in deep SNNs. Our codes are available at https://github.com/Yanqi-Chen/Gradient-Rewiring.","doi":"10.24963/ijcai.2021/236","url":"https://ijcai.org/proceedings/2021/236","pdf":"https://ijcai.org/proceedings/2021/0236.pdf","markdown":"https://ijcai.org/proceedings/2021/236.md","bibtex":"https://ijcai.org/proceedings/2021/bibtex/236","volume":{"slug":"2021","title":"Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2021","url":"https://ijcai.org/proceedings/2021/","year":2021,"location":"Montreal","dates":"19-27 August 2021","editors":"Prof. Zhi-Hua Zhou","isbn":"978-0-9992411-9-6","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2021"},"section":"Humans and AI","track":"Main Track","number":236,"pages":{"first":1713,"last":1721},"keywords":["Humans and AI: Cognitive Modeling","Humans and AI: Brain Sciences","Machine Learning: Classification"],"videos":[],"references":[],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}