{"type":"ScholarlyArticle","title":"Spike Count Maximization for Neuromorphic Vision Recognition","authors":[{"name":"Jianxiong Tang","slug":"jianxiong-tang","given":"Jianxiong","family":"Tang","affiliation":"School of Computer Science and Engineering , Sun Yat-sen University , Guangzhou , China"},{"name":"Jian-Huang Lai","slug":"jian-huang-lai","given":"Jian-Huang","family":"Lai","affiliation":"School of Computer Science and Engineering , Sun Yat-sen University , Guangzhou , China; Guangdong Province Key Laboratory of Information Security Technology , Guangzhou , China; Key Laboratory of Machine Intelligence and Advanced Computing , Ministry of Education , China"},{"name":"Xiaohua Xie","slug":"xiaohua-xie","given":"Xiaohua","family":"Xie","affiliation":"School of Computer Science and Engineering , Sun Yat-sen University , Guangzhou , China; Guangdong Province Key Laboratory of Information Security Technology , Guangzhou , China; Key Laboratory of Machine Intelligence and Advanced Computing , Ministry of Education , China"},{"name":"Lingxiao Yang","slug":"lingxiao-yang","given":"Lingxiao","family":"Yang","affiliation":"School of Computer Science and Engineering , Sun Yat-sen University , Guangzhou , China; Guangdong Province Key Laboratory of Information Security Technology , Guangzhou , China; Key Laboratory of Machine Intelligence and Advanced Computing , Ministry of Education , China"}],"abstract":"Spiking Neural Networks (SNNs) are the promising models of neuromorphic vision recognition. The mean square error (MSE) and cross-entropy (CE) losses are widely applied to supervise the training of SNNs on neuromorphic datasets. However, the relevance between the output spike counts and predictions is not well modeled by the existing loss functions. This paper proposes a Spike Count Maximization (SCM) training approach for the SNN-based neuromorphic vision recognition model based on optimizing the output spike counts. The SCM is achieved by structural risk minimization (SRM) and a specially designed spike counting loss. The spike counting loss counts the output spikes of the SNN by using the L0-norm, and the SRM maximizes the distance between the margin boundaries of the classifier to ensure the generalization of the model. The SCM is non-smooth and non-differentiable, and we design a two-stage algorithm with fast convergence to solve the problem. Experiment results demonstrate that the SCM performs satisfactorily in most cases. Using the output spikes for prediction, the accuracies of SCM are 2.12%~16.50% higher than the popular training losses on the CIFAR10-DVS dataset. The code is available at https://github.com/TJXTT/SCM-SNN.","doi":"10.24963/ijcai.2023/473","url":"https://ijcai.org/proceedings/2023/473","pdf":"https://ijcai.org/proceedings/2023/0473.pdf","markdown":"https://ijcai.org/proceedings/2023/473.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/473","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Machine Learning","track":"Main Track","number":473,"pages":{"first":4253,"last":4261},"keywords":["Computer Vision: CV: Machine learning for vision","Machine Learning: ML: Classification"],"videos":[],"references":[{"position":1,"raw":"[Amir et al., 2017] Arnon Amir, Brian Taba, David Berg, Timothy Melano, Jeffrey McKinstry, Carmelo Di Nolfo, Tapan Nayak, Alexander Andreopoulos, Guillaume Gar- reau, Marcela Mendoza, et al. A low power, fully event- based gesture recognition system. In CVPR, 2017.","title":"A low power, fully eventbased gesture recognition system","authors":"Arnon Amir, Brian Taba, David Berg, Timothy Melano, Jeffrey Mckinstry, Carmelo Di Nolfo, Tapan Nayak, Alexander Andreopoulos, Guillaume Garreau, Marcela Mendoza","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Baldwin et al., 2022] Raymond Baldwin, Ruixu Liu, Mo- hammed Mutlaq Almatrafi, Vijayan K Asari, and Keigo Hirakawa. Time-ordered recent event (tore) volumes for event cameras. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1-1, 2022.","title":"Time-ordered recent event (tore) volumes for event cameras","authors":"Raymond Baldwin, Ruixu Liu, Mohammed Mutlaq Almatrafi, Keigo Vijayan K Asari, Hirakawa","year":"2022","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bi et al., 2019] Yin Bi, Aaron Chadha, Alhabib Abbas, Ei- rina Bourtsoulatze, and Yiannis Andreopoulos. Graph- based object classification for neuromorphic vision sens- ing. In ICCV, 2019.","title":"Graphbased object classification for neuromorphic vision sensing","authors":"Yin Bi, Aaron Chadha, Alhabib Abbas, Eirina Bourtsoulatze, Yiannis Andreopoulos","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bu et al., 2021] Tong Bu, Wei Fang, Jianhao Ding, PengLin Dai, Zhaofei Yu, and Tiejun Huang. Optimal ann-snn conversion for high-accuracy and ultra-low-latency spik- ing neural networks. In ICLR, 2021.","title":"Optimal ann-snn conversion for high-accuracy and ultra-low-latency spiking neural networks","authors":"Tong Bu, Wei Fang, Jianhao Ding, Penglin Dai, Zhaofei Yu, Tiejun Huang","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2021] Yanqi Chen, Zhaofei Yu, Wei Fang, Tiejun Huang, and Yonghong Tian. Pruning of deep spik- ing neural networks through gradient rewiring. In IJCAI, 2021.","title":"Pruning of deep spiking neural networks through gradient rewiring","authors":"Yanqi Chen, Zhaofei Yu, Wei Fang, Tiejun Huang, Yonghong Tian","year":"2021","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2021/236"},{"position":6,"raw":"[Deng and Gu, 2021] Shikuang Deng and Shi Gu. Optimal conversion of conventional artificial neural networks to spiking neural networks. In ICLR, 2021.","title":"Optimal conversion of conventional artificial neural networks to spiking neural networks","authors":"Gu ; ] Shikuang Deng, Shi Deng, Gu","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Deng et al., 2021] Yongjian Deng, Hao Chen, and Youfu Li. Mvf-net: A multi-view fusion network for event-based ob- ject classification. IEEE Transactions on Circuits and Sys- tems for Video Technology, pages 1-1, 2021.","title":"Mvf-net: A multi-view fusion network for event-based object classification","authors":"Yongjian Deng, Hao Chen, Youfu Li","year":"2021","venue":"IEEE Transactions on Circuits and Systems for Video Technology","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Deng et al., 2022a] Shikuang Deng, Yuhang Li, Shanghang Zhang, and Shi Gu. Temporal efficient training of spiking neural network via gradient re-weighting. In ICLR, 2022.","title":"Temporal efficient training of spiking neural network via gradient re-weighting","authors":"Deng","year":"2022","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Deng et al., 2022b] Yongjian Deng, Hao Chen, Hai Liu, and Youfu Li. A voxel graph cnn for object classification with event cameras. In CVPR, 2022.","title":"A voxel graph cnn for object classification with event cameras","authors":"Deng","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ding et al., 2021] Jianhao Ding, Zhaofei Yu, Yonghong Tian, and Tiejun Huang. Optimal ANN-SNN conversion for fast and accurate inference in deep spiking neural net- works. In IJCAI, 2021.","title":"Optimal ANN-SNN conversion for fast and accurate inference in deep spiking neural networks","authors":"Ding","year":"2021","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2021/321"},{"position":11,"raw":"[Fang et al., 2020] Wei Fang, Yanqi Chen, Jianhao Ding, Ding Chen, Zhaofei Yu, Huihui Zhou, Yonghong Tian, and other contributors. Spikingjelly. https://github.com/ fangwei123456/spikingjelly, 2020. Accessed: 2021-11- 21.","title":"Spikingjelly","authors":"Wei Fang, Yanqi Chen, Jianhao Ding, Ding Chen, Zhaofei Yu, Huihui Zhou, Yonghong Tian","year":"2020","venue":null,"doi":null,"url":"https://github.com/fangwei123456/spikingjelly","internal_url":null},{"position":12,"raw":"[Fang et al., 2021a] Wei Fang, Zhaofei Yu, Yanqi Chen, Tiejun Huang, Timothée Masquelier, and Yonghong Tian. Deep residual learning in spiking neural networks. In NeurIPS, 2021.","title":"Deep residual learning in spiking neural networks","authors":"Fang","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Fang et al., 2021b] Wei Fang, Zhaofei Yu, Yanqi Chen, Timothée Masquelier, Tiejun Huang, and Yonghong Tian. Incorporating learnable membrane time constant to en- hance learning of spiking neural networks. In ICCV, 2021.","title":"Incorporating learnable membrane time constant to enhance learning of spiking neural networks","authors":"Fang","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Feng et al., 2022] Lang Feng, Qianhui Liu, Huajin Tang, De Ma, and Gang Pan. Multi-level firing with spiking ds- resnet: Enabling better and deeper directly-trained spiking neural networks. In IJCAI, 2022.","title":"Multi-level firing with spiking dsresnet: Enabling better and deeper directly-trained spiking neural networks","authors":"Lang Feng, Qianhui Liu, Huajin Tang, De Ma, Gang Pan","year":"2022","venue":"IJCAI","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kaiser et al., 2020] Jacques Kaiser, Hesham Mostafa, and Emre Neftci. Synaptic plasticity dynamics for deep contin- uous local learning (decolle). Frontiers in Neuroscience, 14:424, 2020.","title":"Synaptic plasticity dynamics for deep continuous local learning (decolle)","authors":"Jacques Kaiser, Hesham Mostafa, Emre Neftci","year":"2020","venue":"Frontiers in Neuroscience","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lee et al., 2020] Chankyu Lee, Syed Shakib Sarwar, Priyadarshini Panda, Gopalakrishnan Srinivasan, and Kaushik Roy. Enabling spike-based backpropagation for training deep neural network architectures. Frontiers in Neuroscience, 14:119, 2020.","title":"Enabling spike-based backpropagation for training deep neural network architectures","authors":"Lee","year":"2020","venue":"Frontiers in Neuroscience","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Li et al., 2017] Hongmin Li, Hanchao Liu, Xiangyang Ji, Guoqi Li, and Luping Shi. Cifar10-dvs: an event-stream dataset for object classification. Frontiers in neuroscience, 11:309, 2017.","title":"Cifar10-dvs: an event-stream dataset for object classification","authors":"Li","year":"2017","venue":"Frontiers in neuroscience","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Li et al., 2021] Yuhang Li, Yufei Guo, Shanghang Zhang, Shikuang Deng, Yongqing Hai, and Shi Gu. Differen- tiable spike: Rethinking gradient-descent for training spik- ing neural networks. In NeurIPS, 2021.","title":"Differentiable spike: Rethinking gradient-descent for training spiking neural networks","authors":"Yuhang Li, Yufei Guo, Shanghang Zhang, Shikuang Deng, Yongqing Hai, Shi Gu","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Meng et al., 2022] Qingyan Meng, Mingqing Xiao, Shen Yan, Yisen Wang, Zhouchen Lin, and Zhi-Quan Luo. Training high-performance low-latency spiking neural net- works by differentiation on spike representation. In CVPR, 2022.","title":"Training high-performance low-latency spiking neural networks by differentiation on spike representation","authors":"Qingyan Meng, Mingqing Xiao, Shen Yan, Yisen Wang, Zhouchen Lin, Zhi-Quan Luo","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Pei et al., 2019] Jing Pei, Lei Deng, Sen Song, Mingguo Zhao, Youhui Zhang, Shuang Wu, Guanrui Wang, Zhe Zou, Zhenzhi Wu, Wei He, et al. Towards artificial general intelligence with hybrid tianjic chip architecture. Nature, 572(7767):106-111, 2019.","title":"Towards artificial general intelligence with hybrid tianjic chip architecture","authors":"Jing Pei, Lei Deng, Sen Song, Mingguo Zhao, Youhui Zhang, Shuang Wu, Guanrui Wang, Zhe Zou, Zhenzhi Wu, Wei He","year":"2019","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Rahiminejad et al., 2022] Ehsan Rahiminejad, Fatemeh Azad, Adel Parvizi-Fard, Mahmood Amiri, and Bernabé Linares-Barranco. A neuromorphic cmos circuit with self-repairing capability. IEEE Transactions on Neural Networks and Learning Systems, 33(5):2246-2258, 2022.","title":"A neuromorphic cmos circuit with self-repairing capability","authors":"Ehsan Rahiminejad, Fatemeh Azad, Adel Parvizi-Fard, Mahmood Amiri, Bernabé Linares-Barranco","year":"2022","venue":"IEEE Transactions on Neural Networks and Learning Systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Shrestha and Orchard, 2018] Sumit B Shrestha and Garrick Orchard. Slayer: Spike layer error reassignment in time. NeurIPS, 2018.","title":"Slayer: Spike layer error reassignment in time","authors":"B Sumit, Garrick Shrestha, Orchard","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Shrestha et al., 2022] Sumit Bam Shrestha, Longwei Zhu, and Pengfei Sun. Spikemax: Spike-based loss methods for classification. In IJCNN, 2022.","title":"Spikemax: Spike-based loss methods for classification","authors":"Bam Sumit, Longwei Shrestha, Pengfei Zhu, Sun","year":"2022","venue":"IJCNN","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Tang et al., 2018] Jianxiong Tang, Na Zhang, and Qia Li. Robust binary classification via ℓ 0 -svm. In ICDM Work- shops, 2018.","title":"Robust binary classification via ℓ 0 -svm","authors":"Jianxiong Tang, Na Zhang, Qia Li","year":"2018","venue":"ICDM Workshops","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Tavanaei et al., 2019] Amirhossein Tavanaei, Masoud Gho- drati, Saeed Reza Kheradpisheh, Timothée Masquelier, and Anthony Maida. Deep learning in spiking neural net- works. Neural Networks, 111:47-63, 2019.","title":"Deep learning in spiking neural networks","authors":"Tavanaei","year":"2019","venue":"Neural Networks","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Tseng, 2001] Paul Tseng. Convergence of a block co- ordinate descent method for nondifferentiable minimiza- tion. Journal of optimization theory and applications, 109(3):475, 2001.","title":"Convergence of a block coordinate descent method for nondifferentiable minimization","authors":"Paul Tseng","year":"2001","venue":"Journal of optimization theory and applications","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Vapnik, 2000] Vladimir Naumovich Vapnik. The Nature of Statistical Learning Theory, Second Edition. Springer, 2000.","title":"The Nature of Statistical Learning Theory, Second Edition","authors":"Vladimir Naumovich, Vapnik","year":"2000","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wang et al., 2022] Huajun Wang, Yuan-Hai Shao, Sheng- long Zhou, Ce Zhang, and Naihua Xiu. Support vector machine classifier via l 0/1 soft-margin loss. IEEE Trans- actions on Pattern Analysis and Machine Intelligence, 44(10):7253-7265, 2022.","title":"Support vector machine classifier via l 0/1 soft-margin loss","authors":"Huajun Wang, Yuan-Hai Shao, Shenglong Zhou, Ce Zhang, Naihua Xiu","year":"2022","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wu et al., 2018] Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, and Luping Shi. Spatio-temporal backpropagation for training high-performance spiking neural networks. Fron- tiers in neuroscience, 12:331, 2018.","title":"Spatio-temporal backpropagation for training high-performance spiking neural networks","authors":"Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, Luping Shi","year":"2018","venue":"Frontiers in neuroscience","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wu et al., 2019a] Jibin Wu, Yansong Chua, Malu Zhang, Qu Yang, Guoqi Li, and Haizhou Li. Deep spiking neural network with spike count based learning rule. In IJCNN, 2019.","title":"Deep spiking neural network with spike count based learning rule","authors":"Wu","year":"2019","venue":"IJCNN","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Wu et al., 2019b] Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, Yuan Xie, and Luping Shi. Direct training for spiking neu- ral networks: Faster, larger, better. In AAAI, 2019.","title":"Direct training for spiking neural networks: Faster, larger, better","authors":"Wu","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Wu et al., 2021] Zhenzhi Wu, Hehui Zhang, Yihan Lin, Guoqi Li, Meng Wang, and Ye Tang. Liaf-net: Leaky integrate and analog fire network for lightweight and effi- cient spatiotemporal information processing. IEEE Trans- actions on Neural Networks and Learning Systems, pages 1-14, 2021.","title":"Liaf-net: Leaky integrate and analog fire network for lightweight and efficient spatiotemporal information processing","authors":"Wu","year":"2021","venue":"IEEE Transactions on Neural Networks and Learning Systems","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhang et al., 2022] Duzhen Zhang, Shuncheng Jia, and Qingyu Wang. Recent advances and new frontiers in spik- ing neural networks. In IJCAI, 2022.","title":"Recent advances and new frontiers in spiking neural networks","authors":"Zhang","year":"2022","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2022/790"},{"position":34,"raw":"[Zheng et al., 2021] Hanle Zheng, Yujie Wu, Lei Deng, Yi- fan Hu, and Guoqi Li. Going deeper with directly-trained larger spiking neural networks. In AAAI, 2021.","title":"Going deeper with directly-trained larger spiking neural networks","authors":"Hanle Zheng, Yujie Wu, Lei Deng, Yifan Hu, Guoqi Li","year":"2021","venue":"AAAI","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}