{"type":"ScholarlyArticle","title":"Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks","authors":[{"name":"Yang He","slug":"yang-he","given":"Yang","family":"He","affiliation":"SUSTech-UTS Joint Centre of CIS , Southern University of Science and Technology; CAI , University of Technology Sydney"},{"name":"Guoliang Kang","slug":"guoliang-kang","given":"Guoliang","family":"Kang","affiliation":"CAI , University of Technology Sydney"},{"name":"Xuanyi Dong","slug":"xuanyi-dong","given":"Xuanyi","family":"Dong","affiliation":"CAI , University of Technology Sydney"},{"name":"Yanwei Fu","slug":"yanwei-fu","given":"Yanwei","family":"Fu","affiliation":"The School of Data Science , Fudan University"},{"name":"Yi Yang","slug":"yi-yang","given":"Yi","family":"Yang","affiliation":"SUSTech-UTS Joint Centre of CIS , Southern University of Science and Technology; CAI , University of Technology Sydney"}],"abstract":"This paper proposed a Soft Filter Pruning (SFP) method to accelerate the inference procedure of deep Convolutional Neural Networks (CNNs). Specifically, the proposed SFP enables the pruned filters to be updated when training the model after pruning. SFP has two advantages over previous works: (1) Larger model capacity. Updating previously pruned filters provides our approach with larger optimization space than fixing the filters to zero. Therefore, the network trained by our method has a larger model capacity to learn from the training data. (2) Less dependence on the pretrained model. Large capacity enables SFP to train from scratch and prune the model simultaneously. In contrast, previous filter pruning methods should be conducted on the basis of the pre-trained model to guarantee their performance. Empirically, SFP from scratch outperforms the previous filter pruning methods. Moreover, our approach has been demonstrated effective for many advanced CNN architectures. Notably, on ILSCRC-2012, SFP reduces more than 42% FLOPs on ResNet-101 with even 0.2% top-5 accuracy improvement, which has advanced the state-of-the-art. Code is publicly available on GitHub: https://github.com/he-y/softfilter-pruning","doi":"10.24963/ijcai.2018/309","url":"https://ijcai.org/proceedings/2018/309","pdf":"https://ijcai.org/proceedings/2018/0309.pdf","markdown":"https://ijcai.org/proceedings/2018/309.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/309","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Machine Learning","track":"Main track","number":309,"pages":{"first":2234,"last":2240},"keywords":["Machine Learning: Neural Networks","Machine Learning: Deep Learning","Computer Vision: Computer Vision"],"videos":[],"references":[{"position":1,"raw":"[Dong et al., 2017a] Xuanyi Dong, Junshi Huang, Yi Yang, and Shuicheng Yan. More is less: A more complicated network with less inference complexity. In CVPR, 2017.","title":"More is less: A more complicated network with less inference complexity","authors":"Dong","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Dong et al., 2017b] Xuanyi Dong, Deyu Meng, Fan Ma, and Yi Yang. A dual-network progressive approach to weakly supervised object detection. In ACM Multimedia, 2017.","title":"A dual-network progressive approach to weakly supervised object detection","authors":"Dong","year":"2017","venue":"ACM Multimedia","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Dong et al., 2018] Xuanyi Dong, Shoou-I Yu, Xinshuo Weng, Shih-En Wei, Yi Yang, and Yaser Sheikh. Supervision-by-Registration: An unsupervised approach to improve the precision of facial landmark detectors. In CVPR, 2018.","title":"Supervision-by-Registration: An unsupervised approach to improve the precision of facial landmark detectors","authors":"Dong","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Guo et al., 2016] Yiwen Guo, Anbang Yao, and Yurong Chen. Dynamic network surgery for efficient DNNs. In NIPS, 2016.","title":"Dynamic network surgery for efficient DNNs","authors":"Yiwen Guo, Anbang Yao, Yurong Chen","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Han et al., 2015a] Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural net- works with pruning, trained quantization and huffman cod- ing. In ICLR, 2015.","title":"Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding","authors":"Han","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Han et al., 2015b] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In NIPS, 2015.","title":"Learning both weights and connections for efficient neural network","authors":"Han","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[He et al., 2016a] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.","title":"Deep residual learning for image recognition","authors":"He","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[He et al., 2016b] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In ECCV, 2016.","title":"Identity mappings in deep residual networks","authors":"He","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[He et al., 2017] Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural net- works. In ICCV, 2017.","title":"Channel pruning for accelerating very deep neural networks","authors":"Yihui He, Xiangyu Zhang, Jian Sun","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jaderberg et al., 2014] Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. Speeding up convolutional neural net- works with low rank expansions. In BMVC, 2014.","title":"Speeding up convolutional neural networks with low rank expansions","authors":"Jaderberg","year":"2014","venue":"BMVC","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kang et al., 2017] Guoliang Kang, Jun Li, and Dacheng Tao. Shakeout: A new approach to regularized deep neural network training. IEEE T-PAMI, 2017.","title":"Shakeout: A new approach to regularized deep neural network training","authors":"Guoliang Kang, Jun Li, Dacheng Tao","year":"2017","venue":"IEEE T-PAMI","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Krizhevsky and Hinton, 2009] Alex Krizhevsky and Geof- frey Hinton. Learning multiple layers of features from tiny images. 2009.","title":"Learning multiple layers of features from tiny images","authors":"Alex Krizhevsky, Geoffrey Hinton","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lebedev and Lempitsky, 2016] Vadim Lebedev and Victor Lempitsky. Fast ConvNets using group-wise brain dam- age. In CVPR, 2016.","title":"Vadim Lebedev and Victor Lempitsky. Fast ConvNets using group-wise brain damage","authors":"Lempitsky Lebedev","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li et al., 2017] Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient ConvNets. In ICLR, 2017.","title":"Pruning filters for efficient ConvNets","authors":"Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, Hans Peter Graf","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu et al., 2017] Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through net- work slimming. In ICCV, 2017.","title":"Learning efficient convolutional networks through network slimming","authors":"Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, Changshui Zhang","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Luo et al., 2017] Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. ThiNet: A filter level pruning method for deep neural network compression. In ICCV, 2017.","title":"ThiNet: A filter level pruning method for deep neural network compression","authors":"Jian-Hao Luo, Jianxin Wu, Weiyao Lin","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Molchanov et al., 2017] Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolu- tional neural networks for resource efficient transfer learn- ing. In ICLR, 2017.","title":"Pruning convolutional neural networks for resource efficient transfer learning","authors":"Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, Jan Kautz","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Paszke et al., 2017] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary De- Vito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NIPS-W, 2017.","title":"Automatic differentiation in pytorch","authors":"Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary De-Vito, Zeming Lin, Alban Desmaison, Luca Antiga, Adam Lerer","year":"2017","venue":"NIPS-W","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Ren et al., 2015] Shaoqing Ren, Kaiming He, Ross Gir- shick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NIPS, 2015.","title":"Faster r-cnn: Towards real-time object detection with region proposal networks","authors":"Ren","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Russakovsky et al., 2015] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. ImageNet large scale visual recognition challenge. IJCV, 2015.","title":"ImageNet large scale visual recognition challenge","authors":"Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein","year":"2015","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Shen et al., 2018a] Tao Shen, Tianyi Zhou, Guodong Long, Jing Jiang, Shirui Pan, and Chengqi Zhang. Disan: Di- rectional self-attention network for rnn/cnn-free language understanding. In AAAI, 2018.","title":"Disan: Directional self-attention network for rnn/cnn-free language understanding","authors":"Shen","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Shen et al., 2018b] Tao Shen, Tianyi Zhou, Guodong Long, Jing Jiang, and Chengqi Zhang. Bi-directional block self- attention for fast and memory-efficient sequence model- ing. In ICLR, 2018.","title":"Bi-directional block selfattention for fast and memory-efficient sequence modeling","authors":"Shen","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Szegedy et al., 2015] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabi- novich. Going deeper with convolutions. In CVPR, 2015.","title":"Going deeper with convolutions","authors":"Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, Andrew Rabinovich","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Tai et al., 2016] Cheng Tai, Tong Xiao, Yi Zhang, Xiaogang Wang, et al. Convolutional neural networks with low-rank regularization. In ICLR, 2016.","title":"Convolutional neural networks with low-rank regularization","authors":"Cheng Tai, Tong Xiao, Yi Zhang, Xiaogang Wang","year":"2016","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wen et al., 2016] Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. In NIPS, 2016.","title":"Learning structured sparsity in deep neural networks","authors":"Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, Hai Li","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yang et al., 2010] Yi Yang, Dong Xu, Feiping Nie, Shuicheng Yan, and Yueting Zhuang. Image clustering us- ing local discriminant models and global integration. IEEE T-IP, 2010.","title":"Image clustering using local discriminant models and global integration","authors":"Yi Yang, Dong Xu, Feiping Nie, Shuicheng Yan, Yueting Zhuang","year":"2010","venue":"IEEE T-IP","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zagoruyko and Komodakis, 2016] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC, 2016.","title":"Wide residual networks","authors":"Sergey Zagoruyko, Nikos Komodakis","year":"2016","venue":"BMVC","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhang et al., 2016] Xiangyu Zhang, Jianhua Zou, Kaiming He, and Jian Sun. Accelerating very deep convolutional networks for classification and detection. IEEE T-PAMI, 2016.","title":"Accelerating very deep convolutional networks for classification and detection","authors":"Xiangyu Zhang, Jianhua Zou, Kaiming He, Jian Sun","year":"2016","venue":"IEEE T-PAMI","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhou et al., 2017] Aojun Zhou, Anbang Yao, Yiwen Guo, Lin Xu, and Yurong Chen. Incremental network quantiza- tion: Towards lossless cnns with low-precision weights. In ICLR, 2017.","title":"Incremental network quantization: Towards lossless cnns with low-precision weights","authors":"Zhou","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhu et al., 2017] Chenzhuo Zhu, Song Han, Huizi Mao, and William J Dally. Trained ternary quantization. In ICLR, 2017.","title":"Trained ternary quantization","authors":"Zhu","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}