{"type":"ScholarlyArticle","title":"Efficient DNN Neuron Pruning by Minimizing Layer-wise Nonlinear Reconstruction Error","authors":[{"name":"Chunhui Jiang","slug":"chunhui-jiang","given":"Chunhui","family":"Jiang","affiliation":"Anhui Province Key Lab of Big Data Analysis and Application , University of Science and Technology of China , Hefei 230027 , China"},{"name":"Guiying Li","slug":"guiying-li","given":"Guiying","family":"Li","affiliation":"Anhui Province Key Lab of Big Data Analysis and Application , University of Science and Technology of China , Hefei 230027 , China"},{"name":"Chao Qian","slug":"chao-qian","given":"Chao","family":"Qian","affiliation":"Anhui Province Key Lab of Big Data Analysis and Application , University of Science and Technology of China , Hefei 230027 , China"},{"name":"Ke Tang","slug":"ke-tang","given":"Ke","family":"Tang","affiliation":"Shenzhen Key Lab of Computational Intelligence , Department of Computer Science and Engineering , Southern University of Science and Technology , Shenzhen 518055 , China"}],"abstract":"Deep neural networks (DNNs) have achieved great success, but the applications to mobile devices are limited due to their huge model size and low inference speed. Much effort thus has been devoted to pruning DNNs. Layer-wise neuron pruning methods have shown their effectiveness, which minimize the reconstruction error of linear response with a limited number of neurons in each single layer pruning. In this paper, we propose a new layer-wise neuron pruning approach by minimizing the reconstruction error of nonlinear units, which might be more reasonable since the error before and after activation can change significantly. An iterative optimization procedure combining greedy selection with gradient decent is proposed for single layer pruning. Experimental results on benchmark DNN models show the superiority of the proposed approach. Particularly, for VGGNet, the proposed approach can compress its disk space by 13.6× and bring a speedup of 3.7×; for AlexNet, it can achieve a compression rate of 4.1× and a speedup of 2.2×, respectively.","doi":"10.24963/ijcai.2018/318","url":"https://ijcai.org/proceedings/2018/318","pdf":"https://ijcai.org/proceedings/2018/0318.pdf","markdown":"https://ijcai.org/proceedings/2018/318.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/318","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Machine Learning","track":"Main track","number":318,"pages":{"first":2298,"last":2304},"keywords":["Machine Learning: Feature Selection ; Learning Sparse Models","Machine Learning: Neural Networks","Machine Learning Applications: Applications of Supervised Learning","Machine Learning: Deep Learning"],"videos":[],"references":[{"position":1,"raw":"[Alvarez and Salzmann, 2016] J. M. Alvarez and M. Salz- mann. Learning the number of neurons in deep networks. In Advances in Neural Information Processing Systems 29 (NIPS'16), pages 2270-2278, Barcelona, Spain, 2016.","title":"Learning the number of neurons in deep networks","authors":"J M Alvarez, M Salzmann","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Courbariaux et al., 2016] M. Courbariaux, I. Hubara, D. Soudry, R. El-Yaniv, and Y. Bengio. Binarized neural networks: Training deep neural networks with weights and activations constrained to + 1 or -1. arXiv preprint arXiv:1602.02830, 2016.","title":"Binarized neural networks: Training deep neural networks with weights and activations constrained to + 1 or -1","authors":"M Courbariaux, I Hubara, D Soudry, R El-Yaniv, Y Bengio","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Dong et al., 2017] X. Dong, S.-Y. Chen, and S. Pan. Learn- ing to prune deep neural networks via layer-wise optimal brain surgeon. In Advances in Neural Information Pro- cessing Systems 30 (NIPS'17), pages 4860-4874, Long Beach, CA, 2017.","title":"Learning to prune deep neural networks via layer-wise optimal brain surgeon","authors":"X Dong, S.-Y Chen, S Pan","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Guo et al., 2016] Y.-W. Guo, A.-B. Yao, and Y.-R. Chen. Dynamic network surgery for efficient dnns. In Advances In Neural Information Processing Systems 29 (NIPS'16), pages 1379-1387, Barcelona, Spain, 2016.","title":"Dynamic network surgery for efficient dnns","authors":"Y.-W Guo, A.-B Yao, Y.-R Chen","year":"2016","venue":"Advances In Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Han et al., 2015] S. Han, J. Pool, J. Tran, and W. Dally. Learning both weights and connections for efficient neu- ral network. In Advances in Neural Information Process- ing Systems 28 (NIPS'15), pages 1135-1143, Montreal, Canada, 2015.","title":"Learning both weights and connections for efficient neural network","authors":"S Han, J Pool, J Tran, W Dally","year":"2015","venue":"Advances in Neural Information Processing Systems 28 (NIPS'15)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[He et al., 2017] Y.-H. He, X.-Y. Zhang, and J. Sun. Chan- nel pruning for accelerating very deep neural networks. In Proceedings of the 16th IEEE International Conference on Computer Vision (ICCV'17), pages 1398-1406, Venice, Italy, 2017.","title":"Channel pruning for accelerating very deep neural networks","authors":"Y.-H He, X.-Y Zhang, J Sun","year":"2017","venue":"Proceedings of the 16th IEEE International Conference on Computer Vision (ICCV'17)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Ioffe and Szegedy, 2015] S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on Machine Learning (ICML'15), pages 448-456, Lille, France, 2015.","title":"Batch normalization: Accelerating deep network training by reducing internal covariate shift","authors":"S Ioffe, C Szegedy","year":"2015","venue":"Proceedings of the 32nd International Conference on Machine Learning (ICML'15)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Jaderberg et al., 2014] M. Jaderberg, A. Vedaldi, and A. Zisserman. Speeding up convolutional neural net- works with low rank expansions. arXiv preprint arXiv:1405.3866, 2014.","title":"Speeding up convolutional neural net-works with low rank expansions","authors":"M Jaderberg, A Vedaldi, A Zisserman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jia et al., 2014] Y.-Q. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. B. Girshick, S. Guadarrama, and T. Darrell. Caffe: Convolutional architecture for fast fea- ture embedding. In Proceedings of the 22nd ACM inter- national conference on Multimedia (MM'14), pages 675- 678, Orlando, FL, 2014.","title":"Caffe: Convolutional architecture for fast feature embedding","authors":"Y.-Q Jia, E Shelhamer, J Donahue, S Karayev, J Long, R B Girshick, S Guadarrama, T Darrell","year":"2014","venue":"Proceedings of the 22nd ACM international conference on Multimedia (MM'14)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Krizhevsky and Hinton, 2009] A. Krizhevsky and G. Hin- ton. Learning multiple layers of features from tiny images. 2009.","title":"Learning multiple layers of features from tiny images","authors":"A Krizhevsky, G Hinton","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[LeCun et al., 1998] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2324, 1998.","title":"Gradient-based learning applied to document recognition","authors":"Y Lecun, L Bottou, Y Bengio, P Haffner","year":"1998","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2016] H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf. Pruning filters for efficient convnets. arXiv preprint arXiv:1608.08710, 2016.","title":"Pruning filters for efficient convnets","authors":"H Li, A Kadav, I Durdanovic, H Samet, H P Graf","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Liu et al., 2017] Z. Liu, J.-G. Li, Z.-Q. Shen, G. Huang, S.- M. Yan, and C.-S. Zhang. Learning efficient convolutional networks through network slimming. In Proceedings of the 16th IEEE International Conference on Computer Vision (ICCV'17), pages 2755-2763, Venice, Italy, 2017.","title":"Learning efficient convolutional networks through network slimming","authors":"Z Liu, J.-G Li, Z.-Q Shen, G Huang, S.-M Yan, C.-S Zhang","year":"2017","venue":"Proceedings of the 16th IEEE International Conference on Computer Vision (ICCV'17)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Luo et al., 2017] J.-H. Luo, J.-X. Wu, and W.-Y. Lin. Thinet: A filter level pruning method for deep neural net- work compression. In Proceedings of the 16th IEEE In- ternational Conference on Computer Vision (ICCV'17), pages 5068-5076, Venice, Italy, 2017.","title":"Thinet: A filter level pruning method for deep neural network compression","authors":"J.-H Luo, J.-X Wu, W.-Y Lin","year":"2017","venue":"Proceedings of the 16th IEEE International Conference on Computer Vision (ICCV'17)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Nair and Hinton, 2010] V. Nair and G. E. Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th International Conference on Ma- chine Learning (ICML'10), pages 807-814, Haifa, Israel, 2010.","title":"Rectified linear units improve restricted boltzmann machines","authors":"V Nair, G E Hinton","year":"2010","venue":"Proceedings of the 27th International Conference on Machine Learning (ICML'10)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Russakovsky et al., 2015] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z.-H. Huang, A. Karpathy, A. Khosla, M. S. Bernstein, A. C. Berg, and F.-F. Li. Im- agenet large scale visual recognition challenge. Interna- tional Journal of Computer Vision, 115(3):211-252, 2015.","title":"Imagenet large scale visual recognition challenge","authors":"O Russakovsky, J Deng, H Su, J Krause, S Satheesh, S Ma, Z.-H Huang, A Karpathy, A Khosla, M S Bernstein, A C Berg, F.-F Li","year":"2015","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Wen et al., 2016] W. Wen, C.-P. Wu, Y.-D. Wang, Y.-R. Chen, and H. Li. Learning structured sparsity in deep neu- ral networks. In Advances in Neural Information Process- ing Systems 29 (NIPS'16), pages 2074-2082, Barcelona, Spain, 2016.","title":"Learning structured sparsity in deep neural networks","authors":"W Wen, C.-P Wu, Y.-D Wang, Y.-R Chen, H Li","year":"2016","venue":"Advances in Neural Information Processing Systems 29 (NIPS'16)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Zhang et al., 2015] X.-Y. Zhang, J.-H. Zou, X. Ming, K.- M. He, and J. Sun. Efficient and accurate approximations of nonlinear convolutional networks. In Proceedings of the 28th IEEE Conference on Computer Vision and Pattern Recognition (CVPR'15), pages 1984-1992, Boston, MA, 2015.","title":"Efficient and accurate approximations of nonlinear convolutional networks","authors":"Zhang","year":"2015","venue":"Proceedings of the 28th IEEE Conference on Computer Vision and Pattern Recognition (CVPR'15)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}