{"type":"ScholarlyArticle","title":"Optimization based Layer-wise Magnitude-based Pruning for DNN Compression","authors":[{"name":"Guiying Li","slug":"guiying-li","given":"Guiying","family":"Li","affiliation":"Anhui Province Key Lab of Big Data Analysis and Application , University of Science and Technology of China , Hefei 230027 , China"},{"name":"Chao Qian","slug":"chao-qian","given":"Chao","family":"Qian","affiliation":"Anhui Province Key Lab of Big Data Analysis and Application , University of Science and Technology of China , Hefei 230027 , China"},{"name":"Chunhui Jiang","slug":"chunhui-jiang","given":"Chunhui","family":"Jiang","affiliation":"Anhui Province Key Lab of Big Data Analysis and Application , University of Science and Technology of China , Hefei 230027 , China"},{"name":"Xiaofen Lu","slug":"xiaofen-lu","given":"Xiaofen","family":"Lu","affiliation":"CERCIA , School of Computer Science , University of Birmingham , Birmingham B15 2TT , UK"},{"name":"Ke Tang","slug":"ke-tang","given":"Ke","family":"Tang","affiliation":"Shenzhen Key Lab of Computational Intelligence , Department of Computer Science and Engineering , Southern University of Science and Technology , Shenzhen 518055 , China"}],"abstract":"Layer-wise magnitude-based pruning (LMP) is a very popular method for deep neural network (DNN) compression. However, tuning the layer-specific thresholds is a difficult task, since the space of threshold candidates is exponentially large and the evaluation is very expensive. Previous methods are mainly by hand and require expertise. In this paper, we propose an automatic tuning approach based on optimization, named OLMP. The idea is to transform the threshold tuning problem into a constrained optimization problem (i.e., minimizing the size of the pruned model subject to a constraint on the accuracy loss), and then use powerful derivative-free optimization algorithms to solve it. To compress a trained DNN, OLMP is conducted within a new iterative pruning and adjusting pipeline. Empirical results show that OLMP can achieve the best pruning ratio on LeNet-style models (i.e., 114 times for LeNet-300-100 and 298 times for LeNet-5) compared with some state-of-the- art DNN pruning methods, and can reduce the size of an AlexNet-style network up to 82 times without accuracy loss.","doi":"10.24963/ijcai.2018/330","url":"https://ijcai.org/proceedings/2018/330","pdf":"https://ijcai.org/proceedings/2018/0330.pdf","markdown":"https://ijcai.org/proceedings/2018/330.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/330","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Machine Learning","track":"Main track","number":330,"pages":{"first":2383,"last":2389},"keywords":["Constraints and SAT: Constraint Optimisation","Heuristic Search and Game Playing: Heuristic Search and Machine Learning","Machine Learning: Neural Networks","Machine Learning: Deep Learning","Heuristic Search and Game Playing: Combinatorial Search and Optimisation","Machine Learning: Feature Selection ; Learning Sparse Models"],"videos":[],"references":[{"position":1,"raw":"[Brochu et al., 2010] E. Brochu, V. M. Cora, and N. de Fre- itas. A tutorial on bayesian optimization of expensive cost functions, with application to active user modeling and hi- erarchical reinforcement learning. arXiv:1012.2599, 2010.","title":"A tutorial on bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning","authors":"E Brochu, V M Cora, N De Freitas","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Castellano et al., 1997] G. Castellano, A. M. Fanelli, and M. Pelillo. An iterative pruning algorithm for feedforward neural networks. IEEE Transactions on Neural Networks, 8(3):519-531, 1997.","title":"An iterative pruning algorithm for feedforward neural networks","authors":"G Castellano, A M Fanelli, M Pelillo","year":"1997","venue":"IEEE Transactions on Neural Networks","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Denil et al., 2013] M. Denil, B. Shakibi, L. Dinh, M. A. Ranzato, and N. Freitas. Predicting parameters in deep learning. In Advances in Neural Information Processing Systems 26 (NIPS'13), pages 2148-2156, Lake Tahoe, NV, 2013.","title":"Predicting parameters in deep learning","authors":"M Denil, B Shakibi, L Dinh, M A Ranzato, N Freitas","year":"2013","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dong et al., 2017] X. Dong, S.-Y. Chen, and S. J. Pan. Learning to prune deep neural networks via layer-wise optimal brain surgeon. In Advances in Neural Informa- tion Processing Systems 30 (NIPS'17), pages 4860-4874, Long Beach, CA, 2017.","title":"Learning to prune deep neural networks via layer-wise optimal brain surgeon","authors":"X Dong, S.-Y Chen, S J Pan","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Goldberg, 1989] D. E. Goldberg. Genetic Algorithms in Search Optimization and Machine Learning. Addison- Wesley, Boston, MA, 1989.","title":"Genetic Algorithms in Search Optimization and Machine Learning","authors":"D E Goldberg","year":"1989","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Griffin et al., 2006] G. Griffin, AD. Holub, and P. Perona. The caltech 256. Caltech Technical Report, 2006.","title":"The caltech 256","authors":"G Griffin, A D Holub, P Perona","year":"2006","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Guo et al., 2016] Y.-W. Guo, A.-B. Yao, and Y.-R. Chen. Dynamic network surgery for efficient dnns. In Advances in Neural Information Processing Systems 29 (NIPS'16), pages 1379-1387, Barcelona, Spain, 2016.","title":"Dynamic network surgery for efficient dnns","authors":"Y.-W Guo, A.-B Yao, Y.-R Chen","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Han et al., 2015] S. Han, J. Pool, J. Tran, and W. J. Dally. Learning both weights and connections for efficient neu- ral network. In Advances in Neural Information Process- ing Systems 28 (NIPS'15), pages 1135-1143, Montreal, Canada, 2015.","title":"Learning both weights and connections for efficient neural network","authors":"S Han, J Pool, J Tran, W J Dally","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hassibi and Stork, 1992] B. Hassibi and D. G. Stork. Sec- ond order derivatives for network pruning: Optimal brain surgeon. In Advances in Neural Information Processing Systems 5 (NIPS'92), pages 164-171, Denver, CO, 1992.","title":"Second order derivatives for network pruning: Optimal brain surgeon","authors":"B Hassibi, D G Stork","year":"1992","venue":"Advances in Neural Information Processing Systems 5 (NIPS'92)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[He et al., 2014] T.-X. He, Y.-C. Fan, Y.-M. Qian, T. Tan, and K. Yu. Reshaping deep neural network for fast de- coding by node-pruning. In Proceedings of International Conference on Acoustics, Speech and Signal Processing (ICASSP'14), pages 245-249, Florence, Italy, 2014.","title":"Reshaping deep neural network for fast decoding by node-pruning","authors":"T.-X He, Y.-C Fan, Y.-M Qian, T Tan, K Yu","year":"2014","venue":"Proceedings of International Conference on Acoustics, Speech and Signal Processing (ICASSP'14)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Jia et al., 2014] Y.-Q. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. Caffe: Convolutional architecture for fast fea- ture embedding. arXiv:1408.5093, 2014.","title":"Caffe: Convolutional architecture for fast feature embedding","authors":"Y.-Q Jia, E Shelhamer, J Donahue, S Karayev, J Long, R Girshick, S Guadarrama, T Darrell","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kim et al., 2016] Y.-D. Kim, E. Park, S.-J. Yoo, T. Choi, L. Yang, and D.-J. Shin. Compression of deep convolu- tional neural networks for fast and low power mobile ap- plications. In Proceedings of the 4th International Confer- ence on Learning Representations (ICLR'16), pages 1-16, San Juan, Puerto Rico, 2016.","title":"Compression of deep convolutional neural networks for fast and low power mobile applications","authors":"Y.-D Kim, E Park, S.-J Yoo, T Choi, L Yang, D.-J Shin","year":"2016","venue":"Proceedings of the 4th International Conference on Learning Representations (ICLR'16)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Krizhevsky et al., 2012] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolu- tional neural networks. In Advances in Neural Information Processing Systems 25 (NIPS'12), pages 1106-1114, Lake Tahoe, NV, 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"A Krizhevsky, I Sutskever, G E Hinton","year":"2012","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[LeCun et al., 1989] Y. LeCun, J. S. Denker, and S. A. Solla. Optimal brain damage. In Advances in Neural Information Processing Systems 2 (NIPS'89), pages 598-605, Denver, CO, 1989.","title":"Optimal brain damage","authors":"Y Lecun, J S Denker, S A Solla","year":"1989","venue":"Advances in Neural Information Processing Systems 2 (NIPS'89)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[LeCun et al., 2015] Y. LeCun, Y. Bengio, and G. E. Hinton. Deep learning. Nature, 521(7553):436-444, 2015.","title":"Deep learning","authors":"Y Lecun, Y Bengio, G E Hinton","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Molchanov et al., 2017] D. Molchanov, A. Ashukha, and D. P. Vetrov. Variational dropout sparsifies deep neural networks. In Proceedings of the 34th International Confer- ence on Machine Learning (ICML'17), pages 2498-2507, Sydney, Australia, 2017.","title":"Variational dropout sparsifies deep neural networks","authors":"D Molchanov, A Ashukha, D P Vetrov","year":"2017","venue":"Proceedings of the 34th International Conference on Machine Learning (ICML'17)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Nowlan and Hinton, 1992] S. J. Nowlan and G. E. Hinton. Simplifying neural networks by soft weight-sharing. Neu- ral Computation, 4(4):473-493, 1992.","title":"Simplifying neural networks by soft weight-sharing","authors":"S J Nowlan, G E Hinton","year":"1992","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Qian et al., 2015] C. Qian, Y. Yu, and Z.-H. Zhou. Subset selection by Pareto optimization. In Advances in Neu- ral Information Processing Systems 28 (NIPS'15), pages 1774-1782, Montreal, Canada, 2015.","title":"Subset selection by Pareto optimization","authors":"C Qian, Y Yu, Z.-H Zhou","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Real et al., 2017] E. Real, S. Moore, A. Selle, S. Saxena, Y. L. Suematsu, J. Tan, Q. V. Le, and A. Kurakin. Large- scale evolution of image classifiers. In Proceedings of the 34th International Conference on Machine Learning (ICML'17), pages 2902-2911, Sydney, Australia, 2017.","title":"Largescale evolution of image classifiers","authors":"E Real, S Moore, A Selle, S Saxena, Y L Suematsu, J Tan, Q V Le, A Kurakin","year":"2017","venue":"Proceedings of the 34th International Conference on Machine Learning (ICML'17)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Reed, 1993] R. Reed. Pruning algorithms-a survey. IEEE Transactions on Neural Networks, 4(5):740-747, 1993.","title":"Pruning algorithms-a survey","authors":"R Reed","year":"1993","venue":"IEEE Transactions on Neural Networks","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[See et al., 2016] A. See, M.-T. Luong, and C. D. Man- ning. Compression of neural machine translation models via pruning. In Proceedings of the 20th Conference on Computational Natural Language Learning (CoNLL'16), pages 291-301, Berlin, Germany, 2016.","title":"Compression of neural machine translation models via pruning","authors":"A See, M.-T Luong, C D Manning","year":"2016","venue":"Proceedings of the 20th Conference on Computational Natural Language Learning (CoNLL'16)","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Tang et al., 2016] K. Tang, P. Yang, and X. Yao. Nega- tively correlated search. IEEE Journal on Selected Areas in Communications, 34(3):542-550, 2016.","title":"Negatively correlated search","authors":"K Tang, P Yang, X Yao","year":"2016","venue":"IEEE Journal on Selected Areas in Communications","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Ullrich et al., 2017] K. Ullrich, E. Meeds, and M. Welling. Soft weight-sharing for neural network compression. In Proceedings of the 5th International Conference on Learning Representations (ICLR'17), pages 1-16, Toulon, France, 2017.","title":"Soft weight-sharing for neural network compression","authors":"K Ullrich, E Meeds, M Welling","year":"2017","venue":"Proceedings of the 5th International Conference on Learning Representations (ICLR'17)","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yao and Liu, 1997] X. Yao and Y. Liu. A new evolution- ary system for evolving artificial neural networks. IEEE Transactions on Neural Networks, 8(3):694-713, 1997.","title":"A new evolutionary system for evolving artificial neural networks","authors":"Liu Yao, X Yao, Y Liu","year":"1997","venue":"IEEE Transactions on Neural Networks","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yu et al., 2016] Y. Yu, H. Qian, and Y.-Q. Hu. Derivative- free optimization via classification. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence (AAAI'16), pages 2286-2292, Phoenix, AZ, 2016.","title":"Derivativefree optimization via classification","authors":"Y Yu, H Qian, Y.-Q Hu","year":"2016","venue":"Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence (AAAI'16)","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zeiler and Fergus, 2014] M. D. Zeiler and R. Fergus. Visu- alizing and understanding convolutional networks. In Pro- ceedings of the 13th European Conference on Computer Vision (ECCV'14), pages 818-833, Zurich, Switzerland, 2014.","title":"Visualizing and understanding convolutional networks","authors":"M D Zeiler, R Fergus","year":"2014","venue":"Proceedings of the 13th European Conference on Computer Vision (ECCV'14)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}