{"type":"ScholarlyArticle","title":"Accelerating Convolutional Networks via Global & Dynamic Filter Pruning","authors":[{"name":"Shaohui Lin","slug":"shaohui-lin","given":"Shaohui","family":"Lin","affiliation":"Fujian Key Laboratory of Sensing and Computing for Smart City , Xiamen University , China; School of Information Science and Engineering , Xiamen University , China"},{"name":"Rongrong Ji","slug":"rongrong-ji","given":"Rongrong","family":"Ji","affiliation":"Fujian Key Laboratory of Sensing and Computing for Smart City , Xiamen University , China; School of Information Science and Engineering , Xiamen University , China"},{"name":"Yuchao Li","slug":"yuchao-li","given":"Yuchao","family":"Li","affiliation":"Fujian Key Laboratory of Sensing and Computing for Smart City , Xiamen University , China; School of Information Science and Engineering , Xiamen University , China"},{"name":"Yongjian Wu","slug":"yongjian-wu","given":"Yongjian","family":"Wu","affiliation":"BestImage , Tencent Technology (Shanghai) Co. , Ltd , China"},{"name":"Feiyue Huang","slug":"feiyue-huang","given":"Feiyue","family":"Huang","affiliation":"BestImage , Tencent Technology (Shanghai) Co. , Ltd , China"},{"name":"Baochang Zhang","slug":"baochang-zhang","given":"Baochang","family":"Zhang","affiliation":"School of Automation Science and Electrical Engineering , Beihang University , China"}],"abstract":"Accelerating convolutional neural networks has recently received ever-increasing research focus. Among various approaches proposed in the literature, filter pruning has been regarded as a promising solution, which is due to its advantage in significant speedup and memory reduction of both network model and intermediate feature maps. To this end, most approaches tend to prune filters in a layer-wise fixed manner, which is incapable to dynamically recover the previously removed filter, as well as jointly optimize the pruned network across layers. In this paper, we propose a novel global & dynamic pruning (GDP) scheme to prune redundant filters for CNN acceleration. In particular, GDP first globally prunes the unsalient filters across all layers by proposing a global discriminative function based on prior knowledge of filters. Second, it dynamically updates the filter saliency all over the pruned sparse network, and then recover the mistakenly pruned filter, followed by a retraining phase to improve the model accuracy. Specially, we effectively solve the corresponding non-convex optimization problem of the proposed GDP via stochastic gradient descent with greedy alternative updating. Extensive experiments show that, comparing to the state-of-the-art filter pruning methods, the proposed approach achieves superior performance to accelerate several cutting-edge CNNs on the ILSVRC 2012 benchmark.","doi":"10.24963/ijcai.2018/336","url":"https://ijcai.org/proceedings/2018/336","pdf":"https://ijcai.org/proceedings/2018/0336.pdf","markdown":"https://ijcai.org/proceedings/2018/336.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/336","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Machine Learning","track":"Main track","number":336,"pages":{"first":2425,"last":2432},"keywords":["Machine Learning: Classification","Machine Learning: Deep Learning"],"videos":[],"references":[{"position":1,"raw":"[Abadi et al., 2016] Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, et al. Tensorflow: Large-scale machine learn- ing on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467, 2016.","title":"Tensorflow: Large-scale machine learning on heterogeneous distributed systems","authors":"Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Anwar et al., 2015] Sajid Anwar, Kyuyeon Hwang, and Wonyong Sung. Structured pruning of deep convolutional neural networks. arXiv preprint arXiv:1512.08571, 2015.","title":"Structured pruning of deep convolutional neural networks","authors":"Sajid Anwar, Kyuyeon Hwang, Wonyong Sung","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Courbariaux and Bengio, 2016] M. Courbariaux and Y. Bengio. Binarynet: Training deep neural networks with weights and activations constrained to+ 1 or-1. arXiv preprint arXiv:1602.02830, 2016.","title":"Binarynet: Training deep neural networks with weights and activations constrained to+ 1 or-1","authors":"M Courbariaux, Y Bengio","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Courbariaux et al., 2015] M. Courbariaux, Y. Bengio, and J. David. Binaryconnect: Training deep neural networks with binary weights during propagations. In NIPS, 2015.","title":"Binaryconnect: Training deep neural networks with binary weights during propagations","authors":"M Courbariaux, Y Bengio, J David","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Denton et al., 2014] Emily L Denton, Wojciech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus. Exploiting linear structure within convolutional networks for efficient evaluation. In NIPS, pages 1269-1277, 2014.","title":"Exploiting linear structure within convolutional networks for efficient evaluation","authors":"Wojciech Emily L Denton, Joan Zaremba, Yann Bruna, Rob Lecun, Fergus","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Girshick et al., 2014] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR, pages 580-587, 2014.","title":"Rich feature hierarchies for accurate object detection and semantic segmentation","authors":"Ross Girshick, Jeff Donahue, Trevor Darrell, Jitendra Malik","year":"2014","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gong et al., 2014] Yunchao Gong, Liu Liu, Ming Yang, and Lubomir Bourdev. Compressing deep convolu- tional networks using vector quantization. arXiv preprint arXiv:1412.6115, 2014.","title":"Compressing deep convolutional networks using vector quantization","authors":"Gong","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Guo et al., 2016] Yiwen Guo, Anbang Yao, and Yurong Chen. Dynamic network surgery for efficient dnns. In NIPS, pages 1379-1387, 2016.","title":"Dynamic network surgery for efficient dnns","authors":"Yiwen Guo, Anbang Yao, Yurong Chen","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Han et al., 2015a] Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural net- work with pruning, trained quantization and huffman cod- ing. CoRR, abs/1510.00149, 2, 2015.","title":"Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding","authors":"Han","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Han et al., 2015b] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In NIPS, pages 1135-1143, 2015.","title":"Learning both weights and connections for efficient neural network","authors":"Han","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Han et al., 2016] Song Han, Xingyu Liu, Huizi Mao, Jing Pu, Ardavan Pedram, Mark A Horowitz, and William J Dally. Eie: efficient inference engine on compressed deep neural network. In ISCA, 2016.","title":"Eie: efficient inference engine on compressed deep neural network","authors":"Han","year":"2016","venue":"ISCA","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hassibi and Stork, 1993] Babak Hassibi and David G Stork. Second order derivatives for network pruning: Optimal brain surgeon. In NIPS, 1993.","title":"Second order derivatives for network pruning: Optimal brain surgeon","authors":"Babak Hassibi, David G Stork","year":"1993","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hu et al., 2016] Hengyuan Hu, Rui Peng, Yu-Wing Tai, and Chi-Keung Tang. Network trimming: A data-driven neu- ron pruning approach towards efficient deep architectures. arXiv preprint arXiv:1607.03250, 2016.","title":"Network trimming: A data-driven neuron pruning approach towards efficient deep architectures","authors":"Hengyuan Hu, Rui Peng, Yu-Wing Tai, Chi-Keung Tang","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Jaderberg et al., 2014] Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. Speeding up convolutional neu- ral networks with low rank expansions. arXiv preprint arXiv:1405.3866, 2014.","title":"Speeding up convolutional neural networks with low rank expansions","authors":"Jaderberg","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Jia et al., 2014] Yangqing Jia, Evan Shelhamer, Jeff Don- ahue, Sergey Karayev, Jonathan Long, Ross Girshick, Ser- gio Guadarrama, and Trevor Darrell. Caffe: Convolutional architecture for fast feature embedding. In ACM MM, pages 675-678. ACM, 2014.","title":"Caffe: Convolutional architecture for fast feature embedding","authors":"Jia","year":"2014","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Kim et al., 2015] Yong-Deok Kim, Eunhyeok Park, Sungjoo Yoo, Taelim Choi, Lu Yang, and Dongjun Shin. Compression of deep convolutional neural networks for fast and low power mobile applications. arXiv preprint arXiv:1511.06530, 2015.","title":"Compression of deep convolutional neural networks for fast and low power mobile applications","authors":"Yong-Deok Kim, Eunhyeok Park, Sungjoo Yoo, Taelim Choi, Lu Yang, Dongjun Shin","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton","year":"2012","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Lebedev and Lempitsky, 2016] Vadim Lebedev and Victor Lempitsky. Fast convnets using group-wise brain damage. In CVPR, pages 2554-2564, 2016.","title":"Vadim Lebedev and Victor Lempitsky. Fast convnets using group-wise brain damage","authors":"Lempitsky Lebedev","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Lebedev et al., 2014] Vadim Lebedev, Yaroslav Ganin, Maksim Rakhuba, Ivan Oseledets, and Victor Lempitsky. Speeding-up convolutional neural networks using fine- tuned cp-decomposition. arXiv preprint arXiv:1412.6553, 2014.","title":"Speeding-up convolutional neural networks using finetuned cp-decomposition","authors":"Vadim Lebedev, Yaroslav Ganin, Maksim Rakhuba, Ivan Oseledets, Victor Lempitsky","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[LeCun et al., 1989] Yann LeCun, John S Denker, Sara A Solla, Richard E Howard, and Lawrence D Jackel. Op- timal brain damage. In NIPS, 1989.","title":"Optimal brain damage","authors":"Yann Lecun, John S Denker, Sara A Solla, Richard E Howard, Lawrence D Jackel","year":"1989","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Li et al., 2016] Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for ef- ficient convnets. arXiv preprint arXiv:1608.08710, 2016.","title":"Pruning filters for efficient convnets","authors":"Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, Hans Peter Graf","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Lin et al., 2016] Shaohui Lin, Rongrong Ji, Xiaowei Guo, and Xuelong Li. Towards convolutional neural networks compression via global error reconstruction. In IJCAI, pages 1573-1759, 2016.","title":"Towards convolutional neural networks compression via global error reconstruction","authors":"Lin","year":"2016","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2016/251"},{"position":24,"raw":"[Lin et al., 2017] Shaohui Lin, Rongrong Ji, Chao Chen, and Feiyue Huang. Espace: Accelerating convolutional neural networks via eliminating spatial & channel redundancy. In AAAI, pages 1424-1430, 2017.","title":"Espace: Accelerating convolutional neural networks via eliminating spatial & channel redundancy","authors":"Shaohui Lin, Rongrong Ji, Chao Chen, Feiyue Huang","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Liu et al., 2015] Baoyuan Liu, Min Wang, Hassan Foroosh, Marshall Tappen, and Marianna Pensky. Sparse convolu- tional neural networks. In CVPR, pages 806-814, 2015.","title":"Sparse convolutional neural networks","authors":"Liu","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Long et al., 2015] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for seman- tic segmentation. In CVPR, pages 3431-3440, 2015.","title":"Fully convolutional networks for semantic segmentation","authors":"Jonathan Long, Evan Shelhamer, Trevor Darrell","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Luo et al., 2017] Jianhao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural net- work compression. In ICCV, 2017.","title":"Thinet: A filter level pruning method for deep neural network compression","authors":"Luo","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Molchanov et al., 2017] Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convo- lutional neural networks for resource efficient inference. In ICLR, 2017.","title":"Pruning convolutional neural networks for resource efficient inference","authors":"Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, Jan Kautz","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Park et al., 2017] Jongsoo Park, Sheng Li, Wei Wen, Ping Tak Peter Tang, Hai Li, Yiran Chen, and Pradeep Dubey. Faster cnns with direct sparse convolutions and guided pruning. In IJCAI, 2017.","title":"Faster cnns with direct sparse convolutions and guided pruning","authors":"Jongsoo Park, Sheng Li, Wei Wen, Ping Tak, Peter Tang, Hai Li, Yiran Chen, Pradeep Dubey","year":"2017","venue":"IJCAI","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Rastegari et al., 2016] M. Rastegari, V. Ordonez, J. Red- mon, and A. Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. arXiv preprint arXiv:1603.05279, 2016.","title":"Xnor-net: Imagenet classification using binary convolutional neural networks","authors":"M Rastegari, V Ordonez, J Redmon, A Farhadi","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Russakovsky et al., 2015] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. Ima- geNet Large Scale Visual Recognition Challenge. Inter- national Journal of Computer Vision (IJCV), 115(3):211- 252, 2015.","title":"Ima-geNet Large Scale Visual Recognition Challenge","authors":"Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C Berg, Li Fei-Fei","year":"2015","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Wen et al., 2016] Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. In NIPS, 2016.","title":"Learning structured sparsity in deep neural networks","authors":"Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, Hai Li","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Wu et al., 2016] Jiaxiang Wu, Cong Leng, Yuhang Wang, Qinghao Hu, and Jian Cheng. Quantized convolutional neural networks for mobile devices. In CVPR, 2016.","title":"Quantized convolutional neural networks for mobile devices","authors":"Jiaxiang Wu, Cong Leng, Yuhang Wang, Qinghao Hu, Jian Cheng","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}