{"type":"ScholarlyArticle","title":"Where to Prune: Using LSTM to Guide End-to-end Pruning","authors":[{"name":"Jing Zhong","slug":"jing-zhong","given":"Jing","family":"Zhong","affiliation":"† Beijing National Laboratory for Information Science and Technology (BNList) School of Software , Tsinghua University , Beijing 100084 , China"},{"name":"Guiguang Ding","slug":"guiguang-ding","given":"Guiguang","family":"Ding","affiliation":"† Beijing National Laboratory for Information Science and Technology (BNList) School of Software , Tsinghua University , Beijing 100084 , China"},{"name":"Yuchen Guo","slug":"yuchen-guo","given":"Yuchen","family":"Guo","affiliation":"† Beijing National Laboratory for Information Science and Technology (BNList) School of Software , Tsinghua University , Beijing 100084 , China"},{"name":"Jungong Han","slug":"jungong-han","given":"Jungong","family":"Han","affiliation":"‡ School of Computing & Communications , Lancaster University , UK"},{"name":"Bin Wang","slug":"bin-wang","given":"Bin","family":"Wang","affiliation":""}],"abstract":"Recent years have witnessed the great success of convolutional neural networks (CNNs) in many related fields. However, its huge model size and computation complexity bring in difficulty when deploying CNNs in some scenarios, like embedded system with low computation power. To address this issue, many works have been proposed to prune filters in CNNs to reduce computation. However, they mainly focus on seeking which filters are unimportant in a layer and then prune filters layer by layer or globally. In this paper, we argue that the pruning order is also very significant for model pruning. We propose a novel approach to figure out which layers should be pruned in each step.  First, we utilize a long short-term memory (LSTM) to learn the hierarchical characteristics of a network and generate a pruning decision for each layer, which is the main difference from previous works. Next, a channel-based method is adopted to evaluate the importance of filters in a to-be-pruned layer, followed by an accelerated recovery step. Experimental results demonstrate that our approach is capable of reducing 70.1% FLOPs for VGG and 47.5% for Resnet-56 with comparable accuracy. Also, the learning results seem to reveal the sensitivity of each network layer.","doi":"10.24963/ijcai.2018/445","url":"https://ijcai.org/proceedings/2018/445","pdf":"https://ijcai.org/proceedings/2018/0445.pdf","markdown":"https://ijcai.org/proceedings/2018/445.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/445","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Machine Learning","track":"Main track","number":445,"pages":{"first":3205,"last":3211},"keywords":["Machine Learning: Deep Learning","Computer Vision: Computer Vision","Machine Learning: Reinforcement Learning"],"videos":[],"references":[{"position":1,"raw":"[Ba and Caruana, 2014] Jimmy Ba and Rich Caruana. Do deep nets really need to be deep? In Advances in neural information processing systems, pages 2654-2662, 2014.","title":"Do deep nets really need to be deep?","authors":"Caruana Ba, Jimmy Ba, Rich Caruana","year":"2014","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Baker et al., 2016] Bowen Baker, Otkrist Gupta, Nikhil Naik, and Ramesh Raskar. Designing neural network ar- chitectures using reinforcement learning. In ICLR, 2016.","title":"Designing neural network architectures using reinforcement learning","authors":"Bowen Baker, Otkrist Gupta, Nikhil Naik, Ramesh Raskar","year":"2016","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cun et al., 1990] Yann Le Cun, John S. Denker, and Sara A. Solla. Optimal brain damage. In Advances in neural in- formation processing systems, volume 2, pages 598-605, 1990.","title":"Optimal brain damage","authors":"Yann Le Cun, John S Denker, Sara A Solla","year":"1990","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Denton et al., 2014] Emily L Denton, Wojciech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus. Exploiting linear structure within convolutional networks for efficient evaluation. In Advances in neural information processing systems, pages 1269-1277, 2014.","title":"Exploiting linear structure within convolutional networks for efficient evaluation","authors":"Wojciech Emily L Denton, Joan Zaremba, Yann Bruna, Rob Lecun, Fergus","year":"2014","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Girshick, 2015] Ross Girshick. Fast r-cnn. In ICCV, pages 1440-1448, 2015.","title":"Fast r-cnn","authors":"Ross Girshick","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Han et al., 2016] Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural net- works with pruning, trained quantization and huffman cod- ing. In ICLR, 2016.","title":"Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding","authors":"Han","year":"2016","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hassibi and Stork, 1993] Babak Hassibi and David G Stork. Second order derivatives for network pruning: Optimal brain surgeon. In Advances in neural information process- ing systems, pages 164-171, 1993.","title":"Second order derivatives for network pruning: Optimal brain surgeon","authors":"Babak Hassibi, David G Stork","year":"1993","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[He et al., 2015] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpass- ing human-level performance on imagenet classification. In ICCV, 2015.","title":"Delving deep into rectifiers: Surpassing human-level performance on imagenet classification","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735-1780, 1997.","title":"Long short-term memory","authors":"Schmidhuber Hochreiter, Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hu et al., 2016] Hengyuan Hu, Rui Peng, Yu-Wing Tai, and Chi-Keung Tang. Network trimming: A data-driven neu- ron pruning approach towards efficient deep architectures. arXiv preprint arXiv:1607.03250, 2016.","title":"Network trimming: A data-driven neuron pruning approach towards efficient deep architectures","authors":"Hengyuan Hu, Rui Peng, Yu-Wing Tai, Chi-Keung Tang","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Ioffe and Szegedy, 2015] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep net- work training by reducing internal covariate shift. In ICML, pages 448-456, 2015.","title":"Batch normalization: Accelerating deep network training by reducing internal covariate shift","authors":"Szegedy Ioffe, Sergey Ioffe, Christian Szegedy","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Krizhevsky and Hinton, 2009] Alex Krizhevsky and Geof- frey Hinton. Learning multiple layers of features from tiny images. Tech Report, 2009.","title":"Learning multiple layers of features from tiny images","authors":"Alex Krizhevsky, Geoffrey Hinton","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural in- formation processing systems, pages 1097-1105, 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton","year":"2012","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[LeCun et al., 1990] Yann LeCun, Bernhard E Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne E Hubbard, and Lawrence D Jackel. Handwritten digit recognition with a back-propagation network. In Advances in neural information processing systems, pages 396-404, 1990.","title":"Handwritten digit recognition with a back-propagation network","authors":"Yann Lecun, Bernhard E Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne E Hubbard, Lawrence D Jackel","year":"1990","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Li et al., 2017] Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. In ICLR, 2017.","title":"Pruning filters for efficient convnets","authors":"Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, Hans Peter Graf","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liu et al., 2017] Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through net- work slimming. arXiv preprint arXiv:1708.06519, 2017.","title":"Learning efficient convolutional networks through network slimming","authors":"Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, Changshui Zhang","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Luo et al., 2017] Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural network compression. arXiv preprint arXiv:1707.06342, 2017.","title":"Thinet: A filter level pruning method for deep neural network compression","authors":"Jian-Hao Luo, Jianxin Wu, Weiyao Lin","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Molchanov et al., 2017] Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolu- tional neural networks for resource efficient transfer learn- ing. In ICLR, 2017.","title":"Pruning convolutional neural networks for resource efficient transfer learning","authors":"Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, Jan Kautz","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Rastegari et al., 2016] Mohammad Rastegari, Vicente Or- donez, Joseph Redmon, and Ali Farhadi. Xnor-net: Ima- genet classification using binary convolutional neural net- works. In ECCV, pages 525-542. Springer, 2016.","title":"Xnor-net: Imagenet classification using binary convolutional neural networks","authors":"Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, Ali Farhadi","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Scardapane et al., 2017] Simone Scardapane, Danilo Com- miniello, Amir Hussain, and Aurelio Uncini. Group sparse regularization for deep neural networks. Neurocomputing, 2017.","title":"Group sparse regularization for deep neural networks","authors":"Simone Scardapane, Danilo Comminiello, Amir Hussain, Aurelio Uncini","year":"2017","venue":"Neurocomputing","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Schroff et al., 2015] Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In CVPR, pages 815-823, 2015.","title":"Facenet: A unified embedding for face recognition and clustering","authors":"Schroff","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Simonyan and Zisserman, 2015] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wen et al., 2016] Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. In Advances in neural information processing systems, pages 2074-2082, 2016.","title":"Learning structured sparsity in deep neural networks","authors":"Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, Hai Li","year":"2016","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Williams, 1992] Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforce- ment learning. Machine learning, 8(3-4):229-256, 1992.","title":"Simple statistical gradient-following algorithms for connectionist reinforcement learning","authors":"Williams Ronald","year":"1992","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Xie and Yuille, 2017] Lingxi Xie and Alan Yuille. Genetic cnn. arXiv preprint arXiv:1703.01513, 2017.","title":"Lingxi Xie and Alan Yuille","authors":"Yuille Xie","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang et al., 2017] Xuan Zhang, Hao Luo, Xing Fan, Weilai Xiang, Yixiao Sun, Qiqi Xiao, Wei Jiang, Chi Zhang, and Jian Sun. Alignedreid: Surpassing human- level performance in person re-identification. arXiv preprint arXiv:1711.08184, 2017.","title":"Alignedreid: Surpassing humanlevel performance in person re-identification","authors":"Xuan Zhang, Hao Luo, Xing Fan, Weilai Xiang, Yixiao Sun, Qiqi Xiao, Wei Jiang, Chi Zhang, Jian Sun","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zoph and Le, 2017] Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. In ICLR, 2017.","title":"Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning","authors":"Le Zoph","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zoph et al., 2017] Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable archi- tectures for scalable image recognition. arXiv preprint arXiv:1707.07012, 2017.","title":"Learning transferable architectures for scalable image recognition","authors":"Barret Zoph, Vijay Vasudevan, Jonathon Shlens, Quoc V Le","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}