{"type":"ScholarlyArticle","title":"Play and Prune: Adaptive Filter Pruning for Deep Model Compression","authors":[{"name":"Pravendra Singh","slug":"pravendra-singh","given":"Pravendra","family":"Singh","affiliation":"Department of Computer Science and Engineering , IIT Kanpur , India"},{"name":"Vinay Kumar Verma","slug":"vinay-kumar-verma","given":"Vinay","family":"Kumar Verma","affiliation":"Department of Computer Science and Engineering , IIT Kanpur , India"},{"name":"Piyush Rai","slug":"piyush-rai","given":"Piyush","family":"Rai","affiliation":"Department of Computer Science and Engineering , IIT Kanpur , India"},{"name":"Vinay P. Namboodiri","slug":"vinay-p-namboodiri","given":"Vinay P.","family":"Namboodiri","affiliation":"Department of Computer Science and Engineering , IIT Kanpur , India"}],"abstract":"While convolutional neural networks (CNN) have achieved impressive performance on various classification/recognition tasks, they typically consist of a massive number of parameters. This results in significant memory requirement as well as computational overheads. Consequently, there is a growing need for filter-level pruning approaches for compressing CNN based models that not only reduce the total number of parameters but reduce the overall computation as well. We present a new min-max framework for filter-level pruning of CNNs. Our framework, called Play and Prune (PP), jointly prunes and fine-tunes CNN model parameters, with an adaptive pruning rate, while maintaining the model's predictive performance. Our framework consists of two modules: (1) An adaptive filter pruning (AFP) module, which minimizes the number of filters in the model; and (2) A pruning rate controller (PRC) module, which maximizes the accuracy during pruning. Moreover, unlike most previous approaches, our approach allows directly specifying the desired error tolerance instead of pruning level. Our compressed models can be deployed at run-time, without requiring any special libraries or hardware. Our approach reduces the number of parameters of VGG-16 by an impressive factor of 17.5X, and number of FLOPS by 6.43X, with no loss of accuracy, significantly outperforming other state-of-the-art filter pruning methods.","doi":"10.24963/ijcai.2019/480","url":"https://ijcai.org/proceedings/2019/480","pdf":"https://ijcai.org/proceedings/2019/0480.pdf","markdown":"https://ijcai.org/proceedings/2019/480.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/480","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning M-Z","track":"Main track","number":480,"pages":{"first":3460,"last":3466},"keywords":["Machine Learning: Feature Selection ; Learning Sparse Models","Machine Learning: Deep Learning","Computer Vision: Computer Vision","Machine Learning: Classification"],"videos":[],"references":[{"position":1,"raw":"[Alvarez and Salzmann, 2016] Jose M Alvarez and Mathieu Salzmann. Learning the number of neurons in deep net- works. In NIPS, pages 2270-2278, 2016.","title":"Learning the number of neurons in deep networks","authors":"Salzmann ; Alvarez, M Jose, Mathieu Alvarez, Salzmann","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2015] Wenlin Chen, James Wilson, Stephen Tyree, Kilian Weinberger, and Yixin Chen. Compressing neural networks with the hashing trick. In ICML, pages 2285-2294, 2015.","title":"Compressing neural networks with the hashing trick","authors":"Wenlin Chen, James Wilson, Stephen Tyree, Kilian Weinberger, Yixin Chen","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chollet, 2017] Francois Chollet. Xception: Deep learning with depthwise separable convolutions. In CVPR, pages 1251-1258, 2017.","title":"Xception: Deep learning with depthwise separable convolutions","authors":"Francois Chollet","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Ding et al., 2018] Xiaohan Ding, Guiguang Ding, Jungong Han, and Sheng Tang. Auto-balanced filter pruning for efficient convolutional neural networks. AAAI, 2018.","title":"Auto-balanced filter pruning for efficient convolutional neural networks","authors":"Xiaohan Ding, Guiguang Ding, Jungong Han, Sheng Tang","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Han et al., 2015a] Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural net- works with pruning, trained quantization and huffman cod- ing. arXiv preprint arXiv:1510.00149, 2015.","title":"Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding","authors":"Han","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Han et al., 2015b] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In NIPS, pages 1135-1143, 2015.","title":"Learning both weights and connections for efficient neural network","authors":"Han","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[He et al., 2017] Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural net- works. In ICCV, volume 2, page 6, 2017.","title":"Channel pruning for accelerating very deep neural networks","authors":"Yihui He, Xiangyu Zhang, Jian Sun","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[He et al., 2018] Yang He, Guoliang Kang, Xuanyi Dong, Yanwei Fu, and Yi Yang. Soft filter pruning for acceler- ating deep convolutional neural networks. IJCAI, 2018.","title":"Soft filter pruning for accelerating deep convolutional neural networks","authors":"Yang He, Guoliang Kang, Xuanyi Dong, Yanwei Fu, Yi Yang","year":"2018","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2018/309"},{"position":10,"raw":"[Hu et al., 2016] Hengyuan Hu, Rui Peng, Yu-Wing Tai, and Chi-Keung Tang. Network trimming: A data-driven neu- ron pruning approach towards efficient deep architectures. arXiv preprint arXiv:1607.03250, 2016.","title":"Network trimming: A data-driven neuron pruning approach towards efficient deep architectures","authors":"Hengyuan Hu, Rui Peng, Yu-Wing Tai, Chi-Keung Tang","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Krizhevsky and Hinton, 2009] Alex Krizhevsky and Geof- frey Hinton. Learning multiple layers of features from tiny images. 2009.","title":"Learning multiple layers of features from tiny images","authors":"Alex Krizhevsky, Geoffrey Hinton","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, pages 1097- 1105, 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton","year":"2012","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[LeCun et al., 1998] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning ap- plied to document recognition. Proceedings of the IEEE, pages 2278-2324, 1998.","title":"Gradient-based learning applied to document recognition","authors":"Yann Lecun, Léon Bottou, Yoshua Bengio, Patrick Haffner","year":"1998","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li et al., 2017] Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. ICLR, 2017.","title":"Pruning filters for efficient convnets","authors":"Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, Hans Peter Graf","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lin et al., 2014] Tsung-Yi Lin, Michael Maire, Serge Be- longie, James Hays, Pietro Perona, Deva Ramanan, Pi- otr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, pages 740-755. Springer, 2014.","title":"Microsoft coco: Common objects in context","authors":"Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, C Lawrence, Zitnick","year":"2014","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lin et al., 2017a] Ji Lin, Yongming Rao, Jiwen Lu, and Jie Zhou. Runtime neural pruning. In Advances in Neural Information Processing Systems, pages 2181-2191, 2017.","title":"Runtime neural pruning","authors":"Lin","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lin et al., 2017b] Tsung-Yi Lin, Piotr Dollár, Ross B Gir- shick, Kaiming He, Bharath Hariharan, and Serge J Be- longie. Feature pyramid networks for object detection. In CVPR, page 4, 2017.","title":"Feature pyramid networks for object detection","authors":"Lin","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Luo and Wu, 2017] Jian-Hao Luo and Jianxin Wu. An entropy-based pruning method for cnn compression. arXiv preprint arXiv:1706.05791, 2017.","title":"An entropy-based pruning method for cnn compression","authors":"Jian-Hao Luo, Jianxin Wu","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Luo et al., 2017] Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural network compression. ICCV, 2017.","title":"Thinet: A filter level pruning method for deep neural network compression","authors":"Jian-Hao Luo, Jianxin Wu, Weiyao Lin","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Molchanov et al., 2017] Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convo- lutional neural networks for resource efficient inference. ICLR, 2017.","title":"Pruning convolutional neural networks for resource efficient inference","authors":"Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, Jan Kautz","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Neklyudov et al., 2017] Kirill Neklyudov, Dmitry Molchanov, Arsenii Ashukha, and Dmitry P Vetrov. Structured bayesian pruning via log-normal multiplicative noise. In NIPS, pages 6778-6787, 2017.","title":"Structured bayesian pruning via log-normal multiplicative noise","authors":"Kirill Neklyudov, Dmitry Molchanov, Arsenii Ashukha, Dmitry P Vetrov","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Rastegari et al., 2016] Mohammad Rastegari, Vicente Or- donez, Joseph Redmon, and Ali Farhadi. Xnor-net: Ima- genet classification using binary convolutional neural net- works. In ECCV, pages 525-542. Springer, 2016.","title":"Xnor-net: Imagenet classification using binary convolutional neural networks","authors":"Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, Ali Farhadi","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Ren et al., 2015] Shaoqing Ren, Kaiming He, Ross Gir- shick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NIPS, pages 91-99, 2015.","title":"Faster r-cnn: Towards real-time object detection with region proposal networks","authors":"Ren","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Russakovsky et al., 2015] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. IJCV, 115(3):211-252, 2015.","title":"Imagenet large scale visual recognition challenge","authors":"Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein","year":"2015","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Simonyan and Zisserman, 2015] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. ICLR, 2015.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wen et al., 2016] Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. In NIPS, pages 2074-2082, 2016.","title":"Learning structured sparsity in deep neural networks","authors":"Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, Hai Li","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Yu et al., 2018] Ruichi Yu, Ang Li, Chun-Fu Chen, Jui- Hsin Lai, Vlad I Morariu, Xintong Han, Mingfei Gao, Ching-Yung Lin, and Larry S Davis. Nisp: Pruning net- works using neuron importance score propagation. CVPR, 2018.","title":"Nisp: Pruning networks using neuron importance score propagation","authors":"Yu","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhou et al., 2016] Hao Zhou, Jose M Alvarez, and Fatih Porikli. Less is more: Towards compact cnns. In ECCV, pages 662-677. Springer, 2016.","title":"Less is more: Towards compact cnns","authors":"Hao Zhou, Jose M Alvarez, Fatih Porikli","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}