{"type":"ScholarlyArticle","title":"COP: Customized Deep Model Compression via Regularized Correlation-Based Filter-Level Pruning","authors":[{"name":"Wenxiao Wang","slug":"wenxiao-wang","given":"Wenxiao","family":"Wang","affiliation":"State Key Lab of CAD&CG , Zhejiang University , Hangzhou , China; Fabu Inc. , Hangzhou , China"},{"name":"Cong Fu","slug":"cong-fu","given":"Cong","family":"Fu","affiliation":"State Key Lab of CAD&CG , Zhejiang University , Hangzhou , China; Alibaba-Zhejiang University Joint Institute of Frontier Technologies , Hangzhou , China"},{"name":"Jishun Guo","slug":"jishun-guo","given":"Jishun","family":"Guo","affiliation":"GAC R&D Center , Guangzhou , China"},{"name":"Deng Cai","slug":"deng-cai","given":"Deng","family":"Cai","affiliation":"State Key Lab of CAD&CG , Zhejiang University , Hangzhou , China; Fabu Inc. , Hangzhou , China"},{"name":"Xiaofei He","slug":"xiaofei-he","given":"Xiaofei","family":"He","affiliation":"State Key Lab of CAD&CG , Zhejiang University , Hangzhou , China; Fabu Inc. , Hangzhou , China"}],"abstract":"Neural network compression empowers the effective yet unwieldy deep convolutional neural networks (CNN) to be deployed in resource-constrained scenarios. Most state-of-the-art approaches prune the model in filter-level according to the \"importance\" of filters. Despite their success, we notice they suffer from at least two of the following problems: 1) The redundancy among filters is not considered because the importance is evaluated independently. 2) Cross-layer filter comparison is unachievable since the importance is defined locally within each layer. Consequently, we must manually specify layer-wise pruning ratios. 3) They are prone to generate sub-optimal solutions because they neglect the inequality between reducing parameters and reducing computational cost. Reducing the same number of parameters in different positions in the network may reduce different computational cost. To address the above problems, we develop a novel algorithm named as COP (correlation-based pruning), which can detect the redundant filters efficiently. We enable the cross-layer filter comparison through global normalization. We add parameter-quantity and computational-cost regularization terms to the importance, which enables the users to customize the compression according to their preference (smaller or faster). Extensive experiments have shown COP outperforms the others significantly. The code is released at https://github.com/ZJULearning/COP.","doi":"10.24963/ijcai.2019/525","url":"https://ijcai.org/proceedings/2019/525","pdf":"https://ijcai.org/proceedings/2019/0525.pdf","markdown":"https://ijcai.org/proceedings/2019/525.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/525","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning M-Z","track":"Main track","number":525,"pages":{"first":3785,"last":3791},"keywords":["Computer Vision: Computer Vision","Machine Learning: Deep Learning"],"videos":[],"references":[{"position":1,"raw":"[Denil et al., 2013] Misha Denil, Babak Shakibi, Laurent Dinh, Nando De Freitas, et al. Predicting parameters in deep learning. In Advances in Neural Information Pro- cessing Systems, pages 2148-2156, 2013.","title":"Predicting parameters in deep learning","authors":"Misha Denil, Babak Shakibi, Laurent Dinh, Nando De Freitas","year":"2013","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Dong et al., 2017] Xin Dong, Shangyu Chen, and Sinno Pan. Learning to prune deep neural networks via layer- wise optimal brain surgeon. In Advances in Neural Infor- mation Processing Systems, pages 4857-4867, 2017.","title":"Learning to prune deep neural networks via layerwise optimal brain surgeon","authors":"Xin Dong, Shangyu Chen, Sinno Pan","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Guo et al., 2016] Yiwen Guo, Anbang Yao, and Yurong Chen. Dynamic network surgery for efficient dnns. In Ad- vances In Neural Information Processing Systems, pages 1379-1387, 2016.","title":"Dynamic network surgery for efficient dnns","authors":"Yiwen Guo, Anbang Yao, Yurong Chen","year":"2016","venue":"Advances In Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Han et al., 2015] Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural net- works with pruning, trained quantization and huffman cod- ing. arXiv preprint arXiv:1510.00149, 2015.","title":"Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding","authors":"Han","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European Conference on Computer Vision, pages 630-645. Springer, 2016.","title":"Identity mappings in deep residual networks","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[He et al., 2017] Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural net- works. In International Conference on Computer Vision, volume 2, 2017.","title":"Channel pruning for accelerating very deep neural networks","authors":"Yihui He, Xiangyu Zhang, Jian Sun","year":"2017","venue":"International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[He et al., 2018a] Yang He, Guoliang Kang, Xuanyi Dong, Yanwei Fu, and Yi Yang. Soft filter pruning for accelerat- ing deep convolutional neural networks. In International Joint Conference on Artificial Intelligence, pages 2234- 2240. AAAI Press, 2018.","title":"Soft filter pruning for accelerating deep convolutional neural networks","authors":"He","year":"2018","venue":"International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2018/309"},{"position":8,"raw":"[He et al., 2018b] Yang He, Ping Liu, Ziwei Wang, and Yi Yang. Pruning filter via geometric median for deep convolutional neural networks acceleration. arXiv preprint arXiv:1811.00250, 2018.","title":"Pruning filter via geometric median for deep convolutional neural networks acceleration","authors":"He","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Howard et al., 2017] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mo- bilenets: Efficient convolutional neural networks for mo- bile vision applications. arXiv preprint arXiv:1704.04861, 2017.","title":"Mobilenets: Efficient convolutional neural networks for mobile vision applications","authors":"Menglong Andrew G Howard, Bo Zhu, Dmitry Chen, Weijun Kalenichenko, Tobias Wang, Marco Weyand, Hartwig Andreetto, Adam","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jiang et al., 2018] Chunhui Jiang, Guiying Li, Chao Qian, and Ke Tang. Efficient dnn neuron pruning by minimizing layer-wise nonlinear reconstruction error. In International Joint Conference on Artificial Intelligence, volume 2018, pages 2-2, 2018.","title":"Efficient dnn neuron pruning by minimizing layer-wise nonlinear reconstruction error","authors":"Chunhui Jiang, Guiying Li, Chao Qian, Ke Tang","year":"2018","venue":"International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2018/318"},{"position":11,"raw":"[Krizhevsky and Hinton, 2009] Alex Krizhevsky and Geof- frey Hinton. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.","title":"Learning multiple layers of features from tiny images","authors":"Alex Krizhevsky, Geoffrey Hinton","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2016] Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for ef- ficient convnets. arXiv preprint arXiv:1608.08710, 2016.","title":"Pruning filters for efficient convnets","authors":"Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, Hans Peter Graf","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Liu et al., 2017] Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through net- work slimming. In IEEE International Conference on Computer Vision, pages 2755-2763. IEEE, 2017.","title":"Learning efficient convolutional networks through network slimming","authors":"Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, Changshui Zhang","year":"2017","venue":"IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Luo et al., 2017] Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural network compression. In IEEE International Conference on Computer Vision, pages 5068-5076. IEEE, 2017.","title":"Thinet: A filter level pruning method for deep neural network compression","authors":"Jian-Hao Luo, Jianxin Wu, Weiyao Lin","year":"2017","venue":"IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Molchanov et al., 2016] Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convo- lutional neural networks for resource efficient inference. arXiv preprint arXiv:1611.06440, 2016.","title":"Pruning convolutional neural networks for resource efficient inference","authors":"Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, Jan Kautz","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Russakovsky et al., 2015] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recogni- tion challenge. International Journal of Computer Vision, 115(3):211-252, 2015.","title":"Imagenet large scale visual recognition challenge","authors":"Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein","year":"2015","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Suau et al., 2018] Xavier Suau, Luca Zappella, and Nicholas Apostoloff. Network compression using cor- relation analysis of layer responses. arXiv preprint arXiv:1807.10585, 2018.","title":"Network compression using correlation analysis of layer responses","authors":"Xavier Suau, Luca Zappella, Nicholas Apostoloff","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Yu et al., 2018] Ruichi Yu, Ang Li, Chun-Fu Chen, Jui- Hsin Lai, Vlad I Morariu, Xintong Han, Mingfei Gao, Ching-Yung Lin, and Larry S Davis. Nisp: Pruning net- works using neuron importance score propagation. In IEEE Conference on Computer Vision and Pattern Recog- nition, pages 9194-9203, 2018.","title":"Nisp: Pruning networks using neuron importance score propagation","authors":"Yu","year":"2018","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}