{"type":"ScholarlyArticle","title":"Feature Statistics Guided Efficient Filter Pruning","authors":[{"name":"Hang Li","slug":"hang-li","given":"Hang","family":"Li","affiliation":"School of Computer Science , McGill University"},{"name":"Chen Ma","slug":"chen-ma","given":"Chen","family":"Ma","affiliation":"School of Computer Science , McGill University"},{"name":"Wei Xu","slug":"wei-xu","given":"Wei","family":"Xu","affiliation":"Institute for Interdisciplinary Information Sciences , Tsinghua University"},{"name":"Xue Liu","slug":"xue-liu","given":"Xue","family":"Liu","affiliation":"School of Computer Science , McGill University"}],"abstract":"Building compact convolutional neural networks (CNNs) with reliable performance is a critical but challenging task, especially when deploying them in real-world applications. As a common approach to reduce the size of CNNs, pruning methods delete part of the CNN filters according to some metrics such as l1-norm. However, previous methods hardly leverage the information variance in a single feature map and the similarity characteristics among feature maps. In this paper, we propose a novel filter pruning method, which incorporates two kinds of feature map selections: diversity-aware selection (DFS) and similarity-aware selection (SFS). DFS aims to discover features with low information diversity while SFS removes features that have high similarities with others. We conduct extensive empirical experiments with various CNN architectures on publicly available datasets. The experimental results demonstrate that our model obtains up to 91.6% parameter decrease and 83.7% FLOPs reduction with almost no accuracy loss.","doi":"10.24963/ijcai.2020/363","url":"https://ijcai.org/proceedings/2020/363","pdf":"https://ijcai.org/proceedings/2020/0363.pdf","markdown":"https://ijcai.org/proceedings/2020/363.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/363","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":363,"pages":{"first":2619,"last":2625},"keywords":["Machine Learning: Deep Learning","Machine Learning: Feature Selection; Learning Sparse Models","Machine Learning: Deep Learning: Convolutional networks","Data Mining: Feature Extraction, Selection and Dimensionality Reduction"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24923/file_24923.m3u8","length_sec":280,"provider":"hls"}],"references":[{"position":1,"raw":"[Ba and Caruana, 2014] Jimmy Ba and Rich Caruana. Do deep nets really need to be deep? In Advances in neural information processing systems, pages 2654-2662, 2014.","title":"Do deep nets really need to be deep?","authors":"Caruana Ba, Jimmy Ba, Rich Caruana","year":"2014","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Galton, 1886] Francis Galton. Regression towards medi- ocrity in hereditary stature. The Journal of the Anthropo- logical Institute of Great Britain and Ireland, 15:246-263, 1886.","title":"Regression towards mediocrity in hereditary stature","authors":"Francis Galton, Galton","year":"1886","venue":"The Journal of the Anthropological Institute of Great Britain and Ireland","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Girshick et al., 2014] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 580-587, 2014.","title":"Rich feature hierarchies for accurate object detection and semantic segmentation","authors":"Ross Girshick, Jeff Donahue, Trevor Darrell, Jitendra Malik","year":"2014","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Guo et al., 2016] Yiwen Guo, Anbang Yao, and Yurong Chen. Dynamic network surgery for efficient dnns. In Ad- vances In Neural Information Processing Systems, pages 1379-1387, 2016.","title":"Dynamic network surgery for efficient dnns","authors":"Yiwen Guo, Anbang Yao, Yurong Chen","year":"2016","venue":"Advances In Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Han et al., 2015] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in neural informa- tion processing systems, pages 1135-1143, 2015.","title":"Learning both weights and connections for efficient neural network","authors":"Han","year":"2015","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[He et al., 2016a] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"He","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[He et al., 2016b] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual net- works. In European conference on computer vision, pages 630-645. Springer, 2016.","title":"Identity mappings in deep residual networks","authors":"He","year":"2016","venue":"European conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[He et al., 2017] Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural net- works. In Proceedings of the IEEE International Confer- ence on Computer Vision, pages 1389-1397, 2017.","title":"Channel pruning for accelerating very deep neural networks","authors":"Yihui He, Xiangyu Zhang, Jian Sun","year":"2017","venue":"Proceedings of the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[He et al., 2018] Yihui He, Ji Lin, Zhijian Liu, Hanrui Wang, Li-Jia Li, and Song Han. Amc: Automl for model com- pression and acceleration on mobile devices. In Pro- ceedings of the European Conference on Computer Vision (ECCV), pages 784-800, 2018.","title":"Amc: Automl for model compression and acceleration on mobile devices","authors":"Yihui He, Ji Lin, Zhijian Liu, Hanrui Wang, Li-Jia Li, Song Han","year":"2018","venue":"Proceedings of the European Conference on Computer Vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[He et al., 2019] Yang He, Ping Liu, Ziwei Wang, Zhilan Hu, and Yi Yang. Filter pruning via geometric median for deep convolutional neural networks acceleration. In Proceed- ings of the IEEE Conference on Computer Vision and Pat- tern Recognition, pages 4340-4349, 2019.","title":"Filter pruning via geometric median for deep convolutional neural networks acceleration","authors":"Yang He, Ping Liu, Ziwei Wang, Zhilan Hu, Yi Yang","year":"2019","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Howard et al., 2017] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mo- bilenets: Efficient convolutional neural networks for mo- bile vision applications. arXiv preprint arXiv:1704.04861, 2017.","title":"Mobilenets: Efficient convolutional neural networks for mobile vision applications","authors":"Menglong Andrew G Howard, Bo Zhu, Dmitry Chen, Weijun Kalenichenko, Tobias Wang, Marco Weyand, Hartwig Andreetto, Adam","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hu et al., 2016] Hengyuan Hu, Rui Peng, Yu-Wing Tai, and Chi-Keung Tang. Network trimming: A data-driven neu- ron pruning approach towards efficient deep architectures. CoRR, abs/1607.03250, 2016.","title":"Network trimming: A data-driven neuron pruning approach towards efficient deep architectures","authors":"Hengyuan Hu, Rui Peng, Yu-Wing Tai, Chi-Keung Tang","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kim, 2014] Yoon Kim. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Con- ference on Empirical Methods in Natural Language Pro- cessing (EMNLP), pages 1746-1751, Doha, Qatar, Octo- ber 2014. Association for Computational Linguistics.","title":"Convolutional neural networks for sentence classification","authors":"Yoon Kim","year":"2014","venue":"Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Krizhevsky et al., 2009] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny im- ages. Technical report, Citeseer, 2009.","title":"Learning multiple layers of features from tiny images","authors":"Alex Krizhevsky, Geoffrey Hinton","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural in- formation processing systems, pages 1097-1105, 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton","year":"2012","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[LeCun et al., 1990] Yann LeCun, John S Denker, and Sara A Solla. Optimal brain damage. In Advances in neu- ral information processing systems, pages 598-605, 1990.","title":"Optimal brain damage","authors":"Yann Lecun, John S Denker, Sara A Solla","year":"1990","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Li et al., 2016] Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for ef- ficient convnets. arXiv preprint arXiv:1608.08710, 2016.","title":"Pruning filters for efficient convnets","authors":"Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, Hans Peter Graf","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Liu et al., 2017] Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through net- work slimming. In Proceedings of the IEEE International Conference on Computer Vision, pages 2736-2744, 2017.","title":"Learning efficient convolutional networks through network slimming","authors":"Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, Changshui Zhang","year":"2017","venue":"Proceedings of the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Liu et al., 2018] Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. arXiv preprint arXiv:1810.05270, 2018.","title":"Rethinking the value of network pruning","authors":"Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, Trevor Darrell","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Liu et al., 2019] Zechun Liu, Haoyuan Mu, Xiangyu Zhang, Zichao Guo, Xin Yang, Kwang-Ting Cheng, and Jian Sun. Metapruning: Meta learning for automatic neural network channel pruning. In Proceedings of the IEEE International Conference on Computer Vision, pages 3296-3305, 2019.","title":"Metapruning: Meta learning for automatic neural network channel pruning","authors":"Liu","year":"2019","venue":"Proceedings of the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Luo and Wu, 2018] Jian-Hao Luo and Jianxin Wu. Au- topruner: An end-to-end trainable filter pruning method for efficient deep model inference. arXiv preprint arXiv:1805.08941, 2018.","title":"Autopruner: An end-to-end trainable filter pruning method for efficient deep model inference","authors":"Jian-Hao Luo, Jianxin Wu","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Luo et al., 2017] Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural network compression. In Proceedings of the IEEE interna- tional conference on computer vision, pages 5058-5066, 2017.","title":"Thinet: A filter level pruning method for deep neural network compression","authors":"Jian-Hao Luo, Jianxin Wu, Weiyao Lin","year":"2017","venue":"Proceedings of the IEEE international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Sutskever et al., 2013] Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. In International conference on machine learning, pages 1139-1147, 2013.","title":"On the importance of initialization and momentum in deep learning","authors":"Sutskever","year":"2013","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang et al., 2019] Wenxiao Wang, Cong Fu, Jishun Guo, Deng Cai, and Xiaofei He. Cop: Customized deep model compression via regularized correlation-based filter-level pruning. In International Joint Conference on Artificial Intelligence, volume 2019, 2019.","title":"Cop: Customized deep model compression via regularized correlation-based filter-level pruning","authors":"Wang","year":"2019","venue":"International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2019/525"},{"position":25,"raw":"[Yamamoto and Maeno, 2018] Kohei Yamamoto and Kurato Maeno. Pcas: Pruning channels with attention statis- tics for deep network compression. arXiv preprint arXiv:1806.05382, 2018.","title":"Pcas: Pruning channels with attention statistics for deep network compression","authors":"Maeno Yamamoto, Kohei Yamamoto, Kurato Maeno","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}