{"type":"ScholarlyArticle","title":"Crafting Efficient Neural Graph of Large Entropy","authors":[{"name":"Minjing Dong","slug":"minjing-dong","given":"Minjing","family":"Dong","affiliation":"School of Computer Science , Faculty of Engineering , University of Sydney , Australia"},{"name":"Hanting Chen","slug":"hanting-chen","given":"Hanting","family":"Chen","affiliation":"Key Laboratory of Machine Perception (Ministry of Education) , Peking University , China; Huawei Noah's Ark Lab"},{"name":"Yunhe Wang","slug":"yunhe-wang","given":"Yunhe","family":"Wang","affiliation":"Huawei Noah's Ark Lab"},{"name":"Chang Xu","slug":"chang-xu","given":"Chang","family":"Xu","affiliation":"School of Computer Science , Faculty of Engineering , University of Sydney , Australia"}],"abstract":"Network pruning is widely applied to deep CNN models due to their heavy computation costs and achieves high performance by keeping important weights while removing the redundancy. Pruning redundant weights directly may hurt global information flow, which suggests that an efficient sparse network should take graph properties into account. Thus, instead of paying more attention to preserving important weight, we focus on the pruned architecture itself. We propose to use graph entropy as the measurement, which shows useful properties to craft high-quality neural graphs and enables us to propose efficient algorithm to construct them as the initial network architecture. Our algorithm can be easily implemented and deployed to different popular CNN models and achieve better trade-offs.","doi":"10.24963/ijcai.2019/311","url":"https://ijcai.org/proceedings/2019/311","pdf":"https://ijcai.org/proceedings/2019/0311.pdf","markdown":"https://ijcai.org/proceedings/2019/311.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/311","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning A-L","track":"Main track","number":311,"pages":{"first":2244,"last":2250},"keywords":["Machine Learning: Deep Learning","Machine Learning: Unsupervised Learning"],"videos":[],"references":[{"position":1,"raw":"[Bansal et al., 2018] Nitin Bansal, Xiaohan Chen, and Zhangyang Wang. Can we gain more from orthogonal- ity regularizations in training deep networks? In Advances in Neural Information Processing Systems, pages 4261- 4271, 2018.","title":"Can we gain more from orthogonality regularizations in training deep networks?","authors":"Nitin Bansal, Xiaohan Chen, Zhangyang Wang","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Braunstein et al., 2004] Samuel Braunstein, Sibasish Ghosh, and Simone Severini. The laplacian of a graph as a density matrix: A basic combinatorial approach to separability of mixed states. Annals of Combinatorics, 10, 07 2004.","title":"The laplacian of a graph as a density matrix: A basic combinatorial approach to separability of mixed states","authors":"Braunstein","year":"2004","venue":"Annals of Combinatorics","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Deng et al., 2009] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierar- chical Image Database. In CVPR09, 2009.","title":"ImageNet: A Large-Scale Hierarchical Image Database","authors":"J Deng, W Dong, R Socher, L.-J Li, K Li, L Fei-Fei","year":"2009","venue":"CVPR09","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Han et al., 2015] Song Han, Jeff Pool, John Tran, and William J. Dally. Learning both weights and connec- tions for efficient neural networks. CoRR, abs/1506.02626, 2015.","title":"Learning both weights and connections for efficient neural networks","authors":"Han","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Hassibi and Stork, 1993] Babak Hassibi and David G. Stork. Second order derivatives for network pruning: Opti- mal brain surgeon. In S. J. Hanson, J. D. Cowan, and C. L. Giles, editors, Advances in Neural Information Processing Systems 5, pages 164-171.","title":"Second order derivatives for network pruning: Optimal brain surgeon","authors":"Babak Hassibi, David G Stork","year":null,"venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. CoRR, abs/1603.05027, 2016.","title":"Identity mappings in deep residual networks","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[He et al., 2017] Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural net- works. CoRR, abs/1707.06168, 2017.","title":"Channel pruning for accelerating very deep neural networks","authors":"Yihui He, Xiangyu Zhang, Jian Sun","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Huang et al., 2016] Gao Huang, Zhuang Liu, and Kilian Q. Weinberger. Densely connected convolutional networks. CoRR, abs/1608.06993, 2016.","title":"Densely connected convolutional networks","authors":"Gao Huang, Zhuang Liu, Kilian Q Weinberger","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Ioffe and Szegedy, 2015] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep net- work training by reducing internal covariate shift. CoRR, abs/1502.03167, 2015.","title":"Batch normalization: Accelerating deep network training by reducing internal covariate shift","authors":"Szegedy Ioffe, Sergey Ioffe, Christian Szegedy","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Krizhevsky, 2009] Alex Krizhevsky. Learning multiple lay- ers of features from tiny images. 2009.","title":"Learning multiple layers of features from tiny images","authors":"Alex Krizhevsky","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2016] Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. CoRR, abs/1608.08710, 2016.","title":"Pruning filters for efficient convnets","authors":"Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, Hans Peter Graf","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lin and Zhou, 2018] Hongying Lin and Bo Zhou. On the von neumann entropy of a graph. Discrete Applied Math- ematics, 247:448 -455, 2018.","title":"On the von neumann entropy of a graph","authors":"Hongying Lin, Bo Zhou","year":"2018","venue":"Discrete Applied Mathematics","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Liu et al., 2017] Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through net- work slimming. CoRR, abs/1708.06519, 2017.","title":"Learning efficient convolutional networks through network slimming","authors":"Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, Changshui Zhang","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu et al., 2018] Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. CoRR, abs/1810.05270, 2018.","title":"Rethinking the value of network pruning","authors":"Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, Trevor Darrell","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Luo et al., 2017] Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural network compression. CoRR, abs/1707.06342, 2017.","title":"Thinet: A filter level pruning method for deep neural network compression","authors":"Jian-Hao Luo, Jianxin Wu, Weiyao Lin","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Passerini and Severini, 2008] Filippo Passerini and Simone Severini. The von neumann entropy of networks. Univer- sity Library of Munich, Germany, MPRA Paper, 12 2008.","title":"The von neumann entropy of networks","authors":"Severini Passerini, Filippo Passerini, Simone Severini","year":"2008","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR, abs/1409.1556, 2014.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Wu et al., 2018] Junru Wu, Yue Wang, Zhenyu Wu, Zhangyang Wang, Ashok Veeraraghavan, and Yingyan Lin. Deep k-means: Re-training and parameter sharing with harder cluster assignments for compressing deep con- volutions. CoRR, abs/1806.09228, 2018.","title":"Deep k-means: Re-training and parameter sharing with harder cluster assignments for compressing deep convolutions","authors":"Junru Wu, Yue Wang, Zhenyu Wu, Zhangyang Wang, Ashok Veeraraghavan, Yingyan Lin","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Yang et al., 2016] Tien-Ju Yang, Yu-Hsin Chen, and Vivi- enne Sze. Designing energy-efficient convolutional neural networks using energy-aware pruning. CoRR, abs/1611.05128, 2016.","title":"Designing energy-efficient convolutional neural networks using energy-aware pruning","authors":"Tien-Ju Yang, Yu-Hsin Chen, Vivienne Sze","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[You et al., 2017] Shan You, Chang Xu, Chao Xu, and Dacheng Tao. Learning from multiple teacher networks. In Proceedings of the 23rd ACM SIGKDD International Con- ference on Knowledge Discovery and Data Mining, KDD '17, pages 1285-1294, New York, NY, USA, 2017. ACM.","title":"Learning from multiple teacher networks","authors":"Shan You, Chang Xu, Chao Xu, Dacheng Tao","year":"2017","venue":"Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '17","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Yu et al., 2017] Ruichi Yu, Ang Li, Chun-Fu Chen, Jui- Hsin Lai, Vlad I. Morariu, Xintong Han, Mingfei Gao, Ching-Yung Lin, and Larry S. Davis. NISP: pruning net- works using neuron importance score propagation. CoRR, abs/1711.05908, 2017.","title":"NISP: pruning networks using neuron importance score propagation","authors":"Yu","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zhang et al., 2018] Tianyun Zhang, Shaokai Ye, Kaiqi Zhang, Jian Tang, Wujie Wen, Makan Fardad, and Yanzhi Wang. A systematic DNN weight pruning framework us- ing alternating direction method of multipliers. CoRR, abs/1804.03294, 2018.","title":"A systematic DNN weight pruning framework using alternating direction method of multipliers","authors":"Zhang","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhou et al., 2016] Hao Zhou, Jose M. Alvarez, and Fatih Porikli. Less is more: Towards compact cnns. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision -ECCV 2016, pages 662-677, Cham, 2016. Springer International Publishing.","title":"Less is more: Towards compact cnns","authors":"Hao Zhou, Jose M Alvarez, Fatih Porikli","year":"2016","venue":"Computer Vision -ECCV 2016","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}