{"type":"ScholarlyArticle","title":"Improving Deep Neural Network Sparsity through Decorrelation Regularization","authors":[{"name":"Xiaotian Zhu","slug":"xiaotian-zhu","given":"Xiaotian","family":"Zhu","affiliation":"CAS Key Laboratory of Technology in Geo-spatial Information Processing and Application System , EEIS Department , University of Science and Technology of China"},{"name":"Wengang Zhou","slug":"wengang-zhou","given":"Wengang","family":"Zhou","affiliation":"CAS Key Laboratory of Technology in Geo-spatial Information Processing and Application System , EEIS Department , University of Science and Technology of China"},{"name":"Houqiang Li","slug":"houqiang-li","given":"Houqiang","family":"Li","affiliation":"CAS Key Laboratory of Technology in Geo-spatial Information Processing and Application System , EEIS Department , University of Science and Technology of China"}],"abstract":"Modern deep learning models usually suffer high complexity in model size and computation when transplanted to resource constrained platforms. To this end, many works are dedicated to compressing deep neural networks. Adding group LASSO regularization is one of the most effective model compression methods since it generates structured sparse networks. We investigate the deep neural networks trained by group LASSO constraint and observe that even with strong sparsity regularization imposed, there still exists substantial filter correlation among the convolution filters, which is undesired for a compact neural network. We propose to suppress such correlation with a new kind of constraint called decorrelation regularization, which explicitly forces the network to learn a set of less correlated filters. The experiments on CIFAR10/100 and ILSVRC2012 datasets show that when combined our decorrelation regularization with group LASSO, the correlation between filters could be effectively weakened, which increases the sparsity of the resulting model and leads to better compressing performance.","doi":"10.24963/ijcai.2018/453","url":"https://ijcai.org/proceedings/2018/453","pdf":"https://ijcai.org/proceedings/2018/0453.pdf","markdown":"https://ijcai.org/proceedings/2018/453.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/453","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Machine Learning","track":"Main track","number":453,"pages":{"first":3264,"last":3270},"keywords":["Machine Learning: Feature Selection ; Learning Sparse Models","Machine Learning: Neural Networks","Machine Learning: Deep Learning"],"videos":[],"references":[{"position":1,"raw":"[Cai et al., 2016] Xingyang Cai, Wengang Zhou, Lei Wu, Jiebo Luo, and Houqiang Li. Effective active skeleton representation for low latency human action recognition. IEEE TMM, 18(2):141-154, 2016.","title":"Effective active skeleton representation for low latency human action recognition","authors":"Cai","year":"2016","venue":"IEEE TMM","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Courbariaux et al., 2015] Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Binaryconnect: Training deep neural networks with binary weights during propaga- tions. In NIPS, pages 3123-3131, 2015.","title":"Binaryconnect: Training deep neural networks with binary weights during propagations","authors":"Matthieu Courbariaux, Yoshua Bengio, Jean-Pierre David","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Deng et al., 2009] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierar- chical Image Database. In CVPR, 2009.","title":"ImageNet: A Large-Scale Hierarchical Image Database","authors":"J Deng, W Dong, R Socher, L.-J Li, K Li, L Fei-Fei","year":"2009","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Han et al., 2015] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In NIPS, pages 1135-1143, 2015.","title":"Learning both weights and connections for efficient neural network","authors":"Han","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[He et al., 2015] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpass- ing human-level performance on imagenet classification. In ICCV, pages 1026-1034, 2015.","title":"Delving deep into rectifiers: Surpassing human-level performance on imagenet classification","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pages 770-778. IEEE, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[He et al., 2017] Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural net- works. In ICCV, volume 2, page 6, 2017.","title":"Channel pruning for accelerating very deep neural networks","authors":"Yihui He, Xiangyu Zhang, Jian Sun","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Howard et al., 2017] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mo- bilenets: Efficient convolutional neural networks for mo- bile vision applications. arXiv preprint arXiv:1704.04861, 2017.","title":"Mobilenets: Efficient convolutional neural networks for mobile vision applications","authors":"Menglong Andrew G Howard, Bo Zhu, Dmitry Chen, Weijun Kalenichenko, Tobias Wang, Marco Weyand, Hartwig Andreetto, Adam","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hu et al., 2016] Hengyuan Hu, Rui Peng, Yu-Wing Tai, and Chi-Keung Tang. Network trimming: A data-driven neu- ron pruning approach towards efficient deep architectures. arXiv preprint arXiv:1607.03250, 2016.","title":"Network trimming: A data-driven neuron pruning approach towards efficient deep architectures","authors":"Hengyuan Hu, Rui Peng, Yu-Wing Tai, Chi-Keung Tang","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Huang and Wang, 2017] Zehao Huang and Naiyan Wang. Data-driven sparse structure selection for deep neural net- works. In NIPS, 2017.","title":"Data-driven sparse structure selection for deep neural networks","authors":"Wang ; Huang, Zehao Huang, Naiyan Wang","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kim et al., 2015] Yong-Deok Kim, Eunhyeok Park, Sungjoo Yoo, Taelim Choi, Lu Yang, and Dongjun Shin. Compression of deep convolutional neural networks for fast and low power mobile applications. arXiv preprint arXiv:1511.06530, 2015.","title":"Compression of deep convolutional neural networks for fast and low power mobile applications","authors":"Yong-Deok Kim, Eunhyeok Park, Sungjoo Yoo, Taelim Choi, Lu Yang, Dongjun Shin","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Krizhevsky and Hinton, 2009] Alex Krizhevsky and Geof- frey Hinton. Learning multiple layers of features from tiny images. Master's thesis, Department of Computer Science, University of Toronto, 2009.","title":"Learning multiple layers of features from tiny images","authors":"Alex Krizhevsky, Geoffrey Hinton","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, pages 1097- 1105, 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton","year":"2012","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lebedev et al., 2015] Vadim Lebedev, Yaroslav Ganin, Maksim Rakhuba, Ivan Oseledets, and Victor Lempit- sky. Speeding-up convolutional neural networks using fine-tuned cp-decomposition. In ICLR, 2015.","title":"Speeding-up convolutional neural networks using fine-tuned cp-decomposition","authors":"Vadim Lebedev, Yaroslav Ganin, Maksim Rakhuba, Ivan Oseledets, Victor Lempitsky","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2017] Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. In ICLR, 2017.","title":"Pruning filters for efficient convnets","authors":"Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, Hans Peter Graf","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lin et al., 2016] Shaohui Lin, Rongrong Ji, Xiaowei Guo, Xuelong Li, et al. Towards convolutional neural networks compression via global error reconstruction. In IJCAI, pages 1753-1759, 2016.","title":"Towards convolutional neural networks compression via global error reconstruction","authors":"Lin","year":"2016","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2016/251"},{"position":17,"raw":"[Liu et al., 2017] Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through net- work slimming. In ICCV, pages 2755-2763. IEEE, 2017.","title":"Learning efficient convolutional networks through network slimming","authors":"Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, Changshui Zhang","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Luo and Wu, 2017] Jian-Hao Luo and Jianxin Wu. An entropy-based pruning method for cnn compression. arXiv preprint arXiv:1706.05791, 2017.","title":"An entropy-based pruning method for cnn compression","authors":"Jian-Hao Luo, Jianxin Wu","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Luo et al., 2017] Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural network compression. In ICCV, 2017.","title":"Thinet: A filter level pruning method for deep neural network compression","authors":"Jian-Hao Luo, Jianxin Wu, Weiyao Lin","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Rastegari et al., 2016] Mohammad Rastegari, Vicente Or- donez, Joseph Redmon, and Ali Farhadi. Xnor-net: Ima- genet classification using binary convolutional neural net- works. In ECCV, pages 525-542. Springer, 2016.","title":"Xnor-net: Imagenet classification using binary convolutional neural networks","authors":"Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, Ali Farhadi","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sun et al., 2016] Shaoyan Sun, Wengang Zhou, Qi Tian, and Houqiang Li. Scalable object retrieval with compact image representation from generic object regions. ACM TOMM, 12(2):29, 2016.","title":"Scalable object retrieval with compact image representation from generic object regions","authors":"Sun","year":"2016","venue":"ACM TOMM","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wen et al., 2016] Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. In NIPS, pages 2074-2082, 2016.","title":"Learning structured sparsity in deep neural networks","authors":"Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, Hai Li","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wu et al., 2017] Bichen Wu, Alvin Wan, Xiangyu Yue, Pe- ter Jin, Sicheng Zhao, Noah Golmant, Amir Gholamine- jad, Joseph Gonzalez, and Kurt Keutzer. Shift: A zero flop, zero parameter alternative to spatial convolutions. arXiv preprint arXiv:1711.08141, 2017.","title":"Shift: A zero flop, zero parameter alternative to spatial convolutions","authors":"Bichen Wu, Alvin Wan, Xiangyu Yue, Peter Jin, Sicheng Zhao, Noah Golmant, Amir Gholaminejad, Joseph Gonzalez, Kurt Keutzer","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yoon and Hwang, 2017] Jaehong Yoon and Sung Ju Hwang. Combined group and exclusive sparsity for deep neural networks. In ICML, pages 3958-3966, 2017.","title":"Combined group and exclusive sparsity for deep neural networks","authors":"Hwang ; ] Jaehong Yoon, Sung Ju Yoon, Hwang","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yuan and Lin, 2006] Ming Yuan and Yi Lin. Model selec- tion and estimation in regression with grouped variables. Journal of the Royal Statistical Society: Series B (Statisti- cal Methodology), 68(1):49-67, 2006.","title":"Model selection and estimation in regression with grouped variables","authors":"Lin ; Yuan, Ming Yuan, Yi Lin","year":"2006","venue":"Journal of the Royal Statistical Society: Series B (Statistical Methodology","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhou et al., 2014] Wengang Zhou, Ming Yang, Houqiang Li, Xiaoyu Wang, Yuanqing Lin, and Qi Tian. Towards codebook-free: Scalable cascaded hashing for mobile im- age search. IEEE TMM, 16(3):601-611, 2014.","title":"Towards codebook-free: Scalable cascaded hashing for mobile image search","authors":"Wengang Zhou, Ming Yang, Houqiang Li, Xiaoyu Wang, Yuanqing Lin, Qi Tian","year":"2014","venue":"IEEE TMM","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhou et al., 2018] Zhengguang Zhou, Wengang Zhou, and Houqiang Li. Online filter weakening and pruning for ef- ficient convnets. In ICME, 2018.","title":"Online filter weakening and pruning for efficient convnets","authors":"Zhou","year":"2018","venue":"ICME","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhu et al., 2018] Xiaotian Zhu, Wengang Zhou, and Houqiang Li. Adaptive layerwise quantization for deep neural network compression. In ICME, 2018.","title":"Adaptive layerwise quantization for deep neural network compression","authors":"Xiaotian Zhu, Wengang Zhou, Houqiang Li","year":"2018","venue":"ICME","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}