{"type":"ScholarlyArticle","title":"TRP: Trained Rank Pruning for Efficient Deep Neural Networks","authors":[{"name":"Yuhui Xu","slug":"yuhui-xu","given":"Yuhui","family":"Xu","affiliation":"Shanghai Jiao Tong University"},{"name":"Yuxi Li","slug":"yuxi-li","given":"Yuxi","family":"Li","affiliation":"Shanghai Jiao Tong University"},{"name":"Shuai Zhang","slug":"shuai-zhang","given":"Shuai","family":"Zhang","affiliation":"Qualcomm AI Research"},{"name":"Wei Wen","slug":"wei-wen","given":"Wei","family":"Wen","affiliation":"Duke University"},{"name":"Botao Wang","slug":"botao-wang","given":"Botao","family":"Wang","affiliation":"Qualcomm AI Research"},{"name":"Yingyong Qi","slug":"yingyong-qi","given":"Yingyong","family":"Qi","affiliation":"Qualcomm AI Research"},{"name":"Yiran Chen","slug":"yiran-chen","given":"Yiran","family":"Chen","affiliation":"Duke University"},{"name":"Weiyao Lin","slug":"weiyao-lin","given":"Weiyao","family":"Lin","affiliation":"Shanghai Jiao Tong University"},{"name":"Hongkai Xiong","slug":"hongkai-xiong","given":"Hongkai","family":"Xiong","affiliation":"Shanghai Jiao Tong University"}],"abstract":"To enable DNNs on edge devices like mobile phones, low-rank approximation has been widely adopted because of its solid theoretical rationale and efficient implementations. Several previous works attempted to directly approximate a pre-trained model by low-rank decomposition; however, small approximation errors in parameters can ripple over a large prediction loss. As a result, performance usually drops significantly and a sophisticated effort on fine-tuning is required to recover accuracy. Apparently, it is not optimal to separate low-rank approximation from training. Unlike previous works, this paper integrates low rank approximation and regularization into the training process. We propose Trained Rank Pruning (TRP), which alternates between low rank approximation and training. TRP maintains the capacity of the original network while imposing low-rank constraints during training. A nuclear regularization optimized by stochastic sub-gradient descent is utilized to further promote low rank in TRP. The TRP trained network inherently has a low-rank structure, and is approximated with negligible performance loss, thus eliminating the fine-tuning process after low rank decomposition. The proposed method is comprehensively evaluated on CIFAR-10 and ImageNet, outperforming previous compression methods using low rank approximation.","doi":"10.24963/ijcai.2020/136","url":"https://ijcai.org/proceedings/2020/136","pdf":"https://ijcai.org/proceedings/2020/0136.pdf","markdown":"https://ijcai.org/proceedings/2020/136.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/136","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Computer Vision","track":"Main track","number":136,"pages":{"first":977,"last":983},"keywords":["Machine Learning: Deep Learning: Convolutional networks","Computer Vision: Recognition: Detection, Categorization, Indexing, Matching, Retrieval, Semantic Interpretation","Machine Learning: Deep Learning"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24913/file_24913.m3u8","length_sec":299,"provider":"hls"}],"references":[{"position":1,"raw":"[Alvarez and Salzmann, 2017] Jose M Alvarez and Mathieu Salz- mann. Compression-aware training of deep networks. In NIPS, pages 856-867, 2017.","title":"Compression-aware training of deep networks","authors":"Salzmann ; Alvarez, M Jose, Mathieu Alvarez, Salzmann","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Avron et al., 2012] Haim Avron, Satyen Kale, Shiva Prasad Ka- siviswanathan, and Vikas Sindhwani. Efficient and practical stochastic subgradient descent for nuclear norm regularization. In ICML, 2012.","title":"Efficient and practical stochastic subgradient descent for nuclear norm regularization","authors":"Satyen Haim Avron, Shiva Kale, Vikas Prasad Kasiviswanathan, Sindhwani","year":"2012","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2015] Wenlin Chen, James Wilson, Stephen Tyree, Kilian Weinberger, and Yixin Chen. Compressing neural net- works with the hashing trick. In ICML, pages 2285-2294, 2015.","title":"Compressing neural networks with the hashing trick","authors":"Wenlin Chen, James Wilson, Stephen Tyree, Kilian Weinberger, Yixin Chen","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Courbariaux and Bengio, 2016] Matthieu Courbariaux and Yoshua Bengio. Binarynet: Training deep neural networks with weights and activations constrained to +1 or -1. arXiv preprint arXiv:1602.02830, 2016.","title":"Binarynet: Training deep neural networks with weights and activations constrained to +1 or -1","authors":"Matthieu Courbariaux, Yoshua Bengio","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical im- age database. CVPR, pages 248-255, 2009.","title":"Imagenet: A large-scale hierarchical image database","authors":"Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Li Fei-Fei","year":"2009","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Denton et al., 2014] Emily Denton, Wojciech Zaremba, Joan Bruna, Yann Lecun, and Rob Fergus. Exploiting linear structure within convolutional networks for efficient evaluation. In NIPS, pages 1269-1277, 2014.","title":"Exploiting linear structure within convolutional networks for efficient evaluation","authors":"Emily Denton, Wojciech Zaremba, Joan Bruna, Yann Lecun, Rob Fergus","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Guo et al., 2018] Jianbo Guo, Yuxi Li, Weiyao Lin, Yurong Chen, and Jianguo Li. Network decoupling: From regular to depthwise separable convolutions. In BMVC, 2018.","title":"Network decoupling: From regular to depthwise separable convolutions","authors":"Jianbo Guo, Yuxi Li, Weiyao Lin, Yurong Chen, Jianguo Li","year":"2018","venue":"BMVC","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Han et al., 2015a] Song Han, Huizi Mao, and William J. Dally. Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding. CoRR, abs/1510.00149, 2015.","title":"Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding","authors":"Han","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Han et al., 2015b] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In NIPS, pages 1135-1143, 2015.","title":"Learning both weights and connections for efficient neural network","authors":"Han","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[He et al., 2017] Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In ICCV, pages 1389-1397, 2017.","title":"Channel pruning for accelerating very deep neural networks","authors":"Yihui He, Xiangyu Zhang, Jian Sun","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Jaderberg et al., 2014] Max Jaderberg, Andrea Vedaldi, and An- drew Zisserman. Speeding up convolutional neural networks with low rank expansions. arXiv preprint arXiv:1405.3866, 2014.","title":"Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. Speeding up convolutional neural networks with low rank expansions","authors":"Jaderberg","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Krizhevsky et al., 2009] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.","title":"Learning multiple layers of features from tiny images","authors":"Alex Krizhevsky, Geoffrey Hinton","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li et al., 2016] Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient con- vnets. arXiv preprint arXiv:1608.08710, 2016.","title":"Pruning filters for efficient convnets","authors":"Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, Hans Peter Graf","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2017] Hao Li, Soham De, Zheng Xu, Christoph Studer, Hanan Samet, and Tom Goldstein. Training quantized nets: A deeper understanding. In NIPS, pages 5811-5821, 2017.","title":"Training quantized nets: A deeper understanding","authors":"Hao Li, Soham De, Zheng Xu, Christoph Studer, Hanan Samet, Tom Goldstein","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liu et al., 2017] Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through network slimming. In ICCV, pages 2736-2744, 2017.","title":"Learning efficient convolutional networks through network slimming","authors":"Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, Changshui Zhang","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Luo et al., 2017] Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural network compression. ICCV, pages 5058-5066, 2017.","title":"Thinet: A filter level pruning method for deep neural network compression","authors":"Jian-Hao Luo, Jianxin Wu, Weiyao Lin","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Luo et al., 2018] Jian-Hao Luo, Hao Zhang, Hong-Yu Zhou, Chen-Wei Xie, Jianxin Wu, and Weiyao Lin. Thinet: pruning cnn filters for a thinner net. TPAMI, 41(10):2525-2538, 2018.","title":"Thinet: pruning cnn filters for a thinner net","authors":"Jian-Hao Luo, Hao Zhang, Hong-Yu Zhou, Chen-Wei Xie, Jianxin Wu, Weiyao Lin","year":"2018","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Ma et al., 2018] Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient cnn architecture design. arXiv preprint arXiv:1807.11164, 2018.","title":"Shufflenet v2: Practical guidelines for efficient cnn architecture design","authors":"Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, Jian Sun","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Mirsky, 1960] Leon Mirsky. Symmetric gauge functions and uni- tarily invariant norms. The quarterly journal of mathematics, 11(1):50-59, 1960.","title":"Symmetric gauge functions and unitarily invariant norms","authors":"Leon Mirsky","year":"1960","venue":"The quarterly journal of mathematics","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Rastegari et al., 2016] Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classi- fication using binary convolutional neural networks. In ECCV, pages 525-542, 2016.","title":"Xnor-net: Imagenet classification using binary convolutional neural networks","authors":"Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, Ali Farhadi","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sandler et al., 2018] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In CVPR, pages 4510- 4520, 2018.","title":"Mobilenetv2: Inverted residuals and linear bottlenecks","authors":"Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, Liang-Chieh Chen","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Stewart, 1990] Gilbert W Stewart. Matrix perturbation theory. 1990.","title":"Matrix perturbation theory","authors":"Gilbert W Stewart","year":"1990","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Watson, 1992] G Alistair Watson. Characterization of the subdif- ferential of some matrix norms. Linear algebra and its applica- tions, 170:33-45, 1992.","title":"Characterization of the subdifferential of some matrix norms","authors":"Watson Alistair","year":"1992","venue":"Linear algebra and its applications","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wen et al., 2016] Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. In NIPS, pages 2074-2082, 2016.","title":"Learning structured sparsity in deep neural networks","authors":"Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, Hai Li","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wen et al., 2017] Wei Wen, Cong Xu, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Coordinating filters for faster deep neural networks. In ICCV, pages 658-666, 2017.","title":"Coordinating filters for faster deep neural networks","authors":"Wei Wen, Cong Xu, Chunpeng Wu, Yandan Wang, Yiran Chen, Hai Li","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Xu et al., 2018] Yuhui Xu, Yongzhuang Wang, Aojun Zhou, Weiyao Lin, and Hongkai Xiong. Deep neural network compres- sion with single and multiple level quantization. In AAAI, pages 4335-4342, 2018.","title":"Deep neural network compression with single and multiple level quantization","authors":"Yuhui Xu, Yongzhuang Wang, Aojun Zhou, Weiyao Lin, Hongkai Xiong","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Xu et al., 2019] Yuhui Xu, Yuxi Li, Shuai Zhang, Wei Wen, Bo- tao Wang, Yingyong Qi, Yiran Chen, Weiyao Lin, and Hongkai Xiong. Trained rank pruning for efficient deep neural networks. In NIPS EMC2 workshop, 2019.","title":"Trained rank pruning for efficient deep neural networks","authors":"Yuhui Xu, Yuxi Li, Shuai Zhang, Wei Wen, Botao Wang, Yingyong Qi, Yiran Chen, Weiyao Lin, Hongkai Xiong","year":"2019","venue":"NIPS EMC2 workshop","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhang et al., 2016] Xiangyu Zhang, Jianhua Zou, Kaiming He, and Jian Sun. Accelerating very deep convolutional networks for classification and detection. TPAMI, 38(10):1943-1955, 2016.","title":"Accelerating very deep convolutional networks for classification and detection","authors":"Xiangyu Zhang, Jianhua Zou, Kaiming He, Jian Sun","year":"2016","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhou et al., 2017] Aojun Zhou, Anbang Yao, Yiwen Guo, Lin Xu, and Yurong Chen. Incremental network quantization: To- wards lossless cnns with low-precision weights. arXiv preprint arXiv:1702.03044, 2017.","title":"Incremental network quantization: Towards lossless cnns with low-precision weights","authors":"Aojun Zhou, Anbang Yao, Yiwen Guo, Lin Xu, Yurong Chen","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhou et al., 2019] Yuefu Zhou, Ya Zhang, Yanfeng Wang, and Qi Tian. Accelerate cnn via recursive bayesian pruning. In ICCV, pages 3306-3315, 2019.","title":"Accelerate cnn via recursive bayesian pruning","authors":"Yuefu Zhou, Ya Zhang, Yanfeng Wang, Qi Tian","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}