{"type":"ScholarlyArticle","title":"Learning Instance-wise Sparsity for Accelerating Deep Models","authors":[{"name":"Chuanjian Liu","slug":"chuanjian-liu","given":"Chuanjian","family":"Liu","affiliation":"Huawei Noah's Ark Lab"},{"name":"Yunhe Wang","slug":"yunhe-wang","given":"Yunhe","family":"Wang","affiliation":"Huawei Noah's Ark Lab"},{"name":"Kai Han","slug":"kai-han","given":"Kai","family":"Han","affiliation":"Huawei Noah's Ark Lab"},{"name":"Chunjing Xu","slug":"chunjing-xu","given":"Chunjing","family":"Xu","affiliation":"Huawei Noah's Ark Lab"},{"name":"Chang Xu","slug":"chang-xu","given":"Chang","family":"Xu","affiliation":"School of Computer Science , FEIT , University of Sydney , Australia"}],"abstract":"Exploring deep convolutional neural networks of high efficiency and low memory usage is very essential for a wide variety of machine learning tasks. Most of existing approaches used to accelerate deep models by manipulating parameters or filters without data, e.g., pruning and decomposition. In contrast, we study this problem from a different perspective by respecting the difference between data. An instance-wise feature pruning is developed by identifying informative features for different instances. Specifically, by investigating a feature decay regularization, we expect intermediate feature maps of each instance in deep neural networks to be sparse while preserving the overall network performance. During online inference, subtle features of input images extracted by intermediate layers of a well-trained neural network can be eliminated to accelerate the subsequent calculations. We further take coefficient of variation as a measure to select the layers that are appropriate for acceleration. Extensive experiments conducted on benchmark datasets and networks demonstrate the effectiveness of the proposed method.","doi":"10.24963/ijcai.2019/416","url":"https://ijcai.org/proceedings/2019/416","pdf":"https://ijcai.org/proceedings/2019/0416.pdf","markdown":"https://ijcai.org/proceedings/2019/416.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/416","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning A-L","track":"Main track","number":416,"pages":{"first":3001,"last":3007},"keywords":["Computer Vision: Recognition: Detection, Categorization, Indexing, Matching, Retrieval, Semantic Interpretation","Machine Learning: Deep Learning","Machine Learning: Feature Selection ; Learning Sparse Models","Machine Learning: Explainable Machine Learning"],"videos":[],"references":[{"position":1,"raw":"[Almahairi et al., 2016] Amjad Almahairi, Nicolas Ballas, Tim Cooijmans, Yin Zheng, Hugo Larochelle, and Aaron Courville. Dynamic capacity networks. In ICML, 2016.","title":"Dynamic capacity networks","authors":"Amjad Almahairi, Nicolas Ballas, Tim Cooijmans, Yin Zheng, Hugo Larochelle, Aaron Courville","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bolukbasi et al., 2017] Tolga Bolukbasi, Joseph Wang, Ofer Dekel, and Venkatesh Saligrama. Adaptive neural networks for efficient inference. In ICML, 2017.","title":"Adaptive neural networks for efficient inference","authors":"Tolga Bolukbasi, Joseph Wang, Ofer Dekel, Venkatesh Saligrama","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Courbariaux et al., 2015] Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre Binaryconnect David. Training deep neural networks with binary weights during propaga- tions. arXiv preprint arXiv:1511.00363, 2015.","title":"Training deep neural networks with binary weights during propagations","authors":"Matthieu Courbariaux, Yoshua Bengio, Jean-Pierre Binaryconnect David","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li- Jia Li, Kai Li, and Fei-Fei Li. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.","title":"Imagenet: A large-scale hierarchical image database","authors":"Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Fei-Fei Li","year":"2009","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Denton et al., 2014] Emily L Denton, Wojciech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus. Exploiting linear structure within convolutional networks for efficient evaluation. In NIPS, 2014.","title":"Exploiting linear structure within convolutional networks for efficient evaluation","authors":"Wojciech Emily L Denton, Joan Zaremba, Yann Bruna, Rob Lecun, Fergus","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dong et al., 2017] Xuanyi Dong, Junshi Huang, Yi Yang, and Shuicheng Yan. More is less: A more complicated network with less inference complexity. In CVPR, 2017.","title":"More is less: A more complicated network with less inference complexity","authors":"Dong","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Figurnov et al., 2017] Michael Figurnov, Maxwell D. Collins, Yukun Zhu, Li Zhang, Jonathan Huang, Dmitry Vetrov, and Ruslan Salakhutdinov. Spatially adaptive computation time for residual networks. In CVPR, 2017.","title":"Spatially adaptive computation time for residual networks","authors":"Michael Figurnov, Maxwell D Collins, Yukun Zhu, Li Zhang, Jonathan Huang, Dmitry Vetrov, Ruslan Salakhutdinov","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gao et al., 2018] Xitong Gao, Yiren Zhao, Łukasz Dudziak, Robert Mullins, and Cheng-zhong Xu. Dynamic channel pruning: Feature boosting and suppression. arXiv preprint arXiv:1810.05331, 2018.","title":"Dynamic channel pruning: Feature boosting and suppression","authors":"Gao","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Han et al., 2016] Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural net- works with pruning, trained quantization and huffman cod- ing. In ICLR, 2016.","title":"Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding","authors":"Han","year":"2016","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hua et al., 2018] Weizhe Hua, Christopher De Sa, Zhiru Zhang, and G. Edward Suh. Channel gating neural net- works. arXiv preprint arXiv:1805.12549, 2018.","title":"Channel gating neural networks","authors":"Hua","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kim et al., 2016] Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. Accurate image super-resolution using very deep convolutional networks. In CVPR, 2016.","title":"Accurate image super-resolution using very deep convolutional networks","authors":"Jiwon Kim, Jung Kwon Lee, Kyoung Mu, Lee","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton","year":"2012","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Krizhevsky, 2009] Alex Krizhevsky. Learning multiple lay- ers of features from tiny images. Technical report, Cite- seer, 2009.","title":"Learning multiple layers of features from tiny images","authors":"Alex Krizhevsky","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lin et al., 2017] Ji Lin, Yongming Rao, Jiwen Lu, and Jie Zhou. Runtime neural pruning. In NIPS, 2017.","title":"Runtime neural pruning","authors":"Ji Lin, Yongming Rao, Jiwen Lu, Jie Zhou","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liu and Deng, 2018] Lanlan Liu and Jia Deng. Dynamic deep neural networks: Optimizing accuracy-efficiency trade-offs by selective execution. In AAAI, 2018.","title":"Dynamic deep neural networks: Optimizing accuracy-efficiency trade-offs by selective execution","authors":"Lanlan Liu, Jia Deng","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liu et al., 2017] Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through net- work slimming. In ICCV, 2017.","title":"Learning efficient convolutional networks through network slimming","authors":"Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, Changshui Zhang","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Luo et al., 2017] Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural network compression. In ICCV, 2017.","title":"Thinet: A filter level pruning method for deep neural network compression","authors":"Jian-Hao Luo, Jianxin Wu, Weiyao Lin","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[McGill and Perona, 2017] Mason McGill and Pietro Per- ona. Deciding how to decide: Dynamic routing in artificial neural networks. In ICML, 2017.","title":"Deciding how to decide: Dynamic routing in artificial neural networks","authors":"Perona Mcgill, Mason Mcgill, Pietro Perona","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Rastegari et al., 2016] Mohammad Rastegari, Vicente Or- donez, Joseph Redmon, and Ali Farhadi. Xnor-net: Ima- genet classification using binary convolutional neural net- works. In ECCV, 2016.","title":"Xnor-net: Imagenet classification using binary convolutional neural networks","authors":"Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, Ali Farhadi","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Ren et al., 2015] Shaoqing Ren, Kaiming He, Ross Gir- shick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NIPS, 2015.","title":"Faster r-cnn: Towards real-time object detection with region proposal networks","authors":"Ren","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Ren et al., 2018] Mengye Ren, Andrei Pokrovsky, Bin Yang, and Raquel Urtasun. Sbnet: Sparse blocks network for fast inference. In CVPR, 2018.","title":"Sbnet: Sparse blocks network for fast inference","authors":"Ren","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Sandler et al., 2018] Mark Sandler, Andrew Howard, Men- glong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In CVPR, 2018.","title":"Mobilenetv2: Inverted residuals and linear bottlenecks","authors":"Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, Liang-Chieh Chen","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Simonyan and Zisserman, 2015] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Teerapittayanon et al., 2016] Surat Teerapittayanon, Bradley McDanel, and H.T. Kung. Branchynet: Fast inference via early exiting from deep neural networks. In ICPR, 2016.","title":"Branchynet: Fast inference via early exiting from deep neural networks","authors":"Teerapittayanon","year":"2016","venue":"ICPR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Teja Mullapudi et al., 2018] Ravi Teja Mullapudi, William R. Mark, Noam Shazeer, and Kayvon Fata- halian. Hydranets: Specialized dynamic architectures for efficient inference. In CVPR, 2018.","title":"Hydranets: Specialized dynamic architectures for efficient inference","authors":"Ravi Teja Mullapudi, William R Mark, Noam Shazeer, Kayvon Fatahalian","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Vanhoucke et al., 2011] Vincent Vanhoucke, Andrew Se- nior, and Mark Z Mao. Improving the speed of neural networks on cpus. In NIPS, 2011.","title":"Improving the speed of neural networks on cpus","authors":"Vincent Vanhoucke, Andrew Senior, Mark Z Mao","year":"2011","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Veit and Belongie, 2018] Andreas Veit and Serge Belongie. Convolutional networks with adaptive inference graphs. In ECCV, 2018.","title":"Convolutional networks with adaptive inference graphs","authors":"Belongie Veit, Andreas Veit, Serge Belongie","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wang et al., 2016] Yunhe Wang, Chang Xu, Shan You, Dacheng Tao, and Chao Xu. Cnnpack: Packing convolu- tional neural networks in the frequency domain. In NIPS, 2016.","title":"Cnnpack: Packing convolutional neural networks in the frequency domain","authors":"Yunhe Wang, Chang Xu, Shan You, Dacheng Tao, Chao Xu","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wang et al., 2018] Xin Wang, Fisher Yu, Zi-Yi Dou, Trevor Darrell, and Joseph E. Gonzalez. Skipnet: Learning dy- namic routing in convolutional networks. In ECCV, 2018.","title":"Skipnet: Learning dynamic routing in convolutional networks","authors":"Xin Wang, Fisher Yu, Zi-Yi Dou, Trevor Darrell, Joseph E Gonzalez","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Wen et al., 2016] Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. In NIPS, 2016.","title":"Learning structured sparsity in deep neural networks","authors":"Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, Hai Li","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Yang et al., 2011] Yi Yang, Heng Tao Shen, Zhigang Ma, Zi Huang, and Xiaofang Zhou. l2, 1-norm regularized dis- criminative feature selection for unsupervised learning. In IJCAI, 2011.","title":"l2, 1-norm regularized discriminative feature selection for unsupervised learning","authors":"Yi Yang, Heng Tao Shen, Zhigang Ma, Zi Huang, Xiaofang Zhou","year":"2011","venue":"IJCAI","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}