{"type":"ScholarlyArticle","title":"Beyond Network Pruning: a Joint Search-and-Training Approach","authors":[{"name":"Xiaotong Lu","slug":"xiaotong-lu","given":"Xiaotong","family":"Lu","affiliation":"Xidian University"},{"name":"Han Huang","slug":"han-huang","given":"Han","family":"Huang","affiliation":"Xidian University"},{"name":"Weisheng Dong","slug":"weisheng-dong","given":"Weisheng","family":"Dong","affiliation":"Xidian University"},{"name":"Xin Li","slug":"xin-li","given":"Xin","family":"Li","affiliation":"West Virginia University"},{"name":"Guangming Shi","slug":"guangming-shi","given":"Guangming","family":"Shi","affiliation":"Xidian University"}],"abstract":"Network pruning has been proposed as a remedy for alleviating the over-parameterization problem of deep neural networks. However, its value has been recently challenged especially from the perspective of neural architecture search (NAS). We challenge the conventional wisdom of pruning-after-training by proposing a joint search-and-training approach that directly learns a compact network from the scratch. By treating pruning as a search strategy, we present two new insights in this paper: 1) it is possible to expand the search space of networking pruning by associating each filter with a learnable weight; 2) joint search-and-training can be conducted iteratively to maximize the learning efficiency. More specifically, we propose a coarse-to-fine tuning strategy to iteratively sample and update compact sub-network to approximate the target network. The weights associated with network filters will be accordingly updated by joint search-and-training to reflect learned knowledge in NAS space. Moreover, we introduce strategies of random perturbation (inspired by Monte Carlo) and flexible thresholding (inspired by Reinforcement Learning) to adjust the weight and size of each layer. Extensive experiments on ResNet and VGGNet demonstrate the superior performance of our proposed method on popular datasets including CIFAR10, CIFAR100 and ImageNet.","doi":"10.24963/ijcai.2020/358","url":"https://ijcai.org/proceedings/2020/358","pdf":"https://ijcai.org/proceedings/2020/0358.pdf","markdown":"https://ijcai.org/proceedings/2020/358.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/358","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":358,"pages":{"first":2583,"last":2590},"keywords":["Machine Learning: Deep Learning: Convolutional networks","Computer Vision: Recognition: Detection, Categorization, Indexing, Matching, Retrieval, Semantic Interpretation","Machine Learning: Deep Learning"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25073/file_25073.m3u8","length_sec":300,"provider":"hls"}],"references":[{"position":1,"raw":"[Alvarez and Salzmann, 2016] Jose M Alvarez and Mathieu Salzmann. Learning the number of neurons in deep net- works. In Advances in Neural Information Processing Sys- tems, pages 2270-2278, 2016.","title":"Learning the number of neurons in deep networks","authors":"Salzmann ; Alvarez, M Jose, Mathieu Alvarez, Salzmann","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Ashok et al., 2017] Anubhav Ashok, Nicholas Rhinehart, Fares Beainy, and Kris M Kitani. N2n learning: Network to network compression via policy gradient reinforcement learning. arXiv preprint arXiv:1709.06030, 2017.","title":"N2n learning: Network to network compression via policy gradient reinforcement learning","authors":"Anubhav Ashok, Nicholas Rhinehart, Fares Beainy, Kris M Kitani","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cai et al., 2018] Han Cai, Ligeng Zhu, and Song Han. Prox- ylessnas: Direct neural architecture search on target task and hardware. arXiv preprint arXiv:1812.00332, 2018.","title":"Proxylessnas: Direct neural architecture search on target task and hardware","authors":"Han Cai, Ligeng Zhu, Song Han","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dai et al., 2019] Xiaoliang Dai, Peizhao Zhang, Bichen Wu, Hongxu Yin, Fei Sun, Yanghan Wang, Marat Dukhan, Yunqing Hu, Yiming Wu, Yangqing Jia, et al. Chamnet: Towards efficient network design through platform-aware model adaptation. In Proceedings of the IEEE Confer- ence on Computer Vision and Pattern Recognition, pages 11398-11407, 2019.","title":"Chamnet: Towards efficient network design through platform-aware model adaptation","authors":"Xiaoliang Dai, Peizhao Zhang, Bichen Wu, Hongxu Yin, Fei Sun, Yanghan Wang, Marat Dukhan, Yunqing Hu, Yiming Wu, Yangqing Jia","year":"2019","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dettmers and Zettlemoyer, 2019] Tim Dettmers and Luke Zettlemoyer. Sparse networks from scratch: Faster training without losing performance. arXiv preprint arXiv:1907.04840, 2019.","title":"Sparse networks from scratch: Faster training without losing performance","authors":"Zettlemoyer Dettmers, Tim Dettmers, Luke Zettlemoyer","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dong and Yang, 2019a] Xuanyi Dong and Yi Yang. Net- work pruning via transformable architecture search. In Ad- vances in Neural Information Processing Systems, pages 759-770, 2019.","title":"Network pruning via transformable architecture search","authors":"Dong, Yang ;, Xuanyi Dong, Yi Yang","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dong and Yang, 2019b] Xuanyi Dong and Yi Yang. One- shot neural architecture search via self-evaluated template network. In Proceedings of the IEEE International Con- ference on Computer Vision, pages 3681-3690, 2019.","title":"Oneshot neural architecture search via self-evaluated template network","authors":"Dong, Yang ;, Xuanyi Dong, Yi Yang","year":"2019","venue":"Proceedings of the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dong and Yang, 2019c] Xuanyi Dong and Yi Yang. Search- ing for a robust neural architecture in four gpu hours. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1761-1770, 2019.","title":"Searching for a robust neural architecture in four gpu hours","authors":"Dong, Yang ;, Xuanyi Dong, Yi Yang","year":"2019","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Frankle and Carbin, 2018] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv:1803.03635, 2018.","title":"The lottery ticket hypothesis: Finding sparse, trainable neural networks","authors":"Carbin Frankle, Jonathan Frankle, Michael Carbin","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gu et al., 2018] Bin Gu, Miao Xin, Zhouyuan Huo, and Heng Huang. Asynchronous doubly stochastic sparse ker- nel learning, 2018.","title":"Asynchronous doubly stochastic sparse kernel learning","authors":"Bin Gu, Miao Xin, Zhouyuan Huo, Heng Huang","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[He et al., 2018a] Yang He, Guoliang Kang, Xuanyi Dong, Yanwei Fu, and Yi Yang. Soft filter pruning for accelerat- ing deep convolutional neural networks. In IJCAI Interna- tional Joint Conference on Artificial Intelligence, 2018.","title":"Soft filter pruning for accelerating deep convolutional neural networks","authors":"He","year":"2018","venue":"IJCAI International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2018/309"},{"position":12,"raw":"[He et al., 2018b] Yihui He, Ji Lin, Zhijian Liu, Hanrui Wang, Li-Jia Li, and Song Han. Amc: Automl for model compression and acceleration on mobile devices. In Pro- ceedings of the European Conference on Computer Vision (ECCV), pages 784-800, 2018.","title":"Amc: Automl for model compression and acceleration on mobile devices","authors":"He","year":"2018","venue":"Proceedings of the European Conference on Computer Vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[He et al., 2019] Yang He, Ping Liu, Ziwei Wang, Zhilan Hu, and Yi Yang. Filter pruning via geometric median for deep convolutional neural networks acceleration. In Proceed- ings of the IEEE Conference on Computer Vision and Pat- tern Recognition, pages 4340-4349, 2019.","title":"Filter pruning via geometric median for deep convolutional neural networks acceleration","authors":"Yang He, Ping Liu, Ziwei Wang, Zhilan Hu, Yi Yang","year":"2019","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hu et al., 2018] Yiming Hu, Siyang Sun, Jianquan Li, Xin- gang Wang, and Qingyi Gu. A novel channel prun- ing method for deep neural network compression. arXiv preprint arXiv:1805.11394, 2018.","title":"A novel channel pruning method for deep neural network compression","authors":"Yiming Hu, Siyang Sun, Jianquan Li, Xingang Wang, Qingyi Gu","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Huang and Wang, 2018] Zehao Huang and Naiyan Wang. Data-driven sparse structure selection for deep neural net- works. In Proceedings of the European Conference on Computer Vision (ECCV), pages 304-320, 2018.","title":"Data-driven sparse structure selection for deep neural networks","authors":"Wang ; ] Zehao Huang, Naiyan Huang, Wang","year":"2018","venue":"Proceedings of the European Conference on Computer Vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kim et al., 2019] Hyeji Kim, Muhammad Umar Karim Khan, and Chong-Min Kyung. Efficient neural network compression. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 12569- 12577, 2019.","title":"Efficient neural network compression","authors":"Kim","year":"2019","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Li et al., 2016] Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for ef- ficient convnets. arXiv preprint arXiv:1608.08710, 2016.","title":"Pruning filters for efficient convnets","authors":"Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, Hans Peter Graf","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lin et al., 2019] Shaohui Lin, Rongrong Ji, Yuchao Li, Cheng Deng, and Xuelong Li. Toward compact convnets via structure-sparsity regularized filter pruning. IEEE transactions on neural networks and learning systems, 2019.","title":"Toward compact convnets via structure-sparsity regularized filter pruning","authors":"Lin","year":"2019","venue":"IEEE transactions on neural networks and learning systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Liu et al., 2018a] Hanxiao Liu, Karen Simonyan, and Yim- ing Yang. Darts: Differentiable architecture search. arXiv preprint arXiv:1806.09055, 2018.","title":"Darts: Differentiable architecture search","authors":"Liu","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Liu et al., 2018b] Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. arXiv preprint arXiv:1810.05270, 2018.","title":"Rethinking the value of network pruning","authors":"Liu","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Liu et al., 2019] Zechun Liu, Haoyuan Mu, Xiangyu Zhang, Zichao Guo, Xin Yang, Kwang-Ting Cheng, and Jian Sun. Metapruning: Meta learning for automatic neural network channel pruning. In Proceedings of the IEEE International Conference on Computer Vision, pages 3296-3305, 2019.","title":"Metapruning: Meta learning for automatic neural network channel pruning","authors":"Liu","year":"2019","venue":"Proceedings of the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Pham et al., 2018] Hieu Pham, Melody Y Guan, Barret Zoph, Quoc V Le, and Jeff Dean. Efficient neural ar- chitecture search via parameter sharing. arXiv preprint arXiv:1802.03268, 2018.","title":"Efficient neural architecture search via parameter sharing","authors":"Hieu Pham, Melody Y Guan, Barret Zoph, Jeff Quoc V Le, Dean","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Real et al., 2019] Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le. Regularized evolution for image classifier architecture search. In Proceedings of the aaai conference on artificial intelligence, volume 33, pages 4780-4789, 2019.","title":"Regularized evolution for image classifier architecture search","authors":"Esteban Real, Alok Aggarwal, Yanping Huang, Quoc V Le","year":"2019","venue":"Proceedings of the aaai conference on artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Ye et al., 2018] Jianbo Ye, Xin Lu, Zhe Lin, and James Z Wang. Rethinking the smaller-norm-less-informative as- sumption in channel pruning of convolution layers. arXiv preprint arXiv:1802.00124, 2018.","title":"Rethinking the smaller-norm-less-informative assumption in channel pruning of convolution layers","authors":"Jianbo Ye, Xin Lu, Zhe Lin, James Z Wang","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[You et al., 2019] Haoran You, Chaojian Li, Pengfei Xu, Yonggan Fu, Yue Wang, Xiaohan Chen, Yingyan Lin, Zhangyang Wang, and Richard G Baraniuk. Drawing early-bird tickets: Towards more efficient training of deep networks. arXiv preprint arXiv:1909.11957, 2019.","title":"Drawing early-bird tickets: Towards more efficient training of deep networks","authors":"Haoran You, Chaojian Li, Pengfei Xu, Yonggan Fu, Yue Wang, Xiaohan Chen, Yingyan Lin, Zhangyang Wang, Richard G Baraniuk","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhou et al., 2016] Hao Zhou, Jose M Alvarez, and Fatih Porikli. Less is more: Towards compact cnns. In European Conference on Computer Vision, pages 662-677. Springer, 2016.","title":"Less is more: Towards compact cnns","authors":"Hao Zhou, Jose M Alvarez, Fatih Porikli","year":"2016","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zoph et al., 2018] Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architec- tures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 8697-8710, 2018.","title":"Learning transferable architectures for scalable image recognition","authors":"Barret Zoph, Vijay Vasudevan, Jonathon Shlens, Quoc V Le","year":"2018","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}