{"type":"ScholarlyArticle","title":"Pruning-as-Search: Efficient Neural Architecture Search via Channel Pruning and Structural Reparameterization","authors":[{"name":"Yanyu Li","slug":"yanyu-li","given":"Yanyu","family":"Li","affiliation":"Northeastern University"},{"name":"Pu Zhao","slug":"pu-zhao","given":"Pu","family":"Zhao","affiliation":"Northeastern University"},{"name":"Geng Yuan","slug":"geng-yuan","given":"Geng","family":"Yuan","affiliation":"Northeastern University"},{"name":"Xue Lin","slug":"xue-lin","given":"Xue","family":"Lin","affiliation":"Northeastern University"},{"name":"Yanzhi Wang","slug":"yanzhi-wang","given":"Yanzhi","family":"Wang","affiliation":"Northeastern University"},{"name":"Xin Chen","slug":"xin-chen","given":"Xin","family":"Chen","affiliation":"Intel Corp."}],"abstract":"Neural architecture search (NAS) and network pruning are widely studied efficient AI techniques, but not yet perfect.\r\nNAS performs exhaustive candidate architecture search, incurring tremendous search cost.\r\nThough (structured) pruning can simply shrink model dimension, it remains unclear how to decide the per-layer sparsity automatically and optimally.\r\nIn this work, we revisit the problem of layer-width optimization and propose Pruning-as-Search (PaS), an end-to-end channel pruning method to  search out desired sub-network automatically and efficiently.\r\nSpecifically, we add a depth-wise binary convolution to learn pruning policies directly through gradient descent.\r\nBy combining the structural reparameterization and PaS, we successfully searched out a new family of VGG-like and lightweight networks, which enable the flexibility of arbitrary width with respect to each layer instead of each stage.\r\nExperimental results show that our proposed architecture outperforms prior arts by around 1.0% top-1 accuracy under similar inference speed on ImageNet-1000 classification task.\r\nFurthermore, we demonstrate the effectiveness of our width search on complex tasks including instance segmentation and image translation.\r\nCode and models are released.","doi":"10.24963/ijcai.2022/449","url":"https://ijcai.org/proceedings/2022/449","pdf":"https://ijcai.org/proceedings/2022/0449.pdf","markdown":"https://ijcai.org/proceedings/2022/449.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/449","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Machine Learning","track":"Main Track","number":449,"pages":{"first":3236,"last":3242},"keywords":["Machine Learning: Convolutional Networks","Machine Learning: Automated Machine Learning"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986641","length_sec":87,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984784","length_sec":352,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Bengio et al., 2013] Yoshua Bengio, Nicholas Léonard, and Aaron C. Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. CoRR, abs/1308.3432, 2013.","title":"Estimating or propagating gradients through stochastic neurons for conditional computation","authors":"Yoshua Bengio, Nicholas Léonard, Aaron C Courville","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bolya et al., 2019] Daniel Bolya, Chong Zhou, et al. Yolact: Real-time instance segmentation. In ICCV, 2019.","title":"Yolact: Real-time instance segmentation","authors":"Daniel Bolya, Chong Zhou","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Ding et al., 2021] Xiaohan Ding, Xiangyu Zhang, et al. Repvgg: Making vgg-style convnets great again. In CVPR, pages 13733-13742, 2021.","title":"Repvgg: Making vgg-style convnets great again","authors":"Xiaohan Ding, Xiangyu Zhang","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Guan et al., 2022] Yushuo Guan, Ning Liu, et al. Dais: Au- tomatic channel pruning via differentiable annealing indi- cator search. IEEE Transactions on Neural Networks and Learning Systems, 2022.","title":"Dais: Automatic channel pruning via differentiable annealing indicator search","authors":"Guan","year":"2022","venue":"IEEE Transactions on Neural Networks and Learning Systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Guo et al., 2020] Shaopeng Guo, Yujie Wang, et al. Dmcp: Differentiable markov channel pruning for neural networks. In CVPR, 2020.","title":"Dmcp: Differentiable markov channel pruning for neural networks","authors":"Guo","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Guo et al., 2021] Yi Guo, Huan Yuan, et al. Gdp: Stabi- lized neural network pruning via gates with differentiable polarization. In ICCV, pages 5239-5250, October 2021.","title":"Gdp: Stabilized neural network pruning via gates with differentiable polarization","authors":"Yi Guo, Huan Yuan","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Han et al., 2015] Song Han, Jeff Pool, et al. Learning both weights and connections for efficient neural network. In NeurIPS, 2015.","title":"Learning both weights and connections for efficient neural network","authors":"Han","year":"2015","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Han et al., 2016] Song Han, Huizi Mao, et al. Deep com- pression: Compressing deep neural networks with pruning, trained quantization and huffman coding. ICLR, 2016.","title":"Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding","authors":"Han","year":"2016","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, et al. Deep residual learning for image recognition. In CVPR, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[He et al., 2017] Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In ICCV, 2017.","title":"Channel pruning for accelerating very deep neural networks","authors":"Yihui He, Xiangyu Zhang, Jian Sun","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[He et al., 2018] Yihui He, Ji Lin, et al. Amc: Automl for model compression and acceleration on mobile devices. In ECCV, 2018.","title":"Amc: Automl for model compression and acceleration on mobile devices","authors":"Yihui He, Ji Lin","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hu et al., 2016] Hengyuan Hu, Rui Peng, et al. Network trimming: A data-driven neuron pruning approach towards efficient deep architectures. arXiv preprint, 2016.","title":"Network trimming: A data-driven neuron pruning approach towards efficient deep architectures","authors":"Hengyuan Hu, Rui Peng","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hu et al., 2018] Jie Hu, Li Shen, and Gang Sun. Squeeze- and-excitation networks. In CVPR, 2018.","title":"Squeezeand-excitation networks","authors":"Jie Hu, Li Shen, Gang Sun","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Jin et al., 2021] Qing Jin, Jian Ren, et al. Teachers do more than teach: Compressing image-to-image models. In CVPR, pages 13600-13611, 2021.","title":"Teachers do more than teach: Compressing image-to-image models","authors":"Qing Jin, Jian Ren","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2019] Tuanhui Li, Baoyuan Wu, et al. Compressing convolutional neural networks via factorized convolutional filters. In CVPR, pages 3977-3986, 2019.","title":"Compressing convolutional neural networks via factorized convolutional filters","authors":"Li","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Li et al., 2020] Muyang Li, Ji Lin, et al. Gan compression: Efficient architectures for interactive conditional gans. In CVPR, pages 5284-5294, 2020.","title":"Gan compression: Efficient architectures for interactive conditional gans","authors":"Li","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liu et al., 2017] Zhuang Liu, Jianguo Li, et al. Learning efficient convolutional networks through network slimming. In ICCV, 2017.","title":"Learning efficient convolutional networks through network slimming","authors":"Zhuang Liu, Jianguo Li","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Liu et al., 2018] Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint arXiv:1806.09055, 2018.","title":"Darts: Differentiable architecture search","authors":"Liu","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Liu et al., 2019] Zechun Liu, Haoyuan Mu, et al. Metaprun- ing: Meta learning for automatic neural network channel pruning. In ICCV, 2019.","title":"Metapruning: Meta learning for automatic neural network channel pruning","authors":"Liu","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Luo et al., 2017] Jian-Hao Luo, Jianxin Wu, et al. Thinet: A filter level pruning method for deep neural network com- pression. In ICCV, 2017.","title":"Thinet: A filter level pruning method for deep neural network compression","authors":"Jian-Hao Luo, Jianxin Wu","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Marnerides et al., 2018] Demetris Marnerides, Thomas Bashford-Rogers, et al. Expandnet: A deep convolutional neural network for high dynamic range expansion from low dynamic range content. In Computer Graphics Forum, volume 37, pages 37-49. Wiley Online Library, 2018.","title":"Expandnet: A deep convolutional neural network for high dynamic range expansion from low dynamic range content","authors":"Demetris Marnerides, Thomas Bashford-Rogers","year":"2018","venue":"Computer Graphics Forum","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Ramachandran et al., 2018] Prajit Ramachandran, Barret Zoph, and Quoc Le. Searching for activation functions. In ICLR, 2018.","title":"Searching for activation functions","authors":"Ramachandran","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Sandler et al., 2018] Mark Sandler, Andrew Howard, et al. Mobilenetv2: Inverted residuals and linear bottlenecks. In CVPR, 2018.","title":"Mobilenetv2: Inverted residuals and linear bottlenecks","authors":"Mark Sandler, Andrew Howard","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Su et al., 2021] Xiu Su, Shan You, et al. Bcnet: Searching for network width with bilaterally coupled network. CoRR, abs/2105.10533, 2021.","title":"Bcnet: Searching for network width with bilaterally coupled network","authors":"Xiu Su, Shan You","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Tan and Le, 2019] Mingxing Tan and Quoc Le. Efficient- net: Rethinking model scaling for convolutional neural networks. In ICML, pages 6105-6114. PMLR, 2019.","title":"Efficientnet: Rethinking model scaling for convolutional neural networks","authors":"Le ; Tan, Mingxing Tan, Quoc Le","year":"2019","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Tan et al., 2019] Mingxing Tan, Bo Chen, et al. Mnasnet: Platform-aware neural architecture search for mobile. In CVPR, 2019.","title":"Mnasnet: Platform-aware neural architecture search for mobile","authors":"Mingxing Tan, Bo Chen","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wu et al., 2019] Bichen Wu, Xiaoliang Dai, et al. Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search. In CVPR, 2019.","title":"Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search","authors":"Wu","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhang et al., 2018a] Tianyun Zhang, Shaokai Ye, et al. Structadmm: A systematic, high-efficiency framework of structured weight pruning for dnns. arXiv preprint arXiv:1807.11091, 2018.","title":"Structadmm: A systematic, high-efficiency framework of structured weight pruning for dnns","authors":"Zhang","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhang et al., 2018b] Tianyun Zhang, Shaokai Ye, et al. Sys- tematic weight pruning of dnns using alternating direction method of multipliers. arXiv preprint arXiv:1802.05747, 2018.","title":"Systematic weight pruning of dnns using alternating direction method of multipliers","authors":"Zhang","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhong et al., 2018] Zhao Zhong, Junjie Yan, et al. Practi- cal block-wise neural network architecture generation. In CVPR, 2018.","title":"Practical block-wise neural network architecture generation","authors":"Zhao Zhong, Junjie Yan","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhu et al., 2017] Jun-Yan Zhu, Taesung Park, et al. Unpaired image-to-image translation using cycle-consistent adversar- ial networks. In ICCV, pages 2223-2232, 2017.","title":"Unpaired image-to-image translation using cycle-consistent adversarial networks","authors":"Zhu","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zoph and Le, 2017] Barret Zoph and Quoc V. Le. Neural architecture search with reinforcement learning. In ICLR, 2017.","title":"Neural architecture search with reinforcement learning","authors":"Le ; Zoph, Barret Zoph, Quoc V Le","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}