{"type":"ScholarlyArticle","title":"Neural Network Pruning by Cooperative Coevolution","authors":[{"name":"Haopu Shang","slug":"haopu-shang","given":"Haopu","family":"Shang","affiliation":"State Key Laboratory for Novel Software Technology , Nanjing University , Nanjing 210023 , China"},{"name":"Jia-Liang Wu","slug":"jia-liang-wu","given":"Jia-Liang","family":"Wu","affiliation":"State Key Laboratory for Novel Software Technology , Nanjing University , Nanjing 210023 , China"},{"name":"Wenjing Hong","slug":"wenjing-hong","given":"Wenjing","family":"Hong","affiliation":"Department of Computer Science and Engineering Southern University of Science and Technology , Shenzhen 518055 , China"},{"name":"Chao Qian","slug":"chao-qian","given":"Chao","family":"Qian","affiliation":"State Key Laboratory for Novel Software Technology , Nanjing University , Nanjing 210023 , China"}],"abstract":"Neural network pruning is a popular model compression method which can significantly reduce the computing cost with negligible loss of accuracy. Recently, filters are often pruned directly by designing proper criteria or using auxiliary modules to measure their importance, which, however, requires expertise and trial-and-error. Due to the advantage of automation, pruning by evolutionary algorithms (EAs) has attracted much attention, but the performance is limited for deep neural networks as the search space can be quite large. In this paper, we propose a new filter pruning algorithm CCEP by cooperative coevolution, which prunes the filters in each layer by EAs separately. That is, CCEP reduces the pruning space by a divide-and-conquer strategy. The experiments show that CCEP can achieve a competitive performance with the state-of-the-art pruning methods, e.g., prune ResNet56 for 63.42% FLOPs on CIFAR10 with -0.24% accuracy drop, and ResNet50 for 44.56% FLOPs on ImageNet with 0.07% accuracy drop.","doi":"10.24963/ijcai.2022/667","url":"https://ijcai.org/proceedings/2022/667","pdf":"https://ijcai.org/proceedings/2022/0667.pdf","markdown":"https://ijcai.org/proceedings/2022/667.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/667","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Search","track":"Main Track","number":667,"pages":{"first":4814,"last":4820},"keywords":["Search: Evolutionary Computation","Machine Learning: Evolutionary Learning","Search: Heuristic Search","Search: Search and Machine Learning"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987028","length_sec":84,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985171","length_sec":679,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Bäck, 1996] T. Bäck. Evolutionary Algorithms in Theory and Practice: Evolution Strategies, Evolutionary Programming, Ge- netic Algorithms. Oxford University Press, 1996.","title":"Evolutionary Algorithms in Theory and Practice: Evolution Strategies, Evolutionary Programming, Genetic Algorithms","authors":"T Bäck","year":"1996","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Fernandes and Yen, 2021] F. E. Fernandes and G. G. Yen. Pruning deep convolutional neural networks architectures with evolution strategy. Information Sciences, 552:29-47, 2021.","title":"Pruning deep convolutional neural networks architectures with evolution strategy","authors":"F E Fernandes, G G Yen","year":"2021","venue":"Information Sciences","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Han et al., 2016] S. Han, H. Mao, and W. J. Dally. Deep compres- sion: Compressing deep neural network with pruning, trained quantization and huffman coding. In Proceedings of the 4th In- ternational Conference on Learning Representations (ICLR), San Juan, Puerto Rico, 2016.","title":"Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding","authors":"S Han, H Mao, W J Dally","year":"2016","venue":"Proceedings of the 4th International Conference on Learning Representations (ICLR)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[He et al., 2016] K. He, X. Zhang, S. Ren, and J. Sun. Deep resid- ual learning for image recognition. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, Las Vegas, NV, 2016.","title":"Deep residual learning for image recognition","authors":"K He, X Zhang, S Ren, J Sun","year":"2016","venue":"Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[He et al., 2018a] Y. He, G. Kang, X. Dong, Y. Fu, and Y. Yang. Soft filter pruning for accelerating deep convolutional neural net- works. In Proceedings of the 27th International Joint Conference on Artificial Intelligence (IJCAI), pages 2234-2240, Stockholm, Sweden, 2018.","title":"Soft filter pruning for accelerating deep convolutional neural networks","authors":"He","year":"2018","venue":"Proceedings of the 27th International Joint Conference on Artificial Intelligence (IJCAI)","doi":null,"url":null,"internal_url":"/proceedings/2018/309"},{"position":6,"raw":"[He et al., 2018b] Y. He, J. Lin, Z. Liu, H. Wang, L. Li, and S. Han. AMC: Automl for model compression and acceleration on mo- bile devices. In Proceedings of the 15th European Conference on Computer Vision (ECCV), pages 815-832, Munich, Germany, 2018.","title":"AMC: Automl for model compression and acceleration on mobile devices","authors":"He","year":"2018","venue":"Proceedings of the 15th European Conference on Computer Vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[He et al., 2019] Y. He, P. Liu, Z. Wang, Z. Hu, and Y. Yang. Fil- ter pruning via geometric median for deep convolutional neural networks acceleration. In Proceedings of the 2019 IEEE Confer- ence on Computer Vision and Pattern Recognition (CVPR), pages 4340-4349, Long Beach, CA, 2019.","title":"Filter pruning via geometric median for deep convolutional neural networks acceleration","authors":"Y He, P Liu, Z Wang, Z Hu, Y Yang","year":"2019","venue":"Proceedings of the 2019 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[He et al., 2020] Y. He, Y. Ding, P. Liu, L. Zhu, H. Zhang, and Y. Yang. Learning filter pruning criteria for deep convolu- tional neural networks acceleration. In Proceedings of the 2020 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2006-2015, Seattle, WA, 2020.","title":"Learning filter pruning criteria for deep convolutional neural networks acceleration","authors":"Y He, Y Ding, P Liu, L Zhu, H Zhang, Y Yang","year":"2020","venue":"Proceedings of the 2020 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hinton et al., 2015] G. Hinton, O. Vinyals, and J. Dean. Dis- tilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.","title":"Distilling the knowledge in a neural network","authors":"G Hinton, O Vinyals, J Dean","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hong et al., 2019] W. Hong, K. Tang, A. Zhou, H. Ishibuchi, and X. Yao. A scalable indicator-based evolutionary algorithm for large-scale multiobjective optimization. IEEE Transactions on Evolutionary Computation, 23(3):525-537, 2019.","title":"A scalable indicator-based evolutionary algorithm for large-scale multiobjective optimization","authors":"W Hong, K Tang, A Zhou, H Ishibuchi, X Yao","year":"2019","venue":"IEEE Transactions on Evolutionary Computation","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Krizhevsky and Hinton, 2009] A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Techni- cal report, 2009.","title":"Learning multiple layers of features from tiny images","authors":"A Krizhevsky, G Hinton","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2017] H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf. Pruning filters for efficient convnets. In Proceedings of the 5th International Conference on Learning Representations (ICLR), Toulon, France, 2017.","title":"Pruning filters for efficient convnets","authors":"H Li, A Kadav, I Durdanovic, H Samet, H P Graf","year":"2017","venue":"Proceedings of the 5th International Conference on Learning Representations (ICLR)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2018] G. Li, C. Qian, C. Jiang, X. Lu, and K. Tang. Op- timization based layer-wise magnitude-based pruning for DNN compression. In Proceedings of the 27th International Joint Conference on Artificial Intelligence (IJCAI), pages 2383-2389, Stockholm, Sweden, 2018.","title":"Optimization based layer-wise magnitude-based pruning for DNN compression","authors":"G Li, C Qian, C Jiang, X Lu, K Tang","year":"2018","venue":"Proceedings of the 27th International Joint Conference on Artificial Intelligence (IJCAI)","doi":null,"url":null,"internal_url":"/proceedings/2018/330"},{"position":14,"raw":"[Lin et al., 2019] S. Lin, R. Ji, C. Yan, B. Zhang, L. Cao, Q. Ye, F. Huang, and D. S. Doermann. Towards optimal structured CNN pruning via generative adversarial learning. In Proceedings of the 2019 IEEE Conference on Computer Vision and Pattern Recog- nition (CVPR), pages 2790-2799, Long Beach, CA, 2019.","title":"Towards optimal structured CNN pruning via generative adversarial learning","authors":"S Lin, R Ji, C Yan, B Zhang, L Cao, Q Ye, F Huang, D S Doermann","year":"2019","venue":"Proceedings of the 2019 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lin et al., 2020] M. Lin, R. Ji, Y. Wang, Y. Zhang, B. Zhang, Y. Tian, and L. Shao. Hrank: Filter pruning using high-rank feature map. In Proceedings of the 2020 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1526- 1535, Los Alamitos, CA, 2020.","title":"Hrank: Filter pruning using high-rank feature map","authors":"M Lin, R Ji, Y Wang, Y Zhang, B Zhang, Y Tian, L Shao","year":"2020","venue":"Proceedings of the 2020 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liu et al., 2019] Z. Liu, H. Mu, X. Zhang, Z. Guo, X. Yang, K. Cheng, and J. Sun. Metapruning: Meta learning for au- tomatic neural network channel pruning. In Proceedings of the 2019 IEEE International Conference on Computer Vision (ICCV), pages 3295-3304, Seoul, Korea (South), 2019.","title":"Metapruning: Meta learning for automatic neural network channel pruning","authors":"Z Liu, H Mu, X Zhang, Z Guo, X Yang, K Cheng, J Sun","year":"2019","venue":"Proceedings of the 2019 IEEE International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liu et al., 2020] L. Liu, W. Ouyang, X. Wang, P. W. Fieguth, J. Chen, X. Liu, and M. Pietikäinen. Deep learning for generic object detection: A survey. International Journal of Computer Vision, 128(2):261-318, 2020.","title":"Deep learning for generic object detection: A survey","authors":"L Liu, W Ouyang, X Wang, P W Fieguth, J Chen, X Liu, M Pietikäinen","year":"2020","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Luo and Wu, 2020] J. Luo and J. Wu. Autopruner: An end-to-end trainable filter pruning method for efficient deep model inference. Pattern Recognition, 107:107461, 2020.","title":"Autopruner: An end-to-end trainable filter pruning method for efficient deep model inference","authors":"Luo, ] J Wu, J Luo, Wu","year":"2020","venue":"Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Luo et al., 2019] J. Luo, H. Zhang, H. Zhou, C. Xie, J. Wu, and W. Lin. Thinet: Pruning CNN filters for a thinner net. IEEE Transactions on Pattern Analysis and Machine Intelli- gence, 41(10):2525-2538, 2019.","title":"Thinet: Pruning CNN filters for a thinner net","authors":"J Luo, H Zhang, H Zhou, C Xie, J Wu, W Lin","year":"2019","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Ma et al., 2019] X. Ma, X. Li, Q. Zhang, K. Tang, Z. Liang, W. Xie, and Z. Zhu. A survey on cooperative co-evolutionary algorithms. IEEE Transactions on Evolutionary Computation, 23(3):421-441, 2019.","title":"A survey on cooperative co-evolutionary algorithms","authors":"Ma","year":"2019","venue":"IEEE Transactions on Evolutionary Computation","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Potter and Jong, 2000] M. A. Potter and K. A. De Jong. Cooper- ative coevolution: An architecture for evolving coadapted sub- components. Evolutionary Computation, 8(1):1-29, 2000.","title":"Cooperative coevolution: An architecture for evolving coadapted subcomponents","authors":"M A Potter, K A De, Jong","year":"2000","venue":"Evolutionary Computation","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Russakovsky et al., 2015] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei. ImageNet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211-252, 2015.","title":"ImageNet large scale visual recognition challenge","authors":"O Russakovsky, J Deng, H Su, J Krause, S Satheesh, S Ma, Z Huang, A Karpathy, A Khosla, M Bernstein, A C Berg, L Fei-Fei","year":"2015","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Simonyan and Zisserman, 2015] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recog- nition. In Proceedings of the 3rd International Conference on Learning Representations (ICLR), San Diego, CA, 2015.","title":"Very deep convolutional networks for large-scale image recognition","authors":"K Simonyan, A Zisserman","year":"2015","venue":"Proceedings of the 3rd International Conference on Learning Representations (ICLR)","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Su et al., 2021] X. Su, S. You, F. Wang, C. Qian, C. Zhang, and C. Xu. Bcnet: Searching for network width with bilaterally cou- pled network. In Proceedings of the 2021 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2175- 2184, Virtual Event, 2021.","title":"Bcnet: Searching for network width with bilaterally coupled network","authors":"Su","year":"2021","venue":"Proceedings of the 2021 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Tang et al., 2021] Y. Tang, Y. Wang, Y. Xu, Y. Deng, C. Xu, D. Tao, and C. Xu. Manifold regularized dynamic network prun- ing. In Proceedings of the 2021 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5018-5028, Vir- tual Event, 2021.","title":"Manifold regularized dynamic network pruning","authors":"Y Tang, Y Wang, Y Xu, Y Deng, C Xu, D Tao, C Xu","year":"2021","venue":"Proceedings of the 2021 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wang et al., 2020] Y. Wang, X. Zhang, L. Xie, J. Zhou, H. Su, B. Zhang, and X. Hu. Pruning from scratch. In Proceedings of the 34th AAAI Conference on Artificial Intelligence (AAAI), pages 12273-12280, New York, NY, 2020.","title":"Pruning from scratch","authors":"Y Wang, X Zhang, L Xie, J Zhou, H Su, B Zhang, X Hu","year":"2020","venue":"Proceedings of the 34th AAAI Conference on Artificial Intelligence (AAAI)","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wang et al., 2021] H. Wang, C. Qin, Y. Zhang, and Y. Fu. Neural pruning via growing regularization. In Proceedings of the 9th International Conference on Learning Representations (ICLR), Virtual Event, 2021.","title":"Neural pruning via growing regularization","authors":"H Wang, C Qin, Y Zhang, Y Fu","year":"2021","venue":"Proceedings of the 9th International Conference on Learning Representations (ICLR), Virtual Event","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Yao, 1999] X. Yao. Evolving artificial neural networks. Proceed- ings of the IEEE, 87(9):1423-1447, 1999.","title":"Evolving artificial neural networks","authors":"X Yao","year":"1999","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhou et al., 2021] Y. Zhou, G. G. Yen, and Z. Yi. Evolutionary shallowing deep neural networks at block levels. IEEE Transac- tions on Neural Networks and Learning Systems, in press, 2021.","title":"Evolutionary shallowing deep neural networks at block levels","authors":"Y Zhou, G G Yen, Z Yi","year":"2021","venue":"IEEE Transactions on Neural Networks and Learning Systems","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}