{"type":"ScholarlyArticle","title":"Towards Lossless Head Pruning through Automatic Peer Distillation for Language Models","authors":[{"name":"Bingbing Li","slug":"bingbing-li","given":"Bingbing","family":"Li","affiliation":"University of Connecticut"},{"name":"Zigeng Wang","slug":"zigeng-wang","given":"Zigeng","family":"Wang","affiliation":"University of Connecticut"},{"name":"Shaoyi Huang","slug":"shaoyi-huang","given":"Shaoyi","family":"Huang","affiliation":"University of Connecticut"},{"name":"Mikhail Bragin","slug":"mikhail-bragin","given":"Mikhail","family":"Bragin","affiliation":"University of Connecticut"},{"name":"Ji Li","slug":"ji-li","given":"Ji","family":"Li","affiliation":"Microsoft Corporation"},{"name":"Caiwen Ding","slug":"caiwen-ding","given":"Caiwen","family":"Ding","affiliation":"University of Connecticut"}],"abstract":"Pruning has been extensively studied in Transformer-based language models to improve efficiency. Typically, we zero (prune) unimportant model weights and train a derived compact model to improve final accuracy. For pruned weights, we treat them as useless and discard them. This usually leads to significant model accuracy degradation. In this paper, we focus on attention head pruning as head attention is a key component of the transformer-based language models and provides interpretable knowledge meaning. We reveal the relationship between pruned attention heads and retained heads and provide a solution to recycle the discarded knowledge from the pruned heads, named peer distillation. We also develop an automatic framework to locate the to-be-pruned attention heads in each layer, freeing the time-consuming human labor in tuning hyperparameters.Experimental results on the General Language Understanding Evaluation  (GLUE)  benchmark are provided using BERT model. By recycling discarded knowledge from pruned heads, the proposed method maintains model performance across all nine tasks while reducing heads by over 58% on average and outperforms state-of-the-art techniques (e.g., Random, HISP, L0 Norm, SMP).","doi":"10.24963/ijcai.2023/568","url":"https://ijcai.org/proceedings/2023/568","pdf":"https://ijcai.org/proceedings/2023/0568.pdf","markdown":"https://ijcai.org/proceedings/2023/568.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/568","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Natural Language Processing","track":"Main Track","number":568,"pages":{"first":5113,"last":5121},"keywords":["Natural Language Processing: NLP: Language models"],"videos":[],"references":[{"position":1,"raw":"[An et al., 2020] Bang An, Jie Lyu, Zhenyi Wang, Chunyuan Li, Changwei Hu, Fei Tan, Ruiyi Zhang, Yifan Hu, and Changyou Chen. Repulsive attention: Rethinking multi- head attention as bayesian inference. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 236-255, 2020.","title":"Repulsive attention: Rethinking multihead attention as bayesian inference","authors":"Bang An, Jie Lyu, Zhenyi Wang, Chunyuan Li, Changwei Hu, Fei Tan, Ruiyi Zhang, Yifan Hu, Changyou Chen","year":"2020","venue":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Boyd et al., 2011] Stephen Boyd, Neal Parikh, and Eric Chu. Distributed optimization and statistical learning via the alternating direction method of multipliers. Now Pub- lishers Inc, 2011.","title":"Distributed optimization and statistical learning via the alternating direction method of multipliers","authors":"Stephen Boyd, Neal Parikh, Eric Chu","year":"2011","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Devlin et al., 2018] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint arXiv:1810.04805, 2018.","title":"Bert: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Han et al., 2015] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in Neural Information Processing Systems, 28, 2015.","title":"Learning both weights and connections for efficient neural network","authors":"Han","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735-1780, 1997.","title":"Long short-term memory","authors":"Schmidhuber Hochreiter, Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hu et al., 2018] Minghao Hu, Yuxing Peng, Furu Wei, Zhen Huang, Dongsheng Li, Nan Yang, and Ming Zhou. Attention-guided answer distillation for machine reading comprehension. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2077-2086, 2018.","title":"Attention-guided answer distillation for machine reading comprehension","authors":"Minghao Hu, Yuxing Peng, Furu Wei, Zhen Huang, Dongsheng Li, Nan Yang, Ming Zhou","year":"2018","venue":"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Huang et al., 2022] Shaoyi Huang, Dongkuan Xu, Ian Yen, Yijue Wang, Sung-En Chang, Bingbing Li, Shiyang Chen, Mimi Xie, Sanguthevar Rajasekaran, Hang Liu, et al. Sparse progressive distillation: Resolving overfitting un- der pretrain-and-finetune paradigm. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 190- 200, 2022.","title":"Sparse progressive distillation: Resolving overfitting under pretrain-and-finetune paradigm","authors":"Huang","year":"2022","venue":"Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hubara et al., 2016] Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Bina- rized neural networks. In Advances in neural information processing systems, pages 4107-4115, 2016.","title":"Binarized neural networks","authors":"Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, Yoshua Bengio","year":"2016","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kovaleva et al., 2019] Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. Revealing the dark secrets of bert. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNLP), pages 4365-4374, 2019.","title":"Revealing the dark secrets of bert","authors":"Olga Kovaleva, Alexey Romanov, Anna Rogers, Anna Rumshisky","year":"2019","venue":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lee et al., 2020] Hyun Dong Lee, Seongmin Lee, and U Kang. Auber: Automated bert regularization. arXiv preprint arXiv:2009.14409, 2020.","title":"Auber: Automated bert regularization","authors":"Lee","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2016] Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for ef- ficient convnets. arXiv preprint arXiv:1608.08710, 2016.","title":"Pruning filters for efficient convnets","authors":"Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, Hans Peter Graf","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2020] Bingbing Li, Zhenglun Kong, Tianyun Zhang, Ji Li, Zhengang Li, Hang Liu, and Caiwen Ding. Efficient transformer-based large scale language repre- sentations using hardware-friendly block structured prun- ing. In Proceedings of the 2020 Conference on Empiri- cal Methods in Natural Language Processing: Findings, pages 3187-3199, 2020.","title":"Efficient transformer-based large scale language representations using hardware-friendly block structured pruning","authors":"Bingbing Li, Zhenglun Kong, Tianyun Zhang, Ji Li, Zhengang Li, Hang Liu, Caiwen Ding","year":"2020","venue":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Michel et al., 2019] Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? In Ad- vances in Neural Information Processing Systems, pages 14014-14024, 2019.","title":"Are sixteen heads really better than one? In Advances in Neural Information Processing Systems","authors":"Paul Michel, Omer Levy, Graham Neubig","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Raganato et al., 2020] Alessandro Raganato, Yves Scher- rer, and Jörg Tiedemann. Fixed encoder self-attention pat- terns in transformer-based machine translation. In Pro- ceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 556-568, 2020.","title":"Fixed encoder self-attention patterns in transformer-based machine translation","authors":"Alessandro Raganato, Yves Scherrer, Jörg Tiedemann","year":"2020","venue":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Srinivas et al., 2017] Suraj Srinivas, Akshayvarun Subra- manya, and R Venkatesh Babu. Training sparse neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 138-145, 2017.","title":"Training sparse neural networks","authors":"Suraj Srinivas, Akshayvarun Subramanya, R Venkatesh, Babu","year":"2017","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Voita et al., 2019] Elena Voita, David Talbot, Fedor Moi- seev, Rico Sennrich, and Ivan Titov. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5797-5808, 2019.","title":"Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned","authors":"Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, Ivan Titov","year":"2019","venue":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Wang et al., 2018] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353-355, 2018.","title":"Glue: A multi-task benchmark and analysis platform for natural language understanding","authors":"Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, Samuel Bowman","year":"2018","venue":"Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wang et al., 2020] Hanrui Wang, Zhekai Zhang, and Song Han. Spatten: Efficient sparse attention architecture with cascade token and head pruning. arXiv preprint arXiv:2012.09852, 2020.","title":"Spatten: Efficient sparse attention architecture with cascade token and head pruning","authors":"Wang","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wolf et al., 2019] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's transformers: State- of-the-art natural language processing. arXiv preprint arXiv:1910.03771, 2019.","title":"Huggingface's transformers: Stateof-the-art natural language processing","authors":"Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Xiao et al., 2019] Xia Xiao, Zigeng Wang, and Sanguthevar Rajasekaran. Autoprune: Automatic network pruning by regularizing auxiliary parameters. Advances in neural in- formation processing systems, 32, 2019.","title":"Autoprune: Automatic network pruning by regularizing auxiliary parameters","authors":"Xia Xiao, Zigeng Wang, Sanguthevar Rajasekaran","year":"2019","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Yang et al., 2019] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for lan- guage understanding. In Advances in neural information processing systems, pages 5754-5764, 2019.","title":"Xlnet: Generalized autoregressive pretraining for language understanding","authors":"Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, Quoc V Le","year":"2019","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yin et al., 2019] Penghang Yin, Jiancheng Lyu, Shuai Zhang, Stanley J. Osher, Yingyong Qi, and Jack Xin. Un- derstanding straight-through estimator in training activa- tion quantized neural nets. In International Conference on Learning Representations, 2019.","title":"Understanding straight-through estimator in training activation quantized neural nets","authors":"Penghang Yin, Jiancheng Lyu, Shuai Zhang, Stanley J Osher, Yingyong Qi, Jack Xin","year":"2019","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhang et al., 2021] Zhengyan Zhang, Fanchao Qi, Zhiyuan Liu, Qun Liu, and Maosong Sun. Know what you don't need: Single-shot meta-pruning for attention heads. AI Open, 2:36-42, 2021.","title":"Know what you don't need: Single-shot meta-pruning for attention heads","authors":"Zhang","year":"2021","venue":"AI Open","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhou et al., 2020] Shanglin Zhou, Bingbing Li, Caiwu Ding, Lu Lu, and Caiwen Ding. An efficient deep re- inforcement learning framework for uavs. In 2020 21st International Symposium on Quality Electronic Design (ISQED), pages 323-328. IEEE, 2020.","title":"An efficient deep reinforcement learning framework for uavs","authors":"Zhou","year":"2020","venue":"21st International Symposium on Quality Electronic Design (ISQED)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}