{"type":"ScholarlyArticle","title":"Parameter-Efficient Sparsity for Large Language Models Fine-Tuning","authors":[{"name":"Yuchao Li","slug":"yuchao-li","given":"Yuchao","family":"Li","affiliation":"Alibaba Group"},{"name":"Fuli Luo","slug":"fuli-luo","given":"Fuli","family":"Luo","affiliation":"Alibaba Group"},{"name":"Chuanqi Tan","slug":"chuanqi-tan","given":"Chuanqi","family":"Tan","affiliation":"Alibaba Group"},{"name":"Mengdi Wang","slug":"mengdi-wang","given":"Mengdi","family":"Wang","affiliation":"Alibaba Group"},{"name":"Songfang Huang","slug":"songfang-huang","given":"Songfang","family":"Huang","affiliation":"Alibaba Group"},{"name":"Shen Li","slug":"shen-li","given":"Shen","family":"Li","affiliation":"Alibaba Group"},{"name":"Junjie Bai","slug":"junjie-bai","given":"Junjie","family":"Bai","affiliation":"Alibaba Group"}],"abstract":"With the dramatically increased number of parameters in language models, sparsity methods have received ever-increasing research focus to compress and accelerate the models. While most research focuses on how to accurately retain appropriate weights while maintaining the performance of the compressed model, there are challenges in the computational overhead and memory footprint of sparse training when compressing large-scale language models. To address this problem, we propose a Parameter-efficient Sparse Training (PST) method to reduce the number of trainable parameters during sparse-aware training in downstream tasks. Specifically, we first combine the data-free and data-driven criteria to efficiently and accurately measure the importance of weights. Then we investigate the intrinsic redundancy of data-driven weight importance and derive two obvious characteristics i.e. low-rankness and structuredness. Based on that, two groups of small matrices are introduced to compute the data-driven importance of weights, instead of using the original large importance score matrix, which therefore makes the sparse training resource-efficient and parameter-efficient. Experiments with diverse networks (i.e. BERT, RoBERTa and GPT-2) on dozens of datasets demonstrate PST performs on par or better than previous sparsity methods, despite only training a small number of parameters. For instance, compared with previous sparsity methods, our PST only requires 1.5% trainable parameters to achieve comparable performance on BERT.","doi":"10.24963/ijcai.2022/586","url":"https://ijcai.org/proceedings/2022/586","pdf":"https://ijcai.org/proceedings/2022/0586.pdf","markdown":"https://ijcai.org/proceedings/2022/586.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/586","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Natural Language Processing","track":"Main Track","number":586,"pages":{"first":4223,"last":4229},"keywords":["Machine Learning: Learning Sparse Models","Natural Language Processing: Language Models"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986993","length_sec":74,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985136","length_sec":635,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Devlin et al., 2019] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understand- ing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, 2019.","title":"Bert: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2019","venue":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Gordon et al., 2020] Mitchell A Gordon, Kevin Duh, and Nicholas Andrews. Compressing bert: Studying the ef- fects of weight pruning on transfer learning. Association for Computational Linguistics, page 143, 2020.","title":"Compressing bert: Studying the effects of weight pruning on transfer learning","authors":"Kevin Mitchell A Gordon, Nicholas Duh, Andrews","year":"2020","venue":"Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Guo et al., 2020] Demi Guo, Alexander M Rush, and Yoon Kim. Parameter-efficient transfer learning with diff prun- ing. arXiv preprint arXiv:2012.07463, 2020.","title":"Parameter-efficient transfer learning with diff pruning","authors":"Demi Guo, Alexander M Rush, Yoon Kim","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Han et al., 2015] Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural net- works with pruning, trained quantization and huffman cod- ing. arXiv preprint arXiv:1510.00149, 2015.","title":"Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding","authors":"Han","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[He et al., 2021] Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. Towards a unified view of parameter-efficient transfer learning. arXiv preprint arXiv:2110.04366, 2021.","title":"Towards a unified view of parameter-efficient transfer learning","authors":"He","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Houlsby et al., 2019] Neil Houlsby, Andrei Giurgiu, Stanis- law Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In Inter- national Conference on Machine Learning, pages 2790- 2799. PMLR, 2019.","title":"Parameter-efficient transfer learning for nlp","authors":"Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, Sylvain Gelly","year":"2019","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hu et al., 2021] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021.","title":"Lora: Low-rank adaptation of large language models","authors":"J Edward, Yelong Hu, Phillip Shen, Zeyuan Wallis, Yuanzhi Allen-Zhu, Shean Li, Lu Wang, Weizhu Wang, Chen","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Lester et al., 2021] Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691, 2021.","title":"The power of scale for parameter-efficient prompt tuning","authors":"Brian Lester, Rami Al-Rfou, Noah Constant","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Li et al., 2020] Zhuohan Li, Eric Wallace, Sheng Shen, Kevin Lin, Kurt Keutzer, Dan Klein, and Joey Gonzalez. Train big, then compress: Rethinking model size for ef- ficient training and inference of transformers. In ICML, pages 5958-5968, 2020.","title":"Train big, then compress: Rethinking model size for efficient training and inference of transformers","authors":"Li","year":"2020","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li et al., 2021] Yuchao Li, Shaohui Lin, Jianzhuang Liu, Qixiang Ye, Mengdi Wang, Fei Chao, Fan Yang, Jincheng Ma, Qi Tian, and Rongrong Ji. Towards compact cnns via collaborative compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6438-6447, 2021.","title":"Towards compact cnns via collaborative compression","authors":"Li","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Liu et al., 2019] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019.","title":"Roberta: A robustly optimized bert pretraining approach","authors":"Liu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[McCarley et al., 2019] JS McCarley, Rishav Chakravarti, and Avirup Sil. Structured pruning of a bert-based ques- tion answering model. arXiv preprint arXiv:1910.06360, 2019.","title":"Structured pruning of a bert-based ques-tion answering model","authors":"Mccarley","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Molchanov et al., 2019] Pavlo Molchanov, Arun Mallya, Stephen Tyree, Iuri Frosio, and Jan Kautz. Importance estimation for neural network pruning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 11264-11272, 2019.","title":"Importance estimation for neural network pruning","authors":"Pavlo Molchanov, Arun Mallya, Stephen Tyree, Iuri Frosio, Jan Kautz","year":"2019","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Oymak et al., 2019] Samet Oymak, Zalan Fabian, Mingchen Li, and Mahdi Soltanolkotabi. General- ization guarantees for neural networks via harnessing the low-rank structure of the jacobian. arXiv preprint arXiv:1906.05392, 2019.","title":"Generalization guarantees for neural networks via harnessing the low-rank structure of the jacobian","authors":"Samet Oymak, Zalan Fabian, Mingchen Li, Mahdi Soltanolkotabi","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Radford et al., 2019] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.","title":"Language models are unsupervised multitask learners","authors":"Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever","year":"2019","venue":"OpenAI blog","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Sanh et al., 2020] Victor Sanh, Thomas Wolf, and Alexan- der M Rush. Movement pruning: Adaptive sparsity by fine-tuning. In Advances in Neural Information Process- ing Systems, 2020.","title":"Movement pruning: Adaptive sparsity by fine-tuning","authors":"Victor Sanh, Thomas Wolf, Alexander M Rush","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Tanaka et al., 2020] Hidenori Tanaka, Daniel Kunin, Daniel L Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. Advances in Neural Information Processing Systems, 33, 2020.","title":"Pruning neural networks without any data by iteratively conserving synaptic flow","authors":"Hidenori Tanaka, Daniel Kunin, Surya Daniel L Yamins, Ganguli","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Wang et al., 2018] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and In- terpreting Neural Networks for NLP, 2018.","title":"Glue: A multi-task benchmark and analysis platform for natural language understanding","authors":"Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, Samuel Bowman","year":"2018","venue":"Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wang et al., 2020a] Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. In International Conference on Learning Representations, 2020.","title":"Picking winning tickets before training by preserving gradient flow","authors":"Wang","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wang et al., 2020b] Ziheng Wang, Jeremy Wohlwend, and Tao Lei. Structured pruning of large language models. In Proceedings of the 2020 Conference on Empirical Meth- ods in Natural Language Processing, 2020.","title":"Structured pruning of large language models","authors":"Wang","year":"2020","venue":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Zhang and He, 2020] Minjia Zhang and Yuxiong He. Ac- celerating training of transformer-based language models with progressive layer dropping. Advances in Neural In- formation Processing Systems, 33, 2020.","title":"Accelerating training of transformer-based language models with progressive layer dropping","authors":"He Zhang, Minjia Zhang, Yuxiong He","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zhang et al., 2021] Yuxin Zhang, Mingbao Lin, Fei Chao, Yan Wang, Yongjian Wu, Feiyue Huang, Mingliang Xu, Yonghong Tian, and Rongrong Ji. Lottery jackpots exist in pre-trained models. arXiv:2104.08700, 2021.","title":"Lottery jackpots exist in pre-trained models","authors":"Yuxin Zhang, Mingbao Lin, Fei Chao, Yan Wang, Yongjian Wu, Feiyue Huang, Mingliang Xu, Yonghong Tian, Rongrong Ji","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhao et al., 2020] Mengjie Zhao, Tao Lin, Fei Mi, Martin Jaggi, and Hinrich Schütze. Masking as an efficient alter- native to finetuning for pretrained language models. arXiv preprint arXiv:2004.12406, 2020.","title":"Masking as an efficient alternative to finetuning for pretrained language models","authors":"Mengjie Zhao, Tao Lin, Fei Mi, Martin Jaggi, Hinrich Schütze","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhu and Gupta, 2018] Michael H Zhu and Suyog Gupta. To prune, or not to prune: Exploring the efficacy of prun- ing for model compression. International Conference on Learning Representations, 2018.","title":"To prune, or not to prune: Exploring the efficacy of pruning for model compression","authors":"H Michael, Suyog Zhu, Gupta","year":"2018","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}