{"type":"ScholarlyArticle","title":"Faster and Lighter LLMs: A Survey on Current Challenges and Way Forward","authors":[{"name":"Arnav Chavan","slug":"arnav-chavan","given":"Arnav","family":"Chavan","affiliation":"Transmute AI Lab (Texmin Hub) , IIT (ISM) Dhanbad , India"},{"name":"Raghav Magazine","slug":"raghav-magazine","given":"Raghav","family":"Magazine","affiliation":"Nyun AI , India"},{"name":"Shubham Kushwaha","slug":"shubham-kushwaha","given":"Shubham","family":"Kushwaha","affiliation":"Nyun AI , India"},{"name":"Merouane Debbah","slug":"merouane-debbah","given":"Merouane","family":"Debbah","affiliation":"KU 6G Research Center , Khalifa University of Science and Technology , Abu Dhabi , UAE"},{"name":"Deepak Gupta","slug":"deepak-gupta","given":"Deepak","family":"Gupta","affiliation":"Transmute AI Lab (Texmin Hub) , IIT (ISM) Dhanbad , India"}],"abstract":"Despite the impressive performance of LLMs, their widespread adoption faces challenges due to substantial computational and memory requirements during inference. Recent advancements in model compression and system-level optimization methods aim to enhance LLM inference. This survey offers an overview of these methods, emphasizing recent developments. Through experiments on LLaMA(/2)-7B, we evaluate various compression techniques, providing practical insights for efficient LLM deployment in a unified setting. The empirical analysis on LLaMA(/2)-7B highlights the effectiveness of these methods. Drawing from survey insights, we identify current limitations and discuss potential future directions to improve LLM inference efficiency. We release the codebase to reproduce the results presented in this paper at https://github.com/nyunAI/Faster-LLM-Survey","doi":"10.24963/ijcai.2024/883","url":"https://ijcai.org/proceedings/2024/883","pdf":"https://ijcai.org/proceedings/2024/0883.pdf","markdown":"https://ijcai.org/proceedings/2024/883.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/883","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Survey Track","track":"Survey Track","number":883,"pages":{"first":7980,"last":7988},"keywords":["Natural Language Processing: NLP: Resources and evaluation","Natural Language Processing: NLP: Language models"],"videos":[],"references":[{"position":1,"raw":"[Agarwal et al., 2023] R. Agarwal, N. Vieillard, Y. Zhou, P. Stanczyk, S. Ramos, M. Geist, and O. Bachem. Gener- alized knowledge distillation for auto-regressive language models. arXiv, 2023.","title":"Generalized knowledge distillation for auto-regressive language models","authors":"R Agarwal, N Vieillard, Y Zhou, P Stanczyk, S Ramos, M Geist, O Bachem","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[An et al., 2023] Y. An, X. Zhao, T. Yu, M. Tang, and J. Wang. Fluctuation-based adaptive structured pruning for large language models. arXiv, 2023.","title":"Fluctuation-based adaptive structured pruning for large language models","authors":"Y An, X Zhao, T Yu, M Tang, J Wang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Banner et al., 2019] R. Banner, Y. Nahshan, E. Hoffer, and D. Soudry. Post-training 4-bit quantization of convolution networks for rapid-deployment. arXiv, 2019.","title":"Post-training 4-bit quantization of convolution networks for rapid-deployment","authors":"R Banner, Y Nahshan, E Hoffer, D Soudry","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Brown et al., 2020] T Brown, B Mann, N Ryder, M Sub- biah, J D Kaplan, P Dhariwal, A Neelakantan, P Shyam, G Sastry, A Askell, et al. Language models are few-shot learners. NeurIPS, 33:1877-1901, 2020.","title":"Language models are few-shot learners","authors":"Brown, N Mann, M Ryder, J D Subbiah, P Kaplan, Dhariwal, P Neelakantan, G Shyam, Sastry, Askell","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chavan et al., 2022] A Chavan, Z Shen, Z Liu, Z Liu, K T Cheng, and E P Xing. Vision transformer slim- ming: Multi-dimension searching in continuous optimiza- tion space. In CVPR, pages 4931-4941, 2022.","title":"Vision transformer slimming: Multi-dimension searching in continuous optimization space","authors":"Chavan, Z Shen, Z Liu, K T Liu, E P Cheng, Xing","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chavan et al., 2023] A. Chavan, N. Lele, and D. Gupta. Re- thinking compression: Reduced order modelling of latent features in large language models. arXiv, 2023.","title":"Rethinking compression: Reduced order modelling of latent features in large language models","authors":"A Chavan, N Lele, D Gupta","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chen et al., 2023a] T. Chen, T. Ding, B. Yadav, I. Zharkov, and L. Liang. Lorashear: Efficient large language model structured pruning and knowledge recovery. arXiv, 2023.","title":"Lorashear: Efficient large language model structured pruning and knowledge recovery","authors":"Chen","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chen et al., 2023b] Z. Chen, Q. Gao, A. Bosselut, A. Sab- harwal, and K. Richardson. Disco: Distilling counterfac- tuals with large language models. arXiv, 2023.","title":"Disco: Distilling counterfactuals with large language models","authors":"Chen","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Choukroun et al., 2019] Y. Choukroun, E. Kravchik, F. Yang, and P. Kisilev. Low-bit quantization of neural networks for efficient inference. arXiv, 2019.","title":"Low-bit quantization of neural networks for efficient inference","authors":"Y Choukroun, E Kravchik, F Yang, P Kisilev","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Dao et al., 2022] T. Dao, D. Fu, S. Ermon, A. Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. NeurIPS, 35:16344- 16359, 2022.","title":"Flashattention: Fast and memory-efficient exact attention with io-awareness","authors":"T Dao, D Fu, S Ermon, A Rudra, Christopher Ré","year":"2022","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Dao, 2023] Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv, 2023.","title":"Flashattention-2: Faster attention with better parallelism and work partitioning","authors":"Tri Dao","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Denton et al., 2014] E. Denton, W. Zaremba, J. Bruna, Y. LeCun, and R. Fergus. Exploiting linear structure within convolutional networks for efficient evaluation. arXiv, 2014.","title":"Exploiting linear structure within convolutional networks for efficient evaluation","authors":"E Denton, W Zaremba, J Bruna, Y Lecun, R Fergus","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Dettmers et al., 2022] T. Dettmers, M. Lewis, Y. Belkada, and L. Zettlemoyer. Llm.int8(): 8-bit matrix multiplication for transformers at scale. arXiv, 2022.","title":"8-bit matrix multiplication for transformers at scale","authors":"T Dettmers, M Lewis, Y Belkada, L Zettlemoyer","year":"2022","venue":"Llm.int8","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Dettmers et al., 2023] T. Dettmers, A. Pagnoni, A. Holtz- man, and L. Zettlemoyer. Qlora: Efficient finetuning of quantized llms. arXiv, 2023.","title":"Qlora: Efficient finetuning of quantized llms","authors":"T Dettmers, A Pagnoni, A Holtzman, L Zettlemoyer","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Esser et al., 2019] S. K Esser, J. L McKinstry, D. Bablani, R. Appuswamy, and D. S Modha. Learned step size quan- tization. arXiv, 2019.","title":"Learned step size quantization","authors":"S Esser, J Mckinstry, D Bablani, R Appuswamy, D S Modha","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Frankle and Carbin, 2019] J. Frankle and M. Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv, 2019. [Frantar and Alistarh, 2023]","title":"The lottery ticket hypothesis: Finding sparse, trainable neural networks","authors":"Carbin Frankle, ] J Frankle, M Carbin","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"E. Frantar and D. Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. arXiv, 2023.","title":"Sparsegpt: Massive language models can be accurately pruned in one-shot","authors":"E Frantar, D Alistarh","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Frantar et al., 2023] E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh. Gptq: Accurate post-training quantiza- tion for generative pre-trained transformers. arXiv, 2023.","title":"Gptq: Accurate post-training quantization for generative pre-trained transformers","authors":"E Frantar, S Ashkboos, T Hoefler, D Alistarh","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Hassibi et al., 1993] B. Hassibi, D. G Stork, and G. J Wolff. Optimal brain surgeon and general network pruning. In ICNN, pages 293-299. IEEE, 1993.","title":"Optimal brain surgeon and general network pruning","authors":"B Hassibi, D G Stork, G J Wolff","year":"1993","venue":"ICNN","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Hinton et al., 2015] G. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in a neural network. arXiv, 2015.","title":"Distilling the knowledge in a neural network","authors":"G Hinton, O Vinyals, J Dean","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Hu et al., 2021] E. J Hu, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen, et al. Lora: Low-rank adap- tation of large language models. In ICLR, 2021.","title":"Low-rank adaptation of large language models","authors":"E Hu, P Wallis, Z Allen-Zhu, Y Li, S Wang, L Wang, W Chen","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Jaderberg et al., 2014] M. Jaderberg, A. Vedaldi, and A. Zisserman. Speeding up convolutional neural networks with low rank expansions. arXiv, 2014.","title":"Speeding up convolutional neural networks with low rank expansions","authors":"M Jaderberg, A Vedaldi, A Zisserman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Janowsky, 1989] S. A. Janowsky. Pruning versus clipping in neural networks. Phys. Rev. A, 39:6600-6603, 1989.","title":"Pruning versus clipping in neural networks","authors":"S A Janowsky","year":"1989","venue":"Phys. Rev. A","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Jha et al., 2023] A. H. Jha, T. S., E. P. Walsh, D. Groen- eveld, E. Strubell, and I. Beltagy. How to train your (com- pressed) large language model. arXiv, 2023.","title":"How to train your (compressed) large language model","authors":"A H Jha, T S, E P Walsh, D Groeneveld, E Strubell, I Beltagy","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Jiang et al., 2023a] T. Jiang, D. Wang, F. Zhuang, R. Xie, and F. Xia. Pruning pre-trained language models without fine-tuning. arXiv, 2023.","title":"Pruning pre-trained language models without fine-tuning","authors":"Jiang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Jiang et al., 2023b] Y. Jiang, C. Chan, M. Chen, and W. Wang. Lion: Adversarial distillation of proprietary large language models. arXiv, 2023.","title":"Lion: Adversarial distillation of proprietary large language models","authors":"Jiang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Kaushal et al., 2023] A. Kaushal, T. Vaidhya, and I. Rish. Lord: Low rank decomposition of monolingual code llms for one-shot compression. arXiv, 2023.","title":"Lord: Low rank decomposition of monolingual code llms for one-shot compression","authors":"A Kaushal, T Vaidhya, I Rish","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Kenton and Toutanova, 2019] Jacob Devlin Ming- Wei Chang Kenton and L. Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for lan- guage understanding. In Proceedings of NAACL-HLT, pages 4171-4186, 2019.","title":"Bert: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton, L, Kristina Toutanova","year":"2019","venue":"Proceedings of NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Kim et al., 2023] S. Kim, C. Hooper, A. Gholami, Z. Dong, X. Li, S. Shen, M. W. Mahoney, and K. Keutzer. Squeezellm: Dense-and-sparse quantization. arXiv, 2023.","title":"Squeezellm: Dense-and-sparse quantization","authors":"S Kim, C Hooper, A Gholami, Z Dong, X Li, S Shen, M W Mahoney, K Keutzer","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Kwon et al., 2023] W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pages 611- 626, 2023.","title":"Efficient memory management for large language model serving with pagedattention","authors":"W Kwon, Z Li, S Zhuang, Y Sheng, L Zheng, C H Yu, J Gonzalez, H Zhang, I Stoica","year":"2023","venue":"Proceedings of the 29th Symposium on Operating Systems Principles","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Lagunas et al., 2021] F. Lagunas, E. Charlaix, V. Sanh, and A. M. Rush. Block pruning for faster transformers. arXiv, 2021.","title":"Block pruning for faster transformers","authors":"F Lagunas, E Charlaix, V Sanh, A M Rush","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Lebedev et al., 2015] V. Lebedev, Y. Ganin, M. Rakhuba, I. Oseledets, and V. Lempitsky. Speeding-up convolu- tional neural networks using fine-tuned cp-decomposition. arXiv, 2015.","title":"Speeding-up convolutional neural networks using fine-tuned cp-decomposition","authors":"V Lebedev, Y Ganin, M Rakhuba, I Oseledets, V Lempitsky","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[LeCun et al., 1989] Y. LeCun, J. Denker, and S. Solla. Op- timal brain damage. NeurIPS, 2, 1989.","title":"Optimal brain damage","authors":"Y Lecun, J Denker, S Solla","year":"1989","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Leviathan et al., 2023] Y. Leviathan, M. Kalman, and Y. Matias. Fast inference from transformers via specu- lative decoding. In ICML, pages 19274-19286. PMLR, 2023.","title":"Fast inference from transformers via speculative decoding","authors":"Y Leviathan, M Kalman, Y Matias","year":"2023","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Li et al., 2016] H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf. Pruning filters for efficient convnets. arXiv, 2016.","title":"Pruning filters for efficient convnets","authors":"H Li, A Kadav, I Durdanovic, H Samet, H P Graf","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Li et al., 2023] Y. Li, Y. Yu, Q. Zhang, C. Liang, P. He, W. Chen, and T. Zhao. Losparse: Structured compression of large language models based on low-rank and sparse approximation. arXiv, 2023.","title":"Losparse: Structured compression of large language models based on low-rank and sparse approximation","authors":"Y Li, Y Yu, Q Zhang, C Liang, P He, W Chen, T Zhao","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Liang et al., 2023] C. Liang, S. Zuo, Q. Zhang, P. He, W. Chen, and T. Zhao. Less is more: Task-aware layer- wise distillation for language model compression. arXiv, 2023.","title":"Less is more: Task-aware layerwise distillation for language model compression","authors":"C Liang, S Zuo, Q Zhang, P He, W Chen, T Zhao","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Lin et al., 2023] J. Lin, J. Tang, H. Tang, S. Yang, X. Dang, C. Gan, and S. Han. Awq: Activation-aware weight quan- tization for llm compression and acceleration. arXiv, 2023.","title":"Awq: Activation-aware weight quantization for llm compression and acceleration","authors":"J Lin, J Tang, H Tang, S Yang, X Dang, C Gan, S Han","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Liu et al., 2017] Z. Liu, J. Li, Z. Shen, G. Huang, S. Yan, and C. Zhang. Learning efficient convolutional networks through network slimming. arXiv, 2017.","title":"Learning efficient convolutional networks through network slimming","authors":"Z Liu, J Li, Z Shen, G Huang, S Yan, C Zhang","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Liu et al., 2023] Z. Liu, B. Oguz, C. Zhao, E. Chang, P. Stock, Y. Mehdad, Y. Shi, R. Krishnamoorthi, and V. Chandra. Llm-qat: Data-free quantization aware train- ing for large language models. arXiv, 2023.","title":"Llm-qat: Data-free quantization aware training for large language models","authors":"Z Liu, B Oguz, C Zhao, E Chang, P Stock, Y Mehdad, Y Shi, R Krishnamoorthi, V Chandra","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Ma et al., 2023] X. Ma, G. Fang, and X. Wang. Llm-pruner: On the structural pruning of large language models. arXiv, 2023.","title":"Llm-pruner: On the structural pruning of large language models","authors":"Ma","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Molchanov et al., 2016] P. Molchanov, S. Tyree, T. Karras, T. Aila, and J. Kautz. Pruning convolutional neural net- works for resource efficient inference. In ICLR, 2016.","title":"Pruning convolutional neural networks for resource efficient inference","authors":"P Molchanov, S Tyree, T Karras, T Aila, J Kautz","year":"2016","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Ni et al., 2020] R. Ni, H. M. Chu, O. Castañeda, P. Y. Chi- ang, C. Studer, and T. Goldstein. Wrapnet: Neural net in- ference with ultra-low-resolution arithmetic. arXiv, 2020.","title":"Wrapnet: Neural net inference with ultra-low-resolution arithmetic","authors":"R Ni, H M Chu, O Castañeda, P Y Chiang, C Studer, T Goldstein","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":44,"raw":"[S. et al., 2023] Hang S., Bei L., and Yanmin Q. One-Shot Sensitivity-Aware Mixed Sparsity Pruning for Large Lan- guage Models. arXiv, 2023.","title":"One-Shot Sensitivity-Aware Mixed Sparsity Pruning for Large Language Models","authors":"Hang S Bei, L, Yanmin Q","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":45,"raw":"[Sainath et al., 2013] T. N. Sainath, B. Kingsbury, V. Sind- hwani, E. Arisoy, and B. Ramabhadran. Low-rank matrix factorization for deep neural network training with high- dimensional output targets. In ICASSP, pages 6655-6659, 2013.","title":"Low-rank matrix factorization for deep neural network training with highdimensional output targets","authors":"T N Sainath, B Kingsbury, V Sindhwani, E Arisoy, B Ramabhadran","year":"2013","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":46,"raw":"[Sanh et al., 2020] V. Sanh, T. Wolf, and A. M. Rush. Move- ment pruning: Adaptive sparsity by fine-tuning. arXiv, 2020.","title":"Movement pruning: Adaptive sparsity by fine-tuning","authors":"V Sanh, T Wolf, A M Rush","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":47,"raw":"[Shao et al., 2023] W. Shao, M. Chen, Z. Zhang, P. Xu, L. Zhao, Z. Li, K. Zhang, P. Gao, Y. Qiao, and P. Luo. Omniquant: Omnidirectionally calibrated quantization for large language models. arXiv, 2023.","title":"Omniquant: Omnidirectionally calibrated quantization for large language models","authors":"W Shao, M Chen, Z Zhang, P Xu, L Zhao, Z Li, K Zhang, P Gao, Y Qiao, P Luo","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":48,"raw":"[Sharma et al., 2023] P. Sharma, J. T. Ash, and D. Misra. The truth is in there: Improving reasoning in language models with layer-selective rank reduction. arXiv, 2023.","title":"The truth is in there: Improving reasoning in language models with layer-selective rank reduction","authors":"P Sharma, J T Ash, D Misra","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":49,"raw":"[Song et al., 2023] Y. Song, Z. Mi, H. Xie, and H. Chen. Powerinfer: Fast large language model serving with a consumer-grade gpu. arXiv, 2023.","title":"Powerinfer: Fast large language model serving with a consumer-grade gpu","authors":"Y Song, Z Mi, H Xie, H Chen","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":50,"raw":"[Sun et al., 2023] M. Sun, Z. Liu, A. Bair, and J. Z. Kolter. A simple and effective pruning approach for large language models. arXiv, 2023.","title":"A simple and effective pruning approach for large language models","authors":"M Sun, Z Liu, A Bair, J Z Kolter","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":51,"raw":"[Tai et al., 2016] C. Tai, T. Xiao, Y. Zhang, X. Wang, and Weinan E. Convolutional neural networks with low-rank regularization. arXiv, 2016.","title":"Convolutional neural networks with low-rank regularization","authors":"C Tai, T Xiao, Y Zhang, X Wang, Weinan E","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":52,"raw":"[Tailor et al., 2021] S. A. Tailor, J. F. Marques, and N. D. Lane. Degree-quant: Quantization-aware training for graph neural networks. arXiv, 2021.","title":"Degree-quant: Quantization-aware training for graph neural networks","authors":"S A Tailor, J F Marques, N D Lane","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":53,"raw":"[Tao et al., 2022] C. Tao, L. Hou, W. Zhang, L. Shang, X. Jiang, Q. Liu, P. Luo, and N. Wong. Compression of generative pre-trained language models via quantization. arXiv, 2022.","title":"Compression of generative pre-trained language models via quantization","authors":"C Tao, L Hou, W Zhang, L Shang, X Jiang, Q Liu, P Luo, N Wong","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":54,"raw":"[Tian et al., 2022] Yo. Tian, D. Krishnan, and P. Isola. Con- trastive representation distillation. arXiv, 2022.","title":"Contrastive representation distillation","authors":"Tian","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":55,"raw":"[Touvron et al., 2023a] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M. A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv, 2023.","title":"Llama: Open and efficient foundation language models","authors":"Touvron","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":56,"raw":"[Touvron et al., 2023b] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv, 2023.","title":"Llama 2: Open foundation and fine-tuned chat models","authors":"Touvron","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":57,"raw":"[Wang et al., 2023a] H. Wang, S. Ma, L. Dong, S. Huang, H. Wang, L. Ma, F. Yang, R. Wang, Y. Wu, and F. Wei. Bitnet: Scaling 1-bit transformers for large language mod- els. arXiv, 2023.","title":"Bitnet: Scaling 1-bit transformers for large language models","authors":"Wang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":58,"raw":"[Wang et al., 2023b] P. Wang, Z. Wang, Z. Li, Y. Gao, B. Yin, and X. Ren. Scott: Self-consistent chain-of- thought distillation. arXiv, 2023.","title":"Scott: Self-consistent chain-ofthought distillation","authors":"Wang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":59,"raw":"[Wu et al., 2023] X. Wu, Z. Yao, and Y. He. Zeroquant-fp: A leap forward in llms post-training w4a8 quantization using floating-point formats. arXiv, 2023.","title":"Zeroquant-fp: A leap forward in llms post-training w4a8 quantization using floating-point formats","authors":"Wu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":60,"raw":"[Xia et al., 2023] M. Xia, T. Gao, Z. Zeng, and D. Chen. Sheared llama: Accelerating language model pre-training via structured pruning. arXiv, 2023.","title":"Sheared llama: Accelerating language model pre-training via structured pruning","authors":"M Xia, T Gao, Z Zeng, D Chen","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":61,"raw":"[Xiao et al., 2023] G. Xiao, J. Lin, M. Seznec, H. Wu, J. De- mouth, and S. Han. Smoothquant: Accurate and effi- cient post-training quantization for large language models. arXiv, 2023.","title":"Smoothquant: Accurate and efficient post-training quantization for large language models","authors":"G Xiao, J Lin, M Seznec, H Wu, J Demouth, S Han","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":62,"raw":"[Xu et al., 2023] M. Xu, Y. Lei Xu, and D. P. Mandic. Ten- sorgpt: Efficient compression of the embedding layer in llms based on the tensor-train decomposition. arXiv, 2023.","title":"Tensorgpt: Efficient compression of the embedding layer in llms based on the tensor-train decomposition","authors":"M Xu, Y Lei Xu, D P Mandic","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":63,"raw":"[Zhang et al., 2023] M. Zhang, H. Chen, C. Shen, Z. Yang, L. Ou, X. Yu, and B. Zhuang. Loraprune: Pruning meets low-rank parameter-efficient fine-tuning. arXiv, 2023.","title":"Loraprune: Pruning meets low-rank parameter-efficient fine-tuning","authors":"M Zhang, H Chen, C Shen, Z Yang, L Ou, X Yu, B Zhuang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":64,"raw":"[Zhu et al., 2023] X. Zhu, J. Li, Y. Liu, C. Ma, and W. Wang. A survey on model compression for large language mod- els. arXiv, 2023.","title":"A survey on model compression for large language models","authors":"Zhu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}