{"type":"ScholarlyArticle","title":"Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Models","authors":[{"name":"Xindi Wang","slug":"xindi-wang","given":"Xindi","family":"Wang","affiliation":"Huawei Technologies Canada , Canada; University of Western Ontario , Canada; Vector Institute for Artificial Intelligence , Canada"},{"name":"Mahsa Salmani","slug":"mahsa-salmani","given":"Mahsa","family":"Salmani","affiliation":"Huawei Technologies Canada , Canada"},{"name":"Parsa Omidi","slug":"parsa-omidi","given":"Parsa","family":"Omidi","affiliation":"Huawei Technologies Canada , Canada"},{"name":"Xiangyu Ren","slug":"xiangyu-ren","given":"Xiangyu","family":"Ren","affiliation":"Huawei Technologies Canada , Canada"},{"name":"Mehdi Rezagholizadeh","slug":"mehdi-rezagholizadeh","given":"Mehdi","family":"Rezagholizadeh","affiliation":"Huawei Technologies Canada , Canada"},{"name":"Armaghan Eshaghi","slug":"armaghan-eshaghi","given":"Armaghan","family":"Eshaghi","affiliation":"Huawei Technologies Canada , Canada"}],"abstract":"Recently, large language models (LLMs) have shown remarkable capabilities including understanding context, engaging in logical reasoning, and generating responses. However, this is achieved at the expense of stringent computational and memory requirements, hindering their ability to effectively support long input sequences. This survey provides an inclusive review of the recent techniques and methods devised to extend the sequence length in LLMs, thereby enhancing their capacity for long-context understanding. In particular, we review and categorize a wide range of techniques including architectural modifications, such as modified positional encoding and altered attention mechanisms, which are designed to enhance the processing of longer sequences while avoiding a proportional increase in computational cost. The diverse methodologies investigated in this study can be leveraged across different phases of LLMs, i.e., training, fine-tuning and inference. This enables LLMs to efficiently process extended sequences. The limitations of the current methodologies is discussed in the last section along with the suggestions for future research directions, underscoring the importance of sequence length in the continued advancement of LLMs.","doi":"10.24963/ijcai.2024/917","url":"https://ijcai.org/proceedings/2024/917","pdf":"https://ijcai.org/proceedings/2024/0917.pdf","markdown":"https://ijcai.org/proceedings/2024/917.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/917","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Survey Track","track":"Survey Track","number":917,"pages":{"first":8299,"last":8307},"keywords":["Natural Language Processing: NLP: Language models","Natural Language Processing: General"],"videos":[],"references":[{"position":1,"raw":"[Ainslie et al., 2023] Joshua Ainslie, James Lee-Thorp, et al. GQA: Training generalized multi-query transformer mod- els from multi-head checkpoints. In EMNLP, 2023.","title":"GQA: Training generalized multi-query transformer models from multi-head checkpoints","authors":"Joshua Ainslie, James Lee-Thorp","year":"2023","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Alizadeh et al., 2023] Keivan Alizadeh, Iman Mirzadeh, et al. LLM in a flash: Efficient large language model in- ference with limited memory. arXiv:2312.11514, 2023.","title":"LLM in a flash: Efficient large language model inference with limited memory","authors":"Keivan Alizadeh, Iman Mirzadeh","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Anil et al., 2023] Rohan Anil, Andrew M Dai, et al. PaLM 2 technical report. arXiv:2305.10403, 2023.","title":"PaLM 2 technical report","authors":"Rohan Anil, Andrew M Dai","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Beltagy et al., 2020] Iz Beltagy, Matthew E Peters, and Ar- man Cohan. Longformer: The long-document trans- former. arXiv:2004.05150, 2020.","title":"Iz Beltagy, Matthew E Peters, and Arman Cohan","authors":"Beltagy","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Brown et al., 2020] Tom B. Brown, Benjamin Mann, et al. Language models are few-shot learners. In NeurIPS, 2020.","title":"Language models are few-shot learners","authors":"B Tom, Benjamin Brown, Mann","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen et al., 2021] Pu-Chin Chen, Henry Tsai, et al. A sim- ple and effective positional encoding for transformers. In EMNLP, 2021.","title":"A simple and effective positional encoding for transformers","authors":"Pu-Chin Chen, Henry Tsai","year":"2021","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chen et al., 2023] Shouyuan Chen, Sherman Wong, et al. Extending context window of large language models via positional interpolation. arXiv:2306.15595, 2023.","title":"Extending context window of large language models via positional interpolation","authors":"Chen","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chen et al., 2024] Guanzheng Chen, Xin Li, et al. CLEX: Continuous length extrapolation for large language mod- els. In ICLR, 2024.","title":"CLEX: Continuous length extrapolation for large language models","authors":"Chen","year":"2024","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Choromanski et al., 2021] Krzysztof M. Choromanski, Va- lerii Likhosherstov, et al. Rethinking attention with per- formers. In ICLR, 2021.","title":"Valerii Likhosherstov, et al. Rethinking attention with performers","authors":"M Krzysztof, Choromanski","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Chowdhery et al., 2024] Aakanksha Chowdhery, Sharan Narang, et al. PaLM: scaling language modeling with pathways. JMLR, 2024.","title":"PaLM: scaling language modeling with pathways","authors":"Aakanksha Chowdhery, Sharan Narang","year":"2024","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Dao et al., 2022] Tri Dao, Dan Fu, et al. Flashatten- tion: Fast and memory-efficient exact attention with io- awareness. In NeurIPS, 2022.","title":"Flashattention: Fast and memory-efficient exact attention with ioawareness","authors":"Tri Dao, Dan Fu","year":"2022","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Dao et al., 2023] Tri Dao, Daniel Haziza, et al. Flash- Decoding for long-context inference, 2023.","title":"Flash-Decoding for long-context inference","authors":"Tri Dao, Daniel Haziza","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[de Jong et al., 2023] Michiel de Jong, Yury Zemlyanskiy, et al. FiDO: Fusion-in-decoder optimized for stronger per- formance and faster inference. In Findings of ACL, 2023.","title":"FiDO: Fusion-in-decoder optimized for stronger performance and faster inference","authors":"Jong Michiel De, Yury Zemlyanskiy","year":"2023","venue":"Findings of ACL","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Dettmers et al., 2022] Tim Dettmers, Mike Lewis, et al. LLM.int8(): 8-bit matrix multiplication for transformers at scale. In NeurIPS, 2022.","title":"8-bit matrix multiplication for transformers at scale","authors":"Tim Dettmers, Mike Lewis","year":"2022","venue":"LLM.int8","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ding et al., 2023] Jiayu Ding, Shuming Ma, et al. LongNet: Scaling transformers to 1,000,000,000 to- kens. arXiv:2307.02486, 2023.","title":"LongNet: Scaling transformers to 1","authors":"Ding","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Frantar and Alistarh, 2023] Elias Frantar and Dan Alistarh. SparseGPT: Massive language models can be accurately pruned in one-shot. In ICML, 2023.","title":"SparseGPT: Massive language models can be accurately pruned in one-shot","authors":"Alistarh Frantar, Elias Frantar, Dan Alistarh","year":"2023","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Frantar et al., 2023] Elias Frantar, Saleh Ashkboos, et al. OPTQ: Accurate quantization for generative pre-trained transformers. In ICLR, 2023.","title":"OPTQ: Accurate quantization for generative pre-trained transformers","authors":"Elias Frantar, Saleh Ashkboos","year":"2023","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Fu et al., 2023] Daniel Y Fu, Tri Dao, et al. Hungry Hun- gry Hippos: Towards language modeling with state space models. In ICLR, 2023.","title":"Hungry Hungry Hippos: Towards language modeling with state space models","authors":"Tri Daniel Y Fu, Dao","year":"2023","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Gholami et al., 2022] Amir Gholami, Sehoon Kim, et al. A survey of quantization methods for efficient neural net- work inference. In Low-Power Computer Vision, 2022.","title":"A survey of quantization methods for efficient neural network inference","authors":"Gholami","year":"2022","venue":"Low-Power Computer Vision","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Gu and Dao, 2023] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv:2312.00752, 2023.","title":"Mamba: Linear-time sequence modeling with selective state spaces","authors":"Dao Gu, Albert Gu, Tri Dao","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"Gu et al., ] A. Gu, T. Dao, et al. HiPPO: Recurrent memory with optimal polynomial projections. In NeurIPS,2020.","title":"HiPPO: Recurrent memory with optimal polynomial projections","authors":"Gu","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Gu et al., 2022] Albert Gu, Karan Goel, and Christopher Re. Efficiently modeling long sequences with structured state spaces. In ICLR, 2022.","title":"Efficiently modeling long sequences with structured state spaces","authors":"Albert Gu, Karan Goel, Christopher Re","year":"2022","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Hao et al., 2022] Yaru Hao, Yutao Sun, et al. Structured prompting: Scaling in-context learning to 1,000 examples. arXiv:2212.06713, 2022.","title":"Structured prompting: Scaling in-context learning to 1","authors":"Yaru Hao, Yutao Sun","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Hong et al., 2023] Ke Hong, Guohao Dai, et al. FlashDe- coding++: Faster large language model inference on gpus. arXiv:2311.01282, 2023.","title":"FlashDe-coding++: Faster large language model inference on gpus","authors":"Ke Hong, Guohao Dai","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Huang et al., 2023] Yunpeng Huang, Jingwei Xu, et al. Advancing transformer architecture in long-context large language models: A comprehensive survey. arXiv:2311.12351, 2023.","title":"Advancing transformer architecture in long-context large language models: A comprehensive survey","authors":"Yunpeng Huang, Jingwei Xu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Jiang et al., 2023a] Huiqiang Jiang, Qianhui Wu, et al. LLMLingua: Compressing prompts for accelerated infer- ence of large language models. In EMNLP, 2023.","title":"LLMLingua: Compressing prompts for accelerated inference of large language models","authors":"Jiang","year":"2023","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Jiang et al., 2023b] Huiqiang Jiang, Qianhui Wu, et al. LongLLMLingua: Accelerating and enhancing LLMs in long context scenarios via prompt compression. arXiv:2310.06839, 2023.","title":"LongLLMLingua: Accelerating and enhancing LLMs in long context scenarios via prompt compression","authors":"Jiang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Kitaev et al., 2020] Nikita Kitaev, Lukasz Kaiser, et al. Re- former: The efficient transformer. In ICLR, 2020.","title":"Reformer: The efficient transformer","authors":"Nikita Kitaev, Lukasz Kaiser","year":"2020","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Koohpayegani and Pirsiavash, 2024] Soroush A. Kooh- payegani and Hamed Pirsiavash. SimA: Simple softmax- free attention for vision transformers. In WACV, 2024.","title":"SimA: Simple softmaxfree attention for vision transformers","authors":"A Soroush, Hamed Koohpayegani, Pirsiavash","year":"2024","venue":"WACV","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Kwan et al., 2023] Wai-Chung Kwan, Xingshan Zeng, et al. M4LE: A multi-ability multi-range multi-task multi- domain long-context evaluation benchmark for large lan- guage models. arXiv:2310.19240, 2023.","title":"M4LE: A multi-ability multi-range multi-task multidomain long-context evaluation benchmark for large language models","authors":"Wai-Chung Kwan, Xingshan Zeng","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Kwon et al., 2023] Woosuk Kwon, Zhuohan Li, et al. Effi- cient memory management for large language model serv- ing with pagedattention. In SOSP, 2023.","title":"Efficient memory management for large language model serving with pagedattention","authors":"Kwon","year":"2023","venue":"SOSP","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Lee et al., 2024] Changhun Lee, Jungyu Jin, et al. OWQ: Outlier-aware weight quantization for efficient fine-tuning and inference of large language models. In AAAI, 2024.","title":"OWQ: Outlier-aware weight quantization for efficient fine-tuning and inference of large language models","authors":"Lee","year":"2024","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Li et al., 2019] Shiyang Li, Xiaoyong Jin, et al. Enhancing the locality and breaking the memory bottleneck of trans- former on time series forecasting. In NeurIPS, 2019.","title":"Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting","authors":"Shiyang Li, Xiaoyong Jin","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Li et al., 2023a] Raymond Li, Loubna Ben allal, et al. Star- Coder: may the source be with you! TMLR, 2023.","title":"Star-Coder: may the source be with you! TMLR","authors":"Li","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Li et al., 2023b] Yucheng Li, Bo Dong, et al. Compressing context to enhance inference efficiency of large language models. In EMNLP, 2023.","title":"Compressing context to enhance inference efficiency of large language models","authors":"Li","year":"2023","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Lin et al., 2024] Ji Lin, Jiaming Tang, et al. AWQ: Activation-aware weight quantization for LLM compres- sion and acceleration. In MLSys, 2024.","title":"AWQ: Activation-aware weight quantization for LLM compression and acceleration","authors":"Ji Lin, Jiaming Tang","year":"2024","venue":"MLSys","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Liu and Abbeel, 2023] Hao Liu and Pieter Abbeel. Block- wise parallel transformers for large context models. In NeurIPS, 2023.","title":"Blockwise parallel transformers for large context models","authors":"Hao Liu, Pieter Abbeel","year":"2023","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Liu et al., 2023a] Hao Liu, Matei Zaharia, and Pieter Abbeel. Ring attention with blockwise transformers for near-infinite context. In NeurIPS FMDM Workshop, 2023.","title":"Ring attention with blockwise transformers for near-infinite context","authors":"Liu","year":"2023","venue":"NeurIPS FMDM Workshop","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Liu et al., 2023b] Zechun Liu, Barlas Oguz, et al. LLM- QAT: Data-free quantization aware training for large lan- guage models. arXiv:2305.17888, 2023.","title":"LLM-QAT: Data-free quantization aware training for large language models","authors":"Liu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Lu et al., 2021] Jiachen Lu, Jinghan Yao, et al. SOFT: Softmax-free transformer with linear complexity. In NeurIPS, 2021.","title":"SOFT: Softmax-free transformer with linear complexity","authors":"Lu","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Ma et al., 2023a] Xinyin Ma, Gongfan Fang, and Xinchao Wang. LLM-Pruner: On the structural pruning of large language models. In NeurIPS, 2023.","title":"LLM-Pruner: On the structural pruning of large language models","authors":"Ma","year":"2023","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Ma et al., 2023b] Xuezhe Ma, C. Zhou, et al. MEGA: Mov- ing average equipped gated attention. In ICLR, 2023.","title":"MEGA: Moving average equipped gated attention","authors":"Ma","year":"2023","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Naveed et al., 2023] Humza Naveed, Asad Ullah Khan, et al. A comprehensive overview of large language mod- els. arXiv:2307.06435, 2023.","title":"A comprehensive overview of large language models","authors":"Humza Naveed, Asad Ullah Khan","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":44,"raw":"[Orvieto et al., 2023] Antonio Orvieto, Samuel L Smith, et al. Resurrecting recurrent neural networks for long se- quences. In ICML, 2023.","title":"Resurrecting recurrent neural networks for long sequences","authors":"Orvieto","year":"2023","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":45,"raw":"[Packer et al., 2023] Charles Packer, Vivian Fang, et al. MemGPT: Towards LLMs as operating systems. arXiv:2310.08560, 2023.","title":"MemGPT: Towards LLMs as operating systems","authors":"Charles Packer, Vivian Fang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":46,"raw":"[Peng et al., 2023a] Bo Peng, Eric Alcaide, et al. RWKV: Reinventing RNNs for the transformer era. In Findings of EMNLP, 2023.","title":"RWKV: Reinventing RNNs for the transformer era","authors":"Peng","year":"2023","venue":"Findings of EMNLP","doi":null,"url":null,"internal_url":null},{"position":47,"raw":"[Peng et al., 2023b] Bowen Peng, Jeffrey Quesnelle, et al. YaRN: Efficient context window extension of large lan- guage models. In ICLR, 2023.","title":"YaRN: Efficient context window extension of large language models","authors":"Peng","year":"2023","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":48,"raw":"[Poli et al., 2023] Michael Poli, Stefano Massaroli, et al. Hyena Hierarchy: towards larger convolutional language models. In ICML, 2023.","title":"Hyena Hierarchy: towards larger convolutional language models","authors":"Michael Poli, Stefano Massaroli","year":"2023","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":49,"raw":"[Pope et al., 2023] Reiner Pope, Sholto Douglas, et al. Effi- ciently scaling transformer inference. MLSys, 2023.","title":"Efficiently scaling transformer inference","authors":"Reiner Pope, Sholto Douglas","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":50,"raw":"[Press et al., 2022] Ofir Press, Noah Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. In ICLR, 2022.","title":"Train short, test long: Attention with linear biases enables input length extrapolation","authors":"Ofir Press, Noah Smith, Mike Lewis","year":"2022","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":51,"raw":"[Qin et al., 2022] Zhen Qin, Weixuan Sun, et al. cosFormer: Rethinking softmax in attention. In ICLR, 2022.","title":"cosFormer: Rethinking softmax in attention","authors":"Zhen Qin, Weixuan Sun","year":"2022","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":52,"raw":"[Qiu et al., 2020] Jiezhong Qiu, Hao Ma, et al. Blockwise self-attention for long document understanding. In Find- ings of EMNLP, 2020.","title":"Blockwise self-attention for long document understanding","authors":"Qiu","year":"2020","venue":"Findings of EMNLP","doi":null,"url":null,"internal_url":null},{"position":53,"raw":"[Ratner et al., 2023] Nir Ratner, Yoav Levine, et al. Parallel context windows for large language models. In ACL, 2023.","title":"Parallel context windows for large language models","authors":"Nir Ratner, Yoav Levine","year":"2023","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":54,"raw":"[Roy et al., 2021] Aurko Roy, Mohammad Saffar, et al. Ef- ficient content-based sparse attention with routing trans- formers. TACL, 2021.","title":"Efficient content-based sparse attention with routing transformers","authors":"Roy","year":"2021","venue":"TACL","doi":null,"url":null,"internal_url":null},{"position":55,"raw":"[Saragadam et al., 2022] Vishwanath Saragadam, Randall Balestriero, et al. DeepTensor: Low-rank tensor decompo- sition with deep network priors. arXiv:2204.03145, 2022.","title":"DeepTensor: Low-rank tensor decomposition with deep network priors","authors":"Vishwanath Saragadam, Randall Balestriero","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":56,"raw":"[Shaw et al., 2018] Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representa- tions. In NAACL, 2018.","title":"Self-attention with relative position representations","authors":"Shaw","year":"2018","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":57,"raw":"[Shazeer, 2019] Noam Shazeer. Fast transformer decoding: One write-head is all you need. arXiv:1911.02150, 2019.","title":"Noam Shazeer. Fast transformer decoding: One write-head is all you need","authors":"Shazeer","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":58,"raw":"[Shen et al., 2023] Kai Shen, J. Guo, et al. A study on ReLU and softmax in transformer. arXiv:2302.06461, 2023.","title":"A study on ReLU and softmax in transformer","authors":"Kai Shen, J Guo","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":59,"raw":"[Su et al., 2024] Jianlin Su, Murtadha Ahmed, et al. Ro- former: Enhanced transformer with rotary position embed- ding. Neurocomputing, 2024.","title":"Roformer: Enhanced transformer with rotary position embedding","authors":"Jianlin Su, Murtadha Ahmed","year":"2024","venue":"Neurocomputing","doi":null,"url":null,"internal_url":null},{"position":60,"raw":"[Sun et al., 2023a] Mingjie Sun, Zhuang Liu, et al. A simple and effective pruning approach for large language models. In Workshop on ES-FoMo @ ICML2023, 2023.","title":"A simple and effective pruning approach for large language models","authors":"Sun","year":"2023","venue":"Workshop on ES-FoMo @ ICML2023","doi":null,"url":null,"internal_url":null},{"position":61,"raw":"[Sun et al., 2023b] Yutao Sun, Li Dong, et al. A length- extrapolatable transformer. In ACL, 2023.","title":"A lengthextrapolatable transformer","authors":"Sun","year":"2023","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":62,"raw":"[Sun et al., 2023c] Yutao Sun, Li Dong, et al. Retentive Net- work: A successor to transformer for large language mod- els. arXiv:2307.08621, 2023.","title":"Retentive Network: A successor to transformer for large language models","authors":"Sun","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":63,"raw":"[Touvron et al., 2023] Hugo Touvron, Thibaut Lavril, et al. LLaMA: Open and efficient foundation language models. ArXiv, abs/2302.13971, 2023.","title":"Hugo Touvron, Thibaut Lavril, et al. LLaMA: Open and efficient foundation language models","authors":"Touvron","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":64,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, et al. Attention is all you need. In NeurIPS, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":65,"raw":"[Wan et al., 2023] Z. Wan, X. Wang, et al. Efficient large language models: A survey. arXiv:2312.03863, 2023.","title":"Efficient large language models: A survey","authors":"Z Wan, X Wang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":66,"raw":"[Wang et al., 2020] Sinong Wang, Belinda Z Li, et al. Linformer: Self-attention with linear complexity. arXiv:2006.04768, 2020.","title":"Linformer: Self-attention with linear complexity","authors":"Sinong Wang, Belinda Z Li","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":67,"raw":"[Wang et al., 2023] Xindi Wang, Yufei Wang, et al. Inves- tigating the learning behaviour of in-context learning: A comparison with supervised learning. ECAI, 2023.","title":"Investigating the learning behaviour of in-context learning: A comparison with supervised learning","authors":"Wang","year":"2023","venue":"ECAI","doi":null,"url":null,"internal_url":null},{"position":68,"raw":"[Winata et al., 2020] Genta Indra Winata, Samuel Cahyaw- ijaya, et al. Lightweight and efficient end-to-end speech recognition using low-rank transformer. In ICASSP, 2020.","title":"Lightweight and efficient end-to-end speech recognition using low-rank transformer","authors":"Winata","year":"2020","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":69,"raw":"[Wu et al., 2021] Haixu Wu, Jiehui Xu, et al. Auto- former: Decomposition transformers with auto-correlation for long-term series forecasting. In NeurIPS, 2021.","title":"Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting","authors":"Wu","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":70,"raw":"[Xia et al., 2024] Mengzhou Xia, Tianyu Gao, et al. Sheared LLaMA: Accelerating language model pre-training via structured pruning. In ICLR, 2024.","title":"Sheared LLaMA: Accelerating language model pre-training via structured pruning","authors":"Xia","year":"2024","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":71,"raw":"[Xiao et al., 2024] G. Xiao, Y. Tian, et al. Efficient streaming language models with attention sinks. In ICLR, 2024.","title":"Efficient streaming language models with attention sinks","authors":"G Xiao, Y Tian","year":"2024","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":72,"raw":"[Zhai et al., 2021] Shuangfei Zhai, Walter Talbott, et al. An attention free transformer. arXiv:2105.14103, 2021.","title":"An attention free transformer","authors":"Zhai","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":73,"raw":"[Zhao et al., 2023] Wayne Xin Zhao, Kun Zhou, et al. A sur- vey of large language models, 2023.","title":"A survey of large language models","authors":"Kun Wayne Xin Zhao, Zhou","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":74,"raw":"[Zhu et al., 2023] Xunyu Zhu, Jian Li, et al. A sur- vey on model compression for large language models. arXiv:2308.07633, 2023.","title":"A survey on model compression for large language models","authors":"Zhu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":75,"raw":"[Zhuang et al., 2023] Bohan Zhuang, Jing Liu, et al. A sur- vey on efficient training of transformers. In IJCAI, 2023.","title":"A survey on efficient training of transformers","authors":"Bohan Zhuang, Jing Liu","year":"2023","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2023/764"}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}