{"type":"ScholarlyArticle","title":"X-former Elucidator: Reviving Efficient Attention for Long Context Language Modeling","authors":[{"name":"Xupeng Miao","slug":"xupeng-miao","given":"Xupeng","family":"Miao","affiliation":"School of CS & Key Lab of High Confidence Software Technologies (MOE) , Peking University; Carnegie Mellon University"},{"name":"Shenhan Zhu","slug":"shenhan-zhu","given":"Shenhan","family":"Zhu","affiliation":"School of CS & Key Lab of High Confidence Software Technologies (MOE) , Peking University"},{"name":"Fangcheng Fu","slug":"fangcheng-fu","given":"Fangcheng","family":"Fu","affiliation":"School of CS & Key Lab of High Confidence Software Technologies (MOE) , Peking University"},{"name":"Ziyu Guo","slug":"ziyu-guo","given":"Ziyu","family":"Guo","affiliation":"The Chinese University of Hong Kong"},{"name":"Zhi Yang","slug":"zhi-yang","given":"Zhi","family":"Yang","affiliation":"School of CS & Key Lab of High Confidence Software Technologies (MOE) , Peking University"},{"name":"Yaofeng Tu","slug":"yaofeng-tu","given":"Yaofeng","family":"Tu","affiliation":"ZTE Corporation"},{"name":"Zhihao Jia","slug":"zhihao-jia","given":"Zhihao","family":"Jia","affiliation":"Carnegie Mellon University"},{"name":"Bin Cui","slug":"bin-cui","given":"Bin","family":"Cui","affiliation":"School of CS & Key Lab of High Confidence Software Technologies (MOE) , Peking University"}],"abstract":"Transformer-based LLMs are becoming increasingly important in various AI applications. However, apart from the success of LLMs, the explosive demand of long context handling capabilities is a key and in-time problem for both academia and industry.\r\nDue to the limitations from the quadratic complexity of the attention mechanism, long context scenarios require much more resources for LLM development and deployment, bringing huge challenges to the underlying AI infrastructure.\r\nMeanwhile, we observe that there is a trend of reviving previous efficient attention mechanisms to latest LLMs.\r\nHowever, it still remains an open question about how to select from these diverse approaches in practice.\r\nIn this paper, we answer this question from several aspects.\r\nFirst, we revisit these latest long-context LLM innovations and discuss their relationship with prior approaches with a novel and comprehensive taxonomy.\r\nNext, we conduct a thorough evaluation over various types of workloads considering both efficiency and effectiveness.\r\nFinally, we provide an in-depth analysis, summarize our key findings, and offer insightful suggestions on the trade-offs of designing and deploying efficient attention mechanisms for Transformer-based LLMs.","doi":"10.24963/ijcai.2024/904","url":"https://ijcai.org/proceedings/2024/904","pdf":"https://ijcai.org/proceedings/2024/0904.pdf","markdown":"https://ijcai.org/proceedings/2024/904.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/904","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Survey Track","track":"Survey Track","number":904,"pages":{"first":8179,"last":8187},"keywords":["Natural Language Processing: NLP: Language models","Machine Learning: ML: Attention models","Multidisciplinary Topics and Applications: MTA: Computational sustainability","Natural Language Processing: General"],"videos":[],"references":[{"position":1,"raw":"[Ainslie et al., 2023] Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi- query transformer models from multi-head checkpoints. In EMNLP, 2023.","title":"Gqa: Training generalized multiquery transformer models from multi-head checkpoints","authors":"Joshua Ainslie, James Lee-Thorp, Michiel De Jong, Yury Zemlyanskiy, Federico Lebrón, Sumit Sanghai","year":"2023","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Anagnostidis et al., 2023] Sotiris Anagnostidis, Dario Pavllo, Luca Biggio, Lorenzo Noci, Aurelien Lucchi, and Thomas Hoffmann. Dynamic context pruning for efficient and interpretable autoregressive transformers. In NeurIPS, 2023.","title":"Dynamic context pruning for efficient and interpretable autoregressive transformers","authors":"Sotiris Anagnostidis, Dario Pavllo, Luca Biggio, Lorenzo Noci, Aurelien Lucchi, Thomas Hoffmann","year":"2023","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Beltagy et al., 2020] Iz Beltagy, Matthew E Peters, and Ar- man Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020.","title":"Iz Beltagy, Matthew E Peters, and Arman Cohan","authors":"Beltagy","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chevalier et al., 2023] Alexis Chevalier, Alexander Wettig, Anirudh Ajith, and Danqi Chen. Adapting language models to compress contexts. In EMNLP, 2023.","title":"Adapting language models to compress contexts","authors":"Alexis Chevalier, Alexander Wettig, Anirudh Ajith, Danqi Chen","year":"2023","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Child et al., 2019] Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019.","title":"Rewon Child","authors":"Child","year":"2019","venue":"Generating long sequences with sparse transformers","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Choromanski et al., 2020] Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. arXiv preprint arXiv:2009.14794, 2020.","title":"Rethinking attention with performers","authors":"Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Clevert et al., 2015] Djork-Arné Clevert, Thomas Un- terthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv:1511.07289, 2015.","title":"Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units","authors":"Clevert","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Correia et al., 2019] Gonçalo M Correia, Vlad Niculae, and André FT Martins. Adaptively sparse transformers. In EMNLP-IJCNLP, pages 2174-2184, 2019.","title":"Adaptively sparse transformers","authors":"M Gonçalo, Vlad Correia, André Ft Niculae, Martins","year":"2019","venue":"EMNLP-IJCNLP","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Ding et al., 2023] Jiayu Ding, Shuming Ma, Li Dong, Xingx- ing Zhang, Shaohan Huang, Wenhui Wang, and Furu Wei. Longnet: Scaling transformers to 1,000,000,000 tokens. arXiv:2307.02486, 2023.","title":"Longnet: Scaling transformers to 1","authors":"Jiayu Ding, Shuming Ma, Li Dong, Xingxing Zhang, Shaohan Huang, Wenhui Wang, Furu Wei","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ge et al., 2023] Tao Ge, Jing Hu, Xun Wang, Si-Qing Chen, and Furu Wei. In-context autoencoder for context com- pression in a large language model. arXiv preprint arXiv:2307.06945, 2023.","title":"In-context autoencoder for context compression in a large language model","authors":"Tao Ge, Jing Hu, Xun Wang, Si-Qing Chen, Furu Wei","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Gray et al., 2017] Scott Gray, Alec Radford, and Diederik P Kingma. Gpu kernels for block-sparse weights. arXiv preprint arXiv:1711.09224, 3:2, 2017.","title":"Gpu kernels for block-sparse weights","authors":"Scott Gray, Alec Radford, Diederik P Kingma","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Gu and Dao, 2023] Albert Gu and Tri Dao. Mamba: Linear- time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023.","title":"Mamba: Lineartime sequence modeling with selective state spaces","authors":"Dao Gu, Albert Gu, Tri Dao","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Gupta et al., 2021] Ankit Gupta, Guy Dar, Shaya Goodman, David Ciprut, and Jonathan Berant. Memory-efficient trans- formers via top-k attention. In SustaiNLP@EMNLP, 2021.","title":"Memory-efficient transformers via top-k attention","authors":"Ankit Gupta, Guy Dar, Shaya Goodman, David Ciprut, Jonathan Berant","year":"2021","venue":"SustaiNLP@EMNLP","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Han et al., 2023] Insu Han, Rajesh Jayaram, Amin Karbasi, Vahab Mirrokni, David P Woodruff, and Amir Zandieh. Hyperattention: Long-context attention in near-linear time. arXiv preprint arXiv:2310.05869, 2023.","title":"Hyperattention: Long-context attention in near-linear time","authors":"Insu Han, Rajesh Jayaram, Amin Karbasi, Vahab Mirrokni, David P Woodruff, Amir Zandieh","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Jiang et al., 2023a] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.","title":"Mistral 7b","authors":"Jiang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Jiang et al., 2023b] Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. Llmlingua: Compressing prompts for accelerated inference of large language models. arXiv preprint arXiv:2310.05736, 2023.","title":"Llmlingua: Compressing prompts for accelerated inference of large language models","authors":"Jiang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Jiang et al., 2023c] Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. Longllmlingua: Accelerating and enhancing llms in long context scenarios via prompt compression. arXiv preprint arXiv:2310.06839, 2023.","title":"Longllmlingua: Accelerating and enhancing llms in long context scenarios via prompt compression","authors":"Jiang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Katharopoulos et al., 2020] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In ICML, 2020.","title":"Transformers are rnns: Fast autoregressive transformers with linear attention","authors":"Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, François Fleuret","year":"2020","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Kitaev et al., 2019] Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In ICLR, 2019.","title":"Reformer: The efficient transformer","authors":"Nikita Kitaev, Lukasz Kaiser, Anselm Levskaya","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Li et al., 2023a] Longhai Li, Lei Duan, Junchen Wang, Chengxin He, Zihao Chen, Guicai Xie, Song Deng, and Zhaohang Luo. Memory-enhanced transformer for repre- sentation learning on temporal heterogeneous graphs. Data Sci. Eng., 8(2):98-111, 2023.","title":"Memory-enhanced transformer for representation learning on temporal heterogeneous graphs","authors":"Li","year":"2023","venue":"Data Sci. Eng","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Li et al., 2023b] Yucheng Li, Bo Dong, Frank Guerin, and Chenghua Lin. Compressing context to enhance inference efficiency of large language models. In EMNLP, 2023.","title":"Compressing context to enhance inference efficiency of large language models","authors":"Li","year":"2023","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Lingle, 2023] Lucas D Lingle. Transformer-vq: Linear- time transformers via vector quantization. arXiv preprint arXiv:2309.16354, 2023.","title":"Transformer-vq: Lineartime transformers via vector quantization","authors":"Lucas D Lingle","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Liu et al., 2023a] Yuhan Liu, Hanchen Li, Kuntai Du, Ji- ayi Yao, Yihua Cheng, Yuyang Huang, Shan Lu, Michael Maire, Henry Hoffmann, Ari Holtzman, Ganesh Anantha- narayanan, and Junchen Jiang. Cachegen: Fast context loading for language model applications. arXiv preprint arXiv:2310.07240, 2023.","title":"Cachegen: Fast context loading for language model applications","authors":"Liu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Liu et al., 2023b] Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anasta- sios Kyrillidis, and Anshumali Shrivastava. Scissorhands: Exploiting the persistence of importance hypothesis for llm kv cache compression at test time. In NeurIPS, 2023.","title":"Scissorhands: Exploiting the persistence of importance hypothesis for llm kv cache compression at test time","authors":"Liu","year":"2023","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Liu et al., 2023c] Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Ré, and Beidi Chen. Deja vu: Contextual sparsity for efficient llms at inference time. In ICML, pages 22137-22176. PMLR, 2023.","title":"Deja vu: Contextual sparsity for efficient llms at inference time","authors":"Liu","year":"2023","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Miao et al., 2023a] Xupeng Miao, Xiaonan Nie, Hailin Zhang, Tong Zhao, and Bin Cui. Hetu: A highly effi- cient automatic parallel distributed deep learning system. Sci. China Inf. Sci., 66(1):117101, 2023.","title":"Hetu: A highly efficient automatic parallel distributed deep learning system","authors":"Miao","year":"2023","venue":"Sci. China Inf. Sci","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Miao et al., 2023b] Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Hongyi Jin, Tianqi Chen, and Zhi- hao Jia. Towards efficient generative large language model serving: A survey from algorithms to systems. arXiv preprint arXiv:2312.15234, 2023.","title":"Towards efficient generative large language model serving: A survey from algorithms to systems","authors":"Miao","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Mohtashami and Jaggi, 2023] Amirkeivan Mohtashami and Martin Jaggi. Landmark attention: Random-access in- finite context length for transformers. arXiv preprint arXiv:2305.16300, 2023.","title":"Amirkeivan Mohtashami and Martin Jaggi. Landmark attention: Random-access in-finite context length for transformers","authors":"Jaggi Mohtashami","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Mu et al., 2023] Jesse Mu, Xiang Lisa Li, and Noah Good- man. Learning to compress prompts with gist tokens. arXiv preprint arXiv:2304.08467, 2023.","title":"Learning to compress prompts with gist tokens","authors":"Jesse Mu, Xiang, Lisa Li, Noah Goodman","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[mya, 2023] Technical Report. https://github.com/Hsword/ X-former-Elucidator/blob/main/Report.pdf, 2023.","title":"mya","authors":null,"year":"2023","venue":null,"doi":null,"url":"https://github.com/Hsword/X-former-Elucidator/blob/main/Report.pdf","internal_url":null},{"position":31,"raw":"[Pagliardini et al., 2023] Matteo Pagliardini, Daniele Paliotta, Martin Jaggi, and François Fleuret. Faster causal atten- tion over large sequences through sparse flash attention. arXiv:2306.01160, 2023.","title":"Faster causal attention over large sequences through sparse flash attention","authors":"Matteo Pagliardini, Daniele Paliotta, Martin Jaggi, François Fleuret","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Peng et al., 2021] Hao Peng, Nikolaos Pappas, Dani Yo- gatama, Roy Schwartz, Noah A. Smith, and Lingpeng Kong. Random feature attention. In ICLR, 2021.","title":"Random feature attention","authors":"Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah A Smith, Lingpeng Kong","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Peng et al., 2023] Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, et al. Rwkv: Reinventing rnns for the transformer era. In EMNLP (Findings), 2023.","title":"Rwkv: Reinventing rnns for the transformer era","authors":"Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho","year":"2023","venue":"EMNLP (Findings)","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Qin et al., 2022] Zhen Qin, XiaoDong Han, Weixuan Sun, Dongxu Li, Lingpeng Kong, Nick Barnes, and Yiran Zhong. The devil in linear transformer. In EMNLP, 2022.","title":"The devil in linear transformer","authors":"Zhen Qin, Xiaodong Han, Weixuan Sun, Dongxu Li, Lingpeng Kong, Nick Barnes, Yiran Zhong","year":"2022","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Roy et al., 2021] Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. Efficient content-based sparse attention with routing transformers. In TACL, 2021.","title":"Efficient content-based sparse attention with routing transformers","authors":"Roy","year":"2021","venue":"TACL","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Shao et al., 2024] Yunfan Shao, Zhichao Geng, Yitao Liu, Junqi Dai, Hang Yan, Fei Yang, Li Zhe, Hujun Bao, and Xipeng Qiu. Cpt: A pre-trained unbalanced transformer for both chinese language understanding and generation. Sci. China Inf. Sci., 67(5):1-13, 2024.","title":"Cpt: A pre-trained unbalanced transformer for both chinese language understanding and generation","authors":"Yunfan Shao, Zhichao Geng, Yitao Liu, Junqi Dai, Hang Yan, Fei Yang, Li Zhe, Hujun Bao, Xipeng Qiu","year":"2024","venue":"Sci. China Inf. Sci","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Su et al., 2024] Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomput- ing, 2024.","title":"Roformer: Enhanced transformer with rotary position embedding","authors":"Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, Yunfeng Liu","year":"2024","venue":null,"doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Tay et al., 2020] Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: A survey. ACM Computing Surveys (CSUR), 2020.","title":"Efficient transformers: A survey","authors":"Yi Tay, Mostafa Dehghani, Dara Bahri, Donald Metzler","year":"2020","venue":"ACM Computing Surveys (CSUR)","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Tay et al., 2021a] Yi Tay, Dara Bahri, Donald Metzler, Da- Cheng Juan, Zhe Zhao, and Che Zheng. Synthesizer: Re- thinking self-attention for transformer models. In ICML, pages 10183-10192. PMLR, 2021.","title":"Synthesizer: Rethinking self-attention for transformer models","authors":"Tay","year":"2021","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Tay et al., 2021b] Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena : A benchmark for efficient transformers. In ICLR, 2021.","title":"Long range arena : A benchmark for efficient transformers","authors":"Tay","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Tsai et al., 2019] Yao-Hung Hubert Tsai, Shaojie Bai, Makoto Yamada, Louis-Philippe Morency, and Ruslan Salakhutdinov. Transformer dissection: An unified under- standing for transformer's attention via the lens of kernel. In EMNLP-IJCNLP, pages 4344-4353, 2019.","title":"Transformer dissection: An unified understanding for transformer's attention via the lens of kernel","authors":"Yao-Hung Hubert Tsai, Shaojie Bai, Makoto Yamada, Louis-Philippe Morency, Ruslan Salakhutdinov","year":"2019","venue":"EMNLP-IJCNLP","doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS 2017, pages 5998-6008, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, Illia Polosukhin","year":"2017","venue":"NeurIPS 2017","doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Vyas et al., 2020] Apoorv Vyas, Angelos Katharopoulos, and François Fleuret. Fast transformers with clustered attention. In NeurIPS, 2020.","title":"Fast transformers with clustered attention","authors":"Apoorv Vyas, Angelos Katharopoulos, François Fleuret","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":44,"raw":"[Wang and Zhang, 2013] Shusen Wang and Zhihua Zhang. Improving cur matrix decomposition and the nyström ap- proximation via adaptive sampling. JMLR, 2013.","title":"Improving cur matrix decomposition and the nyström approximation via adaptive sampling","authors":"Wang, Zhang, Shusen Wang, Zhihua Zhang","year":"2013","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":45,"raw":"[Wang et al., 2020] Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768, 2020.","title":"Linformer: Self-attention with linear complexity","authors":"Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, Hao Ma","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":46,"raw":"[Xiao et al., 2024] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In ICLR, 2024.","title":"Efficient streaming language models with attention sinks","authors":"Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, Mike Lewis","year":"2024","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":47,"raw":"[Xiong et al., 2021] Yunyang Xiong, Zhanpeng Zeng, Rudra- sis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, and Vikas Singh. Nyströmformer: A nyström-based algorithm for approximating self-attention. In AAAI, 2021.","title":"Nyströmformer: A nyström-based algorithm for approximating self-attention","authors":"Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, Vikas Singh","year":"2021","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":48,"raw":"[Xu et al., 2023] Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Sub- ramanian, Evelina Bakhturina, Mohammad Shoeybi, and Bryan Catanzaro. Retrieval meets long context large lan- guage models. arXiv preprint arXiv:2310.03025, 2023.","title":"Retrieval meets long context large language models","authors":"Peng Xu, Wei Ping, Xianchao Wu, Lawrence Mcafee, Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammad Shoeybi, Bryan Catanzaro","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":49,"raw":"[Ye et al., 2023] Zihao Ye, Ruihang Lai, Junru Shao, Tianqi Chen, and Luis Ceze. Sparsetir: Composable abstractions for sparse compilation in deep learning. In ASPLOS, 2023.","title":"Sparsetir: Composable abstractions for sparse compilation in deep learning","authors":"Ye","year":"2023","venue":"ASPLOS","doi":null,"url":null,"internal_url":null},{"position":50,"raw":"[Zaheer et al., 2020] Manzil Zaheer, Guru Guruganesh, Ku- mar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santi- ago Ontañón, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. Big bird: Transformers for longer sequences. NeurIPS, 33:17283-17297, 2020.","title":"Big bird: Transformers for longer sequences","authors":"Manzil Zaheer, Guru Guruganesh, Avinava Kumar, Joshua Dubey, Chris Ainslie, Santiago Alberti, Philip Ontañón, Anirudh Pham, Qifan Ravula, Li Wang, Amr Yang, Ahmed","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":51,"raw":"[Zhang et al., 2023a] Qingru Zhang, Dhananjay Ram, Cole Hawkins, Sheng Zha, and Tuo Zhao. Efficient long-range transformers: You need to attend more, but not necessarily at every layer. In EMNLP, 2023.","title":"Efficient long-range transformers: You need to attend more, but not necessarily at every layer","authors":"Zhang","year":"2023","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":52,"raw":"[Zhang et al., 2023b] Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark W. Barrett, Zhangyang Wang, and Beidi Chen. H _2 o: Heavy-hitter oracle for efficient generative inference of large language models. In NeurIPS, 2023.","title":"H _2 o: Heavy-hitter oracle for efficient generative inference of large language models","authors":"Zhang","year":"2023","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":53,"raw":"[Zhen et al., 2022] Q Zhen, W Sun, H Deng, D Li, Y Wei, B Lv, J Yan, L Kong, and Y Zhong. cosformer: Rethinking softmax in attention. In ICLR, 2022.","title":"cosformer: Rethinking softmax in attention","authors":"Zhen, H Sun, D Deng, Y Li, B Wei, J Lv, L Yan, Y Kong, Zhong","year":"2022","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":54,"raw":"[Zheng et al., 2023] Lin Zheng, Jianbo Yuan, Chong Wang, and Lingpeng Kong. Efficient attention via control variates. arXiv preprint arXiv:2302.04542, 2023.","title":"Efficient attention via control variates","authors":"Lin Zheng, Jianbo Yuan, Chong Wang, Lingpeng Kong","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":55,"raw":"[Zhong et al., 2022] Haiting Zhong, Wei He, Lizhen Cui, Lei Liu, Zhongmin Yan, and Kun Zhao. Joint atten- tion networks with inherent and contextual preference- awareness for successive poi recommendation. Data Sci. Eng., 7(4):370-382, 2022.","title":"Joint attention networks with inherent and contextual preferenceawareness for successive poi recommendation","authors":"Haiting Zhong, Wei He, Lizhen Cui, Lei Liu, Zhongmin Yan, Kun Zhao","year":"2022","venue":"Data Sci. Eng","doi":null,"url":null,"internal_url":null},{"position":56,"raw":"[Zhou et al., 2021] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In AAAI, 2021.","title":"Informer: Beyond efficient transformer for long sequence time-series forecasting","authors":"Zhou","year":"2021","venue":"AAAI","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}