{"type":"ScholarlyArticle","title":"LongVQ: Long Sequence Modeling with Vector Quantization on Structured Memory","authors":[{"name":"Zicheng Liu","slug":"zicheng-liu","given":"Zicheng","family":"Liu","affiliation":"Zhejiang University , Hangzhou , China; AI Lab , Research Center for Industries of the Future , Westlake University , Hangzhou , China"},{"name":"Li Wang","slug":"li-wang","given":"Li","family":"Wang","affiliation":"AI Lab , Research Center for Industries of the Future , Westlake University , Hangzhou , China"},{"name":"Siyuan Li","slug":"siyuan-li","given":"Siyuan","family":"Li","affiliation":"Zhejiang University , Hangzhou , China; AI Lab , Research Center for Industries of the Future , Westlake University , Hangzhou , China"},{"name":"Zedong Wang","slug":"zedong-wang","given":"Zedong","family":"Wang","affiliation":"AI Lab , Research Center for Industries of the Future , Westlake University , Hangzhou , China"},{"name":"Haitao Lin","slug":"haitao-lin","given":"Haitao","family":"Lin","affiliation":"Zhejiang University , Hangzhou , China; AI Lab , Research Center for Industries of the Future , Westlake University , Hangzhou , China"},{"name":"Stan Z. Li","slug":"stan-z-li","given":"Stan Z.","family":"Li","affiliation":"AI Lab , Research Center for Industries of the Future , Westlake University , Hangzhou , China"}],"abstract":"Transformer models have been successful in various sequence processing tasks, but the self-attention mechanism's computational cost limits its practicality for long sequences. Although there are existing attention variants that improve computational efficiency, they have a limited ability to abstract global information effectively based on their hand-crafted mixing strategies. On the other hand, state-space models (SSMs) are tailored for long sequences but cannot capture complicated local information. Therefore, the combination of them as a unified token mixer is a trend in recent long-sequence models. However, the linearized attention degrades performance significantly even when equipped with SSMs. To address the issue, we propose a new method called LongVQ. LongVQ uses the vector quantization (VQ) technique to compress the global abstraction as a length-fixed codebook, enabling the linear-time computation of the attention matrix. This technique effectively maintains dynamic global and local patterns, which helps to complement the lack of long-range dependency issues. Our experiments on the Long Range Arena benchmark, autoregressive language modeling, and image and speech classification demonstrate the effectiveness of LongVQ. Our model achieves significant improvements over other sequence models, including variants of Transformers, Convolutions, and recent State Space Models.","doi":"10.24963/ijcai.2024/510","url":"https://ijcai.org/proceedings/2024/510","pdf":"https://ijcai.org/proceedings/2024/0510.pdf","markdown":"https://ijcai.org/proceedings/2024/510.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/510","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Machine Learning","track":"Main Track","number":510,"pages":{"first":4614,"last":4622},"keywords":["Machine Learning: ML: Deep learning architectures","Machine Learning: ML: Representation learning","Natural Language Processing: NLP: Embeddings"],"videos":[],"references":[{"position":1,"raw":"[Ainslie et al., 2020] Joshua Ainslie, Santiago Ontanon, Chris Alberti, Vaclav Cvicek, Zachary Fisher, Philip Pham, Anirudh Ravula, Sumit Sanghai, Qifan Wang, and Li Yang. ETC: Encoding long and structured inputs in transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 268-284, Online, 2020. Association for Computational Linguistics.","title":"ETC: Encoding long and structured inputs in transformers","authors":"Joshua Ainslie, Santiago Ontanon, Chris Alberti, Vaclav Cvicek, Zachary Fisher, Philip Pham, Anirudh Ravula, Sumit Sanghai, Qifan Wang, Li Yang","year":"2020","venue":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Baevski and Auli, 2019] Alexei Baevski and Michael Auli. Adaptive input representations for neural language model- ing, 2019.","title":"Alexei Baevski and Michael Auli. Adaptive input representations for neural language modeling","authors":"Auli Baevski","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bai et al., 2018] Shaojie Bai, J Zico Kolter, and Vladlen Koltun. Trellis networks for sequence modeling. arXiv preprint arXiv:1810.06682, 2018.","title":"Trellis networks for sequence modeling","authors":"Bai","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Beltagy et al., 2020] Iz Beltagy, Matthew E Peters, and Ar- man Cohan. Longformer: The long-document trans- former. arXiv preprint arXiv:2004.05150, 2020.","title":"Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer","authors":"Beltagy","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Child et al., 2020] R Child, S Gray, A Radford, and I Sutskever. Generating long sequences with sparse trans- formers. arXiv preprint arXiv:1904.10509, 2020.","title":"Generating long sequences with sparse transformers","authors":"Child, Gray, Radford, Sutskever","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Choromanski et al., 2020] Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. arXiv preprint arXiv:2009.14794, 2020.","title":"Rethinking attention with performers","authors":"Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chung et al., 2014] Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evalua- tion of gated recurrent neural networks on sequence mod- eling. arXiv preprint arXiv:1412.3555, 2014.","title":"Empirical evaluation of gated recurrent neural networks on sequence modeling","authors":"Chung","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dosovitskiy et al., 2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Min- derer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Trans- formers for image recognition at scale, 2021.","title":"An image is worth 16x16 words: Transformers for image recognition at scale","authors":"Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Erichson et al., 2020] N Benjamin Erichson, Omri Azencot, Alejandro Queiruga, Liam Hodgkinson, and Michael W Mahoney. Lipschitz recurrent neural networks. arXiv preprint arXiv:2006.12070, 2020.","title":"Lipschitz recurrent neural networks","authors":"Omri Benjamin Erichson, Alejandro Azencot, Liam Queiruga, Michael W Hodgkinson, Mahoney","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Fu et al., 2023a] Daniel Y. Fu, Tri Dao, Khaled K. Saab, Armin W. Thomas, Atri Rudra, and Christopher Ré. Hun- gry hungry hippos: Towards language modeling with state space models, 2023.","title":"Hungry hungry hippos: Towards language modeling with state space models","authors":"Fu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Fu et al., 2023b] Daniel Y. Fu, Elliot L. Epstein, Eric Nguyen, Armin W. Thomas, Michael Zhang, Tri Dao, Atri Rudra, and Christopher Ré. Simple hardware-efficient long convolutions for sequence modeling, 2023.","title":"Simple hardware-efficient long convolutions for sequence modeling","authors":"Fu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Gu et al., 2020a] Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré. Hippo: Recurrent memory with optimal polynomial projections. Advances in neural information processing systems, 33:1474-1487, 2020.","title":"Hippo: Recurrent memory with optimal polynomial projections","authors":"Gu","year":"2020","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Gu et al., 2020b] Albert Gu, Caglar Gulcehre, Thomas Paine, Matt Hoffman, and Razvan Pascanu. Improving the gating mechanism of recurrent neural networks. In Inter- national Conference on Machine Learning, pages 3800- 3809. PMLR, 2020.","title":"Improving the gating mechanism of recurrent neural networks","authors":"Gu","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Gu et al., 2021] Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. ArXiv preprint, abs/2111.00396, 2021.","title":"Efficiently modeling long sequences with structured state spaces","authors":"Albert Gu, Karan Goel, Christopher Ré","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Gu et al., 2022] Albert Gu, Karan Goel, Ankit Gupta, and Christopher Ré. On the parameterization and initialization of diagonal state space models. Advances in Neural Infor- mation Processing Systems, 35:35971-35983, 2022.","title":"On the parameterization and initialization of diagonal state space models","authors":"Albert Gu, Karan Goel, Ankit Gupta, Christopher Ré","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Gupta et al., 2022] Ankit Gupta, Albert Gu, and Jonathan Berant. Diagonal state spaces are as effective as structured state spaces. Advances in Neural Information Processing Systems, 35:22982-22994, 2022.","title":"Diagonal state spaces are as effective as structured state spaces","authors":"Ankit Gupta, Albert Gu, Jonathan Berant","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Hasani et al., 2022] Ramin Hasani, Mathias Lechner, Tsun- Hsuan Wang, Makram Chahine, Alexander Amini, and Daniela Rus. Liquid structural state-space models. arXiv preprint arXiv:2209.12951, 2022.","title":"Liquid structural state-space models","authors":"Ramin Hasani, Mathias Lechner, Tsun-Hsuan Wang, Makram Chahine, Alexander Amini, Daniela Rus","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735-1780, 1997.","title":"Long short-term memory","authors":"Schmidhuber Hochreiter, Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Hua et al., 2022a] Weizhe Hua, Zihang Dai, Hanxiao Liu, and Quoc Le. Transformer quality in linear time. In Inter- national Conference on Machine Learning, pages 9099- 9117. PMLR, 2022.","title":"Transformer quality in linear time","authors":"Hua","year":"2022","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Hua et al., 2022b] Weizhe Hua, Zihang Dai, Hanxiao Liu, and Quoc V. Le. Transformer quality in linear time, 2022.","title":"Transformer quality in linear time","authors":"Hua","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Jaegle et al., 2021] Andrew Jaegle, Felix Gimeno, Andrew Brock, Andrew Zisserman, Oriol Vinyals, and Joao Car- reira. Perceiver: General perception with iterative atten- tion, 2021.","title":"Perceiver: General perception with iterative attention","authors":"Andrew Jaegle, Felix Gimeno, Andrew Brock, Andrew Zisserman, Oriol Vinyals, Joao Carreira","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Katharopoulos et al., 2020] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and Franc ¸ois Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In International conference on machine learning, pages 5156-5165. PMLR, 2020.","title":"Transformers are rnns: Fast autoregressive transformers with linear attention","authors":"Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, Franc ¸ois, Fleuret","year":"2020","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Kitaev et al., 2020] Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. arXiv preprint arXiv:2001.04451, 2020.","title":"Reformer: The efficient transformer","authors":"Nikita Kitaev, Łukasz Kaiser, Anselm Levskaya","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Krizhevsky et al., 2009] Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 (canadian institute for advanced research). 2009.","title":"Cifar-10 (canadian institute for advanced research","authors":"Alex Krizhevsky, Vinod Nair, Geoffrey Hinton","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Li et al., 2022] Yuhong Li, Tianle Cai, Yi Zhang, Deming Chen, and Debadeepta Dey. What makes convolutional models great on long sequence modeling? arXiv preprint arXiv:2210.09298, 2022.","title":"What makes convolutional models great on long sequence modeling","authors":"Yuhong Li, Tianle Cai, Yi Zhang, Deming Chen, Debadeepta Dey","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Lingle, 2023] Lucas D. Lingle. Transformer-vq: Linear- time transformers via vector quantization, 2023.","title":"Transformer-vq: Lineartime transformers via vector quantization","authors":"Lucas D Lingle","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Liu et al., 2022] Zicheng Liu, Siyuan Li, Di Wu, Zihan Liu, Zhiyuan Chen, Lirong Wu, and Stan Z. Li. Automix: Un- veiling the power of mixup for stronger classifiers, 2022.","title":"Automix: Unveiling the power of mixup for stronger classifiers","authors":"Zicheng Liu, Siyuan Li, Di Wu, Zihan Liu, Zhiyuan Chen, Lirong Wu, Stan Z Li","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Ma et al., 2021] Xuezhe Ma, Xiang Kong, Sinong Wang, Chunting Zhou, Jonathan May, Hao Ma, and Luke Zettle- moyer. Luna: Linear unified nested attention. Advances in Neural Information Processing Systems, 34:2441-2453, 2021.","title":"Luna: Linear unified nested attention","authors":"Ma","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Ma et al., 2022] Xuezhe Ma, Chunting Zhou, Xiang Kong, Junxian He, Liangke Gui, Graham Neubig, Jonathan May, and Luke Zettlemoyer. Mega: Moving average equipped gated attention. ArXiv preprint, abs/2209.10655, 2022.","title":"Mega: Moving average equipped gated attention","authors":"Ma","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Mehta et al., 2022] Harsh Mehta, Ankit Gupta, Ashok Cutkosky, and Behnam Neyshabur. Long range language modeling via gated state spaces, 2022.","title":"Long range language modeling via gated state spaces","authors":"Harsh Mehta, Ankit Gupta, Ashok Cutkosky, Behnam Neyshabur","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Merity et al., 2016] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016.","title":"Pointer sentinel mixture models","authors":"Stephen Merity, Caiming Xiong, James Bradbury, Richard Socher","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Pascual et al., 2019] Santiago Pascual, Mirco Ravanelli, Joan Serrà, Antonio Bonafonte, and Yoshua Bengio. Learning problem-agnostic speech representations from multiple self-supervised tasks, 2019.","title":"Learning problem-agnostic speech representations from multiple self-supervised tasks","authors":"Pascual","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Peng et al., 2023] Bo Peng, Eric Alcaide, Quentin An- thony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, Kranthi Kiran GV, Xuzheng He, Haowen Hou, Jiaju Lin, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Bart- lomiej Koptyra, Hayden Lau, Krishna Sri Ipsit Mantri, Ferdinand Mom, Atsushi Saito, Guangyu Song, Xiangru Tang, Bolun Wang, Johan S. Wind, Stanislaw Wozniak, Ruichong Zhang, Zhenyuan Zhang, Qihang Zhao, Peng Zhou, Qinghua Zhou, Jian Zhu, and Rui-Jie Zhu. Rwkv: Reinventing rnns for the transformer era, 2023.","title":"Rwkv: Reinventing rnns for the transformer era","authors":"Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, Kranthi Kiran, G V, Xuzheng He, Haowen Hou, Jiaju Lin, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Bartlomiej Koptyra, Hayden Lau, Krishna Sri, Ipsit Mantri, Ferdinand Mom, Atsushi Saito, Guangyu Song, Xiangru Tang, Bolun Wang, Johan S Wind, Stanislaw Wozniak, Ruichong Zhang, Zhenyuan Zhang, Qihang Zhao, Peng Zhou, Qinghua Zhou, Jian Zhu, Rui-Jie Zhu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Poli et al., 2023] Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y. Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré. Hyena hi- erarchy: Towards larger convolutional language models, 2023.","title":"Hyena hierarchy: Towards larger convolutional language models","authors":"Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, Christopher Ré","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Rae et al., 2019] Jack W. Rae, Anna Potapenko, Sid- dhant M. Jayakumar, and Timothy P. Lillicrap. Com- pressive transformers for long-range sequence modelling, 2019.","title":"Compressive transformers for long-range sequence modelling","authors":"Jack W Rae, Anna Potapenko, M Siddhant, Timothy P Jayakumar, Lillicrap","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Ramachandran et al., 2017] Prajit Ramachandran, Barret Zoph, and Quoc V. Le. Searching for activation functions, 2017.","title":"Searching for activation functions","authors":"Ramachandran","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Ren et al., 2021] Hongyu Ren, Hanjun Dai, Zihang Dai, Mengjiao Yang, Jure Leskovec, Dale Schuurmans, and Bo Dai. Combiner: Full attention transformer with sparse computation cost, 2021.","title":"Combiner: Full attention transformer with sparse computation cost","authors":"Hanjun Hongyu Ren, Zihang Dai, Mengjiao Dai, Jure Yang, Dale Leskovec, Bo Schuurmans, Dai","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Romero et al., 2021] David W Romero, Robert-Jan Bruin- tjes, Jakub M Tomczak, Erik J Bekkers, Mark Hoogen- doorn, and Jan C van Gemert. Flexconv: Continuous ker- nel convolutions with differentiable kernel sizes. arXiv preprint arXiv:2110.08059, 2021.","title":"Flexconv: Continuous kernel convolutions with differentiable kernel sizes","authors":"Robert-Jan David W Romero, Bruintjes, M Jakub, Erik J Tomczak, Mark Bekkers, Jan C Hoogendoorn, Van Gemert","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Roy et al., 2020] Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. Efficient content-based sparse attention with routing transformers, 2020.","title":"Efficient content-based sparse attention with routing transformers","authors":"Roy","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Smith et al., 2023] Jimmy TH Smith, Andrew Warrington, and Scott W Linderman. Simplified state space layers for sequence modeling. International Conference on Learning Representations, 2023.","title":"Simplified state space layers for sequence modeling","authors":"Smith","year":"2023","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Tay et al., 2020a] Yi Tay, Dara Bahri, Liu Yang, Donald Metzler, and Da-Cheng Juan. Sparse sinkhorn attention. In International Conference on Machine Learning. PMLR, 2020.","title":"Sparse sinkhorn attention","authors":"Tay","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Tay et al., 2020b] Yi Tay, Mostafa Dehghani, Samira Ab- nar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena: A benchmark for efficient transformers. arXiv preprint arXiv:2011.04006, 2020.","title":"Long range arena: A benchmark for efficient transformers","authors":"Tay","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Trinh et al., 2018] Trieu Trinh, Andrew Dai, Thang Luong, and Quoc Le. Learning longer-term dependencies in rnns with auxiliary losses. In International Conference on Ma- chine Learning, pages 4965-4974. PMLR, 2018.","title":"Learning longer-term dependencies in rnns with auxiliary losses","authors":"Trieu Trinh, Andrew Dai, Thang Luong, Quoc Le","year":"2018","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":44,"raw":"[van den Oord et al., 2018] Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete repre- sentation learning, 2018.","title":"Neural discrete representation learning","authors":"Oord Van Den","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":45,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":46,"raw":"[Vyas et al., 2020] Apoorv Vyas, Angelos Katharopoulos, and Franc ¸ois Fleuret. Fast transformers with clustered at- tention. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, edi- tors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Process- ing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.","title":"Fast transformers with clustered attention","authors":"Apoorv Vyas, Angelos Katharopoulos, Franc ¸ois, Fleuret","year":"2020","venue":"Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020","doi":null,"url":null,"internal_url":null},{"position":47,"raw":"[Wang et al., 2020] Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768, 2020.","title":"Linformer: Self-attention with linear complexity","authors":"Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, Hao Ma","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":48,"raw":"[Warden, 2018] Pete Warden. Speech commands: A dataset for limited-vocabulary speech recognition, 2018.","title":"Pete Warden. Speech commands: A dataset for limited-vocabulary speech recognition","authors":"Warden","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":49,"raw":"[Zaheer et al., 2020] Manzil Zaheer, Guru Guruganesh, Ku- mar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santi- ago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer se- quences. Advances in neural information processing sys- tems, 33:17283-17297, 2020.","title":"Big bird: Transformers for longer sequences","authors":"Manzil Zaheer, Guru Guruganesh, Avinava Kumar, Joshua Dubey, Chris Ainslie, Santiago Alberti, Philip Ontanon, Anirudh Pham, Qifan Ravula, Li Wang, Yang","year":"2020","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":50,"raw":"[Zhu and Soricut, 2021] Zhenhai Zhu and Radu Soricut. H- transformer-1d: Fast one-dimensional hierarchical atten- tion for sequences. arXiv preprint arXiv:2107.11906, 2021.","title":"Htransformer-1d: Fast one-dimensional hierarchical attention for sequences","authors":"Zhenhai Zhu, Radu Soricut","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":51,"raw":"[Zhu et al., 2021] Chen Zhu, Wei Ping, Chaowei Xiao, Mo- hammad Shoeybi, Tom Goldstein, Anima Anandkumar, and Bryan Catanzaro. Long-short transformer: Efficient transformers for language and vision. Advances in Neural Information Processing Systems, 34, 2021.","title":"Long-short transformer: Efficient transformers for language and vision","authors":"Chen Zhu, Wei Ping, Chaowei Xiao, Mohammad Shoeybi, Tom Goldstein, Anima Anandkumar, Bryan Catanzaro","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":52,"raw":"[Zuo et al., 2022] Simiao Zuo, Xiaodong Liu, Jian Jiao, De- nis Charles, Eren Manavoglu, Tuo Zhao, and Jianfeng Gao. Efficient long sequence modeling via state space aug- mented transformer, 2022.","title":"Efficient long sequence modeling via state space augmented transformer","authors":"Zuo","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}