{"type":"ScholarlyArticle","title":"Singularformer: Learning to Decompose Self-Attention to Linearize the Complexity of Transformer","authors":[{"name":"Yifan Wu","slug":"yifan-wu","given":"Yifan","family":"Wu","affiliation":"The School of Computer Science and Engineering , Central South University , Changsha , China"},{"name":"Shichao Kan","slug":"shichao-kan","given":"Shichao","family":"Kan","affiliation":"The School of Computer Science and Engineering , Central South University , Changsha , China"},{"name":"Min Zeng","slug":"min-zeng","given":"Min","family":"Zeng","affiliation":"The School of Computer Science and Engineering , Central South University , Changsha , China"},{"name":"Min Li","slug":"min-li","given":"Min","family":"Li","affiliation":"The School of Computer Science and Engineering , Central South University , Changsha , China"}],"abstract":"Transformers achieve excellent performance in a variety of domains since they can capture long-distance dependencies through the self-attention mechanism. However, self-attention is computationally costly due to its quadratic complexity and high memory consumption. In this paper, we propose a novel Transformer variant (Singularformer) that uses neural networks to learn the singular value decomposition process of the attention matrix to design a linear-complexity and memory-efficient global self-attention mechanism. Specifically, we decompose the attention matrix into the product of three matrix factors based on singular value decomposition and design neural networks to learn these matrix factors, then the associative law of matrix multiplication is used to linearize the calculation of self-attention. The above procedure allows us to compute self-attention as two-dimensional reduction processes in the first and second token dimensional spaces, followed by a multi-head self-attention computational process on the first dimensional reduced token features. Experimental results on 8 real-world datasets demonstrate that Singularformer performs favorably against the other Transformer variants with lower time and space complexity. Our source code is publicly available at https://github.com/CSUBioGroup/Singularformer.","doi":"10.24963/ijcai.2023/493","url":"https://ijcai.org/proceedings/2023/493","pdf":"https://ijcai.org/proceedings/2023/0493.pdf","markdown":"https://ijcai.org/proceedings/2023/493.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/493","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Machine Learning","track":"Main Track","number":493,"pages":{"first":4433,"last":4441},"keywords":["Machine Learning: ML: Attention models"],"videos":[],"references":[{"position":1,"raw":"[Beltagy et al., 2020] Iz Beltagy, Matthew E Peters, and Ar- man Cohan. Longformer: The long-document trans- former. arXiv preprint arXiv:2004.05150, 2020.","title":"Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer","authors":"Beltagy","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Child et al., 2019] Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019.","title":"Rewon Child","authors":"Child","year":"2019","venue":"Generating long sequences with sparse transformers","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Choromanski et al., 2020] Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. arXiv preprint arXiv:2009.14794, 2020.","title":"Rethinking attention with performers","authors":"Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Debole and Sebastiani, 2005] Franca Debole and Fabrizio Sebastiani. An analysis of the relative hardness of reuters- 21578 subsets. Journal of the American Society for Infor- mation Science and technology, 56(6):584-596, 2005.","title":"Franca Debole and Fabrizio Sebastiani. An analysis of the relative hardness of reuters-21578 subsets","authors":"Sebastiani Debole","year":"2005","venue":"Journal of the American Society for Information Science and technology","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dosovitskiy et al., 2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Min- derer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.","title":"An image is worth 16x16 words: Transformers for image recognition at scale","authors":"Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fox et al., 2014] Naomi K Fox, Steven E Brenner, and John-Marc Chandonia. Scope: Structural classification of proteins-extended, integrating scop and astral data and classification of new structures. Nucleic acids research, 42(D1):D304-D309, 2014.","title":"Scope: Structural classification of proteins-extended, integrating scop and astral data and classification of new structures","authors":"Naomi K Fox, Steven E Brenner, John-Marc Chandonia","year":"2014","venue":"Nucleic acids research","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gionis et al., 1999] Aristides Gionis, Piotr Indyk, Rajeev Motwani, et al. Similarity search in high dimensions via hashing. In Vldb, volume 99, pages 518-529, 1999.","title":"Similarity search in high dimensions via hashing","authors":"Gionis","year":"1999","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Greene and Cunningham, 2006] Derek Greene and Pádraig Cunningham. Practical solutions to the problem of di- agonal dominance in kernel document clustering. In Proc. 23rd International Conference on Machine learning (ICML'06), pages 377-384. ACM Press, 2006.","title":"Practical solutions to the problem of diagonal dominance in kernel document clustering","authors":"Cunningham Greene, Derek Greene, Pádraig Cunningham","year":"2006","venue":"Proc. 23rd International Conference on Machine learning (ICML'06)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[He et al., 2021] Ruining He, Anirudh Ravula, Bhargav Kanagal, and Joshua Ainslie. Realformer: Transformer likes residual attention. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 929-943, 2021.","title":"Realformer: Transformer likes residual attention","authors":"He","year":"2021","venue":"Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hua et al., 2022] Weizhe Hua, Zihang Dai, Hanxiao Liu, and Quoc Le. Transformer quality in linear time. In Inter- national Conference on Machine Learning, pages 9099- 9117. PMLR, 2022.","title":"Transformer quality in linear time","authors":"Hua","year":"2022","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Johnson et al., 2016] Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo An- thony Celi, and Roger G Mark. Mimic-iii, a freely ac- cessible critical care database. Scientific data, 3(1):1-9, 2016.","title":"Mimic-iii, a freely accessible critical care database","authors":"E W Alistair, Tom J Johnson, Lu Pollard, Li-Wei H Shen, Mengling Lehman, Mohammad Feng, Benjamin Ghassemi, Peter Moody, Leo Szolovits, Roger G Anthony Celi, Mark","year":"2016","venue":"Scientific data","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Jumper et al., 2021] John Jumper, Richard Evans, Alexan- der Pritzel, Tim Green, Michael Figurnov, Olaf Ron- neberger, Kathryn Tunyasuvunakool, Russ Bates, Au- gustin Žídek, Anna Potapenko, et al. Highly accu- rate protein structure prediction with alphafold. Nature, 596(7873):583-589, 2021.","title":"Highly accurate protein structure prediction with alphafold","authors":"John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko","year":"2021","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Katharopoulos et al., 2020] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and Franc ¸ois Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In International Conference on Machine Learning, pages 5156-5165. PMLR, 2020.","title":"Transformers are rnns: Fast autoregressive transformers with linear attention","authors":"Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, Franc ¸ois, Fleuret","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kenton and Toutanova, 2019] Jacob Devlin Ming- Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for lan- guage understanding. In Proceedings of NAACL-HLT, pages 4171-4186, 2019.","title":"Bert: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton, Lee Kristina, Toutanova","year":"2019","venue":"Proceedings of NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kitaev et al., 2020] Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In International Conference on Learning Representations, 2020.","title":"Reformer: The efficient transformer","authors":"Nikita Kitaev, Lukasz Kaiser, Anselm Levskaya","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Krizhevsky and Hinton, 2009] Alex Krizhevsky and Geof- frey Hinton. Learning multiple layers of features from tiny images. Master's thesis, Department of Computer Science, University of Toronto, 2009.","title":"Learning multiple layers of features from tiny images","authors":"Alex Krizhevsky, Geoffrey Hinton","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lang, 1995] Ken Lang. Newsweeder: Learning to filter net- news. In Machine Learning Proceedings 1995, pages 331- 339. Elsevier, 1995.","title":"Newsweeder: Learning to filter netnews","authors":"Lang ;, Ken Lang","year":"1995","venue":"Machine Learning Proceedings 1995","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Liu et al., 2019] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019.","title":"Roberta: A robustly optimized bert pretraining approach","authors":"Liu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Liu et al., 2021] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 10012- 10022, 2021.","title":"Swin transformer: Hierarchical vision transformer using shifted windows","authors":"Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, Baining Guo","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Loshchilov and Hutter, 2018] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In Inter- national Conference on Learning Representations, 2018.","title":"Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization","authors":"Hutter Loshchilov","year":"2018","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Ma et al., 2022] Xuezhe Ma, Chunting Zhou, Xiang Kong, Junxian He, Liangke Gui, Graham Neubig, Jonathan May, and Luke Zettlemoyer. Mega: moving average equipped gated attention. arXiv preprint arXiv:2209.10655, 2022.","title":"Mega: moving average equipped gated attention","authors":"Ma","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Maas et al., 2011] Andrew L. Maas, Raymond E. Daly, Pe- ter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Tech- nologies, pages 142-150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics.","title":"Learning word vectors for sentiment analysis","authors":"Andrew L Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, Christopher Potts","year":"2011","venue":"Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Micikevicius et al., 2018] Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. Mixed precision training. In International Conference on Learning Representations, 2018.","title":"Mixed precision training","authors":"Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh","year":"2018","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Shen et al., 2021] Zhuoran Shen, Mingyuan Zhang, Haiyu Zhao, Shuai Yi, and Hongsheng Li. Efficient attention: Attention with linear complexities. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 3531-3539, 2021.","title":"Efficient attention: Attention with linear complexities","authors":"Shen","year":"2021","venue":"Proceedings of the IEEE/CVF winter conference on applications of computer vision","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[So et al., 2021] David So, Wojciech Mańke, Hanxiao Liu, Zihang Dai, Noam Shazeer, and Quoc V Le. Searching for efficient transformers for language modeling. Advances in Neural Information Processing Systems, 34:6010-6022, 2021.","title":"Searching for efficient transformers for language modeling","authors":"David So, Wojciech Mańke, Hanxiao Liu, Zihang Dai, Noam Shazeer, Quoc V Le","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wang et al., 2020] Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768, 2020.","title":"Linformer: Self-attention with linear complexity","authors":"Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, Hao Ma","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Xiong et al., 2020] Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. On layer nor- malization in the transformer architecture. In International Conference on Machine Learning, pages 10524-10533.","title":"On layer normalization in the transformer architecture","authors":"Xiong","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"PMLR, 2020.","title":null,"authors":"Pmlr","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Xiong et al., 2021] Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, and Vikas Singh. Nyströmformer: A nyström-based algorithm for approximating self-attention. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 35, pages 14138-14148, 2021.","title":"Nyströmformer: A nyström-based algorithm for approximating self-attention","authors":"Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, Vikas Singh","year":"2021","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zaheer et al., 2020] Manzil Zaheer, Guru Guruganesh, Ku- mar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santi- ago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer se- quences. Advances in Neural Information Processing Sys- tems, 33:17283-17297, 2020.","title":"Big bird: Transformers for longer sequences","authors":"Manzil Zaheer, Guru Guruganesh, Avinava Kumar, Joshua Dubey, Chris Ainslie, Santiago Alberti, Philip Ontanon, Anirudh Pham, Qifan Ravula, Li Wang, Yang","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhang et al., 2021] Ningyu Zhang, Zhen Bi, Xiaozhuan Liang, Siyuan Cheng, Haosen Hong, Shumin Deng, Qiang Zhang, Jiazhang Lian, and Huajun Chen. Ontoprotein: Protein pretraining with gene ontology embedding. In International Conference on Learning Representations, 2021.","title":"Ontoprotein: Protein pretraining with gene ontology embedding","authors":"Zhang","year":"2021","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhu et al., 2021] Chen Zhu, Wei Ping, Chaowei Xiao, Mo- hammad Shoeybi, Tom Goldstein, Anima Anandkumar, and Bryan Catanzaro. Long-short transformer: Efficient transformers for language and vision. Advances in Neural Information Processing Systems, 34:17723-17736, 2021.","title":"Long-short transformer: Efficient transformers for language and vision","authors":"Chen Zhu, Wei Ping, Chaowei Xiao, Mohammad Shoeybi, Tom Goldstein, Anima Anandkumar, Bryan Catanzaro","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}