{"type":"ScholarlyArticle","title":"KDLGT: A Linear Graph Transformer Framework via Kernel Decomposition Approach","authors":[{"name":"Yi Wu","slug":"yi-wu","given":"Yi","family":"Wu","affiliation":"Academy for Advanced Interdisciplinary Studies , Peking University"},{"name":"Yanyang Xu","slug":"yanyang-xu","given":"Yanyang","family":"Xu","affiliation":"School of Software and Microelectronics , Peking University"},{"name":"Wenhao Zhu","slug":"wenhao-zhu","given":"Wenhao","family":"Zhu","affiliation":"National Key Lab of General AI , School of Intelligence Science and Technology , Peking University"},{"name":"Guojie Song","slug":"guojie-song","given":"Guojie","family":"Song","affiliation":"National Key Lab of General AI , School of Intelligence Science and Technology , Peking University"},{"name":"Zhouchen Lin","slug":"zhouchen-lin","given":"Zhouchen","family":"Lin","affiliation":"National Key Lab of General AI , School of Intelligence Science and Technology , Peking University; Institute for Artificial Intelligence , Peking University; Peng Cheng Laboratory"},{"name":"Liang Wang","slug":"liang-wang","given":"Liang","family":"Wang","affiliation":"Alibaba Group , Beijing , China"},{"name":"Shaoguo Liu","slug":"shaoguo-liu","given":"Shaoguo","family":"Liu","affiliation":"Alibaba Group , Beijing , China"}],"abstract":"In recent years, graph Transformers (GTs) have been demonstrated as a robust architecture for a wide range of graph learning tasks. However, the quadratic complexity of GTs limits their scalability on large-scale data, in comparison to Graph Neural Networks (GNNs). In this work, we propose the Kernel Decomposition Linear Graph Transformer (KDLGT), an accelerating framework for building scalable and powerful GTs. KDLGT employs the kernel decomposition approach to rearrange the order of matrix multiplication, thereby reducing complexity to linear. Additionally, it categorizes GTs into three distinct types and provides tailored accelerating methods for each category to encompass all types of GTs. Furthermore, we provide a theoretical analysis of the performance gap between KDLGT and self-attention to ensure its effectiveness. Under this framework, we select two representative GTs to design our models. Experiments on both real-world and synthetic datasets indicate that KDLGT not only achieves state-of-the-art performance on various datasets but also reaches an acceleration ratio of approximately 10 on graphs of certain sizes.","doi":"10.24963/ijcai.2023/263","url":"https://ijcai.org/proceedings/2023/263","pdf":"https://ijcai.org/proceedings/2023/0263.pdf","markdown":"https://ijcai.org/proceedings/2023/263.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/263","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Data Mining","track":"Main Track","number":263,"pages":{"first":2370,"last":2378},"keywords":["Data Mining: DM: Big data and scalability","Data Mining: DM: Mining graphs"],"videos":[],"references":[{"position":1,"raw":"[Alon and Yahav, 2021] Uri Alon and Eran Yahav. On the bottleneck of graph neural networks and its practical im- plications. In International Conference on Learning Rep- resentations, 2021.","title":"On the bottleneck of graph neural networks and its practical implications","authors":"Alon, Yahav, Uri Alon, Eran Yahav","year":"2021","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Beltagy et al., 2020] Iz Beltagy, Matthew E Peters, and Ar- man Cohan. Longformer: The long-document trans- former. arXiv preprint arXiv:2004.05150, 2020.","title":"Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer","authors":"Beltagy","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2022] Dexiong Chen, Leslie O'Bray, and Karsten Borgwardt. Structure-aware transformer for graph representation learning. In International Conference on Machine Learning, pages 3469-3489. PMLR, 2022.","title":"Structure-aware transformer for graph representation learning","authors":"Chen","year":"2022","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Child et al., 2019] Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019.","title":"Rewon Child","authors":"Child","year":"2019","venue":"Generating long sequences with sparse transformers","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Choromanski et al., 2020] Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, An- dreea Gane, Tamas Sarlos, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. In International Conference on Learning Representations, 2020.","title":"Rethinking attention with performers","authors":"Valerii Krzysztof Marcin Choromanski, David Likhosherstov, Xingyou Dohan, Andreea Song, Tamas Gane, Peter Sarlos, Jared Quincy Hawkins, Afroz Davis, Lukasz Mohiuddin, Kaiser","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Choromanski et al., 2022] Krzysztof Choromanski, Han Lin, Haoxian Chen, Tianyi Zhang, Arijit Sehanobish, Va- lerii Likhosherstov, Jack Parker-Holder, Tamas Sarlos, Adrian Weller, and Thomas Weingarten. From block- toeplitz matrices to differential equations on graphs: to- wards a general theory for scalable masked transformers. In International Conference on Machine Learning, pages 3962-3983. PMLR, 2022.","title":"From blocktoeplitz matrices to differential equations on graphs: towards a general theory for scalable masked transformers","authors":"Krzysztof Choromanski, Han Lin, Haoxian Chen, Tianyi Zhang, Arijit Sehanobish, Valerii Likhosherstov, Jack Parker-Holder, Tamas Sarlos, Adrian Weller, Thomas Weingarten","year":"2022","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dwivedi et al., 2020] Vijay Prakash Dwivedi, Chaitanya K Joshi, Thomas Laurent, Yoshua Bengio, and Xavier Bres- son. Benchmarking graph neural networks. arXiv preprint arXiv:2003.00982, 2020.","title":"Benchmarking graph neural networks","authors":"Vijay Prakash Dwivedi, K Chaitanya, Thomas Joshi, Yoshua Laurent, Xavier Bengio, Bresson","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hamilton et al., 2017] Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Advances in Neural Information Processing Sys- tems, 30:1024-1034, 2017.","title":"Inductive representation learning on large graphs","authors":"Hamilton","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hu et al., 2020] Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. Advances in Neural Informa- tion Processing Systems, 33:22118-22133, 2020.","title":"Open graph benchmark: Datasets for machine learning on graphs","authors":"Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, Jure Leskovec","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Huang et al., 2020] Kexin Huang, Cao Xiao, Lucas M Glass, Marinka Zitnik, and Jimeng Sun. Skipgnn: pre- dicting molecular interactions with skip-graph networks. Scientific reports, 10(1):1-16, 2020.","title":"Skipgnn: predicting molecular interactions with skip-graph networks","authors":"Kexin Huang, Cao Xiao, Lucas M Glass, Marinka Zitnik, Jimeng Sun","year":"2020","venue":"Scientific reports","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Katharopoulos et al., 2020] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and Franc ¸ois Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In International Conference on Machine Learning, pages 5156-5165. PMLR, 2020.","title":"Transformers are rnns: Fast autoregressive transformers with linear attention","authors":"Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, Franc ¸ois, Fleuret","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kingma and Ba, 2015] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Interna- tional Conference on Learning Representations, 2015.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2015","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kipf and Welling, 2017] Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Rep- resentations, 2017.","title":"Semi-supervised classification with graph convolutional networks","authors":"Thomas N Kipf, Max Welling","year":"2017","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kreuzer et al., 2021] Devin Kreuzer, Dominique Beaini, Will Hamilton, Vincent Létourneau, and Prudencio Tossou. Rethinking graph transformers with spectral atten- tion. Advances in Neural Information Processing Systems, 34:21618-21629, 2021.","title":"Rethinking graph transformers with spectral attention","authors":"Devin Kreuzer, Dominique Beaini, Will Hamilton, Vincent Létourneau, Prudencio Tossou","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2021] Yangyang Li, Yipeng Ji, Shaoning Li, Shu- long He, Yinhao Cao, Yifeng Liu, Hong Liu, Xiong Li, Jun Shi, and Yangchao Yang. Relevance-aware anoma- lous users detection in social network via graph neural net- work. In 2021 International Joint Conference on Neural Networks (IJCNN), pages 1-8. IEEE, 2021.","title":"Relevance-aware anomalous users detection in social network via graph neural network","authors":"Yangyang Li, Yipeng Ji, Shaoning Li, Shulong He, Yinhao Cao, Yifeng Liu, Hong Liu, Xiong Li, Jun Shi, Yangchao Yang","year":"2021","venue":"International Joint Conference on Neural Networks (IJCNN)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liu et al., 2021] Xiyang Liu, Huobin Tan, Qinghong Chen, and Guangyan Lin. Ragat: Relation aware graph attention network for knowledge graph completion. IEEE Access, 9:20840-20849, 2021.","title":"Ragat: Relation aware graph attention network for knowledge graph completion","authors":"Xiyang Liu, Huobin Tan, Qinghong Chen, Guangyan Lin","year":"2021","venue":"IEEE Access","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Maron et al., 2019] Haggai Maron, Heli Ben-Hamu, Hadar Serviansky, and Yaron Lipman. Provably powerful graph networks. Advances in Neural Information Processing Systems, 32:2153-2164, 2019.","title":"Provably powerful graph networks","authors":"Maron","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Mialon et al., 2021] Grégoire Mialon, Dexiong Chen, Mar- got Selosse, and Julien Mairal. Graphit: Encod- ing graph structure in transformers. arXiv preprint arXiv:2106.05667, 2021.","title":"Graphit: Encoding graph structure in transformers","authors":"Grégoire Mialon, Dexiong Chen, Margot Selosse, Julien Mairal","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Morris et al., 2019] Christopher Morris, Martin Ritzert, Matthias Fey, William L Hamilton, Jan Eric Lenssen, Gau- rav Rattan, and Martin Grohe. Weisfeiler and leman go neural: Higher-order graph neural networks. In Proceed- ings of the AAAI conference on artificial intelligence, vol- ume 33, pages 4602-4609, 2019.","title":"Weisfeiler and leman go neural: Higher-order graph neural networks","authors":"Christopher Morris, Martin Ritzert, Matthias Fey, Jan William L Hamilton, Gaurav Eric Lenssen, Martin Rattan, Grohe","year":"2019","venue":"Proceedings of the AAAI conference on artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Morris et al., 2020] Christopher Morris, Nils M Kriege, Franka Bause, Kristian Kersting, Petra Mutzel, and Mar- ion Neumann. Tudataset: A collection of bench- mark datasets for learning with graphs. arXiv preprint arXiv:2007.08663, 2020.","title":"Tudataset: A collection of benchmark datasets for learning with graphs","authors":"Christopher Morris, Nils M Kriege, Franka Bause, Kristian Kersting, Petra Mutzel, Marion Neumann","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Park et al., 2022] Jinyoung Park, Seongjun Yun, Hyeonjin Park, Jaewoo Kang, Jisu Jeong, Kyung-Min Kim, Jung- woo Ha, and Hyunwoo J Kim. Deformable graph trans- former. arXiv preprint arXiv:2206.14337, 2022.","title":"Deformable graph transformer","authors":"Jinyoung Park, Seongjun Yun, Hyeonjin Park, Jaewoo Kang, Jisu Jeong, Kyung-Min Kim, Jungwoo Ha, Hyunwoo J Kim","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Parmar et al., 2018] Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In International Con- ference on Machine Learning, pages 4055-4064. PMLR, 2018.","title":"Image transformer","authors":"Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, Dustin Tran","year":"2018","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Pavan and Pelillo, 2006] Massimiliano Pavan and Marcello Pelillo. Dominant sets and pairwise clustering. IEEE transactions on pattern analysis and machine intelligence, 29(1):167-172, 2006.","title":"Dominant sets and pairwise clustering","authors":"Massimiliano Pavan, Marcello Pelillo","year":"2006","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Peng et al., 2020] Hao Peng, Nikolaos Pappas, Dani Yo- gatama, Roy Schwartz, Noah Smith, and Lingpeng Kong. Random feature attention. In International Conference on Learning Representations, 2020.","title":"Random feature attention","authors":"Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah Smith, Lingpeng Kong","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Qiu et al., 2020] Jiezhong Qiu, Hao Ma, Omer Levy, Wen- tau Yih, Sinong Wang, and Jie Tang. Blockwise self- attention for long document understanding. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2555-2565, 2020.","title":"Blockwise selfattention for long document understanding","authors":"Jiezhong Qiu, Hao Ma, Omer Levy, Wentau Yih, Sinong Wang, Jie Tang","year":"2020","venue":"Findings of the Association for Computational Linguistics: EMNLP 2020","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Rampášek et al., 2022] Ladislav Rampášek, Mikhail Galkin, Vijay Prakash Dwivedi, Anh Tuan Luu, Guy Wolf, and Dominique Beaini. Recipe for a general, powerful, scalable graph transformer. arXiv preprint arXiv:2205.12454, 2022.","title":"Recipe for a general, powerful, scalable graph transformer","authors":"Ladislav Rampášek, Mikhail Galkin, Vijay Prakash Dwivedi, Anh Tuan Luu, Guy Wolf, Dominique Beaini","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Rozemberczki and Sarkar, 2020] Benedek Rozemberczki and Rik Sarkar. Characteristic functions on graphs: Birds of a feather, from statistical descriptors to parametric models. In Proceedings of the 29th ACM international conference on information & knowledge management, pages 1325-1334, 2020.","title":"Benedek Rozemberczki and Rik Sarkar. Characteristic functions on graphs: Birds of a feather, from statistical descriptors to parametric models","authors":"Sarkar Rozemberczki","year":"2020","venue":"Proceedings of the 29th ACM international conference on information & knowledge management","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Sys- tems, 30:5998-6008, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Veličković et al., 2018] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In Interna- tional Conference on Learning Representations, 2018.","title":"Graph attention networks","authors":"Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, Yoshua Bengio","year":"2018","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wang et al., 2020] Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768, 2020.","title":"Linformer: Self-attention with linear complexity","authors":"Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, Hao Ma","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Wang et al., 2021] Yuyang Wang, Jianren Wang, Zhonglin Cao, and Amir Barati Farimani. Molclr: Molecular con- trastive learning of representations via graph neural net- works. arXiv preprint arXiv:2102.10056, 2021.","title":"Molclr: Molecular contrastive learning of representations via graph neural networks","authors":"Yuyang Wang, Jianren Wang, Zhonglin Cao, Amir Barati, Farimani","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Wu et al., 2022] Qitian Wu, Wentao Zhao, Zenan Li, David P Wipf, and Junchi Yan. Nodeformer: A scalable graph structure learning transformer for node classifica- tion. volume 35, pages 27387-27401, 2022.","title":"Nodeformer: A scalable graph structure learning transformer for node classification","authors":"Qitian Wu, Wentao Zhao, Zenan Li, David P Wipf, Junchi Yan","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Xu et al., 2019] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural net- works? In International Conference on Learning Rep- resentations, 2019.","title":"How powerful are graph neural networks","authors":"Xu","year":"2019","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Yang et al., 2016] Zhilin Yang, William W. Cohen, and Ruslan Salakhutdinov. Revisiting semi-supervised learn- ing with graph embeddings. In International Conference on Machine Learning, pages 40-48. PMLR, 2016.","title":"Revisiting semi-supervised learning with graph embeddings","authors":"Zhilin Yang, William W Cohen, Ruslan Salakhutdinov","year":"2016","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Yasunaga et al., 2021] Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut, Percy Liang, and Jure Leskovec. Qa- gnn: Reasoning with language models and knowledge graphs for question answering. In North American Chapter of the Association for Computational Linguistics (NAACL), 2021.","title":"Qagnn: Reasoning with language models and knowledge graphs for question answering","authors":"Yasunaga","year":"2021","venue":"North American Chapter of the Association for Computational Linguistics (NAACL)","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Ying et al., 2021] Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie-Yan Liu. Do transformers really perform badly for graph representation? Advances in Neural Information Processing Systems, 34:28877-28888, 2021.","title":"Do transformers really perform badly for graph representation?","authors":"Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, Tie-Yan Liu","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Zhang et al., 2020] Jiawei Zhang, Haopeng Zhang, Con- gying Xia, and Li Sun. Graph-bert: Only attention is needed for learning graph representations. arXiv preprint arXiv:2001.05140, 2020.","title":"Graph-bert: Only attention is needed for learning graph representations","authors":"Jiawei Zhang, Haopeng Zhang, Congying Xia, Li Sun","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Zhong et al., 2020] Ting Zhong, Tianliang Wang, Jiahao Wang, Jin Wu, and Fan Zhou. Multiple-aspect attentional graph neural networks for online social network user lo- calization. IEEE Access, 8:95223-95234, 2020.","title":"Multiple-aspect attentional graph neural networks for online social network user localization","authors":"Ting Zhong, Tianliang Wang, Jiahao Wang, Jin Wu, Fan Zhou","year":"2020","venue":"IEEE Access","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}