{"type":"ScholarlyArticle","title":"AnchorGT: Efficient and Flexible Attention Architecture for Scalable Graph Transformers","authors":[{"name":"Wenhao Zhu","slug":"wenhao-zhu","given":"Wenhao","family":"Zhu","affiliation":"National Key Laboratory of General Artificial Intelligence , School of Intelligence Science and Technology , Peking University"},{"name":"Guojie Song","slug":"guojie-song","given":"Guojie","family":"Song","affiliation":"National Key Laboratory of General Artificial Intelligence , School of Intelligence Science and Technology , Peking University"},{"name":"Liang Wang","slug":"liang-wang","given":"Liang","family":"Wang","affiliation":"Alibaba Group"},{"name":"Shaoguo Liu","slug":"shaoguo-liu","given":"Shaoguo","family":"Liu","affiliation":"Alibaba Group"}],"abstract":"Graph Transformers (GTs) have significantly advanced the field of graph representation learning by overcoming the limitations of message-passing graph neural networks (GNNs) and demonstrating promising performance and expressive power. However, the quadratic complexity of self-attention mechanism in GTs has limited their scalability, and previous approaches to address this issue often suffer from expressiveness degradation or lack of versatility. To address this issue, we propose AnchorGT, a novel attention architecture for GTs with global receptive field and almost linear complexity, which serves as a flexible building block to improve the scalability of a wide range of GT models. Inspired by anchor-based GNNs, we employ structurally important k-dominating node set as anchors and design an attention mechanism that focuses on the relationship between individual nodes and anchors, while retaining the global receptive field for all nodes. With its intuitive design, AnchorGT can easily replace the attention module in various GT models with different network architectures and structural encodings, resulting in reduced computational overhead without sacrificing performance. In addition, we theoretically prove that AnchorGT attention can be strictly more expressive than Weisfeiler-Lehman test, showing its superiority in representing graph structures. Our experiments on three state-of-the-art GT models demonstrate that their AnchorGT variants can achieve similar results while being faster and significantly more memory efficient.","doi":"10.24963/ijcai.2024/631","url":"https://ijcai.org/proceedings/2024/631","pdf":"https://ijcai.org/proceedings/2024/0631.pdf","markdown":"https://ijcai.org/proceedings/2024/631.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/631","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Machine Learning","track":"Main Track","number":631,"pages":{"first":5707,"last":5715},"keywords":["Machine Learning: ML: Sequence and graph learning"],"videos":[],"references":[{"position":1,"raw":"[Bresson and Laurent, 2017] Xavier Bresson and Thomas Laurent. Residual gated graph convnets. arXiv preprint arXiv:1711.07553, 2017.","title":"Residual gated graph convnets","authors":"Xavier Bresson, Thomas Laurent","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2012] Duanbing Chen, Linyuan Lü, Ming- Sheng Shang, Yi-Cheng Zhang, and Tao Zhou. Identifying influential nodes in complex networks. Physica a: Statis- tical mechanics and its applications, 391(4):1777-1787, 2012.","title":"Identifying influential nodes in complex networks","authors":"Chen","year":"2012","venue":"Physica a: Statistical mechanics and its applications","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2020a] Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. Simple and deep graph convolutional networks. In International conference on machine learning, pages 1725-1735. PMLR, 2020.","title":"Simple and deep graph convolutional networks","authors":"Chen","year":"2020","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2020b] Yu Chen, Lingfei Wu, and Mohammed Zaki. Iterative deep graph learning for graph neural net- works: Better and robust node embeddings. Advances in neural information processing systems, 33:19314-19326, 2020.","title":"Iterative deep graph learning for graph neural networks: Better and robust node embeddings","authors":"Chen","year":"2020","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2022] Dexiong Chen, Leslie O'Bray, and Karsten Borgwardt. Structure-aware transformer for graph representation learning. In International Conference on Machine Learning, pages 3469-3489. PMLR, 2022.","title":"Structure-aware transformer for graph representation learning","authors":"Chen","year":"2022","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dong et al., 2021] Lijun Dong, Hong Yao, Dan Li, Yi Wang, Shengwen Li, and Qingzhong Liang. Improving graph neural network via complex-network-based anchor structure. Knowledge-Based Systems, 233:107528, 2021.","title":"Improving graph neural network via complex-network-based anchor structure","authors":"Lijun Dong, Hong Yao, Dan Li, Yi Wang, Shengwen Li, Qingzhong Liang","year":"2021","venue":"Knowledge-Based Systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dwivedi and Bresson, 2020] Vijay Prakash Dwivedi and Xavier Bresson. A generalization of transformer networks to graphs. arXiv preprint arXiv:2012.09699, 2020.","title":"A generalization of transformer networks to graphs","authors":"Vijay Prakash, Dwivedi, Xavier Bresson","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dwivedi et al., 2020] Vijay Prakash Dwivedi, Chaitanya K Joshi, Thomas Laurent, Yoshua Bengio, and Xavier Bres- son. Benchmarking graph neural networks. arXiv preprint arXiv:2003.00982, 2020.","title":"Benchmarking graph neural networks","authors":"Vijay Prakash Dwivedi, K Chaitanya, Thomas Joshi, Yoshua Laurent, Xavier Bengio, Bresson","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Dwivedi et al., 2022] Vijay Prakash Dwivedi, Ladislav Rampášek, Mikhail Galkin, Ali Parviz, Guy Wolf, Anh Tuan Luu, and Dominique Beaini. Long range graph benchmark. arXiv preprint arXiv:2206.08164, 2022.","title":"Long range graph benchmark","authors":"Vijay Prakash Dwivedi, Ladislav Rampášek, Mikhail Galkin, Ali Parviz, Guy Wolf, Anh Tuan Luu, Dominique Beaini","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hu et al., 2019] Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovec. Strategies for pre-training graph neural net- works. arXiv preprint arXiv:1905.12265, 2019.","title":"Strategies for pre-training graph neural networks","authors":"Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, Jure Leskovec","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hu et al., 2020] Weihua Hu, Matthias Fey, Marinka Zit- nik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. arXiv preprint arXiv:2005.00687, 2020.","title":"Open graph benchmark: Datasets for machine learning on graphs","authors":"Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, Jure Leskovec","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hu et al., 2021] Weihua Hu, Matthias Fey, Hongyu Ren, Maho Nakata, Yuxiao Dong, and Jure Leskovec. Ogb-lsc: A large-scale challenge for machine learning on graphs. arXiv preprint arXiv:2103.09430, 2021.","title":"Ogb-lsc: A large-scale challenge for machine learning on graphs","authors":"Weihua Hu, Matthias Fey, Hongyu Ren, Maho Nakata, Yuxiao Dong, Jure Leskovec","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hussain et al., 2022] Md Shamim Hussain, Mohammed J Zaki, and Dharmashankar Subramanian. Global self- attention as a replacement for graph convolution. In Pro- ceedings of the 28th ACM SIGKDD Conference on Knowl- edge Discovery and Data Mining, pages 655-665, 2022.","title":"Global selfattention as a replacement for graph convolution","authors":"Md Shamim Hussain, Mohammed J Zaki, Dharmashankar Subramanian","year":"2022","venue":"Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kipf and Welling, 2016] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907, 2016.","title":"Semi-supervised classification with graph convolutional networks","authors":"N Thomas, Max Kipf, Welling","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Klicpera et al., 2018] Johannes Klicpera, Aleksandar Bo- jchevski, and Stephan Günnemann. Predict then propa- gate: Graph neural networks meet personalized pagerank. arXiv preprint arXiv:1810.05997, 2018.","title":"Predict then propagate: Graph neural networks meet personalized pagerank","authors":"Klicpera","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kong et al., 2023] Kezhi Kong, Jiuhai Chen, John Kirchen- bauer, Renkun Ni, C. Bayan Bruss, and Tom Goldstein. GOAT: A global transformer on large-scale graphs. In An- dreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Ma- chine Learning, volume 202 of Proceedings of Machine Learning Research, pages 17375-17390. PMLR, 23-29 Jul 2023.","title":"GOAT: A global transformer on large-scale graphs","authors":"Kong","year":"2023","venue":"Proceedings of the 40th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Kreuzer et al., 2021] Devin Kreuzer, Dominique Beaini, William L Hamilton, Vincent Létourneau, and Prudencio Tossou. Rethinking graph transformers with spectral at- tention. arXiv preprint arXiv:2106.03893, 2021.","title":"Rethinking graph transformers with spectral attention","authors":"Devin Kreuzer, Dominique Beaini, Vincent William L Hamilton, Prudencio Létourneau, Tossou","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Liu et al., 2020] Chao Liu, Xinchuan Li, Dongyang Zhao, Shaolong Guo, Xiaojun Kang, Lijun Dong, and Hong Yao. A-gnn: Anchors-aware graph neural networks for node embedding. In International Conference on Hetero- geneous Networking for Quality, Reliability, Security and Robustness, pages 141-153. Springer, 2020.","title":"A-gnn: Anchors-aware graph neural networks for node embedding","authors":"Chao Liu, Xinchuan Li, Dongyang Zhao, Shaolong Guo, Xiaojun Kang, Lijun Dong, Hong Yao","year":"2020","venue":"International Conference on Heterogeneous Networking for Quality, Reliability, Security and Robustness","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Müller et al., 2023] Luis Müller, Mikhail Galkin, Christo- pher Morris, and Ladislav Rampášek. Attending to graph transformers. arXiv preprint arXiv:2302.04181, 2023.","title":"Attending to graph transformers","authors":"Luis Müller, Mikhail Galkin, Christopher Morris, Ladislav Rampášek","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Nguyen et al., 2020] Minh Hai Nguyen, Minh Hoàng Hà, Diep N Nguyen, et al. Solving the k-dominating set prob- lem on very large-scale networks. Computational Social Networks, 7(1):1-15, 2020.","title":"Solving the k-dominating set problem on very large-scale networks","authors":"Nguyen","year":"2020","venue":"Computational Social Networks","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Park et al., 2022] Wonpyo Park, Woong-Gi Chang, Donggeon Lee, Juntae Kim, et al. Grpe: Relative po- sitional encoding for graph transformer. In ICLR2022 Machine Learning for Drug Discovery, 2022.","title":"Grpe: Relative positional encoding for graph transformer","authors":"Park","year":"2022","venue":"ICLR2022 Machine Learning for Drug Discovery","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Rampášek et al., 2022] Ladislav Rampášek, Mikhail Galkin, Vijay Prakash Dwivedi, Anh Tuan Luu, Guy Wolf, and Dominique Beaini. Recipe for a general, powerful, scalable graph transformer. arXiv preprint arXiv:2205.12454, 2022.","title":"Recipe for a general, powerful, scalable graph transformer","authors":"Rampášek","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Rong et al., 2020] Yu Rong, Yatao Bian, Tingyang Xu, Weiyang Xie, Ying Wei, Wenbing Huang, and Junzhou Huang. Self-supervised graph transformer on large-scale molecular data. arXiv preprint arXiv:2007.02835, 2020.","title":"Self-supervised graph transformer on large-scale molecular data","authors":"Yu Rong, Yatao Bian, Tingyang Xu, Weiyang Xie, Ying Wei, Wenbing Huang, Junzhou Huang","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing sys- tems, pages 5998-6008, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Veličković et al., 2017] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv:1710.10903, 2017.","title":"Graph attention networks","authors":"Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, Yoshua Bengio","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wasserman et al., 1994] Stanley Wasserman, Katherine Faust, et al. Social network analysis: Methods and applications. 1994.","title":"Social network analysis: Methods and applications","authors":"Wasserman","year":"1994","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"Wu et al., ] Qitian Wu, Wentao Zhao, Zenan Li, David Wipf, and Junchi Yan. Nodeformer: A scalable graph structure learning transformer for node classification. In Advances in Neural Information Processing Systems.","title":"Nodeformer: A scalable graph structure learning transformer for node classification","authors":"Wu","year":null,"venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wu et al., 2018] Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S Pappu, Karl Leswing, and Vijay Pande. Moleculenet: a benchmark for molecular machine learning. Chemical sci- ence, 9(2):513-530, 2018.","title":"Moleculenet: a benchmark for molecular machine learning","authors":"Wu","year":"2018","venue":"Chemical science","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Xu et al., 2018] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural net- works? arXiv preprint arXiv:1810.00826, 2018.","title":"How powerful are graph neural networks","authors":"Xu","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Ying et al., 2021] Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie-Yan Liu. Do transformers really perform bad for graph representation? arXiv preprint arXiv:2106.05234, 2021.","title":"Do transformers really perform bad for graph representation","authors":"Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, Tie-Yan Liu","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[You et al., 2019] Jiaxuan You, Rex Ying, and Jure Leskovec. Position-aware graph neural networks. In International conference on machine learning, pages 7134-7143. PMLR, 2019.","title":"Position-aware graph neural networks","authors":"You","year":"2019","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhang et al., 2022] Zaixi Zhang, Qi Liu, Qingyong Hu, and Chee-Kong Lee. Hierarchical graph transformer with adaptive node sampling. arXiv preprint arXiv:2210.03930, 2022.","title":"Hierarchical graph transformer with adaptive node sampling","authors":"Zhang","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhao et al., 2021] Jianan Zhao, Chaozhuo Li, Qianlong Wen, Yiqi Wang, Yuming Liu, Hao Sun, Xing Xie, and Yanfang Ye. Gophormer: Ego-graph transformer for node classification. arXiv preprint arXiv:2110.13094, 2021.","title":"Gophormer: Ego-graph transformer for node classification","authors":"Jianan Zhao, Chaozhuo Li, Qianlong Wen, Yiqi Wang, Yuming Liu, Hao Sun, Xing Xie, Yanfang Ye","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}