{"type":"ScholarlyArticle","title":"LGI-GT: Graph Transformers with Local and Global Operators Interleaving","authors":[{"name":"Shuo Yin","slug":"shuo-yin","given":"Shuo","family":"Yin","affiliation":"College of Computer Science and Technology , Ocean University of China"},{"name":"Guoqiang Zhong","slug":"guoqiang-zhong","given":"Guoqiang","family":"Zhong","affiliation":"College of Computer Science and Technology , Ocean University of China"}],"abstract":"Since Transformers can alleviate some critical and fundamental problems of graph neural networks (GNNs), such as over-smoothing, over-squashing and limited expressiveness, they have been successfully applied to graph representation learning and achieved impressive results. However, although there are many works dedicated to make graph Transformers (GTs) aware of the structure and edge information by specifically tailored attention forms or graph-related positional and structural encodings, few works address the problem of how to construct high-performing GTs with modules of GNNs and Transformers. In this paper, we propose a novel graph Transformer with local and global operators interleaving (LGI-GT), in which we further design a new method propagating embeddings of the [CLS] token for global information representation. Additionally, we propose an effective message passing module called edge enhanced local attention (EELA), which makes LGI-GT a full-attention GT. Extensive experiments demonstrate that LGI-GT performs consistently better than previous state-of-the-art GNNs and GTs, while ablation studies show the effectiveness of the proposed LGI scheme and EELA. The source code of LGI-GT is available at https://github.com/shuoyinn/LGI-GT.","doi":"10.24963/ijcai.2023/501","url":"https://ijcai.org/proceedings/2023/501","pdf":"https://ijcai.org/proceedings/2023/0501.pdf","markdown":"https://ijcai.org/proceedings/2023/501.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/501","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Machine Learning","track":"Main Track","number":501,"pages":{"first":4504,"last":4512},"keywords":["Machine Learning: ML: Representation learning","Machine Learning: ML: Sequence and graph learning","Data Mining: DM: Mining graphs"],"videos":[],"references":[{"position":1,"raw":"[Alon and Yahav, 2021] Uri Alon and Eran Yahav. On the bottleneck of graph neural networks and its practical im- plications. In ICLR, 2021.","title":"On the bottleneck of graph neural networks and its practical implications","authors":"Alon, Yahav, Uri Alon, Eran Yahav","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Baek et al., 2021] Jinheon Baek, Minki Kang, and Sung Ju Hwang. Accurate learning of graph representations with graph multiset pooling. In ICLR, 2021.","title":"Accurate learning of graph representations with graph multiset pooling","authors":"Baek","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Beaini et al., 2021] Dominique Beaini, Saro Passaro, Vin- cent Létourneau, William L. Hamilton, Gabriele Corso, and Pietro Lió. Directional graph networks. In ICML, pages 748-758. PMLR, 2021.","title":"Directional graph networks","authors":"Dominique Beaini, Saro Passaro, Vincent Létourneau, William L Hamilton, Gabriele Corso, Pietro Lió","year":"2021","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2020] Deli Chen, Yankai Lin, Wei Li, Peng Li, Jie Zhou, and Xu Sun. Measuring and relieving the over- smoothing problem for graph neural networks from the topological view. In AAAI, 2020.","title":"Measuring and relieving the oversmoothing problem for graph neural networks from the topological view","authors":"Deli Chen, Yankai Lin, Wei Li, Peng Li, Jie Zhou, Xu Sun","year":"2020","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2022] Dexiong Chen, Leslie O'Bray, and Karsten Borgwardt. Structure-aware transformer for graph representation learning. In ICML, pages 3469-3489.","title":"Structure-aware transformer for graph representation learning","authors":"Chen","year":"2022","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"PMLR, 2022.","title":null,"authors":"Pmlr","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Choromanski et al., 2021] Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, An- dreea Gane, Tamás Sarlós, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, David Benjamin Belanger, Lucy J. Colwell, and Adrian Weller. Rethinking attention with performers. In ICLR, 2021.","title":"Rethinking attention with performers","authors":"Valerii Krzysztof Marcin Choromanski, David Likhosherstov, Xingyou Dohan, Andreea Song, Tamás Gane, Peter Sarlós, Jared Quincy Hawkins, Afroz Davis, Lukasz Mohiuddin, David Benjamin Kaiser, Lucy J Belanger, Adrian Colwell, Weller","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Corso et al., 2020] Gabriele Corso, Luca Cavalleri, Do- minique Beaini, Pietro Liò, and Petar Veličković. Prin- cipal neighbourhood aggregation for graph nets. NeurIPS, pages 13260-13271, 2020.","title":"Principal neighbourhood aggregation for graph nets","authors":"Gabriele Corso, Luca Cavalleri, Dominique Beaini, Pietro Liò, Petar Veličković","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Devlin et al., 2018] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint arXiv:1810.04805, 2018.","title":"Bert: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Dwivedi and Bresson, 2020] Vijay Prakash Dwivedi and Xavier Bresson. A generalization of transformer networks to graphs. CoRR, abs/2012.09699, 2020.","title":"A generalization of transformer networks to graphs","authors":"Vijay Prakash, Dwivedi, Xavier Bresson","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Dwivedi et al., 2020] Vijay Prakash Dwivedi, Chaitanya K. Joshi, Thomas Laurent, Yoshua Bengio, and Xavier Bres- son. Benchmarking graph neural networks. CoRR, abs/2003.00982, 2020.","title":"Benchmarking graph neural networks","authors":"Vijay Prakash Dwivedi, Chaitanya K Joshi, Thomas Laurent, Yoshua Bengio, Xavier Bresson","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Elman, 1990] Jeffrey L Elman. Finding structure in time. Cognitive science, 14(2):179-211, 1990.","title":"Finding structure in time","authors":"Jeffrey L Elman","year":"1990","venue":"Cognitive science","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Gilmer et al., 2017] Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In ICML, pages 1263-1272. PMLR, 2017.","title":"Neural message passing for quantum chemistry","authors":"Gilmer","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hamilton et al., 2017] William L. Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In NeurIPS, pages 1024-1034, 2017.","title":"Inductive representation learning on large graphs","authors":"William L Hamilton, Zhitao Ying, Jure Leskovec","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Hu et al., 2020a] Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. NeurIPS, pages 22118- 22133, 2020.","title":"Open graph benchmark: Datasets for machine learning on graphs","authors":"Hu","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Hu et al., 2020b] Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay S. Pande, and Jure Leskovec. Strategies for pre-training graph neural net- works. In ICLR, 2020.","title":"Strategies for pre-training graph neural networks","authors":"Hu","year":"2020","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Ivanov et al., 2019] Sergei Ivanov, Sergei Sviridov, and Evgeny Burnaev. Understanding isomorphism bias in graph data sets. CoRR, abs/1910.12091, 2019.","title":"Understanding isomorphism bias in graph data sets","authors":"Sergei Ivanov, Sergei Sviridov, Evgeny Burnaev","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Kipf and Welling, 2017] Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In ICLR, 2017.","title":"Semi-supervised classification with graph convolutional networks","authors":"Thomas N Kipf, Max Welling","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Klicpera et al., 2019] Johannes Klicpera, Aleksandar Bo- jchevski, and Stephan Günnemann. Predict then propa- gate: Graph neural networks meet personalized pagerank. In ICLR, 2019.","title":"Predict then propagate: Graph neural networks meet personalized pagerank","authors":"Klicpera","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Kreuzer et al., 2021] Devin Kreuzer, Dominique Beaini, William L. Hamilton, Vincent Létourneau, and Prudencio Tossou. Rethinking graph transformers with spectral at- tention. In NeurIPS, pages 21618-21629, 2021.","title":"Rethinking graph transformers with spectral attention","authors":"Devin Kreuzer, Dominique Beaini, William L Hamilton, Vincent Létourneau, Prudencio Tossou","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[LeCun et al., 1989] Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence D Jackel. Backpropagation ap- plied to handwritten zip code recognition. Neural compu- tation, 1(4):541-551, 1989.","title":"Backpropagation applied to handwritten zip code recognition","authors":"Yann Lecun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, Lawrence D Jackel","year":"1989","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Li et al., 2018] Qimai Li, Zhichao Han, and Xiao-Ming Wu. Deeper insights into graph convolutional networks for semi-supervised learning. In AAAI, 2018.","title":"Deeper insights into graph convolutional networks for semi-supervised learning","authors":"Li","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Morris et al., 2019] Christopher Morris, Martin Ritzert, Matthias Fey, William L. Hamilton, Jan Eric Lenssen, Gaurav Rattan, and Martin Grohe. Weisfeiler and leman go neural: Higher-order graph neural networks. In AAAI, 2019.","title":"Weisfeiler and leman go neural: Higher-order graph neural networks","authors":"Christopher Morris, Martin Ritzert, Matthias Fey, William L Hamilton, Jan Eric Lenssen, Gaurav Rattan, Martin Grohe","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Oono and Suzuki, 2020] Kenta Oono and Taiji Suzuki. Graph neural networks exponentially lose expressive power for node classification. In ICLR, 2020.","title":"Graph neural networks exponentially lose expressive power for node classification","authors":"Suzuki ; ] Kenta Oono, Taiji Oono, Suzuki","year":"2020","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Park et al., 2022] Wonpyo Park, Woong-Gi Chang, Donggeon Lee, Juntae Kim, and seung-won hwang. GRPE: Relative positional encoding for graph trans- former. In ICLR2022 Machine Learning for Drug Discovery, 2022.","title":"and seung-won hwang. GRPE: Relative positional encoding for graph transformer","authors":"Park","year":"2022","venue":"ICLR2022 Machine Learning for Drug Discovery","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Rampasek et al., 2022] Ladislav Rampasek, Mikhail Galkin, Vijay Prakash Dwivedi, Anh Tuan Luu, Guy Wolf, and Dominique Beaini. Recipe for a general, powerful, scalable graph transformer. In NeurIPS, 2022.","title":"Recipe for a general, powerful, scalable graph transformer","authors":"Ladislav Rampasek, Mikhail Galkin, Vijay Prakash Dwivedi, Anh Tuan Luu, Guy Wolf, Dominique Beaini","year":"2022","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Rong et al., 2020] Yu Rong, Yatao Bian, Tingyang Xu, Weiyang Xie, Ying Wei, Wenbing Huang, and Junzhou Huang. Self-supervised graph transformer on large-scale molecular data. In NeurIPS, 2020.","title":"Self-supervised graph transformer on large-scale molecular data","authors":"Yu Rong, Yatao Bian, Tingyang Xu, Weiyang Xie, Ying Wei, Wenbing Huang, Junzhou Huang","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Shi et al., 2021] Yunsheng Shi, Zhengjie Huang, Shikun Feng, Hui Zhong, Wenjing Wang, and Yu Sun. Masked label prediction: Unified message passing model for semi- supervised classification. In IJCAI, 2021.","title":"Masked label prediction: Unified message passing model for semisupervised classification","authors":"Yunsheng Shi, Zhengjie Huang, Shikun Feng, Hui Zhong, Wenjing Wang, Yu Sun","year":"2021","venue":"IJCAI","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Tönshoff et al., 2021] Jan Tönshoff, Martin Ritzert, Hin- rikus Wolf, and Martin Grohe. Graph learning with 1d convolutions on random walks. CoRR, abs/2102.08786, 2021.","title":"Graph learning with 1d convolutions on random walks","authors":"Tönshoff","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, pages 5998-6008, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, Illia Polosukhin","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Velickovic et al., 2018] Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In ICLR, 2018.","title":"Graph attention networks","authors":"Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, Yoshua Bengio","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Wale et al., 2008] Nikil Wale, Ian A. Watson, and George Karypis. Comparison of descriptor spaces for chemical compound retrieval and classification. Knowl. Inf. Syst., 14(3):347-375, 2008.","title":"Comparison of descriptor spaces for chemical compound retrieval and classification","authors":"Nikil Wale, Ian A Watson, George Karypis","year":"2008","venue":"Knowl. Inf. Syst","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Wijesinghe and Wang, 2022] Asiri Wijesinghe and Qing Wang. A new perspective on\" how graph neural networks go beyond weisfeiler-lehman?\". In ICLR, 2022.","title":"A new perspective on\" how graph neural networks go beyond weisfeiler-lehman?","authors":"Asiri Wijesinghe, Qing Wang","year":"2022","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Wu et al., 2021] Zhanghao Wu, Paras Jain, Matthew Wright, Azalia Mirhoseini, Joseph E Gonzalez, and Ion Stoica. Representing long-range context for graph neural networks with global attention. NeurIPS, pages 13266- 13279, 2021.","title":"Representing long-range context for graph neural networks with global attention","authors":"Wu","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Xu et al., 2019] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural net- works? In ICLR, 2019.","title":"How powerful are graph neural networks?","authors":"Keyulu Xu, Weihua Hu, Jure Leskovec, Stefanie Jegelka","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Ying et al., 2021] Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie-Yan Liu. Do transformers really perform badly for graph representation? In NeurIPS, pages 28877-28888, 2021.","title":"Do transformers really perform badly for graph representation?","authors":"Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, Tie-Yan Liu","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[You et al., 2021] Jiaxuan You, Jonathan M Gomes-Selman, Rex Ying, and Jure Leskovec. Identity-aware graph neural networks. In AAAI, 2021.","title":"Identity-aware graph neural networks","authors":"Jiaxuan You, Jonathan M Gomes-Selman, Rex Ying, Jure Leskovec","year":"2021","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Zaheer et al., 2020] Manzil Zaheer, Guru Guruganesh, Ku- mar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santi- ago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer se- quences. NeurIPS, pages 17283-17297, 2020.","title":"Big bird: Transformers for longer sequences","authors":"Manzil Zaheer, Guru Guruganesh, Avinava Kumar, Joshua Dubey, Chris Ainslie, Santiago Alberti, Philip Ontanon, Anirudh Pham, Qifan Ravula, Li Wang, Yang","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Zhang and Li, 2021] Muhan Zhang and Pan Li. Nested graph neural networks. In NeurIPS, pages 15734-15747, 2021.","title":"Nested graph neural networks","authors":"Li ; Zhang, Muhan Zhang, Pan Li","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Zhao et al., 2022] Lingxiao Zhao, Wei Jin, Leman Akoglu, and Neil Shah. From stars to subgraphs: Uplifting any GNN with local structure awareness. In ICLR, 2022.","title":"From stars to subgraphs: Uplifting any GNN with local structure awareness","authors":"Zhao","year":"2022","venue":"ICLR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}