{"type":"ScholarlyArticle","title":"Gradformer: Graph Transformer with Exponential Decay","authors":[{"name":"Chuang Liu","slug":"chuang-liu","given":"Chuang","family":"Liu","affiliation":"School of Computer Science , Wuhan University , Wuhan , China"},{"name":"Zelin Yao","slug":"zelin-yao","given":"Zelin","family":"Yao","affiliation":"School of Computer Science , Wuhan University , Wuhan , China"},{"name":"Yibing Zhan","slug":"yibing-zhan","given":"Yibing","family":"Zhan","affiliation":"JD Explore Academy , JD .com , China"},{"name":"Xueqi Ma","slug":"xueqi-ma","given":"Xueqi","family":"Ma","affiliation":"School of Computing and Information Systems , The University of Melbourne , Melbourne , Australia"},{"name":"Shirui Pan","slug":"shirui-pan","given":"Shirui","family":"Pan","affiliation":"School of Information and Communication Technology , Griffith University , Brisbane , Australia"},{"name":"Wenbin Hu","slug":"wenbin-hu","given":"Wenbin","family":"Hu","affiliation":"School of Computer Science , Wuhan University , Wuhan , China"}],"abstract":"Graph Transformers (GTs) have demonstrated their advantages across a wide range of tasks. However, the self-attention mechanism in GTs overlooks the graph's inductive biases, particularly biases related to structure, which are crucial for the graph tasks. Although some methods utilize positional encoding and attention bias to model inductive biases, their effectiveness is still suboptimal analytically. Therefore, this paper presents Gradformer, a method innovatively integrating GT with the intrinsic inductive bias by applying an exponential decay mask to the attention matrix. Specifically, the values in the decay mask matrix diminish exponentially, correlating with the decreasing node proximities within the graph structure. This design enables Gradformer to retain its ability to capture information from distant nodes while focusing on the graph's local details. Furthermore, Gradformer introduces a learnable constraint into the decay mask, allowing different attention heads to learn distinct decay masks. Such an design diversifies the attention heads, enabling a more effective assimilation of diverse structural information within the graph. Extensive experiments on various benchmarks demonstrate that Gradformer consistently outperforms the Graph Neural Network and GT baseline models in various graph classification and regression tasks. Additionally, Gradformer has proven to be an effective method for training deep GT models, maintaining or even enhancing accuracy compared to shallow models as the network deepens, in contrast to the significant accuracy drop observed in other GT models. Codes are available at https://github.com/LiuChuang0059/Gradformer.","doi":"10.24963/ijcai.2024/240","url":"https://ijcai.org/proceedings/2024/240","pdf":"https://ijcai.org/proceedings/2024/0240.pdf","markdown":"https://ijcai.org/proceedings/2024/240.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/240","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Data Mining","track":"Main Track","number":240,"pages":{"first":2171,"last":2179},"keywords":["Data Mining: DM: Mining graphs","Machine Learning: ML: Classification"],"videos":[],"references":[{"position":1,"raw":"[Cai et al., 2022] Weishan Cai, Wenjun Ma, Jieyu Zhan, and Yuncheng Jiang. Entity alignment with reliable path reasoning and relation-aware heterogeneous graph trans- former. In IJCAI, 2022.","title":"Entity alignment with reliable path reasoning and relation-aware heterogeneous graph transformer","authors":"Weishan Cai, Wenjun Ma, Jieyu Zhan, Yuncheng Jiang","year":"2022","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2022/268"},{"position":2,"raw":"[Chen and Li, 2022] Sijia Chen and Baochun Li. Multi- modal dynamic graph transformer for visual grounding. In CVPR, 2022.","title":"Multimodal dynamic graph transformer for visual grounding","authors":"Chen, Li ; ] Sijia Chen, Baochun Li","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2021] Jianwen Chen, Shuangjia Zheng, Ying Song, Jiahua Rao, and Yuedong Yang. Learning attributed graph representation with communicative message passing transformer. In IJCAI, 2021.","title":"Learning attributed graph representation with communicative message passing transformer","authors":"Jianwen Chen, Shuangjia Zheng, Ying Song, Jiahua Rao, Yuedong Yang","year":"2021","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2021/309"},{"position":4,"raw":"[Chen et al., 2022] Dexiong Chen, Leslie O'Bray, and Karsten Borgwardt. Structure-aware transformer for graph representation learning. In ICML, 2022.","title":"Structure-aware transformer for graph representation learning","authors":"Chen","year":"2022","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Choromanski et al., 2022] Krzysztof Choromanski, Han Lin, Haoxian Chen, Tianyi Zhang, Arijit Sehanobish, Va- lerii Likhosherstov, Jack Parker-Holder, Tamas Sarlos, Adrian Weller, and Thomas Weingarten. From block- toeplitz matrices to differential equations on graphs: to- wards a general theory for scalable masked transformers. In ICML, 2022.","title":"From blocktoeplitz matrices to differential equations on graphs: towards a general theory for scalable masked transformers","authors":"Krzysztof Choromanski, Han Lin, Haoxian Chen, Tianyi Zhang, Arijit Sehanobish, Valerii Likhosherstov, Jack Parker-Holder, Tamas Sarlos, Adrian Weller, Thomas Weingarten","year":"2022","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Diao and Loynd, 2023] Cameron Diao and Ricky Loynd. Relational attention: Generalizing transformers for graph- structured tasks. In ICLR, 2023.","title":"Relational attention: Generalizing transformers for graphstructured tasks","authors":"Cameron Diao, Ricky Loynd","year":"2023","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Ding et al., 2020] Liang Ding, Longyue Wang, and Dacheng Tao. Self-attention with cross-lingual position representation. In ACL, 2020.","title":"Self-attention with cross-lingual position representation","authors":"Liang Ding, Longyue Wang, Dacheng Tao","year":"2020","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dwivedi and Bresson, 2021] Vijay Prakash Dwivedi and Xavier Bresson. A generalization of transformer networks to graphs. In AAAI Workshop, 2021.","title":"A generalization of transformer networks to graphs","authors":"Vijay Prakash, Dwivedi, Xavier Bresson","year":"2021","venue":"AAAI Workshop","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Dwivedi et al., 2023] Vijay Prakash Dwivedi, Chaitanya K Joshi, Anh Tuan Luu, Thomas Laurent, Yoshua Bengio, and Xavier Bresson. Benchmarking graph neural net- works. Journal of Machine Learning Research, 2023.","title":"Benchmarking graph neural networks","authors":"Vijay Prakash Dwivedi, K Chaitanya, Anh Tuan Joshi, Thomas Luu, Yoshua Laurent, Xavier Bengio, Bresson","year":"2023","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Guo et al., 2022] Lingbing Guo, Qiang Zhang, and Huajun Chen. Unleashing the power of transformer for graphs. arXiv:2202.10581, 2022.","title":"Unleashing the power of transformer for graphs","authors":"Guo","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[He and Wang, 2023] Xuehai He and Xin Wang. Multimodal graph transformer for multimodal question answering. In ACL, 2023.","title":"Xuehai He and Xin Wang. Multimodal graph transformer for multimodal question answering","authors":"Wang He","year":"2023","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hu et al., 2020] Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. arXiv:2005.00687, 2020.","title":"Open graph benchmark: Datasets for machine learning on graphs","authors":"Hu","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hu et al., 2023] Shengchao Hu, Li Shen, Ya Zhang, and Dacheng Tao. Graph decision transformer. arXiv preprint arXiv:2303.03747, 2023.","title":"Graph decision transformer","authors":"Shengchao Hu, Li Shen, Ya Zhang, Dacheng Tao","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hussain et al., 2022] Md Shamim Hussain, Mohammed J. Zaki, and Dharmashankar Subramanian. Global self- attention as a replacement for graph convolution. In SIGKDD, 2022.","title":"Global selfattention as a replacement for graph convolution","authors":"Md Shamim Hussain, Mohammed J Zaki, Dharmashankar Subramanian","year":"2022","venue":"SIGKDD","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kipf and Welling, 2017] Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In ICLR, 2017.","title":"Semi-supervised classification with graph convolutional networks","authors":"Thomas N Kipf, Max Welling","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kreuzer et al., 2021] Devin Kreuzer, Dominique Beaini, William L. Hamilton, Vincent Létourneau, and Prudencio Tossou. Rethinking graph transformers with spectral at- tention. In NeurIPS, 2021.","title":"Rethinking graph transformers with spectral attention","authors":"Devin Kreuzer, Dominique Beaini, William L Hamilton, Vincent Létourneau, Prudencio Tossou","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lai et al., 2023] Xin Lai, Yang Liu, Rui Qian, Yong Lin, and Qiwei Ye. Deeper exploiting graph structure infor- mation by discrete ricci curvature in a graph transformer. Entropy, 2023.","title":"Deeper exploiting graph structure information by discrete ricci curvature in a graph transformer","authors":"Xin Lai, Yang Liu, Rui Qian, Yong Lin, Qiwei Ye","year":"2023","venue":"Entropy","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Li et al., 2016] Yujia Li, Richard Zemel, Marc Brockschmidt, and Daniel Tarlow. Gated graph se- quence neural networks. In ICLR, 2016.","title":"Gated graph sequence neural networks","authors":"Yujia Li, Richard Zemel, Marc Brockschmidt, Daniel Tarlow","year":"2016","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Li et al., 2023] Chaoliu Li, Lianghao Xia, Xubin Ren, Yaowen Ye, Yong Xu, and Chao Huang. Graph trans- former for recommendation. In SIGIR, 2023.","title":"Graph transformer for recommendation","authors":"Li","year":"2023","venue":"SIGIR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Liu et al., 2023] Chuang Liu, Yibing Zhan, Xueqi Ma, Liang Ding, Dapeng Tao, Jia Wu, and Wenbin Hu. Gap- former: Graph transformer with graph pooling for node classification. In IJCAI, 2023.","title":"Gapformer: Graph transformer with graph pooling for node classification","authors":"Chuang Liu, Yibing Zhan, Xueqi Ma, Liang Ding, Dapeng Tao, Jia Wu, Wenbin Hu","year":"2023","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2023/244"},{"position":21,"raw":"[Liu et al., 2024] Chuang Liu, Yibing Zhan, Xueqi Ma, Liang Ding, Dapeng Tao, Jia Wu, Wenbin Hu, and Bo Du. Exploring sparsity in graph transformers. Neural Net- works, 2024.","title":"Exploring sparsity in graph transformers","authors":"Chuang Liu, Yibing Zhan, Xueqi Ma, Liang Ding, Dapeng Tao, Jia Wu, Wenbin Hu, Bo Du","year":"2024","venue":"Neural Networks","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Min et al., 2022] Erxue Min, Runfa Chen, Yatao Bian, Tingyang Xu, Kangfei Zhao, Wenbing Huang, Peilin Zhao, Junzhou Huang, Sophia Ananiadou, and Yu Rong. Transformer for graphs: An overview from architecture perspective. arXiv:2202.08455, 2022.","title":"Transformer for graphs: An overview from architecture perspective","authors":"Min","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Morris et al., 2020] Christopher Morris, Nils M Kriege, Franka Bause, Kristian Kersting, Petra Mutzel, and Mar- ion Neumann. Tudataset: A collection of benchmark datasets for learning with graphs. arXiv:2007.08663, 2020.","title":"Tudataset: A collection of benchmark datasets for learning with graphs","authors":"Christopher Morris, Nils M Kriege, Franka Bause, Kristian Kersting, Petra Mutzel, Marion Neumann","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Park et al., 2022] Jinyoung Park, Seongjun Yun, Hyeonjin Park, Jaewoo Kang, Jisu Jeong, Kyung-Min Kim, Jung- woo Ha, and Hyunwoo J Kim. Deformable graph trans- former. arXiv:2206.14337, 2022.","title":"Deformable graph transformer","authors":"Jinyoung Park, Seongjun Yun, Hyeonjin Park, Jaewoo Kang, Jisu Jeong, Kyung-Min Kim, Jungwoo Ha, Hyunwoo J Kim","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Rampasek et al., 2022] Ladislav Rampasek, Mikhail Galkin, Vijay Prakash Dwivedi, Anh Tuan Luu, Guy Wolf, and Dominique Beaini. Recipe for a general, powerful, scalable graph transformer. In NeurIPS, 2022.","title":"Recipe for a general, powerful, scalable graph transformer","authors":"Ladislav Rampasek, Mikhail Galkin, Vijay Prakash Dwivedi, Anh Tuan Luu, Guy Wolf, Dominique Beaini","year":"2022","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Rong et al., 2020] Yu Rong, Yatao Bian, Tingyang Xu, Weiyang Xie, Ying WEI, Wenbing Huang, and Junzhou Huang. Self-supervised graph transformer on large-scale molecular data. In NeurIPS, 2020.","title":"Self-supervised graph transformer on large-scale molecular data","authors":"Yu Rong, Yatao Bian, Tingyang Xu, Weiyang Xie, Wei Ying, Wenbing Huang, Junzhou Huang","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Sun et al., 2023] Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor to transformer for large language models. arXiv preprint arXiv:2307.08621, 2023.","title":"Retentive network: A successor to transformer for large language models","authors":"Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, Furu Wei","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Tang et al., 2023] Hao Tang, Zhenyu Zhang, Humphrey Shi, Bo Li, Ling Shao, Nicu Sebe, Radu Timofte, and Luc Van Gool. Graph transformer gans for graph-constrained house generation. In CVPR, 2023.","title":"Graph transformer gans for graph-constrained house generation","authors":"Hao Tang, Zhenyu Zhang, Humphrey Shi, Bo Li, Ling Shao, Nicu Sebe, Radu Timofte, Luc Van Gool","year":"2023","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł Ukasz Kaiser, Illia Polosukhin","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Veličković et al., 2018] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In ICLR, 2018.","title":"Graph attention networks","authors":"Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, Yoshua Bengio","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Wu et al., 2021] Zhanghao Wu, Paras Jain, Matthew Wright, Azalia Mirhoseini, Joseph E Gonzalez, and Ion Stoica. Representing long-range context for graph neural networks with global attention. In NeurIPS, 2021.","title":"Representing long-range context for graph neural networks with global attention","authors":"Wu","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Wu et al., 2022] Qitian Wu, Wentao Zhao, Zenan Li, David Wipf, and Junchi Yan. Nodeformer: A scalable graph structure learning transformer for node classification. In NeurIPS, 2022.","title":"Nodeformer: A scalable graph structure learning transformer for node classification","authors":"Qitian Wu, Wentao Zhao, Zenan Li, David Wipf, Junchi Yan","year":"2022","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Wu et al., 2023a] Qitian Wu, Chenxiao Yang, Wentao Zhao, Yixuan He, David Wipf, and Junchi Yan. DIFFormer: Scalable (graph) transformers induced by energy con- strained diffusion. In ICLR, 2023.","title":"DIFFormer: Scalable (graph) transformers induced by energy constrained diffusion","authors":"Wu","year":"2023","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Wu et al., 2023b] Yi Wu, Yanyang Xu, Wenhao Zhu, Guo- jie Song, Zhouchen Lin, Liang Wang, and Shaoguo Liu. Kdlgt: A linear graph transformer framework via kernel decomposition approach. In IJCAI, 2023.","title":"Kdlgt: A linear graph transformer framework via kernel decomposition approach","authors":"Wu","year":"2023","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2023/263"},{"position":35,"raw":"[Xu et al., 2019a] Chengfeng Xu, Pengpeng Zhao, Yanchi Liu, Victor S. Sheng, Jiajie Xu, Fuzhen Zhuang, Junhua Fang, and Xiaofang Zhou. Graph contextualized self- attention network for session-based recommendation. In IJCAI, 2019.","title":"Graph contextualized selfattention network for session-based recommendation","authors":"Xu","year":"2019","venue":"IJCAI","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Xu et al., 2019b] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural net- works? In ICLR, 2019.","title":"How powerful are graph neural networks?","authors":"Xu","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Yin and Zhong, 2023] Shuo Yin and Guoqiang Zhong. Lgi- gt: Graph transformers with local and global operators in- terleaving. In IJCAI, 2023.","title":"Lgigt: Graph transformers with local and global operators interleaving","authors":"Shuo Yin, Guoqiang Zhong","year":"2023","venue":"IJCAI","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Ying et al., 2021] Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie-Yan Liu. Do transformers really perform badly for graph representation? In NeurIPS, 2021.","title":"Do transformers really perform badly for graph representation?","authors":"Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, Tie-Yan Liu","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Zhao et al., 2021] Jianan Zhao, Chaozhuo Li, Qianlong Wen, Yiqi Wang, Yuming Liu, Hao Sun, Xing Xie, and Yanfang Ye. Gophormer: Ego-graph transformer for node classification. arXiv:2110.13094, 2021.","title":"Gophormer: Ego-graph transformer for node classification","authors":"Jianan Zhao, Chaozhuo Li, Qianlong Wen, Yiqi Wang, Yuming Liu, Hao Sun, Xing Xie, Yanfang Ye","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Zhao et al., 2023] Haiteng Zhao, Shuming Ma, Dongdong Zhang, Zhi-Hong Deng, and Furu Wei. Are more layers beneficial to graph transformers? In ICLR, 2023.","title":"Are more layers beneficial to graph transformers?","authors":"Zhao","year":"2023","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Zhu et al., 2021] Yiran Zhu, Xing Xu, Fumin Shen, Yanli Ji, Lianli Gao, and Heng Tao Shen. Posegtac: Graph trans- former encoder-decoder with atrous convolution for 3d hu- man pose estimation. In IJCAI, 2021.","title":"Posegtac: Graph transformer encoder-decoder with atrous convolution for 3d human pose estimation","authors":"Yiran Zhu, Xing Xu, Fumin Shen, Yanli Ji, Lianli Gao, Heng Tao Shen","year":"2021","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2021/188"}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}