{"type":"ScholarlyArticle","title":"Graph Propagation Transformer for Graph Representation Learning","authors":[{"name":"Zhe Chen","slug":"zhe-chen","given":"Zhe","family":"Chen","affiliation":"State Key Lab for Novel Software Technology , Nanjing Univerisity"},{"name":"Hao Tan","slug":"hao-tan","given":"Hao","family":"Tan","affiliation":"OPPO Research Institute"},{"name":"Tao Wang","slug":"tao-wang","given":"Tao","family":"Wang","affiliation":"State Key Lab for Novel Software Technology , Nanjing Univerisity"},{"name":"Tianrun Shen","slug":"tianrun-shen","given":"Tianrun","family":"Shen","affiliation":"State Key Lab for Novel Software Technology , Nanjing Univerisity"},{"name":"Tong Lu","slug":"tong-lu","given":"Tong","family":"Lu","affiliation":"State Key Lab for Novel Software Technology , Nanjing Univerisity"},{"name":"Qiuying Peng","slug":"qiuying-peng","given":"Qiuying","family":"Peng","affiliation":"OPPO Research Institute"},{"name":"Cheng Cheng","slug":"cheng-cheng","given":"Cheng","family":"Cheng","affiliation":"OPPO Research Institute"},{"name":"Yue Qi","slug":"yue-qi","given":"Yue","family":"Qi","affiliation":"OPPO Research Institute"}],"abstract":"This paper presents a novel transformer architecture for graph representation learning. The core insight of our method is to fully consider the information propagation among nodes and edges in a graph when building the attention module in the transformer blocks. Specifically, we propose a new attention mechanism called Graph Propagation Attention (GPA). It explicitly passes the information among nodes and edges in three ways, i.e. node-to-node, node-to-edge, and edge-to-node, which is essential for learning graph-structured data. On this basis, we design an effective transformer architecture named Graph Propagation Transformer (GPTrans) to further help learn graph data. We verify the performance of GPTrans in a wide range of graph learning experiments on several benchmark datasets. These results show that our method outperforms many state-of-the-art transformer-based graph models with better performance. The code will be released at https://github.com/czczup/GPTrans.","doi":"10.24963/ijcai.2023/396","url":"https://ijcai.org/proceedings/2023/396","pdf":"https://ijcai.org/proceedings/2023/0396.pdf","markdown":"https://ijcai.org/proceedings/2023/396.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/396","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Machine Learning","track":"Main Track","number":396,"pages":{"first":3559,"last":3567},"keywords":["Machine Learning: ML: Sequence and graph learning","Machine Learning: ML: Applications","Machine Learning: ML: Attention models"],"videos":[],"references":[{"position":1,"raw":"[Ba et al., 2016] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.","title":"Layer normalization","authors":"Jimmy Lei Ba, Jamie Ryan Kiros, Geoffrey E Hinton","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Beaini et al., 2021] Dominique Beaini, Saro Passaro, Vincent Létourneau, Will Hamilton, Gabriele Corso, and Pietro Liò. Di- rectional graph networks. In Proceedings of International Con- ference on Machine Learning, pages 748-758. PMLR, 2021.","title":"Directional graph networks","authors":"Dominique Beaini, Saro Passaro, Vincent Létourneau, Will Hamilton, Gabriele Corso, Pietro Liò","year":"2021","venue":"Proceedings of International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bresson and Laurent, 2017] Xavier Bresson and Thomas Lau- rent. Residual gated graph convnets. arXiv preprint arXiv:1711.07553, 2017.","title":"Residual gated graph convnets","authors":"Xavier Bresson, Thomas Laurent","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Brossard et al., 2020] Rémy Brossard, Oriel Frigo, and David De- haene. Graph convolutions that can finally model local structure. arXiv preprint arXiv:2011.15069, 2020.","title":"Graph convolutions that can finally model local structure","authors":"Rémy Brossard, Oriel Frigo, David Dehaene","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Brown et al., 2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural In- formation Processing Systems, 33:1877-1901, 2020.","title":"Language models are few-shot learners","authors":"Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Bruna et al., 2013] Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann LeCun. Spectral networks and locally connected net- works on graphs. arXiv preprint arXiv:1312.6203, 2013.","title":"Spectral networks and locally connected networks on graphs","authors":"Joan Bruna, Wojciech Zaremba, Arthur Szlam, Yann Lecun","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Cai and Lam, 2020] Deng Cai and Wai Lam. Graph transformer for graph-to-sequence learning. In Proceedings of the AAAI Con- ference on Artificial Intelligence, pages 7464-7471, 2020.","title":"Graph transformer for graph-to-sequence learning","authors":"Lam ; ] Deng Cai, Wai Cai, Lam","year":"2020","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chen et al., 2017] Jianfei Chen, Jun Zhu, and Le Song. Stochastic training of graph convolutional networks with variance reduction. arXiv preprint arXiv:1710.10568, 2017.","title":"Stochastic training of graph convolutional networks with variance reduction","authors":"Jianfei Chen, Jun Zhu, Le Song","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Chen et al., 2018] Jie Chen, Tengfei Ma, and Cao Xiao. Fastgcn: fast learning with graph convolutional networks via importance sampling. In Proceedings of International Conference on Learn- ing Representations, 2018.","title":"Fastgcn: fast learning with graph convolutional networks via importance sampling","authors":"Jie Chen, Tengfei Ma, Cao Xiao","year":"2018","venue":"Proceedings of International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Chen et al., 2020] Deli Chen, Yankai Lin, Wei Li, Peng Li, Jie Zhou, and Xu Sun. Measuring and relieving the over-smoothing problem for graph neural networks from the topological view. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 3438-3445, 2020.","title":"Measuring and relieving the over-smoothing problem for graph neural networks from the topological view","authors":"Deli Chen, Yankai Lin, Wei Li, Peng Li, Jie Zhou, Xu Sun","year":"2020","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Chen et al., 2022] Guo Chen, Sen Xing, Zhe Chen, Yi Wang, Kun- chang Li, Yizhuo Li, Yi Liu, Jiahao Wang, Yin-Dong Zheng, Bingkun Huang, et al. Internvideo-ego4d: A pack of champion solutions to ego4d challenges. arXiv preprint arXiv:2211.09529, 2022.","title":"Internvideo-ego4d: A pack of champion solutions to ego4d challenges","authors":"Guo Chen, Sen Xing, Zhe Chen, Yi Wang, Kunchang Li, Yizhuo Li, Yi Liu, Jiahao Wang, Yin-Dong Zheng, Bingkun Huang","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Chen et al., 2023] Zhe Chen, Yuchen Duan, Wenhai Wang, Jun- jun He, Tong Lu, Jifeng Dai, and Yu Qiao. Vision transformer adapter for dense predictions. In International Conference on Learning Representations, 2023.","title":"Vision transformer adapter for dense predictions","authors":"Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, Yu Qiao","year":"2023","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Corso et al., 2020] Gabriele Corso, Luca Cavalleri, Dominique Beaini, Pietro Liò, and Petar Veličković. Principal neighbour- hood aggregation for graph nets. Proceedings of Advances in Neural Information Processing Systems, 33:13260-13271, 2020.","title":"Principal neighbourhood aggregation for graph nets","authors":"Gabriele Corso, Luca Cavalleri, Dominique Beaini, Pietro Liò, Petar Veličković","year":"2020","venue":"Proceedings of Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Defferrard et al., 2016] Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. In Proceedings of Advances in Neural Information Processing Systems, 2016.","title":"Convolutional neural networks on graphs with fast localized spectral filtering","authors":"Defferrard","year":"2016","venue":"Proceedings of Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Dosovitskiy et al., 2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In Pro- ceedings of International Conference on Machine Learning, 2021.","title":"An image is worth 16x16 words: Transformers for image recognition at scale","authors":"Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly","year":"2021","venue":"Proceedings of International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Dwivedi and Bresson, 2020] Vijay Prakash Dwivedi and Xavier Bresson. A generalization of transformer networks to graphs. arXiv preprint arXiv:2012.09699, 2020.","title":"A generalization of transformer networks to graphs","authors":"Vijay Prakash, Dwivedi, Xavier Bresson","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Dwivedi et al., 2020] Vijay Prakash Dwivedi, Chaitanya K Joshi, Thomas Laurent, Yoshua Bengio, and Xavier Bresson. Benchmarking graph neural networks. arXiv preprint arXiv:2003.00982, 2020.","title":"Benchmarking graph neural networks","authors":"Vijay Prakash Dwivedi, K Chaitanya, Thomas Joshi, Yoshua Laurent, Xavier Bengio, Bresson","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Gilmer et al., 2017] Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In Proceedings of In- ternational Conference on Machine Learning, pages 1263-1272, 2017.","title":"Neural message passing for quantum chemistry","authors":"Justin Gilmer, S Samuel, Patrick F Schoenholz, Oriol Riley, George E Vinyals, Dahl","year":"2017","venue":"Proceedings of International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Hamilton et al., 2017] Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In Proceedings of Advances in Neural Information Processing Sys- tems, 2017.","title":"Inductive representation learning on large graphs","authors":"Hamilton","year":"2017","venue":"Proceedings of Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Henaff et al., 2015] Mikael Henaff, Joan Bruna, and Yann LeCun. Deep convolutional networks on graph-structured data. arXiv preprint arXiv:1506.05163, 2015.","title":"Deep convolutional networks on graph-structured data","authors":"Mikael Henaff, Joan Bruna, Yann Lecun","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Hu et al., 2020] Weihua Hu, Matthias Fey, Marinka Zitnik, Yux- iao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learn- ing on graphs. In Proceedings of Advances in Neural Information Processing Systems, 2020.","title":"Open graph benchmark: Datasets for machine learning on graphs","authors":"Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, Jure Leskovec","year":"2020","venue":"Proceedings of Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Hu et al., 2021] Weihua Hu, Matthias Fey, Hongyu Ren, Maho Nakata, Yuxiao Dong, and Jure Leskovec. Ogb-lsc: A large-scale challenge for machine learning on graphs. In Proceedings of Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2021.","title":"Ogb-lsc: A large-scale challenge for machine learning on graphs","authors":"Weihua Hu, Matthias Fey, Hongyu Ren, Maho Nakata, Yuxiao Dong, Jure Leskovec","year":"2021","venue":"Proceedings of Conference on Neural Information Processing Systems Datasets and Benchmarks Track","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Huang et al., 2022] Xuanwen Huang, Yang Yang, Yang Wang, Chunping Wang, Zhisheng Zhang, Jiarong Xu, and Lei Chen. Dgraph: A large-scale financial dataset for graph anomaly detec- tion. arXiv preprint arXiv:2207.03579, 2022.","title":"Dgraph: A large-scale financial dataset for graph anomaly detection","authors":"Xuanwen Huang, Yang Yang, Yang Wang, Chunping Wang, Zhisheng Zhang, Jiarong Xu, Lei Chen","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Hussain et al., 2021] Md Shamim Hussain, Mohammed J Zaki, and Dharmashankar Subramanian. Global self-attention as a replacement for graph convolution. arXiv preprint arXiv:2108.03348, 2021.","title":"Global self-attention as a replacement for graph convolution","authors":"Md Shamim Hussain, Mohammed J Zaki, Dharmashankar Subramanian","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Ji et al., 2023] Yuanfeng Ji, Zhe Chen, Enze Xie, Lanqing Hong, Xihui Liu, Zhaoqiang Liu, Tong Lu, Zhenguo Li, and Ping Luo. Ddp: Diffusion model for dense visual prediction. arXiv preprint arXiv:2303.17559, 2023.","title":"Ddp: Diffusion model for dense visual prediction","authors":"Yuanfeng Ji, Zhe Chen, Enze Xie, Lanqing Hong, Xihui Liu, Zhaoqiang Liu, Tong Lu, Zhenguo Li, Ping Luo","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Kim et al., 2022] Jinwoo Kim, Tien Dat Nguyen, Seonwoo Min, Sungjun Cho, Moontae Lee, Honglak Lee, and Seunghoon Hong. Pure transformers are powerful graph learners. arXiv preprint arXiv:2207.02505, 2022.","title":"Pure transformers are powerful graph learners","authors":"Jinwoo Kim, Tien Dat Nguyen, Seonwoo Min, Sungjun Cho, Moontae Lee, Honglak Lee, Seunghoon Hong","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Kipf and Welling, 2016] Thomas N Kipf and Max Welling. Semi- supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907, 2016.","title":"Semisupervised classification with graph convolutional networks","authors":"N Thomas, Max Kipf, Welling","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Kreuzer et al., 2021] Devin Kreuzer, Dominique Beaini, Will Hamilton, Vincent Létourneau, and Prudencio Tossou. Rethink- ing graph transformers with spectral attention. In Proceedings of Advances in Neural Information Processing Systems, pages 21618-21629, 2021.","title":"Rethinking graph transformers with spectral attention","authors":"Devin Kreuzer, Dominique Beaini, Will Hamilton, Vincent Létourneau, Prudencio Tossou","year":"2021","venue":"Proceedings of Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Le et al., 2021] Tuan Le, Marco Bertolini, Frank Noé, and Djork- Arné Clevert. Parameterized hypercomplex graph neural net- works for graph classification. In Proceedings of International Conference on Artificial Neural Networks, pages 204-216, 2021.","title":"Parameterized hypercomplex graph neural networks for graph classification","authors":"Tuan Le, Marco Bertolini, Frank Noé, Djork-Arné Clevert","year":"2021","venue":"Proceedings of International Conference on Artificial Neural Networks","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Li et al., 2020] Guohao Li, Chenxin Xiong, Ali Thabet, and Bernard Ghanem. Deepergcn: All you need to train deeper gcns. arXiv preprint arXiv:2006.07739, 2020.","title":"Deepergcn: All you need to train deeper gcns","authors":"Li","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Liu et al., 2021a] Meng Liu, Zhengyang Wang, and Shuiwang Ji. Non-local graph neural networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.","title":"Non-local graph neural networks","authors":"Liu","year":"2021","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Liu et al., 2021b] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin trans- former: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE International Conference on Com- puter Vision, pages 10012-10022, 2021.","title":"Swin transformer: Hierarchical vision transformer using shifted windows","authors":"Liu","year":"2021","venue":"Proceedings of the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Loshchilov and Hutter, 2018] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In Proceedings of In- ternational Conference on Learning Representations, 2018.","title":"Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization","authors":"Hutter Loshchilov","year":"2018","venue":"Proceedings of International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Park et al., 2022] Wonpyo Park, Woong-Gi Chang, Donggeon Lee, Juntae Kim, et al. Grpe: Relative positional encoding for graph transformer. In Proceedings of ICLR2022 Machine Learn- ing for Drug Discovery, 2022.","title":"Grpe: Relative positional encoding for graph transformer","authors":"Park","year":"2022","venue":"Proceedings of ICLR2022 Machine Learning for Drug Discovery","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Parmar et al., 2018] Niki Parmar, Ashish Vaswani, Jakob Uszko- reit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In Proceedings of the International Conference on Machine Learning, pages 4055-4064, 2018.","title":"Image transformer","authors":"Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, Dustin Tran","year":"2018","venue":"Proceedings of the International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Perozzi et al., 2014] Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. Deepwalk: Online learning of social representations. In Proceedings of ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 701-710, 2014.","title":"Deepwalk: Online learning of social representations","authors":"Bryan Perozzi, Rami Al-Rfou, Steven Skiena","year":"2014","venue":"Proceedings of ACM SIGKDD International Conference on Knowledge Discovery and Data Mining","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Radford et al., 2019] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.","title":"Language models are unsupervised multitask learners","authors":"Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever","year":"2019","venue":"OpenAI blog","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Rampášek et al., 2022] Ladislav Rampášek, Mikhail Galkin, Vi- jay Prakash Dwivedi, Anh Tuan Luu, Guy Wolf, and Dominique Beaini. Recipe for a general, powerful, scalable graph trans- former. arXiv preprint arXiv:2205.12454, 2022.","title":"Recipe for a general, powerful, scalable graph transformer","authors":"Ladislav Rampášek, Mikhail Galkin, Vijay Prakash Dwivedi, Anh Tuan Luu, Guy Wolf, Dominique Beaini","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Scarselli et al., 2008] Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE Transactions on Neural Net- works, 20(1):61-80, 2008.","title":"The graph neural network model","authors":"Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, Gabriele Monfardini","year":"2008","venue":"IEEE Transactions on Neural Networks","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Shi et al., 2022] Yu Shi, Shuxin Zheng, Guolin Ke, Yifei Shen, Ji- acheng You, Jiyan He, Shengjie Luo, Chang Liu, Di He, and Tie- Yan Liu. Benchmarking graphormer on large-scale molecular modeling datasets. arXiv preprint arXiv:2203.04810, 2022.","title":"Benchmarking graphormer on large-scale molecular modeling datasets","authors":"Yu Shi, Shuxin Zheng, Guolin Ke, Yifei Shen, Jiacheng You, Jiyan He, Shengjie Luo, Chang Liu, Di He, Tie-Yan Liu","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Touvron et al., 2021] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In Proceedings of International Conference on Machine Learning, pages 10347-10357, 2021.","title":"Training data-efficient image transformers & distillation through attention","authors":"Touvron","year":"2021","venue":"Proceedings of International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Par- mar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Pro- ceedings of Advances in Neural Information Processing Systems, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Proceedings of Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Veličković et al., 2017] Petar Veličković, Guillem Cucurull, Aran- txa Casanova, Adriana Romero, Pietro Lio, and Yoshua Ben- gio. Graph attention networks. arXiv preprint arXiv:1710.10903, 2017.","title":"Graph attention networks","authors":"Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, Yoshua Bengio","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":44,"raw":"[Wang et al., 2021] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In Proceedings of the IEEE International Conference on Computer Vision, pages 568- 578, 2021.","title":"Pyramid vision transformer: A versatile backbone for dense prediction without convolutions","authors":"Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, Ling Shao","year":"2021","venue":"Proceedings of the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":45,"raw":"[Wang et al., 2022a] Wenhai Wang, Jifeng Dai, Zhe Chen, Zhen- hang Huang, Zhiqi Li, Xizhou Zhu, Xiaowei Hu, Tong Lu, Lewei Lu, Hongsheng Li, et al. Internimage: Exploring large-scale vision foundation models with deformable convolutions. arXiv preprint arXiv:2211.05778, 2022.","title":"Internimage: Exploring large-scale vision foundation models with deformable convolutions","authors":"Wang","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":46,"raw":"[Wang et al., 2022b] Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, et al. Internvideo: General video foundation mod- els via generative and discriminative learning. arXiv preprint arXiv:2212.03191, 2022.","title":"Internvideo: General video foundation models via generative and discriminative learning","authors":"Wang","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":47,"raw":"[Xu et al., 2018] Keyulu Xu, Weihua Hu, Jure Leskovec, and Ste- fanie Jegelka. How powerful are graph neural networks? arXiv preprint arXiv:1810.00826, 2018.","title":"How powerful are graph neural networks","authors":"Keyulu Xu, Weihua Hu, Jure Leskovec, Stefanie Jegelka","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":48,"raw":"[Ying et al., 2021] Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie-Yan Liu. Do transformers really perform badly for graph represen- tation? In Proceedings of Advances in Neural Information Pro- cessing Systems, pages 28877-28888, 2021.","title":"Do transformers really perform badly for graph representation?","authors":"Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, Tie-Yan Liu","year":"2021","venue":"Proceedings of Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":49,"raw":"[Zhang et al., 2019] Chuxu Zhang, Ananthram Swami, and Nitesh V Chawla. Shne: Representation learning for semantic- associated heterogeneous networks. In Proceedings of ACM International Conference on Web Search and Data Mining, pages 690-698, 2019.","title":"Shne: Representation learning for semanticassociated heterogeneous networks","authors":"Zhang","year":"2019","venue":"Proceedings of ACM International Conference on Web Search and Data Mining","doi":null,"url":null,"internal_url":null},{"position":50,"raw":"[Zhang et al., 2020] Jiawei Zhang, Haopeng Zhang, Congying Xia, and Li Sun. Graph-bert: Only attention is needed for learning graph representations. arXiv preprint arXiv:2001.05140, 2020.","title":"Graph-bert: Only attention is needed for learning graph representations","authors":"Jiawei Zhang, Haopeng Zhang, Congying Xia, Li Sun","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}