{"type":"ScholarlyArticle","title":"Fine-Tuning Graph Neural Networks via Graph Topology Induced Optimal Transport","authors":[{"name":"Jiying Zhang","slug":"jiying-zhang","given":"Jiying","family":"Zhang","affiliation":"Tencent AI Lab; Shenzhen International Graduate School , Tsinghua University"},{"name":"Xi Xiao","slug":"xi-xiao","given":"Xi","family":"Xiao","affiliation":"Shenzhen International Graduate School , Tsinghua University"},{"name":"Long-Kai Huang","slug":"long-kai-huang","given":"Long-Kai","family":"Huang","affiliation":"Tencent AI Lab"},{"name":"Yu Rong","slug":"yu-rong","given":"Yu","family":"Rong","affiliation":"Tencent AI Lab"},{"name":"Yatao Bian","slug":"yatao-bian","given":"Yatao","family":"Bian","affiliation":"Tencent AI Lab"}],"abstract":"Recently, the pretrain-finetuning paradigm has attracted tons of attention in graph learning community due to its power of alleviating the lack of labels problem in many real-world applications. Current studies use existing techniques, such as weight constraint, representation constraint, which are derived from images or text data, to transfer the invariant knowledge from the pre-train stage to fine-tuning stage. However, these methods failed to preserve invariances from graph structure and Graph Neural Network (GNN) style models. In this paper, we present a novel optimal transport-based fine-tuning framework called GTOT-Tuning, namely, Graph Topology induced Optimal Transport fine-Tuning, for GNN style backbones. GTOT-Tuning is required to utilize the property of graph data to enhance the preservation of representation produced by fine-tuned networks. Toward this goal, we formulate graph local knowledge transfer as an Optimal Transport (OT) problem with a structural prior and construct the GTOT regularizer to constrain the fine-tuned model behaviors. By using the adjacency relationship amongst nodes, the GTOT regularizer achieves node-level optimal transport procedures and reduces redundant transport procedures, resulting in efficient knowledge transfer from the pre-trained models. We evaluate GTOT-Tuning on eight downstream tasks with various GNN backbones and demonstrate that it achieves state-of-the-art fine-tuning performance for GNNs.","doi":"10.24963/ijcai.2022/518","url":"https://ijcai.org/proceedings/2022/518","pdf":"https://ijcai.org/proceedings/2022/0518.pdf","markdown":"https://ijcai.org/proceedings/2022/518.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/518","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Machine Learning","track":"Main Track","number":518,"pages":{"first":3730,"last":3736},"keywords":["Data Mining: Mining Graphs","Machine Learning: Multi-task and Transfer Learning","Machine Learning: Sequence and Graph Learning"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986730","length_sec":84,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984873","length_sec":331,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Agrawal et al., 2014] Pulkit Agrawal, Ross Girshick, and Jitendra Malik. Analyzing the performance of multilayer neural networks for object recognition. In ECCV, 2014.","title":"Analyzing the performance of multilayer neural networks for object recognition","authors":"Pulkit Agrawal, Ross Girshick, Jitendra Malik","year":"2014","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bécigneul et al., 2020] Gary Bécigneul, Octavian-Eugen Ganea, and et al. Optimal transport graph neural networks. arXiv preprint arXiv:2006.04804, 2020.","title":"Optimal transport graph neural networks","authors":"Gary Bécigneul, Octavian-Eugen, Ganea","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Ben-David et al., 2007] Shai Ben-David, John Blitzer, Koby Crammer, Fernando Pereira, et al. Analysis of representations for domain adaptation. NeurIPS, 2007.","title":"Analysis of representations for domain adaptation","authors":"Shai Ben-David, John Blitzer, Koby Crammer, Fernando Pereira","year":"2007","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bousquet and Elisseeff, 2002] Olivier Bousquet and André Elisseeff. Stability and generalization. JMLR, 2002.","title":"Stability and generalization","authors":"Elisseeff Bousquet, Olivier Bousquet, André Elisseeff","year":"2002","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen and Gan, 2020] Liqun Chen and Zhe et al. Gan. Graph optimal transport for cross-domain alignment. In ICML, pages 1542-1553, 2020.","title":"Graph optimal transport for cross-domain alignment","authors":"Liqun Chen, Zhe","year":"2020","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen and Wang, 2021] Liqun Chen and Dong et al. Wang. Wasserstein contrastive representation distillation. In CVPR, pages 16296-16305, 2021.","title":"Wasserstein contrastive representation distillation","authors":"Liqun Chen, Dong","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chen and Zhang, 2019] Liqun Chen and Yizhe et al. Zhang. Improving sequence-to-sequence learning via optimal transport. In ICLR, 2019.","title":"Improving sequence-to-sequence learning via optimal transport","authors":"Chen, Zhang, Liqun Chen, Yizhe","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chen et al., 2019] Xinyang Chen, Sinan Wang, and et al. Catastrophic forgetting meets negative transfer: Batch spectral shrinkage for safe transfer learning. NeurIPS, 32:1908-1918, 2019.","title":"Catastrophic forgetting meets negative transfer: Batch spectral shrinkage for safe transfer learning","authors":"Chen","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Courty et al., 2016] Nicolas Courty, Rémi Flamary, Devis Tuia, and Alain Rakotomamonjy. Optimal transport for domain adaptation. TPAMI, 39(9):1853-1865, 2016.","title":"Optimal transport for domain adaptation","authors":"Nicolas Courty, Rémi Flamary, Devis Tuia, Alain Rakotomamonjy","year":"2016","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Donahue and Jia, 2014] Jeff Donahue and Yangqing et al. Jia. Decaf: A deep convolutional activation feature for generic visual recognition. In ICML, pages 647-655, 2014.","title":"Decaf: A deep convolutional activation feature for generic visual recognition","authors":"Jia Donahue, Jeff Donahue, Yangqing","year":"2014","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Ganea et al., 2022] Octavian-Eugen Ganea, Xinyuan Huang, and et al. Independent SE(3)-equivariant models for end-to-end rigid protein docking. In ICLR, 2022.","title":"Independent SE(3)-equivariant models for end-to-end rigid protein docking","authors":"Ganea","year":"2022","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[He et al., 2019] Kaiming He, Ross Girshick, and Piotr Dollár. Rethinking imagenet pre-training. In ICCV, pages 4918-4927, 2019.","title":"Rethinking imagenet pre-training","authors":"He","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hu et al., 2020] W Hu, B Liu, J Gomes, M Zitnik, P Liang, V Pande, and J Leskovec. Strategies for pre-training graph neural networks. In ICLR (ICLR), 2020.","title":"Strategies for pre-training graph neural networks","authors":"B Hu, J Liu, M Gomes, P Zitnik, V Liang, Pande, Leskovec","year":"2020","venue":"ICLR (ICLR)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Ji et al., 2022] Yuanfeng Ji, Lu Zhang, and et al. DrugOOD: Out-of-Distribution (OOD) Dataset Curator and Bench- mark for AI-aided Drug Discovery -A Focus on Affin- ity Prediction Problems with Noise Annotations. arXiv e-prints, page arXiv:2201.09637, January 2022.","title":"DrugOOD: Out-of-Distribution (OOD) Dataset Curator and Benchmark for AI-aided Drug Discovery -A Focus on Affinity Prediction Problems with Noise Annotations","authors":"Yuanfeng Ji, Lu Zhang","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kou et al., 2020] Zhi Kou, Kaichao You, and et al. Stochas- tic normalization. NeurIPS, 33, 2020.","title":"Stochastic normalization","authors":"Kou","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Li et al., 2018a] Qimai Li, Zhichao Han, and Xiao-Ming Wu. Deeper insights into graph convolutional networks for semi-supervised learning. In AAAI, 2018.","title":"Deeper insights into graph convolutional networks for semi-supervised learning","authors":"Li","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Li et al., 2018b] Xingjian Li, Haoyi Xiong, and et al. Delta: Deep learning transfer using feature map with attention for convolutional networks. In ICLR, 2018.","title":"Delta: Deep learning transfer using feature map with attention for convolutional networks","authors":"Li","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Li et al., 2020] Xuhong Li, Yves Grandvalet, Rémi Fla- mary, Nicolas Courty, and Dejing Dou. Represen- tation transfer by optimal transport. arXiv preprint arXiv:2007.06737, 2020.","title":"Representation transfer by optimal transport","authors":"Li","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Oquab et al., 2014] Maxime Oquab, Leon Bottou, Ivan Laptev, and Josef Sivic. Learning and transferring mid- level image representations using convolutional neural net- works. In CVPR, pages 1717-1724, 2014.","title":"Learning and transferring midlevel image representations using convolutional neural networks","authors":"Maxime Oquab, Leon Bottou, Ivan Laptev, Josef Sivic","year":"2014","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Peng et al., 2020] Zhen Peng, Wenbing Huang, and et al. Graph representation learning via graphical mutual infor- mation maximization. In The WebConf, 2020.","title":"Graph representation learning via graphical mutual information maximization","authors":"Zhen Peng, Wenbing Huang","year":"2020","venue":"The WebConf","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Peyré and Cuturi, 2019] Gabriel Peyré and Marco Cuturi. Computational optimal transport. Foundations and Trends in Machine Learning, 11 (5-6):355-602, 2019.","title":"Gabriel Peyré and Marco Cuturi. Computational optimal transport","authors":"Cuturi Peyré","year":"2019","venue":"Foundations and Trends in Machine Learning","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Rong et al., 2020] Yu Rong, Yatao Bian, and et al. Self- supervised graph transformer on large-scale molecular data. In NeurIPS, 2020.","title":"Selfsupervised graph transformer on large-scale molecular data","authors":"Yu Rong, Yatao Bian","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wu et al., 2018] Zhenqin Wu, Bharath Ramsundar, and et al. Moleculenet: a benchmark for molecular machine learning. Chemical science, 9(2):513-530, 2018.","title":"Moleculenet: a benchmark for molecular machine learning","authors":"Wu","year":"2018","venue":"Chemical science","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Xu and et al., 2020] Renjun Xu and et al. Reliable weighted optimal transport for unsupervised domain adaptation. In CVPR, pages 4394-4403, 2020.","title":"Reliable weighted optimal transport for unsupervised domain adaptation","authors":"Xu","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Xu and et al., 2021] Minghao Xu and et al. Self-supervised graph-level representation learning with local and global structure. ICML, 2021.","title":"Self-supervised graph-level representation learning with local and global structure","authors":"Xu","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Xu et al., 2018] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural net- works? In ICLR, 2018.","title":"How powerful are graph neural networks?","authors":"Xu","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Xu et al., 2019] Hongteng Xu, Dixin Luo, and et al. Gromov-wasserstein learning for graph matching and node embedding. In ICML. PMLR, 2019.","title":"Gromov-wasserstein learning for graph matching and node embedding","authors":"Xu","year":"2019","venue":"ICML. PMLR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Xuhong et al., 2018] LI Xuhong, Yves Grandvalet, and Franck Davoine. Explicit inductive bias for transfer learn- ing with convolutional networks. In ICML, 2018.","title":"Explicit inductive bias for transfer learning with convolutional networks","authors":"Yves Li Xuhong, Franck Grandvalet, Davoine","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Yosinski et al., 2014] Jason Yosinski, Jeff Clune, and et al. How transferable are features in deep neural networks? NeurIPS, 27:3320-3328, 2014.","title":"How transferable are features in deep neural networks","authors":"Jason Yosinski, Jeff Clune","year":"2014","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[You et al., 2020] Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph con- trastive learning with augmentations. NeurIPS, 2020.","title":"Graph contrastive learning with augmentations","authors":"Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, Yang Shen","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhang et al., 2022] Jiying Zhang, Fuyang Li, Xi Xiao, Tingyang Xu, Yu Rong, Junzhou Huang, and Yatao Bian. Hypergraph convolutional networks via equivalency be- tween hypergraphs and undirected graphs. ICML TAG-ML Workshop, 2022.","title":"Hypergraph convolutional networks via equivalency between hypergraphs and undirected graphs","authors":"Zhang","year":"2022","venue":"ICML TAG-ML Workshop","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}