{"type":"ScholarlyArticle","title":"Rethinking the Setting of Semi-supervised Learning on Graphs","authors":[{"name":"Ziang Li","slug":"ziang-li","given":"Ziang","family":"Li","affiliation":"Department of Computer Science and Technology , Tsinghua University {li-za19 ,"},{"name":"Ming Ding","slug":"ming-ding","given":"Ming","family":"Ding","affiliation":"Department of Computer Science and Technology , Tsinghua University {li-za19 ,"},{"name":"Weikai Li","slug":"weikai-li","given":"Weikai","family":"Li","affiliation":"Department of Computer Science and Technology , Tsinghua University {li-za19 ,"},{"name":"Zihan Wang","slug":"zihan-wang","given":"Zihan","family":"Wang","affiliation":"Department of Computer Science and Technology , Tsinghua University {li-za19 ,"},{"name":"Ziyu Zeng","slug":"ziyu-zeng","given":"Ziyu","family":"Zeng","affiliation":"Department of Computer Science and Technology , Tsinghua University {li-za19 ,"},{"name":"Yukuo Cen","slug":"yukuo-cen","given":"Yukuo","family":"Cen","affiliation":"Department of Computer Science and Technology , Tsinghua University {li-za19 ,"},{"name":"Jie Tang","slug":"jie-tang","given":"Jie","family":"Tang","affiliation":"Department of Computer Science and Technology , Tsinghua University {li-za19 ,"}],"abstract":"We argue that the present setting of semisupervised learning on graphs may result in unfair comparisons, due to its potential risk of over-tuning hyper-parameters for models. In this paper, we highlight the significant influence of tuning hyper-parameters, which leverages the label information in the validation set to improve the performance. To explore the limit of over-tuning hyperparameters, we propose ValidUtil, an approach to fully utilize the label information in the validation set through an extra group of hyper-parameters. With ValidUtil, even GCN can easily get high accuracy of 85.8% on Cora.\r\nTo avoid over-tuning, we merge the training set and the validation set and construct an i.i.d. graph benchmark (IGB) consisting of 4 datasets. Each dataset contains 100 i.i.d. graphs sampled from a large graph to reduce the evaluation variance. Our experiments suggest that IGB is a more stable benchmark than previous datasets for semisupervised learning on graphs. Our code and data are released at https://github.com/THUDM/IGB/.","doi":"10.24963/ijcai.2022/450","url":"https://ijcai.org/proceedings/2022/450","pdf":"https://ijcai.org/proceedings/2022/0450.pdf","markdown":"https://ijcai.org/proceedings/2022/450.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/450","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Machine Learning","track":"Main Track","number":450,"pages":{"first":3243,"last":3249},"keywords":["Machine Learning: Semi-Supervised Learning","Machine Learning: Sequence and Graph Learning","Multidisciplinary Topics and Applications: Web  and Social Networks"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986843","length_sec":85,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984986","length_sec":363,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Abu-El-Haija et al., 2019] Sami Abu-El-Haija, Bryan Per- ozzi, Amol Kapoor, Hrayr Harutyunyan, Nazanin Alipour- fard, Kristina Lerman, Greg Ver Steeg, and Aram Gal- styan. Mixhop: Higher-order graph convolutional archi- tectures via sparsified neighborhood mixing. In ICML, 2019.","title":"Mixhop: Higher-order graph convolutional architectures via sparsified neighborhood mixing","authors":"Sami Abu-El-Haija, Bryan Perozzi, Amol Kapoor, Hrayr Harutyunyan, Nazanin Alipourfard, Kristina Lerman, Greg Ver Steeg, Aram Galstyan","year":"2019","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Carlson et al., 2010] Andrew Carlson, Justin Betteridge, Bryan Kisiel, Burr Settles, Estevam R. Hruschka, and Tom Michael Mitchell. Toward an architecture for never- ending language learning. In AAAI, 2010.","title":"Toward an architecture for neverending language learning","authors":"Andrew Carlson, Justin Betteridge, Bryan Kisiel, Burr Settles, Estevam R Hruschka, Tom Michael, Mitchell","year":"2010","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cen et al., 2021] Yukuo Cen, Zhenyu Hou, Yan Wang, Qibin Chen, Yizhen Luo, Xingcheng Yao, Aohan Zeng, Shiguang Guo, Peng Zhang, Guohao Dai, et al. Cogdl: An extensive toolkit for deep learning on graphs. arXiv preprint arXiv:2103.00959, 2021.","title":"Cogdl: An extensive toolkit for deep learning on graphs","authors":"Cen","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2020] Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. Simple and deep graph convolutional networks. In ICML, 2020.","title":"Simple and deep graph convolutional networks","authors":"Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, Yaliang Li","year":"2020","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Feng et al., 2020] Wenzheng Feng, Jie Zhang, Yuxiao Dong, Yu Han, Huanbo Luan, Qian Xu, Qiang Yang, and Jie Tang. Graph random neural network for semi- supervised learning on graphs. In NeurIPS, 2020.","title":"Graph random neural network for semisupervised learning on graphs","authors":"Wenzheng Feng, Jie Zhang, Yuxiao Dong, Yu Han, Huanbo Luan, Qian Xu, Qiang Yang, Jie Tang","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fey and Lenssen, 2019] Matthias Fey and Jan E. Lenssen. Fast graph representation learning with PyTorch Geomet- ric. In ICLR Workshop on Representation Learning on Graphs and Manifolds, 2019.","title":"Fast graph representation learning with PyTorch Geometric","authors":"Matthias Fey, Jan E Lenssen","year":"2019","venue":"ICLR Workshop on Representation Learning on Graphs and Manifolds","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gori et al., 2005] Marco Gori, Gabriele Monfardini, and Franco Scarselli. A new model for learning in graph do- mains. In IJCNN, volume 2, pages 729-734. IEEE, 2005.","title":"A new model for learning in graph domains","authors":"Marco Gori, Gabriele Monfardini, Franco Scarselli","year":"2005","venue":"IJCNN","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hamilton et al., 2017] William L. Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In NeurIPS, 2017.","title":"Inductive representation learning on large graphs","authors":"William L Hamilton, Zhitao Ying, Jure Leskovec","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hu et al., 2020] Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. ArXiv, abs/2005.00687, 2020.","title":"Open graph benchmark: Datasets for machine learning on graphs","authors":"Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, Jure Leskovec","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Huang et al., 2020] Qian Huang, Horace He, Abhay Singh, Ser-Nam Lim, and Austin R Benson. Combining label propagation and simple models out-performs graph neural networks. arXiv preprint arXiv:2010.13993, 2020.","title":"Combining label propagation and simple models out-performs graph neural networks","authors":"Qian Huang, Horace He, Abhay Singh, Ser-Nam Lim, Austin R Benson","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kingma and Ba, 2014] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kipf and Welling, 2016] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907, 2016.","title":"Semi-supervised classification with graph convolutional networks","authors":"N Thomas, Max Kipf, Welling","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Klicpera et al., 2019a] Johannes Klicpera, Aleksandar Bo- jchevski, and Stephan Günnemann. Predict then propa- gate: Graph neural networks meet personalized pagerank. In ICLR, 2019.","title":"Predict then propagate: Graph neural networks meet personalized pagerank","authors":"Klicpera","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Klicpera et al., 2019b] Johannes Klicpera, Stefan Weißen- berger, and Stephan Günnemann. Diffusion improves graph learning. In NeurIPS, 2019.","title":"Diffusion improves graph learning","authors":"Klicpera","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lv et al., 2021] Qingsong Lv, Ming Ding, Qiang Liu, Yuxi- ang Chen, Wenzheng Feng, Siming He, Chang Zhou, Jian- guo Jiang, Yuxiao Dong, and Jie Tang. Are we really mak- ing much progress? revisiting, benchmarking and refining heterogeneous graph neural networks. In KDD, 2021.","title":"Are we really making much progress? revisiting, benchmarking and refining heterogeneous graph neural networks","authors":"Lv","year":"2021","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Rozemberczki et al., 2019] Benedek Rozemberczki, Carl Allen, and Rik Sarkar. Multi-scale attributed node em- bedding. CoRR, abs/1909.13021, 2019.","title":"Multi-scale attributed node embedding","authors":"Carl Benedek Rozemberczki, Rik Allen, Sarkar","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Sen et al., 2008] Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Gallagher, and Tina Eliassi- Rad. Collective classification in network data. AI Mag., 29:93-106, 2008.","title":"Collective classification in network data","authors":"Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Gallagher, Tina Eliassi-Rad","year":"2008","venue":"AI Mag","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Tang et al., 2008] Jie Tang, Jing Zhang, Limin Yao, Juan-Zi Li, Li Zhang, and Zhong Su. Arnetminer: extraction and mining of academic social networks. In KDD, 2008.","title":"Arnetminer: extraction and mining of academic social networks","authors":"Jie Tang, Jing Zhang, Limin Yao, Juan-Zi Li, Li Zhang, Zhong Su","year":"2008","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Velickovic et al., 2018] Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio', and Yoshua Bengio. Graph attention networks. In ICLR, 2018.","title":"Graph attention networks","authors":"Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, ', Yoshua Bengio","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wang et al., 2021] Zhitao Wang, Yong Zhou, Litao Hong, Yuanhang Zou, and Hanjing Su. Pairwise learning for neural link prediction. arXiv preprint arXiv:2112.02936, 2021.","title":"Pairwise learning for neural link prediction","authors":"Wang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Yang et al., 2016] Zhilin Yang, William Cohen, and Ruslan Salakhudinov. Revisiting semi-supervised learning with graph embeddings. In ICML, 2016.","title":"Revisiting semi-supervised learning with graph embeddings","authors":"Zhilin Yang, William Cohen, Ruslan Salakhudinov","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Yang et al., 2020] Zhen Yang, Ming Ding, Chang Zhou, Hongxia Yang, Jingren Zhou, and Jie Tang. Understand- ing negative sampling in graph representation learning. In KDD 2020, pages 1666-1676, 2020.","title":"Understanding negative sampling in graph representation learning","authors":"Zhen Yang, Ming Ding, Chang Zhou, Hongxia Yang, Jingren Zhou, Jie Tang","year":"2020","venue":"KDD 2020","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zeng et al., 2019] Hanqing Zeng, Hongkuan Zhou, Ajitesh Srivastava, Rajgopal Kannan, and Viktor K. Prasanna. Graphsaint: Graph sampling based inductive learning method. CoRR, abs/1907.04931, 2019.","title":"Graphsaint: Graph sampling based inductive learning method","authors":"Hanqing Zeng, Hongkuan Zhou, Ajitesh Srivastava, Rajgopal Kannan, Viktor K Prasanna","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhao et al., 2021] Jialin Zhao, Yuxiao Dong, Ming Ding, Evgeny Kharlamov, and Jie Tang. Adaptive diffusion in graph neural networks. In NeurIPS, 2021.","title":"Adaptive diffusion in graph neural networks","authors":"Jialin Zhao, Yuxiao Dong, Ming Ding, Evgeny Kharlamov, Jie Tang","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zheng et al., 2021] Yanan Zheng, Jing Zhou, Yujie Qian, Ming Ding, Jian Li, Ruslan Salakhutdinov, Jie Tang, Se- bastian Ruder, and Zhilin Yang. Fewnlu: Benchmarking state-of-the-art methods for few-shot natural language un- derstanding. arXiv preprint arXiv:2109.12742, 2021.","title":"Fewnlu: Benchmarking state-of-the-art methods for few-shot natural language understanding","authors":"Yanan Zheng, Jing Zhou, Yujie Qian, Ming Ding, Jian Li, Ruslan Salakhutdinov, Jie Tang, Sebastian Ruder, Zhilin Yang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}