{"type":"ScholarlyArticle","title":"Learning Continuous Graph Structure with Bilevel Programming for Graph Neural Networks","authors":[{"name":"Minyang Hu","slug":"minyang-hu","given":"Minyang","family":"Hu","affiliation":"Key Laboratory of Intelligent Information Processing of Chinese Academy of Sciences (CAS) , Institute of Computing Technology , CAS , China; University of Chinese Academy of Sciences , China"},{"name":"Hong Chang","slug":"hong-chang","given":"Hong","family":"Chang","affiliation":"Key Laboratory of Intelligent Information Processing of Chinese Academy of Sciences (CAS) , Institute of Computing Technology , CAS , China; University of Chinese Academy of Sciences , China"},{"name":"Bingpeng Ma","slug":"bingpeng-ma","given":"Bingpeng","family":"Ma","affiliation":"University of Chinese Academy of Sciences , China"},{"name":"Shiguang Shan","slug":"shiguang-shan","given":"Shiguang","family":"Shan","affiliation":"Key Laboratory of Intelligent Information Processing of Chinese Academy of Sciences (CAS) , Institute of Computing Technology , CAS , China; University of Chinese Academy of Sciences , China"}],"abstract":"Learning graph structure for graph neural networks (GNNs) is crucial to facilitate the GNN-based downstream learning tasks. It is challenging due to the non-differentiable discrete graph structure and lack of ground-truth. In this paper, we address these problems and propose a novel graph structure learning framework for GNNs. Firstly, we directly model the continuous graph structure with dual-normalization, which implicitly imposes sparse constraint and reduces the influence of noisy edges. Secondly, we formulate the whole training process as a bilevel programming problem, where the inner objective is to optimize the GNNs given learned graphs, while the outer objective is to optimize the graph structure to minimize the generalization error of downstream task. Moreover, for bilevel optimization, we propose an improved Neumann-IFT algorithm to obtain an approximate solution, which is more stable and accurate than existing optimization methods. Besides, it makes the bilevel optimization process memory-efficient and scalable to large graphs. Experiments on node classification and scene graph generation show that our method can outperform related methods, especially with noisy graphs.","doi":"10.24963/ijcai.2022/424","url":"https://ijcai.org/proceedings/2022/424","pdf":"https://ijcai.org/proceedings/2022/0424.pdf","markdown":"https://ijcai.org/proceedings/2022/424.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/424","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Machine Learning","track":"Main Track","number":424,"pages":{"first":3057,"last":3063},"keywords":["Computer Vision: Scene analysis and understanding","Machine Learning: Hyperparameter Optimization","Data Mining: Mining Graphs"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986878","length_sec":83,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985021","length_sec":351,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Bengio et al., 2013] Y. Bengio, N. Léonard, and A. Courville. Es- timating or propagating gradients through stochastic neurons for conditional computation. arXiv:1308.3432, 2013.","title":"Estimating or propagating gradients through stochastic neurons for conditional computation","authors":"Y Bengio, N Léonard, A Courville","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2020] Y. Chen, L. Wu, and M. Zaki. Iterative deep graph learning for graph neural networks: Better and robust node embeddings. In NeurIPS, 2020.","title":"Iterative deep graph learning for graph neural networks: Better and robust node embeddings","authors":"Y Chen, L Wu, M Zaki","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Colson et al., 2007] B. Colson, P. Marcotte, and G. Savard. An overview of bilevel optimization. Annals of Operations Research, 2007.","title":"An overview of bilevel optimization","authors":"B Colson, P Marcotte, G Savard","year":"2007","venue":"Annals of Operations Research","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Fatemi et al., 2021] B. Fatemi, L. El Asri, and S.M. Kazemi. Slaps: Self-supervision improves structure learning for graph neural networks. In NeurIPS, 2021.","title":"Slaps: Self-supervision improves structure learning for graph neural networks","authors":"B Fatemi, L El Asri, S M Kazemi","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Franceschi et al., 2019] L. Franceschi, M. Niepert, M. Pontil, and X. He. Learning discrete structures for graph neural networks. In ICML, 2019.","title":"Learning discrete structures for graph neural networks","authors":"L Franceschi, M Niepert, M Pontil, X He","year":"2019","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Franklin, 2012] J.N. Franklin. Matrix theory. Courier Corporation, 2012.","title":"Matrix theory","authors":"J N Franklin","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gao and Ji, 2019] H. Gao and S. Ji. Graph u-nets. In ICML, 2019.","title":"Graph u-nets","authors":"H Gao, S Ji","year":"2019","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Ghadimi and Wang, 2018] S. Ghadimi and M. Wang. Approxima- tion methods for bilevel programming. arXiv:1802.02246, 2018.","title":"Approximation methods for bilevel programming","authors":"S Ghadimi, M Wang","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Golub and Loan, 2013] G.H. Golub and C.F. Van Loan. Matrix computations. JHU press, 2013.","title":"Matrix computations","authors":"G H Golub, C F Van Loan","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Grazzi et al., 2020] R. Grazzi, L. Franceschi, M. Pontil, and S. Salzo. On the iteration complexity of hypergradient compu- tation. In ICML, 2020.","title":"On the iteration complexity of hypergradient computation","authors":"R Grazzi, L Franceschi, M Pontil, S Salzo","year":"2020","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hamilton et al., 2017] W. Hamilton, Z. Ying, and J. Leskovec. In- ductive representation learning on large graphs. In NeurIPS, 2017.","title":"Inductive representation learning on large graphs","authors":"W Hamilton, Z Ying, J Leskovec","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[He et al., 2020] X. He, K. Deng, X. Wang, Y. Li, Y. Zhang, and M. Wang. Lightgcn: Simplifying and powering graph convolu- tion network for recommendation. In SIGIR, 2020.","title":"Lightgcn: Simplifying and powering graph convolution network for recommendation","authors":"He","year":"2020","venue":"SIGIR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Henaff et al., 2015] M. Henaff, J. Bruna, and Y. LeCun. Deep con- volutional networks on graph-structured data. arXiv:1506.05163, 2015.","title":"Deep convolutional networks on graph-structured data","authors":"M Henaff, J Bruna, Y Lecun","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Huang and Carley, 2019] B. Huang and K. Carley. Syntax-aware aspect level sentiment classification with graph attention net- works. In EMNLP, 2019.","title":"Syntax-aware aspect level sentiment classification with graph attention networks","authors":"B Huang, K Carley","year":"2019","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Jang et al., 2017] E. Jang, S. Gu, and B. Poole. Categorical repa- rameterization with gumbel-softmax. In ICLR, 2017.","title":"Categorical reparameterization with gumbel-softmax","authors":"E Jang, S Gu, B Poole","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Jiang et al., 2019] B. Jiang, Z. Zhang, D. Lin, J. Tang, and B. Luo. Semi-supervised learning with graph learning-convolutional net- works. In CVPR, 2019.","title":"Semi-supervised learning with graph learning-convolutional networks","authors":"B Jiang, Z Zhang, D Lin, J Tang, B Luo","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Jin et al., 2020] W. Jin, Y. Ma, X. Liu, X. Tang, S. Wang, and J. Tang. Graph structure learning for robust graph neural net- works. In SIGKDD, 2020.","title":"Graph structure learning for robust graph neural networks","authors":"W Jin, Y Ma, X Liu, X Tang, S Wang, J Tang","year":"2020","venue":"SIGKDD","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Kazi et al., 2020] A. Kazi, L. Cosmo, N. Navab, and M. Bronstein. Differentiable graph module (dgm) for graph convolutional net- works. arXiv:2002.04999, 2020.","title":"Differentiable graph module (dgm) for graph convolutional networks","authors":"A Kazi, L Cosmo, N Navab, M Bronstein","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Kingma and Welling, 2013] D.P. Kingma and M. Welling. Auto- encoding variational bayes. arXiv:1312.6114, 2013.","title":"Autoencoding variational bayes","authors":"D P Kingma, M Welling","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Kipf and Welling, 2017] T.N. Kipf and M. Welling. Semi- supervised classification with graph convolutional networks. In ICLR, 2017.","title":"Semisupervised classification with graph convolutional networks","authors":"T N Kipf, M Welling","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Kool et al., 2019] W. Kool, H. Van Hoof, and M. Welling. Stochas- tic beams and where to find them: The gumbel-top-k trick for sampling sequences without replacement. In ICML, 2019.","title":"Stochastic beams and where to find them: The gumbel-top-k trick for sampling sequences without replacement","authors":"W Kool, H Van Hoof, M Welling","year":"2019","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Li et al., 2016] Y. Li, D. Tarlow, M. Brockschmidt, and R. Zemel. Gated graph sequence neural networks. In ICLR, 2016.","title":"Gated graph sequence neural networks","authors":"Y Li, D Tarlow, M Brockschmidt, R Zemel","year":"2016","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Li et al., 2021] R. Li, S. Zhang, B. Wan, and X. He. Bipartite graph network with adaptive message passing for unbiased scene graph generation. In CVPR, 2021.","title":"Bipartite graph network with adaptive message passing for unbiased scene graph generation","authors":"R Li, S Zhang, B Wan, X He","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Lorraine et al., 2020] J. Lorraine, P. Vicol, and D. Duvenaud. Op- timizing millions of hyperparameters by implicit differentiation. In AISTATS, 2020.","title":"Optimizing millions of hyperparameters by implicit differentiation","authors":"J Lorraine, P Vicol, D Duvenaud","year":"2020","venue":"AISTATS","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Pearlmutter, 1994] B.A. Pearlmutter. Fast exact multiplication by the Hessian. Neural Computation, 1994.","title":"Fast exact multiplication by the Hessian","authors":"B A Pearlmutter","year":"1994","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Qu et al., 2020] M. Qu, T. Gao, L. Xhonneux, and J. Tang. Few- shot relation extraction via bayesian meta-learning on relation graphs. In ICML, 2020.","title":"Fewshot relation extraction via bayesian meta-learning on relation graphs","authors":"M Qu, T Gao, L Xhonneux, J Tang","year":"2020","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Rudin, 1964] W. Rudin. Principles of mathematical analysis. McGraw-hill New York, 1964.","title":"Principles of mathematical analysis","authors":"W Rudin","year":"1964","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Veličković et al., 2018] P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Liò, and Y. Bengio. Graph attention networks. In ICLR, 2018.","title":"Graph attention networks","authors":"P Veličković, G Cucurull, A Casanova, A Romero, P Liò, Y Bengio","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wan and Kokel, 2021] G. Wan and H. Kokel. Graph sparsification via meta-learning. DLG@AAAI, 2021.","title":"Graph sparsification via meta-learning","authors":"G Wan, H Kokel","year":"2021","venue":"DLG@AAAI","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wang et al., 2020] X. Wang, M. Zhu, D. Bo, P. Cui, C. Shi, and J. Pei. Am-gcn: Adaptive multi-channel graph convolutional net- works. In SIGKDD, 2020.","title":"Am-gcn: Adaptive multi-channel graph convolutional networks","authors":"Wang","year":"2020","venue":"SIGKDD","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Wu et al., 2018] Z. Wu, B. Ramsundar, E.N. Feinberg, J. Gomes, C. Geniesse, A.S. Pappu, K. Leswing, and V. Pande. Molecu- lenet: a benchmark for molecular machine learning. Chemical science, 2018.","title":"Moleculenet: a benchmark for molecular machine learning","authors":"Z Wu, B Ramsundar, E N Feinberg, J Gomes, C Geniesse, A S Pappu, K Leswing, V Pande","year":"2018","venue":"Chemical science","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Wu et al., 2019] H. Wu, C. Wang, Y. Tyshetskiy, A. Docherty, K. Lu, and L. Zhu. Adversarial examples on graph data: Deep insights into attack and defense. In IJCAI, 2019.","title":"Adversarial examples on graph data: Deep insights into attack and defense","authors":"H Wu, C Wang, Y Tyshetskiy, A Docherty, K Lu, L Zhu","year":"2019","venue":"IJCAI","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Xu et al., 2017] D. Xu, Y. Zhu, C.B. Choy, and L. Fei-Fei. Scene graph generation by iterative message passing. In CVPR, 2017.","title":"Scene graph generation by iterative message passing","authors":"D Xu, Y Zhu, C B Choy, L Fei-Fei","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Xu et al., 2018] K. Xu, C. Li, Y. Tian, T. Sonobe, K. Kawarabayashi, and S. Jegelka. Representation learning on graphs with jumping knowledge networks. In ICML, 2018.","title":"Representation learning on graphs with jumping knowledge networks","authors":"K Xu, C Li, Y Tian, T Sonobe, K Kawarabayashi, S Jegelka","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Xu et al., 2019] B. Xu, H. Shen, Q. Cao, Y. Qiu, and X. Cheng. Graph wavelet neural network. In ICLR, 2019.","title":"Graph wavelet neural network","authors":"B Xu, H Shen, Q Cao, Y Qiu, X Cheng","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Yu et al., 2020] F. Yu, Y. Zhu, Q. Liu, S. Wu, L. Wang, and T. Tan. Tagnn: Target attentive graph neural networks for session-based recommendation. In SIGIR, 2020.","title":"Tagnn: Target attentive graph neural networks for session-based recommendation","authors":"F Yu, Y Zhu, Q Liu, S Wu, L Wang, T Tan","year":"2020","venue":"SIGIR","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Zhang and Zitnik, 2020] X. Zhang and M. Zitnik. Gnnguard: De- fending graph neural networks against adversarial attacks. In NeurIPS, 2020.","title":"Gnnguard: Defending graph neural networks against adversarial attacks","authors":"Zitnik Zhang, X Zhang, M Zitnik","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Zheng et al., 2020] C. Zheng, B. Zong, W. Cheng, D. Song, J. Ni, W. Yu, H. Chen, and W. Wang. Robust graph representation learning via neural sparsification. In ICML, 2020.","title":"Robust graph representation learning via neural sparsification","authors":"C Zheng, B Zong, W Cheng, D Song, J Ni, W Yu, H Chen, W Wang","year":"2020","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Zügner and Günnemann, 2019] D. Zügner and S. Günnemann. Adversarial attacks on graph neural networks via meta learning. In ICLR, 2019.","title":"Adversarial attacks on graph neural networks via meta learning","authors":"D Zügner, S Günnemann","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}