{"type":"ScholarlyArticle","title":"Stability and Generalization of lp-Regularized Stochastic Learning for GCN","authors":[{"name":"Shiyu Liu","slug":"shiyu-liu","given":"Shiyu","family":"Liu","affiliation":"University of Electronic Science and Technology of China , China"},{"name":"Linsen Wei","slug":"linsen-wei","given":"Linsen","family":"Wei","affiliation":"School of Astronautics , Northwestern Polytechnical University , China"},{"name":"Shaogao Lv","slug":"shaogao-lv","given":"Shaogao","family":"Lv","affiliation":"Department of Statistics and Data Science , Nanjing Audit University , China"},{"name":"Ming Li","slug":"ming-li","given":"Ming","family":"Li","affiliation":"Key Laboratory of Intelligent Education Technology and Application of Zhejiang Province , Zhejiang Normal University , China"}],"abstract":"Graph convolutional networks (GCN) are viewed as one of the most popular representations among the variants of graph neural networks over graph data and have shown powerful performance in empirical experiments. That l2-based graph smoothing enforces the global smoothness of GCN, while (soft) l1-based sparse graph learning tends to promote signal sparsity to trade for discontinuity. This paper aims to quantify the trade-off of GCN between smoothness and sparsity, with the help of a general lp-regularized (1<p<= 2) stochastic learning proposed within. While stability-based generalization analyses have been given in prior work for a second derivative objectiveness function, our lp-regularized learning scheme does not satisfy such a smooth condition. To tackle this issue, we propose a novel SGD proximal algorithm for GCNs with an inexact operator. For a single-layer GCN, we establish an explicit theoretical understanding of GCN with the lp-regularized stochastic learning by analyzing the stability of our SGD proximal algorithm. We conduct multiple empirical experiments to validate our theoretical findings.","doi":"10.24963/ijcai.2023/631","url":"https://ijcai.org/proceedings/2023/631","pdf":"https://ijcai.org/proceedings/2023/0631.pdf","markdown":"https://ijcai.org/proceedings/2023/631.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/631","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Uncertainty in AI","track":"Main Track","number":631,"pages":{"first":5685,"last":5693},"keywords":["Uncertainty in AI: UAI: Graphical models"],"videos":[],"references":[{"position":1,"raw":"[Barrett et al., 2018] David Barrett, Felix Hill, Adam Santoro, Ari Morcos, and Timothy Lillicrap. Measuring abstract reasoning in neural networks. In International Conference on Machine Learning, pages 511-520. PMLR, 2018.","title":"Measuring abstract reasoning in neural networks","authors":"David Barrett, Felix Hill, Adam Santoro, Ari Morcos, Timothy Lillicrap","year":"2018","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Battaglia et al., 2016] Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, et al. Interaction networks for learning about objects, relations and physics. In Advances in Neural Information Processing Systems, pages 4502-4510, 2016.","title":"Interaction networks for learning about objects, relations and physics","authors":"Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bottou and Bousquet, 2007] Léon Bottou and Olivier Bous- quet. The tradeoffs of large scale learning. In Advances in Neural Information Processing Systems, pages 161-168, 2007.","title":"The tradeoffs of large scale learning","authors":"Bousquet Bottou, Léon Bottou, Olivier Bousquet","year":"2007","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bousquet and Elisseeff, 2002] Olivier Bousquet and André Elisseeff. Stability and generalization. The Journal of Machine Learning Research, 2:499-526, 2002.","title":"Stability and generalization","authors":"Elisseeff Bousquet, Olivier Bousquet, André Elisseeff","year":"2002","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bruna et al., 2013] Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann LeCun. Spectral networks and lo- cally connected networks on graphs. arXiv preprint arXiv:1312.6203, 2013.","title":"Spectral networks and locally connected networks on graphs","authors":"Joan Bruna, Wojciech Zaremba, Arthur Szlam, Yann Lecun","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Defferrard et al., 2016] Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. In Advances in neural information processing systems, volume 29, pages 3844-3852, 2016.","title":"Convolutional neural networks on graphs with fast localized spectral filtering","authors":"Michaël Defferrard, Xavier Bresson, Pierre Vandergheynst","year":"2016","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Duvenaud et al., 2015] David K Duvenaud, Dougal Maclau- rin, Jorge Iparraguirre, Rafael Bombarell, Timothy Hirzel, Alan Aspuru-Guzik, and Ryan P Adams. Convolutional networks on graphs for learning molecular fingerprints. In Advances in Neural Information Processing Systems, vol- ume 28, pages 2224-2232, 2015.","title":"Convolutional networks on graphs for learning molecular fingerprints","authors":"Dougal David K Duvenaud, Jorge Maclaurin, Rafael Iparraguirre, Timothy Bombarell, Alan Hirzel, Ryan P Aspuru-Guzik, Adams","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Garg et al., 2020] Vikas Garg, Stefanie Jegelka, and Tommi Jaakkola. Generalization and representational limits of graph neural networks. In International Conference on Machine Learning, pages 3419-3430. PMLR, 2020.","title":"Generalization and representational limits of graph neural networks","authors":"Vikas Garg, Stefanie Jegelka, Tommi Jaakkola","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gilmer et al., 2017] Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neu- ral message passing for quantum chemistry. In Interna- tional Conference on Machine Learning, pages 1263-1272. PMLR, 2017.","title":"Neural message passing for quantum chemistry","authors":"Justin Gilmer, S Samuel, Patrick F Schoenholz, Oriol Riley, George E Vinyals, Dahl","year":"2017","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hamilton et al., 2017] William L Hamilton, Rex Ying, and Jure Leskovec. Inductive representation learning on large graphs. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pages 1025- 1035, 2017.","title":"Inductive representation learning on large graphs","authors":"Rex William L Hamilton, Jure Ying, Leskovec","year":"2017","venue":"Proceedings of the 31st International Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hardt et al., 2016] Moritz Hardt, Ben Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent. In International Conference on Machine Learning, pages 1225-1234. PMLR, 2016.","title":"Train faster, generalize better: Stability of stochastic gradient descent","authors":"Moritz Hardt, Ben Recht, Yoram Singer","year":"2016","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Jin et al., 2018] Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Junction tree variational autoencoder for molec- ular graph generation. In International Conference on Machine Learning, pages 2323-2332. PMLR, 2018.","title":"Junction tree variational autoencoder for molecular graph generation","authors":"Jin","year":"2018","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kearns and Ron, 1999] Michael Kearns and Dana Ron. Al- gorithmic stability and sanity-check bounds for leave-one- out cross-validation. Neural Computation, 11(6):1427- 1453, 1999.","title":"Algorithmic stability and sanity-check bounds for leave-oneout cross-validation","authors":"Michael Kearns, Dana Ron","year":"1999","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kipf and Welling, 2016a] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907, 2016.","title":"Semi-supervised classification with graph convolutional networks","authors":"Welling ; Kipf, N Thomas, Max Kipf, Welling","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kipf and Welling, 2016b] Thomas N Kipf and Max Welling. Variational graph auto-encoders. arXiv preprint arXiv:1611.07308, 2016.","title":"Variational graph auto-encoders","authors":"Welling ; Kipf, N Thomas, Max Kipf, Welling","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Koltchinskii, 2009] Vladimir Koltchinskii. Sparsity in pe- nalized empirical risk minimization. In Annales de l'IHP Probabilités et statistiques, volume 45, pages 7-57, 2009.","title":"Vladimir Koltchinskii. Sparsity in penalized empirical risk minimization","authors":"Koltchinskii","year":"2009","venue":"Annales de l'IHP Probabilités et statistiques","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liao et al., 2020] Renjie Liao, Raquel Urtasun, and Richard Zemel. A pac-bayesian approach to generalization bounds for graph neural networks. In International Conference on Learning Representations, 2020.","title":"A pac-bayesian approach to generalization bounds for graph neural networks","authors":"Renjie Liao, Raquel Urtasun, Richard Zemel","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Liu et al., 2021] Xiaorui Liu, Wei Jin, Yao Ma, Yaxin Li, Hua Liu, Yiqi Wang, Ming Yan, and Jiliang Tang. Elastic graph neural networks. In International Conference on Machine Learning, pages 6837-6849. PMLR, 2021.","title":"Elastic graph neural networks","authors":"Xiaorui Liu, Wei Jin, Yao Ma, Yaxin Li, Hua Liu, Yiqi Wang, Ming Yan, Jiliang Tang","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Ma et al., 2021] Yao Ma, Xiaorui Liu, Tong Zhao, Yozen Liu, Jiliang Tang, and Neil Shah. A unified view on graph neural networks as graph signal denoising. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pages 1202-1211, 2021.","title":"A unified view on graph neural networks as graph signal denoising","authors":"Yao Ma, Xiaorui Liu, Tong Zhao, Yozen Liu, Jiliang Tang, Neil Shah","year":"2021","venue":"Proceedings of the 30th ACM International Conference on Information & Knowledge Management","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Nie et al., 2011] Feiping Nie, Hua Wang, Heng Huang, and Chris Ding. Unsupervised and semi-supervised learning via 1-norm graph. In 2011 International Conference on Computer Vision, pages 2268-2273. IEEE, 2011.","title":"Unsupervised and semi-supervised learning via 1-norm graph","authors":"Nie","year":"2011","venue":"2011 International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Oono and Suzuki, 2020] Kenta Oono and Taiji Suzuki. Op- timization and generalization analysis of transduction through gradient boosting and application to multi-scale graph neural networks. Advances in Neural Information Processing Systems, 33:18917-18930, 2020.","title":"Optimization and generalization analysis of transduction through gradient boosting and application to multi-scale graph neural networks","authors":"Suzuki Oono, Kenta Oono, Taiji Suzuki","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Schlichtkrull et al., 2018] Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling. Modeling relational data with graph convo- lutional networks. In European semantic web conference, pages 593-607. Springer, 2018.","title":"Modeling relational data with graph convolutional networks","authors":"Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den, Ivan Berg, Max Titov, Welling","year":"2018","venue":"European semantic web conference","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Sen et al., 2008] Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Rad. Collective classification in network data. AI magazine, 29(3):93-93, 2008.","title":"Collective classification in network data","authors":"Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, Tina Eliassi-Rad","year":"2008","venue":"AI magazine","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Shalev-Shwartz et al., 2010] Shai Shalev-Shwartz, Ohad Shamir, Nathan Srebro, and Karthik Sridharan. Learn- ability, stability and uniform convergence. The Journal of Machine Learning Research, 11:2635-2670, 2010.","title":"Learnability, stability and uniform convergence","authors":"Shai Shalev-Shwartz, Ohad Shamir, Nathan Srebro, Karthik Sridharan","year":"2010","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Tibshirani, 2014] Ryan J Tibshirani. Adaptive piecewise polynomial estimation via trend filtering. The Annals of Statistics, 42(1):285-323, 2014.","title":"Adaptive piecewise polynomial estimation via trend filtering","authors":"J Ryan, Tibshirani","year":"2014","venue":"The Annals of Statistics","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Veličković et al., 2017] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv:1710.10903, 2017.","title":"Graph attention networks","authors":"Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, Yoshua Bengio","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Verma and Zhang, 2019] Saurabh Verma and Zhi-Li Zhang. Stability and generalization of graph convolutional neural networks. In Proceedings of the 25th ACM SIGKDD In- ternational Conference on Knowledge Discovery & Data Mining, pages 1539-1548, 2019.","title":"Stability and generalization of graph convolutional neural networks","authors":"Saurabh Verma, Zhi-Li Zhang","year":"2019","venue":"Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wang et al., 2015] Yu-Xiang Wang, James Sharpnack, Alex Smola, and Ryan Tibshirani. Trend filtering on graphs. In Artificial Intelligence and Statistics, pages 1042-1050. PMLR, 2015.","title":"Trend filtering on graphs","authors":"Yu-Xiang Wang, James Sharpnack, Alex Smola, Ryan Tibshirani","year":"2015","venue":"Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wibisono et al., 2009] Andre Wibisono, Lorenzo Rosasco, and Tomaso Poggio. Sufficient conditions for uniform stability of regularization algorithms. Computer Science and Artificial Intelligence Laboratory Technical Report, MIT-CSAIL-TR-2009-060, 2009.","title":"Sufficient conditions for uniform stability of regularization algorithms","authors":"Andre Wibisono, Lorenzo Rosasco, Tomaso Poggio","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Xu et al., 2011] Huan Xu, Constantine Caramanis, and Shie Mannor. Sparse algorithms are not stable: A no-free-lunch theorem. IEEE transactions on Pattern Analysis and Ma- chine Intelligence, 34(1):187-193, 2011.","title":"Sparse algorithms are not stable: A no-free-lunch theorem","authors":"Huan Xu, Constantine Caramanis, Shie Mannor","year":"2011","venue":"IEEE transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Xu et al., 2018] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? arXiv preprint arXiv:1810.00826, 2018.","title":"How powerful are graph neural networks","authors":"Xu","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Yun et al., 2019] Seongjun Yun, Minbyul Jeong, Raehyun Kim, Jaewoo Kang, and Hyunwoo J Kim. Graph trans- former networks. Advances in Neural Information Process- ing Systems, 32:11983-11993, 2019.","title":"Graph transformer networks","authors":"Yun","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhang et al., 2020] Yuyu Zhang, Xinshi Chen, Yuan Yang, Arun Ramamurthy, Bo Li, Yuan Qi, and Le Song. Efficient probabilistic logic reasoning with graph neural networks. arXiv preprint arXiv:2001.11850, 2020.","title":"Efficient probabilistic logic reasoning with graph neural networks","authors":"Yuyu Zhang, Xinshi Chen, Yuan Yang, Arun Ramamurthy, Bo Li, Yuan Qi, Le Song","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}