{"type":"ScholarlyArticle","title":"An Efficient Prototype-Based Clustering Approach for Edge Pruning in Graph Neural Networks to Battle Over-Smoothing","authors":[{"name":"Yuyang Huang","slug":"yuyang-huang","given":"Yuyang","family":"Huang","affiliation":"Department of Computer Science and Engineering , Shanghai Jiao Tong University , Shanghai , China"},{"name":"Wenjing Lu","slug":"wenjing-lu","given":"Wenjing","family":"Lu","affiliation":"Department of Computer Science and Engineering , Shanghai Jiao Tong University , Shanghai , China"},{"name":"Yang Yang","slug":"yang-yang","given":"Yang","family":"Yang","affiliation":"Department of Computer Science and Engineering , Shanghai Jiao Tong University , Shanghai , China; Key Laboratory of Shanghai Education Commission for Intelligent Interaction and Cognitive Engineering , Shanghai , China"}],"abstract":"Topology augmentation is a popular strategy to address the issue of over-smoothing in graph neural networks (GNNs). To prevent potential distortion of node representations, an essential principle is to enhance the separability between embeddings of nodes from different classes while preserving smoothness among nodes of the same class. However, differentiating between inter-class and intra-class edges becomes arduous when class labels are unavailable or the graph is partially labeled. While clustering offers an alternative for identifying closely connected groups of nodes, traditional clustering methods face challenges when applied to GNNs in terms of accuracy, efficiency, adaptability, and scalability to diverse graphs. To address these limitations, we introduce ClusterDrop, which uses learnable prototypes for efficient clustering and incorporates supervised signals to enhance accuracy and adaptability across different graphs. Experiments on six datasets with varying graph structures demonstrate its effectiveness in alleviating over-smoothing and enhancing GNN performance.","doi":"10.24963/ijcai.2024/464","url":"https://ijcai.org/proceedings/2024/464","pdf":"https://ijcai.org/proceedings/2024/0464.pdf","markdown":"https://ijcai.org/proceedings/2024/464.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/464","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Machine Learning","track":"Main Track","number":464,"pages":{"first":4201,"last":4209},"keywords":["Data Mining: DM: Mining graphs","Machine Learning: ML: Sequence and graph learning","Data Mining: DM: Networks"],"videos":[],"references":[{"position":1,"raw":"[Asano et al., 2020] Yuki Markus Asano, Christian Rup- precht, and Andrea Vedaldi. Self-labelling via simulta- neous clustering and representation learning. In 8th Inter- national Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenRe- view.net, 2020.","title":"Self-labelling via simultaneous clustering and representation learning","authors":"Yuki Markus Asano, Christian Rupprecht, Andrea Vedaldi","year":"2020","venue":"8th International Conference on Learning Representations, ICLR 2020","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Caron et al., 2020] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Bal- can, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.","title":"Unsupervised learning of visual features by contrasting cluster assignments","authors":"Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, Armand Joulin","year":"2020","venue":"Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2020a] Deli Chen, Yankai Lin, Wei Li, Peng Li, Jie Zhou, and Xu Sun. Measuring and relieving the over-smoothing problem for graph neural networks from the topological view. In Proceedings of the AAAI con- ference on artificial intelligence, volume 34, pages 3438- 3445, 2020.","title":"Measuring and relieving the over-smoothing problem for graph neural networks from the topological view","authors":"Chen","year":"2020","venue":"Proceedings of the AAAI conference on artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2020b] Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. Simple and deep graph convolutional networks. In International conference on machine learning, pages 1725-1735. PMLR, 2020.","title":"Simple and deep graph convolutional networks","authors":"Chen","year":"2020","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2023] Dong Chen, Xiang Zhao, Wei Wang, Zhen Tan, and Weidong Xiao. Graph self-supervised learning with augmentation-aware contrastive learning. In Proceedings of the ACM Web Conference 2023, pages 154-164, 2023.","title":"Graph self-supervised learning with augmentation-aware contrastive learning","authors":"Dong Chen, Xiang Zhao, Wei Wang, Zhen Tan, Weidong Xiao","year":"2023","venue":"Proceedings of the ACM Web Conference 2023","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Ding et al., 2019] Ming Ding, Chang Zhou, Qibin Chen, Hongxia Yang, and Jie Tang. Cognitive graph for multi- hop reading comprehension at scale. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2694-2703. Association for Computa- tional Linguistics, July 2019.","title":"Cognitive graph for multihop reading comprehension at scale","authors":"Ming Ding, Chang Zhou, Qibin Chen, Hongxia Yang, Jie Tang","year":"2019","venue":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gasteiger et al., 2019] Johannes Gasteiger, Aleksandar Bo- jchevski, and Stephan Günnemann. Combining neural networks with personalized pagerank for classification on graphs. In International Conference on Learning Repre- sentations, 2019.","title":"Combining neural networks with personalized pagerank for classification on graphs","authors":"Gasteiger","year":"2019","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Guo et al., 2023] Xiaojun Guo, Yifei Wang, Tianqi Du, and Yisen Wang. Contranorm: A contrastive learning perspec- tive on oversmoothing and beyond. In The Eleventh Inter- national Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023, 2023.","title":"Contranorm: A contrastive learning perspective on oversmoothing and beyond","authors":"Xiaojun Guo, Yifei Wang, Tianqi Du, Yisen Wang","year":"2023","venue":"The Eleventh International Conference on Learning Representations, ICLR 2023","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hassani and Khasahmadi, 2020] Kaveh Hassani and Amir Hosein Khasahmadi. Contrastive multi-view repre- sentation learning on graphs. In International conference on machine learning, pages 4116-4126. PMLR, 2020.","title":"Contrastive multi-view repre-sentation learning on graphs","authors":"Kaveh Hassani, Amir Hosein, Khasahmadi","year":"2020","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hou et al., 2023] Zhenyu Hou, Yufei He, Yukuo Cen, Xiao Liu, Yuxiao Dong, Evgeny Kharlamov, and Jie Tang. Graphmae2: A decoding-enhanced masked self- supervised graph learner. In Proceedings of the ACM Web Conference 2023, pages 737-746, 2023.","title":"Graphmae2: A decoding-enhanced masked selfsupervised graph learner","authors":"Zhenyu Hou, Yufei He, Yukuo Cen, Xiao Liu, Yuxiao Dong, Evgeny Kharlamov, Jie Tang","year":"2023","venue":"Proceedings of the ACM Web Conference 2023","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hu et al., 2020] Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.","title":"Open graph benchmark: Datasets for machine learning on graphs","authors":"Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, Jure Leskovec","year":"2020","venue":"Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hui et al., 2021] Bo Hui, Da Yan, and Wei-Shinn Ku. Node- polysemy aware recommendation by matrix completion with side information. In 2021 IEEE International Con- ference on Big Data (Big Data), Orlando, FL, USA, De- cember 15-18, 2021, pages 636-642. IEEE, 2021.","title":"Nodepolysemy aware recommendation by matrix completion with side information","authors":"Bo Hui, Da Yan, Wei-Shinn Ku","year":"2021","venue":"IEEE International Conference on Big Data (Big Data)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Jiang et al., 2022] Xuan Jiang, Zhiyong Yang, Peisong Wen, Li Su, and Qingming Huang. A sparse-motif ensem- ble graph convolutional network against over-smoothing. In Luc De Raedt, editor, Proceedings of the Thirty-First In- ternational Joint Conference on Artificial Intelligence, IJ- CAI 2022, Vienna, Austria, 23-29 July 2022, pages 2094- 2100. ijcai.org, 2022.","title":"A sparse-motif ensemble graph convolutional network against over-smoothing","authors":"Xuan Jiang, Zhiyong Yang, Peisong Wen, Li Su, Qingming Huang","year":"2022","venue":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJ-CAI 2022","doi":null,"url":null,"internal_url":"/proceedings/2022/291"},{"position":14,"raw":"[Keriven, 2022] Nicolas Keriven. Not too little, not too much: a theoretical analysis of graph (over)smoothing. In NeurIPS, 2022.","title":"Not too little, not too much: a theoretical analysis of graph (over)smoothing","authors":"Nicolas Keriven","year":"2022","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kipf and Welling, 2017] Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017.","title":"Semi-supervised classification with graph convolutional networks","authors":"Thomas N Kipf, Max Welling","year":"2017","venue":"5th International Conference on Learning Representations, ICLR 2017","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kipf et al., 2018] Thomas Kipf, Ethan Fetaya, Kuan-Chieh Wang, Max Welling, and Richard Zemel. Neural relational inference for interacting systems. In International con- ference on machine learning, pages 2688-2697. PMLR, 2018.","title":"Neural relational inference for interacting systems","authors":"Thomas Kipf, Ethan Fetaya, Kuan-Chieh Wang, Max Welling, Richard Zemel","year":"2018","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Li et al., 2018] Qimai Li, Zhichao Han, and Xiao-Ming Wu. Deeper insights into graph convolutional networks for semi-supervised learning. In Proceedings of the AAAI con- ference on artificial intelligence, volume 32, 2018.","title":"Deeper insights into graph convolutional networks for semi-supervised learning","authors":"Li","year":"2018","venue":"Proceedings of the AAAI conference on artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Liu et al., 2023] Yang Liu, Chuan Zhou, Shirui Pan, Jia Wu, Zhao Li, Hongyang Chen, and Peng Zhang. Curvdrop: A ricci curvature based approach to prevent graph neural net- works from over-smoothing and over-squashing. In Pro- ceedings of the ACM Web Conference 2023, WWW 2023, Austin, TX, USA, 30 April 2023 -4 May 2023, pages 221- 230, 2023.","title":"Curvdrop: A ricci curvature based approach to prevent graph neural networks from over-smoothing and over-squashing","authors":"Yang Liu, Chuan Zhou, Shirui Pan, Jia Wu, Zhao Li, Hongyang Chen, Peng Zhang","year":"2023","venue":"Proceedings of the ACM Web Conference 2023","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Newman, 2006] M. E. J. Newman. Modularity and com- munity structure in networks. Proceedings of the National Academy of Sciences, 103(23):8577-8582, 2006.","title":"Modularity and community structure in networks","authors":"M E J Newman","year":"2006","venue":"Proceedings of the National Academy of Sciences","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Pei et al., 2020] Hongbin Pei, Bingzhe Wei, Kevin Chen- Chuan Chang, Yu Lei, and Bo Yang. Geom-gcn: Geo- metric graph convolutional networks. In 8th International Conference on Learning Representations, ICLR 2020, Ad- dis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.","title":"Geom-gcn: Geometric graph convolutional networks","authors":"Hongbin Pei, Bingzhe Wei, Kevin Chen-Chuan, Yu Chang, Bo Lei, Yang","year":"2020","venue":"8th International Conference on Learning Representations, ICLR 2020, Addis","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Rong et al., 2020] Yu Rong, Wenbing Huang, Tingyang Xu, and Junzhou Huang. Dropedge: Towards deep graph con- volutional networks on node classification. In 8th Inter- national Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, 2020.","title":"Dropedge: Towards deep graph convolutional networks on node classification","authors":"Yu Rong, Wenbing Huang, Tingyang Xu, Junzhou Huang","year":"2020","venue":"8th International Conference on Learning Representations, ICLR 2020","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Rousseeuw, 1987] Peter J. Rousseeuw. Silhouettes: A graphical aid to the interpretation and validation of cluster analysis. Journal of Computational and Applied Mathe- matics, 20:53-65, 1987.","title":"Silhouettes: A graphical aid to the interpretation and validation of cluster analysis","authors":"J Peter, Rousseeuw","year":"1987","venue":"Journal of Computational and Applied Mathematics","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Sinkhorn and Knopp, 1967] Richard Sinkhorn and Paul Knopp. Concerning nonnegative matrices and doubly stochastic matrices. Pacific Journal of Mathematics, 21(2):343-348, 1967.","title":"Richard Sinkhorn and Paul Knopp. Concerning nonnegative matrices and doubly stochastic matrices","authors":"Knopp Sinkhorn","year":"1967","venue":"Pacific Journal of Mathematics","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Tan et al., 2023] Qiaoyu Tan, Ninghao Liu, Xiao Huang, Soo-Hyun Choi, Li Li, Rui Chen, and Xia Hu. S2gae: Self-supervised graph autoencoders are generalizable learners with graph masking. In Proceedings of the Six- teenth ACM International Conference on Web Search and Data Mining, pages 787-795, 2023.","title":"S2gae: Self-supervised graph autoencoders are generalizable learners with graph masking","authors":"Qiaoyu Tan, Ninghao Liu, Xiao Huang, Soo-Hyun Choi, Li Li, Rui Chen, Xia Hu","year":"2023","venue":"Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Villani, 2021] Cédric Villani. Topics in optimal transporta- tion, volume 58. American Mathematical Soc., 2021.","title":"Topics in optimal transportation","authors":"Cédric Villani, Villani","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wang et al., 2022] Jie Wang, Jianqing Liang, Jiye Liang, and Kaixuan Yao. Guide: Training deep graph neural net- works via guided dropout over edges. IEEE Transactions on Neural Networks and Learning Systems, 2022.","title":"Guide: Training deep graph neural networks via guided dropout over edges","authors":"Jie Wang, Jianqing Liang, Jiye Liang, Kaixuan Yao","year":"2022","venue":"IEEE Transactions on Neural Networks and Learning Systems","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wong and Easton, 1980] Chak-Kuen Wong and Malcolm C. Easton. An efficient method for weighted sampling with- out replacement. SIAM Journal on Computing, 9(1):111- 113, 1980.","title":"An efficient method for weighted sampling without replacement","authors":"Chak-Kuen Wong, Malcolm C Easton","year":"1980","venue":"SIAM Journal on Computing","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wu et al., 2019] Jun Wu, Jingrui He, and Jiejun Xu. Demo- net: Degree-specific graph neural networks for node and graph classification. In Ankur Teredesai, Vipin Kumar, Ying Li, Rómer Rosales, Evimaria Terzi, and George Karypis, editors, Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2019, Anchorage, AK, USA, August 4-8, 2019, pages 406-415. ACM, 2019.","title":"Demonet: Degree-specific graph neural networks for node and graph classification","authors":"Wu","year":"2019","venue":"Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2019","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wu et al., 2023a] Gongce Wu, Shukuan Lin, Xiaoxue Shao, Peng Zhang, and Jianzhong Qiao. QPGCN: graph con- volutional network with a quadratic polynomial filter for overcoming over-smoothing. Appl. Intell., 53(6):7216- 7231, 2023.","title":"QPGCN: graph convolutional network with a quadratic polynomial filter for overcoming over-smoothing","authors":"Wu","year":"2023","venue":"Appl. Intell","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wu et al., 2023b] Xinyi Wu, Zhengdao Chen, William Wei Wang, and Ali Jadbabaie. A non-asymptotic analysis of oversmoothing in graph neural networks. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenRe- view.net, 2023.","title":"A non-asymptotic analysis of oversmoothing in graph neural networks","authors":"Wu","year":"2023","venue":"The Eleventh International Conference on Learning Representations, ICLR 2023","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Xu et al., 2018] Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, and Stefanie Jegelka. Representation learning on graphs with jumping knowledge networks. In International conference on ma- chine learning, pages 5453-5462. PMLR, 2018.","title":"Representation learning on graphs with jumping knowledge networks","authors":"Xu","year":"2018","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zachary, 1977] Wayne W Zachary. An information flow model for conflict and fission in small groups. Journal of anthropological research, 33(4):452-473, 1977.","title":"An information flow model for conflict and fission in small groups","authors":"Zachary Wayne","year":"1977","venue":"Journal of anthropological research","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhang et al., 2023] Yifei Zhang, Hao Zhu, Zixing Song, Pi- otr Koniusz, and Irwin King. Spectral feature augmen- tation for graph contrastive learning and beyond. pages 11289-11297. AAAI Press, 2023.","title":"Spectral feature augmentation for graph contrastive learning and beyond","authors":"Yifei Zhang, Hao Zhu, Zixing Song, Piotr Koniusz, Irwin King","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zhao and Akoglu, 2020] Lingxiao Zhao and Leman Akoglu. Pairnorm: Tackling oversmoothing in gnns. In International Conference on Learning Representations, 2020.","title":"Pairnorm: Tackling oversmoothing in gnns","authors":"Akoglu Zhao, Lingxiao Zhao, Leman Akoglu","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Zhao et al., 2021] Tong Zhao, Yozen Liu, Leonardo Neves, Oliver Woodford, Meng Jiang, and Neil Shah. Data aug- mentation for graph neural networks. In Proceedings of the aaai conference on artificial intelligence, volume 35, pages 11015-11023, 2021.","title":"Data augmentation for graph neural networks","authors":"Tong Zhao, Yozen Liu, Leonardo Neves, Oliver Woodford, Meng Jiang, Neil Shah","year":"2021","venue":"Proceedings of the aaai conference on artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Zheng et al., 2020] Chuanpan Zheng, Xiaoliang Fan, Cheng Wang, and Jianzhong Qi. GMAN: A graph multi-attention network for traffic prediction. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 1234-1241. AAAI Press, 2020.","title":"GMAN: A graph multi-attention network for traffic prediction","authors":"Chuanpan Zheng, Xiaoliang Fan, Cheng Wang, Jianzhong Qi","year":"2020","venue":"The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}