{"type":"ScholarlyArticle","title":"A Sparse-Motif Ensemble Graph Convolutional Network against Over-smoothing","authors":[{"name":"Xuan Jiang","slug":"xuan-jiang","given":"Xuan","family":"Jiang","affiliation":"School of Computer Science and Tech. , University of Chinese Academy of Sciences , Beijing , China"},{"name":"Zhiyong Yang","slug":"zhiyong-yang","given":"Zhiyong","family":"Yang","affiliation":"School of Computer Science and Tech. , University of Chinese Academy of Sciences , Beijing , China"},{"name":"Peisong Wen","slug":"peisong-wen","given":"Peisong","family":"Wen","affiliation":"School of Computer Science and Tech. , University of Chinese Academy of Sciences , Beijing , China; Key Lab of Intell. Info. Process. , Institute of Computing Technology , CAS , Beijing , China"},{"name":"Li Su","slug":"li-su","given":"Li","family":"Su","affiliation":"School of Computer Science and Tech. , University of Chinese Academy of Sciences , Beijing , China; Peng Cheng Laboratory , Shenzhen , China"},{"name":"Qingming Huang","slug":"qingming-huang","given":"Qingming","family":"Huang","affiliation":"School of Computer Science and Tech. , University of Chinese Academy of Sciences , Beijing , China; Key Lab of Intell. Info. Process. , Institute of Computing Technology , CAS , Beijing , China; Peng Cheng Laboratory , Shenzhen , China"}],"abstract":"The over-smoothing issue is a well-known challenge for Graph Convolutional Networks (GCN). Specifically, it is often observed that increasing the depth of GCN ends up in a trivial embedding subspace where the difference among node embeddings belonging to the same cluster tends to vanish. This paper believes that the main cause lies in the limited diversity along the message passing pipeline. Inspired by this, we propose a Sparse-Motif Ensemble Graph Convolutional Network (SMEGCN). We argue that merely employing the original graph Laplacian as the spectrum of the graph cannot capture the diversified local structure of complex graphs. Hence, to improve the diversity of the graph spectrum, we introduce local topological structures of complex graphs into GCN by employing the so-called graph motifs or the small network subgraphs. Moreover, we find that the motif connections are much denser than the edge connections, which might converge to an all-one matrix within a few times of message-passing. To fix this, we first propose the notion of sparse motif to avoid spurious motif connections. Subsequently, we propose a hierarchical motif aggregation mechanism to integrate the graph spectral information from a series of different sparse-motif message passing paths. Finally, we conduct a series of theoretical and experimental analyses to demonstrate the superiority of the proposed method.","doi":"10.24963/ijcai.2022/291","url":"https://ijcai.org/proceedings/2022/291","pdf":"https://ijcai.org/proceedings/2022/0291.pdf","markdown":"https://ijcai.org/proceedings/2022/291.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/291","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Data Mining","track":"Main Track","number":291,"pages":{"first":2094,"last":2100},"keywords":["Data Mining: Networks"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986487","length_sec":79,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984630","length_sec":244,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Ahmed et al., 2015] Nesreen K Ahmed, Jennifer Neville, Ryan A Rossi, and Nick Duffield. Efficient graphlet count- ing for large networks. In ICDM, 2015.","title":"Efficient graphlet counting for large networks","authors":"Jennifer Nesreen K Ahmed, Ryan A Neville, Nick Rossi, Duffield","year":"2015","venue":"ICDM","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Benson et al., 2016] Austin R Benson, David F Gleich, and Jure Leskovec. Higher-order organization of complex net- works. Science, 2016.","title":"Higher-order organization of complex networks","authors":"David F Austin R Benson, Jure Gleich, Leskovec","year":"2016","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2020] Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. Simple and deep graph convolutional networks. In ICML, 2020.","title":"Simple and deep graph convolutional networks","authors":"Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, Yaliang Li","year":"2020","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chien et al., 2020] Eli Chien, Jianhao Peng, Pan Li, and Ol- gica Milenkovic. Adaptive universal generalized pagerank graph neural network. In ICLR, 2020.","title":"Adaptive universal generalized pagerank graph neural network","authors":"Eli Chien, Jianhao Peng, Pan Li, Olgica Milenkovic","year":"2020","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Elsner and Tsonis, 1996] James B Elsner and Anastasios A Tsonis. Singular spectrum analysis: a new tool in time series analysis. Springer Science & Business Media, 1996.","title":"Singular spectrum analysis: a new tool in time series analysis","authors":"Tsonis ; Elsner, B James, Anastasios A Elsner, Tsonis","year":"1996","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gao et al., 2020] Difei Gao, Ke Li, Ruiping Wang, Shiguang Shan, and Xilin Chen. Multi-modal graph neu- ral network for joint reasoning on vision and scene text. In CVPR, 2020.","title":"Multi-modal graph neural network for joint reasoning on vision and scene text","authors":"Gao","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Guo and Wang, 2020] Zhiwei Guo and Heng Wang. A deep graph neural network-based mechanism for social recom- mendations. TII, 2020.","title":"A deep graph neural network-based mechanism for social recommendations","authors":"Zhiwei Guo, Heng Wang","year":"2020","venue":"TII","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Huang et al., 2020] Wenbing Huang, Yu Rong, Tingyang Xu, Fuchun Sun, and Junzhou Huang. Tackling over- smoothing for general graph convolutional networks. arXiv preprint arXiv:2008.09864, 2020.","title":"Tackling oversmoothing for general graph convolutional networks","authors":"Wenbing Huang, Yu Rong, Tingyang Xu, Fuchun Sun, Junzhou Huang","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jiang and Luo, 2021] Weiwei Jiang and Jiayun Luo. Graph neural network for traffic forecasting: A survey. arXiv preprint arXiv:2101.11174, 2021.","title":"Graph neural network for traffic forecasting: A survey","authors":"Luo Jiang, Weiwei Jiang, Jiayun Luo","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kipf and Welling, 2017] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In ICLR, 2017.","title":"Semi-supervised classification with graph convolutional networks","authors":"N Thomas, Max Kipf, Welling","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Klicpera et al., 2018] Johannes Klicpera, Aleksandar Bo- jchevski, and Stephan Günnemann. Predict then propa- gate: Graph neural networks meet personalized pagerank. In ICLR, 2018.","title":"Predict then propagate: Graph neural networks meet personalized pagerank","authors":"Klicpera","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Liu and others, 2020] Meng Liu et al. Towards deeper graph neural networks. In SIGKDD, 2020.","title":"Towards deeper graph neural networks","authors":"Meng Liu","year":"2020","venue":"SIGKDD","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Liu et al., 2021] Yujia Liu, Kang Zeng, Haiyang Wang, Xin Song, and Bin Zhou. Content matters: a gnn-based model combined with text semantics for social network cascade prediction. In KDD, 2021.","title":"Content matters: a gnn-based model combined with text semantics for social network cascade prediction","authors":"Yujia Liu, Kang Zeng, Haiyang Wang, Xin Song, Bin Zhou","year":"2021","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Narasimhan et al., 2018] Medhini Narasimhan, Svetlana Lazebnik, and Alexander G Schwing. Out of the box: Reasoning with graph convolution nets for factual visual question answering. In NeurIPS, 2018.","title":"Out of the box: Reasoning with graph convolution nets for factual visual question answering","authors":"Narasimhan","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Pei et al., 2020] Hongbin Pei, Bingzhe Wei, Kevin Chen- Chuan Chang, Yu Lei, and Bo Yang. Geom-gcn: Geo- metric graph convolutional networks. In ICLR, 2020.","title":"Geom-gcn: Geometric graph convolutional networks","authors":"Hongbin Pei, Bingzhe Wei, Kevin Chen-Chuan, Yu Chang, Bo Lei, Yang","year":"2020","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Rahaman and Gagliardi, 2020] Obaidur Rahaman and Alessio Gagliardi. Deep learning total energies and orbital energies of large organic molecules using hybridization of molecular fingerprints. JCIM, 2020.","title":"Deep learning total energies and orbital energies of large organic molecules using hybridization of molecular fingerprints","authors":"Gagliardi Rahaman, Obaidur Rahaman, Alessio Gagliardi","year":"2020","venue":"JCIM","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Shchur et al., 2018] Oleksandr Shchur, Maximil- ian Mumme, Aleksandar Bojchevski, and Stephan Günnemann. Pitfalls of graph neural network evaluation. arXiv preprint arXiv:1811.05868, 2018.","title":"Pitfalls of graph neural network evaluation","authors":"Oleksandr Shchur, Maximilian Mumme, Aleksandar Bojchevski, Stephan Günnemann","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Shen and others, 2021] Zi-Ang Shen et al. Npi-gnn: Pre- dicting ncrna-protein interactions with deep graph neural networks. BIB, 2021.","title":"Npi-gnn: Predicting ncrna-protein interactions with deep graph neural networks","authors":"Zi-Ang Shen","year":"2021","venue":"BIB","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Shui and Karypis, 2020] Zeren Shui and George Karypis. Heterogeneous molecular graph neural networks for pre- dicting molecule properties. In ICDM, 2020.","title":"Heterogeneous molecular graph neural networks for predicting molecule properties","authors":"Karypis Shui, Zeren Shui, George Karypis","year":"2020","venue":"ICDM","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Tsubaki et al., 2019] Masashi Tsubaki, Kentaro Tomii, and Jun Sese. Compound-protein interaction prediction with end-to-end learning of neural networks for graphs and se- quences. Bioinformatics, 2019.","title":"Compound-protein interaction prediction with end-to-end learning of neural networks for graphs and sequences","authors":"Tsubaki","year":"2019","venue":"Bioinformatics","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Veličković et al., 2018] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. In ICLR, 2018.","title":"Graph attention networks","authors":"Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, Yoshua Bengio","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Windels and others, 2019] Sam FL Windels et al. Graphlet laplacians for topology-function and topology-disease re- lationships. Bioinformatics, 2019.","title":"Graphlet laplacians for topology-function and topology-disease relationships","authors":"F L Sam, Windels","year":"2019","venue":"Bioinformatics","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wu et al., 2019] Felix Wu, Amauri Souza, Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Weinberger. Sim- plifying graph convolutional networks. In ICML, 2019.","title":"Simplifying graph convolutional networks","authors":"Felix Wu, Amauri Souza, Tianyi Zhang, Christopher Fifty, Tao Yu, Kilian Weinberger","year":"2019","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Xie et al., 2020] Yi Xie, Yun Xiong, and Yangyong Zhu. Sast-gnn: A self-attention based spatio-temporal graph neural network for traffic prediction. In DASFAA, 2020.","title":"Sast-gnn: A self-attention based spatio-temporal graph neural network for traffic prediction","authors":"Yi Xie, Yun Xiong, Yangyong Zhu","year":"2020","venue":"DASFAA","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Xu et al., 2018] Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, and Stefanie Jegelka. Representation learning on graphs with jumping knowledge networks. In ICML, 2018.","title":"Representation learning on graphs with jumping knowledge networks","authors":"Xu","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhu and others, 2020] Jiong Zhu et al. Beyond homophily in graph neural networks: Current limitations and effective designs. In NeurIPS, 2020.","title":"Beyond homophily in graph neural networks: Current limitations and effective designs","authors":"Jiong Zhu","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}