{"type":"ScholarlyArticle","title":"Escaping Feature Twist: A Variational Graph Auto-Encoder for Node Clustering","authors":[{"name":"Nairouz Mrabah","slug":"nairouz-mrabah","given":"Nairouz","family":"Mrabah","affiliation":"University of Quebec at Montreal , Montreal , QC , Canada"},{"name":"Mohamed Bouguessa","slug":"mohamed-bouguessa","given":"Mohamed","family":"Bouguessa","affiliation":"University of Quebec at Montreal , Montreal , QC , Canada"},{"name":"Riadh Ksantini","slug":"riadh-ksantini","given":"Riadh","family":"Ksantini","affiliation":"University of Bahrain , Kingdom of Bahrain"}],"abstract":"Most recent graph clustering methods rely on pretraining graph auto-encoders using self-supervision techniques (pretext task) and finetuning based on pseudo-supervision (main task). However, the transition from self-supervision to pseudo-supervision has never been studied from a geometric perspective. Herein, we establish the first systematic exploration of the latent manifolds' geometry under the deep clustering paradigm; we study the evolution of their intrinsic dimension and linear intrinsic dimension. We find that the embedded manifolds undergo coarse geometric transformations under the transition regime: from curved low-dimensional to flattened higher-dimensional. Moreover, we find that this inappropriate flattening leads to clustering deterioration by twisting the curved structures. To address this problem, which we call Feature Twist, we propose a variational graph auto-encoder that can smooth the local curves before gradually flattening the global structures. Our results show a notable improvement over multiple state-of-the-art approaches by escaping Feature Twist.","doi":"10.24963/ijcai.2022/465","url":"https://ijcai.org/proceedings/2022/465","pdf":"https://ijcai.org/proceedings/2022/0465.pdf","markdown":"https://ijcai.org/proceedings/2022/465.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/465","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Machine Learning","track":"Main Track","number":465,"pages":{"first":3351,"last":3357},"keywords":["Machine Learning: Representation learning","Machine Learning: Unsupervised Learning","Machine Learning: Clustering","Machine Learning: Self-supervised Learning","Machine Learning: Autoencoders"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986755","length_sec":101,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984898","length_sec":319,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Ansuini et al., 2019] Alessio Ansuini, Alessandro Laio, Jakob H Macke, and Davide Zoccolan. Intrinsic dimen- sion of data representations in deep neural networks. In NeurIPS, pages 6111-6122, 2019.","title":"Intrinsic dimension of data representations in deep neural networks","authors":"Alessio Ansuini, Alessandro Laio, Jakob H Macke, Davide Zoccolan","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bengio et al., 2013] Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE TPAMI, 35(8):1798-1828, 2013.","title":"Representation learning: A review and new perspectives","authors":"Yoshua Bengio, Aaron Courville, Pascal Vincent","year":"2013","venue":"IEEE TPAMI","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cui et al., 2020] Ganqu Cui, Jie Zhou, Cheng Yang, and Zhiyuan Liu. Adaptive graph encoder for attributed graph embedding. In KDD, pages 976-985, 2020.","title":"Adaptive graph encoder for attributed graph embedding","authors":"Ganqu Cui, Jie Zhou, Cheng Yang, Zhiyuan Liu","year":"2020","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Davies and Bouldin, 1979] David L Davies and Donald W Bouldin. A cluster separation measure. IEEE TPAMI, 1(2):224-227, 1979.","title":"A cluster separation measure","authors":"L David, Donald W Davies, Bouldin","year":"1979","venue":"IEEE TPAMI","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Facco et al., 2017] Elena Facco, Maria d'Errico, Alex Ro- driguez, and Alessandro Laio. Estimating the intrinsic di- mension of datasets by a minimal neighborhood informa- tion. Scientific Reports, 7(1):1-8, 2017.","title":"Estimating the intrinsic dimension of datasets by a minimal neighborhood information","authors":"Elena Facco, Maria Errico, Alex Rodriguez, Alessandro Laio","year":"2017","venue":"Scientific Reports","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hui et al., 2020] Binyuan Hui, Pengfei Zhu, and Qinghua Hu. Collaborative graph convolutional networks: Un- supervised learning meets semi-supervised learning. In AAAI, volume 34, pages 4215-4222, 2020.","title":"Collaborative graph convolutional networks: Unsupervised learning meets semi-supervised learning","authors":"Hui","year":"2020","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Kingma and Welling, 2014] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2014.","title":"Auto-encoding variational bayes","authors":"P Diederik, Max Kingma, Welling","year":"2014","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kipf and Welling, 2016] Thomas N Kipf and Max Welling. Variational graph auto-encoders. In NeurIPS Workshop, pages 1-3, 2016.","title":"Variational graph auto-encoders","authors":"N Thomas, Max Kipf, Welling","year":"2016","venue":"NeurIPS Workshop","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kipf and Welling, 2017] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In ICLR, 2017.","title":"Semi-supervised classification with graph convolutional networks","authors":"N Thomas, Max Kipf, Welling","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Liu et al., 2018] Qi Liu, Miltiadis Allamanis, Marc Brockschmidt, and Alexander L Gaunt. Constrained graph variational autoencoders for molecule design. In NeurIPS, pages 7806-7815, 2018.","title":"Constrained graph variational autoencoders for molecule design","authors":"Qi Liu, Miltiadis Allamanis, Marc Brockschmidt, Alexander L Gaunt","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Liu et al., 2021] Yixin Liu, Shirui Pan, Ming Jin, Chuan Zhou, Feng Xia, and Philip S Yu. Graph self-supervised learning: A survey. arXiv preprint arXiv:2103.00111, 2021.","title":"Graph self-supervised learning: A survey","authors":"Yixin Liu, Shirui Pan, Ming Jin, Chuan Zhou, Feng Xia, Philip S Yu","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Mavromatis and Karypis, 2021] Costas Mavromatis and George Karypis. Graph infoclust: Maximizing coarse- grain mutual information in graphs. In PAKDD, pages 541-553, 2021.","title":"Graph infoclust: Maximizing coarsegrain mutual information in graphs","authors":"Karypis Mavromatis, Costas Mavromatis, George Karypis","year":"2021","venue":"PAKDD","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Mrabah et al., 2020] Nairouz Mrabah, Naimul Mefraz Khan, Riadh Ksantini, and Zied Lachiri. Deep clustering with a dynamic autoencoder: From reconstruction towards centroids construction. Neural Networks, 130:206-228, 2020.","title":"Deep clustering with a dynamic autoencoder: From reconstruction towards centroids construction","authors":"Mrabah","year":"2020","venue":"Neural Networks","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mrabah et al., 2021] Nairouz Mrabah, Mohamed Bouguessa, Mohamed Fawzi Touati, and Riadh Ksantini. Rethinking graph auto-encoder models for attributed graph clustering. arXiv preprint arXiv:2107.08562, 2021.","title":"Rethinking graph auto-encoder models for attributed graph clustering","authors":"Mrabah","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mrabah et al., 2022] Nairouz Mrabah, Mohamed Bouguessa, and Riadh Ksantini. Adversarial deep embedded clustering: on a better trade-off between feature randomness and feature drift. IEEE TKDE, 34(4):1603-1617, 2022.","title":"Nairouz Mrabah, Mohamed Bouguessa, and Riadh Ksantini. Adversarial deep embedded clustering: on a better trade-off between feature randomness and feature drift","authors":"Mrabah","year":"2022","venue":"IEEE TKDE","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Ng et al., 2002] Andrew Y Ng, Michael I Jordan, and Yair Weiss. On spectral clustering: Analysis and an algorithm. In NeurIPS, pages 849-856, 2002.","title":"On spectral clustering: Analysis and an algorithm","authors":"Y Andrew, Michael I Ng, Yair Jordan, Weiss","year":"2002","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Pan et al., 2018] Shirui Pan, Ruiqi Hu, Guodong Long, Jing Jiang, Lina Yao, and Chengqi Zhang. Adversarially regu- larized graph autoencoder for graph embedding. In IJCAI, pages 2609-2615, 2018.","title":"Adversarially regularized graph autoencoder for graph embedding","authors":"Pan","year":"2018","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2018/362"},{"position":18,"raw":"[Perozzi et al., 2014] Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. Deepwalk: Online learning of social rep- resentations. In KDD, pages 701-710, 2014.","title":"Deepwalk: Online learning of social representations","authors":"Bryan Perozzi, Rami Al-Rfou, Steven Skiena","year":"2014","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sen et al., 2008] Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Rad. Collective classification in network data. AI Magazine, 29(3):93-93, 2008.","title":"Collective classification in network data","authors":"Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, Tina Eliassi-Rad","year":"2008","venue":"AI Magazine","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Veličković et al., 2019] Petar Veličković, William Fedus, William L. Hamilton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm. Deep graph infomax. In ICLR, 2019.","title":"Deep graph infomax","authors":"Petar Veličković, William Fedus, William L Hamilton, Pietro Liò, Yoshua Bengio, Devon Hjelm","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang et al., 2019] Chun Wang, Shirui Pan, Ruiqi Hu, Guodong Long, Jing Jiang, and Chengqi Zhang. At- tributed graph clustering: A deep attentional embedding approach. In IJCAI, pages 3670-3676, 2019.","title":"Attributed graph clustering: A deep attentional embedding approach","authors":"Chun Wang, Shirui Pan, Ruiqi Hu, Guodong Long, Jing Jiang, Chengqi Zhang","year":"2019","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2019/509"},{"position":22,"raw":"[Xie et al., 2016] Junyuan Xie, Ross Girshick, and Ali Farhadi. Unsupervised deep embedding for clustering analysis. In ICML, pages 478-487, 2016.","title":"Unsupervised deep embedding for clustering analysis","authors":"Junyuan Xie, Ross Girshick, Ali Farhadi","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yang et al., 2021] Fan Yang, Rui Meng, Hyuna Cho, Guorong Wu, and Won Hwa Kim. Disentangled sequen- tial graph autoencoder for preclinical alzheimer's disease characterizations from adni study. In MICCAI, pages 362- 372, 2021.","title":"Disentangled sequential graph autoencoder for preclinical alzheimer's disease characterizations from adni study","authors":"Fan Yang, Rui Meng, Hyuna Cho, Guorong Wu, Won Hwa, Kim","year":"2021","venue":"MICCAI","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhang et al., 2019] Xiaotong Zhang, Han Liu, Qimai Li, and Xiao-Ming Wu. Attributed graph clustering via adap- tive graph convolution. In IJCAI, pages 4327-4333, 2019.","title":"Attributed graph clustering via adaptive graph convolution","authors":"Xiaotong Zhang, Han Liu, Qimai Li, Xiao-Ming Wu","year":"2019","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2019/601"}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}