{"type":"ScholarlyArticle","title":"Gromov-Wasserstein Discrepancy with Local Differential Privacy for Distributed Structural Graphs","authors":[{"name":"Hongwei Jin","slug":"hongwei-jin","given":"Hongwei","family":"Jin","affiliation":"University of Illinois at Chicago"},{"name":"Xun Chen","slug":"xun-chen","given":"Xun","family":"Chen","affiliation":"Samsung Research America ,"}],"abstract":"Learning the similarity between structured data, especially the graphs, is one of the essential problems. Besides the approach like graph kernels, Gromov-Wasserstein (GW) distance recently draws a big attention due to its flexibility to capture both topological and feature characteristics, as well as handling the permutation invariance. However, structured data are widely distributed for different data mining and machine learning applications. With privacy concerns, accessing the decentralized data is limited to either individual clients or different silos. \r\nTo tackle these issues, we propose a privacy-preserving framework to analyze the GW discrepancy of node embedding learned locally from graph neural networks in a federated flavor, and then explicitly place local differential privacy (LDP) based on Multi-bit Encoder to protect sensitive information. Our experiments show that, with strong privacy protection guaranteed by ε-LDP algorithm, the proposed framework not only preserves privacy in graph learning, but also presents a noised structural metric under GW distance, resulting in comparable and even better performance in classification and clustering tasks. Moreover, we reason the rationale behind the LDP-based GW distance analytically and empirically.","doi":"10.24963/ijcai.2022/294","url":"https://ijcai.org/proceedings/2022/294","pdf":"https://ijcai.org/proceedings/2022/0294.pdf","markdown":"https://ijcai.org/proceedings/2022/294.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/294","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Data Mining","track":"Main Track","number":294,"pages":{"first":2115,"last":2121},"keywords":["Data Mining: Federated Learning","Data Mining: Privacy Preserving Data Mining","Data Mining: Mining Graphs"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986542","length_sec":106,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984685","length_sec":269,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Barbe et al., 2020] Amélie Barbe, Marc Sebban, Paulo Gonc ¸alves, Pierre Borgnat, and Rémi Gribonval. Graph diffusion wasserstein distances. In Joint European Conference on Machine Learn- ing and Knowledge Discovery in Databases, pages 577-592. Springer, 2020.","title":"Graph diffusion wasserstein distances","authors":"Barbe","year":"2020","venue":"Joint European Conference on Machine Learning and Knowledge Discovery in Databases","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Borgwardt and Kriegel, 2005] Karsten M Borgwardt and Hans- Peter Kriegel. Shortest-path kernels on graphs. In ICDM, pages 8-pp. IEEE, 2005.","title":"Shortest-path kernels on graphs","authors":"M Karsten, Hans-Peter Borgwardt, Kriegel","year":"2005","venue":"ICDM","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bukaty, 2019] Preston Bukaty. The California Consumer Privacy Act (CCPA): An Implementation Guide. IT Governance Publish- ing, 2019.","title":"The California Consumer Privacy Act (CCPA): An Implementation Guide","authors":"Preston Bukaty","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chan et al., 2000] Tony F Chan, Jason Cong, Tianming Kong, and Joseph R Shinnerl. Multilevel optimization for large-scale circuit placement. In ICCAD, pages 171-176. IEEE, 2000.","title":"Multilevel optimization for large-scale circuit placement","authors":"Jason Tony F Chan, Tianming Cong, Joseph R Kong, Shinnerl","year":"2000","venue":"ICCAD","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Coley et al., 2019] Connor W Coley, Wengong Jin, Luke Rogers, Timothy F Jamison, Tommi S Jaakkola, William H Green, Regina Barzilay, and Klavs F Jensen. A graph-convolutional neural net- work model for the prediction of chemical reactivity. Chemical science, 10(2):370-377, 2019.","title":"A graph-convolutional neural network model for the prediction of chemical reactivity","authors":"Wengong Connor W Coley, Luke Jin, Timothy F Rogers, Tommi S Jamison, William H Jaakkola, Regina Green, Klavs F Barzilay, Jensen","year":"2019","venue":"Chemical science","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Demetci et al., 2020] Pinar Demetci, Rebecca Santorella, Björn Sandstede, William Stafford Noble, and Ritambhara Singh. Gromov-wasserstein optimal transport to align single-cell multi- omics data. bioRxiv preprint bioRxiv:2020.04.28.066787, 2020.","title":"Gromov-wasserstein optimal transport to align single-cell multiomics data","authors":"Pinar Demetci, Rebecca Santorella, Björn Sandstede, William Stafford Noble, Ritambhara Singh","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Fan et al., 2019] Wenqi Fan, Yao Ma, Qing Li, Yuan He, Eric Zhao, Jiliang Tang, and Dawei Yin. Graph neural networks for social recommendation. In The World Wide Web Conference, pages 417-426, 2019.","title":"Graph neural networks for social recommendation","authors":"Wenqi Fan, Yao Ma, Qing Li, Yuan He, Eric Zhao, Jiliang Tang, Dawei Yin","year":"2019","venue":"The World Wide Web Conference","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hamilton et al., 2017] William L Hamilton, Rex Ying, and Jure Leskovec. Inductive representation learning on large graphs. In NeurIPS, pages 1025-1035, 2017.","title":"Inductive representation learning on large graphs","authors":"Rex William L Hamilton, Jure Ying, Leskovec","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[He et al., 2020] Chaoyang He, Songze Li, Jinhyun So, Xiao Zeng, Mi Zhang, Hongyi Wang, Xiaoyang Wang, Praneeth Vepakomma, Abhishek Singh, Hang Qiu, et al. Fedml: A re- search library and benchmark for federated machine learning. arXiv preprint arXiv:2007.13518, 2020.","title":"Fedml: A research library and benchmark for federated machine learning","authors":"He","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[He et al., 2021] Chaoyang He, Keshav Balasubramanian, Emir Ceyani, Yu Rong, Peilin Zhao, Junzhou Huang, Murali An- navaram, and Salman Avestimehr. Fedgraphnn: A federated learning system and benchmark for graph neural networks. arXiv preprint arXiv:2104.07145, 2021.","title":"Fedgraphnn: A federated learning system and benchmark for graph neural networks","authors":"He","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Jin et al., 2017] Chi Jin, Rong Ge, Praneeth Netrapalli, Sham M Kakade, and Michael I Jordan. How to escape saddle points effi- ciently. In ICML, pages 1724-1732. PMLR, 2017.","title":"How to escape saddle points efficiently","authors":"Jin","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Koopmans and Beckmann, 1957] Tjalling C Koopmans and Mar- tin Beckmann. Assignment problems and the location of eco- nomic activities. Econometrica: journal of the Econometric So- ciety, pages 53-76, 1957.","title":"Tjalling C Koopmans and Martin Beckmann. Assignment problems and the location of economic activities","authors":"Beckmann Koopmans","year":"1957","venue":"Econometrica: journal of the Econometric Society","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2014] Mu Li, David G Andersen, Jun Woo Park, Alexan- der J Smola, Amr Ahmed, Vanja Josifovski, James Long, Eu- gene J Shekita, and Bor-Yiing Su. Scaling distributed machine learning with the parameter server. In OSDI, pages 583-598, 2014.","title":"Scaling distributed machine learning with the parameter server","authors":"Mu Li, Jun Woo David G Andersen, Alexander J Park, Amr Smola, Vanja Ahmed, James Josifovski, Eugene J Long, Bor-Yiing Shekita, Su","year":"2014","venue":"OSDI","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li et al., 2020] Xiangfeng Li, Shenghua Liu, Zifeng Li, Xiaotian Han, Chuan Shi, Bryan Hooi, He Huang, and Xueqi Cheng. Flowscope: Spotting money laundering based on graphs. In AIS- TATS, volume 34, pages 4731-4738, 2020.","title":"Flowscope: Spotting money laundering based on graphs","authors":"Xiangfeng Li, Shenghua Liu, Zifeng Li, Xiaotian Han, Chuan Shi, Bryan Hooi, He Huang, Xueqi Cheng","year":"2020","venue":"AIS-TATS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liao et al., 2021] Peiyuan Liao, Han Zhao, Keyulu Xu, Tommi Jaakkola, Geoffrey J Gordon, Stefanie Jegelka, and Ruslan Salakhutdinov. Information obfuscation of graph neural net- works. In International Conference on Machine Learning, pages 6600-6610. PMLR, 2021.","title":"Information obfuscation of graph neural networks","authors":"Liao","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[McMahan et al., 2017] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decen- tralized data. In AISTATS, pages 1273-1282. PMLR, 2017.","title":"Communication-efficient learning of deep networks from decentralized data","authors":"Brendan Mcmahan, Eider Moore, Daniel Ramage, Seth Hampson, Blaise Aguera Y Arcas","year":"2017","venue":"AISTATS","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Mémoli and Sapiro, 2006] Facundo Mémoli and Guillermo Sapiro. Computing with point cloud data. In Statistics and analysis of shapes, pages 201-229. Springer, 2006.","title":"Facundo Mémoli and Guillermo Sapiro. Computing with point cloud data","authors":"Sapiro Mémoli","year":"2006","venue":"Statistics and analysis of shapes","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Mémoli, 2011] Facundo Mémoli. Gromov-wasserstein distances and the metric approach to object matching. Foundations of com- putational mathematics, 11(4):417-487, 2011.","title":"Gromov-wasserstein distances and the metric approach to object matching","authors":"Facundo Mémoli","year":"2011","venue":"Foundations of computational mathematics","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Morris et al., 2020] Christopher Morris, Nils M Kriege, Franka Bause, Kristian Kersting, Petra Mutzel, and Marion Neumann. Tudataset: A collection of benchmark datasets for learning with graphs. arXiv preprint arXiv:2007.08663, 2020.","title":"Tudataset: A collection of benchmark datasets for learning with graphs","authors":"Christopher Morris, Nils M Kriege, Franka Bause, Kristian Kersting, Petra Mutzel, Marion Neumann","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Niu et al., 2020] Chenhao Niu, Yang Song, Jiaming Song, Shengjia Zhao, Aditya Grover, and Stefano Ermon. Permutation invariant graph generation via score-based generative modeling. In AISTATS, pages 4474-4484. PMLR, 2020.","title":"Permutation invariant graph generation via score-based generative modeling","authors":"Chenhao Niu, Yang Song, Jiaming Song, Shengjia Zhao, Aditya Grover, Stefano Ermon","year":"2020","venue":"AISTATS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Peyré et al., 2016] Gabriel Peyré, Marco Cuturi, and Justin Solomon. Gromov-wasserstein averaging of kernel and distance matrices. In ICML, pages 2664-2672. PMLR, 2016.","title":"Gromov-wasserstein averaging of kernel and distance matrices","authors":"Gabriel Peyré, Marco Cuturi, Justin Solomon","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Regulation, 2018] General Data Protection Regulation. General data protection regulation (gdpr). Intersoft Consulting, 24(1), 2018.","title":"General Data Protection Regulation. General data protection regulation (gdpr)","authors":"Regulation","year":"2018","venue":"Intersoft Consulting","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Sajadmanesh and Gatica-Perez, 2020] Sina Sajadmanesh and Daniel Gatica-Perez. Locally private graph neural networks. arXiv preprint arXiv:2006.05535, 2020.","title":"Locally private graph neural networks","authors":"Gatica-Perez ; Sajadmanesh, Sina Sajadmanesh, Daniel Gatica-Perez","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Shervashidze et al., 2011] Nino Shervashidze, Pascal Schweitzer, Erik Jan Van Leeuwen, Kurt Mehlhorn, and Karsten M Borg- wardt. Weisfeiler-lehman graph kernels. Journal of Machine Learning Research, 12(9), 2011.","title":"Weisfeiler-lehman graph kernels","authors":"Nino Shervashidze, Pascal Schweitzer, Erik, Jan Van Leeuwen, Kurt Mehlhorn, Karsten M Borgwardt","year":"2011","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Smith et al., 2017] Virginia Smith, Chao-Kai Chiang, Maziar San- jabi, and Ameet Talwalkar. Federated multi-task learning. In AISTATS, pages 4427-4437, 2017.","title":"Federated multi-task learning","authors":"Smith","year":"2017","venue":"AISTATS","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Titouan et al., 2019] Vayer Titouan, Nicolas Courty, Romain Tave- nard, and Rémi Flamary. Optimal transport for structured data with application on graphs. In ICML, pages 6275-6284. PMLR, 2019.","title":"Optimal transport for structured data with application on graphs","authors":"Nicolas Vayer Titouan, Romain Courty, Rémi Tavenard, Flamary","year":"2019","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Vishwanathan et al., 2010] Vichy Vishwanathan, Nicol Schrau- dolph, Risi Kondor, and Karsten Borgwardt. Graph kernels. JMLR, 11:1201-1242, 2010.","title":"Vichy Vishwanathan","authors":"Vishwanathan","year":"2010","venue":"Graph kernels","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wang et al., 2020] Binghui Wang, Ang Li, Hai Li, and Yiran Chen. Graphfl: A federated learning framework for semi-supervised node classification on graphs. arXiv preprint arXiv:2012.04187, 2020.","title":"Graphfl: A federated learning framework for semi-supervised node classification on graphs","authors":"Binghui Wang, Ang Li, Hai Li, Yiran Chen","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wang et al., 2021] Chunnan Wang, Bozhou Chen, Geng Li, and Hongzhi Wang. Fl-agcns: Federated learning framework for automatic graph convolutional network search. arXiv preprint arXiv:2104.04141, 2021.","title":"Fl-agcns: Federated learning framework for automatic graph convolutional network search","authors":"Wang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wu et al., 2020] Shiwen Wu, Fei Sun, Wentao Zhang, and Bin Cui. Graph neural networks in recommender systems: a survey. arXiv preprint arXiv:2011.02260, 2020.","title":"Graph neural networks in recommender systems: a survey","authors":"Shiwen Wu, Fei Sun, Wentao Zhang, Bin Cui","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Xu et al., 2019a] Hongteng Xu, Dixin Luo, and Lawrence Carin. Scalable gromov-wasserstein learning for graph partitioning and matching. In NeurIPS, pages 3052-3062, 2019.","title":"Scalable gromov-wasserstein learning for graph partitioning and matching","authors":"Xu","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Xu et al., 2019b] Hongteng Xu, Dixin Luo, Hongyuan Zha, and Lawrence Carin Duke. Gromov-wasserstein learning for graph matching and node embedding. In ICML, pages 6932-6941, 2019.","title":"Gromov-wasserstein learning for graph matching and node embedding","authors":"Xu","year":"2019","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Xu et al., 2019c] Keyulu Xu, Weihua Hu, Jure Leskovec, and Ste- fanie Jegelka. How powerful are graph neural networks? In International Conference on Learning Representations (ICLR), 2019.","title":"How powerful are graph neural networks","authors":"Xu","year":"2019","venue":"International Conference on Learning Representations (ICLR)","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Yang et al., 2016] Zhilin Yang, William Cohen, and Ruslan Salakhudinov. Revisiting semi-supervised learning with graph embeddings. In ICML, pages 40-48. PMLR, 2016.","title":"Revisiting semi-supervised learning with graph embeddings","authors":"Zhilin Yang, William Cohen, Ruslan Salakhudinov","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Zügner et al., 2018] Daniel Zügner, Amir Akbarnejad, and Stephan Günnemann. Adversarial attacks on neural networks for graph data. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2847-2856, 2018.","title":"Adversarial attacks on neural networks for graph data","authors":"Daniel Zügner, Amir Akbarnejad, Stephan Günnemann","year":"2018","venue":"Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining","doi":null,"url":null,"internal_url":"/proceedings/2019/872"}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}