{"type":"ScholarlyArticle","title":"Data-Free Adversarial Knowledge Distillation for Graph Neural Networks","authors":[{"name":"Yuanxin Zhuang","slug":"yuanxin-zhuang","given":"Yuanxin","family":"Zhuang","affiliation":"Beijing University of Posts and Telecommunications"},{"name":"Lingjuan Lyu","slug":"lingjuan-lyu","given":"Lingjuan","family":"Lyu","affiliation":"Sony AI"},{"name":"Chuan Shi","slug":"chuan-shi","given":"Chuan","family":"Shi","affiliation":"Beijing University of Posts and Telecommunications"},{"name":"Carl Yang","slug":"carl-yang","given":"Carl","family":"Yang","affiliation":"Emory University"},{"name":"Lichao Sun","slug":"lichao-sun","given":"Lichao","family":"Sun","affiliation":"Lehigh University"}],"abstract":"Graph neural networks (GNNs) have been widely used in modeling graph structured data, owing to its impressive performance in a wide range of practical applications. Recently, knowledge distillation (KD) for GNNs has enabled remarkable progress in graph model compression and knowledge transfer. However, most of the existing KD methods require a large volume of real data, which are not readily available in practice, and may preclude their applicability in scenarios where the teacher model is trained on rare or hard to acquire datasets. To address this problem, we propose the first end-to-end framework for data-free adversarial knowledge distillation on graph structured data (DFAD-GNN). To be specific, our DFAD-GNN employs a generative adversarial network, which mainly consists of three components: a pre-trained teacher model and a student model are regarded as two discriminators, and a generator is utilized for deriving training graphs to distill knowledge from the teacher model into the student model. Extensive experiments on various benchmark models and six representative datasets demonstrate that our DFAD-GNN significantly surpasses state-of-the-art data-free baselines in the graph classification task.","doi":"10.24963/ijcai.2022/339","url":"https://ijcai.org/proceedings/2022/339","pdf":"https://ijcai.org/proceedings/2022/0339.pdf","markdown":"https://ijcai.org/proceedings/2022/339.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/339","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Data Mining","track":"Main Track","number":339,"pages":{"first":2441,"last":2447},"keywords":["Machine Learning: Adversarial Machine Learning","Data Mining: Mining Graphs","Machine Learning: Classification"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986645","length_sec":81,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984788","length_sec":273,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Arjovsky and Bottou, 2017] Martin Arjovsky and Léon Bottou. Towards principled methods for training generative adversarial networks. arXiv preprint arXiv:1701.04862, 2017.","title":"Towards principled methods for training generative adversarial networks","authors":"Bottou Arjovsky, Martin Arjovsky, Léon Bottou","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bahri et al., 2021] Mehdi Bahri, Gaétan Bahl, and Stefanos Zafeiriou. Binary graph neural networks. In CVPR, pages 9492-9501, 2021.","title":"Binary graph neural networks","authors":"Mehdi Bahri, Gaétan Bahl, Stefanos Zafeiriou","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bojchevski et al., 2018] Aleksandar Bojchevski, Oleksandr Shchur, Daniel Zügner, and Stephan Günnemann. Net- gan: Generating graphs via random walks. In ICML, pages 610-619. PMLR, 2018.","title":"Netgan: Generating graphs via random walks","authors":"Aleksandar Bojchevski, Oleksandr Shchur, Daniel Zügner, Stephan Günnemann","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chang and Lin, 2011] Chih-Chung Chang and Chih-Jen Lin. Libsvm: a library for support vector machines. ACM transactions on intelligent systems and technology (TIST), 2(3):1-27, 2011.","title":"Libsvm: a library for support vector machines","authors":"Chang, Lin ; ] Chih-Chung Chang, Chih-Jen Lin","year":"2011","venue":"ACM transactions on intelligent systems and technology","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[De Cao and Kipf, 2018] Nicola De Cao and Thomas Kipf. Molgan: An implicit generative model for small molecular graphs. arXiv preprint arXiv:1805.11973, 2018.","title":"Molgan: An implicit generative model for small molecular graphs","authors":"Nicola De, Cao, Thomas Kipf","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Deng and Zhang, 2021] Xiang Deng and Zhongfei Zhang. Graph-free knowledge distillation for graph neural net- works. ArXiv, abs/2105.07519, 2021.","title":"Graph-free knowledge distillation for graph neural networks","authors":"Xiang Deng, Zhongfei Zhang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2021/320"},{"position":7,"raw":"[Fang et al., 2019] Gongfan Fang, Jie Song, Chengchao Shen, Xinchao Wang, Da Chen, and Mingli Song. Data-free adversarial distillation. arXiv preprint arXiv:1912.11006, 2019.","title":"Data-free adversarial distillation","authors":"Gongfan Fang, Jie Song, Chengchao Shen, Xinchao Wang, Da Chen, Mingli Song","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Fang et al., 2021] Gongfan Fang, Kanya Mo, Xinchao Wang, Jie Song, Shitao Bei, Haofei Zhang, and Mingli Song. Up to 100x faster data-free knowledge distillation. arXiv preprint arXiv:2112.06253, 2021.","title":"Up to 100x faster data-free knowledge distillation","authors":"Fang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gilmer et al., 2017] Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In ICML, pages 1263-1272. PMLR, 2017.","title":"Neural message passing for quantum chemistry","authors":"Justin Gilmer, S Samuel, Patrick F Schoenholz, Oriol Riley, George E Vinyals, Dahl","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Goodfellow et al., 2014] Ian Goodfellow, Jean Pouget- Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Gen- erative adversarial nets. NeurIPS, 27, 2014.","title":"Generative adversarial nets","authors":"Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio","year":"2014","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Gou et al., 2021] Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey. International Journal of Computer Vision, 129(6):1789-1819, 2021.","title":"Knowledge distillation: A survey","authors":"Jianping Gou, Baosheng Yu, Stephen J Maybank, Dacheng Tao","year":"2021","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hamilton et al., 2017] William L Hamilton, Rex Ying, and Jure Leskovec. Inductive representation learning on large graphs. In Proceedings of the 31st International Con- ference on Neural Information Processing Systems, pages 1025-1035, 2017.","title":"Inductive representation learning on large graphs","authors":"Rex William L Hamilton, Jure Ying, Leskovec","year":"2017","venue":"Proceedings of the 31st International Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hinton et al., 2015] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.","title":"Distilling the knowledge in a neural network","authors":"Geoffrey Hinton, Oriol Vinyals, Jeff Dean","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Jing et al., 2021] Yongcheng Jing, Yiding Yang, Xinchao Wang, Mingli Song, and Dacheng Tao. Amalgamating knowledge from heterogeneous graph neural networks. In CVPR, pages 15709-15718, 2021.","title":"Amalgamating knowledge from heterogeneous graph neural networks","authors":"Jing","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kipf and Welling, 2016] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907, 2016.","title":"Semi-supervised classification with graph convolutional networks","authors":"N Thomas, Max Kipf, Welling","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liu et al., 2021] Yuang Liu, Wei Zhang, Jun Wang, and Jianyong Wang. Data-free knowledge transfer: A survey. arXiv preprint arXiv:2112.15278, 2021.","title":"Data-free knowledge transfer: A survey","authors":"Yuang Liu, Wei Zhang, Jun Wang, Jianyong Wang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lopes et al., 2017] Raphael Gontijo Lopes, Stefano Fenu, and Thad Starner. Data-free knowledge distillation for deep neural networks. arXiv preprint arXiv:1710.07535, 2017.","title":"Data-free knowledge distillation for deep neural networks","authors":"Gontijo Raphael, Stefano Lopes, Thad Fenu, Starner","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lu et al., 2021] Yuanfu Lu, Xunqiang Jiang, Yuan Fang, and Chuan Shi. Learning to pre-train graph neural net- works. In AAAI, volume 35, pages 4276-4284, 2021.","title":"Learning to pre-train graph neural networks","authors":"Lu","year":"2021","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Niepert et al., 2016] Mathias Niepert, Mohamed Ahmed, and Konstantin Kutzkov. Learning convolutional neural networks for graphs. In International conference on ma- chine learning, pages 2014-2023. PMLR, 2016.","title":"Learning convolutional neural networks for graphs","authors":"Mathias Niepert, Mohamed Ahmed, Konstantin Kutzkov","year":"2016","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Sun et al., 2018] Lichao Sun, Yingtong Dou, Carl Yang, Ji Wang, Philip S Yu, Lifang He, and Bo Li. Adversarial attack and defense on graph data: A survey. arXiv preprint arXiv:1812.10528, 2018.","title":"Adversarial attack and defense on graph data: A survey","authors":"Sun","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Veličković et al., 2017] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv:1710.10903, 2017.","title":"Graph attention networks","authors":"Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, Yoshua Bengio","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wu et al., 2020] Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip. A com- prehensive survey on graph neural networks. IEEE trans- actions on neural networks and learning systems, 32(1):4- 24, 2020.","title":"A comprehensive survey on graph neural networks","authors":"Wu","year":"2020","venue":"IEEE transactions on neural networks and learning systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wu et al., 2021] Chuhan Wu, Fangzhao Wu, Yang Cao, Lingjuan Lyu, Yongfeng Huang, and Xing Xie. Fedgnn: Federated graph neural network for privacy-preserving recommendation. arXiv preprint arXiv:2102.04925, 2021.","title":"Fedgnn: Federated graph neural network for privacy-preserving recommendation","authors":"Wu","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wu et al., 2022] Chuhan Wu, Fangzhao Wu, Lingjuan Lyu, Yongfeng Huang, and Xing Xie. Communication-efficient federated learning via knowledge distillation. Nature Communications, 13(1):1-8, 2022.","title":"Communication-efficient federated learning via knowledge distillation","authors":"Wu","year":"2022","venue":"Nature Communications","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Xu et al., 2018] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural net- works? arXiv preprint arXiv:1810.00826, 2018.","title":"How powerful are graph neural networks","authors":"Xu","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yang et al., 2021] Cheng Yang, Jiawei Liu, and Chuan Shi. Extract the knowledge of graph neural networks and go beyond it: An effective knowledge distillation framework. In WWW, pages 1227-1237, 2021.","title":"Extract the knowledge of graph neural networks and go beyond it: An effective knowledge distillation framework","authors":"Cheng Yang, Jiawei Liu, Chuan Shi","year":"2021","venue":"WWW","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang et al., 2018] Muhan Zhang, Zhicheng Cui, Marion Neumann, and Yixin Chen. An end-to-end deep learn- ing architecture for graph classification. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.","title":"An end-to-end deep learning architecture for graph classification","authors":"Zhang","year":"2018","venue":"Thirty-Second AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhou et al., 2020] Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. Graph neural networks: A review of methods and applications. AI Open, 1:57-81, 2020.","title":"Graph neural networks: A review of methods and applications","authors":"Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, Maosong Sun","year":"2020","venue":"AI Open","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}