{"type":"ScholarlyArticle","title":"Private Semi-Supervised Federated Learning","authors":[{"name":"Chenyou Fan","slug":"chenyou-fan","given":"Chenyou","family":"Fan","affiliation":"School of Artificial Intelligence , South China Normal University , China"},{"name":"Junjie Hu","slug":"junjie-hu","given":"Junjie","family":"Hu","affiliation":"Shenzhen Institute of Artificial Intelligence and Robotics for Society , China"},{"name":"Jianwei Huang","slug":"jianwei-huang","given":"Jianwei","family":"Huang","affiliation":"Shenzhen Institute of Artificial Intelligence and Robotics for Society , China; School of Science and Engineering , The Chinese University of Hong Kong , Shenzhen , China"}],"abstract":"We study a federated learning (FL) framework to effectively train models from scarce and skewly distributed labeled data. We consider a challenging yet practical scenario: a few data sources own a  small amount of labeled data, while the rest mass sources own purely unlabeled data. Classical FL requires each client to have enough labeled data for local training, thus is not applicable in this scenario. In this work, we design an effective federated semi-supervised learning framework (FedSSL) to fully leverage both labeled and unlabeled data sources. We establish a unified data space across all participating agents, so that each agent can generate mixed data samples to boost semi-supervised learning (SSL), while keeping data locality. We further show that FedSSL can integrate differential privacy protection techniques to prevent labeled data leakage at the cost of minimum performance degradation. On SSL tasks with as small as 0.17% and 1% of MNIST and CIFAR-10 datasets as labeled data, respectively, our approach can achieve 5-20% performance boost over the state-of-the-art methods.","doi":"10.24963/ijcai.2022/279","url":"https://ijcai.org/proceedings/2022/279","pdf":"https://ijcai.org/proceedings/2022/0279.pdf","markdown":"https://ijcai.org/proceedings/2022/279.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/279","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Data Mining","track":"Main Track","number":279,"pages":{"first":2009,"last":2015},"keywords":["Data Mining: Federated Learning","Machine Learning: Generative Adverserial Networks","Machine Learning: Semi-Supervised Learning","Data Mining: Privacy Preserving Data Mining"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986477","length_sec":91,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984620","length_sec":321,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Abadi et al., 2016] Martin Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In SIGSAC, 2016.","title":"Deep learning with differential privacy","authors":"Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan Mcmahan, Ilya Mironov, Kunal Talwar, Li Zhang","year":"2016","venue":"SIGSAC","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Berthelot et al., 2019] David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raf- fel. Mixmatch: A holistic approach to semi-supervised learning. NeurIPS, 2019.","title":"Mixmatch: A holistic approach to semi-supervised learning","authors":"David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, Colin Raffel","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Caldas et al., 2018] Sebastian Caldas, Peter Wu, Tian Li, Jakub Konecný, H. Brendan McMahan, Virginia Smith, and Ameet Tal- walkar. Leaf: A benchmark for federated settings. arXiv preprint arXiv:1812.01097, 2018.","title":"Leaf: A benchmark for federated settings","authors":"Sebastian Caldas, Peter Wu, Tian Li, Jakub Konecný, H Brendan Mcmahan, Virginia Smith, Ameet Talwalkar","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chapelle et al., 2006] Olivier Chapelle, Bernhard Schölkopf, and Alexander Zien. Semi-Supervised Learning. 2006.","title":"Semi-Supervised Learning","authors":"Olivier Chapelle, Bernhard Schölkopf, Alexander Zien","year":"2006","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen and et al., 2018] Fei Chen and et al. Federated meta-learning with fast convergence and efficient communication. arXiv preprint arXiv:1802.07876, 2018.","title":"Federated meta-learning with fast convergence and efficient communication","authors":"Chen","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dwork and Roth, 2014] Cynthia Dwork and Aaron Roth. The al- gorithmic foundations of differential privacy. Found. Trends Theor. Comput. Sci., 9(3-4):211-407, 2014.","title":"The algorithmic foundations of differential privacy","authors":"Roth ; Dwork, Cynthia Dwork, Aaron Roth","year":"2014","venue":"Found. Trends Theor. Comput. Sci","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Fan and Huang, 2021] Chenyou Fan and Jianwei Huang. Feder- ated few-shot learning with adversarial learning. arXiv preprint arXiv:2104.00365, 2021.","title":"Chenyou Fan and Jianwei Huang. Federated few-shot learning with adversarial learning","authors":"Fan, Huang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Fan and Liu, 2020] Chenyou Fan and Ping Liu. Federated gen- erative adversarial learning. In Chinese Conference on Pattern Recognition and Computer Vision (PRCV), 2020.","title":"Chenyou Fan and Ping Liu. Federated generative adversarial learning","authors":"Fan, Liu","year":"2020","venue":"Chinese Conference on Pattern Recognition and Computer Vision (PRCV)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Goodfellow et al., 2014] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurIPS, 2014.","title":"Generative adversarial nets","authors":"Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio","year":"2014","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Grandvalet and Bengio, 2004] Yves Grandvalet and Yoshua Ben- gio. Semi-supervised learning by entropy minimization. In NIPS, 2004.","title":"Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization","authors":"Bengio Grandvalet","year":"2004","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Itahara et al., 2020] Sohei Itahara, Takayuki Nishio, Yusuke Koda, Masahiro Morikura, and Koji Yamamoto. Distillation-based semi-supervised federated learning for communication-efficient collaborative training with non-iid private data. arXiv preprint arXiv:2008.06180, 2020.","title":"Distillation-based semi-supervised federated learning for communication-efficient collaborative training with non-iid private data","authors":"Sohei Itahara, Takayuki Nishio, Yusuke Koda, Masahiro Morikura, Koji Yamamoto","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Jeong et al., 2020] Wonyong Jeong, Jaehong Yoon, Eunho Yang, and Sung Ju Hwang. Federated semi-supervised learning with inter-client consistency. arXiv preprint arXiv:2006.12097, 2020.","title":"Federated semi-supervised learning with inter-client consistency","authors":"Jeong","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Jin et al., 2020a] Yilun Jin, Xiguang Wei, Yang Liu, and Qiang Yang. A survey towards federated semi-supervised learning. arXiv preprint arXiv:2002.11545, 2020.","title":"A survey towards federated semi-supervised learning","authors":"Jin","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Jin et al., 2020b] Yilun Jin, Xiguang Wei, Yang Liu, and Qiang Yang. Towards utilizing unlabeled data in federated learning: A survey and prospective. arXiv preprint arXiv:2002.11545, 2020.","title":"Towards utilizing unlabeled data in federated learning: A survey and prospective","authors":"Jin","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Krizhevsky, 2009] Alex Krizhevsky. Learning multiple layers of features from tiny images. Master's thesis, University of Tront, 2009.","title":"Alex Krizhevsky. Learning multiple layers of features from tiny images","authors":"Krizhevsky","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Laine and Aila, 2017] Samuli Matias Laine and Timo Oskari Aila. Temporal ensembling for semi-supervised learning. In ICLR, 2017.","title":"Temporal ensembling for semi-supervised learning","authors":"Laine, Aila ; Samuli, Matias Laine, Timo Oskari, Aila","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[LeCun et al., 1998] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to docu- ment recognition. Proceedings of the IEEE, 1998.","title":"Gradient-based learning applied to document recognition","authors":"Yann Lecun, Léon Bottou, Yoshua Bengio, Patrick Haffner","year":"1998","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lee and others, 2013] Dong-Hyun Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In ICML Workshop, 2013.","title":"Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks","authors":"Dong-Hyun Lee","year":"2013","venue":"ICML Workshop","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Li and others, 2019] Tian Li et al. Federated learning: Challenges, methods, and future directions. arXiv preprint arXiv:1908.07873, 2019.","title":"Federated learning: Challenges, methods, and future directions","authors":"Tian Li","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[McMahan et al., 2017] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas. Communication-efficient learning of deep networks from decen- tralized data. In AISTATS, 2017.","title":"Communication-efficient learning of deep networks from decentralized data","authors":"Brendan Mcmahan, Eider Moore, Daniel Ramage, Seth Hampson, Blaise Agüera Y Arcas","year":"2017","venue":"AISTATS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Sajjadi et al., 2016] Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transformations and per- turbations for deep semi-supervised learning. In NIPS, 2016.","title":"Regularization with stochastic transformations and perturbations for deep semi-supervised learning","authors":"Mehdi Sajjadi, Mehran Javanmardi, Tolga Tasdizen","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sattler et al., 2019] Felix Sattler, Simon Wiedemann, Klaus- Robert Müller, and Wojciech Samek. Robust and communication-efficient federated learning from non-iid data. IEEE TNNLS, 2019.","title":"Robust and communication-efficient federated learning from non-iid data","authors":"Felix Sattler, Simon Wiedemann, Klaus-Robert Müller, Wojciech Samek","year":"2019","venue":"IEEE TNNLS","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Sohn et al., 2020] Kihyuk Sohn, David Berthelot, Nicholas Car- lini, Zizhao Zhang, Han Zhang, Colin A. Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simpli- fying semi-supervised learning with consistency and confidence. In NeurIPS, 2020.","title":"Fixmatch: Simplifying semi-supervised learning with consistency and confidence","authors":"Sohn","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wei and others, 2020] Kang Wei et al. Federated learning with dif- ferential privacy: Algorithms and performance analysis. IEEE Transactions on Information Forensics and Security, 2020.","title":"Federated learning with differential privacy: Algorithms and performance analysis","authors":"Kang Wei","year":"2020","venue":"IEEE Transactions on Information Forensics and Security","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wu et al., 2020] Qiong Wu, Kaiwen He, and Xu Chen. Personal- ized federated learning for intelligent iot applications: A cloud- edge based framework. IEEE Computer Graphics and Applica- tions, 2020.","title":"Personalized federated learning for intelligent iot applications: A cloudedge based framework","authors":"Qiong Wu, Kaiwen He, Xu Chen","year":"2020","venue":"IEEE Computer Graphics and Applications","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Xin et al., 2020] Bangzhou Xin, Wei Yang, Yangyang Geng, Sheng Chen, Shaowei Wang, and Liusheng Huang. Private fl- gan: Differential privacy synthetic data generation based on fed- erated learning. In ICASSP, 2020.","title":"Private flgan: Differential privacy synthetic data generation based on federated learning","authors":"Wei Bangzhou Xin, Yangyang Yang, Sheng Geng, Shaowei Chen, Liusheng Wang, Huang","year":"2020","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang et al., 2018] Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR, 2018.","title":"mixup: Beyond empirical risk minimization","authors":"Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, David Lopez-Paz","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhang et al., 2020] Zhengming Zhang, Yaoqing Yang, Zhewei Yao, Yujun Yan, Joseph E Gonzalez, and Michael W Mahoney. Improving semi-supervised federated learning by reducing the gradient diversity of models. arXiv preprint arXiv:2008.11364, 2020.","title":"Improving semi-supervised federated learning by reducing the gradient diversity of models","authors":"Zhengming Zhang, Yaoqing Yang, Zhewei Yao, Yujun Yan, Joseph E Gonzalez, Michael W Mahoney","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhao et al., 2018] Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-iid data. arXiv preprint arXiv:1806.00582, 2018.","title":"Federated learning with non-iid data","authors":"Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, Vikas Chandra","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhou and Li, 2010] Zhi-Hua Zhou and Ming Li. Semi-supervised learning by disagreement. Knowledge and Information Systems, 2010.","title":"Semi-supervised learning by disagreement","authors":"Li ; ] Zhi-Hua Zhou, Ming Zhou, Li","year":"2010","venue":"Knowledge and Information Systems","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhu et al., 2020] Jianchao Zhu, Liangliang Shi, Junchi Yan, and Hongyuan Zha. Automix: Mixup networks for sample interpola- tion via cooperative barycenter learning. ECCV, 2020.","title":"Automix: Mixup networks for sample interpolation via cooperative barycenter learning","authors":"Jianchao Zhu, Liangliang Shi, Junchi Yan, Hongyuan Zha","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}