{"type":"ScholarlyArticle","title":"Estimating before Debiasing: A Bayesian Approach to Detaching Prior Bias in Federated Semi-Supervised Learning","authors":[{"name":"Guogang Zhu","slug":"guogang-zhu","given":"Guogang","family":"Zhu","affiliation":"State Key Laboratory of Virtual Reality Technology and Systems , School of Computer Science and Engineering , Beihang University , Beijing , China"},{"name":"Xuefeng Liu","slug":"xuefeng-liu","given":"Xuefeng","family":"Liu","affiliation":"State Key Laboratory of Virtual Reality Technology and Systems , School of Computer Science and Engineering , Beihang University , Beijing , China; Zhongguancun Laboratory , Beijing , China"},{"name":"Xinghao Wu","slug":"xinghao-wu","given":"Xinghao","family":"Wu","affiliation":"State Key Laboratory of Virtual Reality Technology and Systems , School of Computer Science and Engineering , Beihang University , Beijing , China"},{"name":"Shaojie Tang","slug":"shaojie-tang","given":"Shaojie","family":"Tang","affiliation":"Jindal School of Management , The University of Texas at Dallas , Richardson , TX , USA"},{"name":"Chao Tang","slug":"chao-tang","given":"Chao","family":"Tang","affiliation":"State Key Laboratory of Virtual Reality Technology and Systems , School of Computer Science and Engineering , Beihang University , Beijing , China"},{"name":"Jianwei Niu","slug":"jianwei-niu","given":"Jianwei","family":"Niu","affiliation":"State Key Laboratory of Virtual Reality Technology and Systems , School of Computer Science and Engineering , Beihang University , Beijing , China; Zhongguancun Laboratory , Beijing , China"},{"name":"Hao Su","slug":"hao-su","given":"Hao","family":"Su","affiliation":"State Key Laboratory of Virtual Reality Technology and Systems , School of Computer Science and Engineering , Beihang University , Beijing , China"}],"abstract":"Federated Semi-Supervised Learning (FSSL) leverages both labeled and unlabeled data on clients to collaboratively train a model. \r\nIn FSSL, the heterogeneous data can introduce prediction bias into the model, causing the model's prediction to skew towards some certain classes. \r\nExisting FSSL methods primarily tackle this issue by enhancing consistency in model parameters or outputs. However, as the models themselves are biased, merely constraining their consistency is not sufficient to alleviate prediction bias. In this paper, we explore this bias from a Bayesian perspective and demonstrate that it principally originates from label prior bias within the training data. Building upon this insight, we propose a debiasing method for FSSL named FedDB. FedDB utilizes the Average Prediction Probability of Unlabeled Data (APP-U) to approximate the biased prior. During local training, FedDB employs APP-U to refine pseudo-labeling through Bayes' theorem, thereby significantly reducing the label prior bias. Concurrently, during the model aggregation, FedDB uses APP-U from participating clients to formulate unbiased aggregate weights, thereby effectively diminishing bias in the global model. \r\nExperimental results show that FedDB can surpass existing FSSL methods. The code is available at https://github.com/GuogangZhu/FedDB.","doi":"10.24963/ijcai.2024/290","url":"https://ijcai.org/proceedings/2024/290","pdf":"https://ijcai.org/proceedings/2024/0290.pdf","markdown":"https://ijcai.org/proceedings/2024/290.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/290","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Data Mining","track":"Main Track","number":290,"pages":{"first":2625,"last":2633},"keywords":["Machine Learning: ML: Semi-supervised learning","Data Mining: DM: Parallel, distributed and cloud-based high performance mining","Data Mining: DM: Privacy-preserving data mining"],"videos":[],"references":[{"position":1,"raw":"[Acar et al., 2020] Durmus Alp Emre Acar, Yue Zhao, Ra- mon Matas, Matthew Mattina, Paul Whatmough, and Venkatesh Saligrama. Federated learning based on dy- namic regularization. In International Conference on Learning Representations, 2020.","title":"Federated learning based on dynamic regularization","authors":"Durmus Alp, Emre Acar, Yue Zhao, Ramon Matas, Matthew Mattina, Paul Whatmough, Venkatesh Saligrama","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bai et al., 2023] Sikai Bai, Shuaicheng Li, Weiming Zhuang, Kunlin Yang, Jun Hou, Shuai Yi, Shuai Zhang, Junyu Gao, Jie Zhang, and Song Guo. Combating data imbalances in federated semi-supervised learning with dual regulators. arXiv preprint arXiv:2307.05358, 2023.","title":"Combating data imbalances in federated semi-supervised learning with dual regulators","authors":"Bai","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Berthelot et al., 2019] David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach to semi- supervised learning. Advances in neural information pro- cessing systems, 32, 2019.","title":"Mixmatch: A holistic approach to semisupervised learning","authors":"David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, Colin A Raffel","year":"2019","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Collins et al., 2021] Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. Exploiting shared repre- sentations for personalized federated learning. In Interna- tional conference on machine learning, pages 2089-2099.","title":"Exploiting shared representations for personalized federated learning","authors":"Liam Collins, Hamed Hassani, Aryan Mokhtari, Sanjay Shakkottai","year":null,"venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"PMLR, 2021.","title":null,"authors":"Pmlr","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Diao et al., 2022] Enmao Diao, Jie Ding, and Vahid Tarokh. Semifl: Semi-supervised federated learning for unlabeled clients with alternate training. Advances in Neural Infor- mation Processing Systems, 35:17871-17884, 2022.","title":"Semifl: Semi-supervised federated learning for unlabeled clients with alternate training","authors":"Diao","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Guo and Li, 2022] Lan-Zhe Guo and Yu-Feng Li. Class- imbalanced semi-supervised learning with adaptive thresholding. In International Conference on Machine Learning, pages 8082-8094. PMLR, 2022.","title":"Classimbalanced semi-supervised learning with adaptive thresholding","authors":"Lan-Zhe Guo, Yu-Feng Li","year":"2022","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hong et al., 2021] Youngkyu Hong, Seungju Han, Kwanghee Choi, Seokjun Seo, Beomsu Kim, and Buru Chang. Disentangling label distribution for long- tailed visual recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6626-6636, 2021.","title":"Disentangling label distribution for longtailed visual recognition","authors":"Hong","year":"2021","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jeong et al., 2021] Wonyong Jeong, Jaehong Yoon, Eunho Yang, and Sung Ju Hwang. Federated semi-supervised learning with inter-client consistency & disjoint learning. In International Conference on Learning Representations, 2021.","title":"Federated semi-supervised learning with inter-client consistency & disjoint learning","authors":"Jeong","year":"2021","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jiang et al., 2022] Meirui Jiang, Hongzheng Yang, Xiaox- iao Li, Quande Liu, Pheng-Ann Heng, and Qi Dou. Dy- namic bank learning for semi-supervised federated im- age diagnosis with class imbalance. In International Conference on Medical Image Computing and Computer- Assisted Intervention, pages 196-206. Springer, 2022.","title":"Dynamic bank learning for semi-supervised federated image diagnosis with class imbalance","authors":"Jiang","year":"2022","venue":"International Conference on Medical Image Computing and Computer-Assisted Intervention","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kairouz et al., 2021] Peter Kairouz, H Brendan McMa- han, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Ar- jun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. Foundations and Trends® in Machine Learning, 14(1-2):1-210, 2021.","title":"Advances and open problems in federated learning","authors":"Peter Kairouz, Brendan Mcmahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Nitin Arjun, Kallista Bhagoji, Zachary Bonawitz, Graham Charles, Rachel Cormode, Cummings","year":"2021","venue":"Foundations and Trends® in Machine Learning","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Karimireddy et al., 2020] Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In Interna- tional conference on machine learning, pages 5132-5143.","title":"Scaffold: Stochastic controlled averaging for federated learning","authors":"Praneeth Sai, Satyen Karimireddy, Mehryar Kale, Sashank Mohri, Sebastian Reddi, Ananda Stich, Suresh Theertha","year":null,"venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"PMLR, 2020.","title":null,"authors":"Pmlr","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lee and others, 2013] Dong-Hyun Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML, page 896, 2013.","title":"Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks","authors":"Dong-Hyun Lee","year":"2013","venue":"Workshop on challenges in representation learning","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2020a] Tian Li, Anit Kumar Sahu, Ameet Tal- walkar, and Virginia Smith. Federated learning: Chal- lenges, methods, and future directions. IEEE signal pro- cessing magazine, 37(3):50-60, 2020.","title":"Federated learning: Challenges, methods, and future directions","authors":"Li","year":"2020","venue":"IEEE signal processing magazine","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Li et al., 2020b] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Pro- ceedings of Machine learning and systems, 2:429-450, 2020.","title":"Federated optimization in heterogeneous networks","authors":"Li","year":"2020","venue":"Proceedings of Machine learning and systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Li et al., 2023] Ming Li, Qingli Li, and Yan Wang. Class balanced adaptive pseudo labeling for federated semi- supervised learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16292-16301, 2023.","title":"Class balanced adaptive pseudo labeling for federated semisupervised learning","authors":"Ming Li, Qingli Li, Yan Wang","year":"2023","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Liang et al., 2022] Xiaoxiao Liang, Yiqun Lin, Huazhu Fu, Lei Zhu, and Xiaomeng Li. Rscfed: random sampling con- sensus federated semi-supervised learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 10154-10163, 2022.","title":"Rscfed: random sampling consensus federated semi-supervised learning","authors":"Xiaoxiao Liang, Yiqun Lin, Huazhu Fu, Lei Zhu, Xiaomeng Li","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Liao et al., 2023] Xinting Liao, Weiming Liu, Chaochao Chen, Pengyang Zhou, Huabin Zhu, Yanchao Tan, Jun Wang, and Yue Qi. Hyperfed: hyperbolic prototypes exploration with consistent aggregation for non-iid data in federated learning. arXiv preprint arXiv:2307.14384, 2023.","title":"Hyperfed: hyperbolic prototypes exploration with consistent aggregation for non-iid data in federated learning","authors":"Liao","year":"2023","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2023/440"},{"position":20,"raw":"[Lin et al., 2021] Haowen Lin, Jian Lou, Li Xiong, and Cyrus Shahabi. Semifed: Semi-supervised federated learning with consistency and pseudo-labeling. arXiv preprint arXiv:2108.09412, 2021.","title":"Semifed: Semi-supervised federated learning with consistency and pseudo-labeling","authors":"Lin","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Lin, 1991] Jianhua Lin. Divergence measures based on the shannon entropy. IEEE Transactions on Information the- ory, 37(1):145-151, 1991.","title":"Divergence measures based on the shannon entropy","authors":"Jianhua Lin","year":"1991","venue":"IEEE Transactions on Information theory","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Liu et al., 2023] Jiahao Liu, Jiang Wu, Jinyu Chen, Miao Hu, Yipeng Zhou, and Di Wu. Feddwa: personalized fed- erated learning with dynamic weight adjustment. In Pro- ceedings of the Thirty-Second International Joint Confer- ence on Artificial Intelligence, pages 3993-4001, 2023.","title":"Feddwa: personalized federated learning with dynamic weight adjustment","authors":"Jiahao Liu, Jiang Wu, Jinyu Chen, Miao Hu, Yipeng Zhou, Di Wu","year":"2023","venue":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2023/444"},{"position":23,"raw":"[McMahan et al., 2017] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Ar- cas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273-1282. PMLR, 2017.","title":"Communication-efficient learning of deep networks from decentralized data","authors":"Brendan Mcmahan, Eider Moore, Daniel Ramage, Seth Hampson, Blaise Aguera Y Arcas","year":"2017","venue":"Artificial intelligence and statistics","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Miyato et al., 2018] Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: a regularization method for supervised and semi-supervised learning. IEEE transactions on pattern analysis and machine intelligence, 41(8):1979-1993, 2018.","title":"Virtual adversarial training: a regularization method for supervised and semi-supervised learning","authors":"Miyato","year":"2018","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Reddi et al., 2021] Sashank J Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečnỳ, Sanjiv Kumar, and Hugh Brendan McMahan. Adaptive federated optimization. In International Confer- ence on Learning Representations, 2021.","title":"Adaptive federated optimization","authors":"J Sashank, Zachary Reddi, Manzil Charles, Zachary Zaheer, Keith Garrett, Jakub Rush, Sanjiv Konečnỳ, Hugh Brendan Kumar, Mcmahan","year":"2021","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Sohn et al., 2020] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with con- sistency and confidence. Advances in neural information processing systems, 33:596-608, 2020.","title":"Fixmatch: Simplifying semi-supervised learning with consistency and confidence","authors":"Sohn","year":"2020","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Tan et al., 2022] Yue Tan, Guodong Long, Lu Liu, Tianyi Zhou, Qinghua Lu, Jing Jiang, and Chengqi Zhang. Fed- proto: Federated prototype learning across heterogeneous clients. In Proceedings of the AAAI Conference on Artifi- cial Intelligence, volume 36, pages 8432-8440, 2022.","title":"Fedproto: Federated prototype learning across heterogeneous clients","authors":"Yue Tan, Guodong Long, Lu Liu, Tianyi Zhou, Qinghua Lu, Jing Jiang, Chengqi Zhang","year":"2022","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Tian et al., 2020] Junjiao Tian, Yen-Cheng Liu, Nathaniel Glaser, Yen-Chang Hsu, and Zsolt Kira. Posterior re- calibration for imbalanced datasets. Advances in Neural Information Processing Systems, 33:8101-8113, 2020.","title":"Posterior recalibration for imbalanced datasets","authors":"Tian","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wang et al., 2020] Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor. Tackling the objective in- consistency problem in heterogeneous federated optimiza- tion. Advances in neural information processing systems, 33:7611-7623, 2020.","title":"Tackling the objective inconsistency problem in heterogeneous federated optimization","authors":"Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, H Vincent Poor","year":"2020","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wang et al., 2023] Yidong Wang, Hao Chen, Qiang Heng, Wenxin Hou, Yue Fan, , Zhen Wu, Jindong Wang, Marios Savvides, Takahiro Shinozaki, Bhiksha Raj, Bernt Schiele, and Xing Xie. Freematch: Self-adaptive thresholding for semi-supervised learning. 2023.","title":"Freematch: Self-adaptive thresholding for semi-supervised learning","authors":"Yidong Wang, Hao Chen, Qiang Heng, Wenxin Hou, Zhen Yue Fan, Jindong Wu, Marios Wang, Takahiro Savvides, Bhiksha Shinozaki, Bernt Raj, Xing Schiele, Xie","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Wei et al., 2021] Chen Wei, Kihyuk Sohn, Clayton Mellina, Alan Yuille, and Fan Yang. Crest: A class-rebalancing self-training framework for imbalanced semi-supervised learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10857- 10866, 2021.","title":"Crest: A class-rebalancing self-training framework for imbalanced semi-supervised learning","authors":"Chen Wei, Kihyuk Sohn, Clayton Mellina, Alan Yuille, Fan Yang","year":"2021","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhang et al., 2021] Zhengming Zhang, Yaoqing Yang, Zhewei Yao, Yujun Yan, Joseph E Gonzalez, Kannan Ramchandran, and Michael W Mahoney. Improving semi- supervised federated learning by reducing the gradient di- versity of models. In 2021 IEEE International Conference on Big Data (Big Data), pages 1214-1225. IEEE, 2021.","title":"Improving semisupervised federated learning by reducing the gradient diversity of models","authors":"Zhengming Zhang, Yaoqing Yang, Zhewei Yao, Yujun Yan, Joseph E Gonzalez, Kannan Ramchandran, Michael W Mahoney","year":"2021","venue":"IEEE International Conference on Big Data (Big Data)","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhao et al., 2018] Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Fed- erated learning with non-iid data. arXiv preprint arXiv:1806.00582, 2018.","title":"Federated learning with non-iid data","authors":"Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, Vikas Chandra","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zhu et al., 2023] Guogang Zhu, Xuefeng Liu, Shaojie Tang, and Jianwei Niu. Aligning before aggregating: Enabling communication efficient cross-domain federated learning via consistent feature extraction. IEEE Transactions on Mobile Computing, 2023.","title":"Aligning before aggregating: Enabling communication efficient cross-domain federated learning via consistent feature extraction","authors":"Zhu","year":"2023","venue":"IEEE Transactions on Mobile Computing","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}