{"type":"ScholarlyArticle","title":"Dual Calibration-based Personalised Federated Learning","authors":[{"name":"Xiaoli Tang","slug":"xiaoli-tang","given":"Xiaoli","family":"Tang","affiliation":"College of Computing and Data Science , Nanyang Technological University , Singapore"},{"name":"Han Yu","slug":"han-yu","given":"Han","family":"Yu","affiliation":"College of Computing and Data Science , Nanyang Technological University , Singapore"},{"name":"Run Tang","slug":"run-tang","given":"Run","family":"Tang","affiliation":"South China University of Technology , China"},{"name":"Chao Ren","slug":"chao-ren","given":"Chao","family":"Ren","affiliation":"College of Computing and Data Science , Nanyang Technological University , Singapore"},{"name":"Anran Li","slug":"anran-li","given":"Anran","family":"Li","affiliation":"College of Computing and Data Science , Nanyang Technological University , Singapore"},{"name":"Xiaoxiao Li","slug":"xiaoxiao-li","given":"Xiaoxiao","family":"Li","affiliation":"Department of Electrical and Computer Engineering , The University of British Columbia , Canada"}],"abstract":"Personalized federated learning (PFL) is designed for scenarios with non-independent and identically distributed (non-IID) client data. Existing model mixup-based methods, one of the main approaches of PFL, can only extract either global or personalized features during training, thereby limiting effective knowledge sharing among clients. To address this limitation, we propose the Dual Calibration-based PFL (DC-PFL). It divides local models into a heterogeneous feature extractor and a homogeneous classifier. The FL server utilizes mean and covariance representations from clients' feature extractors to train a global generalized classifier, facilitating information exchange while preserving privacy. To enhance personalization and convergence, we design a feature extractor-level calibration method with an auxiliary loss for local models to refine feature extractors using global knowledge. Furthermore, DC-PFL refines the global classifier through the global classifier-level calibration, utilizing sample representations derived from an approximate Gaussian distribution model specific to each class. This method precludes the need to transmit original data representations, further enhancing privacy preservation. Extensive experiments on widely used benchmark datasets demonstrate that DC-PFL outperforms eight state-of-the-art methods, surpassing the best-performing baseline by 1.22% and 9.22% in terms of accuracy on datasets CIFAR-10 and CIFAR-100, respectively.","doi":"10.24963/ijcai.2024/551","url":"https://ijcai.org/proceedings/2024/551","pdf":"https://ijcai.org/proceedings/2024/0551.pdf","markdown":"https://ijcai.org/proceedings/2024/551.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/551","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Machine Learning","track":"Main Track","number":551,"pages":{"first":4982,"last":4990},"keywords":["Machine Learning: ML: Federated learning"],"videos":[],"references":[{"position":1,"raw":"[Chang et al., 2019] Hongyan Chang, Virat Shejwalkar, Reza Shokri, and Amir Houmansadr. Cronus: Robust and heterogeneous collaborative learning with black-box knowledge transfer. arXiv preprint arXiv:1912.11279, 2019.","title":"Cronus: Robust and heterogeneous collaborative learning with black-box knowledge transfer","authors":"Chang","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2021] Jiangui Chen, Ruqing Zhang, Jiafeng Guo, Yixing Fan, and Xueqi Cheng. Fedmatch: feder- ated learning over heterogeneous question answering data. In Proceedings of the 30th ACM International Conference on Information and Knowledge Management (CIKM'21), pages 181-190, 2021.","title":"Fedmatch: federated learning over heterogeneous question answering data","authors":"Chen","year":"2021","venue":"Proceedings of the 30th ACM International Conference on Information and Knowledge Management (CIKM'21)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cheng et al., 2021] Sijie Cheng, Jingwen Wu, Yanghua Xiao, and Yang Liu. Fedgems: Federated learning of larger server models via selective knowledge fusion. arXiv preprint arXiv:2110.11027, 2021.","title":"Fedgems: Federated learning of larger server models via selective knowledge fusion","authors":"Sijie Cheng, Jingwen Wu, Yanghua Xiao, Yang Liu","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cho et al., 2022] Yae Jee Cho, Andre Manoel, Gauri Joshi, Robert Sim, and Dimitrios Dimitriadis. Heterogeneous ensemble knowledge transfer for training large models in federated learning. In Proceedings of the 31st Inter- national Joint Conference on Artificial Intelligence (IJ- CAI'22), pages 1881-1887, 2022.","title":"Heterogeneous ensemble knowledge transfer for training large models in federated learning","authors":"Yae Jee Cho, Andre Manoel, Gauri Joshi, Robert Sim, Dimitrios Dimitriadis","year":"2022","venue":"Proceedings of the 31st International Joint Conference on Artificial Intelligence (IJ-CAI'22)","doi":null,"url":null,"internal_url":"/proceedings/2022/399"},{"position":5,"raw":"[Collins et al., 2021] Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. Exploiting shared rep- resentations for personalized federated learning. In Pro- ceedings of the 38th International Conference on Machine Learning (ICML'21), pages 2089-2099, 2021.","title":"Exploiting shared representations for personalized federated learning","authors":"Liam Collins, Hamed Hassani, Aryan Mokhtari, Sanjay Shakkottai","year":"2021","venue":"Proceedings of the 38th International Conference on Machine Learning (ICML'21)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Huang et al., 2022a] Wenke Huang, Mang Ye, and Bo Du. Learn from others and be yourself in heterogeneous fed- erated learning. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR'22), pages 10143-10153, 2022.","title":"Learn from others and be yourself in heterogeneous federated learning","authors":"Huang","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR'22)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Huang et al., 2022b] Wenke Huang, Mang Ye, Bo Du, and Xiang Gao. Few-shot model agnostic federated learning. In Proceedings of the 30th ACM International Conference on Multimedia (ACM MM'22), pages 7309-7316, 2022.","title":"Few-shot model agnostic federated learning","authors":"Huang","year":"2022","venue":"Proceedings of the 30th ACM International Conference on Multimedia (ACM MM'22)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Itahara et al., 2021] Sohei Itahara, Takayuki Nishio, Yusuke Koda, Masahiro Morikura, and Koji Yamamoto. Distillation-based semi-supervised federated learning for communication-efficient collaborative training with non-iid private data. IEEE Transactions on Mobile Computing, 22(1):191-205, 2021.","title":"Distillation-based semi-supervised federated learning for communication-efficient collaborative training with non-iid private data","authors":"Itahara","year":"2021","venue":"IEEE Transactions on Mobile Computing","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jang et al., 2022] Jaehee Jang, Heoneok Ha, Dahuin Jung, and Sungroh Yoon. Fedclassavg: Local representation learning for personalized federated learning on heteroge- neous neural networks. In Proceedings of the 51st In- ternational Conference on Parallel Processing (ICPP'22), pages 1-10, 2022.","title":"Fedclassavg: Local representation learning for personalized federated learning on heterogeneous neural networks","authors":"Jaehee Jang, Heoneok Ha, Dahuin Jung, Sungroh Yoon","year":"2022","venue":"Proceedings of the 51st In-ternational Conference on Parallel Processing (ICPP'22)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jeong et al., 2018] Eunjeong Jeong, Seungeun Oh, Hye- sung Kim, Jihong Park, Mehdi Bennis, and Seong-Lyun Kim. Communication-efficient on-device machine learn- ing: Federated distillation and augmentation under non-iid private data. arXiv preprint arXiv:1811.11479, 2018.","title":"Communication-efficient on-device machine learning: Federated distillation and augmentation under non-iid private data","authors":"Jeong","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li and Wang, 2019] Daliang Li and Junpu Wang. Fedmd: Heterogenous federated learning via model distillation. arXiv preprint arXiv:1910.03581, 2019.","title":"Fedmd: Heterogenous federated learning via model distillation","authors":"Wang ; Li, Daliang Li, Junpu Wang","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2021] Qinbin Li, Bingsheng He, and Dawn Song. Practical one-shot federated learning for cross-silo setting. In Proceedings of the 30th International Joint Conference on Artificial Intelligence (IJCAI'21), pages 1484-1490, 2021.","title":"He, and Dawn Song. Practical one-shot federated learning for cross-silo setting","authors":"Li","year":"2021","venue":"Proceedings of the 30th International Joint Conference on Artificial Intelligence (IJCAI'21)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Liang et al., 2020] Paul Pu Liang, Terrance Liu, Liu Ziyin, Nicholas B Allen, Randy P Auerbach, David Brent, Rus- lan Salakhutdinov, and Louis-Philippe Morency. Think locally, act globally: Federated learning with local and global representations. arXiv preprint arXiv:2001.01523, 2020.","title":"Think locally, act globally: Federated learning with local and global representations","authors":"Paul Pu Liang, Terrance Liu, Liu Ziyin, Randy P Nicholas B Allen, David Auerbach, Ruslan Brent, Louis-Philippe Salakhutdinov, Morency","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lin et al., 2020] Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. In Proceedings of the 34th Conference on Neural Information Processing Systems (NeurIPS'20), pages 2351-2363, 2020.","title":"Ensemble distillation for robust model fusion in federated learning","authors":"Tao Lin, Lingjing Kong, Sebastian U Stich, Martin Jaggi","year":"2020","venue":"Proceedings of the 34th Conference on Neural Information Processing Systems (NeurIPS'20)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu et al., 2022] Chang Liu, Yuwen Yang, Xun Cai, Yue Ding, and Hongtao Lu. Completely heterogeneous fed- erated learning. arXiv preprint arXiv:2210.15865, 2022.","title":"Completely heterogeneous federated learning","authors":"Chang Liu, Yuwen Yang, Xun Cai, Yue Ding, Hongtao Lu","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liu et al., 2024] Rui Liu, Pengwei Xing, Zichao Deng, An- ran Li, Cuntai Guan, and Han Yu. Federated graph neural networks: Overview, techniques, and challenges. IEEE Transactions on Neural Networks and Learning Systems, 2024.","title":"Federated graph neural networks: Overview, techniques, and challenges","authors":"Rui Liu, Pengwei Xing, Zichao Deng, Anran Li, Cuntai Guan, Han Yu","year":"2024","venue":"IEEE Transactions on Neural Networks and Learning Systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Luo et al., 2021] Mi Luo, Fei Chen, Dapeng Hu, Yifan Zhang, Jian Liang, and Jiashi Feng. No fear of hetero- geneity: Classifier calibration for federated learning with non-iid data. Advances in Neural Information Processing Systems, 34:5972-5984, 2021.","title":"No fear of heterogeneity: Classifier calibration for federated learning with non-iid data","authors":"Mi Luo, Fei Chen, Dapeng Hu, Yifan Zhang, Jian Liang, Jiashi Feng","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Makhija et al., 2022] Disha Makhija, Xing Han, Nhat Ho, and Joydeep Ghosh. Architecture agnostic federated learn- ing for neural networks. In Proceedings of the 39th In- ternational Conference on Machine Learning (ICML'22), pages 14860-14870, 2022.","title":"Architecture agnostic federated learning for neural networks","authors":"Disha Makhija, Xing Han, Nhat Ho, Joydeep Ghosh","year":"2022","venue":"Proceedings of the 39th International Conference on Machine Learning (ICML'22)","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[McMahan et al., 2017] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Ar- cas. Communication-efficient learning of deep networks from decentralized data. In Proceedings of the 20th Inter- national Conference on Artificial Intelligence and Statis- tics (AISTATS'17), pages 1273-1282, 2017.","title":"Communication-efficient learning of deep networks from decentralized data","authors":"Brendan Mcmahan, Eider Moore, Daniel Ramage, Seth Hampson, Blaise Aguera Y Arcas","year":"2017","venue":"Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (AISTATS'17)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Oh et al., 2022] Jaehoon Oh, Sangmook Kim, and Se- Young Yun. Fedbabu: Towards enhanced representation for federated image classification. In Proceedings of the 10th International Conference on Learning Representa- tions (ICLR'22), 2022.","title":"Fedbabu: Towards enhanced representation for federated image classification","authors":"Jaehoon Oh, Sangmook Kim, Se-Young Yun","year":"2022","venue":"Proceedings of the 10th International Conference on Learning Representations (ICLR'22)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Pillutla et al., 2022] Krishna Pillutla, Kshitiz Malik, Abdel- Rahman Mohamed, Mike Rabbat, Maziar Sanjabi, and Lin Xiao. Federated learning with partial model person- alization. In Proceedings of the 39th International Con- ference on Machine Learning (ICML'22), pages 17716- 17758, 2022.","title":"Federated learning with partial model personalization","authors":"Krishna Pillutla, Kshitiz Malik, Abdel-Rahman Mohamed, Mike Rabbat, Maziar Sanjabi, Lin Xiao","year":"2022","venue":"Proceedings of the 39th International Conference on Machine Learning (ICML'22)","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Robbins and Monro, 1951] Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathematical statistics, pages 400-407, 1951.","title":"The annals of mathematical statistics","authors":"Monro Robbins, Herbert Robbins, Sutton Monro","year":"1951","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Sattler et al., 2021a] Felix Sattler, Tim Korjakow, Roman Rischke, and Wojciech Samek. Fedaux: Leveraging un- labeled auxiliary data in federated learning. IEEE Trans- actions on Neural Networks and Learning Systems, 2021.","title":"Fedaux: Leveraging unlabeled auxiliary data in federated learning","authors":"Sattler","year":"2021","venue":"IEEE Transactions on Neural Networks and Learning Systems","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Sattler et al., 2021b] Felix Sattler, Arturo Marban, Roman Rischke, and Wojciech Samek. Cfd: Communication- efficient federated distillation via soft-label quantization and delta coding. IEEE Transactions on Network Science and Engineering, 9(4):2025-2038, 2021.","title":"Cfd: Communicationefficient federated distillation via soft-label quantization and delta coding","authors":"Sattler","year":"2021","venue":"IEEE Transactions on Network Science and Engineering","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Shen et al., 2020] Tao Shen, Jie Zhang, Xinkang Jia, Fengda Zhang, Gang Huang, Pan Zhou, Kun Kuang, Fei Wu, and Chao Wu. Federated mutual learning. arXiv preprint arXiv:2006.16765, 2020.","title":"Federated mutual learning","authors":"Tao Shen, Jie Zhang, Xinkang Jia, Fengda Zhang, Gang Huang, Pan Zhou, Kun Kuang, Fei Wu, Chao Wu","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Tan et al., 2022a] Alysa Ziying Tan, Han Yu, Lizhen Cui, and Qiang Yang. Towards personalized federated learn- ing. IEEE Transactions on Neural Networks and Learning Systems, 2022.","title":"Towards personalized federated learning","authors":"Tan","year":"2022","venue":"IEEE Transactions on Neural Networks and Learning Systems","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Tan et al., 2022b] Yue Tan, Guodong Long, Lu Liu, Tianyi Zhou, Qinghua Lu, Jing Jiang, and Chengqi Zhang. Fed- proto: Federated prototype learning across heterogeneous clients. In Proceedings of the AAAI Conference on Artifi- cial Intelligence, volume 36, pages 8432-8440, 2022.","title":"Fedproto: Federated prototype learning across heterogeneous clients","authors":"Tan","year":"2022","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Tan et al., 2024] Yue Tan, Chen Chen, Weiming Zhuang, Xin Dong, Lingjuan Lyu, and Guodong Long. Is hetero- geneity notorious? taming heterogeneity to handle test- time shift in federated learning. Advances in Neural Infor- mation Processing Systems, 36, 2024.","title":"Is heterogeneity notorious? taming heterogeneity to handle testtime shift in federated learning","authors":"Yue Tan, Chen Chen, Weiming Zhuang, Xin Dong, Lingjuan Lyu, Guodong Long","year":"2024","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wu et al., 2022] Chuhan Wu, Fangzhao Wu, Lingjuan Lyu, Yongfeng Huang, and Xing Xie. Communication-efficient federated learning via knowledge distillation. Nature com- munications, 13(1):2032, 2022.","title":"Communication-efficient federated learning via knowledge distillation","authors":"Wu","year":"2022","venue":"Nature communications","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Yi et al., 2023] Liping Yi, Gang Wang, Xiaoguang Liu, Zhuan Shi, and Han Yu. FedGH: Heterogeneous federated learning with generalized global header. In Proceedings of the 31st ACM International Conference on Multimedia (ACM MM'23), 2023.","title":"FedGH: Heterogeneous federated learning with generalized global header","authors":"Liping Yi, Gang Wang, Xiaoguang Liu, Zhuan Shi, Han Yu","year":"2023","venue":"Proceedings of the 31st ACM International Conference on Multimedia (ACM MM'23)","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Yu et al., 2022] Sixing Yu, Wei Qian, and Ali Jan- nesari. Resource-aware federated learning using knowl- edge extraction and multi-model fusion. arXiv preprint arXiv:2208.07978, 2022.","title":"Resource-aware federated learning using knowledge extraction and multi-model fusion","authors":"Yu","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhang et al., 2022] Lan Zhang, Dapeng Wu, and Xiaoy- ong Yuan. Fedzkt: Zero-shot knowledge transfer to- wards resource-constrained federated learning with het- erogeneous on-device models. In Proceedings of the IEEE 42nd International Conference on Distributed Computing Systems (ICDCS'22), pages 928-938. IEEE, 2022.","title":"Fedzkt: Zero-shot knowledge transfer towards resource-constrained federated learning with heterogeneous on-device models","authors":"Lan Zhang, Dapeng Wu, Xiaoyong Yuan","year":"2022","venue":"Proceedings of the IEEE 42nd International Conference on Distributed Computing Systems (ICDCS'22)","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhu et al., 2021] Zhuangdi Zhu, Junyuan Hong, and Jiayu Zhou. Data-free knowledge distillation for heterogeneous federated learning. In Proceedings of the 38th Interna- tional Conference on Machine Learning (ICML'21), pages 12878-12889, 2021.","title":"Data-free knowledge distillation for heterogeneous federated learning","authors":"Zhu","year":"2021","venue":"Proceedings of the 38th International Conference on Machine Learning (ICML'21)","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zhuang et al., 2023] Weiming Zhuang, Chen Chen, and Lingjuan Lyu. When foundation model meets federated learning: Motivations, challenges, and future directions. arXiv preprint arXiv:2306.15546, 2023.","title":"When foundation model meets federated learning: Motivations, challenges, and future directions","authors":"Weiming Zhuang, Chen Chen, Lingjuan Lyu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}