{"type":"ScholarlyArticle","title":"From Optimization to Generalization: Fair Federated Learning against Quality Shift via Inter-Client Sharpness Matching","authors":[{"name":"Nannan Wu","slug":"nannan-wu","given":"Nannan","family":"Wu","affiliation":"School of Electronic Information and Communications , Huazhong University of Science and Technology"},{"name":"Zhuo Kuang","slug":"zhuo-kuang","given":"Zhuo","family":"Kuang","affiliation":"School of Electronic Information and Communications , Huazhong University of Science and Technology"},{"name":"Zengqiang Yan","slug":"zengqiang-yan","given":"Zengqiang","family":"Yan","affiliation":"School of Electronic Information and Communications , Huazhong University of Science and Technology"},{"name":"Li Yu","slug":"li-yu","given":"Li","family":"Yu","affiliation":"School of Electronic Information and Communications , Huazhong University of Science and Technology"}],"abstract":"Due to escalating privacy concerns, federated learning has been recognized as a vital approach for training deep neural networks with decentralized medical data. In practice, it is challenging to ensure consistent imaging quality across various institutions, often attributed to equipment malfunctions affecting a minority of clients. This imbalance in image quality can cause the federated model to develop an inherent bias towards higher-quality images, thus posing a severe fairness issue. In this study, we pioneer the identification and formulation of this new fairness challenge within the context of the imaging quality shift. Traditional methods for promoting fairness in federated learning predominantly focus on balancing empirical risks across diverse client distributions. This strategy primarily facilitates fair optimization across different training data distributions, yet neglects the crucial aspect of generalization. To address this, we introduce a solution termed Federated learning with Inter-client Sharpness Matching (FedISM). FedISM enhances both local training and global aggregation by incorporating sharpness-awareness, aiming to harmonize the sharpness levels across clients for fair generalization. Our empirical evaluations, conducted using the widely-used ICH and ISIC 2019 datasets, establish FedISM's superiority over current state-of-the-art federated learning methods in promoting fairness. Code is available at https://github.com/wnn2000/FFL4MIA.","doi":"10.24963/ijcai.2024/575","url":"https://ijcai.org/proceedings/2024/575","pdf":"https://ijcai.org/proceedings/2024/0575.pdf","markdown":"https://ijcai.org/proceedings/2024/575.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/575","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Machine Learning","track":"Main Track","number":575,"pages":{"first":5199,"last":5207},"keywords":["Computer Vision: CV: Biomedical image analysis","Machine Learning: ML: Federated learning","AI Ethics, Trust, Fairness: ETF: Fairness and diversity","Computer Vision: CV: Bias, fairness and privacy"],"videos":[],"references":[{"position":1,"raw":"[Caldarola et al., 2022] Debora Caldarola, Barbara Caputo, and Marco Ciccone. Improving generalization in federated learning by seeking flat minima. In ECCV, 2022.","title":"Improving generalization in federated learning by seeking flat minima","authors":"Debora Caldarola, Barbara Caputo, Marco Ciccone","year":"2022","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2023] Zhen Chen, Wuyang Li, Xiaohan Xing, and Yixuan Yuan. Medical federated learning with joint graph purification for noisy label learning. Medical Image Anal., 90:102976, 2023.","title":"Medical federated learning with joint graph purification for noisy label learning","authors":"Zhen Chen, Wuyang Li, Xiaohan Xing, Yixuan Yuan","year":"2023","venue":"Medical Image Anal","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Codella and others, 2018] Noel CF Codella et al. Skin le- sion analysis toward melanoma detection: A challenge at the 2017 international symposium on biomedical imaging (ISBI), hosted by the international skin imaging collabora- tion (ISIC). In ISBI, pages 168-172, 2018.","title":"Skin lesion analysis toward melanoma detection: A challenge at the 2017 international symposium on biomedical imaging (ISBI), hosted by the international skin imaging collaboration (ISIC)","authors":"C F Noel, Codella","year":"2018","venue":"ISBI","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Combalia et al., 2019] Marc Combalia, Noel CF Codella, Veronica Rotemberg, Brian Helba, Veronica Vilaplana, Ofer Reiter, Cristina Carrera, Alicia Barreiro, Allan C Halpern, Susana Puig, et al. BCN20000: Dermoscopic lesions in the wild. arXiv:1908.02288, 2019.","title":"BCN20000: Dermoscopic lesions in the wild","authors":"Marc Combalia, C F Noel, Veronica Codella, Brian Rotemberg, Veronica Helba, Ofer Vilaplana, Cristina Reiter, Alicia Carrera, Allan C Barreiro, Susana Halpern, Puig","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dou et al., 2021] Qi Dou, Tiffany Y So, Meirui Jiang, Quande Liu, Varut Vardhanabhuti, Georgios Kaissis, Zeju Li, Weixin Si, Heather HC Lee, Kevin Yu, et al. Federated deep learning for detecting COVID-19 lung abnormalities in CT: a privacy-preserving multinational validation study. npj Digit. Medicine, 4(1):60, 2021.","title":"Federated deep learning for detecting COVID-19 lung abnormalities in CT: a privacy-preserving multinational validation study","authors":"Qi Dou, Tiffany Y So, Meirui Jiang, Quande Liu, Varut Vardhanabhuti, Georgios Kaissis, Zeju Li, Weixin Si, H C Heather, Kevin Lee, Yu","year":"2021","venue":"npj Digit. Medicine","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Ezzeldin et al., 2023] Yahya H Ezzeldin, Shen Yan, Chaoyang He, Emilio Ferrara, and A Salman Avestimehr. Fairfed: Enabling group fairness in federated learning. In AAAI, 2023.","title":"Fairfed: Enabling group fairness in federated learning","authors":"Shen Yahya H Ezzeldin, Chaoyang Yan, Emilio He, Salman Ferrara, Avestimehr","year":"2023","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Fang et al., 2023] Xiuwen Fang, Mang Ye, and Xiyuan Yang. Robust heterogeneous federated learning under data corruption. In ICCV, 2023.","title":"Robust heterogeneous federated learning under data corruption","authors":"Xiuwen Fang, Mang Ye, Xiyuan Yang","year":"2023","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Flanders et al., 2020] Adam E Flanders, Luciano M Prevedello, George Shih, Safwan S Halabi, Jayashree Kalpathy-Cramer, Robyn Ball, John T Mongan, Anouk Stein, Felipe C Kitamura, Matthew P Lungren, et al. Construction of a machine learning dataset through collaboration: The RSNA 2019 brain CT hemorrhage challenge. Radiology: Artificial Intelligence, 2(3), 2020.","title":"Construction of a machine learning dataset through collaboration: The RSNA 2019 brain CT hemorrhage challenge","authors":"Luciano M Adam E Flanders, George Prevedello, Shih, S Safwan, Jayashree Halabi, Robyn Kalpathy-Cramer, Ball, Anouk John T Mongan, Felipe C Stein, Kitamura, Matthew P Lungren","year":"2020","venue":"Radiology: Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Foret et al., 2021] Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware mini- mization for efficiently improving generalization. In ICLR, 2021.","title":"Sharpness-aware minimization for efficiently improving generalization","authors":"Foret","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hendrycks and Dietterich, 2019] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robust- ness to common corruptions and perturbations. In ICLR, 2019.","title":"Benchmarking neural network robust-ness to common corruptions and perturbations","authors":"Dietterich Hendrycks, Dan Hendrycks, Thomas Dietterich","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Huang et al., 2022] Wenke Huang, Mang Ye, and Bo Du. Learn from others and be yourself in heterogeneous feder- ated learning. In CVPR, 2022.","title":"Learn from others and be yourself in heterogeneous federated learning","authors":"Wenke Huang, Mang Ye, Bo Du","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Huang et al., 2023a] Wenke Huang, Mang Ye, Zekun Shi, and Bo Du. Generalizable heterogeneous federated cross- correlation and instance similarity learning. IEEE Trans. Pattern Anal. Mach. Intell., 2023.","title":"Generalizable heterogeneous federated crosscorrelation and instance similarity learning","authors":"Huang","year":"2023","venue":"IEEE Trans. Pattern Anal. Mach. Intell","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Huang et al., 2023b] Wenke Huang, Mang Ye, Zekun Shi, He Li, and Bo Du. Rethinking federated learning with domain shift: A prototype view. In CVPR, 2023.","title":"Rethinking federated learning with domain shift: A prototype view","authors":"Huang","year":"2023","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Huang et al., 2023c] Wenke Huang, Mang Ye, Zekun Shi, Guancheng Wan, He Li, Bo Du, and Qiang Yang. Fed- erated learning for generalization, robustness, fairness: A survey and benchmark. arXiv:2311.06750, 2023.","title":"Federated learning for generalization, robustness, fairness: A survey and benchmark","authors":"Huang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Huang et al., 2023d] Zhuo Huang, Miaoxi Zhu, Xiaobo Xia, Li Shen, Jun Yu, Chen Gong, Bo Han, Bo Du, and Tongliang Liu. Robust generalization against photon- limited corruptions via worst-case sharpness minimiza- tion. In CVPR, 2023.","title":"Robust generalization against photonlimited corruptions via worst-case sharpness minimization","authors":"Huang","year":"2023","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Jiang et al., 2022] Meirui Jiang, Hongzheng Yang, Xiaox- iao Li, Quande Liu, Pheng-Ann Heng, and Qi Dou. Dy- namic bank learning for semi-supervised federated image diagnosis with class imbalance. In MICCAI, pages 196- 206, 2022.","title":"Dynamic bank learning for semi-supervised federated image diagnosis with class imbalance","authors":"Jiang","year":"2022","venue":"MICCAI","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Jiang et al., 2023a] Meirui Jiang, Holger R Roth, Wenqi Li, Dong Yang, Can Zhao, Vishwesh Nath, Daguang Xu, Qi Dou, and Ziyue Xu. Fair federated medical image seg- mentation via client contribution estimation. In CVPR, pages 16302-16311, 2023.","title":"Fair federated medical image segmentation via client contribution estimation","authors":"Jiang","year":"2023","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Jiang et al., 2023b] Meirui Jiang, Hongzheng Yang, Chen Cheng, and Qi Dou. IOP-FL: Inside-outside personal- ization for federated medical image segmentation. IEEE Trans. Medical Imaging, 2023.","title":"IOP-FL: Inside-outside personalization for federated medical image segmentation","authors":"Jiang","year":"2023","venue":"IEEE Trans. Medical Imaging","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Li et al., 2020] Tian Li, Maziar Sanjabi, Ahmad Beirami, and Virginia Smith. Fair resource allocation in federated learning. In ICLR, 2020.","title":"Fair resource allocation in federated learning","authors":"Tian Li, Maziar Sanjabi, Ahmad Beirami, Virginia Smith","year":"2020","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Li et al., 2021] Xiaoxiao Li, Meirui Jiang, Xiaofei Zhang, Michael Kamp, and Qi Dou. FedBN: Federated learning on non-iid features via local batch normalization. In ICLR, 2021.","title":"FedBN: Federated learning on non-iid features via local batch normalization","authors":"Xiaoxiao Li, Meirui Jiang, Xiaofei Zhang, Michael Kamp, Qi Dou","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Liu et al., 2021] Quande Liu, Cheng Chen, Jing Qin, Qi Dou, and Pheng-Ann Heng. FedDG: Federated do- main generalization on medical image segmentation via episodic learning in continuous frequency space. In CVPR, pages 1013-1023, 2021.","title":"FedDG: Federated domain generalization on medical image segmentation via episodic learning in continuous frequency space","authors":"Quande Liu, Cheng Chen, Jing Qin, Qi Dou, Pheng-Ann Heng","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Lyu et al., 2020] Lingjuan Lyu, Xinyi Xu, Qian Wang, and Han Yu. Collaborative fairness in federated learning. Fed- erated Learning: Privacy and Incentive, pages 189-204, 2020.","title":"Collaborative fairness in federated learning","authors":"Lyu","year":"2020","venue":"Federated Learning: Privacy and Incentive","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[McMahan et al., 2017] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Ar- cas. Communication-efficient learning of deep networks from decentralized data. In AISTATS, pages 1273-1282, 2017.","title":"Communication-efficient learning of deep networks from decentralized data","authors":"Brendan Mcmahan, Eider Moore, Daniel Ramage, Seth Hampson, Blaise Aguera Y Arcas","year":"2017","venue":"AISTATS","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Menon et al., 2021] Aditya Krishna Menon, Sadeep Jaya- sumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit ad- justment. In ICLR, 2021.","title":"Long-tail learning via logit adjustment","authors":"Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, Sanjiv Kumar","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Mohri et al., 2019] Mehryar Mohri, Gary Sivek, and Ananda Theertha Suresh. Agnostic federated learning. In ICML, pages 4615-4625, 2019.","title":"Agnostic federated learning","authors":"Mehryar Mohri, Gary Sivek, Ananda Theertha, Suresh","year":"2019","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Papadaki et al., 2022] Afroditi Papadaki, Natalia Martinez, Martin Bertran, Guillermo Sapiro, and Miguel Rodrigues. Minimax demographic group fairness in federated learn- ing. In ACM FAccT, pages 142-159, 2022.","title":"Minimax demographic group fairness in federated learning","authors":"Afroditi Papadaki, Natalia Martinez, Martin Bertran, Guillermo Sapiro, Miguel Rodrigues","year":"2022","venue":"ACM FAccT","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Qu et al., 2022] Zhe Qu, Xingyu Li, Rui Duan, Yao Liu, Bo Tang, and Zhuo Lu. Generalized federated learning via sharpness aware minimization. In ICML, 2022.","title":"Generalized federated learning via sharpness aware minimization","authors":"Zhe Qu, Xingyu Li, Rui Duan, Yao Liu, Bo Tang, Zhuo Lu","year":"2022","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Sagawa et al., 2020] Shiori Sagawa, Pang Wei Koh, Tat- sunori B Hashimoto, and Percy Liang. Distributionally ro- bust neural networks for group shifts: On the importance of regularization for worst-case generalization. In ICLR, 2020.","title":"Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization","authors":"Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, Percy Liang","year":"2020","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Sun et al., 2023] Yan Sun, Li Shen, Shixiang Chen, Liang Ding, and Dacheng Tao. Dynamic regularized sharpness aware minimization in federated learning: Approaching global consistency and smooth landscape. In ICML, 2023.","title":"Dynamic regularized sharpness aware minimization in federated learning: Approaching global consistency and smooth landscape","authors":"Yan Sun, Li Shen, Shixiang Chen, Liang Ding, Dacheng Tao","year":"2023","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Tschandl et al., 2018] Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. The HAM10000 dataset, a large col- lection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific Data, 5(1):1-9, 2018.","title":"The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions","authors":"Philipp Tschandl, Cliff Rosendahl, Harald Kittler","year":"2018","venue":"Scientific Data","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Wu et al., 2023a] Nannan Wu, Li Yu, Xuefeng Jiang, Kwang-Ting Cheng, and Zengqiang Yan. Fednoro: To- wards noise-robust federated learning by addressing class imbalance and label noise heterogeneity. In IJCAI, 2023.","title":"Fednoro: Towards noise-robust federated learning by addressing class imbalance and label noise heterogeneity","authors":"Wu","year":"2023","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2023/492"},{"position":33,"raw":"[Wu et al., 2023b] Nannan Wu, Li Yu, Xin Yang, Kwang- Ting Cheng, and Zengqiang Yan. Fediic: Towards ro- bust federated learning for class-imbalanced medical im- age classification. In MICCAI, 2023.","title":"Fediic: Towards robust federated learning for class-imbalanced medical image classification","authors":"Wu","year":"2023","venue":"MICCAI","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Wu et al., 2024] Nannan Wu, Zhaobin Sun, Zengqiang Yan, and Li Yu. Feda3i: Annotation quality-aware aggregation for federated medical image segmentation against hetero- geneous annotation noise. In AAAI, 2024.","title":"Feda3i: Annotation quality-aware aggregation for federated medical image segmentation against heterogeneous annotation noise","authors":"Nannan Wu, Zhaobin Sun, Zengqiang Yan, Li Yu","year":"2024","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Xu et al., 2021] Xinyi Xu, Lingjuan Lyu, Xingjun Ma, Chenglin Miao, Chuan Sheng Foo, and Bryan Kian Hsiang Low. Gradient driven rewards to guarantee fairness in col- laborative machine learning. 2021.","title":"Gradient driven rewards to guarantee fairness in collaborative machine learning","authors":"Xinyi Xu, Lingjuan Lyu, Xingjun Ma, Chenglin Miao, Chuan Sheng Foo, Bryan Kian, Hsiang Low","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Ye et al., 2023] Mang Ye, Xiuwen Fang, Bo Du, Pong C Yuen, and Dacheng Tao. Heterogeneous federated learn- ing: State-of-the-art and research challenges. ACM Com- puting Surveys, 56(3):1-44, 2023.","title":"Heterogeneous federated learning: State-of-the-art and research challenges","authors":"Mang Ye, Xiuwen Fang, Bo Du, Dacheng Pong C Yuen, Tao","year":"2023","venue":"ACM Computing Surveys","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Zhang et al., 2022] Jie Zhang, Zhiqi Li, Bo Li, Jianghe Xu, Shuang Wu, Shouhong Ding, and Chao Wu. Federated learning with label distribution skew via logits calibration. In ICML, 2022.","title":"Federated learning with label distribution skew via logits calibration","authors":"Jie Zhang, Zhiqi Li, Bo Li, Jianghe Xu, Shuang Wu, Shouhong Ding, Chao Wu","year":"2022","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Zhang et al., 2023] Ruipeng Zhang, Qinwei Xu, Jiangchao Yao, Ya Zhang, Qi Tian, and Yanfeng Wang. Federated domain generalization with generalization adjustment. In CVPR, 2023.","title":"Federated domain generalization with generalization adjustment","authors":"Zhang","year":"2023","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Zhou et al., 2023] Yixuan Zhou, Yi Qu, Xing Xu, and Hengtao Shen. Imbsam: A closer look at sharpness-aware minimization in class-imbalanced recognition. In ICCV, pages 11345-11355, 2023.","title":"Imbsam: A closer look at sharpness-aware minimization in class-imbalanced recognition","authors":"Yixuan Zhou, Yi Qu, Xing Xu, Hengtao Shen","year":"2023","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Zhuang et al., 2022] Juntang Zhuang, Boqing Gong, Liangzhe Yuan, Yin Cui, Hartwig Adam, Nicha Dvornek, Sekhar Tatikonda, James Duncan, and Ting Liu. Surrogate gap minimization improves sharpness-aware training. In ICLR, 2022.","title":"Surrogate gap minimization improves sharpness-aware training","authors":"Zhuang","year":"2022","venue":"ICLR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}