{"type":"ScholarlyArticle","title":"FBLG: A Local Graph Based Approach for Handling Dual Skewed Non-IID Data in Federated Learning","authors":[{"name":"Yi Xu","slug":"yi-xu","given":"Yi","family":"Xu","affiliation":"School of Computer Science and Technology , Anhui University , Hefei 230601 , China"},{"name":"Ying Li","slug":"ying-li","given":"Ying","family":"Li","affiliation":"School of Computer Science and Technology , Anhui University , Hefei 230601 , China"},{"name":"Haoyu Luo","slug":"haoyu-luo","given":"Haoyu","family":"Luo","affiliation":"College of Mathematics and Informatics , South China Agricultural University , Guangzhou , China"},{"name":"Xiaoliang Fan","slug":"xiaoliang-fan","given":"Xiaoliang","family":"Fan","affiliation":"Fujian Key Laboratory of Sensing and Computing for Smart Cities , School of Informatics , Xiamen University , Xiamen , China"},{"name":"Xiao Liu","slug":"xiao-liu","given":"Xiao","family":"Liu","affiliation":"School of Information Technology , Deakin University , Geelong , Australia"}],"abstract":"In real-world situations, federated learning often needs to process non-IID (non-independent and identically distributed) data with multiple skews, causing inadequate model performance. Existing federated learning methods mainly focus on addressing the problem with a single skew of non-IID, and hence the performance of global models can be degraded when faced with dual skewed non-IID data caused by heterogeneous label distributions and sample sizes among clients. To address the problem with dual skewed non-IID data, in this paper, we propose a federated learning algorithm based on local graph, named FBLG. Specifically, to address the label distribution skew, we firstly construct a local graph based on clients' local losses and Jensen-Shannon (JS) divergence, so that similar clients can be selected for aggregation to ensure a highly consistent global model. Afterwards, to address the sample size skew, we design the objective function to favor clients with more samples as models trained with more samples tend to carry more useful information. Experiments on four datasets with dual skewed non-IID data demonstrate FBLG outperforms nine baseline methods and achieves up to 9% improvement in accuracy. Simultaneously, both theoretical analysis and experiments show FBLG can converge quickly.","doi":"10.24963/ijcai.2024/585","url":"https://ijcai.org/proceedings/2024/585","pdf":"https://ijcai.org/proceedings/2024/0585.pdf","markdown":"https://ijcai.org/proceedings/2024/585.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/585","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Machine Learning","track":"Main Track","number":585,"pages":{"first":5289,"last":5297},"keywords":["Machine Learning: ML: Federated learning","Machine Learning: ML: Classification","Machine Learning: ML: Supervised Learning","Machine Learning: ML: Optimization"],"videos":[],"references":[{"position":1,"raw":"[Bubeck, 2015] Sébastien Bubeck. Convex optimization: Algorithms and complexity. Foundations and Trends in Machine Learning, pages 231-357, 2015.","title":"Convex optimization: Algorithms and complexity","authors":"Sébastien Bubeck","year":"2015","venue":"Foundations and Trends in Machine Learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cho et al., 2020] Yae Jee Cho, Jianyu Wang, and Gauri Joshi. Client selection in federated learning: Convergence analysis and power-of-choice selection strategies. arXiv preprint arXiv:2010.01243, 2020.","title":"Client selection in federated learning: Convergence analysis and power-of-choice selection strategies","authors":"Yae Jee Cho, Jianyu Wang, Gauri Joshi","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Diao et al., 2023] Yiqun Diao, Qinbin Li, and Bingsheng He. Towards addressing label skews in one-shot federated learning. In ICLR, 2023.","title":"Towards addressing label skews in one-shot federated learning","authors":"Yiqun Diao, Qinbin Li, Bingsheng He","year":"2023","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Hsieh et al., 2020] Kevin Hsieh, Amar Phanishayee, Onur Mutlu, and Phillip Gibbons. The non-iid data quagmire of decentralized machine learning. In ICML, pages 4387- 4398, 2020.","title":"The non-iid data quagmire of decentralized machine learning","authors":"Kevin Hsieh, Amar Phanishayee, Onur Mutlu, Phillip Gibbons","year":"2020","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Hsu et al., 2019] Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.06335, 2019.","title":"Measuring the effects of non-identical data distribution for federated visual classification","authors":"Hsu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Huang et al., 2022] Wenke Huang, Mang Ye, and Bo Du. Learn from others and be yourself in heterogeneous feder- ated learning. In CVPR, pages 10143-10153, 2022.","title":"Learn from others and be yourself in heterogeneous federated learning","authors":"Wenke Huang, Mang Ye, Bo Du","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Kairouz et al., 2021] Peter Kairouz, H Brendan McMa- han, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Ar- jun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. Foundations and Trends in Machine Learning, 14(1-2):1-210, 2021.","title":"Advances and open problems in federated learning","authors":"Peter Kairouz, Brendan Mcmahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Nitin Arjun, Kallista Bhagoji, Zachary Bonawitz, Graham Charles, Rachel Cormode, Cummings","year":"2021","venue":"Foundations and Trends in Machine Learning","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Karimireddy et al., 2020] Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In ICML, pages 5132-5143, 2020.","title":"Scaffold: Stochastic controlled averaging for federated learning","authors":"Praneeth Sai, Satyen Karimireddy, Mehryar Kale, Sashank Mohri, Sebastian Reddi, Ananda Stich, Suresh Theertha","year":"2020","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Krizhevsky et al., 2009] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny im- ages. Handbook of Systemic Autoimmune Diseases, 2009.","title":"Learning multiple layers of features from tiny images","authors":"Alex Krizhevsky, Geoffrey Hinton","year":"2009","venue":"Handbook of Systemic Autoimmune Diseases","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[LeCun et al., 1998] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning ap- plied to document recognition. Proceedings of the IEEE, 86(11):2278-2324, 1998.","title":"Gradient-based learning applied to document recognition","authors":"Yann Lecun, Léon Bottou, Yoshua Bengio, Patrick Haffner","year":"1998","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2020a] Li Li, Yuxi Fan, Mike Tse, and Kuo-Yi Lin. A review of applications in federated learning. Computers & Industrial Engineering, 149:106854, 2020.","title":"A review of applications in federated learning","authors":"Li","year":"2020","venue":"Computers & Industrial Engineering","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2020b] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. In ML- Sys, volume 2, pages 429-450, 2020.","title":"Federated optimization in heterogeneous networks","authors":"Li","year":"2020","venue":"ML-Sys","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2020c] Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the convergence of fedavg on non-iid data. In ICLR, 2020.","title":"On the convergence of fedavg on non-iid data","authors":"Li","year":"2020","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li et al., 2021a] Qinbin Li, Bingsheng He, and Dawn Song. Model-contrastive federated learning. In CVPR, pages 10713-10722, 2021.","title":"Bingsheng He, and Dawn Song. Model-contrastive federated learning","authors":"Li","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2021b] Xiaoxiao Li, Meirui Jiang, Xiaofei Zhang, Michael Kamp, and Qi Dou. Fedbn: Federated learning on non-iid features via local batch normalization. In ICLR, 2021.","title":"Fedbn: Federated learning on non-iid features via local batch normalization","authors":"Li","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Li et al., 2022] Qinbin Li, Yiqun Diao, Quan Chen, and Bingsheng He. Federated learning on non-iid data silos: An experimental study. In ICDE, pages 965-978, 2022.","title":"Federated learning on non-iid data silos: An experimental study","authors":"Li","year":"2022","venue":"ICDE","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liao et al., 2023] Xinting Liao, Weiming Liu, Chaochao Chen, Pengyang Zhou, Huabin Zhu, Yanchao Tan, Jun Wang, and Yue Qi. Hyperfed: hyperbolic prototypes ex- ploration with consistent aggregation for non-iid data in federated learning. In IJCAI, pages 3957-3965, 2023.","title":"Hyperfed: hyperbolic prototypes exploration with consistent aggregation for non-iid data in federated learning","authors":"Liao","year":"2023","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2023/440"},{"position":18,"raw":"[McMahan et al., 2017] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Ar- cas. Communication-efficient learning of deep networks from decentralized data. In AISTATS, pages 1273-1282, 2017.","title":"Communication-efficient learning of deep networks from decentralized data","authors":"Brendan Mcmahan, Eider Moore, Daniel Ramage, Seth Hampson, Blaise Aguera Y Arcas","year":"2017","venue":"AISTATS","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Netzer et al., 2011] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learn- ing. In NIPS, 2011.","title":"Reading digits in natural images with unsupervised feature learning","authors":"Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, Andrew Y Ng","year":"2011","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Ramakrishna and Dán, 2022] Raksha Ramakrishna and György Dán. Inferring class-label distribution in federated learning. In AISec, pages 45-56, 2022.","title":"Raksha Ramakrishna and György Dán. Inferring class-label distribution in federated learning","authors":"Dán Ramakrishna","year":"2022","venue":"AISec","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Shang et al., 2022] Xinyi Shang, Yang Lu, Gang Huang, and Hanzi Wang. Federated learning on heterogeneous and long-tailed data via classifier re-training with feder- ated features. In IJCAI, pages 2218-2224, 2022.","title":"Federated learning on heterogeneous and long-tailed data via classifier re-training with federated features","authors":"Xinyi Shang, Yang Lu, Gang Huang, Hanzi Wang","year":"2022","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2022/308"},{"position":22,"raw":"[Tian et al., 2022] Pu Tian, Weixian Liao, Wei Yu, and Erik Blasch. Wscc: A weight-similarity-based client clustering approach for non-iid federated learning. IEEE Internet of Things Journal, 9(20):20243-20256, 2022.","title":"Wscc: A weight-similarity-based client clustering approach for non-iid federated learning","authors":"Pu Tian, Weixian Liao, Wei Yu, Erik Blasch","year":"2022","venue":"IEEE Internet of Things Journal","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Tijani et al., 2021] Saheed A. Tijani, Xingjun Ma, Ran Zhang, Frank Jiang, and Robin Doss. Federated learning with extreme label skew: A data extension approach. In IJCNN, pages 1-8, 2021.","title":"Federated learning with extreme label skew: A data extension approach","authors":"A Saheed, Xingjun Tijani, Ran Ma, Frank Zhang, Robin Jiang, Doss","year":"2021","venue":"IJCNN","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang et al., 2020a] Hao Wang, Zakhary Kaplan, Di Niu, and Baochun Li. Optimizing federated learning on non- iid data with reinforcement learning. In INFOCOM, pages 1698-1707, 2020.","title":"Optimizing federated learning on noniid data with reinforcement learning","authors":"Wang","year":"2020","venue":"INFOCOM","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wang et al., 2020b] Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor. Tackling the objective in- consistency problem in heterogeneous federated optimiza- tion. Advances in neural information processing systems, 33:7611-7623, 2020.","title":"Tackling the objective inconsistency problem in heterogeneous federated optimization","authors":"Wang","year":"2020","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wang et al., 2021] Lixu Wang, Shichao Xu, Xiao Wang, and Qi Zhu. Addressing class imbalance in federated learning. In AAAI, pages 10165-10173, 2021.","title":"Addressing class imbalance in federated learning","authors":"Wang","year":"2021","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wang et al., 2023] Zheng Wang, Xiaoliang Fan, Jianzhong Qi, Haibing Jin, Peizhen Yang, Siqi Shen, and Cheng Wang. Fedgs: Federated graph-based sampling with ar- bitrary client availability. In AAAI, pages 10271-10278, 2023.","title":"Fedgs: Federated graph-based sampling with arbitrary client availability","authors":"Zheng Wang, Xiaoliang Fan, Jianzhong Qi, Haibing Jin, Peizhen Yang, Siqi Shen, Cheng Wang","year":"2023","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wu et al., 2023] Nannan Wu, Li Yu, Xuefeng Jiang, Kwang-Ting Cheng, and Zengqiang Yan. Fednoro: To- wards noise-robust federated learning by addressing class imbalance and label noise heterogeneity. In IJCAI, pages 4424-4432, 2023.","title":"Fednoro: Towards noise-robust federated learning by addressing class imbalance and label noise heterogeneity","authors":"Nannan Wu, Li Yu, Xuefeng Jiang, Kwang-Ting Cheng, Zengqiang Yan","year":"2023","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2023/492"},{"position":29,"raw":"[Xiao et al., 2017] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for bench- marking machine learning algorithms. arXiv preprint arXiv:1708.07747, 2017.","title":"Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms","authors":"Han Xiao, Kashif Rasul, Roland Vollgraf","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Yan and Zhu, 2023] Yunlu Yan and Lei Zhu. A simple data augmentation for feature distribution skewed feder- ated learning. arXiv preprint arXiv:2306.09363, 2023.","title":"A simple data augmentation for feature distribution skewed federated learning","authors":"Yan, Zhu ; ] Yunlu Yan, Lei Zhu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhang et al., 2021a] Chen Zhang, Yu Xie, Hang Bai, Bin Yu, Weihong Li, and Yuan Gao. A survey on federated learning. Knowledge-Based Systems, 216:106775, 2021.","title":"A survey on federated learning","authors":"Zhang","year":"2021","venue":"Knowledge-Based Systems","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhang et al., 2021b] Shulai Zhang, Zirui Li, Quan Chen, Wenli Zheng, Jingwen Leng, and Minyi Guo. Dubhe: To- wards data unbiasedness with homomorphic encryption in federated learning client selection. In ICPP, pages 1-10, 2021.","title":"Dubhe: Towards data unbiasedness with homomorphic encryption in federated learning client selection","authors":"Zhang","year":"2021","venue":"ICPP","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhang et al., 2022] Jie Zhang, Zhiqi Li, Bo Li, Jianghe Xu, Shuang Wu, Shouhong Ding, and Chao Wu. Federated learning with label distribution skew via logits calibration. In ICML, pages 26311-26329, 2022.","title":"Federated learning with label distribution skew via logits calibration","authors":"Jie Zhang, Zhiqi Li, Bo Li, Jianghe Xu, Shuang Wu, Shouhong Ding, Chao Wu","year":"2022","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zhao et al., 2018] Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Fed- erated learning with non-iid data. arXiv preprint arXiv:1806.00582, 2018.","title":"Federated learning with non-iid data","authors":"Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, Vikas Chandra","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Zhu et al., 2021] Hangyu Zhu, Jinjin Xu, Shiqing Liu, and Yaochu Jin. Federated learning on non-iid data: A survey. Neurocomputing, 465:371-390, 2021.","title":"Federated learning on non-iid data: A survey","authors":"Zhu","year":"2021","venue":"Neurocomputing","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}