{"type":"ScholarlyArticle","title":"Feature Norm Regularized Federated Learning: Utilizing Data Disparities for Model Performance Gains","authors":[{"name":"Ke Hu","slug":"ke-hu","given":"Ke","family":"Hu","affiliation":"School of Cyber Science and Engineering , Shanghai Jiao Tong University"},{"name":"Liyao Xiang","slug":"liyao-xiang","given":"Liyao","family":"Xiang","affiliation":"John Hopcroft Center for Computer Science , Shanghai Jiao Tong University"},{"name":"Peng Tang","slug":"peng-tang","given":"Peng","family":"Tang","affiliation":"School of Cyber Science and Engineering , Shanghai Jiao Tong University"},{"name":"Weidong Qiu","slug":"weidong-qiu","given":"Weidong","family":"Qiu","affiliation":"School of Cyber Science and Engineering , Shanghai Jiao Tong University"}],"abstract":"Federated learning (FL) is a machine learning paradigm that aggregates knowledge and utilizes computational power from multiple participants to train a global model. However, a commonplace challenge—non-independent and identically distributed (non-i.i.d.) data across participants—can lead to significant divergence in model updates, thus diminishing training efficacy. In this paper, we propose the Feature Norm Regularized Federated Learning (FNR-FL) algorithm to tackle the non-i.i.d challenge. FNR-FL incorporates class average feature norms into the loss function by a straightforward yet effective regularization strategy. The core idea of FNR-FL is to penalize the deviations in the update directions of local models caused by the non-i.i.d data. Theoretically, we provide convergence guarantees for FNR-FL when training under non-i.i.d scenarios. Practically, our comprehensive experimental evaluations demonstrate that FNR-FL significantly outperforms existing FL algorithms in terms of test accuracy, and maintains a competitive convergence rate with lower communication overhead and shorter duration. Compared to FedAvg, FNR-FL exhibits a 66.24% improvement in accuracy and an 11.40% reduction in training time, underscoring its enhanced effectiveness and efficiency. The code is available on GitHub at: https://github.com/LonelyMoonDesert/FNR-FL.","doi":"10.24963/ijcai.2024/457","url":"https://ijcai.org/proceedings/2024/457","pdf":"https://ijcai.org/proceedings/2024/0457.pdf","markdown":"https://ijcai.org/proceedings/2024/457.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/457","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Machine Learning","track":"Main Track","number":457,"pages":{"first":4136,"last":4146},"keywords":["Machine Learning: ML: Federated learning","Machine Learning: ML: Robustness","Machine Learning: ML: Supervised Learning","Machine Learning: ML: Optimization"],"videos":[],"references":[{"position":1,"raw":"[Acar et al., 2021] Durmus Alp Emre Acar, Yue Zhao, Ra- mon Matas Navarro, Matthew Mattina, Paul N. What- mough, and Venkatesh Saligrama. Federated learning based on dynamic regularization, 2021.","title":"Whatmough, and Venkatesh Saligrama","authors":"Durmus Alp, Emre Acar, Yue Zhao, Ramon Matas Navarro, Matthew Mattina, N Paul","year":"2021","venue":"Federated learning based on dynamic regularization","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Archetti et al., 2023] Alberto Archetti, Eugenio Lomurno, Francesco Lattari, André Martin, and Matteo Matteucci. Heterogeneous datasets for federated survival analysis sim- ulation. In Companion of the 2023 ACM/SPEC Interna- tional Conference on Performance Engineering, pages 173- 180, 2023.","title":"Heterogeneous datasets for federated survival analysis simulation","authors":"Alberto Archetti, Eugenio Lomurno, Francesco Lattari, André Martin, Matteo Matteucci","year":"2023","venue":"Companion of the 2023 ACM/SPEC International Conference on Performance Engineering","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Banabilah et al., 2022] Syreen Banabilah, Moayad Aloqaily, Eitaa Alsayed, Nida Malik, and Yaser Jararweh. Federated learning review: Fundamentals, enabling technologies, and future applications. Information processing & management, 59(6):103061, 2022.","title":"Federated learning review: Fundamentals, enabling technologies, and future applications","authors":"Syreen Banabilah, Moayad Aloqaily, Eitaa Alsayed, Nida Malik, Yaser Jararweh","year":"2022","venue":"Information processing & management","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bao et al., 2022] Yajie Bao, Michael Crawshaw, Shan Luo, and Mingrui Liu. Fast composite optimization and statis- tical recovery in federated learning. In International Con- ference on Machine Learning, pages 1508-1536. PMLR, 2022.","title":"Fast composite optimization and statistical recovery in federated learning","authors":"Yajie Bao, Michael Crawshaw, Shan Luo, Mingrui Liu","year":"2022","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Brownlee, 2018] Jason Brownlee. Gentle introduction to vector norms in machine learning. Machine Learning Mas- tery, 2018.","title":"Gentle introduction to vector norms in machine learning","authors":"Jason Brownlee","year":"2018","venue":"Machine Learning Mastery","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Clauset, 2011] Aaron Clauset. A brief primer on probability distributions. In Santa Fe Institute, 2011.","title":"A brief primer on probability distributions","authors":"Aaron Clauset","year":"2011","venue":"Santa Fe Institute","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Du et al., 2022] Zhixu Du, Jingwei Sun, Ang Li, Pin-Yu Chen, Jianyi Zhang, Hai\" Helen\" Li, and Yiran Chen. Re- thinking normalization methods in federated learning. In Proceedings of the 3rd International Workshop on Dis- tributed Machine Learning, pages 16-22, 2022.","title":"Rethinking normalization methods in federated learning","authors":"Du","year":"2022","venue":"Proceedings of the 3rd International Workshop on Distributed Machine Learning","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gao et al., 2022] Liang Gao, Huazhu Fu, Li Li, Yingwen Chen, Ming Xu, and Cheng-Zhong Xu. Feddc: Federated learning with non-iid data via local drift decoupling and correction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10112- 10121, 2022.","title":"Feddc: Federated learning with non-iid data via local drift decoupling and correction","authors":"Liang Gao, Huazhu Fu, Li Li, Yingwen Chen, Ming Xu, Cheng-Zhong Xu","year":"2022","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Haddadpour and Mahdavi, 2019] Farzin Haddadpour and Mehrdad Mahdavi. On the convergence of local de- scent methods in federated learning. arXiv preprint arXiv:1910.14425, 2019.","title":"On the convergence of local descent methods in federated learning","authors":"Mahdavi Haddadpour, Farzin Haddadpour, Mehrdad Mahdavi","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hsieh et al., 2020] Kevin Hsieh, Amar Phanishayee, Onur Mutlu, and Phillip Gibbons. The non-iid data quagmire of decentralized machine learning. In International Con- ference on Machine Learning, pages 4387-4398. PMLR, 2020.","title":"The non-iid data quagmire of decentralized machine learning","authors":"Kevin Hsieh, Amar Phanishayee, Onur Mutlu, Phillip Gibbons","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Jin et al., 2023] Cheng Jin, Xuandong Chen, Yi Gu, and Qun Li. Feddyn: A dynamic and efficient federated distillation approach on recommender system. In 2022 IEEE 28th In- ternational Conference on Parallel and Distributed Systems (ICPADS), pages 786-793. IEEE, 2023.","title":"Feddyn: A dynamic and efficient federated distillation approach on recommender system","authors":"Jin Cheng, Xuandong Chen, Yi Gu, Qun Li","year":"2022","venue":"IEEE 28th International Conference on Parallel and Distributed Systems (ICPADS)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Karimireddy et al., 2020] Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International confer- ence on machine learning, pages 5132-5143. PMLR, 2020.","title":"Scaffold: Stochastic controlled averaging for federated learning","authors":"Karimireddy","year":"2020","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Krizhevsky, 2009] Alex Krizhevsky. Learning multiple lay- ers of features from tiny images. Technical report, Citeseer, 2009.","title":"Learning multiple layers of features from tiny images","authors":"Alex Krizhevsky","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2019] Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the convergence of fedavg on non-iid data. arXiv preprint arXiv:1907.02189, 2019.","title":"On the convergence of fedavg on non-iid data","authors":"Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, Zhihua Zhang","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Li et al., 2020] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Fed- erated optimization in heterogeneous networks. Proceed- ings of Machine learning and systems, 2:429-450, 2020.","title":"Federated optimization in heterogeneous networks","authors":"Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, Virginia Smith","year":"2020","venue":"Proceedings of Machine learning and systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Li et al., 2021a] Qinbin Li, Bingsheng He, and Dawn Song. Model-contrastive federated learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10713-10722, 2021.","title":"He, and Dawn Song. Model-contrastive federated learning","authors":"Li","year":"2021","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Li et al., 2021b] Xiaoxiao Li, Meirui Jiang, Xiaofei Zhang, Michael Kamp, and Qi Dou. Fedbn: Federated learning on non-iid features via local batch normalization. arXiv preprint arXiv:2102.07623, 2021.","title":"Fedbn: Federated learning on non-iid features via local batch normalization","authors":"Li","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Li et al., 2022] Qinbin Li, Yiqun Diao, Quan Chen, and Bingsheng He. Federated learning on non-iid data silos: An experimental study. In 2022 IEEE 38th International Conference on Data Engineering (ICDE), pages 965-978. IEEE, 2022.","title":"Federated learning on non-iid data silos: An experimental study","authors":"Li","year":"2022","venue":"IEEE 38th International Conference on Data Engineering (ICDE)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Lin et al., 2020] Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fu- sion in federated learning. Advances in Neural Information Processing Systems, 33:2351-2363, 2020.","title":"Ensemble distillation for robust model fusion in federated learning","authors":"Tao Lin, Lingjing Kong, Sebastian U Stich, Martin Jaggi","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[McMahan et al., 2017] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273-1282. PMLR, 2017.","title":"Communication-efficient learning of deep networks from decentralized data","authors":"Brendan Mcmahan, Eider Moore, Daniel Ramage, Seth Hampson, Blaise Aguera Y Arcas","year":"2017","venue":"Artificial intelligence and statistics","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Naik et al., 2023] Shraddha M Naik, Chinnamuthu Subra- mani, Ravi Prasad K Jagannath, and Anand Paul. Expo- nential filtering technique for euclidean norm-regularized extreme learning machines. Pattern Analysis and Applica- tions, pages 1-10, 2023.","title":"Exponential filtering technique for euclidean norm-regularized extreme learning machines","authors":"Chinnamuthu Shraddha M Naik, Ravi Subramani, Anand Prasad K Jagannath, Paul","year":"2023","venue":"Pattern Analysis and Applications","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Qi et al., 2023] Tao Qi, Fangzhao Wu, Chuhan Wu, Liang He, Yongfeng Huang, and Xing Xie. Differentially private knowledge transfer for federated learning. Nature Commu- nications, 14(1):3785, 2023.","title":"Differentially private knowledge transfer for federated learning","authors":"Tao Qi, Fangzhao Wu, Chuhan Wu, Liang He, Yongfeng Huang, Xing Xie","year":"2023","venue":"Nature Communications","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Robbins and Monro, 1951] Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathematical statistics, pages 400-407, 1951.","title":"The annals of mathematical statistics","authors":"Monro Robbins, Herbert Robbins, Sutton Monro","year":"1951","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Shao et al., 2023] Jiawei Shao, Zijian Li, Wenqiang Sun, Tailin Zhou, Yuchang Sun, Lumin Liu, Zehong Lin, and Jun Zhang. A survey of what to share in federated learning: Perspectives on model utility, privacy leakage, and com- munication efficiency. arXiv preprint arXiv:2307.10655, 2023.","title":"A survey of what to share in federated learning: Perspectives on model utility, privacy leakage, and communication efficiency","authors":"Jiawei Shao, Zijian Li, Wenqiang Sun, Tailin Zhou, Yuchang Sun, Lumin Liu, Zehong Lin, Jun Zhang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[T Dinh et al., 2020] Canh T Dinh, Nguyen Tran, and Josh Nguyen. Personalized federated learning with moreau en- velopes. Advances in Neural Information Processing Sys- tems, 33:21394-21405, 2020.","title":"Personalized federated learning with moreau envelopes","authors":"Nguyen Canh T Dinh, Josh Tran, Nguyen","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wang et al., 2019] Shiqiang Wang, Tiffany Tuor, Theodoros Salonidis, Kin K Leung, Christian Makaya, Ting He, and Kevin Chan. Adaptive federated learning in resource con- strained edge computing systems. IEEE journal on selected areas in communications, 37(6):1205-1221, 2019.","title":"Adaptive federated learning in resource constrained edge computing systems","authors":"Wang","year":"2019","venue":"IEEE journal on selected areas in communications","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wang et al., 2020] Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimiza- tion. Advances in neural information processing systems, 33:7611-7623, 2020.","title":"Tackling the objective inconsistency problem in heterogeneous federated optimization","authors":"Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, H Vincent Poor","year":"2020","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wei and Huang, 2023] Xiao-Xiang Wei and Hua Huang. Edge devices clustering for federated visual classification: A feature norm based framework. IEEE Transactions on Image Processing, 32:995-1010, 2023.","title":"Edge devices clustering for federated visual classification: A feature norm based framework","authors":"Xiao-Xiang Wei, Hua Huang","year":"2023","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhang et al., 2022a] Jie Zhang, Zhiqi Li, Bo Li, Jianghe Xu, Shuang Wu, Shouhong Ding, and Chao Wu. Federated learning with label distribution skew via logits calibration. In International Conference on Machine Learning, pages 26311-26329. PMLR, 2022.","title":"Federated learning with label distribution skew via logits calibration","authors":"Zhang","year":"2022","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhang et al., 2022b] Lin Zhang, Li Shen, Liang Ding, Dacheng Tao, and Ling-Yu Duan. Fine-tuning global model via data-free knowledge distillation for non-iid federated learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10174- 10183, 2022.","title":"Fine-tuning global model via data-free knowledge distillation for non-iid federated learning","authors":"Zhang","year":"2022","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhao et al., 2018] Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Fed- erated learning with non-iid data. arXiv preprint arXiv:1806.00582, 2018.","title":"Federated learning with non-iid data","authors":"Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, Vikas Chandra","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhou, 2018] Xingyu Zhou. On the fenchel duality between strong convexity and lipschitz continuous gradient. arXiv preprint arXiv:1803.06573, 2018.","title":"On the fenchel duality between strong convexity and lipschitz continuous gradient","authors":"Xingyu Zhou, Zhou","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zhu et al., 2021] Zhuangdi Zhu, Junyuan Hong, and Jiayu Zhou. Data-free knowledge distillation for heterogeneous federated learning. In International conference on machine learning, pages 12878-12889. PMLR, 2021.","title":"Data-free knowledge distillation for heterogeneous federated learning","authors":"Zhu","year":"2021","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}