{"type":"ScholarlyArticle","title":"Federated Learning on Heterogeneous and Long-Tailed Data via Classifier Re-Training with Federated Features","authors":[{"name":"Xinyi Shang","slug":"xinyi-shang","given":"Xinyi","family":"Shang","affiliation":"Fujian Key Laboratory of Sensing and Computing for Smart Cities , School of Informatics , Xiamen University , Xiamen , China"},{"name":"Yang Lu","slug":"yang-lu","given":"Yang","family":"Lu","affiliation":"Fujian Key Laboratory of Sensing and Computing for Smart Cities , School of Informatics , Xiamen University , Xiamen , China"},{"name":"Gang Huang","slug":"gang-huang","given":"Gang","family":"Huang","affiliation":"Zhejiang Lab , Hangzhou , China"},{"name":"Hanzi Wang","slug":"hanzi-wang","given":"Hanzi","family":"Wang","affiliation":"Fujian Key Laboratory of Sensing and Computing for Smart Cities , School of Informatics , Xiamen University , Xiamen , China"}],"abstract":"Federated learning (FL) provides a privacy-preserving solution for distributed machine learning tasks. One challenging problem that severely damages the performance of FL models is the co-occurrence of data heterogeneity and long-tail distribution, which frequently appears in real FL applications. In this paper, we reveal an intriguing fact that the biased classifier is the primary factor leading to the poor performance of the global model. Motivated by the above finding, we propose a novel and privacy-preserving FL method for heterogeneous and long-tailed data via Classifier Re-training with Federated Features (CReFF). The classifier re-trained on federated features can produce comparable performance as the one re-trained on real data in a privacy-preserving manner without information leakage of local data or class distribution. Experiments on several benchmark datasets show that the proposed CReFF is an effective solution to obtain a promising FL model under heterogeneous and long-tailed data. Comparative results with the state-of-the-art FL methods also validate the superiority of CReFF. Our code is available at https://github.com/shangxinyi/CReFF-FL.","doi":"10.24963/ijcai.2022/308","url":"https://ijcai.org/proceedings/2022/308","pdf":"https://ijcai.org/proceedings/2022/0308.pdf","markdown":"https://ijcai.org/proceedings/2022/308.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/308","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Data Mining","track":"Main Track","number":308,"pages":{"first":2218,"last":2224},"keywords":["Computer Vision: Machine Learning for Vision","Data Mining: Class Imbalance and Unequal Cost","Data Mining: Privacy Preserving Data Mining","Data Mining: Federated Learning"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986699","length_sec":88,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984842","length_sec":296,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Cao et al., 2019] Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label-distribution-aware margin loss. In NeurIPS, pages 1567-1578, 2019.","title":"Learning imbalanced datasets with label-distribution-aware margin loss","authors":"Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, Tengyu Ma","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen and Chao, 2021] Hong-You Chen and Wei-Lun Chao. Fedbe: Making bayesian model ensemble applicable to federated learning. In ICLR, 2021.","title":"Fedbe: Making bayesian model ensemble applicable to federated learning","authors":"Chen, Chao ; Hong-You Chen, Wei-Lun Chao","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Duan et al., 2020] Moming Duan, Duo Liu, Xianzhang Chen, Renping Liu, Yujuan Tan, and Liang Liang. Self- balancing federated learning with global imbalanced data in mobile systems. IEEE Transactions on Parallel and Distributed Systems, 32(1):59-71, 2020.","title":"Selfbalancing federated learning with global imbalanced data in mobile systems","authors":"Moming Duan, Duo Liu, Xianzhang Chen, Renping Liu, Yujuan Tan, Liang Liang","year":"2020","venue":"IEEE Transactions on Parallel and Distributed Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Hsu et al., 2019] Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.06335, 2019.","title":"Measuring the effects of non-identical data distribution for federated visual classification","authors":"Hsu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Huang et al., 2021] Hua Huang, Fanhua Shang, Yuanyuan Liu, and Hongying Liu. Behavior mimics distribution: Combining individual and group behaviors for federated learning. In IJCAI, pages 2556-2552, 2021.","title":"Behavior mimics distribution: Combining individual and group behaviors for federated learning","authors":"Hua Huang, Fanhua Shang, Yuanyuan Liu, Hongying Liu","year":"2021","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2021/352"},{"position":6,"raw":"[Jiang et al., 2021] Ziyue Jiang, Yi Ren, Ming Lei, and Zhou Zhao. Fedspeech: Federated text-to-speech with continual learning. In IJCAI, pages 3829-3835, 2021.","title":"Fedspeech: Federated text-to-speech with continual learning","authors":"Ziyue Jiang, Yi Ren, Ming Lei, Zhou Zhao","year":"2021","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2021/527"},{"position":7,"raw":"[Kang et al., 2020] Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classi- fier for long-tailed recognition. In ICLR, 2020.","title":"Decoupling representation and classifier for long-tailed recognition","authors":"Kang","year":"2020","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kim et al., 2020] Jaehyung Kim, Jongheon Jeong, and Jin- woo Shin. M2m: Imbalanced classification via major-to- minor translation. In CVPR, pages 13896-13905, 2020.","title":"M2m: Imbalanced classification via major-tominor translation","authors":"Kim","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Krizhevsky and Hinton, 2009] A. Krizhevsky and G. Hin- ton. Learning multiple layers of features from tiny images. In Technical report, pages 32-33. University of Toronto, 2009.","title":"Learning multiple layers of features from tiny images","authors":"A Krizhevsky, G Hinton","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li et al., 2020a] Tian Li, Anit Kumar Sahu, Ameet Tal- walkar, and Virginia Smith. Federated learning: Chal- lenges, methods, and future directions. IEEE Signal Pro- cessing Magazine, 37(3):50-60, 2020.","title":"Federated learning: Challenges, methods, and future directions","authors":"Li","year":"2020","venue":"IEEE Signal Processing Magazine","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2020b] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. In ML- Sys, pages 429-450, 2020.","title":"Federated optimization in heterogeneous networks","authors":"Li","year":"2020","venue":"ML-Sys","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lin et al., 2020] Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. In NeurIPS, pages 2351- 2363, 2020.","title":"Ensemble distillation for robust model fusion in federated learning","authors":"Lin","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Luo et al., 2021] Mi Luo, Fei Chen, Dapeng Hu, Yifan Zhang, and Jiashi Feng. No fear of heterogeneity: Classi- fier calibration for federated learning with non-iid data. In NeurIPS, 2021.","title":"No fear of heterogeneity: Classifier calibration for federated learning with non-iid data","authors":"Mi Luo, Fei Chen, Dapeng Hu, Yifan Zhang, Jiashi Feng","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[McMahan et al., 2017] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Ar- cas. Communication-efficient learning of deep networks from decentralized data. In AISTATS, pages 1273-1282, 2017.","title":"Communication-efficient learning of deep networks from decentralized data","authors":"Brendan Mcmahan, Eider Moore, Daniel Ramage, Seth Hampson, Blaise Aguera Y Arcas","year":"2017","venue":"AISTATS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Russakovsky et al., 2015] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, and M. Bernstein. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211-252, 2015.","title":"Imagenet large scale visual recognition challenge","authors":"O Russakovsky, J Deng, H Su, J Krause, S Satheesh, S Ma, Z Huang, A Karpathy, A Khosla, M Bernstein","year":"2015","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Sarkar et al., 2020] Dipankar Sarkar, Ankur Narang, and Sumit Rai. Fed-focal loss for imbalanced data classification in federated learning. arXiv preprint arXiv:2011.06283, 2020.","title":"Fed-focal loss for imbalanced data classification in federated learning","authors":"Dipankar Sarkar, Ankur Narang, Sumit Rai","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Wang et al., 2020] Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H. Vincent Poor. Tackling the objec- tive inconsistency problem in heterogeneous federated op- timization. In NeurIPS, pages 7611-7623, 2020.","title":"Tackling the objective inconsistency problem in heterogeneous federated optimization","authors":"Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, H Vincent Poor","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Wang et al., 2021a] Lixu Wang, Shichao Xu, Xiao Wang, and Qi Zhu. Addressing class imbalance in federated learning. In AAAI, pages 10165-10173, 2021.","title":"Addressing class imbalance in federated learning","authors":"Wang","year":"2021","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wang et al., 2021b] Xudong Wang, Long Lian, Zhongqi Miao, Ziwei Liu, and Stella X. Yu. Long-tailed recog- nition by routing diverse distribution-aware experts. In ICLR, 2021.","title":"Long-tailed recognition by routing diverse distribution-aware experts","authors":"Wang","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Xiang et al., 2020] Liuyu Xiang, Guiguang Ding, and Jun- gong Han. Learning from multiple experts: Self-paced knowledge distillation for long-tailed classification. In ECCV, pages 247-263, 2020.","title":"Liuyu Xiang, Guiguang Ding, and Jungong Han. Learning from multiple experts: Self-paced knowledge distillation for long-tailed classification","authors":"Xiang","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Yang et al., 2020] Miao Yang, Akitanoshou Wong, Hong- bin Zhu, Haifeng Wang, and Hua Qian. Federated learning with class imbalance reduction. arXiv preprint arXiv:2011.11266, 2020.","title":"Federated learning with class imbalance reduction","authors":"Miao Yang, Akitanoshou Wong, Hongbin Zhu, Haifeng Wang, Hua Qian","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zang et al., 2021] Yuhang Zang, Chen Huang, and Chen Change Loy. Fasa: Feature augmentation and sam- pling adaptation for long-tailed instance segmentation. In ICCV, 2021.","title":"Fasa: Feature augmentation and sampling adaptation for long-tailed instance segmentation","authors":"Yuhang Zang, Chen Huang, Chen Change Loy","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhang et al., 2021] Yifan Zhang, Bingyi Kang, Bryan Hooi, Shuicheng Yan, and Jiashi Feng. Deep long-tailed learn- ing: A survey. arXiv preprint arXiv:2110.04596, 2021.","title":"Deep long-tailed learning: A survey","authors":"Yifan Zhang, Bingyi Kang, Bryan Hooi, Shuicheng Yan, Jiashi Feng","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhao et al., 2021] Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient match- ing. In ICLR, 2021.","title":"Dataset condensation with gradient matching","authors":"Bo Zhao, Konda Reddy Mopuri, Hakan Bilen","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zheng et al., 2020] Wenbo Zheng, Lan Yan, Chao Gou, and Fei-Yue Wang. Federated meta-learning for fraudulent credit card detection. In IJCAI, pages 4654-4660, 2020.","title":"Federated meta-learning for fraudulent credit card detection","authors":"Wenbo Zheng, Lan Yan, Chao Gou, Fei-Yue Wang","year":"2020","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2020/642"}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}