{"type":"ScholarlyArticle","title":"FedSampling: A Better Sampling Strategy for Federated Learning","authors":[{"name":"Tao Qi","slug":"tao-qi","given":"Tao","family":"Qi","affiliation":"Department of Electronic Engineering , Tsinghua University , Beijing 100084 , China"},{"name":"Fangzhao Wu","slug":"fangzhao-wu","given":"Fangzhao","family":"Wu","affiliation":"Microsoft Research Asia , Beijing 100080 , China"},{"name":"Lingjuan Lyu","slug":"lingjuan-lyu","given":"Lingjuan","family":"Lyu","affiliation":"Sony AI , 1-7-1 Konan Minato-ku Tokyo 108-0075 , Japan"},{"name":"Yongfeng Huang","slug":"yongfeng-huang","given":"Yongfeng","family":"Huang","affiliation":"Department of Electronic Engineering , Tsinghua University , Beijing 100084 , China; Zhongguancun Laboratory , Beijing 100094 , China; Institute for Precision Medicine of Tsinghua University , Beijing 102218 , China"},{"name":"Xing Xie","slug":"xing-xie","given":"Xing","family":"Xie","affiliation":"Microsoft Research Asia , Beijing 100080 , China"}],"abstract":"Federated learning (FL) is an important technique for learning models from decentralized data in a privacy-preserving way. Existing FL methods usually uniformly sample clients for local model learning in each round. However, different clients may have significantly different data sizes, and the clients with more data cannot have more opportunities to contribute to model training, which may lead to inferior performance. In this paper, instead of client uniform sampling, we propose a novel data uniform sampling strategy for federated learning (FedSampling), which can effectively improve the performance of federated learning especially when client data size distribution is highly imbalanced across clients. In each federated learning round, local data on each client is randomly sampled for local model learning according to a probability based on the server desired sample size and the total sample size on all available clients. Since the data size on each client is privacy-sensitive, we propose a privacy-preserving way to estimate the total sample size with a differential privacy guarantee. Experiments on four benchmark datasets show that FedSampling can effectively improve the performance of federated learning.","doi":"10.24963/ijcai.2023/462","url":"https://ijcai.org/proceedings/2023/462","pdf":"https://ijcai.org/proceedings/2023/0462.pdf","markdown":"https://ijcai.org/proceedings/2023/462.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/462","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Machine Learning","track":"Main Track","number":462,"pages":{"first":4154,"last":4162},"keywords":["Data Mining: DM: Privacy-preserving data mining","Machine Learning: ML: Federated learning"],"videos":[],"references":[{"position":1,"raw":"[Achituve et al., 2021] Idan Achituve, Aviv Shamsian, Aviv Navon, Gal Chechik, and Ethan Fetaya. Personalized fed- erated learning with gaussian processes. NeurIPS, 2021.","title":"Personalized federated learning with gaussian processes","authors":"Idan Achituve, Aviv Shamsian, Aviv Navon, Gal Chechik, Ethan Fetaya","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Avdiukhin and Kasiviswanathan, 2021] Dmitrii Avdiukhin and Shiva Kasiviswanathan. Federated learning under ar- bitrary communication patterns. In ICML, pages 425-435, 2021.","title":"Dmitrii Avdiukhin and Shiva Kasiviswanathan. Federated learning under arbitrary communication patterns","authors":"Kasiviswanathan Avdiukhin","year":"2021","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bahr and Esseen, 1965] Bengt von Bahr and Carl-Gustav Esseen. Inequalities for the rth absolute moment of a sum of random variables, 1 ≤ r ≤ 2. The Annals of Mathemat- ical Statistics, pages 299-303, 1965.","title":"Inequalities for the rth absolute moment of a sum of random variables, 1 ≤ r ≤ 2","authors":"Esseen ; Bahr, Von Bengt, Carl-Gustav Bahr, Esseen","year":"1965","venue":"The Annals of Mathematical Statistics","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bai et al., 2021] Xiang Bai, Hanchen Wang, Liya Ma, Yongchao Xu, Jiefeng Gan, Ziwei Fan, Fan Yang, Ke Ma, Jiehua Yang, Song Bai, et al. Advancing covid-19 diagno- sis with privacy-preserving collaboration in artificial intel- ligence. Nature Machine Intelligence, pages 1-9, 2021.","title":"Advancing covid-19 diagnosis with privacy-preserving collaboration in artificial intelligence","authors":"Xiang Bai, Hanchen Wang, Liya Ma, Yongchao Xu, Jiefeng Gan, Ziwei Fan, Fan Yang, Ke Ma, Jiehua Yang, Song Bai","year":"2021","venue":"Nature Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cohen et al., 2017] Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre Van Schaik. Emnist: Extending mnist to handwritten letters. In IJCNN, pages 2921-2926, 2017.","title":"Emnist: Extending mnist to handwritten letters","authors":"Gregory Cohen, Saeed Afshar, Jonathan Tapson, Andre Van Schaik","year":"2017","venue":"IJCNN","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dayan et al., 2021] Ittai Dayan, Holger R Roth, Aoxiao Zhong, Ahmed Harouni, Amilcare Gentili, Anas Z Abidin, Andrew Liu, Anthony Beardsworth Costa, Bradford J Wood, Chien-Sung Tsai, et al. Federated learning for pre- dicting clinical outcomes in patients with covid-19. Nature Medicine, pages 1735-1743, 2021.","title":"Federated learning for predicting clinical outcomes in patients with covid-19","authors":"Ittai Dayan, Aoxiao Holger R Roth, Ahmed Zhong, Amilcare Harouni, Gentili, Andrew Anas Z Abidin, Anthony Beardsworth Liu, Bradford J Costa, Chien-Sung Wood, Tsai","year":"2021","venue":"Nature Medicine","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Fraboni et al., 2021a] Yann Fraboni, Richard Vidal, Laeti- tia Kameni, and Marco Lorenzi. Clustered sam- pling: Low-variance and improved representativity for clients selection in federated learning. arXiv preprint arXiv:2105.05883, 2021.","title":"Clustered sampling: Low-variance and improved representativity for clients selection in federated learning","authors":"Fraboni","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Fraboni et al., 2021b] Yann Fraboni, Richard Vidal, Laetitia Kameni, and Marco Lorenzi. On the impact of client sam- pling on federated learning convergence. arXiv preprint arXiv:2107.12211, 2021.","title":"On the impact of client sampling on federated learning convergence","authors":"Fraboni","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gerber et al., 2018] Nina Gerber, Paul Gerber, and Melanie Volkamer. Explaining the privacy paradox: A systematic review of literature investigating privacy attitude and be- havior. Computers & security, pages 226-261, 2018.","title":"Explaining the privacy paradox: A systematic review of literature investigating privacy attitude and behavior","authors":"Nina Gerber, Paul Gerber, Melanie Volkamer","year":"2018","venue":"Computers & security","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hamer et al., 2020] Jenny Hamer, Mehryar Mohri, and Ananda Theertha Suresh. Fedboost: A communication- efficient algorithm for federated learning. In ICML, pages 3973-3983, 2020.","title":"Fedboost: A communicationefficient algorithm for federated learning","authors":"Jenny Hamer, Mehryar Mohri, Ananda Theertha, Suresh","year":"2020","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Huang et al., 2021] Baihe Huang, Xiaoxiao Li, Zhao Song, and Xin Yang. Fl-ntk: A neural tangent kernel-based framework for federated learning analysis. In ICML, pages 4423-4434, 2021.","title":"Fl-ntk: A neural tangent kernel-based framework for federated learning analysis","authors":"Baihe Huang, Xiaoxiao Li, Zhao Song, Xin Yang","year":"2021","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Jiang et al., 2021] Ziyue Jiang, Yi Ren, Ming Lei, and Zhou Zhao. Fedspeech: Federated text-to-speech with continual learning. In IJCAI, pages 3829-3835, 2021.","title":"Fedspeech: Federated text-to-speech with continual learning","authors":"Ziyue Jiang, Yi Ren, Ming Lei, Zhou Zhao","year":"2021","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2021/527"},{"position":14,"raw":"[Jumper et al., 2021] John Jumper, Richard Evans, Alexan- der Pritzel, Tim Green, Michael Figurnov, Olaf Ron- neberger, Kathryn Tunyasuvunakool, Russ Bates, Au- gustin Žídek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, pages 583-589, 2021.","title":"Highly accurate protein structure prediction with alphafold","authors":"Jumper","year":"2021","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kim, 2014] Yoon Kim. Convolutional neural networks for sentence classification. In EMNLP, pages 1746-1751, 2014.","title":"Convolutional neural networks for sentence classification","authors":"Yoon Kim","year":"2014","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kingma and Ba, 2015] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Li et al., 2019] Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the convergence of fedavg on non-iid data. In ICLR, 2019.","title":"On the convergence of fedavg on non-iid data","authors":"Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, Zhihua Zhang","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Li et al., 2020] Tian Li, Anit Kumar Sahu, Ameet Tal- walkar, and Virginia Smith. Federated learning: Chal- lenges, methods, and future directions. IEEE Signal Pro- cessing Magazine, pages 50-60, 2020.","title":"Federated learning: Challenges, methods, and future directions","authors":"Tian Li, Anit Kumar Sahu, Ameet Talwalkar, Virginia Smith","year":"2020","venue":"IEEE Signal Processing Magazine","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Liang et al., 2021] Feng Liang, Weike Pan, and Zhong Ming. Fedrec++: Lossless federated recommendation with explicit feedback. In AAAI, pages 4224-4231, 2021.","title":"Fedrec++: Lossless federated recommendation with explicit feedback","authors":"Feng Liang, Weike Pan, Zhong Ming","year":"2021","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Lydia and Francis, 2019] Agnes Lydia and Sagayaraj Fran- cis. Adagrad-an optimizer for stochastic gradient de- scent. IJICS, 2019.","title":"Agnes Lydia and Sagayaraj Francis. Adagrad-an optimizer for stochastic gradient descent","authors":"Lydia, Francis","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Lyu et al., 2020] Lingjuan Lyu, Han Yu, Xingjun Ma, Lichao Sun, Jun Zhao, Qiang Yang, and Philip S Yu. Pri- vacy and robustness in federated learning: Attacks and de- fenses. arXiv preprint arXiv:2012.06337, 2020.","title":"Privacy and robustness in federated learning: Attacks and defenses","authors":"Lyu","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[McAuley et al., 2015] Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton Van Den Hengel. Image-based recommendations on styles and substitutes. In SIGIR, pages 43-52, 2015.","title":"Image-based recommendations on styles and substitutes","authors":"Julian Mcauley, Christopher Targett, Qinfeng Shi, Anton Van Den, Hengel","year":"2015","venue":"SIGIR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[McMahan et al., 2017] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Ar- cas. Communication-efficient learning of deep networks from decentralized data. In AISTATS, pages 1273-1282, 2017.","title":"Communication-efficient learning of deep networks from decentralized data","authors":"Brendan Mcmahan, Eider Moore, Daniel Ramage, Seth Hampson, Blaise Aguera Y Arcas","year":"2017","venue":"AISTATS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Reddi et al., 2018] S Reddi, Manzil Zaheer, Devendra Sachan, Satyen Kale, and Sanjiv Kumar. Adaptive meth- ods for nonconvex optimization. In NIPS, 2018.","title":"Adaptive methods for nonconvex optimization","authors":"Manzil Reddi, Devendra Zaheer, Satyen Sachan, Sanjiv Kale, Kumar","year":"2018","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Reddi et al., 2020] Sashank J Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečnỳ, Sanjiv Kumar, and Hugh Brendan McMahan. Adaptive federated optimization. In ICLR, 2020.","title":"Adaptive federated optimization","authors":"J Sashank, Zachary Reddi, Manzil Charles, Zachary Zaheer, Keith Garrett, Jakub Rush, Sanjiv Konečnỳ, Hugh Brendan Kumar, Mcmahan","year":"2020","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Rothchild et al., 2020] Daniel Rothchild, Ashwinee Panda, Enayat Ullah, Nikita Ivkin, Ion Stoica, Vladimir Braver- man, Joseph Gonzalez, and Raman Arora. Fetchsgd: Communication-efficient federated learning with sketch- ing. In ICML, pages 8253-8265, 2020.","title":"Fetchsgd: Communication-efficient federated learning with sketching","authors":"Daniel Rothchild, Ashwinee Panda, Enayat Ullah, Nikita Ivkin, Ion Stoica, Vladimir Braverman, Joseph Gonzalez, Raman Arora","year":"2020","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Sun et al., 2021] Lichao Sun, Jianwei Qian, and Xun Chen. Ldp-fl: Practical private aggregation in federated learning with local differential privacy. In Zhi-Hua Zhou, editor, IJCAI, pages 1571-1578, 2021.","title":"Ldp-fl: Practical private aggregation in federated learning with local differential privacy","authors":"Lichao Sun, Jianwei Qian, Xun Chen","year":"2021","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2021/217"},{"position":28,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, pages 5998-6008, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wang et al., 2020] Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor. Tackling the objective in- consistency problem in heterogeneous federated optimiza- tion. NeurIPS, 2020.","title":"Tackling the objective inconsistency problem in heterogeneous federated optimization","authors":"Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, H Vincent Poor","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wang et al., 2021] Zheng Wang, Xiaoliang Fan, Jianzhong Qi, Chenglu Wen, Cheng Wang, and Rongshan Yu. Fed- erated learning with fair averaging. In IJCIA, pages 1615- 1623, 2021.","title":"Federated learning with fair averaging","authors":"Zheng Wang, Xiaoliang Fan, Jianzhong Qi, Chenglu Wen, Cheng Wang, Rongshan Yu","year":"2021","venue":"IJCIA","doi":null,"url":null,"internal_url":"/proceedings/2021/223"},{"position":31,"raw":"[Wu et al., 2020] Fangzhao Wu, Ying Qiao, Jiun-Hung Chen, Chuhan Wu, Tao Qi, Jianxun Lian, Danyang Liu, Xing Xie, Jianfeng Gao, Winnie Wu, et al. Mind: A large- scale dataset for news recommendation. In ACL, pages 3597-3606, 2020.","title":"Mind: A largescale dataset for news recommendation","authors":"Fangzhao Wu, Ying Qiao, Jiun-Hung Chen, Chuhan Wu, Tao Qi, Jianxun Lian, Danyang Liu, Xing Xie, Jianfeng Gao, Winnie Wu","year":"2020","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Yang et al., 2018] Timothy Yang, Galen Andrew, Hubert Eichner, Haicheng Sun, Wei Li, Nicholas Kong, Daniel Ramage, and Franc ¸oise Beaufays. Applied federated learning: Improving google keyboard query suggestions. arXiv preprint arXiv:1812.02903, 2018.","title":"Applied federated learning: Improving google keyboard query suggestions","authors":"Timothy Yang, Galen Andrew, Hubert Eichner, Haicheng Sun, Wei Li, Nicholas Kong, Daniel Ramage, Franc, Beaufays","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Yoon et al., 2021] Jaehong Yoon, Wonyong Jeong, Gi- woong Lee, Eunho Yang, and Sung Ju Hwang. Federated continual learning with weighted inter-client transfer. In ICML, pages 12073-12086, 2021.","title":"Federated continual learning with weighted inter-client transfer","authors":"Yoon","year":"2021","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Yuan et al., 2021] Honglin Yuan, Manzil Zaheer, and Sashank Reddi. Federated composite optimization. In ICML, pages 12253-12266, 2021.","title":"Federated composite optimization","authors":"Honglin Yuan, Manzil Zaheer, Sashank Reddi","year":"2021","venue":"ICML","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}