{"type":"ScholarlyArticle","title":"FedFa: A Fully Asynchronous Training Paradigm for Federated Learning","authors":[{"name":"Haotian Xu","slug":"haotian-xu","given":"Haotian","family":"Xu","affiliation":"The Hong Kong Polytechnic University , Hong Kong"},{"name":"Zhaorui Zhang","slug":"zhaorui-zhang","given":"Zhaorui","family":"Zhang","affiliation":"The Hong Kong Polytechnic University , Hong Kong"},{"name":"Sheng Di","slug":"sheng-di","given":"Sheng","family":"Di","affiliation":"Argonne National Laboratory , USA"},{"name":"Benben Liu","slug":"benben-liu","given":"Benben","family":"Liu","affiliation":"The University of Hong Kong , Hong Kong"},{"name":"Khalid Ayed Alharthi","slug":"khalid-ayed-alharthi","given":"Khalid Ayed","family":"Alharthi","affiliation":"Department of Computer Science , College Of Computing , University of Bisha , Bisha 61922 , P.O. Box 551 , Saudi Arabia."},{"name":"Jiannong Cao","slug":"jiannong-cao","given":"Jiannong","family":"Cao","affiliation":"The Hong Kong Polytechnic University , Hong Kong"}],"abstract":"Federated learning has been identified as an efficient decentralized training paradigm for scaling the machine learning model training on a large number of devices while guaranteeing the data privacy of the trainers. FedAvg has become a foundational parameter update strategy for federated learning, which has been promising to eliminate the effect of the heterogeneous data across clients and guarantee convergence. However, the synchronization parameter update barriers for each communication round during the training significant time on waiting, slowing down the training procedure. Therefore, recent state-of-the-art solutions propose using semi-asynchronous approaches to mitigate the waiting time cost with guaranteed convergence. Nevertheless, emerging semi-asynchronous approaches are unable to eliminate the waiting time completely. \r\n\r\nWe propose a full asynchronous training paradigm called FedFa, which can guarantee model convergence and eliminate the waiting time completely for federated learning by using a few buffered results on the server for parameter updating. Further, we provide theoretical proof of the convergence rate for our proposed FedFa. Extensive experimental results indicate our approach effectively improves the training performance of federated learning by up to 6x and 4x speedup compared to the state-of-the-art synchronous and semi-asynchronous strategies while retaining high accuracy in both IID and Non-IID scenarios.","doi":"10.24963/ijcai.2024/584","url":"https://ijcai.org/proceedings/2024/584","pdf":"https://ijcai.org/proceedings/2024/0584.pdf","markdown":"https://ijcai.org/proceedings/2024/584.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/584","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Machine Learning","track":"Main Track","number":584,"pages":{"first":5281,"last":5288},"keywords":["Multidisciplinary Topics and Applications: MTA: Software engineering","Machine Learning: ML: Federated learning"],"videos":[],"references":[{"position":1,"raw":"[Bhargava et al., 2021] Prajjwal Bhargava, Aleksandr Drozd, and Anna Rogers. Generalization in nli: Ways (not) to go beyond simple heuristics, 2021.","title":"Generalization in nli: Ways (not) to go beyond simple heuristics","authors":"Bhargava","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bonawitz et al., 2016] Keith Bonawitz, Vladimir Ivanov, Ben Kreuter, Antonio Marcedone, H Brendan McMa- han, Sarvar Patel, Daniel Ramage, Aaron Segal, and Karn Seth. Practical secure aggregation for federated learning on user-held data. arXiv preprint arXiv:1611.04482, 2016.","title":"Practical secure aggregation for federated learning on user-held data","authors":"Keith Bonawitz, Vladimir Ivanov, Ben Kreuter, Antonio Marcedone, Brendan Mcmahan, Sarvar Patel, Daniel Ramage, Aaron Segal, Karn Seth","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Caldas et al., 2018] Sebastian Caldas, Sai Meher Karthik Duddu, Peter Wu, Tian Li, Jakub Konečnỳ, H Bren- dan McMahan, Virginia Smith, and Ameet Talwalkar. Leaf: A benchmark for federated settings. arXiv preprint:1812.01097, 2018.","title":"Leaf: A benchmark for federated settings","authors":"Sebastian Caldas, Sai Meher, Karthik Duddu, Peter Wu, Tian Li, Jakub Konečnỳ, Brendan Mcmahan, Virginia Smith, Ameet Talwalkar","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chai et al., 2020] Zheng Chai, Yujing Chen, Liang Zhao, Yue Cheng, and Huzefa Rangwala. Fedat: A communication-efficient federated learning method with asynchronous tiers under non-iid data. ArXivorg, 2020.","title":"Fedat: A communication-efficient federated learning method with asynchronous tiers under non-iid data","authors":"Zheng Chai, Yujing Chen, Liang Zhao, Yue Cheng, Huzefa Rangwala","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cho et al., 2022] Yae Jee Cho, Andre Manoel, Gauri Joshi, Robert Sim, and Dimitriadis. Heterogeneous ensemble knowledge transfer for training large models in federated learning. arXiv preprint arXiv:2204.12703, 2022.","title":"Heterogeneous ensemble knowledge transfer for training large models in federated learning","authors":"Yae Jee Cho, Andre Manoel, Gauri Joshi, Robert Sim, Dimitriadis","year":"2022","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2022/399"},{"position":6,"raw":"[Fraboni et al., 2023] Yann Fraboni, Richard Vidal, Laetitia Kameni, and Marco Lorenzi. A general theory for fed- erated optimization with asynchronous and heterogeneous clients updates. Journal of Machine Learning Research, 24(110):1-43, 2023.","title":"A general theory for federated optimization with asynchronous and heterogeneous clients updates","authors":"Yann Fraboni, Richard Vidal, Laetitia Kameni, Marco Lorenzi","year":"2023","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hsu et al., 2019] Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.06335, 2019.","title":"Measuring the effects of non-identical data distribution for federated visual classification","authors":"Hsu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hu et al., 2021] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021.","title":"Lora: Low-rank adaptation of large language models","authors":"J Edward, Yelong Hu, Phillip Shen, Zeyuan Wallis, Yuanzhi Allen-Zhu, Shean Li, Lu Wang, Weizhu Wang, Chen","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[K. and Geoffrey, 2009] Alex K. and H. Geoffrey. Learning multiple layers of features from tiny images. 2009.","title":"Learning multiple layers of features from tiny images","authors":"Alex K, H Geoffrey","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kairouz et al., 2021] Peter Kairouz, H Brendan McMa- han, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Ar- jun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. Foundations and Trends® in Machine Learning, 14(1-2):1-210, 2021.","title":"Advances and open problems in federated learning","authors":"Peter Kairouz, Brendan Mcmahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Nitin Arjun, Kallista Bhagoji, Zachary Bonawitz, Graham Charles, Rachel Cormode, Cummings","year":"2021","venue":"Foundations and Trends® in Machine Learning","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Karl et al., 2020] Ryan Karl, Jonathan Takeshita, Nirajan Koirla, and Taeho Jung. Cryptonite: A framework for flex- ible time-series secure aggregation with online fault toler- ance. Cryptology ePrint Archive, Paper 2020/1561, 2020.","title":"Cryptonite: A framework for flexible time-series secure aggregation with online fault tolerance","authors":"Ryan Karl, Jonathan Takeshita, Nirajan Koirla, Taeho Jung","year":"2020","venue":"Cryptology ePrint Archive","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Koloskova et al., 2022] Anastasiia Koloskova, Sebastian U Stich, and Martin Jaggi. Sharper convergence guarantees for asynchronous sgd for distributed and federated learn- ing. Advances in Neural Information Processing Systems, 35:17202-17215, 2022.","title":"Sharper convergence guarantees for asynchronous sgd for distributed and federated learning","authors":"Anastasiia Koloskova, Sebastian U Stich, Martin Jaggi","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li et al., 2020] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Pro- ceedings of MLSys, 2:429-450, 2020.","title":"Federated optimization in heterogeneous networks","authors":"Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, Virginia Smith","year":"2020","venue":"Proceedings of MLSys","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mania et al., 2015] Horia Mania, Xinghao Pan, Dimitris Papailiopoulos, Benjamin Recht, Kannan Ramchandran, and Michael I Jordan. Perturbed iterate analysis for asynchronous stochastic optimization. arXiv preprint arXiv:1507.06970, 2015.","title":"Perturbed iterate analysis for asynchronous stochastic optimization","authors":"Horia Mania, Xinghao Pan, Dimitris Papailiopoulos, Benjamin Recht, Kannan Ramchandran, Michael I Jordan","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[McMahan et al., 2017] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Ar- cas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273-1282. PMLR, 2017.","title":"Communication-efficient learning of deep networks from decentralized data","authors":"Brendan Mcmahan, Eider Moore, Daniel Ramage, Seth Hampson, Blaise Aguera Y Arcas","year":"2017","venue":"Artificial intelligence and statistics","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Nguyen et al., 2022] John Nguyen, Kshitiz Malik, Hongyuan Zhan, Ashkan Yousefpour, Mike Rabbat, Mani Malek, and Dzmitry Huba. Federated learning with buffered asynchronous aggregation. In International Conference on Artificial Intelligence and Statistics, pages 3581-3607. PMLR, 2022.","title":"Federated learning with buffered asynchronous aggregation","authors":"John Nguyen, Kshitiz Malik, Hongyuan Zhan, Ashkan Yousefpour, Mike Rabbat, Mani Malek, Dzmitry Huba","year":"2022","venue":"International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[So et al., 2021] Jinhyun So, Bas ¸ak Güler, and A Salman Avestimehr. Turbo-aggregate: Breaking the quadratic ag- gregation barrier in secure federated learning. IEEE Jour- nal on Selected Areas in Information Theory, 2021.","title":"and A Salman Avestimehr. Turbo-aggregate: Breaking the quadratic aggregation barrier in secure federated learning","authors":"So","year":"2021","venue":"IEEE Journal on Selected Areas in Information Theory","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Su and Li, 2022] Ningxin Su and Baochun Li. How asyn- chronous can federated learning be? In 2022 IEEE/ACM 30th International Symposium on Quality of Service (IWQoS), pages 1-11. IEEE, 2022.","title":"How asynchronous can federated learning be?","authors":"Li ; Su, Ningxin Su, Baochun Li","year":"2022","venue":"IEEE/ACM 30th International Symposium on Quality of Service (IWQoS)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[U, 2018] Sebastian U. Local sgd converges fast and com- municates little. arXiv preprint arXiv:1805.09767, 2018.","title":"Local sgd converges fast and communicates little","authors":"U Sebastian","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang et al., 2020] Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor. Tackling the objective in- consistency problem in heterogeneous federated optimiza- tion. NeurIPS, 2020.","title":"Tackling the objective inconsistency problem in heterogeneous federated optimization","authors":"Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, H Vincent Poor","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zhang and Wang, 2021] Zhaorui Zhang and Choli Wang. Sapus: Self-adaptive parameter update strategy for dnn training on multi-gpu clusters. IEEE Transactions on Par- allel and Distributed Systems, 33(7):1569-1580, 2021.","title":"Sapus: Self-adaptive parameter update strategy for dnn training on multi-gpu clusters","authors":"Wang ; ] Zhaorui Zhang, Choli Zhang, Wang","year":"2021","venue":"IEEE Transactions on Parallel and Distributed Systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhang and Wang, 2022] Zhaorui Zhang and Choli Wang. Mipd: An adaptive gradient sparsification framework for distributed dnns training. IEEE Transactions on Parallel and Distributed Systems, 33(11):3053-3066, 2022.","title":"Mipd: An adaptive gradient sparsification framework for distributed dnns training","authors":"Wang ; ] Zhaorui Zhang, Choli Zhang, Wang","year":"2022","venue":"IEEE Transactions on Parallel and Distributed Systems","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhang et al., 2022] Zhaorui Zhang, Zhuoran Ji, and Choli Wang. Momentum-driven adaptive synchronization model for distributed dnn training on hpc clusters. Journal of Parallel and Distributed Computing, 159:65-84, 2022.","title":"Momentum-driven adaptive synchronization model for distributed dnn training on hpc clusters","authors":"Zhang","year":"2022","venue":"Journal of Parallel and Distributed Computing","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhao et al., 2018] Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Fed- erated learning with non-iid data. arXiv preprint arXiv:1806.00582, 2018.","title":"Federated learning with non-iid data","authors":"Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, Vikas Chandra","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}