{"type":"ScholarlyArticle","title":"A Survey on Efficient Federated Learning Methods for Foundation Model Training","authors":[{"name":"Herbert Woisetschläger","slug":"herbert-woisetschlager","given":"Herbert","family":"Woisetschläger","affiliation":"Technical University of Munich"},{"name":"Alexander Erben","slug":"alexander-erben","given":"Alexander","family":"Erben","affiliation":"Technical University of Munich"},{"name":"Shiqiang Wang","slug":"shiqiang-wang","given":"Shiqiang","family":"Wang","affiliation":"IBM Research"},{"name":"Ruben Mayer","slug":"ruben-mayer","given":"Ruben","family":"Mayer","affiliation":"University of Bayreuth"},{"name":"Hans-Arno Jacobsen","slug":"hans-arno-jacobsen","given":"Hans-Arno","family":"Jacobsen","affiliation":"University of Toronto"}],"abstract":"Federated Learning (FL) has become an established technique to facilitate privacy-preserving collaborative training across a multitude of clients. However, new approaches to FL often discuss their contributions involving small deep-learning models only and focus on training full models on clients. In the wake of Foundation Models (FM), the reality is different for many deep learning applications. Typically, FMs have already been pre-trained across a wide variety of tasks and can be fine-tuned to specific downstream tasks over significantly smaller datasets than required for full model training. However, access to such datasets is often challenging. By its design, FL can help to open data silos. With this survey, we introduce a novel taxonomy focused on computational and communication efficiency, the vital elements to make use of FMs in FL systems. We discuss the benefits and drawbacks of parameter-efficient fine-tuning (PEFT) for FL applications, elaborate on the readiness of FL frameworks to work with FMs and provide future research opportunities on how to evaluate generative models in FL as well as the interplay of privacy and PEFT.","doi":"10.24963/ijcai.2024/919","url":"https://ijcai.org/proceedings/2024/919","pdf":"https://ijcai.org/proceedings/2024/0919.pdf","markdown":"https://ijcai.org/proceedings/2024/919.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/919","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Survey Track","track":"Survey Track","number":919,"pages":{"first":8317,"last":8325},"keywords":["Data Mining: DM: Big data and scalability","Machine Learning: ML: Federated learning"],"videos":[],"references":[{"position":1,"raw":"[Aledhari et al., 2020] Mohammed Aledhari, Rehma Razzak, Reza M. Parizi, and Fahad Saeed. Federated learning: A survey on enabling technologies, protocols, and applications. IEEE Access, 8:140699-140725, 2020.","title":"Federated learning: A survey on enabling technologies, protocols, and applications","authors":"Aledhari","year":"2020","venue":"IEEE Access","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Alistarh et al., 2017] Dan Alistarh, Demjan Grubic, et al. Qsgd: Communication-efficient sgd via gradient quantization and en- coding. In Advances in Neural Information Processing Systems, volume 30, 2017.","title":"Qsgd: Communication-efficient sgd via gradient quantization and encoding","authors":"Dan Alistarh, Demjan Grubic","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Babakniya et al., 2023] Sara Babakniya, Ahmed R. Elkordy, et al. Slora: Federated parameter efficient fine-tuning of language models, 2023.","title":"Slora: Federated parameter efficient fine-tuning of language models","authors":"Sara Babakniya, Ahmed R Elkordy","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Banabilah et al., 2022] Syreen Banabilah, Moayad Aloqaily, et al. Federated learning review: Fundamentals, enabling technologies, and future applications. Information Processing & Management, 59(6), 2022.","title":"Federated learning review: Fundamentals, enabling technologies, and future applications","authors":"Syreen Banabilah, Moayad Aloqaily","year":"2022","venue":"Information Processing & Management","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Beutel et al., 2020] Daniel J. Beutel, Taner Topal, et al. Flower: A friendly federated learning research framework, 2020.","title":"Flower: A friendly federated learning research framework","authors":"J Daniel, Taner Beutel, Topal","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Bommasani et al., 2021] Rishi Bommasani, Drew A. Hudson, et al. On the opportunities and risks of foundation models, 2021.","title":"On the opportunities and risks of foundation models","authors":"Rishi Bommasani, Drew A Hudson","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chen et al., 2023] Yuanyuan Chen, Zichen Chen, et al. Fedobd: Opportunistic block dropout for efficiently training large-scale neural networks through federated learning. In Proceedings of the Thirty-Second International Joint Conference on Artificial In- telligence, IJCAI-23. IJCAI Org., 2023.","title":"Fedobd: Opportunistic block dropout for efficiently training large-scale neural networks through federated learning","authors":"Yuanyuan Chen, Zichen Chen","year":"2023","venue":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI-23","doi":null,"url":null,"internal_url":"/proceedings/2023/394"},{"position":8,"raw":"[Diao et al., 2020] Enmao Diao, Jie Ding, et al. Heterofl: Compu- tation and communication efficient federated learning for hetero- geneous clients, 2020.","title":"Heterofl: Computation and communication efficient federated learning for heterogeneous clients","authors":"Diao","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Ding et al., 2023] Ning Ding, Yujia Qin, et al. Parameter-efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence, 5(3):220-235, March 2023.","title":"Parameter-efficient fine-tuning of large-scale pre-trained language models","authors":"Ning Ding, Yujia Qin","year":"2023","venue":"Nature Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Dosovitskiy et al., 2020] Alexey Dosovitskiy, Lucas Beyer, et al. An image is worth 16x16 words: Transformers for image recog- nition at scale, 2020.","title":"An image is worth 16x16 words: Transformers for image recognition at scale","authors":"Alexey Dosovitskiy, Lucas Beyer","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Erben et al., 2023] Alexander Erben, Ruben Mayer, and Hans- Arno Jacobsen. How can we train deep learning models across clouds and continents? an experimental study, 2023.","title":"How can we train deep learning models across clouds and continents? an experimental study","authors":"Alexander Erben, Ruben Mayer, Hans-Arno Jacobsen","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Fan et al., 2023] Tao Fan, Yan Kang, et al. Fate-llm: A industrial grade federated learning framework for large language models, 2023.","title":"Fate-llm: A industrial grade federated learning framework for large language models","authors":"Tao Fan, Yan Kang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Foley et al., 2022] Patrick Foley, Micah J Sheller, et al. Openfl: the open federated learning library. Physics in Medicine & Biology, 67(21), 2022.","title":"Openfl: the open federated learning library","authors":"Patrick Foley, Micah J Sheller","year":"2022","venue":"Physics in Medicine & Biology","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Frankle and Carbin, 2019] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In ICLR, 2019.","title":"The lottery ticket hypothesis: Finding sparse, trainable neural networks","authors":"Carbin Frankle, Jonathan Frankle, Michael Carbin","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Galtier and Marini, 2019] Mathieu N Galtier and Camille Marini. Substra: a framework for privacy-preserving, traceable and col- laborative machine learning, 2019.","title":"Substra: a framework for privacy-preserving, traceable and collaborative machine learning","authors":"N Mathieu, Camille Galtier, Marini","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"Inc. Google. Tensorflow federated: Machine learn- ing on decentralized data. https://www.tensorflow.org/federated, 2019. Accessed: 2023-12-01.","title":"Tensorflow federated: Machine learning on decentralized data","authors":null,"year":"2019","venue":null,"doi":null,"url":"https://www.tensorflow.org/federated","internal_url":null},{"position":17,"raw":"[Habib et al., 2023] Gousia Habib, Tausifa Jan Saleem, and Brejesh Lall. Knowledge distillation in vision transformers: A critical review, 2023.","title":"Knowledge distillation in vision transformers: A critical review","authors":"Habib","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[He et al., 2020] Chaoyang He, Songze Li, et al. Fedml: A research library and benchmark for federated machine learning, 2020.","title":"Fedml: A research library and benchmark for federated machine learning","authors":"He","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[He et al., 2021] Haoyu He, Xingjian Shi, et al. Distiller: A sys- tematic study of model distillation methods in natural language processing, 2021.","title":"Distiller: A systematic study of model distillation methods in natural language processing","authors":"Haoyu He, Xingjian Shi","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Horváth et al., 2021] Samuel Horváth, Stefanos Laskaridis, et al. FjORD: Fair and accurate federated learning under heteroge- neous targets with ordered dropout. In Advances in Neural In- formation Processing Systems, 2021.","title":"FjORD: Fair and accurate federated learning under heterogeneous targets with ordered dropout","authors":"Horváth","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Houlsby et al., 2019] Neil Houlsby, Andrei Giurgiu, et al. Parameter-efficient transfer learning for nlp, 2019.","title":"Parameter-efficient transfer learning for nlp","authors":"Neil Houlsby, Andrei Giurgiu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Hu et al., 2021] Edward J. Hu, Yelong Shen, et al. Lora: Low-rank adaptation of large language models, 2021.","title":"Low-rank adaptation of large language models","authors":"Edward J Hu, Yelong Shen","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Huang et al., 2022] Hong Huang, Lan Zhang, et al. Distributed pruning towards tiny neural networks in federated learning, 2022.","title":"Distributed pruning towards tiny neural networks in federated learning","authors":"Huang","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Isik et al., 2022] Berivan Isik, Francesco Pase, et al. Sparse ran- dom networks for communication-efficient federated learning, 2022.","title":"Sparse random networks for communication-efficient federated learning","authors":"Isik","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Jia et al., 2022] Menglin Jia, Luming Tang, et al. Visual prompt tuning, 2022.","title":"Visual prompt tuning","authors":"Jia","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Jiang et al., 2022] Yuang Jiang, Shiqiang Wang, et al. Model prun- ing enables efficient federated learning on edge devices. IEEE Transactions on Neural Networks and Learning Systems, 2022.","title":"Model pruning enables efficient federated learning on edge devices","authors":"Yuang Jiang, Shiqiang Wang","year":"2022","venue":"IEEE Transactions on Neural Networks and Learning Systems","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Kuang et al., 2023] Weirui Kuang, Bingchen Qian, et al. Federatedscope-llm: A comprehensive package for fine-tuning large language models in federated learning, 2023.","title":"Federatedscope-llm: A comprehensive package for fine-tuning large language models in federated learning","authors":"Kuang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Lagunas et al., 2021] Francois Lagunas, Ella Charlaix, et al. Block pruning for faster transformers. In Proceedings of the 2021 Con- ference on EMNLP, Online and Punta Cana, Dominican Repub- lic, November 2021. ACL.","title":"Block pruning for faster transformers","authors":"Francois Lagunas, Ella Charlaix","year":"2021","venue":"Proceedings of the 2021 Conference on EMNLP","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Lester et al., 2021] Brian Lester, Rami Al-Rfou, and Noah Con- stant. The power of scale for parameter-efficient prompt tuning, 2021.","title":"The power of scale for parameter-efficient prompt tuning","authors":"Brian Lester, Rami Al-Rfou, Noah Constant","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Li et al., 2020] Ang Li, Jingwei Sun, et al. Lotteryfl: Personal- ized and communication-efficient federated learning with lottery ticket hypothesis on non-iid datasets, 2020.","title":"Lotteryfl: Personalized and communication-efficient federated learning with lottery ticket hypothesis on non-iid datasets","authors":"Ang Li, Jingwei Sun","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Li et al., 2022] Zhize Li, Haoyu Zhao, et al. Soteriafl: A uni- fied framework for private federated learning with communica- tion compression. In Advances in Neural Information Processing Systems, volume 35, 2022.","title":"Soteriafl: A unified framework for private federated learning with communication compression","authors":"Li","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Li et al., 2023] Qinbin Li, Zeyi Wen, Zhaomin Wu, Sixu Hu, Naibo Wang, Yuan Li, Xu Liu, and Bingsheng He. A survey on federated learning systems: Vision, hype and reality for data privacy and protection. IEEE Transactions on Knowledge and Data Engineering, 35(4):3347-3366, April 2023.","title":"A survey on federated learning systems: Vision, hype and reality for data privacy and protection","authors":"Qinbin Li, Zeyi Wen, Zhaomin Wu, Sixu Hu, Naibo Wang, Yuan Li, Xu Liu, Bingsheng He","year":"2023","venue":"IEEE Transactions on Knowledge and Data Engineering","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Liu et al., 2021] Yang Liu, Tao Fan, et al. Fate: An industrial grade platform for collaborative learning with data protection. JMLR, 22(1), 2021.","title":"Fate: An industrial grade platform for collaborative learning with data protection","authors":"Yang Liu, Tao Fan","year":"2021","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Liu et al., 2022] Ji Liu, Jizhou Huang, et al. From distributed ma- chine learning to federated learning: a survey. Knowledge and Information Systems, 64(4), 2022.","title":"From distributed machine learning to federated learning: a survey","authors":"Ji Liu, Jizhou Huang","year":"2022","venue":"Knowledge and Information Systems","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Longpre et al., 2023] Shayne Longpre, Le Hou, et al. The flan col- lection: Designing data and methods for effective instruction tun- ing, 2023.","title":"The flan collection: Designing data and methods for effective instruction tuning","authors":"Shayne Longpre, Le Hou","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Lu et al., 2023] Wang Lu, Xixu Hu, et al. Fedclip: Fast general- ization and personalization for clip in federated learning, 2023.","title":"Fedclip: Fast generalization and personalization for clip in federated learning","authors":"Wang Lu, Xixu Hu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Ludwig et al., 2020] Heiko Ludwig, Nathalie Baracaldo, et al. Ibm federated learning: an enterprise framework white paper v0.1, 2020.","title":"Ibm federated learning: an enterprise framework white paper","authors":"Heiko Ludwig, Nathalie Baracaldo","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[McMahan et al., 2017] Brendan McMahan, Eider Moore, et al. Communication-Efficient Learning of Deep Networks from De- centralized Data. In Proceedings of the 20th International Con- ference on Artificial Intelligence and Statistics, volume 54 of Pro- ceedings of Machine Learning Research. PMLR, 2017.","title":"Communication-Efficient Learning of Deep Networks from Decentralized Data","authors":"Brendan Mcmahan, Eider Moore","year":"2017","venue":"Proceedings of the 20th International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Mishchenko et al., 2019] Konstantin Mishchenko, Eduard Gor- bunov, et al. Distributed learning with compressed gradient dif- ferences, 2019.","title":"Distributed learning with compressed gradient differences","authors":"Konstantin Mishchenko, Eduard Gorbunov","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Mitra et al., 2021] Aritra Mitra, Rayana Jaafar, George J. Pappas, and Hamed Hassani. Linear convergence in federated learning: Tackling client heterogeneity and sparse gradients. In Advances in Neural Information Processing Systems, 2021.","title":"Linear convergence in federated learning: Tackling client heterogeneity and sparse gradients","authors":"Aritra Mitra, Rayana Jaafar, George J Pappas, Hamed Hassani","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Nguyen et al., 2021] Dinh C. Nguyen, Ming Ding, et al. Federated learning for internet of things: A comprehensive survey. IEEE Communications Surveys & Tutorials, 23(3), 2021.","title":"Federated learning for internet of things: A comprehensive survey","authors":"C Dinh, Ming Nguyen, Ding","year":"2021","venue":"IEEE Communications Surveys & Tutorials","doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[OpenAI, 2023] OpenAI. GPT-4 technical report, 2023.","title":null,"authors":"Openai","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Penedo et al., 2023] Guilherme Penedo, Quentin Malartic, et al. The refinedweb dataset for falcon llm: Outperforming curated corpora with web data, and web data only, 2023.","title":"The refinedweb dataset for falcon llm: Outperforming curated corpora with web data, and web data only","authors":"Guilherme Penedo, Quentin Malartic","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":44,"raw":"[Peng et al., 2020] Xingchao Peng, Zijun Huang, et al. Federated adversarial domain adaptation. In ICLR, 2020.","title":"Federated adversarial domain adaptation","authors":"Peng","year":"2020","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":45,"raw":"[Perez et al., 2020] Andres Perez, Valentina Sanguineti, et al. Audio-visual model distillation using acoustic images. In Pro- ceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2020.","title":"Audio-visual model distillation using acoustic images","authors":"Andres Perez, Valentina Sanguineti","year":"2020","venue":"Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)","doi":null,"url":null,"internal_url":null},{"position":46,"raw":"[Qin et al., 2023] Zhen Qin, Daoyuan Chen, et al. Federated full- parameter tuning of billion-sized language models with commu- nication cost under 18 kilobytes, 2023.","title":"Federated fullparameter tuning of billion-sized language models with communication cost under 18 kilobytes","authors":"Zhen Qin, Daoyuan Chen","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":47,"raw":"[Radford et al., 2019] Alec Radford, Jeff Wu, et al. Language mod- els are unsupervised multitask learners. 2019.","title":"Language models are unsupervised multitask learners","authors":"Alec Radford, Jeff Wu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":48,"raw":"[Reisizadeh et al., 2020] Amirhossein Reisizadeh, Aryan Mokhtari, et al. Fedpaq: A communication-efficient feder- ated learning method with periodic averaging and quantization. In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108. PMLR, 2020.","title":"Amirhossein Reisizadeh, Aryan Mokhtari, et al. Fedpaq: A communication-efficient federated learning method with periodic averaging and quantization","authors":"Reisizadeh","year":"2020","venue":"Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":49,"raw":"[Roth et al., 2022] Holger R. Roth, Yan Cheng, et al. Nvidia flare: Federated learning from simulation to real-world. 2022.","title":"Nvidia flare: Federated learning from simulation to real-world","authors":"R Holger, Yan Roth, Cheng","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":50,"raw":"[Ryabinin et al., 2023] Max Ryabinin, Tim Dettmers, et al. Swarm parallelism: Training large models can be surprisingly communication-efficient, 2023.","title":"Swarm parallelism: Training large models can be surprisingly communication-efficient","authors":"Ryabinin","year":"2023","venue":"Max Ryabinin","doi":null,"url":null,"internal_url":null},{"position":51,"raw":"[Sanh et al., 2020] Victor Sanh, Thomas Wolf, and Alexander Rush. Movement pruning: Adaptive sparsity by fine-tuning. In Advances in Neural Information Processing Systems, volume 33.","title":"Movement pruning: Adaptive sparsity by fine-tuning","authors":"Victor Sanh, Thomas Wolf, Alexander Rush","year":null,"venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":52,"raw":"Curran Associates, Inc., 2020.","title":null,"authors":"Curran Associates, Inc","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":53,"raw":"[Sun et al., 2023] Guangyu Sun, Matias Mendieta, et al. Explor- ing parameter-efficient fine-tuning for improving communication efficiency in federated learning, 2023.","title":"Exploring parameter-efficient fine-tuning for improving communication efficiency in federated learning","authors":"Guangyu Sun, Matias Mendieta","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":54,"raw":"[Taori et al., 2023] Rohan Taori, Ishaan Gulrajani, et al. Stanford alpaca: An instruction-following llama model. https://github. com/tatsu-lab/stanford alpaca, 2023.","title":"Stanford alpaca: An instruction-following llama model","authors":"Rohan Taori, Ishaan Gulrajani","year":"2023","venue":null,"doi":null,"url":"https://github.com/tatsu-lab/stanfordalpaca","internal_url":null},{"position":55,"raw":"[Tian et al., 2022] Yuanyishu Tian, Yao Wan, et al. Fedbert: When federated learning meets pre-training. ACM Transactions on In- telligent Systems and Technology, 13(4), 2022.","title":"Fedbert: When federated learning meets pre-training","authors":"Tian","year":"2022","venue":"ACM Transactions on Intelligent Systems and Technology","doi":null,"url":null,"internal_url":null},{"position":56,"raw":"[Woisetschläger et al., 2023] Herbert Woisetschläger, Alexander Erben, et al. Federated fine-tuning of llms on the very edge: The good, the bad, the ugly, 2023.","title":"Federated fine-tuning of llms on the very edge: The good, the bad, the ugly","authors":"Woisetschläger","year":"2023","venue":"Herbert Woisetschläger","doi":null,"url":null,"internal_url":null},{"position":57,"raw":"[Xie et al., 2023] Yuexiang Xie, Zhen Wang, et al. Federatedscope: A flexible federated learning platform for heterogeneity. Pro- ceedings of the VLDB Endowment, 16(5), 2023.","title":"Federatedscope: A flexible federated learning platform for heterogeneity","authors":"Xie","year":"2023","venue":"Proceedings of the VLDB Endowment","doi":null,"url":null,"internal_url":null},{"position":58,"raw":"[Xu et al., 2023] Zheng Xu, Yanxiang Zhang, et al. Federated learning of gboard language models with differential privacy. 2023.","title":"Federated learning of gboard language models with differential privacy","authors":"Zheng Xu, Yanxiang Zhang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":59,"raw":"[Yang et al., 2023] Dongchao Yang, Jinchuan Tian, et al. Uniaudio: An audio foundation model toward universal audio generation, 2023.","title":"Uniaudio: An audio foundation model toward universal audio generation","authors":"Yang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":60,"raw":"[Yang et al., 2024] Yiyuan Yang, Guodong Long, Taoshu Shen, Jing Jiang, and Michael Blumenstein. Dual-personalizing adapter for federated foundation models. ArXiv, abs/2403.19211, 2024.","title":"Dual-personalizing adapter for federated foundation models","authors":"Yiyuan Yang, Guodong Long, Taoshu Shen, Jing Jiang, Michael Blumenstein","year":"2024","venue":null,"doi":null,"url":null,"internal_url":null},{"position":61,"raw":"[Yousefpour et al., 2023] Ashkan Yousefpour, Shen Guo, et al. Green federated learning, 2023.","title":"Green federated learning","authors":"Ashkan Yousefpour, Shen Guo","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":62,"raw":"[Yu et al., 2023] Sixing Yu, J. Pablo Muñoz, and Ali Jannesari. Federated foundation models: Privacy-preserving and collabo- rative learning for large models, 2023.","title":"Federated foundation models: Privacy-preserving and collaborative learning for large models","authors":"Sixing Yu, J Pablo Muñoz, Ali Jannesari","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":63,"raw":"[Zaken et al., 2021] Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models, 2021.","title":"Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models","authors":"Elad Ben Zaken, Shauli Ravfogel, Yoav Goldberg","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":64,"raw":"[Zhang et al., 2021] Chen Zhang, Yu Xie, et al. A survey on feder- ated learning. Knowledge-Based Systems, 216, 2021.","title":"A survey on federated learning","authors":"Chen Zhang, Yu Xie","year":"2021","venue":"Knowledge-Based Systems","doi":null,"url":null,"internal_url":null},{"position":65,"raw":"[Zhang et al., 2023a] Jianyi Zhang, Saeed Vahidian, et al. Towards building the federated gpt: Federated instruction tuning, 2023.","title":"Towards building the federated gpt: Federated instruction tuning","authors":"Zhang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":66,"raw":"[Zhang et al., 2023b] Zhuo Zhang, Yuanhang Yang, et al. Fed- PETuning: When federated learning meets the parameter- efficient tuning methods of pre-trained language models. In Findings of the Association for Computational Linguistics: ACL 2023. ACL, 2023.","title":"Fed-PETuning: When federated learning meets the parameterefficient tuning methods of pre-trained language models","authors":"Zhang","year":"2023","venue":"Findings of the Association for Computational Linguistics: ACL 2023. ACL","doi":null,"url":null,"internal_url":null},{"position":67,"raw":"[Zhao et al., 2022] Haodong Zhao, Wei Du, et al. Fedprompt: Communication-efficient and privacy preserving prompt tuning in federated learning, 2022.","title":"Fedprompt: Communication-efficient and privacy preserving prompt tuning in federated learning","authors":"Zhao","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":68,"raw":"[Zheng et al., 2023] Rui Zheng, Shihan Dou, et al. Secrets of rlhf in large language models part i: Ppo, 2023.","title":"Secrets of rlhf in large language models part i: Ppo","authors":"Rui Zheng, Shihan Dou","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":69,"raw":"[Zhou et al., 2018] Yiren Zhou, Seyed Moosavi-Dezfooli, et al. Adaptive quantization for deep neural network. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), 2018.","title":"Adaptive quantization for deep neural network","authors":"Yiren Zhou, Seyed Moosavi-Dezfooli","year":"2018","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":70,"raw":"[Zhu and Gupta, 2017] Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy of pruning for model com- pression, 2017.","title":"To prune, or not to prune: exploring the efficacy of pruning for model compression","authors":"Michael Zhu, Suyog Gupta","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":71,"raw":"[Zhu et al., 2019] Maohua Zhu, Tao Zhang, et al. Sparse tensor core: Algorithm and hardware co-design for vector-wise sparse neural networks on modern gpus. In Proceedings of the 52nd An- nual IEEE/ACM International Symposium on Microarchitecture, MICRO '52. ACM, 2019.","title":"Sparse tensor core: Algorithm and hardware co-design for vector-wise sparse neural networks on modern gpus","authors":"Zhu","year":"2019","venue":"Proceedings of the 52nd Annual IEEE/ACM International Symposium on Microarchitecture, MICRO '52","doi":null,"url":null,"internal_url":null},{"position":72,"raw":"[Zhuang et al., 2023] Weiming Zhuang, Chen Chen, and Lingjuan Lyu. When foundation model meets federated learning: Motiva- tions, challenges, and future directions, 2023.","title":"When foundation model meets federated learning: Motivations, challenges, and future directions","authors":"Weiming Zhuang, Chen Chen, Lingjuan Lyu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":73,"raw":"[Ziller et al., 2021] Alexander Ziller, Andrew Trask, et al. PySyft: A Library for Easy Federated Learning. Springer International Publishing, 2021.","title":"PySyft: A Library for Easy Federated Learning","authors":"Alexander Ziller, Andrew Trask","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}