{"type":"ScholarlyArticle","title":"FedBFPT: An Efficient Federated Learning Framework for Bert Further Pre-training","authors":[{"name":"Xin'ao Wang","slug":"xin-ao-wang","given":"Xin'ao","family":"Wang","affiliation":"Key Lab of Intelligent Computing Based Big Data of Zhejiang Province"},{"name":"Huan Li","slug":"huan-li","given":"Huan","family":"Li","affiliation":"Key Lab of Intelligent Computing Based Big Data of Zhejiang Province"},{"name":"Ke Chen","slug":"ke-chen","given":"Ke","family":"Chen","affiliation":"Zhejiang University , Hangzhou , China"},{"name":"Lidan Shou","slug":"lidan-shou","given":"Lidan","family":"Shou","affiliation":"Zhejiang University , Hangzhou , China"}],"abstract":"This study proposes FEDBFPT (Federated BERT Further Pre-Training), a Federated Learning (FL) framework for further pre-training the BERT language model in specialized domains while addressing privacy concerns. FEDBFPT enables multiple clients to collaboratively train the shallower layers of BERT, which are crucial in the pre-training stage, without the need to share private data. To achieve this, FEDBFPT involves building a local model for each client, progressively training the shallower layers of local models while sampling deeper layers, and aggregating trained parameters on a server to create the final global model. This approach utilizes multiple smaller local models to further pre-train a global model targeted at specific tasks via fine-tuning, resulting in a reduction in resource usage while maintaining model accuracy. Theoretical analysis is conducted to support the efficiency of FEDBFPT, and experiments are conducted on corpora across domains such as medicine, biology, and computer science. Results indicate that FEDBFPT achieves performance levels comparable to traditional FL methods while reducing computation and communication costs by 46.70% and 7.04%, respectively, even approaching the performance of centralized training models. The Source code is released at https://github.com/Hanzhouu/FedBFPT.","doi":"10.24963/ijcai.2023/483","url":"https://ijcai.org/proceedings/2023/483","pdf":"https://ijcai.org/proceedings/2023/0483.pdf","markdown":"https://ijcai.org/proceedings/2023/483.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/483","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Machine Learning","track":"Main Track","number":483,"pages":{"first":4344,"last":4352},"keywords":["Machine Learning: ML: Federated learning","Natural Language Processing: NLP: Applications"],"videos":[],"references":[{"position":1,"raw":"[Alistarh et al., 2017] Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, and Milan Vojnovic. QSGD: Communication-efficient SGD via gradient quantization and encoding. NeurIPS, 30, 2017.","title":"QSGD: Communication-efficient SGD via gradient quantization and encoding","authors":"Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, Milan Vojnovic","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Beltagy et al., 2019] Iz Beltagy, Kyle Lo, and Arman Co- han. Scibert: A pretrained language model for scientific text. In EMNLP-IJCNLP, pages 3615-3620, 2019.","title":"Scibert: A pretrained language model for scientific text","authors":"Beltagy","year":"2019","venue":"EMNLP-IJCNLP","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Collier and Kim, 2004] Nigel Collier and Jin-Dong Kim. Introduction to the bio-entity recognition task at JNLPBA. In NLPBA/BioNLP, pages 73-78, 2004.","title":"Introduction to the bio-entity recognition task at JNLPBA","authors":"Kim ; Collier, Nigel Collier, Jin-Dong Kim","year":"2004","venue":"NLPBA/BioNLP","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Devlin et al., 2019] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understand- ing. In NAACL-HLT, pages 4171-4186, 2019.","title":"BERT: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2019","venue":"NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Fu et al., 2020] Fangcheng Fu, Yuzheng Hu, Yihan He, Ji- awei Jiang, Yingxia Shao, Ce Zhang, and Bin Cui. Don't waste your bits! squeeze activations and gradients for deep neural networks via tinyscript. In ICML, pages 3304- 3314, 2020.","title":"Don't waste your bits! squeeze activations and gradients for deep neural networks via tinyscript","authors":"Fangcheng Fu, Yuzheng Hu, Yihan He, Jiawei Jiang, Yingxia Shao, Ce Zhang, Bin Cui","year":"2020","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hao et al., 2019] Yaru Hao, Li Dong, Furu Wei, and Ke Xu. Visualizing and understanding the effectiveness of bert. In EMNLP-IJCNLP, pages 4143-4152, 2019.","title":"Visualizing and understanding the effectiveness of bert","authors":"Yaru Hao, Li Dong, Furu Wei, Ke Xu","year":"2019","venue":"EMNLP-IJCNLP","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hinton et al., 2015] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.","title":"Distilling the knowledge in a neural network","authors":"Geoffrey Hinton, Oriol Vinyals, Jeff Dean","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Jawahar et al., 2019] Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. What does bert learn about the structure of language? In ACL, 2019.","title":"What does bert learn about the structure of language? In ACL","authors":"Ganesh Jawahar, Benoît Sagot, Djamé Seddah","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jiao et al., 2020] Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling BERT for natural language un- derstanding. In EMNLP, 2020.","title":"Tinybert: Distilling BERT for natural language understanding","authors":"Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, Qun Liu","year":"2020","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Karras et al., 2018] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. In ICLR, 2018.","title":"Progressive growing of gans for improved quality, stability, and variation","authors":"Tero Karras, Timo Aila, Samuli Laine, Jaakko Lehtinen","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Konečnỳ et al., 2016] Jakub Konečnỳ, H Brendan McMa- han, X Yu Felix, Peter Richtarik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:1610.05492, 2016.","title":"Federated learning: Strategies for improving communication efficiency","authors":"Jakub Konečnỳ, Brendan Mcmahan, Yu Felix, Peter Richtarik, Ananda Theertha Suresh, Dave Bacon","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lee et al., 2020] Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. BioBERT: A pre-trained biomedical lan- guage representation model for biomedical text mining. Bioinformatics, 36(4):1234-1240, 2020.","title":"BioBERT: A pre-trained biomedical language representation model for biomedical text mining","authors":"Lee","year":"2020","venue":"Bioinformatics","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li and Wang, 2019] Daliang Li and Junpu Wang. Fedmd: Heterogenous federated learning via model distillation. arXiv preprint arXiv:1910.03581, 2019.","title":"Fedmd: Heterogenous federated learning via model distillation","authors":"Wang ; Li, Daliang Li, Junpu Wang","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li et al., 2020a] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. ML- Sys, 2:429-450, 2020.","title":"Federated optimization in heterogeneous networks","authors":"Li","year":"2020","venue":"ML-Sys","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2020b] Zhiyuan Li, Jaideep Vitthal Murkute, Prashnna Kumar Gyawali, and Linwei Wang. Progressive learning and disentanglement of hierarchical representa- tions. In ICLR, 2020.","title":"Progressive learning and disentanglement of hierarchical representations","authors":"Li","year":"2020","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lin et al., 2018] Yujun Lin, Song Han, Huizi Mao, Yu Wang, and Bill Dally. Deep gradient compression: Re- ducing the communication bandwidth for distributed train- ing. In ICLR, 2018.","title":"Deep gradient compression: Reducing the communication bandwidth for distributed training","authors":"Yujun Lin, Song Han, Huizi Mao, Yu Wang, Bill Dally","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lin et al., 2020] Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. NeurIPS, 33:2351-2363, 2020.","title":"Ensemble distillation for robust model fusion in federated learning","authors":"Tao Lin, Lingjing Kong, Sebastian U Stich, Martin Jaggi","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Liu et al., 2019] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692, 2019.","title":"Roberta: A robustly optimized BERT pretraining approach","authors":"Liu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Lo et al., 2020] Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel Weld. S2ORC: The semantic scholar open research corpus. In ACL, pages 4969-4983, 2020.","title":"S2ORC: The semantic scholar open research corpus","authors":"Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, Daniel Weld","year":"2020","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Luan et al., 2018] Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. Multi-task identification of en- tities, relations, and coreference for scientific knowledge graph construction. In EMNLP, pages 3219-3232, 2018.","title":"Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction","authors":"Yi Luan, Luheng He, Mari Ostendorf, Hannaneh Hajishirzi","year":"2018","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Manginas et al., 2020] Nikolaos Manginas, Ilias Chalkidis, and Prodromos Malakasiotis. Layer-wise guided training for BERT: Learning incrementally refined document rep- resentations. arXiv preprint arXiv:2010.05763, 2020.","title":"Layer-wise guided training for BERT: Learning incrementally refined document representations","authors":"Nikolaos Manginas, Ilias Chalkidis, Prodromos Malakasiotis","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[McMahan et al., 2017] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Ar- cas. Communication-efficient learning of deep networks from decentralized data. In AISTATS, pages 1273-1282, 2017.","title":"Communication-efficient learning of deep networks from decentralized data","authors":"Brendan Mcmahan, Eider Moore, Daniel Ramage, Seth Hampson, Blaise Aguera Y Arcas","year":"2017","venue":"AISTATS","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Paszke et al., 2019] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. NeurIPS, 32, 2019.","title":"Pytorch: An imperative style, highperformance deep learning library","authors":"Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Peters et al., 2018] Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In ACL, 2018.","title":"Deep contextualized word representations","authors":"Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, Luke Zettlemoyer","year":"2018","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Radford et al., 2018] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language un- derstanding by generative pre-training. Technical report, OpenAI, 2018.","title":"Improving language understanding by generative pre-training","authors":"Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Raghu et al., 2017] Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. Svcca: Singular vec- tor canonical correlation analysis for deep learning dynam- ics and interpretability. NeurIPS, 30, 2017.","title":"Svcca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability","authors":"Raghu","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Reddi et al., 2020] Sashank Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečnỳ, Sanjiv Kumar, and H Brendan McMa- han. Adaptive federated optimization. arXiv preprint arXiv:2003.00295, 2020.","title":"Adaptive federated optimization","authors":"Sashank Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečnỳ, Sanjiv Kumar, Brendan Mcmahan","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Sanh et al., 2019] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled ver- sion of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108, 2019.","title":"Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter","authors":"Victor Sanh, Lysandre Debut, Julien Chaumond, Thomas Wolf","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Stich et al., 2018] Sebastian U Stich, Jean-Baptiste Cordon- nier, and Martin Jaggi. Sparsified SGD with memory. NeurIPS, 31, 2018.","title":"Sparsified SGD with memory","authors":"Jean-Baptiste Sebastian U Stich, Martin Cordonnier, Jaggi","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurIPS, 30, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Wang et al., 2018] Yifan Wang, Federico Perazzi, Brian McWilliams, Alexander Sorkine-Hornung, Olga Sorkine- Hornung, and Christopher Schroers. A fully progres- sive approach to single-image super-resolution. In CVPR Workshops, pages 864-873, 2018.","title":"A fully progressive approach to single-image super-resolution","authors":"Yifan Wang, Federico Perazzi, Brian Mcwilliams, Alexander Sorkine-Hornung, Olga Sorkine-Hornung, Christopher Schroers","year":"2018","venue":"CVPR Workshops","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Wang et al., 2022a] Hui-Po Wang, Sebastian Stich, Yang He, and Mario Fritz. ProgFed: Effective, communication, and computation efficient federated learning by progres- sive training. In ICML, pages 23034-23054, 2022.","title":"ProgFed: Effective, communication, and computation efficient federated learning by progressive training","authors":"Wang","year":"2022","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Wang et al., 2022b] Jue Wang, Ke Chen, Gang Chen, Lidan Shou, and Julian McAuley. SkipBERT: Efficient inference with shallow layer skipping. In ACL, pages 7287-7301, 2022.","title":"SkipBERT: Efficient inference with shallow layer skipping","authors":"Wang","year":"2022","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Wang et al., 2023] Xin'ao Wang, Huan Li, Ke Chen, and Lidan Shou. Code, datasets, and supplementary materials. https://github.com/Hanzhouu/FedBFPT, 2023. Accessed: 2023-05-23.","title":"Code, datasets, and supplementary materials","authors":"Wang","year":"2023","venue":null,"doi":null,"url":"https://github.com/Hanzhouu/FedBFPT","internal_url":null},{"position":35,"raw":"[Wu et al., 2020] Rongliang Wu, Gongjie Zhang, Shijian Lu, and Tao Chen. Cascade EF-GAN: Progressive facial ex- pression editing with local focuses. In CVPR, pages 5021- 5030, 2020.","title":"Cascade EF-GAN: Progressive facial expression editing with local focuses","authors":"Rongliang Wu, Gongjie Zhang, Shijian Lu, Tao Chen","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}