{"type":"ScholarlyArticle","title":"FedOBD: Opportunistic Block Dropout for Efficiently Training Large-scale Neural Networks through Federated Learning","authors":[{"name":"Yuanyuan Chen","slug":"yuanyuan-chen","given":"Yuanyuan","family":"Chen","affiliation":"School of Computer Science and Engineering , Nanyang Technological University , Singapore"},{"name":"Zichen Chen","slug":"zichen-chen","given":"Zichen","family":"Chen","affiliation":"University of California , Santa Barbara , CA , USA"},{"name":"Pengcheng Wu","slug":"pengcheng-wu","given":"Pengcheng","family":"Wu","affiliation":"School of Computer Science and Engineering , Nanyang Technological University , Singapore"},{"name":"Han Yu","slug":"han-yu","given":"Han","family":"Yu","affiliation":"School of Computer Science and Engineering , Nanyang Technological University , Singapore"}],"abstract":"Large-scale neural networks possess considerable expressive power. They are well-suited for complex learning tasks in industrial applications. However, large-scale models pose significant challenges for training under the current Federated Learning (FL) paradigm. Existing approaches for efficient FL training often leverage model parameter dropout. However, manipulating individual model parameters is not only inefficient in meaningfully reducing the communication overhead when training  large-scale FL models, but may also be detrimental to the scaling efforts and model performance as shown by recent research. To address these issues, we propose the Federated Opportunistic Block Dropout (FedOBD) approach. The key novelty is that it decomposes large-scale models into semantic blocks so that FL participants can opportunistically upload quantized blocks, which are deemed to be significant towards training the model, to the FL server for aggregation. Extensive experiments evaluating FedOBD against four state-of-the-art approaches based on multiple real-world datasets show that it reduces the overall communication overhead by more than 88% compared to the best performing baseline approach, while achieving the highest test accuracy. To the best of our knowledge, FedOBD is the first approach to perform dropout on FL models at the block level rather than at the individual parameter level.","doi":"10.24963/ijcai.2023/394","url":"https://ijcai.org/proceedings/2023/394","pdf":"https://ijcai.org/proceedings/2023/0394.pdf","markdown":"https://ijcai.org/proceedings/2023/394.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/394","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Machine Learning","track":"Main Track","number":394,"pages":{"first":3541,"last":3549},"keywords":["Machine Learning: ML: Learning sparse models","Machine Learning: ML: Federated learning","Machine Learning: ML: Optimization"],"videos":[],"references":[{"position":1,"raw":"[Alistarh et al., 2017] Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, and Milan Vojnovic. QSGD: Communication-efficient SGD via gradient quantization and encoding. NeurIPS, 30:1709-1720, 2017.","title":"QSGD: Communication-efficient SGD via gradient quantization and encoding","authors":"Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, Milan Vojnovic","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Amiri et al., 2020] Mohammad Mohammadi Amiri, Deniz Gunduz, Sanjeev R Kulkarni, and H Vincent Poor. Feder- ated learning with quantized global model updates. arXiv preprint arXiv:2006.10672, 2020.","title":"Federated learning with quantized global model updates","authors":"Mohammad Mohammadi Amiri, Deniz Gunduz, H Sanjeev R Kulkarni, Vincent Poor","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bernstein et al., 2018] Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, and Animashree Anandkumar. SignSGD: Compressed optimisation for non-convex prob- lems. In ICML, pages 560-569, 2018.","title":"SignSGD: Compressed optimisation for non-convex problems","authors":"Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, Animashree Anandkumar","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bouacida et al., 2021] Nader Bouacida, Jiahui Hou, Hui Zang, and Xin Liu. Adaptive federated dropout: Improv- ing communication efficiency and generalization for feder- ated learning. In INFOCOM Workshops, pages 1-6, 2021.","title":"Adaptive federated dropout: Improving communication efficiency and generalization for federated learning","authors":"Nader Bouacida, Jiahui Hou, Hui Zang, Xin Liu","year":"2021","venue":"INFOCOM Workshops","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cheng et al., 2022] Gary Cheng, Zachary Charles, Zachary Garrett, and Keith Rushg. Does federated dropout actually work? In CVPR Workshops, pages 3387-3395, 2022.","title":"Does federated dropout actually work?","authors":"Gary Cheng, Zachary Charles, Zachary Garrett, Keith Rushg","year":"2022","venue":"CVPR Workshops","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Elsken et al., 2019] Thomas Elsken, Jan Hendrik Metzen, Frank Hutter, et al. Neural architecture search: A sur- vey. Journal of Machine Learning Research, 20(55):1-21, 2019.","title":"Neural architecture search: A survey","authors":"Thomas Elsken, Jan Hendrik Metzen, Frank Hutter","year":"2019","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gunesli et al., 2021] Gozde N. Gunesli, Mohsin Bilal, Shan E Ahmed Raza, and Nasir M. Rajpoot. FedDropoutAvg: Generalizable federated learning for histopathology image classification. arXiv preprint arXiv:2111.13230, 2021.","title":"FedDropoutAvg: Generalizable federated learning for histopathology image classification","authors":"N Gozde, Mohsin Gunesli, Shan E Bilal, Ahmed Raza, Nasir M Rajpoot","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Horváth et al., 2021] Samuel Horváth, Stefanos Laskaridis, Mario Almeida, Ilias Leontiadis, Stylianos Venieris, and Nicholas Lane. FjORD: Fair and accurate federated learn- ing under heterogeneous targets with ordered dropout. In NeurIPS, 2021.","title":"FjORD: Fair and accurate federated learning under heterogeneous targets with ordered dropout","authors":"Horváth","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hou et al., 2018] Lu Hou, Ruiliang Zhang, and James T Kwok. Analysis of quantized models. In ICLR, 2018.","title":"Analysis of quantized models","authors":"Lu Hou, Ruiliang Zhang, James T Kwok","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Huang et al., 2017] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely con- nected convolutional networks. In CVPR, pages 4700- 4708, 2017.","title":"Densely connected convolutional networks","authors":"Gao Huang, Zhuang Liu, Laurens Van Der Maaten, Kilian Q Weinberger","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kairouz et al., 2021] Peter Kairouz, H. Brendan McMahan, et al. Advances and open problems in federated learning. Foundations and Trends in Machine Learning, 14(1-2):1- 210, 2021.","title":"Advances and open problems in federated learning","authors":"H Peter Kairouz, Brendan Mcmahan","year":"2021","venue":"Foundations and Trends in Machine Learning","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Konečnỳ et al., 2016] Jakub Konečnỳ, H Brendan McMa- han, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:1610.05492, 2016.","title":"Federated learning: Strategies for improving communication efficiency","authors":"Jakub Konečnỳ, Brendan Mcmahan, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, Dave Bacon","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Krizhevsky and Hinton, 2009] Alex Krizhevsky and Geof- frey Hinton. Learning multiple layers of features from tiny images. Technical Report 0, University of Toronto, Toronto, Ontario, 2009.","title":"Learning multiple layers of features from tiny images","authors":"Hinton Krizhevsky","year":"2009","venue":"Alex Krizhevsky and Geoffrey Hinton","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liao et al., 2021] Dongping Liao, Xitong Gao, Yiren Zhao, Hao Dai, Li Li, Kafeng Wang, Kejiang Ye, Yang Wang, and Cheng-Zhong Xu. Fed- Drop: Trajectory-weighted dropout for efficient feder- ated learning. https://paperswithcode.com/paper/feddrop- trajectory-weighted-dropout-for, 2021.","title":"Fed-Drop: Trajectory-weighted dropout for efficient federated learning","authors":"Dongping Liao, Xitong Gao, Yiren Zhao, Hao Dai, Li Li, Kafeng Wang, Kejiang Ye, Yang Wang, Cheng-Zhong Xu","year":null,"venue":null,"doi":null,"url":"https://paperswithcode.com/paper/feddrop-trajectory-weighted-dropout-for,2021","internal_url":null},{"position":16,"raw":"[Lin et al., 2017] Yujun Lin, Song Han, Huizi Mao, Yu Wang, and William J Dally. Deep gradient compres- sion: Reducing the communication bandwidth for dis- tributed training. arXiv preprint arXiv:1712.01887, 2017.","title":"Deep gradient compression: Reducing the communication bandwidth for distributed training","authors":"Yujun Lin, Song Han, Huizi Mao, Yu Wang, William J Dally","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liu et al., 2020] Yang Liu, Anbu Huang, Yun Luo, He Huang, Youzhi Liu, Yuanyuan Chen, Lican Feng, Tian- jian Chen, Han Yu, and Qiang Yang. FedVision: An on- line visual object detection platform powered by federated learning. In IAAI, pages 13172-13179, 2020.","title":"FedVision: An online visual object detection platform powered by federated learning","authors":"Yang Liu, Anbu Huang, Yun Luo, He Huang, Youzhi Liu, Yuanyuan Chen, Lican Feng, Tianjian Chen, Han Yu, Qiang Yang","year":"2020","venue":"IAAI","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Liu et al., 2022] Zelei Liu, Yuanyuan Chen, Yansong Zhao, Han Yu, Yang Liu, Renyi Bao, Jinpeng Jiang, Zaiqing Nie, Qian Xu, and Qiang Yang. Contribution-aware feder- ated learning for smart healthcare. In IAAI, pages 12396- 12404, 2022.","title":"Contribution-aware federated learning for smart healthcare","authors":"Zelei Liu, Yuanyuan Chen, Yansong Zhao, Han Yu, Yang Liu, Renyi Bao, Jinpeng Jiang, Zaiqing Nie, Qian Xu, Qiang Yang","year":"2022","venue":"IAAI","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Loshchilov and Hutter, 2016] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983, 2016.","title":"Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts","authors":"Hutter Loshchilov","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Maas et al., 2011] Andrew L. Maas, Raymond E. Daly, Pe- ter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In ACL, pages 142-150, 2011.","title":"Learning word vectors for sentiment analysis","authors":"Andrew L Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, Christopher Potts","year":"2011","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[McMahan et al., 2017] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Ar- cas. Communication-Efficient Learning of Deep Networks from Decentralized Data. In AISTATS, pages 1273-1282, 2017.","title":"Communication-Efficient Learning of Deep Networks from Decentralized Data","authors":"Brendan Mcmahan, Eider Moore, Daniel Ramage, Seth Hampson, Blaise Aguera Y Arcas","year":"2017","venue":"AISTATS","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Paszke et al., 2017] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary De- Vito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch. In NIPS-W, 2017.","title":"Automatic differentiation in PyTorch","authors":"Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary De-Vito, Zeming Lin, Alban Desmaison, Luca Antiga, Adam Lerer","year":"2017","venue":"NIPS-W","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Pennington et al., 2014] Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In EMNLP, pages 1532-1543, 2014.","title":"Glove: Global vectors for word representation","authors":"Jeffrey Pennington, Richard Socher, Christopher D Manning","year":"2014","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Reisizadeh et al., 2020] Amirhossein Reisizadeh, Aryan Mokhtari, Hamed Hassani, Ali Jadbabaie, and Ramtin Pedarsani. FedPAQ: A communication-efficient federated learning method with periodic averaging and quantization. In AISTATS, pages 2021-2031, 2020.","title":"FedPAQ: A communication-efficient federated learning method with periodic averaging and quantization","authors":"Reisizadeh","year":"2020","venue":"AISTATS","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Rothchild et al., 2020] Daniel Rothchild, Ashwinee Panda, Enayat Ullah, Nikita Ivkin, Ion Stoica, Vladimir Braver- man, Joseph Gonzalez, and Raman Arora. FetchSGD: Communication-efficient federated learning with sketch- ing. In ICML, pages 8253-8265, 2020.","title":"FetchSGD: Communication-efficient federated learning with sketching","authors":"Daniel Rothchild, Ashwinee Panda, Enayat Ullah, Nikita Ivkin, Ion Stoica, Vladimir Braverman, Joseph Gonzalez, Raman Arora","year":"2020","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wen et al., 2021] Dingzhu Wen, Ki-Jun Jeon, and Kaibin Huang. Federated dropout -a simple approach for en- abling federated learning on resource constrained devices. arXiv preprint arXiv:2109.15258, 2021.","title":"Federated dropout -a simple approach for enabling federated learning on resource constrained devices","authors":"Wen","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Xie et al., 2022] Yuan-Ai Xie, Jiawen Kang, Dusit Niyato, Nguyen Thi Thanh Van, Nguyen Cong Luong, Zhixin Liu, and Han Yu. Securing federated learning: A covert communication-based approach. IEEE Network, 37(1):118-124, 2022.","title":"Securing federated learning: A covert communication-based approach","authors":"Yuan-Ai Xie, Jiawen Kang, Dusit Niyato, Nguyen Thi, Thanh Van, Cong Nguyen, Zhixin Luong, Han Liu, Yu","year":"2022","venue":"IEEE Network","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Yang et al., 2018] Timothy Yang, Galen Andrew, Hubert Eichner, Haicheng Sun, Wei Li, Nicholas Kong, Daniel Ramage, and Franc ¸oise Beaufays. Applied federated learning: Improving google keyboard query suggestions. arXiv preprint arXiv:1812.02903, 2018.","title":"Applied federated learning: Improving google keyboard query suggestions","authors":"Timothy Yang, Galen Andrew, Hubert Eichner, Haicheng Sun, Wei Li, Nicholas Kong, Daniel Ramage, Franc, Beaufays","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Yang et al., 2019] Qiang Yang, Yang Liu, Tianjian Chen, and Yongxin Tong. Federated machine learning: Concept and applications. ACM Transactions on Intelligent Systems and Technology, 10(2):1-19, 2019.","title":"Federated machine learning: Concept and applications","authors":"Qiang Yang, Yang Liu, Tianjian Chen, Yongxin Tong","year":"2019","venue":"ACM Transactions on Intelligent Systems and Technology","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Yu et al., 2017] Han Yu, Chunyan Miao, Yiqiang Chen, Si- mon Fauvel, Xiaoming Li, and Victor R. Lesser. Algorith- mic management for improving collective productivity in crowdsourcing. Scientific Reports, 7(12541), 2017.","title":"Algorithmic management for improving collective productivity in crowdsourcing","authors":"Han Yu, Chunyan Miao, Yiqiang Chen, Simon Fauvel, Xiaoming Li, Victor R Lesser","year":"2017","venue":"Scientific Reports","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}