{"type":"ScholarlyArticle","title":"Heterogeneous Ensemble Knowledge Transfer for Training Large Models in Federated Learning","authors":[{"name":"Yae Jee Cho","slug":"yae-jee-cho","given":"Yae Jee","family":"Cho","affiliation":"Carnegie Mellon University"},{"name":"Andre Manoel","slug":"andre-manoel","given":"Andre","family":"Manoel","affiliation":"Microsoft Research"},{"name":"Gauri Joshi","slug":"gauri-joshi","given":"Gauri","family":"Joshi","affiliation":"Carnegie Mellon University"},{"name":"Robert Sim","slug":"robert-sim","given":"Robert","family":"Sim","affiliation":"Microsoft Research"},{"name":"Dimitrios Dimitriadis","slug":"dimitrios-dimitriadis","given":"Dimitrios","family":"Dimitriadis","affiliation":"Microsoft Research"}],"abstract":"Federated learning (FL) enables edge-devices to collaboratively learn a model without disclosing their private data to a central aggregating server. Most existing FL algorithms require models of identical architecture to be deployed across the clients and server, making it infeasible to train large models due to clients' limited system resources. In this work, we propose a novel ensemble knowledge transfer method named Fed-ET in which small models (different in architecture) are trained on clients, and used to train a larger model at the server. Unlike in conventional ensemble learning, in FL the ensemble can be trained on clients' highly heterogeneous data. Cognizant of this property, Fed-ET uses a weighted consensus distillation scheme with diversity regularization that efficiently extracts reliable consensus from the ensemble while improving generalization by exploiting the diversity within the ensemble. We show the generalization bound for the ensemble of weighted models trained on heterogeneous datasets that supports the intuition of Fed-ET. Our experiments on image and language tasks show that Fed-ET significantly outperforms other state-of-the-art FL algorithms with fewer communicated parameters, and is also robust against high data-heterogeneity.","doi":"10.24963/ijcai.2022/399","url":"https://ijcai.org/proceedings/2022/399","pdf":"https://ijcai.org/proceedings/2022/0399.pdf","markdown":"https://ijcai.org/proceedings/2022/399.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/399","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Machine Learning","track":"Main Track","number":399,"pages":{"first":2881,"last":2887},"keywords":["Knowledge Representation and Reasoning: Applications","Machine Learning: Ensemble Methods"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986941","length_sec":83,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985084","length_sec":353,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Allen-Zhu and Li, 2021] Zeyuan Allen-Zhu and Yuanzhi Li. Towards understanding ensemble, knowledge distil- lation and self-distillation in deep learning. arXiv preprint arXiv:2012.09816, July 2021.","title":"Towards understanding ensemble, knowledge distillation and self-distillation in deep learning","authors":"Allen-Zhu, Li ; Zeyuan Allen-Zhu, Yuanzhi Li","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bhargava et al., 2021] Prajjwal Bhargava, Aleksandr Drozd, and Anna Rogers. Generalization in nli: Ways (not) to go beyond simple heuristics. arXiv preprint arXiv:2110.01518, 2021.","title":"Generalization in nli: Ways (not) to go beyond simple heuristics","authors":"Bhargava","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bonawitz et al., 2019] Keith Bonawitz, Hubert Eichner, Wolfgang Grieskamp, Dzmitry Huba, Alex Ingerman, Vladimir Ivanov, Chloe Kiddon, Jakub Konecny, Stefano Mazzocchi, H. Brendan McMahan, Timon Van Overveldt, David Petrou, Daniel Ramage, and Jason Roselander. Towards Federated Learning at Scale: System Design. SysML, April 2019.","title":"Towards Federated Learning at Scale: System Design","authors":"Keith Bonawitz, Hubert Eichner, Wolfgang Grieskamp, Dzmitry Huba, Alex Ingerman, Vladimir Ivanov, Chloe Kiddon, Jakub Konecny, Stefano Mazzocchi, H Brendan Mcmahan, Timon Van Overveldt, David Petrou, Daniel Ramage, Jason Roselander","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Camacho-Gómez et al., 2021] Carlos Camacho-Gómez, Sancho Salcedo-Sanz, and David Camacho. A Review on Ensemble Methods and their Applications to Optimization Problems, pages 25-45. Springer Singapore, Singapore, 2021.","title":"A Review on Ensemble Methods and their Applications to Optimization Problems","authors":"Carlos Camacho-Gómez, Sancho Salcedo-Sanz, David Camacho","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cheng et al., 2021] Sijie Cheng, Jingwen Wu, Yanghua Xiao, Yang Liu, and Yang Liu. Fedgems: Federated learn- ing of larger server models via selective knowledge fusion. arXiv preprint arXiv:2110.11027, December 2021.","title":"Fedgems: Federated learning of larger server models via selective knowledge fusion","authors":"Sijie Cheng, Jingwen Wu, Yanghua Xiao, Yang Liu, Yang Liu","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Cho et al., 2021] Yae Jee Cho, Jianyu Wang, Tarun Chiru- volu, and Gauri Joshi. Personalized federated learning for heterogeneous clients with clustered knowledge transfer. arXiv preprint arXiv:2109.08119, 2021.","title":"Personalized federated learning for heterogeneous clients with clustered knowledge transfer","authors":"Yae Jee Cho, Jianyu Wang, Tarun Chiruvolu, Gauri Joshi","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[He et al., 2020] Chaoyang He, Murali Annavaram, and Salman Avestimehr. Group knowledge transfer: Federated learning of large cnns at the edge. In Advances in Neural Information Processing Systems, 2020.","title":"Group knowledge transfer: Federated learning of large cnns at the edge","authors":"He","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hinton et al., 2015] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, March 2015.","title":"Distilling the knowledge in a neural network","authors":"Geoffrey Hinton, Oriol Vinyals, Jeff Dean","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hong et al., 2021] Zhang-Wei Hong, Prabhat Nagarajan, and Guilherme Maeda. Periodic intra-ensemble knowl- edge distillation for reinforcement learning. In Euro- pean Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, 2021.","title":"Periodic intra-ensemble knowledge distillation for reinforcement learning","authors":"Zhang-Wei Hong, Prabhat Nagarajan, Guilherme Maeda","year":"2021","venue":"European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hsu et al., 2019] Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribution for federated visual classification. In International Workshop on Federated Learning for User Privacy and Data Confidentiality in Conjunction with NeurIPS 2019 (FL-NeurIPS'19), December 2019.","title":"Measuring the effects of non-identical data distribution for federated visual classification","authors":"Hsu","year":"2019","venue":"International Workshop on Federated Learning for User Privacy and Data Confidentiality in Conjunction with NeurIPS 2019 (FL-NeurIPS'19)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Itahara et al., 2021] Sohei Itahara, Takayuki Nishio, Yusuke Koda, Masahiro Morikura, and Koji Yamamoto. Distillation-based semi-supervised federated learning for communication-efficient collaborative training with non-iid private data. arXiv preprint arXiv:2008.06180, 2021.","title":"Distillation-based semi-supervised federated learning for communication-efficient collaborative training with non-iid private data","authors":"Itahara","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kairouz et al., 2019] Peter Kairouz, H. Brendan McMahan, Brendan Avent, and Aurelien Bellet et. al. Advances and open problems in federated learning. arXiv preprint arXiv:1912.04977, 2019.","title":"Advances and open problems in federated learning","authors":"H Brendan Peter Kairouz, Brendan Mcmahan, Aurelien Avent, Bellet","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Karimireddy et al., 2020] Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank J Reddi, Sebas- tian U Stich, and Ananda Theertha Suresh. SCAFFOLD: Stochastic controlled averaging for on-device federated learning. In Proceedings of the International Conference on Machine Learning (ICML), 2020.","title":"SCAFFOLD: Stochastic controlled averaging for on-device federated learning","authors":"Praneeth Sai, Satyen Karimireddy, Mehryar Kale, Mohri, J Sashank, Sebastian U Reddi, Ananda Stich, Suresh Theertha","year":"2020","venue":"Proceedings of the International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lan et al., 2018] X. Lan, X. Zhu, and S. Gong. Knowledge distillation by on-the-fly native ensemble. In Proceedings of the 32nd International Conference on Neural Informa- tion Processing Systems., pages 7528-7538, 2018.","title":"Knowledge distillation by on-the-fly native ensemble","authors":"X Lan, X Zhu, S Gong","year":"2018","venue":"Proceedings of the 32nd International Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Li et al., 2021] Qinbin Li, Bingsheng He, and Dawn Song. Model-contrastive federated learning. In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2021.","title":"Qinbin Li, Bingsheng He, and Dawn Song. Model-contrastive federated learning","authors":"Li","year":"2021","venue":"IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lin et al., 2020] Tao Lin, Lingjing Kong, Sebastian U. Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. In Advances in Neural Information Processing Systems, 2020.","title":"Ensemble distillation for robust model fusion in federated learning","authors":"Tao Lin, Lingjing Kong, Sebastian U Stich, Martin Jaggi","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[McMahan et al., 2017] H. Brendan McMahan, Ei- der Moore, Daniel Ramage, Seth Hampson, and Blaise Agøura y Arcas. Communication-Efficient Learning of Deep Networks from Decentralized Data. International Conference on Artificial Intelligenece and Statistics (AISTATS), April 2017.","title":"Communication-Efficient Learning of Deep Networks from Decentralized Data","authors":"H, Brendan Mcmahan, Eider Moore, Daniel Ramage, Seth Hampson, Blaise Agøura Y Arcas","year":"2017","venue":"International Conference on Artificial Intelligenece and Statistics (AISTATS)","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Ozkara et al., 2021] Kaan Ozkara, Navjot Singh, Deepesh Data, and Suhas Diggavi. Quped: Quantized personaliza- tion via distillation with applications to federated learning. In Advances in Neural Information Processing Systems, volume 34, pages 3622-3634, 2021.","title":"Quped: Quantized personalization via distillation with applications to federated learning","authors":"Kaan Ozkara, Navjot Singh, Deepesh Data, Suhas Diggavi","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Park and Kwak, 2020] SeongUk Park and Nojun Kwak. Feature-level ensemble knowledge distillation for aggre- gating knowledge from multiple networks. In European Conference on Artificial Intelligence (ECAI), 2020.","title":"Feature-level ensemble knowledge distillation for aggregating knowledge from multiple networks","authors":"Kwak ; Park, Seonguk Park, Nojun Kwak","year":"2020","venue":"European Conference on Artificial Intelligence (ECAI)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Sahu et al., 2020] Anit Kumar Sahu, Tian Li, Maziar San- jabi, Manzil Zaheer, Ameet Talwalkar, and Virginia Smith. Federated optimization for heterogeneous networks. In Proceedings of the 3rd MLSys Conference, January 2020.","title":"Federated optimization for heterogeneous networks","authors":"Anit Kumar Sahu, Tian Li, Maziar Sanjabi, Manzil Zaheer, Ameet Talwalkar, Virginia Smith","year":"2020","venue":"Proceedings of the 3rd MLSys Conference","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR, abs/1409.1556, 2014.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Tran et al., 2020] Linh Tran, Bastiaan S. Veeling, and Kevin Roth et. al. Hydra: Preserving ensemble diversity for model distillation. In ICML Workshop on Uncertainty and Robustness in Deep Learning, 2020.","title":"Hydra: Preserving ensemble diversity for model distillation","authors":"Linh Tran, Bastiaan S Veeling, Kevin Roth","year":"2020","venue":"ICML Workshop on Uncertainty and Robustness in Deep Learning","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang et al., 2021] Jianyu Wang, Zachary Charles, Zheng Xu, Gauri Joshi, H Brendan McMahan, Maruan Al- Shedivat, Galen Andrew, Salman Avestimehr, Katharine Daly, Deepesh Data, et al. A field guide to federated opti- mization. arXiv preprint arXiv:2107.06917, 2021.","title":"A field guide to federated optimization","authors":"Jianyu Wang, Zachary Charles, Zheng Xu, Gauri Joshi, H Brendan Mcmahan, Maruan Al-Shedivat, Galen Andrew, Salman Avestimehr, Katharine Daly, Deepesh Data","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}