{"type":"ScholarlyArticle","title":"Knowledge Distillation in Federated Learning: A Practical Guide","authors":[{"name":"Alessio Mora","slug":"alessio-mora","given":"Alessio","family":"Mora","affiliation":"University of Bologna"},{"name":"Irene Tenison","slug":"irene-tenison","given":"Irene","family":"Tenison","affiliation":"Massachusetts Institute of Technology (MIT)"},{"name":"Paolo Bellavista","slug":"paolo-bellavista","given":"Paolo","family":"Bellavista","affiliation":"University of Bologna"},{"name":"Irina Rish","slug":"irina-rish","given":"Irina","family":"Rish","affiliation":"Mila , University of Montreal"}],"abstract":"Federated Learning (FL) enables the training of Deep Learning models without centrally collecting possibly sensitive raw data. The most used algorithms for FL are parameter-averaging based schemes (e.g., Federated Averaging) that, however, have well known limits, i.e., model homogeneity, high communication cost, poor performance in presence of heterogeneous data distributions. Federated adaptations of regular Knowledge Distillation (KD) can solve or mitigate the weaknesses of parameter-averaging FL algorithms while possibly introducing other trade-offs. In this article, we originally present a focused review of the state-of-the-art KD-based algorithms specifically tailored for FL, by providing both a novel classification of the existing approaches and a detailed technical description of their pros, cons, and tradeoffs.","doi":"10.24963/ijcai.2024/905","url":"https://ijcai.org/proceedings/2024/905","pdf":"https://ijcai.org/proceedings/2024/0905.pdf","markdown":"https://ijcai.org/proceedings/2024/905.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/905","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Survey Track","track":"Survey Track","number":905,"pages":{"first":8188,"last":8196},"keywords":["Machine Learning: ML: Ensemble methods","Machine Learning: ML: Federated learning"],"videos":[],"references":[{"position":1,"raw":"[Anil et al., 2018] Rohan Anil, Gabriel Pereyra, Alexan- dre Passos, Robert Ormandi, George E Dahl, and Ge- offrey E Hinton. Large scale distributed neural net- work training through online distillation. arXiv preprint arXiv:1804.03235, 2018.","title":"Large scale distributed neural network training through online distillation","authors":"Rohan Anil, Gabriel Pereyra, Alexandre Passos, Robert Ormandi, George E Dahl, Geoffrey E Hinton","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bellavista et al., 2021] Paolo Bellavista, Luca Foschini, and Alessio Mora. Decentralised learning in federated deploy- ment environments: A system-level survey. ACM Comput- ing Surveys (CSUR), 54(1):1-38, 2021.","title":"Decentralised learning in federated deployment environments: A system-level survey","authors":"Paolo Bellavista, Luca Foschini, Alessio Mora","year":"2021","venue":"ACM Computing Surveys (CSUR)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chang et al., 2019] Hongyan Chang, Virat Shejwalkar, Reza Shokri, and Amir Houmansadr. Cronus: Robust and heterogeneous collaborative learning with black-box knowledge transfer. arXiv preprint arXiv:1912.11279, 2019.","title":"Cronus: Robust and heterogeneous collaborative learning with black-box knowledge transfer","authors":"Hongyan Chang, Virat Shejwalkar, Reza Shokri, Amir Houmansadr","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen and Chao, 2020] Hong-You Chen and Wei-Lun Chao. Fedbe: Making bayesian model ensemble applicable to federated learning. arXiv preprint arXiv:2009.01974, 2020.","title":"Fedbe: Making bayesian model ensemble applicable to federated learning","authors":"Chen, Chao ; Hong-You Chen, Wei-Lun Chao","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2023] Zihan Chen, Howard Hao Yang, Tony Quek, and K. F. Ernest Chong. Spectral co-distillation for personalized federated learning. In Thirty-seventh Confer- ence on Neural Information Processing Systems, 2023.","title":"Spectral co-distillation for personalized federated learning","authors":"Zihan Chen, Howard Hao Yang, Tony Quek, K F, Ernest Chong","year":"2023","venue":"Thirty-seventh Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Cheng et al., 2021] Sijie Cheng, Jingwen Wu, Yanghua Xiao, and Yang Liu. Fedgems: Federated learning of larger server models via selective knowledge fusion. arXiv preprint arXiv:2110.11027, 2021.","title":"Fedgems: Federated learning of larger server models via selective knowledge fusion","authors":"Sijie Cheng, Jingwen Wu, Yanghua Xiao, Yang Liu","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Diakonikolas et al., 2017] Ilias Diakonikolas, Gautam Ka- math, Daniel M Kane, Jerry Li, Ankur Moitra, and Alistair Stewart. Being robust (in high dimensions) can be practi- cal. In International Conference on Machine Learning, pages 999-1008. PMLR, 2017.","title":"Being robust (in high dimensions) can be practical","authors":"Ilias Diakonikolas, Gautam Kamath, Jerry Daniel M Kane, Ankur Li, Alistair Moitra, Stewart","year":"2017","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dwork et al., 2014] Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations of differential privacy. Founda- tions and Trends® in Theoretical Computer Science, 9(3- 4):211-407, 2014.","title":"The algorithmic foundations of differential privacy","authors":"Cynthia Dwork, Aaron Roth","year":"2014","venue":"Foundations and Trends® in Theoretical Computer Science","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gong et al., 2021] Xuan Gong, Abhishek Sharma, Srikr- ishna Karanam, Ziyan Wu, Terrence Chen, David Doer- mann, and Arun Innanje. Ensemble attention distillation for privacy-preserving federated learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15076-15086, 2021.","title":"Ensemble attention distillation for privacy-preserving federated learning","authors":"Xuan Gong, Abhishek Sharma, Srikrishna Karanam, Ziyan Wu, Terrence Chen, David Doermann, Arun Innanje","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gou et al., 2021] Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey. International Journal of Computer Vision, 129(6):1789-1819, 2021.","title":"Knowledge distillation: A survey","authors":"Jianping Gou, Baosheng Yu, Stephen J Maybank, Dacheng Tao","year":"2021","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Guo et al., 2024] Kuangpu Guo, Yuhe Ding, Jian Liang, Ran He, Zilei Wang, and Tieniu Tan. Not all minorities are equal: Empty-class-aware distillation for heterogeneous federated learning. arXiv preprint arXiv:2401.02329, 2024.","title":"Not all minorities are equal: Empty-class-aware distillation for heterogeneous federated learning","authors":"Guo","year":"2024","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[He et al., 2020] Chaoyang He, Murali Annavaram, and Salman Avestimehr. Group knowledge transfer: Federated learning of large cnns at the edge. Advances in Neural Information Processing Systems, 33:14068-14080, 2020.","title":"Group knowledge transfer: Federated learning of large cnns at the edge","authors":"He","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[He et al., 2022a] Yuting He, Yiqiang Chen, XiaoDong Yang, Hanchao Yu, Yi-Hua Huang, and Yang Gu. Learn- ing critically: Selective self-distillation in federated learn- ing on non-iid data. IEEE Transactions on Big Data, 2022.","title":"Learning critically: Selective self-distillation in federated learning on non-iid data","authors":"He","year":"2022","venue":"IEEE Transactions on Big Data","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[He et al., 2022b] Yuting He, Yiqiang Chen, Xiaodong Yang, Yingwei Zhang, and Bixiao Zeng. Class-wise adap- tive self distillation for heterogeneous federated learning. 2022.","title":"Class-wise adaptive self distillation for heterogeneous federated learning","authors":"He","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Hinton et al., 2015] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.","title":"Distilling the knowledge in a neural network","authors":"Geoffrey Hinton, Oriol Vinyals, Jeff Dean","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Hu et al., 2021] Li Hu, Hongyang Yan, Lang Li, Zijie Pan, Xiaozhang Liu, and Zulong Zhang. Mhat: an effi- cient model-heterogenous aggregation training scheme for federated learning. Information Sciences, 560:493-503, 2021.","title":"Mhat: an efficient model-heterogenous aggregation training scheme for federated learning","authors":"Li Hu, Hongyang Yan, Lang Li, Zijie Pan, Xiaozhang Liu, Zulong Zhang","year":"2021","venue":"Information Sciences","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Itahara et al., 2021] Sohei Itahara, Takayuki Nishio, Yusuke Koda, Masahiro Morikura, and Koji Yamamoto. Distillation-based semi-supervised federated learning for communication-efficient collaborative training with non-iid private data. IEEE Transactions on Mobile Computing, 22(1):191-205, 2021.","title":"Distillation-based semi-supervised federated learning for communication-efficient collaborative training with non-iid private data","authors":"Itahara","year":"2021","venue":"IEEE Transactions on Mobile Computing","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Jeong et al., 2018] Eunjeong Jeong, Seungeun Oh, Hye- sung Kim, Jihong Park, Mehdi Bennis, and Seong-Lyun Kim. Communication-efficient on-device machine learn- ing: Federated distillation and augmentation under non-iid private data. arXiv preprint arXiv:1811.11479, 2018.","title":"Communication-efficient on-device machine learning: Federated distillation and augmentation under non-iid private data","authors":"Jeong","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Jin et al., 2022] Hai Jin, Dongshan Bai, Dezhong Yao, Yu- tong Dai, Lin Gu, Chen Yu, and Lichao Sun. Personalized edge intelligence via federated self-knowledge distillation. IEEE Transactions on Parallel and Distributed Systems, 34(2):567-580, 2022.","title":"Personalized edge intelligence via federated self-knowledge distillation","authors":"Hai Jin, Dongshan Bai, Dezhong Yao, Yutong Dai, Lin Gu, Chen Yu, Lichao Sun","year":"2022","venue":"IEEE Transactions on Parallel and Distributed Systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Karimireddy et al., 2020] Sai P Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and A Theertha Suresh. Scaffold: Stochastic controlled aver- aging for federated learning. In International Conference on Machine Learning, pages 5132-5143. PMLR, 2020.","title":"Scaffold: Stochastic controlled averaging for federated learning","authors":"Satyen Sai P Karimireddy, Mehryar Kale, Sashank Mohri, Sebastian Reddi, Stich, Theertha, Suresh","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Kim et al., 2022] Jinkyu Kim, Geeho Kim, and Bohyung Han. Multi-level branched regularization for federated learning. In International Conference on Machine Learn- ing, pages 11058-11073. PMLR, 2022.","title":"Multi-level branched regularization for federated learning","authors":"Kim","year":"2022","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Lee et al., 2022] Gihun Lee, Minchan Jeong, Yongjin Shin, Sangmin Bae, and Se-Young Yun. Preservation of the global knowledge by not-true distillation in federated learning. In Advances in Neural Information Processing Systems, 2022.","title":"Preservation of the global knowledge by not-true distillation in federated learning","authors":"Lee","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Li and Wang, 2019] Daliang Li and Junpu Wang. Fedmd: Heterogenous federated learning via model distillation. arXiv preprint arXiv:1910.03581, 2019.","title":"Fedmd: Heterogenous federated learning via model distillation","authors":"Wang ; Li, Daliang Li, Junpu Wang","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Lin et al., 2020] Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. Advances in Neural Informa- tion Processing Systems, 33:2351-2363, 2020.","title":"Ensemble distillation for robust model fusion in federated learning","authors":"Tao Lin, Lingjing Kong, Sebastian U Stich, Martin Jaggi","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Liu et al., 2023a] Chenghao Liu, Xiaoyang Qu, Jianzong Wang, and Jing Xiao. Fedet: a communication-efficient federated class-incremental learning framework based on enhanced transformer. In Proceedings of the Thirty- Second International Joint Conference on Artificial Intel- ligence, pages 3984-3992, 2023.","title":"Fedet: a communication-efficient federated class-incremental learning framework based on enhanced transformer","authors":"Liu","year":"2023","venue":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2023/443"},{"position":26,"raw":"[Liu et al., 2023b] Jianchun Liu, Qingmin Zeng, Hongli Xu, Yang Xu, Zhiyuan Wang, and He Huang. Adaptive block- wise regularization and knowledge distillation for enhanc- ing federated learning. IEEE/ACM Transactions on Net- working, 2023.","title":"Adaptive blockwise regularization and knowledge distillation for enhancing federated learning","authors":"Liu","year":"2023","venue":"IEEE/ACM Transactions on Networking","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Lu et al., 2023] Jianghu Lu, Shikun Li, Kexin Bao, Pengju Wang, Zhenxing Qian, and Shiming Ge. Federated learn- ing with label-masking distillation. In Proceedings of the 31st ACM International Conference on Multimedia, pages 222-232, 2023.","title":"Federated learning with label-masking distillation","authors":"Jianghu Lu, Shikun Li, Kexin Bao, Pengju Wang, Zhenxing Qian, Shiming Ge","year":"2023","venue":"Proceedings of the 31st ACM International Conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Luo et al., 2021] Mi Luo, Fei Chen, Dapeng Hu, Yifan Zhang, Jian Liang, and Jiashi Feng. No fear of hetero- geneity: Classifier calibration for federated learning with non-iid data. Advances in Neural Information Processing Systems, 34:5972-5984, 2021.","title":"No fear of heterogeneity: Classifier calibration for federated learning with non-iid data","authors":"Mi Luo, Fei Chen, Dapeng Hu, Yifan Zhang, Jian Liang, Jiashi Feng","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[McMahan et al., 2016] H Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, et al. Communication- efficient learning of deep networks from decentralized data. arXiv preprint arXiv:1602.05629, 2016.","title":"Communicationefficient learning of deep networks from decentralized data","authors":"Eider H Brendan Mcmahan, Daniel Moore, Seth Ramage, Hampson","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Mendieta et al., 2022] Matias Mendieta, Taojiannan Yang, Pu Wang, Minwoo Lee, Zhengming Ding, and Chen Chen. Local learning matters: Rethinking data heterogeneity in federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8397-8406, 2022.","title":"Local learning matters: Rethinking data heterogeneity in federated learning","authors":"Matias Mendieta, Taojiannan Yang, Pu Wang, Minwoo Lee, Zhengming Ding, Chen Chen","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Ni et al., 2022] Xuanming Ni, Xinyuan Shen, and Huimin Zhao. Federated optimization via knowledge codistilla- tion. Expert Systems with Applications, 191:116310, 2022.","title":"Federated optimization via knowledge codistillation","authors":"Ni","year":"2022","venue":"Expert Systems with Applications","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Papernot et al., 2016] Nicolas Papernot, Martín Abadi, Ul- far Erlingsson, Ian Goodfellow, and Kunal Talwar. Semi- supervised knowledge transfer for deep learning from pri- vate training data. arXiv preprint arXiv:1610.05755, 2016.","title":"Semisupervised knowledge transfer for deep learning from private training data","authors":"Nicolas Papernot, Martín Abadi, Ulfar Erlingsson, Ian Goodfellow, Kunal Talwar","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Poirot et al., 2019] Maarten G Poirot, Praneeth Vepakomma, Ken Chang, Jayashree Kalpathy-Cramer, Rajiv Gupta, and Ramesh Raskar. Split learning for collaborative deep learning in healthcare. arXiv preprint arXiv:1912.12115, 2019.","title":"Split learning for collaborative deep learning in healthcare","authors":"Praneeth Maarten G Poirot, Ken Vepakomma, Jayashree Chang, Rajiv Kalpathy-Cramer, Ramesh Gupta, Raskar","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Reddi et al., 2020] Sashank Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečnỳ, Sanjiv Kumar, and H Brendan McMa- han. Adaptive federated optimization. arXiv preprint arXiv:2003.00295, 2020.","title":"Adaptive federated optimization","authors":"Sashank Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečnỳ, Sanjiv Kumar, Brendan Mcmahan","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Sattler et al., 2019] Felix Sattler, Simon Wiedemann, Klaus-Robert Müller, and Wojciech Samek. Robust and communication-efficient federated learning from non-iid data. IEEE transactions on neural networks and learning systems, 2019.","title":"Robust and communication-efficient federated learning from non-iid data","authors":"Felix Sattler, Simon Wiedemann, Klaus-Robert Müller, Wojciech Samek","year":"2019","venue":"IEEE transactions on neural networks and learning systems","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Sattler et al., 2021a] Felix Sattler, Tim Korjakow, Roman Rischke, and Wojciech Samek. Fedaux: Leveraging un- labeled auxiliary data in federated learning. IEEE Trans- actions on Neural Networks and Learning Systems, 2021.","title":"Fedaux: Leveraging unlabeled auxiliary data in federated learning","authors":"Sattler","year":"2021","venue":"IEEE Transactions on Neural Networks and Learning Systems","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Sattler et al., 2021b] Felix Sattler, Arturo Marban, Roman Rischke, and Wojciech Samek. Cfd: Communication- efficient federated distillation via soft-label quantization and delta coding. IEEE Transactions on Network Science and Engineering, 2021.","title":"Cfd: Communicationefficient federated distillation via soft-label quantization and delta coding","authors":"Sattler","year":"2021","venue":"IEEE Transactions on Network Science and Engineering","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Selvaraju et al., 2017] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on com- puter vision, pages 618-626, 2017.","title":"Grad-cam: Visual explanations from deep networks via gradient-based localization","authors":"Michael Ramprasaath R Selvaraju, Abhishek Cogswell, Ramakrishna Das, Devi Vedantam, Dhruv Parikh, Batra","year":"2017","venue":"Proceedings of the IEEE international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Seo et al., 2020] Hyowoon Seo, Jihong Park, Seungeun Oh, Mehdi Bennis, and Seong-Lyun Kim. Federated knowl- edge distillation. arXiv preprint arXiv:2011.02367, 2020.","title":"Federated knowledge distillation","authors":"Seo","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Wang et al., 2023] Haozhao Wang, Yichen Li, Wenchao Xu, Ruixuan Li, Yufeng Zhan, and Zhigang Zeng. Dafkd: Domain-aware federated knowledge distillation. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20412-20421, 2023.","title":"Dafkd: Domain-aware federated knowledge distillation","authors":"Haozhao Wang, Yichen Li, Wenchao Xu, Ruixuan Li, Yufeng Zhan, Zhigang Zeng","year":"2023","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Wu et al., 2022a] Chen Wu, Sencun Zhu, and Prasenjit Mi- tra. Federated unlearning with knowledge distillation. arXiv preprint arXiv:2201.09441, 2022.","title":"Federated unlearning with knowledge distillation","authors":"Wu","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Wu et al., 2022b] Zhiyuan Wu, Sheng Sun, Yuwei Wang, Min Liu, Quyang Pan, Junbo Zhang, Zeju Li, and Qingxi- ang Liu. Exploring the distributed knowledge congruence in proxy-data-free federated distillation. ACM Transac- tions on Intelligent Systems and Technology, 2022.","title":"Exploring the distributed knowledge congruence in proxy-data-free federated distillation","authors":"Wu","year":"2022","venue":"ACM Transactions on Intelligent Systems and Technology","doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Wu et al., 2024] Zhiyuan Wu, Tianliu He, Sheng Sun, Yuwei Wang, Min Liu, Bo Gao, and Xuefeng Jiang. Feder- ated class-incremental learning with new-class augmented self-distillation. arXiv preprint arXiv:2401.00622, 2024.","title":"Federated class-incremental learning with new-class augmented self-distillation","authors":"Zhiyuan Wu, Tianliu He, Sheng Sun, Yuwei Wang, Min Liu, Bo Gao, Xuefeng Jiang","year":"2024","venue":null,"doi":null,"url":null,"internal_url":null},{"position":44,"raw":"[Yang et al., 2020] Taojiannan Yang, Sijie Zhu, and Chen Chen. Gradaug: A new regularization method for deep neural networks. Advances in Neural Information Process- ing Systems, 33:14207-14218, 2020.","title":"Gradaug: A new regularization method for deep neural networks","authors":"Yang","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":45,"raw":"[Yao et al., 2023] Dezhong Yao, Wanning Pan, Yutong Dai, Yao Wan, Xiaofeng Ding, Chen Yu, Hai Jin, Zheng Xu, and Lichao Sun. F ed gkd: Towards heterogeneous fed- erated learning via global knowledge distillation. IEEE Transactions on Computers, 2023.","title":"F ed gkd: Towards heterogeneous federated learning via global knowledge distillation","authors":"Yao","year":"2023","venue":"IEEE Transactions on Computers","doi":null,"url":null,"internal_url":null},{"position":46,"raw":"[Zhang et al., 2022a] Lan Zhang, Dapeng Wu, and Xiaoy- ong Yuan. Fedzkt: Zero-shot knowledge transfer to- wards resource-constrained federated learning with het- erogeneous on-device models. In 2022 IEEE 42nd In- ternational Conference on Distributed Computing Systems (ICDCS), pages 928-938. IEEE, 2022.","title":"Fedzkt: Zero-shot knowledge transfer towards resource-constrained federated learning with heterogeneous on-device models","authors":"Zhang","year":"2022","venue":"IEEE 42nd International Conference on Distributed Computing Systems (ICDCS)","doi":null,"url":null,"internal_url":null},{"position":47,"raw":"[Zhang et al., 2022b] Lin Zhang, Li Shen, Liang Ding, Dacheng Tao, and Ling-Yu Duan. Fine-tuning global model via data-free knowledge distillation for non-iid fed- erated learning. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pages 10174-10183, 2022.","title":"Fine-tuning global model via data-free knowledge distillation for non-iid federated learning","authors":"Zhang","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":48,"raw":"[Zhu et al., 2021] Zhuangdi Zhu, Junyuan Hong, and Jiayu Zhou. Data-free knowledge distillation for heterogeneous federated learning. In International Conference on Ma- chine Learning, pages 12878-12889. PMLR, 2021.","title":"Data-free knowledge distillation for heterogeneous federated learning","authors":"Zhu","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}