{"type":"ScholarlyArticle","title":"Continual Federated Learning Based on Knowledge Distillation","authors":[{"name":"Yuhang Ma","slug":"yuhang-ma","given":"Yuhang","family":"Ma","affiliation":"College of Computer Science and Technology , Zhejiang University"},{"name":"Zhongle Xie","slug":"zhongle-xie","given":"Zhongle","family":"Xie","affiliation":"College of Computer Science and Technology , Zhejiang University"},{"name":"Jue Wang","slug":"jue-wang","given":"Jue","family":"Wang","affiliation":"College of Computer Science and Technology , Zhejiang University"},{"name":"Ke Chen","slug":"ke-chen","given":"Ke","family":"Chen","affiliation":"College of Computer Science and Technology , Zhejiang University"},{"name":"Lidan Shou","slug":"lidan-shou","given":"Lidan","family":"Shou","affiliation":"College of Computer Science and Technology , Zhejiang University; State Key Laboratory of CAD&CG , Zhejiang University"}],"abstract":"Federated learning (FL) is a promising approach for learning a shared global model on decentralized data owned by multiple clients without exposing their privacy. In real-world scenarios, data accumulated at the client-side varies in distribution over time. As a consequence, the global model tends to forget the knowledge obtained from previous tasks while learning new tasks, showing signs of \"catastrophic forgetting\". Previous studies in centralized learning use techniques such as data replay and parameter regularization to mitigate catastrophic forgetting. Unfortunately, these techniques cannot adequately solve the non-trivial problem in FL. We propose Continual Federated Learning with Distillation (CFeD) to address catastrophic forgetting under FL. CFeD performs knowledge distillation on both the clients and the server, with each party independently having an unlabeled surrogate dataset, to mitigate forgetting. Moreover, CFeD assigns different learning objectives, namely learning the new task and reviewing old tasks, to different clients, aiming to improve the learning ability of the model. The results show that our method performs well in mitigating catastrophic forgetting and achieves a good trade-off between the two objectives.","doi":"10.24963/ijcai.2022/303","url":"https://ijcai.org/proceedings/2022/303","pdf":"https://ijcai.org/proceedings/2022/0303.pdf","markdown":"https://ijcai.org/proceedings/2022/303.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/303","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Data Mining","track":"Main Track","number":303,"pages":{"first":2182,"last":2188},"keywords":["Machine Learning: Incremental Learning","Data Mining: Federated Learning"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986680","length_sec":77,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984823","length_sec":328,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Chaudhry et al., 2018] Arslan Chaudhry, Marc'Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem. arXiv preprint arXiv:1812.00420, 2018.","title":"Efficient lifelong learning with a-gem","authors":"Arslan Chaudhry, Marc'aurelio Ranzato, Marcus Rohrbach, Mohamed Elhoseiny","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Delange et al., 2021] Matthias Delange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Greg Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE Transactions on Pattern Analysis and Machine Intelli- gence, pages 1-1, 2021.","title":"A continual learning survey: Defying forgetting in classification tasks","authors":"Matthias Delange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Greg Slabaugh, Tinne Tuytelaars","year":"2021","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Griffin et al., 2007] Gregory Griffin, Alex Holub, and Pietro Perona. Caltech-256 object category dataset. Technical report, California Institute of Technology, 2007.","title":"Caltech-256 object category dataset","authors":"Gregory Griffin, Alex Holub, Pietro Perona","year":"2007","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Hinton et al., 2015] Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2(7), 2015.","title":"Distilling the knowledge in a neural network","authors":"Geoffrey Hinton, Oriol Vinyals, Jeff Dean","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hung et al., 2019] Ching-Yi Hung, Cheng-Hao Tu, Cheng- En Wu, Chien-Hung Chen, Yi-Ming Chan, and Chu-Song Chen. Compacting, picking and growing for unforgetting continual learning. In Advances in Neural Information Processing Systems, pages 13647-13657, 2019.","title":"Compacting, picking and growing for unforgetting continual learning","authors":"Ching-Yi Hung, Cheng-Hao Tu, Cheng-En Wu, Chien-Hung Chen, Yi-Ming Chan, Chu-Song Chen","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Itahara et al., 2020] Sohei Itahara, Takayuki Nishio, Yusuke Koda, Masahiro Morikura, and Koji Yamamoto. Distillation-based semi-supervised federated learning for communication-efficient collaborative training with non-iid private data. arXiv preprint arXiv:2008.06180, 2020.","title":"Distillation-based semi-supervised federated learning for communication-efficient collaborative training with non-iid private data","authors":"Sohei Itahara, Takayuki Nishio, Yusuke Koda, Masahiro Morikura, Koji Yamamoto","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Jeong et al., 2018] Eunjeong Jeong, Seungeun Oh, Hye- sung Kim, Jihong Park, Mehdi Bennis, and Seong-Lyun Kim. Communication-efficient on-device machine learn- ing: Federated distillation and augmentation under non-iid private data. arXiv preprint arXiv:1811.11479, 2018.","title":"Communication-efficient on-device machine learning: Federated distillation and augmentation under non-iid private data","authors":"Jeong","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kim, 2014] Yoon Kim. Convolutional neural networks for sentence classification. CoRR, abs/1408.5882, 2014.","title":"Convolutional neural networks for sentence classification","authors":"Yoon Kim","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kirkpatrick et al., 2017] James Kirkpatrick, Razvan Pas- canu, Neil Rabinowitz, Joel Veness, Guillaume Des- jardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcom- ing catastrophic forgetting in neural networks. Proceed- ings of the national academy of sciences, 114(13):3521- 3526, 2017.","title":"Overcoming catastrophic forgetting in neural networks","authors":"James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska","year":"2017","venue":"Proceedings of the national academy of sciences","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Krizhevsky, 2009] A. Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.","title":"Learning multiple layers of features from tiny images","authors":"A Krizhevsky","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lee et al., 2019] Kibok Lee, Kimin Lee, Jinwoo Shin, and Honglak Lee. Overcoming catastrophic forgetting with unlabeled data in the wild. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 312-321, 2019.","title":"Overcoming catastrophic forgetting with unlabeled data in the wild","authors":"Lee","year":"2019","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li and Hoiem, 2017] Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935-2947, 2017.","title":"Learning without forgetting","authors":"Zhizhong Li, Derek Hoiem","year":"2017","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li and Wang, 2019] Daliang Li and Junpu Wang. Fedmd: Heterogenous federated learning via model distillation. arXiv preprint arXiv:1910.03581, 2019.","title":"Fedmd: Heterogenous federated learning via model distillation","authors":"Wang ; Li, Daliang Li, Junpu Wang","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2006] Jingyang Li, Maosong Sun, and Xian Zhang. A comparison and semi-quantitative analysis of words and character-bigrams as features in chinese text categorization. In Proceedings of the International Con- ference on ACL, pages 545-552, 2006.","title":"A comparison and semi-quantitative analysis of words and character-bigrams as features in chinese text categorization","authors":"Jingyang Li, Maosong Sun, Xian Zhang","year":"2006","venue":"Proceedings of the International Conference on ACL","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lin et al., 2020] Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. Advances in Neural Informa- tion Processing Systems, 33:2351-2363, 2020.","title":"Ensemble distillation for robust model fusion in federated learning","authors":"Tao Lin, Lingjing Kong, Sebastian U Stich, Martin Jaggi","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[McMahan et al., 2017] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Ar- cas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273-1282. PMLR, 2017.","title":"Communication-efficient learning of deep networks from decentralized data","authors":"Brendan Mcmahan, Eider Moore, Daniel Ramage, Seth Hampson, Blaise Aguera Y Arcas","year":"2017","venue":"Artificial intelligence and statistics","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[NLPIR, 2017] NLPIR. Nlpir weibo corpus. http://www. nlpir.org/, 2017. Accessed: 2017-12-03.","title":"NLPIR. Nlpir weibo corpus","authors":"Nlpir","year":"2017","venue":null,"doi":null,"url":"http://www.nlpir.org/,2017","internal_url":null},{"position":19,"raw":"[Shoham et al., 2019] Neta Shoham, Tomer Avidor, Aviv Keren, Nadav Israel, Daniel Benditkis, Liron Mor-Yosef, and Itai Zeitak. Overcoming forgetting in federated learn- ing on non-iid data. arXiv preprint arXiv:1910.07796, 2019.","title":"Overcoming forgetting in federated learning on non-iid data","authors":"Neta Shoham, Tomer Avidor, Aviv Keren, Nadav Israel, Daniel Benditkis, Liron Mor-Yosef, Itai Zeitak","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[SogouLabs, 2012] SogouLabs. Sohu news data(sogoucs). http://www.sogou.com/labs/resource/cs.php, 2012. Ac- cessed: 2020-06-22.","title":"SogouLabs. Sohu news data(sogoucs)","authors":"Sogoulabs","year":"2012","venue":null,"doi":null,"url":"http://www.sogou.com/labs/resource/cs.php,2012.Ac-cessed:2020-06-22","internal_url":null},{"position":21,"raw":"[Toneva et al., 2018] Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Ben- gio, and Geoffrey J Gordon. An empirical study of exam- ple forgetting during deep neural network learning. arXiv preprint arXiv:1812.05159, 2018.","title":"An empirical study of example forgetting during deep neural network learning","authors":"Mariya Toneva, Alessandro Sordoni, Remi Tachet Des Combes, Adam Trischler, Yoshua Bengio, Geoffrey J Gordon","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Usmanova et al., 2021] Anastasiia Usmanova, Franc ¸ois Portet, Philippe Lalanda, and German Vega. A distillation- based approach integrating continual learning and fed- erated learning for pervasive services. arXiv preprint arXiv:2109.04197, 2021.","title":"A distillationbased approach integrating continual learning and federated learning for pervasive services","authors":"Anastasiia Usmanova, Philippe Franc ¸ois Portet, German Lalanda, Vega","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yoon et al., 2021] Jaehong Yoon, Wonyong Jeong, Gi- woong Lee, Eunho Yang, and Sung Ju Hwang. Feder- ated continual learning with weighted inter-client transfer. In International Conference on Machine Learning, pages 12073-12086. PMLR, 2021.","title":"Federated continual learning with weighted inter-client transfer","authors":"Yoon","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhang et al., 2020] Junting Zhang, Jie Zhang, Shalini Ghosh, Dawei Li, Serafettin Tasci, Larry Heck, Heming Zhang, and C-C Jay Kuo. Class-incremental learning via deep model consolidation. In The IEEE Winter Conference on Applications of Computer Vision, pages 1131-1140, 2020.","title":"Class-incremental learning via deep model consolidation","authors":"Junting Zhang, Jie Zhang, Shalini Ghosh, Dawei Li, Serafettin Tasci, Larry Heck, Heming Zhang, C-C Jay Kuo","year":"2020","venue":"The IEEE Winter Conference on Applications of Computer Vision","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}