{"type":"ScholarlyArticle","title":"FedET: A Communication-Efficient Federated Class-Incremental Learning Framework Based on Enhanced Transformer","authors":[{"name":"Chenghao Liu","slug":"chenghao-liu","given":"Chenghao","family":"Liu","affiliation":"Ping An Technology (Shenzhen) Co., Ltd. , Shenzhen , China; The Shenzhen International Graduate School , Tsinghua University , China"},{"name":"Xiaoyang Qu","slug":"xiaoyang-qu","given":"Xiaoyang","family":"Qu","affiliation":"Ping An Technology (Shenzhen) Co., Ltd. , Shenzhen , China"},{"name":"Jianzong Wang","slug":"jianzong-wang","given":"Jianzong","family":"Wang","affiliation":"Ping An Technology (Shenzhen) Co., Ltd. , Shenzhen , China"},{"name":"Jing Xiao","slug":"jing-xiao","given":"Jing","family":"Xiao","affiliation":"Ping An Technology (Shenzhen) Co., Ltd. , Shenzhen , China"}],"abstract":"Federated Learning (FL) has been widely concerned for it enables decentralized learning while ensuring data privacy. However, most existing methods unrealistically assume that the classes encountered by local clients are fixed over time. After learning new classes, this impractical assumption will make the model's catastrophic forgetting of old classes significantly severe. Moreover, due to the limitation of communication cost, it is challenging to use large-scale models in FL, which will affect the prediction accuracy. To address these challenges, we propose a novel framework, Federated Enhanced Transformer (FedET), which simultaneously achieves high accuracy and low communication cost. Specifically, FedET uses Enhancer, a tiny module, to absorb and communicate new knowledge, and applies pre-trained Transformers combined with different Enhancers to ensure high precision on various tasks. To address local forgetting caused by new classes of new tasks and global forgetting brought by non-i.i.d class imbalance across different local clients, we proposed an Enhancer distillation method to modify the imbalance between old and new knowledge and repair the non-i.i.d. problem. Experimental results demonstrate that FedET's average accuracy on a representative benchmark dataset is 14.1% higher than the state-of-the-art method, while FedET saves 90% of the communication cost compared to the previous method.","doi":"10.24963/ijcai.2023/443","url":"https://ijcai.org/proceedings/2023/443","pdf":"https://ijcai.org/proceedings/2023/0443.pdf","markdown":"https://ijcai.org/proceedings/2023/443.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/443","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Machine Learning","track":"Main Track","number":443,"pages":{"first":3984,"last":3992},"keywords":["Machine Learning: ML: Federated learning","Machine Learning: ML: Incremental learning"],"videos":[],"references":[{"position":1,"raw":"[Ahn et al., 2021] Hongjoon Ahn, Jihwan Kwak, Subin Lim, Hyeonsu Bang, Hyojun Kim, and Taesup Moon. SS- IL: separated softmax for incremental learning. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pages 824-833, 2021.","title":"SS-IL: separated softmax for incremental learning","authors":"Ahn","year":"2021","venue":"IEEE/CVF International Conference on Computer Vision, ICCV 2021","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Ba and Caruana, 2014] Jimmy Ba and Rich Caruana. Do deep nets really need to be deep? In Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, Decem- ber 8-13 2014, Montreal, Quebec, Canada, pages 2654- 2662, 2014.","title":"Do deep nets really need to be deep?","authors":"Caruana Ba, Jimmy Ba, Rich Caruana","year":"2014","venue":"Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2020] Jiaao Chen, Zichao Yang, and Diyi Yang. Mixtext: Linguistically-informed interpolation of hidden space for semi-supervised text classification. In Proceed- ings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 2147-2157, 2020.","title":"Mixtext: Linguistically-informed interpolation of hidden space for semi-supervised text classification","authors":"Chen","year":"2020","venue":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[de Masson d'Autume et al., 2019] Cyprien de Mas- son d'Autume, Sebastian Ruder, Lingpeng Kong, and Dani Yogatama. Episodic memory in lifelong language learning. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 13122-13131, 2019.","title":"Cyprien de Masson d'Autume, Sebastian Ruder, Lingpeng Kong, and Dani Yogatama. Episodic memory in lifelong language learning","authors":"De Masson D'autume","year":"2019","venue":"Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li- Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Computer So- ciety Conference on Computer Vision and Pattern Recog- nition (CVPR 2009), 20-25 June 2009, Miami, Florida, USA, pages 248-255, 2009.","title":"Imagenet: A large-scale hierarchical image database","authors":"Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Li Fei-Fei","year":"2009","venue":"IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dong et al., 2022] Jiahua Dong, Lixu Wang, Zhen Fang, Gan Sun, Shichao Xu, Xiao Wang, and Qi Zhu. Feder- ated class-incremental learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 10154- 10163, 2022.","title":"Federated class-incremental learning","authors":"Jiahua Dong, Lixu Wang, Zhen Fang, Gan Sun, Shichao Xu, Xiao Wang, Qi Zhu","year":"2022","venue":"IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dong et al., 2023] Jiahua Dong, Yang Cong, Gan Sun, Yu- lun Zhang, Bernt Schiele, and Dengxin Dai. No one left behind: Real-world federated class-incremental learning. abs/2302.00903, 2023.","title":"No one left behind: Real-world federated class-incremental learning","authors":"Jiahua Dong, Yang Cong, Gan Sun, Yulun Zhang, Bernt Schiele, Dengxin Dai","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Douillard et al., 2020] Arthur Douillard, Matthieu Cord, Charles Ollion, Thomas Robert, and Eduardo Valle. Pod- net: Pooled outputs distillation for small-tasks incremental learning. In Computer Vision -ECCV 2020 -16th Euro- pean Conference, Glasgow, UK, August 23-28, 2020, Pro- ceedings, Part XX, pages 86-102, 2020.","title":"Podnet: Pooled outputs distillation for small-tasks incremental learning","authors":"Arthur Douillard, Matthieu Cord, Charles Ollion, Thomas Robert, Eduardo Valle","year":"2020","venue":"Computer Vision -ECCV 2020 -16th European Conference","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Ermis et al., 2022] Beyza Ermis, Giovanni Zappella, Mar- tin Wistuba, and Cédric Archambeau. Memory ef- ficient continual learning for neural text classification. abs/2203.04640, 2022.","title":"Memory efficient continual learning for neural text classification","authors":"Ermis","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[He et al., 2022] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross B. Girshick. Masked autoencoders are scalable vision learners. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 15979-15988, 2022.","title":"Masked autoencoders are scalable vision learners","authors":"Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, Ross B Girshick","year":"2022","venue":"IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hendryx et al., 2021] Sean M. Hendryx, Dharma Raj KC, Bradley Walls, and Clayton T. Morrison. Federated recon- naissance: Efficient, distributed, class-incremental learn- ing. abs/2109.00150, 2021.","title":"Federated reconnaissance: Efficient, distributed, class-incremental","authors":"Sean M Hendryx, Dharma Raj, K C, Bradley Walls, Clayton T Morrison","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hong et al., 2021] Zhenhou Hong, Jianzong Wang, Xi- aoyang Qu, Jie Liu, Chendong Zhao, and Jing Xiao. Federated learning with dynamic transformer for text to speech. In Interspeech 2021, 22nd Annual Conference of the International Speech Communication Association, Brno, Czechia, 30 August -3 September 2021, pages 3590-3594, 2021.","title":"Federated learning with dynamic transformer for text to speech","authors":"Hong","year":"2021","venue":"Interspeech 2021, 22nd Annual Conference of the International Speech Communication Association","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hu et al., 2021] Xinting Hu, Kaihua Tang, Chunyan Miao, Xian-Sheng Hua, and Hanwang Zhang. Distilling causal effect of data in class-incremental learning. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 3957-3966, 2021.","title":"Distilling causal effect of data in class-incremental learning","authors":"Hu","year":"2021","venue":"IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Huang et al., 2021] Yufan Huang, Yanzhe Zhang, Jiaao Chen, Xuezhi Wang, and Diyi Yang. Continual learning for text classification with information disentanglement based regularization. In Proceedings of the 2021 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo- gies, NAACL-HLT 2021, Online, June 6-11, 2021, pages 2736-2746, 2021.","title":"Continual learning for text classification with information disentanglement based regularization","authors":"Huang","year":"2021","venue":"Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Krizhevsky et al., 2009] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny im- ages. Citeseer, 2009.","title":"Learning multiple layers of features from tiny images","authors":"Alex Krizhevsky, Geoffrey Hinton","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lai et al., 2015] Siwei Lai, Liheng Xu, Kang Liu, and Jun Zhao. Recurrent convolutional neural networks for text classification. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, January 25- 30, 2015, Austin, Texas, USA, pages 2267-2273, 2015.","title":"Recurrent convolutional neural networks for text classification","authors":"Siwei Lai, Liheng Xu, Kang Liu, Jun Zhao","year":"2015","venue":"Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lange et al., 2022] Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Gre- gory G. Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. 44(7):3366-3385, 2022.","title":"A continual learning survey: Defying forgetting in classification tasks","authors":"Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Gregory G Slabaugh, Tinne Tuytelaars","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Liu et al., 2020] Yaoyao Liu, Yuting Su, An-An Liu, Bernt Schiele, and Qianru Sun. Mnemonics training: Multi- class incremental learning without forgetting. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 12242-12251, 2020.","title":"Mnemonics training: Multiclass incremental learning without forgetting","authors":"Liu","year":"2020","venue":"IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Loshchilov and Hutter, 2019] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019.","title":"Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization","authors":"Hutter Loshchilov","year":"2019","venue":"7th International Conference on Learning Representations, ICLR 2019","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Qu et al., 2020] Xiaoyang Qu, Jianzong Wang, and Jing Xiao. Quantization and knowledge distillation for efficient federated learning on edge devices. In 22nd IEEE Inter- national Conference on High Performance Computing and Communications, HPCC 2020, Yanuca Island, December 14-16, 2020, pages 967-972, 2020.","title":"Quantization and knowledge distillation for efficient federated learning on edge devices","authors":"Xiaoyang Qu, Jianzong Wang, Jing Xiao","year":"2020","venue":"22nd IEEE International Conference on High Performance Computing and Communications, HPCC 2020","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Rebuffi et al., 2017] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H. Lampert. icarl: Incremental classifier and representation learning. In 2017 IEEE Conference on Computer Vision and Pattern Recog- nition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 5533-5542, 2017.","title":"icarl: Incremental classifier and representation learning","authors":"Sylvestre-Alvise, Alexander Rebuffi, Georg Kolesnikov, Christoph H Sperl, Lampert","year":"2017","venue":"IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Rücklé et al., 2021] Andreas Rücklé, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, and Iryna Gurevych. Adapterdrop: On the efficiency of adapters in transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pages 7930- 7946, 2021.","title":"Adapterdrop: On the efficiency of adapters in transformers","authors":"Andreas Rücklé, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, Iryna Gurevych","year":"2021","venue":"Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Shmelkov et al., 2017] Konstantin Shmelkov, Cordelia Schmid, and Karteek Alahari. Incremental learning of object detectors without catastrophic forgetting. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pages 3420-3429, 2017.","title":"Incremental learning of object detectors without catastrophic forgetting","authors":"Konstantin Shmelkov, Cordelia Schmid, Karteek Alahari","year":"2017","venue":"IEEE International Conference on Computer Vision, ICCV 2017","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Simon et al., 2021] Christian Simon, Piotr Koniusz, and Mehrtash Harandi. On learning the geodesic path for in- cremental learning. In IEEE Conference on Computer Vi- sion and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 1591-1600, 2021.","title":"On learning the geodesic path for incremental learning","authors":"Simon","year":"2021","venue":"IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Tan and Le, 2019] Mingxing Tan and Quoc V. Le. Efficient- net: Rethinking model scaling for convolutional neural networks. In Proceedings of the 36th International Confer- ence on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, pages 6105-6114, 2019.","title":"Efficientnet: Rethinking model scaling for convolutional neural networks","authors":"Mingxing Tan, V Quoc, Le","year":"2019","venue":"Proceedings of the 36th International Conference on Machine Learning, ICML 2019","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wolf et al., 2020] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Fun- towicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of- the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing: System Demonstrations, EMNLP 2020 -Demos, Online, November 16-20, 2020, pages 38-45, 2020.","title":"Transformers: State-ofthe-art natural language processing","authors":"Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Clara Patrick Von Platen, Yacine Ma, Julien Jernite, Canwen Plu, Teven Xu, Sylvain Le Scao, Mariama Gugger, Quentin Drame, Alexander M Lhoest, Rush","year":"2020","venue":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, EMNLP 2020 -Demos","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wu et al., 2019] Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale incremental learning. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 374-382, 2019.","title":"Large scale incremental learning","authors":"Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, Yun Fu","year":"2019","venue":"IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Yang et al., 2019] Qiang Yang, Yang Liu, Yong Cheng, Yan Kang, Tianjian Chen, and Han Yu. Federated Learning. Synthesis Lectures on Artificial Intelligence and Machine Learning. Morgan & Claypool Publishers, 2019.","title":"Federated Learning","authors":"Qiang Yang, Yang Liu, Yong Cheng, Yan Kang, Tianjian Chen, Han Yu","year":"2019","venue":"Synthesis Lectures on Artificial Intelligence and Machine Learning","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Yogatama et al., 2019] Dani Yogatama, Cyprien de Mas- son d'Autume, Jerome T. Connor, Tomás Kociský, Mike Chrzanowski, Lingpeng Kong, Angeliki Lazaridou, Wang Ling, Lei Yu, Chris Dyer, and Phil Blunsom. Learning and evaluating general linguistic intelligence. abs/1901.11373, 2019.","title":"Learning and evaluating general linguistic intelligence","authors":"Dani Yogatama, Cyprien De Masson D'autume, Jerome T Connor, Tomás Kociský, Mike Chrzanowski, Lingpeng Kong, Angeliki Lazaridou, Wang Ling, Lei Yu, Chris Dyer, Phil Blunsom","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Yoon et al., 2021] Jaehong Yoon, Wonyong Jeong, Gi- woong Lee, Eunho Yang, and Sung Ju Hwang. Feder- ated continual learning with weighted inter-client transfer. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, pages 12073-12086, 2021.","title":"Federated continual learning with weighted inter-client transfer","authors":"Yoon","year":"2021","venue":"Proceedings of the 38th International Conference on Machine Learning, ICML 2021","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhang et al., 2020] Junting Zhang, Jie Zhang, Shalini Ghosh, Dawei Li, Serafettin Tasci, Larry P. Heck, Hem- ing Zhang, and C.-C. Jay Kuo. Class-incremental learning via deep model consolidation. In IEEE Winter Conference on Applications of Computer Vision, WACV 2020, Snow- mass Village, CO, USA, March 1-5, 2020, pages 1120- 1129, 2020.","title":"Class-incremental learning via deep model consolidation","authors":"Junting Zhang, Jie Zhang, Shalini Ghosh, Dawei Li, Serafettin Tasci, Larry P Heck, Heming Zhang, C.-C. Jay Kuo","year":"2020","venue":"IEEE Winter Conference on Applications of Computer Vision, WACV 2020","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}