{"type":"ScholarlyArticle","title":"Dynamically Anchored Prompting for Task-Imbalanced Continual Learning","authors":[{"name":"Chenxing Hong","slug":"chenxing-hong","given":"Chenxing","family":"Hong","affiliation":"Key Laboratory of Multimedia Trusted Perception and Efficient Computing , Ministry of Education of China , Xiamen University , Xiamen , China"},{"name":"Yan Jin","slug":"yan-jin","given":"Yan","family":"Jin","affiliation":"Key Laboratory of Multimedia Trusted Perception and Efficient Computing , Ministry of Education of China , Xiamen University , Xiamen , China; Fujian Key Laboratory of Sensing and Computing for Smart City , School of Informatics , Xiamen University , Xiamen , China"},{"name":"Zhiqi Kang","slug":"zhiqi-kang","given":"Zhiqi","family":"Kang","affiliation":"Univ. Grenoble Alpes , Inria , CNRS , Grenoble INP , LJK , 38000 Grenoble , France."},{"name":"Yizhou Chen","slug":"yizhou-chen","given":"Yizhou","family":"Chen","affiliation":"Key Laboratory of Multimedia Trusted Perception and Efficient Computing , Ministry of Education of China , Xiamen University , Xiamen , China"},{"name":"Mengke Li","slug":"mengke-li","given":"Mengke","family":"Li","affiliation":"Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ) , Shenzhen , China"},{"name":"Yang Lu","slug":"yang-lu","given":"Yang","family":"Lu","affiliation":"Key Laboratory of Multimedia Trusted Perception and Efficient Computing , Ministry of Education of China , Xiamen University , Xiamen , China; Fujian Key Laboratory of Sensing and Computing for Smart City , School of Informatics , Xiamen University , Xiamen , China"},{"name":"Hanzi Wang","slug":"hanzi-wang","given":"Hanzi","family":"Wang","affiliation":"Key Laboratory of Multimedia Trusted Perception and Efficient Computing , Ministry of Education of China , Xiamen University , Xiamen , China; Fujian Key Laboratory of Sensing and Computing for Smart City , School of Informatics , Xiamen University , Xiamen , China"}],"abstract":"Existing continual learning literature relies heavily on a strong assumption that tasks arrive with a balanced data stream, which is often unrealistic in real-world applications. In this work, we explore task-imbalanced continual learning (TICL) scenarios where the distribution of task data is non-uniform across the whole learning process. We find that imbalanced tasks significantly challenge the capability of models to control the trade-off between stability and plasticity from the perspective of recent prompt-based continual learning methods. On top of the above finding, we propose Dynamically Anchored Prompting (DAP), a prompt-based method that only maintains a single general prompt to adapt to the shifts within a task stream dynamically. This general prompt is regularized in the prompt space with two specifically designed prompt anchors, called boosting anchor and stabilizing anchor, to balance stability and plasticity in TICL. Remarkably, DAP achieves this balance by only storing a prompt across the data stream, therefore offering a substantial advantage in rehearsal-free CL. Extensive experiments demonstrate that the proposed DAP results in 4.5% to 15% absolute improvements over state-of-the-art methods on benchmarks under task-imbalanced settings. Our code is available at https://github.com/chenxing6666/DAP.","doi":"10.24963/ijcai.2024/456","url":"https://ijcai.org/proceedings/2024/456","pdf":"https://ijcai.org/proceedings/2024/0456.pdf","markdown":"https://ijcai.org/proceedings/2024/456.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/456","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Machine Learning","track":"Main Track","number":456,"pages":{"first":4127,"last":4135},"keywords":["Machine Learning: ML: Classification","Machine Learning: ML: Incremental learning","Data Mining: DM: Class imbalance and unequal cost","Computer Vision: CV: Recognition (object detection, categorization)"],"videos":[],"references":[{"position":1,"raw":"[Aljundi et al., 2018] Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuyte- laars. Memory aware synapses: Learning what (not) to forget. In ECCV, pages 139-154, 2018.","title":"Memory aware synapses: Learning what (not) to forget","authors":"Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, Tinne Tuytelaars","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cao et al., 2019] Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbal- anced datasets with label-distribution-aware margin loss. NeurIPS, 32, 2019.","title":"Learning imbalanced datasets with label-distribution-aware margin loss","authors":"Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, Tengyu Ma","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Castro et al., 2018] Francisco M Castro, Manuel J Marin- Jimenez, Nicolas Guil, Cordelia Schmid, and Karteek Ala- hari. End-to-end incremental learning. In ECCV, 2018.","title":"End-to-end incremental learning","authors":"Castro","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cha et al., 2021] Hyuntak Cha, Jaeho Lee, and Jinwoo Shin. Co2l: Contrastive continual learning. In CVPR, pages 9516-9525, 2021.","title":"Co2l: Contrastive continual learning","authors":"Cha","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chaudhry et al., 2021] Arslan Chaudhry, Albert Gordo, Puneet Dokania, Philip Torr, and David Lopez-Paz. Using hindsight to anchor past knowledge in continual learning. In AAAI, volume 35, pages 6993-7001, 2021.","title":"Using hindsight to anchor past knowledge in continual learning","authors":"Arslan Chaudhry, Albert Gordo, Puneet Dokania, Philip Torr, David Lopez-Paz","year":"2021","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen et al., 2023] Xiaohua Chen, Yucan Zhou, Dayan Wu, Chule Yang, Bo Li, Qinghua Hu, and Weiping Wang. Area: adaptive reweighting via effective area for long-tailed clas- sification. In ICCV, pages 19277-19287, 2023.","title":"Area: adaptive reweighting via effective area for long-tailed classification","authors":"Xiaohua Chen, Yucan Zhou, Dayan Wu, Chule Yang, Bo Li, Qinghua Hu, Weiping Wang","year":"2023","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[De Lange et al., 2021] Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gre- gory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. TPAMI, 44(7):3366-3385, 2021.","title":"A continual learning survey: Defying forgetting in classification tasks","authors":"Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, Tinne Tuytelaars","year":"2021","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dosovitskiy et al., 2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Min- derer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.","title":"An image is worth 16x16 words: Transformers for image recognition at scale","authors":"Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Douillard et al., 2020] Arthur Douillard, Matthieu Cord, Charles Ollion, Thomas Robert, and Eduardo Valle. Pod- net: Pooled outputs distillation for small-tasks incremental learning. In ECCV, pages 86-102. Springer, 2020.","title":"Podnet: Pooled outputs distillation for small-tasks incremental learning","authors":"Arthur Douillard, Matthieu Cord, Charles Ollion, Thomas Robert, Eduardo Valle","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[He et al., 2020] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsu- pervised visual representation learning. In CVPR, pages 9729-9738, 2020.","title":"Momentum contrast for unsupervised visual representation learning","authors":"Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, Ross Girshick","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Huang et al., 2019] Chen Huang, Yining Li, Chen Change Loy, and Xiaoou Tang. Deep imbalanced learning for face recognition and attribute prediction. TPAMI, 42(11):2781- 2794, 2019.","title":"Deep imbalanced learning for face recognition and attribute prediction","authors":"Chen Huang, Yining Li, Chen Change Loy, Xiaoou Tang","year":"2019","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kang et al., 2022] Haeyong Kang, Rusty John Lloyd Mina, Sultan Rizky Hikmawan Madjid, Jaehong Yoon, Mark Hasegawa-Johnson, Sung Ju Hwang, and Chang D Yoo. Forget-free continual learning with winning subnetworks. In ICML, pages 10734-10750. PMLR, 2022.","title":"Forget-free continual learning with winning subnetworks","authors":"Kang","year":"2022","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kang et al., 2023] Zhiqi Kang, Enrico Fini, Moin Nabi, Elisa Ricci, and Karteek Alahari. A soft nearest-neighbor framework for continual semi-supervised learning. In ICCV, pages 11868-11877, 2023.","title":"A soft nearest-neighbor framework for continual semi-supervised learning","authors":"Kang","year":"2023","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kemker and Kanan, 2017] Ronald Kemker and Christopher Kanan. Fearnet: Brain-inspired model for incremental learning. arXiv preprint arXiv:1711.10563, 2017.","title":"Fearnet: Brain-inspired model for incremental learning","authors":"Kanan Kemker, Ronald Kemker, Christopher Kanan","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kemker et al., 2018] Ronald Kemker, Marc McClure, An- gelina Abitino, Tyler Hayes, and Christopher Kanan. Mea- suring catastrophic forgetting in neural networks. In AAAI, volume 32, 2018.","title":"Measuring catastrophic forgetting in neural networks","authors":"Ronald Kemker, Marc Mcclure, Angelina Abitino, Tyler Hayes, Christopher Kanan","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Khan et al., 2023] Muhammad Gul Zain Ali Khan, Muham- mad Ferjad Naeem, Luc Van Gool, Didier Stricker, Fed- erico Tombari, and Muhammad Zeshan Afzal. Introducing language guidance in prompt-based continual learning. In ICCV, pages 11463-11473, 2023.","title":"Introducing language guidance in prompt-based continual learning","authors":"Khan","year":"2023","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Kim et al., 2020] Chris Dongjoo Kim, Jinseo Jeong, and Gunhee Kim. Imbalanced continual learning with partition- ing reservoir sampling. In ECCV, pages 411-428. Springer, 2020.","title":"Imbalanced continual learning with partitioning reservoir sampling","authors":"Chris Dongjoo Kim, Jinseo Jeong, Gunhee Kim","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Kirkpatrick et al., 2017] James Kirkpatrick, Razvan Pas- canu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catas- trophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521-3526, 2017.","title":"Overcoming catastrophic forgetting in neural networks","authors":"James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska","year":"2017","venue":"Proceedings of the national academy of sciences","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Krizhevsky et al., 2009] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.","title":"Learning multiple layers of features from tiny images","authors":"Alex Krizhevsky, Geoffrey Hinton","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Liu et al., 2022] Xialei Liu, Yu-Song Hu, Xu-Sheng Cao, Andrew D Bagdanov, Ke Li, and Ming-Ming Cheng. Long- tailed class incremental learning. In European Conference on Computer Vision, pages 495-512. Springer, 2022.","title":"Longtailed class incremental learning","authors":"Liu","year":"2022","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Lopez-Paz et al., 2017] David Lopez-Paz, Robert Nishihara, Soumith Chintala, Bernhard Scholkopf, and Léon Bottou. Discovering causal signals in images. In CVPR, pages 6979-6987, 2017.","title":"Discovering causal signals in images","authors":"David Lopez-Paz, Robert Nishihara, Soumith Chintala, Bernhard Scholkopf, Léon Bottou","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Peng et al., 2023] Hanyu Peng, Weiguo Pian, Mingming Sun, and Ping Li. Dynamic re-weighting for long-tailed semi-supervised learning. In WACV, pages 6464-6474, 2023.","title":"Dynamic re-weighting for long-tailed semi-supervised learning","authors":"Hanyu Peng, Weiguo Pian, Mingming Sun, Ping Li","year":"2023","venue":"WACV","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Rebuffi et al., 2017] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In CVPR, pages 2001-2010, 2017.","title":"icarl: Incremental classifier and representation learning","authors":"Sylvestre-Alvise, Alexander Rebuffi, Georg Kolesnikov, Christoph H Sperl, Lampert","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Smith et al., 2023] James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante-Bonilla, Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira. Coda-prompt: Continual decomposed attention-based prompting for rehearsal-free continual learning. In CVPR, pages 11909-11919, 2023.","title":"Coda-prompt: Continual decomposed attention-based prompting for rehearsal-free continual learning","authors":"James Seale, Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante-Bonilla, Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, Zsolt Kira","year":"2023","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Sun et al., 2022] Qing Sun, Fan Lyu, Fanhua Shang, Wei Feng, and Liang Wan. Exploring example influence in continual learning. NIPS, 35:27075-27086, 2022.","title":"Exploring example influence in continual learning","authors":"Qing Sun, Fan Lyu, Fanhua Shang, Wei Feng, Liang Wan","year":"2022","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Vaishnavh et al., 2018] N Vaishnavh, C Raffel, and IJ Good- fellow. Theoretical insights into memorization in gans. In NeurIPS, 2018.","title":"Theoretical insights into memorization in gans","authors":"Vaishnavh, Raffel, Goodfellow","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wang et al., 2022a] Fu-Yun Wang, Da-Wei Zhou, Han-Jia Ye, and De-Chuan Zhan. Foster: Feature boosting and com- pression for class-incremental learning. In ECCV, pages 398-414. Springer, 2022.","title":"Foster: Feature boosting and compression for class-incremental learning","authors":"Wang","year":"2022","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wang et al., 2022b] Liyuan Wang, Xingxing Zhang, Qian Li, Jun Zhu, and Yi Zhong. Coscl: Cooperation of small continual learners is stronger than a big one. In ECCV, pages 254-271. Springer, 2022.","title":"Coscl: Cooperation of small continual learners is stronger than a big one","authors":"Wang","year":"2022","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wang et al., 2022c] Zhen Wang, Liu Liu, Yiqun Duan, Ya- jing Kong, and Dacheng Tao. Continual learning with life- long vision transformer. In CVPR, pages 171-181, 2022.","title":"Continual learning with lifelong vision transformer","authors":"Wang","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wang et al., 2022d] Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen-Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. Du- alprompt: Complementary prompting for rehearsal-free continual learning. In ECCV, pages 631-648, 2022.","title":"Dualprompt: Complementary prompting for rehearsal-free continual learning","authors":"Wang","year":"2022","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Wang et al., 2022e] Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In CVPR, pages 139-149, 2022.","title":"Learning to prompt for continual learning","authors":"Wang","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Wang et al., 2023] Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application. arXiv preprint arXiv:2302.00487, 2023.","title":"A comprehensive survey of continual learning: Theory, method and application","authors":"Liyuan Wang, Xingxing Zhang, Hang Su, Jun Zhu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Wu et al., 2019] Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale incremental learning. In CVPR, pages 374-382, 2019.","title":"Large scale incremental learning","authors":"Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, Yun Fu","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zhang et al., 2020] Junting Zhang, Jie Zhang, Shalini Ghosh, Dawei Li, Serafettin Tasci, Larry Heck, Heming Zhang, and C-C Jay Kuo. Class-incremental learning via deep model consolidation. In CVPR, pages 1131-1140, 2020.","title":"Class-incremental learning via deep model consolidation","authors":"Junting Zhang, Jie Zhang, Shalini Ghosh, Dawei Li, Serafettin Tasci, Larry Heck, Heming Zhang, C-C Jay Kuo","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Zhang et al., 2023a] Gengwei Zhang, Liyuan Wang, Guo- liang Kang, Ling Chen, and Yunchao Wei. Slca: Slow learner with classifier alignment for continual learning on a pre-trained model. arXiv preprint arXiv:2303.05118, 2023.","title":"Slca: Slow learner with classifier alignment for continual learning on a pre-trained model","authors":"Zhang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Zhang et al., 2023b] Yifan Zhang, Bingyi Kang, Bryan Hooi, Shuicheng Yan, and Jiashi Feng. Deep long-tailed learning: A survey. TPAMI, 2023.","title":"Deep long-tailed learning: A survey","authors":"Zhang","year":"2023","venue":"TPAMI","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}