{"type":"ScholarlyArticle","title":"Learning from Students: Online Contrastive Distillation Network for General Continual Learning","authors":[{"name":"Jin Li","slug":"jin-li","given":"Jin","family":"Li","affiliation":"School of Electrical and Information Engineering , Tianjin University"},{"name":"Zhong Ji","slug":"zhong-ji","given":"Zhong","family":"Ji","affiliation":"School of Electrical and Information Engineering , Tianjin University"},{"name":"Gang Wang","slug":"gang-wang","given":"Gang","family":"Wang","affiliation":"School of Electrical and Information Engineering , Tianjin University; CETC Key Laboratory of Aerospace Information Applications"},{"name":"Qiang Wang","slug":"qiang-wang","given":"Qiang","family":"Wang","affiliation":"School of Electrical and Information Engineering , Tianjin University"},{"name":"Feng Gao","slug":"feng-gao","given":"Feng","family":"Gao","affiliation":"CETC Key Laboratory of Aerospace Information Applications"}],"abstract":"The goal of General Continual Learning (GCL) is to preserve learned knowledge and learn new knowledge with constant memory from an infinite data stream where task boundaries are blurry. Distilling the model's response of reserved samples between the old and the new models is an effective way to achieve promise performance on GCL. However, it accumulates the inherent old model's response bias and is not robust to model changes. To this end, we propose an Online Contrastive Distillation Network (OCD-Net) to tackle these problems, which explores the merit of the student model in each time step to guide the training process of the student model. Concretely, the teacher model is devised to help the student model to consolidate the learned knowledge, which is trained online via integrating the model weights of the student model to accumulate the new knowledge. Moreover, our OCD-Net incorporates both relation and adaptive response to help the student model alleviate the catastrophic forgetting, which is also beneficial for the teacher model preserves the learned knowledge. Extensive experiments on six benchmark datasets demonstrate that our proposed OCD-Net significantly outperforms state-of-the-art approaches in 3.26%~8.71% with various buffer sizes. Our code is available at https://github.com/lijincm/OCD-Net.","doi":"10.24963/ijcai.2022/446","url":"https://ijcai.org/proceedings/2022/446","pdf":"https://ijcai.org/proceedings/2022/0446.pdf","markdown":"https://ijcai.org/proceedings/2022/446.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/446","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Machine Learning","track":"Main Track","number":446,"pages":{"first":3215,"last":3221},"keywords":["Machine Learning: Incremental Learning","Computer Vision: Recognition (object detection, categorization)"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986692","length_sec":62,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984835","length_sec":313,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Ahn et al., 2021] Hongjoon Ahn, Jihwan Kwak, Subin Lim, Hyeonsu Bang, Hyojun Kim, and Taesup Moon. Ss-il: Separated softmax for incremental learning. In ICCV, pages 844-853, 2021.","title":"Ss-il: Separated softmax for incremental learning","authors":"Ahn","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Brahma et al., 2021] Dhanajit Brahma, Vinay Kumar Verma, and Piyush Rai. Hypernetworks for contin- ual semi-supervised learning. In IJCAI Workshop on Continual Semi-Supervised Learning, 2021.","title":"Dhanajit Brahma, Vinay Kumar Verma, and Piyush Rai. Hypernetworks for continual semi-supervised learning","authors":"Brahma","year":"2021","venue":"IJCAI Workshop on Continual Semi-Supervised Learning","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Buzzega et al., 2020] Pietro Buzzega, Matteo Boschini, An- gelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: a strong, simple baseline. In NeurIPS, pages 15920-15930, 2020.","title":"Dark experience for general continual learning: a strong, simple baseline","authors":"Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, Simone Calderara","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cha et al., 2021] Hyuntak Cha, Jaeho Lee, and Jinwoo Shin. Co2l: Contrastive continual learning. In ICCV, pages 9516-9525, 2021.","title":"Co2l: Contrastive continual learning","authors":"Cha","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chaudhry et al., 2019] Arslan Chaudhry, Marc'Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-GEM. In ICLR, 2019.","title":"Efficient lifelong learning with a-GEM","authors":"Arslan Chaudhry, Marc'aurelio Ranzato, Marcus Rohrbach, Mohamed Elhoseiny","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen et al., 2021] Liqun Chen, Dong Wang, Zhe Gan, Jingjing Liu, Ricardo Henao, and Lawrence Carin. Wasserstein contrastive representation distillation. In CVPR, pages 16296-16305, 2021.","title":"Wasserstein contrastive representation distillation","authors":"Liqun Chen, Dong Wang, Zhe Gan, Jingjing Liu, Ricardo Henao, Lawrence Carin","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[David and Akansel, 2018] Isele David and Cosgun Akansel. Selective experience replay for lifelong learning. In AAAI, pages 3302-3309, 2018.","title":"Selective experience replay for lifelong learning","authors":"Isele David, Cosgun Akansel","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Delange et al., 2021] Matthias Delange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Greg Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE TPAMI, 2021.","title":"A continual learning survey: Defying forgetting in classification tasks","authors":"Matthias Delange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Greg Slabaugh, Tinne Tuytelaars","year":"2021","venue":"IEEE TPAMI","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hadi and Saman, 2015] Pouransari Hadi and Ghili Saman. Tiny imagenet visual recognition challenge. CS231N course, Stanford University, Stanford, CA, USA, 2015.","title":"Pouransari Hadi and Ghili Saman. Tiny imagenet visual recognition challenge","authors":"Saman Hadi","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[He et al., 2020] Kaiming He, Haoqi Fan, Yuxin Wu, Sain- ing Xie, and Ross Girshick. Momentum contrast for un- supervised visual representation learning. In CVPR, pages 9239-9738, 2020.","title":"Momentum contrast for unsupervised visual representation learning","authors":"Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, Ross Girshick","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Ji et al., 2021a] Zhong Ji, Jin Li, Qiang Wang, and Zhongfei Zhang. Complementary calibration: Boosting general continual learning with collaborative distillation and self- supervision. arXiv preprint arXiv: 2109.02426, 2021.","title":"Complementary calibration: Boosting general continual learning with collaborative distillation and selfsupervision","authors":"Ji","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Ji et al., 2021b] Zhong Ji, Jiayi Liu, Qiang Wang, and Zhongfei Zhang. Coordinating experience replay: A harmonious experience retention approach for continual learning. Knowledge-Based Systems, 234:107589, 2021.","title":"Coordinating experience replay: A harmonious experience retention approach for continual learning","authors":"Ji","year":"2021","venue":"Knowledge-Based Systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Khosla et al., 2020] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised con- trastive learning. In NeurIPS, pages 18661-18673, 2020.","title":"Supervised contrastive learning","authors":"Khosla","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Krizhevsky and Hinton, 2009] A. Krizhevsky and G. Hin- ton. Learning multiple layers of features from tiny images. Handbook of systemic autoimmune diseases, 1(4), 2009.","title":"Learning multiple layers of features from tiny images","authors":"A Krizhevsky, G Hinton","year":"2009","venue":"Handbook of systemic autoimmune diseases","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liu et al., 2021] Yaoyao Liu, Bernt Schiele, and Qianru Sun. Adaptive aggregation networks for class-incremental learning. In CVPR, pages 2544-2553, 2021.","title":"Adaptive aggregation networks for class-incremental learning","authors":"Liu","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[McCloskey and Cohen, 1989] Michael McCloskey and Neal J. Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. Psychology of Learning and Motivation, 24:109-165, 1989.","title":"Catastrophic interference in connectionist networks: The sequential learning problem","authors":"Michael Mccloskey, Neal J Cohen","year":"1989","venue":"Psychology of Learning and Motivation","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Mi et al., 2020] Fei Mi, Lingjing Kong, Tao Lin, Kaicheng Yu, and Boi Faltings. Generalized class incremental learn- ing. In CVPR Workshop, pages 970-974, 2020.","title":"Generalized class incremental learning","authors":"Fei Mi, Lingjing Kong, Tao Lin, Kaicheng Yu, Boi Faltings","year":"2020","venue":"CVPR Workshop","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Quang et al., 2021] Pham Quang, Liu Chenghao, and Hoi Steven. Dualnet: Continual learning, fast and slow. In NeurIPS, pages 16131-16144, 2021.","title":"Dualnet: Continual learning, fast and slow","authors":"Pham Quang, Liu Chenghao, Hoi Steven","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Rahaf et al., 2019] Aljundi Rahaf, Lin Min, Goujaud Bap- tiste, and Bengio Yoshua. Gradient based sample selection for online continual learning. In NeurIPS, pages 11816- 11825, 2019.","title":"Gradient based sample selection for online continual learning","authors":"Rahaf","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Simon et al., 2021] Christian Simon, Piotr Koniusz, and Mehrtash Harandi. On learning the geodesic path for in- cremental learning. In CVPR, pages 1591-1600, 2021.","title":"On learning the geodesic path for incremental learning","authors":"Simon","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Tao et al., 2020] Xiaoyu Tao, Xiaopeng Hong, Xinyuan Chang, Songlin Dong, Xing Wei, and Yihong Gong. Few- shot class-incremental learning. In CVPR, pages 12183- 12192, 2020.","title":"Fewshot class-incremental learning","authors":"Xiaoyu Tao, Xiaopeng Hong, Xinyuan Chang, Songlin Dong, Xing Wei, Yihong Gong","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Vitter, 1985] Jeffrey S. Vitter. Random sampling with a reservoir. ACM Transactions on Mathematical Software, 11(1):37-57, 1985.","title":"Random sampling with a reservoir","authors":"Jeffrey S Vitter","year":"1985","venue":"ACM Transactions on Mathematical Software","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wu et al., 2019] Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale incremental learning. In CVPR, pages 374- 382, 2019.","title":"Large scale incremental learning","authors":"Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, Yun Fu","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhang et al., 2020] Youcai Zhang, Zhonghao Lan, Yuchen Dai, Fangao Zeng, Yan Bai, Jie Chang, and Yichen Wei. Prime-aware adaptive distillation. In ECCV, pages 658- 674, 2020.","title":"Prime-aware adaptive distillation","authors":"Zhang","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhao et al., 2020] Bowen Zhao, Xi Xiao, Guojun Gan, Bin Zhang, and Shu-Tao Xia. Maintaining discrimination and fairness in class incremental learning. In CVPR, pages 13205-13214, 2020.","title":"Maintaining discrimination and fairness in class incremental learning","authors":"Bowen Zhao, Xi Xiao, Guojun Gan, Bin Zhang, Shu-Tao Xia","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhao et al., 2021] Hanbin Zhao, Yongjian Fu, Mintong Kang, Qi Tian, Fei Wu, and Xi Li. Mgsvf: Multi-grained slow vs. fast framework for few-shot class-incremental learning. IEEE TPAMI, 2021.","title":"Mgsvf: Multi-grained slow vs. fast framework for few-shot class-incremental learning","authors":"Hanbin Zhao, Yongjian Fu, Mintong Kang, Qi Tian, Fei Wu, Xi Li","year":"2021","venue":"IEEE TPAMI","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhu et al., 2021a] Fei Zhu, Xuyao Zhang, Chuang Wang, Fei Yin, and Chenglin Liu. Prototype augmentation and self-supervision for incremental learning. In CVPR, pages 5871-5880, 2021.","title":"Prototype augmentation and self-supervision for incremental learning","authors":"Zhu","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhu et al., 2021b] Jinguo Zhu, Shixiang Tang, Dapeng Chen, Shijie Yu, Yakun Liu, Mingzhe Rong, Aijun Yang, and Xiaohua Wang. Complementary relation contrastive distillation. In CVPR, pages 9260-9269, 2021.","title":"Complementary relation contrastive distillation","authors":"Zhu","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}