{"type":"ScholarlyArticle","title":"Navigating Continual Test-time Adaptation with Symbiosis Knowledge","authors":[{"name":"Xu Yang","slug":"xu-yang","given":"Xu","family":"Yang","affiliation":"Xidian University"},{"name":"Moqi Li","slug":"moqi-li","given":"Moqi","family":"Li","affiliation":"Xidian University"},{"name":"Jie Yin","slug":"jie-yin","given":"Jie","family":"Yin","affiliation":"Xidian University"},{"name":"Kun Wei","slug":"kun-wei","given":"Kun","family":"Wei","affiliation":"Xidian University"},{"name":"Cheng Deng","slug":"cheng-deng","given":"Cheng","family":"Deng","affiliation":"Xidian University"}],"abstract":"Continual test-time domain adaptation seeks to adapt the source pre-trained model to a continually changing target domain without incurring additional data acquisition or labeling costs. Unfortunately, existing mainstream methods may result in a detrimental cycle. This is attributed to noisy pseudo-labels caused by the domain shift, which immediately negatively impacts the model's knowledge. The long-term accumulation of these negative effects exacerbates the model's difficulty in generalizing to future domain shifts and contributes to catastrophic forgetting. To address these challenges, this paper introduces a Dual-stream Network that independently optimizes different parameters in each stream to capture symbiotic knowledge from continual domains, thereby ensuring generalization while enhancing instantaneous discrimination. Furthermore, to prevent catastrophic forgetting, a weighted soft parameter alignment method is designed to leverage knowledge from the source model. Finally, efforts are made to calibrate and explore reliable supervision signals to mitigate instantaneous negative optimization. These include label calibration with prior knowledge, label selection using self-adaptive confidence thresholds, and a soft-weighted contrastive module for capturing potential semantics. Extensive experimental results demonstrate that our method achieves state-of-the-art performance on several benchmark datasets.","doi":"10.24963/ijcai.2024/589","url":"https://ijcai.org/proceedings/2024/589","pdf":"https://ijcai.org/proceedings/2024/0589.pdf","markdown":"https://ijcai.org/proceedings/2024/589.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/589","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Machine Learning","track":"Main Track","number":589,"pages":{"first":5326,"last":5334},"keywords":["Computer Vision: CV: Transfer, low-shot, semi- and un- supervised learning","Machine Learning: ML: Unsupervised learning"],"videos":[],"references":[{"position":1,"raw":"[Bai et al., 2021] Yingbin Bai, Erkun Yang, Bo Han, Yanhua Yang, Jiatong Li, Yinian Mao, Gang Niu, and Tongliang Liu. Understanding and improving early stopping for learning with noisy labels. NeurIPS, 34:24392-24403, 2021.","title":"Understanding and improving early stopping for learning with noisy labels","authors":"Yingbin Bai, Erkun Yang, Bo Han, Yanhua Yang, Jiatong Li, Yinian Mao, Gang Niu, Tongliang Liu","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cao et al., 2022] Zhangjie Cao, Kaichao You, Ziyang Zhang, Jianmin Wang, and Mingsheng Long. From big to small: adaptive learning to partial-set domains. IEEE Trans. Pattern Anal. Mach. Intell., 45(2):1766-1780, 2022.","title":"From big to small: adaptive learning to partial-set domains","authors":"Cao","year":"2022","venue":"IEEE Trans. Pattern Anal. Mach. Intell","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2022] Dian Chen, Dequan Wang, Trevor Dar- rell, and Sayna Ebrahimi. Contrastive test-time adaptation. arXiv preprint arXiv:2204.10377, 2022.","title":"Contrastive test-time adaptation","authors":"Dian Chen, Dequan Wang, Trevor Darrell, Sayna Ebrahimi","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Croce et al., 2020] Francesco Croce, Maksym An- driushchenko, Vikash Sehwag, Edoardo Debenedetti, Nicolas Flammarion, Mung Chiang, Prateek Mittal, and Matthias Hein. Robustbench: a standardized adversarial robustness benchmark. arXiv preprint arXiv:2010.09670, 2020.","title":"Robustbench: a standardized adversarial robustness benchmark","authors":"Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Debenedetti, Nicolas Flammarion, Mung Chiang, Prateek Mittal, Matthias Hein","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cui et al., 2020] Shuhao Cui, Shuhui Wang, Junbao Zhuo, Liang Li, Qingming Huang, and Qi Tian. Towards dis- criminability and diversity: Batch nuclear-norm maxi- mization under label insufficient situations. In CVPR, pages 3941-3950, 2020.","title":"Towards discriminability and diversity: Batch nuclear-norm maximization under label insufficient situations","authors":"Cui","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Ding et al., 2023] Yifei Ding, Minping Jia, Jichao Zhuang, Yudong Cao, Xiaoli Zhao, and Chi-Guhn Lee. Deep im- balanced domain adaptation for transfer learning fault di- agnosis of bearings under multiple working conditions. Reliability Engineering & System Safety, 230:108890, 2023.","title":"Deep imbalanced domain adaptation for transfer learning fault diagnosis of bearings under multiple working conditions","authors":"Yifei Ding, Minping Jia, Jichao Zhuang, Yudong Cao, Xiaoli Zhao, Chi-Guhn Lee","year":"2023","venue":"Reliability Engineering & System Safety","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Döbler et al., 2023] Mario Döbler, Robert A Marsden, and Bin Yang. Robust mean teacher for continual and gradual test-time adaptation. In CVPR, pages 7704-7714, 2023.","title":"Robust mean teacher for continual and gradual test-time adaptation","authors":"Mario Döbler, Robert A Marsden, Bin Yang","year":"2023","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Frankle et al., 2020] Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In ICML, pages 3259-3269. PMLR, 2020.","title":"Linear mode connectivity and the lottery ticket hypothesis","authors":"Jonathan Frankle, Karolina Gintare, Daniel Dziugaite, Michael Roy, Carbin","year":"2020","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gan et al., 2022] Yulu Gan, Xianzheng Ma, Yihang Lou, Yan Bai, Renrui Zhang, Nian Shi, and Lin Luo. Decorate the newcomers: Visual domain prompt for continual test time adaptation. arXiv preprint arXiv:2212.04145, 2022.","title":"Decorate the newcomers: Visual domain prompt for continual test time adaptation","authors":"Yulu Gan, Xianzheng Ma, Yihang Lou, Yan Bai, Renrui Zhang, Nian Shi, Lin Luo","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ganin and Lempitsky, 2015] Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backprop- agation. In ICML, pages 1180-1189. PMLR, 2015.","title":"Unsupervised domain adaptation by backpropagation","authors":"Yaroslav Ganin, Victor Lempitsky","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Ganin et al., 2016] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, Franc ¸ois Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. Journal of Machine Learning Research, 17(1):2096-2030, 2016.","title":"Domain-adversarial training of neural networks","authors":"Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, Mario Franc ¸ois Laviolette, Victor Marchand, Lempitsky","year":"2016","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"Gong et al., ] Taesik Gong, Jongheon Jeong, Taewon Kim, Yewon Kim, Jinwoo Shin, and Sung-Ju Lee. Note: Ro- bust continual test-time adaptation against temporal corre- lation. In NeurIPS. [Hendrycks and Dietterich, 2019] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network ro- bustness to common corruptions and perturbations. arXiv preprint arXiv:1903.12261, 2019.","title":"Note: Robust continual test-time adaptation against temporal correlation","authors":"Gong","year":"2019","venue":"Benchmarking neural network robustness to common corruptions and perturbations","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hendrycks et al., 2019] Dan Hendrycks, Norman Mu, Ekin D Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan. Augmix: A simple data processing method to improve robustness and uncertainty. arXiv preprint arXiv:1912.02781, 2019.","title":"Augmix: A simple data processing method to improve robustness and uncertainty","authors":"Dan Hendrycks, Norman Mu, D Ekin, Barret Cubuk, Justin Zoph, Balaji Gilmer, Lakshminarayanan","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Jiang et al., 2021] Junguang Jiang, Baixu Chen, Jianmin Wang, and Mingsheng Long. Decoupled adaptation for cross-domain object detection. arXiv preprint arXiv:2110.02578, 2021.","title":"Decoupled adaptation for cross-domain object detection","authors":"Jiang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lee and others, 2013] Dong-Hyun Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In ICMLW, volume 3, page 896, 2013.","title":"Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks","authors":"Dong-Hyun Lee","year":"2013","venue":"ICMLW","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Li et al., 2016] Yanghao Li, Naiyan Wang, Jianping Shi, Ji- aying Liu, and Xiaodi Hou. Revisiting batch normal- ization for practical domain adaptation. arXiv preprint arXiv:1603.04779, 2016.","title":"Revisiting batch normalization for practical domain adaptation","authors":"Yanghao Li, Naiyan Wang, Jianping Shi, Jiaying Liu, Xiaodi Hou","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Li et al., 2020] Rui Li, Qianfen Jiao, Wenming Cao, Hau- San Wong, and Si Wu. Model adaptation: Unsupervised domain adaptation without source data. In CVPR, pages 9641-9650, 2020.","title":"Model adaptation: Unsupervised domain adaptation without source data","authors":"Rui Li, Qianfen Jiao, Wenming Cao, Hau-San, Si Wong, Wu","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Liang et al., 2020] Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? source hy- pothesis transfer for unsupervised domain adaptation. In ICML, pages 6028-6039. PMLR, 2020.","title":"Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation","authors":"Jian Liang, Dapeng Hu, Jiashi Feng","year":"2020","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Liu et al., 2021] Yuang Liu, Wei Zhang, and Jun Wang. Source-free domain adaptation for semantic segmentation. In CVPR, pages 1215-1224, 2021.","title":"Source-free domain adaptation for semantic segmentation","authors":"Yuang Liu, Wei Zhang, Jun Wang","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Marsden et al., 2023] Robert A Marsden, Mario Döbler, and Bin Yang. Universal test-time adaptation through weight ensembling, diversity weighting, and prior correc- tion. arXiv preprint arXiv:2306.00650, 2023.","title":"Universal test-time adaptation through weight ensembling, diversity weighting, and prior correction","authors":"Mario Robert A Marsden, Bin Döbler, Yang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Prabhu et al., 2021] Viraj Prabhu, Shivam Khare, Deeksha Kartik, and Judy Hoffman. Sentry: Selective entropy opti- mization via committee consistency for unsupervised do- main adaptation. In ICCV, pages 8558-8567, 2021.","title":"Sentry: Selective entropy optimization via committee consistency for unsupervised domain adaptation","authors":"Viraj Prabhu, Shivam Khare, Deeksha Kartik, Judy Hoffman","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Saito et al., 2019] Kuniaki Saito, Donghyun Kim, Stan Sclaroff, Trevor Darrell, and Kate Saenko. Semi- supervised domain adaptation via minimax entropy. In ICCV, pages 8050-8058, 2019.","title":"Semisupervised domain adaptation via minimax entropy","authors":"Kuniaki Saito, Donghyun Kim, Stan Sclaroff, Trevor Darrell, Kate Saenko","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Schneider et al., 2020] Steffen Schneider, Evgenia Rusak, Luisa Eck, Oliver Bringmann, Wieland Brendel, and Matthias Bethge. Improving robustness against com- mon corruptions by covariate shift adaptation. NeurIPS, 33:11539-11551, 2020.","title":"Improving robustness against common corruptions by covariate shift adaptation","authors":"Steffen Schneider, Evgenia Rusak, Luisa Eck, Oliver Bringmann, Wieland Brendel, Matthias Bethge","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Sun et al., 2022] Tao Sun, Cheng Lu, and Haibin Ling. Prior knowledge guided unsupervised domain adaptation. In ECCV, pages 639-655. Springer, 2022.","title":"Prior knowledge guided unsupervised domain adaptation","authors":"Tao Sun, Cheng Lu, Haibin Ling","year":"2022","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Tzeng et al., 2017] Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In CVPR, pages 7167-7176, 2017.","title":"Adversarial discriminative domain adaptation","authors":"Eric Tzeng, Judy Hoffman, Kate Saenko, Trevor Darrell","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wang et al., 2020] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. arXiv preprint arXiv:2006.10726, 2020.","title":"Tent: Fully test-time adaptation by entropy minimization","authors":"Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, Trevor Darrell","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wang et al., 2022] Qin Wang, Olga Fink, Luc Van Gool, and Dengxin Dai. Continual test-time domain adaptation. arXiv preprint arXiv:2203.13591, 2022.","title":"Continual test-time domain adaptation","authors":"Qin Wang, Olga Fink, Luc Van Gool, Dengxin Dai","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wang et al., 2023a] Mengzhu Wang, Shanshan Wang, Wei Wang, Li Shen, Xiang Zhang, Long Lan, and Zhigang Luo. Reducing bi-level feature redundancy for unsuper- vised domain adaptation. Pattern Recognit., page 109319, 2023.","title":"Reducing bi-level feature redundancy for unsupervised domain adaptation","authors":"Wang","year":"2023","venue":"Pattern Recognit","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wang et al., 2023b] Xu Wang, Dezhong Peng, Peng Hu, Yunhong Gong, and Yong Chen. Cross-domain alignment for zero-shot sketch-based image retrieval. IEEE Trans. Circuits Syst. Video Technol., 2023.","title":"Cross-domain alignment for zero-shot sketch-based image retrieval","authors":"Wang","year":"2023","venue":"IEEE Trans. Circuits Syst. Video Technol","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wang et al., 2023c] Xu Wang, Dezhong Peng, Ming Yan, and Peng Hu. Correspondence-free domain alignment for unsupervised cross-domain image retrieval. arXiv preprint arXiv:2302.06081, 2023.","title":"Correspondence-free domain alignment for unsupervised cross-domain image retrieval","authors":"Wang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Wolsey, 2020] Laurence A Wolsey. Integer programming. John Wiley & Sons, 2020.","title":"Integer programming","authors":"Laurence A Wolsey","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Xie et al., 2017] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. In CVPR, pages 1492-1500, 2017.","title":"Aggregated residual transformations for deep neural networks","authors":"Xie","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Xie et al., 2022] Binhui Xie, Longhui Yuan, Shuang Li, Chi Harold Liu, Xinjing Cheng, and Guoren Wang. Ac- tive learning for domain adaptation: An energy-based ap- proach. In AAAI, volume 36, pages 8708-8716, 2022.","title":"Active learning for domain adaptation: An energy-based approach","authors":"Xie","year":"2022","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Yang et al., 2021] Shiqi Yang, Yaxing Wang, Joost van de Weijer, Luis Herranz, and Shangling Jui. Generalized source-free domain adaptation. In CVPR, pages 8978- 8987, 2021.","title":"Generalized source-free domain adaptation","authors":"Shiqi Yang, Yaxing Wang, Joost Van De Weijer, Luis Herranz, Shangling Jui","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Yang et al., 2022] Xu Yang, Cheng Deng, Tongliang Liu, and Dacheng Tao. Heterogeneous graph attention net- work for unsupervised multiple-target domain adaptation. IEEE Trans. Pattern Anal. Mach. Intell., 44(4):1992-2003, 2022.","title":"Heterogeneous graph attention network for unsupervised multiple-target domain adaptation","authors":"Xu Yang, Cheng Deng, Tongliang Liu, Dacheng Tao","year":"2022","venue":"IEEE Trans. Pattern Anal. Mach. Intell","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Yang et al., 2023] Xu Yang, Yanan Gu, Kun Wei, and Cheng Deng. Exploring safety supervision for continual test-time domain adaptation. In IJCAI, pages 1649-1657, 2023.","title":"Exploring safety supervision for continual test-time domain adaptation","authors":"Xu Yang, Yanan Gu, Kun Wei, Cheng Deng","year":"2023","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2023/183"},{"position":37,"raw":"[Yuan et al., 2023] Longhui Yuan, Binhui Xie, and Shuang Li. Robust test-time adaptation in dynamic scenarios. In CVPR, pages 15922-15932, 2023.","title":"Robust test-time adaptation in dynamic scenarios","authors":"Yuan","year":"2023","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Zagoruyko and Komodakis, 2016] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv:1605.07146, 2016.","title":"Wide residual networks","authors":"Sergey Zagoruyko, Nikos Komodakis","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}