{"type":"ScholarlyArticle","title":"Exploring Safety Supervision for Continual Test-time Domain Adaptation","authors":[{"name":"Xu Yang","slug":"xu-yang","given":"Xu","family":"Yang","affiliation":"Xidian University"},{"name":"Yanan Gu","slug":"yanan-gu","given":"Yanan","family":"Gu","affiliation":"Xidian University"},{"name":"Kun Wei","slug":"kun-wei","given":"Kun","family":"Wei","affiliation":"Xidian University"},{"name":"Cheng Deng","slug":"cheng-deng","given":"Cheng","family":"Deng","affiliation":"Xidian University"}],"abstract":"Continual test-time domain adaptation aims to adapt a source pre-trained model to a continually changing target domain without using any source data. Unfortunately, existing methods based on pseudo-label learning suffer from the changing target domain environment, and the quality of generated pseudo-labels is attenuated due to the domain shift, leading to instantaneous negative learning and long-term knowledge forgetting. To solve these problems, in this paper, we propose a simple yet effective framework for exploring safety supervision with three elaborate strategies: Label Safety, Sample Safety, and Parameter Safety. Firstly, to select reliable pseudo-labels, we define and adjust the confidence threshold in a self-adaptive manner according to the test-time learning status. Secondly, a soft-weighted contrastive learning module is presented to explore the highly-correlated samples and discriminate uncorrelated ones, improving the instantaneous efficiency of the model. Finally, we frame a Soft Weight Alignment strategy to normalize the distance between the parameters of the adapted model and the source pre-trained model, which alleviates the long-term problem of knowledge forgetting and significantly improves the accuracy of the adapted model in the late adaptation stage. Extensive experimental results demonstrate that our method achieves state-of-the-art performance on several benchmark datasets.","doi":"10.24963/ijcai.2023/183","url":"https://ijcai.org/proceedings/2023/183","pdf":"https://ijcai.org/proceedings/2023/0183.pdf","markdown":"https://ijcai.org/proceedings/2023/183.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/183","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Computer Vision","track":"Main Track","number":183,"pages":{"first":1649,"last":1657},"keywords":["Computer Vision: CV: Transfer, low-shot, semi- and un- supervised learning","Computer Vision: CV: Representation learning"],"videos":[],"references":[{"position":1,"raw":"[Chen et al., 2022] Dian Chen, Dequan Wang, Trevor Dar- rell, and Sayna Ebrahimi. Contrastive test-time adaptation. arXiv preprint arXiv:2204.10377, 2022.","title":"Contrastive test-time adaptation","authors":"Dian Chen, Dequan Wang, Trevor Darrell, Sayna Ebrahimi","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Croce et al., 2020] Francesco Croce, Maksym An- driushchenko, Vikash Sehwag, Edoardo Debenedetti, Nicolas Flammarion, Mung Chiang, Prateek Mittal, and Matthias Hein. Robustbench: a standardized adversarial robustness benchmark. arXiv preprint arXiv:2010.09670, 2020.","title":"Robustbench: a standardized adversarial robustness benchmark","authors":"Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Debenedetti, Nicolas Flammarion, Mung Chiang, Prateek Mittal, Matthias Hein","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cui et al., 2020] Shuhao Cui, Shuhui Wang, Junbao Zhuo, Liang Li, Qingming Huang, and Qi Tian. Towards dis- criminability and diversity: Batch nuclear-norm maxi- mization under label insufficient situations. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3941-3950, 2020.","title":"Towards discriminability and diversity: Batch nuclear-norm maximization under label insufficient situations","authors":"Cui","year":"2020","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dong et al., 2020] Jiahua Dong, Yang Cong, Gan Sun, Bi- neng Zhong, and Xiaowei Xu. What can be transferred: Unsupervised domain adaptation for endoscopic lesions segmentation. In IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR), pages 4022-4031, June 2020.","title":"What can be transferred: Unsupervised domain adaptation for endoscopic lesions segmentation","authors":"Jiahua Dong, Yang Cong, Gan Sun, Bineng Zhong, Xiaowei Xu","year":"2020","venue":"IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dong et al., 2021] Jiahua Dong, Yang Cong, Gan Sun, Zhen Fang, and Zhengming Ding. Where and how to transfer: Knowledge aggregation-induced transferability perception for unsupervised domain adaptation. pages 1-1, 2021.","title":"Where and how to transfer: Knowledge aggregation-induced transferability perception for unsupervised domain adaptation","authors":"Jiahua Dong, Yang Cong, Gan Sun, Zhen Fang, Zhengming Ding","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Ganin and Lempitsky, 2015] Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backprop- agation. In International conference on machine learning, pages 1180-1189. PMLR, 2015.","title":"Unsupervised domain adaptation by backpropagation","authors":"Yaroslav Ganin, Victor Lempitsky","year":"2015","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Ganin et al., 2016] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, Franc ¸ois Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. The jour- nal of machine learning research, 17(1):2096-2030, 2016.","title":"Domain-adversarial training of neural networks","authors":"Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, Mario Franc ¸ois Laviolette, Victor Marchand, Lempitsky","year":"2016","venue":"The journal of machine learning research","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Grandvalet and Bengio, 2004] Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimiza- tion. Advances in neural information processing systems, 17, 2004.","title":"Semi-supervised learning by entropy minimization","authors":"Bengio Grandvalet, Yves Grandvalet, Yoshua Bengio","year":"2004","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hendrycks and Dietterich, 2019] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network ro- bustness to common corruptions and perturbations. arXiv preprint arXiv:1903.12261, 2019.","title":"Benchmarking neural network robustness to common corruptions and perturbations","authors":"Dietterich Hendrycks, Dan Hendrycks, Thomas Dietterich","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hendrycks et al., 2019] Dan Hendrycks, Norman Mu, Ekin D Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan. Augmix: A simple data processing method to improve robustness and uncertainty. arXiv preprint arXiv:1912.02781, 2019.","title":"Augmix: A simple data processing method to improve robustness and uncertainty","authors":"Dan Hendrycks, Norman Mu, D Ekin, Barret Cubuk, Justin Zoph, Balaji Gilmer, Lakshminarayanan","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kundu et al., 2020] Jogendra Nath Kundu, Naveen Venkat, R Venkatesh Babu, et al. Universal source-free domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4544- 4553, 2020.","title":"Universal source-free domain adaptation","authors":"Kundu","year":"2020","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kurmi et al., 2021] Vinod K Kurmi, Venkatesh K Subrama- nian, and Vinay P Namboodiri. Domain impression: A source data free domain adaptation method. In Proceed- ings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 615-625, 2021.","title":"Domain impression: A source data free domain adaptation method","authors":"Vinod K Kurmi, K Venkatesh, Subramanian, P Vinay, Namboodiri","year":"2021","venue":"Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lee and others, 2013] Dong-Hyun Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML, volume 3, page 896, 2013.","title":"Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks","authors":"Dong-Hyun Lee","year":"2013","venue":"Workshop on challenges in representation learning, ICML","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li et al., 2016] Yanghao Li, Naiyan Wang, Jianping Shi, Ji- aying Liu, and Xiaodi Hou. Revisiting batch normal- ization for practical domain adaptation. arXiv preprint arXiv:1603.04779, 2016.","title":"Revisiting batch normalization for practical domain adaptation","authors":"Yanghao Li, Naiyan Wang, Jianping Shi, Jiaying Liu, Xiaodi Hou","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2020] Rui Li, Qianfen Jiao, Wenming Cao, Hau- San Wong, and Si Wu. Model adaptation: Unsupervised domain adaptation without source data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 9641-9650, 2020.","title":"Model adaptation: Unsupervised domain adaptation without source data","authors":"Rui Li, Qianfen Jiao, Wenming Cao, Hau-San, Si Wong, Wu","year":"2020","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liang et al., 2020] Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In International Conference on Machine Learning, pages 6028-6039. PMLR, 2020.","title":"Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation","authors":"Jian Liang, Dapeng Hu, Jiashi Feng","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liu et al., 2021] Yuang Liu, Wei Zhang, and Jun Wang. Source-free domain adaptation for semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1215-1224, 2021.","title":"Source-free domain adaptation for semantic segmentation","authors":"Yuang Liu, Wei Zhang, Jun Wang","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lu et al., 2022] Wang Lu, Jindong Wang, Haoliang Li, Yiqiang Chen, and Xing Xie. Domain-invariant feature exploration for domain generalization. arXiv preprint arXiv:2207.12020, 2022.","title":"Domain-invariant feature exploration for domain generalization","authors":"Wang Lu, Jindong Wang, Haoliang Li, Yiqiang Chen, Xing Xie","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Polyak and Juditsky, 1992] Boris T Polyak and Anatoli B Juditsky. Acceleration of stochastic approximation by averaging. SIAM journal on control and optimization, 30(4):838-855, 1992.","title":"Acceleration of stochastic approximation by averaging","authors":"T Boris, Anatoli B Polyak, Juditsky","year":"1992","venue":"SIAM journal on control and optimization","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Prabhu et al., 2021] Viraj Prabhu, Shivam Khare, Deeksha Kartik, and Judy Hoffman. Sentry: Selective entropy opti- mization via committee consistency for unsupervised do- main adaptation. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pages 8558-8567, 2021.","title":"Sentry: Selective entropy optimization via committee consistency for unsupervised domain adaptation","authors":"Viraj Prabhu, Shivam Khare, Deeksha Kartik, Judy Hoffman","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Saito et al., 2019] Kuniaki Saito, Donghyun Kim, Stan Sclaroff, Trevor Darrell, and Kate Saenko. Semi- supervised domain adaptation via minimax entropy. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8050-8058, 2019.","title":"Semisupervised domain adaptation via minimax entropy","authors":"Kuniaki Saito, Donghyun Kim, Stan Sclaroff, Trevor Darrell, Kate Saenko","year":"2019","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Schneider et al., 2020] Steffen Schneider, Evgenia Rusak, Luisa Eck, Oliver Bringmann, Wieland Brendel, and Matthias Bethge. Improving robustness against common corruptions by covariate shift adaptation. Advances in Neural Information Processing Systems, 33:11539-11551, 2020.","title":"Improving robustness against common corruptions by covariate shift adaptation","authors":"Steffen Schneider, Evgenia Rusak, Luisa Eck, Oliver Bringmann, Wieland Brendel, Matthias Bethge","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Sun et al., 2019] Yu Sun, Eric Tzeng, Trevor Darrell, and Alexei A Efros. Unsupervised domain adaptation through self-supervision. arXiv preprint arXiv:1909.11825, 2019.","title":"Unsupervised domain adaptation through self-supervision","authors":"Yu Sun, Eric Tzeng, Trevor Darrell, Alexei A Efros","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Tarvainen and Valpola, 2017] Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight- averaged consistency targets improve semi-supervised deep learning results. Advances in neural information pro- cessing systems, 30, 2017.","title":"Mean teachers are better role models: Weightaveraged consistency targets improve semi-supervised deep learning results","authors":"Valpola Tarvainen, Antti Tarvainen, Harri Valpola","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Tzeng et al., 2017] Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEEE confer- ence on computer vision and pattern recognition, pages 7167-7176, 2017.","title":"Adversarial discriminative domain adaptation","authors":"Eric Tzeng, Judy Hoffman, Kate Saenko, Trevor Darrell","year":"2017","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wang et al., 2020] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. arXiv preprint arXiv:2006.10726, 2020.","title":"Tent: Fully test-time adaptation by entropy minimization","authors":"Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, Trevor Darrell","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wang et al., 2022] Qin Wang, Olga Fink, Luc Van Gool, and Dengxin Dai. Continual test-time domain adaptation. arXiv preprint arXiv:2203.13591, 2022.","title":"Continual test-time domain adaptation","authors":"Qin Wang, Olga Fink, Luc Van Gool, Dengxin Dai","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Xie et al., 2017] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 1492-1500, 2017.","title":"Aggregated residual transformations for deep neural networks","authors":"Xie","year":"2017","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Xu et al., 2021] Qinwei Xu, Ruipeng Zhang, Ya Zhang, Yanfeng Wang, and Qi Tian. A fourier-based frame- work for domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14383-14392, 2021.","title":"A fourier-based framework for domain generalization","authors":"Xu","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Yang et al., 2019] Xu Yang, Cheng Deng, Feng Zheng, Junchi Yan, and Wei Liu. Deep spectral clustering us- ing dual autoencoder network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.","title":"Deep spectral clustering using dual autoencoder network","authors":"Xu Yang, Cheng Deng, Feng Zheng, Junchi Yan, Wei Liu","year":"2019","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Yang et al., 2021] Shiqi Yang, Yaxing Wang, Joost van de Weijer, Luis Herranz, and Shangling Jui. Generalized source-free domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8978-8987, 2021.","title":"Generalized source-free domain adaptation","authors":"Shiqi Yang, Yaxing Wang, Joost Van De Weijer, Luis Herranz, Shangling Jui","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Yang et al., 2022] Xu Yang, Cheng Deng, Tongliang Liu, and Dacheng Tao. Heterogeneous graph attention net- work for unsupervised multiple-target domain adaptation. IEEE Transactions on Pattern Analysis and Machine In- telligence, 44(4):1992-2003, 2022.","title":"Heterogeneous graph attention network for unsupervised multiple-target domain adaptation","authors":"Xu Yang, Cheng Deng, Tongliang Liu, Dacheng Tao","year":"2022","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Yeh et al., 2021] Hao-Wei Yeh, Baoyao Yang, Pong C Yuen, and Tatsuya Harada. Sofa: Source-data-free feature alignment for unsupervised domain adaptation. In Pro- ceedings of the IEEE/CVF Winter Conference on Applica- tions of Computer Vision, pages 474-483, 2021. [Zagoruyko and Komodakis, 2016] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv:1605.07146, 2016.","title":"Sofa: Source-data-free feature alignment for unsupervised domain adaptation","authors":"Hao-Wei Yeh, Baoyao Yang, Pong C Yuen, Tatsuya Harada","year":"2016","venue":"Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zhou and Levine, 2021] Aurick Zhou and Sergey Levine. Training on test data with bayesian adaptation for covari- ate shift. arXiv preprint arXiv:2109.12746, 2021.","title":"Training on test data with bayesian adaptation for covariate shift","authors":"Levine Zhou, Aurick Zhou, Sergey Levine","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}