{"type":"ScholarlyArticle","title":"Towards Dynamic-Prompting Collaboration for Source-Free Domain Adaptation","authors":[{"name":"Mengmeng Zhan","slug":"mengmeng-zhan","given":"Mengmeng","family":"Zhan","affiliation":"School of Computer Science and Engineering , University of Electronic Science and Technology of China"},{"name":"Zongqian Wu","slug":"zongqian-wu","given":"Zongqian","family":"Wu","affiliation":"School of Computer Science and Engineering , University of Electronic Science and Technology of China"},{"name":"Rongyao Hu","slug":"rongyao-hu","given":"Rongyao","family":"Hu","affiliation":"School of Computer Science and Engineering , University of Electronic Science and Technology of China"},{"name":"Ping Hu","slug":"ping-hu","given":"Ping","family":"Hu","affiliation":"School of Computer Science and Engineering , University of Electronic Science and Technology of China"},{"name":"Heng Tao Shen","slug":"heng-tao-shen","given":"Heng Tao","family":"Shen","affiliation":"School of Computer Science and Engineering , University of Electronic Science and Technology of China"},{"name":"Xiaofeng Zhu","slug":"xiaofeng-zhu","given":"Xiaofeng","family":"Zhu","affiliation":"School of Computer Science and Engineering , University of Electronic Science and Technology of China"}],"abstract":"In domain adaptation, challenges such as data privacy constraints can impede access to source data, catalyzing the development of source-free domain adaptation (SFDA) methods. However, current approaches heavily rely on models trained on source data, posing the risk of overfitting and suboptimal generalization.This paper introduces a dynamic prompt learning paradigm that harnesses the power of large-scale vision-language models to enhance the semantic transfer of source models. Specifically, our approach fosters robust and adaptive collaboration between the source-trained model and the vision-language model, facilitating the reliable extraction of domain-specific information from unlabeled target data, while consolidating domain-invariant knowledge. Without the need for accessing source data, our method amalgamates the strengths inherent in both traditional SFDA approaches and vision-language models, formulating a collaborative framework for addressing SFDA challenges. Extensive experiments conducted on three benchmark datasets showcase the superiority of our framework over previous SOTA methods.","doi":"10.24963/ijcai.2024/182","url":"https://ijcai.org/proceedings/2024/182","pdf":"https://ijcai.org/proceedings/2024/0182.pdf","markdown":"https://ijcai.org/proceedings/2024/182.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/182","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Computer Vision","track":"Main Track","number":182,"pages":{"first":1643,"last":1651},"keywords":["Computer Vision: CV: Multimodal learning","Computer Vision: CV: Transfer, low-shot, semi- and un- supervised learning","Computer Vision: CV: Representation learning"],"videos":[],"references":[{"position":1,"raw":"[Chen et al., 2022] Dian Chen, Dequan Wang, Trevor Dar- rell, and Sayna Ebrahimi. Contrastive test-time adaptation. In CVPR, pages 295-305, 2022.","title":"Contrastive test-time adaptation","authors":"Dian Chen, Dequan Wang, Trevor Darrell, Sayna Ebrahimi","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Dosovitskiy et al., 2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Min- derer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.","title":"An image is worth 16x16 words: Transformers for image recognition at scale","authors":"Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Fahes et al., 2023] Mohammad Fahes, Tuan-Hung Vu, An- drei Bursuc, Patrick Pérez, and Raoul de Charette. Poda: Prompt-driven zero-shot domain adaptation. In ICCV, pages 18623-18633, 2023.","title":"Poda: Prompt-driven zero-shot domain adaptation","authors":"Mohammad Fahes, Tuan-Hung Vu, Andrei Bursuc, Patrick Pérez, Raoul De Charette","year":"2023","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Fang et al., 2022] Yuqi Fang, Pew-Thian Yap, Weili Lin, Hongtu Zhu, and Mingxia Liu. Source-free unsuper- vised domain adaptation: A survey. arXiv preprint arXiv:2301.00265, 2022.","title":"Source-free unsupervised domain adaptation: A survey","authors":"Yuqi Fang, Pew-Thian, Weili Yap, Hongtu Lin, Mingxia Zhu, Liu","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Ge et al., 2023] Chunjiang Ge, Rui Huang, Mixue Xie, Zi- hang Lai, Shiji Song, Shuang Li, and Gao Huang. Do- main adaptation via prompt learning. IEEE Transactions on Neural Networks and Learning Systems, 2023.","title":"Domain adaptation via prompt learning","authors":"Chunjiang Ge, Rui Huang, Mixue Xie, Zihang Lai, Shiji Song, Shuang Li, Gao Huang","year":"2023","venue":"IEEE Transactions on Neural Networks and Learning Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Huang et al., 2021] Jiaxing Huang, Dayan Guan, Aoran Xiao, and Shijian Lu. Model adaptation: Historical con- trastive learning for unsupervised domain adaptation with- out source data. In NeurIPS, volume 34, pages 3635-3649, 2021.","title":"Model adaptation: Historical contrastive learning for unsupervised domain adaptation without source data","authors":"Jiaxing Huang, Dayan Guan, Aoran Xiao, Shijian Lu","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Jia et al., 2022] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In ECCV, 2022.","title":"Visual prompt tuning","authors":"Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, Ser-Nam Lim","year":"2022","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Jiang et al., 2024] Xin Jiang, Hao Tang, Junyao Gao, Xi- aoyu Du, Shengfeng He, and Zechao Li. Delving into mul- timodal prompting for fine-grained visual classification. In AAAI, pages 2570-2578, 2024.","title":"Delving into multimodal prompting for fine-grained visual classification","authors":"Xin Jiang, Hao Tang, Junyao Gao, Xiaoyu Du, Shengfeng He, Zechao Li","year":"2024","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Lai et al., 2023] Zhengfeng Lai, Noranart Vesdapunt, Ning Zhou, Jun Wu, Cong Phuoc Huynh, Xuelu Li, Kah Kuen Fu, and Chen-Nee Chuah. Padclip: Pseudo-labeling with adaptive debiasing in clip for unsupervised domain adap- tation. In ICCV, pages 16155-16165, 2023.","title":"Padclip: Pseudo-labeling with adaptive debiasing in clip for unsupervised domain adaptation","authors":"Lai","year":"2023","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lee et al., 2022] Jonghyun Lee, Dahuin Jung, Junho Yim, and Sungroh Yoon. Confidence score for source-free un- supervised domain adaptation. In ICML, pages 12365- 12377, 2022.","title":"Confidence score for source-free unsupervised domain adaptation","authors":"Lee","year":"2022","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2020] Rui Li, Qianfen Jiao, Wenming Cao, Hau- San Wong, and Si Wu. Model adaptation: Unsupervised domain adaptation without source data. In CVPR, pages 9641-9650, 2020.","title":"Model adaptation: Unsupervised domain adaptation without source data","authors":"Rui Li, Qianfen Jiao, Wenming Cao, Hau-San, Si Wong, Wu","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Liang et al., 2020] Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? source hy- pothesis transfer for unsupervised domain adaptation. In ICML, pages 6028-6039, 2020.","title":"Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation","authors":"Jian Liang, Dapeng Hu, Jiashi Feng","year":"2020","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Litrico et al., 2023] Mattia Litrico, Alessio Del Bue, and Pietro Morerio. Guiding pseudo-labels with uncertainty estimation for source-free unsupervised domain adapta- tion. In CVPR, pages 7640-7650, 2023.","title":"Guiding pseudo-labels with uncertainty estimation for source-free unsupervised domain adaptation","authors":"Mattia Litrico, Alessio Del Bue, Pietro Morerio","year":"2023","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Nejjar et al., 2023] Ismail Nejjar, Qin Wang, and Olga Fink. Dare-gram: Unsupervised domain adaptation regression by aligning inverse gram matrices. In CVPR, pages 11744- 11754, 2023.","title":"Dare-gram: Unsupervised domain adaptation regression by aligning inverse gram matrices","authors":"Nejjar","year":"2023","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Peng et al., 2019] Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment match- ing for multi-source domain adaptation. In ICCV, pages 1406-1415, 2019.","title":"Moment matching for multi-source domain adaptation","authors":"Peng","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Permuter et al., 2006] Haim Permuter, Joseph Francos, and Ian Jermyn. A study of gaussian mixture models of color and texture features for image classification and segmen- tation. Pattern recognition, 39(4):695-706, 2006.","title":"A study of gaussian mixture models of color and texture features for image classification and segmentation","authors":"Haim Permuter, Joseph Francos, Ian Jermyn","year":"2006","venue":"Pattern recognition","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Qu et al., 2022] Sanqing Qu, Guang Chen, Jing Zhang, Zhi- jun Li, Wei He, and Dacheng Tao. Bmd: A general class-balanced multicentric dynamic prototype strategy for source-free domain adaptation. In ECCV, pages 165-182, 2022.","title":"Bmd: A general class-balanced multicentric dynamic prototype strategy for source-free domain adaptation","authors":"Sanqing Qu, Guang Chen, Jing Zhang, Zhijun Li, Wei He, Dacheng Tao","year":"2022","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Radford et al., 2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from nat- ural language supervision. In ICML, pages 8748-8763, 2021.","title":"Learning transferable visual models from natural language supervision","authors":"Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark","year":"2021","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Roy et al., 2022] Subhankar Roy, Martin Trapp, Andrea Pilzer, Juho Kannala, Nicu Sebe, Elisa Ricci, and Arno Solin. Uncertainty-guided source-free domain adaptation. In ECCV, pages 537-555, 2022.","title":"Uncertainty-guided source-free domain adaptation","authors":"Subhankar Roy, Martin Trapp, Andrea Pilzer, Juho Kannala, Nicu Sebe, Elisa Ricci, Arno Solin","year":"2022","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Saenko et al., 2010] Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In ECCV, pages 213-226, 2010.","title":"Adapting visual category models to new domains","authors":"Saenko","year":"2010","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Sanyal et al., 2023] Sunandini Sanyal, Ashish Ramayee Asokan, Suvaansh Bhambri, Akshay Kulkarni, Jogen- dra Nath Kundu, and R Venkatesh Babu. Domain- specificity inducing transformers for source-free domain adaptation. In ICCV, pages 18928-18937, 2023.","title":"Domainspecificity inducing transformers for source-free domain adaptation","authors":"Sanyal","year":"2023","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Selvaraju et al., 2017] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In ICCV, pages 618-626, 2017.","title":"Grad-cam: Visual explanations from deep networks via gradient-based localization","authors":"Michael Ramprasaath R Selvaraju, Abhishek Cogswell, Ramakrishna Das, Devi Vedantam, Dhruv Parikh, Batra","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Singha et al., 2023] Mainak Singha, Harsh Pal, Ankit Jha, and Biplab Banerjee. Ad-clip: Adapting domains in prompt space using clip. In ICCV, pages 4355-4364, 2023.","title":"Ad-clip: Adapting domains in prompt space using clip","authors":"Mainak Singha, Harsh Pal, Ankit Jha, Biplab Banerjee","year":"2023","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Tang et al., 2020] Hao Tang, Zechao Li, Zhimao Peng, and Jinhui Tang. Blockmix: meta regularization and self- calibrated inference for metric-based meta-learning. In ACMMM, pages 610-618, 2020.","title":"Blockmix: meta regularization and selfcalibrated inference for metric-based meta-learning","authors":"Hao Tang, Zechao Li, Zhimao Peng, Jinhui Tang","year":"2020","venue":"ACMMM","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Tang et al., 2023] Longxiang Tang, Kai Li, Chunming He, Yulun Zhang, and Xiu Li. Consistency regularization for generalizable source-free domain adaptation. In ICCV, pages 4323-4333, 2023.","title":"Consistency regularization for generalizable source-free domain adaptation","authors":"Longxiang Tang, Kai Li, Chunming He, Yulun Zhang, Xiu Li","year":"2023","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Venkateswara et al., 2017] Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Pan- chanathan. Deep hashing network for unsupervised do- main adaptation. In CVPR, pages 5018-5027, 2017.","title":"Deep hashing network for unsupervised domain adaptation","authors":"Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, Sethuraman Pan-Chanathan","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wang et al., 2022a] Fan Wang, Zhongyi Han, Yongshun Gong, and Yilong Yin. Exploring domain-invariant param- eters for source free domain adaptation. In CVPR, pages 7151-7160, 2022.","title":"Exploring domain-invariant parameters for source free domain adaptation","authors":"Wang","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wang et al., 2022b] Zengmao Wang, Chaoyang Zhou, Bo Du, and Fengxiang He. Self-paced supervision for multi-source domain adaptation. In IJCAI, 2022.","title":"Self-paced supervision for multi-source domain adaptation","authors":"Wang","year":"2022","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2022/493"},{"position":29,"raw":"[Xu et al., 2021] Tongkun Xu, Weihua Chen, WANG Pichao, Fan Wang, Hao Li, and Rong Jin. Cdtrans: Cross- domain transformer for unsupervised domain adaptation. In ICLR, 2021.","title":"Cdtrans: Crossdomain transformer for unsupervised domain adaptation","authors":"Xu","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Yang et al., 2021] Shiqi Yang, Joost van de Weijer, Luis Herranz, Shangling Jui, et al. Exploiting the intrinsic neighborhood structure for source-free domain adaptation. In NeurIPS, volume 34, pages 29393-29405, 2021.","title":"Exploiting the intrinsic neighborhood structure for source-free domain adaptation","authors":"Shiqi Yang, Joost Van De Weijer, Luis Herranz, Shangling Jui","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Yang et al., 2022] Shiqi Yang, Shangling Jui, Joost van de Weijer, et al. Attracting and dispersing: A simple approach for source-free domain adaptation. In NeurIPS, volume 35, pages 5802-5815, 2022.","title":"Attracting and dispersing: A simple approach for source-free domain adaptation","authors":"Shiqi Yang, Shangling Jui, Joost Van De Weijer","year":"2022","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Yang et al., 2023] Jinyu Yang, Jingjing Liu, Ning Xu, and Junzhou Huang. Tvt: Transferable vision transformer for unsupervised domain adaptation. In ICCV, pages 520- 530, 2023.","title":"Tvt: Transferable vision transformer for unsupervised domain adaptation","authors":"Jinyu Yang, Jingjing Liu, Ning Xu, Junzhou Huang","year":"2023","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Yu et al., 2023] Zhiqi Yu, Jingjing Li, Zhekai Du, Lei Zhu, and Heng Tao Shen. A comprehensive survey on source- free domain adaptation. arXiv preprint arXiv:2302.11803, 2023.","title":"A comprehensive survey on sourcefree domain adaptation","authors":"Yu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zara et al., 2023] Giacomo Zara, Subhankar Roy, Paolo Rota, and Elisa Ricci. Autolabel: Clip-based framework for open-set video domain adaptation. In CVPR, pages 11504-11513, 2023.","title":"Autolabel: Clip-based framework for open-set video domain adaptation","authors":"Giacomo Zara, Subhankar Roy, Paolo Rota, Elisa Ricci","year":"2023","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Zhang et al., 2023a] Wenyu Zhang, Li Shen, and Chuan- Sheng Foo. Rethinking the role of pre-trained networks in source-free domain adaptation. In ICCV, pages 18841- 18851, 2023.","title":"Rethinking the role of pre-trained networks in source-free domain adaptation","authors":"Zhang","year":"2023","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Zhang et al., 2023b] Yixin Zhang, Zilei Wang, and Weinan He. Class relationship embedded learning for source-free unsupervised domain adaptation. In CVPR, pages 7619- 7629, 2023.","title":"Class relationship embedded learning for source-free unsupervised domain adaptation","authors":"Zhang","year":"2023","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Zhou et al., 2022a] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In CVPR, pages 16816-16825, 2022.","title":"Conditional prompt learning for vision-language models","authors":"Zhou","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Zhou et al., 2022b] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 130(9):2337-2348, 2022.","title":"Learning to prompt for vision-language models","authors":"Zhou","year":"2022","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Zinkevich et al., 2010] Martin Zinkevich, Markus Weimer, Lihong Li, and Alex Smola. Parallelized stochastic gradi- ent descent. In NeurIPS, volume 23, 2010.","title":"Parallelized stochastic gradient descent","authors":"Martin Zinkevich, Markus Weimer, Lihong Li, Alex Smola","year":"2010","venue":"NeurIPS","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}