An Empirical Study of Federated Prompt Learning for Vision Language Model

Wang, Zhihao; Huang, Wenke; Chen, Tian; Shi, Zekun; Wan, Guancheng; Qiao, Yu; Yang, Bin; Wang, Jian; Li, Bing; Ye, Mang

doi:10.24963/ijcai.2025/1188

An Empirical Study of Federated Prompt Learning for Vision Language Model

Zhihao Wang, Wenke Huang, Tian Chen, Zekun Shi, Guancheng Wan, Yu Qiao, Bin Yang, Jian Wang, Bing Li, Mang Ye

Download BibTeX Cite

Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence. Survey Track. Pages 10705-10713.

https://doi.org/10.24963/ijcai.2025/1188

Keywords

Computer Vision: CV: Vision, language and reasoning
Machine Learning: ML: Federated learning

Abstract

The Vision Language Model (VLM) excels in aligning vision and language representations, and prompt learning has emerged as a key technique for adapting such models to downstream tasks. However, the application of prompt learning with VLM in federated learning (FL) scenarios remains underexplored. This paper systematically investigates the behavioral differences between language prompt learning (LPT) and vision prompt learning (VPT) under data heterogeneity challenges, including label skew and domain shift. We conduct extensive experiments to evaluate the impact of various FL and prompt configurations, such as client scale, aggregation strategies, and prompt length, to assess the robustness of Federated Prompt Learning (FPL). Furthermore, we explore strategies for enhancing prompt learning in complex scenarios where label skew and domain shift coexist, including leveraging both prompt types when computational resources allow. Our findings offer practical insights into optimizing prompt learning in federated settings, contributing to the broader deployment of VLMs in privacy-preserving environments.

Zhihao Wang, Wenke Huang, Tian Chen, Zekun Shi, Guancheng Wan, Yu Qiao, Bin Yang, Jian Wang, Bing Li, and Mang Ye. 2025. An Empirical Study of Federated Prompt Learning for Vision Language Model. In Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, IJCAI-25. IJCAI, 10705–10713. https://doi.org/10.24963/ijcai.2025/1188

@inproceedings{ijcai2025p1188,
  title     = {An Empirical Study of Federated Prompt Learning for Vision Language Model},
  author    = {Wang, Zhihao and Huang, Wenke and Chen, Tian and Shi, Zekun and Wan, Guancheng and Qiao, Yu and Yang, Bin and Wang, Jian and Li, Bing and Ye, Mang},
  booktitle = {Proceedings of the Thirty-Fourth International Joint Conference on
               Artificial Intelligence, {IJCAI-25}},
  publisher = {International Joint Conferences on Artificial Intelligence Organization},
  editor    = {James Kwok},
  pages     = {10705--10713},
  year      = {2025},
  month     = {8},
  note      = {Survey Track},
  doi       = {10.24963/ijcai.2025/1188},
  url       = {https://doi.org/10.24963/ijcai.2025/1188},
}

TY  - CPAPER
TI  - An Empirical Study of Federated Prompt Learning for Vision Language Model
AU  - Wang, Zhihao
AU  - Huang, Wenke
AU  - Chen, Tian
AU  - Shi, Zekun
AU  - Wan, Guancheng
AU  - Qiao, Yu
AU  - Yang, Bin
AU  - Wang, Jian
AU  - Li, Bing
AU  - Ye, Mang
T2  - Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, IJCAI-25
PB  - International Joint Conferences on Artificial Intelligence Organization
PY  - 2025
SP  - 10705
EP  - 10713
DO  - 10.24963/ijcai.2025/1188
UR  - https://doi.org/10.24963/ijcai.2025/1188
ER  -