Screening, Rectifying, and Re-Screening: A Unified Framework for Tuning Vision-Language Models with Noisy Labels

Fang, Chaowei; Ma, Hangfei; Li, Zhihao; Cheng, De; Zhang, Yue; Li, Guanbin

doi:10.24963/ijcai.2025/568

Screening, Rectifying, and Re-Screening: A Unified Framework for Tuning Vision-Language Models with Noisy Labels

Chaowei Fang, Hangfei Ma, Zhihao Li, De Cheng, Yue Zhang, Guanbin Li

Download BibTeX Cite

Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence. Main Track. Pages 5101-5109.

https://doi.org/10.24963/ijcai.2025/568

Keywords

Machine Learning: ML: Cost-sensitive learning
Computer Vision: CV: Machine learning for vision
Machine Learning: ML: Classification
Computer Vision: CV: Transfer, low-shot, semi- and un- supervised learning

Abstract

Pre-trained vision-language models have shown remarkable potential for downstream tasks. However, their fine-tuning under noisy labels remains an open problem due to challenges like self-confirmation bias and the limitations of conventional small-loss criteria. In this paper, we propose a unified framework to address these issues, consisting of three key steps: Screening, Rectifying, and Re-Screening. First, a dual-level semantic matching mechanism is introduced to categorize samples into clean, ambiguous, and noisy samples by leveraging both macro-level and micro-level textual prompts. Second, we design tailored pseudo-labeling strategies to rectify noisy and ambiguous labels, enabling their effective incorporation into the training process. Finally, a re-screening step, utilizing cross-validation with an auxiliary vision-language model, mitigates self-confirmation bias and enhances the robustness of the framework. Extensive experiments across ten datasets demonstrate that the proposed method significantly outperforms existing approaches for tuning vision-language pre-trained models with noisy labels.

Chaowei Fang, Hangfei Ma, Zhihao Li, De Cheng, Yue Zhang, and Guanbin Li. 2025. Screening, Rectifying, and Re-Screening: A Unified Framework for Tuning Vision-Language Models with Noisy Labels. In Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, IJCAI-25. IJCAI, 5101–5109. https://doi.org/10.24963/ijcai.2025/568

@inproceedings{ijcai2025p568,
  title     = {Screening, Rectifying, and Re-Screening: A Unified Framework for Tuning Vision-Language Models with Noisy Labels},
  author    = {Fang, Chaowei and Ma, Hangfei and Li, Zhihao and Cheng, De and Zhang, Yue and Li, Guanbin},
  booktitle = {Proceedings of the Thirty-Fourth International Joint Conference on
               Artificial Intelligence, {IJCAI-25}},
  publisher = {International Joint Conferences on Artificial Intelligence Organization},
  editor    = {James Kwok},
  pages     = {5101--5109},
  year      = {2025},
  month     = {8},
  note      = {Main Track},
  doi       = {10.24963/ijcai.2025/568},
  url       = {https://doi.org/10.24963/ijcai.2025/568},
}

TY  - CPAPER
TI  - Screening, Rectifying, and Re-Screening: A Unified Framework for Tuning Vision-Language Models with Noisy Labels
AU  - Fang, Chaowei
AU  - Ma, Hangfei
AU  - Li, Zhihao
AU  - Cheng, De
AU  - Zhang, Yue
AU  - Li, Guanbin
T2  - Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, IJCAI-25
PB  - International Joint Conferences on Artificial Intelligence Organization
PY  - 2025
SP  - 5101
EP  - 5109
DO  - 10.24963/ijcai.2025/568
UR  - https://doi.org/10.24963/ijcai.2025/568
ER  -