Fine-grained Prompt Screening: Defending Against Backdoor Attack on Text-to-Image Diffusion Models

Xu, Yiran; Zhong, Nan; Li, Guobiao; Cheng, Anda; Wang, Yinggui; Qian, Zhenxing; Zhang, Xinpeng

doi:10.24963/ijcai.2025/68

Fine-grained Prompt Screening: Defending Against Backdoor Attack on Text-to-Image Diffusion Models

Yiran Xu, Nan Zhong, Guobiao Li, Anda Cheng, Yinggui Wang, Zhenxing Qian, Xinpeng Zhang

Download BibTeX Cite

Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence. Main Track. Pages 601-609.

https://doi.org/10.24963/ijcai.2025/68

Keywords

AI Ethics, Trust, Fairness: ETF: Trustworthy AI
AI Ethics, Trust, Fairness: ETF: Safety and robustness

Abstract

Text-to-image (T2I) diffusion models exhibit impressive generation capabilities in recently studies. However, they are vulnerable to backdoor attacks, where model outputs are manipulated by malicious triggers. In this paper, we propose a novel input-level defense method, called Fine-grained Prompt Screening (GrainPS). Our method is motivated by the phenomenon, i.e., Semantics Misalignment, where the backdoor trigger causes the inconsistency between the cross-attention projections of object words (the key words to determine the main content of the generated image) and their true semantics. In particular, we divide each prompt into pieces and conduct fine-grained analysis by examining the impact of the trigger on object words in the cross-attention layers rather than their global influence on the entire generated image. To assess the impact of each word on object words, we formulate "semantics alignment score'' as the metric with a carefully crafted detection strategy to identify the trigger. Therefore, our implementation can detect backdoor input prompts and localize of triggers simultaneously. Evaluations across four advanced backdoor attack scenarios demonstrate the effectiveness of our proposed defense method.

Yiran Xu, Nan Zhong, Guobiao Li, Anda Cheng, Yinggui Wang, Zhenxing Qian, and Xinpeng Zhang. 2025. Fine-grained Prompt Screening: Defending Against Backdoor Attack on Text-to-Image Diffusion Models. In Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, IJCAI-25. IJCAI, 601–609. https://doi.org/10.24963/ijcai.2025/68

@inproceedings{ijcai2025p68,
  title     = {Fine-grained Prompt Screening: Defending Against Backdoor Attack on Text-to-Image Diffusion Models},
  author    = {Xu, Yiran and Zhong, Nan and Li, Guobiao and Cheng, Anda and Wang, Yinggui and Qian, Zhenxing and Zhang, Xinpeng},
  booktitle = {Proceedings of the Thirty-Fourth International Joint Conference on
               Artificial Intelligence, {IJCAI-25}},
  publisher = {International Joint Conferences on Artificial Intelligence Organization},
  editor    = {James Kwok},
  pages     = {601--609},
  year      = {2025},
  month     = {8},
  note      = {Main Track},
  doi       = {10.24963/ijcai.2025/68},
  url       = {https://doi.org/10.24963/ijcai.2025/68},
}

TY  - CPAPER
TI  - Fine-grained Prompt Screening: Defending Against Backdoor Attack on Text-to-Image Diffusion Models
AU  - Xu, Yiran
AU  - Zhong, Nan
AU  - Li, Guobiao
AU  - Cheng, Anda
AU  - Wang, Yinggui
AU  - Qian, Zhenxing
AU  - Zhang, Xinpeng
T2  - Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, IJCAI-25
PB  - International Joint Conferences on Artificial Intelligence Organization
PY  - 2025
SP  - 601
EP  - 609
DO  - 10.24963/ijcai.2025/68
UR  - https://doi.org/10.24963/ijcai.2025/68
ER  -