---
title: "REVEAL: Multi-turn Evaluation of Image-Input Harms for Vision LLMs"
authors:
  - "Madhur Jindal"
  - "Saurabh Deshpande"
doi: "10.24963/ijcai.2025/1081"
url: "https://ijcai.org/proceedings/2025/1081"
pdf: "https://ijcai.org/proceedings/2025/1081.pdf"
venue: "Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence"
year: 2025
pages: "9728-9736"
track: "AI and Social Good"
keywords:
  - "Natural Language Processing: General"
  - "AI Ethics, Trust, Fairness: General"
  - "Agent-based and Multi-agent Systems: General"
  - "Multidisciplinary Topics and Applications: General"
---

# REVEAL: Multi-turn Evaluation of Image-Input Harms for Vision LLMs

Madhur Jindal, Saurabh Deshpande

*Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence*, pages 9728-9736. DOI: [10.24963/ijcai.2025/1081](https://doi.org/10.24963/ijcai.2025/1081)

## Abstract

Vision Large Language Models (VLLMs) represent a significant advancement in artificial intelligence by integrating image-processing capabilities with textual understanding, thereby enhancing user interactions and expanding application domains. However, their increased complexity introduces novel safety and ethical challenges, particularly in multi-modal and multi-turn conversations. Traditional safety evaluation frameworks, designed for text-based, single-turn interactions, are inadequate for addressing these complexities. To bridge this gap, we introduce the REVEAL (Responsible Evaluation of Vision-Enabled AI LLMs) Framework, a scalable and automated pipeline for evaluating image-input harms in VLLMs. REVEAL includes automated image mining, synthetic adversarial data generation, multi-turn conversational expansion using crescendo attack strategies, and comprehensive harm assessment through evaluators like GPT-4o.

We extensively evaluated five state-of-the-art VLLMs, GPT-4o, Llama-3.2, Qwen2-VL, Phi3.5V, and Pixtral, across three important harm categories: sexual harm, violence, and misinformation. Our findings reveal that multi-turn interactions result in significantly higher defect rates compared to single-turn evaluations, highlighting deeper vulnerabilities in VLLMs. Notably, GPT-4o demonstrated the most balanced performance as measured by our Safety-Usability Index (SUI) followed closely by Pixtral. Additionally, misinformation emerged as a critical area requiring enhanced contextual defenses. Llama-3.2 exhibited the highest MT defect rate (16.55%) while Qwen2-VL showed the highest MT refusal rate (19.1%).