{"type":"ScholarlyArticle","title":"REVEAL: Multi-turn Evaluation of Image-Input Harms for Vision LLMs","authors":[{"name":"Madhur Jindal","slug":"madhur-jindal","given":"Madhur","family":"Jindal","affiliation":""},{"name":"Saurabh Deshpande","slug":"saurabh-deshpande","given":"Saurabh","family":"Deshpande","affiliation":""}],"abstract":"Vision Large Language Models (VLLMs) represent a significant advancement in artificial intelligence by integrating image-processing capabilities with textual understanding, thereby enhancing user interactions and expanding application domains. However, their increased complexity introduces novel safety and ethical challenges, particularly in multi-modal and multi-turn conversations. Traditional safety evaluation frameworks, designed for text-based, single-turn interactions, are inadequate for addressing these complexities. To bridge this gap, we introduce the REVEAL (Responsible Evaluation of Vision-Enabled AI LLMs) Framework, a scalable and automated pipeline for evaluating image-input harms in VLLMs. REVEAL includes automated image mining, synthetic adversarial data generation, multi-turn conversational expansion using crescendo attack strategies, and comprehensive harm assessment through evaluators like GPT-4o.\r\n\r\nWe extensively evaluated five state-of-the-art VLLMs, GPT-4o, Llama-3.2, Qwen2-VL, Phi3.5V, and Pixtral, across three important harm categories: sexual harm, violence, and misinformation. Our findings reveal that multi-turn interactions result in significantly higher defect rates compared to single-turn evaluations, highlighting deeper vulnerabilities in VLLMs. Notably, GPT-4o demonstrated the most balanced performance as measured by our Safety-Usability Index (SUI) followed closely by Pixtral. Additionally, misinformation emerged as a critical area requiring enhanced contextual defenses. Llama-3.2 exhibited the highest MT defect rate (16.55%) while Qwen2-VL showed the highest MT refusal rate (19.1%).","doi":"10.24963/ijcai.2025/1081","url":"https://ijcai.org/proceedings/2025/1081","pdf":"https://ijcai.org/proceedings/2025/1081.pdf","markdown":"https://ijcai.org/proceedings/2025/1081.md","bibtex":"https://ijcai.org/proceedings/2025/bibtex/1081","volume":{"slug":"2025","title":"Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2025","url":"https://ijcai.org/proceedings/2025/","year":2025,"location":"Montreal, Canada","dates":"16-22 August 2025","editors":"James Kwok","isbn":"978-1-956792-06-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2025"},"section":"AI and Social Good","track":"AI and Social Good","number":1081,"pages":{"first":9728,"last":9736},"keywords":["Natural Language Processing: General","AI Ethics, Trust, Fairness: General","Agent-based and Multi-agent Systems: General","Multidisciplinary Topics and Applications: General"],"videos":[],"references":[],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}