{"type":"ScholarlyArticle","title":"Explaining Jailbreaks: Structured and Interpretable Safety Assessment for Large Language Models","authors":[{"name":"Sunghee Dong","slug":"sunghee-dong","given":"Sunghee","family":"Dong","affiliation":""},{"name":"Sungwon Yi","slug":"sungwon-yi","given":"Sungwon","family":"Yi","affiliation":""},{"name":"Kangmin Bae","slug":"kangmin-bae","given":"Kangmin","family":"Bae","affiliation":""},{"name":"Jaeyoon Kim","slug":"jaeyoon-kim","given":"Jaeyoon","family":"Kim","affiliation":""}],"abstract":"Large Language Models (LLMs) remain highly vulnerable to jailbreak attacks, yet existing evaluations rely primarily on outcome-level metrics such as Attack Success Rate (ASR), providing limited insight into how and why safety failures occur.\r\nWe propose an explanation-aware safety framework that augments binary harmfulness detection with structured, human-interpretable explanations capturing severity, strategies, trigger spans, rationales, and derived safety factors.\r\nTo enable scalable and consistent supervision, we introduce a human--LLM hybrid annotation and canonicalization pipeline. We then fine-tune a compact model to generate canonical explanations alongside harmfulness decisions.\r\nAcross both seen and unseen benchmark settings, our method improves robustness and explanation fidelity.\r\nIn jailbreak defense evaluation, our approach reduces ASR to 0.44% on Vicuna-7B and 1.30% on GPT-3.5, outperforming existing defense baselines while also achieving the lowest StrongREJECT scores.\r\nBeyond outcome-level gains, the model more accurately recovers diagnostic attributes (e.g., attack strategy, trigger spans, and safety factors) than strong general-purpose LLM baselines.\r\nOverall, explanation-aware learning exposes diagnostic dimensions that ASR alone cannot capture and provides a more faithful and actionable foundation for robust LLM safety assessment.","doi":"10.24963/ijcai.2026/63","url":"https://ijcai.org/proceedings/2026/63","pdf":"https://ijcai.org/proceedings/2026/0063.pdf","markdown":"https://ijcai.org/proceedings/2026/63.md","bibtex":"https://ijcai.org/proceedings/2026/bibtex/63","volume":{"slug":"2026","title":"Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2026","url":"https://ijcai.org/proceedings/2026/","year":2026,"location":"Bremen, Germany","dates":"15-21 August 2026","editors":"Diego Calvanese","isbn":"978-1-956792-09-6","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2026"},"section":"AI Ethics, Trust, Fairnes","track":"Main Track","number":63,"pages":{"first":553,"last":561},"keywords":["Natural Language Processing: Language models","AI Ethics, Trust, Fairnes: Safety and robustness","AI Ethics, Trust, Fairnes: Explainability and interpretability","AI Ethics, Trust, Fairnes: Trustworthy AI"],"videos":[],"references":[],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}