---
title: "Hierarchical Conditional Energy Modeling for Medical Vision–Language Pretraining"
authors:
  - "Chengsheng Mao"
  - "Yuan Luo"
doi: "10.24963/ijcai.2026/759"
url: "https://ijcai.org/proceedings/2026/759"
pdf: "https://ijcai.org/proceedings/2026/0759.pdf"
venue: "Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence"
year: 2026
pages: "6823-6831"
track: "AI and Health"
keywords:
  - "Multimodal data: Multimodal data"
  - "Biomedical NLP: Biomedical NLP"
  - "LLM in medicine: LLM in medicine"
  - "Medical imaging: Medical imaging"
---

# Hierarchical Conditional Energy Modeling for Medical Vision–Language Pretraining

Chengsheng Mao, Yuan Luo

*Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence*, pages 6823-6831. DOI: [10.24963/ijcai.2026/759](https://doi.org/10.24963/ijcai.2026/759)

## Abstract

Contrastive vision–language pretraining models such as CLIP align images and text in a shared embedding space but do not explicitly model or evaluate the hierarchical semantics common in medical image interpretation. We propose HCE-CLIP (Hierarchical Conditional Energy CLIP), a vision–language pretraining framework that formulates medical image–text alignment as a hierarchical label-conditional energy modeling problem. HCE-CLIP encodes an image series using transformer-based aggregation and aligns it with free-text reports and structured label state prompts across multiple semantic levels. At each level, conditional energy functions favor clinically consistent label states while suppressing contradictory alternatives, enabling uncertainty-aware inference. To assess semantic coherence, we introduce a hierarchical contradiction-based metric that quantifies logical inconsistencies between fine-grained disease predictions and higher-level clinical summaries. Experiments on MIMIC-CXR and other public benchmarks show that HCE-CLIP outperforms existing medical vision–language pretraining methods in seen-label, zero-shot and linear-probe settings, while producing substantially fewer hierarchical contradictions.