Hierarchical Conditional Energy Modeling for Medical Vision–Language Pretraining

1 Introduction

Contrastive language-image pre-training (CLIP) [Radford et al., 2021] has shown remarkable success in learning transferable representations by aligning images and text within a shared embedding space. In the medical domain, recent adaptations of CLIP demonstrate promising performance for tasks such as disease classification and report retrieval [Wang et al., 2022; Zhang et al., 2023; Wu et al., 2023; Huang et al., 2021a]. However, medical image interpretation differs from general vision-language settings in two important ways that remain insufficiently addressed. First, medical semantics are inherently hierarchical. Clinical findings are often described at multiple levels of abstraction, ranging from fine-grained disease entities (e.g., atelectasis) to higher-level diagnostic summaries (e.g., abnormal chest disease). These levels are not independent; rather, they obey structured semantic relationships that conventional CLIP-style alignment does not explicitly model. Second, medical reports typically describe an entire imaging study rather than individual images. A single radiology report may summarize findings across multiple views or slices, and specific abnormalities may only be visible in a subset of images. Treating each image-text pair independently therefore introduces semantic mismatch and weakens the alignment signal.

Most existing medical vision-language models typically align a single image embedding with multiple textual descriptions corresponding to different semantic levels (Fig. 1a), implicitly assuming that one representation can simultaneously capture all levels of abstraction. This assumption can lead to under-representation of higher-level clinical semantics and inconsistent predictions across levels (e.g., predicting a disease as present while simultaneously predicting "No Finding"). Such consistency is particularly important for clinical reliability and interpretability.

To address these challenges, we propose HCE-CLIP (Hierarchical Conditional Energy CLIP), a hierarchical conditional energy-based framework for medical vision-language pretraining. HCE-CLIP represents an imaging study as a unified image series embedding using a transformer-based aggregation mechanism [Vaswani et al., 2017], and models the compatibility between image series, texts, and structured clinical label states using temperature-scaled energy functions across multiple semantic levels (Fig. 1b). This formulation explicitly favors clinically consistent label states while suppressing contradictory alternatives, providing a principled mechanism for enforcing hierarchical semantic coherence.

To quantitatively assess semantic coherence, we further introduce a hierarchical contradiction-based evaluation metric that measures the extent to which model predictions violate predefined clinical hierarchy constraints. This metric directly evaluates whether fine-grained disease predictions are consistent with higher-level diagnostic summaries.

We pretrain HCE-CLIP on the MIMIC-CXR dataset [Johnson et al., 2019] and evaluate it on multiple public benchmarks, including MIMIC-CXR, CheXpert [Irvin et al., 2019], and ChestX-ray14 [Wang et al., 2017]. Experimental results demonstrate that HCE-CLIP improves classfication performance on both seen and unseen disease categories, enhances representation transferability, and substantially reduces hierarchical contradictions compared to existing medical vision-language models. These findings suggest that hierarchical conditional energy modeling provides an effective and general framework for medical vision-language representation learning. Our codes are available at https://github.com/ mocherson/HCE-CLIP

In summary, our contributions are threefold. (1) We propose HCE-CLIP, a hierarchical conditional energy-based framework for medical vision-language pretraining that explicitly models the semantic compatibility between image series, radiology reports, and structured clinical label states across multiple levels of abstraction. (2) We introduce a Hierarchical Contradiction Rate (HCR) metric to quantitatively evaluate semantic consistency across label hierarchies, enabling systematic analysis of logically coherent clinical predictions. (3) Extensive experiments demonstrate that HCECLIP achieves superior classification performance on seen and unseen diseases, improved representation transferability, and substantially reduced hierarchical contradictions compared to existing medical vision-language models.

2 RelatedWork

2.1 Vision-Language Representation

Vision-language representation learning has been widely studied using both dual-stream and single-stream architectures for multimodal fusion [Jia et al., 2021; Li et al., 2021; Chen et al., 2020]. Early work focused on aligning visual and textual modalities through contrastive objectives, enabling models to learn transferable representations from large-scale image-text data. Subsequent studies explored incorporating external knowledge [Cui et al., 2021; Li et al., 2020; Yu et al., 2021] and refining pretraining objectives such as masked language and visual modeling [Huang et al., 2021b; Wang et al., 2023].

More recently, CLIP-style contrastive learning has become a dominant paradigm for scalable vision-language pretraining [Radford et al., 2021; Li et al., 2022b; Mu et al., 2022; Yao et al., 2022; Chen et al., 2023]. Some works have investigated hierarchical or structured representations within contrastive frameworks. For example, HiCLIP [Geng et al., 2023] introduces hierarchy-aware attention mechanisms, and other approaches improve robustness or generalization by modeling semantic relationships across modalities [Ge et al., 2023]. Instruction-tuned multimodal models like LLaVAMed [Li et al., 2023] and Dr-LLaVA [Sun et al., 2024] extend CLIP-style architectures with biomedical-specific tuning, enabling better generalization in clinical settings. LoGra-Med [Nguyen et al., 2024] further augments alignment through multi-graph modeling to handle complex long-context scenarios in medical reports. However, these methods primarily focus on architectural or attention-based designs and do not explicitly model label-state semantics or structured opposition between clinically consistent and inconsistent concepts. In contrast, our work formulates vision-language alignment as a hierarchical label-conditional energy modeling problem, where semantic consistency across abstraction levels is enforced through structured energy functions rather than attention mechanisms alone.

2.2 Medical Image Understanding

Medical image understanding is a pivotal area in healthcare, and extensive research has been devoted to advancing the capabilities of computer systems in this domain [Wang et al., 2017; Mao et al., 2018; Mao et al., 2022; Yang et al., 2022]. Medical vision-language models have been adapted to align radiology images with free-text reports for tasks such as disease classification and report retrieval. MedCLIP [Wang et al., 2022] decouples image-text pairing to increase training scalability, while MedKLIP [Wu et al., 2023] incorporates structured medical knowledge into the alignment process. GLoRIA [Huang et al., 2021a] learns both global and local representations to improve label efficiency in medical image recognition. CheXzero [Zhang et al., 2023] leverages domain-specific knowledge to guide pretraining on chest radiographs. MedMax [Bansal et al., 2024] introduced mixed-modal instruction tuning to scale vision-language assistants for biomedicine.

While these approaches improve medical image-text alignment, they typically rely on aligning individual images with textual descriptions and do not explicitly account for the fact that radiology reports summarize entire imaging studies. Moreover, hierarchical clinical semantics are usually treated implicitly through prompt design or supervision, rather than being modeled as structured semantic constraints.

Our method differs by encoding an image series as a unified representation and aligning it jointly with reports and structured label prompts across multiple semantic levels. By modeling label states through conditional energy functions, HCE-CLIP explicitly enforces consistency between supported and contradictory clinical concepts, enabling more structured and interpretable alignment.

3 Method

3.1 Hierarchical Label-Conditional Energy

Energy-based models (EBMs) provide a flexible framework for modeling structured relationships by assigning a scalar energy to each configuration of variables, where lower energy indicates higher compatibility [LeCun et al., 2006]. EBMs define unnormalized compatibility functions and can be trained using discriminative objectives [Xie et al., 2016; Du and Mordatch, 2019]. This property makes them wellsuited for modeling multimodal and structured prediction problems.

We formulate medical vision-language pretraining as a hierarchical label-conditional energy modeling problem. Given an image series x, its associated report t, a medical label l, and a clinical state s ∈ S (positive, negative, or uncertain), the model assigns an energy value to each configuration, where lower energy indicates stronger semantic compatibility.

At each hierarchy level k, HCE-CLIP defines temperaturescaled energy functions for both the image-label-state and text-label-state relationships:

where E(k) x and E(k) t denote the image-series and report embeddings, E(k) l,s denotes the corresponding label-state embedding, and τk is a learnable temperature parameter that controls the sharpness of the energy distribution at level k. ⟨, ⟩ is the dot product of two vectors. At each hierarchy level k, the label l is interpreted according to the corresponding semantic granularity (e.g., specific disease-level or coarse categorylevel), with distinct prompt sets and embeddings used to represent the label states at that level.

The conditional distributions over label states are defined as:

Learning proceeds by minimizing the negative loglikelihood of the ground-truth state s∗ for both modalities:

which enforces semantic consistency between the image, the report, and the corresponding clinical label state.

This objective encourages the model to assign lower energy to clinically consistent label states while increasing the energy of contradictory or uncertain alternatives. Such conditional energy formulations have been shown to be effective for structured classification and discriminative modeling [Grathwohl et al., 2021; Xie et al., 2021].

In addition to label-conditional energy modeling, HCECLIP preserves the contrastive alignment between image series x and free-text reports t through a temperature-scaled cross-modal energy formulation:

where τ0 is a learnable temperature parameter.

The symmetric CLIP objective minimizes both the image-to-text and text-to-image conditional negative loglikelihoods:

where

The full training loss combines cross-modal alignment and hierarchical label-conditional energy learning:

which jointly optimizes image-report alignment and hierarchical label-conditional energy separation across both visual and textual modalities.

3.2 HCE-CLIP Architecture

We now describe how the proposed hierarchical labelconditional energy model is instantiated within a practical vision-language pretraining architecture. This section introduces the HCE-CLIP model design, including the image series encoder, text encoder, and hierarchical representation modules used to realize the energy functions defined above.

Figure 2 illustrates the overall architecture and training pipeline of HCE-CLIP. Given an image series x, its associated report t, and a set of medical labels {l} with corresponding clinical states at each semantic level, the model learns to align visual, textual, and structured label information within a unified energy-based framework.

The image series is first encoded into a unified representation using a transformer-based aggregation module, while the report is encoded by a text encoder. Both representations are then projected into a shared embedding space and further transformed into hierarchical representations that capture progressively abstract medical semantics. In parallel, each label is converted into multiple natural-language prompts corresponding to different clinical states (e.g., positive, negative, uncertain) and encoded into label-state embeddings.

HCE-CLIP defines hierarchical label-conditional energy functions between image series, reports, and label-state embeddings, encouraging clinically consistent configurations to have low energy while suppressing contradictory alternatives.

Image Series Encoding. Radiology reports describe an entire imaging study rather than individual images. We therefore encode an image series x = (x1, · · · , xj) as a unified representation. In our framework, the image series encoder consists of an image encoder and a multi-layer transformer. The image encoder processes each image in the series to produce an individual image embedding. The transformer then aggregates these individual embeddings into a unified image series embedding. A projection head is applied to transform the raw embedding to a specified dimension. The process is denoted as

where Enci is the image encoder; xj is the jth image in the input image series; E′x is the the raw output embedding of the image series through the transformer; fs is the projection head that maps the output embeddings E′x to a specified dimension Ex.

Text Encoding. The text encoder is to encode a text into a raw text embedding that is also projected to a specified dimension by the projection head, denoted as

where Enct is the text encoder; t is the input text; ft is the projection head that maps the raw output embeddings E′t to a specified dimension Et. Et should have the same dimension with Ex for contrastive learning. The text encoder is also used to encode label-state prompts.

Hierarchical Representations. Medical semantics are inherently hierarchical, with fine-grained disease descriptions forming the basis for more abstract clinical summaries. To capture this structure, HCE-CLIP constructs multi-level representations for both image series and reports using cascaded transformation networks. Starting from the base embeddings Ex and Et obtained from the image-series encoder and text encoder, respectively, we apply a sequence of multilayer perceptrons (MLPs) to progressively extract higher-level semantic features:

Each transformation produces representations corresponding to a different level of semantic abstraction. Lower levels retain fine-grained visual and textual details, while higher levels encode more abstract clinical concepts. These hierarchical embeddings are subsequently aligned with label-state prompt embeddings using the proposed label-conditional energy formulation, enabling structured semantic consistency across abstraction levels. This cascaded representation design allows higher-level clinical concepts to regularize lower-level representations, promoting coherent and interpretable multimodal alignment.

Label-State Prompt Encoding. Medical labels often admit multiple semantic states, reflecting the presence, absence, or uncertainty of a clinical finding. In HCE-CLIP, we model this structure explicitly by associating each medical label l with a set of states corresponding to S = {positive, negative, uncertain}.

In our experiments on the MIMIC-CXR dataset, we consider 14 predefined labels, including 12 disease categories, support devices, and no finding. The label no finding, which summarizes the absence of pathology, is treated as a higherlevel (category-level) semantic concept, while the remaining labels are treated as disease-level concepts.

For each label l and state s ∈ S, we construct a naturallanguage prompt to represent the label in that state. For example, for the label Atelectasis, the prompts to represent positive, negative, and uncertain clinical states are:

• Disease Atelectasis is found.
• Disease Atelectasis is not found.
• Not sure if Disease Atelectasis is found.

Each label-state prompt is encoded using the text encoder to obtain a corresponding label-state embedding:

where Prompt(·) generates the label-state prompts, Enct denotes the text encoder, and fl is a projection head that maps the embeddings to the appropriate dimensionality.

To ensure compatibility with the hierarchical energy formulation, the embedding dimension of disease-level labelstate representations E(1) l,s matches that of the corresponding image and text representations E(1) x and E(1) t , while the category-level label-state representations E(2) l,s match the dimensionality of E(2) x and E(2) t . This design enables structured alignment between multimodal representations and hierarchically organized clinical semantics.

Energy Modules. HCE-CLIP employs a set of energy modules to compute scalar compatibility scores between image representations, text embeddings, and label-state prompts at different semantic levels. Each energy module implements an energy function defined in Eq. 1, Eq. 2, or Eq. 6, depending on the input modalities. During training, energies corresponding to the ground-truth label state are minimized, while energies that contradict the ground-truth label state are maximized. This formulation enables HCE-CLIP to explicitly enforce semantic compatibility and hierarchical consistency across disease-level and category-level concepts.

3.3 Training Objective

With the above architecture, HCE-CLIP produces aligned representations for image series (E(k) x ), report texts (E(k) t ), and structured label prompts (E(k) l,s ) across multiple semantic levels. Based on these representations, the corresponding energy functions are defined under the hierarchical labelconditional energy modeling framework described in Section 3.1. The training objective of HCE-CLIP jointly optimizes cross-modal image-text alignment and hierarchical label-conditional energy learning.

Image-Text Alignment. At the base level, HCE-CLIP preserves the original CLIP objective by minimizing the symmetric image-to-text and text-to-image conditional negative log-likelihoods defined by the cross-modal energy E (0) (x, t) (Eq. 7). This encourages globally consistent alignment between image series and their associated free-text reports.

Hierarchical Label-Conditional Learning. At each hierarchy level k ≥ 1, the model is trained to assign lower energy to the ground-truth clinical label state and higher energy to contradictory or uncertain alternatives using the labelconditional energy functions E (k) x (x, l, s) and E (k) t (t, l, s) (Eqs. 1-5). This enforces semantic consistency among the image series, the report, and the structured label prompts across multiple levels of clinical abstraction.

Joint Optimization. The overall training loss combines the cross-modal alignment objective and the hierarchical labelconditional energy losses across all labels and hierarchy levels (Eq. 10). By optimizing these objectives jointly, HCECLIP learns representations that are simultaneously aligned across modalities and constrained by structured clinical semantics.

3.4 Inference

At inference time, HCE-CLIP predicts the clinical state of a medical label by minimizing the hierarchical labelconditional energy. For image classification, given an image series x and a target label l, the model evaluates the compatibility between the input image series and each possible clinical state s ∈ S by computing the image-label-state energy Ex(x, l, s) using Eq. 1. The final prediction is obtained by selecting the state with the minimum energy at the hierarchy level corresponding to l:

Text-based classification can be performed analogously by minimizing the text-label-state energy Et(t, l, s).

This energy-based inference formulation provides a unified mechanism for zero-shot classification, uncertainty-aware prediction, and hierarchical semantic consistency. New medical concepts can be introduced at test time through textual label descriptions without retraining the image encoder.

3.5 Hierarchical ConsistencyMetric

Standard classification metrics (AUC/ACC/F1) do not measure whether predictions are logically consistent across the label hierarchy. For example, predicting 'Atelectasis = positive' while predicting 'No Finding = positive' constitutes a hierarchical contradiction. To quantify semantic inconsistency, we define the Hierarchical Contradiction Rate (HCR), the fraction of violated hierarchical constraints.

In our hierarchy, level-2 corresponds to No Finding / Normal (denoted as NF), which summarizes the absence of any abnormalities. Therefore, a logically consistent prediction should not simultaneously indicate that a specific disease is present and that the study is normal.

Let L1 denote the set of level-1 disease labels (excluding NF). For a test sample xi, let ˆ si,l ∈ S be the predicted clinical state of label l, where S = {positive, negative, uncertain}. We define a contradiction for disease label l as:

We report the HCR as the fraction of disease-normal contradictions across labels and samples:

where N is the number of test samples. Lower HCR indicates better hierarchical semantic consistency. Following common practice for uncertainty-aware evaluation, we do not count ˆ si,NF = uncertain as a contradiction.

4 Experiments

4.1 Datasets

Pretraining Dataset. We pretrain HCE-CLIP on MIMICCXR [Johnson et al., 2019], a large public dataset of 377,110 chest radiographs, corresponding to 227,835 imaging studies paired with free-text radiology reports. MIMIC-CXR predefined 14 labels, including 12 disease, support devices, and no findings. We follow the official patient-level split and train on 222,758 studies, validating on 1,808 studies to select the best model.

Evaluation Dataset. (1) MIMIC-CXR test set is a split from MIMIC-CXR dataset. It contains 5,159 images from 3,269 studies with the same disease label set as the training set. This evaluates the performance on seen labels. (2) CheXpert [Irvin et al., 2019], the test set contains 500 radiologistannotated test studies with the same label set as MIMICCXR, enabling external validation on seen diseases. (3) ChestX-ray14 [Wang et al., 2017] contains 112,120 frontalview X-rays with 14 disease labels. Seven labels overlap with MIMIC-CXR (seen), and seven are unseen, enabling zeroshot evaluation on novel disease categories. Each image is treated as a single-image study.

4.2 Implementation Details

We benchmarked on the Mask R-CNN [He et al., 2017] with SwinTransformer [Liu et al., 2021] followed by a feature pyramid networks (FPN) [Lin et al., 2017] as the image encoder. We used the BERT model [Devlin et al., 2019] as the backbone of text encoder. The image encoder and text encoder are initialized with the GLIP-T [Li et al., 2022a] model.

The original text embedding dimension output from text encoder is 768. The original image series embedding dimension output from image series encoder is 1280 and is then reduced to 768 by a linear projection head. The level 1 (disease-level) and level 2 (category-level) embeddings are 128 and 64, respectively. The transformer to aggregate the image embeddings consists 2 layers of Multihead Attention. All the learnable temperature τk is initialized to 0.07. The training batch size is 32 and the max epoch is 30. we use AdamW optimizer [Loshchilov and Hutter, 2019] with both initial weight decay and initial learning rate equal to 0.0001 during pre-training.

We evaluate several variants of HCE-CLIP, including models with one or two hierarchical levels (H1/H2), with or without energy-based modeling (E/NE), and using either full image series or single images as input (F/S). We name the models like HCE-H2-E-F. Baselines include MedCLIP [Wang et al., 2022] and MedKLIP [Wu et al., 2023], evaluated using the same prompt format and inference procedure for fair comparison.

4.3 Classification Results

Tables 1 and 2 report classification performance using prompt-based, classifier-free inference. MIMIC-CXR Test and CheXpert use the same label set as training (seen labels), while ChestX-ray14 additionally contains unseen disease categories for zero-shot evaluation.

For seen-label classification, HCE-H2-E-F achieves the best overall performance, with the highest AUC and F1 scores across all three datasets. All HCE-CLIP variants substantially outperform MedCLIP and MedKLIP, demonstrating the effectiveness of hierarchical energy-based alignment for prompt-based classification.

For unseen diseases on ChestX-ray14, HCE-H2-E-F consistently outperforms all baselines, achieving the highest AUC, accuracy, and F1. In contrast, MedCLIP and MedKLIP exhibit weaker generalization, highlighting the advantage of hierarchical energy modeling for zero-shot medical classification.

4.4 Linear-Probe Evaluation

Table 3 reports linear-probe evaluation results on MIMICCXR and ChestX-ray14. In this setting, the image encoder is frozen and a linear classifier is trained on top of the learned representations, allowing us to assess feature transferability.

HCE-CLIP with two hierarchical levels achieves the best performance on both datasets, with the highest AUC, accuracy, and F1 scores, outperforming both MedKLIP and Med- CLIP by a clear margin. These results indicate that HCECLIP learns more discriminative and transferable visual representations than existing medical vision-language pretraining methods, further validating the effectiveness of hierarchical conditional energy modeling.

4.5 Hierarchical Consistency Evaluation

Table 4 summarizes the HCR (Eq. 20) results across MIMICCXR Test, CheXpert, and ChestX-ray14. HCE-H2-E-F and HCE-H2-E-S achieve near-zero contradiction rates on all datasets, demonstrating that two-level hierarchical energy modeling enforces strong semantic consistency. In contrast, removing hierarchical energy constraints (HCE-H2-NE-F) leads to higher contradiction rates, confirming that architectural hierarchy alone is insufficient without explicit energybased supervision.

The single-level variants show substantially worse consistency. HCE-H1-E-F exhibits noticeably higher HCR, while HCE-H1-NE-F suffers from severe inconsistency across all datasets. This indicates that both hierarchical depth and energy-based modeling are necessary to maintain semantic coherence.

Baseline methods MedCLIP and MedKLIP show significantly higher contradiction rates than all HCE-CLIP variants. Overall, these results demonstrate that hierarchical conditional energy modeling is crucial for enforcing clinically meaningful semantic consistency, substantially reducing contradictory predictions compared to both non-energy variants and existing medical vision-language models.

4.6 Ablation Insights

Effect of Hierarchical Energy Modeling. We compare HCE-H2-E-F with its non-energy counterpart HCE-H2-NEF, as well as HCE-H1-E-F with HCE-H1-NE-F. Across all datasets, models with hierarchical energy supervision achieve higher classification performance and substantially lower HCR. In particular, HCE-H2-E-F consistently outperforms HCE-H2-NE-F in AUC and F1 while reducing contradiction rates to near zero. These results confirm that explicit hierarchical energy modeling is essential for both predictive accuracy and logical coherence.

Effect of Image-Series Encoding. We compare HCE-H2E-F with the single-image variant HCE-H2-E-S to evaluate the impact of modeling complete imaging studies. While both variants achieve low HCR, image-series encoding improves performance on most datasets, indicating that aggregating multiple views better captures clinical context than relying on individual images alone.

Effect of Hierarchical Depth. We compare two-level models (HCE-H2-E-F) with single-level models (HCE-H1-E-F). The two-level model consistently achieves higher AUC and F1 scores and lower HCR. In contrast, increasing hierarchical depth without energy modeling (HCE-H2-NE-F vs. HCEH1-NE-F) does not improve classification performance and only marginally improves consistency. This indicates that hierarchical depth alone is insufficient to improve performance without explicit energy-based supervision.

Overall, these ablations validate the design choices of HCE-CLIP and highlight the importance of structured energy modeling for reliable medical vision-language understanding.

5 Discussion and Conclusion

Our results demonstrate that hierarchical conditional energy modeling provides a strong structural guidance for learning hierarchically coherent medical representations. Across MIMIC-CXR, CheXpert, and ChestX-ray14, HCE-CLIP consistently improves both classification performance and hierarchical semantic consistency.

A key finding is that hierarchical energy supervision, rather than hierarchical structure alone, drives these improvements. Ablation studies show that increasing hierarchical depth without energy constraints does not improve performance or consistency, while energy-based supervision substantially reduces hierarchical contradictions. This indicates that structured energy modeling is essential for enforcing meaningful semantic relationships between fine-grained disease findings and higher-level clinical concepts.

HCE-CLIP also achieves near-zero Hierarchical Contradiction Rate, producing logically coherent predictions across semantic levels. Such consistency is particularly important in clinical settings, where contradictory outputs (e.g., detecting a disease while predicting "No Finding") may undermine trust. In addition, while single-imagemodels can also achieve low contradiction rates, HCE-CLIP with image-series encoding improves robustness by aggregating multi-view information into a unified representation.

Despite these strengths, our approach relies on predefined label hierarchies and prompt-based label descriptions, and our evaluation focuses on chest X-ray data. Future work may explore learning hierarchical structures automatically, extending to other imaging modalities, and integrating uncertainty-aware decision thresholds for clinical deployment.

Overall, HCE-CLIP provides a principled and effective framework for hierarchical conditional energy-based medical vision-language pretraining, enabling more accurate, consistent, and interpretable clinical predictions.