ClinAlign: Clinical Workflow Aligned Memory Retrieval for Radiology Report Generation

1 Introduction

Radiology report generation aims to connect computer vision and natural language processing to automatically generate structurally standardized, professional, and clinically consistent diagnostic reports from medical images [Nooralahzadeh et al., 2021; Yan and Pei, 2022; Wang et al., 2023a] . This task has significant practical value for clinical decision support and reducing the workload of radiologists [Tanida et al., 2023; Bu et al., 2024; Tanno et al., 2025] . Compared to general image description tasks, radiology report generation is much more challenging: reports are typically lengthy, structurally complex, and contain a large amount of technical terminology. Furthermore, in radiology images, key lesions often present subtle appearance features, have low contrast characteristics, and are easily obscured by complex anatomical structures , making accurate perception and description of lesions extremely difficult [Xie et al., 2023].

* Corresponding author.

In recent years, various research methods have emerged in the field of radiology report generation, including encoderdecoder frameworks enhanced with knowledge graphs [Liu et al., 2021; Chen et al., 2022; Li et al., 2023; Huang et al., 2023], memory mechanisms [Jin et al., 2024; Wang et al., 2024c; Tao et al., 2024], and large language models [Wang et al., 2023b; Chen et al., 2023; Liu et al., 2024b]. These methods have significantly improved the quality of generated reports. However, existing methods still have shortcomings in realistically reflecting the clinical diagnostic process, especially in fine-grained lesion modeling and clinical semantic consistency. Specifically, these limitations manifest in the following two aspects: First, existing research mainly focuses on the reuse of textual memory, neglecting visual pattern memory at the image patch level [Jin et al., 2024; Wang et al., 2024c; Tao et al., 2024]. In the complex realities of clinical diagnosis, radiologists rely heavily on visual memory templates to conduct in-depth and meticulous analyses of lesions' morphology, texture, and local characteristics. However, the problem is that existing models lack explicit modeling of lesion regions at the image patch level, which makes it extremely difficult for them to accurately perceive and clearly distinguish lesion regions.

Second, current retrieval enhancement models have significant shortcomings when it comes to offering clinical semantic guidance. Usually, these approaches directly match global image features with textual memory [Jin et al., 2024; Wang et al., 2024c]. This leads to an over-reliance on superficial similarity calculations and a lack of essential clinical reasoning support during the retrieval process . Such a design fails to take into account the well-recognized medical diagnostic principle of "identify first, then refine." Radiologists first identify diseases and determine their states, and then retrieve relevant knowledge. Skipping this crucial semantic step might unintentionally incorporate pathologically irrelevant information into the retrieval results, thus reducing the clinical effectiveness of the generated reports. Figure 1 offers a brief overview of the real-world clinical workflow adopted by radiologists: they first spot abnormalities, match them with visual pattern memory, infer disease states, and then retrieve clinically relevant knowledge and templates to produce structured reports.

To tackle these limitations, we introduce ClinAlign, a memory retrieval framework for radiology report generation that is aligned with clinical workflows. From the visual perspective, ClinAlign constructs a disease-aware visual memory bank and then enhances patch-level representations via Memory-based Patch Pattern Augmentation (MPPA), thereby improving the perception and discrimination of pathological regions. In terms of text, ClinAlign constructs a disease-aware textual memory bank and proposes Classification-Guided Prompt Augmentation (CGPA). This module translates the results of disease state prediction into structured diagnostic prompts, offering explicit semantic guidance for textual memory retrieval. This unified design ensures that both the visual perception and textual retrieval processes are in line with clinical diagnostic workflows, ultimately leading to the generation of more accurate and clinically coherent radiology reports.

We conduct extensive experiments on two medical report generation benchmarks. The results show that the proposed framework achieves competitive performance in report accuracy and clinical consistency, which effectively validates its efficacy and practical significance in automatic radiology report generation.

2 RelatedWork

2.1 Medical Radiology Report Generation

Radiology report generation aims to integrate computer vision and natural language processing technologies to automatically produce structured, professional reports frommedical images . Compared to general image description tasks, radiology report generation is much more challenging: reports are typically lengthy, structurally complex, and laden with specialized terminology. Furthermore, in radiology images, key lesions often present subtle appearance features, have low contrast characteristics, and are easily obscured by complex anatomical structures [Xie et al., 2023], making accurate perception and description of lesions extremely difficult.

To overcome these limitations, researchers have explored various approaches. Mainstream methods generally employ an encoder-decoder architecture and enhance the professionalism and completeness of reports by incorporating strategies such as knowledge graphs [Liu et al., 2021; Chen et al., 2022; Yang et al., 2022; Huang et al., 2023; Li et al., 2023], contrastive learning [Yang et al., 2023; Hou et al., 2023; Wang et al., 2024a], and memory mechanisms [Jin et al., 2024; Wang et al., 2024c; Tao et al., 2024]. Furthermore, methods optimized based on human feedback [Qin and Song, 2022; Xiao et al., 2025] and the application of large language models have also been proven to effectively improve the quality of generated reports [Chen et al., 2023; Liu et al., 2024b; Li et al., 2024a; Wang et al., 2025]. However, these methods still have shortcomings in the accurate perception of subtle lesions and semantic guidance that aligns with clinical diagnostic logic, which may reduce the clinical consistency of the generated reports.

2.2 Retrieval-Augmented Generation

Retrieval-augmented generation (RAG) has become an important technological paradigm in the field of natural language processing, demonstrating significant value in specialized tasks such as medical report generation [Singh et al., 2025]. Its core mechanism involves retrieving relevant memory knowledge units from large-scale professional corpora to provide reliable references for the generation process, thereby ensuring the professional accuracy of the output content . Current retrieval-augmented RRG methods mainly retrieve text from past reports [Yang et al., 2022; Jin et al., 2024; Wang et al., 2024c; Tao et al., 2024]. As a result, these methods deviate from the clinical workflow in two aspects. First, they rarely model local visual pattern memory, which weakens the perception and discrimination of pathological regions. Second, they often jump straight to global image-text matching, with no clear clinical guidance. Without first identifying the disease and its state, retrieval can bring in irrelevant details and make the report less clinically consistent.

3 Method

3.1 Disease-Aware Multimodal Memory Bank

Disease-Aware Visual Memory Bank As illustrated in Figure 3, given a chest X-ray image of size, visual features are extracted using a pre-trained medical Vision Transformer (ViT). The image is divided into fixed non-overlapping 16×16 patches, which are then linearly projected into token embeddings and processed by Transformer encoder layers.The resulting representations are fed into a multi-label classification head to predict disease categories in the MIMIC-CXR

dataset, with labels derived from expert-annotated radiology reports. During training, the ViT backbone remains frozen, and only the final linear classifier is fine-tuned.

Inspired by previous methods [Lin et al., 2023; Wang et al., 2024b], we leverage the class activation map (CAM) technique to localize disease-aware visual patches. In our work, HiResCAM is applied to highlight image regions most relevant to disease prediction. Compared with GradCAM, HiResCAM produces more spatially precise heatmaps [Draelos and Carin, 2020], enabling clearer localization of clinically significant pathological regions . Specifically, for an input X-ray image I and each of its positive disease labels, we generate a corresponding heatmap S ∈ RH×W . For each image patch pi in the heatmap S, we compute the average value vi within the patch region. vi is computed as follows:

where |pi| is the total number of pixels in patch pi. S(x, y) is the activation value at pixel (x, y) in the heatmap S .

Only when vi exceeds a predefined threshold θ is the patch pi retained and stored in the visual memory bank corresponding to that disease label.

where Vd denotes the visual memory bank for disease d. Following prior work, we set the threshold θ to 0.5.

Directly utilizing the complete visual memory bank Vd for training and retrieval poses significant efficiency bottlenecks and resource challenges, primarily due to its vast overall sample size. To address this issue, we compute the mean feature centroid cd for all samples within Vd :

where |Vd| is the total number of samples in Vd , and f denotes the feature vector of an individual sample. Based on this centroid, we then select the k-nearest samples to construct a compact subset.

We apply this selection process to all disease categories, extracting a compact subset for each. The final visual memory bank Vm is then formed by merging all these subsets.

Disease-Aware Textual Memory Bank To leverage the rich semantic information in medical reports, we construct a textual memory bank that operates in parallel with the visual one. Specifically, we employ BioClinicalBERT [Alsentzer et al., 2019] which is pre-trained on large-scale biomedical corpora - as the text encoder to extract features from all radiology reports in the training set. These textual features are then categorized according to the disease label associated with each report. For each disease d, we build its corresponding textual memory bank Td , formally defined as the set of textual feature vectors belonging to that disease category.

To limit the scale of the textual memory bank, we apply a parallel core sample selection strategy to all disease categories, extracting a compact and representative textual subset for each. The final textual memory bank Tm used for multimodal training is then constructed by merging all these disease-specific subsets.

3.2 Memory-Based Patch Pattern Augmentation

To address the lack of patch-level visual memory in existing methods. We leverage the disease-aware visual memory bank Vm to enhance local image features fp extracted by the visual encoder via Hopfield-based retrieval. Following prior work [Mao et al., 2025; Wang et al., 2025], we adopt a modern Hopfield network as the retrieval module due to its highcapacity associative memory, robust convergence behavior, and efficient matching of high-dimensional continuous representations [Ramsauer et al., 2020]. The Hopfield network performs associative retrieval between the input features and the stored disease-aware visual prototypes, yielding an enhanced representation:

Specifically, the Hopfield network matches each local feature fp against the memory patterns in Vm and aggregates relevant disease-aware visual information to update the feature representation. The resulting enhanced feature f ∗ p explicitly incorporates disease-specific visual priors, improving model's ability to perceive and discriminate pathological visual patterns.

3.3 Classification-Guided Prompt Augmentation

Directly retrieving textual knowledge from global image features often lacks explicit clinical semantics, leading to ambiguous or irrelevant retrieval results. To provide structured semantic guidance for textual retrieval and generation, we introduce a classification-guided prompt mechanism based on disease state recognition.

We use the global image feature fg for disease state classification, with the classification loss defined as:

where M denotes the number of diseases, N the number of samples, and C the number of states per disease. yi,m,c and ˆ yi,m,c represent the ground-truth label (one-hot) and the predicted probability, respectively.

The predicted disease states are used to fill predefined prompt templates following the pattern "The [disease name]

is [disease state] in the image.", thereby converting structured classification outputs into natural language descriptions. The resulting prompts are encoded by BioClinicalBERT and further enhanced via Hopfield-based retrieval over the constructed disease-aware textual memory bank Tm:

where fprompt denotes the original prompt text feature.

Through associative retrieval from the disease-aware textual memory bank Tm, the Hopfield network injects diseaseand state-specific clinical semantics as well as typical expression patterns into the prompt representation, resulting in an enhanced textual feature that provides more informative semantic guidance for subsequent report generation.

3.4 Overall Training Objectives

Finally, the enhanced prompt feature f ∗ t is fused with the retrieval-enhanced local visual feature f ∗ p via a feed-forward network with SwishGLU activation to form the joint representation X. The classification results are incorporated as diagnostic prompt tokens D = {d1, d2, . . . , dS}, where S denotes the fixed number of prompt tokens. Conditioned on the joint representation X and the diagnostic prompts D, the text decoder Fd generates the diagnostic report R. The decoding process is formulated as:

and the report generation is optimized by minimizing the language modeling loss:

where rt denotes the token predicted at time step t, and T is the total length of the generated report.

The overall training objective is defined as:

where λ is a balancing coefficient that controls the relative importance of the language modeling loss and the classification loss.

4 Experiments and Analysis

4.1 Experiments Settings

Datasets We evaluate our model on two public RRG benchmark datasets: IU X-Ray [Demner-Fushman et al., 2015] and MIMIC-CXR [Johnson et al., 2019]. IU X-Ray contains 7,470 chest X-ray image-report pairs released by Indiana University. MIMIC-CXR is a widely adopted large-scale benchmark, providing paired radiology images and diagnostic reports. For data partitioning, MIMIC-CXR uses the official split (270,790/2,130/3,858 samples). IU X-Ray splits data in a 7:2:1 ratio for reliable and fair evaluation.

Evaluation Metrics We evaluate our model with both natural language generation (NLG) metrics and clinical efficacy (CE) metrics. NLG metrics (BLEU [Papineni et al., 2002], METEOR [Denkowski and Lavie, 2011], ROUGE-L [Lin, 2004]) reflect report text fluency and similarity to reference reports.

CE metrics (precision, recall, F1-score) quantify clinical correctness by converting reports into 14 disease labels using CheXbert tool. All CE metric values reported are computed using example-based averaging.

Implementation Details The visual encoder is a medical pretrained 12-layer ViT-B/16 [Dosovitskiy, 2020], and the decoder is based on BERT-base. The training objective combines the language modeling loss and the classification loss, and the balancing coefficient λ is set to 4 based on hyperparameter ablation. We use BioClinicalBERT as the textual feature extractor. To improve retrieval efficiency, we compress both the visual and textual memory banks for each disease category by retaining the 500 samples closest to the feature centroid. This value is also determined via hyperparameter ablation. We train the model using the AdamW optimizer with a weight decay of 0.05. The initial learning rate is set to 5e-5 and decayed with a cosine schedule. We train for 15 epochs with a batch size of 16 and an input resolution of 224×224. All experiments are conducted on a single NVIDIA A40 GPU (48GB), and training takes approximately 24 hours.

4.2 Experimental Results

We compare our proposed method with several representative state-of-the-art radiology report generation models, including DCL [Li et al., 2023], KiUT [Huang et al., 2023] , PromptMRG [Jin et al., 2024], TokenMixer [Yang et al., 2024], OaD [Li et al., 2024b] , SILC [Liu et al., 2024a], and RRG-Mamba [Hou et al., 2025]. Table 1 reports the results on the MIMIC-CXR dataset using both CE and NLG metrics, while Table 2 presents the NLG results on the IU X-Ray dataset.

Results on MIMIC-CXR As shown in Table 1, Our method achieves the best results on all three CE metrics, with 0.525 Precision, 0.546 Recall, and 0.505 F1, significantly outperforming prior models. These results indicate that our model can identify clinical entities more accurately and more completely. Specifically, compared with DCL (0.373 F1), our method yields an absolute improvement of 13.2% in F1; compared with KiUT (0.321 F1), the gain further increases to 18.4%. Even against the strongest competitor RRG-Mamba (0.475 F1) and PromptMRG (0.476 F1), our method still achieves a 3.0% absolute improvement, highlighting its superior clinical entity-level performance. Meanwhile, ClinAlign is also competitive on NLG metrics. It achieves the best BLEU-1 score (0.412) and maintains strong performance on BLEU-4, METEOR, and ROUGE-L.

In our experiments, greater emphasis is placed on clinical efficacy metrics, as these metrics truly measure whether diagnostic entities are accurate and comprehensive-which is what matters in real-world clinical settings. Text generation metrics like BLEU and ROUGE primarily assess superficial similarity between sentences. However, they often fail to reflect whether critical lesion descriptions in reports are accurate and complete. Therefore, even if some methods achieve high text scores, our approach significantly outperforms them in clinical metrics. This demonstrates its true utility and reliability in real-world medical settings. In summary, our approach balances clinical correctness and language quality, significantly improving clinical efficacy while preserving the fluency of the report.

Results on IU X-Ray Since IU X-Ray does not provide standardized annotations or an established protocol for CE evaluation, prior work typically reports only NLG metrics. To guarantee an equitable comparison , we evaluate our method on IU X-Ray using NLG metrics only. As shown in Table 2, our method achieves 0.495 BLEU-1, 0.193 BLEU-4, 0.208 METEOR, and 0.397 ROUGE-L, where BLEU-1, METEOR, and ROUGE-L are the best or tied for the best. For instance, compared with R2GenCMN [Chen et al., 2021], our method improves BLEU-1 from 0.475 to 0.495 and ROUGE-L from 0.375 to 0.397. In addition, our method outperforms strong recent baselines such as TokenMixer on BLEU-1 (0.495 vs 0.483) and achieves comparable performance on other NLG metrics.

Overall, our method substantially improves clinical entitylevel correctness on MIMIC-CXR and achieves strong NLG performance on IU X-Ray, supporting the effectiveness of ClinAlign in improving both clinical accuracy and report generation quality.

4.3 Ablation Study

Analysis on proposed components Table 3 reports ablation results on the MIMIC test set, analyzing the contributions of MPPA and CGPA. It is important to note that without CGPA, retrieval is conducted by similarity matching between global image features and the textual memory bank. The baseline achieves CE scores of 0.495/0.509/0.472 (Precision/Recall/F1). Adding MPPA improves CE to 0.505/0.524/0.485, showing that disease-aware visual pattern memory strengthens patch-level representations and helps capture subtle lesions. Using CGPA alone boosts CE to 0.513/0.528/0.492, indicating that disease-state-based semantic cues provide clearer clinical guidance for retrieval and reduce irrelevant noise. Enabling both yields the best CE performance (0.525/0.546/0.505), demonstrating their complementarity in "local lesion perception + semantically guided retrieval." NLG metrics remain stable, suggesting improved clinical accuracy without sacrificing fluency. Overall, these results support the effectiveness of our design consistent with radiologists' diagnostic workflow.

Analysis of different sampling strategy We compare two retrieval sampling strategies: Random and KNN. Table 3 shows that KNN consistently yields better CE performance under the same component setting. For instance, without CGPA, switching from Random to KNN increases F1 from 0.485 to 0.496; with CGPA only, F1 improves from 0.486 to 0.492. This suggests that KNN retrieves more pathologyconsistent references, reducing retrieval noise and improving entity-level clinical accuracy. With the full MPPA + CGPA setup, KNN achieves the best result (F1 = 0.505), showing the importance of a compact and relevant retrieved subset for clinical consistency.

4.4 Hyperparameter Analysis

We study the impact of the balancing coefficient λ and the memory compression size k on the MIMIC-CXR test set. λ balances clinical semantic alignment and report generation. As shown in Fig. 4(a), performance improves as λ increases from 2 to 4 and reaches its best at λ = 4, suggesting that stronger classification guidance helps retrievalenhanced prompt learning. However, when λ is too large, performance drops because classification is over-weighted and language modeling is weakened, which reduces both CE and NLG scores. In contrast, a smaller λ provides too little semantic guidance. We further evaluate k under the default setting λ = 4, where k denotes the number of centroidnearest samples retained in each disease-specific memory bank. As shown in Fig. 4(b), the best performance is achieved at k = 500. A smaller k reduces retrieval diversity and weakens the retrieved cues, whereas a larger k introduces redundant or less representative samples, injecting noise that harms retrieval enhancement and generation accuracy. Overall, appropriate λ and k are crucial for balancing semantic guidance, retrieval efficiency, and generation quality. Therefore, we adopt λ = 4 and k = 500 as the default settings in all subsequent experiments.

5 Visualization Case

To evaluate the effectiveness of ClinAlign in generating radiology reports, we randomly select one case from MIMICCXR for qualitative analysis. From figure 5, we can find that the report generated by ClinAlign performs better in terms of clinical consistency and detailed description, focusing its attention on diagnostically relevant areas. Specifically, ClinAlign correctly captures postoperative findings (e.g., median sternotomy) and subtle imaging patterns (e.g., linear opacities), while excluding other key abnormalities such as consolidation, pleural effusion, or pneumothorax. Notably, in this case, ClinAlign covers all key medical disease entities mentioned in the actual report, demonstrating improved entity-level coverage and consistency, consistent with the method's significant improvement in CE metrics. In contrast, PromptMRG fails to identify these critical postoperative and subtle findings, its attention is scattered to coarse or non-specific areas, and it generates unsupported statements (highlighted in red), such as emphasizing mild cardiac hypertrophy or surgical clips not mentioned in the actual report or lacking sufficient imaging evidence. These errors further reduce the reliability and clinical utility of the generated reports. Overall, the results indicate that ClinAlign achieves more accurate localization of abnormalities and more reliable clinical semantic expression, thereby improving the accuracy and specificity of radiological report generation.

6 Conclusion

We propose ClinAlign, a clinical workflow-aligned memory retrieval framework for RRG task. The framework achieves multimodal integration by jointly modeling visual pattern memory and disease-aware textual memory: the MPPA module enhances local feature representation at the visual level, while the CGPA module converts disease predictions into structured prompts to guide text retrieval and report generation. Extensive experiments demonstrate that ClinAlign achieves competitive performance across multiple evaluation metrics and effectively improves the clinical consistency and quality of generated reports. In future work, we plan to extend ClinAlign to temporal and longitudinal radiology data and integrate it with larger foundation models, so as to better capture disease progression and further improve robustness and generalization for real-world clinical deployment.

Acknowledgements

This work was supported by the National Natural Science Foundation of China (Grant No.62576064), Chongqing Natural Science Foundation for Innovative Development Joint Fund (Grant No.CSTB2025NSCQ-LZX0147) and Chongqing Key Laboratory of Precision Diagnosis and Treatment for Kidney Diseases.