DERI: Cross-Modal ECG Representation Learning with Deep ECG-Report Interaction

1 Introduction

Electrocardiogram (ECG) is a widely used data for reflecting heart electrical activity [Attia et al., 2019], which is of great importance for cardiac conditions classification. Supervised learning methods have obtained effective performance in ECG signal classification with high-quality annotations [Huang et al., 2022; Chen et al., 2024a]. However, there are a large number of unlabeled ECG signals in the real world, and supervised learning methods have difficulty utilizing this resource effectively. To reduce the dependence on labeled data, ECG representation learning methods based on selfsupervised learning (SSL) have demonstrated their powerful performance [Oh et al., 2022]. Compared to supervised learning methods, SSL methods aim to learn effective representations from ECG signal data without labels and thus tend to be more generalizable and adaptable to different downstream tasks, showing great potential.

∗ Corresponding authors.

ECG self-supervised learning (SSL) methods are mainly generative or contrastive. Generative models reconstruct inputs via masked modeling, while contrastive methods distinguish between similar and dissimilar samples. However, their single-modal nature limits the capture of rich clinical semantics. Generative methods focus on reconstructing low-level signal patterns, often missing clinical semantics [Zhang et al., 2023], while contrastive methods rely on input-level augmentations, which risk distorting ECG semantics [Na et al., 2024]. Multi-modal learning has emerged as a promising solution for these limitations due to its great learning ability with multiple data sources [Chen et al., 2024b; Chen et al., 2024c]. Compared to ECG signals, clinical reports offer direct high-level semantic insights. Inspired by advances in medical imaging and radiology reports [Liu et al., 2023a], Liu et al. proposed a multi-modal representation learning approach called MERL by aligning ECG signals with clinical reports [Liu et al., 2024b]. However, their method aligns ECG features with report features in the feature space, drawing inspiration from CLIP [Radford et al., 2021], but the interaction between modalities is relatively shallow. Furthermore, although the ECG representations learned by MERL performwell in classification tasks, they fail to effectively convey the underlying semantics of ECG recordings, which are crucial for understanding cardiac conditions.

To overcome these limitations, we proposed a novel Deep ECG Report Interaction (DERI) framework for cross-modal representation learning. To better capture the clinical semantics of ECG signals, we design an encoder-decoder structure to conduct multiple cross-modal alignments and mutual feature reconstruction. Specifically, ECG signals and the corresponding clinical reports are first encoded and projected into a shared alignment space to achieve an initial alignment. To enhance interaction, two specialized decoders are employed to reconstruct features by decoding the aligned representations into the other modality. This reconstruction process captures the latent semantics in both modalities, enabling the learning of richer cross-modal representations. Subsequently, the decoded features are fused with the modality-specific aligned features to create mixed representations incorporating both ECG signals and clinical reports semantics. These mixed representations are further utilized for a second alignment. Additionally, we introduce a Random Masked Enhancement Module (RME-Module) to improve ECG representation learning. Furthermore, the proposed DERI framework is integrated with language models to generate reports, providing a way to assess the learned clinical semantics. Extensive experiments across various settings and datasets are conducted to demonstrate the effectiveness of DERI. The main contributions are summarized as follows:

• To learn effective ECG representation from reports, we propose a novel cross-modal framework of ECG-Report via multiple feature alignment and mutual feature reconstruction. An RME-Module is also designed for ECG representation learning enhancement.
• To better illustrate the clinical semantics learned by DERI, we combine it with a language model for report generation. The pre-trained model provides effective ECG representation and a language model is used to generate clinical reports for clinical semantics.
• Comprehensive experiments on downstream datasets are conducted to evaluate the proposed DERI method, including zero-shot classification, linear probing, and even report generation. Experimental results illustrate that our DERI method surpasses all SOTA methods.

2 RelatedWork

Single-modal ECG Representation Learning. There are various SSL methods for ECG representation learning. Most of these methods are single-modal, which conduct generative learning or contrastive learning on unannotated ECG signals. CLOCS [Kiyasseh et al., 2021] and ASTCL [Wang et al., 2023] are the SOTA single-modal contrastive learning methods that explore the spatial-temporal correlation of ECG signals. Similarly, ST-MEM [Na et al., 2024] proposes to learn ECG representation by spatial-temporal masking modeling and reconstruction of 12-lead ECG signals. Although all these unimodal methods have achieved good performance, they still fall short in learning the clinical semantics of ECG signals [Liu et al., 2024b]. Single-modal contrastive and generative methods extract representations only from ECG signals without diagnostic reports.

Multi-modal ECG Representation Learning. Several works conduct ECG multi-modal learning for better classification. Ref. [Raghu et al., 2022] proposes to learn representations from ECG signals and structured data from labs and vitals by contrastive learning. Ref. [Lalam et al., 2023] combines ECG signals with structured Electronic Health Records (EHRs) to conduct contrastive learning. BPNet fuses ECG signals with PPG signals to better conduct blood pressure estimation [Long and Wang, 2023]. However, these methods do not use diagnostic report data, making it difficult to learn the clinical semantics effectively. To learn the clinical semantics of ECG signals, Liu et al. proposed to align ECG features with clinical reports inspired by multi-modal learning in medical images and radiology reports [Liu et al., 2024a; Liu et al., 2024b]. Introducing corresponding diagnostic reports for ECG representation learning greatly improves their performance on the downstream cardiac condition classification tasks, but these multimodal approaches only achieve shallow modal interaction. The learned representations can not efficiently incorporate the semantics in the reports. Therefore, we design our DERI framework to conduct deep crossmodal interaction and then expand the model to report generation with great meaning for clinical diagnosis.

Clinical Report Generation. Clinical report generation in radiology has obtained great performance inspired by imaging captioning [You et al., 2021]. R2Gen [Chen et al., 2020b] uses a memory-driven Transformer to generate a radiology report directly with the representation of the medical image. CvT2DistilGPT2 [Liu et al., 2023b] demonstrates that pretrained NLP models can provide benefits for radiology report generation as well. X-REM is proposed to fuse the imagetext multi-modal representation and then used retrieval-based methods to generate the predicted reports from the retrieval corpus [Jeong et al., 2024]. Inspired by these image-based clinical report generation methods, we extend our ECG representation learning method to ECG-based clinical report generation, which can help understand the clinical semantics of the cardiac conditions from the ECG signals.

3 Methodology

3.1 Overview

Our DERI is a dual-encoder framework for learning effective multimodal representations from ECG signals and clinical reports. As shown in Fig. 1, it leverages feature alignment and reconstruction for cross-modal interaction, with an RME module further enhancing representation quality.

3.2 Multiple ECG-Report Alignment

The Multiple ECG-Report Alignment in DERI contains two strategies: modal-specify and mix-modal feature alignment. Given an ECG signal recording ei with corresponding clinical report ri, we construct an ECG-Report pair as (ei,ri), with i = 1, 2, 3, ...,N where N is the number of recordings. Two distinct encoders Fe and Ft are used to learn the latent encoding of ECG signals and report texts respectively, represented as ze,i and zt,i. Specifically, the latent encoding is obtained by ze,i = Fe(ei) and zt,i = Ft(ri). To align the ECG encoding and text encoding, we use two linear projectors Pe and Pt to map them into an alignment space of the same dimension, which can be represented as ˆ Ae,i = Pe(ze,i) and ˆ At,i = Pt(zt,i). The align loss Lalign, which is inspired by the CLIP loss, is used to close the distance between the representations of ECG signals and reports in the alignment space. Specifically, each ECG signal and the corresponding report are regarded as a positive pair and others as negative pairs. The loss function Lalign is shown as Eq. 1 to Eq. 3.

where B is the batch size, x ∈ [e, t], y ∈ [e, t] represents the ECG modal or the text modal, τ is the temperature coefficient which is set as 0.7, and I is the indicator function. Through calculating the mutual align loss of ECG-report Le,t i,j and report-ECG Lt,e i,j respectively, themodel can performbetter feature alignment.

Furthermore, to conduct deep ECG-report interaction, we adopt Cross-modal Mutual Reconstruction (depicted in Section 3.3) to decode the representation of another mode (De and Dt represent decoded ECG feature and decoded text feature) from the aligned modal representation. Then the decoded features are added to the aligned modal representation to obtain the mix-modal encoding. By combining alignment and reconstruction, mixed-modal can better achieve deep ECG-Report interaction. Specifically, we will use the ˆ Ae,i and ˆ At,i as the core to obtain the mix-modal encoding ˆM e i = ˆ Ae,i ⊕ Dt and ˆM t i = ˆ At,i ⊕ De, and then conduct the second encoding alignment in mixed space, thus obtaining the final multimodal representation. The mixed alignment loss can be calculated as Eq. 4.

where L′e,t i,j and L′t,e i,j is calculated by using ˆM e i and ˆM t i to replace ˆ Ae,i and ˆ At,i as Eq. 2 and Eq. 3. Therefore, our proposed method can effectively extract mixed modal representations with report characteristics by using only ECG signals after completing the pre-training stage. This can help the model better complete the task of zero-shot classification and report generation.

In conclusion, the whole loss for multiple ECG-report alignment can be written as Eq. 5:

3.3 Cross-modal Mutual Reconstruction

To better guide the model in achieving deeper modal interactions between ECG signals and diagnostic reports, we introduce cross-modal mutual reconstruction. Specifically, after we obtained the aligned ECG feature ˆ Ae,i and the aligned text features ˆ At,i, we aim to facilitate modal interactions by reconstructing the target modality while bringing them closer to each other in space. We introduce transformers as decoders to decode the representations of one modal in the alignment space to another modal. Considering that reports offer intuitive semantic information valuable for heart state classification but are often unavailable without cardiologists, we introduce a shared embedding derived from the textual modality decoder for better learning textual semantics during pretraining. This shared embedding is combined with the ECG features, enriching them with additional textual features to enhance cardiac condition classification. After completing the pre-training in this manner, the final representation obtained from inputting only the ECG data effectively encapsulates the semantic information of the corresponding report text. This process is represented as Eq. 6:

Ae,i, SEt]), (6) where De and Dt are the decoder transformers to obtain ECG encoding ˆDe,i and report encoding ˆDt,i respectively, and SEt is the shared embedding. Then we use standard contrastive loss on the original feature embeddings and the decoded embeddings for cross-modal mutual reconstruction as Eq. 7:

where LeD and LtD represents the loss of ECG and report feature reconstruction respectively, Lze,de i,j and Lzt,dt i,j represent to use De and Dt with the original features ze,i and zt,i to calculate the similarity as the same of Eq. 3.

3.4 Latent RandomMasking Enhancement

We further conduct an RME-Module on latent ECG encoding to facilitate representation learning. Considering that augmentation directly at the data level entails the loss of semantic information about the signal, we propose an RME-Module conducting on the ECG encoding. Specifically, as the encoder extracts encodings, it tends to focus on the local features of the signal to form an encoding sequence. Rather than applying global average pooling to the ECG encoding sequence and using two separate dropout operations to create augmented views, we instead randomly mask the sequence twice, independently. Then, a multi-head attention mechanism is employed to aggregate the sequence, producing two augmented views of the encoding as a positive pair. This random masking approach helps preserve sequence-level semantic features while enabling the model to learn global features more effectively. We then use standard contrastive loss on these two augmented encoding views. The whole process can be illustrated in Eq. 8:

where si,i = z1 ⊤ e,i z2 e,i , MHA is multi-head attention, Mask is the random mask strategy, which generates random mask sets M1 and M2. M1 and M2 with each entry independently sampled with masking ratio p = 0.1 are in Rb×n where b is the batch size and n is the length of the embedding sequence. Each item inM is either 0 or 1, indicating whether the corresponding patch should be masked. We add a global average on the MHA to obtain the global representation of the masked embedding. Importantly, the random masks are generated by two independent random noises.

In summary, our proposed DERI learns representative ECG features with the help of clinical reports by jointly minimizing LERA, LCMR and LRME. The overall training loss of pretraining can shown as Eq. 9:

3.5 DownstreamTasks on DERI Framework

After training the proposed DERI model, we can obtain an effective representation of ECG signals that contains clinical report information. Then we can use the representation to conduct zero-shot classification and report generation. Considering the quality of the category prompts for zero-shot classification will have a great impact on the performance [Maniparambil et al., 2023], we adopt the CKEPE prompts which are constructed by combining large language model (LLM) and clinical knowledge [Liu et al., 2024b]. The whole process of these two tasks is illustrated in Fig. 2.

Zero-shot Classification. We adopt CKEPE as the category prompts and use the trained report encoder Ft and the projector to obtain the prompt embeddings of all categories. We then use the trained DERI model to obtain the Mix Encoding with ECG signals alone, which contains both ECG signal features and the corresponding clinical report features. Finally, we calculate the similarity between the Mix Encoding and the prompt encoding and then conduct an optimal classification threshold search, all categories above this threshold are considered to be predicted. Importantly, all the parameters of the proposed DERI are frozen in this process.

Report Generation. After we obtain the final representation of ECG, we adopt GPT-2 as the text decoder to construct an encoder-decoder structure since DistilGPT2 [Sanh, 2019] has shown its great performance on report generation [Wang et al., 2024]. We adopt a trainable linear layer to transform the input encoding dimension to meet the dimension of the GPT-2 and perform fine-tuning on the GPT-2 to minimize a cross-entropy loss LCE between the generated report and ground truth reports. After fine-tuning, we can generate corresponding diagnostic reports with ECG signals alone.

4 Experiment

4.1 Datasets

MIMIC-ECG. The pre-training process of our proposed DERI model is conducted on the MIMIC-ECG dataset [Gow et al., 2023] with 800,035 paired ECG-report from 161,352 subjects. We removed all the samples without reports containing more than 3 words and replaced 'NaN' or 'Inf' in the ECG signal with mean interpolation. Finally, the dataset used for pre-training has 771,693 samples.

PTBXL [Wagner et al., 2020] contains 21,837 12-lead ECG signals from 18,885 patients at a sampling rate of 500 Hz with a duration of 10s. It contains four multi-label classification tasks: Superclass, Subclass, Form, and Rhythm.

CPSC2018 [Liu et al., 2018] contains 6,877 12-lead ECG recordings with a sampling rate of 500 Hz. The duration of these signals ranges from 6 to 60 seconds with one corresponding label within nine categories.

Chapman-Shaoxing-Ningbo (CSN) [Zheng et al., 2022] contains 45,152 12-lead ECG recordings with a sampling rate of 500 Hz. Each recording has a duration of 10 seconds, and signals with "unknown" annotation are removed. 23,026 ECG records with 38 categories are used for classification.

4.2 Experimental Setup

Pre-training. For the encoders used for ECG signals and reports, we adopt a randomly initialized 1D-ResNet18 and the Med-CPT [Jin et al., 2023], respectively. For decoders, we adopt two transformers with 8 attention heads, a depth of 2, and a hidden size of 256, respectively, for ECG encoding reconstruction and report encoding reconstruction. We use the AdamW optimizer with a learning rate of 1e-3 and a weight decay of 1e-8. The epoch for pre-training is set as 50 with a cosine annealing scheduler to adjust the learning rate. We conduct all the pre-trained experiments on 4 NVIDIA GeForce RTX 4090 GPUs with a batch size of 512.

Classification. We freeze the whole DERI and conduct zero-shot classification as illustrated in Section 3.5. For linear probing, we add a new linear classifier and freeze all other parameters in our DERI. We adopt three different settings, which utilize 1%, 10%, and 100% of the training data. Since these tasks are all classifications that contain many categories, we adopt the macro AUC as the evaluated metric. We conduct these experiments on one NVIDIA GeForce RTX 4090 GPU. The baselines we compared include SimCLR [Chen et al., 2020a], BYOL [Grill et al., 2020], BarlowTwins [Zbontar et al., 2021], MoCo-v3 [Chen et al., 2021], SimSiam [Chen and He, 2021], TS-TCC [Eldele et al., 2021], CLOCS [Kiyasseh et al., 2021], ASTCL [Wang et al., 2023], CRT [Zhang et al., 2023], STMEM [Na et al., 2024], and MERL [Liu et al., 2024b]. More details about implementation settings and baselines can be found in the Supplementary.

Report generation. We conduct report generation on the MIMIC-ECG dataset first. We adopt the Natural Language Generation (NLG) metrics which include BLEU-n and ROUGE-L as metrics. We further integrated Clinical Efficiency (CE) metrics inspired by zero-shot classification, as seen in our Supplementary. We conduct the experiments on 2 NVIDIA GeForce RTX 4090 GPUs.

4.3 Experimental Results

Classification. Since most of the existing ECG representation learning methods are proposed without a text encoder for zero-shot learning, we compared our proposed DERI with MERL to verify the cross-modal ECG representations learned from clinical reports. The comparison results are illustrated in Fig. 3. It is evident that our proposed method, DERI, significantly outperforms MERL across all tasks. The dotted line in the figure indicates the average performance of the two zero-sample methods on the six classification tasks. DERI achieves an average macro AUC of 78.73, while MERL attains only 75.25. This underscores DERI's capability to learn clinically relevant representations through deep cross-modal interactions. Compared to MERL which just aligns the ECG and report encoding, our proposed DERI achieves deep crossmodal interaction by multiple alignment and feature reconstruction, which enables the model to learn more effective representation for zero-shot clinical classification.

We evaluate our DERI framework using linear probing, a widely adopted protocol in SSL [Wang et al., 2023], and compare it with existing ECG SSL methods. As shown in Table 1, DERI outperforms all baselines, including the multi-modal method MERL and state-of-the-art single-modal methods, across all datasets and training data ratios. "Random Init" denotes using DERI in a purely supervised setting without pre-training. Notably, DERI achieves the largest performance gain when trained with only 1% of labeled data, indicating strong generalization with minimal supervision. On the PTBXL-Super task, DERI, with just 1% data even surpasses all single-modal methods trained with 100% data. Moreover, both MERL and DERI exhibit strong performance across settings, confirming the benefit of integrating clinical reports. DERI's consistent advantage over MERL highlights its superior capability in cross-modal ECG representation learning for cardiac condition classification.

To assess the robustness of learned representations under distribution shift, we conduct linear probing with 100% training data on three classification tasks: PTBXL-Super, CPSC2018, and CSN. For zero-shot capable models like MERL and DERI, we reclassify their fixed representations using trained classifiers. Specifically, models are trained on one dataset (source domain) and evaluated on another (target domain). As shown in Table 2, DERI consistently outperforms MERL across all six transfer settings. Compared to single-modal SSL baselines, only STMEM achieves better performance than DERI in the PTBXL-to-CSN setting, while DERI outperforms all others in the remaining cases. MERL generally ranks second, demonstrating the benefit of incorporating clinical reports. The consistent improvements of DERI over MERL further validate the effectiveness of its enhanced cross-modal interaction in improving generalization and robustness across domains.

Report Generation. Beyond cardiac condition classification, DERI enables deep cross-modal interactions that allow the extracted ECG representations to support diagnostic report generation. To evaluate this capability, we conduct report generation experiments on the MIMIC-ECG dataset. We use pre-trained DERI and MERL as encoders and integrate them with DistilGPT2 in an encoder-decoder framework. To isolate the contribution of our cross-modal reconstruction strategy, we introduce a variant of DERI-align, which uses only the aligned ECG representations. As shown in Table 3, DERI outperforms MERL in both NLG and clinical efficacy (CE) metrics, indicating that it captures richer clinical semantics. Moreover, DERI also surpasses DERI-align, demonstrating the effectiveness of our cross-modal feature reconstruction in incorporating diagnostic report information and enhancing semantic fidelity in generated reports.

To better verify the CE metrics calculation method, we also adopt large language models LLaMA2-7b [Touvron et al., 2023] and vicuna-7b [Zheng et al., 2023] to conduct report classification. Specifically, we feed the original reports and generated reports to the LLMs respectively and then ask the LLMs to choose the best class from six given categories: Normal ECG, Myocardial Infarction, ST/T Change, Conduction Disturbance, Hypertrophy, and Others. The answers of the original reports are regarded as ground truth and the answers of the generated reports are predicted labels. We then calculate the CE metrics as Fig. 4.

We can observe that the results of report classification using LLMs are basically the same as the results of our zeroshot categorization method: DERI is the best and DERIalign is the second best, while both methods outperform MERL on F1. Meanwhile, the calculation of CE using vicuna performs better results than LLaMA2. We also provide example-generated reports and report-generation tasks on PTB-XL compared with MEIT [Wan et al., 2024] and ECG-Chat [Zhao et al., 2024] in Supplementary.

4.4 Ablation Study

To better verify the performance of the key components/design choices of our DERI, we conduct comprehensive ablation studies on zero-shot classification and linear probing with 1% training data across different classification datasets. All results are proposed with average AUC on six downstream datasets for ECG classification.

Multiple Alignment and Feature Reconstruction. We realize the validation of the effect of these compositions by ablating the corresponding loss functions separately, and the experimental results are reported in Table 4. Table 4 shows that for zero-shot classification, Lmixed brings the best improvement while LeRME for linear probing. This suggests that the second alignment of mixed encoding can better fuse the clinical information from reports because zero-shot classification learning is conducted to calculate the similarity between the learned representations and prompt representations. The effect of LeRME on linear probing exemplifies the effectiveness of the RME-Module we designed to improve the learning ability of ECG representations. The completed DERI obtains the best performance, which illustrates the effectiveness of our DERI method.

RME-Module. We compare the effect of the RMEModule and its variant that uses linear projectors instead of the attention mechanism (which is set as RME-Linear) and the Latent Dropout strategy used by MERL. The experimental results are reported in Table 5. We can observe that the random masking strategy obtains better performance than dropout while using the multi-head attention mechanism instead of global meaning can achieve the best performance, enhancing the model's ability to learn ECG representation for classification. Masking encourages the model to learn context-aware representations, focusing on understanding relationships within the input. Random dropout removes (i.e., zeroes out) a fraction of the neurons (units) or edges in a network layer during training, but it does not apply this to the input itself. In dropout, neurons are randomly dropped independently at each forward pass. The designed RME-Module is used to enhance the global feature of the ECG signals, so we adopt random masking rather than a random dropout.

We then further explore the impact of the masking ratio p on the performance by changing it from 0.1 to 0.5 with a step of 0.1. The experimental results are shown in Table 6. We observe that the masking ratio of 0.1 obtains the best performance among other masking ratios in both zero-shot classification and linear probing. Therefore, we adopt the masking ratio of 0.1 in our RME-Module.

Shared Embedding and Mix Encoding. We also conduct experiments to verify the effect of the shared embedding used in cross-modal reconstruction. We remove the shared embedding from Dt to De as a variant of DERI. In addition, we use the pre-trained aligned ECG features to conduct zeroshot classification and linear probing to verify whether the mix encoding performs better than the aligned encoding for classification. This means that we adopt the same pre-training model but use the aligned ECG encoding instead of the mixed encoding for downstream tasks.

The experimental results are shown in Table 7. It can be observed that removing the shared embedding from Dt to De during text encoding reconstruction leads to a decline in model performance for both zero-shot and linear probing tasks. Furthermore, using the mixed encoding, which includes the decoded report features, outperforms using only the aligned ECG features. These findings underscore the strong representation learning capability of DERI.

5 Conclusion

In this study, we proposed DERI, an innovative deep ECGReport interaction framework for cross-modal representation learning. To obtain deep ECG-Report interaction for better representation learning, we design multiple alignments and cross-modal mutual reconstruction. Besides, an RMEModule is conducted on the ECG latent encoding for representation learning enhancement. Moreover, we extended ECG representation learning to clinical diagnostic report generation, aiming to deliver more intuitive ECG clinical insights. Our extensive experiments demonstrate the DERI's capability to learn the clinical semantics of ECG signals with the help of reports, which achieves the best performance on ECG classification and report generation.

Ethical Statement

There are no ethical issues.

Acknowledgments

This work was supported in part by the Sustainable Development Science and Technology Project of Shenzhen Science and Technology Innovation Commission under Grant KCXFZ20201221173411032.