MedFiTRG: Jointly Learning Dynamic Temporal and Cross-Patient Graphs for Clinical Outcome Prediction

1 Introduction

Electronic Health Records (EHRs) have become a crucial component in modern healthcare, facilitating accurate and timely clinical decision-making through the integration of heterogeneous patient data. With the advancement of deep learning techniques, EHRs now power predictive models across tasks such as risk assessment [Wang et al., 2024; Bai et al., 2018; Luo et al., 2020], clinical decision support [Rajkomar et al., 2018; Gupta et al., 2025; Chen et al., 2024a], and personalized medicine [Zhang et al., 2023; Ahmed et al., 2023; Zhu et al., 2024]. To achieve this, clinical decision support (CDS) systems increasingly rely on the fusion of diverse modalities, including structured EHR, medical imaging, and unstructured clinical narratives [Al-Zoghby et al., 2025; Jandoubi and Akhloufi, 2025]. Each modality provides a distinct and complementary view of patient health, EHR captures longitudinal physiological measurements and interventions, clinical text encodes diagnostic reasoning, and medical images such as chest X-rays (CXR) reveal latent pathological patterns. While modality-specific models have advanced significantly, real-world clinical decision making requires effective integration across modalities and time [Siam et al., 2025; Teles et al., 2025]. In critical care settings in particular, accurate prediction of outcomes such as multi-disease risk, length of stay (LOS), and mortality depends on jointly modeling temporal dynamics, heterogeneous data types, and complex patient-level interactions. Despite growing interest in multimodal clinical prediction, several fundamental challenges remain:

∗Corresponding author

Modality Heterogeneity and Semantic Misalignment: Multimodal medical data consist of fundamentally different representations, where EHR time-series encode longitudinal clinical events, chest X-rays capture spatial visual patterns, and clinical notes describe subjective and contextual observations. These modalities operate at different semantic granularities and levels of reliability, making direct fusion a challenging task.

Modality Sparsity and Imbalanced Clinical Evidence: In real-world clinical settings, modality availability is highly imbalanced, with structured EHR data present for nearly all visits, while chest X-ray images are available for only a small subset of encounters. As a result, conventional models trained with uniform aggregation or static fusion strategies tend to ignore or underweight sparse modalities, effectively treating them as auxiliary noise rather than informative clinical evidence. This imbalance results in the underuse of critical visual information, particularly in acute scenarios where imaging provides essential diagnostic cues, posing a significant challenge for learning well-calibrated multimodal representations.

Dynamic Integration of Temporal and relational Dependency: Existing studies on multimodal clinical prediction typically decouple temporal and relational reasoning, first modeling intra-patient evolution, then applying a static or global patient graph. Such sequential or hierarchical designs overlook synchronous temporal-relational dependencies, where visits from different patients evolve and interact simultaneously over time. The absence of frameworks capturing these visit-level, cross-patient temporal connections limits current models' ability to reflect real-world clinical dynamics and population-level disease progression.

To address the above-mentioned challenges, we introduce modality-enhanced dynamic temporal relational graph (MedFiTRG), a unified framework to integrate modality-aware GNN encoding with a learnable temporal relational graph (TRG) for dynamic, multimodal patient modeling across time and population. MedFiTRG jointly encodes EHR, text, and imaging into a global context vector and refines each modality using modulated graph neural networks (MGNN) inspired by GNN-FiLM [Brockschmidt, 2020]. Unlike conventional GNN-FiLM, which statically modulates node features, our MGNN dynamically learns FiLM parameters from fused multimodal and relational contexts, enabling contextaware, hierarchical modulation across modalities, time, and patient graphs. This modulation enables context-aware alignment of heterogeneous feature spaces and meaningful utilization of sparse modalities. Subsequently, MedFiTRG builds a TRG, where each node represents a patient visit, with directed intra-patient edges capturing temporal evolution and learnable inter-patient edges encoding dynamic cross-patient similarity. This design allows synchronized reasoning across time and population, capturing multi-dimensional temporal and relational dependencies essential for holistic clinical prediction. The key contributions of this work are summarized as follows:

• MGNN: We employ a modulated GNN to learn modality-specific representations for EHR, chest X-rays, and clinical text. This adaptive modulation mechanism performs feature-wise scaling and shifting based on global contextual information, effectively aligning heterogeneous feature spaces and mitigating semantic misalignment across modalities.
• TRG: We construct a TRG, where intra-patient edges capture longitudinal evolution and inter-patient edges are dynamically learned through a similarity-based threshold, facilitating adaptive relational reasoning across patients.
• Contextually Modulated Message Passing: We perform message passing using MGNN over both temporal and relational edges, jointly capturing temporal progression, population-level relations, and modality interactions in an end-to-end manner.
• Extensive Experimental Validation: Extensive experiments on large-scale real-world datasets, MIMIC-IV, MIMIC-CXR-JPG, and MIMIC-IV-NOTE, show that MedFiTRG achieves superior or comparable performance against strong multimodal baselines, improving

LOS Macro-F1 from 0.155 to 0.310 and achieving an AUROC of 0.939 for mortality prediction.

2 RelatedWork

In this section, we conduct a comprehensive review of the literature.

Multimodal Clinical Representation Learning: Recent advances in multimodal learning have demonstrated that jointly leveraging imaging, clinical text, and structured EHR data substantially improves clinical prediction and patient modeling [Xu et al., 2023; Zhong et al., 2024; Zhang et al., 2022; Yao et al., 2024; Ruan et al., 2025]. Earlyfusion frameworks such as HAIM [Soenksen et al., 2022] and HEALNet [Hemker et al., 2024] integrate pre-trained modality encoders with attention-based fusion to align heterogeneous features while preserving modality-specific semantics. Subsequent models like VecoCare [Xu et al., 2023], M3Care [Zhang et al., 2022], and DrFuse [Yao et al., 2024] further refine cross-modal learning through adaptive weighting and contrastive consistency objectives, enhancing robustness across diverse clinical outcomes. However, most existing approaches struggle with incomplete or irregularly sampled modalities, a common limitation in real-world EHR systems. MedFuse [Hayat et al., 2022] addresses this by employing LSTM-based fusion to handle partial inputs, while Lee et al. [Lee et al., 2023] design modality-aware attention mechanisms to mitigate EHR data missingness. Similarly, DrFuse [Yao et al., 2024] introduces shared-distinct embedding spaces to improve modality alignment under sparse conditions. Despite these efforts, current methods often fuse modalities at feature level without explicitly modeling temporal progression or inter-patient relationships.

Temporal and Graph-based Clinical Prediction: Temporal and graph-based models have become pivotal for capturing both the dynamic evolution and relational structure of clinical data. Sequence-based architectures such as HiTANet [Luo et al., 2020] and StageNet [Gao et al., 2020] effectively model intra-patient temporal dependencies but lack mechanisms to represent population-level relationships. To jointly capture sequential and structural signals, TRANS [Chen et al., 2024b] formulates heterogeneous temporal graphs that represent visits and medical events as interconnected nodes, while Multi-TA [Sohn et al., 2024] enhances temporal robustness via BERT-based contextual embeddings for early disease detection. Beyond patient timelines, Tang et al. [Tang et al., 2023] introduce multimodal spatiotemporal GNNs that leverage similarity graphs to propagate information across related patients, and Xu et al. [Xu et al., 2024a] design temporal neighbor transformers to handle asynchronous and incomplete data using missingness-aware prompts.

In summary, despite advances in multimodal fusion, temporal modeling, and graph-based learning, existing methods still face challenges with heterogeneous and incomplete clinical data. Most frameworks struggle to generalize under modality imbalance and lack scalable, interpretable architectures for fine-grained cross-modal reasoning, limiting their effectiveness in real-world healthcare settings.

3 ProblemFormulation

Let each patient pi (i = 1, 2, . . . ,N) have a chronological sequence of clinical visits Vi = {vi,1, vi,2, . . . , vi,Ti }, where Ti denotes the total number of visits for patient pi. Each visit vi,t consists of multimodal data, including structured EHR features xehr i,t , clinical text embeddings xtext i,t , and imaging features ximg i,t .

Our goal is to learn a predictive function fθ that, for each patient pi and visit vi,t, estimates the clinical outcome ˆ yi,t using all available information up to and including visit t:

In this formulation, the model performs per-visit prediction, where each prediction at time t leverages the patient's historical trajectory up to that point. For the first visit (t = 1), the model relies solely on the current visit's data, whereas subsequent predictions progressively incorporate both past and current multimodal representations. During training, the model learns from all patients {V1, V2, . . . , VN} to optimize θ for accurate visit-level outcome prediction. It predicts a clinical outcome ˆ yi,t (e.g., in-hospital mortality (IHM), readmission (REA), length of stay (LOS), and phenotyping (PHE)) by jointly modeling temporal evolution and relational dependencies across patients.

4 Methodology

MedFiTRG is a unified multimodal framework that jointly models modality-specific representations, longitudinal patient trajectories, and population-level dependencies. The architecture follows a hierarchical fusion design based on feature-wise linear modulation (FiLM), enabling contextaware interaction across modalities, time, and patients. Figure 1 illustrates the overall architecture.

4.1 Modality-Aware Representation Learning

Consider a patient cohort P = {p1, . . . , pN}, where each patient pi has a sequence of visits Vi = {vi,1, vi,2, . . . , vi,Ti }. Each visit consists of structured EHR features xehr i , chest Xray images ximg i , and clinical text xtxt i . Modality-specific encoders transform each input into a latent embedding emi,t ∈ R512 , where m ∈ {ehr, img, text}. To robustly handle missing modalities, we replace absent inputs with learnable modality-specific missing tokens, allowing the model to optimize default representations rather than relying on zeroimputation. Auxiliary prediction heads are attached to each unimodal encoder to provide deep supervision and enforce discriminative feature learning prior to fusion.

4.2 Symmetric Modality Fusion via FiLM

To achieve coherent cross-modal alignment, we adopt a symmetric FiLM-based fusion strategy that treats each modality equally while conditioning on the shared multimodal context. For each visit, modality embeddings are concatenated and passed through a lightweight multilayer perceptron (MLP) to produce a global visit context:

This global context captures interactions and dependencies across modalities, effectively summarizing the visit as a unified latent descriptor. The FiLM generator gϕ then produces modality-specific scaling and shifting coefficients (γm, βm) based on this shared context:

Each modality embedding is subsequently modulated as:

Through this modulation, the FiLM mechanism adaptively emphasizes or suppresses modality features according to their relevance in the overall clinical context. The key advantage of this symmetric conditioning is that all modalities influence one another through the same contextual signal, promoting balanced contribution and preventing dominance by high-dimensional modalities such as imaging. The modulated embeddings are finally aggregated to yield a fused visit representation zi,t, which encodes both modality-specific evidence and their interdependencies in a compact form suitable for downstream temporal reasoning. The final fused visit representation is obtained via element-wise aggregation of the modulated modality embeddings:

where the summation denotes point-wise addition across modalities of equal dimensionality. This formulation preserves modality-specific semantics while enabling joint representation learning in a shared latent space.

4.3 Temporal MGNN for Longitudinal Modeling

While visit-level fusion provides a comprehensive snapshot of multimodal information, patient trajectories evolve dynamically over time. Diseases progress gradually, and their manifestations in EHR, imaging, and text often exhibit nonlinear temporal dependencies. To model such evolution, we adopt a first-order Markov assumption, where the patient's current clinical state depends primarily on its immediate historical context [Choi et al., 2016]. This assumption aligns with common temporal patterns in longitudinal EHR data, where each visit reflects an incremental transition from the most recent medical state.

Under this Markovian formulation, we introduce a temporal MGNN that performs context-aware reasoning over sequential visits. Specifically, each visit embedding zi,t is conditioned on its immediate predecessor zi,t-1 through a FiLMbased modulation mechanism:

where the FiLM operation applies a feature-wise affine transformation conditioned on the previous visit embedding:

with γ(·) and β(·) being learnable functions (e.g., MLPs) that generate feature-wise scaling and shifting parameters from the context zi,t-1. The previous visit representation zi,t-1 thus serves as a dynamic conditioning context, modulating the current visit embedding zi,t to capture first-order dependencies between consecutive visits while filtering redundant temporal noise and emphasizing clinically meaningful deviations.

The residual connection preserves current visit information, enabling stable gradient flow and preventing historical overdominance. This formulation disentangles short-term variations from long-term progression, capturing clinically relevant transitions such as treatment response or acute deterioration. Unlike recurrent models (e.g., LSTMs or GRUs) that implicitly encode sequential memory, our Markov-based Temporal MGNN explicitly parameterizes inter-visit influence through learnable modulation functions, enhancing interpretability and robustness to irregular sampling intervals.

4.4 Adaptive Social Relational Fusion

Beyond temporal self-dependence, patients within a population exhibit shared clinical patterns, identified through embedding-level similarities that reflect underlying demographic, phenotypic, and comorbidity factors. To capture these cross-patient relationships, we construct a dynamically learned patient similarity graph that allows information propagation across related individuals. For each pair of patients (i, j) within a mini-batch, we compute an adaptive edge weight:

where cos(·) denotes cosine similarity, θ is a learnable threshold controlling edge sparsity, and τ is a temperature parameter regulating the sharpness of connectivity. This adaptive formulation enables the graph topology to evolve with training, reflecting emergent clinical clusters or patient subtypes. The aggregated social context is then integrated via MGNN:

In this stage, each patient representation htemp i is refined by adaptively modulating information aggregated from its clinically similar neighbors. The FiLM function acts as a contextual gate, determining how social information should alter the patient's individual trajectory. The inclusion of tanh(∑j Aij) provides a soft normalization that controls the influence of neighborhood density. This design ensures that population-level patterns, such as treatment responses or risk factors shared among subgroups, inform individual-level predictions without overwhelming personalized signals.

Overall, this adaptive social relational fusion bridges individual and cohort-level learning: temporal FiLM captures intra-patient evolution, while graph-based FiLM models inter-patient associations, resulting in a robust and interpretable framework for multimodal clinical reasoning. The combined effect is a system capable of symmetric multimodal alignment, dynamic temporal conditioning, and socially aware graph reasoning, all unified through FiLMdriven modulation principles.

4.5 Outcome Prediction

The final representation hsocial i integrates information across multiple dimensions, capturing multimodal clinical features, longitudinal temporal evolution, and population-level relational dependencies. This embedding serves as a comprehensive patient state descriptor that reflects both individual health dynamics and contextual similarities within the cohort. To perform downstream clinical tasks such as REA, IHM, PHE, and LOS prediction, we employ a task-specific output head implemented as a linear projection followed by an appropriate activation function:

where Wo and bo denote learnable parameters of the output layer, hsocial i represents the social-aware patient representation, and f(·) denotes the task-specific activation function. Specifically, sigmoid activation is used for binary and multilabel tasks (REA, IHM, and PHE), while softmax activation is employed for LOS classification.

5 Loss Function Formulation

To address severe class imbalance across heterogeneous prediction tasks, we adopt a focal loss-based optimization scheme. The learning objectives are categorized into three groups: binary, multi-label and multi-class tasks.

Binary Tasks (IHM, REA): For binary classification, the model outputs a logit z converted to probability p = σ(z). The focal loss is defined as:

where pt = p if y = 1 else 1 - p, γ = 2 is the focusing factor, αt ∈ {α, 1 - α} balances positive/negative samples, and λcls = 1 + y(wpos - 1) upweights rare positive cases.

Multi-Label Task (PHE): For phenotyping, each of the C labels is optimized independently using the same focal loss:

Multi-Class Task (LOS) For LOS, the task is formulated as a K-class classification problem with K = 10 discrete time buckets. Given logits z ∈ RK , class probabilities are computed using the softmax function pk = exp(zk)/ ∑Kj=1 exp(zj). With one-hot encoded targets y ∈ {0, 1}K and a precomputed class-weight vector w, the multiclass robust focal loss is defined as

where wk denotes the inverse class-frequency weight, computed as wk = N/(K · Nk), with N representing the total number of training samples, K the number of classes, and Nk the number of samples belonging to class k.

To promote modality-specific discriminative representations, auxiliary classifiers are attached to unimodal encoders and trained via deep supervision. Auxiliary losses are computed only over samples where the corresponding modality is available. The final objective minimizes the weighted sum of the fused prediction loss and auxiliary losses:

where λcxr and λtext are learnable parameters.

6 Experiments

In this section, we detail the experimental setup and present a comprehensive analysis of the results.

6.1 Experimental Settings

Datasets

In this study, we leverage three complementary components of the MIMIC database ecosystem: MIMIC-IV, MIMICCXR-JPG, and MIMIC-IV-Note datasets [Johnson et al., 2019; Johnson et al., 2023]. These datasets share a common patient cohort, enabling seamless alignment across modalities. We extract longitudinal structured EHR data from MIMIC-IV, chest X-ray images fromMIMIC-CXR, and clinical narratives from MIMIC-IV-Note to construct a unified, multimodal patient representation that captures temporal, visual, and textual clinical information. The resulting dataset is divided into training, validation, and testing subsets following a 70:10:20 ratio based on unique patient identifiers to ensure patient-level independence across splits. Summary statistics are reported in Table 1.

Tasks

To comprehensively evaluate our framework, we benchmark it on four clinically relevant prediction tasks derived from the MIMIC-IV dataset:

• In-hospital mortality (IHM): Predicts whether a patient dies during the current ICU stay based on the first 48 hours of data (binary classification).
• Length of stay (LOS): Estimates the remaining ICU stay, categorized into 10 discrete intervals: one for stays under a day, seven for each subsequent day up to a week, one for stays between one and two weeks, and one for stays exceeding two weeks. (multi-class classification).
• Phenotyping (PHE): Identifies 25 acute conditions associated with an ICU episode (multi-label classification).
• Readmission (REA): Predicts hospital readmission within 30 days post-discharge (binary classification).

6.2 Baselines

To rigorously evaluate our proposed MedFiTRG framework, we benchmark it against several state-of-the-art multimodal models addressing missing modality and temporal integration. MedFuse [Hayat et al., 2022] introduces an LSTMbased fusion module capable of handling both unimodal and multimodal inputs, enabling flexible integration across heterogeneous data sources. MT-Transformer [Ma et al., 2022] employs a transformer architecture with attention masks to effectively fuse multimodal data under missing modality conditions. M3Care [Zhang et al., 2022] imputes missing modality information within a latent representation space by leveraging neighboring patients with similar clinical patterns. MMF [Lee et al., 2023] adopts a modality-aware attention mechanism combined with a skip bottleneck structure to learn robust EHR representations despite incomplete modalities. MultiModN [Swamy et al., 2023] sequentially processes any subset of available modalities through modality-specific encoders, dynamically skipping absent inputs during inference. FlexCare [Xu et al., 2024b] dynamically weights available modalities to handle missing data with continual learning on different tasks, and MoEHealth [Wang and Yang, 2025] uses a mixture-of-experts design for adaptive modality selection. For consistency, all baselines were adapted or extended to align with our task definitions and multimodal dataset configuration.

6.3 EvaluationMetrics

To comprehensively assess model performance across different clinical prediction tasks, we employ a combination of classification and ranking-based metrics. For binary classification tasks, we report the area Under the receiver operating characteristic curve (AUROC) and the area under the precision-recall curve (AUPRC). For multi-class prediction task, we compute both macro-AUROC and micro-AUROC. For the multi-label prediction task, we evaluate using macro-F1 and micro-F1 scores to capture both per-class and overall balance between precision and recall. Together, these metrics provide a robust evaluation of the model across different prediction tasks.

7 Experimental Results

In this section, we present and analyze the experimental findings to assess the effectiveness of our model.

7.1 Baseline Comparison

Table 2 presents the performance comparison between the proposed MedFiTRG model and various baseline models across various tasks. All experiments are repeated five times with different random seeds, and we report the mean and standard deviation of performance metrics. The best-performing scores are highlighted in bold, while the second-best are underlined for clarity. For LOS classification, it improves Macro-F1 from 0.155 to 0.310, demonstrating superior handling of minority classes. It also achieves an AUROC of 0.939 for mortality prediction (↑6.45%) and maintains competitive performance on readmission and phenotyping tasks, underscoring its strength in leveraging sparse multimodal clinical data for decision support.

7.2 Ablation Study

To evaluate the contribution of individual components in MedFiTRG, we conducted a comprehensive ablation study by systematically removing key modules from the full model. Specifically, we tested four variants: (1) w/o social graph, where inter-patient relational edges were removed, (2) w/o temporal graph, where intra-patient temporal connections were omitted, (3) w/o global context vector, where modality modulation with the global context vector was removed, and (4) w/o temporal and social graph, where both intra- and inter-patient dependencies were excluded. As summarized in Table 3, across all configurations, we observed consistent performance drops compared to the full MedFiTRG model. For LOS the removal of the temporal and social graph caused the largest decline, underscoring the importance of dynamic relational reasoning. For IHM the elimination of global context vector caused the lowest performance, indicating that MGNN-based feature modulation is crucial for aligning and balancing sparse modalities. These results collectively validate the effectiveness of each component in enhancing multimodal clinical prediction.

7.3 Impact of Different GNNs

To further assess the effectiveness of the MGNN architecture, we replaced it with standard GCN and GAT layers in both the global context generation stage and the temporal-social graph message-passing module. A noticeable degradation in performance was observed across different prediction tasks. The results indicate that conventional graph layers fail to capture fine-grained feature-wise dependencies and modality-specific dynamics, leading to weaker fusion and relational reasoning. In contrast, MGNN provides adaptive feature modulation through learnable scaling and shifting, enabling more expressive and context-aware information propagation, which proves essential for robust multimodal clinical prediction (Table 4).

7.4 t-SNE Plot

To further interpret the learned representations, we visualize the modality-fused embeddings of MedFiTRG using t- SNE for the IHM and LOS prediction tasks. The resulting plots reveal well-separated and compact clusters across outcome categories, indicating that the model effectively captures discriminative patient representations. This clear separation highlights MedFiTRG's ability to integrate multimodal and temporal-relational information into a coherent latent space, enhancing both prediction accuracy and interpretability in complex clinical settings. Figure 2 shows the t-SNE plot.

7.5 Hyperparameter Sensitivity Analysis

We tune hyperparameters via grid search over learning rate {1e-2, 5e-3, 5e-5}, batch size {32, 64, 128}, dropout rate {0.1, 0.2, 0.3}, transformers layers {1, 2, 4, 6}, and hidden dimensions {64, 128, 256}. To evaluate sensitivity, we fix the optimal settings and vary the batch size from 64 to 128. The resulting performance change is modest (0.2-0.5%), suggesting that MedFiTRG is not overly sensitive to hyperparameters. This reflects strong generalization capacity and reduced overfitting risk.

7.6 Scalability and Computational Complexity

We assess the scalability and computational efficiency of MedFiTRG, as summarized in Table 5. The model trains in approximately 770.94 seconds per epoch with 126M parameters, utilizing up to 10.7GB GPU memory. To maintain scalability, inter-patient edges in the social graph are computed within mini-batches rather than the entire patient population. During inference, it achieves an average latency of 1.15 seconds per batch (about 0.014 seconds per patient), demonstrating efficient scalability and suitability for real-time and largescale clinical applications.

7.7 Explainability

To enhance explainability, MedFiTRG offers multi-level explanations across modalities. For medical imaging, we apply Grad-CAM to visualize salient regions in chest X-rays that most influence model predictions. For structured EHR data, Integrated Gradients are used to quantify feature importance and identify key clinical variables contributing to each outcome. Beyond modality-level attribution, the learned relational graph further improves interpretability: edges connect patients exhibiting similar patterns in EHR, text, or imaging embeddings, revealing clinically meaningful relationships across the population as shown in figure 3.

8 Conclusion and FutureWork

In this work, we introduced MedFiTRG, a unified framework for dynamic multimodal clinical prediction that integrates modality-aware representation learning and temporal-relational reasoning. By employing MGNN to learn adaptive, feature-wise modulated embeddings, MedFiTRG effectively mitigates modality sparsity and aligns heterogeneous feature spaces from EHR, imaging, and text. The subsequent temporal relational graph enables joint modeling of intrapatient temporal evolution and inter-patient relational dynamics, capturing multi-dimensional dependencies that conventional static or unimodal models fail to represent. Extensive experiments across three large-scale MIMIC datasets demonstrate that MedFiTRG achieves superior or comparable performance against strong multimodal baselines, achieving LOS Macro-F1 from 0.155 to 0.310, and IHM AUROC of 0.939 for mortality prediction (↑6.45%) and maintains competitive performance on readmission and phenotyping tasks. While MedFiTRG achieves superior or comparable performance against strong multimodal baselines, future work may extend toward population-level modeling using hierarchical or global graph representations, the integration of additional clinical modalities such as vital signs, lab time-series, and pathology, and prospective evaluations to assess real-world clinical utility.

Figure 3: Multi-level explainability in MedFiTRG across imaging, EHR, and patient relations.