Leveraging Implicit Contexts via LLM–Graph Fusion for Temporal Knowledge Graph Reasoning

1 Introduction

Temporal Knowledge Graph (TKG) reasoning aims to model and predict how real-world events evolve over time, and it underpins many time-critical forecasting tasks such as stock market prediction [Zhu et al., 2025], political stability assessment [Gwak et al., 2024], and crisis response planning [Chen et al., 2024], where future outcomes depend on the structured progression of past events. In practice, most TKGs are built from raw text through an information extraction pipeline that converts event mentions into structured temporal facts, typically represented as (subject, relation, object, timestamp) quadruples [Cai et al., 2023a].

However, this pipeline inherently limits what evidence can be used for reasoning. It retains only text that can be explicitly extracted into a fixed quadruple form (e.g., via Named Entity Recognition or Relation Extraction). We term such extractable textual evidence as explicit context. Many informative cues in the source passage, such as explanations, thematic signals, and regional conditions, cannot be cast into a single quadruple. We term such non-extractable yet informative evidence as implicit contexts. Prior studies on automatic knowledge graph construction have observed that extractionbased pipelines often fail to preserve reasoning-relevant implicit contextual semantics from the original corpus [Zhong et al., 2023], leaving them unused by structure-only TKG reasoning models and limiting predictive performance in complex scenarios. Fig. 1 illustrates how implicit cues can constrain subsequent temporal reasoning queries. Therefore, acquiring and utilizing implicit contexts for TKG reasoning remains an important problem.

To enable effective acquisition and utilization of implicit contexts in TKG reasoning, we need to address the following significant challenges:

CH1. How can implicit contexts be systematically acquired and effectively utilized to enhance query representations for temporal prediction? The first challenge concerns how to reliably identify, acquire and distill useful implicit signals from unstructured text for a given temporal query. Unlike explicit contexts that are directly recorded as graph facts, implicit contexts are scattered, heterogeneous, and often noisy, with relevance that varies across time and queries. Effectively exploiting such information therefore requires temporally aligned context selection as well as compact and noise-robust representation learning, so that the distilled semantics can consistently strengthen temporal link prediction.

CH2. How can semantic representations from implicit contexts be fused with structure-constrained temporal representations? The second challenge concerns coherently fusing semantic representations from implicit contexts with graph-based temporal representations that are constrained by both graph structure and temporal dynamics. Temporal graph models produce structure-aware embeddings by explicitly encoding temporal and relational dependencies, whereas representations derived from implicit text are typically highdimensional and lack explicit structural constraints [Cai et al., 2023b]. Bridging this gap requires a fusion mechanism that respects structural and temporal constraints, ensuring that implicit semantic information complements graph-based reasoning and leads to stable temporal predictions.

Our Approach and Contributions. To address these challenges, we propose the Textual-Temporal Graph Fusion Network (TTGFN), a framework that enhances TKG reasoning by acquiring implicit contexts from text and integrating its LLM-encoded representations with graph-based query representations. Rather than augmenting the graph with additional extracted facts or modifying its structure, TTGFN enhances the original prediction query through context-aware semantic fusion. The main contributions of our work are summarized as follows:

• To address CH1, we propose a context-aware inference module that acquires implicit contexts and encodes it into compact semantic priors using LLMs, thereby enhancing query representations for temporal link prediction.
• To address CH2, we propose a structure-constrained fusion mechanism that aligns LLM-derived semantic representations with temporal graph embeddings, enabling stable and synergistic enhanced reasoning.
• Extensive experiments on three TKG reasoning benchmarks demonstrate that TTGFN achieves state-of-theart performance. We also release three implicit-contextaugmented benchmark datasets 1 to facilitate future research in the community.

2 RelatedWork

2.1 Temporal Knowledge Graph Reasoning

Recent advances in TKG reasoning have been largely driven by neural temporal modeling over observed quadruples. Representative methods capture historical dependencies through autoregressive or recurrent dynamics (e.g., RE-NET [Jin et al., 2019]), incorporate relation-aware temporal message passing and updates (e.g., REGCN [Li et al., 2021a]), and further enhance temporal modeling by combining local and global temporal patterns (e.g., TiRGN [Li et al., 2022]). More recent work explores richer temporal signals, such as trendaware representations for evolving event dynamics (Trend [Huang et al., 2024]), or formulates temporal reasoning as long-horizon decision making to improve dependency modeling (LogiRL [Chen et al., 2025a]). Despite these advances, most existing methods still operate primarily on structured quadruples and do not systematically exploit the rich textual context available during graph construction, limiting their ability to leverage implicit background knowledge for temporal prediction.

1 Our code and three implicit-context-augmented datasets are publicly available at https://github.com/tianzeshu/TTGFN.

2.2 Integration of LLMs with TKGs

Recent studies have explored integrating LLMs with graphbased models for prediction and reasoning [Li et al., 2023b; Jin et al., 2024]. In TKG settings, most approaches use LLMs as auxiliary components, such as generating temporal rules, modeling logical constraints, or reformulating forecasting as language-based inference [Wang et al., 2024a; Bai et al., 2025a; Pan et al., 2025]. However, they often overlook leveraging implicit textual context that is discarded during graph construction to systematically strengthen the prediction query. In contrast, our work reuses such implicit contexts and aligns their semantic representations with structure-constrained temporal representations for more expressive TKG reasoning.

3 ProblemDefinition

A TKG is denoted as G = {E, R, T , Q}, where E is the entity set, R is the relation set, T is the timestamp set, and Q is the set of temporal quadruples. Each quadruple is represented as (es, r, eo, t) ∈ Q, where es, eo ∈ E denote the head and tail entities, r ∈ R denotes the relation, and t ∈ T denotes the timestamp. Given a query (es, r, t), the task of TKG reasoning is formulated as predicting the tail entity eo.

4 Methodology

Fig. 2 illustrates our framework for temporal link prediction as a unified three-stage pipeline. Stage I retrieves implicit context snippets for the query entity using reproducible strategies. Stage II encodes the retrieved context and query information into dense, context-aware representations via a LLM. Stage III integrates these representations with graph-guided reasoning to predict target entities under temporal and structural constraints.

4.1 Stage I: Implicit Context Acquisition

In this work, we define implicit contexts as auxiliary semantic information from unstructured or weakly structured textual sources related to an entity that is not explicitly captured by knowledge graph quadruples, yet remains informative for temporal reasoning. This stage aims to recover such contextual snippets in a concise and reproducible manner.

Implicit Context Repository. We construct an offline Implicit Context Repository (ICR) that stores context snippets associated with each query anchor (es, t) and temporally preceding t (e.g., within [t - w, t), where w denotes a temporal window size), obtained either from available unstructured texts or via entity-anchored retrieval from external corpora.

Each snippet is represented as a pair (t′j , xj), where xj is a natural language snippet related to es and t′j ∈ T denotes its timestamp; such snippets do not necessarily mention the target entity or queried relation and may contain noise. For each anchor (es, t), snippets are as follows:

where Nes,t denotes the number of snippets. Each snippet is indexed by (es, t′j ), enabling entity-conditioned and timeaware organization, while the coexistence of informative and irrelevant snippets is handled in Stage II via a two-step training strategy (Section 4.2).

Time-aware Context Selection. At training or inference time, given a temporal query (es, r, t), we retrieve its associated snippet set M[es, t] from the ICR and select only snippets whose timestamps precede the query time:

To emphasize temporal relevance, we retain the K snippets closest to t:

This design prevents temporal leakage and preserves the most relevant implicit contexts for subsequent semantic encoding, where TOPK(·) ranks snippets by their timestamps.

4.2 Stage II: Implicit Context LLMEncoder

Stage II encodes implicit contexts associated with each temporal query into compact, reusable semantic representations using a large language model, providing context-aware information for downstream graph-based temporal reasoning.

Prompt Construction

For each temporal query (es, r, t), we construct a taskoriented prompt by instantiating a fixed template with the query tuple and the retrieved implicit contexts:

where superscripts ictx indicate whether implicit context (ictx) snippets are included in the prompt.

During training, we follow a teacher-forced formulation and append the ground-truth tail entity text to the query prompt, yielding an answer-augmented input P ictx es,r,t,eo , while inference and downstream fusion use P ictx es,r,t alone.

Context-Enhanced Representation Generation

Given the input prompt ˆ P (i.e., Pes,r,t,eo for training and Pes,r,t for inference), the LoRA-adapted encoder fθ produces token-level representations:

Let Iqry, Ians ⊆ {1, . . . , L} denote the token index sets of the query and answer segments, respectively. Segment representations are obtained via masked mean pooling:

Importantly, qllm is computed exclusively from the query segment to prevent label leakage and ensure train-test consistency.

Multi-Objective Optimization

We optimize the prompt encoder using a multi-objective formulation to handle noisy and weakly aligned implicit contexts.

Query Retrieval Loss. Given the context-aware query representation qllm and a fixed entity embedding dictionary E that stores embeddings for all candidate entities e ∈ E, preencoded by the same base language model used for prompt encoding:

we compute retrieval logits s(e) = q⊤llmee/τq for e ∈ E, where τq > 0 is a temperature parameter, and define the retrieval loss as follows:

Language Modeling Loss. Under the teacher-forced formulation adopted during training, let Pes,r,t,eo = (x1, . . . , xL) denote the answer-augmented prompt, where xℓ is the ℓ-th input token. We apply a language modeling loss only on tokens corresponding to the appended ground-truth tail entity, indexed by Ians to regularize hidden representations under retrieval supervision as follows:

where pθ(·) denotes the conditional token probability induced by the language model with parameters θ.

Contrastive Loss. To encourage the query representation to selectively attend to context cues that are discriminative for identifying the correct tail entity, we introduce a contrastive objective between the query representation qllm and the tail entity representation zeo extracted from the same forward computation. For a mini-batch of size B, the contrastive loss is formulated as:

Algorithm 1: Stage II Training Procedure

Input: Temporal training queries Q, implicit contexts index M

Output: Trained encoder fθ

• 1 Step 1: Train with context-free prompts;
• 2 foreach (es, r, t, eo) ∈ Q do
• 3 Construct the answer-augmented prompt Pes,r,t,eo ;
• 4 Compute H= fθ(Pes,r,t,eo ) (Eq. (5));
• 5 Obtain qllm and zeo by masked pooling (Eq. (6));
• 6 Compute Lqry,Llm,Lcon, and optimize Lllm;
• 7 Step 2: Fine-tune with implicit contexts;
• 8 foreach (es, r, t, eo) ∈ Q do
• 9 Retrieve C (K) es,t fromMand construct the answer-augmented ictx-aware prompt P ictx es,r,t,eo ; 10 Compute H= fθ(P ictx es,r,t,eo ) (Eq. (5));
• 11 Obtain qllm and zeo by masked pooling (Eq. (6));
• 12 Compute Lqry,Llm,Lcon, and optimize Lllm;
• 13 return fθ;

where aij denotes the batch-wise similarity between the i-th LLM-based query representation q i llm and the j-th tail entity representation z j eo , and τc > 0 is a temperature parameter.

Overall Objective. The overall training objective of Stage II is defined as follows:

where λlm and λcon control the contributions of semantic regularization and context disambiguation, respectively.

Two-Step Training Strategy

Implicit context snippets are weakly aligned and may contain noise, and directly incorporating them from the beginning of training can cause the model to overfit spurious correlations. To mitigate this issue, Stage II adopts a two-step training strategy based on whether implicit contexts are included. The overall training and inference procedure of Stage II is summarized in Algorithm 1.

Specifically, we first train the model with context-free prompts that exclude implicit contexts, allowing the model to focus on the core temporal prediction task without interference from noisy contextual signals. We then fine-tune the model using context-aware prompts P ictx es,r,t with C (K) es,t to selectively incorporate implicit contextual semantics.

4.3 Stage III: Graph-LLMRepresentation Fusion

We treat the temporal graph encoder as a structure- and timeconstrained semantic module that complements the languagebased query representation with inductive bias. Entities and relations are embedded in the same semantic space as the entity dictionary E, where both entities and relations are initialized by projecting their LLM-induced embeddings, yielding the initial entity embedding v(0) e and the relation embedding vr, respectively.

Temporal Graph Encoding. Given a temporal query (es, r, t), the temporal neighborhood N(es, t) consists of all historical interactions involving es that occurred strictly before t, each represented as (eu, ru→s, tu→s), where eu ∈ E is a neighboring entity, ru→s ∈ R denotes the relation type, and tu→s ∈ T is the interaction timestamp. To capture temporal dynamics, the time gap between each interaction and t is encoded using a continuous time encoding function τ(·). Following TGAT [Xu et al., 2020], relation-aware and timeconditioned messages are aggregated from N(es, t), with each message defined as:

whereWmsg is a learnable weight matrix.

After Lg aggregation layers, the head entity temporal embedding v(Lg) es,t is combined with the query relation embedding vr to form the graph-side semantic query, defined as:

where ϕ(·) denotes a linear projection followed by normalization; qg is subsequently used for gated fusion with the language-based query.

Gated Fusion of Language and Graph Queries. The language-based query qllm and the graph-based query qg are implicitly aligned through shared supervision over the entity embedding dictionary E and consistent semantic initialization, yielding representations in a common semantic space. They are combined via a gated fusion mechanism, where the gate vector g controls the dimension-wise contribution of each query. The fused query representation is obtained as:

where Norm(·) denotes ℓ2 normalization, and ⊙ denotes element-wise multiplication.

Predictions are made by ranking candidate entities according to q⊤f ee, and the model is trained using a softmax-based query loss:

5 Experiments

We conduct extensive experiments to answer the following research questions (RQs):

• RQ1: How does TTGFN perform when compared with state-of-the-art methods on TKG reasoning tasks?
• RQ2: How do individual components of TTGFN contribute to overall performance and robustness against hallucinated contextual signals?
• RQ3: How does the quality of implicit contexts, in terms of relevance and temporal proximity, affect TTGFN and its sensitivity to hallucinated contexts?
• RQ4: How robust is TTGFN under long-tail relations and long-range temporal reasoning?
• RQ5: How sensitive is TTGFN to loss-weighting parameters balancing semantic modeling and structureconstrained alignment?

5.1 Experimental Settings

Datasets

We conduct experiments on three widely used ICEWS TKG benchmarks 2 , which record fine-grained socio-political events as temporal quadruples. Specifically, we use ICEWS14, ICEWS05-15, and ICEWS18, following standard 80%/10%/10% train/validation/test splits. All models are evaluated on the link prediction task, with dataset statistics summarized in Table 1. Additional dataset statistics, including temporal coverage, graph scale and three implicitcontext-augmented datasets constructed in this work, are provided in Appendix Section 3.

Evaluation Metrics

We evaluate performance using standard ranking-based metrics, including Mean Reciprocal Rank (MRR) and Hits@k for k ∈ {1, 3}. We focus on temporal link prediction, where the model predicts the tail entity o given (s, r, ?, t). Following common practice [Cai et al., 2023b], we adopt the timeaware filtered evaluation protocol provided by xERTE [Han et al., 2021] to ensure fair and consistent evaluation.

Parameter Settings

Experiments are conducted on a single NVIDIA RTX 5090 GPU, while the codebase also supports multi-GPU training via Distributed Data Parallel (DDP). Unless otherwise stated, all runs are executed with fixed random seeds to ensure reproducibility.

We train TTGFN with AdamW (initial learning rate 1 × 10-4 ) for up to 50 epochs and apply early stopping (patience = 3) based on time-aware filtered validation MRR. To avoid frequent checkpoint updates caused by marginal fluctuations, we discretize the monitored MRR by flooring to 10-3 precision. We optionally use gradient accumulation and a cosine annealing learning-rate schedule for stable optimization. This discretization stabilizes early stopping without affecting final rankings. All results are averaged over five independent runs to ensure statistical robustness. Detailed parameter settings are provided in Appendix Section 1.

2 https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi: 10.7910/DVN/28075

Baselines for Comparison

To comprehensively evaluate the effectiveness of TTGFN, we compare it with representative baseline methods summarized in Table 2. All baselines are evaluated using publicly available implementations or by following the experimental settings reported in their original papers. Detailed descriptions of the baseline methods are provided in Appendix Section 2.

5.2 Result 1: Comparison with Baselines

To answer RQ1, we evaluate TTGFN on temporal link prediction over ICEWS14, ICEWS05-15, and ICEWS18, with results reported in Table 2. Overall, TTGFN consistently outperforms all baseline methods across all datasets and all Hits@k metrics, demonstrating robust and comprehensive improvements in ranking performance. In particular, TTGFN achieves gains of 8.82% in MRR and 20.85% in Hits@1 on ICEWS14, and achieves relative gains of up to 15.34% in MRR and 31.14% in Hits@1 on ICEWS05-15, as well as 5.64% in MRR and 24.22% in Hits@1 on ICEWS18, indicating more accurate and reliable tail entity prediction.

These consistent gains suggest that reusing implicit contextual information primarily enhances the discriminative power of query representations, rather than merely reshaping the ranking distribution in isolated cases. By integrating semantic cues recovered by LLMs with structure-constrained temporal representations, TTGFN produces more informative query embeddings, leading to improved top-ranked prediction accuracy across diverse TKGs. We also provide a detailed analysis of training and inference efficiency in Appendix Section 6.

5.3 Result 2: Ablation Study

To answer RQ2, we conduct an ablation study under identical hyper-parameter settings, with results summarized in Table 3. This experiment evaluates the contribution of four key designs in TTGFN: language-based semantic reasoning, temporal graph reasoning, implicit context retrieval, and gated representation fusion. We compare the full TTGFN with five ablated variants. Specifically, LLM-only removes the graphbased reasoning path and relies on language-derived representations, while Graph-only disables the language branch and performs reasoning only on temporal graph structure. The w/o ICR variant removes implicit context retrieval and forces the model to operate without external textual context. To examine the fusion design, AvgFusion replaces the gated fusion module with uniform averaging of the two query representations, whereas ScalarGate adopts a simplified fusion mechanism using a learnable scalar gate.

The results show that removing any core component consistently degrades performance across the three datasets. The declines of LLM-only and Graph-only indicate that languagederived semantics and temporal graph structure provide complementary evidence for temporal link prediction. The degradation of w/o ICR further confirms that implicit context retrieval contributes useful external semantic information beyond explicit temporal quadruples. Moreover, although AvgFusion and ScalarGate still combine both representation sources, they remain inferior to the full TTGFN, suggesting that simple averaging or scalar-level weighting cannot fully capture fine-grained interactions between graph-side and language-side signals.

These ablation results demonstrate that TTGFN improves temporal reasoning through coordinated semantic enhancement and structure-constrained fusion, rather than by simply introducing LLM-encoded features. They also address the hallucination concern associated with LLM-based semantics, since TTGFN consistently outperforms both the LLM-only and naive fusion variants. This suggests that semantic cues are injected in a controlled manner, while the gated fusion mechanism helps suppress spurious or hallucinated signals through temporal graph constraints.

5.4 Result 3: Analysis of Implicit Context Quality

To answer RQ3, we analyze the impact of implicit context quality on temporal link prediction using a binned evaluation (Fig. 3). Implicit context snippets are grouped by their average time gap ∆t and LLM-estimated relevance score ρ, aggregated over the test queries that select each context. This setting allows us to examine how temporal proximity and semantic relevance jointly affect the usefulness of retrieved implicit contexts. Performance gains are measured as ∆MRR and ∆Hits@1 relative to the no-context setting.

Across all datasets, implicit contexts consistently improve performance, with larger gains for contexts that are temporally closer and more semantically relevant. This trend demonstrates that TTGFN can effectively exploit high-quality implicit contexts to enhance temporal query representations. Meanwhile, the performance gains become smaller when contexts are less relevant or temporally distant, indicating that the model remains robust under limited relevance or temporal proximity rather than being overly sensitive to noisy contextual information.

Regarding hallucination, the binned analysis indicates that improvements concentrate in temporally proximate and highrelevance contexts, while low-relevance contexts yield negligible impact. This selectivity suggests that TTGFN does not blindly amplify contextual semantics and is relatively robust to unreliable or hallucinated signals. Overall, the analysis

(c) ICEWS05-15 (∆MRR)

(b) ICEWS14 (∆H@1)

(d) ICEWS05-15 (∆H@1)

(e) ICEWS18 (∆MRR)

(f) ICEWS18 (∆H@1)

Figure 3: Binned analysis of implicit context effectiveness on temporal link prediction. Bins are defined by the time gap ∆t and LLMestimated relevance ρ. 3D columns denote instance counts, and surfaces show performance gains (∆MRR on the left and ∆Hits@1 on the right) on ICEWS14, ICEWS05-15, and ICEWS18.

supports that implicit contexts are beneficial when they provide relevant temporal evidence, while low-quality contexts have limited influence on prediction. Detailed analyses are provided in Appendix Section 5.

5.5 Result 4: Long-Tail and Long-Range Reasoning Analysis

To answer RQ4, we analyze the robustness of TTGFN under long-tail relations and long-range temporal dependencies, with results reported in Table 4. Test queries are analyzed along two dimensions: relation frequency and temporal distance to the most recent historical observation. For relation frequency, relation types are ranked by occurrence frequency and split into three groups with similar numbers of relation types. For temporal distance, queries are grouped into temporal-gap tertiles based on the empirical distribution, with no-history queries considered separately.

Across both analyses, TTGFN maintains relatively stable MRR as relation frequency decreases or temporal gaps increase. In the relation-frequency analysis, the model still achieves reasonable performance on low-frequency relations, indicating its ability to handle long-tail reasoning cases despite limited observations. In the temporal-distance analysis, TTGFN shows moderate degradation from short-gap to longgap settings, and it remains effective in no-history scenarios where recent structural evidence is unavailable.

These results demonstrate that TTGFN is robust under sparse structural supervision. Its performance in long-tail, long-range, and no-history settings suggests that the model can still make reliable predictions when explicit graph evidence becomes limited. This observation is consistent with the role of implicit contextual semantics in complementing sparse structural information, while the temporal graph component provides necessary structural constraints for prediction.

5.6 Result 5: Loss Weight Sensitivity Study

To address RQ5, we investigate the sensitivity of TTGFN to the loss-weight parameters λlm and λcon in the overall Stage II training objective (Eq. (11)). These two parameters control the contributions of the language modeling loss and the contrastive loss, respectively. Fig. 4 shows the MRR performance surfaces on three datasets under different weight combinations.

Across all datasets, TTGFN exhibits a smooth performance landscape with a broad high-performance region, indicating low sensitivity to precise weight tuning. The model maintains strong performance under a range of moderate weight combinations, suggesting that the language modeling loss and contrastive loss can jointly improve the quality of LLM-based query representations. In contrast, extreme imbalance between λlm and λcon causes moderate degradation, showing that overemphasizing either objective may weaken the overall representation quality.

These results demonstrate that TTGFN is robust to lossweight selection and does not require precise dataset-specific parameter tuning. Balanced configurations consistently yield strong performance, supporting the joint use of language modeling supervision and contrastive learning in Stage II training. Accordingly, we adopt a balanced default setting λlm = λcon for all experiments, and fix both to 0.5 in practice unless otherwise specified.

6 Conclusion

In this paper, we propose TTGFN, which integrates LLMs with temporal graph networks for temporal knowledge graph reasoning. By aligning implicit contexts with structureconstrained temporal semantics, TTGFN enhances query representations without altering the graph structure. Experiments on ICEWS benchmarks show consistent gains, especially in long-tail and long-range settings.