Attention as Selection: Semantic-Guided Time Series Forecasting

1 Introduction

Time series forecasting (TSF) has broad applications across a wide range of domains, such as energy [Iftikhar et al., 2024], environment [Vadurin et al., 2025], economy [Kynigakis and Panopoulou, 2025], and other fields. Many conventional TSF models often lack explicit mechanisms to incorporate semantic information, which limits their ability to model complex real-world scenarios. In recent years, LLMs have demonstrated strong representation and reasoning abilities across diverse domains. To reduce reliance on extensive training datasets [Yang, 2025], LLMs have also been deployed in the TSF task to provide high-level semantic priors, such as strucc. Semantic-Guided Temporal Selection (Ours)

∗Corresponding author

Figure 1: Comparison of cross-modal interaction methods between time series (in blue) and semantic prompts (in orange). (a) Concatenation-based methods keep modalities separate but provide weak semantic guidance, leading to poorly aligned temporal representations. (b) Fusion-based methods entangle temporal and semantic features, where semantic dominance may obscure fine-grained temporal patterns. (c) Our method employs a semantic-guided selection mechanism to modulate salient temporal features with semantic embeddings under explicit alignment constraints, yielding more discriminative and semantically aligned temporal representations.

tured descriptions, trend summaries and contextual knowledge [Cao et al., 2024; Xue and Salim, 2023].

Existing TSF methods based on LLMs typically treat semantic representations as auxiliary features, incorporating them into temporal modeling through two primary approaches: As shown in Figure 1, concatenation-based methods directly connect semantic and temporal features at the dimensional level to maintain relative decoupling between modalities [Zhou et al., 2025], which may introduce misalignment issues. Fusion-based methods achieve crossmodal interaction through feature-level fusion, such as crossattention mechanisms [Liu et al., 2025b; Liu et al., 2025a], shared embedding spaces [Pan et al., 2024], or joint Transformer architectures [Jin et al., 2024]. Such fusion-based methods allow semantic tokens to directly participate in constructing temporal representations, with the fused features then used for downstream prediction or generation tasks. Nevertheless, this method can lead to modal entanglement, and even local temporal structures may be obscured by semantic signals [Rehman et al., 2025; Jackson, 2021]. Since accurate forecasting critically depends on preserving intricate local temporal patterns, their distortion can decrease predictive performance [Yang et al., 2025].

Multimodal studies also indicate that when cross-modal correlations are weak or inconsistent, strong fusion tends to amplify noise propagation and induce modality dominance [Luo et al., 2024; Zhang et al., 2025; Xing et al., 2024]. Driven by these limitations, multimodal studies in vision and affective computing have increasingly explored the higherlevel semantic signals as guidance or constraints to preserve the integrity of core modalities, such as [Wang et al., 2025b] and [Liu et al., 2025c]. Nevertheless, the efficient and controlled utilization of semantic information remains underexplored in the TSF.

To address these issues, we propose the Attention as Selection (AAS), which interprets cross-attention as a selection operator rather than a feature fusion mechanism. The hierarchical prompt representations produced by HPE map semantic information into importance assessments for different temporal positions by correlating semantic and temporal features, thereby generating sparse temporal attribution distributions. To prevent semantic interference, this mechanism operates solely on attention weights, adjusting the response intensity of temporal features without directly participating in their construction or representation. Moreover, lowentropy regularization (LER) is introduced to shape the selection distribution, ensuring that it remains sparse and controlled. Furthermore, we incorporate a global cross-modal consistency objective to adaptively refine semantic prompts via gradient feedback from the forecasting objective. By aligning these prompts with temporally discriminative structures, AAS achieves stable and effective selection. The contributions of our work are summarized as follows:

• We propose a new semantic-temporal modeling paradigm for LLM-based TSF, where semantic information is treated as a guiding and constraining signal rather than a feature to be directly fused with temporal representations, thereby avoiding modal entanglement.
• We propose AAS, a semantically guided selection mechanism that employs attention as a selection operator. It maps semantic representations onto sparse temporal attribution distributions. AAS also modulates temporal responses solely within the weight space without interfering with temporal feature construction.
• We introduce a low-entropy regularization along with a global temporal-semantic consistency objective to promote sparse, focused, and task-aligned temporal selection, enabling controllable and interpretable semantic attribution.
• Extensive experiments on multiple real-world datasets demonstrate that AAS outperforms state-of-the-art forecasting methods.

2 RelatedWork

2.1 Time Series Forecasting Using LLMs

Recent work has explored the integration of LLMs into the TSF task to incorporate high-level semantic and contextual information. Early approaches treated LLMs as the primary modeling backbone, such as LLM4TS [Chang et al., 2025] and AutoTimes [Liu et al., 2024b], which encoded numerical time series into tokenized sequences and performed direct forecasting within the LLMs. Despite their empirical success, these methods often neglect the modality gap between time series and textual semantics [Tao et al., 2025].

In contrast, recent research has increasingly adopted decoupled dual-branch architectures, where dedicated temporal models capture time-series dynamics and LLMs encode structured prompts or textual descriptions to provide complementary semantic context. Representative methods such as CALF [Liu et al., 2025b] and TimeCMA [Liu et al., 2025a] combine the two modalities through feature-level fusion. Despite progress, most existing LLM-based prediction frameworks still primarily treat semantic representations as auxiliary features fused with temporal embeddings, without explicitly modeling how semantic information should guide or constrain temporal reasoning. As a result, the role of semantics in controllable and interpretable temporal modeling remains insufficiently explored in the TSF.

2.2 Cross-Modal Feature Integration

Cross-attention is widely used to model cross-modal interactions in multimodal learning [Ding et al., 2025; Wang et al., 2024], and has been adopted in recent LLM-based TSF to integrate semantic and temporal information. Most existing approaches treat cross-attention as a feature-level fusion operator, encoding time series and textual prompts separately and aligning them to produce joint representations. Representative methods, including TimeCMA [Liu et al., 2025a] and LLMTPF [Pan et al., 2025], follow this dual-branch design and use the fused features for downstream forecasting [Lee et al., 2025]. While these fusion-based methods enable strong integration between modalities, they also raise the risk of semantic-temporal entanglement [Sun et al., 2025], and may even interfere with the preservation of intrinsic temporal structures.

To mitigate these issues, previous multimodal research has explored more regulated forms of cross-modal interaction. In computer vision, semantic-guided methods often modulate visual representations via attention reweighting, masking, or auxiliary objectives, using textual descriptions as highlevel guidance rather than fused features [Liu et al., 2025c]. Similar ideas appear in affective computing, where semantic graphs or knowledge structures guide temporal or emotional modeling through graph attention [Hu et al., 2025], alignment, or gating mechanisms [Cen et al., 2024]. However, in LLM-based TSF, Cross-attention modules typically pursue full fusion between time-series and textual embeddings [Wang et al., 2025a]. Therefore, how to leverage crossattention to enable more interpretable and regulated interaction mechanisms remains insufficiently explored in this domain.

3 Method

Figure 2 illustrates the architecture of AAS. From semantic modeling (Section 3.2) and selection execution (Section 3.3) to constraint optimization (Section 3.4), the design forms a closed-loop structure. Semantic representations provide structured and learnable guidance for temporal selection. Additionally, explicit constraints and alignment objectives are employed to regulate the sparsity, stability, and task relevance of the outputs from the selector.

3.1 ProblemFormulation

We study the task of multivariate time series forecasting (MTSF), which aims to predict future values of multiple variables based on historical observations and semantic context.

Time Series Data. Let X ∈ RTraw×N denote the observed multivariate time series, where Traw is the total length and N is the number of variables. At time index t, the model takes a look-back windowXt-L+1:t of length L as input and predicts the future sequence Y = Xt+1:t+H ∈ RH×N with prediction horizon H.

Textual Data. Prompts are constructed from statistical summaries and learnable prompt tokens to provide semantic context. They are encoded by a pre-trained language model into semantic embeddings Eprompt ∈ RNp×dllm , where Np is the number of prompt tokens and dllm is the embedding dimension.

Forecasting Objective. The goal is to learn a forecasting function fθ that predicts future values of multivariate time series by leveraging historical temporal information together with auxiliary semantic context. Formally, the forecasting process is defined as

3.2 Hierarchical Prompt EncodingModule (HPE)

HPE combines cached static prompts with learnable prompts to provide stable and fine-grained semantic signals for temporal selection.

Offline Static Prompts. For each variable, given a time series window x ∈ RTraw , A set of descriptive statistical attributes is extracted and formatted into a natural language description s(x). The description is encoded by a frozen pretrained language model, and a static semantic embedding is obtained by mean pooling over the final-layer hidden states:

where tokenize(·) denotes the tokenization function that converts text into a sequence of input tokens for the language model,LLM(last) denotes the final-layer hidden states of the frozen language model, Ns is the number of static prompt tokens. These embeddings are computed offline and cached, providing a stable, non-trainable semantic anchor and avoiding repeated LLM forward passes during training.

Online Learnable Prompts. To enable task-specific adaptation, a set of learnable prompt embeddings Plearn ∈

RM×dllm is introduced, whereM denotes the number of learnable prompt tokens. These embeddings are randomly initialized. To avoid unstable prompt drift, gradients are restricted to the learnable prompt embeddings.

Hprompt ∈ RNp×dllm , where Np = M+Lp denotes the total number of prompt tokens. The concatenated prompt embeddings are then fed into the LLM

The resulting semantic representation combines both global stability and local adaptability, mitigating the interference of semantic noise in temporal modeling.

3.3 Semantic-Guided Selection (SGS)

To decouple semantic attribution from temporal representation learning, cross-attention is restricted to operate as a selector rather than a fusion module. Specifically, SGS is constrained to: (i) generate feature attribution weights solely at temporal positions, (ii) discard all value-based feature aggregations, and (iii) enforce sparsity in the attribution distribution through explicit regularization.

Input Representations. Given a normalized multivariate time series, patch-level temporal representations are extracted using the temporal encoder:

where T denotes the number of temporal patches and d is the hidden dimension of the temporal encoder. To align temporal features with the semantic embedding space of the language model, they are further projected into the LLM hidden space via a linear projection layer:

The semantic representations are obtained as Eprompt ∈ RNp×dllm from the HPE described in Section 3.2.

Cross-Attention as Temporal Attribution. We define a semantic selector S(·) that maps temporal and semantic representations to a normalized temporal attribution distribution:

The selector S(·) is formulated as a constrained multi-head attention score operator, which computes normalized similarity scores between temporal queries and semantic keys, while explicitly abandoning all value-based feature aggregation. Specifically,

where WQ,WK ∈ Rdllm×dk are learnable projection matrices and dk denotes the per-head query/key dimension. The multi-head attention scores are computed as:

where H denotes the number of attention heads. Importantly, only the attention weights are retained and treated as semantic attribution scores for temporal positions. In this manner, semantic prompts influence temporal modeling solely through attribution, rather than directly participating in the construction of temporal features.

Temporal Saliency Aggregation. To obtain a unique attribution decision for each temporal token, the multi-head and multi-prompt attention tensor is collapsed into a single normalized attribution distribution:

where h indexes attention heads, t indexes temporal tokens, and p indexes prompt tokens. This aggregation explicitly removes head-specific and token-specific variability, forcing the selector to produce a global and comparable temporal attribution distribution over temporal positions.

Selection-Based Temporal Enhancement. Unlike traditional attention mechanisms that generate fused representations, the attribution vector w is treated as a decision signal. Specifically, the selector adjusts temporal representations through constrained element-wise rescaling:

where γ controls the modulation strength and ⊙ denotes broadcasting over the feature dimension.

Importantly, this operation preserves the original temporal representation space and merely redistributes the activation energy over time, enforcing a strict separation between semantic attribution and temporal representation learning. As a result, semantic prompts influence where the model attends and how important each time step is emphasized, while the temporal encoder remains solely responsible for modeling temporal dynamics.

3.4 Training Objectives and Regularization

Low-Entropy Regularization (LER). To explicitly shape the behavior of the semantic selector and encourage sparse yet stable temporal selection, the LER term is introduced over the attribution distribution. Here, the entropy of the attribution distribution characterizes the uncertainty and dispersion of temporal selection induced by the semantic selector. Given the attribution score vector ˆw ∈ RT obtained by SGS, it is normalized into a probability distribution:

To construct a sharpened target distribution that favors peaked selections, the attribution distribution is exponentiated with an entropy control parameter α > 1:

The regularization objective minimizes the KL divergence between the normalized attribution distribution and the sharpened prior:

where B denotes the batch size. This regularization encourages the model to concentrate attribution mass on a small subset of informative temporal positions. Thus, the selector can learn sparse, controllable and interpretable selection strategies, enabling semantic guidance to maintain focus without overriding the inherent temporal dynamics.

Consistency Loss. An InfoNCE-based contrastive loss is applied to encourage global consistency between temporal and semantic representations. Let zti and zsi denote the pooled temporal and semantic embeddings of the i-th sample, respectively. The alignment objective is defined as:

where sim(·, ·) denotes cosine similarity, τ is a temperature hyperparameter, (zti , zsi ) forms the positive temporal-semantic pair, and all mismatched pairs {(zti , zsj ) | j ̸= i} within the mini-batch are treated as negative pairs.

By aligning semantic prompts with the selected and enhanced temporal representations, this objective calibrates the semantic space to remain consistent with task-relevant and discriminative temporal structures. This alignment mitigates semantic drift and establishes a stable global constraint that reinforces reliable and interpretable temporal selection.

Overall Objective. Themodel is trained by jointly optimizing the forecasting objective together with auxiliary semantic regularization terms. The overall training objective is defined as:

where Lpre denotes the forecasting loss, LInfoNCE encourages cross-modal temporal-semantic consistency, LLE enforces sparse and controlled temporal selection, and λLE is a learnable weight that is jointly optimized with the model parameters to adaptively balance the regularization strength.

4 Experimental

4.1 Experimental Setup

Datasets. We conduct experiments on multiple widelyused benchmark datasets from different domains. For longterm time series forecasting, we evaluate our method on the ETT benchmark [Zeng et al., 2023], Weather and Exchange datasets [Zhou et al., 2021].

Baselines. We compare our method with a diverse set of representative baselines, including: (1) LLM-based methods: Time-LLM [Jin et al., 2024], TimeCMA [Liu et al., 2025a], and CALF [Liu et al., 2025b]; (2) Transformer-based methods: iTransformer [Liu et al., 2024a] and PatchTST [Nie et al., 2022]; (3) CNN-based methods: TimesNet [Wu et al., 2023]; (4) Linear model-based methods: DLinear [Zeng et al., 2023]. For all baselines, the official implementations or scripts released by the original authors are used.

Implementation Details. For all LLM-based methods, we adopt GPT-2 with six Transformer layers as the language model backbone, with all backbone parameters frozen during training. For fair comparison, all methods are trained under the same experimental settings. Unless otherwise specified, the input sequence length is fixed to L = 512 across all methods and datasets. We evaluate all models under both full-data and few-shot settings. In the few-shot setting, we randomly sample 10% of the training data, with validation and test sets unchanged. All datasets are evaluated under multiple forecasting horizons H ∈ {96, 192, 336, 720} to assess robustness across different prediction scales. Each experiment of AAS is repeated three times with different random seeds, and the averaged results are reported. For additional experimental details, please refer to the supplementary material.

4.2 Long-termForecasting Results

Table 1 summarizes the long-term forecasting performance across all datasets and horizons. The best or second-best average accuracy is achieved by AAS across most datasets, with particularly strong performance observed on the ETT benchmarks. Compared to DLinear, a strong non-LLM baseline, AAS attains an average reduction of 12.6% in MSE and 8.2% in MAE across the four ETT datasets, computed over all forecasting horizons. Against the leading LLM-based forecaster CALF, AAS consistently improves accuracy, reducing average MSE and MAE by 5.1% and 4.3%, respectively, with gains becoming more pronounced at longer horizons. On highly non-stationary datasets such as Exchange, AAS achieves stable performance by leveraging semanticguided selection to preserve discriminative temporal structures. In addition, the global consistency objective calibrates the semantic space to remain aligned with task-relevant temporal patterns. This alignment mitigates semantic drift and suppresses the accumulation of long-horizon errors, thereby enabling robust and reliable forecasting under complex and evolving temporal dynamics.

4.3 Few-shot Forecasting Results

In the few-shot forecasting experiments, the AAS model demonstrates clear and robust superiority. As shown in Table 2, the experiments are conducted under the strict setting of using only 10% of the training data. AAS maintains consistently lower MSE and MAE than all baseline models across all ETT datasets and all forecasting horizons, indicating strong robustness and generalization under datascarce conditions. Notably, while MSE and MAE increase for all models as the prediction horizon extends from 96 to 720, AAS exhibits the slowest error growth, indicating superior stability in long-term forecasting. In scenarios of sparse data, AAS leverages frozen LLM priors to reduce reliance on extensive training datasets, employing SGS for efficient semantically guided temporal selection. Combined with lowentropy regularization that promotes sparse and focused attribution, alongside an InfoNCE-based cross-modal alignment consistency objective, this design mitigates overfitting while preserving robust representational capabilities.

4.4 Ablation Study

To validate the effectiveness of the key components in the AAS, an ablation study is conducted across all datasets. All ablation variants were evaluated under identical experimental settings as the full model. Taking ETTh2 and Exchange as examples, we analyze the model from the following perspectives:

(1) w/o Fusion-based Cross-Attention. The SGS module is replaced by a standard cross-attention mechanism for cross-modal fusion. Temporal features serve as queries, while semantic prompt embeddings function as keys and values. Unlike SGS, this variant does not introduce an explicit temporal selection mechanism, but directly produces fused temporal representations via cross-attention.

(2) w/o LER. The SGS module is retained, but the LER on the attribution weights is removed. In this setting, semantic guidance is still applied through cross-attention-based temporal selection, while the attribution distribution is unconstrained.

(3) w/o SGS. This variant serves as the backbone baseline, obtained by removing the SGS and the LER from the full model.

As shown in Table 3, the full AAS model consistently achieves the best performance on both the ETTh2 and Exchange datasets. Introducing SGS into the temporal forecasting backbone leads to the most significant performance improvement, demonstrating the effectiveness of explicitly modeling semantic-guided temporal attribution. In contrast, replacing SGS with a fusion-based alignment strategy fails to improve performance and even leads to degradation, indicating that directly integrating semantic representations into temporal features may introduce noise and interfere with temporal dynamics. This observation is further supported by Figure 3. When employing AAS, the feature response exhibits pronounced locality and heterogeneity in the temporal dimension. In contrast, standard fusion-based cross-attention produces markedly smoother feature maps, with feature responses tending toward consistency across time steps. This indicates that direct feature fusion weakens discriminative information at the temporal step level and suppresses finegrained temporal variations.

Further adding LER on top of SGS yields additional improvements across datasets. As a regularization mechanism, LER stabilizes training dynamics and encourages more targeted temporal selection. Overall, these results indicate that semantic information is more effective than the direct feature fusion in guiding temporal selection, and the combination of SGS and LER achieves the best overall performance.

4.5 Hyperparameter Sensitivity

Look-backWindow. The impact of the look-backwindow size is further analyzed under different forecasting horizons, with ETTh2 used as a representative dataset. As illustrated in Figure 4, increasing the look-back window consistently improves performance across most prediction lengths, indicating that a richer historical context is beneficial for modeling long-range temporal dependencies. However, the best results are achieved when the window size is set to 512, while further extending the window leads to marginal or negative performance gains. This trend suggests that although longer input sequences provide additional temporal information, excessively large windows may introduce redundant or irrelevant history, especially under long-horizon forecasting settings. Overall, the results demonstrate that a moderately large look-back window strikes a favorable balance between contextual richness and effective temporal modeling.

Entropy Control Parameter. This section analyzes the sensitivity of the proposed LER with respect to the entropy control parameter α in the KL-based attention prior. α is varied over 1, 2, 4, 8 while all other training configurations are fixed. Figure 5 reports forecasting performance on the ETTh2 and Weather datasets in terms of MSE and MAE under different values of α. Across all datasets, the model consistently achieves better performance when α is set to a moderate value (e.g., α = 4), indicating that a fixed moderate setting provides a reliable inductive bias without requiring dataset-specific tuning. This setting encourages the attribution distribution to become sufficiently concentrated, while still preserving discriminative temporal patterns. When α is too small, the sharpened target distribution degenerates into the original at- MSE

MAE

Figure 5: Forecasting performance of AAS with varying entropy control parameters on the ETTh2 and Weather datasets. MSE and MAE are averaged over four prediction horizons.

tribution distribution, resulting in a weaker regularization signal and diminished effectiveness of semantic-guided selection. On the other hand, excessively large values of α impose overly sharp target distributions, which can overly constrain the selector and reduce its flexibility in capturing diverse temporal dependencies. Overall, the performance variation across different values of α remains relatively limited, indicating that the proposed regularization strategy is robust to the choice of α within a reasonable range.

5 Conclusion

In this work, we propose AAS, an attention-as-selection framework for semantic-guided time series forecasting that explicitly redefines cross-attention as a temporal selection mechanism. Through sparse attribution-based modulation and low-entropy regularization, AAS enables controlled semantic guidance while preserving intrinsic temporal structure. This design mitigates modality interference and maintains fine-grained temporal dynamics. Extensive experiments demonstrate strong performance in both few-shot and longterm forecasting scenarios. These results indicate that treating attention as selection provides a principled and effective paradigm for integrating LLM semantics into time series prediction. Beyond predictive accuracy, AAS further improves interpretability by explicitly revealing how semantic information modulates temporal attribution and selection patterns.

Acknowledgments

This research is supported by the National Natural Science Foundation of China (No.62372153, No.62572168) and the Anhui Provincial Natural Science Foundation (No.2308085MF216, No.2508085MF151).