Skip-Timeformer: Skip-Time Interaction Transformer for Long Sequence Time-Series Forecasting

1 Introduction

Long sequence time-series forecasting (LSTF) holds significant practical relevance in complex real-world scenarios [Demirel et al., 2012; Angryk et al., 2020; Patton, 2013; Zhang et al., 2023b; Wang et al., 2023]. For instance, stock prices are influenced by factors such as the company's performance reports over the past few months, industry trends, news events, and investor sentiment, making the advance prediction of stock prices crucial [Liu et al., 2022]. Given its immense practical value, long sequence time-series forecasting has garnered widespread research interest [Lim and Zohren, 2021]. Diverging from other sequential data types, such as language or video, time series distinguishes itself by its continuous record, where each time step stores only limited scalar information. Since individual time steps often fall short in providing sufficient semantic information for indepth analysis, researchers have shifted their focus towards the temporal dynamics. This temporal evolution encapsulates richer information, more authentically reflecting the inherent properties of time series, such as continuity, periodicity, and trendiness. Real-world time series variations frequently involve intricate temporal patterns, encompassing diverse forms of changes (e.g., ascent, descent, fluctuation) that intertwine and overlap, posing a substantial challenge in modeling temporal dynamics. Confronted with this complexity, in-depth exploration of time series forecasting methods, especially in the realm of long sequence forecasting model design, becomes an imperative need to address practical challenges. This necessitates delving into the underlying deep patterns to enhance the accuracy of forecasting future trends.

∗Corresponding Author: zhangfan@sdtbu.edu.cn

In recent years, deep learning has not only demonstrated outstanding performance in predictive tasks but has also excelled in representation learning. It can extract abstract representations and transfer them to various downstream tasks, such as classification and anomaly detection, yielding stateof-the-art performance. Notably, Transformer-based models have achieved tremendous success in natural language processing [Kalyan et al., 2021] and computer vision [Khan et al., 2022; Zhang et al., 2023a; Zhang et al., 2024a], showing great potential in capturing time dependencies in the field of time series [Zhou et al., 2021; Wu et al., 2021; Nie et al., 2022; Zhang et al., 2024b].

Recently, researchers have raised concerns about the effectiveness of Transformer-based models in long sequence forecasting [Zeng et al., 2023]. Existing studies leveraging Transformer for long-term multivariate time series forecasting have exhibited suboptimal performance. This is largely due to the fact that most existing LSTF methods primarily focus on reducing the computational cost of univariate predictions, lacking research specifically tailored to the characteristics of LSTF data. Applying Transformer models to long-term sequence forecasting faces two main challenges.

Initially, the Transformer was originally designed for language analysis [Devlin et al., 2018], where each word could be regarded as an individual sequence of data. However, in the context of time series forecasting, as illustrated in Figure 1, there are periodic and irregular fluctuations between long-term time sequences. Tokens represented at single time step interval struggle to unveil valuable information, and the model's performance may even decline due to irregular, unforeseen events. Consequently, the model's ability to analyze interactions between different data sequences is limited. Secondly, LSTF is more susceptible to overfitting compared to short-term predictions. Transformer-based models are not suitable for training on longer sequences at once. After multiple rounds of training, these models tend to memorize intricate details from the training data, failing to accurately capture real trends and patterns.

To address the aforementioned issues, we focus is on enhancing the capabilities of transformer-based models in long sequence forecasting at the sequence data level. In the context of long sequence time-series problems, capturing substantial historical information is crucial, especially in the LSTF setting. Specifically, we introduce the multi-skip sequence token embedding, as depicted in Figure 1, where sequences in each dimension are segmented and then embedded into feature vectors. The output of the multi-skip sequence token embedding is a one-dimensional array of multiple sequences. Subsequently, we propose skip-time interaction forecasting to capture dependencies between different skipping times among multiple subsequences based on features at various scales. Utilizing skip-time interaction conditional layer normalization, we combine multi-skip sequence tokens with original sequence tokens to analyze dependencies and global patterns missed by multi-skip time steps. Here, we employ a dynamic Dropout scheme, progressively introducing Bernoulli noise to the training data to prevent model overfitting and enhance robustness. Our contributions are as follows:

• We conducted a critical examination of the Transformer architecture and identified that the inherent capabilities of the native Transformer structure for long sequence time-series have not been fully explored. These models embed data points for all dimensions at each time step, sequentially aggregating them into a single vector, focusing on capturing dependencies between variables across different time steps. Without explicitly and adequately mining and utilizing the dependencies between different time steps in long sequences, their predictive capabilities are limited. To address this, we propose the method of multi-skip sequence token embedding.
• We introduce Skip-Timeformer, a Transformer-based model designed for long sequence forecasting that leverages the dependency on skip-time dimensions. SkipTimeformer is one of the few Transformer models explicitly exploring and utilizing skip-time dependencies for long sequence forecasting.
• Extensive experimental results on eight real-world benchmarks demonstrate that Skip-Timeformer consistently achieves state-of-the-art performance on realworld prediction benchmarks. We conduct in-depth analyses of token embedding methods and architecture choices, providing new insights for the future development of Transformer-based models in long sequence forecasting.

2 RelatedWork

In recent years, Transformer-based models have achieved state-of-the-art results in numerous time series tasks [Zeng et al., 2023], particularly in long short-term forecasting (LSTF) problems [Wu et al., 2020; Wu et al., 2021; Jin et al., 2023]. Here, we summarize some of the mainstream models. LogTrans [Li et al., 2019] utilizes convolutional self-attention layers with LogSparse design to capture local information, reducing spatial complexity. Informer [Zhou et al., 2021] introduces ProbSparse self-attention, effectively extracting the most important keywords using extraction techniques. Autoformer [Wu et al., 2021] incorporates decomposition and autocorrelation ideas from traditional time series analysis methods. FEDformer [Zhou et al., 2022] employs a Fourierenhanced structure to achieve linear complexity. Pyraformer [Liu et al., 2021] applies pyramid attention modules with both inter- and intra-scale connections, also maintaining linear complexity. These models predominantly focus on designing novel mechanisms to reduce the complexity of the original attention mechanism, thereby achieving better predictive performance, especially for longer prediction horizons. PatchTST [Nie et al., 2022] takes a different approach by segmenting time series into subsequence-level patches, serving as input tokens for the Transformer. This enhances the local semantic information of the sequence.

It is noteworthy that, unlike previous approaches, we analyze the applicability of Transformer in handling long sequence time-series. Our approach aims to make the Transformer suitable for long sequence time-series forecasting by exploring sequence token embeddings.

3 Skip-Timeformer

In the context of multivariate long sequence time series forecasting, given historical observations X = {x1, ..., xT} ∈ RT×D with T time steps and D variables, we aim to predict the future U time steps Y = {yT+1, ..., yT+U} ∈ RU×D . For convenience, let's denote x1:T as the set of T time steps representing recorded multivariate variables and ¯ yT:T+U as the set of multiple variables predicting the future U time steps.

3.1 Multi-skip Sequence Token Embedding

For the Transformer, tokens embedded in it are formed only by single time steps, resulting in weaker interaction capability for long sequential data. To address this limitation, we decided to partition the time series into multiple subsequences based on different time intervals, as illustrated in Figure 1. We propose a method for embedding tokens, known as multi-skip sequence token embedding. This method involves dividing the multi-variable long sequence into various subsequences based on a skip step length µ. Subsequently, to emphasize edge semantic information, we use padding to generate a series of sequences S1, S2, ..., Sµ. This transformation converts tokens formed by a single time step into tokens formed by multiple skip time steps. It enables the transformer to capture time-related dependencies at different time scales when processing long sequential data. For instance, certain subsequences can aid in capturing short-term variations, while others may be more helpful in capturing longterm trends. The tokens with multiple skip time steps not only assist the model in adapting to variations across various time scales but also enhance the model's generalization ability, helping prevent performance degradation on other time scales while overfitting to a specific one.

3.2 Skip-Time Interaction Forecasting

To further enhance the outstanding performance of our proposed method of multi-skip sequence token embedding in long sequence time series forecasting tasks, we introduce skip-time interaction forecasting. This conceptual framework is specifically designed for Transformer models in the context of long sequence forecasting. Our objective is to augment the Transformer's ability to handle the data complexity, prolonged temporal dependencies, and the identification of hidden temporal patterns within multiple highly similar long sequences. Hence, we propose skip-time interaction forecasting, as illustrated in Figure 2.

Given the output S ∈ Rµ×T×D of a multi-skip sequence token embedding layer as input to the skip-time interaction, where µ and D represent the number of subsequences and dimensions, respectively. For simplicity, we use Sµ i in the following text to denote the vector of all dimensions of the i-th sub-sequence at time step t. In skip-time interaction forecasting, we directly apply multi-head self-attention (MSA) to all dimensions of each sub-sequence:

To further refine the concept of skip-time interaction forecasting, combining the obtained information with the original sequence input helps analyze the dependencies between sequences missed by multi-skip time steps. We employ skiptime at the task level to adapt to time series characteristics. The connection of time interaction information to the original sequence input is achieved through the skip-time interaction conditional layer normalization (STICLN) layer. The general formula for this connection can be expressed as:

Here, STICLN (·) represents skip-time interaction conditional layer normalization, MSA(·) denotes the attention layer in the encoder block, and Sl-1 signifies the output state of the preceding layer l-1 and S0 indicates the multiple subsequences obtained in the embedding layer S1, S2, ..., Sµ. hl represents the embedding token of the original sequence.

3.3 Skip-Time Interaction Conditional Layer Normalization

The conditional layer normalization of skip time interaction integrates the multi-skip sequence tokens into the encoder. In this context, the encoder's input comprises the output from the self-attention layer of the multi-skip sequence tokens and the embedding token information provided from the original sequence. Longer sequences often encompass more time steps, and the model can benefit from global statistical information across the entire sequence. By considering the entire sequence when computing mean and variance, the model can better adapt to the global characteristics of the sequence, aiding in capturing long-term dependencies within the sequence. We normalize the features of the entire sequence information to ensure each temporal feature carries the same weight for generating predictions.

Within STICLN, we update α and β based on the multiskip sequence token embedding, as they are the two most crucial parameters for normalization. For each time step t and each sample S(i) t,j with D feature dimensions, the proposed ST is expressed as follows:

Here, k (·) represents the linear layer. Scaling and shifting are applied to the normalized values:

Where θ and σ represent the mean and standard deviation of hl-1 .

Taking the above considerations into account, the modeling process for predicting the future sequence of each specific

D

Figure 2: Skip-time interaction layer. Used to process arrays of vectors representing multiple subsequence segments of long sequence timeseries, where each vector denotes a fragment of the original sequence. The entire vector array undergoes skip-time interaction to obtain corresponding dependencies (left). Multi-head self-attention (MSA) is employed within the skip-time interaction to establish connections between multiple subsequences (right).

variable based on backtracking sequences can be succinctly described as follows:

Where H = {h1, ..., ht} ,H ∈ Rt×D consists of t embedding tokens with D dimensions, hl is the embedding token of the original sequence, the superscript denotes the layer index, [S0 1 , S0 2 , ..., S 0n ] represents the generated n multi-skip sequence tokens. These tokens are connected through a fully connected layer in the embedding, transitioning from RT to RD , and from RD to RC in the projection. The multi-skip sequence tokens and the output of the previous layer STICLN interact through an MSA layer, and the obtained information is passed into the l-th layer along with the original sequence tokens. We propose an approach that gradually introduces Bernoulli noise to the training data through dropout, progressively increasing the difficulty of the training task and the diversity of the data. This can effectively reduce overfitting and enhance model performance. The output of the L-th layer is linearly projected, denoted as hLt to obtain the final prediction. The overall framework is illustrated in Figure 3, where the proposed Skip-Timeformer leverages the simpler encoder architecture of Transformer [Vaswani et al., 2017], consisting of embedding, STICLN, projection, and Transformer blocks.

4 Experiments

We thoroughly evaluated the proposed Skip-Timeformer across various time series forecasting applications, validating the framework's generality. Additionally, we conducted an in-depth investigation into the impact of the Transformer components' skip-time interaction on the long sequence timeseries dimension.

4.1 Datasets

In our experiment, we extensively incorporated eight realworld datasets, comprising four ETT datasets employed by Autoformer [Wu et al., 2021] (ETTh1, ETTh2, ETTm1, ETTm2), as well as datasets related to Weather, Electricity, Exchange and Traffic. These datasets have been widely utilized for benchmarking.

4.2 Baselines

We meticulously selected eight widely recognized predictive models as our benchmarks, encompassing (1) Transformerbased approaches: Informer [Zhou et al., 2021], Autoformer [Wu et al., 2021], FEDformer [Zhou et al., 2022], ETSformer [Woo et al., 2022], PatchTST [Nie et al., 2022], Crossformer [Zhang and Yan, 2022]; (2) Linear-based methods: DLinear [Zeng et al., 2023]; (3) TCN-based approaches: TimesNet [Wu et al., 2022a].

4.3 Main Results

The comprehensive prediction results are presented in Table 1, with the superior outcomes highlighted in red. Lower values of MSE and MAE indicate more accurate predictions. The proposed Skip-Timeformer achieves consistently stateof-the-art performance. It is noteworthy that PatchTST, considered the current best model for time series forecasting, falls short on multiple datasets compared to our model. This discrepancy can be attributed to the high-frequency fluctuations in long sequences of the datasets, and the patching mechanism of PatchTST may emphasize local attention, leading to failures in predicting long-term trends.

In contrast, our proposed approach, leveraging an aggregation-based representation of the entire series variation, proves more adept at handling such scenarios. Additionally, as a representative explicitly designed to capture periodicity, TimesNet's performance still lags behind that of SkipTimeformer. This suggests that representing time series as a 2D structure still falls short in effectively capturing the long sequence trends and patterns in the sequences.

Therefore, Transformer-based components demonstrate competence in long sequence time-series modeling, and the proposed Skip-Timeformer's architecture with skip-time interactions proves effective in real-world long sequence timeseries forecasting scenarios.

4.4 Skip-Timeformer Generality

In this section, we evaluate the Skip-Timeformer by applying our framework to Transformer and its variants, which commonly deal with the secondary complexity of self-attention mechanisms, including Transformer, Informer, Flowformer [Wu et al., 2022b], and FlashAttention [Dao et al., 2022]. Surprising and promising findings are demonstrated, indicating that skip-time forecasting can enhance the performance and efficiency of transformer-based predictors, generalize to unseen variables, and better utilize historical observations.

We assess the Transformer and the corresponding SkipTimeformer based on the performance improvements reported in Table 2 and 3. It is noteworthy that the framework continuously improves various Transformers. Overall, the average enhancement rates for Transformer, Informer, Flowformer, and Flashformer on the ETTh1, Weather, and Exchange datasets are 57.3%, 42.6%, 28.5%, 57.8%, and 47.8%, respectively, revealing the inappropriate use of the Transformer architecture in time series forecasting. Furthermore, due to the skip-time interaction forecasting structure, our framework makes it easier for attention mechanisms to capture periodic patterns and trends between long sequences. Therefore, the idea of Skip-Timeformer can be widely applied to transformer-based predictors to leverage the flourishing efficient attention mechanisms.

4.5 Increasing Lookback Length

Some previous studies have found that the predictive performance of the Transformer may not necessarily improve with an increase in lookback length [Nie et al., 2022; Zeng et al., 2023]. This can be attributed to the continuous growth of data, making it challenging for attention mechanisms to capture true trends when dealing with long sequences. Existing research focuses on performance improvement, usually based on linear predictions, with theoretical statistical support [Box and Jenkins, 1968], utilizing expanded historical information to enhance predictive performance. As we partition the sequence into multiple subsequences controlled by skip-time length, we evaluate the performance of both the Transformer and Skip-Timeformer as the lookback length increases, as shown in Table 4.

The results surprisingly validate the rationale of utilizing multi-skip sequence embeddings in the time dimension. This allows the Transformer to benefit from an extended lookback window, leading to more accurate predictions.

4.6 Model Analysis

In Table 5, we investigate the impact of multi-skip sequence tokens and skip-time interaction forecasting. We use TimesNet as the state-of-the-art model benchmark for long sequence time-series forecasting. By comparing the results with and without the corresponding multi-skip sequence tokens (MST) and skip-time interaction forecasting (STIF), we observe that both are crucial factors in improving predictive performance.

Series Representation Analysis

To further substantiate the claim that skip-time interaction units are more adept at extracting series representations, we conduct representation analysis using Centered Kernel Alignment (CKA) similarity [Kornblith et al., 2019]. Higher CKA values indicate a greater degree of similarity in representations. For both Transformer variants and Skip-Timeformer, we calculate the CKA between the output features after the first layer and those after the last layer, both embedded with multi-skip sequence tokens.

It is important to note that previous studies have demonstrated a preference for higher CKA similarity in time series forecasting, considering it as a low-level generation task, to achieve improved performance [Wu et al., 2022a; Dong et al., 2023]. As illustrated in Figure 4, Skip-Timeformer, STInformer, STFlowformer, and STFlashformer, through skiptime interaction, learn representations more conducive to long sequence series contexts, consequently leading to more accurate predictions. The results further indicate that SkipTimeformer merits a fundamental reconsideration of the predictive framework.

Partial Sequence Token Analysis

In our designed Skip-Timeformer, training can become cumbersome and challenging when dealing with input sequences that are excessively long, owing to the secondary complexity of self-attention. To address this challenge, in addition to introducing the skip-time interaction forecasting mechanism, we also leverage advanced techniques from previously embedded multi-skip sequence tokens. Specifically, we strategically opt to train the model using only a subset of the sequences.

Experimental results, as demonstrated in Table 6, prove that the performance of the model with partial sequence tokens is only slightly inferior to that with the full sequence when handling exchange rate datasets. Simultaneously, this approach successfully reduces memory consumption, providing robust support for the efficient training of the model.

5 Conclusion

In this paper, we highlight that the conventional architecture of Transformer is not suitable for discovering fundamental sequence representations and inter-sequence correlations in long sequence time-series forecasting. Our approach is specifically designed to address the unique data characteristics of the LSTF dataset. Specifically, the multi-skip sequence token embedding embeds input data into a one-dimensional vector array, preserving information across multiple time dimensions. The skip-time interaction forecasting aims to capture dependencies across time dimensions of the embedded array. Through extensive experiments, we demonstrate that skip-time interaction forecasting enhances the model's understanding of long sequence patterns from highly similar training samples. Additionally, skip-time interaction forecasting effectively addresses overfitting issues in LSTF. Across eight popular datasets, Skip-Timeformer achieves state-ofthe-art performance in long sequence forecasting. Compared to previous work, it benefits from longer lookback windows and demonstrates remarkable framework generality in various model scenarios. In the future, we plan to explore the application of Skip-Timeformer in diverse time series analysis tasks.

Acknowledgments

This work was supported in part by the following: the National Natural Science Foundation of China under Grant No. 62272281, the Special Funds for Taishan Scholars Project(tsqn202306274), the Youth Innovation Technology Project of Higher School in Shandong Province under Grant No. 2023KJ212, and the NSFC Joint Fund with Zhejiang Integration of Informatization and Industrialization under Key Project(U22A2033).

Contribution Statement

Wenchang Zhang and Hua Wang contributed equally to this work and served as co-first authors† . All authors were actively involved in methodology design, research execution, data analysis, and manuscript preparation.