---
title: "Triformer: Triangular, Variable-Specific Attentions for Long Sequence Multivariate Time Series Forecasting"
authors:
  - "Razvan-Gabriel Cirstea"
  - "Chenjuan Guo"
  - "Bin Yang"
  - "Tung Kieu"
  - "Xuanyi Dong"
  - "Shirui Pan"
doi: "10.24963/ijcai.2022/277"
url: "https://ijcai.org/proceedings/2022/277"
pdf: "https://ijcai.org/proceedings/2022/0277.pdf"
venue: "Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence"
year: 2022
pages: "1994-2001"
track: "Main Track"
keywords:
  - "Machine Learning: Recurrent Networks"
  - "Machine Learning: Regression"
  - "Machine Learning: Time-series; Data Streams"
  - "Data Mining: Mining Spatial and/or Temporal Data"
---

# Triformer: Triangular, Variable-Specific Attentions for Long Sequence Multivariate Time Series Forecasting

Razvan-Gabriel Cirstea, Chenjuan Guo, Bin Yang, Tung Kieu, Xuanyi Dong, Shirui Pan

*Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence*, pages 1994-2001. DOI: [10.24963/ijcai.2022/277](https://doi.org/10.24963/ijcai.2022/277)

## Abstract

A variety of real-world applications rely on far future information to make decisions, thus calling for efficient and accurate long sequence multivariate time series forecasting. While recent attention-based forecasting models show strong abilities in
capturing long-term dependencies, they still suffer from two key limitations. First, canonical self attention has a quadratic complexity w.r.t. the input time series length, thus falling short in efficiency. Second, different variables’ time series often have
distinct temporal dynamics, which existing studies fail to capture, as they use the same model parameter space, e.g., projection matrices, for all variables’ time series, thus falling short in accuracy. To ensure high efficiency and accuracy, we propose Triformer, a triangular, variable-specific attention. (i) Linear complexity: we introduce a novel patch attention with linear complexity. When stacking multiple layers of the patch attentions, a triangular structure is proposed such that the
layer sizes shrink exponentially, thus maintaining linear complexity. (ii) Variable-specific parameters: we propose a light-weight method to enable distinct sets of model parameters for different variables’ time series to enhance accuracy
without compromising efficiency and memory usage. Strong empirical evidence on four datasets from multiple domains justifies our design choices, and it demonstrates that Triformer outperforms state-of-the-art methods w.r.t. both accuracy and
efficiency. Source code is publicly available at https://github.com/razvanc92/triformer.

## Full text

## 1 Introduction

Long sequence multivariate time series forecasting plays an essential role in planning and managing complex systems across diverse domains [Liu et al., 2018; Guo et al., 2020; Pedersen et al., 2020a; Pedersen et al., 2020b; Yang et al., 2020]. In such settings, multiple sensors are often deployed to collect diverse information related to a complex system, thus giving rise to multivariate time series [Cirstea et al., 2019]. Figure 1 shows an example of a 3-variate time series indicating the power consumption of three clients from a power grid system, where each variable has its own time series.

∗Corresponding Author

Figure 1: Example of 3-variate time series, where each variable has a distinct temporal pattern. Variable-agnostic modeling: same projection matrices (see the black matrices) are employed for all variables. Variable-specific modeling: distinct projection matrices (see the colorful matrices) are used for different variables, thus enabling the capture of distinct temporal patterns of different variables.

![Image](https://ijcai.org/proceedings/2022/fulltext/277/image_000000_8e4280e00dbc9eb2c91df87da429b3a314400c8ff956e7638bcf15a39cdaa0ff.png)

While extensive studies on short term forecasting exist, e.g., dozens steps ahead forecasting [Hochreiter and Schmidhuber, 1997; Qin et al., 2017; Wu et al., 2019; Wu et al., 2020], a limited number of studies focus on long term forecasting, e.g., hundreds steps ahead. Recent studies show that attentions [Vaswani et al., 2017; Wu et al., 2021b] are able to capture better long term dependencies, comparing to Recurrent Neural Networks RNNs [Li et al., 2018; Hu et al., 2020; Kieu et al., 2022a] or Temporal Convolutional Networks TCNs [Wu et al., 2019; Campos et al., 2022; Kieu et al., 2022b]. However, two main limitations still exist.

High complexity. For a time series of H timestamps, canonical self-attention has a quadratic complexity of O(H2 ). Recent studies on long-term forecasting propose different "sparse" versions of self-attentions [Zhou et al., 2021; Kitaev et al., 2020], aiming at reducing the high complexity. We propose Triformer with linear complexity O(H). We first break the time series into small patches. For each patch, we define a novel Patch Attention (PA) with linear complexity (cf. the small white triangles in Figure 2). Specifically, we introduce a pseudo timestamp for a patch and we compute attentions of the timestamps in the patch only to the single Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence (IJCAI-22)

## 2 RelatedWork

We categorize multivariate time series forecasting methods in Table 1 along two dimensions-short vs. long term forecasting and variable-agnostic vs. variable-specific modeling.

K ∈ ℝd×d×N Short vs. Long Term Forecasting. Most studies focus on short-term forecasting, e.g., 12 to 48 steps ahead. Short-term forecasting models often rely on two types of models that are good at capturing short-term temporal dependenciesrecurrent neural networks (RNNs), e.g., LSTM or GRU [Bai et al., 2019; Li et al., 2018; Seo et al., 2018], and temporal convolutions networks (TCNs), e.g., 1D convolution and WaveNet [Cao et al., 2020; Wu et al., 2019]. Both RNNs and TCNs have limited capabilities when dealing with longrange dependencies since they can not directly access the whole input time series, as they rely on intermediate representations [Khandelwal et al., 2018]. Thus, they fall short on long-term forecasting.

![Image](https://ijcai.org/proceedings/2022/fulltext/277/image_000001_01b5685d099a08a74fbfe4f970678e468baea7da38f96a91448150e5f5a3cf23.png)

For long term forecasting, self-attention based models achieve superior accuracy, but they suffer from quadratic memory and runtime overhead w.r.t. the input time series length H. To reduce the complexity, recent long term forecasting studies propose sparse attentions: LogTrans [Li et al., 2019] is with O(H(logH)2 ), and Informer [Zhou et al., 2021] further reduces the complexity to O(H logH). There also exist other efficient versions of attentions [Beltagy et al., 2020; Wang et al., 2020; Katharopoulos et al., 2020], but they are not designed and verified for time series forecasting. When stacking multiple layers of attentions, an additional pooling layer is often used after each attention layer. This helps shrink the input size to the next attention layer, thus reducing overall complexity [Dai et al., 2020; Zhou et al., 2021]. We propose a novel efficient patch attention with linear complexity, which is able to shrink the layer size by itself without using additional pooling, making a multi-layer patch attention structure also linear.

Variable-agnostic vs. variable-specific modeling. Most studies are variable agnostic, meaning that, although different variables' time series may exhibit distinct temporal patterns, they share the same sets of model parameters, e.g., the same weight matrices in RNNs, the same convolution kernels in TCNs, or the same projection matrices in attentions. Some studies propose variable-specific modeling for RNNs. [Pan et al., 2019] generates variable-specific weight matrices using additional meta-information, e.g., categories of points of interest around the deployed sensor locations. However,

the layer sizes shrink exponentially, making a triangular structure as shown in Figure 2. This ensures a multi-layer Triformer still have linear complexity O(H).

![Image](https://ijcai.org/proceedings/2022/fulltext/277/image_000002_c8be9ef59b3931fa4d1058a279331ce7c35f611935e4a08826f4547aaa57ec24.png)

Figure 2: Triformer has a triangular structure as the layer input sizes shrink exponentially, making a multi-layer Triformer still with O(H).

![Image](https://ijcai.org/proceedings/2022/fulltext/277/image_000003_50d88b905c12cc3f51d4e7005b07bb3197b800c694202b68863e1c2d8eb8309f.png)

Variable-agnostic parameters. Existing forecasting models often use variable-agnostic parameters, although different variables may exhibit distinct temporal patterns [Wu et al., 2019; Li et al., 2018; Wu et al., 2021b]. For example, as shown in Figure 1, although variable 1 has a unique temporal pattern, which is different from the patterns of variables 2 and 3, the same model parameters, e.g., the projection matrices WQ,WK, and WV used in self-attentions (cf. the black matrices in Figure 1), are applied to all three variables. This forces the learned parameters to capture an "average" pattern among all the variables, thus failing to capture distinct temporal pattern of each variable and hurting accuracy.

We propose a light-weight method to enable variable specific parameters, e.g., a distinct set of matrices W(i) Q ,W(i) K , and W(i) V for the i-th variable (cf. the colorful matrices in Figure 1), such that it is possible to capture distinct temporal patterns for different variables. Specifically, we factorize the projection matrices into variable-agnostic and variablespecific matrices, where the former is shared among all variables and the latter is specific to different variables. We make the variable-specific matrices very compact, thus avoiding increasing the parameter space and computation overhead.

We summarize the main contributions as follows: (i) We propose a novel, efficient attention mechanism, namely Patch Attention, along with its triangular, multi-layer structure. This ensures an overall linear complexity, thus achieving high efficiency. (ii) We propose a light-weight approach to enable variable-specific model parameters, making it possible to capture distinct temporal patterns from different variables, thus enhancing accuracy. (iii) We conduct extensive experiments on four public, commonly used multivariate time series data sets from different domains, justifying our design choices and demonstrating that the proposal outperforms state-of-the-art methods.

such additional meta-information may not be always available. [Bai et al., 2020] learns variable-specific weight matrices using a purely data-driven manner without additional information. We empirically compare with it in the experiments. [Cirstea et al., 2022b; Cirstea et al., 2021] consider variable-specific modeling for networks other than RNNs, but not focusing on long term forecasting. {M (i) } i = 1 N {M (i) } i = 1 N

block. Second, we propose a triangular structure when stacking multiple layers of patch attentions, such that the layer sizes shrink exponentially. This ensures linear complexity for multi-layer patch attentions and also enables extracting multiscale features. Third, we propose a light-weight method to enable variable specific modeling, thus being able to capture distinct temporal patterns from different variables, without TP+1 TP TP+1 TP

x(p -1)s+1

![Image](https://ijcai.org/proceedings/2022/fulltext/277/image_000004_f544c031d62c814bf72bf99906a599f5eeab8f1e93e060f087219f93301cf061.png)

<!-- formula-not-decoded -->

![Image](https://ijcai.org/proceedings/2022/fulltext/277/image_000005_7d8f52b47e654c5d8936386393921b7a26e2d3f6f823d10c44a0a91830f34b34.png)

where ϕ denotes the learnable parameters of the forecasting model and ˆxj is the predicted values at timestamp j.

Self-attention. Self-attention is a core operation in attention based models. Given a sequence, e.g., a time series, each timestamp attends to all timestamps in the time series, thus deriving a representation of the entire time series by capturing both long- and short-term dependencies. Consider a time series with H timestamps, e.g., time series x(i) ∈ RH from the i-th variable. Canonical self-attention [Vaswani et al., 2017] first transforms the time series into a query matrix Q(i) = x(i)WQ ∈ RH×d , a key matrix K(i) = x(i)WK ∈ RH×d and a value matrix V(i) = x(i)WV ∈ RH×d , where d represents the hidden representation andWQ,WK, WV ∈ Rd×d are projection matrices, which are learnable. Next, the output is represented as a weighted sum of the values in the value matrix V(i) , where the weights, a.k.a., attention scores, are computed based on the query matrix Q(i) and the key matrix K(i) , as shown in Equation 2.

<!-- formula-not-decoded -->

where φ represents the softmax activation function. Computing the attention scores requires quadratic O(H2 ) time complexity and memory usage, which is a major drawback. Sparse versions of self-attentions exist, [Li et al., 2019; Kitaev et al., 2020; Zhou et al., 2021], reducing the complexity to O(Hlog(H)2 ) and O(H· logH). We strive for linear complexity. In addition, in related attention based works, the same projection matrices WQ, WK and WV are applied to all variables x(i) for 1 ≤ i ≤ N. In contrast, we strive for variable-specific projection matrices.

## 4 Triformer

We propose Triformer for learning long-term and multiscale dependencies in multivariate time series. An overview of the Triformer is shown in Figure 2. The design choices of Triformer are three-fold. First, we propose an efficient Patch Attention with linear complexity as the basic building Based on the patches, we compute attention scores per patch. If we naively use self-attention in a patch, we still have quadratic complexity (cf. Figure 3a). To reduce the complexity to linear, for each patch p, we introduce a learnable, pseudo timestamp Tp ∈ RN×d (cf. Figure 3b). The pseudo timestamp acts as a data placeholder where all the timestamps from the patch can write useful information which is then passed to the next layers. In Triformer, we choose to use the attention mechanism to update the pseudo timestamp, where the pseudo timestamp works as the Query in self-attentions. The pseudo timestamp queries all the "real" timestamps in the patch, thus only computing a single attention score for each real timestamp, giving rise to linear complexity. We call this Patch Attention (PA). TP+1 TP θ1,b1 θ2,b2 gσ PA PA TP+1 TP θ1,b1 θ2,b2 gσ PA PA

![Image](https://ijcai.org/proceedings/2022/fulltext/277/image_000006_c54cdf6851547f57e9a32fab2c9cf0137783f5220b28a235ba9054074ab9b86d.png)

Figure 3: Computing self-attention (SA) vs. patch attention (PA) on patch p. In SA, each timestamp attends to all other timestamps, thus having O(S2 ). In PA, each timestamp only attends to the pseudo timestamp, thus having O(S).

![Image](https://ijcai.org/proceedings/2022/fulltext/277/image_000007_45987c5b28f83edc75e816a5c9ec28526cd4dd1d38edc30547480c764af88e1b.png)

Equation 3 elaborates the computations of PA for pseudo timestamp Tp. Note that Tp = {T(i) p }Ni=1 and each variable has specific pseudo timestamp T(i) p , thus being variablespecific.

<!-- formula-not-decoded -->

The reduced complexity of PA comes with a price that the temporal receptive filed of each timestamp is reduced to the patch size S. In contrast, in the canonical attention, it is H

Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence (IJCAI-22)

![Image](https://ijcai.org/proceedings/2022/fulltext/277/image_000008_81e43bd4deb0eb86e4161089d0aac4aff4e9e7c1526eebacce38796cbc92f34f.png)

![Image](https://ijcai.org/proceedings/2022/fulltext/277/image_000009_be0081347dfcdce5e5ed7f4d8d8e1b90db141b2648d301b59e0daf158082bd3c.png)

of the l-th layer. This leads to Lemma 1.

Lemma 1. An L-layer Triformer has a linear time complexity O(H) if patch size Sl ≥ 2 where 1 ≤ l ≤ L.

Proof. Presented in the technical report [Cirstea et al., 2022a].

In a multi-layer Triformer, each layer consists of different numbers of patches and thus having different number of outputs, i.e., pseudo timestamps. Instead of only using the last pseudo timestamp per layer, we aggregate all pseudo timestamps per layer into an aggregated output. More specifically, the aggregate output Ol at the l-th layer is defined as

<!-- formula-not-decoded -->

where θl is a neural network, Tlp denotes the pseudo timestamp for patch p ∈ [1, P] at the l-th layer.

Finally, the aggregate outputs from all layers are connected to the predictor. This brings two benefits than just using the aggregate output of the last layer. First, the aggregate outputs represent features from different temporal scales, contributing to different temporal views. Second, it provides multiple gradient feedback short-paths, thus easing the learning processes. In the ablation study, we empirically justify this design choice (cf. Table 3 in Experiments).

Predictor. We use a fully connected neural network as the predictor due to its high efficiency w.r.t. longer term forecasting.

Variable-specific modeling can be achieved in a naive way by introducing different projection matrices for each variable, which leads to a very large parameter space. Figure 4 (a) shows that the naive approach needs to learn N·d2 parameters for each projection matrix. This may lead to over-fitting, incurs high memory usage, and does not scale well w.r.t. the number of variables N.

![Image](https://ijcai.org/proceedings/2022/fulltext/277/image_000010_d7b2b20e19e46ff49dd8febd3c55295d42aa1c1f1d48d66ca0c30115a1b07714.png)

To contend with the above challenges, we propose a lightweight method to generate variable specific parameters. In addition, the method is purely data-driven that only relies on the time series themselves and does not require any additional prior knowledge. The overall process is illustrated in Figure 4 (b). First, we introduce a m-dimensional memory vector M(i) ∈ Rm for each variable with i ∈ [1,N]. The memory is randomly initialized and learnable. This makes the method purely data-driven and can learn the most prominent characteristics of each variable.

Figure 4: Enabling Variable-Specific Projection Matrices. Different colors represent matrices for different variables and L and R are variable-agnostic, thus without colors. For the i-th variable, the naive method learns directly matrix K(i) , while the light-weight method learns K(i) by L × B(i) × R, where B(i) is generated by memory M(i) through a generator G.

![Image](https://ijcai.org/proceedings/2022/fulltext/277/image_000011_f06f0b76aa7cce6857cfd9402081ad0f136751c23fa3ce7075ad3e8dc71eb196.png)

Next, we propose to factorize a projection matrix, e.g., a Key matrix W(i) K ∈ Rd×d into three matrices-a left variableagnostic matrix LK ∈ Rd×a , a middle variable-specific matrix B(i) ∈ Ra×a , and a right variable-agnostic matrix RK ∈ Ra×d . We intend to make the middle matrix compact, i.e., a ≪ d, thus making the method light-weight.

The left and right matrices are variable-agnostic, thus being shared with all variables. Different variables have their own middle matrices {B(i) }Ni=1 , thus making the middle matrices variable-specific. More specifically, the i-th variable's middle matrix B(i) is generated from its memory M(i) using a generator G(·), e.g., a 1-layer neural network. This step is also essential to reduce the number of parameters to be learned. Learning a full matrix B={B(i) }Ni=1 directly requires N·a2 parameters. In contrast, when using a generator, it requires N·m for the memories, and an additional overhead of m·a2 for the generator.

In addition to the reduced parameter sizes, the factorization also contributes to improved accuracy (cf. Table 3

Table 2: Overall accuracy. Bold highlights the best results. Underline highlights the second best results.

| Method   |   Method - F |   Reformer - MSE | Reformer - MAE   |   LogTrans - MSE |   LogTrans - MAE |   StemGNN - MSE |   StemGNN - MAE |   AGCRN - MSE |   AGCRN - MAE |   Informer - MSE |   Informer - MAE |   Autoformer - MSE |   Autoformer - MAE |   Triformer - MSE |   Triformer - MAE |
|----------|--------------|------------------|------------------|------------------|------------------|-----------------|-----------------|---------------|---------------|------------------|------------------|--------------------|--------------------|-------------------|-------------------|
| ETTh1    |           24 |            0.991 | 0.754 0.686      |                  |            0.604 |           0.488 |           0.508 |         0.438 |         0.461 |            0.577 |            0.549 |              0.399 |              0.429 |             0.328 |             0.380 |
| ETTh1    |           48 |            1.313 | 0.906            |            0.766 |            0.757 |           0.473 |           0.505 |         0.488 |         0.486 |            0.685 |            0.625 |              0.405 |              0.439 |             0.359 |             0.401 |
| ETTh1    |          168 |            1.824 | 1.138            |            1.002 |            0.846 |           0.630 |           0.577 |         0.553 |         0.540 |            0.931 |            0.752 |              0.498 |              0.486 |             0.433 |             0.449 |
| ETTh1    |          336 |            2.117 | 1.280            |            1.362 |            0.952 |           0.658 |           0.598 |         0.611 |         0.566 |            1.128 |            0.873 |              0.500 |              0.480 |             0.487 |             0.475 |
| ETTh1    |          720 |            2.415 | 1.520            |            1.397 |            1.291 |           0.764 |           0.675 |         0.778 |         0.643 |            1.215 |            0.896 |              0.498 |              0.500 |             0.488 |             0.475 |
| ETTm1    |           24 |            0.724 | 0.607            |            0.419 |            0.412 |           0.302 |           0.381 |         0.318 |         0.358 |            0.323 |            0.369 |              0.377 |              0.417 |             0.252 |             0.329 |
| ETTm1    |           48 |            1.098 | 0.777            |            0.507 |            0.583 |           0.382 |           0.436 |         0.426 |         0.429 |            0.494 |            0.503 |              0.429 |              0.442 |             0.275 |             0.337 |
| ETTm1    |           96 |            1.433 | 0.945            |            0.768 |            0.792 |           0.419 |           0.461 |         0.409 |         0.433 |            0.678 |            0.614 |              0.458 |              0.460 |             0.314 |             0.371 |
| ETTm1    |          288 |            1.820 | 1.094            |            1.462 |            1.320 |           0.522 |           0.518 |         0.545 |         0.531 |            1.056 |            0.786 |              0.632 |              0.526 |             0.385 |             0.427 |
| ETTm1    |          672 |            2.187 | 1.232            |            1.669 |            1.461 |           0.644 |           0.590 |         0.567 |         0.570 |            1.192 |            0.926 |              0.602 |              0.540 |             0.437 |             0.448 |
| Weather  |           24 |            0.655 | 0.583            |            0.435 |            0.477 |           0.377 |           0.416 |         0.947 |         0.774 |            0.335 |            0.381 |              0.408 |              0.445 |             0.323 |             0.364 |
| Weather  |           48 |            0.729 | 0.666            |            0.426 |            0.495 |           0.438 |           0.467 |         0.948 |         0.777 |            0.395 |            0.459 |              0.475 |              0.487 |             0.390 |             0.429 |
| Weather  |          168 |            1.318 | 0.855            |            0.727 |            0.671 |           0.554 |           0.545 |         0.950 |         0.774 |            0.608 |            0.567 |              0.576 |              0.550 |             0.497 |             0.501 |
| Weather  |          336 |            1.930 | 1.167            |            0.754 |            0.670 |           0.598 |           0.573 |         0.946 |         0.772 |            0.702 |            0.620 |              0.593 |              0.558 |             0.538 |             0.531 |
| Weather  |          720 |            2.726 | 1.575            |            0.885 |            0.773 |           0.676 |           0.619 |         0.938 |         0.770 |            0.831 |            0.731 |              0.682 |              0.634 |             0.587 |             0.563 |
| ECL      |           48 |            1.404 | 0.999            |            0.355 |            0.418 |           0.209 |           0.311 |         0.385 |         0.398 |            0.344 |            0.393 |              0.193 |              0.310 |             0.183 |             0.279 |
| ECL      |          168 |            1.515 | 1.069            |            0.368 |            0.432 |           0.265 |           0.355 |         0.481 |         0.509 |            0.368 |            0.424 |              0.223 |              0.334 |             0.182 |             0.288 |
| ECL      |          336 |            1.601 | 1.104            |            0.373 |            0.439 |           0.291 |           0.380 |         0.564 |         0.561 |            0.381 |            0.431 |              0.232 |              0.345 |             0.202 |             0.309 |
| ECL      |          720 |            2.009 | 1.170            |            0.409 |            0.454 |           0.317 |           0.400 |         0.725 |         0.678 |            0.406 |            0.443 |              0.257 |              0.361 |             0.251 |             0.335 |
| ECL      |          960 |            2.141 | 1.387            |            0.477 |            0.589 |           0.329 |           0.411 |         1.005 |         0.829 |            0.460 |            0.548 |              0.265 |              0.364 |             0.248 |             0.339 |

in Experiments). Since all variables' time series share the variable-agnostic matrices L and R, they act as an implicit regularizer-the amount of combinations that can be produced using them is limited. It also facilitates implicit knowledge sharing among variables.

Equation 6 summarizes the generation of variable-specific Key and Value matrices W(i) K and W(i) V , which replace WK and WV in Equation 3 to make Patch Attention variablespecific. We do not need the Query matrix as PA does not need it and employs the pseudo timestamp instead.

<!-- formula-not-decoded -->

## 5 Experiments

We report on a comprehensive empirical study on four realworld, commonly used time series long term forecasting data sets to justify our design choices and demonstrate that Triformer outperforms the state-of-the-art methods.

## 5.1 Experimental Setup

Datasets. We consider four data sets and follow the setup from the-state-of-the-art method for long term forecasting [Zhou et al., 2021].

ETTh1 , ETTm1 . (Electricity Transformer Temperature) 1 : ETTh1 and ETTm1 has observations every 15 minutes. Each observation consists of 6 power load features, making them 6variate time series. To ensure fair comparisons with existing studies, following [Zhou et al., 2021], the train/validation/test data cover 12/4/4 months.

1 https://github.com/zhouhaoyi/ETDataset

ECL. (Electricity Consumption Load) 2 is a 321-variate time series, which records the hourly electricity consumption (Kwh) of 321 clients. Following [Zhou et al., 2021], the train/validation/test data cover 15/3/4 months.

Weather. 3 is a 12-variate time series which record 12 different climate features, e.g., temperature, humidity, etc, from 1,600 U.S. locations. The data is collected hourly. Following [Zhou et al., 2021], the train/validation/test data cover 28/10/10 months.

Forecasting Setups. We use historical H timestamps to forecast the future F timestamps. We vary H and F for different data sets, by following a commonly used long term forecasting setup [Zhou et al., 2021]. We vary F progressively in {24, 48, 168, 336, 720, 960} for the hourly data sets ETTh1 , ECL, and Weather, corresponding to 1, 2, 7, 14, 30, and 40 days ahead; and we vary F in {24, 48, 96, 288, 672} for ETTm1 corresponding to 6, 12, 24, 72 and 168 hours ahead. We also vary the input historical timestamps H. For ETTh1 , Weather and ECL, we vary H from {24, 48, 96, 168, 336, 720}, and from {24, 48, 96, 192, 288, 672} for ETTm1 . Following [Zhou et al., 2021], for each F value, we iterate all possible H values and report the best values in Table 2. A more detailed description of the setup can be found in the technical report [Cirstea et al., 2022a].

Baselines. We select six recent and strong baselines from different categories shown in Table 1, including StemGNN [Cao et al., 2020], AGCRN [Bai et al., 2020], Informer [Zhou et al., 2021], Reformer [Kitaev et al., 2020], LogTrans [Li et al., 2019], and Autoformer [Wu et al., 2021a].

2 https://archive.ics.uci.edu/ml/datasets/ ElectricityLoadDiagrams20112014

3 https://ncei.noaa.gov/data/local-climatological-data

Implementation Details. Due to space limitation the implementation details are listed in the technical report [Cirstea et al., 2022a].

## 5.2 Experimental Results

Table 2 shows the overall accuracy. The results of Informer, Reformer and LogTrans are collected from [Zhou et al., 2021]. For AGCRN, StemGNN, and Autoformer we have used their original implementations which are publicly available. Triformer outperforms all baselines on all datasets. In most settings, AGCRN achieves better results when compared to the three attention based methods that are variable-agnostic Reformer, LogTrans and Informer. This highlights the needs of variable-specific modeling. Autoformer achieves the second best accuracy which highlights the importance of auto-correlation. Finally our proposed method, Triformer, achieves the best accuracy in all settings.

Ablation Study. We perform an ablation study on the ECL dataset, as shown in Table 3. First, we remove the variablespecific modeling (VSM), where the same projection matrices are shared across all variables. We observe a significant loss of accuracy, which well justifies our design considerations to capture distinct temporal patterns of different variables. A shared parameter space for all variables fails to do so. Second, we replace the light-weight variable specific modeling by the naive way (cf. Figure 4(a)). We observe that the naive method is less accurate and incurs significant more parameters to be learned, which are well aligned with our design considerations. The generator G in Triformer takes very limited additional time to generate matrix B. Third, we do not stack multiple layers patch attentions but use only 1 layer of patch attentions. We observe a significant drop in accuracy, suggesting that the proposed multi-layer, triangular structure, is very effective. Fourth, we remove the multiscale modeling such that only the top most layer's output, rather than the outputs from all layers, is fed into the predictor. The accuracy drops significantly, which justifies our design choices of using multi-scale representations in the predictor. Finally, we remove the recurrent connections that connect two consecutive pseudo timestamps within PAs. The temporal information flow is thus broken, meaning that each patch is computed independently of the others. This results in sub-optimal accuracy. However, we note that the accuracy loss is much smaller than removing other components. One of the benefit of this variant is that this makes the computations of all PAs per layer fully parallelizable thus improving the running time.

Table 3: Ablation Study, ECL.

|                |   MSE |   MAE | #Param   |   s/epoch |
|----------------|-------|-------|----------|-----------|
| Triformer      | 0.183 | 0.279 | 347k     |     73.11 |
| w/o VSM        | 0.215 | 0.273 | 303k     |     33.25 |
| wNaive VSM     | 0.191 | 0.288 | 826k     |     63.81 |
| w/o Stacking   | 0.203 | 0.295 | 258k     |     56.26 |
| w/o Multiscale | 0.266 | 0.352 | 285k     |     71.58 |
| w/o Recurrent  | 0.191 | 0.290 | 346k     |     65.75 |

Impact of m. To study the impact of m, i.e., the memory size of M(i) (cf. the section "Variable-Specific Modeling" and Figure 5 in the main paper), we ran multiple experiments in which we vary m among {3, 5, 16, 32}. The results are shown in Table 4. We observe insignificant variations in terms of accuracy, which indicates the proposed method is insensitive w.r.t. the memory size.

Table 4: Effect of m, ECL.

|   m |   MSE |   MAE |
|-----|-------|-------|
|   3 | 0.185 | 0.281 |
|   5 | 0.183 | 0.279 |
|  16 | 0.186 | 0.285 |
|  32 | 0.188 | 0.284 |

Impact of a. We study the impact of a, i.e., the size of the middle matrix B(i) (cf. the section "Variable-SpecificModeling" and Figure 4b). We vary a among the following values {3, 5, 16, 32} and the results are shown in Table 5. We observe that when a is too small, e.g., a = 3, the accuracy drops significantly, which is understandable as the variable-specific matrix B(i) has a shape of a × a, which may be too compact to capture sufficient variable-specific temporal patterns. In addition, we observe that a = 5 is sufficient and increasing further a to 32 does not contribute to improved accuracy.

Table 5: Effect of a, ECL.

|   a |   MSE |   MAE |
|-----|-------|-------|
|   3 | 0.191 | 0.288 |
|   5 | 0.183 | 0.279 |
|  16 | 0.184 | 0.280 |
|  32 | 0.186 | 0.281 |

Efficiency. Figure 5 shows the training runtime (seconds/epoch) of Triformer against other four baselines that show the second best accuracy in some data sets, i.e., Informer, AGCRN, StemGNN and Autoformer, as shown in Table 2. When varying H: (i) Triformer is faster than Informer and Autoformer, which is in accordance with the complexities of the two methods and also due to the efficient fully connected neural network based predictor. (ii) StemGNN increases faster as H increases, while Triformer almost keeps steady. (iii) we observe that the recurrent based method AGCRN falls behind when compared with attention based methods, especially when the input time series is long, i.e., large H. When varying F , Triformer is the fastest in all settings. In addition, for inference time, all methods are less than 13.3 ms per inference, which are sufficiently efficient to support real-time forecasting.

Visualization of Learned Memories. We visualize the learned memories M(i) to investigate whether they may capture distinct and the most prominent patterns of different variable's time series. We select 8 time series from the ECL data sets, as shown in Figure 6. We use t-SNE [Maaten and Hinton, 2008] to compress each variable memory M(i) to a 2D point, which is also shown in Figure 6. We observe that the points are spread over the space, indicating that different time series have their own unique patterns. In addition, we observe that when the variables have similar temporal patterns, their learned memories are close by. For example, the time series of variables 6, 7, and 8 are similar, and their corresponding memories are also clustered together, i.e., in the right, top corner.

Figure 5: Training Runtime, seconds/epoch

![Image](https://ijcai.org/proceedings/2022/fulltext/277/image_000012_2ae1534dd04acc2b986446e8a0ce396b86d010519a47bc0f571cd268974fdcdc.png)

Figure 6: Visualization of the Learned Memories M(i) for different variables along with their time series, ECL

![Image](https://ijcai.org/proceedings/2022/fulltext/277/image_000013_c6969f8c3e98f4d8c642c0e67df611d03fb26cfb519c451512983fa0d9aa043d.png)

Additional experiments for very long sequence forecasting, hyper-parameter sensitivity analysis for number of layers and different patches sizes, and empirical evidence that the proposed VSM can generalize to existing methods are presented in the technical report [Cirstea et al., 2022a].

## 6 Conclusion and Outlook

We propose Triformer, a triangular structure that employs novel patch attentions, which ensures linear complexity. Furthermore, we propose a light-weight method to generate variable-specific projection matrices which are tailored to capture distinct temporal patterns for each variable's time series. Extensive experiments on four data sets show that our proposal outperforms other state-of-the-art methods for long sequence multivariate time series forecasting. In future work, it is of interest to explore different ways of supporting dynamic input lengths and to enhance model training using curriculum learning [Yang et al., 2021; Yang et al., 2022].

## Acknowledgements

This work was supported in part by Independent Research Fund Denmark under agreements 8022-00246B and 8048-

00038B, VILLUM FONDEN under agreements 34328 and 40567, and Innovation Fund Denmark centre, DIREC.

## References

1. [Bai et al., 2019] Lei Bai, Lina Yao, Salil S Kanhere, Zheng Yang, Jing Chu, and Xianzhi Wang. Passenger demand forecasting with multi-task convolutional recurrent neural networks. In PAKDD, pages 29-42, 2019.
2. [Bai et al., 2020] Lei Bai, Lina Yao, Can Li, Xianzhi Wang, and Can Wang. Adaptive graph convolutional recurrent network for traffic forecasting. In NeurIPS, pages 1-16, 2020.
3. [Beltagy et al., 2020] Iz Beltagy, Matthew E Peters, and Ar- man Cohan. Longformer: The long-document trans- former. arXiv preprint arXiv:2004.05150, 2020.
4. [Campos et al., 2022] David Campos, Tung Kieu, Chenjuan Guo, Feiteng Huang, Kai Zheng, Bin Yang, and Chris- tian S. Jensen. Unsupervised time series outlier detection with diversity-driven convolutional ensembles. PVLDB, 15(3):611-623, 2022.
5. [Cao et al., 2020] Defu Cao, Yujing Wang, Juanyong Duan, Ce Zhang, Xia Zhu, Congrui Huang, Yunhai Tong, Bix- iong Xu, Jing Bai, Jie Tong, and Qi Zhang. Spectral temporal graph neural network for multivariate time-series forecasting. In NeurIPS, pages 17766-17778, 2020.
6. [Cirstea et al., 2019] Razvan-Gabriel Cirstea, Bin Yang, and Chenjuan Guo. Graph attention recurrent neural networks for correlated time series forecasting. In MileTS, 2019.
7. [Cirstea et al., 2021] Razvan-Gabriel Cirstea, Tung Kieu, Chenjuan Guo, Bin Yang, and Sinno Jialin Pan. En- hanceNet: Plugin neural networks for enhancing corre- lated time series forecasting. In ICDE, pages 1739-1750, 2021.
8. [Cirstea et al., 2022a] Razvan-Gabriel Cirstea, Chenjuan Guo, Bin Yang, Tung Kieu, Xuanyi Dong, and Shirui Pan. Triformer: Triangular, variable-specific attentions for long sequence multivariate time series forecasting-full version. arXiv, abs/2204.13767, 2022.
9. [Cirstea et al., 2022b] Razvan-Gabriel Cirstea, Bin Yang, Chenjuan Guo, Tung Kieu, and Shirui Pan. Towards spatio-temporal aware traffic time series forecasting. In ICDE, 2022.
10. [Dai et al., 2020] Zihang Dai, Guokun Lai, Yiming Yang, and Quoc V Le. Funnel-transformer: Filtering out se- quential redundancy for efficient language processing. In NeurIPS, 2020.
11. [Dauphin et al., 2017] Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In ICML, pages 933-941, 2017.
12. [Guo et al., 2020] Chenjuan Guo, Bin Yang, Jilin Hu, Chris- tian S. Jensen, and Lu Chen. Context-aware, preference- based vehicle routing. VLDB J., 29(5):1149-1170, 2020.
13. [Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Comput., 9(8):1735-1780, 1997.
14. [Hu et al., 2020] Jilin Hu, Bin Yang, Chenjuan Guo, Chris- tian S. Jensen, and Hui Xiong. Stochastic origin- destination matrix forecasting using dual-stage graph con- volutional, recurrent neural networks. In ICDE, pages 1417-1428, 2020.
15. [Katharopoulos et al., 2020] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and Franc ¸ois Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In International Conference on Machine Learning, pages 5156-5165. PMLR, 2020.
16. [Khandelwal et al., 2018] Urvashi Khandelwal, He He, Peng Qi, and Dan Jurafsky. Sharp nearby, fuzzy far away: How neural language models use context. ACL, pages 284-294, 2018.
17. [Kieu et al., 2022a] Tung Kieu, Bin Yang, Chenjuan Guo, Razvan-Gabriel Cirstea, Yan Zhao, Yale Song, and Chris- tian S. Jensen. Anomaly detection in time series with ro- bust variational quasi-recurrent autoencoders. In ICDE, 2022.
18. [Kieu et al., 2022b] Tung Kieu, Bin Yang, Chenjuan Guo, Christian S. Jensen, Yan Zhao, Feiteng Huang, and Kai Zheng. Robust and explainable autoencoders for time se- ries outlier detection. In ICDE, 2022.
19. [Kitaev et al., 2020] Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. ICLR, pages 1-12, 2020.
20. [Li et al., 2018] Yaguang Li, Rose Yu, Cyrus Shahabi, and Yan Liu. Diffusion convolutional recurrent neural net- work: Data-driven traffic forecasting. In ICLR, pages 1- 16, 2018.
21. [Li et al., 2019] Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. En- hancing the locality and breaking the memory bottleneck of transformer on time series forecasting. In NeurIPS, pages 5244-5254, 2019.
22. [Liu et al., 2018] Huiping Liu, Cheqing Jin, Bin Yang, and Aoying Zhou. Finding top-k optimal sequenced routes. In ICDE, pages 569-580, 2018.
23. [Maaten and Hinton, 2008] Laurens van der Maaten and Ge- offrey Hinton. Visualizing data using t-SNE. J. Mach. Learn. Res., 9(11):2579-2605, 2008.
24. [Pan et al., 2019] Zheyi Pan, Yuxuan Liang, Weifeng Wang, Yong Yu, Yu Zheng, and Junbo Zhang. Urban traffic pre- diction from spatio-temporal data using deep meta learn- ing. In SIGKDD, pages 1720-1730, 2019.
25. [Pan et al., 2021] Zizheng Pan, Bohan Zhuang, Jing Liu, Haoyu He, and Jianfei Cai. Scalable vision transformers with hierarchical pooling. ICCV, 2021.
26. [Pedersen et al., 2020a] Simon A. Pedersen, Bin Yang, and Christian S. Jensen. Fast stochastic routing under time- varying uncertainty. VLDB J., 29(4):819-839, 2020.
27. [Pedersen et al., 2020b] Simon Aagaard Pedersen, Bin Yang, and Christian S. Jensen. Anytime stochastic routing with hybrid learning. PVLDB, 13(9):1555-1567, 2020.
28. [Qin et al., 2017] Yao Qin, Dongjin Song, Haifeng Chen, Wei Cheng, Guofei Jiang, and Garrison Cottrell. A dual- stage attention-based recurrent neural network for time se- ries prediction. IJCAI, pages 2627-2633, 2017.
29. [Seo et al., 2018] Youngjoo Seo, Michaël Defferrard, Pierre Vandergheynst, and Xavier Bresson. Structured sequence modeling with graph convolutional recurrent networks. In ICONIP, pages 362-373, 2018.
30. [Stoller et al., 2020] Daniel Stoller, Mi Tian, Sebastian Ew- ert, and Simon Dixon. Seq-u-net: A one-dimensional causal u-net for efficient sequence modelling. In IJCAI, pages 2893-2900, 2020.
31. [Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. NIPS, page 6000-6010, 2017.
32. [Wang et al., 2020] Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768, 2020.
33. [Wu et al., 2019] Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, and Chengqi Zhang. Graph wavenet for deep spatial-temporal graph modeling. In IJCAI, pages 1907- 1913, 2019.
34. [Wu et al., 2020] Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, Xiaojun Chang, and Chengqi Zhang. Con- necting the dots: Multivariate time series forecasting with graph neural networks. In SIGKDD, pages 753-763, 2020.
35. [Wu et al., 2021a] Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transform- ers with auto-correlation for long-term series forecasting. In NIPS, 2021.
36. [Wu et al., 2021b] Xinle Wu, Dalin Zhang, Chenjuan Guo, Chaoyang He, Bin Yang, and Christian S. Jensen. Au- toCTS: Automated correlated time series forecasting. PVLDB, 15(4):971-983, 2021.
37. [Yang et al., 2020] Sean Bin Yang, Chenjuan Guo, and Bin Yang. Context-aware path ranking in road networks. IEEE Trans. Knowl. Data Eng., 2020.
38. [Yang et al., 2021] Sean Bin Yang, Chenjuan Guo, Jilin Hu, Jian Tang, and Bin Yang. Unsupervised path representa- tion learning with curriculum negative sampling. In IJCAI, pages 3286-3292, 2021.
39. [Yang et al., 2022] Sean Bin Yang, Chenjuan Guo, Jilin Hu, Bin Yang, Jian Tang, and Christian S. Jensen. Temporal path representation learning with weakly-supervised con- trastive curriculum learning. In ICDE, 2022.
40. [Zhou et al., 2021] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In AAAI, pages 11106- 11115, 2021.