Hierarchical Multi-Scale Gaussian Transformer for Stock Movement Prediction

1 Introduction

With the development of stock markets all around the world, the overall capitalization of stock markets worldwide has exceed 68 trillion U.S. dollar by 2018 1 . Recent years, more and more quantitative researchers get involved in predicting the future trends of stocks, and they help investors make profitable decisions using state-of-the-art trading strategies. However, the uncertainty of stock prices make it an extremely challenging problem in the field of data science.

Prediction of stock price movement belongs to the area of time series analysis which models rich contextual dependencies using statics or machine learning methods. Traditional approaches for stock price prediction are mainly based on fundamental factors technical indices or statistical time series models, which captures explicit or implicit patterns from historical financial data. However, the performance of those methods are limited by two aspects. Firstly, they usually require expertise in finance. Secondly, these methods only capture simple patterns and simple dependence structures of financial time series. With the rise of artificial intelligence technology, more and more researchers attempt to solve this problem using machine learning algorithms, such as SVM [Cortes and Vapnik, 1995], Nearest Neighbors [Altman, 1992], Random Forest [Breiman, 2001]. Recently, since deep neural networks empirically exhibited its powerful capabilities in solving highly uncertain and nonlinear problems, the stock prediction research based on deep learning technique has become more and more popular in recent years and show significant advantages over traditional approaches.

∗Equal contribution.

†Corresponding author.

1 https://data.worldbank.org/indicator/CM.MKT.TRAD.CD?end= 2018&start=2018&view=bar

The stock prediction research based on deep learning technique can roughly be grouped to two categories: (1) Fundamental analysis, and (2) Technical analysis. Fundamental analysis constructs prediction signals using fundamental information such as news text, finance report and analyst report. For example, [Schumaker and Chen, 2009; Xu and Cohen, 2018; Chen et al., 2019] use natural language processing approaches to predict stock price movement by extracting latent features from market-related texts information,such as news, reports, and even rumors. On the other hand, technical analysis predicts finance market using historical data of stocks. One natural choice is the RNN family, such as RNN [Rumelhart et al., 1986], LSTM [Hochreiter and Schmidhuber, 1997], Conv-LSTM [Xingjian et al., 2015], and ALSTM [Qin et al., 2017]. However, the primary drawback of these methods is that RNN family struggles to capture extremely long-term dependencies [Li et al., 2019], such as the dependencies across several months on financial time series.

Recently, a well-known sequence-to-sequence model called Transformer [Vaswani et al., 2017] has achieved great success on natural machine translation tasks. Distinct from RNN-based models, Transformer employs a multi-head selfattention mechanism to learn the relationship among different positions globally, thereby the capacity of learning long-term dependencies is enhanced. Nevertheless, canonical Transformer is designed for natural language tasks, and therefore it has a number of limitations in tackling finance prediction: (1) Locality imperception: the global self-attention mechanism in canonical Transformer is insensitive to local context, whose dependencies are much important in financial time series. (2) Hierarchy poverty: the point-wise dot-product selfattention mechanism lacks the capability of utilizing hierarchical structure of financial time series (e.g. learning intraday, intra-week and intra-month features in financial time series independently). Intuitively, addressing those drawbacks will improve the robustness of the model and lead to better performance in the task of financial time series prediction.

In this paper, we propose a new Transformer-based method for stock movement prediction. The primary highlight of the proposed model is the capability of capturing long-term, short-term as well as hierarchical dependencies of financial time series. For these aims, we propose several enhancements for the Transformer-based model: (1) Multi-Scale Gaussian Prior enhances the locality of Transformer. (2) Orthogonal Regularization avoids learning redundant heads in the multimead self-attention mechanism. (3) Trading Gap Splitter enables Transformer to learn intra-day features and intra-week features independently. Numerical results comparing with other competitive methods for time series show the advantages of the proposed method.

In summary, the main contributions of our paper include:

• We propose a Transformer-based method for stock movement prediction. To the best of our knowledge, this is the first work using Transformer model to tackle financial time series forecasting problems.
• We propose several enhancements for Transformer model include Multi-Scale Gaussian Prior, Orthogonal Regularization, and Trading Gap Splitter.
• In experiments, the proposed Transformer-basedmethod significantly outperform several state-of-the-art baselines, such as CNN, LSTM and ALSTM, on two realworld exchange market.

2 RelatedWork

Fundamental Analysis Machine learning for fundamental analysis developed with the explosion of finance alternative data, such as news, location, e-commerce. [Schumaker and Chen, 2009] proposes a predictive machine learning approach for financial news articles analysis using several different textual representations. [Weng et al., 2017] outlines a novel methodology to predict the future movements in the value of securities after tapping data from disparate sources. [Xu and Cohen, 2018] uses a stochastic recurrent model (SRM) with an extra discriminator and an attention mechanism to address the adaptability of stock markets. [Chen et al., 2019] proposes to learn event extraction and stock prediction jointly.

Technical Analysis On the other hand, technical analysis methods extract price-volume information from historical trading data and use machine learning algorithms for prediction. For instances, [Lin et al., 2013] proposes an SVMbased approach for stock market trend prediction. Meanwhile, LSTM neural network [Hochreiter and Schmidhuber, 1997] has been employed to model stock price movement. [Nelson et al., 2017] proposes an LSTM model to predict stock movement based on the technical analysis indicators.

[Zhang et al., 2017] proposes an LSTM model on historical data to discover multi-frequency trading. [Wang et al., 2019] proposes a ConvLSTM-based Seq2Seq framework for stock movement prediction. [Qin et al., 2017] proposes an Attentive-LSTM model with an attention mechanism to predict stock price movement and [Feng et al., 2019] further introduces an data augmentation approach with the idea of adversarial training. However, [Li et al., 2019] points out that LSTM can only distinguish 50 positions nearby with an effective context size of about 200. That means that LSTM-based models suffer from the difficulty in capturing extremely longterm dependencies in time series. To tackle this issue, we propose a Transformer-based method to better mine the intrinsic long-term and complex structures in financial time series.

3 ProblemFormulation

Since the exact price of a stock is extremely hard to be predicted accurately, we follow the setup of [Walczak, 2001] and predict the stock price movement instead. Usually the stock movement prediction is treated as a binary classification problem - e.g., discretizing the stock movement into two classes (Rise or Fall). Formally, given the stock features X = [xT-∆t+1, xT-∆t+2, ..., xT ] ∈ R∆t×F (also represented as X = [x1, x2, ..., xN] ∈ RN×F in the rest of the paper for simplicity) in the latest ∆t time-steps, the prediction model fθ(X) with parameters θ can output the predicted movement label y = I(pT+k > pT ), where T denotes the target trading time, F denotes the dimension of stock features and pt denotes the close price at time-step t. Briefly, the proposed model utilizes the historical data of a stock s in the lag [T -∆t + 1, T ] (where ∆t is a fixed lag size) to predict the movement class y (0 for Fall, 1 for Rise) of the future k time-steps.

4 ProposedMethod

In this section, we first describe the basic Transformer model we designed. Then we introduce the proposed enhancements of Transformer for financial time series.

4.1 Basic Transformer for StockMovement Prediction

In our work, we instantiate fθ(·) with Transformer-based model. To adapt the stock movement prediction task which takes time series as inputs, we design a variant of Transformer with encoder-only structure which consists of L blocks of multi-head self-attention layers and position-wise feed forward layers (see Figure 1). Given the input time series X = [x1, x2, ..., xN] ∈ RN×F , we first add the position encoding and adopt an linear layer with tanh activation function

Then multi-head self-attention layers take ¯X as input, and are computed by

where h = 1, ..., H and W(Q) h ,W(K) h and W(V ) h are learnable weight matrices for Query, Key and Value, respectively

(refers to [Vaswani et al., 2017] for more details). Then the attention score matrix ah ∈ RN×N of the hth head is computed by

where M is a position-wise mask matrix to filter out temporal attention, so as to avoid future information leakage. Afterwards, the output of the hth head is a weighted sum defined as follows:

The final output of multi-head attention layers is the concatenation of all heads by O = [O1,O2, ...,OH]. Afterward, the position-wise feed forward layer takes O as input and transforms it to Z by two fully-connected layers and a ReLU activation layer. Upon the output zi of the last selfattention layer, a temporal attention layer [Qin et al., 2017] is deployed to aggregate the latent features from each position as m = ∑Ni=1 αizi. Then the scalar prediction score ˆ y is computed by a fully-connected layer and a sigmoid transformation:

Our ultimate goal is to maximize the log-likelihood between ˆ y and y via the following loss function:

4.2 Enhancing Locality withMulti-Scale Gaussian Prior

Recently, Transformer exhibits its powerful capability of extracting global patterns in natural language processing fields. However, the self-attention mechanism in Transformer considers the global dependencies with very weak position information. Note that, the position information serves as the temporal variant patterns in time series, which is much important. To address it, we incorporate Multi-Scale Gaussian Prior into the canonical multi-head self-attention mechanism with the intuition that the relevance of data in two positions is directly proportional to the temporal distance between them.

To pay more attention to the closer time-steps, we add biases of Gaussian prior to the attention score matrices based on the assumption that such scores would obey Gaussian distributions. Note that this operation is equivalent to multiplying the original attention distribution with a Gaussian distribution mask (See [Guo et al., 2019] for the proof). In details, we transform Eq.3 to the following form by adding Gaussian biases:

where B(G) h ∈ RN×N is a matrix computed by

Note that we allow σh in B(G) h are different for different heads in the multi-head self-attention layer.

Besides, we also give an empirical approximation for σh. Suppose we want to pay more attention to the Dh closest time-steps, the variance can be empirically set as σh = Dh. By this way, we allow different Dh in different attention heads in order to provide Multi-Scale Gaussian Prior.

In finance, the temporal features from last 5-day, 10-day, 20-day or 40-day are usually considered in trading strategies. That means, for a 4-head self-attention layer, we can empirically assign the window-size set D = {5, 10, 20, 40} to σh with h = 1, ..., 4, respectively as is shown in Figure 2. In conclusion, the proposed Multi-Scale Gaussian Prior enables Transformer to learn multi-scale localities from financial time series.

4.3 Orthogonal Regularization for Multi-Head Self-Attention Mechanism

With the proposed Multi-Scale Gaussian Prior, we let different heads learn different temporal patterns in the multi-head attention layer. However, some previous research [Tao et al., 2018][Li et al., 2018][Lee et al., 2019] claims that canonical multi-head self-attention mechanism tend to learn redundant heads. To enhance the diversity between each head, we induce an orthogonal regularization with regard to the weight tensor W(V ) h in Eq.2. Specifically, we first calculate the

(b) Hierarchical self-attention mechanism

Figure 3: Trading Gap Splitter overview.

tensor W(V ) = [W(V ) 1 ,W(V ) 2 , ...,W(V ) H ] by concatenating W(V ) h of all heads. Note that the size ofW(V ) is H×F ×dv where dv denotes the last dimension of Vh. Then we flatten the tensor W(V ) to a matrix A with size of H×(F ∗dv), and further normalize it as ˜A = A/||A||2. Finally, the penalty loss is computed by

where || ◦ ||F denotes the Frobenius norm of a matrix and I stands for an identity matrix. We briefly add the penalty loss to the original loss with a trade-off hyper-parameter γ as follows:

For simplicity in expression, here we omit the number of multi-head self-attention layers in the model. In our experiment, we sum up the penalty losses from each multi-head self-attention layer as the final penalty loss as follows:

4.4 Trading Gap Splitter

As is known that the input of the model is a continuous time series. However, due to the trading gaps, the input time series is essentially NOT continuous. Takes the 15-minute data from NASDAQ stock market 2 as an example, of which one trading day contains 26 15-minute time-steps and one trading week contains 5 trading days. This means there are inter-day and inter-week trading gaps. However, when the basic Transformer model is applied to this data, the self-attention layer inside treats all time-steps equally and omit the implicit interday and inter-week trading gaps. To solve this problem, we design a new hierarchical self-attention mechanism for the Transformer model to learn the hierarchical features of stock data (see Figure 3 (a)).

2 the NASDAQ Stock Exchange Market is open 5 days per week for 6.5 hours per day

Takes a 3-block Transformer model as an example, we aim to learn the hierarchical features of stock data by the order "intra-day→intra-week→global". In order to do so, we set two extra position-wise masks to the first and second selfattention blocks respectively in order to limit the attention scopes. Formally, we modify Eq.7 to the following form:

where M(H) is an N × N matrix filled with -inf whose diagonal is composed of continuous sub-matrices filled with 0. TheM(H) for the first and second self-attention blocks are shown in Figure 3 (c). Specifically, the size of sub-matrices in M(H) for the first block is 26×26 since one trading day contains 26 15-minute time-steps, and the size of sub-matrices for the second block changes to 130 × 130 (26 ∗ 5) since one trading week contains 5 trading days. By this way, the first and second self-attention blocks will learn the intra-day and intra-week features of stock data, respectively. Moreover, for the last self-attention block, we keep the original attention mechanism without M(H) to learn global features of stock data. As a result, the Transformer model with the proposed hierarchical attention mechanism avoids suffering from the trading gaps. Note that, all attention heads in the same multihead self-attention layer share the sameM(H) .

(c) Hierarchical masks

5 Experiments

To evaluate the proposed methods, we use two stock data: one from NASDAQ market and the other from China A-shares market. The details of the two data are listed in Table 1. In the following subsections, we will introduce the data collection process and show our empirical results from numerical experiments. We also conduct an incremental analysis to explore the effectiveness of each proposed enhancements for Transformer.

5.1 Data Collection

We collect the daily quote data of all 3243 stocks from NASDAQ stock market from July 1 st , 2010 to July 1 st , 2019 and the 15-min quote data of 500 CSI-500 component stocks from China A-shares market from December 1 st , 2015 to December 1 st , 2019. We move a lag window with size of N timesteps along these time series to construct candidate examples. For the NASDAQ data, we construct 5 datasets with window sizes N = 5, 10, 20, 40 (denoting 5-, 10-, 20-, 40day, respectively), and the lag strides are all fixed to 1. For the China A-shares data, we also construct 5 datasets with window sizes N = 5 ∗ 16, 10 ∗ 16, 20 ∗ 16, 40 ∗ 16 (denoting 5-, 10-, 20-, 40-day since one trading day contains 16 15-minute in China A-shares market), and the lag strides are all fixed to 16 (i.e. 1 day). The features used in all datasets consist of open, high, low, close and volume, which are all adjusted and normalized following [Feng et al., 2019; Xu and Cohen, 2018]. We initially label both datasets by the strategy mentioned in Section 3.1 (i.e y = I(pT+k > pT )), where k is set to 1 and 16 (both denote 1 day) for the NASDAQ data and the China A-shares data, respectively. Furthermore, we set two threshold parameter βrise, βfall to the labels in each dataset in order to balance the number of positive and negative samples to roughly 1 : 1 as follows:

To avoid the data leakage problem, we strictly follow the sequential order to split training/validation/test sets. For instances, we split the NASDAQ data and the China A-shares data into training/validation/test sets by 8-year/1-year/1-year and 3-year/6-month/6-month, respectively.

5.2 EvaluationMetrics

Following previous research [Feng et al., 2019; Xu and Cohen, 2018], we evaluate the prediction performance with two metrics: Accuracy (Acc) and Matthews Correlation Coefficient (MCC), which is defined as below:

where tp, tn, fp, fn denote the number of samples classified as true positive, true negative, false positive and false negative, respectively.

5.3 Numerical Experiments

Approaches in comparison. We compare our approaches B-TF, MG-TF and HMG-TF with the baselines CNN, LSTM and ALSTM:

• CNN [Selvin et al., 2017]: Here we use 1D-CNN with the kernel size of 1 × 3.
• LSTM [Nelson et al., 2017]: A variant of recurrent neural network with feedback connections.
• Attentive LSTM (ALSTM) [Feng et al., 2019]: A variant of LSTM model with a temporal attentive aggregation layer.
• Basic Transformer (B-TF): The proposed Basic Transformer model introduced in Section 4.1.

• Multi-Scale Gaussian Transformer (MG-TF): The B-TF model with enhancements of Multi-Gaussian Prior and Orthogonal Regularization introduced in Section 4.2 and Section 4.3, respectively.
• Hierarchical Multi-Scale Gaussian Transformer (HMGTF): The MG-TF model with the enhancement of Trading Gap Splitter introduced in Section 4.4.

Settings. We implement B-TF, MG-TF and HMG-TF with PyTorch framework on Nvidia Tesla V100 GPU. We use Adam optimizer with an initial learning-rate of 1e -4. The size of mini-batch is set to 256. The trade-off hyperparameter γ is set to 0.05. All TF-based models have 3 multihead self-attention blocks each with 4 heads. We train the model in an end-to-end manner from raw quote data without any data augmentation. Since Trading Gap Splitter is less appropriate for low-frequency data like daily quote data, we only perform MG-TF on our NASDAQ dataset.

Results. The performance of comparison experiments on both data are shown in Table 2, from which we have the following observations:

• The proposed approaches B-TF, MG-TF and HMGTF show significant gains on both metrics Acc and MCC compared with baselines in all cases. It exhibits that Transformer-based approaches have significant performance advantages over RNN- and CNN-based approaches.
• Transformer-based approaches have better capabilities of learning long-term dependencies. Especially on China A-shares data, in which the window size of 40day comtains 640 (40*16) time-steps, and it is hard for RNN-based approaches to learn the dependencies across so many time-steps (see Figure 4). While the advantages of the self-attention mechanism enables Transformerbased approaches achieve consistently better performance as the window size becomes larger.
• The modified Transformer model MG-TF and HMGTF have better performance than the basic Transformer model. More detailed analysis will be shown in the next section.

5.4 Incremental Analysis

To explore the effectiveness of the proposed components Multi-Scale Guassian Prior, Orthogonal Regularization and Trading Gap Splitter, we further conduct an incremental analysis on different settings of the Transformer-based models.

(a) Daily attention scores

As is shown in Table 3, these components all contribute to the performance of Transformer-based method. Moreover, we can further observe that the performance improvement mainly benefits from Multi-Scale Guassian Prior and Trading Gap Splitter, while the gain from Orthogonal Regularization is relatively insignificant.

Besides, we illustrate the effectiveness of Trading Gap Splitter by visualizing the attention score matrix learned by HMG-TF model with N = 160 (is equivalent to K = 10-day or 2-week). As is shown in Figure 5, the attention scope becomes larger gradually (16→80→160) from (a) to (c). That means, with the proposed hierarchical self-attentiom mechanism, the time-steps only attend to those belonging to the same day in the first self-attention block, the time-steps only attend to those belonging to the same week in the second block, and the attention scopes of time-steps are no limit in the last block.

6 Discussion &FutureWork

In this paper, we propose to apply Transformer model for stock movement prediction in which the attention mechanism can help to capture extremely long-term dependencies of finance time series. Furthermore, equipped with the proposed enhancements Multi-Scale Gaussian Prior, Orthogonal Regularization and Trading Gap Splitter, our Transformer-based model achieves significant gains over several state-of-the-art baselines on two real-world market dataset. In the future, except for the model itself, the following aspects can be investigated for further improvements: (1) cross-sectional features of financial data can be engaged to improve the model, (2) regularization methods can be explored to avoid suffering from overfitting on financial data, (3) data augmentation such as adversarial and stochastic perturbations can be adopted to improve the robustness of the model. Also, it is significant to investigate the theoretical guarantee for our method.