Modeling the Stock Relation with Graph Network for Overnight Stock Movement Prediction

1 Introduction

Stock movement prediction is one of the most attractive topics in the Fintech area [Bollen et al., 2011]. Many researches are devoted to predicting the movement trend of stocks based on news or historic market information. Researchers try to predict the stock price based on historic market data [Feng et al., 2019], the stock related news [Hu et al., 2018] or the combination of both [Xu and Cohen, 2018]. These researches all focus on predicting on the level of a trading day. However, it is a widely accepted fact that the stock movement is highly stochastic and can be influenced by complicated factors [Malkiel, 1999]. Experts in the financial area agree that the time for the market to absorb the impact of news is uncertain, which ranges from a few minutes to hours, but usually less than a day. Therefore, using the news signal to predict the stock movement for the next day is not very reliable.

∗Contact Author

1 The code and dataset will be available in https://github.com/ liweitj47/overnight-stock-movement-prediction

In this paper, we explore the prediction of the stock movement in a more practicable way. We propose to predict the overnight stock movement based on the overnight financial news. Overnight movement means the movement between the closing price of the previous day and the opening price of the next day. Only the news happened after the market being closed is considered. This way, the reaction of the market on the news can be more precisely reflected because there is no trading operation during the closing hours of the market.

When predicting the stock movement of a company, previous works only consider the news and market data of a single company. This omits the connection among related stocks. It is a common knowledge for market participants that the stock price of a company is often related to others that have business connection. For example, the stock of Toyota is related to the stock of Honda, because they are both in the automobile industry. Therefore, in this work, we propose to consider the information of related stocks when predicting the stock movement instead of treating them as isolated ones.

To represent the connection between two companies, we propose to adopt the correlation matrix among companies, which market participants often refer to. This correlation matrix is calculated based on the correlation of historic market data, which introduces very valuable information. Inspired by the success of graph neural networks, we propose a Long Short Term Memory Relational Graph Convolution Networks model (LSTM-RGCN) to represent the correlation among stocks. In the graph, each stock is a node, and the stock nodes are connected by the correlation between the two stocks filtered by a threshold.

To test the effectiveness of our proposed model, we collect and publish an overnight stock movement prediction dataset of Reuters Financial News, which is widely used in the financial industry. In the dataset, there are the financial news and market data from Reuters during 01-01-2013 to 09-282018 for Tokyo Stock Exchange (TSE). The experiment results show that our model outperforms various strong baseline models. Moreover, the introduction of the graph structure indeed helps predict the stock movement. Further analysis suggest that our model can infer the price movement of stocks which are not associated with any news and the whole market because of the graph representation.

We conclude our contributions as follows:

• We propose a more practical objective that aims to predict the overnight movement. One big obstacle in stock prediction is the lack of data, in this work we publish the corresponding dataset from a professional content provider Reuters Financial News.
• We propose to consider the connection among stocks when predicting the stock movement and propose a LSTM-RGCNmodel to represent the connection.
• Extensive experiment results show that our model outperforms all the baseline models. Further analysis suggests that the introduction of the graph makes our model able to infer the price movement of related stocks that do not have news as well as the whole market.

2 Task Formulation and Dataset

In this section, we describe the task formulation and the dataset. This task aims to predict the overnight stock movement as positive or negative given the overnight news. By overnight movement, we mean the movement between the opening price of the current trading day pto and the closing price of the previous trading day pt-1 c :

Because stock price is volatile in normal cases, we consider the movement as positive or negative only when it exceeds 0.5 times of hourly standard deviation of the stock movement. By overnight news we mean the news that take place after the trading market being closed. We choose overnight news because the effect of normal news tends to be absorbed by the market within an hour or even few minutes during the trading hours of the day. On the contrary, the effect of the overnight news would be reflected on the overnight movement.

The dataset consists of the headline of the news and the target stock overnight movement. The news are associated with the stocks based on the "RIC" labels provided by Reuters. We choose the data during 2013-01-01 to 2018-09-28. Some statistics of the dataset is shown in Table 1.

3 Approach

In this section, we describe our Long Short Term Memory Relational Graph Convolutional Networks. Given the overnight news text of the stocks at one day, we want to predict the overnight stock price movement of the stocks that are attached with news. Our model first encodes news headline with a text encoder. Then we merge the news vector and the node embedding as the node vector. After that, we feed the node vectors to the LSTM-RGCN to get the final representation of the node. Finally, we predict the stock movement based on the node representation in the graph.

3.1 Stock Correlation Graph

To model the correlation among stocks, we build a stock correlation graph. In the graph, each node represents a stock. Each node is attached with some news text data. The nodes are connected in reference to a correlation matrix, which is calculated based on the historic market price. The correlation matrix will be published with the dataset. The historic market price considers the market movement information. Therefore, this correlation matrix provides very valuable information about the inter-stock relation. The correlation values can be either positive (including 0) or negative. Therefore, we define two kinds of relationships between nodes depending on the polarity of the value, positively correlated (correlation ≥ threshold) or negatively correlated (correlation < -threshold). To reduce the noise of the correlation matrix, we connect the two nodes only when the absolute value of the correlation score is above a threshold in the matrix.

3.2 Node News Encoder

LSTM has been successfully applied in encoding the context information of text data. Therefore, we propose to encode the news headline of a node with LSTM:

where xw is the word embedding, dt is the t-th word token in the news, htw is the hidden state of word dt .

Since different words are not equally important in the news, we propose to represent the sentence with attention mechanism. We choose the stock embedding as the query and do attention on the hidden vectors of the news words:

where xs is the stock embedding of the node. Ws is a learnable parameter matrix, [;;] means concatenation of vectors.

To represent the node feature, we combine the news text vector hn and the company embedding together:

where Wv is a learnable parameter matrix.

3.3 Graph Encoder

In this section, we describe our proposed LSTM RGCN based graph encoder.

GCN [Kipf and Welling, 2017] is able to model the graph structure, which is the correlation among stocks in this case. In our correlation matrix, there are two kinds of relationships representing positive and negative correlation relations. The original GCN is designed for the case where there is only one kind of relation. Therefore, we propose to adopt Relational Graph Convolutional Networks (RGCN) [Schlichtkrull et al., 2018] to encode the graph structure:

where Ar is the adjacency matrix of relation r, D- 12 AD- 1 2 is the normalized symmetric adjacency matrix. W l r is the learnable parameter matrix of the l-th layer for relation r. Wh is the learnable parameter matrix for the node vector. In our model, the parameter matrices are shared across layers. Hl represents the hidden representations of all the nodes in the l-th layer. N l+1 is the aggregated neighbor information for the (l + 1)-th layer.

Li et al. [2018] claim that GCN is vulnerable to the oversmoothing problem, which means that the value of different nodes would be very close after multiple layers of propagation. To alleviate this over-smoothing problem, we propose to add LSTM mechanism between RGCN layers so that the gate mechanism can dynamically select which part of the information should be transmitted to upper layers. Furthermore, we argue that the movement of one stock is related to the movement trend of the whole market. To model the movement trend of the whole market, we propose to add a global node to the graph, which can interact with each stock node. The LSTM process is calculated as follows:

where hv is the aggregated vector calculated with the RGCN, fθ is a one-layer feed forward network with sigmoid activation function and parameters θ. i, f, o indicate input, forget and output gates respectively. Different from the original design of LSTM, we also take the node embedding and vv and global node vector g into consideration. Vv serves as the role similar to residual connection, while g can provide the information of the whole market.

3.4 Global Node

To calculate the hidden state of the global node, we first aggregate the hidden information of all the nodes with attentive pooling:

where Wa, u are learnable parameters.

Then, we use LSTM mechanism to filter the aggregated global information based on the hidden state of the global node in the previous layer and the updated node representations of the current layer:

where fg, fi, og are the forget gate, input gate and the output gate of the global node, respectively.

3.5 Objective

After we get the hidden state of each node in the graph, we can predict the movement label:

where W is a learnable matrix, q is the gold label.

The task is modeled as a two-class classification problem. We use the standard cross entropy as the objective function.

4 Experiment

In this section, we describe the experiment setting, results and give detailed analysis.

4.1 Data

We choose the stocks within the TPX500 and TPX100 index. 2 Because the news data contains noisy news that do not influence the stock market, we first filter the news with the "RIC" label provided in the data by Reuters, which are the stock codes that the news may influence. Then we filter the news with some financial keywords described in the paper of Chen et al. [2019b]. Because much of the news is not related to the market price, we choose the keywords in the category of earnings, affairs, business, ratings and corporate. News that do not contain these keywords are filtered out. In the task, we only predict the movement when the both news is available and the price movement exceeds 0.5 times of hourly standard deviation. As the results, there are 10,367 positive movements and 8,050 negative ones in TPX500. There are 4,867 positive movements and 3,647 negative ones in TPX100. We choose the data in the period of 01-01-2018 ∼ 04-30-2018 as the development set and the data in the period of 05-01-2018 ∼ 0930-2018 as the test set. Some details of the data are described in Table 2.

2 Tokyo Stock Price Index, commonly known as TOPIX or TPX, is an important stock market index for the Tokyo Stock Exchange. TPX500 and TPX100 are the indexes for the top 500 and 100 stocks in TPX.

4.2 Baseline Models

In this part, we describe the baseline models.

• Random: this model is the random guess that randomly predict the movement to be improve or decline.
• Random Forest [Pagolu et al., 2016]: this model takes word embedding of the news headline as the input feature and applies Random Forest classifier 3 to predict the movement label. The word embedding is learned with GloVe 4 on the bloomberg news data.
• Naive Bayes: this model also takes the word embedding of the news headline as the input features, but applies Naive Bayes classifier 5 to predict the movement label.
• Linear Regression: this model also takes the word embedding of the news headline as the input features, but applies linear logistic regression 6 to predict the movement label.
• Hierarchical Attention Networks (HAN) [Yang et al., 2016]: a state of the art text classification model using hierarchical bidirectional LSTM structure with attentive pooling to encode the word and sentence. In our task, each headline is treated as a sentence in the HAN model.
• S-LSTM [Zhang et al., 2018]: a state of the art text representation model using LSTM to encode text. A global node is inserted to interact with each word.

3 https://scikit-learn.org/stable/modules/ensemble.html#forest

4 https://nlp.stanford.edu/software/GloVe-1.2.zip

5 https://scikit-learn.org/stable/modules/naive bayes.html# gaussian-naive-bayes

6 https://scikit-learn.org/stable/modules/linear model.html# logistic-regression

• Transformer [Vaswani et al., 2017]: a self-attention based model uses attention to encode context information of each word. A special "CLS" token is inserted in the front of the text, the hidden vector of which represents the whole text.

We use two kinds of word embeddings (GloVe and BERT) as the input features for the Random Forest, Naive Bayes and Linear regression three classifiers. For GloVe, we use the sum of the word embeddings. For BERT, we use the sentence vector. We do not use the sentence vector of BERT in our model because the vocabulary in the financial news headlines is very different from the vocabulary of the pre-trained BERT.

4.3 Setting

In the experiment, we set the layer number of S-LSTM and the proposed LSTM RGCN to be 3. The layer number of Transformer (baseline model) is 6. The headline length is truncated to 50. The maximum sentence number in hierarchical attention networks is truncated to 10. We set the threshold of correlation edge to 0.6, that is, only when the weight of the edge exceeds 0.6, there is an edge built between the two nodes. The embedding size of GloVe [Pennington et al., 2014] is 50. We use BERT (base) model to get the sentence vector, whose dimension is 768. We use Adam optimizer to train the model parameters. The learning rate is initially set to 0.001 and decayed by half after each iteration. The hidden size is 300.

4.4 Results

In Table 3 we show the experiment results. From the results we can see that our proposed model outperforms all other baseline models. The random guess generally results in an accuracy of around 50. Simple models can produce similar results compared with deep learning based baseline models. We assume that this is because the expression form of the financial news is relatively simple, which makes the deep learning based text classifiers do not have big advantage over the simple models.

Both the simple models and deep learning based baseline models do not perform as well as our proposed one. We argue that this is because the news in the market is still not enough to infer the movement of a stock. Even filtered with the topic keywords, there is still much noisy news that do not influence the price of the stock. Therefore, by introducing the information of relevant companies, our model can figure out the trend of the stock from the neighboring nodes and further validate the effect of the attached news.

4.5 The Effect of Graph

In Figure 2, we show the experiment result with and without the graph structure. From the figure we can see that the accuracy on both TPX500 and TPX100 increase by a big margin when adding the graph. We assume that this is because that the information from the related companies can supplement the news information of the current stock. Without the neighboring news, the model would suffer from information deficiency problem. Furthermore, by introducing the information of the related companies, our model can cross validate the effect of the news on the stock price.

4.6 Associative Inference

Because of the graph structure, our model is aware of the information of related companies. This makes our model able to learn the representation of a stock even though there is no directly attached news, which is realized by the information propagation via the graph edges. We call this ability the associative inference ability, which is the ability to predict the price movement of a stock where there is no attached news signal. In Table 4 we show the accuracy of associative inference. From the results we can observe that on both TPX500 and TPX100, our model can yield accuracy better than random guess (50%). The accuracy is especially high on TPX100, we assume that this is because the correlation among the big stocks provides more useful information. Other models cannot infer the movement of these stocks, because there is no available information.

4.7 Whole Index Inference

To test whether our model can capture the price movement of the whole market, we design an experiment that predicts the price movement of the whole TPX index. The training process remains the same, while during evaluation, we predict the index price movement Pindex based on the global graphlevel representation g (depicted in section 3.4).

The prediction process is the same as ordinary stocks. The parameter W is shared with the ordinary prediction in Eqn. 17. We use the news of TPX500 stocks. The prediction accuracy is 55.74, which is rather satisfactory.

Actually, the prediction of the index price movement is also an attractive objective. However, it is hard to infer the index price because there is no directly attached news and the data is quite limited compared with ordinary stock. In this paper, we provide a view that predicts the market level price movement based on the global node in the graph. The global node is calculated with attentive pooling on the stock nodes, which gives the model the ability to dynamically select the important information from the stock nodes.

4.8 Effect of Different News Topics

In Table 5 we show the results of data eliminating news of one specific topic. In the experiment, we iteratively eliminate the filtering keywords from one topic out of the five topics we use, which are "ratings, affairs, corporate, business and earnings". From the results we can see that the topic of "affairs" generally has the biggest influence on the stock especially for the TPX100. We assume that this is because the content of news in the "affairs" topic is generally negative and the pattern is easier to catch. As long as the model can detect a negative affair happening to a company, its stock is likely to be predicted to decline. When consulting to the domain experts, we expect that news of "earnings" topic would have the biggest influence on the price. However, the results show that news of "earnings" only has medium influence on the price. We assume that this is because the pattern behind the "earnings" topic is more complicated, which involves the digit numbers, the comparison between the published earnings and the earning expectation from the market and so on. By now, this kind of problem is still very difficult to solve especially with very limited data.

5 RelatedWork

5.1 Stock Prediction

Traditional researches use human engineered features to predict stock price movement. Schumaker and Chen [2009] propose to use features like noun phrases and entities to predict the stock price. Oliveira et al. [2013] propose to use several sentiment indicators to predict stock market variables. Ding et al. [2015] further propose to use the result of information retrieval systems to predict the event-based stock price. Qin et al. [2017] and Feng et al. [2019] propose to predict the stock price trend based solely on the market data sequence. However, the stock market is highly stochastic, it is almost impossible to predict the stock movement with only the historic market price. Hu et al. [2018] propose to predict stock price movement based on sequential news. Xu and Cohen [2018] propose to use sequential tweets and market data to predict the stock movement. Apart from stock movement prediction, Chen et al. [2019a] propose a hierarchical framework to predict the Forex movement by grouping and summarizing a large amount of finance text.

5.2 Graph Neural Networks on Text Representation

Graph neural networks are designed for graph structured data, which have been widely applied in various NLP tasks. Yao et al. [2019] propose to organize documents and words into a unified heterogeneous graph and apply GCN to learn the document node representation, which are later used to predict the label of the documents. Peng et al. [2018] propose to organize the text into local text windows that can be easily modeled by convolution operations. These works organize the text with word co-occurrence information and focus on building the graph based on the local intra-sentence information. Yasunaga et al. [2017] and Sahu et al. [2019] propose to capture the local and non-local dependencies the graph constructed out of the inter- and intra-sentence dependencies. Yin et al. [2019] and Li et al. [2019] propose to model the inter-sentence dependency with the shared entities between sentences.

6 Conclusion

In this paper, we propose the objective of overnight stock price movement prediction, which aims to predict the overnight stock price movement based on the overnight news that take place during the stock market closing time. Because the stocks are not independent, we propose to consider the information of related companies by introducing the stock correlation matrix, which is built based on the historic market information. To make better use of the graph structure, we propose the LSTM-RGCN model that can handle both the positive and negative correlation. Furthermore, the LSTM module makes the proposed model less vulnerable to the oversmoothing problem, which is faced by many graph based neural networks. Experiment results show that our model outperforms the strong baselines. Moreover, our model can infer the price movement of stocks that are not attached with any news as well as the whole market.

Acknowledgements

This work is supported by a Research Grant from Mizuho Securities Co., Ltd. Mizuho Securities also provide very valuable domain experts suggestions. The experiment data is provided by Mizuho Securities and Reuters.