Relation-Aware Transformer for Portfolio Policy Learning

1 Introduction

Portfolio selection (PS) is an important research problem in computational finance [Li and Hoi, 2014; Li and Ng, 2000]. PS aims to maximize the long-term returns of wealth by dynamically allocating the wealth among a set of assets, e.g., stocks and crypto-currencies. Despite making great financial sense, it is very difficult for investors to handle such a laborious task, since even domain experts have to spend a large amount of effort/time in investigating each asset and managing portfolios. Recently, machine learning based methods have been proposed to address this task and have shown empirical improvements at PS [Agarwal et al., 2006; Das et al., 2014]. However, due to the complex nature of PS, existing methods may not be able to achieve promising performance in practice. Specifically, it is very hard to represent the non-stationary price series of assets, as they often contain significant noises and oscillations [Zhang et al., 2020].

∗Equal contributions. Work done as interns at Tencent AI Lab.

†Corresponding author.

1 The source code is available: https://github.com/Ivsxk/RAT.

The majority of existing PS methods [Cover and others, 1991; Shen and others, 2015] heavily rely on handcrafted features, such as moving average [Li and Hoi, 2012], stochastic technical indicators [Neely et al., 2014], etc. These features, however, have shown limited representation ability over practical price series [Zhang et al., 2020]. Recently, deep neural networks (DNNs) have become popular in sequential modeling and shown stronger sequential representation ability [LeCun et al., 2015]. However, directly applying existing DNNs for portfolios cannot extract price sequential features well due to two practical challenges listed below.

First, price sequences of assets often follow complex financial laws that are hard to capture using existing DNNs. For instance, asset prices contain complex short-term trends [Atsalakis and others, 2009], which potentially shed light on local patterns of price series. Here, the local pattern indicates the sequential pattern of a local price subsequence. Moreover, asset prices generally satisfy the long-term mean reversion principle [Poterba and Summers, 1988]. That is, the price of an asset will finally reflect its true value. In this sense, both long-term and local sequential patterns of price series are important for PS. To handle this, one may employ recurrent neural networks (RNNs) (such as LSTM [Hochreiter and others, 1997] and GRU [Cho et al., 2014]) to model price series. However, empirical studies [Li et al., 2019b; Zhang et al., 2020] have found that these methods are struggling to capture (very) long-term dependencies. Thus, how to exploit DNNs to effectively capture the sequential patterns for portfolio assets remains an open challenge.

Second, financial assets in portfolios contain complex correlations that may vary rapidly over time [Stefanova and Elkamhi, 2011]. Such asset correlations are important in mining financial data [Frye, 2008; Lopez, 2004], as they are helpful for the estimation of investment risk, thereby guiding more effective portfolio management. For instance, during economic downturns, investors may reduce investment risk by increasing portfolio diversification [Tasca et al., 2017] based on asset correlations; while during economic upturns, they may increase portfolio returns by exploiting the synergy of upward/related assets. However, existing DNNs for series modeling generally extract features for each temporal series, without particular considerations in modeling the correlations among assets. Hence, how to capture asset correlations in price sequences is another under-explored problem in PS.

In recent years, the attention mechanism [Vaswani et al., 2017] has gained great attention in compelling sequence modeling. More recently, Transformer [Vaswani et al., 2017], which relies on the attention mechanism, has shown great potential in grasping repeating sequential patterns with very long-term dependencies. However, Transformer cannot be directly applied to PS due to two reasons. First, in Transformer, the similarities between queries and keys in self-attention layers are computed based on point-wise values, which makes it incapable of capturing local context information and easily confused by local noisy points [Li et al., 2019b]. Moreover, as Transformer is primarily designed for word-level sequence translation, it is unable to capture the assets correlations that are useful in making portfolio decisions.

In this paper, we propose to exploit and improve Transformer for long-term sequence modeling in portfolios, so that it can well capture local sequential patterns of price series and the correlations among assets. Specifically, we propose a Relation-aware Transformer (RAT), which serves as a policy network for PS. RAT is structurally innovated to capture sequential patterns and asset correlations via newly proposed attention-based modules, and makes profitable portfolio decisions through a newly devised decision-making module. By exploiting reinforcement learning to train the policy network, RAT yields significant improvements in PS performance.

Our main contributions are summarized as follows:

• We propose a novel RAT method for PS. According to our best knowledge, this is the first attention-based method to simultaneously model complex sequential patterns and varying asset correlations for PS.
• By exploring financial leverage, we resolve a decisionmaking limitation in existing deep reinforcement learning methods for PS. Based on a new decision-making module, RAT makes more profitable portfolio decisions under the deep reinforcement learning framework.
• Extensive experiments on real-world datasets verify the significant superiority of our method, compared with state-of-the-art methods in PS, including both online learning and reinforcement learning based methods.

2 ProblemFormulation

Without loss of generality, we consider a portfolio selection (PS) task with m assets during a total number of n trading periods. Assume each asset has d kinds of prices. Here, we consider 4 kinds of prices, namely opening, highest, lowest and close prices (which means d = 4). As the prices often change over time, we use pt,i∈Rd+ to denote the price of asset i at period t (where t=1, .., n) and let Pt∈Rm×d be the prices for all assets at this period. Moreover, let Pt={Pt-k, ..,Pt-1}∈Rk×m×d be the price series of previous k-moment prices regarding period t.

Portfolio Selection as a Markov Decision Process. The price of assets is determined by many factors (such as the market), so it is impossible to annotate the data in advance. As a result, the traditional supervised learning paradigm is not suitable for modeling the PS process. In fact, given the decision nature of PS, it is more natural and convenient to model it as a Markov Decision Process (MDP). An MDP can be defined as a tuple (S,A, T ,R), where S denotes a finite state space, A denotes a finite set of actions, T (s′ |s, a) is a state transition function that defines the next state s′ given the current state s and action a, and R(s, a) is a reward function. Moreover, a policy π(a|s) determines an action a given the current state s. In the context of PS, the MDP model will be slightly different from the standard ones. Specifically, the action is specified by a portfolio vector at=[at,1, at,2, ...at,m]⊤∈Rm, where at,i indicates the wealth proportion regarding asset i and ∑mi=1 at,i=1. To construct a PS policy, at period t, an agent observes a state of price series st=Pt∈S, and takes an action at=π(Pt)∈A. Afterwards, the agent receives a reward rt∈R(s, a), while the next state st+1 is reached based on st+1∼T (st, at). In practice, if st+1 is mainly determined by the market, then st+1∼T (st).

In this paper, we aim to devise a policy network (served as π) to maximize the accumulated reward (e.g., the overall wealth of portfolios) via reinforcement learning (RL). However, it is non-trivial to devise such a policy network, as nonstationary price series and complex asset correlations make sequential modeling very difficult. Existing RL-based methods for PS directly use existing DNN models for PS, which, however, cannot extract the complex information well. To solve this, we propose a Relation-Aware Transformer.

3 Relation-Aware Transformer

In portfolio selection (PS), both price series patterns and asset correlations are significant for correct decision-making of portfolios and it is important to capture both types of information in the learning process. To this end, we propose a Relation-Aware Transformer (RAT), which will serve as a policy network for PS. The goal of such a policy network is to extract informative features for asset prices, and make portfolio decisions based on these features. To this end, we extend the standard Transformer structure by making it be relationaware and handle this task with reinforcement learning (RL).

3.1 General Architecture

RAT follows an encoder-decoder structure [Vaswani et al., 2017]. As shown in Figure 1, RAT consists of an encoder and a decoder, where the encoder is for sequential feature extraction, while the decoder is for decision making.

Encoder: In PS, both sequential patterns and asset correlations are necessary for correct decision-making of portfolios. Therefore, we propose two task-specific attention modules for the encoder. To be specific, a sequential attention layer is devised to capture sequential patterns for asset prices (See Section 3.2), and a relation attention layer is devised for capturing asset correlations (See Section 3.3).

Decoder: The decoder has network modules similar to the encoder, apart from a new decision-making layer (See Section 3.4). Such a layer aims to select portfolios by comprehensively considering the extracted features, local price context, and the decision of last period. Here, a new leverage operation is designed to enhance the decision making. Other components of the encoder-decoder structure, such as the positional encoding, feed forward layer and layer normalization, are identical to Transformer [Vaswani et al., 2017].

3.2 Sequential Attention Layer

Financial price series generally follows the mean reversion principle in the long term. Besides, a price series is often affected by surrounding events, such as interest rate cutting, leading to oscillations in the short term. Hence, it is nontrivial to devise DNNs for capturing such complex sequential patterns. Although self-attention works well in modeling long-term dependencies in Transformer, it falls short of exploiting local context since its query-key matching is computed based on point-wise values (as shown in Figure 2 (a)). As a result, standard self-attention may be confused by local noisy points, causing underlying optimization issues. To solve this, we devise a sequential attention layer, which enhances multi-head attention via new devised context attention. First, we introduce the multi-head attention scheme.

Multi-head Attention. With Pt∈Rk×m×d as the input, a multi-head attention processes the price series Pt,i∈Rk×d for each asset i∈{1, 2, ..,m}. Specifically, it first transforms Pt,i into H distinct query matrices Qht,i=Pt,iWQ h , key matrices Kht,i=Pt,iWK h and value matrices V h t,i=Pt,iWV h , respectively. Here, h∈{1, 2, ..,H} denotes the head index, while WQ h , WK h ,WV h ∈Rd×df denote parameters of linear projections and df indicates the dimension of the projected feature space. After linear projections, the scaled dot-product attention is adopted to compute the output values as follows:

where √df is a scale term [Vaswani et al., 2017]. We then concatenate the outputs of all heads, i.e., Ot,i∈Rk×Hdf . The final output is to concatenate all assets, i.e., Ot∈Rk×m×Hdf .

Mulit-head Attention

Linear Linear Linear

(a) Standard self-attentionMulti-head Attention

Linear Linear Linear

(b) Context-aware self-attention

Figure 2: The comparison between different self-attention mechanisms. (a) standard self-attention in Transformer can be confused by noisy points due to point-wise matching. Instead, (b) context-aware self-attention uses context attention to transform local price context into queries/keys, thus being more robust to local price noise.

Note that the dot-product attention in Eq. (1) cannot well exploit the context information. To address this, as shown in Figure 2 (b), instead of using dot-point projections for querykey matching, we propose a context attention scheme to transform the local context into queries and keys.

Context Attention. To enhance the queries and keys with locality, we explore short-term dependencies between the current price and local price context. Specifically, at period t, we first introduce how to obtain the query matrix Qht,i regarding head h in Eq. (1) with linear projection matrix ˆWQ h ∈Rd×df .

Without loss of generality, at any moment τ∈{1, 2, ..., k} in the price series Pt∈Rk×m×d , we input local price context P lτ ={Pτ-l, .., Pτ-1}∈Rl×m×d (instead of only the current price) into context attention. Here, l is the length of context, and we use padding when τ≤l. For asset i, the context attention first transforms local context P lτ,i∈Rl×d into the context-aware key matrix and value matrix ˆKhτ,i=ˆV h τ,i=P lτ,i ˆWQ h ∈Rl×df , while transforming the current price Pτ-1,i into context-agnostic query matrix ˆQhτ,i=Pτ-1,i ˆWQ h ∈R1×df . After projection, the query matrix can be obtained by exploiting the dependencies between context-aware key and context-agnostic query through scaled attention as follows:

We then concatenate the outputs of all moments to obtain the queries Qht,i∈Rk×df . The keys Kht,i can be computed in the same manner with different parameters ˆWK h . In this way, the queries and keys can be more aware of the local context and thus more robust to local price noise. When l=1, the sequential attention degrades to standard multi-head attention.

3.3 Relation Attention Layer

Assets in a portfolio inherently contain complex and varying correlations. Such correlations are important for PS since they reveal macro market trends and help to address investment risk by adjusting portfolio diversification. To capture the correlations, we devise a relation attention layer to enhance features after the sequential attention layer. With Oh t ∈Rk×m×df as the output of the h-th head in the previous sequential attention layer, the relation attention uses scaled self-attention to model asset correlations and enhance features Oh,j t ∈Rm×df for each time point j∈{1, .., k} as below:

The output of the h-th attention head is the concatenation of all time points by Z h t =Concat{Z h,1 t , ..,Z h,k t }∈Rk×m×df , while the output of the relation attention layer is to concatenate all heads by Zt=Concat{Z 1 t , ..,Z H t }∈Rk×m×Hdf .

3.4 Decision-making Layer

It is non-trivial to design a decision-making module for RAT due to the complexity of PS. Existing RL based methods for PS [Jiang et al., 2017; Liang and others, 2018] decide the portfolio through a fully connected layer with softmax. However, using a softmax layer inevitably enforces the proportion of assets to be positive, i.e., at,i∈(0, 1) for asset i, where ∑mi=1 at,i=1. This may make the agent suffer a huge loss of wealth when the asset prices decrease in the future. The critical problem is that even if the agent can predict the decrease, it cannot avoid the loss due to at,i∈(0, 1).

To enhance the decision-making ability of RAT, motivated by leverage [Mandelker and others, 1984], we introduce short sale [Shen and others, 2015; Shen and Wang, 2016] into the decision making. The short sale means that the policy network can first borrow some assets (whose prices are predicted to decrease by the network) for sale, and then reinvest the liquidated money into other assets (whose prices are predicted to increase). In this way, RAT is able to make more accurate decisions regarding each asset.

To this end, we devise a new leverage operation with three independent softmax fully-connected decision-making layer. Specifically, one head outputs an initial portfolio vector ˆat, and one head outputs a short sale vector ˆast , while the last one outputs a reinvestment vector ˆart . Based on the three heads, we improve the decision by considering both short sale and reinvestment. That is, the final portfolio vector is decided by at=ˆat-ˆast +ˆart . In this way, the proportion regarding asset i becomes at,i∈(-1, 2), where ∑mi=1 at,i=1. The negative sign of the weight indicates that investors will make money if asset price drops, while suffering a loss if the price rises.

In addition, since PS is also influenced by transaction costs, we adopt a recursive mechanism [Moody and Saffell, 2001] to avoid heavy transaction fees. That is, the decision-making requires taking into consideration the action from last period, which discourages very large changes between portfolios. To this end, we concatenate the portfolio vector from last period at-1 into feature maps so that RAT can make profitable portfolio decisions while constraining aggressive trading.

3.5 Learning with Reinforcement Learning

Since the asset prices are determined by many factors such as the financial market, it is impossible for us to annotate the data in advance and train RAT with supervised learning. To address this, we resort to reinforcement learning (RL). Specifically, we adopt a classical direct policy gradient algorithm [Moody and Saffell, 2001] to train the policy network (RAT) by maximizing a reward function. Here, we first denote the price change of all assets by a price relative vector yt:= pct pct-1 ∈Rm regarding period t, where pct indicates the close price. Based on the price relative vector, we devise the reward relying on the log-return of portfolios as follows:

where ct denotes the transaction cost, computed using the method presented in [Jiang et al., 2017]. The motivation lies in that the direct policy gradient algorithm guarantees at least a theoretical log-optimal strategy with the log-return as the reward. This can be proved by combining the previous theoretical results [Gy¨orfi and Vajda, 2008; Sutton et al., 2000]. One can also use more advanced RL methods, e.g., DDPG [Lillicrap and others, 2016] and PPO [Schulman and others, 2017]. However, we found that they usually fail to converge in PS in our preliminary studies. Since RL is not our main focus, we leave the study of more task-specific RL to future work.

4 Related work

Portfolio selection (PS) has attracted extensive research focus from the AI community [Shen and others, 2015; Shen and Wang, 2016]. Following the Kelly principle [Kelly, 1956], many types of PS methods have been proposed, including online learning and reinforcement learning based methods.

Online learning based methods aim to maximize the expected log-return in sequential decision-making [Zhao et al., 2018; Zhang et al., 2018]. Pioneering studies include UCRP [Cover and others, 1991], Anticor [Borodin et al., 2004], and ONS [Agarwal et al., 2006]. Recently, several methods have exploited the mean reversion property to select the portfolio, e.g., OLMAR [Li and Hoi, 2012] RMR [Huang et al., 2013] and SSR [Shen and Wang, 2017]. However, these methods failed to take into account the learning of sequential features and only used handcrafted features, such as moving average and stochastic technical indicators. This can lead to unsatisfactory PS performance due to limited feature representations [Deng and others, 2016].

Reinforcement learning (RL) based methods aim to optimize specific utility functions and learn comprehensive policies via RL algorithms [Moody and Saffell, 2001; Neuneier, 1998]. Very recently, several RL-based studies have used deep learning [Niu et al., 2020; Zhang et al., 2019] to extract features for assets [Guo et al., 2018; Jiang et al., 2017; Kang et al., 2018] and have achieved promising performance for PS. To be specific, the state-of-the-art ones are EIIE [Jiang et al., 2017] and the adversarial deep RL method [Liang and others, 2018]. These methods resort to deep learning techniques [Cao et al., 2019], e.g., CNNs or RNNs, as the sequential feature extractor for PS.

By comparison, our proposed RAT explores attention mechanism to capture in-depth the task-specific information, i.e., correlations among assets and long-term/local sequential patterns of assets. As a result, RAT learns more representative sequential features for PS.

Transformer is a powerful attention model primarily used in the field of NLP [Vaswani et al., 2017]. Note that transformer employs a encoder-decoder structure to capture the sequential dependencies between the source and the target sequence. It uses multiple attention heads (with different parameters of linear projections) to capture diverse aspects of sequential patterns. The outputs of all heads are concatenated and then fed into a fully connected feed-forward layer. In this paper, we exploit it for modeling long-term series dependencies and devise a Relation-aware Transformer for PS.

5 Experimental Results

To verify our proposed method, we evaluate RAT in terms of three main aspects: (1) the profitability on real-world datasets, (2) the feature representation ability for portfolios, and (3) the benefits of the financial leverage.

5.1 Experimental Settings

We first describe the experimental settings.

Datasets. We examine RAT on real-world crypto-currency and stock datasets. The statistics of all datasets are summarized in Table 1. All crypto-currency datasets are accessed with Poloniex 2 , where data selection is based on the method in [Jiang et al., 2017]. To be specific, we select the assets according to the crypto-currencies with top month trading volumes in Poloniex. We also evaluate our methods on the S&P500 stock dataset obtained from Kaggle 3 . Since decision-making of PS relies on the relative price, we normalize the price series of each asset by element-wise dividing the prices regarding the last period in the price series.

Baselines. We compare RAT with several advanced methods, including (1) online learning based methods: UCRP [Cover and others, 1991], Anticor [Borodin et al., 2004], SSR [Shen and Wang, 2017] OLMAR [Li and Hoi, 2012] and RMR [Huang et al., 2013]; (2) reinforcement learning (RL) based methods: ADDPG [Liang and others, 2018], EIIE [Jiang et al., 2017] and MTL [Li et al., 2019a]. All these RL methods use either LSTM or CNNs as the feature extractor. To evaluate the decision-making module, we also present a degenerate variant (RAT-B) that only uses a basic softmax in the decision-making layer without the leverage operation.

2 Poloniex's official API: https://poloniex.com/support/api/.

3 https://www.kaggle.com/camnugent/sandp500.

Metrics. Following [Shen and Wang, 2017], we use three metrics to evaluate performance. The first is the accumulated portfolio value: APV=Sn=S0 ∏nt=1 a⊤t yt(1 - ct), where S0=1 is the initial wealth. Such a metric evaluates the profitability when considering the transaction cost. A major drawback of APV is that it neglects the risk factor. To take risk into account, the second metric is the Sharp Ratio: SR= Average(rt-1) Standard Deviation(rt-1) . Although SR considers the volatility of portfolio values, it treats upward and downward movements equally, while downward movements are usually more important. To highlight the influence of downward deviations, the third metric is the Calmar Ratio: CR= Sn MDD , where MDD denotes the biggest loss from a peak to a trough and is calculated via MDD=maxt:τ>t St-Sτ St .

Implementation Details. RAT is implemented via pytorch. The number of attention heads is set to H=2, and the dimension of the feature space is set to df=12. In the training process, we adopt Adam optimizer on a single NVIDIA Tesla P40 GPU. The training step is 80000 for crypto-currency data and 20000 for stock data, where the batch size is 128. We set learning rate to 10-4 and weight decay of l2 regularizer to 10-7 . The transaction cost rate is 0.25%. The temporal length of the local context is set to l=5, while the length of the price series is k=30. We will show the parameter sensitivity analysis of RAT in a future long paper. In addition, the portfolio vector a0 is initialized by the average assignment. For all RL based methods, results are averaged over 5 runs with random initialization seeds.

5.2 Evaluation on Portfolio Selection

We report the results of all methods in Table 2. Overall, RAT outperforms all other baselines, which demonstrates strong profitability of our method in PS. In comparison, online learning based methods fail to perform well, since these methods do not consider the learning of sequential features for assets, which may lead to unsatisfactory decisions for portfolios. As for deep reinforcement learning methods, ADDPG fails in PS because its Q network is hard to train, leading to a poor portfolio policy. Although EIIE and MTL perform better than ADDPG, They are still worse than RAT-B, since their structure of policy network (e.g., LSTM or CNNs) suffers from limited sequential modeling ability for portfolio prices. Finally, RAT outperforms RAT-B by a large margin in terms of APV, which demonstrates the contribution of the decisionmaking layer and the leverage operation in PS.

5.3 Ablation Studies

To evaluate the proposed attention components, i.e., context attention (CA) and relation attention (RA), we compare RATB with three degenerate variants, i.e., Transformer (without CA and RA), RAT-B-CA (without CA) and RAT-B-RA (without RA). Here, we use RAT-B to eliminate the influence of leverage. According to the results reported in Table 3, both CA and RA contribute to the performance of RAT. This verifies the importance of capturing local sequential patterns and asset correlations. More specifically, CA contributes slightly more than RA, while combining both modules yields a significant improvement in performance.

5.4 Evaluation on Feature Representation

To evaluate the representation ability of the proposed method, we compare RAT-B (without the leverage operation) with three variants of RAT equipped with different feature extractors, i.e., CNN, LSTM, and CNN-LSTM (series connection). All variants in these experiments have the same optimization and decision-making mechanisms as RAT yet different network architectures. Specifically, the network architectures of CNN and LSTM use the architectures presented in EIIE [Jiang et al., 2017], while CNN-LSTM follows the network architecture in RWCLDNN [Sainath and others, 2015]. As shown in Table 4, RAT-B outperforms all variants, which verifies the strong series modeling ability of the attentionbased model. Moreover, by combining the results in Table 3, we come to a conclusion that the proposed policy network has superior representation ability for asset price series.

5.5 Evaluation on Leverage

We next evaluate the effectiveness of the leverage operation on RAT and another deep reinforcement learning method, i.e., EIIE, named as EIIE-L (with leverage). As shown in Figure 3, the leverage operation is beneficial to both RAT and EIIE, leading to significant improvement in terms of APV and CR. Moreover, the leverage operation also contributes slightly to the SR value. In conclusion, these results demonstrate the effectiveness of the leverage operation and confirm its importance for profitable decision-making for PS.

6 Conclusion

This paper has presented a novel Relation-aware Transformer (RAT) for learning portfolio policy. RAT is structurally novel to simultaneously capture sequential patterns, grasp asset correlations, and make profitable portfolio decisions. By exploiting reinforcement learning to train the policy network (i.e., RAT), our method yields significant performance improvements for portfolio selection. Extensive experiments, on real-world crypto-currency and stock datasets, demonstrate the superiority of RAT. According to the experimental results, we argue that (1) the attention mechanism can be regarded as a dominant scheme for PS, and (2) the leverage operation is of great benefit to the decision making of reinforcement learning based methods for PS.

Acknowledgments

This work was supported by Key-Area Research and Development Program of Guangdong Province (2018B010107001, 2019B010155002, 2019B010155001), National Natural Science Foundation of China (NSFC) 61836003 (key project), 2017ZT07X183, Tencent AI Lab Rhino-Bird Focused Research Program (No. JR201902), Fundamental Research Funds for the Central Universities D2191240.