{"type":"ScholarlyArticle","title":"Relation-Aware Transformer for Portfolio Policy Learning","authors":[{"name":"Ke Xu","slug":"ke-xu","given":"Ke","family":"Xu","affiliation":"South China University of Technology , Guangzhou , China; Pazhou Lab , Guangzhou , China"},{"name":"Yifan Zhang","slug":"yifan-zhang","given":"Yifan","family":"Zhang","affiliation":"South China University of Technology , Guangzhou , China; Pazhou Lab , Guangzhou , China"},{"name":"Deheng Ye","slug":"deheng-ye","given":"Deheng","family":"Ye","affiliation":"Tencent AI Lab , Shenzhen , China"},{"name":"Peilin Zhao","slug":"peilin-zhao","given":"Peilin","family":"Zhao","affiliation":"Tencent AI Lab , Shenzhen , China"},{"name":"Mingkui Tan","slug":"mingkui-tan","given":"Mingkui","family":"Tan","affiliation":"South China University of Technology , Guangzhou , China"}],"abstract":"Portfolio selection   is an important yet challenging task in AI for FinTech. One of the key issues is how to represent the non-stationary price series of assets in a portfolio, which is important for portfolio decisions. The existing methods, however, fall short of capturing: 1) the complicated sequential patterns for asset price series and 2) the price correlations among multiple assets.  In this paper, under a deep reinforcement learning paradigm for portfolio selection, we propose a novel Relation-aware Transformer (RAT) to handle these aspects. Specifically, being equipped with our newly developed attention modules, RAT is structurally innovated  to capture both sequential patterns and asset correlations for portfolio selection. Based on the extracted sequential features, RAT is able to make profitable portfolio decisions regarding each asset via a newly devised leverage operation. Extensive experiments on real-world crypto-currency and stock datasets verify the state-of-the-art performance of RAT.","doi":"10.24963/ijcai.2020/641","url":"https://ijcai.org/proceedings/2020/641","pdf":"https://ijcai.org/proceedings/2020/0641.pdf","markdown":"https://ijcai.org/proceedings/2020/641.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/641","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Special Track on AI in FinTech","track":"Special Track on AI in FinTech","number":641,"pages":{"first":4647,"last":4653},"keywords":["AI for trading: AI for portfolio analytics","AI for wealth: AI for digital asset management","Foundation for AI in FinTech: Reinforcement learning for FinTech"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24940/file_24940.m3u8","length_sec":276,"provider":"hls"}],"references":[{"position":1,"raw":"[Agarwal et al., 2006] Amit Agarwal, Elad Hazan, et al. Algo- rithms for portfolio management based on the newton method. In ICML, 2006.","title":"Algorithms for portfolio management based on the newton method","authors":"Amit Agarwal, Elad Hazan","year":"2006","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Atsalakis and others, 2009] George Atsalakis et al. Forecast- ing stock market short-term trends using a neuro-fuzzy based methodology. Expert Systems with Applications, 2009.","title":"Forecasting stock market short-term trends using a neuro-fuzzy based methodology","authors":"George Atsalakis","year":"2009","venue":"Expert Systems with Applications","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Borodin et al., 2004] Allan Borodin, Ran El-Yaniv, and Vincent Gogan. Can we learn to beat the best stock. In NeurIPS, 2004.","title":"Can we learn to beat the best stock","authors":"Allan Borodin, Ran El-Yaniv, Vincent Gogan","year":"2004","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cao et al., 2019] Jiezhang Cao, Langyuan Mo, Yifan Zhang, et al. Multi-marginal wasserstein gan. In Advances in Neural Informa- tion Processing Systems, pages 1774-1784, 2019.","title":"Multi-marginal wasserstein gan","authors":"Cao","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cho et al., 2014] Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural ma- chine translation: Encoder-decoder approaches. arXiv, 2014.","title":"On the properties of neural machine translation: Encoder-decoder approaches","authors":"Cho","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"Thomas M Cover et al. Universal portfo- lios. In Mathematical Finance. World Scientific, 1991.","title":"Universal portfolios","authors":"M Thomas, Cover","year":"1991","venue":"Mathematical Finance","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Das et al., 2014] Puja Das, Nicholas Johnson, et al. Online portfo- lio selection with group sparsity. In AAAI, 2014.","title":"Online portfolio selection with group sparsity","authors":"Das","year":"2014","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Deng and others, 2016] Yue Deng et al. Deep direct reinforcement learning for financial signal representation and trading. IEEE Transactions on Neural Networks and Learning Systems, 2016.","title":"Deep direct reinforcement learning for financial signal representation and trading","authors":"Yue Deng","year":"2016","venue":"IEEE Transactions on Neural Networks and Learning Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Frye, 2008] Jon Frye. Correlation and asset correlation in the structural portfolio model. The Journal of Credit Risk, 4, 2008.","title":"Correlation and asset correlation in the structural portfolio model","authors":"Jon Frye, Frye","year":"2008","venue":"The Journal of Credit Risk","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Guo et al., 2018] Yifeng Guo, Xingyu Fu, et al. Robust log- optimal strategy with reinforcement learning. arXiv, 2018.","title":"Robust logoptimal strategy with reinforcement learning","authors":"Yifeng Guo, Xingyu Fu","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Györfi and Vajda, 2008] László Györfi and István Vajda. Growth optimal investment with transaction costs. In COLT, 2008.","title":"Growth optimal investment with transaction costs","authors":"Vajda Györfi, László Györfi, István Vajda","year":"2008","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hochreiter and others, 1997] Sepp Hochreiter et al. Long short- term memory. Neural Computation, 1997.","title":"Long shortterm memory","authors":"Sepp Hochreiter","year":"1997","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Huang et al., 2013] Dingjiang Huang, Junlong Zhou, Bin Li, Steven HOI, and Shuigeng Zhou. Robust median reversion strat- egy for on-line portfolio selection. In IJCAI, 2013.","title":"Robust median reversion strategy for on-line portfolio selection","authors":"Huang","year":"2013","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2013/295"},{"position":14,"raw":"[Jiang et al., 2017] Zhengyao Jiang, Dixing Xu, and Jinjun Liang. A deep reinforcement learning framework for the financial port- folio management problem. arXiv, 2017.","title":"A deep reinforcement learning framework for the financial portfolio management problem","authors":"Jiang","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kang et al., 2018] Qinma Kang, Huizhuo Zhou, et al. An asyn- chronous advantage actor-critic reinforcement learning method for stock selection and portfolio management. In ICBDR, 2018.","title":"An asynchronous advantage actor-critic reinforcement learning method for stock selection and portfolio management","authors":"Kang","year":"2018","venue":"ICBDR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kelly, 1956] JL Kelly. A new interpretation of information rate. Bell System Technical Journal, 1956.","title":"A new interpretation of information rate","authors":"Kelly","year":"1956","venue":"Bell System Technical Journal","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[LeCun et al., 2015] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521, 2015.","title":"Deep learning","authors":"Yann Lecun, Yoshua Bengio, Geoffrey Hinton","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Li and Hoi, 2012] Bin Li and Steven CH Hoi. On-line portfolio selection with moving average reversion. arXiv, 2012.","title":"On-line portfolio selection with moving average reversion","authors":"Hoi ; Li, Bin Li, Steven Ch Hoi","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Li and Hoi, 2014] Bin Li and Steven CH Hoi. Online portfolio selection: A survey. ACM Computing Surveys, 46, 2014.","title":"Online portfolio selection: A survey","authors":"Hoi ; Li, Bin Li, Steven Ch Hoi","year":"2014","venue":"ACM Computing Surveys","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Li and Ng, 2000] Duan Li and Wan-Lung Ng. Optimal dy- namic portfolio selection: Multiperiod mean-variance formula- tion. Mathematical Finance, 2000.","title":"Optimal dynamic portfolio selection: Multiperiod mean-variance formulation","authors":"Li, Ng ; ] Duan Li, Wan-Lung Ng","year":"2000","venue":"Mathematical Finance","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Li et al., 2019a] Jianquan Li, Xiaokang Liu, et al. An empirical evaluation of multi-task learning in deep neural networks for nat- ural language processing. arXiv, 2019.","title":"An empirical evaluation of multi-task learning in deep neural networks for natural language processing","authors":"Li","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Li et al., 2019b] Shiyang Li, Xiaoyong Jin, et al. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. In NeurIPS, 2019.","title":"Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting","authors":"Li","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Liang and others, 2018] Zhipeng Liang et al. Adversarial deep re- inforcement learning in portfolio management. arXiv, 2018.","title":"Adversarial deep reinforcement learning in portfolio management","authors":"Others ; Zhipeng Liang, Liang","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Lillicrap and others, 2016] Timothy Lillicrap et al. Continuous control with deep reinforcement learning. ICLR, 2016.","title":"Continuous control with deep reinforcement learning","authors":"Timothy Lillicrap","year":"2016","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Lopez, 2004] Jose A Lopez. The empirical relationship between average asset correlation, firm probability of default, and asset size. Journal of Financial Intermediation, 13, 2004.","title":"The empirical relationship between average asset correlation, firm probability of default, and asset size","authors":"Jose A Lopez","year":"2004","venue":"Journal of Financial Intermediation","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Mandelker and others, 1984] G Mandelker et al. The impact of the degrees of operating and financial leverage on systematic risk of common stock. Financial and Quantitative Analysis, 1984.","title":"The impact of the degrees of operating and financial leverage on systematic risk of common stock","authors":"Mandelker","year":"1984","venue":"Financial and Quantitative Analysis","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Moody and Saffell, 2001] John Moody and Matthew Saffell. Learning to trade via direct reinforcement. IEEE Transactions on Neural Networks, 2001.","title":"Learning to trade via direct reinforcement","authors":"Saffell Moody, John Moody, Matthew Saffell","year":"2001","venue":"IEEE Transactions on Neural Networks","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Neely et al., 2014] Christopher J Neely, David E Rapach, Jun Tu, and Guofu Zhou. Forecasting the equity risk premium: the role of technical indicators. Management Science, 60, 2014.","title":"Forecasting the equity risk premium: the role of technical indicators","authors":"J Christopher, David E Neely, Jun Rapach, Guofu Tu, Zhou","year":"2014","venue":"Management Science","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Neuneier, 1998] Ralph Neuneier. Enhancing q-learning for opti- mal asset allocation. In NeurIPS, 1998.","title":"Ralph Neuneier. Enhancing q-learning for optimal asset allocation","authors":"Neuneier","year":"1998","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Niu et al., 2020] Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Yong Guo, Peilin Zhao, et al. Disturbance-immune weight shar- ing for neural architecture search. arXiv, 2020.","title":"Disturbance-immune weight sharing for neural architecture search","authors":"Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Yong Guo, Peilin Zhao","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Poterba and Summers, 1988] James M Poterba and Lawrence H Summers. Mean reversion in stock prices: Evidence and im- plications. Journal of Financial Economics, 22, 1988.","title":"Mean reversion in stock prices: Evidence and implications","authors":"M James, Lawrence H Poterba, Summers","year":"1988","venue":"Journal of Financial Economics","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Sainath and others, 2015] Tara Sainath et al. Learning the speech front-end with raw waveform cldnns. In INTERSPEECH, 2015.","title":"Learning the speech front-end with raw waveform cldnns","authors":"Tara Sainath","year":"2015","venue":"INTERSPEECH","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Schulman and others, 2017] John Schulman et al. Proximal policy optimization algorithms. arXiv, 2017.","title":"Proximal policy optimization algorithms","authors":"John Schulman","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Shen and others, 2015] Weiwei Shen et al. Portfolio choices with orthogonal bandit learning. In IJCAI, 2015.","title":"Portfolio choices with orthogonal bandit learning","authors":"Others ; Weiwei Shen, Shen","year":"2015","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2015/142"},{"position":35,"raw":"[Shen and Wang, 2016] Weiwei Shen and Jun Wang. Portfolio blending via thompson sampling. In IJCAI, 2016.","title":"Portfolio blending via thompson sampling","authors":"Wang ; Shen, Weiwei Shen, Jun Wang","year":"2016","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2016/283"},{"position":36,"raw":"[Shen and Wang, 2017] Weiwei Shen and Jun Wang. Portfolio se- lection via subset resampling. In AAAI, 2017.","title":"Portfolio selection via subset resampling","authors":"Wang ; Shen, Weiwei Shen, Jun Wang","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Stefanova and Elkamhi, 2011] Denitsa Stefanova and Redouane Elkamhi. Dynamic correlation or tail dependence hedging for portfolio selection. In AFA Chicago Meetings Paper, 2011.","title":"Dynamic correlation or tail dependence hedging for portfolio selection","authors":"Denitsa Stefanova, Redouane Elkamhi","year":"2011","venue":"AFA Chicago Meetings Paper","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Sutton et al., 2000] Richard S Sutton, David A McAllester, et al. Policy gradient methods for reinforcement learning with function approximation. In NeurIPS, 2000.","title":"Policy gradient methods for reinforcement learning with function approximation","authors":"David A Richard S Sutton, Mcallester","year":"2000","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Tasca et al., 2017] Paolo Tasca, Stefano Battiston, and Andrea Deghi. Portfolio diversification and systemic risk in interbank networks. Journal of Economic Dynamics and Control, 2017.","title":"Portfolio diversification and systemic risk in interbank networks","authors":"Paolo Tasca, Stefano Battiston, Andrea Deghi","year":"2017","venue":"Journal of Economic Dynamics and Control","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, et al. At- tention is all you need. In NeurIPS, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Zhang et al., 2018] Yifan Zhang, Peilin Zhao, Jiezhang Cao, et al. Online adaptive asymmetric active learning for budgeted imbal- anced data. In SIGKDD, pages 2768-2777, 2018.","title":"Online adaptive asymmetric active learning for budgeted imbalanced data","authors":"Yifan Zhang, Peilin Zhao, Jiezhang Cao","year":"2018","venue":"SIGKDD","doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Zhang et al., 2019] Yifan Zhang, Ying Wei, et al. Collaborative unsupervised domain adaptation for medical image diagnosis. In Medical Imaging meets NeurIPS, 2019.","title":"Collaborative unsupervised domain adaptation for medical image diagnosis","authors":"Yifan Zhang, Ying Wei","year":"2019","venue":"Medical Imaging meets NeurIPS","doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Zhang et al., 2020] Yifan Zhang, Peilin Zhao, et al. Cost-sensitive portfolio selection via deep reinforcement learning. IEEE Trans- actions on Knowledge and Data Engineering, 2020.","title":"Cost-sensitive portfolio selection via deep reinforcement learning","authors":"Yifan Zhang, Peilin Zhao","year":"2020","venue":"IEEE Transactions on Knowledge and Data Engineering","doi":null,"url":null,"internal_url":null},{"position":44,"raw":"[Zhao et al., 2018] Peilin Zhao, Yifan Zhang, et al. Adaptive cost- sensitive online classification. IEEE Transactions on Knowledge and Data Engineering, 31(2):214-228, 2018.","title":"Adaptive costsensitive online classification","authors":"Peilin Zhao, Yifan Zhang","year":"2018","venue":"IEEE Transactions on Knowledge and Data Engineering","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}