{"type":"ScholarlyArticle","title":"StockFormer: Learning Hybrid Trading Machines with Predictive Coding","authors":[{"name":"Siyu Gao","slug":"siyu-gao","given":"Siyu","family":"Gao","affiliation":"MoE Key Lab of Artificial Intelligence , AI Institute , Shanghai Jiao Tong University"},{"name":"Yunbo Wang","slug":"yunbo-wang","given":"Yunbo","family":"Wang","affiliation":"MoE Key Lab of Artificial Intelligence , AI Institute , Shanghai Jiao Tong University"},{"name":"Xiaokang Yang","slug":"xiaokang-yang","given":"Xiaokang","family":"Yang","affiliation":"MoE Key Lab of Artificial Intelligence , AI Institute , Shanghai Jiao Tong University"}],"abstract":"Typical RL-for-finance solutions directly optimize trading policies over the noisy market data, such as stock prices and trading volumes, without explicitly considering the future trends and correlations of different investment assets as we humans do. In this paper, we present StockFormer, a hybrid trading machine that integrates the forward modeling capabilities of predictive coding with the advantages of RL agents in policy flexibility. The predictive coding part consists of three Transformer branches with modified structures, which respectively extract effective latent states of long-/short-term future dynamics and asset relations. The RL agent adaptively fuses these states and then executes an actor-critic algorithm in the unified state space. The entire model is jointly trained by propagating the critic's gradients back to the predictive coding module. StockFormer significantly outperforms existing approaches across three publicly available financial datasets in terms of portfolio returns and Sharpe ratios.","doi":"10.24963/ijcai.2023/530","url":"https://ijcai.org/proceedings/2023/530","pdf":"https://ijcai.org/proceedings/2023/0530.pdf","markdown":"https://ijcai.org/proceedings/2023/530.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/530","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Multidisciplinary Topics and Applications","track":"Main Track","number":530,"pages":{"first":4766,"last":4774},"keywords":["Machine Learning: ML: Deep reinforcement learning","Multidisciplinary Topics and Applications: MDA: Finance"],"videos":[],"references":[{"position":1,"raw":"[Basak and Chabakauri, 2010] Suleyman Basak and Georgy Chabakauri. Dynamic mean-variance asset allocation. The Review of Financial Studies, 23(8):2970-3016, 2010.","title":"Dynamic mean-variance asset allocation","authors":"Suleyman Basak, Georgy Chabakauri","year":"2010","venue":"The Review of Financial Studies","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Benhamou et al., 2020] Eric Benhamou, David Saltiel, Jean Jacques Ohana, Jamal Atif, and Rida Laraki. Deep reinforcement learning (DRL) for portfolio allocation. In ECMLPKDD, 2020.","title":"Deep reinforcement learning (DRL) for portfolio allocation","authors":"Eric Benhamou, David Saltiel, Jean Jacques Ohana, Jamal Atif, Rida Laraki","year":"2020","venue":"ECMLPKDD","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cho et al., 2019] Chun-Hung Cho, Guan-Yi Lee, Yueh-Lin Tsai, and Kun-Chan Lan. Toward stock price prediction using deep learning. In UCC, 2019.","title":"Toward stock price prediction using deep learning","authors":"Chun-Hung Cho, Guan-Yi Lee, Yueh-Lin Tsai, Kun-Chan Lan","year":"2019","venue":"UCC","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Duan et al., 2022] Yitong Duan, Lei Wang, Qizhong Zhang, and Jian Li. FactorVAE: A probabilistic dynamic fac- tor model based on variational autoencoder for predicting cross-sectional stock returns. In AAAI, 2022.","title":"FactorVAE: A probabilistic dynamic factor model based on variational autoencoder for predicting cross-sectional stock returns","authors":"Yitong Duan, Lei Wang, Qizhong Zhang, Jian Li","year":"2022","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Elias, 1955] Peter Elias. Predictive coding-I. IRE transac- tions on information theory, 1(1):16-24, 1955.","title":"Predictive coding-I","authors":"Peter Elias","year":"1955","venue":"IRE transactions on information theory","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fedus et al., 2021] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv preprint arXiv:2101.03961, 2021.","title":"Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity","authors":"William Fedus, Barret Zoph, Noam Shazeer","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Feng et al., 2019] Fuli Feng, Xiangnan He, Xiang Wang, Cheng Luo, Yiqun Liu, and Tat-Seng Chua. Temporal relational ranking for stock prediction. ACM Transactions on Information Systems, 37(2):1-30, 2019.","title":"Temporal relational ranking for stock prediction","authors":"Fuli Feng, Xiangnan He, Xiang Wang, Cheng Luo, Yiqun Liu, Tat-Seng Chua","year":"2019","venue":"ACM Transactions on Information Systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Haarnoja et al., 2018] Tuomas Haarnoja, Aurick Zhou, Kris- tian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905, 2018.","title":"Soft actor-critic algorithms and applications","authors":"Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hoseinzade and Haratizadeh, 2019] Ehsan Hoseinzade and Saman Haratizadeh. CNNpred: CNN-based stock market prediction using a diverse set of variables. Expert Systems with Applications, 129:273-285, 2019.","title":"CNNpred: CNN-based stock market prediction using a diverse set of variables","authors":"Ehsan Hoseinzade, Saman Haratizadeh","year":"2019","venue":"Expert Systems with Applications","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hu and Lin, 2019] Yuh-Jong Hu and Shang-Jen Lin. Deep reinforcement learning for optimizing finance portfolio management. In AICAI, 2019.","title":"Deep reinforcement learning for optimizing finance portfolio management","authors":"Lin ; Yuh-Jong Hu, Shang-Jen Hu, Lin","year":"2019","venue":"AICAI","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hu et al., 2018] Ziniu Hu, Weiqing Liu, Jiang Bian, Xu- anzhe Liu, and Tie-Yan Liu. Listening to chaotic whispers: A deep learning framework for news-oriented stock trend prediction. In WSDM, 2018.","title":"Listening to chaotic whispers: A deep learning framework for news-oriented stock trend prediction","authors":"Ziniu Hu, Weiqing Liu, Jiang Bian, Xuanzhe Liu, Tie-Yan Liu","year":"2018","venue":"WSDM","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Huotari et al., 2020] Tommi Huotari, Jyrki Savolainen, and Mikael Collan. Deep reinforcement learning agent for S&P 500 stock selection. Axioms, 9(4):130, 2020.","title":"Deep reinforcement learning agent for S&P 500 stock selection","authors":"Tommi Huotari, Jyrki Savolainen, Mikael Collan","year":"2020","venue":"Axioms","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kitaev et al., 2020] Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In ICLR, 2020.","title":"Reformer: The efficient transformer","authors":"Kitaev","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lee et al., 2020] Alex X Lee, Anusha Nagabandi, Pieter Abbeel, and Sergey Levine. Stochastic latent actor-critic: Deep reinforcement learning with a latent variable model. In NeurIPS, 2020.","title":"Stochastic latent actor-critic: Deep reinforcement learning with a latent variable model","authors":"Alex X Lee, Anusha Nagabandi, Pieter Abbeel, Sergey Levine","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2018] Hao Li, Yanyan Shen, and Yanmin Zhu. Stock price prediction using attention-based multi-input LSTM. In ACML, 2018.","title":"Stock price prediction using attention-based multi-input LSTM","authors":"Hao Li, Yanyan Shen, Yanmin Zhu","year":"2018","venue":"ACML","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liang et al., 2018] Zhipeng Liang, Hao Chen, Junhao Zhu, Kangkang Jiang, and Yanran Li. Adversarial deep rein- forcement learning in portfolio management. arXiv preprint arXiv:1808.09940, 2018.","title":"Adversarial deep reinforcement learning in portfolio management","authors":"Zhipeng Liang, Hao Chen, Junhao Zhu, Kangkang Jiang, Yanran Li","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lillicrap et al., 2016] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In ICLR, 2016.","title":"Continuous control with deep reinforcement learning","authors":"Jonathan J Timothy P Lillicrap, Alexander Hunt, Nicolas Pritzel, Tom Heess, Yuval Erez, David Tassa, Daan Silver, Wierstra","year":"2016","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Liu et al., 2021] Xiao-Yang Liu, Hongyang Yang, Jiechao Gao, and Christina Dan Wang. FinRL: Deep reinforcement learning framework to automate trading in quantitative fi- nance. In ICAIF, 2021.","title":"FinRL: Deep reinforcement learning framework to automate trading in quantitative finance","authors":"Xiao-Yang Liu, Hongyang Yang, Jiechao Gao, Christina Dan Wang","year":"2021","venue":"ICAIF","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Mikolov et al., 2013] Tomas Mikolov, Kai Chen, Greg Cor- rado, and Jeffrey Dean. Efficient estimation of word repre- sentations in vector space. In ICLR, 2013.","title":"Efficient estimation of word representations in vector space","authors":"Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean","year":"2013","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Oord et al., 2018] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018.","title":"Representation learning with contrastive predictive coding","authors":"Aaron Van Den Oord, Yazhe Li, Oriol Vinyals","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Patil et al., 2020] Pratik Patil, Ching-Seh Mike Wu, Kate- rina Potika, and Marjan Orang. Stock market prediction using ensemble of graph theory, machine learning and deep learning models. In ICSIM, 2020.","title":"Stock market prediction using ensemble of graph theory, machine learning and deep learning models","authors":"Pratik Patil, Ching-Seh Mike Wu, Katerina Potika, Marjan Orang","year":"2020","venue":"ICSIM","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Puterman, 2014] Martin L Puterman. Markov decision pro- cesses: discrete stochastic dynamic programming. John Wiley & Sons, 2014.","title":"Markov decision processes: discrete stochastic dynamic programming","authors":"Martin L Puterman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Qin et al., 2017] Yao Qin, Dongjin Song, Haifeng Chen, Wei Cheng, Guofei Jiang, and Garrison Cottrell. A dual-stage attention-based recurrent neural network for time series prediction. arXiv preprint arXiv:1704.02971, 2017.","title":"A dual-stage attention-based recurrent neural network for time series prediction","authors":"Dongjin Yao Qin, Haifeng Song, Wei Chen, Guofei Cheng, Garrison Jiang, Cottrell","year":"2017","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2017/366"},{"position":24,"raw":"[Rao and Ballard, 1999] Rajesh PN Rao and Dana H Ballard. Predictive coding in the visual cortex: A functional interpre- tation of some extra-classical receptive-field effects. Nature neuroscience, 2(1):79-87, 1999.","title":"Predictive coding in the visual cortex: A functional interpretation of some extra-classical receptive-field effects","authors":"P N Rajesh, Dana H Rao, Ballard","year":"1999","venue":"Nature neuroscience","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Spratling, 2017] Michael W Spratling. A review of predic- tive coding algorithms. Brain and cognition, 112:92-97, 2017.","title":"A review of predictive coding algorithms","authors":"W Michael, Spratling","year":"2017","venue":"Brain and cognition","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Suri et al., 2021] Karush Suri, Xiao Qi Shi, Konstantinos Plataniotis, and Yuri Lawryshyn. TradeR: Practical deep hi- erarchical reinforcement learning for trade execution. arXiv preprint arXiv:2104.00620, 2021.","title":"TradeR: Practical deep hierarchical reinforcement learning for trade execution","authors":"Karush Suri, Xiao Qi Shi, Konstantinos Plataniotis, Yuri Lawryshyn","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Théate and Ernst, 2021] Thibaut Théate and Damien Ernst. An application of deep reinforcement learning to algorith- mic trading. Expert Systems with Applications, 173:114632, 2021.","title":"Thibaut Théate and Damien Ernst. An application of deep reinforcement learning to algorithmic trading","authors":"Ernst Théate","year":"2021","venue":"Expert Systems with Applications","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wang et al., 2021] Heyuan Wang, Shun Li, Tengjiao Wang, and Jiayi Zheng. Hierarchical adaptive temporal-relational modeling for stock trend prediction. In IJCAI, 2021.","title":"Hierarchical adaptive temporal-relational modeling for stock trend prediction","authors":"Heyuan Wang, Shun Li, Tengjiao Wang, Jiayi Zheng","year":"2021","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2021/508"},{"position":30,"raw":"[Wen et al., 2019] Min Wen, Ping Li, Lingfei Zhang, and Yan Chen. Stock market trend prediction using high-order information of time series. IEEE Access, 7:28299-28308, 2019.","title":"Stock market trend prediction using high-order information of time series","authors":"Wen","year":"2019","venue":"IEEE Access","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Weng et al., 2020] Liguo Weng, Xudong Sun, Min Xia, Jia Liu, and Yiqing Xu. Portfolio trading system of digital currencies: A deep reinforcement learning with multidi- mensional attention gating mechanism. Neurocomputing, 402:171-182, 2020.","title":"Portfolio trading system of digital currencies: A deep reinforcement learning with multidimensional attention gating mechanism","authors":"Liguo Weng, Xudong Sun, Min Xia, Jia Liu, Yiqing Xu","year":"2020","venue":"Neurocomputing","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Wu et al., 2019] Jheng-Long Wu, Chi-Sheng Yang, Kai- Hsuan Liu, and Min-Tzu Huang. A deep learning model for dimensional valencearousal intensity prediction in stock market. In iCAST, 2019.","title":"A deep learning model for dimensional valencearousal intensity prediction in stock market","authors":"Jheng-Long Wu, Chi-Sheng Yang, Kai-Hsuan Liu, Min-Tzu Huang","year":"2019","venue":"iCAST","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Wu et al., 2021] Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transform- ers with auto-correlation for long-term series forecasting. In NeurIPS, 2021.","title":"Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting","authors":"Wu","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Yarats et al., 2021] Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Improving sample efficiency in model-free reinforcement learning from images. In AAAI, 2021.","title":"Improving sample efficiency in model-free reinforcement learning from images","authors":"Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, Rob Fergus","year":"2021","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Ye et al., 2020] Yunan Ye, Hengzhi Pei, Boxin Wang, Pin- Yu Chen, Yada Zhu, Ju Xiao, and Bo Li. Reinforcement- learning based portfolio management with augmented asset movement prediction states. In AAAI, 2020.","title":"Reinforcementlearning based portfolio management with augmented asset movement prediction states","authors":"Yunan Ye, Hengzhi Pei, Boxin Wang, Pin-Yu Chen, Yada Zhu, Ju Xiao, Bo Li","year":"2020","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Zhang et al., 2017] Liheng Zhang, Charu Aggarwal, and Guo-Jun Qi. Stock price prediction via discovering multi- frequency trading patterns. In KDD, 2017.","title":"Stock price prediction via discovering multifrequency trading patterns","authors":"Zhang","year":"2017","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Zhong et al., 2020] Yueyang Zhong, YeeMan Bergstrom, and Amy R Ward. Data-driven market-making via model- free learning. In IJCAI, 2020.","title":"Data-driven market-making via modelfree learning","authors":"Zhong","year":"2020","venue":"IJCAI","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Zhou et al., 2021] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In AAAI, 2021.","title":"Informer: Beyond efficient transformer for long sequence time-series forecasting","authors":"Zhou","year":"2021","venue":"AAAI","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}