{"type":"ScholarlyArticle","title":"Contrastive Learning and Reward Smoothing for Deep Portfolio Management","authors":[{"name":"Yun-Hsuan Lien","slug":"yun-hsuan-lien","given":"Yun-Hsuan","family":"Lien","affiliation":"National Yang Ming Chiao Tung University , Taiwan"},{"name":"Yuan-Kui Li","slug":"yuan-kui-li","given":"Yuan-Kui","family":"Li","affiliation":"National Yang Ming Chiao Tung University , Taiwan"},{"name":"Yu-Shuen Wang","slug":"yu-shuen-wang","given":"Yu-Shuen","family":"Wang","affiliation":"National Yang Ming Chiao Tung University , Taiwan"}],"abstract":"In this study, we used reinforcement learning (RL) models to invest assets in order to earn returns. The models were trained to interact with a simulated environment based on historical market data and learn trading strategies. However, using deep neural networks based on the returns of each period can be challenging due to the unpredictability of financial markets. As a result, the policies learned from training data may not be effective when tested in real-world situations. To address this issue, we incorporated contrastive learning and reward smoothing into our training process. Contrastive learning allows the RL models to recognize patterns in asset states that may indicate future price movements. Reward smoothing, on the other hand, serves as a regularization technique to prevent the models from seeking immediate but uncertain profits. We tested our method against various traditional financial techniques and other deep RL methods, and found it to be effective in both the U.S. stock market and the cryptocurrency market. Our source code is available at https://github.com/sophialien/FinTech-DPM.","doi":"10.24963/ijcai.2023/441","url":"https://ijcai.org/proceedings/2023/441","pdf":"https://ijcai.org/proceedings/2023/0441.pdf","markdown":"https://ijcai.org/proceedings/2023/441.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/441","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Machine Learning","track":"Main Track","number":441,"pages":{"first":3966,"last":3974},"keywords":["Machine Learning: ML: Relational learning","Machine Learning: ML: Representation learning","Machine Learning: ML: Reinforcement learning"],"videos":[],"references":[{"position":1,"raw":"[Almahdi and Yang, 2017] Saud Almahdi and Steve Y Yang. An adaptive portfolio trading system: A risk-return port- folio optimization using recurrent reinforcement learning with expected maximum drawdown. Expert Systems with Applications, 87:267-279, 2017.","title":"An adaptive portfolio trading system: A risk-return portfolio optimization using recurrent reinforcement learning with expected maximum drawdown","authors":"Yang ; Almahdi, Saud Almahdi, Steve Y Yang","year":"2017","venue":"Expert Systems with Applications","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Buehler et al., 2019] Hans Buehler, Lukas Gonon, Josef Te- ichmann, and Ben Wood. Deep hedging. Quantitative Fi- nance, 19(8):1271-1291, 2019.","title":"Deep hedging","authors":"Buehler","year":"2019","venue":"Quantitative Finance","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2020] Ting Chen, Simon Kornblith, Moham- mad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In In- ternational conference on machine learning, pages 1597- 1607, 2020.","title":"A simple framework for contrastive learning of visual representations","authors":"Ting Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey Hinton","year":"2020","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"Thomas M Cover. Universal portfolios. In The Kelly Capital Growth Investment Criterion: Theory and Practice, pages 181-209, 2011.","title":"Universal portfolios","authors":"M Thomas, Cover","year":"2011","venue":"The Kelly Capital Growth Investment Criterion: Theory and Practice","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Crawshaw, 2020] Michael Crawshaw. Multi-task learning with deep neural networks: A survey. arXiv preprint arXiv:2009.09796, 2020.","title":"Multi-task learning with deep neural networks: A survey","authors":"Michael Crawshaw","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dempster and Leemans, 2006] Michael AH Dempster and Vasco Leemans. An automated fx trading system using adaptive reinforcement learning. Expert Systems with Ap- plications, 30(3):543-552, 2006.","title":"An automated fx trading system using adaptive reinforcement learning","authors":"A H Michael, Vasco Dempster, Leemans","year":"2006","venue":"Expert Systems with Applications","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Deng et al., 2016] Yue Deng, Feng Bao, Youyong Kong, Zhiquan Ren, and Qionghai Dai. Deep direct reinforce- ment learning for financial signal representation and trad- ing. IEEE transactions on neural networks and learning systems, 28(3):653-664, 2016.","title":"Deep direct reinforcement learning for financial signal representation and trading","authors":"Yue Deng, Feng Bao, Youyong Kong, Zhiquan Ren, Qionghai Dai","year":"2016","venue":"IEEE transactions on neural networks and learning systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gao and Zhang, 2013] Li Gao and Weiguo Zhang. Weighted moving average passive aggressive algo- rithm for online portfolio selection. In International Conference on Intelligent Human-Machine Systems and Cybernetics, volume 1, pages 327-330, 2013.","title":"Weighted moving average passive aggressive algorithm for online portfolio selection","authors":"Zhang ; ] Gao, Li Gao, Weiguo Zhang","year":"2013","venue":"International Conference on Intelligent Human-Machine Systems and Cybernetics","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Guo et al., 2018] Yifeng Guo, Xingyu Fu, Yuyan Shi, and Mingwen Liu. Robust log-optimal strategy with reinforce- ment learning. arXiv preprint arXiv:1805.00205, 2018.","title":"Robust log-optimal strategy with reinforcement learning","authors":"Yifeng Guo, Xingyu Fu, Yuyan Shi, Mingwen Liu","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Györfi et al., 2006] László Györfi, Gábor Lugosi, and Fred- eric Udina. Nonparametric kernel-based sequential in- vestment strategies. Mathematical Finance: An Interna- tional Journal of Mathematics, Statistics and Financial Economics, 16(2):337-357, 2006.","title":"Nonparametric kernel-based sequential investment strategies","authors":"Györfi","year":"2006","venue":"Mathematical Finance: An International Journal of Mathematics, Statistics and Financial Economics","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Huang, 2018] Chien Yi Huang. Financial trading as a game: A deep reinforcement learning approach. arXiv preprint arXiv:1807.02787, 2018.","title":"Financial trading as a game: A deep reinforcement learning approach","authors":"Yi Chien, Huang","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Jiang et al., 2017] Zhengyao Jiang, Dixing Xu, and Jinjun Liang. A deep reinforcement learning framework for the financial portfolio management problem. arXiv preprint arXiv:1706.10059, 2017.","title":"A deep reinforcement learning framework for the financial portfolio management problem","authors":"Jiang","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kipf et al., 2018] Thomas Kipf, Ethan Fetaya, Kuan-Chieh Wang, Max Welling, and Richard Zemel. Neural relational inference for interacting systems. In International Confer- ence on Machine Learning, pages 2688-2697, 2018.","title":"Neural relational inference for interacting systems","authors":"Kipf","year":"2018","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li and Hoi, 2012] Bin Li and Steven CH Hoi. On-line port- folio selection with moving average reversion. In Interna- tional Conference on Machine Learning, 2012.","title":"On-line portfolio selection with moving average reversion","authors":"Hoi ; ] Bin Li, Steven Ch Li, Hoi","year":"2012","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liang et al., 2018] Zhipeng Liang, Hao Chen, Junhao Zhu, Kangkang Jiang, and Yanran Li. Adversarial deep re- inforcement learning in portfolio management. arXiv preprint arXiv:1808.09940, 2018.","title":"Adversarial deep reinforcement learning in portfolio management","authors":"Zhipeng Liang, Hao Chen, Junhao Zhu, Kangkang Jiang, Yanran Li","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Loshchilov and Hutter, 2017] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.","title":"Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization","authors":"Hutter Loshchilov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lu, 2017] David W Lu. Agent inspired trading using re- current reinforcement learning and lstm neural networks. arXiv preprint arXiv:1707.07338, 2017.","title":"Agent inspired trading using recurrent reinforcement learning and lstm neural networks","authors":"Lu David","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Maddison et al., 2017] Chris J Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continu- ous relaxation of discrete random variables. International Conference on Learning Representations (ICLR), 2017.","title":"The concrete distribution: A continuous relaxation of discrete random variables","authors":"Andriy Chris J Maddison, Yee Whye Mnih, Teh","year":"2017","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Moody and Saffell, 2001] John Moody and Matthew Saf- fell. Learning to trade via direct reinforcement. IEEE transactions on neural Networks, 12(4):875-889, 2001.","title":"John Moody and Matthew Saffell. Learning to trade via direct reinforcement","authors":"Saffell Moody","year":"2001","venue":"IEEE transactions on neural Networks","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Neyshabur et al., 2018] Behnam Neyshabur, Zhiyuan Li, Srinadh Bhojanapalli, Yann LeCun, and Nathan Srebro. The role of over-parametrization in generalization of neu- ral networks. In International Conference on Learning Representations, 2018.","title":"The role of over-parametrization in generalization of neural networks","authors":"Zhiyuan Behnam Neyshabur, Srinadh Li, Yann Bhojanapalli, Nathan Lecun, Srebro","year":"2018","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Ruder, 2017] Sebastian Ruder. An overview of multi- task learning in deep neural networks. arXiv preprint arXiv:1706.05098, 2017.","title":"An overview of multitask learning in deep neural networks","authors":"Sebastian Ruder","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sutton and Barto, 2018] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.","title":"Reinforcement learning: An introduction","authors":"S Richard, Andrew G Sutton, Barto","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Sutton et al., 2000] Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approxi- mation. In Advances in neural information processing sys- tems, pages 1057-1063, 2000.","title":"Policy gradient methods for reinforcement learning with function approximation","authors":"David A Richard S Sutton, Mcallester, P Satinder, Yishay Singh, Mansour","year":"2000","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang et al., 2019] Jingyuan Wang, Yang Zhang, Ke Tang, Junjie Wu, and Zhang Xiong. Alphastock: A buying- winners-and-selling-losers investment strategy using inter- pretable deep reinforcement attention networks. In ACM SIGKDD International Conference on Knowledge Discov- ery & Data Mining, pages 1900-1908, 2019.","title":"Alphastock: A buyingwinners-and-selling-losers investment strategy using interpretable deep reinforcement attention networks","authors":"Jingyuan Wang, Yang Zhang, Ke Tang, Junjie Wu, Zhang Xiong","year":"2019","venue":"ACM SIGKDD International Conference on Knowledge Discovery & Data Mining","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wei et al., 2019] Haoran Wei, Yuanbo Wang, Lidia Mangu, and Keith Decker. Model-based reinforcement learning for predictions and control for limit order books. arXiv preprint arXiv:1910.03743, 2019.","title":"Model-based reinforcement learning for predictions and control for limit order books","authors":"Haoran Wei, Yuanbo Wang, Lidia Mangu, Keith Decker","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yang et al., 2020] Zitong Yang, Yaodong Yu, Chong You, Jacob Steinhardt, and Yi Ma. Rethinking bias-variance trade-off for generalization of neural networks. In Inter- national Conference on Machine Learning, pages 10767- 10777. PMLR, 2020.","title":"Rethinking bias-variance trade-off for generalization of neural networks","authors":"Yang","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Ye et al., 2020] Yunan Ye, Hengzhi Pei, Boxin Wang, Pin- Yu Chen, Yada Zhu, Ju Xiao, and Bo Li. Reinforcement- learning based portfolio management with augmented as- set movement prediction states. In AAAI Conference on Artificial Intelligence, volume 34, pages 1112-1119, 2020.","title":"Reinforcementlearning based portfolio management with augmented asset movement prediction states","authors":"Yunan Ye, Hengzhi Pei, Boxin Wang, Pin-Yu Chen, Yada Zhu, Ju Xiao, Bo Li","year":"2020","venue":"AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Yue et al., 2021] Zhihan Yue, Yujing Wang, Juanyong Duan, Tianmeng Yang, Congrui Huang, Yunhai Tong, and Bixiong Xu. Ts2vec: Towards universal representation of time series. arXiv preprint arXiv:2106.10466, 2021.","title":"Ts2vec: Towards universal representation of time series","authors":"Yue","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}