{"type":"ScholarlyArticle","title":"MAPS: Multi-Agent reinforcement learning-based Portfolio management System.","authors":[{"name":"Jinho Lee","slug":"jinho-lee","given":"Jinho","family":"Lee","affiliation":"Department of Computer Science and Engineering , Korea University"},{"name":"Raehyun Kim","slug":"raehyun-kim","given":"Raehyun","family":"Kim","affiliation":"Department of Computer Science and Engineering , Korea University"},{"name":"Seok-Won Yi","slug":"seok-won-yi","given":"Seok-Won","family":"Yi","affiliation":"Department of Computer Science and Engineering , Korea University"},{"name":"Jaewoo Kang","slug":"jaewoo-kang","given":"Jaewoo","family":"Kang","affiliation":"Department of Computer Science and Engineering , Korea University"}],"abstract":"Generating an investment strategy using advanced deep learning methods in stock markets has recently been a topic of interest. Most existing deep learning methods focus on proposing an optimal model or network architecture by maximizing return. However, these models often fail to consider and adapt to the continuously changing market conditions. In this paper, we propose the Multi-Agent reinforcement learning-based Portfolio management System (MAPS). MAPS is a cooperative system in which each agent is an independent \"investor\" creating its own portfolio. In the training procedure, each agent is guided to act as diversely as possible while maximizing its own return with a carefully designed loss function. As a result, MAPS as a system ends up with a diversified portfolio. Experiment results with 12 years of US market data show that MAPS outperforms most of the baselines in terms of Sharpe ratio. Furthermore, our results show that adding more agents to our system would allow us to get a higher Sharpe ratio by lowering risk with a more diversified portfolio.","doi":"10.24963/ijcai.2020/623","url":"https://ijcai.org/proceedings/2020/623","pdf":"https://ijcai.org/proceedings/2020/0623.pdf","markdown":"https://ijcai.org/proceedings/2020/623.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/623","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Special Track on AI in FinTech","track":"Special Track on AI in FinTech","number":623,"pages":{"first":4520,"last":4526},"keywords":["AI for trading: AI for portfolio analytics","AI for trading: AI for predictive trading","AI for trading: AI for novel financial models"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24613/file_24613.m3u8","length_sec":286,"provider":"hls"}],"references":[{"position":1,"raw":"[Abarbanell and Bushee, 1997] Jeffrey S Abarbanell and Brian J Bushee. Fundamental analysis, future earn- ings, and stock prices. Journal of Accounting Research, 35(1):1-24, 1997.","title":"Fundamental analysis, future earnings, and stock prices","authors":"S Jeffrey, Brian J Abarbanell, Bushee","year":"1997","venue":"Journal of Accounting Research","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bacoyannis et al., 2018] Vangelis Bacoyannis, Vacslav Glukhov, Tom Jin, Jonathan Kochems, and Doo Re Song. Idiosyncrasies and challenges of data driven learning in electronic trading. arXiv preprint arXiv:1811.09549, 2018.","title":"Idiosyncrasies and challenges of data driven learning in electronic trading","authors":"Vangelis Bacoyannis, Vacslav Glukhov, Tom Jin, Jonathan Kochems, Doo Re Song","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Deng et al., 2016] Yue Deng, Feng Bao, Youyong Kong, Zhiquan Ren, and Qionghai Dai. Deep direct reinforce- ment learning for financial signal representation and trad- ing. IEEE transactions on neural networks and learning systems, 28(3):653-664, 2016.","title":"Deep direct reinforcement learning for financial signal representation and trading","authors":"Yue Deng, Feng Bao, Youyong Kong, Zhiquan Ren, Qionghai Dai","year":"2016","venue":"IEEE transactions on neural networks and learning systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Di Persio and Honchar, 2016] Luca Di Persio and Olek- sandr Honchar. Artificial neural networks architectures for stock price prediction: Comparisons and applications. In- ternational journal of circuits, systems and signal process- ing, 10(2016):403-413, 2016.","title":"Artificial neural networks architectures for stock price prediction: Comparisons and applications","authors":"Di Persio, ; Honchar, Di Luca, Oleksandr Persio, Honchar","year":"2016","venue":"International journal of circuits, systems and signal processing","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Ding et al., 2015] Xiao Ding, Yue Zhang, Ting Liu, and Junwen Duan. Deep learning for event-driven stock pre- diction. In Twenty-fourth international joint conference on artificial intelligence, 2015.","title":"Deep learning for event-driven stock prediction","authors":"Xiao Ding, Yue Zhang, Ting Liu, Junwen Duan","year":"2015","venue":"Twenty-fourth international joint conference on artificial intelligence","doi":null,"url":null,"internal_url":"/proceedings/2015/329"},{"position":6,"raw":"[Feng et al., 2019] Fuli Feng, Huimin Chen, Xiangnan He, Ji Ding, Maosong Sun, and Tat-Seng Chua. Enhancing stock movement prediction with adversarial training. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, pages 5843-5849. AAAI Press, 2019.","title":"Enhancing stock movement prediction with adversarial training","authors":"Fuli Feng, Huimin Chen, Xiangnan He, Ji Ding, Maosong Sun, Tat-Seng Chua","year":"2019","venue":"Proceedings of the 28th International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2019/810"},{"position":7,"raw":"[Ioffe and Szegedy, 2015] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep net- work training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.","title":"Batch normalization: Accelerating deep network training by reducing internal covariate shift","authors":"Szegedy Ioffe, Sergey Ioffe, Christian Szegedy","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Jegadeesh and Titman, 1993] Narasimhan Jegadeesh and Sheridan Titman. Returns to buying winners and selling losers: Implications for stock market efficiency. The Jour- nal of finance, 48(1):65-91, 1993.","title":"Returns to buying winners and selling losers: Implications for stock market efficiency","authors":"Narasimhan Jegadeesh, Sheridan Titman","year":"1993","venue":"The Journal of finance","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kaya et al., 2016] Orc ¸un Kaya, Jan Schildbach, and Deutsche Bank Ag. High-frequency trading. Reaching the limits, Automated trader magazine, 41:23-27, 2016.","title":"Orc ¸un Kaya, Jan Schildbach, and Deutsche Bank Ag. High-frequency trading. Reaching the limits","authors":"Kaya","year":"2016","venue":"Automated trader magazine","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kim et al., 2019] Raehyun Kim, Chan Ho So, Minbyul Jeong, Sanghoon Lee, Jinkyu Kim, and Jaewoo Kang. Hats: A hierarchical graph attention network for stock movement prediction. arXiv preprint arXiv:1908.07999, 2019.","title":"Hats: A hierarchical graph attention network for stock movement prediction","authors":"Raehyun Kim, Chan Ho So, Minbyul Jeong, Sanghoon Lee, Jinkyu Kim, Jaewoo Kang","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kingma and Ba, 2014] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lo et al., 2000] Andrew W Lo, Harry Mamaysky, and Jiang Wang. Foundations of technical analysis: Computational algorithms, statistical inference, and empirical implemen- tation. The journal of Finance, 55(4):1705-1765, 2000.","title":"Foundations of technical analysis: Computational algorithms, statistical inference, and empirical implementation","authors":"Andrew W Lo, Harry Mamaysky, Jiang Wang","year":"2000","venue":"The journal of Finance","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Markowitz, 1952] Harry Markowitz. Portfolio selection. The journal of finance, 7(1):77-91, 1952.","title":"Portfolio selection","authors":"Markowitz","year":"1952","venue":"The journal of finance","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidje- land, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Mnih","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Park and Irwin, 2007] Cheolho Park and Scott H Irwin. What do we know about the profitability of technical anal- ysis? Journal of Economic Surveys, 21(4):786-826, 2007.","title":"Cheolho Park and Scott H Irwin. What do we know about the profitability of technical analysis?","authors":"Irwin Park","year":"2007","venue":"Journal of Economic Surveys","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Poterba and Summers, 1988] James M Poterba and Lawrence H Summers. Mean reversion in stock prices: Evidence and implications. Journal of financial eco- nomics, 22(1):27-59, 1988.","title":"Mean reversion in stock prices: Evidence and implications","authors":"M James, Lawrence H Poterba, Summers","year":"1988","venue":"Journal of financial economics","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Qin et al., 2017] Yao Qin, Dongjin Song, Haifeng Chen, Wei Cheng, Guofei Jiang, and Garrison Cottrell. A dual- stage attention-based recurrent neural network for time se- ries prediction. arXiv preprint arXiv:1704.02971, 2017.","title":"A dualstage attention-based recurrent neural network for time series prediction","authors":"Dongjin Yao Qin, Haifeng Song, Wei Chen, Guofei Cheng, Garrison Jiang, Cottrell","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Sutton and Barto, 2018] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. 2018.","title":"Reinforcement learning: An introduction","authors":"S Richard, Andrew G Sutton, Barto","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Xiong et al., 2018] Zhuoran Xiong, Xiao-Yang Liu, Shan Zhong, Hongyang Yang, and Anwar Walid. Practical deep reinforcement learning approach for stock trading. arXiv preprint arXiv:1811.07522, 2018.","title":"Practical deep reinforcement learning approach for stock trading","authors":"Xiong","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Xu and Cohen, 2018] Yumo Xu and Shay B Cohen. Stock movement prediction from tweets and historical prices. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1970-1979, 2018.","title":"Stock movement prediction from tweets and historical prices","authors":"Cohen ; ] Yumo Xu, Shay B Xu, Cohen","year":"2018","venue":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}