{"type":"ScholarlyArticle","title":"Risk-Averse Trust Region Optimization for Reward-Volatility Reduction","authors":[{"name":"Lorenzo Bisi","slug":"lorenzo-bisi","given":"Lorenzo","family":"Bisi","affiliation":"Politecnico di Milano; ISI Foundation"},{"name":"Luca Sabbioni","slug":"luca-sabbioni","given":"Luca","family":"Sabbioni","affiliation":"Politecnico di Milano; ISI Foundation"},{"name":"Edoardo Vittori","slug":"edoardo-vittori","given":"Edoardo","family":"Vittori","affiliation":"Politecnico di Milano; Banca IMI"},{"name":"Matteo Papini","slug":"matteo-papini","given":"Matteo","family":"Papini","affiliation":"Politecnico di Milano"},{"name":"Marcello Restelli","slug":"marcello-restelli","given":"Marcello","family":"Restelli","affiliation":"Politecnico di Milano"}],"abstract":"The use of reinforcement learning in algorithmic trading is of growing interest, since it offers the opportunity of making profit through the development of autonomous artificial traders, that do not depend on hard-coded rules. In such a framework, keeping uncertainty under control is as important as maximizing expected returns. Risk aversion has been addressed in reinforcement learning through measures related to the distribution of returns. However, in trading it is essential to keep under control the risk of portfolio positions in the intermediate steps. In this paper, we define a novel measure of risk, which we call reward volatility, consisting of the variance of the rewards under the state-occupancy measure. This new risk measure is shown to bound the return variance so that reducing the former also constrains the latter. We derive a policy gradient theorem with a new objective function that exploits the mean-volatility relationship. Furthermore, we adapt TRPO, the well-known policy gradient algorithm with monotonic improvement guarantees, in a risk-averse manner. Finally, we test the proposed approach in two financial environments using real market data.","doi":"10.24963/ijcai.2020/632","url":"https://ijcai.org/proceedings/2020/632","pdf":"https://ijcai.org/proceedings/2020/0632.pdf","markdown":"https://ijcai.org/proceedings/2020/632.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/632","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Special Track on AI in FinTech","track":"Special Track on AI in FinTech","number":632,"pages":{"first":4583,"last":4589},"keywords":["Foundation for AI in FinTech: Reinforcement learning for FinTech","AI for trading: AI for algorithmic trading"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25415/file_25415.m3u8","length_sec":275,"provider":"hls"}],"references":[{"position":1,"raw":"[Chow et al., 2017] Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. Risk-constrained reinforcement learning with percentile risk criteria. JMLR, 18(1):6070-6120, 2017.","title":"Risk-constrained reinforcement learning with percentile risk criteria","authors":"Chow","year":"2017","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Deisenroth et al., 2013] Marc Peter Deisenroth, Gerhard Neumann, Jan Peters, et al. A survey on policy search for robotics. Foundations and Trends R in Robotics, 2(1- 2):1-142, 2013.","title":"A survey on policy search for robotics","authors":"Marc Peter Deisenroth, Gerhard Neumann, Jan Peters","year":"2013","venue":"Foundations and Trends R in Robotics","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Di Castro et al., 2012] Dotan Di Castro, Aviv Tamar, and Shie Mannor. Policy gradients with variance related risk criteria. ICML, 1, 06 2012.","title":"Policy gradients with variance related risk criteria","authors":"Di Castro","year":"2012","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[García and Fernández, 2015] Javier García and Fernando Fernández. A comprehensive survey on safe reinforcement learning. JMLR, 16:1437-1480, 2015.","title":"Javier García and Fernando Fernández. A comprehensive survey on safe reinforcement learning","authors":"Fernández García","year":"2015","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Heess et al., 2017] Nicolas Heess, Dhruva TB, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, S. M. Ali Eslami, Martin A. Riedmiller, and David Silver. Emergence of locomotion behaviours in rich environments. CoRR, abs/1707.02286, 2017.","title":"Emergence of locomotion behaviours in rich environments","authors":"Nicolas Heess, T B Dhruva, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, S M Ali Eslami, Martin A Riedmiller, David Silver","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Kakade and Langford, 2002] Sham Kakade and John Lang- ford. Approximately optimal approximate reinforcement learning. In ICML, volume 2, pages 267-274, 2002.","title":"Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning","authors":"Langford Kakade","year":"2002","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Moody and Saffell, 2001] John Moody and Matthew Saf- fell. Learning to trade via direct reinforcement. IEEE transactions on neural Networks, 12(4):875-889, 2001.","title":"John Moody and Matthew Saffell. Learning to trade via direct reinforcement","authors":"Saffell Moody","year":"2001","venue":"IEEE transactions on neural Networks","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Morimura et al., 2010] Tetsuro Morimura, Masashi Sugiyama, Hisashi Kashima, Hirotaka Hachiya, and Toshiyuki Tanaka. Nonparametric return distribution approximation for reinforcement learning. In ICML, 2010. [","title":"Nonparametric return distribution approximation for reinforcement learning","authors":"Tetsuro Morimura, Masashi Sugiyama, Hisashi Kashima, Hirotaka Hachiya, Toshiyuki Tanaka","year":"2010","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"OpenAI, 2018 accessed May 2020] OpenAI. Openai five. https://blog.openai.com/openai-five/, 2018 -accessed May 2020.","title":"OpenAI. Openai five","authors":"Openai","year":"2018","venue":null,"doi":null,"url":"https://blog.openai.com/openai-five/,2018-","internal_url":null},{"position":10,"raw":"[Papini et al., 2017] Matteo Papini, Matteo Pirotta, and Mar- cello Restelli. Adaptive batch size for safe policy gradi- ents. In NeurIPS, pages 3591-3600, 2017.","title":"Adaptive batch size for safe policy gradients","authors":"Matteo Papini, Matteo Pirotta, Marcello Restelli","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Papini et al., 2019] Matteo Papini, Matteo Pirotta, and Mar- cello Restelli. Smoothing policies and safe policy gradi- ents, 2019.","title":"Smoothing policies and safe policy gradients","authors":"Matteo Papini, Matteo Pirotta, Marcello Restelli","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Peters and Schaal, 2008] Jan Peters and Stefan Schaal. Re- inforcement learning of motor skills with policy gradients. Neural networks, 21(4):682-697, 2008.","title":"Reinforcement learning of motor skills with policy gradients","authors":"Schaal Peters, Jan Peters, Stefan Schaal","year":"2008","venue":"Neural networks","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Pirotta et al., 2013a] Matteo Pirotta, Marcello Restelli, and Luca Bascetta. Adaptive step-size for policy gradient methods. In NeurIPS 26, pages 1394-1402.","title":"Adaptive step-size for policy gradient methods","authors":"Pirotta","year":"2013","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Pirotta et al., 2013b] Matteo Pirotta, Marcello Restelli, Alessio Pecorino, and Daniele Calandriello. Safe policy iteration. In ICML, pages 307-315, 2013.","title":"Safe policy iteration","authors":"Pirotta","year":"2013","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Pirotta et al., 2015] Matteo Pirotta, Marcello Restelli, and Luca Bascetta. Policy gradient in lipschitz markov de- cision processes. Machine Learning, 100(2-3):255-283, 2015.","title":"Policy gradient in lipschitz markov decision processes","authors":"Matteo Pirotta, Marcello Restelli, Luca Bascetta","year":"2015","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Prashanth and Ghavamzadeh, 2014a] L. A. Prashanth and Mohammad Ghavamzadeh. Actor-critic algorithms for risk-sensitive reinforcement learning. arXiv preprint arXiv:1403.6530, 2014.","title":"Actor-critic algorithms for risk-sensitive reinforcement learning","authors":"Ghavamzadeh Prashanth, L A Prashanth, Mohammad Ghavamzadeh","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Prashanth and Ghavamzadeh, 2014b] L. A. Prashanth and Mohammad Ghavamzadeh. Variance-constrained actor- critic algorithms for discounted and average reward mdps. CoRR, abs/1403.6530, 2014.","title":"Variance-constrained actorcritic algorithms for discounted and average reward mdps","authors":"Ghavamzadeh Prashanth, L A Prashanth, Mohammad Ghavamzadeh","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Schulman et al., 2015] John Schulman, Sergey Levine, Pieter Abbeel, Michael I. Jordan, and Philipp Moritz. Trust region policy optimization. In ICML, pages 1889-1897, 2015.","title":"Trust region policy optimization","authors":"John Schulman, Sergey Levine, Pieter Abbeel, Michael I Jordan, Philipp Moritz","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Schulman et al., 2017] John Schulman, Filip Wolski, Pra- fulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.06347, 2017.","title":"Proximal policy optimization algorithms","authors":"John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Shen et al., 2014] Yun Shen, Ruihong Huang, Chang Yan, and Klaus Obermayer. Risk-averse reinforcement learning for algorithmic trading. pages 391-398, March 2014.","title":"Risk-averse reinforcement learning for algorithmic trading","authors":"Yun Shen, Ruihong Huang, Chang Yan, Klaus Obermayer","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Sobel, 1982] Matthew J. Sobel. The variance of discounted Markov decision processes. Journal of Applied Probabil- ity, 19(4):794-802, 1982.","title":"The variance of discounted Markov decision processes","authors":"J Matthew, Sobel","year":"1982","venue":"Journal of Applied Probability","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Steinbach, 2001] Marc C Steinbach. Markowitz revis- ited: Mean-variance models in financial portfolio analysis. SIAM review, 43(1):31-85, 2001.","title":"Markowitz revisited: Mean-variance models in financial portfolio analysis","authors":"C Marc, Steinbach","year":"2001","venue":"SIAM review","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Sutton and Barto, 1998] Richard S. Sutton and Andrew G. Barto. Introduction to Reinforcement Learning. MIT Press, Cambridge, MA, USA, 1st edition, 1998.","title":"Introduction to Reinforcement Learning","authors":"Richard S Sutton, Andrew G Barto","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Sutton et al., 2000] Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approx- imation. In NeurIPS, pages 1057-1063, 2000.","title":"Policy gradient methods for reinforcement learning with function approximation","authors":"David A Richard S Sutton, Mcallester, P Satinder, Yishay Singh, Mansour","year":"2000","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Tamar and Mannor, 2013] Aviv Tamar and Shie Mannor. Variance adjusted actor critic algorithms. arXiv preprint arXiv:1310.3697, 2013.","title":"Variance adjusted actor critic algorithms","authors":"Tamar, Mannor ;, Aviv Tamar, Shie Mannor","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Tamar et al., 2015] Aviv Tamar, Yinlam Chow, Mohammad Ghavamzadeh, and Shie Mannor. Policy Gradient for Co- herent Risk Measures. CoRR, page 9, 2015.","title":"Policy Gradient for Coherent Risk Measures","authors":"Tamar","year":"2015","venue":"CoRR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Tamar et al., 2016] Aviv Tamar, Dotan Di Castro, and Shie Mannor. Learning the variance of the reward-to-go. The Journal of Machine Learning Research, 17(1):361-396, 2016.","title":"Learning the variance of the reward-to-go","authors":"Tamar","year":"2016","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Tamar et al., 2017] Aviv Tamar, Yinlam Chow, Mohammad Ghavamzadeh, and Shie Mannor. Sequential Decision Making With Coherent Risk. IEEE Transactions on Au- tomatic Control, 62(7):3323-3338, July 2017.","title":"Sequential Decision Making With Coherent Risk","authors":"Tamar","year":"2017","venue":"IEEE Transactions on Automatic Control","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}