{"type":"ScholarlyArticle","title":"Mean-Semivariance Policy Optimization via Risk-Averse Reinforcement Learning (Extended Abstract)","authors":[{"name":"Xiaoteng Ma","slug":"xiaoteng-ma","given":"Xiaoteng","family":"Ma","affiliation":"Department of Automation , Tsinghua University"},{"name":"Shuai Ma","slug":"shuai-ma","given":"Shuai","family":"Ma","affiliation":"School of Business , Sun Yat-sen University"},{"name":"Li Xia","slug":"li-xia","given":"Li","family":"Xia","affiliation":"School of Business , Sun Yat-sen University"},{"name":"Qianchuan Zhao","slug":"qianchuan-zhao","given":"Qianchuan","family":"Zhao","affiliation":"Department of Automation , Tsinghua University"}],"abstract":"Keeping risk under control is often more crucial than maximizing expected rewards in real-world decision-making situations, such as finance, robotics, autonomous driving, etc. The most natural choice of risk measures is variance, while it penalizes the upside volatility as much as the downside part. Instead, the (downside) semivariance, which captures negative deviation of a random variable under its mean, is more suitable for risk-averse proposes. This paper aims at optimizing the mean-semivariance (MSV) criterion in reinforcement learning w.r.t. steady reward distribution. Since semivariance is time-inconsistent and does not satisfy the standard Bellman equation, the traditional dynamic programming methods are inapplicable to MSV problems directly. To tackle this challenge, we resort to Perturbation Analysis (PA) theory and establish the performance difference formula for MSV. We reveal that the MSV problem can be solved by iteratively solving a sequence of RL problems with a policy-dependent reward function. Further, we propose two on-policy algorithms based on the policy gradient theory and the trust region method. Finally, we conduct diverse experiments from simple bandit problems to continuous control tasks in MuJoCo, which demonstrate the effectiveness of our proposed methods.","doi":"10.24963/ijcai.2023/784","url":"https://ijcai.org/proceedings/2023/784","pdf":"https://ijcai.org/proceedings/2023/0784.pdf","markdown":"https://ijcai.org/proceedings/2023/784.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/784","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Journal Track","track":"Journal Track","number":784,"pages":{"first":6925,"last":6930},"keywords":["Uncertainty in AI: UAI: Sequential decision making","Machine Learning: ML: Reinforcement learning","Planning and Scheduling: PS: Markov decisions processes","Machine Learning: ML: Deep reinforcement learning"],"videos":[],"references":[{"position":1,"raw":"[Achiam et al., 2017] Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimiza- tion. In International Conference on Machine Learning, volume 70, pages 22-31, 2017.","title":"Constrained policy optimization","authors":"Achiam","year":"2017","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Berner et al., 2019] Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemysław Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforce- ment learning. ArXiv preprint, abs/1912.06680, 2019.","title":"Dota 2 with large scale deep reinforcement learning","authors":"Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemysław Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bisi et al., 2020] Lorenzo Bisi, Luca Sabbioni, Edoardo Vit- tori, Matteo Papini, and Marcello Restelli. Risk-averse trust region optimization for reward-volatility reduction. In International Joint Conference on Artificial Intelligence, pages 4583-4589, 2020.","title":"Risk-averse trust region optimization for reward-volatility reduction","authors":"Lorenzo Bisi, Luca Sabbioni, Edoardo Vittori, Matteo Papini, Marcello Restelli","year":"2020","venue":"International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2020/632"},{"position":4,"raw":"[Bollerslev et al., 2020] Tim Bollerslev, Sophia Zhengzi Li, and Bingzhi Zhao. Good volatility, bad volatility, and the cross section of stock returns. Journal of Financial and Quantitative Analysis, 55(3):751-781, 2020.","title":"Good volatility, bad volatility, and the cross section of stock returns","authors":"Tim Bollerslev, Sophia Zhengzi Li, Bingzhi Zhao","year":"2020","venue":"Journal of Financial and Quantitative Analysis","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cao, 2007] Xi-Ren Cao. Stochastic Learning and Optimiza- tion: A Sensitivity-Based Approach. Springer, 2007.","title":"Stochastic Learning and Optimization: A Sensitivity-Based Approach","authors":"Xi-Ren Cao","year":"2007","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen et al., 2019] Wei Chen, Dandan Li, Shan Lu, and Weiyi Liu. Multi-period mean-semivariance portfolio op- timization based on uncertain measure. Soft Computing, 23(15):6231-6247, 2019.","title":"Multi-period mean-semivariance portfolio optimization based on uncertain measure","authors":"Wei Chen, Dandan Li, Shan Lu, Weiyi Liu","year":"2019","venue":"Soft Computing","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dulac-Arnold et al., 2019] Gabriel Dulac-Arnold, Daniel Mankowitz, and Todd Hester. Challenges of real-world reinforcement learning. ArXiv preprint, abs/1904.12901, 2019.","title":"Challenges of real-world reinforcement learning","authors":"Gabriel Dulac-Arnold, Daniel Mankowitz, Todd Hester","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Estrada, 2007] Javier Estrada. Mean-semivariance behav- ior: Downside risk and capital asset pricing. International Review of Economics & Finance, 16(2):169-185, 2007.","title":"Mean-semivariance behavior: Downside risk and capital asset pricing","authors":"Javier Estrada","year":"2007","venue":"International Review of Economics & Finance","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Filar et al., 1989] Jerzy A Filar, Lodewijk CM Kallenberg, and Huey-Miin Lee. Variance-penalized Markov decision processes. Mathematics of Operations Research, 14(1):147- 161, 1989.","title":"Variance-penalized Markov decision processes","authors":"Jerzy A Filar, C M Lodewijk, Huey-Miin Kallenberg, Lee","year":"1989","venue":"Mathematics of Operations Research","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Garcıa and Fernández, 2015] Javier Garcıa and Fernando Fernández. A comprehensive survey on safe reinforce- ment learning. Journal of Machine Learning Research, 16(1):1437-1480, 2015.","title":"Javier Garcıa and Fernando Fernández. A comprehensive survey on safe reinforcement learning","authors":"Fernández Garcıa","year":"2015","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Janner et al., 2019] Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. In Advances in Neu- ral Information Processing Systems, pages 12498-12509, 2019.","title":"When to trust your model: Model-based policy optimization","authors":"Michael Janner, Justin Fu, Marvin Zhang, Sergey Levine","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li and Ng, 2000] Duan Li and Wan-Lung Ng. Optimal dy- namic portfolio selection: Multiperiod mean-variance for- mulation. Mathematical Finance, 10(3):387-406, 2000.","title":"Optimal dynamic portfolio selection: Multiperiod mean-variance formulation","authors":"Duan Li, Wan-Lung Ng","year":"2000","venue":"Mathematical Finance","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Liu and Zhang, 2015] Yong-Jun Liu and Wei-Guo Zhang. A multi-period fuzzy portfolio optimization model with min- imum transaction lots. European Journal of Operational Research, 242(3):933-941, 2015.","title":"A multi-period fuzzy portfolio optimization model with minimum transaction lots","authors":"Zhang ; Yong-Jun Liu, Wei-Guo Liu, Zhang","year":"2015","venue":"European Journal of Operational Research","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Ma et al., 2021] Xiaoteng Ma, Xiaohang Tang, Li Xia, Jun Yang, and Qianchuan Zhao. Average-reward reinforcement learning with trust region methods. In International Joint Conference on Artificial Intelligence, pages 2797-2803, 2021.","title":"Average-reward reinforcement learning with trust region methods","authors":"Ma","year":"2021","venue":"International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2021/385"},{"position":15,"raw":"[Ma et al., 2022a] Shuai Ma, Xiaoteng Ma, and Li Xia. An optimistic value iteration for mean-variance optimization in discounted markov decision processes. Results in Control and Optimization, 8:100165, 2022.","title":"An optimistic value iteration for mean-variance optimization in discounted markov decision processes","authors":"Ma","year":"2022","venue":"Results in Control and Optimization","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Ma et al., 2022b] Shuai Ma, Xiaoteng Ma, and Li Xia. A uni- fied algorithm framework for mean-variance optimization in discounted Markov decision processes. ArXiv preprint, abs/2201.05737, 2022.","title":"A unified algorithm framework for mean-variance optimization in discounted Markov decision processes","authors":"Ma","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Ma et al., 2022c] Xiaoteng Ma, Shuai Ma, Li Xia, and Qianchuan Zhao. Mean-semivariance policy optimization via risk-averse reinforcement learning. Journal of Artificial Intelligence Research, 75:569-595, 2022.","title":"Mean-semivariance policy optimization via risk-averse reinforcement learning","authors":"Ma","year":"2022","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Markowitz, 1952] Harry M Markowitz. Portfolio selection. Journal of Finance, 7:77-91, 1952.","title":"Portfolio selection","authors":"M Harry, Markowitz","year":"1952","venue":"Journal of Finance","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Markowitz, 1959] Harry M Markowitz. Portfolio Selection: Efficient Diversification of Investments. John Wiley & Sons, New York, 1959.","title":"Portfolio Selection: Efficient Diversification of Investments","authors":"M Harry, Markowitz","year":"1959","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Nagabandi et al., 2020] Anusha Nagabandi, Kurt Konolige, Sergey Levine, and Vikash Kumar. Deep dynamics models for learning dexterous manipulation. In Conference on Robot Learning, pages 1101-1112, 2020.","title":"Deep dynamics models for learning dexterous manipulation","authors":"Anusha Nagabandi, Kurt Konolige, Sergey Levine, Vikash Kumar","year":"2020","venue":"Conference on Robot Learning","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Schulman et al., 2015] John Schulman, Sergey Levine, Pieter Abbeel, Michael I. Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference on Machine Learning, volume 37, pages 1889-1897, 2015.","title":"Trust region policy optimization","authors":"John Schulman, Sergey Levine, Pieter Abbeel, Michael I Jordan, Philipp Moritz","year":"2015","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Silver et al., 2017] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 550(7676):354-359, 2017.","title":"Mastering the game of go without human knowledge","authors":"David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton","year":"2017","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Sobel, 1982] Matthew J Sobel. The variance of discounted Markov decision processes. Journal of Applied Probability, 19(4):794-802, 1982.","title":"The variance of discounted Markov decision processes","authors":"J Matthew, Sobel","year":"1982","venue":"Journal of Applied Probability","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Vinyals et al., 2019] Oriol Vinyals, Igor Babuschkin, Woj- ciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in star- craft ii using multi-agent reinforcement learning. Nature, 575(7782):350-354, 2019.","title":"Grandmaster level in starcraft ii using multi-agent reinforcement learning","authors":"Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev","year":"2019","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wei, 2019] Qingda Wei. Mean-semivariance optimality for continuous-time Markov decision processes. Systems & Control Letters, 125:67-74, 2019.","title":"Mean-semivariance optimality for continuous-time Markov decision processes","authors":"Qingda Wei, Wei","year":"2019","venue":"Systems & Control Letters","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Xia, 2020] Li Xia. Risk-sensitive Markov decision processes with combined metrics of mean and variance. Production and Operations Management, 29(12):2808-2827, 2020.","title":"Risk-sensitive Markov decision processes with combined metrics of mean and variance","authors":"Li Xia","year":"2020","venue":"Production and Operations Management","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Xie et al., 2018] Tengyang Xie, Bo Liu, Yangyang Xu, Mo- hammad Ghavamzadeh, Yinlam Chow, Daoming Lyu, and Daesub Yoon. A block coordinate ascent algorithm for mean-variance optimization. In Advances in Neural Infor- mation Processing Systems, volume 31, pages 1073-1083, 2018.","title":"A block coordinate ascent algorithm for mean-variance optimization","authors":"Tengyang Xie, Bo Liu, Yangyang Xu, Mohammad Ghavamzadeh, Yinlam Chow, Daoming Lyu, Daesub Yoon","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Yan et al., 2007] Wei Yan, Rong Miao, and Shurong Li. Multi-period semi-variance portfolio selection: Model and numerical solution. Applied Mathematics and Computation, 194(1):128-134, 2007.","title":"Multi-period semi-variance portfolio selection: Model and numerical solution","authors":"Wei Yan, Rong Miao, Shurong Li","year":"2007","venue":"Applied Mathematics and Computation","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhang et al., 2012] Wei-Guo Zhang, Yong-Jun Liu, and Wei- Jun Xu. A possibilistic mean-semivariance-entropy model for multi-period portfolio selection with transaction costs. European Journal of Operational Research, 222(2):341- 349, 2012.","title":"A possibilistic mean-semivariance-entropy model for multi-period portfolio selection with transaction costs","authors":"Wei-Guo Zhang, Yong-Jun Liu, Wei-Jun Xu","year":"2012","venue":"European Journal of Operational Research","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhang et al., 2021] Shangtong Zhang, Bo Liu, and Shimon Whiteson. Mean-variance policy iteration for risk-averse reinforcement learning. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 35, pages 10905- 10913, 2021.","title":"Mean-variance policy iteration for risk-averse reinforcement learning","authors":"Zhang","year":"2021","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}