{"type":"ScholarlyArticle","title":"Lexicographic Multi-Objective Reinforcement Learning","authors":[{"name":"Joar Skalse","slug":"joar-skalse","given":"Joar","family":"Skalse","affiliation":"Department of Computer Science , University of Oxford"},{"name":"Lewis Hammond","slug":"lewis-hammond","given":"Lewis","family":"Hammond","affiliation":"Department of Computer Science , University of Oxford"},{"name":"Charlie Griffin","slug":"charlie-griffin","given":"Charlie","family":"Griffin","affiliation":"Department of Computer Science , University of Oxford"},{"name":"Alessandro Abate","slug":"alessandro-abate","given":"Alessandro","family":"Abate","affiliation":"Department of Computer Science , University of Oxford"}],"abstract":"In this work we introduce reinforcement learning techniques for solving lexicographic multi-objective problems.  These are problems that involve multiple reward signals, and where the goal is to learn a policy that maximises the first reward signal, and subject to this constraint also maximises the second reward signal, and so on. We present a family of both action-value and policy gradient algorithms that can be used to solve such problems, and prove that they converge to policies that are lexicographically optimal. We evaluate the scalability and performance of these algorithms empirically, and demonstrate their applicability in practical settings. As a more specific application, we show how our algorithms can be used to impose safety constraints on the behaviour of an agent, and compare their performance in this context with that of other constrained reinforcement learning algorithms.","doi":"10.24963/ijcai.2022/476","url":"https://ijcai.org/proceedings/2022/476","pdf":"https://ijcai.org/proceedings/2022/0476.pdf","markdown":"https://ijcai.org/proceedings/2022/476.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/476","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Machine Learning","track":"Main Track","number":476,"pages":{"first":3430,"last":3436},"keywords":["Machine Learning: Reinforcement Learning","AI Ethics, Trust, Fairness: Safety & Robustness","Constraint Satisfaction and Optimization: Constraints and Machine Learning"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987055","length_sec":81,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985198","length_sec":712,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Achiam et al., 2017] Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimiza- tion. In Proceedings of the 34th International Conference on Machine Learning, pages 22-31, 2017.","title":"Constrained policy optimization","authors":"Achiam","year":"2017","venue":"Proceedings of the 34th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Amari, 1998] Shun-ichi Amari. Natural gradient works ef- ficiently in learning. Neural Computation, 10(2):251-276, 1998.","title":"Shun-ichi Amari. Natural gradient works efficiently in learning","authors":"Amari","year":"1998","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bertsekas, 1999] Dimitri Bertsekas. Nonlinear Program- ming. Athena Scientific, 1999.","title":"Nonlinear Programming","authors":"Dimitri Bertsekas","year":"1999","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bhatnagar et al., 2009] Shalabh Bhatnagar, Richard S. Sut- ton, Mohammad Ghavamzadeh, and Mark Lee. Natural actor-critic algorithms. Automatica, 45(11):2471-2482, 2009.","title":"Natural actor-critic algorithms","authors":"Shalabh Bhatnagar, Richard S Sutton, Mohammad Ghavamzadeh, Mark Lee","year":"2009","venue":"Automatica","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Borkar, 2008] Vivek S. Borkar. Stochastic Approximation. Hindustan Book Agency, 2008.","title":"Stochastic Approximation","authors":"S Vivek, Borkar","year":"2008","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chow et al., 2017] Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. Risk-constrained reinforcement learning with percentile risk criteria. Journal of Machine Learning Research, 18(1):6070-6120, 2017.","title":"Risk-constrained reinforcement learning with percentile risk criteria","authors":"Chow","year":"2017","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chow et al., 2018] Yinlam Chow, Ofir Nachum, Edgar Duenez-Guzman, and Mohammad Ghavamzadeh. A lyapunov-based approach to safe reinforcement learn- ing. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 8103- 8112, 2018.","title":"A lyapunov-based approach to safe reinforcement learning","authors":"Chow","year":"2018","venue":"Proceedings of the 32nd International Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gábor et al., 1998] Zoltán Gábor, Zsolt Kalmár, and Csaba Szepesvári. Multi-criteria reinforcement learning. In Pro- ceedings of the 15th International Conference on Machine Learning, pages 197-205, 1998.","title":"Multi-criteria reinforcement learning","authors":"Zoltán Gábor, Zsolt Kalmár, Csaba Szepesvári","year":"1998","venue":"Proceedings of the 15th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"Hado V. Hasselt. Double q-learning. In Pro- ceedings of the 24th International Conference on Neural Information Processing Systems, pages 2613-2621, 2010.","title":"Double q-learning","authors":"V Hado, Hasselt","year":"2010","venue":"Proceedings of the 24th International Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hsu et al., 2020] Chloe Ching-Yun Hsu, Celestine Mendler- Dünner, and Moritz Hardt. Revisiting design choices in proximal policy optimization. arXiv:2009.10897, 2020.","title":"Revisiting design choices in proximal policy optimization","authors":"Chloe Ching-Yun, Celestine Hsu, Moritz Mendler-Dünner, Hardt","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kakade, 2001] Sham Kakade. A natural policy gradient. In Proceedings of the 14th International Conference on Neu- ral Information Processing Systems, pages 1531-1538, 2001.","title":"A natural policy gradient","authors":"Sham Kakade","year":"2001","venue":"Proceedings of the 14th International Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Konda and Tsitsiklis, 2000] Vijay R. Konda and John N. Tsitsiklis. Actor-critic algorithms. In Proceedings of the 13th International Conference on Neural Information Pro- cessing Systems, pages 1008-1014, 2000.","title":"Actor-critic algorithms","authors":"Vijay R Konda, John N Tsitsiklis","year":"2000","venue":"Proceedings of the 13th International Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lesser and Abate, 2018] K. Lesser and A. Abate. Multi- objective optimal control with safety as a priority. IEEE Transactions on Control Systems Technology, 26(3):1015- 1027, 2018.","title":"Multiobjective optimal control with safety as a priority","authors":"K Lesser, A Abate","year":"2018","venue":"IEEE Transactions on Control Systems Technology","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu et al., 2015] C. Liu, X. Xu, and D. Hu. Multiobjective reinforcement learning: A comprehensive overview. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 45(3):385-398, 2015.","title":"Multiobjective reinforcement learning: A comprehensive overview","authors":"C Liu, X Xu, D Hu","year":"2015","venue":"IEEE Transactions on Systems, Man, and Cybernetics: Systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu et al., 2019] Boyi Liu, Qi Cai, Zhuoran Yang, and Zhaoran Wang. Neural trust region/proximal policy op- timization attains globally optimal policy. In Proceedings of the 33rd International Conference on Neural Informa- tion Processing Systems, pages 10564-10575, 2019.","title":"Neural trust region/proximal policy optimization attains globally optimal policy","authors":"Boyi Liu, Qi Cai, Zhuoran Yang, Zhaoran Wang","year":"2019","venue":"Proceedings of the 33rd International Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Miryoosefi et al., 2019] Sobhan Miryoosefi, Kianté Brant- ley, Hal Daumé III, Miroslav Dudík, and Robert E. Schapire. Reinforcement learning with convex constraints. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, pages 14070- 14079, 2019.","title":"Reinforcement learning with convex constraints","authors":"Sobhan Miryoosefi, Kianté Brantley, Hal Daumé, Iii, Miroslav Dudík, Robert E Schapire","year":"2019","venue":"Proceedings of the 33rd International Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Mitten, 1974] L. G. Mitten. Preference order dynamic pro- gramming. Management Science, 21(1):43-46, 1974.","title":"Preference order dynamic programming","authors":"L G Mitten","year":"1974","venue":"Management Science","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Polymenakos et al., 2019] K. Polymenakos, A. Abate, and S. Roberts. Safe policy search using gaussian process models. In Proceedings of the 18th International Con- ference on Autonomous Agents and Multiagent Systems, pages 1565-1573, 2019.","title":"Safe policy search using gaussian process models","authors":"K Polymenakos, A Abate, S Roberts","year":"2019","venue":"Proceedings of the 18th International Conference on Autonomous Agents and Multiagent Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Rentmeesters et al., 1996] M.J. Rentmeesters, W.K. Tsai, and Kwei-Jay Lin. A theory of lexicographic multi-criteria optimization. In Proceedings of the 2nd IEEE Interna- tional Conference on Engineering of Complex Computer Systems, 1996.","title":"A theory of lexicographic multi-criteria optimization","authors":"M J Rentmeesters, W K Tsai, Kwei-Jay Lin","year":"1996","venue":"Proceedings of the 2nd IEEE International Conference on Engineering of Complex Computer Systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Roijers et al., 2013] D. M. Roijers, P. Vamplew, S. White- son, and R. Dazeley. A survey of multi-objective sequen- tial decision-making. Journal of Artificial Intelligence Re- search, 48:67-113, 2013.","title":"A survey of multi-objective sequential decision-making","authors":"D M Roijers, P Vamplew, S Whiteson, R Dazeley","year":"2013","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Rummery and Niranjan, 1994] Gavin Rummery and Mahe- san Niranjan. On-line q-learning using connectionist sys- tems. Technical report, University of Cambridge, 1994.","title":"On-line q-learning using connectionist systems","authors":"Gavin Rummery, Mahesan Niranjan","year":"1994","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Schulman et al., 2017] John Schulman, Filip Wolski, Pra- fulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv:1707.06347, 2017.","title":"Proximal policy optimization algorithms","authors":"John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Slater, 1950] Morton Slater. Lagrange multipliers revisited. Cowles Commission Discussion Paper No. 403, 1950.","title":"Lagrange multipliers revisited","authors":"Morton Slater","year":"1950","venue":"Cowles Commission Discussion Paper","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Sutton and Barto, 2018] Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, 2018.","title":"Reinforcement Learning: An Introduction","authors":"Richard S Sutton, Andrew G Barto","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Sutton et al., 1999] Richard S. Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approx- imation. In Proceedings of the 12th International Con- ference on Neural Information Processing Systems, pages 1057-1063, 1999.","title":"Policy gradient methods for reinforcement learning with function approximation","authors":"Richard S Sutton, David Mcallester, Satinder Singh, Yishay Mansour","year":"1999","venue":"Proceedings of the 12th International Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Tessler et al., 2019] Chen Tessler, Daniel J. Mankowitz, and Shie Mannor. Reward constrained policy optimization. In Proceedings of the 7th International Conference on Learn- ing Representations, 2019.","title":"Reward constrained policy optimization","authors":"Chen Tessler, Daniel J Mankowitz, Shie Mannor","year":"2019","venue":"Proceedings of the 7th International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Thomas et al., 2013] Philip S. Thomas, William Dabney, Sridhar Mahadevan, and Stephen Giguere. Projected natu- ral actor-critic. In Proceedings of the 26th International Conference on Neural Information Processing Systems, pages 2337-2345, 2013.","title":"Projected natural actor-critic","authors":"Philip S Thomas, William Dabney, Stephen Sridhar Mahadevan, Giguere","year":"2013","venue":"Proceedings of the 26th International Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[van Seijen et al., 2009] Harm van Seijen, Hado van Hasselt, Whiteson Shimon, and Marco Wiering. A theoretical and empirical analysis of expected sarsa. IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning, pages 177-184, 2009.","title":"A theoretical and empirical analysis of expected sarsa","authors":"Van Seijen","year":"2009","venue":"IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Watkins, 1986] Chris Watkins. Learning from Delayed Re- wards. PhD thesis, University of Cambridge, 1986.","title":"Learning from Delayed Rewards","authors":"Chris Watkins","year":"1986","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wray and Zilberstein, 2015] Kyle Hollins Wray and Shlomo Zilberstein. Multi-objective pomdps with lexico- graphic reward preferences. In Proceedings of the 24th International Conference on Artificial Intelligence, pages 1719-1725, 2015.","title":"Multi-objective pomdps with lexicographic reward preferences","authors":"Zilberstein Wray, Kyle Hollins Wray, Shlomo Zilberstein","year":"2015","venue":"Proceedings of the 24th International Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2015/245"}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}