{"type":"ScholarlyArticle","title":"Distributional Multi-Objective Decision Making","authors":[{"name":"Willem Röpke","slug":"willem-ropke","given":"Willem","family":"Röpke","affiliation":"Vrije Universiteit Brussel , Brussels , Belgium"},{"name":"Conor F. Hayes","slug":"conor-f-hayes","given":"Conor F.","family":"Hayes","affiliation":"University of Galway , Galway , Ireland"},{"name":"Patrick Mannion","slug":"patrick-mannion","given":"Patrick","family":"Mannion","affiliation":"University of Galway , Galway , Ireland"},{"name":"Enda Howley","slug":"enda-howley","given":"Enda","family":"Howley","affiliation":"University of Galway , Galway , Ireland"},{"name":"Ann Nowé","slug":"ann-nowe","given":"Ann","family":"Nowé","affiliation":"Vrije Universiteit Brussel , Brussels , Belgium"},{"name":"Diederik M. Roijers","slug":"diederik-m-roijers","given":"Diederik M.","family":"Roijers","affiliation":"Vrije Universiteit Brussel , Brussels , Belgium; City of Amsterdam , Amsterdam , The Netherlands"}],"abstract":"For effective decision support in scenarios with conflicting objectives, sets of potentially optimal solutions can be presented to the decision maker. We explore both what policies these sets should contain and how such sets can be computed efficiently. With this in mind, we take a distributional approach and introduce a novel dominance criterion relating return distributions of policies directly. Based on this criterion, we present the distributional undominated set and show that it contains optimal policies otherwise ignored by the Pareto front. In addition, we propose the convex distributional undominated set and prove that it comprises all policies that maximise expected utility for multivariate risk-averse decision makers. We propose a novel algorithm to learn the distributional undominated set and further contribute pruning operators to reduce the set to the convex distributional undominated set. Through experiments, we demonstrate the feasibility and effectiveness of these methods, making this a valuable new approach for decision support in real-world problems.","doi":"10.24963/ijcai.2023/634","url":"https://ijcai.org/proceedings/2023/634","pdf":"https://ijcai.org/proceedings/2023/0634.pdf","markdown":"https://ijcai.org/proceedings/2023/634.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/634","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Uncertainty in AI","track":"Main Track","number":634,"pages":{"first":5711,"last":5719},"keywords":["Machine Learning: ML: Reinforcement learning","Uncertainty in AI: UAI: Sequential decision making","Uncertainty in AI: UAI: Other"],"videos":[],"references":[{"position":1,"raw":"[Bawa et al., 1985] Vijay S. Bawa, James N. Bodurtha, M. R. Rao, and Hira L. Suri. On determination of stochas- tic dominance optimal sets. The Journal of Finance, 40(2):417-431, 1985.","title":"On determination of stochastic dominance optimal sets","authors":"Vijay S Bawa, James N Bodurtha, M R Rao, Hira L Suri","year":"1985","venue":"The Journal of Finance","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bellemare et al., 2023] Marc G. Bellemare, Will Dabney, and Mark Rowland. Distributional Reinforcement Learn- ing. MIT Press, 2023. http://www.distributional-rl.org.","title":"Distributional Reinforcement Learning","authors":"G Marc, Will Bellemare, Mark Dabney, Rowland","year":"2023","venue":null,"doi":null,"url":"http://www.distributional-rl.org","internal_url":null},{"position":3,"raw":"[Denuit et al., 2013] Michel Denuit, Louis Eeckhoudt, Ilia Tsetlin, and Robert L. Winkler. Multivariate Concave and Convex Stochastic Dominance. In Francesca Biagini, An- dreas Richter, and Harris Schlesinger, editors, Risk Mea- sures and Attitudes, pages 11-32. Springer London, Lon- don, 2013.","title":"Multivariate Concave and Convex Stochastic Dominance","authors":"Michel Denuit, Louis Eeckhoudt, Ilia Tsetlin, Robert L Winkler","year":"2013","venue":"Risk Measures and Attitudes","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Epshteyn and DeJong, 2006] Arkady Epshteyn and Gerald DeJong. Qualitative reinforcement learning. In Proceed- ings of the 23rd International Conference on Machine Learning, ICML '06, page 305-312, New York, NY, USA, 2006. Association for Computing Machinery.","title":"Qualitative reinforcement learning","authors":"Dejong Epshteyn, Arkady Epshteyn, Gerald Dejong","year":"2006","venue":"Proceedings of the 23rd International Conference on Machine Learning, ICML '06","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Fishburn, 1974] Peter C Fishburn. Convex stochastic dom- inance with continuous distribution functions. Journal of Economic Theory, 7(2):143-158, February 1974.","title":"Convex stochastic dominance with continuous distribution functions","authors":"Peter C Fishburn","year":"1974","venue":"Journal of Economic Theory","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Guerreiro et al., 2021] Andreia P. Guerreiro, Carlos M. Fonseca, and Luís Paquete. The hypervolume indicator: Computational problems and algorithms. ACM Comput. Surv., 54(6), jul 2021.","title":"The hypervolume indicator: Computational problems and algorithms","authors":"P Andreia, Carlos M Guerreiro, Luís Fonseca, Paquete","year":"2021","venue":"ACM Comput. Surv","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hayes et al., 2021] Conor F Hayes, Mathieu Reymond, Diederik M Roijers, Enda Howley, and Patrick Mannion. Distributional monte carlo tree search for risk-aware and multi-objective reinforcement learning. In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems, pages 1530-1532, 2021.","title":"Distributional monte carlo tree search for risk-aware and multi-objective reinforcement learning","authors":"F Conor, Mathieu Hayes, Reymond, M Diederik, Enda Roijers, Patrick Howley, Mannion","year":"2021","venue":"Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hayes et al., 2022a] Conor F. Hayes, Roxana Rȃdulescu, Eugenio Bargiacchi, Johan Källström, Matthew Macfar- lane, Mathieu Reymond, Timothy Verstraeten, Luisa M. Zintgraf, Richard Dazeley, Fredrik Heintz, Enda How- ley, Athirai A. Irissappane, Patrick Mannion, Ann Nowé, Gabriel Ramos, Marcello Restelli, Peter Vamplew, and Diederik M. Roijers. A practical guide to multi-objective reinforcement learning and planning. Autonomous Agents and Multi-Agent Systems, 36(1):26, April 2022.","title":"Roijers. A practical guide to multi-objective reinforcement learning and planning","authors":"Hayes","year":"2022","venue":"Autonomous Agents and Multi-Agent Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hayes et al., 2022b] Conor F Hayes, Diederik M Roijers, Enda Howley, and Patrick Mannion. Decision-theoretic planning for the expected scalarised returns. In Proceed- ings of the 21st International Conference on Autonomous Agents and Multiagent Systems, pages 1621-1623, 2022.","title":"Decision-theoretic planning for the expected scalarised returns","authors":"Hayes","year":"2022","venue":"Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hayes et al., 2022c] Conor F. Hayes, Timothy Verstraeten, Diederik M. Roijers, Enda Howley, and Patrick Mannion. Expected scalarised returns dominance: A new solution concept for multi-objective decision making. Neural Com- puting and Applications, July 2022.","title":"Expected scalarised returns dominance: A new solution concept for multi-objective decision making","authors":"Hayes","year":"2022","venue":"Neural Computing and Applications","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Levy, 2016a] Haim Levy. Bivariate FSD (BFSD). In Stochastic Dominance: Investment Decision Making un- der Uncertainty, pages 441-465. Springer International Publishing, Cham, 2016.","title":"Stochastic Dominance: Investment Decision Making under Uncertainty","authors":"Levy","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Levy, 2016b] Haim Levy. Stochastic dominance decision rules. In Stochastic Dominance: Investment Decision Making under Uncertainty, pages 41-124. Springer Inter- national Publishing, Cham, 2016.","title":"Stochastic dominance decision rules","authors":"Levy ; Haim Levy","year":"2016","venue":"Stochastic Dominance: Investment Decision Making under Uncertainty","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Mandow et al., 2022] L. Mandow, J. L. Perez-de-la-Cruz, and N. Pozas. Multi-objective dynamic programming with limited precision. Journal of Global Optimization, 82(3):595-614, March 2022.","title":"Multi-objective dynamic programming with limited precision","authors":"L Mandow, J L Perez-De-La-Cruz, N Pozas","year":"2022","venue":"Journal of Global Optimization","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Martin et al., 2020] John Martin, Michal Lyskawinski, Xi- aohu Li, and Brendan Englot. Stochastically dominant dis- tributional reinforcement learning. In International Con- ference on Machine Learning, pages 6745-6754. PMLR, 2020.","title":"Stochastically dominant distributional reinforcement learning","authors":"John Martin, Michal Lyskawinski, Xiaohu Li, Brendan Englot","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Reymond et al., 2023] Mathieu Reymond, Conor F. Hayes, Denis Steckelmacher, Diederik M. Roijers, and Ann Nowé. Actor-critic multi-objective reinforcement learning for non-linear utility functions. Autonomous Agents and Multi-Agent Systems, 37(2):23, April 2023.","title":"Actor-critic multi-objective reinforcement learning for non-linear utility functions","authors":"Conor F Mathieu Reymond, Denis Hayes, Steckelmacher, M Diederik, Ann Roijers, Nowé","year":"2023","venue":"Autonomous Agents and Multi-Agent Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Richard, 1975] Scott F. Richard. Multivariate Risk Aver- sion, Utility Independence and Separable Utility Func- tions. Management Science, 22(1):12-21, September 1975.","title":"Multivariate Risk Aversion, Utility Independence and Separable Utility Functions","authors":"F Scott, Richard","year":"1975","venue":"Management Science","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Roijers and Whiteson, 2017] Diederik M. Roijers and Shi- mon Whiteson. Multi-objective decision making. In Synthesis Lectures on Artificial Intelligence and Machine Learning, volume 34, pages 129-129.","title":"Multi-objective decision making","authors":"M Diederik, Shimon Roijers, Whiteson","year":null,"venue":"Synthesis Lectures on Artificial Intelligence and Machine Learning","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"Morgan and Clay- pool, 2017.","title":"Morgan and Claypool","authors":null,"year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Roijers et al., 2013] Diederik M. Roijers, Peter Vamplew, Shimon Whiteson, and Richard Dazeley. A survey of multi-objective sequential decision-making. Journal of Ar- tificial Intelligence Research, 48:67-113, 2013.","title":"A survey of multi-objective sequential decision-making","authors":"M Diederik, Peter Roijers, Shimon Vamplew, Richard Whiteson, Dazeley","year":"2013","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Rowland et al., 2018] Mark Rowland, Marc Bellemare, Will Dabney, Remi Munos, and Yee Whye Teh. An analy- sis of categorical distributional reinforcement learning. In Amos Storkey and Fernando Perez-Cruz, editors, Proceed- ings of the Twenty-First International Conference on Arti- ficial Intelligence and Statistics, volume 84 of Proceed- ings of Machine Learning Research, pages 29-37. PMLR, April 2018.","title":"An analysis of categorical distributional reinforcement learning","authors":"Mark Rowland, Marc Bellemare, Will Dabney, Remi Munos, Yee Whye Teh","year":"2018","venue":"Proceedings of the Twenty-First International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Scarsini, 1988] Marco Scarsini. Dominance Conditions for Multivariate Utility Functions. Management Science, 34(4):454-460, 1988.","title":"Dominance Conditions for Multivariate Utility Functions","authors":"Marco Scarsini","year":"1988","venue":"Management Science","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Taboada et al., 2007] Heidi A. Taboada, Fatema Baheran- wala, David W. Coit, and Naruemon Wattanapongsakorn. Practical solutions for multi-objective optimization: An application to system reliability design problems. Selected Papers Presented at the Fourth International Conference on Quality and Reliability, 92(3):314-322, March 2007.","title":"Practical solutions for multi-objective optimization: An application to system reliability design problems","authors":"Heidi A Taboada, Fatema Baheranwala, David W Coit, Naruemon Wattanapongsakorn","year":"2007","venue":"Selected Papers Presented at the Fourth International Conference on Quality and Reliability","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Vamplew et al., 2009] Peter Vamplew, Richard Dazeley, Ewan Barker, and Andrei Kelarev. Constructing Stochastic Mixture Policies for Episodic Multiobjective Reinforce- ment Learning Tasks. In Ann Nicholson and Xiaodong Li, editors, AI 2009: Advances in Artificial Intelligence, pages 340-349, Berlin, Heidelberg, 2009. Springer Berlin Heidelberg.","title":"Constructing Stochastic Mixture Policies for Episodic Multiobjective Reinforcement Learning Tasks","authors":"Vamplew","year":"2009","venue":"AI 2009: Advances in Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Vamplew et al., 2022] Peter Vamplew, Cameron Foale, and Richard Dazeley. The impact of environmental stochas- ticity on value-based multiobjective reinforcement learn- ing. Neural Computing and Applications, 34(3):1783- 1799, February 2022.","title":"The impact of environmental stochasticity on value-based multiobjective reinforcement learning","authors":"Vamplew","year":"2022","venue":"Neural Computing and Applications","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Van Moffaert and Nowé, 2014] Kristof Van Moffaert and Ann Nowé. Multi-objective reinforcement learning using sets of pareto dominating policies. Journal of Machine Learning Research, 15(107):3663-3692, 2014.","title":"Multi-objective reinforcement learning using sets of pareto dominating policies","authors":"Van Moffaert, Nowé ; Kristof Van Moffaert, Ann Nowé","year":"2014","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Van Moffaert et al., 2013] Kristof Van Moffaert, Madalina M. Drugan, and Ann Nowé. Scalarized multi-objective reinforcement learning: Novel design techniques. In 2013 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL), pages 191-199, 2013.","title":"Scalarized multi-objective reinforcement learning: Novel design techniques","authors":"Kristof Van Moffaert, Madalina M Drugan, Ann Nowé","year":"2013","venue":"IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}