{"type":"ScholarlyArticle","title":"Modeling Moral Choices in Social Dilemmas with Multi-Agent Reinforcement Learning","authors":[{"name":"Elizaveta Tennant","slug":"elizaveta-tennant","given":"Elizaveta","family":"Tennant","affiliation":"University College London"},{"name":"Stephen Hailes","slug":"stephen-hailes","given":"Stephen","family":"Hailes","affiliation":"University College London"},{"name":"Mirco Musolesi","slug":"mirco-musolesi","given":"Mirco","family":"Musolesi","affiliation":"University College London; University of Bologna"}],"abstract":"Practical uses of Artificial Intelligence (AI) in the real world have demonstrated the importance of embedding moral choices into intelligent agents. They have also highlighted that defining top-down ethical constraints on AI according to any one type of morality is extremely challenging and can pose risks. A bottom-up learning approach may be more appropriate for studying and developing ethical behavior in AI agents. In particular, we believe that an interesting and insightful starting point is the analysis of emergent behavior of Reinforcement Learning (RL) agents that act according to a predefined set of moral rewards in social dilemmas.\r\n\r\nIn this work, we present a systematic analysis of the choices made by intrinsically-motivated RL agents whose rewards are based on moral theories. We aim to design reward structures that are simplified yet representative of a set of key ethical systems. Therefore, we first define moral reward functions that distinguish between consequence- and norm-based agents, between morality based on societal norms or internal virtues, and between single- and mixed-virtue (e.g., multi-objective) methodologies. Then, we evaluate our approach by modeling repeated dyadic interactions between learning moral agents in three iterated social dilemma games (Prisoner's Dilemma, Volunteer's Dilemma and Stag Hunt). We analyze the impact of different types of morality on the emergence of cooperation, defection or exploitation, and the corresponding social outcomes. Finally, we discuss the implications of these findings for the development of moral agents in artificial and mixed human-AI societies.","doi":"10.24963/ijcai.2023/36","url":"https://ijcai.org/proceedings/2023/36","pdf":"https://ijcai.org/proceedings/2023/0036.pdf","markdown":"https://ijcai.org/proceedings/2023/36.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/36","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Agent-based and Multi-agent Systems","track":"Main Track","number":36,"pages":{"first":317,"last":325},"keywords":["AI Ethics, Trust, Fairness: ETF: Moral decision making","Agent-based and Multi-agent Systems: MAS: Multi-agent learning"],"videos":[],"references":[{"position":1,"raw":"[Abel et al., 2016] David Abel, James MacGlashan, and Michael L. Littman. Reinforcement learning as a frame- work for ethical decision making. In AAAI Workshop: AI, Ethics, and Society, 2016.","title":"Reinforcement learning as a framework for ethical decision making","authors":"David Abel, James Macglashan, Michael L Littman","year":"2016","venue":"AAAI Workshop: AI, Ethics","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Ajmeri et al., 2020] Nirav Ajmeri, Hui Guo, Pradeep K. Murukannaiah, and Munindar P. Singh. Elessar: Ethics in norm-aware agents. In AAMAS'20, pages 16-24, 2020.","title":"Elessar: Ethics in norm-aware agents","authors":"Nirav Ajmeri, Hui Guo, K Pradeep, Munindar P Murukannaiah, Singh","year":"2020","venue":"AAMAS'20","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Amodei et al., 2016] Dario Amodei, Chris Olah, Jacob Steinhardt, et al. Concrete problems in ai safety. arXiv preprint arXiv:1606.06565, 2016.","title":"Concrete problems in ai safety","authors":"Dario Amodei, Chris Olah, Jacob Steinhardt","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Anastassacos et al., 2020] Nicolas Anastassacos, Stephen Hailes, and Mirco Musolesi. Partner selection for the emergence of cooperation in multi-agent systems using reinforcement learning. In AAAI '20, volume 34, pages 7047-7054, Apr. 2020.","title":"Partner selection for the emergence of cooperation in multi-agent systems using reinforcement learning","authors":"Anastassacos","year":"2020","venue":"AAAI '20","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Anastassacos et al., 2021] Nicolas Anastassacos, Julian García, Stephen Hailes, and Mirco Musolesi. Cooperation and reputation dynamics with reinforcement learning. In AAMAS'21, pages 115-123, 2021.","title":"Cooperation and reputation dynamics with reinforcement learning","authors":"Nicolas Anastassacos, Julian García, Stephen Hailes, Mirco Musolesi","year":"2021","venue":"AAMAS'21","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Anderson et al., 2005] Michael Anderson, Susan Anderson, and Chris Armen. Towards machine ethics: implementing two action-based ethical theories. In AAAI 2005 Fall Sym- posium on Machine Ethics, pages 1-7, 2005.","title":"Towards machine ethics: implementing two action-based ethical theories","authors":"Michael Anderson, Susan Anderson, Chris Armen","year":"2005","venue":"AAAI 2005 Fall Symposium on Machine Ethics","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Aristotle, 2009] Aristotle. The Nicomachean Ethics. Oxford world's classics. Oxford University Press, 2009.","title":"Aristotle. The Nicomachean Ethics. Oxford world's classics","authors":"Aristotle","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Awad et al., 2018] Edmond Awad, Sohan Dsouza, Richard Kim, et al. The moral machine experiment. Nature, 563(7729):59-64, 2018.","title":"The moral machine experiment","authors":"Edmond Awad, Sohan Dsouza, Richard Kim","year":"2018","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Awad et al., 2022] Edmond Awad, Sydney Levine, Michael Anderson, et al. Computational ethics. Trends in Cognitive Sciences, 26(5):388-405, 2022.","title":"Computational ethics","authors":"Awad","year":"2022","venue":"Trends in Cognitive Sciences","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Axelrod and Hamilton, 1981] Robert Axelrod and William D Hamilton. The evolution of cooperation. Science, 211(4489):1390-1396, 1981.","title":"The evolution of cooperation","authors":"Robert Axelrod, William D Hamilton","year":"1981","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Bengio et al., 2009] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In ICML'09, pages 41-48, 2009.","title":"Curriculum learning","authors":"Yoshua Bengio, Jérôme Louradour, Ronan Collobert, Jason Weston","year":"2009","venue":"ICML'09","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Bentahila et al., 2021] Lina Bentahila, Roger Fontaine, and Valérie Pennequin. Universality and cultural diversity in moral reasoning and judgment. Frontiers in Psychology, 12, 2021.","title":"Universality and cultural diversity in moral reasoning and judgment","authors":"Lina Bentahila, Roger Fontaine, Valérie Pennequin","year":"2021","venue":"Frontiers in Psychology","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Bentham, 1996] Jeremy Bentham. An Introduction to the Principles of Morals and Legislation: the Collected Works of Jeremy Bentham. Oxford University Press, 1996.","title":"An Introduction to the Principles of Morals and Legislation: the Collected Works of Jeremy Bentham","authors":"Jeremy Bentham","year":"1996","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Binmore, 2005] Ken Binmore. Natural Justice. Oxford University Press, 2005.","title":"Natural Justice","authors":"Ken Binmore","year":"2005","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Carroll et al., 2019] Micah Carroll, Rohin Shah, Mark K Ho, et al. On the utility of learning about humans for human-ai coordination. In NeurIPS'19, volume 32, pages 5174-5185, 2019.","title":"On the utility of learning about humans for human-ai coordination","authors":"Micah Carroll, Rohin Shah, Mark K Ho","year":"2019","venue":"NeurIPS'","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Cavagnetto and Gahir, 2014] Stefano Cavagnetto and Bruce Gahir. Game theory -its applications to ethical decision making. CRIS -Bulletin of the Centre for Research and Interdisciplinary Study, 2014(1), 2014.","title":"Game theory -its applications to ethical decision making","authors":"Stefano Cavagnetto, Bruce Gahir","year":"2014","venue":"CRIS -Bulletin of the Centre for Research and Interdisciplinary Study","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Chentanez et al., 2004] Nuttapong Chentanez, Andrew Barto, and Satinder Singh. Intrinsically motivated rein- forcement learning. In NeurIPS'04, pages 1281-1288, 2004.","title":"Intrinsically motivated reinforcement learning","authors":"Chentanez","year":"2004","venue":"NeurIPS'04","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Cunningham, 1967] R L Cunningham. Ethics and game the- ory: The prisoner's dilemma. Papers on Non-market De- cision Making, 2(1):11-26, 1967.","title":"Ethics and game theory: The prisoner's dilemma","authors":"R L Cunningham","year":"1967","venue":"Papers on Non-market Decision Making","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[de Cote et al., 2006] Enrique Munoz de Cote, Alessandro Lazaric, and Marcello Restelli. Learning to cooperate in multi-agent social dilemmas. In AAMAS'06, pages 783- 785, 2006.","title":"Learning to cooperate in multi-agent social dilemmas","authors":"Enrique Munoz De Cote, Alessandro Lazaric, Marcello Restelli","year":"2006","venue":"AAMAS'06","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Dignum, 2017] Virginia Dignum. Responsible autonomy. In IJCAI'17, pages 4698-4704, 2017.","title":"Virginia Dignum. Responsible autonomy","authors":"Dignum","year":"2017","venue":"IJCAI'17","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Fehr and Fischbacher, 2002] Ernst Fehr and Urs Fis- chbacher. Why social preferences matter-the impact of non-selfish motives on competition, cooperation and incentives. The Economic Journal, 112(478):1-33, 2002.","title":"Why social preferences matter-the impact of non-selfish motives on competition, cooperation and incentives","authors":"Fischbacher Fehr","year":"2002","venue":"The Economic Journal","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Fehr and Fischbacher, 2004] Ernst Fehr and Urs Fis- chbacher. Social norms and human cooperation. Trends in Cognitive Sciences, 8(4):185-190, 2004.","title":"Social norms and human cooperation","authors":"Fischbacher Fehr","year":"2004","venue":"Trends in Cognitive Sciences","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Foerster et al., 2018] Jakob Foerster, Richard Y. Chen, Maruan Al-Shedivat, et al. Learning with opponent- learning awareness. In AAMAS'18, page 122-130, 2018.","title":"Learning with opponentlearning awareness","authors":"Jakob Foerster, Richard Y Chen, Maruan Al-Shedivat","year":"2018","venue":"AAMAS'18","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Gini, 1912] Corrado Gini. Variabilità e Mutabilità: Con- tributo allo studio delle distribuzioni e delle relazioni statistiche. [Fasc. I.]. Tipografia di P. Cuppini, 1912.","title":"Variabilità e Mutabilità: Contributo allo studio delle distribuzioni e delle relazioni statistiche","authors":"Corrado Gini","year":"1912","venue":"Tipografia di P. Cuppini","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Graham et al., 2009] Jesse Graham, Jonathan Haidt, and Brian A Nosek. Liberals and conservatives rely on dif- ferent sets of moral foundations. Journal of Personality and Social Psychology, 96(5):1029, 2009.","title":"Liberals and conservatives rely on different sets of moral foundations","authors":"Jesse Graham, Jonathan Haidt, Brian A Nosek","year":"2009","venue":"Journal of Personality and Social Psychology","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Graham et al., 2013] Jesse Graham, Jonathan Haidt, Sena Koleva, et al. Moral foundations theory: The pragmatic validity of moral pluralism. volume 47 of Advances in Ex- perimental Social Psychology, pages 55-130. Academic Press, 2013.","title":"Moral foundations theory: The pragmatic validity of moral pluralism","authors":"Jesse Graham, Jonathan Haidt, Sena Koleva","year":"2013","venue":"Advances in Experimental Social Psychology","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Harper et al., 2017] Marc Harper, Vincent Knight, Martin Jones, et al. Reinforcement learning produces dominant strategies for the iterated prisoner's dilemma. PLOS ONE, 12(12), 2017.","title":"Reinforcement learning produces dominant strategies for the iterated prisoner's dilemma","authors":"Marc Harper, Vincent Knight, Martin Jones","year":"2017","venue":"PLOS ONE","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Hegde et al., 2020] Aditya Hegde, Vibhav Agarwal, and Shrisha Rao. Ethics, prosperity, and society: moral eval- uation using virtue ethics and utilitarianism. In IJCAI'20, pages 167-174, 2020.","title":"Ethics, prosperity, and society: moral evaluation using virtue ethics and utilitarianism","authors":"Aditya Hegde, Vibhav Agarwal, Shrisha Rao","year":"2020","venue":"IJCAI'20","doi":null,"url":null,"internal_url":"/proceedings/2020/24"},{"position":29,"raw":"[Hofbauer and Sigmund, 1998] Josef Hofbauer and Karl Sigmund. Evolutionary Games and Population Dynamics. Cambridge University Press, 1998.","title":"Evolutionary Games and Population Dynamics","authors":"Sigmund Hofbauer","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Hughes et al., 2018] Edward Hughes, Joel Z Leibo, Matthew Phillips, et al. Inequity aversion improves coop- eration in intertemporal social dilemmas. In NeurIPS'18, pages 3330-3340, 2018.","title":"Inequity aversion improves cooperation in intertemporal social dilemmas","authors":"Edward Hughes, Joel Z Leibo, Matthew Phillips","year":"2018","venue":"NeurIPS'","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Jaques et al., 2019] Natasha Jaques, Angeliki Lazaridou, Edward Hughes, et al. Social influence as intrinsic mo- tivation for multi-agent deep reinforcement learning. In ICML'19, pages 3040-3049, 2019.","title":"Social influence as intrinsic motivation for multi-agent deep reinforcement learning","authors":"Natasha Jaques, Angeliki Lazaridou, Edward Hughes","year":"2019","venue":"ICML'19","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Kant, 1981] Immanuel Kant. Grounding for the meta- physics of morals. 1785. Trans. James W. Ellington. In- dianapolis: Hackett, 1981.","title":"Immanuel Kant. Grounding for the metaphysics of morals","authors":"Kant","year":"1785","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Kilbride, 1993] Daniel Kilbride. Slavery and utilitarianism: Thomas cooper and the mind of the old south. The Journal of Southern History, 59(3):469-486, 1993.","title":"Slavery and utilitarianism: Thomas cooper and the mind of the old south","authors":"Daniel Kilbride, Kilbride","year":"1993","venue":"The Journal of Southern History","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Kleiman-Weiner et al., 2017] Max Kleiman-Weiner, Re- becca Saxe, and Joshua B. Tenenbaum. Learning a com- monsense moral theory. Cognition, 167:107-123, 2017. Moral Learning.","title":"Learning a commonsense moral theory","authors":"Kleiman-Weiner","year":"2017","venue":"Moral Learning","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Leibo et al., 2017] Joel Z. Leibo, Vinicius Zambaldi, Marc Lanctot, et al. Multi-agent reinforcement learning in se- quential social dilemmas. In AAMAS'17, page 464-473, 2017.","title":"Multi-agent reinforcement learning in sequential social dilemmas","authors":"Joel Z Leibo, Vinicius Zambaldi, Marc Lanctot","year":"2017","venue":"AAMAS'17","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Littman, 1994] Michael L. Littman. Markov games as a framework for multi-agent reinforcement learning. In ICML'94, page 157-163, 1994.","title":"Markov games as a framework for multi-agent reinforcement learning","authors":"L Michael, Littman","year":"1994","venue":"ICML'94","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Mabille and Stoker, 2021] Louise Mabille and Henk Stoker. The Morality Wars: The Ongoing Debate Over the Origin of Human Goodness. Fortress Academic, 2021.","title":"The Morality Wars: The Ongoing Debate Over the Origin of Human Goodness","authors":"Stoker Mabille, Louise Mabille, Henk Stoker","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[McKee et al., 2020] Kevin R. McKee, Ian Gemp, Brian McWilliams, et al. Social diversity and social preferences in mixed-motive reinforcement learning. In AAMAS'20, page 869-877, 2020.","title":"Social diversity and social preferences in mixed-motive reinforcement learning","authors":"Kevin R Mckee, Ian Gemp, Brian Mcwilliams","year":"2020","venue":"AAMAS'20","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Peschl et al., 2022] Markus Peschl, Arkady Zgonnikov, Frans A. Oliehoek, and Luciano C. Siebert. MORAL: aligning AI with human norms through multi-objective re- inforced active learning. In AAMAS'22, page 1038-1046, 2022.","title":"MORAL: aligning AI with human norms through multi-objective reinforced active learning","authors":"Markus Peschl, Arkady Zgonnikov, Frans A Oliehoek, Luciano C Siebert","year":"2022","venue":"AAMAS'22","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Peysakhovich and Lerer, 2018a] Alexander Peysakhovich and Adam Lerer. Consequentialist conditional coopera- tion in social dilemmas with imperfect information. In ICLR'18, 2018.","title":"Consequentialist conditional cooperation in social dilemmas with imperfect information","authors":"Lerer Peysakhovich, Alexander Peysakhovich, Adam Lerer","year":"2018","venue":"ICLR'18","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Peysakhovich and Lerer, 2018b] Alexander Peysakhovich and Adam Lerer. Prosocial learning agents solve gener- alized stag hunts better than selfish ones. In AAMAS'18, page 2043-2044, 2018.","title":"Prosocial learning agents solve generalized stag hunts better than selfish ones","authors":"Lerer Peysakhovich, Alexander Peysakhovich, Adam Lerer","year":"2018","venue":"AAMAS'18","doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Poundstone, 1993] William Poundstone. Prisoner's Dilemma: John von Neumann, Game Theory, and the Puzzle of the Bomb. Anchor Books, 1993.","title":"Prisoner's Dilemma: John von Neumann, Game Theory, and the Puzzle of the Bomb","authors":"William Poundstone, Poundstone","year":"1993","venue":null,"doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Rahwan et al., 2019] Iyad Rahwan, Manuel Cebrian, Nick Obradovich, et al. Machine behaviour. Nature, 568(7753):477-486, 2019.","title":"Machine behaviour","authors":"Rahwan","year":"2019","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":44,"raw":"[Rapoport, 1974] Anatol Rapoport. Prisoner's dilemma - recollections and observations. In Game Theory as a The- ory of a Conflict Resolution, pages 17-34. Springer, 1974.","title":"Prisoner's dilemmarecollections and observations","authors":"Anatol Rapoport","year":"1974","venue":"Game Theory as a Theory of a Conflict Resolution","doi":null,"url":null,"internal_url":null},{"position":45,"raw":"[Rodriguez-Soto et al., 2021] Manel Rodriguez-Soto, Maite Lopez-Sanchez, and Juan A. Rodriguez Aguilar. Multi- objective reinforcement learning for designing ethical en- vironments. In IJCAI'21, pages 545-551, 2021.","title":"Multiobjective reinforcement learning for designing ethical environments","authors":"Manel Rodriguez-Soto, Maite Lopez-Sanchez, Juan A Rodriguez Aguilar","year":"2021","venue":"IJCAI'21","doi":null,"url":null,"internal_url":null},{"position":46,"raw":"[Rodriguez-Soto et al., 2022] Manel Rodriguez-Soto, Marc Serramia, Maite Lopez-Sanchez, and Juan Antonio Rodriguez-Aguilar. Instilling moral value alignment by means of multi-objective reinforcement learning. Ethics and Information Technology, 24(1):1-17, 2022.","title":"Instilling moral value alignment by means of multi-objective reinforcement learning","authors":"Rodriguez-Soto","year":"2022","venue":"Ethics and Information Technology","doi":null,"url":null,"internal_url":null},{"position":47,"raw":"[Salahshour, 2022] Mohammad Salahshour. Interaction be- tween games give rise to the evolution of moral norms of cooperation. PLOS Computational Biology, 18(9):1-35, 2022.","title":"Interaction between games give rise to the evolution of moral norms of cooperation","authors":"Mohammad Salahshour","year":"2022","venue":"PLOS Computational Biology","doi":null,"url":null,"internal_url":null},{"position":48,"raw":"[Sandholm and Crites, 1996] Tuomas W Sandholm and Robert H Crites. Multiagent reinforcement learning in the iterated prisoner's dilemma. Biosystems, 37(1-2):147- 166, 1996.","title":"Multiagent reinforcement learning in the iterated prisoner's dilemma","authors":"Crites ; ] Sandholm, W Tuomas, Robert H Sandholm, Crites","year":"1996","venue":"Biosystems","doi":null,"url":null,"internal_url":null},{"position":49,"raw":"[Sigmund, 2010] Karl Sigmund. The Calculus of Selfishness. Princeton University Press, 2010.","title":"The Calculus of Selfishness","authors":"Karl Sigmund","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":50,"raw":"[Skyrms, 2001] Brian Skyrms. The stag hunt. Proceedings and Addresses of the American Philosophical Association, 75(2):31-41, 2001.","title":"The stag hunt","authors":"Brian Skyrms","year":"2001","venue":"Proceedings and Addresses of the American Philosophical Association","doi":null,"url":null,"internal_url":null},{"position":51,"raw":"[Sutton and Barto, 2018] Richard S Sutton and Andrew G Barto. Reinforcement Learning, Second Edition: An In- troduction. MIT Press, 2018.","title":"Reinforcement Learning, Second Edition: An Introduction","authors":"S Richard, Andrew G Sutton, Barto","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":52,"raw":"[Tolmeijer et al., 2021] Suzanne Tolmeijer, Markus Kneer, Cristina Sarasua, et al. Implementations in machine ethics: a survey. ACM Computing Surveys, 53(6), 2021.","title":"Implementations in machine ethics: a survey","authors":"Suzanne Tolmeijer, Markus Kneer, Cristina Sarasua","year":"2021","venue":"ACM Computing Surveys","doi":null,"url":null,"internal_url":null},{"position":53,"raw":"[Vamplew et al., 2018] Peter Vamplew, Richard Dazeley, Cameron Foale, et al. Human-aligned artificial intelli- gence is a multiobjective problem. Ethics and Information Technology, 20(1):27-40, 2018.","title":"Human-aligned artificial intelligence is a multiobjective problem","authors":"Peter Vamplew, Richard Dazeley, Cameron Foale","year":"2018","venue":"Ethics and Information Technology","doi":null,"url":null,"internal_url":null},{"position":54,"raw":"[Wallach and Allen, 2009] Wendell Wallach and Colin Allen. Moral Machines: Teaching Robots Right from Wrong. Oxford University Press, feb 2009.","title":"Moral Machines: Teaching Robots Right from Wrong","authors":"Allen ; Wallach, Wendell Wallach, Colin Allen","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":55,"raw":"[Wallach, 2010] Wendell Wallach. Robot minds and hu- man ethics: the need for a comprehensive model of moral decision making. Ethics and Information Technology, 12(3):243-250, 2010.","title":"Robot minds and human ethics: the need for a comprehensive model of moral decision making","authors":"Wendell Wallach","year":"2010","venue":"Ethics and Information Technology","doi":null,"url":null,"internal_url":null},{"position":56,"raw":"[Watkins and Dayan, 1992] Christopher JCH Watkins and Peter Dayan. Q-learning. Machine Learning, 8(3):279- 292, 1992.","title":"Q-learning","authors":"Jch Christopher, Peter Watkins, Dayan","year":"1992","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":57,"raw":"[Yu et al., 2018] Han Yu, Zhiqi Shen, Chunyan Miao, et al. Building ethics into artificial intelligence. In IJCAI'18, pages 5527-5533, 2018.","title":"Building ethics into artificial intelligence","authors":"Han Yu, Zhiqi Shen, Chunyan Miao","year":"2018","venue":"IJCAI'18","doi":null,"url":null,"internal_url":"/proceedings/2018/779"}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}