{"type":"ScholarlyArticle","title":"Normative Testimony and Belief Functions: A Formal Theory of Norm Learning","authors":[{"name":"Taylor Olson","slug":"taylor-olson","given":"Taylor","family":"Olson","affiliation":""},{"name":"Kenneth D. Forbus","slug":"kenneth-d-forbus","given":"Kenneth D.","family":"Forbus","affiliation":""}],"abstract":"The ability to learn another’s moral beliefs is necessary for all social agents. It allows us to predict their behavior and is a prerequisite to correcting their beliefs if they are incorrect. To make AI systems more socially competent, a formal theory for learning internal normative beliefs is thus needed. However, to the best of our knowledge, a philosophically justified formal theory for this process does not yet exist. This paper begins the development of such a theory, focusing on learning from testimony. We make four main contributions. First, we provide a set of axioms that any such theory must satisfy. Second, we provide justification for belief functions, as opposed to traditional probability theory, for modeling norm learning. Third, we construct a novel learning function that satisfies these axioms. Fourth, we provide a complexity analysis of this formalism and proof that deontic rules are sound under its semantics. This paper thus serves as a theoretical contribution towards modeling learning norms from testimony, paving the road towards more social AI systems.","doi":"10.24963/ijcai.2024/53","url":"https://ijcai.org/proceedings/2024/53","pdf":"https://ijcai.org/proceedings/2024/0053.pdf","markdown":"https://ijcai.org/proceedings/2024/53.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/53","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"AI Ethics, Trust, Fairness","track":"Main Track","number":53,"pages":{"first":476,"last":484},"keywords":["Uncertainty in AI: UAI: Uncertainty representations","Knowledge Representation and Reasoning: KRR: Learning and reasoning","AI Ethics, Trust, Fairness: ETF: Values","Agent-based and Multi-agent Systems: MAS: Normative systems"],"videos":[],"references":[{"position":1,"raw":"[Andrighetto, G. et al., 2010] Giulia Andrighetto, Marco Campennì, Federico Cecconi, and Rosaria Conte. The complex loop of norm emergence: A simulation model. In Simulating Interacting Agents and Social Phenomena: The second world congress: 19-35. Springer Japan. doi.org/10.1007/978-4-431-99781-8_2. 2010.","title":"The complex loop of norm emergence: A simulation model","authors":"Giulia Andrighetto, Marco Campennì, Federico Cecconi, Rosaria Conte","year":null,"venue":"Simulating Interacting Agents and Social Phenomena: The second world congress","doi":".org/10.1007/978-4-431-99781-8_2.2010","url":null,"internal_url":null},{"position":2,"raw":"[Awad, E. et al., 2018] Edmond Awad, Sohan Dsouza, Rich- ard Kim, Jonathan Schulz, Joseph Henrich, Azim Shariff, Jean-François Bonnefon, and Iyad Rahwan. \"The moral machine experiment.\" Nature 563, no. 7729: 59-64. 2018.","title":"The moral machine experiment","authors":"Edmond Awad, Sohan Dsouza, Richard Kim, Jonathan Schulz, Joseph Henrich, Azim Shariff, Jean-François Bonnefon, Iyad Rahwan","year":"2018","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Borg J.S. et al., 2006] Jana Schaich Borg, Catherine Hynes, John Van Horn, Scott Grafton, and Walter Sinnott-Arm- strong. \"Consequences, action, and intention as factors in moral judgments: An fMRI investigation.\" Journal of cognitive neuroscience 18, no. 5: 803-817. 2006.","title":"Consequences, action, and intention as factors in moral judgments: An fMRI investigation","authors":"J S Borg","year":"2006","venue":"Journal of cognitive neuroscience","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Buckholtz, J.W. et al., 2015] Joshua W. Buckholtz, Justin W. Martin, Michael T. Treadway, Katherine Jan, David H. Zald, Owen Jones, and Rene Marois. \"From blame to punishment: disrupting prefrontal cortex activity reveals norm enforcement mechanisms.\" Neuron 87, no. 6: 1369- 1380. 2015.","title":"From blame to punishment: disrupting prefrontal cortex activity reveals norm enforcement mechanisms","authors":"Joshua W Buckholtz, Justin W Martin, Michael T Treadway, Katherine Jan, David H Zald, Owen Jones, Rene Marois","year":"2015","venue":"Neuron","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dalla Pozza, G. et al., 2011] Giorgio Dalla Pozza, Maria Sil- via Pini, Francesca Rossi, and K. Brent Venable. \"Multi- agent soft constraint aggregation via sequential voting.\" In Twenty-Second International Joint Conference on Ar- tificial Intelligence. 2011.","title":"Multiagent soft constraint aggregation via sequential voting","authors":"G Dalla Pozza","year":"2011","venue":"Twenty-Second International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"Dempster, A.P. 1967]","title":null,"authors":"A P Dempster","year":"1967","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"Arthur P. Dempster. Upper and Lower Probabilities Induced by a Multivalued Mapping. In An- nals of Mathematical Statistics 38, no. 2, 325-339. doi:10.1214/aoms/1177698950. 1967.","title":"Upper and Lower Probabilities Induced by a Multivalued Mapping","authors":"Arthur P Dempster","year":null,"venue":"Annals of Mathematical Statistics","doi":"10.1214/aoms/1177698950.1967","url":null,"internal_url":null},{"position":8,"raw":"Dubois, D., & Prade, H., 1992]. On the combination of evi- dence in various mathematical frameworks. In Reliability data collection and analysis (pp. 213-241). Dordrecht: Springer Netherlands. 1992.","title":"On the combination of evidence in various mathematical frameworks","authors":"D Dubois, H Prade","year":"1992","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Freitas dos Santos, T. et al., 2023] Thiago Freitas dos Santos, Nardine Osman, and Marco Schorlemmer. \"A multi-sce- nario approach to continuously learn and understand norm violations.\" Autonomous Agents and Multi-Agent Sys- tems 37, no. 2: 38. 2023.","title":"A multi-scenario approach to continuously learn and understand norm violations","authors":"T Freitas Dos Santos","year":"2023","venue":"Autonomous Agents and Multi-Agent Systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gold, N. et al., 2015] Natalie Gold, Briony D. Pulford, and Andrew M. Colman. \"Do as I say, don't do as I do: Dif- ferences in moral judgments do not translate into differ- ences in decisions in real-life trolley problems.\" Journal of Economic Psychology 47: 50-61. 2015.","title":"Do as I say, don't do as I do: Differences in moral judgments do not translate into differences in decisions in real-life trolley problems","authors":"Natalie Gold, D Briony, Andrew M Pulford, Colman","year":"2015","venue":"Journal of Economic Psychology","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hamid, N. H. A., et al., 2015] Nurzeatul Hamimah Abdul Hamid, Mohd Sharifuddin Ahmad, Azhana Ahmad, Aida Mustapha, Moamin A. Mahmoud, and Mohd Zaliman Mohd Yusoff. \"Trusting norms: A conceptual norms' trust framework for norms adoption in open normative multi-agent systems.\" In Distributed Computing and Ar- tificial Intelligence, 12th International Conference, pp. 149-157. Springer International Publishing, 2015.","title":"Trusting norms: A conceptual norms' trust framework for norms adoption in open normative multi-agent systems","authors":"N H A Hamid","year":"2015","venue":"Distributed Computing and Artificial Intelligence, 12th International Conference","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hills, A., 2009] Alison Hills. Moral Testimony and Moral Epistemology. Ethics, 120(1), 94-127. doi.org/10.1086/648610. 2009.","title":"Alison Hills. Moral Testimony and Moral Epistemology","authors":"A Hills","year":"2009","venue":"Ethics","doi":".org/10.1086/648610.2009","url":null,"internal_url":null},{"position":13,"raw":"[Jiang, L. et al., 2021] Liwei Jiang, Jena D. Hwang, Chandra Bhagavatula, Ronan Le Bras, Maxwell Forbes, Jon Bor- chardt, Jenny Liang, Oren Etzioni, Maarten Sap, and Yejin Choi. \"Delphi: Towards machine ethics and norms.\" arXiv preprint arXiv:2110.07574 6. 2021.","title":"Delphi: Towards machine ethics and norms","authors":"Liwei Jiang, Jena D Hwang, Chandra Bhagavatula, Le Ronan, Maxwell Bras, Jon Forbes, Jenny Borchardt, Oren Liang, Maarten Etzioni, Yejin Sap, Choi","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kadir, M. R. A. and Selamat, A., 2018] Mohd Rashdan Ab- dul Kadir and Ali Selamat. A Categorization of Runtime Norm Synthesis in Normative Multi-Agent Systems. In 2018 IEEE Conference on e-Learning, e-Management and e-Services (IC3e): 128-133. IEEE. 2018.","title":"A Categorization of Runtime Norm Synthesis in Normative Multi-Agent Systems","authors":"M R A Kadir, A Selamat, Abdul Mohd Rashdan, Ali Kadir, Selamat","year":"2018","venue":"IEEE Conference on e-Learning, e-Management and e-Services (IC3e)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Levy, K. M., 2019] Ken M. Levy. Normative Ignorance: A Critical Connection Between the Insanity and Mistake of Law Defenses. Fla. St. UL Rev., 47, 411. 2019.","title":"Normative Ignorance: A Critical Connection Between the Insanity and Mistake of Law Defenses","authors":"Ken M Levy","year":"2019","venue":"Fla. St. UL Rev","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[McNamara, P., 1996] Paul McNamara. Making Room for Going Beyond the Call. Mind, 105(419): 415-450. doi.org/10.1093/mind/105.419.415. 1996.","title":"Making Room for Going Beyond the Call","authors":"P Mcnamara","year":"1996","venue":"Mind","doi":".org/10.1093/mind/105.419.415.1996","url":null,"internal_url":null},{"position":17,"raw":"[Meltzoff, A.N., 2007] Andrew N. Meltzoff. 'Like me': a foundation for social cognition. Dev Sci. Jan;10(1):126- 34. doi: 10.1111/j.1467-7687.2007.00574.x. PMID: 17181710; PMCID: PMC1852489. 2007.","title":"Like me': a foundation for social cognition","authors":"Andrew N Meltzoff","year":null,"venue":"Dev Sci. Jan","doi":"10.1111/j.1467-7687.2007.00574.x","url":null,"internal_url":null},{"position":18,"raw":"[Olson, T. and Forbus, K., 2021] Taylor Olson and Kenneth Forbus. Learning Norms via Natural Language Teach- ings. In Proceedings of the 9th Annual Conference on Ad- vances in Cognitive Systems. 2021.","title":"Learning Norms via Natural Language Teachings","authors":"T Olson, K Forbus, Taylor Olson, Kenneth Forbus","year":"2021","venue":"Proceedings of the 9th Annual Conference on Advances in Cognitive Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Olson, T. and Forbus, K., 2023] Taylor Olson and Kenneth Forbus. Mitigating Adversarial Norm Training With Moral Axioms. In Proceedings of AAAI-23. 2023.","title":"Mitigating Adversarial Norm Training With Moral Axioms","authors":"T Olson, K Forbus, Taylor Olson, Kenneth Forbus","year":"2023","venue":"Proceedings of AAAI-23","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Olson, T., 2024] Towards Unifying the Descriptive and Pre- scriptive for Machine Ethics. In P. Wu, M. Salpukas, H.F. Wu, S. Ellsworth (Eds.), Trolley Crash: Approaching Key Metrics for Ethical AI Practitioners, Researchers, and Pol- icy Makers. (Chapter 5). Elsevier. 2024.","title":"Towards Unifying the Descriptive and Prescriptive for Machine Ethics","authors":"T Olson","year":"2024","venue":"Trolley Crash: Approaching Key Metrics for Ethical AI Practitioners, Researchers, and Policy Makers","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Premaratne, K. et al., 2009] Kamal Premaratne, Manohar N. Murthi, Jinsong Zhang, Matthias Scheutz, and Peter H. Bauer. A Dempster-Shafer theoretic conditional approach to evidence updating for fusion of hard and soft data. In Proceedings of the 12th International Conference on In- formation Fusion (pp. 2122-2129). 2009.","title":"A Dempster-Shafer theoretic conditional approach to evidence updating for fusion of hard and soft data","authors":"Kamal Premaratne, N Manohar, Jinsong Murthi, Matthias Zhang, Peter H Scheutz, Bauer","year":"2009","venue":"Proceedings of the 12th International Conference on Information Fusion","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Rabkina, I. et al., 2017] Irina Rabkina, Clifton McFate, Ken- neth D. Forbus, and Christian Hoyos. Towards a Compu- tational Analogical Theory of Mind. In Proceedings of the 39th Annual Conference of the Cognitive Science Soci- ety, 2949-2954. 2017","title":"Towards a Computational Analogical Theory of Mind","authors":"Irina Rabkina, Clifton Mcfate, Kenneth D Forbus, Christian Hoyos","year":"2017","venue":"Proceedings of the 39th Annual Conference of the Cognitive Science Society","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Ross, A., 1941] Alf Ross. Imperatives and Logic (in Discus- sions). Theoria, 7(1): 53-71. doi:10.1111/j.1755- 2567.1941.tb00034.x. 1941.","title":"Imperatives and Logic (in Discussions)","authors":"Alf Ross","year":null,"venue":"Theoria","doi":"10.1111/j.1755-2567.1941.tb00034.x.1941","url":null,"internal_url":null},{"position":24,"raw":"[Santos, J. S. et al., 2017] Santos, Jéssica S., Jean O. Zahn, Eduardo A. Silvestre, Viviane T. Silva, and Wamberto W. Vasconcelos. \"Detection and resolution of normative con- flicts in multi-agent systems: a literature survey.\" Auton- omous agents and multi-agent systems 31: 1236-1282. 2017.","title":"Detection and resolution of normative conflicts in multi-agent systems: a literature survey","authors":"Jéssica S Santos, O Jean, Eduardo A Zahn, Silvestre, T Viviane, Wamberto W Silva, Vasconcelos","year":"2017","venue":"Autonomous agents and multi-agent systems","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Sarathy, V. et al., 2017] Vasanth Sarathy, Matthias Scheutz, Yoed N. Kenett, Mowafak Allaham, Joseph L. Auster- weil, and Bertram F. Malle. Mental representations and computational modeling of context-specific human norm systems. In Proceedings of the 39th Annual Meeting of the Cognitive Science Society. 2017.","title":"Mental representations and computational modeling of context-specific human norm systems","authors":"Vasanth Sarathy, Matthias Scheutz, Yoed N Kenett, Mowafak Allaham, Joseph L Austerweil, Bertram F Malle","year":"2017","venue":"Proceedings of the 39th Annual Meeting of the Cognitive Science Society","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Sentz, K. & Ferson, S., 2002] Karl Sentz and Scott Ferson. \"Combination of Evidence in Dempster-Shafer Theory\". United States. doi.org/10.2172/800792. 2002.","title":"Combination of Evidence in Dempster-Shafer Theory","authors":"K Sentz, S Ferson, Karl Sentz, Scott Ferson","year":"2002","venue":null,"doi":".org/10.2172/800792.2002","url":null,"internal_url":null},{"position":27,"raw":"[Shafer, G., 1976] Glenn Shafer. A Mathematical Theory of Evidence. Princeton, New Jersey: Princeton University Press. 1976.","title":"A Mathematical Theory of Evidence","authors":"Glenn Shafer","year":"1976","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Shafer, G., 1981] Glenn Shafer. Constructive probability. Synthese, 1-60. 1981.","title":"Constructive probability","authors":"G Shafer","year":"1981","venue":"Synthese","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Slobogin, C., 2008] Christopher Slobogin. Experts, mental states, and acts. Seton Hall L. Rev., 38, 1009. 2008.","title":"Experts, mental states, and acts","authors":"Christopher Slobogin","year":"2008","venue":"Seton Hall L. Rev","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wen, R. et al., 2020] Ruchen Wen, Mohammed Aun Sid- diqui, and Tom Williams. Dempster-Shafer Theoretic Learning of Indirect Speech Act Comprehension Norms. Proceedings of the AAAI Conference on Artificial Intel- ligence (pp. 10410-10417). 2020.","title":"Dempster-Shafer Theoretic Learning of Indirect Speech Act Comprehension Norms","authors":"R Wen","year":"2020","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zadeh, L.A., 1984] Lotfi A. Zadeh. Review of a mathemati- cal theory of evidence. AI magazine, 5(3), 81-81. 1984.","title":"Review of a mathematical theory of evidence","authors":"A Lotfi, Zadeh","year":"1984","venue":"AI magazine","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}