{"type":"ScholarlyArticle","title":"Governance by Glass-Box: Implementing Transparent Moral Bounds for AI Behaviour","authors":[{"name":"Andrea Aler Tubella","slug":"andrea-aler-tubella","given":"Andrea","family":"Aler Tubella","affiliation":"Umeå University"},{"name":"Andreas Theodorou","slug":"andreas-theodorou","given":"Andreas","family":"Theodorou","affiliation":"Umeå University"},{"name":"Frank Dignum","slug":"frank-dignum","given":"Frank","family":"Dignum","affiliation":"Umeå University"},{"name":"Virginia Dignum","slug":"virginia-dignum","given":"Virginia","family":"Dignum","affiliation":"Umeå University"}],"abstract":"Artificial Intelligence (AI) applications are being used to predict and assess behaviour in multiple domains which directly affect human well-being. However, if AI is to improve people’s lives, then\r\npeople must be able to trust it, by being able to understand what the system is doing and why. Although transparency is often seen as the requirement in this case, realistically it might not always\r\nbe possible, whereas the need to ensure that the system operates within set moral bounds remains. \r\nIn this paper, we present an approach to evaluate the moral bounds of an AI system based on the monitoring of its inputs and outputs. We place a ‘Glass-Box’ around the system by mapping moral values into explicit verifiable norms that constrain inputs and outputs, in such a way that if these remain within the box we can guarantee that the system adheres to the value. The focus on inputs and outputs allows for the verification and comparison of vastly different intelligent systems; from deep neural networks to agent-based systems.\r\nThe explicit transformation of abstract moral values into concrete norms brings great benefits in terms of explainability; stakeholders know exactly how the system is interpreting and employing relevant abstract moral human values and calibrate their trust accordingly. Moreover, by operating at a higher level we can check the compliance of the system with different interpretations of the same value.","doi":"10.24963/ijcai.2019/802","url":"https://ijcai.org/proceedings/2019/802","pdf":"https://ijcai.org/proceedings/2019/0802.pdf","markdown":"https://ijcai.org/proceedings/2019/802.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/802","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"AI for Improving Human Well-being","track":"AI for Improving Human Well-being","number":802,"pages":{"first":5787,"last":5793},"keywords":["Special Track on AI for Improving Human-Well Being: AI ethics (Special Track on AI and Human Wellbeing)","Special Track on AI for Improving Human-Well Being: AI safety (Special Track on AI and Human Wellbeing)"],"videos":[],"references":[{"position":1,"raw":"[Aldewereld et al., 2010] Huib Aldewereld, Sergio Álvarez- Napagao, Frank Dignum, and Javier Vázquez-Salceda. Making Norms Concrete. In Proc. of 9th Int. Conf. on Au- tonomous Agents and Multiagent Systems (AAMAS 2010), pages 807-814, Toronto, Canada, 2010.","title":"Making Norms Concrete","authors":"Huib Aldewereld, Sergio Álvarez-Napagao, Frank Dignum, Javier Vázquez-Salceda","year":"2010","venue":"Proc. of 9th Int. Conf. on Autonomous Agents and Multiagent Systems (AAMAS 2010)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Aldewereld et al., 2014] Huib Aldewereld, Virginia Dignum, and Yao-hua Tan. Design for Values in Soft- ware Development, pages 1-12. Springer Netherlands, Dordrecht, 2014.","title":"Design for Values in Software Development","authors":"Huib Aldewereld, Virginia Dignum, Yao-Hua Tan","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Ananny and Crawford, 2018] M. Ananny and K. Crawford. Seeing without knowing: Limitations of the transparency ideal and its application to algorithmic accountability. New Media & Society, 20(3):973-989, 2018.","title":"Seeing without knowing: Limitations of the transparency ideal and its application to algorithmic accountability","authors":"M Ananny, K Crawford","year":"2018","venue":"New Media & Society","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bryson and Theodorou, 2019] J. Bryson, Joanna and An- dreas Theodorou. How Society Can Maintain Human- Centric Artificial Intelligence. In Human-centered digi- talization and services. Springer, 2019.","title":"How Society Can Maintain Human-Centric Artificial Intelligence. In Human-centered digitalization and services","authors":"Theodorou Bryson, J Bryson","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bryson and Winfield, 2017] Joanna J. Bryson and Alan Winfield. Standardizing Ethical Design for Artificial Intel- ligence and Autonomous Systems. Computer, 50(5):116- 119, 5 2017.","title":"Standardizing Ethical Design for Artificial Intelligence and Autonomous Systems","authors":"Joanna J Bryson, Alan Winfield","year":"2017","venue":"Computer","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Cranefield et al., 2017] S. Cranefield, M. Winikoff, V. Dignum, and F. Dignum. No pizza for you: Value- based plan selection in BDI agents. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, pages 178-184. ijcai.org, 2017.","title":"No pizza for you: Valuebased plan selection in BDI agents","authors":"S Cranefield, M Winikoff, V Dignum, F Dignum","year":"2017","venue":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dignum, 2017] Virginia Dignum. Responsible autonomy. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence (IJCAI'2017), pages 4698-4704, 2017.","title":"Virginia Dignum. Responsible autonomy","authors":"Dignum","year":"2017","venue":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence (IJCAI'2017)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dignum, 2018] Virginia Dignum. Ethics in artificial intelli- gence: introduction to the special issue. Ethics and Infor- mation Technology, 20(1):1-3, 3 2018.","title":"Virginia Dignum. Ethics in artificial intelligence: introduction to the special issue","authors":"Dignum","year":"2018","venue":"Ethics and Information Technology","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Dwork et al., 2012] Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard S. Zemel. Fairness through awareness. In Innovations in Theoretical Com- puter Science 2012, Cambridge, MA, USA, January 8-10, 2012, pages 214-226, 2012.","title":"Fairness through awareness","authors":"Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, Richard S Zemel","year":"2012","venue":"Innovations in Theoretical Computer Science 2012","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Floridi et al., 2018] L. Floridi, J. Cowls, M. Beltram- etti, R. Chatila, P. Chazerand, V. Dignum, C. Luetge, R. Madelin, U. Pagallo, F. Rossi, B. Schafer, P. Valcke, and E. Vayena. Ai4people-an ethical framework for a good ai society: Opportunities, risks, principles, and rec- ommendations. Minds and Machines, 28(4):689-707, Dec 2018.","title":"Ai4people-an ethical framework for a good ai society: Opportunities, risks, principles, and recommendations","authors":"L Floridi, J Cowls, M Beltrametti, R Chatila, P Chazerand, V Dignum, C Luetge, R Madelin, U Pagallo, F Rossi, B Schafer, P Valcke, E Vayena","year":"2018","venue":"Minds and Machines","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Floridi, 2018] Luciano Floridi. Soft ethics and the gover- nance of the digital. Phil. & Tech., 31(1):1-8, 2018.","title":"Luciano Floridi. Soft ethics and the governance of the digital","authors":"Floridi","year":"2018","venue":"Phil. & Tech","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Friedman et al., 2006] Batya Friedman, Peter Kahn, and Alan Borning. Value sensitive design and information systems. Advances in Management Information Systems, 6:348 -372, 2006.","title":"Value sensitive design and information systems","authors":"Batya Friedman, Peter Kahn, Alan Borning","year":"2006","venue":"Advances in Management Information Systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Greene et al., 2019] D. Greene, A. Hoffmann, and L. Stark. Better, nicer, clearer, fairer: A critical assessment of the movement for ethical artificial intelligence and machine learning. In Proceedings of the 52nd Hawaii International Conference on System Sciences, 2019.","title":"Better, nicer, clearer, fairer: A critical assessment of the movement for ethical artificial intelligence and machine learning","authors":"D Greene, A Hoffmann, L Stark","year":"2019","venue":"Proceedings of the 52nd Hawaii International Conference on System Sciences","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Grossi et al., 2006a] Davide Grossi, Huib Aldewereld, Javier Vázquez-Salceda, and Frank Dignum. Ontologi- cal aspects of the implementation of norms in agent-based electronic institutions. Computational & Mathematical Organization Theory, 12(2-3):251-275, 10 2006.","title":"Ontological aspects of the implementation of norms in agent-based electronic institutions","authors":"Grossi","year":"2006","venue":"Computational & Mathematical Organization Theory","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Grossi et al., 2006b] Davide Grossi, John Jules Ch Meyer, and Frank Dignum. Counts-as: Classification or consti- tution? An answer using modal logic. In LNAI, volume 4048, pages 115-130, 2006.","title":"Counts-as: Classification or constitution? An answer using modal logic","authors":"Grossi","year":"2006","venue":"LNAI","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Grossi, 2007] D. Grossi. Designing Invisible Handcuffs: Formal Investigations in Institutions and Organizations for Multi-agent Systems. PhD thesis, Universiteit Utrecht, 2007.","title":"Designing Invisible Handcuffs: Formal Investigations in Institutions and Organizations for Multi-agent Systems","authors":"D Grossi","year":"2007","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[ISO, 2015] ISO. Quality management systems -require- ments. Standard, International Organization for Standard- ization, Geneva, CH, September 2015.","title":"ISO. Quality management systems -requirements. Standard, International Organization for Standardization","authors":"Iso","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Jones and Sergot, 1995] Andrew J. I. Jones and Marek Ser- got. A Formal Characterisation of Institutionalised Power. Logic Journal of IGPL, 4(3):427-443, 6 1995.","title":"A Formal Characterisation of Institutionalised Power","authors":"J I Andrew, Marek Jones, Sergot","year":"1995","venue":"Logic Journal of IGPL","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Lepri et al., 2018] B. Lepri, N. Oliver, E. Letouzé, A. Pent- land, and P. Vinck. Fair, transparent, and accountable al- gorithmic decision-making processes. Philosophy & Tech- nology, 31(4):611-627, 2018.","title":"Fair, transparent, and accountable algorithmic decision-making processes","authors":"B Lepri, N Oliver, E Letouzé, A Pentland, P Vinck","year":"2018","venue":"Philosophy & Technology","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Pedreschi et al., 2008] Dino Pedreschi, Salvatore Ruggieri, and Franco Turini. Discrimination-aware data mining. In Proceedings of the 14th ACM SIGKDD International Con- ference on Knowledge Discovery and Data Mining, Las Vegas, Nevada, USA, August 24-27, 2008, pages 560-568, 2008.","title":"Discrimination-aware data mining","authors":"Dino Pedreschi, Salvatore Ruggieri, Franco Turini","year":"2008","venue":"Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Searle, 1995] J Searle. The construction of social reality. Free Press, New York, 1995.","title":"The construction of social reality","authors":"Searle","year":"1995","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Theodorou et al., 2017] Andreas Theodorou, Robert H. Wortham, and Joanna J. Bryson. Designing and im- plementing transparency for real time inspection of au- tonomous robots. Connection Science, 29(3):230-241, 7 2017.","title":"Designing and implementing transparency for real time inspection of autonomous robots","authors":"Andreas Theodorou, Robert H Wortham, Joanna J Bryson","year":"2017","venue":"Connection Science","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Turiel, 2002] E. Turiel. The culture of morality: Social de- velopment, context, and conflict. Cambridge University Press, 2002.","title":"The culture of morality: Social development, context, and conflict","authors":"E Turiel","year":"2002","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Van de Poel, 2013] Ibo Van de Poel. Translating values into design requirements. In Philosophy and engineering: Re- flections on practice, principles and process, pages 253- 266. Springer, 2013.","title":"Ibo Van de Poel. Translating values into design requirements","authors":"Poel Van De","year":"2013","venue":"Philosophy and engineering: Reflections on practice, principles and process","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[van den Hoven, 2005] Jeroen van den Hoven. Design for values and values for design. Information Age +, Journal of the Australian Computer Society, 7(2):4-7, 2005.","title":"Design for values and values for design. Information Age +","authors":"Van Den Hoven ; Jeroen Van Den Hoven","year":"2005","venue":"Journal of the Australian Computer Society","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Vázquez-Salceda et al., 2007] Javier Vázquez-Salceda, Huib Aldewereld, Davide Grossi, and Frank Dignum. From human regulations to regulated software agents' behavior. Artificial Intelligence and Law, 16(1):73-87, 3 2007.","title":"From human regulations to regulated software agents' behavior","authors":"Javier Vázquez-Salceda, Huib Aldewereld, Davide Grossi, Frank Dignum","year":"2007","venue":"Artificial Intelligence and Law","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}