{"type":"ScholarlyArticle","title":"Teaching AI Agents Ethical Values Using Reinforcement Learning and Policy Orchestration","authors":[{"name":"Ritesh Noothigattu","slug":"ritesh-noothigattu","given":"Ritesh","family":"Noothigattu","affiliation":"Carnegie Mellon University , Pittsburgh , PA , USA"},{"name":"Djallel Bouneffouf","slug":"djallel-bouneffouf","given":"Djallel","family":"Bouneffouf","affiliation":"IBM Research , Yorktown Heights , NY , USA"},{"name":"Nicholas Mattei","slug":"nicholas-mattei","given":"Nicholas","family":"Mattei","affiliation":"Tulane University , New Orleans , LA , USA"},{"name":"Rachita Chandra","slug":"rachita-chandra","given":"Rachita","family":"Chandra","affiliation":"IBM Research , Cambridge , MA , USA"},{"name":"Piyush Madan","slug":"piyush-madan","given":"Piyush","family":"Madan","affiliation":"IBM Research , Cambridge , MA , USA"},{"name":"Kush R. Varshney","slug":"kush-r-varshney","given":"Kush R.","family":"Varshney","affiliation":"IBM Research , Yorktown Heights , NY , USA"},{"name":"Murray Campbell","slug":"murray-campbell","given":"Murray","family":"Campbell","affiliation":"IBM Research , Yorktown Heights , NY , USA"},{"name":"Moninder Singh","slug":"moninder-singh","given":"Moninder","family":"Singh","affiliation":"IBM Research , Yorktown Heights , NY , USA"},{"name":"Francesca Rossi","slug":"francesca-rossi","given":"Francesca","family":"Rossi","affiliation":"IBM Research , Yorktown Heights , NY , USA"}],"abstract":"Autonomous cyber-physical agents play an increasingly large role in our lives. To ensure that they behave in ways aligned with the values of society, we must develop techniques that allow these agents to not only maximize their reward in an environment, but also to learn and follow the implicit constraints of society.  \r\nWe detail a novel approach that uses inverse reinforcement learning to learn a set of unspecified constraints from demonstrations and reinforcement learning to learn to maximize environmental rewards.  A contextual bandit-based orchestrator then picks between the two policies: constraint-based and environment reward-based. The contextual bandit orchestrator allows the agent to mix policies in novel ways, taking the best actions from either a reward-maximizing or constrained policy.  In addition, the orchestrator is transparent on  which policy is being employed at each time step. We test our algorithms using Pac-Man and show that the agent is able to learn to act optimally, act within the demonstrated constraints, and mix these two functions in complex ways.","doi":"10.24963/ijcai.2019/891","url":"https://ijcai.org/proceedings/2019/891","pdf":"https://ijcai.org/proceedings/2019/0891.pdf","markdown":"https://ijcai.org/proceedings/2019/891.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/891","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Journal track","track":"Journal track","number":891,"pages":{"first":6377,"last":6381},"keywords":["Machine Learning Applications: Applications of Reinforcement Learning"],"videos":[],"references":[{"position":1,"raw":"[Abbeel and Ng, 2004] P. Abbeel and A. Y. Ng. Apprentice- ship learning via inverse reinforcement learning. In Pro- ceedings of the 21st International Conference on Machine Learning (ICML), 2004.","title":"Apprenticeship learning via inverse reinforcement learning","authors":"P Abbeel, A Y Ng","year":"2004","venue":"Proceedings of the 21st International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Allen et al., 2005] C. Allen, I. Smit, and W. Wallach. Ar- tificial morality: Top-down, bottom-up, and hybrid ap- proaches. Ethics and Information Technology, 7(3):149- 155, 2005.","title":"Artificial morality: Top-down, bottom-up, and hybrid approaches","authors":"C Allen, I Smit, W Wallach","year":"2005","venue":"Ethics and Information Technology","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Anderson and Anderson, 2011] M. Anderson and S. L. An- derson. Machine Ethics. Cambridge University Press, 2011.","title":"Machine Ethics","authors":"M Anderson, S L","year":"2011","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Arnold et al., 2017] T. Arnold, D. Kasenberg, and M. Scheutzs. Value alignment or misalignment - what will keep systems accountable? In AI, Ethics, and Society, Papers from the 2017 AAAI Workshops, 2017.","title":"Value alignment or misalignmentwhat will keep systems accountable?","authors":"T Arnold, D Kasenberg, M Scheutzs","year":"2017","venue":"Papers from the 2017 AAAI Workshops","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Balakrishnan et al., 2018] A. Balakrishnan, D. Bouneffouf, N. Mattei, and F. Rossi. Using contextual bandits with be- havioral constraints for constrained online movie recom- mendation. In Proc. of the 27th Intl. Joint Conference on AI (IJCAI), 2018.","title":"Using contextual bandits with behavioral constraints for constrained online movie recommendation","authors":"A Balakrishnan, D Bouneffouf, N Mattei, F Rossi","year":"2018","venue":"Proc. of the 27th Intl. Joint Conference on AI (IJCAI)","doi":null,"url":null,"internal_url":"/proceedings/2018/843"},{"position":6,"raw":"[Balakrishnan et al., 2019] A. Balakrishnan, D. Bouneffouf, N. Mattei, and F. Rossi. Incorporating behavioral con- straints in online AI systems. In Proc. of the 33rd AAAI Conference on Artificial Intelligence (AAAI), 2019.","title":"Incorporating behavioral constraints in online AI systems","authors":"A Balakrishnan, D Bouneffouf, N Mattei, F Rossi","year":"2019","venue":"Proc. of the 33rd AAAI Conference on Artificial Intelligence (AAAI)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Bertsekas and Tsitsiklis, 1996] D. Bertsekas and J. Tsitsik- lis. Neuro-dynamic programming. Athena Scientific, 1996.","title":"Neuro-dynamic programming","authors":"D Bertsekas, J Tsitsiklis","year":"1996","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Bouneffouf and Rish, 2019] D. Bouneffouf and I. Rish. A survey on practical applications of multi-armed and con- textual bandits. CoRR, abs/1904.10040, 2019.","title":"A survey on practical applications of multi-armed and contextual bandits","authors":"D Bouneffouf, I Rish","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Bouneffouf et al., 2017] D. Bouneffouf, I. Rish, G. A. Cec- chi, and R. Féraud. Context attentive bandits: Contextual bandit with restricted context. In Proc. of the 26th Intl. Joint Conference on AI (IJCAI), pages 1468-1475, 2017.","title":"Context attentive bandits: Contextual bandit with restricted context","authors":"D Bouneffouf, I Rish, G A Cecchi, R Féraud","year":"2017","venue":"Proc. of the 26th Intl. Joint Conference on AI (IJCAI)","doi":null,"url":null,"internal_url":"/proceedings/2017/203"},{"position":10,"raw":"[Leike et al., 2017] J. Leike, M. Martic, V. Krakovna, P.A. Ortega, T. Everitt, A. Lefrancq, L. Orseau, and S. Legg. AI safety gridworlds. arXiv preprint arXiv:1711.09883, 2017.","title":"AI safety gridworlds","authors":"J Leike, M Martic, V Krakovna, P A Ortega, T Everitt, A Lefrancq, L Orseau, S Legg","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Loreggia et al., 2018a] A. Loreggia, N. Mattei, F. Rossi, and K. B. Venable. Preferences and ethical principles in deci- sion making. In Proc. of the 1st AAAI/ACM Conference on AI, Ethics, and Society (AIES), 2018.","title":"Preferences and ethical principles in decision making","authors":"Loreggia","year":"2018","venue":"Proc. of the 1st AAAI/ACM Conference on AI, Ethics, and Society (AIES)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Loreggia et al., 2018b] A. Loreggia, N. Mattei, F. Rossi, and K. B. Venable. Value alignment via tractable prefer- ence distance. In R. V. Yampolskiy, editor, Artificial Intel- ligence Safety and Security, chapter 18. CRC Press, 2018.","title":"Value alignment via tractable preference distance","authors":"Loreggia","year":"2018","venue":"Artificial Intelligence Safety and Security, chapter 18","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Noothigattu et al., 2019] R. Noothigattu, D. Bouneffouf, N. Mattei, R. Chandra, P. Madan, K. Varshney, M. Camp- bell, M. Singh, and F. Rossi. Teaching AI agents ethical values using reinforcement learning and policy orchestra- tion. IBM Journal of Research & Development, 2019. To Appear.","title":"Teaching AI agents ethical values using reinforcement learning and policy orchestration","authors":"R Noothigattu, D Bouneffouf, N Mattei, R Chandra, P Madan, K Varshney, M Campbell, M Singh, F Rossi","year":"2019","venue":"IBM Journal of Research & Development","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Rossi and Mattei, 2019] F. Rossi and N. Mattei. Building ethically bounded AI. In Proc. of the 33rd AAAI Confer- ence on Artificial Intelligence (AAAI), 2019.","title":"Building ethically bounded AI","authors":"F Rossi, N Mattei","year":"2019","venue":"Proc. of the 33rd AAAI Conference on Artificial Intelligence (AAAI)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Russell et al., 2015] S. Russell, D. Dewey, and M. Tegmark. Research priorities for robust and beneficial artificial intel- ligence. AI Magazine, 36(4):105-114, 2015.","title":"Research priorities for robust and beneficial artificial intelligence","authors":"S Russell, D Dewey, M Tegmark","year":"2015","venue":"AI Magazine","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Sen, 1974] A. Sen. Choice, ordering and morality. In S. Körner, editor, Practical Reason. Blackwell, Oxford, 1974.","title":"Choice, ordering and morality","authors":"A Sen","year":"1974","venue":"Practical Reason","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Simonite, 2018] T. Simonite. When bots teach themselves to cheat. Wired Magazine, August 2018.","title":"When bots teach themselves to cheat","authors":"T Simonite","year":"2018","venue":"Wired Magazine","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Ventura and Gates, 2018] D. Ventura and D. Gates. Ethics as aesthetic: A computational creativity approach to ethi- cal behavior. In Proc. Int. Conference on Computational Creativity (ICCC), pages 185-191, 2018.","title":"Ethics as aesthetic: A computational creativity approach to ethical behavior","authors":"D Ventura, D Gates","year":"2018","venue":"Proc. Int. Conference on Computational Creativity (ICCC)","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wallach and Allen, 2008] W. Wallach and C. Allen. Moral Machines: Teaching Robots Right From Wrong. Oxford University Press, 2008.","title":"Moral Machines: Teaching Robots Right From Wrong","authors":"W Wallach, C Allen","year":"2008","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Yu et al., 2018] H. Yu, Z. Shen, C. Miao, C. Leung, V. R. Lesser, and Q. Yang. Building ethics into artificial intel- ligence. In Proc. of the 27th Intl. Joint Conference on AI (IJCAI), pages 5527-5533, 2018.","title":"Building ethics into artificial intelligence","authors":"H Yu, Z Shen, C Miao, C Leung, V R Lesser, Q Yang","year":"2018","venue":"Proc. of the 27th Intl. Joint Conference on AI (IJCAI)","doi":null,"url":null,"internal_url":"/proceedings/2018/779"}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}