{"type":"ScholarlyArticle","title":"Explanation-Guided Reward Alignment","authors":[{"name":"Saaduddin Mahmud","slug":"saaduddin-mahmud","given":"Saaduddin","family":"Mahmud","affiliation":"University of Massachusetts Amherst , USA"},{"name":"Sandhya Saisubramanian","slug":"sandhya-saisubramanian","given":"Sandhya","family":"Saisubramanian","affiliation":"Oregon State University , USA"},{"name":"Shlomo Zilberstein","slug":"shlomo-zilberstein","given":"Shlomo","family":"Zilberstein","affiliation":"University of Massachusetts Amherst , USA"}],"abstract":"Agents often need to infer a reward function from observations to learn desired behaviors. However, agents may infer a reward function that does not align with the original intent because there can be multiple reward functions consistent with its observations. Operating based on such misaligned rewards can be risky. Furthermore, black-box representations make it difficult to verify the learned rewards and prevent harmful behavior. We present a framework for verifying and improving reward alignment using explanations and show how explanations can help detect misalignment and reveal failure cases in novel scenarios. The problem is formulated as inverse reinforcement learning from ranked trajectories. Verification tests created from the trajectory dataset are used to iteratively validate and improve reward alignment. The agent explains its learned reward and a tester signals whether the explanation passes the test. In cases where the explanation fails, the agent offers alternative explanations to gather feedback, which is then used to improve the learned reward. We analyze the efficiency of our approach in improving reward alignment using different types of explanations and demonstrate its effectiveness in five domains.","doi":"10.24963/ijcai.2023/53","url":"https://ijcai.org/proceedings/2023/53","pdf":"https://ijcai.org/proceedings/2023/0053.pdf","markdown":"https://ijcai.org/proceedings/2023/53.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/53","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"AI Ethics, Trust, Fairness","track":"Main Track","number":53,"pages":{"first":473,"last":482},"keywords":["Machine Learning: ML: Reinforcement learning","AI Ethics, Trust, Fairness: ETF: Safety and robustness","AI Ethics, Trust, Fairness: ETF: Explainability and interpretability"],"videos":[],"references":[{"position":1,"raw":"[Abbeel and Ng, 2004] Pieter Abbeel and Andrew Y. Ng. Apprenticeship learning via inverse reinforcement learn- ing. In Proceedings of the 21st International Conference on Machine learning, 2004.","title":"Apprenticeship learning via inverse reinforcement learning","authors":"Abbeel, ; Ng, Pieter Abbeel, Andrew Y Ng","year":"2004","venue":"Proceedings of the 21st International Conference on Machine learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Alqaraawi et al., 2020] Ahmed Alqaraawi, M. Schuessler, Philipp Weiß, Enrico Costanza, and Nadia Bianchi- Berthouze. Evaluating saliency map explanations for con- volutional neural networks: A user study. In Proceedings of the 25th International Conference on Intelligent User Interfaces, pages 275-285, 2020.","title":"Evaluating saliency map explanations for convolutional neural networks: A user study","authors":"M Ahmed Alqaraawi, Philipp Schuessler, Enrico Weiß, Nadia Costanza, Bianchi-Berthouze","year":"2020","venue":"Proceedings of the 25th International Conference on Intelligent User Interfaces","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Amir et al., 2019] Ofra Amir, Finale Doshi-Velez, and David Sarne. Summarizing agent strategies. Autonomous Agents and Multi-Agent Systems, 33:628-644, 2019.","title":"Summarizing agent strategies","authors":"Ofra Amir, Finale Doshi-Velez, David Sarne","year":"2019","venue":"Autonomous Agents and Multi-Agent Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Argall et al., 2009] Brenna D Argall, Sonia Chernova, Manuela Veloso, and Brett Browning. A survey of robot learning from demonstration. Robotics and Autonomous Systems, 57(5):469-483, 2009.","title":"A survey of robot learning from demonstration","authors":"Sonia Brenna D Argall, Manuela Chernova, Brett Veloso, Browning","year":"2009","venue":"Robotics and Autonomous Systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Basu et al., 2018] Chandrayee Basu, Mukesh Singhal, and Anca D. Dragan. Learning from richer human guid- ance: Augmenting comparison-based learning with feature queries. In Proceedings of the 13th International Confer- ence on Human-Robot Interaction, pages 132-140, 2018.","title":"Learning from richer human guidance: Augmenting comparison-based learning with feature queries","authors":"Basu","year":"2018","venue":"Proceedings of the 13th International Conference on Human-Robot Interaction","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Brown and Niekum, 2018] Daniel S. Brown and Scott Niekum. Efficient Probabilistic Performance Bounds for Inverse Reinforcement Learning. In AAAI Conference on Artificial Intelligence, 2018.","title":"Efficient Probabilistic Performance Bounds for Inverse Reinforcement Learning","authors":"Daniel S Brown, Scott Niekum","year":"2018","venue":"AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Brown et al., 2019a] Daniel S. Brown, Wonjoon Goo, and Scott Niekum. Better-than-demonstrator imitation learn- ing via automaticaly-ranked demonstrations. In Confer- ence on Robot Learning, 2019.","title":"Better-than-demonstrator imitation learning via automaticaly-ranked demonstrations","authors":"Brown","year":"2019","venue":"Conference on Robot Learning","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Brown et al., 2019b] Daniel S. Brown, Wonjoon Goo, Na- garajan Prabhat, and Scott Niekum. Extrapolating be- yond suboptimal demonstrations via inverse reinforcement learning from observations. In Proceedings of the 36th In- ternational Conference on Machine Learning, pages 783- 792, 2019.","title":"Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations","authors":"Brown","year":"2019","venue":"Proceedings of the 36th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Brown et al., 2020] Daniel S. Brown, Scott Niekum, Rus- sell Coleman, and Ravi Srinivasan. Safe imitation learn- ing via fast bayesian reward inference from preferences. In Proceedings of the 37th International Conference on Ma- chine Learning, pages 1165-1177, 2020.","title":"Safe imitation learning via fast bayesian reward inference from preferences","authors":"S Daniel, Scott Brown, Russell Niekum, Ravi Coleman, Srinivasan","year":"2020","venue":"Proceedings of the 37th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Brown et al., 2021] Daniel S. Brown, Jordan Jack Schnei- der, and Scott Niekum. Value alignment verification. In Proceedings of the 38th International Conference on Ma- chine Learning, pages 1105-1115, 2021.","title":"Value alignment verification","authors":"S Daniel, Jordan Jack Brown, Scott Schneider, Niekum","year":"2021","venue":"Proceedings of the 38th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Chakraborti et al., 2017] Tathagata Chakraborti, Sarath Sreedharan, Yu Zhang, and Subbarao Kambhampati. Plan explanations as model reconciliation: Moving beyond ex- planation as soliloquy. arXiv preprint arXiv:1701.08317, 2017.","title":"Plan explanations as model reconciliation: Moving beyond explanation as soliloquy","authors":"Tathagata Chakraborti, Sarath Sreedharan, Yu Zhang, Subbarao Kambhampati","year":"2017","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2017/23"},{"position":12,"raw":"[Dietterich, 2017] Thomas G. Dietterich. Steps toward ro- bust artificial intelligence. AI Magazine, 38(3):3-24, 2017.","title":"Steps toward robust artificial intelligence","authors":"G Thomas, Dietterich","year":"2017","venue":"AI Magazine","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Guan et al., 2021] Lin Guan, Mudit Verma, Sihang Guo, Ruohan Zhang, and Subbarao Kambhampati. Widening the pipeline in human-guided reinforcement learning with explanation and context-aware data augmentation. In Pro- ceedings of the Annual Conference on Neural Information Processing Systems, pages 21885-21897, 2021.","title":"Widening the pipeline in human-guided reinforcement learning with explanation and context-aware data augmentation","authors":"Lin Guan, Mudit Verma, Sihang Guo, Ruohan Zhang, Subbarao Kambhampati","year":"2021","venue":"Proceedings of the Annual Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hadfield-Menell et al., 2017] Dylan Hadfield-Menell, Smitha Milli, Pieter Abbeel, Stuart J. Russell, and Anca Dragan. Inverse reward design. In Advances in Neural Information Processing Systems, 2017.","title":"Inverse reward design","authors":"Hadfield-Menell","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Huang et al., 2018] Sandy H. Huang, Kush Bhatia, Pieter Abbeel, and Anca D. Dragan. Establishing appropriate trust via critical states. In Proceedings of the IEEE/RSJ In- ternational Conference on Intelligent Robots and Systems, pages 3929-3936, 2018.","title":"Establishing appropriate trust via critical states","authors":"Sandy H Huang, Kush Bhatia, Pieter Abbeel, Anca D Dragan","year":"2018","venue":"Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Langosco et al., 2022] Lauro Langosco, Jack Koch, Lee Sharkey, Jacob Pfau, and David Krueger. Goal misgen- eralization in deep reinforcement learning. In Proceedings of the 39th International Conference on Machine Learn- ing, pages 12004-12019, 2022.","title":"Goal misgeneralization in deep reinforcement learning","authors":"Lauro Langosco, Jack Koch, Lee Sharkey, Jacob Pfau, David Krueger","year":"2022","venue":"Proceedings of the 39th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Leike et al., 2017] Jan Leike, Miljan Martic, Victoria Krakovna, Pedro A. Ortega, Tom Everitt, Andrew Lefrancq, Laurent Orseau, and Shane Legg. Ai safety grid- worlds. ArXiv, abs/1711.09883, 2017.","title":"Ai safety gridworlds","authors":"Leike","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Molnar, 2022] Christoph Molnar. Interpretable Machine Learning: A Guide For Making Black Box Models Explain- able. Lulu.com, 2022.","title":"Interpretable Machine Learning: A Guide For Making Black Box Models Explainable","authors":"Christoph Molnar","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Nashed et al., 2022] Samer B. Nashed, Saaduddin Mah- mud, Claudia V. Goldman, and Shlomo Zilberstein. A unifying framework for causal explanation of sequential decision making. ArXiv, abs/2205.15462, 2022.","title":"A unifying framework for causal explanation of sequential decision making","authors":"B Samer, Saaduddin Nashed, Claudia V Mahmud, Shlomo Goldman, Zilberstein","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Ng and Russell, 2000] Andrew Y. Ng and Stuart J. Russell. Algorithms for inverse reinforcement learning. In Pro- ceedings of the 17th International Conference on Machine Learning, pages 663-670, 2000.","title":"Algorithms for inverse reinforcement learning","authors":"Andrew Y Ng, Stuart J Russell","year":"2000","venue":"Proceedings of the 17th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Palan et al., 2019] Malayandi Palan, Nicholas C. Landolfi, Gleb Shevchuk, and Dorsa Sadigh. Learning reward func- tions by integrating human demonstrations and prefer- ences. In Proceedings of Robotics: Science and Systems, June 2019.","title":"Learning reward functions by integrating human demonstrations and preferences","authors":"Palan","year":"2019","venue":"Proceedings of Robotics: Science and Systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Ramachandran and Amir, 2007] Deepak Ramachandran and Eyal Amir. Bayesian inverse reinforcement learning. In Proceedings of the 20th International Joint Conference on Artifical intelligence, pages 2586-2591, 2007.","title":"Bayesian inverse reinforcement learning","authors":"Deepak Ramachandran, Eyal Amir","year":"2007","venue":"Proceedings of the 20th International Joint Conference on Artifical intelligence","doi":null,"url":null,"internal_url":"/proceedings/2007/416"},{"position":23,"raw":"[Ribeiro et al., 2016] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. \"Why should I trust you?\" Explain- ing the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowl- edge Discovery and Data Mining, pages 1135-1144, 2016.","title":"Why should I trust you?\" Explaining the predictions of any classifier","authors":"Marco Tulio Ribeiro, Sameer Singh, Carlos Guestrin","year":"2016","venue":"Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Saisubramanian et al., 2022] Sandhya Saisubramanian, Shlomo Zilberstein, and Ece Kamar. Avoiding negative side effects due to incomplete knowledge of AI systems. AI Magazine, 42(4):62-71, 2022.","title":"Avoiding negative side effects due to incomplete knowledge of AI systems","authors":"Sandhya Saisubramanian, Shlomo Zilberstein, Ece Kamar","year":"2022","venue":"AI Magazine","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Settles, 2012] Burr Settles. Active learning. Synthesis Lectures on Artificial Intelligence and Machine Learning, 6(1):1-114, 2012.","title":"Burr Settles. Active learning","authors":"Settles","year":"2012","venue":"Synthesis Lectures on Artificial Intelligence and Machine Learning","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Simonyan et al., 2014] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. CoRR, abs/1312.6034, 2014.","title":"Deep inside convolutional networks: Visualising image classification models and saliency maps","authors":"Karen Simonyan, Andrea Vedaldi, Andrew Zisserman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Sohn, 2016] Kihyuk Sohn. Improved deep metric learning with multi-class N-pair loss objective. In Proceedings of the Annual Conference on Neural Information Processing Systems, pages 1849-1857, 2016.","title":"Improved deep metric learning with multi-class N-pair loss objective","authors":"Kihyuk Sohn, Sohn","year":"2016","venue":"Proceedings of the Annual Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Sutton and Barto, 1998] Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. MIT Press, 1998.","title":"Reinforcement Learning: An Introduction","authors":"Richard S Sutton, Andrew G Barto","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Tabrez et al., 2020] Aaquib Tabrez, Shivendra Agrawal, and Bradley Hayes. Explanation-based reward coaching to im- prove human performance via reinforcement learning. In Proceedings of the 14th ACM/IEEE International Confer- ence on Human-Robot Interaction, pages 249-257, 2020.","title":"Explanation-based reward coaching to improve human performance via reinforcement learning","authors":"Tabrez","year":"2020","venue":"Proceedings of the 14th ACM/IEEE International Conference on Human-Robot Interaction","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Tayyub et al., 2022] Jawad Tayyub, Muhammad Sarmad, and Nicolas Schönborn. Explaining deep neural networks for point clouds using gradient-based visualisations. arXiv preprint arXiv:2207.12984, 2022.","title":"Explaining deep neural networks for point clouds using gradient-based visualisations","authors":"Jawad Tayyub, Muhammad Sarmad, Nicolas Schönborn","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Wray and Zilberstein, 2016] Kyle Hollins Wray and Shlomo Zilberstein. A POMDP formulation of proactive learning. In Proceedings of the 30th AAAI Conference on Artificial Intelligence, pages 3202-3208, 2016.","title":"A POMDP formulation of proactive learning","authors":"Zilberstein Wray, Kyle Hollins Wray, Shlomo Zilberstein","year":"2016","venue":"Proceedings of the 30th AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Ziebart et al., 2008] Brian D. Ziebart, Andrew L. Maas, J. Andrew Bagnell, and Anind K. Dey. Maximum entropy inverse reinforcement learning. In Proceedings of the 23rd AAAI Conference on Artificial Intelligence, pages 1433- 1438, 2008.","title":"Maximum entropy inverse reinforcement learning","authors":"Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey","year":"2008","venue":"Proceedings of the 23rd AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zilberstein, 2015] Shlomo Zilberstein. Building strong semi-autonomous systems. In Proceedings of the 29th AAAI Conference on Artificial Intelligence, pages 4088- 4092, 2015.","title":"Building strong semi-autonomous systems","authors":"Shlomo Zilberstein","year":"2015","venue":"Proceedings of the 29th AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}