{"type":"ScholarlyArticle","title":"Local and Global Explanations of Agent Behavior: Integrating Strategy Summaries with Saliency Maps (Extended Abstract)","authors":[{"name":"Tobias Huber","slug":"tobias-huber","given":"Tobias","family":"Huber","affiliation":"Universität Augsburg , Universitätsstraße 6a , Augsburg , Germany"},{"name":"Katharina Weitz","slug":"katharina-weitz","given":"Katharina","family":"Weitz","affiliation":"Universität Augsburg , Universitätsstraße 6a , Augsburg , Germany"},{"name":"Elisabeth André","slug":"elisabeth-andre","given":"Elisabeth","family":"André","affiliation":"Universität Augsburg , Universitätsstraße 6a , Augsburg , Germany"},{"name":"Ofra Amir","slug":"ofra-amir","given":"Ofra","family":"Amir","affiliation":"Technion -Israel Institute of Technology"}],"abstract":"With advances in reinforcement learning (RL), agents are now being developed in high-stakes application domains such as healthcare and transportation. Explaining the behavior of these agents is challenging, as they act in large state spaces, and their decision-making can be affected by delayed rewards. In this paper, we explore a combination of explanations that attempt to convey the global behavior of the agent and local explanations which provide information regarding the agent's decision-making in a particular state. Specifically, we augment strategy summaries that demonstrate the agent's actions in a range of states with saliency maps highlighting the information it attends to. Our user study shows that intelligently choosing what states to include in the summary (global information) results in an improved analysis of the agents. We find mixed results with respect to augmenting summaries with saliency maps (local information).","doi":"10.24963/ijcai.2022/803","url":"https://ijcai.org/proceedings/2022/803","pdf":"https://ijcai.org/proceedings/2022/0803.pdf","markdown":"https://ijcai.org/proceedings/2022/803.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/803","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Journal Track","track":"Journal Track","number":803,"pages":{"first":5747,"last":5751},"keywords":["Computer Vision: Interpretability and Transparency","Machine Learning: Deep Reinforcement Learning","AI Ethics, Trust, Fairness: Trustworthy AI","Machine Learning: Explainable/Interpretable Machine Learning","AI Ethics, Trust, Fairness: Explainability and Interpretability"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987243","length_sec":87,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985386","length_sec":683,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Alqaraawi et al., 2020] Ahmed Alqaraawi, Martin Schuessler, Philipp Weiß, Enrico Costanza, and Na- dia Berthouze. Evaluating saliency map explanations for convolutional neural networks: A user study. In Proceed- ings of the 25th International Conference on Intelligent User Interfaces, IUI '20, page 275-285, New York, NY, USA, 2020. Association for Computing Machinery.","title":"Evaluating saliency map explanations for convolutional neural networks: A user study","authors":"Ahmed Alqaraawi, Martin Schuessler, Philipp Weiß, Enrico Costanza, Nadia Berthouze","year":"2020","venue":"Proceedings of the 25th International Conference on Intelligent User Interfaces, IUI '20","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Amir and Amir, 2018] Dan Amir and Ofra Amir. High- lights: Summarizing agent behavior to people. In Proc. of the 17th International conference on Autonomous Agents and Multi-Agent Systems (AAMAS), 2018.","title":"Highlights: Summarizing agent behavior to people","authors":"Amir Amir, Dan Amir, Ofra Amir","year":"2018","venue":"Proc. of the 17th International conference on Autonomous Agents and Multi-Agent Systems (AAMAS)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Amir et al., 2019] Ofra Amir, Finale Doshi-Velez, and David Sarne. Summarizing agent strategies. Autonomous Agents and Multi-Agent Systems, 33(5):628-644, 2019.","title":"Summarizing agent strategies","authors":"Ofra Amir, Finale Doshi-Velez, David Sarne","year":"2019","venue":"Autonomous Agents and Multi-Agent Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Anderson et al., 2019] Andrew Anderson, Jonathan Dodge, Amrita Sadarangani, Zoe Juozapaitis, Evan Newman, Jed Irvine, Souti Chattopadhyay, Alan Fern, and Margaret Burnett. Explaining reinforcement learning to mere mor- tals: An empirical study. In Proceedings of the Twenty- Eighth International Joint Conference on Artificial Intel- ligence, IJCAI-19, pages 1328-1334. International Joint Conferences on Artificial Intelligence Organization, 7 2019.","title":"Explaining reinforcement learning to mere mortals: An empirical study","authors":"Andrew Anderson, Jonathan Dodge, Amrita Sadarangani, Zoe Juozapaitis, Evan Newman, Jed Irvine, Souti Chattopadhyay, Alan Fern, Margaret Burnett","year":"2019","venue":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19","doi":null,"url":null,"internal_url":"/proceedings/2019/184"},{"position":5,"raw":"[Bellemare et al., 2013] M. G. Bellemare, Y. Naddaf, J. Ve- ness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Ar- tificial Intelligence Research, 47:253-279, jun 2013.","title":"The arcade learning environment: An evaluation platform for general agents","authors":"M G Bellemare, Y Naddaf, J Veness, M Bowling","year":"2013","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dhariwal et al., 2017] Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plap- pert, Alec Radford, John Schulman, Szymon Sidor, Yuhuai Wu, and Peter Zhokhov. Openai baselines. https://github.com/openai/baselines, 2017. Accessed: 2020-05-01.","title":"Openai baselines","authors":"Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, Yuhuai Wu, Peter Zhokhov","year":"2017","venue":null,"doi":null,"url":"https://github.com/openai/baselines","internal_url":null},{"position":7,"raw":"[Halasz and Moran, 1983] Frank G Halasz and Thomas P Moran. Mental models and problem solving in using a cal- culator. In Proceedings of the SIGCHI conference on Hu- man Factors in Computing Systems, pages 212-216, 1983.","title":"Mental models and problem solving in using a calculator","authors":"G Frank, Thomas P Halasz, Moran","year":"1983","venue":"Proceedings of the SIGCHI conference on Human Factors in Computing Systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hoffman et al., 2018] Robert R Hoffman, Shane T Mueller, Gary Klein, and Jordan Litman. Metrics for explain- able AI: Challenges and prospects. arXiv preprint arXiv:1812.04608, 2018.","title":"Metrics for explainable AI: Challenges and prospects","authors":"Shane T Robert R Hoffman, Gary Mueller, Jordan Klein, Litman","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hsieh and Shannon, 2005] Hsiu-Fang Hsieh and Sarah E Shannon. Three approaches to qualitative content analy- sis. Qualitative health research, 15(9):1277-1288, 2005.","title":"Three approaches to qualitative content analysis","authors":"Hsiu-Fang Hsieh, Sarah E Shannon","year":"2005","venue":"Qualitative health research","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Huber et al., 2019] Tobias Huber, Dominik Schiller, and Elisabeth André. Enhancing explainability of deep rein- forcement learning through selective layer-wise relevance propagation. In Joint German/Austrian Conference on Ar- tificial Intelligence (Künstliche Intelligenz), pages 188- 202. Springer, 2019.","title":"Enhancing explainability of deep reinforcement learning through selective layer-wise relevance propagation","authors":"Huber","year":"2019","venue":"Joint German/Austrian Conference on Artificial Intelligence (Künstliche Intelligenz)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Huber et al., 2021] Tobias Huber, Katharina Weitz, Elisa- beth André, and Ofra Amir. Local and global explana- tions of agent behavior: Integrating strategy summaries with saliency maps. Artificial Intelligence, 301:103571, 2021.","title":"Local and global explanations of agent behavior: Integrating strategy summaries with saliency maps","authors":"Huber","year":"2021","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lee and See, 2004] John D Lee and Katrina A See. Trust in automation: Designing for appropriate reliance. Human factors, 46(1):50-80, 2004.","title":"Trust in automation: Designing for appropriate reliance","authors":"D John, Katrina A Lee, See","year":"2004","venue":"Human factors","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Madumal et al., 2020] Prashan Madumal, Tim Miller, Liz Sonenberg, and Frank Vetere. Explainable reinforcement learning through a causal lens. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, pages 2493-2500. AAAI Press, 2020.","title":"Explainable reinforcement learning through a causal lens","authors":"Madumal","year":"2020","venue":"The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[McKnight and Najab, 2010] Patrick E McKnight and Julius Najab. Mann-whitney u test. The Corsini encyclopedia of psychology, pages 1-1, 2010.","title":"The Corsini encyclopedia of psychology","authors":"E Patrick, Julius Mcknight, Najab","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidje- land, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Molnar, 2019] Christoph Molnar. Interpretable machine learning: A guide for making black box models explain- able. https://christophm.github.io/interpretable-ml-book, 2019. Accessed: 2020-05-01.","title":"Interpretable machine learning: A guide for making black box models explainable","authors":"Christoph Molnar","year":"2019","venue":null,"doi":null,"url":"https://christophm.github.io/interpretable-ml-book","internal_url":null},{"position":17,"raw":"[Norman, 2014] Donald A Norman. Some observations on mental models. In Mental models, pages 15-22. Psychol- ogy Press, 2014.","title":"Some observations on mental models","authors":"Donald A Norman","year":"2014","venue":"Mental models","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Selvaraju et al., 2020] Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-CAM: Visual explanations from deep networks via gradient-based localization. Int. J. Comput. Vis., 128(2):336-359, 2020.","title":"Grad-CAM: Visual explanations from deep networks via gradient-based localization","authors":"R Ramprasaath, Michael Selvaraju, Abhishek Cogswell, Ramakrishna Das, Devi Vedantam, Dhruv Parikh, Batra","year":"2020","venue":"Int. J. Comput. Vis","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sequeira et al., 2019] Pedro Sequeira, Eric Yeh, and Melinda T Gervasio. Interestingness elements for explain- able reinforcement learning through introspection. In IUI Workshops, page 7, 2019.","title":"Interestingness elements for explainable reinforcement learning through introspection","authors":"Pedro Sequeira, Eric Yeh, Melinda T Gervasio","year":"2019","venue":"IUI Workshops","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Stone et al., 2016] Peter Stone, Rodney Brooks, Erik Bryn- jolfsson, Ryan Calo, Oren Etzioni, Greg Hager, Julia Hirschberg, Shivaram Kalyanakrishnan, Ece Kamar, Sarit Kraus, Kevin Leyton-Brown, David Parkes, Press William, Saxenian AnnaLee, Shah Julie, Tambe Milind, and Teller Astro. Artificial intelligence and life in 2030. One Hun- dred Year Study on Artificial Intelligence: Report of the 2015-2016 Study Panel, 2016.","title":"Artificial intelligence and life in 2030","authors":"Peter Stone, Rodney Brooks, Erik Brynjolfsson, Ryan Calo, Oren Etzioni, Greg Hager, Julia Hirschberg, Shivaram Kalyanakrishnan, Ece Kamar, Sarit Kraus, Kevin Leyton-Brown, David Parkes, Press William, Saxenian Annalee, Julie Shah, Tambe Milind, Teller Astro","year":"2016","venue":"One Hundred Year Study on Artificial Intelligence: Report of the 2015-2016 Study Panel","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}