{"type":"ScholarlyArticle","title":"Exploring Computational User Models for Agent Policy Summarization","authors":[{"name":"Isaac Lage","slug":"isaac-lage","given":"Isaac","family":"Lage","affiliation":"Harvard University"},{"name":"Daphna Lifschitz","slug":"daphna-lifschitz","given":"Daphna","family":"Lifschitz","affiliation":"Technion -Israel Institute of Technology"},{"name":"Finale Doshi-Velez","slug":"finale-doshi-velez","given":"Finale","family":"Doshi-Velez","affiliation":"Harvard University"},{"name":"Ofra Amir","slug":"ofra-amir","given":"Ofra","family":"Amir","affiliation":"Technion -Israel Institute of Technology"}],"abstract":"AI agents support high stakes decision-making processes from driving cars to prescribing drugs, making it increasingly important for human users to understand their behavior. Policy summarization methods aim to convey strengths and weaknesses of such agents by demonstrating their behavior in a subset of informative states. Some policy summarization methods extract a summary that optimizes the ability to reconstruct the agent's policy under the assumption that users will deploy inverse reinforcement learning. In this paper, we explore the use of different models for extracting summaries. We introduce an imitation learning-based approach to policy summarization; we demonstrate through computational simulations that a mismatch between the model used to extract a summary and the model used to reconstruct the policy results in worse reconstruction quality; and we demonstrate through a human-subject study that people use different models to reconstruct policies in different contexts, and that matching the summary extraction model to these can improve performance. Together, our results suggest that it is important to carefully consider user models in policy summarization.","doi":"10.24963/ijcai.2019/194","url":"https://ijcai.org/proceedings/2019/194","pdf":"https://ijcai.org/proceedings/2019/0194.pdf","markdown":"https://ijcai.org/proceedings/2019/194.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/194","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Humans and AI","track":"Main track","number":194,"pages":{"first":1401,"last":1407},"keywords":["Machine Learning: Explainable Machine Learning","Humans and AI: Human-AI Collaboration","Agent-based and Multi-agent Systems: Human-Agent Interaction"],"videos":[],"references":[{"position":1,"raw":"[Adams et al., 2005] BM Adams, HT Banks, M Davidian, Hee-Dae Kwon, HT Tran, SN Wynne, and ES Rosenberg. Hiv dynamics: modeling, data analysis, and optimal treatment protocols. Jour- nal of Computational and Applied Mathematics, 184(1):10-49, 2005.","title":"Hiv dynamics: modeling, data analysis, and optimal treatment protocols","authors":"Adams","year":"2005","venue":"Journal of Computational and Applied Mathematics","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Aha et al., 2017] David W. Aha, Trevor Darrell, Michael Pazzani, Darryn Reid, Claude Sammut, and Peter Stone. Ijcai-17 work- shop on explainable ai (xai). In IJCAI-17 Workshop on Explain- able AI (XAI), 2017.","title":"Ijcai-17 workshop on explainable ai (xai)","authors":"David W Aha, Trevor Darrell, Michael Pazzani, Darryn Reid, Claude Sammut, Peter Stone","year":"2017","venue":"IJCAI-17 Workshop on Explainable AI (XAI)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Albrecht and Stone, 2018] Stefano V. Albrecht and Peter Stone. Autonomous agents modelling other agents: A comprehensive survey and open problems. Artificial Intelligence, 258:66 -95, 2018.","title":"Autonomous agents modelling other agents: A comprehensive survey and open problems","authors":"Stefano V Albrecht, Peter Stone","year":"2018","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Amir and Amir, 2018] Dan Amir and Ofra Amir. Highlights: Summarizing agent behavior to people. In Proc. of the 17th In- ternational conference on Autonomous Agents and Multi-Agent Systems (AAMAS), 2018.","title":"Highlights: Summarizing agent behavior to people","authors":"Amir Amir, Dan Amir, Ofra Amir","year":"2018","venue":"Proc. of the 17th International conference on Autonomous Agents and Multi-Agent Systems (AAMAS)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Amir et al., 2018] Ofra Amir, Finale Doshi-Velez, and David Sarne. Agent strategy summarization. In Proceedings of the 17th International Conference on Autonomous Agents and Mul- tiAgent Systems, pages 1203-1207. International Foundation for Autonomous Agents and Multiagent Systems, 2018.","title":"Agent strategy summarization","authors":"Ofra Amir, Finale Doshi-Velez, David Sarne","year":"2018","venue":"Proceedings of the 17th International Conference on Autonomous Agents and Mul-tiAgent Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Baker et al., 2009] Chris L Baker, Rebecca Saxe, and Joshua B Tenenbaum. Action understanding as inverse planning. Cog- nition, 113(3):329-349, 2009.","title":"Action understanding as inverse planning","authors":"Chris L Baker, Rebecca Saxe, Joshua B Tenenbaum","year":"2009","venue":"Cognition","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Baker et al., 2011] Chris Baker, Rebecca Saxe, and Joshua Tenen- baum. Bayesian theory of mind: Modeling joint belief-desire attribution. In Proceedings of the annual meeting of the cognitive science society, volume 33, 2011.","title":"Bayesian theory of mind: Modeling joint belief-desire attribution","authors":"Chris Baker, Rebecca Saxe, Joshua Tenenbaum","year":"2011","venue":"Proceedings of the annual meeting of the cognitive science society","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Broekens et al., 2010] Joost Broekens, Maaike Harbers, Koen Hindriks, Karel van den Bosch, Catholijn Jonker, and John-Jules Meyer. Do you get it? user-evaluated explainable bdi agents. In Jürgen Dix and Cees Witteveen, editors, Multiagent System Tech- nologies, pages 28-39, Berlin, Heidelberg, 2010. Springer Berlin Heidelberg.","title":"Do you get it? user-evaluated explainable bdi agents","authors":"Joost Broekens, Maaike Harbers, Koen Hindriks, Karel Van Den, Catholijn Bosch, John-Jules Jonker, Meyer","year":"2010","venue":"Multiagent System Technologies","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Brown and Niekum, 2018] Daniel S. Brown and Scott Niekum. Machine teaching for inverse reinforcement learning: Algorithms and applications. arXiv, abs/1805.07687, 2018.","title":"Machine teaching for inverse reinforcement learning: Algorithms and applications","authors":"Daniel S Brown, Scott Niekum","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Daw et al., 2005] Nathaniel D. Daw, Yael Niv, and Peter Dayan. Uncertainty-based competition between prefrontal and dorsolat- eral striatal systems for behavioral control. Nature Neuroscience, 8:1704 EP -, 11 2005.","title":"Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control","authors":"Nathaniel D Daw, Yael Niv, Peter Dayan","year":"2005","venue":"Nature Neuroscience","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Dodson et al., 2011] Thomas Dodson, Nicholas Mattei, and Judy Goldsmith. A natural language argumentation interface for ex- planation generation in markov decision processes. In Interna- tional Conference on Algorithmic DecisionTheory, pages 42-55. Springer, 2011.","title":"A natural language argumentation interface for explanation generation in markov decision processes","authors":"Thomas Dodson, Nicholas Mattei, Judy Goldsmith","year":"2011","venue":"International Conference on Algorithmic DecisionTheory","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Dragan and Srinivasa, 2014] Anca Dragan and Siddhartha Srini- vasa. Familiarization to robot motion. In Proceedings of the 2014 ACM/IEEE international conference on Human-robot in- teraction, pages 366-373. ACM, 2014.","title":"Anca Dragan and Siddhartha Srinivasa","authors":"Srinivasa Dragan","year":"2014","venue":"Proceedings of the 2014 ACM/IEEE international conference on Human-robot interaction","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Ernst et al., 2006] Damien Ernst, Guy-Bart Stan, Jorge Goncalves, and Louis Wehenkel. Clinical data based optimal sti strategies for hiv: a reinforcement learning approach. In Decision and Control, 2006 45th IEEE Conference on, pages 667-672. IEEE, 2006.","title":"Clinical data based optimal sti strategies for hiv: a reinforcement learning approach","authors":"Damien Ernst, Guy-Bart Stan, Jorge Goncalves, Louis Wehenkel","year":"2006","venue":"45th IEEE Conference on","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Greydanus et al., 2017] Sam Greydanus, Anurag Koul, Jonathan Dodge, and Alan Fern. Visualizing and understanding atari agents. arXiv, abs/1711.00138, 2017.","title":"Visualizing and understanding atari agents","authors":"Sam Greydanus, Anurag Koul, Jonathan Dodge, Alan Fern","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Huang et al., 2017] Sandy H. Huang, David Held, Pieter Abbeel, and Anca D. Dragan. Enabling robots to communicate their ob- jectives. arXiv, abs/1702.03465, 2017.","title":"Enabling robots to communicate their objectives","authors":"Sandy H Huang, David Held, Pieter Abbeel, Anca D Dragan","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Huang et al., 2018] Sandy H Huang, Kush Bhatia, Pieter Abbeel, and Anca D Dragan. Establishing appropriate trust via critical states. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3929-3936. IEEE, 2018.","title":"Establishing appropriate trust via critical states","authors":"H Sandy, Kush Huang, Pieter Bhatia, Anca D Abbeel, Dragan","year":"2018","venue":"IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Khan et al., 2009] Omar Zia Khan, Pascal Poupart, and James P Black. Minimal sufficient explanations for factored markov de- cision processes. In ICAPS, 2009.","title":"Minimal sufficient explanations for factored markov decision processes","authors":"Zia Omar, Pascal Khan, James P Poupart, Black","year":"2009","venue":"ICAPS","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lage et al., 2019] Isaac Lage, Daphna Lifschitz, Finale Doshi- Velez, and Ofra Amir. Exploring computational user models for agent policy summarization. arXiv, abs/1905.13271, 2019.","title":"Exploring computational user models for agent policy summarization","authors":"Isaac Lage, Daphna Lifschitz, Finale Doshi-Velez, Ofra Amir","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Lomas et al., 2012] Meghann Lomas, Robert Chevalier, Ernest Vincent Cross II, Robert Christopher Garrett, John Hoare, and Michael Kopack. Explaining robot actions. In Proceedings of the seventh annual ACM/IEEE international conference on Human-Robot Interaction, pages 187-188. ACM, 2012.","title":"Explaining robot actions","authors":"Meghann Lomas, Robert Chevalier, Ernest Vincent Cross, I I, Robert Christopher Garrett, John Hoare, Michael Kopack","year":"2012","venue":"Proceedings of the seventh annual ACM/IEEE international conference on Human-Robot Interaction","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Medin and Shaffer, 1978] Douglas L. Medin and Marguerite M. Shaffer. Context theory of classification learning. Psychologi- cal Review, 85(33):2017-238, 1978.","title":"Context theory of classification learning","authors":"Douglas L Medin, Marguerite M Shaffer","year":"1978","venue":"Psychological Review","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Miller, 2018] Tim Miller. Explanation in artificial intelligence: In- sights from the social sciences. Artificial Intelligence, 2018.","title":"Explanation in artificial intelligence: Insights from the social sciences","authors":"Tim Miller","year":"2018","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Ng et al., 2000] Andrew Y Ng, Stuart J Russell, et al. Algorithms for inverse reinforcement learning. In Icml, pages 663-670, 2000.","title":"Algorithms for inverse reinforcement learning","authors":"Y Andrew, Stuart J Ng, Russell","year":"2000","venue":"Icml","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Ramakrishnan et al., 2018] Ramya Ramakrishnan, Ece Kamar, Debadeepta Dey, Julie A. Shah, and Eric Horvitz. Discovering blind spots in reinforcement learning. arXiv, abs/1805.08966, 2018.","title":"Discovering blind spots in reinforcement learning","authors":"Ramya Ramakrishnan, Ece Kamar, Debadeepta Dey, Julie A Shah, Eric Horvitz","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[van der Waa et al., 2018] Jasper van der Waa, Jurriaan van Digge- len, Karel van den Bosch, and Mark A. Neerincx. Contrastive explanations for reinforcement learning in terms of expected con- sequences. arXiv, abs/1807.08706, 2018.","title":"Contrastive explanations for reinforcement learning in terms of expected consequences","authors":"Waa Van Der","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhu et al., 2003a] Xiaojin Zhu, Zoubin Ghahramani, and John Lafferty. Semi-supervised learning using gaussian fields and har- monic functions. In Proceedings of the Twentieth International Conference on International Conference on Machine Learning, ICML'03, pages 912-919. AAAI Press, 2003.","title":"Semi-supervised learning using gaussian fields and harmonic functions","authors":"Zhu","year":"2003","venue":"Proceedings of the Twentieth International Conference on International Conference on Machine Learning, ICML'03","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhu et al., 2003b] Xiaojin Zhu, John Lafferty, and Zoubin Ghahra- mani. Combining active learning and semi-supervised learning using gaussian fields and harmonic functions. ICML 2003 Work- shop on The Continuum from Labeled to Unlabeled Data in Ma- chine Learning and Data Mining, pages 58-65, 2003.","title":"Combining active learning and semi-supervised learning using gaussian fields and harmonic functions","authors":"Zhu","year":"2003","venue":"ICML 2003 Workshop on The Continuum from Labeled to Unlabeled Data in Machine Learning and Data Mining","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhu, 2015] Xiaojin Zhu. Machine teaching: An inverse problem to machine learning and an approach toward optimal education. In AAAI, pages 4083-4087, 2015.","title":"Machine teaching: An inverse problem to machine learning and an approach toward optimal education","authors":"Xiaojin Zhu","year":"2015","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Ziebart et al., 2008] Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, and Anind K Dey. Maximum entropy inverse reinforce- ment learning. In AAAI, volume 8, pages 1433-1438. Chicago, IL, USA, 2008.","title":"Maximum entropy inverse reinforcement learning","authors":"Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey","year":"2008","venue":"AAAI","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}