{"type":"ScholarlyArticle","title":"Explaining Reinforcement Learning to Mere Mortals: An Empirical Study","authors":[{"name":"Andrew Anderson","slug":"andrew-anderson","given":"Andrew","family":"Anderson","affiliation":"Oregon State University {anderan2 , dodgej ,"},{"name":"Jonathan Dodge","slug":"jonathan-dodge","given":"Jonathan","family":"Dodge","affiliation":"Oregon State University {anderan2 , dodgej ,"},{"name":"Amrita Sadarangani","slug":"amrita-sadarangani","given":"Amrita","family":"Sadarangani","affiliation":"Oregon State University {anderan2 , dodgej ,"},{"name":"Zoe Juozapaitis","slug":"zoe-juozapaitis","given":"Zoe","family":"Juozapaitis","affiliation":"Oregon State University {anderan2 , dodgej ,"},{"name":"Evan Newman","slug":"evan-newman","given":"Evan","family":"Newman","affiliation":"Oregon State University {anderan2 , dodgej ,"},{"name":"Jed Irvine","slug":"jed-irvine","given":"Jed","family":"Irvine","affiliation":"Oregon State University {anderan2 , dodgej ,"},{"name":"Souti Chattopadhyay","slug":"souti-chattopadhyay","given":"Souti","family":"Chattopadhyay","affiliation":"Oregon State University {anderan2 , dodgej ,"},{"name":"Alan Fern","slug":"alan-fern","given":"Alan","family":"Fern","affiliation":"Oregon State University {anderan2 , dodgej ,"},{"name":"Margaret Burnett","slug":"margaret-burnett","given":"Margaret","family":"Burnett","affiliation":"Oregon State University {anderan2 , dodgej ,"}],"abstract":"We present a user study to investigate the impact of explanations on non-experts? understanding of reinforcement learning (RL) agents. We investigate both a common RL visualization, saliency maps (the focus of attention), and a more recent explanation type, reward-decomposition bars (predictions of future types of rewards). We designed a 124 participant, four-treatment experiment to compare participants? mental models of an RL agent in a simple Real-Time Strategy (RTS) game. Our results show that the combination of both saliency and reward bars were needed to achieve a statistically significant improvement in mental model score over the control. In addition, our qualitative analysis of the data reveals a number of effects for further study.","doi":"10.24963/ijcai.2019/184","url":"https://ijcai.org/proceedings/2019/184","pdf":"https://ijcai.org/proceedings/2019/0184.pdf","markdown":"https://ijcai.org/proceedings/2019/184.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/184","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Humans and AI","track":"Main track","number":184,"pages":{"first":1328,"last":1334},"keywords":["Humans and AI: Human-Computer Interaction","Humans and AI: Intelligent User Interfaces"],"videos":[],"references":[{"position":1,"raw":"[Adebayo et al., 2018] Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. In Advances in Neural Information Processing Systems 31, 2018.","title":"Sanity checks for saliency maps","authors":"Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, Been Kim","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Ancona et al., 2018] Marco Ancona, Enea Ceolini, Cengiz Öztireli, and Markus Gross. Towards better understanding of gradient-based attribution methods for deep neural net- works. In Intl. Conf. on Learning Representations, 2018.","title":"Towards better understanding of gradient-based attribution methods for deep neural networks","authors":"Marco Ancona, Enea Ceolini, Cengiz Öztireli, Markus Gross","year":"2018","venue":"Intl. Conf. on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cramer and Howitt, 2004] Duncan Cramer and Dennis Howitt. The Sage dictionary of statistics: a practical re- source for students in the social sciences. Sage, 2004.","title":"The Sage dictionary of statistics: a practical resource for students in the social sciences","authors":"Howitt Cramer, Duncan Cramer, Dennis Howitt","year":"2004","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dodge et al., 2018] Jonathan Dodge, Sean Penney, Claudia Hilderbrand, Andrew Anderson, and Margaret Burnett. How the experts do it: Assessing and explaining agent be- haviors in real-time strategy games. In ACM Conf. on Hu- man Factors in Computing Systems, CHI '18. ACM, 2018.","title":"How the experts do it: Assessing and explaining agent behaviors in real-time strategy games","authors":"Jonathan Dodge, Sean Penney, Claudia Hilderbrand, Andrew Anderson, Margaret Burnett","year":"2018","venue":"ACM Conf. on Human Factors in Computing Systems, CHI '18","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Erwig et al., 2018] Martin Erwig, Alan Fern, Magesh Mu- rali, and Anurag Koul. Explaining deep adaptive programs via reward decomposition. In IJCAI Workshop on Explain- able Artificial Intelligence, 2018.","title":"Explaining deep adaptive programs via reward decomposition","authors":"Martin Erwig, Alan Fern, Magesh Murali, Anurag Koul","year":"2018","venue":"IJCAI Workshop on Explainable Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fong and Vedaldi, 2017] Ruth Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful perturbation. In IEEE International Conference on Com- puter Vision (ICCV). IEEE, 2017.","title":"Interpretable explanations of black boxes by meaningful perturbation","authors":"Ruth Fong, Andrea Vedaldi","year":"2017","venue":"IEEE International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Greydanus et al., 2018] Sam Greydanus, Anurag Koul, Jonathan Dodge, and Alan Fern. Visualizing and under- standing Atari agents. In Proceedings of the 35th Inter- national Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research. PMLR, 2018.","title":"Visualizing and understanding Atari agents","authors":"Sam Greydanus, Anurag Koul, Jonathan Dodge, Alan Fern","year":"2018","venue":"Proceedings of the 35th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hoffman et al., 2018] Robert Hoffman, Shane Mueller, Gary Klein, and Jordan Litman. Metrics for explainable ai: Challenges and prospects. arXiv:1812.04608, 2018.","title":"Metrics for explainable ai: Challenges and prospects","authors":"Robert Hoffman, Shane Mueller, Gary Klein, Jordan Litman","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hsieh and Shannon, 2005] Hsiu-Fang Hsieh and Sarah Shannon. Three approaches to qualitative content analy- sis. Qualitative health research, 15(9), 2005.","title":"Three approaches to qualitative content analysis","authors":"Hsiu-Fang Hsieh, Sarah Shannon","year":"2005","venue":"Qualitative health research","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jaccard, 1908] Paul Jaccard. Nouvelles recherches sur la distribution florale. Bull. Soc. Vaud. Sci. Nat., 44, 1908.","title":"Nouvelles recherches sur la distribution florale","authors":"Paul Jaccard","year":"1908","venue":"Bull. Soc. Vaud. Sci. Nat","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kim et al., 2018] Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, and Rory Sayres. Interpretability beyond feature attribu- tion: Quantitative testing with concept activation vectors (TCAV). In Intl. Conf. on Machine Learning, volume 80. PMLR, 2018.","title":"Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (TCAV)","authors":"Kim","year":"2018","venue":"Intl. Conf. on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kulesza et al., 2015] Todd Kulesza, Margaret Burnett, Weng-Keen Wong, and Simone Stumpf. Principles of explanatory debugging to personalize interactive ma- chine learning. In ACM Intl. Conf. on Intelligent User Interfaces, IUI '15. ACM, 2015.","title":"Principles of explanatory debugging to personalize interactive machine learning","authors":"Todd Kulesza, Margaret Burnett, Weng-Keen Wong, Simone Stumpf","year":"2015","venue":"ACM Intl. Conf. on Intelligent User Interfaces, IUI '15","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lippa et al., 2008] Katherine Lippa, Helen Klein, and Va- lerie Shalin. Everyday expertise: cognitive demands in diabetes self-management. Human Factors, 50(1), 2008.","title":"Everyday expertise: cognitive demands in diabetes self-management","authors":"Katherine Lippa, Helen Klein, Valerie Shalin","year":"2008","venue":"Human Factors","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Muramatsu and Pratt, 2001] Jack Muramatsu and Wanda Pratt. Transparent queries: investigation users' mental models of search engines. In Intl. ACM SIGIR Conf. on Research and Development in Info. Retrieval. ACM, 2001.","title":"Transparent queries: investigation users' mental models of search engines","authors":"Pratt Muramatsu, Jack Muramatsu, Wanda Pratt","year":"2001","venue":"Intl. ACM SIGIR Conf. on Research and Development in Info. Retrieval","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Newn et al., 2016] Joshua Newn, Eduardo Velloso, Mar- cus Carter, and Frank Vetere. Exploring the effects of gaze awareness on multiplayer gameplay. In ACM Symp. on Computer-Human Interaction in Play Companion Ex- tended Abstracts. ACM, 2016.","title":"Exploring the effects of gaze awareness on multiplayer gameplay","authors":"Newn","year":"2016","venue":"ACM Symp. on Computer-Human Interaction in Play Companion Extended Abstracts","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Newn et al., 2017] Joshua Newn, Eduardo Velloso, Fraser Allison, Yomna Abdelrahman, and Frank Vetere. Evalu- ating real-time gaze representations to infer intentions in competitive turn-based strategy games. In ACM Symp. on Computer-Human Interaction in Play. ACM, 2017.","title":"Evaluating real-time gaze representations to infer intentions in competitive turn-based strategy games","authors":"Joshua Newn, Eduardo Velloso, Fraser Allison, Yomna Abdelrahman, Frank Vetere","year":"2017","venue":"ACM Symp. on Computer-Human Interaction in Play","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Norman and Gentner, 1983] Donald Norman and Dedra Gentner. Mental models. Lawrence Erlbaum Associates, Hillsdale, NJ, 1983.","title":"Mental models","authors":"Donald Norman, Dedra Gentner","year":"1983","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Penney et al., 2018] Sean Penney, Jonathan Dodge, Clau- dia Hilderbrand, Andrew Anderson, Logan Simpson, and Margaret Burnett. Toward foraging for understanding of StarCraft agents: An empirical study. In ACM Intl. Conf. on Intelligent User Interfaces, IUI '18. ACM, 2018.","title":"Toward foraging for understanding of StarCraft agents: An empirical study","authors":"Sean Penney, Jonathan Dodge, Claudia Hilderbrand, Andrew Anderson, Logan Simpson, Margaret Burnett","year":"2018","venue":"ACM Intl. Conf. on Intelligent User Interfaces, IUI '18","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Riche et al., 2013] Nicolas Riche, Matthieu Duvinage, Matei Mancas, Bernard Gosselin, and Thierry Dutoit. Saliency and human fixations: state-of-the-art and study of comparison metrics. In IEEE Intl. Conf. on Computer Vision, 2013.","title":"Saliency and human fixations: state-of-the-art and study of comparison metrics","authors":"Nicolas Riche, Matthieu Duvinage, Matei Mancas, Bernard Gosselin, Thierry Dutoit","year":"2013","venue":"IEEE Intl. Conf. on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Russell and Zimdars, 2003] Stuart Russell and Andrew Zimdars. Q-decomposition for reinforcement learning agents. In Intl. Conf. on Machine Learning, 2003.","title":"Q-decomposition for reinforcement learning agents","authors":"Stuart Russell, Andrew Zimdars","year":"2003","venue":"Intl. Conf. on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Simonyan et al., 2013] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. CoRR, 2013.","title":"Deep inside convolutional networks: Visualising image classification models and saliency maps","authors":"Karen Simonyan, Andrea Vedaldi, Andrew Zisserman","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Springenberg et al., 2014] Jost Springenberg, Alexey Doso- vitskiy, Thomas Brox, and Martin A. Riedmiller. Striving for simplicity: The all convolutional net. CoRR, 2014.","title":"Striving for simplicity: The all convolutional net","authors":"Jost Springenberg, Alexey Dosovitskiy, Thomas Brox, Martin A Riedmiller","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Van Seijen et al., 2017] HARM Van Seijen, Mehdi Fatemi, Joshua Romoff, Romain Laroche, Tavian Barnes, and Jef- frey Tsang. Hybrid reward architecture for reinforcement learning. In Advances in Neural Information Processing Systems, 2017.","title":"Hybrid reward architecture for reinforcement learning","authors":"Van Seijen","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Vinyals et al., 2019] Oriol Vinyals, David Silver, et al. Al- phaStar: Mastering the Real-Time Strategy Game Star- Craft II. https://deepmind.com/blog/alphastar-mastering- real-time-strategy-game-starcraft-ii/, 2019.","title":"Al-phaStar: Mastering the Real-Time Strategy Game Star-Craft II","authors":"Oriol Vinyals, David Silver","year":null,"venue":null,"doi":null,"url":"https://deepmind.com/blog/alphastar-mastering-real-time-strategy-game-starcraft-ii/,2019","internal_url":null},{"position":25,"raw":"[Wohlin et al., 2000] Claes Wohlin, Per Runeson, Martin Höst, Magnus Ohlsson, Björn Regnell, and Anders Wesslén. Experimentation in Software Engineering: An Introduction. Kluwer Academic Publishers, 2000.","title":"Experimentation in Software Engineering: An Introduction","authors":"Wohlin","year":"2000","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zeiler and Fergus, 2014] Matthew Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European Conf. on Computer Vision. Springer, 2014.","title":"Visualizing and understanding convolutional networks","authors":"Matthew Zeiler, Rob Fergus","year":"2014","venue":"European Conf. on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang et al., 2018] Jianming Zhang, Sarah Adel Bargal, Zhe Lin, Jonathan Brandt, Xiaohui Shen, and Stan Sclaroff. Top-down neural attention by excitation back- prop. Int. J. Comput. Vision, 126(10), October 2018.","title":"Top-down neural attention by excitation backprop","authors":"Jianming Zhang, Sarah Adel Bargal, Zhe Lin, Jonathan Brandt, Xiaohui Shen, Stan Sclaroff","year":"2018","venue":"Int. J. Comput. Vision","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}