{"type":"ScholarlyArticle","title":"Interpretable Models for Understanding Immersive Simulations","authors":[{"name":"Nicholas Hoernle","slug":"nicholas-hoernle","given":"Nicholas","family":"Hoernle","affiliation":"University of Edinburgh"},{"name":"Kobi Gal","slug":"kobi-gal","given":"Kobi","family":"Gal","affiliation":"University of Edinburgh; Ben-Gurion University"},{"name":"Barbara Grosz","slug":"barbara-grosz","given":"Barbara","family":"Grosz","affiliation":"Harvard University"},{"name":"Leilah Lyons","slug":"leilah-lyons","given":"Leilah","family":"Lyons","affiliation":""},{"name":"Ada Ren","slug":"ada-ren","given":"Ada","family":"Ren","affiliation":"TERC"},{"name":"Andee Rubin","slug":"andee-rubin","given":"Andee","family":"Rubin","affiliation":"TERC"}],"abstract":"This paper describes methods for comparative evaluation of the interpretability of models of high dimensional time series data inferred by unsupervised machine learning algorithms. The time series data used in this investigation were logs from an immersive simulation like those commonly used in education and healthcare training. The structures learnt by the models provide representations of participants' activities in the simulation which are intended to be meaningful to people's interpretation. To choose the model that induces the best representation, we designed two interpretability tests, each of which evaluates the extent to which a model’s output aligns with people’s expectations or intuitions of what has occurred in the simulation. We compared the performance of the models on these interpretability tests to their performance on statistical information criteria. We show that the models that optimize interpretability quality differ from those that optimize (statistical) information theoretic criteria. Furthermore, we found that a model using a fully Bayesian approach performed well on both the statistical and human-interpretability measures. The Bayesian approach is a good candidate for fully automated model selection, i.e., when direct empirical investigations of interpretability are costly or infeasible.","doi":"10.24963/ijcai.2020/321","url":"https://ijcai.org/proceedings/2020/321","pdf":"https://ijcai.org/proceedings/2020/0321.pdf","markdown":"https://ijcai.org/proceedings/2020/321.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/321","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":321,"pages":{"first":2319,"last":2325},"keywords":["AI Ethics: Explainability","Machine Learning: Interpretability","Humans and AI: Human-AI Collaboration"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24252/file_24252.m3u8","length_sec":296,"provider":"hls"}],"references":[{"position":1,"raw":"[Alinier et al., 2014] Guillaume Alinier, Colin Harwood, Pa- tricia Harwood, Susan Montague, Eileen Huish, Krishna Ruparelia, and Melina Antuofermo. Immersive clinical simulation in undergraduate health care interprofessional education: Knowledge and perceptions. Clinical Simula- tion in Nursing, 10(4):e205-e216, 2014.","title":"Immersive clinical simulation in undergraduate health care interprofessional education: Knowledge and perceptions","authors":"Guillaume Alinier, Colin Harwood, Patricia Harwood, Susan Montague, Eileen Huish, Krishna Ruparelia, Melina Antuofermo","year":"2014","venue":"Clinical Simulation in Nursing","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Amir and Gal, 2013] Ofra Amir and Ya'akov Kobi Gal. Plan recognition and visualization in exploratory learning environments. ACM Transactions on Interactive Intelli- gent Systems (TiiS), 3(3):16, 2013.","title":"Plan recognition and visualization in exploratory learning environments","authors":"Gal Amir, Ofra Amir, Ya'akov Kobi Gal","year":"2013","venue":"ACM Transactions on Interactive Intelligent Systems (TiiS)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Caruana et al., 2015] Rich Caruana, Yin Lou, Johannes Gehrke, Paul Koch, Marc Sturm, and Noemie Elhadad. Intelligible models for healthcare: Predicting pneumo- nia risk and hospital 30-day readmission. In Proceed- ings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1721- 1730. ACM, 2015.","title":"Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission","authors":"Rich Caruana, Yin Lou, Johannes Gehrke, Paul Koch, Marc Sturm, Noemie Elhadad","year":"2015","venue":"Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chang et al., 2009] Jonathan Chang, Sean Gerrish, Chong Wang, Jordan L Boyd-Graber, and David M Blei. Read- ing tea leaves: How humans interpret topic models. In Advances in neural information processing systems, pages 288-296, 2009.","title":"Reading tea leaves: How humans interpret topic models","authors":"Jonathan Chang, Sean Gerrish, Chong Wang, Jordan L Boyd-Graber, David M Blei","year":"2009","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Choi et al., 2016] Edward Choi, Mohammad Taha Ba- hadori, Jimeng Sun, Joshua Kulas, Andy Schuetz, and Walter Stewart. Retain: An interpretable predictive model for healthcare using reverse time attention mechanism. In Advances in Neural Information Processing Systems, pages 3504-3512, 2016.","title":"Retain: An interpretable predictive model for healthcare using reverse time attention mechanism","authors":"Edward Choi, Mohammad Taha Bahadori, Jimeng Sun, Joshua Kulas, Andy Schuetz, Walter Stewart","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Doshi-Velez and Kim, 2017] Finale Doshi-Velez and Been Kim. Towards a rigorous science of interpretable machine learning. In eprint arXiv:1702.08608, 2017.","title":"Finale Doshi-Velez and Been Kim. Towards a rigorous science of interpretable machine learning","authors":"Kim Doshi-Velez","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Fox et al., 2008] Emily B Fox, Erik B Sudderth, Michael I Jordan, and Alan S Willsky. An hdp-hmm for systems with state persistence. In Proceedings of the 25th international conference on Machine learning, pages 312-319. ACM, 2008.","title":"An hdp-hmm for systems with state persistence","authors":"Emily B Fox, Erik B Sudderth, Michael I Jordan, Alan S Willsky","year":"2008","venue":"Proceedings of the 25th international conference on Machine learning","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Fox, 2009] Emily Beth Fox. Bayesian nonparametric learn- ing of complex dynamical phenomena. PhD thesis, Mas- sachusetts Institute of Technology, 2009.","title":"Bayesian nonparametric learning of complex dynamical phenomena","authors":"Emily Beth, Fox","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gelman et al., 2013] Andrew Gelman, John B Carlin, Hal S Stern, David B Dunson, Aki Vehtari, and Donald B Rubin. Bayesian data analysis. Chapman and Hall/CRC, 2013.","title":"Bayesian data analysis","authors":"Andrew Gelman, Hal S John B Carlin, David B Stern, Aki Dunson, Donald B Vehtari, Rubin","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hoernle et al., 2018] Nicholas Hoernle, Kobi Gal, Barbara Grosz, Pavlos Protopapas, and Andee Rubin. Modeling the effects of students' interactions with immersive simu- lations using markov switching systems. In Educational Data Mining (EDM), Buffalo, USA, July 2018, 2018.","title":"Modeling the effects of students' interactions with immersive simulations using markov switching systems","authors":"Nicholas Hoernle, Kobi Gal, Barbara Grosz, Pavlos Protopapas, Andee Rubin","year":"2018","venue":"Educational Data Mining (EDM)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lage et al., 2018] Isaac Lage, Andrew Ross, Samuel J Ger- shman, Been Kim, and Finale Doshi-Velez. Human-in- the-loop interpretability prior. In Advances in Neural In- formation Processing Systems, pages 10159-10168, 2018.","title":"Human-inthe-loop interpretability prior","authors":"Isaac Lage, Andrew Ross, Been Samuel J Gershman, Finale Kim, Doshi-Velez","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lipton, 2016] Zachary C Lipton. The mythos of model in- terpretability. arXiv preprint arXiv:1606.03490, 2016.","title":"The mythos of model interpretability","authors":"Zachary C Lipton","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Mallavarapu et al., 2019] Aditi Mallavarapu, Leilah Lyons, Stephen Uzzo, Wren Thompson, Rinat Levy-Cohen, and Brian Slattery. Connect-to-connected worlds: Piloting a mobile, data-driven reflection tool for an open-ended sim- ulation at a museum. In Proceedings of the 2019 CHI Con- ference on Human Factors in Computing Systems, page 7. ACM, 2019.","title":"Connect-to-connected worlds: Piloting a mobile, data-driven reflection tool for an open-ended simulation at a museum","authors":"Aditi Mallavarapu, Leilah Lyons, Stephen Uzzo, Wren Thompson, Rinat Levy-Cohen, Brian Slattery","year":"2019","venue":"Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Ribeiro et al., 2016] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Why should i trust you?: Explain- ing the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowl- edge discovery and data mining, pages 1135-1144. ACM, 2016.","title":"Why should i trust you?: Explaining the predictions of any classifier","authors":"Marco Tulio Ribeiro, Sameer Singh, Carlos Guestrin","year":"2016","venue":"Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Rosenfeld and Richardson, 2019] Avi Rosenfeld and Ariella Richardson. Explainability in human-agent systems. Auton. Agents Multi Agent Syst., 33(6):673-705, 2019.","title":"Explainability in human-agent systems","authors":"Richardson Rosenfeld, Avi Rosenfeld, Ariella Richardson","year":"2019","venue":"Auton. Agents Multi Agent Syst","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Smørdal et al., 2012] Ole Smørdal, Jim Slotta, Tom Moher, Michelle Lui, and Alfredo Jornet. Hybrid spaces for sci- ence learning: New demands and opportunities for re- search. In International Conference of the Learning Sci- ences. Sydney, Australia., 2012.","title":"Hybrid spaces for science learning: New demands and opportunities for research","authors":"Ole Smørdal, Jim Slotta, Tom Moher, Michelle Lui, Alfredo Jornet","year":"2012","venue":"International Conference of the Learning Sciences","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Teh et al., 2005] Yee W Teh, Michael I Jordan, Matthew J Beal, and David M Blei. Sharing clusters among related groups: Hierarchical dirichlet processes. In Advances in neural information processing systems, pages 1385-1392, 2005.","title":"Sharing clusters among related groups: Hierarchical dirichlet processes","authors":"Yee W Teh, Michael I Jordan, Matthew J Beal, David M Blei","year":"2005","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Wu et al., 2018] Mike Wu, Michael C Hughes, Sonali Parb- hoo, Maurizio Zazzi, Volker Roth, and Finale Doshi-Velez. Beyond sparsity: Tree regularization of deep models for interpretability. In Thirty-Second AAAI Conference on Ar- tificial Intelligence, 2018.","title":"Beyond sparsity: Tree regularization of deep models for interpretability","authors":"Mike Wu, C Michael, Sonali Hughes, Maurizio Parbhoo, Zazzi, Roth Volker, Finale Doshi-Velez","year":"2018","venue":"Thirty-Second AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}