{"type":"ScholarlyArticle","title":"Explainable Inference on Sequential Data via Memory-Tracking","authors":[{"name":"Biagio La Rosa","slug":"biagio-la-rosa","given":"Biagio","family":"La Rosa","affiliation":"Sapienza University of Rome , Italy"},{"name":"Roberto Capobianco","slug":"roberto-capobianco","given":"Roberto","family":"Capobianco","affiliation":"Sapienza University of Rome , Italy; Sony AI"},{"name":"Daniele Nardi","slug":"daniele-nardi","given":"Daniele","family":"Nardi","affiliation":"Sapienza University of Rome , Italy"}],"abstract":"In this paper we present a novel mechanism to\r\nget explanations that allow to better understand\r\nnetwork predictions when dealing with sequential\r\ndata. Specifically, we adopt memory-based networks\r\n— Differential Neural Computers — to exploit \r\ntheir capability of storing data in memory and\r\nreusing it for inference. By tracking both the memory \r\naccess at prediction time, and the information\r\nstored by the network at each step of the input\r\nsequence, we can retrieve the most relevant input\r\nsteps associated to each prediction. We validate\r\nour approach (1) on a modified T-maze, which is a\r\nnon-Markovian discrete control task evaluating an\r\nalgorithm’s ability to correlate events far apart in\r\nhistory, and (2) on the Story Cloze Test, which is\r\na commonsense reasoning framework for evaluating \r\nstory understanding that requires a system to\r\nchoose the correct ending to a four-sentence story.\r\nOur results show that we are able to explain agent’s\r\ndecisions in (1) and to reconstruct the most relevant\r\nsentences used by the network to select the story\r\nending in (2). Additionally, we show not only that\r\nby removing those sentences the network prediction\r\nchanges, but also that the same are sufficient to\r\nreproduce the inference.","doi":"10.24963/ijcai.2020/278","url":"https://ijcai.org/proceedings/2020/278","pdf":"https://ijcai.org/proceedings/2020/0278.pdf","markdown":"https://ijcai.org/proceedings/2020/278.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/278","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":278,"pages":{"first":2006,"last":2013},"keywords":["Machine Learning: Interpretability","Machine Learning: Deep Learning","Machine Learning: Explainable Machine Learning","Machine Learning: Deep Learning: Sequence Modeling"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24861/file_24861.m3u8","length_sec":294,"provider":"hls"}],"references":[{"position":1,"raw":"[Adebayo et al., 2018] Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. In Proc. of the 32nd Int. Conf. on Neural Information Processing Systems, page 9525-9536, Red Hook, NY, USA, 2018.","title":"Sanity checks for saliency maps","authors":"Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, Been Kim","year":"2018","venue":"Proc. of the 32nd Int. Conf. on Neural Information Processing Systems, page 9525-9536","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bach et al., 2015] Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojciech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLOS ONE, 10(7):e0130140, 2015.","title":"On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation","authors":"Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, Wojciech Samek","year":"2015","venue":"PLOS ONE","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Baehrens et al., 2010] David Baehrens, Timon Schroeter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, and Klaus-Robert Müller. How to explain individual classi- fication decisions. J. Mach. Learn. Res., 11:1803-1831, 2010.","title":"How to explain individual classification decisions","authors":"David Baehrens, Timon Schroeter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, Klaus-Robert Müller","year":"2010","venue":"J. Mach. Learn. Res","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bakker, 2002] Bram Bakker. Reinforcement learning with long short-term memory. In Advances in Neural Informa- tion Processing Systems, pages 1475-1482, 2002.","title":"Reinforcement learning with long short-term memory","authors":"Bram Bakker, Bakker","year":"2002","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cai et al., 2018] Qi Cai, Yingwei Pan, Ting Yao, Cheng- gang Yan, and Tao Mei. Memory matching networks for one-shot image recognition. In 2018 IEEE/CVF Conf. on Computer Vision and Pattern Recognition, 2018.","title":"Memory matching networks for one-shot image recognition","authors":"Qi Cai, Yingwei Pan, Ting Yao, Chenggang Yan, Tao Mei","year":"2018","venue":"IEEE/CVF Conf. on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Franke et al., 2018] Jörg Franke, Jan Niehues, and Alex Waibel. Robust and scalable differentiable neural com- puter for question answering. In Proc. of the Workshop on Machine Reading for Question Answering, pages 47-59, 2018.","title":"Robust and scalable differentiable neural computer for question answering","authors":"Franke","year":"2018","venue":"Proc. of the Workshop on Machine Reading for Question Answering","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Graves et al., 2016] Alex Graves, Greg Wayne, Mal- colm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska-Barwińska, Sergio Gómez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, et al. Hybrid computing using a neural network with dynamic external memory. Nature, 538(7626):471, 2016.","title":"Hybrid computing using a neural network with dynamic external memory","authors":"Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska-Barwińska, Sergio Gómez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou","year":"2016","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Guidotti et al., 2018] Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. A survey of methods for explaining black box models. ACM Computing Surveys, 51(5):1-42, 2018.","title":"A survey of methods for explaining black box models","authors":"Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, Dino Pedreschi","year":"2018","venue":"ACM Computing Surveys","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hausknecht and Stone, 2015] Matthew Hausknecht and Pe- ter Stone. Deep recurrent q-learning for partially observ- able mdps. In 2015 AAAI Fall Symposium Series, 2015.","title":"Deep recurrent q-learning for partially observable mdps","authors":"Matthew Hausknecht, Peter Stone","year":"2015","venue":"AAAI Fall Symposium Series","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lundberg and Lee, 2017] Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Advances in Neural Information Processing Systems, pages 4765-4774, 2017.","title":"A unified approach to interpreting model predictions","authors":"M Scott, Su-In Lundberg, Lee","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Ma et al., 2018] Chao Ma, Chunhua Shen, Anthony Dick, Qi Wu, Peng Wang, Anton van den Hengel, and Ian Reid. Visual question answering with memory-augmented net- works. In 2018 IEEE/CVF Conf. on Computer Vision and Pattern Recognition, 2018.","title":"Visual question answering with memory-augmented networks","authors":"Chao Ma, Chunhua Shen, Anthony Dick, Qi Wu, Peng Wang, Anton Van Den, Ian Hengel, Reid","year":"2018","venue":"IEEE/CVF Conf. on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Manning et al., 2014] Christopher Manning, Mihai Sur- deanu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky. The stanford corenlp natural language processing toolkit. In Proc. of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 55-60, 2014.","title":"The stanford corenlp natural language processing toolkit","authors":"Christopher Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, David Mcclosky","year":"2014","venue":"Proc. of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Mihaylov and Frank, 2017] Todor Mihaylov and Anette Frank. Story cloze ending selection baselines and data ex- amination. LSDSem 2017, page 87, 2017.","title":"Story cloze ending selection baselines and data examination","authors":"Todor Mihaylov, Anette Frank","year":"2017","venue":"LSDSem","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mikolov et al., 2013] Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word representations. In Proc. of the 2013 Conf. of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, pages 746-751, 2013.","title":"Linguistic regularities in continuous space word representations","authors":"Mikolov","year":"2013","venue":"Proc. of the 2013 Conf. of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Montavon et al., 2019] Grégoire Montavon, Alexander Binder, Sebastian Lapuschkin, Wojciech Samek, and Klaus-Robert Müller. Layer-wise relevance propagation: An overview. In Explainable AI: Interpreting, Explaining and Visualizing Deep Learning, pages 193-209. Springer International Publishing, 2019.","title":"Layer-wise relevance propagation: An overview","authors":"Grégoire Montavon, Alexander Binder, Sebastian Lapuschkin, Wojciech Samek, Klaus-Robert Müller","year":"2019","venue":"Explainable AI: Interpreting, Explaining and Visualizing Deep Learning","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Mostafazadeh et al., 2016] Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proc. of the 2016 Conf. of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Language Technologies, pages 839-849, San Diego, California, 2016.","title":"A corpus and cloze evaluation for deeper understanding of commonsense stories","authors":"Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, James Allen","year":"2016","venue":"Proc. of the 2016 Conf. of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Ribeiro et al., 2016] Marco Ribeiro, Sameer Singh, and Carlos Guestrin. \"why should i trust you?\": Explaining the predictions of any classifier. In Proc. of the 2016 Conf. of the North American Chapter of the Association for Com- putational Linguistics: Demonstrations, 2016.","title":"why should i trust you?\": Explaining the predictions of any classifier","authors":"Ribeiro","year":"2016","venue":"Proc. of the 2016 Conf. of the North American Chapter of the Association for Computational Linguistics: Demonstrations","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Ribeiro et al., 2018] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Anchors: High-precision model- agnostic explanations. In AAAI Conf. on Artificial Intel- ligence, 2018.","title":"Anchors: High-precision modelagnostic explanations","authors":"Marco Tulio Ribeiro, Sameer Singh, Carlos Guestrin","year":"2018","venue":"AAAI Conf. on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Samek and Müller, 2019] Wojciech Samek and Klaus- Robert Müller. Towards explainable artificial intelligence. In Explainable AI: Interpreting, Explaining and Visual- izing Deep Learning, pages 5-22. Springer International Publishing, 2019.","title":"Towards explainable artificial intelligence","authors":"Müller ; Samek, Klaus-Robert Müller","year":"2019","venue":"Explainable AI: Interpreting, Explaining and Visualizing Deep Learning","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Samek et al., 2018] W. Samek, T. Wiegand, and K. Müller. Explainable artificial intelligence: Understanding, visual- izing and interpreting deep learning models. ITU Journal: ICT Discoveries -Special Issue 1 -The Impact of Arti- ficial Intelligence (AI) on Communication Networks and Services, 1(1):39-48, 2018.","title":"Explainable artificial intelligence: Understanding, visualizing and interpreting deep learning models","authors":"W Samek, T Wiegand, K Müller","year":"2018","venue":"ITU Journal: ICT Discoveries -Special Issue 1 -The Impact of Artificial Intelligence (AI) on Communication Networks and Services","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Santoro et al., 2016] Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy P. Lillicrap. One-shot learning with memory-augmented neural networks. CoRR, abs/1605.06065, 2016.","title":"One-shot learning with memory-augmented neural networks","authors":"Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, Timothy P Lillicrap","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Shih et al., 2018] Andy Shih, Arthur Choi, and Adnan Dar- wiche. A symbolic approach to explaining bayesian net- work classifiers. In Proc. of the Twenty-Seventh Interna- tional Joint Conf. on Artificial Intelligence, 2018.","title":"A symbolic approach to explaining bayesian network classifiers","authors":"Andy Shih, Arthur Choi, Adnan Darwiche","year":"2018","venue":"Proc. of the Twenty-Seventh International Joint Conf. on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2018/708"},{"position":23,"raw":"[Springenberg et al., 2014] Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin A. Ried- miller. Striving for simplicity: The all convolutional net. CoRR, abs/1412.6806, 2014.","title":"Striving for simplicity: The all convolutional net","authors":"Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, Martin A Riedmiller","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Srinivasan et al., 2018] S. Srinivasan, R. Arora, and M. Riedl. A simple and effective approach to the story cloze test. In Proc. of the 2018 Conf. of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, volume 2, pages 92-96, 2018.","title":"A simple and effective approach to the story cloze test","authors":"S Srinivasan, R Arora, M Riedl","year":"2018","venue":"Proc. of the 2018 Conf. of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wierstra et al., 2007] Daan Wierstra, Alexander Foerster, Jan Peters, and Juergen Schmidhuber. Solving deep mem- ory pomdps with recurrent policy gradients. In Int. Conf. on Artificial Neural Networks, pages 697-706. Springer, 2007.","title":"Solving deep memory pomdps with recurrent policy gradients","authors":"Daan Wierstra, Alexander Foerster, Jan Peters, Juergen Schmidhuber","year":"2007","venue":"Int. Conf. on Artificial Neural Networks","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zeiler and Fergus, 2014] Matthew D. Zeiler and Rob Fer- gus. Visualizing and understanding convolutional net- works. In European Conf. on Computer Vision -ECCV 2014, pages 818-833. Springer International Publishing, 2014.","title":"Visualizing and understanding convolutional networks","authors":"Matthew D Zeiler, Rob Fergus","year":"2014","venue":"European Conf. on Computer Vision -ECCV 2014","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhou et al., 2016] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In 2016 IEEE Conf. on Computer Vision and Pattern Recognition, 2016.","title":"Learning deep features for discriminative localization","authors":"Zhou","year":"2016","venue":"IEEE Conf. on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}