{"type":"ScholarlyArticle","title":"Right for the Right Reasons: Training Differentiable Models by Constraining their Explanations","authors":[{"name":"Andrew Slavin Ross","slug":"andrew-slavin-ross","given":"Andrew Slavin","family":"Ross","affiliation":"Paulson School of Engineering and Applied Sciences , Harvard University , Cambridge , MA 02138 , USA"},{"name":"Michael C. Hughes","slug":"michael-c-hughes","given":"Michael C.","family":"Hughes","affiliation":"Paulson School of Engineering and Applied Sciences , Harvard University , Cambridge , MA 02138 , USA"},{"name":"Finale Doshi-Velez","slug":"finale-doshi-velez","given":"Finale","family":"Doshi-Velez","affiliation":"Paulson School of Engineering and Applied Sciences , Harvard University , Cambridge , MA 02138 , USA"}],"abstract":"Expressive classifiers such as neural networks are among the most accurate supervised learning methods in use today, but their opaque decision boundaries make them difficult to trust in critical applications. We propose a method to explain the predictions of any differentiable model via the gradient of the class label with respect to the input (which provides a normal to the decision boundary). Not only is this approach orders of magnitude faster at identifying input dimensions of high sensitivity than sample-based perturbation methods (e.g. LIME), but it also lends itself to efficiently discovering multiple qualitatively different decision boundaries as well as decision boundaries that are consistent with expert annotation. On multiple datasets, we show our approach generalizes much better when test conditions differ from those in training.","doi":"10.24963/ijcai.2017/371","url":"https://ijcai.org/proceedings/2017/371","pdf":"https://ijcai.org/proceedings/2017/0371.pdf","markdown":"https://ijcai.org/proceedings/2017/371.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/371","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning A-R","track":"Main track","number":371,"pages":{"first":2662,"last":2670},"keywords":["Multidisciplinary Topics and Applications: Validation and Verification","Machine Learning: Neural Networks","Machine Learning: Machine Learning","Machine Learning: Learning Theory"],"videos":[],"references":[{"position":1,"raw":"[Adler et al., 2016] Philip Adler, Casey Falk, Sorelle A Friedler, Gabriel Rybeck, Carlos Scheidegger, Brandon Smith, and Suresh Venkatasubramanian. Auditing black-box models by obscuring features. arXiv preprint arXiv:1602.07043, 2016.","title":"Auditing black-box models by obscuring features","authors":"Philip Adler, Casey Falk, Sorelle A Friedler, Gabriel Rybeck, Carlos Scheidegger, Brandon Smith, Suresh Venkatasubramanian","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bach et al., 2015] Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wo- jciech Samek. On pixel-wise explanations for non-linear clas- sifier decisions by layer-wise relevance propagation. PloS one, 10(7):e0130140, 2015.","title":"On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation","authors":"Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, Wojciech Samek","year":"2015","venue":"PloS one","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Baehrens et al., 2010] David Baehrens, Timon Schroeter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, and Klaus-Robert Müller. How to explain individual classification decisions. Journal of Machine Learning Research, 11(Jun):1803-1831, 2010.","title":"How to explain individual classification decisions","authors":"David Baehrens, Timon Schroeter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, Klaus-Robert Müller","year":"2010","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Caruana et al., 2015] Rich Caruana, Yin Lou, Johannes Gehrke, Paul Koch, Marc Sturm, and Noemie Elhadad. Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission. In Proceedings of the 21th ACM SIGKDD Inter- national Conference on Knowledge Discovery and Data Mining, pages 1721-1730. ACM, 2015.","title":"Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission","authors":"Rich Caruana, Yin Lou, Johannes Gehrke, Paul Koch, Marc Sturm, Noemie Elhadad","year":"2015","venue":"Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Donahue and Grauman, 2011] Jeff Donahue and Kristen Grauman. Annotator rationales for visual recognition. In 2011 International Conference on Computer Vision, pages 1395-1402. IEEE, 2011.","title":"Annotator rationales for visual recognition","authors":"Grauman Donahue, Jeff Donahue, Kristen Grauman","year":"2011","venue":"2011 International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Doshi-Velez and Kim, 2017] Finale Doshi-Velez and Been Kim. Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08608, 2017.","title":"Finale Doshi-Velez and Been Kim. Towards a rigorous science of interpretable machine learning","authors":"Kim Doshi-Velez","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Drucker and Le Cun, 1992] Harris Drucker and Yann Le Cun. Im- proving generalization performance using double backpropagation. IEEE Transactions on Neural Networks, 3(6):991-997, 1992.","title":"Harris Drucker and Yann Le Cun. Improving generalization performance using double backpropagation","authors":"Le Drucker, Cun","year":"1992","venue":"IEEE Transactions on Neural Networks","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dwork et al., 2012] Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Fairness through awareness. In Proceedings of the 3rd Innovations in Theoretical Computer Science Conference, pages 214-226. ACM, 2012.","title":"Fairness through awareness","authors":"Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, Richard Zemel","year":"2012","venue":"Proceedings of the 3rd Innovations in Theoretical Computer Science Conference","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Fong and Vedaldi, 2017] Ruth Fong and Andrea Vedaldi. Inter- pretable explanations of black boxes by meaningful perturbation. arXiv preprint arXiv:1704.03296, 2017.","title":"Interpretable explanations of black boxes by meaningful perturbation","authors":"Vedaldi Fong, Ruth Fong, Andrea Vedaldi","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hechtlinger, 2016] Yotam Hechtlinger. Interpretation of prediction models using the input gradient. arXiv preprint arXiv:1611.07634, 2016.","title":"Interpretation of prediction models using the input gradient","authors":"Yotam Hechtlinger","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Keil, 2006] Frank C Keil. Explanation and understanding. Annu. Rev. Psychol., 57:227-254, 2006.","title":"Explanation and understanding","authors":"Keil Frank","year":"2006","venue":"Annu. Rev. Psychol","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kingma and Ba, 2014] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"Diederik Kingma, Jimmy Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[LeCun et al., 2010] Yann LeCun, Corinna Cortes, and Christo- pher J.C. Burges. The MNIST database of handwritten digits. http://yann.lecun.com/exdb/mnist/, 2010.","title":"The MNIST database of handwritten digits","authors":"Yann Lecun, Corinna Cortes, Christopher J C Burges","year":null,"venue":null,"doi":null,"url":"http://yann.lecun.com/exdb/mnist/,2010","internal_url":null},{"position":14,"raw":"[Lei et al., 2016] Tao Lei, Regina Barzilay, and Tommi Jaakkola. Ra- tionalizing neural predictions. arXiv preprint arXiv:1606.04155, 2016.","title":"Rationalizing neural predictions","authors":"Tao Lei, Regina Barzilay, Tommi Jaakkola","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2015] Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Juraf- sky. Visualizing and understanding neural models in NLP. arXiv preprint arXiv:1506.01066, 2015.","title":"Visualizing and understanding neural models in NLP","authors":"Jiwei Li, Xinlei Chen, Eduard Hovy, Dan Jurafsky","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Li et al., 2016] Jiwei Li, Will Monroe, and Dan Jurafsky. Under- standing neural networks through representation erasure. arXiv preprint arXiv:1612.08220, 2016.","title":"Understanding neural networks through representation erasure","authors":"Jiwei Li, Will Monroe, Dan Jurafsky","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lichman, 2013] M. Lichman. UCI machine learning repository. http://archive.ics.uci.edu/ml, 2013.","title":"UCI machine learning repository","authors":"M Lichman","year":null,"venue":null,"doi":null,"url":"http://archive.ics.uci.edu/ml,2013","internal_url":null},{"position":18,"raw":"[Lipton, 2016] Zachary C Lipton. The mythos of model inter- pretability. arXiv preprint arXiv:1606.03490, 2016.","title":"The mythos of model interpretability","authors":"Zachary C Lipton","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Lundberg and Lee, 2016] Scott Lundberg and Su-In Lee. An unex- pected unity among methods for interpreting model predictions. arXiv preprint arXiv:1611.07478, 2016.","title":"An unexpected unity among methods for interpreting model predictions","authors":"Lee ; Lundberg, Scott Lundberg, Su-In Lee","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Mclaurin et al., 2017] Dougal Mclaurin, David Duvenaud, and Matt Johnson. Autograd. https://github.com/HIPS/ autograd, 2017.","title":"Autograd","authors":"Dougal Mclaurin, David Duvenaud, Matt Johnson","year":"2017","venue":null,"doi":null,"url":"https://github.com/HIPS/autograd","internal_url":null},{"position":21,"raw":"[Montavon et al., 2017] Grégoire Montavon, Sebastian Lapuschkin, Alexander Binder, Wojciech Samek, and Klaus-Robert Müller. Explaining nonlinear classification decisions with deep Taylor decomposition. Pattern Recognition, 65:211-222, 2017.","title":"Explaining nonlinear classification decisions with deep Taylor decomposition","authors":"Grégoire Montavon, Sebastian Lapuschkin, Alexander Binder, Wojciech Samek, Klaus-Robert Müller","year":"2017","venue":"Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Papernot et al., 2016] Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. Distillation as a defense to adversarial perturbations against deep neural networks. In Security and Privacy (SP), 2016 IEEE Symposium on, pages 582- 597. IEEE, 2016.","title":"Distillation as a defense to adversarial perturbations against deep neural networks","authors":"Nicolas Papernot, Patrick Mcdaniel, Xi Wu, Somesh Jha, Ananthram Swami","year":"2016","venue":"Security and Privacy (SP)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Ribeiro et al., 2016] Marco Tulio Ribeiro, Sameer Singh, and Car- los Guestrin. Why should I trust you?: Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD Inter- national Conference on Knowledge Discovery and Data Mining, pages 1135-1144. ACM, 2016.","title":"Why should I trust you?: Explaining the predictions of any classifier","authors":"Marco Tulio Ribeiro, Sameer Singh, Carlos Guestrin","year":"2016","venue":"Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Ribeiro, 2016] Marco Tulio Ribeiro. LIME. https://github. com/marcotcr/lime, 2016.","title":"LIME","authors":"Marco Tulio, Ribeiro","year":null,"venue":null,"doi":null,"url":"https://github.com/marcotcr/lime,2016","internal_url":null},{"position":25,"raw":"[Selvaraju et al., 2016] Ramprasaath R Selvaraju, Abhishek Das, Ramakrishna Vedantam, Michael Cogswell, Devi Parikh, and Dhruv Batra. Grad-CAM: Why did you say that? arXiv preprint arXiv:1611.07450, 2016.","title":"Grad-CAM: Why did you say that","authors":"Abhishek Ramprasaath R Selvaraju, Ramakrishna Das, Michael Vedantam, Devi Cogswell, Dhruv Parikh, Batra","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Shrikumar et al., 2016] Avanti Shrikumar, Peyton Greenside, Anna Shcherbina, and Anshul Kundaje. Not just a black box: Learn- ing important features through propagating activation differences. arXiv preprint arXiv:1605.01713, 2016.","title":"Not just a black box: Learning important features through propagating activation differences","authors":"Avanti Shrikumar, Peyton Greenside, Anna Shcherbina, Anshul Kundaje","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Shrikumar et al., 2017] Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. arXiv preprint arXiv:1704.02685, 2017.","title":"Learning important features through propagating activation differences","authors":"Avanti Shrikumar, Peyton Greenside, Anshul Kundaje","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Simonyan et al., 2013] Karen Simonyan, Andrea Vedaldi, and An- drew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034, 2013.","title":"Deep inside convolutional networks: Visualising image classification models and saliency maps","authors":"Karen Simonyan, Andrea Vedaldi, Andrew Zisserman","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Singh et al., 2016] Sameer Singh, Marco Tulio Ribeiro, and Carlos Guestrin. Programs as black-box explanations. arXiv preprint arXiv:1611.07579, 2016.","title":"Programs as black-box explanations","authors":"Sameer Singh, Marco Tulio Ribeiro, Carlos Guestrin","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Sundararajan et al., 2016] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Gradients of counterfactuals. arXiv preprint arXiv:1611.02639, 2016.","title":"Gradients of counterfactuals","authors":"Mukund Sundararajan, Ankur Taly, Qiqi Yan","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zafar et al., 2016] Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rodriguez, and Krishna P Gummadi. Fairness beyond disparate treatment & disparate impact: Learning classification without disparate mistreatment. arXiv preprint arXiv:1610.08452, 2016.","title":"Fairness beyond disparate treatment & disparate impact: Learning classification without disparate mistreatment","authors":"Zafar","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zaidan et al., 2007] Omar Zaidan, Jason Eisner, and Christine D Piatko. Using \"annotator rationales\" to improve machine learning for text categorization. In HLT-NAACL, pages 260-267, 2007.","title":"Using \"annotator rationales\" to improve machine learning for text categorization","authors":"Omar Zaidan, Jason Eisner, Christine D Piatko","year":"2007","venue":"HLT-NAACL","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zeiler and Fergus, 2014] Matthew D Zeiler and Rob Fergus. Visu- alizing and understanding convolutional networks. In European conference on computer vision, pages 818-833. Springer, 2014.","title":"Visualizing and understanding convolutional networks","authors":"D Matthew, Rob Zeiler, Fergus","year":"2014","venue":"European conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zhang et al., 2016] Ye Zhang, Iain Marshall, and Byron C Wal- lace. Rationale-augmented convolutional neural networks for text classification. arXiv preprint arXiv:1605.04469, 2016.","title":"Rationale-augmented convolutional neural networks for text classification","authors":"Zhang","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}