{"type":"ScholarlyArticle","title":"AttExplainer: Explain Transformer via Attention by Reinforcement Learning","authors":[{"name":"Runliang Niu","slug":"runliang-niu","given":"Runliang","family":"Niu","affiliation":"School of Artificial Intelligence , Jilin University"},{"name":"Zhepei Wei","slug":"zhepei-wei","given":"Zhepei","family":"Wei","affiliation":"School of Artificial Intelligence , Jilin University"},{"name":"Yan Wang","slug":"yan-wang","given":"Yan","family":"Wang","affiliation":"School of Artificial Intelligence , Jilin University; Key Laboratory of Symbol Computation and Knowledge Engineering of Ministry of Education , College of Computer Science and Technology , Jilin University"},{"name":"Qi Wang","slug":"qi-wang","given":"Qi","family":"Wang","affiliation":"School of Artificial Intelligence , Jilin University"}],"abstract":"Transformer and its variants, built based on attention mechanisms, have recently achieved remarkable performance in many NLP tasks. Most existing works on Transformer explanation tend to reveal and utilize the attention matrix with human subjective intuitions in a qualitative manner. However, the huge size of dimensions directly challenges these methods to quantitatively analyze the attention matrix. Therefore, in this paper, we propose a novel reinforcement learning (RL) based framework for Transformer explanation via attention matrix, namely AttExplainer. The RL agent learns to perform step-by-step masking operations by observing the change in attention matrices. We have adapted our method to two scenarios, perturbation-based model explanation and text adversarial attack. Experiments on three widely used text classification benchmarks validate the effectiveness of the proposed method compared to state-of-the-art baselines. Additional studies show that our method is highly transferable and consistent with human intuition. The code of this paper is available at https://github.com/niuzaisheng/AttExplainer .","doi":"10.24963/ijcai.2022/102","url":"https://ijcai.org/proceedings/2022/102","pdf":"https://ijcai.org/proceedings/2022/0102.pdf","markdown":"https://ijcai.org/proceedings/2022/102.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/102","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"AI Ethics, Trust, Fairness","track":"Main Track","number":102,"pages":{"first":724,"last":731},"keywords":["Natural Language Processing: Text Classification","Natural Language Processing: Interpretability and Analysis of Models for NLP","AI Ethics, Trust, Fairness: Explainability and Interpretability"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987068","length_sec":73,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985211","length_sec":715,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Abnar and Zuidema, 2020] Samira Abnar and Willem Zuidema. Quantifying attention flow in transformers. In ACL, 2020. [Alzantot and others, 2018] Moustafa Alzantot et al. Gener- ating natural language adversarial examples. In EMNLP, 2018.","title":"Quantifying attention flow in transformers","authors":"Zuidema Abnar, Samira Abnar, Willem Zuidema","year":"2018","venue":"Generating natural language adversarial examples","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bowman and others, 2015] Samuel R. Bowman et al. A large annotated corpus for learning natural language in- ference. In EMNLP, 2015.","title":"A large annotated corpus for learning natural language inference","authors":"R Samuel, Bowman","year":"2015","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Castro et al., 2009] Javier Castro, Daniel Gómez, and Juan Tejada. Polynomial calculation of the shapley value based on sampling. Computers & Operations Research, 2009.","title":"Polynomial calculation of the shapley value based on sampling","authors":"Javier Castro, Daniel Gómez, Juan Tejada","year":"2009","venue":"Computers & Operations Research","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chefer et al., 2021] Hila Chefer, Shir Gur, and Lior Wolf. Transformer interpretability beyond attention visualiza- tion. In CVPR, 2021.","title":"Transformer interpretability beyond attention visualization","authors":"Hila Chefer, Shir Gur, Lior Wolf","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Devlin and others, 2019] Jacob Devlin et al. BERT: Pre- training of deep bidirectional transformers for language understanding. In NAACL-HLT, 2019.","title":"BERT: Pretraining of deep bidirectional transformers for language understanding","authors":"Jacob Devlin","year":"2019","venue":"NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Ebrahimi et al., 2018] Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. HotFlip: White-box adversarial examples for text classification. In ACL, 2018.","title":"HotFlip: White-box adversarial examples for text classification","authors":"Javid Ebrahimi, Anyi Rao, Daniel Lowd, Dejing Dou","year":"2018","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Feng and others, 2018] Shi Feng et al. Pathologies of neural models make interpretations difficult. In EMNLP, 2018.","title":"Pathologies of neural models make interpretations difficult","authors":"Feng Shi","year":"2018","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gao et al., 2018] Ji Gao, Jack Lanchantin, Mary Lou Soffa, and Yanjun Qi. Black-box generation of adversarial text sequences to evade deep learning classifiers. In 2018 IEEE Security and Privacy Workshops (SPW), 2018.","title":"Black-box generation of adversarial text sequences to evade deep learning classifiers","authors":"Ji Gao, Jack Lanchantin, Mary Lou Soffa, Yanjun Qi","year":"2018","venue":"2018 IEEE Security and Privacy Workshops (SPW)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hewitt and Manning, 2019] John Hewitt and Christo- pher D. Manning. A structural probe for finding syntax in word representations. In NAACL-HLT, 2019.","title":"A structural probe for finding syntax in word representations","authors":"John Hewitt, Christopher D Manning","year":"2019","venue":"NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Htut et al., 2019] Phu Mon Htut, Jason Phang, Shikha Bor- dia, and Samuel R. Bowman. Do attention heads in bert track syntactic dependencies? arXiv preprint arXiv: 1911.12246, 2019.","title":"Do attention heads in bert track syntactic dependencies","authors":"Jason Phu Mon Htut, Shikha Phang, Samuel R Bordia, Bowman","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Iyyer et al., 2018] Mohit Iyyer, John Wieting, Kevin Gim- pel, and Luke Zettlemoyer. Adversarial example genera- tion with syntactically controlled paraphrase networks. In NAACL-HLT, 2018.","title":"Adversarial example generation with syntactically controlled paraphrase networks","authors":"Mohit Iyyer, John Wieting, Kevin Gimpel, Luke Zettlemoyer","year":"2018","venue":"NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Jain and Wallace, 2019] Sarthak Jain and Byron C. Wallace. Attention is not Explanation. In NAACL-HLT, 2019.","title":"Attention is not Explanation","authors":"Sarthak Jain, Byron C Wallace","year":"2019","venue":"NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kingma and Ba, 2015] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kokhlikyan et al., 2020] Narine Kokhlikyan, Vivek Miglani, Miguel Martin, Edward Wang, Bilal Alsal- lakh, Jonathan Reynolds, Alexander Melnikov, Natalia Kliushkina, Carlos Araya, Siqi Yan, and Orion Reblitz- Richardson. Captum: A unified and generic model interpretability library for pytorch. arXiv preprint arXiv:2009.07896, 2020.","title":"Captum: A unified and generic model interpretability library for pytorch","authors":"Narine Kokhlikyan, Vivek Miglani, Miguel Martin, Edward Wang, Bilal Alsallakh, Jonathan Reynolds, Alexander Melnikov, Natalia Kliushkina, Carlos Araya, Siqi Yan, Orion Reblitz-Richardson","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kovaleva et al., 2019] Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. Revealing the dark secrets of BERT. In EMNLP, 2019.","title":"Revealing the dark secrets of BERT","authors":"Olga Kovaleva, Alexey Romanov, Anna Rogers, Anna Rumshisky","year":"2019","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Li et al., 2016a] Jiwei Li, Will Monroe, and Dan Juraf- sky. Understanding neural networks through representa- tion erasure. arXiv preprint arXiv:1612.08220, 2016.","title":"Understanding neural networks through representation erasure","authors":"Li","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Li et al., 2016b] Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao. Deep rein- forcement learning for dialogue generation. In EMNLP, 2016.","title":"Deep reinforcement learning for dialogue generation","authors":"Li","year":"2016","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Li et al., 2019] Jinfeng Li, Shouling Ji, Tianyu Du, Bo Li, and Ting Wang. TextBugger: Generating adversarial text against real-world applications. In Proceedings 2019 Net- work and Distributed System Security Symposium. Internet Society, 2019.","title":"TextBugger: Generating adversarial text against real-world applications","authors":"Jinfeng Li, Shouling Ji, Tianyu Du, Bo Li, Ting Wang","year":"2019","venue":"Proceedings 2019 Network and Distributed System Security Symposium","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Li et al., 2020] Linyang Li, Ruotian Ma, Qipeng Guo, Xi- angyang Xue, and Xipeng Qiu. BERT-ATTACK: Adver- sarial attack against BERT using BERT. In EMNLP, 2020.","title":"BERT-ATTACK: Adversarial attack against BERT using BERT","authors":"Li","year":"2020","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Lundberg and Lee, 2017] Scott M. Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In NIPS, 2017.","title":"A unified approach to interpreting model predictions","authors":"M Scott, Su-In Lundberg, Lee","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Papernot et al., 2016] Nicolas Papernot, Patrick Mcdaniel, Ananthram Swami, and Richard E. Harang. Crafting ad- versarial input sequences for recurrent neural networks. MILCOM, 2016.","title":"Crafting adversarial input sequences for recurrent neural networks","authors":"Nicolas Papernot, Patrick Mcdaniel, Ananthram Swami, Richard E Harang","year":"2016","venue":"MILCOM","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Ren et al., 2019] Shuhuai Ren, Yihe Deng, Kun He, and Wanxiang Che. Generating natural language adversarial examples through probability weighted word saliency. In ACL, 2019.","title":"Generating natural language adversarial examples through probability weighted word saliency","authors":"Ren","year":"2019","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Ribeiro et al., 2016] Marco Túlio Ribeiro, Sameer Singh, and Carlos Guestrin. \"why should I trust you?\": Explain- ing the predictions of any classifier. In SIGKDD, 2016.","title":"why should I trust you?\": Explaining the predictions of any classifier","authors":"Marco Túlio Ribeiro, Sameer Singh, Carlos Guestrin","year":"2016","venue":"SIGKDD","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Saravia et al., 2018] Elvis Saravia, Hsien-Chi Toby Liu, Yen-Hao Huang, Junlin Wu, and Yi-Shin Chen. CARER: Contextualized affect representations for emotion recogni- tion. In EMNLP, 2018.","title":"CARER: Contextualized affect representations for emotion recognition","authors":"Saravia","year":"2018","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Schaul et al., 2016] Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience re- play. In ICLR, 2016.","title":"Prioritized experience replay","authors":"Tom Schaul, John Quan, Ioannis Antonoglou, David Silver","year":"2016","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Serrano and Smith, 2019] Sofia Serrano and Noah A. Smith. Is attention interpretable? In ACL, 2019.","title":"Is attention interpretable?","authors":"Smith ; Serrano, Sofia Serrano, Noah A Smith","year":"2019","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Vaswani and others, 2017] Ashish Vaswani et al. Attention is all you need. In NIPS, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wallace and others, 2019] Eric Wallace et al. AllenNLP in- terpret: A framework for explaining predictions of NLP models. In EMNLP, 2019.","title":"AllenNLP interpret: A framework for explaining predictions of NLP models","authors":"Eric Wallace","year":"2019","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wang and others, 2019] Alex Wang et al. GLUE: A multi- task benchmark and analysis platform for natural language understanding. In ICLR, 2019.","title":"GLUE: A multitask benchmark and analysis platform for natural language understanding","authors":"Alex Wang","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wiegreffe and Pinter, 2019] Sarah Wiegreffe and Yuval Pin- ter. Attention is not not explanation. In EMNLP, 2019.","title":"Attention is not not explanation","authors":"Sarah Wiegreffe, Yuval Pinter","year":"2019","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Wolf et al., 2019] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771, 2019.","title":"Huggingface's transformers: State-of-the-art natural language processing","authors":"Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Clara Patrick Von Platen, Yacine Ma, Julien Jernite, Canwen Plu, Teven Xu, Sylvain Le Scao, Mariama Gugger, Quentin Drame, Alexander M Lhoest, Rush","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Wu et al., 2020] Zhiyong Wu, Yun Chen, Ben Kao, and Qun Liu. Perturbed masking: Parameter-free probing for ana- lyzing and interpreting BERT. In ACL, 2020.","title":"Perturbed masking: Parameter-free probing for analyzing and interpreting BERT","authors":"Zhiyong Wu, Yun Chen, Ben Kao, Qun Liu","year":"2020","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Xiong et al., 2018] Caiming Xiong, Victor Zhong, and Richard Socher. DCN+: mixed objective and deep residual coattention for question answering. In ICLR, 2018.","title":"DCN+: mixed objective and deep residual coattention for question answering","authors":"Xiong","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Xu et al., 2019] Jingjing Xu, Liang Zhao, Hanqi Yan, Qi Zeng, Yun Liang, and Xu Sun. LexicalAT: Lexical- based adversarial reinforcement training for robust senti- ment classification. In EMNLP, 2019.","title":"LexicalAT: Lexicalbased adversarial reinforcement training for robust sentiment classification","authors":"Jingjing Xu, Liang Zhao, Hanqi Yan, Qi Zeng, Yun Liang, Xu Sun","year":"2019","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Zang et al., 2020] Yuan Zang, Fanchao Qi, Chenghao Yang, Zhiyuan Liu, Meng Zhang, Qun Liu, and Maosong Sun. Word-level textual adversarial attacking as combinatorial optimization. In ACL, 2020.","title":"Word-level textual adversarial attacking as combinatorial optimization","authors":"Yuan Zang, Fanchao Qi, Chenghao Yang, Zhiyuan Liu, Meng Zhang, Qun Liu, Maosong Sun","year":"2020","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Zeiler and Fergus, 2014] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In ECCV, 2014.","title":"Visualizing and understanding convolutional networks","authors":"D Matthew, Rob Zeiler, Fergus","year":"2014","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Zeng and others, 2021] Guoyang Zeng et al. OpenAttack: An open-source textual adversarial attack toolkit. In ACL, 2021.","title":"OpenAttack: An open-source textual adversarial attack toolkit","authors":"Guoyang Zeng, Zeng","year":"2021","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Zhao et al., 2018] Zhengli Zhao, Dheeru Dua, and Sameer Singh. Generating natural adversarial examples. In ICLR, 2018.","title":"Generating natural adversarial examples","authors":"Zhao","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Zhong et al., 2017] Victor Zhong, Caiming Xiong, and Richard Socher. Seq2sql: Generating structured queries from natural language using reinforcement learning. arXiv preprint arXiv:1709.00103, 2017.","title":"Seq2sql: Generating structured queries from natural language using reinforcement learning","authors":"Victor Zhong, Caiming Xiong, Richard Socher","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Zou et al., 2020] Wei Zou, Shujian Huang, Jun Xie, Xinyu Dai, and Jiajun Chen. A reinforced generation of adver- sarial examples for neural machine translation. In ACL, 2020.","title":"A reinforced generation of adversarial examples for neural machine translation","authors":"Wei Zou, Shujian Huang, Jun Xie, Xinyu Dai, Jiajun Chen","year":"2020","venue":"ACL","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}