{"type":"ScholarlyArticle","title":"Interpretable Adversarial Perturbation in Input Embedding Space for Text","authors":[{"name":"Motoki Sato","slug":"motoki-sato","given":"Motoki","family":"Sato","affiliation":"Preferred Networks, Inc. ,"},{"name":"Jun Suzuki","slug":"jun-suzuki","given":"Jun","family":"Suzuki","affiliation":"NTT Communication Science Laboratories ,; RIKEN Center for Advanced Intelligence Project; Tohoku University."},{"name":"Hiroyuki Shindo","slug":"hiroyuki-shindo","given":"Hiroyuki","family":"Shindo","affiliation":"Nara Institute of Science and Technology ,; RIKEN Center for Advanced Intelligence Project"},{"name":"Yuji Matsumoto","slug":"yuji-matsumoto","given":"Yuji","family":"Matsumoto","affiliation":"Nara Institute of Science and Technology ,; RIKEN Center for Advanced Intelligence Project"}],"abstract":"Following great success in the image processing field, the idea of adversarial training has been applied to tasks in the natural language processing (NLP) field.\r\nOne promising approach directly applies adversarial training developed in the image processing field to the input word embedding space instead of the discrete input space of texts.\r\nHowever, this approach abandons such interpretability as generating adversarial texts to significantly improve the performance of NLP tasks.\r\nThis paper restores interpretability to such methods by restricting the directions of perturbations toward the existing words in the input embedding space.\r\nAs a result, we can straightforwardly reconstruct each input with perturbations to an actual text by considering the perturbations to be the replacement of words in the sentence while maintaining or even improving the task performance.","doi":"10.24963/ijcai.2018/601","url":"https://ijcai.org/proceedings/2018/601","pdf":"https://ijcai.org/proceedings/2018/0601.pdf","markdown":"https://ijcai.org/proceedings/2018/601.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/601","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Natural Language Processing","track":"Main track","number":601,"pages":{"first":4323,"last":4330},"keywords":["Natural Language Processing: Natural Language Processing","Natural Language Processing: Sentiment Analysis and Text Mining","Machine Learning: Interpretability"],"videos":[],"references":[{"position":1,"raw":"[Belinkov and Bisk, 2018] Yonatan Belinkov and Yonatan Bisk. Synthetic and natural noise both break neural ma- chine translation. In ICLR, 2018.","title":"Synthetic and natural noise both break neural machine translation","authors":"Bisk Belinkov, Yonatan Belinkov, Yonatan Bisk","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bengio et al., 2000] Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. A neural proba- bilistic language model. Journal of Machine Learning Re- search, 3:1137-1155, 2000.","title":"A neural probabilistic language model","authors":"Yoshua Bengio, Réjean Ducharme, Pascal Vincent, Christian Janvin","year":"2000","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Caruana et al., 2000] Rich Caruana, Steve Lawrence, and C. Lee Giles. Overfitting in neural nets: Backpropagation, conjugate gradient, and early stopping. In NIPS, pages 402-408, 2000.","title":"Overfitting in neural nets: Backpropagation, conjugate gradient, and early stopping","authors":"Rich Caruana, Steve Lawrence, C Lee Giles","year":"2000","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dai and Le, 2015] Andrew M. Dai and Quoc V. Le. Semi- supervised sequence learning. In NIPS, pages 3079-3087, 2015.","title":"Semisupervised sequence learning","authors":"Andrew M Dai, V Quoc, Le","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Goodfellow et al., 2015] Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In ICLR, 2015.","title":"Explaining and harnessing adversarial examples","authors":"Ian J Goodfellow, Jonathon Shlens, Christian Szegedy","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Grave et al., 2017] Edouard Grave, Armand Joulin, Moustapha Cissé, David Grangier, and Hervé Jégou. Efficient softmax approximation for gpus. In ICML, 2017.","title":"Efficient softmax approximation for gpus","authors":"Grave","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735-1780, 1997.","title":"Long short-term memory","authors":"Schmidhuber Hochreiter, Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hosseini et al., 2017] Hossein Hosseini, Sreeram Kannan, Baosen Zhang, and Radha Poovendran. Deceiving google's perspective api built for detecting toxic com- ments. CoRR, abs/1702.08138, 2017.","title":"Deceiving google's perspective api built for detecting toxic comments","authors":"Hossein Hosseini, Sreeram Kannan, Baosen Zhang, Radha Poovendran","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jia and Liang, 2017] Robin Jia and Percy Liang. Adversar- ial examples for evaluating reading comprehension sys- tems. In EMNLP, pages 2021-2031, 2017.","title":"Adversarial examples for evaluating reading comprehension systems","authors":"Liang Jia, Robin Jia, Percy Liang","year":"2017","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Johnson and Zhang, 2015] Rie Johnson and Tong Zhang. Semi-supervised convolutional neural networks for text categorization via region embedding. In NIPS, volume 28, pages 919-927, 2015.","title":"Semi-supervised convolutional neural networks for text categorization via region embedding","authors":"Rie Johnson, Tong Zhang","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Johnson and Zhang, 2016] Rie Johnson and Tong Zhang. Supervised and semi-supervised text categorization using lstm for region embeddings. In ICML, pages 526-534, 2016.","title":"Supervised and semi-supervised text categorization using lstm for region embeddings","authors":"Rie Johnson, Tong Zhang","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kaneko et al., 2017] Masahiro Kaneko, Yuya Sakaizawa, and Mamoru Komachi. Grammatical error detection us- ing error-and grammaticality-specific word embeddings. In IJCNLP, volume 1, pages 40-48, 2017.","title":"Grammatical error detection using error-and grammaticality-specific word embeddings","authors":"Masahiro Kaneko, Yuya Sakaizawa, Mamoru Komachi","year":"2017","venue":"IJCNLP","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kingma and Ba, 2014] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Le and Mikolov, 2014] Quoc V. Le and Tomas Mikolov. Distributed representations of sentences and documents. In ICML, pages 1188-1196, 2014.","title":"Distributed representations of sentences and documents","authors":"V Quoc, Tomas Le, Mikolov","year":"2014","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lehmann et al., 2015] Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N. Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick van Kleef, Sören Auer, and Christian Bizer. Dbpedia - a large-scale, multilingual knowledge base extracted from wikipedia. Semantic Web, 6:167-195, 2015.","title":"Dbpediaa large-scale, multilingual knowledge base extracted from wikipedia","authors":"Lehmann","year":"2015","venue":"Semantic Web","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lewis et al., 2004] David D. Lewis, Yiming Yang, Tony G. Rose, and Fan Li. Rcv1: A new benchmark collection for text categorization research. Journal of Machine Learning Research, 5:361-397, 2004.","title":"Rcv1: A new benchmark collection for text categorization research","authors":"David D Lewis, Yiming Yang, Tony G Rose, Fan Li","year":"2004","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Maas et al., 2011] Andrew L. Maas, Raymond E. Daly, Pe- ter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In ACL, pages 142-150, 2011.","title":"Learning word vectors for sentiment analysis","authors":"Andrew L Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, Christopher Potts","year":"2011","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Miyato et al., 2016] Takeru Miyato, Shin ichi Maeda, Masanori Koyama, Ken Nakae, and Shin Ishii. Distri- butional smoothing with virtual adversarial training. In ICLR, 2016.","title":"Distributional smoothing with virtual adversarial training","authors":"Miyato","year":"2016","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Miyato et al., 2017] Takeru Miyato, Andrew M Dai, and Ian Goodfellow. Adversarial training methods for semi- supervised text classification. In ICLR, 2017.","title":"Adversarial training methods for semisupervised text classification","authors":"Miyato","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Nagata and Nakatani, 2010] Ryo Nagata and Kazuhide Nakatani. Evaluating performance of grammatical error detection to maximize learning effect. In COLING, pages 894-900, 2010.","title":"Evaluating performance of grammatical error detection to maximize learning effect","authors":"Nakatani ; ] Ryo Nagata, Kazuhide Nagata, Nakatani","year":"2010","venue":"COLING","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Pang and Lee, 2005] Bo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In ACL, pages 115-124, 2005.","title":"Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales","authors":"Lee ; Pang, Bo Pang, Lillian Lee","year":"2005","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Rei and Yannakoudakis, 2016] Marek Rei and Helen Yan- nakoudakis. Compositional sequence labeling models for error detection in learner writing. In ACL, pages 1181- 1191, 2016.","title":"Compositional sequence labeling models for error detection in learner writing","authors":"Yannakoudakis Rei, Marek Rei, Helen Yannakoudakis","year":"2016","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Rozovskaya et al., 2014] Alla Rozovskaya, Kai-Wei Chang, Mark Sammons, Dan Roth, and Nizar Habash. The illinois-columbia system in the conll-2014 shared task. In CoNLL Shared Task, pages 34-42, 2014.","title":"The illinois-columbia system in the conll-2014 shared task","authors":"Alla Rozovskaya, Kai-Wei Chang, Mark Sammons, Dan Roth, Nizar Habash","year":"2014","venue":"CoNLL Shared Task","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Samanta and Mehta, 2017] Suranjana Samanta and Sameep Mehta. Towards crafting text adversarial samples. arXiv preprint arXiv:1707.02812, 2017.","title":"Towards crafting text adversarial samples","authors":"Mehta Samanta","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Srivastava et al., 2014] Nitish Srivastava, Geoffrey E. Hin- ton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15:1929-1958, 2014.","title":"Dropout: a simple way to prevent neural networks from overfitting","authors":"Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan Salakhutdinov","year":"2014","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Szegedy et al., 2014] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Er- han, Ian J. Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In ICLR, 2014.","title":"Intriguing properties of neural networks","authors":"Szegedy","year":"2014","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Tokui et al., 2015] Seiya Tokui, Kenta Oono, Shohei Hido, and Justin Clayton. Chainer: a next-generation open source framework for deep learning. In Proceedings of Workshop on Machine Learning Systems (LearningSys) in The Twenty-ninth Annual Conference on Neural Informa- tion Processing Systems (NIPS), volume 5, 2015.","title":"Chainer: a next-generation open source framework for deep learning","authors":"Tokui","year":"2015","venue":"Proceedings of Workshop on Machine Learning Systems (LearningSys) in The Twenty-ninth Annual Conference on Neural Information Processing Systems (NIPS)","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Yannakoudakis et al., 2011] Helen Yannakoudakis, Ted Briscoe, and Ben Medlock. A new dataset and method for automatically grading esol texts. In ACL, pages 180-189. ACL, 2011.","title":"A new dataset and method for automatically grading esol texts","authors":"Helen Yannakoudakis, Ted Briscoe, Ben Medlock","year":"2011","venue":"ACL","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}