{"type":"ScholarlyArticle","title":"Deep Text Classification Can be Fooled","authors":[{"name":"Bin Liang","slug":"bin-liang","given":"Bin","family":"Liang","affiliation":"School of Information , Renmin University of China , Beijing , China"},{"name":"Hongcheng Li","slug":"hongcheng-li","given":"Hongcheng","family":"Li","affiliation":"School of Information , Renmin University of China , Beijing , China"},{"name":"Miaoqiang Su","slug":"miaoqiang-su","given":"Miaoqiang","family":"Su","affiliation":"School of Information , Renmin University of China , Beijing , China; Key laboratory of Data Engineering and Knowledge Engineering , MOE , Beijing , China"},{"name":"Pan Bian","slug":"pan-bian","given":"Pan","family":"Bian","affiliation":""},{"name":"Xirong Li","slug":"xirong-li","given":"Xirong","family":"Li","affiliation":"School of Information , Renmin University of China , Beijing , China"},{"name":"Wenchang Shi","slug":"wenchang-shi","given":"Wenchang","family":"Shi","affiliation":"School of Information , Renmin University of China , Beijing , China"}],"abstract":"In this paper, we present an effective method to craft text adversarial samples, revealing one important yet underestimated fact that DNN-based text classifiers are also prone to adversarial sample attack. Specifically, confronted with different adversarial scenarios, the text items that are important for classification are identified by computing the cost gradients of the input (white-box attack) or generating a series of occluded test samples (black-box attack). Based on these items, we design three perturbation strategies, namely insertion, modification, and removal, to generate adversarial samples. The experiment results show that the adversarial samples generated by our method can successfully fool both state-of-the-art character-level and word-level DNN-based text classifiers. The adversarial samples can be perturbed to any desirable classes without compromising their utilities. At the same time, the introduced perturbation is difficult to be perceived.","doi":"10.24963/ijcai.2018/585","url":"https://ijcai.org/proceedings/2018/585","pdf":"https://ijcai.org/proceedings/2018/0585.pdf","markdown":"https://ijcai.org/proceedings/2018/585.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/585","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Natural Language Processing","track":"Main track","number":585,"pages":{"first":4208,"last":4215},"keywords":["Natural Language Processing: Text Classification","Multidisciplinary Topics and Applications: Security and Privacy"],"videos":[],"references":[{"position":1,"raw":"[Arras et al., 2016] Leila Arras, Franziska Horn, Grégoire Montavon, Klaus-Robert Müller, and Wojciech Samek. Explaining predictions of non-linear classifiers in nlp. arXiv preprint arXiv:1606.07298, 2016.","title":"Explaining predictions of non-linear classifiers in nlp","authors":"Leila Arras, Franziska Horn, Grégoire Montavon, Klaus-Robert Müller, Wojciech Samek","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Atallah et al., 2001] Mikhail Atallah, Victor Raskin, Michael Crogan, Christian Hempelmann, Florian Ker- schbaum, Dina Mohamed, and Sanket Naik. Natural language watermarking: Design, analysis, and a proof-of- concept implementation. In Information Hiding, pages 185-200. Springer, 2001.","title":"Natural language watermarking: Design, analysis, and a proof-ofconcept implementation","authors":"Mikhail Atallah, Victor Raskin, Michael Crogan, Christian Hempelmann, Florian Kerschbaum, Dina Mohamed, Sanket Naik","year":"2001","venue":"Information Hiding","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Barzilay and McKeown, 2001] Regina Barzilay and Kath- leen R McKeown. Extracting paraphrases from a paral- lel corpus. In Proceedings of the 39th annual meeting on Association for Computational Linguistics, pages 50-57. Association for Computational Linguistics, 2001.","title":"Extracting paraphrases from a parallel corpus","authors":"Mckeown Barzilay, Regina Barzilay, Kathleen R Mckeown","year":"2001","venue":"Proceedings of the 39th annual meeting on Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Carlini and Wagner, 2017] Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In Security and Privacy (SP), 2017 IEEE Symposium on, pages 39-57. IEEE, 2017.","title":"Towards evaluating the robustness of neural networks","authors":"Wagner Carlini, Nicholas Carlini, David Wagner","year":"2017","venue":"Security and Privacy (SP)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Gao et al., 2018] Ji Gao, Jack Lanchantin, Mary Lou Soffa, and Yanjun Qi. Black-box generation of adversarial text sequences to evade deep learning classifiers. arXiv preprint arXiv:1801.04354, 2018.","title":"Black-box generation of adversarial text sequences to evade deep learning classifiers","authors":"Ji Gao, Jack Lanchantin, Mary Lou Soffa, Yanjun Qi","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gong et al., 2018] Zhitao Gong, Wenlu Wang, Bo Li, Dawn Song, and Wei-Shinn Ku. Adversarial texts with gradient methods. arXiv preprint arXiv:1801.07175, 2018.","title":"Adversarial texts with gradient methods","authors":"Gong","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Goodfellow et al., 2015] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harness- ing adversarial examples. In Proceedings of the 2015 International Conference on Learning Representations. Computational and Biological Learning Society, 2015.","title":"Explaining and harnessing adversarial examples","authors":"Ian J Goodfellow, Jonathon Shlens, Christian Szegedy","year":"2015","venue":"Proceedings of the 2015 International Conference on Learning Representations. Computational and Biological Learning Society","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Jia and Liang, 2017] Robin Jia and Percy Liang. Adversar- ial examples for evaluating reading comprehension sys- tems. arXiv preprint arXiv:1707.07328, 2017.","title":"Adversarial examples for evaluating reading comprehension systems","authors":"Liang Jia, Robin Jia, Percy Liang","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kereliuk et al., 2015] Corey Kereliuk, Bob L Sturm, and Jan Larsen. Deep learning and music adversaries. IEEE Transactions on Multimedia, 17(11):2059-2071, 2015.","title":"Deep learning and music adversaries","authors":"Corey Kereliuk, Bob L Sturm, Jan Larsen","year":"2015","venue":"IEEE Transactions on Multimedia","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kim, 2014] Yoon Kim. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Confer- ence on Empirical Methods in Natural Language Process- ing (EMNLP), pages 1746-1751. Association for Compu- tational Linguistics, 2014.","title":"Convolutional neural networks for sentence classification","authors":"Yoon Kim","year":"2014","venue":"Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Liang et al., 2017] Bin Liang, Hongcheng Li, Miaoqiang Su, Xirong Li, Wenchang Shi, and Xiaofeng Wang. De- tecting adversarial examples in deep networks with adap- tive noise reduction. arXiv preprint arXiv:1705.08378, 2017.","title":"Detecting adversarial examples in deep networks with adap-tive noise reduction","authors":"Bin Liang, Hongcheng Li, Miaoqiang Su, Xirong Li, Wenchang Shi, Xiaofeng Wang","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Mikolov et al., 2013] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed repre- sentations of words and phrases and their compositional- ity. In Advances in neural information processing systems, pages 3111-3119, 2013.","title":"Distributed representations of words and phrases and their compositionality","authors":"Mikolov","year":"2013","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Mitton, 1985] Roger Mitton. Cor- pora of misspellings for download. http://www.dcs.bbk.ac.uk/ ROGER/corpora.html, 1985.","title":"Corpora of misspellings for download","authors":"Roger Mitton","year":"1985","venue":null,"doi":null,"url":"http://www.dcs.bbk.ac.uk/ROGER/corpora.html","internal_url":null},{"position":14,"raw":"[Moosavi-Dezfooli et al., 2016] Seyed-Mohsen Moosavi- Dezfooli, Alhussein Fawzi, and Pascal Frossard. Deep- fool: a simple and accurate method to fool deep neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2574-2582, 2016.","title":"Deepfool: a simple and accurate method to fool deep neural networks","authors":"Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Pascal Frossard","year":"2016","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Nguyen et al., 2015] Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High con- fidence predictions for unrecognizable images. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 427-436, 2015.","title":"Deep neural networks are easily fooled: High confidence predictions for unrecognizable images","authors":"Anh Nguyen, Jason Yosinski, Jeff Clune","year":"2015","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Papernot et al., 2016] Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. Distillation as a defense to adversarial perturbations against deep neu- ral networks. In Security and Privacy (SP), 2016 IEEE Symposium on, pages 582-597. IEEE, 2016.","title":"Distillation as a defense to adversarial perturbations against deep neural networks","authors":"Nicolas Papernot, Patrick Mcdaniel, Xi Wu, Somesh Jha, Ananthram Swami","year":"2016","venue":"Security and Privacy (SP)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Slawski, 2014] Bill Slawski. Google turns to deep learning classification to fight web spam. http://www.seobythesea.com/2014/09/google-turns- deep-learning-classification-fight-web-spam/, sep 2014.","title":"Bill Slawski. Google turns to deep learning classification to fight web spam","authors":"Slawski","year":"2014","venue":null,"doi":null,"url":"http://www.seobythesea.com/2014/09/google-turns-deep-learning-classification-fight-web-spam/,sep2014","internal_url":null},{"position":18,"raw":"[Suchanek et al., 2007] Fabian M Suchanek, Gjergji Kas- neci, and Gerhard Weikum. Yago: A core of semantic knowledge unifying wordnet and wikipedia. In Proceed- ings of the 16th international conference on World Wide Web, pages 697-706. ACM, 2007.","title":"Yago: A core of semantic knowledge unifying wordnet and wikipedia","authors":"Gjergji Fabian M Suchanek, Gerhard Kasneci, Weikum","year":"2007","venue":"Proceedings of the 16th international conference on World Wide Web","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sutton et al., 2007] Michael Sutton, Adam Greene, and Pe- dram Amini. Fuzzing: brute force vulnerability discovery. Pearson Education, 2007.","title":"Fuzzing: brute force vulnerability discovery","authors":"Michael Sutton, Adam Greene, Pedram Amini","year":"2007","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Szegedy et al., 2014] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. 2014.","title":"Intriguing properties of neural networks","authors":"Szegedy","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Szegedy et al., 2015] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabi- novich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1-9, 2015.","title":"Going deeper with convolutions","authors":"Szegedy","year":"2015","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Topkara et al., 2006] Umut Topkara, Mercan Topkara, and Mikhail J Atallah. The hiding virtues of ambiguity: quan- tifiably resilient watermarking of natural language text through synonym substitutions. In Proceedings of the 8th workshop on Multimedia and security, pages 164-174. ACM, 2006.","title":"The hiding virtues of ambiguity: quantifiably resilient watermarking of natural language text through synonym substitutions","authors":"Umut Topkara, Mercan Topkara, Mikhail J Atallah","year":"2006","venue":"Proceedings of the 8th workshop on Multimedia and security","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Topkara et al., 2007] Mercan Topkara, Umut Topkara, and Mikhail J Atallah. Information hiding through errors: a confusing approach. In Security, Steganography, and Wa- termarking of Multimedia Contents, page 65050V, 2007.","title":"Information hiding through errors: a confusing approach","authors":"Topkara","year":"2007","venue":"Security, Steganography, and Watermarking of Multimedia Contents","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Vybornova and Macq, 2007] Olga Vybornova and Benoit Macq. Natural language watermarking and robust hashing based on presuppositional analysis. In Information Reuse and Integration, 2007. IRI 2007. IEEE International Con- ference on, pages 177-182. IEEE, 2007.","title":"Natural language watermarking and robust hashing based on presuppositional analysis","authors":"Olga Vybornova, Benoit Macq","year":"2007","venue":"Information Reuse and Integration","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Whittaker et al., 2010] Colin Whittaker, Brian Ryner, and Marria Nazif. Large-scale automatic classification of phishing pages. In NDSS, volume 10, page 2010, 2010.","title":"Large-scale automatic classification of phishing pages","authors":"Colin Whittaker, Brian Ryner, Marria Nazif","year":"2010","venue":"NDSS","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhang et al., 2015] Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In Advances in neural information process- ing systems, pages 649-657, 2015.","title":"Character-level convolutional networks for text classification","authors":"Xiang Zhang, Junbo Zhao, Yann Lecun","year":"2015","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}