{"type":"ScholarlyArticle","title":"A Survey on Out-of-Distribution Evaluation of Neural NLP Models","authors":[{"name":"Xinzhe Li","slug":"xinzhe-li","given":"Xinzhe","family":"Li","affiliation":"Deakin University"},{"name":"Ming Liu","slug":"ming-liu","given":"Ming","family":"Liu","affiliation":"Deakin University"},{"name":"Shang Gao","slug":"shang-gao","given":"Shang","family":"Gao","affiliation":"Deakin University"},{"name":"Wray Buntine","slug":"wray-buntine","given":"Wray","family":"Buntine","affiliation":"VinUniversity"}],"abstract":"Adversarial robustness, domain generalization and dataset biases are three active lines of research contributing to out-of-distribution (OOD) evaluation on neural NLP models.\r\n\r\nHowever, a comprehensive, integrated discussion of the three research lines is still lacking in the literature. This survey will 1) compare the three lines of research under a unifying definition; 2) summarize their data-generating processes and evaluation protocols for each line of research; and 3) emphasize the challenges and opportunities for future work.","doi":"10.24963/ijcai.2023/749","url":"https://ijcai.org/proceedings/2023/749","pdf":"https://ijcai.org/proceedings/2023/0749.pdf","markdown":"https://ijcai.org/proceedings/2023/749.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/749","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Survey Track","track":"Survey Track","number":749,"pages":{"first":6683,"last":6691},"keywords":["Survey: Natural Language Processing","Survey: AI Ethics, Trust, Fairness","Survey: Machine Learning","Survey: Uncertainty in AI"],"videos":[],"references":[{"position":1,"raw":"[Alzantot et al., 2018] Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, and Kai- Wei Chang. Generating natural language adversarial ex- amples. In EMNLP, pages 2890-2896, 2018.","title":"Generating natural language adversarial examples","authors":"Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, Kai-Wei Chang","year":"2018","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Arora et al., 2021] Udit Arora, William Huang, and He He. Types of out-of-distribution texts and how to detect them. In EMNLP, pages 10687-10701, 2021.","title":"Types of out-of-distribution texts and how to detect them","authors":"Udit Arora, William Huang, He He","year":"2021","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Barrett et al., 2019] Maria Barrett, Yova Kementchedjhieva, Yanai Elazar, Desmond Elliott, and Anders Søgaard. Ad- versarial removal of demographic attributes revisited. In EMNLP-IJCNLP, pages 6330-6335, 2019.","title":"Adversarial removal of demographic attributes revisited","authors":"Maria Barrett, Yova Kementchedjhieva, Yanai Elazar, Desmond Elliott, Anders Søgaard","year":"2019","venue":"EMNLP-IJCNLP","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Behjati et al., 2019] Melika Behjati, Seyed-Mohsen Moosavi-Dezfooli, Mahdieh Soleymani Baghshah, and Pascal Frossard. Universal adversarial attacks on text classifiers. In ICASSP, pages 7345-7349, 2019.","title":"Universal adversarial attacks on text classifiers","authors":"Melika Behjati, Seyed-Mohsen Moosavi-Dezfooli, Mahdieh Soleymani Baghshah, Pascal Frossard","year":"2019","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Borkan et al., 2019] Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Nuanced metrics for measuring unintended bias with real data for text classification. In World Wide Web Conference, page 491-500, 2019.","title":"Nuanced metrics for measuring unintended bias with real data for text classification","authors":"Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, Lucy Vasserman","year":"2019","venue":"World Wide Web Conference","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Bras et al., 2020] Ronan Le Bras, Swabha Swayamdipta, Chandra Bhagavatula, Rowan Zellers, Matthew Peters, Ashish Sabharwal, and Yejin Choi. Adversarial filters of dataset biases. In ICML, pages 1078-1088, 2020.","title":"Adversarial filters of dataset biases","authors":"Bras","year":"2020","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chen et al., 2022] Hannah Chen, Yangfeng Ji, and David Evans. Balanced adversarial training: Balancing trade- offs between fickleness and obstinacy in NLP models. In EMNLP, 2022.","title":"Balanced adversarial training: Balancing tradeoffs between fickleness and obstinacy in NLP models","authors":"Hannah Chen, Yangfeng Ji, David Evans","year":"2022","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Cheng et al., 2020] Minhao Cheng, Jinfeng Yi, Pin-Yu Chen, Huan Zhang, and Cho-Jui Hsieh. Seq2sick: Evalu- ating the robustness of sequence-to-sequence models with adversarial examples. In AAAI, pages 3601-3608, 2020.","title":"Seq2sick: Evaluating the robustness of sequence-to-sequence models with adversarial examples","authors":"Cheng","year":"2020","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Ebrahimi et al., 2018] Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. Hotflip: White-box adversarial ex- amples for text classification. In ACL, pages 31-36, 2018.","title":"Hotflip: White-box adversarial examples for text classification","authors":"Javid Ebrahimi, Anyi Rao, Daniel Lowd, Dejing Dou","year":"2018","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Emelin et al., 2020] Denis Emelin, Ivan Titov, and Rico Sennrich. Detecting word sense disambiguation biases in machine translation for model-agnostic adversarial at- tacks. In EMNLP, pages 7635-7653, 2020.","title":"Detecting word sense disambiguation biases in machine translation for model-agnostic adversarial attacks","authors":"Denis Emelin, Ivan Titov, Rico Sennrich","year":"2020","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Gao et al., 2018] Ji Gao, Jack Lanchantin, Mary Lou Soffa, and Yanjun Qi. Black-box generation of adversarial text sequences to evade deep learning classifiers. In 2018 IEEE Security and Privacy Workshops (SPW), pages 50- 56, 2018.","title":"Black-box generation of adversarial text sequences to evade deep learning classifiers","authors":"Ji Gao, Jack Lanchantin, Mary Lou Soffa, Yanjun Qi","year":"2018","venue":"2018 IEEE Security and Privacy Workshops (SPW)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Gardner et al., 2020] Matt Gardner, Yoav Artzi, Victoria Basmov, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Got- tumukkala, Nitish Gupta, Hannaneh Hajishirzi, Gabriel Il- harco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nel- son F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A. Smith, Sanjay Subramanian, Reut Tsarfaty, Eric Wallace, Ally Zhang, and Ben Zhou. Evaluating models' local decision boundaries via contrast sets. In EMNLP, pages 1307-1323, 2020.","title":"Evaluating models' local decision boundaries via contrast sets","authors":"Matt Gardner, Yoav Artzi, Victoria Basmov, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hannaneh Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A Smith, Sanjay Subramanian, Reut Tsarfaty, Eric Wallace, Ally Zhang, Ben Zhou","year":"2020","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Geva et al., 2019] Mor Geva, Yoav Goldberg, and Jonathan Berant. Are we modeling the task or the annotator? an investigation of annotator bias in natural language under- standing datasets. In EMNLP-IJCNLP, pages 1161-1166, 2019.","title":"Are we modeling the task or the annotator? an investigation of annotator bias in natural language understanding datasets","authors":"Mor Geva, Yoav Goldberg, Jonathan Berant","year":"2019","venue":"EMNLP-IJCNLP","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Gokhale et al., 2022] Tejas Gokhale, Swaroop Mishra, Man Luo, Bhavdeep Sachdeva, and Chitta Baral. Generalized but not Robust? comparing the effects of data modification methods on out-of-domain generalization and adversarial robustness. In ACL, pages 2705-2718, 2022.","title":"Generalized but not Robust? comparing the effects of data modification methods on out-of-domain generalization and adversarial robustness","authors":"Tejas Gokhale, Swaroop Mishra, Man Luo, Bhavdeep Sachdeva, Chitta Baral","year":"2022","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Gong et al., 2018] Zhitao Gong, Wenlu Wang, Bo Li, Dawn Song, and Wei-Shinn Ku. Adversarial texts with gradient methods. arXiv preprint arXiv:1801.07175, 2018.","title":"Adversarial texts with gradient methods","authors":"Gong","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Goodfellow et al., 2015] Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In ICLR, 2015.","title":"Explaining and harnessing adversarial examples","authors":"Ian J Goodfellow, Jonathon Shlens, Christian Szegedy","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Gururangan et al., 2018] Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R. Bowman, and Noah A. Smith. Annotation artifacts in natural language inference data. In NAACL-HLT, pages 107-112, 2018.","title":"Annotation artifacts in natural language inference data","authors":"Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R Bowman, Noah A Smith","year":"2018","venue":"NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Gururangan et al., 2020] Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Belt- agy, Doug Downey, and Noah A. Smith. Don't stop pretraining: Adapt language models to domains and tasks. In ACL, pages 8342-8360, July 2020.","title":"Don't stop pretraining: Adapt language models to domains and tasks","authors":"Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, Noah A Smith","year":"2020","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Hendrycks et al., 2020] Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, Adam Dziedzic, Rishabh Krishnan, and Dawn Song. Pretrained transformers improve out-of- distribution robustness. In ACL, pages 2744-2751, 2020.","title":"Pretrained transformers improve out-ofdistribution robustness","authors":"Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, Adam Dziedzic, Rishabh Krishnan, Dawn Song","year":"2020","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Ilyas et al., 2019] Andrew Ilyas, Shibani Santurkar, Dim- itris Tsipras, Logan Engstrom, Brandon Tran, and Alek- sander Madry. Adversarial examples are not bugs, they are features. In Advances in NIPS, 2019.","title":"Adversarial examples are not bugs, they are features","authors":"Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, Aleksander Madry","year":"2019","venue":"Advances in NIPS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Iyyer et al., 2018] Mohit Iyyer, John Wieting, Kevin Gim- pel, and Luke Zettlemoyer. Adversarial example genera- tion with syntactically controlled paraphrase networks. In NAACL-HLT, pages 1875-1885, 2018.","title":"Adversarial example generation with syntactically controlled paraphrase networks","authors":"Mohit Iyyer, John Wieting, Kevin Gimpel, Luke Zettlemoyer","year":"2018","venue":"NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Jia and Liang, 2017] Robin Jia and Percy Liang. Adversar- ial examples for evaluating reading comprehension sys- tems. In EMNLP, pages 2021-2031, 2017.","title":"Adversarial examples for evaluating reading comprehension systems","authors":"Liang Jia, Robin Jia, Percy Liang","year":"2017","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Jin et al., 2020] Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. Is bert really robust? a strong baseline for natural language attack on text classification and en- tailment. In AAAI, pages 8018-8025, 2020.","title":"Is bert really robust? a strong baseline for natural language attack on text classification and entailment","authors":"Di Jin, Zhijing Jin, Joey Tianyi Zhou, Peter Szolovits","year":"2020","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Kaushik et al., 2020] Divyansh Kaushik, Eduard Hovy, and Zachary Lipton. Learning the difference that makes a dif- ference with counterfactually-augmented data. In ICLR, 2020.","title":"Learning the difference that makes a difference with counterfactually-augmented data","authors":"Kaushik","year":"2020","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Kedzie et al., 2018] Chris Kedzie, Kathleen McKeown, and Hal Daumé III. Content selection in deep learning models of summarization. In EMNLP, pages 1818-1828, 2018.","title":"Content selection in deep learning models of summarization","authors":"Chris Kedzie, Kathleen Mckeown, Hal Daumé, Iii","year":"2018","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Lai et al., 2021] Yuxuan Lai, Chen Zhang, Yansong Feng, Quzhe Huang, and Dongyan Zhao. Why machine reading comprehension models learn shortcuts? In ACL-IJCNLP, pages 989-1002, 2021.","title":"Why machine reading comprehension models learn shortcuts?","authors":"Yuxuan Lai, Chen Zhang, Yansong Feng, Quzhe Huang, Dongyan Zhao","year":"2021","venue":"ACL-IJCNLP","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Li et al., 2019] Jinfeng Li, Shouling Ji, Tianyu Du, Bo Li, and Ting Wang. Textbugger: Generating adversarial text against real-world applications. In NDSS, 2019.","title":"Textbugger: Generating adversarial text against real-world applications","authors":"Jinfeng Li, Shouling Ji, Tianyu Du, Bo Li, Ting Wang","year":"2019","venue":"NDSS","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Li et al., 2020] Linyang Li, Ruotian Ma, Qipeng Guo, Xi- angyang Xue, and Xipeng Qiu. BERT-ATTACK: Adver- sarial attack against BERT using BERT. In EMNLP, pages 6193-6202, 2020.","title":"BERT-ATTACK: Adversarial attack against BERT using BERT","authors":"Li","year":"2020","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Liang et al., 2018] Bin Liang, Hongcheng Li, Miaoqiang Su, Pan Bian, Xirong Li, and Wenchang Shi. Deep text classification can be fooled. In IJCAI, pages 4208-4215, 7 2018.","title":"Deep text classification can be fooled","authors":"Bin Liang, Hongcheng Li, Miaoqiang Su, Pan Bian, Xirong Li, Wenchang Shi","year":"2018","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2018/585"},{"position":30,"raw":"[Liang et al., 2021] Kaizhao Liang, Jacky Y Zhang, Boxin Wang, Zhuolin Yang, Sanmi Koyejo, and Bo Li. Uncover- ing the connections between adversarial transferability and knowledge transferability. In ICML, pages 6577-6587, 2021.","title":"Uncovering the connections between adversarial transferability and knowledge transferability","authors":"Kaizhao Liang, Jacky Y Zhang, Boxin Wang, Zhuolin Yang, Sanmi Koyejo, Bo Li","year":"2021","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Liu et al., 2022] Aiwei Liu, Honghai Yu, Xuming Hu, Shu'ang Li, Li Lin, Fukun Ma, Yawen Yang, and Li- jie Wen. Character-level white-box adversarial attacks against transformers via attachable subwords substitution. In EMNLP, pages 7664-7676, December 2022.","title":"Character-level white-box adversarial attacks against transformers via attachable subwords substitution","authors":"Liu","year":"2022","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[McCoy et al., 2019] Tom McCoy, Ellie Pavlick, and Tal Linzen. Right for the wrong reasons: Diagnosing syntac- tic heuristics in natural language inference. In ACL, pages 3428-3448, 2019.","title":"Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference","authors":"Mccoy","year":"2019","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Moosavi-Dezfooli et al., 2017] Seyed-Mohsen Moosavi- Dezfooli, Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. Universal adversarial perturbations. In CVPR, pages 1765-1773, 2017.","title":"Universal adversarial perturbations","authors":"Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Omar Fawzi, Pascal Frossard","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Naik et al., 2018] Aakanksha Naik, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose, and Graham Neubig. Stress test evaluation for natural language inference. In CoLing, pages 2340-2353, 2018.","title":"Stress test evaluation for natural language inference","authors":"Aakanksha Naik, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose, Graham Neubig","year":"2018","venue":"CoLing","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Nie et al., 2020] Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adver- sarial NLI: A new benchmark for natural language under- standing. In ACL, pages 4885-4901, 2020.","title":"Adversarial NLI: A new benchmark for natural language understanding","authors":"Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, Douwe Kiela","year":"2020","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Oprea and Magdy, 2020] Silviu Vlad Oprea and Walid Magdy. The effect of sociocultural variables on sarcasm communication online. In ACM, page 22, 2020.","title":"The effect of sociocultural variables on sarcasm communication online","authors":"Silviu Vlad, Oprea, Walid Magdy","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Qi et al., 2021] Fanchao Qi, Yangyi Chen, Xurui Zhang, Mukai Li, Zhiyuan Liu, and Maosong Sun. Mind the style of text! adversarial and backdoor attacks based on text style transfer. In EMNLP, pages 4569-4580, 2021.","title":"Mind the style of text! adversarial and backdoor attacks based on text style transfer","authors":"Qi","year":"2021","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Ribeiro et al., 2018] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Semantically equivalent adversarial rules for debugging NLP models. In ACL, pages 856-865, 2018.","title":"Semantically equivalent adversarial rules for debugging NLP models","authors":"Marco Tulio Ribeiro, Sameer Singh, Carlos Guestrin","year":"2018","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Sakaguchi et al., 2020] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial Winograd schema challenge at scale. In AAAI, pages 8732-8740, 2020.","title":"Winogrande: An adversarial Winograd schema challenge at scale","authors":"Keisuke Sakaguchi, Le Ronan, Chandra Bras, Yejin Bhagavatula, Choi","year":"2020","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Schuster et al., 2019] Tal Schuster, Darsh Shah, Yun Jie Serene Yeo, Daniel Roberto Filizzola Ortiz, Enrico Santus, and Regina Barzilay. Towards debiasing fact ver- ification models. In EMNLP-IJCNLP, pages 3419-3425, 2019.","title":"Towards debiasing fact verification models","authors":"Tal Schuster, Darsh Shah, Yun Jie, Serene Yeo, Daniel Roberto Filizzola, Enrico Ortiz, Regina Santus, Barzilay","year":"2019","venue":"EMNLP-IJCNLP","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Song et al., 2020] Congzheng Song, Alexander Rush, and Vitaly Shmatikov. Adversarial semantic collisions. In EMNLP, pages 4198-4210, 2020.","title":"Congzheng Song, Alexander Rush, and Vitaly Shmatikov. Adversarial semantic collisions","authors":"Song","year":"2020","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Song et al., 2021] Liwei Song, Xinwei Yu, Hsuan-Tung Peng, and Karthik Narasimhan. Universal adversarial attacks with natural triggers for text classification. In NAACL-HLT, pages 3724-3733, 2021.","title":"Universal adversarial attacks with natural triggers for text classification","authors":"Liwei Song, Xinwei Yu, Hsuan-Tung Peng, Karthik Narasimhan","year":"2021","venue":"NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Sugawara et al., 2018] Saku Sugawara, Kentaro Inui, Satoshi Sekine, and Akiko Aizawa. What makes reading comprehension questions easier? In EMNLP, pages 4208-4219, 2018.","title":"What makes reading comprehension questions easier?","authors":"Saku Sugawara, Kentaro Inui, Satoshi Sekine, Akiko Aizawa","year":"2018","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":44,"raw":"[Wallace et al., 2019a] Eric Wallace, Shi Feng, Nikhil Kand- pal, Matt Gardner, and Sameer Singh. Universal adversar- ial triggers for attacking and analyzing NLP. In EMNLP- IJCNLP, pages 2153-2162, 2019.","title":"Universal adversarial triggers for attacking and analyzing NLP","authors":"Wallace","year":"2019","venue":"EMNLP-IJCNLP","doi":null,"url":null,"internal_url":null},{"position":45,"raw":"[Wallace et al., 2019b] Eric Wallace, Pedro Rodriguez, Shi Feng, Ikuya Yamada, and Jordan Boyd-Graber. Trick me if you can: Human-in-the-loop generation of adversarial examples for question answering. TACL, 7:387-401, 2019.","title":"Trick me if you can: Human-in-the-loop generation of adversarial examples for question answering","authors":"Wallace","year":"2019","venue":"TACL","doi":null,"url":null,"internal_url":null},{"position":46,"raw":"[Wallace et al., 2020] Eric Wallace, Mitchell Stern, and Dawn Song. Imitation attacks and defenses for black-box machine translation systems. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, EMNLP, pages 5531-5546, 2020.","title":"Imitation attacks and defenses for black-box machine translation systems","authors":"Eric Wallace, Mitchell Stern, Dawn Song","year":"2020","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":47,"raw":"[Wei and Zou, 2019] Jason Wei and Kai Zou. EDA: Easy data augmentation techniques for boosting performance on text classification tasks. In EMNLP-IJCNLP, pages 6382- 6388, 2019.","title":"EDA: Easy data augmentation techniques for boosting performance on text classification tasks","authors":"Zou ; ] Wei, Jason Wei, Kai Zou","year":"2019","venue":"EMNLP-IJCNLP","doi":null,"url":null,"internal_url":null},{"position":48,"raw":"[Yi et al., 2021] Mingyang Yi, Lu Hou, Jiacheng Sun, Lifeng Shang, Xin Jiang, Qun Liu, and Zhiming Ma. Improved ood generalization via adversarial training and pretraing. In Marina Meila and Tong Zhang, editors, ICML, volume 139 of Proceedings of Machine Learning Research, pages 11987-11997. PMLR, 18-24 Jul 2021.","title":"Improved ood generalization via adversarial training and pretraing","authors":"Mingyang Yi, Lu Hou, Jiacheng Sun, Lifeng Shang, Xin Jiang, Qun Liu, Zhiming Ma","year":"2021","venue":"Proceedings of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":49,"raw":"[Zang et al., 2020] Yuan Zang, Fanchao Qi, Chenghao Yang, Zhiyuan Liu, Meng Zhang, Qun Liu, and Maosong Sun. Word-level textual adversarial attacking as combinatorial optimization. In ACL, pages 6066-6080, 2020.","title":"Word-level textual adversarial attacking as combinatorial optimization","authors":"Yuan Zang, Fanchao Qi, Chenghao Yang, Zhiyuan Liu, Meng Zhang, Qun Liu, Maosong Sun","year":"2020","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":50,"raw":"[Zellers et al., 2018] Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. Swag: A large-scale adversarial dataset for grounded commonsense inference. In EMNLP, pages 93-104, 2018.","title":"Swag: A large-scale adversarial dataset for grounded commonsense inference","authors":"Rowan Zellers, Yonatan Bisk, Roy Schwartz, Yejin Choi","year":"2018","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":51,"raw":"[Zhang et al., 2019] Yuan Zhang, Jason Baldridge, and Luheng He. PAWS: Paraphrase adversaries from word scrambling. In NAACL, pages 1298-1308, 2019.","title":"PAWS: Paraphrase adversaries from word scrambling","authors":"Yuan Zhang, Jason Baldridge, Luheng He","year":"2019","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":52,"raw":"[Zhao et al., 2018] Zhengli Zhao, Dheeru Dua, and Sameer Singh. Generating natural adversarial examples. In ICLR, 2018.","title":"Generating natural adversarial examples","authors":"Zhao","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}