{"type":"ScholarlyArticle","title":"Less Learn Shortcut: Analyzing and Mitigating Learning of Spurious Feature-Label Correlation","authors":[{"name":"Yanrui Du","slug":"yanrui-du","given":"Yanrui","family":"Du","affiliation":"Harbin Institute of Technology , Harbin , China"},{"name":"Jing Yan","slug":"jing-yan","given":"Jing","family":"Yan","affiliation":"Baidu Inc. , Beijing , China"},{"name":"Yan Chen","slug":"yan-chen","given":"Yan","family":"Chen","affiliation":"Baidu Inc. , Beijing , China"},{"name":"Jing Liu","slug":"jing-liu","given":"Jing","family":"Liu","affiliation":"Baidu Inc. , Beijing , China"},{"name":"Sendong Zhao","slug":"sendong-zhao","given":"Sendong","family":"Zhao","affiliation":"Harbin Institute of Technology , Harbin , China"},{"name":"Qiaoqiao She","slug":"qiaoqiao-she","given":"Qiaoqiao","family":"She","affiliation":"Baidu Inc. , Beijing , China"},{"name":"Hua Wu","slug":"hua-wu","given":"Hua","family":"Wu","affiliation":"Baidu Inc. , Beijing , China"},{"name":"Haifeng Wang","slug":"haifeng-wang","given":"Haifeng","family":"Wang","affiliation":"Baidu Inc. , Beijing , China"},{"name":"Bing Qin","slug":"bing-qin","given":"Bing","family":"Qin","affiliation":"Harbin Institute of Technology , Harbin , China"}],"abstract":"Recent research has revealed that deep neural networks often take dataset biases as a shortcut to make decisions rather than understand tasks, leading to failures in real-world applications. In this study, we focus on the spurious correlation between word features and labels that models learn from the biased data distribution of training data. In particular, we define the word highly co-occurring with a specific label as biased word, and the example containing biased word as biased example. Our analysis shows that biased examples are easier for models to learn, while at the time of prediction, biased words make a significantly higher contribution to the models' predictions, and models tend to assign predicted labels over-relying on the spurious correlation between words and labels. To mitigate models' over-reliance on the shortcut (i.e. spurious correlation), we propose a training strategy Less-Learn-Shortcut (LLS): our strategy quantifies the biased degree of the biased examples and down-weights them accordingly. Experimental results on Question Matching, Natural Language Inference and Sentiment Analysis tasks show that LLS is a task-agnostic strategy and can improve the model performance on adversarial data while maintaining good performance on in-domain data.","doi":"10.24963/ijcai.2023/560","url":"https://ijcai.org/proceedings/2023/560","pdf":"https://ijcai.org/proceedings/2023/0560.pdf","markdown":"https://ijcai.org/proceedings/2023/560.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/560","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Natural Language Processing","track":"Main Track","number":560,"pages":{"first":5039,"last":5048},"keywords":["Natural Language Processing: NLP: Interpretability and analysis of models for NLP","Natural Language Processing: NLP: Question answering"],"videos":[],"references":[{"position":1,"raw":"[Alzantot et al., 2018] Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, and Kai-Wei Chang. Generating natural language adversar- ial examples. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2890-2896, Brussels, Belgium, 2018. Association for Computational Linguistics.","title":"Generating natural language adversarial examples","authors":"Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, Kai-Wei Chang","year":"2018","venue":"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Balkir et al., 2022] Esma Balkir, Svetlana Kiritchenko, Isar Nejadgholi, and Kathleen C Fraser. Challenges in apply- ing explainability methods to improve the fairness of nlp models. arXiv preprint arXiv:2206.03945, 2022.","title":"Challenges in applying explainability methods to improve the fairness of nlp models","authors":"Esma Balkir, Svetlana Kiritchenko, Isar Nejadgholi, Kathleen C Fraser","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bender and Koller, 2020] Emily M Bender and Alexander Koller. Climbing towards nlu: On meaning, form, and understanding in the age of data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5185-5198, 2020.","title":"Climbing towards nlu: On meaning, form, and understanding in the age of data","authors":"M Emily, Alexander Bender, Koller","year":"2020","venue":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bolukbasi et al., 2016] Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. Man is to computer programmer as woman is to home- maker? debiasing word embeddings. Advances in neural information processing systems, 29:4349-4357, 2016.","title":"Man is to computer programmer as woman is to homemaker? debiasing word embeddings","authors":"Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, Adam T Kalai","year":"2016","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bowman et al., 2015] Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. A large annotated corpus for learning natural language inference. arXiv preprint arXiv:1508.05326, 2015.","title":"A large annotated corpus for learning natural language inference","authors":"Gabor Samuel R Bowman, Christopher Angeli, Christopher D Potts, Manning","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Clark et al., 2019] Christopher Clark, Mark Yatskar, and Luke Zettlemoyer. Don't take the easy way out: Ensemble based methods for avoiding known dataset biases. arXiv preprint arXiv:1909.03683, 2019.","title":"Don't take the easy way out: Ensemble based methods for avoiding known dataset biases","authors":"Christopher Clark, Mark Yatskar, Luke Zettlemoyer","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Cui et al., 2016] Yiming Cui, Zhipeng Chen, Si Wei, Shi- jin Wang, Ting Liu, and Guoping Hu. Attention-over- attention neural networks for reading comprehension. arXiv preprint arXiv:1607.04423, 2016.","title":"Attention-overattention neural networks for reading comprehension","authors":"Yiming Cui, Zhipeng Chen, Si Wei, Shijin Wang, Ting Liu, Guoping Hu","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dawkins, 2021] Hillary Dawkins. Marked attribute bias in natural language inference. arXiv preprint arXiv:2109.14039, 2021.","title":"Hillary Dawkins. Marked attribute bias in natural language inference","authors":"Dawkins","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Devlin et al., 2018] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint arXiv:1810.04805, 2018.","title":"Bert: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Du et al., 2021] Mengnan Du, Varun Manjunatha, R. Jain, Ruchi Deshpande, Franck Dernoncourt, Jiuxiang Gu, Tong Sun, and Xia Hu. Towards interpreting and mitigat- ing shortcut learning behavior of nlu models. In NAACL, 2021.","title":"Towards interpreting and mitigating shortcut learning behavior of nlu models","authors":"Mengnan Du, R Varun Manjunatha, Ruchi Jain, Franck Deshpande, Jiuxiang Dernoncourt, Tong Gu, Xia Sun, Hu","year":"2021","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Gardner et al., 2021] Matt Gardner, William Merrill, Jesse Dodge, Matthew E Peters, Alexis Ross, Sameer Singh, and Noah Smith. Competency problems: On finding and removing artifacts in language data. arXiv preprint arXiv:2104.08646, 2021.","title":"Competency problems: On finding and removing artifacts in language data","authors":"Gardner","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Geirhos et al., 2020] Robert Geirhos, Jörn-Henrik Jacob- sen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learn- ing in deep neural networks. Nature Machine Intelligence, 2(11):665-673, 2020.","title":"Shortcut learning in deep neural networks","authors":"Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, Felix A Wichmann","year":"2020","venue":"Nature Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Goyal et al., 2017] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6904-6913, 2017.","title":"Making the v in vqa matter: Elevating the role of image understanding in visual question answering","authors":"Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, Devi Parikh","year":"2017","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Gu et al., 2020] Shuhao Gu, Jinchao Zhang, Fandong Meng, Yang Feng, Wanying Xie, Jie Zhou, and Dong Yu. Token-level adaptive training for neural machine transla- tion. arXiv preprint arXiv:2010.04380, 2020.","title":"Token-level adaptive training for neural machine translation","authors":"Gu","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Gururangan et al., 2018] Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R Bowman, and Noah A Smith. Annotation artifacts in natural language inference data. arXiv preprint arXiv:1803.02324, 2018.","title":"Annotation artifacts in natural language inference data","authors":"Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Noah A Samuel R Bowman, Smith","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Jia and Liang, 2017] Robin Jia and Percy Liang. Adversar- ial examples for evaluating reading comprehension sys- tems. arXiv preprint arXiv:1707.07328, 2017.","title":"Adversarial examples for evaluating reading comprehension systems","authors":"Liang Jia, Robin Jia, Percy Liang","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Jiao et al., 2018] Zhenyu Jiao, Shuqi Sun, and Ke Sun. Chi- nese lexical analysis with deep bi-gru-crf network. arXiv preprint arXiv:1807.01882, 2018.","title":"Chinese lexical analysis with deep bi-gru-crf network","authors":"Zhenyu Jiao, Shuqi Sun, Ke Sun","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Jin et al., 2020] Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. Is BERT really robust? A strong base- line for natural language attack on text classification and entailment. In The Thirty-Fourth AAAI Conference on Ar- tificial Intelligence, AAAI 2020, The Thirty-Second Inno- vative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 8018-8025. AAAI Press, 2020.","title":"Is BERT really robust? A strong baseline for natural language attack on text classification and entailment","authors":"Di Jin, Zhijing Jin, Joey Tianyi Zhou, Peter Szolovits","year":"2020","venue":"The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Kaneko and Bollegala, 2021] Masahiro Kaneko and Danushka Bollegala. Debiasing pre-trained contextualised embeddings. arXiv preprint arXiv:2101.09523, 2021.","title":"Debiasing pre-trained contextualised embeddings","authors":"Masahiro Kaneko, Danushka Bollegala","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Kaushik and Lipton, 2018] Divyansh Kaushik and Zachary C Lipton. How much reading does reading comprehension require? a critical investigation of popular benchmarks. arXiv preprint arXiv:1808.04926, 2018.","title":"How much reading does reading comprehension require? a critical investigation of popular benchmarks","authors":"Lipton Kaushik, Divyansh Kaushik, Zachary C Lipton","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Kavumba et al., 2021] Pride Kavumba, Benjamin Heinz- erling, Ana Brassard, and Kentaro Inui. Learning to learn to be right for the right reasons. arXiv preprint arXiv:2104.11514, 2021.","title":"Learning to learn to be right for the right reasons","authors":"Kavumba","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Lai et al., 2021] Yuxuan Lai, Chen Zhang, Yansong Feng, Quzhe Huang, and Dongyan Zhao. Why machine reading comprehension models learn shortcuts? arXiv preprint arXiv:2106.01024, 2021.","title":"Why machine reading comprehension models learn shortcuts","authors":"Yuxuan Lai, Chen Zhang, Yansong Feng, Quzhe Huang, Dongyan Zhao","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Liu et al., 2018] Xin Liu, Qingcai Chen, Chong Deng, Hua- jun Zeng, Jing Chen, Dongfang Li, and Buzhou Tang. Lcqmc: A large-scale chinese question matching corpus. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1952-1962, 2018.","title":"Lcqmc: A large-scale chinese question matching corpus","authors":"Xin Liu, Qingcai Chen, Chong Deng, Huajun Zeng, Jing Chen, Dongfang Li, Buzhou Tang","year":"2018","venue":"Proceedings of the 27th International Conference on Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Liu et al., 2019a] Nelson F Liu, Roy Schwartz, and Noah A Smith. Inoculation by fine-tuning: A method for analyz- ing challenge datasets. arXiv preprint arXiv:1904.02668, 2019.","title":"Inoculation by fine-tuning: A method for analyzing challenge datasets","authors":"Liu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Liu et al., 2019b] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019.","title":"Roberta: A robustly optimized bert pretraining approach","authors":"Liu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[May et al., 2019] Chandler May, Alex Wang, Shikha Bor- dia, Samuel R Bowman, and Rachel Rudinger. On mea- suring social biases in sentence encoders. arXiv preprint arXiv:1903.10561, 2019.","title":"On measuring social biases in sentence encoders","authors":"May","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[McCoy et al., 2019] R Thomas McCoy, Ellie Pavlick, and Tal Linzen. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. arXiv preprint arXiv:1902.01007, 2019.","title":"Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference","authors":"Thomas Mccoy, Ellie Pavlick, Tal Linzen","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Morris et al., 2020] John X Morris, Eli Lifland, Jack Lan- chantin, Yangfeng Ji, and Yanjun Qi. Reevaluating ad- versarial examples in natural language. arXiv preprint arXiv:2004.14174, 2020.","title":"Reevaluating adversarial examples in natural language","authors":"Eli John X Morris, Jack Lifland, Yangfeng Lanchantin, Yanjun Ji, Qi","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Niven and Kao, 2019] Timothy Niven and Hung-Yu Kao. Probing neural network comprehension of natural lan- guage arguments. arXiv preprint arXiv:1907.07355, 2019.","title":"Probing neural network comprehension of natural language arguments","authors":"Niven, Kao, Timothy Niven, Hung-Yu Kao","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Ott et al., 2018] Myle Ott, Michael Auli, David Grangier, and Marc'Aurelio Ranzato. Analyzing uncertainty in neu- ral machine translation. In International Conference on Machine Learning, pages 3956-3965. PMLR, 2018.","title":"Analyzing uncertainty in neural machine translation","authors":"Myle Ott, Michael Auli, David Grangier, Marc'aurelio Ranzato","year":"2018","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Ravfogel et al., 2020] Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. Null it out: Guarding protected attributes by iterative nullspace projec- tion. arXiv preprint arXiv:2004.07667, 2020.","title":"Null it out: Guarding protected attributes by iterative nullspace projection","authors":"Ravfogel","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Ren et al., 2019] Shuhuai Ren, Yihe Deng, Kun He, and Wanxiang Che. Generating natural language adversarial examples through probability weighted word saliency. In Proceedings of the 57th Annual Meeting of the Associa- tion for Computational Linguistics, pages 1085-1097, Flo- rence, Italy, 2019. Association for Computational Linguis- tics.","title":"Generating natural language adversarial examples through probability weighted word saliency","authors":"Ren","year":"2019","venue":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Ribeiro et al., 2016] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. \" why should i trust you?\" explain- ing the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowl- edge discovery and data mining, pages 1135-1144, 2016.","title":"why should i trust you?\" explaining the predictions of any classifier","authors":"Marco Tulio Ribeiro, Sameer Singh, Carlos Guestrin","year":"2016","venue":"Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Schuster et al., 2019] Tal Schuster, Darsh Shah, Yun Jie Serene Yeo, Daniel Roberto Filizzola Ortiz, Enrico Santus, and Regina Barzilay. Towards debiasing fact veri- fication models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNLP), pages 3419-3425, 2019.","title":"Towards debiasing fact verification models","authors":"Tal Schuster, Darsh Shah, Yun Jie, Serene Yeo, Daniel Roberto Filizzola, Enrico Ortiz, Regina Santus, Barzilay","year":"2019","venue":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Stacey et al., 2020] Joe Stacey, Pasquale Minervini, Haim Dubossarsky, Sebastian Riedel, and Tim Rocktäschel. Avoiding the hypothesis-only bias in natural language in- ference via ensemble adversarial training. arXiv preprint arXiv:2004.07790, 2020.","title":"Avoiding the hypothesis-only bias in natural language inference via ensemble adversarial training","authors":"Joe Stacey, Pasquale Minervini, Haim Dubossarsky, Sebastian Riedel, Tim Rocktäschel","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Sun et al., 2019] Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxi- ang Zhu, Hao Tian, and Hua Wu. Ernie: Enhanced rep- resentation through knowledge integration. arXiv preprint arXiv:1904.09223, 2019.","title":"Ernie: Enhanced representation through knowledge integration","authors":"Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hua Hao Tian, Wu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Utama et al., 2020] Prasetya Ajie Utama, Nafise Sadat Moosavi, and Iryna Gurevych. Towards debiasing nlu models from unknown biases. arXiv preprint arXiv:2009.12303, 2020.","title":"Towards debiasing nlu models from unknown biases","authors":"Nafise Prasetya Ajie Utama, Iryna Sadat Moosavi, Gurevych","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Wang et al., 2021] Lijie Wang, Hao Liu, Shuyuan Peng, Hongxuan Tang, Xinyan Xiao, Ying Chen, Hua Wu, and Haifeng Wang. Dutrust: A sentiment analysis dataset for trustworthiness evaluation. arXiv preprint arXiv:2108.13140, 2021.","title":"Dutrust: A sentiment analysis dataset for trustworthiness evaluation","authors":"Lijie Wang, Hao Liu, Shuyuan Peng, Hongxuan Tang, Xinyan Xiao, Ying Chen, Hua Wu, Haifeng Wang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Webster et al., 2020] Kellie Webster, Xuezhi Wang, Ian Tenney, Alex Beutel, Emily Pitler, Ellie Pavlick, Jilin Chen, Ed Chi, and Slav Petrov. Measuring and reduc- ing gendered correlations in pre-trained models. arXiv preprint arXiv:2010.06032, 2020.","title":"Measuring and reducing gendered correlations in pre-trained models","authors":"Kellie Webster, Xuezhi Wang, Ian Tenney, Alex Beutel, Emily Pitler, Ellie Pavlick, Jilin Chen, Ed Chi, Slav Petrov","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Williams et al., 2017] Adina Williams, Nikita Nangia, and Samuel R Bowman. A broad-coverage challenge cor- pus for sentence understanding through inference. arXiv preprint arXiv:1704.05426, 2017.","title":"A broad-coverage challenge corpus for sentence understanding through inference","authors":"Adina Williams, Nikita Nangia, Samuel R Bowman","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Yaghoobzadeh et al., 2021] Yadollah Yaghoobzadeh, Soroush Mehri, Remi Tachet des Combes, Timothy J Hazen, and Alessandro Sordoni. Increasing robustness to spurious correlations using forgettable examples. In EACL, 2021.","title":"Increasing robustness to spurious correlations using forgettable examples","authors":"Yaghoobzadeh","year":"2021","venue":"EACL","doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Ye and Kovashka, 2021] Keren Ye and Adriana Kovashka. A case study of the shortcut effects in visual commonsense reasoning. In Proceedings of the AAAI Conference on Ar- tificial Intelligence, volume 35, pages 3181-3189, 2021.","title":"A case study of the shortcut effects in visual commonsense reasoning","authors":"Kovashka Ye, Keren Ye, Adriana Kovashka","year":"2021","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Zhu et al., 2021] Hongyu Zhu, Yan Chen, Jing Yan, Jing Liu, Yu Hong, Ying Chen, Hua Wu, and Haifeng Wang. Duqm: A chinese dataset of linguistically perturbed nat- ural questions for evaluating the robustness of question matching models. arXiv preprint arXiv:2112.08609, 2021.","title":"Duqm: A chinese dataset of linguistically perturbed natural questions for evaluating the robustness of question matching models","authors":"Hongyu Zhu, Yan Chen, Jing Yan, Jing Liu, Yu Hong, Ying Chen, Hua Wu, Haifeng Wang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}