{"type":"ScholarlyArticle","title":"On the Paradox of Learning to Reason from Data","authors":[{"name":"Honghua Zhang","slug":"honghua-zhang","given":"Honghua","family":"Zhang","affiliation":"University of California , Los Angeles"},{"name":"Liunian Harold Li","slug":"liunian-harold-li","given":"Liunian Harold","family":"Li","affiliation":"University of California , Los Angeles"},{"name":"Tao Meng","slug":"tao-meng","given":"Tao","family":"Meng","affiliation":"University of California , Los Angeles"},{"name":"Kai-Wei Chang","slug":"kai-wei-chang","given":"Kai-Wei","family":"Chang","affiliation":"University of California , Los Angeles"},{"name":"Guy Van den Broeck","slug":"guy-van-den-broeck","given":"Guy","family":"Van den Broeck","affiliation":"University of California , Los Angeles"}],"abstract":"Logical reasoning is needed in a wide range of NLP tasks. Can a BERT model be trained end-to-end to solve logical reasoning problems presented in natural language? We attempt to answer this question in a confined problem space where there exists a set of parameters that perfectly simulates logical reasoning. We make observations that seem to contradict each other: BERT attains near-perfect accuracy on in-distribution test examples while failing to generalize to other data distributions over the exact same problem space. Our study provides an explanation for this paradox: instead of learning to emulate the correct reasoning function, BERT has, in fact, learned statistical features that inherently exist in logical reasoning problems. We also show that it is infeasible to jointly remove statistical features from data, illustrating the difficulty of learning to reason in general. Our result naturally extends to other neural models (e.g. T5) and unveils the fundamental difference between learning to reason and learning to achieve high performance on NLP benchmarks using statistical features.","doi":"10.24963/ijcai.2023/375","url":"https://ijcai.org/proceedings/2023/375","pdf":"https://ijcai.org/proceedings/2023/0375.pdf","markdown":"https://ijcai.org/proceedings/2023/375.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/375","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Knowledge Representation and Reasoning","track":"Main Track","number":375,"pages":{"first":3365,"last":3373},"keywords":["Natural Language Processing: NLP: Interpretability and analysis of models for NLP","Knowledge Representation and Reasoning: KRR: Learning and reasoning","AI Ethics, Trust, Fairness: ETF: Trustworthy AI"],"videos":[],"references":[{"position":1,"raw":"[Abdelaziz et al., 2020] Ibrahim Abdelaziz, Veronika Thost, Maxwell Crouse, and Achille Fokoue. An experimental study of formula embeddings for automated theorem prov- ing in first-order logic. CoRR, abs/2002.00423, 2020.","title":"An experimental study of formula embeddings for automated theorem proving in first-order logic","authors":"Ibrahim Abdelaziz, Veronika Thost, Maxwell Crouse, Achille Fokoue","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bowman et al., 2015] Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large annotated corpus for learning natural language inference. In EMNLP. The Association for Computational Linguis- tics, 2015.","title":"A large annotated corpus for learning natural language inference","authors":"R Samuel, Gabor Bowman, Christopher Angeli, Christopher D Potts, Manning","year":"2015","venue":"EMNLP. The Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Clark et al., 2019] Christopher Clark, Mark Yatskar, and Luke Zettlemoyer. Don't take the easy way out: Ensem- ble based methods for avoiding known dataset biases. In EMNLP/IJCNLP (1), pages 4067-4080. Association for Computational Linguistics, 2019.","title":"Don't take the easy way out: Ensemble based methods for avoiding known dataset biases","authors":"Christopher Clark, Mark Yatskar, Luke Zettlemoyer","year":"2019","venue":"EMNLP/IJCNLP (1)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Clark et al., 2020] Peter Clark, Oyvind Tafjord, and Kyle Richardson. Transformers as soft reasoners over language. In IJCAI. ijcai.org, 2020.","title":"Transformers as soft reasoners over language","authors":"Clark","year":"2020","venue":"IJCAI. ijcai.org","doi":null,"url":null,"internal_url":"/proceedings/2020/537"},{"position":5,"raw":"[Cook, 1971] Stephen A. Cook. The complexity of theorem- proving procedures. In STOC, pages 151-158. ACM, 1971.","title":"The complexity of theoremproving procedures","authors":"Stephen A Cook","year":"1971","venue":"STOC","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Crouse et al., 2019] Maxwell Crouse, Ibrahim Abdelaziz, Cristina Cornelio, Veronika Thost, Lingfei Wu, Ken- neth D. Forbus, and Achille Fokoue. Improving graph neu- ral network representations of logical formulae with sub- graph pooling. CoRR, abs/1911.06904, 2019.","title":"Improving graph neural network representations of logical formulae with subgraph pooling","authors":"Maxwell Crouse, Ibrahim Abdelaziz, Cristina Cornelio, Veronika Thost, Lingfei Wu, Kenneth D Forbus, Achille Fokoue","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Darwiche and Marquis, 2002] Adnan Darwiche and Pierre Marquis. A knowledge compilation map. Journal of Arti- ficial Intelligence Research, 17:229-264, 2002.","title":"A knowledge compilation map","authors":"Adnan Darwiche, Pierre Marquis","year":"2002","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Devlin et al., 2019] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understand- ing. In NAACL-HLT (1). Association for Computational Linguistics, 2019.","title":"BERT: pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2019","venue":"Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Elazar et al., 2021] Yanai Elazar, Hongming Zhang, Yoav Goldberg, and Dan Roth. Back to square one: Artifact detection, training and commonsense disentanglement in the winograd schema. arXiv preprint arXiv:2104.08161, 2021.","title":"Back to square one: Artifact detection, training and commonsense disentanglement in the winograd schema","authors":"Yanai Elazar, Hongming Zhang, Yoav Goldberg, Dan Roth","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gontier et al., 2020] Nicolas Gontier, Koustuv Sinha, Siva Reddy, and Chris Pal. Measuring systematic gen- eralization in neural proof generation with transform- ers. Advances in Neural Information Processing Systems, 33:22231-22242, 2020.","title":"Measuring systematic generalization in neural proof generation with transformers","authors":"Nicolas Gontier, Koustuv Sinha, Siva Reddy, Chris Pal","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Gururangan et al., 2018] Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R. Bowman, and Noah A. Smith. Annotation artifacts in natural language inference data. In NAACL-HLT (2). Association for Computational Linguistics, 2018.","title":"Annotation artifacts in natural language inference data","authors":"Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R Bowman, Noah A Smith","year":"2018","venue":"Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hahn et al., 2021] Christopher Hahn, Frederik Schmitt, Jens U. Kreber, Markus Norman Rabe, and Bernd Finkbeiner. Teaching temporal logics to neural networks. In ICLR. OpenReview.net, 2021.","title":"Teaching temporal logics to neural networks","authors":"Christopher Hahn, Frederik Schmitt, Jens U Kreber, Markus Norman Rabe, Bernd Finkbeiner","year":"2021","venue":"ICLR. OpenReview.net","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[He et al., 2019] He He, Sheng Zha, and Haohan Wang. Un- learn dataset bias in natural language inference by fitting the residual. In DeepLo@EMNLP-IJCNLP, pages 132- 142. Association for Computational Linguistics, 2019.","title":"Unlearn dataset bias in natural language inference by fitting the residual","authors":"He","year":"2019","venue":"DeepLo@EMNLP-IJCNLP","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Johnson et al., 2017] Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross B. Girshick. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. In CVPR. IEEE Computer Society, 2017.","title":"CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning","authors":"Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, Ross B Girshick","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Khardon and Roth, 1997] Roni Khardon and Dan Roth. Learning to reason. Journal of the ACM (JACM), 44(5):697-725, 1997.","title":"Roni Khardon and Dan Roth. Learning to reason","authors":"Roth Khardon","year":"1997","venue":"Journal of the ACM (JACM)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Koh et al., 2021] Pang Wei Koh, Shiori Sagawa, Hen- rik Marklund, Sang Michael Xie, Marvin Zhang, Ak- shay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: A benchmark of in-the-wild distribution shifts. In Inter- national Conference on Machine Learning, pages 5637- 5664. PMLR, 2021.","title":"Wilds: A benchmark of in-the-wild distribution shifts","authors":"Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lample and Charton, 2020] Guillaume Lample and Franc ¸ois Charton. Deep learning for symbolic mathemat- ics. In ICLR. OpenReview.net, 2020.","title":"Guillaume Lample and Franc ¸ois Charton. Deep learning for symbolic mathematics","authors":"Charton Lample","year":"2020","venue":"ICLR. OpenReview.net","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lin et al., 2019] Kevin Lin, Oyvind Tafjord, Peter Clark, and Matt Gardner. Reasoning over paragraph effects in situations. arXiv preprint arXiv:1908.05852, 2019.","title":"Reasoning over paragraph effects in situations","authors":"Kevin Lin, Oyvind Tafjord, Peter Clark, Matt Gardner","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Liu et al., 2020] Jian Liu, Leyang Cui, Hanmeng Liu, Dan- dan Huang, Yile Wang, and Yue Zhang. Logiqa: A chal- lenge dataset for machine reading comprehension with logical reasoning. In IJCAI. ijcai.org, 2020.","title":"Logiqa: A challenge dataset for machine reading comprehension with logical reasoning","authors":"Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, Yue Zhang","year":"2020","venue":"IJCAI. ijcai.org","doi":null,"url":null,"internal_url":"/proceedings/2020/501"},{"position":20,"raw":"[Maas et al., 2011] Andrew L. Maas, Raymond E. Daly, Pe- ter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Tech- nologies, pages 142-150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics.","title":"Learning word vectors for sentiment analysis","authors":"Andrew L Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, Christopher Potts","year":"2011","venue":"Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[McCoy et al., 2019] Tom McCoy, Ellie Pavlick, and Tal Linzen. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computa- tional Linguistics, pages 3428-3448, 2019.","title":"Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference","authors":"Mccoy","year":"2019","venue":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Raffel et al., 2020] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(140):1-67, 2020.","title":"Exploring the limits of transfer learning with a unified text-to-text transformer","authors":"Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu","year":"2020","venue":"J. Mach. Learn. Res","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Rajpurkar et al., 2016] Pranav Rajpurkar, Jian Zhang, Kon- stantin Lopyrev, and Percy Liang. Squad: 100, 000+ ques- tions for machine comprehension of text. In EMNLP, pages 2383-2392. The Association for Computational Linguistics, 2016.","title":"Squad: 100, 000+ questions for machine comprehension of text","authors":"Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, Percy Liang","year":"2016","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Saxton et al., 2019] David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. Analysing mathemati- cal reasoning abilities of neural models. In ICLR (Poster). OpenReview.net, 2019.","title":"Analysing mathematical reasoning abilities of neural models","authors":"David Saxton, Edward Grefenstette, Felix Hill, Pushmeet Kohli","year":"2019","venue":"ICLR (Poster). OpenReview.net","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Selsam et al., 2019] Daniel Selsam, Matthew Lamm, Benedikt Bünz, Percy Liang, Leonardo de Moura, and David L. Dill. Learning a SAT solver from single-bit supervision. In ICLR (Poster). OpenReview.net, 2019.","title":"Learning a SAT solver from single-bit supervision","authors":"Daniel Selsam, Matthew Lamm, Benedikt Bünz, Percy Liang, Leonardo De Moura, David L Dill","year":"2019","venue":"ICLR (Poster)","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Sinha et al., 2019] Koustuv Sinha, Shagun Sodhani, Jin Dong, Joelle Pineau, and William L. Hamilton. CLUTRR: A diagnostic benchmark for inductive reasoning from text. In EMNLP/IJCNLP (1), pages 4505-4514. Association for Computational Linguistics, 2019.","title":"CLUTRR: A diagnostic benchmark for inductive reasoning from text","authors":"Sinha","year":"2019","venue":"EMNLP/IJCNLP (1)","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Talmor et al., 2019] Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge. In NAACL-HLT (1), pages 4149-4158. Association for Computational Linguistics, 2019.","title":"Commonsenseqa: A question answering challenge targeting commonsense knowledge","authors":"Alon Talmor, Jonathan Herzig, Nicholas Lourie, Jonathan Berant","year":"2019","venue":"NAACL-HLT (1)","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Talmor et al., 2020] Alon Talmor, Oyvind Tafjord, Peter Clark, Yoav Goldberg, and Jonathan Berant. Leap-of- thought: Teaching pre-trained models to systematically reason over implicit knowledge. Advances in Neural In- formation Processing Systems, 33:20227-20237, 2020.","title":"Leap-ofthought: Teaching pre-trained models to systematically reason over implicit knowledge","authors":"Alon Talmor, Oyvind Tafjord, Peter Clark, Yoav Goldberg, Jonathan Berant","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Tian et al., 2021] Jidong Tian, Yitian Li, Wenqing Chen, Liqiang Xiao, Hao He, and Yaohui Jin. Diagnosing the first-order logical reasoning ability through logicnli. In EMNLP (1). Association for Computational Linguistics, 2021.","title":"Diagnosing the first-order logical reasoning ability through logicnli","authors":"Jidong Tian, Yitian Li, Wenqing Chen, Liqiang Xiao, Hao He, Yaohui Jin","year":"2021","venue":"Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Torralba and Efros, 2011] Antonio Torralba and Alexei A. Efros. Unbiased look at dataset bias. In CVPR, 2011.","title":"Unbiased look at dataset bias","authors":"Efros Torralba, Antonio Torralba, Alexei A Efros","year":"2011","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, pages 5998-6008, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, Illia Polosukhin","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Wald et al., 2021] Yoav Wald, Amir Feder, Daniel Green- feld, and Uri Shalit. On calibration and out-of-domain generalization. In NeurIPS, pages 2215-2227, 2021.","title":"On calibration and out-of-domain generalization","authors":"Yoav Wald, Amir Feder, Daniel Greenfeld, Uri Shalit","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Wang et al., 2019] Po-Wei Wang, Priya L. Donti, Bryan Wilder, and J. Zico Kolter. Satnet: Bridging deep learning and logical reasoning using a differentiable satisfiability solver. In ICML, Proceedings of Machine Learning Re- search. PMLR, 2019.","title":"Satnet: Bridging deep learning and logical reasoning using a differentiable satisfiability solver","authors":"Po-Wei Wang, Priya L Donti, Bryan Wilder, J Zico Kolter","year":"2019","venue":"ICML, Proceedings of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Welleck et al., 2021] Sean Welleck, Jiacheng Liu, Ro- nan Le Bras, Hannaneh Hajishirzi, Yejin Choi, and Kyunghyun Cho. Naturalproofs: Mathematical the- orem proving in natural language. arXiv preprint arXiv:2104.01112, 2021.","title":"Naturalproofs: Mathematical theorem proving in natural language","authors":"Sean Welleck, Jiacheng Liu, Ronan Le Bras, Hannaneh Hajishirzi, Yejin Choi, Kyunghyun Cho","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Williams et al., 2018] Adina Williams, Nikita Nangia, and Samuel R. Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In NAACL- HLT. Association for Computational Linguistics, 2018.","title":"A broad-coverage challenge corpus for sentence understanding through inference","authors":"Adina Williams, Nikita Nangia, Samuel R Bowman","year":"2018","venue":"NAACL-HLT. Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Xu et al., 2019] Keyulu Xu, Jingling Li, Mozhi Zhang, Si- mon S Du, Ken-ichi Kawarabayashi, and Stefanie Jegelka. What can neural networks reason about? arXiv preprint arXiv:1905.13211, 2019.","title":"What can neural networks reason about","authors":"Keyulu Xu, Jingling Li, Mozhi Zhang, Simon S Du, Ken-Ichi Kawarabayashi, Stefanie Jegelka","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Yang et al., 2018] Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdi- nov, and Christopher D. Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In EMNLP, pages 2369-2380. Association for Computa- tional Linguistics, 2018.","title":"Hotpotqa: A dataset for diverse, explainable multi-hop question answering","authors":"Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, Christopher D Manning","year":"2018","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Yin et al., 2020] Fan Yin, Quanyu Long, Tao Meng, and Kai-Wei Chang. On the robustness of language encoders against grammatical errors. In ACL, pages 3386-3403. As- sociation for Computational Linguistics, 2020.","title":"On the robustness of language encoders against grammatical errors","authors":"Fan Yin, Quanyu Long, Tao Meng, Kai-Wei Chang","year":"2020","venue":"Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Yin et al., 2021] Da Yin, Liunian Harold Li, Ziniu Hu, Nanyun Peng, and Kai-Wei Chang. Broaden the vision: Geo-diverse visual commonsense reasoning. In EMNLP (1), pages 2115-2129. Association for Computational Lin- guistics, 2021.","title":"Broaden the vision: Geo-diverse visual commonsense reasoning","authors":"Liunian Da Yin, Ziniu Harold Li, Nanyun Hu, Kai-Wei Peng, Chang","year":"2021","venue":"EMNLP (1)","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Yu et al., 2020] Weihao Yu, Zihang Jiang, Yanfei Dong, and Jiashi Feng. Reclor: A reading comprehension dataset re- quiring logical reasoning. In International Conference on Learning Representations (ICLR), April 2020.","title":"Reclor: A reading comprehension dataset requiring logical reasoning","authors":"Yu","year":"2020","venue":"International Conference on Learning Representations (ICLR)","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Zellers et al., 2018] Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. SWAG: A large-scale ad- versarial dataset for grounded commonsense inference. In EMNLP. Association for Computational Linguistics, 2018.","title":"SWAG: A large-scale adversarial dataset for grounded commonsense inference","authors":"Rowan Zellers, Yonatan Bisk, Roy Schwartz, Yejin Choi","year":"2018","venue":"EMNLP. Association for Computational Linguistics","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}