{"type":"ScholarlyArticle","title":"Large Language Models Are Not Strong Abstract Reasoners","authors":[{"name":"Gaël Gendron","slug":"gael-gendron","given":"Gaël","family":"Gendron","affiliation":"University of Auckland"},{"name":"Qiming Bao","slug":"qiming-bao","given":"Qiming","family":"Bao","affiliation":"University of Auckland"},{"name":"Michael Witbrock","slug":"michael-witbrock","given":"Michael","family":"Witbrock","affiliation":"University of Auckland"},{"name":"Gillian Dobbie","slug":"gillian-dobbie","given":"Gillian","family":"Dobbie","affiliation":"University of Auckland"}],"abstract":"Large Language Models have shown tremendous performance on a large variety of natural language processing tasks, ranging from text comprehension to common sense reasoning. \r\nHowever, the mechanisms responsible for this success remain opaque, and it is unclear whether LLMs can achieve human-like cognitive capabilities or whether these models are still fundamentally circumscribed. Abstract reasoning is a fundamental task for cognition, consisting of finding and applying a general pattern from few data. Evaluating deep neural architectures on this task could give insight into their potential limitations regarding reasoning and their broad generalisation abilities, yet this is currently an under-explored area. In this paper, we introduce a new benchmark for evaluating language models beyond memorization on abstract reasoning tasks. We perform extensive evaluations of state-of-the-art LLMs, showing that they currently achieve very limited performance in contrast with other natural language tasks, even when applying techniques that have been shown to improve performance on other NLP tasks. We argue that guiding LLM generation to follow causal paths could help improve the generalisation and reasoning abilities of LLMs.","doi":"10.24963/ijcai.2024/693","url":"https://ijcai.org/proceedings/2024/693","pdf":"https://ijcai.org/proceedings/2024/0693.pdf","markdown":"https://ijcai.org/proceedings/2024/693.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/693","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Natural Language Processing","track":"Main Track","number":693,"pages":{"first":6270,"last":6278},"keywords":["Machine Learning: ML: Robustness","Machine Learning: ML: Evaluation","Natural Language Processing: NLP: Question answering","Natural Language Processing: NLP: Language models"],"videos":[],"references":[{"position":1,"raw":"[Acquaviva et al., 2021] Samuel Acquaviva, Yewen Pu, Marta Kryven, Catherine Wong, Gabrielle E. Ecanow, Maxwell I. Nye, Theodoros Sechopoulos, Michael Henry Tessler, and Joshua B. Tenenbaum. Communicating natural programs to humans and machines. CoRR, abs/2106.07824, 2021.","title":"Communicating natural programs to humans and machines","authors":"Samuel Acquaviva, Yewen Pu, Marta Kryven, Catherine Wong, Gabrielle E Ecanow, Maxwell I Nye, Theodoros Sechopoulos, Michael Henry Tessler, Joshua B Tenenbaum","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bender et al., 2021] Emily M. Bender, Timnit Gebru, An- gelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In FAccT '21: 2021 ACM Conference on Fairness, Accountability, and Transparency, Virtual Event / Toronto, Canada, March 3-10, 2021, pages 610-623. ACM, 2021.","title":"Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big?","authors":"Emily M Bender, Timnit Gebru","year":"2021","venue":"FAccT '21: 2021 ACM Conference on Fairness, Accountability, and Transparency, Virtual Event","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Booch et al., 2021] Grady Booch, Francesco Fabiano, Lior Horesh, Kiran Kate, Jonathan Lenchner, Nick Linck, An- dreas Loreggia, Keerthiram Murgesan, Nicholas Mattei, Francesca Rossi, et al. Thinking fast and slow in AI. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, 2021.","title":"Thinking fast and slow in AI","authors":"Grady Booch, Francesco Fabiano, Lior Horesh, Kiran Kate, Jonathan Lenchner, Nick Linck, Andreas Loreggia, Keerthiram Murgesan, Nicholas Mattei, Francesca Rossi","year":"2021","venue":"Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Brown et al., 2020] Tom Brown, Benjamin Mann, Nick Ry- der, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In NeurIPS, 2020.","title":"Language models are few-shot learners","authors":"Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bubeck et al., 2023] Sébastien Bubeck, Varun Chan- drasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with GPT-4. CoRR, abs/2303.12712, 2023.","title":"Sparks of artificial general intelligence: Early experiments with GPT-4","authors":"Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Carpenter et al., 1990] Patricia A Carpenter, Marcel A Just, and Peter Shell. What one intelligence test measures: a theoretical account of the processing in the raven progres- sive matrices test. Psychological review, 97(3):404, 1990.","title":"What one intelligence test measures: a theoretical account of the processing in the raven progressive matrices test","authors":"Patricia A Carpenter, Marcel A Just, Peter Shell","year":"1990","venue":"Psychological review","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chen et al., 2022] Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. CoRR, abs/2211.12588, 2022.","title":"Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks","authors":"Chen","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chollet, 2019] Franc ¸ois Chollet. On the measure of intelli- gence. CoRR, abs/1911.01547, 2019.","title":"Franc ¸ois Chollet. On the measure of intelligence","authors":"Chollet","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Devlin et al., 2019] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understand- ing. In Proceedings of NAACL-HLT, pages 4171-4186. Association for Computational Linguistics, 2019.","title":"BERT: pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2019","venue":"Proceedings of NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ellis et al., 2020] Kevin Ellis, Catherine Wong, Maxwell I. Nye, Mathias Sablé-Meyer, Luc Cary, Lucas Morales, Luke B. Hewitt, Armando Solar-Lezama, and Joshua B. Tenenbaum. Dreamcoder: Growing generalizable, in- terpretable knowledge with wake-sleep bayesian program learning. CoRR, abs/2006.08381, 2020.","title":"Dreamcoder: Growing generalizable, interpretable knowledge with wake-sleep bayesian program learning","authors":"Kevin Ellis, Catherine Wong, Maxwell I Nye, Mathias Sablé-Meyer, Luc Cary, Lucas Morales, Luke B Hewitt, Armando Solar-Lezama, Joshua B Tenenbaum","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Gendron et al., 2023] Gaël Gendron, Michael Witbrock, and Gillian Dobbie. A survey of methods, challenges and perspectives in causality. CoRR, abs/2302.00293, 2023.","title":"A survey of methods, challenges and perspectives in causality","authors":"Gaël Gendron, Michael Witbrock, Gillian Dobbie","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Goyal and Bengio, 2020] Anirudh Goyal and Yoshua Ben- gio. Inductive biases for deep learning of higher-level cog- nition. CoRR, abs/2011.15091, 2020.","title":"Inductive biases for deep learning of higher-level cognition","authors":"Anirudh Goyal, Yoshua Bengio","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hu et al., 2022] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large lan- guage models. In The Tenth International Conference on Learning Representations, ICLR 2022. OpenReview.net, 2022.","title":"Lora: Low-rank adaptation of large language models","authors":"Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen","year":"2022","venue":"The Tenth International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Jiang et al., 2023] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. CoRR, abs/2310.06825, 2023.","title":"Mistral 7b","authors":"Jiang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Jiao et al., 2022] Fangkai Jiao, Yangyang Guo, Xuemeng Song, and Liqiang Nie. Merit: Meta-path guided con- trastive learning for logical reasoning. In Findings of the Association for Computational Linguistics: ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 3496-3509. As- sociation for Computational Linguistics, 2022.","title":"Merit: Meta-path guided contrastive learning for logical reasoning","authors":"Jiao","year":"2022","venue":"Findings of the Association for Computational Linguistics: ACL 2022","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Jin et al., 2023] Zhijing Jin, Jiarui Liu, Zhiheng Lyu, Spencer Poff, Mrinmaya Sachan, Rada Mihalcea, Mona T. Diab, and Bernhard Schölkopf. Can large language models infer causation from correlation? CoRR, abs/2306.05836, 2023.","title":"Can large language models infer causation from correlation?","authors":"Jin","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Kiciman et al., 2023] Emre Kiciman, Robert Ness, Amit Sharma, and Chenhao Tan. Causal reasoning and large language models: Opening a new frontier for causality. CoRR, abs/2305.00050, 2023.","title":"Causal reasoning and large language models: Opening a new frontier for causality","authors":"Emre Kiciman, Robert Ness, Amit Sharma, Chenhao Tan","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lake et al., 2015] Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learn- ing through probabilistic program induction. Science, 350(6266):1332-1338, 2015.","title":"Human-level concept learning through probabilistic program induction","authors":"Ruslan Brenden M Lake, Joshua B Salakhutdinov, Tenenbaum","year":"2015","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Li and Flanigan, 2024] Changmao Li and Jeffrey Flanigan. Task contamination: Language models may not be few- shot anymore. In Michael J. Wooldridge, Jennifer G. Dy, and Sriraam Natarajan, editors, Thirty-Eighth AAAI Con- ference on Artificial Intelligence, AAAI 2024, 2024.","title":"Task contamination: Language models may not be fewshot anymore","authors":"Li, Flanigan, Changmao Li, Jeffrey Flanigan","year":"2024","venue":"Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Li et al., 2022] Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. On the advance of making language models better reasoners. CoRR, abs/2206.02336, 2022.","title":"On the advance of making language models better reasoners","authors":"Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, Weizhu Chen","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Liu et al., 2019] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692, 2019.","title":"Roberta: A robustly optimized BERT pretraining approach","authors":"Liu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Madaan et al., 2023] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. CoRR, abs/2303.17651, 2023.","title":"Self-refine: Iterative refinement with self-feedback","authors":"Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[OpenAI, 2023] OpenAI. GPT-4 technical report. CoRR, abs/2303.08774, 2023.","title":"OpenAI. GPT-4 technical report","authors":"Openai","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Ouyang et al., 2022] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. In NeurIPS, 2022.","title":"Training language models to follow instructions with human feedback","authors":"Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray","year":"2022","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Qiu et al., 2023] Linlu Qiu, Liwei Jiang, Ximing Lu, Melanie Sclar, Valentina Pyatkin, Chandra Bhagavatula, Bailin Wang, Yoon Kim, Yejin Choi, Nouha Dziri, et al. Phenomenal yet puzzling: Testing inductive reasoning ca- pabilities of language models with hypothesis refinement. CoRR, abs/2310.08559, 2023.","title":"Phenomenal yet puzzling: Testing inductive reasoning capabilities of language models with hypothesis refinement","authors":"Qiu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Radford et al., 2019] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.","title":"Language models are unsupervised multitask learners","authors":"Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever","year":"2019","venue":"OpenAI blog","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Raven, 1938] John C Raven. Raven standard progressive matrices. Journal of Cognition and Development, 1938.","title":"Raven standard progressive matrices","authors":"John C Raven","year":"1938","venue":"Journal of Cognition and Development","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Rule, 2020] Joshua Stewart Rule. The child as hacker: building more human-like models of learning. PhD the- sis, Massachusetts Institute of Technology, 2020.","title":"The child as hacker: building more human-like models of learning","authors":"Joshua Stewart, Rule","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Santoro et al., 2018] Adam Santoro, Felix Hill, David G. T. Barrett, Ari S. Morcos, and Timothy P. Lillicrap. Measur- ing abstract reasoning in neural networks. In Proceedings of the 35th International Conference on Machine Learn- ing, ICML 2018, 2018.","title":"Measuring abstract reasoning in neural networks","authors":"Adam Santoro, Felix Hill, G T David, Ari S Barrett, Timothy P Morcos, Lillicrap","year":"2018","venue":"Proceedings of the 35th International Conference on Machine Learning, ICML 2018","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Schölkopf et al., 2021] Bernhard Schölkopf, Francesco Lo- catello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbren- ner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proc. IEEE, 109(5):612-634, 2021.","title":"Toward causal representation learning","authors":"Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, Yoshua Bengio","year":"2021","venue":"Proc. IEEE","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Srivastava et al., 2022] Aarohi Srivastava, Abhinav Ras- togi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. Beyond the imita- tion game: Quantifying and extrapolating the capabilities of language models. CoRR, abs/2206.04615, 2022.","title":"Beyond the imitation game: Quantifying and extrapolating the capabilities of language models","authors":"Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal, Md Shoeb, Abubakar Abid, Adam Fisch, Adam Adam R Brown, Aditya Santoro, Adrià Gupta, Garriga-Alonso","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Stolfo et al., 2022] Alessandro Stolfo, Zhijing Jin, Kumar Shridhar, Bernhard Schölkopf, and Mrinmaya Sachan. A causal framework to quantify the robustness of math- ematical reasoning with language models. CoRR, abs/2210.12023, 2022.","title":"A causal framework to quantify the robustness of mathematical reasoning with language models","authors":"Alessandro Stolfo, Zhijing Jin, Kumar Shridhar, Bernhard Schölkopf, Mrinmaya Sachan","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Talmor et al., 2020] Alon Talmor, Oyvind Tafjord, Peter Clark, Yoav Goldberg, and Jonathan Berant. Leap-of- thought: Teaching pre-trained models to systematically reason over implicit knowledge. In NeurIPS, 2020.","title":"Leap-ofthought: Teaching pre-trained models to systematically reason over implicit knowledge","authors":"Alon Talmor, Oyvind Tafjord, Peter Clark, Yoav Goldberg, Jonathan Berant","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Tang et al., 2023] Xiaojuan Tang, Zilong Zheng, Jiaqi Li, Fanxu Meng, Song-Chun Zhu, Yitao Liang, and Muhan Zhang. Large language models are in-context seman- tic reasoners rather than symbolic reasoners. CoRR, abs/2305.14825, 2023.","title":"Large language models are in-context semantic reasoners rather than symbolic reasoners","authors":"Xiaojuan Tang, Zilong Zheng, Jiaqi Li, Fanxu Meng, Song-Chun Zhu, Yitao Liang, Muhan Zhang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Taori et al., 2023] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model, 2023.","title":"Stanford alpaca: An instruction-following llama model","authors":"Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, Tatsunori B Hashimoto","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Tirumala et al., 2022] Kushal Tirumala, Aram H. Markosyan, Luke Zettlemoyer, and Armen Agha- janyan. Memorization without overfitting: Analyzing the training dynamics of large language models. In NeurIPS, 2022.","title":"Memorization without overfitting: Analyzing the training dynamics of large language models","authors":"Kushal Tirumala, Aram H Markosyan, Luke Zettlemoyer, Armen Aghajanyan","year":"2022","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Touvron et al., 2023a] Hugo Touvron, Thibaut Lavril, Gau- tier Izacard, Xavier Martinet, Marie-Anne Lachaux, Tim- othée Lacroix, Baptiste Rozière, Naman Goyal, Eric Ham- bro, Faisal Azhar, et al. Llama: Open and efficient foun- dation language models. CoRR, abs/2302.13971, 2023.","title":"Llama: Open and efficient foundation language models","authors":"Touvron","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Touvron et al., 2023b] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine- tuned chat models. CoRR, abs/2307.09288, 2023.","title":"Llama 2: Open foundation and finetuned chat models","authors":"Touvron","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Tunstall et al., 2023a] Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. Zephyr: Direct distillation of LM alignment. CoRR, abs/2310.16944, 2023.","title":"Zephyr: Direct distillation of LM alignment","authors":"Tunstall","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Tunstall et al., 2023b] Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Alexander M. Rush, and Thomas Wolf. The alignment handbook. https: //github.com/huggingface/alignment-handbook, 2023.","title":"The alignment handbook","authors":"Tunstall","year":"2023","venue":null,"doi":null,"url":"https://github.com/huggingface/alignment-handbook,2023","internal_url":null},{"position":41,"raw":"[Wang et al., 2022] Yizhong Wang, Yeganeh Kordi, Swa- roop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning lan- guage model with self generated instructions. CoRR, abs/2212.10560, 2022.","title":"Self-instruct: Aligning language model with self generated instructions","authors":"Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, Hannaneh Hajishirzi","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Wang et al., 2023] Ruocheng Wang, Eric Zelikman, Gabriel Poesia, Yewen Pu, Nick Haber, and Noah D. Goodman. Hypothesis search: Inductive reasoning with language models. CoRR, abs/2309.05660, 2023.","title":"Hypothesis search: Inductive reasoning with language models","authors":"Wang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Wei et al., 2022] Jason Wei, Xuezhi Wang, Dale Schuur- mans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits rea- soning in large language models. In NeurIPS, 2022.","title":"Chain-of-thought prompting elicits reasoning in large language models","authors":"Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, V Quoc, Denny Le, Zhou","year":"2022","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":44,"raw":"[Wu et al., 2023] Zhaofeng Wu, Linlu Qiu, Alexis Ross, Ekin Akyürek, Boyuan Chen, Bailin Wang, Najoung Kim, Jacob Andreas, and Yoon Kim. Reasoning or reciting? ex- ploring the capabilities and limitations of language mod- els through counterfactual tasks. CoRR, abs/2307.02477, 2023.","title":"Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks","authors":"Wu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":45,"raw":"[Xu et al., 2023] Fangzhi Xu, Qika Lin, Jiawei Han, Tianzhe Zhao, Jun Liu, and Erik Cambria. Are large language models really good logical reasoners? A comprehensive evaluation from deductive, inductive and abductive views. CoRR, abs/2306.09841, 2023.","title":"Are large language models really good logical reasoners? A comprehensive evaluation from deductive, inductive and abductive views","authors":"Xu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":46,"raw":"[Zecevic et al., 2023] Matej Zecevic, Moritz Willig, Deven- dra Singh Dhami, and Kristian Kersting. Causal par- rots: Large language models may talk causality but are not causal. CoRR, abs/2308.13067, 2023.","title":"Causal parrots: Large language models may talk causality but are not causal","authors":"Matej Zecevic, Moritz Willig, Devendra Singh Dhami, Kristian Kersting","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":47,"raw":"[Zhang et al., 2019] Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song-Chun Zhu. RAVEN: A dataset for re- lational and analogical visual reasoning. In IEEE Confer- ence on Computer Vision and Pattern Recognition, CVPR 2019. Computer Vision Foundation / IEEE, 2019.","title":"RAVEN: A dataset for relational and analogical visual reasoning","authors":"Zhang","year":"2019","venue":"IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019","doi":null,"url":null,"internal_url":null},{"position":48,"raw":"[Zhang et al., 2021a] Chi Zhang, Baoxiong Jia, Mark Ed- monds, Song-Chun Zhu, and Yixin Zhu. ACRE: abstract causal reasoning beyond covariation. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021. Computer Vision Foundation / IEEE, 2021.","title":"ACRE: abstract causal reasoning beyond covariation","authors":"Zhang","year":"2021","venue":"IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021","doi":null,"url":null,"internal_url":null},{"position":49,"raw":"[Zhang et al., 2021b] Chiyuan Zhang, Maithra Raghu, Jon M. Kleinberg, and Samy Bengio. Pointer value retrieval: A new benchmark for understanding the limits of neural network generalization. CoRR, abs/2107.12580, 2021.","title":"Pointer value retrieval: A new benchmark for understanding the limits of neural network generalization","authors":"Zhang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}