{"type":"ScholarlyArticle","title":"It Ain’t That Bad: Understanding the Mysterious Performance Drop in OOD Generalization for Generative Transformer Models","authors":[{"name":"Xingcheng Xu","slug":"xingcheng-xu","given":"Xingcheng","family":"Xu","affiliation":"Shanghai Artificial Intelligence Laboratory"},{"name":"Zihao Pan","slug":"zihao-pan","given":"Zihao","family":"Pan","affiliation":"ShanghaiTech University"},{"name":"Haipeng Zhang","slug":"haipeng-zhang","given":"Haipeng","family":"Zhang","affiliation":"ShanghaiTech University"},{"name":"Yanqing Yang","slug":"yanqing-yang","given":"Yanqing","family":"Yang","affiliation":"Shanghai Artificial Intelligence Laboratory; Fudan University"}],"abstract":"Large language models (LLMs) have achieved remarkable proficiency on solving diverse problems. However, their generalization ability is not always satisfying and the generalization problem is common for generative transformer models in general. Researchers take basic mathematical tasks like n-digit addition or multiplication as important perspectives for investigating their generalization behaviors. It is observed that when training models on n-digit operations (e.g., additions) in which both input operands are n-digit in length, models generalize successfully on unseen n-digit inputs (in-distribution (ID) generalization), but fail miserably on longer, unseen cases (out-of-distribution (OOD) generalization). We bring this unexplained performance drop into attention and ask whether there is systematic OOD generalization. Towards understanding LLMs, we train various smaller language models which may share the same underlying mechanism. We discover that the strong ID generalization stems from structured representations, while behind the unsatisfying OOD performance, the models still exhibit clear learned algebraic structures. Specifically, these models map unseen OOD inputs to outputs with learned equivalence relations in the ID domain, which we call the equivalence generalization. These findings deepen our knowledge regarding the generalizability of generative models including LLMs, and provide insights into potential avenues for improvement.","doi":"10.24963/ijcai.2024/727","url":"https://ijcai.org/proceedings/2024/727","pdf":"https://ijcai.org/proceedings/2024/0727.pdf","markdown":"https://ijcai.org/proceedings/2024/727.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/727","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Natural Language Processing","track":"Main Track","number":727,"pages":{"first":6578,"last":6586},"keywords":["AI Ethics, Trust, Fairness: ETF: Explainability and interpretability","Natural Language Processing: NLP: Interpretability and analysis of models for NLP","Knowledge Representation and Reasoning: KRR: Learning and reasoning"],"videos":[],"references":[{"position":1,"raw":"Nadeem Akhtar and U Ragavendran. Interpretation of intelli- gence in cnn-pooling processes: a methodological survey. Neural computing and applications, 32(3):879-898, 2020.","title":"Interpretation of intelligence in cnn-pooling processes: a methodological survey","authors":"Nadeem Akhtar, U Ragavendran","year":"2020","venue":"Neural computing and applications","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"Cem Anil, Yuhuai Wu, Anders Andreassen, Aitor Lewkowycz, Vedant Misra, Vinay Ramasesh, Ambrose Slone, Guy Gur-Ari, Ethan Dyer, and Behnam Neyshabur. Exploring length generalization in large language models. Advances in Neural Information Processing Systems, 35:38546-38556, 2022.","title":"Exploring length generalization in large language models","authors":"Cem Anil, Yuhuai Wu, Anders Andreassen, Aitor Lewkowycz, Vedant Misra, Vinay Ramasesh, Ambrose Slone, Guy Gur-Ari, Ethan Dyer, Behnam Neyshabur","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"Anthropic. Model card and evaluations for claude models. 2023.","title":"Model card and evaluations for claude models","authors":"Anthropic","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"Pietro Barbiero, Gabriele Ciravegna, Francesco Giannini, Pietro Lió, Marco Gori, and Stefano Melacci. Entropy- based logic explanations of neural networks. In Proceed- ings of the AAAI Conference on Artificial Intelligence, vol- ume 36, pages 6046-6054, 2022.","title":"Entropybased logic explanations of neural networks","authors":"Pietro Barbiero, Gabriele Ciravegna, Francesco Giannini, Pietro Lió, Marco Gori, Stefano Melacci","year":"2022","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 610-623, 2021.","title":"On the dangers of stochastic parrots: Can language models be too big?","authors":"Emily M Bender, Timnit Gebru, Angelina Mcmillan-Major, Shmargaret Shmitchell","year":"2021","venue":"Proceedings of the 2021 ACM conference on fairness, accountability, and transparency","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Lan- guage models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.","title":"Language models are few-shot learners","authors":"Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell","year":"2020","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712, 2023.","title":"Sparks of artificial general intelligence: Early experiments with gpt-4","authors":"Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-113, 2023.","title":"Palm: Scaling language modeling with pathways","authors":"Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann","year":"2023","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"Yann Dubois, Gautier Dagan, Dieuwke Hupkes, and Elia Bruni. Location attention for extrapolation to longer se- quences. arXiv preprint arXiv:1911.03872, 2019.","title":"Location attention for extrapolation to longer sequences","authors":"Yann Dubois, Gautier Dagan, Dieuwke Hupkes, Elia Bruni","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"Samy Jelassi, Stéphane d'Ascoli, Carles Domingo-Enrich, Yuhuai Wu, Yuanzhi Li, and Franc ¸ois Charton. Length generalization in arithmetic transformers. arXiv preprint arXiv:2306.15400, 2023.","title":"Length generalization in arithmetic transformers","authors":"Samy Jelassi, Carles Stéphane D'ascoli, Yuhuai Domingo-Enrich, Yuanzhi Wu, Franc Li, ¸ois Charton","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"Andrej Karpathy. A minimal pytorch re-implementation of the openai gpt (generative pretrained transformer) training. GitHub https://github.com/karpathy/minGPT, 2022.","title":"A minimal pytorch re-implementation of the openai gpt (generative pretrained transformer) training","authors":"Andrej Karpathy","year":null,"venue":null,"doi":null,"url":"https://github.com/karpathy/minGPT,2022","internal_url":null},{"position":12,"raw":"Nayoung Lee, Kartik Sreenivasan, Jason D Lee, Kangwook Lee, and Dimitris Papailiopoulos. Teaching arithmetic to small transformers. arXiv preprint arXiv:2307.03381, 2023.","title":"Teaching arithmetic to small transformers","authors":"Nayoung Lee, Kartik Sreenivasan, Jason D Lee, Kangwook Lee, Dimitris Papailiopoulos","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"Ziming Liu, Ouail Kitouni, Niklas S Nolte, Eric Michaud, Max Tegmark, and Mike Williams. Towards understand- ing grokking: An effective theory of representation learn- ing. Advances in Neural Information Processing Systems, 35:34651-34663, 2022.","title":"Towards understanding grokking: An effective theory of representation learning","authors":"Ziming Liu, Ouail Kitouni, Eric Niklas S Nolte, Max Michaud, Mike Tegmark, Williams","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"Woo-Jeoung Nam, Shir Gur, Jaesik Choi, Lior Wolf, and Seong-Whan Lee. Relative attributing propagation: Inter- preting the comparative contributions of individual units in deep neural networks. In Proceedings of the AAAI con- ference on artificial intelligence, volume 34, pages 2501- 2508, 2020.","title":"Relative attributing propagation: Interpreting the comparative contributions of individual units in deep neural networks","authors":"Woo-Jeoung Nam, Shir Gur, Jaesik Choi, Lior Wolf, Seong-Whan Lee","year":"2020","venue":"Proceedings of the AAAI conference on artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"Neel Nanda and Tom Lieberum. A mechanistic interpretabil- ity analysis of grokking. In Alignment Forum, 2022.","title":"A mechanistic interpretability analysis of grokking","authors":"Neel Nanda, Tom Lieberum","year":"2022","venue":"Alignment Forum","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"Rodrigo Nogueira, Zhiying Jiang, and Jimmy Lin. Investigat- ing the limitations of transformers with simple arithmetic tasks. arXiv preprint arXiv:2102.13019, 2021.","title":"Investigating the limitations of transformers with simple arithmetic tasks","authors":"Rodrigo Nogueira, Zhiying Jiang, Jimmy Lin","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"OpenAI. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023.","title":"Gpt-4 technical report","authors":"Openai","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Car- roll Wainwright, Pamela Mishkin, Chong Zhang, Sand- hini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feed- back. Advances in Neural Information Processing Systems, 35:27730-27744, 2022.","title":"Training language models to follow instructions with human feedback","authors":"Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"Jing Qian, Hong Wang, Zekun Li, Shiyang Li, and Xifeng Yan. Limitations of language models in arithmetic and symbolic induction. arXiv preprint arXiv:2208.05051, 2022.","title":"Limitations of language models in arithmetic and symbolic induction","authors":"Jing Qian, Hong Wang, Zekun Li, Shiyang Li, Xifeng Yan","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Mar- tinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.","title":"Llama: Open and efficient foundation language models","authors":"Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.","title":"Llama 2: Open foundation and fine-tuned chat models","authors":"Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"Immanuel Trummer. Codexdb: Synthesizing code for query processing from natural language instructions us- ing gpt-3 codex. Proceedings of the VLDB Endowment, 15(11):2921-2928, 2022.","title":"Codexdb: Synthesizing code for query processing from natural language instructions using gpt-3 codex","authors":"Immanuel Trummer","year":"2022","venue":"Proceedings of the VLDB Endowment","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"Han Xuanyuan, Pietro Barbiero, Dobrik Georgiev, Lu- cie Charlotte Magister, and Pietro Lió. Global concept- based interpretability for graph neural networks via neuron analysis. In Proceedings of the AAAI Conference on Artifi- cial Intelligence, volume 37, pages 10675-10683, 2023.","title":"Global conceptbased interpretability for graph neural networks via neuron analysis","authors":"Han Xuanyuan, Pietro Barbiero, Dobrik Georgiev, Lucie, Charlotte Magister, Pietro Lió","year":"2023","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"Hao Yuan, Yongjun Chen, Xia Hu, and Shuiwang Ji. Inter- preting deep models for text analysis via optimization and regularization methods. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 33, pages 5717- 5724, 2019.","title":"Interpreting deep models for text analysis via optimization and regularization methods","authors":"Yongjun Hao Yuan, Xia Chen, Shuiwang Hu, Ji","year":"2019","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"Hao Yuan, Jiliang Tang, Xia Hu, and Shuiwang Ji. Xgnn: Towards model-level explanations of graph neural net- works. In Proceedings of the 26th ACM SIGKDD Interna- tional Conference on Knowledge Discovery & Data Min- ing, pages 430-438, 2020.","title":"Xgnn: Towards model-level explanations of graph neural networks","authors":"Jiliang Hao Yuan, Xia Tang, Shuiwang Hu, Ji","year":"2020","venue":"Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"Ziqian Zhong, Ziming Liu, Max Tegmark, and Jacob An- dreas. The clock and the pizza: Two stories in mech- anistic explanation of neural networks. arXiv preprint arXiv:2306.17844, 2023.","title":"The clock and the pizza: Two stories in mechanistic explanation of neural networks","authors":"Ziqian Zhong, Ziming Liu, Max Tegmark, Jacob Andreas","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"Mingyu Zong and Bhaskar Krishnamachari. Solving math word problems concerning systems of equations with gpt- 3. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 15972-15979, 2023.","title":"Solving math word problems concerning systems of equations with gpt-3","authors":"Mingyu Zong, Bhaskar Krishnamachari","year":"2023","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}