{"type":"ScholarlyArticle","title":"Never Train from Scratch: Fair Comparison of Long-Sequence Models Requires Data-Driven Priors (Extended Abstract)","authors":[{"name":"Ido Amos","slug":"ido-amos","given":"Ido","family":"Amos","affiliation":"Tel Aviv University"},{"name":"Jonathan Berant","slug":"jonathan-berant","given":"Jonathan","family":"Berant","affiliation":"Tel Aviv University"},{"name":"Ankit Gupta","slug":"ankit-gupta","given":"Ankit","family":"Gupta","affiliation":"IBM Research"}],"abstract":"This paper is an extended abstract of our ICLR 2024 Outstanding Paper Award work. Modeling long-range dependencies across sequences is a longstanding goal in machine learning. While state space models reportedly outperform Transformers on benchmarks like Long Range Arena, we show that random initialization significantly overestimates architectural differences. Pretraining with standard denoising objectives on downstream task data leads to dramatic gains across architectures and minimal performance gaps between Transformers and state space models (SSMs). We demonstrate that properly pretrained vanilla Transformers match S4 performance on Long Range Arena and improve SSM results on PathX-256 by 20 absolute points. Our analysis shows previously-proposed structured parameterizations for SSMs become largely redundant with pretraining. When evaluating architectures on supervised tasks, incorporating data-driven priors via pretraining is essential for reliable performance estimation.","doi":"10.24963/ijcai.2025/1203","url":"https://ijcai.org/proceedings/2025/1203","pdf":"https://ijcai.org/proceedings/2025/1203.pdf","markdown":"https://ijcai.org/proceedings/2025/1203.md","bibtex":"https://ijcai.org/proceedings/2025/bibtex/1203","volume":{"slug":"2025","title":"Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2025","url":"https://ijcai.org/proceedings/2025/","year":2025,"location":"Montreal, Canada","dates":"16-22 August 2025","editors":"James Kwok","isbn":"978-1-956792-06-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2025"},"section":"Sister Conferences Best Papers","track":"Sister Conferences Best Papers","number":1203,"pages":{"first":10846,"last":10851},"keywords":["Sister Conferences Best Papers: Machine Learning","Sister Conferences Best Papers: Knowledge Representation and Reasoning","Sister Conferences Best Papers: Computer Vision","Sister Conferences Best Papers: Natural Language Processing"],"videos":[],"references":[{"position":1,"raw":"[Amos et al., 2024] Ido Amos, Jonathan Berant, and Ankit Gupta. Never train from scratch: Fair comparison of long- sequence models requires data-driven priors. In Proc. of ICLR, 2024.","title":"Never train from scratch: Fair comparison of long-sequence models requires data-driven priors","authors":"Amos","year":"2024","venue":"Proc. of ICLR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Baevski et al., 2020] Alexei Baevski, Yuhao Zhou, Abdel- rahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech represen- tations. In Proc. of NeurIPS, 2020.","title":"wav2vec 2.0: A framework for self-supervised learning of speech representations","authors":"Baevski","year":"2020","venue":"Proc. of NeurIPS","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Dao et al., 2022] Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. In Proc. of NeurIPS, 2022.","title":"Flashattention: Fast and memory-efficient exact attention with io-awareness","authors":"Tri Dao, Daniel Y Fu, Stefano Ermon, Atri Rudra, Christopher Ré","year":"2022","venue":"Proc. of NeurIPS","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Delétang et al., 2023] Grégoire Delétang, Anian Ruoss, Jordi Grau-Moya, Tim Genewein, Li Kevin Wenliang, El- liot Catt, Chris Cundy, Marcus Hutter, Shane Legg, Joel Veness, and Pedro A. Ortega. Neural networks and the chomsky hierarchy. In Proc. of ICLR, 2023.","title":"Neural networks and the chomsky hierarchy","authors":"Grégoire Delétang, Anian Ruoss, Jordi Grau-Moya, Tim Genewein, Kevin Li, Elliot Wenliang, Chris Catt, Marcus Cundy, Shane Hutter, Joel Legg, Pedro A Veness, Ortega","year":"2023","venue":"Proc. of ICLR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Diao and Loynd, 2023] Cameron Diao and Ricky Loynd. Relational attention: Generalizing transformers for graph- structured tasks. In Proc. of ICLR, 2023.","title":"Relational attention: Generalizing transformers for graphstructured tasks","authors":"Cameron Diao, Ricky Loynd","year":"2023","venue":"Proc. of ICLR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dwivedi et al., 2022] Vijay Prakash Dwivedi, Ladislav Rampásek, Michael Galkin, Ali Parviz, Guy Wolf, Anh Tuan Luu, and Dominique Beaini. Long range graph benchmark. In Proc. of NeurIPS, 2022.","title":"Long range graph benchmark","authors":"Vijay Prakash Dwivedi, Ladislav Rampásek, Michael Galkin, Ali Parviz, Guy Wolf, Anh Tuan Luu, Dominique Beaini","year":"2022","venue":"Proc. of NeurIPS","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[El-Nouby et al., 2021] Alaaeldin El-Nouby, Gautier Izac- ard, Hugo Touvron, Ivan Laptev, Hervé Jegou, and Edouard Grave. Are large-scale datasets necessary for self-supervised pre-training? ArXiv preprint, 2021.","title":"Are large-scale datasets necessary for self-supervised pre-training","authors":"El-Nouby","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Fu et al., 2023] Daniel Y. Fu, Elliot L. Epstein, Eric Nguyen, Armin W. Thomas, Michael Zhang, Tri Dao, Atri Rudra, and Christopher Ré. Simple hardware-efficient long convo- lutions for sequence modeling. In Proc. of ICML, Proceed- ings of Machine Learning Research, 2023.","title":"Simple hardware-efficient long convolutions for sequence modeling","authors":"Y Daniel, Elliot L Fu, Eric Epstein, Armin W Nguyen, Michael Thomas, Tri Zhang, Atri Dao, Christopher Rudra, Ré","year":"2023","venue":"Proc. of ICML, Proceedings of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gu et al., 2020] Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré. Hippo: Recurrent memory with optimal polynomial projections. In Proc. of NeurIPS, 2020.","title":"Hippo: Recurrent memory with optimal polynomial projections","authors":"Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, Christopher Ré","year":"2020","venue":"Proc. of NeurIPS","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gu et al., 2022] Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. In Proc. of ICLR, 2022.","title":"Efficiently modeling long sequences with structured state spaces","authors":"Albert Gu, Karan Goel, Christopher Ré","year":"2022","venue":"Proc. of ICLR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Gupta et al., 2022a] Ankit Gupta, Albert Gu, and Jonathan Berant. Diagonal state spaces are as effective as structured state spaces. In Proc. of NeurIPS, 2022.","title":"Diagonal state spaces are as effective as structured state spaces","authors":"Gupta","year":"2022","venue":"Proc. of NeurIPS","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Gupta et al., 2022b] Ankit Gupta, Harsh Mehta, and Jonathan Berant. Simplifying and understanding state space models with diagonal linear rnns. ArXiv preprint, 2022.","title":"Simplifying and understanding state space models with diagonal linear rnns","authors":"Gupta","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[He et al., 2022] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross B. Girshick. Masked autoencoders are scalable vision learners. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, 2022.","title":"Masked autoencoders are scalable vision learners","authors":"He","year":"2022","venue":"IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Jumper et al., 2021] John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, Alex Bridgland, Clemens Meyer, Si- mon A. A. Kohl, Andrew J. Ballard, Andrew Cowie, Bernardino Romera-Paredes, Stanislav Nikolov, Rishub Jain, Jonas Adler, Trevor Back, Stig Petersen, David Reiman, Ellen Clancy, Michal Zielinski, Martin Steinegger, Michalina Pacholska, Tamas Berghammer, Sebastian Bo- denstein, David Silver, Oriol Vinyals, Andrew W. Senior, Koray Kavukcuoglu, Pushmeet Kohli, and Demis Hassabis. Highly accurate protein structure prediction with alphafold. Nature, (7873), 2021.","title":"Highly accurate protein structure prediction with alphafold","authors":"John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, Alex Bridgland, Clemens Meyer, A A Simon, Andrew J Kohl, Andrew Ballard, Bernardino Cowie, Stanislav Romera-Paredes, Rishub Nikolov, Jonas Jain, Trevor Adler, Stig Back, David Petersen, Ellen Reiman, Michal Clancy, Martin Zielinski, Michalina Steinegger, Tamas Pacholska, Sebastian Berghammer, David Bodenstein, Oriol Silver, Andrew W Vinyals, Koray Senior, Pushmeet Kavukcuoglu, Demis Kohli, Hassabis","year":"7873","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kim et al., 2020] Junkyung Kim, Drew Linsley, Kalpit Thakkar, and Thomas Serre. Disentangling neural mecha- nisms for perceptual grouping. In Proc. of ICLR, 2020.","title":"Disentangling neural mechanisms for perceptual grouping","authors":"Kim","year":"2020","venue":"Proc. of ICLR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Krishna et al., 2023] Kundan Krishna, Saurabh Garg, Jeffrey Bigham, and Zachary Lipton. Downstream datasets make surprisingly good pretraining corpora. In Proc. of ACL, 2023.","title":"Downstream datasets make surprisingly good pretraining corpora","authors":"Kundan Krishna, Saurabh Garg, Jeffrey Bigham, Zachary Lipton","year":"2023","venue":"Proc. of ACL","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Li et al., 2023] Yuhong Li, Tianle Cai, Yi Zhang, Deming Chen, and Debadeepta Dey. What makes convolutional models great on long sequence modeling? In Proc. of ICLR, 2023.","title":"What makes convolutional models great on long sequence modeling?","authors":"Yuhong Li, Tianle Cai, Yi Zhang, Deming Chen, Debadeepta Dey","year":"2023","venue":"Proc. of ICLR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Linsley et al., 2018] Drew Linsley, Junkyung Kim, Vijay Veerabadran, Charles Windolf, and Thomas Serre. Learn- ing long-range spatial dependencies with horizontal gated recurrent units. In Proc. of NeurIPS, 2018.","title":"Learning long-range spatial dependencies with horizontal gated recurrent units","authors":"Linsley","year":"2018","venue":"Proc. of NeurIPS","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Liu et al., 2019] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. ArXiv preprint, 2019.","title":"Roberta: A robustly optimized bert pretraining approach","authors":"Liu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Ma et al., 2023] Xuezhe Ma, Chunting Zhou, Xiang Kong, Junxian He, Liangke Gui, Graham Neubig, Jonathan May, and Luke Zettlemoyer. Mega: Moving average equipped gated attention. In Proc. of ICLR, 2023.","title":"Mega: Moving average equipped gated attention","authors":"Ma","year":"2023","venue":"Proc. of ICLR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Maas et al., 2011] Andrew L. Maas, Raymond E. Daly, Pe-[Raffel et al., 2020] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 2020.","title":"Exploring the limits of transfer learning with a unified text-to-text transformer","authors":"Andrew L Maas, Raymond E Daly, Pe-[ Raffel","year":"2020","venue":"J. Mach. Learn. Res","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Reed et al., 2022] Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley D. Edwards, Nicolas Manfred Otto Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas. A generalist agent. Trans. Mach. Learn. Res., 2022.","title":"A generalist agent","authors":"Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley D Edwards, Nicolas Manfred, Otto Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, Nando De Freitas","year":"2022","venue":"Trans. Mach. Learn. Res","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Shirzad et al., 2023] Hamed Shirzad, Ameya Velingker, Bal- aji Venkatachalam, Danica J. Sutherland, and Ali Kemal Sinop. Exphormer: Sparse transformers for graphs. In Proc. of ICML, Proceedings of Machine Learning Research, 2023.","title":"Exphormer: Sparse transformers for graphs","authors":"Hamed Shirzad, Ameya Velingker, Balaji Venkatachalam, Danica J Sutherland, Ali Kemal, Sinop","year":"2023","venue":"Proc. of ICML, Proceedings of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Su et al., 2021] Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. ArXiv preprint, 2021.","title":"Roformer: Enhanced transformer with rotary position embedding","authors":"Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, Yunfeng Liu","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Tay et al., 2021] Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena : A benchmark for efficient transformers. In Proc. of ICLR, 2021.","title":"Long range arena : A benchmark for efficient transformers","authors":"Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, Donald Metzler","year":"2021","venue":"Proc. of ICLR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Touvron et al., 2023] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Can- ton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schel- ten, Ruan Silva, Eric Michael Smith, Ranjan Subrama- nian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. ArXiv preprint, 2023.","title":"Llama 2: Open foundation and fine-tuned chat models","authors":"Touvron","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Velickovic et al., 2022] Petar Velickovic, Adrià Puig- domènech Badia, David Budden, Razvan Pascanu, Andrea Banino, Misha Dashevskiy, Raia Hadsell, and Charles Blundell. The CLRS algorithmic reasoning benchmark. In Proc. of ICML, Proceedings of Machine Learning Research, 2022.","title":"The CLRS algorithmic reasoning benchmark","authors":"Petar Velickovic, Adrià Puigdomènech Badia, David Budden, Razvan Pascanu, Andrea Banino, Misha Dashevskiy, Raia Hadsell, Charles Blundell","year":"2022","venue":"Proc. of ICML, Proceedings of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Warden, 2018] Pete Warden. Speech commands: A dataset for limited-vocabulary speech recognition. ArXiv preprint, 2018.","title":"Pete Warden. Speech commands: A dataset for limited-vocabulary speech recognition","authors":"Warden","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zuo et al., 2022] Simiao Zuo, Xiaodong Liu, Jian Jiao, De- nis Charles, Eren Manavoglu, Tuo Zhao, and Jianfeng Gao. Efficient long sequence modeling via state space augmented transformer. ArXiv preprint, 2022.","title":"Efficient long sequence modeling via state space augmented transformer","authors":"Zuo","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}