Never Train from Scratch: Fair Comparison of Long-Sequence Models Requires Data-Driven Priors (Extended Abstract)

1 Introduction

Self-supervised pretraining is widespread across machine learning, with pretrained model finetuning now standard practice for downstream tasks [Touvron et al., 2023; Baevski et al., 2020; Reed et al., 2022; Raffel et al., 2020]. However, when developing architectures with better inductive biases for specific skills, training from scratch with random initialization remains common [Tay et al., 2021; Del´etang et al., 2023; Velickovic et al., 2022; Dwivedi et al., 2022]. This practice stems from computational constraints and attempts to enable fair comparisons without requiring a standard pretraining corpus.

Long Range Arena (LRA) exemplifies this pattern, with Transformers showing poor performance on these stress tests [Tay et al., 2021], spurring development of new architectures biased toward capturing long-range dependencies [Gu et al., 2022; Gupta et al., 2022a; Li et al., 2023; Ma et al., 2023]. These results contradict the success of pretrained Transformers on long-range tasks like text summarization and protein folding [Touvron et al., 2023; Jumper et al., 2021]. Despite advances in long sequence modeling, the reasons for this discrepancy remain unexplored, while competitive methods rely on architectural modifications [Ma et al., 2023; Zuo et al., 2022].

∗ Ido Amos, Jonathan Berant, and Ankit Gupta. Never train from scratch: Fair comparison of long-sequence models requires datadriven priors. In The Twelfth International Conference on Learning Representations, 2024.

We demonstrate this discrepancy stems from inadequate training and evaluation practices and propose a simple solution. While avoiding large-corpus pretraining is understandable, random initialization overlooks how pretraining objectives create beneficial inductive biases. Recent work shows pretraining solely on downstream training data (self pretraining or SPT) often yields gains comparable to largecorpus pretraining [El-Nouby et al., 2021; He et al., 2022; Krishna et al., 2023]. This suggests SPT offers a more realistic performance estimate while serving as a data-driven initialization method that enables fair comparisons using only the task data.

Our empirical evidence shows that priors learned through SPT with denoising objectives effectively capture long-range dependencies across architectures, often eliminating the need for complex hand-crafted modeling biases [Gu et al., 2022;

Ma et al., 2023; Li et al., 2023; Orvieto et al., 2023]. On Long Range Arena (LRA), SPT improves vanilla Transformer performance by more than 30%, allowing them to match stateof-the-art results without architectural changes (Figure 1), contradicting prior works showing significantly lower Transformer performance.

SPT also benefits State Space models (SSMs), which use modified linear RNNs in place of attention. S4 achieves impressive performance on long sequence tasks through specialized parameterization and initialization [Gu et al., 2022]. With SPT, S4 shows gains in 5 of 6 LRA tasks and solves the challenging PathX-256 task with 20% improved accuracy (Figure 1). Our analysis reveals that data-driven priors from SPT make many hand-crafted modeling biases redundant [Gupta et al., 2022b], enabling competitive performance with simple diagonal linear RNNs without manual modifications [Orvieto et al., 2023].

Our findings demonstrate that beneficial priors for capturing distant dependencies can be learned directly from task data through standard denoising objectives. The benefits of SPT become more pronounced with scarcer data. For SSMs, our analysis of convolution kernels reveals that, depending on the modality, rapidly decaying kernels sometimes outperform the slowly decaying ones used in native S4, highlighting the advantages of learning data-specific priors [Gu et al., 2020].

Our main contributions are:

• (i) Demonstrating that reported performance on long-range benchmarks is severely underestimated, and proposing an inexpensive data-driven approach for accurate evaluation, without requiring any additional data.
• (ii) Reporting significant empirical gains across architectures on LRA, improving the best reported accuracy on PathX-256 by 20 absolute points (67 → 87).
• (iii) Showing that manually-designed biases become redundant with pretraining, enabling simpler models to match sophisticated architectures. We achieve competitive performance on LRA with Transformers and diagonal linear RNNs.

The substantial improvements from SPT across LRA's multi-modal tasks suggest including pretraining when evaluating models for multidimensional inputs [Nguyen et al., 2022], algorithmic reasoning [Diao and Loynd, 2023], or graphs [Shirzad et al., 2023].

Our code & data are available at https://github.com/ IdoAmos/not-from-scratch.

2 Experimental Setup

We evaluate Transformers and SSMs on Long Range Arena (LRA), a benchmark for testing long-range dependency modeling [Tay et al., 2021]. It contains 6 sequence classification tasks:

1. ListOps: Nested lists with operations (MAX, MEAN, etc.) applied to multiple token arguments [Nangia and Bowman, 2018]. 10-way classification with 2K sequence length. INPUT:[MAX 4 3[MIN 2 3]1 0[MEDIAN 1 5 8 9 2]] OUTPUT: 5
2. Text: Character-level IMDb reviews [Maas et al., 2011]. Binary sentiment classification with up to 2048 sequence length.
3. Retrieval: Character-level AAN dataset [Radev et al., 2009] for document similarity. Binary classification with 8K total tokens.
4. Image: Flattened grayscale CIFAR10 images as 1D sequences. 10-way classification with 1024 sequence length.
5. Pathfinder, PathX: Synthetic visual path-tracing tasks [Linsley et al., 2018; Kim et al., 2020]. Binary classification with lengths 1024 and 16384.

We also test on PathX-256 (sequence length 2562 = 65536) and additional datasets described in Section 3.4.

Self Pretraining (SPT) We perform SPT using only the downstream task training set, with causal/autoregressive objectives for unidirectional models and masked modeling for bidirectional models. Masking ratios: 50% for visual tasks [He et al., 2022], 15% for language tasks [Liu et al., 2019], and 10% for ListOps. We use FLASH attention [Dao et al., 2022] for Transformers, with blocked attention (block size 4096) for sequences over 16K. Our code builds on the official S4 repository. For details on hyperparameters and compute, refer to the original paper [Amos et al., 2024].

3 Results

Section 3.1 shows SPT results on LRA with official configurations. Section 3.2 compares SPT for Transformers and S4. Section 3.3 evaluates SSM design choices with SPT. Section 3.4 presents additional experiments on different modalities.

3.1 Underestimation of Long-range Abilities of Transformers

We investigate the reliability of historically-reported LRA model performances with pretraining. We repeat the Transformer experiments from [Tay et al., 2021], first pretraining models on task data before finetuning, strictly following their configurations. We test both next token prediction and masked token prediction objectives with masking ratios as described in Section 2.

As Table 1 shows, both pretraining objectives dramatically improve Transformer performance compared to random initialization, with average test accuracy increasing by roughly 30%. Causal and masked pretraining yield similar results even for visual tasks and ListOps (where arguments are randomly sampled, making token prediction from context difficult). Since we made no architectural changes and used no additional data, these improvements come entirely from priors learned during SPT, demonstrating its importance for reliable evaluation.

3.2 Comparing S4 and Transformers

In the above set-up we strictly adhered to the model sizes used by [Tay et al., 2021] and consequently the absolute performances are still low compared to the current state-of-the-art on LRA. In this section, we scale the model sizes and evaluate the utility of SPT for the best performing architectures including S4 [Gu et al., 2022]. For Transformers, we replace the positional embeddings with the more commonly used rotary embeddings [Su et al., 2021] and only train bidirectional models in line with prior works reporting high performance.

As summarized in Table 2, SPT leads to dramatic performance gains for Transformers with performance gains ranging from 8-15% across tasks, even surpassing the average performance of a well-tuned S4 (86.2 vs 86.1). SPT Transformers surpass the performance of both trained from scratch and SPT versions of S4 on 3 out of 6 tasks. The results in Table 2 defy current understanding, with prior works citing the subpar LRA performance of Transformers as a prime motivating factor for new methods. Yet we show that, while architectural developments indeed lead to remarkable performance gains, most of the priors essential to high performance can already be learned from data directly.

In case of S4, while SPT leads to modest gains on most tasks, a substantial gain of 20% is observed on the challenging PathX-256 task with input length of 65K, significantly improving over the best reported performance of 63.1% by [Dao et al., 2022] who, in addition, used extra data from the Pathfinder-64 task.

The additionally reported models, SPADE and MEGA, are Transformer variants that augment the model with a single or several state space layers. SPADE combines the outputs of a frozen S4 layer and local attention in the first block, while MEGA incorporates a learned exponential moving average, an instance of diagonal SSMs, into gated attention blocks. To the best of our knowledge, we are the first to show that purely attention-based methods, without any architectural modifications, can achieve competitive results on LRA. While incorporating SSMs can be important in terms of scalability to longer sequences due to their log-linear complexity with respect to input length, we show that in terms of model performance, pretraining leads to biases that are as effective as manual designs.

An important aspect of SPT is the use of additional compute compared to the trained from scratch baseline and it is natural to investigate if similar gains can be obtained by training from scratch for longer. For all our trained from scratch baselines, we ensured that the validation performance had converged and did not improve for several consecutive epochs. We examine the aspect of the computational overhead of SPT in detail in the original paper [Amos et al., 2024], where we show that SPT leads to significant gains, even in the setting where the same amount of compute is used for SPT models and the ones that are trained from scratch.

3.3 The Role of Explicit Priors

Having established that SPT enables more reliable architectural evaluation and improves SSM performance, we now examine S4's design complexity. S4's theoretically-motivated design enables long-range signal propagation, explaining its slight advantage over SPT Transformers. While various S4 simplifications have been proposed, we show that SPT enables even simpler diagonal linear RNNs to match S4 performance.

For SSMs (see [Gu et al., 2022] for details), given input scalar sequence u, a linear recurrence generates hidden state

⃗xn at timestep n and a scalar output sequence y as:

This can be computed by convolving u with kernel defined by Kk = C T AkB. S4 uses a specialized parameterization with transformations:

with learnable parameters Λ,P,Q,B,C,∆ where Λ ∈ Diag(CN×N ),P,Q ∈ CN×1 . S4 uses principled initialization for slow kernel decay to capture long-range dependencies.

[Gupta et al., 2022b] proposed a simplified Diagonal Linear RNN (DLR):

where 1 is the all-ones vector. DLR is computationally simpler than S4 yet reportedly matches state-of-the-art SSMs on token-level tasks. We investigate when S4's complex design can be replaced by DLR, testing both on the hardest LRA tasks (ListOps, Text, Image, PathX) with two initialization schemes: random and "structured" (designed for long-range dependencies).

Results in Figure 2 show that when trained from scratch, DLR lags behind S4 (77 vs 83), confirming that S4's specialized initialization and parameterization are critical. However, with SPT, DLR outperforms from-scratch S4 (83.4 vs 82.8) and approaches SPT S4 (83.4 vs 84.5) suggesting that the data-driven priors from pretraining can largely replace manual biases.

These findings have broader implications. First, this is the first demonstration of vanilla diagonal linear RNNs achieving competitive LRA performance without normalization or specialized initialization [Orvieto et al., 2023]. Second, many global convolution designs follow similar principles to SSMs, like generating smooth decaying kernels [Li et al., 2023] or applying deterministic transformations [Fu et al., 2023]. Our results suggest these explicit design steps become less critical with self-pretraining.

3.4 Additional Experiments

We tested SPT on three additional datasets across different modalities:

• Speech Commands (SC): Raw speech waveforms (16K length) for 35-way classification [Warden, 2018], testing both causal and bidirectional models.
• sCIFAR: Sequential CIFAR-10 with RGB channels as features (richer than LRA's grayscale Image task).
• BIDMC: Three regression tasks predicting health metrics (RR, HR, SpO2) from 4K-length physiological signals.

Results in Table 3 further support our claims. On SC and sCIFAR, SPT significantly improves Transformer performance while modestly enhancing S4, substantially narrowing their performance gap. For SC, Transformers gain 5% with SPT, and the gap between causal and bidirectional S4 variants diminishes, similar to our LRA observations. On sCIFAR, Transformers improve by 11% with SPT, nearly matching S4 (90.3 vs 91.7). BIDMC results show minimal gains as both models already achieve near-perfect performance. These findings suggest that performance underestimation likely affects other domains where training from scratch remains standard practice [Del´etang et al., 2023; Velickovic et al., 2022; Dwivedi et al., 2022].