HT-Transformer: Event Sequences Classification by Accumulating Prefix Information with History Tokens

1 Introduction

Many real-world problems involve predicting future events from historical observations. In generative tasks, the goal is to forecast events similar to those previously observed [Xue et al., 2024]. However, many practical applications require anticipating events that do not explicitly appear in the training history. Examples include high-impact tasks in fintech, e-commerce, and medical analytics, such as predicting loan default, customer churn, and disease onset. These scenarios are typically addressed using classical machine learning models, such as logistic regression or gradient boosting, applied to handcrafted features or unsupervised model-based embeddings derived from historical data [Osin et al., 2025; Synerise, 2025].

Deep learning has achieved significant success in modeling sequential structures, including event sequences, temporal point processes, and time series. A prominent trend is the adoption of pretrained Transformer architectures due to their ability to capture long-range dependencies and complex temporal patterns [Padhi et al., 2021; Zuo et al., 2020]. Unlike recurrent neural networks, however, Transformers lack a canonical mechanism for extracting a fixed-size embedding from a sequence, as information is distributed across all token activations. This issue is commonly mitigated through auxiliary objectives during pretraining, such as contrastive learning [BehnamGhader et al., 2024], sentence order prediction [Lan et al., 2020], or next-sentence prediction [Devlin et al., 2019]. However, each of these approaches introduces limitations. For instance, it is well documented that contrastive pretraining may overemphasize "easy features", compromising downstream quality [Robinson et al., 2021]. Consequently, the problem of learning robust and informative sequence embeddings, including methods based on the next-token prediction (NTP) objective [Yenduri et al., 2024], remains an open research question.

Figure 1: History tokens accumulate prefix information during pretraining via next-token-prediction. The embedding of the History Token is later used in downstream tasks.

In this work, we propose a novel approach to pretraining Transformer-based embeddings without relying on auxiliary tasks. Our method draws inspiration from recurrent architectures and leverages sparse attention masks to guide the accumulation of historical information [Bulatov et al., 2022]. Specifically, we introduce history tokens that gather and summarize contextual information during training via a standard next-token prediction objective, as illustrated in Figure 1. We empirically evaluate the resulting embeddings across multiple domains, including finance, e-commerce, and healthcare, and show that they offer strong predictive performance, especially for tasks focused on forecasting future events rather than global sequence classification.

The contributions of this paper are as follows:

1. We propose a novel HT-Transformer architecture that employs history tokens to accumulate past information during pretraining using only the next-token prediction objective.
2. We develop advanced strategies for selecting history token positions and applying attention masking to improve downstream performance.
3. We introduce strong baselines for embedding extraction by utilizing adapted variants of the Recurrent Memory Transformer and Longformer.
4. We demonstrate that the proposed HT-Transformer is especially effective for forecasting future events, consistently outperforming widely used transformer models and achieving strong results across multiple benchmarks in finance, e-commerce, and healthcare.

2 Preliminaries on Event Sequences

Consider sequences of discrete events S = {si}Ni=1 , where each event si is represented by a collection of fields, including a timestamp ti, optional numerical attributes, and categorical variables. Each sequence typically corresponds to a single entity, such as a user or client, and the events are ordered chronologically by their timestamps: t1 ≤ t2 ≤ · · · ≤ tN. An illustration of event sequences is provided in Figure 2.

Data preprocessing. Before feeding data into a deep model, each event in the sequence must be transformed into an embedding in a latent space. In a typical pipeline, each data field is encoded independently, and the resulting embeddings are concatenated to form a single event representation [Gorishniy et al., 2021]. Categorical features are transformed by assigning a trainable embedding vector to each possible value. Numerical features are incorporated directly into the event embedding without additional preprocessing.

We apply time-based positional encoding when using Transformer models, following the approach proposed in prior work [Yang et al., 2022]. Specifically, for each timestamp t, we compute a positional embedding PEi(t) of dimension d as:

where m and M are constants determined from the distribution of timestamp values. The precise hyperparameters are listed in Appendix D in the supplementary materials.

Pretraining. We consider sequence-level classification tasks, where each sequence S is associated with a single target label l. Because labels are assigned at the sequence level rather than the event level, the number of labeled examples is often much smaller than the total number of events. This imbalance motivates the development of unsupervised pretraining algorithms that can leverage the abundance of unlabeled sequential data to improve downstream performance [Osin et al., 2025].

Unsupervised pretraining on event sequences is typically based on either generative or contrastive learning objectives.

In the generative approach, the model is trained to predict the next event si+1 given the historical context s1, . . . , si, thereby encouraging it to capture temporal dependencies and sequential structure. A typical NTP objective is formulated as a weighted sum of individual losses over each data field [Shchur et al., 2020; Padhi et al., 2021; McDermott et al., 2023]:

where ˆ s = (ˆ s1 , . . . , ˆ sF ) denotes the predicted event with F fields, s = (s1 , . . . , sF ) is the ground-truth event, and Li is a field-specific loss determined by the type of the i-th field. The individual losses are combined using weights αi.

Timestamps can be predicted using standard regression losses such as Mean Absolute Error (MAE) or Mean Squared Error (MSE), or through more expressive temporal point process models based on intensity functions [Rizoiu et al., 2017; Zuo et al., 2020]. In the current work, we use MAE to predict inter-event times:

where ∆ˆ t is the predicted inter-event time and ∆t is the ground truth. We also apply MAE to other numerical fields, such as transaction amount. For categorical fields, we use cross-entropy:

where ˆ p(·) is the predicted label distribution and l is the ground-truth category. The exact combination of MAE and cross-entropy terms depends on the dataset. The total number of fields F for each dataset is reported in Table 1.

An alternative to generative modeling is contrastive learning [Babaev et al., 2022], which aims to learn sequence representations by maximizing agreement between different augmented views of the same sequence and pushing apart views from different sequences. Typically, each sequence is divided into multiple, possibly overlapping, subsequences Rk ⊂ S

Positional encoding

Figure 2: Event sequences preprocessing.

for k = 1, . . . ,K. Let ID(R) denote the index of the original sequence from which a chunk R was derived. Then the contrastive loss [Chopra et al., 2005] is defined as:

where f(R) ∈ Rd is the embedding of a subsequence R produced by the model. Following prior works [Babaev et al., 2022; Sakhno et al., 2025], we use ϵ = 0.5 and K = 5 subsequences per sequence.

Both generative and contrastive paradigms have been successfully adapted to neural architectures such as recurrent neural networks (RNNs) and Transformers. However, while effective for specific tasks, these approaches exhibit notable limitations, especially when the goal is to anticipate future events rather than to summarize past behavior. Overcoming these limitations is a key motivation behind the approach proposed in this work.

Downstream tasks. Embeddings extracted from pretrained models are widely used for downstream classification and regression tasks, either via a fully connected prediction head [Synerise, 2025] or in combination with gradient boosting methods [Babaev et al., 2022]. Following prior work on event sequences [Babaev et al., 2022; Osin et al., 2025], we apply gradient boosting on top of pretrained embeddings for downstream classification. We discuss the motivation for this choice and compare gradient boosting with an MLP head in Appendix E.

3 ProposedMethod

The core idea of the proposed method is to introduce special history tokens into Transformer models. These tokens are designed to accumulate information from preceding tokens in the sequence. A carefully constructed attention mask ensures that these tokens act as an information bottleneck, similar in function to the hidden states in RNNs. In the following, we describe the training procedure and the application of history tokens to downstream classification tasks.

3.1 Unsupervised Pretraining with History Tokens

The proposed approach is compatible with any Transformer architecture that employs a causal attention mask (Fig. 3a), where each token attends only to preceding tokens. Transformers for event sequences typically have three primary components: an event embedder, a backbone, and a prediction head. History tokens are inserted into the backbone input after event embeddings are computed, as shown in Figure 1a. Each history token is assigned the timestamp of a preceding event. These timestamps are then used for positional encoding.

To enable history tokens to serve as memory units, we modify the backbone's attention mask. Each history token can attend to all preceding event tokens (except other history tokens), thereby accumulating prefix information. In contrast, event tokens can attend only to history tokens and to events between the current position and the most recent history token. This attention pattern is illustrated in Figure 3d. When multiple history tokens are present, we introduce two attention strategies. In the Last strategy (Figure 3d), each event token is restricted to attend only to the most recent preceding history token. In the Random strategy, illustrated in Figure 3e, each event token randomly selects one of the prior history tokens during attention computation. As demonstrated in our experiments, the Random strategy consistently yields better performance across various tasks.

The proposed method provides considerable flexibility in selecting both the number and the placement of history tokens. For a given sequence of length L, the number of history tokens is computed as max(1, ⌊fL⌉), where ⌊·⌉ is a rounding operator and f is a frequency hyperparameter.

We also implement two strategies for inserting history tokens into the sequence. The Uniform strategy inserts history tokens at uniformly sampled positions. However, this approach can lead to a discrepancy between training and inference, as history tokens are typically positioned near the end of the sequence during evaluation. To address this issue, we introduce the Bias-End strategy, which places history tokens closer to the end of the sequence. Specifically, it samples positions uniformly within the range [µ/2,L], where µ is the mean sequence length in the batch, and L is the maximum sequence length. Our experiments demonstrate that the Bias-End strategy improves downstream performance in the considered tasks.

Finally, to align pretraining and inference, we introduce one more modification. At inference time, a single history token is appended to the end of the sequence and attends to the entire prefix through the causal masking scheme. The hidden representation of this token is then used as the sequence

(b) Rec. Memory.

(c) Longformer.

embedding for downstream classification. This creates a bottleneck analogous to the pretraining setup, although inference uses only one history token instead of multiple intermediate tokens. To reduce the mismatch between training and inference, we apply history tokens only to a subset of pretraining batches using application probability p = 0.5.

3.2 Classification

During embedding extraction, a single history token is appended to the end of the input sequence, and the average of its hidden activations across Transformer layers is used as the sequence-level embedding (Figure 1b). This embedding then serves as an input feature for a downstream gradient boosting classifier.

4 RelatedWork

Transformer models have been successfully applied to sequence classification tasks, especially in natural language processing (NLP) [Vaswani et al., 2017]. One of the main challenges in this setting is the limited availability of labeled data, which has driven the development of effective unsupervised pretraining strategies [Muennighoff et al., 2023]. A notable early approach is BERT [Devlin et al., 2019], which introduced a masked language modeling (MLM) objective alongside a next sentence prediction (NSP) task to enable powerful representation learning. Fine-tuned BERT models have been widely adopted for various classification tasks such as natural language understanding (NLU) [Wang et al., 2019].

A central issue in applying Transformers to NLU is extracting a compact, semantically meaningful representation of an entire sequence. In BERT, this was addressed by introducing a special classification token trained via the NSP objective. However, subsequent work such as RoBERTa [Liu et al., 2019] challenged the necessity of the NSP task, demonstrating that it could be omitted without degrading performance.

Beyond BERT-style objectives, other works have explored contrastive pretraining techniques, such as LLM2Vec [BehnamGhader et al., 2024], which aim to bring semantically similar sequences closer in embedding space. Although contrastive learning can perform well across a range of tasks, it has notable limitations. In particular, models can rely on "easy" features, such as surface-level similarities, to distinguish positive pairs, bypassing the need for deeper semantic understanding. Furthermore, contrastive pretraining tends to bias models toward capturing global sequence properties at the expense of local or up-to-date information. This bias poses a particular challenge in event sequence modeling, where the most recent context is often essential for accurate prediction, in contrast to many natural language processing tasks emphasizing global semantics.

Alternative methods for sequence embedding extraction include averaging token activations across specific layers or using the final token's activation [Stankeviˇcius and Lukoˇseviˇcius, 2024]. However, these approaches generally underperform specialized embedding pretraining techniques, particularly for tasks that require nuanced or fine-grained representations.

In contrast, RNNs naturally summarize sequences through the hidden state at the final timestep, which compactly encodes the information needed for future prediction. This property has inspired the integration of recurrent principles into Transformer architectures, particularly for modeling long sequences. For instance, Recurrent-Memory Transformers (RMT) recursively apply a Transformer model to chunks of the input sequence, using special tokens to pass information between successive segments [Bulatov et al., 2022]. The corresponding attention mask is shown in Figure 3b. Similar ideas are adopted in architectures such as Longformer [Beltagy et al., 2020], where global tokens aggregate and propagate information across extended contexts (Figure 3c). More recently, recurrent-style Transformers have been combined with contrastive learning objectives to attain strong results on natural language understanding (NLU) tasks, while retaining the generative abilities of causal models [Zhang et al., 2025].

Our work extends recurrent-style Transformer architectures toward representation learning for event-sequence classification. While prior approaches such as RMT and Longformer primarily target efficient long-context modeling, HTTransformer uses history tokens as predictive bottlenecks during standard next-token prediction pretraining. Unlike RMT, the HT-Transformer processes a sequence in a single pass while retaining activations from all layers, rather than only the last, for subsequent token processing. In contrast to Longformer, HT-Transformer employs special tokens instead of solely modifying the attention matrix. Moreover, it restricts attention to future tokens, forcing history tokens to accumulate local information.

A detailed comparison of the proposed history tokens with other special tokens, such as [CLS], [SEP], [SOS], and [EOS], is provided in Appendix B.

Longformer [Beltagy et al., 2020], were initially proposed to address the limited scalability of standard Transformers, whose complexity grows quadratically with input length. These models were not explicitly designed for embedding extraction and therefore require additional modifications when applied to classification tasks.

5 Experimental Results

Setup. We conduct experiments on datasets spanning multiple domains. The Churn 1 , AgePred 2 , and Alfabattle 3 datasets represent a range of downstream tasks in the financial domain. The Taobao dataset 4 represents user interactions in e-commerce, and MIMIC-III [Johnson et al., 2016] is a widely used collection of medical records. Summary statistics for these datasets are provided in Table 1.

We evaluate four primary baseline approaches: supervised learning, NTP pretraining [Radford et al., 2018], BERT [Devlin et al., 2019], and contrastive learning (CoLES) [Babaev et al., 2022]. Each method is applied to two backbone architectures. For RNN-based models, we use a GRU backbone [Cho et al., 2014], while Transformer-based models employ a decoder-only architecture [Radford et al., 2018].

All models are trained using the Adam optimizer [Kingma and Ba, 2015] with a fixed learning rate of 0.001. The maximum number of training epochs varies by dataset and ranges from 60 to 120 (Appendix D). Early stopping is applied based on validation performance to prevent overfitting. Experiments were conducted on NVIDIA A100 GPUs. All datasets were trained on a single GPU, except Alfabattle, for which we used two GPUs in some experiments.

Hyperparameters, including the loss weights for the NTP objective and model size (Appendix D), are optimized using a Bayesian optimizer [Snoek et al., 2012] applied to the NTP RNN baseline. The resulting configurations are reused across all other RNN settings. For Transformer models, we separately tune the number of layers and the hidden dimension using the NTP configuration and apply these settings consistently across all Transformer-based variants.

We report the mean and standard deviation across five seeds for all methods, except Alfabattle, for which we use three seeds due to computational constraints.

To assess the quality of extracted embeddings, we train a gradient boosting classifier for each downstream task using the LightGBM library [Ke et al., 2017]. The classifier is trained on frozen embeddings and uses the same hyperparameters as in the CoLES baseline [Babaev et al., 2022].

Appendix A contains further reproducibility details.

Baselines implementation details. Two of our baselines, Recurrent Memory Transformer (RMT) [Synerise, 2025] and RMT was originally applied to fixed-length chunks of the input sequence. We follow the same procedure during pretraining but append memory tokens to the end of each chunk for embedding extraction. The final embedding is obtained by averaging the memory token activations.

1 https://boosters.pro/championship/rosbank1/

2 https://ods.ai/competitions/sberbank-sirius-lesson

3 https://boosters.pro/championship/alfabattle2/overview

4 https://tianchi.aliyun.com/dataset/46

The Longformer architecture was introduced for bidirectional Transformers, which are incompatible with next-token prediction pretraining. To adapt it, we modify the attention mask to enforce causality for regular tokens while allowing global tokens to attend to all tokens, including future ones. Conversely, all tokens can attend to global tokens regardless of relative position. An illustration of our modified Longformer mask is provided in Figure 3c. In addition, we observed that convolutional attention leads to suboptimal performance. Instead, we integrate global tokens directly into the causal attention mask. We further found that using a single global token at the end of each sequence consistently outperforms configurations with multiple tokens placed at regular or random positions. As a result, our final Longformer variant employs a single global token without convolutional masking, and the sequence embedding is obtained by averaging the activations of the last token.

To the best of our knowledge, we are applying RMT and Longformer to extract embeddings for event sequence classification tasks for the first time.

5.1 Classification of Event Sequences

Classification results are presented in Table 2. Among baselines in the unsupervised setting, the standard NTP Transformer significantly outperforms its RNN counterpart only on the MIMIC-III dataset, while performing worse on Churn, AgePred, and Alfabattle. This highlights the limitations of traditional Transformer architectures in learning compact and informative sequence representations for downstream tasks.

The proposed HT-Transformer effectively addresses these limitations. It consistently outperforms the NTP Transformer in all comparisons. At the same time, contrastively pretrained RNN (CoLES) surpasses HT-Transformer on AgePred and Taobao datasets. However, on Taobao, the difference is not statistically significant. The same is true for BERT, which uses CoLES for [CLS] pretraining (Appendix C).

The AgePred task differs from the others because it requires predicting a global property, specifically the client's age group, using historical event data. As discussed in the following section, history tokens are designed to capture recent and predictive information, which is beneficial for forecasting future events [Karpukhin and Savchenko, 2026] but less effective for tasks that require encoding global sequence properties. As a result, embeddings extracted from HTTransformer underperform CoLES RNN.

5.2 Global Classification and Future-Oriented Tasks

While the concept of history tokens is broadly applicable, we observe certain limitations when combined with next token prediction during pretraining. The next token prediction objective encourages the model to focus on extracting recent, directly relevant information for forecasting upcoming events. In contrast, downstream tasks involving classification based on global or persistent properties, such as long-term user characteristics, may benefit more from contrastive pretraining or simpler aggregation strategies, such as averaging Transformer outputs across the sequence.

To investigate this effect, we conduct experiments on a synthetic dataset specifically designed to evaluate the suitability of different representation learning methods for local versus global tasks. In this dataset, we sample ten distinct transition matrices, each defining a Markov process by specifying the probability of transitioning from one label to another. We then construct nonstationary sequences by concatenating multiple segments, each generated using a different transition matrix, as illustrated in Figure 4. Note that we use regular timestamps here for simplicity.

We introduce two classification tasks for our synthetic dataset. The global classification task is to predict the total number of transition matrices used in a sequence, ranging from 1 to 5. This task requires the model to capture information across the entire sequence. In contrast, the local classification task involves identifying the index of the transition matrix used in the final segment, which depends only on the most recent data.

The results of classification experiments with Transformerbased models are reported in Table 3. The NTP Last and NTP Avg baselines correspond to Transformer models with different embedding extraction strategies. NTP Last uses the activations of the final token, whereas NTP Avg computes the average activations across all input tokens. The results show that the Last aggregation and HT-Transformer achieve superior accuracy on the local task, while average aggregation and contrastive pretraining yield the highest accuracy on global classification. These findings suggest that history tokens are particularly effective for tasks that depend on recent context, whereas contrastive pretraining and embedding averaging are more suitable for global classification tasks that require holistic sequence understanding.

5.3 Qualitative Analysis of History Tokens

To verify that history tokens are used to predict future tokens and to assess their relative contribution, we analyze attention patterns from the fifth layer of an HT-Transformer trained on MIMIC-III. Figure 5.a shows that the history token receives substantially higher attention weights than standard event tokens. This suggests that the history token serves as an accumulator of prefix information and actively influences the model's next-token predictions.

• (a) HT-Transformer
• (b) NTP Transformer.

Table 4: Comparison of history token placement and selection strategies.

However, the increased attention to the history token could be a side effect of the masking scheme that restricts access to earlier tokens in the HT Transformer. To test this alternative, we analyze a standard NTP Transformer under an analogous masking regime, in which prefix tokens are manually excluded from attention. In this setting, shown in Figure 5.b, the last unmasked token does not receive comparably high attention. This contrast indicates that the prefix-accumulation behavior of the history token is not induced solely by masking, but instead emerges during pretraining.

5.4 Ablation Studies

In the introduction of HT-Transformer, we outlined alternative strategies for history token placement and selection. Table 4 compares these alternatives with the default HT-Transformer configuration, which employs the Bias-End placement strategy and Last selection of history tokens. The results indicate that the default configuration yields superior downstream performance on the considered datasets.

Additional ablation studies in Appendix D demonstrate that the proposed method is largely insensitive to the choice of p and f . Notably, the model trains successfully even with a single history token per sequence. Performance is best when the application probability p is between 0.25 and 0.75.

6 Limitations and FutureWork

In this paper, we demonstrate the effectiveness of using history tokens for event sequence classification. However, several aspects of the method remain open for further exploration. First, as shown in our ablation studies, the placement of history tokens affects downstream performance. We compared two strategies: uniform placement and the Bias-End approach. Future work could pursue a more detailed analysis of token positioning and sampling policies.

Second, our current implementation relies on the standard PyTorch multi-head attention module. This component may not be optimal for working with the custom attention masks required by the HT-Transformer. Future technical improvements could focus on optimizing the attention mechanism, particularly by exploiting the sparsity of the mask. Although HT-Transformer preserves the same asymptotic complexity as standard causal Transformers in the current implementation, the sparse masking structure could enable future optimized implementations with reduced memory and computational cost.

Our experiments demonstrate that HT-Transformer may be suboptimal for global classification tasks. Future research could explore the design of architectures that effectively address both global and local tasks within a unified framework. One promising direction is to combine embeddings extracted from the model using different algorithms, thereby leveraging their complementary strengths.

Overall, the proposed architecture offers a promising direction for highly efficient and accurate modeling of event sequences. Future work can continue to improve the method's predictive performance and computational scalability.

7 Conclusion

This paper introduces HT-Transformer, a novel architecture that enhances Transformer-based models for event sequence classification by explicitly accumulating historical information via learnable history tokens. We identified the inherent limitations of standard Transformers in tasks requiring future event prediction, specifically the lack of a unified representation that effectively captures sequential context. To address this limitation, we proposed a simple yet effective mechanism where history tokens act as information bottlenecks during NTP pretraining, analogous to hidden states in recurrent neural networks.

Our method eliminates the need for auxiliary objectives, such as contrastive learning, and instead leverages sparse attention patterns to ensure efficient information aggregation. Extensive empirical evaluations across real-world financial, healthcare, and e-commerce datasets demonstrate that HTTransformer consistently outperforms conventional Transformer baselines. Moreover, the proposed method achieves the highest accuracy on three datasets, surpassing all competing methods.

Overall, HT-Transformer represents a significant step toward bridging the performance gap between recurrent and Transformer-based models for future-oriented sequence modeling, combining high-quality embeddings with the longcontext modeling capabilities of Transformer architectures.

Acknowledgments

The work of A. Savchenko was partially prepared within the framework of the Basic Research Program at the National Research University Higher School of Economics (HSE).