Rethinking Convolutional Networks for Attribute-Aware Sequential Recommendation

1 Introduction

In the era of explosive growth of social media networks, video streaming, and e-commerce platforms, recommendation systems have become an indispensable tool to help users navigate the vast digital landscape by providing personalized suggestions that align with their unique preferences and past behaviors. More recently, sequential recommendation has emerged as a dominant recommendation task. Unlike traditional collaborative filtering (CF) methods, such as Matrix Factorization (MF) [Koren et al., 2009; Rendle et al., 2012] that solely focus on static user-item interactions, sequential recommendation models are designed to capture the temporal dynamics of user behaviors.

† The extended version of this paper, including supplementary appendices, is available at https://arxiv.org/pdf/2605.04723.

Earlier approaches in sequential recommendation domain were based on Markov Chains [Shani et al., 2005], [Rendle et al., 2010], which could capture short-term dependencies but struggled with longer-range patterns. With the arrival of deep learning, a new wave of model architectures that are capable of handling high-order, nonlinear, and longrange data sequences was introduced. Models using Recurrent Neural Networks (RNNs) [Cho et al., 2014], [Hidasi and Karatzoglou, 2018] could effectively capture longer evolving sequential dependencies, but suffer from training and inference inefficiency due to the lack of parallelization. On the other hand, Transformer-based models managed to address these limitations by leveraging self-attention to model dependencies across all items in a sequence at the same time, which eliminates the need for sequential processing and enables full parallelization during training. Models, such as SASRec [Kang and McAuley, 2018], BERT4Rec [Sun et al., 2019], helped establish self-attention as a dominant modeling paradigm in the field of sequential recommendation. Despite parallelization capabilities, the computational and memory complexity of self-attention scales quadratically with the sequence length, which offsets much of the training efficiency it provides.

In this paper, we revisit Convolutional Neural Networks (CNNs) as a competitive framework for temporal and attribute-aware sequential recommendation. Although early CNN-based models like Caser [Tang and Wang, 2018] demonstrated the ability to capture local transition patterns, its receptive field is strictly bounded within the fixed history window of the last few items. Thus, it is unable to capture the global view of the long-range sequential dependencies in the way Transformers can. To overcome this limitation, we propose a hierarchical convolutional encoder that progressively downsamples the entire interaction sequence (via striding/pooling) to a single node, thereby distilling the temporal features into a compact "intent" representation (Figure 1) with a global view of the sequence. This design takes inspiration from the pyramidal and patch-based techniques that were successfully used in time-series and vision models [Wang et al., 2024], [Lin et al., 2017].

We introduce ConvRec, a hierarchical down-scaling convolutional model for attribute-aware sequential recommendation. First, it fuses each item embedding in the user's interaction history with its corresponding attribute embeddings and contextual embeddings into a joint representation. These embeddings form a chronological interaction sequence for each user, which is fed through a series of convolutional blocks. In each block, a strided convolution operation aggregates local neighboring interactions and downsamples the sequence. By repeating this down-scaling procedure in a pyramidal fashion, ConvRec progressively distills the entire user interaction history into a single vector with rich representation of the entire sequence, which is then used to score and rank next-item candidates via a dot-product operation.

Our primary contributions are as follows:

• We propose a novel convolutional encoder for sequential recommendation, which leverages item attributes and context, and applying a hierarchical down-scaling convolutions to capture both local and long-range user interaction patterns.
• Experiments across widely used sequential recommendation datasets, demonstrating superior performance of ConvRec in comparison to state-of-the-art baselines.
• We show that ConvRec scales linearly in memory and computation time with sequence length, and offers substantial efficiency improvements over existing best performing Transformer-based models on long user sequences.

2 RelatedWork

Sequential Recommendation. Sequential recommendation aims to predict the next item a user will interact with and is central to modern recommender systems. Earlier models like Caser [Tang and Wang, 2018], CosRec [Yan et al., 2019], and RCNN [Xu et al., 2019] used convolutions and recurrent layers to capture sequential patterns. GRU4Rec [Jannach and Ludewig, 2017] and BERT4Rec [Sun et al., 2019] introduced recurrent and transformer architectures, while SASRec [Kang and McAuley, 2018] employed multi-head self-attention. Extensions such as TiSASRec [Li et al., 2020], CoSeRec [Liu et al., 2021b], and TiCoSeRec [Dang et al., 2023] incorporate temporal and contrastive learning signals to further improve performance. Attribute-aware Sequential Recommendation. Incorporating item attributes is essential for capturing user preferences more accurately. S3Rec [Zhou et al., 2020] leverages pre-training to enhance data representations. Other models, such as NOVA [Liu et al., 2021a], DIFSR [Xie et al., 2022], and MSSR [Lin et al., 2024], propose various strategies for feature fusion. CARCA [Rashed et al., 2022] introduces cross-attention mechanisms to compute item relevance scores more effectively. ProxyRCA [Seol et al., 2024] builds upon CARCA by incorporating proxy-based item embeddings, significantly improving representation quality. Additionally, HGTL [Xu et al., 2025] applies cross-domain transfer learning to leverage user preferences from multiple domains.

Despite these advancements, convolutional architectures remain relatively underexplored in the context of sequential recommendation, even though they have shown strong potential for capturing local patterns in sequences [Liu et al., 2022; Wang et al., 2024]. In this work, we propose a lightweight, hierarchical convolutional approach for down-scaling and encoding user sequences, offering an efficient and effective alternative for sequence representation.

3 Problem Setting

In the attribute-aware sequential recommendation setting, we consider a set of users U interacting with a set of items I. Each user u ∈ U has a chronologically ordered interaction sequence Su = [iu1 , . . . , iu|Su | ], where each item is associated with a set of descriptive features/attributes A (e.g., title, brand, etc.), the items features sequence for user u is denoted as [au1 , . . . , au|Su | ]. Additionally, each interaction is accompanied by contextual information C, such as the timestamp of the interaction, which can be decomposed into temporal components (e.g., year, month, day). We denote the contextual sequence for user u as [cu1 , . . . , cu|Su | ]. The primary objective of the next-item recommendation task is to predict the most likely item a user will interact with at the next time step.

4 ConvRec

Considering the similarity between time-series forecasting and next-item recommendation tasks, sequential recommendation can similarly benefit from the hierarchical scaling properties used in time series. Specifically, a user's input sequence can be aggregated in a hierarchically downscaled manner to derive a meaningful sequence representation. Given that neighboring items in a sequence are often more closely related than distant ones [Wang et al., 2020; Kang and McAuley, 2018], unlike the self-attention mechanism, we propose a representation method where items are gradually merged with their neighbors to form a unified representation of the entire sequence.

4.1 ItemEncoding

For item encoding, we rely not only on item IDs, but also emphasize the importance of incorporating item attributes/features. These attributes provide insight into user preferences and enhance the model's ability to accurately predict the next item. For a given input sequence of fixed length L (obtained through either padding or truncation), each item's features are passed through a fully connected layer:

Wa ∈ R|A|×da is a weight matrix, |A| is the length of the item's features set, da is the embedding dimension of the input features and ba is the bias term. In addition to item features, we incorporate interaction context, such as time and location of interactions. In this work, we focus on temporal context, which is also encoded via a fully connected layer:

Wc ∈ R|C|×dc is the weight matrix for the context features, |C| is the number of contextual elements, dc is the context embedding dimension, and bc is the bias term. The encoded item features and context vectors are then concatenated and passed through another fully connected layer to generate the interaction embedding:

Here, Wf ∈ R(da+dc)×df , df is the output embedding dimension, and bf is the bias term.

While various strategies exist for incorporating item IDs in recommendation systems, we adopt the approach introduced in [Seol et al., 2024], which assigns unique embeddings only to the most frequent items. Less frequent items are assigned a shared generic ID. This approach shifts the model's learning focus toward high-frequency items while relying on feature-based representations for infrequent ones. Accordingly, we embed item IDs using a lookup table: i′i u = LookupTable(ii u ). This embedding is concatenated with the previously obtained feature-context embedding and passed through a linear layer:

Wv ∈ R(di+df)×dv , dv is the embedding dimension of the combined item IDs and side information, and bv is the bias term.

In sequential recommendation, temporal dynamics play a critical role. The time gap between interactions can vary significantly, so time not only contributes to the representation of individual interactions but can also help assess their relative importance. To this end, we compute time intervals (year, month, day) between each two consecutive interactions, and concatenate the intervals to the input sequence, where for the first interaction time interval is set to zero:

Here, c is the interaction context where we extract only the (year, month and day) for the interval calculation, V u = [v1, v2, . . . , vL] is the sequence of item representations, and ∆cu = [∆cu1 ,∆cu2 , . . . ,∆cuL] are the corresponding intervals, and σ denotes the GELU activation function [Hendrycks and Gimpel, 2016]. Then it is fed into a fully connected layer to obtain a combined representation.

WQ ∈ R(dv+3)×dv is a weight matrix, and bQ is the bias term, and σ denotes the GELU activation function.

4.2 Convolution Down-scaling Component (CDS)

Building on the idea of gradual aggregation within a user's historical interaction sequence, we design a hierarchy of convolutional layers to construct the final sequence representation. Given an encoded sequence Z u ∈ RL×dv , where L denotes the maximum sequence length and dv is the dimensionality of the interaction embeddings, our objective is to reduce the sequence length L to 1. This is achieved by progressively aggregating information through a series of hierarchical convolutional downscaling layers that downscale the sequence over multiple ConvBlocks:

Here, Z u denotes the input embeddings, initially shaped as L× dv, where L is the sequence length and dv is the embedding dimension, and Xu denotes the output representation. To enable convolutional operations across neighboring items, the embeddings are reshaped such that the convolutional kernel can operate along the sequence dimension with dv input channels.

Convolution Block (ConvBlock). To perform the convolution operations effectively, the input sequence Z u is transposed to shape dv × L, treating dv as the number of input channels. This reshaping allows 1D convolution to be applied along the sequence dimension L:

The 1D convolution is applied across the sequence dimension (L) using a specified kernel size and stride. In most cases, the stride is set equal to the kernel size to ensure nonoverlapping windows, which empirically yielded better performance. When the sequence length is not divisible by the kernel size, appropriate padding is applied to maintain dimensional consistency. After convolution, K(j) u is transposed back to the shape L(j) ×dv, where L(j) is the sequence length after applying the convolution operation. A final fully connected layer is then applied:

Here, WG ∈ RdK(j)×dG(j) is the weight matrix of the output projection layer, bG is the bias term, and dG is the output embedding dimension, which is also typically set equal to dv for consistency and σ is a GELU activation.

Subsequently, layer normalization is performed on the output. Additionally, we incorporate two types of weighted residual connections to enhance training stability:

The first residual connection is derived by applying average pooling to the original input. This helps preserve the original signal and mitigates the risk of vanishing gradients. The second, ProgRes(j) , is a progressive residual connection from the previous convolutional layer's output. It is also downsampled using average pooling to match the output dimensions.

In the first convolutional layer, both residual connections originate from the same input, as no prior layer exists. However, in subsequent layers, ProgRes continues to evolve, allowing for smoother gradient flow and improved model convergence. α1 and α2 are learnable parameters that are initialized to 0.5.

Consequently, the downscaling convolutional blocks can be defined as follows:

Here, M denotes the total number of convolutional layers, resulting in the final output O(M) u with an expected shape of dv × 1. However, in cases where the convolutional layers do not completely collapse (using average pooling), the sequence length is reduced to 1. An additional average pooling layer is applied to ensure the final output has the desired dimension.

4.3 Model Training and Optimization

For item ranking, we calculate the final score by taking the dot product of the target items' embedding Z u target obtained from the item encoder (Equation 7) and the transpose of the sequence embedding (Xu )T as follows:

Given a positive item sampled from a user's sequence and a set of negative items drawn from a candidate pool (excluding items in the user's sequence), we train the model using the binary cross-entropy loss defined as follows:

Algorithm 1 Forward pass of the ConvRec model

Require: input encoded sequence Z u , list of layer specs M = [(kernel1, stride1), . . . , (kernelm, stridem)]

Require: activation function σ(·), pooling function Pool(·)

Ensure: logits ˆ Y

• 1: ProgRes(0) ←Z u
• 2: O(0) u ←Z u
• 3: for j = 1 →M do
• 4: K(j) u ← 1D-Conv(Padding(O(j-1) uT ), kernelj, stridej) (Eq: 9)
• 5: G(j) u ← σ(K(j) uT WG + bG) (Eq: 10)
• 6: O(j) u ← LayerNorm(G(j) u + α1 · AvgPool(Z u ) + α2 · ProgRes(j-1) ) (Eq: 11)
• 7: ProgRes(j) ←O(j) u
• 8: end for
• 9: Xu ←Collapse(O(M) u ) (Eq: 13)
• 10: ˆ Y ←Z u target · (Xu )T (Eq: 14)
• 11: return ˆ Y

where ˆ Y (+) are the output scores for the positive sample and ˆ Y (-) n are the output scores for Ntrain negative samples in our case we set Ntrain to 100. S is the set of all sequences.

Many recommender models select the most recent part of the input sequence to represent the input and the shifted sequence to represent the positive items of the user [Kang and McAuley, 2018], [Rashed et al., 2022]. While effective, this approach narrowly focuses on recent interactions, potentially overlooking valuable information present in earlier parts of the user's history. In contrast, we adopt the training strategy proposed in [Seol et al., 2024], where instead of always selecting the latest subsequence, a random subsequence of length L is sampled from the full user sequence Su , and the final item in that sampled subsequence is used as the positive target. This allows the model to learn from various segments of the user's behavior, not just the most recent interactions, and promotes better generalization.

5 Experiments

In this section we aim to answer the following research questions:

• RQ1: What is the model performance versus state-ofthe-art methods?
• RQ2: How effective is each model component?
• RQ3: What is the effect of varying model hyperparameters such as kernel sizes and strides?
• RQ4: What are the computation and memory costs of the proposed model versus attention-based approaches?

Datasets. To comprehensively evaluate our proposed model, we conduct experiments on four widely used Amazon datasets. The Fashion and Men datasets [Elsayed et al., 2022], [Rashed et al., 2022], [Seol et al., 2024] are derived from different segments of the Amazon fashion domain. The Fashion dataset includes six categories: men's and women's tops, bottoms, and shoes. In contrast, the Amazon Men dataset comprises a broader range of subcategories, such as gloves, scarves, and sunglasses. For both datasets, item attributes are derived from visual features extracted using a pre-trained ResNet-50 model. In addition, we include the Amazon Games dataset [Rashed et al., 2022], [Seol et al., 2024], which contains video game items along with attributes such as category and brand. Lastly, the Amazon Beauty dataset [Rashed et al., 2022], [Seol et al., 2024] includes all items in the beauty category, where item attributes are also based on category and brand information. Table 1 summarizes the statistics of the datasets.

Evaluation Protocol. We adopt a widely-used evaluation protocol based on the leave-one-out strategy [Kang and McAuley, 2018], [Rashed et al., 2022], [Seol et al., 2024]. Specifically, for each user, the last interacted item is held out for testing, the second-to-last item is used for validation, and the remaining sequence is used for training. For a fair comparison, we adopt the same negative sampling strategy used in prior studies [Rashed et al., 2022], [Seol et al., 2024], in which 100 negative items are sampled for evaluation.

We report standard top-K metrics, Hit Rate (HR), and Normalized Discounted Cumulative Gain (NDCG) to effectively capture the performance of our model.

Baselines. We selected several state-of-the-art sequential and attribute-aware sequential recommendation models to assess the performance of our proposed method.

• BERT4Rec [Sun et al., 2019]: A sequential recommendation framework based on a bidirectional Transformer, trained to predict randomly masked items within the input sequence.
• SASRec [Kang and McAuley, 2018]: A self-attentionbased model that learns item dependencies in the sequence to generate next-item.
• CosRec [Yan et al., 2019]: A 2D convolutional model that encodes item sequences to capture complex pairwise and high-order item interactions.
• TiSASRec [Li et al., 2020]: An extension of SASRec that integrates temporal information (e.g., time intervals between interactions) into the attention mechanism.
• S3 -Rec [Zhou et al., 2020]: An attribute-aware sequential recommendation model that employs self-supervised pre-training to enhance item and user representations.
• SASRec++ [Rashed et al., 2022]: An enhanced version of SASRec that incorporates item attribute information to improve recommendation accuracy.
• DIF-SR [Xie et al., 2022]: A state-of-the-art approach that injects item features into the attention layers rather than restricting them to the input layer.

• CARCA [Rashed et al., 2022]: A context-aware recommendation method that employs cross-attention to compute item scores.
• CARCA-LIP [Seol et al., 2024]: ACARCAmodel variant proposed in [Seol et al., 2024] introduces an improved training approach and enhanced item embeddings, resulting in better performance.
• ProxyRCA [Seol et al., 2024]: An advancement over CARCA that introduces proxy-based item embeddings, further improving item representation quality.

5.1 Model Performance Against Baselines

Table 2 reports the HR@10 and NDCG@10 of all methods on the four benchmark datasets, evaluated with 100 negative samples per user. For each dataset, the mean and standard deviation are reported based on three independent runs. The baseline models were trained and tuned according to the guidelines and hyperparameter settings in their original papers. ConvRec achieves the best results in both HR@10 and NDCG@10 metrics across all datasets. ProxyRCA is the strongest baseline and ranks second in terms of overall performance in most of the cases. Relative to ProxyRCA, ConvRec yields improvements ranging from 0.65% to 4.08% in HR@10 and from 2% to 7.82% in NDCG@10. In overall, attribute-aware models outperform sequence-only baselines. Among them, all recent top performers are attention-based models, whereas, to the best of our knowledge, ConvRec is the only convolution-based attribute-aware model.

Table 3 compares model performance on the Beauty and Games datasets using the all-item evaluation setting. While baseline sequential models (e.g., BERT4Rec, SASRec) perform modestly, stronger baselines such as TiSASRec and ProxyRCA achieve competitive results. Our proposed ConvRec surpasses all existing methods, delivering the best HR@10 and NDCG@10 on both datasets. ConvRec achieves improvements of up to 3.6% in HR@10 and 8.0% in NDCG@10 over the best baseline, indicating its enhanced ability to model sequential dependencies and user-item interactions.

5.2 Ablation Studies

The ablation studies are organized into four main parts:

1. Analyzing the impact of individual model components on overall performance.
2. Comparing memory and runtime efficiency between the proposed convolution-based model and a multi-head attention-based model, such as ProxyRCA.
3. Exploring alternative convolutional architectures by testing different kernel sizes and strides. Additionally, testing the hyperparameter sensitivity of the model.
4. Evaluating the effect of varying sequence lengths on model performance, using the Games dataset, which has the longest average user sequence length.

Effect of Model Components

Table 4 presents the impact of various model components on overall performance. One key factor examined is the concatenation of the time intervals between successive interactions. This component shows a particularly notable impact on the Beauty dataset, highlighting the importance of temporal information in modeling user behavior.

Another important component is the residual connection. Its effect is evident on all datasets for example in the Men dataset, removing the two residual connections leads to a significant performance drop from 0.618 to 0.574. This underscores the value of incorporating residual connections into the model architecture.

Furthermore, we evaluate a variant of the model that uses a single convolutional layer with both kernel size and stride equal to the sequence length. This configuration effectively acts as a fully connected layer over the entire input sequence. While its performance is slightly lower than the proposed model, the drop is not substantial.

Additionally, we consider a baseline that computes the average pooling of item embeddings without any additional layers. This approach generally results in a performance decline across most datasets. However, the drop is relatively minor in the Men dataset, where the performance decreases to 0.611.

Memory and Runtime Efficiency

We evaluated the training memory and runtime efficiency on Beauty dataset of our model and ProxyRCA using a batch size of 128 on NVIDIA A40 GPU. As shown in Figure 4, ProxyRCA exhibits a quadratic increase in memory consumption with respect to sequence length, rising from 3,878 MB for sequences of length 50 to 14,280 MB for sequences of length 500. In contrast, our model demonstrates linear memory growth, scaling from 2,849 MB to 6,096 MB under the same conditions.

Runtime performance follows a similar trend: ProxyRCA requires 0.0893 seconds per batch for sequences of length 50 and 0.382 seconds for length 500, while our model completes the same tasks in 0.076 and 0.202 seconds, respectively. These results highlight our model's superior scalability in both memory efficiency and computational speed.

Hyperparameter sensitivity

We explore alternative network architectures by experimenting with different kernel sizes. Furthermore, we analyze the impact of various hyperparameter settings, such as the embedding size and the dropout rate, to assess the model's sensitivity to different configurations (Appendix A† ). Lastly, we examined the performance across user groups with varying sequence lengths to evaluate the effectiveness of our convolution-based model compared to an attention-based model such as ProxyRCA (Appendix B† ).

Kernel size. Since kernel size plays a crucial role in convolutional models, tuning it directly impacts the network architecture. As shown in Table 5, we experimented with various kernel sizes across different numbers of layers. The choice of kernel size for the first layer proved particularly important for the final model performance. The best results were obtained with a first-layer kernel size of 2, where each pair of items is aggregated together, followed by subsequent layers that aggregate the next 5 groups, and so on. Additionally, we experimented with different strides, such as a stride of 1 to combine each item with its preceding and succeeding items for a richer representation. However, this did not improve performance, and using a stride equal to the kernel size yielded the best results.

6 Conclusion

This paper introduces ConvRec, a convolution-based model for attribute and context-aware sequential recommendation. By leveraging adaptive convolution layers, the model compresses user historical sequences into compact yet highly expressive representations. Experimental results demonstrate that ConvRec outperforms state-of-the-art methods in recommendation accuracy. Moreover, compared to traditional multi-head attention approaches, the proposed model achieves superior efficiency in both memory usage and runtime, making it scalable for real-world applications.

Contribution Statement

Shereen Elsayed and Ngoc Son Le contributed equally to this work.