gSASRec: Reducing Overconfidence in Sequential Recommendation Trained with Negative Sampling (Extended Abstract)

1 Overconfidence in Sequential Recommender Systems

State-of-the-art sequential recommendation models, such as BERT4Rec [Sun et al., 2019], SASRec [Kang and McAuley, 2018] or ALBERT4Rec [Petrov and Macdonald, 2022] are based on the Transformer [Vaswani et al., 2017] architecture and use item IDs in place of token IDs in the original transformer architecture. However, the number of items in the catalogue of a recommender system can be much larger compared to the number of tokens in language models: it is not uncommon to have millions of items in recommender system datasets [Schedl, 2016; Cho et al., 2011], while vocabulary size of language models typically limited by tens of thousands of tokens (e.g. ≈ 30k tokens in BERT [Devlin et al., 2019]).

Therefore, to train recommender models with large catalogues, sequential recommender systems frequently employ negative sampling - a technique in which the model is trained on all positive samples (i.e. items interacted by a user), but only a small proportion of negative samples (i.e. noninteracted items). In this case, the model task is usually considered a binary classification task with the goal of predicting the chances of an item being positive; therefore, many models use Binary Cross-Entropy (BCE) as a loss function (in contrast to the Multiclass Cross-Entropy loss typically used by the language models).

In the original paper, we prove mathematically and show empirically that negative sampling causes overconfidence: the interaction probability predicted by the model becomes much larger compared to the actual probability of a user interacting with an item. For example, Figure 1 illustrates overconfidence in the popular SASRec model. The figure shows interaction probabilities for user 963 with all items in the MovieLens-1M dataset [Harper and Konstan, 2015] (ranked according to their predicted probability). As we can see from the figure, SASRec predicts that the user will interact with all items at ranks from 1 to 25 with a probability close to one. This can not be true - in sequential recommendation, the user only interacts with one item at a time, so the sum of the probabilities should be equal to one and not individual probability. Therefore, most of these predictions are false positives.

In the original paper, we show that false positives (i.e. negative items with predicted probability close to 1) are problematic for the BCE loss, as in these examples, the value of the loss function tends to ∞, causing model effectiveness degradation.

2 gBCE

Consider a scenario when we train a recommendation model using k randomly sampled negatives per positive, i.e. using the sampling rate α = k |I|-1 (where |I| is the catalogue size). To mitigate the overconfidence problem caused by negative sampling, we propose Generalised Binary Cross-Entropy loss (gBCE)):

where I - k is a set of k randomly sampled negatives, si+ is the score predicted by the model for the positive item, si are predicted negative scores, σ is the logistic sigmoid function, α < β ≤ 1 is a hyperparameter (sigmoid for the positive item is raised to the power of β). The main difference between gBCE and the standard BCE loss is that the sigmoid of the positive item is raised to the power of β. Note that gBCE is a generalisation over standard BCE because when we set β = 1, gBCE turns into standard BCE loss.

The range of possible values of the parameter β depends on the sampling rate α, which in turn depends on the catalogue size |I|. To work with gBCE more conveniently, we control power parameter β indirectly with the help of a calibration parameter t that does not depend on the catalogue size. β can be calculated from t using a simple linear transformation:

Possible values of t range between 0 and 1 independent of the dataset size. The case of t = 0 corresponds to standard BCE loss (which, as we argue, causes overconfidence); higher values of t allow for the reduction of overconfidence. In particular, we theoretically prove that by setting t = 1.0, the value of the loss function is minimised when it predicts actual user-item interaction probabilities.

We use gBCE to train gSASRec - a model based on SASRec [Kang and McAuley, 2018] architecture. Compared to standard SASRec, gSASRec has two modifications in the training process: (i) it uses gBCE loss instead of standard BCE, and (ii) it uses multiple negatives per positive instead of one negative.

3 Experiments

In the original paper, through an extensive evaluation using three public datasets, we answer the following research questions:

RQ1 How does negative sampling affect BERT4Rec's performance gains over SASRec?

RQ2 What is the effect of gBCE on predicted item probabilities?

RQ3 What is the effect of negative sampling rate and parameter t on the performance of gSASRec?

RQ4 How does gBCE loss affect the performance of SASRec and BERT4Rec models trained with negative sampling?

RQ5 How does gSASRec perform in comparison to state-ofthe-art sequential recommendation models?

In answer to RQ1, we find that negative sampling causes overconfidence in SASRec, which is the main reason for SASRec's poorer performance compared to BERT4Rec. When controlled for negative sampling, the performance of the models is similar. This is contrary to the original BERT4Rec author's findings [Sun et al., 2019], who attributed the improved performance to BERT4Rec's bidirectional architecture and itemmasking training task.

In answer to RQ2, we experimentally confirm that gBCE can reduce overconfidence in the trained model, and it does it consistently with our theoretical analysis. In particular, we show that by setting calibration parameter t = 1.0, we can train a calibrated model, i.e. the model which predicts interaction probabilities that closely match actual interaction probabilities.

Answering RQ3, we find that both a high number of negatives and high values of t have a positive effect on the model's effectiveness. However, we find that when setting the value of t too close to 1 leads to an increase in training time. Therefore, we propose to use a recommended value of t = 0.75 by default. Similarly, we find that a large number of negatives always leads to improved performance; however, it also increases the required computational resources and memory; therefore, we recommend using 128 or 256 negatives by default.

For RQ4, we find that gBCE increases the effectiveness of both SASRec and BERT4Rec by a large margin compared to standard BCE when the number of sampled negatives per positive is small. For example, when training BERT4Rec on the MovieLens-1M dataset with 4 sampled negatives per positive, gBCE allows to improve Recall@1 metric by 7.2% compared to the standard BCE Loss.

Finally, answering RQ5, we find that gSASRec model trained with gBCE loss matches or outperforms state-of-theart results. At the same time, in contrast to the state-of-the-art models, such as BERT4Rec, gSASRec retains negative sampling that, as we argue, is unavoidable when training recommendation models with large catalogues.

4 Conclusion and Discussion

In the original paper [Petrov and Macdonald, 2023], we showed that negative sampling, which is frequently used for training recommendation models with large catalogues, causes the overconfidence problem, which in turn leads to effectiveness degradation in sequential recommender systems. We showed that overconfidence induced by negative sampling is the key reason for SASRec's lower effectiveness when compared to BERT4Rec. To mitigate overconfidence, we proposed the gBCE loss function and the gSASRec model that uses gBCE. We prove mathematically and demonstrate empirically that gBCE and gSASRec can solve the overconfidence problem while retaining negative sampling. Our proposed gSASRec matches the state-of-the-art results on the small datasets; however, in contrast to other state-of-the-art models, it is suitable for training models on catalogues with more than 1 million items that are common in real life. We hope that gBCE and gSASRec will help to bridge the gap between academic research, where transformer-based models dominate the field of sequential recommendation, and industry, where the application of transformer-based models is still limited due to large catalogues.

Despite the original paper [Petrov and Macdonald, 2023] being specific to the recommender systems domain, the theory and methodology developed in the paper can be interesting to the larger community of researchers. Indeed, the main result of the paper - the methodology for countering problems induced by negative sampling - can be applied to many AI-related tasks that involve multi-class classification with a very large number of classes; examples of such applications may include language modelling with large vocabulary, information retrieval, image classification and many others. For instance, we have already successfully applied the training methodology described in the paper to improve the effectiveness of textual document retrieval in low-latency conditions [Petrov et al., 2024]. Hence, IJCAI, as one of the leading conferences in the field of Artificial Intelligence, is a great venue for sharing the results of the paper with a broad community of AI researchers. At the conference, we will not only present the main results of the original RecSys paper but also show examples of how the results can be adapted for other domains beyond recommender systems.