Beyond Top-1: Addressing Inconsistencies in Evaluating Counterfactual Explanations for Recommender Systems (Extended Abstract)

1 Introduction

As machine learning systems are increasingly deployed in decision-support settings, the need for transparent and trustworthy model behavior has become central to modern AI research [Ghorbani et al., 2019]. Among post-hoc explainability techniques, counterfactual explanations (CE) [DoshiVelez and Kim, 2017] have emerged as a particularly intuitive paradigm. Rather than describing why a model produced a certain output, counterfactual explanations identify minimal changes to the input that would lead to a different outcome. This makes them especially attractive in domains where actionable feedback and user understanding are important [Karimi et al., 2021].

Recent years have seen growing interest in applying counterfactual explanations to systems that produce ranked outputs, such as recommender systems, retrieval engines [Mozafari et al., 2026], and personalized ranking models [Zhong and Negre, 2022; Yu et al., 2025]. In these settings, the objective is typically not to predict a single class label, but to generate an ordered list of items according to estimated relevance. Explaining such systems therefore introduces distinct methodological challenges that differ from traditional classification settings. One of the most fundamental open challenges concerns evaluation. While many methods have been proposed to generate counterfactual explanations, considerably less attention has been paid to how they should be compared reliably [Bauer et al., 2024]. Existing evaluation protocols in recommender systems often assess explanations based on whether perturbing the identified important inputs changes the top-1 recommended item. Although convenient, this practice implicitly reduces a ranked prediction task to a single-item outcome and overlooks the list-wise nature of the model output [Mohammadi et al., 2024]. We argue that top1 evaluation can produce unstable conclusions. In ranking systems, the highest-ranked item may be sensitive to small score fluctuations, ties, or modest changes in model quality. As a result, explanation methods may appear better or worse depending not on their true explanatory power, but on incidental variation in the underlying model. This creates an important threat to fair comparison and reproducible benchmarking [Mohammadi et al., 2025].

In this paper, we revisit the evaluation of counterfactual explanations through the lens of ranked outputs. We investigate whether extending evaluation beyond the top-1 item to top-k ranked outputs leads to more consistent assessments. In this setting, consistency denotes the capacity of an evaluation metric to preserve a reliable ordering of CE methods even when the recommender model undergoes performance variations. For instance, if explanation method E1 performs better than method E2 for recommender R1, then after slight changes in the recommender system, the absolute CE scores may vary, but E1 should remain superior to E2 in relative ranking. While [Mohammadi et al., 2024] identified such inconsistencies in current evaluation protocols, the root causes remain unresolved. Intuitively, evaluating multiple highly ranked items should better reflect the structure of recommendation tasks while reducing sensitivity to local ranking noise.

To study this question, we evaluate six representative explanation methods across multiple datasets, recommender architectures, and model performance checkpoints. Our results show that list-wise top-k protocols substantially improve consistency compared with conventional top-1 evaluation. These findings suggest that explanation benchmarks for ranking systems should align with the ranked nature of model outputs rather than inherit evaluation assumptions from single-label prediction tasks.

2 RelatedWork

Research in interpretable machine learning has advanced significantly in recent years [Ranjbar et al., 2024; Ghorbani and Zou, 2020]. Initial efforts largely concentrated on developing inherently interpretable models, where transparency is embedded directly into the model design, enabling human-understandable reasoning without additional processing [Peintner et al., 2023; Peintner et al., 2025]. Alongside this line of work, post-hoc explainability has become a widely adopted alternative, focusing on explaining model behavior after training while treating the underlying system as a black box [Wachter et al., 2017; Kaffes et al., 2021].

Despite rapid progress in explanation methods, recent studies reveal that evaluation practices remain inconsistent and underdeveloped. A comprehensive survey covering hundreds of XAI papers shows that many approaches rely on qualitative or anecdotal validation, while rigorous quantitative evaluation often addresses only a narrow subset of explanation characteristics [Nauta et al., 2023]. The authors introduce a set of conceptual properties (Co-12), such as Correctness, Compactness, and Consistency, emphasizing that explanation quality is inherently multi-faceted. However, existing work typically focuses on isolated criteria, overlooking interactions between them. This limitation becomes particularly pronounced in settings where model outputs are structured, such as ranked predictions, where explanations depend not only on individual outputs but also on their relative ordering. Motivated by this gap, we focus on consistency as a central property and analyze it in the context of counterfactual explanations for recommender systems.

Explanation Methods for Recommender Systems Explainability in recommender systems has attracted considerable attention due to its role in improving user trust, transparency, and accountability. Existing approaches can be broadly grouped into model-intrinsic, post-hoc, and auxiliary-information-based methods. Model-intrinsic approaches incorporate interpretability directly into the recommendation model. Early work on explainable matrix factorization aligns latent representations with interpretable components [Abdollahi and Nasraoui, 2016]. More recent architectures, such as ProtoMF [Melchiorre et al., 2022], leverage prototypes or attention mechanisms to identify influential user-item interactions. While these methods provide structured explanations, they are often tightly coupled to specific model designs. Post-hoc approaches, in contrast, aim to explain predictions without modifying the underlying recommender. However, these approaches primarily capture correlational relationships and do not explicitly account for the sensitivity of ranked outputs. A separate line of work exploits auxiliary information such as textual reviews, aspects, or knowledge graphs. Aspect-based models like the Explicit Factor Model (EFM) [Zhang et al., 2014] align latent factors with interpretable attributes, while path-based approaches justify recommendations through semantic connections in knowledge graphs [Chen et al., 2020]. More recent studies question the reliability of attention-based explanations and propose causal reasoning over graph paths as a more faithful alternative [Li et al., 2024]. Although these methods provide rich explanations, they rely on additional data sources that are not always available.

Counterfactual Explanations Counterfactual explanations (CEs) provide insights by identifying minimal changes to the input that would alter the model's output [Wachter et al., 2017]. Due to their actionable nature and intuitive interpretation, they have become a prominent approach for explaining recommendation models. Early methods for generating CEs relied on heuristic search or perturbation strategies to identify influential interactions [Mohammadi, 2023]. More recent approaches employ reinforcement learning to efficiently explore the combinatorial space of possible modifications. For example, frameworks such as CERec [Wang et al., 2024] learn policies over attributes or graph structures, often incorporating fairness considerations. Conversational recommender systems further integrate counterfactual reasoning to iteratively refine explanations through user interaction [Yu et al., 2024; M¨uller et al., 2026]. In parallel, recourse-oriented methods such as RecRec [Verma et al., 2023] focus on actionable modifications that can influence ranking outcomes. Model-agnostic approaches aim to improve scalability and generality. Notably, LXR [Barkan et al., 2024] learns explanation masks through self-supervised objectives, avoiding explicit combinatorial search over perturbations. While effective, such methods do not explicitly address the stability of explanations across ranked outputs.

Evaluation of Counterfactual Explanations Despite substantial progress in generating counterfactual explanations, their evaluation remains limited and often simplistic. Most existing approaches assess explanation quality based on whether the top-ranked item changes after perturbation [Tan et al., 2021; Barkan et al., 2024; Zhong and Negre, 2022; Ser et al., 2024]. This practice effectively reduces a ranked prediction problem to a single outcome, disregarding the structure of the full ranking. Recent studies show that such evaluation protocols can lead to unstable conclusions: small variations in model performance may alter the relative ranking of explanation methods [Mohammadi et al., 2024; Mohammadi et al., 2025]. This instability raises concerns about the reliability and reproducibility of current benchmarks. In contrast, our work adopts a list-wise perspective by evaluating explanations across multiple top-ranked items. By aligning evaluation protocols with the ranked nature of model outputs, we aim to provide a more stable and principled basis for comparing counterfactual explanation methods.

3 Methodology

To investigate the impact of extending the evaluation from top-1 items to top-k items, we propose the following approach.

3.1 Problem Setup and Preliminaries

Let U denote the set of users and I the set of items. Each user u ∈ U is represented by a binary interaction vector xu ∈ {0, 1}|I| , where xu[i] = 1 indicates that user u has interacted with item i. A recommender model fθ maps this input to a predicted score vector, with fθ(xu)[i] representing the predicted relevance of item i. The RS generates a ranked list of recommendations for each user. We denote the top-k recommended items as Rku = {i∗1 , i∗2 , . . . , i∗k }, where i∗j is the j-th item in the ranked recommendation list. These are the items for which we seek to evaluate CEs.

Given a counterfactual explainer e, we obtain a relevance ranking over user history items with respect to each item i∗j in Rku . Based on this relevance, we iteratively perturb the user input vector xu by removing one item at a time, yielding a sequence x(t) u of perturbed inputs. Following the literature [Barkan et al., 2024], We compute the following metric:

Positive Perturbation (POS-P@T) This metric quantifies how quickly the top-k recommended items fall below the rank threshold T during perturbation. For user u, the POS-P@T score is:

where rank(i∗j ; x(t) u ) is the rank of i∗j in the output of fθ given the perturbed input x(t) u and the indicator function 1[·] checks whether the item still appears within the rankd threshold T. The key difference between this formulation and the literature [Barkan et al., 2024; Mohammadi et al., 2024] lies in computing the inner value over the top-k items (∑kj=1 ) rather than only the top-1 item.

The final POS-P@T score is the average over all users:

3.2 Evaluation Setup

We use the MovieLens 1M (ML-1M) [Harper and Konstan, 2015], Yahoo! Music [Dror et al., 2012] and Pinterest [He et al., 2017] datasets following the literature [Mohammadi et al., 2024], focusing on collaborative filtering models based on implicit user-item interactions. For each user, we generate a ranked list of recommendations and evaluate the explanation of the top-k recommendations. For each k value, we use four levels of recommender performance by training the recommender based on Hit Rate and saving model checkpoints after 25%, 50%, 75%, and full training, to simulate recommendations of various quality levels. We evaluate 24 configurations (3 dataset x 2 Recommenders x 4 performance levels), repeat our experiments three times, and report the average performance. We share our dataset processing scripts, the source code, and the hyper-parameters 1 .

3.3 EvaluatedMethods

Our evaluation includes a range of CE methods, from established baselines to recent advancements.

Jaccard [Bag et al., 2019]/Cosine [Singh et al., 2020] Similarity generate explanations by comparing an item to those in the user's history using Jaccard [Bag et al., 2019] or Cosine [Singh et al., 2020] similarity metrics based on cointeracted users.

LIME-RS [N´obrega and Marinho, 2019] is a modified version of the LIME framework designed specifically for recommender systems.

SHAP (SHapley Additive exPlanations) [Zhong and Negre, 2022] quantifies the influence of individual features on a model's predictions. By leveraging game theory, it assigns a value to each feature according to its contribution to the final output.

ACCENT (Action-based Counterfactual Explanations for Neural Recommenders for Tangibility) [Tran et al., 2021] is a CE framework that extends influence functions to provide actionable, model-agnostic insights for neural recommenders.

LXR [Barkan et al., 2024], the state-of-the-art CE approach, employs self-supervised learning to generate explanation masks that highlight critical user data without perturbations using a gradient learning approach.

Our study includes two collaborative filtering recommenders: (1) Matrix Factorization (MF) [Abdi et al., 2018] and (2) Variational Autoencoder (VAE), a generative model that encodes and decodes data while learning a probabilistic latent representation [Liang et al., 2018].

4 Experiments and Results

Based on our hypothesis that extending evaluations to include lower-ranked recommendations-such as the top-3 items-enhances consistency, we incorporate lower-ranked items in the evaluation. Particularly, we compute performance separately for top-1, top-2, top-3, and higher-ranked recommendations, then aggregate these performance values to obtain a comprehensive assessment. For instance, computing the value for top-3 involves selectively masking data from the user's history to ensure that the third-highest recommended item drops to a lower rank (rank ≥ 4). This method ensures that the evaluation captures the influence of multiple recommendation ranks rather than relying solely on the highest-ranked item, leading to a more stable comparison of CE methods.

To test this hypothesis, we evaluate six representative CE methods using the POS metric, analyzing its performance across four thresholds of recommendation length (top-k) and four levels of recommender performance. As shown in Figure 1, evaluations based solely on the top-1 recommendation (Fig. 1a (a)) exhibit significant fluctuations on every two recommenders, particularly when transitioning from the lowest-performing recommender (TP25%) to

1 https://github.com/dbis-uibk/CE4RS-Eval

(b) MF recommender on Pinterest dataset

Figure 1: Comparison of CE methods based on (a) POS@5 ↓ (lower value is the better) and (b) POS@10 ↓ across different performance levels. The figure shows the impact of going beyond top-1 (a) and considering top-k (b-d) recommendations on improving consistency when evaluating CE models. To facilitate comparisons, the values are normalized, and shading is used to represent the variance in the results.

TP50%. However, when extending the evaluation to the top-2 recommendations (Fig. 1a (b)), we can already observe more consistency, especially for high-performing recommenders (TP50%-TP100%). Following this, setting the threshold to include the top-5 recommendations (Fig. 1a (d)) stabilizes the evaluation outcomes for the highest performing recommenders (TP50%-TP100%). We observed a similar pattern in other configurations, albeit with different thresholds. For the MF recommender, the ranking of methods remained stable even at smaller k values across all datasets, in contrast to the VAE (Fig. 1b) . The observed consistency when considering only the top-3 items stems from the simplicity of the MF model, which results in a more straightforward recommendation task and lower performance variability among recommenders, ultimately leading to less variability in explainers. More detailed results are available in the accompanying Git repository presented in section 3.2. Overall, this evidence highlights the importance of considering listwise recommendations in CE evaluations. Adopting such an approach not only enhances evaluation consistency but also ensures better alignment with the inherent characteristics of recommendation systems. Importantly, our findings show that evaluation consistency is not uniformly affected across all metrics. POS is particularly sensitive to the quality of the recommender. These results suggest that the choice of metric-and the value of k-should be treated as a tunable hyperparameter, influenced by both the dataset and the recommender architecture. Moreover, the use of multiple performance checkpoints for the recommender further reinforces the robustness of our evaluation framework. By capturing performance across training stages, we ensure that the evaluation of explainers is not biased by any single fixed model state. This approach not only provides a more comprehensive view of the explainer's effectiveness but also reveals how susceptible existing metrics are to fluctuations in model quality.

5 Conclusion and FutureWork

In this paper, we address a gap in the evaluation of counterfactual explanations for recommender systems, focusing on evaluation consistency. Our findings highlight the importance of considering top-k recommendations when assessing CEs. By extending evaluations beyond the top-1 recommendation, we demonstrate consistency on current mainstream metrics for evaluation. These findings not only address key methodological gaps in counterfactual evaluation but also lay the groundwork for future research into domain-adaptive CE metrics. An important direction for future work is the development of unified evaluation metrics that jointly consider both recommender performance and explanation quality.