Knowing When Not to Predict: Self Supervised Learning and Abstention for Safer DR Screening

1 Introduction

Diabetic retinopathy (DR) screening is a safety-critical clinical task. In a screening workflow, a model that makes a confident but incorrect prediction can be more harmful than a model that defers uncertain cases to expert review. In real-world settings, screening systems must handle nongradable images, borderline disease stages, and variability in labeling and grading protocols [Gulshan et al., 2016; Krause et al., 2017; Tsiknakis et al., 2021]. These factors make reliability, not only average accuracy, essential for clinical deployment.

Self-supervised learning (SSL) has become a common approach for representation learning in medical image analysis, particularly when labeled data are limited [Albelwi, 2022; Chen et al., 2020; He et al., 2020; Grill et al., 2020; Zbontar et al., 2021; Bardes et al., 2022]. In ophthalmology and fundus-based DR grading, SSL and transfer learning are widely used to improve performance under limited annotation budgets [Li et al., 2017; Tsiknakis et al., 2021]. Despite this adoption, evaluation in medical imaging studies still largely centers on downstream accuracy or AUROC [Wang et al., 2021]. Such metrics provide an incomplete picture for screening applications, where uncertainty and referral decisions play a central role.

Selective prediction and calibration provide a more deployment-aligned evaluation paradigm. Under selective prediction, the model predicts only for high-confidence cases and abstains on uncertain inputs [Geifman and El-Yaniv, 2017], while calibration aligns predicted confidence with empirical correctness [Guo et al., 2017]. These components are often applied post hoc to a trained model, and representation learning choices are rarely examined through the lens of selective behavior, despite their direct influence on confidence distributions.

A frequently overlooked design aspect is SSL pretraining length. In practice, the number of pretraining epochs is often set by compute budgets or marginal gains in downstream accuracy after performance saturates. Whether this choice also affects reliability properties such as calibration and selective prediction remains unclear, even though they are central to screening workflows.

In this work, we study SSL pretraining length as a reliability factor and analyze how the duration of pretraining affects downstream confidence behavior and selective prediction in DR screening. We evaluate checkpoints spanning early, intermediate, and late pretraining stages, fine-tune each under an identical supervised protocol, and assess both standard performance and reliability-aware metrics under confidence-based abstention [Geifman and El-Yaniv, 2017; Guo et al., 2017].

We use APTOS as the primary benchmark [Tsiknakis et al., 2021] and additionally evaluate transfer to Messidor and a 7-class fundus dataset using the same pipeline. To isolate representation-level effects on confidence behavior, CAMbased refinement [Zhou et al., 2015] is excluded from selective prediction experiments.

Our main finding is that longer SSL pretraining does not monotonically improve reliability. While downstream accuracy often saturates early, selective macro-F1 and confidence behavior can vary substantially across checkpoints, and in some cases selective performance decreases at clinically relevant operating points. These effects are not apparent from accuracy alone and emerge only under selective prediction, which better reflects real screening use.

Contributions. We make three contributions. First, we analyze how SSL pretraining duration affects reliability-aware evaluation in diabetic retinopathy screening. Second, we evaluate checkpoints using selective prediction metrics, including coverage, selective accuracy, and selective macro-F1. Third, using fixed downstream splits, calibration, and training settings, we isolate representation-level effects on confidence behavior.

2 RelatedWork

Our work bridges two areas that are often studied separately: self-supervised representation learning for medical imaging, and reliability-aware evaluation via calibration and selective prediction. We summarize relevant work and position our contribution in diabetic retinopathy (DR) screening.

2.1 Self-Supervised Learning inMedical Imaging

Self-supervised learning (SSL) learns visual representations from unlabeled data. Contrastive methods such as SimCLR and MoCo use positive pairs from augmented views and negatives from other images [Chen et al., 2020; He et al., 2020], while non-contrastive methods avoid explicit negatives through architectural asymmetry or redundancy reduction, for example BYOL, Barlow Twins, and VICReg [Grill et al., 2020; Zbontar et al., 2021; Bardes et al., 2022]. Surveys review these families and their medical imaging adaptations [Albelwi, 2022; Wang et al., 2021].

SSL has been applied across medical imaging modalities, including histopathology, X-ray imaging, and retinal fundus photography [Ciga et al., 2022; Shen et al., 2019; Li et al., 2021]. In ophthalmology, SSL is widely used for DR grading under limited labels and domain shift [Li et al., 2017; Tsiknakis et al., 2021]. However, evaluation still largely focuses on downstream accuracy or AUROC rather than reliability under selective prediction.

2.2 Calibration and Selective Prediction

Confidence calibration aims to align predicted probabilities with empirical correctness [Guo et al., 2017]. Selective prediction allows models to abstain on low-confidence inputs, trading coverage for reduced risk [Geifman and El-Yaniv, 2017]. Learned abstention methods such as SelectiveNet optimize coverage jointly with prediction [Geifman and ElYaniv, 2019], but add modeling and optimization complexity. A widely used baseline is confidence-threshold abstention, often applied after calibration, following the classical reject-option rule [Chow, 2003; Franc et al., 2023].

In medical imaging, selective prediction is closely related to human-AI collaboration and referral workflows, including DR screening systems that defer uncertain or ungradable cases [De Fauw et al., 2018]. Still, many studies treat abstention as a post hoc decision layer, and the interaction between representation learning choices and resulting confidence distributions is not analyzed in depth [Angelopoulos et al., 2024; Cattelan and Silva, 2023].

2.3 Positioning of This Work

We treat selective prediction performance as an emergent downstream property influenced by representations learned during SSL pretraining. Abstention is not part of pretraining in our setup. We sample checkpoints at different pretraining lengths, fine-tune each checkpoint under an identical downstream protocol, and evaluate confidence-threshold abstention using calibrated confidence and selective metrics. Rather than focusing primarily on algorithmic novelty in SSL or abstention mechanisms, we study how SSL pretraining duration affects calibration and risk-coverage behavior in DR screening, addressing an evaluation gap that is directly relevant to deployment.

3 Methodology

We study how the length of self-supervised pretraining influences downstream confidence behavior and selective prediction in diabetic retinopathy screening. To isolate the effect of representation learning, we adopt a controlled protocol in which only the SSL checkpoint varies, while downstream training, calibration, and evaluation settings are kept fixed.

We consider two self-supervised objectives under a common pipeline: SiCoVa (Self-Informed Cross-Correlation for Variance and Invariance), a hybrid non-contrastive objective combining variance, invariance, and cross-correlation regularization, and a contrastive triplet-loss baseline.

3.1 Pretraining Setup

Self-supervised pretraining is performed on unlabeled EyePACS fundus images. From each input image x, we generate two stochastic views v1 = t1(x) and v2 = t2(x) using standard augmentations, including random cropping, horizontal flipping, color jittering, and Gaussian blur. A lightweight Jigsaw perturbation [Park and Ryu, 2024] is applied at the input to encourage patch-level discrimination and fine-grained spatial sensitivity.

Each view is processed by a shared encoder fθ, implemented as either a ResNet50 [He et al., 2016] or a ViT-B/16 [Dosovitskiy, 2020], followed by a three-layer MLP projector gϕ. For any view v, this produces an embedding

For a mini-batch of size N, we write the paired embeddings as Z,Z ′ ∈ RN×D , where the i-th row corresponds to the ith image in the batch, for i ∈ {1, . . . ,N}. Concretely, we denote the embeddings of the two views of image xi by

During SSL training, checkpoints are saved at regular intervals spanning early, intermediate, and late stages of convergence. Since all checkpoints share the same architecture and pretraining configuration, observed downstream differences are primarily associated with pretraining length, although some variability due to stochastic optimization and fine-tuning randomness is expected.

3.2 Non-Contrastive SSL with SiCoVa

SiCoVa is a non-contrastive objective that encourages informative, non-collapsing, and decorrelated representations without using negative samples. It combines intra-view variance and covariance regularization with inter-view invariance and cross-correlation alignment. The objective integrates principles used in VICReg [Bardes et al., 2022] and Barlow Twins [Zbontar et al., 2021], within a unified formulation that jointly optimizes intra-view and inter-view criteria. We weight the different regularization terms using non-negative scalars λintra, λinv, and λcorr, which control their relative contribution to the total loss.

Intra-view regularization. We apply variance and covariance regularization independently to each view. Let Z (j) ∈ RN denote the j-th embedding dimension across the batch, and let Std(Z (j) ) denote its empirical standard deviation. The variance term enforces a minimum standard deviation γ > 0:

where the standard deviation is computed with a small constant added inside the square root for numerical stability.

The covariance regularization penalizes correlations between different embedding dimensions. Let CZ ∈ RD×D denote the sample covariance matrix of Z (computed after centering each dimension). The covariance loss is:

We apply these terms to both views and sum:

Inter-view regularization. To enforce invariance across views, we minimize the mean squared distance between paired embeddings:

To promote alignment of corresponding embedding dimensions while discouraging redundancy across dimensions, we use a cross-correlation criterion. Let ˜ Z and ˜ Z ′ denote column-wise centered and l2-normalized embeddings, and define the cross-correlation matrix:

The correlation loss is:

Overall objective. The SiCoVa objective is a weighted sum of intra-view and inter-view terms:

All weights are fixed across experiments.

3.3 Contrastive Baseline: Triplet Loss

As a contrastive baseline, we use a batch-all triplet loss [Hermans et al., 2017]. For each anchor embedding zi from Z, the positive sample is the corresponding embedding z′i from Z ′ , and all other embeddings z′k with k ̸= i act as negatives. With margin m > 0 and hinge function [·]+ = max(·, 0), the loss is:

(8)

This objective emphasizes inter-instance discrimination and serves as a contrastive reference to SiCoVa.

3.4 DownstreamFine-Tuning

Each SSL checkpoint is fine-tuned on labeled diabetic retinopathy data using an identical supervised protocol. We use fixed training, validation, and calibration splits across all experiments to ensure fair comparison. The downstream task is multi-class DR severity classification.

Class activation map (CAM) based refinement is applied during downstream fine-tuning to improve localization and raw classification performance. However, CAM refinement is not used when evaluating selective prediction and abstention. This separation ensures that abstention behavior and confidence-based deferral are driven by the learned representations and calibrated probabilities, rather than by additional spatial regularization introduced during fine-tuning.

3.5 Calibration and Selective Prediction

After fine-tuning, predicted class probabilities are calibrated using temperature scaling on a held-out calibration split [Guo et al., 2017]. Calibration is performed independently for each SSL checkpoint using identical data and procedures. In particular, we fit a single temperature parameter on the calibration split and apply it to logits at inference time to obtain calibrated class probabilities.

Selective prediction is implemented via confidencethreshold abstention [Geifman and El-Yaniv, 2017] as a post hoc decision rule, and does not affect training. For an input x, let pmax(x) = maxk p(y = k | x) denote the maximum calibrated predicted probability. Given a threshold τ , the model predicts if pmax(x) ≥ τ and abstains otherwise. Coverage is defined as the fraction of samples that are retained (not abstained on).

To support comparisons across checkpoints and methods, we report selective results at a fixed coverage of 70%. This operating point was chosen as a clinically realistic trade-off that retains the majority of predictions while still allowing substantial abstention of uncertain cases for expert review. Concretely, for each checkpoint, we sweep a fixed grid of thresholds τ and select the operating point whose coverage is closest to 70%, then report selective accuracy and selective macro-F1 at that operating point. We additionally observe qualitatively similar trends across neighboring operating points in the risk-coverage analysis.

4 Experimental Setup

This section describes the controlled experimental setup used to isolate the effect of SSL pretraining length on confidence behavior and selective prediction.

4.1 Datasets

We study diabetic retinopathy (DR) severity classification using retinal fundus images. Our primary evaluation benchmark is APTOS-19, a widely used dataset in the DR literature that provides relatively consistent annotations compared to other public benchmarks [Tsiknakis et al., 2021]. APTOS-19 contains five disease severity levels, ranging from no DR to proliferative DR.

For representation learning, models are pretrained on EyePACS using unlabeled images only. EyePACS provides largescale and diverse retinal data and is commonly used for learning general-purpose fundus representations. All downstream fine-tuning and primary evaluation are performed on APTOS19.

To assess robustness and generalization, we additionally evaluate performance on Messidor and a 7-class fundus dataset. These datasets are not used during self-supervised pretraining. For downstream evaluation, we fine-tune and evaluate models on each dataset using the same controlled protocol and labeled-data regimes as for APTOS-19.

Table 1 summarizes the datasets used in our experiments.

4.2 Data Splits and Training Protocol

For downstream training, we use fixed training, validation, and calibration splits for all SSL checkpoints. The calibration split is used exclusively for temperature scaling, while the validation split is used for evaluation both with and without abstention. Identical splits are used across all experiments to ensure fair comparison across pretraining lengths and methods.

We evaluate multiple labeled data regimes by varying the fraction of APTOS-19 training data used for fine-tuning. This allows us to assess whether the impact of SSL pretraining length on reliability persists under limited supervision, which is common in medical imaging settings.

4.3 EvaluationMetrics

We report standard downstream performance without abstention using accuracy and macro-F1. To assess reliability under selective prediction, we additionally report selective accuracy and selective macro-F1 after applying confidence-threshold abstention. Coverage is defined as the fraction of samples for which the model produces a prediction (i.e., not abstained on). For fair comparison across checkpoints and methods, we report selective results at a fixed coverage of 70%.

Macro-averaged metrics are essential in diabetic retinopathy screening due to class imbalance and severity-dependent rejection effects. In particular, selective prediction changes the evaluated set and can skew class frequencies, making accuracy alone potentially misleading. We therefore use selective macro-F1 as the primary metric for ranking checkpoints at the target coverage.

Beyond aggregate scores, we analyze coverage-risk trends and class-wise rejection patterns to characterize how pretraining length reshapes deployment-relevant trade-offs and abstention behavior across disease stages.

4.4 Implementation Details

All experiments use the same architecture and optimization settings across SSL checkpoints. Training and evaluation are performed on a single NVIDIA A100 GPU. Self-supervised pretraining requires approximately 10 hours per model, while downstream fine-tuning takes approximately 1.25 hours per model. At inference time, the abstention mechanism introduces negligible computational overhead, as it relies solely on confidence-based post-processing.

5 Results

We now present experimental results analyzing the effect of self-supervised pretraining length on downstream performance and reliability in diabetic retinopathy screening. We first examine standard classification performance without abstention, and then analyze selective prediction behavior under a fixed operating point.

5.1 DownstreamPerformance

We begin by analyzing downstream classification performance as a function of SSL pretraining length. Figure 1 shows downstream macro-F1 across SSL checkpoints for SiCoVa and Triplet, evaluated at downstream fine-tuning epochs 50 and 200 on APTOS-19 dataset.

Across methods, macro-F1 improves rapidly during early stages of pretraining and then saturates. Extending pretraining beyond this point yields marginal or inconsistent gains, and in some cases mild degradation. This trend is consistent across both fine-tuning durations, indicating that longer downstream training does not compensate for saturation in representation learning.

These observations align with prior work showing that SSL representations converge early with respect to downstream accuracy [Truong et al., 2021]. While some checkpoint-level variability is expected due to stochastic fine-tuning, we focus next on whether pretraining length systematically affects reliability.

Table 2 reports downstream performance of SiCoVa across datasets and labeled data regimes. While accuracy and macro-F1 improve with increased supervision, gains diminish at higher data fractions, particularly on APTOS.

To contextualize these results, Table 3 compares SiCoVa against Triplet, supervised baselines, and prior approaches on APTOS without abstention. SiCoVa achieves the strongest overall performance, particularly when combined with CAM refinement. We next examine whether these accuracy differences translate to improved reliability under selective prediction.

1 ClementP fundus grading collection: https://huggingface.co/

5.2 Selective Prediction at Fixed Coverage

We next evaluate selective prediction using confidence-based abstention. All selective results are reported at a fixed coverage of 70%, reflecting a realistic screening setting in which uncertain cases are deferred for expert review.

Table 4 summarizes selective performance at this operating point for representative SSL checkpoints (with pretraining epochs shown). While selective accuracy remains high across checkpoints, selective macro-F1 varies substantially, even among models with similar downstream performance. Notably, longer SSL pretraining does not consistently improve selective macro-F1: intermediate checkpoints can outperform later checkpoints despite comparable accuracy, suggesting that extended pretraining reshapes confidence allocation rather than raw predictive performance.

Figure 2 complements Table 4 by sweeping the confidence threshold to vary coverage and obtain a risk-coverage curve. SiCoVa consistently achieves higher selective macroF1 across operating points, whereas Triplet is more sensitive to the choice of coverage. The highlighted point corresponds to the 70% coverage operating point used in the table.

5.3 Accuracy versus Macro-AveragedMetrics under Abstention

As confidence thresholds increase, selective accuracy rises sharply due to the removal of ambiguous samples and collections/ClementP/fundus-grading minority-class instances. In contrast, selective macro-F1 improves more gradually and often plateaus.

This discrepancy arises because accuracy is dominated by majority classes and benefits from aggressive rejection, whereas macro-F1 penalizes uneven class-wise performance. As a result, selective macro-F1 provides a more informative measure of reliability under abstention for imbalanced screening tasks.

5.4 Representation Structure and Qualitative Analysis

Figure 3 provides qualitative Grad-CAM visualizations across APTOS, Messidor, and the 7-class fundus dataset. Accepted predictions typically exhibit more coherent activation patterns, while rejected cases more often show diffuse or unstable responses. These visualizations serve as qualitative sanity checks rather than localization claims.

Figure 4 visualizes representation structure and severitydependent rejection using PaCMAP and class-wise acceptance statistics. Disease severity classes form a continuous and partially overlapping manifold, reflecting the progressive nature of diabetic retinopathy. After abstention, accepted samples occupy denser and more class-consistent regions, while rejected samples concentrate near inter-class boundaries rather than forming isolated outliers, suggesting that abstention primarily filters semantically ambiguous cases.

5.5 Cross-Dataset Transfer

Finally, we evaluate selective prediction on Messidor and the 7-class fundus dataset. While downstream accuracy remains relatively stable across SSL checkpoints, selective macro-F1 continues to vary, indicating that the impact of pretraining length on confidence behavior generalizes beyond a single benchmark.

Overall, these results demonstrate that SSL pretraining length has a limited effect on downstream accuracy but a pronounced impact on confidence calibration and selective prediction behavior, with direct implications for safe deployment in clinical screening systems.

6 Discussion

Our results demonstrate that the length of self-supervised pretraining plays a meaningful role in shaping confidence behavior and selective prediction performance in diabetic retinopathy screening. While downstream accuracy often saturates early, reliability-aware evaluation reveals substantial variation across SSL checkpoints that is not captured by standard metrics. In this section, we discuss the implications of these findings for representation learning, clinical interpretation, and deployment.

6.1 Why Longer Pretraining Does Not Guarantee Better Reliability

A common assumption in SSL-based medical imaging is that longer pretraining is either beneficial or, at worst, neutral once downstream accuracy has converged. Our results challenge this assumption. Although extended pretraining does not consistently degrade standard downstream performance, it can still reshape confidence distributions, which may affect selective macro-F1 at a fixed coverage. We examine this relationship directly by reporting selective performance across pretraining epochs in the subsequent analysis.

One plausible explanation is that later stages of SSL pretraining emphasize invariances that improve global discrimination while compressing features relevant for borderline or minority classes. In diabetic retinopathy, disease severity evolves along a continuum rather than forming well-separated categories. As a result, representations that appear strong under accuracy-based evaluation may still yield poorly calibrated confidence near class boundaries. These effects remain largely invisible unless selective prediction behavior is explicitly analyzed.

Importantly, our findings do not suggest that longer pretraining is inherently harmful. Rather, they indicate that pretraining length interacts with downstream reliability in nonmonotonic ways, and that accuracy alone is insufficient to identify favorable operating points.

6.2 Clinical Interpretation of Accepted and Rejected Cases

To assess the clinical plausibility of abstention decisions, a clinician inspected a subset of accepted and rejected predictions. Rejected cases often exhibited factors known to complicate automated grading, including decentration, shadowing, media opacities, and confounding retinal conditions, consistent with criteria for non-gradable images in screening workflows. The clinician also noted occasional inconsistencies within the accepted set and a small number of rejected images of acceptable quality. This underscores that confidence-based abstention reflects model uncertainty, not clinical ground truth or image quality, and may conservatively reject borderline but usable cases. Overall, abstention should be interpreted as decision support rather than a replacement for expert judgment [De Fauw et al., 2018].

6.3 Accuracy versus Reliability in Screening Systems

Our analysis highlights limitations of accuracy as a deployment-oriented metric. Under selective prediction, accuracy can increase as uncertain samples are rejected even when performance on clinically relevant minority or borderline classes degrades. Macro-averaged metrics expose this effect by penalizing uneven class-wise performance and severity-dependent rejection. Selective macro-F1 therefore, provides a more balanced assessment of reliability under abstention and aligns better with equitable performance across severity levels.

6.4 Implications for Deployment and Evaluation

First, SSL pretraining length should be treated as a design choice with direct consequences for confidence behavior. While selective performance varies across checkpoints, we do not observe a clear monotonic trend with pretraining length, motivating further experiments. Second, benchmarking SSL methods solely on downstream accuracy can miss clinically relevant differences in uncertainty handling. Third, confidence-based abstention is a simple deferral mechanism, but its behavior should be assessed across operating points and disease severities.

7 Conclusion

We study how self-supervised pretraining length influences reliability and selective prediction in diabetic retinopathy screening. Although downstream accuracy saturates early, confidence behavior and selective macro-F1 vary substantially across SSL checkpoints, affecting deployment decisions under abstention. These findings motivate reliabilityaware evaluation and treating pretraining length as a primary design choice.

Limitations. We focus on a single SSL framework and a limited set of architectures, and the clinical inspection is qualitative and based on a small sample. We rely on confidencebased abstention rather than learned deferral. In addition, we do not perform statistical significance testing across checkpoints or repeated fine-tuning runs. Future work should extend to additional SSL objectives and architectures, analyze reliability trends across multiple random seeds, and include larger-scale human-in-the-loop evaluation.

Acknowledgments

This research has been funded by the Federal Ministry of Education and Research of Germany and the state of NorthRhine Westphalia as part of the Lamarr Institute for Machine Learning and Artificial Intelligence.