Less Is More: Proportional Memory-Guided Differential-Attention MIL for Whole-Slide Image Classification

1 Introduction

Histopathological examination of tissue specimens is the gold standard for cancer diagnosis, subtyping, and grading, providing critical insights into tissue morphology and cellular organization [Campanella et al., 2019; Di et al., 2022]. With the advent of digital pathology, conventional glass slides can now be scanned at high resolution to produce whole-slide images (WSIs), enabling large-scale computational analysis of histopathological data [He et al., 2012]. However, learning effective representations from WSIs remains fundamentally challenging. WSIs are gigapixel-scale images that exhibit extreme variability in tissue area, pronounced intra-slide heterogeneity, and sparse diagnostically relevant regions embedded in large amounts of normal or weakly informative background tissue [Srinidhi et al., 2021]. Moreover, acquiring fine-grained annotations at the pixel or patch level is prohibitively expensive and rarely available in routine clinical practice [Lu et al., 2021]. These factors jointly make direct supervised learning on WSIs impractical.

∗Corresponding authors sq.ma@siat.ac.cn, guofei@csu.edu.cn

To address these limitations, multiple instance learning (MIL) has emerged as the dominant framework for WSI classification. In MIL, each WSI is modeled as a bag of patch-level instances, and learning is performed using only slide-level labels [Zheng et al., 2023; Zheng et al., 2024; He et al., 2026]. Early MIL approaches relied on simple pooling strategies, while subsequent works introduced attention-based mechanisms and more sophisticated aggregation architectures to learn instance importance and improve slide-level predictions [Ilse et al., 2018; Li et al., 2021; Xiang and Zhang, 2023; Zheng et al., 2025].

However, existing MIL approaches still suffer from fundamental limitations. Most methods either aggregate all available instances or retain a fixed number of top-ranked patches based on local classifier confidence. Such designs are poorly aligned with the intrinsic properties of WSIs: fixed instance budgets fail to adapt to the extreme variability in slide size and lesion burden, while confidence-based instance selection is often unstable and locally myopic. As a consequence, informative regions may be discarded on large slides, whereas redundant or irrelevant patches may dominate the representation on smaller or less pathological slides.

More importantly, recent studies suggest that the core challenge of WSI analysis is not merely how to aggregate instance representations, but which instances should be treated as reliable evidence for slide-level prediction [Wang et al., 2025]. Not all patches contribute equally to the final decision: visually dominant or high-confidence regions are not necessarily the most informative, while small and spatially localized regions may carry decisive diagnostic cues.

In this work, we propose Proportional Memory-guided Differential-attention Multiple Instance Learning network (PMDMIL), a scalable MIL framework that reconsiders how evidence should be selected and organized for whole-slide image classification (Figure 1). Motivated by the sparsity and spatial structure of diagnostic evidence in WSIs, PMDMIL performs group-wise proportional instance selection to adaptively retain informative patches across slides with varying sizes and lesion extents. A class-wise prototype memory is introduced to guide instance selection, encouraging the retention of patches that are both informative and consistent at the class level. The selected instances are then integrated through a differential-attention mechanism [Ye et al., 2024; Hammoud and Ghanem, 2025] with coordinate-based positional encoding [Vaswani et al., 2017], enabling effective modeling of spatially distributed evidence using a compact token set. Experiments on five public WSI classification benchmarks demonstrate that PMDMIL consistently outperforms representative MIL baselines, while ablation and efficiency analyses confirm that its components play complementary roles in improving performance and scalability. Our main contributions are summarized as follows:

• We propose PMDMIL, a proportional memory-guided differential-attention MIL framework for scalable WSI classification.
• We introduce a memory-guided instance sampler that maintains a class-wise prototype memory bank and enables adaptive, proportional instance selection.
• We design a differential-attention refiner operating on spatially grouped tokens with coordinate-based rotary positional encoding to capture inter-region relations and global tissue context.

2 RelatedWork

2.1 MIL-basedWhole-Slide Image Classification

MIL is widely adopted for WSI classification task [Campanella et al., 2019; Chen et al., 2022; Coudray et al., 2018], where only slide-level labels are available. Most existing MIL

approaches focus on designing aggregation mechanisms to combine patch-level embeddings into a slide-level representation. Representative methods include attention-based MIL models, hierarchical aggregation frameworks, and more recent transformer- or graph-based architectures for modeling instance relationships across WSIs [Yu et al., 2025].

Several studies investigate attention mechanisms to estimate instance importance and emphasize discriminative patches for slide-level prediction. Beyond simple attentionbased pooling, structured MIL architectures have been proposed to improve robustness and incorporate contextual information among instances, for example by modeling spatial or relational dependencies [Zhang et al., 2024; Fang et al., 2024]. While effective, these approaches generally operate on all instances or rely on fixed aggregation schemes, which can be inefficient for large-scale WSIs.

2.2 Memory-Based and Prototype-GuidedModels

Memory mechanisms [Wu et al., 2018; Caron et al., 2020; Wei et al., 2025; Zhou et al., 2023] and prototype [Song et al., 2024] representations have been extensively studied in metric learning, few-shot learning, and self-supervised representation learning, where representative features are maintained to provide stable semantic references and improve generalization [Snell et al., 2017; Fang et al., 2025; Ho et al., 2023]. In computational pathology, prototype-based ideas have been explored to capture characteristic morphological patterns or to regularize feature learning across heterogeneous tissue regions [Adnan et al., 2022; Yu et al., 2023; Yao et al., 2020].

Despite these efforts, most existing MIL methods for WSIs do not explicitly maintain a class-level memory across slides to guide instance selection or aggregation. As a result, instance importance is often estimated independently for each slide, without leveraging accumulated class-consistent evidence from the dataset.

3 Methodology

We propose PMDMIL for supervised WSI classification. It adopts a group-first processing strategy with memory-guided proportional instance selection and differential-attention refinement, enabling robust and scalable slide-level representation learning. Figure 2 illustrates the process setting and contrasts PMDMIL with conventional MIL pipelines, while Figure 3 details the overall architecture of PMDMIL.

3.1 ProblemFormulation

A WSI is represented as a bag of patch features and their spatial coordinates,

where xi ∈ Rdin denotes a patch feature extracted by a pretrained patch encoder and pi ∈ R2 denotes its spatial location on the slide. Each slide is associated with a slide-level label Y ∈ {1, . . . ,C}, while instance-level labels are unavailable. The objective is to predict the slide label by identifying and integrating informative regions within the bag.

3.2 Group-Wise Feature Adaptation

Directly operating on all patches of a WSI is computationally expensive and prone to noise accumulation. PMDMIL therefore follows a group-first strategy. Patch features are first adapted to a shared embedding space using a linear adapter,

where ϕ(·) consists of normalization, linear projection, and non-linear activation. This adapter serves to harmonize features extracted from a frozen patch encoder and prepares them for subsequent aggregation.

The embedded patches are then partitioned into a set of spatial groups based on their coordinates.

Grouping localizes instance selection and aggregation, ensuring that spatially dense regions do not dominate sparse but informative areas.

3.3 Memory-Guided Instance Sampler

A core component of PMDMIL is memory-guided instance sampler, which aims to retain informative patches while suppressing redundant or weakly informative regions.

Within each spatial group, the number of retained instances is determined proportionally to the group size rather than being fixed across slides. This proportional strategy naturally adapts to whole-slide images with highly variable tissue coverage and lesion burden.

To stabilize instance selection, PMDMIL maintains a class-wise prototype memory bank,

where L denotes the memory length per class. For efficient similarity computation, we use the mean prototype for each class ¯Mc = 1L ∑Ls=1 Mc,s.

Within each spatial group Gg, patches are first pre-scored based on their cosine similarity to the mean class-wise prototypes. Given a patch embedding hi ∈ RD , its memory-guided pre-score is computed as

Based on these pre-scores, a proportion of patches is retained from each group,

where q ∈ (0, 1] is a predefined sampling ratio and Sg ⊂ Gg denotes the selected instance set.

Instance-level class evidence is then computed only for the selected patches. Specifically, for each patch j ∈ Sg, we compute instance-level logits as

where finst(·) denotes an instance classification head. The bag-level class probability for the group is estimated using a LogSumExp [Blanchard et al., 2021] aggregation over the selected instances,

Based on the resulting class distribution, a group-level memory token is constructed as a weighted combination of mean class prototypes,

which represents the dominant semantic content of the group. The memory token is prepended to the selected instance tokens and assigned the mean spatial coordinate of the group for subsequent processing.

During training, the class-wise prototype memory is updated using an exponential moving average. Let ¯hSg denote the normalized mean feature of the selected instances. The mean prototype of the most probable class is updated as

Finally, for each spatial group, the sampler outputs a compact set of tokens composed of the group-level memory token and the selected patch tokens. Specifically, the output feature set is formed by concatenating the memory token with the selected instance features,

To preserve spatial information, the corresponding coordinates are constructed by assigning the mean coordinate of the group to the memory token and retaining the original coordinates of the selected patches,

where ¯ pg = 1 |Gg | ∑i∈Gg pi. These group-level outputs are then passed to the subsequent aggregation and refinement modules.

3.4 Differential-Attention Refiner

After the sampler, each spatial group Gg outputs a compact set of tokens composed of a group-level memory token and the selected patch tokens and coordinates (Hg, Pg). To summarize local evidence within each group, PMDMIL applies a simple mean pooling layer over the sampler outputs, yielding a single group-level representation for Gg.

The aggregated representations from all spatial groups are concatenated to form a bag-level token sequence Hb ∈ RG×D . Let Pb ∈ RG×2 denote the corresponding group-level coordinates, obtained by averaging the coordinates of patches within each group.

Given the token sequence and their coordinates, 2D rotary embeddings are generated from the spatial coordinates and injected into the attention mechanism. The refiner first applies differential multi-head attention with a residual connection, and the attention output is further refined using a feedforward network with normalization and a residual connection,

where R(·) denotes the rotary positional encoding derived from the 2D coordinates.

Finally, a learnable attention-based pooling operation is applied to aggregate the refined group representations into a single slide-level embedding. Specifically, The pooling weights and slide-level representation are computed as,

where w ∈ RD is a learnable projection vector.

The resulting slide-level embedding z is fed into a classification head to obtain the final prediction. Specifically, z is first normalized and then passed through a two-layer multilayer perceptron with GELU activation and dropout to produce slide-level logits ˆ y ∈ RC . The predicted class probabilities are computed using a softmax function, and the final slide label is obtained by selecting the class with the highest logit value.

3.5 Training Objective

Our model is trained primarily with cross-entropy loss at the slide level. Let ˆ y ∈ RC denote the slide-level logits predicted from the final slide representation z. The primary objective of PMDMIL is the slide-level cross-entropy loss,

In addition, the sampler produces instance-level logits for the selected patches within each group. We optionally use them as an auxiliary supervision signal by propagating the slide label to instances. Let {ℓk}Kk=1 be the instance logits

(Eq. 8) returned by the sampler for a slide. The auxiliary instance loss is defined as

The overall training objective is

where λ ≥ 0 controls the contribution of the auxiliary instance-level supervision.

4 Experiments and Results

4.1 Dataset Description

We evaluate PMDMIL on five publicly available WSI classification datasets: TCGA-BRCA, BRACS [Brancati et al., 2022], PANDA [Bulten et al., 2022], DHMC [Zhu et al., 2021] and TCGA-NSCLC. These H&E-stained cohorts span multiple organs, including breast, prostate, lung and kidney, and cover a wide range of dataset scales, with the number of WSIs per cohort varying from 261 to 10,614. Figure 4 illustrates the distribution of the number of patches per WSI across the five datasets. For fair comparison, we follow the same dataset splits as used in previous study [Zhang et al., 2025b].

4.2 Implementation Details

Whole-slide images are processed using the TRIDENT framework [Zhang et al., 2025a; Vaidya et al., 2025]. Tissue regions are first segmented to exclude background areas, after which non-overlapping patches are extracted at 20× magnification. The patch size is fixed to 512 × 512 pixels. This preprocessing pipeline follows standard practice in computational pathology and is consistent across all datasets. Patchlevel features are extracted using a frozen foundation model (UNI2-h [Chen et al., 2024]) from TRIDENT framework.

All models are optimized using AdamW with a batch size of one slide per iteration. Training is performed for 20 epochs. The initial learning rate is set to 1 × 10-4 and is scheduled using cosine annealing. All experiments are conducted following the general experimental protocol of 2DMamba [Zhang et al., 2025b], including data split, optimization strategy, and training schedule, to ensure fair and reproducible comparison. Unless otherwise specified, all hyperparameters are kept consistent across datasets. All experiments are conducted using the RTX 6000 Ada GPU. Model selection is based on validation performance.

4.3 Results

Table 1 summarizes the quantitative comparison between PMDMIL and representative multiple instance learning methods on five public WSI classification benchmarks. The compared baselines include classical attention-based MIL approaches as well as recent sequence modeling methods based on transformers and Mamba architectures. Performance is evaluated using accuracy (Acc), F1-score, and AUC, with all results averaged over five runs.

PMDMIL achieves strong and consistent performance across all datasets and outperforms most competing methods on the majority of evaluation metrics. Compared with classical MIL baselines such as AB-MIL, CLAM, and DSMIL, PMDMIL shows clear improvements, particularly on datasets with high intra-slide heterogeneity. Moreover, while recent sequence modeling approaches (e.g., TransMIL and Mambabased methods) achieve competitive results by modeling long-range dependencies, PMDMIL further improves performance on challenging datasets such as PANDA and NSCLC, especially in terms of accuracy and F1-score, indicating that explicit instance selection remains beneficial even with strong sequence models.

On DHMC, PMDMIL achieves the best performance across all three metrics, demonstrating its ability to capture subtle morphological differences. On BRCA and BRACS, PMDMIL consistently improves F1-score, reflecting enhanced class balance in multi-class settings. Notably, these gains are obtained without processing all available patches, providing quantitative evidence that effective instance selection can be more important than exhaustive aggregation for WSI classification.

4.4 Analysis of Sampling Ratio

To analyze whether retaining more patches consistently leads to better performance, or whether a compact subset of informative patches is sufficient for accurate WSI classification, we investigate the behavior of PMDMIL under different sampling ratios q, which control the proportion of patches retained for each whole-slide image. Specifically, we vary q from 0.1 to 1.0 and evaluate the resulting classification performance, computational cost, and learned slide-level representations on the DHMC dataset.

Figure 5 summarizes the experimental results. As shown in the performance curves, the classification metrics do not exhibit a monotonic improvement as the sampling ratio increases. Instead, PMDMIL achieves strong and stable performance at moderate sampling ratios, with performance around q = 0.1 being comparable to or better than that obtained using significantly larger ratios. Further increasing the sampling ratio yields diminishing or inconsistent performance gains, and in some cases slightly degrades performance. These results indicate that exhaustively processing all available patches is not necessary to achieve high classification accuracy.

In contrast to the relatively stable performance trends, the computational cost measured by FLOPs increases approximately linearly with the sampling ratio. This behavior reflects the direct relationship between the number of retained instances and the computational burden of downstream processing. Consequently, moderate sampling ratios provide a substantially more favorable trade-off between accuracy and efficiency compared to full sampling, highlighting the scalability advantage of proportional instance selection in largescale WSI analysis.

To further understand the effect of the sampling ratio on representation quality, we visualize the learned slide-level embeddings using t-SNE. As illustrated in Figure 5, representations learned with a moderate sampling ratio (q = 0.2) form more compact and better-separated class clusters than those obtained with a high sampling ratio (q = 0.9). When a large proportion of patches is retained, the embeddings become more scattered and exhibit increased overlap between classes, suggesting that excessive inclusion of weakly informative patches may dilute discriminative structure in the learned representations.

This analysis demonstrates that PMDMIL benefits from a principled instance selection strategy. By retaining only a compact subset of informative patches, the model is able to preserve discriminative representations while significantly reducing computational cost. This analysis provides empirical support for the central "less is more" principle underlying PMDMIL, showing that effective instance selection is more important than exhaustive aggregation for whole-slide image classification.

4.5 Ablation Study

We conduct ablation experiments on the DHMC dataset to analyze the contribution of key components in PMDMIL, with results summarized in Table 2. Removing memory-guided sampling leads to a substantial performance drop across all metrics, indicating that the class-wise prototype memory provides an important global reference for identifying informative patches beyond proportional sampling alone. Eliminating the auxiliary instance-level supervision also consistently degrades performance, suggesting that instance-level scores serve as a useful regularization signal under slide-level supervision. Replacing differential attention with standard attention results in a smaller but noticeable performance decrease, highlighting the benefit of explicitly modeling relative differences between group representations. These findings indicate that memory guidance, instance-level supervision, and differential-attention refinement play complementary roles in improving the robustness and effectiveness of PMDMIL.

5 Conclusion

In this work, we proposed PMDMIL, a proportional memoryguided multiple instance learning framework for whole-slide image classification. Motivated by the sparsity and spatial structure of diagnostic evidence in WSIs, PMDMIL performs group-wise proportional instance selection to reduce redundant computation while preserving spatial organization, following a principled "less is more" paradigm. Experiments on five public benchmarks demonstrate that PMDMIL consistently outperforms representative MIL baselines while processing only a fraction of patches, and ablation analyses confirm the complementary roles of memory-guided sampling and differential-attention refinement. In conclusion, PMDMIL provides an effective and scalable solution for WSI classification.

Acknowledgments

This work is supported by grants from the National Natural Science Foundation of China (Grants No. 62322215, 62532017 and 62402488), and Natural Science Foundation of Hunan Province (Grants No. 2026JJ30018).