PMCE: Probabilistic Multi-Granularity Semantics with Caption-Guided Enhancement for Few-Shot Learning

1 Introduction

Few-shot learning studies the recognition of novel categories from only a handful of labeled examples. Although deep networks perform well with large-scale supervision, they often overfit when only 1-5 supports per class are available. As a result, the gains over simple baselines are often modest, especially in the 1-shot regime and on visually diverse benchmarks [Fei-Fei et al., 2003; Chen et al., 2019].

∗Corresponding author

To alleviate data scarcity, metric-based and distributionbased methods learn transferable embedding spaces or reuse base-class statistics. Matching Networks [Vinyals et al., 2016], Prototypical Networks [Snell et al., 2017], MAML [Finn et al., 2017] and later strong baselines [Chen et al., 2019] improve the metric, while distribution calibration approaches [Yang et al., 2022; Wu et al., 2021; Wu and Hu, 2022; Xu et al., 2026] transfer base statistics to novel classes by adjusting empirical feature distributions or sampling from Gaussian priors. These methods show the benefit of a probabilistic view, but their priors are defined and selected purely in the visual space, mainly according to feature similarity and hand-designed rules.

Semantic-based methods introduce attributes, word vectors or text embeddings to compensate for limited visual data. AM3 [Xing et al., 2019] adapts visual prototypes with class-level semantics. With CLIP [Radford et al., 2021], BMI [Li and Wang, 2023], SP-CLIP [Chen et al., 2023b], SemFew [Zhang et al., 2024], PRSN [Dong et al., 2025] and related work encode class names with prompts and inject the resulting text features into metric learning pipelines or prototype resynthesis, leading to strong and robust baselines. However, most methods rely on coarse class-level text and mainly modify support prototypes or classifier weights, while query features remain purely visual or are adjusted in a transductive manner. Instance-level semantics such as captions are rarely modeled under the standard inductive few-shot protocol.

In this work, we propose PMCE motivated by a simple yet critical observation, that is, in the inductive setting, improving the support prototype alone is often not sufficient when the query representation remains purely visual and deviates from the prototype manifold. We address this by organizing semantics at two complementary granularities. At the class level, text embeddings act as retrieval keys to identify relevant base-class statistics as priors for MAP prototype estimation. At the instance level, captions provide label-free descriptions that optimize both support and query representations into a more coherent space. This design is particularly helpful in 1-shot regime, where support evidence is scarce and supportquery mismatch becomes a major source of errors.

In summary, for this paper, the main contributions are:

• We present PMCE, which connects semantic-based and distribution-based few-shot learning by using classname embeddings to retrieve class-specific priors from a non-parametric knowledge bank and injecting them into support prototypes under a MAP view.
• We design a caption-guided dual-stream enhancer that integrates BLIP-generated captions with visual features for both supports and queries in an inductive setting, yielding a more coherent space for prototype matching.
• We introduce a lightweight caption consistency regularization and demonstrate consistent gains on four benchmarks with two backbones, with the largest improvements in the 1-shot regime.

2 RelatedWork

In this section, we briefly review three lines of work that are most related to PMCE and clarify how our framework fits into this landscape.

Metric- and distribution-based few-shot learning. Traditional few-shot methods often learn a transferable embedding space or a good initialization, such as Matching Networks [Vinyals et al., 2016], Prototypical Networks [Snell et al., 2017], and MAML [Finn et al., 2017], as well as strong baselines that carefully revisit pre-training and episodic training [Chen et al., 2019]. Recent work further improves the representations or matching rules, for example FGFL [Cheng et al., 2023] and CPEA [Hao et al., 2023]. Another line transfers base-class statistics to novel classes by modeling feature distributions, including distribution calibration [Yang et al., 2022] and MAP-based prototype estimation [Wu et al., 2021; Wu and Hu, 2022]. These methods are effective, but priors are usually selected only in the visual space based on feature similarity or simple heuristics.

Semantic-based few-shot learning. Semantic-basedmethods introduce attributes or text embeddings to compensate for limited visual data [Guo and Guo, 2021; Guo et al., 2023; Guo et al., 2024]. AM3 [Xing et al., 2019] adapts prototypes with class-level semantics, while CLIP-based [Radford et al.,

2021], approaches such as BMI [Li and Wang, 2023], SPCLIP [Chen et al., 2023b], SemFew [Zhang et al., 2024], and PRSN [Dong et al., 2025] encode class names with prompts and inject text features into metric learning or prototype construction. Most of them mainly modify support prototypes or classifier weights; query features are often left purely visual, or refined transductively using query-set statistics.

Vision-language models and caption-based semantics. Vision-language pre-training offers general-purpose text cues for images [Lu et al., 2023]. CLIP learns aligned imagetext embeddings [Radford et al., 2021], and BLIP can generate high-quality captions [Li et al., 2022]. In few-shot learning, these cues are commonly used at the class levels [Chen et al., 2023b; Zhang et al., 2024], whereas automatic captions are less explored under the inductive setting. PMCE keeps CLIP and BLIP frozen as off-the-shelf semantic extractors, using class names for stable prior retrieval and captions for query-side refinement.

3 Preliminaries

In this section, we explain the few-shot setting and review the MAP-based prototype estimation that our method builds on.

3.1 Problem Setting

In the standard few-shot setting, we have a base set Db = {(xi, yi)}|Db| i=1 ⊂ X × Cb and a novel set Dn = {(xi, yi)}|Dn| i=1 ⊂ X × Cn, where Cb ∩ Cn = ∅. Db provides abundant annotated data needed to train the visual backbone and the enhancement module, while Dn is used only at metatest time.

At meta-test time, we evaluate the performance on the Nway K-shot tasks. Each task contains a support set S = {(xi, yi)}N×K i=1 and a query set Q = {(xj, yj)}N×M j=1 , where M is the number of query samples per class. Given support set S and a query image xq, the goal is to predict its label by

where C denotes the set of N classes sampled in the current episode.

3.2 MAP-Based Prototype Estimation

Following [Wu et al., 2021; Wu and Hu, 2022], we construct a visual knowledge bank based on the priors from base set. Specifically, for each base class j ∈ Cb, we store its mean feature

where Dj b = {(xi, yi) ∈ Db | yi = j}.

For a novel class c in an episode, pcinit is the empirical prototype calculated from the mean of its support features. Given a class-specific prior mean µcprior obtained from the knowledge bank, we calibrate the prototype by a MAP estimator under the Gaussian assumptions, which admits a closed-form convex combination:˜ pc = α pcinit + (1 - α) µcprior , (3)

where α ∈ [0, 1] controls the strength of the prior.

4 Method

In this section, we describe semantics-guided prior retrieval for MAP prototype estimation, the caption-guided dualstream enhancer, and the training objective. An overview is shown in Fig. 2.

4.1 Semantics-Guided Prior Selection

We demonstrate the prior selection with CLIP class-name semantics for MAP prototype estimation in Eq. (3).

Class-level semantics on base classes. Let ny be the natural language name of class y. We follow the CLIP-style prompting strategy and encode

which provides a coarse description of category y. For each base class j, we store its mean feature µj and class-name embedding sjbase = sjclass in a non-parametric semantic knowledge bank

Prior selection for novel classes. At meta-test time, for a novel class c with class name nc and embedding scclass , we measure its similarity to each base class via cosine similarity:

We then select the indices of the top-K base classes to obtain a neighbor set Ntop and aggregate their visual means into a class-specific prior. Instead of uniform averaging, we use a similarity-weighted aggregation:

where τ is a temperature.

MAP view with spherical uncertainties. We interpret the support prototype pcinit as a noisy observation of the true class mean µc under a spherical Gaussian likelihood pcinit ∼ N (µc , σ2 like I), while the retrieved prior mean provides a spherical Gaussian prior p(µc ) = N(µcprior , σ2 prior I). Under Gaussian-Gaussian conjugacy, the MAP estimate has a closed form that reduces to a convex combination:

Thus, α is determined by the relative confidence between the support evidence and the retrieved semantic prior.

4.2 Caption-Guided Dual-StreamEnhancement

In this section, we introduce the enhancer in our framenwork. We optimize representations with instance-level captions by applying the same enhancer Φ to the support and query streams. Given a visual feature and its caption-derived semantic representation, Φ outputs an enhanced visual embedding used for prototype construction and query matching.

Instance-level semantics. To provide finer-grained cues and cover unlabeled queries, we use image captions as instance-level semantics. For any image x, we first generate a caption with a frozen BLIP captioner,

and then encode it with the same text encoder gϕ(·) to obtain a single sentence embedding:

where dt is the text feature dimension. sinst is label-free and thus available for both supports and queries, serving as the semantic input to Φ on the two streams.

Enhancement module. We define a mapping

where vin ∈ R1×dv is a visual anchor (e.g., a prototype or a query feature), sin ∈ Rdt is a semantic context, and Θ are learnable parameters. Since visual and textual features live in different spaces, we first project semantics to the visual

with Wp ∈ Rdt×dv , bp ∈ Rdv and LN(·) layer normalization. We then perform cross-attention by using the visual anchor as query and the projected semantics as key/value:

where WQ,WK,WV ∈ Rdv×dk are learnable projections. The attention output is computed as

A residual connection with a learnable scale β (initialized to a small value) preserves the original visual structure:

Support and query streams. We apply Φ(·) to supports and queries in the same form. For class c, let Sc be its support set and s(i) inst be the caption embedding of support xi ∈ Sc. We average support caption embeddings to form a class-level semantic context:

We then use the MAP-estimated prototype ˜ pc as the visual anchor to obtain the enhanced prototype:

For a query image xq with visual feature zq = fθ(xq) and caption embedding sqinst , we compute

Each query is processed independently, so the overall framework remains strictly inductive.

4.3 Training ObjectiveWith Consistency Regularization

In this section, we present the objective we trained in our framework. We freeze fθ and gϕ and train only the enhancer Φ together with an auxiliary classifier on base classes. Let v(i) out be the enhanced feature of a training sample xi with label yi.

Classification and prototype-preserving regularization. We use a standard cross-entropy loss

and a lightweight prototype-preserving term that discourages excessive drift from the frozen backbone structure:

We use a small λrec so that the enhancer can still introduce caption-guided adaptation while keeping base-class geometry stable.

Caption consistency via a contrastive loss. Caption quality may vary, so we add a consistency regularization on caption semantics in the projected space. For sample xi, let s(i) proj ∈ Rdv be the projected caption embedding (Eq. (12)). We normalize it as

We encourage captions from the same class to be coherent while maintaining inter-class separation via a supervised contrastive loss [Khosla et al., 2020]:

where si,a = cos (vcap i , vcap a ) denotes the cosine similarity between the normalized projected caption embeddings of samples i and a. Here, P(i) = {p ̸= i | yp = yi} is the positive set of sample i in the mini-batch, and τc is the temperature.

Overall objective. The final objective is

with λrec and λcon tuned on the validation set. During metatest, fθ, gϕ and Φ are all frozen, and classification is performed with logistic regression using enhanced support and query features.

5 Experiments

In this section, we present the datasets and implementation details in the experiments, followed by experimental results, ablation, and visualization analysis.

5.1 Datasets

To evaluate the performance of proposed method, we conduct experiments on MiniImageNet [Vinyals et al., 2016] , TieredImageNet [Ren et al., 2018], CIFAR-FS [Bertinetto et al., 2019], and FC100 [Oreshkin et al., 2018]. They cover both ImageNet-style datasets with richer visual details and CIFAR-style datasets with lower resolution.

MiniImageNet contains 100 classes with 600 images per class. We follow the standard split of 64, 16, and 20 classes for training, validation and test, respectively.

TieredImageNet is a larger subset of ImageNet with 608 classes grouped into 34 higher-level categories. We use the standard split with 351, 97, and 160 classes for training, validation, and test, respectively.

CIFAR-FS is created by randomly splitting 100 classes of CIFAR-100 into 64 training, 16 validation and 20 test classes with image size of 32 ×32.

FC100 is another CIFAR-100 derived benchmark with 100 classes grouped into 20 super-classes. We follow the standard split of 60, 20, and 20 classes for training, validation, and test, respectively.

5.2 Implementation Details

Following [Zhao et al., 2024], we use pre-trained ResNet-12 and Swin-T as visual backbones. We generate captions with a frozen BLIP [Li et al., 2022] captioner and encode both class names and captions using the frozen ViT-B/16 CLIP text encoder [Radford et al., 2021], obtaining 512-d sentence embeddings. We intentionally adopt BLIP to keep the captioner modest in size, while stronger captioners (e.g., BLIP2) can be used as drop-in replacements. The enhancer Φ is a projection block (Linear+LayerNorm+ReLU) followed by multi-head cross-attention, with a residual scale β initialized to 0.1. We use h = 4 heads and dropout 0; the key dimension is dk = 160 for ResNet-12 (dv = 640) and dk = 192 for Swin-T (dv = 768). All backbones and BLIP/CLIP modules remain frozen, and inference is strictly inductive.

We optimize only Φ and a shallow auxiliary classifier for 50 epochs using Adam (lr 10-4 , batch size 128) with StepLR (step 30, gamma 0.1). At test time, we evaluate 5-way 1-shot and 5-way 5-shot tasks over 600 episodes and report mean accuracy with 95% confidence intervals; logistic regression is used by default, and nearest-prototype yields similar trends. Unless otherwise specified, we set α = 0.33 (1-shot) and α = 0.7 (5-shot) in Eq. (3), use top-k retrieval with k = 7 and temperature τ = 1.0, and set τc = 0.1, λrec = 1.0, and λcon = 1.0 for training. We use class-name embeddings for retrieval since they are stable and class-consistent, while captions can be noisier on low-resolution images. We also experimented with token-level caption features, but they lead to significantly larger intermediate files and storage overhead, so we report sentence-embedding results by default.

5.3 ComparisonWith State-of-the-Art Methods

To evaluate the performance of our method, we make a comprehensive comparison with the state-of-the-art methods. These include metric-based methods MatchNet [Vinyals et al., 2016], ProtoNet [Snell et al., 2017], RelationNet [Sung et al., 2018], FGFL [Cheng et al., 2023], CPEA [Hao et al., 2023], MCNet [Chen et al., 2024], Cads [Zhang et al., 2025], and SRE-ProtoNet* [Chen et al., 2025b]; distributionbased methods ImprovedMAP [Wu and Hu, 2022], TDO [Liu et al., 2023], PBML [Fu et al., 2025], and DDC [Chen et al., 2025a]; semantic-based methods KTN [Peng et al., 2019], AM3 [Xing et al., 2019], BMI [Li and Wang, 2023], MAVSI [Zhao et al., 2024], and PRSN [Dong et al., 2025]; and Vision-language based methods SP-CLIP [Chen et al., 2023b], SP-SBERT [Chen et al., 2023b], SP-Glove [Chen et al., 2023b], SemFew-Trans [Zhang et al., 2024], and SYNTRANS [Tang et al., 2025]. Results on MiniImageNet and TieredImageNet are reported in Tab. 1, and results on CIFARFS and FC100 in Tab. 2. The results of these compared methods are quoted from their original work or relevant references.

Tab. 1 and Tab. 2 show the main comparison on four benchmarks with different lines of state-of-the-art methods. On MiniImageNet, PMCE achieves the best performance under both backbones. With Swin-T, PMCE reaches 85.03% and 92.77%, improving over SemFew-Trans by 7.71% and 7.26%, respectively. On TieredImageNet, PMCE also gives clear gains on the 1-shot setting and is consistently stronger in 5-shot. On CIFAR-style benchmarks, PMCE remains strong, it achieves the best 1-shot accuracy, which is 2.79 higher than PRSN, while it is slightly lower in 5-shot. On FC100, PMCE sets the best results with 52.46% and 67.00%, improving over PRSN by 9.5% and 6.43%, respectively.

We believe the improvements mainly come from two design choices. First, semantics-guided prior retrieval provides a class-specific prior that stabilizes prototype estimation when supports are extremely limited, which is especially important in 1-shot. Second, PMCE optimizes the query side with label-free captions under the inductive protocol, so support and query features are better aligned before matching.

5.4 Ablation Study

To test whether our performance is solely brought by MAP or Enhancer, we evaluate their importance separately. Tab. 3 shows the results of our method when trained with or without each component. Starting from the visual baseline, adding only MAP estimation already brings a 5.51% gain on MiniImageNet, showing that semantics-guided priors effectively correct prototype bias. Caption enhancement on either the support side (Enh.(S)) or the query side (Enh.(Q)) yields further improvements, and enabling both sides jointly brings clear gains on both benchmarks. With all components activated, PMCE improves the baseline by up to 19.73% on MiniImageNet and 15.88% on FC100. Therefore, both MAP estimation and Enhancer for both supports and queries play important roles in our method.

5.5 Impact of Hyperparameters

We study the sensitivity to α in Eq. (3) and the neighbor count k. Fig. 3a and Fig. 3b show that the optimal α lies around 0.3-0.4 in the 1-shot case and around 0.7 in the 5-shot case, which matches the intuition that the prior should be stronger when supports are scarce. Fig. 3c and Fig. 3d indicate that the accuracy is stable for a wide range of k, suggesting that PMCE is robust to the exact number of retrieved base classes.

5.6 Visualization Analysis

In this section, we present visualization results of both t-SNE and Grad-CAM on MiniImageNet.

Prototype evolution. Fig. 4 illustrates how prototypes and features evolve on a 5-way 1-shot episode from MiniImageNet. Initial prototypes lie near cluster boundaries and are far from the ground-truth centers, indicating strong bias under very few supports. After MAP estimation they move closer to the centers, and caption-guided enhancement further tightens and separates the clusters; the final prototypes almost coincide with the ground-truth centers. Overall, the visualization suggests that PMCE turns scarce supports into stable, semantically anchored prototypes while improving the separability of the surrounding query features.

Effect of caption guidance. We further visualize GradCAM activation maps for several MiniImageNet episodes. The comparison between w/o caption and w/ caption (Fig. 5) shows that caption-guided enhancement suppresses background responses and concentrates attention on the target object for both supports and queries. This happens because instance captions act as semantic cues, highlighting key visual parts that global features often miss.

6 Conclusion

In this paper, we propose PMCE, a probabilistic few-shot framework that couples semantics-guided prior retrieval with caption-guided dual-stream enhancement. By calibrating prototypes with class-specific priors and enhancing both supports and queries using label-free captions under the inductive protocol, PMCE effectively incorporates multi-granularity semantic knowledge into the adaptation process and consistently improves few-shot classification. Experiments on four benchmarks with ResNet-12 and Swin-T validate the effectiveness of PMCE, with the largest gains observed in 1-shot settings.

Acknowledgements

This work was supported by the Hong Kong RGC General Research Fund under Grant Nos. 15221123, 15216424, and 15211525, and the Hong Kong PolyU Internal Research Fund under Grant Nos. P0058468 and P0056171.