Learning with Semantic Priors: Stabilizing Point-Supervised Infrared Small Target Detection via Hierarchical Knowledge Distillation

1 Introduction

Single-frame Infrared Small Target Detection (ISTD) is a vital technique in infrared search and tracking systems, owing to its superior capabilities of passive detection and allweather operation. The goal is to detect faint signals within intricate settings characterized by severe background clutter and real-world degradations [Liu et al., 2025; Wang et al., 2024; Wang et al., 2023], with broad applications in early warning and maritime search and rescue. Despite the success of fully supervised methods, their scalability is severely limited by the prohibitive cost of annotation. Consequently, point supervision, which requires only one coordinate annotation per target, has emerged as a challenging direction.

∗Corresponding author.

In the early stage, methods were dominated by handcrafted priors. Representative techniques, such as filter-based methods (e.g., Top-Hat [Bai and Zhou, 2010]) and Human Visual System (HVS)-based methods (e.g., LCM [Chen et al., 2014]), utilize local contrast mechanisms to suppress background clutter. However, these methods rely on the adjustments of hyper-parameters and specific assumptions, limiting generalization in complex scenarios [Liu et al., 2023a]. Recently, with the paradigm shift toward data-driven semantic segmentation, deep learning-based methods [Wang et al., 2025; Wang et al., 2026] have dominated the ISTD field. To preserve the signatures of tiny targets from corrupted features, diverse mechanisms are proposed: multi-scale feature fusion (e.g., UIUNet [Wu et al., 2023b]) effectively integrates high-level semantics with low-level spatial details; contextual modulation schemes (e.g., ACM [Dai et al., 2021]) utilize local contrast and attention to highlight dim targets against complex backgrounds; and dense nested interactions (e.g., DNANet [Li et al., 2023]) enable repetitive feature enhancement to maintain signal strength. However, fully supervised annotation is labor-intensive, limiting scalability.

To address the annotation bottleneck, point-supervised methods have recently emerged as a promising direction. LESPS [Ying et al., 2023] pioneered this paradigm, introducing a label evolution strategy to iteratively update pseudomasks from point signals. But the unconstrained evolution in LESPS often suffers from optimization instability. To mitigate this, PAL [Yu et al., 2025] proposed an easy-to-hard active learning curriculum to stabilize the training trajectory. Although these methods are effective, a fundamental issue remains unaddressed: the semantic deficiency in lightweight ISTD networks. In the online label evolution framework, the generation of pseudo-labels heavily depends on the feature extraction capability. However, shallow features lack highlevel semantic discrimination, making it difficult to distinguish dim targets from high-frequency noise. This semantic gap leads to the generation of noisy pseudo-labels, causing training oscillation and performance degradation.

The emergence of Vision Foundation Model (VFM) [Radford et al., 2021; Kirillov et al., 2023; Oquab et al., 2023] has catalyzed a paradigm shift in ISTD, formally characterized as a foundation-driven paradigm. Capitalizing on this, SAMbased adaptations like IRSAM [Zhang et al., 2024] exploit tailored prompt encoders and self-prompting mechanisms to transfer open-world segmentation capabilities to the infrared domain. Concurrently, vision-language approaches such as SAIST [Zhang et al., 2025] leverage CLIP to align textual semantics with visual features for enhanced discrimination ability. However, despite their superior representation power, these methods inevitably incur a prohibitive inference burden due to their reliance on heavy, over-parameterized backbones. This heavy-encoder design results in redundant computation and high latency, fundamentally hindering their deployment on resource-constrained edge devices for real-time use.

Driven by these observations, a critical research question arises: How can we leverage the robust semantic priors of VFMs to rectify the feature insufficiency of point supervision, while preserving the inference efficiency? This objective presents a dual challenge. On one hand, the network can effectively distill the rich semantic representation and generalization capability from frozen VFMs to bridge the semantic gap of point-supervised networks, thereby stabilizing the label evolution process. On the other hand, it can achieve a complete decoupling of the heavy parameters from the inference pipeline. Furthermore, direct knowledge distillation straightforwardly minimizes the discrepancy between students and teachers on the training set. However, we argue that this naive alignment creates a generalization bottleneck: the student tends to overfit to the training domain bias.

To address the instability and generalization bottlenecks in online point supervision, we propose a hierarchical VFMdriven knowledge distillation framework. By reformulating the learning paradigm as a hierarchical optimization process, we enable the model to dynamically optimize for representation fitting (lower level) and generalization feedback (upper level). In detail, through explicitly optimizing for performance on an unseen validation distribution in the upper level, we generate VFM-guided feedback that prevents the student network from overfitting to the bias of training set. Furthermore, we design the semantic-conditioned affine modulation to inject robust object-centric semantics from frozen VFMs into the student network, rectifying the semantic deficiency of shallow features. Moreover, to combat overfitting risks, we introduce a dynamic collaborative learning strategy with cluster-level sample reweighting. This mechanism dynamically down-weights outliers while emphasizing informative hard samples, significantly enhancing both the robustness of label evolution and the detection performance in complex scenarios. Our contributions can be summarized as:

• We propose a VFM-driven knowledge distillation framework for point-supervised ISTD by reformulating online point supervision as a bilevel optimization problem. We prioritize validation-based generalization feedback, improving training stability under evolving pseudo-masks.
• To bridge the semantic gap between VFM priors and the fine-grained semantics required by tiny-target localization, we design the semantic-conditioned affine modulation, a lightweight interface that injects VFM semantics into the shallow ISTD network.
• A dynamic collaborative learning strategy is introduced to enhance the robustness of label evolution under noisy pseudo-masks and severe data imbalance. By clusterlevel sample reweighting, it automatically down-weights noisy pseudo-labels without manual tuning.
• Experiments on SIRST benchmarks demonstrate consistent improvements across multiple backbones, while keeping the student network efficient at inference.

2 The ProposedMethod

2.1 VFM-driven Distillation Framework

As mentioned above, pseudo-masks are inevitably noisy in the early phase of online evolution, and their evolution quality is strongly coupled with the feature extractor. Lightweight CNNs may entangle weak targets with high-frequency clutter, causing unstable mask updates and fragile representations. VFMs possess stronger target-centered semantics and better cross-domain robustness [Radford et al., 2021], making them suitable for addressing the semantic deficiency of point supervision. However, directly deploying VFMs is often impractical due to high inference costs and a mismatch with the semantic granularity of small targets. A naive alternative is to distill students on the training set, but this direct alignment may become a generalization bottleneck: under the induction of evolving pseudo-masks, students tend to imitate the teacher on the training distribution, overfitting to datasetspecific textures instead of inheriting transferable semantics.

Therefore, motivated by recent advances in bilevel optimization [Liu et al., 2023b; Liu et al., 2024c], we propose a hierarchical VFM knowledge distillation framework that uses a frozen VFM only during training and explicitly optimizes for generalization capabilities, which can be formulated as:

Specifically, we introduce the base ISTD network, denoted as Sθ, and a VFM-embedded teacher network, denoted as Tθ,ϕ, where ϕ denotes the teacher-side adaptation parameters, and α denotes the parameters of the sample reweighting function wα(·). The teacher shares the student weights θ but additionally equips a SCAMmodulator (reported in Sec. 2.2) parameterized by ϕ, which injects semantic information into student layers from the VFM. The inner objective Lin is to fit a representation on weighted training data and update teacher-side adaptation parameters ϕ, enabling it to generate more reliable soft labels and stabilize label evolution:

where Rgate = ∑ℓ ∥uℓ∥1 is set to encourage sparsity in the gating variables of semantic modulation uℓ, so that the teacher selectively injects VFM semantics only when beneficial, yielding a more compressible guidance signal for the CNN student. Here, pt and ps denote the teacher and student predictions, respectively. We set Ltask(p, ˜ y) = BCE(p, ˜ y). For knowledge distillation, we use the temperature-scaled sigmoid distributions qs = σ(zs/τ) and qt = σ(zt/τ) with τ = 4, and define Lkd as the mean-squared error between qs and qt. λin controls the inner KD strength and λgate weights the sparsity regularizer Rgate. We set λin to 0.1 and λgate to 5 × 10-3 . The outer objective Lout conducts generalizationdriven distillation on validation distributions, jointly optimizing the student and sample weights, so that knowledge transfer is more inclined to improve validation performance rather than merely matching the training set:

where λout controls the outer KD strength and is set to 1.0 in all experiments. We detach the student prediction in the inner-level KD and detach the teacher prediction in the outerlevel KD, since the teacher and student share θ.

2.2 Semantic-Conditioned Affine Modulation

The practical bottleneck of introducing a VFM into pointsupervised ISTD lies in the semantic granularity mismatch: Transformer token features have strong semantics but coarse spatial resolution, while infrared small targets rely on fragile local cues and multi-scale CNN priors. Existing fusion methods are either very heavy (e.g., cross-attention) or overly coarse (e.g., concatenation / global pooling), which can easily lead to blurring of small targets, unstable gradients under noisy pseudo-masks, and significantly increase training and inference overhead. Therefore, we aim to find a trainingphase mechanism to inject VFM semantics into CNN detectors in a lightweight and controllable manner. We propose Semantic-Conditioned Affine Modulation (SCAM), which converts the frozen VFM features into multi-layer affine parameters for modulating CNN backbone features. For an input image x ∈ RH×W , the frozen VFM outputs token embeddings. We denote the token features from the k-th selected transformer block as F(k) (x) ∈ RN×d , where N is the number of tokens, and d is the token embedding dimension. We then learn a soft fusion across depths fromK blocks:

where π = {πk}Kk=1 are learnable weights with ∑ πk = 1, and K is set to 12 to maximize the number of selected layers.

Inspired by DynamicViT [Rao et al., 2021], we introduce Token-aware Attention Pooling (TAP) to aggregate Fvfm into a global semantic vector g. Specifically, given Fvfm = [t1, . . . , tN] ∈ RN×d , TAP assigns an attention weight to each token via a lightweight scoring head:

where w is a learnable parameter vector and aj are normalized attention scores. Unlike global average pooling, this design better preserves tokens related to the target, facilitating semantic stability for small targets in complex backgrounds.

Subsequently, SCAM maps g to the affine parameters (γ, β) of each layer in the CNN. For the feature fℓ of the layer ℓ, a compact MLP generates (γℓ, βℓ) with inputs g, where γℓ, βℓ ∈ RCℓ represent the channel scale and bias. We introduce a channel gating vector uℓ to filter γℓ, enabling selective semantic injection and providing an interface for subsequent sparse regularization. To ensure stable training under online pseudo-masking, we adopt a residual form of scale transformation: γℓ ← 1 + tanh(γℓ ⊙ uℓ). Finally, after γℓ, βℓ are broadcast along spatial dimensions, we modulate the features through an affine transformation in a residual form:

where ˜fℓ are modulated CNN features.

Overall, the design adds a small modulator on top of frozen VFM and performs layer-by-layer affine operations on CNN features. Therefore, it has the following advantages: (i) computational overhead is much lower than that of heavy feature fusion; (ii) token scoring and lightweight modulation bring stability; (iii) it is plug-and-play for different CNN backbones, requiring only the acquisition of layer features and reuse of the original detection head.

2.3 Dynamic Collaborative Learning Strategy

ISTD datasets are extremely imbalanced in terms of target size, shape, SNR, and background complexity. If uniform sampling and loss averaging are adopted, the model is prone to overfitting simple samples and reducing robustness. Furthermore, dividing the training into a staged process (first evolving pseudo-labels, then distilling) fails to utilize validation feedback to correct which samples are more suitable for semantic transfer. We propose a dynamic collaborative learning strategy, which couples (i) cluster-level reweighting with (ii) unified bilevel optimization [Liu et al., 2026; Liu et al., 2024b] with Gauss-Newton hypergradient. Specifically, we compute hand-crafted priors (e.g., intensity statistics, texture complexity, spectral sharpness) for each image and cluster the training and validation samples. Each cluster c has a learnable logit αc, and the weight for each sample i is obtained through softmax within the batch:

Algorithm 1 Dynamic Collaborative Learning Strategy

Input:

Training set Dtr, validation set Dval, frozen VFM; student parameters θ, SCAM parameters ϕ, cluster logits α

Initialize: pseudo-masks ˜ y from point annotations

for each epoch do

Update pseudo-masks ˜ y using teacher prediction pt

Sample mini-batch Btr ∼ Dtr and compute weights wα

Inner step: ϕ ← ϕ - η∇ϕLin(Btr)

Sample mini-batch Bval ∼ Dval and compute weights wα

Outer step: update θ, α by ∇θ,αLout(Bval) using Eq. (8) end for

where B denotes the current mini-batch and c(i) is the cluster index of sample xi. Cluster-level learning avoids overfitting to a small number of simple samples and promotes balanced improvement across different levels of scenarios.

To enhance the efficiency of bilevel learning, we adopt a Gauss-Newton style hypergradient approximation based on gradient alignment. Specifically, the hypergradient for α is corrected by an implicit term induced by the inner update:

where η is the inner-step learning rate, ⟨·, ·⟩ denotes the inner product, and ϵ is a small constant for numerical stability. This Gauss-Newton correction avoids explicit Hessian inversion and maintains stability as the pseudo-mask continuously evolves. The whole algorithm is summarized in Alg. 1.

3 Experiments

3.1 Implementation Configurations

Datasets. The comprehensive dataset SIRST3 [Ying et al., 2023] consists of three sub-datasets SIRST-v1 [Dai et al., 2021], NUDT-SIRST [Li et al., 2023], and IRSTD-1k [Zhang et al., 2022]. Following the LESPS split protocol, all models are trained on SIRST3, with 10% of the samples held out for validation (1402/274/1079 for train/val/test).

We further report results on a test split into Salient, Filamentary, Faint, and Camouflaged, defined by target shape difficulty and target-to-background contrast. Specifically, Salient indicates regular and high-contrast targets, Filamentary indicates morphology-challenging but high-contrast targets, Faint indicates low-contrast but less complex targets, and Camouflaged represents the hardest cases with both challenging morphology and low contrast. This split enables a more interpretable, challenge-aware comparison.

Experimental settings. Unless otherwise specified, we train for 300 epochs with AdamW (batch size 16). The base learning rate is 1 × 10-3 for both inner and outer updates; for backbones that are less stable under weak supervision, we use 5 × 10-4 . We adopt DINOv3 [Sim´eoni et al., 2025] ViTS+/16 pretrained on LVD-1689M and keep the VFM backbone frozen throughout. We use PAL [Yu et al., 2025] as the default pseudo-mask evolution engine, while LESPS [Ying et al., 2023] is reported as a baseline. Bilevel updates are performed every 5 epochs: we sample mini-batches from the training and validation sets and run 4 Gauss-Newton hypergradient steps to update the ISTD CNN, SCAM parameters, and cluster weights α. We report IoU and nIoU for mask quality, Pd for detection probability (recall), and Fa for false alarm rate, following standard ISTD evaluation.

3.2 Experimental Results

Quantitative results. Table 1 reports results on SIRST3 over the overall test set and four characteristic partitions, including Salient, Filamentary, Faint, and Camouflaged, across seven backbones. Compared with single-point baselines such as LESPS and PAL, our method achieves consistent improvements across metrics, while the deployable student (Ours-S) remains close to the VFM-embedded teacher (Ours-T). PAL is a stronger baseline than LESPS but still suffers from missed targets and higher false alarms, especially on harder subsets, whereas our gains are most evident on Faint and Camouflaged samples, indicating improved robustness under distribution shift. For example, on DNANet, Ours-T improves overall IoU from 62.67 to 63.09 and reduces Fa from 17.59 to 16.18; on SCTransNet, Ours-T further boosts IoU from 65.95 to 69.08 with lower Fa, and achieves a large gain on Faint targets. Consistent improvements are observed across additional backbones, including ALCLNet, GGLNet, and MLCLNet, where both Ours-T and Ours-S improve IoU/nIoU and Pd while keeping Fa competitive. Overall, these results validate that our bilevel, generalization-driven knowledge distillation effectively transfers VFM priors into a lightweight CNN student for robust detection across diverse target characteristics.

Qualitative results. Fig. 3 presents qualitative comparisons produced by different methods under two backbones, DNANet and SCTransNet. Our approach yields consistently more complete and compact target responses with clearer boundaries in both the teacher and the student. Notably, even under extremely low contrast, tiny target size, or heavy background clutter, Ours-S remains capable of activating the true target region while effectively suppressing spurious background responses. In contrast, LESPS and PAL frequently suffer from missed detections, which are marked as Missed Target, or fragmented/shifted predictions across multiple cases. Despite discarding the VFM branch at inference, Ours-S remains close to Ours-T, indicating that our bilevel knowledge distillation effectively transfers VFM generalizable semantics into the lightweight student for robust ISTD.

3.3 Ablation Studies

Training Stability Analysis. ISTD is challenging due to extremely small targets and scarce annotations. Under pointsupervised label evolution, the pseudo-mask ˜ y is noisy and constantly changing, which often amplifies training instability and overfitting. Our VFM-embedded teacher injects robust global semantics via SCAM, while the bilevel optimization leans more towards transferable cues. Fig. 4 shows higher final IoU on both train and test, with smoother optimization and reduced overfitting. Overall, these observations validate our design goals: enhancing VFM-driven optimization, stabilizing training under incomplete supervision conditions, and distilling this stability into an ISTD CNN student.

Impact of VFM on CNN Representations. To visually demonstrate how VFM priors reshape ISTD detectors, we visualize the intermediate representations of the CNN encoder before and after SCAM modulation. Specifically, for three encoder layers, we calculate the channel mean intensity maps of the original features fℓ and modulated features ˜fℓ. Additionally, we also visualize the VFM patch grid score map generated by token-aware aggregation, which directly reflects the attention distribution of DINO patch tokens. Fig. 5 shows SCAM boosts target responses and suppresses clutter across layers, producing clearer spatial contrast. The VFM score maps often focus on subtle target cues that are easily overlooked by purely local CNN priors (e.g., faint or irregular targets). Therefore, VFM can provide complementary global guidance. This effect ultimately translates into more accurate and compact predicted shapes in difficult samples and improves overall detection performance.

Computational Overhead. We benchmark our framework on four backbones using a single NVIDIA A40 GPU with AMP enabled. We perform 20 warm-up iterations and report averaged latency over 100 measured iterations, repeated 3 times. The student is unchanged for inference, while the teacher is equipped with a frozen DINOv3 ViT-S+/16 backbone with about 29M parameters. As summarized in Table 2, the teacher introduces a moderate additional cost with a latency increase ranging from 1.04× to 1.63× and a peak memory increase from 1.10× to 1.33×, depending on the backbone. For SCTransNet, overhead adds 1.19× latency and 1.33× memory. On lightweight backbones (e.g., MLCLNet / ALCLNet), latency overhead becomes more visible but memory remains efficient. Importantly, since all teacher components are removed for deployment, our improvements come from the proposed training strategy without increasing inference-time complexity of the student model.

Efficiency of Proposed Mechanisms. Despite adding a frozen VFM branch and hierarchical training, the gains come from a few lightweight designs. Table 3 shows that removing VFM and SCAM decreases IoU to 63.06, suggesting pure CNNs lack semantic discrimination with evolving pseudomasks. The two knowledge distillation terms are complementary: removing either lowers IoU and raises Fa. The inner term Lkd(pt, ps) stabilizes teacher adaptation, while the outer term Lkd(ps, pt) distills validation semantics from Dval into the student. Removing the validation set causes one of the largest performance drops, showing validation feedback counters cross-scene bias; while disabling cluster-level reweighting reduces Pd and increases Fa, indicating it mitigates imbalance. Fig. 6 (a) shows performance saturates with 10%Dval and 4 outer steps; Fig.6 (b) indicates that the hidden layer width of the MLP in SCAM depends on the DINOv3 variant due to different semantic dimensions, so the overhead is controllable without sensitive tuning.

SCAM Design. In Table 4, we compared six VFM-CNN fusion strategies to better integrate VFM priors with ISTD CNN detectors. Replacing TAP with Global Average Pooling (GAP) or using a single VFM depth degrades performance; GAP increases Fa to 33.95. This suggests that selective token aggregation and multi-depth fusion are crucial in cluttered infrared backgrounds. For semantic injection, naive concatenation or single-layer modulation is inferior, while multi-layer affine modulation brings stronger and more stable gains, better aligning global semantics with fine-grained localization.

Removing the layer-wise gate reduces IoU to 66.18 and Pd to 93.69, implying that gated semantic control is important under evolving pseudo-masks ˜ y. Table 5 shows that camouflaged targets have lower Hnorm, smaller EffN, and larger pmax, i.e., attention becomes more selective under heavy clutter and irregular shapes. Overall, SCAM balances performance and robustness via token-aware multi-depth aggregation, multi-layer affine modulation, and gated injection, enabling layer-wise guidance for stable bilevel transfer.

4 Conclusion

This paper proposes a bilevel knowledge distillation framework guided by a VFM for point-supervised ISTD. During training, the VFM is frozen and its knowledge is distilled into a lightweight detector. SCAM and dynamic collaborative learning reduce the impact of noisy pseudo-masks and stabilize training. Experiments on SIRST3 show consistent improvements across different backbones and data splits.

Acknowledgments

This work is partially supported by the National Natural Science Foundation of China (Nos. 62306059, 624B2033), the Fundamental Research Funds for the Central Universities.