Rethinking Multimodal Few-Shot 3D Point Cloud Segmentation: From Fused Refinement to Decoupled Arbitration

1 Introduction

3D point cloud analysis is following the development of Label-efficient learning [Xiao et al., 2024; Chen et al., 2023] and the advent of Large Vision-Language Models (VLMs) and 3D-LLMs [Hong et al., 2023] has triggered a revolution in 3D scene understanding [Xu et al., 2024; Xue et al., 2024; Guo et al., 2023; Hong et al., 2023;

∗Corresponding author

Liang et al., 2024]. Through establishing a connection between point clouds and human language, some methods have broken the recognition bottleneck imposed by the closed label sets of traditional supervised learning [Qi et al., 2017a; Wang et al., 2019; Wu et al., 2024b]. Meanwhile, the paradigm shift towards leveraging foundation models is exemplified by Point-Bind [Guo et al., 2023] and ULIP-2 [Xue et al., 2024; Xue et al., 2023] which are two examples of aggregated frameworks that have successfully created a shared embedding space for 3D and human language. For instance, OpenScene [Peng et al., 2023], OV3D [Jiang et al., 2024], RegionPLC [Yang et al., 2024], UniM-OV3D [He et al., 2024], Open3DIS [Nguyen et al., 2024], and PLA [Ding et al., 2023] all demonstrate solid zero-shot generalization by building upon these semantic priors. Despite these advances, FewShot Learning (FSL) [Snell et al., 2017; Zhao et al., 2021; Hong et al., 2022] is still a critical challenge due to annotated scarce data.

In the realm of MM-FS-PCS, our key insight is that MMFSS [An et al., 2025b] is the pioneering SOTA that validated the utility of VLM priors. However, we've found that MM-FSS suffers from a Plasticity-Stability Dilemma which creates a Gradient Domination. This is caused by CLIP's category confusion [Li et al., 2025; Zhu et al., 2023; Tu et al., 2023; Qiu et al., 2024], which leads to semantic hallucinations in geometrically ambiguous regions (e.g., confusing a white wall with a picture).

Specifically, CLIP (Contrastive Language-Image Pretraining) produces semantic features that have been trained at a large scale. Its vector norms are typically large and stable. MM-FSS chooses to freeze the parameters of CLIP (IF Head) to preserve semantic features, leveraging the VLM, aligning with the trend of parameter-efficient fine-tuning [Tang et al., 2024; Zhang et al., 2022]. This leads to the problem that the model will discover that "I can achieve a high score simply by copying CLIP's semantic outputs." Thereby causing the 3D encoder to learn from 2D features in a simplistic manner, resulting in feature representations with relatively smaller norms, with semantic features dominating gradient flow.

Motivated by these important observations, a pertinent question arises: How can we limit gradient conflict and prevent category confusion? In this paper, the idea comes from the philosophy of Logits DeConfusion [Li et al., 2025], which physically separates confusion noise through residual learning. So we suggest a new paradigm "Decoupled Arbitration" to resolve this problem. We believe that geometric adaptation and semantic preservation need pathways that are completely separate from each other. Interaction should be limited to two laws: Soft Regularization which guide the geometric expert without messing up its input and Late Arbitration which only combines decisions after the decoupled decisions have been independently refined. Specifically, we introduce a novel model, Decoupled-experts Arbitration Few-Shot SegNet (DA-FSS), to leverage multimodal information more efficiently. Concretely, we develop an Adaptive Geometric Expert and a Static Semantic Expert to form the Decoupled-experts. The following Stacked Arbitration Module (SAM) performs convolutional fusion of the independently refined features from both pathways, building upon utilizing semantic guidance from textual information. Additionally, we propose Decoupled Alignment Module (DAM) enforcing the geometric expert to align with the semantic expert at the representation level so as to prevent the decoupling from causing each path to deviate excessively.

We systematically compare our DA-FSS against existing methods AttMPTI [Zhao et al., 2021], QGE [Ning et al., 2023], COSeg [An et al., 2024], and MM-FSS [An et al., 2025b] on S3DIS [Armeni et al., 2016] and ScanNet [Dai et al., 2017] datasets 4.2, suggesting the superiority of DA-FSS across various settings.

The contributions of our work are threefold: (i) We design a novel model, DA-FSS, to more effectively exploit information from different modalities. To the best of our knowledge, this is the first work to explore decoupling in MM-FSS. (ii) Learning from the baseline's novel cost-free multimodal FSPCS design philosophy, our DA-FSS efficiently reuses the pre-trained alignment already completed by the baseline to achieve optimization without any extra parameter overhead. (iii) Extensive experimental results demonstrate the value of our proposed framework and its consistent efficacy across different few-shot settings. This work will provide a new perspective for other multimodal fusion approaches.

2 RelatedWork

2.1 Few-Shot 3D Point Cloud Segmentation

Although fully-supervised methods have achieved promising results in 3D point cloud segmentation [Qi et al., 2017a; Qi et al., 2017b; Wang et al., 2019; Lai et al., 2022; Wu et al., 2024b; Yu et al., 2022], the expensive annotation costs have shifted research attention toward few-shot learning. As a pioneer, AttMPTI [Zhao et al., 2021] proposed a transductive label propagation method. Later works mainly concentrate on narrowing the semantic gap [Zhang et al., 2023; Ning et al., 2023; Ning et al., 2023; Xiong et al., 2024] and improving feature representation [An et al., 2024]. Specifically, AttMPTI [Zhao et al., 2021] and COSeg [An et al., 2024] optimize prototype correlations to reduce intraclass variance. In contrast, SCAT [Zhang et al., 2023] and QGE [Ning et al., 2023] apply advanced cross-attention mechanisms for fine-grained alignment between support and query features. However, recent observations indicate that these methods rely mainly on geometric features, which limits their generalization in complex open-world scenes (e.g., distinguishing objects with similar geometry).

2.2 Multimodal 3D Point Cloud Segmentation

With the rapid development of unimodal baselines, researchers are increasingly exploring multimodality for further performance gains. Language is currently the most popular auxiliary modality. Unified frameworks [Guo et al., 2023; Xue et al., 2024; Xu et al., 2024; Zhou et al., 2024] align 3D point clouds with image-text spaces to boost performance . Furthermore, open-vocabulary methods (e.g., OV3D [Jiang et al., 2024], RegionPLC [Yang et al., 2024], Open3DIS [Nguyen et al., 2024], UniM-OV3D [He et al., 2024], PointTFA [Wu et al., 2024a]) utilize VLM guidance to achieve open-set segmentation. Nevertheless, these approaches are tailored primarily for fully supervised or zeroshot settings. In the context of FS-PCS, existing works still predominantly use unimodal input. Recently, MM-FSS [An et al., 2025b] and Generalized FS-PCS [An et al., 2025a] attempt to leverage multimodal information. Critically, these methods follow a "Fused Refinement" paradigm. We observe that the aggressive fusion in these works leads to Gradient Domination. To the best of our knowledge, we are the first to introduce decoupled expert arbitration [Li et al., 2025; Zhang et al., 2022; Zhao et al., 2022] into multimodal FSPCS.

3 Methodology

3.1 Problem Setup

Multimodal FS-PCS. Following prior works [An et al., 2024; An et al., 2025b], we formulate this task as the popular episodic paradigm. Each N-way K-shot episode comprises a support set S = {Xn,k s , Y n,k s } and a query set Q = {Xq, Yq}, where X and Y denote the point cloud and corresponding labels, respectively. The goal is to segment query samples Xq into N target classes and 'background' by leveraging knowledge from S. Following the multimodal setting, our DA-FSS introduces textual and 2D modalities.

3.2 Motivation

Current SOTA methods [An et al., 2025b] typically adopt an early fusion strategy, obtaining a joint representation H = σ(W · [Fgeo ⊕Fsem]). Although this is a simple and intuitive approach, we argue that this paradigm creates a fundamental optimization conflict : during meta-training, the gradient flow to the geometric encoder called UF Head [An et al., 2025b] is always coupled with the semantic branch called IF Head [An et al., 2025b]:

We identify two issues in this interaction: (1) The static semantic features Fsem (from frozen VLMs) typically exhibit significantly larger norms than the adaptive geometric features, numerically overwhelming the sparse geometric gradients. (2) Frozen VLMs suffer from inherent texture bias [Li et al., 2025; Tu et al., 2023]. In geometrically ambiguous regions, the optimization shortcut provided by texture semantics suppresses the learning of structural details.

Consequently, the UF Head fails to adapt effectively, a phenomenon we term Plasticity Decay:

We provide concrete evidence of this phenomenon in Figure 1. By monitoring the training dynamics, we observe that the baseline's geometric gradients (Orange) are strictly suppressed by the semantic branch, effectively rendering the backbone "lazy" so that the UF Head will only obey the VLM's guidance from the IF Head (Frozen) without learning to optimize itself. This observation motivates us to physically decouple the optimization pathways to restore the backbone's vitality.

3.3 Method Overview

The overall architecture of the proposed DA-FSS is depicted in Figure 2. Given support and query point clouds, we first generate two sets of dense correlations: intermodal correlations from the Intermodal Feature (IF) head and unimodal correlations from the Unimodal Feature (UF) head. Both intermodal and unimodal correlations are then forwarded to the Parallel Expert Refinement module to independently produce refined geometric and semantic features. Beyond independent refinement, we use the Decoupled Alignment Module (DAM) to bridge the modal gap. This module acts as a soft regularizer to align the experts without gradient interference. We then exploit useful boundary-injected guidance to synthesize the final decision in the Stacked Arbitration Module (SAM).

Existing MultiModal FS-PCS approaches [An et al., 2025b] typically have two training steps: a pretraining step for obtaining an effective feature extractor, and a metalearning step towards few-shot segmentation tasks. Our method follows this two-step training paradigm. First, we pretrain the backbone and IF head using 3D point clouds and 2D images. To more objectively demonstrate our improvement results, we chose to directly use the same method as MM-FSS does, or even download officially released pretrained weights directly. Second, we conduct meta-learning to train the model end-to-end while freezing the backbone and IF head to maintain stability. In the following, we elaborate on Parallel Experts, DAM, and SAM modules.

3.4 Parallel Experts Refinement

Inspired by Mixture-of-Experts (MoE) [Jacobs et al., 1991], we propose the Parallel Experts module, as illustrated in Figure 2 to preserve the unique characteristics of each modality. Following [An et al., 2025b], we first merge correlations to capture support-query interactions. The Unimodal Correlation C u ∈ RNq×Ns captures geometric similarity, while the Intermodal Correlation C i ∈ RNq×Ns captures semantic affinity using VLM projections. This enhanced comprehension promotes the transfer of knowledge from the support set to the query point cloud, thereby refining segmentation outcomes.MM-FSS chose to sum C u and C i , but we do not opt for this approach; conversely, we propose that:

Adaptive Geometric Expert (Tgeo). This expert is fully learnable and responsible for Plasticity by receiving C u , which focuses on capturing task-specific structural variations. To this end, we employ a standard Transformer Layer (incorporating MHSA) to capture the holistic feature tensor of Hgeo (192-dimensional):

Static Semantic Expert (Tsem). This expert is fully learnable and responsible for Stability by receiving C i , which leverages the frozen VLM priors. This is to provide a consistent semantic reference robust to few-shot noise. Although this expert's functions differ, we have chosen an identical design but Hsem (512-dimensional):

Crucially, by isolating these pathways (I(Rgeo;Rsem) ≈ 0 during refinement), we ensure the experts are optimized solely based on features unique to their respective responsibilities, thereby protecting UF head against semantic gradient domination.

3.5 Decoupled Alignment Module (DAM)

While the Parallel Experts module effectively isolates interference between modalities, the consistency would largely disappear due to decoupling, which could lead to feature divergence. However, we cannot forcibly intervene to rigidly align the features of the two modules. Therefore, we propose the DAM, as illustrated in Fig. 2. DAM integrates structural consensus from the semantic expert to regularize the geometric expert. Additionally, since raw semantic priors contain fixed confusion patterns [Li et al., 2025], DAM employs a stop-gradient strategy to filter out texture noise, accounting for the reliability gap between adaptive and frozen features.

Prototype Loss Regularization (PLR). Given learnable geometric prototypes Pu and frozen semantic prototypes Pi, since the semantic prototypes Pi originate from stable VLM priors, they provide informative guidance on the true semantic manifold. Therefore, we first compute the alignment distance between them to generate structural constraints LPLR. Specifically, we project Pu (192-dimensional) to the dimension of Pi (512-dimensional) via a linear layer Fproj and minimize:

where sg(·) denotes the stop-gradient operator. Note that this effectively uses the semantic prototype as a "Teacher" anchor, ensuring the geometric encoder learns the semantic structure without inheriting confusion gradients.

Decoupled Consistency Regularization (DCR). Next, to enforce consistency at the decision boundary, we minimize the divergence between the probability distributions of the two experts (pgeo, psem). Since confusion patterns are often manifested in the logits distribution, the regularization term is computed as follows:

Through this interaction, Tsem can also learn the specialized adaptation of the other branch, rather than solely optimizing itself. This is akin to how a "teacher" typically influences a student, yet the "student" can also subtly impact their own "teacher."

This DAM module fully leverages the stability of the semantic expert to guide the plasticity of the geometric expert, helping to maintain feature coherence. Note that it uses stopgradient operators to prevent the propagation of confusion noise, improving the effective collaboration of decoupled experts.

3.6 Stacked ArbitrationModule (SAM)

While the two experts can make predictions independently; without fusing and refining these two predictions, the multimodal system cannot fully realize its potential for crosschecking and compensating for deficiencies. To synthesize the final prediction from the refined experts, we propose the Stacked Arbitration Module (SAM). Unlike prior works that rely on simple concatenation, we opt for a Stacked Transformer Arbitration architecture to model global interactions between geometric and semantic evidences.

Feature Merging Layer. We first project the concatenated expert outputs into a unified space (192-dimensional). Consistent with our implementation, Batch Normalization (BN) is applied pre-fusion to calibrate the scale discrepancy between adaptive and static features, followed by a 1 × 1 convolution and ReLU activation:

Dual-Guidance Injection. We inject inductive biases strictly at the boundaries using a "Negative-Suppression, Positive-Enhancement" strategy:

• Input: Background Suppression (Gbase). At the input of every Transformer layer, we explicitly inject Base-Class Guidance (Gbase) to refine the background representation. Specifically, we concatenate Gbase with the background token Rbg to suppress easy negatives early, while leaving foreground tokens Rfg untouched:

This can lend the model a more hierarchical structure, enabling further refinement of the feature space.

• Output: Semantic Gating (Gq). At the final layer, we employ a multiplicative gating mechanism derived from the target class embedding (Gq). This acts as a semantic gate to enhance foreground precision by scaling the feature magnitude based on textual affinity:

where ⊙ denotes element-wise multiplication and σ is a scaling function.

This effectively mitigates issues of over-suppression or overamplification while also reducing the parameter overhead in deep stacks beyond two layers.

3.7 Optimization Objective

After the Stacked Arbitration Module (SAM), the synthesized feature tensor Rfinal is transformed into the final prediction ˆ Pq ∈ RNq×(Nway+1) by a decoder comprising a KPConv [Thomas et al., 2019] layer and an MLP classifier. The whole model is optimized end-to-end by minimizing a compound objective function. To ensure expert coordination without compromising individual plasticity, we integrate the segmentation task with decoupled alignment constraints:

where Lseg is the standard cross-entropy loss between ˆ Pq and the ground-truth Yq, and Lbase checks the predictions for the base class. This formulation ensures a cooperative optimization: vertical gradients from Lseg optimize the decision boundary, while horizontal gradients from the alignment terms maintain expert coordination.

4 Experiments

In this section, we conduct extensive experiments to validate our proposed Decoupled-experts Arbitration (DA-FSS) framework. We first introduce the experimental setup. Then, we compare our method with state-of-the-art (SOTA) approaches on two challenging benchmarks: S3DIS [Armeni et al., 2016] and ScanNet [Dai et al., 2017]. Finally, we perform detailed ablation studies to demonstrate the effectiveness of each key component in our architecture.

4.1 Experimental Setup

Datasets. We evaluate our method on two standard fewshot 3D point cloud segmentation benchmarks: S3DIS [Armeni et al., 2016] and ScanNet [Dai et al., 2017]. S3DIS contains 3D scans from 6 large-scale indoor areas. ScanNet consists of 1513 scanned indoor scenes. We adopt the official benchmark splits [Dai et al., 2017], which contain 20 semantic classes. Their semantic classes are split into 2 folds (S0 and S1 ) for cross-validation. Following prior works [An et al., 2025b; Zhao et al., 2021], we maintain a maximum of 20,480 points per block using a 0.02m grid size.

Evaluation Protocol. We follow the standard N-way Kshot episodic evaluation protocol. The primary evaluation metric is the mean Intersection-over-Union (mIoU) over all foreground classes. As in [An et al., 2025b; An et al., 2024], we report results on 1-way 1-shot, 1-way 5-shot, 2-way 1shot, and 2-way 5-shot settings where the evaluation sets consist of 1,000 episodes per class in the 1-way setting and 100 episodes per class combination in the 2-way setting.

Implementation Details. We adopt the same backbone (Stratified Transformer) [Lai et al., 2022] and feature head architecture as MM-FSS [An et al., 2025b] for fair comparison. Following the standard protocol [An et al., 2025b], we initialize the 3D backbone with pre-trained weights that use the AdamW optimizer, setting a weight decay of 0.01 and a learning rate of 0.006 during pretraining in 100 epochs. Isolating the source of improvement, we choose to download officially released pre-trained weights directly. During the meta-learning phase, the model is trained using the AdamW optimizer with an initial learning rate of 0.0001 and a weight decay of 0.01. For our DA-FSS architecture, we set the number of layers in the Stacked Arbitration Module (SAM) to N = 1 for S3DIS and N = 2 for ScanNet. For the Decoupled Alignment Module (DAM), the loss weights are set to λPLR = 0.001 and λDCR = 0.5 based on empirical tuning.

4.2 Comparison with State-of-the-Art

We compare our DA-FSS with previous state-of-the-art methods on S3DIS [Armeni et al., 2016] and ScanNet [Dai et al., 2017], as detailed in Table 1 and Table 2. To ensure a fair assessment, we reproduced the leading baseline MM-FSS [An et al., 2025b] (denoted as MM-FSS† ) using its officially released pre-trained weights under our strictly controlled evaluation protocol. In contrast, DA-FSS consistently outperforms the former state-of-the-art across all settings, demonstrating the improved stable generalization brought by decoupling, which enhances the utilization rate of information from pretrained weights.

Specifically, on the S3DIS dataset, DA-FSS records mIoU increases of +1.21% in the 1-way-1-shot setting-which relies heavily on intrinsic geometric generalization. Furthermore, in 1-way-1-shot setting on the ScanNet dataset, DA- FSS has also achieved +1.00% improvement over MM-FSS. Notably, DA-FSS surpasses the baseline by +1.33% in the 2way 5-shot setting (see Supp. Material for full tables) that our Decoupled Arbitration mechanism effectively disentangles inter-class semantics under high interference, while simultaneously enforcing intra-class prototype stability through alignment constraints.

Table 3: Comprehensive analysis on ScanNet (1-way 1-shot). We analyze (a) architectural effectiveness, (b) computational complexity, and (c) the specific impact of decoupling. Note: DA-FSS achieves a massive surge in mAcc.

Remarkably, beyond standard mIoU metrics, our DA-FSS demonstrates a substantial improvement in Mean Accuracy (mAcc) on ScanNet. In settings where mIoU gains appear moderate (e.g., ScanNet 1-way 1-shot), our mAcc often surges by approximately +10% compared to the baseline shown in Table 3(c).

Although in some cases ours (DA-FSS) may induce oversegmentation due to its tendency to preserve geometric completeness with high-variance class confusion (which penalizes IoU heavily), this very tendency actually aids the model in more accurately segmenting points that should have been correctly identified. We prioritize semantic completeness over boundary precision, arguing that successfully identifying the target outweighs the cost of minor over-segmentation in 1-way 1-shot setting.

4.3 Ablation Studies

In this section, we conduct a series of ablation studies. Unless specified otherwise, all ablation experiments are performed on the ScanNet dataset under the 1-way 1-shot setting, as this setting is most sensitive to feature quality and generalization capability.

Effectiveness of DA-FSS Architecture. Table 3(a) validates our core hypothesis: "Decoupling" is structurally superior to "Fusion." Comparing (A) and (B), we observe that mIoU increases by approximately +0.49% solely through the decoupled design, even when all other components are disabled. This confirms that simply removing the early fusion bottleneck brings gains by mitigating the plasticity-stability conflict. Comparing (B) and (C), we achieved a +0.51% improvement after enabling all modules. This proves that while decoupling is essential, the experts must be coordinated via regularization (our DAM) to prevent divergence and achieve optimal alignment thereby optimizing the utilization of information interaction across various modalities..

Complexity Analysis. Finally, Table 3(b) highlights the efficiency of our design. Our DA-FSS achieves a significant performance gain (+1.00% mIoU) over the MM-FSS baseline while actually reducing computational overhead (-0.30 GFLOPs) and parameters (-0.27 M). This confirms that our gains stem from the superior decoupled architecture, not from increased model capacity.

5 Conclusion

In this paper, we identify a critical conflict in FS-PCS: the "Plasticity-Stability Dilemma," which has undermined the validity of previous progress in fusion-based paradigms. To rectify this issue, we present DA-FSS designed to decouple geometric adaptation from semantic preservation to maximize its adaptability across modalities. DA-FSS combines Parallel Experts to effectively aggregate features, enriching the comprehension of novel concepts from both geometric and semantic perspectives, which are mutually important for recovering missing parts. Furthermore, to ensure expert coordination despite decoupling, we introduce the DAM, which mutually regularizes gradients via prototype alignment. Finally, we introduce the SAM to fuse dualpath features and dynamically synthesize final decisions using boundary-injected guidance. DA-FSS achieves significant improvements over existing methods across all settings, particularly in boosting point-wise accuracy (mAcc). Overall, our research provides valuable insights into the importance of Decoupled Arbitration in FS-PCS and suggests promising directions for future studies.