Rethinking Multi-Modal Point Cloud Completion: Query-Aware Gating Attention and Gramian Volume Alignment

1 Introduction

Point clouds serve as a fundamental representation in 3D vision , widely applied in autonomous driving [Geiger et al., 2012; Huang et al., 2020; Guo et al., 2026b], robotic manipulation, and digital twinning [Liang et al., 2023; Song et al., 2017]. However, real-world scans are inevitably constrained by self-occlusion [Du et al., 2024b; Liang et al., 2025b;

∗Corresponding author

Liang et al., 2026], inter-object occlusion, and sensor resolution. Consequently, observed point clouds are often sparse and incomplete [Tesema et al., 2024], making them insufficient for downstream tasks and necessitating robust algorithms to recover complete geometric structures from partial observations [Liang et al., 2025a]. Early single-modal completion methods relied solely on partial inputs for inference. However, they struggle to resolve topological ambiguities when information is severely deficient, often resulting in structural discontinuities or blurred details [Tchapmi et al., 2019; Huang et al., 2020; Xie et al., 2020; Li et al., 2023; Yu et al., 2021; Guo et al., 2024; Guo et al., 2026a].

To address the limitations of single-modal inference, recent research has shifted toward multi-modal point cloud completion, leveraging single-view RGB images to provide complementary appearance cues and object-level semantic priors. ViPC pioneered sequential fusion based on image reconstruction priors [Zhang et al., 2021]. XMFnet successfully fused visual texture and geometric structure using cross-modal attention mechanisms, significantly improving completion accuracy [Aiello et al., 2022]. EGIINet further optimized this process via explicit guided information interaction [Xu et al., 2024]. Building upon previous cross-modal fusion strategies, IAET recently introduced an interleaved attention mechanism designed to significantly enhance the bidirectional information flow between modalities [Fang et al., 2025].

Despite the promising prospects of multi-modal fusion, two key challenges remain unresolved. First, the potential of natural language remains largely underexplored. Unlike images, which provide dense pixel-level texture, language offers abstract, high-level semantic descriptions that are invariant to viewpoint changes and lighting conditions [Zhou et al., 2026]. Ignoring text limits the model's ability to capture the global semantic essence of objects. Second, since point clouds contain only partial geometric information while images and text depict the complete object form, directly fusing the three may lead to unstable matching in regions where geometric information is missing. Traditional CrossAttention mechanisms force every Query feature to find a corresponding Key in the auxiliary modality and assign probability weights [Vaswani et al., 2017]. When geometric features correspond to occluded or missing regions, this "Forced Alignment" causes the network to erroneously attend to irrelevant areas, reducing the effectiveness and controllability of cross-modal information [Fang et al., 2026; Fang et al., 2024; He et al., 2026].

To tackle these challenges, we propose Query-Aware Gating Attention and Gramian Volume Alignment (QGGA), explicitly modeling multi-modal point cloud completion as a problem involving both "Adaptive Modulation" and "Geometric Alignment." For active selection, we introduce a Query-Dependent Gate within the interleaved cross-modal interaction. This mechanism uses query-driven gating scores to modulate attention outputs, dynamically controlling the injection intensity of multi-modal information. When specific visual or textual cues lack constructiveness for the current partial geometry, the gate suppresses their influence, preventing noise propagation caused by invalid alignment.For geometric alignment, we introduce an alignment metric based on Gramian Volume. Consistency is characterized by the volume of the parallelotope spanned by multi-modal feature vectors; this volume is calculated via the determinant of the Gram matrix, where a smaller volume indicates better alignment. As shown in Figure 1, we impose a Gramian Volume alignment constraint at the local level by minimizing the Gramian volume of triplets formed by local features of point clouds, images, and text. Furthermore, we calculate the Gramian consistency of tri-modal features at the global level to derive a confidence score for scaling the fusion gate. When overall trimodal consistency is low, the system automatically reduces the intensity of cross-modal injection, making the completion process more robust. Additionally, we utilize a Large Language Model to automatically generate structured text descriptions for ShapeNet-ViPC images, introducing ShapeNetViPC-Desc. This enables QGGA to operate as a tri-modal system. In summary, our contributions are as follows:

• We introduce natural language descriptions into multimodal point cloud completion, enabling the model to simultaneously leverage visual appearance cues and highlevel textual semantics to assist in the geometric recovery of partial point clouds.
• We propose a fusion and alignment framework for trimodal completion. For fusion, we employ interleaved cross-modal interaction with a query-dependent gating mechanism to make information injection more controllable. For alignment, we introduce a local consistency constraint based on Gramian Volume, combined with
• global consistency confidence to regulate fusion intensity, explicitly enhancing the stability of cross-modal interaction.
• We construct the ShapeNet-ViPC-Desc dataset and conduct systematic experiments on this benchmark. Extensive experiments demonstrate that our method achieves state-of-the-art (SOTA) performance.

2 RelatedWork

2.1 Single-modal Point Cloud Completion

Early data-driven methods primarily relied on 3D self-priors to recover geometric structures from incomplete inputs. PCN [Yuan et al., 2018] pioneered a folding-based decoder but struggled to recover fine local details. To address this, SnowflakeNet [Xiang et al., 2023] introduced Snowflake Point Deconvolution, progressively revealing geometric details through a coarse-to-fine generation process. Entering the Transformer era, PoinTr [Yu et al., 2021] and its adaptive variant AdaPoinTr [Yu et al., 2023] modeled the completion task as a set-to-set translation problem, utilizing geometryaware attention mechanisms to capture long-range dependencies. Recently, SVDFormer [Zhu et al., 2023] leveraged SelfView Decomposition to understand global shape structure, while SymmCompletion [Yan et al., 2025] explicitly incorporated symmetry priors to guide high-fidelity recovery. However, single-modal methods are fundamentally limited by semantic ambiguity when facing severe occlusion, lacking external guidance to determine the true topology of missing regions [Xiu et al., 2025].

2.2 Multi-modal Point Cloud Completion

To eliminate geometric ambiguity, researchers introduced auxiliary image modalities to provide semantic guidance. ViPC [Zhang et al., 2021] established the view-guided completion paradigm but relied solely on simple global feature concatenation. Subsequent works like XMFnet [Aiello et al., 2022] and CSDN [Zhu et al., 2024] significantly improved performance by projecting modalities into a shared latent space for cross-attention fusion and style transfer. EGIINet [Xu et al., 2024] and CDPNet [Du et al., 2024a] further refined this interaction through explicit guidance and dual-stage mechanisms. Recent advances focus on alignment robustness: IAET [Fang et al., 2025] employs an interleaved attention mechanism to enable bidirectional information flow. Recently, GRAM [Cicchetti et al., 2025] proposed a novel alignment metric that enforces synchronous geometric consistency across n modalities in high-dimensional space by minimizing the Gramian volume of the feature parallelotope. In this work, we apply this theoretical insight to 3D completion by proposing QGGA. This method aims to enforce structural consistency among linguistic, visual, and geometric modalities, thereby effectively bridging the "partial-complete" gap.

3 Method

3.1 Overview

The proposed QGGA framework reconstructs complete and high-fidelity point clouds Pout from partial observations Pin by leveraging single-view images and text descriptions as complementary auxiliary modalities. As illustrated in Figure 2, the framework consists of three stages. First, in the multimodal fusion stage, PointNet++ extracts point tokens FP [Qi et al., 2017], while frozen CLIP encoders produce image tokens FI and text tokens FT [Radford et al., 2021]. These tokens are progressively fused by the Interleaved Gated Transformer (IGT), which performs controlled cross-modal interaction through Query-Dependent Attention (QDA). Second, in the Gramian alignment stage, we impose tri-modal consistency during training by minimizing the Gramian volume induced by aligned local feature groups. Third, a coarse-to-fine decoder progressively reconstructs dense complete geometry from the fused point features.

3.2 Multi-Modal Feature Fusion

In multi-modal point cloud completion, the input point cloud is partial, whereas the image and text modalities describe the object in a more complete manner. Direct fusion or standard cross-attention fusion may result in insufficient integration of useful information from various modalities. To address this, we propose the Interleaved Gated Transformer (IGT).

Query Dependent Attention. The core computational unit of IGT is Query Dependent Attention (QDA). Given a query matrix Q, a key matrix K, and a value matrix V , the standard attention is calculated as

We introduce a learnable gating mechanism based on the query Q at the output of the attention module, as shown in Figure 3. The gating coefficient g is derived via a linear projection and a Sigmoid activation function

where Wgate and bgate are learnable parameters. The final output of QDA is the element-wise product of the standard attention output and the gating coefficient

This gating operation Ygated explicitly controls the amount of cross-modal information allowed for each query token, thereby effectively suppressing the injection of erroneous information and enhancing the fusion robustness of multimodal completion.

Interleaved Gated Transformer. To fully exploit visual texture and textual semantics, we construct the fusion structure in an interleaved manner, as shown in Figure 3. Since both images and point clouds possess spatial structures, their interaction first establishes a robust geometric skeleton. Abstract textual semantics, which lack spatial coordinates, are subsequently injected to refine the visually enhanced features. First, we set Q = FI and K = V = FP to compute the updated image features F ′ I

Subsequently, the point cloud queries the image to inject reliable visual information into the geometric features. We set Q = FP and K = V = F ′ I to compute the updated point cloud features F ′P

Next, we perform similar operations to interact the visually enhanced point cloud tokens F ′P with the text tokens FT to integrate high-level semantics. We set Q = FT and K = V = F ′P to compute the updated text features F ′T

Finally, we set Q = F ′P and K = V = F ′T to compute the fully updated point cloud features F ′′ P

The final outputs of IGT are the fused point cloud features F ′′ P , image features F ′ I , and text features F ′T .

3.3 Gramian Alignment

Recent advances in multi-modal representation learning demonstrate a strong correlation between geometric alignment in high-dimensional space and semantic consistency. We introduce a novel local-global Gramian mechanism for point cloud completion. Unlike the original GRAM which focuses solely on global representation alignment, we extend this theory to fine-grained local feature alignment and global confidence gating, ensuring the completion process is guided by geometrically consistent multi-modal cues. As demonstrated by [Cicchetti et al., 2025], the degree of alignment among a set of k modality vectors V = {v1, . . . , vk} ∈ Rd can be quantified by the volume of the k-dimensional paralleltope they span. This volume is calculated via the determinant of their Gram matrix G, where Gij = ⟨vi, vj⟩

We apply L2 normalization to all feature vectors before constructing the Gram matrix to prevent trivial solutions where feature magnitudes collapse to zero. A smaller volume indicates stronger semantic correlation and alignment among the modality vectors. We leverage this geometric property to enforce consistency among the partial point cloud, the reference image, and the text description.

Soft Alignment and Top-k Filtering. Since there is naturally no one-to-one correspondence between point cloud tokens, image tokens, and text tokens, we must first establish a shared feature space where every local point cloud token possesses corresponding visual and textual representations before performing geometric alignment. To address this, we propose an Attention-Soft Alignment strategy. As shown in Figure 4, let F′′P = {f P 1 , . . . , f PNP } ∈ RNP×d , F′I = {f I 1 , . . . , f INI } ∈ RNI×d , and F′T = {f T 1 , . . . , f TNT } ∈ RNT×d denote the fused features of the point cloud, image, and text, respectively. We employ two parallel SoftAlign modules to aggregate auxiliary information into the point cloud domain. Specifically, we use the point cloud features F′′P as the Query (Q) and the auxiliary modalities as Keys (K) and Values (V )

Here, FPI and FPT represent point-wise features enriched by visual and textual contexts, while API and APT are the attention weights representing the attention degree between each point cloud token and the auxiliary tokens. For each point token f P i , its visual prototype ˆvi and textual prototype ˆti are calculated as

This step effectively projects the discrete feature spaces of images and texts into the geometric space of the point cloud, facilitating the construction of effective geometric triplets (f P i , ˆvi, ˆti) for volume calculation. Since not all triplets contribute meaningfully to the loss computation, we introduce a Top-k filtering strategy to select the K most critical triplets. We utilize the peak values of attention weights to measure the confidence of modal alignment. When a point cloud token attends to specific regions of the image and text with high weights, it implies not only more accurate semantic alignment but also higher multi-modal consistency; consequently, such tokens are considered highly reliable. Based on this, we define a reliability score si for each point cloud token

We select the index set K of the top-k points with the highest scores si.

Local GRAM. For the selected point i ∈ K, we utilize the Gram determinant to calculate the positive sample volume V (i) pos of the aligned triplet (f P i , ˆvi, ˆti). Minimizing V (i) pos alone might lead the model to concentrate attention meaninglessly to achieve a trivially small volume. To mitigate this, we adopt a Hinge Ranking Loss. We construct negative sample triplets by shuffling the aligned triplets to obtain V (i) neg , forcing the volume formed by matched tri-modal local features (Vpos) to be as small as possible, and the volume of mismatched features (Vneg) to be as large as possible. The local loss is defined as

where µ is a margin hyperparameter.

Global Confidence Gating. Complementing the finegrained local alignment, we employ a Global Gramian Gating mechanism to handle overall semantic conflicts. As shown in Figure 2, We extract global features (gI , gT ) for images and text via CLIP, and generate the global point cloud feature gP via average pooling of the point cloud tokens FP . We estimate the global confidence γ ∈ (0, 1] by calculating the Gramian volume Vglobal of the global features (gP , gI , gT ). A smaller volume indicates higher global consistency, yielding a higher confidence score

Next, we generate a local feature gate. We concatenate the point features with the aggregated auxiliary contexts to form Fcat = [F ′′ P ; FPI; FPT ]. This combined representation is passed through a gating network (MLP) to produce pointwise gates α ∈ [0, 1]NP ×d

Finally, we leverage both the learned local gate and the global confidence scalar to modulate feature injection. The updated point features ˆ F P are computed via a residual connection

3.4 Coarse-to-Fine Decoding

We employ a hierarchical coarse-to-fine decoder that takes only the fused point tokens ˆ FP as input, without additional image or text skip features. The decoding process starts with a coarse decoder, which regresses a sparse structural scaffold from the latent embeddings using a transposed-convolution module and a residual MLP. The coarse output is then fed into cascaded upsampling layers to progressively densify point coordinates and recover fine-grained surface details until the target resolution is reached. Existing multi-modal completion networks often inject auxiliary visual features into the decoder. In contrast, our decoder relies solely on the fused point tokens produced by the proposed fusion and alignment modules. This design highlights the effectiveness of Gramianguided fusion, suggesting that the fused point features encode sufficient cross-modal semantics for reconstruction and reduce the need for extra modality-specific skip features.

3.5 Loss Functions

We adopt Chamfer Distance (CD) as the primary reconstruction objective to measure the geometric discrepancy between two point sets. Let P be the predicted point cloud and G be the ground truth point cloud. The CD loss is defined as:

Since our decoder employs a "Coarse-to-Fine" multi-stage strategy comprising four resolution levels, the total objective function aggregates structural supervision from all stages and incorporates the Gramian alignment regularization term proposed in this paper:

where Pk and Gk denote the predicted and ground truth point clouds at stage k, respectively. Llocal is the reliability-aware Gramian alignment loss defined in Section 3.3, and λ is a balancing hyperparameter.

4 Experiments

4.1 Datasets

ShapeNet-ViPC. We evaluate our method on the widely used ShapeNet-ViPC benchmark [Zhang et al., 2021]. Derived from ShapeNet, this dataset contains 38,328 objects across 8 categories. Each object is rendered from 24 different viewpoints to generate partial views, totaling 919,872 samples. Following the official split, we use 80% of the data for training and 20% for testing.

4.2 Experimental Settings

Implementation Details. All experiments are conducted using the PyTorch framework on an NVIDIA H100 GPU. We use the Adam optimizer with an initial learning rate of 1×10-3 and weight decay of 1×10-4 . The batch size is set to 64. The learning rate decays by a factor of 0.7 at fixed milestones. Following the ViPC benchmark [Zhang et al., 2021], we set the input image height H and width W to 224. The input partial point cloud contains 2,048 points, which are downsampled via Farthest Point Sampling (FPS) to NP = 256 center point tokens before being fed into the encoder. The feature embedding dimension C for all modalities is uniformly set to 256. We employ a pre-trained and frozen CLIP (ViTB/16) model as the visual and text encoder [Radford et al., 2021], with the maximum text token sequence length set to 77. The core multi-modal fusion module consists of 4 Transformer blocks, each containing 8 attention heads. The decoder progressively restores the point count to 2,048 via three upsampling stages (factor of 2) for final evaluation. For the proposed Gramian alignment, based on ablation results, we set the balance weight λ = 0.05, the margin µ = 0.07, the τ = 0.1 and the top-k ratio to 25%.

Metrics. Following standard protocols, we employ L2 Chamfer Distance (CD) and F-Score as the primary evaluation metrics. CD measures the geometric fidelity between the predicted shape and the ground truth, while F-Score evaluates the recall and precision of the reconstruction.

4.3 Evaluation on ShapeNet-ViPC

We compare our method with some multi-modal completion methods, as shown in Table 1 and Table 2. Our method achieves superior performance in CD metrics across most categories. Notably, compared to the strong baseline IAET [Fang et al., 2025], our method significantly reduces the average CD, validating the effectiveness of Gramian-guided alignment in mitigating semantic ambiguity. Figure 5 visualizes reconstruction results on challenging samples. Due to modal misalignment, baseline methods often produce noisy artifacts or fail to recover fine structures. In contrast, our method generates clean, high-fidelity shapes while preserving fine-grained local details.

4.4 Ablation Study

To validate the effectiveness of each proposed contribution, we conducted ablation studies on the cabinet category of the ShapeNet-ViPC dataset, as shown in Table 3. First, replacing QDA with standard attention (w/o QDA) degraded the performance to 1.522 CD and 0.753 F-Score, demonstrating the importance of query-dependent gating for controlling crossmodal information injection. Second, removing the local Gramian alignment loss (w/o L-GRAM) further increased the CD to 1.571 and decreased the F-Score to 0.744, showing that reconstruction supervision alone is insufficient for reliable fine-grained cross-modal alignment. Furthermore, excluding global confidence gating (w/o Gram-Gate) raised the CD to 1.808 and lowered the F-Score to 0.723, confirming that global reliability estimation helps suppress inconsistent auxiliary information. Overall, the full QGGA model achieves the best performance on the cabinet category, demonstrating the effectiveness of combining query-dependent gated fusion, local Gramian alignment, and global confidence gating.

5 Conclusion

In this paper, we addressed the problem of multi-modal point cloud completion, leveraging image appearance cues and textual semantic priors to assist geometric recovery under incomplete point cloud observations. To tackle the heterogeneity in representation forms and granularity across modalities, as well as unstable cross-modal correspondences, we proposed QGGA. For fusion, we employed interleaved crossmodal interaction and introduced query-dependent gating to achieve controllable information injection at channel levels. For alignment, we constructed soft correspondences based on Cross-Attention weights, utilized Gramian volumes to constrain local consistency, and incorporated global consistency confidence to adaptively regulate fusion intensity. The fused and aligned features are fed into a coarse-to-fine decoder to progressively generate the complete point cloud. Finally, we established the ShapeNet-ViPC-Desc multi-modal benchmark with text descriptions, providing unified data support and an evaluation foundation for future research.

Acknowledgments

This work is supported by the National Natural Science Foundation of China (No.U25A202453, 62536006, 62376142, 62406180).