PointGP: Geometry-Primed Attention for Point Cloud Analysis

1 Introduction

The rapid evolution of 3D sensing technologies has established point clouds as a fundamental representation for digitizing the physical world. The increasing accessibility of high-quality 3D data has enabled a wide range of applications, including autonomous driving, robotic navigation, AR/VR, and large-scale reconstruction.

In response to this growing demand, extensive research has been devoted to developing learning architectures that can capture fine-grained geometry and semantic context in unstructured 3D spaces. Existing methods include convolutional neural networks (CNNs) [Thomas et al., 2019], graph neural networks (GNNs) [Shi and Rajkumar, 2020], attention-based models [Zhao et al., 2021], and recent Mamba-based frameworks [Bahri et al., 2025] that leverage state-space representations for efficient sequence modeling.

∗Corresponding authors.

Among these methods, Transformer-based architectures have shown strong effectiveness by modeling contextual dependencies through self-attention. However, most existing 3D Transformers still follow a feature-centric design paradigm, where attention weights are mainly derived from semantic feature similarity and geometric information is often introduced as auxiliary positional encoding. While this de-

similarity-based attention to point clouds may be suboptimal. Unlike text tokens or image patches, whose positions are defined on regular domains, point cloud coordinates are irregular, unordered, and closely tied to the underlying local shape structure. Therefore, local geometric relations provide informative structural cues that should play a more direct role in attention generation.

As illustrated in Fig. 1, our empirical analysis shows that geometric cues tend to receive higher gating weights than semantic features across Set Abstraction stages, including deeper stages. This observation suggests that local geometry can provide useful structural cues in settings where feature representations may become progressively smoothed, motivating a geometry-primed perspective for point cloud attention. Based on this observation, we propose PointGP, a geometry-primed framework for point cloud analysis. Instead of directly using semantic feature similarity to determine attention weights, PointGP generates attention weights primarily from geometry-aware structural cues.

Our contributions are summarized as follows:

• We revisit feature-centric attention in 3D point cloud Transformers and provide empirical observations showing that local geometric cues can serve as stable structural anchors for 3D attention generation.
• We propose PointGP, a geometry-primed attention framework that reduces reliance on explicit query-key feature matching and generates attention weights from geometry-aware structural cues, enabled by featureconditioned local coordinate rectification and attention logit computation from both raw and rectified geometric cues.
• Extensive experiments on ModelNet40 [Wu et al., 2015], ScanObjectNN [Uy et al., 2019], ShapeNetPart [Chang et al., 2015], S3DIS [Armeni et al., 2016], and ScanNet v2 [Dai et al., 2017] demonstrate that PointGP achieves competitive performance with strong parameter and computational efficiency compared with representative strong baselines.

2 RelatedWork

2.1 AttentionMechanisms in Point Clouds

Transformers have been widely explored for 3D point cloud understanding due to their ability to model contextual dependencies in unordered point sets. PCT [Guo et al., 2021] adapted global self-attention to point clouds with offsetattention, but its global design introduces quadratic complexity with respect to the number of points. To improve local feature aggregation, Point Transformer (PTv1) [Zhao et al., 2021] introduced vector attention within local k-nearest neighborhoods. PTv2 [Wu et al., 2022] further improved this design with Grouped Vector Attention (GVA) and partitionbased pooling, while PTv3 [Wu et al., 2024] scaled point Transformers using serialized representations based on spacefilling curves.

2.2 Adaptive Geometry and Position Awareness

Geometric modeling is a central problem in point cloud analysis. KPConv [Thomas et al., 2019] introduced kernel point convolution with deformable kernels to adapt receptive fields to local geometric structures. AdaptConv [Zhou et al., 2021] dynamically generates graph convolution kernels from point-pair features, improving the flexibility of local aggregation. Position-aware operators, such as relative position encoding [Wu et al., 2021] and point-specific positional encodings [Xiao et al., 2025], further demonstrate the importance of spatial information in local feature learning. In point Transformers, PTv2 [Wu et al., 2022] explicitly integrates relative position encodings into attention computation, strengthening spatial awareness in local neighborhoods. In addition, PointNet-style symmetric aggregation [Qi et al., 2017a] has been widely used to summarize local or global point-set context, providing a simple yet effective way to encode neighborhood-level structural information.

2.3 Discussion and Positioning of PointGP.

PointGP is closely related to point Transformers and adaptive geometric modeling methods, but differs in the way attention weights are generated. Most point Transformer variants derive attention weights primarily from semantic feature interactions, such as dot products, feature subtraction, or MLPbased comparisons, while geometry is typically incorporated as positional encoding or an additive spatial term. In contrast, PointGP uses semantic features to rectify local geometric coordinates before attention computation, allowing geometryaware structural cues to directly guide attention generation. In this way, PointGP shifts geometry from a supplementary positional term to a primary source for computing attention distributions, while still retaining semantic guidance through local geometric rectification.

3 Analysis: Geometry-Semantic Duality

In this section, we revisit feature-centric attention in point cloud Transformers and analyze why local geometry can provide complementary structural cues for attention generation. These analyses motivate a practical geometry-primed design, where semantic features modulate local geometric relations and attention weights are computed primarily from geometryaware cues.

3.1 Motivation: Limitations of Feature-Centric Attention

Many point Transformer variants compute attention weights from semantic feature interactions, such as dot products, subtraction, or MLP-based feature comparisons. This design is effective when point features remain sufficiently discriminative. However, in deep networks, repeated local aggregation may progressively smooth feature representations. This phenomenon is closely related to feature homogenization and rank collapse [Dong et al., 2021], and may make featuresimilarity-based attention less selective.

Let F(l) ∈ RN×C denote the point feature matrix at layer l, where f (l) i is the feature of point i. When features within a local neighborhood become overly similar, i.e.,

the attention logits computed from feature similarity also become less distinguishable. For a standard dot-product attention formulation,

similar neighboring features lead to logits with small relative variation. In the limiting case where sij ≈ S for all j ∈ N(i), the Softmax-normalized attention weights approach a uniform distribution:

This simplified analysis indicates that feature-centric attention may lose local selectivity when semantic features are excessively smoothed, making the aggregation behave similarly to uniform local averaging. For clarity, we refer to this tendency as attention collapse in this paper.

In contrast, relative coordinates

remain directly tied to the spatial arrangement of local neighborhoods. Although raw Euclidean geometry is not always sufficient to describe task-relevant semantic relations, it provides explicit structural cues that do not depend on feature similarity alone. This motivates a geometry-primed attention design, where local geometric relations serve as the primary input for attention-logit generation, while semantic features are used to guide geometric rectification.

3.2 Feature-Conditioned Local Coordinate Rectification

Raw Euclidean neighborhoods can be imperfect for point cloud analysis. For example, points that are close in Euclidean space may belong to different object parts or surfaces, while points on the same semantic structure may be separated by curvature, sparsity, occlusion, or sampling noise. Therefore, directly using raw coordinates may introduce a mismatch between Euclidean proximity and task-relevant local relations.

To describe this mismatch, let the point cloud P = {pi}Ni=1 ⊂ R3 be a discrete observation of an underlying surface or local structure. Practical point cloud encoders usually construct neighborhoods using the ambient Euclidean distance:

For curved, sparse, or noisy point clouds, this Euclidean neighborhood can deviate from the desired task-relevant neighborhood. We therefore introduce a learnable local coordinate rectification mechanism to adjust the geometric relation before attention generation.

Given a center point pi and one of its neighbors pj, we predict a local coordinate offset from their feature difference:

where Hθ is a learnable projection function. The rectified relative coordinate is then defined as

This formulation provides a task-driven local topology or metric adjustment: semantic feature differences guide the rectification of local geometric relations before attention generation. The resulting rectified coordinates remain in a compact geometric space suitable for efficient attention-logit computation.

Based on both raw and rectified geometric cues, PointGP computes geometry-aware attention logits:

where Ψθ denotes the geometric kernel for attention-logit generation. In this way, semantic information affects attention indirectly through coordinate rectification, while the final attention distribution is derived from geometry-aware structural cues. This design reduces reliance on explicit query-key feature matching and provides an efficient alternative for local point aggregation.

3.3 Empirical Observation: Geometry Remains Informative

To empirically examine the roles of semantic and geometric cues, we design a probe attention module with two decoupled streams. The semantic stream computes logits from feature interactions, while the geometric stream computes logits from relative coordinates. Their contributions are controlled by learnable scalar gates wfeat and wgeo, which are normalized by a softmax constraint:

The combined probe logit is computed as

The corresponding attention weight is then obtained by Softmax normalization:

This decoupled design allows us to examine the relative contributions of semantic and geometric cues under the same local aggregation framework.

As shown in Fig. 1, in this probe setting, the learned gates suggest that geometric cues tend to receive higher weights than semantic features across Set Abstraction stages on ModelNet40 [Wu et al., 2015]. This observation suggests that local geometry remains informative for attention generation, especially in settings where semantic representations may become progressively smoothed. Meanwhile, the non-negligible semantic gates suggest that semantic information remains complementary to geometry. These observations motivate the design of PointGP, where semantic features guide local coordinate rectification and attention weights are generated from both raw and rectified geometric cues.

4 Methodology

4.1 Overview

PointGP adopts a hierarchical encoder-decoder architecture for point cloud analysis. Given an input point cloud P ∈ RN×3 and associated point features F ∈ RN×C , the basic building block is the Local Aggregation module. The core of this module is the proposed Geometry-Primed Attention

(GPA) mechanism, which generates attention weights primarily from geometry-aware structural relations. Semantic features do not directly form query-key attention logits; instead, they influence attention generation through local coordinate rectification, as illustrated in Fig. 2.

4.2 Semantic-GuidedManifold Rectifier

Standard k-NN grouping is usually constructed in the raw Euclidean space, which may not always align well with taskrelevant local relations under semantic inconsistency, curvature, sparsity, occlusion, or sampling irregularity. To alleviate this mismatch, we introduce the Semantic-Guided Manifold Rectifier (SGMR), a learnable local topology/metric rectifier that predicts feature-conditioned coordinate offsets before attention-logit generation.

Let N(i) denote the k-nearest neighbors of center point pi in the current abstraction layer. For each neighbor pj , SGMR first computes the local feature difference:

which provides semantic-aware guidance for local geometric rectification. A lightweight MLP Φ(·) then maps this feature difference to a 3D coordinate offset:

where ΘΦ denotes learnable parameters.

Given the raw relative coordinate

the rectified relative coordinate is obtained as:

We initialize the last projection layer of Φ(·) with zeros, so SGMR starts from the raw Euclidean geometry and gradually learns task-adaptive coordinate rectification during training. The rectified coordinate ∆˜pij then serves as a featureconditioned geometric cue for subsequent attention-logit generation, helping the attention kernel better differentiate taskrelevant neighbors from spatially close but semantically inconsistent points without explicit query-key feature matching.

4.3 Dual-StreamGeometric Kernel

To generate geometry-aware attention logits, we introduce the Dual-Stream Geometric Kernel (DSGK). DSGK first concatenates the raw and rectified relative coordinates into a composite geometric descriptor:

where ⊕ denotes channel-wise concatenation. The raw coordinate preserves the original local spatial organization, while the rectified coordinate provides task-adaptive geometric refinement.

DSGK processes gij with two parallel streams. The local stream models fine-grained edge-level geometry using a shared MLP ψloc:

The contextual stream summarizes neighborhood-level geometric structure by applying a shared MLP ψctx followed by symmetric max pooling:

The context vector hctx i is broadcast back to each neighbor and fused with the local feature to produce vector attention logits:

where ψfuse is a fusion MLP and Aij ∈ RCv matches the channel dimension of the value feature.

Through this design, DSGK combines pairwise geometric details with compact neighborhood-level structural context for attention-logit generation. Although the contextual stream adopts PointNet-style symmetric pooling, it operates on geometric descriptors rather than directly aggregating point features.

4.4 Geometry-Primed Attention Aggregation

Given the attention logits generated by DSGK, GPA aggregates local features in a geometry-primed manner. Input features are first projected into value vectors:

where vj ∈ RCv . To preserve explicit spatial information in the aggregated content, we add value-side positional encoding from the raw relative coordinate:

where ζ(·) maps ∆pij to the value-channel dimension. Raw geometry is used for value encoding to retain physical spatial information, while rectified geometry guides attention-weight generation.

The vector attention weights are obtained by applying softmax normalization over the neighbor dimension:

where normalization is performed over j ∈ N(i) for each channel. In the multi-head implementation, value channels are split into multiple heads and normalized independently.

The output feature of point i is computed as:

where ⊙ denotes element-wise multiplication.

In GPA, attention weights are not directly computed from query-key feature similarity. Instead, semantic features influence attention through SGMR, while DSGK derives logits from raw and rectified geometric cues, making geometryaware structural relations the primary cues for local attention generation.

4.5 Computational Characteristics

Compared with feature-centric attention mechanisms based on high-dimensional query-key interactions, GPA generates attention logits from compact low-dimensional geometric descriptors. DSGK operates on the 6-D descriptor gij through shared MLPs, while SGMR adds only lightweight offset prediction from local feature differences. Thus, the main attention-logit generation cost comes from efficient geometric MLP operations rather than full pairwise feature-channel comparisons. This enables PointGP to preserve adaptive local aggregation with favorable parameter and computational efficiency.

5 Experiments

5.1 Experimental Setup

We evaluate PointGP on five widely used point cloud benchmarks covering three representative tasks. For shape classification, we use ModelNet40 [Wu et al., 2015] and ScanObjectNN [Uy et al., 2019]; for part segmentation, we use ShapeNetPart [Chang et al., 2015]; for indoor scene semantic segmentation, we use S3DIS [Armeni et al., 2016] and ScanNet v2 [Dai et al., 2017]. These datasets cover clean synthetic objects, noisy real-world objects, fine-grained partlevel segmentation, and large-scale indoor scene understanding, providing a comprehensive evaluation of the proposed geometry-primed attention design. The network configurations for different datasets are summarized in Table 1. The number of parameters is measured over all learnable weights, and FLOPs are measured with a batch size of 1 under the corresponding input resolution for each benchmark.

5.2 Shape Classification

ModelNet40. ModelNet40 [Wu et al., 2015] is a standard benchmark for 3D shape classification, containing 12,311 CAD models from 40 categories, with 9,843 samples for training and 2,468 samples for testing. As shown in Table 2, PointGP achieves 94.1% OA and 91.6% mAcc with 4.9M parameters. Compared with PTv2 [Wu et al., 2022], PointGP obtains comparable accuracy while using substantially fewer parameters. These results suggest that geometry-primed attention provides an effective and parameter-efficient design for clean synthetic shape classification.

ScanObjectNN. ScanObjectNN [Uy et al., 2019] is a challenging real-world benchmark containing noisy and incomplete point clouds with background clutter and occlusion. We evaluate PointGP on the hardest PB T50 RS split, which better reflects real-world object recognition scenarios than clean synthetic datasets. As shown in Table 3, PointGP achieves 90.3% OA and 89.6% mAcc with 4.9M parameters.

5.3 Part Segmentation

ShapeNetPart. ShapeNetPart [Chang et al., 2015] is a large-scale benchmark for fine-grained 3D part segmentation. It contains 16,881 shapes from 16 categories with 50 part labels. The task requires models to capture local geometric details and part boundaries, making it suitable for evaluating the effectiveness of local attention mechanisms. As shown in Table 4, PointGP achieves 87.3% instance-averaged mIoU and 85.3% class-averaged mIoU with 6.5M parameters. PointGP outperforms several representative point-based backbones while maintaining a moderate model size.

5.4 Semantic Segmentation

S3DIS Area 5. The S3DIS dataset [Armeni et al., 2016] contains 271 rooms across six areas with 13 semantic categories. We follow the standard Area 5 evaluation protocol, training on Areas 1-4 and 6 and testing on Area 5. This benchmark evaluates scene-level semantic understanding under complex indoor layouts and diverse object scales. As shown in Table 5, PointGP achieves 91.7% OA and 73.2% mIoU.

ScanNet v2. ScanNet v2 [Dai et al., 2017] is a large-scale indoor RGB-D dataset containing 1,513 reconstructed 3D scenes annotated with 20 semantic categories. We follow the standard evaluation protocol, using 1,201 scenes for training, 312 scenes for validation, and 100 hidden scenes for testing. Compared with object-level benchmarks, ScanNet v2 contains larger scenes, more diverse spatial layouts, and stronger variations in point density. As shown in Table 6, PointGP achieves 75.8% mIoU on the validation set and 75.4% mIoU on the hidden test set.

5.5 Ablation Studies

We conduct ablation studies on ModelNet40 to analyze the effectiveness and efficiency of the proposed design, as it provides a standard and efficient benchmark for comparing architectural variants under consistent training conditions. All variants follow the same network configuration and training protocol, and Overall Accuracy (OA) is reported as the primary metric.

Effect of Geometry-Primed Attention. To compare GPA with feature-centric attention mechanisms, we replace GPA with several representative attention modules, including standard scalar attention [Vaswani et al., 2017], PCT-style offset attention [Guo et al., 2021], PTv1-style vector attention [Zhao et al., 2021], and PTv2-style grouped vector attention [Wu et al., 2022]. As shown in Table 7, GPA achieves the best accuracy with fewer parameters and lower computational cost. Compared with grouped vector attention, GPA improves OA from 93.5% to 94.1%, while reducing parameters from 6.4M to 4.9M and FLOPs from 5.3G to 0.8G. This comparison suggests that deriving attention logits from geometry-aware cues provides an efficient alternative to direct feature-similarity-based attention.

Efficiency Analysis. Table 7 also shows that GPA substantially reduces computational cost. Compared with standard scalar attention, GPA reduces FLOPs from 2.9G to 0.8G. The efficiency gain mainly comes from avoiding explicit highdimensional query-key feature matching for attention-logit generation. Instead, GPA computes attention logits from low-dimensional geometric descriptors, while semantic information is introduced through feature-conditioned coordinate rectification. This design improves the accuracy-efficiency trade-off of local attention.

Component-wise Effectiveness. To evaluate the individual contributions of SGMR and DSGK, we construct a baseline that uses only raw relative coordinates and a single-stream MLP for attention-logit generation. As shown in Table 8, adding SGMR improves OA from 92.8% to 93.5%, showing that feature-conditioned coordinate rectification provides useful geometric cues. Adding DSGK improves OA to 93.6%, indicating the benefit of combining edge-level geometric details with neighborhood-level context. Combining both components achieves the best result of 94.1%, suggesting that SGMR and DSGK are complementary.

6 Conclusion and Limitations

In this paper, we presented PointGP, a geometry-primed framework for 3D point cloud analysis. Different from conventional feature-centric attention mechanisms that mainly rely on semantic feature similarity, PointGP derives attention weights from geometry-aware local relations. Specifically, the proposed Semantic-Guided Manifold Rectifier predicts feature-conditioned local coordinate offsets, while the DualStream Geometric Kernel computes attention logits from both raw and rectified geometric cues. In this formulation, semantic information guides local geometric rectification, and attention generation is driven by geometry-aware structural relations. Extensive experiments on classification and segmentation benchmarks demonstrate that PointGP achieves competitive performance with strong parameter efficiency.

Limitations. Despite its effectiveness, PointGP still has several limitations. First, the proposed design relies on local neighborhoods constructed by k-NN. When point clouds are extremely sparse, highly noisy, or dominated by outliers, the neighborhood structure may become unreliable, which can affect the quality of coordinate rectification and attention generation. Second, PointGP mainly emphasizes geometric relations. For scenarios where semantic differences depend heavily on non-geometric attributes such as color, texture, or intensity, geometric cues alone may not always be sufficient to distinguish ambiguous regions. Future work will explore multi-modal extensions that combine geometry with richer appearance or sensor cues for more robust point cloud understanding.

Acknowledgments

The paper is supported by the National Natural Science Foundation of China (No. 62072388), Fujian Provincial Science and Technology Major Project (No. 2024HZ022003), Jiangxi Provincial Natural Science Foundation Key Project (No. 20244BAB28039), Xiamen Public Technology Service Platform (No. 3502Z20231043), and Fujian Sunshine Charity Public Welfare Foundation.