Visual Implicit Geometry Transformer for Autonomous Driving

1 Introduction

Accurate, metric geometric perception models are crucial for reliable and safe autonomous driving. Learning 3D-aware geometric representations from images and other sensor modalities has become central to progress in both autonomous driv-

Recent advances in multi-view reconstruction [Wang et al., 2024a; Lin et al., 2025] have enabled a new class of end-toend geometric models that directly estimate dense 3D point maps from uncalibrated image collections, implicitly recovering camera parameters. While these methods achieve strong performance on monocular and multi-view reconstruction in general domain, their application to autonomous driving scenarios remains challenging. Firstly, surround-view camera rigs commonly used in autonomous vehicles prioritize full 360◦ coverage with limited redundancy, resulting in reduced overlap between camera views and weakening the multi-view geometric assumptions underlying these approaches. Secondly, many recent multi-view reconstruction models do not explicitly enforce accurate metric scale, which is critical for autonomous driving applications. Thirdly, per-pixel depth and point predictions are typically aligned with individual images rather than the global scene, making it difficult to reconcile inconsistencies across views. In contrast, discretized 3D voxel representations that explicitly encode occupied and free space have become widely adopted in modern autonomous driving systems due to their direct compatibility with downstream tasks such as collision checking and motion planning.

However, voxel-based approaches often face scalability challenges, as they typically require dense and costly ground-truth annotations for training.

Occupancy fields provide an attractive alternative by modeling 3D scenes as continuous functions over space, rather than relying on explicit voxel grids or per-view depth maps. Such representations are compact, differentiable, resolutionand sensor-agnostic, and naturally align geometry from multiple views into a unified scene-centric coordinate frame, making them well suited for autonomous driving applications.

In this work, we introduce the Visual Implicit Geometry Transformer, which estimates a continuous 3D occupancy field directly from surround-view camera images. Our model learns a compact continuous bird's-eye-view (BEV) field that can be decoded into binary occupancy using a point-wise decoder. This field provides a flexible geometric representation of the scene and can be further transformed into downstream representations by querying point-wise occupancy probabilities and accumulating them through rendering. In contrast to previous approaches, we simplify the image-to-BEV transformation by making it calibration-free and fully data-driven. This design reduces inductive bias and improves the generality and scalability of the model. Finally, we adopt a scalable, sensor-agnostic, self-supervised training procedure that leverages synchronized but uncalibrated multi-view images and LiDAR data without requiring additional annotations.

2 RelatedWork

General geometric models. Foundational geometric models have been an active area of research for several decades. Learning-based monocular depth estimation was shown to generalize better when scaled with large and diverse datasets [Ranftl et al., 2022; Patakin et al., 2022]. In particular, pixel-aligned depth prediction models trained with uncalibrated [Ranftl et al., 2022; Patakin et al., 2022], or metric supervision [Bhat et al., 2023] were shown to produce geometry estimates that generalize effectively to in-the-wild scenarios. However, monocular depth estimation remains fundamentally ill-posed due to depth ambiguity, which limits generalization when relying on single-view inputs [Yin et al., 2023]. To address this limitation, recent works proposed transformer-based architectures [Wang et al., 2024a; Leroy et al., 2024; Zhang et al., 2025] that predict dense point maps instead of per-pixel depth. By leveraging multi-view supervision during training, these models reconstruct imagealigned visible geometry [Wang et al., 2024a; Leroy et al., 2024; Lin et al., 2025] and, in some cases, estimate camera parameters from uncalibrated image collections [Wang et al., 2025a]. Subsequent approaches [Wang et al., 2025b; Wu et al., 2025] introduced latent memory embedding that can be rendered from novel viewpoints, albeit with a limited number of views. While these methods significantly improve scalability with respect to both training data and the number of input images, they still rely heavily on multi-view correspondences and therefore exhibit substantial performance degradation under low-overlap image collections [Li et al., 2025]. Moreover, most existing approaches regress scaleinvariant depth or point representations, which are insufficient for recovering 3D metric volumetric geometry required by downstream applications such as autonomous driving. In this work, we introduce a scalable transformer-based model that estimates a continuous 3D occupancy field at metric scale directly from surround-view images. Differently, our method is designed for surround-view driving scenarios with limited camera overlap, alleviating multi-view constraints.

Geometric models for autonomous driving. Prior geometric models for autonomous driving have predominantly focused on semantic occupancy estimation, as it provides a structured representation suitable for downstream tasks. Vision-based perception models in this domain build upon earlier works [Li et al., 2022; Philion and Fidler, 2020] that introduced forward [Li et al., 2022] and backward [Philion and Fidler, 2020] mappings from images to bird's-eyeview (BEV) space, where subsequent estimations are performed. More recent methods [Marinello et al., 2025; Pan et al., 2024; Huang et al., 2024; Tang et al., 2024] typically lift multi-view image features into voxel grids using explicit geometric projections [Li et al., 2022; Philion and Fidler, 2020], followed by refinement through 3D convolutional or transformer-based fusion [Zhou and Kr¨ahenb¨uhl, 2022]. Alternative approaches [Liu et al., 2023b; Zhang et al., 2024] incorporate explicit LiDAR encoders to improve prediction accuracy. Despite their effectiveness, these architectures rely on precise camera calibration, which limits generalization to novel camera configurations or scenarios with calibration errors. Additionally, they require expensive annotations, such as voxel-level semantic labels, which constrains scalability. In contrast, we introduce the simple transformerbased model architecture that does not use camera calibration for BEV construction. Our model learns a continuous 3D occupancy field in BEV representation that incorporates depth, point clouds, and occupancy from images, enabling scalable and transferable 3D perception across diverse selfdriving datasets.

Recent advances [Agro et al., 2024; Boulch et al., 2023; Yang et al., 2024] have introduced continuous occupancy forecasting in both space and time, leveraging pretext geometric tasks to support downstream applications. Early methods [Boulch et al., 2023; Agro et al., 2024; Khurana et al., 2023] operate on LiDAR point clouds and employ convolutional architectures to predict binary occupancy at the point level. Subsequent works [Agro et al., 2024; Liu et al., 2024] extend this paradigm by incorporating temporal modeling. Similarly, visual point cloud forecasting methods, such as ViDAR [Yang et al., 2024], predict future LiDAR-like point clouds from monocular image sequences rather than estimating occupancy directly. However, the majority of these models remain biased by calibration-dependent architectures, limiting their ability to generalize across diverse sensor configurations and environments. We leverage continuous occupancy estimation task for training our models since it provides annotation-free supervision from raw data.

3 Method

In this work, we introduce the Visual Implicit Geometry Transformer (ViGT) that estimates continuous 3D occupancy field directly from surround-view camera images. The remainder of this section is organized as follows: section 3.1 presents ViGT architecture that incorporates scalable blocks without inductive biases; section 3.2 describes unsupervised occupancy estimation problem as the scalable task for training; finally, section 3.3 details the label-free training procedure enabling learning continuous 3D occupancy fields endto-end.

3.1 Visual Implicit Geometry Transformer

We step towards general foundational geometric models for autonomous driving by designing our model architecture, prioritizing scalability, simplicity, and generalization across diverse sensor configurations. Our architecture consists of three main components (Fig. 2): an image encoder that independently processes each image, a calibration-free implicit camera-to-BEV projection module that unifies independent image features in a scene-centric BEV representation and a query-based implicit decoder that predicts occupancy values for 3D points.

Image Encoder. Following principles of simplicity and scalability, we employ a ViT-Large backbone [Dosovitskiy et al., 2021] to extract visual features from camera images. In contrast to prior autonomous driving models that rely on convolutional neural networks with limited receptive fields [Li et al., 2022; Philion and Fidler, 2020], transformer architectures capture long-range contextual dependencies through self-attention while imposing minimal inductive bias. Each image is encoded independently by the ViT-Large model, producing a compact set of token embeddings that serve as input to subsequent processing stages.

Implicit BEV projection. Autonomous driving applications require scene-centric geometric representations defined at metric scale within a unified coordinate frame. Unlike image-aligned representations, such as depth or point maps, scene-centric representations enable direct reasoning about spatial occupancy, which is critical for autonomous driving tasks. To this end, we adopt a bird's-eye-view (BEV) representation, as it provides comprehensive scene coverage while being more computationally and memory efficient than dense 3D volumetric grids. Existing self-driving approaches [Li et al., 2022; Philion and Fidler, 2020] rely on known camera intrinsics and extrinsics to explicitly project image features into BEV space. Guided by principles of simplicity and scalability, we seek a calibration-free transformation from multiview visual features to BEV space that can be applied across diverse camera setups.

We design our implicit camera-to-BEV projection module to learn the geometric transformation from data (Figure 3, Figure 5). Specifically, the implicit camera-to-BEV projection employs two sequential cross-attention blocks between image patches and BEV latent queries. The number of BEV latent queries is smaller than the number of image patch tokens, addressing computationally efficiency. To capture multiscale geometric information, we project outputs of the four last layers of ViT-L encoder independently to BEV space, using our implicit BEV projection with nonshared weights and individual BEV queries. The resulting multi-layer BEV representations are then aggregated and upscaled using DPT [Ranftl et al., 2021] to obtain the final fine-grained scene-level BEV representation. We validate our design choices through ablation studies comparing different projection structures and encoder layer selections (Table 3).

Implicit decoder. To transform the discrete BEV grid into a continuous occupancy field, we employ an implicit decoder inspired by ImplicitIO [Agro et al., 2023] that maps 3D query points to occupancy probabilities. We linearly interpolate query features and concatenate them with normalized query points. The concatenated features are decoded using Convolutional Occupancy Network [Peng et al., 2020] to predict the occupancy probabilities for the query point.

3.2 Unsupervised Occupancy Estimation

To build a scalable training framework, we leverage a training objective that relies exclusively on data that can be collected efficiently on real-world autonomous platforms. Specifically, we assume the availability of synchronized multi-view images and corresponding LiDAR point clouds. Unlike conventional occupancy estimation approaches that operate on discretized voxel grids, we avoid explicit spatial discretization and instead formulate the task in continuous space, operating directly on points. LiDAR is a high-precision active sensor that emits laser pulses and measures the return time from reflecting surfaces to estimate distances. By construction, LiDAR measurements guarantee that the space between the sensor origin and the first surface intersection along a ray is free. Accordingly, we construct two sets of training points sampled along each LiDAR ray. Points located before the reflection point are labeled as free space and thus assigned negative occupancy labels, while points near reflection point correspond to solid surfaces and are assigned positive occupancy labels. More formally, given a normalized LiDAR ray parameterized by origin o and direction d, we define two occupancy classes as follows:

where d is the distance from the ray origin o to the reflected surface and τ is the thickness hyperparameter that regulates the minimal surface depth along rays. This value is usually small for models that should perceive thin surfaces.

Overall, the training objective is formulated as a binary occupancy classification problem, where the model predicts the occupancy label of each queried 3D point given the available input context. Prior works [Boulch et al., 2023; Agro et al., 2024] primarily rely on single or multiple LiDAR point clouds as input, resulting in LiDAR-only models. In contrast, we adopt a vision-only paradigm and leverage this training objective to supervise the Visual Implicit Geometry Transformer.

3.3 Training

Following the unsupervised occupancy estimation formulation in Section 3.2, we sample query points along LiDAR rays to construct positives (occupied space) and negatives (free space) occupancy labels. We employ a balanced sampling strategy, that ensures equal representation of negative and positive targets. We divide the ray segment [0, d) into K bins and sample uniformly within each bin, ensuring coverage across the entire free space. However, simple uniform sampling for negative queries can lead to inefficient supervision near object surfaces, where accurate boundary predictions are crucial. To this end, we augment this with symmetric negative sampling from the region [d - τ, d) where τ is the same hyperparameter used for positive sampling, better utilizing object borders and improving surface localization. This symmetric approach samples negative points near the surface boundary, providing explicit supervision for the model to learn object borders. We validate our query sampling strategy through ablation studies comparing random, stratified, and stratified with symmetric interval sampling (Table 3).

We train the model using binary cross-entropy loss. For a batch of query points {pi}Ni=1 with ground truth occupancy labels {yi ∈ {0, 1}}Ni=1 and predicted occupancy probabilities {oi ∈ [0, 1]}Ni=1 , the loss is defined as:

3.4 Rendering

Our model estimates a continuous occupancy field, which provides a flexible geometric representation that can be transformed into various downstream representations by querying point-wise occupancy probabilities and accumulating them through volume rendering. This unified representation enables conversion to point clouds and occupancy voxel grids, facilitating application to diverse autonomous driving tasks while maintaining a single, consistent geometric model.

Ray-based rendering. To extract point clouds, we leverage volumetric integration to render LiDAR rays. Given the ray with origin o and direction d, we sample points

{pi = o + tid}Ni=1 , where ti are evenly spaced sampling distances. We decode occupancy probability o(pi) for every point and integrate the depth value accordingly:

where Ti is the transmittance accumulated from predicted occupancy.

Voxel grid rendering. To generate binary voxel grids from continuous occupancy field, we query M points randomly sampled within each voxel and aggregate their occupancy predictions. For a voxel V the occupancy is computed as the maximum response across all predictions:

4 Experiments

4.1 Training details

We train the model with the AdamW optimizer for 200K iterations, using a cosine learning rate scheduler with a peak learning rate of 5 × 10-5 and a warmup of 10K iterations. To ensure diverse training data, we use balanced sampling across five autonomous driving datasets: NuScenes [Caesar et al., 2020], ONCE [Mao et al., 2021], Waymo [Sun et al., 2020], Argoverse 2 [Wilson et al., 2021], and NuPlan [Caesar et al., 2021]. For each batch, we sample equally from all datasets to prevent bias towards larger datasets. We use a batch size of 6 per GPU. The input images are resized with the shortest side set to 192 pixels. We use 1024 BEV queries with 1024 channels in each Implicit BEV projector, aggregating features to 256×256 resolution with 256 channels after DPT. The training runs on 128 A100 GPUs over five days. We employ gradient norm clipping with a threshold of 1.0 to ensure training stability.

4.2 Implicit BEV projection

We demonstrate that our calibration-free implicit BEV projection correctly learns geometric transformations without explicit camera parameters. Figure 3 illustrates attention responses between BEV queries and camera tokens, showing consistent correspondences in both forward (camera-to-BEV) and inverse (BEV-to-camera) attention directions. Figure 5 demonstrates that the projection correctly aligns image features to their corresponding spatial sectors in BEV space for different camera rigs (6 cameras on NuScenes and 7 cameras on Argoverse 2) from different datasets, with each camera's field of view accurately mapped to its correct frustum region. These figures empirically confirm that the implicit projection successfully learns geometrically correct camerato-BEV transformations in a calibration-free manner across diverse datasets with different camera rig configurations.

4.3 Occupancy estimation

The first experiment compares our model with recent geometric models designed for autonomous driving applications.

Evaluation protocol. We evaluate all models on the Occ3D-NuScenes benchmark [Tian et al., 2023] which comprises 150 validation scenes with voxelized occupancy annotations defined within a region of interest of [-40m,-40m,-1m] × [+40m,+40m,+5.4m] in ego-vehicle coordinate frame with voxel resolution [0.4m, 0.4m, 0.4m]. Unlike the original benchmark, which provides semantic occupancy annotations across 17 categories, we focus exclusively on geometric occupancy estimation. To this end, all semantic classes are mapped to a single occupied label, reducing the task to binary occupancy prediction. We evaluate occupancy estimation using two metrics: F1-score and Intersection over Union (IoU). Both metrics are computed over the binary occupancy predictions (occupied vs. free) within the camera's visible region.

Baselines. We compare our method against recent visionbased occupancy prediction approaches as shown in Table 1. All models are grouped by their supervision type. Most approaches employ dense voxel grids with explicit 3D occupancy annotations: FB-Occ [Li et al., 2023], PanoOcc [Wang et al., 2024b], Sparse-Occ [Liu et al., 2023a], and OffsetOcc [Marinello et al., 2025]. Methods with reduced supervision include RenderOcc [Pan et al., 2024], which trains with 2D supervision, and Self-Occ [Huang et al., 2024], which learns occupancy in a self-supervised manner without voxel labels, similar to our approach.

Experimental results. We evaluate our model without voxel finetuning to demonstrate that our learned representation can generalize to new geometric tasks. Our model achieves third-best overall performance with an F1-score of 0.7115 and IoU of 0.5658, while being the only approach that is both fully self-supervised and calibration-free. Unlike top-performing methods such as PanoOcc and FB-Occ that require expensive 3D annotations, our method uses only raw LiDAR point clouds for supervision, eliminating the need for costly manual labeling. Compared to the previous best selfsupervised method (Self-Occ), we achieve improvements of 0.056 in F1-score and 0.07 in IoU, while additionally removing the calibration requirement.

4.4 Pointmap Estimation

Second, we evaluate our model against general-domain vision-based methods that are similarly calibration-free but produce pixel-aligned predictions.

Evaluation protocol. We evaluate all methods on the public test splits of widely used autonomous driving datasets, including NuScenes, Argoverse 2, Waymo, ONCE, and NuPlan, using LiDAR point clouds as ground-truth targets. Due to the large scale of nuPlan, we subsample the official test split by selecting every 100th LiDAR frame for evaluation. We report two evaluation metrics: Absolute Relative Error (AbsRel↓), computed along LiDAR rays, and Chamfer Distance (CD↓). For pixel-aligned methods, predictions are sampled at image pixels corresponding to the projections of LiDAR points onto the image plane. To provide an aggregate measure of overall performance across all datasets and metrics, we compute the Average Rank (Avg Rank↓) for each method as the mean of performance places (1st, 2nd, 3rd,

4th, etc.) assigned to each metric-dataset combination, where lower rank values indicate better overall performance.

Baselines Table 2 presents a comprehensive comparison of our method against state-of-the-art approaches. The evaluated methods can be categorized into two groups: (1) general geometry methods that produce image-aligned representations, and (2) self-driving domain methods that estimate 3D scene representation. For general geometry methods (VGGT [Wang et al., 2025a], DUSt3R [Wang et al., 2024a], Mast3R [Leroy et al., 2024], Monst3R [Zhang et al., 2025], Stream3R [Lan et al., 2025], Cut3R [Wang et al., 2025b], and DA3 [Lin et al., 2025]), we evaluate them using two variants: depth maps and pointmaps, both filtered by LiDAR ray masks to extract valid 3D points for comparison. Note that we use metric-scale DA3 that only produces depth maps, so we evaluate it using the depth map variant only. For scene-centric methods including ours and RenderOcc [Pan et al., 2024], we render depth by LiDAR rays using Eq. (3).

Experimental results. As shown in Table 2, our method achieves the best average rank (1.8) across all metrics and datasets. Figure 6 provides a visual comparison of the top3 methods by Average Rank. We outperform all competing methods in both AbsRel and CD metrics on NuScenes and Argoverse 2. On NuScenes, we achieve AbsRel of 0.068 and CD of 1.807, representing improvements of 0.105 in AbsRel and 1.83 in CD over the second-best method (Stream3R with depth-maps). On Argoverse 2, we achieve AbsRel of 0.131, outperforming the best competitor by 0.031 (Stream3R with depth-maps); and we achieve CD of 2.965, outperforming the best competitor by 0.716 (Cut3R with depth-maps). On Waymo, we achieve the second-best CD performance (2.431), trailing the best method by only 0.05. On ONCE and NuPlan, we achieve the best AbsRel performance, with third-best and second-best CD performance respectively.

Compared to RenderOcc, the most similar autonomous driving model, we consistently outperform it across all evaluated metrics. These results demonstrate the effectiveness of our calibration-free, self-supervised approach in accurately estimating 3D geometry from camera rig images across diverse autonomous driving datasets.

4.5 Ablation Studies

We evaluate three main components: the implicit BEV projector architecture, encoder output layer selection, and query sampling strategy (see Table 3.). All ablation experiments are trained on NuScenes only, using 4 GPUs for 100k iterations.

BEV Projector Architecture We compare three variants of implicit BEV projection module: a single cross-attention block (CA), two sequential cross-attention blocks (CA + CA), and a variant with self-attention (CA + SA + CA). The CA + CA configuration achieves the best performance, demonstrating that two sequential cross-attention operations provide sufficient capacity to learn the geometric transformation without the added complexity of self-attention.

Encoder Layer Selection We evaluate which ViT-L encoder layers to project to BEV space for DPT aggregation and upsampling. We compare using the last four layers versus default ViT-L output layers (5, 11, 16, 23). The last four layers selection shows better performance.

Query Sampling Strategy We ablate the strategy for sampling query points along LiDAR rays for occupancy supervision. We compare random sampling, stratified bin-based sampling, and stratified sampling augmented with symmetric interval sampling near object boundaries. The stratified + symmetric interval strategy performs best, confirming that explicit supervision near object boundaries is crucial for accurate occupancy estimation.

5 Conclusion

In this paper, we propose ViGT, a scalable vision-based geometric model for autonomous driving that learns a continuous 3D occupancy field directly from uncalibrated multicamera images. ViGT is trained jointly on five large-scale datasets using a self-supervised learning procedure based on synchronized image-LiDAR pairs. We show that the resulting unified representation can be directly rendered for multiple downstream tasks, including pointmap estimation and occupancy prediction, without task-specific retraining. Extensive evaluations demonstrate that ViGT generalizes effectively, achieving state-of-the-art performance on pointmap estimation across five datasets with diverse camera rig configurations while remaining competitive with supervised methods on voxel-based occupancy benchmark. We consider such models representing a promising research direction toward building foundational geometric perception systems for autonomous driving applications.

Acknowledgements

This work was supported by the The Ministry of Economic Development of the Russian Federation in accordance with the subsidy agreement (agreement identifier 000000C313925P4H0002; grant No 139-15-2025-012).