Unified Sensor Simulation for Autonomous Driving

1 Introduction

Development of safe and reliable autonomous vehicles necessitates access to large-scale and diverse datasets for both training and evaluation. However, acquiring diverse real-world data remains prohibitively expensive and laborintensive. Simulation based on real data has emerged as a promising alternative, enabling the creation of augmented datasets in a cost and time efficient manner for various downstream applications [Yuan et al., 2024; Adamkiewicz et al., 2022; Ljungbergh et al., 2025]. Recently, 3D Gaussian Splatting (3DGS) [Kerbl et al., 2023; Wu et al., 2025; Hess et al., 2024] has introduced a photorealistic rendering engine lowering sim-to-real gap and offering an effective balance among visual fidelity and computational efficiency.

While existing simulators predominantly rely on EWA splatting [Zwicker et al., 2001], the recently proposed 3DGUT framework [Wu et al., 2025] offers benefits for sensor simulation in autonomous driving scenarios. Real-world driving data is typically captured using cameras with highly nonlinear and dynamic effects, such as rolling shutter and optical distortions. These effects are difficult to model accurately with EWA splatting due to its reliance on linearized particle projection during rasterization, resulting in specialized rendering procedures for each camera type. In contrast, 3DGUT projects Gaussians through arbitrary camera models using the Unscented Transform (UT), effectively emulating ray-based rendering within a rasterization framework and enabling accurate modeling of complex distortions. Building on this capability, we introduce a sensor simulation framework that adapts 3DGUT splatting to autonomous driving data. We further extend the framework with a generalized rolling-shutter modeling, enabling accurate simulation of LiDAR and arbitrary camera sensors in unified manner. However, we observe that UT-based projection for spherical cameras introduces challenges for Gaussian particles spanning the azimuth boundary. Naive processing of such particles leads to missing projections or distorted positions and shapes due to cyclic azimuth parameterization and temporal discontinuities at azimuth edges. To address this issue, we propose a phase modeling mechanism that explicitly accounts for these effects, enabling accurate rendering of spherical cameras with rolling shutter.

Autonomous driving scenarios require jointly modeling geometric and appearance sensors. Accurate appearance modeling typically necessitates multiple transparent Gaussian particles to capture complex lighting effects, whereas geometric sensors provide precise surface measurements that can be faithfully represented using a single opaque Gaussian. To address this distribution mismatch, we extend scene parameterization with two distinct opacity parameters per Gaussian, jointly optimized for color and geometry distributions. Such parameterization alleviates potential mismatches within a unified representation and results in enhanced quality.

Eventually, our main contributions are three-fold:

• We introduce XSIM - the sensor simulation framework for autonomous driving extending 3DGUT splatting and enabling rendering LiDAR and camera sensors in a unified manner with generalized rolling shutter modeling.
• We propose a phase modeling mechanism for spherical rasterization that explicitly accounts temporal and shape discontinuities of Gaussians projected by the Unscented

Transform at azimuth borders.

• We introduce an extended 3D Gaussian representation that incorporates two distinct opacity parameters per Gaussian, jointly optimized for color and geometry distributions, mitigating distribution mismatches within a unified representation.

2 RelatedWork

3DGS and NeRF for automotive scenes. In automotive applications Neural Radiance Field (NeRF) [Mildenhall et al., 2020; Barron et al., 2023; M¨uller et al., 2022; Wu et al., 2023; Xie et al., 2023] and 3D Gaussian Splatting (3DGS) [Kerbl et al., 2023; Wu et al., 2025; Hess et al., 2024; Zhang et al., 2024] gathered significant attention since it provide flexible tools for photorealistic sensor modeling. Initial research efforts [Ost et al., 2021; Fu et al., 2022; Kundu et al., 2022; Kerbl et al., 2023; Moenne-Loccoz et al., 2024] emphasized the reconstruction of scene semantic and appearance guided by RGB camera images. Later works enriched scene representation by decoupled modeling of dynamic actors and static layout leveraging either explicit scene graphs [Yang et al., 2023; Tonderski et al., 2024; Yang et al., 2024; Hess et al., 2024; Zhou et al., 2024c; Yan et al., 2024; Zhou et al., 2024b; Zhou et al., 2024a; Jiang et al., 2025] or time-dependent implicit representations [Go et al., 2025; Chen et al., 2023; Peng et al., 2025]. Recent approaches further extended prior methods to encompass accurate sensor simulations including modeling of complex camera distortions [Xie et al., 2023; Yang et al., 2023; Tonderski et al., 2024; Chen et al., 2025a; Zhang et al., 2024; Wu et al., 2025; Moenne-Loccoz et al., 2024] and LiDAR [Huang et al., 2023; Wu et al., 2024; Yang et al., 2023;

Tonderski et al., 2024; Chen et al., 2025a; Hess et al., 2024; Zhou et al., 2025; Chen et al., 2025b]. 3DGS provides a better trade-off between photorealism, physical accuracy and computational efficiency. However, the conventional 3DGS based on EWA [Zwicker et al., 2001] formulation introduces challenges for automotive scenarios limiting the ability to render accurately complex sensors [Huang et al., 2025]. The promising alternative 3DGUT [Wu et al., 2025] provides flexibility to render arbitrary cameras without strong approximation. In this work we introduced the sensor simulation framework based on 3DGUT [Wu et al., 2025]. In contrast to previous 3DGS-based methods [Hess et al., 2024; Chen et al., 2025b], our framework provides a unified rendering formulation for rolling-shutter camera and LiDAR sensors, resulting in improved cross-sensor consistency.

3 Method

We introduce XSIM - the sensor simulation framework for autonomous driving. Our framework is based on 3DGUT formulation that we overview in Section 3.1. In Section 3.2 we introduce extended 3D Gaussian representation to alleviate geometry and color distribution mismatches. Finally, we describe our rolling shutter sensor modeling approach along with the proposed phase mode mechanism in Section 3.3.

3.1 Gaussian Splatting Preliminary

Scene representation 3DGS [Kerbl et al., 2023; Zwicker et al., 2001; Wu et al., 2025] represents an arbitrary scene as a set of transparent Gaussian particles. Each particle is parameterized by its mean position µ ∈ R3 and a shape encoded by a covariance matrix Σ ∈ R3×3 . The contribution of a particle is defined by a Gaussian kernel function:

Covariances Σ = RSS⊤R⊤ in practice are represented and optimized via decoupled scaling vector s ∈ R3 and rotation quaternion q ∈ R4 . Each particle is associated with opacity value σ ∈ R, diffuse color cd ∈ R3 , and viewdependent appearance feature vector cs ∈ Rf . While original 3DGS [Kerbl et al., 2023] uses spherical harmonics for encoding view-dependent appearance and evaluate them into color before volumetric integration, recent autonomous driving simulation frameworks [Tonderski et al., 2024; Hess et al., 2024] render both cd and cs features into an image. We follow their approach, and use small trainable network consisting of few convolution layers to decode RGB color.

Dynamic scenes are typically represented as the graph [Chen et al., 2025b] with static and dynamic actor nodes, consisting of individual Gaussian particles. Dynamic actors are associated with their bounding boxes and an optimizable trajectory (sequence of SE3 poses). As pedestrians and cyclists represent a vulnerable road users category, we follow [Chen et al., 2025b] and represent humans as SMPL [Loper et al., 2015] bodies to improve reconstruction quality and provide better controllability.

Unscented transform. A key stage that enables efficient rendering in Gaussian splatting is tiling, in which particles are assigned to screen-space tiles based on the 2D projection of their shape. The EWA splatting formulation relies on a linearized projection approximation based on a single point, which makes rendering highly non-linear cameras (e.g. with rolling shutter or optical distortions) challenging. Alternatively, 3DGUT projects particles onto an arbitrary camera via Unscented Transform (UT). Given the mean and covariance of a particle in 3D, UT constructs a set of sigma points, which are individually projected through the camera model. The resulting 2D Gaussian conics are then approximated by weighting projected points. As a result, UT enables tiling for a wide range of complex camera models without requiring any modifications. Finally, particles assigned to each tile are sorted by depth to ensure correct ordering during volumetric rendering.

Volumetric integration. The final stage of rendering is rasterization, which performs volumetric integration. In EWA

splatting, 2D Gaussian conics are used explicitly to compute particle responses during volumetric rendering, introducing projection approximation errors into the rendering process. In contrast, 3DGUT uses 2D conics only for tiling, while volumetric rendering is performed by computing particle responses directly in 3D space. Specifically, for a given camera ray with origin o ∈ R3 and direction d ∈ R3 , the particle is evaluated at the point of maximum response along the ray:

Given the maximum response point, the overall volumetric integration follows standard formulation with αi = ρi(xmax):

Similarly, for depth and range image rendering we use the same equation as above, but replace particle colors ci with maximum response τmax distances along the ray.

3.2 Extended 3DGaussian Representation

While using geometric guidance is generally beneficial for 3DGS scene reconstuction, in some cases the two sensing modalities may impose different requirements on opacity modeling. Accurate geometric modelling requires representing surfaces with precisely located opaque gaussians, as guided by LiDAR measurements. In contrast, appearance modelling must account for view-dependent effects such as specular reflections and translucency, which often require multiple semi-transparent Gaussians along a viewing ray. Furthermore, surface opacity can be wavelength-dependent, as materials such as glass exhibit different transparency properties for LiDAR and visible light. To accommodate these effects, we augment each Gaussian with separate opacity parameters: σc for camera and σL for LiDAR rendering, which are jointly optimized within the unified representation. These two opacity values are then regularized during optimization to ensure consistency. We experimentally demonstrate that this extended representation effectively resolves distribution mismatches and improves both camera and LiDAR modelling for the novel-view synthesis problem.

3.3 Sensor Modeling

Autonomous driving perception and planning algorithms primarily rely on two sensor modalities: cameras and LiDARs. Both sensors typically operate in rolling shutter mode, in which sensor readings are acquired sequentially over time in a row-by-row fashion. As highlited by multiple previous works [Tonderski et al., 2024; Hess et al., 2024], due to high velocities experienced in autonomous driving scenarios, modeling rolling shutter is essential for accurate reconstruction. Whereas previous work [Hess et al., 2024] addressed rolling-shutter effects using separate, sensor-specific models, we describe generalized rolling-shutter modeling approach. We also identify spherical cameras (i.e. LiDARs) as an edge case of 3DGUT approach and propose phase modeling mechanism to mitigate arising rendering issues.

General rolling shutter modeling. Due to the rollingshutter mechanism, sensor readings are not captured instantaneously, and the observation time varies across pixels. Consequently, each image-space point (u, v) is associated with a distinct capture time τ(u, v). While in practice the rolling shutter time is linear and aligned with either horizontal or vertical image axis, we can express it as:

where τu, τv define the scan direction and speed, and denote middle of exposure time as τmid = τ(0.5, 0.5).

As time progresses during image acquisition, both the sensor and the scene evolve. We assume that, over the duration of capture, the camera and all dynamic actors move with constant linear and angular velocities. Under this assumption, the position of a 3D world point xw ∈ R3 at time η is given by:

where va ∈ R3 and wa ∈ R3 denote linear and angular velocities of the dynamic actor, respectively, and r ∈ R3 is radius vector defined by point position in actor coordinates.

Similarly, the camera pose evolves over time according to constant-velocity motion. Let q(η) ∈ R4 and t(η) ∈ R3 denote the camera orientation and position at time η. Camera movement is then modeled as:

where vc and wc are the camera's linear and angular velocities, and ⊗ denotes the Hamilton product. Given the arbitrary static camera projection function π(·) : R3 -→ [0, 1]2 and known point time η projection of world point xw ∈ R3 into image coordinates (u, v) has the closed form:

However, since the observation time η of the point is typically not known prior to projection, the projection problem formulates as estimating world time η which is consistent with image space point observation time:

Although this dependency is nonlinear and does not admit a closed-form solution, it can be efficiently solved using iterative methods. Following [Sun et al., 2020], we use the Newton-Raphson method and observe that only a few iterations are sufficient in practice.

Phase modeling. Our framework builds upon 3DGUT splatting, which leverages the Unscented Transform (UT) to approximate the projection of a 3D Gaussian. The UT performs a posterior approximation under the assumption that the projection of a 3D Gaussian can be adequately represented by a single 2D Gaussian. While this assumption holds in many practical settings, it breaks down in scenarios involving spherical rolling-shutter sensors. In particular, spinning spherical LiDAR sensors may observe a single Gaussian particle multiple times when it spans the azimuthal boundary or ignore it depending on the world dynamics (see Fig. 4). Moreover, this can lead to a same particle projecting into two separate shapes near boundaries with different covariances and depths (see ground-truth projections in Fig. 6). In such cases, the projected distribution becomes multimodal, violating the assumptions inherent to the UT.

To address this limitation, we introduce a phase modeling mechanism for spherical camera rendering. Under the spherical projection, the azimuth is a periodic function defined as:

While conventional static spherical mappings retain only the principal solution (k = 0), we explicitly account for phase wrapping by considering k ∈ {-1, 0,+1}. Since additional projections may differ in covariance and depth to camera, in addition to the standard central projection, we perform auxiliary projections shifted by ±π (see Fig. 6). For each interval, we constrain projections of sigma points into it by initializing τ in Eq. 8 with an interval middle of exposure time, and shift projected point azimuths by 2πk if they fall outside of interval. 2D Gaussian conics and depth are individually estimated for each interval based on projected sigma points, and valid projections are passed to the next tiling stage. Compared to original 3DGUT projection, our mechanism results in accurate tiling with no false tile-particle intersections and less depth sorting errors for particles near azimuth boundaries.

3.4 Camera and LiDAR supervision

Our scene representation consisting of multiple object nodes is optimized simultaneously from driving logs by randomly sampling images and closest by time LiDAR sweeps at each iteration. We supervise it using combination of losses:

Following common practice we set λ = 0.2. LiDAR guidance loss is L1 loss between rendered and ground-truth ray lengths. To ensure consistency between optimized opacities we impose Lopacity = ∑i |σc,i - σL,i| regularization. Details on other regularization terms Lreg are listed in appendix.

4 Experiments

Implementation details. We implement camera and LiDAR rendering using custom CUDA kernels. A unified rendering pipeline allows sharing rasterization forward and backward passes across all camera models. To handle non-uniform LiDAR beam angles during tiling, particle assignment iterates over elevation tile boundaries [Hess et al., 2024]. Human modeling follows OmniRE [Chen et al., 2025b], using their deformable and SMPL-based scene nodes. We adopt the specular Gaussian configuration and CNN post-processing from [Hess et al., 2024; Tonderski et al., 2024]. Hyperparameters largely follow SplatAD [Hess et al., 2024], while Gaussian splitting and densification use the 3DGUT [Wu et al., 2025] strategy with minor modifications. Full hyperparameter details are provided in the appendix.

Datasets. To evaluate our framework we use three popular datasets - Waymo Open Dataset [Sun et al., 2020], Argoverse2 [Wilson et al., 2021] and PandaSet [Xiao et al., 2021]. Following OmniRE [Chen et al., 2025b], we use 12 scenes fromWaymo, featuring ego-vehicle motion, dynamic and diverse classes of vehicles and pedestrians. As for Argoverse2 and PandaSet we adopt SplatAD [Hess et al., 2024] partition without modifications. Both training and evaluation are performed using full-resolution images and point clouds.

Baselines. For experimental evaluation we choose a wide range of baselines, featuring both recent NeRF-based methods (UniSim [Yang et al., 2023], NeuRAD [Tonderski et al., 2024], EmerNerf [Yang et al., 2024]) and 3DGS-based PVG [Chen et al., 2023], StreetGaussians [Yan et al., 2024], OmniRe [Chen et al., 2025b], HUGS [Zhou et al., 2024b; Zhou et al., 2024a] and SplatAD [Hess et al., 2024]. For PVG, StreetGaussians and OmniRe we use implementation based on drivestudio. As UniSim has no official repository, we use reimplementation by neurad-studio.

Evaluation metrics. For rendered image quality assessment we use standard novel-view synthesis metrics - PSNR ↑, SSIM ↑, and LPIPS↓. To measure LiDAR simulation quality we compute Chamfer Distance (CD↓) metric between rendered and ground-truth point clouds.

4.1 Image rendering

Scene Reconstruction. Following common practice, we evaluate the upper bound of reconstruction and modeling capacity of frameworks by reconstructing scenes using all available RGB images and LiDAR sweeps. Quantitative results on three datasets are reported in Table 1. On the Waymo dataset, XSIM achieves substantial improvements over the previous state of the art, SplatAD, with gains of +3.01 PSNR and +3.8% SSIM, while reducing LPIPS by 20.6%. Across all three datasets, XSIM consistently demonstrates superior RGB image reconstruction quality compared to prior baselines. On PandaSet, performance remains competitive across all metrics, with LPIPS showing a minor performance drop.

Novel-view synthesis. As our framework targets simulation of novel scenarios and trajectories, we evaluate rendering quality on views unseen during reconstruction (Table 1). We train on every second sensor frame and evaluate on the remaining frames. In this setting, XSIM achieves gains of +2.74 PSNR on Waymo and +1.04 PSNR on Argoverse over previous state-of-the-art method. We further assess extrapolation capability in Figure 2 by rendering ego-vehicle cameras under a lateral shift of 3 meters. Compared to prior methods, our framework produces more consistent renderings.

4.2 LiDAR and depth rendering

We evaluate LiDAR rendering quality on three datasets in both scene reconstruction and novel-view synthesis settings using Chamfer Distance (CD) (Table 1). XSIM achieves state-of-the-art results on all datasets, with an x8.8 CD error reduction on Waymo reconstruction and an x4.5 error reduction on Waymo NVS. As shown in Figure 7, XSIM better preserves characteristic LiDAR ring patterns and accurately renders pedestrians. We further assess geometric quality via RGB-camera depth rendering (Figure 3), where our method produces smooth, dense depth maps.

4.3 Ablation study

We illustrate the effect of our phase modeling mechanism via synthetic example (fig. 1) with a single car mesh captured by rolling shutter LiDAR camera. As the LiDAR moves along the vehicle, it becomes visible twice near the azimuthal boundaries of the range image due to rolling shutter. While 3DGUT default projection produces artifacts at the boundaries, our framework with phase modeling precisely reconstructs and renders the scene. We further ablate our framework components in a novel-view synthesis setting on six scenes fromWaymo dataset by individually disabling features (Tab. 2). Camera rolling-shutter modeling (c vs. a) leads to significant gains in PSNR. Removing the separate LiDAR opacity parameter (c vs. a) degrades both image and LiDAR metrics, with qualitative effects shown in Figure 5. Modeling LiDAR rolling shutter (d vs. c) with our phase modeling mechanism (e vs. c) further improves image and LiDAR rendering demonstrating effectiveness of our contributions.

5 Conclusion

In this paper, we presented XSIM, a unified sensor simulation framework for autonomous driving that extends 3DGUT splatting with generalized rolling-shutter modeling, enabling modeling of complex sensors in dynamic environments in a unified manner. Our phase modeling mechanism enables robust rendering for spherical rolling-shutter sensors by accounting for azimuthal discontinuities. Extensive experiments on three autonomous driving benchmarks demonstrate that our approach consistently improves geometric accuracy and photorealism, outperforming strong previous baselines.

Acknowledgements

This work was supported by the The Ministry of Economic Development of the Russian Federation in accordance with the subsidy agreement (agreement identifier 000000C313925P4H0002; grant No 139-15-2025-012).