2D Gaussian Splatting for Outdoor Scene Decomposition and Relighting

1 Introduction

Inverse rendering is a fundamental task in computer vision and graphics. It aims to decompose the 3D scene into intrinsic properties (e.g., material, geometry, and environment lighting) only from multi-view images, then allowing all properties can be manipulated independently.

Recently, advanced novel view synthesis techniques like neural radiance field (NeRF) [Mildenhall et al., 2021] provide a new approach for inverse rendering. Several NeRF-derived researches decompose scenes by multiple multi-layer perceptions (MLP) to learn intrinsic properties separately [Srinivasan et al., 2021; Zhang et al., 2021b; Rudnev et al., 2022]. However, the execution of MLP networks leads to

∗Corresponding author.

(c) Scene Properties Learning

Figure 1: (a) OSDR-GS takes multi-view outdoor images with changing lighting conditions as input. (b) It clusters images into several groups by their lighting cues, and (c) learns scene properties (e.g., geometry, material, and lighting) within lighting groups, (d) enabling the scene decomposition and relighting.

high computation, which significantly suffers from rendering efficiency. Recent 3D Gaussian splatting (3DGS) [Kerbl et al., 2023] has exhibited outstanding efficient performance for novel view synthesis. It utilizes a set of Gaussian primitives (e.g., position and covariance) to represent scene explicitly and employs GPU-accelerated rasterization for real-time rendering. Several GS-derived researches attempt to extend Gaussian primitives with normal, material, or other properties, which play the same role as multi-MLPs of NeRF and fulfill the scene efficient inverse rendering [Liang et al., 2024; Gao et al., 2023; Bi et al., 2024]. However, these methods are required to receive multi-view images with the fixed lighting conditions. Notably, it is costly and even impractical to control a fixed lighting in outdoor scenes (i.e., the changing sun or weather) during the dense view inputs capturing.

The main challenges of outdoor scene decomposition and relighting with Gaussian splatting are: (1) Images are captured under changing lighting conditions, causing the inconsistency performance of scene's visual appearance; (2) Lighting outdoors is highly complex and changing, leading to the wide variety of cast shadows. The first challenge causes the same coordinate of the scene to exhibit different pixel colors in the same viewpoints. It induces the uncertainty of spherical harmonic (SH) coefficients for each Gaussian to perform varying colors, which causes inaccurate scene inverse rendering ultimately. The second challenge is the sophisticated mix of multiple light sources in outdoor scene, adding to the complexity of lighting conditions. The relative positions of objects within the scene produce intricate cast shadows, making it more difficult to simulate scene details precisely. Specifically, the mentioned challenges are common and inevitable during the input views capturing in outdoor scenes.

In this work, we propose OSDR-GS, a novel method for Outdoor Scene Decomposition and Relighting leveraging 2D Gaussian Splatting. OSDR-GS consists of three key components: Lighting-based Group Learning, Fine-grained Outdoor Lighting Modeling, and Visibility-driven Shadow Module, to decompose the 3D scene into geometry, material, and lighting from multi-view images captured under changing and unknown lighting conditions. For the lightingbased group learning, we cluster images into multiple lighting groups by the elaborately distilled lighting cues, where each image within the same group is considered to have similar lighting and share the same lighting parameters. Specifically, the misgrouped images will be updated regularly according to the presented groups refinement strategy. For outdoor lighting modeling, we adopt a fine-grained parameterized scheme to simulate sun-light and sky-light separately. Then we assign the exposure factors to the input images for compensating the discrepancies adaptively of exposure settings for each capture. For visibility-driven shadow module, we measure the visibility of incident light in the scene to synthesize realistic cast shadows, and formulate an efficient regularization scheme based on the physical of light propagation to optimize more generalized scene shadows.

Extensive experiments on various challenging outdoor scenes, including NeRF-OSR [Rudnev et al., 2022] and MipNeRF 360 datasets [Barron et al., 2022] with several state-ofthe-art baselines, demonstrate the effectiveness of OSDR-GS. The main contributions are summarized as follows:

• We propose OSDR-GS, a novel scene decomposition and relighting method with Gaussian splatting from multi-view images captured in changing lighting.
• We present a lighting-based group learning framework that clusters input images into multiple lighting groups, and performs group-specific learning to optimize the lighting from each group.
• We introduce a fine-grained lighting scheme with a visibility-based shadow synthesis to measure the visibility of incident light, from arbitrary directions for simulating nuanced interplay of light and shadow.

2 RelatedWork

3D Scene Representation. In recent years, 3D scene representation technology advanced rapidly [Gao et al., 2022]. Several methods shown impressive performance in highquality novel view synthesis [Barron et al., 2021; Barron et al., 2022; Chen et al., 2022]. NeRF [Mildenhall et al., 2021] utilizes Multi Layer Perceptron (MLP) to model color and density of sampled 3D coordinates to represent scene implicitly. A number of researchers have endeavored to improve the performance of original NeRF in numerous dimensions. Several approaches work towards improving the quality of image rendering using enhanced neural network architectures [Zhang et al., 2020; Barron et al., 2021]. Some researchers seek to reconstruct scenes without the specific initialized camera pose prior by optimizing poses and MLP simultaneously [Lin et al., 2021; Bian et al., 2023; Zhang et al., 2024]. However, the execution of MLP networks lead to high computation, which significantly suffers from efficiency in rendering [Chen and Wang, 2024]. Recently, 3D Gaussian Splatting [Kerbl et al., 2023] significantly reduce rendering time while maintaining quality comparable to NeRF. It utilize a set of learnable 3D Gaussian primitive to explicitly model scene's color and render images. 3DGS employ a unique tile-based rasterizer tailored for Gaussian splatting, which facilitates real-time rendering for novel view synthesis [Yu et al., 2024; Sun et al., 2024; Xie et al., 2024; Lin et al., 2024]. While the volumetric radiance representation of 3D Gaussians conflicts with the thin nature of surfaces in most scenes. 2D Gaussian Splatting (2DGS) [Huang et al., 2024] propose to utilize a set of plane 2D Gaussian disks to represent surface of scene, which is benefit for high-quality surface reconstruction.

Different from the above methods, we represent the 3D scene by material, visibility and lighting rather than just color, which enables scene decomposition and relighting.

Scene Inverse Rendering. Scene inverse rendering is a fundamental task in computer vision and graphics, which aims to separate scene's geometry, material and lighting from multi-view images. Previous methods [Yu and Smith, 2019; Yu et al., 2020; Xu et al., 2018] utilize several convolution neural networks (CNN) to predict normal, albedo and lighting of an input image. With the rapid development of NeRF, some methods take as input multi-view images captured under known lighting condition and model scene using additional MLPs [Bi et al., 2020; Srinivasan et al., 2021; Xu et al., 2023; Li et al., 2023]. While these methods rely on known lighting, they can not work in most natural scenes. To deal with this, several methods model environmental illumination to decompose and relighting scene under unknown lighting [Zhang et al., 2021b; Zhang et al., 2022; Zhang et al., 2021a; Wu et al., 2023; Boss et al., 2021; Yang et al., 2022]. Other methods attempt to relight outdoor scene by modeling scene properties and learning irrelative environmental light in each views [Rudnev et al., 2022; Sun et al., 2023; Gardner et al., 2024]. Although these NeRFbased methods show infusive performance in outdoor scene inverse rendering, they still suffer from terrible efficient during training and rendering. Recently, 3D Gaussian splatting based method provide novel approach for inverse rendering [Guo et al., 2024; Chen et al., 2024]. GS-IR [Liang et al., 2024] proposes a 3DGS-based inverse rendering framework to deduce the physical attributes of scene. GS3 [Bi et al., 2024] presents a spatial and angular Gaussian based representation and a triple splatting process for indoor scene relighting. R3DG [Gao et al., 2023] proposes a material and lighting decomposition scheme by introducing additional components for each 3D Gaussian. However, these methods struggle with outdoor scenes especially those under changing and unknown lighting conditions.

In this work, we propose OSDR-GS that clusters input images into multiple lighting groups to learn lighting within groups, enabling the relighting of outdoor scene.

3 Preliminaries

2D Gaussian Splatting. 2D Gaussian Splatting [Huang et al., 2024] represents the scene by directly modeling the object's surface using 2D planar Gaussians, in contrast to 3DGS [Kerbl et al., 2023], which employs anisotropic 3D Gaussian primitives. The 2D Gaussian disk defined on a local tangent plane is parameterized by its center p, two orthogonal tangent vectors tu and tv, and a scaling vector S = (su, sv). The normal vector of Gaussian disk is natively represented as the cross product tw = tu × tv. The orientation is arranged into a rotation matrix R = [tu, tv, tw], and the scaling factors are encoded in a 3 × 3 diagonal matrix S = [su, sv, 0]. The parameterization of a 2D Gaussian is as follows:

where H ∈ 4 × 4 is a homogeneous transformation matrix representing the geometry of the 2D Gaussian, uniquely determining a 2D Gaussian in space. Its inhomogeneous form is given by ˆH = [RS p], where RS = [sutu svtv 0]. The 2D Gaussian value at the point u = (u, v) in the uv space can be computed by standard 2D Gaussian function G(u, v).

The center p, scaling (su, sv), and rotation (tu, tv) are learnable parameters. Similar to 3DGS, each 2D Gaussian primitive has opacity α and viewpoint-dependent color c parameterized by spherical harmonic coefficients. During rendering, it employs volume rendering to integrate alphaweighted color from near to far:

where x represents a ray emitted from the camera, u(x) denotes the uv space coordinates of the intersection point where ray x passes through the plane of the Gaussian.

Lambertian Reflectance. Lambertian reflectance model is a classic and widely used model for approximating diffuse reflection on surfaces [Kajiya, 1986]. It uses diffuse albedo as the key parameter of the bidirectional reflectance distribution function (BRDF), with the assumption that the reflected radiance is proportional to the cosine of the angle between incident light direction and surface normal. The outgoing light Lo at surface point x is formulated as:

where Li(x, ωi) refers to the incident light arriving at surface point x from the direction ωi. a is the albedo at point x, and (n · ωi) is the cosine of the angle between incident light direction ωi and the surface normal n at point x.

4 Method

In this section, we introduce OSDR-GS, which is comprised of lighting-based group learning framework (Sec. 4.1), finegrained outdoor lighting modeling (Sec. 4.2), and visibilitydriven shadow module (Sec. 4.3). Eventually, we describe the rendering process and training framework (Sec. 4.4).

4.1 Lighting-based Group Learning

Different lighting conditions can result in varying appearances of multi-view images captured in outdoor scenes. Previous approaches [Rudnev et al., 2022; Sun et al., 2023] assume that these images have totally inconsistent and unrelated lighting conditions, processing each one independently. This will introduce additional lighting uncertainty of the scene since the lighting in different views may be related, which hinders the learning of accurate lighting. In this part, we aim to fully utilize the relationship among multi-view images. As shown in Fig. 2(a), we propose a lighting-based group learning framework that first cluster them into multiple lighting groups with similar lighting conditions, and then assign lighting parameters to each group to learn changing lighting.

Lighting Cues Acquisition. To cluster multi-view images with changing lighting conditions into different lighting groups, we propose an elaborately distilled approach to acquire the lighting cues from each image. The lighting cues is considered to represent lighting condition, and demonstrate a certain level of robustness to changes in viewpoint [Tian et al., 2018]. We start the acquisition by a preprocessing that use pre-trained semantic segmentation model to filter out the background elements (e.g., sky). Next, we convert the image to the HSV (i.e., Hue, Saturation, and Value) color space, and extract the Hue channel. After that, we divide the Hue channel image into k equally spaced bins based on pixels' value, and calculate the frequency of pixels within each bin:

where hi denotes the frequency of the ith bin (i = 1, 2, . . . , k). P is the set of all pixel values in Hue channel image. IB(p) is an indicator function that equals 1 when the pixel's value p satisfies condition B, and 0 otherwise. Subsequently, we normalize all h and concatenate them into:

where h′ is the normalized frequency. Vector flight ∈ R1×k is the lighting cues of an input image. Since it is a statistical vector, flight demonstrates a certain level of robustness to changes in camera viewpoint, as shown in the left of Fig. 3. This enables it is applicable to images with different views. Lighting Group Learning. Following the acquisition of lighting cues from each image, we apply the simple Kmeans clustering algorithm [MacQueen, 1967] using the Bhattacharyya distance [Bhattacharyya, 1943] as the metric to obtain several clusters, which represent lighting groups. Images within the same lighting group have similar lighting conditions, while those in different lighting groups exhibit significantly different lighting conditions. Next, we assign a set of learnable lighting parameters (detailed in Sec. 4.2) to each lighting group, which means images in the same group share the same lighting parameters. This provides the lighting consistency across different viewpoints. These lighting parameters are jointly optimized with scene's geometry and material through backpropagation via our carefully designed loss function (detailed in Sec. 4.4).

Refining Lighting Groups. Although our lighting grouping strategy is applicable to most situations, there are several misgrouped images inevitably. These misgrouped images will contribute distorted gradients to the current lighting group during optimizing, which disturbs the optimization process and results in inaccurate lighting learning. Inspired by [Feng et al., 2015; Zhang et al., 2018], we assess the deviation between image rendered under particular lighting and the ground truth by the structural similarity item str(I1, I2) from structural similarity index (SSIM) [Wang et al., 2004] to solve this problem. We regularly iterate through all lighting groups for each image to find the group with the highest structural similarity str(I1, I2) as the optimal lighting group. This achieves the refinement of lighting groups.

4.2 Fine-grained Outdoor LightingModeling

Lighting Representation. Outdoor scenes are primarily illuminated by two types of light: direct sun-light and diffuse sky-light. The sun-light comes from the sun itself has strong directional quality. While the sky-light originates from the scattered light in atmosphere and diffuses across the entire sky. Inspired by [Sun et al., 2023], we adopt a finegrained approach to separately model these two light sources, as shown in Fig. 2(b). For the sun-light, we simply model it as a directional light source, parameterized by a learnable direction ωsun ∈ R3 and a learnable color-intensity Lsun i ∈ R3 For sky-light, we parameterize by first-order spherical harmonic (SH). It describes the global lighting distribution on the sphere by a set of basis functions Yl,m and corresponding learnable coefficients ysky l,m ∈ R, where l ∈ {0, 1},

-l ≤ m ≤ l, and l ∈ Z. The incident sky-light Lsky i ∈ R3 from direction ωi ∈ R3 can be described as:

where deg is the degree of SH that equals 1 here. Notably, as we described in Sec. 4.1, we assign lighting parameters to each lighting group, including both sky-light and sun-light.

Adaptive Exposure Compensation. Since the lighting cues is performed in the HSV color space, it is also robust to brightness (typically due to differences in camera exposure settings), as shown in the right of Fig. 3. This causes that within one lighting group, there can be images with the same lighting condition but different brightness. To adaptively compensate each image brightness in the same lighting group, we employ an exposure compensation strategy as follows:

where Lo(x) represents the outgoing light which comes from 3D point x in scene and arrives at the camera. γ ∈ R is a learnable exposure factors assigned to each image. C(x) is the color of pixel that corresponding to point x.

4.3 Visibility-driven ShadowModule

ShadowModeling. Synthesizing cast shadows is crucial for high-quality outdoor scene rendering, which not only determines the realism of rendering, but also impacts the accuracy of lighting groups refinement. In the real world, shadows are primarily caused by the occlusion of sun-light. To simulate this effect, we introduce a second-order learnable spherical harmonics (SH) coefficients yvis l,m ∈ R into each Gaussian primitive that measure the visibility of incident light (i.e., sunlight) and block it. The formula is shown as follows:

where V(x, ωi) ∈ R is the visibility that represents whether the light from direction ωi is visible at surface point x. It equals 0 if invisible (i.e., the light is occluded) and 1 if visible. deg is the degree of SH that equals 2 here. Y vis l,m ∈ R3 is the spherical harmonic coefficient at surface point x, which obtained by volume rendering using Eq. (2). Then, the occluded incident sun-light ˆLsun i can be formulated as:

where ωsun is the direction of incident sun-light Lsun i . Visibility Regularization. In order to improve the generalization of the visibility, we introduce a negative normal loss function based on a physical light propagation: the lower hemisphere of the object surface normal is occluded. Thus we formulate the negative normal Lnn as below:

where p is the number of pixels in an image. Ni represent the surface normal of 3D point xi corresponding to the i-th pixel, which is obtained by volume rendering from 2D Gaussians. ϵ ∈ R3 is a random noise that added in negative normal to represent the direction of negative normal hemisphere.

Practically, the realistic cast shadow will accompany by the binary visibility V. We present a binarization loss function Lbin to enforce a binary visibility pattern, as shown in below:

(11)

where F(x) = 1 - exp(-mx2 ) is an empirical exponentialbased function that use to adjust the strength of binarization. The other variables follow the explanation given in Eq. (10).

4.4 Rendering and Training

Physically Based Rendering. In order to decompose the scene, we introduce albedo a ∈ R3 into each Gaussian to render albedo image A by volume rendering, so that we can employ physically based rendering model to synthesize image. We define the outgoing light Lo at point x as follows:

where Lsun o and Lsky o represent the outgoing sun-light and outgoing sky-light, respectively. For the outgoing sky-light, we assume that it exclusively comes from the direction of surface normal N, which we formulate it as:

where Lsky i (ωi) is the incident sky-light from Eq. (6). ωi is aligned with normal, which means the cosine item (N · ωi) is equal to 1. For outgoing sun-light Lsun o , it is formulated as:

where ˆLsun i is the occluded sun-light from Eq. (9).

Overall, the aforementioned rendering process is illustrated in Fig. 4. After that, we extend Eq. (7) to adjust the exposure and obtain pixel color C using outgoing light Lsun o and Lsky o :

Training Framework. We follow 2DGS [Huang et al., 2024] to mainly employ color loss Lc and normal loss Ln to drive the training. To summarize, our loss function is formulated by combining several components, as expressed below:

where λ1 = 0.05, λ2 = 0.01 and λ3 = 0.001 are predefined weighting hyperparameters for each loss terms. By minimizing Eq. (16), our approach returns the 3D scene representation and the optimized lighting parameters.

5 Experiments

5.1 Experimental Settings

Datasets. We evaluate our OSDR-GS with SOTA baselines on both NeRF-OSR [Rudnev et al., 2022] and Mip-NeRF 360 [Barron et al., 2022] dataset. NeRF-OSR dataset consists of 8 outdoor scenes. Each scene includes multi-view images under multiple lighting conditions, with no overlap in lighting between the training and testing sets. Mip-NeRF 360 dataset consists of 5 outdoor scenes with fixed lighting conditions. Following GS-IR [Liang et al., 2024], we use images downsampled by a factor of 4, and pick every eighth image as a test image for dataset splitting.

Baselines and Metrics. We compare our OSDR-GS with three SOTA baselines: NeRF-OSR [Rudnev et al., 2022], GSIR [Liang et al., 2024], and R3DG [Gao et al., 2023]. In order to measure the quality of rendered images, we adopt Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Measure (SSIM), and Learned Perceptual Image Patch Similarity (LPIPS) as our evaluation metrics.

Implementation Details. To adapt the changing lighting conditions for GS-IR and R3DG, we revise them that assign lighting parameters individually to each image rather than to the entire scene. Moreover, due to the different lighting modeling among these methods, ground truth environment maps in NeRF-OSR dataset are not suitable for all of them. Thus, we obtain the necessary lighting parameters of test set by optimizing them exclusively under fixed geometry and material of scene. For each scene, training for 30k iterations on a single NVIDIA RTX 4090 GPU takes approximately 30 minutes. The test-time optimization requires around 5 minutes.

5.2 Comparing with SOTAMethods

In this part, we perform relighting evaluation on NeRF-OSR dataset and novel view sythesis on Mip-NeRF 360 dataset.

Comparison on NeRF-OSR Dataset. We report the relighting performance of our OSDR-GS compared with other baselines. As shown in Tab. 1, our method exhibits outstanding performance in novel view synthesis under novel lighting conditions. Specifically, compared with the best baseline GSIR [Liang et al., 2024], our method increases by 1.53, 0.55, and 1 on PSNR in all three scenes.

This demonstrates our method achieves more accurate decomposition of intrinsic components, resulting in more realistic images rendered under novel lighting condition. We also perform qualitative comparison. As shown in Fig. 5, our method produces more natural relighting effect and more clear image compared with other baselines. Moreover, unlike GS-IR or R3DG that requires multiple stages optimization, our method uses only one-stage optimization process, which significantly reduces the time cost of training.

Comparison on Mip-NeRF 360 Dataset. We report the novel view synthesis (NVS) performance of OSDR-GS on Mip-NeRF 360 dataset. As shown in Tab. 2, we demonstrate results on three outdoor scenes (i.e., Bicycle, Garden, and Stump). Our method achieves slight improvements over GS-IR in terms of PSNR and SSIM, and results comparable performance in LPIPS.

Note that the images in Mip-NeRF 360 dataset are captured under fixed lighting condition. This demonstrates the efficiency of proposed fine-grained lighting modeling and visibility-based shadow module, which are particularly suitable for inverse rendering of outdoor scenes. Moreover, we also conduct qualitative comparison. As shown in Fig. 6, we can observe that our method produces more refined surface details of Garden scene and more realistic colors in Bicycle scene. This indicates the capability of OSDR-GS to decompose scene into precise geometry and lighting.

5.3 Method Analysis

In this part, we conduct elaborate experiments on NeRF-OSR dataset to evaluate the efficacy of our proposed components. Effect of Proposed Components. We ablate the proposed components of our OSDR-GS, including lightingbased group learning (w/o LBGL), fine-grained lighting modeling (w/o FGLM), visibility-driven shadow module (w/o VDSM), and regularization of visibility (w/o RoV).

As illustrated in Tab. 3, we observe that the last row achieves remarkable performance, which demonstrates that the proposed group learning framework and shadow module are effective for modeling outdoor scene accurately. Moreover, the variant without shadow modeling result in inferior performance. This indicates that our visibility-driven shadow module effectively responds to the changing lighting conditions in the scene and produces realistic shadow effects.

Impact of Lighting-based Group Learning Framework. As summarized in Tab. 4, the first row of the table shows the baseline performance when neither the lighting grouping strategy nor the group refinement approach is applied. The second row indicates the performance improvement when only the lighting grouping strategy is used, and the third row presents the results when both the lighting grouping strategy and the group refinement approach are employed.

These results suggest that both the proposed lighting grouping strategy and the group refinement approach contribute significantly to enhancing the model's performance. Notably, we can observe that variant of only grouping demonstrates results almost as good as the best. This indicates the initialization of lighting groups using lighting cues is quite precise, nearly eliminating the necessity of lighting groups refinement process.

Impact of Visibility-driven Shadow Module. We remove the visibility-driven shadow modeling (w/o VDSM), negative normal regularization (w/o Lnn), and visibility binaryzation regularization (w/o Lbin) from OSDR-GS to evaluate the efficiency of these presented components.

The results are illustrated in Tab. 5. We can observe that the variant of last raw achieves the best results. This demonstrates the outstanding performance of our visibility-driven shadow module in rendering photorealistic images. Both the variant of w/o Lnn and the variant of w/o Lbin show slightly worse results, which indicates that both of them play important roles in high-quality visibility modeling.

6 Conclusion

This paper proposes outdoor scene decomposition and relighting with 2D Gaussian splatting (OSDR-GS), a novel inverse rendering strategy under outdoor scenes with changing and unknown lighting conditions. The presented OSDR-GS consists of lighting-based group learning framework, finedgrained outdoor lighting modeling component and visibilitydriven shadow module. We conduct extensive experiments to evaluate our method with several state-of-the-art baselines on multiple challenging outdoor datasets. The results demonstrate the effectiveness of OSDR-GS for outdoor decomposition and relighting. Since it is challenging to accurately obtain camera poses in outdoor scene with changing lighting conditions, We plan to consider camera-pose-free decomposition and relighting of outdoor scenes in future research.

Acknowledgments

This work was supported by the National Key R&D Program of China under Grants 2023YFF0906200.