Label-efficient Semantic Scene Completion with Scribble Annotations

1 Introduction

Semantic scene completion (SSC), also known as semantic occupancy estimation, aims to predict 3D geometric and semantic information about the whole scene. Recent studies have shown that occupancy grid can model the environments in a general representation [Tian et al., 2023] and provide essential guidance for downstream tasks [Sima et al., 2023; Liu et al., 2024], which motivate a series of camera-based methods [Cao and de Charette, 2022; Li et al., 2023c].

Current research on semantic scene completion mainly focuses on designing effective network structures under the fully supervised settings [Cao and de Charette, 2022; Li et al., 2023c; Xia et al., 2023], while few research work provides the solution to learn from less expensive sparse labels. Meanwhile, existing semantic scene completion [Behley et al., 2019; Li et al., 2023b] or occupancy estimation benchmarks [Wang et al., 2023c; Wei et al., 2023] all heavily rely on fully point-wise annotations for semantic segmentation on LiDAR point cloud, which not only incurs the expensive and tedious manual labeling but also limits their application in new scenarios. In this work, we firstly revisit the annotation on 3D semantic scene completion, and then propose the scribble-supervised paradigm for this task.

* Corresponding authors.

The ground truth for semantic scene completion contains both geometric and semantic parts that are obtained by accumulating multi-scan point clouds with point-wise semantic labels [Behley et al., 2019; Wang et al., 2022]. The geometric part can be directly extracted from raw LiDAR scans while the semantic one needs densely annotated labels. We aim to construct a label-efficient benchmark with only sparse semantic labels. Besides, previous efforts [Li et al., 2023c; Xia et al., 2023; Shi et al., 2024] indicate that it is challenging for fully-supervised models to estimate geometry. It is even more severe to learn from sparse semantic labels, as most of the non-empty voxels are unlabeled without semantic information. To tackle this issue, we investigate the potential of dense geometry offline, thus significantly mitigating the reliance on dense semantic labels.

In this work, we make full use of the sparse annotations in ScribbleKITTI [Unal et al., 2022] to generate scribblebased semantic occupancy labels combined with the dense geometric structure to construct a new benchmark called ScribbleSC. Specially, we develop a simple yet effective approach, dubbed as Scribble2Scene, which is the first weaklysupervised scheme for 3D semantic occupancy estimation. More importantly, it achieves similar performance compared with the existing fully supervised methods. Our proposed Scribble2Scene consists of two stages, including geometryaware auto-labelers construction (Stage-I) and online model training with distillation (Stage-II).

At Stage-I, we construct geometry-aware auto-labelers with scribble annotations, including Dean-Labeler and Teacher-Labeler. Dean-Labeler treats the complete geometric structure as input, which converts this task into an easier semantic segmentation problem to obtain high-quality voxelwise segmentation results. Teacher-Labeler is also trained in offline mode with both input image and complete geometry, which has the same network architecture as the online model. It has the capability to extract more accurate features and semantic logits for the online model. At Stage-II, we train the online completion network in a fully-supervised manner based on the pseudo labels provided by Dean-Labeler. In particular, a new range-guided offline-to-online distillation scheme is proposed for large-scale semantic scene understanding, which enhances the performance of the online model with the features from the trained Teacher-Labeler. Fig. 1 shows some qualitative results and comparisons.

Our main contributions are summarized as below:

• We revisit the annotation of semantic scene completion and propose a scribble-based label-efficient benchmark named ScribbleSC, which provides both sparse semantic annotations and dense geometric labels.
• We propose Scribble2Scene, the first weakly-supervised approach for semantic scene completion, designed to handle sparse scribble annotations. Geometry-aware auto-labelers construction and offline-to-online distillation training are devised to accurately predict 3D semantic occupancy.
• Under our presented Scribble2Scene framework, the camera-based scribble-supervised model achieves up to a competitive 99% performance of the fully-supervised one on SemanticKITTI without incurring the computational cost during inference. Additional experiments on SemanticPOSS demonstrate the generalization capability and robustness of our proposed scheme.

2 RelatedWork

Semantic Scene Completion. Semantic scene completion (SSC) is firstly proposed in SSCNet [Song et al., 2017] to construct the complete 3D occupancy with voxel-wise semantic labels from a single-view observation. At the early stage, researchers mainly focus on the indoor scenarios [Liu et al., 2018; Zhang et al., 2018; Li et al., 2020; Cai et al., 2021] with RGB image or depth map as input. SemanticKITTI [Behley et al., 2019] provides the first large dataset and benchmark in the outdoor for autonomous driving. The subsequent works mainly utilize the occupancy grid voxelized from the current LiDAR frame [Roldao et al., 2020; Wilson et al., 2022] or point cloud directly [Yan et al., 2021;

Cheng et al., 2021; Xia et al., 2023; Mei et al., 2023] as input and obtain promising performance. Recently, camera-based methods [Cao and de Charette, 2022; Huang et al., 2023; Li et al., 2023c; Zhang et al., 2023; Yao et al., 2023] attract more research attention due to their lower sensor costs. VoxFormer [Li et al., 2023c] estimates the coarse geometry firstly and adopts the non-empty proposals to perform deformable cross-attention [Zhu et al., 2021b] with image features, which achieves the best performance among camera-based models. Along this line, we mainly focus on vision only methods.

Sparsely Annotated Learning. Sparse annotations for 2D image segmentation are widely explored including scribble [Lin et al., 2016; Liang et al., 2022; Li et al., 2024b], box [Tian et al., 2021; Li et al., 2022; Li et al., 2024a], point [Bearman et al., 2016; Fan et al., 2022; Li et al., 2023a] and etc. In 3D scene understanding, ScribbleKITTI [Unal et al., 2022] re-annotates the KITTI Odometry dataset [Geiger et al., 2012] and provides the scribble-supervised benchmark for LiDAR segmentation on SemanticKITTI [Behley et al., 2019]. Box2Mask [Chibane et al., 2022] adopts 3D bounding boxes to train dense segmentation models and achieves 97% performance of current fully-supervised models. We explore the potential of sparse annotations on the geometrically and semantically challenging task of semantic scene completion.

Teacher-Student Network. Knowledge Distillation (KD) is proposed initially to transfer the dark knowledge from a large trained teacher model to a small student one for model compression [Hinton et al., 2015]. Following researchers achieve this goal in 2D image at not only logit-level [Cho and Hariharan, 2019; Furlanello et al., 2018; Zhao et al., 2022; Liu et al., 2022] but also the feature-level [Romero et al., 2015; Heo et al., 2019a; Heo et al., 2019b; Yang et al., 2022] for in-depth exploration. Teacher-Student architectures in KD are widely adopted in various tasks and applications [Ye and Bors, 2022; Ye and Bors, 2023; Wang et al., 2023b; Wang et al., 2024]. In SSC, SCPNet [Xia et al., 2023] is proposed to distill dense knowledge from a multi-scan model to a single one with pairwise relational information while its design is exclusively tailored for LiDAR-based methods. CleanerS [Wang et al., 2023a] generates a perfect visible surface with ground truth voxels and trains a teacher model having cleaner knowledge in indoor scenarios. In this paper, we present a new offline-to-online distillation scheme, which is specially designed for 3D semantic scene completion in selfdriving environments.

3 The ScribbleSC Benchmark

The supervision of semantic scene completion can be split into two parts, including geometric structure and semantic label. The geometric information can be easily obtained by accumulating exhaustive LiDAR scans and voxelizing the points that fall on the predefined region in front of the car. Meanwhile, the semantic label of each voxel is determined by the majority of labeled points within the voxel. The annotation of semantic part is highly dependent on the dense point-wise semantic segmentation labels, which requires an expensive and complicated labeling process. To construct a label-efficient benchmark, we make use of sparse annotations from ScribbleKITTI [Unal et al., 2022] to replace the original full annotations provided by SemanticKITTI [Behley et al., 2019] and achieve the construction of the semantic part. In ScribbleKITTI [Unal et al., 2022], line-scribbles are adopted to label the accumulated point clouds and cover only 8.06% labeled points of the total training set including 10 sequences, which contribute to a 90% time saving 1 .

Label Construction. Like SemanticKITTI [Behley et al., 2019], we superimpose 70 future LiDAR scans to get the dense geometric structure and choose the volume of 51.2m in the forward of car, 25.6m to the left/right side and 6.4m in height with an off-the-shelf voxelizer tool 2 . The voxel resolution is set to 0.2m and a volume V of 256×256×32 voxels can be obtained. We assign the empty label to voxels that are devoid of any points. For non-empty voxels, the semantic label is ascertained by conducting a majority vote across the scribble labels of the points situated within the voxel. If no labeled points exist in a non-empty voxel, we annotate it as unlabeled. Fig. 2 provides typical examples from ScribbleSC compared with the fully-annotated SemanticKITTI [Behley et al., 2019]. Further information on the label construction for ScribbleSC is included in the Supplementary Material. Label Usage Instruction. The label V ∈ RX×Y×Z (X, Y, Z = 256, 256, 32) in ScribbleSC contains 19 seman-

tic classes, an empty class, and an unlabeled class, which can be split into full geometric annotation G and sparse semantic annotation S. The geometric annotation G ∈ RX×Y×Z is a binary voxel grid map. Each voxel is marked as 0 if it is empty else 1 for occupied no matter with labeled or unlabeled points. The semantic annotation S ∈ RX×Y×Z contains a small number of voxels that are labeled with semantic classes. The vast majority of voxels in S are 0, including empty and unlabeled ones. Training directly with ScribbleSC inevitably encounters an imbalance between geometric and semantic supervision. We will introduce our solution in Sec. 4.

1 Scribble labels cost around 10-25 minutes per tile, while full annotations cost 1.5-4.5 hours.

2 https://github.com/jbehley/voxelizer, MIT License

Voxel Labeling Statistics. We have conducted the distribution analysis of semantic labels in ScribbleSC and compared it with the fully annotated SemanticKITTI Benchmark as shown in Fig. 3. Our proposed semantic annotations only contain 13.5% labeled voxels over SemanticKITTI. ScribbleSC is a more challenging benchmark as it not only contains a substantial count of empty voxels but also a large number of unlabeled voxels among non-empty ones. More statistical analyses are given in the Supplementary Material.

4 ProposedMethod

4.1 Overview of Scribble2Scene

Fig. 4 provides the overview of our proposed Scribble2Scene method for scribble-supervised semantic scene completion, which can be divided into two stages, i.e. geometry-aware auto-labelers construction (Stage-I) and online model training with distillation (Stage-II). At Stage-I, we make full use of complete geometric structure G and image I ∈ RH×W×3 observed from the current frame to construct Dean-Labeler and Teacher-Labeler with sparse scribble annotations, as shown in the left of Fig. 4. At Stage-II, we then adopt the pseudo labels generated by Dean-Labeler and perform the presented rangeguided offline-to-online distillation with Teacher-Labeler to train the online model, as illustrated in the right of Fig. 4. All the models need to predict the complete 3D occupancy information O ∈ RX×Y ×Z×C in the predefined voxel space, where C is the number of total categories including empty and semantic classes. VoxFormer [Li et al., 2023c] is employed as the baseline model, which is the state-of-the-art (SOTA) semantic scene completion network with only camera input. We delve into the specifics of Stage-I in Sec. 4.2

and Stage-II in Sec. 4.3.

4.2 Geometry-Aware Auto-Labelers

To fully investigate the potential of the dense geometric structure and sparse semantic labels, we construct two geometryaware auto-labelers (GA2L) in the offline mode at Stage-I. The offline mode means that we can leverage the complete geometry G from the whole sequences to train a more performant model.

Dean-Labeler. Existing SOTA semantic scene completion models [Li et al., 2023c; Mei et al., 2023; Xia et al., 2023] employ different branches to process geometric structure and semantic information, respectively. Moreover, their overall performance is often greatly limited by the inaccuracy of geometry estimation. If we directly treat the complete geometry G as input, the semantic scene completion can be converted into a voxel-wise semantic segmentation problem. Motivated by this, we adopt the sparse convolutional network (SCN) as the backbone of Dean-Labeler to obtain voxel-wise semantic prediction from the complete geometry, as illustrated in the top left of Fig. 4. Since the SCN does not change the geometric structure of the input, we only need to process the semantic part with scribble supervision. The partial cross-entropy loss function CE(·, ·) to train Dean-Labeler is formulated as

where O(D) is the predicted occupancy information from Dean-Labeler. We only perform optimization on voxels that are non-empty and labeled.

Leveraging complete geometry input, Dean-Labeler obtains the promising 3D semantic predictions P (D) (Sec. 5.3), which eliminates the imbalance between geometry (dense) and scribble-based semantic supervision (sparse) with ScribbleSC. The high-quality semantic predictions serve as the pseudo-labels for online model training at Stage-II.

Teacher-Labeler. To further explore the role of complete geometry S, we design Teacher-Labeler with the modalityspecific model VoxFormer [Li et al., 2023c]. We replace the noisy coarse geometry ˆ G12 ∈ R X2 × Y2 × Z2 estimated from depth prediction as in VoxFormer with the down-sampling complete geometry G12 ∈ R X2 × Y2 × Z2 . The deformable cross attention (DCA) is employed to sample image features with the precise non-empty proposals, as shown in the bottom left of Fig. 4. The SSC Decoder is made of a deformable selfattention (DSA) module and a completion head. The partial cross entropy loss in Eq. 1 is also adopted to train TeacherLabeler. Additionally, we employ the geometric loss from MonoScene [Cao and de Charette, 2022] with G to alleviate the geometry change from the non-empty proposals when dense convolutions are used in the completion head.

Teacher-Labeler is trained with the precise non-empty proposals provided by complete geometry, which is able to extract more accurate features from input images. Therefore, the completion model can focus on the semantic part to make this task easier with only sparse scribble semantic annotations. In our experiments (Sec. 5.3), Teacher-Labeler achieves extremely higher performance compared to online models using noisy coarse geometry, which is further leveraged for online model training at Stage-II.

4.3 Online Model Training with Distillation

At Stage-II, we perform the online model training, which only employs currently observed information as inference input. The accurate pseudo labels provided by Dean-Labeler are utilized to replace the scribble labels so that we can optimize the model in a fully-supervised paradigm.

The online model as the student network has the same architecture as Teacher-Labeler while it cannot use the complete geometry G as input. To take advantage of the features and predictions obtained from models with the full geometry, we propose a novel range-guided offline-to-online distillation (RGO2D) module that instructs the online model to learn auxiliary modality-specific knowledge at Stage-II. As shown in the right half of Fig. 4, we adopt the well-trained Teacher-Labeler as the offline teacher model and freeze the network weights to perform offline-to-online distillation for the online model, which alleviates the interference of inaccurate pseudo-labels on training.

Range-Guided Offline-to-Online Distillation. When there is a large difference in the input and network performance of the teacher and student models, directly minimizing the Kullback-Leibler (KL) divergence or other metrics to align the outputs, i.e. O(S) ∈ RX×Y×Z×C from the student and O(T) ∈ RX×Y×Z×C from the teacher, often does not work well [Huang et al., 2022]. Inspired by CleanerS [Wang et al., 2023a], we adopt global semantic logit combined with local semantic affinity rather than the original predictions to perform distillation between the teacher and student at the logit level. Considering that the outdoor driving scene involves a wider range, only using the semantic logit and affinity of the whole scene cannot reflect the distribution of each class well. Moreover, the semantic logit and affinity of the area closer to the ego-vehicle are inherently more amenable to learning processes and endow more significance in ensuring vehicular safety assurances. Therefore, we consider introducing range information as guidance to model the distribution of each semantic class.

Specifically, we take the ego-vehicle as the center and divide the whole volume in the predictions of teacher O(T) and student O(S) into three ranges (including near, middle and far) according to the distance as shown in Fig. 5. Then the global semantic logits (GSL) of teacher {o(T,r) }3r=1 and student {o(S,r) }3r=1 at different ranges are calculated with scribble annotations S. To adequately measure the difference of global semantic logits between student and teacher, we introduce the inter-relation and intra-relation loss to perform global distillation

where dp is Pearson's distance, and r = 1, 2, 3 denote different ranges. α and β are the balanced weights of the inter- and intra-relation loss, respectively. Moreover, we compute the local semantic affinity (LSA) with the global semantic logit and the prediction at each range for teacher {A(T,r) }3r=1 and student {A(S,r) }3r=1 . Then local distillation loss is computed below

where MSE(·, ·) denotes the mean square error function.

Additionally, we adopt MSE(·, ·) as the feature-level distillation loss Lfeat. We choose the 2D features from the image encoder as the targets to align. Overall, the range-guided offline-to-online distillation is composed of the above items

where wr denotes the range loss coefficients, and r = 1, 2, 3. Lglobal,r and Llocal,r represent the global and local distillation losses at near, middle, and far range, respectively.

4.4 Training and Inference

Overall Loss. The total loss Ltotal for the online model training consists of semantic loss Lsem, geometric loss Lgeo and distillation loss Ldistill as below

where Lsem is the commonly used weighted cross-entropy loss. We employ the pseudo labels P (D) from Dean-Labelers as the full semantic supervision. Lgeo is the geometric scene-class affinity loss proposed in MonoScene [Cao and de Charette, 2022].

Inference. At the inference stage, we only need to preserve the student branch, which can obtain similar performance on accuracy while retaining efficient inference as the fullysupervised model.

5 Experiments

5.1 Experimental Setup

Dataset. Our models are trained on ScribbleSC. Unless specified, the performance is mainly evaluated on the validation set of the fully-annotated SemanticKITTI [Behley et al., 2019], which is a highly challenging benchmark. All input images come from the KITTI Odometry Benchmark [Geiger et al., 2012] consisting of 22 sequences. Following the official setting, we use the sequences 00-10 except 08 for training with ScribbleSC while sequence 08 is preserved as the validation set. We submit the predictions of sequences 1121 to the online evaluation website and obtain the scores on the hidden test set. Additionally, we have conducted extra experiments on the SemanticPOSS [Pan et al., 2020], which is another challenging dataset collected in a campus-based environment. Since the scribble-based annotations on point clouds of SemanticPOSS are unavailable, we randomly sample 10% of its full annotations to obtain similar sparse labels as scribbles. Then we construct semantic scene completion labels, including sparse semantic labels and dense geometric labels as described in Sec. 3. Adhering to the original configuration, the sequences (00-01, 03-05) / 02 are split as training and validation set, respectively.

Evaluation Protocol. We follow the official evaluation benchmark and employ intersection over union (IoU) to evaluate the scene completion performance, which only measures the class-agnostic geometric quality. The standard mean intersection over union metric (mIoU) of 19 semantic classes is reported for semantic scene completion. We choose the class-wise mIoU as the dominant evaluation metric. To comprehensively compare with the fully-supervised methods, we provide evaluation scores from three different ranges on validation set including 12.8×12.8×6.4m3 , 25.6×25.6×6.4m3 , and 51.2 × 51.2 × 6.4m3 .

Implementation Details. For Dean-Labeler, we adopt Cylinder3D [Zhu et al., 2021a] as the SCN backbone and use a single GPU to train the network with a batch size of 4. For Teacher-Labeler and student model, we use the same backbone of VoxFormer-T [Li et al., 2023c], which takes the current and previous 4 images as input. All models based on VoxFormer are trained on 4 GPUs with 20 epochs, a batch size of 1 (containing 5 images) per GPU. Our baseline model is directly trained with partial cross-entropy loss and geometric loss under the available scribble- and geometrybased supervisions. For our proposed range-guided distillation scheme, we choose three different ranges, which are the same as the evaluation part to perform the global and local distillation, i.e. 12.8m, 25.6m, and 51.2m. More implementation details and model complexity analyses are provided in the Supplementary Material.

5.2 Main Results

We firstly compare Scribble2Scene with state-of-the-art fullysupervised camera-based methods on the validation set of SemanticKITTI, including VoxFormer [Li et al., 2023c], TPVFormer [Huang et al., 2023], and MonoScene [Cao and de Charette, 2022]. As shown in Tab. 1, Scribble2Scene obtains 99% performance (13.27% mIoU v.s. 13.35% mIoU) of fully-supervised VoxFormer at full-range 51.2m, which only uses 13.5% of the labeled voxels. The competitive accuracy is also achieved against other camera-based models at different ranges. Compared with VoxFormer trained directly using scribble annotations, our method has a significant improvement in the most of categories. To further examine the effectiveness of our method, we submit results on the extremely challenging test set of SemanticKITTI without extra tricks. As illustrated in Tab. 2, our scribble-based method achieves 13.33% mIoU and 42.60% IoU, which outperforms most of the fully-supervised models and demonstrates generalization capability in more scenarios.

Additional experiments are conducted on SemanticPOSS with our Scribble2Scene framework. Due to limited space, the corresponding experimental table and implementation details are provided in the Supplementary Material.

5.3 Ablation Studies

Effectiveness of Auto-Labelers. Firstly, we verify the effectiveness of scribbles as annotations to train the DeanLabeler and Teacher-Labeler. As shown in Tab. 3, the promising performance is achieved with our training pipeline compared with SOTA fully-supervised completion methods including LiDAR-based models. Our Dean-Labeler obtains 100% IoU with the highest score of 42.28% mIoU on ScribbleSC, which ensures that the quality of our pseudo-labels is sufficient to provide reliable supervision for the student model. The Teacher-Labeler also achieves a comparable performance with LiDAR-based methods and outperforms the camera-based models by a large margin. This observation reveals that the main bottleneck of current camera-based methods lies in the estimation of geometry.

Impact of Each Module. Secondly, we study the impact of each module in the whole framework as shown in Tab. 4. The Scribble2Scene with Dean-Labeler (DL) means that we replace the scribble annotations with the pseudo labels provided by DL. It can be seen that the models trained with DL obtain obviously higher geometric performance than those trained with scribbles directly. Moreover, we directly add the Teacher-Labeler (TL) with the Vanilla KD [Hinton et al., 2015], and a slight precision improvement is achieved. Furthermore, we perform range-guided offline-to-online distillation (RGO2D) with TL and the best performance is obtained. Effect of Offline-to-Online Distillation. Finally, we compare our RGO2D with other knowledge distillation methods including MGD [Yang et al., 2022], DIST [Huang et al., 2022] for common KD and CleanerS [Wang et al., 2023a] designed for SSC. As shown in Tab. 5, our proposed offlineto-online distillation scheme outperforms all other methods. We further conduct the ablation analysis on each item of our distillation module as described in Tab. 6. Compared to local distillation, the global one generates a larger performance impact. The range-guided information further enhances the performance by computing semantic logit and affinity at different ranges. With feature distillation, the online model obtains the best semantic scene completion accuracy.

6 Conclusion

In this work, we have presented a scribble-based labelefficient benchmark ScribbleSC for semantic scene completion in autonomous driving. To enhance the performance in this setting, an effective scribble-supervised approach Scribble2Scene has been developed. The offline Dean-Labeler provides dense semantic supervision and Teacher-Labeler guides the online model to learn structured occupancy information with new range-guided offline-to-online distillation. Extensive experiments demonstrate that our Scribble2Scene closes the gap between the sparse scribble-based approach and densely annotated methods, which shows competitive performance against the fully-supervised counterparts.

Acknowledgments

This work is supported by National Natural Science Foundation of China under Grants (62376244). It is also supported by Information Technology Center and State Key Lab of CAD&CG, Zhejiang University.