SceneEncoder: Scene-Aware Semantic Segmentation of Point Clouds with A Learnable Scene Descriptor

1 Introduction

As a basic task in 3D vision, semantic segmentation of point clouds has drawn more and more attention. Due to the irregularity and disorder of point clouds, many previous works convert point clouds into the grid representation through voxelization or projection to leverage the effectiveness of grid convolution [Graham et al., 2018; Su et al., 2015]. These methods inevitably destroy the original geometric information of point clouds. Therefore, PointNet [Qi et al., 2017a] directly processes the raw point clouds and extracts features with shared Multi-Layer Perceptrons (MLPs). However, because of the unusual properties of point clouds and the complexity of scene segmentation, semantic segmentation of point clouds remains a challenging issue.

Global information, which is commonly extracted by the encoder network, usually contains the essential knowledge that directly summarizes the information of the whole scene, thus should play a more significant role in the semantic segmentation. For human beings, the prior knowledge towards a scene could directly influence the semantic comprehension. For instance, as in Figure 1, when being in a bathroom, it is easy to distinguish the "shower curtain" category from the "curtain" category even they look similar. Therefore, the global information of point clouds can play the role of prior knowledge for semantic segmentation.

∗Equal Contribution.

†Corresponding Author.

Many previous works utilize the U-Net architecture to extract global features with growing receptive field, and then mix these global features with local features by concatenation hierarchically [Wu et al., 2019; Graham et al., 2018]. However, such a kind of mixture not only attenuates the guidance of global information, but also degrades the representation capability of local features. SpiderCNN [Xu et al., 2018] adopts a more direct way to take the global label of a point cloud as a part of elements in feature vectors to enhance the effect of the global information. However, it is based on an impractical assumption that the global label is always available, especially in testing phase.

Therefore, we propose a SceneEncoder module to predict a multi-hot scene descriptor for every point cloud, a novel representation of global label rather than the manually annotated one-hot scene label, to perform a scene-level guidance. An ideal scene descriptor is designed to be a binary vector with each element representing the existence of corresponding category of object. As shown in Figure 1(a), the scene descriptor of a bedroom, consisting of floor, bed, table, curtain and some other furniture, can be represented by a 5-hot vector.

Different from concatenating global features to local features, we utilize the scene descriptor as a mask/attention to aid the point-level semantic segmentation by filtering out irrelevant object categories that are impossible to exist in the current scene. For example, there is a curtain and a shower curtain in Figure 1(a) and (b) respectively, which are difficult to distinguish due to the similar appearance. However, our descriptors can directly help to exclude the classification option of the shower curtain in bedroom scenes and exclude the curtain in the bathroom. Besides, compared with a onehot scene label, our well-designed multi-hot scene descriptor is also able to subdivide each category of scenes. As shown in Figure 1(c) and (d), meeting rooms which are designed for different purposes can be further represented by different scene descriptors. To train the SceneEncoder module, we dynamically generate the ground truth of scene descriptors by checking which categories of objects exist in the input training scene point clouds based on labels of all points.

Since the segmentation task is a point-level task, global information is not enough for point-level classification. Therefore, an increasing number of works [Zhao et al., 2019; Wu et al., 2019] focus on exploiting the local context which is important for recognizing fine-grained patterns and generalizing to complex scenes. A problem in the prevailing methods is that the extracted feature usually involves the features of different categories in the local region. For example, the point clouds belonging to a chair might be nearby a table, such that the extracted local feature would incorporate features of both the chair and table. This would make point features less distinguishing because these features encode more than one class information, and it also results in poor object contours in the segmentation task. To tackle this problem, we design a novel loss, namely region similarity loss, to propagate distinguishing point features to ambiguous features in local regions. That is, we only propagate each distinguishing feature to its adjacent points (i.e., points in its neighborhood) with same labels since the points in the same local region but with different categories would differ on their features. Overall, the major contributions can be summarized as follows:

• We design a SceneEncoder module with a scene descriptor to provide a scene-level guidance, leading to an effective collaboration of the global information and local features for semantic segmentation.
• We propose a novel region similarity loss to propagate distinguishing point features to ambiguous features in local regions so as to alleviate the segmentation noise effectively.
• We integrate our SceneEncoder module into two prevailing networks by involving the region similarity loss. The experimental results on two benchmark datasets demonstrate that our method outperforms many state-of-the-art competitors.

2 RelatedWork

Segmentation on point clouds. Impressively, PointNet [Qi et al., 2017a] processed the raw point clouds directly and extracted point features with MLPs. Built upon it, PointNet++ [Qi et al., 2017b] designed a hierarchical framework to exploit local context with growing receptive fields. However, the key operation to aggregate features in both methods is max-pooling, which leads to great loss of context information. To utilize the local context efficiently, PointWeb [Zhao et al., 2019] densely connected each pair of points in the local region and adjusted each point feature adaptively. An independent edge branch was introduced in [Jiang et al., 2019] to exploit the relation between neighboring points at different scale and interwove with the point branch to provide more context information. PointConv [Wu et al., 2019] focused on the distribution of points in the local region and utilized density information. Furthermore, an octree guided selection strategy was proposed in [Lei et al., 2019] to guide the convolution process of point clouds in a more reasonable way. Compared with these methods which utilize the scene information by fusing local and global features together, we directly predict a scene descriptor from global features to guide the point-level segmentation so as to enhance the effect of the global information.

Segmentation refinement. Additionally, noises and poor contours are usually caused by ambiguous features because they are mixed with the features of different categories in the local region. SEGCloud [Tchapmi et al., 2017] proposed to combine networks with fine-grained representation of point clouds using 3D Conditional Random Fields (CRF). GAC [Wang et al., 2019] determined how each point feature contribute to the extracted feature by the similarity between these two features to improve the distinguishability of features. By contrast, we propose a more general loss to refine the results of segmentation by propagating distinguishing features in the local region. This strategy could be adopted in the training process of most networks in semantic segmentation task.

Loss function. In [De Brabandere et al., 2017], a widely used loss was proposed to minimize the difference between point features of the same instance object. Similarly, Pairwise Similarity Loss [Engelmann et al., 2018] maximized the similarity among point features of the same semantic object. It is noteworthy that points which are far apart should have different features even if they belong to the same category. Therefore, our loss just focus on forcing adjacent point features of the same category to be similar.

3 Method

First we introduce the overall architecture in Sec. 3.1. Then, the SceneEncoder module and the scene descriptor will be described in details in Sec. 3.2. In Sec. 3.3, the novel region similarity loss would be depicted. Finally, we summarize different losses used in the training process of the whole network in Sec. 3.4.

3.1 Overview

Figure 2 illustrates our overall architecture. We encode the raw point cloud into few global features, which will be taken as the input into the proposed SceneEncoder module (Figure 2(a)). The output of this module, i.e., the predicted scene descriptor, would be used to point-wise multiply with classification probability vector of each point, so as to exclude the impossible category options and get final prediction. Furthermore, during the training process, we select M distinguishing points that are correctly classified and their corresponding features from the feature map. Then, we define a region similarity loss (Figure 2(b)) to propagate distinguishing features by improving the feature similarity between each distinguishing point and its neighboring points with the same label.

3.2 SceneEncoder Module

The global information can give a top-down guidance to point-level classifier. To make full use of the global information as the scene prior knowledge, we propose a SceneEncoder module with a well-designed multi-hot scene descriptor to enhance the effect of global information in the semantic comprehension. This descriptor is able to substitute the global scene label, and acts as a mask/attention to eliminate impossible results when classifying each point.

Multi-hot Scene Descriptor

As the recognition of a scene can directly influence the result of point-level semantic segmentation, we propose to use a scene descriptor to represent the global scene information. Similar to [Gupta et al., 2013], we also use the objects existing in the scene to represent global information. By contrast, our scene descriptor focuses on whether the corresponding object exists rather than the proportion and work as a mask to filter out impossible categories rather than an additional feature. For scene semantic segmentation task with n object categories, we design a multi-hot vector of length n with each element representing the probability of existence of the corresponding category in the point cloud. Specifically, let ˜g denote the predicted descriptor, the i-th element of this descriptor ˜gi will be 0 if the i-th category does not exist in this point cloud. Then, in semantic segmentation, we could apply softmax on the output to obtain the classification probability map ˜ P ∈ RN×n , where N denotes the number of points, n is the number of classes, and ˜ P(i, j) represents the probability that point i belongs to class j. To make the scene descriptor directly guide the semantic segmentation, we directly multiply the probability map ˜ P with the predicted descriptor ˜g to achieve the refined probability map ˜ Pref as follow.

where ˜ Pref(i, j) is the final predicted probability that point i belongs to class j, and ˜gj indicates the probability of existence of the class j in the whole scene.

In this way, global information plays an important and influential role of prior knowledge in semantic segmentation by filtering out impossible results. Obviously, two scenes with the same descriptor are likely to be the same type of scene, and this makes our descriptor be able to substitute the scene label. Furthermore, compared with the manually annotated label, our scene descriptor could subdivide each scene according to the object composition and provide a more concrete global information.

Supervision for Scene Descriptor

Compared with the SpiderCNN [Xu et al., 2018] that requires the ground truth of global label during both training and testing, we generate the scene descriptor through the SceneEncoder module. To effectively train the SceneEcoder module, we regularize the predicted scene descriptor ˜g. Instead of labeling the ground truth of each scene descriptor g manually, we generate it on-the-fly from the labels of all points in the point cloud during the training process. Note that, the ground truth of the scene descriptor is a binary vector of length n, and each element represents the existence of corresponding category in this scene. Therefore, through checking the labels of all the points, we can confirm which classes appear in this point cloud easily. Then, the regularization is as follow:

In the training phase, the SceneEncoder module could learn to represent the type of each scene by encoding the global information into the scene descriptor. As for the validation and testing process, the scene descriptor could be predicted by the well-trained SceneEncoder module and directly aid pointlevel prediction through Eq. (1).

3.3 Region Similarity Loss

Compared with global information, local details could help to generalize to complex scenes. However, during feature aggregation process, previous works ignore the difference among points with different labels in each local context. Such a kind of label inconsistency will lead to the lack of distinguishing ability of point features. Consequently, there are usually poor contours and noisy regions in the segmentation results. Therefore, we propose a novel region-based loss to propagate distinguishing features in local regions, thus those nondistinguishing features of neighbors with the same label can be refined.

Distinguishing Feature Selection

Compared with features achieved from intermediate layers, point features of the last layer encode more local and global information, and directly affect the segmentation results. Hence, our proposed loss directly refines point features of the last feature abstraction layer.

As shown in Figure 2, to select distinguishing point features, we first choose a set of correctly classified points P. Then, among these points, we analyze two strategies to select a fixed number M of points. The first strategy is randomly selection, while the second one is to pick points with the top M classification confidence from P and classification confidence refers to the predicted probability. However, the first strategy is too arbitrary so that some correctly classified points with low confidence may also be selected, leading to incorporate indistinguishable features.

By contrast, the second strategy is more reasonable, which will be confirmed in Sec. 4.3. In practice, if the amount of correctly classified points is less than the given number M, which usually happens in the beginning of training process, the point feature with the highest classification confidence is sampled repeatedly.

Feature Adjustment

Based on the selected distinguishing point features, we adjust other features to boost their distinguishing ability. First, since points of different categories should have different features, we only propagate distinguishing features to points of the same category. Then, we achieve the feature propagation by reducing the difference between each distinguishing feature and features of points in its local neighborhood instead of the whole scene. Because in most cases, features of points that are far away from each other are likely to be dissimilar even if they belong to the same category. If we force to reduce the difference between two features but their corresponding points are far apart, it would hinder the feature abstraction layers from learning the correct point features, making the deep network hard to train.

Therefore, as shown in Figure 2(b), we utilize a KNN with label strategy to select neighboring points of same category, and only propagate each distinguishing feature in corresponding neighborhood. Concretely, for each point pi with distinguishing feature, its k nearest neighbors pni1 , pni2 , ..., pnik with the same label are adjusted. To minimize the difference between each neighboring point feature and the center distinguishing point feature, we use the cosine similarity to define the region similarity loss as follows:

where M is the number of distinguishing point features and k is the number of selected neighboring points. fpi and fpnij are features of center point and neighboring point, respectively.

In addition to cosine similarity, Euclidean distance can also be used as a distance measurement. However, it is sensitive to the value of each dimension of each point feature, reducing generalization ability of the loss. On the contrary, cosine similarity could make our proposed loss more robust to point clouds in different scales.

3.4 Total Loss

In summary, the total loss function consists of three parts: Lcls, Ldes, and Lrs. Lcls is the cross entropy loss applied to constrain the point-level predictions. Ldes is defined in Eq. (2) to improve the performance of SceneEncoder module. Furthermore, Lrs is the proposed region similarity loss to boost the distinguishing ability of point features as in Eq. (3). Therefore, the total loss function is as follows:

where λ1, λ2 and λ3 adjust ratio of the three losses.

In the training process, we inhibit the gradient back propagation from Lcls to ˜ g, that is to prevent the scene descriptor from being regularized by Lcls. Because, Lcls is a point-level classification loss while our scene descriptor is a global descriptor, which would lower the training performance. In our experiments, λ1 and λ2 are set the same, but we dynamically change λ3 during training. In the beginning of training process the selected features are not distinguishing enough, so the impact to other features should be weakened. Therefore, we assign a small weight to λ3 in the beginning and increase the weight gradually until it is the same as λ1 and λ2. In the testing process, we do not generate the ground truth of scene descriptor any more.

4 Experiments

Experiments consist of three parts. First, we demonstrate the effectiveness of our method on semantic segmentation task in Sec 4.1. Second, we generalize our method to part segmentation and prove its effectiveness on part segmentation in Sec 4.2. Finally, we conduct ablation study to demonstrate the effectiveness of the proposed SceneEncoder module and region similarity loss in Sec 4.3.

Metric. Like previous works, we take the intersection-overunion (IoU) as our metric, i.e., mean IoU over categories for semantic segmentation task, class average IoU and instance average IoU for part segmentation task.

4.1 Scene Semantic Segmentation

Dataset. We evaluate the performance of scene semantic segmentation task on ScanNet v2 [Dai et al., 2017]. ScanNet v2 consists of 1,201 scanned scenes for training and 312 scenes for validation. Another 100 scenes are provided as the testing dataset. Different from most previous works using both geometry and color information, we just take the geometry information (i.e., xyz) as the input.

Implementation details. We insert our SceneEncoder module into PointConv [Wu et al., 2019], in which point features of the last layer are regularized by our region similarity loss during training process. Following [Wu et al., 2019], training samples are generated by randomly sampling 3m×1.5m×1.5m cubes from rooms, and then we test over the entire scan. The network is trained by using Adam optimizer with batch size 8 on a single GTX 1080Ti GPU.

Results. We report mean IoU (mIoU) over categories in Table 1, where a clear improvement over backbone (PointConv) can be observed, i.e., 7.2% in mIoU, and our method achieves a state-of-the-art performance in ScanNet benchmark. As aforementioned, we only take the location information as our input and it also performs better than lots of methods that use extra color information. Figure 3 visualizes the scene semantic segmentation results of PointConv [Wu et al., 2019] and our methods.

4.2 Part Segmentation

Similarly, our methods can also improve the performance on part segmentation. In part segmentation task that purposes to segment the functional part of each object, we can treat each object as the entire scene point cloud, and the parts of each object can be considered as the different objects in the scene. Therefore, to confirm the generalization ability, we evaluate our method on part segmentation.

Dataset. We evaluate the performance on ShapeNet part data [Chang et al., 2015]. ShapeNet part data consists of 16,881 point clouds from 16 categories and totally contains 50 parts, with xyz and norm vectors being taken as the input.

Implementation details. We still choose PointConv as the baseline and integrate our method into it. We directly take the raw point cloud as input without preprocessing. The Adam optimizer is employed to train the model with our region similarity loss on a single GTX 1080Ti GPU.

Results. We report class average IoU (mcIoU) and instance average IoU (mIoU) for part segmentation in Table 2. The 0.6% increment over PointConv in term of mcIoU shows the great potential of the proposed method in part segmentation. Overall, the improvements on both scene semantic segmentation task and part segmentation task show the generality of our method. However, even obvious improvements have been shown in mcIoU, there is little increase in mIoU, seeing the rightmost column in Table 2. The reason might be attributed to the fact that: the target scene descriptors are very similar for instance of the same categories, while the predict scene descriptor is able to learn the pattern for one categories but not good at distinguishing the subtle difference among instances. The visualization of some results is shown in Figure 4.

4.3 Ablation Study

To prove that scene descriptor could describe the global information and substitute the global category label, we conduct experiments on ShapeNet with SpiderCNN [Xu et al., 2018] as the backbone, since it directly adopts category label. Then, without loss of generality, we conduct more ablation study on semantic segmentation task on ScanNet and choose PointConv as the baseline.

SceneEncoder module versus category information. For part segmentation, category labels are not always available, as for scene semantic segmentation, it is even hard to give a label to a scene. By contrast, our SceneEncoder module learns to represent the global information with the scene descriptor extracted from the point cloud, even with no extra information of category label. To show the effectiveness of scene descriptor in representing global information, we conduct the ablation study for different combinations with SpiderCNN as backbone. [Xu et al., 2018] releases the result of SpiderCNN with category information, and we conduct three extra experiments under the same setting, i.e., SpiderCNN with no category information, SpiderCNN with SceneEncoder module, and SpiderCNN with both category information and SceneEncoder module. As shown in Table 3, SpiderCNN with SceneEncoder module but without category information perform as well as SpiderCNN with only category information. Therefore, it indicates that the scene descriptor could also represent the global information and substitute the global category label. Additionally, compared with [Xu et al., 2018], combining the SceneEncoder and category label can increase the class average IoU by 0.7%. This result illustrates that the SceneEncoder enhances the effect of global information.

SceneEncoder module and region similarity loss. To evaluate the impact of the SceneEncoder module and the region similarity loss, we conduct two experiments on ScanNet. First, we insert our SceneEncoder module into the PointConv and train the model without region similarity loss. Second, we just introduce the region similarity loss into training process of the PointConv without the SceneEncoder module. The results are shown in Table 4, where we can observe that the SceneEncoder module and region similarity loss individually improve the performance. The combination can deal with both the global and local feature in a better way, hence further improve the performance of PointConv by a large margin.

Selection strategy of region similarity loss. In order to show the effectiveness of our selection strategy of distinguishing point features, we design experiments to study the performance of the region similarity loss using another selection strategy. Namely, we select distinguishing point features among correctly classified points randomly. As shown in Table 5, selecting distinguishing point features with high confidence could help to improve 2.5% on mIoU, which proves the effectiveness of our proposed selection strategy.

5 Conclusion

In this paper, we propose a scene-aware semantic segmentation method with a SceneEncoder module and a region similarity loss. The SceneEncoder module makes full utilization of global information to predict a scene descriptor and this scene descriptor can aid the point level semantic segmentation by filtering out impossible results. To further refine the contour of segmentation, we propose the region similarity loss to propagate distinguishing features by forcing points in each local region with same labels to have similar final features. Overall, the proposed methods achieve the state-ofthe-art performance on ScanNet dataset for semantic segmentation and ShapeNet dataset for part segmentation.

Acknowledgments

We thank for the support from National Natural Science Foundation of China (61972157, 61902129, 61772524, 61876161, 61701235, 61373077, 61602482), National R&D Program (SQ2019YFC150159), Shanghai Pujiang Talent Program (19PJ1403100), Beijing Municipal Natural Science Foundation (4182067), Economy and Information Commission of Shanghai (XX-RGZN-01-19-6348), Fundamental Research Funds for the Central Universities associated with Shanghai Key Laboratory of Trustworthy Computing. Xin Tan is also supported by the Postgraduate Studentship (by Mainland Schemes) from City University of Hong Kong. Jingyu Gong is also supported by Wu Wen Jun Honorary Doctoral Scholarship, AI Institute, Shanghai Jiao Tong University.