Importance-Aware Semantic Segmentation for Autonomous Driving System

1 Introduction

Semantic Segmentation (SS) partitions an image into regions that represent meaningful objects, which serves as an important tool for the subsequent image analysis such as scene understanding. In recent years, autonomous driving system has gained much popularity, in which SS has played an important role in detecting obstacles and recognizing road conditions.

Apparently, good SS results in autonomous driving system will help it precisely understand the scene, and thus leading to safe decision-making and vehicle control. However, we argue that the SS for autonomous driving system is quite different from the conventional SS problems. For conventional SS, all the objects appeared in an image are of equal importance and one should segment all of them from the image as accurately as possible. In contrast, the objects in the scene are not equally important for autonomous vehicles. For instance, the selfdriving system should pay more attention to the objects that are closely related to safe-driving than those that are not often used for vehicle control. In other words, the SS algorithm in autonomous vehicles should segment the major obstacles and potential driving risks (e.g. pedestrians, cyclists, other vehicles, and traffic signs) with a high precision, while reducing the attention on processing less important objects such as sky, grassland and sun.

In this sense, current SS methods are not suitable for dealing with autonomous driving problem. For example, the traditional works [Shotton et al., 2008; Ladicky et al., 2010; Gong et al., 2015] based on handcrafted features and recent Deep Convolutional Neural Network (DCNN) based methods [Long et al., 2015; Vijay Badrinarayanan and Cipolla, 2017; Paszke et al., 2016] equally treat all the classes. As a result, they generate very low accuracy on segmenting the important objects as mentioned above.

From above analyses, we see that existing methodologies cannot render reliable segmentation results for autonomous driving, as they all adopt the cross-entropy [de Boer et al., 2005] loss function for model training, which equally evaluates the errors incurred by all image pixels without focusing on the important objects. Therefore, a novel importanceaware loss function should be specifically designed for the application of automatic driving. To this end, we introduce the notion of class importance where pedestrians, vehicles and other objects on the road are more important for driving than other classes such as sky and remote buildings that are off the road. Based on this notion, we design a novel loss function termed "Importance-Aware Loss" (IAL) that is able to put more emphasis on accurately segmenting the important objects than less important ones.

Inspired by [Szegedy et al., 2013], we propose a novel loss function with hierarchical structure as shown in Figure 3. In this structure, the objects with different importance are located in different levels, and the more important an object is, the higher level it stands. Consequently, the important objects are in higher levels than the unimportant ones, and thus they are multiplied by larger importance factors for computing the final loss. To validate our proposed loss function, we replace the cross-entropy loss utilized by representative deep learning methods [Long et al., 2015;

Vijay Badrinarayanan and Cipolla, 2017; Paszke et al., 2016] with our proposed importance-aware loss. The experimental results on two typical autonomous driving datasets including CamVid [Brostow et al., 2009] and Cityscapes [Cordts et al., 2016] demonstrate that the important objects can be segmented more precisely than existing approaches.

The rest of this paper is organized as follows. Some related works are reviewed in Section 2. After that, we describe the proposed loss function and also the relationship with existing cross-entropy loss in Section 3. In Section 4, we derive the forward and backward propagation rules for our proposed loss. In Section 5, we provide the experimental results on the representative traffic datasets including CamVid and Cityscapes. Finally, our paper is concluded in Section 6.

2 RelatedWork

SS has been intensively studied for a long time as it is an important tool for understanding a scene. For example, some traditional methods focus on designing powerful handcrafted features and using Random Forest [Shotton et al., 2008; Brostow et al., 2008; Silberman et al., 2012] or boostingbased [Sturgess et al., 2009; Kontschieder et al., 2011; Zhang et al., 2010] classifiers for predicting the class of image pixels. To improve the segmentation accuracy, some postprocessing strategies have been developed to improve the initial segmentation results. For instance, the techniques based on Conditional Random Fields (CRF) [Sturgess et al., 2009; Ladicky et al., 2010; Ren et al., 2012] are used to suppress the per-pixel prediction noise output by the classifiers.

With the rapid development of deep learning, various deep neural networks have been applied to SS and achieved stateof-the-art performance. The works such as [Farabet et al., 2012; Grangier et al., 2011; Gatta et al., 2014] employ the features extracted by DCNN for class prediction. To make SS an end-to-end process, a fully convolutional network (FCN) [Long et al., 2015] is applied and shows very promising results. Based on [Long et al., 2015], many other methods [Chen et al., 2016; Zheng et al., 2015; Shen et al., 2016; Eigen and Fergus, 2015] are proposed which further incorporate multi-scale manipulation or post-processing based on CRF. Another important architecture for segmentation is based on the structure of encoder-decoder. SegNet [Vijay Badrinarayanan and Cipolla, 2017] and some other works like [Noh et al., 2015; Hong et al., 2015; Yang et al., 2016] belong to this type.

Recently, several works have been done to apply SS to autonomous driving. [Pohlen et al., 2016] develops a deep neural network for segmenting the major object classes in street scenes and reaches state-of-the-art results on the Cityscapes benchmark [Cordts et al., 2016]. To further improve the efficiency and achieve real-time segmentation, more neural networks are designed for self-driving system such as ENet [Paszke et al., 2016] and the work [Treml et al., 2016].

Although above SS algorithms targeting self-driving have achieved encouraging performance, none of them take the importance of different classes into account, so their results are not reliable for autonomous driving. Therefore, this paper presents the concept of class importance and proposes a novel loss function with hierarchical structure. By embedding the proposed loss to three representative deep networks such as FCN, SegNet and ENet, we will show that our loss is able to drive the networks attention to important objects during self-driving.

3 The Proposed Loss Function

As mentioned in the introduction, different object classes have different levels of importance for autonomous driving, so this section introduces our proposed loss function that takes the importance information into consideration. CamVid [Brostow et al., 2009] is a widely used dataset for evaluating the self-driving performance, in which the image data is captured from the perspective of a driving automobile. This dataset suggests 11 meaningful object classes that are often appeared in a driving scenario, and in this section we use these 11 suggested classes for explanation. First of all, safety is the most critical issue for driving and the collisions with car, pedestrian and bicyclist are strongly opposed, so these objects show the top level importance in our algorithm. In contrast, road, sidewalk, and sign/symbol are less important as they only guarantee the normal driving. Sky is not essential here as it is seldom used as a cue for car control, so it is the least important among all above 11 classes. The detailed importance levels of all the investigated classes are depicted in Figure 1. Besides, it is worth mentioning that the users can re-define the objects' importance levels according to different criteria or their own prior knowledge.

To characterize the importance of objects in our method, we first define some notations for the ease of following descriptions. We define C as the number of classes in driving environment. The final output of an SS algorithm can be represented by a tensor X ∈ RC×Himg×Wimg where its height and width correspond to a Himg ×Wimg input image, and its depth targets the one-hot encoding of the ground truth and indicates the class of each of the Himg ×Wimg pixels (see Figure 2(a)). Here the one-hot encoding has the formation [0, · · · , 0, 1, 0, · · · , 0]T with the position of the correct label being 1. Besides, the segmentation ground truth of an image is denoted by a matrix Y ∈ NHimg×Wimg with the (i, j)-th element Yi,j ∈ {1, 2, · · · ,C} representing the real class label of the (i, j)-th pixel.

According to above mathematical definitions and the importance levels as shown in Figure 1, we propose a novel importance-aware loss with hierarchical structure as shown in Figure 3, in which different levels represent the objects with different importance. In Figure 3, the vectors lG1 , lG2 , lG3 and lG4 encode the values of cross-entropy loss [de Boer et al., 2005] of the objects in Group 1, Group 2, Group 3, and Group 4, respectively, and the j-th element (lGi )j (i = 1, 2, 3, 4) is calculated by

where pc = exp(Xc,i,j)/ ∑Ck=1 exp(Xk,i,j) is the probability of the (i, j)-th pixel belonging to the c-th class (c takes a value from 1, 2, · · · ,C) based on the output X, and q is a one-hot encoding with the c-th element qc being 1. Similar to the formation of lGi , we use the vectors wG1 , wG2 , wG3 and wG4 to record the corresponding weights of the objects in the four groups for avoiding class imbalance, and the object with fewer pixels is assigned larger weight [Eigen and Fergus, 2015]. The j-th element in wGi (i = 1, 2, 3, 4) are

where freq(i, j) is the number of pixels of the j-th class in Group i divided by the total number of pixels in images where the class (Gi)j is present, and median freq is the median of these frequencies. Therefore, the weighted cross-entropy losses for Group 1 to Group 4 are wTG1 lG1 , wTG2 lG2 , wTG3 lG3 and wTG4 lG4 correspondingly.

Besides, for the four groups defined in Figure 3, we introduce three Himg ×Wimg matrices Mt (t = 1, 2, 3) to model the importance relationship of four groups. For example, the M1 for comparing Group 1 and Groups 2, 3, and 4 is presented in Figure 2(b), in which the elements corresponding to the classes in Group 1 are set to 0, and the elements corresponding to Groups 2∼4 are 1 indicating that they are more important than Group 1. To further compare the importance of Group 2 and Groups 3 and 4, the elements of M2 (see Figure 2(c)) regarding Group 2 are set to 0, and the elements of Groups 3 and 4 are defined as 1 because they are more important than Group 2. In M2, the elements indicating Group 1 are denoted as "x" which means that the comparison of Group 1 and other groups has been done before. Similarly, the M3 for comparing Group 3 and Group 4 is shown in Figure 2(d), where the elements of Group 3 and Group 4 are 0 and 1, respectively. The elements representing Group 1 and Group 2 are "x" since their importance comparisons with other groups have been studied.

Based on Mt (t = 1, 2, 3), we define ft(X) + α as importance factor where α is a tuning parameter with default value 1, so the ft(X) (t = 1, 2, 3) in Figure 3 are computed by

where E is an all-one matrix, and I{Mt ̸= "x"} returns a matrix where its element is 1 if the corresponding element (Mt)i,j is not "x", and 0 otherwise. The notation "⊙" denotes the element-wise product of two matrices. X is a matrix with the same dimension of Y and its (i, j)-th element is defined by Xi,j = Xc,i,j 1 with c = Yi,j. In (3), λt ∈ R+ (t = 1, 2, 3) are tuning parameters and in this paper we set λ1 = λ2 = λ3 = 0.5. Note that if λt is small, the value of ft(X) will be large due to the error between Xi,j and (Mt)i,j when (Mt)i,j = 1 (i.e. the corresponding class is important). By this way, Eq. (3) encourages the model to focus on the classifications of important classes.

1 Here all elements belonging to the (i, j)-th pixel (i.e. "X:,i,j" in Matlab expression) have been normalized to [0, 1].

Therefore, the loss of the objects in the four groups can be computed by following the arrows in Figure 3. For instance, Group 1 has the lowest importance level, of which the importance-aware loss is wTG1 lG1 ; The weighted crossentropy loss of Group 2 should be multiplied by an importance factor f1(X) +α, so its importance-aware loss should be (f1(X) + α)(wTG2 lG2 ). Similarly, the loss value of the classes in Group 3 is (f1(X) + α)(f2(X) + α)(wTG3 lG3 ). The classes in Group 4 are the most important and thus its weighted cross-entropy loss wTG4 lG4 should be augmented by three importance factors. Consequently, the loss of Group 4 is (f1(X)+α)(f2(X)+α)(f3(X)+α)(wTG4 lG4 ). Finally, the total value of our importance-aware loss is the sum of the loss values contributed by the four groups, which is

From above analyses, we see that the cross-entropy losses of important classes will be augmented by more importance factors than the less importance ones. As a result, the more important an object class is, the greater the importance-aware loss value it will obtain. Moreover, from Eq. (4) we see that if we set all importance factors ft(X)+α (t = 1, 2, 3) to 1, our proposed IAL function will degrade into the existing crossentropy loss with all classes sharing the equal importance.

4 Forward and Backward Propagation Rules

Here, we give a general description of the proposed loss function, and then deduce its related forward and backward propagation rules.

Suppose we have totally C classes that are grouped into g groups G = {G1,G2, · · · ,Gg} which satisfy Gi ̸= ∅ and Gi ∩ Gj = ∅. For these g groups, their cross-entropy losses and corresponding weights avoiding class imbalance are {lG1 , lG2 , · · · , lGg } and {wG1 ,wG2 , · · · ,wGg }, respectively.

According to the above description, the forward propagation rule of the proposed loss function is

· · · · · ·

where Qt+1 = (ft+1(X) + α)(wTGt+2 lGt+2 ) corresponds to the most important group. Therefore, the compact formation of the forward propagation rule regarding our IAL is

As a consequence, the backward propagation rules of IAL corresponding to Eqs. (8), (5), (6), and (7) are

· · · · · ·

where

and

By denoting

we have

where c = Yi,j .

For ∂lGt ∂X , if the (i, j)-th pixel belongs to class (Gt)r (i.e. the r-th class in Group Gt), and its corresponding weight is (wGt )r, we obtain

and then (wTGt ∗ ∂lGt ∂X ):,i,j is represented by

5 Experimental Results

To verify the effectiveness of our proposed importance-aware loss (IAL) function, we apply IAL to three existing deep neural networks, i.e. FCN [Cordts et al., 2016], SegNet [Vijay Badrinarayanan and Cipolla, 2017] and ENet [Paszke et al., 2016], to deal with SS problem. Among them, FCN and SegNet are popular deep methods for conventional SS, and ENet is a recently proposed deep network specifically for autonomous driving application. The cross-entropy loss adopted by these models will be replaced with our IAL during the training stage, and we term them as "FCN+IAL", "SegNet+IAL", and "ENet+IAL", respectively. Besides, we follow [Cordts et al., 2016; Everingham et al., 2015] and use the intersection-over-union (IoU) score to evaluate the performances of compared methods on different object classes.

We use the CamVid dataset [Brostow et al., 2009] mentioned in Section 3 and a recent Cityscapes [Cordts et al., 2016] dataset for our experiments. CamVid contains 367 training images, 26 validation images, and 233 test images. The resolution of images in this dataset is 960 × 720. Cityscapes is also a high-quality dataset for semantic scene understanding captured from the view of cockpit, which contains 2975 color training images, 500 validation images, and 1525 test images. The resolution of all images is 2048×1024. In Cityscapes dataset, we pick up 19 the most frequently occurred classes from the original 35 classes, and their importance groupings from trivial to important are

Group 1 = { Sky };

Group 2 = { Building, Wall, Fence, Vegetation, Terrain };

Group 3 = { Road, Sidewalk, Train };

Group 4 = { Person, Rider, Car, Truck, Bus, Motorcycle, Bicycle, Traffic light, Traffic sign, Pole }.

On both datasets, we train the neural networks on training sets, and observe their IoU scores on test sets. The experimental results of compared methods on the investigated classes of the two datasets are shown in Tables 1∼2 and Tables 3∼4, respectively. For the CamVid dataset, the results of ENet and SegNet are directly originated from [Paszke et al., 2016], and we implement FCN by ourselves as no prior results on this dataset have been reported. For the Cityscapes dataset, the results of ENet, SegNet and FCN in Tables 3 and 4 are provided by the original papers [Paszke et al., 2016; Pohlen et al., 2016; Cordts et al., 2016].

From the results shown in Tables 1 and 2, we observe that by employing our IAL, the IoU scores of important classes like pedestrain, bicyclist, car and sign/symbol can be significantly improved when compared with the settings without IAL. Not surprisingly, the IoU scores on some unimportant classes such as building and sky drop because they are trained with small weights by our IAL. However, if we compute the IoU scores averaged over all classes for all compared methods (see the last column in Table 2), we find that the networks with IAL are still able to achieve better performance than the original networks with cross-entropy loss, and the improvements are 1.6 for ENet, 1.5 for SegNet, and 1.6 for FCN.

From the results in Table 3 and Table 4, we see that the important classes in Group 4 are segmented with very high IoU scores by FCN+IAL, ENet+IAL and SegNet+IAL, such as person, rider, car, truck, bus, and bicycle. Specifically, the IoU scores of person and truck generated by ENet+IAL are as high as 87.7 and 73.5, which are significantly better than the results of ENet, FCN, and SegNet. For some unimportant classes in Group 2, the performances of IAL-based models are inferior to the original models. However, they will not have large impact on safe-driving as explained above. Furthermore, the last column of Table 4 reveals that the segmentation results of ENet, SegNet and FCN on the entire image are improved by utilizing our IAL. Besides, another interesting finding is that sky, which is inessential, is also segmented more precisely by IAL-based models than the corresponding baselines. Perhaps this is because the accurate segmentations of other objects also render valuable cues for partitioning the unimportant regions.

To intuitively present the effectiveness of our proposed loss function, we provide some representative segmentation results of ENet and ENet+IAL in Figure 4. For the important objects with large size (e.g. truck, bus, and road), we see that the regions segmented by ENet+IAL are very compact and most pixels of the corresponding regions are correctly classified. Comparatively, the original ENet yields much worse outputs than ENet+IAL such as the incomplete truck, bus, and road. For the important objects with small size (e.g. traffic light, person, and pole), the ENet+IAL also generates more similar segmentation results to ground truth than ENet. For example, the traffic light indicated by a circle is rather small, and it is missed by ENet. However, our ENet+IAL successfully picks it up and renders accurate segmentation. The pole in the last row is so tiny that it is completely misclassified by ENet. In contrast, ENet+IAL clearly identifies the pole from the background as indicated by the white circle. Here we only present the results related to ENet as ENet is the state-of-theart deep neural network specifically designed for the application of self-driving. However, the comparisons between FCN vs. FCN+IAL and SegNet vs. SegNet+IAL also reveal the similar results.

According to above qualitative and quantitative results, we conclude that the proposed hierarchical importance-aware loss can improve the segmentation quality of the important objects with a large margin in terms of IoU score. Therefore, IAL is quite suitable for the application of autonomous driving.

6 Conclusion

Semantic segmentation in driving environment is quite different from its traditional implementations, as various classes might have different levels of importance for safety driving. Based on this argument, this paper proposes a novel hierarchical importance-aware loss (IAL) so that the object classes with different importance are adaptively allocated different weights during the model training stage. As a result, the objects that are critical for safe-driving can be segmented more accurately than the traditional SS methods as revealed by the experiments. Moreover, our loss function IAL is general in nature and can be easily combined with many other existing SS algorithms for various applications with the consideration of class importance.

Acknowledgements

This work was supported by the National Natural Science Foundation of China (Nos. 61602246, 91420201, and 61472187), the 973 Program (No. 2014CB349303), and the Program for Changjiang Scholars.