Towards Cross-View Consistency in Semantic Segmentation While Varying View Direction

1 Introduction

Semantic segmentation, which aims to label each pixel of a given image with a certain semantic class, is one of the fundamental missions in computer vision. Benefiting from the strong feature learning ability of the Convolutional Neural Networks (CNNs) and rich datasets, previous semantic segmentation methods have obtained promising performances. However, the generalization ability related to varying view direction of semantic segmentation algorithms has not been paid enough attention yet. When it comes to semantic segmentation, an intuitive motivation is that if we observe the same object via different view directions at the same viewpoint, the semantic labels of the object should be the same.

∗Corresponding Author

We call it cross-view consistency while varying view direction. The consistency is crucial to keep stable and robust results in higher-level computer vision tasks such as 3D reconstructions, autonomous driving, AR/VR, etc. Unfortunately, as shown in Fig. 1, we find the existing semantic segmentation algorithms are often not able to keep cross-view consistency while varying view direction. Moreover, images in popular urban scene datasets such as Cityscapes [Cordts et al., 2016] and CamVid [Brostow et al., 2008] are acquired by mounted cameras in moving vehicles. View directions of these images are often near urban street directions, i.e., along driving directions, meanwhile images with other view directions are a minority. The CNN-based models trained on these datasets will be constrained and overfit in specific view directions due to the lack of data with diverse view directions. Specifically, the segmentation performance drops a lot on images with richer view directions. Meanwhile, the inconsistency problem is more severe in urban scenes.

In this work, we aim to modify existing CNN-based semantic segmentation algorithms by enhancing their ability related to cross-view consistency while varying view direction. The modification can be easily applied in any CNN-based semantic segmentation methods. No extra annotated images are necessarily needed during training, since new manually annotated images for semantic segmentation are usually costly and time-consuming to acquire. In the modification, we propose three general add-on modules. We first present an online varying view direction (VVD) data generator to provide images with abundant view directions for training. The key insight is to utilize carefully generated homography to generate new image of the same scene in another view direction and get correspondent pixel pairs. Then we propose gradient guided deformable convolution module, which can learn dynamic proper receptive field from feature gradients. Furthermore, a novel consistency loss is proposed to enforce the consistency between features at corresponding positions in different feature maps. For evaluation, We derive VVD datasets which contain annotated images with diverse view directions from common-used urban datasets. The proposed method efficiently improves semantic segmentation methods, especially in terms of the cross-view consistency.

The contributions of this paper can be summarized below: (1) Our work is the first one that focuses on the problem that previous semantic segmentation methods may not keep crossview consistency even if the viewpoint is unchanged and only the view direction is varied; (2) To alleviate the inconsistency, we propose a novel online varying view direction (VVD) data generator to dynamically generate images in different view directions using carefully generated homography. The generated images can be treated as real photos of the scene taken by the camera with a new direction exactly. We also construct VVD datasets that contain more challenging annotated images in plentiful view directions for evaluation; (3) We propose a gradient guided deformable convolution module and a novel consistency loss, which can be added to segmentation network easily and enforce network to predict consistent semantic predictions; (4) The proposed method achieves significant improvement on images with abundant directions in cross-view consistency and segmentation performance, while keeping comparable or better performance on common-used datasets.

2 RelatedWork

Semantic segmentation algorithms. Semantic segmentation methods based on convolutional neural networks have got promising performance. The seminal work [Long et al., 2015] proposed fully convolutional network (FCN) to train the segmentation network end to end and fit for images of any size. [Chen et al., 2017a] and [Yang et al., 2018] applied Atrous/Dilated convolution to enlarge receptive field. Global information and multi-scale features were incorporated to boost the network in [Zhao et al., 2017; Ronneberger et al., 2015; Chen et al., 2016]. Conditional random field (CRF) was used as post processing in [Chen et al., 2017a] to refine the predictions or embedded into the network in [Zheng et al., 2015] to enable end-to-end training. Recent approaches used attention [Huang et al., 2019; Fu et al., 2019; Li et al., 2019; Zhao et al., 2018; Yu et al., 2018], dictionary learning [Zhang et al., 2018], context relationship [Yuan and Wang, 2018; Zhang et al., 2019] and get better results. We focus on improving the cross-view consistency in CNN-based image semantic segmentation, which has not been paid enough attention in the previous methods.

Cross-view consistency. Cross-view consistency is a basic and natural constrain in computer vision. It requests each projection of the same 3D object represents the same semantic meaning. Xiao et al. [Xiao and Quan, 2009] proposed a graph-based optimization approach to enforce consistency of the segmentation result across multiple views. Ma et al. [Ma et al., 2017] warped feature maps into a common reference view and enforced multi-view consistency with various constraints based on RGB-D images. In this work, we use the cross-view consistency in the condition that the view direction is varied and the viewpoint is unchanged. The consistency can be defined by a homography transformation. We apply it in semantic segmentation of 2D RGB images by evaluating and enhancing the cross-view consistency while varying view direction of any given CNN-based algorithm.

Deformation of convolution. Deform the convolution for a better receptive field has been well studied. [Jaderberg et al., 2015] proposed a Spatial Transformer Network (STN) which estimates a group of global parameters to warp the input feature maps. [Jeon and Kim, 2017] proposed a convolution unit with learned offsets to obtain better receptive field for object classification, by learning fixed offsets for feature sampling on each convolution. [Dai et al., 2017] proposed a a more dynamically deformable convolution unit where the image offsets are learned through a set of parameters. The offsets in above methods are learned from the feature maps directly. For our task, when view direction varies, the local receptive fields are supposed to have similar semantic information with a local spatial deformation, e.g., homography transformation. The changes of feature map gradients can reflect the deformation directly. Thus, the offset is learned from the gradient of the feature maps in our approach.

3 Algorithm

In this section, we introduce three proposed modules for improving cross-view consistency in semantic segmentation. Training data are generated by online VVD data generator, then processed by the network modified with gradient guided deformable convolution, and finally supervised by additional consistency loss.

3.1 Online VVDData Generator

Under the assumption of the pinhole model, we make the origin of the camera coordinate coincide with the origin of the world coordinate. The relation between a 3D point P and its 2D projection in an image p can be described as

where [xp, yp, 1]T and [Xp, Yp,Zp, 1]T represent the homogeneous coordinate of p and P , respectively. The matrix containing focal length fx, fy, principal point offset (u0, v0) and axis skew s is camera intrinsic matrix. R represents the rotation matrix between the world coordinate system and the camera coordinate system. λ is a normalization factor.

When 3D point P appears in two images with different views directions, the relation between the two projections (x′p , y′p ) and (xp, yp) can be described as

where Rz(α), Ry(β), Rx(γ) represents the rotation matrix of the two view directions and can be decomposed into three rotation matrices based on three Euler Angles, respectively. H is the so-called homography with shape of 3×3. As shown in Fig. 2, two images of different views are connected with a homography H. Equation (2) can be used to exhaustively and accurately find out corresponding pixel pairs that should have the same semantic label.

In order to get corresponding pixel pairs in images of different view directions, we propose an online VVD data generator. During training, the proposed online VVD data generator is used to produce a pair of image patches and the homography between them. We randomly crop a patch with the size of 2a × 2a in the original image for the first one and record the crop center o. As to the second image, we firstly warp the original image with a generated homography H. And we find the correspondent position of o in the warped image, which is denoted as o′ . Then a patch of the same size is cropped in the warped image with center o′ . Thus, the homography matrix between the two image patches can be described as

where xo, yo denote the x and y coordinate of point o. xo′ , yo′ represent the x and y coordinate of point o′ , which can be calculated by

3.2 Gradient Guided Deformable Convolution

Convolution can be seen as a feature sampling followed by a weighted sum operation. It gathers information from the receptive field of the convolution kernel in the input feature maps. After convolution operation, the value of location p on the output feature map y can be calculated as

where w is the weight of the convolution kernel and x is the input feature map. δp enumerates the pixel relative location in grid R. For a standard 3 × 3 convolution, the regular grid R is represented as

For deformable convolution, R is not constrained in a rigid formulation and may vary when the position p changes. Thus, Eq. (5) becomes

where F(x, p, δp) represents the learnable augmented offset. In standard deformable convolution, the grid is more likely to be driven by semantic information. For two images containing the same object via different view directions, the local receptive fields are supposed to have the similar semantic information with a local spatial deformation. The changes of feature map gradients reflect the deformation directly. Thus, we design a gradient guided deformable convolution module, in which the grid offset is driven by gradient information from the input feature maps. The output of the proposed gradient guided deformable convolution can be represented as

where G(x) represents the gradients of the input feature maps x. The gradient guided deformable convolution is illustrated in Fig. 3. In practise, the gradient extractor G and offset generator F are respectively implemented using Sobel operator and convolutions for end-to-end training. The novel gradient guided deformable convolution can readily replace standard convolution in existing segmentation networks.

3.3 Consistency Loss

To enforce the corresponding pixel pairs have the same inferred labels, we constrain output feature maps with a novel consistency loss in a deeply self-supervised way during training. When two images in different view directions are fed into the network, the feature columns at the corresponding positions in two feature maps are supposed to be the same. For them-th selected layer, let xmi and ym i be the i-th positions of corresponding pixel pairs. We use cosine similarity to calculate the distance between two correspondence columns of the feature maps, followed by a linear operation to normalize and a log operation to magnify the penalty for the error. The consistency loss for the m-th selected layer can be represented as

where N is the number of corresponding pixel pairs which appear in both the images. Assuming that we consider total M layers in calculating the consistency loss, the final consistency loss can be represented as

Finally, the total loss L is the sum of the standard cross entropy loss in semantic segmentation Lseg and the proposed consistency loss Lcons with a weighting factor α, which can be written as

4 Experiments

4.1 Implementation Details

Our training and evaluation is implemented in PyTorch. Resnet50 with the dilated network strategy is used as our backbone. We replace the last 3×3 convolution layer of each residual block with the proposed gradient guided deformable convolution module. Outputs of all residual blocks are selected to be supervised by the consistency loss and α is selected to 1 after a parameter study. For training, we use SGD optimizer and employ the polynomial learning rate policy [Chen et al., 2017a; Liu et al., 2015] where current learning rate equals to the initial one multiplying (1- iter max iter )power . The initial learning rate and the power are set to 0.01 and 0.9, while the momentum and weight decay are set to 0.9 and 0.0001 respectively. Due to GPU memory limitations, we use a batch size of 8 and crop size of 776 during training.

4.2 Ablation Study on Cityscapes

We conduct an ablation study on a widely adopted urban semantic segmentation dataset Cityscapes [Cordts et al., 2016] to verify the effectiveness of our method. Cityscapes dataset contains 5000 high quality pixel-level finely annotated images including 2975 images for training and 500 for validation. For evaluating the cross-view consistency and the segmentation performance of images in diverse view directions, we derive varying view direction (VVD) dataset from Cityscapes. New images and annotations are generated by a warp followed by a crop operation. For each image in the datasets, we randomly generate a homography matrix according to Equation (2). The yaw, pitch and roll angles used in homography generation are set in range of [-30, 30], [-15, 15] and [-3, 3] respectively. The focal length f is set to 2262 following [Godard et al., 2017]. (u, v) are set to half of the width and length of the images. We use the matrix to warp the previous image and crop the maximum inscribed rectangle of it. The same operation is applied to the annotation. Correspondent pixel pairs of the previous image and the generated one can be found out using the homography. The constructed VVD dataset including annotated images containing the same scenes and in different view directions from Cityscapes and the corresponding pixel pairs between each image and its original image.

We employ mean intersection-over-union metric (mIoU) and pixel accuracy for evaluating the consistency. mIoU measures them in a class-balance way while pixel accuracy does in a pixel-balance way. After inferring an image pair from the VVD datasets with a given algorithm, we can get two segmentation maps. As the positions of all correspondent pixel pairs are known according to the homography and the semantic labels of them should be the same, the cross-view consistency can be measured by considering one prediction as ground truth and evaluating another.

In the ablation study, we use PSPNet50 [Zhao et al., 2017] consisted of Resnet50 and a pyramid pooling module as our baseline method. The baseline method is trained with standard cross entropy loss. We gradually add the proposed modules including online VVD data generator, consistency loss and gradient guided deformable convolution. We evaluate the cross-view consistency using VVD dataset and the segmentation performance using both Cityscapes and VVD dataset. For the method only using cross entropy loss, the training data is trained for 90K iterations. For the other methods, to ensure the network provides nontrivial solutions and make it easier to converge, the consistency losses are added after 45K iterations warm-up training. The total iterations are set to 90K to guarantee the fairness of comparison.

The semantic segmentation and cross-view consistency results are listed in Table 1. We find the segmentation performance of the baseline method drops severely when dealing with images in abundant view directions in VVD dataset. All the metrics, especially in VVD dataset are improved with the use of different novel modules. The proposed method with all three modules gets the best performance in the comparison. Meanwhile, it obviously reduces the gap between the segmentation performance on VVD dataset and Cityscapes. For visual comparison, some segmentation examples are shown in Fig. 4. The proposed method improves the segmentation performance in vehicles and traffic signs compared with the baseline method. We also visualize the cross-view consistency results in Fig. 5. We gather the statistic of improvement for each class in Cityscapes. We get improvement in all 19 classes and present the highest 9 classes for clear visualization, as shown in Fig. 6.

A parameter study of α is performed for the consistency loss to investigate its impact on the performance in VVD dataset deriving from Cityscapes. Semantic segmentation performance is considered in this part. We use different α from 0.1 to 5 in the proposed consistency loss and keep other experiment details the same. The results are listed in Table 2. The best choice for α is 1 with respect to segmentation.

4.3 Performance on Different Architectures

We evaluate our method with different network architectures in this section. The three novel modules are applied on DeepLabV3 [Chen et al., 2017b] and CCNet [Huang et al., 2019], respectively. The results are listed in Table 3.

We also evaluate our method in CamVid dataset. CamVid dataset contains 701 images and their pixel-level segmentation annotation with size of 720×960. We use 468 images to train and 233 images to validate following [Badrinarayanan et al., 2017]. We derive new VVD dataset from CamVid for evaluating the performance. The intrinsic matrix is obtained using [Li et al., 2010]. Model pretrained on Cityscapes in used as a start point in the experiment. We train both the baseline method and our method for 10K iterations. The results are shown in Table 4.

4.4 Consistent Video Semantic Segmentation

To evaluate the effectiveness and applicability of the proposed method, we conduct an experiment on video semantic segmentation on Cityscapes dataset. We select 4 frame sequences with total 160 frames from the video demos. In this case the positions of the two cameras can be considered having a small baseline. Although a homography can approximately describe the relationship of the consecutive frames, we use dense optical flow to find the corresponding pixel pairs for more accurate measurement. We measure the consistency of the corresponding pixel pairs between consecutive video frames. Result of one sequence is shown in Fig. 7. The proposed method achieves an average of 70.4% on total sequences, which gains 8.2% over the baseline method. Results show our approach can improve the consistency in more general cases even the position of the camera is changed, though only view directions of images are varied in the training.

5 Conclusion

This paper focuses on improving cross-view consistency in semantic segmentation while varying view direction. We find that the view directions in previous urban datasets are often near directions of streets. In order to alleviate the bias, we propose an online VVD data generator that can modify annotated images to new plentiful view directions and calculate correspondent pixel pair indices, with carefully generated homography transformations. Furthermore, to improve the consistency, we present gradient guided deformable convolution that can readily replace standard convolution in existing segmentation networks. A novel consistency loss is also proposed for extra supervision during training. With the proposed consistency metrics, we evaluate the performance in the generated VVD datasets containing images with abundant view directions compared to the common-used datasets. Experimental results show that our method improves the segmentation performance of the convolutional neural network, making it more robust to view direction changes.

Acknowledgments

This work was supported in part by State Key Development Program Grand No. 2020YFB1708000, and NNSFC Grant No. 61971008.