Self-supervised Monocular Depth and Visual Odometry Learning with Scale-consistent Geometric Constraints

1 Introduction

The depth and pose estimation from images is essential for many applications, such as augmented reality and self-driving cars in robotics and computer vision. Traditional methods are mainly hand-crafted features systems. With the progress of deep learning, depth can be predicted from a single image by Convolutional Neural Network (CNN) in a supervised paradigm [Eigen et al., 2014], [Laina et al., 2016], [Li et al., 2018a]. However, these methods are limited by requiring large amounts of labeled data and it is challenging to collect ground truth depth especially in outdoor environments. Without the need for annotated depth, self-supervised monocular depth estimation methods from stereo images have been proposed in [Garg et al., 2016] and [Godard et al., 2017]. [Zhou et al., 2017] shows a promising direction that depth and pose estimation models can be jointly trained from single view video sequences without ground truth. These self-supervised methods mainly use view synthesis [Zhou et al., 2016] from adjacent frames as supervision. Previous works [Zhou et al., 2017], [Wang et al., 2018] utilize photometric error or gradient-based losses to penalize the differences among these synthetic views. However, these appearance losses are fragile in illumination variations environments and lack relevant constraints to get global scale consistent visual odometry results as proposed in [Bian et al., 2019]. In this paper, to tackle these issues, we leverage more geometric information to formulate supervision beyond using appearance losses. The proposed method is also based on a self-supervised learning framework of depth and pose estimation from monocular videos. Different from the recent work [Bian et al., 2019] directly using the predicted depth, we first derive the reconstructed depth maps estimated from consecutive frames and then use a simple but efficient approach to enforce the depth scale consistency. Besides depth reconstruction loss, we simultaneously consider the forward-backward relative poses error in our geometric loss function to get more accurate results. Depth and pose networks are jointly trained by the proposed loss functions and tested separately in the respective task. Qualitative evaluation on the KITTI Eigen test split is shown in Fig. 1 and the depth estimation results of the proposed method have significant improvement. Extensive qualitative and quantitative evaluation experiments are conducted in Sec. 4.

∗Corresponding Author

Our contributions are summarized as follows: i) we propose novel scale-consistent geometric constraints by simultaneously considering depth reconstruction error and forwardbackward relative poses consistency as supervision; ii) a novel self-supervised learning framework with the proposed loss functions is presented to get accurate depth estimation and scale-consistent visual odometry results; iii) the proposed models can achieve not only SOTA depth estimation performance on KITTI and Make3D datasets but also SOTA pose estimation results on the KITTI Odometry dataset comparing with self-supervised methods.

2 RelatedWork

Traditionally, depth and pose estimation are mostly solved by hand-crafted features methods, which are the basics of many SLAM and SfM algorithms. ORB-SLAM [Mur-Artal et al., 2015] is typically a visual SLAM system that is based on ORB [Rublee et al., 2011] features. [Schonberger and Frahm, 2016] revisits SfM algorithms and develops the COLMAP system. Recently, with the development of deep learning, CNNs have been successfully applied to estimate the monocular depth and ego-motion.

2.1 SupervisedMethods

[Eigen et al., 2014] firstly introduces deep learning to estimate single-view depth in a supervised fashion. Their network can refine the coarse global prediction locally. [Laina et al., 2016] proposes a fully convolutional residual network to model the mapping between RGB images and depth maps. They use the berHu loss and propose a new up-sampling method. [Li et al., 2018a] tackles monocular depth estimation as a multi-category formulation. They fuse the outputs from their dilated CNN in a hierarchical way and utilize the soft-weighted-sum inference to get continuous depth results.

For VO, [Konda and Memisevic, 2015] firstly develops a deep-learning-based VO system. The approach predicts changes in velocity and direction by using the softmax layer. [Wang et al., 2017] formulates VO as a sequence learning problem. Historical information is introduced to infer current relative motions through a recurrent convolutional network. [Xue et al., 2019] presents a VO framework composed of Memory and Refining components. To distill features from previous results in the Memory unit, they adopt a spatial-temporal attention mechanism to model the Refining component.

These methods are supervised by ground truth and therefore have the limitations of demanding enormous labeled data for training.

2.2 Self-supervisedMethods

By leveraging view synthesis [Zhou et al., 2016] as supervision, the depth and pose estimators can be jointly learned in a self-supervised paradigm. [Zhou et al., 2017] proposes a novel framework for estimating depth and ego-motion using monocular video sequences. The framework consists of two separate deep networks that use the photometric error as a supervisory signal without depth or pose ground truth. Following this work, by considering the consistency of consecutive 3D point clouds, [Mahjourian et al., 2018] introduces an ICP loss for aligning 3D structures to further improve the depth and pose estimation with combining 2D photometrybased losses. UnDeepVO [Li et al., 2018b] can infer absolute scale results from monocular image sequences by utilizing stereo image pairs as training datasets. DDVO [Wang et al., 2018] combines the traditional direct visual odometry with the learning framework to refine the depth estimation results. However, these methods still suffer from the scale inconsistent problem as proposed in recent work [Bian et al., 2019]. Similarly, [Bian et al., 2019] also considers geometric constraints to cope with this issue. Different from their work, we explicitly align the depth scale and enforce pose consistency in the proposed constraints.

These self-supervised learning methods have shown a promising way for single view depth and pose estimation without ground truth. The details of our loss functions are described in Sec. 3 and comparisons with SOTA approaches are shown in Sec. 4.

3 Method

In this section, we first describe the main idea behind selfsupervised monocular depth and pose estimation, and then introduce our framework and loss functions. Finally, we present the baseline and final learning methods.

3.1 ProblemFormulation

To learn depth and pose from monocular videos without ground truth, following previous methods [Zhou et al., 2017], [Mahjourian et al., 2018], [Bian et al., 2019], we use photometric reprojection error as supervision. Given a pixel coordinate point x = (x, y)T and its estimated depth d = d(x) predicted from the depth network, we can reconstruct its 3D point X = (X, Y, Z, 1)T based on the pinhole camera model by the inverse projection function

where fx, fy represent focal lengths and cx, cy stand for optical centers. The projection function is performed as

Considering two consecutive frames {It-1, It}, we can use the relative pose T = Tt→t-1 predicted by pose network to get warping transform function

With this warping transformation, the synthesized images It-1(ω(xi,j ,T)) are generated by using a differentiable bilinear sampling mechanism [Jaderberg et al., 2015] and photometric consistency loss is formulated as

where N is the number of valid points.

3.2 Scale-consistent Geometric Constraints

Besides minimizing photometric inconsistency, in this section, we consider decreasing depth maps and pose differences among consecutive frames as geometric supervision. The details of our loss functions will be introduced respectively.

Scale-consistent Depth Reconstruction Loss

We use {dt-1, dt} to represent the estimated depth of images {It-1, It}. Similarly to photometric error, warping function Eq. 3 is used to get synthesized depth map d′t-1 . Its equation is shown as follows:

Rather than directly utilizing dt predicted from depth net, the transformation matrix T is applied to get the reconstructed depth map d′t [Kerl et al., 2013]. The equation is given as

where [·]d represents d component of a 3D point.

Learning depth only from monocular sequences suffers from scale inconsistent problem [Bian et al., 2019], that is to say, the estimated depth maps from different images could have a different scale. Therefore, to penalize the inconsistent between synthesized and reconstructed depth maps, the scale inconsistent issue is reconsidered here. In details, we firstly align the depth scale by using their mean { ¯ d′ t-1, ¯ d′ t} to normalize depth maps, and then apply a term to penalize the error between ¯ d′ t-1 and ¯ d′ t to keep scale consistent. Finally, our scale-consistent depth reconstruction loss is written as

where N represents the number of valid points in the depth map.

Forward-backward Pose Consistency Loss

We use the vectors r and t to represent the rotational and translational items of the relative pose T. Inspired by [Godard et al., 2017] which uses mirror consistency between left and right views, we propose forward-backward pose constraint to get more accurate results. As Fig. 2 shows, two concat images are the input of pose network. The relative poses {Tt→t-1,Tt+1→t,Tt+1→t-1} can be predicted from the images pairs {{It-1, It} , {It, It+1} , {It-1, It+1}}. If the estimated relative poses are accurate, the product of Tt→t-1 and Tt+1→t should be equal to Tt+1→t-1 and vice versa. For simplicity, we use the rotational and translation vectors to represent the estimated relative poses. Our forward-backward pose consistency loss is formulated as

where m represents the forward-backward operation.

Note that the relative poses have been used to get the synthesized and reconstructed depth maps in Eq. 5, 6. So the scale inconsistent problem does not need to be considered in pose estimation. The forward-backward relative poses have been predicted by pose network, therefore the forwardbackward idea can be easily extended to depth reconstruction loss function. For simplicity, we do not show this operation in our depth reconstruction loss and our scale-consistent geometric constraint is given as

where empirical parameters λd and λpo have been used in our formulation and more details can be found in Sec. 4.2.

3.3 Total Training Loss and Framework

Similar to existing work [Godard et al., 2017], [Mahjourian et al., 2018], [Bian et al., 2019], structural similarity (SSIM) [Wang et al., 2004] loss has been used in our work to solve the complex illumination variations problem. Considering two images patches p1 and p2, SSIM(p1, p2) is defined

Following previous methods [Godard et al., 2017], [Mahjourian et al., 2018], [Bian et al., 2019], it is assumed that the estimated depth maps should be locally smooth. An edge-aware term is used to weight the cost by using depth maps gradients. The depth smoothness loss is shown as follow:

To demonstrate that the proposed constraint can contribute to the depth and pose estimation, we use photometric consistency loss with additional appearance losses as the baseline method which is represented by

Via adding our proposed scale-consistent geometric constraint to baseline formulation, the total loss is written as

As shown in Fig. 2, our depth network is based on the DispNet [Zhou et al., 2017] which is an encoder-decoder architecture. We replace the original encoder-decoder convolutional blocks with residual blocks [He et al., 2016]. The input of this network is an RGB image and the output is a disparity map (the inverse of a depth map). For pose network, we adopt the architecture PoseNet proposed in [Bian et al., 2019]. The network uses a concat of two images as input then output rotational and translational vectors of the relative pose. In Fig. 2, we present the main idea of our proposed scale-consistent geometric constraint. Three sequential images are formulated as a training sample and their depth maps are predicted respectively. The relative poses from image pairs are utilized in Eq. 5 and 6. To get backward pose results, the image sequences need to be reversed.

4 Experiments

4.1 Datasets

Our models are mainly trained on KITTI datasets [Geiger et al., 2012]. For monocular depth estimation, we use Eigen split [Eigen et al., 2014] of Raw data for a fair comparison with previous methods. The split selects 697 images as test datasets for monocular depth estimation and the others are applied for training. The ground truth depth maps of test datasets are obtained from lidar sensors. The original image size is 1242×375 and we resize it as 416×128 or 832×256 to formulate training datasets. For pose estimation, we train our networks on the KITTI Odometry dataset [Geiger et al., 2012], which contains 11 sequences with public ground truth poses. The sequences 00-08 are utilized for training and 0910 are test sets. The ground truth poses are not used in our training framework. Note that for the depth or pose evaluation, we separately train our networks on two datasets (Raw data and Odometry data). Because there are overlapping scenes in these datasets.

As presented in previous works [Zhou et al., 2017], [Bian et al., 2019], depth and pose estimation results can be improved by pretraining on Cityscapes [Cordts et al., 2016] datasets. To compare with SOTA methods, we pre-train networks on Cityscapes and finetune on KITTI with the parameters in Sec. 4.2.

The Make3D datasets [Saxena et al., 2009] are used to evaluate the generalization ability of the depth estimation model. We directly test our models on this set without training or fine-tuning. The datasets contain 534 single-view images with ground truth depth maps and 134 images are used for evaluation.

4.2 Implementation Details

We use PyTorch [Paszke et al., 2019] to implement our framework and train it with a TITAN XP GPU. Adam optimizer is adopted and parameters are set as β1 = 0.9 and β2 = 0.999. In Eq. 12 and 13, the combination [λph, λS, λsm, λd, λpo] = [0.15, 0.85, 0.1, 0.001, 0.1] is used. We utilize three sequential frames as training samples and batch size is set to 4. The learning rate is set as 10-4 . Our models are trained in 100 epochs and we randomly select 1000 samples in every epoch. We pre-train the model on Cityscapes and then finetune KITTI datasets. The data is augmented with random brightness, contrast, and saturation.

4.3 Depth Estimation Results

Our method is evaluated on standard Eigen split [Eigen et al., 2014] by using the metric proposed in [Eigen et al., 2014]. As shown in Tab. 1, we compare the depth estimation results with SOTA self-supervised methods. These models are trained on KITTI or Cityscapes datasets. The evaluation results of previous works are taken from their public papers. Two maximum depth range 80m and 50m have been used in the evaluation and most models are tested at 80m depth range. Without the utilization of traditional direct visual odometry, the proposed method outperforms DDVO [Wang et al., 2018]. Different from the methods using complex auxiliary tasks [Yin and Shi, 2018] [Ranjan et al., 2019] or complicated networks [Vankadari et al., 2019], our approach mainly focus on leveraging geometric information. It is important to note that although the models vid2depth [Mahjourian et al., 2018] and SC-SfMLearner [Bian et al., 2019] have considered the depth consistency in geometry, our methods not only mitigate depth scale inconsistency but also take into account relative pose consistency during the training phase and thus our model can achieve more accurate depth estimation results.

Our model is directly tested on Make3D datasets without finetuning to verify the generalization ability. In Tab. 3, we compare our models with the online public SOTA models. In this unseen environment, the proposed method still has significant improvement compared with the other methods.

4.4 Pose Estimation Results

Although depth and pose nets are jointly learned during training, they are an independent model in the testing phase. Frame-to-frame pose estimation results without postprocessing have been integrated over sequence 09 and 10 of the KITTI Odometry dataset. As shown in Tab. 4, we evaluate average translation and rotation error of (100,200,...,800) meters sub-sequences by using the standard evaluation metrics proposed in [Geiger et al., 2012]. Fig. 4 shows the qualitative results of trajectories. Our estimated trajectory scale only needs to be aligned once with ground truth rather than aligning every estimated pose results scale as SfMLearner [Zhou et al., 2017]. Different from Depth-VO-Feat [Zhan et al., 2018] using a stereo sequence for training, only monocular videos are utilized in the proposed method. By simultaneously considering pose and aligned depth consistency, our model achieves better pose estimation results than SCSfMLearner [Bian et al., 2019].

4.5 Ablation Studies

We present ablation studies to show the importance of our methods in depth estimation. The experiments are conducted on KITTI Eigen split with variations of our models and use the same metrics mentioned in Sec. 4.3.

Tab. 2 presents the quantitative results and Fig. 1 shows qualitative comparisons. Our baseline method is similar to the basic approach of [Bian et al., 2019] and is based on the loss proposed in Eq. 12. We apply the network architecture mentioned in Sec. 3.3 to ablation studies. It can be found that, without any contributions of our methods, the baseline model performs worst. The combination of depth reconstruction and pose consistency loss proposed in Eq. 9 makes a significant improvement. Excluding one of our loss functions decline the performance of depth estimation. Moreover, we use larger image resolution 832 × 256 and pre-train our models on the Cityscapes dataset to get better results.

5 Conclusion

In this paper, we present a self-supervised learning framework of depth and pose estimation with scale-consistent geometric constraints. Beyond using photometric consistency as supervision, robust geometric information is utilized and scale alignment operation is also conducted in the proposed method. The scale-consistent depth and visual odometry results can be predicted by our models, which is essential for many applications such as obstacle avoidance. Comparing with prior self-supervised learning frameworks, the proposed monocular depth and pose estimation models obtain SOTA results on KITTI and Make3D datasets. To the best of our knowledge, it is the first work to simultaneously use depth and pose consistency to formulate the self-supervised signal from monocular video sequences.

Moreover, little current work uses higher-level features as supervision. It is interesting to fuse semantic knowledge with the proposed method for future work. Traditional VO systems have been widely used in many applications, we will try to incorporate self-supervised learning ideas into traditional VO or visual SLAM to get more robust and accurate pose estimation results.

Acknowledgments

This work is supported by the National Natural Science Foundation of China under Grant 61673274.