Collaborative Learning of Depth Estimation, Visual Odometry and Camera Relocalization from Monocular Videos

1 Introduction

As the basis of various key applications such as autonomous driving, VR/AR, robot vision and drones, researches on scene perceiving and understanding including depth estimation, visual odometry and camera relocalization have attracted significant attention from the community. Due to the development of deep learning, many approaches have been proposed recently to solve these three individual tasks by using CNNs or RNNs to replace traditional methods in supervised or unsupervised manners.

*The work was done during Haimei Zhao's visit at UBTECH Sydney AI Centre, School of Computer Science, Faculty of Engineering, The University of Sydney.

Depth Estimation has been extensively studied recently due to its crucial role in 3D scene understanding. Supervised monocular learning is first proposed as a regression problem to learn a mapping from RGB images to per-pixel depth maps using labelled datasets [Eigen et al., 2014]. DORN [Fu et al., 2018] turns the regression procedure into a multiclass classification problem with discrete depth values. Although these supervised methods can achieve impressive results, collecting a dataset with ground truth is both challenging and expensive, especially for outdoor scenarios. Unsupervised approaches [Zhou et al., ; Wang et al., 2018] make use of the photometric consistency between adjacent frames to provide self-supervision. This pipeline relies on a spatial transformer network [Jaderberg et al., 2015] to synthesize reference frames using target frames, which can simultaneously optimize the pose transformation between them. Stereo methods [Godard et al., 2017; Zhan et al., 2018] employ the consensus between the left and right cameras in image or feature level. These self-supervised methods tend to suffer from the violation of moving vehicles or people. To tackle this issue, several works [Casser et al., 2019; Xu et al., 2019] use the Mask R-CNN [He et al., 2017] to separate dynamic objects from the scene and deal with them separately to improve the estimation accuracy. MD2 [Godard et al., 2019] employs an auto-masking strategy to handle static camera and moving objects.

Visual Odometry is one of the most essential techniques in computer vision and robotic localization. Traditional methods generally follow a pipeline [Fraundorfer and Scaramuzza, 2012] including camera calibration, feature detection, feature matching, outlier rejection, motion estimation, scale estimation and Bundle Adjustment. Recently, deep learning based methods have been presented, which replace the original visual odometry (VO) process with an end-to-end neural network. DeepVO [Wang et al., 2017] combines CNNs and Long Short-Term Memory Networks (LSTMs) to obtain pose estimation from image sequences by conducting sequential modelling. UndeepVO [Li et al., 2018] employs a framework similar to [Zhou et al., ] to estimate VO with recovered scales from stereo sequences. CTCNet [Iyer et al., 2018] adds a set of transformation constraints across a series of frames to enforce the geometric consistency of the trajectory. At present, VO estimation is relatively accurate locally but it is still confronted with the issue of long-distance drift.

Camera Relocalization aims to infer the global pose of a camera from visual scene representations, which is crucial for navigation applications. PoseNet [Kendall et al., 2015] is one of the early attempts to train CNNs in the end-to-end manner to infer the camera's 6-DoF pose from a single image. MapNet [Brahmbhatt et al., 2018] combines camera relocalization with traditional VO and GPS data to improve the estimation accuracy. With the self-attention mechanism, AtLoc [Wang et al., 2019] devotes to helping CNNs focus on geometrically robust objects or features, producing more robust estimation. Currently, the quality of camera relocalization is reasonably satisfactory in the global range but there is still a big margin for improvement concerning local precision.

In previous work, the three tasks (depth estimation, VO and camera relocalization) are accomplished using separate neural networks from a single image or video clips. Since they all fall into the domain of scene understanding from input images, it is reasonably to argue that the abundant inter-task geometric correlations should be exploited to bring extra benefits to each other.

In this paper, we propose a collaborative learning network to jointly conduct the tasks of depth estimation, VO and camera relocalization from monocular videos. We leverage three branches (DepthNet, LocalPoseNet and GlobalPoseNet) to perceive the environment from different perspectives, which are particularly suitable for exploiting 3D geometry structure, local pose transformation and global pose, respectively. With a purposefully designed joint optimization loss, our three branches are able to provide complementary benefits and overcome the individual defects (e.g., drift in long distance for LocalPoseNet and low precision in local range for GlobalPoseNet) during learning. In addition, the Geometric Attention Guidance Model (GAGM) is introduced to extract valuable 3D geometric information from the depth estimation branch to enhance the VO and camera relocalization branches, which is also helpful for the improvement of depth estimation in turn. The visualization of the attention maps shows that our GAGM does have the ability to learn meaningful guidance from the depth information.

The main contributions of our work are as follows:

1. We present a collaborative learning framework to jointly conduct the tasks of depth estimation, visual odometry and camera relocalization to leverage the complementary advantages among tasks. To the best of our knowledge, this is the first work to jointly solve these three problems.
2. We introduce a Geometric Attention Guidance Model (GAGM) as an inter-task interaction mechanism to help LocalPoseNet and GlobalPoseNet acquire valuable geometric information from the depth data to improve the accuracy.
3. We conduct extensive experiments on KITTI and our method outperforms SOTA methods by 10.4% in depth estimation and 15.2% and 27.4% in camera relocalization according to two major performance metrics, respectively.

2 Methodology

This section presents our collaborative learning framework for depth estimation, visual odometry and camera relocalization from monocular videos. Taking a sequence of video frames as input, the aim of our framework is to produce depth map Dt and camera global pose Pt from each frame and the ego-motion of each frame pair Tt+1→t simultaneously. For this purpose, our framework consists of three main network branches DepthNet, LocalPoseNet and GlobalPoseNet with an interaction mechanism GAGM.

2.1 Depth and Visual Odometry

The estimation of depth and VO employs the self-supervised pipeline [Zhou et al., ], which is built based on the photometric consistency among adjacent frames.

As shown in Figure 2, our DepthNet and LocalPoseNet both adopt encoder-decoder architecture, while skip connections are used in DepthNet to utilize both shallow information and deep abstract features to get accurate depth maps. To ensure a fair comparison with other methods, we modified the ResNet18 network pretrained on ImageNet [Russakovsky et al., 2015] as our encoder, ED and EL. The DepthNet Θ takes frame sequence < It-n, It, It+n > as input and outputs corresponding depth maps < Dt-n,Dt,Dt+n >, f(I,Θ) = D. Meanwhile, from continuous frames, the LocalPoseNet Φ produces the 6-DoF relative pose transformation < Tt+n→t, Tt-n→t > of adjacent frame pairs, g(< It, It±n >, Φ) =< Tt±n→t >.

During learning, the predicted depth map Dt and pose transformation T are used to warp the source frame It+n to obtain the synthesized target frame ˆ It+n→t. By adopting the differential bilinear sampling mechanism [Jaderberg et al., 2015], we can obtain corresponding reprojected pixel coordinate ˆ p in ˆ It+n→t for each pixel p in frame It:

where K is the known camera intrinsics and h(p) = (x, y, 1) means the homogeneous coordinates of pixel p. By constraining the difference between the synthetic frame ˆ It+n→t and the original target frame It, depth and visual odometry can be optimized in an self-supervised manner:

The difference measurement ℓ is a weighted combination of l1 loss and SSIM following prior work [Godard et al., 2017]:

To overcome the effect of occlusion and dynamic vehicles moving at the same speed as the camera, the minimum dissimilarities and the stationary mask κ are adopted as proposed in [Godard et al., 2019]:

where, N is the collection of the reference frames.

An image-aware smoothing item is used to regularize the depth discontinuity as previous work [Godard et al., 2017] :

where µDt is the normalized inverse depth by mean value.

2.2 Camera Relocalization

Our GlobalPoseNet Ψ aims to learn the global camera pose from each image on the training set C = (I, P ∗ ) in a supervised manner, g(I;Ψ) = P. Given an input frame It, the pretrained ResNet34 network is first used in GlobalPoseNet to extract valuable feature FPt. The following Multilayers Perceptrons (MLPs) then map feature FPt to global pose P.

We represent the camera pose P as [p, q] with p ∈ R3 for position and a unit quaternion q ∈ R4 for orientation to regress it with l1 or l2 norm properly. Because any rotations in 3D space can be effectively mapped to valid and unique unit quaternions by normalizing 4D quaternions to unit length and restricted to the same hemisphere according to [Brahmbhatt et al., 2018; Wang et al., 2019]. To regularize the network, we adopt the l1 loss between the predicted [p, q] and the label [p∗ , q∗ ] :

where η and ϕ are weights for balancing the position loss and rotation loss, learning from initial values η0 and ϕ0 during training simultaneously. logq is the logarithmic form of q:

v and u are unit quaternion q's real and imaginary parts.

2.3 Collaborative Learning

Geometric AttentionGuidanceModel We propose a selflearning Geometric Attention Guidance Model (GAGM) to learn and leverage the latent geometric correlation between DepthNet and two pose networks LocalPoseNet and GlobalPoseNet to acquire more accurate estimation. Our GAGM takes the deepest feature of depth encoder FDe and the second last feature of depth decoder FDd2 as input and outputs two attention scale maps for LocalPoseNet and GlobalPoseNet, respectively. As shown in Figure 2 (green area), in GAGM, we first combine FDe and FDd2 to get the depth feature FD via a concatenation and a convolution layer, to utilize both abstract and explicit 3D geometric features from the depth learning branch. Next, FD is passed to an average pooling and a max pooling operations along the channel axis and then concatenated to obtain a highly compressed expression, since pooling layers can help feature be concentrated as shown in prior studies [Woo et al., 2018]. Lastly, a subsequent convolution layer is employed to get the final attention scale maps. During collaborative learning, the corresponding learned attention maps in GAGM are introduced as guidance to scale the deepest feature of LocalPoseNet and GlobalPoseNet, FL and FG, by multiplication. The scaled features are regarded as a residual item to be added to the original feature FL and FG, respectively. By introducing the condensed feature from DepthNet, our framework allows LocalPoseNet and GlobalPoseNet branches to acquire useful information from not only 2D input images but also 3D geometry to improve accuracy, which is also beneficial for the depth estimation in turn. GAGM can be formulated as:

here, F ′L and F ′G denote the guided feature of LocalPoseNet and GlobalPoseNet, respectively. Wc, Wl and Wg are learnable weights of the corresponding convolution layers.

The Joint Optimization Loss We design a joint optimization loss ℓc for LocalPoseNet and GlobalPoseNet to regularize them mutually. We can calculate a local pose transformation Tt+n→t-n within input images window from pairwise pose transformation learned from LocalPoseNet:

Besides, the local pose transformation can also be produced from camera pose [p, q] estimated in GlobalPoseNet:

where Γ is the transmutation function to change the representation mode for pose. Therefore, we can use the l1 loss to regularize the consistency between the poses predicted from LocalPoseNet and GlobalPoseNet to optimize them jointly within the time window whose size is 2n + 1, taking n as 1 in our work:

To summarize, the integrated loss for our collaborative learning framework is:

where s denotes different scales and S means the number of the scales (4 in our work).

Training Details We trained our models using the KITTI dataset [Geiger et al., 2012], which is one of the most commonly used dataset in autonomous driving. We took sequence 00-08 from the KITTI Odometry dataset as training data, sequence 09 and 10 as test data following prior works [Zhou et al., ; Zou et al., 2018]. Our DepthNet and LocalPoseNet were first pretrained with input images resized to 640 × 192, a batch size of 12 and parameter α and β set to 1 and 0.05. After that, we started to train the collaborative framework with input images resized to 1024 × 320, a batch size of 4 and the loss weight α, β, γ and δ set to 0.5, 0.05, 0.85 and 0.1, respectively. The learning rate was first set to 10-4 for the first 10 epochs and then dropped to 10-5 for the remaining 40 epochs. We took three consecutive frames as input. With the ability of locally optimizing relative pose and global pose within a time window during learning, our framework should perform better if given longer sequences as input. Our method was implemented in Pytorch.

3 Experiments

Comprehensive evaluations were implemented towards all three tasks (depth estimation, VO and camera relocalization). Our method achieved excellent results in the comparison with existing SOTA researches, which demonstrates the effectiveness of our collaborative learning method.

3.1 Evaluation of Depth Estimation

We conducted extensive evaluation experiments to compare the depth estimation performance of our method with previous works. The quantitative results are reported in Table 1, which clearly demonstrate that our model outperforms current SOTA approaches trained in self-supervised manner. Moreover, although being trained with monocular sequences only, our model surpasse other methods learned from stereo videos. As shown in Figure 3, our method can generate satisfying depth maps with clear instance boundary and perform properly in some challenging situations including delicate structures (e.g. traffic signs) and low-texture regions (e.g. the surface of tankers or carriages).

Ablation To ensure the fairness of the comparisons, we list ablation study in Table 1. Ours (baseline) means method without collaborative learning with GlobalPoseNet and the joint optimization loss ℓc, but with GAGM between DepthNet and LocalPoseNet, which is totally self-supervised. It is clear that our method is superior to other methods even without the supervision of poses for GlobalPoseNet. Ours (baseline w/o GAGM) means further removing the GAGM between DepthNet and LocalPoseNet. From the comparisons, the effect of our collaborative learning framework and GAGM for depth estimation can be demonstrated obviously.

Generalization Ability Although being trained on KITTI only, our method can also achieve promising results on unseen datasets without any fine-tuning. We conducted inference experiments on Make3D and Cityscapes to verify the generalization ability as shown in Figure 4. Compared with SOTA approaches, our method is capable to produce more accurate depth maps with sharper object boundary and better perception of distant instances even on unseen dataset.

3.2 Evaluation of Visual Odometry

Previous studies on Visual Odometry suffer severely from long-distance drift. To overcome it, our collaborative learning framework takes advantage of the rich 3D geometry information from depth estimation branch and the global localization auxiliary from relocalization network. In Table 2, we conducted quantitative evaluation on KITTI Odometry test sequence 09 and 10. The Absolute Trajectory Error (ATE) was calculated on 5-frame snippets and averaged over the full sequences following the protocol of [Zhou et al., ].

Ablation We also trained our models without collaborative learning with camera relocalization task, which was taken as the baseline in Table 3 and Figure 5 (right). The comparisons of our predicted trajectory with other methods and our baseline in Figure 5 can demonstrate our method is valuable for alleviating drift in long range.

3.3 Evaluation of Camera Relocalization

The evaluation of camera relocalization was also conducted on KITTI Odometry sequence 09 and 10. As shown in Table 2, we summarized the mean and median value of position and rotation error with corresponding ground truth. Benefited from the 3D geometry guidance of depth network and local constraints of visual odometry branch, our relocalization results exceed prior SOTA methods in both position and rotation accuracy. We visualized the prediction trajectory of relevant methods using a subsampling factor 5 to show results more clearly, as listed in Figure 6. The green, red and blue lines denote the ground truth, estimation and error, respectively. Compared with other methods, our trajectories are more accurate and noiseless.

AttentionMap Analysis The visualization of the attention guidance maps generated by GAGM is shown in Figure 7. It is clear that GAGM is useful to teach the GlobalPoseNet to focus on stable geometric features and regions such as road signs, trees and the ground instead of dynamic cars for remembering and recognizing scenes, which is highly valuable for camera relocalization task.

4 Conclusion

The motivation of this work is to highlight the importance of exploiting the inherent correlation among three classical scene understanding tasks: depth estimation, VO and camera relocalization. With the proposed collaborative learning framework and the joint optimization loss along with the GAGM, we show the great prospect of boosting the performance of the three tasks simultaneously through collaborative learning. In the experimental studies, we show that our method is superior to existing SOTA methods in depth estimation and camera relocalization, and can achieve highly competitive results in VO. As to future work, we will further train and evaluate our method on other datasets to fully investigate its potential in various application scenarios and provide more insights into its working mechanism.

Acknowledgments

The work is supported by Australian Research Council Projects FL-170100117, IH-180100002, the Natural Science Foundation of China (NSFC)(No.U1713214) and Shenzhen Fundamental Research Fund(No.JCYJ20170412170602564).