Unsupervised Monocular Visual-inertial Odometry Network

1 Introduction

VO or VIO is a fundamental task that aims to track the incremental motion of the sensor and simultaneously build a map of the environment. Traditional monocular VO methods [Mur-Artal and Tard´os, 2017, Geiger et al., 2011, Engel et al., 2017] utilize handcrafted features or photometric matches to calculate the trajectory from a monocular image sequence. However, these methods are impressionable to motion blur, occlusion, and textureless regions. As a complementary sensor of visual cameras, inertial measurement unit (IMU) has been widely adopted in VIO methods [Huang and Liu, 2018,Bloesch et al., 2015,Leutenegger et al., 2013,Qin et al., 2018] for its high-frequency motion measurement and relatively low cost. The use of IMU can help to increase the robustness as well as improving the accuracy.

∗Equal contribution

†Corresponding Author

1 https://github.com/Ironbrotherstyle/UnVIO

With the development of CNN and RNN, various learningbased VO or VIO methods have been proposed. Although many supervised methods [Wang et al., 2018, Clark et al., 2017, Chen et al., 2019] have been revealed more competitive than traditional methods, the demands of a large number of labeled data, i.e., the ground truth poses acquired from high-precision devices, limit the application of the technology. Self-supervised methods [Shamwell et al., 2018,Han et al., 2019] release the pressure of collecting large quantities of ground truth, but they still require other expensive data, e.g., depth map, which degrades the flexibility. In contrast, unsupervised VO methods [Zhou et al., 2017, Bian et al., 2019] only utilize image sequences to achieve pose estimation, requiring no ground truth label nor expensive data input. However, existing unsupervised VO methods suffer from poor capability on long-term odometry task, due to the inherent limitation of only relying on visual data that may degrade in some cases. Besides, the error accumulation problem in long-term trajectory was ignored in previous methods, thus causing the mediocre result.

In this paper, an unsupervised visual-inertial odometry framework (UnVIO) is proposed. As shown in Fig. 1, by taking the consecutive images and IMU measurements as input, UnVIO is able to predict the depth map and estimate the ego-motion. In particular, a heuristic fusion module is introduced to self-adaptively fuse the visual and inertial features, enabling the model to handle data pollution. The entire framework is trained in an unsupervised end-to-end fashion, through a proposed sliding window optimization strategy. A sliding window is utilized to traverse through the sequence, where the geometric constraint and sequential constraint are exploited to optimize the geometric inferences within and among windows. The contributions can be listed as follows:

• An end-to-end unsupervised visual-inertial odometry framework (UnVIO) is proposed for estimating the egomotion as well as predicting the depth map.

• A visual-inertial feature fusion module is designed to select the most discriminative motion features for camera pose regression. The module improves the robustness to the contamination of image-IMU input.
• A sliding window optimization strategy, consisting of an intra-window optimization and an inter-window optimization, is proposed for unsupervised VIO to tackle the error accumulation and scale ambiguity problems.

2 RelatedWork

Traditional methods. Traditional visual odometry methods can be divided into two categories, feature-based methods and direct methods. ORB-SLAM2 [Mur-Artal and Tard´os, 2017] is a classical feature-based method which extracts hand-crafted features and utilizes bundle adjustment to estimate ego-motion in real-time. DSO [Engel et al., 2017] is a sparse direct method that performs the epipolar matching to achieve camera tracking, based on the assumption of photometric consistency.

In order to increase the robustness and improve the performance, researchers exploit IMU measurements as supplemental information, hence extending VO methods to VIO methods [Bloesch et al., 2015,Huang and Liu, 2018]. OKVIS [Leutenegger et al., 2013] is a tightly-coupled method which optimizes the reprojection error and IMU error at the same time. VINS-Mono [Qin et al., 2018] fuses preintegrated IMU measurements with visual feature observations to achieve accurate pose estimation.

Supervised/Self-Supervised learning methods. By harnessing the deep convolutional and recurrent neural networks, Wang et al. [Wang et al., 2018] designed a supervised architecture to estimate camera pose from the monocular image sequence. VINet [Clark et al., 2017] firstly tackled VIO in a supervised manner. Chen et al. [Chen et al., 2019] exploited two masking strategies for visual-inertial sensor fusion.

Some self-supervised learning methods were proposed for releasing the pressure of collecting quantities of ground truth labels for supervised learning. Shamwell et al. [Shamwell et al., 2018] presented VIOLearner that carried out online error correction in multiple scales to refine the pose estimation. But VIOLearner required the depth map as input, thus limiting its applicability on other scenes without supplying depth data. DeepVIO [Han et al., 2019] is a self-supervised VIO method that uses 3D geometric constraint as supervision. However, DeepVIO needs an awesome pretrained stereo network PSMNet [Chang and Chen, 2018] to provide the accurate and dense disparity map for training.

Unsupervised learning methods. Zhou et al. [Zhou et al., 2017] proposed an unsupervised framework of pose estimation and depth prediction. The framework can be trained by only using image sequences. Shen et al. [Shen et al., 2019] proposed a matching loss constrained by epipolar geometry and improved the odometry performance. In addition to the photometric matching loss, PatchGAN [Vankadari et al., 2019] adopted the generative adversarial approach to promote the depth prediction and pose estimation results. Different from these unsupervised VO methods, we propose an unsupervised VIO method that significantly improves the odometry performance. In contrast to self-supervised VIOmethods, ours requires no extra expensive data as input, but achieves competitive results through the sliding window optimization strategy and visual-inertial fusion module.

3 Unsupervised Visual-inertial Odometry

An overview of the unsupervised visual-inertial odometry framework is shown in Fig. 1. The DepthNet learns a mapping from single RGB image to depth map (see Sec.3.1). By taking the raw monocular image sequence and IMU measurements as input, the visual-inertial odometry networks estimate the ego-motion (see Sec.3.2). The whole framework is trained in a sliding window optimization strategy that includes two parts: intra-window optimization and interwindow optimization (see Sec.3.3).

3.1 Depth Estimation

Given an image I ∈ R3×H×W , the DepthNet learns a mapping function FD that infers the scene depth of per pixel, i.e., D = FD(I). The DepthNet is designed based on an encoder-decoder architecture where the encoder part maps the RGB image into a high dimensional feature space, and the decoder remaps these features into depth values. The pretrained ResNet18 [He et al., 2016] is adopted as the encoder and the skip-connection is exploited between the encoder and decoder for reserving structure details. For the decoder, the nearest neighboring upsampling operation followed by a Conv layer is used to expand the resolution. Exponential linear units are appended after each Conv layer as recommended in [Godard et al., 2017].

3.2 Visual-inertial Odometry

Two parallel networks are designed to extract visual features and inertial features, followed by a visual-inertial fusion module to select the most efficient features. Then, the PoseNet takes the fused temporal visual-inertial features as input to regress 6 DOF poses.

Visual feature extraction. Two adjacent frames It-1, It from the image sequence are concatenated along the channel dimension as the input of VisualNet. The architecture of VisualNet is made up of the first 7 Conv layers of FlowNet [Dosovitskiy et al., 2015] and a global average pooling. The process of visual motion feature extraction can be formulated as:

where ⊕ denotes the concatenation in the channel dimension, Φ is the feature extraction function of VisualNet.

Inertial feature extraction. IMU measures the linear acceleration and angular velocity of the embedded body at a faster rate than the visual measurement. The sampled raw IMU measurements from time t - 1 to t are arrayed in the following form:

where α, ω ∈ R3 are the linear acceleration and angular velocity respectively, n is the number of IMU samples. The sequential IMU measurements are then sent into a two-layer LSTM [Hochreiter and Schmidhuber, 1997] to get the inertial motion features:

where R represents the recurrent function of IMUNet, Hi is the hidden state. By this way, sequential IMU measurements are integrated into the final inertial motion feature F I t .

Visual-inertial feature fusion. A straightforward but effective fusion strategy is designed to fuse visual features and inertial features. The concatenated visual-inertial features F = F V t ⊕ F I t along channel dimension will be firstly aggregated into squeezed features F ′ , by a learned basis vector group G and a learned bias vector b:

Then, F ′ are decoded to a weight vector W that indicates the importances of channel-wise visual and inertial features:

where FF is the decoding function of the fusion module, and σ represents the sigmoid function. The recalibrated visualinertial features F are obtained through the Hadamard product of W and F : F = F ⊙W.

Pose estimation. Given an image-IMU stream, the motion feature set {F10 ,F21 , · · · ,Fs-2 s-3 , Fs-1 s-2} where each item represents the visual-inertial features between two adjacent times, is fed into PoseNet to dig the temporal relevance:

where Hi is the hidden state output, and R is the refining function of PoseNet. Tii-1 is the refined motion features between frame i - 1 and frame i, which is subsequently sent to a linear layer to obtain 6 DOF camera pose pii-1 .

3.3 SlidingWindowOptimization

The key supervision of the unsupervised visual-inertial odometry framework comes from the multi-view geometric constraint and sequential constraint. Given a sequence of visual-inertial measurements at different times, {⟨I0,M0⟩ , · · · , ⟨Is-1,Ms-1⟩}, a sliding window traverses through the sequence, with consistency check leveraged to optimize the geometric inferences (depth and camera pose). An example of sliding window optimization with step size 1 and window size 3 is shown in Fig. 2. In each window W, the depth D and camera poses ˆ p∗ are independently predicted from the windowed visual-inertial measurements through our framework. The photometric consistency check is utilized to achieve individual intra-window optimization. To handle the error accumulation and scale ambiguity problems, additional inter-window optimization is designed to constrain predictions of different windows through checking 3D geometric consistency and trajectory consistency.

Prior knowledge of multi-view geometry. When a camera moves in a scene, objects that can be seen in adjacent views form the geometric constraint. Denote Is, It are two adjacent frames of source view and target view respectively, and ps, pt are two pixel points that correspond to the same 3D map point of the scene. With the depth maps Ds, Dt and the egomotion transform matrix Tt→s available, the 3D geometric consistency can be set by:

where K is the camera intrinsic matrix. Also, Equ. (7) can be converted to indicate 2D reprojection constraint:

where ∼ means 'equal in the homogeneous coordinate. According to Equ. (8), a sample grid can be generated and used to warp Is into synthesized target-view image ˜ Is, through a bilinear sampling. The photometric consistency check is defined by the appearance similarity between ˜ Is and It.

Intra-window optimization. The image-IMU stream {⟨Ii,Mi⟩ , · · · , ⟨Ii+w-1,Mi+w-1⟩} of each sliding window is utilized for geometric inference generation and intrawindow optimization. The middle frame of the window is taken as the target view, while others are source views. The predicted depth map D of the target view and estimated camera poses ⟨ ˆ pi+1 i , · · · , ˆ pi+w-1 i+w-2 ⟩ between adjacent frames within the window, are checked by photometric consistency:

where ⟨s, t⟩ denotes all the source-target pairs, SSIM [Wang et al., 2004] represents the structural similarity metric. Additional smoothness loss is also adopted for alleviating the shortage of photometric consistency on the textureless regions as recommended in [Shen et al., 2019]:

The intra-window optimization loss can be summarized as:

where α1 and α2 are weighting factors.

Inter-window optimization. It is prone to fall into a local optimum by only relying on the optimization within windowed frames, due to the lack of sequential constraint that may cause the universal scale ambiguity and the accumulated error problems in monocular odometry. We consider the inter-window optimization, including trajectory consistency check and 3D geometric consistency check.

Partial information of the sequence is exploited to estimate the windowed ego-motion in intra-window optimization, i.e., {Ti+1 i , · · · ,Ti+w-1 i+w-2} → ⟨ ˆ pi+1 i , · · · , ˆ pi+w-1 i+w-2 ⟩. To take the inter-window relevance into account, the integrated information is also exploited to estimate the camera poses for the entire sequence, i.e. {T10 , · · · ,Ts-1 s-2} → ⟨p10 , · · · , ps-1 s-2 ⟩. The estimated camera poses ˆ p∗ that aggregated from the windowed estimation and the corresponding p∗ that estimated from the integrated sequential information are checked for the trajectory consistency by:

With the sliding window traverses through the sequence, the middle-frame depth map that determines the scale of each window is predicted. Therefore, to ensure the uniform scale of contiguous windows, we project the depth map into 3D point clouds and then perform the 3D transform based on Equ. (7) to check the 3D geometric consistency. The 3D geometric consistency loss L3D for inter-window optimization is defined as:

where ˜Di is the warped depthmap fromDi. The loss function of inter-window optimization is then concluded as:

(a) KITTI 05

(b) KITTI 07

(c) KITTI 09

(d) KITTI 10

(e) Malaga 03

(g) Malaga 09

Figure 3: Trajectory estimation on KITTI and Malaga dataset. (a), (b) are KITTI 05, 07 in the training set. (c), (d) are KITTI 09, 10 that are used for testing. (e), (f), (g) are the test trajectories of Malaga 03, 07, 09 overlaid on Google Map (GPS is served as reference instead). Best viewed in the colored electronic version.

To summarize, the loss function of sliding window optimization can be written in:

4 Experiments

In this section, both quantitative and qualitative results compared with traditional and learning-based VO/VIO methods are presented. The ablation study is employed to demonstrate the effectiveness of each component of our method.

4.1 Datasets

KITTI Dataset. KITTI dataset [Geiger et al., 2012] serves as a prevalent driving dataset, with stereo images at 10Hz, IMU data at 100Hz, accurate pose and laser scan. Seqs 00-10 of the odometry partition are used, except for 03 where IMU data is not acquirable. Seqs 00-08 excluding 03 are adopted for training and 09-10 are utilized for testing.

Malaga Dataset. Malaga [Blanco-Claraco et al., 2014] is an outdoor dataset. Stereo images at 20Hz, IMU measurements at 100Hz and GPS are provided. In our implementation, rectified left images are downsampled to 10Hz. Seqs 01, 02, 04, 05, 06, 08 are adopted for training and Seqs 03, 07, 09 are used for qualitatively evaluating since no ground truth pose is offered.

4.2 Training Details

All the models are implemented by using the Pytorch framework on a computer equipped with an Nvidia GeForce GTX1080 Ti GPU. Adam optimizer with learning rate 10-4 ,

β1 = 0.9, β2 = 0.999 is utilized. Images for training on both datasets are resized to 832 × 256, meanwhile, the IMU samples n is set to 11. The training process converges after about 100000 iterations with a batch size of 4. Besides, the length of training sequence s and window size w are 5 and 3 respectively in our experiment. The weights for loss functions are empirically given as: α1 = 1, α2 = 0.1, α3 = 0.1, α4 = 0.1, λ1 = 0.15, λ2 = 0.85.

4.3 Odometry Evaluation

The evaluation of odometry is carried out among traditional VO methods VISO-M [Geiger et al., 2011] (monocular version), VISO-S [Geiger et al., 2011] (stereo version), ORBSLAM2 [Mur-Artal and Tard´os, 2017], VIO methods VINSMono [Qin et al., 2018], OKVIS [Leutenegger et al., 2013], self-supervised VIO methods VIOLearner [Shamwell et al., 2018], DeepVIO [Han et al., 2019], and unsupervised VO methods SfM [Zhou et al., 2017], SC [Bian et al., 2019]. All the monocular methods need to be evaluated after making 7 DOF (6 DOF + scale) alignment with ground truth, apart from VINS and OKVIS that can recover the scale. Notably, we implemented the above open-source methods except for VIOLearner and DeepVIO to get the odometry results. KITTI benchmark [Geiger et al., 2013] is utilized as the evaluation criterion, where trel is the average translational RMSE drift (%) on length of 100m-800m, and rrel is the average rotational RMSE drift (◦ /100m) on length of 100m-800m.

The quantitative results of odometry evaluation on KITTI dataset are summarized in Table 1. Seqs 00, 02, 05, 07, 08 of the training set are selected for evaluation in [Shamwell et al., 2018], therefore, we calculate the average errors of these sequences in column Avg(sub-t). The results of Avg(sub-t) show that our method outperforms other self-supervised VIO methods in both trel and rrel metrics, although without using extra depth data. The average results of all the training set except Seq 03 whose IMU data is not available are used for the complete comparison on the training set (see column Avg(train)). Our method clearly performs better in trel than the unsupervised VO methods and traditional methods that may hold larger accumulated errors. Additionally, average errors on the test set (see column Avg(test)) are provided. It can be observed that the proposed method significantly improves the translational performance compared with the unsupervised VO methods and traditional methods on unseen scenes, validating the superiority of UnVIO. Compared with self-supervised VIO methods, UnVIO also achieves competitive results with lower rotational error rrel on the test set. The reason that other self-supervised VIO methods gain better trel may be that the extra depth data can provide more determinate information for geometric inference. Besides, our vision-only method, i.e, Ours (NoIMU), performs better than other unsupervised VO methods, which indicates the predominance of our framework when implemented without IMU data.

Fig.3 illustrates the trajectories generated by various methods on KITTI and Malaga dataset. The proposed UnVIO can predict more accurate trajectories than other learning-based methods on KITTI and is superior to traditional monocular VO methods. They may hold large drift due to scale ambiguity and accumulated error. It is obvious that our method outperforms reference methods on Malaga dataset, where VINS and OKVIS are likely to fail at the begining part of the trajectories because of long-time initialization.

4.4 VIORobustness Evaluation

Four settings that simulate sensor data collapse due to physical and thermal changes in the VIO system are conducted to test the robustness of UnVIO. Specifically, Mis-c:10◦ indicates adding 10◦ to the rotation matrix of camera-IMU extrinsic parameters. Unsyn:20ms means randomly adding 20ms to the IMU time-stamp. IMU-D:30% represents adding white noise to the accelerometer data and random walk noise to the gyroscope measurements at a rate of 30%. Cam-D:30% means adding blur, partial occlusion or full occlusion on the input images with a probability of 30%. Table 2 details the performance of VINS and the proposed method trained on KITTI on the abovementioned conditions. The proposed method has no performance deterioration on Mis-c because the extrinsic parameters are not required in our method. However, VINS seems to be confused and holds large errors as VINS relies on delicate calibration. On Unsyn, our method shows the advantages of handling the unsynchronized image- IMU stream than VINS. Even when the input data is polluted, i.e. IMU-D and Cam-D, the proposed method still achieves better performance, showing better robustness to polluted input data.

4.5 Depth Evaluation

Since depth and pose estimation are coupled tasks, we test the performance of DepthNet following the odometry split. Table 3 gives quantified comparison with SfM, SC, and the proposed UnVIO on KITTI dataset. The results show that our method achieves the best performance in all metrics. Fig. 4 sketches the predicted depth maps. The first row lists the monocular inputs of KITTI and Malaga dataset, and the second to the last rows are the predicted depth maps corresponding to each input of the three methods. Intuitively, our results retain more details of the edges and contours of the predicted objects, e.g., the road sign and distant cars.

4.6 Ablation Study

Ablation study has been done to demonstrate the effectiveness of each proposed component, as shown in Table 4. By extending VO methods to unsupervised VIO methods, the performance gains remarkable improvements via taking both image and IMU data as input. It can also be concluded that the proposed sliding window optimization is able to promote the translational and rotational performance. This is because the sliding window optimization strategy not only considers the intra-window photometric consistency, but also focuses on inter-window 3D geometric consistency and trajectory consistency to handle the widespread problems in monocular odometry. By self-adaptively fusing the visual-inertial features through the visual-inertial fusion module, the performance is further improved.

5 Conclusions

An unsupervised visual-inertial odometry framework (UnVIO) which only utilizes the monocular image-IMU stream for training and testing, is proposed in this paper. A visualinertial feature fusion module is introduced to enable UnVIO to show robustness to polluted data. Besides, a novel sliding window optimization strategy with the advantages of overcoming scale ambiguity and error accumulation is proposed. Experimental results show that our method not only outperforms other unsupervised methods and traditional methods but also performs competitively with self-supervised VIO methods that need extra expensive depth data.

Acknowledgments

This work is supported by National Natural Science Foundation of China (No.U1613209, 61906103), National Natural Science Foundation of Shenzhen (No.JCYJ20190808182209321)