{"type":"ScholarlyArticle","title":"Unsupervised Monocular Visual-inertial Odometry Network","authors":[{"name":"Peng Wei","slug":"peng-wei","given":"Peng","family":"Wei","affiliation":"Key Laboratory of Machine Perception , Peking University , Shenzhen Graduate School , China; Peng Cheng Laboratory , Shenzhen , China"},{"name":"Guoliang Hua","slug":"guoliang-hua","given":"Guoliang","family":"Hua","affiliation":"Key Laboratory of Machine Perception , Peking University , Shenzhen Graduate School , China"},{"name":"Weibo Huang","slug":"weibo-huang","given":"Weibo","family":"Huang","affiliation":"Key Laboratory of Machine Perception , Peking University , Shenzhen Graduate School , China"},{"name":"Fanyang Meng","slug":"fanyang-meng","given":"Fanyang","family":"Meng","affiliation":"Peng Cheng Laboratory , Shenzhen , China"},{"name":"Hong Liu","slug":"hong-liu","given":"Hong","family":"Liu","affiliation":"Key Laboratory of Machine Perception , Peking University , Shenzhen Graduate School , China; Peng Cheng Laboratory , Shenzhen , China"}],"abstract":"Recently, unsupervised methods for monocular visual odometry (VO), with no need for quantities of expensive labeled ground truth, have attracted much attention. However, these methods are inadequate for long-term odometry task, due to the inherent limitation of only using monocular visual data and the inability to handle the error accumulation problem. By utilizing supplemental low-cost inertial measurements, and exploiting the multi-view geometric constraint and sequential constraint, an unsupervised visual-inertial odometry framework (UnVIO) is proposed in this paper. Our method is able to predict the per-frame depth map, as well as extracting and self-adaptively fusing visual-inertial motion features from image-IMU stream to achieve long-term odometry task. A novel sliding window optimization strategy, which consists of an intra-window and an inter-window optimization, is introduced for overcoming the error accumulation and scale ambiguity problem.  The intra-window optimization restrains the geometric inferences within the window through checking the photometric consistency. And the inter-window optimization checks the 3D geometric consistency and trajectory consistency among predictions of separate windows. Extensive experiments have been conducted on KITTI and Malaga datasets to demonstrate the superiority of UnVIO over other state-of-the-art VO / VIO methods. The codes are open-source.","doi":"10.24963/ijcai.2020/325","url":"https://ijcai.org/proceedings/2020/325","pdf":"https://ijcai.org/proceedings/2020/0325.pdf","markdown":"https://ijcai.org/proceedings/2020/325.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/325","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":325,"pages":{"first":2347,"last":2354},"keywords":["Machine Learning: Unsupervised Learning","Robotics: Localization, Mapping, State Estimation","Robotics: Vision and Perception"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24242/file_24242.m3u8","length_sec":798,"provider":"hls"}],"references":[{"position":1,"raw":"[Bian et al., 2019] J. Bian, Z. Li, N. Wang, H. Zhan, C. Shen, M. Cheng, and I. Reid. Unsupervised scale-consistent depth and ego-motion learning from monocular video. In Advances in Neural Information Processing Systems (NeurIPS), pages 35-45, 2019.","title":"Unsupervised scale-consistent depth and ego-motion learning from monocular video","authors":"J Bian, Z Li, N Wang, H Zhan, C Shen, M Cheng, I Reid","year":"2019","venue":"Advances in Neural Information Processing Systems (NeurIPS)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Blanco-Claraco et al., 2014] J. L. Blanco-Claraco, F. Á. Moreno- Dueñas, and J. González-Jiménez. The málaga urban dataset: High-rate stereo and lidar in a realistic urban scenario. The In- ternational Journal of Robotics Research (IJRR), 33(2):207-214, 2014.","title":"The málaga urban dataset: High-rate stereo and lidar in a realistic urban scenario","authors":"J L Blanco-Claraco, F Á Moreno-Dueñas, J González-Jiménez","year":"2014","venue":"The International Journal of Robotics Research (IJRR)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bloesch et al., 2015] M. Bloesch, S. Omari, M. Hutter, and R. Siegwart. Robust visual inertial odometry using a direct ekf- based approach. In IEEE/RSJ international conference on intel- ligent robots and systems (IROS), pages 298-304, 2015.","title":"Robust visual inertial odometry using a direct ekfbased approach","authors":"M Bloesch, S Omari, M Hutter, R Siegwart","year":"2015","venue":"IEEE/RSJ international conference on intelligent robots and systems (IROS)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chang and Chen, 2018] J. Chang and Y. Chen. Pyramid stereo matching network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5410- 5418, 2018.","title":"Pyramid stereo matching network","authors":"Chang, Chen ;, ] J Chang, Y Chen","year":"2018","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2019] C. Chen, S. Rosa, Y. Miao, C. X. Lu, W. Wu, A. Markham, and N. Trigoni. Selective sensor fusion for neu- ral visual-inertial odometry. In Proceedings of the IEEE Confer- ence on Computer Vision and Pattern Recognition (CVPR), pages 10542-10551, 2019.","title":"Selective sensor fusion for neural visual-inertial odometry","authors":"C Chen, S Rosa, Y Miao, C X Lu, W Wu, A Markham, N Trigoni","year":"2019","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Clark et al., 2017] R. Clark, S. Wang, H. Wen, A. Markham, and N. Trigoni. Vinet: Visual-inertial odometry as a sequence-to- sequence learning problem. In Thirty-First AAAI Conference on Artificial Intelligence (AAAI), pages 3995-4001, 2017.","title":"Vinet: Visual-inertial odometry as a sequence-tosequence learning problem","authors":"R Clark, S Wang, H Wen, A Markham, N Trigoni","year":"2017","venue":"Thirty-First AAAI Conference on Artificial Intelligence (AAAI)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dosovitskiy et al., 2015] A. Dosovitskiy, P. Fischer, E. Ilg, P. Husser, C. Hazirbas, V. Golkov, P. v. d. Smagt, D. Cremers, and T. Brox. Flownet: Learning optical flow with convolutional networks. In IEEE International Conference on Computer Vision (ICCV), pages 2758-2766, 2015.","title":"Flownet: Learning optical flow with convolutional networks","authors":"A Dosovitskiy, P Fischer, E Ilg, P Husser, C Hazirbas, V Golkov, P V D Smagt, D Cremers, T Brox","year":"2015","venue":"IEEE International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Engel et al., 2017] J. Engel, V. Koltun, and D. Cremers. Direct sparse odometry. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 40(3):611-625, 2017.","title":"Direct sparse odometry","authors":"J Engel, V Koltun, D Cremers","year":"2017","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Geiger et al., 2011] A. Geiger, J. Ziegler, and C. Stiller. Stere- oscan: Dense 3d reconstruction in real-time. In IEEE Intelligent Vehicles Symposium (IV), pages 963-968, 2011.","title":"Stereoscan: Dense 3d reconstruction in real-time","authors":"A Geiger, J Ziegler, C Stiller","year":"2011","venue":"IEEE Intelligent Vehicles Symposium","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Geiger et al., 2012] A. Geiger, P. Lenz, and R. Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3354-3361, 2012.","title":"Are we ready for autonomous driving? the kitti vision benchmark suite","authors":"A Geiger, P Lenz, R Urtasun","year":"2012","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Geiger et al., 2013] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun. Vision meets robotics: The kitti dataset. The International Jour- nal of Robotics Research (IJRR), 32(11):1231-1237, 2013.","title":"Vision meets robotics: The kitti dataset","authors":"A Geiger, P Lenz, C Stiller, R Urtasun","year":"2013","venue":"The International Journal of Robotics Research (IJRR)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Godard et al., 2017] C. Godard, O. Mac Aodha, and G. J. Brostow. Unsupervised monocular depth estimation with left-right consis- tency. In Proceedings of the IEEE Conference on Computer Vi- sion and Pattern Recognition (CVPR), pages 270-279, 2017.","title":"Unsupervised monocular depth estimation with left-right consistency","authors":"C Godard, O Mac Aodha, G J Brostow","year":"2017","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Han et al., 2019] L. Han, Y. Lin, G. Du, and S. Lian. Deep- vio: Self-supervised deep learning of monocular visual iner- tial odometry using 3d geometric constraints. arXiv preprint arXiv:1906.11435, 2019.","title":"Deepvio: Self-supervised deep learning of monocular visual inertial odometry using 3d geometric constraints","authors":"L Han, Y Lin, G Du, S Lian","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[He et al., 2016] K. He, X. Zhang, S. Ren, and J. Sun. Deep resid- ual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"K He, X Zhang, S Ren, J Sun","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Hochreiter and Schmidhuber, 1997] S. Hochreiter and J. Schmid- huber. Long short-term memory. Neural computation, 9(8):1735-1780, 1997.","title":"Long short-term memory","authors":"S Hochreiter, J Schmidhuber","year":"1997","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Huang and Liu, 2018] W. Huang and H. Liu. Online initialization and automatic camera-imu extrinsic calibration for monocular visual-inertial SLAM. In International Conference on Robotics and Automation (ICRA), pages 5182-5189, 2018.","title":"Online initialization and automatic camera-imu extrinsic calibration for monocular visual-inertial SLAM","authors":"W Huang, H Liu","year":"2018","venue":"International Conference on Robotics and Automation (ICRA)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Leutenegger et al., 2013] S. Leutenegger, P. T. Furgale, V. Rabaud, M. Chli, K. Konolige, and R. Siegwart. Keyframe-based visual- inertial slam using nonlinear optimization. In Proceedings of Robotis Science and Systems (RSS), 2013.","title":"Keyframe-based visualinertial slam using nonlinear optimization","authors":"S Leutenegger, P T Furgale, V Rabaud, M Chli, K Konolige, R Siegwart","year":"2013","venue":"Proceedings of Robotis Science and Systems (RSS)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Mur-Artal and Tardós, 2017] R. Mur-Artal and J. D. Tardós. ORB- SLAM2: an open-source SLAM system for monocular, stereo and RGB-D cameras. IEEE Transactions on Robotics (TRO), 33(5):1255-1262, 2017.","title":"ORB-SLAM2: an open-source SLAM system for monocular, stereo and RGB-D cameras","authors":"R Mur-Artal, J D Tardós","year":"2017","venue":"IEEE Transactions on Robotics (TRO)","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Qin et al., 2018] T. Qin, P. Li, and S. Shen. Vins-mono: A ro- bust and versatile monocular visual-inertial state estimator. IEEE Transactions on Robotics (TRO), 34(4):1004-1020, 2018.","title":"Vins-mono: A robust and versatile monocular visual-inertial state estimator","authors":"T Qin, P Li, S Shen","year":"2018","venue":"IEEE Transactions on Robotics (TRO)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Shamwell et al., 2018] E. J. Shamwell, S. Leung, and W. D. Noth- wang. Vision-aided absolute trajectory estimation using an unsupervised deep network with online error correction. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 2524-2531, 2018.","title":"Vision-aided absolute trajectory estimation using an unsupervised deep network with online error correction","authors":"E J Shamwell, S Leung, W D Nothwang","year":"2018","venue":"IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Shen et al., 2019] T. Shen, Z. Luo, L. Zhou, H. Deng, R. Zhang, T. Fang, and L. Quan. Beyond photometric loss for self- supervised ego-motion estimation. In International Conference on Robotics and Automation (ICRA), pages 6359-6365, 2019.","title":"Beyond photometric loss for selfsupervised ego-motion estimation","authors":"T Shen, Z Luo, L Zhou, H Deng, R Zhang, T Fang, L Quan","year":"2019","venue":"International Conference on Robotics and Automation (ICRA)","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Vankadari et al., 2019] M. B. Vankadari, S. Kumar, A. Majumder, and K. Das. Unsupervised learning of monocular depth and ego-motion using conditional patchgans. In International Joint Conference on Artificial Intelligence (IJCAI), pages 5677-5684, 2019.","title":"Unsupervised learning of monocular depth and ego-motion using conditional patchgans","authors":"M B Vankadari, S Kumar, A Majumder, K Das","year":"2019","venue":"International Joint Conference on Artificial Intelligence (IJCAI)","doi":null,"url":null,"internal_url":"/proceedings/2019/787"},{"position":23,"raw":"[Wang et al., 2004] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing (TIP), 13(4):600-612, 2004.","title":"Image quality assessment: from error visibility to structural similarity","authors":"Z Wang, A C Bovik, H R Sheikh, E P Simoncelli","year":"2004","venue":"IEEE transactions on image processing","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang et al., 2018] S. Wang, R. Clark, H. Wen, and N. Trigoni. End-to-end, sequence-to-sequence probabilistic visual odome- try through deep neural networks. The International Journal of Robotics Research (IJRR), 37(4-5):513-542, 2018.","title":"End-to-end, sequence-to-sequence probabilistic visual odometry through deep neural networks","authors":"S Wang, R Clark, H Wen, N Trigoni","year":"2018","venue":"The International Journal of Robotics Research (IJRR)","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhou et al., 2017] T. Zhou, M. Brown, N. Snavely, and D. G. Lowe. Unsupervised learning of depth and ego-motion from video. In Proceedings of the IEEE Conference on Computer Vi- sion and Pattern Recognition (CVPR), pages 1851-1858, 2017.","title":"Unsupervised learning of depth and ego-motion from video","authors":"T Zhou, M Brown, N Snavely, D G Lowe","year":"2017","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}