{"type":"ScholarlyArticle","title":"Self-supervised Monocular Depth and Visual Odometry Learning with Scale-consistent Geometric Constraints","authors":[{"name":"Mingkang Xiong","slug":"mingkang-xiong","given":"Mingkang","family":"Xiong","affiliation":"Shanghai Key Laboratory of Intelligent Sensing and Recognition , Shanghai Jiao Tong University , Shanghai , China"},{"name":"Zhenghong Zhang","slug":"zhenghong-zhang","given":"Zhenghong","family":"Zhang","affiliation":"Shanghai Key Laboratory of Intelligent Sensing and Recognition , Shanghai Jiao Tong University , Shanghai , China"},{"name":"Weilin Zhong","slug":"weilin-zhong","given":"Weilin","family":"Zhong","affiliation":"Shanghai Key Laboratory of Intelligent Sensing and Recognition , Shanghai Jiao Tong University , Shanghai , China"},{"name":"Jinsheng Ji","slug":"jinsheng-ji","given":"Jinsheng","family":"Ji","affiliation":"Shanghai Key Laboratory of Intelligent Sensing and Recognition , Shanghai Jiao Tong University , Shanghai , China"},{"name":"Jiyuan Liu","slug":"jiyuan-liu","given":"Jiyuan","family":"Liu","affiliation":"Shanghai Key Laboratory of Intelligent Sensing and Recognition , Shanghai Jiao Tong University , Shanghai , China"},{"name":"Huilin Xiong","slug":"huilin-xiong","given":"Huilin","family":"Xiong","affiliation":"Shanghai Key Laboratory of Intelligent Sensing and Recognition , Shanghai Jiao Tong University , Shanghai , China"}],"abstract":"The self-supervised learning-based depth and visual odometry (VO) estimators trained on monocular videos without ground truth have drawn significant attention recently. Prior works use photometric consistency as supervision, which is fragile under complex realistic environments due to illumination variations. More importantly, it suffers from scale inconsistency in the depth and pose estimation results. In this paper, robust geometric losses are proposed to deal with this problem. Specifically, we first align the scales of two reconstructed depth maps estimated from the adjacent image frames, and then enforce forward-backward relative pose consistency to formulate scale-consistent geometric constraints. Finally, a novel training framework is constructed to implement the proposed losses. Extensive evaluations on KITTI and Make3D datasets demonstrate that, i) by incorporating the proposed constraints as supervision, the depth estimation model can achieve state-of-the-art (SOTA) performance among the self-supervised methods, and ii) it is effective to use the proposed training framework to obtain a uniform global scale VO model.","doi":"10.24963/ijcai.2020/134","url":"https://ijcai.org/proceedings/2020/134","pdf":"https://ijcai.org/proceedings/2020/0134.pdf","markdown":"https://ijcai.org/proceedings/2020/134.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/134","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Computer Vision","track":"Main track","number":134,"pages":{"first":963,"last":969},"keywords":["Computer Vision: 2D and 3D Computer Vision","Robotics: Robotics and Vision"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24100/file_24100.m3u8","length_sec":285,"provider":"hls"}],"references":[{"position":1,"raw":"[Bian et al., 2019] Jiawang Bian, Zhichao Li, Naiyan Wang, Huangying Zhan, Chunhua Shen, Ming-Ming Cheng, and Ian Reid. Unsupervised scale-consistent depth and ego- motion learning from monocular video. In NeurIPS, 2019.","title":"Unsupervised scale-consistent depth and egomotion learning from monocular video","authors":"Bian","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cordts et al., 2016] Marius Cordts, Mohamed Omran, Se- bastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene under- standing. In CVPR, 2016.","title":"The cityscapes dataset for semantic urban scene understanding","authors":"Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, Bernt Schiele","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Eigen et al., 2014] David Eigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from a single image using a multi-scale deep network. In NeurIPS, 2014.","title":"Depth map prediction from a single image using a multi-scale deep network","authors":"David Eigen, Christian Puhrsch, Rob Fergus","year":"2014","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Garg et al., 2016] Ravi Garg, Vijay Kumar BG, Gustavo Carneiro, and Ian Reid. Unsupervised cnn for single view depth estimation: Geometry to the rescue. In ECCV, 2016.","title":"Unsupervised cnn for single view depth estimation: Geometry to the rescue","authors":"Ravi Garg, Vijay Kumar, B G, Gustavo Carneiro, Ian Reid","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Geiger et al., 2012] Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In CVPR, 2012.","title":"Are we ready for autonomous driving? the kitti vision benchmark suite","authors":"Andreas Geiger, Philip Lenz, Raquel Urtasun","year":"2012","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Godard et al., 2017] Clément Godard, Oisin Mac Aodha, and Gabriel J Brostow. Unsupervised monocular depth es- timation with left-right consistency. In CVPR, 2017.","title":"Unsupervised monocular depth estimation with left-right consistency","authors":"Clément Godard, Oisin Mac Aodha, Gabriel J Brostow","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Jaderberg et al., 2015] Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al. Spatial transformer networks. In NeurIPS, 2015.","title":"Spatial transformer networks","authors":"Jaderberg","year":"2015","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kerl et al., 2013] Christian Kerl, Jürgen Sturm, and Daniel Cremers. Dense visual slam for rgb-d cameras. In IROS, 2013.","title":"Dense visual slam for rgb-d cameras","authors":"Kerl","year":"2013","venue":"IROS","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Konda and Memisevic, 2015] Kishore Reddy Konda and Roland Memisevic. Learning visual odometry with a con- volutional network. In VISAPP (1), 2015.","title":"Learning visual odometry with a convolutional network","authors":"Kishore Reddy, Konda, Roland Memisevic","year":"2015","venue":"VISAPP","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Laina et al., 2016] Iro Laina, Christian Rupprecht, Vasileios Belagiannis, Federico Tombari, and Nassir Navab. Deeper depth prediction with fully convolutional residual net- works. In 3DV, 2016.","title":"Deeper depth prediction with fully convolutional residual networks","authors":"Iro Laina, Christian Rupprecht, Vasileios Belagiannis, Federico Tombari, Nassir Navab","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2018a] Bo Li, Yuchao Dai, and Mingyi He. Monocular depth estimation with hierarchical fusion of dilated cnns and soft-weighted-sum inference. Pattern Recognition, 83:328-339, 2018.","title":"Monocular depth estimation with hierarchical fusion of dilated cnns and soft-weighted-sum inference","authors":"Li","year":"2018","venue":"Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2018b] Ruihao Li, Sen Wang, Zhiqiang Long, and Dongbing Gu. Undeepvo: Monocular visual odometry through unsupervised deep learning. In ICRA, 2018.","title":"Undeepvo: Monocular visual odometry through unsupervised deep learning","authors":"Li","year":"2018","venue":"ICRA","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mahjourian et al., 2018] Reza Mahjourian, Martin Wicke, and Anelia Angelova. Unsupervised learning of depth and ego-motion from monocular video using 3d geomet- ric constraints. In CVPR, 2018.","title":"Unsupervised learning of depth and ego-motion from monocular video using 3d geometric constraints","authors":"Reza Mahjourian, Martin Wicke, Anelia Angelova","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mur-Artal et al., 2015] Raul Mur-Artal, Jose Maria Mar- tinez Montiel, and Juan D Tardos. Orb-slam: a versatile and accurate monocular slam system. IEEE transactions on robotics, 31(5):1147-1163, 2015.","title":"Orb-slam: a versatile and accurate monocular slam system","authors":"Mur-Artal","year":"2015","venue":"IEEE transactions on robotics","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Paszke et al., 2019] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. In NeurIPS, 2019.","title":"Pytorch: An imperative style, highperformance deep learning library","authors":"Paszke","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Ranjan et al., 2019] Anurag Ranjan, Varun Jampani, Lukas Balles, Kihwan Kim, Deqing Sun, Jonas Wulff, and Michael J Black. Competitive collaboration: Joint unsu- pervised learning of depth, camera motion, optical flow and motion segmentation. In CVPR, 2019.","title":"Competitive collaboration: Joint unsupervised learning of depth, camera motion, optical flow and motion segmentation","authors":"Anurag Ranjan, Varun Jampani, Lukas Balles, Kihwan Kim, Deqing Sun, Jonas Wulff, Michael J Black","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Rublee et al., 2011] Ethan Rublee, Vincent Rabaud, Kurt Konolige, and Gary Bradski. Orb: An efficient alternative to sift or surf. In ICCV, 2011.","title":"Orb: An efficient alternative to sift or surf","authors":"Rublee","year":"2011","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Saxena et al., 2009] A. Saxena, M. Sun, and A. Y. Ng. Make3d: Learning 3d scene structure from a single still image. IEEE Transactions on Pattern Analysis and Ma- chine Intelligence, 31(5):824-840, 2009.","title":"Make3d: Learning 3d scene structure from a single still image","authors":"A Saxena, M Sun, A Y Ng","year":"2009","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Schonberger and Frahm, 2016] Johannes L. Schonberger and Jan-Michael Frahm. Structure-from-motion revisited. In CVPR, 2016.","title":"Structure-from-motion revisited","authors":"Johannes L Schonberger, Jan-Michael Frahm","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Vankadari et al., 2019] Madhu Vankadari, Swagat Kumar, Anima Majumder, and Kaushik Das. Unsupervised learn- ing of monocular depth and ego-motion using conditional patchgans. In IJCAI, 2019.","title":"Unsupervised learning of monocular depth and ego-motion using conditional patchgans","authors":"Madhu Vankadari, Swagat Kumar, Anima Majumder, Kaushik Das","year":"2019","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2019/787"},{"position":22,"raw":"[Wang et al., 2004] Zhou Wang, Alan C Bovik, Hamid R Sheikh, Eero P Simoncelli, et al. Image quality assess- ment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600-612, 2004.","title":"Image quality assessment: from error visibility to structural similarity","authors":"Zhou Wang, Alan C Bovik, Eero P Hamid R Sheikh, Simoncelli","year":"2004","venue":"IEEE transactions on image processing","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wang et al., 2017] Sen Wang, Ronald Clark, Hongkai Wen, and Niki Trigoni. Deepvo: Towards end-to-end visual odometry with deep recurrent convolutional neural net- works. In ICRA, 2017.","title":"Deepvo: Towards end-to-end visual odometry with deep recurrent convolutional neural networks","authors":"Sen Wang, Ronald Clark, Hongkai Wen, Niki Trigoni","year":"2017","venue":"ICRA","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang et al., 2018] Chaoyang Wang, José Miguel Buena- posada, Rui Zhu, and Simon Lucey. Learning depth from monocular videos using direct methods. In CVPR, 2018.","title":"Learning depth from monocular videos using direct methods","authors":"Wang","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Xue et al., 2019] Fei Xue, Xin Wang, Shunkai Li, Qiuyuan Wang, Junqiu Wang, and Hongbin Zha. Beyond track- ing: Selecting memory and refining poses for deep visual odometry. In CVPR, 2019.","title":"Beyond tracking: Selecting memory and refining poses for deep visual odometry","authors":"Fei Xue, Xin Wang, Shunkai Li, Qiuyuan Wang, Junqiu Wang, Hongbin Zha","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yin and Shi, 2018] Zhichao Yin and Jianping Shi. Geonet: Unsupervised learning of dense depth, optical flow and camera pose. In CVPR, 2018.","title":"Geonet: Unsupervised learning of dense depth, optical flow and camera pose","authors":"Zhichao Yin, Jianping Shi","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhan et al., 2018] Huangying Zhan, Ravi Garg, Chamara Saroj Weerasekera, Kejie Li, Harsh Agarwal, and Ian Reid. Unsupervised learning of monocular depth estima- tion and visual odometry with deep feature reconstruction. In CVPR, 2018.","title":"Unsupervised learning of monocular depth estimation and visual odometry with deep feature reconstruction","authors":"Huangying Zhan, Ravi Garg, Chamara Saroj Weerasekera, Kejie Li, Harsh Agarwal, Ian Reid","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhou et al., 2016] Tinghui Zhou, Shubham Tulsiani, Weilun Sun, Jitendra Malik, and Alexei A Efros. View synthesis by appearance flow. In ECCV, 2016.","title":"View synthesis by appearance flow","authors":"Zhou","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhou et al., 2017] Tinghui Zhou, Matthew Brown, Noah Snavely, and David G Lowe. Unsupervised learning of depth and ego-motion from video. In CVPR, 2017.","title":"Unsupervised learning of depth and ego-motion from video","authors":"Zhou","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}