{"type":"ScholarlyArticle","title":"Collaborative Learning of Depth Estimation, Visual Odometry and Camera Relocalization from Monocular Videos","authors":[{"name":"Haimei Zhao","slug":"haimei-zhao","given":"Haimei","family":"Zhao","affiliation":"Shenzhen International Graduate School , Tsinghua University; UBTECH Sydney AI Centre , School of CS , Faculty of Engineering , The University of Sydney , Australia"},{"name":"Wei Bian","slug":"wei-bian","given":"Wei","family":"Bian","affiliation":"Center for Artificial Intelligence , University of Technology Sydney"},{"name":"Bo Yuan","slug":"bo-yuan","given":"Bo","family":"Yuan","affiliation":"Shenzhen International Graduate School , Tsinghua University"},{"name":"Dacheng Tao","slug":"dacheng-tao","given":"Dacheng","family":"Tao","affiliation":"UBTECH Sydney AI Centre , School of CS , Faculty of Engineering , The University of Sydney , Australia"}],"abstract":"Scene perceiving and understanding tasks including depth estimation, visual odometry (VO) and camera relocalization are fundamental for applications such as autonomous driving, robots and drones. Driven by the power of deep learning, significant progress has been achieved on individual tasks but the rich correlations among the three tasks are largely neglected. In previous studies, VO is generally accurate in local scope yet suffers from drift in long distances. By contrast, camera relocalization performs well in the global sense but lacks local precision. We argue that these two tasks should be strategically combined to leverage the complementary advantages, and be further improved by exploiting the 3D geometric information from depth data, which is also beneficial for depth estimation in turn. Therefore, we present a collaborative learning framework, consisting of DepthNet, LocalPoseNet and GlobalPoseNet with a joint optimization loss to estimate depth, VO and camera localization unitedly. Moreover, the Geometric Attention Guidance Model is introduced to exploit the geometric relevance among three branches during learning. Extensive experiments demonstrate that the joint learning scheme is useful for all tasks and our method outperforms current state-of-the-art techniques in depth estimation and camera relocalization with highly competitive performance in VO.","doi":"10.24963/ijcai.2020/68","url":"https://ijcai.org/proceedings/2020/68","pdf":"https://ijcai.org/proceedings/2020/0068.pdf","markdown":"https://ijcai.org/proceedings/2020/68.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/68","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Computer Vision","track":"Main track","number":68,"pages":{"first":488,"last":494},"keywords":["Robotics: Localization, Mapping, State Estimation","Computer Vision: 2D and 3D Computer Vision"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24385/file_24385.m3u8","length_sec":280,"provider":"hls"}],"references":[{"position":1,"raw":"[Brahmbhatt et al., 2018] Samarth Brahmbhatt, Jinwei Gu, Kihwan Kim, James Hays, and Jan Kautz. Geometry- aware learning of maps for camera localization. In CVPR, pages 2616-2625, 2018.","title":"Geometryaware learning of maps for camera localization","authors":"Samarth Brahmbhatt, Jinwei Gu, Kihwan Kim, James Hays, Jan Kautz","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Casser et al., 2019] Vincent Casser, Soeren Pirk, Reza Mahjourian, and Anelia Angelova. Depth prediction with- out the sensors: Leveraging structure for unsupervised learning from monocular videos. In AAAI, pages 8001- 8008, 2019.","title":"Depth prediction without the sensors: Leveraging structure for unsupervised learning from monocular videos","authors":"Vincent Casser, Soeren Pirk, Reza Mahjourian, Anelia Angelova","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Eigen et al., 2014] David Eigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from a single image using a multi-scale deep network. In NIPS, pages 2366- 2374, 2014.","title":"Depth map prediction from a single image using a multi-scale deep network","authors":"David Eigen, Christian Puhrsch, Rob Fergus","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Fraundorfer and Scaramuzza, 2012] Friedrich Fraundorfer and Davide Scaramuzza. Visual odometry: Part ii: Match- ing, robustness, optimization, and applications. IEEE Robotics & Automation Magazine, 19(2):78-90, 2012.","title":"Friedrich Fraundorfer and Davide Scaramuzza. Visual odometry: Part ii: Matching, robustness, optimization, and applications","authors":"Scaramuzza Fraundorfer","year":"2012","venue":"IEEE Robotics & Automation Magazine","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Fu et al., 2018] Huan Fu, Mingming Gong, Chaohui Wang, Kayhan Batmanghelich, and Dacheng Tao. Deep ordinal regression network for monocular depth estimation. In CVPR, pages 2002-2011, 2018.","title":"Deep ordinal regression network for monocular depth estimation","authors":"Huan Fu, Mingming Gong, Chaohui Wang, Kayhan Batmanghelich, Dacheng Tao","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Garg et al., 2016] Ravi Garg, Vijay Kumar BG, Gustavo Carneiro, and Ian Reid. Unsupervised cnn for single view depth estimation: Geometry to the rescue. In ECCV, pages 740-756, 2016.","title":"Unsupervised cnn for single view depth estimation: Geometry to the rescue","authors":"Ravi Garg, Vijay Kumar, B G, Gustavo Carneiro, Ian Reid","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Geiger et al., 2012] Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In CVPR, pages 3354- 3361, 2012.","title":"Are we ready for autonomous driving? the kitti vision benchmark suite","authors":"Andreas Geiger, Philip Lenz, Raquel Urtasun","year":"2012","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Godard et al., 2017] Clément Godard, Oisin Mac Aodha, and Gabriel J Brostow. Unsupervised monocular depth es- timation with left-right consistency. In CVPR, pages 270- 279, 2017.","title":"Unsupervised monocular depth estimation with left-right consistency","authors":"Clément Godard, Oisin Mac Aodha, Gabriel J Brostow","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Godard et al., 2019] Clément Godard, Oisin Mac Aodha, Michael Firman, and Gabriel J Brostow. Digging into self- supervised monocular depth estimation. In ICCV, pages 3828-3838, 2019.","title":"Digging into selfsupervised monocular depth estimation","authors":"Clément Godard, Oisin Mac Aodha, Michael Firman, Gabriel J Brostow","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Grupp, 2017] Michael Grupp. evo: Python package for the evaluation of odometry and slam. https://github.com/ MichaelGrupp/evo, 2017.","title":"evo: Python package for the evaluation of odometry and slam","authors":"Michael Grupp","year":"2017","venue":null,"doi":null,"url":"https://github.com/MichaelGrupp/evo","internal_url":null},{"position":11,"raw":"[He et al., 2017] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, pages 2961-2969, 2017.","title":"Kaiming He, Georgia Gkioxari","authors":"He","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Iyer et al., 2018] Ganesh Iyer, J. Krishna Murthy, Gunshi Gupta, Madhava Krishna, and Liam Paull. Geometric con- sistency for self-supervised end-to-end visual odometry. In CVPR Workshops, 2018.","title":"Geometric consistency for self-supervised end-to-end visual odometry","authors":"J Krishna Ganesh Iyer, Gunshi Murthy, Madhava Gupta, Liam Krishna, Paull","year":"2018","venue":"CVPR Workshops","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Jaderberg et al., 2015] Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al. Spatial transformer networks. In NIPS, pages 2017-2025, 2015.","title":"Spatial transformer networks","authors":"Jaderberg","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kendall et al., 2015] Alex Kendall, Matthew Grimes, and Roberto Cipolla. Posenet: A convolutional network for real-time 6-dof camera relocalization. In ICCV, 2015.","title":"Posenet: A convolutional network for real-time 6-dof camera relocalization","authors":"Alex Kendall, Matthew Grimes, Roberto Cipolla","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2018] Ruihao Li, Sen Wang, Zhiqiang Long, and Dongbing Gu. Undeepvo: Monocular visual odome- try through unsupervised deep learning. In ICRA, pages 7286-7291. IEEE, 2018.","title":"Undeepvo: Monocular visual odometry through unsupervised deep learning","authors":"Li","year":"2018","venue":"ICRA","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Mur-Artal et al., 2015] Raul Mur-Artal, Jose Maria Mar- tinez Montiel, and Juan D Tardos. Orb-slam: a versatile and accurate monocular slam system. IEEE transactions on robotics, 31(5):1147-1163, 2015.","title":"Orb-slam: a versatile and accurate monocular slam system","authors":"Mur-Artal","year":"2015","venue":"IEEE transactions on robotics","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Russakovsky et al., 2015] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. IJCV, 115(3):211-252, 2015.","title":"Imagenet large scale visual recognition challenge","authors":"Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein","year":"2015","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Tosi et al., 2019] Fabio Tosi, Filippo Aleotti, Matteo Poggi, and Stefano Mattoccia. Learning monocular depth esti- mation infusing traditional stereo knowledge. In CVPR, pages 9799-9809, 2019.","title":"Learning monocular depth estimation infusing traditional stereo knowledge","authors":"Fabio Tosi, Filippo Aleotti, Matteo Poggi, Stefano Mattoccia","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wang et al., 2017] Sen Wang, Ronald Clark, Hongkai Wen, and Niki Trigoni. Deepvo: Towards end-to-end visual odometry with deep recurrent convolutional neural net- works. In ICRA, pages 2043-2050, 2017.","title":"Deepvo: Towards end-to-end visual odometry with deep recurrent convolutional neural networks","authors":"Sen Wang, Ronald Clark, Hongkai Wen, Niki Trigoni","year":"2017","venue":"ICRA","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wang et al., 2018] Chaoyang Wang, José Miguel Buena- posada, Rui Zhu, and Simon Lucey. Learning depth from monocular videos using direct methods. In CVPR, pages 2022-2030, 2018.","title":"Learning depth from monocular videos using direct methods","authors":"Wang","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang et al., 2019] Bing Wang, Changhao Chen, Chris Xi- aoxuan Lu, Peijun Zhao, Niki Trigoni, and Andrew Markham. Atloc: Attention guided camera localization. arXiv preprint arXiv:1909.03557, 2019.","title":"Atloc: Attention guided camera localization","authors":"Bing Wang, Changhao Chen, Chris Xiaoxuan Lu, Peijun Zhao, Niki Trigoni, Andrew Markham","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Woo et al., 2018] Sanghyun Woo, Jongchan Park, Joon- Young Lee, and In So Kweon. Cbam: Convolutional block attention module. In ECCV, pages 3-19, 2018.","title":"Cbam: Convolutional block attention module","authors":"Sanghyun Woo, Jongchan Park, Joon-Young Lee, In So Kweon","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Xu et al., 2019] Haofei Xu, Jianmin Zheng, Jianfei Cai, and Juyong Zhang. Region deformer networks for unsu- pervised depth estimation from unconstrained monocular videos. In IJCAI, 2019.","title":"Region deformer networks for unsupervised depth estimation from unconstrained monocular videos","authors":"Haofei Xu, Jianmin Zheng, Jianfei Cai, Juyong Zhang","year":"2019","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2019/788"},{"position":24,"raw":"[Yin and Shi, 2018] Zhichao Yin and Jianping Shi. Geonet: Unsupervised learning of dense depth, optical flow and camera pose. In CVPR, pages 1983-1992, 2018.","title":"Geonet: Unsupervised learning of dense depth, optical flow and camera pose","authors":"Zhichao Yin, Jianping Shi","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhan et al., 2018] Huangying Zhan, Ravi Garg, Chamara Saroj Weerasekera, Kejie Li, Harsh Agarwal, and Ian Reid. Unsupervised learning of monocular depth estima- tion and visual odometry with deep feature reconstruction. In CVPR, pages 340-349, 2018.","title":"Unsupervised learning of monocular depth estimation and visual odometry with deep feature reconstruction","authors":"Huangying Zhan, Ravi Garg, Chamara Saroj Weerasekera, Kejie Li, Harsh Agarwal, Ian Reid","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"Zhou et al., ] Tinghui Zhou, Matthew Brown, Noah Snavely, and David G Lowe. Unsupervised learning of depth and ego-motion from video. In CVPR.","title":"Unsupervised learning of depth and ego-motion from video","authors":"Zhou","year":null,"venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhou et al., 2019] Junsheng Zhou, Yuwang Wang, Kaihuai Qin, and Wenjun Zeng. Unsupervised high-resolution depth learning from videos with dual networks. In ICCV, pages 6872-6881, 2019.","title":"Unsupervised high-resolution depth learning from videos with dual networks","authors":"Junsheng Zhou, Yuwang Wang, Kaihuai Qin, Wenjun Zeng","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zou et al., 2018] Yuliang Zou, Zelun Luo, and Jia-Bin Huang. Df-net: Unsupervised joint learning of depth and flow using cross-task consistency. In ECCV, pages 36-53, 2018.","title":"Df-net: Unsupervised joint learning of depth and flow using cross-task consistency","authors":"Yuliang Zou, Zelun Luo, Jia-Bin Huang","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}