{"type":"ScholarlyArticle","title":"Region Deformer Networks for Unsupervised Depth Estimation from Unconstrained Monocular Videos","authors":[{"name":"Haofei Xu","slug":"haofei-xu","given":"Haofei","family":"Xu","affiliation":"University of Science and Technology of China"},{"name":"Jianmin Zheng","slug":"jianmin-zheng","given":"Jianmin","family":"Zheng","affiliation":"Nanyang Technological University"},{"name":"Jianfei Cai","slug":"jianfei-cai","given":"Jianfei","family":"Cai","affiliation":"Nanyang Technological University"},{"name":"Juyong Zhang","slug":"juyong-zhang","given":"Juyong","family":"Zhang","affiliation":"University of Science and Technology of China"}],"abstract":"While learning based depth estimation from images/videos has achieved substantial progress, there still exist intrinsic limitations. Supervised methods are limited by a small amount of ground truth  or labeled data  and unsupervised methods for monocular videos are mostly based on the static scene assumption, not performing well on real world scenarios with the presence of dynamic objects. In this paper, we propose a new learning based method consisting of DepthNet, PoseNet and Region Deformer Networks (RDN) to estimate depth from unconstrained monocular videos without ground truth supervision. The core contribution lies in RDN for proper handling of rigid and non-rigid motions of various objects such as rigidly moving cars and deformable humans. In particular, a deformation based motion representation is proposed to model individual object motion on 2D images. This representation enables our method to be applicable to diverse unconstrained monocular videos. Our method can not only achieve the state-of-the-art results on standard benchmarks KITTI and Cityscapes, but also show promising results on a crowded pedestrian tracking dataset, which demonstrates the effectiveness of the deformation based motion representation. Code and trained models are available at https://github.com/haofeixu/rdn4depth.","doi":"10.24963/ijcai.2019/788","url":"https://ijcai.org/proceedings/2019/788","pdf":"https://ijcai.org/proceedings/2019/0788.pdf","markdown":"https://ijcai.org/proceedings/2019/788.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/788","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Robotics","track":"Main track","number":788,"pages":{"first":5685,"last":5691},"keywords":["Robotics: Localization, Mapping, State Estimation","Computer Vision: 2D and 3D Computer Vision","Computer Vision: Motion and Tracking","Robotics: Robotics and Vision"],"videos":[],"references":[{"position":1,"raw":"[Casser et al., 2019] Vincent Casser, Soeren Pirk, Reza Mahjourian, and Anelia Angelova. Depth prediction with- out the sensors: Leveraging structure for unsupervised learning from monocular videos. In AAAI, 2019.","title":"Depth prediction without the sensors: Leveraging structure for unsupervised learning from monocular videos","authors":"Vincent Casser, Soeren Pirk, Reza Mahjourian, Anelia Angelova","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2016] Weifeng Chen, Zhao Fu, Dawei Yang, and Jia Deng. Single-image depth perception in the wild. In NIPS, pages 730-738, 2016.","title":"Single-image depth perception in the wild","authors":"Weifeng Chen, Zhao Fu, Dawei Yang, Jia Deng","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cordts et al., 2016] Marius Cordts, Mohamed Omran, Se- bastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene under- standing. In CVPR, pages 3213-3223, 2016.","title":"The cityscapes dataset for semantic urban scene understanding","authors":"Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, Bernt Schiele","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dai et al., 2017] Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. De- formable convolutional networks. CoRR, abs/1703.06211, 1(2):3, 2017.","title":"Deformable convolutional networks","authors":"Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, Yichen Wei","year":"2017","venue":"CoRR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Eigen et al., 2014] David Eigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from a single image using a multi-scale deep network. In NIPS, pages 2366- 2374, 2014.","title":"Depth map prediction from a single image using a multi-scale deep network","authors":"David Eigen, Christian Puhrsch, Rob Fergus","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Ess et al., 2009] Andreas Ess, Bastian Leibe, Konrad Schindler, and Luc Van Gool. Robust multiperson track- ing from a mobile platform. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(10):1831-1846, 2009.","title":"Robust multiperson tracking from a mobile platform","authors":"Andreas Ess, Bastian Leibe, Konrad Schindler, Luc Van Gool","year":"2009","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Garg et al., 2016] Ravi Garg, Vijay Kumar BG, Gustavo Carneiro, and Ian Reid. Unsupervised cnn for single view depth estimation: Geometry to the rescue. In ECCV, pages 740-756. Springer, 2016.","title":"Unsupervised cnn for single view depth estimation: Geometry to the rescue","authors":"Ravi Garg, Vijay Kumar, B G, Gustavo Carneiro, Ian Reid","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Geiger et al., 2012] Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In CVPR, pages 3354- 3361. IEEE, 2012.","title":"Are we ready for autonomous driving? the kitti vision benchmark suite","authors":"Andreas Geiger, Philip Lenz, Raquel Urtasun","year":"2012","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Godard et al., 2017] Clément Godard, Oisin Mac Aodha, and Gabriel J Brostow. Unsupervised monocular depth es- timation with left-right consistency. In CVPR, volume 2, page 7, 2017.","title":"Unsupervised monocular depth estimation with left-right consistency","authors":"Clément Godard, Oisin Mac Aodha, Gabriel J Brostow","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Godard et al., 2018] Clément Godard, Oisin Mac Aodha, and Gabriel Brostow. Digging into self-supervised monoc- ular depth estimation. arXiv preprint arXiv:1806.01260, 2018.","title":"Digging into self-supervised monocular depth estimation","authors":"Clément Godard, Oisin Mac Aodha, Gabriel Brostow","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[He et al., 2017] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, pages 2980-2988. IEEE, 2017.","title":"Kaiming He, Georgia Gkioxari","authors":"He","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Jaderberg et al., 2015] Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al. Spatial transformer networks. In NIPS, pages 2017-2025, 2015.","title":"Spatial transformer networks","authors":"Jaderberg","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kingma and Ba, 2014] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li and Snavely, 2018] Zhengqi Li and Noah Snavely. Megadepth: Learning single-view depth prediction from internet photos. In CVPR, pages 2041-2050, 2018.","title":"Megadepth: Learning single-view depth prediction from internet photos","authors":"Snavely ; Li, Zhengqi Li, Noah Snavely","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2015] Bo Li, Chunhua Shen, Yuchao Dai, Anton Van Den Hengel, and Mingyi He. Depth and surface nor- mal estimation from monocular images using regression on deep features and hierarchical crfs. In CVPR, pages 1119-1127, 2015.","title":"Depth and surface normal estimation from monocular images using regression on deep features and hierarchical crfs","authors":"Bo Li, Chunhua Shen, Yuchao Dai, Anton Van Den, Mingyi Hengel, He","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liu et al., 2016] Fayao Liu, Chunhua Shen, Guosheng Lin, and Ian Reid. Learning depth from single monocular im- ages using deep convolutional neural fields. IEEE Trans. Pattern Anal. Mach. Intell., 38(10):2024-2039, 2016.","title":"Learning depth from single monocular images using deep convolutional neural fields","authors":"Fayao Liu, Chunhua Shen, Guosheng Lin, Ian Reid","year":"2016","venue":"IEEE Trans. Pattern Anal. Mach. Intell","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Mahjourian et al., 2018] Reza Mahjourian, Martin Wicke, and Anelia Angelova. Unsupervised learning of depth and ego-motion from monocular video using 3d geomet- ric constraints. In CVPR, pages 5667-5675, 2018.","title":"Unsupervised learning of depth and ego-motion from monocular video using 3d geometric constraints","authors":"Reza Mahjourian, Martin Wicke, Anelia Angelova","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Mayer et al., 2018] Nikolaus Mayer, Eddy Ilg, Philipp Fis- cher, Caner Hazirbas, Daniel Cremers, Alexey Dosovit- skiy, and Thomas Brox. What makes good synthetic train- ing data for learning disparity and optical flow estima- tion? International Journal of Computer Vision, pages 1-19, 2018.","title":"What makes good synthetic training data for learning disparity and optical flow estimation?","authors":"Nikolaus Mayer, Eddy Ilg, Philipp Fischer, Caner Hazirbas, Daniel Cremers, Alexey Dosovitskiy, Thomas Brox","year":"2018","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Ranftl et al., 2016] Rene Ranftl, Vibhav Vineet, Qifeng Chen, and Vladlen Koltun. Dense monocular depth esti- mation in complex dynamic scenes. In CVPR, pages 4058- 4066, 2016.","title":"Dense monocular depth estimation in complex dynamic scenes","authors":"Rene Ranftl, Vibhav Vineet, Qifeng Chen, Vladlen Koltun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wang et al., 2004] Zhou Wang, Alan C Bovik, Hamid R Sheikh, Eero P Simoncelli, et al. Image quality assess- ment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600-612, 2004.","title":"Image quality assessment: from error visibility to structural similarity","authors":"Zhou Wang, Alan C Bovik, Eero P Hamid R Sheikh, Simoncelli","year":"2004","venue":"IEEE transactions on image processing","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang et al., 2018] Chaoyang Wang, José Miguel Buena- posada, Rui Zhu, and Simon Lucey. Learning depth from monocular videos using direct methods. In CVPR, pages 2022-2030, 2018.","title":"Learning depth from monocular videos using direct methods","authors":"Wang","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Xie et al., 2016] Junyuan Xie, Ross Girshick, and Ali Farhadi. Deep3d: Fully automatic 2d-to-3d video conver- sion with deep convolutional neural networks. In ECCV, pages 842-857. Springer, 2016.","title":"Deep3d: Fully automatic 2d-to-3d video conversion with deep convolutional neural networks","authors":"Junyuan Xie, Ross Girshick, Ali Farhadi","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yin and Shi, 2018] Zhichao Yin and Jianping Shi. Geonet: Unsupervised learning of dense depth, optical flow and camera pose. In CVPR, volume 2, 2018.","title":"Geonet: Unsupervised learning of dense depth, optical flow and camera pose","authors":"Zhichao Yin, Jianping Shi","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zheng et al., 2018] Chuanxia Zheng, Tat-Jen Cham, and Jianfei Cai. T2net: Synthetic-to-realistic translation for solving single-image depth estimation tasks. In ECCV, pages 767-783, 2018.","title":"T2net: Synthetic-to-realistic translation for solving single-image depth estimation tasks","authors":"Zheng","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhou et al., 2017] Tinghui Zhou, Matthew Brown, Noah Snavely, and David G Lowe. Unsupervised learning of depth and ego-motion from video. In CVPR, volume 2, page 7, 2017.","title":"Unsupervised learning of depth and ego-motion from video","authors":"Zhou","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zou et al., 2018] Yuliang Zou, Zelun Luo, and Jia-Bin Huang. Df-net: Unsupervised joint learning of depth and flow using cross-task consistency. In ECCV, pages 38-55. Springer, 2018.","title":"Df-net: Unsupervised joint learning of depth and flow using cross-task consistency","authors":"Yuliang Zou, Zelun Luo, Jia-Bin Huang","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}