{"type":"ScholarlyArticle","title":"Unsupervised Learning of Monocular Depth and Ego-Motion using Conditional PatchGANs","authors":[{"name":"Madhu Vankadari","slug":"madhu-vankadari","given":"Madhu","family":"Vankadari","affiliation":"TCS Research and Innovation , Bangalore , India"},{"name":"Swagat Kumar","slug":"swagat-kumar","given":"Swagat","family":"Kumar","affiliation":"TCS Research and Innovation , Bangalore , India"},{"name":"Anima Majumder","slug":"anima-majumder","given":"Anima","family":"Majumder","affiliation":"TCS Research and Innovation , Bangalore , India"},{"name":"Kaushik Das","slug":"kaushik-das","given":"Kaushik","family":"Das","affiliation":"TCS Research and Innovation , Bangalore , India"}],"abstract":"This paper presents a new GAN-based deep learning framework for estimating absolute scale awaredepth and ego motion from monocular images using a completely unsupervised mode of learning.The proposed architecture uses two separate generators to learn the distribution of depth and posedata for a given input image sequence. The depth and pose data, thus generated, are then evaluated bya patch-based discriminator using the reconstructed image and its corresponding actual image. Thepatch-based GAN (or PatchGAN) is shown to detect high frequency local structural defects in thereconstructed image, thereby improving the accuracy of overall depth and pose estimation. Unlikeconventional GANs, the proposed architecture uses a conditioned version of input and output of thegenerator for training the whole network. The resulting framework is shown to outperform all existing deep networks in this field and beating the current state-of-the-art method by 8.7% in absoluteerror and 5.2% in RMSE metric. To the best of our knowledge, this is first deep network based modelto estimate both depth and pose simultaneously using a conditional patch-based GAN paradigm.The efficacy of the proposed approach is demonstrated through rigorous ablation studies and exhaustive performance comparison on the popular KITTI outdoor driving dataset.","doi":"10.24963/ijcai.2019/787","url":"https://ijcai.org/proceedings/2019/787","pdf":"https://ijcai.org/proceedings/2019/0787.pdf","markdown":"https://ijcai.org/proceedings/2019/787.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/787","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Robotics","track":"Main track","number":787,"pages":{"first":5677,"last":5684},"keywords":["Robotics: Localization, Mapping, State Estimation","Computer Vision: 2D and 3D Computer Vision","Robotics: Robotics and Vision"],"videos":[],"references":[{"position":1,"raw":"[Aleotti et al., 2018] Filippo Aleotti, Fabio Tosi, Matteo Poggi, and Stefano Mattoccia. Generative adversarial net- works for unsupervised monocular depth prediction. In 15th European Conference on Computer Vision (ECCV) Workshops, volume 1, page 8, 2018.","title":"Generative adversarial networks for unsupervised monocular depth prediction","authors":"Filippo Aleotti, Fabio Tosi, Matteo Poggi, Stefano Mattoccia","year":"2018","venue":"15th European Conference on Computer Vision (ECCV) Workshops","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Almalioglu et al., 2019] Yasin Almalioglu, Muhamad Risqi U Saputra, Pedro PB de Gusmao, Andrew Markham, and Niki Trigoni. Ganvo: Unsupervised deep monocular visual odometry and depth estimation with generative adversarial networks. IEEE International Conference on Robotics and Automation (ICRA), 2019.","title":"Ganvo: Unsupervised deep monocular visual odometry and depth estimation with generative adversarial networks","authors":"Yasin Almalioglu, Muhamad Risqi, U Saputra, Pedro Pb De Gusmao, Andrew Markham, Niki Trigoni","year":"2019","venue":"IEEE International Conference on Robotics and Automation (ICRA)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Babu et al., 2018] V Madhu Babu, Kaushik Das, Anima Majumdar, and Swagat Kumar. Undemon: Unsuper- vised deep network for depth and ego-motion estimation. In 2018 IEEE/RSJ International Conference on Intelli- gent Robots and Systems (IROS), pages 1082-1088. IEEE, 2018.","title":"Undemon: Unsupervised deep network for depth and ego-motion estimation","authors":"Kaushik V Madhu Babu, Anima Das, Swagat Majumdar, Kumar","year":"2018","venue":"IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Eigen et al., 2014] David Eigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from a single image using a multi-scale deep network. In Advances in neural information processing systems, pages 2366-2374, 2014.","title":"Depth map prediction from a single image using a multi-scale deep network","authors":"Eigen","year":"2014","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Geiger et al., 2011] Andreas Geiger, Julius Ziegler, and Christoph Stiller. Stereoscan: Dense 3d reconstruction in real-time. In IEEE Intelligent Vehicles Symposium, Baden- Baden, Germany, June 2011.","title":"Stereoscan: Dense 3d reconstruction in real-time","authors":"Andreas Geiger, Julius Ziegler, Christoph Stiller","year":"2011","venue":"IEEE Intelligent Vehicles Symposium","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Geiger et al., 2013] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. The International Journal of Robotics Re- search, 32(11):1231-1237, 2013.","title":"Vision meets robotics: The kitti dataset","authors":"Andreas Geiger, Philip Lenz, Christoph Stiller, Raquel Urtasun","year":"2013","venue":"The International Journal of Robotics Research","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Godard et al., 2017] Clément Godard, Oisin Mac Aodha, and Gabriel J Brostow. Unsupervised monocular depth estimation with left-right consistency. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6602-6611. IEEE, 2017.","title":"Unsupervised monocular depth estimation with left-right consistency","authors":"Clément Godard, Oisin Mac Aodha, Gabriel J Brostow","year":"2017","venue":"IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Goodfellow et al., 2014] Ian Goodfellow, Jean Pouget- Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Gen- erative adversarial nets. In Advances in neural information processing systems, pages 2672-2680, 2014.","title":"Generative adversarial nets","authors":"Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio","year":"2014","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Handa et al., 2014] Ankur Handa, Thomas Whelan, John McDonald, and Andrew J Davison. A benchmark for rgb- d visual odometry, 3d reconstruction and slam. In Robotics and automation (ICRA), 2014 IEEE international confer- ence on, pages 1524-1531. IEEE, 2014.","title":"A benchmark for rgbd visual odometry, 3d reconstruction and slam","authors":"Ankur Handa, Thomas Whelan, John Mcdonald, Andrew J Davison","year":"2014","venue":"Robotics and automation (ICRA), 2014 IEEE international conference on","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Isola et al., 2017] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with con- ditional adversarial networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5967-5976. IEEE, 2017.","title":"Image-to-image translation with conditional adversarial networks","authors":"Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, Alexei A Efros","year":"2017","venue":"IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Jaderberg et al., 2015] Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al. Spatial transformer networks. In Advances in neural information processing systems, pages 2017-2025, 2015.","title":"Spatial transformer networks","authors":"Jaderberg","year":"2015","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kingma and Ba, 2014] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. Proceed- ings of the 3rd International Conference on Learning Rep- resentations (ICLR), 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":"Proceedings of the 3rd International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kumar et al., 2018] Aran CS Kumar, Suchendra M Bhan- darkar, and Mukta Prasad. Monocular depth prediction using generative adversarial networks. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 413-4138. IEEE, 2018.","title":"Monocular depth prediction using generative adversarial networks","authors":"Kumar","year":"2018","venue":"IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kuznietsov et al., 2017] Yevhen Kuznietsov, Jörg Stückler, and Bastian Leibe. Semi-supervised deep learning for monocular depth map prediction. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6647-6655, 2017.","title":"Semi-supervised deep learning for monocular depth map prediction","authors":"Kuznietsov","year":"2017","venue":"Proc. of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2018] Ruihao Li, Sen Wang, Zhiqiang Long, and Dongbing Gu. Undeepvo: Monocular visual odometry through unsupervised deep learning. In 2018 IEEE Inter- national Conference on Robotics and Automation (ICRA), pages 7286-7291. IEEE, 2018.","title":"Undeepvo: Monocular visual odometry through unsupervised deep learning","authors":"Ruihao Li, Sen Wang, Zhiqiang Long, Dongbing Gu","year":"2018","venue":"IEEE International Conference on Robotics and Automation (ICRA)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liu et al., 2016] Fayao Liu, Chunhua Shen, Guosheng Lin, and Ian Reid. Learning depth from single monocular images using deep convolutional neural fields. IEEE transactions on pattern analysis and machine intelligence, 38(10):2024-2039, 2016.","title":"Learning depth from single monocular images using deep convolutional neural fields","authors":"Liu","year":"2016","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Luo et al., 2018] Yue Luo, Jimmy Ren, Mude Lin, Jiahao Pang, Wenxiu Sun, Hongsheng Li, and Liang Lin. Single view stereo matching. In Proceedings of the IEEE Confer- ence on Computer Vision and Pattern Recognition, pages 155-163, 2018.","title":"Single view stereo matching","authors":"Yue Luo, Jimmy Ren, Mude Lin, Jiahao Pang, Wenxiu Sun, Hongsheng Li, Liang Lin","year":"2018","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Mahjourian et al., 2018] Reza Mahjourian, Martin Wicke, and Anelia Angelova. Unsupervised learning of depth and ego-motion from monocular video using 3d geomet- ric constraints. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5667- 5675, 2018.","title":"Unsupervised learning of depth and ego-motion from monocular video using 3d geometric constraints","authors":"Reza Mahjourian, Martin Wicke, Anelia Angelova","year":"2018","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Marchand et al., 2016] Eric Marchand, Hideaki Uchiyama, and Fabien Spindler. Pose estimation for augmented real- ity: a hands-on survey. IEEE transactions on visualization and computer graphics, 22(12):2633-2651, 2016.","title":"Pose estimation for augmented reality: a hands-on survey","authors":"Eric Marchand, Hideaki Uchiyama, Fabien Spindler","year":"2016","venue":"IEEE transactions on visualization and computer graphics","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Mirza and Osindero, 2014] Mehdi Mirza and Simon Osin- dero. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784, 2014.","title":"Conditional generative adversarial nets","authors":"Mehdi Mirza, Simon Osindero","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Pilzer et al., 2018] Andrea Pilzer, Dan Xu, Mihai Puscas, Elisa Ricci, and Nicu Sebe. Unsupervised adversarial depth estimation using cycled generative networks. In 2018 International Conference on 3D Vision (3DV), pages 587-595. IEEE, 2018.","title":"Unsupervised adversarial depth estimation using cycled generative networks","authors":"Andrea Pilzer, Dan Xu, Mihai Puscas, Elisa Ricci, Nicu Sebe","year":"2018","venue":"International Conference on 3D Vision (3DV)","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Reed et al., 2016] Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. Generative adversarial text to image synthesis. Proceed- ings of The 33rd International Conference on Machine Learning, 2016.","title":"Generative adversarial text to image synthesis","authors":"Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, Honglak Lee","year":"2016","venue":"Proceedings of The 33rd International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Xu et al., 2015] Bing Xu, Naiyan Wang, Tianqi Chen, and Mu Li. Empirical evaluation of rectified activations in convolutional network. arXiv preprint arXiv:1505.00853, 2015.","title":"Empirical evaluation of rectified activations in convolutional network","authors":"Bing Xu, Naiyan Wang, Tianqi Chen, Mu Li","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhan et al., 2018] Huangying Zhan, Ravi Garg, Chamara Saroj Weerasekera, Kejie Li, Harsh Agar- wal, and Ian Reid. Unsupervised learning of monocular depth estimation and visual odometry with deep feature reconstruction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 340-349, 2018.","title":"Unsupervised learning of monocular depth estimation and visual odometry with deep feature reconstruction","authors":"Huangying Zhan, Ravi Garg, Chamara Saroj Weerasekera, Kejie Li, Harsh Agarwal, Ian Reid","year":"2018","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhou et al., 2017] Tinghui Zhou, Matthew Brown, Noah Snavely, and David G. Lowe. Unsupervised learning of depth and ego-motion from video. In CVPR, 2017.","title":"Unsupervised learning of depth and ego-motion from video","authors":"Zhou","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhu et al., 2017] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Computer Vision (ICCV), 2017 IEEE International Conference on, 2017.","title":"Unpaired image-to-image translation using cycle-consistent adversarial networks","authors":"Zhu","year":"2017","venue":"Computer Vision (ICCV), 2017 IEEE International Conference on","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}