{"type":"ScholarlyArticle","title":"Unsupervised Learning of Scene Flow Estimation Fusing with Local Rigidity","authors":[{"name":"Liang Liu","slug":"liang-liu","given":"Liang","family":"Liu","affiliation":"Institute of Cyber-Systems and Control , Zhejiang University"},{"name":"Guangyao Zhai","slug":"guangyao-zhai","given":"Guangyao","family":"Zhai","affiliation":"Institute of Cyber-Systems and Control , Zhejiang University"},{"name":"Wenlong Ye","slug":"wenlong-ye","given":"Wenlong","family":"Ye","affiliation":"Institute of Cyber-Systems and Control , Zhejiang University"},{"name":"Yong Liu","slug":"yong-liu","given":"Yong","family":"Liu","affiliation":"Institute of Cyber-Systems and Control , Zhejiang University"}],"abstract":"Scene flow estimation in the dynamic scene remains a challenging task. Computing scene flow by a combination of 2D optical flow and depth has shown to be considerably faster with acceptable performance. In this work, we present a unified framework for joint unsupervised learning of stereo depth and optical flow with explicit local rigidity to estimate scene flow. We estimate camera motion directly by a Perspective-n-Point method from the optical flow and depth predictions, with RANSAC outlier rejection scheme. In order to disambiguate the object motion and the camera motion in the scene, we distinguish the rigid region by the re-project error and the photometric similarity. By joint learning with the local rigidity, both depth and optical networks can be refined. This framework boosts all four tasks: depth, optical flow, camera motion estimation, and object motion segmentation. Through the evaluation on the KITTI benchmark, we show that the proposed framework achieves state-of-the-art results amongst unsupervised methods. Our models and code are available at https://github.com/lliuz/unrigidflow.","doi":"10.24963/ijcai.2019/123","url":"https://ijcai.org/proceedings/2019/123","pdf":"https://ijcai.org/proceedings/2019/0123.pdf","markdown":"https://ijcai.org/proceedings/2019/123.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/123","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Computer Vision","track":"Main track","number":123,"pages":{"first":876,"last":882},"keywords":["Computer Vision: Computer Vision","Computer Vision: 2D and 3D Computer Vision"],"videos":[],"references":[{"position":1,"raw":"[Basha et al., 2013] Tali Basha, Yael Moses, and Nahum Kiryati. Multi-view scene flow estimation: A view cen- tered variational approach. IJCV, 101(1):6-21, 2013.","title":"Multi-view scene flow estimation: A view centered variational approach","authors":"Basha","year":"2013","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chang and Chen, 2018] Jia-Ren Chang and Yong-Sheng Chen. Pyramid stereo matching network. In CVPR, 2018.","title":"Pyramid stereo matching network","authors":"Jia-Ren Chang, Yong-Sheng Chen","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Eigen et al., 2014] David Eigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from a single image using a multi-scale deep network. In NIPS, 2014.","title":"Depth map prediction from a single image using a multi-scale deep network","authors":"David Eigen, Christian Puhrsch, Rob Fergus","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Fischler and Bolles, 1981] Martin A. Fischler and Robert C. Bolles. Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM, 24(6):381-395, 1981.","title":"Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography","authors":"Martin A Fischler, Robert C Bolles","year":"1981","venue":"Commun. ACM","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Geiger et al., 2012] Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In CVPR, 2012.","title":"Are we ready for autonomous driving? the kitti vision benchmark suite","authors":"Andreas Geiger, Philip Lenz, Raquel Urtasun","year":"2012","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Geiger et al., 2013] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. IJRR, 2013.","title":"Vision meets robotics: The kitti dataset","authors":"Andreas Geiger, Philip Lenz, Christoph Stiller, Raquel Urtasun","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Godard et al., 2017] Clément Godard, Oisin Mac Aodha, and Gabriel J Brostow. Unsupervised monocular depth es- timation with left-right consistency. In CVPR, 2017.","title":"Unsupervised monocular depth estimation with left-right consistency","authors":"Clément Godard, Oisin Mac Aodha, Gabriel J Brostow","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Ilg et al., 2017] Eddy Ilg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, and Thomas Brox. Flownet 2.0: Evolution of optical flow estimation with deep networks. In CVPR, 2017.","title":"Flownet 2.0: Evolution of optical flow estimation with deep networks","authors":"Eddy Ilg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, Thomas Brox","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jaderberg et al., 2015] Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al. Spatial transformer networks. In NIPS, 2015.","title":"Spatial transformer networks","authors":"Jaderberg","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jaimez et al., 2017] Mariano Jaimez, Christian Kerl, Javier Gonzalez-Jimenez, and Daniel Cremers. Fast odometry and scene flow from rgb-d cameras based on geometric clustering. In ICRA, 2017.","title":"Fast odometry and scene flow from rgb-d cameras based on geometric clustering","authors":"Mariano Jaimez, Christian Kerl, Javier Gonzalez-Jimenez, Daniel Cremers","year":"2017","venue":"ICRA","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kingma and Ba, 2015] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lee and Fowlkes, 2018] Minhaeng Lee and Charless C Fowlkes. Cemnet: Self-supervised learning for accu- rate continuous ego-motion estimation. arXiv preprint arXiv:1806.10309, 2018.","title":"Cemnet: Self-supervised learning for accurate continuous ego-motion estimation","authors":"Lee, Fowlkes, Minhaeng Lee, Charless C Fowlkes","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lv et al., 2018] Zhaoyang Lv, Kihwan Kim, Alejandro Troccoli, Deqing Sun, James M Rehg, and Jan Kautz. Learning rigidity in dynamic scenes with a moving camera for 3d motion field estimation. In ECCV, pages 468-484, 2018.","title":"Learning rigidity in dynamic scenes with a moving camera for 3d motion field estimation","authors":"Lv","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Meister et al., 2018] Simon Meister, Junhwa Hur, and Ste- fan Roth. Unflow: Unsupervised learning of optical flow with a bidirectional census loss. In AAAI, 2018.","title":"Unflow: Unsupervised learning of optical flow with a bidirectional census loss","authors":"Meister","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Menze and Geiger, 2015] Moritz Menze and Andreas Geiger. Object scene flow for autonomous vehicles. In CVPR, 2015.","title":"Object scene flow for autonomous vehicles","authors":"Geiger Menze, Moritz Menze, Andreas Geiger","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Newcombe et al., 2015] Richard A Newcombe, Dieter Fox, and Steven M Seitz. Dynamicfusion: Reconstruction and tracking of non-rigid scenes in real-time. In CVPR, pages 343-352, 2015.","title":"Dynamicfusion: Reconstruction and tracking of non-rigid scenes in real-time","authors":"Dieter Richard A Newcombe, Steven M Fox, Seitz","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Paszke et al., 2017] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary De- Vito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NIPS-W, 2017.","title":"Automatic differentiation in pytorch","authors":"Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary De-Vito, Zeming Lin, Alban Desmaison, Luca Antiga, Adam Lerer","year":"2017","venue":"NIPS-W","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Quiroga et al., 2014] Julian Quiroga, Thomas Brox, Frédéric Devernay, and James Crowley. Dense semi-rigid scene flow estimation from rgbd images. In ECCV, 2014.","title":"Dense semi-rigid scene flow estimation from rgbd images","authors":"Julian Quiroga, Thomas Brox, Frédéric Devernay, James Crowley","year":"2014","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sun et al., 2018] Deqing Sun, Xiaodong Yang, Ming-Yu Liu, and Jan Kautz. Pwc-net: Cnns for optical flow us- ing pyramid, warping, and cost volume. In CVPR, pages 8934-8943, 2018.","title":"Pwc-net: Cnns for optical flow using pyramid, warping, and cost volume","authors":"Deqing Sun, Xiaodong Yang, Ming-Yu Liu, Jan Kautz","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Valgaerts et al., 2010] Levi Valgaerts, Andrés Bruhn, Hen- ning Zimmer, Joachim Weickert, Carsten Stoll, and Chris- tian Theobalt. Joint estimation of motion, structure and geometry from stereo sequences. In ECCV, 2010.","title":"Joint estimation of motion, structure and geometry from stereo sequences","authors":"Valgaerts","year":"2010","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Vedula et al., 1999] Sundar Vedula, Simon Baker, Peter Rander, Robert Collins, and Takeo Kanade. Three- dimensional scene flow. In ICCV, 1999.","title":"Threedimensional scene flow","authors":"Sundar Vedula, Simon Baker, Peter Rander, Robert Collins, Takeo Kanade","year":"1999","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang et al., 2018a] Sen Wang, Ronald Clark, Hongkai Wen, and Niki Trigoni. End-to-end, sequence-to-sequence probabilistic visual odometry through deep neural net- works. IJRR, 37(4-5):513-542, 2018.","title":"End-to-end, sequence-to-sequence probabilistic visual odometry through deep neural networks","authors":"Wang","year":"2018","venue":"IJRR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wang et al., 2018b] Yang Wang, Yi Yang, Zhenheng Yang, Liang Zhao, Peng Wang, and Wei Xu. Occlusion aware unsupervised learning of optical flow. In CVPR, 2018.","title":"Occlusion aware unsupervised learning of optical flow","authors":"Wang","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang et al., 2018c] Yang Wang, Zhenheng Yang, Peng Wang, Yi Yang, Chenxu Luo, and Wei Xu. Joint unsu- pervised learning of optical flow and depth by watching stereo videos. ECCV, 2018.","title":"Joint unsupervised learning of optical flow and depth by watching stereo videos","authors":"Wang","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wulff et al., 2017] Jonas Wulff, Laura Sevilla-Lara, and Michael J Black. Optical flow in mostly rigid scenes. In CVPR, 2017.","title":"Optical flow in mostly rigid scenes","authors":"Wulff","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yang et al., 2018] Zhenheng Yang, Peng Wang, Yang Wang, Wei Xu, and Ram Nevatia. Every pixel counts: Unsupervised geometry learning with holistic 3d motion understanding. In ECCV, 2018.","title":"Every pixel counts: Unsupervised geometry learning with holistic 3d motion understanding","authors":"Zhenheng Yang, Peng Wang, Yang Wang, Wei Xu, Ram Nevatia","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Yin and Shi, 2018] Zhichao Yin and Jianping Shi. Geonet: Unsupervised learning of dense depth, optical flow and camera pose. In CVPR, pages 1983-1992, 2018.","title":"Geonet: Unsupervised learning of dense depth, optical flow and camera pose","authors":"Zhichao Yin, Jianping Shi","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhou et al., 2017] Tinghui Zhou, Matthew Brown, Noah Snavely, and David G Lowe. Unsupervised learning of depth and ego-motion from video. In CVPR, 2017.","title":"Unsupervised learning of depth and ego-motion from video","authors":"Zhou","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}