{"type":"ScholarlyArticle","title":"Learning to Estimate Object Poses without Real Image Annotations","authors":[{"name":"Haotong Lin","slug":"haotong-lin","given":"Haotong","family":"Lin","affiliation":"State Key Lab of CAD and CG , Zhejiang University"},{"name":"Sida Peng","slug":"sida-peng","given":"Sida","family":"Peng","affiliation":"State Key Lab of CAD and CG , Zhejiang University"},{"name":"Zhize Zhou","slug":"zhize-zhou","given":"Zhize","family":"Zhou","affiliation":"State Key Lab of CAD and CG , Zhejiang University"},{"name":"Xiaowei Zhou","slug":"xiaowei-zhou","given":"Xiaowei","family":"Zhou","affiliation":"State Key Lab of CAD and CG , Zhejiang University"}],"abstract":"This paper presents a simple yet effective approach for learning 6DoF object poses without real image annotations. Previous methods have attempted to train pose estimators on synthetic data, but they do not generalize well to real images due to the sim-to-real domain gap and produce inaccurate pose estimates. We find that, in most cases, the synthetically trained pose estimators are able to provide reasonable initialization for depth-based pose refinement methods which yield accurate pose estimates. Motivated by this, we propose a novel learning framework, which utilizes the accurate results of depth-based pose refinement methods to supervise the RGB-based pose estimator. Our method significantly outperforms previous self-supervised methods on several benchmarks. Even compared with fully-supervised methods that use real annotated data, we achieve competitive results without using any real annotation. The code is available at https://github.com/zju3dv/pvnet-depth-sup.","doi":"10.24963/ijcai.2022/162","url":"https://ijcai.org/proceedings/2022/162","pdf":"https://ijcai.org/proceedings/2022/0162.pdf","markdown":"https://ijcai.org/proceedings/2022/162.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/162","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":162,"pages":{"first":1159,"last":1165},"keywords":["Computer Vision: 3D Computer Vision","Computer Vision: Recognition (object detection, categorization)"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986555","length_sec":86,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984698","length_sec":279,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Brachmann et al., 2014] Eric Brachmann, Alexander Krull, Frank Michel, Stefan Gumhold, Jamie Shotton, and Carsten Rother. Learning 6d object pose estimation using 3d object coordinates. In ECCV, pages 536-551, 2014.","title":"Learning 6d object pose estimation using 3d object coordinates","authors":"Brachmann","year":"2014","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2019] Wenzheng Chen, Huan Ling, Jun Gao, Edward J. Smith, Jaakko Lehtinen, Alec Jacobson, and Sanja Fidler. Learning to predict 3d objects with an interpolation-based differentiable renderer. In NeurIPS, volume 32, 2019.","title":"Learning to predict 3d objects with an interpolation-based differentiable renderer","authors":"Chen","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Deng et al., 2020] Xinke Deng, Yu Xiang, Arsalan Mousa- vian, Clemens Eppner, Timothy Bretl, and Dieter Fox. Self-supervised 6d object pose estimation for robot ma- nipulation. In ICRA, pages 3665-3671, 2020.","title":"Self-supervised 6d object pose estimation for robot manipulation","authors":"Deng","year":"2020","venue":"ICRA","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[He et al., 2022] Yisheng He, Haoqiang Fan, Haibin Huang, Qifeng Chen, and Jian Sun. Towards self-supervised category-level object pose and size estimation. arXiv preprint arXiv:2203.02884, 2022.","title":"Towards self-supervised category-level object pose and size estimation","authors":"Yisheng He, Haoqiang Fan, Haibin Huang, Qifeng Chen, Jian Sun","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Hinterstoisser et al., 2012] Stefan Hinterstoisser, Vincent Lepetit, Slobodan Ilic, Stefan Holzer, Gary Bradski, Kurt Konolige, and Nassir Navab. Model based training, de- tection and pose estimation of texture-less 3d objects in heavily cluttered scenes. In ACCV, pages 548-562, 2012.","title":"Model based training, detection and pose estimation of texture-less 3d objects in heavily cluttered scenes","authors":"Stefan Hinterstoisser, Vincent Lepetit, Slobodan Ilic, Stefan Holzer, Gary Bradski, Kurt Konolige, Nassir Navab","year":"2012","venue":"ACCV","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hodaň et al., 2019] Tomáš Hodaň, Vibhav Vineet, Ran Gal, Emanuel Shalev, Jon Hanzelka, Treb Connell, Pedro Urbina, Sudipta N Sinha, and Brian Guenter. Photorealis- tic image synthesis for object instance detection. In ICIP, pages 66-70, 2019.","title":"Photorealistic image synthesis for object instance detection","authors":"Tomáš Hodaň, Vibhav Vineet, Ran Gal, Emanuel Shalev, Jon Hanzelka, Treb Connell, Pedro Urbina, Sudipta N Sinha, Brian Guenter","year":"2019","venue":"ICIP","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hu et al., 2020] Yinlin Hu, Pascal Fua, Wei Wang, and Mathieu Salzmann. Single-stage 6d object pose estima- tion. In CVPR, pages 2930-2939, 2020.","title":"Single-stage 6d object pose estimation","authors":"Hu","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hu et al., 2021] Yinlin Hu, Sebastien Speierer, Wenzel Jakob, Pascal Fua, and Mathieu Salzmann. Wide-depth- range 6d object pose estimation in space. In CVPR, pages 15870-15879, 2021.","title":"Wide-depthrange 6d object pose estimation in space","authors":"Hu","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Li et al., 2019] Zhigang Li, Gu Wang, and Xiangyang Ji. CDPN: coordinates-based disentangled pose network for real-time rgb-based 6-dof object pose estimation. In ICCV, pages 7678-7687, 2019.","title":"CDPN: coordinates-based disentangled pose network for real-time rgb-based 6-dof object pose estimation","authors":"Zhigang Li, Gu Wang, Xiangyang Ji","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Manhardt et al., 2019] Fabian Manhardt, Diego Martín Ar- royo, Christian Rupprecht, Benjamin Busam, Tolga Birdal, Nassir Navab, and Federico Tombari. Explaining the ambiguity of object detection and 6d pose from visual data. In ICCV, pages 6841-6850, 2019.","title":"Explaining the ambiguity of object detection and 6d pose from visual data","authors":"Fabian Manhardt, Diego Martín Arroyo, Christian Rupprecht, Benjamin Busam, Tolga Birdal, Nassir Navab, Federico Tombari","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Park et al., 2019] Kiru Park, Timothy Patten, and Markus Vincze. Pix2pose: Pixel-wise coordinate regression of ob- jects for 6d pose estimation. In ICCV, pages 7668-7677, 2019.","title":"Pix2pose: Pixel-wise coordinate regression of objects for 6d pose estimation","authors":"Kiru Park, Timothy Patten, Markus Vincze","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Peng et al., 2019] Sida Peng, Yuan Liu, Qixing Huang, Xi- aowei Zhou, and Hujun Bao. Pvnet: Pixel-wise voting network for 6dof pose estimation. In CVPR, pages 4561- 4570, 2019.","title":"Pvnet: Pixel-wise voting network for 6dof pose estimation","authors":"Peng","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Rad and Lepetit, 2017] Mahdi Rad and Vincent Lepetit. BB8: A scalable, accurate, robust to partial occlusion method for predicting the 3d poses of challenging objects without using depth. In ICCV, pages 3828-3836, 2017.","title":"Mahdi Rad and Vincent Lepetit. BB8: A scalable, accurate, robust to partial occlusion method for predicting the 3d poses of challenging objects without using depth","authors":"Rad, Lepetit","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Sock et al., 2020] Juil Sock, Guillermo Garcia-Hernando, Anil Armagan, and Tae-Kyun Kim. Introducing pose con- sistency and warp-alignment for self-supervised 6d object pose estimation in color images. In 3DV, pages 291-300, 2020.","title":"Introducing pose consistency and warp-alignment for self-supervised 6d object pose estimation in color images","authors":"Sock","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Song et al., 2020] Chen Song, Jiaru Song, and Qixing Huang. Hybridpose: 6d object pose estimation under hy- brid representations. In CVPR, pages 431-440, 2020.","title":"Hybridpose: 6d object pose estimation under hybrid representations","authors":"Chen Song, Jiaru Song, Qixing Huang","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Sundermeyer et al., 2018] Martin Sundermeyer, Zoltan- Csaba Marton, Maximilian Durner, Manuel Brucker, and Rudolph Triebel. Implicit 3d orientation learning for 6d object detection from rgb images. In ECCV, pages 699- 715, 2018.","title":"Implicit 3d orientation learning for 6d object detection from rgb images","authors":"Martin Sundermeyer, Zoltan-Csaba Marton, Maximilian Durner, Manuel Brucker, Rudolph Triebel","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Wang et al., 2019] He Wang, Srinath Sridhar, Jingwei Huang, Julien Valentin, Shuran Song, and Leonidas J Guibas. Normalized object coordinate space for category- level 6d object pose and size estimation. In CVPR, pages 2642-2651, 2019.","title":"Normalized object coordinate space for categorylevel 6d object pose and size estimation","authors":"Wang","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Wang et al., 2020] Gu Wang, Fabian Manhardt, Jianzhun Shao, Xiangyang Ji, Nassir Navab, and Federico Tombari. Self6d: Self-supervised monocular 6d object pose estima- tion. In ECCV, pages 108-125, 2020.","title":"Self6d: Self-supervised monocular 6d object pose estimation","authors":"Gu Wang, Fabian Manhardt, Jianzhun Shao, Xiangyang Ji, Nassir Navab, Federico Tombari","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wang et al., 2021a] Gu Wang, Fabian Manhardt, Xingyu Liu, Xiangyang Ji, and Federico Tombari. Occlusion- aware self-supervised monocular 6d object pose estima- tion. TPAMI, 2021.","title":"Occlusionaware self-supervised monocular 6d object pose estimation","authors":"Wang","year":"2021","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wang et al., 2021b] Gu Wang, Fabian Manhardt, Federico Tombari, and Xiangyang Ji. Gdr-net: Geometry-guided direct regression network for monocular 6d object pose es- timation. In CVPR, pages 16611-16621, 2021.","title":"Gdr-net: Geometry-guided direct regression network for monocular 6d object pose estimation","authors":"Wang","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wen et al., 2021] Yilin Wen, Xiangyu Li, Hao Pan, Lei Yang, Zheng Wang, Taku Komura, and Wenping Wang. Disentangled implicit shape and pose learning for scal- able 6d pose estimation. arXiv preprint arXiv:2107.12549, 2021.","title":"Disentangled implicit shape and pose learning for scalable 6d pose estimation","authors":"Yilin Wen, Xiangyu Li, Hao Pan, Lei Yang, Zheng Wang, Taku Komura, Wenping Wang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Xiang et al., 2018] Yu Xiang, Tanner Schmidt, Venkatra- man Narayanan, and Dieter Fox. Posecnn: A convolu- tional neural network for 6d object pose estimation in clut- tered scenes. In RSS, 2018.","title":"Posecnn: A convolutional neural network for 6d object pose estimation in cluttered scenes","authors":"Yu Xiang, Tanner Schmidt, Venkatraman Narayanan, Dieter Fox","year":"2018","venue":"RSS","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zakharov et al., 2019] Sergey Zakharov, Ivan Shugurov, and Slobodan Ilic. DPOD: 6d pose object detector and refiner. In ICCV, pages 1941-1950, 2019.","title":"DPOD: 6d pose object detector and refiner","authors":"Sergey Zakharov, Ivan Shugurov, Slobodan Ilic","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhang, 1994] Zhengyou Zhang. Iterative point matching for registration of free-form curves and surfaces. IJCV, 1994.","title":"Iterative point matching for registration of free-form curves and surfaces","authors":"Zhengyou Zhang","year":"1994","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhou et al., 2019] Xingyi Zhou, Dequan Wang, and Philipp Krähenbühl. Objects as points. In arXiv preprint arXiv:1904.07850, 2019.","title":"Objects as points","authors":"Zhou","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}