{"type":"ScholarlyArticle","title":"Semi-supervised Three-dimensional Reconstruction Framework with GAN","authors":[{"name":"Chong Yu","slug":"chong-yu","given":"Chong","family":"Yu","affiliation":"NVIDIA Corporation"}],"abstract":"Because of the intrinsic complexity in computation, three-dimensional (3D) reconstruction is an essential and challenging topic in computer vision research and applications. The existing methods for 3D reconstruction often produce holes, distortions and obscure parts in the reconstructed 3D models, or can only reconstruct voxelized 3D models for simple isolated objects. So they are not adequate for real usage. From 2014, the Generative Adversarial Network (GAN) is widely used in generating unreal dataset and semi-supervised learning. So the focus of this paper is to achieve high quality 3D reconstruction performance by adopting GAN principle. We propose a novel semi-supervised 3D reconstruction framework, namely SS-3D-GAN, which can iteratively improve any raw 3D reconstruction models by training the GAN models to converge. This new model only takes real-time 2D observation images as the weak supervision, and doesn't rely on prior knowledge of shape models or any referenced observations. Finally, through the qualitative and quantitative experiments & analysis, this new method shows compelling advantages over the current state-of-the-art methods on Tanks & Temples reconstruction benchmark dataset.","doi":"10.24963/ijcai.2019/582","url":"https://ijcai.org/proceedings/2019/582","pdf":"https://ijcai.org/proceedings/2019/0582.pdf","markdown":"https://ijcai.org/proceedings/2019/582.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/582","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning M-Z","track":"Main track","number":582,"pages":{"first":4192,"last":4198},"keywords":["Machine Learning: Semi-Supervised Learning","Computer Vision: 2D and 3D Computer Vision"],"videos":[],"references":[{"position":1,"raw":"Algorithms Family Francis Horse Lighthouse M60 Panther Playground Train Auditorium Ballroom Courtroom Museum Palace Temple COLMAP 45.82 16.46 18.79 49.34 59.69 57.01 66.61 42.15 10.73 26.29 31.40 38.44 13.36 23.56 MVE 68.52 32.75 14.74 68.59 8.14 75.40 3.83 49.32 2.92 18.26 40.21 52.05 14.79 19.51 MVE + SMVS 30.47 15.62 7.82 41.06 45.20 52.71 1.34 20.86 0.51 4.96 14.13 21.03 5.84 5.80 OpenMVG + MVE 69.70 37.91 24.01 73.21 71.15 77.41 84.71 57.69 15.22 39.72 43.42 55.74 2.20 31.41 OpenMVG + PMVS 30.85 10.77 7.73 28.73 30.04 24.19 25.88 22.58 2.48 7.63 13.93 20.99 3.94 8.18 OpenMVG + SMVS 31.99 18.66 13.65 43.16 45.51 54.02 39.91 24.02 4.41 9.54 17.46 24.11 6.82 10.35 SS-3D-GAN 69.31 38.11 25.12 72.89 69.97 77.60 83.55 55.72 15.47 37.66 43.59 54.83 14.74 32.28 VisualSfM + PMVS 28.02 7.77 6.73 27.83 34.36 25.07 28.86 8.25 2.49 6.63 10.20 13.30 4.15 1.13 Table 2: Recall (%) for Tanks and Temple Dataset Algorithms Family Francis Horse Lighthouse M60 Panther Playground Train Auditorium Ballroom Courtroom Museum Palace Temple COLMAP 50.41 22.26 25.64 44.83 56.43 46.97 48.53 42.04 16.02 25.23 34.71 41.51 18.06 27.93 MVE 48.60 23.84 12.70 39.63 5.07 38.17 5.81 29.19 4.11 12.63 27.93 34.67 13.58 16.79 MVE + SMVS 30.41 16.64 10.44 34.35 39.16 37.90 2.40 21.44 0.96 6.76 16.97 19.72 7.73 7.98 OpenMVG + MVE 49.92 28.19 20.75 44.51 43.35 44.76 36.57 35.95 14.70 26.36 32.48 37.57 3.65 22.84 OpenMVG + PMVS 41.04 17.70 12.83 35.92 36.68 33.19 31.78 28.10 4.54 12.09 21.01 29.17 6.76 12.72 OpenMVG + SMVS 31.93 19.92 15.02 36.51 39.38 41.60 35.89 25.12 6.96 11.58 19.82 21.89 8.90 12.27 SS-3D-GAN 67.94 42.87 31.48 59.36 60.54 62.91 59.58 47.38 21.99 32.82 43.31 51.32 19.89 32.85 VisualSfM + PMVS 38.02 12.94 11.30 35.48 41.75 34.19 35.47 13.25 4.67 10.84 16.36 20.01 7.32 2.13 Choy et al., 2016]","title":"Algorithms Family Francis Horse Lighthouse M60 Panther Playground Train Auditorium Ballroom","authors":null,"year":null,"venue":"and Temple Dataset Algorithms Family Francis Horse Lighthouse M60 Panther Playground Train Auditorium Ballroom Courtroom Museum Palace Temple COLMAP","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"Christopher B Choy, Danfei Xu, JunY- oung Gwak, Kevin Chen, and Silvio Savarese. 3d-r2n2: A unified approach for single and multi-view 3d objec- t reconstruction. In European Conference on Computer Vision, pages 628-644. Springer, 2016.","title":"3d-r2n2: A unified approach for single and multi-view 3d object reconstruction","authors":"Danfei Christopher B Choy, Junyoung Xu, Kevin Gwak, Silvio Chen, Savarese","year":"2016","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Dai et al., 2017] Angela Dai, Matthias Nießner, Michael Zollhöfer, Shahram Izadi, and Christian Theobalt. Bundle- fusion: Real-time globally consistent 3d reconstruction us- ing on-the-fly surface reintegration. ACM Transactions on Graphics (TOG), 36(4):76a, 2017.","title":"Bundlefusion: Real-time globally consistent 3d reconstruction using on-the-fly surface reintegration","authors":"Angela Dai, Matthias Nießner, Michael Zollhöfer, Shahram Izadi, Christian Theobalt","year":"2017","venue":"ACM Transactions on Graphics (TOG)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dominguez et al., 2017] Miguel Dominguez, Felipe Petros- ki Such, Shagan Sah, and Raymond Ptucha. Towards 3d convolutional neural networks with meshes. In Image Pro- cessing (ICIP), 2017 IEEE International Conference on, pages 3929-3933. IEEE, 2017.","title":"Miguel Dominguez, Felipe Petroski Such, Shagan Sah, and Raymond Ptucha. Towards 3d convolutional neural networks with meshes","authors":"Dominguez","year":"2017","venue":"Image Processing (ICIP), 2017 IEEE International Conference on","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Fuhrmann et al., 2014] Simon Fuhrmann, Fabian Langguth, and Michael Goesele. Mve-a multi-view reconstruction environment. In GCH, pages 11-18, 2014.","title":"Mve-a multi-view reconstruction environment","authors":"Simon Fuhrmann, Fabian Langguth, Michael Goesele","year":"2014","venue":"GCH","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Furukawa and Ponce, 2010] Yasutaka Furukawa and Jean Ponce. Accurate, dense, and robust multiview stereopsis. IEEE transactions on pattern analysis and machine intel- ligence, 32(8):1362-1376, 2010.","title":"Accurate, dense, and robust multiview stereopsis","authors":"Ponce Furukawa, Yasutaka Furukawa, Jean Ponce","year":"2010","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Galliani et al., 2015] Silvano Galliani, Katrin Lasinger, and Konrad Schindler. Massively parallel multiview stereopsis by surface normal diffusion. In Proceedings of the IEEE International Conference on Computer Vision, pages 873- 881, 2015.","title":"Massively parallel multiview stereopsis by surface normal diffusion","authors":"Silvano Galliani, Katrin Lasinger, Konrad Schindler","year":"2015","venue":"Proceedings of the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Guizilini and Ramos, 2016] Vitor Guizilini and Fabio Ramos. Large-scale 3d scene reconstruction with hilbert maps. In Intelligent Robots and Systems (IROS), 2016 IEEE/RSJ International Conference on, pages 3247-3254. IEEE, 2016.","title":"Vitor Guizilini and Fabio Ramos. Large-scale 3d scene reconstruction with hilbert maps","authors":"Ramos Guizilini","year":"2016","venue":"Intelligent Robots and Systems (IROS), 2016 IEEE/RSJ International Conference on","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Johnson-Roberson et al., 2017] Matthew Johnson- Roberson, Mitch Bryson, Ariell Friedman, Oscar Pizarro, Giancarlo Troni, Paul Ozog, and Jon C Henderson. High-resolution underwater robotic vision-based mapping and three-dimensional reconstruction for archaeology. Journal of Field Robotics, 34(4):625-643, 2017.","title":"High-resolution underwater robotic vision-based mapping and three-dimensional reconstruction for archaeology","authors":"Matthew Johnson-Roberson, Mitch Bryson, Ariell Friedman, Oscar Pizarro, Giancarlo Troni, Paul Ozog, Jon C Henderson","year":"2017","venue":"Journal of Field Robotics","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Knapitsch et al., 2017] Arno Knapitsch, Jaesik Park, Qian- Yi Zhou, and Vladlen Koltun. Tanks and temples: Bench- marking large-scale scene reconstruction. ACM Transac- tions on Graphics (ToG), 36(4):78, 2017.","title":"Tanks and temples: Benchmarking large-scale scene reconstruction","authors":"Knapitsch","year":"2017","venue":"ACM Transactions on Graphics (ToG)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Langguth et al., 2016] Fabian Langguth, Kalyan Sunkaval- li, Sunil Hadap, and Michael Goesele. Shading-aware multi-view stereo. In European Conference on Comput- er Vision, pages 469-485. Springer, 2016.","title":"Shading-aware multi-view stereo","authors":"Fabian Langguth, Kalyan Sunkavalli, Sunil Hadap, Michael Goesele","year":"2016","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Loper and Black, 2014] Matthew M Loper and Michael J Black. Opendr: An approximate differentiable renderer. In European Conference on Computer Vision, pages 154- 169. Springer, 2014. [Moulon et al., 2016] Pierre Moulon, Pascal Monasse, Ro- muald Perrot, and Renaud Marlet. Openmvg: Open mul- tiple view geometry. In International Workshop on Re- producible Research in Pattern Recognition, pages 60-74. Springer, 2016.","title":"Opendr: An approximate differentiable renderer","authors":"M Matthew, Michael J Loper, Black","year":"2014","venue":"International Workshop on Reproducible Research in Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Newcombe et al., 2011] Richard A Newcombe, Shahram Izadi, Otmar Hilliges, David Molyneaux, David Kim, An- drew J Davison, Pushmeet Kohi, Jamie Shotton, Steve Hodges, and Andrew Fitzgibbon. Kinectfusion: Real- time dense surface mapping and tracking. In Mixed and augmented reality (ISMAR), 2011 10th IEEE internation- al symposium on, pages 127-136. IEEE, 2011.","title":"Kinectfusion: Realtime dense surface mapping and tracking","authors":"Shahram Richard A Newcombe, Otmar Izadi, David Hilliges, David Molyneaux, Andrew J Kim, Pushmeet Davison, Jamie Kohi, Steve Shotton, Andrew Hodges, Fitzgibbon","year":"2011","venue":"10th IEEE international symposium on","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Nex and Remondino, 2014] Francesco Nex and Fabio Re- mondino. Uav for 3d mapping applications: a review. Ap- plied geomatics, 6(1):1-15, 2014.","title":"Francesco Nex and Fabio Remondino. Uav for 3d mapping applications: a review","authors":"Remondino Nex","year":"2014","venue":"Applied geomatics","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Pillai et al., 2016] Sudeep Pillai, Srikumar Ramalingam, and John J Leonard. High-performance and tunable stere- o reconstruction. In Robotics and Automation (ICRA), 2016 IEEE International Conference on, pages 3188- 3195. IEEE, 2016.","title":"High-performance and tunable stereo reconstruction","authors":"Sudeep Pillai, Srikumar Ramalingam, John J Leonard","year":"2016","venue":"Robotics and Automation (ICRA), 2016 IEEE International Conference on","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Qi et al., 2016] Charles R Qi, Hao Su, Matthias Nießner, Angela Dai, Mengyuan Yan, and Leonidas J Guibas. Vol- umetric and multi-view cnns for object classification on 3d data. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5648-5656, 2016.","title":"Volumetric and multi-view cnns for object classification on 3d data","authors":"Hao Charles R Qi, Matthias Su, Angela Nießner, Mengyuan Dai, Leonidas J Yan, Guibas","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Schönberger and Frahm, 2016] Johannes Lutz Schönberger and Jan-Michael Frahm. Structure-from-motion revisited. In Conference on Computer Vision and Pattern Recogni- tion (CVPR), 2016.","title":"Structure-from-motion revisited","authors":"Frahm Schönberger, Johannes Lutz Schönberger, Jan-Michael Frahm","year":"2016","venue":"Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sra et al., 2016] Misha Sra, Sergio Garrido-Jurado, Chris Schmandt, and Pattie Maes. Procedurally generated vir- tual reality from 3d reconstructed physical space. In Pro- ceedings of the 22nd ACM Conference on Virtual Reality Software and Technology, pages 191-200. ACM, 2016.","title":"Procedurally generated virtual reality from 3d reconstructed physical space","authors":"Misha Sra, Sergio Garrido-Jurado, Chris Schmandt, Pattie Maes","year":"2016","venue":"Proceedings of the 22nd ACM Conference on Virtual Reality Software and Technology","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wang et al., 2004] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transac- tions on image processing, 13(4):600-612, 2004.","title":"Image quality assessment: from error visibility to structural similarity","authors":"Zhou Wang, Alan C Bovik, Hamid R Sheikh, Eero P Simoncelli","year":"2004","venue":"IEEE transactions on image processing","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Whelan et al., 2016] Thomas Whelan, Renato F Salas- Moreno, Ben Glocker, Andrew J Davison, and Stefan Leutenegger. Elasticfusion: Real-time dense slam and light source estimation. The International Journal of Robotics Research, 35(14):1697-1716, 2016.","title":"Elasticfusion: Real-time dense slam and light source estimation","authors":"Thomas Whelan, Renato F Salas-Moreno, Ben Glocker, Andrew J Davison, Stefan Leutenegger","year":"2016","venue":"The International Journal of Robotics Research","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wu and others, 2011] Changchang Wu et al. Visualsfm: A visual structure from motion system. 2011.","title":"Visualsfm: A visual structure from motion system","authors":"Changchang Wu, Wu","year":"2011","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wu et al., 2016] Jiajun Wu, Chengkai Zhang, Tianfan Xue, Bill Freeman, and Josh Tenenbaum. Learning a proba- bilistic latent space of object shapes via 3d generative- adversarial modeling. In Advances in Neural Information Processing Systems, pages 82-90, 2016.","title":"Learning a probabilistic latent space of object shapes via 3d generativeadversarial modeling","authors":"Jiajun Wu, Chengkai Zhang, Tianfan Xue, Bill Freeman, Josh Tenenbaum","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yu, 2016] Chong Yu. Steganography of digital watermark based on artificial neural networks in image communica- tion and intellectual property protection. Neural Process- ing Letters, 44(2):307-316, 2016.","title":"Steganography of digital watermark based on artificial neural networks in image communication and intellectual property protection","authors":"Chong Yu","year":"2016","venue":"Neural Processing Letters","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}