{"type":"ScholarlyArticle","title":"View-Volume Network for Semantic Scene Completion from a Single Depth Image","authors":[{"name":"Yuxiao Guo","slug":"yuxiao-guo","given":"Yuxiao","family":"Guo","affiliation":"University of Electronic Science and Technology of China; Microsoft Research Asia"},{"name":"Xin Tong","slug":"xin-tong","given":"Xin","family":"Tong","affiliation":"Microsoft Research Asia"}],"abstract":"We introduce a View-Volume convolutional neural network (VVNet) for inferring the occupancy and semantic labels of a volumetric 3D scene from a single depth image. Our method extracts the detailed geometric features from the input depth image with a 2D view CNN and then projects the features into a 3D volume according to the input depth map via a projection layer. After that, we learn the 3D context information of the scene with a 3D volume CNN for computing the result volumetric occupancy and semantic labels. With combined 2D and 3D representations, the VVNet efficiently reduces the computational cost, enables feature extraction from multi-channel high resolution inputs, and thus significantly improve the result accuracy. We validate our method and demonstrate its efficiency and effectiveness on both synthetic SUNCG and real NYU dataset.","doi":"10.24963/ijcai.2018/101","url":"https://ijcai.org/proceedings/2018/101","pdf":"https://ijcai.org/proceedings/2018/0101.pdf","markdown":"https://ijcai.org/proceedings/2018/101.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/101","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Computer Vision","track":"Main track","number":101,"pages":{"first":726,"last":732},"keywords":["Computer Vision: Recognition: Detection, Categorization, Indexing, Matching, Retrieval, Semantic Interpretation","Computer Vision: 2D and 3D Computer Vision"],"videos":[],"references":[{"position":1,"raw":"[Atapour-Abarghouei and Breckon, 2017] A. Atapour-Abarghouei and T.P. Breckon. Depthcomp: Real-time depth image com- pletion based on prior semantic scene segmentation. In BMVC, pages 1-13, 2017.","title":"Depthcomp: Real-time depth image completion based on prior semantic scene segmentation","authors":"A Atapour-Abarghouei, T P Breckon","year":"2017","venue":"BMVC","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Choy et al., 2016] Christopher B Choy, Danfei Xu, JunYoung Gwak, Kevin Chen, and Silvio Savarese. 3d-r2n2: A unified ap- proach for single and multi-view 3d object reconstruction. In ECCV, pages 628-644, 2016.","title":"3d-r2n2: A unified approach for single and multi-view 3d object reconstruction","authors":"Danfei Christopher B Choy, Junyoung Xu, Kevin Gwak, Silvio Chen, Savarese","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Dai et al., 2017] Angela Dai, Daniel Ritchie, Martin Bokeloh, Scott Reed, Jürgen Sturm, and Matthias Nießner. Scancomplete: Large-scale scene completion and semantic segmentation for 3d scans. arXiv preprint arXiv:1712.10215, 2017.","title":"Scancomplete: Large-scale scene completion and semantic segmentation for 3d scans","authors":"Angela Dai, Daniel Ritchie, Martin Bokeloh, Scott Reed, Jürgen Sturm, Matthias Nießner","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Firman et al., 2016] Michael Firman, Oisin Mac Aodha, Simon Julier, and Gabriel J. Brostow. Structured prediction of unob- served voxels from a single depth image. In CVPR, pages 5431- 5440, 2016.","title":"Structured prediction of unobserved voxels from a single depth image","authors":"Oisin Michael Firman, Simon Mac Aodha, Gabriel J Julier, Brostow","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Geiger and Wang, 2015] Andreas Geiger and Chaohui Wang. Joint 3d object and layout inference from a single rgb-d image. In GCPR, pages 183-195, 2015.","title":"Joint 3d object and layout inference from a single rgb-d image","authors":"Wang ; Geiger, Andreas Geiger, Chaohui Wang","year":"2015","venue":"GCPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Girdhar et al., 2016] R. Girdhar, D.F. Fouhey, M. Rodriguez, and A. Gupta. Learning a predictable and generative vector represen- tation for objects. In ECCV, pages 484-499, 2016.","title":"Learning a predictable and generative vector representation for objects","authors":"R Girdhar, D F Fouhey, M Rodriguez, A Gupta","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Guedes et al., 2018] Andre Bernardes Soares Guedes, Te- ofilo Emidio de Campos, and Adrian Hilton. Semantic scene completion combining colour and depth: preliminary experiments. arXiv preprint arXiv:1802.04735, 2018.","title":"Semantic scene completion combining colour and depth: preliminary experiments","authors":"Andre Bernardes, Soares Guedes, Teofilo Emidio De Campos, Adrian Hilton","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Guo et al., 2015] Ruiqi Guo, Chuhang Zou, and Derek Hoiem. Predicting complete 3d models of indoor scenes. arXiv preprint arXiv:1504.02437, 2015.","title":"Predicting complete 3d models of indoor scenes","authors":"Ruiqi Guo, Chuhang Zou, Derek Hoiem","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gupta et al., 2013] Saurabh Gupta, Pablo Arbelaez, and Jitendra Malik. Perceptual organization and recognition of indoor scenes from RGB-D images. In CVPR. 2013.","title":"Perceptual organization and recognition of indoor scenes from RGB-D images","authors":"Saurabh Gupta, Pablo Arbelaez, Jitendra Malik","year":"2013","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gupta et al., 2014] Saurabh Gupta, Ross Girshick, Pablo Ar- beláez, and Jitendra Malik. Learning rich features from rgb-d images for object detection and segmentation. In ECCV, pages 345-360, 2014.","title":"Learning rich features from rgb-d images for object detection and segmentation","authors":"Saurabh Gupta, Ross Girshick, Pablo Arbeláez, Jitendra Malik","year":"2014","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Gupta et al., 2015] Saurabh Gupta, Pablo Andrés Arbeláez, Ross B. Girshick, and Jitendra Malik. Aligning 3D models to RGB-D images of cluttered scenes. In CVPR, pages 4731-4740, 2015.","title":"Aligning 3D models to RGB-D images of cluttered scenes","authors":"Saurabh Gupta, Pablo Andrés Arbeláez, Ross B Girshick, Jitendra Malik","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Häne et al., 2017] Christian Häne, Shubham Tulsiani, and Jitendra Malik. Hierarchical surface prediction for 3d object reconstruc- tion. In arXiv preprint arXiv:1704.00710. 2017.","title":"Hierarchical surface prediction for 3d object reconstruction","authors":"Häne","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Ioannidou et al., 2017] Anastasia Ioannidou, Elisavet Chatzilari, Spiros Nikolopoulos, and Ioannis Kompatsiaris. Deep learning advances in computer vision with 3d data: A survey. ACM Com- puting Surveys, 50(2):20:1-20:38, 2017.","title":"Deep learning advances in computer vision with 3d data: A survey","authors":"Anastasia Ioannidou, Elisavet Chatzilari, Spiros Nikolopoulos, Ioannis Kompatsiaris","year":"2017","venue":"ACM Computing Surveys","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Jimenez Rezende et al., 2016] Danilo Jimenez Rezende, S. M. Ali Eslami, Shakir Mohamed, Peter Battaglia, Max Jaderberg, and Nicolas Heess. Unsupervised learning of 3d structure from im- ages. In NIPS, pages 4996-5004. 2016.","title":"Unsupervised learning of 3d structure from images","authors":"Danilo Jimenez Rezende, S M Ali Eslami, Shakir Mohamed, Peter Battaglia, Max Jaderberg, Nicolas Heess","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kalogerakis et al., 2017] Evangelos Kalogerakis, Melinos Averkiou, Subhransu Maji, and Siddhartha Chaudhuri. 3D shape segmentation with projective convolutional networks. In CVPR, pages 3779-3788, 2017.","title":"3D shape segmentation with projective convolutional networks","authors":"Evangelos Kalogerakis, Melinos Averkiou, Subhransu Maji, Siddhartha Chaudhuri","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lin et al., 2013] Dahua Lin, Sanja Fidler, and Raquel Urtasun. Holistic scene understanding for 3d object detection with rgbd cameras. In ICCV, pages 1417-1424, 2013.","title":"Holistic scene understanding for 3d object detection with rgbd cameras","authors":"Lin","year":"2013","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liu et al., 2017] Fangyu Liu, Shuaipeng Li, Liqiang Zhang, Chenghu Zhou, Rongtian Ye, Yuebin Wang, and Jiwen Lu. 3dcnn-dqn-rnn: A deep reinforcement learning framework for semantic parsing of large-scale 3d point clouds. In ICCV, pages 5678-5687, 2017.","title":"3dcnn-dqn-rnn: A deep reinforcement learning framework for semantic parsing of large-scale 3d point clouds","authors":"Fangyu Liu, Shuaipeng Li, Liqiang Zhang, Chenghu Zhou, Rongtian Ye, Yuebin Wang, Jiwen Lu","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Nguyen et al., 2016] D. T. Nguyen, B. S. Hua, M. K. Tran, Q. H. Pham, and S. K. Yeung. A field model for repairing 3d shapes. In CVPR, pages 5676-5684, 2016.","title":"A field model for repairing 3d shapes","authors":"D T Nguyen, B S Hua, M K Tran, Q H Pham, S K Yeung","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Qi et al., 2016] Charles Ruizhongtai Qi, Hao Su, Kaichun Mo, and Leonidas J. Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In CVPR, pages 652-660, 2016.","title":"Pointnet: Deep learning on point sets for 3d classification and segmentation","authors":"Charles Ruizhongtai, Qi, Hao Su, Kaichun Mo, Leonidas J Guibas","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Qi et al., 2017] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J. Guibas. Pointnet++: Deep hierarchical feature learn- ing on point sets in a metric space. In NIPS, pages 5105-5114, 2017.","title":"Pointnet++: Deep hierarchical feature learning on point sets in a metric space","authors":"Charles Ruizhongtai, Qi, Li Yi, Hao Su, Leonidas J Guibas","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Ren and Sudderth, 2016] Zhile Ren and Erik B. Sudderth. Three- dimensional object detection and layout prediction using clouds of oriented gradients. In CVPR, pages 1525-1533, 2016.","title":"Threedimensional object detection and layout prediction using clouds of oriented gradients","authors":"Sudderth ; ] Zhile Ren, Erik B Ren, Sudderth","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Rock et al., 2015] Jason Rock, Tanmay Gupta, Justin Thorsen, JunYoung Gwak, Daeyun Shin, and Derek Hoiem. Completing 3d object shape from one depth image. In CVPR, pages 2484- 2493, 2015.","title":"Completing 3d object shape from one depth image","authors":"Jason Rock, Tanmay Gupta, Justin Thorsen, Junyoung Gwak, Daeyun Shin, Derek Hoiem","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Silberman et al., 2012] Nathan Silberman, Derek Hoiem, Push- meet Kohli, and Rob Fergus. Indoor segmentation and support inference from rgbd images. In ECCV, pages 746-760, 2012.","title":"Indoor segmentation and support inference from rgbd images","authors":"Nathan Silberman, Derek Hoiem, Pushmeet Kohli, Rob Fergus","year":"2012","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Song and Xiao, 2016] Shuran Song and Jianxiong Xiao. Deep Sliding Shapes for amodal 3D object detection in RGB-D im- ages. In CVPR, pages 808-816, 2016.","title":"Shuran Song and Jianxiong Xiao. Deep Sliding Shapes for amodal 3D object detection in RGB-D images","authors":"Xiao Song","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Song et al., 2017] Shuran Song, Fisher Yu, Andy Zeng, Angel X Chang, Manolis Savva, and Thomas Funkhouser. Semantic scene completion from a single depth image. In CVPR, pages 1746- 1754, 2017.","title":"Semantic scene completion from a single depth image","authors":"Song","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Tulsiani et al., 2017] Shubham Tulsiani, Tinghui Zhou, Alexei A. Efros, and Jitendra Malik. Multi-view supervision for single- view reconstruction via differentiable ray consistency. In CVPR, pages 2626-2634, 2017.","title":"Multi-view supervision for singleview reconstruction via differentiable ray consistency","authors":"Shubham Tulsiani, Tinghui Zhou, Alexei A Efros, Jitendra Malik","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wang et al., 2017] Weiyue Wang, Qiangui Huang, Suya You, Chao Yang, and Ulrich Neumann. Shape inpainting using 3d generative adversarial network and recurrent convolutional net- works. In ICCV, pages 2298-2306, 2017.","title":"Shape inpainting using 3d generative adversarial network and recurrent convolutional networks","authors":"Wang","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wu et al., 2015] Zhirong Wu, Shuran Song, Aditya Khosla, Lin- guang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shape modeling. In CVPR, pages 1912-1920, 2015.","title":"3d shapenets: A deep representation for volumetric shape modeling","authors":"Zhirong Wu, Shuran Song, Aditya Khosla, Linguang Zhang, Xiaoou Tang, Jianxiong Xiao","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Yan et al., 2016] X. Yan, J. Yang, E. Yumer, Y. Guo, and H. Lee. Perspective transformer nets: Learning single-view 3d object re- construction without 3d supervision. In NIPS 2016, pages 1696- 1704, 2016.","title":"Perspective transformer nets: Learning single-view 3d object reconstruction without 3d supervision","authors":"Yan","year":"2016","venue":"NIPS 2016","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Yang et al., 2017] Bo Yang, Hongkai Wen, Sen Wang, Ronald Clark, Andrew Markham, and Niki Trignoi. 3d object recon- struction from a single depth view with adversarial learning. In ICCV, pages 679-688, 2017.","title":"3d object reconstruction from a single depth view with adversarial learning","authors":"Bo Yang, Hongkai Wen, Sen Wang, Ronald Clark, Andrew Markham, Niki Trignoi","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zheng et al., 2013] Bo Zheng, Yibiao Zhao, Joey C. Yu, Katsushi Ikeuchi, and Song-Chun Zhu. Beyond point clouds: Scene un- derstanding by reasoning geometry and physics. In CVPR, pages 3127-3134, 2013.","title":"Beyond point clouds: Scene understanding by reasoning geometry and physics","authors":"Bo Zheng, Yibiao Zhao, Joey C Yu, Katsushi Ikeuchi, Song-Chun Zhu","year":"2013","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}