{"type":"ScholarlyArticle","title":"Error-aware Sampling in Adaptive Shells for Neural Surface Reconstruction","authors":[{"name":"Qi Wang","slug":"qi-wang","given":"Qi","family":"Wang","affiliation":"State Key Lab of CAD&CG , Zhejiang University"},{"name":"Yuchi Huo","slug":"yuchi-huo","given":"Yuchi","family":"Huo","affiliation":"State Key Lab of CAD&CG , Zhejiang University; Zhejiang Lab"},{"name":"Qi Ye","slug":"qi-ye","given":"Qi","family":"Ye","affiliation":"State Key Lab of CAD&CG , Zhejiang University"},{"name":"Rui Wang","slug":"rui-wang","given":"Rui","family":"Wang","affiliation":"State Key Lab of CAD&CG , Zhejiang University"},{"name":"Hujun Bao","slug":"hujun-bao","given":"Hujun","family":"Bao","affiliation":"State Key Lab of CAD&CG , Zhejiang University"}],"abstract":"Neural implicit surfaces with signed distance functions (SDFs) achieve superior quality in 3D geometry reconstruction. However, training SDFs is time-consuming because it requires a great number of samples to calculate accurate weight distributions and a considerable amount of samples sampled from the distribution for integrating the rendering results. Some existing sampling strategies focus on this problem. During the training, they assume a spatially-consistent convergence speed of kernel size, thus still suffering from low convergence or errors. Instead, we introduce an error-aware sampling method based on thin intervals of valid weight distributions, dubbed adaptive shells, to reduce the number of samples while still maintaining the reconstruction accuracy. To this end, we first extend Laplace-based neural implicit surfaces with learned spatially-varying kernel sizes which indicates the range of valid weight distributions. Then, the adaptive shell for each ray is determined by an efficient double-clipping strategy with spatially-varying SDF values and kernel sizes, fitting larger kernel sizes to wider shells. Finally, we calculate the error-bounded cumulative distribution functions (CDFs) of shells to conduct efficient importance sampling, achieving low-variance rendering with fewer calculations. Extensive results in various scenes demonstrate the superiority of our sampling technique, including significantly reducing sample counts and training time, even improving the reconstruction quality. The code is available at https://github.com/erernan/ESampling.","doi":"10.24963/ijcai.2024/153","url":"https://ijcai.org/proceedings/2024/153","pdf":"https://ijcai.org/proceedings/2024/0153.pdf","markdown":"https://ijcai.org/proceedings/2024/153.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/153","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Computer Vision","track":"Main Track","number":153,"pages":{"first":1380,"last":1388},"keywords":["Computer Vision: CV: 3D computer vision","Computer Vision: CV: Applications"],"videos":[],"references":[{"position":1,"raw":"[Azinović et al., 2022] Dejan Azinović, Ricardo Martin- Brualla, Dan B Goldman, Matthias Nießner, and Justus Thies. Neural rgb-d surface reconstruction. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6290-6301, 2022.","title":"Neural rgb-d surface reconstruction","authors":"Dejan Azinović, Ricardo Martin-Brualla, Dan B Goldman, Matthias Nießner, Justus Thies","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Barron et al., 2021] Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P. Srinivasan. Mip-nerf: A multiscale represen- tation for anti-aliasing neural radiance fields, 2021.","title":"Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields","authors":"Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, Pratul P Srinivasan","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Barron et al., 2022] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip- nerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5470-5479, 2022.","title":"Mipnerf 360: Unbounded anti-aliased neural radiance fields","authors":"Jonathan T Barron, Ben Mildenhall, Dor Verbin, Peter Pratul P Srinivasan, Hedman","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Fan et al., 2023] Yue Fan, Ivan Skorokhodov, Oleg Voynov, Savva Ignatyev, Evgeny Burnaev, Peter Wonka, and Yiqun Wang. Factored-neus: Reconstructing surfaces, illumi- nation, and materials of possibly glossy objects. arXiv preprint arXiv:2305.17929, 2023.","title":"Factored-neus: Reconstructing surfaces, illumination, and materials of possibly glossy objects","authors":"Ivan Yue Fan, Oleg Skorokhodov, Savva Voynov, Evgeny Ignatyev, Peter Burnaev, Yiqun Wonka, Wang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Fridovich-Keil et al., 2022] Sara Fridovich-Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pages 5501-5510, 2022.","title":"Plenoxels: Radiance fields without neural networks","authors":"Sara Fridovich-Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, Angjoo Kanazawa","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fu et al., 2022] Qiancheng Fu, Qingshan Xu, Yew Soon Ong, and Wenbing Tao. Geo-neus: Geometry-consistent neural implicit surfaces learning for multi-view recon- struction. Advances in Neural Information Processing Sys- tems, 35:3403-3416, 2022.","title":"Geo-neus: Geometry-consistent neural implicit surfaces learning for multi-view reconstruction","authors":"Fu","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Ge et al., 2023] Wenhang Ge, Tao Hu, Haoyu Zhao, Shu Liu, and Ying-Cong Chen. Ref-neus: Ambiguity-reduced neural implicit surface learning for multi-view reconstruc- tion with reflection. arXiv preprint arXiv:2303.10840, 2023.","title":"Ref-neus: Ambiguity-reduced neural implicit surface learning for multi-view reconstruction with reflection","authors":"Wenhang Ge, Tao Hu, Haoyu Zhao, Shu Liu, Ying-Cong Chen","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Jensen et al., 2014] Rasmus Jensen, Anders Dahl, George Vogiatzis, Engin Tola, and Henrik Aanæs. Large scale multi-view stereopsis evaluation. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 406-413, 2014.","title":"Large scale multi-view stereopsis evaluation","authors":"Rasmus Jensen, Anders Dahl, George Vogiatzis, Engin Tola, Henrik Aanæs","year":"2014","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jiang et al., 2023] Hanqi Jiang, Cheng Zeng, Runnan Chen, Shuai Liang, Yinhe Han, Yichao Gao, and Conglin Wang. Depth-neus: Neural implicit surfaces learning for multi- view reconstruction based on depth information optimiza- tion. arXiv preprint arXiv:2303.17088, 2023.","title":"Depth-neus: Neural implicit surfaces learning for multiview reconstruction based on depth information optimization","authors":"Hanqi Jiang, Cheng Zeng, Runnan Chen, Shuai Liang, Yinhe Han, Yichao Gao, Conglin Wang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kurz et al., 2022] Andreas Kurz, Thomas Neff, Zhaoyang Lv, Michael Zollhöfer, and Markus Steinberger. Adanerf: Adaptive sampling for real-time rendering of neural radi- ance fields. In European Conference on Computer Vision, pages 254-270. Springer, 2022.","title":"Adanerf: Adaptive sampling for real-time rendering of neural radiance fields","authors":"Andreas Kurz, Thomas Neff, Zhaoyang Lv, Michael Zollhöfer, Markus Steinberger","year":"2022","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2022] Ruilong Li, Matthew Tancik, and Angjoo Kanazawa. Nerfacc: A general nerf acceleration toolbox. arXiv preprint arXiv:2210.04847, 2022.","title":"Nerfacc: A general nerf acceleration toolbox","authors":"Li","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2023] Ruilong Li, Hang Gao, Matthew Tancik, and Angjoo Kanazawa. Nerfacc: Efficient sampling acceler- ates nerfs. arXiv preprint arXiv:2305.04966, 2023.","title":"Nerfacc: Efficient sampling accelerates nerfs","authors":"Li","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lindell et al., 2021] David B Lindell, Julien NP Martel, and Gordon Wetzstein. Autoint: Automatic integration for fast neural volume rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14556-14565, 2021.","title":"Autoint: Automatic integration for fast neural volume rendering","authors":"Julien Np David B Lindell, Gordon Martel, Wetzstein","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mildenhall et al., 2020] Ben Mildenhall, Pratul P. Srini- vasan, Matthew Tancik, Jonathan T. Barron, Ravi Ra- mamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.","title":"Nerf: Representing scenes as neural radiance fields for view synthesis","authors":"Ben Mildenhall, P Pratul, Matthew Srinivasan, Jonathan T Tancik, Ravi Barron, Ren Ramamoorthi, Ng","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Müller et al., 2022] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Trans. Graph., 41(4):102:1-102:15, July 2022.","title":"Instant neural graphics primitives with a multiresolution hash encoding","authors":"Thomas Müller, Alex Evans, Christoph Schied, Alexander Keller","year":"2022","venue":"ACM Trans. Graph","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Piala and Clark, 2021] Martin Piala and Ronald Clark. Ter- minerf: Ray termination prediction for efficient neural ren- dering. In 2021 International Conference on 3D Vision (3DV), pages 1106-1114. IEEE, 2021.","title":"Terminerf: Ray termination prediction for efficient neural rendering","authors":"Martin Piala, Ronald Clark","year":"2021","venue":"International Conference on 3D Vision (3DV)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Rosu and Behnke, 2023] Radu Alexandru Rosu and Sven Behnke. Permutosdf: Fast multi-view reconstruction with implicit surfaces using permutohedral lattices. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8466-8475, 2023.","title":"Permutosdf: Fast multi-view reconstruction with implicit surfaces using permutohedral lattices","authors":"Alexandru Radu, Sven Rosu, Behnke","year":"2023","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Sun et al., 2022] Jiaming Sun, Xi Chen, Qianqian Wang, Zhengqi Li, Hadar Averbuch-Elor, Xiaowei Zhou, and Noah Snavely. Neural 3d reconstruction in the wild. In ACM SIGGRAPH 2022 Conference Proceedings, pages 1- 9, 2022.","title":"Neural 3d reconstruction in the wild","authors":"Jiaming Sun, Xi Chen, Qianqian Wang, Zhengqi Li, Hadar Averbuch-Elor, Xiaowei Zhou, Noah Snavely","year":"2022","venue":"ACM SIGGRAPH 2022 Conference Proceedings","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Takikawa et al., 2021] Towaki Takikawa, Joey Litalien, Kangxue Yin, Karsten Kreis, Charles Loop, Derek Nowrouzezahrai, Alec Jacobson, Morgan McGuire, and Sanja Fidler. Neural geometric level of detail: Real-time rendering with implicit 3D shapes. 2021.","title":"Neural geometric level of detail: Real-time rendering with implicit 3D shapes","authors":"Takikawa","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wang et al., 2021] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume ren- dering for multi-view reconstruction. NeurIPS, 2021.","title":"Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction","authors":"Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, Wenping Wang","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang et al., 2023a] Yiming Wang, Qin Han, Marc Haber- mann, Kostas Daniilidis, Christian Theobalt, and Lingjie Liu. Neus2: Fast learning of neural implicit surfaces for multi-view reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.","title":"Neus2: Fast learning of neural implicit surfaces for multi-view reconstruction","authors":"Wang","year":"2023","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang et al., 2023b] Zian Wang, Tianchang Shen, Mer- lin Nimier-David, Nicholas Sharp, Jun Gao, Alexander Keller, Sanja Fidler, Thomas Müller, and Zan Gojcic. Adaptive shells for efficient neural radiance field render- ing. ACM Transactions on Graphics (TOG), 42(6):1-15, 2023.","title":"Adaptive shells for efficient neural radiance field rendering","authors":"Wang","year":"2023","venue":"ACM Transactions on Graphics (TOG)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Xu et al., 2023] Jiale Xu, Xintao Wang, Weihao Cheng, Yan-Pei Cao, Ying Shan, Xiaohu Qie, and Shenghua Gao. Dream3d: Zero-shot text-to-3d synthesis using 3d shape prior and text-to-image diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 20908-20918, June 2023.","title":"Dream3d: Zero-shot text-to-3d synthesis using 3d shape prior and text-to-image diffusion models","authors":"Jiale Xu, Xintao Wang, Weihao Cheng, Yan-Pei Cao, Ying Shan, Xiaohu Qie, Shenghua Gao","year":"2023","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yariv et al., 2021] Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. Volume rendering of neural implicit sur- faces. In Thirty-Fifth Conference on Neural Information Processing Systems, 2021.","title":"Volume rendering of neural implicit surfaces","authors":"Lior Yariv, Jiatao Gu, Yoni Kasten, Yaron Lipman","year":"2021","venue":"Thirty-Fifth Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yariv et al., 2023] Lior Yariv, Peter Hedman, Christian Reiser, Dor Verbin, Pratul P. Srinivasan, Richard Szeliski, Jonathan T. Barron, and Ben Mildenhall. Bakedsdf: Mesh- ing neural sdfs for real-time view synthesis. arXiv, 2023.","title":"Bakedsdf: Meshing neural sdfs for real-time view synthesis","authors":"Lior Yariv, Peter Hedman, Christian Reiser, Dor Verbin, P Pratul, Richard Srinivasan, Jonathan T Szeliski, Ben Barron, Mildenhall","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yu et al., 2022a] Zehao Yu, Anpei Chen, Bozidar An- tic, Songyou Peng, Apratim Bhattacharyya, Michael Niemeyer, Siyu Tang, Torsten Sattler, and Andreas Geiger. Sdfstudio: A unified framework for surface reconstruction, 2022.","title":"Sdfstudio: A unified framework for surface reconstruction","authors":"Yu","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Yu et al., 2022b] Zehao Yu, Songyou Peng, Michael Niemeyer, Torsten Sattler, and Andreas Geiger. Monosdf: Exploring monocular geometric cues for neural implicit surface reconstruction. Advances in neural information processing systems, 35:25018-25032, 2022.","title":"Monosdf: Exploring monocular geometric cues for neural implicit surface reconstruction","authors":"Yu","year":"2022","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zheng et al., 2022] Xin-Yang Zheng, Yang Liu, Peng-Shuai Wang, and Xin Tong. Sdf-stylegan: Implicit sdf-based stylegan for 3d shape generation. In Comput. Graph. Fo- rum (SGP), 2022.","title":"Sdf-stylegan: Implicit sdf-based stylegan for 3d shape generation","authors":"Xin-Yang Zheng, Yang Liu, Peng-Shuai Wang, Xin Tong","year":"2022","venue":"Comput. Graph. Forum (SGP)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}