{"type":"ScholarlyArticle","title":"Learning Implicit Body Representations from Double Diffusion Based Neural Radiance Fields","authors":[{"name":"Guangming Yao","slug":"guangming-yao","given":"Guangming","family":"Yao","affiliation":"Zhejiang University"},{"name":"Hongzhi Wu","slug":"hongzhi-wu","given":"Hongzhi","family":"Wu","affiliation":"Zhejiang University"},{"name":"Yi Yuan","slug":"yi-yuan","given":"Yi","family":"Yuan","affiliation":"NetEase Fuxi AI Lab"},{"name":"Lincheng Li","slug":"lincheng-li","given":"Lincheng","family":"Li","affiliation":"NetEase Fuxi AI Lab"},{"name":"Kun Zhou","slug":"kun-zhou","given":"Kun","family":"Zhou","affiliation":"Zhejiang University"},{"name":"Xin Yu","slug":"xin-yu","given":"Xin","family":"Yu","affiliation":"University of Technology Sydney"}],"abstract":"In this paper, we present a novel double diffusion based neural radiance field, dubbed DD-NeRF, to reconstruct human body geometry and render the human body appearance in novel views from a sparse set of images. We first propose a double diffusion mechanism to achieve expressive representations of input images by fully exploiting human body priors and image appearance details at two levels. At the coarse level, we first model the coarse human body poses and shapes via an unclothed 3D deformable vertex model as guidance. At the fine level, we present a multi-view sampling network to capture subtle geometric deformations and image detailed appearances, such as clothing and hair, from multiple input views. Considering the sparsity of the two level features, we diffuse them into feature volumes in the canonical space to construct neural radiance fields. Then, we present a signed distance function (SDF) regression network to construct body surfaces from the diffused features. Thanks to our double diffused representations, our method can even synthesize novel views of unseen subjects. Experiments on various datasets demonstrate that our approach outperforms the state-of-the-art in both geometric reconstruction and novel view synthesis.","doi":"10.24963/ijcai.2022/218","url":"https://ijcai.org/proceedings/2022/218","pdf":"https://ijcai.org/proceedings/2022/0218.pdf","markdown":"https://ijcai.org/proceedings/2022/218.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/218","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":218,"pages":{"first":1566,"last":1572},"keywords":["Computer Vision: Applications","Computer Vision: 3D Computer Vision","Computer Vision: Neural generative models, auto encoders, GANs"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986460","length_sec":87,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984603","length_sec":345,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Chen et al., 2021] Anpei Chen, Zexiang Xu, and Hao Su. Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo. arXiv preprint arXiv:2103.15595, 2021.","title":"Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo","authors":"Chen","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Dong and Fang, 2021] Junting Dong and Qi Fang. Fast and robust multi-person 3d pose estimation and tracking from multiple views. In T-PAMI, 2021.","title":"Junting Dong and Qi Fang. Fast and robust multi-person 3d pose estimation and tracking from multiple views","authors":"Fang Dong","year":"2021","venue":"T-PAMI","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Gilbert et al., 2018] Andrew Gilbert, Marco Volino, John Collomosse, and Adrian Hilton. Volumetric performance capture from minimal camera viewpoints. In ECCV, pages 566-581, 2018.","title":"Volumetric performance capture from minimal camera viewpoints","authors":"Andrew Gilbert, Marco Volino, John Collomosse, Adrian Hilton","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Graham and Engelcke, 2018] Benjamin Graham and Engel- cke. 3d semantic segmentation with submanifold sparse convolutional networks. In CVPR, pages 9224-9232, 2018.","title":"3d semantic segmentation with submanifold sparse convolutional networks","authors":"Benjamin Graham, Engelcke","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Gropp and Yariv, 2020] Amos Gropp and Lior Yariv. Im- plicit geometric regularization for learning shapes. arXiv preprint arXiv:2002.10099, 2020.","title":"Implicit geometric regularization for learning shapes","authors":"Yariv Gropp, Amos Gropp, Lior Yariv","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Kellnhofer et al., 2021] Petr Kellnhofer, Lars C Jebe, and Gordon Wetzstein. Neural lumigraph rendering. In CVPR, pages 4287-4297, 2021.","title":"Neural lumigraph rendering","authors":"Kellnhofer","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Kingma and Ba, 2014] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kwon et al., 2021] Youngjoong Kwon, Dahun Kim, Duygu Ceylan, and Henry Fuchs. Neural human performer: Learning generalizable radiance fields for human perfor- mance rendering. Advances in Neural Information Pro- cessing Systems, 34, 2021.","title":"Neural human performer: Learning generalizable radiance fields for human performance rendering","authors":"Kwon","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Loper et al., 2015] Matthew Loper, Naureen Mahmood, and Javier Romero. Smpl: A skinned multi-person linear model. ACM transactions on graphics (TOG), 34(6):1-16, 2015.","title":"Smpl: A skinned multi-person linear model","authors":"Matthew Loper, Naureen Mahmood, Javier Romero","year":"2015","venue":"ACM transactions on graphics (TOG)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lorensen and Cline, 1987] William E Lorensen and Har- vey E Cline. Marching cubes: A high resolution 3d sur- face construction algorithm. ACM siggraph, 21(4):163- 169, 1987.","title":"Marching cubes: A high resolution 3d surface construction algorithm","authors":"E William, Harvey E Lorensen, Cline","year":"1987","venue":"ACM siggraph","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Mildenhall et al., 2020] Ben Mildenhall, Pratul P Srini- vasan, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, pages 405- 421. Springer, 2020.","title":"Nerf: Representing scenes as neural radiance fields for view synthesis","authors":"Ben Mildenhall, Ren Pratul P Srinivasan, Ng","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Newell et al., 2016] Alejandro Newell, Kaiyu Yang, and Jia Deng. Stacked hourglass networks for human pose esti- mation. In ECCV, pages 483-499. Springer, 2016.","title":"Stacked hourglass networks for human pose estimation","authors":"Alejandro Newell, Kaiyu Yang, Jia Deng","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Niemeyer et al., 2020] Michael Niemeyer, Lars Mescheder, and Andreas Geiger. Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervi- sion. In CVPR, pages 3504-3515, 2020.","title":"Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision","authors":"Michael Niemeyer, Lars Mescheder, Andreas Geiger","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Peng et al., 2021] Sida Peng, Yuanqing Zhang, Hujun Bao, and Xiaowei Zhou. Neural body: Implicit neural repre- sentations with structured latent codes for novel view syn- thesis of dynamic humans. In CVPR, pages 9054-9063, 2021.","title":"Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans","authors":"Peng","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Saito et al., 2019] Shunsuke Saito, Zeng Huang, and Hao Li. Pifu: Pixel-aligned implicit function for high- resolution clothed human digitization. In CVPR, pages 2304-2314, 2019.","title":"Pifu: Pixel-aligned implicit function for highresolution clothed human digitization","authors":"Shunsuke Saito, Zeng Huang, Hao Li","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Saito et al., 2020] Shunsuke Saito, Tomas Simon, and Han- byul Joo. Pifuhd: Multi-level pixel-aligned implicit func- tion for high-resolution 3d human digitization. In CVPR, pages 84-93, 2020.","title":"Pifuhd: Multi-level pixel-aligned implicit function for high-resolution 3d human digitization","authors":"Shunsuke Saito, Tomas Simon, Hanbyul Joo","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Shao et al., 2021] Ruizhi Shao, Hongwen Zhang, Tao Yu, and Yebin Liu. Doublefield: Bridging the neural sur- face and radiance fields for high-fidelity human rendering. arXiv preprint arXiv:2106.03798, 2021.","title":"Doublefield: Bridging the neural surface and radiance fields for high-fidelity human rendering","authors":"Shao","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Vaswani and Shazeer, 2017] Ashish Vaswani and Noam Shazeer. Attention is all you need. In NeurIPS, pages 5998-6008, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wang et al., 2021a] Peng Wang, Lingjie Liu, Yuan Liu, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint arXiv:2106.10689, 2021.","title":"Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction","authors":"Wang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wang et al., 2021b] Qianqian Wang, Zhicheng Wang, and Kyle Genova. Ibrnet: Learning multi-view image-based rendering. In CVPR, 2021.","title":"Ibrnet: Learning multi-view image-based rendering","authors":"Wang","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Yariv et al., 2020] Lior Yariv, Yoni Kasten, and Dror Moran. Multiview neural surface reconstruction by dis- entangling geometry and appearance. arXiv preprint arXiv:2003.09852, 2020.","title":"Multiview neural surface reconstruction by disentangling geometry and appearance","authors":"Lior Yariv, Yoni Kasten, Dror Moran","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Yu et al., 2018] Tao Yu, Zerong Zheng, and Yebin Liu. Doublefusion: Real-time capture of human performances with inner body shapes from a single depth sensor. In CVPR, pages 7287-7296, 2018.","title":"Doublefusion: Real-time capture of human performances with inner body shapes from a single depth sensor","authors":"Tao Yu, Zerong Zheng, Yebin Liu","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yu et al., 2021] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from one or few images. In CVPR, pages 4578-4587, 2021.","title":"pixelnerf: Neural radiance fields from one or few images","authors":"Alex Yu, Vickie Ye, Matthew Tancik, Angjoo Kanazawa","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhang et al., 2018] Richard Zhang, Phillip Isola, and Alexei Efros. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018.","title":"The unreasonable effectiveness of deep features as a perceptual metric","authors":"Richard Zhang, Phillip Isola, Alexei Efros","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zheng et al., 2021] Yang Zheng, Ruizhi Shao, Qionghai Dai, and Yebin Liu. Deepmulticap: Performance capture of multiple characters using sparse multiview cameras. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6239-6249, 2021.","title":"Deepmulticap: Performance capture of multiple characters using sparse multiview cameras","authors":"Yang Zheng, Ruizhi Shao, Qionghai Dai, Yebin Liu","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}