{"type":"ScholarlyArticle","title":"RePaint-NeRF: NeRF Editting via Semantic Masks and Diffusion Models","authors":[{"name":"Xingchen Zhou","slug":"xingchen-zhou","given":"Xingchen","family":"Zhou","affiliation":"College of Computer Science and Software Engineering , Shenzhen University; Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)"},{"name":"Ying He","slug":"ying-he","given":"Ying","family":"He","affiliation":"College of Computer Science and Software Engineering , Shenzhen University; Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)"},{"name":"F. Richard Yu","slug":"f-richard-yu","given":"F. Richard","family":"Yu","affiliation":"College of Computer Science and Software Engineering , Shenzhen University; Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)"},{"name":"Jianqiang Li","slug":"jianqiang-li","given":"Jianqiang","family":"Li","affiliation":"College of Computer Science and Software Engineering , Shenzhen University; National Engineering Laboratory for Big Data System Computing Technology , Shenzhen University"},{"name":"You Li","slug":"you-li","given":"You","family":"Li","affiliation":"Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)"}],"abstract":"The emergence of Neural Radiance Fields (NeRF) has promoted the development of synthesized high-fidelity views of the intricate real world. However, it is still a very demanding task to repaint the content in NeRF. In this paper, we propose a novel framework that can take RGB images as input and alter the 3D content in neural scenes. Our work leverages existing diffusion models to guide changes in the designated 3D content. Specifically, we semantically select the target object and a pre-trained diffusion model will guide the NeRF model to generate new 3D objects, which can improve the editability, diversity, and application range of NeRF. Experiment results show that our algorithm is effective for editing 3D objects in NeRF under different text prompts, including editing appearance, shape, and more. We validate our method on both real-world datasets and synthetic-world datasets for these editing tasks. Please visit https://repaintnerf.github.io for a better view of our results.","doi":"10.24963/ijcai.2023/201","url":"https://ijcai.org/proceedings/2023/201","pdf":"https://ijcai.org/proceedings/2023/0201.pdf","markdown":"https://ijcai.org/proceedings/2023/201.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/201","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Computer Vision","track":"Main Track","number":201,"pages":{"first":1813,"last":1821},"keywords":["Computer Vision: CV: Neural generative models, auto encoders, GANs","Computer Vision: CV: Applications","Computer Vision: CV: 3D computer vision"],"videos":[],"references":[{"position":1,"raw":"[Aharchi and Ait Kbir, 2020] M Aharchi and M Ait Kbir. A review on 3d reconstruction techniques from 2d images. In Proceedings of the Third International Conference on Smart City Applications, pages 510-522, 2020.","title":"A review on 3d reconstruction techniques from 2d images","authors":"M Aharchi, M Ait, Kbir","year":"2020","venue":"Proceedings of the Third International Conference on Smart City Applications","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Barron et al., 2021] Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P Srinivasan. Mip-nerf: A multiscale represen- tation for anti-aliasing neural radiance fields. In Proceed- ings of the IEEE/CVF International Conference on Com- puter Vision, pages 5855-5864, 2021.","title":"Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields","authors":"Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, Pratul P Srinivasan","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Deng et al., 2022] Kangle Deng, Andrew Liu, Jun-Yan Zhu, and Deva Ramanan. Depth-supervised nerf: Fewer views and faster training for free. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12882-12891, 2022.","title":"Depth-supervised nerf: Fewer views and faster training for free","authors":"Kangle Deng, Andrew Liu, Jun-Yan Zhu, Deva Ramanan","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Fridovich-Keil et al., 2022] Sara Fridovich-Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pages 5501-5510, 2022.","title":"Plenoxels: Radiance fields without neural networks","authors":"Sara Fridovich-Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, Angjoo Kanazawa","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Ho et al., 2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.","title":"Denoising diffusion probabilistic models","authors":"Jonathan Ho, Ajay Jain, Pieter Abbeel","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Jain et al., 2022] Ajay Jain, Ben Mildenhall, Jonathan T Barron, Pieter Abbeel, and Ben Poole. Zero-shot text- guided object generation with dream fields. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 867-876, 2022.","title":"Zero-shot textguided object generation with dream fields","authors":"Ajay Jain, Ben Mildenhall, Jonathan T Barron, Pieter Abbeel, Ben Poole","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Kingma and Ba, 2014] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kobayashi et al., 2022] Sosuke Kobayashi, Eiichi Mat- sumoto, and Vincent Sitzmann. Decomposing nerf for editing via feature field distillation. arXiv preprint arXiv:2205.15585, 2022.","title":"Decomposing nerf for editing via feature field distillation","authors":"Kobayashi","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kundu et al., 2022] Abhijit Kundu, Kyle Genova, Xiaoqi Yin, Alireza Fathi, Caroline Pantofaru, Leonidas J Guibas, Andrea Tagliasacchi, Frank Dellaert, and Thomas Funkhouser. Panoptic neural fields: A semantic object- aware neural scene representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12871-12881, 2022.","title":"Panoptic neural fields: A semantic objectaware neural scene representation","authors":"Abhijit Kundu, Kyle Genova, Xiaoqi Yin, Alireza Fathi, Caroline Pantofaru, Leonidas J Guibas, Andrea Tagliasacchi, Frank Dellaert, Thomas Funkhouser","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li et al., 2019] Wei Li, CW Pan, Rong Zhang, JP Ren, YX Ma, Jin Fang, FL Yan, QC Geng, XY Huang, HJ Gong, et al. Aads: Augmented autonomous driving simulation using data-driven algorithms. Science robotics, 4(28):eaaw0863, 2019.","title":"Aads: Augmented autonomous driving simulation using data-driven algorithms","authors":"Wei Li, Rong Cw Pan, Zhang, Jp Ren, Jin Ma, Fang, Fl Yan, Geng, Huang, Hj Gong","year":"2019","venue":"Science robotics","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2022] Boyi Li, Kilian Q Weinberger, Serge Be- longie, Vladlen Koltun, and René Ranftl. Language-driven semantic segmentation. arXiv preprint arXiv:2201.03546, 2022.","title":"Language-driven semantic segmentation","authors":"Boyi Li, Q Kilian, Serge Weinberger, Vladlen Belongie, René Koltun, Ranftl","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lin et al., 2022] Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, and Tsung-Yi Lin. Magic3d: High-resolution text-to-3d content creation. arXiv preprint arXiv:2211.10440, 2022.","title":"Magic3d: High-resolution text-to-3d content creation","authors":"Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, Tsung-Yi Lin","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Liu et al., 2021] Steven Liu, Xiuming Zhang, Zhoutong Zhang, Richard Zhang, Jun-Yan Zhu, and Bryan Russell. Editing conditional radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5773-5783, 2021.","title":"Editing conditional radiance fields","authors":"Steven Liu, Xiuming Zhang, Zhoutong Zhang, Richard Zhang, Jun-Yan Zhu, Bryan Russell","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Metzer et al., 2022] Gal Metzer, Elad Richardson, Or Patashnik, Raja Giryes, and Daniel Cohen-Or. Latent-nerf for shape-guided generation of 3d shapes and textures. arXiv preprint arXiv:2211.07600, 2022.","title":"Latent-nerf for shape-guided generation of 3d shapes and textures","authors":"Metzer","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mildenhall et al., 2019] Ben Mildenhall, Pratul P. Srini- vasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar. Local light field fusion: Practical view synthesis with prescrip- tive sampling guidelines. ACM Transactions on Graphics (TOG), 2019.","title":"Local light field fusion: Practical view synthesis with prescriptive sampling guidelines","authors":"Ben Mildenhall, P Pratul, Rodrigo Srinivasan, Nima Ortiz-Cayon, Ravi Khademi Kalantari, Ren Ramamoorthi, Abhishek Ng, Kar","year":"2019","venue":"ACM Transactions on Graphics","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Mildenhall et al., 2021] Ben Mildenhall, Pratul P Srini- vasan, Matthew Tancik, Jonathan T Barron, Ravi Ra- mamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.","title":"Nerf: Representing scenes as neural radiance fields for view synthesis","authors":"Ben Mildenhall, P Pratul, Matthew Srinivasan, Jonathan T Tancik, Ravi Barron, Ren Ramamoorthi, Ng","year":"2021","venue":"Communications of the ACM","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Mirzaei et al., 2022] Ashkan Mirzaei, Yash Kant, Jonathan Kelly, and Igor Gilitschenski. Laterf: Label and text driven object radiance fields. In Proceedings of European Con- ference on Computer Vision, pages 20-36, 2022.","title":"Laterf: Label and text driven object radiance fields","authors":"Ashkan Mirzaei, Yash Kant, Jonathan Kelly, Igor Gilitschenski","year":"2022","venue":"Proceedings of European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Müller et al., 2022] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. arXiv preprint arXiv:2201.05989, 2022.","title":"Instant neural graphics primitives with a multiresolution hash encoding","authors":"Thomas Müller, Alex Evans, Christoph Schied, Alexander Keller","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Nichol and Dhariwal, 2021] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion proba- bilistic models. In International Conference on Machine Learning, pages 8162-8171, 2021.","title":"Improved denoising diffusion probabilistic models","authors":"Nichol, Dhariwal, Alexander Quinn, Nichol, Prafulla Dhariwal","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Poole et al., 2022] Ben Poole, Ajay Jain, Jonathan T Bar- ron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209.14988, 2022.","title":"Dreamfusion: Text-to-3d using 2d diffusion","authors":"Ben Poole, Ajay Jain, Jonathan T Barron, Ben Mildenhall","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Pumarola et al., 2021] Albert Pumarola, Enric Corona, Ger- ard Pons-Moll, and Francesc Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 10318-10327, 2021.","title":"D-nerf: Neural radiance fields for dynamic scenes","authors":"Pumarola","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Radford et al., 2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In Proceedings of Inter- national Conference on Machine Learning, pages 8748- 8763, 2021.","title":"Learning transferable visual models from natural language supervision","authors":"Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark","year":"2021","venue":"Proceedings of International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Ramesh et al., 2022] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.","title":"Hierarchical text-conditional image generation with clip latents","authors":"Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, Mark Chen","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Ranftl et al., 2022] René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(3), 2022.","title":"Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer","authors":"Ranftl","year":"2022","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Rombach et al., 2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.","title":"High-resolution image synthesis with latent diffusion models","authors":"Rombach","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Saharia et al., 2022] Chitwan","title":"Chitwan","authors":"Saharia","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language un- derstanding. Advances in Neural Information Processing Systems, 35:36479-36494, 2022.","title":"Photorealistic text-to-image diffusion models with deep language understanding","authors":"William Saharia, Saurabh Chan, Lala Saxena, Jay Li, Emily L Whang, Kamyar Denton, Raphael Gontijo Ghasemipour, Burcu Lopes, Tim Karagol Ayan, Salimans","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Schonberger and Frahm, 2016] Johannes L Schonberger and Jan-Michael Frahm. Structure-from-motion revisited. In Proceedings of the IEEE Conference on Computer vVsion and Pattern Recognition, pages 4104-4113, 2016.","title":"Structure-from-motion revisited","authors":"Frahm ; Schonberger, L Johannes, Jan-Michael Schonberger, Frahm","year":"2016","venue":"Proceedings of the IEEE Conference on Computer vVsion and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Tancik et al., 2022] Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul P Srinivasan, Jonathan T Barron, and Henrik Kretzschmar. Block-nerf: Scalable large scene neural view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8248-8258, 2022.","title":"Block-nerf: Scalable large scene neural view synthesis","authors":"Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, P Pratul, Jonathan T Srinivasan, Henrik Barron, Kretzschmar","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wang et al., 2022] Can Wang, Menglei Chai, Mingming He, Dongdong Chen, and Jing Liao. Clip-nerf: Text-and- image driven manipulation of neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3835-3844, 2022.","title":"Clip-nerf: Text-andimage driven manipulation of neural radiance fields","authors":"Wang","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Weder et al., 2022] Silvan Weder, Guillermo Garcia- Hernando, Aron Monszpart, Marc Pollefeys, Gabriel Brostow, Michael Firman, and Sara Vicente. Removing objects from neural radiance fields. arXiv preprint arXiv:2212.11966, 2022.","title":"Removing objects from neural radiance fields","authors":"Weder","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Xu and Harada, 2022] Tianhan Xu and Tatsuya Harada. De- forming radiance fields with cages. In European Confer- ence on Computer Vision, pages 159-175, 2022.","title":"Deforming radiance fields with cages","authors":"Harada Xu, Tianhan Xu, Tatsuya Harada","year":"2022","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Yang et al., 2021] Bangbang Yang, Yinda Zhang, Yinghao Xu, Yijin Li, Han Zhou, Hujun Bao, Guofeng Zhang, and Zhaopeng Cui. Learning object-compositional neural ra- diance field for editable scene rendering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13779-13788, 2021.","title":"Learning object-compositional neural radiance field for editable scene rendering","authors":"Bangbang Yang, Yinda Zhang, Yinghao Xu, Yijin Li, Han Zhou, Hujun Bao, Guofeng Zhang, Zhaopeng Cui","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Yuan et al., 2022] Yu-Jie Yuan, Yang-Tian Sun, Yu-Kun Lai, Yuewen Ma, Rongfei Jia, and Lin Gao. Nerf-editing: geometry editing of neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 18353-18364, 2022.","title":"Nerf-editing: geometry editing of neural radiance fields","authors":"Yu-Jie Yuan, Yang-Tian Sun, Yu-Kun Lai, Yuewen Ma, Rongfei Jia, Lin Gao","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Zhi et al., 2021] Shuaifeng Zhi, Tristan Laidlow, Stefan Leutenegger, and Andrew J Davison. In-place scene la- belling and understanding with implicit scene representa- tion. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 15838-15847, 2021.","title":"In-place scene labelling and understanding with implicit scene representation","authors":"Zhi","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}