{"type":"ScholarlyArticle","title":"Invertible Residual Neural Networks with Conditional Injector and Interpolator for Point Cloud Upsampling","authors":[{"name":"Aihua Mao","slug":"aihua-mao","given":"Aihua","family":"Mao","affiliation":"South China University of Technology"},{"name":"Yaqi Duan","slug":"yaqi-duan","given":"Yaqi","family":"Duan","affiliation":"South China University of Technology"},{"name":"Yu-Hui Wen","slug":"yu-hui-wen","given":"Yu-Hui","family":"Wen","affiliation":"Beijing Jiaotong University"},{"name":"Zihui Du","slug":"zihui-du","given":"Zihui","family":"Du","affiliation":"South China University of Technology"},{"name":"Hongmin Cai","slug":"hongmin-cai","given":"Hongmin","family":"Cai","affiliation":"South China University of Technology"},{"name":"Yong-Jin Liu","slug":"yong-jin-liu","given":"Yong-Jin","family":"Liu","affiliation":"Tsinghua University"}],"abstract":"Point clouds obtained by LiDAR and other sensors are usually sparse and irregular. Low-quality point clouds have serious influence on the final performance of downstream tasks. Recently, a point cloud upsampling network with normalizing flows has been proposed to address this problem. However, the network heavily relies on designing specialized architectures to achieve invertibility. In this paper, we propose a novel invertible residual neural network for point cloud upsampling, called PU-INN, which allows unconstrained architectures to learn more expressive feature transformations. Then, we propose a conditional injector to improve nonlinear transformation ability of the neural network while guaranteeing invertibility. Furthermore, a lightweight interpolator is proposed based on semantic similarity distance in the latent space, which can intuitively reflect the interpolation changes in Euclidean space. Qualitative and quantitative results show that our method outperforms the state-of-the-art works in terms of distribution uniformity, proximity-to-surface accuracy, 3D reconstruction quality, and computation efficiency.","doi":"10.24963/ijcai.2023/141","url":"https://ijcai.org/proceedings/2023/141","pdf":"https://ijcai.org/proceedings/2023/0141.pdf","markdown":"https://ijcai.org/proceedings/2023/141.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/141","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Computer Vision","track":"Main Track","number":141,"pages":{"first":1267,"last":1275},"keywords":["Computer Vision: CV: 3D computer vision","Computer Vision: CV: Neural generative models, auto encoders, GANs","Machine Learning: ML: Probabilistic machine learning"],"videos":[],"references":[{"position":1,"raw":"[Alexa et al., 2003] Marc Alexa, Johannes Behr, Daniel Cohen-Or, Shachar Fleishman, David Levin, and Clau- dio T. Silva. Computing and rendering point set surfaces. IEEE Transactions on visualization and computer graph- ics, 9(1):3-15, 2003.","title":"Computing and rendering point set surfaces","authors":"Alexa","year":"2003","venue":"IEEE Transactions on visualization and computer graphics","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Behrmann et al., 2019] Jens Behrmann, Will Grathwohl, Ricky TQ Chen, David Duvenaud, and Jörn-Henrik Jacob- sen. Invertible residual networks. In International Confer- ence on Machine Learning, pages 573-582. PMLR, 2019.","title":"Invertible residual networks","authors":"Behrmann","year":"2019","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Butt, 1998] Butt. Optimum design of chamfer distance transforms. IEEE Transactions on Image Processing, 7(10):1477-1484, 1998.","title":"Optimum design of chamfer distance transforms","authors":"Butt","year":"1998","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2018] Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary dif- ferential equations. Advances in neural information pro- cessing systems, 31, 2018.","title":"Neural ordinary differential equations","authors":"Yulia Ricky Tq Chen, Jesse Rubanova, David K Bettencourt, Duvenaud","year":"2018","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2019] Tian Qi Chen, Jens Behrmann, David Duvenaud, and Jörn-Henrik Jacobsen. Residual flows for invertible generative modeling. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché- Buc, Emily B. Fox, and Roman Garnett, editors, Ad- vances in Neural Information Processing Systems 32: An- nual Conference on Neural Information Processing Sys- tems 2019, NeurIPS 2019, December 8-14, 2019, Vancou- ver, BC, Canada, pages 9913-9923, 2019.","title":"Residual flows for invertible generative modeling","authors":"Tian Qi, Chen, Jens Behrmann, David Duvenaud, Jörn-Henrik Jacobsen","year":"2019","venue":"Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dinh et al., 2014] Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent compo- nents estimation. arXiv preprint arXiv:1410.8516, 2014.","title":"Nice: Non-linear independent components estimation","authors":"Laurent Dinh, David Krueger, Yoshua Bengio","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dinh et al., 2016] Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint arXiv:1605.08803, 2016.","title":"Density estimation using real nvp","authors":"Laurent Dinh, Jascha Sohl-Dickstein, Samy Bengio","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Fan et al., 2017] Haoqiang Fan, Hao Su, and Leonidas J Guibas. A point set generation network for 3d object re- construction from a single image. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 605-613, 2017.","title":"A point set generation network for 3d object reconstruction from a single image","authors":"Fan","year":"2017","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Geiger et al., 2013] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. The International Journal of Robotics Re- search, 32(11):1231-1237, 2013.","title":"Vision meets robotics: The kitti dataset","authors":"Andreas Geiger, Philip Lenz, Christoph Stiller, Raquel Urtasun","year":"2013","venue":"The International Journal of Robotics Research","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hu et al., 2020] Qingyong Hu, Bo Yang, Linhai Xie, Ste- fano Rosa, Yulan Guo, Zhihua Wang, Niki Trigoni, and Andrew Markham. Randla-net: Efficient semantic seg- mentation of large-scale point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 11108-11117, 2020.","title":"Randla-net: Efficient semantic segmentation of large-scale point clouds","authors":"Qingyong Hu, Bo Yang, Linhai Xie, Stefano Rosa, Yulan Guo, Zhihua Wang, Niki Trigoni, Andrew Markham","year":"2020","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Huang et al., 2009] Hui Huang, Dan Li, Hao Zhang, Uri Ascher, and Daniel Cohen-Or. Consolidation of unorga- nized point clouds for surface reconstruction. ACM trans- actions on graphics (TOG), 28(5):1-7, 2009.","title":"Consolidation of unorganized point clouds for surface reconstruction","authors":"Hui Huang, Dan Li, Hao Zhang, Uri Ascher, Daniel Cohen-Or","year":"2009","venue":"ACM transactions on graphics","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Huang et al., 2013] Hui Huang, Shihao Wu, Minglun Gong, Daniel Cohen-Or, Uri Ascher, and Hao Zhang. Edge- aware point set resampling. ACM transactions on graphics (TOG), 32(1):1-12, 2013.","title":"Edgeaware point set resampling","authors":"Hui Huang, Shihao Wu, Minglun Gong, Daniel Cohen-Or, Uri Ascher, Hao Zhang","year":"2013","venue":"ACM transactions on graphics (TOG)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Huttenlocher, 1993] Huttenlocher. Comparing images us- ing the hausdorff distance. IEEE Transactions on pattern analysis and machine intelligence, 15(9):850-863, 1993.","title":"Huttenlocher. Comparing images using the hausdorff distance","authors":"Huttenlocher","year":"1993","venue":"IEEE Transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Jacobsen et al., 2018] Jörn-Henrik Jacobsen, Arnold Smeulders, and Edouard Oyallon. i-revnet: Deep in- vertible networks. arXiv preprint arXiv:1802.07088, 2018.","title":"i-revnet: Deep invertible networks","authors":"Jacobsen","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kingma and Dhariwal, 2018] Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 con- volutions. Advances in neural information processing sys- tems, 31, 2018.","title":"Glow: Generative flow with invertible 1x1 convolutions","authors":"Dhariwal ; Kingma, P Durk, Prafulla Kingma, Dhariwal","year":"2018","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Klokov et al., 2020] Roman Klokov, Edmond Boyer, and Jakob Verbeek. Discrete point flow networks for efficient point cloud generation. In European Conference on Com- puter Vision. Springer, 2020.","title":"Discrete point flow networks for efficient point cloud generation","authors":"Klokov","year":"2020","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Li et al., 2018] Yangyan Li, Rui Bu, Mingchao Sun, Wei Wu, Xinhan Di, and Baoquan Chen. Pointcnn: Convo- lution on x-transformed points. Advances in neural infor- mation processing systems, 31:820-830, 2018.","title":"Pointcnn: Convolution on x-transformed points","authors":"Yangyan Li, Rui Bu, Mingchao Sun, Wei Wu, Xinhan Di, Baoquan Chen","year":"2018","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Li et al., 2019] Ruihui Li, Xianzhi Li, Chi-Wing Fu, Daniel Cohen-Or, and Pheng-Ann Heng. Pu-gan: a point cloud upsampling adversarial network. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7203-7212, 2019.","title":"Pu-gan: a point cloud upsampling adversarial network","authors":"Li","year":"2019","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Li et al., 2021] Ruihui Li, Xianzhi Li, Pheng-Ann Heng, and Chi-Wing Fu. Point cloud upsampling via disentan- gled refinement. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pages 344-353, 2021.","title":"Point cloud upsampling via disentangled refinement","authors":"Li","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Lipman et al., 2007] Yaron Lipman, Daniel Cohen-Or, David Levin, and Hillel Tal-Ezer. Parameterization-free projection for geometry reconstruction. ACM Transactions on Graphics (TOG), 26(3):22-es, 2007.","title":"Parameterization-free projection for geometry reconstruction","authors":"Yaron Lipman, Daniel Cohen-Or, David Levin, Hillel Tal-Ezer","year":"2007","venue":"ACM Transactions on Graphics (TOG)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Liu et al., 2019a] Yongcheng Liu, Bin Fan, Gaofeng Meng, Jiwen Lu, Shiming Xiang, and Chunhong Pan. Dense- point: Learning densely contextual representation for ef- ficient point cloud processing. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5239-5248, 2019.","title":"Densepoint: Learning densely contextual representation for efficient point cloud processing","authors":"Liu","year":"2019","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Liu et al., 2019b] Yongcheng Liu, Bin Fan, Shiming Xiang, and Chunhong Pan. Relation-shape convolutional neural network for point cloud analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8895-8904, 2019.","title":"Relation-shape convolutional neural network for point cloud analysis","authors":"Liu","year":"2019","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Long et al., 2022] Chen Long, WenXiao Zhang, Ruihui Li, Hao Wang, Zhen Dong, and Bisheng Yang. Pc2-pu: Patch correlation and point correlation for effective point cloud upsampling. In Proceedings of the 30th ACM International Conference on Multimedia, pages 2191-2201, 2022.","title":"Pc2-pu: Patch correlation and point correlation for effective point cloud upsampling","authors":"Chen Long, Wenxiao Zhang, Ruihui Li, Hao Wang, Zhen Dong, Bisheng Yang","year":"2022","venue":"Proceedings of the 30th ACM International Conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Mao et al., 2022] Aihua Mao, Zihui Du, Junhui Hou, Yaqi Duan, Yong-jin Liu, and Ying He. Pu-flow: a point cloud upsampling network with normalizing flows. IEEE Transactions on Visualization and Computer Graphics, DOI:10.1109/TVCG.2022.3196334, 2022.","title":"Pu-flow: a point cloud upsampling network with normalizing flows","authors":"Aihua Mao, Zihui Du, Junhui Hou, Yaqi Duan, Yong-Jin Liu, Ying He","year":"2022","venue":"IEEE Transactions on Visualization and Computer Graphics","doi":"10.1109/tvcg.2022.3196334","url":null,"internal_url":null},{"position":25,"raw":"[Miyato et al., 2018] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normal- ization for generative adversarial networks. arXiv preprint arXiv:1802.05957, 2018.","title":"Spectral normalization for generative adversarial networks","authors":"Miyato","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Peng et al., 2021] Songyou Peng, Chiyu \"Max\" Jiang, Yiyi Liao, Michael Niemeyer, Marc Pollefeys, and Andreas Geiger. Shape as points: A differentiable poisson solver. CoRR, abs/2106.03452, 2021.","title":"Shape as points: A differentiable poisson solver","authors":"Peng","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Qi et al., 2017a] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652-660, 2017.","title":"Pointnet: Deep learning on point sets for 3d classification and segmentation","authors":"Qi","year":"2017","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Qi et al., 2017b] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical fea- ture learning on point sets in a metric space. Advances in neural information processing systems, 30, 2017.","title":"Pointnet++: Deep hierarchical feature learning on point sets in a metric space","authors":"Qi","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Qian et al., 2020] Yue Qian, Junhui Hou, Sam Kwong, and Ying He. Pugeo-net: A geometry-centric network for 3d point cloud upsampling. In European Conference on Com- puter Vision, pages 752-769. Springer, 2020.","title":"Pugeo-net: A geometry-centric network for 3d point cloud upsampling","authors":"Junhui Yue Qian, Sam Hou, Ying Kwong, He","year":"2020","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Qian et al., 2021a] Guocheng Qian, Abdulellah Abual- shour, Guohao Li, Ali Thabet, and Bernard Ghanem. Pu- gcn: Point cloud upsampling using graph convolutional networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11683- 11692, 2021.","title":"Pugcn: Point cloud upsampling using graph convolutional networks","authors":"Qian","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Qian et al., 2021b] Yue Qian, Junhui Hou, Sam Kwong, and Ying He. Deep magnification-flexible upsampling over 3d point clouds. IEEE Trans. Image Process., 30:8354-8367, 2021.","title":"Deep magnification-flexible upsampling over 3d point clouds","authors":"Qian","year":"2021","venue":"IEEE Trans. Image Process","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Sketchfab, 2011] Inc. Sketchfab. Sketchfab -the best 3d viewer on the web. https://sketchfab.com/, 2011. Ac- cessed: 2021-11-25.","title":"Sketchfab -the best 3d viewer on the web","authors":null,"year":null,"venue":null,"doi":null,"url":"https://sketchfab.com/,2011.Ac-cessed:2021-11-25","internal_url":null},{"position":33,"raw":"[Thomas et al., 2019] Hugues Thomas, Charles R Qi, Jean- Emmanuel Deschaud, Beatriz Marcotegui, Franc ¸ois Goulette, and Leonidas J Guibas. Kpconv: Flexible and deformable convolution for point clouds. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6411-6420, 2019.","title":"Kpconv: Flexible and deformable convolution for point clouds","authors":"Hugues Thomas, Charles R Qi, Jean-Emmanuel Deschaud, Beatriz Marcotegui, Leonidas J Franc ¸ois Goulette, Guibas","year":"2019","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Wang et al., 2019] Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph cnn for learning on point clouds. Acm Transactions On Graphics (tog), 38(5):1-12, 2019.","title":"Dynamic graph cnn for learning on point clouds","authors":"Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, Justin M Solomon","year":"2019","venue":"Acm Transactions On Graphics (tog)","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Wu et al., 2015] Shihao Wu, Hui Huang, Minglun Gong, Matthias Zwicker, and Daniel Cohen-Or. Deep points consolidation. ACM Transactions on Graphics (ToG), 34(6):1-13, 2015.","title":"Deep points consolidation","authors":"Shihao Wu, Hui Huang, Minglun Gong, Matthias Zwicker, Daniel Cohen-Or","year":"2015","venue":"ACM Transactions on Graphics (ToG)","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Wu et al., 2019] Wenxuan Wu, Zhongang Qi, and Li Fuxin. Pointconv: Deep convolutional networks on 3d point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9621- 9630, 2019.","title":"Pointconv: Deep convolutional networks on 3d point clouds","authors":"Wenxuan Wu, Zhongang Qi, Li Fuxin","year":"2019","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Yang et al., 2019] Guandao Yang, Xun Huang, Zekun Hao, Ming-Yu Liu, Serge Belongie, and Bharath Hariharan. Pointflow: 3d point cloud generation with continuous nor- malizing flows. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pages 4541-4550, 2019.","title":"Pointflow: 3d point cloud generation with continuous normalizing flows","authors":"Guandao Yang, Xun Huang, Zekun Hao, Ming-Yu Liu, Serge Belongie, Bharath Hariharan","year":"2019","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Yifan et al., 2019] Wang Yifan, Shihao Wu, Hui Huang, Daniel Cohen-Or, and Olga Sorkine-Hornung. Patch- based progressive 3d point set upsampling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 5958-5967, 2019.","title":"Patchbased progressive 3d point set upsampling","authors":"Wang Yifan, Shihao Wu, Hui Huang, Daniel Cohen-Or, Olga Sorkine-Hornung","year":"2019","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Yu et al., 2018] Lequan Yu, Xianzhi Li, Chi-Wing Fu, Daniel Cohen-Or, and Pheng-Ann Heng. Pu-net: Point cloud upsampling network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2790-2799, 2018.","title":"Pu-net: Point cloud upsampling network","authors":"Lequan Yu, Xianzhi Li, Chi-Wing Fu, Daniel Cohen-Or, Pheng-Ann Heng","year":"2018","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}