{"type":"ScholarlyArticle","title":"PRNet: Point-Range Fusion Network for Real-Time LiDAR Semantic Segmentation","authors":[{"name":"Xiaoyan Li","slug":"xiaoyan-li","given":"Xiaoyan","family":"Li","affiliation":"University of Chinese Academy of Sciences; Institute of Computing Technology , Chinese Academy of Sciences"},{"name":"Gang Zhang","slug":"gang-zhang","given":"Gang","family":"Zhang","affiliation":"Damo Academy , Alibaba Group"},{"name":"Tao Jiang","slug":"tao-jiang","given":"Tao","family":"Jiang","affiliation":"Institute of Computing Technology , Chinese Academy of Sciences"},{"name":"Xufen Cai","slug":"xufen-cai","given":"Xufen","family":"Cai","affiliation":"Beijing School"},{"name":"Zhenhua Wang","slug":"zhenhua-wang","given":"Zhenhua","family":"Wang","affiliation":"Damo Academy , Alibaba Group"}],"abstract":"Accurate and real-time LiDAR semantic segmentation is necessary for advanced autonomous driving systems. To guarantee a fast inference speed, previous methods utilize the highly optimized 2D convolutions to extract features on the range view (RV), which is the most compact representation of the LiDAR point clouds. However, these methods often suffer from lower accuracy for two reasons: 1) the information loss during the projection from 3D points to the RV, 2) the semantic ambiguity when 3D points labels are assigned according to the RV predictions. In this work, we introduce an end-to-end point-range fusion network (PRNet) that extracts semantic features mainly on the RV and iteratively fuses the RV features back to the 3D points for the final prediction. Besides, a novel range view projection (RVP) operation is designed to alleviate the information loss during the projection to the RV, and a point-range convolution (PRConv) is proposed to automatically mitigate the semantic ambiguity during transmitting features from the RV back to 3D points. Experiments on the SemanticKITTI and nuScenes benchmarks demonstrate that the PRNet pushes the range-based methods to a new state-of-the-art, and achieves a better speed-accuracy trade-off.","doi":"10.24963/ijcai.2022/156","url":"https://ijcai.org/proceedings/2022/156","pdf":"https://ijcai.org/proceedings/2022/0156.pdf","markdown":"https://ijcai.org/proceedings/2022/156.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/156","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":156,"pages":{"first":1116,"last":1122},"keywords":["Computer Vision: Scene analysis and understanding","Computer Vision: Segmentation","Robotics: Applications","Robotics: Perception","Computer Vision: 3D Computer Vision"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986630","length_sec":88,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984773","length_sec":268,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Aksoy et al., 2020] Eren Erdal Aksoy, Saimir Baci, and Sel- cuk Cavdar. Salsanet: Fast road and vehicle segmenta- tion in lidar point clouds for autonomous driving. In 2020 IEEE Intelligent Vehicles Symposium (IV), pages 926-932. IEEE, 2020.","title":"Salsanet: Fast road and vehicle segmentation in lidar point clouds for autonomous driving","authors":"Eren Erdal Aksoy, Saimir Baci, Selcuk Cavdar","year":"2020","venue":"IEEE Intelligent Vehicles Symposium (IV)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Behley et al., 2019] Jens Behley, Martin Garbade, Andres Milioto, Jan Quenzel, Sven Behnke, Cyrill Stachniss, and Jurgen Gall. Semantickitti: A dataset for semantic scene understanding of lidar sequences. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9297-9307, 2019.","title":"Semantickitti: A dataset for semantic scene understanding of lidar sequences","authors":"Behley","year":"2019","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Berman et al., 2018] Maxim Berman, Amal Rannen Triki, and Matthew B Blaschko. The lovász-softmax loss: A tractable surrogate for the optimization of the intersection- over-union measure in neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4413-4421, 2018.","title":"The lovász-softmax loss: A tractable surrogate for the optimization of the intersectionover-union measure in neural networks","authors":"Berman","year":"2018","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Caesar et al., 2020] Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11621-11631, 2020.","title":"nuscenes: A multimodal dataset for autonomous driving","authors":"Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, Oscar Beijbom","year":"2020","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cheng et al., 2021] Ran Cheng, Ryan Razani, Ehsan Taghavi, Enxu Li, and Bingbing Liu. 2-s3net: Attentive feature fusion with adaptive feature selection for sparse semantic segmentation network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12547-12556, 2021.","title":"2-s3net: Attentive feature fusion with adaptive feature selection for sparse semantic segmentation network","authors":"Ran Cheng, Ryan Razani, Ehsan Taghavi, Enxu Li, Bingbing Liu","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Choy et al., 2019] Christopher Choy, JunYoung Gwak, and Silvio Savarese. 4d spatio-temporal convnets: Minkowski convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3075-3084, 2019.","title":"4d spatio-temporal convnets: Minkowski convolutional neural networks","authors":"Christopher Choy, Junyoung Gwak, Silvio Savarese","year":"2019","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Cortinhal et al., 2020] Tiago Cortinhal, George Tzelepis, and Eren Erdal Aksoy. Salsanext: Fast, uncertainty-aware semantic segmentation of lidar point clouds. In Interna- tional Symposium on Visual Computing, pages 207-222. Springer, 2020.","title":"Salsanext: Fast, uncertainty-aware semantic segmentation of lidar point clouds","authors":"Tiago Cortinhal, George Tzelepis, Eren Erdal, Aksoy","year":"2020","venue":"International Symposium on Visual Computing","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hu et al., 2020] Qingyong Hu, Bo Yang, Linhai Xie, Ste- fano Rosa, Yulan Guo, Zhihua Wang, Niki Trigoni, and Andrew Markham. Randla-net: Efficient semantic seg- mentation of large-scale point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 11108-11117, 2020.","title":"Randla-net: Efficient semantic segmentation of large-scale point clouds","authors":"Qingyong Hu, Bo Yang, Linhai Xie, Stefano Rosa, Yulan Guo, Zhihua Wang, Niki Trigoni, Andrew Markham","year":"2020","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Liong et al., 2020] Venice Erin Liong, Thi Ngoc Tho Nguyen, Sergi Widjaja, Dhananjai Sharma, and Zhuang Jie Chong. Amvnet: Assertion-based multi- view fusion network for lidar semantic segmentation. arXiv preprint arXiv:2012.04934, 2020.","title":"Amvnet: Assertion-based multiview fusion network for lidar semantic segmentation","authors":"Liong","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Milioto et al., 2019] Andres Milioto, Ignacio Vizzo, Jens Behley, and Cyrill Stachniss. Rangenet++: Fast and ac- curate lidar semantic segmentation. In 2019 IEEE/RSJ In- ternational Conference on Intelligent Robots and Systems (IROS), pages 4213-4220. IEEE, 2019.","title":"Rangenet++: Fast and accurate lidar semantic segmentation","authors":"Andres Milioto, Ignacio Vizzo, Jens Behley, Cyrill Stachniss","year":"2019","venue":"IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Qi et al., 2017a] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652-660, 2017.","title":"Pointnet: Deep learning on point sets for 3d classification and segmentation","authors":"Qi","year":"2017","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Qi et al., 2017b] Charles R Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. arXiv preprint arXiv:1706.02413, 2017.","title":"Pointnet++: Deep hierarchical feature learning on point sets in a metric space","authors":"Qi","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Razani et al., 2021] Ryan Razani, Ran Cheng, Ehsan Taghavi, and Liu Bingbing. Lite-hdseg: Lidar semantic segmentation using lite harmonic dense convolutions. In 2021 IEEE International Conference on Robotics and Au- tomation (ICRA), pages 9550-9556. IEEE, 2021.","title":"Lite-hdseg: Lidar semantic segmentation using lite harmonic dense convolutions","authors":"Ryan Razani, Ran Cheng, Ehsan Taghavi, Liu Bingbing","year":"2021","venue":"IEEE International Conference on Robotics and Automation (ICRA)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Tang et al., 2020] Haotian Tang, Zhijian Liu, Shengyu Zhao, Yujun Lin, Ji Lin, Hanrui Wang, and Song Han. Searching efficient 3d architectures with sparse point- voxel convolution. In European Conference on Computer Vision, pages 685-702. Springer, 2020.","title":"Searching efficient 3d architectures with sparse pointvoxel convolution","authors":"Tang","year":"2020","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Thomas et al., 2019] Hugues Thomas, Charles R Qi, Jean- Emmanuel Deschaud, Beatriz Marcotegui, Franc ¸ois Goulette, and Leonidas J Guibas. Kpconv: Flexible and deformable convolution for point clouds. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6411-6420, 2019.","title":"Kpconv: Flexible and deformable convolution for point clouds","authors":"Hugues Thomas, Charles R Qi, Jean-Emmanuel Deschaud, Beatriz Marcotegui, Leonidas J Franc ¸ois Goulette, Guibas","year":"2019","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Xu et al., 2020] Chenfeng Xu, Bichen Wu, Zining Wang, Wei Zhan, Peter Vajda, Kurt Keutzer, and Masayoshi Tomizuka. Squeezesegv3: Spatially-adaptive convolution for efficient point-cloud segmentation. In European Con- ference on Computer Vision, pages 1-19. Springer, 2020.","title":"Squeezesegv3: Spatially-adaptive convolution for efficient point-cloud segmentation","authors":"Xu","year":"2020","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Xu et al., 2021] Jianyun Xu, Ruixiang Zhang, Jian Dou, Yushi Zhu, Jie Sun, and Shiliang Pu. Rpvnet: A deep and efficient range-point-voxel fusion network for lidar point cloud segmentation. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision, pages 16024- 16033, 2021.","title":"Rpvnet: A deep and efficient range-point-voxel fusion network for lidar point cloud segmentation","authors":"Xu","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Ye et al., 2021] Maosheng Ye, Shuangjie Xu, Tongyi Cao, and Qifeng Chen. Drinet: A dual-representation iterative learning network for point cloud segmentation. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pages 7447-7456, 2021.","title":"Drinet: A dual-representation iterative learning network for point cloud segmentation","authors":"Ye","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Zhang et al., 2020] Yang Zhang, Zixiang Zhou, Philip David, Xiangyu Yue, Zerong Xi, Boqing Gong, and Has- san Foroosh. Polarnet: An improved grid representation for online lidar point clouds semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9601-9610, 2020.","title":"Polarnet: An improved grid representation for online lidar point clouds semantic segmentation","authors":"Yang Zhang, Zixiang Zhou, Philip David, Xiangyu Yue, Zerong Xi, Boqing Gong, Hassan Foroosh","year":"2020","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Zhu et al., 2021] Xinge Zhu, Hui Zhou, Tai Wang, Fangzhou Hong, Yuexin Ma, Wei Li, Hongsheng Li, and Dahua Lin. Cylindrical and asymmetrical 3d convolution networks for lidar segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9939-9948, 2021.","title":"Cylindrical and asymmetrical 3d convolution networks for lidar segmentation","authors":"Xinge Zhu, Hui Zhou, Tai Wang, Fangzhou Hong, Yuexin Ma, Wei Li, Hongsheng Li, Dahua Lin","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}