{"type":"ScholarlyArticle","title":"Aggregation and Purification: Dual Enhancement Network for Point Cloud Few-shot Segmentation","authors":[{"name":"Guoxin Xiong","slug":"guoxin-xiong","given":"Guoxin","family":"Xiong","affiliation":"University of Science and Technology of China"},{"name":"Yuan Wang","slug":"yuan-wang","given":"Yuan","family":"Wang","affiliation":"University of Science and Technology of China"},{"name":"Zhaoyang Li","slug":"zhaoyang-li","given":"Zhaoyang","family":"Li","affiliation":"University of Science and Technology of China"},{"name":"Wenfei Yang","slug":"wenfei-yang","given":"Wenfei","family":"Yang","affiliation":"University of Science and Technology of China"},{"name":"Tianzhu Zhang","slug":"tianzhu-zhang","given":"Tianzhu","family":"Zhang","affiliation":"University of Science and Technology of China; Institute of Artificial Intelligence , Hefei Comprehensive National Science Center , China; Deep Space Exploration Lab"},{"name":"Xu Zhou","slug":"xu-zhou","given":"Xu","family":"Zhou","affiliation":"Sangfor Technologies Inc"},{"name":"Shifeng Zhang","slug":"shifeng-zhang","given":"Shifeng","family":"Zhang","affiliation":"Sangfor Technologies Inc"},{"name":"Yongdong Zhang","slug":"yongdong-zhang","given":"Yongdong","family":"Zhang","affiliation":"University of Science and Technology of China; Institute of Artificial Intelligence , Hefei Comprehensive National Science Center , China"}],"abstract":"Point cloud few-shot semantic segmentation (PC-FSS) aims to segment objects within query samples of new categories given only a handful of annotated support samples.  Although PC-FSS demonstrates enhanced category generalization capabilities compared to the fully supervised paradigm,  the prevalent significant scene discrepancies, which can be systematically summarized into intra-semantic diversity and semantic inconsistency, have posed substantial challenges to the area. In this work, we design a novel Dual Enhancement Network (DENet) to comprehensively tackle different kinds of scene discrepancies in a coherent and synergistic framework. The proposed DENet enjoys several merits. First, we design a mutual aggregation module to reconcile the intrinsic tension between the support prototypes and query point features, and the intra-semantic diversity is diminished in a bidirectional manner. Second, the consistent purification strategy is introduced to eliminate ambiguous prototypes, thereby reducing the mismatches brought by semantic inconsistency. Extensive experiments on S3DIS and ScanNet under different settings demonstrate that DENet significantly outperforms previous SOTAs.","doi":"10.24963/ijcai.2024/164","url":"https://ijcai.org/proceedings/2024/164","pdf":"https://ijcai.org/proceedings/2024/0164.pdf","markdown":"https://ijcai.org/proceedings/2024/164.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/164","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Computer Vision","track":"Main Track","number":164,"pages":{"first":1480,"last":1488},"keywords":["Computer Vision: CV: Transfer, low-shot, semi- and un- supervised learning","Computer Vision: CV: 3D computer vision"],"videos":[],"references":[{"position":1,"raw":"[Armeni et al., 2016] Iro Armeni, Ozan Sener, Amir R Za- mir, Helen Jiang, Ioannis Brilakis, Martin Fischer, and Sil- vio Savarese. 3d semantic parsing of large-scale indoor spaces. In Proceedings of the IEEE conference on com- puter vision and pattern recognition, pages 1534-1543, 2016.","title":"3d semantic parsing of large-scale indoor spaces","authors":"Iro Armeni, Ozan Sener, Helen Amir R Zamir, Ioannis Jiang, Martin Brilakis, Silvio Fischer, Savarese","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cheng et al., 2022] Bowen Cheng, Ishan Misra, Alexan- der G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pages 1290-1299, 2022.","title":"Masked-attention mask transformer for universal image segmentation","authors":"Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, Rohit Girdhar","year":"2022","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Dai et al., 2017] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5828- 5839, 2017.","title":"Scannet: Richly-annotated 3d reconstructions of indoor scenes","authors":"Angela Dai, X Angel, Manolis Chang, Maciej Savva, Thomas Halber, Matthias Funkhouser, Nießner","year":"2017","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Hu et al., 2020] Qingyong Hu, Bo Yang, Linhai Xie, Ste- fano Rosa, Yulan Guo, Zhihua Wang, Niki Trigoni, and Andrew Markham. Randla-net: Efficient semantic seg- mentation of large-scale point clouds. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11108-11117, 2020.","title":"Randla-net: Efficient semantic segmentation of large-scale point clouds","authors":"Qingyong Hu, Bo Yang, Linhai Xie, Stefano Rosa, Yulan Guo, Zhihua Wang, Niki Trigoni, Andrew Markham","year":"2020","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Huang et al., 2018] Qiangui Huang, Weiyue Wang, and Ul- rich Neumann. Recurrent slice networks for 3d segmen- tation of point clouds. In Proceedings of the IEEE con- ference on computer vision and pattern recognition, pages 2626-2635, 2018.","title":"Recurrent slice networks for 3d segmentation of point clouds","authors":"Huang","year":"2018","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Lai et al., 2022] Xin Lai, Jianhui Liu, Li Jiang, Liwei Wang, Hengshuang Zhao, Shu Liu, Xiaojuan Qi, and Jiaya Jia. Stratified transformer for 3d point cloud segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8500-8509, 2022.","title":"Stratified transformer for 3d point cloud segmentation","authors":"Xin Lai, Jianhui Liu, Li Jiang, Liwei Wang, Hengshuang Zhao, Shu Liu, Xiaojuan Qi, Jiaya Jia","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Landrieu and Simonovsky, 2018] Loic Landrieu and Martin Simonovsky. Large-scale point cloud semantic segmenta- tion with superpoint graphs. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4558-4567, 2018.","title":"Large-scale point cloud semantic segmentation with superpoint graphs","authors":"Simonovsky Landrieu, Loic Landrieu, Martin Simonovsky","year":"2018","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Li et al., 2018] Yangyan Li, Rui Bu, Mingchao Sun, Wei Wu, Xinhan Di, and Baoquan Chen. Pointcnn: Convo- lution on x-transformed points. Advances in neural infor- mation processing systems, 31, 2018.","title":"Pointcnn: Convolution on x-transformed points","authors":"Yangyan Li, Rui Bu, Mingchao Sun, Wei Wu, Xinhan Di, Baoquan Chen","year":"2018","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Long et al., 2015] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for seman- tic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3431- 3440, 2015.","title":"Fully convolutional networks for semantic segmentation","authors":"Long","year":"2015","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Luo et al., 2023] Naisong Luo, Yuwen Pan, Rui Sun, Tianzhu Zhang, Zhiwei Xiong, and Feng Wu. Camou- flaged instance segmentation via explicit de-camouflaging. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17918-17927, 2023.","title":"Camouflaged instance segmentation via explicit de-camouflaging","authors":"Luo","year":"2023","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Luo et al., 2024] Naisong Luo, Rui Sun, Yuwen Pan, Tianzhu Zhang, and Feng Wu. Electron microscopy im- ages as set of fragments for mitochondrial segmentation. In Proceedings of the AAAI Conference on Artificial Intel- ligence, 2024.","title":"Electron microscopy images as set of fragments for mitochondrial segmentation","authors":"Luo","year":"2024","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Mai et al., 2023] Huayu Mai, Rui Sun, Tianzhu Zhang, Zhi- wei Xiong, and Feng Wu. Dualrel: Semi-supervised mitochondria segmentation from a prototype perspec- tive. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19617- 19626, 2023.","title":"Dualrel: Semi-supervised mitochondria segmentation from a prototype perspective","authors":"Mai","year":"2023","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Mai et al., 2024a] Huayu Mai, Rui Sun, Yuan Wang, Tianzhu Zhang, and Feng Wu. Pay attention to target: Relation-aware temporal consistency for domain adaptive video semantic segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, 2024.","title":"Pay attention to target: Relation-aware temporal consistency for domain adaptive video semantic segmentation","authors":"Mai","year":"2024","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mai et al., 2024b] Huayu Mai, Rui Sun, Tianzhu Zhang, and Feng Wu. Rankmatch: Exploring the better con- sistency regularization for semi-supervised semantic seg- mentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.","title":"Rankmatch: Exploring the better consistency regularization for semi-supervised semantic segmentation","authors":"Mai","year":"2024","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ning et al., 2023] Zhenhua Ning, Zhuotao Tian, Guang- ming Lu, and Wenjie Pei. Boosting few-shot 3d point cloud segmentation via query-guided enhancement. In Proceedings of the 31st ACM International Conference on Multimedia, pages 1895-1904, 2023.","title":"Boosting few-shot 3d point cloud segmentation via query-guided enhancement","authors":"Zhenhua Ning, Zhuotao Tian, Guangming Lu, Wenjie Pei","year":"2023","venue":"Proceedings of the 31st ACM International Conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Pan et al., 2023] Yuwen Pan, Naisong Luo, Rui Sun, Meng Meng, Tianzhu Zhang, Zhiwei Xiong, and Yongdong Zhang. Adaptive template transformer for mitochondria segmentation in electron microscopy images. In Proceed- ings of the IEEE/CVF International Conference on Com- puter Vision, pages 21474-21484, 2023.","title":"Adaptive template transformer for mitochondria segmentation in electron microscopy images","authors":"Yuwen Pan, Naisong Luo, Rui Sun, Meng Meng, Tianzhu Zhang, Zhiwei Xiong, Yongdong Zhang","year":"2023","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Qi et al., 2017a] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652-660, 2017.","title":"Pointnet: Deep learning on point sets for 3d classification and segmentation","authors":"Qi","year":"2017","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Qi et al., 2017b] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical fea- ture learning on point sets in a metric space. Advances in neural information processing systems, 30, 2017.","title":"Pointnet++: Deep hierarchical feature learning on point sets in a metric space","authors":"Qi","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sun et al., 2021] Rui Sun, Yihao Li, Tianzhu Zhang, Zhen- dong Mao, Feng Wu, and Yongdong Zhang. Lesion-aware transformers for diabetic retinopathy grading. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10938-10947, 2021.","title":"Lesion-aware transformers for diabetic retinopathy grading","authors":"Rui Sun, Yihao Li, Tianzhu Zhang, Zhendong Mao, Feng Wu, Yongdong Zhang","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Sun et al., 2023a] Rui Sun, Naisong Luo, Yuwen Pan, Huayu Mai, Tianzhu Zhang, Zhiwei Xiong, and Feng Wu. Appearance prompt vision transformer for connec- tome reconstruction. In Edith Elkind, editor, Proceedings of the Thirty-Second International Joint Conference on Ar- tificial Intelligence, IJCAI-23, pages 1423-1431. Interna- tional Joint Conferences on Artificial Intelligence Organi- zation, 8 2023. Main Track.","title":"Appearance prompt vision transformer for connectome reconstruction","authors":"Sun","year":"2023","venue":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI-23","doi":null,"url":null,"internal_url":"/proceedings/2023/158"},{"position":21,"raw":"[Sun et al., 2023b] Rui Sun, Huayu Mai, Naisong Luo, Tianzhu Zhang, Zhiwei Xiong, and Feng Wu. Structure- decoupled adaptive part alignment network for domain adaptive mitochondria segmentation. In International Conference on Medical Image Computing and Computer- Assisted Intervention, pages 523-533. Springer, 2023.","title":"Structuredecoupled adaptive part alignment network for domain adaptive mitochondria segmentation","authors":"Sun","year":"2023","venue":"International Conference on Medical Image Computing and Computer-Assisted Intervention","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sun et al., 2023c] Rui Sun, Huayu Mai, Tianzhu Zhang, and Feng Wu. Daw: Exploring the better weighting function for semi-supervised semantic segmentation. In Advances in Neural Information Processing Systems, 2023.","title":"Daw: Exploring the better weighting function for semi-supervised semantic segmentation","authors":"Sun","year":"2023","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Sun et al., 2023d] Rui Sun, Yuan Wang, Huayu Mai, Tianzhu Zhang, and Feng Wu. Alignment before aggrega- tion: trajectory memory retrieval network for video object segmentation. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pages 1218-1228, 2023.","title":"Alignment before aggregation: trajectory memory retrieval network for video object segmentation","authors":"Sun","year":"2023","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Vinyals et al., 2016] Oriol Vinyals, Charles Blundell, Timo- thy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in neural information pro- cessing systems, 29, 2016.","title":"Matching networks for one shot learning","authors":"Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra","year":"2016","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wang et al., 2019] Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph cnn for learning on point clouds. ACM Transactions on Graphics (tog), 38(5):1-12, 2019.","title":"Dynamic graph cnn for learning on point clouds","authors":"Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, Justin M Solomon","year":"2019","venue":"ACM Transactions on Graphics (tog)","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wang et al., 2022] Yuan Wang, Rui Sun, Zhe Zhang, and Tianzhu Zhang. Adaptive agent transformer for few-shot segmentation. In European Conference on Computer Vi- sion, pages 36-52. Springer, 2022.","title":"Adaptive agent transformer for few-shot segmentation","authors":"Yuan Wang, Rui Sun, Zhe Zhang, Tianzhu Zhang","year":"2022","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wang et al., 2023a] Yuan Wang, Naisong Luo, and Tianzhu Zhang. Focus on query: Adversarial mining transformer for few-shot segmentation. In Advances in Neural Infor- mation Processing Systems, 2023.","title":"Focus on query: Adversarial mining transformer for few-shot segmentation","authors":"Wang","year":"2023","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wang et al., 2023b] Yuan Wang, Rui Sun, and Tianzhu Zhang. Rethinking the correlation in few-shot segmen- tation: A buoys view. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7183-7192, 2023.","title":"Rethinking the correlation in few-shot segmentation: A buoys view","authors":"Wang","year":"2023","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wang et al., 2024] Yuan Wang, Rui Sun, Naisong Luo, Yuwen Pan, and Tianzhu Zhang. Image-to-image match- ing via foundation models: A new perspective for open- vocabulary semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.","title":"Image-to-image matching via foundation models: A new perspective for openvocabulary semantic segmentation","authors":"Yuan Wang, Rui Sun, Naisong Luo, Yuwen Pan, Tianzhu Zhang","year":"2024","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wu et al., 2019] Wenxuan Wu, Zhongang Qi, and Li Fuxin. Pointconv: Deep convolutional networks on 3d point clouds. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pages 9621- 9630, 2019.","title":"Pointconv: Deep convolutional networks on 3d point clouds","authors":"Wu","year":"2019","venue":"Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Xiong et al., 2024] Guoxin Xiong, Meng Meng, Tianzhu Zhang, Dongming Zhang, and Yongdong Zhang. Reference-aware adaptive network for image-text match- ing. IEEE Transactions on Circuits and Systems for Video Technology, pages 1-1, 2024.","title":"Reference-aware adaptive network for image-text matching","authors":"Guoxin Xiong, Meng Meng, Tianzhu Zhang, Dongming Zhang, Yongdong Zhang","year":"2024","venue":"IEEE Transactions on Circuits and Systems for Video Technology","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Ye et al., 2018] Xiaoqing Ye, Jiamao Li, Hexiao Huang, Liang Du, and Xiaolin Zhang. 3d recurrent neural net- works with context fusion for point cloud semantic seg- mentation. In Proceedings of the European conference on computer vision (ECCV), pages 403-417, 2018.","title":"3d recurrent neural networks with context fusion for point cloud semantic segmentation","authors":"Xiaoqing Ye, Jiamao Li, Hexiao Huang, Liang Du, Xiaolin Zhang","year":"2018","venue":"Proceedings of the European conference on computer vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhang et al., 2023] Qieshi Zhang, Tichao Wang, Fusheng Hao, Fuxiang Wu, and Jun Cheng. Prototype expansion and feature calibration for few-shot point cloud semantic segmentation. Neurocomputing, 558:126732, 2023.","title":"Prototype expansion and feature calibration for few-shot point cloud semantic segmentation","authors":"Zhang","year":"2023","venue":"Neurocomputing","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zhao et al., 2021a] Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr, and Vladlen Koltun. Point transformer. In Proceedings of the IEEE/CVF international conference on computer vision, pages 16259-16268, 2021.","title":"Point transformer","authors":"Zhao","year":"2021","venue":"Proceedings of the IEEE/CVF international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Zhao et al., 2021b] Na Zhao, Tat-Seng Chua, and Gim Hee Lee. Few-shot 3d point cloud semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8873-8882, 2021.","title":"Few-shot 3d point cloud semantic segmentation","authors":"Zhao","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Zhu et al., 2023] Guanyu Zhu, Yong Zhou, Rui Yao, and Hancheng Zhu. Cross-class bias rectification for point cloud few-shot segmentation. IEEE Transactions on Mul- timedia, 2023.","title":"Cross-class bias rectification for point cloud few-shot segmentation","authors":"Guanyu Zhu, Yong Zhou, Rui Yao, Hancheng Zhu","year":"2023","venue":"IEEE Transactions on Multimedia","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}