{"type":"ScholarlyArticle","title":"PointTFA: Training-Free Clustering Adaption for Large 3D Point Cloud Models","authors":[{"name":"Jinmeng Wu","slug":"jinmeng-wu","given":"Jinmeng","family":"Wu","affiliation":"School of Electrical and Information Engineering , Wuhan Institute of Technology , Wuhan , China"},{"name":"Chong Cao","slug":"chong-cao","given":"Chong","family":"Cao","affiliation":"School of Electrical and Information Engineering , Wuhan Institute of Technology , Wuhan , China"},{"name":"Hao Zhang","slug":"hao-zhang","given":"Hao","family":"Zhang","affiliation":"Center for Frontier AI Research (CFAR) , Agency for Science, Technology and Research (A*STAR)"},{"name":"Basura Fernando","slug":"basura-fernando","given":"Basura","family":"Fernando","affiliation":"Center for Frontier AI Research (CFAR) , Agency for Science, Technology and Research (A*STAR)"},{"name":"Yanbin Hao","slug":"yanbin-hao","given":"Yanbin","family":"Hao","affiliation":"University of Science and Technology of China"},{"name":"Hanyu Hong","slug":"hanyu-hong","given":"Hanyu","family":"Hong","affiliation":"School of Electrical and Information Engineering , Wuhan Institute of Technology , Wuhan , China"}],"abstract":"The success of contrastive learning models like CLIP, known for aligning 2D image-text pairs, has inspired the development of triplet alignment for Large 3D Point Cloud Models (3D-PCM). Examples like ULIP integrate images, text, and point clouds into a unified semantic space. However, despite showing impressive zero-shot capabilities, frozen 3D-PCM still falls short compared to fine-tuned methods, especially when downstream 3D datasets are significantly different from upstream data. Addressing this, we propose a Data-Efficient, Training-Free 3D Adaptation method named PointTFA that adjusts ULIP outputs with representative samples. PointTFA comprises the Representative Memory Cache (RMC) for selecting a representative support set, Cloud Query Refactor (CQR) for reconstructing a query cloud using the support set, and Training-Free 3D Adapter (3D-TFA) for inferring query categories from the support set. A key advantage of PointTFA is that it introduces no extra training parameters, yet outperforms vanilla frozen ULIP, closely approaching few-shot fine-tuning training methods in downstream cloud classification tasks like ModelNet10 & 40 and ScanObjectNN. The code is available at: https://github.com/CaoChong-git/PointTFA.","doi":"10.24963/ijcai.2024/159","url":"https://ijcai.org/proceedings/2024/159","pdf":"https://ijcai.org/proceedings/2024/0159.pdf","markdown":"https://ijcai.org/proceedings/2024/159.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/159","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Computer Vision","track":"Main Track","number":159,"pages":{"first":1434,"last":1442},"keywords":["Computer Vision: CV: 3D computer vision","Computer Vision: CV: Recognition (object detection, categorization)"],"videos":[],"references":[{"position":1,"raw":"[Alayrac et al., 2022] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Pro- cessing Systems, 35:23716-23736, 2022.","title":"Flamingo: a visual language model for few-shot learning","authors":"Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bellman, 1966] Richard Bellman. Dynamic programming. Science, 153(3731):34-37, 1966.","title":"Dynamic programming","authors":"Richard Bellman","year":"1966","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Brown et al., 2020] Tom Brown, Benjamin Mann, Nick Ry- der, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing sys- tems, 33:1877-1901, 2020.","title":"Language models are few-shot learners","authors":"Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell","year":"2020","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chang et al., 2015] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012, 2015.","title":"Shapenet: An information-rich 3d model repository","authors":"Thomas Angel X Chang, Leonidas Funkhouser, Pat Guibas, Qixing Hanrahan, Zimo Huang, Silvio Li, Manolis Savarese, Shuran Savva, Hao Song, Su","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Goyal et al., 2021] Ankit Goyal, Hei Law, Bowei Liu, Ale- jandro Newell, and Jia Deng. Revisiting point cloud shape classification with a simple and effective baseline. In Inter- national Conference on Machine Learning, pages 3809- 3820. PMLR, 2021.","title":"Revisiting point cloud shape classification with a simple and effective baseline","authors":"Ankit Goyal, Hei Law, Bowei Liu, Alejandro Newell, Jia Deng","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hu et al., 2020] Qingyong Hu, Bo Yang, Linhai Xie, Ste- fano Rosa, Yulan Guo, Zhihua Wang, Niki Trigoni, and Andrew Markham. Randla-net: Efficient semantic seg- mentation of large-scale point clouds. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11108-11117, 2020.","title":"Randla-net: Efficient semantic segmentation of large-scale point clouds","authors":"Qingyong Hu, Bo Yang, Linhai Xie, Stefano Rosa, Yulan Guo, Zhihua Wang, Niki Trigoni, Andrew Markham","year":"2020","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Huang et al., 2023] Tianyu Huang, Bowen Dong, Yunhan Yang, Xiaoshui Huang, Rynson WH Lau, Wanli Ouyang, and Wangmeng Zuo. Clip2point: Transfer clip to point cloud classification with image-depth pre-training. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pages 22157-22167, 2023.","title":"Clip2point: Transfer clip to point cloud classification with image-depth pre-training","authors":"Tianyu Huang, Bowen Dong, Yunhan Yang, Xiaoshui Huang, W H Rynson, Wanli Lau, Wangmeng Ouyang, Zuo","year":"2023","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Jiao et al., 2024] Yang Jiao, Zequn Jie, Shaoxiang Chen, Lechao Cheng, Jingjing Chen, Lin Ma, and Yu-Gang Jiang. Instance-aware multi-camera 3d object detection with structural priors mining and self-boosting learning. In Proceedings of the AAAI Conference on Artificial Intel- ligence, volume 38, pages 2598-2606, 2024.","title":"Instance-aware multi-camera 3d object detection with structural priors mining and self-boosting learning","authors":"Yang Jiao, Zequn Jie, Shaoxiang Chen, Lechao Cheng, Jingjing Chen, Lin Ma, Yu-Gang Jiang","year":"2024","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Khandelwal et al., 2019] Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Gen- eralization through memorization: Nearest neighbor lan- guage models. arXiv preprint arXiv:1911.00172, 2019.","title":"Generalization through memorization: Nearest neighbor language models","authors":"Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, Mike Lewis","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lei et al., 2023] Weixian Lei, Yixiao Ge, Jianfeng Zhang, Dylan Sun, Kun Yi, Ying Shan, and Mike Zheng Shou. Vit-lens: Towards omni-modal representations. arXiv preprint arXiv:2308.10185, 2023.","title":"Vit-lens: Towards omni-modal representations","authors":"Lei","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2021] Zhichao Li, Feng Wang, and Naiyan Wang. Lidar r-cnn: An efficient and universal 3d object detector. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7546-7555, 2021.","title":"Lidar r-cnn: An efficient and universal 3d object detector","authors":"Zhichao Li, Feng Wang, Naiyan Wang","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2024] Hao Li, Dingwen Zhang, Yalun Dai, Nian Liu, Lechao Cheng, Jingfeng Li, Jingdong Wang, and Junwei Han. Gp-nerf: Generalized perception nerf for context-aware 3d scene understanding. CVPR, 2024.","title":"Gp-nerf: Generalized perception nerf for context-aware 3d scene understanding","authors":"Hao Li, Dingwen Zhang, Yalun Dai, Nian Liu, Lechao Cheng, Jingfeng Li, Jingdong Wang, Junwei Han","year":"2024","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Liu et al., 2023] Minghua Liu, Ruoxi Shi, Kaiming Kuang, Yinhao Zhu, Xuanlin Li, Shizhong Han, Hong Cai, Fatih Porikli, and Hao Su. Openshape: Scaling up 3d shape representation towards open-world understanding, 2023.","title":"Openshape: Scaling up 3d shape representation towards open-world understanding","authors":"Minghua Liu, Ruoxi Shi, Kaiming Kuang, Yinhao Zhu, Xuanlin Li, Shizhong Han, Hong Cai, Fatih Porikli, Hao Su","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Ma et al., 2022] Xu Ma, Can Qin, Haoxuan You, Haoxi Ran, and Yun Fu. Rethinking network design and local ge- ometry in point cloud: A simple residual mlp framework. arXiv preprint arXiv:2202.07123, 2022.","title":"Rethinking network design and local geometry in point cloud: A simple residual mlp framework","authors":"Xu Ma, Can Qin, Haoxuan You, Yun Haoxi Ran, Fu","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Qi et al., 2017] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Advances in neu- ral information processing systems, 30, 2017.","title":"Pointnet++: Deep hierarchical feature learning on point sets in a metric space","authors":"Charles Ruizhongtai, Qi, Li Yi, Hao Su, Leonidas J Guibas","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Qi et al., 2023] Zekun Qi, Runpei Dong, Guofan Fan, Zheng Ge, Xiangyu Zhang, Kaisheng Ma, and Li Yi. Contrast with reconstruct: Contrastive 3d representation learning guided by generative pretraining. arXiv preprint arXiv:2302.02318, 2023.","title":"Contrast with reconstruct: Contrastive 3d representation learning guided by generative pretraining","authors":"Qi","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Qian et al., 2022] Guocheng Qian, Yuchen Li, Houwen Peng, Jinjie Mai, Hasan Hammoud, Mohamed Elhoseiny, and Bernard Ghanem. Pointnext: Revisiting pointnet++ with improved training and scaling strategies. Advances in Neural Information Processing Systems, 35:23192-23204, 2022.","title":"Pointnext: Revisiting pointnet++ with improved training and scaling strategies","authors":"Guocheng Qian, Yuchen Li, Houwen Peng, Jinjie Mai, Hasan Hammoud, Mohamed Elhoseiny, Bernard Ghanem","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Qiu et al., 2021] Shi Qiu, Saeed Anwar, and Nick Barnes. Geometric back-projection network for point cloud clas- sification. IEEE Transactions on Multimedia, 24:1943- 1955, 2021.","title":"Geometric back-projection network for point cloud classification","authors":"Saeed Shi Qiu, Nick Anwar, Barnes","year":"2021","venue":"IEEE Transactions on Multimedia","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Radford et al., 2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from nat- ural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.","title":"Learning transferable visual models from natural language supervision","authors":"Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark","year":"2021","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Song et al., 2023] Linxin Song, Jieyu Zhang, Lechao Cheng, Pengyuan Zhou, Tianyi Zhou, and Irene Li. Nlp- bench: Evaluating large language models on solving nlp problems. arXiv preprint arXiv:2309.15630, 2023.","title":"Nlpbench: Evaluating large language models on solving nlp problems","authors":"Song","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Uy et al., 2019] Mikaela Angelina Uy, Quang-Hieu Pham, Binh-Son Hua, Thanh Nguyen, and Sai-Kit Yeung. Revis- iting point cloud classification: A new benchmark dataset and classification model on real-world data. In Proceed- ings of the IEEE/CVF international conference on com- puter vision, pages 1588-1597, 2019.","title":"Revisiting point cloud classification: A new benchmark dataset and classification model on real-world data","authors":"Angelina Mikaela, Quang-Hieu Uy, Binh-Son Pham, Thanh Hua, Sai-Kit Nguyen, Yeung","year":"2019","venue":"Proceedings of the IEEE/CVF international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wortsman et al., 2022] Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al. Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7959-7971, 2022.","title":"Robust fine-tuning of zero-shot models","authors":"Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wu et al., 2015] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE conference on com- puter vision and pattern recognition, pages 1912-1920, 2015.","title":"3d shapenets: A deep representation for volumetric shapes","authors":"Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, Jianxiong Xiao","year":"2015","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wu et al., 2023] Jinmeng Wu, Tingting Mu, Jeyan Thiya- galingam, and John Y Goulermas. Memory-aware at- tentive control for community question answering with knowledge-based dual refinement. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 2023.","title":"Memory-aware attentive control for community question answering with knowledge-based dual refinement","authors":"Wu","year":"2023","venue":"IEEE Transactions on Systems, Man, and Cybernetics","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Xue et al., 2023a] Le Xue, Mingfei Gao, Chen Xing, Roberto Martín-Martín, Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos Niebles, and Silvio Savarese. Ulip: Learning a unified representation of language, images, and point clouds for 3d understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1179-1189, 2023.","title":"Ulip: Learning a unified representation of language, images, and point clouds for 3d understanding","authors":"Xue","year":"2023","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Xue et al., 2023b] Le Xue, Ning Yu, Shu Zhang, Junnan Li, Roberto Martín-Martín, Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos Niebles, and Silvio Savarese. Ulip-2: To- wards scalable multimodal pre-training for 3d understand- ing. arXiv preprint arXiv:2305.08275, 2023.","title":"Ulip-2: Towards scalable multimodal pre-training for 3d understanding","authors":"Xue","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Yu et al., 2022] Xumin Yu, Lulu Tang, Yongming Rao, Tiejun Huang, Jie Zhou, and Jiwen Lu. Point-bert: Pre- training 3d point cloud transformers with masked point modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19313- 19322, 2022.","title":"Point-bert: Pretraining 3d point cloud transformers with masked point modeling","authors":"Yu","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhang et al., 2022a] Renrui Zhang, Ziyu Guo, Wei Zhang, Kunchang Li, Xupeng Miao, Bin Cui, Yu Qiao, Peng Gao, and Hongsheng Li. Pointclip: Point cloud understanding by clip. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8552- 8562, 2022.","title":"Pointclip: Point cloud understanding by clip","authors":"Zhang","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhang et al., 2022b] Renrui Zhang, Wei Zhang, Rongyao Fang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-adapter: Training-free adaption of clip for few-shot classification. In European Conference on Computer Vision, pages 493-510. Springer, 2022.","title":"Tip-adapter: Training-free adaption of clip for few-shot classification","authors":"Zhang","year":"2022","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhang et al., 2023] Renrui Zhang, Liuhui Wang, Yali Wang, Peng Gao, Hongsheng Li, and Jianbo Shi. Parameter is not all you need: Starting from non-parametric networks for 3d point cloud analysis. arXiv preprint arXiv:2303.08134, 2023.","title":"Parameter is not all you need: Starting from non-parametric networks for 3d point cloud analysis","authors":"Zhang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhou et al., 2024] Pengyuan Zhou, Lin Wang, Zhi Liu, Yanbin Hao, Pan Hui, Sasu Tarkoma, and Jussi Kan- gasharju. A survey on generative ai and llm for video generation, understanding, and streaming. arXiv preprint arXiv:2404.16038, 2024.","title":"A survey on generative ai and llm for video generation, understanding, and streaming","authors":"Pengyuan Zhou, Lin Wang, Zhi Liu, Yanbin Hao, Pan Hui, Sasu Tarkoma, Jussi Kangasharju","year":"2024","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhu et al., 2023] Xiangyang Zhu, Renrui Zhang, Bowei He, Ziyu Guo, Ziyao Zeng, Zipeng Qin, Shanghang Zhang, and Peng Gao. Pointclip v2: Prompting clip and gpt for powerful 3d open-world learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2639-2650, 2023.","title":"Pointclip v2: Prompting clip and gpt for powerful 3d open-world learning","authors":"Zhu","year":"2023","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}