{"type":"ScholarlyArticle","title":"Open-Vocabulary Fine-Grained Hand Action Detection","authors":[{"name":"Ting Zhe","slug":"ting-zhe","given":"Ting","family":"Zhe","affiliation":""},{"name":"Mengya Han","slug":"mengya-han","given":"Mengya","family":"Han","affiliation":""},{"name":"Xiaoshuai Hao","slug":"xiaoshuai-hao","given":"Xiaoshuai","family":"Hao","affiliation":""},{"name":"Yong Luo","slug":"yong-luo","given":"Yong","family":"Luo","affiliation":""},{"name":"Zheng He","slug":"zheng-he","given":"Zheng","family":"He","affiliation":""},{"name":"Xiantao Cai","slug":"xiantao-cai","given":"Xiantao","family":"Cai","affiliation":""},{"name":"Jing Zhang","slug":"jing-zhang","given":"Jing","family":"Zhang","affiliation":""}],"abstract":"In this work, we address the new challenge of open-vocabulary fine-grained hand action detection, which aims to recognize hand actions from both known and novel categories using textual descriptions. Traditional hand action detection methods are limited to closed-set detection, making it difficult for them to generalize to new, unseen hand action categories. While current open-vocabulary detection (OVD) methods are effective at detecting novel objects, they face challenges with fine-grained action recognition, particularly when data is limited and heterogeneous. This often leads to poor generalization and performance bias between base and novel categories. To address these issues, we propose a novel approach, Open-FGHA (Open-vocabulary Fine-Grained Hand Action), which learns to distinguish fine-grained features across multiple modalities from limited heterogeneous data. It then identifies optimal matching relationships among these features, enabling accurate open-vocabulary fine-grained hand action detection. Specifically, we introduce three key components: Hierarchical Heterogeneous Low-Rank Adaptation, Bidirectional Selection and Fusion Mechanism, and Cross-Modality Query Generator. These components work in unison to enhance the alignment and fusion of multimodal fine-grained features. Extensive experiments demonstrate that Open-FGHA outperforms existing OVD methods, showing its strong potential for open-vocabulary hand action detection. The source code is available at OV-FGHAD.","doi":"10.24963/ijcai.2025/276","url":"https://ijcai.org/proceedings/2025/276","pdf":"https://ijcai.org/proceedings/2025/0276.pdf","markdown":"https://ijcai.org/proceedings/2025/276.md","bibtex":"https://ijcai.org/proceedings/2025/bibtex/276","volume":{"slug":"2025","title":"Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2025","url":"https://ijcai.org/proceedings/2025/","year":2025,"location":"Montreal, Canada","dates":"16-22 August 2025","editors":"James Kwok","isbn":"978-1-956792-06-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2025"},"section":"Computer Vision","track":"Main Track","number":276,"pages":{"first":2476,"last":2484},"keywords":["Computer Vision: CV: Action and behavior recognition","Computer Vision: CV: Recognition (object detection, categorization)"],"videos":[],"references":[],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}