A3fford-HOI: Anatomy-Aligned Affordance Disentanglement for Fine-grained and Generalizable Hand Object Interaction

Abstract

Hand Object Interaction (HOI) generation provides an efficient solution for virtual reality simulation and embodied AI deployment.Recent studies have explored instruction-driven HOI synthesis, yet they overlooked the fine-grained interactive contact and struggled with robust generalization in data-scarce scenarios. In addition, existing datasets are frequently constrained with limited object categories and monotonous motion patterns. To address these limitations, we first integrate a large-scale HOI dataset, namely HOI-X, which contains diverse objects, rich motion patterns, and part-level contact annotations. Along with HOI-X, we propose A3fford-HOI, a fine-grained and generalizable Hand Object Interaction framework with Anatomy-Aligned Affordance Disentanglement. Specifically, we define a Disentangle-2-Interact paradigm in A3fford-HOI. Considering the various interactions of distinct hand anatomical structures,

in the Disentangle stage, we devise an Anatomy-Aligned Affordance Predictor, which leverages the external plausible physical priors in 3D multimodal large language models for affordance disentanglement prediction. Then, the disentangled affordance is employed into the Interact stage, i.e. Affordance Driven Interactor. The interactor integrates textual instructions, object geometric features, and the external knowledge enhanced (MLLMs) affordance via a Diffusion Transformer (DiT) for generalizable HOI generation. Extensive experiments demonstrate that A3fford-HOI maintains superior generation quality and contact plausibility for fine-grained and generalizable hand object interaction.