RealDex: Towards Human-like Grasping for Robotic Dexterous Hand

1 Introduction

Compared to standard parallel grippers with 7 degrees of freedom (DoF) [Fang et al., 2022; Fang et al., 2020; Dai et al., 2023], five-fingered dexterous hands [Sha, 2005] boasting over 24 DoF closely mimic human hand structure and movement. This sophistication enhances their ability to adapt to human-centered environments and execute more intricate and nuanced operations, such as using complex tools, with greater ease. Furthermore, robots that mirror human appearance and behavior tend to be more quickly embraced and comprehended by people, making them particularly advantageous in service and healthcare sectors.

Dexterous grasping represents an initial step in the interaction between the dexterous hand and the real world, forming the foundation for research and application of human-like service robots. Due to the high DoF of dexterous hand, traditional motion planning-based methods [Andrews and Kry, 2013; Bai and Liu, 2014; Dogar and Srinivasa, 2010] struggle to handle such complex hand joint movements. In the absence of real-world data, prevalent methodologies pivot towards reinforcement learning (RL) for training dexterous hand grasping behaviors. Considering real-world applications, some of them [Chen et al., 2022; Qin et al., 2022; Wan et al., 2023] have started using raw visual data, such as image or point cloud, as inputs to directly train grasping strategies for dexterous hand. However, reward functions in RL are artificially devised and cannot encompass all desired outcomes. In particular, it is difficult to model human behavioral habits in the reward mechanism. As a result, RLtrained dexterous hand grasping behaviors are only partially physically plausible and fail to truly mimic human-object interaction habits, not fully harnessing the inherent potential of humanoid robots. For example, people usually pick up a cup by its handle instead of sticking their thumb inside it, even though both ways can lift the cup physiccally. Without modeling human motion priors, current methods face notable challenges in replicating human-like grasping motions for robotic dexterous hands.

To overcome the challenges posed by data scarcity, some works [Wei et al., 2022; Li et al., 2023; Wang et al., 2023; Liu et al., 2020] have proposed synthesized dexterous grasping pose datasets. Models trained on these datasets with the supervision of ground truth can generate improved goal poses for dexterous grasping. However, these datasets, derived from simulation environments via optimization methods, rely on artificially defined energy functions and necessitate detailed physical and geometric information about objects, which are impractical for real-world applications. Furthermore, the substantial disparity between synthesized and real data makes the trained algorithms exhibit poor real-world applicability and limited generalization abilities. Additionally, the inability of simulated data to effectively model human-like behavior results in grasping poses devoid of human behavioral insights.

The teleoperation system of ShadowHand [Tel, 2022] with integrated signal synchronization and control mechanisms enables the alignment of poses between the dexterous hand and human hand in real time. Utilizing this system, we have created a comprehensive real-world dataset, RealDex, for dexterous hand grasping, ensuring the actions closely mirror human grasping patterns. Moreover, we have established a multi-view vision system with multiple RGB-D cameras to support research in vision-based grasping algorithms. RealDex contains 52 objects with varied scales, shapes, and materials, 2.6k sequences of grasping motions with diverse initial positions and directions, and around 955k frames of visual data, including images and point clouds. This groundbreaking dataset, driven by human hands and gathered in real-world settings, is significant for robotic dexterous handobject grasping. It stands apart from previous datasets in three critical ways: it replicates human hand-object interactions, enhancing the training of dexterous hands to more accurately emulate human movements, thereby advancing humanoid robotics; its precise ground truth data for real dexterous hand motions bridges the gap between model training and real application; and its multi-view, multimodal visual data pave the way for the development of diverse vision-based manipulation algorithms, fostering automated perception, cognition, and action of dexterous hands in real worlds.

Leveraging the RealDex dataset, we have developed a novel and effective method for dexterous grasping, which involves two key stages: grasp pose generation and motion synthesis, using only the point cloud data of objects captured by real vision sensor as input. Thanks to the dataset's rich representation of human-like dexterous hand motions, our model can innately learn human behavior patterns, producing practical grasping motions. Recognizing the robust generalization capabilities of Multimodal Large Language Models (MLLMs) [Google, 2023; OpenAI, 2023], we have integrated an MLLM Selection Module. This module is adept at choosing the most natural, physically plausible, and humanlike grasp pose from multiple generated options, enhancing the final motion synthesis. The inherent general knowledge within large models bolsters our method's generalization capacity, ensuring effective performance even with unseen objects and thus significantly benefiting practical applications. We have conducted extensive experiments on RealDex and other open datasets, including dexterous grasping dataset and human grasping dataset, and also have tested our method on real robot hand. Quantitative and qualitative results demonstrate that our method outperforms others obviously for generation human-like practicable dexterous grasping motions.

2 RelatedWork

2.1 Dataset for Dexterous Hand Grasping

The complexity of dexterous hands, while allowing for precise operations, poses challenges in annotating grasping motions. Many datasets [Liu et al., 2020; Hasson et al., 2019a; Lundell et al., 2021; Goldfeder et al., 2009] use the planner [Miller and Allen, 2004] to generate grasping poses. However, this method's reliance on a restricted eigengrasp space limits the diversity of the data, failing to capture the full range of a multi-fingered hand's dexterity. Some datasets try to improve the quality and diversity of the grasp poses. DVGG [Wei et al., 2022] utilizes MuJoCo physics simulator [Todorov et al., 2012] to synthesize data and filtering out unstable grasp poses through a hand-shaking test. DexGraspNet [Wang et al., 2023] leverages a deeply accelerated differentiable force closure estimator and synthesize stable and diverse grasp poses on a large scale. MultiDex [Li et al., 2023] synthesizes versatile dexterous grasp poses across five different robot hands. Nevertheless, existing datasets for dexterous grasping consist solely of synthesized data created through optimization, resulting in a significant disconnect with real robotic hand data and the absence of human-like behavior modeling. It is noted that these datasets only contain the static grasp poses and without any dynamic grasping motion sequences. Although many datasets [Yang et al., 2022; Chao, 2021; Taheri et al., 2020] focusing on human-object interaction have emerged recently, the discrepancy between the representations of robot hand and human hand make these datasets difficult to be directly used for dexterous hand manipulation.

2.2 Dexterous Grasping Generation

Traditional dexterous grasping [Andrews and Kry, 2013; Bai and Liu, 2014; Dogar and Srinivasa, 2010] usually leverage analytical methods to model the kinematics and dynamics of both hands and objects, performing limited for high-DoF robot hands and complex objects. Current mainstream method for dexterous grasping is utilizing reinforcement learning (RL) for training dexterous hand grasping behaviors. Many algorithms [Andrychowicz et al., 2020; Christen et al., 2022; Huang et al., 2021; She et al., 2022; Wu et al., 2023] in this sphere demand exact object geometric and pose data, markedly curtailing their practical applicability. To better apply for real-world scenarios, some methods [Chen et al., 2022; Qin et al., 2022; Wan et al., 2023] have begun leveraging raw visual data, like images or point clouds, to train dexterous hand grasping motions. Yet, the reward functions in RL are artificially designed and fail to model human motion patterns, making RL-based method challenging to achieve human-like grasping motions. Because our dataset provides accurate ground truth of human-like dexterous grasping motions, inspired by human motion generation methods [Taheri et al., 2022; Wu et al., 2022], it is more appropriate to utilize supervised methods to guide the learning process.

3 RealDex Dataset

To advance research and practical applications of human-like robotic dexterous grasping in real-world environments, we introduce RealDex, the first extensive dataset that captures real dexterous hand grasping motions incorporating human behavioral patterns. RealDex features a diverse collection of 52 objects varying in scale, shape, and material, along with 2.6k sequences of grasping motions involving different initial object positions and grasping directions, and approximately 955k frames of raw visual data, including images and point clouds. We use the commercial 3D scanner, the EinScanPro+ [3D, 2018], to reconstruct detailed 3D mesh models for collected objects. We have divided our dataset into training, validation, and test sets, ensuring that each object appears exclusively in one of these subsets. The three sets contain 2114 grasping motion sequences for 40 objects, 245 grasping motions for 6 objects, and 271 grasping motions for 6 objects, respectively. Detailed information about the division will be provided in the supplementary materials.

We will delve into the specifics of RealDex, covering aspects such as the hardware setup of the capture system in Section 3.1, the calibration and synchronization of all sensors in Section 3.2, and the characteristics of RealDex in Section 3.3.

3.1 Hardware System Setup

To fulfill the requirements of high-quality data collection and effective real-world application deployment, we have established a sophisticated dexterous system comprising three key components: a vision capture system designed to acquire multi-view and multimodal visual data, a manipulation system equipped with a robotic arm and a five-fingered robtic hand, and a teleoperation system that facilitates the control of the dexterous hand to emulate human-like movements. Them communicate via Robot Operation System (ROS) network.

Vision Capture System. Much like human eyes, vision sensors are crucial for robots to perceive and understand their surroundings, thereby enabling appropriate action. Imagebased and point cloud-based deep learning methods [Qi et al., 2017a; Qi et al., 2017b; Xu et al., 2023b; Zhu et al., 2021; Wang et al., 2021] have achieved significant breakthroughs in scene understanding, propelling the advancement of robotic perception systems. To support the practical deployment of dexterous hands, we have established a vision system equipped with four Azure Kinect RGB-D cameras with 15Hz capture frequency. These cameras are strategically positioned at four different locations, offering a quartet of perspectives focused on the operating desk, as illustrated in Figure 2. This setup provides RGB images and point clouds from multiple angles, not only aiding in delivering precise object pose annotations for our datasets, but also enabling research and practical applications of vision-based dexterous grasping under various visual configurations, including monocular-based, multiview-based, point-based, multimodal-based approaches, etc.

Dexterous Manipulation System. The dexterous manipulation system is designed to execute grasping operations akin to human actions. It features a UR10e robotic arm with 6 DoF and a tendon-driven Shadow Right Hand robot [Sha, 2005]. This robot hand is equipped with 20 motors, enabling adduction and abduction movements across 24 degrees of freedom. Additionally, the dexterous hand is outfitted with over 100 sensors operating at up to 1 kHz.

Teleoperation System. For the teleoperation system [Tel, 2022], users wear a lightweight glove to intuitively control the Shadow Hand and robotic arm, allowing them to mimic natural human movements in real time. Human hand poses are seamlessly transmitted to the robot hand through the glove, vision tracking devices, and WiFi services. This system is performed by well-trained individuals to collect data on human-like dexterous grasping motions.

3.2 Calibration and Synchronization

The integral dexterous system, with diverse sensors and communication networks, is inherently complex. Precise calibration and synchronization are vital to ensure seamless coordination among all subsystems for executing fine-grained operations and capturing accurate data. We introduce the most important camera-camera and robot-camera calibration and synchronization in this section.

Camera-camera Calibration. Camera intrinsics comes from manufacturer, while camera extrinsics are carefully measured through a two-stage process. Firstly, we use AprilTag [Krogius et al., 2019] to roughly calculate extrinsics for cameras. Then, we apply the Iterative Closet Point method (ICP) [Besl and McKay, 1992; stytim, 2023] between two neighbouring cameras to get more accurate alignment among point clouds from different views.

Camera-camera Synchronization. TheKinect depth camera measures depth by timing the travel of emitted infrared light to objects and back. Interference from simultaneous infrared emissions by multiple cameras leads to depth inaccuracies, which we address by staggering camera sampling times and synchronizing auxiliary camera data streams with the main camera using aligned timestamps.

Robot-camera Calibration. To synchronize the robotic hand's motion with the vision system, we use hand-eye calibration [IFL-CAMP, 2021]. This process starts with the Shadow Hand holding a calibration board, followed by moving it to over 30 locations for initial static transformation. We then refine this alignment by applying ICP method [Besl and McKay, 1992] between the robot's mesh vertices and the main camera's point cloud. This thorough calibration, as shown in Figure 3, ensures accurate alignment between the robotic and vision systems, enhancing data precision.

Robot-camera Synchronization. The robot's timestamp, derived from the ROS TF tree, continuously updates the pose of each dexterous hand joint. To ensure precise time alignment, we match the initial camera frame's point cloud with the robot's mesh, reconstructed for each timestamp within a proximate time window. We then identify the robot frame most similar to the point cloud for synchronization. With this initial aligned timestamp established, we can extrapolate and align the remaining timestamps, ensuring a consistent and accurate correlation across the entire data set.

3.3 Characteristics

Robotic grasping has been extensively studied for decades, serving as an essential skill for agents to engage with their surroundings and conduct manipulation tasks. Some datasets focusing on dexterous grasping, including DVGG [Wei et al., 2022], MultiDex [Li et al., 2023], DexGraspNet [Wang et al., 2023], and DDGdata [Liu et al., 2020], have already been proposed. However, our dataset, RealDex, stands apart from these existing datasets in terms of its capture process, data types, annotations, and task orientation. We draw comparisons in Table 1 and highlight three key novel features of RealDex in the following discussion.

Human-likeGraspingMotion. In the data capture process of our dataset, the dexterous hand is operated via a teleoperation system directly controlled by a human hand. This unique approach ensures that all grasping motions in RealDex authentically mirror human behavioral patterns. In contrast, previous datasets only have static grasp poses derived from physical optimization, lacking dynamic motion sequences and any real human behavioral insights. The inclusion of realistic, human-like motion data in RealDex is profoundly significant for advancing intelligent dexterous hand development, imbued with human habits. This dataset holds the potential to propel the evolution of humanoid robotics.

Real Robotic Dexterous Poses. Using a real dexterous robotic hand for data collection, our RealDex dataset accurately captures the ground truth of dexterous poses in every frame. This allows for the direct and effective application of models trained on RealDex in real-world scenarios. In contrast, previous datasets with synthesized grasp poses exhibit a substantial disparity from real-world data, leading to performance gap in translating models trained on such data into practical applications.

Rich Real Visual Data. We have set up a vision capture system to obtain multi-view and multimodal visual data. This wealth of information is dexterous grasping for advancing research in vision-based dexterous grasping methods, which are typically more applicable and effective in real-world scenarios. The lack of genuine visual data in previous datasets significantly limits their practical utility.

4 Methodology

A dexterous hand is designed to execute motions that closely resemble those performed by human hands and interactions with objects, necessitating the use of flexible strategies in different external conditions, similar to how a human would do. For example, the way in which humans hold a cup can vary depending on several factors, including whether there is a handle, the contents of the cup, and the temperature of the liquid inside. In this section, we propose a unified framework for generating complete dexterous hand motions to grasp objects, closely aligning with the human experience in various environmental settings, as depicted in Figure 4. Our framework primarily consists of two parts: grasping pose generation in Section 4.2 and pose-guided motion synthesis in Section 4.3. The former involves generating grasping poses for the input objects by using a conditional Variational Autoencoder (cVAE) to generate candidate poses and aligning them with human preferences through Multi-modal Large Language Models (MLLMs). The latter is responsible for predicting complete hand motion sequences for each pose through auto-regressive motion trajectory prediction.

4.1 ProblemFormulation

Dexterous Hand Representation. A dexterous hand is driven by revolute joints that only allow rotation around a fixed single axis. Hence, the pose of a dexterous hand is represented by the joint angle configuration, denoted as θ ∈ R22 . The global translation and orientation of the hand are represented as η ∈ R6 , where the orientation is represented in angle-axis form. The pose of a robotic hand ϕ = (θ, η) ∈ R28 is the combination of joint angle configuration and global 6D pose of hand. The mesh representation of a dexterous hand can be generated by employing forward kinematics to calculate the global transformation for each segment. To represent the shape of hand mesh, a set of points is sampled, which we denote as Ph .

Problem Formulation. Given an observed object point cloud Po , we aim to generate the preferred grasping poses {ϕk }Kk=1 and synthesize the corresponding motion sequence Φ = {ϕkt }Tt=1 of dexterous hand to approach and grasp object for each grasping pose ϕk .

4.2 Grasping Pose Generation

Pose Candidate Generation. Inspired by the work in human grasping generation [Hasson et al., 2019b; Jiang et al., 2021], we use a contact-aware generative model for dexterous grasp generation. This model uses a module to predict the contact map and a cVAE to learn a latent embedding grasping space. By sampling from the Gaussian variational distribution, the decoder of the cVAE is capable of generating a diverse array of candidate grasping poses {ϕk }Nk=1 for any input object point cloud. Subsequently, these poses undergo optimization to align with the predicted contact map, ensuring they are feasible. We then filter out unstable grasps that have insufficient contact with the object, and a human-like selection module is utilized to determine the optimal target poses {ϕk }Kk=1 for execution, where K < N.

Human-like Pose Selection. Despite the acquisition of a diverse range of candidate grasping poses through consideration of the interaction between the object and the hand, this set of candidate poses fails to account for human grasping preferences and priors. Since MLLMs [Google, 2023; OpenAI, 2023] have already demonstrated their capability to encode rich world knowledge, we propose using MLLMs to discern the most natural, physically plausible, and humanlike grasp poses from the candidate poses, thereby aligning it with human experience. Specifically, we begin by acquiring mesh representations of both the robotic hand and the object for each candidate grasping pose, followed by rendering the image depicting their interaction. Next, we prompt Gemini [Google, 2023] using the specific prompt shown in Figure 4. The prompt emphasizes a comprehensive understanding of the grasping posture, highlighting the nuanced interactions between the fingers and the object. Gemini scores the rendered image, which contains intricate hand-object interaction details, in terms of naturalness, physical plausibility, human-likeness, and preference in order to select the target poses {ϕk }Kk=1 .

4.3 Pose-guided HandMotion Synthesis

The synthesis of intermediate motion for the human hand in an auto-regressive manner, given a starting pose ϕ0 and a target pose ϕtgt ∈ {ϕk }Kk=1 , has been studied in previous works [Zhang et al., 2021; Taheri et al., 2022]. Like them, we also employ an autoregressive approach, namely MotionNet, to learn how to predict future movements based on past trajectories and the current state to reach the target object. However, unlike parameterized human hand models such as MANO [Romero et al., 2022], which estimate joint positions via skinning, a robotic hand's joints link its components with fixed articulatory relationships. By strengthening the modeling of the interdependence among these joints, we can improve dexterous grasping. Therefore, we encode the joints' coordinates and utilize self-attention to model their spatial relationships.

Specifically, for each time frame t, let J PE t denote the sinusoidal positional encoding to the joints' coordinates. The encoded joint positions are given by

where the query Q and the key-value pair (K, V ) are all positional code J PE t , and Attn means the self-attention mechanism [Vaswani et al., 2017].

Next, we utilize the hand pose information from the previous five timesteps in conjunction with the current timestep's hand and object point cloud features to forecast the hand poses for the subsequent ten timesteps following [Taheri et al., 2022]. At timestep t, the static state of a dexterous hand is represented by hand poses ϕt, sampled points on the hand's mesh Pht , and the joint feature F J t . Joint features and hand poses from the preceding five timesteps, denoted as F J t-5:t and ϕt-5:t respectively, collectively characterize the motion trajectory of the hand. The velocity of hand points at time t, represented by ˙Pht is instrumental since the velocity is directly correlated with the motion in subsequent frames. Let F h tgt represent the global feature of the target hand points. It is a guidance signal to encourage motion progress towards the predefined target. However, this guidance, when provided in isolation, is incomplete as it does not account for the current state of motion. To address this, we compute the displacement of the hand points from the current frame to the target frame, denoted by dht , which serves to quantify the spatial difference and provide more contextually relevant guidance toward the target. In the end, the input of MotionNet is given by:

The MotionNet integrates a gating mechanism to encode the motion phase and utilizes MLP layers to predict the change of poses [Taheri et al., 2022; Starke et al., 2019]. The change can capture the temporal dynamic of movement. Representing the changes of parameters relative to current frame t as ∆, the output MotionNet produce is:

whereMout is used to reconstruct future motion and compute the input in the next timestep.

5 Experiments

We developed and executed our algorithm using Python with PyTorch framework. Models were both trained and tested on an Ubuntu server, equipped with eight NVIDIA GeForce RTX 3090 GPU cards. We leveraged Gemini [Google, 2023] for our MLLM selection module. More details for the training process, inference process, loss functions are introduced in the supplementary material. In the following, we first show the comparision results for vision-based grasping methods, and then evaluate each stage of our method in detail. Especially, test result on real robot is also demonstrated.

5.1 Comparison for Dexterous Grasping

With the same point cloud data as input, we generate grasping motion sequences with different methods for the object in the test set and visualize these motions in videos. We carry out a user study to evaluate these motions based on their humanoid characteristics, grasping stability, and hand-object interaction quality, etc. The user score for each method is derived by averaging the ratings across all samples, where rank 1 gets 3 points, rank 2 gets 2 points, and rank 3 gets 1 point. A higher score indicates better performance. We have 40 users involved. Due to a prior shortage of real-world robotic hand grasping data, no supervised models exist for fair comparison. Therefore, we adapt methods, SAGA [Wu et al., 2021] and GOAL [Taheri et al., 2022], designed for human grasp generation to produce motions for robotic dexterous hands. In addition, to verify the generalization capability of our method, we also conduct evaluation on human hand grasping dataset GRAB [Taheri et al., 2020]. Results in Table 2 shows that our method outperforms others obviously.

5.2 Evaluation for Grasping Pose Generation

The self-intersection of robot hand and the intersection between hand and object are physically implausible, which are measured by the intersection volume (s.i.vol.) and the penetration distance (p.dist.). To measure the stability of grasping, we compute the simulation displacement (sim.disp.) following [Tzionas et al., 2015; Hasson et al., 2019b], which is defined as the average displacement of the object when the hand is stationary and the object is subjected to gravity. These metrics provide a quantitative basis for determining the realism and viability of the grasp. We generate 10 grasp poses for each object in the test set for the evaluation. In addition, we also use the user score by rendering images for randomly selected 3 grasp poses of each object for more comprehensive evaluation.

We compare our pose generation module with UniDexGrasp [Xu et al., 2023a], the latest work for dexterous grasping. UniDexGrasp uses conditional normalizing flows to learn the distribution of plausible grasp poses. We use its released code for experiments. We evaluate on GRAB [Taheri et al., 2020], DexGraspNet [Wang et al., 2023] and our dataset RealDex. DexGraspNet is a synthetic dataset including various objects and grasp poses for ShadowHand. As shown in Table 3, grasp poses produced by our method demonstrate the highest stability (sim.disp.) and achieves the highest user acceptance across all datasets. The ablation result (ours w/o MLLM) demonstrates the effectiveness of our MLLM selection module in taking advantage of more general knowledge in MLLM. Qualitative result is shown in Figure 5, where our result is more approaching human-like grasp poses, more natural, and more stable.

5.3 Evaluation for Motion Synthesis

To purely evaluate the motion sequence quality, we input the GT goal pose and sample the same number of steps with the GT motion for evaluation. The results are shown in Table 4. Mean Per-Joint Positional Error (MPJPE) and Average Variance Error (AVE) are calculated, quantifying the mean and variance errors in predicted joint positions compared to ground truth across all frames [Ghosh et al., 2021]. We also use hand mesh vertex offset and minimum distance to the object to measure the difference between final pose and the ground truth. Even with the same goal pose as the input, the motion synthesis stage in our method is still superior due to the consideration of the spatial relationship of joints.

5.4 Test on Real Robot

Because RealDex is collected by real robotic dexterous hand with accurate ground truth, our model trained on it can be directly tested on real robot, enabling fast application deployment. For motion sequences generated by our model, we send them to the simulator for safety testing before the robot execution. The qualified motion sequence is then encoded into a stamped trajectory and seamlessly integrated into the real Shadow Hand, as shown in Figure 6. The execution process from the initial hand pose to holding the object lasts about

20s. The generated human-like grasp and accompanying approaching motion exhibit a natural appearance. Throughout the entire process, the hand and the object maintain a relatively static and stable relationship, aligning with the essential requirements for successful grasping. More videos depicting entire robot grasping process are in the appendix.

6 Conclusion

In this work, we present RealDex, a groundbreaking dataset featuring genuine dexterous hand grasping motions, embedded with human behaviors and rich multi-view, multimodal visual data. Our dataset is instrumental for training dexterous hands in human-like movements, significantly enhancing humanoid robotics in real-world perception, cognition, and manipulation. We also introduce a novel dexterous grasping motion generation method using MLLM, enhancing model adaptability and real-world applicability. Extensive experiments and real robot testing show remarkable performance and practical value of our method and our dataset.

Acknowledgements

This work was supported by NSFC (No.62206173), MoE Key Laboratory of Intelligent Perception and Human-Machine Collaboration (ShanghaiTech University), Shanghai Frontiers Science Center of Human-centered Artificial Intelligence (ShangHAI), Shanghai Engineering Research Center of Intelligent Vision and Imaging.

Contribution Statement

• Yumeng Liu, Yaxun Yang and Youzhuo Wang contributed equally to this work.
• † Yuexin Ma supervised the project.