PECHC: Robust Tactile Grasping Stabilization in Vision-Denied Peripersonal Space

1 Introduction

Robust dexterous manipulation is a milestone in embodied intelligence. While vision-based methods excel [Billard and Kragic, 2019; Andrychowicz et al., 2020; Yang et al., 2024], they fail in extreme scenarios like occlusion, darkness, or transparent surfaces. In such vision-denied scenarios, tactile feedback serves as the last line of defense, providing the most direct and authentic physical observations of contact states [Liu et al., 2024; Li et al., 2023; Jia et al., 2025; Lin et al., 2025].

∗Corresponding author.

Thus, achieving tactile-driven grasp stabilization serves as a robust "Last-Centimeter" reflex policy [Kang et al., 2025]. Rather than replacing global navigation, it complements vision by handling occlusion-induced localization errors within the peripersonal workspace, where tactile feedback becomes the sole reliable modality [Chen et al., 2025b].

In recent years, Sim-to-Real techniques based on reinforcement learning (RL) have shown great potential in dexterous manipulation [Andrychowicz et al., 2020; Kumar et al., 2021].

However, Sim-to-Real for tactile-based systems faces more severe challenges than vision-the "Tactile Reality Gap", which encompasses two core challenges: 1) Force Adaptation: Real-world properties (e.g., mass, friction) are invisible and variable, risking slippage or damage; 2) Simulation Fidelity: Simulators struggle to model complex contact dynamics (e.g., soft-body deformation), causing policy failure due to distribution shifts. These two factors collectively make policies trained solely in simulation fragile and poorly adaptable when facing complex interactions in the real world [Niu et al., 2024; Chen et al., 2025a].

To address the above core challenges, this paper proposes the PECHC algorithm as an integrated solution to bridge the gaps between simulation and reality.

To bridge the physical gap, we draw inspiration from the privileged distillation paradigm [Kumar et al., 2021]. However, directly applying such methods to blind grasping faces a unique "Geometric Gap": unlike locomotion, a grasper cannot adapt to dynamics if it fails to establish contact due to pose uncertainty. Therefore, PECHC integrates Cascaded Constraint Scheduling (CCS) to first guide geometric exploration, serving as a prerequisite for the subsequent dynamic adaptation (THED). We observe that the Tactile Reality Gap comprises two distinct components: 1) Systemic Bias (e.g., global sensor sensitivity, simulation friction scaling offsets), which is consistent across tasks, and 2) Instancelevel Dynamics (e.g., specific object mass, compliance). While pure Domain Randomization struggles with the former, blindly adapting to everything risks overfitting. Abandoning the traditional full-time demonstration paradigm, Hybrid Correction Imitation Learning (HCIL) innovatively designs a failure-triggered human-machine collaborative mechanism. When the policy fails to adjust grasp force due to object uncertainty (e.g., slippage) or exhibits abnormal actions due to model deviations during real-robot execution, human experts intervene to provide sparse residual corrections. This mechanism not only directly corrects model errors (the model gap) but, more importantly, injects expert physical intuition for handling uncertainty into the policy, correcting systemic simulation biases via sparse human intervention. This calibration aligns the fundamental physics engine parameters, allowing the THED module to handle instance-level object variations (mass, friction) through online inference without further human guidance.

To maximize the correction efficiency of HCIL and ensure the policy possesses strong initial capabilities, we design two key supporting modules:

1. Cascaded Constraint Scheduling (CCS) mitigates the geometric gap by decomposing grasping into geometric approach, force closure, and dynamic stability. Physical constraints guide the policy toward plausible tactile primitives and provide a reliable baseline for human corrections.
2. Temporal Heterogeneous Distillation (THED) bridges the physical gap by learning to infer privileged physical parameters, such as friction and mass, from tactile history. This enables adaptive force modulation under object uncertainty.

Unlike blind manipulation approaches targeting full-scene navigation or object search, our work strictly focuses on grasping stability within the end-effector's workspace. PECHC specifically addresses the most intractable challenge in dexterous manipulation: dynamic force adaptation after contact establishment.

Scope and Assumption: It is important to clarify that our work focuses on grasp stabilization rather than global object search. We assume a coarse visual system or heuristics has guided the wrist to the object's vicinity (peripersonal space). PECHC addresses the critical challenge that follows: establishing a stable grasp under local pose uncertainty and dynamic mismatch, where vision is no longer reliable.

2 RelatedWork

2.1 Tactile and Proprioception-Based Dexterous Grasping

Blind manipulation relying on tactile and proprioceptive feedback offers unique advantages in vision-denied or highprecision force control scenarios. The core challenge lies in deriving adaptive policies from sparse, noisy signals. Suresh et al. [Suresh et al., 2023] addressed this via a Monte Carlo inference framework to recover object geometry from sliding feedback. OpenAI's Dactyl achieved blind object rotation using large-scale RL and domain randomization [Andrychowicz et al., 2020], though this approach requires immense computational resources. To mitigate reliance on randomization, Lin et al. [Lin et al., 2022] developed Tactile Gym 2.0, a high-fidelity simulator that narrows the sim-to-real gap. Regarding feature extraction, Hu et al. [Hu et al., 2025] utilized low-dimensional features for pose tracking, while Guzey et al. [Guzey et al., 2023] introduced self-supervised learning to derive general representations from interaction data. For stability prediction, Zhao et al. [Zhao et al., 2024] modeled spatiotemporal correlations using "pseudo-3D" tactile images and NDConv. Guo et al. [Guo et al., 2025] employed graph convolutional networks to enable skill transfer and verify tactile grasping of soft, slippery objects. Furthermore, Church et al. [Church et al., 2022] achieved millimeter-level tactile servoing via Real-to-Sim optimization. Despite these advances, existing works still lack effective online adaptation mechanisms for significant physical dynamic discrepancies between simulation and reality [Chari et al., 2024; Hang et al., 2024].

2.2 Human-Machine Interaction Correction and Policy Adaptation

Introducing Human-in-the-Loop (HITL) is effective for bridging sim-to-real gaps. Early methods like Behavior Cloning (BC) relied on full-time teleoperation, which is constrained by data quality and struggles to correct minor deviations [Mandlekar et al., 2022]. Recently, residual learning has gained attention: Johannink et al. [Johannink et al., 2019] compensated for model errors via learned residual actions, while Losey et al. [Losey et al., 2020] utilized physical contact for real-time correction. In vision domains, TRANSIC [Jiang et al., 2024] adapts via online expert corrections but relies on visual feedback for intervention timing, making it infeasible for Blind Grasp Stabilization. Meanwhile, Xue et al. [Xue et al., 2023] focused on preference-based task optimization rather than physical dynamics adaptation. Distinct from these, our Hybrid Correction Imitation Learning (HCIL) is tailored for tactile Blind Grasp Stabilization. Inspired by "Recovery Zones" [Thananjeyan et al., 2021], we replace visual judgment with an automatic failure-triggered mechanism based on Tactile Safety Thresholds. Expert intervention is requested only upon detecting anomalies (e.g., a slip). This on-demand correction significantly reduces annotation burden and guides the policy to focus on learning real-world physical dynamics missed by simulators, facilitating efficient transfer [Kojima et al., 2025].

3 Method

The PECHC algorithm aims to solve the problem of general dexterous grasping relying solely on sparse tactile feedback in unstructured environments. As shown in Figure 1, the algorithm consists of three progressive stages: first, learning physically plausible grasp primitives in fully observable simulations via Cascaded Constraint Scheduling (CCS); second, compressing the teacher's privileged physical information into tactile history representations through Temporal Heterogeneous Distillation (THED); finally, correcting long-tail dynamic errors not covered by simulations via human-machine interaction (HCIL).

3.1 Preliminaries

We model the problem as a Partially Observable Markov Decision Process (POMDP), defined as M = (S, Otac , A, T , R, γ).

Privileged State Space S ∈ R139 : st = [qt, ˙qt ∈ R12 , pobj ∈ R7 , vobj ∈ R6 , ϕphy], where qt, ˙qt are joint positions/velocities (12-DoF), pobj is object pose (3D position + quaternion), vobj is velocity (linear+angular), and ϕphy ∈ R102 contains contact geometry, physical properties (friction µ, mass m), and fine-grained contact states.

Tactile Observation Space Otac ∈ R39 : otac t = [qt, ˙qt ∈ R12 , f tip ∈ R15 ], where f tip represents 5-fingertip 3D forces. Joint torques τ are computed via inverse dynamics. The action space A ∈ R12 consists of joint position increments. Goal: learn πS(at|otac t-k:t ) maximizing E[∑ γ t rt] inMreal.

3.2 Stage I: Constraint-Aware Teacher Learning via CCS

(Resolving the geometric gap) In the simulation stage (Msim), we train the teacher policy πT (at|st) using privileged information. To address low exploration efficiency in high-dimensional action spaces and "non-physical violent grasping", we design the Cascaded Constraint Scheduling (CCS) mechanism. CCS models the grasping task as a physically constrained optimization problem, dynamically adjusting the reward function structure through state machine logic. Unlike traditional scalar rewards, CCS decouples the grasping process into three hierarchical constraints K ∈ {Kapp,Kcon,Ksta} and guides the policy to focus on specific physical sub-goals at different stages via dynamic weights αi.

(a) Geometric Approach Constraint (Kapp): Aims to guide the palm to envelope the object with a reasonable configuration. The reward function rapp is designed to guide the hand configuration without penalizing contact force, avoiding policy shrinking (fear of contact):

Here, dmin is the fingertip-object distance, E(q) aligns palm orientation, and Icontact indicates contact; λd, λe are scaling coefficients. α1 is high in the early training stage and gradually decreases as the stage progresses (as shown in the upper right of Figure 1), preventing the policy from overfitting to geometric positions while ignoring force control. Note that dmin and object pose pobj are privileged information available only during simulation training. The Student Policy πS does not have access to these states and must implicitly infer contact geometry solely from proprioception and tactile history during the approach phase.

(b) Force Closure and Balance Constraint (Kcon): After geometric contact is established, the goal shifts to achieving mechanically stable grasping. We introduce the minimum singular value σmin(G) of the grasp matrix G as a force closure quality metric and penalize the variance Var(∥ftip∥) of fingertip force distribution to induce human-like compliant multi-finger collaboration instead of relying on single-finger heavy squeezing.

Here, σmin(G) quantifies force closure quality, Var(∥ftip∥) ensures load distribution among fingers, and the last term prevents motor overload.

The coefficients and thresholds in CCS are physically grounded rather than arbitrarily tuned. For instance, the safety threshold fmax is derived from the sensor's saturation limit, and the geometric weights evolve based on the kinematic workspace of the XHand. We observed that this physically-constrained reward formulation is robust to minor hyperparameter variations compared to pure scalar reward shaping.

(c) Dynamic Stability Constraint (Ksta): During the lifting stage, the object must maintain a stable state when disturbed. The policy must keep the object's linear velocity ∥vo∥ and angular velocity ∥ωo∥ below thresholds while the object is subjected to random external force disturbances or being lifted (h > htgt).

This term rewards lifting the object to the target height htgt while penalizing jitter in linear velocity vo and angular velocity ωo during lifting. The weight vector α = [α1, α2, α3] dynamically evolves with learning steps. In the early stage, α1 dominates to quickly explore contact states; in the later stage, α2, α3 dominate to optimize the force control policy, ultimately learning the teacher policy πT (at|st). State switching follows an irreversible logic (unless the task fails and resets), and the policy strictly adheres to the physical causal chain of "geometric approach → mechanical balance → dynamic stability". The core role of CCS is to use "contact itself" as a geometric sensor in the absence of visual priors, guiding the policy to smoothly transition from "exploratory contact" to "force closure establishment".

3.3 Stage II: Temporal Heterogeneous Distillation (THED)

(Resolving the physical gap) Building upon the success of Rapid Motor Adaptation [Kumar et al., 2021] in locomotion, THED adopts an asymmetric actor-critic structure. While [Kumar et al., 2021] focuses on body dynamics, THED is specifically tailored to reconstruct contact parameters (e.g., friction, compliance) from sparse tactile history.

To transfer the teacher policy πT relying on privileged information st to the student policy πS with only otac t , this paper proposes Temporal Heterogeneous Distillation (THED).

Its core idea is to implicitly reconstruct environmental physical parameters from tactile temporal history (e.g., sliding history implies friction coefficient information). We explicitly design the student policy as a decoupled tactile reflex layer. In real-robot deployment, the wrist trajectory is handled by a separate kinematic planner, while the student policy assumes full authority over the finger articulations to stabilize the grasp. This decoupling strategy isolates the high-frequency contact stabilization problem from the low-frequency global transport, allowing PECHC to serve as a plug-and-play reflex module compatible with various arm planners.

The teacher encodes st into features zTt via an MLP. The student processes tactile history Ht = {(otac k , ak-1)}tk=t-H using an LSTM to extract hSt , followed by an action MLP. The distillation objective includes two loss functions:

(a) Behavior Cloning Loss (LBC): Collect demonstration trajectories Ddemo in simulation via the teacher policy, forcing the student action distribution to fit the teacher's actions:

(b) Latent Physical Representation Alignment (Llat): Force the projected LSTM latent state hSt of the student network to approximate the teacher's privileged feature zTt . This compels the student network to decouple key physical information from tactile history (System Identification). By minimizing the latent Llat, we force the recurrent latent state of the student network to reconstruct the teacher's beliefs about explicit physical parameters. This process goes beyond behavior cloning; it endows the student with online system identification capabilities, enabling it to infer invisible environmental dynamics (e.g., object mass distribution) from contact force history:

The total optimization objective is L = LBC + βLlat, where β is a balance coefficient.

3.4 Stage III: Hybrid Correction Imitation Learning (HCIL)

(Resolving the model gap) Although simulation training endows the policy with basic capabilities, complex dynamic characteristics in the real environment (e.g., cable pulling, nonlinear friction) may still lead to failures. We design a Hybrid Correction Imitation Learning (HCIL) system, introducing human experts as correctors. Our HCIL proceeds in two stages: (1) Adaptation: at initial deployment, human experts intervene upon failure detection to provide correction data; (2) Autonomous deployment: the fine-tuned π∗S operates independently on unseen objects without further human intervention.

We empirically observe that a significant portion of the Sim-to-Real gap manifests as systematic sensor-motor biases (e.g., global friction scaling, sensor thresholds) rather than purely object-specific dynamics. Therefore, HCIL utilizes a representative object to perform system-level alignment. Our experiments suggest that calibrating these funda-

Algorithm 1 PECHC Training Pipeline

Require: Simulator Msim, Real Robot Mreal, Expert Human E

Ensure: Robust Tactile Policy π∗S

• 1: ▷ Stage I: Teacher Training
• 2: Initialize πT , CCS Rewards {rapp, rcon, rsta}
• 3: while not converged do
• 4: Collect trajectories ζ ∼ πT inMsim
• 5: Update πT using PPOwith curriculum state K(s)
• 6: end while

7:

• 8: ▷ Stage II: Distillation
• 9: Collect dataset Dsim = {(Ht, aTt , zTt )} using πT
• 10: Initialize πS (LSTM-based)
• 11: while not converged do
• 12: Sample batch B ∼ Dsim
• 13: Update πS by minimizing LBC +βLlat
• 14: end while

15:

• 16: ▷ Stage III: Real-World Adaptation (HCIL)
• 17: Dmix ←∅
• 18: for iteration k = 1 to N do
• 19: Rollout πS inMreal
• 20: if Failure Detected (∥˙ f∥ > ˙ fslip or τ > τlimit) then
• 21: Expert E intervenes, collect correction ζexp
• 22: Dmix ←Dmix ∪ ζexp
• 23: else
• 24: Store successful trajectory ζself
• 25: Dmix ←Dmix ∪ Sample(ζself)
• 26: end if
• 27: Fine-tune πS on Dmix
• 28: end for
• 29: return π∗S

RMA [Kumar et al., 2021]. This baseline utilizes the identical LSTM architecture and domain randomization ranges as PECHC but lacks the CCS, THED, and HCIL components. By maintaining architectural parity, we isolate the performance improvements derived specifically from our proposed physics-evolving constraints and human-correction framework rather than general backbone capacity.

mental physics engine offsets significantly improves the policy's zero-shot generalization to unseen objects, reducing the need for per-instance fine-tuning.

Unlike full-time teleoperation, HCIL adopts a "failuretriggered" mechanism via tactile safety thresholds Φs = {fmax = 15N, ˙ fslip = 50N/s, τlimit = 1.5Nm}, derived from hardware specs (sensor saturation, motor rating) rather than object-specific tuning. When ∥˙ f∥ > ˙ fslip (slip detection) or τ > τlimit (overload), the robot pauses and requests expert correction via Manus Pro.

To prevent catastrophic forgetting, we construct a mixed dataset Dmix:

where Dexp is human correction data, Dpolicy is selfexploration data, and ρ = 0.3 (30% self-exploration + 70% human corrections). By fine-tuning πS on Dmix, the policy can quickly adapt to the physical distribution of the real world while retaining the general force control primitives learned in the simulation stage.

The complete algorithm flow is shown in Algorithm 1.

4 Experiments

4.1 Experimental Setup and Baseline Description

Experiments use a 12-DoF XHand with 3D force sensors and Manus Pro gloves. Simulations (Isaac Lab, 60Hz physics) require lifting objects to htgt = 6 cm for >2 s. Training employs basic geometric shapes, while testing evaluates generalization on 150 complex objects from the Visual Dexterity Dataset and fragile models.

For "Blind Grasp Stabilization", we assume that objects are initialized within a 2 cm×2 cm workspace. The wrist follows a fixed trajectory, relying on mechanical compliance and tactile feedback to accommodate positional errors. This definition decouples global navigation from grasping, focusing specifically on assessing policy robustness against geometric and physical uncertainties within peripersonal space.

To strictly quantify the marginal gain of our tactile adaptation modules, we employ Vanilla PPO with Domain Randomization (DR) as the primary baseline, consistent with the protocols in Dactyl [Andrychowicz et al., 2020] and

4.2 Simulation Performance Analysis

We first evaluate the learning efficiency of the teacher policy in the simulation environment. Figure 2 shows temporal snapshots of successful grasping and lifting objects in the simulation environment. The sequence demonstrates the three stages guided by CCS: approach (t = 0-3), where the hand pre-configures and approaches the object; grasping (t = 7-8), where stable multi-finger contact is established; lifting (t = 9-19), where the object is lifted to the target height while maintaining its posture. The quantitative impact of CCS is shown in Figure 3, which indicates that the CCS mechanism increases the grasping success rate from 3.8% of the unconstrained baseline to 98.9%. Funnel analysis shows that CCS effectively guides the policy to cross the exploration gap from "contact" to "force closure", proving the necessity of physical constraints in long-horizon task learning.

4.3 Real-World Evaluation

We set 6 difficulty levels for the Visual Dexterity Dataset [Chen et al., 2023] based on [Lum et al., 2024]. Adopting a 'calibrate-then-generalize' protocol, HCIL adaptation is performed once on a single calibration object from the 150-object evaluation set to align physical gaps. The policy is then frozen and evaluated on the full 150-object set, where the remaining 149 objects are unseen during calibration and no further intervention is allowed. As shown in Table 1, PECHC achieved a 97.3% success rate (146/150), demonstrating excellent post-calibration generalization (see Figure 4 and Figure 5(a)). We benchmark against external vision-based SOTA methods (DemoGrasp [Yuan et al., 2025], DextrAH-G [Lum et al., 2024]) and an internal baseline (Vanilla PPO + DR), representing standard Sim-to-Real RL without our proposed modules.

Table 2 presents results across two distinct evaluation paradigms. Vision-based methods (rows 1-3) achieve 80.0%-99.1% in idealized simulation with perfect visual perception, establishing the theoretical upper bound when geometry is fully observable. These results contextualize task difficulty but are NOT direct competitors since our setting deliberately excludes vision. Our primary contribution lies in the tactile-only comparison (rows 4-5): PECHC achieves 97.3% success on 150 real-world objects using only proprioceptive and tactile feedback-a substantial +42.8% improvement over standard Sim-to-Real RL (Vanilla PPO + DR: 54.5%). This gap demonstrates that domain randomization alone is insufficient for tactile-driven manipulation, validating the necessity of our CCS+THED+HCIL framework.

4.4 Ablation Study and Physical Perception Verification

To quantify the marginal contribution of each module and verify whether the policy truly possesses "physical perception" capabilities, we conduct a detailed ablation study (Figure 5(b)).

1. HCIL and Systematic Alignment: Removing HCIL drops the success rate from 97.3% to 72.7%. Notably, performance is restored by calibrating on just a single representative object. This confirms that a dominant component of the Sim-to-Real gap in our setup stems from global systematic biases (e.g., sensor sensitivity, friction scaling) rather than per-object geometric variance. HCIL effectively aligns the student's physical beliefs, facilitating zero-shot transfer to unseen rigid objects.
2. THED and Implicit Perception: Without tactile history, the success rate plummets to 54.5%, aligning with the vanilla domain randomization baseline. This 42.8%

Note: Vision methods evaluated in idealized simulation; tactile in real hardware.

Direct comparison not applicable due to different evaluation domains.

Figure 6: Experimental results on the private dataset

performance gain confirms that THED endows the agent with implicit system identification capabilities. By dynamically inferring invisible environmental parameters (e.g., friction, slip trends) from tactile history, the policy achieves true closed-loop physical adaptation rather than simple open-loop control.

4.5 Fine-Grained Force Control and Failure Analysis

To further evaluate the fine-grained force control capability of the policy, we selected 11 high-fidelity models of fruits with different hardness and shapes (e.g., lemons, red apples, grapes, starfruits) as well as fragile test tubes for testing. Experimental results show that PECHC achieves a 100% success rate (see Figure 6 for details). The policy successfully modulates fingertip force to prevent crushing, a capability often lacking in vision-only policies. Despite the high success rate, we analyzed rare failure cases to understand system limitations:

1. Geometric outliers-objects exceeding the hand aperture can fail during the initial "geometric approach" stage, as the blind policy struggles to locate boundaries.
2. Sensor saturation-high-speed grasping of heavy objects (> 500 grams) can saturate tactile readings, causing state-estimation loss and slip.

5 Conclusion

The proposed PECHC algorithm addresses the limits of vision-based planning by introducing a tactile-driven "Tactile Reflex" layer that decouples wrist navigation from fingerlevel interaction. With one-object calibration, PECHC closes the "Last-Centimeter" gap in the visual blind zone and achieves a 97.3% success rate on the 150-object benchmark. HCIL verifies that tactile feedback need not replace vision; instead, a human-calibrated tactile reflex loop with physical perception is sufficient for robust last-centimeter dexterous grasping. Future work will explore "In-Hand Reorientation" and vision-tactile fusion control.

Acknowledgments

The experiments, simulations, and model training were performed on the robotic AI-Scientist platform of the Chinese Academy of Sciences. This work was supported by the Major Project of Anhui Province's Science and Technology Innovation Breakthrough Plan (Grant No. 202423h08050003).