Morphometric Imitation: From Morphology and Contact Aware Hand Retargeting to Sim-to-Real Visuomotor Policy

The authors propose Morphometric Imitation, a three-stage framework transforming reconstructed human hand-object interactions (HOIs) into zero-shot sim-to-real visuomotor policies. First, morphometric optimization (MMO) kinematically retargets human motion across hand morphologies while preserving demonstrated contacts. Second, residual reinforcement learning (RL) refines the kinematic reference using object pose and contact data. Third, these demonstrations are distilled into visuomotor policies. Across three robot hands and ten HOIs, MMO improves contact F1 over five baselines by at least 8 points, and the final visuomotor policies achieve 89.3% zero-shot success in 300 real-world trials on 30 objects.

MMO improves contact F1 over the strongest of five baselines by at least 8 points for every hand.

Residual reinforcement learning refines the kinematic reference into dynamically feasible robot demonstrations.

Visuomotor policies achieve 89.3% zero-shot success in 300 real-world trials on 30 objects.