Shanghai AI Lab Open-Sources InfiniHand: Streaming World-Coordinate Hand Reconstruction at Double the Speed of HaWoR

InfiniHand: More Accurate and Faster Hand Reconstruction

Shanghai AI Laboratory | Submission

QbitAI | WeChat Official Account: QbitAI

Humans perform rich hand manipulations every day. Recording these processes via head-mounted cameras can provide massive demonstrations for embodied intelligence.

However, extracting 3D motion from this data lacks a crucial layer of information: Where exactly are the hands in space? How do the fingers bend? What trajectories do they follow?

The difficulty lies in the fact that the camera is also moving. A hand appearing to move right in the frame might simply be due to a head turn; object occlusion and bimanual interaction further cause critical finger details to disappear from view.

To address this issue, Shanghai AI Laboratory, in collaboration with teams from Shanghai Jiao Tong University and other institutions, proposed InfiniHand. This streaming model jointly estimates hand position, hand shape, and camera trajectory, restoring continuous actions into a shared world coordinate system.

The team aggregated approximately 5,000 hours of public first-person video, performing two-stage training after data cleaning. This allows the model to simultaneously learn fine-grained gestures and cross-frame spatial relationships. In reported prediction speed comparisons, InfiniHand achieved 11.19 FPS, more than double that of HaWoR; reconstruction accuracy also improved across multiple test sets.

△ Overview of InfiniHand's hand reconstruction.

First Know Where the Hands Are, Then See How Fingers Move

Reconstructing hand motion cannot rely solely on local images of the hands. The same palm size may correspond to different distances, and local appearance alone struggles to explain camera rotation. To determine the hand's position in space, it must be understood within the context of the surrounding scene and previous/next frames.

Traditional world-coordinate reconstruction methods typically delegate tasks to multiple systems: detectors find hands, hand networks estimate pose, SLAM systems track the camera, and results are then aligned. Errors can occur at each stage, with detection jitter, trajectory drift, and scale bias propagating to subsequent phases. When processing large-scale videos, the computational and coordination costs of multiple models are also non-negligible.

InfiniHand chooses to share the geometric context required by these stages. The pre-trained LingBot-Map backbone extracts geometric features from the full frame, based on which the localization module predicts left and right hand regions; the hand reconstruction module then fuses local appearance to supplement finger and joint details. Global features help determine spatial relationships, while local features help restore fine-grained shapes.

The output uses the MANO parametric hand model, describing joint pose, hand shape, and orientation with a set of parameters. Auxiliary 2D keypoint predictions constrain the projection of 3D results in the image, ensuring the reconstructed hands align with visible positions. The model subsequently combines camera trajectories to transform per-frame hand results into the same world coordinate system, yielding action trajectories comparable across frames.

△ Geometric information from the full frame and local hand details jointly participate in reconstruction; two-stage training progressively connects gestures with global motion.

5,000 Hours of Video: First Address Annotation Reliability

How much video the model has seen is important, but erroneous annotations enter training along with scale. Left/right hand ordering, coordinate definitions, and annotation types vary across datasets; some annotations, when projected back onto the image, do not even match the actual hand position or pose.

The team aggregated approximately 5,000 hours of video from nine public datasets including ARCTIC, HOT3D, and EgoDex, unifying images, hand parameters, joints, masks, and bounding boxes. Existing 3D keypoints were used to fill missing MANO parameters, while sequences with camera trajectories retained additional supervision in the world coordinate system.

During cleaning, the system used Sapiens 2 to predict 2D hand keypoints in the image and compared them with projections of original annotations. Sequences with persistent inconsistencies were filtered out, and uncertain samples were reserved for manual inspection. This step removed approximately 30%–40% of candidate training data, providing quality control for scaling up.

Based on this, training was divided into two steps. The first stage utilized valid camera-space annotations to first learn hand localization and fine-grained reconstruction; the second stage introduced camera trajectories and temporal supervision to jointly learn hand and camera motion. This way, data with varying annotation conditions could all participate in training, allowing fine-grained gesture learning and global trajectory estimation to have distinct focuses.

Continuous Video Input, Continuous Forward Reconstruction

Long-video processing faces another practical problem: how to utilize past information while controlling computational cost? InfiniHand continuously receives video in windows, retaining the initial reference frame, recent detail features, and compressed memory from earlier moments. The model thus extends geometric context without needing to reprocess the entire history each time.

During inference, two task streams work in parallel: one extracts backbone features for the next window, while the other decodes hand and camera results from the previous window, reducing serial waiting. The paper reports a prediction throughput of 11.19 FPS, approximately 2.04 times that of HaWoR. For batch video annotation, this means more frames can be processed in the same amount of time.

To address residual drift in long sequences, the system also performs sparse bundle adjustment in the background, correcting camera trajectories through geometric relationships between keyframes. A binary keyframe pool retains both recent observations and some earlier frames, avoiding loss of long-term connections by focusing only on the immediate present. This optimization step still incurs additional computational overhead; the aforementioned 11.19 FPS refers specifically to prediction throughput.

Fewer Missed Hands Under Occlusion, Continuous Actions Closer to Reference

Performance improvements are first evident in scenes prone to failure. The paper demonstrates examples such as opening boxes, bimanual object manipulation, and natural scenarios like kitchens and tool benches: comparative methods might miss a hand or misestimate palm position and finger pose, whereas InfiniHand recovers these more completely in such samples. The video below allows further observation of changes across continuous frames.

△ Comparison in Ego4D natural scenes. Top-left: InfiniHand; Top-right: ViDiHand; Bottom-left: HaWoR; Bottom-right: WiLoR. Note: In this offline example, InfiniHand uses an external hand detector.

Quantitative tests cover 34 ARCTIC, 10 HOT3D, 166 HOI4D, and 100 EgoDex scenes. Measured by the PA-p metric (which considers both pose error and missed-detection penalties), InfiniHand achieved the lowest error among compared methods on all four datasets. Compared to ViDiHand, error on ARCTIC dropped from 9.82 mm to 7.72 mm, a 21.4% reduction; on EgoDex, it dropped from 17.22 mm to 7.29 mm, a 57.7% reduction.

After placing hands in the world coordinate system, InfiniHand achieved the lowest global joint position error (W-MPJPE) among compared methods on ARCTIC, HOT3D, and EgoDex. Specifically, on EgoDex, W-MPJPE dropped from 78.41 mm (Dyn-HaMR) to 28.77 mm, a 63.3% reduction. Aligned continuous pose visualizations also show that its action range and spatial arrangement are closer to the reference results.

△ Overlaying hands from consecutive timestamps allows comparison of action range, inter-hand distance, and pose changes. Blue represents InfiniHand; Green represents the reference. The figure shows qualitative comparisons after alignment.

Ego4D’s natural-scene videos offer another test: can the model handle viewpoints, occlusions, and interactions outside its training distribution? The reconstructions shown in the examples demonstrate generalization potential, achieving precise and stable results even in heavily occluded regions.

Making More Human Operation Videos Usable as 3D Demonstrations

InfiniHand integrates hand localization, fine-grained reconstruction, and camera estimation into a unified streaming framework, providing a more efficient path for large-scale annotation of first-person videos. After quality screening and adaptation to robotic morphologies, these 3D motions hold promise for dexterous hand data augmentation, motion transfer, and imitation learning, thereby expanding the sources of human demonstrations available for embodied training.

This pathway still requires refinement: the system needs an additional model to recover physical scale; large motions and intermittent occlusions remain challenging; and supervision generated from monocular reconstruction may introduce scale drift. Higher-quality and more diverse training data could extend reliable annotation to more complex daily activities.

Authors: Kerui Ren, Kaiwen Song, Weiguang Zhao, Yuxi Wang, Yufei Liu, Bo Dai, Haoyu Guo, Chunhua Shen, Mulin Yu, Tao Lu, Junting Dong. Affiliations: Shanghai AI Laboratory, Shanghai Jiao Tong University, University of Science and Technology of China, University of Liverpool, Nanyang Technological University, University of Hong Kong, Zhejiang University. The paper was released as a preprint on arXiv on September 28, 2026. Paper: https://arxiv.org/abs/2609.35743 Code Repository: https://github.com/infinihand/InfiniHand

Like, Share, and Favorite!

We welcome your thoughts in the comments section!

— End —

[Academic Submissions] Please send emails to [email protected] on weekdays. Mark [Submission] in the subject line and tell us: Who you are, Where you come from, Your submission content, attach Project/Homepage links, and provide Contact information.

🎓 We will (try to) reply promptly :)

🌟 Star Us 🌟

Daily Updates on Cutting-Edge Tech Progress