Real2Gym Framework Converts Videos to Sim Environments for Robot Skill Transfer, Cutting Token Use by 74.9%

1 reports archived1 independent sourcesupdated 10/13/2026, 19:12:00

Synthesis & Latest Updates

1 Sources Cross-Validated

Shanghai AI Lab and partners introduced Real2Gym, a framework that converts human or robot videos into interactive simulation environments using geometric constraints and physics engine validation. This approach addresses the gap between visual realism and physical feasibility in video reconstruction, enabling agents to learn skills in simulation and transfer them to real robots, resulting in a 74.9% reduction in token usage.

LATEST/Shanghai AI Lab and partners introduced Real2Gym, a framework that converts human or robot videos into interactive simulation environments using geometric constraints and physics engine validation. This approach addresses the gap between visual realism and physical feasibility in video reconstruction, enabling agents to learn skills in simulation and transfer them to real robots, resulting in a 74.9% reduction in token usage.

HEAT TRENDHourly heat curve

2 fully observed hours

Fewer than 3 fully observed hours; trend not plotted yet.

TIMELINECoverage timeline

Total 1 reports · Latest first

量子位 (微信公众号)T2·Oct 13, 19:1278 pts
Real2Gym Framework Converts Videos to Sim Environments for Robot Skill Transfer, Cutting Token Use by 74.9%
- Real2Gym uses Pi3X geometric cues and SAM2 segmentation to reconstruct multi-view videos into Blender scenes and corresponding MuJoCo physics environments.
- It adopts 'physical executability' as a scene acceptance criterion, correcting contact, support, and stability errors via keyframe inspection.
- The framework supports randomizing object positions, materials, and lighting in validated simulations to generate diverse tasks for agent generalization.
- Experiments show effective skill transfer from simulation to real robots, reducing token usage during training by 74.9%.
View Evidence Original Source