Real2Gym Framework Converts Videos to Sim Environments for Robot Skill Transfer, Cutting Token Use by 74.9%

Turning Videos into Training Grounds for Robots

Shanghai AI Laboratory Submission

QbitAI | Official Account: QbitAI

Placing a cup in the microwave is a daily action humans perform almost without thinking. However, robots often fail on subtle details, such as an unstable grip or the gripper hitting the door frame.

While learning from repeated failures is feasible, relying solely on real-world trial-and-error incurs significant costs due to execution time, reset procedures, and hardware maintenance.

Moving exploration to simulation introduces new challenges: even if the scene reconstruction looks realistic, can it truly reproduce the operations seen in the video?

To address this issue, a research team from Shanghai AI Laboratory, Shanghai Jiao Tong University, Zhejiang University, University of Science and Technology of China, The Chinese University of Hong Kong, and The University of Hong Kong proposed Real2Gym. This framework builds interactive simulation environments from human or robot videos, allowing agents to explore and accumulate skills within them before transferring that experience to real robots. The paper and code are now publicly available.

△ Real2Gym Overview: Building environments from video demonstrations, exploring in simulation, and applying skills to real robots.

Why Visual Fidelity Alone Is Not Enough

Multimodal models can already invoke modeling tools to generate detailed 3D scenes. However, robotic manipulation is particularly sensitive to geometric errors. A slightly narrow cup rim, object offset, or collision models blocking the interior of a container can cause actions that look reasonable in the video to fail during physical execution.

Real2Gym incorporates whether actions can be completed within the physics engine as part of the scene acceptance criteria. The system must not only preserve the objects, layout, and viewpoints from the video but also ensure the robot can successfully perform grasping, placing, and contact interactions.

Two Key Steps Transforming Video into Practice Environments

Constraining Scenes with Geometric Cues, Then Correcting via Interaction

For multi-view videos, Real2Gym leverages geometric cues provided by Pi3X, combined with SAM2 segmentation to associate objects across different views. It constrains their scale and position, then constructs an editable Blender scene and a corresponding MuJoCo environment. For single-view inputs, appropriate geometric initialization schemes are employed.

Next, the system focuses on moments where interaction changes: when an object is grasped, when it contacts a support surface, and when the gripper releases. Centered around these keyframes, the agent self-checks viewpoint, layout, contact, and appearance, combining local corrections based on adjacent frames.

Take pouring a cup as an example: even if the table and objects are aligned, size errors near the cup rim may still lead to penetration issues. Keyframe inspection concentrates corrections on areas that truly impact operation, avoiding reliance on a single overall rendered image to judge reconstruction quality.

Executing Actions in the Physics Engine for Verification

Corrected actions are executed in MuJoCo to check task completion, as well as support, contact, and stability. The execution trajectories are then exported back to Blender and compared against the original video to verify visual correspondence and physical feasibility.

△ Comparison of real and simulated execution for the "block-in-cup" task. From left to right: real video, Blender rendering, and MuJoCo execution, showing the correspondence of the same operation across three views.

Verified scenarios can further vary object positions, table height, materials, background, lighting, or distractors. Changes that affect motion space require action adaptation and re-verification, thereby providing agents with more practice environments.

△ From geometric initialization and interaction keyframe correction to physical execution verification, Real2Gym expands reconstructed scenes into environments for agent exploration.

Advancing One Manipulation Phase per Decision

With a training ground established, agents need to explore efficiently. Real2Gym elevates the goal of each decision round from generating a single action to generating code that executes a full manipulation phase.

For instance, a single response can plan approaching the cup, closing the gripper, lifting it, and checking for a secure grasp, organizing multiple actions and verification conditions together to reduce the number of model requests. SAM3 provides object localization cues, while GraspNet offers grasp candidates, helping the agent translate task intent into executable actions.

After execution, the agent organizes successful steps, failure causes, and recovery methods based on feedback, forming reusable skills. These skill descriptions detail how to act relative to objects, when to check, and how to correct after failure; upon reuse, they must be re-localized and adjusted according to current observations.

Self-improvement here is achieved through updating the skill library, while the underlying model weights remain unchanged. Each new task starts with an independent context and receives selected skills, allowing the effectiveness of the experience itself to be evaluated.

Higher Success Rates and Reduced Model Round-Trips

The team selected 24 scenarios from DROID and EgoDex, covering both robot demonstrations and human manipulation videos. In reconstruction evaluation, Real2Gym outperformed the compared GPT baselines in content alignment, viewpoint alignment, action fidelity, and simulation success scores. Qualitative comparisons show it better preserves cabinet structures, relative object sizes and positions, and original camera viewpoints.

△ Compared to the original video, scenes reconstructed by Real2Gym are closer to actual layouts and task-relevant structures. From top to bottom: GPT-5.6 Sol, GPT-6 Astra, Real2Gym, and the real video.

In agent evaluations within these environments, the average success rate reached 87.5% on the second execution after reusing first-round experience, compared to 70.8% for the GPT-6 Astra Direct Mode baseline. Token usage for policy execution decreased by approximately 74.9%. The average number of model responses also dropped from about 31 to about 14.

Phase-level code allows one decision to advance more actions, while skill reuse reduces redundant exploration. The paper also presents cases where Real2Gym succeeded in bowl-stacking tasks while two GPT baselines failed, as well as more direct cabinet operations and adapter placements after experience reuse. Note that the overhead statistics mentioned above pertain to policy execution and exclude scene construction and separate skill extraction.

Fixing Failures in Simulation Before Returning to Reality

On a Franka robotic arm, the team tested four tasks: block-to-tray, hanging a cup, placing a cup in the microwave and closing the door, and shelving a plate. In three attempts each for the first three tasks, Real2Gym succeeded every time; the GPT-6 Astra baseline succeeded 1, 3, and 1 times respectively.

The plate-shelving task posed difficulties for both methods: grasp depth, plate rim orientation, and placement precision in narrow spaces had to meet requirements simultaneously. Neither method succeeded in the initial executions.

Subsequently, the team recorded human demonstrations and rebuilt an interactive digital twin. The agent failed its first two attempts in simulation but acquired usable skills on the third iteration. Combining these skills with real-time observations on the physical robot allowed it to complete the originally failed task. This demonstrates the complete process of starting from real demonstrations, improving skills through simulation exploration, and validating back in the real world.

△ Real-robot execution of plate shelving: Skills accumulated in simulation helped the robotic arm complete a task that initially failed. Left and right show external and wrist-camera views, respectively; clips are sped up.

Making Manipulation Videos the Starting Point for Robot Experience Accumulation

Real2Gym offers a pathway to leverage manipulation videos: building environments from demonstrations, allowing robots to receive feedback through execution, and saving experiences as reusable skills. As more videos are converted into interactive tasks, robots are expected to accumulate richer manipulation experience in simulation.

Currently, point cloud noise still affects reconstruction, and stage-level control has room for improvement in fine-grained operations; real-robot validation is limited to Franka arms and parallel grippers. Extending to dexterous hands and humanoid robots requires further experimentation.

Paper Title: Real2Gym: Building Gyms from Videos, Bringing Skills to Robots Paper: https://arxiv.org/abs/2609.37089 Real2Sim Code: https://github.com/real2gym/Real2Gym Agent Code: https://github.com/real2gym/R2G_Agent

Like, Share, and Favorite!

We welcome your thoughts in the comments section!

— End —

[Academic Submissions] Please send emails on weekdays to: [email protected], with "[Submission]" in the subject line, and tell us: Who you are, Where you come from, Submission content attached with Project/Homepage links, and Contact information.

🎓 We will (try our best to) reply to you promptly :)

🌟 Star Us 🌟

Daily updates on cutting-edge tech progress