Fudan Proposes RoboSkill for Self-Evolving Robot Agents

From Task Completion to Efficient Execution: What Is Still Missing for Robotic Agents?

Contributed by the RoboSkill Team

QbitAI | WeChat Official Account QbitAI

GPT-6-Astra has delivered a significant shock to the field of embodied intelligence.

Today, general-purpose agents represented by Astra and others have overcome the historical bottleneck in robotics where data collection was a prerequisite for task execution.

Without pre-collecting demonstration data for target tasks or conducting targeted training, robots can now rely on agents to observe, reason, and write code on-site, figuring out how to complete tasks through exploration.

However, although on-site exploration saves the cost of data collection, agents still spend considerable time repeatedly cycling through "observation-reasoning-execution-result verification."

The gap between "being able to complete" and "completing efficiently" remains a hurdle that must be crossed.

To address this, the team from the Institute of Trusted Embodied Intelligence at Fudan University proposed RoboSkill: saving effective methods and code discovered during exploration as skills, allowing robots to avoid detours in future tasks.

How Much Data Do Robots Still Need When Entering New Scenarios?

△ VLA/WAM learned to hammer nails vertically but cannot hammer them diagonally without adding new data. Agents can solve this task through reasoning, but the process is slow.

Vision-Language-Action models (VLA) and World Action Models (WAM) are typically pre-trained on robot data and then adapted to specific tasks. However, when actually deployed, they often still require collecting a batch of data in the target scenario for post-training. The range of tasks a model can perform largely depends on what is included in this data.

For example, training on data involving grasping blocks does not imply that the model will know how to open drawers. How much help pre-training provides for new tasks remains a question that this approach needs to answer.

Currently, common practice involves continuing to collect data in deployment scenarios: supplementing data for missing tasks, adding demonstrations for new situations, hoping that if the model sees enough examples, it can handle them upon deployment. But this means that every time tasks are added or environments change, another batch of data may need to be collected.

Robot data, however, is notoriously difficult to collect. Comparing with autonomous driving, people already need to drive, so autonomous driving data can be continuously acquired during daily commutes. General-purpose robots do not yet have such consumer demand; currently, operators must be hired to teleoperate robots, repeatedly placing objects, demonstrating actions, and resetting scenes. As data volume increases, investments in manpower, equipment, and facilities must also increase accordingly.

Moreover, spending money to collect more data does not necessarily cover situations the model has never seen. Operators have their own habits, and the same task might always be completed in a few fixed ways; items, arrangements, and tasks within the facility are relatively limited. Even if the number of collections increases, the model may still only see similar operations. Upon actual deployment, changing an object or its arrangement can lead to problems again.

Tesla relies on a massive fleet of user vehicles and has accumulated data over approximately ten years. Robots lack a similar user base while facing a wider variety of tasks and operational methods.

If every new situation requires solving via additional data collection, how long will it take before robots can adapt to unseen new scenarios?

Robotic Agents Are Disrupting the Embodied Paradigm

Recently, the emergence of capabilities in agents like GPT-6-Astra to directly control robots to complete tasks has pushed robotic manipulation research into its next phase.

In the paper "Explore, Execute, Evolve: A Skill Acquisition and Reuse Loop for Embodied Agents," the Fudan University team and their collaborators systematically evaluated the operational capabilities of current general-purpose multimodal agents on both simulated and real robots. They found that even without pre-collecting demonstration data for target tasks or performing targeted model training, agents could still control robots to complete various manipulation tasks through on-site observation, reasoning, and code writing.

In the LIBERO-10 simulation environment, allowing multiple attempts within a four-hour budget, GPT-6-Astra and Opus 5 achieved a final success rate of 100% without using existing skills, while Fable 5.1 and GPT-5.6-Sol reached 97.5% and 82.5%, respectively. On real robots, GPT-6-Astra achieved success rates of 91.7% and 80% across six simple tasks and six high-difficulty tasks, respectively. These results indicate that agents, through on-site reasoning and interactive exploration, are already capable of solving diverse robotic manipulation tasks.

However, behind these high success rates lies lengthy execution time. In simulations, GPT-5.6-Sol averaged 94.5 minutes, and Opus 5 required 80.9 minutes; on real robots, GPT-6-Astra took an average of 20.3 minutes for simple tasks and 27 minutes for high-difficulty tasks to achieve success. Moving from "being able to complete" to "completing efficiently" remains a problem that robotic agents must solve.

Why Does It Take So Long for Agents to Complete a Task?

The research team found that agents typically undergo repeated cycles of "observation—reasoning—execution—result verification." Each time a new uncertainty arises, the agent may need to stop and re-evaluate. While a single movement of the robotic arm may be brief, the reasoning between actions can last several minutes, causing time to accumulate rapidly over multiple rounds.

To address this, the team proposed RoboSkill, reducing such overhead in two ways: using tactile feedback to confirm physical states more quickly, and leveraging skill accumulation to reuse effective methods discovered during exploration.

The former helps agents judge whether actions are properly executed more quickly, while the latter preserves strategies and code from previous executions, reducing repetitive exploration and reasoning in subsequent tasks.

Adding Tactile Feedback to Help Agents Confirm Action Results Faster

Has the gripper touched the object? Has the robotic arm contacted the table surface when moving downward? Relying solely on images, these states are not always easy to determine. Agents may need to observe repeatedly, make minor adjustments, and then verify if the action is correct.

Tactile feedback provides direct evidence for these judgments. Contact information such as pressure, force, and torque helps agents understand what actually happened, allowing them to decide promptly whether to continue or adjust.

In experiments, GPT-6-Astra relying only on vision eventually completed all simulation tasks, but after adding tactile feedback, the average time decreased from 25.35 minutes to 15.93 minutes. For the test tube retrieval task on real robots, the average time for successful trials dropped from 33.63 minutes to 20 minutes. Tactile feedback allows agents to confirm states faster, thereby completing tasks more quickly.

Accumulating Skills to Preserve Explored Methods

Beyond faster state confirmation, another source of time overhead comes from redundant exploration. When an Agent discovers an effective method during a task, it may need to re-derive and retry that method for similar subsequent problems if the solution is not saved.

RoboSkill enables Agents to review their execution process after completing a task, organizing effective methods into Skills and continuously updating them based on subsequent execution results. For example, after determining table height through tactile probing, the Agent can save this measurement procedure. In a new scenario, the height must be measured again, but the measurement steps already have a established method to follow.

Skill accumulation yields tangible efficiency gains: GPT-5.6-Sol’s average completion time in simulation dropped from 94.5 minutes to 35.9 minutes, with acceleration also observed for both simple and complex tasks on physical robots.

Saving Code in Skills Enables Direct Method Reuse

What is stored within a Skill determines how much work it saves the Agent. A directive like “measure the table height first” provides strategy, but specifics such as movement control, feedback reading, and contact detection still require the Agent to write code.

Therefore, RoboSkill saves textual guidance alongside the actual code used during execution: text explains applicability conditions and operational strategies, while code preserves concrete implementations, allowing the Agent to modify and invoke them according to the current context.

In experiments, when Skills evolved from “text-only” to “text + code,” average completion times decreased across all four tested Agents. Notably, GPT-5.6-Sol’s completion time fell from 67.1 minutes to 39.9 minutes, and its one-shot success rate improved from 15% to 67.5%. Retaining code provides a practically validated starting point for subsequent executions, reducing the overhead of deriving actions anew from textual strategies.

Further experiments demonstrate that Skills can save execution time across tasks, improve performance across models, and continue evolving through subsequent executions.

Using tactile feedback to enhance exploration efficiency, preserving exploration outcomes via code, and continuously updating Skills through execution. RoboSkill bridges single-task efficiency challenges with long-term skill accumulation, turning exploration into the foundation for future capability growth toward Embodied Recursive Self-Improvement (RSI).

Author Introduction

Xie Sicheng and Chen Yitong are PhD students at the Institute of Trustworthy Embodied AI, Fudan University. Xie Sicheng’s research focuses on robotic Agents, vision-action-language models, and cross-embodiment learning. This study was conducted in collaboration with Shanghai XinZhi Embodied Intelligence Technology Co., Ltd. The corresponding authors are Professor Wu Zuxuan and Professor Jiang Yugang from Fudan University.

Paper: arxiv.org/abs/2609.37810 Code: github.com/SII-dannyXSC/RoboSkill

Like, Share, and Subscribe!

We welcome your thoughts in the comments section!

— End —

【Academic Submissions】Please send emails to [email protected] on weekdays. Include 【Submission】 in the subject line and tell us: Who you are, Where you come from, Your submission content, attach Project/Homepage links, and provide Contact information.

🎓 We will reply as promptly as possible :)

🌟 Star Us 🌟

Daily updates on cutting-edge tech progress