Tsinghua's SkillMaster Trains Agents to Evolve Their Own Skills

Original title: NeurIPS 2026|不再依赖外挂:清华把「自我进化技能」训进Agent大脑
78AI Score

A Tsinghua team led by Prof. Li Yong proposes SkillMaster, which folds skill creation, updating and retention into the agent's own policy and learns it jointly with task execution via reinforcement learning, removing reliance on external teacher models or handcrafted rules. Skill edits are rewarded when previously failed similar tasks later succeed and solved tasks need fewer steps; the paper is accepted to NeurIPS 2026.

Full Report

SkillMaster: Teaching Agents to Evolve Their Own Skills

With the rise of Recursive Self-Improvement (RSI) research, the long-term capability evolution of agents has attracted increasing attention. Skills can distill interaction experience into reusable operational procedures and task strategies, allowing the gains from one task to serve subsequent tasks. Enabling agents to autonomously create and improve these skills provides a concrete path toward their continuous evolution.

Figure 1 Left: External modules manage skills, and the agent retrieves and uses them. Right: SkillMaster incorporates skill management into the agent's own policy, learning it jointly with task execution.

However, skill creation and updating in existing methods typically rely on external teacher models, predefined rules, or auxiliary modules (Figure 1, left). The agent executing tasks learns to use skills but is rarely trained directly to improve them. To achieve autonomous improvement along this path, "how to evolve skills" must become a capability that the agent itself can learn (Figure 1, right).

The team of Professor Yong Li at Tsinghua University proposed SkillMaster, and the related paper has been accepted by NeurIPS 2026. The first author is Min Yang, a PhD student at Beijing Zhongguancun Academy.

The research starts from an intuitive insight: a valuable skill improvement should make previously failed similar tasks easier to succeed in subsequent attempts, and make previously successful tasks completable in fewer steps. SkillMaster converts these two types of effects into learning signals for skill editing, training the same agent to create, update, and retain skills based on interaction experience, so that skill evolution and task execution are jointly incorporated into the reinforcement learning process.

  • Paper Title: SKILLMASTER: Toward Autonomous Skill Mastery in LLM Agents
  • Author: Min Yang, Jinghua Piao, Xu Xia, Xiaochong Lan, Jiaju Chen, Yongshun Gong, Yong Li
  • Affiliation: Shandong University, Zhongguancun Academy, Tsinghua University, Southeast University, University of Science and Technology of China
  • arXiv: https://arxiv.org/abs/2605.08693
  • GitHub: https://github.com/sduyangmin/Skill-Master

Core Insight: Make Failed Tasks Easier to Succeed

and Successful Tasks More Efficient to Complete

To train an agent to improve skills, it is first necessary to know what a modification brings to subsequent tasks. Using only whether the task ultimately succeeds as the reward makes it difficult to distinguish execution capability from skill quality, and even harder to provide clear feedback for a specific skill edit.

To this end, SkillMaster focuses on the actual impact of skill modifications on subsequent related tasks. If a skill fills a gap in existing knowledge, it should increase the chances of success for previously failed tasks; if it improves an existing operational strategy, it should reduce unnecessary steps for tasks that can already be solved. These two observable changes provide a basis for measuring the effect of skill modifications.

Method Framework:

Jointly Learning Task Execution and Skill Evolution

To enable an agent to autonomously create, revise, and retain skills, three problems need to be solved: what to base modifications on, how to evaluate the value of modifications, and how to jointly optimize task execution and skill management. SkillMaster addresses these three aspects through trajectory-based skill review, downstream utility rewards, and DualAdv-GRPO, respectively.

Figure 2 The three components of SkillMaster: trajectory-based skill review, counterfactual skill utility rewards, and DualAdv-GRPO joint optimization.

Learning Skill Management from Interaction Trajectories

In the skill review stage, the task goal, retrieved skills, action and observation trajectories, and final feedback are provided to the model together. The model performs creation, updating, and retention operations through three types of tool calls: propose_skill, update_skill, and keep_skill. For example, when it finds that existing skills do not cover a certain search strategy, it can add a new skill; when it observes that actual feedback conflicts with existing operational rules, it can revise the original skill.

Task actions and skill tool calls are generated by the same policy, so skill management is also directly incorporated into model training. SkillMaster first uses supervised fine-tuning to equip the model with basic task execution and tool-calling capabilities, and then uses skill utility feedback to optimize decisions on creating, updating, and retaining skills through reinforcement learning.

Evaluating Skill Utility through Counterfactual Comparison

For a candidate skill edit, the system selects a set of related probe tasks and executes them separately with the original skill library and the temporarily modified skill library, directly comparing the downstream changes brought by the modification. Whether previously failed tasks turn into successes and whether previously successful tasks reduce execution steps are both reflected in the utility score, providing direct training feedback for skill edits.

Probe tasks come from the training task pool and are selected based on task category, product category, or question semantic similarity. The system computes a utility reward by considering the average score change of probe tasks as well as the number of tasks that improve and degrade, and optimizes skill editing decisions accordingly. During training, only modifications with positive utility rewards are written into the persistent skill library.

DualAdv-GRPO Jointly Optimizes Two Types of Decisions

Task execution mainly relies on environmental result feedback, while skill management uses tool-call effectiveness and downstream utility feedback; the two types of rewards have different scales and meanings. To this end, SkillMaster proposes DualAdv-GRPO, which separately computes the advantages of task execution and skill management, then assigns them to the corresponding generated content, jointly updating the same policy. The model thereby simultaneously learns to execute tasks effectively and improve reusable skills.

Experimental results:

From task performance to skill evolution capability

The experiments are based on Qwen2.5-7B-Instruct and cover ALFWorld, WebShop, and SearchQA. Among them, ALFWorld contains six types of simulated household tasks, WebShop evaluates multi-step product search and purchase, and SearchQA covers seven single-hop and multi-hop question-answering datasets.

Table 1 Main experimental results on ALFWorld and WebShop.

As shown in Table 1, SkillMaster achieves the best performance among the compared methods on both ALFWorld and WebShop, with task success rates reaching 98.7% and 82.0%, respectively, improving over SkillRL by 8.8 and 9.3 percentage points. Four of the six ALFWorld task types reach a 100% success rate, and the remaining two both exceed 95%.

Table 2 Experimental results on the seven SearchQA datasets.

As shown in Table 2, on SearchQA, SkillMaster achieves an overall score of 48.0%, the highest among the compared methods, improving over SkillRL by 0.9 percentage points. It outperforms SkillRL on five of the seven datasets, demonstrating the effectiveness of this framework in search-augmented question-answering tasks.

Figure 3 Left: component ablation results. Middle: changes in the proportion of skill management operations. Right: changes in the size of the skill library.

As shown on the left side of Figure 3, removing the skill utility reward, switching to merged advantage normalization, or randomly selecting probe tasks all degrade performance, supporting the roles of utility feedback, separate advantage estimation, and related task selection. The middle and right sides of Figure 3 show changes in skill management behavior and skill library size, respectively.

During ALFWorld training, creation and update operations are more active in the early stage, and in the later stage the proportion of retained skills rises from about 81.25% to 97.66%. This process reflects the model's behavioral shift from supplementing and revising skills to gradually retaining accumulated skills. The authors also fixed the trained model parameters to test its ability to build skills during test-time interaction. In this setting, counterfactual probing and utility feedback are not used; valid skill management operations are executed directly and reused for subsequent tasks. On WebShop, starting from an empty skill library and enabling both skill construction and retrieval, the success rate increases from 65.63% to 71.88%, indicating that the trained policy can generate and reuse useful skills based on new interactions.

Case analysis:

Creating and revising skills based on execution evidence

Figure 4 The agent discovers skill gaps from task trajectories, corrects existing rules, and validates the effect of modifications on related tasks.

As shown in Figure 4, in an ALFWorld cooling task, the agent spends a large number of steps on cabinets and drawers with a low probability of containing food, and ultimately fails to complete the task. During review, it identifies that the existing skill lacks a search priority rule, so it creates a new skill instructing subsequent searches to prioritize high-probability locations related to the target object.

Another case demonstrates the revision of an existing skill. According to ALFWorld's interaction interface, heating must be performed while the agent is holding the target object, but the old skill required placing the object into the microwave first. The model revises this operating condition based on feedback from the trajectory. The comparison on the right side of Figure 4 shows that in related probe tasks, the heating phase is shortened from 8 steps to 2 steps, eliminating six redundant operations and directly reflecting the downstream utility of the skill modification.

Research outlook:

Exploring recursive self-improvement through learning skill evolution

Starting from skills as a reusable knowledge carrier, SkillMaster explores one path toward recursive self-improvement of agents. The agent discovers knowledge gaps or inefficient strategies during interaction, changes subsequent behavior through skill editing, and then learns more valuable editing decisions based on actual effects. Task execution and skill improvement are interconnected, enabling the agent both to accumulate skills and to improve its ability to improve skills through training.

The current method has been validated in structured interaction environments and search-augmented question answering. Moving further toward open environments, it is necessary to explore more effective probe task selection and lower-cost utility evaluation, and to extend to skill deletion, cross-agent sharing, and continual learning under distribution shifts. These directions are expected to enable the skill library and the agent's capabilities to co-evolve through long-term interaction.

Image

© THE END

For reprinting, please contact this official account for authorization.

Submissions or report requests: [email protected]

Open in WeChat