Stanford Fixes Robot Instruction Failures by Editing 65 Embedding Rows

Across 335 held-out instructions containing the edited tokens, the success rate rose from 74.3% to 87.5%; for the other 430 episodes, the success or failure outcome remained unchanged in every single case.

By changing only 65 rows of word embeddings, GPT-6 Astra fixed a class of robot failures where the robot "fails as soon as you rephrase the instruction."

The task is to turn on the stove. The instruction given to the robot was "activate the cooking appliances heat source." The original policy, however, reached for a plate and still had not completed the task after 300 steps.

With the modified weights, in the same scene, with the same instruction and the same random seed, the robot completed the task at step 78. No gradient updates were performed during the editing process, and no new robot demonstration data was added.

Animation 1 | Robot behavior before and after the edit on the same held-out instruction. The original policy on the left timed out at 300 steps; the edited policy on the right succeeded at step 78 and then stayed on the final frame. The scene, random seed 7, and initial state 0 are identical, and the footage comes from real simulation recordings. This is a selected repair case; the full statistics also include degraded episodes.

This research was conducted jointly by Jusheng Zhang, Keze Wang, and Liang Lin, and the paper is titled "Can LLMs Debug Robots by Editing Their Weights?" In the best-performing experiment, across 335 held-out instructions containing the edited tokens, the success rate increased from 74.3% to 87.5%; for the other 430 episodes, the success or failure outcome remained unchanged in every single case. Here, "zero change" refers to the success or failure outcome of each episode.

The paper, the actual evaluated weight modifications, the code, and all 16 experiment records have been made public. The research addresses a specific question: can the manipulation capabilities a robot already possesses be invoked more reliably by different linguistic expressions through local weight modifications?

• Paper: https://arxiv.org/abs/2611.09376
• GitHub: https://github.com/jushengzhang/robot-weight-debugging
• Project introduction and process replay: https://jushengzhang.github.io/posts/robot-weight-debugging/zh.html

Putting the LLM in the Debugger's Seat

The research uses the π₀.₅ LIBERO checkpoint released by OpenPI. It is a vision-language-action model: it receives camera images, language instructions, and robot state, and outputs the next action to execute. The base model has already been pretrained and fine-tuned; this editing uses LIBERO manipulation tasks and LIBERO-Plus instruction rewrites as the test environment.[1][5][6]

In this experimental setup, Astra is responsible for inspecting and modifying the policy's weights. It is provided with a stateful Python workspace and callable forward-pass tools. Astra can inspect tensors, write replacement values, and then check the results, actions, or internal signals of simulation tasks. Once the modifications are finalized, actions are still computed by the edited robot policy.

Each language edit has a budget of 50 tool requests, and a single request can include multiple turns or inputs. Astra relies on feedback from practice samples to select candidates, and the final held-out test results are not returned to it. Therefore, "no gradient updates" does not mean "no feedback," nor does it mean the robot has never been trained.

This shifts the question from "has the robot learned this action" to "how is this instruction represented in the model, and can it more reliably invoke existing behaviors."

Animation 2 | A real process replay of GPT-6 Astra debugging robot weights. The researchers released the key nodes among the 50 tool requests, showing how Astra adjusts word embeddings based on simulation feedback, checks candidate effects, and ultimately retains 65 rows of modifications. On the left is the robot behavior before and after the edit on the same instruction; at the bottom are the practice sample results and tool feedback.

65 Rows, Changed Where Language Enters the Model

When an instruction enters the model, it is first split into tokens. A token does not necessarily equal a complete English word. The model then looks up the corresponding row vector from the embedding table by token ID, and the subsequent network processes these numerical representations.

The modifications Astra ultimately retained are concentrated in this embedding table. It selects a subset of tokens and moves their vectors toward more common task expressions, for example cooker→stove, basin→bowl, fermented→wine. Some rows use the target vector, some retain part of the original vector; when ambiguity arises, the mean of multiple target vectors can also be used.

The word cooking in the opening stove instruction is a concrete example. The embedding of the corresponding token was changed to 25% of the original vector plus 75% of the stove vector. The text input to the robot still contains cooking, but the numerical representation entering the subsequent network has changed. This row is part of the full set of 65 modifications, and the research did not evaluate its effect separately.

The remaining cases also preserve verifiable correspondences: in the wine bottle instruction, fermented and grapes are shifted toward wine; in the bowl-fetching instruction, rounded is shifted toward bowl, and uppermost is shifted toward top. The complete token mappings and paired recordings have all been made public.

The 65 rows change a total of 132,768 parameter values, accounting for 0.00367% of the exported checkpoint. Here, "rows" refers to the rows of the embedding table—not 65 parameters, nor 65 lines of code. The submitted language fix changes only these embedding values; the specific token IDs, target rows, and mixing ratios have all been made public.

Prior research has explored having general-purpose LLMs directly read and rewrite neural network weights, known as Weight-Space Programming.[2] Works such as ROME, MEMIT, and MEND have also studied the model editing problem.[8][9][10] This study examines the feasibility of an LLM constructing robot policy modifications with the aid of weight inspection and simulation feedback; it does not make direct comparisons with these editing algorithms.

765 held-out instructions: 48 fixes, 4 regressions

A single successful video cannot answer whether the modification is reliable. The study starts three separate edits from the same original checkpoint, with the three searches not inheriting notes or modifications from one another. Once candidates are determined, they are compared pairwise against the original policy on the same set of 765 held-out paraphrased instructions.

Table 1 | Held-out results of three independent edit searches. Testing uses the same 765 instructions and random seed 7; the three main tests remain significant after Holm multiple-comparison correction. The three searches share the same test set and cannot be treated as three independent test datasets.

The best-performing one, C, raises the success count from 660 to 704: 48 originally failing episodes succeed, while 4 originally successful episodes fail, for a net increase of 44 successes. Overall success rate rises from 86.3% to 92.0%, an improvement of 5.8 percentage points, with an exact paired test p=1.31×10⁻¹⁰.

Splitting instructions by whether they read the modified embedding rows makes the location of the change clearer: among the 335 episodes containing the modified tokens, the success count rises from 249 to 293, corresponding to 74.3%→87.5%; among the 430 episodes not containing these tokens, the success/failure outcomes remain unchanged one by one.

Table 2 | Token coverage grouping for edit C. The complete results count both fixes and regressions; the "zero change" for the 430 episodes refers only to per-episode success/failure outcomes.

Figure 2 | The outcome changes of edit C are concentrated in instructions containing the modified tokens. The success rate on the hardest level of paraphrasing rises from 59.8% to 78.2%; the numbers in the figure come from per-episode records.

This locality has a direct explanation: if an instruction does not read the modified rows, the text vectors obtained via lookup are identical. When inputs such as camera observations and sampling noise are fixed, the remaining unmodified network receives the same computational inputs. However, other new contexts containing these tokens may still be affected, so this cannot guarantee that the model has no side effects on all tasks.

From the stove to bowl-fetching and wine bottles: where do the fixes occur?

In the bowl-fetching task, the instruction describes the target and location using "darkcolored rounded container" and "uppermost storage compartment." The modified tokens rounded and uppermost are shifted toward the embeddings of bowl and top, respectively. In the same scene, the original policy times out at 220 steps, while the edited policy succeeds in 138 steps.

Animation 3 | Bowl-fetching: the original policy times out at 220 steps, and succeeds in 138 steps after editing.

The wine bottle task describes the target as a glass container holding "fermented grapes." The embeddings corresponding to fermented and grapes are shifted toward wine; this instruction also involves an atop→top modification. The original policy times out at 300 steps, and succeeds in 90 steps after editing. The above effects come from the entire 65-row modification; no separate ablation evaluation was performed for individual tokens.

Animation 4 | Wine bottle: the original policy times out at 300 steps, and succeeds in 90 steps after editing.

These are fix cases selected from the held-out tests. They help explain which modifications the model read; the overall conclusion still rests on the paired results across all 765 episodes, including the 4 regressions. The complete instructions, token mappings, and per-episode results can all be viewed in the public code repository.

After one high score, the modification must still hold up on the next batch of samples

The final 65-row modification also went through stages that looked successful and then exposed problems. During one editing process, the original model first succeeded 49 times on 80 harder paraphrases of the target task. Astra proposed modifications around lexical expression, and an early candidate achieved 69/80 on this diagnostic sample.

After switching to another batch of samples, this candidate fixed 8 failing episodes but also caused 6 originally successful episodes to regress. The high score on the old sample did not end this search. Astra continued to adjust ambiguous lexical mappings, forming a candidate involving 65 embedding rows, first freezing the modification, and then validating it with reserved practice samples.

After retrying runtime errors with the same seed and deduplicating, the number of successes among the 460 practice samples increased from 412 to 429. In the end, Astra kept this modification. Which tokens to select, which target vectors to move toward, and how much to move each row were all proposed and checked by Astra during the session.

These nodes illustrate how candidates were filtered, but scores from different practice batches cannot be connected into a training curve, and gains in practice cannot replace the final evaluation on the 765 held-out instructions. The complete trial-and-error process can be reviewed step by step in the public experiment logs.

Can this be achieved by simply swapping in a different set of synonyms?

A natural question is: does the gain come from Astra finding meaningful lexical correspondences, or would a random parameter perturbation also improve the results? The study ran two controls on the same 65 source rows.

The first moved along random directions, with movement distances matched to the original modifications. The second used the lexical correspondences found by Astra, fully replacing the source embeddings with the target embeddings, while also retaining the multi-target mean setting. What is compared here is a complete replacement of embedding values; the input strings were not rewritten.

Table 3 | Controls on the same set of 765 episodes. Full replacement had 2 runtime errors and the random control had 3, counted as failures in the table; the paper also reports an analysis excluding invalid pairs. The random direction is one control instance constructed for this study.

This random-direction control brought no improvement. Full replacement, however, already explained a substantial portion of the gain: the success rate reached 90.5%. This means that identifying which tokens need to be changed and where they should be moved is itself an important part of the fix.

Astra's edit C achieved a success rate of 92.0%, with a higher point estimate and fewer degradations; however, its direct comparison with full replacement on 763 valid pairs gave p=0.0614, which has not reached the 0.05 significance threshold. Therefore, the current results do not confirm that "partial mixing is significantly better than full replacement," nor do they prove that it is superior to independently compiled human word lists, prompt rewriting, or fine-tuning methods.

What could not be fixed also left a record

The 16 public robot experiments also covered camera viewpoints, backgrounds, language perturbations, instruction suffixes, as well as LIBERO-RECOVER failure recovery and LIBERO-PRO robustness search. [11][12] According to the paper's final decisions, all three language fixes were retained, and all had significant held-out gains; among the remaining 13 searches, 12 abandoned their candidates, and the only one counted as retained, the suffix follow-up, also had no significant held-out gain, p=0.542.

The failure recovery experiment illustrates the distinction well. One modification made the robot more often reopen its gripper after a failed grasp, but in a set of practice evaluations, the number of actual task completions instead dropped from 213/352 to 208/352. Some internal signal changed as expected, but the overall task did not improve.

In a reserved practice validation for one recovery search, the final round of reserved practice validation had 288 valid pairs, and the number of successes dropped from 166 to 164, so the modification was ultimately withdrawn. The study does not claim on this basis to have fixed failure recovery capability on the held-out set.

This makes retaining or withdrawing a candidate a result that requires checking just as much as proposing a modification. However, 12/13 only describes the decisions of this set of runs and is not enough to prove that LLMs generally know "which failures can be fixed." Historical runs also included cases where cleanup code was not executed and withdrawn candidates remained on disk; the framework bugs have been fixed, and the materials distinguish between archived files and final decisions.

Independent finding: numbers at the end of instructions also change success rates

In the FastWAM experiments, the study also found a kind of instruction-format sensitivity: adding metadata such as "view 0 0 100 2 354 initstate 0" to the end of a task description noticeably changed policy performance. [14]

On the same 210 paired episodes, FastWAM achieved a 70.0% success rate with pure task instructions, which dropped to 48.6% after adding the suffix. As a control, the tested π₀.₅ had an overall success rate of 88.6% both times, with 9 improvements and 9 degradations, so not all episode-level results were unchanged.

Figure 3 | Instruction-format sensitivity. Each checkpoint was compared on the same 210 episodes using pure task instructions versus suffixed instructions.

This finding has not yet become a stable weight fix. The first round of FastWAM archived candidates came with a withdrawal request, and subsequent experiments did not confirm a significant gain. Therefore, it needs to be viewed separately from the earlier π₀.₅ lexical fix: one is an observed evaluation phenomenon, and the other is an editing result supported by held-out evidence.

The comparison involves only two released checkpoints. World action models such as DreamZero and E0.5 provide a broader research context but are outside the scope of this suffix test. [15][16] It leaves evaluation with a very concrete question: when reporting success rates, what exactly is the complete instruction the model receives?

The first version is open-sourced, and real-robot validation is underway

The paper's title asks: can LLMs debug robots by modifying weights? This set of experiments gives a positive result for one class of lexical understanding failures: a small number of direct weight modifications enabled a batch of held-out instructions to more reliably invoke manipulation behaviors the policy already possessed. The complete causal pathway from language to action has not yet been localized layer by layer.

The scope of validation also has clear boundaries. The main held-out test used the same random seed; an earlier 200-instruction held-out subset did not detect a significant improvement, and the same candidate was subsequently evaluated on the existing full held-out split. The supplementary multi-seed checks came from the practice set, and the gains were uneven. All publicly released results come from simulation, and real-robot transfer or the acquisition of new motor skills has not yet been validated.

The currently open-sourced version is the first edition (v1); research and real-robot experiments for the second edition are ongoing. GitHub provides the actually evaluated 65-line sparse modifications and application scripts, the runtime framework, all 16 sessions, and turn-by-turn evidence. Readers can both inspect the final scores and follow the records to see how candidates were formed, which modifications were abandoned, and under what conditions the conclusions hold.

References

1. π₀.₅: a Vision-Language-Action Model with Open-World Generalization

2. Sparks of Weight-Space Programming: LLMs can natively read and write neural network weights

3. π₀: A Vision-Language-Action Flow Model for General Robot Control

4. OpenVLA: An Open-Source Vision-Language-Action Model

5. LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning

6. LIBERO-Plus: In-depth Robustness Analysis of Vision-Language-Action Models

7. Stable Language Guidance for Vision–Language–Action Models

8. Locating and Editing Factual Associations in GPT

9. Mass-Editing Memory in a Transformer

10. Fast Model Editing at Scale

11. LIBERO-RECOVER: Beyond Task Success Towards Failure Recovery in Robotic Manipulation Models

12. LIBERO-PRO: Towards Robust and Fair Evaluation of Vision-Language-Action Models Beyond Memorization

13. LoRA: Low-Rank Adaptation of Large Language Models

14. Fast-WAM: Do World Action Models Need Test-time Future Imagination?

15. World Action Models are Zero-shot Policies

16. E0.5: From Reconstructive to Future-Predictive 3D Foundations for Robotic World-Action Modeling

© THE END

For reprinting, please contact this official account for authorization.

Submissions or report inquiries: [email protected]

Open in WeChat