MIT Proposes Weight-Space Programming: LLMs Directly Edit Model Weights to Boost Math Skills
MIT CSAIL researchers introduced 'Weight-Space Programming,' a method where GPT-6 Astra directly reads and edits the neural network weights of Qwen3.5-0.8B. Without gradient descent or traditional training loops, adjusting just ~0.016% of parameters eliminated reasoning loops, boosting MATH-500 scores from 7.9% to 48.1%, offering a new paradigm for model optimization and interpretability.
Related Entities
Full Report
IT Home November 3 News — Researchers from the MIT Computer Science and Artificial Intelligence Laboratory (MIT CSAIL) have proposed a new method called "Weight-Space Programming."
They enabled OpenAI's GPT-6 Astra to directly analyze and modify the neural network weights of Alibaba Cloud's popular Tongyi Qianwen model, Qwen3.5-0.8B. By adjusting only approximately 0.016% of the parameters, they improved the model's score on the MATH-500 benchmark from 7.9% to 48.1%, achieving roughly a 6.1x increase. Reportedly, the entire process did not use gradient descent or traditional training loops.
This research was conducted jointly by Rikiya Takehi, Jayden Teoh, Alex Zhang, and Omar Khattab. Takehi published the study on his blog on November 2 local time, terming it "Weight-Space Programming." The core idea is to allow large language models to directly read, understand, and modify the weights of another neural network, rather than adjusting model parameters through conventional training pipelines.
In specific experiments, GPT-6 Astra did not simply randomly adjust a small number of parameters; instead, it first identified reasoning loop issues within Qwen3.5-0.8B.
Researchers found that the model tended to repeatedly generate identical content when answering math problems, causing many responses to exhaust the 32,768-token length limit. In the initial 96 test responses, 85 hit this upper limit.
The research team provided GPT-6 Astra with a Python environment capable of reading and modifying model weights, allowing it to perform a limited number of forward inferences using practice questions from the GSM8K and MATH training sets. Astra subsequently located a repetitive direction written by the MLPs (Multi-Layer Perceptrons) at layers 15 and 19. By modifying partial weights in these two layers, it weakened the signals causing the reasoning loops.
Ultimately, Astra modified 131,072 weights, accounting for approximately 0.016% of Qwen3.5-0.8B's roughly 800 million parameters. In experiments allowing up to 50 rounds of forward inference, the model's score on MATH-500 rose from 7.9% to 48.1%, and its score on GSM8K increased from 13.8% to 67.1%.
Simultaneously, the instruction-following capability of the modified model also improved. On the IFEval test, which measures model instruction-following performance, the modified model's score rose from 21.0% to 62.3%, approximately tripling the original result. Meanwhile, the proportion of math test responses that failed to complete due to hitting the generation length limit dropped from 93% to 33.1%.
It is important to note that this experiment was not entirely devoid of computational processes. GPT-6 Astra used a Python environment to analyze and modify weights and performed up to 50 rounds of forward inference; however, it did not use gradient descent, backpropagation, or traditional training loops to update parameters.
Researchers also conducted control experiments using Gemma 3 270M. Since this model rarely exhibited reasoning loops originally, Astra could not significantly improve its mathematical capabilities through similar local weight modifications. The MATH-500 score increased only slightly from 6.8% to 7.7% at most. Based on this, the research team pointed out that Weight-Space Programming can remove barriers encountered during the utilization of a model's existing capabilities but may not necessarily create capabilities that the model inherently lacks.
The study included other experiments, such as enabling GPT-6 Astra to identify games trained on via neural network weights alone, determine the training language of small language models, and directly write neural networks capable of running games. Researchers believe this approach offers new perspectives for neural network interpretability research and model optimization.
References:
Advertisement Disclaimer: External links contained in this article (including but not limited to hyperlinks, QR codes, passwords, etc.) are provided to convey additional information and save selection time. Results are for reference only. All IT Home articles containing external links include this disclaimer.