Termsslug: model-souping

Model Souping

模型汤

Merging weights of multiple fine-tuned models, guided by benchmarks, to improve generalization.

Curated Intelligence

1 items · Page 1/1
Hacker News AIT2·Models
74 pts
Editing Just 0.016% of Weights Cures Model Reasoning Loops

The author built an interactive WebGPU walkthrough of "weight-space programming": without gradient descent, editing only 0.016% (131,072 of ~800M) of a small Qwen model's ffn_down output weights across layers 15 and 19 largely cures its self-verification loop. On identical prompts, runs hitting the 32k-token cap fell from 93.0% to 33.1%, while MATH-500 accuracy rose from 7.9% to 48.1%. Unlike broad model souping, the method targets a few specific failure cases and the neurons behind them, and is pitched as a fit for agentic propose-test-feedback loops.