机器之心 (微信公众号)T2·78 pts
- Core insight is read-write asymmetry: writing memories for generation needs far less deep computation than next-token prediction
- Prefill runs only 31 of 48 layers for non-final tokens while decoding stays full-depth, with frozen weights and distillation-trained add-ons
- Recurrent states stored as dominant direction plus low-rank decomposition, cutting per-request state from 113.2MB to 29.2MB compressed
- 16%–42% higher per-GPU peak throughput on GB300 with SGLang, and 23%–42% at equal P90 output speed