Nvidia's SEED exploits the read-write asymmetry of hybrid models: writing memories for future generation needs far less deep computation than predicting the next token. Non-final input tokens run only the first 31 of 48 layers, with a lightweight memory-write module producing KV cache and recurrent states for the remaining 17 layers, while decoding still uses all 48. On the decode side, recurrent states are stored in a dominant-direction plus low-rank decomposition, shrinking per-request state from 113.2MB to 29.2MB compressed. On GB300 GPUs with SGLang, Qwen3.8-Flash-Next gains 16%–42% per-GPU peak throughput, with frozen base weights and distillation-trained add-ons.
Termsslug: recurrent-state
Recurrent State
循环状态
Fixed-size state maintained by linear-attention and state-space layers, read and updated per token.
Curated Intelligence
机器之心 (微信公众号)T2·Infra
78 pts