Nvidia SEED Cuts Prefill Depth to Boost Per-GPU Throughput by Up to 42%
Nvidia's SEED exploits the read-write asymmetry of hybrid models: writing memories for future generation needs far less deep computation than predicting the next token. Non-final input tokens run only the first 31 of 48 layers, with a lightweight memory-write module producing KV cache and recurrent states for the remaining 17 layers, while decoding still uses all 48. On the decode side, recurrent states are stored in a dominant-direction plus low-rank decomposition, shrinking per-request state from 113.2MB to 29.2MB compressed. On GB300 GPUs with SGLang, Qwen3.8-Flash-Next gains 16%–42% per-GPU peak throughput, with frozen base weights and distillation-trained add-ons.
Full Report

NVIDIA breaks down the "read-write asymmetry" of hybrid models.

Qwen3.8-Flash-Next has 48 layers. When processing a piece of input, SEED makes all tokens except the last one run only the first 31 layers, and then a lightweight memory-write component generates the KV Cache and recurrent state for the subsequent 17 layers. In the generation phase, each token still runs through the full 48 layers.
This change exploits a distinction: writing memories for subsequent generation requires far less deep computation than predicting the next token. SEED measured this difference across five trained hybrid large models and used it to reduce prefill computation.
There is another opportunity on the decoding side. The recurrent state that the model reads and writes at each step often contains a principal component along a nearly fixed direction, while the remaining content has a low-rank structure. SEED stores the state in this decomposed form, letting the model read and update it directly. On Qwen3.8-Flash-Next, the per-request recurrent state drops from 113.2MB to 29.2MB immediately after compression, averaging 43.4MB over the entire compression cycle.
On GB300 GPUs, with SGLang and coding-agent session traces, SEED improves Qwen3.8-Flash-Next's peak per-GPU throughput by 16%–42%; at the comparison point of equal P90 user end-to-end output rate, throughput improves by 23%–42%. Quality evaluations across five models and six checkpoints examined the accuracy cost of these changes. The original model weights remain frozen throughout, and the new components are trained via distillation.
The paper "Shallow Encoding Emits Deep Memories" has been released as an arXiv preprint. Mingyuan Ma and Yangmin Li co-led the work with equal contribution; the remaining authors are Harris Nover, Zheyu Fu, Po-Han Huang, Alex Settle, and Ashwin Nanjappa, all from NVIDIA.

- Paper link: https://arxiv.org/abs/2611.06084v1
Fixed-size state can still be expensive
Hybrid architectures use a small number of softmax attention layers to hold the KV Cache, and a large number of linear attention or state space layers to maintain recurrent state. The KV Cache grows with context, while the recurrent state is fixed in size—but the latter is not small: for Qwen3.8-Flash-Next, the per-request state occupies as much space as the attention cache of about 4500 tokens.
These states must be updated for every generated token, and they also contend for GPU memory with active requests and cached prefixes. SEED addresses the costs of the two phases separately: less deep computation during prefill, and less recurrent state storage during decoding. Both optimizations stem from memory structures that existing models already exhibit.
Why the first 31 layers suffice to write memories for the last 17 layers
During prefill, only the output of the last input token is used to predict the first response token. The remaining tokens contribute KV Cache and recurrent state written by each layer for subsequent generation. The researchers therefore examined the write path of the deep network separately.
The researchers fed shallow hidden states directly into the write paths of subsequent layers, skipping other deep computation, and compared this against early exit at the same depth: feeding that hidden state directly into the output head to predict the next token.

Table 1 Untrained comparison of Qwen3.8-Flash-Next at anchor depth k=31. Lower KL means the output distribution is closer to the original model. The write path uses the original layer weights directly; this is not the final distilled result. Source: paper Figure 2 (c).
The two results differ by about 141×. Shallow features are not yet sufficient to directly predict the next token, but they can already write usable memories for deep layers. This is the write-read asymmetry.
Figure 1 SEED's shallow prefill. Emitters write the KV Cache and recurrent state of subsequent layers from the anchor layer's output; the last input token still passes through the full network. Source: paper Figure 1 (a).
SEED implements this path with emitters. Attention layers retain the generation of keys, values, and necessary indices; recurrent layers additionally retain write strength, decay, short convolution, and scan. These components are initialized from the original layer parameters and adapted to shallow inputs via distillation, skipping the full deep attention readout and feedforward computation.
Including the emitters, Qwen3.8-Flash-Next's parameter-related computation for all input tokens except the last drops from 12.1 GFLOP to 9.1 GFLOP, a reduction of about 25%. Generation still uses the full network, and the deep memories are preserved.
Anchor depth cannot be chosen based on average loss alone. GLM-5.3-Flash at k=27 already has a very small distribution deviation, but it can only reproduce 269 of the original model's 595 identifiers; moving to k=31 allows 530 to be reproduced. SEED therefore selects depth using both distribution deviation and exact-reproduction tests, avoiding average metrics masking retrieval failures.
The highest-energy part of the state does not necessarily carry the most context
Qwen3.8-Flash-Next has 1728 Gated DeltaNet heads, each maintaining a 128×128 state matrix. After applying SVD to these matrices, the median energy share of the first singular component is about 85%, and the principal direction remains similar across uncorrelated inputs.
The researchers further estimate the average value direction of each head on general text, and call the projection of the state along this fixed direction the state sink. The median energy share of this projection is about 75%, which is a different statistic from the 85% given by SVD. After removing this direction, the remaining content is still very compact; the median head needs only about 6 directions to retain 90% of its energy.
Figure 2: The original state, the sink projection along the fixed direction, and the remaining content. In the example head shown, the sink accounts for 94% of the energy; the 85% and 75% in the main text are cross-head medians of different statistics. Energy refers to the sum of squares of the matrix elements. Source: paper Figure 3 (a–c) and Section 4.
High energy alone is not enough to show which parts are worth preserving. The paper conducted replacement experiments: replacing each head's sink with the average sink over other inputs increases the subsequent prediction loss by only about 0.002 nats/token; but if only the sink is kept at every subsequent step, the loss increases by about 1.66 nats/token. The former is a one-time intervention at the end of prefill, while the latter is a continuous intervention; together they show that the remaining working memory cannot be discarded.
After normalization, the sink mainly provides bias and gain, while the content useful for the current input mainly lies in the remaining part. SEED therefore stores the two separately: the fixed direction is shared by all requests, and each request stores only the sink coefficients and the low-rank factors of the working memory.

Figure 3: State decomposition and direct readout. v̄ is the fixed direction shared by each head; a, U, and V belong to the current request. The model computes the output using this set of factors, without needing to reconstruct the full state matrix for readout. Formulas are adapted from Section 5.1 of the paper.
Taking rank R=16 as an example, the storage per head immediately after compression changes from 128×128=16384 numbers to 17×128+16×128=4224 numbers. At the same precision, this is about one quarter of the original. The nearly 4× compression comes from the representation structure, not from converting the state to lower precision.
Compression alone is not enough
This set of factors must be continuously updatable
The recurrent state must be updated at every token. If the full matrix were reconstructed at every step and SVD were performed again, the storage savings would likely be offset by computation. SEED exploits the structure of the original update rule: at each step, it updates the sink coefficients, applies decay to the existing factors, and then appends one column to each of the two factors, thereby representing the new working memory.
The paper proves that the updates of Gated DeltaNet, Kimi Delta Attention, and Mamba-2 can all preserve this decomposition form. Between two compressions, the updates on the factors are mathematically exact; only the low-rank truncation introduces approximation. In Qwen3.8-Flash-Next, both R and the compression period W are 16; after generating 16 tokens, the grown factors are compressed back to the target rank.
This also explains why 29.2MB is not a constant footprint: the appended columns gradually increase storage, averaging 43.4MB over a period. Across five models, the compression ratio immediately after state compression is 2.0–3.9×, and the period average is 1.3–2.6×. These numbers measure the recurrent state representation; the attention KV Cache keeps its original format.
The error analysis targets the open loop: the compressed recurrence and the exact recurrence receive the same key, value, and other inputs, and the decay satisfies the contraction condition. In this case, the error bound does not grow with the number of compressions. Free generation also changes subsequent inputs; the paper additionally tracks long decoding: in models tracked to 16K steps, from the first 4K to the last 4K, the number of compressions increases to 4×, while the distribution shift caused by the state increases only to about 1.2–1.4×.
How much capability does the distilled model actually lose
SEED trains only the emitters and an optional prompt code, with the original model as a frozen teacher. Training first distills on fixed text, then runs one to two rounds on responses generated by the teacher and the current components; Nemotron-3.5-Super uses only the first-stage components. Anchor depth and state rank are selected using a unified probing criterion, independent of exam scores.
Quality tests cover Qwen3.8-Flash-Next, Nemotron-3.5-Lightning, Kimi-Linear, GLM-5.3-Flash, and Nemotron-3.5-Super, totaling six checkpoints and 11 benchmarks. Each question is compared pairwise with the original model using the same sampling seed; among the 66 "checkpoint × benchmark" combinations, 63 scores fall within two standard errors of the original model or higher.

Figure 4: Quality changes of six checkpoints on 11 benchmarks. The right panel separately examines the perplexity change caused by state compression, using exact prefill. The results support that evaluation performance is close to the original model, which does not mean all outputs are completely identical. Source: paper Figure 6.
Memory is saved
Why the system can actually serve more users
SEED's end-to-end validation on SGLang uses GB300 GPUs, replaying coding agent sessions recorded by InferenceX. Each concurrency point runs for 900 seconds; the original model and SEED use the same hardware, recurrent states are stored in FP32, and the measurement includes emitters and periodic compression overhead. The table below enables both shallow prefill and shallow state.

Table 2 Per-GPU serving throughput of Qwen3.8-Flash-Next. Peaks are compared at each configuration's highest measured throughput; the P90 column is compared at the same user-side end-to-end output rate point selected in the paper, with specific points given in Section 7 of the paper. TP denotes tensor parallelism; 1P consists of one prefill instance and one decode instance, each using two GPUs. Source: Table 5 of the paper.
Under NVFP4 single-GPU aggregated deployment, the original model peaks at about 80.2K tokens/s/GPU, and SEED at about 96.0K. Under bf16 prefill-decode disaggregated deployment, the peak rises from about 33.9K to 48.1K. The throughput statistics in the table include both input and output tokens, and cache-hit input is also counted; they cannot be treated as the generation speed for a single user.
Under the test workloads for these MoE models, weight reads during the decode phase still account for a large share, and the smaller state mainly improves serving through capacity rather than merely moving less state data.
SGLang's hybrid-model prefix cache and in-flight requests share a single recurrent state pool. When the state pool is full, the system evicts cached prefixes; if a new request uses that context again, it must be prefilled anew. The higher the concurrency, the more pronounced the link between state occupancy and recomputation.
At the measurement point of NVFP4 TP1 with 40 concurrent sessions, the original model's prefix cache hit rate is about 55%, and SEED's is about 85%. The smaller state frees up space for both active requests and reusable prefixes, reducing recomputation; shallow prefill in turn lowers the computation for the missed portion.
In PD-disaggregated deployment, the main bottleneck shifts to the prefill instance. Enabling only shallow prefill raises peak throughput by about 35%; compressing only the state raises it by about 5%; enabling both raises it by about 42%. The source of gains across deployments can thus be distinguished: aggregated serving is more affected by state pool capacity, while PD disaggregation relies more on reduced prefill computation.
The P90 user-side end-to-end output rate also counts queuing and time-to-first-token. Comparing throughput under the same P90 condition is meant to compare how much load can be handled under a similar interactive experience; the 23%–42% corresponds to the comparison points selected in the paper, not a fixed increase at arbitrary rates. At low load, the throughput difference between the two configurations is very small, and end-to-end serving validation is currently concentrated on Qwen3.8-Flash-Next.
How can memory for hybrid models be designed?
SwiftKV and POP previously explored generating Transformer deep-layer KV Cache from shallow features. SEED extends the problem to hybrid models: prefill needs to write deep recurrent states, decode needs to continue updating on the compressed representation, and the original model weights remain frozen.
These two findings respectively change how memory is computed and how it is stored. Deep networks still play a role during generation, but writing memory for input tokens does not necessarily require the same computational depth; the full state matrix still defines the model update, but an implementation can store and update a smaller set of factors. The model already possesses these structures, and SEED turns them into measurable serving gains.
The gains also have clear scope. SEED does not compress the attention KV Cache; in long contexts, if the KV Cache dominates GPU memory, state compression will contribute less to total memory. The prompt code prefix storage discussed in the paper is still a future direction and is not included in the serving throughput in this article. Code, newly added components, and training data are listed in the paper as materials planned for release.
Author Bios
Mingyuan Ma
From the NVIDIA Compute Architecture / DL Compute team, focusing on performance modeling, inference optimization, and performance prediction for next-generation models of large-model inference. Participated in inference performance modeling and prediction for projects such as Nemotron 3 Super, Nemotron 3 Ultra, and Nemotron 3.5 Lightning, Super, and Bolt, providing quantitative basis for model design and system optimization.
Yangmin Li
From the NVIDIA Compute Architecture / LLM Inference Performance team, focusing on performance optimization of large-model inference systems, and one of the main contributors to the open-source inference engine SGLang. Works on directions such as PD disaggregation, speculative decoding, KV Cache management, and fused kernels to improve throughput under real workloads and reduce interactive latency. Participated in Day-0 support and inference optimization for mainstream open-source models such as DeepSeek, Qwen, Kimi, and GLM, promoting out-of-the-box and efficient operation of these models on NVIDIA's latest GPUs.

© THE END
For reprinting, please contact this official account for authorization.
Submission or reporting: [email protected]