HOT RADAR
computeHEAT 9.0°

AI CURATED HOT STORY · 11/19/2026

Nvidia SEED Cuts Prefill Depth to Boost Per-GPU Throughput by Up to 42%

1 reports archived1 independent sourcesupdated 11/19/2026, 12:09:00
Synthesis & Latest Updates
1 Sources Cross-Validated

Nvidia's SEED exploits the read-write asymmetry of hybrid models: writing memories for future generation needs far less deep computation than predicting the next token. Non-final input tokens run only the first 31 of 48 layers, with a lightweight memory-write module producing KV cache and recurrent states for the remaining 17 layers, while decoding still uses all 48. On the decode side, recurrent states are stored in a dominant-direction plus low-rank decomposition, shrinking per-request state from 113.2MB to 29.2MB compressed. On GB300 GPUs with SGLang, Qwen3.8-Flash-Next gains 16%–42% per-GPU peak throughput, with frozen base weights and distillation-trained add-ons.

LATEST/Nvidia's SEED exploits the read-write asymmetry of hybrid models: writing memories for future generation needs far less deep computation than predicting the next token. Non-final input tokens run only the first 31 of 48 layers, with a lightweight memory-write module producing KV cache and recurrent states for the remaining 17 layers, while decoding still uses all 48. On the decode side, recurrent states are stored in a dominant-direction plus low-rank decomposition, shrinking per-request state from 113.2MB to 29.2MB compressed. On GB300 GPUs with SGLang, Qwen3.8-Flash-Next gains 16%–42% per-GPU peak throughput, with frozen base weights and distillation-trained add-ons.

Hourly heat curve

2 fully observed hours
Fewer than 3 fully observed hours; trend not plotted yet.

Coverage timeline

Total 1 reports · Latest first
  1. 机器之心 (微信公众号)T2·78 pts
    • Core insight is read-write asymmetry: writing memories for generation needs far less deep computation than next-token prediction
    • Prefill runs only 31 of 48 layers for non-final tokens while decoding stays full-depth, with frozen weights and distillation-trained add-ons
    • Recurrent states stored as dominant direction plus low-rank decomposition, cutting per-request state from 113.2MB to 29.2MB compressed
    • 16%–42% higher per-GPU peak throughput on GB300 with SGLang, and 23%–42% at equal P90 output speed