AWS introduced distributed KV caching built on Amazon ElastiCache, adding a shared cache tier beside GPU fleets so any server can reuse KV state for days across sessions, routing shifts and failures. AWS claims the approach cuts self-hosted LLM inference costs by up to 40% and time to first token by up to 80%, directly addressing redundant prefill work in long-context agents.
Prefill
预填充
Inference phase that processes the input context at once and builds the KV cache, governing time-to-first-token.
Curated Intelligence
Nvidia's SEED exploits the read-write asymmetry of hybrid models: writing memories for future generation needs far less deep computation than predicting the next token. Non-final input tokens run only the first 31 of 48 layers, with a lightweight memory-write module producing KV cache and recurrent states for the remaining 17 layers, while decoding still uses all 48. On the decode side, recurrent states are stored in a dominant-direction plus low-rank decomposition, shrinking per-request state from 113.2MB to 29.2MB compressed. On GB300 GPUs with SGLang, Qwen3.8-Flash-Next gains 16%–42% per-GPU peak throughput, with frozen base weights and distillation-trained add-ons.
Cerebras has moved disaggregated inference into production, assigning prompt prefill to GPUs and output decode to its wafer-scale engines, yielding five times more tokens per megawatt at the same ultrafast speeds. The setup now runs on Cerebras Inference Cloud and is available to infrastructure partners for their own datacenters, alongside CS-4 and a planned 700MW of capacity.
Artificial Analysis has updated its API performance benchmarking methodology, establishing 'Time per Task' as the headline metric over traditional streaming speed. This shift reflects the growing importance of end-to-end completion efficiency in agentic workloads, incorporating full request latency including prefill and reasoning waits into the measurement.
A new study reveals that reasoning models' poor chain-of-thought control stems largely from a habitual start with the word "Okay". By banning this token and prefilling one compliant token, average instruction compliance increased by 34-41 percentage points. This suggests that CoT monitoring should distinguish between opening-word bias and general controllability.
TokenSpeed introduces a next-generation LLM inference engine optimized for agentic workloads, addressing multi-turn short requests and long-context reuse. On the Blackwell platform, it achieves 11% higher throughput than TensorRT-LLM while significantly improving KV Cache reuse efficiency to reduce costs.
Baseten has open-sourced joey-1, a 2B-parameter decision model that outputs calibrated probabilities via a single prefill pass without generating any text tokens. Scoring 63.30 on the Jev Decision Index, it ranks as the top model under 4B parameters with a P50 latency of just 20ms, offering a highly efficient solution for agent routing and classification tasks.
viiwork introduces KV-cache sharing to synchronize inference key-value caches between two RTX 5090 hosts over Ethernet. This mechanism eliminates the latency penalty of repeated prefilling when sessions migrate across nodes during load balancing, significantly improving response efficiency in multi-node deployments.
DeepSeek has open-sourced DeepInfer, its production-grade inference engine, under the Apache 2.0 license. Optimized for DeepSeek-V4.1-Flash on NVIDIA SM100 and Huawei Ascend 950 chips, it focuses on offline inference core technologies like DSpark speculative decoding and Bounded Replay prefill optimization. Although it lacks full serving stack components such as KV cache management and load balancing, the project garnered over 6,300 GitHub stars within 8 hours of release, highlighting significant community interest in high-performance underlying inference techniques.
Shi Technology unveiled TokenFleet, a distributed scheduling framework at the CCF HPC conference, designed to boost effective output of domestic compute clusters via optimized request routing, cross-datacenter capacity coordination, and mixed workload governance. Featuring three core technologies—TokenPilot, FleetMind, and SLOMax—the framework claims to reduce Time-to-First-Token (TTFT) latency by 32.4% while improving cost-efficiency per unit of valid output.