Termsslug: prefill

Prefill

预填充

Inference phase that processes the input context at once and builds the KV cache, governing time-to-first-token.

Curated Intelligence

19 items · Page 1/2
AWS Machine Learning BlogT1·Infra
72 pts
AWS Launches Distributed KV Caching to Cut LLM Inference Costs

AWS introduced distributed KV caching built on Amazon ElastiCache, adding a shared cache tier beside GPU fleets so any server can reuse KV state for days across sessions, routing shifts and failures. AWS claims the approach cuts self-hosted LLM inference costs by up to 40% and time to first token by up to 80%, directly addressing redundant prefill work in long-context agents.

机器之心 (微信公众号)T2·Infra
78 pts
Nvidia SEED Cuts Prefill Depth to Boost Per-GPU Throughput by Up to 42%

Nvidia's SEED exploits the read-write asymmetry of hybrid models: writing memories for future generation needs far less deep computation than predicting the next token. Non-final input tokens run only the first 31 of 48 layers, with a lightweight memory-write module producing KV cache and recurrent states for the remaining 17 layers, while decoding still uses all 48. On the decode side, recurrent states are stored in a dominant-direction plus low-rank decomposition, shrinking per-request state from 113.2MB to 29.2MB compressed. On GB300 GPUs with SGLang, Qwen3.8-Flash-Next gains 16%–42% per-GPU peak throughput, with frozen base weights and distillation-trained add-ons.

Hacker News AIT2·Infra
78 pts
Cerebras Deploys Disaggregated Inference Mixing GPUs and Wafer-Scale Engines

Cerebras has moved disaggregated inference into production, assigning prompt prefill to GPUs and output decode to its wafer-scale engines, yielding five times more tokens per megawatt at the same ultrafast speeds. The setup now runs on Cerebras Inference Cloud and is available to infrastructure partners for their own datacenters, alongside CS-4 and a planned 700MW of capacity.

Artificial Analysis (@ArtificialAnlys)T1·Tools
68 pts
Artificial Analysis Shifts Benchmarking to Time per Task Metric

Artificial Analysis has updated its API performance benchmarking methodology, establishing 'Time per Task' as the headline metric over traditional streaming speed. This shift reflects the growing importance of end-to-end completion efficiency in agentic workloads, incorporating full request latency including prefill and reasoning waits into the measurement.

Rohan Paul (@rohanpaul_ai)T2·Models
78 pts
Harvard and MIT Study: Banning 'Okay' Boosts Reasoning Model Control

A new study reveals that reasoning models' poor chain-of-thought control stems largely from a habitual start with the word "Okay". By banning this token and prefilling one compliant token, average instruction compliance increased by 34-41 percentage points. This suggests that CoT monitoring should distinguish between opening-word bias and general controllability.

InfoQ 中文T2·Infra
68 pts
TokenSpeed Unveils Next-Gen LLM Inference Engine for Agents

TokenSpeed introduces a next-generation LLM inference engine optimized for agentic workloads, addressing multi-turn short requests and long-context reuse. On the Blackwell platform, it achieves 11% higher throughput than TensorRT-LLM while significantly improving KV Cache reuse efficiency to reduce costs.

Marktechpost (@Marktechpost)T2·Models
78 pts
Baseten Open-Sources joey-1: A 2B Decision Model for Zero-Token Routing

Baseten has open-sourced joey-1, a 2B-parameter decision model that outputs calibrated probabilities via a single prefill pass without generating any text tokens. Scoring 63.30 on the Jev Decision Index, it ranks as the top model under 4B parameters with a P50 latency of just 20ms, offering a highly efficient solution for agent routing and classification tasks.

Hacker News AIT2·Infra
78 pts
viiwork Enables Cross-Host KV Cache Sharing over Ethernet

viiwork introduces KV-cache sharing to synchronize inference key-value caches between two RTX 5090 hosts over Ethernet. This mechanism eliminates the latency penalty of repeated prefilling when sessions migrate across nodes during load balancing, significantly improving response efficiency in multi-node deployments.

Rohan Paul (@rohanpaul_ai)T2·Open Source
85 pts
DeepSeek Open-Sources DeepInfer Inference Engine, Hits 6.3K Stars in 8 Hours

DeepSeek has open-sourced DeepInfer, its production-grade inference engine, under the Apache 2.0 license. Optimized for DeepSeek-V4.1-Flash on NVIDIA SM100 and Huawei Ascend 950 chips, it focuses on offline inference core technologies like DSpark speculative decoding and Bounded Replay prefill optimization. Although it lacks full serving stack components such as KV cache management and load balancing, the project garnered over 6,300 GitHub stars within 8 hours of release, highlighting significant community interest in high-performance underlying inference techniques.

量子位官网T2·Infra
48 pts
Shi Technology Launches TokenFleet Scheduling Framework, Cutting TTFT by 32.4%

Shi Technology unveiled TokenFleet, a distributed scheduling framework at the CCF HPC conference, designed to boost effective output of domestic compute clusters via optimized request routing, cross-datacenter capacity coordination, and mixed workload governance. Featuring three core technologies—TokenPilot, FleetMind, and SLOMax—the framework claims to reduce Time-to-First-Token (TTFT) latency by 32.4% while improving cost-efficiency per unit of valid output.