Termsslug: prefill-decode-disaggregation

Prefill/Decode Disaggregation

预填充/解码分离

Architecture that places prefill and decode on separate resource pools to keep long inputs from blocking streaming.

Curated Intelligence

10 items · Page 1/1
Marktechpost (@Marktechpost)T2·Infra
62 pts
Sarvam Opens Token Factory Inference Platform in India

Indian AI firm Sarvam has opened its Sarvam Token Factory inference platform to all users, serving its own models and leading open-weight models from GPU clusters in India. The stack centers on a shared prefix cache with a cache-aware router, plus separate B200 pools for prefill and decode; Sarvam self-reports 6,177 tokens/s per GPU on Kimi K3.1 Pro. However, it trails Together on streaming throughput in some cases, and all figures are self-reported.

IT之家 · 智能时代T2·Infra
58 pts
LG U+ to Enter AI Token Business Next Year with Heterogeneous Chips

South Korean carrier LG U+ will enter the B2B AI token business next year, targeting enterprise AI transformation costs with three technologies: Harness Engineering for agent task paths, intelligent model routing, and a heterogeneous chip mix. Its in-house on-device model shrinks by a third while retaining over 95% of a general cloud model's performance, and a small-model-generates, large-model-verifies flow more than doubles token generation speed. The move means all three major Korean carriers will compete in the token economy.

Artificial Analysis (@ArtificialAnlys)T1·Infra
72 pts
DeepSeek V4.1 Flash Gains Up to 1.5x Per-Agent Speed on GB300 NVL72

Artificial Analysis' AA-AgentPerf benchmark shows DeepSeek V4.1 Flash on GB300 NVL72 rising from 596 to 848 tok/s for a single agent (1.42x), with roughly 1.6x throughput per GPU at ~80 tok/s per agent. The gains stem from a newer vLLM release with GB300-specific kernels, FlashInfer autotuning, fuller CUDA graph coverage, tuned speculative decoding, and co-locating prefill and decode on the same GPUs in NVIDIA's updated Dynamo + vLLM stack.

InfoQ 中文T2·Infra
68 pts
Iluvatar CoreX Releases TianGai 300 Benchmarks: Matches H200 in Specific Inference Scenarios

Iluvatar CoreX unveiled the TianGai 300 GPU, with third-party tests showing end-to-end throughput reaching 1.05x that of NVIDIA's H200 under PD-disaggregated deployment. The chip demonstrates strong performance in long-context, MoE, and online recommendation scenarios, while the company also previewed the next-gen edge device TY2000 with 600T compute power.

InfoQ 中文T2·Tools
48 pts
Paradigm Open-Sources ModelSphere for AI Deployment

Paradigm and Ruisi Zhilian launched ModelSphere, a free open-source one-stop AI deployment platform. It integrates heterogeneous chip management, model deployment, and runtime optimization, claiming a 285% throughput increase via AutoTune. The platform aims to lower barriers for enterprises using domestic chips (e.g., Ascend, Hygon) by providing graphical interfaces and enterprise-grade high availability.

机器之心 (微信公众号)T2·Infra
68 pts
HKUST and Huawei Propose BalanceRoute to Boost LLM Inference Throughput via Load Balancing

To address throughput bottlenecks caused by synchronization waits in data-parallel MoE deployments, HKUST and Huawei proposed BalanceRoute. By modeling load imbalance and converting short-term predictions into online routing decisions, the method improved decoding throughput by ~15% in a 96-GPU experiment. The work was accepted at NeurIPS 2026 and deployed in the MathBot-Pie system.

vLLM 官方博客T1·Tools
78 pts
vLLM Optimizes Agentic Serving with AgentX: Achieving 130K Tokens/GPU-sec on DeepSeek V4 Pro

vLLM optimizes its serving stack for agentic workloads by enhancing KV cache management, parallelism, and prefill/decode disaggregation to handle multi-turn sessions and long contexts. Benchmarked on SemiAnalysis's AgentX, vLLM achieves up to 130K total tokens per GPU-second on DeepSeek V4 Pro and 376 tokens/sec interactivity on MiniMax M3. This configuration delivers a 14.6x to 106x cost advantage over Opus 5 API pricing across major models.