Indian AI firm Sarvam has opened its Sarvam Token Factory inference platform to all users, serving its own models and leading open-weight models from GPU clusters in India. The stack centers on a shared prefix cache with a cache-aware router, plus separate B200 pools for prefill and decode; Sarvam self-reports 6,177 tokens/s per GPU on Kimi K3.1 Pro. However, it trails Together on streaming throughput in some cases, and all figures are self-reported.
Prefill/Decode Disaggregation
预填充/解码分离
Architecture that places prefill and decode on separate resource pools to keep long inputs from blocking streaming.
Curated Intelligence
South Korean carrier LG U+ will enter the B2B AI token business next year, targeting enterprise AI transformation costs with three technologies: Harness Engineering for agent task paths, intelligent model routing, and a heterogeneous chip mix. Its in-house on-device model shrinks by a third while retaining over 95% of a general cloud model's performance, and a small-model-generates, large-model-verifies flow more than doubles token generation speed. The move means all three major Korean carriers will compete in the token economy.
Artificial Analysis' AA-AgentPerf benchmark shows DeepSeek V4.1 Flash on GB300 NVL72 rising from 596 to 848 tok/s for a single agent (1.42x), with roughly 1.6x throughput per GPU at ~80 tok/s per agent. The gains stem from a newer vLLM release with GB300-specific kernels, FlashInfer autotuning, fuller CUDA graph coverage, tuned speculative decoding, and co-locating prefill and decode on the same GPUs in NVIDIA's updated Dynamo + vLLM stack.
Iluvatar CoreX unveiled the TianGai 300 GPU, with third-party tests showing end-to-end throughput reaching 1.05x that of NVIDIA's H200 under PD-disaggregated deployment. The chip demonstrates strong performance in long-context, MoE, and online recommendation scenarios, while the company also previewed the next-gen edge device TY2000 with 600T compute power.
vLLM, utilizing P/D disaggregation and Wide-EP, achieves 954K tokens/s per GPU on an NVIDIA GB300 NVL72 rack for DeepSeek-V4.1-Flash. Its decode throughput is within 6% of DeepSeek's specialized engine, DeepInfer, demonstrating that a general-purpose inference engine can match model-specific performance through targeted engineering at scale.
vLLM introduces a push-based KV cache transfer mechanism via the NIXL connector to address communication bottlenecks in prefill-decode disaggregation. By leveraging RDMA for direct data pushing, this approach significantly reduces Time-to-First-Token (TTFT) and Time-per-Output-Token (TPOT), enhancing efficiency for large-scale LLM serving.
Paradigm and Ruisi Zhilian launched ModelSphere, a free open-source one-stop AI deployment platform. It integrates heterogeneous chip management, model deployment, and runtime optimization, claiming a 285% throughput increase via AutoTune. The platform aims to lower barriers for enterprises using domestic chips (e.g., Ascend, Hygon) by providing graphical interfaces and enterprise-grade high availability.
To address throughput bottlenecks caused by synchronization waits in data-parallel MoE deployments, HKUST and Huawei proposed BalanceRoute. By modeling load imbalance and converting short-term predictions into online routing decisions, the method improved decoding throughput by ~15% in a 96-GPU experiment. The work was accepted at NeurIPS 2026 and deployed in the MathBot-Pie system.
AWS introduces Ray LLM Deep Learning Containers enabling single LLM deployment across multiple GPU nodes on Amazon EKS. The solution exposes a unified OpenAI-compatible endpoint while natively supporting prefill/decode disaggregation to simplify distributed inference architecture setup and maintenance.
vLLM optimizes its serving stack for agentic workloads by enhancing KV cache management, parallelism, and prefill/decode disaggregation to handle multi-turn sessions and long contexts. Benchmarked on SemiAnalysis's AgentX, vLLM achieves up to 130K total tokens per GPU-second on DeepSeek V4 Pro and 376 tokens/sec interactivity on MiniMax M3. This configuration delivers a 14.6x to 106x cost advantage over Opus 5 API pricing across major models.