vLLM Adds Push-Based KV Transfer to Cut Latency in Disaggregated Inference

2 reports archived1 independent sourcesupdated 10/20/2026, 00:05:17

Synthesis & Latest Updates

1 Sources Cross-Validated

vLLM introduces a push-based KV cache transfer mechanism via the NIXL connector to address communication bottlenecks in prefill-decode disaggregation. By leveraging RDMA for direct data pushing, this approach significantly reduces Time-to-First-Token (TTFT) and Time-per-Output-Token (TPOT), enhancing efficiency for large-scale LLM serving.

LATEST/vLLM introduces a push-based KV cache transfer mechanism via the NIXL connector to address communication bottlenecks in prefill-decode disaggregation. By leveraging RDMA for direct data pushing, this approach significantly reduces Time-to-First-Token (TTFT) and Time-per-Output-Token (TPOT), enhancing efficiency for large-scale LLM serving.

HEAT TRENDHourly heat curve

2 fully observed hours

Fewer than 3 fully observed hours; trend not plotted yet.

TIMELINECoverage timeline

Total 2 reports · Latest first

AWS Machine Learning BlogT1·Oct 20, 00:0568 pts
vLLM Adds Push-Based KV Transfer to Cut Latency in Disaggregated Inference
- Replaces pull-based mechanisms with push-mode transfers using RDMA to accelerate KV cache movement from prefill to decode instances.
- Reduces TTFT and TPOT, improving resource utilization and throughput in prefill-decode disaggregated architectures.
View Related Reports & Analysis Original Source
AWS Machine Learning BlogT1·Oct 20, 00:0378 pts
vLLM Adds Bi-directional KV Transfer to Cut TTFT for Agentic Workloads
- Traditional P-D disaggregation uses one-way KV transfer, forcing recomputation of history in multi-turn chats
- New mechanism enables Decode-to-Prefill KV return, reusing session state
- Significantly reduces Time To First Token (TTFT) for multi-turn agent tasks
View Related Reports & Analysis Original Source

HEAT TRENDHourly heat curve

TIMELINECoverage timeline

vLLM Adds Push-Based KV Transfer to Cut Latency in Disaggregated Inference

vLLM Adds Bi-directional KV Transfer to Cut TTFT for Agentic Workloads