Termsslug: disaggregated-prefill-and-decode

Disaggregated Prefill and Decode

预填充解码分离

Splitting prompt prefill and token-by-token decode onto separate resources to improve throughput.

Curated Intelligence

1 items · Page 1/1
AWS Machine Learning BlogT1·Infra
62 pts
Amazon SageMaker HyperPod Adds Managed NVIDIA Dynamo Inference

Amazon SageMaker HyperPod Inference now supports managed NVIDIA Dynamo, pairing Dynamo's distributed inference with HyperPod's infrastructure management to address single-instance GPU memory limits and long-prompt slowdowns. The integration enables multi-node deployment, disaggregated prefill and decode, and KV-aware routing across engines such as vLLM, SGLang, and TensorRT-LLM, backed by EFA networking, coordinated GPU scheduling, and node health monitoring.