AWS introduced the SageMaker HyperPod Global Inference Gateway, extending its GPU-aware routing from a single cluster to a single managed endpoint spanning multiple clusters and Regions. The gateway selects clusters using real-time signals such as KV cache utilization and queue depth, carries requests over the AWS backbone, and provides automatic failover with SigV4 and IAM authentication. It targets capacity waste when one cluster queues requests while GPUs elsewhere sit idle.
SageMaker HyperPod
AWS managed cluster service for large-scale model training and inference with elastic GPU capacity and resilient scheduling.
Curated Intelligence
AWS reviewed its 2026 SageMaker Inference updates, including new routing strategies for HyperPod Inference, broader model-serving framework support and expanded multi-Region networking, with the Inference Gateway handling routing and managed NVIDIA Dynamo added as a new serving framework. For managed endpoints, inference recommendations now run from SageMaker AI Studio, the Python SDK and coding agents, and endpoints can route requests across Regions and accounts, helping teams re-select configurations after traffic shifts, pack more models onto paid-for GPUs and fix idle capacity across Regions.
Amazon SageMaker HyperPod Inference now supports managed NVIDIA Dynamo, pairing Dynamo's distributed inference with HyperPod's infrastructure management to address single-instance GPU memory limits and long-prompt slowdowns. The integration enables multi-node deployment, disaggregated prefill and decode, and KV-aware routing across engines such as vLLM, SGLang, and TensorRT-LLM, backed by EFA networking, coordinated GPU scheduling, and node health monitoring.
AWS has added built-in Data Capture to SageMaker HyperPod, recording live inference inputs and outputs to S3 across three independent tiers with zero impact on model availability. A home-appliance support router walkthrough shows the full flywheel: capture traffic, surface a new-product blind spot via Athena, label records with SQL, then fine-tune and measure accuracy before and after.