Toolsslug: global-inference-gateway

Global Inference Gateway

全局推理网关

Managed SageMaker HyperPod gateway that routes inference requests across clusters and Regions with automatic failover.

Curated Intelligence

1 items · Page 1/1
AWS Machine Learning BlogT1·Infra
58 pts
AWS Launches SageMaker HyperPod Global Inference Gateway

AWS introduced the SageMaker HyperPod Global Inference Gateway, extending its GPU-aware routing from a single cluster to a single managed endpoint spanning multiple clusters and Regions. The gateway selects clusters using real-time signals such as KV cache utilization and queue depth, carries requests over the AWS backbone, and provides automatic failover with SigV4 and IAM authentication. It targets capacity waste when one cluster queues requests while GPUs elsewhere sit idle.