Termsslug: kv-cache-utilization

KV Cache Utilization

KV 缓存利用率

Metric of key-value cache occupancy in inference serving, commonly used to gauge instance load and routing decisions.

Curated Intelligence

1 items · Page 1/1
AWS Machine Learning BlogT1·Infra
58 pts
AWS Launches SageMaker HyperPod Global Inference Gateway

AWS introduced the SageMaker HyperPod Global Inference Gateway, extending its GPU-aware routing from a single cluster to a single managed endpoint spanning multiple clusters and Regions. The gateway selects clusters using real-time signals such as KV cache utilization and queue depth, carries requests over the AWS backbone, and provides automatic failover with SigV4 and IAM authentication. It targets capacity waste when one cluster queues requests while GPUs elsewhere sit idle.