AWS Machine Learning BlogT1·Infra
62 pts
Amazon SageMaker AI introduced KV cache-aware routing for real-time endpoints, using cache state reported by the model container to send each request to an instance holding a long cached prefix match that is not overloaded. It targets multi-turn chat, long shared prompts and RAG workloads, and is enabled via one environment variable plus the KV_AWARE routing strategy in the endpoint configuration, with no client API changes.