Termsslug: kv-cache-aware-routing

KV Cache-Aware Routing

KV 缓存感知路由

Inference routing strategy that dispatches requests based on KV cache prefix hits and instance load.

Curated Intelligence

1 items · Page 1/1
AWS Machine Learning BlogT1·Infra
62 pts
Amazon SageMaker Adds KV Cache-Aware Routing for Real-Time Endpoints

Amazon SageMaker AI introduced KV cache-aware routing for real-time endpoints, using cache state reported by the model container to send each request to an instance holding a long cached prefix match that is not overloaded. It targets multi-turn chat, long shared prompts and RAG workloads, and is enabled via one environment variable plus the KV_AWARE routing strategy in the endpoint configuration, with no client API changes.