AWS Machine Learning BlogT1·Infra
72 pts
AWS introduced distributed KV caching built on Amazon ElastiCache, adding a shared cache tier beside GPU fleets so any server can reuse KV state for days across sessions, routing shifts and failures. AWS claims the approach cuts self-hosted LLM inference costs by up to 40% and time to first token by up to 80%, directly addressing redundant prefill work in long-context agents.