Context Caching

Context Caching is an optimization technique where the server persists computed KV Cache for long prompt prefixes (such as system instructions, codebases, or reference documents). Subsequent requests sharing the same prefix reuse the cached states, drastically reducing time-to-first-token (TTFT) and inference compute costs.