KV Cache

Key-Value Cache (KV Cache) stores the computed attention Key and Value tensor representations of historical tokens in GPU VRAM during autoregressive generation, avoiding redundant recalculations for each newly generated token. While essential for low-latency generation, KV Cache footprint scales linearly with context length and batch concurrency.