AI engineering · 03 of 42

KV Cache

Reuse past attention work

Scroll

Reuse past attention work

Attention works by comparing the token being generated against every token before it. Done naively, producing the thousandth token means redoing the work for the previous nine hundred and ninety-nine.

A KV cache stores that work. The keys and values for each token are computed once and kept, so each new token reads the cache and adds its own entry rather than rebuilding the history.

The cost moved rather than vanished: it is now memory. A longer conversation means a larger cache, and cache is what fills the GPU. That is the real reason a provider caps how long your context can be and how many people it can serve at once — not the arithmetic, the storage.

Inference
DO THE WORK ONCE, KEEP IT past tokens t1 t2 ... t(n-1) tn KV cache keys and values, stored once reused next token updated cache appends its own state Without it, every new token re-reads the whole conversation. The cost is memory: a longer context needs a bigger cache, and that is what limits how many people one GPU can serve at once.
Past tokens computed once and stored, then reused as each new token appends its own state.