Reuse the prefix
Many requests share a prefix: a fixed system prompt, a block of few-shot examples, a long document you ask several questions about. Prefilling that prefix is the expensive part — and it's identical every time. So do it once: prefill the prefix, save the resulting cache, and for each request that starts with that prefix, reuse the saved cache and only decode the suffix. The shared work happens a single time, no matter how many suffixes follow. That's why a long system prompt doesn't cost you a full prefill every turn.
prefix = [system prompt + few-shot] (P tokens)
savePrefixCache(prefix) ──► saved cache: positions 0 .. P-1 populated, all 12 layers
request A: reuse(saved) ► decode suffix_A at positions P, P+1, ...
request B: reuse(saved) ► decode suffix_B at positions P, P+1, ...
request C: reuse(saved) ► decode suffix_C at positions P, P+1, ...
^ the P-token prefill is paid once, then skipped
The catch: positions must continue
The saved cache already occupies positions 0 .. P-1. When you decode the suffix on top of it, the suffix's first token is the P-th token in the sequence — its absolute position is P, not 0. GPT-2 adds a learned absolute position embedding wpe[position] at the input, and every cached K/V was computed with the prefix's positions baked in. Continue the numbering from P and the cached path reproduces a from-scratch recompute of the full prefix + suffix prompt exactly. Reset it and you double-use the early positions — the suffix thinks it's at the start of the sequence while the cache thinks otherwise.
A saved prefix cache of length P is valid for a suffix only if the suffix's first token gets position P, not 0. Reset the positions and you silently corrupt the output — no crash, no error, just wrong logits, because GPT-2's wpe[position] term no longer matches the positions the cached K/V were built for. We grade a “wrong if positions reset” case in the next step so this rule isn't merely assumed — the position-reset variant must fail to match recompute.
There's a second, quieter rule. Decoding a suffix appends the suffix's K/V to the cache. If you reuse the same saved-cache object across two suffixes, the second request inherits the first request's appended tokens — its prefix is no longer just the prefix. So each reuse must work on its own copy of the saved cache (clone it). The saved cache stays pristine; the suffix's growth lives on the throwaway copy.