Where this goes next

Everything that makes modern inference fast is a strategy for managing the cache you just built, not replacing it:

• Continuous batching runs many sequences at once, each with its own cache, and swaps a finished sequence out for a waiting one mid-batch instead of waiting for the whole batch to drain — keeping the hardware busy.
• PagedAttention (vLLM) treats the cache like OS virtual memory: it splits each sequence's K/V into fixed-size pages so memory isn't pre-reserved in one contiguous block, which kills the fragmentation that otherwise wastes most of the cache.
• RadixAttention (SGLang) stores cached prefixes in a radix tree so sequences that share a prefix — a system prompt, a few-shot preamble — share the cached K/V instead of each holding its own copy. It's your Chapter 6 prefix reuse, generalized across an entire fleet of requests.
• Speculative decoding has a small draft model propose several tokens, then verifies them in one cached forward pass — turning several sequential decode steps into one, all riding on a cache that stays correct.

The serving vocabulary maps directly onto the two phases you built:

• TTFT (time to first token) is your prefill — the one-pass cost of filling the cache over the whole prompt before the first token comes out. Long prompts make TTFT large.
• ITL (inter-token latency) is your per-step decode — the cost of each cached step after the first. This is the O(n) work you proved, and it's why decode is cheap once the cache is warm.
• The memory bill is why long context is expensive: the cache grows linearly with sequence length (that kv_bytes figure), and at scale it, not compute, is usually the limit on how many sequences fit at once.
• The identical-prefix rule — same tokens, same cached K/V — is exactly the cache-hit condition behind prompt caching: reuse only holds when the prefix matches bit-for-bit, which is the invariant your Chapter 6 step enforced.