Prefill is decode's batched twin
Decode handles one token at a time: a single new query, attending over a cache you've already filled. Prefill handles the whole prompt in one batched forward under the full lower-triangular mask — every prompt position computes its own K/V, and they all land in the cache at once. The two passes are the same attention math at different shapes: decode is prefill with a one-row sequence. When prefill finishes you keep only the last token's logits — the prediction for the first token you'll generate — and throw away the rest. Positions run 0..prompt_len-1.
Two metrics
Prefill and decode map directly onto the two latencies every serving stack reports. Prefill ≈ time to first token (TTFT): you pay it once per request, it scans the entire prompt, and it's compute-bound — a big batched matmul over prompt_len positions. Per-step decode ≈ inter-token latency (ITL): paid once per generated token, processing a single new query against the cache, and it's memory-bandwidth-bound — you stream the whole cache through the math unit just to add one row. A long prompt makes TTFT hurt; a long generation makes ITL add up.
prompt ids: [ t0 t1 t2 t3 t4 ] positions 0..4, wpe[0..4]
│ │ │ │ │
▼ ▼ ▼ ▼ ▼
┌───────────────────────────┐
│ one masked attention pass │ full lower-triangular mask:
│ (token i attends to ≤ i) │ row i sees columns 0..i
└───────────────────────────┘
│ │ │ │ │
K/V for every position written → cache filled to length 5
│
▼
keep ONLY last row's logits → logits[50257]
(prediction for the first generated token,
which gets position 5 = cache length)
Decode in this course used a trivial mask — one new query attends to all cached keys, no triangle needed. Prefill needs the full lower-triangular mask because every row is a query at once: row i must see positions 0..i and not i+1..L-1, or earlier tokens would peek at the future and the cached K/V you write would be wrong.