Build your own GPT-2
A single attention head·2 min

Recap

0 / 1

You just built one attention head end to end: score → mask → softmax → weighted sum. That single function is the engine of the whole model.

You are here. Engine ✓. Next you'll build the parts that surround it: identity for tokens (Ch 3 embeddings), stability (Ch 4 LayerNorm), the per-token compute (Ch 5 MLP) — then assemble 12 heads (Ch 6) into the full block (Ch 7) and stack it 12× (Ch 8).

A real GPT-2 block runs 12 of these heads in parallel over different 64-dim slices of each token's vector, then mixes their outputs — that's Chapter 6. First, the pieces that surround attention inside a block: embeddings (Chapter 3), LayerNorm (Chapter 4), and the MLP (Chapter 5).

Save your `runAttention`

It's called unchanged by runHead in Chapter 6 — you won't rewrite the mask/softmax again. Keep it a clean (q, k, v) -> out function.