Pre-LN + residual stream
You now have every piece of a block: LayerNorm (Chapter 4), the MLP (Chapter 5), and multi-head attention (Chapter 6). This chapter snaps them together into one GPT-2 block — the unit that gets stacked 12 times to make the whole model.
Here's the one idea to hold onto. Think of x [T, 768] as a shared notebook that flows through all 12 blocks like a river. Each sublayer reads a clean (normalized) copy of the notebook, computes a small correction, and adds it back to the stream — it never overwrites the notebook, only annotates it. That running notebook is the residual stream, and it is the single most important idea in modern transformers.
How a block works
GPT-2 uses pre-norm blocks with residual connections. For input x [T, 768] and block layer:
a = MHA( LayerNorm(x, ln_1.weight, ln_1.bias), layer )
x = x + a # residual 1
m = MLP( LayerNorm(x, ln_2.weight, ln_2.bias), layer )
x = x + m # residual 2
return x # [T, 768]
x ──┬───────────────────────────(+)──┬────────────────────────────(+)──▶ x'
│ ▲ │ ▲
└─ ln_1 ─▶ MHA ──────────────┘ └─ ln_2 ─▶ MLP ────────────────┘
(read a clean copy, add a correction back to the stream — shape stays [T,768] throughout)
Two things to notice. The norm happens before each sublayer (pre-norm), and the residual adds the sublayer's output back to its input — the value before the norm, not after. The shape is [T, 768] from start to finish; that invariance is exactly what lets you stack the block 12 times without ever changing a dimension.
It's x = x + sublayer(ln(x)), not x = ln(x) + sublayer(ln(x)). The identity path carries the un-normalized stream forward — normalizing it would throw away everything the stream has accumulated so far and defeat the whole purpose. The normalized copy is a throwaway input to the sublayer; the thing you add back into is the original x.