Why normalize
Picture each token's 768 numbers as a noisy audio signal whose volume drifts as it passes through 12 layers — quietly louder here, lopsided there. LayerNorm is the automatic gain control: re-center each token to mean 0, rescale to unit spread, then let learned γ/β set the 'correct' level back. It runs before every sublayer — 25 times total (ln_1 + ln_2 per block × 12, plus one final ln_f).
How LayerNorm works
It works per row — each token's 768 numbers are normalized on their own, with the row's own mean and variance. Rows never look at each other. With learned γ and β (each length 768) and ε = 1e-5:
μ = mean(x) # over the 768 features of THIS row
σ² = mean((x - μ)²) # population variance (÷ N, not N-1)
x_hat = (x - μ) / sqrt(σ² + ε) # ε is INSIDE the sqrt
out = x_hat * γ + β # elementwise, per feature
The + ε lives inside the square root, guarding against division by zero when a row is perfectly flat.
Work one by hand
Take a tiny 4-feature row x = [2, 4, 4, 6] (we'll reuse it next step):
- mean = (2+4+4+6)/4 = 4
- deviations =
[−2, 0, 0, 2] - variance = (4 + 0 + 0 + 4) / 4 = 2 ← divide by N=4, the population variance
- denom = √(2 + 1e-5) ≈ 1.41422
- x_hat =
[−2, 0, 0, 2] / 1.41422≈[-1.4142, 0, 0, 1.4142]
With γ = [1,1,1,1], β = [0,0,0,0] the affine step changes nothing, so that's the normalized row.
If you got variance 2.667 instead of 2, you divided by N−1 (sample variance) — GPT-2 uses the population variance, ÷N. And ε goes inside the root: √(σ² + ε), never √(σ²) + ε. Both mistakes pass small smoke tests and then silently fail the atol 1e-5 check downstream.