Affine LayerNorm
# grade this stage (from your course directory)
npx theinference test --stage affine-layernorm --watchThe second half is the manual knob on the gain control: rescale and shift the standardized values with the learned γ (scale) and β (shift). Standardize re-centered everything to 0 and unit spread; γ/β set the level the model actually wants. Together they make the full LayerNorm GPT-2 uses.
One affine step by hand
Continue the running example. Standardize gave x_hat = [-1.4142, 0, 0, 1.4142]. Apply γ = [2, 2, 2, 2], β = [1, 1, 1, 1], elementwise per feature:
-1.4142·2 + 1= −1.82840·2 + 1= 10·2 + 1= 11.4142·2 + 1= 3.8284
Result: [-1.8284, 1, 1, 3.8284]. γ stretches the spread, β shifts the level — each feature gets its own pair.
What to build
Implement runLayerNorm
Implement runLayerNorm(x, gamma, beta) → standardize(x) * gamma + beta, per row. gamma and beta are passed in directly — this step is a pure function, no weights file needed.
export function runLayerNorm(x: number[][], gamma: number[], beta: number[]): number[][] {
// x: [T, 768], gamma/beta: [768] -> [T, 768]
}
No weights file here: γ and β arrive as arguments. Reuse your runStandardize, then apply the affine map elementwise: out[i][j] = x_hat[i][j] * gamma[j] + beta[j]. The real γ/β (ln_1.weight/bias, ln_2.weight/bias, ln_f.weight/bias) get wired in from the weights file at Chapter 7.
Your [T, 768] output is compared to the reference with allclose, atol 1e-5.