LayerNorm·10 min

Standardize a row

0 / 1

# grade this stage (from your course directory)
npx theinference test --stage standardize --watch

LayerNorm has two halves. The first — standardize — is the gain control's automatic part: re-center each row to zero mean and rescale to unit spread. No learned parameters yet, just the row's own statistics.

Standardize one row

x_hat = (x - mean(x)) / sqrt(var(x) + 1e-5)   # var is population (÷ N), ε inside the root

Per-row independence

x [T,768]   →   each row standardized on its own (rows never interact)   →   x_hat [T,768]

  row 0:  (x[0] - mean(x[0])) / sqrt(var(x[0]) + 1e-5)   ┐
  row 1:  (x[1] - mean(x[1])) / sqrt(var(x[1]) + 1e-5)   ├─ T independent computations
   ...                                                    │
  row T-1: ...                                            ┘

Reuse the row from the last step to self-check: x = [2, 4, 4, 6] → mean 4, variance 2, denom √(2+1e-5) ≈ 1.41422 → [-1.4142, 0, 0, 1.4142]. If your code reproduces that, your standardize is right.

What to build

Your task

Implement runStandardize

Implement runStandardize(x): for each row, subtract the mean and divide by sqrt(var + 1e-5). No gamma/beta. Variance is the population variance (÷ N), and ε goes inside the square root.

tests/adapters.ts

export function runStandardize(x: number[][]): number[][] {
  // x: [T, d] -> [T, d], each row zero-mean / unit-variance
}

Fastest debug: print one row

After standardizing, every row has mean ≈ 0 and variance ≈ 1 (not exactly 1 — the +ε shaves it a hair). A quick console.error of one row's mean and variance is the fastest sanity check. Debug goes to stderr; the grader only reads stdout.

How it's graded

Your [T, d] output is compared with allclose, atol 1e-5.

Why normalize Gamma