Backprop, by hand·30 min

L2-norm backward

0 / 1

# grade this stage (from your course directory)
npx theinference test --stage grad-into-weights --watch

The chain from a sentence embedding back to the table runs through five Jacobians: L2-normalize → linear2 → tanh → linear1 → mean-pool. Each is a one-line vector-Jacobian product. The trickiest by far is the L2-norm backward — so you'll implement and grade that one in isolation here, then assemble the full chain in the next step.

the five backward steps

L2-norm backward:  dx = (dy − (dy·ŷ)·ŷ) / ‖x‖        ŷ = x/‖x‖
tanh backward:     dz = dy · (1 − tanh(z)²)
linear backward:   dW = xᵀ·dy ;  db = dy ;  dx = dy·Wᵀ
pool backward:     each non-pad token row gets dPooled / (#non-pad)

What to build

Your task

Implement the L2-norm backward

Implement runL2NormBackward(x, dy): given the pre-normalize vector x and the upstream gradient dy (w.r.t. the normalized output ŷ = x/‖x‖), return dx. The formula removes the component of dy along ŷ:

dx = (dy − (dy·ŷ) ŷ) / ‖x‖.

tests/adapters.ts

export function runL2NormBackward(x: number[], dy: number[]): number[] {
  // ŷ = x/‖x‖ ;  dx = (dy − (dy·ŷ)·ŷ) / ‖x‖
}

How it's graded

Adapter mode, single-op: dx is compared to the reference with allclose, atol 1e-5, rtol 1e-4. A single op accumulates no long float chain, so the tolerance stays tight. The fixture is finite-diff-validated (ε = 1e-4).

Finite-difference your own work

The starter ships gradCheck (central differences, ε = 1e-4) in src/gradcheck.ts. Run it on your backward before submitting — it is exactly how the reference fixtures were validated.

Gradient into the embeddings The full encoder backward