Backprop, by hand·50 min

The full encoder backward

0 / 1

# grade this stage (from your course directory)
npx theinference test --stage full-chain-grad --watch

Assemble the whole thing. Given a batch of (anchor, positive) pairs and a seeded init, run the full contrastive backward: loss → dScores → dQ/dP → through L2-norm, linear2, tanh, linear1, mean-pool, and scatter into the table rows. This is the intellectual peak of the course — every Jacobian you built, chained.

what the full backward produces

contrastiveBackward(qCaches, pCaches, dScores, params)
   →  { dE, dW1, db1, dW2, db2 }

dE is SPARSE: only rows for tokens that appeared in the batch are nonzero.

What to build

Your task

Implement the full encoder backward

Wire up contrastiveBackward so the encoder's forward caches its intermediates (pooled, h1, a1, h2, norm, ids, mask) and the backward produces all five parameter gradients. The graded adapter runGradW2(anchors, positives, seed) re-inits from seed, runs forward+backward on the fixture batch, and returns dW2.

tests/adapters.ts

export function runGradW2(
  anchors: string[], positives: string[], seed: number,
): number[][] {
  // initParams(seed) → encode both sides → infonceBackward → contrastiveBackward → .W2
}

How it's graded — and why the tolerance is looser here

This is the full composed chain — L2-norm + two linears + tanh + masked pool + sparse table scatter — so a long float32 reduction accumulates differently across implementations. The grader compares dW2 to the reference with allclose, atol 1e-4, rtol 1e-3 (looser than the single-op checks), and the reference itself is double-anchored by finite differences (rel error ≤ 1e-2). A correct gradient clears both; a sign error or a missing L2-norm term fails both.

The table gradient is sparse

Only the rows of E for tokens that actually appeared in the batch get a nonzero gradient. Each non-pad token row receives dPooled / (#non-pad) from the mean-pool backward. Tokens that never appear stay exactly zero — initializing dE to a full zero matrix and scattering into it is the clean way.

When in doubt, grad-check

If runGradW2 is off, run gradCheck over W2 against the InfoNCE loss. The finite-diff estimate is assumption-free — if it disagrees with your analytic dW2, the analytic one is wrong (it almost always is).

L2-norm backward Why finite differences?