Linear + tanh
# grade this stage (from your course directory)
npx theinference test --stage linear-and-tanh --watchThe head is built from two primitives: an affine layer xW + b and the tanh activation. Build the linear layer (you'll grade it), and write tanh element-wise — together they're everything the projection head needs.
linear(x, W, b)[j] = b[j] + Σᵢ x[i]·W[i][j] // W is [in × out]
tanh(z) = (eᶻ − e⁻ᶻ) / (eᶻ + e⁻ᶻ) // element-wise
What to build
Implement linear (and tanh)
Implement runLinear(x, W, b) returning xW + b as a number[]. x is [in], W is [in × out], b is [out], so the result is [out]. Also implement tanh element-wise in src/tensor.ts (JavaScript ships Math.tanh).
export function runLinear(x: number[], W: number[][], b: number[]): number[] {
// out[j] = b[j] + Σᵢ x[i]·W[i][j]
}
Adapter mode: your [out] vector from runLinear is compared to the reference with allclose, atol 1e-5, rtol 1e-4.
W is [in × out] here — index it as W[i][j] with i over inputs and j over outputs. Transposing it (treating it as [out × in]) gives a wrong-shaped or wrong-valued result. The bias adds per output: b has length out.
tanh's derivative is 1 − tanh²(z) — you already have tanh(z) from the forward pass, so the backward costs one multiply. That clean Jacobian is why we chose it over GELU.