The MLP block·8 min

Down-project

0 / 1

# grade this stage (from your course directory)
npx theinference test --stage down-project --watch

The second matmul compresses the 3072-wide scratchpad back down to 768 — the residual width — completing the MLP. Expand, think, compress.

text

out = h @ c_proj.weight + c_proj.bias   # [T, 3072] -> [T, 768]

The full MLP — half a transformer block

[T,768] ─c_fc[768,3072]+b─▶ [T,3072] ─gelu─▶ [T,3072] ─c_proj[3072,768]+b─▶ [T,768]
                                                                      └─ this is the MLP sublayer (Chapter 7)

What to build

Your task

Implement runMLP

Implement runMLP(x, layer) → the full MLP: up-project + GELU (reuse runMLPHidden), then down-project.

Load h.{layer}.mlp.c_fc.{weight,bias} and h.{layer}.mlp.c_proj.{weight,bias} from the weights file (THEINFERENCE_WEIGHTS / ./weights/model.safetensors).

tests/adapters.ts

export function runMLP(x: number[][], layer: number): number[][] {
  // x: [T, 768], layer: which block's weights  ->  [T, 768]
}

Don't reimplement the first half

You already built the up-project + GELU as runMLPHidden in step 5.4. Call it to get h [T, 3072], then do the single down-projection: h @ c_proj.weight + c_proj.bias. No need to re-load c_fc or re-run GELU.

Compared with allclose

Run on block 0; your [T, 768] output is compared with allclose, atol 1e-4 (the same looser tolerance as the up-projection — these are 768- and 3072-wide accumulations).

References

Up-project + activate The full MHA