The MLP block·10 min

Up-project + activate

0 / 2

# grade this stage (from your course directory)
npx theinference test --stage up-project --watch

The first matmul of the MLP: project each token from 768 up to the 3072-wide scratchpad, add the bias, and apply gelu_new. This is the "expand and think" half.

text

h = gelu_new( x @ c_fc.weight + c_fc.bias )   # [T, 768] -> [T, 3072]

Up-project shapes (Conv1D layout, no transpose)

x [T,768]  @  c_fc.weight [768,3072]  +  bias [3072]  ──gelu_new──▶  h [T,3072]

What to build

Your task

Implement runMLPHidden

Implement runMLPHidden(x, layer) → gelu(x @ c_fc.weight + c_fc.bias), shape [T, 3072]. Conv1D layout: c_fc.weight is [768, 3072], so it's x @ W with no transpose. Load h.{layer}.mlp.c_fc.{weight,bias} and run on the given block.

tests/adapters.ts

export function runMLPHidden(x: number[][], layer: number): number[][] {
  // x: [T, 768], layer: which block  ->  [T, 3072]
}

Where to find the weights

For block layer, load:

h.${layer}.mlp.c_fc.weight — shape [768, 3072]
h.${layer}.mlp.c_fc.bias — shape [3072]

Grading runs on block 0. Reuse your runGelu from step 5.2 for the activation.

How it's graded (and why 1e-4, not 1e-5)

Run on block 0; your [T, 3072] output is compared with allclose, atol 1e-4.

This is the first step graded at the looser 1e-4 (attention used 1e-5). That's intentional, not sloppy: each output element here is a sum of 768 products from a [T,768] @ [768,3072] matmul, and float32 is not associative — the order in which those 768 terms are added changes the last few bits. The wider tolerance simply acknowledges that accumulation, while still being thousands of times tighter than any real difference.

Knowledge check

GPT-2 stores c_fc.weight with shape [768, 3072]. How do you apply it to an input x of shape [T, 768]?

GELU vs ReLU Down-project