Build your own GPT-2
A full transformer block·10 min

The full block

0 / 1
# grade this stage (from your course directory)
npx theinference test --stage full-block --watch

Add the second half — the MLP sublayer — and you have a complete GPT-2 block: the unit that gets stacked 12 times. It is the same residual move as the attention sublayer, repeated once more.

The full block: two sublayers, two residual adds
x ─ln_1─▶MHA─(+)─▶ h ─ln_2─▶MLP─(+)─▶ out      all arrows are [T,768]
    └──────────┘        └──────────┘
      sublayer 1          sublayer 2     ← one block; stack 12 (Ch 8)

The first residual produces h = x + MHA(ln_1(x)) — the post-attention stream. The second residual adds the MLP correction to h, not back to the original x: out = h + MLP(ln_2(h)). Every arrow is [T, 768].

What to build

Your task

Implement runBlock

Implement runBlock(x, layer): the attention sublayer (reuse runAttnSublayer), then x = x + MLP(LayerNorm(x, ln_2)). Load h.{layer}.ln_2.{weight,bias}; the attention and MLP weights are loaded by the helpers you already wrote. No new math — just compose runLayerNorm, runMHA, and runMLP with two residual adds.

tests/adapters.ts
export function runBlock(x: number[][], layer: number): number[][] {
  // x: [T, 768], layer: which block  ->  [T, 768]
}
Two LayerNorms, two sets of weights

ln_1 (before attention) and ln_2 (before the MLP) each have their own weight/bias — both length-768 — and you must not share them. Load h.{layer}.ln_1.* for the attention sublayer and h.{layer}.ln_2.* for the MLP sublayer. And the second residual adds the MLP output back to h (the post-attention stream), not to the original x.

How it's graded

Run on block 0; your [T, 768] output is compared with allclose, atol 1e-4.

Why residual / pre-LNStack → norm → head