Forward → logits·10 min

Run the stack

0 / 1

# grade this stage (from your course directory)
npx theinference test --stage run-the-stack --watch

Thread the embeddings through all 12 blocks. This is the body of GPT-2 — the hidden states [T, 768] just before the final norm and head. Same shape in, same shape out, twelve times over.

Chaining the blocks

ids[T] ─embed─▶ [T,768] ─block 0─▶ [T,768] ─block 1─▶ … ─block 11─▶ [T,768]
                    x            x            x                  x   (return this)
                    └── each block's output becomes the next block's input ──┘

What to build

Your task

Implement runHidden

Implement runHidden(inputIds): embed the input (reuse runEmbeddings), then run all 12 blocks in order (reuse runBlock for layers 0–11), and return the [T, 768] hidden states. No final norm yet.

tests/adapters.ts

export function runHidden(inputIds: number[]): number[][] {
  // embed -> 12 blocks  ->  [T, 768]
}

Thread x in order — reuse the same x

Chain it: let x = embed(ids); for (let l = 0; l < 12; l++) x = runBlock(x, l); return x;. Feeding the original embedding into every block (instead of the running x) still produces a [T, 768] of the right shape but wrong values — and atol 1e-4 will catch it.

How it's graded

Your [T, 768] output is compared with allclose, atol 1e-4. (Errors accumulate across 12 blocks, so the tolerance is looser than the single-block steps.)

Stack → norm → head Final norm + logits