What you're building
In 2019 OpenAI published GPT-2 as a paper and a file of 124 million numbers. No code that runs it on a CPU, no narration — just weights.
Over the next ~40 steps you'll write, by hand in TypeScript, the exact arithmetic that turns those numbers into a next-word prediction. No PyTorch, no autograd, no GPU. Think of it as forensic reconstruction: OpenAI shipped the evidence, and you're going to make it talk, value for value. At the end, your logits will match the reference — to the decimal.
The whole model, in one line
GPT-2 is a pipeline you build one piece at a time:
token ids → embeddings → 12 transformer blocks → final LayerNorm → logits
Each transformer block is just attention (every token looks back at earlier tokens) followed by an MLP (a small per-token network), with a LayerNorm and a residual add around each. That's the entire architecture — GPT-2 is that block, repeated twelve times, wrapped in an embedding lookup at the front and a vocabulary projection at the back.
ids[T] → [T,768] → [T,768] → … → [T,768] → [T,768] → logits[50257]
embed block 0 block 11 ln_f tied head
(×12 identical blocks: attention + MLP)
12 layers · 12 heads · 768 wide · 50257 vocab · 1024 context.
Memorize these — every shape you write is built from them:
768= 12 heads × 64 (head width)3072= 4 × 768 (the MLP scratchpad)2304= 3 × 768 (fused Q, K, V)
Each chapter breaks one idea into small steps: read the idea, sometimes answer a quick quiz, and on build steps write one small function and watch the grader turn it green. You only move on once the previous build passes — so the model is always correct as it grows. Everything runs locally; no API keys, no GPU.