Build your own GPT-2

Load real GPT-2 weights and produce logits that match the reference, byte for byte.

What you'll build

You'll build the forward pass of GPT-2 124M from scratch in TypeScript: load the real published weights and run them through every layer until you get logits that match the reference exactly. No framework, no autograd, no GPU — just arrays and the math, so nothing stays a black box.

A loader for GPT-2's 124M published parameters, mapped into typed arrays
Numerically exact building blocks — embeddings, layernorm, multi-head attention, and the MLP
The full 12-layer transformer block wired into a complete forward pass
A logit projection that matches the reference byte-for-byte, verified by the grader

What you'll learn

How transformer inference actually works end to end — as code you wrote, not a diagram
How real model weights are stored on disk and mapped into memory
What each layer does and why — attention, layernorm, the residual stream, and the MLP
How to debug a forward pass to numerical exactness against a reference

Prerequisites

Comfortable reading and running TypeScript
Basic linear algebra — vectors and matrix multiplication (we re-derive the rest)
No machine-learning background required

Get started

Everything runs locally — no GPU, no API keys. Clear stage 1 in under ten minutes.

# 1. log in (approve in your browser — required)
npx theinference login

# 2. scaffold the course
npx theinference course start llm-inference ./llm-inference

# 3. install dependencies
cd llm-inference && npm install

# 4. fetch the GPT-2 weights (~548 MB)
./scripts/fetch-weights.sh ./weights

# 5. grade the first stage
npx theinference test --stage run-the-grader --watch

Each lesson shows the exact --stage command for that stage — passing it marks the stage complete here automatically.