You've built GPT-2
That length-50257 vector is a score for every token GPT-2 knows. Take the argmax and you have the model's predicted next token; feed it back in and loop, and you have text generation. You built that — the real GPT-2 forward pass, from safetensors bytes to logits, every number matching the reference. No framework did the work; you did.
Look back at the pipeline you assembled: token + position embeddings → 12 pre-norm transformer blocks (each = multi-head attention + MLP, with LayerNorm and residuals) → final LayerNorm → a tied LM head. That is GPT-2 124M, end to end, as code you wrote.
Watch your model predict a word
Run the prompt Hello, world — which tokenizes to [15496, 11, 995] under r50k_base — through the runLogits you just finished, and pull out the top scores:
// Greedy next-token prediction from YOUR runLogits:
const ids = [15496, 11, 995]; // "Hello, world" (r50k_base)
const logits = runLogits(ids); // length 50257 — the code you just wrote
const top = logits
.map((v, id) => [id, v] as const)
.sort((a, b) => b[1] - a[1])
.slice(0, 5);
console.log(top); // top-5 (token id, logit) — the model's next-word guesses
argmax(logits) is GPT-2's single best next token. Feed it back in and loop, and you have text generation. We don't ship a tokenizer in this course, so you're looking at token ids, not words — decode them with a tokenizer (the next course!) and you'll read real text. The point is the numbers: your logits match HF's to 1e-3. This is the actual model, not a lookalike.
Each of these starts from exactly the runLogits you just finished:
- Sampling (greedy / temperature / top-k / top-p) — turn one logit vector into varied, fluent text.
- A KV cache — make looped decoding fast without changing a single logit.
- Quantization (int8 / int4) — same forward pass, much smaller weights.