Forward → logits·6 min
Stack → norm → head
0 / 1
The finish line. Every piece is built. Now stack the block 12 times, normalize the stream once more, and ask all 50,257 candidate tokens a single question: how much do you look like where this sequence ended up? The answers are the logits — GPT-2's score for every word it could say next.
How the full forward works
Given inputIds (a list of token IDs):
text
x = embed(inputIds) # Chapter 3: [T, 768]
for layer in 0..11: # Chapter 7, stacked
x = block(x, layer)
x = LayerNorm(x, ln_f.weight, ln_f.bias) # final norm, ONCE
logits = x[last] @ wteᵀ # LM head, tied to wte -> [50257]
The full forward pass, with shapes
ids[T] ─embed─▶ [T,768] ─block 0─▶ … ─block 11─▶ [T,768] ─ln_f─▶ [T,768]
│ take last row
▼ x_last[768]
logits[t] = x_last · wte[t] → logits[50257]
Two details:
- Final LayerNorm
ln_fis applied once, after the last block — not per block. - The LM head is tied to the token embeddings. There's no separate output matrix in the file — the logits are the last position's vector dotted against every row of
wte([50257, 768]), giving one score per vocabulary token.
Only the last position matters
For next-token prediction you need just x[T-1]'s logits — a vector of length 50257, not the full [T, 50257]. nanoGPT does exactly this at inference (lm_head(x[:, [-1], :])), which saves a whole T×50257×768 matmul.