The feed-forward network
Attention moves information between tokens. The MLP does the per-token thinking: take each token's 768-number vector, expand it into a 3072-wide scratchpad, bend it through a nonlinearity, then compress it back to 768.
It's position-wise — every token is processed independently, the same way — and it's where roughly two-thirds of GPT-2's parameters live. If attention is the routing, this is where most of the knowledge is stored.
How the MLP works
For an input x of shape [T, 768], using block layer's weights:
h = gelu_new( x @ c_fc.weight + c_fc.bias ) # [T, 3072]
out = h @ c_proj.weight + c_proj.bias # [T, 768]
x [T,768] ──@ c_fc[768,3072] +b──▶ [T,3072] ──gelu_new──▶ [T,3072] ──@ c_proj[3072,768] +b──▶ out [T,768]
▲ up-project (5.4) ▲ nonlinearity (5.2) ▲ down-project (5.5)
The hidden width is 4 × 768 = 3072. The nonlinearity is GPT-2's tanh approximation of GELU, gelu_new — we build it next step:
gelu_new(x) = 0.5 · x · (1 + tanh( √(2/π) · (x + 0.044715 · x³) ))
This is the single most error-prone fact in the whole course. GPT-2 stores these as Conv1D weights, laid out [in, out] — so c_fc.weight is [768, 3072] and the op is x @ W with NO transpose.
A PyTorch Linear stores [out, in] and needs x @ Wᵀ. GPT-2 does not. Transpose here and one of two things happens:
- Best case: your shapes don't even line up —
[T,768] @ [3072,768]fails loudly. Annoying, but the error tells you. - Worst case: the dimensions coincidentally match and you get silent garbage that fails grading deep downstream with no obvious cause. People lose an afternoon right here.
Say it out loud: c_fc.weight is [768, 3072], the op is x @ W, no transpose.