A full transformer block·2 min
Why residual / pre-LN
0 / 2
Two checks on the structure of a block — the overall shape, and exactly what the residual adds to.
Knowledge check
Which expression correctly describes one GPT-2 transformer block?
Knowledge check
In x = x + attn(ln_1(x)), what is the residual added back to, and how?