The MLP block·2 min
GELU vs ReLU
0 / 2
Why gelu_new and not the simpler ReLU? And what exactly is the MLP doing around it?
Knowledge check
GPT-2 uses gelu_new instead of ReLU. What's the practical difference?
Knowledge check
For the up-projection in GPT-2's MLP, what is the hidden width and which nonlinearity is used?