LayerNorm·2 min
Gamma
0 / 2
Two checks on the details that trip people up: which axis is normalized, and exactly how the variance and ε are computed.
Knowledge check
For an input of shape [T, 768], which axis does LayerNorm normalize over?
Knowledge check
How is the variance computed, and where does ε go?