Multi-head attention·2 min
Why multiple heads
0 / 1
Before assembling all the heads, one conceptual question: why split attention into 12 heads at all?
Knowledge check
Why split attention into 12 heads instead of one big 768-wide head?
Build your own GPT-2