A single attention head·2 min
Why divide by √d
0 / 2
One detail in that formula matters a lot: the /√d. Here is why it is there.
Knowledge check
Why are the raw scores q[i] · k[j] divided by √d before softmax?
Knowledge check
At d = 64 with no 1/√d scaling, what happens to a softmax over the raw scores?