Backprop, by hand·7 min
The clean gradient
0 / 1
Here's the beautiful part: the gradient of cross-entropy-after-softmax is just softmax − onehot. For the one-directional InfoNCE, dScores[i] = softmax(scores[i]) − e_i, where e_i is the one-hot on the diagonal — then scaled by 1/τ on the way out (chain rule through the temperature). Because we grade the symmetric loss, the row-direction term is averaged with the transposed column-direction term.
dL/dS for the symmetric loss
# symmetric loss = (row-CE + col-CE) / 2 → add both directions, then halve:
dS = 0.5 · ( (softmax(S/τ) − I) + (softmax(Sᵀ/τ) − I)ᵀ ) / τ
# the one-directional default would be simply (softmax(S/τ) − I) / τ
# dS then flows into the cosine matrix → embeddings → MLP → table
One equation to remember
If you take a single formula away from this course, make it dScores = softmax − onehot. Everything else in the backward pass is standard matmul / activation / pooling Jacobians chained onto it.