The contrastive loss·5 min
Temperature
0 / 1
Cosines live in [−1, 1] — far too flat for softmax to separate. So before the softmax we scale by 1/τ (temperature τ = 0.05, i.e. multiply by 20) to stretch the cosines into real logits. SimCSE and sentence-transformers both default to τ = 0.05 / scale 20.
why scaling matters
cosines [0.9, 0.1]
× 20 [18, 2] → softmax ≈ [0.9999, 0.0001] ← sharp, useful signal
without scaling:
softmax([0.9, 0.1]) ≈ [0.69, 0.31] ← barely separated
Knowledge check
Why multiply the cosine-similarity matrix by 20 (divide by τ = 0.05) before the softmax?