The projection head·5 min
Why a projection head?
0 / 1
Mean-pooled token vectors live in 'token space' — they capture which words a sentence used. But two paraphrases share meaning, not words. A small MLP — Linear → tanh → Linear — learns a projection into 'sentence-similarity space', where cosine actually tracks meaning. SimCSE adds exactly such an MLP head during contrastive training.
the head, with shapes
[32] ──Linear W1[32×64], b1──▶ [64] ──tanh──▶ [64]
──Linear W2[64×32], b2──▶ [32] ──L2-norm──▶ [32], ‖v‖ = 1
We use tanh, not GELU or ReLU, for the activation. It's bounded, smooth, and its derivative is dead simple — 1 − tanh²(z). That makes the hand-derived backward pass in chapter 4 clean, and it keeps finite-difference grad-checks stable. (Real models often use GELU; tanh is the right call for a from-scratch, finite-diff-validated build.)