The projection head·2 min
Recap: the encoder
0 / 1
You've built the entire encoder forward pass: tokenize → lookup → mean-pool → linear → tanh → linear → L2-normalize. Given any string, you can now produce a 32-d unit vector. That's model.encode(), demystified.
the full forward path
text → ids → [T×32] rows → [32] pooled → head → [32] embedding (‖·‖=1)
This untrained encoder is your baseline
Right now the table E and the MLP are random, so these vectors are essentially noise — paraphrases land no closer than unrelated sentences. That random-init encoder is exactly the baseline you'll beat in chapter 6. Keep it in mind: the gap between it and the trained model is the whole point.
Next: a loss that makes these vectors mean something — pull paraphrases together, push everything else apart. That's the contrastive loss, and it's the heart of the course.