Train your own embedding model

What you'll build

You'll train a real text-embedding model from a random init on your CPU in TypeScript: a mean-pooled token encoder with a tiny projection head, optimized by InfoNCE contrastive loss over in-batch negatives — and you'll derive and implement the backward pass by hand, no autograd. By the end you embed a query, retrieve its paraphrases by cosine, and prove the trained model beats a bag-of-words baseline on recall@5 against a frozen, grader-verified pool.

• A from-scratch sentence encoder — frozen-vocab tokenizer, learned embedding table, masked mean pooling, a Linear-tanh-Linear projection head, and L2-normalization — that turns text into a unit 64-d vector
• The full InfoNCE contrastive loss over a B-by-B cosine matrix with in-batch negatives and temperature scaling, plus its hand-derived backward pass through every layer into the embedding table
• A fixed-seed, single-thread training loop wired to a plain-Adam optimizer that drives the loss down reproducibly on real Quora/STS-B text pairs
• An embed and nearest CLI backed by your trained checkpoint, with recall@5 verified by the grader to clear a pinned threshold and beat the baseline

What you'll learn

• How an embedding model is actually trained — contrastive gradient descent pulling paraphrases together and pushing the rest of the batch apart under cosine
• Why each design choice exists — L2-normalization, temperature, mean pooling, and in-batch negatives as free negatives instead of a sampler
• How to derive a full backward pass by hand and validate it with finite differences, including the L2-norm Jacobian people get wrong
• How to evaluate a representation model honestly with recall@k against a frozen pool, instead of eyeballing nearest neighbors

Prerequisites

• Comfortable reading and running TypeScript
• Basic linear algebra — vectors, dot products, and matrix multiplication
• Familiar with a forward and backward pass, an Adam step, and a training loop (as built in train-gpt)