Build your own semantic search
Setup & orientation·4 min

What's in the fixture pack

0 / 1

Frozen, deterministic, on disk

course start already placed the fixture pack on disk under fixtures/pack/. Let's walk it, because every later step reads from it:

  • corpus_emb.f32 (200 × 64 float32) + corpus_emb.shape.json — the document vectors.
  • query_emb.f32 (40 × 64) + query_emb.shape.json — the query vectors.
  • corpus.json — doc ids + titles + text, index-aligned with the embedding rows.
  • qrels.json — the relevance judgments (which docs answer which query).
  • gt_knn.json — the exact float64 top-100 nearest neighbors per query (the oracle).
The float32 blob convention
corpus_emb.f32        = raw little-endian Float32Array, row-major
corpus_emb.shape.json = [200, 64]   (numDocs × dim)

We ship raw Float32Array blobs + a sidecar shape file — the same convention the GPT-2 course uses for tensors. A one-line helper loadF32(path, shape) (in the provided src/data.ts) reads a blob into a Float32Array; loadPack(dir) loads the whole pack. That is the only file I/O you ever need.

These vectors are seeded-synthetic — on purpose

The vectors here are deterministic Gaussian noise (seeded), L2-normalized just like a real embedding model's output, with each query built near a "gold" document so retrieval is meaningful. The algorithms you write — cosine, kNN, recall, nDCG — are exactly what you'd run over real all-MiniLM-L6-v2 embeddings of a benchmark like BEIR/SciFact. Swapping in real vectors is a fixture regeneration, nothing more (see NOTES.md). Synthetic data buys bit-identical grading on every machine, zero setup.