Tokens to vectors·20 min

Tokenize + embedding lookup

0 / 1

# grade this stage (from your course directory)
npx theinference test --stage tokenize-and-lookup --watch

Text isn't numbers. The frozen tokenizer (shipped and identical for everyone) maps each word to an integer id; the embedding table E of shape [vocab × 32] gives every id a learnable 32-d row. The bottom of the encoder is just: tokenize the sentence, then gather the rows of E for those ids.

The frozen tokenizer

Our tokenizer is deliberately tiny and seedless: lowercase, split on whitespace, then hash each word's character-trigrams into one of 256 buckets (ids 0 and 1 are reserved for padding and unknown). No BPE file to download, no vocabulary drift — every learner produces byte-identical ids. It's provided for you in src/vocab.ts as tokenize(sentence): number[].

tokenize → ids → rows of E

"reset my password"
   │ tokenize  (lowercase, split, hash trigrams)
   ▼
[ 88, 203, 17 ]                       ← token ids
   │ gather rows of E [256 × 32]
   ▼
[ E[88], E[203], E[17] ]              ← a [3 × 32] matrix

What to build

Your task

Implement the embedding lookup

Implement runEmbedLookup(tokenIds, E): return the [T, 32] matrix of embedding rows for the given ids — row t is E[tokenIds[t]]. This is a pure gather; no math yet. (Slice/copy the rows so you never alias the table.)

tests/adapters.ts

export function runEmbedLookup(tokenIds: number[], E: number[][]): number[][] {
  return tokenIds.map((id) => E[id].slice());
}

How it's graded

Adapter mode: the gathered [T, 32] matrix is compared to the reference with allclose, atol 1e-6. The fixture ships a fixed E slice and a fixed id list.

Why normalize?Mean pooling