Build your own GPT-2
Token + position embeddings·8 min

Add positions

0 / 1
# grade this stage (from your course directory)
npx theinference test --stage add-positions --watch

Attention has no built-in sense of order — to it, a sentence is a bag of tokens. Without position information, "dog bites man" and "man bites dog" produce the identical set of vectors, and attention can't tell which mauled which. Adding wpe[t] stamps each slot with a unique signature, and that sum is the input to the first block.

What to build

Your task

Implement runEmbeddings

Implement runEmbeddings(inputIds) → wte[inputIds[t]] + wpe[t] per position, shape [T, 768]. Reuse your token lookup from the last step and add the position row. Load wte/wpe with your stage-1 loader.

tests/adapters.ts
export function runEmbeddings(inputIds: number[]): number[][] {
  // -> [T, 768]
}
It's wpe[t] — the absolute position, not wpe[inputIds[t]]

This is the #1 embedding bug, and it's subtle because the two tables are indexed differently:

  • the token row uses the id: wte[inputIds[t]]
  • the position row uses the index in the sequence: wpe[t]

For IDs [15496, 11, 995], position 1 is wte[11] + wpe[1] — not wte[11] + wpe[11]. Loop over t, use t straight into wpe. (The quiz on the next step tests exactly this.)

How it's graded

Your [T, 768] output is compared to the reference with allclose, atol 1e-5.

Token embeddingsWhy position embeddings