Token + position embeddings·5 min

Ids to vectors

0 / 1

A transformer doesn't think in token IDs — it thinks in vectors. An ID like 15496 is just an index; it carries no meaning a matrix multiply could use. This chapter turns a flat list of IDs into the [T, 768] matrix that flows into block 0. This is where the residual stream begins.

Two lookup tables

GPT-2 carries two embedding tables, both loaded from the weights file you opened in Chapter 1:

wte.weight — the token table, shape [50257, 768]. Row id is the 768-dim vector for token id. What the word is.
wpe.weight — the position table, shape [1024, 768]. Row pos is the vector for absolute position pos. Where in the sequence it sits.

The input representation is just their sum, computed independently per position:

text

x[t] = wte[ inputIds[t] ] + wpe[t]

A concrete trace

Tokenize "Hello, world" under r50k_base and you get IDs [15496, 11, 995], so T = 3. Each position pulls one row from each table and adds them:

position 0 → wte[15496] + wpe[0]
position 1 → wte[11] + wpe[1]
position 2 → wte[995] + wpe[2]

Two lookups, summed, per position

ids = [15496, 11, 995]            (T=3)
      wte[15496]  +  wpe[0]  =  x[0]   ┐
      wte[11]     +  wpe[1]  =  x[1]   ├─ x : [3, 768]  ← the residual stream begins
      wte[995]    +  wpe[2]  =  x[2]   ┘

Notice what happens when the same token appears at two positions: it reads the same wte row but a different wpe row, so the two vectors come out different. That tiny offset is the entire reason the model can tell "the dog and the cat" apart by position — identical words at different places stay distinguishable.

Recap Token embeddings