Build your own GPT-2
A single attention head·10 min

Scaled scores

0 / 1
# grade this stage (from your course directory)
npx theinference test --stage scaled-scores --watch

Step one of attention: decide how much each position should attend to each other position. Score every query against every key — a dot product measures how well what one token is looking for matches what another token advertises.

Score for query i against key j
scores[i][j] = (q[i] · k[j]) / √d
Shapes
q [T,d] , k [T,d]   ──(q·kᵀ)/√d──▶   scores [T,T]
Hand-compute these two rows

Take d = 2, q = [[1,0],[0,1]], k = [[1,1],[2,0]], so √d = √2 ≈ 1.4142:

  • scores[0] = [ (1·1+0·1)/1.4142, (1·2+0·0)/1.4142 ] = [0.7071, 1.4142]
  • scores[1] = [ (0·1+1·1)/1.4142, (0·2+1·0)/1.4142 ] = [0.7071, 0]

Your runScores should reproduce these exactly.

What to build

Your task

Implement runScores

Implement runScores(q, k). q and k are [T, d]; return the full [T, T] score matrix (q·k)/√d — every query against every key, no masking yet.

tests/adapters.ts
export function runScores(q: number[][], k: number[][]): number[][] {
  // q, k: [T, d]  ->  [T, T]
}
How it's graded

Your [T, T] matrix is compared to the reference with allclose, atol 1e-5. d is the head dimension — the width of each q/k row.

`d` is the head width, not `T`

d = q[0].length — the inner dimension of each q/k row (16 in this chapter's fixture, 64 in real GPT-2). It is not the number of tokens T. Dividing by √T instead of √d silently fails the atol 1e-5 check.

What attention computesWhy divide by √d