Multi-head attention·10 min
QKV projection
0 / 2
# grade this stage (from your course directory)
npx theinference test --stage qkv-projection --watchOne matmul produces all 12 heads' queries, keys, and values at once. Build that fused projection first — everything downstream slices into its output.
Shape
x[T,768] @ c_attn.weight[768,2304] + bias[2304] --> qkv[T,2304] (= 3 * 768)
What to build
Your task
Implement runQKV
Implement runQKV(x, layer) → x @ c_attn.weight + c_attn.bias, shape [T, 2304] (= 3 × 768). The columns are [all-q (768) | all-k (768) | all-v (768)]. Load h.{layer}.attn.c_attn.{weight,bias}.
tests/adapters.ts
export function runQKV(x: number[][], layer: number): number[][] {
// x: [T, 768] -> [T, 2304]
}
Same Conv1D rule as the MLP
c_attn.weight is stored [in, out] = [768, 2304], so the op is x @ W with no transpose — exactly like c_fc in Chapter 5. Load h.{layer}.attn.c_attn.weight [768, 2304] and h.{layer}.attn.c_attn.bias [2304].
How it's graded
Run on block 0; your [T, 2304] output is compared with allclose, atol 1e-4.
Knowledge check
The c_attn projection outputs 2304 columns. How are those columns organized?