Setup & orientation·4 min

Tensors & weights

0 / 1

Everything in GPT-2 is a tensor — a grid of floating-point numbers. In this course a 2-D tensor is just a number[][] (rows × columns) and a 1-D tensor a number[]. A 768-long vector is one token's representation; a [T, 768] matrix is T tokens stacked, one per row.

A [2,3] tensor is just a list of rows

A [2,3] tensor is just:  [[1, 2, 3],
                          [4, 5, 6]]

row 0 = [1,2,3]   element [1][2] = 6   shape = [rows, cols] = [2,3]

Where the knowledge lives

The model's knowledge is 124,439,808 of these floats — the weights — learned once during training and frozen. The biggest single tensor is wte.weight, shape [50257, 768] = 38.6M numbers (50257 × 768 = 38,597,376), almost a third of the whole model, and it's nothing but the token vocabulary.

You won't train any of them. You'll load these exact floats and do arithmetic with them — every chapter is one more piece of that arithmetic. The next chapter cracks open the weights file.

The shapes you’ll see everywhere

T — the number of tokens in the input (the sequence length).
768 — GPT-2’s hidden width (d_model): every token is a 768-dim vector through the whole model. d_model = 768 is the one width that never changes between blocks — that constancy is exactly what lets you stack 12 of them.
[T, 768] — the residual stream: the running representation that flows from the embeddings, through all 12 blocks, to the final norm.

Fetch the weights now

Most build steps need the real GPT-2 weights (~548 MB). The download script ships with your course — run it once from your course directory (the one course start created) so you're never blocked:

./scripts/fetch-weights.sh ./weights

It downloads model.safetensors + config.json into ./weights/, which is exactly where the grader looks for them.

Run the grader The safetensors format