Build your own GPT-2
Load the GPT-2 weights·5 min

The safetensors format

0 / 1

Time to crack open the vault. GPT-2 124M ships as a single model.safetensors file — ~548 MB, every value an F32 — and inside it are the 124 million numbers OpenAI published in 2019. No code, no narration, just weights. Before you can run a single forward pass you have to reach into those raw bytes and pull tensors out by name. This step is the map of where everything sits; the next one reads it.

The safetensors layout

A safetensors file is three parts laid out back to back:

  1. 8 bytes — the header length N, a little-endian unsigned 64-bit integer.
  2. N bytes — a JSON object mapping each tensor name to its metadata: { "dtype", "shape", "data_offsets": [begin, end] }.
  3. The rest — raw tensor bytes, little-endian, row-major (C order).

data_offsets are relative to the start of the data region (byte 8 + N), and end is exclusive. So a tensor's bytes live at [8 + N + begin, 8 + N + end). GPT-2's tensors are all F32 (4 bytes per value).

The real gpt2 file, to scale
┌──────────┬───────────────────────────┬────────────────────────────┐
│ 8 bytes  │  N = 14283 bytes of JSON  │   ~548 MB of raw F32 bytes  │
│ uint64LE │  {"wte.weight":{...}, …}  │   little-endian, row-major  │
└──────────┴───────────────────────────┴────────────────────────────┘
 byte 0      byte 8                       byte 8+N = 14291

These are the actual numbers in the file on disk. The first 8 bytes decode to N = 14283, so every tensor's data starts at byte 8 + N = 14291. A tensor whose header says data_offsets: [b, e] therefore lives at bytes [14291 + b, 14291 + e) (end exclusive).

Token 0's soul, byte for byte

wte.weight (the token embedding table, shape [50257, 768]) has data_offsets: [287209472, 441598976]. Walk to byte 14291 + 287209472, read the first four bytes as a little-endian F32, and out falls -0.1101… — the first feature of token 0.

Read the next four: -0.0393. Then 0.0331, 0.1338, -0.0485. Those five floats are not metadata or padding — they are what GPT-2 knows about token 0 before it has seen any context. You will watch these exact numbers flow forward through every chapter until, at the very end, they help produce a logit.

Two off-by-N mistakes that produce pure garbage

Both of these shift every value you read and the floats come back as noise:

  • Offsets are relative to byte 8 + N, not byte 0. Forget the 8 + N and you start reading 14291 bytes too early — inside the JSON header.
  • end is exclusive. data_offsets: [b, e] means the bytes [b, e); the value count is (e − b) / 4, not (e − b) / 4 + 1.
Tensors & weightsParse the header