Prove the win·10 min

Identity held, work slashed

0 / 1

# grade this stage (from your course directory)
npx theinference test --stage identity-and-speedup --watch

This is the screenshot. One command runs your prompt and prints the whole story in three numbers: the cached op-count, the recompute op-count, and the memory the cache costs at 1024 tokens. The two counts come from the profiles you just built; the byte figure from kvBytes. Put them on one line and you have the receipt — identical logits, a fraction of the work — that turns a toy decoder into something you'd actually serve from.

The line you are shipping

$ npx tsx program.ts capstone 4 4
cached=[5,6,7,8] recompute=[10,15,21,28] kv_bytes_1024=37748736

What to build

Your task

Wire the capstone command

In program.ts, handle capstone <promptLen> <nSteps>. Parse both args as integers, call cachedProfile, recomputeProfile, and kvBytes from src/kvcache.ts, and print exactly one summary line:

cached=[...] recompute=[...] kv_bytes_1024=<bytes>

The arrays are comma-joined with no spaces; kv_bytes_1024 is the cache's total byte cost for a 1024-token sequence (use the model's dtype — float32, 4 bytes). This is the deterministic op-count and memory summary the grader reads.

program.ts

import { cachedProfile, recomputeProfile, kvBytes } from './src/kvcache.ts'

if (command === 'capstone') {
  const promptLen = Number(process.argv[3])
  const nSteps = Number(process.argv[4])
  const cached = cachedProfile(promptLen, nSteps)
  const recompute = recomputeProfile(promptLen, nSteps)
  const { total } = kvBytes(1024, 4)
  console.log(`cached=[${cached}] recompute=[${recompute}] kv_bytes_1024=${total}`)
  process.exit(0)
}

The grader reads the line literally

Match the format character-for-character: keys in lower snake_case, arrays bracketed and comma-joined with no spaces (a bare ${array} does this for you), one space between fields, the byte count as a plain integer. A stray space, a label typo, or scientific notation on the byte figure and the parser misses it. Print one line and nothing else on stdout.

The identity is already yours

You proved logit-allclose against the recompute oracle back in Chapter 4 and held it through generate and prefix reuse — cached GPT-2 produces the same tokens recompute does, step for step. This step doesn't re-litigate that; it stands on it. The summary is the consequence: same output, with the cached curve doing a fraction of recompute's work and a fixed, knowable memory bill.

The prefix-cache flex

The same engine carries the Chapter 6 trick: save a long system prompt's cache once, then reuse it across requests and skip re-prefilling it — identical output, the prefill work paid a single time. That kv_bytes_1024 figure is exactly the memory a saved prefix occupies, which is why reuse is a memory-for-compute trade, not a free lunch.

How it's graded

The grader runs program.ts capstone <promptLen> <nSteps> and parses the summary line. cached must equal the O(n) profile, recompute the O(n²) profile (both integer-exact), and kv_bytes_1024 must equal kvBytes(1024, 4).total. Any wall-clock output is ignored — never gated.

Prove the work reduction Where this goes next