The generate CLI
# grade this stage (from your course directory)
npx theinference test --stage generate-cli --watchThe greedy loop works under the grader's adapter. Now give it a front door. program.ts is the process entrypoint, and you'll add a generate command that takes a token budget and a prompt — as token IDs on the command line — runs the cached greedy decode, and prints the generated token IDs. This is the moment your cache stops being a test fixture and becomes something you can actually invoke.
# generate <maxTokens> <id...> — greedy cached decode, prints generated ids
npx tsx program.ts generate 4 40 716 257 1175
# -> 4293 13 314 716 (the 4 generated ids, space-separated)
What to build
Wire the generate command in program.ts
In program.ts, add a branch for command === 'generate'. Parse maxTokens from the first argument and the remaining arguments as the prompt token IDs (.map(Number)). Load the weights, call generateTokens(promptIds, maxTokens, w) from src/kvcache.ts, and print the returned IDs joined by single spaces — nothing else on the line. Exit 0.
// in program.ts — references generateTokens + loadWeights from ./src/kvcache.ts
if (command === "generate") {
const maxTokens = Number(rest[0]);
const promptIds = rest.slice(1).map(Number);
const w = await loadWeights(WEIGHTS);
const ids = generateTokens(promptIds, maxTokens, w);
console.log(ids.join(" "));
process.exit(0);
}
The CLI deals in raw token IDs, not text — there's no tokenizer in the loop. The prompt arrives as integers and you print integers. That keeps grading exact and the engine focused on the cache mechanics; turning IDs back into readable text is a tokenizer's job, not the generator's.
Right now generate always prefills the prompt from scratch. In the next chapter you'll add a prefix-cache path so a shared prefix is prefilled once and reused — wired through this same command. You don't need it yet; just leave room for it.
It's tempting to print how fast cached decode runs versus recompute, and you can — but timing is machine-dependent and noisy. No step is ever graded on a clock. The proof of the win is the deterministic op-count contrast you'll build in the capstone, not the stopwatch.
This is a process step: the grader runs program.ts generate <maxTokens> <id...> and checks that stdout exactly matches the expected line of space-separated token IDs and the process exits 0. Print only the IDs — a stray log line, trailing punctuation, or commas will fail the byte-for-byte match.