Run the grader
# grade this stage (from your course directory)
npx theinference test --stage run-the-grader --watchThis is your first green check. You won't write any agent code yet — you'll run the harness once, prove the toolchain works, and watch the determinism machinery boot. Every later step plugs into this same grader, so it's worth landing the green here and seeing how the pieces move before you depend on them.
Run it
npx theinference login # once — approves in your browser
npx theinference test --stage run-the-grader --watch
That's the whole command. It builds your code, runs the current step's grader, and prints a pass or fail. From here on you'll keep it running in a split pane next to your editor and watch steps turn green as you write them.
The first cc test does two things before it grades anything:
- Starts a local cassette proxy on
127.0.0.1:$PORT— this stands in for the Claude API. It replays recorded responses instead of calling a model, so the same step always behaves the same way. - Copies a seed repo into
$TASK_DIR— a fresh, throwaway working tree your tools act on for real.
It's CPU-only: no API key, no network, no GPU.
The three things that make a green check trustworthy
┌─ cassette proxy (127.0.0.1:$PORT) ─┐ ┌─ sandbox ($TASK_DIR) ─┐
│ replays recorded responses │ │ seed repo, real files │
│ → the MODEL is deterministic │ │ → the TOOLS run for │
└────────────────────────────────────┘ │ real │
│ └────────────────────────┘
▼ ▼
same tool_use every run files written / commands run
╲ ╱
▼ ▼
grade = test exit code + byte diff
(no model judges another model)
- The cassette fixes the model. The proxy replays recorded API responses, so the model's
tool_usecalls are identical on every run. The behavior under test is your code, not the model's mood. - The tools run for real. Your
view,create,bash, and the rest act on the seed repo in$TASK_DIR. The side effects are genuine, not mocked. - The side effects are asserted. "Done" is a test's exit code plus a byte diff of the resulting files against a reference. Never an LLM's verdict.
Freeze the model, run the tools for real, assert the side effects: that's why a green check here means the same thing every time, on every machine.
Grading compares stdout exactly. Send debug output to stderr (console.error) — a stray console.log will fail an exact-match check. Keep the grader open in a split pane; it re-runs as you work.
How it's graded
This step is a process grade: the harness runs your toolchain and compares stdout, exactly, against fixtures/expected.txt. The whole bar to clear is a single line:
theinference: claude-code toolchain OK