The multi-feature suite
# grade this stage (from your course directory)
npx theinference test --stage the-suite --watchThis is it — the capstone. Step 01 proved the wiring on the simplest possible task. Now you run a frozen multi-feature suite, where each task exercises several subsystems at once instead of one in isolation. A single task fixes a bug that needs an MCP tool call, runs under a settings.json carrying a deny rule (Bash(rm *)) and a PostToolUse lint hook, and is kicked off by a skill (/fix). Your agent plans, asks permission, calls the MCP server, fires the hook after each edit, edits the source, runs the test, sees it go green, and stops.
Nothing about this is new code. Every organ is the one you already built and graded. The capstone proves they hold together when they all fire on the same turn.
/fix "rotation bug" ── skill: loads the body, expands $ARGUMENTS
▶ plan: view test, view src ── read tools
▶ tools/call get_spec ── MCP client: fetch the token-rotation spec
▶ Edit src/auth/refresh.ts ── permission: ask→allow (in-tree edit)
└─ PostToolUse hook: run the linter ── hook fires after the edit lands
▶ Bash "rm -rf node_modules" ── DENY ── settings.json deny rule blocks it
▶ Bash "npm test" ── PASS ── oracle goes green
▶ end_turn ── bounded stop
What to build
Run the frozen suite
Implement runSuite(tasks) — loop your Chapter-15 runTask over every task in the frozen suite and collect a result per task. Each task carries its own seed (files), its own cassette, its own settings.json (deny rule + hook), the paths to report, and its own oracle. Run them in fixture order; for each, return { trajectory, files, oracleExit }. Because runTask already wires permissions, hooks, MCP, and skills under the loop, the suite is a thin driver over it — the work is making sure each subsystem's decision lands and is recorded.
Graded by program.ts run-suite: it runs the whole suite against the frozen cassettes and compares the array of per-task results to the reference.
import type { SuiteTask, SuiteResult } from '../src/types.ts'
export async function runSuite(tasks: SuiteTask[]): Promise<SuiteResult> {
const results = []
for (const task of tasks) {
// each task is a fresh sandbox + its own settings.json (deny rule + hook) + cassette
// runTask wires permissions, hooks, MCP, and skills under the loop
const { trajectory, files, oracleExit } = await runTask(task)
results.push({ trajectory, files, oracleExit })
}
return { results }
}
It is not enough for the tests to pass. The grader asserts the deny decision and the hook decision are the expected ones: the planted Bash(rm *) must be blocked before any side effect (its tool_result is an error, the directory is untouched, and it is not a trajectory entry), and the PostToolUse lint hook must have fired after the edit. An agent that quietly skips the rm instead of letting the engine deny it, or that never triggers the hook, fails even with a green oracle. The whole course's thesis is that the decisions are real — so the capstone checks them.
Build a new sandbox per task and load that task's settings.json, .mcp.json, and skills — no state leaks between tasks. Run them in fixture order so results[i] lines up with expected.results[i]. If a single MCP server is shared across the suite, still re-handshake per task or reset its state; a result that depends on a prior task's leftover files is exactly the flakiness the determinism contract forbids.
Process mode: your_program.sh run-suite runs program.ts run-suite against fixtures/case.json (the frozen task array) and compares your { results: [...] } to fixtures/expected.json. Per task it checks: trajectory equal (the real executed calls, after permission/hooks), files byte-equal to the reference fix (expected/), and oracleExit: 0. The cassette fixes the model; your tools, permission engine, hooks, and MCP client run for real — so a wrong edit, a missed hook, a mis-applied deny, or a malformed JSON-RPC frame each yield a deterministic, located failure. Green across every task means the whole machine works the same way every run.