Pipeline scorecard
# grade this stage (from your course directory)
npx theinference test --stage scorecard --watchThe final report — the screenshot you can defend. Run the best chunker over all QA pairs and report every number that matters: recall@5, MRR@5, hit-rate@5, citation-precision, mean context tokens, and mean chunks per query. One schema, every number reproducible, no LLM judge anywhere.
What to build
Implement the scorecard command
The scorecard <k> <budget> command runs the recursive chunker over every question — retrieve, assemble, generate, cite — and aggregates the six metrics into one JSON object. It composes everything you've built; the wiring is in cli.ts.
out({ recall, mrr, hitRate, citationPrecision,
meanContextTokens, meanChunksPerQuery });
./your_program.sh scorecard 5 200
# -> { recall:1, mrr:1, hitRate:0.9355, citationPrecision:1, meanContextTokens, meanChunksPerQuery }
process mode, stdout_json_match. The report schema and every number must match the frozen reference. Real-world floors: recall@5 >= 0.80, MRR@5 >= 0.55, hit-rate@5 >= 0.80, citation-precision >= 0.85. Here the grade is an exact match to the measured reference.
recall@5 = 1.0 · MRR = 1.0 · hit-rate@5 = 0.94 · citation-precision = 1.0 — PASS. Every number is reproducible and LLM-judge-free. You can point at it and say exactly how often your retrieval surfaces the answer and how often your citations are right.