Bake-off, CLI & scorecard·25 min

End-to-end RAG CLI

0 / 1

# grade this stage (from your course directory)
npx theinference test --stage rag-cli --watch

Wire the whole thing into one command. ask runs the full pipeline — chunk → embed → retrieve → assemble → buildPrompt → generate (cassette) → cite — and prints the context block, the grounded answer, the citation list, and the integrity flags answer_in_context and citation_correct. This is the capstone deliverable.

What to build

Your task

Wire the ask command

The ask <questionIndex> <k> <budget> command runs retrieve → assemble → prompt → generate, parses citations, and computes the two boolean flags. answer_in_context = any used chunk contains the gold span; citation_correct = every cited chunk is a gold chunk that contains the span. It prints one JSON object. Every piece is already built — this assembles the loop.

cli.ts

out({ question, context: ctx.contextString, answer,
      citations: cited.map((id) => ({ chunkId: id, sourceId: paraById.get(id)! })),
      answer_in_context: answerInContext, citation_correct: citationCorrect });

How the grader runs it

./your_program.sh ask 0 5 200
# -> { question, context, answer, citations, answer_in_context: true, citation_correct: true }

How it's graded

process mode, stdout_json_match. The full block — context, answer, citations, and both flags — must match the reference exactly for the demo question. The assembled prompt is byte-identical to the cassette recording, so the answer lookup never misses.

Reuse your assembly output verbatim

Two ways to break the byte-exact match: (1) reformatting or pretty-printing the context/prompt for display before hashing — reuse your assembleContext/buildPrompt output exactly as the cassette key; (2) changing the retrieval path so a different chunk set assembles a different prompt. The cassette is keyed on the exact prompt; any drift is a miss.

Run the bake-off Pipeline scorecard