Citation correctness
# grade this stage (from your course directory)
npx theinference test --stage citation-correctness --watchA right answer with a wrong citation is still a bug — and an invisible one to LLM-judge evals that only check the answer. This step grades citation correctness: every cited chunk must (a) belong to the gold paragraph and (b) contain the gold span. Compute the fraction of citations that pass.
What to build
Implement citationPrecision
Implement citationPrecision(records): across all cited chunks, count how many are both a gold chunk and contain the normalized gold span; divide by the total cited. An out-of-range [doc i] marker (a hallucinated doc) counts as wrong. The citation-precision <k> <budget> command generates over the QA set and prints {citationPrecision}.
export function citationPrecision(records: {
citedChunkIds: string[]; goldChunks: string[]; goldAnswer: string;
chunkText: Record<string, string>;
}[]): number {
// for each cited id: correct if goldChunks.has(id) && normalize(text).includes(goldAnswer)
// return correct / totalCited
}
./your_program.sh citation-precision 5 200
# -> { citationPrecision } (1.0 on this corpus)
process mode, stdout_json_match. The printed precision must equal the frozen reference (1.0 here). Real-world target: >= 0.85. Any hallucinated [doc i] id (out of range) drags precision down — exactly the failure mode this catches.
The model answers correctly and cites [doc 3], but the answer actually came from [doc 1] — right answer, wrong citation. LLM-judge evals that only score the answer miss this entirely. The span check doesn't: it requires the cited chunk to literally contain the gold span.