Retrieve, measure, de-duplicate·30 min

recall@k & MRR

0 / 1

# grade this stage (from your course directory)
npx theinference test --stage recall-mrr --watch

Now score retrieval across the whole QA set. Run retrieve for every question, then compute mean recall@5 and mean MRR@5 against the gold chunks. This is the first number you can defend: how often does my retriever surface a relevant chunk, and how high?

What to build

Your task

Implement recallAtK + mrrAtK

Implement recallAtK(retrieved, gold, k) — the fraction of queries whose top-k include a gold chunk — and mrrAtK(retrieved, gold, k) — the mean reciprocal rank of the first gold chunk (1-indexed). The recall-mrr <k> command retrieves over all questions and prints {recall, mrr} (rounded to 4 decimals).

src/retrieve.ts

export function recallAtK(retrieved: string[][], gold: string[][], k: number): number
export function mrrAtK(retrieved: string[][], gold: string[][], k: number): number

How the grader runs it

./your_program.sh recall-mrr 5
# -> { recall, mrr }  (recall@5 = 1.0, MRR@5 = 1.0 on this corpus)

How it's graded

process mode, stdout_json_match (tight tolerance). The printed recall and mrr must equal the frozen reference values. Because retrieval is deterministic, your numbers equal the reference's — this is an exact match to a frozen value, not a >= threshold. On this low-distractor corpus both come out to 1.0; the real target (recall@5 >= 0.8) is the lesson framing.

First gold chunk only, 1-indexed

MRR uses the rank of the first gold chunk in the list — don't sum reciprocals of every gold chunk. And rank is 1-indexed: a gold chunk at array index 0 contributes 1/1 = 1, not 1/0.

Retrieve top-k Redundant retrieval