Mean MRR over all queries
# grade this stage (from your course directory)
npx theinference test --stage mrr-eval --watchNow mean it. Run search over every query at full depth (top-100), take the reciprocalRank of each against the qrels, and average. This is the first of your two quality numbers (the second is nDCG, next step).
What to build
Wire the MRR eval
In src/eval.ts, implement evalMrr(pack, rankDepth): for each query run search(..., rankDepth) (rankDepth = 100), get the relevant doc indices from pack.qrels.get(q), take reciprocalRank(ranking, relevant), and return the mean. The CLI command eval-mrr prints { mrr }.
export function evalMrr(pack: Pack, rankDepth: number): number {
// mean reciprocalRank(search(...,rankDepth), [...qrels keys]) over all queries
}
./your_program.sh eval-mrr fixtures/pack # prints {"mrr": ...}
process — your printed { mrr } is exact-matched to the frozen reference value (atol 1e-6). With these qrels the reference is ≈ 0.766563. On a real benchmark you'd set a "≥ measured − headroom" threshold; here the grade is equality to the reference's deterministic mean.
Feed reciprocalRank the full top-100 ranking, not a top-10 slice. If a query's first relevant doc sits at rank 14, a top-10 ranking reports RR = 0 and drags the mean down — a depth bug, not a model bug. rankDepth = 100 (≥ the metric cut, deep enough to contain the first relevant doc) removes it.