Is it any good? (qrels, MRR, nDCG)·12 min

MRR: how high is the first hit?

0 / 1

# grade this stage (from your course directory)
npx theinference test --stage mrr --watch

The simplest ranking-aware metric: Mean Reciprocal Rank. For one query, find the rank of the first relevant doc and take 1/rank. First result relevant → 1.0; second → 0.5; third → 0.33; none in the list → 0. Average over queries → MRR.

MRR

RR = 1 / (rank of first relevant doc)     MRR = mean(RR over queries)

What to build

Your task

Implement reciprocalRank

Implement reciprocalRank(ranking, relevant) in src/metrics.ts: walk ranking (a number[] of doc indices), return 1/(position+1) of the first index that's in relevant, or 0 if none. (Position is 0-based; rank is position+1.)

src/metrics.ts

export function reciprocalRank(ranking: number[], relevant: number[]): number {
  // 1/rank of first relevant; rank is 1-based; 0 if none found
}

Pass the FULL ranking, not a top-10 slice

reciprocalRank must receive the full ranked list (we use the top-100), not a top-10 array. RR is defined over the whole ranking: if a query's first relevant doc sits at rank 14, the correct RR is 1/14, but a top-10 array would wrongly report 0. The eval driver runs search(..., 100) and hands that to the metric.

1-based rank, 0-based array

The single most common bug here: ranks in the formula start at 1 (the top result), but your array is 0-indexed. RR = 1/(i+1), not 1/i. Off-by-one here silently doubles every reciprocal rank.

How it's graded

adapter — allclose vs reference on a hand-built (ranking, relevant) case, atol=1e-6. (The end-to-end mean MRR is graded two steps later, in mrr-eval.)

Qrels: the human ground truth DCG