Is it any good? (qrels, MRR, nDCG)·12 min

nDCG@k

0 / 1

# grade this stage (from your course directory)
npx theinference test --stage ndcg --watch

Put it together: nDCG@k = DCG@k / IDCG@k, a 0–1 score where 1 = a perfect ranking. The "n" is normalized — dividing by IDCG makes scores comparable across queries with different numbers of relevant docs.

nDCG@k

nDCG@k = DCG@k / IDCG@k          (0 if IDCG@k is 0)

What to build

Your task

Implement ndcgAtK

Implement ndcgAtK(ranking, relevant, k) in src/metrics.ts — ranking is the full ranked list (top-100); relevant is a Map<docIndex, relevance>. Take the first k ranked docs, map each to its relevance (the Map value, or 0 if absent) to get gains, then return dcgAtK(gains, k) / idcgAtK([...relevant.values()], k) (return 0 if IDCG is 0).

src/metrics.ts

export function ndcgAtK(ranking: number[], relevant: Map<number, number>, k: number): number {
  // gains = ranking.slice(0,k).map(d => relevant.get(d) ?? 0)
  // return dcgAtK(gains,k) / idcgAtK([...relevant.values()], k)   (0 if idcg===0)
}

How it's graded

adapter — allclose vs reference on a hand-built (ranking, relevant) case, atol=1e-6. (The end-to-end mean nDCG@10 is graded two steps later, in ndcg-eval.)

Gains come from the ranking, IDCG from all relevant docs

A frequent slip: the gains list is indexed by rank position (the first k ranked docs, scored 1 if relevant else 0). The IDCG denominator uses all the relevant docs' values, regardless of where (or whether) they were retrieved. Mixing those two up tanks your nDCG.

IDCG Mean MRR over all queries