Retrieve, measure, de-duplicate·5 min
Gold chunks & metrics
0 / 1
To measure retrieval you need a notion of relevant. Define it: a chunk is relevant to a question if its paragraphId is the question's goldParagraphId. The set of relevant chunk ids per question is its gold chunks — computed from the chunking, so it tracks whichever chunker you ran.
Two metrics over the gold chunks:
- recall@k — the fraction of questions whose top-k retrieved chunks include at least one gold chunk. Did a gold chunk make the cut?
- MRR@k — the mean of
1/rankof the first gold chunk in the top-k (0 if none). How high did it rank? (1-indexed.)
Worked example
query top-5 ids: [ c9, c2(gold), c7, c1, c4 ]
recall@5: a gold chunk is present -> contributes 1
MRR@5: first gold chunk is at rank 2 -> reciprocal rank 1/2 = 0.5