Is it any good? (qrels, MRR, nDCG)·5 min
Qrels: the human ground truth
0 / 1
recall=1.0 only says "my brute force matches my brute force." The real question is whether the retrieved docs answer the query. For that we need humans. qrels ("query relevance judgments") are exactly that: for each query, which docs a human marked relevant. Our qrels.json holds those judgments — binary (relevant = 1) for this dataset, like the real SciFact benchmark.
qrels.json shape — { queryIndex: { docIndex: relevance } }
{ "0": { "47": 1 }, "1": { "12": 1, "133": 1 }, ... }
This is what BEIR scores
On the real SciFact benchmark, all-MiniLM-L6-v2 scores nDCG@10 ≈ 0.645 against these human judgments — a number published in the literature. The metric you're about to build computes that exact figure with your own code. (Here, on synthetic qrels, you'll land at a frozen value around 0.77 — see NOTES.md.)