nDCG@10 vs the real qrels
# grade this stage (from your course directory)
npx theinference test --stage ndcg-eval --watchThe headline quality number. Run your engine over every query against the real qrels and you get the same kind of figure a dense-retrieval paper reports. On the real SciFact benchmark this model scores nDCG@10 ≈ 0.645; on our synthetic qrels you land at a frozen value around 0.77.
What to build
Wire the nDCG eval
In src/eval.ts, implement evalNdcg(pack, k, rankDepth): for each query run search(..., rankDepth) (rankDepth = 100), get the relevance Map from pack.qrels.get(q), take ndcgAtK(ranking, relevant, k) with k=10, and return the mean. The CLI command eval-ndcg prints { ndcg }.
export function evalNdcg(pack: Pack, k: number, rankDepth: number): number {
// mean ndcgAtK(search(...,rankDepth), qrels.get(q), k) over all queries
}
./your_program.sh eval-ndcg fixtures/pack # prints {"ndcg": ...}
process — your printed { ndcg } is exact-matched to the frozen reference value (atol 1e-6), which is ≈ 0.772590 on this data. The "≥ 0.60" framing you'd use for the real model lives in the prose; the grade is equality to the reference's deterministic mean.
If you're at 0.0, your index→doc_id mapping (or the gains→relevance lookup) is off. At ~0.3, suspect the discount base (ln vs log2) or a tie-break. Near the reference, ship it.