The answer-span oracle·20 min

Answer-span hit-rate@k

0 / 1

# grade this stage (from your course directory)
npx theinference test --stage hit-rate --watch

Implement the answer-span oracle as a metric. For each question, check whether any retrieved top-k chunk contains the gold answer span (normalized). The mean over all questions is hit-rate@k — the "is the answer even retrievable" gate.

What to build

Your task

Implement hitRateAtK

Implement hitRateAtK(retrievedTexts, goldAnswers, k): for each question normalize the gold answer and each of the top-k chunk texts, mark a hit if any normalized chunk text includes the normalized gold span, and return the mean hit fraction. The hit-rate <k> command runs it over the whole QA set.

src/retrieve.ts

export function hitRateAtK(
  retrievedTexts: string[][], goldAnswers: string[], k: number,
): number {
  // hit if some normalize(top-k text).includes(normalize(goldAnswer)); return mean
}

How the grader runs it

./your_program.sh hit-rate 5
# -> { hitRate }   (0.9355 on this corpus)

How it's graded

process mode, stdout_json_match (tight tolerance). Your printed hitRate must equal the frozen reference value exactly. The real-world target is hit-rate@5 >= 0.78; the grade is an exact match to the measured reference (0.9355 here).

The answer-span oracle Query rewriting