Bake-off, CLI & scorecard·30 min

Run the bake-off

0 / 1

# grade this stage (from your course directory)
npx theinference test --stage run-bakeoff --watch

Run the ablation. For each chunker, chunk the corpus, embed, retrieve over all questions, and compute recall@5 + hit-rate@5. Emit the comparison table. Hold everything else fixed so the only variable is the chunker.

What to build

Your task

Implement the bakeoff command

The bakeoff <k> command loops over ["fixed", "recursive", "sentence-window"], rebuilds the index per chunker, retrieves over the QA set, and prints {table: [{chunker, recall, hitRate}, ...]}. The pipeline pieces are all in place from earlier chapters — this wires them into the loop.

cli.ts

const table = ["fixed","recursive","sentence-window"].map((chunker) => {
  const { qa, index, gold, textById } = buildPipeline(chunker);
  const ids = retrieveAll(qa, index, k);
  const txts = /* top-k texts per question */;
  return { chunker, recall: round(recallAtK(ids, gold, k)),
           hitRate: round(hitRateAtK(txts, qa.map(q=>q.goldAnswer), k)) };
});

How the grader runs it

./your_program.sh bakeoff 5
# -> { table: [ {chunker:"fixed",...}, {chunker:"recursive",...}, {chunker:"sentence-window",...} ] }

How it's graded

process mode, stdout_json_match. The table must match the reference exactly — same chunker order, same recall and hit-rate per row. Sentence-window's hit-rate beats fixed-size's, which is the point of the exercise.

Hold everything else fixed

Same k, same query embeddings, same metric — only the chunker changes. That's what makes it an ablation, not a confound. If you also changed k between rows, you couldn't attribute the difference to the chunker.

The chunking bake-off End-to-end RAG CLI