Quantized search + rescore
# grade this stage (from your course directory)
npx theinference test --stage quantized-search --watchPut quantization to work. Pull an oversampled candidate pool (k × oversample) from the index, then rescore it two ways — by int8 distance (no rescore) and by exact float32 distance (rescore) — and compare recall. Rescoring is what earns back the precision quantization gave away.
What to build
Print quantized recall, with and without rescore
Wire the quant-recall subcommand. For each query, take the index's top k × oversample candidates; rank them by int8 distance (no-rescore top-k) and by float32 distance (rescored top-k); average recall@10 of each against brute-force ground truth. Print:
{ "rescoredRecall": ..., "noRescoreRecall": ..., "floatBytes": ..., "int8Bytes": ... }
./your_program.sh quant-recall <fixture>/chapters/08-quantization/steps/03-quantized-search/fixtures/data.json
Set the candidate pool to k × oversample before rescoring — rescoring can only reorder what the search surfaced. If recall is stuck, raise oversample, not efSearch. And note int8Bytes is exactly a quarter of floatBytes — the 4x memory win, asserted from byte counts.
Process mode: stdout_json_match against the frozen reference (atol 1e-6 on recall). The reference shows rescoredRecall meaningfully above noRescoreRecall — proving rescoring earns its keep — and int8Bytes = floatBytes / 4.