Retrieve, measure, de-duplicate·4 min
Redundant retrieval
0 / 1
Overlap-chunking and near-duplicate passages mean your top-5 can be five rephrasings of the same sentence — wasting budget and starving the prompt of diverse evidence. Maximal Marginal Relevance (MMR) fixes this: pick the next chunk that is relevant to the query and dissimilar to what you've already picked.
MMR (Carbonell & Goldstein 1998)
MMR = argmax over d_i in (R \ S) of
[ lambda * Sim1(d_i, q) - (1 - lambda) * max over d_j in S of Sim2(d_i, d_j) ]