Recursive & sentence-window chunking·5 min
Structure-aware splitting
0 / 1
Fixed-size chunking cuts mid-sentence. Recursive splitting does better: it tries the most semantic separator first (\n\n paragraphs), and only falls to finer separators (\n → . sentences → words) for pieces still over budget. The result respects natural boundaries while staying under maxTokens.
Recursion tree (maxTokens = 120)
300-token text
split on "\n\n" -> [150, 150] (both still over budget)
150 -> split on ". " -> [70, 80] (pack greedily, both fit)
150 -> split on ". " -> [60, 90]
result: 70, 80, 60, 90 (all <= 120, all on sentence boundaries)