Recursive chunking
# grade this stage (from your course directory)
npx theinference test --stage recursive-chunking --watchTime to build the recursive splitter — the LangChain RecursiveCharacterTextSplitter algorithm. Split on the separator hierarchy, greedily pack pieces up to maxTokens, and recurse with the next separator into anything still over budget.
What to build
Implement chunkRecursive
Split text on the separator hierarchy ["\n\n", "\n", ". ", " "]. At each level: split on the current separator, then greedily merge adjacent pieces (re-joined by that separator) while the merged countTokens stays <= maxTokens. Close a chunk the moment the next piece would exceed the budget. Recurse into any single piece that is itself over budget using the next separator. Finally map each fragment back onto token offsets in the paragraph.
export function chunkRecursive(
paragraphId: string, text: string, maxTokens: number,
separators: string[] = ["\n\n", "\n", ". ", " "],
): Chunk[] {
// 1. recursively split text into fragments each <= maxTokens (greedy merge)
// 2. walk the paragraph token array, assigning each fragment its [startTok,endTok)
}
./your_program.sh chunk recursive "moon-landing#p1" 20
# -> chunk records, every chunk countTokens(text) <= 20, boundaries exact
process mode, stdout_json_match. Every chunk must satisfy countTokens(text) <= maxTokens, and the boundaries must match the reference exactly — including not splitting mid-sentence when a . split would have fit.
When merging pieces, add the next one only if the merged token count is still <= maxTokens. The moment it would exceed, close the current chunk and start a new one. Off-by-one here is the #1 boundary-mismatch bug — close before exceeding, never after.