Recursive & sentence-window chunking·4 min
Sentence segmentation
0 / 1
Both recursive (. level) and sentence-window chunking need to split text into sentences deterministically. We do not use an NLP model — we use a specified regex plus a shipped abbreviation guard list so Dr. King isn't split into two sentences. This is re-implementable and bit-reproducible.
src/chunk.ts
export function splitSentences(text: string): string[]
// 1. collapse whitespace to single spaces, trim.
// 2. split on /(?<=[.!?])\s+/ (sentence-final punctuation + whitespace).
// 3. abbreviation guard: if a piece ends in a known abbreviation
// (mr, dr, u.s, e.g, ...), keep accumulating instead of closing.
Determinism > linguistic perfection
Real systems use spaCy or NLTK Punkt for sentence segmentation. We pin a regex so your boundaries are bit-identical to the reference. For a graded course, reproducibility beats edge-case accuracy.