The chunk record

0 / 1

Every chunk you emit is a record: {id, paragraphId, text, startTok, endTok}. The id is ${paragraphId}::${startTok}-${endTok}. The offsets are token offsets into the paragraph's token array, half-open [startTok, endTok) — so boundaries grade integer-exactly no matter how you slice the string.

src/chunk.ts

export interface Chunk {
  id: string;          // `${paragraphId}::${startTok}-${endTok}`
  paragraphId: string;
  text: string;        // = detokenize(tokens.slice(startTok, endTok))
  startTok: number;    // inclusive
  endTok: number;      // exclusive
}

Token offsets, not character offsets

This is grade-critical. tokenize(text) gives the token array; detokenize(tokens) joins tokens with single spaces. Your chunk's text must equal detokenize(tokens.slice(startTok, endTok)) exactly — so build text from the offsets, don't substring the raw string.

Why chunk at all Fixed-size chunking