Build your own RAG pipeline
Fixed-size chunking·25 min

Fixed-size chunking

0 / 1
# grade this stage (from your course directory)
npx theinference test --stage fixed-size-chunking --watch

The simplest splitter: fixed-size token windows with overlap. Slide a window of size tokens across the paragraph, stepping by size - overlap each time. The last window may be short.

What to build

Your task

Implement chunkFixed

Given a paragraph's tokens: string[], a size, and an overlap, emit chunks of size tokens stepping by stride = size - overlap. The final chunk may be short; emit no empty chunks. Build each chunk's id and text from its offsets.

src/chunk.ts
export function chunkFixed(
  paragraphId: string, tokens: string[], size: number, overlap: number,
): Chunk[] {
  // stride = size - overlap; for start = 0; start < tokens.length; start += stride
  //   end = min(start + size, tokens.length)
  //   id = `${paragraphId}::${start}-${end}`,  text = detokenize(tokens.slice(start, end))
}
How the grader runs it
./your_program.sh chunk fixed "eiffel-tower#p1" 12 4
# -> JSON array of {id, paragraphId, startTok, endTok, text}, compared exactly
How it's graded

process mode, stdout_json_match. The chunk fixed <paragraphId> <size> <overlap> command prints the chunk records; your token offsets, ids, and text must match the reference exactly.

Stride is size minus overlap

With size=12, overlap=4 you advance 8 tokens each step — adjacent chunks share their last/first 4 tokens. Guard against overlap >= size: that makes stride <= 0 and loops forever.

The chunk recordWhy overlap