Load the corpus·3 min
Paragraphs & tokens
0 / 1
The corpus is an array of {paragraphId, title, text}. You'll also meet countTokens(text) — the token counter you'll use for token budgets later. Chars lie; tokens are the real constraint a model's context window and your bill are measured in.
src/data.ts / src/tokenizer.ts
export interface Paragraph { paragraphId: string; title: string; text: string }
// A deterministic word/punctuation tokenizer (no model download).
export function tokenize(text: string): string[] // runs of word chars or single punctuation
export function detokenize(tokens: string[]): string // tokens joined with single spaces
export function countTokens(text: string): number // = tokenize(text).length
Tokens are the real unit
Real RAG counts BPE tokens; we ship a simpler word/punctuation tokenizer so "token budget" is real, deterministic, and re-implementable. The rule of thumb still holds: a chunk's token count is what fills the context window, not its character count.