Cypher-lite front-end: tokenize then parse·12 min

The tokenizer

0 / 1

# grade this stage (from your course directory)
npx theinference test --stage tokenizer --watch

Lex a query string into a token stream — keywords, identifiers, strings, integers, punctuation. The scanner skips whitespace, recognizes the fixed keyword set (case-insensitive, emitted UPPERCASED), single-quoted strings, integers, multi-char punctuation, identifiers, and ends with an EOF token.

Your task

Implement tokenize

Keywords: MATCH WHERE RETURN DISTINCT ORDER BY LIMIT AND ASC DESC TRUE FALSE. Multi-char punctuation: -> <- .. <> <= >=. Single-char: ( ) [ ] { } : , . - * = < >. Return Token[] ending in {kind:'EOF', value:''}.

src/tokenizer.ts

type Token = { kind: 'KEYWORD'|'IDENT'|'STRING'|'INT'|'PUNCT'|'EOF'; value: string };
export function tokenize(query: string): Token[] {
  // skip ws; '...' -> STRING; digits -> INT; ident -> KEYWORD (if reserved) or IDENT;
  // try 2-char punct before 1-char; push { kind:'EOF', value:'' } at the end.
}

Process command

your_program.sh tokenize "MATCH (a)-[:ACTED_IN*1..2]->(b) RETURN a"
# -> [{"kind":"KEYWORD","value":"MATCH"}, ... {"kind":"PUNCT","value":"*"},
#     {"kind":"INT","value":"1"}, {"kind":"PUNCT","value":".."}, ...]

Longest-match-wins

Tokenize -> and .. as single tokens, greedily. If you read - then > separately, the parser can't tell a relationship arrow from a minus sign, and .. from two stray dots. Always try the 2-character punctuation before the 1-character ones.

How it's graded

Process mode, stdout_json_match against the exact token array (kind + value per token). The fixture query includes *, 1, .., 2, -> to exercise the tricky punctuation.

The Cypher-lite grammar Parse a node pattern