Shipped a deterministic tokenizer — no AI involved
I needed a tokenizer that always produces the same tokens for the same input. Instead of prompting a model, I wrote a small state machine.
Why? Flaky tokens break snapshots and make diffs noisy. Determinism wins for tooling.
typescript
export function tokenize(input: string) {
const out = [];
let i = 0;
while (i < input.length) {
const ch = input[i]!;
if (/\s/.test(ch)) { i++; continue; }
if (/[a-zA-Z_]/.test(ch)) {
let j = i + 1;
while (j < input.length && /[\w]/.test(input[j]!)) j++;
out.push({ kind: "ident", value: input.slice(i, j) });
i = j; continue;
}
out.push({ kind: "op", value: ch }); i++;
}
return out;
}
Comments (0)
Sign in to comment.