parseman 0.1.7 → 0.3.0

package/README.md

@@ -6,6 +6,8 @@
 
 Write parsers in TypeScript — fast enough to run as-is, and blazing fast when the bundler macro kicks in. Same code either way; no grammar files, no generated output to check in. Drop the plugin in tests or anywhere a bundler isn't around and everything still works.
 
+_Note: Not necessarily production-ready! I still have test cases and sample parsers I want to build to more rigorously test the API and performance._
+
 ## Benchmarks
 
 Measured on Apple M2 Pro. Bars show µs per parse — shorter is faster.
@@ -14,7 +16,11 @@ Measured on Apple M2 Pro. Bars show µs per parse — shorter is faster.
 
 ![CSV parsing benchmarks](https://raw.githubusercontent.com/matthew-dean/parsecraft/main/assets/bench-csv.svg)
 
-Parséman compiled edges out Peggy on small and medium JSON. At 12 kB Peggy pulls ahead by ~10% — it's been doing this a while. On CSV, where the grammar is non-recursive and fully inlines, Parséman compiled is the clear winner.
+![GraphQL parsing benchmarks](https://raw.githubusercontent.com/matthew-dean/parsecraft/main/assets/bench-graphql.svg)
+
+Parséman has three modes — **interpreter** (zero setup, works anywhere), **macro build** (compiled by the bundler plugin at build time, zero runtime cost), and **`.compile()`** (optional runtime JIT). Most production use lands on one of the first two. The initialization section only shows parsers with a nonzero setup cost: `.compile()` costs 85–680 µs depending on grammar size; Chevrotain always costs 950–1,300 µs. Parsers not listed there start for free.
+
+On JSON, Parséman macro beats Peggy at small and medium sizes; Peggy pulls ahead by ~19% at 12 kB — it's been doing this a while. On CSV and GraphQL, where the grammar is non-recursive or fully inlineable, Parséman macro is the clear winner — CSV macro is now ~6× faster than Peggy on the 500-row fixture. For CST-building grammars (`node()` rules with trivia capture), the interpreter runs ~2× faster than Chevrotain on the JSON CST fixture while building an equivalent tree.
 
 ---
 
@@ -106,7 +112,7 @@ If `with { type: 'macro' }` is stripped (older bundlers, test runners), the attr
 
 ### What gets compiled
 
-Pure combinator trees — `literal`, `regex`, `sequence`, `choice`, `many`, `oneOrMore`, `optional`, `sepBy`, `transform`, `skip`. Parsers using `ref()` for recursion or that close over external variables stay as-is. The plugin compiles what it can and quietly leaves the rest alone.
+Combinator trees — `literal`, `regex`, `sequence`, `choice`, `many`, `oneOrMore`, `optional`, `sepBy`, `transform`, `skip`, `not`, `scanTo`, `balanced` — plus `rules()` factories (including mutually recursive ones), `parser({ trivia })` wrappers, and `node()` rules (CST capture and all). A full grammar built as a `rules()` factory of `node()` rules compiles end to end: each rule becomes an independently-callable function, terminal/trivia capture is emitted inline, and every `build`/`transform` callback is inlined with its source span. Grammars with no `node()` emit zero capture code, so they compile byte-identically. Parsers that close over external variables the evaluator can't resolve stay as-is — the plugin compiles what it can and quietly leaves the rest alone.
 
 ---
 
@@ -115,6 +121,8 @@ Pure combinator trees — `literal`, `regex`, `sequence`, `choice`, `many`, `one
 | Combinator | Description |
 |---|---|
 | `literal(s, opts?)` | Exact string match. `opts.caseInsensitive` for locale-aware comparison. |
+| `word(s, boundary?)` | Single keyword with automatic word-boundary guard. |
+| `wordContext(boundary?)` | Returns a keyword factory with a baked-in boundary (e.g. CSS `A-Za-z0-9_-`). |
 | `regex(pattern)` | Match a regex at the current position. Patterns are optimized via `regexp-tree`. |
 | `sequence(...parsers)` | Match all in order; returns a tuple `[v1, v2, ...]`. Skips trivia between terms when trivia is set. |
 | `choice(...parsers)` | Ordered alternatives (PEG — first match wins). Disjoint first chars → O(1) dispatch. |
@@ -124,29 +132,53 @@ Pure combinator trees — `literal`, `regex`, `sequence`, `choice`, `many`, `one
 | `sepBy(parser, sep)` | Zero or more `parser` separated by `sep`. |
 | `transform(parser, fn)` | Map the result: `fn(value, span) → newValue`. |
 | `skip(main, skipped)` | Match `main` then `skipped`; return `main`'s value. |
-| `parser(factory)` | Mutually recursive grammar rules — no forward declarations needed. |
-| `ref<T>()` | Low-level forward declaration slot (use `parser()` in most cases). |
+| `rules(factory)` | Named grammar rules — no forward declarations needed, handles mutual recursion. |
+| `node(type, parser, build)` | CST/AST rule: captures `parser`'s terminals into `children`/`rawChildren` and trivia offsets into a flat `triviaLog`, then calls `build(children, rawChildren, span, triviaLog)`. |
+| `ref<T>()` | Low-level forward declaration slot (use `rules()` in most cases). |
 | `not(parser)` | Negative lookahead — succeeds (consuming nothing) when `parser` fails. |
 | `guard(predicate)` | Succeeds only when `predicate(ctx)` returns true; used for context-sensitive rules. |
 | `withCtx(extra, parser)` | Merge `extra` into the user context for the duration of `parser`. |
 | `recover(parser, sentinel)` | On failure, skip input until `sentinel` matches; returns a `CSTError` node. |
-| `scanTo(sentinel, skips?, opts?)` | Consume input up to (and including) `sentinel`, optionally skipping balanced pairs. |
-| `balanced(open, close)` | Match a balanced pair (e.g. `(…)`, `[…]`). Used as a `skip` argument to `scanTo`. |
+| `scanTo(sentinel, opts?)` | Scan forward until `sentinel` matches (sentinel not consumed). Pass `opts.skip` to treat certain patterns as opaque blobs that may contain the sentinel character. Pass `opts.orEOF: true` to succeed at end-of-input. |
+| `balanced(open, close, opts?)` | Match a single balanced delimited region — e.g. `(…)` or `[…]` — including the delimiters. Primarily used as an element of `scanTo`'s `opts.skip` list. |
+
+---
+
+## Performance: collapse opaque shapes into one regex
+
+The single biggest grammar-level perf lever is **the number of combinator boundaries on the hot path**. Every `sequence` / `regex` / `oneOrMore` step is a function call plus a result-object allocation plus — in a `node()` rule — a leaf push. Fewer, fatter combinators beat many thin ones.
+
+Measured on a repeated 3-shape group (`name1 1px #111 …`, ~29 KB), parsing the same content three ways:
+
+| Approach | Interpreted | Compiled |
+|---|---|---|
+| `oneOrMore(sequence(ident, sp, num, sp, hex, sp))` | 0.289 ms | 0.167 ms |
+| same, with **inline** `regex(…)` instead of shared refs | 0.281 ms | 0.183 ms |
+| **one** `regex(/…ident…num…hex…/)` per group | **0.055 ms** | **0.042 ms** |
+
+Two takeaways:
+
+- **Shared combinator ref vs inline `regex(…)` literal makes no difference.** Both produce the identical runtime structure (one `regex` combinator either way), and the compiler inlines a `regex` test at every use site regardless of sharing — there's no function-call indirection to eliminate (that only exists for `ref()` / `rules()` entries). Factor out shared terminals for readability; it costs nothing.
+- **Collapsing a fixed multi-token shape into a single `regex` is 4–5× faster** in both the interpreter and compiled output, because it erases the per-step call + allocation overhead. `compile()` is a real but smaller win (~1.7×) and **stacks** with collapsing.
+
+**When to collapse:** only where the CST treats the group as opaque text — a dimension `\d+px`, a hex color, an `nth` expression, a simple ident-run. A single regex yields **one leaf**, not structured sub-nodes.
+
+**When NOT to collapse:** keep the parts as separate combinators wherever the builder needs them as distinct CST children — for per-field source spans (`fieldSpans`/`valueSpans`), for trivia recovered *between* the parts, or for distinct typed nodes. Correctness first; collapse only the genuinely opaque runs.
 
 ---
 
 ## Whitespace and comment skipping
 
-Pass `trivia` to `parse()` and `sequence()` will automatically skip it between terms:
+Wrap your root combinator with `parser()` to declare trivia — whitespace and comments to skip between tokens. This bakes the setting into the combinator tree so all modes (interpreter, `compile()`, macro build) behave identically:
 
 ```ts
-import { regex, sepBy, literal, parse } from 'parseman'
+import { parser, regex, trivia, sepBy, literal } from 'parseman'
 
-const ws   = regex(/\s*/)
+const ws   = trivia(regex(/\s*/))
 const word = regex(/[a-z]+/)
-const list = sepBy(word, literal(','))
+const list = parser({ trivia: ws }, sepBy(word, literal(',')))
 
-parse(list, 'foo ,  bar , baz', { trivia: ws })
+list.parse('foo ,  bar , baz')
 // { ok: true, value: ['foo', 'bar', 'baz'], ... }
 ```
 
@@ -155,15 +187,7 @@ Multiple trivia types — whitespace and comments — combine with `choice()` an
 ```ts
 const lineComment  = sequence(literal('//'), regex(/[^\n]*/))
 const blockComment = sequence(literal('/*'), scanTo(literal('*/'), []))
-const trivia       = many(choice(regex(/\s+/), lineComment, blockComment))
-```
-
-Use `grammar(opts, root)` instead of the `parse()` trivia option when you want trivia only for a subtree within a larger parse:
-
-```ts
-import { grammar } from 'parseman'
-
-const jsonValue = grammar({ trivia: ws }, choice(object, array, str, num, bool, nil))
+const ws           = trivia(many(choice(regex(/\s+/), lineComment, blockComment)))
 ```
 
 ---
@@ -172,7 +196,20 @@ const jsonValue = grammar({ trivia: ws }, choice(object, array, str, num, bool,
 
 `choice()` uses PEG ordered-choice semantics: first match wins. **Order matters.**
 
-For keywords — where `if` should not match the prefix of `ifdef` — use `not()`:
+For keywords — where `if` should not match the prefix of `ifdef` — use `word()` or `wordContext()`:
+
+```ts
+const kw = wordContext()  // bakes in a default identifier boundary once
+
+const token = choice(
+  kw('if'),
+  kw('else'),
+  kw('return'),
+  regex(/[a-zA-Z_]\w*/),  // ident fallback
+)
+```
+
+Or build the boundary guard by hand with `not()`:
 
 ```ts
 const wordChar = regex(/\w/)
@@ -196,20 +233,20 @@ const op = choice(literal('instanceof'), literal('in'), literal('if'))
 
 ---
 
-## Recursive grammars
+## Named and recursive rules
 
-Use `parser()` for mutually recursive rules. Pass a factory that receives all rule names as ready-to-use references and returns the definitions:
+Use `rules()` when your combinators need to reference each other by name. Pass a factory that receives all rule names as ready-to-use references and returns the definitions:
 
 ```ts
-import { parser, choice, sequence, literal, sepBy, transform, regex } from 'parseman'
+import { rules, parser, choice, sequence, literal, sepBy, transform, trivia, regex } from 'parseman'
 import type { Combinator } from 'parseman'
 
 type JSON = null | boolean | number | string | JSON[] | Record<string, JSON>
 
-const ws = regex(/[ \t\n\r]*/)
+const ws = trivia(regex(/[ \t\n\r]*/))
 
-const { value } = parser<{ value: Combinator<JSON> }>(g => {
-  const comma = sequence(ws, literal(','), ws)
+const { value } = rules<{ value: Combinator<JSON> }>(g => {
+  const comma = literal(',')
 
   const array = transform(
     sequence(literal('['), sepBy(g.value, comma), literal(']')),
@@ -225,151 +262,92 @@ const { value } = parser<{ value: Combinator<JSON> }>(g => {
   )
 
   return {
-    value: grammar(
-      { trivia: ws },
-      choice(object, array, jsonString, jsonNumber, jsonBool, jsonNull)
-    ) as Combinator<JSON>,
+    value: choice(object, array, jsonString, jsonNumber, jsonBool, jsonNull) as Combinator<JSON>,
   }
 })
+
+export const jsonParser = parser({ trivia: ws }, value)
+jsonParser.parse('{ "a": 1 }')
 ```
 
-`g.value` is a parser reference that works anywhere inside the factory regardless of order. Local helpers (`comma`, `pair`, `object`) that don't need to be cross-referenced can be plain `const`. Only put a rule in the returned object if other rules need to reach it as `g.xxx`.
+`g.value` is a reference that works anywhere inside the factory regardless of order. Local helpers (`comma`, `pair`, `object`) that don't need to be cross-referenced can be plain `const`. Only put a rule in the returned object if other rules need to reach it as `g.xxx`.
 
-> **Macro note:** Recursive parsers can't be inlined as a single expression, so the plugin leaves them as-is — but you can still call `compile()` at runtime for the same speedups. The codegen emits mutually recursive named functions and handles cycles just fine. Non-recursive leaf parsers in the same file still get inlined.
+> **Macro and `rules()`:** The plugin fully compiles `rules()` factories, including recursive ones. It emits mutually recursive named functions (`_pf0` etc.) so the cycle is broken. Add `with { type: 'macro' }` to your import and the entire grammar — recursive rules included — is inlined at build time. Both binding forms compile: `const { value } = rules(...)` (each rule becomes a top-level function) and `const grammar = rules(...)` (an object literal of compiled rules, so `grammar.value(...)` works).
+>
+> If the plugin meets a macro-imported declaration it can't compile statically (it closes over a runtime value, or isn't a recognized combinator shape), it leaves that declaration for the interpreter, strips the `with { type: 'macro' }` attribute so the import stays valid, and emits a build **warning** (`[parseman] file:line — …`) pointing at it — so a silent fallback never goes unnoticed.
 
 ### `ref<T>()` — low-level forward declaration
 
-`parser()` is the right tool for most recursive grammars. `ref<T>()` is the lower-level primitive it uses internally, exposed for cases where you need a single forward slot outside of a `parser()` call:
+`rules()` handles forward references automatically. `ref<T>()` is the lower-level primitive it uses internally, exposed for cases where you need a single forward slot outside of a `rules()` call:
 
 ```ts
 const value = ref<JSON>()
 // ... build parsers that use value ...
-value.define(grammar({ trivia: ws }, choice(object, array, str, num, bool, nil)))
+value.define(choice(object, array, str, num, bool, nil))
 ```
 
 ---
 
-## Class-based grammars
+## Grammars that build an AST
 
-For grammars that need automatic CST construction, incremental re-parsing, or custom AST nodes, extend `Parser`. Capital-letter rules produce named CST nodes; lowercase rules are transparent helpers whose terminals surface as leaves of the nearest enclosing rule.
+For grammars that produce a typed CST/AST, support incremental re-parsing, or care about trivia, wrap each rule in `node(type, parser, build)`. parseman captures the rule's terminals into `children` / `rawChildren` and records trivia as flat `[start, end, insertIdx, …]` triples in `triviaLog`, then hands all four to `build(children, rawChildren, span, triviaLog)`. **Capture is the library's job:** you don't wrap terminals to recover their spans, and you don't reconstruct trivia — it's collected as the parser runs, in both the interpreter and the compiled build. Use `buildTriviaIndex(tree, input)` to turn `triviaLog` into a before/after lookup table.
 
 ```ts
-import { Parser, parse, regex, literal, choice, sequence, many, sepBy } from 'parseman'
-import type { Refs } from 'parseman'
-
-class ExprParser extends Parser {
-  ws     = regex(/\s*/)
-  digits = regex(/[0-9]+/)
-  ident  = regex(/[a-zA-Z_]\w*/)
-
-  // Plain initializer when no cross-reference needed
-  Str    = sequence(literal('"'), regex(/[^"]*/), literal('"'))
-
-  // Thunk form for forward / mutual references
-  Num    = (g: Refs<ExprParser>) => g.digits
-  Id     = (g: Refs<ExprParser>) => g.ident
-  Add    = (g: Refs<ExprParser>) => sequence(g.Num, many(sequence(literal('+'), g.Num)))
-  Expr   = (g: Refs<ExprParser>) => choice(g.Add, g.Num, g.Id)
-}
-
-const expr = new ExprParser()
-const r = parse(expr.rule('Expr'), '1+2+3')
-// r.value is a CSTNode { _tag: 'node', type: 'Expr', span, children, savedContext }
-```
+import { rules, parser, node, regex, literal, sequence, many, trivia } from 'parseman'
+import type { Combinator } from 'parseman'
 
-### Inheritance
+// Any node shape works as long as it satisfies NodeLike (see below).
+type N = { _tag: 'node'; type: string; span: { start: number; end: number }; state: unknown; children: unknown[] }
+const ws = trivia(regex(/\s+/))
 
-Override any rule by redeclaring it in a subclass — subclass initializers run after the parent's, so the override wins automatically:
+export const { Expr, Num } = rules<{ Expr: Combinator<N>; Num: Combinator<N> }>(g => {
+  const num = node('Num', regex(/[0-9]+/),
+    (children, raw, span, _triviaLog) => ({ _tag: 'node', type: 'Num', span, state: null, children: [...children] }))
+  const expr = node('Expr', parser({ trivia: ws }, sequence(g.Num, many(sequence(literal('+'), g.Num)))),
+    (children, raw, span, _triviaLog) => ({ _tag: 'node', type: 'Expr', span, state: null, children: [...children] }))
+  return { Expr: expr, Num: num }
+})
 
-```ts
-class JSXParser extends ExprParser {
-  // replace just the ident rule; everything else stays
-  ident = regex(/[a-zA-Z_$][\w$]*/)
-}
+Expr.parse('1 + 2 + 3', 0, { trackLines: false })
+// value is a Node whose children are the captured Num sub-nodes and '+' leaves
 ```
 
-### Custom AST nodes (`buildNode`)
-
-Override `buildNode` to return your own node type instead of the default `CSTNode`:
-
-```ts
-import type { CSTLeaf, CSTError, CSTRawChild, Span } from 'parseman'
-
-type MyNode = { _tag: 'node'; type: string; span: Span; savedContext: unknown; children: MyNode[]; text: string }
-
-class MyParser extends Parser<MyNode> {
-  // ... rules ...
+- **`children`** — structural items in source order: spanned `CSTLeaf` terminals (`{ _tag:'leaf', value, span }`) and sub-nodes (whatever a nested `node()`'s `build` returned). A `build` that returns a bare string is recorded by the parent as a spanned leaf, so single-item "collapsing" rules keep their source span.
+- **`rawChildren`** — structural children only (same items as `children`, without trivia tokens).
+- **`triviaLog`** — flat `[start, end, insertIdx, …]` triples for whitespace/comments consumed between terms. `insertIdx` is the `rawChildren` index before which the trivia was consumed. Pass the tree to `buildTriviaIndex(tree, input)` for a `before`/`after` map of trivia tokens — useful for whitespace-sensitive syntax (e.g. CSS `div p` vs `div.p`).
 
-  protected buildNode(
-    type: string,
-    span: Span,
-    children: ReadonlyArray<MyNode | CSTLeaf | CSTError>,
-    savedContext: unknown,
-    rawChildren: ReadonlyArray<CSTRawChild>,
-  ): MyNode {
-    return { _tag: 'node', type, span, savedContext, children: children as MyNode[], text: '...' }
-  }
-}
-```
+Each rule returned from the factory is independently callable — `Expr`, `Num` above are the **rule registry** incremental re-parsing needs. Wrap a rule's inner parser in `parser({ trivia })` so trivia-skipping is baked in regardless of which rule you start from; the macro compiles the wrapper (and all capture) away to flat JS.
 
-`children` contains the structural children (sub-nodes and leaf tokens, no trivia). `rawChildren` contains everything in parse order including trivia tokens — useful for whitespace-sensitive grammars.
-
-### Whitespace-sensitive rules with `rawChildren`
-
-When whitespace is semantically meaningful (e.g. CSS where `div p` is a descendant combinator but `div+p` is adjacent), inspect `rawChildren` inside `buildNode`:
-
-```ts
-import type { CSTTrivia } from 'parseman'
-
-class CssParser extends Parser<SelectorNode> {
-  ident     = regex(/[a-zA-Z-]+/)
-  Selector  = (g: Refs<CssParser>) => sequence(g.ident, g.ident)
-
-  protected buildNode(type, span, children, savedContext, rawChildren) {
-    if (type === 'Selector') {
-      // rawChildren: [Ident("div"), CSTTrivia(" "), Ident("p")]
-      const hasDescendant = rawChildren.some(c => c._tag === 'trivia')
-    }
-    return ...
-  }
-}
-
-// Trivia is set on parse() — the whitespace skip happens globally
-parse(css.rule('Stylesheet'), src, { trivia: many(choice(regex(/\s+/), comment)) })
-```
-
-`CSTTrivia` nodes only appear in `rawChildren`, never in `children`. Zero-length trivia matches (e.g. `\s*` at a non-whitespace position) are not emitted.
+> `transform(p, fn)` is still the tool for plain value-mapping (no children/trivia). `node()` is for CST/AST rules — it adds the capture `transform` doesn't. Both compile under the macro.
 
 ### Incremental re-parsing
 
-`Parser.parse(ruleName, input)` returns a `ParseDoc` — an object holding the tree, any parse errors, and an `edit()` method for incremental re-parsing. The parser itself stays stateless; all the incremental state lives in the doc.
+`makeFunctionalDoc(registry, rootRule, input, opts?)` wraps a parse in a document that re-parses incrementally on edits. The `registry` is the object `rules()` returns (rule name → parser fn); the parser functions stay stateless, all incremental state lives in the doc.
 
 ```ts
-const css = new CSSParser()
+import { makeFunctionalDoc } from 'parseman'
 
-const doc = css.parse('Stylesheet', src)
-doc.tree    // CSTNode root, or null on failure
+const registry = { Expr, Num }              // straight from rules()
+let doc = makeFunctionalDoc(registry, 'Expr', src)
+doc.tree    // your Node root, or null on failure
 doc.errors  // ParseFail[], empty on success
 doc.input   // the source string
 
-// subsequent edits return a new doc — old one is untouched
-// edit(from, to, replacement): select from→to in the old text, replace with replacement
-const doc2 = doc.edit(changeStart, changeStart + changeLength, newText)
+// edit(from, to, replacement) — two byte offsets into the OLD text + the
+// replacement. "Select from→to, type replacement" — the three things every
+// editor knows on each keystroke. Returns a new doc; the old one is untouched.
+doc = doc.edit(changeStart, changeStart + changeLength, newText)
 ```
 
-`edit(from, to, replacement)` takes two byte offsets into the **old** text (`from` and `to`), plus the replacement string. Think of it as "highlight characters from→to, type replacement" — the same three things any editor already knows on every keystroke. Internally it runs `old.slice(0, from) + replacement + old.slice(to)`. It finds the smallest node containing the change, re-parses just that subtree using its saved context, and stops early when the new span end matches the expected position. O(changed region) amortized for typical edits. Nodes unaffected by the edit are structurally shared between old and new docs.
-
-Context-sensitive grammars work correctly: each CST node records a `ctx.user` snapshot at parse time (`savedContext`), so re-parsing resumes from the exact same state. Solid enough for a language server.
+`edit()` finds the smallest node containing the change, re-parses just that rule from its start offset using the node's saved `state`, and grafts the result back in when the new span end lands where the edit's delta predicts. Nodes unaffected by the edit are shared by reference between old and new docs. For class-instance ASTs that can't be shallow-spread, pass `opts.rebuild(node, children)` to control how a parent is reconstructed when a child is replaced.
 
-**In an IDE extension**, hold one parser instance per language, one `ParseDoc` per open document. On each keystroke your editor gives you the changed range as byte offsets — pass those straight to `edit()`:
+**In an IDE extension**, keep one registry per language and one doc per open document. Each keystroke gives you the changed range as byte offsets — pass them straight to `edit()`:
 
 ```ts
-// VS Code example
-const parser = new CSSParser()
-const docs = new Map<string, ParseDoc<CSTNode>>()
+const docs = new Map<string, ReturnType<typeof makeFunctionalDoc<Node>>>()
 
-vscode.workspace.onDidOpenTextDocument(document => {
-  docs.set(document.uri.toString(), parser.parse('Stylesheet', document.getText()))
+vscode.workspace.onDidOpenTextDocument(d => {
+  docs.set(d.uri.toString(), makeFunctionalDoc(registry, 'Stylesheet', d.getText()))
 })
 
 vscode.workspace.onDidChangeTextDocument(event => {
@@ -379,7 +357,23 @@ vscode.workspace.onDidChangeTextDocument(event => {
     doc = doc.edit(change.rangeOffset, change.rangeOffset + change.rangeLength, change.text)
   }
   docs.set(uri, doc)
-  // walk doc.tree to emit diagnostics, folding ranges, semantic tokens, etc.
+  // walk doc.tree for diagnostics, folding ranges, semantic tokens, etc.
+})
+```
+
+### Inheritance — by composition
+
+Share rules across grammar variants by composition: factor common rules into helpers, and let each variant swap the ones it needs. Because every rule is a plain value, a variant is just different pieces passed into the same `rules()` shape — and it all stays macro-compilable.
+
+```ts
+// shared building blocks
+const makeIdent = () => regex(/[a-zA-Z_]\w*/)
+
+export const { Expr } = rules<{ Expr: Combinator<Node> }>(g => {
+  const ident = makeIdent()                       // base
+  // a JSX variant would use regex(/[a-zA-Z_$][\w$]*/) here instead
+  /* ...rules referencing ident... */
+  return { Expr: /* ... */ }
 })
 ```
 
@@ -387,28 +381,26 @@ vscode.workspace.onDidChangeTextDocument(event => {
 
 ## Context-sensitive parsing
 
-`withCtx` and `guard` implement context-sensitive rules without mutating shared state.
+`withCtx` and `guard` implement context-sensitive rules without mutating shared state, and they compose into `rules()` like any other combinator.
 
-`withCtx(extra, parser)` merges `extra` into the user context for the duration of `parser`. `guard(predicate)` succeeds only when `predicate(ctx)` returns true, effectively gating a rule behind runtime context.
+`withCtx(extra, parser)` merges `extra` into the user context for the duration of `parser`. `guard(predicate)` succeeds only when `predicate(ctx)` returns true, gating a rule behind runtime context.
 
 ```ts
-import { withCtx, guard, many, sequence, choice, literal, regex } from 'parseman'
+import { rules, withCtx, guard, many, sequence, choice, literal, regex, trivia, parser } from 'parseman'
+import type { Combinator } from 'parseman'
 
-class LangParser extends Parser {
-  ws = regex(/\s*/)
+const ws = trivia(regex(/\s*/))
 
-  Expr    = regex(/[a-z]+/)
-  Return  = (g: Refs<LangParser>) => sequence(
-    guard((ctx: { inFn?: boolean }) => ctx.inFn === true),
-    literal('return'),
-  )
-  Stmt    = (g: Refs<LangParser>) => choice(g.Return, g.Expr)
-  Body    = (g: Refs<LangParser>) => withCtx({ inFn: true }, many(sequence(g.Stmt, g.ws)))
-  Program = (g: Refs<LangParser>) => many(g.Body)
-}
+export const { Program } = rules<{ Program: Combinator<unknown> }>(g => {
+  const expr   = regex(/[a-z]+/)
+  const ret    = sequence(guard((ctx: { inFn?: boolean }) => ctx.inFn === true), literal('return'))
+  const stmt   = choice(ret, expr)
+  const body   = withCtx({ inFn: true }, many(sequence(stmt, ws)))
+  return { Program: parser({ trivia: ws }, many(body)) }
+})
 ```
 
-`Return` is only reachable inside a `Body` because `guard` rejects it when `inFn` is not set. `ParseDoc.edit()` replays the correct context on incremental edits because `savedContext` captures the `inFn: true` snapshot at the node that originally set it.
+`return` is only reachable inside a body because `guard` rejects it when `inFn` is not set. Incremental `edit()` replays the correct context because each node records the `ctx.state` snapshot (`node.state`) active when it was parsed.
 
 ---
 
@@ -426,7 +418,35 @@ const stmt = recover(g.Stmt, literal(';'))
 const block = scanTo(literal('}'), [balanced(literal('('), literal(')')), balanced(literal('['), literal(']'))])
 ```
 
-`scanTo(sentinel, skips?, opts?)` consumes input character-by-character until `sentinel` matches. Pass `skips` to skip over balanced pairs that might contain the sentinel character. Pass `opts.orEOF: true` to succeed at end-of-input if the sentinel is never found.
+### `scanTo` vs `balanced` — when to use each
+
+Both are **position arithmetic**: pure cursor-advance with zero CST allocation. Neither pushes terminals into the enclosing `node()`'s child list; only the final scanned span appears as a single leaf.
+
+**`scanTo(sentinel, opts?)`** — use when you need to consume an open-ended region whose boundary is a specific token. It scans character-by-character, stopping the moment `sentinel` matches. Pass `opts.skip` to declare patterns that should be treated as opaque blobs so their content never accidentally looks like the sentinel:
+
+```ts
+// Consume a CSS at-rule prelude up to '{' or ';',
+// but don't stop inside parentheses or strings.
+const prelude = scanTo(choice(literal('{'), literal(';')), {
+  skip: [balanced('(', ')'), singleStr, doubleStr],
+})
+```
+
+**`balanced(open, close, opts?)`** — use when you want to match a single self-contained delimited region and get its full text back (delimiters included). This is the natural building block for `scanTo`'s `skip` list:
+
+```ts
+const parenGroup  = balanced('(', ')')   // matches (a + b) including the parens
+const bracketExpr = balanced('[', ']')   // matches [0] including the brackets
+```
+
+`balanced` can itself accept a `skip` list for deeply nested structures:
+
+```ts
+// Match (…) allowing strings inside to contain unbalanced parens
+const parenWithStrings = balanced('(', ')', { skip: [singleStr, doubleStr] })
+```
+
+**The key difference**: `scanTo` scans *until* a boundary; `balanced` matches *across* a known delimited region. Use `balanced` inside `scanTo`'s `skip` to prevent the scanner from stopping at a sentinel that appears inside a nested structure.
 
 ---
 
@@ -479,18 +499,35 @@ type Span = {
 }
 ```
 
-### CST types
+### Node and document types
+
+Any AST your `transform` callbacks produce participates in incremental re-parsing as long as it satisfies `NodeLike` — that's the whole contract:
 
 ```ts
-type CSTNode  = { _tag: 'node';  type: string; span: Span; children: CSTChild[]; savedContext: unknown }
-type CSTLeaf  = { _tag: 'leaf';  value: string; span: Span }
-type CSTError = { _tag: 'error'; type: string; span: Span; expected: string[]; children: CSTChild[]; savedContext: unknown }
-type CSTTrivia = { _tag: 'trivia'; value: string; span: Span }  // only in rawChildren
+type NodeLike = {
+  readonly _tag: 'node'
+  readonly type: string          // the rule name — used as the registry key on re-parse
+  readonly span: Span
+  readonly state: unknown        // ctx.state snapshot at parse time; replayed on edit
+  readonly children: ReadonlyArray<{ readonly _tag: string }>
+}
 
-type CSTChild    = CSTNode | CSTLeaf | CSTError
-type CSTRawChild = CSTNode | CSTLeaf | CSTTrivia | CSTError
+// makeFunctionalDoc<N>(registry, rootRule, input, opts?) → FunctionalDoc<N>
+type Registry<N>    = Record<string, (input: string, pos: number, ctx: ParseContext) => ParseResult<N>>
+interface FunctionalDoc<N extends NodeLike> {
+  readonly tree: N | null
+  readonly errors: ParseFail[]
+  readonly input: string
+  edit(from: number, to: number, replacement: string): FunctionalDoc<N>
+}
+type FunctionalDocOptions<N> = {
+  state?: unknown                                                  // initial ctx.state for the root parse
+  rebuild?: (node: N, children: ReadonlyArray<unknown>) => N       // override for class-instance ASTs
+}
 ```
 
+`children` only needs items carrying a `_tag` so traversal can tell sub-nodes (`_tag: 'node'`) from anything else. The `type` string must match the rule name in the registry so `edit()` can re-parse the right rule.
+
 ---
 
 ## Developing

package/dist/combinators/choice.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"choice.d.ts","sourceRoot":"","sources":["../../src/combinators/choice.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EACV,UAAU,EAC+B,QAAQ,EAClD,MAAM,aAAa,CAAA;AAGpB,KAAK,SAAS,CAAC,CAAC,IAAI,CAAC,SAAS,QAAQ,CAAC,MAAM,CAAC,CAAC,GAAG,UAAU,CAAC,CAAC,CAAC,GAAG,CAAC,SAAS,UAAU,CAAC,MAAM,CAAC,CAAC,GAAG,UAAU,CAAC,CAAC,CAAC,GAAG,KAAK,CAAA;AACvH,KAAK,SAAS,CAAC,CAAC,SAAS,CAAC,UAAU,CAAC,OAAO,CAAC,GAAG,QAAQ,CAAC,OAAO,CAAC,CAAC,EAAE,IAAI;KACrE,CAAC,IAAI,MAAM,CAAC,GAAG,SAAS,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;CAChC,CAAC,MAAM,CAAC,SAAS,UAAU,CAAC,MAAM,CAAC,CAAC,GAAG,CAAC,GAAG,KAAK,CAAA;AAEjD,wBAAgB,MAAM,CAAC,CAAC,SAAS,CAAC,UAAU,CAAC,OAAO,CAAC,GAAG,QAAQ,CAAC,OAAO,CAAC,EAAE,GAAG,CAAC,UAAU,CAAC,OAAO,CAAC,GAAG,QAAQ,CAAC,OAAO,CAAC,CAAC,EAAE,CAAC,EACxH,GAAG,IAAI,EAAE,CAAC,GACT,UAAU,CAAC,SAAS,CAAC,CAAC,CAAC,CAAC,CAqH1B;AAwFD,uEAAuE;AACvE,wBAAgB,mBAAmB,CAAC,CAAC,EAAE,UAAU,CAAC,OAAO,CAAC,GAAG,MAAM,GAAG,IAAI,CAKzE;AAED,qEAAqE;AACrE,wBAAgB,eAAe,CAAC,CAAC,EAAE,UAAU,CAAC,OAAO,CAAC,GAAG;IAAE,MAAM,EAAE,MAAM,CAAC;IAAC,KAAK,EAAE,MAAM,CAAA;CAAE,GAAG,IAAI,CAKhG"}
+{"version":3,"file":"choice.d.ts","sourceRoot":"","sources":["../../src/combinators/choice.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EACV,UAAU,EAC+B,QAAQ,EAClD,MAAM,aAAa,CAAA;AAGpB,KAAK,SAAS,CAAC,CAAC,IAAI,CAAC,SAAS,QAAQ,CAAC,MAAM,CAAC,CAAC,GAAG,UAAU,CAAC,CAAC,CAAC,GAAG,CAAC,SAAS,UAAU,CAAC,MAAM,CAAC,CAAC,GAAG,UAAU,CAAC,CAAC,CAAC,GAAG,KAAK,CAAA;AACvH,KAAK,SAAS,CAAC,CAAC,SAAS,CAAC,UAAU,CAAC,OAAO,CAAC,GAAG,QAAQ,CAAC,OAAO,CAAC,CAAC,EAAE,IAAI;KACrE,CAAC,IAAI,MAAM,CAAC,GAAG,SAAS,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;CAChC,CAAC,MAAM,CAAC,SAAS,UAAU,CAAC,MAAM,CAAC,CAAC,GAAG,CAAC,GAAG,KAAK,CAAA;AAEjD,wBAAgB,MAAM,CAAC,CAAC,SAAS,CAAC,UAAU,CAAC,OAAO,CAAC,GAAG,QAAQ,CAAC,OAAO,CAAC,EAAE,GAAG,CAAC,UAAU,CAAC,OAAO,CAAC,GAAG,QAAQ,CAAC,OAAO,CAAC,CAAC,EAAE,CAAC,EACxH,GAAG,IAAI,EAAE,CAAC,GACT,UAAU,CAAC,SAAS,CAAC,CAAC,CAAC,CAAC,CAiI1B;AAwFD,uEAAuE;AACvE,wBAAgB,mBAAmB,CAAC,CAAC,EAAE,UAAU,CAAC,OAAO,CAAC,GAAG,MAAM,GAAG,IAAI,CAKzE;AAED,qEAAqE;AACrE,wBAAgB,eAAe,CAAC,CAAC,EAAE,UAAU,CAAC,OAAO,CAAC,GAAG;IAAE,MAAM,EAAE,MAAM,CAAC;IAAC,KAAK,EAAE,MAAM,CAAA;CAAE,GAAG,IAAI,CAKhG"}

package/dist/combinators/completions.d.ts

@@ -0,0 +1,17 @@
+import type { Combinator } from '../types.ts';
+/**
+ * Returns the set of expected tokens at the given cursor offset.
+ * Useful for implementing IDE completions.
+ *
+ * Runs the parser on input truncated at `offset` with a probe that tracks the
+ * highest-position failure seen, even when sepBy/many backtrack past the cursor.
+ * Returns the expected tokens from that deepest failure.
+ *
+ * Returns an empty array when the input up to `offset` parses completely
+ * with no failures at or before the cursor.
+ *
+ * The returned strings use the same labels as ParseFail.expected:
+ * quoted literals like `"\"{\""` and regex patterns like `"/[0-9]+/"`.
+ */
+export declare function completionsAt(parser: Combinator<unknown>, input: string, offset: number): string[];
+//# sourceMappingURL=completions.d.ts.map

package/dist/combinators/completions.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"completions.d.ts","sourceRoot":"","sources":["../../src/combinators/completions.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,UAAU,EAA2B,MAAM,aAAa,CAAA;AAEtE;;;;;;;;;;;;;GAaG;AACH,wBAAgB,aAAa,CAC3B,MAAM,EAAE,UAAU,CAAC,OAAO,CAAC,EAC3B,KAAK,EAAE,MAAM,EACb,MAAM,EAAE,MAAM,GACb,MAAM,EAAE,CAYV"}

package/dist/combinators/grammar.d.ts

@@ -1,8 +1,24 @@
-import type { Combinator, ParseResult } from '../types.ts';
+import type { Combinator, ParseContext, ParseResult } from '../types.ts';
 export type ParseOptions = {
-    trivia?: Combinator<unknown>;
     trackLines?: boolean;
+    /**
+     * Enable error recovery. When true, recover() nodes collect their ParseErrors
+     * into a side-channel array rather than (only) embedding them in the value tree.
+     * The returned ParseOk will have an `errors` field listing all recovered errors.
+     * Top-level parse failures (where no recover() node caught the error) still
+     * return ParseFail as usual.
+     */
+    recover?: boolean;
+};
+export type ParserOptions = ParseOptions & {
+    trivia?: Combinator<unknown>;
+    /** Record consumed trivia as CSTTrivia tokens in rawChildren. Default: skip. */
+    captureTrivia?: boolean;
 };
-export declare function grammar<T>(opts: ParseOptions, root: Combinator<T>): Combinator<T>;
-export declare function parse<T>(parser: Combinator<T>, input: string, opts?: ParseOptions): ParseResult<T>;
+export interface ParsemanParser<T> extends Combinator<T> {
+    parse(input: string): ParseResult<T>;
+    parse(input: string, pos: number, ctx: ParseContext): ParseResult<T>;
+}
+export declare function parser<T>(opts: ParserOptions, root: Combinator<T>): ParsemanParser<T>;
+export declare function parse<T>(combinator: Combinator<T>, input: string, opts?: ParseOptions): ParseResult<T>;
 //# sourceMappingURL=grammar.d.ts.map

package/dist/combinators/grammar.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"grammar.d.ts","sourceRoot":"","sources":["../../src/combinators/grammar.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,UAAU,EAAgB,WAAW,EAAE,MAAM,aAAa,CAAA;AAGxE,MAAM,MAAM,YAAY,GAAG;IACzB,MAAM,CAAC,EAAE,UAAU,CAAC,OAAO,CAAC,CAAA;IAC5B,UAAU,CAAC,EAAE,OAAO,CAAA;CACrB,CAAA;AAED,wBAAgB,OAAO,CAAC,CAAC,EAAE,IAAI,EAAE,YAAY,EAAE,IAAI,EAAE,UAAU,CAAC,CAAC,CAAC,GAAG,UAAU,CAAC,CAAC,CAAC,CAuBjF;AAED,wBAAgB,KAAK,CAAC,CAAC,EACrB,MAAM,EAAE,UAAU,CAAC,CAAC,CAAC,EACrB,KAAK,EAAE,MAAM,EACb,IAAI,GAAE,YAAiB,GACtB,WAAW,CAAC,CAAC,CAAC,CAWhB"}
+{"version":3,"file":"grammar.d.ts","sourceRoot":"","sources":["../../src/combinators/grammar.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,UAAU,EAAE,YAAY,EAAE,WAAW,EAAE,MAAM,aAAa,CAAA;AAGxE,MAAM,MAAM,YAAY,GAAG;IACzB,UAAU,CAAC,EAAE,OAAO,CAAA;IACpB;;;;;;OAMG;IACH,OAAO,CAAC,EAAE,OAAO,CAAA;CAClB,CAAA;AAED,MAAM,MAAM,aAAa,GAAG,YAAY,GAAG;IACzC,MAAM,CAAC,EAAE,UAAU,CAAC,OAAO,CAAC,CAAA;IAC5B,gFAAgF;IAChF,aAAa,CAAC,EAAE,OAAO,CAAA;CACxB,CAAA;AAED,MAAM,WAAW,cAAc,CAAC,CAAC,CAAE,SAAQ,UAAU,CAAC,CAAC,CAAC;IACtD,KAAK,CAAC,KAAK,EAAE,MAAM,GAAG,WAAW,CAAC,CAAC,CAAC,CAAA;IACpC,KAAK,CAAC,KAAK,EAAE,MAAM,EAAE,GAAG,EAAE,MAAM,EAAE,GAAG,EAAE,YAAY,GAAG,WAAW,CAAC,CAAC,CAAC,CAAA;CACrE;AAED,wBAAgB,MAAM,CAAC,CAAC,EAAE,IAAI,EAAE,aAAa,EAAE,IAAI,EAAE,UAAU,CAAC,CAAC,CAAC,GAAG,cAAc,CAAC,CAAC,CAAC,CA6BrF;AAED,wBAAgB,KAAK,CAAC,CAAC,EACrB,UAAU,EAAE,UAAU,CAAC,CAAC,CAAC,EACzB,KAAK,EAAE,MAAM,EACb,IAAI,GAAE,YAAiB,GACtB,WAAW,CAAC,CAAC,CAAC,CAehB"}

package/dist/combinators/guard.d.ts

@@ -1,7 +1,7 @@
 import type { Combinator } from '../types.ts';
 /**
  * Zero-width assertion: succeeds (consuming nothing) only when `predicate`
- * returns true for `ctx.user`. Fails otherwise.
+ * returns true for `ctx.state`. Fails otherwise.
  *
  * Intended for use inside sequence() to gate subsequent parsing on runtime
  * context set with withCtx().
@@ -11,5 +11,5 @@ import type { Combinator } from '../types.ts';
  *     literal('return'), optional(expr)
  *   )
  */
-export declare function guard(predicate: (user: unknown) => boolean): Combinator<null>;
+export declare function guard(predicate: (state: unknown) => boolean): Combinator<null>;
 //# sourceMappingURL=guard.d.ts.map

package/dist/combinators/guard.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"guard.d.ts","sourceRoot":"","sources":["../../src/combinators/guard.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,UAAU,EAAyC,MAAM,aAAa,CAAA;AAEpF;;;;;;;;;;;GAWG;AACH,wBAAgB,KAAK,CAAC,SAAS,EAAE,CAAC,IAAI,EAAE,OAAO,KAAK,OAAO,GAAG,UAAU,CAAC,IAAI,CAAC,CAgB7E"}
+{"version":3,"file":"guard.d.ts","sourceRoot":"","sources":["../../src/combinators/guard.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,UAAU,EAAyC,MAAM,aAAa,CAAA;AAEpF;;;;;;;;;;;GAWG;AACH,wBAAgB,KAAK,CAAC,SAAS,EAAE,CAAC,KAAK,EAAE,OAAO,KAAK,OAAO,GAAG,UAAU,CAAC,IAAI,CAAC,CAgB9E"}