ibex 0.1.0 → 0.2.0

data/docs/architecture.md

@@ -3,26 +3,177 @@
 Ibex keeps syntax, grammar meaning, automaton construction, and output concerns behind two versioned immutable contracts.
 
 ```text
-.y Lexer -> Token adapter -> self-hosted LR Parser ─┐
-Ruby DSL ────────┴─> Grammar AST -> Normalizer -> Grammar IR
+.y root/fragments -> Frontend Lexer/CST -> self-hosted LR Parser -> canonical Resolver ─┐
+Ruby DSL ───────────────────────────────────────────────────────┴─> Grammar AST -> Normalizer -> Grammar IR
+                                                                               |              |
+                                                                               |          Lexer IR v1
                                                     |
                                                set analysis
                                                     |
-                                      SLR/LALR/LR1 Builder -> Automaton IR
+                                      SLR/LALR/IELR/LR1 Builder -> Automaton IR
                                                                     |
-                              Ruby/RBS generators / report / DOT / HTML / counterexamples
+                       Ruby/RBS/action-shadow generators / report / DOT / Mermaid / HTML / counterexamples
 ```
 
 Frontend changes stop at the Normalizer. Algorithm strategies consume Grammar IR and produce identical Automaton IR shapes.
 Outputs consume Automaton IR and never call builder internals. The CLI only connects stages and supports JSON resumption.
 
+An optional root-only `lexer` declaration normalizes to independently versioned
+Lexer IR v1 and is embedded unchanged in Grammar IR v2. Its flat rule list
+records state, declaration id, pattern source/options, action, and provenance;
+`--emit=lexer-ir` exposes the same document. Code generation compiles every
+pattern with an internal current-position anchor and emits immutable,
+state-indexed rules. Per-parser mutable input, position, emission, and state
+stacks live in `Runtime::GeneratedLexer`, never in the tables. See
+[ADR 0014](decisions/0014-versioned-generated-lexer.md).
+
+`pragma cst` remains an optional Grammar IR v2 flag. Regenerated format-v6
+tables add deterministic kind and normalized slot metadata. `Runtime::Parser`
+builds a pure-syntax Green entry for every shift and reduction on a stack
+parallel to the semantic stack, so tree shape no longer depends on semantic
+actions. Generated lexer skips become token-owned trivia. Parser failure,
+repair, and lexer failure remain explicit and lossless. See
+[ADR 0016](decisions/0016-red-green-concrete-syntax.md) and the
+[CST guide](cst.md). Batch CST, typed views, editing, and serialization form
+the Stable v1 contract. The runtime accepts structured CST metadata only in
+the current format and rejects older CST tables before reading input; see
+[ADR 0008](decisions/0008-versioned-runtime-package-boundary.md).
+
+## Red/Green CST v2
+
+Green nodes and tokens contain no parents, source objects, absolute positions,
+semantic values, or parser states. Their integer kinds, binary text, trivia,
+flags, widths, children, and descendant counts are immutable and
+Ractor-shareable. A session-owned `NodeCache` interns unannotated values. Lazy
+Red wrappers add occurrence-specific parent, index, offset, span, location, and
+typed-field navigation. The root is `source_file(start, EOF)`, and
+`to_source` is byte exact for the `leading` and `balanced` trivia policies.
+
+Generated `Parser::Syntax::<Name>` classes are typed views over Red nodes using
+the same normalized `@node` metadata as Data AST generation. Persistent edits
+replace one Green occurrence and copy its ancestor path; rewriters, batched
+editors, annotations, and identity-skipping text diffing share that mechanism.
+See [ADR 0017](decisions/0017-persistent-syntax-artifacts.md).
+
+`ibex_cst` schema v1 serializes the Green root, kind metadata, compatibility
+counts, and optional preorder parser memo independently of Grammar IR.
+Validation reconstructs every derived width, flag, and descendant count.
+Non-UTF-8 bytes use canonical Base64. See
+[ADR 0017](decisions/0017-persistent-syntax-artifacts.md).
+
+Incremental sessions are syntax-only: parser production actions do not run.
+The generated lexer first validates token/state resynchronization. `Blender`
+then offers either a fresh token or an old Green nonterminal to the LR driver.
+A subtree is pushed directly through `goto` only when damage, recorded
+left-state, follow-token identity, error flags, and positive width satisfy the
+conservative reuse proof. Token and parse memos remain preorder/occurrence
+state owned by one session; resource exhaustion falls back to the fresh token
+stream. See [ADR 0018](decisions/0018-conservative-incremental-syntax-reuse.md).
+
+Alternative-level `@node` declarations are preserved as Grammar IR v2
+production metadata. Runtime Ruby, static action-shadow Ruby, and generated
+RBS all derive Data node classes and Visitor/Listener hooks from that same
+metadata; action source is never inspected to infer a shape. Symbol semantic
+types and fully annotated nonterminals supply field types.
+
 The text frontend's canonical syntax is `lib/ibex/frontend/grammar.y`. Ibex generates and commits
 `lib/ibex/frontend/generated_parser.rb`; the public `Frontend::Parser` always delegates to that class. Lexer `Token` objects remain
 the semantic values passed through `TokenAdapter`, preserving their `Location` in AST nodes and diagnostics. The explicitly named
 handwritten `BootstrapParser` is excluded from normal loading and exists only to break the regeneration cycle. See
-[ADR 0015](decisions/0015-self-hosted-grammar-frontend.md) for the update procedure and boundary.
+[ADR 0003](decisions/0003-self-hosted-grammar-frontend.md) for the update procedure and boundary.
+`lib/ibex/frontend/shadow_grammar.y` describes the same frontend with parameterized list rules and an inline terminal wrapper.
+It is generated only in tests and must match the production parser's AST across the canonical grammar and extended fixtures;
+see the [development guide](development.md).
+
+The lexer also retains an immutable lexical CST without changing the semantic token stream. `Frontend::Parser#parse_document`
+returns a `SourceDocument` whose source, token-indexed segments, and AST come from that single lexer/parser pass.
+`SourceSpan` uses half-open zero-based byte offsets and one-based Unicode-scalar line/column positions. Actions and user-code
+bodies remain opaque segments; whitespace, line breaks, both comment forms, user-code markers, and EOF remain individually traversable.
+`render`, byte slicing, and byte/line/column conversion provide the common source contract for formatter, documentation, include,
+and language-server layers. `RuleDocumentation` correlates immediately preceding `##` comment-only lines with semantic rule
+locations and copy-enriches generated Root/Fragment nodes; occupied opaque-segment lines are never scanned as comments. See
+[ADR 0004](decisions/0004-shared-semantic-and-lossless-source-model.md).
+
+`Frontend::Formatter` classifies the document's existing semantic tokens, replaces only whitespace/newline trivia, and protects
+token, comment, action, heredoc, marker, and user-code bytes. It reparses the rendered root or fragment in the same frontend mode
+and compares ASTs with an explicit work stack after removing location fields. The CLI's stdout, batch check, and transactional
+in-place surfaces are therefore downstream consumers of `SourceDocument`, not an alternate grammar parser. Existing newline
+segment spellings and blank-line counts survive required line boundaries; new boundaries use the first newline even when it is
+inside opaque text. Batch stages and hard-link backups live beside each resolved target. Alias rejection, reverse rollback, and
+all-directory synchronization preserve full file modes and relative or absolute symlink identities. Backups are synchronized
+before installation; a failed restore preserves its backup, while post-commit cleanup problems are status-0 warnings. See
+[ADR 0004](decisions/0004-shared-semantic-and-lossless-source-model.md).
 
-The RBS generator emits the generated class namespace, superclass, parser-table constants, and `.parser_tables` contract. The
+`Frontend::SourceLoader` is the shared disk/overlay read boundary. Resolver's default loader retains canonical filesystem
+behavior; LSP injects open buffers, including safe new files, while the resolver continues to enforce realpath containment,
+symlink escape rejection, cycle identity, and diamond deduplication. `LSP::DocumentStore` layers monotonic open versions,
+root/include closures, reverse dependencies, and disk restoration over that loader. `PositionCodec` is the only UTF-16 adapter
+over frontend byte/scalar spans. `SymbolIndex` derives navigation and guarded rename edits from parsed nodes and lossless tokens,
+never from opaque Ruby or textual scanning. Content-Length transport, lifecycle handling, and request handlers remain separate
+from workspace semantics; see [ADR 0004](decisions/0004-shared-semantic-and-lossless-source-model.md).
+
+CLI file generation renders every requested output into an immutable `ArtifactSet` before entering `GenerationTransaction`.
+The transaction records the exact root, fragment, IR, and message bytes read through `GenerationInput`, rejects portable target
+collisions and input aliases, takes stable sidecar locks, stages and synchronizes every file, and can restore hard-link backups
+in reverse publication order. Ordinary companions publish first, the parser second, and an opt-in generation manifest last.
+That manifest is the coherence marker: readers verify its listed paths, sizes, and SHA-256 digests and retry from a newly read
+manifest on a mismatch. It is not a claim that several filesystem renames occur atomically. `--watch` feeds the same transaction
+only candidates whose complete canonical source closure and failed include attempts remain unchanged across rendering and
+publication. Portable polling, bounded debounce, failure deduplication, and cancellable nonblocking locks keep the last successful
+generation usable while a source is invalid; see [ADR 0013](decisions/0013-transactional-generation-publication.md).
+The executable's ordinary generation path declares this pipeline directly instead of loading the complete library and every
+subcommand. Optional subcommands and generation outputs load at their invocation boundary while their public constants remain
+autoloadable.
+
+Extended grammar paths cross an explicit `Frontend::Resolver` boundary. The canonical `import` declaration and compatible
+`include` spelling share this boundary. Roots retain class, start, options, and user code;
+fragments contain composable declarations and rules. Canonical realpaths define DFS order, diamond deduplication, cycle identity,
+and the Rake dependency closure. Canonical dirname ancestry keeps every resolved target below the root grammar directory after
+symlink resolution, including when that directory is a filesystem or drive root. The source-only Parser never follows an
+include. A `Resolution` recursively freezes its owned AST and defensive provenance copies while retaining rule identity for
+include-chain lookup. Rake resolves this closure at task definition and refuses invalid graphs before timestamp checks can reuse
+a stale output; see [ADR 0005](decisions/0005-contained-grammar-composition.md).
+
+Extended parameterized definitions remain structural AST templates rather than grammar symbols. The Normalizer validates the
+complete template graph, interns an impossible `$parameter_N` helper for each canonical call, records that helper in a memo
+before expanding its body, and substitutes formals through nested EBNF and calls. Resumable alternative, item, and EBNF
+continuations on an explicit depth-first worklist preserve ordinary lowering order and active-depth semantics without consuming
+the Ruby call stack. Same-argument recursion therefore closes over the memo, while a configurable total-specialization budget
+and structural constructor-growth detection bound argument-growing recursion without an arbitrary depth cutoff. Template actions,
+precedence, metadata, documentation, locations, and definition include chains flow into the specialized productions; see
+[ADR 0006](decisions/0006-bounded-structural-grammar-lowering.md).
+
+Inline definitions are lowered temporarily, then a bounded deterministic post-pass substitutes marked alternatives through
+ordinary, parameterized, and EBNF productions before diagnostics and LR construction. It removes every marked symbol and
+production, remaps the dense symbol/production ids, and retains eliminated semantic reductions as a versioned post-order
+action plan. The plan addresses flattened physical values followed by earlier logical results, records a nullable semantic
+`result_type` on every newly emitted step, reconstructs surrounding stack
+prefixes and semantic spans, and remains executable after IR serialization. Cycle validation covers paths through ordinary
+rules and templates; the default 10,000-production cartesian budget is configurable. See
+[ADR 0006](decisions/0006-bounded-structural-grammar-lowering.md).
+
+The strict generated frontend remains the grammar authority during batch diagnostics. A diagnostic parse retries it after
+suppressing only a whole declaration, whole rule, or outer alternative at balanced delimiters. Each retry must remove a new
+original token and both attempts and emitted diagnostics are bounded. Lexical and syntax phases collect independently before a
+global source-order limit is applied, so an earlier syntax error cannot be hidden by a later lexical error. Machine-readable
+diagnostics retain source spans, defensive locations, and stable codes. A repaired AST is marked partial by `ParseResult`, and
+is not attached to the unchanged source document. After a successful root parse, the CLI reports the first resolver grammar
+failure through the same diagnostic schema, while actual resolution I/O failures remain invocation errors. The CLI exposes this
+analysis only through `ibex diagnose`; see [ADR 0004](decisions/0004-shared-semantic-and-lossless-source-model.md).
+
+The RBS generator emits the generated class namespace, superclass, parser-table constants, `.parser_tables` contract, and
+private reduction and composed-fragment signatures. Declared symbol types refine the RHS tuple and LHS result independently,
+with `untyped` used at undeclared boundaries; composed inputs resolve either a physical symbol type or an earlier plan step's
+`result_type`. Reduction methods also receive a location tuple, surrounding location stack, and optional
+`Runtime::LocationSpan`. Generated tables mark whether actions require locations. The runtime leaves the location stack
+unallocated for ordinary two-element-token parses and creates or backfills it only when an action, tooling observer, or
+three-element token requires it. The public immutable `Ibex::Location` range and `loc`/`result_loc` action helpers sit above the
+same stack contract.
+Default source mapping compiles opaque action methods with `class_eval` when the generated class loads. The opt-in action-shadow
+generator makes those exact method bodies visible to Steep without runtime loading: runtime and shadow output share one method
+source builder, while the shadow omits parser infrastructure and every user-code section. Ibex only generates this source;
+executing Steep remains an application/CI boundary. See
+[ADR 0011](decisions/0011-versioned-semantic-action-boundary.md). The
 gem also ships a one-to-one rbs-inline-generated signature tree under `sig/` for every Ruby source in `lib/`, including the
 self-hosted parser. CI regenerates into an empty temporary directory, compares the complete trees, validates the RBS environment,
 and runs Steep against the entire library. Token/location records, grammar AST nodes, parser classifier and bootstrap state, the
@@ -30,7 +181,7 @@ Ruby DSL, IR records, and automaton actions use concrete domain types. Generated
 cells, decoded JSON values, and user methods embedded as opaque Ruby source remain `untyped`; applications can reopen the generated
 class in their own RBS files to declare embedded methods.
 
-## Grammar IR v1
+## Grammar IR versions 1 and 2
 
 Top-level fields:
 
@@ -38,30 +189,67 @@ Top-level fields:
 |---|---|
 | `ibex_ir`, `schema_version` | `"grammar"`, `1` |
 | `class_name`, `superclass` | Generated Ruby class contract |
-| `start`, `expect`, `options` | Start name, unresolved S/R expectation, result/action flags |
+| `start`, optional `starts`, `expect`, `options` | Primary/ordered start names, unresolved S/R expectation, result/action flags |
+| optional `params`, `printers` | Generated-constructor keywords and symbol-specific debug value formatters |
+| optional `lexer` | Embedded independently versioned Lexer IR v1 |
 | `symbols` | Interned terminals and nonterminals; `$eof` id 0 and `error` id 1 |
 | `productions` | Numeric LHS/RHS ids, action, precedence override, source origin |
 | `user_code`, `conversions`, `warnings` | Concatenated code, external token expressions, structured diagnostics |
 | `user_code_chunks` | Optional opaque chunks with first-code-line locations for compatible source mapping |
 
-Warning records use stable type names (`undeclared_terminal`, `unused_terminal`, `unreachable_nonterminal`,
-`duplicate_production`, and `empty_language`) and retain source locations. The CLI applies display/error policy at the boundary;
+Warning records use the additive type vocabulary `undeclared_terminal`, `unused_terminal`, `unused_precedence`,
+`unreachable_terminal`, `unreachable_nonterminal`, `duplicate_production`, and `empty_language`, and retain source locations.
+Schema-v1 readers must ignore warning types they do not recognize. The CLI applies display/error policy at the boundary;
 normalization and IR serialization do not discard diagnostics.
 
-A symbol has `id`, `name`, `kind`, `reserved`, optional `prec {associativity, level}`, and `loc`. A production has `id`, `lhs`,
-`rhs`, optional `action`, optional `prec_override`, and `origin`. Synthetic EBNF origins include an additive, deterministic
-`expression` label used by text, DOT, and HTML presentation while numeric symbol identities remain unchanged. An action has opaque
-`code`, `loc`, `named_refs [{name,index}]`, and `context_length`; middle-action helpers use the last field to view preceding stack
-values.
+A symbol has `id`, `name`, `kind`, `reserved`, optional `prec {associativity, level}`, `loc`, `display_name`, and
+`semantic_type`. The last two fields are omitted when undeclared, so older schema-v1 documents remain byte-stable and loadable.
+A production has `id`, `lhs`, `rhs`, optional `action`, optional `prec_override`, and `origin`. Synthetic EBNF origins include
+an additive, deterministic `expression` label used by text, DOT, and HTML presentation while numeric symbol identities remain
+unchanged. An action has opaque `code`, `loc`, `named_refs [{name,index}]`, and `context_length`; middle-action helpers use the
+last field to view preceding stack values.
+
+IR objects and nested collections are frozen. JSON keys have deterministic order, so dump/load/dump is byte-stable. The additive
+`user_code_chunks` field remains optional in version 1 so older JSON stays loadable.
+
+New normalized grammars use version 2. It keeps every version-1 semantic field and adds explicit nullable metadata:
+
+| Record | Version-2 metadata |
+|---|---|
+| grammar | `source_provenance {file, root, byte_span {start,end}}` and optional `migration` loss record |
+| symbol | `doc` |
+| production | `doc` and `expansion {parameter, inline, include_chain}` |
+| action | `composition {strategy, fragments, plan {version, physical, steps}}`; new steps include nullable `result_type` |
+
+The source-only text frontend supplies the source filename and leaves unknown metadata null. A resolved grammar also supplies its
+canonical source root and each production's include chain while preserving its original-file origin. Lossless rule comments
+populate symbol and user-production documentation, including through fragment resolution; synthetic EBNF helpers remain
+undocumented. Parameterized specializations populate `expansion.parameter` with the template name and canonical structural
+arguments while retaining the definition's include chain. Version-1 upgrades mark every unrecoverable metadata family in
+`migration.unavailable` instead of guessing.
+For compatibility with earlier version-2 documents, the input schema also accepts an absent composition-step
+`result_type`; generators treat it as `untyped`.
+`Serialize.load` and `Validator.validate` accept both versions; a loaded version-1 object dumps with the original version-1
+shape. `IR::Migration.to_version` upgrades version 1 to 2 and is idempotent at version 2. The CLI exposes this as
+`ibex migrate-ir INPUT --to=2 [-o FILE]`; file output uses an atomic same-directory rename and refuses to alias the input.
+
+The v1 stabilization freeze covers required core fields, meanings, ordering,
+identity, and validation behavior. The future `x-` experimental namespace is
+outside that freeze, but current schemas remain closed and reject unknown
+fields. Experimental data must therefore begin in a new additive schema
+version rather than weakening an existing document. See
+[stability and deprecation](stability.md).
 
-IR objects and nested collections are frozen. JSON keys have deterministic order, so dump/load/dump is byte-stable. Incompatible
-schema changes require a new version. The additive `user_code_chunks` field is optional so older schema-v1 JSON remains loadable;
-new text-front-end IR retains it to make direct and resumed `--line-convert-all` generation identical.
+`Codegen::Documentation` renders normalized user rules and alternatives as escaped Markdown, self-contained accessible HTML, or
+railroad SVG. The railroad renderer includes visible wrapped rule descriptions in its section-height calculation and exposes the
+full escaped text through SVG descriptions. `ibex doc` resolves the same canonical include graph and writes to stdout or an
+atomic file without generating or executing application parser code.
 
-## Automaton IR v1
+## Automaton IR versions 1 and 2
 
-Top-level fields are `ibex_ir: "automaton"`, `schema_version`, `algorithm`, `grammar_digest`, embedded `grammar`, `states`, and
-`conflict_summary`. Embedding Grammar IR makes automaton JSON sufficient for code generation after `--from=automaton-ir`.
+Top-level fields are `ibex_ir: "automaton"`, `schema_version`, `algorithm`, `grammar_digest`, embedded `grammar`, `states`,
+optional `entry_states`, and `conflict_summary`. Embedding Grammar IR makes automaton JSON sufficient for code generation after
+`--from=automaton-ir`. A multi-entry automaton maps every ordered grammar start name to its initial state.
 
 Each state contains:
 
@@ -70,30 +258,136 @@ Each state contains:
 - resolved terminal `actions` and nonterminal `gotos`;
 - an optional reduce `default_action`, selected only when explicit error masks preserve every terminal lookup and reduce the
   total encoded ACTION entries;
-- every conflict, including precedence-resolved conflicts and the resolution reason.
+- every conflict, including precedence-resolved conflicts and the resolution reason; multi-entry conflicts also carry their
+  reachable `entries` and optional `composite` marker.
 
 `conflict_summary.sr` counts unresolved default-shift conflicts for `expect`; `resolved_sr` counts retained precedence or
 associativity decisions; `rr` counts reduce/reduce cells.
 
+New automata use version 2 and always embed Grammar IR version 2. Migration recalculates `grammar_digest` from the upgraded
+canonical grammar. Version-1 automata remain loadable, validatable, and byte-stable. Published Draft 2020-12 contracts for both
+versions live under `schema/`; see
+[ADR 0001](decisions/0001-versioned-ir-pipeline.md).
+
+`--emit=sets` is a deterministic analysis view rather than another IR: it emits lexically sorted nullable nonterminals and
+FIRST/FOLLOW maps for nonterminals. DOT, Mermaid, and the self-contained searchable HTML report are deterministic presentation
+views over Automaton IR.
+
+## Construction algorithms and counterexamples
+
+The `lalr` and `slr` strategies construct LR(0) states directly for a single entry. LALR lookaheads are the least fixed point of deterministic
+shift, spontaneous-FIRST, and nullable-suffix propagation edges over item occurrences; SLR replaces completed lookaheads with
+FOLLOW sets. Multiple entries seed distinct augmented canonical items and use canonical core merging because the direct
+lookahead graph has a single-root contract. Canonical `lr1` retains the canonical collection. An explicit canonical-and-merge
+LALR reference strategy proves byte equivalence without changing the Automaton IR algorithm label. `ielr` conservatively merges action-compatible canonical
+states and refines partitions until outgoing transitions are congruent, avoiding LALR inadequacies without promising a minimum
+state count. `--entry-isolation` instead constructs each start independently and concatenates the resulting state sets with
+deterministic offsets. Shared builds attribute reachable entries to conflicts and compare isolated conflict fingerprints to
+identify merge-created composite conflicts. All strategies use the same conflict resolver and default reduction pass. After a build, frozen diagnostic
+`metrics` record the strategy and construction/final state counts, plus a canonical count only when one was actually built. See
+[ADR 0007](decisions/0007-shared-parser-construction-pipeline.md).
+
+`Ibex::LALR::Counterexample` consumes only Automaton IR. For each conflict it explores parser-stack configurations, forces the
+competing actions, and searches for a common accepting suffix. A successful result contains both complete derivation trees and is
+marked `unifying: true`. Search defaults to 32 tokens and 50,000 configurations; the Ruby and CLI APIs can set both positive
+budgets. If no common sentence is found within them, the result is explicitly marked nonunifying and contains the deterministic
+shortest reachability witness instead of claiming ambiguity.
+
+`Ibex::Codegen::Explain` filters those immutable conflicts by state and canonical token identity before asking
+`Counterexample#for_conflict` to search only the selected entries, then renders text or the versioned `explain` JSON analysis
+shape. `Counterexample#all` retains its original all-conflict behavior. The view performs no additional parser analysis and does
+not extend Grammar or Automaton IR.
+
+The repository's self-authored representative grammar feeds the current versioned `ibex_benchmark` v2 document. Its JSON Schema
+is shipped beside the IR schemas, while committed environment-specific observations live under the matching
+`benchmark/results/vN` directory. Timing and peak RSS remain non-gating; CI reproduces only deterministic structure and digests.
+See the [benchmark guide](../benchmark/README.md).
+
 ## Runtime table contract
 
 Generated subclasses expose `.parser_tables` with a required `format_version`, external `tokens`, display `token_names`, ACTION
 and GOTO tables, per-state default actions, and production `{lhs,length,action}` records. The runtime validates the version before
-reading input and rejects missing or unsupported formats with a regeneration instruction; see
-[ADR 0018](decisions/0018-parser-table-format-version.md). Plain tables are arrays of Hash rows. Compact tables use row
+reading input and rejects missing or unsupported formats with a regeneration instruction. The generator emits v6. For non-CST
+tables, the runtime accepts v1's two-argument actions, v2/v3 explicitly marked five-argument location actions, v3 explicitly
+marked six-argument composed actions, v4 one-Array values actions, and v5/v6 positional actions. Marker contracts are validated
+before input. A CST table is executable only when it uses current format-v6 structured metadata; older or boolean CST shapes
+must be regenerated. See
+[ADR 0008](decisions/0008-versioned-runtime-package-boundary.md). Plain tables are arrays of Hash rows. Compact tables use row
 displacement with offsets, values, and row-ownership checks; both expose equivalent lookups. Default reductions are restricted
 to known token ids, and explicit error masks preserve the pre-optimization result of every declared terminal cell, including
-the synthetic `error` terminal. The deterministic size policy is fixed by
-[ADR 0014](decisions/0014-compatibility-safe-default-reductions.md).
+the synthetic `error` terminal. Extended parser tables opt `expected_tokens` into lookahead correction: the runtime copies only
+the state stack and simulates reductions and gotos for each declared terminal, without evaluating semantic actions. The explicit
+`expected_tokens_exact` API exposes the same result for compatible tables. The deterministic size policy is fixed by
+[ADR 0008](decisions/0008-versioned-runtime-package-boundary.md).
+
+Runtime execution is packaged independently as `ibex-runtime`, with its own version and RBS tree. The generator package depends
+on a compatible runtime series but does not duplicate runtime-owned files. Compact lookup values live in a runtime-safe leaf
+file, while table construction remains generator-only. Normal output requires only `ibex/runtime`; `-E` embeds the same sources.
+See [ADR 0008](decisions/0008-versioned-runtime-package-boundary.md).
 
 The runtime maintains state and value stacks, pulls a lookahead only when required, and applies tagged `shift`, `reduce`,
 `accept`, and `error` actions. Recovery pops to a state that shifts token id 1, suppresses repeated reports for three successful
-shifts, and honors `yyerrok`. No-op `on_shift`, `on_reduce`, and `on_error_recover` extension points observe successfully
+shifts, and honors `yyerrok`. No-op shift, reduce, recovery, location-aware, and discard extension points observe successfully
 committed events without changing parser results; the recovery hook retains the pre-pop error context and is distinct from an
-ordinary token shift. Their ordering and payload contract is fixed by [ADR 0013](decisions/0013-runtime-observation-hooks.md).
+ordinary token shift. A configured value printer affects only human `yydebug` output. Their ordering and payload contract is
+extended additively by
+[ADR 0010](decisions/0010-committed-runtime-observation.md). Grammar-declared
+symbol printers are optional IR v2 metadata compiled into private methods and
+an id-indexed table.
+
+Ordinary generated tables are recursively frozen and made Ractor-shareable. Threads and Ractors share those tables but parse
+through distinct instances; stacks, lookahead, lexer state, callbacks, observers, and semantic values are session-owned. A
+single instance rejects overlapping drivers. Immutable `Runtime::ResourceLimits` values bound every stack push and recovery
+entry with finite defaults. Exhaustion raises the structured `ResourceLimitError`; see
+[ADR 0009](decisions/0009-isolated-parser-sessions.md).
+
+Extended Grammar IR v2 may additionally carry synchronization terminals and ordered `%on_error_reduce` groups. Table
+construction fills only otherwise erroneous ACTION cells with a unique highest-priority completed declared production. At
+runtime, an explicit shift of the synthetic `error` token always wins; only when it is unavailable does panic recovery discard
+through a configured synchronization token and pop to a state that accepts that retained lookahead. Generated parsers without
+sync declarations omit the optional table field. The pull/push ordering and observer contract are fixed by
+the runtime and grammar reference documentation.
+
+The separate `Runtime::Parser#observe` API publishes ordered, immutable schema-v1 events for tooling. Its bounded sanitizer
+copies only JSON data and never retains application identities or private stacks. With no observer, parse transitions construct
+no Event, payload summary, or dispatch snapshot; parser initialization still creates its ownership mutex. Generated tables
+contribute grammar digest, table format, state count, and production count to the `start` event. `Runtime::EventJSONLTracer`
+exposes the versioned stream; the original hook-based `Runtime::JSONLTracer` remains byte-compatible. The protocol and
+exception/threading behavior are fixed by
+[ADR 0010](decisions/0010-committed-runtime-observation.md).
+
+Optional `Runtime::RepairPolicy` drives a bounded Dijkstra search over copied state stacks and buffered token identities. Search
+uses explicit/default ACTION, GOTO, and production shape only; it never executes semantic code. A selected immutable edit plan is
+reported once, then replayed through the ordinary runtime so actions and hooks remain committed-path effects. Pull lookahead,
+push buffering, deterministic tie-breaking, fallback to yacc recovery, and no-policy compatibility are fixed by
+[ADR 0012](decisions/0012-bounded-nonexecuting-analysis.md).
+
+`Coverage::Collector` accepts only contiguous, complete runtime-event sessions with generated parser metadata. It counts entries
+to the initial, shift, reduce-goto, and recovery states and counts committed reductions by production id. `Coverage::Report`
+publishes ascending sparse hit arrays under the versioned runtime-coverage schema. Merge requires identical full grammar digest,
+table format, and totals and uses checked addition. The coverage CLI only reads bounded JSON/JSON Lines and never loads generated
+Ruby or executes semantic actions; collection, merge, threshold, and atomic-output policy are fixed by
+[ADR 0010](decisions/0010-committed-runtime-observation.md).
+
+`TableSimulation::Simulator` is a separate state-stack interpreter over validated Automaton IR. It resolves an explicit ACTION
+cell before a default action, so explicit error masks remain authoritative, and never evaluates the opaque semantic-action
+source stored in Grammar IR. Immutable steps expose state, lookahead, selected action source, reduction/goto metadata, and stack
+depth. Action and stack budgets bound default/epsilon cycles and growth. The text/JSON CLI and versioned output contract are
+fixed by [ADR 0012](decisions/0012-bounded-nonexecuting-analysis.md).
+
+Grammar IR v2 may carry ordered accept/reject source tests without adding them to parser tables. `GrammarTests::Runner` generates
+one embedded parser and executes fresh parser instances in a separate Ruby process, distinguishing `ParseError` rejection from
+lexer/application errors and bounding the whole suite by a timeout. The separate runner loads the generated file, so guarded
+footer programs stay inactive. The source contract, isolation boundary, and CI behavior are fixed by
+the grammar reference documentation.
 
 ## Clean-room boundary
 
 Implementation work uses public racc documentation, CLI black-box behavior, and published LR algorithms only. racc implementation
 sources and generated source are not inputs to the design. Self-authored compatibility grammars execute both outputs in separate
 processes and compare observable results.
+
+`RaccMigration::Checker` treats grammar code as opaque while reporting default-mode parse/normalization errors and known runtime
+coupling. The separate harness generator emits source only; its output refuses an empty corpus and makes bounded child-process
+execution explicit. The boundary is fixed by
+[ADR 0012](decisions/0012-bounded-nonexecuting-analysis.md).

data/docs/cst-migration.md

@@ -0,0 +1,107 @@
+# CST Red/Green migration
+
+The runtime supports only the current format-v6 Red/Green representation for
+`pragma cst`. CST parser tables from formats v1 through v5, and boolean
+`cst: true` tables without structured metadata, fail before the first token is
+read and instruct the application to regenerate. Older non-CST parser tables
+remain executable.
+
+Regenerate with the same command used for the grammar, for example:
+
+```sh
+ibex grammar.y -o parser.rb
+```
+
+Upgrade the generator and runtime together, regenerate the parser, and update
+consumers to the current APIs below. The removed `CST::Trivia`, `CST::Token`,
+`CST::Missing`, `CST::Error`, and `CST::Node` constants are not defined.
+
+## Parsing
+
+Use `parse_with_syntax` when both the semantic result and syntax are needed:
+
+```ruby
+result = parser.parse_with_syntax(source, file: "input.txt")
+value = result.value
+root = result.syntax_root
+diagnostics = result.diagnostics
+```
+
+`parse`, `do_parse`, and `yyparse` keep returning the semantic result. After a
+CST parse, `syntax_root` exposes the most recently built Red root.
+
+## Changed tree shape
+
+The new root is always `source_file`:
+
+```text
+source_file
+├── selected start-symbol node
+└── $eof
+```
+
+The EOF token owns final trivia. Code that previously treated the returned CST
+as the start node should use `result.syntax_root.children.fetch(0)`.
+
+The legacy implementation exposed semantic values inside an actionless
+parent. For example, an action-bearing `term` could appear as a `CST::Token`
+whose symbol was `"term"` and whose value was the action result. Format v6
+always exposes the physical `term` syntax node in that position. Read the
+semantic result from `result.value`; syntax children never contain semantic
+values.
+
+These are the primary incompatible shapes found by the migration
+characterization suite. `deconstruct` continues to return syntax children and
+`deconstruct_keys` retains the compatibility keys. Typed field keys are added
+from `@node` slot metadata.
+
+## Removed public API mapping
+
+The inventory below records the removed values and their format-v6
+replacements.
+
+| Removed value and members | Previous use | Format-v6 replacement |
+| --- | --- | --- |
+| `CST::Trivia#text`, `#location`, `#to_h` | leading/trailing trivia assertions | `GreenTrivia#text` and `#kind`; obtain absolute location from the owning Red token |
+| `CST::Token#symbol`, `#value`, `#location`, `#leading_trivia` | terminal names, lexer action values, positions, skipped text | `SyntaxToken#kind_name`/`#symbol`, `#location`, and `#leading_trivia`; compatibility `#value` returns the same source bytes as `#text`, while semantic lexer values stay in the normal action/value path |
+| `CST::Token#kind`, `#children`, `#deconstruct`, `#deconstruct_keys`, `#to_h` | error classification and pattern matching | integer `#kind`, `#error?`/`#missing?`, empty `#children`/`#deconstruct`, and compatibility pattern/hash keys |
+| `CST::Missing` | bounded-repair insertion checks | `SyntaxToken#missing?`, `GreenToken#expected_kind`, and `CONTAINS_MISSING` |
+| `CST::Error#reason` | lexical, syntax, delete, and discard checks | `SyntaxToken#error?`, `SyntaxNode#error?`, diagnostics, and Green error/skipped flags |
+| `CST::Node#symbol`, `#production_id`, `#children`, `#location`, `#trailing_trivia` | root/reduction shape, source position, final trivia | `SyntaxNode#symbol`, `#children`, `#location`, and compatibility `#trailing_trivia`; `#production_id` is the `-1` sentinel because kinds/slot metadata replace occurrence-local production identity |
+| `CST::Node#each`, `#deconstruct`, `#deconstruct_keys`, `#to_h`, `#with_trailing_trivia` | enumeration, pattern matching, final-trivia attachment | Red `Enumerable`/pattern/hash methods; persistent trivia changes use the owning token's `with_leading`/`with_trailing` |
+
+Current pure-syntax shape and compatibility accessors are executable in
+`test/codegen/cst_characterization_test.rb`. Rejection of obsolete CST table
+shapes is covered by `test/runtime/table_format_test.rb` and
+`test/codegen/cst_runtime_integration_test.rb`. The semantic-value removal is
+the C1 change; the `source_file` root is C2.
+
+## Trivia
+
+`--cst-trivia=attach` remains accepted and means `leading`.
+
+- `leading`: skipped lexer text belongs to the next token.
+- `balanced`: text through the first newline belongs to the preceding token,
+  with the remainder leading the next token.
+- `drop`: trivia is omitted; source-coordinate and incremental APIs raise
+  because offsets no longer correspond to the original source.
+
+Use `to_source` for a binary-exact reconstruction. On early `yyaccept`,
+`incomplete_input?` is true and `to_source` is the consumed prefix.
+
+## Errors and repair
+
+Lexical failures, yacc recovery, panic discards, and bounded repair return a
+syntax result rather than changing the semantic action contract. Inspect
+`diagnostics`, `contains_error?`, `each_error`, missing tokens, and Green flags
+for skipped input. Unrecoverable input is retained under `synthetic_root`.
+
+## New capabilities
+
+Format v6 provides typed `Parser::Syntax` views, persistent
+path-copy editing, annotations, structural text diffing, `ibex_cst` v1
+serialization, and syntax-only incremental sessions. These APIs have no
+equivalent on the removed mixed semantic/syntax tree. Batch CST, views,
+editing, and serialization are Stable; incremental sessions remain
+Experimental. See the [CST guide](cst.md) for examples and the incremental
+action contract.