ibex 0.2.0 → 0.3.0

data/docs/architecture.md

@@ -1,6 +1,11 @@
+---
+title: Architecture
+description: Ibex's grammar, IR, parser construction, runtime, and verification boundaries.
+---
+
 # Architecture and IR schemas
 
-Ibex keeps syntax, grammar meaning, automaton construction, and output concerns behind two versioned immutable contracts.
+Ibex keeps syntax, grammar meaning, automaton construction, and output concerns behind two immutable current contracts.
 
 ```text
 .y root/fragments -> Frontend Lexer/CST -> self-hosted LR Parser -> canonical Resolver ─┐
@@ -8,7 +13,7 @@ Ruby DSL ───────────────────────
                                                                                |              |
                                                                                |          Lexer IR v1
                                                     |
-                                               set analysis
+                                                    set analysis
                                                     |
                                       SLR/LALR/IELR/LR1 Builder -> Automaton IR
                                                                     |
@@ -18,8 +23,56 @@ Ruby DSL ───────────────────────
 Frontend changes stop at the Normalizer. Algorithm strategies consume Grammar IR and produce identical Automaton IR shapes.
 Outputs consume Automaton IR and never call builder internals. The CLI only connects stages and supports JSON resumption.
 
-An optional root-only `lexer` declaration normalizes to independently versioned
-Lexer IR v1 and is embedded unchanged in Grammar IR v2. Its flat rule list
+IELR has two construction strategies. The conservative `partition` strategy
+builds canonical LR(1) and partitions compatible isocores. The opt-in `direct`
+strategy works from LR(0), goto-follow closures, lane annotations, and targeted
+state splitting before rerunning lookahead propagation. Both use the shared
+conflict resolver and downstream Automaton IR pipeline. See the [IELR guide](ielr.md).
+
+## Execution trust boundary
+
+`syntax-only` means that the LR runtime suppresses parser production actions.
+It does not mean that no application Ruby executes.
+
+| Execution path | Parser production actions | Generated lexer actions | User `header` / `inner` / `footer` | Trust |
+| --- | --- | --- | --- | --- |
+| Grammar parse/normalize, format, LSP, reports, conflict/diff/equiv/verify/debug, internal fuzz | No | No | No | Nonexecuting without an external-command option |
+| Generated-lexer semantic: `parse`, `lex(...).do_parse`, `parse_with_syntax(source)` | Yes | Yes | May execute when the generated file loads | Trusted application code; not a sandbox |
+| Handwritten pull semantic: `do_parse`, no-argument `parse_with_syntax` | Yes | No | May execute when the generated file loads | Trusted application code; not a sandbox |
+| Caller-fed semantic: `yyparse`, `push` / `finish` | Yes | No | May execute when the generated file loads | Trusted application code; not a sandbox |
+| Generated syntax-only parse | No | Yes | May execute when the generated file loads | Trusted application code; not a sandbox |
+| Future safe syntax profile | No | Declarative built-ins only | No | Nonexecuting profile; not currently available |
+
+Static paths operate on source, Grammar IR, Lexer IR, Automaton IR, or parser
+tables without requiring the generated application parser. Parser actions,
+lexer actions, conversions, and user sections remain opaque data on those
+paths. Code generation may emit source that compiles them when the artifact is
+loaded, but the generator does not load that artifact.
+
+The static guarantee excludes explicit external-command options.
+`ibex fuzz --against=COMMAND` and `ibex reduce --command=COMMAND` spawn the
+supplied executable and may run arbitrary application code with host permissions.
+Subprocess resource limits and process-group cleanup are not sandboxes and do
+not confine side effects.
+
+Semantic runtime entry points execute committed parser production actions.
+Generated lexer actions execute only when tokens are pulled through
+`GeneratedLexer#next_token`; handwritten `next_token`, `yyparse`, and
+`push` / `finish` paths do not invoke that lexer. Syntax-only entry points
+suppress parser actions but still use the generated lexer to emit tokens,
+convert values, and mutate lexer state. Every generated runtime path crosses
+the trusted application boundary because loading the generated Ruby file may
+execute arbitrary user sections. Resource budgets and process-isolation
+building blocks do not make those paths sandboxes.
+
+A future profile for untrusted syntax work must use data-only parser tables, a
+declarative lexer whose operations are restricted to built-ins, no parser
+actions, no arbitrary conversions, and no `header`, `inner`, or `footer`
+sections. The current generated parser API must not be described as that
+profile.
+
+An optional root-only `lexer` declaration normalizes to the independently versioned
+Lexer IR and its flat rule list, which are embedded unchanged in the current Grammar IR.
 records state, declaration id, pattern source/options, action, and provenance;
 `--emit=lexer-ir` exposes the same document. Code generation compiles every
 pattern with an internal current-position anchor and emits immutable,
@@ -27,7 +80,7 @@ state-indexed rules. Per-parser mutable input, position, emission, and state
 stacks live in `Runtime::GeneratedLexer`, never in the tables. See
 [ADR 0014](decisions/0014-versioned-generated-lexer.md).
 
-`pragma cst` remains an optional Grammar IR v2 flag. Regenerated format-v6
+`pragma cst` remains an optional current Grammar IR flag. Regenerated format-v6
 tables add deterministic kind and normalized slot metadata. `Runtime::Parser`
 builds a pure-syntax Green entry for every shift and reduction on a stack
 parallel to the semantic stack, so tree shape no longer depends on semantic
@@ -61,8 +114,9 @@ Validation reconstructs every derived width, flag, and descendant count.
 Non-UTF-8 bytes use canonical Base64. See
 [ADR 0017](decisions/0017-persistent-syntax-artifacts.md).
 
-Incremental sessions are syntax-only: parser production actions do not run.
-The generated lexer first validates token/state resynchronization. `Blender`
+Incremental sessions are syntax-only: parser production actions do not run,
+but generated lexer actions do. The generated lexer first validates token/state
+resynchronization. `Blender`
 then offers either a fresh token or an old Green nonterminal to the LR driver.
 A subtree is pushed directly through `goto` only when damage, recorded
 left-state, follow-token identity, error flags, and positive width satisfy the
@@ -70,7 +124,18 @@ conservative reuse proof. Token and parse memos remain preorder/occurrence
 state owned by one session; resource exhaustion falls back to the fresh token
 stream. See [ADR 0018](decisions/0018-conservative-incremental-syntax-reuse.md).
 
-Alternative-level `@node` declarations are preserved as Grammar IR v2
+`Runtime::SyntaxSession` is a thin generated-language service boundary over
+that same engine. Current generated classes expose only
+`:trusted_application_code`, and callers must explicitly acknowledge that
+profile because lexer actions still execute. Immutable operation snapshots add
+expected-token and reuse/fallback evidence plus cooperative cancellation and
+service bounds. The façade belongs to `ibex-runtime`; it defines neither LSP
+nor workspace semantics. Its additive `repair` operation projects bounded
+syntax repairs into byte edits and fresh syntax results without exposing
+semantic values or invoking application repair callbacks. See
+[ADR 0019](decisions/0019-runtime-syntax-session-boundary.md).
+
+Alternative-level `@node` declarations are preserved in the current Grammar IR
 production metadata. Runtime Ruby, static action-shadow Ruby, and generated
 RBS all derive Data node classes and Visitor/Listener hooks from that same
 metadata; action source is never inspected to infer a shape. Symbol semantic
@@ -125,6 +190,20 @@ The executable's ordinary generation path declares this pipeline directly instea
 subcommand. Optional subcommands and generation outputs load at their invocation boundary while their public constants remain
 autoloadable.
 
+### Effective configuration boundary
+
+`Ibex::Configuration` gives generation, grammar-test, and analysis paths one closed definition for parser construction and
+build-policy concepts. Each immutable value records its canonical key, typed domain, owner, override policy, selected origin,
+explicitness, and whether the selection is canonical. `Configuration::CLIAdapter` translates the existing internal option
+hash into domain values such as `parser.entries=isolated`, `cst.trivia=leading`, and
+`source.line_mapping=all`; code generators do not interpret legacy CLI shapes directly.
+
+`Configuration::Resolver` applies fixed and monotone-minimum algebra without loading a grammar file or executing user code.
+A fixed grammar declaration accepts a matching request and raises `Configuration::Conflict` for a contradiction. An explicit
+analysis override remains noncanonical and serializes both the declared and selected values. Evidence JSON is emitted in
+canonical-key order. This boundary intentionally adds neither grammar syntax nor a project configuration file; those adapters
+must supply the same closed keys when their separately reviewed formats exist.
+
 Extended grammar paths cross an explicit `Frontend::Resolver` boundary. The canonical `import` declaration and compatible
 `include` spelling share this boundary. Roots retain class, start, options, and user code;
 fragments contain composable declarations and rules. Canonical realpaths define DFS order, diamond deduplication, cycle identity,
@@ -145,7 +224,7 @@ precedence, metadata, documentation, locations, and definition include chains fl
 
 Inline definitions are lowered temporarily, then a bounded deterministic post-pass substitutes marked alternatives through
 ordinary, parameterized, and EBNF productions before diagnostics and LR construction. It removes every marked symbol and
-production, remaps the dense symbol/production ids, and retains eliminated semantic reductions as a versioned post-order
+production, remaps the dense symbol/production ids, and retains eliminated semantic reductions as a stable post-order
 action plan. The plan addresses flattened physical values followed by earlier logical results, records a nullable semantic
 `result_type` on every newly emitted step, reconstructs surrounding stack
 prefixes and semantic spans, and remains executable after IR serialization. Cycle validation covers paths through ordinary
@@ -181,17 +260,19 @@ Ruby DSL, IR records, and automaton actions use concrete domain types. Generated
 cells, decoded JSON values, and user methods embedded as opaque Ruby source remain `untyped`; applications can reopen the generated
 class in their own RBS files to declare embedded methods.
 
-## Grammar IR versions 1 and 2
+## Current Grammar IR
+
+<!-- stable:current-ir:v1 -->
 
 Top-level fields:
 
 | Field | Meaning |
 |---|---|
-| `ibex_ir`, `schema_version` | `"grammar"`, `1` |
+| `ibex_ir`, `schema_version` | `"grammar"`, current format |
 | `class_name`, `superclass` | Generated Ruby class contract |
 | `start`, optional `starts`, `expect`, `options` | Primary/ordered start names, unresolved S/R expectation, result/action flags |
 | optional `params`, `printers` | Generated-constructor keywords and symbol-specific debug value formatters |
-| optional `lexer` | Embedded independently versioned Lexer IR v1 |
+| optional `lexer` | Embedded independently versioned Lexer IR |
 | `symbols` | Interned terminals and nonterminals; `$eof` id 0 and `error` id 1 |
 | `productions` | Numeric LHS/RHS ids, action, precedence override, source origin |
 | `user_code`, `conversions`, `warnings` | Concatenated code, external token expressions, structured diagnostics |
@@ -199,24 +280,23 @@ Top-level fields:
 
 Warning records use the additive type vocabulary `undeclared_terminal`, `unused_terminal`, `unused_precedence`,
 `unreachable_terminal`, `unreachable_nonterminal`, `duplicate_production`, and `empty_language`, and retain source locations.
-Schema-v1 readers must ignore warning types they do not recognize. The CLI applies display/error policy at the boundary;
+The CLI applies display/error policy at the boundary;
 normalization and IR serialization do not discard diagnostics.
 
 A symbol has `id`, `name`, `kind`, `reserved`, optional `prec {associativity, level}`, `loc`, `display_name`, and
-`semantic_type`. The last two fields are omitted when undeclared, so older schema-v1 documents remain byte-stable and loadable.
+`semantic_type`.
 A production has `id`, `lhs`, `rhs`, optional `action`, optional `prec_override`, and `origin`. Synthetic EBNF origins include
 an additive, deterministic `expression` label used by text, DOT, and HTML presentation while numeric symbol identities remain
 unchanged. An action has opaque `code`, `loc`, `named_refs [{name,index}]`, and `context_length`; middle-action helpers use the
 last field to view preceding stack values.
 
-IR objects and nested collections are frozen. JSON keys have deterministic order, so dump/load/dump is byte-stable. The additive
-`user_code_chunks` field remains optional in version 1 so older JSON stays loadable.
+IR objects and nested collections are frozen. JSON keys have deterministic order, so dump/load/dump is byte-stable.
 
-New normalized grammars use version 2. It keeps every version-1 semantic field and adds explicit nullable metadata:
+Normalized grammars use the current format for declaration-free source and carry explicit nullable metadata:
 
-| Record | Version-2 metadata |
+| Record | Current-format metadata |
 |---|---|
-| grammar | `source_provenance {file, root, byte_span {start,end}}` and optional `migration` loss record |
+| grammar | `source_provenance {file, root, byte_span {start,end}}` |
 | symbol | `doc` |
 | production | `doc` and `expansion {parameter, inline, include_chain}` |
 | action | `composition {strategy, fragments, plan {version, physical, steps}}`; new steps include nullable `result_type` |
@@ -225,13 +305,28 @@ The source-only text frontend supplies the source filename and leaves unknown me
 canonical source root and each production's include chain while preserving its original-file origin. Lossless rule comments
 populate symbol and user-production documentation, including through fragment resolution; synthetic EBNF helpers remain
 undocumented. Parameterized specializations populate `expansion.parameter` with the template name and canonical structural
-arguments while retaining the definition's include chain. Version-1 upgrades mark every unrecoverable metadata family in
-`migration.unavailable` instead of guessing.
-For compatibility with earlier version-2 documents, the input schema also accepts an absent composition-step
-`result_type`; generators treat it as `untyped`.
-`Serialize.load` and `Validator.validate` accept both versions; a loaded version-1 object dumps with the original version-1
-shape. `IR::Migration.to_version` upgrades version 1 to 2 and is idempotent at version 2. The CLI exposes this as
-`ibex migrate-ir INPUT --to=2 [-o FILE]`; file output uses an atomic same-directory rename and refuses to alias the input.
+arguments while retaining the definition's include chain.
+The current format adds one closed, root-owned `parser_contract`. Its `algorithm`, `entries`, and `cst_trivia` members each carry
+`value`, `explicit`, and `loc`. An explicit member has an admitted value and source location; an unspecified member has all
+three states fixed to `value: null`, `explicit: false`, and `loc: null`. Unspecified is not an encoding of the current built-in
+default. The contract sits only at the root, so fragments cannot change parser-wide construction while their normalized
+productions and source provenance remain composable.
+
+Extended root grammars can populate `algorithm` and `entries` through one
+domain-specific `parser ... end` declaration. The frontend rejects that block
+in fragments and compatible mode, and rejects duplicate blocks, duplicate or
+unknown keys, unknown values, and `algorithm auto` before normalization. The
+typed configuration resolver treats those values as fixed grammar contracts:
+canonical generation accepts only an absent or matching CLI request. Analysis
+and grammar-test commands may move an explicitly different algorithm into a
+reported noncanonical analysis origin; entry construction remains fixed. The
+configuration inspector reads the same AST and import closure without running
+actions or user sections.
+
+`Serialize.load` and `Validator.validate` accept only the current Grammar IR format. Both declaration-free and
+`parser`-declared sources write that same format, so there is no in-process migration path or compatibility reader.
+Callers construct the current object directly through `IR::Grammar.new`; see
+[ADR 0020](decisions/0020-grammar-ir-parser-contract.md).
 
 The v1 stabilization freeze covers required core fields, meanings, ordering,
 identity, and validation behavior. The future `x-` experimental namespace is
@@ -245,7 +340,7 @@ railroad SVG. The railroad renderer includes visible wrapped rule descriptions i
 full escaped text through SVG descriptions. `ibex doc` resolves the same canonical include graph and writes to stdout or an
 atomic file without generating or executing application parser code.
 
-## Automaton IR versions 1 and 2
+## Current Automaton IR
 
 Top-level fields are `ibex_ir: "automaton"`, `schema_version`, `algorithm`, `grammar_digest`, embedded `grammar`, `states`,
 optional `entry_states`, and `conflict_summary`. Embedding Grammar IR makes automaton JSON sufficient for code generation after
@@ -264,10 +359,17 @@ Each state contains:
 `conflict_summary.sr` counts unresolved default-shift conflicts for `expect`; `resolved_sr` counts retained precedence or
 associativity decisions; `rr` counts reduce/reduce cells.
 
-New automata use version 2 and always embed Grammar IR version 2. Migration recalculates `grammar_digest` from the upgraded
-canonical grammar. Version-1 automata remain loadable, validatable, and byte-stable. Published Draft 2020-12 contracts for both
-versions live under `schema/`; see
-[ADR 0001](decisions/0001-versioned-ir-pipeline.md).
+The current Automaton IR embeds the exact current Grammar IR, includes the parser contract in `grammar_digest`, and records
+`entry_construction` as `shared` or `isolated`. Older documents are rejected rather than migrated. Callers construct the
+current object directly through `IR::Automaton.new`.
+
+`--from=grammar-ir` applies the current parser contract through the typed configuration resolver before constructing tables;
+matching CLI values are allowed and conflicts are rejected with the contract location. `--from=automaton-ir` generates without
+the original source and rejects algorithm/entry construction flags because its tables are already built. A generation
+manifest records the grammar digest and contract separately from the embedded automaton's algorithm/entry facts and codegen-only
+effective configuration. Static IR views validate opaque action strings but never execute them. Published Draft 2020-12
+contracts for both current IR documents live under `schema/`; see
+[ADR 0001](decisions/0001-separate-ir-pipeline.md).
 
 `--emit=sets` is a deterministic analysis view rather than another IR: it emits lexically sorted nullable nonterminals and
 FIRST/FOLLOW maps for nonterminals. DOT, Mermaid, and the self-contained searchable HTML report are deterministic presentation
@@ -298,6 +400,56 @@ shortest reachability witness instead of claiming ambiguity.
 shape. `Counterexample#all` retains its original all-conflict behavior. The view performs no additional parser analysis and does
 not extend Grammar or Automaton IR.
 
+`Ibex::Verify::Verifier` accepts validated Automaton IR and derives LR(0) or
+canonical LR(1) collections from the embedded Grammar IR without calling the
+parser builder. Default checks validate item/action soundness,
+algorithm-specific lookaheads, expanded default reductions and error masks,
+reachability, productivity, epsilon termination, resolver consistency, and
+rebuilt plain/compact row equality. `--strict` adds collection completeness
+and reports that same row equality as V5 rather than V4. The verifier does not
+consume generated Ruby or a supplied executable table artifact. Strict IELR
+verification also runs the bounded V9 canonical acceptance witness; this is
+not an unbounded equivalence proof. Its exact
+shared dependencies, algorithm limits, resource semantics, fault mapping, and
+non-goals are documented in the [verifier trust boundary](verifier-trust-boundary.md).
+`ibex verify` uses exit statuses 0 for valid, 1 for a violation, and 2 for
+reference-budget exhaustion.
+
+`Ibex::Equiv` first compares normalized grammar structure, then generates
+sentences in both directions and explores the product of two immutable LR
+state-stack machines in breadth-first order. The product key includes both
+stacks, statuses, and consumed-action counts; a supplied rule map also adds
+both postorder reduction traces. The first reported token witness is therefore
+shortest within the declared token/configuration bounds. Simulation uses only
+Automaton IR actions and gotos. It validates both inputs with the independent
+verifier and does not evaluate semantic actions. A rule map requests bounded
+tree comparison; without one, only accepted language is compared. Every
+successful bounded report explicitly states that it is not an equivalence
+proof.
+
+`Ibex::Fix` composes those two read-only safety oracles around a finite repair
+space. It rebuilds a candidate through the shared construction pipeline, but
+accepts it only after the target fingerprint disappears, other fingerprints
+do not increase, independent verification succeeds, and bounded language plus
+identity-mapped reduction traces find no difference. User actions remain
+opaque. Source proposals are whole-file deterministic edits; the CLI delegates
+application to the existing transactional writer and refuses aliased inputs.
+`Ibex::Diff` and `Ibex::Metrics` are deterministic views over the same frozen
+Grammar and Automaton IR. They do not add an IR stage or alter generated
+parsers.
+
+`Ibex::BisonImport` is a clean-room adapter in front of that frozen pipeline.
+Its iterative scanner recovers declarations, production alternatives, source
+positions, and opaque C actions under byte/token/rule/action budgets, then
+emits ordinary extended source. It adds no frontend syntax or IR field.
+Deterministic terminal/nonterminal namespaces prevent Bison casing and keyword
+rules from changing symbol kinds. The report separates unsupported directives
+that leave the recovered production graph complete from structural gaps.
+Analysis may continue across either result, but Ruby generation rejects the C
+action sentinel and `Fix` rejects structurally incomplete source. Pinned
+third-party grammars exist only in temporary external CI directories; see the
+[import guide](bison-import.md).
+
 The repository's self-authored representative grammar feeds the current versioned `ibex_benchmark` v2 document. Its JSON Schema
 is shipped beside the IR schemas, while committed environment-specific observations live under the matching
 `benchmark/results/vN` directory. Timing and peak RSS remain non-gating; CI reproduces only deterministic structure and digests.
@@ -332,7 +484,7 @@ committed events without changing parser results; the recovery hook retains the
 ordinary token shift. A configured value printer affects only human `yydebug` output. Their ordering and payload contract is
 extended additively by
 [ADR 0010](decisions/0010-committed-runtime-observation.md). Grammar-declared
-symbol printers are optional IR v2 metadata compiled into private methods and
+symbol printers are optional current IR metadata compiled into private methods and
 an id-indexed table.
 
 Ordinary generated tables are recursively frozen and made Ractor-shareable. Threads and Ractors share those tables but parse
@@ -341,7 +493,7 @@ single instance rejects overlapping drivers. Immutable `Runtime::ResourceLimits`
 entry with finite defaults. Exhaustion raises the structured `ResourceLimitError`; see
 [ADR 0009](decisions/0009-isolated-parser-sessions.md).
 
-Extended Grammar IR v2 may additionally carry synchronization terminals and ordered `%on_error_reduce` groups. Table
+Extended current Grammar IR may additionally carry synchronization terminals and ordered `%on_error_reduce` groups. Table
 construction fills only otherwise erroneous ACTION cells with a unique highest-priority completed declared production. At
 runtime, an explicit shift of the synthetic `error` token always wins; only when it is unavailable does panic recovery discard
 through a configured synchronization token and pop to a state that accepts that retained lookahead. Generated parsers without
@@ -352,7 +504,7 @@ The separate `Runtime::Parser#observe` API publishes ordered, immutable schema-v
 copies only JSON data and never retains application identities or private stacks. With no observer, parse transitions construct
 no Event, payload summary, or dispatch snapshot; parser initialization still creates its ownership mutex. Generated tables
 contribute grammar digest, table format, state count, and production count to the `start` event. `Runtime::EventJSONLTracer`
-exposes the versioned stream; the original hook-based `Runtime::JSONLTracer` remains byte-compatible. The protocol and
+exposes the versioned stream. The protocol and
 exception/threading behavior are fixed by
 [ADR 0010](decisions/0010-committed-runtime-observation.md).
 
@@ -375,7 +527,7 @@ source stored in Grammar IR. Immutable steps expose state, lookahead, selected a
 depth. Action and stack budgets bound default/epsilon cycles and growth. The text/JSON CLI and versioned output contract are
 fixed by [ADR 0012](decisions/0012-bounded-nonexecuting-analysis.md).
 
-Grammar IR v2 may carry ordered accept/reject source tests without adding them to parser tables. `GrammarTests::Runner` generates
+The current Grammar IR may carry ordered accept/reject source tests without adding them to parser tables. `GrammarTests::Runner` generates
 one embedded parser and executes fresh parser instances in a separate Ruby process, distinguishing `ParseError` rejection from
 lexer/application errors and bounding the whole suite by a timeout. The separate runner loads the generated file, so guarded
 footer programs stay inactive. The source contract, isolation boundary, and CI behavior are fixed by

data/docs/bison-import.md

@@ -0,0 +1,103 @@
+# Bison grammar import for analysis
+
+`ibex import bison` converts Bison-shaped grammar structure into ordinary
+extended Ibex source for analysis. It is a one-way adapter, not a parser
+generator compatibility mode. Imported C actions remain opaque, are never
+executed by analysis, and make Ruby generation fail before code is emitted.
+
+```sh
+ibex import bison parser.y -o parser.analysis.y
+ibex import bison --format=json parser.y > import-report.json
+ibex explain --state=STATE parser.y
+ibex metrics parser.y
+```
+
+`explain`, `metrics`, `diff`, and the other read-only grammar analysis paths
+recognize a file with two Bison `%%` section markers and import it in memory.
+Source-repair commands need a canonical imported file because edits against
+the generated analysis source do not map byte-for-byte onto the Bison input.
+
+## Conversion contract
+
+| Classification | Directives | Treatment |
+|---|---|---|
+| converted | `%token`, `%left`, `%right`, `%nonassoc`, `%precedence`, `%start`, `%expect`, `%expect-rr`, `%empty`, `%prec` | existing extended declarations or production syntax |
+| recognized metadata | `%nterm`, `%type`, `%union`, `%destructor`, `%printer`, `%param`, `%parse-param`, `%lex-param`, `%define`, `%code` | positioned report entries; no new Ibex syntax is invented |
+| unsupported | `%dprec`, `%merge`, GLR controls, generator/output controls, and every unknown directive | every occurrence is reported with its original line and column |
+
+`$$`, `$1`, `$<type>1`, and `@$` become `result`, `val[0]`, `val[0]`, and
+`result_loc` inside an opaque hexadecimal action sentinel. Ordinary `@1`
+locations remain `@1`. Balanced action scanning handles nested braces,
+strings, character literals, and C comments iteratively.
+
+Bison permits uppercase nonterminals, lowercase terminals, names that collide
+after punctuation removal, and names that are Ibex declaration words. The
+adapter deterministically namespaces nonterminals as `bison_nt_*`, maps
+lowercase terminals into `BISON_T_*`, and adds stable numeric suffixes to
+sanitization collisions. Named LHS forms such as `expression[result]:` are
+recognized.
+
+The source and JSON report distinguish general unsupported directives from
+structural gaps. Output/language controls and opaque initialization hooks do
+not make the recovered production graph incomplete. Unknown directives,
+`%dprec`, `%merge`, GLR semantics, and grammar-macro directives do. A generated
+header therefore contains exactly one of:
+
+```text
+# ibex-bison-structural-status: complete
+# ibex-bison-structural-status: incomplete
+```
+
+`ibex fix` refuses a structurally incomplete import. This prevents a repair
+from being called safe when the bounded comparison only covered an
+approximation of the source grammar.
+
+## Limits and failure behavior
+
+The defaults are 20 MiB of input, 1,000,000 structural tokens, 50,000 rule
+groups, and 100,000 actions. Override them with `--max-bytes`,
+`--max-tokens`, `--max-rules`, and `--max-actions`; every value must be
+positive. Exhaustion produces a version-1 JSON envelope and exits 2.
+Successful import exits 0 even when unsupported directives are present,
+because the report—not silence—is the result. Invocation and malformed-input
+errors go to stderr and exit 1.
+
+`-o` writes atomically, rejects an input/output alias, and refuses symlink
+targets or files with multiple hard links. Third-party grammar files are not
+packaged or committed.
+
+## External evidence and the `parse.y` boundary
+
+The scheduled external gate downloads five checksum-pinned GNU Bison grammars
+listed in [`gallery/EXTERNAL.md`](../gallery/EXTERNAL.md). It validates the
+import report, parses the generated source, normalizes it, and builds an Ibex
+LALR automaton. Current aggregate evidence is:
+
+| Grammar | Rule groups | Actions | Productions | Ibex states | Unresolved S/R |
+|---|---:|---:|---:|---:|---:|
+| GNU Bison calc | 5 | 7 | 13 | 22 | 0 |
+| jq | 29 | 167 | 167 | 311 | 408 |
+| PHP | 177 | 552 | 635 | 1,203 | 0 |
+| PostgreSQL | 795 | 2,436 | 3,640 | 6,942 | 0 |
+| CRuby Bison-era `parse.y` | 223 | 594 | 781 | 1,303 | 0 |
+
+For the pinned Bison-era CRuby grammar, GNU Bison 3.8.2 and Ibex both recover
+781 non-augmented productions and zero unresolved conflicts. Bison reports
+1,304 states while Ibex reports 1,303. The one-state difference is explained
+by the acceptance convention: Bison shifts end-of-input into a separate
+accept state, while Ibex accepts from the completed augmented start item.
+
+Current CRuby `parse.y` is an Lrama input, not raw GNU Bison input. The pinned
+current file imports 231 rule groups and 486 actions, but reports 22
+structurally unsupported occurrences, including `%rule` grammar macros. The
+approximate automaton has 1,152 states and 27 unresolved S/R conflicts.
+Selecting state 413 keeps `explain` bounded and yields one nonunifying
+reachability witness for the `'rescue' modifier` shift/reduce choice within 8
+tokens and 10,000 configurations. That is useful diagnostic evidence about
+the recovered approximation, not a claim about CRuby's actual Lrama table.
+`fix` refuses this input because proposing a language-preserving repair across
+unexpanded `%rule` definitions would be unsound.
+
+The external gate reads public grammar text and GNU Bison's black-box report
+only. It does not inspect Bison implementation source or generated C, and
+publishes only aggregate counts.

data/docs/comparison-policy.md

@@ -0,0 +1,127 @@
+# Comparative claim policy
+
+This policy defines when Ibex documentation may compare Ibex with another
+parser tool. The machine-readable registry is [`claims.yml`](claims.yml), and
+`bundle exec rake quality:comparative_claims` validates the registry and its
+public claim markers.
+
+The initial comparison set is Racc, Lrama, GNU Bison, Menhir, Tree-sitter, and
+ANTLR. A tool being in the set does not imply that a comparison has been made.
+An unmeasured tool remains `not_compared`; unknown facts remain `unknown`.
+Each entry also owns its canonical public aliases. README scanning recognizes
+only this declared set and aliases; adding a tool or spelling requires a
+reviewed registry change.
+
+Tool-level state is derived from every registered claim that names that tool.
+`pending_claims` is the canonical ordered list of claim IDs that are not yet
+`measured`, and the reason is generated deterministically from those IDs,
+their pending state, and their declared missing evidence. A tool with no claims
+is `not_compared` with an empty pending list and unknown identity. A tool with
+any pending claim is `evidence_pending`, even if another claim for that tool is
+measured. `compared` is valid only when the tool has at least one claim, every
+one is measured, and `pending_claims` is empty.
+
+## Required record
+
+Every measured, evidence-pending, or review-pending record contains:
+
+- an immutable claim ID and narrowly scoped wording;
+- each subject's exact released version and, when applicable, repository
+  revision;
+- one public command represented as an executable and an ordered argv list;
+- fixed corpus paths and external revisions;
+- known environment values and an explicit list of unrecorded values;
+- excluded or unsupported semantics;
+- the subjective review method and its completion state;
+- repository-relative evidence paths and explicit limitations; and
+- the SHA-256 of the canonical marker body, including every table and note; and
+- an exact-revision validity scope, expiry statement, and conditions that
+  require review.
+
+Repository subjects and corpora use immutable 40- or 64-hex revisions. Released
+tools use an exact release version and may use `not_applicable` only for the
+repository revision. It must not conceal missing information. Every required
+environment identity is either recorded as a non-placeholder value or listed
+as `unknown`, and each unknown field is repeated in the public limitations.
+
+No extra JSON Schema is used for this YAML registry. Its cross-file rules—file
+existence, canonical ordering, public markers, and wording restrictions—need an
+explicit repository validator and would not be made clearer by duplicating
+them in a data-only schema.
+
+## Public interface and corpus rules
+
+Comparisons use public commands, public callbacks, or documented artifact
+formats. Do not inspect another tool's implementation or generated source to
+manufacture an advantage. If a public interface cannot expose a value, record
+that limitation instead of inferring it.
+
+Inputs, grammar revisions, lifecycle choices, lexer inclusion, runtime backend,
+warm-up, repetitions, random seeds, and instrumentation must be fixed before a
+measurement. Failure rows stay in the evidence. Results from different
+environments or configurations form separate observations and must not be
+silently merged.
+
+## Subjective review
+
+Diagnostic and repair usefulness are human judgments. The review method must
+fix case IDs and the allowed labels, retain rationale and disagreements, and
+separate maintainer assessment from independent assessment. A record whose
+required independent review is missing stays `review_pending`; a record missing
+its direct result artifact stays `evidence_pending`. Neither state can use a
+public strong-claim marker or be worded as a completed comparative conclusion.
+
+Performance statistics and deterministic behavior digests do not require a
+subjective reviewer, but their scope and limitations still require ordinary
+code review.
+
+## Scoped wording
+
+Comparative wording names the claim ID and states the relevant revision or
+release, corpus, environment, measurement category, and important limitation.
+For example:
+
+> At revision X, on corpus Y and environment Z, metric M had the recorded
+> relationship to tool version V. This does not describe other revisions,
+> workloads, backends, or semantic capabilities.
+
+Do not publish wording such as “faster than Racc” or “better diagnostics”
+without those boundaries. README comparative strength wording is enclosed by
+`comparative-claim` markers. Only a `measured` record with a direct result
+artifact may use that marker. Pending material instead uses a
+`comparative-evidence` marker around the full table or conclusion it records.
+The validator requires each marker body to contain the registry wording exactly
+and every registered table anchor. It also canonicalizes line endings and
+trailing whitespace and requires the complete body SHA-256 to match, so any
+other prose, table-cell, row, or blank-line change fails independently of its
+vocabulary. It always scans README, even when no claim targets README, and
+rejects unmarked paragraphs that combine a declared tool alias with English or
+Japanese strength wording.
+
+## No cross-category ordering
+
+<!-- comparison-policy:forbidden-terms:start -->
+Do not collapse performance, diagnostics, recovery, migration, ecosystem, and
+verification into an aggregate score or tool ranking. The categories have
+different semantics and trust boundaries. Publish each observation and its
+limitations separately. Numeric totals never imply semantic equivalence.
+<!-- comparison-policy:forbidden-terms:end -->
+
+## Review and update workflow
+
+1. Add evidence without deleting unfavorable or failed rows.
+2. Add or update the claim in canonical claim-ID order. Keep corpus and evidence
+   entries ordered by ID and path.
+3. Mark unavailable tools or facts `not_compared` or `unknown`; do not estimate
+   values.
+4. Bind measured wording with claim markers or pending tables and conclusions
+   with evidence markers. The body must retain the exact registered wording.
+5. After reviewing an intentional marker-body diff, run the quality task. Its
+   mismatch reports both the registered and actual SHA-256; copy the actual
+   digest into `body_sha256` only in the same reviewed change. Never refresh a
+   digest merely to make the task pass.
+6. Run `bundle exec rake quality:comparative_claims`, the focused tests, and
+   documentation coverage.
+7. When a review condition fires, either remeasure under a new claim ID or
+   narrow the old wording to its historical scope. Historical observations are
+   retained rather than rewritten as current results.