ibex 0.2.0 → 0.3.0

data/docs/verifier-trust-boundary.md

@@ -0,0 +1,199 @@
+# Verifier trust boundary
+
+`ibex verify` checks a validated Automaton IR document against expectations
+derived from the Grammar IR embedded in that document. It is a semantic
+cross-check for the current Automaton IR. It is not a proof checker for a
+generated parser, a verifier for arbitrary Ruby, or a security sandbox.
+
+## Input and result boundary
+
+The command reads one complete JSON document with `File.binread`.
+`IR::Validator` first parses it and validates its raw shape, schema version,
+symbol/state/production references, and conflict records and summaries. It
+then calls `IR::Serialize.load` to construct immutable Grammar and Automaton
+IR objects. Finally, before returning the Automaton, it recomputes and checks
+the embedded-grammar digest. Those validation and construction steps are a
+prerequisite to, not a substitute for, the semantic checks below.
+
+The embedded Grammar IR is the verifier's semantic source of truth. Supplying
+`--grammar=GRAMMAR.json` requires byte-deterministic serialized equality with
+that embedded Grammar IR, but does not bind either IR to an original `.y`
+file. Recomputing `grammar_digest` detects accidental inconsistency inside the
+document; an author who changes both the embedded grammar and its digest has
+created a different self-consistent input, not a verifier failure.
+
+A semantic violation produces an invalid result and CLI exit status 1. A
+completed run with no named violation produces a valid result and exit status
+0 within this document's boundary. Reference-construction exhaustion is a
+separate result with exit status 2; it is neither validity nor a violation.
+
+## Named invariants
+
+The report's invariant IDs are stable schema values. “Default” means every
+run; “strict” means the ID is listed only for `--strict`.
+
+<!-- verifier-checks:start -->
+| ID | Mode | What the implementation checks |
+| --- | --- | --- |
+| V1 | default | The embedded grammar digest, state item soundness against the algorithm-specific reference, transition shift preservation, nonterminal transition/GOTO agreement, and exact single-candidate ACTION selection. Unsupported programmatic algorithm values are also V1. |
+| V2 | strict | Collection completeness: exact LR(0) core coverage for SLR; exact derived-state/item coverage for LALR(1) and LR(1); and canonical-core lookahead-union coverage for IELR(1). |
+| V3 | default | Every completed SLR item has exactly its independently requested FOLLOW lookaheads (or EOF for an augmented production). Non-completed SLR lookaheads cannot select ACTION cells and are deliberately outside this check. |
+| V4 | default | Default actions are reductions, terminals with no candidate remain errors after expanding defaults and explicit masks, and rebuilt plain/compact ACTION, GOTO, and default rows are equal. A table transformation failure is also V4 in default mode. |
+| V5 | strict | The same rebuilt plain/compact row equality that default mode reports as V4 is classified as V5 in strict mode. Strict mode does not consume a separately supplied table artifact. |
+| V6 | default | Every submitted state is reachable from an entry through submitted transitions, GOTOs, or shift actions, and every embedded-grammar nonterminal derives some terminal sentence. |
+| V7 | default | For each terminal, effective zero-length reductions and their submitted GOTOs do not form a state cycle that can reduce forever without consuming that terminal. |
+| V8 | default | A cell with multiple derived candidates has a declared resolver; resolver records are unique and complete for their token, name only cell candidates, and choose the effective ACTION (or an allowed error). |
+| V9 | strict | For IELR, a bounded canonical-reference witness compares accepted/error outcomes for terminal sequences up to three tokens and fails closed on witness-budget exhaustion. It is a semantic regression witness, not unbounded language equivalence or split-proof verification. Non-IELR algorithms do not execute the witness. |
+<!-- verifier-checks:end -->
+
+Default and strict both construct the algorithm-specific reference and both
+rebuild and compare plain and compact rows. Strict adds the V2 completeness
+requirements. It also changes a row mismatch's identifier from V4 to V5; it
+does not add another table representation or inspect generated table bytes.
+In particular, default-mode item checks are soundness/subset checks and can
+accept an incomplete collection that strict mode rejects. For `ielr1`, strict
+mode additionally runs V9. V9 compares the submitted table with a
+verifier-owned canonical LR(1) machine over a finite breadth-first token set;
+a clean result is evidence only within the recorded token and case bounds.
+
+## Supported algorithms and reference cost
+
+The CLI validator accepts only the following serialized algorithm identities.
+
+<!-- verifier-algorithms:start -->
+| Automaton IR value | Reference and comparison |
+| --- | --- |
+| `slr` | Build the full LR(0) collection. Completed submitted items use FOLLOW sets for lookaheads. Strict mode requires exact LR(0) core coverage. |
+| `lalr1` | Build the full canonical LR(1) collection and union states with the same LR(0) core. Submitted items are subsets by default; strict mode requires the complete merged states. |
+| `ielr1` | Build the full canonical LR(1) collection and its LALR core unions. Each submitted partition must be a subset of its core union; strict mode requires the union of submitted partitions to cover it and runs the bounded V9 acceptance witness. This does not prove unbounded IELR adequacy, the reason for each split, or split witnesses. |
+| `lr1` | Build the full canonical LR(1) collection. Submitted items are subsets by default; strict mode requires the complete canonical states. |
+<!-- verifier-algorithms:end -->
+
+Thus LALR(1), IELR(1), and LR(1) verification enumerate the canonical LR(1)
+collection even in default mode. That collection can grow exponentially with
+the grammar. SLR enumerates LR(0), but is still a whole-collection operation.
+The current verifier is consequently a bounded reference oracle, not a
+scale-independent certificate checker. A serialized unknown algorithm is
+rejected structurally before the CLI verifier runs; an Automaton object made
+directly with an unknown value receives V1.
+
+The supported `ielr1` check is intentionally narrower than a future direct
+IELR verifier: it checks membership, union coverage, and (in strict mode) a
+bounded acceptance witness. It does not establish conflict preservation,
+canonical state correspondence, or split witnesses. Raw ACTION-cell
+correspondence remains a diagnostic: IELR state merging can produce a
+canonical error cell versus a target reduction cell without an observed
+acceptance difference in the bounded witness.
+
+## Limits and exhaustion
+
+The two CLI limits must be positive and apply only to reference collection
+construction.
+
+<!-- verifier-limits:start -->
+| Limit | Default | Meaning |
+| --- | ---: | --- |
+| `--max-states` | `100000` | Maximum distinct reference states inserted. |
+| `--max-items` | `1000000` | Work-account limit for augmented seed items and new closure-expansion items. It is not the final collection's unique-item count and does not count every moved GOTO kernel item. |
+<!-- verifier-limits:end -->
+
+Exceeding either limit raises `Verify::BudgetExceeded`; the CLI emits
+`result: "budget_exhausted"` with the bounds and exits 2. Strict mode uses the
+same two limits and has no separate completeness budget.
+
+These limits do **not** bound input bytes, JSON parsing, IR validation,
+embedded grammar size, submitted state/table size, nullable/FIRST/FOLLOW
+analysis, table compression, all verifier loops, wall-clock time, Ruby heap,
+or process resources. An operating-system failure, timeout, or memory
+exhaustion is not converted into a verifier budget result. Callers handling
+hostile inputs need an outer byte/time/memory/process boundary.
+
+## Trusted computing base and independence
+
+Here, “independent” means only that `Verify::ReferenceCollection` implements
+its own LR(0)/LR(1) closure, GOTO, and collection traversal and does not require
+or call `LALR::Builder`. It does not mean independently authored, third-party
+reviewed, formally verified, cryptographically certified, or free of shared
+implementation defects.
+
+The result trusts all of the following:
+
+| Trusted component or input | Why it is in the TCB |
+| --- | --- |
+| Embedded Grammar IR | It defines the grammar against which the submitted automaton is checked. |
+| `IR::Validator`, `IR::Serialize`, IR migration and IR model classes | They validate, decode, canonicalize, hash, and expose the input. |
+| `Verify::Verifier`, `Verify::ReferenceCollection`, and result classes | They derive and evaluate all named semantic invariants. |
+| `Analysis::Sets` | It supplies nullable, FIRST, and FOLLOW data. The parser builder also uses this component, so defects here can be correlated. |
+| `Tables.build`, `Tables::Compact`, and `Tables::CompactActions` | The verifier rebuilds both in-memory table views with production table code. Plain/compact comparison is therefore not an independent compact-table implementation. |
+| Ruby runtime and standard library (`JSON`, `Set`, `Digest`, file I/O) | They parse, enumerate, compare, hash, and allocate the checked data. |
+| Host resource isolation | The verifier itself supplies only the two reference-work limits above. |
+
+Excluding `LALR::Builder` is useful deliberate duplication, but the shared
+sets, IR, table transformation, language runtime, and embedded input remain in
+the TCB. A valid report is evidence against the implemented fault model, not
+an unqualified correctness proof.
+
+## Explicit non-goals
+
+<!-- verifier-non-goals:start -->
+| Boundary | What is not verified |
+| --- | --- |
+| `source-to-ir` | Correct parsing or normalization of the original grammar source, or provenance from that source to the embedded Grammar IR. |
+| `generated-ruby` | Generated Ruby source, wrapper structure, digests, loading behavior, or binding between generated output and this Automaton IR. |
+| `runtime` | The parser runtime implementation, actual parser execution, recovery behavior under application callbacks, or plain/compact behavior after code generation. |
+| `semantic-actions` | Meaning, type safety, termination, side effects, or correctness of parser actions and user-code sections. Their text is opaque and is never evaluated. |
+| `lexer-actions` | Lexer IR, tokenization, generated lexer actions, handwritten lexers, or parser-to-lexer feedback. |
+| `resolver-policy` | Whether a resolver's claimed `by` reason correctly applies grammar precedence/associativity policy; V8 checks record/candidate/selected-cell consistency. |
+| `grammar-properties` | Unambiguity, language equivalence to another grammar/parser, completeness of `%expect` as a design claim, or application semantics. |
+| `ielr-adequacy` | Unbounded language equivalence, conflict preservation, why an IELR state was split, or a proof that every split has the required canonical correspondence. Strict V9 supplies only a bounded acceptance witness. |
+| `data-only-artifact` | The standalone `ibex verify` command does not consume a supplied executable table artifact, compact-table bytes, generated bundle, manifest/report binding, or artifact digest. The current Automaton IR contains states and actions, and the verifier rebuilds table views from them. The separate bundle report records this verifier result and binds table identities without widening V1-V9. |
+| `security` | Authenticity, trusted publication, signatures, sandboxing, confidentiality, or availability under hostile resource use. |
+<!-- verifier-non-goals:end -->
+
+Opaque action and user-code strings participate in serialized Grammar IR and
+therefore its digest, but checking that self-contained digest is not semantic
+verification of the code and never executes it.
+
+The data-only table and scoped bundle are separate layers around this
+verifier. Their validators check closed formats and digest relationships, but
+they do not make the standalone V1-V9 implementation consume or independently
+verify sidecar bytes. Consequently `ibex verify` must still not be described
+as validating generated Ruby, table artifacts, or manifests.
+
+## Committed fault corpus
+
+`test/verify/verifier_test.rb` mutates a generated document in twenty named
+ways while keeping it acceptable to `IR::Validator`, then requires strict
+verification to reject it. The table records the invariant IDs currently
+observed for each mutation. Multiple IDs mean one mutation violates more than
+one invariant; they do not turn the finite corpus into completeness evidence.
+
+<!-- verifier-faults:start -->
+| Fault | Observed invariant IDs |
+| --- | --- |
+| `add_lookahead` | V1, V2 |
+| `remove_item` | V1, V2 |
+| `remove_lookahead` | V2 |
+| `move_item_dot` | V1, V2 |
+| `remove_transition` | V4 |
+| `redirect_transition` | V1 |
+| `remove_action` | V1 |
+| `redirect_shift` | V1 |
+| `replace_reduction` | V1 |
+| `remove_goto` | V1 |
+| `redirect_goto` | V1 |
+| `add_default` | V4 |
+| `remove_default` | V1 |
+| `remove_error_mask` | V4 |
+| `remove_conflict` | V8 |
+| `duplicate_conflict` | V8 |
+| `redirect_conflict_shift` | V8 |
+| `change_conflict_choice` | V8 |
+| `add_unreachable_state` | V1, V6 |
+| `epsilon_cycle` | V1, V4, V7 |
+<!-- verifier-faults:end -->
+
+The corpus demonstrates detection of these twenty constructed, structurally
+valid mutations on its committed grammars and construction harness. It does
+not establish that all corruptions, all combinations of faults, or faults in
+the TCB are detected.

data/docs/workloads.md

@@ -0,0 +1,132 @@
+# Public workload and problem registry
+
+[`workloads.yml`](workloads.yml) is the machine-readable authority for workload
+identity, source and license evidence, structural measurements, known problems,
+workarounds, and benchmark eligibility. It is a planning registry, not a claim
+that every grammar is fully supported or that every eligible scope has a
+published result.
+
+## Source classes
+
+The `classification` field prevents unlike evidence from being combined:
+
+| Class | Meaning |
+| --- | --- |
+| `public_real` | A public third-party application grammar with an exact repository revision. |
+| `external_real` | A public third-party grammar used by the download-only Bison import gate. |
+| `repository_real` | A production Ibex workload, currently the self-hosted frontend. |
+| `repository_synthetic` | A repository-authored correctness fixture; it is not adoption or public-language evidence. |
+
+In particular, the gallery is `repository_synthetic`. Its results can support
+correctness, fuzzing, and verifier claims, but cannot by themselves establish a
+benefit for Namae, BCDice, Nokogiri, CRuby, jq, PHP, or PostgreSQL. The three
+`public_real` workloads may support only the scopes listed in their eligibility
+records and only after the benchmark's checkout and behavior checks pass.
+
+## Counting contract
+
+`ibex-normalized-lalr-v1` parses the exact pinned bytes in their admitted mode,
+normalizes them to Grammar IR, counts user productions and all normalized
+terminals (including `$eof` and `error`), then builds the direct LALR automaton
+and counts states and unresolved conflicts. Bison-family sources are first
+processed by the Ibex importer, which strips actions and reports unsupported
+directives.
+
+`measured` means that the complete grammar admitted by that method was counted.
+`not_measured` requires a reason and carries no numeric value. The pinned current
+CRuby Lrama grammar has 22 structural import gaps, so its partial importer
+figures are deliberately not promoted to production, token, state, or conflict
+counts.
+
+The registry validator recomputes repository-owned grammar counts, cross-checks
+the three public identities, grammar digests, production/state/token counts,
+and conflicts against
+[`benchmark/public_workloads.json`](../benchmark/public_workloads.json), and
+cross-checks Bison identities and every structurally complete expected metric,
+including tokens, against
+[`tool/quality/bison_external.rb`](../tool/quality/bison_external.rb). It does
+not download third-party source in ordinary CI.
+
+Remote source, license, and permission records have a second reviewed binding in
+[`tool/quality/workloads/evidence.yml`](../tool/quality/workloads/evidence.yml).
+The validator requires exact owner/project/revision/path-derived GitHub
+locators, license expressions, evidence digests, and classification-appropriate
+permission states to match that fixture. Repository-owned evidence is instead
+checked against both the current file and its pinned Git object.
+
+## Registered workloads
+
+Counts are production/state/token. `n/m` means `not_measured`, not zero.
+
+| Stable workload ID | Class | Grammar | Counts | Conflicts SR/RR | Benchmark status and scope |
+| --- | --- | --- | ---: | ---: | --- |
+| `bcdice-command-parser` | `public_real` | BCDice command parser | 34/56/19 | 2/0 | eligible: public generation/runtime |
+| `bison-gnu-calc` | `external_real` | GNU Bison C calculator | 13/22/11 | 0/0 | diagnostic only: Bison import compatibility |
+| `bison-jq-parser` | `external_real` | jq parser | 167/311/100 | 408/0 | diagnostic only: import/conflict analysis |
+| `bison-php-parser` | `external_real` | PHP Zend language parser | 635/1203/185 | 0/0 | diagnostic only: import/scale construction |
+| `bison-postgresql-parser` | `external_real` | PostgreSQL backend SQL grammar | 3640/6942/562 | 0/0 | diagnostic only: import/scale construction |
+| `bison-ruby-bison-era` | `external_real` | CRuby Bison-era parse.y | 781/1303/158 | 0/0 | diagnostic only: import/external state comparison |
+| `bison-ruby-current-lrama` | `external_real` | Pinned current CRuby Lrama parse.y | n/m | n/m | ineligible: structurally incomplete import |
+| `gallery-calc` | `repository_synthetic` | Calculator fixture | 7/15/9 | 0/0 | eligible: correctness/fuzzing/verification |
+| `gallery-json` | `repository_synthetic` | JSON fixture | 17/27/13 | 0/0 | eligible: correctness/fuzzing/verification |
+| `gallery-sql-lite` | `repository_synthetic` | SQL-subset fixture | 14/26/14 | 0/0 | eligible: correctness/fuzzing/verification |
+| `ibex-frontend` | `repository_real` | Production self-hosted grammar | 146/227/60 | 0/0 | eligible: self-host/regression |
+| `namae-parser` | `public_real` | Namae name parser | 51/78/12 | 0/0 | eligible: public generation/runtime |
+| `nokogiri-css-parser` | `public_real` | Nokogiri CSS parser | 81/117/32 | 0/1 | eligible: public generation/runtime |
+
+The exact owner/project, full revision, path, source SHA-256, license expression
+and evidence digest, public-source permission evidence, feature inventory,
+current pain, current workaround, and eligibility reasons live in the YAML
+record. Third-party license locators include the full pinned revision; their
+digests were calculated from those exact bytes. The registry records evidence
+and does not replace a project's license terms or legal review.
+
+## Stable problem IDs
+
+Future design and profiling work should cite a problem and at least one workload
+ID instead of saying only that a change helps “real grammars.”
+
+| Stable problem ID | Current evidence boundary |
+| --- | --- |
+| `problem-bison-dialect-coverage` | Every pinned Bison-family import reports at least one unsupported directive; current CRuby has structural gaps. |
+| `problem-conflict-volume` | `bison-jq-parser` has 408 unresolved shift/reduce conflicts in the measured imported direct-LALR automaton. This is not a claim about jq's production parser behavior. |
+| `problem-public-runtime-performance` | The evidence-pending historical projection retains slower cold-generation and new-instance rows for the three `public_real` workloads. |
+
+For example, a direct-IELR proposal can cite
+`problem-conflict-volume` + `bison-jq-parser`, but it must still establish that
+the imported conflicts are semantically meaningful before claiming an
+application benefit. A runtime optimization can cite
+`problem-public-runtime-performance` plus the affected public workload IDs and
+must publish a new environment-bound comparison artifact before making a
+current performance claim.
+
+The H005 [construction profile](construction-profiling.md) consumes these exact
+identities and keeps unavailable public checkouts as `not_run`. Synthetic and
+repository-real construction observations never stand in for third-party
+evidence.
+
+## Updating the registry
+
+Run:
+
+```sh
+bundle exec rake quality:workloads
+bundle exec ruby -Itest test/quality/workloads_test.rb
+```
+
+For a repository grammar, update its revision to a commit that already contains
+the recorded source bytes, update its digest, and let the validator recompute
+counts. The initial repository entries intentionally use pre-registry revision
+`d901b787c7e10c1ab3344ac0156f802ab29ad207`, avoiding a self-referential
+registry commit.
+
+For a public benchmark grammar, change
+`benchmark/public_workloads.json` and this registry together. For a Bison input,
+change the source constant, digest, expected measurements, exact-revision
+license evidence, and registry together. Any remote source/license change also
+requires an explicitly reviewed update to `tool/quality/workloads/evidence.yml`.
+When a pinned public checkout is supplied, the benchmark recomputes and rejects
+drift in all five structural metric families. A moving branch, tag without a
+resolved commit, short SHA, missing/changed local file, digest mismatch,
+unexplained numeric placeholder, or eligible record without complete count,
+license, and permission evidence fails the quality gate.

data/lib/ibex/analysis/digraph.rb

@@ -0,0 +1,128 @@
+# frozen_string_literal: true
+# rbs_inline: enabled
+
+# steep:ignore:start
+
+# The iterative SCC implementation keeps all graph bookkeeping in one pass.
+# rubocop:disable Metrics/AbcSize, Metrics/CyclomaticComplexity, Metrics/MethodLength, Metrics/PerceivedComplexity, Metrics/BlockLength
+
+module Ibex
+  module Analysis
+    # Computes the transitive bit-set closure used by the LR lookahead
+    # algorithms.  The implementation is iterative so a large grammar cannot
+    # exhaust Ruby's call stack merely because its dependency graph is deep.
+    module Digraph
+      module_function
+
+      # @rbs (Array[Integer] initial, Array[Array[Integer]] edges) -> Array[Integer]
+      def closure(initial, edges)
+        raise ArgumentError, "digraph edge count does not match values" unless initial.length == edges.length
+
+        values = initial.dup
+        components = strongly_connected_components(edges)
+        component_edges = Array.new(components.length) { [] }
+        component_for = Array.new(edges.length)
+        components.each_with_index do |members, component_id|
+          members.each { |vertex| component_for[vertex] = component_id }
+        end
+        edges.each_with_index do |successors, vertex|
+          source = component_for.fetch(vertex)
+          successors.each do |successor|
+            target = component_for.fetch(successor)
+            component_edges[source] << target if source != target
+          end
+        end
+        component_edges.each(&:uniq!)
+
+        order = topological_order(component_edges)
+        order.reverse_each do |component_id|
+          members = components.fetch(component_id)
+          merged = members.reduce(0) { |bits, vertex| bits | values.fetch(vertex) }
+          component_edges.fetch(component_id).each do |successor|
+            merged |= values.fetch(components.fetch(successor).first)
+          end
+          members.each { |vertex| values[vertex] = merged }
+        end
+        values
+      end
+
+      # @rbs (Array[Array[Integer]] edges) -> Array[Array[Integer]]
+      def strongly_connected_components(edges)
+        index = 0
+        indexes = Array.new(edges.length)
+        lowlinks = Array.new(edges.length)
+        on_stack = Array.new(edges.length, false)
+        stack = []
+        components = []
+        visit = lambda do |root|
+          frames = [[root, 0, false]]
+          until frames.empty?
+            vertex, offset, entered = frames[-1]
+            unless entered
+              indexes[vertex] = index
+              lowlinks[vertex] = index
+              index += 1
+              stack << vertex
+              on_stack[vertex] = true
+              frames[-1][2] = true
+            end
+
+            successors = edges.fetch(vertex)
+            if offset < successors.length
+              successor = successors.fetch(offset)
+              frames[-1][1] += 1
+              if indexes[successor].nil?
+                frames << [successor, 0, false]
+              elsif on_stack[successor]
+                lowlinks[vertex] = [lowlinks[vertex], indexes[successor]].min
+              end
+              next
+            end
+
+            if lowlinks[vertex] == indexes[vertex]
+              component = []
+              loop do
+                member = stack.pop
+                on_stack[member] = false
+                component << member
+                break if member == vertex
+              end
+              components << component.sort
+            end
+            frames.pop
+            next if frames.empty?
+
+            parent = frames[-1][0]
+            lowlinks[parent] = [lowlinks[parent], lowlinks[vertex]].min
+          end
+        end
+        edges.each_index { |vertex| visit.call(vertex) if indexes[vertex].nil? }
+        components
+      end
+      private_class_method :strongly_connected_components
+
+      # @rbs (Array[Array[Integer]]) -> Array[Integer]
+      def topological_order(edges)
+        indegree = Array.new(edges.length, 0)
+        edges.each { |successors| successors.each { |successor| indegree[successor] += 1 } }
+        queue = indegree.each_index.select { |vertex| indegree[vertex].zero? }
+        order = []
+        until queue.empty?
+          vertex = queue.shift
+          order << vertex
+          edges.fetch(vertex).each do |successor|
+            indegree[successor] -= 1
+            queue << successor if indegree[successor].zero?
+          end
+        end
+        raise Ibex::Error, "digraph contains a cyclic component graph" unless order.length == edges.length
+
+        order
+      end
+      private_class_method :topological_order
+    end
+  end
+end
+
+# rubocop:enable Metrics/AbcSize, Metrics/CyclomaticComplexity, Metrics/MethodLength, Metrics/PerceivedComplexity, Metrics/BlockLength
+# steep:ignore:end

data/lib/ibex/analysis.rb

@@ -1,6 +1,7 @@
 # frozen_string_literal: true
 
 require_relative "analysis/sets"
+require_relative "analysis/digraph"
 
 module Ibex
   module Analysis