ibex 0.1.0 → 0.3.0

data/docs/racc-migration.md

@@ -1,3 +1,8 @@
+---
+title: Migrating from racc
+description: Move a racc grammar to Ibex with explicit compatibility checks and runtime boundaries.
+---
+
 # Migrating from racc
 
 Ibex targets grammar-file compatibility, generated parser public API compatibility, and the main racc CLI options. It does not
@@ -5,11 +10,14 @@ copy racc's internal table arrays, internal method names, native runtime, or gen
 
 ## Typical migration
 
-1. Run `ibex -o parser.rb grammar.y` in place of `racc -o parser.rb grammar.y`.
-2. Change the generated-file runtime dependency from deployment packaging only; application calls to `do_parse`, `yyparse`,
+1. Run `ibex migrate-check grammar.y` and resolve any errors. Use `--format=json` for CI.
+2. Generate a reviewable parity harness with `ibex migrate-harness -o migration_harness.rb grammar.y`, add explicit token
+   cases, and run it inside an isolation boundary appropriate for the grammar code.
+3. Run `ibex -o parser.rb grammar.y` in place of `racc -o parser.rb grammar.y`.
+4. Change the generated-file runtime dependency from deployment packaging only; application calls to `do_parse`, `yyparse`,
    `next_token`, `on_error`, `token_to_str`, `yyerror`, `yyerrok`, and `yyaccept` remain the same.
-3. Use `-E` if the generated parser must be a single file with no installed Ibex gem.
-4. Keep the default `--mode=racc` until intentionally adopting EBNF or names. Extended grammars can make that choice locally by
+5. Use `-E` if the generated parser must be a single file with no installed Ibex gem.
+6. Keep the default `--mode=default` until intentionally adopting EBNF or names. Extended grammars can make that choice locally by
    placing `pragma extended` immediately after their class header instead of requiring `--mode=extended` at each invocation.
 
 ## CLI mapping
@@ -23,6 +31,7 @@ copy racc's internal table arrays, internal method names, native runtime, or gen
 | `-E`, `--embedded` | Embed the Pure Ruby runtime |
 | `-F`, `--frozen` | Accepted; Ibex always emits frozen-string magic comments |
 | `--rbs[=FILE]` | Ibex extension; emit a generated parser signature |
+| `--action-source[=FILE]` | Ibex extension; emit a non-executable Steep shadow of semantic methods |
 | `--warnings=all,error` | Ibex extension; display or promote structured grammar diagnostics |
 | `--line-convert-all`, `-l` | Map header/inner/footer too, or disable all source mapping |
 | `-a` | Generate methods for implicit actions |
@@ -34,6 +43,38 @@ copy racc's internal table arrays, internal method names, native runtime, or gen
 Ibex defaults to `<input>.rb`; racc 1.8.1 was observed to default to `<input>.tab.rb`. Use `-o` for portable scripts.
 By default, semantic actions and `inner` methods report grammar-file lines while `header` and `footer` retain generated-file
 lines. `--line-convert-all` maps every user-code section; `-l` maps none. The mapping is retained through IR JSON resumption.
+For static action checking, combine `--rbs --action-source`; configure the generated `.rbs` and `.actions.rb` in the
+application's Steep target, and run Steep separately. The shadow is check-only input and must not replace or be required by the
+runtime parser.
+
+`migrate-check` never executes semantic actions or `header`/`inner`/`footer` code. `migrate-harness` also only writes source. The
+generated harness is the explicit execution step: after reviewed cases are added, it invokes both generators and executes both
+generated parsers in bounded child processes. This is not a sandbox; use a container or VM for untrusted grammar code. See
+[ADR 0012](decisions/0012-bounded-nonexecuting-analysis.md).
+
+## Compatibility baseline
+
+The repository probes and their exact reproduction commands are recorded in
+[`racc-migration-evidence.md`](racc-migration-evidence.md). That matrix is an
+internal evidence index: it distinguishes self-authored black-box coverage
+from the separately pinned public-gem migration suites and does not turn either
+set into a claim of third-party adoption or performance parity.
+
+The compatibility claims above were checked on 2026-07-22 against racc 1.8.1 using only its public documentation, `racc --help`,
+and black-box execution. Ibex does not inspect racc implementation files or generated source. Self-authored probes compare
+observable results for arithmetic precedence, empty rules, string tokens, `convert`, `no_result_var`, inline actions,
+dangling-else `expect`, error recovery, source-line conversion, and a generated 500-production grammar.
+
+Precedence-resolved conflicts remain visible in Automaton IR but are excluded from the CLI conflict count and `expect`. Error
+recovery probes compare result values and the public `on_error` arguments. These tests skip when the `racc` executable is absent.
+
+Application-defined parser initializers do not have to call `super`; the runtime completes its isolated session state on first
+use while retaining application-owned instance variables. During parsing, the historical `@vstack` and `@racc_vstack` names
+both refer to the live semantic-value stack for read compatibility. They are internal aliases: applications may inspect them
+while tokenizing but must not mutate, replace, or retain them across parser sessions.
+
+An unqualified `ParseError` in a parser method or semantic action resolves through `Runtime::Parser`, so application actions can
+keep their existing explicit rejection path. Structured failures remain available as `Ibex::Runtime::ParseError`.
 
 ## Known differences
 
@@ -43,5 +84,3 @@ lines. `--line-convert-all` maps every user-code section; `-l` maps none. The ma
   unknown lookahead through `on_error` first, then recovers if the callback returns. Declared invalid tokens match in the
   black-box recovery probe.
 - `require "racc/parser"` replacement and previously generated racc parser table compatibility are out of scope.
-
-See [compatibility observations](compat-notes.md) for the tested version and probe set.

data/docs/release-readiness.md

@@ -0,0 +1,221 @@
+---
+title: Release readiness
+description: Outcome-based release evidence, limitations, and the current publication decision.
+---
+
+# v1.0 readiness report
+
+This is the outcome-based v1.0 decision required by the project design. It
+records evidence current through 2026-08-08. Performance measurements used
+Ruby 4.0.0 on `arm64-darwin24`. Passing the implementation checklist is not
+sufficient for a stable release.
+
+## Decision
+
+**HOLD v1.0.** Compatibility behavior, scale construction, semantic-value
+signatures, and the repository compatibility suite have evidence. The exact
+formal public performance rerun is now recorded for the measured revision
+below. The published ten-case error comparison has not received independent
+third-party review. Its machine-readable
+[`R001` status](error-ux-review-status-v1.json) is
+`HOLD / awaiting_independent_review`; the linked status leads to the complete
+rubric, closed record schema, immutable evidence identity, and imported-record
+registry.
+
+<!-- r001-review-status:start -->
+R001: `HOLD` — [`awaiting_independent_review`](error-ux-review-status-v1.json).
+<!-- r001-review-status:end -->
+
+Feature development is not frozen. The open release gate blocks publication of
+v1.0, not feature development. New work remains subject to the feature budgets,
+maturity rules, compatibility checks, and exact-revision release gates
+documented by the project. The current Grammar IR and Automaton IR contracts
+remain closed and validated; before v1.0, incompatible changes are coordinated
+with their writers, readers, schemas, fixtures, and downstream consumers.
+
+The format-v6 batch Red/Green CST has been selected as part of the initial
+Stable v1 API. Its syntax-only incremental layer remains Experimental. Other
+Preview and Experimental features may continue to ship in prereleases without
+promotion; before v1.0, those surfaces can change between prereleases when the
+release notes, migration note, regressions, and maturity evidence are updated.
+The release lock applies to Stable declarations and versioned persisted
+contracts, not to every Preview implementation detail or evidence capture.
+
+## KPI result
+
+| KPI | Result | Evidence |
+|---|---|---|
+| Three public-gem migrations | Pass with documented adapters | Namae, BCDice, and Nokogiri behavior suites below |
+| Hundreds-of-productions scale | Pass | 501 productions, 503 states, 125.008 ms average complete build |
+| Ten-case error UX comparison | Partial | 10/10 snapshots and 8/10 useful repairs are public; independent review is missing |
+| Public generator/runtime performance baseline | Pass at measured revision | Direct formal ten-run result artifact is published below; the historical projection remains retained separately |
+| Semantic-value RBS and typed ratchet | Pass | Generated reduction signatures include declared RHS/LHS types; the 2026-08-08 evidence revision measured whole-library Steep at 88.4% typed |
+| Compatibility suite unbeaten | Pass at the measured revision | Current black-box, self-host, IR, property, and runtime suites are green |
+| Release basis | Pass without publication | Stable declaration diff is 0 across 48 locked files; both gems build byte-identically under two environments |
+
+## Public-gem migration evidence
+
+Only public grammar files, public commands, and observable test results were
+used. Neither racc implementation files nor generated source were inspected.
+Each repository was cloned at the recorded revision:
+
+| Project | Revision and grammar | Static check | Behavior result |
+|---|---|---|---|
+| Namae | [`d33875a`](https://github.com/berkmancenter/namae/commit/d33875aaf1fc420a8dfe946a3b29cc3e19710061), `lib/namae/parser.y` | compatible | 151 RSpec examples, 85 Cucumber scenarios, 198 steps |
+| BCDice | [`21b4a03`](https://github.com/bcdice/BCDice/commit/21b4a03789bf2080ad41aaf31299b609ee7bda86), `lib/bcdice/command/parser.y` | compatible; runtime-constant warning | 25 tests, 103 assertions |
+| Nokogiri | [`04a4c29`](https://github.com/sparklemotion/nokogiri/commit/04a4c29c6a605ad40a78f4ce343ced0832a1805c), `lib/nokogiri/css/parser.y` | compatible; runtime-constant warning | 6 tests, 14 assertions |
+
+Namae generated and ran as one embedded file. The other two grammars contain a
+user header whose superclass is named `Racc::Parser`. Their PoCs preload this
+explicit migration adapter before loading the generated parser:
+
+```ruby
+require "ibex/runtime"
+
+module Racc
+  Parser = Ibex::Runtime::Parser
+end
+```
+
+This is the adapter recommended by `ibex migrate-check`; it does not load or
+execute the racc runtime. The PoCs uncovered and fixed three compatibility
+gaps: an omitted `end` before a user-code section, an application initializer
+that omits `super` plus historical value-stack reads, and an unqualified
+`ParseError`.
+
+## Formal Pure Ruby performance comparison
+
+<!-- comparative-evidence:racc-public-performance-2026-07-31:start -->
+At clean Ibex revision 984d4ae7a1c96db4c71e3077669842b7bbc3b4ea, the readiness
+projection for three pinned workloads on Ruby 4.0.0, arm64-darwin24, with YJIT disabled
+records slower cold generation and new-instance runtime than Racc 1.8.1's Ruby backend;
+the direct formal result artifact is absent, so this is evidence-pending and non-publishable
+as a comparative claim.
+
+Evidence record ID: `racc-public-performance-2026-07-31`. Its complete scope,
+evidence paths, missing artifact, and limitations are registered in
+[`claims.yml`](claims.yml) under the [comparison policy](comparison-policy.md).
+CPU model, kernel release, and processor count were not retained in this
+readiness projection. Host CPU and host OS were also not recorded as separate
+fields. The measurements below do not describe the current checkout.
+
+The historical readiness projection says the measurement was collected on
+2026-07-31 from clean revision
+`984d4ae7a1c96db4c71e3077669842b7bbc3b4ea` with ten
+alternating isolated runs per implementation, 50 warm-up workloads, 250
+measured workloads, and 10,000 bootstrap samples. Each workload parses the
+five fixed public inputs. Ruby 4.0.0 ran with YJIT disabled; Racc 1.8.1 was
+forced to its Ruby backend and every worker verified the selected runtime.
+Ratios below are Ibex divided by Racc; time cells include the bootstrap 95%
+interval. The harness's `target_met` field marks absolute parity when the
+interval's upper bound is no greater than 1.0. That field is diagnostic:
+v1.0 requires publication of measurements and their conditions, while a
+future performance gate may reject only a relative regression measured under
+an identical environment and configuration.
+
+| Project | Cold generation time | Reuse time | Reuse allocations | New-instance time | New-instance allocations | Generated bytes |
+|---|---:|---:|---:|---:|---:|---:|
+| Namae | 1.238x [1.184, 1.272] | 0.985x [0.966, 1.032] | 0.849x | 1.194x [1.157, 1.248] | 1.190x | 0.862x |
+| BCDice command | 1.329x [1.278, 1.420] | 1.052x [1.008, 1.075] | 0.666x | 1.373x [1.326, 1.415] | 1.103x | 0.981x |
+| Nokogiri CSS | 1.309x [1.277, 1.343] | 0.985x [0.963, 1.029] | 0.662x | 1.186x [1.149, 1.255] | 1.051x | 0.937x |
+
+Result values and ordered result sequences are equivalent in every runtime
+row. The measured parser-time medians in milliseconds per parse were
+0.039443/0.040048 (Ibex/Racc) for Namae reuse,
+0.020925/0.019897 for BCDice reuse, and
+0.024892/0.025280 for Nokogiri reuse. The corresponding new-instance medians
+were 0.048234/0.040413, 0.027523/0.020044, and 0.030420/0.025642.
+
+The results deliberately retain the unfavorable rows: cold generation is
+23.8–32.9% slower, and new-instance parsing is 18.6–37.3% slower. Reuse point
+estimates range from 1.5% faster to 5.2% slower. Ibex allocates fewer objects
+for every reuse workload, while new-instance allocation ratios remain
+1.051–1.190x. Generated output is smaller for all three grammars. These values
+are a release baseline, not portable scores or an assertion of parity.
+<!-- comparative-evidence:racc-public-performance-2026-07-31:end -->
+
+## Exact-revision formal rerun
+
+<!-- comparative-claim:racc-public-performance-2026-08-08:start -->
+At clean Ibex revision 26e94ce631bf5c075ecbca0205a34a8c231c31d9, on Ruby 4.0.0,
+arm64-darwin24, with YJIT disabled, the direct formal comparison of the pinned
+Namae, BCDice command, and Nokogiri CSS workloads against Racc 1.8.1 produced
+equivalent result sequences; Ibex recorded slower cold generation and new-instance
+parsing, fewer reuse allocations, and smaller generated output in every workload.
+
+| Project | Cold generation ratio | Reuse time ratio | Reuse allocations ratio | New-instance time ratio | New-instance allocations ratio | Generated bytes ratio |
+|---|---:|---:|---:|---:|---:|---:|
+| Namae | 1.317830 | 0.996320 | 0.854462 | 1.209687 | 1.207559 | 0.861519 |
+| BCDice command | 1.309690 | 1.010467 | 0.675012 | 1.331780 | 1.129299 | 0.981449 |
+| Nokogiri CSS | 1.288118 | 0.994802 | 0.670484 | 1.193522 | 1.074838 | 0.936806 |
+
+The complete formal result is [`2026-08-08-8c9cef999d09-ruby-4.0.0-arm64-darwin24.json`](../benchmark/results/public/2026-08-08-8c9cef999d09-ruby-4.0.0-arm64-darwin24.json). It records ten alternating isolated runs per implementation and scenario, 50 warm-up workloads, 250 measured workloads, five behavior probes, 10,000 bootstrap samples, exact public checkout revisions, and equivalent result and result-sequence digests.
+<!-- comparative-claim:racc-public-performance-2026-08-08:end -->
+
+## Scale evidence
+
+Run:
+
+```sh
+benchmark/scale.rb --rules 500 --iterations 3
+```
+
+The reproducible synthetic chain shares its grammar shape with the 500-rule
+black-box compatibility test. The measured report contains 501 productions,
+503 direct-LALR construction states, 503 final states, no conflicts, and
+41,332 bytes of compact generated Ruby. Three complete parse-to-codegen builds
+averaged 125.008 ms with a 122.742–127.002 ms range. The Automaton IR digest
+was `fe0e2f0c9bf16e7aa22a7ebcea40b229fedbb573f83af37596b76e0b989c3338`.
+
+This synthetic result establishes the requested scale bound. The 139-production
+representative grammar remains the realistic workload, so neither result is
+presented as a substitute for an application-specific benchmark.
+
+## Error UX and type evidence
+
+[`error-ux.md`](error-ux.md) and its versioned JSON publish the same ten
+malformed JSON inputs against the public racc callback. Eight selected repairs
+were assessed useful. A maintainer assessment is not independent review; at
+least one external reviewer must record a review of the cases and usefulness
+labels before this KPI passes. Reviewers use the versioned
+[`independent rubric`](error-ux-review-rubric-v1.md), generate a draft with the
+exact checkout and environment identity, set all structured consent fields, and
+publish the final permalink-free payload at a full-SHA blob URL. Maintainers
+import the exact bytes and register publication provenance separately, without
+rewriting. Ordinary CI validates the offline kit; the release gate performs
+fail-closed blob checks and requires the source owner, GitHub API author,
+publisher, and reviewer logins to agree. This establishes only GitHub namespace
+control and account metadata, not cryptographic identity or a signature. The
+gate remains intentionally failing while the public status is HOLD.
+
+The 2026-07-31 evidence snapshot recorded 23,517 typed calls and 2,946
+untyped calls out of 26,463, or 88.9% typed. Generated parser RBS refines
+declared terminal, nonterminal, RHS tuple, and reduction result types. Untyped
+values remain explicit at undeclared grammar symbols, decoded JSON, dynamic
+table cells, and opaque application Ruby boundaries. The current repository
+result is maintained in the generated type-statistics block in `README.md`.
+
+## Release-basis evidence
+
+`bundle exec rake release:reproducible` compares 47 normalized Stable RBS
+declaration files with the v0.2.0 baseline. The current result has zero
+differences; the installed-Gem embedded-source helper is an internal packaging
+implementation and is outside that public lock. The task also builds both
+gems twice under distinct locale, timezone, and frozen-string settings and
+requires byte-identical artifacts. SHA-256 digests are emitted for the exact
+candidate revision rather than recorded as if the current HOLD revision were
+a release.
+
+No release signing credential is stored in this repository. A signature or
+provenance attestation must be supplied by a protected external release
+environment after the outcome gates pass; an unsigned local test key would not
+establish publisher identity.
+
+## Actions required to release v1.0
+
+1. Obtain and publish an independent review of the ten error cases using the
+   R001 rubric, then import the record byte-for-byte with its immutable
+   permalink and consent. Preserve disagreements separately from the normative
+   maintainer assessment.
+2. Re-run every compatibility, IR, type, benchmark, and site gate on the exact
+   release revision.

data/docs/repair-semantics.md

@@ -0,0 +1,123 @@
+# Existing bounded repair semantics
+
+Ibex bounded repair is an opt-in Experimental runtime facility. Assign an
+immutable `Ibex::Runtime::RepairPolicy` before starting a parser session. With
+no policy, parsing does not search or prefetch for repairs and existing error
+recovery is unchanged.
+
+## Search state and bounds
+
+`RepairSearch` runs Dijkstra search over configurations containing:
+
+- a copied LR state stack;
+- the buffered-input index;
+- consecutive successful shifts;
+- accumulated edit cost and ordered edits;
+- whether that configuration has reached the proof goal.
+
+Insertion, deletion, and replacement are the only edits. The default costs are
+1, 1, and 2, with maximum cost 3. Search also has independent defaults of 5,000
+simulated configurations, 8 lookahead tokens, 3 proof shifts, and stack depth
+256. Every policy number is positive and proof shifts cannot exceed lookahead.
+
+The queue orders configurations lexicographically by:
+
+1. total cost;
+2. semantic-value risk;
+3. ordered edit keys `(input position, delete|insert|replace rank, token id)`;
+4. goal before non-goal;
+5. more successful shifts;
+6. earlier buffered-input index;
+7. LR state stack.
+
+Word-like token names add semantic-value risk; punctuation does not. This is a
+deterministic tie-break, not a claim that punctuation edits are correct.
+Candidate inserted/replacement token IDs are ordered and exclude EOF and the
+reserved `error` token. A plan succeeds after the configured consecutive
+shifts or accept.
+
+Search simulates table actions and reductions on integer state stacks only. It
+does not call semantic actions, `on_error`, `on_repair`, lexer actions, or
+runtime observers. Reduction-stack cycles, excess stack depth, excess cost,
+and excess configuration count stop a path. Pull parsing may read ahead to the
+lookahead bound. An incomplete push prefix yields the internal `NEED_INPUT`
+sentinel and waits for more input.
+
+The compatibility `RepairSearch#search` projection still returns the legacy
+`RepairPlan | NEED_INPUT | nil` shape. The additive `search_result` method
+retains a frozen typed outcome (`:selected`, `:need_input`, `:exhausted`, or
+`:not_found`) for syntax tooling, so bounded exhaustion is not confused with a
+complete search that found no plan.
+
+## Selection, hooks, and replay
+
+On a reported syntax error, the runtime captures the error context and CST
+error, publishes the runtime `error` event, and calls `on_error`. If search
+selected a plan, `on_repair(plan)` is called next. If either hook raises, its
+exception propagates before edited tokens are applied.
+
+After both hooks return, the runtime rebuilds the buffered token prefix:
+
+| Edit | Replay token value | Replay location | CST effect |
+| --- | --- | --- | --- |
+| insert | `nil` | location at the insertion point | missing token |
+| delete | absent from replay | original source retained separately | skipped/error trivia |
+| replace | original token value | original token location | error-marked replacement |
+
+The rebuilt prefix then passes through ordinary shift/reduce/accept handling.
+Committed semantic actions and observers therefore run normally; speculative
+search never runs them. Replay emits normal parser events and does not invent a
+`recover` event for the selected repair. The original syntax error remains a
+diagnostic even if replay accepts.
+
+Inserted `nil` and a replacement's retained value are compatibility mechanics,
+not evidence of user intent. A syntactically accepted repair can still be
+semantically unsafe.
+
+## Syntax-only editor proposals
+
+`SyntaxSession#repair` is a separate Experimental boundary for callers that
+need byte edits rather than semantic replay. It runs the same bounded search
+on a fresh private parser, captures absolute source ranges, resolves source
+spelling only from caller-provided text or conservative punctuation literals,
+and validates the applied bytes with a fresh syntax session. Production actions
+and the application's `on_repair` callback do not run, and the result has no
+semantic `value`. The originating session remains unchanged.
+
+`SyntaxRepairResult` distinguishes `:accepted`, `:progress`, and `:rejected`
+from bounded `:exhausted` and `:not_found` outcomes and selected-but-unavailable
+cases such as missing token spelling or multiple repair segments. Accepted
+results require a diagnostic-free fresh parse consuming all edited bytes. For
+error results, `updated_source` keeps the exact bytes while the CST may expose
+only the consumed prefix under the existing error CST contract. See [E002
+syntax-only repair](investigations/E002-syntax-only-repair-result.md) and
+[ADR 0023](decisions/0023-syntax-only-repair-results.md).
+
+## CST and source ownership
+
+CST parsing retains the original source bytes. Insertions are represented by
+missing nodes, replacements carry an error flag, and deleted token bytes are
+preserved as skipped-token trivia rather than silently disappearing. These
+representations are part of the existing CST error-path contract, not a text
+edit API. `RepairPlan#position` is a buffered token index, not a source-byte
+offset.
+
+## Executable coverage
+
+<!-- repair-semantics:coverage:start -->
+| Concept | Executable evidence |
+| --- | --- |
+| policy-and-default-off | `test/runtime/repair_test.rb` |
+| dijkstra-priority-and-outcomes | `test/runtime/repair_characterization_test.rb` |
+| typed-search-outcomes | `test/runtime/repair_characterization_test.rb` |
+| values-locations-and-replay | `test/runtime/repair_test.rb` |
+| hooks-actions-and-observers | `test/runtime/repair_test.rb` |
+| pull-and-push-bounds | `test/runtime/repair_test.rb` |
+| cst-representation | `test/codegen/cst_test.rb` |
+| cst-source-fidelity | `test/runtime/cst_fidelity_property_test.rb` |
+| syntax-only-repair-projection | `test/runtime/syntax_repair_test.rb` |
+<!-- repair-semantics:coverage:end -->
+
+The H003 [round-two evidence](error-ux-round2.md) records exact selected plans
+and fresh reparses but keeps external usefulness judgments on HOLD. Neither
+that evidence nor this characterization promotes bounded repair to Stable.