ibex 0.1.0 → 0.3.0

data/docs/error-ux-round2.md

@@ -0,0 +1,111 @@
+# Error UX evidence round 2
+
+H003 expands repository-owned diagnostic and repair evidence beyond R001's ten
+JSON cases. The machine-readable result is
+[`error-ux-round2-v1.json`](error-ux-round2-v1.json), validated by
+[`error-ux-round2-v1.schema.json`](../schema/error-ux-round2-v1.schema.json).
+It does not replace or revise the R001 snapshot, rubric, review records, or
+release gate.
+
+## Status and authority
+
+The deterministic repository capture is **complete**. The external subjective
+gate is separately **HOLD**: there are no external review records, and every
+case has `status: pending` with an empty label list. Repository observations
+cannot be promoted into external `useful`, `misleading`, `unsafe`, or `unclear`
+labels.
+
+The immutable machine capture intentionally has no slot for later subjective
+labels. H003's separate
+[`review status registry`](error-ux-round2-review-status-v1.json) and
+[`review schema`](../schema/error-ux-round2-review-v1.schema.json) provide that
+pathway without rewriting the capture. The registry binds the exact capture
+SHA-256 and required case inventory. It remains `HOLD` with `records: []`.
+
+The fixed corpus covers these distinct dimensions:
+
+| Case | Shape | Observation | Fresh reparse |
+| --- | --- | --- | --- |
+| `H003-DELIMITER-01` | nested delimiter-heavy calls | mismatched closer | accepted |
+| `H003-STATEMENT-01` | assignment statement | missing semantic value | accepted |
+| `H003-STATEFUL-01` | stateful string lexer | EOF in the string state | accepted |
+| `H003-EOF-01` | delimiter EOF | missing closer | progress; the proposed edit over-closes |
+| `H003-MULTI-01` | synchronized statements | two errors at distinct offsets, then continuation | accepted |
+| `H003-UNKNOWN-01` | external token adapter | unknown token after a complete word | accepted |
+| `H003-LEXER-01` | generated lexer | no matching lexer rule | accepted |
+
+Indentation-sensitive input is explicitly `excluded-unimplemented`. Ibex has
+no indentation-tokenization contract, so the corpus does not simulate one.
+
+## Capture semantics
+
+Each case binds the grammar and input digests, exact runtime expected tokens,
+full diagnostic message, source line and caret, bounded repair result, a
+repository-authored byte-range edit, the outcome of applying that edit and
+parsing with a fresh parser, semantic-value risk, and external-review state.
+The capture also binds the exact frontend, normalization, code generator,
+runtime parser, generated lexer, repair implementation, and H003 generator
+source bytes used to produce those observations.
+
+Parser expected-token arrays come directly from the existing runtime API. A
+generated-lexer failure occurs before a parser state exists, so its exact array
+is empty with the closed reason
+`lexer-failure-precedes-parser-state`; it is not presented as a parser
+prediction. Runtime repairs remain distinct from proposed source edits.
+Insertion can supply nil, replacement retains the original runtime value, and
+deletion can discard intent. A successful fresh parse establishes syntax only,
+not that an invented literal, deleted token, or replacement matches the user's
+intended semantics.
+
+The multi-error case overrides the normal error hook only to record the same
+runtime token, expected-token, state, and location fields while allowing the
+grammar's explicit `error ';'` synchronization path to continue. The committed
+capture requires at least two diagnostics at strictly increasing byte offsets
+and a non-nil result after recovery; it cannot be satisfied by duplicating one
+error.
+
+## Trust boundary
+
+The capture compiles and executes generated lexer and parser action code.
+Execution is allowlisted to the four repository-owned fixtures under
+`test/fixtures/error_ux_round2/`. It does not download or execute external
+grammars, reviewer payloads, or user-supplied code. These fixtures are trusted
+test programs, not a sandbox claim.
+
+The existing R001 normative snapshot is bound to its unchanged SHA-256. H003
+generation fails if those bytes change. H003 uses its own corpus, schema,
+evidence ID, generator, and quality gate so comparative R001 evidence and this
+repository study cannot silently become duplicate authorities.
+
+## Limits and kill conditions
+
+This evidence has no external workloads or reviewers, observes one bounded
+repair plan rather than every equal-cost plan, and does not infer intent from
+acceptance. It must fail closed when fixture/corpus bytes drift, a required
+dimension disappears, R001 changes, review state is overstated, synchronized
+diagnostics collapse, or the committed fresh-reparse outcome changes.
+
+The same quality gate validates the external registry fail-closed. `PASS`
+requires at least two complete, independent, external records from
+reviewers whose names remain distinct after Unicode NFKC normalization, full
+case folding, trimming, and whitespace collapse. Every record must assess all seven cases with a
+label, rationale, and semantic-value risk assessment. If reviewer labels
+differ, every reviewer-pair disagreement must remain explicit rather than
+being reduced to a majority conclusion. See the
+[`records workflow`](error-ux-round2-reviews/v1/records/README.md).
+
+Independence and identity remain human attestations: schema validation can
+require the declarations and reject duplicate normalized names, but cannot
+prove who authored a review. A record must never be added merely to clear the
+gate. Evidence drift, incomplete case coverage, duplicate reviewer identity,
+or missing disagreement entries kills `PASS`.
+
+Regenerate only after reviewing intentional parser, lexer, diagnostic, or
+repair changes:
+
+```sh
+bundle exec ruby tool/error_ux_round2.rb --write
+bundle exec rake quality:error_ux_round2
+```
+
+The ordinary quality command never writes evidence.

data/docs/error-ux.md

@@ -0,0 +1,114 @@
+---
+title: Error experience evidence
+description: Bounded diagnostics, repair behavior, and the evidence boundary for parser errors.
+---
+
+# JSON error UX evidence
+
+This report fixes the Phase 14 error-experience baseline against ten malformed
+inputs for the gallery JSON grammar. It is generated from
+[`json-errors-v1.json`](../test/fixtures/error_ux/json-errors-v1.json) by:
+
+```sh
+bundle exec ruby tool/error_ux_snapshot.rb
+```
+
+Use `--write` only after reviewing a deliberate diagnostic, parser-table, racc,
+or repair-policy change. CI regenerates the evidence and requires byte-for-byte
+equality.
+
+## Independent review gate
+
+The fixed observations include a maintainer repair assessment. They do not
+include an independent diagnostic or repair assessment.
+
+<!-- r001-review-status:start -->
+R001: `HOLD` — [`awaiting_independent_review`](error-ux-review-status-v1.json).
+<!-- r001-review-status:end -->
+
+Third-party reviewers use the versioned
+[`rubric`](error-ux-review-rubric-v1.md) and closed
+[`review-record schema`](../schema/error-ux-review-v1.schema.json). Run:
+
+```sh
+bundle exec rake quality:error_ux_review_kit
+bundle exec rake quality:error_ux_review_status
+bundle exec ruby tool/error_ux_review.rb template review.json
+```
+
+The generated payload contains the reviewer identity, reviewed maintainer
+roster, structured consent, labels, rationales, and disagreements, but no
+permalink. Publication metadata is kept separately in the status registry so a
+reviewer can commit the final payload without a self-referential SHA. The first
+command verifies the immutable snapshot, corpus identities, schema, local
+byte/digest bindings, reports, claims, and truthful status entirely offline
+while allowing the expected HOLD in ordinary CI.
+
+`bundle exec rake release:error_ux_review` is the separate promotion gate and
+returns nonzero until a valid published external payload exists. When PASS is
+claimed, it additionally fetches the full-SHA blob without redirects, compares
+the exact imported bytes, and requires the source repository owner, GitHub
+commit API author, status publisher login, and reviewer login to agree
+case-insensitively. This verifies control of the named GitHub namespace and
+GitHub account metadata, not a cryptographic identity or signature. Network or
+account-metadata uncertainty fails closed. Neither path changes the normative
+observation fixture.
+
+## Comparison method
+
+<!-- comparative-evidence:racc-error-ux-json-v1:start -->
+At Ibex revision cc20c5eb799cc218ebea665df64261f10d030f75, on ten fixed malformed
+JSON inputs with the Ruby, OS, CPU, kernel, processor-count, and YJIT environment unrecorded,
+the committed result artifact records Ibex diagnostics,
+Racc 1.8.1's public on_error callback values, and a maintainer assessment of ten bounded repairs;
+independent subjective review is pending, so this is not a completed comparative UX claim.
+
+Evidence record ID: `racc-error-ux-json-v1`. The observations and assessment
+are registered in [`claims.yml`](claims.yml), including every unrecorded
+environment field and the still-pending independent review.
+
+Ibex uses [`examples/json.y`](../examples/json.y), including its generated
+lexer locations. The migration from its previous handwritten lexer is guarded
+by exact location and error-snapshot tests. The comparison uses the self-authored compatible grammar
+[`json_racc.y`](../test/fixtures/error_ux/json_racc.y), racc 1.8.1, and only the
+public `racc` executable and `on_error(token, value, value_stack)` callback.
+Neither racc implementation files nor generated source are inspected. Both
+parsers receive the same source strings and equivalent token streams.
+
+The racc column records the neutral callback wrapper's token argument. It does
+not claim that an application cannot build richer diagnostics around racc; it
+shows what the compared parser runtime exposes at that boundary without an
+additional state-to-message layer.
+
+| Case | Invalid source | Ibex diagnostic evidence | racc callback | Selected repair | Useful |
+|---|---|---|---|---|---|
+| EUX-01 | `{"a":,}` | line 1, column 6; 7 expected values; caret | token 9 | delete `,`, insert `STRING` | yes |
+| EUX-02 | `{"a" 1}` | line 1, column 6; expected `:`; caret | token 3 | insert `:` | yes |
+| EUX-03 | `{"a":1,}` | line 1, column 8; expected `STRING`; caret | token 8 | insert a complete member | no |
+| EUX-04 | `[1,]` | line 1, column 4; 7 expected values; caret | token 12 | insert `STRING` | no |
+| EUX-05 | `[1 2]` | line 1, column 4; expected `,` or `]`; caret | token 3 | insert `,` | yes |
+| EUX-06 | `{"a":[true false]}` | line 1, column 12; expected `,` or `]`; caret | token 5 | insert `,` | yes |
+| EUX-07 | `{"a":null "b":2}` | line 1, column 11; expected `}` or `,`; caret | token 2 | insert `,` | yes |
+| EUX-08 | `{]` | line 1, column 2; expected `STRING` or `}`; caret | token 12 | replace `]` with `}` | yes |
+| EUX-09 | `[}` | line 1, column 2; 8 expected values/closer; caret | token 8 | replace `}` with `]` | yes |
+| EUX-10 | `true false` | line 1, column 6; expected EOF; caret | token 5 | delete extra `FALSE` | yes |
+
+The committed JSON is the normative snapshot and also retains exact messages,
+LR states, expected-token arrays, token values, repair costs, configuration
+counts, and edit positions. Its public structure is validated by
+[`schema/error-ux-v1.schema.json`](../schema/error-ux-v1.schema.json).
+
+## SP-4 decision
+
+The bounded insertion/deletion/replacement search produced a completing plan
+for all ten cases. Eight plans are reasonable representations of the likely
+human edit; two trailing-comma cases invent a missing value/member where
+deleting the comma would usually be preferable. The measured useful rate is
+therefore **8/10 (80%)**, above the required majority.
+
+SP-4 is **go** for the existing opt-in bounded single-plan repair described by
+[ADR 0012](decisions/0012-bounded-nonexecuting-analysis.md). It remains
+experimental: a selected edit can require a nil semantic value, and the search
+does not enumerate every equal-cost CPCT+ repair. The stable default answer
+remains exact expected tokens plus explicit yacc/synchronization recovery.
+<!-- comparative-evidence:racc-error-ux-json-v1:end -->

data/docs/getting-started.md

@@ -0,0 +1,80 @@
+---
+title: Getting started
+description: Install Ibex, generate a first parser, and choose a grammar mode.
+---
+
+# Getting started
+
+Ibex turns a racc-compatible grammar into a Pure Ruby LR parser. This guide
+gets from an installed gem to a working parser before explaining the optional
+extended surface.
+
+## Install the generator
+
+```sh
+gem install ibex
+```
+
+For an application that only runs generated parsers, install `ibex-runtime`
+instead. A project can add either dependency with `bundle add ibex` or
+`bundle add ibex-runtime`.
+
+## Generate a calculator
+
+Create `calculator.y` with a class, token declaration, rules, and a small
+handwritten pull lexer:
+
+```text
+class Calculator
+token NUM
+preclow
+  left '+'
+  left '*'
+prechigh
+rule
+  expr : expr '+' expr { result = val[0] + val[2] }
+       | expr '*' expr { result = val[0] * val[2] }
+       | NUM { result = val[0] }
+end
+---- inner
+def parse_tokens(tokens)
+  @tokens = tokens
+  do_parse
+end
+
+def next_token
+  @tokens.shift
+end
+---- footer
+if $PROGRAM_NAME == __FILE__
+  tokens = [[:NUM, 2], ['+', nil], [:NUM, 3], ['*', nil], [:NUM, 4]]
+  puts Calculator.new.parse_tokens(tokens)
+end
+```
+
+Generate and run it:
+
+```sh
+ibex -o calculator.rb calculator.y
+ruby calculator.rb
+# 14
+```
+
+Run `ibex --check -o calculator.rb calculator.y` in CI. It compares the
+reproducible output without rewriting the committed file.
+
+## Choose a mode
+
+The default mode is the conservative racc migration surface. Add
+`pragma extended` immediately after the class header, or pass
+`--mode=extended`, when the grammar needs EBNF groups, imports, generated
+lexers, typed trees, or other opt-in features. The [grammar reference](grammar-reference.md)
+lists the exact syntax and the [maturity audit](maturity.md) records support
+level per API.
+
+## Next steps
+
+- [Migrate an existing racc grammar](racc-migration.md).
+- [Configure grammar-owned and invocation-owned settings](configuration-model.md).
+- [Inspect a grammar locally in the Playground](https://ydah.github.io/ibex/playground/).
+- [Read the runtime and grammar reference](grammar-reference.md).