ibex 0.1.0 → 0.2.0

data/docs/cst.md

@@ -0,0 +1,213 @@
+# Red/Green concrete syntax trees
+
+`pragma cst` generates a lossless, error-tolerant syntax tree alongside the
+ordinary semantic result. Regenerate the parser with the current generator to
+receive parser-table format v6 and this API. Batch parsing, typed views,
+editing, diffing, and `ibex_cst` serialization are Stable for v1; syntax-only
+incremental sessions are Experimental.
+
+```text
+class Calculator
+pragma cst
+token NUM PLUS
+lexer
+  skip /[[:space:]]+/
+  NUM /[0-9]+/ { lexeme.to_i }
+  PLUS '+'
+end
+rule
+start: expression
+expression: NUM PLUS NUM @node Addition(left, operator, right)
+end
+```
+
+## Parsing and tree shape
+
+`parse_with_syntax` runs semantic actions normally and returns both results:
+
+```ruby
+result = Calculator.new.parse_with_syntax("1 + 2", file: "input.txt")
+value = result.value
+root = result.syntax_root
+diagnostics = result.diagnostics
+```
+
+`parse`, `do_parse`, and `yyparse` keep their semantic return values.
+`parser.syntax_root` exposes the most recent syntax root. Every successful root
+has this physical shape:
+
+```text
+source_file
+├── selected start-symbol node
+└── $eof
+```
+
+The immutable Green layer stores kind ids, source bytes, trivia, flags, widths,
+and children without parents or absolute positions. It is Ractor-shareable.
+The Red `SyntaxNode`/`SyntaxToken` layer adds lazy parent, child-index, offset,
+span, and location navigation for one occurrence. Equal Green values can
+therefore appear at several positions without sharing Red coordinates.
+
+`root.to_source` reconstructs the consumed bytes exactly. Early `yyaccept`
+marks `root.incomplete_input?` and reconstructs the consumed prefix.
+
+## Trivia and coordinates
+
+Select trivia ownership while generating:
+
+- `leading` owns skipped text on the following token;
+- `balanced` owns text through the first newline on the preceding token and
+  the remainder on the following token;
+- `drop` omits trivia.
+
+Pass the policy explicitly when needed:
+
+```sh
+ibex --cst-trivia=leading grammar.y
+ibex --cst-trivia=balanced grammar.y
+ibex --cst-trivia=drop grammar.y
+```
+
+`attach` is accepted as an alias for `leading`. EOF owns final leading trivia.
+`drop` trees deliberately reject span, location, and incremental APIs because
+their offsets cannot describe the original source.
+
+Spans are zero-based, half-open byte ranges. `SourceText#position` and
+`#location` convert them to one-based Unicode-scalar lines and columns while
+preserving byte offsets.
+
+## Typed syntax views
+
+An `@node` annotation generates a view in `Parser::Syntax` without changing the
+physical tree:
+
+```ruby
+addition = Calculator::Syntax::Addition.cast(
+  result.syntax_root.children.fetch(0).children.fetch(0)
+)
+addition.left
+addition.operator
+addition.right
+```
+
+Accessors use normalized physical RHS slots. Lowered repetitions add element
+enumerators; separated lists also expose separators. These views are distinct
+from generated `Parser::AST` Data values: syntax views retain punctuation and
+trivia, while AST values are semantic-action results.
+
+## Errors and repair
+
+Lexical failure, yacc recovery, panic discard, and bounded repair all produce a
+tree. Inspect `diagnostics`, `contains_error?`, error nodes, missing tokens, and
+Green flags. Inserted tokens are zero-width `missing_token` values. Discarded
+input remains under error nodes or `skipped_tokens` trivia. An unrecoverable
+parse uses `synthetic_root`.
+
+## Persistent editing and diffing
+
+Editing returns a new Red root and path-copies only changed ancestors:
+
+```ruby
+old_root = result.syntax_root
+first = old_root.first_token
+new_root = first.with_text("10")
+edits = Ibex::Runtime::CST::Diff.text_edits(old_root, new_root)
+new_source = old_root.source_text.apply(edits)
+```
+
+Nodes support `replace_with`, `with_child`, `insert_child`, and `remove_child`.
+Tokens support `replace_with`, `with_text`, `with_leading`, and
+`with_trailing`. `SyntaxRewriter` performs bottom-up kind dispatch;
+`SyntaxEditor` applies several occurrence-addressed replacements in one pass.
+`SyntaxAnnotation` marks one occurrence and excludes annotated Green values
+from interning.
+
+## Incremental syntax sessions
+
+Incremental parsing is Experimental and deliberately syntax-only:
+
+```ruby
+source = Ibex::Runtime::CST::SourceText.new("1 + 2", file: "input.txt")
+session = Calculator.incremental_session(source)
+result = session.edit([
+  Ibex::Runtime::CST::TextEdit.new(
+    start: 4,
+    delete_length: 1,
+    insert_text: "3"
+  )
+])
+```
+
+The initial parse and every edit suppress parser production actions.
+`SyntaxResult` therefore has `syntax_root`, `diagnostics`, and `reused_ratio`,
+but no semantic value. Generated lexer actions still run because they define
+token emission and lexer-state changes. Run a normal parse separately when a
+semantic value or side effect is required.
+
+Sessions require `SourceText`, a generated lexer, format-v6 CST metadata, and
+non-`drop` trivia. Handwritten token sources and push input are unsupported.
+Parser actions that alone switch lexer state are also unavailable because the
+syntax-only contract forbids executing them.
+
+The default Stage B path relexes soundly from the beginning, validates old
+token boundaries/states, and directly reuses only subtrees whose LR state,
+follow token, flags, width, and damage range are safe. Pass `blender: false` to
+retain the Stage A full LR drive. `ResourceLimits` bounds decomposition and
+memo bytes; a bound failure deterministically falls back to the fresh token
+stream. Observe `cst_built`, `cst_reuse`, and `cst_fallback` runtime events for
+metrics.
+
+## Serialization
+
+`ibex_cst` schema v1 is independent from Grammar IR:
+
+```ruby
+tables = Calculator.parser_tables
+json = Ibex::Runtime::CST::Serialize.dump(
+  session.result.syntax_root,
+  grammar_digest: tables.fetch(:grammar_digest),
+  table_format: tables.fetch(:format_version),
+  state_count: tables.fetch(:state_count),
+  production_count: tables.fetch(:production_count),
+  memo: session.parse_memo
+)
+loaded = Ibex::Runtime::CST::Serialize.load(
+  json,
+  grammar_digest: tables.fetch(:grammar_digest),
+  state_count: tables.fetch(:state_count),
+  production_count: tables.fetch(:production_count)
+)
+```
+
+Valid UTF-8 bytes use JSON strings; other bytes use canonical Base64 objects.
+Loading rebuilds widths, aggregate flags, and descendant counts. A grammar
+digest mismatch raises `ValidationError`. State or production count mismatch
+silently discards only the optional parse memo. Annotations are session-local
+and cannot be serialized.
+
+## Invariants and test coverage
+
+The Red/Green contract is continuously exercised by the following repository
+tests:
+
+| Invariant | Automated evidence |
+| --- | --- |
+| I1 consumed-input fidelity | `test/runtime/cst_fidelity_property_test.rb` is PT1 over random grammars and valid, lexical-error, recovery, repair, and early-accept inputs; `test/fixtures/cst/runtime-paths-v1.json` fixes every error/trivia path, including long unmatched remainders and repair deletion. |
+| I2 Green purity | `test/runtime/cst_green_test.rb` checks derived-only immutable state and Ractor shareability; `test/codegen/cst_characterization_test.rb` keeps semantic values outside syntax children. |
+| I3 lazy Red navigation | `test/runtime/cst_red_test.rb` checks occurrence-local memoization, offsets, parents, cursors, and deterministic traversal. |
+| I4 determinism | PT3 in `test/runtime/cst_green_test.rb` compares cache-on/off structure, source, and dump; PT6 in `test/runtime/cst_serialize_test.rb` checks byte-stable dump/load/dump; generated Ruby and RBS have golden tests. |
+| I5 unchanged action contract | `test/codegen/cst_contract_test.rb` compares CST/no-CST `parse`, `do_parse`, `yyparse`, hooks, observer order, and non-CST table shape; characterization and obsolete-CST-table rejection regressions remain active. |
+| I6 sharing boundary | PT5 in `test/runtime/cst_green_test.rb` checks Green shareability and concurrent Ractor reads; editing and parser-table tests keep Red/session state occurrence-owned. |
+| I7 versioned contracts | `test/runtime/cst_serialize_test.rb`, `test/runtime/table_format_test.rb`, and `test/packaging/schema_files_test.rb` cover schema validation, memo compatibility, old table readers, and schema packaging. |
+| I8 type soundness | CI regenerates the `sig/` tree, runs Steep and RBS validation, and rejects a type-statistics regression. |
+
+`test/runtime/cst_incremental_property_test.rb` is the large PT2 authority. Its
+fixed seed generates four grammar shapes and executes 20,000 random edits
+through Stage A plus 20,000 through Stage B, comparing fresh syntax sessions in
+Green structure, bytes, flags, diagnostics, and memo cardinality after every
+edit. `test/runtime/cst_incremental_test.rb` adds focused stateful-lexer,
+fallback, sharing, minimal-diff, action-suppression, no-value, and unsupported
+configuration cases.
+
+See [the migration guide](cst-migration.md) for removed tree-shape changes and
+[the stability policy](stability.md) for support levels.

data/docs/development.md

@@ -0,0 +1,129 @@
+# Development and quality checks
+
+This document describes contributor workflow and repository quality gates.
+These practices verify the implementation but are not implementation
+architecture decisions.
+
+## Default checks
+
+Use Ruby 3.4 or newer, install the complete development toolchain once, and run
+the unit, integration, documentation, and style suite:
+
+```sh
+bundle install
+bundle exec rake
+```
+
+Additional repository checks are:
+
+```sh
+bundle exec rake frontend:check
+bundle exec rake grammar:test
+bundle exec rake quality:error_ux
+npm ci
+npm run test:site
+actionlint
+zizmor .
+```
+
+CI runs the supported Ruby matrix separately from optional experimental Ruby
+implementations. Workflow and dependency-update configuration are authoritative
+for the current matrix, schedules, permissions, and pinned action versions.
+The matrix uses the intentionally unlocked `gemfiles/compat.Gemfile` so each
+supported Ruby resolves compatible test dependencies. Contributors use the
+root `Gemfile` and its committed lockfile for every development tool.
+
+## Browser site and API documentation
+
+Install the locked browser dependencies and build the same self-hosted site
+bundle exercised in CI:
+
+```sh
+npm ci
+npm run test:site
+bundle exec yard doc
+```
+
+`tool/build_site.rb`, the YARD configuration, and
+`.github/workflows/pages.yml` are authoritative for documentation inputs,
+build versions, artifact retention, deployment permissions, and the protected
+Pages environment. Publication runs from `main`; pull requests validate the
+site without receiving deployment credentials.
+
+Changing those operational settings does not require an implementation ADR.
+A change to the shipped browser analyzer's worker isolation, execution limits,
+or Ruby/JavaScript data boundary does.
+
+## Self-hosted frontend
+
+`lib/ibex/frontend/grammar.y` is the production grammar. Regenerate its
+committed parser after changing the frontend language:
+
+```sh
+bundle exec rake frontend:generate
+bundle exec rake frontend:check
+bundle exec ruby -Itest test/frontend/self_host_test.rb
+```
+
+`lib/ibex/frontend/shadow_grammar.y` independently describes the same language
+using parameterized and inline rules. Tests build it through the bootstrap
+frontend and compare its location-preserving AST with the production parser.
+The shadow source must change with the canonical grammar, but it never replaces
+the committed production parser.
+
+## RBS and Steep
+
+The development bundle includes the type toolchain. Regenerate the committed
+RBS tree, validate it, type-check the library, and refresh the documented
+statistics with:
+
+```sh
+ruby -e '
+  sources = Dir.glob("lib/**/*.rb").sort
+  exec("bundle", "exec", "rbs-inline", "--opt-out", "--base=lib", "--output=sig", *sources)
+'
+bundle exec rbs -r digest -r json -r optparse -r tempfile -r timeout -r tmpdir -r uri -I sig validate
+bundle exec steep check
+bundle exec ruby tool/type_stats.rb --write
+```
+
+CI generates signatures into a clean temporary directory and compares the
+complete tree, so both missing and stale signature files fail.
+
+## Bounded mutation testing
+
+Mutation analysis is part of the development bundle:
+
+```sh
+bundle exec rake quality:mutation
+```
+
+The job uses MRI 4.0, two workers, a ten-minute job budget, and the focused
+`Ibex::Tables::Compact#initialize` matcher. `TablesTest` owns the mutation
+coverage declaration. The mutation environment declares open-source usage and
+applies a one-second timeout to each deterministic mutation. Expanding the
+subject requires an explicit test owner, no surviving non-equivalent mutations,
+and a measured duration within the same bounded job.
+
+The supported runtime matrix uses `gemfiles/compat.Gemfile`, so the mutation
+tool's Ruby floor does not change the library's Ruby floor.
+
+## Benchmarks and evidence
+
+Performance observations are not ordinary CI timing thresholds. Follow the
+[benchmark guide](../benchmark/README.md) for workload identities, formal
+comparison commands, environment matching, artifact validation, and
+append-only result history.
+
+The development bundle includes the exact-version external-grammar profiler:
+
+```sh
+bundle exec ruby benchmark/public_profile.rb --help
+```
+
+Its output is diagnostic-only and must remain outside `benchmark/results/`.
+Use it to locate costs, then repeat the uninstrumented public comparison to
+produce reviewable performance evidence.
+
+The versioned error-experience snapshot and its review procedure live in
+[`docs/error-ux.md`](error-ux.md).

data/docs/editor-setup.md

@@ -0,0 +1,25 @@
+# Editor setup
+
+Ibex ships an LSP 3.17 server for grammar files:
+
+```sh
+ibex lsp --stdio
+```
+
+Configure an editor language client to start that command for `.y` files and pass the project directory as `rootUri` or an
+initial `workspaceFolders` entry. The server uses standard Content-Length framing on stdin/stdout. Do not wrap it with a command
+that writes banners or logs to stdout; Ibex writes its own logs to stderr.
+
+The server negotiates UTF-16 positions and full-document text synchronization. It provides:
+
+- bounded parse and include diagnostics, including unsaved included fragments;
+- definition and references for rules, formal parameters, terminals, and include targets;
+- guarded cross-file prepare-rename and rename; and
+- hover for rule signatures/documentation, terminal metadata/precedence, and include targets.
+
+Only local `file:` URIs below the initialized workspace roots are accepted. Symlink escapes, traversal, remote authorities,
+query/fragment URI components, and ambiguous encoded separators are rejected. Open buffers override disk until `didClose`;
+closing first clears the buffer diagnostics and then restores the disk snapshot.
+
+For a generic editor configuration, use command `ibex`, arguments `lsp --stdio`, language id `ibex`, and file pattern `*.y`.
+Ibex never executes semantic actions or `header`, `inner`, or `footer` code while serving editor requests.

data/docs/error-ux.md

@@ -0,0 +1,60 @@
+# JSON error UX evidence
+
+This report fixes the Phase 14 error-experience baseline against ten malformed
+inputs for the gallery JSON grammar. It is generated from
+[`json-errors-v1.json`](../test/fixtures/error_ux/json-errors-v1.json) by:
+
+```sh
+bundle exec ruby tool/error_ux_snapshot.rb
+```
+
+Use `--write` only after reviewing a deliberate diagnostic, parser-table, racc,
+or repair-policy change. CI regenerates the evidence and requires byte-for-byte
+equality.
+
+## Comparison method
+
+Ibex uses [`examples/json.y`](../examples/json.y), including its generated
+lexer locations. The migration from its previous handwritten lexer is guarded
+by exact location and error-snapshot tests. The comparison uses the self-authored compatible grammar
+[`json_racc.y`](../test/fixtures/error_ux/json_racc.y), racc 1.8.1, and only the
+public `racc` executable and `on_error(token, value, value_stack)` callback.
+Neither racc implementation files nor generated source are inspected. Both
+parsers receive the same source strings and equivalent token streams.
+
+The racc column records the neutral callback wrapper's token argument. It does
+not claim that an application cannot build richer diagnostics around racc; it
+shows what the compared parser runtime exposes at that boundary without an
+additional state-to-message layer.
+
+| Case | Invalid source | Ibex diagnostic evidence | racc callback | Selected repair | Useful |
+|---|---|---|---|---|---|
+| EUX-01 | `{"a":,}` | line 1, column 6; 7 expected values; caret | token 9 | delete `,`, insert `STRING` | yes |
+| EUX-02 | `{"a" 1}` | line 1, column 6; expected `:`; caret | token 3 | insert `:` | yes |
+| EUX-03 | `{"a":1,}` | line 1, column 8; expected `STRING`; caret | token 8 | insert a complete member | no |
+| EUX-04 | `[1,]` | line 1, column 4; 7 expected values; caret | token 12 | insert `STRING` | no |
+| EUX-05 | `[1 2]` | line 1, column 4; expected `,` or `]`; caret | token 3 | insert `,` | yes |
+| EUX-06 | `{"a":[true false]}` | line 1, column 12; expected `,` or `]`; caret | token 5 | insert `,` | yes |
+| EUX-07 | `{"a":null "b":2}` | line 1, column 11; expected `}` or `,`; caret | token 2 | insert `,` | yes |
+| EUX-08 | `{]` | line 1, column 2; expected `STRING` or `}`; caret | token 12 | replace `]` with `}` | yes |
+| EUX-09 | `[}` | line 1, column 2; 8 expected values/closer; caret | token 8 | replace `}` with `]` | yes |
+| EUX-10 | `true false` | line 1, column 6; expected EOF; caret | token 5 | delete extra `FALSE` | yes |
+
+The committed JSON is the normative snapshot and also retains exact messages,
+LR states, expected-token arrays, token values, repair costs, configuration
+counts, and edit positions. Its public structure is validated by
+[`schema/error-ux-v1.schema.json`](../schema/error-ux-v1.schema.json).
+
+## SP-4 decision
+
+The bounded insertion/deletion/replacement search produced a completing plan
+for all ten cases. Eight plans are reasonable representations of the likely
+human edit; two trailing-comma cases invent a missing value/member where
+deleting the comma would usually be preferable. The measured useful rate is
+therefore **8/10 (80%)**, above the required majority.
+
+SP-4 is **go** for the existing opt-in bounded single-plan repair described by
+[ADR 0012](decisions/0012-bounded-nonexecuting-analysis.md). It remains
+experimental: a selected edit can require a nil semantic value, and the search
+does not enumerate every equal-cost CPCT+ repair. The stable default answer
+remains exact expected tokens plus explicit yacc/synchronization recovery.