ibex 0.1.0 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 1cf3b4cceb041622899923a174e36d413adb399c22cdb44f8023f1f199acea73
-  data.tar.gz: 62ac1c3098103ac5270a8c684065424fcfc8bb2759332b3414672454ecbb25ca
+  metadata.gz: 46d7505f93788dea378cffff477d2d881b786445c3fd1fc805de78267784d11a
+  data.tar.gz: 853d1ad44d6224c7feba09fb00e012aa883ad3a86bbcbbaedc878b3f151f7bd6
 SHA512:
-  metadata.gz: ec45347b6c9ec5f427261ade9ea4102a2b7b6583bcec47ee024d920fc624018bae9315a7129657977de47241e867ef23809ccef372169cb8e1d6a9ad858e31b3
-  data.tar.gz: a163ab0fb84918570c2f62a9299d395653a0896c510e43a7672677803fe73f6010d74305868e1f6e5018b13b30d053c1a4d0852e16a0941f56a123bb883e7422
+  metadata.gz: 6b4f1e69a42798bc4b8b81be22ce21ce18a3a63e4fab0684e22e9d092055eba37d7a3d7341c3aec54bd04177553e8d8570419564130e1c8369e27ef742dffee9
+  data.tar.gz: 26d34199203cea82c17da593b06c067e24c661496ea13a1f0ec9bb2a4884730fb439ce5427c15c4bc193400ab365e237677462e540b539c1a31e74fa02b6e9c0

data/README.md

@@ -1,28 +1,71 @@
 # Ibex
 
-Ibex is a Pure Ruby LR parser generator. It accepts racc-compatible grammar files, generates parsers with the familiar
-`do_parse` / `yyparse` API, and requires no C or Java extension. Its staged Grammar IR and Automaton IR can also drive extended
-EBNF syntax, diagnostics, visualizations, and alternate LR construction algorithms.
+[![Gem Version](https://badge.fury.io/rb/ibex.svg)](https://badge.fury.io/rb/ibex)
+[![CI](https://github.com/ydah/ibex/actions/workflows/main.yml/badge.svg)](https://github.com/ydah/ibex/actions/workflows/main.yml)
+
+Ibex is a Pure Ruby LR parser generator. It reads racc-compatible grammar
+files, generates parsers with the familiar `do_parse` / `yyparse` API, and
+requires no C or Java extension.
+
+> [!IMPORTANT]
+> Ibex is pre-1.0. The current v1.0 decision is
+> [**HOLD**](docs/release-readiness.md): compatibility and scale evidence pass,
+> but cold generation and new-instance runtime still trail Racc's Ruby backend,
+> and the error UX evidence still needs independent review. Compatible mode is
+> the stable baseline; opt-in features have the maturity levels documented in
+> the [stability policy](docs/stability.md).
+
+- Try grammar analysis in the
+  [browser playground](https://ydah.github.io/ibex/playground/).
+- Start with the [grammar reference](docs/grammar-reference.md).
+- Build lossless and incremental tooling with the
+  [Red/Green CST guide](docs/cst.md).
+- Migrate an existing parser with the
+  [racc migration guide](docs/racc-migration.md).
+- Browse the [API reference](https://ydah.github.io/ibex/api/).
+
+The playground runs in a local Web Worker. It does not upload grammar source
+or execute semantic action bodies.
+
+## At a glance
+
+| Concern | Ibex default |
+| --- | --- |
+| Grammar input | racc-compatible `.y` source |
+| Construction | Direct LALR(1) |
+| Generated tables | Compact, immutable Ruby data |
+| Parser API | Pull parsing with `do_parse` / `yyparse` |
+| Generated dependency | The separately versioned `ibex-runtime` gem |
+| Ruby support | Ruby 3.0 or later |
+| Extensions | Disabled unless `pragma extended` or `--mode=extended` is used |
+| Concrete syntax | Stable format-v6 Red/Green CST with `pragma cst` |
+| Semantic actions | Opaque during generation and analysis; executed only by a running parser |
+
+The generator and runtime are deliberately separate:
 
-## Requirements and installation
+```text
+grammar.y + lexer contract
+          |
+          v
+        ibex  -----> parser.rb --------> ibex-runtime
+          |
+          +--------> optional RBS, IR, reports, diagrams, and docs
+```
 
-Ibex supports Ruby 3.0 or later and has no runtime gem dependencies. From a source checkout:
+Use `-E` when a generated parser must embed the runtime instead of depending
+on `ibex-runtime`.
 
-```sh
-bundle install
-bundle exec rake
-```
+## Quick start from a checkout
 
-Build and install the local gem when you want the `ibex` executable on your `PATH`:
+Clone the repository and install its locked development dependencies:
 
 ```sh
-gem build ibex.gemspec
-gem install ./ibex-0.1.0.gem
+git clone https://github.com/ydah/ibex.git
+cd ibex
+bundle install
 ```
 
-## Three-minute calculator
-
-Save this as `calculator.y`:
+Save this grammar as `calculator.y`:
 
 <!-- calculator-grammar:start -->
 ```text
@@ -54,141 +97,241 @@ end
 ```
 <!-- calculator-grammar:end -->
 
-Generate and run it:
+Generate and run the parser without installing local gems:
+
+```sh
+bundle exec ruby -Ilib exe/ibex calculator.y
+bundle exec ruby -Ilib calculator.rb
+# 14
+```
+
+With the local gems installed, the equivalent commands are:
 
 ```sh
 ibex calculator.y
 ruby calculator.rb
-# 14
 ```
 
-From a checkout without installing the gem, use `bundle exec ruby -Ilib exe/ibex calculator.y` and
-`bundle exec ruby -Ilib calculator.rb` instead.
+Ibex writes `calculator.rb`, uses compact tables, and generates a parser that
+requires `ibex-runtime`. Use `-o PATH` to choose the output, `--table=plain`
+for inspectable Hash rows, `--check` to compare without rewriting, or `-E` to
+embed the runtime.
+
+## Generated parser contract
+
+### Tokens and lexers
+
+A handwritten pull lexer implements `next_token` and returns:
+
+```ruby
+[token, value]
+[token, value, location]
+```
 
-Ibex generates compact tables by default. Compatibility-safe default reductions shrink profitable states while retaining
-explicit error cells, including recovery and undeclared-token behavior. `--table=plain` produces inspectable Hash rows, while
-`-E` embeds the runtime into a single dependency-free output file.
+`false` or `nil` marks EOF. Bare grammar tokens use Ruby symbols such as
+`:NUM`; quoted terminals use strings such as `'+'`. Locations are
+application-owned objects and remain optional.
 
-## Lexer contract
+Existing lexers can use the [lexer migration guide](docs/lexer-migration.md).
+Extended grammars may instead generate a streaming lexer from declarative
+regular-expression rules.
 
-Ibex does not generate a lexer. A pull parser implements `next_token` and returns `[token, value]`; `false` or `nil` marks EOF.
-Bare grammar tokens normally use Ruby symbols (`:NUM`), and quoted grammar tokens use strings (`'+'`). A push source can call
-`yyparse(receiver, method_name)` where the receiver method yields the same pairs.
+### Semantic actions and locations
 
-The default `on_error(token_id, value, value_stack)` raises `Ibex::ParseError`. Override it to use yacc-style `error` recovery.
-Semantic actions can call `yyerror`, `yyerrok`, or `yyaccept`, and `expected_tokens` reports valid lookaheads in the current state.
-Parser subclasses can also override `on_shift(token_id, value, state)`,
-`on_reduce(production_id, values, result)`, and `on_error_recover(token_id, value, value_stack)` as no-op-by-default observers.
-Ordinary shifts and the synthetic recovery-token shift use separate hooks; observer return values never replace semantic values.
+Actions read RHS values through `val`. `@1`, `@2`, and so on address parallel
+semantic locations, while `@$` is the span of the current reduction. Action
+source remains opaque Ruby: Ibex preserves its source mapping but does not
+parse, type-check, or sandbox it.
 
-## Extended mode
+### Parse lifecycles and errors
 
-`--mode=extended` enables optional, repeated, and separated values plus named references. A grammar can make the same choice
-locally with `pragma extended` immediately after its `class` header:
+Generated parsers support:
+
+- `do_parse` and yielding `yyparse` for racc-compatible pull parsing;
+- `push(token, value, location)` and `finish(location:)` for caller-driven
+  streaming;
+- structured `Ibex::ParseError` values containing the token, expected tokens,
+  state, location, and spelling suggestions;
+- yacc-style recovery through `on_error`, `yyerror`, `yyerrok`, and
+  `yyaccept`; and
+- immutable, shareable generated tables.
+
+Use one parser instance per concurrent parse. Isolated runtime sessions and
+resource budgets are opt-in when application code needs explicit execution
+boundaries.
+
+## Choose a grammar mode
+
+| Mode | Use it for | Contract |
+| --- | --- | --- |
+| `default` | Existing racc grammars and conservative migrations | Compatible syntax and runtime behavior |
+| `extended` | New grammars that need composition or generated tooling | Preview, explicitly enabled |
+
+Enable extended syntax with `pragma extended` in the grammar or
+`--mode=extended` on the command line:
 
 ```text
 class ExtendedParser
 pragma extended
+token NUM KEY VALUE
 rule
-  arguments : separated_list(NUM, ',') { result = val[0] }
+  arguments : separated_list(NUM, ',')
   sum       : NUM:left '+' NUM:right { result = left + right }
   maybe     : NUM?
   many      : NUM*
-  some      : NUM+
-  pairs     : (KEY VALUE)*
+  groups    : (KEY VALUE)+
+  list(X)   : X | list(X) ',' X
+  %inline signed(X): '-' X { result = -val[1] }
+  negative  : signed(NUM)
 end
 ```
 
-The value conventions are `nil` or a value for `?`, and arrays for `*`, `+`, `separated_list`, and
-`separated_nonempty_list`. Parenthesized sequences and alternatives can be nested; multi-item groups produce an Array value.
-Text, DOT, and HTML automaton reports label lowered helper symbols with these source-level EBNF expressions.
+Extended mode includes nested EBNF groups, named references, parameterized and
+inline rules, multiple entry points, canonical fragment imports, generated
+lexers, CST or typed `Data` AST generation, declarative recovery, semantic
+types, and grammar-declared tests. These features do not silently change the
+default compatible frontend. Maturity is assigned per API: format-v6 batch
+Red/Green CST is Stable, while the wider extended grammar surface remains
+Preview and incremental CST sessions remain Experimental.
 
-## Pipeline and diagnostics
+## Choose a construction algorithm
 
-```sh
-ibex --emit=grammar-ir grammar.y > grammar.json
-ibex --from=grammar-ir --emit=automaton-ir grammar.json > automaton.json
-ibex --from=automaton-ir -o parser.rb automaton.json
-ibex -v --dot=states.dot --html=states.html grammar.y
-ibex --algorithm=lr1 grammar.y
-ibex --counterexamples --counterexample-max-tokens=64 --counterexample-max-configurations=100000 grammar.y
-ibex --rbs -o parser.rb grammar.y
-ibex --warnings=all,error -C grammar.y
-```
+| Option | When to use it |
+| --- | --- |
+| `--algorithm=lalr` | Default direct LALR(1) construction |
+| `--algorithm=slr` | Grammars whose FOLLOW-set reductions are sufficient |
+| `--algorithm=ielr` | Preview backend for conflicts introduced by LALR state merging |
+| `--algorithm=lr1` | Canonical LR(1) analysis when the larger automaton is acceptable |
 
-Supported construction algorithms are `slr`, `lalr` (default), and canonical `lr1`. Reports retain precedence-resolved
-conflicts and distinguish unifying counterexamples from nonunifying reachability witnesses. Counterexample searches default to
-32 tokens and 50,000 explored configurations; `--counterexample-max-tokens=N` and
-`--counterexample-max-configurations=N` set positive per-run budgets and request a report. `--rbs` writes a signature beside the
-generated parser; `--rbs=FILE` selects another path. Application methods supplied as opaque `---- inner` code can be declared by
-reopening the generated class in an application RBS file.
+`--suggest-ielr` checks whether IELR removes an unexpected LALR conflict and
+prints an advisory note when it does.
+That note does not change the selected algorithm or exit status.
 
-`--warnings=all` prints unused terminals, unreachable nonterminals, duplicate productions, undeclared terminals, and empty-language
-diagnostics. Add `error` (`--warnings=all,error`, or simply `--warnings=error`) to make any such diagnostic fail the command.
-Action exceptions and `inner` methods point back to their `.y` lines by default. `--line-convert-all` extends that mapping to
-`header` and `footer`; `-l` disables every source-line conversion. User-code chunk locations survive Grammar/Automaton IR JSON
-round trips, so resumed generation has the same backtraces as direct generation.
+Conflict freedom is not the same as a proof of unambiguity. `ibex check
+--ambiguity` searches for a concrete sentence with two interpretations, but
+the token and configuration budgets make every result explicitly bounded.
 
-## Documentation
+## Integrate generation into a build
 
-- [Grammar reference](docs/grammar-reference.md)
-- [racc migration guide](docs/racc-migration.md)
-- [Architecture and IR schemas](docs/architecture.md)
-- [Compatibility observations](docs/compat-notes.md)
-- [Phase 10 extensions](docs/phase10-extensions.md)
+Treat grammar source as the input and generated Ruby as a reproducible build
+artifact:
 
-## Development
+```sh
+ibex --warnings=error -o lib/calculator.rb grammar/calculator.y
+ibex --warnings=error --check -o lib/calculator.rb grammar/calculator.y
+```
 
-Run all unit, integration, documentation, and optional local racc black-box tests with `bundle exec rake test`; run style checks
-with `bundle exec rake lint`. The default `bundle exec rake` runs both. Compatibility tests skip automatically when the `racc`
-command is unavailable.
+The first command writes the parser. The second is suitable for CI and fails
+when the committed output is stale without modifying it.
 
-Ibex's grammar frontend is self-hosted. Edit `lib/ibex/frontend/grammar.y`, then regenerate and verify the committed parser with:
+Rake projects can declare the dependency directly:
 
-```sh
-bundle exec rake frontend:generate
-git diff --exit-code -- lib/ibex/frontend/generated_parser.rb
-bundle exec ruby -Itest test/frontend/self_host_test.rb
+```ruby
+require "ibex/rake_task"
+
+Ibex::RakeTask.new("lib/calculator.rb") do |task|
+  task.grammar = "grammar/calculator.y"
+  task.options = ["--warnings=error"]
+end
 ```
 
-Normal library and CLI execution use the generated parser. The handwritten `BootstrapParser` is loaded only by this regeneration
-workflow, whose direct dependency graph also works when the generated file is absent. Byte-comparison and AST/error parity tests
-prevent generated-source drift.
+The task follows the grammar's fragment dependency closure. Multi-output
+generation can additionally publish a manifest containing input identities,
+output paths, sizes, and SHA-256 digests. Generation renders all requested
+outputs before publishing them; `--watch` retains the last successful output
+when a later edit is invalid.
 
-Fixed-seed property tests exercise SLR, LALR, and LR(1) pipeline invariants, while versioned Grammar and Automaton IR fixtures
-guard byte-stable schema compatibility. Intentional fixture updates are documented in `test/fixtures/ir/README.md`. Run the
-reproducible whole-builder benchmark without a timing pass/fail threshold with:
+Applications that only execute generated parsers need `ibex-runtime`, not the
+generator. To build and install both current local gems:
 
 ```sh
-benchmark/pipeline.rb --rules 200 --iterations 5 --seed 12345
+gem build ibex-runtime.gemspec
+gem build ibex.gemspec
+gem install ./ibex-runtime-0.1.0.gem
+gem install ./ibex-0.1.0.gem
 ```
 
-Signatures for every Ruby source under `lib/` are generated from rbs-inline annotations and checked with Steep, including the
-self-hosted generated parser. To regenerate the committed signature tree and reproduce its validation locally:
+## Diagnose, verify, and inspect
+
+| Goal | Command |
+| --- | --- |
+| Promote grammar warnings to errors | `ibex --warnings=error -C grammar.y` |
+| Collect structured frontend diagnostics | `ibex diagnose --format=json grammar.y` |
+| Explain one conflict | `ibex explain --state=7 --token=ELSE grammar.y` |
+| Search for a concrete ambiguity | `ibex check --ambiguity --algorithm=lr1 grammar.y` |
+| Run grammar-declared tests | `ibex test --coverage=100 grammar.y` |
+| Check canonical formatting | `ibex fmt --check grammar.y` |
+| Render grammar documentation | `ibex doc --format=html -o grammar.html grammar.y` |
+| Emit Automaton IR | `ibex --emit=automaton-ir grammar.y > automaton.json` |
+| Simulate table behavior without actions | `ibex debug automaton.json NUMBER PLUS NUMBER` |
+| Check a racc migration | `ibex migrate-check grammar.y` |
+| Start the language server | `ibex lsp --stdio` |
+
+Grammar IR, Automaton IR, generated table formats, diagnostic JSON, runtime
+events, coverage, and manifests are versioned contracts. Reports can also be
+rendered as text, DOT, Mermaid, HTML, railroad SVG, samples, or conflict
+counterexamples.
+
+## Safety boundaries and non-goals
+
+- Frontend parsing, diagnosis, formatting, documentation, LSP, static
+  migration checks, IR validation, and table simulation do not execute
+  semantic actions or user-code sections.
+- Generated parsers, grammar-declared tests, and explicit migration harnesses
+  do execute application Ruby. They are not sandboxes.
+- Fragment imports are confined to the declared source root and reject
+  traversal, cycles, and symlink escapes.
+- Counterexample, ambiguity, sample, repair, and runtime-budget searches are
+  bounded. Exhausting a budget is distinct from finding no witness within it.
+- Ibex generates deterministic LR parsers. It does not provide GLR or
+  generalized ambiguity handling. Incremental CST sessions are syntax-only
+  and experimental.
+- racc compatibility is a migration surface, not a claim that every generated
+  parser is a byte-for-byte or adapter-free replacement.
+- Preview and experimental features may have weaker compatibility guarantees
+  than the default mode.
+
+## Documentation by task
+
+| When you need to... | Start here |
+| --- | --- |
+| Write a grammar or use the runtime | [Grammar reference](docs/grammar-reference.md) |
+| Build, edit, serialize, or incrementally update syntax trees | [Red/Green CST guide](docs/cst.md) and [migration guide](docs/cst-migration.md) |
+| Learn from executable grammars | [Examples](examples/README.md) |
+| Migrate from racc | [racc migration guide](docs/racc-migration.md) |
+| Adapt a handwritten lexer | [Lexer migration guide](docs/lexer-migration.md) |
+| Configure an editor or LSP client | [Editor setup](docs/editor-setup.md) |
+| Understand the pipeline, IR, algorithms, or tables | [Architecture](docs/architecture.md) |
+| Check API maturity and deprecation rules | [Stability policy](docs/stability.md) |
+| Evaluate readiness, performance, or error UX | [Release readiness](docs/release-readiness.md), [benchmarks](benchmark/README.md), and [error UX](docs/error-ux.md) |
+| Contribute or run every quality gate | [Development guide](docs/development.md) |
+| Review implementation decisions | [Architecture decision records](docs/decisions/README.md) |
+
+The published documentation separates the
+[compatibility contract](https://ydah.github.io/ibex/compatibility/),
+[opt-in extensions](https://ydah.github.io/ibex/extensions/), and
+[experimental surfaces](https://ydah.github.io/ibex/experimental/). The
+[grammar gallery](https://ydah.github.io/ibex/gallery/) contains executable
+examples whose declared tests reach complete production coverage.
+
+## Development
+
+The [development guide](docs/development.md) contains frontend regeneration,
+RBS/Steep validation, grammar coverage, browser checks, mutation analysis, and
+workflow lint commands. The default local gate is:
 
 ```sh
-BUNDLE_GEMFILE=gemfiles/Gemfile bundle install
-BUNDLE_GEMFILE=gemfiles/Gemfile ruby -e '
-  sources = Dir.glob("lib/**/*.rb").sort
-  exec("bundle", "exec", "rbs-inline", "--opt-out", "--base=lib", "--output=sig", *sources)
-'
-BUNDLE_GEMFILE=gemfiles/Gemfile bundle exec rbs -r digest -r json -r optparse -I sig validate
-BUNDLE_GEMFILE=gemfiles/Gemfile bundle exec steep check
-BUNDLE_GEMFILE=gemfiles/Gemfile bundle exec steep stats
-BUNDLE_GEMFILE=gemfiles/Gemfile bundle exec ruby tool/type_stats.rb --write
+bundle install
+bundle exec rake
 ```
 
-CI performs generation in a clean temporary directory and compares the complete trees, so missing source signatures and stale
-signature files both fail the build.
 <!-- type-stats:start -->
-The current whole-library `steep stats` result is 3,771 typed calls and 449 untyped calls out of 4,220 (89.4% typed).
-The generated signature tree contains 397 explicit `untyped` occurrences across 16 files.
+The current whole-library `steep stats` result is 20,629 typed calls and 2,275 untyped calls out of 22,904 (90.1% typed).
+The generated signature tree contains 2,084 explicit `untyped` occurrences across 103 files.
 <!-- type-stats:end -->
 
-Those boundaries are concentrated in generated-parser reduction values, heterogeneous JSON decoding/serialization, runtime
-semantic values and parser-table cells, and embedded user Ruby. Token/location records, the complete grammar AST, parser
-classifier state, IR, the public Ruby DSL, bootstrap parser state, analysis, automaton construction, code generators, table
-construction, and CLI options use concrete domain types. The committed self-hosted parser remains in the Steep target; no library
-directory or generated source is excluded.
+Performance measurements are evidence, not portable scores or CI timing
+thresholds. Follow the [benchmark guide](benchmark/README.md) to reproduce and
+compare results under the same environment and configuration.
 
 Ibex is available under the [MIT License](LICENSE.txt).