ibex 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 1cf3b4cceb041622899923a174e36d413adb399c22cdb44f8023f1f199acea73
+  data.tar.gz: 62ac1c3098103ac5270a8c684065424fcfc8bb2759332b3414672454ecbb25ca
+SHA512:
+  metadata.gz: ec45347b6c9ec5f427261ade9ea4102a2b7b6583bcec47ee024d920fc624018bae9315a7129657977de47241e867ef23809ccef372169cb8e1d6a9ad858e31b3
+  data.tar.gz: a163ab0fb84918570c2f62a9299d395653a0896c510e43a7672677803fe73f6010d74305868e1f6e5018b13b30d053c1a4d0852e16a0941f56a123bb883e7422

data/.rubocop.yml

@@ -0,0 +1,43 @@
+AllCops:
+  CacheRootDirectory: tmp/rubocop_cache
+  NewCops: enable
+  TargetRubyVersion: 3.0
+  SuggestExtensions: false
+  Exclude:
+    - "pkg/**/*"
+    - "tmp/**/*"
+    - "vendor/**/*"
+    - "lib/ibex/frontend/generated_parser.rb"
+
+Layout/LineLength:
+  Max: 120
+
+Layout/LeadingCommentSpace:
+  AllowRBSInlineAnnotation: true
+
+Style/StringLiterals:
+  EnforcedStyle: double_quotes
+
+Metrics/AbcSize:
+  Max: 36
+
+Metrics/ClassLength:
+  Max: 200
+
+Metrics/CyclomaticComplexity:
+  Max: 10
+
+Metrics/MethodLength:
+  Max: 30
+
+Metrics/ModuleLength:
+  Max: 150
+
+Metrics/ParameterLists:
+  Max: 12
+
+Metrics/PerceivedComplexity:
+  Max: 10
+
+Style/Documentation:
+  Enabled: false

data/CHANGELOG.md

@@ -0,0 +1,30 @@
+# Changelog
+
+## Unreleased
+
+- Establish the Pure Ruby gem, Minitest, lint, CI, and CLI foundations.
+- Add a table-driven Pure Ruby LR runtime with pull/push APIs, recovery, and tracing.
+- Tokenize grammar files with positions, opaque Ruby actions, comments, and user-code blocks.
+- Parse compatible and extended grammar syntax into a serializable, location-preserving AST.
+- Normalize grammars into immutable, versioned JSON IR with EBNF expansion and diagnostics.
+- Compute nullable, FIRST, and FOLLOW sets with deterministic integer bitsets.
+- Build deterministic LALR(1) automata, resolve and retain conflicts, and render state reports.
+- Generate plain or compact Ruby parsers with embedded runtime and source-line mapping support.
+- Preserve per-block user-code source maps through IR and implement compatible default, all-code, and disabled line conversion.
+- Complete the compatible CLI surface and add optional racc black-box result comparisons.
+- Add extended EBNF/named-reference integration, source-facing EBNF labels in text/DOT/HTML reports, and resumable IR pipelines.
+- Add selectable SLR, LALR(1), and canonical LR(1) construction strategies.
+- Add a Ruby DSL frontend that converges on the existing Grammar AST and IR pipeline.
+- Add unifying conflict counterexamples with complete competing derivation trees and bounded nonunifying fallback.
+- Add generated parser RBS output, shipped runtime signatures, and strict structured grammar diagnostics.
+- Expand generated signatures and Steep checking to every library source, including concrete frontend/IR/LALR/codegen/CLI domain
+  types and the self-hosted parser.
+- Support quoted, interpolated, and multiple Ruby heredocs plus recursively nested grouped EBNF.
+- Define unknown external tokens to invoke `on_error` before yacc-style recovery.
+- Add post-commit shift, reduction, and successful-recovery observation hooks to the runtime.
+- Add complete quickstart, grammar, migration, architecture, and extension documentation with an executable README test.
+- Self-host the grammar frontend from a committed Ibex grammar with deterministic regeneration and bootstrap parity checks.
+- Version generated parser tables and reject missing or unsupported formats before consuming input.
+- Add grammar-local `pragma extended` selection and document the action and named-reference boundary inside EBNF groups.
+- Add fixed-seed pipeline properties, versioned Grammar/Automaton IR fixtures, and a reproducible whole-builder benchmark.
+- Keep the whole-library Steep statistics in the README synchronized and enforce them in CI.

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2026 Yudai Takada
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,194 @@
+# Ibex
+
+Ibex is a Pure Ruby LR parser generator. It accepts racc-compatible grammar files, generates parsers with the familiar
+`do_parse` / `yyparse` API, and requires no C or Java extension. Its staged Grammar IR and Automaton IR can also drive extended
+EBNF syntax, diagnostics, visualizations, and alternate LR construction algorithms.
+
+## Requirements and installation
+
+Ibex supports Ruby 3.0 or later and has no runtime gem dependencies. From a source checkout:
+
+```sh
+bundle install
+bundle exec rake
+```
+
+Build and install the local gem when you want the `ibex` executable on your `PATH`:
+
+```sh
+gem build ibex.gemspec
+gem install ./ibex-0.1.0.gem
+```
+
+## Three-minute calculator
+
+Save this as `calculator.y`:
+
+<!-- calculator-grammar:start -->
+```text
+class Calculator
+token NUM
+preclow
+  left '+'
+  left '*'
+prechigh
+rule
+  expr : expr '+' expr { result = val[0] + val[2] }
+       | expr '*' expr { result = val[0] * val[2] }
+       | NUM { result = val[0] }
+end
+---- inner
+def parse_tokens(tokens)
+  @tokens = tokens
+  do_parse
+end
+
+def next_token
+  @tokens.shift
+end
+---- footer
+if $PROGRAM_NAME == __FILE__
+  tokens = [[:NUM, 2], ['+', nil], [:NUM, 3], ['*', nil], [:NUM, 4]]
+  puts Calculator.new.parse_tokens(tokens)
+end
+```
+<!-- calculator-grammar:end -->
+
+Generate and run it:
+
+```sh
+ibex calculator.y
+ruby calculator.rb
+# 14
+```
+
+From a checkout without installing the gem, use `bundle exec ruby -Ilib exe/ibex calculator.y` and
+`bundle exec ruby -Ilib calculator.rb` instead.
+
+Ibex generates compact tables by default. Compatibility-safe default reductions shrink profitable states while retaining
+explicit error cells, including recovery and undeclared-token behavior. `--table=plain` produces inspectable Hash rows, while
+`-E` embeds the runtime into a single dependency-free output file.
+
+## Lexer contract
+
+Ibex does not generate a lexer. A pull parser implements `next_token` and returns `[token, value]`; `false` or `nil` marks EOF.
+Bare grammar tokens normally use Ruby symbols (`:NUM`), and quoted grammar tokens use strings (`'+'`). A push source can call
+`yyparse(receiver, method_name)` where the receiver method yields the same pairs.
+
+The default `on_error(token_id, value, value_stack)` raises `Ibex::ParseError`. Override it to use yacc-style `error` recovery.
+Semantic actions can call `yyerror`, `yyerrok`, or `yyaccept`, and `expected_tokens` reports valid lookaheads in the current state.
+Parser subclasses can also override `on_shift(token_id, value, state)`,
+`on_reduce(production_id, values, result)`, and `on_error_recover(token_id, value, value_stack)` as no-op-by-default observers.
+Ordinary shifts and the synthetic recovery-token shift use separate hooks; observer return values never replace semantic values.
+
+## Extended mode
+
+`--mode=extended` enables optional, repeated, and separated values plus named references. A grammar can make the same choice
+locally with `pragma extended` immediately after its `class` header:
+
+```text
+class ExtendedParser
+pragma extended
+rule
+  arguments : separated_list(NUM, ',') { result = val[0] }
+  sum       : NUM:left '+' NUM:right { result = left + right }
+  maybe     : NUM?
+  many      : NUM*
+  some      : NUM+
+  pairs     : (KEY VALUE)*
+end
+```
+
+The value conventions are `nil` or a value for `?`, and arrays for `*`, `+`, `separated_list`, and
+`separated_nonempty_list`. Parenthesized sequences and alternatives can be nested; multi-item groups produce an Array value.
+Text, DOT, and HTML automaton reports label lowered helper symbols with these source-level EBNF expressions.
+
+## Pipeline and diagnostics
+
+```sh
+ibex --emit=grammar-ir grammar.y > grammar.json
+ibex --from=grammar-ir --emit=automaton-ir grammar.json > automaton.json
+ibex --from=automaton-ir -o parser.rb automaton.json
+ibex -v --dot=states.dot --html=states.html grammar.y
+ibex --algorithm=lr1 grammar.y
+ibex --counterexamples --counterexample-max-tokens=64 --counterexample-max-configurations=100000 grammar.y
+ibex --rbs -o parser.rb grammar.y
+ibex --warnings=all,error -C grammar.y
+```
+
+Supported construction algorithms are `slr`, `lalr` (default), and canonical `lr1`. Reports retain precedence-resolved
+conflicts and distinguish unifying counterexamples from nonunifying reachability witnesses. Counterexample searches default to
+32 tokens and 50,000 explored configurations; `--counterexample-max-tokens=N` and
+`--counterexample-max-configurations=N` set positive per-run budgets and request a report. `--rbs` writes a signature beside the
+generated parser; `--rbs=FILE` selects another path. Application methods supplied as opaque `---- inner` code can be declared by
+reopening the generated class in an application RBS file.
+
+`--warnings=all` prints unused terminals, unreachable nonterminals, duplicate productions, undeclared terminals, and empty-language
+diagnostics. Add `error` (`--warnings=all,error`, or simply `--warnings=error`) to make any such diagnostic fail the command.
+Action exceptions and `inner` methods point back to their `.y` lines by default. `--line-convert-all` extends that mapping to
+`header` and `footer`; `-l` disables every source-line conversion. User-code chunk locations survive Grammar/Automaton IR JSON
+round trips, so resumed generation has the same backtraces as direct generation.
+
+## Documentation
+
+- [Grammar reference](docs/grammar-reference.md)
+- [racc migration guide](docs/racc-migration.md)
+- [Architecture and IR schemas](docs/architecture.md)
+- [Compatibility observations](docs/compat-notes.md)
+- [Phase 10 extensions](docs/phase10-extensions.md)
+
+## Development
+
+Run all unit, integration, documentation, and optional local racc black-box tests with `bundle exec rake test`; run style checks
+with `bundle exec rake lint`. The default `bundle exec rake` runs both. Compatibility tests skip automatically when the `racc`
+command is unavailable.
+
+Ibex's grammar frontend is self-hosted. Edit `lib/ibex/frontend/grammar.y`, then regenerate and verify the committed parser with:
+
+```sh
+bundle exec rake frontend:generate
+git diff --exit-code -- lib/ibex/frontend/generated_parser.rb
+bundle exec ruby -Itest test/frontend/self_host_test.rb
+```
+
+Normal library and CLI execution use the generated parser. The handwritten `BootstrapParser` is loaded only by this regeneration
+workflow, whose direct dependency graph also works when the generated file is absent. Byte-comparison and AST/error parity tests
+prevent generated-source drift.
+
+Fixed-seed property tests exercise SLR, LALR, and LR(1) pipeline invariants, while versioned Grammar and Automaton IR fixtures
+guard byte-stable schema compatibility. Intentional fixture updates are documented in `test/fixtures/ir/README.md`. Run the
+reproducible whole-builder benchmark without a timing pass/fail threshold with:
+
+```sh
+benchmark/pipeline.rb --rules 200 --iterations 5 --seed 12345
+```
+
+Signatures for every Ruby source under `lib/` are generated from rbs-inline annotations and checked with Steep, including the
+self-hosted generated parser. To regenerate the committed signature tree and reproduce its validation locally:
+
+```sh
+BUNDLE_GEMFILE=gemfiles/Gemfile bundle install
+BUNDLE_GEMFILE=gemfiles/Gemfile ruby -e '
+  sources = Dir.glob("lib/**/*.rb").sort
+  exec("bundle", "exec", "rbs-inline", "--opt-out", "--base=lib", "--output=sig", *sources)
+'
+BUNDLE_GEMFILE=gemfiles/Gemfile bundle exec rbs -r digest -r json -r optparse -I sig validate
+BUNDLE_GEMFILE=gemfiles/Gemfile bundle exec steep check
+BUNDLE_GEMFILE=gemfiles/Gemfile bundle exec steep stats
+BUNDLE_GEMFILE=gemfiles/Gemfile bundle exec ruby tool/type_stats.rb --write
+```
+
+CI performs generation in a clean temporary directory and compares the complete trees, so missing source signatures and stale
+signature files both fail the build.
+<!-- type-stats:start -->
+The current whole-library `steep stats` result is 3,771 typed calls and 449 untyped calls out of 4,220 (89.4% typed).
+The generated signature tree contains 397 explicit `untyped` occurrences across 16 files.
+<!-- type-stats:end -->
+
+Those boundaries are concentrated in generated-parser reduction values, heterogeneous JSON decoding/serialization, runtime
+semantic values and parser-table cells, and embedded user Ruby. Token/location records, the complete grammar AST, parser
+classifier state, IR, the public Ruby DSL, bootstrap parser state, analysis, automaton construction, code generators, table
+construction, and CLI options use concrete domain types. The committed self-hosted parser remains in the Steep target; no library
+directory or generated source is excluded.
+
+Ibex is available under the [MIT License](LICENSE.txt).

data/Rakefile

@@ -0,0 +1,25 @@
+# frozen_string_literal: true
+
+require "bundler/gem_tasks"
+require "rake/testtask"
+require "rubocop/rake_task"
+
+Rake::TestTask.new(:test) do |task|
+  task.libs << "test"
+  task.pattern = "test/**/*_test.rb"
+  task.warning = true
+end
+
+RuboCop::RakeTask.new(:lint)
+
+namespace :frontend do
+  desc "Regenerate the self-hosted grammar parser"
+  task :generate do
+    require_relative "lib/ibex/frontend/regenerator"
+
+    output = File.expand_path("lib/ibex/frontend/generated_parser.rb", __dir__)
+    File.write(output, Ibex::Frontend::Regenerator.generate)
+  end
+end
+
+task default: %i[test lint]

data/Steepfile

@@ -0,0 +1,10 @@
+# frozen_string_literal: true
+
+target :library do
+  signature "sig"
+  check "lib"
+
+  library "digest"
+  library "json"
+  library "optparse"
+end

data/docs/architecture.md

@@ -0,0 +1,99 @@
+# Architecture and IR schemas
+
+Ibex keeps syntax, grammar meaning, automaton construction, and output concerns behind two versioned immutable contracts.
+
+```text
+.y Lexer -> Token adapter -> self-hosted LR Parser ─┐
+Ruby DSL ────────┴─> Grammar AST -> Normalizer -> Grammar IR
+                                                    |
+                                               set analysis
+                                                    |
+                                      SLR/LALR/LR1 Builder -> Automaton IR
+                                                                    |
+                              Ruby/RBS generators / report / DOT / HTML / counterexamples
+```
+
+Frontend changes stop at the Normalizer. Algorithm strategies consume Grammar IR and produce identical Automaton IR shapes.
+Outputs consume Automaton IR and never call builder internals. The CLI only connects stages and supports JSON resumption.
+
+The text frontend's canonical syntax is `lib/ibex/frontend/grammar.y`. Ibex generates and commits
+`lib/ibex/frontend/generated_parser.rb`; the public `Frontend::Parser` always delegates to that class. Lexer `Token` objects remain
+the semantic values passed through `TokenAdapter`, preserving their `Location` in AST nodes and diagnostics. The explicitly named
+handwritten `BootstrapParser` is excluded from normal loading and exists only to break the regeneration cycle. See
+[ADR 0015](decisions/0015-self-hosted-grammar-frontend.md) for the update procedure and boundary.
+
+The RBS generator emits the generated class namespace, superclass, parser-table constants, and `.parser_tables` contract. The
+gem also ships a one-to-one rbs-inline-generated signature tree under `sig/` for every Ruby source in `lib/`, including the
+self-hosted parser. CI regenerates into an empty temporary directory, compares the complete trees, validates the RBS environment,
+and runs Steep against the entire library. Token/location records, grammar AST nodes, parser classifier and bootstrap state, the
+Ruby DSL, IR records, and automaton actions use concrete domain types. Generated-parser reduction values, dynamic parser-table
+cells, decoded JSON values, and user methods embedded as opaque Ruby source remain `untyped`; applications can reopen the generated
+class in their own RBS files to declare embedded methods.
+
+## Grammar IR v1
+
+Top-level fields:
+
+| Field | Meaning |
+|---|---|
+| `ibex_ir`, `schema_version` | `"grammar"`, `1` |
+| `class_name`, `superclass` | Generated Ruby class contract |
+| `start`, `expect`, `options` | Start name, unresolved S/R expectation, result/action flags |
+| `symbols` | Interned terminals and nonterminals; `$eof` id 0 and `error` id 1 |
+| `productions` | Numeric LHS/RHS ids, action, precedence override, source origin |
+| `user_code`, `conversions`, `warnings` | Concatenated code, external token expressions, structured diagnostics |
+| `user_code_chunks` | Optional opaque chunks with first-code-line locations for compatible source mapping |
+
+Warning records use stable type names (`undeclared_terminal`, `unused_terminal`, `unreachable_nonterminal`,
+`duplicate_production`, and `empty_language`) and retain source locations. The CLI applies display/error policy at the boundary;
+normalization and IR serialization do not discard diagnostics.
+
+A symbol has `id`, `name`, `kind`, `reserved`, optional `prec {associativity, level}`, and `loc`. A production has `id`, `lhs`,
+`rhs`, optional `action`, optional `prec_override`, and `origin`. Synthetic EBNF origins include an additive, deterministic
+`expression` label used by text, DOT, and HTML presentation while numeric symbol identities remain unchanged. An action has opaque
+`code`, `loc`, `named_refs [{name,index}]`, and `context_length`; middle-action helpers use the last field to view preceding stack
+values.
+
+IR objects and nested collections are frozen. JSON keys have deterministic order, so dump/load/dump is byte-stable. Incompatible
+schema changes require a new version. The additive `user_code_chunks` field is optional so older schema-v1 JSON remains loadable;
+new text-front-end IR retains it to make direct and resumed `--line-convert-all` generation identical.
+
+## Automaton IR v1
+
+Top-level fields are `ibex_ir: "automaton"`, `schema_version`, `algorithm`, `grammar_digest`, embedded `grammar`, `states`, and
+`conflict_summary`. Embedding Grammar IR makes automaton JSON sufficient for code generation after `--from=automaton-ir`.
+
+Each state contains:
+
+- merged items `{production, dot, lookaheads}`;
+- named `transitions`;
+- resolved terminal `actions` and nonterminal `gotos`;
+- an optional reduce `default_action`, selected only when explicit error masks preserve every terminal lookup and reduce the
+  total encoded ACTION entries;
+- every conflict, including precedence-resolved conflicts and the resolution reason.
+
+`conflict_summary.sr` counts unresolved default-shift conflicts for `expect`; `resolved_sr` counts retained precedence or
+associativity decisions; `rr` counts reduce/reduce cells.
+
+## Runtime table contract
+
+Generated subclasses expose `.parser_tables` with a required `format_version`, external `tokens`, display `token_names`, ACTION
+and GOTO tables, per-state default actions, and production `{lhs,length,action}` records. The runtime validates the version before
+reading input and rejects missing or unsupported formats with a regeneration instruction; see
+[ADR 0018](decisions/0018-parser-table-format-version.md). Plain tables are arrays of Hash rows. Compact tables use row
+displacement with offsets, values, and row-ownership checks; both expose equivalent lookups. Default reductions are restricted
+to known token ids, and explicit error masks preserve the pre-optimization result of every declared terminal cell, including
+the synthetic `error` terminal. The deterministic size policy is fixed by
+[ADR 0014](decisions/0014-compatibility-safe-default-reductions.md).
+
+The runtime maintains state and value stacks, pulls a lookahead only when required, and applies tagged `shift`, `reduce`,
+`accept`, and `error` actions. Recovery pops to a state that shifts token id 1, suppresses repeated reports for three successful
+shifts, and honors `yyerrok`. No-op `on_shift`, `on_reduce`, and `on_error_recover` extension points observe successfully
+committed events without changing parser results; the recovery hook retains the pre-pop error context and is distinct from an
+ordinary token shift. Their ordering and payload contract is fixed by [ADR 0013](decisions/0013-runtime-observation-hooks.md).
+
+## Clean-room boundary
+
+Implementation work uses public racc documentation, CLI black-box behavior, and published LR algorithms only. racc implementation
+sources and generated source are not inputs to the design. Self-authored compatibility grammars execute both outputs in separate
+processes and compare observable results.

data/docs/compat-notes.md

@@ -0,0 +1,37 @@
+# Compatibility notes
+
+Compatibility observations are recorded here without consulting racc implementation sources or generated source text.
+
+## Observed CLI (2026-07-22)
+
+Black-box command: `racc --help`, version 1.8.1.
+
+Observed options are `-o/--output-file`, `-t/--debug`, obsolete `-g`, `-v/--verbose`, `-O/--log-file`,
+`-e/--executable`, `-E/--embedded`, `-F/--frozen`, `--line-convert-all`, `-l/--no-line-convert`,
+`-a/--no-omit-actions`, `--superclass`, `-C/--check-only`, `-S/--output-status`, profiling `-P`, internal `-D`,
+`--version`, `--runtime-version`, `--copyright`, and `--help`.
+
+Ibex accepts that option set. `-P` and `-D` are accepted compatibility no-ops because they expose generator internals rather
+than parser behavior. Frozen string literals are always emitted, so `-F` is also a no-op. Ibex's default output is `<input>.rb`
+rather than racc's observed `<input>.tab.rb`; `-o` is portable between both tools.
+
+## Behavioral probes
+
+Self-authored grammars are compiled independently and the generated files are executed without inspecting them. Current probes
+cover arithmetic precedence, empty rules, string tokens, `convert`, `no_result_var`, inline actions, dangling-else `expect`, and
+a generated 500-production grammar. Precedence-resolved conflicts remain in Automaton IR diagnostics but are not counted in CLI
+warnings or `expect`, matching the observed calculator behavior.
+
+An inline action's own `val[0]` was observed as `nil`; its result occupies one value position visible to the final action. The
+grammar reference confirms that `convert` takes a quoted string containing Ruby source: for example `NUM ':number'` emits the
+symbol expression, while `NUM '"number"'` emits a Ruby string token.
+
+Source-line probes with raises in self-authored `header`, `inner`, and `footer` blocks show three distinct modes. The default maps
+semantic actions and `inner` code to the grammar file but leaves `header` and `footer` on generated-file lines.
+`--line-convert-all` maps all three user-code sections, while `-l` leaves all of them on generated-file lines.
+
+With an explicitly declared but syntactically invalid `BAD` token, error recovery results and `on_error` observations (token
+string, value, and value-stack length) match. An undeclared `:BAD` was observed to enter racc's `error` production without an
+`on_error` callback. Ibex intentionally treats it as an unknown lookahead and calls `on_error` before attempting the same error
+recovery flow. This preserves the unexpected token object and value for logging and application policy instead of silently
+entering recovery.

data/docs/grammar-reference.md

@@ -0,0 +1,127 @@
+# Grammar reference
+
+Ibex's default `racc` mode accepts the compatible grammar described here. `--mode=extended` or an explicit grammar-file
+`pragma extended` adds the marked syntax; extensions are never inferred from a production.
+
+## File structure
+
+```text
+class Namespace::Parser < OptionalSuperclass
+  pragma extended       # optional; must precede ordinary declarations
+  declarations
+rule
+  productions
+end
+---- header
+Ruby copied before the parser class
+---- inner
+Ruby copied inside the parser class
+---- footer
+Ruby copied after the parser class
+```
+
+The superclass defaults to `Ibex::Runtime::Parser`. Repeated user-code blocks retain their source order and are concatenated.
+Grammar comments use `#` through end of line or `/* ... */`.
+
+## Declarations
+
+- `pragma extended` enables extended syntax for this grammar even when the CLI uses its default or explicit `--mode=racc`.
+  It must immediately follow the class header, before every ordinary declaration. Unknown, duplicate, and misplaced pragmas
+  are positioned errors. The pragma is consumed by the frontend and is not stored in AST or Grammar IR output.
+
+- `token NAME ...` declares terminals for typo diagnostics. It is optional. Uppercase names and quoted strings are terminals;
+  lowercase names are nonterminals unless they are `error`.
+- A `prechigh ... preclow` block lists precedence from high to low; `preclow ... prechigh` lists it from low to high. Each level
+  begins with `left`, `right`, or `nonassoc` followed by one or more terminals.
+- `options no_result_var` makes an action's final expression its value. `omit_action_call` is enabled by default;
+  `no_omit_action_call` disables it.
+- `expect N` suppresses the warning when exactly N unresolved shift/reduce conflicts remain. Conflicts resolved by precedence are
+  retained in Automaton IR but are not counted.
+- `start name` overrides the first rule as the start symbol.
+- `convert ... end` changes external token objects. The second column is a quoted string containing Ruby source, not the value
+  itself: `NUM ':number'` uses `:number`, while `NUM '"number"'` uses the String `"number"`.
+
+## Productions and actions
+
+```text
+rule
+  expression : expression '+' expression { result = val[0] + val[2] }
+             | '-' expression = UMINUS
+             | NUMBER
+             |                         /* empty */
+             ;                         /* optional */
+end
+```
+
+Alternatives use `|`; a trailing semicolon is optional. `= TOKEN` overrides a production's precedence. The `error` terminal
+enables yacc-style recovery.
+
+Actions are opaque Ruby between balanced braces. `val` contains RHS values, `result` begins as `val[0]`, and `_values` is a copy
+of the surrounding value stack. With `no_result_var`, the action's evaluated value is used directly. A middle action becomes an
+empty helper production and consumes one value position in the enclosing RHS.
+
+Action and `inner` backtraces use the original grammar filename and line by default. `--line-convert-all` applies the same mapping
+to `header` and `footer`; `-l` keeps all backtraces on generated-file lines.
+
+The action scanner handles nested braces, quoted/backtick strings and interpolation, `%q/%Q/%w/%W/%i/%I/%x/%r/%s`, regular
+expressions, comments, character literals, and unquoted, single-quoted, double-quoted, or backtick heredocs. Indented, squiggly,
+interpolated, and multiple heredocs on one opener line are supported. See [lexer coverage](lexer-coverage.md).
+
+## Runtime errors
+
+The default `on_error(token_id, value, value_stack)` raises `Ibex::ParseError`. Override it and return to allow an `error`
+production to recover. Unknown external token objects receive a temporary negative internal id, remain printable through
+`token_to_str`, and always invoke `on_error` before recovery is attempted.
+
+Three optional observer methods default to no-ops. `on_shift(token_id, value, state)` follows each ordinary input-token shift;
+`on_reduce(production_id, values, result)` follows a completed semantic action and goto; and
+`on_error_recover(token_id, value, value_stack)` follows a successful synthetic `error` shift while retaining the original
+unexpected-token context. Hook return values are ignored and exceptions propagate. See
+[ADR 0013](decisions/0013-runtime-observation-hooks.md) for exact ordering and snapshot semantics.
+
+## Extended EBNF and names
+
+Extended mode supports:
+
+- `item?`: `nil` or the item value.
+- `item*`: zero or more values as an Array.
+- `item+`: one or more values as an Array.
+- `separated_list(item, separator)`: zero or more item values; separators are omitted.
+- `separated_nonempty_list(item, separator)`: one or more item values.
+- `symbol:name`: binds the corresponding RHS value as a local variable in the final action.
+
+Parenthesized groups may contain sequences, alternatives, and nested EBNF, for example `(KEY VALUE)*`, `(A | B)+`, or
+`separated_list((KEY VALUE), ',')`. A one-item group has that item's value; a multi-item group has an Array of its item values;
+an empty group has `nil`. Named references must be unique in an outer alternative and cannot use `result`, `val`, or `_values`;
+references inside a group are rejected because the group is lowered behind one outer value slot. Text, DOT, and HTML reports
+render lowered helper nonterminals as their original EBNF expressions instead of exposing generated helper names.
+
+Actions and named references are supported on an outer production alternative, but not inside a parenthesized EBNF group.
+Move the action or binding to a separately named ordinary rule and reference that rule from the group.
+
+## Strict diagnostics
+
+Grammar IR retains structured diagnostics for undeclared or unused terminals, unreachable nonterminals, duplicate productions,
+and a start symbol that cannot derive any terminal sentence. They remain silent by default for compatibility. `--warnings=all`
+prints them, `--warnings=all,error` or `--warnings=error` promotes them to command failures, and `--warnings=none` explicitly
+suppresses them.
+
+## Ruby DSL
+
+The DSL builds the same AST and IR without evaluating grammar text:
+
+```ruby
+ast = Ibex::Frontend::DSL.grammar(class_name: "Calculator") do |grammar|
+  grammar.token(:NUM)
+  grammar.precedence { |levels| levels.left("'+'"); levels.left("'*'") }
+  grammar.rule(:expr) do |rule|
+    rule.alt(:expr, "'+'", :expr, action: " result = val[0] + val[2] ")
+    rule.alt(:NUM)
+  end
+end
+
+grammar_ir = Ibex::Normalizer.new(ast).normalize
+```
+
+The builder also provides `options`, `expect`, `start`, `convert`, `user_code`, `ref(as:)`, `optional`, `star`, `plus`,
+`separated_list`, and `inline`.

data/docs/lexer-coverage.md

@@ -0,0 +1,14 @@
+# Lexer coverage
+
+| Construct | Status | Notes |
+|---|---|---|
+| Nested action braces | Supported | Balanced recursively |
+| Single, double, and backtick strings | Supported | Escapes and `#{...}` interpolation are skipped |
+| `%q`, `%Q`, `%w`, `%W`, `%i`, `%I`, `%x`, `%r`, `%s` | Supported | Paired and single-character delimiters |
+| Regular expressions | Supported | Conservative expression-prefix heuristic; character classes and flags supported |
+| `#` comments and `?x` characters | Supported | Ignored for brace balancing |
+| `<<ID`, `<<-ID`, `<<~ID` heredocs | Supported | Terminators obey indentation mode |
+| Single/double/backtick quoted heredocs | Supported | Quoted identifiers may contain any non-quote, non-newline text |
+| Interpolated and multiple heredocs | Supported | Bodies are opaque; multiple openers on one line are consumed in order |
+| Grammar `#` and `/* ... */` comments | Supported | Removed before tokenization |
+| `---- header`, `inner`, `footer` | Supported | Must begin in column one; order and duplicates retained |

data/docs/phase10-extensions.md

@@ -0,0 +1,27 @@
+# Phase 10 extensions
+
+## E3: conflict witnesses
+
+`Ibex::LALR::Counterexample` accepts only Automaton IR. It explores parser-stack configurations, forces each pair of competing
+actions at the requested conflict, and then searches for a common suffix that lets both branches accept the same terminal
+sentence. A successful result contains `unifying: true`, the conflict lookahead position, and both complete derivation trees.
+
+Because unifying-counterexample search is not guaranteed to terminate for every context-free grammar, the search has explicit
+token and configuration budgets. The defaults are 32 tokens and 50,000 explored configurations. Pass `max_tokens:` and
+`max_configurations:` to `Ibex::LALR::Counterexample.new` or `Ibex::Codegen::Report.render` to adjust them through the Ruby API.
+For generated reports, use the positive-integer CLI options `--counterexample-max-tokens=N` and
+`--counterexample-max-configurations=N`; either option also requests a report. The token budget covers each unifying-search
+candidate and any returned unifying counterexample, including a non-EOF conflict lookahead; the EOF marker does not consume a
+token. If no common sentence is found within those budgets, the result is marked `unifying: false` and contains the deterministic
+shortest reachability witness, which may exceed the unifying-search token budget. This distinction prevents a nonunifying
+diagnostic from being presented as proof of ambiguity.
+
+## E7: Ruby DSL
+
+`Ibex::Frontend::DSL.grammar` constructs the existing AST with synthetic locations and joins the normal pipeline at Normalizer.
+Text and DSL inputs therefore share symbol classification, EBNF expansion, validation, algorithms, and outputs.
+
+## E9: construction algorithms
+
+The builder first constructs canonical LR(1). `lr1` retains the states, `lalr` merges equal LR(0) cores, and `slr` applies FOLLOW
+sets to completed items in the LR(0) states. All strategies use the same conflict resolver and Automaton IR.