ibex 0.1.0 → 0.3.0

data/.rubocop.yml

@@ -1,43 +0,0 @@
-AllCops:
-  CacheRootDirectory: tmp/rubocop_cache
-  NewCops: enable
-  TargetRubyVersion: 3.0
-  SuggestExtensions: false
-  Exclude:
-    - "pkg/**/*"
-    - "tmp/**/*"
-    - "vendor/**/*"
-    - "lib/ibex/frontend/generated_parser.rb"
-
-Layout/LineLength:
-  Max: 120
-
-Layout/LeadingCommentSpace:
-  AllowRBSInlineAnnotation: true
-
-Style/StringLiterals:
-  EnforcedStyle: double_quotes
-
-Metrics/AbcSize:
-  Max: 36
-
-Metrics/ClassLength:
-  Max: 200
-
-Metrics/CyclomaticComplexity:
-  Max: 10
-
-Metrics/MethodLength:
-  Max: 30
-
-Metrics/ModuleLength:
-  Max: 150
-
-Metrics/ParameterLists:
-  Max: 12
-
-Metrics/PerceivedComplexity:
-  Max: 10
-
-Style/Documentation:
-  Enabled: false

data/CHANGELOG.md

@@ -1,30 +0,0 @@
-# Changelog
-
-## Unreleased
-
-- Establish the Pure Ruby gem, Minitest, lint, CI, and CLI foundations.
-- Add a table-driven Pure Ruby LR runtime with pull/push APIs, recovery, and tracing.
-- Tokenize grammar files with positions, opaque Ruby actions, comments, and user-code blocks.
-- Parse compatible and extended grammar syntax into a serializable, location-preserving AST.
-- Normalize grammars into immutable, versioned JSON IR with EBNF expansion and diagnostics.
-- Compute nullable, FIRST, and FOLLOW sets with deterministic integer bitsets.
-- Build deterministic LALR(1) automata, resolve and retain conflicts, and render state reports.
-- Generate plain or compact Ruby parsers with embedded runtime and source-line mapping support.
-- Preserve per-block user-code source maps through IR and implement compatible default, all-code, and disabled line conversion.
-- Complete the compatible CLI surface and add optional racc black-box result comparisons.
-- Add extended EBNF/named-reference integration, source-facing EBNF labels in text/DOT/HTML reports, and resumable IR pipelines.
-- Add selectable SLR, LALR(1), and canonical LR(1) construction strategies.
-- Add a Ruby DSL frontend that converges on the existing Grammar AST and IR pipeline.
-- Add unifying conflict counterexamples with complete competing derivation trees and bounded nonunifying fallback.
-- Add generated parser RBS output, shipped runtime signatures, and strict structured grammar diagnostics.
-- Expand generated signatures and Steep checking to every library source, including concrete frontend/IR/LALR/codegen/CLI domain
-  types and the self-hosted parser.
-- Support quoted, interpolated, and multiple Ruby heredocs plus recursively nested grouped EBNF.
-- Define unknown external tokens to invoke `on_error` before yacc-style recovery.
-- Add post-commit shift, reduction, and successful-recovery observation hooks to the runtime.
-- Add complete quickstart, grammar, migration, architecture, and extension documentation with an executable README test.
-- Self-host the grammar frontend from a committed Ibex grammar with deterministic regeneration and bootstrap parity checks.
-- Version generated parser tables and reject missing or unsupported formats before consuming input.
-- Add grammar-local `pragma extended` selection and document the action and named-reference boundary inside EBNF groups.
-- Add fixed-seed pipeline properties, versioned Grammar/Automaton IR fixtures, and a reproducible whole-builder benchmark.
-- Keep the whole-library Steep statistics in the README synchronized and enforce them in CI.

data/Rakefile

@@ -1,25 +0,0 @@
-# frozen_string_literal: true
-
-require "bundler/gem_tasks"
-require "rake/testtask"
-require "rubocop/rake_task"
-
-Rake::TestTask.new(:test) do |task|
-  task.libs << "test"
-  task.pattern = "test/**/*_test.rb"
-  task.warning = true
-end
-
-RuboCop::RakeTask.new(:lint)
-
-namespace :frontend do
-  desc "Regenerate the self-hosted grammar parser"
-  task :generate do
-    require_relative "lib/ibex/frontend/regenerator"
-
-    output = File.expand_path("lib/ibex/frontend/generated_parser.rb", __dir__)
-    File.write(output, Ibex::Frontend::Regenerator.generate)
-  end
-end
-
-task default: %i[test lint]

data/Steepfile

@@ -1,10 +0,0 @@
-# frozen_string_literal: true
-
-target :library do
-  signature "sig"
-  check "lib"
-
-  library "digest"
-  library "json"
-  library "optparse"
-end

data/docs/compat-notes.md

@@ -1,37 +0,0 @@
-# Compatibility notes
-
-Compatibility observations are recorded here without consulting racc implementation sources or generated source text.
-
-## Observed CLI (2026-07-22)
-
-Black-box command: `racc --help`, version 1.8.1.
-
-Observed options are `-o/--output-file`, `-t/--debug`, obsolete `-g`, `-v/--verbose`, `-O/--log-file`,
-`-e/--executable`, `-E/--embedded`, `-F/--frozen`, `--line-convert-all`, `-l/--no-line-convert`,
-`-a/--no-omit-actions`, `--superclass`, `-C/--check-only`, `-S/--output-status`, profiling `-P`, internal `-D`,
-`--version`, `--runtime-version`, `--copyright`, and `--help`.
-
-Ibex accepts that option set. `-P` and `-D` are accepted compatibility no-ops because they expose generator internals rather
-than parser behavior. Frozen string literals are always emitted, so `-F` is also a no-op. Ibex's default output is `<input>.rb`
-rather than racc's observed `<input>.tab.rb`; `-o` is portable between both tools.
-
-## Behavioral probes
-
-Self-authored grammars are compiled independently and the generated files are executed without inspecting them. Current probes
-cover arithmetic precedence, empty rules, string tokens, `convert`, `no_result_var`, inline actions, dangling-else `expect`, and
-a generated 500-production grammar. Precedence-resolved conflicts remain in Automaton IR diagnostics but are not counted in CLI
-warnings or `expect`, matching the observed calculator behavior.
-
-An inline action's own `val[0]` was observed as `nil`; its result occupies one value position visible to the final action. The
-grammar reference confirms that `convert` takes a quoted string containing Ruby source: for example `NUM ':number'` emits the
-symbol expression, while `NUM '"number"'` emits a Ruby string token.
-
-Source-line probes with raises in self-authored `header`, `inner`, and `footer` blocks show three distinct modes. The default maps
-semantic actions and `inner` code to the grammar file but leaves `header` and `footer` on generated-file lines.
-`--line-convert-all` maps all three user-code sections, while `-l` leaves all of them on generated-file lines.
-
-With an explicitly declared but syntactically invalid `BAD` token, error recovery results and `on_error` observations (token
-string, value, and value-stack length) match. An undeclared `:BAD` was observed to enter racc's `error` production without an
-`on_error` callback. Ibex intentionally treats it as an unknown lookahead and calls `on_error` before attempting the same error
-recovery flow. This preserves the unexpected token object and value for logging and application policy instead of silently
-entering recovery.

data/docs/lexer-coverage.md

@@ -1,14 +0,0 @@
-# Lexer coverage
-
-| Construct | Status | Notes |
-|---|---|---|
-| Nested action braces | Supported | Balanced recursively |
-| Single, double, and backtick strings | Supported | Escapes and `#{...}` interpolation are skipped |
-| `%q`, `%Q`, `%w`, `%W`, `%i`, `%I`, `%x`, `%r`, `%s` | Supported | Paired and single-character delimiters |
-| Regular expressions | Supported | Conservative expression-prefix heuristic; character classes and flags supported |
-| `#` comments and `?x` characters | Supported | Ignored for brace balancing |
-| `<<ID`, `<<-ID`, `<<~ID` heredocs | Supported | Terminators obey indentation mode |
-| Single/double/backtick quoted heredocs | Supported | Quoted identifiers may contain any non-quote, non-newline text |
-| Interpolated and multiple heredocs | Supported | Bodies are opaque; multiple openers on one line are consumed in order |
-| Grammar `#` and `/* ... */` comments | Supported | Removed before tokenization |
-| `---- header`, `inner`, `footer` | Supported | Must begin in column one; order and duplicates retained |

data/docs/phase10-extensions.md

@@ -1,27 +0,0 @@
-# Phase 10 extensions
-
-## E3: conflict witnesses
-
-`Ibex::LALR::Counterexample` accepts only Automaton IR. It explores parser-stack configurations, forces each pair of competing
-actions at the requested conflict, and then searches for a common suffix that lets both branches accept the same terminal
-sentence. A successful result contains `unifying: true`, the conflict lookahead position, and both complete derivation trees.
-
-Because unifying-counterexample search is not guaranteed to terminate for every context-free grammar, the search has explicit
-token and configuration budgets. The defaults are 32 tokens and 50,000 explored configurations. Pass `max_tokens:` and
-`max_configurations:` to `Ibex::LALR::Counterexample.new` or `Ibex::Codegen::Report.render` to adjust them through the Ruby API.
-For generated reports, use the positive-integer CLI options `--counterexample-max-tokens=N` and
-`--counterexample-max-configurations=N`; either option also requests a report. The token budget covers each unifying-search
-candidate and any returned unifying counterexample, including a non-EOF conflict lookahead; the EOF marker does not consume a
-token. If no common sentence is found within those budgets, the result is marked `unifying: false` and contains the deterministic
-shortest reachability witness, which may exceed the unifying-search token budget. This distinction prevents a nonunifying
-diagnostic from being presented as proof of ambiguity.
-
-## E7: Ruby DSL
-
-`Ibex::Frontend::DSL.grammar` constructs the existing AST with synthetic locations and joins the normal pipeline at Normalizer.
-Text and DSL inputs therefore share symbol classification, EBNF expansion, validation, algorithms, and outputs.
-
-## E9: construction algorithms
-
-The builder first constructs canonical LR(1). `lr1` retains the states, `lalr` merges equal LR(0) cores, and `slr` applies FOLLOW
-sets to completed items in the LR(0) states. All strategies use the same conflict resolver and Automaton IR.

data/gemfiles/Gemfile

@@ -1,7 +0,0 @@
-# frozen_string_literal: true
-
-source "https://rubygems.org"
-
-gem "rbs", "4.0.3", require: false
-gem "rbs-inline", "0.14.0", require: false
-gem "steep", "2.0.0", require: false

data/gemfiles/Gemfile.lock

@@ -1,98 +0,0 @@
-GEM
-  remote: https://rubygems.org/
-  specs:
-    ast (2.4.3)
-    concurrent-ruby (1.3.8)
-    csv (3.3.5)
-    ffi (1.17.4)
-    ffi (1.17.4-arm64-darwin)
-    fileutils (1.8.0)
-    json (2.21.1)
-    language_server-protocol (3.17.0.6)
-    listen (3.10.0)
-      logger
-      rb-fsevent (~> 0.10, >= 0.10.3)
-      rb-inotify (~> 0.9, >= 0.9.10)
-    logger (1.7.0)
-    parser (3.3.12.0)
-      ast (~> 2.4.1)
-      racc
-    prism (1.9.0)
-    racc (1.8.1)
-    rainbow (3.1.1)
-    rb-fsevent (0.11.2)
-    rb-inotify (0.11.1)
-      ffi (~> 1.0)
-    rbs (4.0.3)
-      logger
-      prism (>= 1.6.0)
-      tsort
-    rbs-inline (0.14.0)
-      prism (>= 0.29)
-      rbs (~> 4.0)
-    securerandom (0.4.1)
-    steep (2.0.0)
-      concurrent-ruby (>= 1.1.10)
-      csv (>= 3.0.9)
-      fileutils (>= 1.1.0)
-      json (>= 2.1.0)
-      language_server-protocol (>= 3.17.0.4, < 4.0)
-      listen (~> 3.0)
-      logger (>= 1.3.0)
-      parser (>= 3.2)
-      prism (>= 0.25.0)
-      rainbow (>= 2.2.2, < 4.0)
-      rbs (~> 4.0)
-      securerandom (>= 0.1)
-      strscan (>= 1.0.0)
-      terminal-table (>= 2, < 5)
-      uri (>= 0.12.0)
-    strscan (3.1.8)
-    terminal-table (4.0.0)
-      unicode-display_width (>= 1.1.1, < 4)
-    tsort (0.2.0)
-    unicode-display_width (3.2.0)
-      unicode-emoji (~> 4.1)
-    unicode-emoji (4.2.0)
-    uri (1.1.1)
-
-PLATFORMS
-  arm64-darwin-24
-  ruby
-
-DEPENDENCIES
-  rbs (= 4.0.3)
-  rbs-inline (= 0.14.0)
-  steep (= 2.0.0)
-
-CHECKSUMS
-  ast (2.4.3) sha256=954615157c1d6a382bc27d690d973195e79db7f55e9765ac7c481c60bdb4d383
-  bundler (4.0.13) sha256=19f08be7f27022cf0b89f27da0b044ae075e8270a9ef44ad248a932614e1ca3b
-  concurrent-ruby (1.3.8) sha256=b2f1be836e968ccc78ccfce277ea79c72a88633f22306782c16ff23fb415d1e1
-  csv (3.3.5) sha256=6e5134ac3383ef728b7f02725d9872934f523cb40b961479f69cf3afa6c8e73f
-  ffi (1.17.4) sha256=bcd1642e06f0d16fc9e09ac6d49c3a7298b9789bcb58127302f934e437d60acf
-  ffi (1.17.4-arm64-darwin) sha256=19071aaf1419251b0a46852abf960e77330a3b334d13a4ab51d58b31a937001b
-  fileutils (1.8.0) sha256=8c6b1df54e2540bdb2f39258f08af78853aa70bad52b4d394bbc6424593c6e02
-  json (2.21.1) sha256=13a43df75d95641443f5702dff350f237164a9d811ff0f2c2800d4d980220583
-  language_server-protocol (3.17.0.6) sha256=5ef2c0c138f8267e1bc631d3328347d354f96724b0af22f2c79516120443b7f0
-  listen (3.10.0) sha256=c6e182db62143aeccc2e1960033bebe7445309c7272061979bb098d03760c9d2
-  logger (1.7.0) sha256=196edec7cc44b66cfb40f9755ce11b392f21f7967696af15d274dde7edff0203
-  parser (3.3.12.0) sha256=21a6d7f755d5a24dfbdc6e6b772e4e879a52e7631a88bc5a3a134606052c9828
-  prism (1.9.0) sha256=7b530c6a9f92c24300014919c9dcbc055bf4cdf51ec30aed099b06cd6674ef85
-  racc (1.8.1) sha256=4a7f6929691dbec8b5209a0b373bc2614882b55fc5d2e447a21aaa691303d62f
-  rainbow (3.1.1) sha256=039491aa3a89f42efa1d6dec2fc4e62ede96eb6acd95e52f1ad581182b79bc6a
-  rb-fsevent (0.11.2) sha256=43900b972e7301d6570f64b850a5aa67833ee7d87b458ee92805d56b7318aefe
-  rb-inotify (0.11.1) sha256=a0a700441239b0ff18eb65e3866236cd78613d6b9f78fea1f9ac47a85e47be6e
-  rbs (4.0.3) sha256=5a7bf70e2628549d9a1f44eae447b2cfe55968a9c60cfff52693a4bdcc020e14
-  rbs-inline (0.14.0) sha256=eaf47b46690d63acddab1f6804c9cc205a4bc78e637cfc421a0b1239874ef51c
-  securerandom (0.4.1) sha256=cc5193d414a4341b6e225f0cb4446aceca8e50d5e1888743fac16987638ea0b1
-  steep (2.0.0) sha256=6eb0ecc09637bbb54f0a5f2cf63daea6d3208ccace64b4f1107d976333605c30
-  strscan (3.1.8) sha256=aae2db611a225559f21ffbb71765c9a4e60fd262534a9ea84f4f11c7f32f679e
-  terminal-table (4.0.0) sha256=f504793203f8251b2ea7c7068333053f0beeea26093ec9962e62ea79f94301d2
-  tsort (0.2.0) sha256=9650a793f6859a43b6641671278f79cfead60ac714148aabe4e3f0060480089f
-  unicode-display_width (3.2.0) sha256=0cdd96b5681a5949cdbc2c55e7b420facae74c4aaf9a9815eee1087cb1853c42
-  unicode-emoji (4.2.0) sha256=519e69150f75652e40bf736106cfbc8f0f73aa3fb6a65afe62fefa7f80b0f80f
-  uri (1.1.1) sha256=379fa58d27ffb1387eaada68c749d1426738bd0f654d812fcc07e7568f5c57c6
-
-BUNDLED WITH
-  4.0.13

data/lib/ibex/frontend/grammar.y

@@ -1,156 +0,0 @@
-class Ibex::Frontend::GeneratedParser < Ibex::Frontend::GeneratedParserBase
-token CLASS TOKEN PRECHIGH PRECLOW OPTIONS EXPECT START CONVERT PRAGMA RULE END
-token LEFT RIGHT NONASSOC IDENTIFIER LITERAL INTEGER ACTION USER_CODE LHS
-token SEPARATED_LIST SEPARATED_NONEMPTY_LIST
-rule
-  grammar
-    : CLASS constant_path superclass pragmas declarations RULE rules END user_code
-      { result = build_root(val[0], val[1], val[2], val[4], val[6], val[8]) }
-
-  constant_path
-    : IDENTIFIER                         { result = [val[0].value] }
-    | constant_path '::' IDENTIFIER      { result = val[0] + [val[2].value] }
-
-  superclass
-    :                                    { result = nil }
-    | '<' constant_path                  { result = val[1] }
-
-  pragmas
-    :
-    | pragmas PRAGMA IDENTIFIER
-
-  declarations
-    :                                    { result = Array.new(0) }
-    | declarations declaration           { result = val[0] + [val[1]] }
-
-  declaration
-    : token_declaration                  { result = val[0] }
-    | precedence_declaration             { result = val[0] }
-    | options_declaration                { result = val[0] }
-    | expect_declaration                 { result = val[0] }
-    | start_declaration                  { result = val[0] }
-    | convert_declaration                { result = val[0] }
-
-  token_declaration
-    : TOKEN symbols                      { result = build_tokens(val[0], val[1]) }
-
-  precedence_declaration
-    : PRECHIGH precedence_levels PRECLOW { result = build_precedence(val[0], :high_to_low, val[1]) }
-    | PRECLOW precedence_levels PRECHIGH { result = build_precedence(val[0], :low_to_high, val[1]) }
-
-  precedence_levels
-    :                                    { result = Array.new(0) }
-    | precedence_levels precedence_level { result = val[0] + [val[1]] }
-
-  precedence_level
-    : association symbols                { result = build_precedence_level(val[0], val[1]) }
-
-  association
-    : LEFT                               { result = val[0] }
-    | RIGHT                              { result = val[0] }
-    | NONASSOC                           { result = val[0] }
-
-  options_declaration
-    : OPTIONS identifiers                { result = build_options(val[0], val[1]) }
-
-  expect_declaration
-    : EXPECT INTEGER                     { result = build_expect(val[0], val[1]) }
-
-  start_declaration
-    : START grammar_symbol               { result = build_start(val[0], val[1].value) }
-
-  convert_declaration
-    : CONVERT conversions END            { result = build_convert(val[0], val[1]) }
-
-  conversions
-    :                                    { result = Array.new(0) }
-    | conversions conversion             { result = val[0] + [val[1]] }
-
-  conversion
-    : grammar_symbol LITERAL              { result = build_conversion(val[0], val[1]) }
-
-  identifiers
-    :                                    { result = Array.new(0) }
-    | identifiers IDENTIFIER              { result = val[0] + [val[1].value] }
-
-  symbols
-    :                                    { result = Array.new(0) }
-    | symbols grammar_symbol              { result = val[0] + [val[1].value] }
-
-  grammar_symbol
-    : IDENTIFIER                         { result = val[0] }
-    | LITERAL                            { result = val[0] }
-
-  rules
-    : rule_definition                    { result = [val[0]] }
-    | rules rule_definition              { result = val[0] + [val[1]] }
-
-  rule_definition
-    : LHS ':' alternatives semicolon     { result = build_rule(val[0], val[2]) }
-
-  semicolon
-    :
-    | ';'
-
-  alternatives
-    : alternative                        { result = [val[0]] }
-    | alternatives '|' alternative       { result = val[0] + [val[2]] }
-
-  alternative
-    : items precedence_override          { result = build_alternative(val[0], val[1]) }
-
-  precedence_override
-    :                                    { result = nil }
-    | '=' grammar_symbol                 { result = val[1] }
-
-  items
-    :                                    { result = Array.new(0) }
-    | items item                         { result = val[0] + [val[1]] }
-
-  item
-    : symbol_item                        { result = val[0] }
-    | ACTION                             { result = build_action(val[0]) }
-    | group_item                         { result = val[0] }
-    | separated_item                     { result = val[0] }
-
-  symbol_item
-    : grammar_symbol named_reference suffixes
-      { result = build_symbol_reference(val[0], val[1], val[2]) }
-
-  named_reference
-    :                                    { result = nil }
-    | ':' IDENTIFIER                     { result = [val[0], val[1]] }
-
-  suffixes
-    :                                    { result = Array.new(0) }
-    | suffixes '?'                       { result = val[0] + [val[1]] }
-    | suffixes '*'                       { result = val[0] + [val[1]] }
-    | suffixes '+'                       { result = val[0] + [val[1]] }
-
-  group_item
-    : '(' group_alternatives ')' suffixes
-      { result = build_group(val[0], val[1], val[3]) }
-
-  group_alternatives
-    : group_items                        { result = [val[0]] }
-    | group_alternatives '|' group_items { result = val[0] + [val[2]] }
-
-  group_items
-    :                                    { result = Array.new(0) }
-    | group_items group_item_element      { result = val[0] + [val[1]] }
-
-  group_item_element
-    : symbol_item                        { result = val[0] }
-    | group_item                         { result = val[0] }
-    | separated_item                     { result = val[0] }
-
-  separated_item
-    : SEPARATED_LIST '(' item ',' item ')'
-      { result = build_separated_list(val[0], val[2], val[4]) }
-    | SEPARATED_NONEMPTY_LIST '(' item ',' item ')'
-      { result = build_separated_list(val[0], val[2], val[4]) }
-
-  user_code
-    :                                    { result = empty_user_code }
-    | user_code USER_CODE                { result = append_user_code(val[0], val[1]) }
-end