ibex 0.1.0

data/docs/racc-migration.md

@@ -0,0 +1,47 @@
+# Migrating from racc
+
+Ibex targets grammar-file compatibility, generated parser public API compatibility, and the main racc CLI options. It does not
+copy racc's internal table arrays, internal method names, native runtime, or generated source layout.
+
+## Typical migration
+
+1. Run `ibex -o parser.rb grammar.y` in place of `racc -o parser.rb grammar.y`.
+2. Change the generated-file runtime dependency from deployment packaging only; application calls to `do_parse`, `yyparse`,
+   `next_token`, `on_error`, `token_to_str`, `yyerror`, `yyerrok`, and `yyaccept` remain the same.
+3. Use `-E` if the generated parser must be a single file with no installed Ibex gem.
+4. Keep the default `--mode=racc` until intentionally adopting EBNF or names. Extended grammars can make that choice locally by
+   placing `pragma extended` immediately after their class header instead of requiring `--mode=extended` at each invocation.
+
+## CLI mapping
+
+| Option | Ibex behavior |
+|---|---|
+| `-o`, `--output-file` | Select generated parser path |
+| `-t`, `--debug`; `-g` | Generate a debug-capable parser; `-g` is accepted as an obsolete alias |
+| `-v`, `-O` | Write the independent state report and optional path |
+| `-e [RUBY]` | Add a shebang and executable permission |
+| `-E`, `--embedded` | Embed the Pure Ruby runtime |
+| `-F`, `--frozen` | Accepted; Ibex always emits frozen-string magic comments |
+| `--rbs[=FILE]` | Ibex extension; emit a generated parser signature |
+| `--warnings=all,error` | Ibex extension; display or promote structured grammar diagnostics |
+| `--line-convert-all`, `-l` | Map header/inner/footer too, or disable all source mapping |
+| `-a` | Generate methods for implicit actions |
+| `--superclass` | Override the grammar superclass |
+| `-C`, `-S` | Check only; show pipeline status |
+| `-P`, `-D` | Accepted no-ops because they expose generator internals |
+| `--version`, `--runtime-version`, `--copyright`, `--help` | Informational output |
+
+Ibex defaults to `<input>.rb`; racc 1.8.1 was observed to default to `<input>.tab.rb`. Use `-o` for portable scripts.
+By default, semantic actions and `inner` methods report grammar-file lines while `header` and `footer` retain generated-file
+lines. `--line-convert-all` maps every user-code section; `-l` maps none. The mapping is retained through IR JSON resumption.
+
+## Known differences
+
+- Generated source and internal table representations are intentionally different.
+- `.output` report formatting is independent and contains additional resolved-conflict and witness data.
+- An undeclared invalid token was observed to enter racc's `error` production without `on_error`. Ibex intentionally reports the
+  unknown lookahead through `on_error` first, then recovers if the callback returns. Declared invalid tokens match in the
+  black-box recovery probe.
+- `require "racc/parser"` replacement and previously generated racc parser table compatibility are out of scope.
+
+See [compatibility observations](compat-notes.md) for the tested version and probe set.

data/exe/ibex

@@ -0,0 +1,6 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require "ibex/cli"
+
+exit Ibex::CLI.start(ARGV)

data/gemfiles/Gemfile

@@ -0,0 +1,7 @@
+# frozen_string_literal: true
+
+source "https://rubygems.org"
+
+gem "rbs", "4.0.3", require: false
+gem "rbs-inline", "0.14.0", require: false
+gem "steep", "2.0.0", require: false

data/gemfiles/Gemfile.lock

@@ -0,0 +1,98 @@
+GEM
+  remote: https://rubygems.org/
+  specs:
+    ast (2.4.3)
+    concurrent-ruby (1.3.8)
+    csv (3.3.5)
+    ffi (1.17.4)
+    ffi (1.17.4-arm64-darwin)
+    fileutils (1.8.0)
+    json (2.21.1)
+    language_server-protocol (3.17.0.6)
+    listen (3.10.0)
+      logger
+      rb-fsevent (~> 0.10, >= 0.10.3)
+      rb-inotify (~> 0.9, >= 0.9.10)
+    logger (1.7.0)
+    parser (3.3.12.0)
+      ast (~> 2.4.1)
+      racc
+    prism (1.9.0)
+    racc (1.8.1)
+    rainbow (3.1.1)
+    rb-fsevent (0.11.2)
+    rb-inotify (0.11.1)
+      ffi (~> 1.0)
+    rbs (4.0.3)
+      logger
+      prism (>= 1.6.0)
+      tsort
+    rbs-inline (0.14.0)
+      prism (>= 0.29)
+      rbs (~> 4.0)
+    securerandom (0.4.1)
+    steep (2.0.0)
+      concurrent-ruby (>= 1.1.10)
+      csv (>= 3.0.9)
+      fileutils (>= 1.1.0)
+      json (>= 2.1.0)
+      language_server-protocol (>= 3.17.0.4, < 4.0)
+      listen (~> 3.0)
+      logger (>= 1.3.0)
+      parser (>= 3.2)
+      prism (>= 0.25.0)
+      rainbow (>= 2.2.2, < 4.0)
+      rbs (~> 4.0)
+      securerandom (>= 0.1)
+      strscan (>= 1.0.0)
+      terminal-table (>= 2, < 5)
+      uri (>= 0.12.0)
+    strscan (3.1.8)
+    terminal-table (4.0.0)
+      unicode-display_width (>= 1.1.1, < 4)
+    tsort (0.2.0)
+    unicode-display_width (3.2.0)
+      unicode-emoji (~> 4.1)
+    unicode-emoji (4.2.0)
+    uri (1.1.1)
+
+PLATFORMS
+  arm64-darwin-24
+  ruby
+
+DEPENDENCIES
+  rbs (= 4.0.3)
+  rbs-inline (= 0.14.0)
+  steep (= 2.0.0)
+
+CHECKSUMS
+  ast (2.4.3) sha256=954615157c1d6a382bc27d690d973195e79db7f55e9765ac7c481c60bdb4d383
+  bundler (4.0.13) sha256=19f08be7f27022cf0b89f27da0b044ae075e8270a9ef44ad248a932614e1ca3b
+  concurrent-ruby (1.3.8) sha256=b2f1be836e968ccc78ccfce277ea79c72a88633f22306782c16ff23fb415d1e1
+  csv (3.3.5) sha256=6e5134ac3383ef728b7f02725d9872934f523cb40b961479f69cf3afa6c8e73f
+  ffi (1.17.4) sha256=bcd1642e06f0d16fc9e09ac6d49c3a7298b9789bcb58127302f934e437d60acf
+  ffi (1.17.4-arm64-darwin) sha256=19071aaf1419251b0a46852abf960e77330a3b334d13a4ab51d58b31a937001b
+  fileutils (1.8.0) sha256=8c6b1df54e2540bdb2f39258f08af78853aa70bad52b4d394bbc6424593c6e02
+  json (2.21.1) sha256=13a43df75d95641443f5702dff350f237164a9d811ff0f2c2800d4d980220583
+  language_server-protocol (3.17.0.6) sha256=5ef2c0c138f8267e1bc631d3328347d354f96724b0af22f2c79516120443b7f0
+  listen (3.10.0) sha256=c6e182db62143aeccc2e1960033bebe7445309c7272061979bb098d03760c9d2
+  logger (1.7.0) sha256=196edec7cc44b66cfb40f9755ce11b392f21f7967696af15d274dde7edff0203
+  parser (3.3.12.0) sha256=21a6d7f755d5a24dfbdc6e6b772e4e879a52e7631a88bc5a3a134606052c9828
+  prism (1.9.0) sha256=7b530c6a9f92c24300014919c9dcbc055bf4cdf51ec30aed099b06cd6674ef85
+  racc (1.8.1) sha256=4a7f6929691dbec8b5209a0b373bc2614882b55fc5d2e447a21aaa691303d62f
+  rainbow (3.1.1) sha256=039491aa3a89f42efa1d6dec2fc4e62ede96eb6acd95e52f1ad581182b79bc6a
+  rb-fsevent (0.11.2) sha256=43900b972e7301d6570f64b850a5aa67833ee7d87b458ee92805d56b7318aefe
+  rb-inotify (0.11.1) sha256=a0a700441239b0ff18eb65e3866236cd78613d6b9f78fea1f9ac47a85e47be6e
+  rbs (4.0.3) sha256=5a7bf70e2628549d9a1f44eae447b2cfe55968a9c60cfff52693a4bdcc020e14
+  rbs-inline (0.14.0) sha256=eaf47b46690d63acddab1f6804c9cc205a4bc78e637cfc421a0b1239874ef51c
+  securerandom (0.4.1) sha256=cc5193d414a4341b6e225f0cb4446aceca8e50d5e1888743fac16987638ea0b1
+  steep (2.0.0) sha256=6eb0ecc09637bbb54f0a5f2cf63daea6d3208ccace64b4f1107d976333605c30
+  strscan (3.1.8) sha256=aae2db611a225559f21ffbb71765c9a4e60fd262534a9ea84f4f11c7f32f679e
+  terminal-table (4.0.0) sha256=f504793203f8251b2ea7c7068333053f0beeea26093ec9962e62ea79f94301d2
+  tsort (0.2.0) sha256=9650a793f6859a43b6641671278f79cfead60ac714148aabe4e3f0060480089f
+  unicode-display_width (3.2.0) sha256=0cdd96b5681a5949cdbc2c55e7b420facae74c4aaf9a9815eee1087cb1853c42
+  unicode-emoji (4.2.0) sha256=519e69150f75652e40bf736106cfbc8f0f73aa3fb6a65afe62fefa7f80b0f80f
+  uri (1.1.1) sha256=379fa58d27ffb1387eaada68c749d1426738bd0f654d812fcc07e7568f5c57c6
+
+BUNDLED WITH
+  4.0.13

data/lib/ibex/analysis/sets.rb

@@ -0,0 +1,194 @@
+# frozen_string_literal: true
+
+module Ibex
+  module Analysis
+    # Computes nullable, FIRST, and FOLLOW sets over Grammar IR using integer bitsets.
+    class Sets
+      attr_reader :nullable_bits #: Integer
+      attr_reader :first_bits #: Array[Integer]
+      attr_reader :follow_bits #: Array[Integer]
+
+      # @rbs @grammar: IR::Grammar
+
+      # @rbs (IR::Grammar grammar) -> void
+      def initialize(grammar)
+        @grammar = grammar
+        @nullable_bits = 0
+        @first_bits = Array.new(grammar.symbols.length, 0)
+        @follow_bits = Array.new(grammar.symbols.length, 0)
+        grammar.terminals.each { |terminal| @first_bits[terminal.id] = bit(terminal.id) }
+        compute_nullable
+        compute_first
+        compute_follow
+      end
+
+      # @rbs (Integer | String | Symbol | IR::GrammarSymbol symbol) -> bool
+      def nullable?(symbol)
+        id = symbol_id(symbol)
+        @nullable_bits.anybits?(bit(id))
+      end
+
+      # @rbs (Integer | String | Symbol | IR::GrammarSymbol symbol) -> Array[String]
+      def first(symbol)
+        terminal_names(@first_bits.fetch(symbol_id(symbol)))
+      end
+
+      # @rbs (Integer | String | Symbol | IR::GrammarSymbol symbol) -> Array[String]
+      def follow(symbol)
+        definition = definition_for(symbol)
+        raise Ibex::Error, "(analysis):1:1: FOLLOW is only defined for nonterminals" unless definition.nonterminal?
+
+        terminal_names(@follow_bits.fetch(definition.id))
+      end
+
+      # @rbs (Array[Integer] symbol_ids) -> Integer
+      def first_of_sequence(symbol_ids)
+        bits = 0
+        symbol_ids.each do |id|
+          bits |= @first_bits.fetch(id)
+          return bits unless nullable_id?(id)
+        end
+        bits
+      end
+
+      # @rbs (Array[Integer] symbol_ids) -> bool
+      def sequence_nullable?(symbol_ids)
+        symbol_ids.all? { |id| nullable_id?(id) }
+      end
+
+      private
+
+      # @rbs () -> void
+      def compute_nullable
+        dependencies, remaining, queue = nullable_worklist
+
+        until queue.empty?
+          id = queue.shift
+          next if nullable_id?(id)
+
+          @nullable_bits |= bit(id)
+          dependencies[id].each do |production|
+            remaining[production.id] -= 1
+            queue << production.lhs if remaining[production.id].zero?
+          end
+        end
+      end
+
+      # @rbs () -> [Array[Array[IR::Production]], Array[Integer], Array[Integer]]
+      def nullable_worklist
+        dependencies = Array.new(@grammar.symbols.length) { [] }
+        remaining = Array.new(@grammar.productions.length, 0)
+        queue = [] #: Array[Integer]
+        @grammar.productions.each do |production|
+          next if production.rhs.any? { |id| required_symbol_by_id(id).terminal? }
+
+          remaining[production.id] = production.rhs.length
+          production.rhs.each { |id| dependencies[id] << production }
+          queue << production.lhs if production.rhs.empty?
+        end
+        [dependencies, remaining, queue]
+      end
+
+      # @rbs () -> void
+      def compute_first
+        dependencies = Array.new(@grammar.symbols.length) { [] }
+        @grammar.productions.each do |production|
+          production.rhs.each do |id|
+            dependencies[id] << production.lhs
+            break unless nullable_id?(id)
+          end
+        end
+
+        propagate_bits(@first_bits, dependencies, @grammar.terminals.map(&:id))
+      end
+
+      # @rbs () -> void
+      def compute_follow
+        start_definition = @grammar.symbol(@grammar.start)
+        raise Ibex::Error, "(analysis):1:1: unknown start symbol #{@grammar.start}" unless start_definition
+
+        start_id = start_definition.id
+        @follow_bits[start_id] |= bit(0)
+        dependencies = Array.new(@grammar.symbols.length) { [] }
+        @grammar.productions.each { |production| initialize_follow(production, dependencies) }
+        seeds = @grammar.nonterminals.filter_map { |symbol| symbol.id unless @follow_bits[symbol.id].zero? }
+        propagate_bits(@follow_bits, dependencies, seeds)
+      end
+
+      # @rbs (IR::Production production, Array[Array[Integer]] dependencies) -> void
+      def initialize_follow(production, dependencies)
+        trailer = 0
+        suffix_nullable = true
+        production.rhs.reverse_each do |id|
+          definition = required_symbol_by_id(id)
+          if definition.nonterminal?
+            @follow_bits[id] |= trailer
+            dependencies[production.lhs] << id if suffix_nullable
+            trailer = @first_bits[id] | (nullable_id?(id) ? trailer : 0)
+            suffix_nullable &&= nullable_id?(id)
+          else
+            trailer = @first_bits[id]
+            suffix_nullable = false
+          end
+        end
+      end
+
+      # @rbs (Array[Integer] sets, Array[Array[Integer]] dependencies, Array[Integer] seeds) -> void
+      def propagate_bits(sets, dependencies, seeds)
+        queue = seeds.dup
+        queued = Array.new(@grammar.symbols.length, false)
+        queue.each { |id| queued[id] = true }
+
+        until queue.empty?
+          source = queue.shift
+          queued[source] = false
+          dependencies[source].each do |target|
+            combined = sets[target] | sets[source]
+            next if combined == sets[target]
+
+            sets[target] = combined
+            next if queued[target]
+
+            queued[target] = true
+            queue << target
+          end
+        end
+      end
+
+      # @rbs (Integer id) -> bool
+      def nullable_id?(id)
+        @nullable_bits.anybits?(bit(id))
+      end
+
+      # @rbs (Integer bits) -> Array[String]
+      def terminal_names(bits)
+        @grammar.terminals.filter_map { |terminal| terminal.name if bits.anybits?(bit(terminal.id)) }
+      end
+
+      # @rbs (Integer | String | Symbol | IR::GrammarSymbol symbol) -> Integer
+      def symbol_id(symbol)
+        definition_for(symbol).id
+      end
+
+      # @rbs (Integer | String | Symbol | IR::GrammarSymbol symbol) -> IR::GrammarSymbol
+      def definition_for(symbol)
+        return symbol if symbol.is_a?(IR::GrammarSymbol)
+
+        definition = symbol.is_a?(Integer) ? @grammar.symbol_by_id(symbol) : @grammar.symbol(symbol.to_s)
+        return definition if definition
+
+        raise Ibex::Error, "(analysis):1:1: unknown symbol #{symbol}"
+      end
+
+      # @rbs (Integer id) -> IR::GrammarSymbol
+      def required_symbol_by_id(id)
+        @grammar.symbol_by_id(id) || raise(Ibex::Error, "(analysis):1:1: unknown symbol id #{id}")
+      end
+
+      # @rbs (Integer id) -> Integer
+      def bit(id)
+        1 << id
+      end
+    end
+  end
+end

data/lib/ibex/analysis.rb

@@ -0,0 +1,8 @@
+# frozen_string_literal: true
+
+require_relative "analysis/sets"
+
+module Ibex
+  module Analysis
+  end
+end

data/lib/ibex/cli/counterexample_options.rb

@@ -0,0 +1,38 @@
+# frozen_string_literal: true
+
+module Ibex
+  # Counterexample report limits and their command-line validation.
+  module CLICounterexampleOptions
+    DEFAULTS = {
+      counterexample_max_tokens: LALR::Counterexample::DEFAULT_MAX_TOKENS,
+      counterexample_max_configurations: LALR::Counterexample::DEFAULT_MAX_CONFIGURATIONS
+    }.freeze #: Hash[Symbol, Integer]
+
+    private
+
+    # @rbs (OptionParser options) -> void
+    def add_counterexample_options(options)
+      options.on("--counterexamples", "include conflict counterexamples in a report") { @options[:verbose] = true }
+      options.on("--counterexample-max-tokens=N", Integer, "maximum counterexample search token budget") do |value|
+        @options[:counterexample_max_tokens] = positive_counterexample_limit(value, "--counterexample-max-tokens")
+      end
+      options.on(
+        "--counterexample-max-configurations=N", Integer, "maximum counterexample search configuration budget"
+      ) do |value|
+        @options[:counterexample_max_configurations] = positive_counterexample_limit(
+          value, "--counterexample-max-configurations"
+        )
+      end
+    end
+
+    # @rbs (Integer value, String option) -> Integer
+    def positive_counterexample_limit(value, option)
+      if value.positive?
+        @options[:verbose] = true
+        return value
+      end
+
+      raise Ibex::Error, "(cli):1:1: #{option} must be positive"
+    end
+  end
+end

data/lib/ibex/cli/outputs.rb

@@ -0,0 +1,88 @@
+# frozen_string_literal: true
+
+module Ibex
+  # Files, warnings, and progress emitted by CLI pipeline stages.
+  module CLIOutputs
+    WARNING_MESSAGES = {
+      undeclared_terminal: ->(warning) { "undeclared terminal #{warning[:symbol]}" },
+      unused_terminal: ->(warning) { "unused terminal #{warning[:symbol]}" },
+      unreachable_nonterminal: ->(warning) { "unreachable nonterminal #{warning[:symbol]}" },
+      duplicate_production: lambda do |warning|
+        "duplicate production #{warning[:production]} (first defined as #{warning[:original]})"
+      end,
+      empty_language: ->(warning) { "start symbol #{warning[:symbol]} derives no terminal sentence" }
+    }.freeze #: Hash[Symbol, ^(IR::grammar_warning) -> String]
+
+    private
+
+    # @rbs (String value) -> Array[Symbol]
+    def warning_categories(value)
+      categories = value.split(",").map(&:strip).reject(&:empty?).map(&:to_sym)
+      unknown = categories - %i[all error none]
+      raise OptionParser::InvalidArgument, "unknown warning category #{unknown.first}" if unknown.any?
+      if categories.empty? || (categories.include?(:none) && categories.length > 1)
+        raise OptionParser::InvalidArgument, "warning category none cannot be combined"
+      end
+
+      categories
+    end
+
+    # @rbs (IR::Grammar grammar, String input_path) -> void
+    def handle_grammar_warnings(grammar, input_path)
+      categories = @options[:warnings]
+      return if categories.nil? || categories.include?(:none) || grammar.warnings.empty?
+
+      messages = grammar.warnings.map { |warning| format_grammar_warning(warning, input_path) }
+      if categories.include?(:error)
+        promoted = messages.map { |message| message.sub(": warning:", ": warning treated as error:") }
+        raise Ibex::Error, promoted.join("\n")
+      end
+
+      messages.each { |message| @stderr.puts(message) }
+    end
+
+    # @rbs (IR::grammar_warning warning, String input_path) -> String
+    def format_grammar_warning(warning, input_path)
+      location = warning[:loc]
+      rendered = if location
+                   "#{location[:file] || input_path}:#{location[:line] || 1}:#{location[:column] || 1}"
+                 else
+                   "#{input_path}:1:1"
+                 end
+      formatter = WARNING_MESSAGES.fetch(warning[:type]) { ->(item) { item[:type].to_s.tr("_", " ") } }
+      "#{rendered}: warning: #{formatter.call(warning)}"
+    end
+
+    # @rbs (IR::Automaton automaton, String input_path) -> void
+    def report_conflicts(automaton, input_path)
+      summary = automaton.conflict_summary
+      unless summary[:expectation_met]
+        @stderr.puts("#{input_path}:1:1: #{summary[:sr]} shift/reduce conflicts; expected #{summary[:expected_sr]}")
+      end
+      @stderr.puts("#{input_path}:1:1: #{summary[:rr]} reduce/reduce conflicts") if summary[:rr].positive?
+    end
+
+    # @rbs (IR::Automaton automaton, String input_path) -> void
+    def write_report(automaton, input_path)
+      path = @options[:log_file] || default_output_path(input_path, ".output")
+      report = Codegen::Report.render(
+        automaton,
+        max_tokens: @options[:counterexample_max_tokens],
+        max_configurations: @options[:counterexample_max_configurations]
+      )
+      File.write(path, report)
+      report_status("wrote #{path}")
+    end
+
+    # @rbs (String input_path, String extension) -> String
+    def default_output_path(input_path, extension)
+      replaced = input_path.sub(/\.[^.]+\z/, extension)
+      replaced == input_path ? "#{input_path}#{extension}" : replaced
+    end
+
+    # @rbs (String message) -> void
+    def report_status(message)
+      @stderr.puts("ibex: #{message}") if @options[:status]
+    end
+  end
+end