flexr 1.0.0 → 1.1.0

data/docs/reference/runtime.md

@@ -10,14 +10,22 @@ Lexer.new(input,
   filename: nil,
   error_mode: :raise,
   max_token_size: 16 * 1024 * 1024,
+  max_lookahead_size: nil,
+  max_buffer_size: 64 * 1024 * 1024,
   max_state_stack: 1024,
+  max_steps: nil,
+  cancellation: nil,
+  retain_input: true,
   chunk_size: 64 * 1024)
 ```
 
 `input` must be a `String` or an object responding to `read`. `filename` is
 copied into locations and lexical errors. `error_mode` is `:raise`, `:token`,
 or `:panic`; unknown values fall back to raising when an unmatched byte is
-handled. Size limits are non-negative, and `chunk_size` must be positive.
+handled. `max_lookahead_size` defaults to `max_token_size`. `max_steps: nil`
+leaves the step count unbounded; `cancellation` may be a callable accepting
+zero arguments or the lexer. Size limits are non-negative, and `chunk_size`
+must be positive.
 
 ## Consumption
 
@@ -27,11 +35,13 @@ handled. Size limits are non-negative, and `chunk_size` must be positive.
 | `tokens` | Array containing every token until EOF |
 | `each_token` | Enumerator without a block; otherwise yields tokens and returns `self` |
 | `racc_next_token` | `[type, value]`, or `[false, "$end"]` at EOF |
-| `input` | The buffered input string |
+| `input` | The currently retained input string |
 | `buffer` | Internal buffered input object; use only for integrations that need it |
 | `filename` | The configured filename |
 | `error_mode` | The configured default input error mode |
 | `max_token_size` | The configured token-size limit |
+| `max_lookahead_size` | The configured trailing-context limit |
+| `steps` | Work units consumed by scanning |
 
 `each_token` yields two arguments instead of one when `token_kind :yield` is
 selected. `tokens` always returns an array of token values.
@@ -46,4 +56,11 @@ class or another lexer instance.
 
 IO input is read lazily in chunks. UTF-8 input is checked at codepoint
 boundaries, while binary input is consumed byte by byte. Invalid UTF-8 becomes
-an unmatched-byte error rather than being silently normalized.
+an unmatched-byte error rather than being silently normalized. With
+`retain_input: false`, bytes before the current token are released after each
+match; `input` and `binary_input` then expose only the retained window.
+
+`max_token_size` is checked before an action can shorten or materialize an
+oversized match. `max_lookahead_size` separately bounds trailing context, and
+`max_buffer_size` bounds bytes retained at one time. `max_steps` and
+`cancellation` bound CPU work cooperatively.

data/docs/reference/tokens-and-locations.md

@@ -24,10 +24,12 @@ token.
 | `line_begin` / `line_end` | One-based line range |
 | `column_begin` / `column_end` | One-based character columns for UTF-8, byte columns for binary input |
 
-Columns are computed lazily by default. `option :eager_columns` fills them while
-the location is created. The end line and column identify the position after
-the token's last byte, so a token ending at the start of the next line may have
-an end column of 1.
+Line and column positions are tracked incrementally while input is consumed,
+so creating locations does not rescan the input prefix. Column members are
+filled lazily by default; `option :eager_columns` fills them while the location
+is created. The end line and column identify the position after the token's
+last byte, so a token ending at the start of the next line may have an end
+column of 1.
 
 `more` extends the location from the first match to the final match. Trailing
 context is not part of the location because it is not consumed.

data/docs/releases/v1.1.0.md

@@ -0,0 +1,44 @@
+# flexr v1.1.0
+
+Release date: 2026-08-13
+
+v1.1.0 strengthens the runtime, generated, and standalone lexer contract while
+keeping the stable public API focused on deterministic, byte-oriented lexing.
+
+## Highlights
+
+- Runtime, generated, and standalone modes now share action precedence,
+  nested-state behavior, static-evaluation rules, immutable compiled
+  specifications, and structured error outcomes.
+- Runtime resource usage is bounded with token, lookahead, buffer, step, state
+  stack, cancellation, and non-progress guards. Streaming input supports
+  sliding-buffer retention and incremental location tracking.
+- Unicode properties and POSIX classes use the vendored Unicode 15.1.0 data
+  through the DFA path, including property-aware longest-match behavior and
+  Unicode-safe empty-match progress.
+- Regexp parsing and normalization now fail closed for unsupported syntax and
+  cover empty-language AST nodes, symbolic repetition, class intersections,
+  trailing hyphens, shorthand classes, and exact first-match counterexamples.
+- Compiler and generated-runtime paths use indexed construction, Hopcroft
+  minimization, lazy transition representations, backend cost selection, and
+  state-specific scanner fast paths.
+- Generated artifact writes are collision-safe, atomic, no-op aware, and carry
+  schema, runtime ABI, compiler, and Unicode metadata.
+
+## Compatibility notes
+
+- Stable backends remain `table`, `direct`, and `auto`.
+- `firstmatch` remains experimental and intentionally does not provide the
+  longest-match guarantee.
+- Generated files should be regenerated with flexr v1.1.0 when upgrading from
+  v1.0.x. Older generated artifacts remain loadable when their artifact
+  metadata is compatible.
+- Unicode behavior is pinned to the vendored Unicode 15.1.0 snapshot rather
+  than the host Ruby version.
+
+## Verification
+
+The release candidate passed the repository's RSpec, RuboCop, documentation,
+generation, parity, differential, fuzz, Unicode, coverage, benchmark, and gem
+build gates on the release environment. See [the release procedure](../RELEASING.md)
+for the commands to repeat before tagging and publishing.

data/lib/flexr/action_resolver.rb

@@ -0,0 +1,15 @@
+# frozen_string_literal: true
+
+module Flexr
+  module ActionResolver
+    module_function
+
+    def resolve(skip:, emit:, block:, default:)
+      return :skip if skip
+      return [:emit, emit.to_sym] if emit
+      return block if block
+
+      default
+    end
+  end
+end

data/lib/flexr/artifact_writer.rb

@@ -0,0 +1,61 @@
+# frozen_string_literal: true
+
+require "tempfile"
+
+module Flexr
+  module ArtifactWriter
+    module_function
+
+    def default_generated_path(source_path)
+      return source_path.sub(/\.flexr\.rb\z/, ".rb") if source_path.end_with?(".flexr.rb")
+      return source_path.sub(/\.rb\z/, ".generated.rb") if source_path.end_with?(".rb")
+
+      "#{source_path}.generated.rb"
+    end
+
+    def write!(path, content, source_path: nil)
+      ensure_distinct!(source_path, path) if source_path
+      return :unchanged if File.file?(path) && File.binread(path) == content.b
+
+      atomic_replace(path, content)
+      :written
+    end
+
+    def ensure_distinct!(source_path, output_path)
+      same_path = File.expand_path(source_path) == File.expand_path(output_path)
+      same_file = File.exist?(source_path) && File.exist?(output_path) && File.identical?(source_path, output_path)
+      return unless same_path || same_file
+
+      raise ArgumentError, "refusing to overwrite input file: #{source_path}"
+    end
+
+    def atomic_replace(path, content)
+      expanded = File.expand_path(path)
+      directory = File.dirname(expanded)
+      basename = File.basename(expanded)
+      mode = File.exist?(expanded) ? File.stat(expanded).mode & 0o7777 : 0o666 & ~File.umask
+      temporary_path = nil
+
+      Tempfile.create([".#{basename}.", ".tmp"], directory, binmode: true) do |temporary|
+        temporary_path = temporary.path
+        temporary.write(content)
+        temporary.flush
+        temporary.fsync
+        temporary.chmod(mode)
+        temporary.close
+        File.rename(temporary_path, expanded)
+        temporary_path = nil
+      end
+      sync_directory(directory)
+    ensure
+      File.unlink(temporary_path) if temporary_path && File.exist?(temporary_path)
+    end
+
+    def sync_directory(directory)
+      File.open(directory, File::RDONLY, &:fsync)
+    rescue SystemCallError, IOError
+      # Some supported filesystems do not allow fsync on directories. The file
+      # itself has already been synced before the atomic rename.
+    end
+  end
+end

data/lib/flexr/automaton/accel.rb

@@ -2,7 +2,7 @@
 
 module Flexr
   module Automaton
-    Region = Struct.new(:state, :bytes, :regexp, keyword_init: true)
+    Region = Struct.new(:state, :bytes, :regexp, :utf8_regexp, keyword_init: true)
 
     module Accel
       module_function
@@ -12,13 +12,18 @@ module Flexr
           bytes = Analysis.self_loop_set(dfa, state)
           next if bytes.empty?
 
-          Region.new(state: state, bytes: bytes.freeze, regexp: regexp_for(bytes))
+          Region.new(
+            state: state, bytes: bytes.freeze,
+            regexp: regexp_for(bytes, binary: true),
+            utf8_regexp: bytes.all? { |byte| byte < 128 } ? regexp_for(bytes, binary: false) : nil
+          )
         end
       end
 
-      def regexp_for(bytes)
+      def regexp_for(bytes, binary: true)
         source = bytes_to_source(bytes)
-        ::Regexp.new("(?:[#{source}])+", ::Regexp::NOENCODING)
+        options = binary ? ::Regexp::NOENCODING : 0
+        ::Regexp.new("(?:[#{source}])+", options)
       end
 
       def bytes_to_source(bytes)

data/lib/flexr/automaton/analysis.rb

@@ -19,13 +19,13 @@ module Flexr
       end
 
       def self_loop_set(dfa, state)
-        dfa.transitions[state].each_with_index.with_object([]) do |(destination, class_id), result|
-          next unless destination == state
-
-          dfa.ec.each_with_index do |value, byte|
-            result << byte if value == class_id
-          end
+        representatives = Array.new(dfa.class_count)
+        dfa.ec.each_with_index { |class_id, byte| representatives[class_id] ||= byte }
+        self_loops = representatives.each_index.map do |class_id|
+          byte = representatives.fetch(class_id)
+          byte && dfa.transition(state, byte) == state
         end
+        dfa.ec.each_index.select { |byte| self_loops.fetch(dfa.ec.fetch(byte)) }
       end
 
       def dead_states(dfa)
@@ -33,6 +33,47 @@ module Flexr
           dfa.accepts[state].empty? && dfa.transitions[state].compact.all? { |destination| destination == state }
         end
       end
+
+      def firstmatch_counterexample(first, second)
+        queue = [[first.start, second.start, false, +"".b]]
+        visited = { [first.start, second.start, false] => true }
+        bytes = joint_byte_representatives(first, second)
+        cursor = 0
+
+        while cursor < queue.length
+          first_state, second_state, first_seen, input = queue.fetch(cursor)
+          cursor += 1
+          bytes.each do |byte|
+            next_first = first_state && first.transition(first_state, byte)
+            next_second = second_state && second.transition(second_state, byte)
+            next unless next_second
+
+            next_input = input + byte.chr(Encoding::BINARY)
+            next_first_seen = first_seen || accepting?(first, next_first)
+            return next_input if next_first_seen && accepting?(second, next_second) && !accepting?(first, next_first)
+
+            key = [next_first, next_second, next_first_seen]
+            next if visited[key]
+
+            visited[key] = true
+            queue << [next_first, next_second, next_first_seen, next_input]
+          end
+        end
+        nil
+      end
+
+      def joint_byte_representatives(first, second)
+        representatives = {}
+        256.times do |byte|
+          key = [first.ec[byte], second.ec[byte]]
+          representatives[key] ||= byte
+        end
+        representatives.values.sort_by { |byte| [byte.between?(32, 126) ? 0 : 1, byte] }
+      end
+
+      def accepting?(dfa, state)
+        state && !dfa.accepts.fetch(state).empty?
+      end
     end
   end
 end

data/lib/flexr/automaton/backend_cost_model.rb

@@ -0,0 +1,40 @@
+# frozen_string_literal: true
+
+module Flexr
+  module Automaton
+    module BackendCostModel
+      Metrics = Struct.new(:dense_bytes, :packed_bytes, :lookup_samples, keyword_init: true) do
+        def direct_score
+          dense_bytes + lookup_samples
+        end
+
+        def table_score
+          packed_bytes + (lookup_samples * 3)
+        end
+      end
+
+      module_function
+
+      def choose(compiled)
+        metrics = compiled.machines.values.map { |machine| metrics_for(machine.dfa) }
+        direct_score = metrics.sum(&:direct_score)
+        table_score = metrics.sum(&:table_score)
+        direct_score < table_score ? :direct : :table
+      end
+
+      def metrics_for(dfa)
+        cells = dfa.states * dfa.class_count
+        overrides = dfa.transitions.sum do |row|
+          default = row.tally.max_by { |_value, count| count }&.first
+          row.count { |value| value != default }
+        end
+        packed_integers = (overrides * 2) + (dfa.states * 2)
+        Metrics.new(
+          dense_bytes: cells * 4,
+          packed_bytes: packed_integers * 4,
+          lookup_samples: dfa.states * [dfa.class_count, 16].min
+        ).freeze
+      end
+    end
+  end
+end

data/lib/flexr/automaton/compiler.rb

@@ -2,9 +2,6 @@
 
 module Flexr
   module Automaton
-    CompiledSpec = Struct.new(:machines, :rules, :states, :stats, :diagnostics, keyword_init: true)
-    Machine = Struct.new(:dfa, :state_name, keyword_init: true)
-
     class Compiler
       def initialize(spec)
         @spec = spec
@@ -48,11 +45,6 @@ module Flexr
         normalized = []
         rules.each do |rule|
           rule.pattern_conditions = []
-          if reference_rule?(rule)
-            validate_reference_patterns(rule)
-            next
-          end
-
           rule.patterns.each_with_index do |pattern, pattern_index|
             regexp = pattern.is_a?(::Regexp) ? pattern : ::Regexp.new(::Regexp.escape(pattern.to_s))
             encoding = regexp.encoding == Encoding::BINARY ? Encoding::BINARY : @spec.encoding
@@ -78,32 +70,6 @@ module Flexr
         DFA.new(transitions: [[nil]], accepts: [[]], ec: Array.new(256, 0), class_count: 1, start: 0, rule_ids: [])
       end
 
-      def reference_rule?(rule)
-        rule.patterns.any? { |pattern| reference_pattern?(pattern) }
-      end
-
-      def reference_pattern?(pattern)
-        return false unless pattern.is_a?(::Regexp)
-        return true if pattern.source.match?(/\\[pP]\{/) || pattern.source.match?(/\[:(?:\^)?[a-z]+:\]/)
-
-        @spec.options[:unicode] == true && @spec.encoding != Encoding::BINARY &&
-          pattern.source.match?(/\\[dDwWsS]/)
-      end
-
-      def validate_reference_patterns(rule)
-        rule.patterns.each_with_index do |pattern, pattern_index|
-          regexp = pattern.is_a?(::Regexp) ? pattern : ::Regexp.new(::Regexp.escape(pattern.to_s))
-          encoding = regexp.encoding == Encoding::BINARY ? Encoding::BINARY : @spec.encoding
-          parser = Regexp::Parser.new(regexp.source, options: regexp.options, encoding: encoding,
-                                      unicode: @spec.options[:unicode] == true)
-          ast = parser.parse
-          _body, bol_only, end_anchor = strip_anchors(ast)
-          rule.pattern_conditions[pattern_index] = Acceptance.new(
-            rule_index: rule.index, pattern_index: pattern_index, bol_only: bol_only, end_anchor: end_anchor
-          )
-        end
-      end
-
       def strip_anchors(ast)
         children = ast.is_a?(Regexp::AST::Seq) ? ast.children.dup : [ast]
         children.shift while children.first.is_a?(Regexp::AST::Empty)
@@ -131,35 +97,38 @@ module Flexr
 
       def contains_anchor?(node)
         return true if node.is_a?(Regexp::AST::Anchor)
+        return contains_anchor?(node.child) if node.is_a?(Regexp::AST::Repeat) || node.is_a?(Regexp::AST::Star)
         return false unless node.respond_to?(:children)
 
         node.children.any? { |child| contains_anchor?(child) }
       end
 
       def subset_construction(nfa, ec, class_count)
-        representatives = Array.new(class_count)
-        ec.each_with_index { |class_id, byte| representatives[class_id] ||= byte }
-        start_set = epsilon_closure(nfa, 1 << nfa.start)
+        closures = {}
+        closure_for = lambda do |state|
+          closures[state] ||= epsilon_closure_for(nfa, state)
+        end
+        transition_closures = indexed_transition_closures(nfa, ec, closure_for)
+        start_set = closure_for.call(nfa.start)
         sets = [start_set]
         ids = { start_set => 0 }
         transitions = []
         accepts = []
         queue = [start_set]
+        queue_index = 0
+        limit = @spec.options.fetch(:max_dfa_states, 100_000)
+        limit = 100_000 unless limit.is_a?(Integer) && limit.positive?
 
-        until queue.empty?
-          set = queue.shift
+        while queue_index < queue.length
+          set = queue.fetch(queue_index)
+          queue_index += 1
           state_id = ids.fetch(set)
           transitions[state_id] ||= Array.new(class_count)
           accepts[state_id] = accepting_rules(nfa, set)
-          class_count.times do |class_id|
-            moved = move(nfa, set, representatives[class_id])
-            next if moved.zero?
-            closure = epsilon_closure(nfa, moved)
+          move_closures(set, transition_closures).each do |class_id, closure|
             destination = ids[closure]
             unless destination
               destination = sets.length
-              limit = @spec.options.fetch(:max_dfa_states, 100_000)
-              limit = 100_000 unless limit.is_a?(Integer) && limit.positive?
               if destination >= limit
                 message = "DFA state limit exceeded while compiling rules #{@active_rule_ids.join(', ')}"
                 diagnostic = Diagnostics.error("FLEXR-E006", message,
@@ -180,13 +149,12 @@ module Flexr
         Minimizer.minimize(dfa)
       end
 
-      def epsilon_closure(nfa, set)
-        closure = set
-        stack = []
-        nfa.states.each_index { |id| stack << id if set.anybits?(1 << id) }
+      def epsilon_closure_for(nfa, state)
+        closure = 1 << state
+        stack = [state]
         until stack.empty?
-          state = stack.pop
-          nfa.states[state].epsilon.each do |target|
+          current = stack.pop
+          nfa.states[current].epsilon.each do |target|
             next if closure.anybits?(1 << target)
 
             closure |= 1 << target
@@ -196,35 +164,52 @@ module Flexr
         closure
       end
 
-      def move(nfa, set, byte)
-        moved = 0
-        nfa.states.each_index do |state|
-          next if set.nobits?(1 << state)
-
-          nfa.states[state].transitions.each do |transition|
-            next unless byte.between?(transition.lo, transition.hi)
+      def indexed_transition_closures(nfa, ec, closure_for)
+        nfa.states.map do |state|
+          raw = {}
+          state.transitions.each do |transition|
+            transition.lo.upto(transition.hi).map { |byte| ec.fetch(byte) }.uniq.each do |class_id|
+              raw[class_id] = raw.fetch(class_id, 0) | (1 << transition.to)
+            end
+          end
+          raw.transform_values do |targets|
+            each_state_id(targets).reduce(0) { |closure, target| closure | closure_for.call(target) }
+          end
+        end
+      end
 
-            moved |= 1 << transition.to
+      def move_closures(set, transition_closures)
+        each_state_id(set).with_object({}) do |state, moves|
+          transition_closures.fetch(state).each do |class_id, closure|
+            moves[class_id] = moves.fetch(class_id, 0) | closure
           end
         end
-        moved
       end
 
       def accepting_rules(nfa, set)
-        rules = []
-        nfa.states.each_index do |state|
-          next if set.nobits?(1 << state)
-
-          rules.concat(nfa.states[state].accepts)
+        rules = each_state_id(set).with_object([]) do |state, result|
+          result.concat(nfa.states[state].accepts)
         end
         rules.uniq.sort_by { |acceptance| [acceptance.rule_index, acceptance.pattern_index] }
       end
 
+      def each_state_id(set)
+        return enum_for(__method__, set) unless block_given?
+
+        remaining = set
+        until remaining.zero?
+          bit = remaining & -remaining
+          yield bit.bit_length - 1
+          remaining ^= bit
+        end
+      end
+
       def validate_rules
         raise CompileError, "firstmatch requires option :experimental" if @spec.backend == :firstmatch && !@spec.options[:experimental]
 
         @spec.rules.each do |rule|
           raise CompileError, "rule #{rule.index} has no pattern" if rule.patterns.empty?
+          parse_regexp(rule.trailing) if rule.trailing.is_a?(::Regexp)
           next if @spec.options[:allow_empty_match]
 
           rule.patterns.each do |pattern|
@@ -249,8 +234,6 @@ module Flexr
         present = compiled.machines.values.flat_map do |machine|
           machine.dfa.accepts.filter_map { |acceptances| acceptances.min_by(&:rule_index)&.rule_index }
         end
-        reference_rules = @spec.rules.select { |rule| reference_rule?(rule) && rule_active_anywhere?(rule) }
-        present.concat(reference_rules.map(&:index))
         shadowers = shadowed_rules(compiled)
         diagnostics.concat(@spec.rules.reject { |rule| present.include?(rule.index) }.map do |rule|
           winners = shadowers.fetch(rule.index, []).uniq.sort
@@ -344,10 +327,6 @@ module Flexr
         rules.combination(2).to_a
       end
 
-      def rule_active_anywhere?(rule)
-        @spec.states.keys.any? { |state_name| rules_for(state_name).include?(rule) }
-      end
-
       def capture_rules
         @spec.rules.select do |rule|
           rule.patterns.any? do |pattern|