lrama-fuzz 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 20f3eb480a5c1390a4bbf41c9fe786d5a2c4d5a6f71d1131a9db21962c58706a
+  data.tar.gz: a8276babf5102702fdc250718298c5f2840ebd2f05691c3a97c98d596ba4252c
+SHA512:
+  metadata.gz: 79b4593ed0821e4e1102e426170132be2de49206e0b12301101e9e374f40be11731c6f937378401bcd1321c047d95791bcc73fe4cea2f8211910ef238aa841fc
+  data.tar.gz: 13a57d3e084070b2d7f557c73a4e4b7ed90290dc16ab66160c2f2145a3a6b77bb1cd6d9ddef8ef054768c0762e212af50d1a7f2bf8c887bd664d0d3b1743f521

data/Gemfile

@@ -0,0 +1,10 @@
+# frozen_string_literal: true
+
+source "https://rubygems.org"
+
+gemspec
+
+gem "prism"
+gem "racc"
+gem "rake"
+gem "test-unit"

data/LICENSE

@@ -0,0 +1,7 @@
+Copyright 2026-present, Kevin Newton
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,228 @@
+# lrama-fuzz
+
+Grammar-based fuzzer for [lrama](https://github.com/ruby/lrama) grammars. Generates random strings from any lrama grammar, with built-in profiles for fuzzing Ruby (via Prism or RubyVM) and JSON.
+
+## Installation
+
+In your Gemfile:
+
+```ruby
+gem "lrama-fuzz"
+```
+
+Or install directly:
+
+```sh
+gem install lrama-fuzz
+```
+
+## Quick start
+
+### Ruby fuzzing (Prism)
+
+```ruby
+require "lrama/fuzz"
+
+session = Lrama::Fuzz.prism(ruby_src_dir: "/path/to/ruby", seed: 42)
+
+# Generate a raw grammar derivation (may or may not be valid Ruby)
+puts session.generate
+
+# Generate a composed, valid Ruby program
+puts session.compose
+
+# Evolve programs over 10 generations, optimizing for complexity
+best = session.evolve(10, population_size: 50)
+best.each { |code, fitness| puts "#{fitness.round(2)}: #{code}" }
+
+# Check grammar rule coverage
+session.generate_full_coverage(max_attempts: 500)
+cov = session.coverage
+puts "#{cov.covered_count}/#{cov.total_count} rules (#{(cov.ratio * 100).round(1)}%)"
+```
+
+### Ruby fuzzing (RubyVM)
+
+Uses `RubyVM::InstructionSequence.compile_parsey` for validation instead of Prism. This tests the lrama-generated parser directly.
+
+```ruby
+session = Lrama::Fuzz.rubyvm(ruby_src_dir: "/path/to/ruby", seed: 42)
+puts session.compose
+```
+
+### JSON fuzzing
+
+Uses the JSON grammar from `examples/json.y` -- no external files needed.
+
+```ruby
+session = Lrama::Fuzz.json(seed: 42)
+
+puts session.generate                          # raw derivation
+puts session.generate_valid(max_retries: 50)   # valid JSON document
+```
+
+## Coverage-guided generation
+
+Generate programs with validity feedback. The generator tracks which grammar rules have appeared in valid programs and biases future generation toward rules that haven't been tested in valid contexts yet.
+
+```ruby
+session = Lrama::Fuzz.json(seed: 42)
+
+# Generate 200 programs with automatic feedback
+valid_programs = session.generate_guided(count: 200)
+puts "#{valid_programs.size} valid out of 200"
+
+# Check valid coverage (rules seen in valid programs)
+cov = session.coverage
+puts "Raw coverage:   #{(cov.ratio * 100).round(1)}%"
+puts "Valid coverage: #{(cov.valid_ratio * 100).round(1)}%"
+
+# Use the block form for per-program handling
+session.generate_guided(count: 100) do |code, valid|
+  File.write("corpus/#{Time.now.to_f}.json", code) if valid
+end
+```
+
+The generator also uses valid coverage in its rule selection: after all rules have been expanded at least once, it prefers rules that haven't yet appeared in any valid program. This drives generation toward under-tested parts of the grammar.
+
+## Shrinking
+
+Minimize a failing input to the smallest version that still triggers the bug. Uses delta debugging (line-level, then character-level).
+
+```ruby
+# Standalone
+small = Lrama::Fuzz::Shrinker.shrink(big_program) { |code| crashes?(code) }
+
+# Via session
+small = session.shrink(big_program) { |code| crashes?(code) }
+```
+
+## CLI
+
+```
+$ lrama-fuzz --help
+Usage: lrama-fuzz [options]
+
+Generates programs from lrama grammars.
+
+    --profile PROFILE            Profile: prism, rubyvm, json (default: prism)
+    -d, --ruby-src-dir DIR       Path to Ruby source (default: $RUBY_SRC_DIR)
+        --grammar-path PATH      Path to grammar file (json only)
+    -n, --count N                Number of programs to generate (default: 10)
+    -m, --mode MODE              Mode: generate, compose, evolve, coverage (default: generate)
+    -g, --generations N          Generations for evolve mode (default: 10)
+    -p, --population N           Population size for evolve mode (default: 50)
+    -s, --seed N                 Random seed for reproducibility
+    -h, --help                   Show this help
+```
+
+Examples:
+
+```sh
+# Generate 5 composed Ruby programs
+RUBY_SRC_DIR=/path/to/ruby lrama-fuzz -m compose -n 5
+
+# Generate valid JSON
+lrama-fuzz --profile json -m generate -n 10
+
+# Evolve Ruby programs for 20 generations
+RUBY_SRC_DIR=/path/to/ruby lrama-fuzz -m evolve -g 20 -p 30
+
+# Measure grammar rule coverage
+RUBY_SRC_DIR=/path/to/ruby lrama-fuzz -m coverage -n 500
+```
+
+## Custom grammars
+
+You can fuzz any lrama grammar by using the core API directly.
+
+```ruby
+require "lrama/fuzz"
+
+# Parse a grammar
+grammar = Lrama::Fuzz.parse("path/to/grammar.y")
+
+# Define terminal generators -- each token name maps to a string or proc
+terminals = {
+  "NUMBER" => -> { rand(1..100).to_s },
+  "STRING" => -> { %w[foo bar baz].sample }
+}
+
+# Create a generator
+generator = Lrama::Fuzz::Generator.new(
+  grammar,
+  terminals: terminals,
+  max_depth: 10,       # depth limit for derivation (default: 10)
+  random: Random.new(42)
+)
+
+# Generate strings
+10.times { puts generator.generate }
+
+# Generate strings that pass a validator
+valid = generator.generate_valid(max_retries: 100) { |s| valid?(s) }
+
+# Track coverage
+generator.generate_full_coverage(max_attempts: 500)
+puts generator.coverage.ratio
+```
+
+### Wrapping in a Session
+
+For access to composition, evolution, and shrinking, wrap a generator in a `Session`:
+
+```ruby
+session = Lrama::Fuzz::Session.new(
+  generator,
+  fitness: ->(code) { code.length > 10 ? 1.5 : 0.3 },
+  validator: ->(code) { code.length > 0 },
+  random: Random.new(42)
+)
+
+session.generate                                    # raw derivation
+session.generate_valid                              # passes validator
+session.evolve(10, population_size: 20)             # evolutionary optimization
+session.shrink(code) { |c| some_predicate?(c) }     # delta debugging
+```
+
+## Architecture
+
+```
+Lrama::Fuzz
+  .prism(ruby_src_dir:, seed:)       # -> Session (Prism profile)
+  .rubyvm(ruby_src_dir:, seed:)      # -> Session (RubyVM profile)
+  .json(seed:)                       # -> Session (JSON profile)
+  .parse(path)                       # -> Grammar
+  .new(path, terminals:, **opts)     # -> Generator
+
+Session                # unified interface
+  #generate            # raw grammar derivation
+  #generate_valid      # derivation that passes validator
+  #generate_guided     # generate with validity feedback loop
+  #compose             # template-composed valid program (Ruby only)
+  #evolve(n)           # evolutionary optimization
+  #shrink(code, &pred) # delta debugging minimizer
+  #coverage            # grammar rule coverage tracker
+
+Generator              # core derivation engine
+  #generate            # random derivation from start symbol
+  #generate_valid      # retry until validator passes
+  #record_result       # feed back validity for coverage guidance
+  #generate_full_coverage  # target uncovered rules
+
+Coverage               # grammar rule coverage tracking
+  #ratio               # raw coverage (rules expanded / reachable)
+  #valid_ratio         # valid coverage (rules in valid programs / reachable)
+  #uncovered_valid_rules  # rules not yet seen in valid programs
+
+Profiles (provide fitness, validator, session factory):
+  Prism                # validates with ::Prism.parse
+  RubyVM               # validates with ::RubyVM::InstructionSequence.compile_parsey
+  Json                 # validates with JSON.parse
+
+Ruby                   # shared Ruby grammar infrastructure (classifier, composer)
+Shrinker               # delta debugging minimizer
+Joiner                 # token spacing/joining
+Evolver                # genome-based evolutionary optimization
+ComposedEvolver        # evolutionary optimization with composer
+```

data/Rakefile

@@ -0,0 +1,11 @@
+# frozen_string_literal: true
+
+require "bundler/gem_tasks"
+require "rake/testtask"
+
+Rake::TestTask.new(:test) do |t|
+  t.libs << "test"
+  t.test_files = FileList["test/**/*_test.rb"]
+end
+
+task default: :test

data/examples/arithmetic.y

@@ -0,0 +1,9 @@
+%token NUMBER
+
+%%
+
+expr: NUMBER
+    | expr '+' expr
+    | expr '*' expr
+    | '(' expr ')'
+    ;

data/examples/json.y

@@ -0,0 +1,31 @@
+%token STRING NUMBER TRUE FALSE NULL
+
+%%
+
+value: object
+     | array
+     | STRING
+     | NUMBER
+     | TRUE
+     | FALSE
+     | NULL
+     ;
+
+object: '{' '}'
+      | '{' members '}'
+      ;
+
+members: pair
+       | members ',' pair
+       ;
+
+pair: STRING ':' value
+    ;
+
+array: '[' ']'
+     | '[' elements ']'
+     ;
+
+elements: value
+        | elements ',' value
+        ;

data/examples/lists.y

@@ -0,0 +1,14 @@
+%token ITEM SEPARATOR
+
+%%
+
+program: list
+       ;
+
+list: %empty
+    | list_items
+    ;
+
+list_items: ITEM
+          | list_items SEPARATOR ITEM
+          ;

data/exe/lrama-fuzz

@@ -0,0 +1,170 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require "bundler/setup"
+require "optparse"
+require "lrama/fuzz"
+
+module Lrama
+  module Fuzz
+    module CLI
+      def self.run(argv = ARGV)
+        options = {
+          mode: :generate,
+          count: 10,
+          generations: 10,
+          population_size: 50,
+          seed: nil,
+          profile: :prism,
+          ruby_src_dir: ENV["RUBY_SRC_DIR"],
+          grammar_path: nil
+        }
+
+        parser = ::OptionParser.new do |opts|
+          opts.banner = "Usage: lrama-fuzz [options]"
+          opts.separator ""
+          opts.separator "Generates programs from lrama grammars."
+          opts.separator ""
+
+          opts.on("--profile PROFILE", %i[prism rubyvm json],
+                  "Profile: prism, rubyvm, json (default: prism)") do |p|
+            options[:profile] = p
+          end
+
+          opts.on("-d", "--ruby-src-dir DIR", "Path to Ruby source (default: $RUBY_SRC_DIR)") do |dir|
+            options[:ruby_src_dir] = dir
+          end
+
+          opts.on("--grammar-path PATH", "Path to grammar file (json only)") do |path|
+            options[:grammar_path] = path
+          end
+
+          opts.on("-n", "--count N", Integer, "Number of programs to generate (default: 10)") do |n|
+            options[:count] = n
+          end
+
+          opts.on("-m", "--mode MODE", %i[generate compose evolve coverage guided],
+                  "Mode: generate, compose, evolve, coverage, guided (default: generate)") do |mode|
+            options[:mode] = mode
+          end
+
+          opts.on("-g", "--generations N", Integer, "Generations for evolve mode (default: 10)") do |n|
+            options[:generations] = n
+          end
+
+          opts.on("-p", "--population N", Integer, "Population size for evolve mode (default: 50)") do |n|
+            options[:population_size] = n
+          end
+
+          opts.on("-s", "--seed N", Integer, "Random seed for reproducibility") do |n|
+            options[:seed] = n
+          end
+
+          opts.on("-h", "--help", "Show this help") do
+            puts opts
+            exit
+          end
+        end
+
+        parser.parse!(argv)
+        session = build_session(options)
+
+        case options[:mode]
+        when :generate
+          emit(options[:count]) { session.generate }
+        when :compose
+          emit(options[:count]) { session.compose }
+        when :evolve
+          run_evolve(session, options)
+        when :coverage
+          run_coverage(session, options)
+        when :guided
+          run_guided(session, options)
+        end
+      end
+
+      def self.build_session(options)
+        case options[:profile]
+        when :prism, :rubyvm
+          unless options[:ruby_src_dir]
+            $stderr.puts "Error: Ruby source directory required."
+            $stderr.puts "Set RUBY_SRC_DIR or pass --ruby-src-dir."
+            exit 1
+          end
+
+          unless File.exist?(File.join(options[:ruby_src_dir], "parse.y"))
+            $stderr.puts "Error: #{options[:ruby_src_dir]}/parse.y not found."
+            exit 1
+          end
+
+          $stderr.puts "Loading Ruby grammar (#{options[:profile]})..."
+          session = Fuzz.public_send(options[:profile],
+                                     ruby_src_dir: options[:ruby_src_dir],
+                                     seed: options[:seed])
+          $stderr.puts "Ready."
+          session
+        when :json
+          $stderr.puts "Loading JSON grammar..."
+          kwargs = { seed: options[:seed] }
+          kwargs[:grammar_path] = options[:grammar_path] if options[:grammar_path]
+          session = Fuzz.json(**kwargs)
+          $stderr.puts "Ready."
+          session
+        end
+      end
+
+      def self.emit(count)
+        count.times do
+          puts yield
+          puts "---"
+        end
+      end
+
+      def self.run_evolve(session, options)
+        $stderr.puts "Evolving #{options[:population_size]} programs for #{options[:generations]} generations..."
+        best = session.evolve(
+          options[:generations],
+          population_size: options[:population_size]
+        )
+
+        best.each do |code, fitness|
+          puts "# fitness: #{fitness.round(4)}"
+          puts code
+          puts "---"
+        end
+
+        $stderr.puts "Best fitness: #{best.first[1].round(4)}"
+      end
+
+      def self.run_coverage(session, options)
+        $stderr.puts "Generating programs to measure coverage..."
+        session.generate_full_coverage(max_attempts: options[:count])
+
+        cov = session.coverage
+        $stderr.puts "Coverage: #{cov.covered_count}/#{cov.total_count} rules (#{(cov.ratio * 100).round(1)}%)"
+      end
+
+      def self.run_guided(session, options)
+        $stderr.puts "Generating #{options[:count]} programs with validity feedback..."
+        valid_count = 0
+
+        session.generate_guided(count: options[:count]) do |code, valid|
+          if valid
+            valid_count += 1
+            puts code
+            puts "---"
+          end
+        end
+
+        cov = session.coverage
+        $stderr.puts "Valid: #{valid_count}/#{options[:count]}"
+        $stderr.puts "Raw coverage:   #{cov.covered_count}/#{cov.total_count} (#{(cov.ratio * 100).round(1)}%)"
+        $stderr.puts "Valid coverage: #{(cov.valid_ratio * 100).round(1)}%"
+      end
+
+      private_class_method :build_session
+    end
+  end
+end
+
+Lrama::Fuzz::CLI.run

data/lib/lrama/fuzz/codon_random.rb

@@ -0,0 +1,48 @@
+# frozen_string_literal: true
+
+module Lrama
+  module Fuzz
+    # A Random-compatible object that reads values from a codon (integer)
+    # sequence. This allows deterministic replay and evolution of any code
+    # path that uses Random — in particular, the Composer's template and
+    # fragment selection.
+    #
+    # Wraps around when the sequence is exhausted, so any genome length
+    # can drive arbitrarily many decisions.
+    class CodonRandom
+      def initialize(codons)
+        @codons = codons
+        @index = 0
+      end
+
+      # Compatible with Random#rand:
+      #   rand()      -> Float in [0, 1)
+      #   rand(n)     -> Integer in [0, n)
+      #   rand(a..b)  -> Integer in [a, b]
+      def rand(max = nil)
+        codon = next_codon
+
+        case max
+        when nil
+          codon.to_f / GeneticOperators::MAX_CODON
+        when Integer
+          max == 0 ? 0 : codon % max
+        when Range
+          min_val = max.min
+          span = max.max - min_val + (max.exclude_end? ? 0 : 1)
+          span <= 0 ? min_val : min_val + (codon % span)
+        else
+          raise ArgumentError, "unexpected argument: #{max.inspect}"
+        end
+      end
+
+      private
+
+      def next_codon
+        codon = @codons[@index % @codons.size]
+        @index += 1
+        codon
+      end
+    end
+  end
+end

data/lib/lrama/fuzz/composed_evolver.rb

@@ -0,0 +1,94 @@
+# frozen_string_literal: true
+
+module Lrama
+  module Fuzz
+    # Evolves genomes that drive Composer template and fragment selection
+    # via CodonRandom. Unlike the grammar-level Evolver, every genome here
+    # produces a structurally valid program (the Composer enforces that),
+    # so evolution optimizes for fitness (complexity, diversity) rather
+    # than basic validity.
+    #
+    # Usage:
+    #   generator = Generator.new(grammar, terminals: ..., ...)
+    #   evolver = ComposedEvolver.new(generator, fitness: Ruby.fitness)
+    #   results = evolver.evolve  # => [[code, fitness], ...]
+    class ComposedEvolver
+      include GeneticOperators
+
+      DEFAULT_POPULATION_SIZE = 50
+      DEFAULT_GENOME_LENGTH = 300
+      DEFAULT_MUTATION_RATE = 0.05
+
+      attr_reader :generation, :best_fitness
+
+      def initialize(generator, fitness:, composer_class: Ruby::Composer, validator: nil, population_size: DEFAULT_POPULATION_SIZE, genome_length: DEFAULT_GENOME_LENGTH, mutation_rate: DEFAULT_MUTATION_RATE, random: Random.new)
+        @generator = generator
+        @fitness = fitness
+        @composer_class = composer_class
+        @validator = validator
+        @population_size = population_size
+        @genome_length = genome_length
+        @mutation_rate = mutation_rate
+        @random = random
+        @generation = 0
+        @best_fitness = 0.0
+        @last_evaluated = nil
+
+        @population = Array.new(population_size) { random_genome }
+      end
+
+      # Run one generation: evaluate, select, crossover, mutate.
+      # Returns an array of [code, fitness] pairs.
+      def evolve
+        evaluated = @population.map do |genome|
+          code = generate_from_genome(genome)
+          score = @fitness.call(code)
+          Individual.new(genome: genome, code: code, fitness: score)
+        end
+
+        evaluated.sort_by! { |ind| -ind.fitness }
+        @best_fitness = evaluated.first.fitness
+
+        new_population = []
+
+        # Elitism: keep top 20%
+        elite_count = [@population_size / 5, 1].max
+        new_population.concat(evaluated.first(elite_count).map(&:genome))
+
+        # Fill rest with crossover + mutation
+        while new_population.size < @population_size
+          p1 = tournament_select(evaluated)
+          p2 = tournament_select(evaluated)
+          child = crossover(p1, p2)
+          mutate!(child)
+          new_population << child
+        end
+
+        @population = new_population.first(@population_size)
+        @last_evaluated = evaluated
+        @generation += 1
+
+        evaluated.map { |ind| [ind.code, ind.fitness] }
+      end
+
+      # Generate a program from a genome by driving the Composer with
+      # a CodonRandom seeded from the genome.
+      def generate_from_genome(genome)
+        codon_random = CodonRandom.new(genome)
+        composer = @composer_class.new(@generator, random: codon_random, validator: @validator)
+        composer.generate
+      end
+
+      # Return the best programs from the current population.
+      def best_programs(n = 10)
+        evaluated = @last_evaluated || @population.map do |genome|
+          code = generate_from_genome(genome)
+          score = @fitness.call(code)
+          Individual.new(genome: genome, code: code, fitness: score)
+        end
+
+        evaluated.sort_by { |ind| -ind.fitness }.first(n).map { |ind| [ind.code, ind.fitness] }
+      end
+    end
+  end
+end