evilution 0.21.0 → 0.22.1

data/docs/mutation_density_benchmark.md

@@ -0,0 +1,91 @@
+# Mutation Density Benchmark Methodology
+
+## Goal
+
+Track and close the mutation density gap between evilution and a reference
+mutation testing tool.
+Current gap: **1.8-2.6x** (evilution generates fewer mutations).
+Target: **< 1.5x** across the benchmark corpus.
+
+## Metric
+
+**Density ratio** = `reference_mutations / evilution_mutations` per file.
+
+A ratio of 1.0 means parity. Values above 1.0 mean the reference tool generates
+more. The aggregate ratio is computed from total mutations across all benchmark
+files (not an average of per-file ratios, which would over-weight small files).
+
+## Measurement Protocol
+
+### Benchmark corpus
+
+Select **10 files** from a real-world Rails project covering diverse patterns:
+
+| Slot | Category             | Example                          |
+|------|----------------------|----------------------------------|
+| 1    | Controller           | `app/controllers/*_controller.rb`|
+| 2    | Model (ActiveRecord) | `app/models/*.rb`                |
+| 3    | Service object       | `app/services/*.rb`              |
+| 4    | Validator            | `app/validators/*.rb`            |
+| 5    | Concern / mixin      | `app/models/concerns/*.rb`       |
+| 6    | Helper               | `app/helpers/*.rb`               |
+| 7    | Formatter / presenter| `app/presenters/*.rb`            |
+| 8    | Lib utility          | `lib/*.rb`                       |
+| 9    | Job / worker         | `app/jobs/*.rb`                  |
+| 10   | Configuration / DSL  | `config/initializers/*.rb`       |
+
+Files should be **50-300 LOC** (enough mutations to be meaningful, small enough
+to run quickly). The exact file list is stored in the benchmark config file
+(`scripts/benchmark_density.yml`).
+
+### Tool configuration
+
+Both tools must run with equivalent settings:
+
+- **evilution**: default operators, no `--skip-heredoc-literals`, no ignore patterns
+- **reference tool**: default operator set, no timeout (we only count, not run)
+
+The benchmark counts **generated mutations**, not killed/survived. This isolates
+operator coverage from test quality.
+
+### Running the benchmark
+
+```bash
+# Count-only mode (fast, no test execution):
+scripts/benchmark_density scripts/benchmark_density.yml
+
+# Full output with per-file breakdown:
+scripts/benchmark_density scripts/benchmark_density.yml --verbose
+```
+
+### Output
+
+The script produces a table:
+
+```
+File                          Evilution  Reference  Ratio
+app/models/user.rb                  42         78  1.86x
+app/services/payment.rb             31         52  1.68x
+...
+TOTAL                              312        534  1.71x
+```
+
+And a summary: `Density ratio: 1.71x (target: < 1.50x)`.
+
+## When to Run
+
+- **Before each release** that adds new operators
+- **After closing operator issues** from the gap analysis epic (GH #515)
+- **On demand** when evaluating whether a proposed operator is worth adding
+
+## Interpreting Results
+
+- **Ratio < 1.5x**: target met
+- **Ratio 1.5-2.0x**: progress, but more operators needed
+- **Ratio > 2.0x**: significant gap remains
+- **Per-file outliers**: files with ratio > 3.0x likely expose a missing operator category
+
+Not all extra mutations from the reference tool are valuable. Some produce
+equivalent mutants (semantically identical code). The head-to-head comparison
+(GH #523) classifies each extra mutation as signal vs noise. The density ratio
+is a **coarse progress metric**, not a quality score.

data/lib/evilution/ast/parser.rb

@@ -70,7 +70,8 @@ module Evilution::AST
              end
 
       loc = node.location
-      method_source = @source[loc.start_offset...loc.end_offset]
+      method_source = @source.byteslice(loc.start_offset, loc.end_offset - loc.start_offset)
+                             .force_encoding(@source.encoding)
 
       @subjects << Evilution::Subject.new(
         name: name,

data/lib/evilution/baseline.rb

@@ -14,9 +14,11 @@ class Evilution::Baseline
     end
   end
 
-  def initialize(spec_resolver: Evilution::SpecResolver.new, timeout: 30)
+  def initialize(spec_resolver: Evilution::SpecResolver.new, timeout: 30, runner: nil, fallback_dir: "spec")
     @spec_resolver = spec_resolver
     @timeout = timeout
+    @runner = runner
+    @fallback_dir = fallback_dir
   end
 
   def call(subjects)
@@ -33,10 +35,14 @@ class Evilution::Baseline
   end
 
   def run_spec_file(spec_file)
+    raise Evilution::Error, "no baseline runner configured" unless @runner
+
     read_io, write_io = IO.pipe
     pid = fork_spec_runner(spec_file, read_io, write_io)
     write_io.close
     read_result(read_io, pid)
+  rescue Evilution::Error
+    raise
   rescue StandardError
     false
   ensure
@@ -45,19 +51,16 @@ class Evilution::Baseline
   end
 
   def fork_spec_runner(spec_file, read_io, write_io)
+    runner = @runner
     Process.fork do
       read_io.close
       $stdout.reopen(File::NULL, "w")
       $stderr.reopen(File::NULL, "w")
 
-      require "rspec/core"
-      RSpec.reset
-      status = RSpec::Core::Runner.run(
-        ["--format", "progress", "--no-color", "--order", "defined", spec_file]
-      )
-      Marshal.dump({ passed: status.zero? }, write_io)
+      passed = runner.call(spec_file)
+      Marshal.dump({ passed: passed }, write_io)
       write_io.close
-      exit!(status.zero? ? 0 : 1)
+      exit!(passed ? 0 : 1)
     end
   end
 
@@ -97,10 +100,10 @@ class Evilution::Baseline
     subjects.map do |s|
       resolved = @spec_resolver.call(s.file_path)
       if resolved.nil? && warned.add?(s.file_path)
-        warn "[evilution] No matching spec found for #{s.file_path}, running full suite. " \
-             "Use --spec to specify the spec file."
+        warn "[evilution] No matching test found for #{s.file_path}, running full suite. " \
+             "Use --spec to specify the test file."
       end
-      resolved || "spec"
+      resolved || @fallback_dir
     end.uniq
   end
 end

data/lib/evilution/cli.rb

@@ -247,9 +247,10 @@ class Evilution::CLI
                            "(default: disabled; if provided without N, uses 1; use --fail-fast=N)") { @options[:fail_fast] ||= 1 }
     opts.on("--no-baseline", "Skip baseline test suite check") { @options[:baseline] = false }
     opts.on("--incremental", "Cache killed/timeout results; skip re-running them on unchanged files") { @options[:incremental] = true }
+    opts.on("--integration NAME", "Test integration: rspec, minitest (default: rspec)") { |i| @options[:integration] = i }
     opts.on("--isolation STRATEGY", "Isolation: auto, fork, in_process (default: auto)") { |s| @options[:isolation] = s }
     opts.on("--stdin", "Read target file paths from stdin (one per line)") { @options[:stdin] = true }
-    opts.on("--suggest-tests", "Generate concrete RSpec test code in suggestions") { @options[:suggest_tests] = true }
+    opts.on("--suggest-tests", "Generate concrete test code in suggestions (RSpec or Minitest)") { @options[:suggest_tests] = true }
     opts.on("--no-progress", "Disable progress bar") { @options[:progress] = false }
     add_extra_flag_options(opts)
   end

data/lib/evilution/config.rb

@@ -125,7 +125,7 @@ class Evilution::Config
       # Minimum mutation score to pass (0.0 to 1.0, default: 0.0)
       # min_score: 0.0
 
-      # Test integration: rspec (default: rspec)
+      # Test integration: rspec, minitest (default: rspec)
       # integration: rspec
 
       # Number of parallel workers (default: 1)
@@ -134,7 +134,7 @@ class Evilution::Config
       # Stop after N surviving mutants (default: disabled)
       # fail_fast: 1
 
-      # Generate concrete RSpec test code in suggestions (default: false)
+      # Generate concrete test code in suggestions, matching integration (default: false)
       # suggest_tests: false
 
       # Skip all string literal mutations inside heredocs (default: false)
@@ -172,7 +172,7 @@ class Evilution::Config
     @format = merged[:format].to_sym
     @target = merged[:target]
     @min_score = merged[:min_score].to_f
-    @integration = merged[:integration].to_sym
+    @integration = validate_integration(merged[:integration])
     @verbose = merged[:verbose]
     @quiet = merged[:quiet]
     @jobs = validate_jobs(merged[:jobs])
@@ -192,6 +192,18 @@ class Evilution::Config
     @hooks = validate_hooks(merged[:hooks])
   end
 
+  def validate_integration(value)
+    raise Evilution::ConfigError, "integration must be rspec or minitest, got nil" if value.nil?
+
+    value = value.to_sym
+    unless %i[rspec minitest].include?(value)
+      raise Evilution::ConfigError,
+            "integration must be rspec or minitest, got #{value.inspect}"
+    end
+
+    value
+  end
+
   def validate_isolation(value)
     raise Evilution::ConfigError, "isolation must be auto, fork, or in_process, got nil" if value.nil?
 

data/lib/evilution/disable_comment.rb

@@ -22,7 +22,8 @@ class Evilution::DisableComment
   def classify_comments(parse_result, source)
     parse_result.comments.filter_map do |comment|
       loc = comment.location
-      text = source[loc.start_offset...loc.end_offset]
+      text = source.byteslice(loc.start_offset, loc.end_offset - loc.start_offset)
+                   .force_encoding(source.encoding)
 
       if text.match?(DISABLE_MARKER)
         line = source.lines[loc.start_line - 1]

data/lib/evilution/integration/base.rb

@@ -1,13 +1,136 @@
 # frozen_string_literal: true
 
+require "fileutils"
+require "tmpdir"
 require_relative "../integration"
+require_relative "../temp_dir_tracker"
 
 class Evilution::Integration::Base
+  def self.baseline_runner
+    raise NotImplementedError, "#{name}.baseline_runner must be implemented"
+  end
+
+  def self.baseline_options
+    raise NotImplementedError, "#{name}.baseline_options must be implemented"
+  end
+
   def initialize(hooks: nil)
     @hooks = hooks
   end
 
   def call(mutation)
-    raise NotImplementedError, "#{self.class}#call must be implemented"
+    @temp_dir = nil
+    ensure_framework_loaded
+    fire_hook(:mutation_insert_pre, mutation: mutation, file_path: mutation.file_path)
+    load_error = apply_mutation(mutation)
+    return load_error if load_error
+
+    fire_hook(:mutation_insert_post, mutation: mutation, file_path: mutation.file_path)
+    run_tests(mutation)
+  ensure
+    restore_original(mutation)
+  end
+
+  private
+
+  def ensure_framework_loaded
+    raise NotImplementedError, "#{self.class}#ensure_framework_loaded must be implemented"
+  end
+
+  def run_tests(_mutation)
+    raise NotImplementedError, "#{self.class}#run_tests must be implemented"
+  end
+
+  def build_args(_mutation)
+    raise NotImplementedError, "#{self.class}#build_args must be implemented"
+  end
+
+  def reset_state
+    raise NotImplementedError, "#{self.class}#reset_state must be implemented"
+  end
+
+  def fire_hook(event, **payload)
+    @hooks.fire(event, **payload) if @hooks
+  end
+
+  def apply_mutation(mutation)
+    @temp_dir = Dir.mktmpdir("evilution")
+    Evilution::TempDirTracker.register(@temp_dir)
+    @displaced_feature = nil
+    subpath = resolve_require_subpath(mutation.file_path)
+
+    if subpath
+      apply_via_require(mutation, subpath)
+    else
+      apply_via_load(mutation)
+    end
+    nil
+  rescue SyntaxError => e
+    {
+      passed: false,
+      error: "syntax error in mutated source: #{e.message}",
+      error_class: e.class.name,
+      error_backtrace: Array(e.backtrace).first(5)
+    }
+  rescue ScriptError, StandardError => e
+    {
+      passed: false,
+      error: "#{e.class}: #{e.message}",
+      error_class: e.class.name,
+      error_backtrace: Array(e.backtrace).first(5)
+    }
+  end
+
+  def apply_via_require(mutation, subpath)
+    dest = File.join(@temp_dir, subpath)
+    FileUtils.mkdir_p(File.dirname(dest))
+    File.write(dest, mutation.mutated_source)
+    $LOAD_PATH.unshift(@temp_dir)
+    displace_loaded_feature(mutation.file_path)
+    require(subpath.delete_suffix(".rb"))
+  end
+
+  def apply_via_load(mutation)
+    absolute = File.expand_path(mutation.file_path)
+    dest = File.join(@temp_dir, absolute)
+    FileUtils.mkdir_p(File.dirname(dest))
+    File.write(dest, mutation.mutated_source)
+    load(dest)
+  end
+
+  def restore_original(_mutation)
+    return unless @temp_dir
+
+    $LOAD_PATH.delete(@temp_dir)
+    $LOADED_FEATURES.reject! { |f| f.start_with?(@temp_dir) }
+    $LOADED_FEATURES << @displaced_feature if @displaced_feature && !$LOADED_FEATURES.include?(@displaced_feature)
+    @displaced_feature = nil
+    FileUtils.rm_rf(@temp_dir)
+    Evilution::TempDirTracker.unregister(@temp_dir)
+    @temp_dir = nil
+  end
+
+  def resolve_require_subpath(file_path)
+    absolute = File.expand_path(file_path)
+    best_subpath = nil
+
+    $LOAD_PATH.each do |entry|
+      dir = File.expand_path(entry)
+      prefix = dir.end_with?("/") ? dir : "#{dir}/"
+      next unless absolute.start_with?(prefix)
+
+      candidate = absolute.delete_prefix(prefix)
+      best_subpath = candidate if best_subpath.nil? || candidate.length < best_subpath.length
+    end
+
+    best_subpath
+  end
+
+  def displace_loaded_feature(file_path)
+    absolute = File.expand_path(file_path)
+    return unless $LOADED_FEATURES.include?(absolute)
+
+    @displaced_feature = absolute
+    $LOADED_FEATURES.delete(absolute)
   end
 end

data/lib/evilution/integration/minitest.rb

@@ -0,0 +1,145 @@
+# frozen_string_literal: true
+
+require "stringio"
+require_relative "base"
+require_relative "minitest_crash_detector"
+require_relative "../spec_resolver"
+
+require_relative "../integration"
+
+class Evilution::Integration::Minitest < Evilution::Integration::Base
+  def self.baseline_runner
+    lambda { |test_file|
+      require "minitest"
+      require "stringio"
+      ::Minitest::Runnable.runnables.clear
+      files = File.directory?(test_file) ? Dir.glob(File.join(test_file, "**/*_test.rb")) : [test_file]
+      files.each { |f| load(File.expand_path(f)) }
+      out = StringIO.new
+      options = ::Minitest.process_args(["--seed", "0"])
+      options[:io] = out
+      reporter = ::Minitest::CompositeReporter.new
+      reporter << ::Minitest::SummaryReporter.new(out, options)
+      reporter.start
+      ::Minitest.__run(reporter, options)
+      reporter.report
+      reporter.passed?
+    }
+  end
+
+  def self.baseline_options
+    {
+      runner: baseline_runner,
+      spec_resolver: Evilution::SpecResolver.new(test_dir: "test", test_suffix: "_test.rb", request_dir: "integration"),
+      fallback_dir: "test"
+    }
+  end
+
+  def initialize(test_files: nil, hooks: nil)
+    @test_files = test_files
+    @minitest_loaded = false
+    @spec_resolver = Evilution::SpecResolver.new(test_dir: "test", test_suffix: "_test.rb", request_dir: "integration")
+    @crash_detector = nil
+    @warned_files = Set.new
+    super(hooks: hooks)
+  end
+
+  private
+
+  attr_reader :test_files
+
+  def ensure_framework_loaded
+    return if @minitest_loaded
+
+    fire_hook(:setup_integration_pre, integration: :minitest)
+    require "minitest"
+    @minitest_loaded = true
+    fire_hook(:setup_integration_post, integration: :minitest)
+  rescue LoadError => e
+    raise Evilution::Error, "minitest is required but not available: #{e.message}"
+  end
+
+  def run_tests(mutation)
+    reset_state
+    files = resolve_test_files(mutation)
+    command = "ruby -Itest #{files.join(" ")}"
+
+    files.each { |f| load(File.expand_path(f)) }
+
+    args = build_args(mutation)
+    detector = reset_crash_detector
+    passed = run_minitest(args, detector)
+
+    build_minitest_result(passed, command, detector)
+  rescue StandardError => e
+    { passed: false, error: e.message, test_command: command }
+  end
+
+  def build_args(_mutation)
+    ["--seed", "0"]
+  end
+
+  def reset_state
+    ::Minitest::Runnable.runnables.clear
+  end
+
+  def run_minitest(args, detector)
+    out = StringIO.new
+    options = ::Minitest.process_args(args)
+    options[:io] = out
+
+    reporter = ::Minitest::CompositeReporter.new
+    reporter << ::Minitest::SummaryReporter.new(out, options)
+    reporter << detector
+
+    reporter.start
+    ::Minitest.__run(reporter, options)
+    reporter.report
+
+    reporter.passed?
+  end
+
+  def reset_crash_detector
+    if @crash_detector
+      @crash_detector.reset
+    else
+      @crash_detector = Evilution::Integration::MinitestCrashDetector.new
+    end
+    @crash_detector
+  end
+
+  def build_minitest_result(passed, command, detector)
+    if passed
+      { passed: true, test_command: command }
+    elsif detector.only_crashes?
+      { passed: false, error: "test crashes: #{detector.crash_summary}", test_command: command }
+    else
+      { passed: false, test_command: command }
+    end
+  end
+
+  def resolve_test_files(mutation)
+    return test_files if test_files
+
+    resolved = @spec_resolver.call(mutation.file_path)
+    unless resolved
+      warn_unresolved_test(mutation.file_path)
+      return glob_test_files
+    end
+
+    [resolved]
+  end
+
+  def glob_test_files
+    files = Dir.glob("test/**/*_test.rb")
+    files.empty? ? ["test"] : files
+  end
+
+  def warn_unresolved_test(file_path)
+    return if @warned_files.include?(file_path)
+
+    @warned_files << file_path
+    warn "[evilution] No matching test found for #{file_path}, running full suite. " \
+         "Use --spec to specify the test file."
+  end
+end

data/lib/evilution/integration/minitest_crash_detector.rb

@@ -0,0 +1,55 @@
+# frozen_string_literal: true
+
+require_relative "../integration"
+
+class Evilution::Integration::MinitestCrashDetector
+  def initialize
+    reset
+  end
+
+  def start
+    # Required by Minitest reporter interface
+  end
+
+  def report
+    # Required by Minitest reporter interface
+  end
+
+  def passed?
+    @crashes.empty?
+  end
+
+  def reset
+    @assertion_failures = 0
+    @crashes = []
+  end
+
+  def record(result)
+    result.failures.each do |failure|
+      if failure.is_a?(::Minitest::UnexpectedError)
+        @crashes << failure.error
+      elsif failure.is_a?(::Minitest::Assertion)
+        @assertion_failures += 1
+      end
+    end
+  end
+
+  def has_assertion_failure? # rubocop:disable Naming/PredicatePrefix
+    @assertion_failures.positive?
+  end
+
+  def has_crash? # rubocop:disable Naming/PredicatePrefix
+    @crashes.any?
+  end
+
+  def only_crashes?
+    @crashes.any? && @assertion_failures.zero?
+  end
+
+  def crash_summary
+    return nil if @crashes.empty?
+
+    types = @crashes.map { |e| e.class.name }.uniq
+    "#{types.join(", ")} (#{@crashes.length} crash#{"es" unless @crashes.length == 1})"
+  end
+end