evilution 0.20.0 → 0.22.0

data/comparison_results/operator_prioritization.md

@@ -0,0 +1,68 @@
+# Operator Addition Prioritization
+
+Date: 2026-04-09
+Based on: EV-250 classification of 3,901 reference tool mutations across 10 files
+
+## Current Status
+
+- Density ratio: **1.34x** (target: < 1.5x) — **already passing**
+- Evilution: 68 operators covering 14/20 reference categories fully, 4 partially
+- Effective signal gap after removing noise: ~1.0-1.1x (near parity)
+
+## Prioritized Operator Additions
+
+Ranked by: (a) signal frequency, (b) implementation complexity, (c) equivalent mutant rate.
+
+### Priority 1: High signal, low complexity
+
+| # | Operator | Signal Count | Complexity | Equiv. Rate | Notes |
+|---|----------|-------------|-----------|-------------|-------|
+| 1 | `[]` → `.at()` substitution | 60 | Low | Low (~10%) | Catches unchecked array/hash access. Single AST node transform. New operator needed. |
+
+**Rationale:** Only uncovered category with real signal. `.at()` returns nil
+instead of raising on out-of-bounds, exposing missing bounds checks. Simple to
+implement — match `CallNode` with name `[]` on collection receivers, emit `.at()`
+variant.
+
+### Priority 2: Improve existing coverage (partial gaps)
+
+| # | Operator | Gap Area | Complexity | Equiv. Rate | Notes |
+|---|----------|----------|-----------|-------------|-------|
+| 2 | Regex simplification (EV-230, #514) | 27 | Medium | Low (~15%) | Quantifier removal, anchor removal, character class simplification. Already scoped. |
+| 3 | Block pass removal (`&...`) | 5 | Low | Medium (~30%) | Remove `&` block pass arguments (`&:symbol`, `&method(:name)`, etc). Marginal count but trivial to add. |
+
+**Rationale:** EV-230 is already scoped with a GH issue. Block pass removal is
+minimal effort for minimal gain — include only if doing a batch of small operators.
+
+### Priority 3: Do not implement
+
+| # | Category | Count | Reason |
+|---|----------|------:|--------|
+| — | Guard clause restructuring | 570 | Pure noise — syntactic reformatting, not semantic mutation |
+| — | Receiver self-swap (remaining) | ~140 | Mostly equivalent — `self.method` vs `method` rarely matters |
+| — | Complex compound mutations | ~288 | Noise portion of multi-statement changes; not decomposable into discrete operators |
+
+## Implementation Order
+
+1. **EV-230** (#514) — Regex simplification operators (already scoped, medium complexity, 27 signal mutations)
+2. **New: `IndexToAt`** — `[]` → `.at()` substitution (60 signal mutations, low complexity)
+3. **New: `BlockPassRemoval`** — `&:method` removal (5 signal mutations, trivial)
+
+## Impact Assessment
+
+| Scenario | Estimated Ratio | Delta |
+|----------|----------------|-------|
+| Current | 1.34x | — |
+| After adding all Priority 1+2 | ~1.31x | -0.03x |
+
+The density gap is already within target. These additions improve **signal
+coverage** (catching real bugs that reference tool catches and evilution misses)
+rather than closing the headline ratio, which is already healthy.
+
+## Recommendation
+
+The density gap research (EV-238) can be considered **successful** — the 1.5x
+target is met at 1.34x. Remaining work should focus on signal quality (regex
+mutations, bounds checking) rather than chasing the ratio lower. The reference
+tool's ~15% noise inflation means its raw count is not a meaningful target for
+exact parity.

data/docs/mutation_density_benchmark.md

@@ -0,0 +1,91 @@
+# Mutation Density Benchmark Methodology
+
+## Goal
+
+Track and close the mutation density gap between evilution and a reference
+mutation testing tool.
+Current gap: **1.8-2.6x** (evilution generates fewer mutations).
+Target: **< 1.5x** across the benchmark corpus.
+
+## Metric
+
+**Density ratio** = `reference_mutations / evilution_mutations` per file.
+
+A ratio of 1.0 means parity. Values above 1.0 mean the reference tool generates
+more. The aggregate ratio is computed from total mutations across all benchmark
+files (not an average of per-file ratios, which would over-weight small files).
+
+## Measurement Protocol
+
+### Benchmark corpus
+
+Select **10 files** from a real-world Rails project covering diverse patterns:
+
+| Slot | Category             | Example                          |
+|------|----------------------|----------------------------------|
+| 1    | Controller           | `app/controllers/*_controller.rb`|
+| 2    | Model (ActiveRecord) | `app/models/*.rb`                |
+| 3    | Service object       | `app/services/*.rb`              |
+| 4    | Validator            | `app/validators/*.rb`            |
+| 5    | Concern / mixin      | `app/models/concerns/*.rb`       |
+| 6    | Helper               | `app/helpers/*.rb`               |
+| 7    | Formatter / presenter| `app/presenters/*.rb`            |
+| 8    | Lib utility          | `lib/*.rb`                       |
+| 9    | Job / worker         | `app/jobs/*.rb`                  |
+| 10   | Configuration / DSL  | `config/initializers/*.rb`       |
+
+Files should be **50-300 LOC** (enough mutations to be meaningful, small enough
+to run quickly). The exact file list is stored in the benchmark config file
+(`scripts/benchmark_density.yml`).
+
+### Tool configuration
+
+Both tools must run with equivalent settings:
+
+- **evilution**: default operators, no `--skip-heredoc-literals`, no ignore patterns
+- **reference tool**: default operator set, no timeout (we only count, not run)
+
+The benchmark counts **generated mutations**, not killed/survived. This isolates
+operator coverage from test quality.
+
+### Running the benchmark
+
+```bash
+# Count-only mode (fast, no test execution):
+scripts/benchmark_density scripts/benchmark_density.yml
+
+# Full output with per-file breakdown:
+scripts/benchmark_density scripts/benchmark_density.yml --verbose
+```
+
+### Output
+
+The script produces a table:
+
+```
+File                          Evilution  Reference  Ratio
+app/models/user.rb                  42         78  1.86x
+app/services/payment.rb             31         52  1.68x
+...
+TOTAL                              312        534  1.71x
+```
+
+And a summary: `Density ratio: 1.71x (target: < 1.50x)`.
+
+## When to Run
+
+- **Before each release** that adds new operators
+- **After closing operator issues** from the gap analysis epic (GH #515)
+- **On demand** when evaluating whether a proposed operator is worth adding
+
+## Interpreting Results
+
+- **Ratio < 1.5x**: target met
+- **Ratio 1.5-2.0x**: progress, but more operators needed
+- **Ratio > 2.0x**: significant gap remains
+- **Per-file outliers**: files with ratio > 3.0x likely expose a missing operator category
+
+Not all extra mutations from the reference tool are valuable. Some produce
+equivalent mutants (semantically identical code). The head-to-head comparison
+(GH #523) classifies each extra mutation as signal vs noise. The density ratio
+is a **coarse progress metric**, not a quality score.

data/lib/evilution/ast/parser.rb

@@ -70,7 +70,8 @@ module Evilution::AST
              end
 
       loc = node.location
-      method_source = @source[loc.start_offset...loc.end_offset]
+      method_source = @source.byteslice(loc.start_offset, loc.end_offset - loc.start_offset)
+                             .force_encoding(@source.encoding)
 
       @subjects << Evilution::Subject.new(
         name: name,

data/lib/evilution/baseline.rb

@@ -14,9 +14,11 @@ class Evilution::Baseline
     end
   end
 
-  def initialize(spec_resolver: Evilution::SpecResolver.new, timeout: 30)
+  def initialize(spec_resolver: Evilution::SpecResolver.new, timeout: 30, runner: nil, fallback_dir: "spec")
     @spec_resolver = spec_resolver
     @timeout = timeout
+    @runner = runner
+    @fallback_dir = fallback_dir
   end
 
   def call(subjects)
@@ -33,10 +35,14 @@ class Evilution::Baseline
   end
 
   def run_spec_file(spec_file)
+    raise Evilution::Error, "no baseline runner configured" unless @runner
+
     read_io, write_io = IO.pipe
     pid = fork_spec_runner(spec_file, read_io, write_io)
     write_io.close
     read_result(read_io, pid)
+  rescue Evilution::Error
+    raise
   rescue StandardError
     false
   ensure
@@ -45,19 +51,16 @@ class Evilution::Baseline
   end
 
   def fork_spec_runner(spec_file, read_io, write_io)
+    runner = @runner
     Process.fork do
       read_io.close
       $stdout.reopen(File::NULL, "w")
       $stderr.reopen(File::NULL, "w")
 
-      require "rspec/core"
-      RSpec.reset
-      status = RSpec::Core::Runner.run(
-        ["--format", "progress", "--no-color", "--order", "defined", spec_file]
-      )
-      Marshal.dump({ passed: status.zero? }, write_io)
+      passed = runner.call(spec_file)
+      Marshal.dump({ passed: passed }, write_io)
       write_io.close
-      exit!(status.zero? ? 0 : 1)
+      exit!(passed ? 0 : 1)
     end
   end
 
@@ -97,10 +100,10 @@ class Evilution::Baseline
     subjects.map do |s|
       resolved = @spec_resolver.call(s.file_path)
       if resolved.nil? && warned.add?(s.file_path)
-        warn "[evilution] No matching spec found for #{s.file_path}, running full suite. " \
-             "Use --spec to specify the spec file."
+        warn "[evilution] No matching test found for #{s.file_path}, running full suite. " \
+             "Use --spec to specify the test file."
       end
-      resolved || "spec"
+      resolved || @fallback_dir
     end.uniq
   end
 end

data/lib/evilution/cli.rb

@@ -247,14 +247,16 @@ class Evilution::CLI
                            "(default: disabled; if provided without N, uses 1; use --fail-fast=N)") { @options[:fail_fast] ||= 1 }
     opts.on("--no-baseline", "Skip baseline test suite check") { @options[:baseline] = false }
     opts.on("--incremental", "Cache killed/timeout results; skip re-running them on unchanged files") { @options[:incremental] = true }
+    opts.on("--integration NAME", "Test integration: rspec, minitest (default: rspec)") { |i| @options[:integration] = i }
     opts.on("--isolation STRATEGY", "Isolation: auto, fork, in_process (default: auto)") { |s| @options[:isolation] = s }
     opts.on("--stdin", "Read target file paths from stdin (one per line)") { @options[:stdin] = true }
-    opts.on("--suggest-tests", "Generate concrete RSpec test code in suggestions") { @options[:suggest_tests] = true }
+    opts.on("--suggest-tests", "Generate concrete test code in suggestions (RSpec or Minitest)") { @options[:suggest_tests] = true }
     opts.on("--no-progress", "Disable progress bar") { @options[:progress] = false }
     add_extra_flag_options(opts)
   end
 
   def add_extra_flag_options(opts)
+    opts.on("--skip-heredoc-literals", "Skip all string literal mutations inside heredocs") { @options[:skip_heredoc_literals] = true }
     opts.on("--show-disabled", "Report mutations skipped by # evilution:disable") { @options[:show_disabled] = true }
     opts.on("--baseline-session PATH", "Compare against a baseline session in HTML report") { |p| @options[:baseline_session] = p }
     opts.on("--save-session", "Save session results to .evilution/results/") { @options[:save_session] = true }
@@ -298,10 +300,11 @@ class Evilution::CLI
 
     registry = Evilution::Mutator::Registry.default
     filter = build_subject_filter(config)
+    operator_options = build_operator_options(config)
     total_mutations = 0
 
     subjects.each do |subj|
-      count = registry.mutations_for(subj, filter: filter).length
+      count = registry.mutations_for(subj, filter: filter, operator_options: operator_options).length
       total_mutations += count
       label = count == 1 ? "1 mutation" : "#{count} mutations"
       $stdout.puts("  #{subj.name}  #{subj.file_path}:#{subj.line_number}  (#{label})")
@@ -317,6 +320,10 @@ class Evilution::CLI
     2
   end
 
+  def build_operator_options(config)
+    { skip_heredoc_literals: config.skip_heredoc_literals? }
+  end
+
   def build_subject_filter(config)
     return nil if config.ignore_patterns.empty?
 
@@ -425,6 +432,7 @@ class Evilution::CLI
       "  suggest_tests: #{config.suggest_tests}",
       "  save_session: #{config.save_session}",
       "  target: #{config.target || "(all files)"}",
+      "  skip_heredoc_literals: #{config.skip_heredoc_literals}",
       "  ignore_patterns: #{config.ignore_patterns.empty? ? "(none)" : config.ignore_patterns.inspect}"
     ]
   end
@@ -481,8 +489,10 @@ class Evilution::CLI
   def run_util_mutation
     source, file_path = resolve_util_mutation_source
     subjects = parse_source_to_subjects(source, file_path)
+    config = Evilution::Config.new(**@options)
     registry = Evilution::Mutator::Registry.default
-    mutations = subjects.flat_map { |s| registry.mutations_for(s) }
+    operator_options = build_operator_options(config)
+    mutations = subjects.flat_map { |s| registry.mutations_for(s, operator_options: operator_options) }
 
     if mutations.empty?
       $stdout.puts("No mutations generated")

data/lib/evilution/config.rb

@@ -25,14 +25,16 @@ class Evilution::Config
     spec_files: [],
     ignore_patterns: [],
     show_disabled: false,
-    baseline_session: nil
+    baseline_session: nil,
+    skip_heredoc_literals: false
   }.freeze
 
   attr_reader :target_files, :timeout, :format,
               :target, :min_score, :integration, :verbose, :quiet,
               :jobs, :fail_fast, :baseline, :isolation, :incremental, :suggest_tests,
               :progress, :save_session, :line_ranges, :spec_files, :hooks,
-              :ignore_patterns, :show_disabled, :baseline_session
+              :ignore_patterns, :show_disabled, :baseline_session,
+              :skip_heredoc_literals
 
   def initialize(**options)
     file_options = options.delete(:skip_config_file) ? {} : load_config_file
@@ -89,6 +91,10 @@ class Evilution::Config
     show_disabled
   end
 
+  def skip_heredoc_literals?
+    skip_heredoc_literals
+  end
+
   def self.file_options
     CONFIG_FILES.each do |path|
       next unless File.exist?(path)
@@ -119,7 +125,7 @@ class Evilution::Config
       # Minimum mutation score to pass (0.0 to 1.0, default: 0.0)
       # min_score: 0.0
 
-      # Test integration: rspec (default: rspec)
+      # Test integration: rspec, minitest (default: rspec)
       # integration: rspec
 
       # Number of parallel workers (default: 1)
@@ -128,9 +134,12 @@ class Evilution::Config
       # Stop after N surviving mutants (default: disabled)
       # fail_fast: 1
 
-      # Generate concrete RSpec test code in suggestions (default: false)
+      # Generate concrete test code in suggestions, matching integration (default: false)
       # suggest_tests: false
 
+      # Skip all string literal mutations inside heredocs (default: false)
+      # skip_heredoc_literals: false
+
       # Hooks: Ruby files returning a Proc, keyed by lifecycle event
       # hooks:
       #   worker_process_start: config/evilution_hooks/worker_start.rb
@@ -163,7 +172,7 @@ class Evilution::Config
     @format = merged[:format].to_sym
     @target = merged[:target]
     @min_score = merged[:min_score].to_f
-    @integration = merged[:integration].to_sym
+    @integration = validate_integration(merged[:integration])
     @verbose = merged[:verbose]
     @quiet = merged[:quiet]
     @jobs = validate_jobs(merged[:jobs])
@@ -179,9 +188,22 @@ class Evilution::Config
     @ignore_patterns = validate_ignore_patterns(merged[:ignore_patterns])
     @show_disabled = merged[:show_disabled]
     @baseline_session = merged[:baseline_session]
+    @skip_heredoc_literals = merged[:skip_heredoc_literals]
     @hooks = validate_hooks(merged[:hooks])
   end
 
+  def validate_integration(value)
+    raise Evilution::ConfigError, "integration must be rspec or minitest, got nil" if value.nil?
+
+    value = value.to_sym
+    unless %i[rspec minitest].include?(value)
+      raise Evilution::ConfigError,
+            "integration must be rspec or minitest, got #{value.inspect}"
+    end
+
+    value
+  end
+
   def validate_isolation(value)
     raise Evilution::ConfigError, "isolation must be auto, fork, or in_process, got nil" if value.nil?
 

data/lib/evilution/disable_comment.rb

@@ -22,7 +22,8 @@ class Evilution::DisableComment
   def classify_comments(parse_result, source)
     parse_result.comments.filter_map do |comment|
       loc = comment.location
-      text = source[loc.start_offset...loc.end_offset]
+      text = source.byteslice(loc.start_offset, loc.end_offset - loc.start_offset)
+                   .force_encoding(source.encoding)
 
       if text.match?(DISABLE_MARKER)
         line = source.lines[loc.start_line - 1]

data/lib/evilution/integration/base.rb

@@ -1,13 +1,110 @@
 # frozen_string_literal: true
 
+require "fileutils"
+require "tmpdir"
 require_relative "../integration"
+require_relative "../temp_dir_tracker"
 
 class Evilution::Integration::Base
+  def self.baseline_runner
+    raise NotImplementedError, "#{name}.baseline_runner must be implemented"
+  end
+
+  def self.baseline_options
+    raise NotImplementedError, "#{name}.baseline_options must be implemented"
+  end
+
   def initialize(hooks: nil)
     @hooks = hooks
   end
 
   def call(mutation)
-    raise NotImplementedError, "#{self.class}#call must be implemented"
+    @temp_dir = nil
+    ensure_framework_loaded
+    fire_hook(:mutation_insert_pre, mutation: mutation, file_path: mutation.file_path)
+    apply_mutation(mutation)
+    fire_hook(:mutation_insert_post, mutation: mutation, file_path: mutation.file_path)
+    run_tests(mutation)
+  ensure
+    restore_original(mutation)
+  end
+
+  private
+
+  def ensure_framework_loaded
+    raise NotImplementedError, "#{self.class}#ensure_framework_loaded must be implemented"
+  end
+
+  def run_tests(_mutation)
+    raise NotImplementedError, "#{self.class}#run_tests must be implemented"
+  end
+
+  def build_args(_mutation)
+    raise NotImplementedError, "#{self.class}#build_args must be implemented"
+  end
+
+  def reset_state
+    raise NotImplementedError, "#{self.class}#reset_state must be implemented"
+  end
+
+  def fire_hook(event, **payload)
+    @hooks.fire(event, **payload) if @hooks
+  end
+
+  def apply_mutation(mutation)
+    @temp_dir = Dir.mktmpdir("evilution")
+    Evilution::TempDirTracker.register(@temp_dir)
+    @displaced_feature = nil
+    subpath = resolve_require_subpath(mutation.file_path)
+
+    if subpath
+      dest = File.join(@temp_dir, subpath)
+      FileUtils.mkdir_p(File.dirname(dest))
+      File.write(dest, mutation.mutated_source)
+      $LOAD_PATH.unshift(@temp_dir)
+      displace_loaded_feature(mutation.file_path)
+    else
+      absolute = File.expand_path(mutation.file_path)
+      dest = File.join(@temp_dir, absolute)
+      FileUtils.mkdir_p(File.dirname(dest))
+      File.write(dest, mutation.mutated_source)
+      load(dest)
+    end
+  end
+
+  def restore_original(_mutation)
+    return unless @temp_dir
+
+    $LOAD_PATH.delete(@temp_dir)
+    $LOADED_FEATURES.reject! { |f| f.start_with?(@temp_dir) }
+    $LOADED_FEATURES << @displaced_feature if @displaced_feature && !$LOADED_FEATURES.include?(@displaced_feature)
+    @displaced_feature = nil
+    FileUtils.rm_rf(@temp_dir)
+    Evilution::TempDirTracker.unregister(@temp_dir)
+    @temp_dir = nil
+  end
+
+  def resolve_require_subpath(file_path)
+    absolute = File.expand_path(file_path)
+    best_subpath = nil
+
+    $LOAD_PATH.each do |entry|
+      dir = File.expand_path(entry)
+      prefix = dir.end_with?("/") ? dir : "#{dir}/"
+      next unless absolute.start_with?(prefix)
+
+      candidate = absolute.delete_prefix(prefix)
+      best_subpath = candidate if best_subpath.nil? || candidate.length < best_subpath.length
+    end
+
+    best_subpath
+  end
+
+  def displace_loaded_feature(file_path)
+    absolute = File.expand_path(file_path)
+    return unless $LOADED_FEATURES.include?(absolute)
+
+    @displaced_feature = absolute
+    $LOADED_FEATURES.delete(absolute)
   end
 end

data/lib/evilution/integration/minitest.rb

@@ -0,0 +1,145 @@
+# frozen_string_literal: true
+
+require "stringio"
+require_relative "base"
+require_relative "minitest_crash_detector"
+require_relative "../spec_resolver"
+
+require_relative "../integration"
+
+class Evilution::Integration::Minitest < Evilution::Integration::Base
+  def self.baseline_runner
+    lambda { |test_file|
+      require "minitest"
+      require "stringio"
+      ::Minitest::Runnable.runnables.clear
+      files = File.directory?(test_file) ? Dir.glob(File.join(test_file, "**/*_test.rb")) : [test_file]
+      files.each { |f| load(File.expand_path(f)) }
+      out = StringIO.new
+      options = ::Minitest.process_args(["--seed", "0"])
+      options[:io] = out
+      reporter = ::Minitest::CompositeReporter.new
+      reporter << ::Minitest::SummaryReporter.new(out, options)
+      reporter.start
+      ::Minitest.__run(reporter, options)
+      reporter.report
+      reporter.passed?
+    }
+  end
+
+  def self.baseline_options
+    {
+      runner: baseline_runner,
+      spec_resolver: Evilution::SpecResolver.new(test_dir: "test", test_suffix: "_test.rb", request_dir: "integration"),
+      fallback_dir: "test"
+    }
+  end
+
+  def initialize(test_files: nil, hooks: nil)
+    @test_files = test_files
+    @minitest_loaded = false
+    @spec_resolver = Evilution::SpecResolver.new(test_dir: "test", test_suffix: "_test.rb", request_dir: "integration")
+    @crash_detector = nil
+    @warned_files = Set.new
+    super(hooks: hooks)
+  end
+
+  private
+
+  attr_reader :test_files
+
+  def ensure_framework_loaded
+    return if @minitest_loaded
+
+    fire_hook(:setup_integration_pre, integration: :minitest)
+    require "minitest"
+    @minitest_loaded = true
+    fire_hook(:setup_integration_post, integration: :minitest)
+  rescue LoadError => e
+    raise Evilution::Error, "minitest is required but not available: #{e.message}"
+  end
+
+  def run_tests(mutation)
+    reset_state
+    files = resolve_test_files(mutation)
+    command = "ruby -Itest #{files.join(" ")}"
+
+    files.each { |f| load(File.expand_path(f)) }
+
+    args = build_args(mutation)
+    detector = reset_crash_detector
+    passed = run_minitest(args, detector)
+
+    build_minitest_result(passed, command, detector)
+  rescue StandardError => e
+    { passed: false, error: e.message, test_command: command }
+  end
+
+  def build_args(_mutation)
+    ["--seed", "0"]
+  end
+
+  def reset_state
+    ::Minitest::Runnable.runnables.clear
+  end
+
+  def run_minitest(args, detector)
+    out = StringIO.new
+    options = ::Minitest.process_args(args)
+    options[:io] = out
+
+    reporter = ::Minitest::CompositeReporter.new
+    reporter << ::Minitest::SummaryReporter.new(out, options)
+    reporter << detector
+
+    reporter.start
+    ::Minitest.__run(reporter, options)
+    reporter.report
+
+    reporter.passed?
+  end
+
+  def reset_crash_detector
+    if @crash_detector
+      @crash_detector.reset
+    else
+      @crash_detector = Evilution::Integration::MinitestCrashDetector.new
+    end
+    @crash_detector
+  end
+
+  def build_minitest_result(passed, command, detector)
+    if passed
+      { passed: true, test_command: command }
+    elsif detector.only_crashes?
+      { passed: false, error: "test crashes: #{detector.crash_summary}", test_command: command }
+    else
+      { passed: false, test_command: command }
+    end
+  end
+
+  def resolve_test_files(mutation)
+    return test_files if test_files
+
+    resolved = @spec_resolver.call(mutation.file_path)
+    unless resolved
+      warn_unresolved_test(mutation.file_path)
+      return glob_test_files
+    end
+
+    [resolved]
+  end
+
+  def glob_test_files
+    files = Dir.glob("test/**/*_test.rb")
+    files.empty? ? ["test"] : files
+  end
+
+  def warn_unresolved_test(file_path)
+    return if @warned_files.include?(file_path)
+
+    @warned_files << file_path
+    warn "[evilution] No matching test found for #{file_path}, running full suite. " \
+         "Use --spec to specify the test file."
+  end
+end