mutation_tester 1.2.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 20fff769b6b8b7efeb874208f5e3c25206ca92a2fc9f220bdd4fcb520eab944c
+  data.tar.gz: 3b630985b285958780d49d356cbe04f629ce50e1aca947d2f40d8bdddd274d42
+SHA512:
+  metadata.gz: f40cf255d91681f4ae191b444ba8f99ba897ea746b2f157190d69bce091e90c4aaec9bde9c73fe69008de17916478ff40227a7e2439de687d4a576d72d664d0a
+  data.tar.gz: 9c8d46ca0d8a98dff41de40b61e8fdd60c68c55ed37303e02d08009968a0263bf932e0892af3e13af12810c27cb61f695f1d01a77e46f2dbc375735fe49b87a4

data/CHANGELOG.md

@@ -0,0 +1,13 @@
+# Changelog
+
+## [1.2.0] - 2026-07-14
+
+- Fixed the in-memory runner reporting false survivors for mutations that only take effect at class-load time (constants consumed by macros, `validates`/`has_many`/`before_save`/`scope`/`attribute`, and anything inside an `included do` block). The mutator now classifies each mutation by AST context, and the in-memory run routes load-time mutations to the file-based path while keeping method-body mutations in memory, so the default (`auto`) score matches a full `fork` run on Rails concerns and models. Measured on a real Rails concern the default score went from a misleading 1.63% to the correct 35.77%.
+
+## [1.1.0] - 2026-07-13
+
+- Added `--worker-env NAME` (and the `MUTATION_TESTER_WORKER_ENV` environment variable) to set a distinct per-worker value of `NAME` before each parallel worker boots, following the `parallel_tests` `TEST_ENV_NUMBER` convention (worker 0 -> "", worker N -> N+1). This lets a Rails app with a `parallel_tests`-style `database.yml` run parallel mutation testing with a per-worker database instead of being forced to serial execution. Supported by the `fork` and `spawn` runners; the `in_memory` runner cannot isolate a per-worker database and falls back to `fork` with an announced notice.
+
+## [1.0.0] - 2026-07-12
+
+- Released first version.

data/Gemfile

@@ -0,0 +1,9 @@
+# Gemfile
+source 'https://rubygems.org'
+
+gemspec
+
+group :development, :test do
+  gem 'pry'
+  gem 'rubocop', require: false
+end

data/Gemfile.lock

@@ -0,0 +1,83 @@
+PATH
+  remote: .
+  specs:
+    mutation_tester (1.2.0)
+      parallel (~> 1.20)
+      parser (~> 3.3)
+      rainbow (~> 3.0)
+      unparser (>= 0.6, < 0.9)
+
+GEM
+  remote: https://rubygems.org/
+  specs:
+    ast (2.4.3)
+    coderay (1.1.3)
+    diff-lcs (1.6.2)
+    json (2.16.0)
+    language_server-protocol (3.17.0.5)
+    lint_roller (1.1.0)
+    method_source (1.1.0)
+    minitest (5.26.2)
+    parallel (1.27.0)
+    parser (3.3.11.1)
+      ast (~> 2.4.1)
+      racc
+    prism (1.6.0)
+    pry (0.15.2)
+      coderay (~> 1.1)
+      method_source (~> 1.0)
+    racc (1.8.1)
+    rainbow (3.1.1)
+    rake (13.3.1)
+    regexp_parser (2.11.3)
+    rspec (3.13.2)
+      rspec-core (~> 3.13.0)
+      rspec-expectations (~> 3.13.0)
+      rspec-mocks (~> 3.13.0)
+    rspec-core (3.13.6)
+      rspec-support (~> 3.13.0)
+    rspec-expectations (3.13.5)
+      diff-lcs (>= 1.2.0, < 2.0)
+      rspec-support (~> 3.13.0)
+    rspec-mocks (3.13.7)
+      diff-lcs (>= 1.2.0, < 2.0)
+      rspec-support (~> 3.13.0)
+    rspec-support (3.13.6)
+    rubocop (1.81.7)
+      json (~> 2.3)
+      language_server-protocol (~> 3.17.0.2)
+      lint_roller (~> 1.1.0)
+      parallel (~> 1.10)
+      parser (>= 3.3.0.2)
+      rainbow (>= 2.2.2, < 4.0)
+      regexp_parser (>= 2.9.3, < 3.0)
+      rubocop-ast (>= 1.47.1, < 2.0)
+      ruby-progressbar (~> 1.7)
+      unicode-display_width (>= 2.4.0, < 4.0)
+    rubocop-ast (1.48.0)
+      parser (>= 3.3.7.2)
+      prism (~> 1.4)
+    ruby-progressbar (1.13.0)
+    unicode-display_width (3.2.0)
+      unicode-emoji (~> 4.1)
+    unicode-emoji (4.2.0)
+    unparser (0.8.1)
+      diff-lcs (~> 1.6)
+      parser (>= 3.3.0)
+      prism (>= 1.5.1)
+
+PLATFORMS
+  arm64-darwin-24
+  ruby
+
+DEPENDENCIES
+  bundler (~> 2.0)
+  minitest (~> 5.0)
+  mutation_tester!
+  pry
+  rake (~> 13.0)
+  rspec (~> 3.0)
+  rubocop
+
+BUNDLED WITH
+   2.6.9

data/LICENSE.txt

@@ -0,0 +1,19 @@
+Copyright (c) 2026 - MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/Rakefile

@@ -0,0 +1,57 @@
+require 'bundler/gem_tasks'
+require 'rspec/core/rake_task'
+require_relative 'lib/mutation_tester/rake_task'
+
+RSpec::Core::RakeTask.new(:spec)
+
+task default: :spec
+
+desc 'Run the full test suite (unit + integration RSpec + Minitest)'
+task :test do
+  puts '--- Running RSpec Suite (unit + integration) ---'
+  sh 'bundle exec rspec'
+
+  puts "\n--- Running Minitest Suite ---"
+  sh 'ruby test/mutator_test.rb'
+end
+
+namespace :test do
+  task :unit do
+    sh 'bundle exec rspec --exclude-pattern "**/integration_spec.rb"'
+  end
+
+  task :integration do
+    sh 'bundle exec rspec spec/integration_spec.rb'
+  end
+
+  task :minitest do
+    sh 'ruby test/mutator_test.rb'
+  end
+end
+
+desc 'Run example mutation test (default: RSpec)'
+task :example do
+  require_relative 'examples/run_example'
+end
+
+namespace :example do
+  desc 'Run RSpec example (serial execution)'
+  task :rspec do
+    require_relative 'examples/run_example_rspec'
+  end
+
+  desc 'Run Minitest example (serial execution)'
+  task :minitest do
+    require_relative 'examples/run_example_minitest'
+  end
+
+  desc 'Run RSpec example (parallel execution)'
+  task :rspec_parallel do
+    require_relative 'examples/run_example_parallel_rspec'
+  end
+
+  desc 'Run Minitest example (parallel execution)'
+  task :minitest_parallel do
+    require_relative 'examples/run_example_parallel_minitest'
+  end
+end

data/docs/ci.md

@@ -0,0 +1,158 @@
+# CI/CD Integration
+
+Ready-to-copy recipes for running MutationTester as a CI quality gate. See the
+[CI/CD integration](../readme.md#cicd-integration) section of the README for the
+overview and the shipped templates under
+[`examples/github_actions/`](../examples/github_actions).
+
+## Mutation testing in CI in 5 minutes
+
+The gem ships a ready-to-copy GitHub Actions workflow at
+[`examples/github_actions/mutation_test.yml`](../examples/github_actions/mutation_test.yml)
+(installed alongside the gem, so you also have it offline). Three steps:
+
+1. Add the gem to your bundle: `bundle add mutation_tester --group development,test`.
+2. Copy the template to `.github/workflows/mutation_test.yml`.
+3. Edit the lines marked `EDIT:` to point at your Ruby version and your source/test files.
+
+The workflow checks out your code, sets up Ruby with `bundler-cache`, runs
+`bundle exec mutation_test SOURCE TEST`, uploads the HTML/JSON reports from
+`tmp/mutation_reports/` as a build artifact (even on failure), and fails the job
+when the mutation score is below your threshold (the CLI exit code is the gate).
+It also carries commented variants for parallel execution
+(`MUTATION_TESTER_PARALLEL_PROCESSES`), for testing several file pairs, and for
+an incremental pull-request gate (`--since`/`--fail-fast`).
+
+## GitHub Actions (minimal inline workflow)
+
+The same thing, condensed to a copy-pasteable minimal workflow. It matches the
+maintained template above; reach for the template when you want the parallel and
+multi-file variants. The quality gate needs no extra configuration: the CLI exits
+non-zero when the mutation score is below the threshold, which fails the step (and
+the job).
+
+```yaml
+name: Mutation Testing
+
+on:
+  push:
+  pull_request:
+
+jobs:
+  mutation_test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Ruby
+        uses: ruby/setup-ruby@v1
+        with:
+          ruby-version: '3.3'   # EDIT: match your project's Ruby (floor is 3.0)
+          bundler-cache: true
+
+      - name: Run mutation tests
+        # EDIT: point at your own source file and its test file.
+        run: bundle exec mutation_test app/models/user.rb spec/models/user_spec.rb
+
+      - name: Upload mutation reports
+        if: always()   # keep the reports even when the gate failed the job
+        uses: actions/upload-artifact@v4
+        with:
+          name: mutation-reports
+          # Default output_dir; change it if you pass a custom --output-dir.
+          path: tmp/mutation_reports/
+```
+
+## GitHub Actions on pull requests (incremental gate)
+
+On a pull request you rarely want to mutate the whole project. Combine
+`--since` and `--fail-fast` to mutate only the files the PR changed and stop at
+the first surviving mutant. The maintained template
+[`examples/github_actions/mutation_test.yml`](../examples/github_actions/mutation_test.yml)
+carries this variant too; the condensed version:
+
+```yaml
+name: Mutation Testing (PR)
+
+on:
+  pull_request:
+
+jobs:
+  mutation_test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0   # full history so git can diff against the base branch
+
+      - name: Set up Ruby
+        uses: ruby/setup-ruby@v1
+        with:
+          ruby-version: '3.3'   # EDIT: match your project's Ruby
+          bundler-cache: true
+
+      - name: Run mutation tests on changed files only
+        # EDIT: point the glob at your own sources.
+        run: |
+          bundle exec mutation_test --glob 'lib/**/*.rb' \
+            --since "origin/${{ github.base_ref }}" --fail-fast
+
+      - name: Upload mutation reports
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: mutation-reports
+          path: tmp/mutation_reports/
+```
+
+A PR that touches nothing under `lib/` exits `0` with a "Nothing to mutate"
+message, so the gate never blocks unrelated changes.
+
+## Machine mode as a CI gate and artifact
+
+`--json` makes the CLI double as the gate (its exit code follows the threshold)
+and the artifact producer. Capture stdout to a file, upload it, and the run
+fails the job automatically when the score drops below the threshold:
+
+```yaml
+    - name: Run mutation tests (JSON report on stdout)
+      run: |
+        bundle exec mutation_test --json \
+          app/models/user.rb spec/models/user_spec.rb > mutation_report.json
+
+    - name: Upload JSON report
+      if: always()   # keep the report even when the gate failed the job
+      uses: actions/upload-artifact@v4
+      with:
+        name: mutation-report-json
+        path: mutation_report.json
+```
+
+## AI workflow (mutation gate)
+
+`--json` turns the report into a worklist for an AI agent (or a script): every
+**surviving** mutant carries `file_path`, `line`, `original` and `mutated`, which
+is a concrete, located test gap. A CI job can run the gate, publish those
+survivors, and hand them to an agent that proposes the missing tests.
+
+The gem ships a ready-to-copy workflow at
+[`examples/github_actions/ai_mutation_gate.yml`](../examples/github_actions/ai_mutation_gate.yml)
+(installed with the gem, so you have it offline). Copy it to
+`.github/workflows/ai_mutation_gate.yml`, add the gem to your bundle, and edit
+the `EDIT:` lines. It:
+
+1. runs `mutation_test --json` and captures the report,
+2. writes the score and a table of surviving mutants to the GitHub job summary
+   (`$GITHUB_STEP_SUMMARY`),
+3. extracts the survivors into a machine-readable `survivors.json` and uploads it
+   (with the full report) as an artifact even on a failing run,
+4. re-fails the job when the score is below the threshold, so it stays a gate.
+
+The survivor extraction is the same `jq` filter documented under
+[Extracting the score and the survived mutants](json-schema.md#extracting-the-score-and-the-survived-mutants),
+written to `survivors.json`. That file is what you feed the agent ("here are
+located test gaps; for each, write a focused test that would kill the mutant");
+the template carries a commented step showing where to wire your agent CLI. Use
+it alongside the plain [5-minute CI template](#mutation-testing-in-ci-in-5-minutes):
+the AI gate adds the surviving-mutant worklist, the plain template is just the
+pass/fail gate.

data/docs/execution-runners.md

@@ -0,0 +1,163 @@
+# Execution Runners and Test Selection
+
+Deep reference for how MutationTester executes each mutant and how two-phase test
+selection speeds up kills. See the [Execution model](../readme.md#execution-model)
+section of the README for the overview, the parallel/serial guidance, and how to
+force a runner.
+
+## Execution runners (fork, spawn, in-memory)
+
+Every mutant is executed by one of three runners:
+
+- **fork**: a helper process preloads the environment
+  once (RubyGems, Bundler and `rspec-core`, without loading the mutated file or
+  the specs), and each mutant runs in a fresh fork of that process. The fork
+  loads the spec only after the mutated source has been written, so every
+  mutant is visible and no state leaks between mutants. This removes most of
+  the fixed per-mutant boot cost, which matters on large suites and in CI.
+- **spawn**: each mutant starts a full new process (`bundle exec rspec ...`).
+  Slower per mutant, but works everywhere.
+- **in_memory** (default where supported): the helper process additionally preloads the spec
+  file and, through it, the original source, once per run. Each mutant then
+  runs in a fresh fork that re-evaluates the mutated source in memory
+  (redefining the loaded methods and class constants, with the
+  "already initialized constant" warning silenced only inside that child) and
+  runs the already-loaded examples. Nothing is written to disk on the mutation
+  hot path: the source file and the project directory stay untouched for the
+  whole run and no shadow workspaces are created. With `-p N` the preloaded
+  process is forked into a pool of clones and each parallel worker runs its
+  mutants in memory against its own clone. This brings the cost per mutation
+  below the fork runner and close to in-process mutation tools, at the price
+  of requiring loadable, re-evaluable code (see the limitations below).
+
+Selection is automatic (`auto`): the fastest safe path is tried first and every
+step down to a slower one prints a single stderr warning with its reason, so a
+fallback is never silent. The order is `in_memory` (RSpec with `Process.fork`
+available and a passing unmutated-source probe), then `fork`, then `spawn`.
+All runners produce identical scores and per-mutant statuses, and all enforce
+the same hard per-mutant timeout (monotonic deadline plus a process-group
+kill).
+
+| Mode        | Picked by `auto` when                                                                                                                                                   | Falls back to                                                                                                                              |
+|-------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------|
+| `in_memory` | the suite is RSpec, the platform has `Process.fork`, the file has no load-time `defined?` guard, and re-applying the unmutated source in a probe child passes the suite | `fork`/`spawn` (whole run) with a stderr warning naming the reason; a single worker dying mid-run falls back only for its share of mutants; a mutant that raises while being applied falls back alone |
+| `fork`      | the suite is RSpec, `Process.fork` is available, but in-memory is unavailable (each reason is printed)                                                                  | `spawn`, with a stderr warning, when the helper process fails to preload the environment                                                   |
+| `spawn`     | the suite is Minitest, or the platform has no `Process.fork`                                                                                                            | nothing; it works everywhere                                                                                                               |
+
+Forcing a mode with `--runner fork|spawn|in_memory` skips the auto attempts and
+uses that mode directly (`in_memory` keeps its own documented safety fallbacks;
+`fork` still degrades to `spawn` where `Process.fork` does not exist).
+
+### When to force --runner fork or spawn
+
+- **`--runner fork`**: your source file is not cleanly re-evaluable in memory
+  (heavy load-time side effects, code that must never be redefined) and you do
+  not want to rely on the automatic probe, but you still want the preloaded
+  environment speed.
+- **`--runner spawn`**: you want maximum isolation (one full pristine process
+  per mutant), you are debugging a suspicious result from a preloaded runner,
+  or your environment misbehaves with forked workers (for example C extensions
+  that do not survive `fork`).
+- Stay on the default `auto` everywhere else: it always announces on stderr
+  which path it took, so CI logs show exactly how the mutants were executed.
+
+Override the automatic choice with any of:
+
+```bash
+# CLI flag
+mutation_test --runner spawn lib/calculator.rb spec/calculator_spec.rb
+
+# Environment variable
+MUTATION_TESTER_RUNNER=spawn mutation_test lib/calculator.rb spec/calculator_spec.rb
+```
+
+```ruby
+MutationTester.configure do |config|
+  config.runner = :spawn
+end
+```
+
+### Limitations of the fork runner
+
+- Minitest suites always use `spawn` (fork support for Minitest is a separate
+  decision after RSpec experience is collected).
+- Platforms without `Process.fork` (for example Windows or JRuby) always use
+  `spawn`, even when `--runner fork` is requested.
+- If the helper process fails to preload the environment, the run warns once
+  and falls back to `spawn`.
+
+### Limitations of the in-memory runner
+
+The in-memory runner never fails silently: each case below falls back to
+file-based execution with a warning, and a mutant is marked `error` only when
+no fallback is possible.
+
+- RSpec only, and the file must be classic loadable code (classes/modules).
+  Minitest suites fall back to the file-based path with a warning.
+- With `-p N` (N > 1) the run stays fully in memory: the environment, the
+  original source and the specs are preloaded once, the preloaded process is
+  forked into N pooled clones, and every parallel worker applies each mutant
+  in a fresh fork of its own clone. If a pooled worker dies mid-run, that
+  worker finishes its share of mutants through the file-based path with a
+  warning; the other workers stay in memory.
+- Before any mutant runs, the runner re-applies the **unmutated** source in a
+  probe child and runs the suite. If that probe fails (for example the file has
+  top-level side effects that break on a second execution, or the class is
+  frozen so it cannot be reopened), the whole run falls back to file-based
+  execution with a warning instead of reporting false kills. In a parallel run
+  this fallback lands on the regular parallel file-based path, after the same
+  shadow-workspace sanity check that path always performs.
+- Files that use `defined?` at load time (for example
+  `X = 1 unless defined?(X)`) are rejected up front with a fallback warning:
+  the guard would silently skip the redefinition and mutants could falsely
+  survive. `defined?` inside method bodies is fine.
+- A mutant that raises while being applied in memory (for example a top-level
+  `raise`, a mutation that makes the file unloadable, or a `FrozenError` on
+  redefinition) falls back to file-based execution for that single mutant with
+  a warning: the mutant is decided from disk in a shadow workspace and that
+  result is final. An objectively unloadable mutant therefore counts as
+  `killed`, exactly as under the fork and spawn runners, while a failure
+  specific to the in-memory mechanics still gets an honest file-based verdict
+  instead of a false kill. Such a mutant is reported as `error` only when the
+  fallback is impossible because no project root (a `Gemfile` or a `.git`
+  directory) is discoverable above the source file.
+- `require_relative` in the mutated file is safe: it is idempotent on the
+  second evaluation because the file is already in `$LOADED_FEATURES`.
+- Two-phase test selection does not apply in this mode (serial or parallel);
+  every mutant runs the full preloaded example set (the examples are already
+  in memory, so the per-mutant cost stays low). Because no selection happens,
+  the `Selection:` summary line is omitted for in-memory runs; it reappears
+  only when a fallback actually executed mutants through the file-based path.
+
+## Test selection (fast kill with full-file confirmation)
+
+For RSpec suites, MutationTester runs each mutant in two phases instead of
+always paying for the whole spec file:
+
+1. **Fast kill (subset)**: for a mutant inside method `foo`, it first runs only
+   the examples whose group description matches the method
+   (`rspec spec_file -e '#foo' -e '.foo'`, following the common
+   `describe '#foo'` / `describe '.foo'` convention). If any of these examples
+   fail or time out, the mutant is finished right there: killed (or timeout)
+   without ever running the rest of the file.
+2. **Full-file confirmation**: if the subset passes, the full spec file is run
+   and only that result decides the status. A mutant can never be reported as
+   survived from the subset alone, so selection cannot introduce false
+   survivors.
+
+The heuristic degrades safely: when the mutant is not inside a method, when the
+spec file contains no `'#foo'` / `".foo"` group description, or when the suite
+is Minitest, the full file runs directly. Scores and per-mutant statuses are
+identical with and without selection on both file-based execution runners (fork
+and spawn); selection only changes how fast killed mutants die. The in-memory
+runner (the default path) performs no selection at all: every mutant runs the
+full preloaded example set, which is why its report omits the `Selection:`
+summary line (see the in-memory limitations above).
+
+Disable it with the `--no-test-selection` CLI flag or in Ruby:
+
+```ruby
+MutationTester.configure do |config|
+  config.test_selection = false
+end
+```