mutation_tester 1.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/CHANGELOG.md +13 -0
- data/Gemfile +9 -0
- data/Gemfile.lock +83 -0
- data/LICENSE.txt +19 -0
- data/Rakefile +57 -0
- data/docs/ci.md +158 -0
- data/docs/execution-runners.md +163 -0
- data/docs/json-schema.md +171 -0
- data/docs/mutation-types.md +207 -0
- data/examples/calculator.rb +35 -0
- data/examples/calculator_100_perc_spec.rb +121 -0
- data/examples/calculator_minitest.rb +53 -0
- data/examples/calculator_spec.rb +105 -0
- data/examples/github_actions/ai_mutation_gate.yml +75 -0
- data/examples/github_actions/mutation_test.yml +87 -0
- data/examples/hooks/pre-push +41 -0
- data/examples/run_example.rb +31 -0
- data/examples/run_example_minitest.rb +38 -0
- data/examples/run_example_parallel_minitest.rb +48 -0
- data/examples/run_example_parallel_rspec.rb +48 -0
- data/examples/run_example_rspec.rb +38 -0
- data/examples/spec_helper.rb +21 -0
- data/exe/mutation_test +345 -0
- data/lib/mutation_tester/batch_runner.rb +286 -0
- data/lib/mutation_tester/configuration.rb +105 -0
- data/lib/mutation_tester/core.rb +193 -0
- data/lib/mutation_tester/fork_runner/worker.rb +202 -0
- data/lib/mutation_tester/fork_runner.rb +382 -0
- data/lib/mutation_tester/framework_detector.rb +25 -0
- data/lib/mutation_tester/in_memory_loader.rb +25 -0
- data/lib/mutation_tester/mutation_runner.rb +644 -0
- data/lib/mutation_tester/mutator.rb +874 -0
- data/lib/mutation_tester/progress_display.rb +130 -0
- data/lib/mutation_tester/railtie.rb +7 -0
- data/lib/mutation_tester/rake_task.rb +18 -0
- data/lib/mutation_tester/reporters/base_reporter.rb +154 -0
- data/lib/mutation_tester/reporters/batch_json_reporter.rb +72 -0
- data/lib/mutation_tester/reporters/console_reporter.rb +130 -0
- data/lib/mutation_tester/reporters/html_reporter.rb +693 -0
- data/lib/mutation_tester/reporters/json_reporter.rb +67 -0
- data/lib/mutation_tester/test_command.rb +165 -0
- data/lib/mutation_tester/version.rb +3 -0
- data/lib/mutation_tester.rb +52 -0
- data/lib/tasks/mutation_tester.rake +80 -0
- data/readme.md +925 -0
- metadata +213 -0
data/readme.md
ADDED
|
@@ -0,0 +1,925 @@
|
|
|
1
|
+
# MutationTester 🧬
|
|
2
|
+
|
|
3
|
+
[](https://github.com/Oxyconit/mutation_tester/actions/workflows/ci.yml)
|
|
4
|
+
|
|
5
|
+
Simple mutation testing framework for Ruby applications with RSpec or Minitest. Improve your AI workflow and test
|
|
6
|
+
quality by identifying weak spots in your test suite through code mutations.
|
|
7
|
+
|
|
8
|
+
```bash
|
|
9
|
+
# Install, then mutation-test a file (its spec is found by convention):
|
|
10
|
+
bundle add mutation_tester
|
|
11
|
+
bundle exec mutation_test lib/calculator.rb
|
|
12
|
+
```
|
|
13
|
+
|
|
14
|
+
See [Getting Started](#getting-started) for the full quick start.
|
|
15
|
+
|
|
16
|
+
## Table of Contents
|
|
17
|
+
|
|
18
|
+
- [Features](#features)
|
|
19
|
+
- [Requirements and Compatibility](#requirements-and-compatibility)
|
|
20
|
+
- [Installation](#installation)
|
|
21
|
+
- [Getting Started](#getting-started)
|
|
22
|
+
- [Usage](#usage)
|
|
23
|
+
- [Configuration](#configuration)
|
|
24
|
+
- [Execution model](#execution-model)
|
|
25
|
+
- [Mutation types](#mutation-types)
|
|
26
|
+
- [Equivalent mutants](#equivalent-mutants)
|
|
27
|
+
- [Reports and output](#reports-and-output)
|
|
28
|
+
- [Pre-push hook](#pre-push-hook)
|
|
29
|
+
- [CI/CD integration](#cicd-integration)
|
|
30
|
+
- [Troubleshooting](#troubleshooting)
|
|
31
|
+
- [Development](#development)
|
|
32
|
+
- [Contributing](#contributing)
|
|
33
|
+
- [License](#license)
|
|
34
|
+
|
|
35
|
+
Reference material lives under [`docs/`](docs): the full
|
|
36
|
+
[mutation catalog](docs/mutation-types.md), the [JSON report schema](docs/json-schema.md),
|
|
37
|
+
and the [CI/CD recipes](docs/ci.md).
|
|
38
|
+
|
|
39
|
+
## Features
|
|
40
|
+
|
|
41
|
+
- ✅ **Multiple Mutation Types**: Arithmetic, comparison, logical, boolean, number, string, conditional, call-removal,
|
|
42
|
+
nil-injection, and argument mutations, plus an opt-in strict-equality mode
|
|
43
|
+
- 📊 **Rich Reporting for you and your AI workflow**: Console, HTML, and JSON reports
|
|
44
|
+
- ⚡ **Fast by Default**: Runs mutants in memory and in parallel on the available CPU cores out of the box, with an
|
|
45
|
+
always-announced fallback to file-based execution
|
|
46
|
+
- 🔧 **Configurable**: Customize mutation types, parallel processes, and thresholds
|
|
47
|
+
- 🚀 **Rails Integration**: Rake tasks for easy integration
|
|
48
|
+
- 💎 **Clean API**: Simple and intuitive interface
|
|
49
|
+
|
|
50
|
+
## Requirements and Compatibility
|
|
51
|
+
|
|
52
|
+
MutationTester keeps its requirements low so it drops into a wide range of projects.
|
|
53
|
+
|
|
54
|
+
| Component | Supported | Notes |
|
|
55
|
+
|-------------------------|-----------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
|
56
|
+
| Ruby | `>= 3.0` | Floor is Ruby 3.0. CI runs the suite on 3.0, 3.1, 3.2 and 3.3. The gem is developed on Ruby 4.0.2, and 4.x is supported. Ruby 4.x has no prebuilt binary on the GitHub-hosted runners yet, so it is verified on the development host rather than in the CI matrix. |
|
|
57
|
+
| RSpec (your project) | `3.x` | The gem shells out to your project's own `rspec`, so any RSpec 3.x works. Both ends of the range are exercised by a real mutation run: the lowest 3.0.x line in the CI framework matrix, and 3.13.x in the example jobs. |
|
|
58
|
+
| Minitest (your project) | `5.x` and `6.x` | The gem shells out to your project's own `ruby test_file.rb`, so both the 5.x and 6.x lines work. Both are exercised by a real mutation run: 5.x in the example jobs, and 6.x in the CI framework matrix. |
|
|
59
|
+
|
|
60
|
+
Notes:
|
|
61
|
+
|
|
62
|
+
- These are the versions the gem runs *your* tests against. It detects the framework, runs a baseline, then runs the
|
|
63
|
+
covering tests for each mutant, all inside your project's environment (through `bundle exec` when your project has a
|
|
64
|
+
`Gemfile`).
|
|
65
|
+
- Every advertised framework version is backed by CI, not just the ones the gem's dev bundle happens to pin. A dedicated
|
|
66
|
+
`framework-matrix` job (see `.github/workflows/ci.yml`) installs an alternate user-project bundle for each boundary
|
|
67
|
+
version (`gemfiles/minitest6.gemfile` and `gemfiles/rspec3_low.gemfile`) and runs an end-to-end mutation run against
|
|
68
|
+
it via `gemfiles/run_matrix.rb`; the job is green only when every mutant is killed on that version.
|
|
69
|
+
- The gem's *own* test suite pins `minitest ~> 5.0`, because Minitest 6.x dropped `minitest/mock`, which those internal
|
|
70
|
+
tests use. That pin does not constrain your project: your project may use Minitest 5.x or 6.x, and the 6.x path is
|
|
71
|
+
verified through the framework matrix above rather than the gem's dev bundle.
|
|
72
|
+
- On Ruby 3.4+/4.x the bundled `parser` gem prints one benign syntax-level warning per run (it recognizes syntax up to
|
|
73
|
+
Ruby 3.3); see [Parser Version Warning on Newer Ruby](#parser-version-warning-on-newer-ruby-supported-syntax-level).
|
|
74
|
+
|
|
75
|
+
## Installation
|
|
76
|
+
|
|
77
|
+
MutationTester requires Ruby >= 3.0 (see [Requirements and Compatibility](#requirements-and-compatibility)). There are
|
|
78
|
+
two supported ways to install it.
|
|
79
|
+
|
|
80
|
+
### In your project's bundle (recommended)
|
|
81
|
+
|
|
82
|
+
Add it to your application's `Gemfile` and install in one step:
|
|
83
|
+
|
|
84
|
+
```bash
|
|
85
|
+
bundle add mutation_tester
|
|
86
|
+
```
|
|
87
|
+
|
|
88
|
+
(this writes `gem "mutation_tester"` into your `Gemfile` and runs `bundle install`).
|
|
89
|
+
|
|
90
|
+
Then run it through Bundler so it uses your project's locked dependency versions:
|
|
91
|
+
|
|
92
|
+
```bash
|
|
93
|
+
bundle exec mutation_test app/models/user.rb spec/models/user_spec.rb
|
|
94
|
+
```
|
|
95
|
+
|
|
96
|
+
### As a global gem
|
|
97
|
+
|
|
98
|
+
Install it once, system-wide:
|
|
99
|
+
|
|
100
|
+
```bash
|
|
101
|
+
gem install mutation_tester
|
|
102
|
+
```
|
|
103
|
+
|
|
104
|
+
Then run the `mutation_test` command directly (no `bundle exec`):
|
|
105
|
+
|
|
106
|
+
```bash
|
|
107
|
+
mutation_test app/models/user.rb spec/models/user_spec.rb
|
|
108
|
+
```
|
|
109
|
+
|
|
110
|
+
This is convenient for a project that does not list the gem in its `Gemfile`.
|
|
111
|
+
The CLI then cannot load itself from that project bundle, so it loads the
|
|
112
|
+
globally installed gem *outside* the bundle and prints one line to stderr:
|
|
113
|
+
|
|
114
|
+
```
|
|
115
|
+
mutation_tester loaded outside the project bundle
|
|
116
|
+
```
|
|
117
|
+
|
|
118
|
+
In this fallback the gem and its own dependencies (parser, unparser, parallel,
|
|
119
|
+
rainbow) come from the global install, so their versions may differ from your
|
|
120
|
+
project's `Gemfile.lock`. Your own tests are unaffected: when your project has a
|
|
121
|
+
`Gemfile`, each mutant still runs through `bundle exec`, in your project's
|
|
122
|
+
environment. Running in a directory with no `Gemfile` at all works too and
|
|
123
|
+
prints no notice. If your project *does* list `mutation_tester`, prefer
|
|
124
|
+
`bundle exec mutation_test ...`, which runs fully inside your bundle with no
|
|
125
|
+
notice.
|
|
126
|
+
|
|
127
|
+
## Getting Started
|
|
128
|
+
|
|
129
|
+
Install the gem, then point `mutation_test` at one or more source files. Each
|
|
130
|
+
file is mapped to its spec by convention (`lib/X.rb` -> `spec/X_spec.rb`,
|
|
131
|
+
override with `--spec-glob`):
|
|
132
|
+
|
|
133
|
+
```bash
|
|
134
|
+
# Install
|
|
135
|
+
bundle add mutation_tester
|
|
136
|
+
|
|
137
|
+
# Test a file: the spec is found by convention (lib/X.rb -> spec/X_spec.rb)
|
|
138
|
+
bundle exec mutation_test lib/calculator.rb
|
|
139
|
+
|
|
140
|
+
# Test several files in one aggregated run
|
|
141
|
+
bundle exec mutation_test lib/calculator.rb lib/parser.rb
|
|
142
|
+
|
|
143
|
+
# Test exactly what you have staged in git (the "test what I changed" flow)
|
|
144
|
+
bundle exec mutation_test --staged
|
|
145
|
+
|
|
146
|
+
# Minitest layout under test/
|
|
147
|
+
bundle exec mutation_test --spec-glob 'test/{name}_test.rb' lib/calculator.rb
|
|
148
|
+
|
|
149
|
+
# Or point at the test file explicitly: exactly two arguments, the second a test file
|
|
150
|
+
bundle exec mutation_test app/models/user.rb spec/models/user_spec.rb
|
|
151
|
+
bundle exec mutation_test lib/calculator.rb test/calculator_test.rb
|
|
152
|
+
|
|
153
|
+
# Or with rake (non-Rails: first add `require 'mutation_tester/rake_task'` to your Rakefile)
|
|
154
|
+
bundle exec rake "mutation_test[app/models/user.rb,spec/models/user_spec.rb]"
|
|
155
|
+
```
|
|
156
|
+
|
|
157
|
+
That's it! 🎉 The gem will:
|
|
158
|
+
|
|
159
|
+
- ✅ Run your original tests to make sure they pass
|
|
160
|
+
- ✅ Generate mutations of your code
|
|
161
|
+
- ✅ Run tests against each mutation
|
|
162
|
+
- ✅ Generate reports showing which mutations survived
|
|
163
|
+
|
|
164
|
+
Press Ctrl+C at any time to stop early: the run cleans up its worker processes and temporary workspaces, prints a single
|
|
165
|
+
interruption line (no backtrace), and exits with status 130.
|
|
166
|
+
|
|
167
|
+
See [Usage](#usage) for the full command reference and [Configuration](#configuration) to tune it.
|
|
168
|
+
|
|
169
|
+
## Usage
|
|
170
|
+
|
|
171
|
+
`mutation_test` is the main CLI executable for running mutation tests. Pass one
|
|
172
|
+
or more source files (specs mapped by convention), the git staging area, an
|
|
173
|
+
explicit pair, or a glob:
|
|
174
|
+
|
|
175
|
+
```bash
|
|
176
|
+
# Main interface: one or more source files, specs mapped by convention
|
|
177
|
+
mutation_test [OPTIONS] FILE...
|
|
178
|
+
|
|
179
|
+
# The files currently staged in git (see File lists and --staged below)
|
|
180
|
+
mutation_test [OPTIONS] --staged
|
|
181
|
+
|
|
182
|
+
# Explicit pair: exactly two arguments where the second is a test file
|
|
183
|
+
mutation_test [OPTIONS] SOURCE_FILE TEST_FILE
|
|
184
|
+
|
|
185
|
+
# Many files in one run: select sources with a glob (see Batch mode below)
|
|
186
|
+
mutation_test [OPTIONS] --glob 'lib/**/*.rb'
|
|
187
|
+
```
|
|
188
|
+
|
|
189
|
+
**Arguments:**
|
|
190
|
+
|
|
191
|
+
- `FILE...` - Ruby source files to mutate; each is mapped to its spec by convention (`lib/X.rb` -> `spec/X_spec.rb`,
|
|
192
|
+
override with `--spec-glob`). Files that cannot be mutated are reported as skipped with a reason (
|
|
193
|
+
see [File lists and --staged](#file-lists-and---staged-test-what-you-changed)).
|
|
194
|
+
- `SOURCE_FILE TEST_FILE` - Explicit pair: with exactly two arguments where the second is recognized as a test file (
|
|
195
|
+
`*_spec.rb`, `*.spec.rb`, `*_test.rb`, `test_*.rb`, or a file requiring minitest), the second is used as the test file
|
|
196
|
+
directly.
|
|
197
|
+
|
|
198
|
+
### CLI options
|
|
199
|
+
|
|
200
|
+
| Flag | Description |
|
|
201
|
+
|---|---|
|
|
202
|
+
| `-p, --parallel N` | Run with N parallel processes (default: auto, derived from the CPU core count with a cap of 8; `-p 1` forces serial execution). |
|
|
203
|
+
| `--runner MODE` | Mutant execution runner: `auto` (default) tries `in_memory` first (RSpec with `Process.fork` available and a passing unmutated-source probe), then falls back to `fork`, then `spawn`, announcing every step down on stderr with its reason; `fork` (preloaded environment, RSpec on platforms with `Process.fork`), `spawn` (one full process per mutant) and `in_memory` (mutations applied in child-process memory, zero file writes per mutant, RSpec only) force the specific mode. See [Execution runners](#execution-runners-fork-spawn-in-memory). |
|
|
204
|
+
| `--staged` | Mutation-test the files staged in git (`git diff --cached --name-only`; files staged as deleted are ignored), mapping each to its spec like a positional `FILE` list. Cannot be combined with positional arguments or `--glob`. See [File lists and --staged](#file-lists-and---staged-test-what-you-changed). |
|
|
205
|
+
| `--glob PATTERN` | Batch mode: mutation-test every source file matching `PATTERN`, mapping each to its spec by convention (see [Batch mode](#batch-mode-run-many-files-in-one-command)). |
|
|
206
|
+
| `--spec-glob TEMPLATE` | Spec-mapping template with a `{name}` placeholder (default: `spec/{name}_spec.rb`). Requires a positional `FILE` list, `--staged`, or `--glob`. |
|
|
207
|
+
| `--since REV` | Incremental batch mode: mutate only the files matched by `--glob` that changed since git revision `REV` (new files count as changed). Requires `--glob`. See [Incremental mode](#incremental-mode-mutate-only-what-changed). |
|
|
208
|
+
| `--fail-fast` | Stop the run at the first surviving mutant and finish with a failing status. Works in single-file mode and with `--glob`. |
|
|
209
|
+
| `--worker-env NAME` | Set environment variable `NAME` to a distinct per-worker value before each parallel worker boots (`parallel_tests` `TEST_ENV_NUMBER` convention: worker 0 -> `""`, worker N -> `N+1`), so a `parallel_tests`-style `database.yml` selects a per-worker database. You provision the databases (e.g. `rake parallel:prepare`). Not supported by the `in_memory` runner (it falls back to `fork`). See [Making parallelism work with Rails](#making-parallelism-work-with-rails). |
|
|
210
|
+
| `--strict-equality` | Enable the opt-in strict-equality probes (`==` → `eql?` and `==` → `equal?`). Default off; expect noise on code that does not distinguish numeric types or object identity. See [Strict Equality Mutations](docs/mutation-types.md#strict-equality-mutations-opt-in). |
|
|
211
|
+
| `-h, --help` | Show help message. |
|
|
212
|
+
| `-v, --version` | Show version. |
|
|
213
|
+
| `--verbose` | Show a per-mutation warning for every skipped mutation (quiet by default; the "Generated N mutations, skipped M" summary always prints when mutants are dropped). |
|
|
214
|
+
| `--no-progress` | Disable progress display. |
|
|
215
|
+
| `--no-test-selection` | Disable two-phase test selection and always run the full test file for every mutant. See [Test selection](#test-selection-fast-kill-with-full-file-confirmation). |
|
|
216
|
+
| `--json` | Machine mode: print ONLY the JSON report to stdout (banner, progress and colours go to stderr). A single file prints the per-file report; a multi-file run (`FILE` list with more than one file, `--staged`, `--glob`) prints one aggregate envelope with a condensed `survivors` list. See [Reports and output](#reports-and-output). |
|
|
217
|
+
| `--reporters LIST` | Comma-separated reporters to run: `console`, `html`, `json` (default: `console,html,json`). An unknown name errors and exits 1. |
|
|
218
|
+
| `--output-dir PATH` | Directory for the generated report files (default: `tmp/mutation_reports`). In batch mode each file writes to its own subdirectory under this path. |
|
|
219
|
+
|
|
220
|
+
```bash
|
|
221
|
+
# Choose which reporters run and where their files land
|
|
222
|
+
bundle exec mutation_test --reporters json,html --output-dir build/mutation \
|
|
223
|
+
app/models/user.rb spec/models/user_spec.rb
|
|
224
|
+
```
|
|
225
|
+
|
|
226
|
+
### Running with rake
|
|
227
|
+
|
|
228
|
+
You can also run mutation tests through rake tasks.
|
|
229
|
+
|
|
230
|
+
In a **non-Rails** project, require the tasks from your `Rakefile`:
|
|
231
|
+
|
|
232
|
+
```ruby
|
|
233
|
+
# Rakefile
|
|
234
|
+
require 'mutation_tester/rake_task'
|
|
235
|
+
```
|
|
236
|
+
|
|
237
|
+
In a **Rails** app the tasks load automatically through the gem's railtie, so no
|
|
238
|
+
Rakefile change is needed.
|
|
239
|
+
|
|
240
|
+
Then run either task. Rake takes the file arguments inside brackets (not
|
|
241
|
+
space-separated), so quote the invocation for your shell:
|
|
242
|
+
|
|
243
|
+
```bash
|
|
244
|
+
# Top-level task
|
|
245
|
+
bundle exec rake "mutation_test[app/models/user.rb,spec/models/user_spec.rb]"
|
|
246
|
+
|
|
247
|
+
# Namespaced task
|
|
248
|
+
bundle exec rake "mutation:test[app/models/user.rb,spec/models/user_spec.rb]"
|
|
249
|
+
```
|
|
250
|
+
|
|
251
|
+
### Programmatic usage
|
|
252
|
+
|
|
253
|
+
You can also run the gem directly from Ruby with `MutationTester.run(source, test)`:
|
|
254
|
+
|
|
255
|
+
```ruby
|
|
256
|
+
# RSpec
|
|
257
|
+
MutationTester.run('examples/calculator.rb', 'examples/calculator_spec.rb')
|
|
258
|
+
|
|
259
|
+
# Minitest
|
|
260
|
+
MutationTester.run('examples/calculator.rb', 'examples/calculator_minitest.rb')
|
|
261
|
+
```
|
|
262
|
+
|
|
263
|
+
### File lists and --staged: test what you changed
|
|
264
|
+
|
|
265
|
+
Passing one or more source files is the main interface. Each file is mapped to
|
|
266
|
+
its spec by convention (`lib/X.rb` -> `spec/X_spec.rb`, override with
|
|
267
|
+
`--spec-glob`), and the whole list runs as one aggregated batch: every file is
|
|
268
|
+
processed, the summary shows one `PASS`/`FAIL` line per file, reports land in
|
|
269
|
+
per-file subdirectories, and the exit code reflects the whole run.
|
|
270
|
+
|
|
271
|
+
```bash
|
|
272
|
+
bundle exec mutation_test lib/calculator.rb lib/parser.rb
|
|
273
|
+
|
|
274
|
+
# The natural "test what I changed" flow:
|
|
275
|
+
bundle exec mutation_test --staged
|
|
276
|
+
bundle exec mutation_test $(git diff --cached --name-only)
|
|
277
|
+
```
|
|
278
|
+
|
|
279
|
+
`--staged` reads the list from the git staging area (`git diff --cached
|
|
280
|
+
--name-only`), so the two commands above are equivalent when run from the
|
|
281
|
+
repository root. Files staged as deleted are ignored. Outside a git repository,
|
|
282
|
+
or when nothing is staged, the CLI prints a readable error instead of running.
|
|
283
|
+
`--staged` cannot be combined with positional arguments or `--glob`.
|
|
284
|
+
|
|
285
|
+
When any mutant survives, the aggregate summary ends with a survivors section:
|
|
286
|
+
one `file:line original -> mutated` line per surviving mutant. A surviving
|
|
287
|
+
mutant is a change to your code that your tests do not detect, so each line is
|
|
288
|
+
a concrete test gap to close. With zero survivors the section is absent.
|
|
289
|
+
|
|
290
|
+
The list may contain anything a real `git diff` produces; unmutable entries are
|
|
291
|
+
reported as `SKIPPED` with an explicit reason and never count as a success:
|
|
292
|
+
|
|
293
|
+
- **file not found** - the path does not exist.
|
|
294
|
+
- **not a Ruby source file** - e.g. a staged `.md` or config file.
|
|
295
|
+
- **a test file, not a mutable source** - a test file passed directly
|
|
296
|
+
(`*_spec.rb`, `*.spec.rb`, `*_test.rb`, `test_*.rb`, or minitest content).
|
|
297
|
+
- **no matching spec file** - the convention (or `--spec-glob`) points at a
|
|
298
|
+
spec that does not exist; the expected path is printed.
|
|
299
|
+
|
|
300
|
+
Exit codes: `0` when at least one file was processed and every processed file
|
|
301
|
+
met the threshold; `1` when any processed file was below threshold or when
|
|
302
|
+
every listed file was skipped (nothing was actually mutation-tested); `2` for
|
|
303
|
+
usage errors (flag conflicts, `--staged` outside a git repository).
|
|
304
|
+
|
|
305
|
+
**Legacy pair heuristic:** exactly two arguments where the second is recognized
|
|
306
|
+
as a test file (by the name patterns above or by minitest content) keep the
|
|
307
|
+
original `SOURCE_FILE TEST_FILE` behavior. Two source files enter list mode. An
|
|
308
|
+
RSpec test file with an unconventional name (no `_spec.rb` suffix) is not
|
|
309
|
+
recognized, so that pair is treated as a file list; rename the test or use the
|
|
310
|
+
conventional layout to get the explicit pair.
|
|
311
|
+
|
|
312
|
+
### Batch mode: run many files in one command
|
|
313
|
+
|
|
314
|
+
`--glob PATTERN` mutation-tests every source file the pattern matches in a single
|
|
315
|
+
run, so you no longer need to script a loop around `mutation_test` or depend on
|
|
316
|
+
the Rails-only `rake mutation:test_models` task.
|
|
317
|
+
|
|
318
|
+
Each matched source file is mapped to its spec by convention: `lib/X.rb` becomes
|
|
319
|
+
`spec/X_spec.rb`. Concretely the spec path is `spec/{name}_spec.rb` where `{name}`
|
|
320
|
+
is the source path with a leading `lib/` segment removed and the `.rb` extension
|
|
321
|
+
stripped, subdirectories preserved (`lib/foo/bar.rb` -> `spec/foo/bar_spec.rb`).
|
|
322
|
+
|
|
323
|
+
Override the convention with `--spec-glob TEMPLATE`, a template containing the
|
|
324
|
+
`{name}` placeholder. For a Minitest project laid out under `test/`:
|
|
325
|
+
|
|
326
|
+
```bash
|
|
327
|
+
bundle exec mutation_test --glob 'lib/**/*.rb' --spec-glob 'test/{name}_test.rb'
|
|
328
|
+
```
|
|
329
|
+
|
|
330
|
+
Behaviour:
|
|
331
|
+
|
|
332
|
+
- **Every file is processed.** A file whose score is below the threshold does not
|
|
333
|
+
abort the batch; the run continues to the next file (the `mutation:test_models`
|
|
334
|
+
pattern).
|
|
335
|
+
- **Reports never overwrite each other.** Each processed file writes its reporter
|
|
336
|
+
output to its own subdirectory under `--output-dir` (a slug derived from the
|
|
337
|
+
source path), so per-file HTML/JSON reports coexist.
|
|
338
|
+
- **A console aggregate summary** is printed at the end: one line per processed
|
|
339
|
+
file with its score and `PASS`/`FAIL` against the threshold, followed by a
|
|
340
|
+
clearly separated `SKIPPED` list.
|
|
341
|
+
- **A source file with no matching spec is `SKIPPED`**, reported explicitly and
|
|
342
|
+
never counted as a success. A skipped file does not by itself fail the run.
|
|
343
|
+
|
|
344
|
+
Exit codes:
|
|
345
|
+
|
|
346
|
+
- `0` - every processed file met the mutation score threshold (including a
|
|
347
|
+
`--since` run where nothing changed, see below).
|
|
348
|
+
- `1` - at least one processed file was below threshold, the glob matched no
|
|
349
|
+
source files at all, or `--fail-fast` stopped the run at a surviving mutant.
|
|
350
|
+
- `2` - a usage error: `--spec-glob` with an explicit `SOURCE_FILE TEST_FILE`
|
|
351
|
+
pair, `--since` given without `--glob`, `--staged` combined with positional
|
|
352
|
+
arguments or `--glob`, or `--since`/`--staged` used outside a git repository
|
|
353
|
+
(for `--since` also an unknown revision).
|
|
354
|
+
|
|
355
|
+
```bash
|
|
356
|
+
# Minitest project, JSON report per file, custom output directory
|
|
357
|
+
bundle exec mutation_test --glob 'lib/**/*.rb' --spec-glob 'test/{name}_test.rb' \
|
|
358
|
+
--reporters json --output-dir build/mutation
|
|
359
|
+
```
|
|
360
|
+
|
|
361
|
+
### Incremental mode: mutate only what changed
|
|
362
|
+
|
|
363
|
+
`--since REV` narrows a `--glob` batch to the files that changed since a git
|
|
364
|
+
revision, which is how you keep mutation testing affordable on pull requests:
|
|
365
|
+
|
|
366
|
+
```bash
|
|
367
|
+
bundle exec mutation_test --glob 'lib/**/*.rb' --since origin/main
|
|
368
|
+
```
|
|
369
|
+
|
|
370
|
+
Behaviour:
|
|
371
|
+
|
|
372
|
+
- A matched file counts as **changed** when `git diff --name-only REV` lists it;
|
|
373
|
+
new files the revision does not know about (committed or still untracked) also
|
|
374
|
+
count as changed.
|
|
375
|
+
- Unchanged matched files are reported as skipped in the batch summary
|
|
376
|
+
(`SKIPPED (unchanged since REV)`), never mutated.
|
|
377
|
+
- When nothing matched by the glob changed since `REV`, the run succeeds with a
|
|
378
|
+
"Nothing to mutate" message and exit code `0`, so a PR that does not touch
|
|
379
|
+
your sources does not fail the gate.
|
|
380
|
+
- Outside a git repository (or without a `git` executable), or with a revision
|
|
381
|
+
the repository does not know, the CLI prints a readable error and exits `2`
|
|
382
|
+
before any mutation runs.
|
|
383
|
+
|
|
384
|
+
### Fail fast: stop at the first surviving mutant
|
|
385
|
+
|
|
386
|
+
`--fail-fast` turns the run into a cheap gate: the run stops as soon as one
|
|
387
|
+
mutant survives, the reports contain the results obtained up to that point with
|
|
388
|
+
a clear interruption notice, and the process finishes with exit code `1`. It
|
|
389
|
+
works in single-file mode, with `--glob` (the batch stops and remaining files
|
|
390
|
+
are not run), and in parallel mode:
|
|
391
|
+
|
|
392
|
+
```bash
|
|
393
|
+
bundle exec mutation_test --glob 'lib/**/*.rb' --since origin/main --fail-fast
|
|
394
|
+
```
|
|
395
|
+
|
|
396
|
+
## Configuration
|
|
397
|
+
|
|
398
|
+
```ruby
|
|
399
|
+
MutationTester.configure do |config|
|
|
400
|
+
# Parallel by default: the process count is derived from the CPU core count
|
|
401
|
+
# (capped at 8, minimum 1). Set an explicit value to override the default,
|
|
402
|
+
# or force serial execution (recommended for Rails apps sharing a database):
|
|
403
|
+
# config.parallel_processes = 1
|
|
404
|
+
config.parallel_processes = 4
|
|
405
|
+
|
|
406
|
+
# Timeout for each test run (seconds)
|
|
407
|
+
config.timeout = 30
|
|
408
|
+
|
|
409
|
+
# Mutant execution runner: :auto (default), :fork, :spawn or :in_memory.
|
|
410
|
+
# :auto picks the fastest safe path and announces any fallback on stderr;
|
|
411
|
+
# setting a specific mode forces it. See the Execution model section below.
|
|
412
|
+
config.runner = :auto
|
|
413
|
+
|
|
414
|
+
# Two-phase test selection (RSpec only): fast-kill on a matching example
|
|
415
|
+
# subset, always confirmed by the full file before a mutant is reported as
|
|
416
|
+
# survived. Set to false to always run the full file. See Execution model.
|
|
417
|
+
config.test_selection = true
|
|
418
|
+
|
|
419
|
+
# Hard deadline for the single baseline run of the whole suite (seconds).
|
|
420
|
+
# Runs every example once, so it is looser than the per-mutant timeout above.
|
|
421
|
+
# Set to nil to disable the baseline deadline.
|
|
422
|
+
config.baseline_timeout = 300
|
|
423
|
+
|
|
424
|
+
# Report formats
|
|
425
|
+
config.reporters = [:console, :html, :json]
|
|
426
|
+
# Where reports are written. Defaults to "tmp/mutation_reports".
|
|
427
|
+
# Set it explicitly to write elsewhere (this example uses "mutation_reports"):
|
|
428
|
+
config.output_dir = "mutation_reports"
|
|
429
|
+
|
|
430
|
+
# Quality thresholds
|
|
431
|
+
config.minimum_score = 80.0
|
|
432
|
+
config.fail_on_threshold = true
|
|
433
|
+
|
|
434
|
+
# Display options
|
|
435
|
+
# Quiet by default. When true, a per-mutation "Warning: skipped ..." line is
|
|
436
|
+
# printed for each skipped mutation. The aggregate "Generated N mutations,
|
|
437
|
+
# skipped M" summary is always printed when mutants are dropped, regardless of
|
|
438
|
+
# this flag. The CLI --verbose flag sets this to true.
|
|
439
|
+
config.verbose = false
|
|
440
|
+
config.show_file_path = true # Show full file path with line number
|
|
441
|
+
config.show_progress = true # Show live progress during mutation testing
|
|
442
|
+
|
|
443
|
+
# Enable/disable mutation types
|
|
444
|
+
config.mutation_types = {
|
|
445
|
+
arithmetic: true,
|
|
446
|
+
comparison: true,
|
|
447
|
+
logical: true,
|
|
448
|
+
boolean: true,
|
|
449
|
+
number: true,
|
|
450
|
+
string: true,
|
|
451
|
+
conditional: true,
|
|
452
|
+
call_removal: true,
|
|
453
|
+
nil_injection: true,
|
|
454
|
+
argument: true,
|
|
455
|
+
strict_equality: false # Opt-in: == -> eql?/equal? probes (see Strict Equality Mutations)
|
|
456
|
+
}
|
|
457
|
+
end
|
|
458
|
+
```
|
|
459
|
+
|
|
460
|
+
## Execution model
|
|
461
|
+
|
|
462
|
+
MutationTester runs in parallel by default and picks the fastest safe execution
|
|
463
|
+
runner automatically. This section covers when to override those defaults, how
|
|
464
|
+
the runners differ, and how two-phase test selection speeds up kills.
|
|
465
|
+
|
|
466
|
+
### Parallel execution (on by default)
|
|
467
|
+
|
|
468
|
+
By default, mutation_tester runs in **parallel**: the process count is derived
|
|
469
|
+
from the number of CPU cores (`Etc.nprocessors`), capped at 8 and never below 1.
|
|
470
|
+
Force a specific count, or serial execution, when your tests need it:
|
|
471
|
+
|
|
472
|
+
```bash
|
|
473
|
+
# Using CLI flag
|
|
474
|
+
bundle exec mutation_test app/models/user.rb spec/models/user_spec.rb --parallel 4
|
|
475
|
+
|
|
476
|
+
# Force serial execution (recommended for Rails apps sharing one test database)
|
|
477
|
+
bundle exec mutation_test app/models/user.rb spec/models/user_spec.rb -p 1
|
|
478
|
+
|
|
479
|
+
# Using environment variable
|
|
480
|
+
MUTATION_TESTER_PARALLEL_PROCESSES=4 bundle exec mutation_test app/models/user.rb spec/models/user_spec.rb
|
|
481
|
+
```
|
|
482
|
+
|
|
483
|
+
Precedence is: an explicit `--parallel/-p` flag overrides `MUTATION_TESTER_PARALLEL_PROCESSES`, which overrides the
|
|
484
|
+
auto-derived core count; an explicit `config.parallel_processes` assignment in Ruby also replaces the auto default. An
|
|
485
|
+
invalid value (less than 1, or non-numeric) falls back to 1 with a warning on stderr. `-p 1` forces serial execution.
|
|
486
|
+
|
|
487
|
+
**⚠️ Force serial execution with `-p 1` if your tests share database state!**
|
|
488
|
+
|
|
489
|
+
In parallel mode each mutant runs in an isolated shadow workspace. Every `.rb`
|
|
490
|
+
file is a physical copy (non-Ruby files stay symlinks for speed), so mutations
|
|
491
|
+
apply correctly even when a spec loads the source indirectly (e.g. via
|
|
492
|
+
`spec_helper`). The parallel mutation score therefore matches serial.
|
|
493
|
+
|
|
494
|
+
### When to use serial vs parallel execution
|
|
495
|
+
|
|
496
|
+
Use **parallel execution** (the default) for:
|
|
497
|
+
|
|
498
|
+
- ⚡ **Pure Ruby classes** - No database, no shared state
|
|
499
|
+
- ⚡ **Unit tests with mocks** - Fast and independent tests
|
|
500
|
+
- ⚡ **Large codebases** - Significant time savings
|
|
501
|
+
- ⚡ **CI/CD with powerful machines** - Make use of available resources
|
|
502
|
+
|
|
503
|
+
Force **serial execution with `-p 1`** for:
|
|
504
|
+
|
|
505
|
+
- ✅ **Rails applications with database** - Avoids conflicts
|
|
506
|
+
- ✅ **Tests that share state** - No interference between test runs
|
|
507
|
+
- ✅ **First time using mutation testing** - Easier to debug
|
|
508
|
+
- ✅ **Limited system resources** - Less memory/CPU usage
|
|
509
|
+
|
|
510
|
+
### Making parallelism work with Rails
|
|
511
|
+
|
|
512
|
+
Parallel workers get an isolated filesystem (each mutant runs in its own shadow workspace), but they share one
|
|
513
|
+
**database** unless you give each worker its own. If your app is already set up for `parallel_tests` (a `database.yml`
|
|
514
|
+
keyed on `TEST_ENV_NUMBER` and per-worker databases created with `rake parallel:prepare`), `--worker-env` bridges the
|
|
515
|
+
gem to that setup so you can run parallel instead of serial.
|
|
516
|
+
|
|
517
|
+
**`--worker-env NAME`** sets the environment variable `NAME` to a distinct value in each worker before it boots its
|
|
518
|
+
test environment, following the `parallel_tests` `TEST_ENV_NUMBER` convention:
|
|
519
|
+
|
|
520
|
+
| Worker | `TEST_ENV_NUMBER` | Database (example) |
|
|
521
|
+
|---|---|---|
|
|
522
|
+
| 0 | `""` (empty) | `myapp_test` |
|
|
523
|
+
| 1 | `"2"` | `myapp_test2` |
|
|
524
|
+
| 2 | `"3"` | `myapp_test3` |
|
|
525
|
+
|
|
526
|
+
You provision the databases; the gem only sets the variable. A single mutant run does not create or migrate anything.
|
|
527
|
+
|
|
528
|
+
**Worked example** (a `parallel_tests`-ready Rails app):
|
|
529
|
+
|
|
530
|
+
```bash
|
|
531
|
+
# 1. Provision one test database per worker (once, and after schema changes)
|
|
532
|
+
RAILS_ENV=test bundle exec rake parallel:prepare
|
|
533
|
+
|
|
534
|
+
# 2. Run mutation testing in parallel, one database per worker
|
|
535
|
+
bundle exec mutation_test app/models/user.rb spec/models/user_spec.rb \
|
|
536
|
+
-p 4 --worker-env TEST_ENV_NUMBER
|
|
537
|
+
|
|
538
|
+
# Batch over a whole directory the same way
|
|
539
|
+
bundle exec mutation_test --glob 'app/models/**/*.rb' --spec-glob 'spec/models/{name}_spec.rb' \
|
|
540
|
+
-p 4 --worker-env TEST_ENV_NUMBER
|
|
541
|
+
```
|
|
542
|
+
|
|
543
|
+
`MUTATION_TESTER_WORKER_ENV=TEST_ENV_NUMBER` is equivalent to passing the flag.
|
|
544
|
+
|
|
545
|
+
**Runner support.** `--worker-env` works with the `fork` and `spawn` runners, where each mutant boots its test
|
|
546
|
+
environment freshly and picks up the variable. The `in_memory` runner clones a single preloaded worker that has already
|
|
547
|
+
connected to one database, so it cannot isolate a per-worker database; when `--worker-env` is set the runner selection
|
|
548
|
+
skips `in_memory` and uses `fork`, announcing the reason on stderr.
|
|
549
|
+
|
|
550
|
+
**Still simplest without a parallel database setup:** if you have not provisioned per-worker databases, keep Rails
|
|
551
|
+
model runs on serial `-p 1`. `--worker-env` is only useful once the databases exist.
|
|
552
|
+
|
|
553
|
+
**Other strategies** if you are not using `parallel_tests`:
|
|
554
|
+
|
|
555
|
+
1. **In-Memory SQLite**: For unit tests that don't need advanced DB features, switch to SQLite in memory.
|
|
556
|
+
2. **Transactional Cleanup**: Ensure `DatabaseCleaner` or Rails transactional fixtures are working correctly across
|
|
557
|
+
processes (though this is often insufficient for parallel processes).
|
|
558
|
+
|
|
559
|
+
### Execution runners (fork, spawn, in-memory)
|
|
560
|
+
|
|
561
|
+
Every mutant is executed by one of three runners, and `auto` (the default) picks
|
|
562
|
+
the fastest safe one, announcing every fallback on stderr:
|
|
563
|
+
|
|
564
|
+
- **in_memory** (default where supported): re-evaluates the mutated source in the
|
|
565
|
+
memory of a fresh fork of a preloaded process, with zero file writes per mutant
|
|
566
|
+
and no shadow workspaces. RSpec only; the fastest path. Mutations that only take
|
|
567
|
+
effect at class-load time (constants consumed by macros, `validates`/`has_many`/
|
|
568
|
+
`before_save`/`scope`/`attribute`, anything inside an `included do` block) cannot
|
|
569
|
+
be observed by re-evaluating source in a preloaded process, so those mutants are
|
|
570
|
+
routed automatically to the file-based path and the rest still run in memory (see
|
|
571
|
+
below); the combined score matches a full `fork` run.
|
|
572
|
+
- **fork**: preloads the environment once (RubyGems, Bundler, `rspec-core`) and
|
|
573
|
+
forks a fresh child per mutant. RSpec on platforms with `Process.fork`; removes
|
|
574
|
+
most of the fixed per-mutant boot cost.
|
|
575
|
+
- **spawn**: starts one full process per mutant (`bundle exec rspec ...`). Slower
|
|
576
|
+
per mutant, but works everywhere (the only runner for Minitest and for
|
|
577
|
+
platforms without `Process.fork`).
|
|
578
|
+
|
|
579
|
+
`auto` tries `in_memory`, then `fork`, then `spawn`; every step down prints one
|
|
580
|
+
stderr warning with its reason, so a fallback is never silent. All runners
|
|
581
|
+
produce identical scores and per-mutant statuses and enforce the same hard
|
|
582
|
+
per-mutant timeout (monotonic deadline plus a process-group kill).
|
|
583
|
+
|
|
584
|
+
**Load-time mutants under in-memory (Rails).** The in-memory runner classifies each
|
|
585
|
+
mutation by its AST context. A mutation inside a method body defined directly in a
|
|
586
|
+
class/module is re-appliable in memory and runs there (fast). A mutation on a
|
|
587
|
+
class/module-body statement (a constant, a `validates`/`has_many`/`before_save`/
|
|
588
|
+
`scope`/`attribute` macro, or anything inside `included do ... end`) is decided
|
|
589
|
+
file-based within the same run, because re-evaluating the source does not re-run
|
|
590
|
+
those class-load registrations. This keeps in-memory speed for the common case
|
|
591
|
+
while matching a full `fork` score on Rails concerns and models. When at least one
|
|
592
|
+
mutant is routed this way, the run prints one stderr notice. It is automatic; you
|
|
593
|
+
do not need to pick `--runner fork` for correctness on load-time code.
|
|
594
|
+
|
|
595
|
+
| Mode | Picked by `auto` when | Falls back to |
|
|
596
|
+
|-------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------|
|
|
597
|
+
| `in_memory` | the suite is RSpec, the platform has `Process.fork`, the file has no load-time `defined?` guard, and re-applying the unmutated source in a probe child passes the suite | `fork`/`spawn` (whole run) with a stderr warning naming the reason; a single worker dying mid-run falls back only for its share of mutants; a mutant that raises while being applied falls back alone |
|
|
598
|
+
| `fork` | the suite is RSpec, `Process.fork` is available, but in-memory is unavailable (each reason is printed) | `spawn`, with a stderr warning, when the helper process fails to preload the environment |
|
|
599
|
+
| `spawn` | the suite is Minitest, or the platform has no `Process.fork` | nothing; it works everywhere |
|
|
600
|
+
|
|
601
|
+
Force a specific runner (skipping the auto attempts) with the `--runner
|
|
602
|
+
fork|spawn|in_memory` flag, the `MUTATION_TESTER_RUNNER` environment variable, or
|
|
603
|
+
`config.runner`:
|
|
604
|
+
|
|
605
|
+
```bash
|
|
606
|
+
mutation_test --runner spawn lib/calculator.rb spec/calculator_spec.rb
|
|
607
|
+
```
|
|
608
|
+
|
|
609
|
+
Reach for `--runner spawn` when you want maximum isolation or are debugging a
|
|
610
|
+
suspicious result from a preloaded runner, and `--runner fork` when your source
|
|
611
|
+
is not cleanly re-evaluable in memory but you still want the preloaded-environment
|
|
612
|
+
speed. For the full per-runner mechanics, when to force each one, and the complete
|
|
613
|
+
fork and in-memory limitation lists (frozen classes, load-time `defined?` guards,
|
|
614
|
+
worker-death fallback, `require_relative` idempotency), see
|
|
615
|
+
[docs/execution-runners.md](docs/execution-runners.md#execution-runners-fork-spawn-in-memory).
|
|
616
|
+
|
|
617
|
+
### Test selection (fast kill with full-file confirmation)
|
|
618
|
+
|
|
619
|
+
For RSpec suites on the file-based runners, MutationTester runs each mutant in
|
|
620
|
+
two phases: it first runs only the examples whose group matches the mutated
|
|
621
|
+
method (`rspec spec_file -e '#foo' -e '.foo'`) to kill it fast, then confirms a
|
|
622
|
+
passing subset against the full spec file before a mutant can be reported as
|
|
623
|
+
survived, so selection never introduces false survivors. It degrades to the full
|
|
624
|
+
file when the mutant is not inside a method, the spec has no matching group, or
|
|
625
|
+
the suite is Minitest; the in-memory runner skips selection entirely (its
|
|
626
|
+
examples are already loaded). Disable it with `--no-test-selection` or
|
|
627
|
+
`config.test_selection = false`. See
|
|
628
|
+
[docs/execution-runners.md](docs/execution-runners.md#test-selection-fast-kill-with-full-file-confirmation)
|
|
629
|
+
for the full behavior.
|
|
630
|
+
|
|
631
|
+
## Mutation types
|
|
632
|
+
|
|
633
|
+
MutationTester generates several families of mutations, enabled by default:
|
|
634
|
+
arithmetic, bitwise compound-assignment, comparison, logical, boolean, number,
|
|
635
|
+
string, conditional, call-removal, nil-injection and argument mutations. A
|
|
636
|
+
strict-equality family is opt-in. Enable or disable individual families through
|
|
637
|
+
`config.mutation_types` (see [Configuration](#configuration)) or turn
|
|
638
|
+
strict-equality on with `--strict-equality`.
|
|
639
|
+
|
|
640
|
+
See [docs/mutation-types.md](docs/mutation-types.md) for the full catalog: every
|
|
641
|
+
operator swap and structural mutation each family generates, its reported `type`,
|
|
642
|
+
and the constructs each family intentionally leaves alone.
|
|
643
|
+
|
|
644
|
+
## Equivalent mutants
|
|
645
|
+
|
|
646
|
+
A mutation score of 100% is not always achievable, and a surviving mutation is
|
|
647
|
+
not always a gap in your tests. Some mutations produce code that behaves
|
|
648
|
+
**identically** to the original for every possible input, an *equivalent
|
|
649
|
+
mutant*, and no test can ever kill it. For example, in a `max` implementation the
|
|
650
|
+
original `a > b ? a : b` and the mutant `a >= b ? a : b` differ only when
|
|
651
|
+
`a == b`, and both return the same value there, so the mutant survives no matter
|
|
652
|
+
how thorough your tests are. Because equivalence is undecidable in the general
|
|
653
|
+
case, treat survivors as *candidates* to review rather than guaranteed test gaps;
|
|
654
|
+
once you confirm a survivor is equivalent, it is reasonable to accept a score
|
|
655
|
+
below 100%.
|
|
656
|
+
|
|
657
|
+
### Excluding a line with `# mutation_tester:disable`
|
|
658
|
+
|
|
659
|
+
Once you have confirmed that a survivor is equivalent, annotate its line with a
|
|
660
|
+
trailing `# mutation_tester:disable` comment (the same style as
|
|
661
|
+
`# rubocop:disable`) so the mutator skips every mutation on that line and the
|
|
662
|
+
excluded line drops out of the score and the report:
|
|
663
|
+
|
|
664
|
+
```ruby
|
|
665
|
+
def max(a, b)
|
|
666
|
+
a > b ? a : b # mutation_tester:disable
|
|
667
|
+
end
|
|
668
|
+
```
|
|
669
|
+
|
|
670
|
+
The marker is honoured only inside a real comment (never inside a string
|
|
671
|
+
literal), and only on the line it sits on. See
|
|
672
|
+
[docs/mutation-types.md](docs/mutation-types.md#excluding-a-line-with--mutation_testerdisable)
|
|
673
|
+
for the console output and the current limits (no block ranges or per-type
|
|
674
|
+
exclusion yet).
|
|
675
|
+
|
|
676
|
+
## Reports and output
|
|
677
|
+
|
|
678
|
+
### Console report
|
|
679
|
+
|
|
680
|
+
Surviving mutants are grouped by file and line (one header per location, all
|
|
681
|
+
mutation variants listed under it) and each group shows a unified diff with a
|
|
682
|
+
few lines of surrounding context:
|
|
683
|
+
|
|
684
|
+
```
|
|
685
|
+
🧬 MUTATION TESTING REPORT
|
|
686
|
+
================================================================================
|
|
687
|
+
📊 Summary:
|
|
688
|
+
Total Mutations: 20
|
|
689
|
+
Killed: 18 ✅
|
|
690
|
+
Survived: 2 ❌
|
|
691
|
+
Mutation Score: 90.0%
|
|
692
|
+
Quality: Excellent 🌟
|
|
693
|
+
|
|
694
|
+
⚠️ Survived Mutations (Need Improvement):
|
|
695
|
+
--------------------------------------------------------------------------------
|
|
696
|
+
|
|
697
|
+
Location: lib/calculator.rb:12 (2 variants)
|
|
698
|
+
#4 [arithmetic] Change + to -
|
|
699
|
+
#5 [arithmetic] Change + to *
|
|
700
|
+
|
|
701
|
+
@@ -10,5 +10,6 @@
|
|
702
|
+
def add(a, b)
|
|
703
|
+
- a + b
|
|
704
|
+
+ a - b (#4)
|
|
705
|
+
+ a * b (#5)
|
|
706
|
+
end
|
|
707
|
+
💡 Suggestion: Add tests to verify behavior for each of the 2 variants above
|
|
708
|
+
```
|
|
709
|
+
|
|
710
|
+
### HTML report
|
|
711
|
+
|
|
712
|
+
Beautiful interactive HTML report with:
|
|
713
|
+
|
|
714
|
+
- Summary statistics
|
|
715
|
+
- Mutation score visualization
|
|
716
|
+
- Filterable mutation list
|
|
717
|
+
- Survivors grouped by file and line, with all mutation variants under one card
|
|
718
|
+
- Unified diffs with surrounding context for survived and timeout mutants
|
|
719
|
+
- Detailed suggestions for improvements
|
|
720
|
+
|
|
721
|
+
### JSON report and machine-readable output
|
|
722
|
+
|
|
723
|
+
Machine-readable report for CI/CD integration and AI agents. Run with `--json`
|
|
724
|
+
to get a single, clean JSON document on **stdout** and nothing else: the banner,
|
|
725
|
+
progress spinner, colours and the "report saved" notice all go to **stderr**, so
|
|
726
|
+
the stream is safe to pipe straight into `jq` or a parser. The process still
|
|
727
|
+
exits `0` when the mutation score meets the configured threshold and `1` when it
|
|
728
|
+
does not, so the exit code remains a pass/fail signal.
|
|
729
|
+
|
|
730
|
+
```bash
|
|
731
|
+
bundle exec mutation_test --json examples/calculator.rb examples/calculator_spec.rb | jq .
|
|
732
|
+
|
|
733
|
+
# A single listed file works too; its spec is mapped by convention
|
|
734
|
+
bundle exec mutation_test --json lib/calculator.rb | jq .
|
|
735
|
+
```
|
|
736
|
+
|
|
737
|
+
The same report is also written to `tmp/mutation_reports/mutation_report.json`
|
|
738
|
+
(see `--output-dir`). A run that resolves exactly one source file, or an explicit
|
|
739
|
+
`SOURCE_FILE TEST_FILE` pair, prints the plain per-file report; a multi-file run
|
|
740
|
+
(a `FILE` list with more than one file, `--staged`, or `--glob`) prints one
|
|
741
|
+
aggregate envelope with a condensed `survivors` array.
|
|
742
|
+
|
|
743
|
+
Every **surviving** mutant carries `file_path`, `line`, `original` and `mutated`,
|
|
744
|
+
which is a concrete, located test gap: the worklist you can hand to an AI agent
|
|
745
|
+
or a CI gate (see [CI/CD integration](#cicd-integration)).
|
|
746
|
+
|
|
747
|
+
Full field-by-field documentation of both shapes (the single-file report and the
|
|
748
|
+
multi-file envelope), the `schema_version` policy, and ready-to-use `jq` recipes
|
|
749
|
+
live in [docs/json-schema.md](docs/json-schema.md).
|
|
750
|
+
|
|
751
|
+
## Pre-push hook
|
|
752
|
+
|
|
753
|
+
Gate your pushes locally: run mutation testing on the file(s) you touched and
|
|
754
|
+
block the push when the score is under your bar, the fast-feedback sibling of
|
|
755
|
+
the CI gate below (see [CI/CD integration](#cicd-integration)).
|
|
756
|
+
|
|
757
|
+
The gem ships a ready-to-copy hook at
|
|
758
|
+
[`examples/hooks/pre-push`](examples/hooks/pre-push) (installed with the gem, so
|
|
759
|
+
you have it offline). It runs `mutation_test --json`, reads the score with
|
|
760
|
+
[`jq`](https://jqlang.github.io/jq/), and on a below-threshold run lists the
|
|
761
|
+
surviving mutants (file, line, what changed) so you see which gaps to close.
|
|
762
|
+
Install it with:
|
|
763
|
+
|
|
764
|
+
```bash
|
|
765
|
+
cp examples/hooks/pre-push .git/hooks/pre-push
|
|
766
|
+
chmod +x .git/hooks/pre-push
|
|
767
|
+
```
|
|
768
|
+
|
|
769
|
+
Then edit the `THRESHOLD` and the `SOURCE TEST` pair(s) at the top of the copied
|
|
770
|
+
hook.
|
|
771
|
+
|
|
772
|
+
Prefer to gate on your project's configured threshold rather than one written
|
|
773
|
+
into the hook? `mutation_test` already exits non-zero when the score is below
|
|
774
|
+
`config.minimum_score` (default 80, see [Configuration](#configuration)), so you
|
|
775
|
+
can drop the `jq` comparison and let the exit code be the gate:
|
|
776
|
+
|
|
777
|
+
```sh
|
|
778
|
+
bundle exec mutation_test app/models/user.rb spec/models/user_spec.rb || exit 1
|
|
779
|
+
```
|
|
780
|
+
|
|
781
|
+
lefthook or overcommit users: call the shipped hook from your `pre-push` step
|
|
782
|
+
instead of writing to `.git/hooks/`.
|
|
783
|
+
|
|
784
|
+
## CI/CD integration
|
|
785
|
+
|
|
786
|
+
Run MutationTester as a CI quality gate: the CLI exits non-zero when the mutation
|
|
787
|
+
score is below the threshold, so it fails the job with no extra configuration.
|
|
788
|
+
|
|
789
|
+
The gem ships ready-to-copy GitHub Actions workflows (installed alongside the
|
|
790
|
+
gem, so you have them offline too):
|
|
791
|
+
|
|
792
|
+
- [`examples/github_actions/mutation_test.yml`](examples/github_actions/mutation_test.yml)
|
|
793
|
+
is the maintained template. Add the gem to your bundle, copy it to
|
|
794
|
+
`.github/workflows/`, edit the `EDIT:` lines, and it runs the gate, uploads the
|
|
795
|
+
HTML/JSON reports from `tmp/mutation_reports/` as an artifact (even on
|
|
796
|
+
failure), and fails the job below threshold. It also carries commented variants
|
|
797
|
+
for parallel execution, several file pairs, and an incremental pull-request gate.
|
|
798
|
+
- [`examples/github_actions/ai_mutation_gate.yml`](examples/github_actions/ai_mutation_gate.yml)
|
|
799
|
+
is the AI gate: the same pass/fail gate, plus it writes the surviving-mutant
|
|
800
|
+
worklist to the GitHub job summary and uploads `survivors.json` for an agent to
|
|
801
|
+
turn into missing tests.
|
|
802
|
+
|
|
803
|
+
See [docs/ci.md](docs/ci.md) for the full recipes: a 5-minute setup, minimal
|
|
804
|
+
inline and pull-request workflows, machine mode as a gate and artifact, and the
|
|
805
|
+
AI workflow.
|
|
806
|
+
|
|
807
|
+
## Troubleshooting
|
|
808
|
+
|
|
809
|
+
### Tests Pass Normally but Fail During Mutation Testing
|
|
810
|
+
|
|
811
|
+
This usually means:
|
|
812
|
+
|
|
813
|
+
- Your tests depend on execution order
|
|
814
|
+
- Your tests share state between runs
|
|
815
|
+
- Database transactions aren't being properly cleaned up
|
|
816
|
+
|
|
817
|
+
**Solution**: Ensure each test is independent and can run in isolation.
|
|
818
|
+
|
|
819
|
+
### Database Conflicts in Parallel Mode
|
|
820
|
+
|
|
821
|
+
If you see errors like "database is locked" or "record not found":
|
|
822
|
+
|
|
823
|
+
**Solution**: Use serial execution (the default):
|
|
824
|
+
|
|
825
|
+
```bash
|
|
826
|
+
bundle exec mutation_test app/models/user.rb spec/models/user_spec.rb
|
|
827
|
+
```
|
|
828
|
+
|
|
829
|
+
Or if using environment variable, make sure it's not set or set to 1:
|
|
830
|
+
|
|
831
|
+
```bash
|
|
832
|
+
MUTATION_TESTER_PARALLEL_PROCESSES=1 bundle exec mutation_test app/models/user.rb spec/models/user_spec.rb
|
|
833
|
+
```
|
|
834
|
+
|
|
835
|
+
### Slow Execution
|
|
836
|
+
|
|
837
|
+
Mutation testing generates many mutations and runs your tests repeatedly, so it's naturally slower than a regular test
|
|
838
|
+
run.
|
|
839
|
+
|
|
840
|
+
**Tips for faster execution**:
|
|
841
|
+
|
|
842
|
+
- Test only critical files (don't test everything)
|
|
843
|
+
- Use parallel execution if your tests support it (`--parallel N`)
|
|
844
|
+
- Consider using faster test databases (SQLite in-memory for unit tests)
|
|
845
|
+
- Focus on high-value code (models, services, core logic)
|
|
846
|
+
|
|
847
|
+
### "No Mutations Generated"
|
|
848
|
+
|
|
849
|
+
If you see "Generated 0 mutations", this usually means:
|
|
850
|
+
|
|
851
|
+
- The file only contains method signatures or delegations
|
|
852
|
+
- The file has no logic to mutate (e.g., only `belongs_to` associations)
|
|
853
|
+
- The file might be empty or only contain constants
|
|
854
|
+
|
|
855
|
+
**Solution**: Choose files with actual logic to test (calculations, conditionals, validations, etc.)
|
|
856
|
+
|
|
857
|
+
### Debugger Statement Hit
|
|
858
|
+
|
|
859
|
+
If mutation testing stops at a `debugger` or `binding.pry` statement:
|
|
860
|
+
|
|
861
|
+
- This is expected in development mode
|
|
862
|
+
- The mutation is testing code that hits a debugger
|
|
863
|
+
|
|
864
|
+
**Solution**: Continue (`c`), quit (`q`), or remove debuggers from your code before running mutation tests
|
|
865
|
+
|
|
866
|
+
### Parser Version Warning on Newer Ruby (Supported Syntax Level)
|
|
867
|
+
|
|
868
|
+
MutationTester parses your source with the `parser` gem. The newest published
|
|
869
|
+
`parser` line recognizes **Ruby 3.3 syntax**; there is not yet a release that
|
|
870
|
+
understands Ruby 3.4+/4.x syntax. So when you run on Ruby 3.4 or newer you may
|
|
871
|
+
see one line on stderr per run:
|
|
872
|
+
|
|
873
|
+
```
|
|
874
|
+
warning: parser/current is loading parser/ruby33, which recognizes 3.3.x-compliant syntax, but you are running 4.0.2.
|
|
875
|
+
```
|
|
876
|
+
|
|
877
|
+
This warning is **benign**. It only means the parser recognizes syntax up to
|
|
878
|
+
Ruby 3.3. The gem itself runs fine on Ruby 3.4+/4.x, and files written in Ruby
|
|
879
|
+
3.3-and-earlier syntax are mutated normally.
|
|
880
|
+
|
|
881
|
+
The only real limitation is a source file that relies on syntax introduced
|
|
882
|
+
after Ruby 3.3. Such a file cannot be parsed, so it reports an explicit
|
|
883
|
+
`Failed to parse source file` and the run fails - it never fakes a passing
|
|
884
|
+
score. The warning is left in place on purpose as an honest signal; it is not
|
|
885
|
+
globally silenced.
|
|
886
|
+
|
|
887
|
+
## Development
|
|
888
|
+
|
|
889
|
+
Run `bundle install`, then run `rake spec` to run the tests.
|
|
890
|
+
|
|
891
|
+
The gem includes working examples for both RSpec and Minitest under the
|
|
892
|
+
`examples/` directory (a sample `Calculator` class with an RSpec spec and a
|
|
893
|
+
Minitest test). Run them directly with the CLI, or through the example rake tasks:
|
|
894
|
+
|
|
895
|
+
```bash
|
|
896
|
+
# Directly with the CLI
|
|
897
|
+
bundle exec mutation_test examples/calculator.rb examples/calculator_spec.rb
|
|
898
|
+
bundle exec mutation_test examples/calculator.rb examples/calculator_minitest.rb
|
|
899
|
+
|
|
900
|
+
# RSpec examples
|
|
901
|
+
rake example:rspec # Serial execution
|
|
902
|
+
rake example:rspec_parallel # Parallel execution
|
|
903
|
+
|
|
904
|
+
# Minitest examples
|
|
905
|
+
rake example:minitest # Serial execution
|
|
906
|
+
rake example:minitest_parallel # Parallel execution
|
|
907
|
+
|
|
908
|
+
# Default example (RSpec)
|
|
909
|
+
rake example
|
|
910
|
+
```
|
|
911
|
+
|
|
912
|
+
Each example will:
|
|
913
|
+
|
|
914
|
+
- Run mutation tests on a sample Calculator class
|
|
915
|
+
- Generate console, HTML, and JSON reports
|
|
916
|
+
- Show mutation score and quality metrics
|
|
917
|
+
- Demonstrate the difference between serial and parallel execution
|
|
918
|
+
|
|
919
|
+
## Contributing
|
|
920
|
+
|
|
921
|
+
Bug reports and pull requests are welcome on GitHub at https://github.com/Oxyconit/mutation_tester.
|
|
922
|
+
|
|
923
|
+
## License
|
|
924
|
+
|
|
925
|
+
The gem is available as open source under the terms of the [MIT License](LICENSE.txt).
|