mutineer 1.4.0 → 1.5.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (43) hide show
  1. checksums.yaml +4 -4
  2. data/CHANGELOG.md +250 -0
  3. data/README.md +198 -27
  4. data/lib/mutineer/changed_lines.rb +24 -1
  5. data/lib/mutineer/cli.rb +112 -16
  6. data/lib/mutineer/config.rb +64 -13
  7. data/lib/mutineer/coverage_map.rb +161 -14
  8. data/lib/mutineer/daemon_backend.rb +3 -0
  9. data/lib/mutineer/daemon_client.rb +170 -13
  10. data/lib/mutineer/daemon_server.rb +2 -1
  11. data/lib/mutineer/external_backend.rb +2 -2
  12. data/lib/mutineer/isolation.rb +91 -6
  13. data/lib/mutineer/kill_channel.rb +284 -0
  14. data/lib/mutineer/kill_matrix.rb +88 -0
  15. data/lib/mutineer/minitest_integration/kill_recorder.rb +301 -0
  16. data/lib/mutineer/minitest_integration/outer_reporter.rb +126 -0
  17. data/lib/mutineer/minitest_integration/stop_at_first_failure.rb +14 -68
  18. data/lib/mutineer/minitest_integration.rb +27 -10
  19. data/lib/mutineer/mutant_id.rb +5 -4
  20. data/lib/mutineer/mutators/array_literal.rb +0 -7
  21. data/lib/mutineer/mutators/base.rb +63 -4
  22. data/lib/mutineer/mutators/chain_link.rb +0 -7
  23. data/lib/mutineer/mutators/collection_method.rb +1 -1
  24. data/lib/mutineer/mutators/condition_forcing.rb +0 -7
  25. data/lib/mutineer/mutators/operand_removal.rb +0 -7
  26. data/lib/mutineer/mutators/operator_assignment.rb +0 -7
  27. data/lib/mutineer/mutators/return_nil.rb +5 -11
  28. data/lib/mutineer/mutators/string_literal.rb +1 -1
  29. data/lib/mutineer/pairing.rb +222 -15
  30. data/lib/mutineer/parser.rb +12 -3
  31. data/lib/mutineer/project.rb +7 -0
  32. data/lib/mutineer/rails_worker_db.rb +1 -1
  33. data/lib/mutineer/reporter.rb +288 -52
  34. data/lib/mutineer/result.rb +37 -1
  35. data/lib/mutineer/runner.rb +133 -16
  36. data/lib/mutineer/statement_lines.rb +143 -0
  37. data/lib/mutineer/test_runners/minitest.rb +4 -2
  38. data/lib/mutineer/test_runners/rspec.rb +243 -3
  39. data/lib/mutineer/test_runners.rb +5 -3
  40. data/lib/mutineer/version.rb +1 -1
  41. data/lib/mutineer/worker_pool.rb +4 -2
  42. data/lib/mutineer.rb +1 -0
  43. metadata +8 -3
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: e18649829a9226ea7a5f654ef38602eabff4005ebf8a516fca4402460a7aac09
4
- data.tar.gz: 9e3c166e5e2e513278aaf210f271ed0f070cd335d0de588318cb8aa71f6b6a06
3
+ metadata.gz: 50113a063a85737ad99476f85e78cb2ac015f267018d19ba6350ba36ff4ac6df
4
+ data.tar.gz: c12343a00d2df6d495859644c326bc650e33d013ddf009f2648c06a199aceb86
5
5
  SHA512:
6
- metadata.gz: 2e10e1214d0b55a4c93c7fa13d4c3b39503d4e2b1e367a995e2d813e98bd55ce2c2774b429bd690f2ae55019a39c7ee539e3ef60ea51bf25e2bc61c602ab72b6
7
- data.tar.gz: 146768f9f599d41af9e6549f65ad7658cdd55e3d57a6f35fa6d094cfef6be73b036bb3dc8ebe8616d03cf358cdaf78e5d8a3b40a996c7a4d18a06898b9454824
6
+ metadata.gz: a170b15c6883a4dcec6a2798c3b102a5f9c9ded1f46c166df98d0e5bdf5b891ca5b24892efe06245680b45a94da38638adc36ca9a48a48d2ac1c07b29126f173
7
+ data.tar.gz: a30272ff6afb48dd666d6945c6532d987ef335e0e78dc422420cf26cdbec639ee1c850118a10ba25e10a426a2644b3926e39698126eaad0646707a4f2be2e719
data/CHANGELOG.md CHANGED
@@ -6,6 +6,255 @@ All notable changes to this project are documented here. The format is based on
6
6
 
7
7
  ## [Unreleased]
8
8
 
9
+ ## [1.5.0] - 2026-10-06
10
+
11
+ ### Added
12
+ - **`--timeout SECONDS` and `--capture-timeout SECONDS`**, also `timeout:` and
13
+ `capture_timeout:` in `.mutineer.yml`. They set the in-process per-mutant
14
+ time limit (default 10s) and the coverage-capture time limit (default 120s),
15
+ which were fixed. On a large Rails suite the clean run of the unmutated
16
+ tests took longer than 120s, so every run that offered the whole suite
17
+ stopped as not green, and a mutant on a line many tests cover took longer
18
+ than 10s and was scored `timeout`. `--daemon` keeps its own per-mutant
19
+ limit but uses `--capture-timeout` for its coverage capture;
20
+ `--test-command` uses neither. Either backend warns about a limit it was
21
+ given and does not use.
22
+ - **`--cache-dir DIR`**, also `cache_dir:` in `.mutineer.yml`, sets where the
23
+ coverage cache is written (default `.mutineer`, as before). Two runs from the
24
+ same project root shared `.mutineer/coverage.json` and could overwrite each
25
+ other's map; each can now have its own directory. `--daemon` uses it too.
26
+ A blank value (an unset CI variable) exits 2 rather than writing the cache
27
+ into the project root.
28
+ - **The README states how timeouts affect the score** (#62): a timed-out
29
+ mutant is left out of the score, and counts as a mutant with no verdict
30
+ under `--threshold`.
31
+ - **`--matrix` reports which tests kill each mutant** (`matrix: true` in
32
+ `.mutineer.yml`). Each mutant runs every test in its covering files instead
33
+ of stopping at the first failure, and the child sends each test's outcome to
34
+ the parent as it is recorded. The report names blind tests, which ran in a
35
+ complete row and killed no mutant, and redundant tests, whose every kill
36
+ another test also makes. Redundancy is judged one test at a time, so delete
37
+ redundant tests one at a time. The human report lists up to 20 of each, the
38
+ HTML report all of them, and the JSON report moves to schema `1.5` with a
39
+ `matrix` block that appears only with the flag. A test is its file and its
40
+ id (the example id under RSpec), so examples that share a description stay
41
+ apart and an example whose generated description changes with the mutant
42
+ stays one test. Verdicts, the score and the exit code do not change: each
43
+ mutant gets the verdict a run without the flag gives. After a serial test
44
+ fails, an exit, a crash or the timeout in a later test, class or group
45
+ leaves it `killed`; an end in the code that run still reaches (the failing
46
+ test's class wrapper or group hooks, the suite's own cleanup) and a failure
47
+ in a `parallelize_me!` class keep the exit status. A row is complete only
48
+ when the child's stream arrived in order and every test reported (under
49
+ Minitest the recorder saw every test, under RSpec every planned example),
50
+ so a caught `Interrupt` leaves it incomplete. The human report names up
51
+ to 20 incomplete rows and the HTML and JSON reports all of them; the human
52
+ and HTML reports warn not to delete a blind test until its rows are
53
+ complete.
54
+ Minitest and RSpec 3.3+ are supported, on the in-process backend only;
55
+ `--daemon`, `--test-command`, `--fail-fast` and `--dry-run` exit 2 with it.
56
+ `--no-matrix` and `--no-fail-fast` beat the matching `.mutineer.yml` key,
57
+ so a file that sets one can still run with a flag that conflicts with it.
58
+ - **`no_coverage[]` and `uncapturable[]` name each mutant.** Every entry in
59
+ the JSON report now has `operator`, `token` and `id`, as `ignored[]` has, so
60
+ mutants on one line can be told apart and an `id` can go in `ignore:`. The
61
+ entries are sorted by `(file, line, operator, id)`. `schema_version` is `1.6`.
62
+
63
+ - **The site and README explain "clean-room" and compare Mutineer with
64
+ Mutant**, and the landing page hero has a one-minute explainer video. It
65
+ plays at 55% volume while in view (not with reduced motion; muted until
66
+ the first click or key where the browser requires it), pauses when it scrolls
67
+ away or the tab is hidden, and has an Enlarge button.
68
+
69
+ ### Changed
70
+ - **The human report has a `Timeout:` row**, and the score line lists timeouts
71
+ apart from errored mutants. Before, the `Errored:` row and the score line
72
+ added the two together, while the JSON report kept them apart. The HTML
73
+ summary now shows a `timeout` count apart from `errored` too.
74
+ - **A test file given as a source after `--test` exits 2.** `--test` takes one
75
+ file, so in `mutineer run app/x.rb --test spec/a_spec.rb spec/b_spec.rb` the
76
+ second spec became a source to mutate, and the run tested with one spec file
77
+ without a word. Now a source named `*_test.rb` or `*_spec.rb` inside a
78
+ `test/` or `spec/` directory, given with `--test`, is a usage error that says
79
+ to repeat `--test`. The usage line now reads `--test <test> [--test <test>...]`.
80
+
81
+ ### Fixed
82
+ - **The RSpec runner keeps what gems configured before the run.** It called
83
+ `RSpec.reset`, which drops the whole configuration. Under `rails: true`, a
84
+ `--boot` file or `require:`, the app's gems load first, so a setting such as
85
+ rspec-retry's `verbose_retry` was gone, and a support file that set it
86
+ raised `NoMethodError`. The run stopped with "the unmutated suite is not
87
+ green". The runner now calls `RSpec.clear_examples`, which keeps the
88
+ configuration. Hooks that gems added now apply too: an example with
89
+ rspec-retry's `retry: 3` runs up to 3 times per mutant, as under `rspec`.
90
+ - **`--since` no longer scores a file that Git ignores** (#168). A file
91
+ outside the index is scored in full, so a new file is not missed. That
92
+ check also caught files that Git ignores on purpose (`.gitignore`,
93
+ `.git/info/exclude` or the global excludes file), such as generated code
94
+ inside a directory run. Those files now get no mutants under `--since`,
95
+ and a warning names each one. Other untracked files are still scored in
96
+ full, and so is a file when Git cannot tell, such as one beyond a symlink or
97
+ inside a submodule.
98
+ - **The human report escapes control characters from the source** (#163).
99
+ A surviving line that held a terminal control byte (for example ESC) was
100
+ printed as that byte, so it could change what the terminal showed. The
101
+ token, the replacement, the diff lines, method names, file paths and the
102
+ kill matrix's test names now print each control character except tab as
103
+ its Ruby escape (`\e`), and a byte that is not valid UTF-8 (a Latin-1
104
+ source) as `\xNN`. The JSON and HTML
105
+ reports already escaped text and do not change.
106
+ - **One edit on a line is scored once** (#159). Some opt-in operators emitted
107
+ two mutants that give the same source: `literal_mutation` on `0` (the
108
+ "change to 1" and "add 1" rules), `negation_removal` on `!!x` (either `!`
109
+ removed), and `chain_link` or `operand_removal` on a repeated part such as
110
+ `a.b.b.c` or `x || x`. Each copy had its own id, so one surviving edit
111
+ counted twice. Now only the first copy on a line runs. Copies that start
112
+ on different lines (a multi-line chain) both stay, so `--since` never
113
+ loses the edit. Ids are assigned before the copy is dropped, so every kept
114
+ mutant keeps its id, and the dropped copy's id simply stops appearing.
115
+ Default runs do not change. If you use these operators with `--baseline`,
116
+ regenerate the baseline after upgrading: the score can move with no code
117
+ change, and in a full (unscoped) comparison a dropped copy that survived
118
+ is listed as fixed. An ignore entry for one copy's id still ignores only
119
+ that copy; the other copy still runs.
120
+ - **`# mutineer:disable-line` inside a string no longer silences a line**
121
+ (#158). The marker was found with a text search, so a string such as
122
+ `"# mutineer:disable-line"` ignored every mutant on its line and left
123
+ them out of the score. Only a real `#` comment counts now. A heredoc or
124
+ regex that holds the text is ignored too.
125
+ - **`./lib/x.rb` and an absolute path pair with a test** (#104). Auto-pairing
126
+ found the test for `lib/calc.rb` but not for `./lib/calc.rb` or the
127
+ absolute path to the same file. It reported "no test found by convention"
128
+ and exited 2. A file argument inside the project is now made relative to
129
+ the project root first, so equivalent spellings (`./`, `..`, absolute)
130
+ pair with the same test and run once. A symlink keeps its own name, so it
131
+ stays a separate source from its target. Reports show that root-relative
132
+ path. A path outside the project stays as typed.
133
+ - **A mutant on another line of a multi-line statement runs its tests**
134
+ instead of being reported as `no_coverage`. Ruby counts one line of a
135
+ statement only. When the mutant's own line has no count, the runner now uses
136
+ the tests that ran the statement that holds the mutation. A heredoc body
137
+ belongs to its statement. This fallback skips a first or last line that also
138
+ holds other code, such as the `def` line, and the body of `x while c` and
139
+ `x until c`. It also gives no tests to a mutant in code of the statement
140
+ that runs only sometimes: a `when` or `in` condition, a `rescue` class
141
+ list, a parameter default, the rescue side of `x rescue y`, the value of
142
+ `||=` or `&&=`, the operand of `defined?`, the right side of `a ||` or
143
+ `a &&`, a branch of `x if c`, `x unless c`, a ternary or `else`, the
144
+ pattern of `v in p` or `v => p`, an
145
+ argument or block of `x&.m(...)`, and the value of `x&.m += v`. Those
146
+ mutants stay `no_coverage`, as before.
147
+ - **A survivor diff for a multi-line mutant applies with `git apply`** (#106).
148
+ The JSON `diff` header always read `@@ -N +N @@`, even when the mutant
149
+ removed two lines and added one, so `git apply` rejected the patch as
150
+ corrupt. The header now carries each side's line count (`@@ -3,2 +3 @@`).
151
+ A one-line diff keeps its old header. A mutant that empties a line now
152
+ shows that empty line as a `+` line. A side whose last line has no final
153
+ newline gets ``. Lines of a CRLF file now keep
154
+ their `\r`, so the patch applies to that file.
155
+ - **Split test files pair with their source** (#87). A source such as
156
+ `app/foo/bar.rb` now also uses `test/foo/bar_upsert_test.rb` and
157
+ `test/foo/bar_guards_test.rb`, together with `test/foo/bar_test.rb` when
158
+ that file exists. The same names under `test/lib/` pair with a `lib/`
159
+ source. Before, only the exact `bar_test.rb` name was tried, so a suite
160
+ split across `bar_*_test.rb` files was skipped and a gate could pass with
161
+ no mutants run. A spec the old rules already find is unchanged. The
162
+ split name is left for `user_session.rb` when that file exists, so
163
+ `user.rb` does not run `user_session_test.rb`. The same rule holds outside
164
+ `app/` and `lib/`: `src/user.rb` does not take `test/src/user_session_test.rb`
165
+ when `src/user_session.rb` exists. A longer source claims a split file
166
+ only when that source itself searches the directory the file is in.
167
+ A failed capture of `test/foo/bar_upsert_test.rb` can mark `app/foo/bar.rb`
168
+ uncapturable only when no successful capture covered that source and
169
+ `bar_upsert.rb` does not exist. A test outside that directory does not,
170
+ and neither does a source that only shares the test basename.
171
+ `user_session__test.rb` stays with `user.rb` when `user_session.rb`
172
+ exists, because that name is not the exact test or a split test for
173
+ `user_session`. A failed exact spec in the mirrored `spec/` or `spec/lib/` directory
174
+ marks an `app/` or `lib/` source the same way. A failed spec in another
175
+ directory does not. A project rooted at `/` still finds split tests
176
+ inside that root. A stored failure keeps the pairing from the moment it
177
+ was checked. Adding or deleting a longer source rebuilds the coverage
178
+ map, so that failure is not moved onto the shorter source.
179
+ - **A blank `operators` key no longer runs zero mutants and exits 0** (#167).
180
+ `operators:` with no value became an empty list. The run made no mutants
181
+ and the gate passed. A blank operator list is now an error, and so is a
182
+ list of only unknown operator names. An empty `require` or `ignore` list
183
+ is still valid. `--operators` on the command line still replaces the
184
+ file list, including a blank or unknown file list.
185
+ - **The daemon no longer runs the tool's Ruby settings inside the app** (#100).
186
+ Open3 keeps a variable when the spawn hash omits it, so a parent `RUBYOPT`
187
+ ran inside the app. The child now gets the environment Bundler saved before
188
+ it activated, passed with `unsetenv_others`. The app keeps its gem home,
189
+ excluded groups, Bundler config, and chruby path. An `RBENV_VERSION` or
190
+ `ASDF_RUBY_VERSION` already in the environment stays. That pin is often
191
+ the Ruby that started the tool. `.ruby-version` applies only when no pin
192
+ is set. The tool's `RUBYOPT` and `RUBYLIB` do not reach the child, including
193
+ a value Bundler had saved. A gem home, `BUNDLE_PATH`, or `BUNDLE_WITHOUT`
194
+ reaches the child only when Bundler saved the app's value. The same
195
+ rule covers `GEM_PATH` and `BUNDLER_VERSION`. An rbenv version
196
+ bin under `RBENV_ROOT`, or under `~/.rbenv` when that variable is unset,
197
+ leaves `PATH` only when that root has a shim directory. An asdf version
198
+ bin leaves only when an asdf shim directory exists, including
199
+ `ASDF_DATA_DIR/shims`. Without that shim, the bin stays and `bundle` can
200
+ still be found. The app Gemfile, an explicit Ruby pin, and Rails env are
201
+ applied after that.
202
+ - **The Action refuses to run when `output` and `baseline` are the same file** (#160).
203
+ That setup used to work, and a failing run copied its report onto the
204
+ baseline. The next run then treated the failed report as the baseline, so
205
+ the gate could pass. The Action now stops before the run and exits 2.
206
+ The baseline may come from the `baseline` input, from `--baseline` in
207
+ `extra-args`, or from `.mutineer.yml`. A numeric path such as `123` is
208
+ read the same way as the CLI. Write the report to a separate path. Copy
209
+ that file onto the baseline in a later step, and only when this step
210
+ exits 0. You can also refresh the baseline with the CLI, and keep the
211
+ file only when the run exits 0. The next release moves the `v1` tag, so
212
+ workflows on `@v1` get this refusal then.
213
+ - **A mutation no longer leaves a heredoc body behind as Ruby** (#134).
214
+ Replacing a node deletes the heredoc opener and not the body. The body
215
+ then parses as code, raises, and counts as a kill. A mutation that drops
216
+ the opener is skipped. A replacement that still contains the opener stays.
217
+ - **A bare `public`, `private` or `protected` ends `module_function` mode**,
218
+ so later `def`s in the module body are named as instance methods, not
219
+ singleton methods (#144).
220
+ - **Turning on reduced motion mid-visit finishes the landing page count-ups**,
221
+ so no number keeps changing after the visitor asks for less motion (#176).
222
+ - **`rake site:build` can no longer delete a directory such as `lib/` or
223
+ `.git`** (#162). The task removed the destination it was given, and only
224
+ refused the checkout, its parents and `docs/`. It now always builds into
225
+ `_site/` at the checkout root, and `rake "site:build[DEST]"` and
226
+ `SITE_BUILD_DEST` are gone.
227
+ - **Documentation matches the 1.4 behavior.** Config examples use supported keys,
228
+ agent loops check the exit code and a real score, and report totals explain
229
+ partial `--fail-fast` runs. The README lists every config key and its validation
230
+ rules. API README links work outside GitHub, repeated changelog headings have
231
+ unique anchors, and the website checks validate built links and anchors.
232
+ Mutant-id wording and the gem's Minitest/RSpec description are also corrected.
233
+ - **A mutant in a nested method counts once** (#157). A `def` inside
234
+ another method is a subject of its own, but most operators also mutated
235
+ its code on the outer method. The same edit then counted twice in the
236
+ score, with two ids. Every operator now skips a nested `def` when it
237
+ mutates the outer method. A `def` inside `class << obj`, where `obj` is
238
+ not `self`, is not a subject, so the outer method still mutates its code.
239
+ `array_literal`, `chain_link`, `condition_true`, `condition_false`,
240
+ `operand_removal`, `operator_assignment` and `return_nil` skipped that
241
+ code before. Their new mutants there can show as new survivors against a
242
+ baseline.
243
+
244
+ A mutant id includes an ordinal among the mutants with the same operator
245
+ and token in a subject. When the duplicates leave the outer method, the
246
+ ordinals of its own mutants change. An old duplicate id can then name a
247
+ different mutant on the outer method, and an `ignore:` entry or a
248
+ baseline entry with that id applies to that mutant with no warning. If
249
+ your code has nested methods, regenerate the `ignore:` entries and the
250
+ baseline for the outer methods.
251
+
252
+ Under `--strategy redefine` (the `--rails` default), a mutant in a nested
253
+ method can survive or error even when a test would catch it. `redefine`
254
+ loads only the inner method, and a call to the outer method defines the
255
+ original inner method again. Before, the copy of the edit on the outer
256
+ method was killed. Now each edit has one verdict, from the inner method.
257
+
9
258
  ## [1.4.0] - 2026-09-30
10
259
 
11
260
  ### Added
@@ -677,6 +926,7 @@ Rails hardening + CI batch (issues #8–#13), all verified Rails-free.
677
926
  - `.mutineer.yml` configuration (CLI > config > default precedence).
678
927
  - Byte-correct source handling for multibyte (UTF-8) sources.
679
928
 
929
+ [1.5.0]: https://github.com/davidteren/mutineer/releases/tag/v1.5.0
680
930
  [1.4.0]: https://github.com/davidteren/mutineer/releases/tag/v1.4.0
681
931
  [1.3.0]: https://github.com/davidteren/mutineer/releases/tag/v1.3.0
682
932
  [1.2.0]: https://github.com/davidteren/mutineer/releases/tag/v1.2.0
data/README.md CHANGED
@@ -9,12 +9,19 @@ change at a time, runs your test suite (Minitest or RSpec) against each mutant,
9
9
  ones your tests failed to catch — the gaps where your suite isn't actually
10
10
  testing anything.
11
11
 
12
+ "Clean-room" means Mutineer was written from scratch. It contains no code from
13
+ [Mutant](https://github.com/mbj/mutant) or any other mutation-testing tool.
14
+ Mutineer's own code is licensed under MIT. See
15
+ [Mutineer and Mutant](https://github.com/davidteren/mutineer#mutineer-and-mutant)
16
+ for how the two tools differ.
17
+
12
18
  - **Prism + stdlib only** — zero runtime dependencies (Ruby ≥ 3.4).
13
19
  - **One mutation per mutant**, validity-checked by re-parsing.
14
20
  - **Fork-isolated**, parallel execution (Linux + macOS).
15
21
  - **Coverage-guided** — each mutant runs only the test files that cover its line.
16
22
  - **Stops at the first failing test** — in-process runs (not `--daemon` or
17
- `--test-command`) stop a mutant's test run at the first failure.
23
+ `--test-command`) stop a mutant's test run at the first failure, unless
24
+ `--matrix` asks for every covering test.
18
25
 
19
26
  📖 **[mutineer.github.io →](https://davidteren.github.io/mutineer/)** — overview, operators, and usage.
20
27
 
@@ -33,7 +40,7 @@ gem "mutineer", group: :test
33
40
  ## Usage
34
41
 
35
42
  ```sh
36
- mutineer run <source...> --test <test...> [options]
43
+ mutineer run <source...> --test <test> [--test <test>...] [options]
37
44
  ```
38
45
 
39
46
  Mutate `lib/calculator.rb`, checking it against its test, and fail CI if the
@@ -47,24 +54,30 @@ mutineer run lib/calculator.rb --test test/calculator_test.rb --threshold 90
47
54
 
48
55
  | Flag | Meaning |
49
56
  |------|---------|
50
- | `--test FILE` | Test file covering the sources (repeatable) |
57
+ | `--test FILE` | Test file covering the sources; one file per flag, so repeat it for each (`--test a_test.rb --test b_test.rb`) |
51
58
  | `--operators LIST` | Comma-separated operator names (default: the Tier-1 set) |
52
59
  | `--threshold FLOAT` | Exit 1 when the score is below FLOAT, or when nothing could be scored and something broke, or more than one mutant produced no verdict and they exceed 10% of those attempted (default: 0 = off) |
53
60
  | `--only NAME` | Restrict to one fully-qualified subject, e.g. `Calculator#add` |
54
61
  | `--framework NAME` | `minitest` (default) or `rspec`; auto-detected as rspec when most `--test` files end in `_spec.rb` |
55
- | `--since REF` | Only mutate lines changed since git `REF` (e.g. `origin/main`) — ideal for PR CI |
62
+ | `--since REF` | Only mutate lines changed since git `REF` (e.g. `origin/main`). Untracked files are scored in full unless Git ignores them. |
56
63
  | `--no-since` | Disable diff scoping; a typed no beats a `.mutineer.yml` `since:` key |
57
- | `--baseline FILE` | Compare against a prior `--format json` run; exit 1 on new survivors / score drop (score drop is skipped under `--since`, whose score covers a different denominator; see [CI](#ci-gating)) |
64
+ | `--baseline FILE` | Compare against a prior `--format json` run; exit 1 on new survivors / score drop (score drop is skipped under `--since`, whose score covers a different denominator; see [CI](https://github.com/davidteren/mutineer#ci-gating)) |
58
65
  | `--baseline-epsilon FLOAT` | Score-drop tolerance for `--baseline` (default: 0) |
59
- | `--jobs N` | Parallel worker count (default: processor count; `1` under `--rails`) |
66
+ | `--jobs N` | Parallel worker count (default: processor count); forced to `1` by `--test-command`, `--fail-fast`, or `--rails` without `--daemon` |
67
+ | `--boot FILE` | Require an app entry point once before forking; select at least one test file |
68
+ | `--rails` | Boot `config/environment` and reconnect ActiveRecord per fork; without `--daemon`, defaults to `redefine` and runs serially |
60
69
  | `--verbose` | Surface the real error when a fork capture fails (alias `--debug`) |
61
70
  | `--strategy NAME` | Mutation application: `reload` whole-file (default) or `redefine` surgical (`7a`/`7b` accepted as deprecated aliases) |
62
- | `--test-command CMD` | Run the suite as a subprocess in the app's own runtime (for apps on Ruby < 3.4); `CMD` must contain `%{files}`. See [Apps on Ruby < 3.4](#apps-on-ruby--34) |
63
- | `--daemon` | Boot the app once in a persistent daemon and fork per mutant, with per-worker DB isolation so `--jobs N` is safe under Rails (needs `--rails`/`--boot`; not with `--test-command`). See [the daemon backend](#faster-parallel-safe-rails-the---daemon-backend) |
71
+ | `--timeout SECONDS` | Per-mutant time limit for in-process runs, in whole seconds (default: 10). A mutant whose tests run longer is a `timeout`; see [Timeouts and the score](https://github.com/davidteren/mutineer#timeouts-and-the-score). `--daemon` and `--test-command` keep their own limits |
72
+ | `--capture-timeout SECONDS` | Time limit for each in-process coverage-capture subprocess and for the clean run of the unmutated tests, in whole seconds (default: 120). A suite slower than this stops the run as not green. `--daemon` uses it for its coverage capture; `--test-command` ignores it and says so |
73
+ | `--cache-dir DIR` | Directory for the coverage cache (default: `.mutineer`). Give two runs from the same project root different directories so they do not share `coverage.json`. `--test-command` builds no cache, so it ignores it and says so |
74
+ | `--test-command CMD` | Run the suite as a subprocess in the app's own runtime (for apps on Ruby < 3.4); `CMD` must contain `%{files}`. See [Apps on Ruby < 3.4](https://github.com/davidteren/mutineer#apps-on-ruby--34) |
75
+ | `--daemon` | Boot the app once in a persistent daemon and fork per mutant, with per-worker DB isolation so `--jobs N` is safe under Rails (needs `--rails`/`--boot`; not with `--test-command`). See [the daemon backend](https://github.com/davidteren/mutineer#faster-parallel-safe-rails-the---daemon-backend) |
64
76
  | `--format human\|json\|html` | Report format (default: human; `html` is a self-contained file) |
65
77
  | `--output FILE` | Write the report to FILE instead of stdout |
66
78
  | `--dry-run` | List candidate mutations without executing (honors suppression) |
67
- | `--fail-fast` | Stop at the first surviving mutant |
79
+ | `--fail-fast` | Stop at the first surviving mutant (`--no-fail-fast` beats a `.mutineer.yml` `fail_fast:` key) |
80
+ | `--matrix` | Run every covering test for each mutant and report the blind and redundant tests; the JSON report also lists each mutant's killers. In-process only; exits 2 with `--daemon`, `--test-command`, `--fail-fast` or `--dry-run`. RSpec needs 3.3 or later. `--no-matrix` beats a `.mutineer.yml` `matrix:` key. See [Kill matrix](https://github.com/davidteren/mutineer#kill-matrix) |
68
81
  | `--list-operators` | List available operators (default vs optional) and exit |
69
82
  | `--version`, `--help` | Print version / usage and exit |
70
83
 
@@ -99,8 +112,8 @@ RAILS_ENV=test bundle exec mutineer run \
99
112
  ```
100
113
 
101
114
  `--rails` boots `config/environment` once in the parent process (every mutant
102
- then forks and inherits it), defaults `--strategy` to `redefine` (surgical — it
103
- avoids reloading files into the app tree), and reconnects ActiveRecord in each
115
+ then forks and inherits it), defaults `--strategy` to `redefine` without `--daemon`
116
+ (surgical — it avoids reloading files into the app tree; `--daemon` keeps `reload`), and reconnects ActiveRecord in each
104
117
  fork so the database connection is fork-safe. Use `--boot FILE` to boot a
105
118
  different entry point. Boot mode requires at least one `--test` file and is
106
119
  coverage-guided — each mutant runs only the test files that exercise its line
@@ -135,9 +148,10 @@ RAILS_ENV=test bundle exec mutineer run \
135
148
  - **One backend at a time** — `--daemon` can't be combined with `--test-command`
136
149
  (choose one), and it needs an app to boot (`--rails` or `--boot`).
137
150
 
138
- Status: **SQLite** today (hermetic, CI-proven). **Postgres** per-worker
139
- provisioning is in progress (#34/#35); until it lands, use `--daemon` with a
140
- SQLite test database, or drop `--jobs` to run serially on other adapters.
151
+ Status: **SQLite** only (hermetic, CI-proven). Per-worker provisioning for
152
+ **Postgres** and other adapters is not supported: on those, `--daemon` scores
153
+ every mutant as `error`. Use `--daemon` with a SQLite test database, or drop
154
+ `--daemon` to run serially on other adapters.
141
155
 
142
156
  ### Apps on Ruby < 3.4
143
157
 
@@ -207,7 +221,23 @@ Tradeoffs — this path is correct but not free:
207
221
  front (a "smoke check") if your unmutated suite isn't green.
208
222
  - **Reload strategy only** (`--strategy redefine` is rejected on this path) and
209
223
  **serial** (`--jobs` is forced to 1). For apps on Ruby ≥ 3.4, `--daemon` gives
210
- safe parallelism instead (see [the daemon backend](#faster-parallel-safe-rails-the---daemon-backend)).
224
+ safe parallelism instead (see [the daemon backend](https://github.com/davidteren/mutineer#faster-parallel-safe-rails-the---daemon-backend)).
225
+
226
+ ## Timeouts and the score
227
+
228
+ The mutation score is `killed / (killed + survived)`. A mutant whose tests run
229
+ past `--timeout` is a `timeout`. It is neither killed nor survived, so it is
230
+ left out of the score, like `no_coverage`, `uncapturable`, `errored`, skipped
231
+ and ignored mutants. A timeout is not counted as a kill, because a hang the
232
+ mutant caused and a suite that is just slow look the same. The human report
233
+ shows the count in its `Timeout:` row, and the JSON report in `summary.timeout`.
234
+
235
+ Under `--threshold`, a timeout is a mutant with no verdict. The run exits 1 when
236
+ more than one mutant produced no verdict and they exceed 10% of those attempted,
237
+ or when nothing could be scored and something broke. If a slow suite times out
238
+ mutants that its tests would catch, raise `--timeout`. A limit set below what
239
+ the suite needs does not raise the score: it turns those mutants into timeouts,
240
+ and once they pass 10% of the attempted mutants the run fails.
211
241
 
212
242
  ## Suppressing equivalent mutants
213
243
 
@@ -268,6 +298,77 @@ Ids are relative to the directory you run mutineer from. mutineer finds
268
298
  (other than your home directory), it warns that the ignore ids will not match
269
299
  and tells you which directory to run from.
270
300
 
301
+ ## Kill matrix
302
+
303
+ `--matrix` reports which tests kill each mutant, and from that, which tests
304
+ add nothing to the suite.
305
+
306
+ ```sh
307
+ mutineer run lib/calculator.rb --test test/calculator_test.rb --matrix
308
+ ```
309
+
310
+ For each mutant, Mutineer runs every test in its covering files instead of
311
+ stopping at the first failure, and records which tests failed. Coverage is
312
+ recorded per test file, so that set also holds tests that never reach the
313
+ mutated line; they pass and count as having run. A mutant's row is complete
314
+ when its whole covering set ran and the suite returned normally (under
315
+ Minitest, an `Interrupt` that cut the run short does not count as returning
316
+ normally). The report
317
+ names two kinds of test:
318
+
319
+ - **Blind:** the test ran in at least one complete row and killed no mutant
320
+ in any row. It would still pass with this run's mutations of the code, so
321
+ it guards none of them. It can still check behavior outside this run's
322
+ mutants (code in other files, or edits no operator makes), so look at what
323
+ it asserts before you delete it, or give it an assertion that can fail.
324
+ - **Redundant:** every mutant the test kills, another test kills too. That
325
+ makes it a candidate for deletion, one test at a time: two redundant tests
326
+ can be the only killers of one mutant, so re-run after each deletion.
327
+
328
+ The answers cover this run's mutants only. A test of code outside the sources
329
+ you passed kills nothing here, so mutate the code a test exercises before you
330
+ call the test blind.
331
+
332
+ The matrix changes no verdict, score or exit code: each mutant gets the
333
+ verdict a run without `--matrix` gives. That run skips every later test,
334
+ test class and example group after the first failure, so when a later one
335
+ exits the process, crashes or runs into the time limit, the mutant is still
336
+ `killed`. The run without `--matrix` still runs the code around the failing
337
+ test (the rest of its Minitest class `run` wrapper, the `after(:all)` hooks of
338
+ its RSpec groups) and the suite's own cleanup (RSpec's `after(:suite)`), so an
339
+ end there keeps the exit status. A failure in a Minitest `parallelize_me!`
340
+ class also keeps it: those run after every serial class and cannot be stopped
341
+ once queued, while a failure in a serial class still stops the run.
342
+
343
+ Under RSpec, `--matrix` needs RSpec 3.3 or later, whose example ids tell
344
+ apart examples on one line. With an older RSpec every row is incomplete.
345
+
346
+ The human report lists up to 20 blind and 20 redundant tests, and the HTML
347
+ report lists them all. `--format json` adds a `matrix` block with every test
348
+ and each mutant's killers (see the
349
+ [JSON schema](https://davidteren.github.io/mutineer/json-schema.html#matrix)).
350
+ A test is its file and its id. The name is `CalculatorTest#test_add` under
351
+ Minitest, or the example's full description under RSpec, where the example id
352
+ (`./spec/calc_spec.rb[1:2]`) tells apart examples that share a description
353
+ and keeps an example whose generated description changes with the mutant as
354
+ one test.
355
+
356
+ Each mutant runs its whole covering set, so every mutant costs what a survivor
357
+ costs. In 1.4, stopping at the first failure cut a full run of rack's
358
+ `lib/rack/utils.rb` from 86 to 89 seconds down to 35 to 41, so expect a matrix
359
+ run to take 2.1 to 2.5 times as long as a normal one. A mutant that reaches the
360
+ per-mutant time limit (`--timeout`, 10 seconds by default) after a test already failed stays `killed`
361
+ with its row marked incomplete. The report names each incomplete row and warns
362
+ that a blind test may have killed one of those mutants, so do not delete a
363
+ blind test until its rows are complete. Raise `--timeout` (or `timeout:` in
364
+ `.mutineer.yml`) to get complete rows.
365
+
366
+ `--matrix` runs on the in-process backend only. It exits 2 with `--daemon`,
367
+ `--test-command`, `--fail-fast` or `--dry-run`, and the message says whether
368
+ each setting came from the command line or `.mutineer.yml`. To run one of
369
+ those with `matrix: true` in the file, pass `--no-matrix`; to run `--matrix`
370
+ with `fail_fast: true` in the file, pass `--no-fail-fast`.
371
+
271
372
  ## CI gating
272
373
 
273
374
  Store a JSON run as a baseline, then fail the build only when a PR makes things
@@ -277,10 +378,15 @@ worse:
277
378
  mutineer run app/ --baseline .mutineer/baseline.json # exit 1 on NEW survivors or a score drop
278
379
  ```
279
380
 
280
- `--baseline` reports which survivors are new (by [mutant id](#mutant-ids)) and any score drop. It
381
+ `--baseline` reports which survivors are new (by [mutant id](https://github.com/davidteren/mutineer#mutant-ids)) and any score drop. It
281
382
  combines with `--threshold` (the worse of the two sets the exit code). Pass a
282
383
  directory (or several sources) to audit a whole layer in one boot — tests are
283
- auto-paired by convention and the report breaks down per source.
384
+ auto-paired by convention and the report breaks down per source. A source
385
+ `app/foo/bar.rb` pairs with `test/foo/bar_test.rb` and with unclaimed
386
+ `test/foo/bar_*_test.rb` files in that directory, such as `bar_upsert_test.rb`.
387
+ It does not take `user_session_test.rb` when `user_session.rb` exists in the same directory.
388
+ It does not take `bar_upsert_guards_test.rb` when `bar_upsert.rb` exists.
389
+ A spec that already pairs is left as that one file.
284
390
 
285
391
  ### GitHub Action
286
392
 
@@ -316,17 +422,17 @@ the format.
316
422
 
317
423
  ## For AI agents & pipelines
318
424
 
319
- Mutineer is built for programmatic use — versioned JSON, [mutant ids](#mutant-ids) that survive unrelated edits,
425
+ Mutineer is built for programmatic use — versioned JSON, [mutant ids](https://github.com/davidteren/mutineer#mutant-ids) that survive unrelated edits,
320
426
  structured exit codes, and diff-scoped runs. See:
321
427
 
322
428
  - **AI agents & CI recipes** — the agent inner-loop and CI-gate recipes (and how
323
429
  to avoid infinite loops on equivalent mutants):
324
430
  [rendered](https://davidteren.github.io/mutineer/agentic-coding.html) ·
325
- [source](docs/agentic-coding.md)
431
+ [source](https://davidteren.github.io/mutineer/agentic-coding.md)
326
432
  - **JSON schema reference** — the `--format json` schema and its versioning
327
433
  contract:
328
434
  [rendered](https://davidteren.github.io/mutineer/json-schema.html) ·
329
- [source](docs/json-schema.md)
435
+ [source](https://davidteren.github.io/mutineer/json-schema.md)
330
436
  - **Ruby API (YARD)** — class reference for the shipped gem:
331
437
  [https://davidteren.github.io/mutineer/api/](https://davidteren.github.io/mutineer/api/)
332
438
 
@@ -335,11 +441,44 @@ structured exit codes, and diff-scoped runs. See:
335
441
  Mutineer reads an optional `.mutineer.yml` from the project root (nearest one,
336
442
  walking up). CLI flags override config; config overrides defaults.
337
443
 
338
- Sources are positional CLI arguments and test files come from `--test`; the
339
- config file accepts these keys: `operators`, `threshold`, `jobs`, `only`,
340
- `require` (extra files to load before mutating), `boot`/`rails`, and
341
- `test_command` (the external-runtime suite command — see
342
- [Apps on Ruby < 3.4](#apps-on-ruby--34)).
444
+ Sources are positional CLI arguments and test files come from `--test`. The
445
+ config file accepts these keys:
446
+
447
+ | Key | Value and purpose |
448
+ |-----|-------------------|
449
+ | `operators` | An operator name or list of names; defaults to the Tier-1 set |
450
+ | `threshold` | A number from 0 to 100; 0 turns the score gate off |
451
+ | `jobs` | A positive integer; the default is the processor count. `test_command`, `fail_fast`, or `--rails` without `--daemon` forces 1. |
452
+ | `only` | A fully-qualified subject name, such as `Calculator#add` |
453
+ | `require` | A path or list of extra files to load before mutating |
454
+ | `boot` | The app entry point to require once before forking |
455
+ | `rails` | `true` or `false`; enables the Rails boot defaults |
456
+ | `since` | A nonblank git ref, or `false` to disable diff scoping |
457
+ | `framework` | `minitest` or `rspec`; an explicit value is kept during test pairing |
458
+ | `verbose` | `true` or `false`; shows capture diagnostics |
459
+ | `ignore` | A mutant id or list of ids to suppress |
460
+ | `baseline` | The path to a prior JSON report |
461
+ | `fail_fast` | `true` or `false`; stops scheduling after the first survivor |
462
+ | `test_command` | The external-runtime suite command, including `%{files}`; see [Apps on Ruby < 3.4](https://github.com/davidteren/mutineer#apps-on-ruby--34) |
463
+ | `daemon` | `true` or `false`; uses the persistent app daemon with worker DB isolation |
464
+ | `timeout` | A positive integer; the per-mutant time limit in seconds (default 10) |
465
+ | `capture_timeout` | A positive integer; the coverage-capture time limit in seconds (default 120) |
466
+ | `cache_dir` | The coverage cache directory (default `.mutineer`) |
467
+ | `matrix` | `true` or `false`; runs every covering test for each mutant and adds the kill matrix to the report |
468
+
469
+ Invalid values for known scalar keys, and a blank `operators` list, exit 2
470
+ with a message naming the file and key. An unknown operator name warns and
471
+ is skipped. If none of the names are known, the run exits 2. An empty
472
+ `require` or `ignore` list is valid. Boolean keys take `true` or `false` (quoted forms also work), not `"yes"`.
473
+ `jobs` must be positive; a string value contains digits only. String values for
474
+ `threshold` and the CLI-only `--baseline-epsilon` use plain decimals such as `90`
475
+ or `0.5`, not `+2`, `1e2`, or `1_0`. String options such as `only` and `baseline`
476
+ cannot be null or boolean. A blank `since` is invalid; use `since: false` to turn
477
+ scoping off. Unknown keys warn and are ignored. Unknown operator names warn and are skipped, and the run exits 2 when none remain. `--operators` replaces a blank or unknown file list.
478
+
479
+ `format`, `strategy`, `output`, `baseline_epsilon`, and `dry_run` are CLI-only.
480
+ For JSON output, use `--format json`, not a `format:` config key. To select RSpec
481
+ in the file, add `framework: rspec`.
343
482
 
344
483
  ```yaml
345
484
  # .mutineer.yml
@@ -351,8 +490,40 @@ require:
351
490
  ```
352
491
 
353
492
  Coverage results are cached in `.mutineer/coverage.json` (digest-keyed; rebuilt
354
- automatically when sources change). Add `.mutineer/` to your `.gitignore`.
493
+ automatically when sources change). Add `.mutineer/` to your `.gitignore`, and
494
+ the directory you set with `--cache-dir` if it is inside the project. Keep
495
+ `.mutineer/` ignored even then: `--test-command` writes lock files to
496
+ `.mutineer/` beside each source file.
497
+
498
+ ## Mutineer and Mutant
499
+
500
+ [Mutant](https://github.com/mbj/mutant), by Markus Schirp, is the established
501
+ mutation-testing tool for Ruby. It has been in development since 2012 and is
502
+ the subject of published research. Mutineer is a separate tool, and it shares
503
+ no code with Mutant.
504
+
505
+ | Aspect | Mutineer | Mutant |
506
+ |---|---|---|
507
+ | License | MIT, for every use | Free for open source (`--usage opensource`); commercial use needs a paid subscription |
508
+ | Runtime dependencies | None (Prism + stdlib) | `parser`, `unparser`, `regexp_parser`, `sorbet-runtime` and others |
509
+ | Ruby | 3.4 and later to run Mutineer; an app on an older Ruby works through `--test-command` (see [Apps on Ruby < 3.4](https://github.com/davidteren/mutineer#apps-on-ruby--34)) | 3.3 and later |
510
+ | Test frameworks | Minitest and RSpec, in one gem | RSpec, Minitest and Test::Unit, one integration gem each |
511
+ | What you mutate | Files (`mutineer run lib/foo.rb --test test/foo_test.rb`), narrowed to one method with `--only 'Foo#bar'` | Subjects named by expression (`mutant run 'Foo#bar'`, `'Foo*'`) |
512
+ | Which tests run | The test files whose coverage reaches the mutated line | The tests that declare the subject (an RSpec description or a Minitest `cover`), or the tests that ran it in a per-test coverage recording |
513
+ | Operators | 20 (5 by default, 15 opt-in) | A larger set; the default `light` set applies almost all of it (`full` adds `#==` to `#eql?`) |
514
+ | CI gating | `--threshold`, `--baseline` deltas, `--since`, and a GitHub Action | Incremental mode (`--since`) and a recorded session history |
515
+ | Machine-readable output | Versioned JSON report schema | Session JSON schema |
516
+
517
+ Choose Mutineer when you want an MIT-licensed tool with no extra gems in your
518
+ bundle, file-based runs, and a JSON contract built for CI gates and AI agents.
519
+ Choose Mutant when you need its deeper operator set, Test::Unit support, or
520
+ subject expressions that select whole namespaces. The two tools do not
521
+ conflict, so you can run both on one project.
522
+
523
+ This comparison describes Mutant 0.17 (checked on 2026-10-06). Check
524
+ [Mutant's README](https://github.com/mbj/mutant#readme) for its current
525
+ license terms and features.
355
526
 
356
527
  ## License
357
528
 
358
- MIT — see [LICENSE](LICENSE).
529
+ MIT — see [LICENSE](https://github.com/davidteren/mutineer/blob/main/LICENSE).