mutineer 1.4.0 → 1.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +475 -0
- data/README.md +224 -28
- data/lib/mutineer/changed_lines.rb +24 -1
- data/lib/mutineer/cli.rb +117 -20
- data/lib/mutineer/config.rb +64 -13
- data/lib/mutineer/coverage_map.rb +533 -100
- data/lib/mutineer/daemon_backend.rb +51 -33
- data/lib/mutineer/daemon_client.rb +212 -22
- data/lib/mutineer/daemon_server.rb +74 -27
- data/lib/mutineer/external_backend.rb +2 -2
- data/lib/mutineer/file_swap.rb +1 -1
- data/lib/mutineer/isolation.rb +92 -6
- data/lib/mutineer/job_plan.rb +352 -0
- data/lib/mutineer/kill_channel.rb +284 -0
- data/lib/mutineer/kill_matrix.rb +88 -0
- data/lib/mutineer/minitest_integration/kill_recorder.rb +301 -0
- data/lib/mutineer/minitest_integration/outer_reporter.rb +126 -0
- data/lib/mutineer/minitest_integration/stop_at_first_failure.rb +14 -68
- data/lib/mutineer/minitest_integration.rb +75 -11
- data/lib/mutineer/mutant_id.rb +5 -4
- data/lib/mutineer/mutators/array_literal.rb +0 -7
- data/lib/mutineer/mutators/base.rb +63 -4
- data/lib/mutineer/mutators/chain_link.rb +0 -7
- data/lib/mutineer/mutators/collection_method.rb +1 -1
- data/lib/mutineer/mutators/condition_forcing.rb +0 -7
- data/lib/mutineer/mutators/operand_removal.rb +0 -7
- data/lib/mutineer/mutators/operator_assignment.rb +0 -7
- data/lib/mutineer/mutators/return_nil.rb +5 -11
- data/lib/mutineer/mutators/string_literal.rb +1 -1
- data/lib/mutineer/orphan_guard.rb +28 -0
- data/lib/mutineer/pairing.rb +222 -15
- data/lib/mutineer/parser.rb +12 -3
- data/lib/mutineer/project.rb +344 -21
- data/lib/mutineer/rails_worker_db.rb +83 -9
- data/lib/mutineer/reporter.rb +306 -53
- data/lib/mutineer/result.rb +73 -5
- data/lib/mutineer/runner.rb +123 -242
- data/lib/mutineer/statement_lines.rb +183 -0
- data/lib/mutineer/subject.rb +6 -2
- data/lib/mutineer/test_runners/minitest.rb +4 -2
- data/lib/mutineer/test_runners/rspec.rb +282 -3
- data/lib/mutineer/test_runners.rb +5 -3
- data/lib/mutineer/version.rb +1 -1
- data/lib/mutineer/worker_pool.rb +4 -2
- data/lib/mutineer.rb +2 -0
- metadata +10 -3
data/README.md
CHANGED
|
@@ -9,12 +9,21 @@ change at a time, runs your test suite (Minitest or RSpec) against each mutant,
|
|
|
9
9
|
ones your tests failed to catch — the gaps where your suite isn't actually
|
|
10
10
|
testing anything.
|
|
11
11
|
|
|
12
|
+
"Clean-room" means Mutineer was written from scratch. It contains no code from
|
|
13
|
+
[Mutant](https://github.com/mbj/mutant) or any other mutation-testing tool.
|
|
14
|
+
Mutineer's own code is licensed under MIT. See
|
|
15
|
+
[Mutineer and Mutant](https://github.com/davidteren/mutineer#mutineer-and-mutant)
|
|
16
|
+
for how the two tools differ.
|
|
17
|
+
|
|
12
18
|
- **Prism + stdlib only** — zero runtime dependencies (Ruby ≥ 3.4).
|
|
13
19
|
- **One mutation per mutant**, validity-checked by re-parsing.
|
|
14
20
|
- **Fork-isolated**, parallel execution (Linux + macOS).
|
|
15
21
|
- **Coverage-guided** — each mutant runs only the test files that cover its line.
|
|
16
22
|
- **Stops at the first failing test** — in-process runs (not `--daemon` or
|
|
17
|
-
`--test-command`) stop a mutant's test run at the first failure
|
|
23
|
+
`--test-command`) stop a mutant's test run at the first failure, unless
|
|
24
|
+
`--matrix` asks for every covering test. The source's paired test files run
|
|
25
|
+
first, then the rest from fastest to slowest in doubling steps (under 1 s,
|
|
26
|
+
1 to 3 s, 3 to 7 s, and so on); files in one step run in path order.
|
|
18
27
|
|
|
19
28
|
📖 **[mutineer.github.io →](https://davidteren.github.io/mutineer/)** — overview, operators, and usage.
|
|
20
29
|
|
|
@@ -33,7 +42,7 @@ gem "mutineer", group: :test
|
|
|
33
42
|
## Usage
|
|
34
43
|
|
|
35
44
|
```sh
|
|
36
|
-
mutineer run <source...> --test <test
|
|
45
|
+
mutineer run <source...> --test <test> [--test <test>...] [options]
|
|
37
46
|
```
|
|
38
47
|
|
|
39
48
|
Mutate `lib/calculator.rb`, checking it against its test, and fail CI if the
|
|
@@ -47,24 +56,30 @@ mutineer run lib/calculator.rb --test test/calculator_test.rb --threshold 90
|
|
|
47
56
|
|
|
48
57
|
| Flag | Meaning |
|
|
49
58
|
|------|---------|
|
|
50
|
-
| `--test FILE` | Test file covering the sources (
|
|
59
|
+
| `--test FILE` | Test file covering the sources; one file per flag, so repeat it for each (`--test a_test.rb --test b_test.rb`) |
|
|
51
60
|
| `--operators LIST` | Comma-separated operator names (default: the Tier-1 set) |
|
|
52
61
|
| `--threshold FLOAT` | Exit 1 when the score is below FLOAT, or when nothing could be scored and something broke, or more than one mutant produced no verdict and they exceed 10% of those attempted (default: 0 = off) |
|
|
53
62
|
| `--only NAME` | Restrict to one fully-qualified subject, e.g. `Calculator#add` |
|
|
54
63
|
| `--framework NAME` | `minitest` (default) or `rspec`; auto-detected as rspec when most `--test` files end in `_spec.rb` |
|
|
55
|
-
| `--since REF` | Only mutate lines changed since git `REF` (e.g. `origin/main`)
|
|
64
|
+
| `--since REF` | Only mutate lines changed since git `REF` (e.g. `origin/main`). Untracked files are scored in full unless Git ignores them. |
|
|
56
65
|
| `--no-since` | Disable diff scoping; a typed no beats a `.mutineer.yml` `since:` key |
|
|
57
|
-
| `--baseline FILE` | Compare against a prior `--format json` run; exit 1 on new survivors / score drop (score drop is skipped under `--since`, whose score covers a different denominator; see [CI](#ci-gating)) |
|
|
66
|
+
| `--baseline FILE` | Compare against a prior `--format json` run; exit 1 on new survivors / score drop (score drop is skipped under `--since`, whose score covers a different denominator; see [CI](https://github.com/davidteren/mutineer#ci-gating)) |
|
|
58
67
|
| `--baseline-epsilon FLOAT` | Score-drop tolerance for `--baseline` (default: 0) |
|
|
59
|
-
| `--jobs N` | Parallel worker count (default: processor count; `1`
|
|
68
|
+
| `--jobs N` | Parallel worker count (default: processor count); forced to `1` by `--test-command`, `--fail-fast`, or `--rails` without `--daemon` |
|
|
69
|
+
| `--boot FILE` | Require an app entry point once before forking; select at least one test file |
|
|
70
|
+
| `--rails` | Boot `config/environment` and reconnect ActiveRecord per fork; without `--daemon`, defaults to `redefine` and runs serially |
|
|
60
71
|
| `--verbose` | Surface the real error when a fork capture fails (alias `--debug`) |
|
|
61
72
|
| `--strategy NAME` | Mutation application: `reload` whole-file (default) or `redefine` surgical (`7a`/`7b` accepted as deprecated aliases) |
|
|
62
|
-
| `--
|
|
63
|
-
| `--
|
|
73
|
+
| `--timeout SECONDS` | Per-mutant time limit for in-process runs, in whole seconds (default: 10). A mutant whose tests run longer is a `timeout`; see [Timeouts and the score](https://github.com/davidteren/mutineer#timeouts-and-the-score). `--daemon` and `--test-command` keep their own limits |
|
|
74
|
+
| `--capture-timeout SECONDS` | Time limit for each in-process coverage-capture subprocess and for the clean run of the unmutated tests, in whole seconds (default: 120). A suite slower than this stops the run as not green. `--daemon` uses it for its coverage capture; `--test-command` ignores it and says so |
|
|
75
|
+
| `--cache-dir DIR` | Directory for the coverage cache (default: `.mutineer`). Give two runs from the same project root different directories so they do not share `coverage.json`. `--test-command` builds no cache, so it ignores it and says so |
|
|
76
|
+
| `--test-command CMD` | Run the suite as a subprocess in the app's own runtime (for apps on Ruby < 3.4); `CMD` must contain `%{files}`. See [Apps on Ruby < 3.4](https://github.com/davidteren/mutineer#apps-on-ruby--34) |
|
|
77
|
+
| `--daemon` | Boot the app once in a persistent daemon and fork per mutant, with per-worker DB isolation so `--jobs N` is safe under Rails (needs `--rails`/`--boot`; not with `--test-command`). See [the daemon backend](https://github.com/davidteren/mutineer#faster-parallel-safe-rails-the---daemon-backend) |
|
|
64
78
|
| `--format human\|json\|html` | Report format (default: human; `html` is a self-contained file) |
|
|
65
79
|
| `--output FILE` | Write the report to FILE instead of stdout |
|
|
66
80
|
| `--dry-run` | List candidate mutations without executing (honors suppression) |
|
|
67
|
-
| `--fail-fast` | Stop at the first surviving mutant |
|
|
81
|
+
| `--fail-fast` | Stop at the first surviving mutant (`--no-fail-fast` beats a `.mutineer.yml` `fail_fast:` key) |
|
|
82
|
+
| `--matrix` | Run every covering test for each mutant and report the blind and redundant tests; the JSON report also lists each mutant's killers. In-process only; exits 2 with `--daemon`, `--test-command`, `--fail-fast` or `--dry-run`. RSpec needs 3.3 or later. `--no-matrix` beats a `.mutineer.yml` `matrix:` key. See [Kill matrix](https://github.com/davidteren/mutineer#kill-matrix) |
|
|
68
83
|
| `--list-operators` | List available operators (default vs optional) and exit |
|
|
69
84
|
| `--version`, `--help` | Print version / usage and exit |
|
|
70
85
|
|
|
@@ -99,13 +114,25 @@ RAILS_ENV=test bundle exec mutineer run \
|
|
|
99
114
|
```
|
|
100
115
|
|
|
101
116
|
`--rails` boots `config/environment` once in the parent process (every mutant
|
|
102
|
-
then forks and inherits it), defaults `--strategy` to `redefine`
|
|
103
|
-
avoids reloading files into the app tree), and reconnects ActiveRecord in each
|
|
117
|
+
then forks and inherits it), defaults `--strategy` to `redefine` without `--daemon`
|
|
118
|
+
(surgical — it avoids reloading files into the app tree; `--daemon` keeps `reload`), and reconnects ActiveRecord in each
|
|
104
119
|
fork so the database connection is fork-safe. Use `--boot FILE` to boot a
|
|
105
120
|
different entry point. Boot mode requires at least one `--test` file and is
|
|
106
121
|
coverage-guided — each mutant runs only the test files that exercise its line
|
|
107
122
|
(coverage is captured by forking the booted app, then cached).
|
|
108
123
|
|
|
124
|
+
Some code runs while the app boots or a class loads, for example a class body
|
|
125
|
+
that calls a method to build a constant, a `to_prepare` initializer, or a
|
|
126
|
+
`--require` file that calls a source method. That run
|
|
127
|
+
happens before the mutant is applied, and the forked test does not repeat it.
|
|
128
|
+
A mutant on such a line is `ran_at_load`, not `survived` or `no_coverage`. It is
|
|
129
|
+
left out of the score and does not fail `--threshold`; a kill still counts.
|
|
130
|
+
This holds under both strategies: `reload` runs the mutated file's class body
|
|
131
|
+
again, but not an initializer or another file's code, so a survivor on such a
|
|
132
|
+
line is not trusted there either.
|
|
133
|
+
Run those mutants with `--test-command`, which boots a fresh process per mutant,
|
|
134
|
+
to get a verdict.
|
|
135
|
+
|
|
109
136
|
Add Mutineer to your Gemfile's test group:
|
|
110
137
|
|
|
111
138
|
```ruby
|
|
@@ -131,13 +158,21 @@ RAILS_ENV=test bundle exec mutineer run \
|
|
|
131
158
|
a mutant on an uncovered line is `no_coverage`, so the score stays comparable to
|
|
132
159
|
the in-process `--rails` score.
|
|
133
160
|
- **Safe `--jobs N`** — each worker routes to its own copy of the test database, so
|
|
134
|
-
parallel verdicts equal serial (no fixture cross-talk).
|
|
161
|
+
parallel verdicts equal serial (no fixture cross-talk). On first use, a
|
|
162
|
+
worker's database is a copy of the test database after the app boots, so rows
|
|
163
|
+
written by initializers or `--require` files are there, as in-process. When
|
|
164
|
+
the copy's schema differs from `db/schema.rb` (another schema version or
|
|
165
|
+
`schema_sha1` checksum, or no stored checksum, as in an empty or out-of-date
|
|
166
|
+
test database, or one set up with a plain `load "db/schema.rb"`), the
|
|
167
|
+
worker loads `db/schema.rb` over it, which drops the copied rows of the
|
|
168
|
+
tables it defines.
|
|
135
169
|
- **One backend at a time** — `--daemon` can't be combined with `--test-command`
|
|
136
170
|
(choose one), and it needs an app to boot (`--rails` or `--boot`).
|
|
137
171
|
|
|
138
|
-
Status: **SQLite**
|
|
139
|
-
|
|
140
|
-
|
|
172
|
+
Status: **SQLite** only (hermetic, CI-proven). Per-worker provisioning for
|
|
173
|
+
**Postgres** and other adapters is not supported: on those, `--daemon` scores
|
|
174
|
+
every mutant as `error`. Use `--daemon` with a SQLite test database, or drop
|
|
175
|
+
`--daemon` to run serially on other adapters.
|
|
141
176
|
|
|
142
177
|
### Apps on Ruby < 3.4
|
|
143
178
|
|
|
@@ -207,7 +242,23 @@ Tradeoffs — this path is correct but not free:
|
|
|
207
242
|
front (a "smoke check") if your unmutated suite isn't green.
|
|
208
243
|
- **Reload strategy only** (`--strategy redefine` is rejected on this path) and
|
|
209
244
|
**serial** (`--jobs` is forced to 1). For apps on Ruby ≥ 3.4, `--daemon` gives
|
|
210
|
-
safe parallelism instead (see [the daemon backend](#faster-parallel-safe-rails-the---daemon-backend)).
|
|
245
|
+
safe parallelism instead (see [the daemon backend](https://github.com/davidteren/mutineer#faster-parallel-safe-rails-the---daemon-backend)).
|
|
246
|
+
|
|
247
|
+
## Timeouts and the score
|
|
248
|
+
|
|
249
|
+
The mutation score is `killed / (killed + survived)`. A mutant whose tests run
|
|
250
|
+
past `--timeout` is a `timeout`. It is neither killed nor survived, so it is
|
|
251
|
+
left out of the score, like `no_coverage`, `uncapturable`, `unplaceable`, `ran_at_load`, `errored`, skipped
|
|
252
|
+
and ignored mutants. A timeout is not counted as a kill, because a hang the
|
|
253
|
+
mutant caused and a suite that is just slow look the same. The human report
|
|
254
|
+
shows the count in its `Timeout:` row, and the JSON report in `summary.timeout`.
|
|
255
|
+
|
|
256
|
+
Under `--threshold`, a timeout is a mutant with no verdict. The run exits 1 when
|
|
257
|
+
more than one mutant produced no verdict and they exceed 10% of those attempted,
|
|
258
|
+
or when nothing could be scored and something broke. If a slow suite times out
|
|
259
|
+
mutants that its tests would catch, raise `--timeout`. A limit set below what
|
|
260
|
+
the suite needs does not raise the score: it turns those mutants into timeouts,
|
|
261
|
+
and once they pass 10% of the attempted mutants the run fails.
|
|
211
262
|
|
|
212
263
|
## Suppressing equivalent mutants
|
|
213
264
|
|
|
@@ -268,6 +319,77 @@ Ids are relative to the directory you run mutineer from. mutineer finds
|
|
|
268
319
|
(other than your home directory), it warns that the ignore ids will not match
|
|
269
320
|
and tells you which directory to run from.
|
|
270
321
|
|
|
322
|
+
## Kill matrix
|
|
323
|
+
|
|
324
|
+
`--matrix` reports which tests kill each mutant, and from that, which tests
|
|
325
|
+
add nothing to the suite.
|
|
326
|
+
|
|
327
|
+
```sh
|
|
328
|
+
mutineer run lib/calculator.rb --test test/calculator_test.rb --matrix
|
|
329
|
+
```
|
|
330
|
+
|
|
331
|
+
For each mutant, Mutineer runs every test in its covering files instead of
|
|
332
|
+
stopping at the first failure, and records which tests failed. Coverage is
|
|
333
|
+
recorded per test file, so that set also holds tests that never reach the
|
|
334
|
+
mutated line; they pass and count as having run. A mutant's row is complete
|
|
335
|
+
when its whole covering set ran and the suite returned normally (under
|
|
336
|
+
Minitest, an `Interrupt` that cut the run short does not count as returning
|
|
337
|
+
normally). The report
|
|
338
|
+
names two kinds of test:
|
|
339
|
+
|
|
340
|
+
- **Blind:** the test ran in at least one complete row and killed no mutant
|
|
341
|
+
in any row. It would still pass with this run's mutations of the code, so
|
|
342
|
+
it guards none of them. It can still check behavior outside this run's
|
|
343
|
+
mutants (code in other files, or edits no operator makes), so look at what
|
|
344
|
+
it asserts before you delete it, or give it an assertion that can fail.
|
|
345
|
+
- **Redundant:** every mutant the test kills, another test kills too. That
|
|
346
|
+
makes it a candidate for deletion, one test at a time: two redundant tests
|
|
347
|
+
can be the only killers of one mutant, so re-run after each deletion.
|
|
348
|
+
|
|
349
|
+
The answers cover this run's mutants only. A test of code outside the sources
|
|
350
|
+
you passed kills nothing here, so mutate the code a test exercises before you
|
|
351
|
+
call the test blind.
|
|
352
|
+
|
|
353
|
+
The matrix changes no verdict, score or exit code: each mutant gets the
|
|
354
|
+
verdict a run without `--matrix` gives. That run skips every later test,
|
|
355
|
+
test class and example group after the first failure, so when a later one
|
|
356
|
+
exits the process, crashes or runs into the time limit, the mutant is still
|
|
357
|
+
`killed`. The run without `--matrix` still runs the code around the failing
|
|
358
|
+
test (the rest of its Minitest class `run` wrapper, the `after(:all)` hooks of
|
|
359
|
+
its RSpec groups) and the suite's own cleanup (RSpec's `after(:suite)`), so an
|
|
360
|
+
end there keeps the exit status. A failure in a Minitest `parallelize_me!`
|
|
361
|
+
class also keeps it: those run after every serial class and cannot be stopped
|
|
362
|
+
once queued, while a failure in a serial class still stops the run.
|
|
363
|
+
|
|
364
|
+
Under RSpec, `--matrix` needs RSpec 3.3 or later, whose example ids tell
|
|
365
|
+
apart examples on one line. With an older RSpec every row is incomplete.
|
|
366
|
+
|
|
367
|
+
The human report lists up to 20 blind and 20 redundant tests, and the HTML
|
|
368
|
+
report lists them all. `--format json` adds a `matrix` block with every test
|
|
369
|
+
and each mutant's killers (see the
|
|
370
|
+
[JSON schema](https://davidteren.github.io/mutineer/json-schema.html#matrix)).
|
|
371
|
+
A test is its file and its id. The name is `CalculatorTest#test_add` under
|
|
372
|
+
Minitest, or the example's full description under RSpec, where the example id
|
|
373
|
+
(`./spec/calc_spec.rb[1:2]`) tells apart examples that share a description
|
|
374
|
+
and keeps an example whose generated description changes with the mutant as
|
|
375
|
+
one test.
|
|
376
|
+
|
|
377
|
+
Each mutant runs its whole covering set, so every mutant costs what a survivor
|
|
378
|
+
costs. In 1.4, stopping at the first failure cut a full run of rack's
|
|
379
|
+
`lib/rack/utils.rb` from 86 to 89 seconds down to 35 to 41, so expect a matrix
|
|
380
|
+
run to take 2.1 to 2.5 times as long as a normal one. A mutant that reaches the
|
|
381
|
+
per-mutant time limit (`--timeout`, 10 seconds by default) after a test already failed stays `killed`
|
|
382
|
+
with its row marked incomplete. The report names each incomplete row and warns
|
|
383
|
+
that a blind test may have killed one of those mutants, so do not delete a
|
|
384
|
+
blind test until its rows are complete. Raise `--timeout` (or `timeout:` in
|
|
385
|
+
`.mutineer.yml`) to get complete rows.
|
|
386
|
+
|
|
387
|
+
`--matrix` runs on the in-process backend only. It exits 2 with `--daemon`,
|
|
388
|
+
`--test-command`, `--fail-fast` or `--dry-run`, and the message says whether
|
|
389
|
+
each setting came from the command line or `.mutineer.yml`. To run one of
|
|
390
|
+
those with `matrix: true` in the file, pass `--no-matrix`; to run `--matrix`
|
|
391
|
+
with `fail_fast: true` in the file, pass `--no-fail-fast`.
|
|
392
|
+
|
|
271
393
|
## CI gating
|
|
272
394
|
|
|
273
395
|
Store a JSON run as a baseline, then fail the build only when a PR makes things
|
|
@@ -277,10 +399,15 @@ worse:
|
|
|
277
399
|
mutineer run app/ --baseline .mutineer/baseline.json # exit 1 on NEW survivors or a score drop
|
|
278
400
|
```
|
|
279
401
|
|
|
280
|
-
`--baseline` reports which survivors are new (by [mutant id](#mutant-ids)) and any score drop. It
|
|
402
|
+
`--baseline` reports which survivors are new (by [mutant id](https://github.com/davidteren/mutineer#mutant-ids)) and any score drop. It
|
|
281
403
|
combines with `--threshold` (the worse of the two sets the exit code). Pass a
|
|
282
404
|
directory (or several sources) to audit a whole layer in one boot — tests are
|
|
283
|
-
auto-paired by convention and the report breaks down per source.
|
|
405
|
+
auto-paired by convention and the report breaks down per source. A source
|
|
406
|
+
`app/foo/bar.rb` pairs with `test/foo/bar_test.rb` and with unclaimed
|
|
407
|
+
`test/foo/bar_*_test.rb` files in that directory, such as `bar_upsert_test.rb`.
|
|
408
|
+
It does not take `user_session_test.rb` when `user_session.rb` exists in the same directory.
|
|
409
|
+
It does not take `bar_upsert_guards_test.rb` when `bar_upsert.rb` exists.
|
|
410
|
+
A spec that already pairs is left as that one file.
|
|
284
411
|
|
|
285
412
|
### GitHub Action
|
|
286
413
|
|
|
@@ -316,30 +443,67 @@ the format.
|
|
|
316
443
|
|
|
317
444
|
## For AI agents & pipelines
|
|
318
445
|
|
|
319
|
-
Mutineer is built for programmatic use — versioned JSON, [mutant ids](#mutant-ids) that survive unrelated edits,
|
|
446
|
+
Mutineer is built for programmatic use — versioned JSON, [mutant ids](https://github.com/davidteren/mutineer#mutant-ids) that survive unrelated edits,
|
|
320
447
|
structured exit codes, and diff-scoped runs. See:
|
|
321
448
|
|
|
322
449
|
- **AI agents & CI recipes** — the agent inner-loop and CI-gate recipes (and how
|
|
323
450
|
to avoid infinite loops on equivalent mutants):
|
|
324
451
|
[rendered](https://davidteren.github.io/mutineer/agentic-coding.html) ·
|
|
325
|
-
[source](
|
|
452
|
+
[source](https://davidteren.github.io/mutineer/agentic-coding.md)
|
|
326
453
|
- **JSON schema reference** — the `--format json` schema and its versioning
|
|
327
454
|
contract:
|
|
328
455
|
[rendered](https://davidteren.github.io/mutineer/json-schema.html) ·
|
|
329
|
-
[source](
|
|
456
|
+
[source](https://davidteren.github.io/mutineer/json-schema.md)
|
|
330
457
|
- **Ruby API (YARD)** — class reference for the shipped gem:
|
|
331
458
|
[https://davidteren.github.io/mutineer/api/](https://davidteren.github.io/mutineer/api/)
|
|
459
|
+
- **Agent skill** ([`skills/mutineer/SKILL.md`](https://github.com/davidteren/mutineer/blob/main/skills/mutineer/SKILL.md)): a short
|
|
460
|
+
card for coding agents with install, the agent loop, and exit codes. Install it
|
|
461
|
+
with `gh skill install davidteren/mutineer mutineer` or
|
|
462
|
+
`npx skills add davidteren/mutineer --skill mutineer`.
|
|
332
463
|
|
|
333
464
|
## Configuration
|
|
334
465
|
|
|
335
466
|
Mutineer reads an optional `.mutineer.yml` from the project root (nearest one,
|
|
336
467
|
walking up). CLI flags override config; config overrides defaults.
|
|
337
468
|
|
|
338
|
-
Sources are positional CLI arguments and test files come from `--test
|
|
339
|
-
config file accepts these keys:
|
|
340
|
-
|
|
341
|
-
|
|
342
|
-
|
|
469
|
+
Sources are positional CLI arguments and test files come from `--test`. The
|
|
470
|
+
config file accepts these keys:
|
|
471
|
+
|
|
472
|
+
| Key | Value and purpose |
|
|
473
|
+
|-----|-------------------|
|
|
474
|
+
| `operators` | An operator name or list of names; defaults to the Tier-1 set |
|
|
475
|
+
| `threshold` | A number from 0 to 100; 0 turns the score gate off |
|
|
476
|
+
| `jobs` | A positive integer; the default is the processor count. `test_command`, `fail_fast`, or `--rails` without `--daemon` forces 1. |
|
|
477
|
+
| `only` | A fully-qualified subject name, such as `Calculator#add` |
|
|
478
|
+
| `require` | A path or list of extra files to load before mutating |
|
|
479
|
+
| `boot` | The app entry point to require once before forking |
|
|
480
|
+
| `rails` | `true` or `false`; enables the Rails boot defaults |
|
|
481
|
+
| `since` | A nonblank git ref, or `false` to disable diff scoping |
|
|
482
|
+
| `framework` | `minitest` or `rspec`; an explicit value is kept during test pairing |
|
|
483
|
+
| `verbose` | `true` or `false`; shows capture diagnostics |
|
|
484
|
+
| `ignore` | A mutant id or list of ids to suppress |
|
|
485
|
+
| `baseline` | The path to a prior JSON report |
|
|
486
|
+
| `fail_fast` | `true` or `false`; stops scheduling after the first survivor |
|
|
487
|
+
| `test_command` | The external-runtime suite command, including `%{files}`; see [Apps on Ruby < 3.4](https://github.com/davidteren/mutineer#apps-on-ruby--34) |
|
|
488
|
+
| `daemon` | `true` or `false`; uses the persistent app daemon with worker DB isolation |
|
|
489
|
+
| `timeout` | A positive integer; the per-mutant time limit in seconds (default 10) |
|
|
490
|
+
| `capture_timeout` | A positive integer; the coverage-capture time limit in seconds (default 120) |
|
|
491
|
+
| `cache_dir` | The coverage cache directory (default `.mutineer`) |
|
|
492
|
+
| `matrix` | `true` or `false`; runs every covering test for each mutant and adds the kill matrix to the report |
|
|
493
|
+
|
|
494
|
+
Invalid values for known scalar keys, and a blank `operators` list, exit 2
|
|
495
|
+
with a message naming the file and key. An unknown operator name warns and
|
|
496
|
+
is skipped. If none of the names are known, the run exits 2. An empty
|
|
497
|
+
`require` or `ignore` list is valid. Boolean keys take `true` or `false` (quoted forms also work), not `"yes"`.
|
|
498
|
+
`jobs` must be positive; a string value contains digits only. String values for
|
|
499
|
+
`threshold` and the CLI-only `--baseline-epsilon` use plain decimals such as `90`
|
|
500
|
+
or `0.5`, not `+2`, `1e2`, or `1_0`. String options such as `only` and `baseline`
|
|
501
|
+
cannot be null or boolean. A blank `since` is invalid; use `since: false` to turn
|
|
502
|
+
scoping off. Unknown keys warn and are ignored. Unknown operator names warn and are skipped, and the run exits 2 when none remain. `--operators` replaces a blank or unknown file list.
|
|
503
|
+
|
|
504
|
+
`format`, `strategy`, `output`, `baseline_epsilon`, and `dry_run` are CLI-only.
|
|
505
|
+
For JSON output, use `--format json`, not a `format:` config key. To select RSpec
|
|
506
|
+
in the file, add `framework: rspec`.
|
|
343
507
|
|
|
344
508
|
```yaml
|
|
345
509
|
# .mutineer.yml
|
|
@@ -351,8 +515,40 @@ require:
|
|
|
351
515
|
```
|
|
352
516
|
|
|
353
517
|
Coverage results are cached in `.mutineer/coverage.json` (digest-keyed; rebuilt
|
|
354
|
-
automatically when sources change). Add `.mutineer/` to your `.gitignore
|
|
518
|
+
automatically when sources change). Add `.mutineer/` to your `.gitignore`, and
|
|
519
|
+
the directory you set with `--cache-dir` if it is inside the project. Keep
|
|
520
|
+
`.mutineer/` ignored even then: `--test-command` writes lock files to
|
|
521
|
+
`.mutineer/` beside each source file.
|
|
522
|
+
|
|
523
|
+
## Mutineer and Mutant
|
|
524
|
+
|
|
525
|
+
[Mutant](https://github.com/mbj/mutant), by Markus Schirp, is the established
|
|
526
|
+
mutation-testing tool for Ruby. It has been in development since 2012 and is
|
|
527
|
+
the subject of published research. Mutineer is a separate tool, and it shares
|
|
528
|
+
no code with Mutant.
|
|
529
|
+
|
|
530
|
+
| Aspect | Mutineer | Mutant |
|
|
531
|
+
|---|---|---|
|
|
532
|
+
| License | MIT, for every use | Free for open source (`--usage opensource`); commercial use needs a paid subscription |
|
|
533
|
+
| Runtime dependencies | None (Prism + stdlib) | `parser`, `unparser`, `regexp_parser`, `sorbet-runtime` and others |
|
|
534
|
+
| Ruby | 3.4 and later to run Mutineer; an app on an older Ruby works through `--test-command` (see [Apps on Ruby < 3.4](https://github.com/davidteren/mutineer#apps-on-ruby--34)) | 3.3 and later |
|
|
535
|
+
| Test frameworks | Minitest and RSpec, in one gem | RSpec, Minitest and Test::Unit, one integration gem each |
|
|
536
|
+
| What you mutate | Files (`mutineer run lib/foo.rb --test test/foo_test.rb`), narrowed to one method with `--only 'Foo#bar'` | Subjects named by expression (`mutant run 'Foo#bar'`, `'Foo*'`) |
|
|
537
|
+
| Which tests run | The test files whose coverage reaches the mutated line | The tests that declare the subject (an RSpec description or a Minitest `cover`), or the tests that ran it in a per-test coverage recording |
|
|
538
|
+
| Operators | 20 (5 by default, 15 opt-in) | A larger set; the default `light` set applies almost all of it (`full` adds `#==` to `#eql?`) |
|
|
539
|
+
| CI gating | `--threshold`, `--baseline` deltas, `--since`, and a GitHub Action | Incremental mode (`--since`) and a recorded session history |
|
|
540
|
+
| Machine-readable output | Versioned JSON report schema | Session JSON schema |
|
|
541
|
+
|
|
542
|
+
Choose Mutineer when you want an MIT-licensed tool with no extra gems in your
|
|
543
|
+
bundle, file-based runs, and a JSON contract built for CI gates and AI agents.
|
|
544
|
+
Choose Mutant when you need its deeper operator set, Test::Unit support, or
|
|
545
|
+
subject expressions that select whole namespaces. The two tools do not
|
|
546
|
+
conflict, so you can run both on one project.
|
|
547
|
+
|
|
548
|
+
This comparison describes Mutant 0.17 (checked on 2026-10-06). Check
|
|
549
|
+
[Mutant's README](https://github.com/mbj/mutant#readme) for its current
|
|
550
|
+
license terms and features.
|
|
355
551
|
|
|
356
552
|
## License
|
|
357
553
|
|
|
358
|
-
MIT — see [LICENSE](LICENSE).
|
|
554
|
+
MIT — see [LICENSE](https://github.com/davidteren/mutineer/blob/main/LICENSE).
|
|
@@ -59,6 +59,8 @@ module Mutineer
|
|
|
59
59
|
# A file that is not in the index has no usable diff, so `new_file_diff`
|
|
60
60
|
# supplies one that marks every line new. That includes a file the ref still
|
|
61
61
|
# has, deleted and then written again: git prints only a deletion hunk for it.
|
|
62
|
+
# A Git-ignored file outside the index contributes no changed lines, with a
|
|
63
|
+
# warning, so the skip does not look like a file with no changes.
|
|
62
64
|
# Paths are literal, so a name such as `file[1].rb` is not read as a glob.
|
|
63
65
|
#
|
|
64
66
|
# A failure is warned, never silent: an empty result means "no changed
|
|
@@ -73,7 +75,13 @@ module Mutineer
|
|
|
73
75
|
out, _err, status = Open3.capture3(
|
|
74
76
|
"git", "--literal-pathspecs", "-C", project_root, "diff", "--unified=0", ref, "--", abs_file
|
|
75
77
|
)
|
|
76
|
-
|
|
78
|
+
if status.success?
|
|
79
|
+
return out if indexed?(abs_file, project_root)
|
|
80
|
+
return new_file_diff(abs_file) unless ignored?(abs_file, project_root)
|
|
81
|
+
|
|
82
|
+
warn "[mutineer] #{abs_file} is ignored by Git; its lines will not be mutated (--since)"
|
|
83
|
+
return ""
|
|
84
|
+
end
|
|
77
85
|
|
|
78
86
|
warn "[mutineer] git diff failed for #{abs_file}; its lines will not be mutated (--since)"
|
|
79
87
|
""
|
|
@@ -95,6 +103,21 @@ module Mutineer
|
|
|
95
103
|
known.success?
|
|
96
104
|
end
|
|
97
105
|
|
|
106
|
+
# Tells whether Git ignores the file (`.gitignore`, `.git/info/exclude` or
|
|
107
|
+
# the global excludes file).
|
|
108
|
+
#
|
|
109
|
+
# @param abs_file [String] absolute path of the file.
|
|
110
|
+
# @param project_root [String] repository root for `git -C`.
|
|
111
|
+
# @return [Boolean] true when Git ignores the file. False when Git cannot
|
|
112
|
+
# tell, such as for a path beyond a symlink or inside a submodule (exit
|
|
113
|
+
# 128): that file is scored in full, as a file Git does not ignore is.
|
|
114
|
+
def ignored?(abs_file, project_root)
|
|
115
|
+
_out, _err, status = Open3.capture3(
|
|
116
|
+
"git", "-C", project_root, "check-ignore", "--quiet", "--", abs_file
|
|
117
|
+
)
|
|
118
|
+
status.success?
|
|
119
|
+
end
|
|
120
|
+
|
|
98
121
|
# Returns a diff that marks a file as entirely new. It returns `""` when
|
|
99
122
|
# the file is empty or cannot be read (after a warning). A file outside the
|
|
100
123
|
# index has every line new; read as "unchanged", `--since` would score
|