mutineer 1.5.0 → 1.7.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +473 -0
- data/README.md +352 -26
- data/lib/mutineer/child_coverage.rb +53 -0
- data/lib/mutineer/cli.rb +184 -24
- data/lib/mutineer/concern_blocks.rb +80 -0
- data/lib/mutineer/config.rb +148 -8
- data/lib/mutineer/coverage_map.rb +541 -92
- data/lib/mutineer/daemon_backend.rb +227 -86
- data/lib/mutineer/daemon_client.rb +72 -9
- data/lib/mutineer/daemon_server.rb +229 -51
- data/lib/mutineer/external_backend.rb +12 -1
- data/lib/mutineer/file_swap.rb +1 -1
- data/lib/mutineer/init.rb +124 -0
- data/lib/mutineer/isolation.rb +15 -2
- data/lib/mutineer/job_plan.rb +458 -0
- data/lib/mutineer/migrate.rb +705 -0
- data/lib/mutineer/minitest_integration.rb +48 -1
- data/lib/mutineer/mutator_registry.rb +6 -2
- data/lib/mutineer/mutators/base.rb +14 -7
- data/lib/mutineer/mutators/call_to_receiver.rb +135 -0
- data/lib/mutineer/mutators/chain_link.rb +1 -1
- data/lib/mutineer/orphan_guard.rb +100 -0
- data/lib/mutineer/project.rb +411 -21
- data/lib/mutineer/rails_run_env.rb +73 -0
- data/lib/mutineer/rails_worker_db.rb +1477 -48
- data/lib/mutineer/redaction.rb +98 -0
- data/lib/mutineer/reporter.rb +83 -12
- data/lib/mutineer/result.rb +49 -8
- data/lib/mutineer/runner.rb +66 -290
- data/lib/mutineer/statement_lines.rb +54 -7
- data/lib/mutineer/subject.rb +24 -2
- data/lib/mutineer/test_runners/rspec.rb +54 -3
- data/lib/mutineer/test_runners.rb +3 -2
- data/lib/mutineer/triage.rb +111 -0
- data/lib/mutineer/version.rb +1 -1
- data/lib/mutineer.rb +5 -0
- metadata +11 -1
data/README.md
CHANGED
|
@@ -1,6 +1,8 @@
|
|
|
1
1
|
# Mutineer
|
|
2
2
|
|
|
3
3
|
[](https://rubygems.org/gems/mutineer)
|
|
4
|
+
[](https://github.com/davidteren/mutineer/actions/workflows/ci.yml)
|
|
5
|
+
[](https://github.com/davidteren/mutineer/actions/workflows/ci.yml)
|
|
4
6
|
[](https://github.com/marketplace/actions/mutineer-ruby)
|
|
5
7
|
[](https://socket.dev/rubygems/package/mutineer)
|
|
6
8
|
|
|
@@ -21,9 +23,206 @@ for how the two tools differ.
|
|
|
21
23
|
- **Coverage-guided** — each mutant runs only the test files that cover its line.
|
|
22
24
|
- **Stops at the first failing test** — in-process runs (not `--daemon` or
|
|
23
25
|
`--test-command`) stop a mutant's test run at the first failure, unless
|
|
24
|
-
`--matrix` asks for every covering test.
|
|
26
|
+
`--matrix` asks for every covering test. The source's paired test files run
|
|
27
|
+
first, then the rest from fastest to slowest in doubling steps (under 1 s,
|
|
28
|
+
1 to 3 s, 3 to 7 s, and so on); files in one step run in path order.
|
|
25
29
|
|
|
26
|
-
📖 **[
|
|
30
|
+
📖 **[davidteren.github.io/mutineer →](https://davidteren.github.io/mutineer/)** — overview, operators, and usage.
|
|
31
|
+
|
|
32
|
+
New to mutation testing? Read
|
|
33
|
+
[What is mutation testing?](https://github.com/davidteren/mutineer#what-is-mutation-testing)
|
|
34
|
+
first. It takes five minutes and uses one small example.
|
|
35
|
+
|
|
36
|
+
Try it on one file and its test:
|
|
37
|
+
|
|
38
|
+
```sh
|
|
39
|
+
gem install mutineer
|
|
40
|
+
mutineer run lib/calculator.rb --test test/calculator_test.rb
|
|
41
|
+
```
|
|
42
|
+
|
|
43
|
+
## Is it production-ready?
|
|
44
|
+
|
|
45
|
+
Read this section before you gate an app on Mutineer.
|
|
46
|
+
Use the Status column to decide.
|
|
47
|
+
Adopt a row whose Status says supported or serial only.
|
|
48
|
+
Not supported in the Parallelism column means `--daemon` cannot run that row.
|
|
49
|
+
|
|
50
|
+
Mutineer itself needs Ruby 3.4 or later.
|
|
51
|
+
In-process runs fork, so use Linux or macOS.
|
|
52
|
+
A plain Ruby project with no database is supported.
|
|
53
|
+
It runs in parallel on those systems.
|
|
54
|
+
|
|
55
|
+
<!-- contract:support-matrix -->
|
|
56
|
+
| Ruby | Database | Framework | Parallelism | Status |
|
|
57
|
+
| --- | --- | --- | --- | --- |
|
|
58
|
+
| 3.4 or later | SQLite file | Minitest | supported with `--daemon` | supported |
|
|
59
|
+
| 3.4 or later | SQLite file | RSpec | not supported | serial only |
|
|
60
|
+
| 3.4 or later | PostgreSQL | Minitest | supported with `--daemon` | supported |
|
|
61
|
+
| 3.4 or later | PostgreSQL | RSpec | not supported | serial only |
|
|
62
|
+
| 3.4 or later | MySQL 8 | Minitest | supported with `--daemon` | supported |
|
|
63
|
+
| 3.4 or later | MySQL 8 | RSpec | not supported | serial only |
|
|
64
|
+
| 3.4 or later | MariaDB | Minitest | not tested | not tested |
|
|
65
|
+
| 3.4 or later | SQLite in memory | Minitest | not supported | serial only |
|
|
66
|
+
| 3.4 or later | SQLite in memory | RSpec | not supported | serial only |
|
|
67
|
+
| 3.4 or later | more than one database | Minitest | one worker | serial only |
|
|
68
|
+
| 3.4 or later | more than one database | RSpec | not supported | serial only |
|
|
69
|
+
|
|
70
|
+
This matrix is for a Rails app on Ruby 3.4 or later.
|
|
71
|
+
`supported` means you can run that cell in parallel.
|
|
72
|
+
`serial only` means the safe run uses one worker.
|
|
73
|
+
`not supported` means `--daemon` cannot run that cell.
|
|
74
|
+
`not tested` means no CI job runs that cell.
|
|
75
|
+
SQLite file means a database file, not `:memory:`.
|
|
76
|
+
Minitest on that file can use `--daemon`.
|
|
77
|
+
RSpec on that file is serial only.
|
|
78
|
+
PostgreSQL with Minitest can use `--daemon`.
|
|
79
|
+
PostgreSQL with RSpec is serial only.
|
|
80
|
+
MySQL 8 with Minitest can use `--daemon`.
|
|
81
|
+
MariaDB is not tested.
|
|
82
|
+
MySQL 5.7 is not supported.
|
|
83
|
+
A `:memory:` database is serial only.
|
|
84
|
+
An app with more than one database runs one worker.
|
|
85
|
+
`--daemon` isolates only the primary connection.
|
|
86
|
+
`--rails` without `--daemon` is serial on every row.
|
|
87
|
+
A Postgres worker database is copied once, at the start of the run.
|
|
88
|
+
A MySQL 8 worker database is created once, at the start of the run, and the rows are copied then.
|
|
89
|
+
A test that commits rows outside a transaction leaves them in that worker database.
|
|
90
|
+
The next mutant on that worker sees those rows.
|
|
91
|
+
Rails parallel tests have the same ceiling.
|
|
92
|
+
|
|
93
|
+
Mutineer sets `PARALLEL_WORKERS=1`, so a `parallelize` call in `test_helper.rb` starts no Rails workers during a mutant run.
|
|
94
|
+
See [Rails parallel tests, Spring, Bootsnap, and Redis](https://github.com/davidteren/mutineer#rails-parallel-tests-spring-bootsnap-and-redis).
|
|
95
|
+
|
|
96
|
+
<!-- /contract:support-matrix -->
|
|
97
|
+
|
|
98
|
+
An app on an older Ruby uses `--test-command`.
|
|
99
|
+
Mutineer still runs on Ruby 3.4 or later.
|
|
100
|
+
That path is serial.
|
|
101
|
+
It does not narrow tests by coverage.
|
|
102
|
+
A crash in the command can count as a kill.
|
|
103
|
+
|
|
104
|
+
What a release will not break is in
|
|
105
|
+
[STABILITY.md](https://github.com/davidteren/mutineer/blob/main/STABILITY.md).
|
|
106
|
+
|
|
107
|
+
## What is mutation testing?
|
|
108
|
+
|
|
109
|
+
Your tests pass. That shows the code does what the tests check. It does not
|
|
110
|
+
show that the tests check enough. Mutation testing finds out by changing your
|
|
111
|
+
code on purpose and seeing whether any test notices.
|
|
112
|
+
|
|
113
|
+
Each change is small, such as `>=` to `>` or `&&` to `||`. One changed copy of
|
|
114
|
+
your code is a **mutant**. Mutineer runs your tests against each mutant:
|
|
115
|
+
|
|
116
|
+
- When a test fails, the tests **killed** the mutant. Good: they noticed.
|
|
117
|
+
- When every test still passes, the mutant **survived**. Your tests ran that
|
|
118
|
+
code but did not check what it does.
|
|
119
|
+
|
|
120
|
+
### A small example
|
|
121
|
+
|
|
122
|
+
```ruby
|
|
123
|
+
# lib/discount.rb
|
|
124
|
+
class Discount
|
|
125
|
+
def rate(qty)
|
|
126
|
+
return 0.1 if qty >= 10
|
|
127
|
+
0
|
|
128
|
+
end
|
|
129
|
+
end
|
|
130
|
+
```
|
|
131
|
+
|
|
132
|
+
```ruby
|
|
133
|
+
# test/discount_test.rb
|
|
134
|
+
require "minitest/autorun"
|
|
135
|
+
require_relative "../lib/discount"
|
|
136
|
+
|
|
137
|
+
class DiscountTest < Minitest::Test
|
|
138
|
+
def test_bulk_order_gets_ten_percent
|
|
139
|
+
assert_equal 0.1, Discount.new.rate(50)
|
|
140
|
+
end
|
|
141
|
+
|
|
142
|
+
def test_small_order_gets_nothing
|
|
143
|
+
assert_equal 0, Discount.new.rate(1)
|
|
144
|
+
end
|
|
145
|
+
end
|
|
146
|
+
```
|
|
147
|
+
|
|
148
|
+
Both tests pass, and every line runs, so line coverage is 100%. Run Mutineer:
|
|
149
|
+
|
|
150
|
+
```sh
|
|
151
|
+
mutineer run lib/discount.rb --test test/discount_test.rb
|
|
152
|
+
```
|
|
153
|
+
|
|
154
|
+
Part of the report:
|
|
155
|
+
|
|
156
|
+
```text
|
|
157
|
+
Mutation score: 50.0% (killed / (killed + survived); ...)
|
|
158
|
+
|
|
159
|
+
Surviving Mutants
|
|
160
|
+
-----------------
|
|
161
|
+
|
|
162
|
+
lib/discount.rb
|
|
163
|
+
Discount#rate (discount.rb:3)
|
|
164
|
+
Operator: comparison (>= -> >)
|
|
165
|
+
- return 0.1 if qty >= 10
|
|
166
|
+
+ return 0.1 if qty > 10
|
|
167
|
+
```
|
|
168
|
+
|
|
169
|
+
Mutineer changed `>=` to `>`, and both tests still passed. No test checks an
|
|
170
|
+
order of exactly 10, so a bug at that boundary would ship. Add that test:
|
|
171
|
+
|
|
172
|
+
```ruby
|
|
173
|
+
def test_ten_is_the_first_bulk_quantity
|
|
174
|
+
assert_equal 0.1, Discount.new.rate(10)
|
|
175
|
+
end
|
|
176
|
+
```
|
|
177
|
+
|
|
178
|
+
Run Mutineer again. Both mutants are killed, and the score is 100%.
|
|
179
|
+
|
|
180
|
+
### Coverage and mutation testing
|
|
181
|
+
|
|
182
|
+
Line coverage tells you which lines ran during the tests. Mutation testing tells
|
|
183
|
+
you whether the tests would fail if those lines were wrong. A test with no
|
|
184
|
+
assertion still adds coverage, but it kills a mutant only when the change makes
|
|
185
|
+
the code raise an error. That gap is common in
|
|
186
|
+
tests written in a hurry, and in tests written by AI coding agents.
|
|
187
|
+
|
|
188
|
+
### What each result means
|
|
189
|
+
|
|
190
|
+
| Report row | What happened | In the score? |
|
|
191
|
+
| --- | --- | --- |
|
|
192
|
+
| Killed | A test failed against the mutant. | Yes |
|
|
193
|
+
| Survived | Every test passed against the mutant. Look at it. | Yes |
|
|
194
|
+
| No coverage | No test runs that line. Write a test for it first. | No |
|
|
195
|
+
| Timeout | The tests ran past `--timeout`. | No |
|
|
196
|
+
| Errored | The test run crashed, so there is no verdict. | No |
|
|
197
|
+
| Ignored | You marked the mutant as equivalent (see below). | No |
|
|
198
|
+
| Skipped | The change made invalid Ruby, so it did not run. | No |
|
|
199
|
+
|
|
200
|
+
Uncapturable, Unplaceable, and Ran at load are rarer. The
|
|
201
|
+
[JSON schema](https://davidteren.github.io/mutineer/json-schema.html#summary)
|
|
202
|
+
explains each one.
|
|
203
|
+
|
|
204
|
+
The **mutation score** is `killed / (killed + survived)`. The other rows do not
|
|
205
|
+
change the score. When nothing can be scored, the score is empty (`null` in
|
|
206
|
+
JSON), not 0. Under `--threshold`, too many Timeout or Errored mutants still
|
|
207
|
+
fail the run; see [Timeouts and the score](https://github.com/davidteren/mutineer#timeouts-and-the-score).
|
|
208
|
+
|
|
209
|
+
### What to do with a survivor
|
|
210
|
+
|
|
211
|
+
1. Read the diff. Ask what behaviour the change breaks.
|
|
212
|
+
2. Write or strengthen a test that fails under the change. Run again.
|
|
213
|
+
3. Sometimes the change breaks nothing. For example, when `x` is a number,
|
|
214
|
+
swapping `x * 1` for `x / 1` gives the same result. That mutant is **equivalent**, and no test can
|
|
215
|
+
kill it. Suppress it with a reason (see
|
|
216
|
+
[Suppressing equivalent mutants](https://github.com/davidteren/mutineer#suppressing-equivalent-mutants)).
|
|
217
|
+
|
|
218
|
+
### Where to start
|
|
219
|
+
|
|
220
|
+
Each mutant runs part of your suite again, so a run takes longer than one test
|
|
221
|
+
run. Start with one file and its test, as above. Use `--dry-run` to list the
|
|
222
|
+
mutants without running tests. When the report makes sense, widen the run to a
|
|
223
|
+
folder, then gate pull requests on the lines they change with `--since`. On a
|
|
224
|
+
Rails app, the [Rails rollout guide](https://davidteren.github.io/mutineer/rails.html)
|
|
225
|
+
walks through each stage.
|
|
27
226
|
|
|
28
227
|
## Install
|
|
29
228
|
|
|
@@ -41,8 +240,17 @@ gem "mutineer", group: :test
|
|
|
41
240
|
|
|
42
241
|
```sh
|
|
43
242
|
mutineer run <source...> --test <test> [--test <test>...] [options]
|
|
243
|
+
mutineer init [--rails] [--force]
|
|
244
|
+
mutineer triage REPORT.json --reason TEXT (--id ID ... | --all)
|
|
245
|
+
mutineer migrate <source...> [--dry-run] [--operators LIST]
|
|
44
246
|
```
|
|
45
247
|
|
|
248
|
+
- `run` mutates the sources, runs the tests, and prints the report.
|
|
249
|
+
- `init` writes a commented `.mutineer.yml` and prints a first run command.
|
|
250
|
+
- `triage` prints `ignore:` entries for survivors in a JSON report. It does not
|
|
251
|
+
edit files. See [Suppressing equivalent mutants](https://github.com/davidteren/mutineer#suppressing-equivalent-mutants).
|
|
252
|
+
- `migrate` rewrites old ignore ids. See [Migrate old ignore ids](https://github.com/davidteren/mutineer#migrate-old-ignore-ids).
|
|
253
|
+
|
|
46
254
|
Mutate `lib/calculator.rb`, checking it against its test, and fail CI if the
|
|
47
255
|
mutation score drops below 90%:
|
|
48
256
|
|
|
@@ -50,6 +258,26 @@ mutation score drops below 90%:
|
|
|
50
258
|
mutineer run lib/calculator.rb --test test/calculator_test.rb --threshold 90
|
|
51
259
|
```
|
|
52
260
|
|
|
261
|
+
### Migrate old ignore ids
|
|
262
|
+
|
|
263
|
+
```sh
|
|
264
|
+
mutineer migrate <source...> [--dry-run] [--operators LIST]
|
|
265
|
+
```
|
|
266
|
+
|
|
267
|
+
`migrate` rewrites old-format ids in `.mutineer.yml` `ignore:` to the current
|
|
268
|
+
ids. It keeps comments and every other line. When one old id matches several
|
|
269
|
+
mutants, it writes every new id. That keeps the 1.x match. An id that matches
|
|
270
|
+
nothing stays, and the command exits 1. In an `{ id:, reason: }` entry, the
|
|
271
|
+
id changes and the reason stays. One entry holds one id, so an old id there
|
|
272
|
+
that now matches several mutants stays as written: the command names the new
|
|
273
|
+
ids, exits 1, and you split the entry by hand. A block scalar under `ignore:`
|
|
274
|
+
is not supported: the command exits 2 and does not change the file.
|
|
275
|
+
`--dry-run` prints each change and writes nothing.
|
|
276
|
+
|
|
277
|
+
Pass the same sources and `--operators` you use for `run`. The command runs
|
|
278
|
+
no tests. It does not rewrite a baseline. Generate a new baseline with
|
|
279
|
+
`--format json`.
|
|
280
|
+
|
|
53
281
|
### Options
|
|
54
282
|
|
|
55
283
|
| Flag | Meaning |
|
|
@@ -59,7 +287,7 @@ mutineer run lib/calculator.rb --test test/calculator_test.rb --threshold 90
|
|
|
59
287
|
| `--threshold FLOAT` | Exit 1 when the score is below FLOAT, or when nothing could be scored and something broke, or more than one mutant produced no verdict and they exceed 10% of those attempted (default: 0 = off) |
|
|
60
288
|
| `--only NAME` | Restrict to one fully-qualified subject, e.g. `Calculator#add` |
|
|
61
289
|
| `--framework NAME` | `minitest` (default) or `rspec`; auto-detected as rspec when most `--test` files end in `_spec.rb` |
|
|
62
|
-
| `--since REF` | Only mutate lines changed since git `REF` (e.g. `origin/main`). Untracked files are scored in full unless Git ignores them. |
|
|
290
|
+
| `--since REF` | Only mutate lines changed since git `REF` (e.g. `origin/main`). Untracked files are scored in full unless Git ignores them. The report lists only suppressed mutants on changed lines, too. |
|
|
63
291
|
| `--no-since` | Disable diff scoping; a typed no beats a `.mutineer.yml` `since:` key |
|
|
64
292
|
| `--baseline FILE` | Compare against a prior `--format json` run; exit 1 on new survivors / score drop (score drop is skipped under `--since`, whose score covers a different denominator; see [CI](https://github.com/davidteren/mutineer#ci-gating)) |
|
|
65
293
|
| `--baseline-epsilon FLOAT` | Score-drop tolerance for `--baseline` (default: 0) |
|
|
@@ -71,12 +299,14 @@ mutineer run lib/calculator.rb --test test/calculator_test.rb --threshold 90
|
|
|
71
299
|
| `--timeout SECONDS` | Per-mutant time limit for in-process runs, in whole seconds (default: 10). A mutant whose tests run longer is a `timeout`; see [Timeouts and the score](https://github.com/davidteren/mutineer#timeouts-and-the-score). `--daemon` and `--test-command` keep their own limits |
|
|
72
300
|
| `--capture-timeout SECONDS` | Time limit for each in-process coverage-capture subprocess and for the clean run of the unmutated tests, in whole seconds (default: 120). A suite slower than this stops the run as not green. `--daemon` uses it for its coverage capture; `--test-command` ignores it and says so |
|
|
73
301
|
| `--cache-dir DIR` | Directory for the coverage cache (default: `.mutineer`). Give two runs from the same project root different directories so they do not share `coverage.json`. `--test-command` builds no cache, so it ignores it and says so |
|
|
302
|
+
| `--coverage-from FILE` | Reuse the `coverage.json` of an earlier `--rails`/`--boot` run with the same tests over these sources or more, so parallel shards capture coverage once. The run skips the capture. It runs each test that covers one of its sources alone, because a mutant runs only those tests, then all tests once together, and writes its own cache to `--cache-dir`. A test that covers none of its sources does not run alone. A capture that does not fit (changed tests or sources, an owner file added or removed, a source it did not cover, a failed capture, another `PARALLEL_WORKERS`, a cache from before this flag) stops the run (exit 2). Under `--strategy redefine`, eager-load in both runs. It has no effect, and says so, without `--rails`/`--boot` or with `--daemon`, `--test-command` or `--dry-run` |
|
|
74
303
|
| `--test-command CMD` | Run the suite as a subprocess in the app's own runtime (for apps on Ruby < 3.4); `CMD` must contain `%{files}`. See [Apps on Ruby < 3.4](https://github.com/davidteren/mutineer#apps-on-ruby--34) |
|
|
75
|
-
| `--daemon` | Boot the app once in a persistent daemon and fork per mutant, with per-worker DB isolation so `--jobs N` is safe under Rails (needs `--rails`/`--boot`; not with `--test-command`). See [the daemon backend](https://github.com/davidteren/mutineer#faster-parallel-safe-rails-the---daemon-backend) |
|
|
304
|
+
| `--daemon` | Boot the app once in a persistent daemon and fork per mutant, with per-worker DB isolation so `--jobs N` is safe under Rails (needs `--rails`/`--boot`; not with `--test-command`). See [the daemon backend](https://github.com/davidteren/mutineer#faster-parallel-safe-rails-the---daemon-backend). `--no-daemon` beats a `.mutineer.yml` `daemon:` key. It keeps `--rails` serial, which is the 1.x default, and in 2.0 it opts out of the parallel default |
|
|
76
305
|
| `--format human\|json\|html` | Report format (default: human; `html` is a self-contained file) |
|
|
77
306
|
| `--output FILE` | Write the report to FILE instead of stdout |
|
|
78
307
|
| `--dry-run` | List candidate mutations without executing (honors suppression) |
|
|
79
308
|
| `--fail-fast` | Stop at the first surviving mutant (`--no-fail-fast` beats a `.mutineer.yml` `fail_fast:` key) |
|
|
309
|
+
| `--allow-empty` | A run with no mutants is expected. Without it, an empty run warns that Mutineer 2.0 will fail it. A `--since` run whose changes hold no mutants does not warn: it reports that the changes hold nothing to test. A `--since` run with no mutants even before scoping still warns. `--no-allow-empty` beats a `.mutineer.yml` `allow_empty:` key |
|
|
80
310
|
| `--matrix` | Run every covering test for each mutant and report the blind and redundant tests; the JSON report also lists each mutant's killers. In-process only; exits 2 with `--daemon`, `--test-command`, `--fail-fast` or `--dry-run`. RSpec needs 3.3 or later. `--no-matrix` beats a `.mutineer.yml` `matrix:` key. See [Kill matrix](https://github.com/davidteren/mutineer#kill-matrix) |
|
|
81
311
|
| `--list-operators` | List available operators (default vs optional) and exit |
|
|
82
312
|
| `--version`, `--help` | Print version / usage and exit |
|
|
@@ -98,7 +328,7 @@ Run `mutineer --list-operators` to see them. Default (Tier 1): `arithmetic`,
|
|
|
98
328
|
Available but off by default (Tier 2, enable via `--operators`): `return_nil`,
|
|
99
329
|
`literal_mutation`, `condition_negation`, `string_literal`, `regex`,
|
|
100
330
|
`collection_method`, `safe_navigation`, `range`, `negation_removal`, `chain_link`,
|
|
101
|
-
`operand_removal`, `array_literal`, `condition_true`,
|
|
331
|
+
`call_to_receiver`, `operand_removal`, `array_literal`, `condition_true`,
|
|
102
332
|
`condition_false`, `operator_assignment`.
|
|
103
333
|
|
|
104
334
|
## Rails apps
|
|
@@ -119,12 +349,68 @@ different entry point. Boot mode requires at least one `--test` file and is
|
|
|
119
349
|
coverage-guided — each mutant runs only the test files that exercise its line
|
|
120
350
|
(coverage is captured by forking the booted app, then cached).
|
|
121
351
|
|
|
352
|
+
Some code runs while the app boots or a class loads, for example a class body
|
|
353
|
+
that calls a method to build a constant, a `to_prepare` initializer, or a
|
|
354
|
+
`--require` file that calls a source method. That run
|
|
355
|
+
happens before the mutant is applied, and the forked test does not repeat it.
|
|
356
|
+
A mutant on such a line is `ran_at_load`, not `survived` or `no_coverage`. It is
|
|
357
|
+
left out of the score and does not fail `--threshold`; a kill still counts.
|
|
358
|
+
This holds under both strategies: `reload` runs the mutated file's class body
|
|
359
|
+
again, but not an initializer or another file's code, so a survivor on such a
|
|
360
|
+
line is not trusted there either.
|
|
361
|
+
Run those mutants with `--test-command`, which boots a fresh process per mutant,
|
|
362
|
+
to get a verdict.
|
|
363
|
+
|
|
122
364
|
Add Mutineer to your Gemfile's test group:
|
|
123
365
|
|
|
124
366
|
```ruby
|
|
125
367
|
gem "mutineer", group: :test, require: false
|
|
126
368
|
```
|
|
127
369
|
|
|
370
|
+
Start from one command. It writes `.mutineer.yml` and prints the first run.
|
|
371
|
+
It does not replace a file you already wrote. Pass `--force` to replace one.
|
|
372
|
+
|
|
373
|
+
```sh
|
|
374
|
+
RAILS_ENV=test bundle exec mutineer init --rails
|
|
375
|
+
```
|
|
376
|
+
|
|
377
|
+
The path from that first report to a CI gate is the
|
|
378
|
+
[Rails rollout guide](https://davidteren.github.io/mutineer/rails.html).
|
|
379
|
+
|
|
380
|
+
### Rails parallel tests, Spring, Bootsnap, and Redis
|
|
381
|
+
|
|
382
|
+
Mutineer sets `PARALLEL_WORKERS` to `1` for every run.
|
|
383
|
+
Rails then runs each mutant's tests in one process.
|
|
384
|
+
Your `parallelize(workers:)` call stays in `test_helper.rb`.
|
|
385
|
+
It does not start Rails workers during a mutant run.
|
|
386
|
+
If you already set `PARALLEL_WORKERS` to another value, Mutineer replaces it and prints one line.
|
|
387
|
+
With `--test-command`, a script your command runs can set `PARALLEL_WORKERS` again.
|
|
388
|
+
Its value wins, and the Rails workers start again.
|
|
389
|
+
Remove that line from the script, or write `PARALLEL_WORKERS=${PARALLEL_WORKERS:-4}` so it keeps the 1 Mutineer passes.
|
|
390
|
+
|
|
391
|
+
Mutineer does not run your `parallelize_setup` hooks.
|
|
392
|
+
Rails registers its own database hook in that same list.
|
|
393
|
+
Running the list would point the tests at `<base>-<n>` instead of Mutineer's database.
|
|
394
|
+
|
|
395
|
+
`--test-command` also sets `DISABLE_SPRING=1`.
|
|
396
|
+
Spring does not start for that command.
|
|
397
|
+
The `--daemon` path loads `config/environment` itself, so Spring does not start there either.
|
|
398
|
+
A wrapper that sets `DISABLE_SPRING` again wins, same as `PARALLEL_WORKERS`.
|
|
399
|
+
|
|
400
|
+
Bootsnap checks its compile cache by file path, size, and modification time in whole seconds.
|
|
401
|
+
When the size and the time match, it uses the cached code without reading the file content.
|
|
402
|
+
The in-process and `--daemon` paths load each mutant from a new file, so the cache cannot mix them up.
|
|
403
|
+
`--test-command` rewrites the source file in place.
|
|
404
|
+
Two mutants of the same size, written in the same second, can then load the first one's cached code.
|
|
405
|
+
To rule that out, set `DISABLE_BOOTSNAP_COMPILE_CACHE=1` on the Mutineer command.
|
|
406
|
+
Bootsnap keeps its cache in `tmp/cache/bootsnap`, or under `$BOOTSNAP_CACHE_DIR` when that is set.
|
|
407
|
+
If a run uses code that no longer matches your files, delete that folder and run again.
|
|
408
|
+
|
|
409
|
+
`redis-client` (which `redis` 5 uses) opens a new connection after a fork on its own.
|
|
410
|
+
You do not set anything in Mutineer for this.
|
|
411
|
+
`redis` 4.x raises `Redis::InheritedError` instead.
|
|
412
|
+
Open that connection after boot, or reconnect it in an after-fork hook.
|
|
413
|
+
|
|
128
414
|
### Faster, parallel-safe Rails (the `--daemon` backend)
|
|
129
415
|
|
|
130
416
|
`--rails` boots your app once but runs mutants **serially** — parallel `--jobs`
|
|
@@ -144,14 +430,37 @@ RAILS_ENV=test bundle exec mutineer run \
|
|
|
144
430
|
a mutant on an uncovered line is `no_coverage`, so the score stays comparable to
|
|
145
431
|
the in-process `--rails` score.
|
|
146
432
|
- **Safe `--jobs N`** — each worker routes to its own copy of the test database, so
|
|
147
|
-
parallel verdicts equal serial (no fixture cross-talk).
|
|
433
|
+
parallel verdicts equal serial (no fixture cross-talk). On first use, a
|
|
434
|
+
worker's database is a copy of the test database after the app boots, so rows
|
|
435
|
+
written by initializers or `--require` files are there, as in-process. When
|
|
436
|
+
the copy's schema differs from `db/schema.rb` (another schema version or
|
|
437
|
+
`schema_sha1` checksum, or no stored checksum, as in an empty or out-of-date
|
|
438
|
+
test database, or one set up with a plain `load "db/schema.rb"`), the
|
|
439
|
+
worker loads `db/schema.rb` over it, which drops the copied rows of the
|
|
440
|
+
tables it defines.
|
|
148
441
|
- **One backend at a time** — `--daemon` can't be combined with `--test-command`
|
|
149
442
|
(choose one), and it needs an app to boot (`--rails` or `--boot`).
|
|
150
443
|
|
|
151
|
-
Status: **SQLite**
|
|
152
|
-
|
|
153
|
-
|
|
154
|
-
|
|
444
|
+
Status: **SQLite**, **PostgreSQL**, and **MySQL 8** (Minitest). Each Postgres worker uses a
|
|
445
|
+
database named `<base>-mutineer-<N>` (shortened when that would pass 63 bytes).
|
|
446
|
+
Mutineer copies the test database into those names before the workers start,
|
|
447
|
+
and leaves them in place when the run ends. Drop one with
|
|
448
|
+
`DROP DATABASE "myapp_test-mutineer-0";` (use your test database name and slot).
|
|
449
|
+
Each MySQL 8 worker uses a database named `<base>-mutineer-<N>` (shortened when that would pass 64 bytes).
|
|
450
|
+
Mutineer creates that database, loads `schema.rb` or `structure.sql`, and copies rows before the workers start.
|
|
451
|
+
It does not copy the database again after an error or a timeout.
|
|
452
|
+
The worker databases stay in place when the run ends. The next run drops and rebuilds them.
|
|
453
|
+
Triggers in the schema do not fire on the copied rows: in each worker database, Mutineer drops them for the copy and creates them again.
|
|
454
|
+
The database user needs CREATE, DROP, TRIGGER, and the table and row privileges a schema load and a row copy use (such as ALTER, INDEX, REFERENCES, and INSERT) on the worker databases, plus SELECT on the test database.
|
|
455
|
+
A trigger whose DEFINER is another account also needs SET_USER_ID.
|
|
456
|
+
An app on `structure.sql` also needs the `mysql` client on `PATH`.
|
|
457
|
+
Drop one with ``DROP DATABASE `myapp_test-mutineer-0`;`` (MySQL quotes the name with backticks).
|
|
458
|
+
A second run on the same test database stops with `another Mutineer run is using <base>` until the first run ends.
|
|
459
|
+
MySQL runs at most 256 workers.
|
|
460
|
+
`mysql2` and `trilogy` are supported.
|
|
461
|
+
MariaDB is not tested.
|
|
462
|
+
MySQL 5.7 is not supported.
|
|
463
|
+
A `:memory:` database is not supported on `--daemon`.
|
|
155
464
|
|
|
156
465
|
### Apps on Ruby < 3.4
|
|
157
466
|
|
|
@@ -180,7 +489,9 @@ RAILS_ENV=test mutineer run app/models/order.rb \
|
|
|
180
489
|
prepends rbenv/asdf shims when a pin was scrubbed. Do not rely on
|
|
181
490
|
`RBENV_VERSION=…` on the Mutineer command for the suite; use `.ruby-version`
|
|
182
491
|
or a wrapper. Don't put `KEY=val` prefixes *inside* `--test-command` (no shell;
|
|
183
|
-
that would be treated as the program name).
|
|
492
|
+
that would be treated as the program name). Mutineer also sets
|
|
493
|
+
`PARALLEL_WORKERS=1` and `DISABLE_SPRING=1` in the child and replaces a value
|
|
494
|
+
you set (see [Rails parallel tests, Spring, Bootsnap, and Redis](https://github.com/davidteren/mutineer#rails-parallel-tests-spring-bootsnap-and-redis)).
|
|
184
495
|
|
|
185
496
|
#### Under a version manager (rbenv / asdf / chruby)
|
|
186
497
|
|
|
@@ -214,10 +525,13 @@ Mutineer also surfaces a targeted smoke-check message when Bundler prints
|
|
|
214
525
|
Tradeoffs — this path is correct but not free:
|
|
215
526
|
|
|
216
527
|
- **Slower:** your app re-boots for every mutant (no shared boot yet).
|
|
217
|
-
- **No coverage narrowing:** every mutant runs the
|
|
218
|
-
|
|
219
|
-
|
|
220
|
-
|
|
528
|
+
- **No coverage narrowing:** every mutant runs the full `--test` set.
|
|
529
|
+
The run is serial.
|
|
530
|
+
A crash in the command can count as a kill.
|
|
531
|
+
Uncovered mutants count as survivors.
|
|
532
|
+
Do not compare a `--test-command` score with an in-process score.
|
|
533
|
+
Exit codes 126 and 127 (the command could not start) are scored `error`.
|
|
534
|
+
Mutineer prints this caveat on every run and aborts up
|
|
221
535
|
front (a "smoke check") if your unmutated suite isn't green.
|
|
222
536
|
- **Reload strategy only** (`--strategy redefine` is rejected on this path) and
|
|
223
537
|
**serial** (`--jobs` is forced to 1). For apps on Ruby ≥ 3.4, `--daemon` gives
|
|
@@ -227,7 +541,7 @@ Tradeoffs — this path is correct but not free:
|
|
|
227
541
|
|
|
228
542
|
The mutation score is `killed / (killed + survived)`. A mutant whose tests run
|
|
229
543
|
past `--timeout` is a `timeout`. It is neither killed nor survived, so it is
|
|
230
|
-
left out of the score, like `no_coverage`, `uncapturable`, `errored`, skipped
|
|
544
|
+
left out of the score, like `no_coverage`, `uncapturable`, `unplaceable`, `ran_at_load`, `errored`, skipped
|
|
231
545
|
and ignored mutants. A timeout is not counted as a kill, because a hang the
|
|
232
546
|
mutant caused and a suite that is just slow look the same. The human report
|
|
233
547
|
shows the count in its `Timeout:` row, and the JSON report in `summary.timeout`.
|
|
@@ -244,9 +558,12 @@ and once they pass 10% of the attempted mutants the run fails.
|
|
|
244
558
|
Some mutants are equivalent (behaviour-identical) and survive forever — keeping a
|
|
245
559
|
file off 100%. Suppress them so the score and `--threshold` gate stay meaningful:
|
|
246
560
|
|
|
247
|
-
- **Inline:** `some_line # mutineer:disable-line` (or scope it: `# mutineer:disable-line comparison`). Put a reason after `--`: `# mutineer:disable-line comparison -- the test checks only 20`.
|
|
248
|
-
- **Config:** a `.mutineer.yml` `ignore:` list
|
|
249
|
-
|
|
561
|
+
- **Inline:** `some_line # mutineer:disable-line` (or scope it: `# mutineer:disable-line comparison`). Put a reason after `--`: `# mutineer:disable-line comparison -- the test checks only 20`. The reason is stored on the ignored mutant and shown in the JSON report and the HTML report. A marker with no reason stores none.
|
|
562
|
+
- **Config:** a `.mutineer.yml` `ignore:` list. A bare mutant id still works. A mapping is `id` plus an optional `reason`. Mutineer 1.7 is the first version that reads a mapping. An older version matches no mutant to a mapping and does not warn, so the mutant it names is not ignored. Add a mapping only after every gate that reads the file runs 1.7 or later. In the Action, the `version:` input sets the version, or your `Gemfile.lock` does when you set `use-bundler: true`. In your own CI job, your `Gemfile.lock` sets it. The reason shows in the same reports as an inline reason. Each survivor's `id` is printed in the JSON report.
|
|
563
|
+
|
|
564
|
+
`mutineer triage REPORT.json --reason TEXT --all` prints one mapping entry per survivor. `--id ID` prints one entry. Repeat `--id` for several. The command does not edit files. Paste the lines under `ignore:`. They are mappings, so every gate that reads the file must run 1.7 or later (see **Config** above).
|
|
565
|
+
|
|
566
|
+
Built-in skips are listed on the [equivalent skips](https://davidteren.github.io/mutineer/equivalent-skips.html) page. There is no switch to turn them off.
|
|
250
567
|
|
|
251
568
|
Suppressed mutants are excluded from the score (so 100% becomes reachable).
|
|
252
569
|
|
|
@@ -279,14 +596,18 @@ working until 2.0, with a warning:
|
|
|
279
596
|
hid mutants you did not mean to ignore. The warning says so. Keep only the ids for the mutant you meant to
|
|
280
597
|
ignore, not all of them. The list covers only the sources and operators in
|
|
281
598
|
that run, so run over every source with every operator set you use (for
|
|
282
|
-
example your Tier-2 `--operators`) for the full list.
|
|
599
|
+
example your Tier-2 `--operators`) for the full list. Or run
|
|
600
|
+
`mutineer migrate` on those sources. It writes every new id. Delete any id
|
|
601
|
+
you did not mean to ignore. See [Migrate old ignore ids](https://github.com/davidteren/mutineer#migrate-old-ignore-ids).
|
|
283
602
|
- **`--baseline`** An old baseline still matches: a survivor matches a stored
|
|
284
603
|
one with the same old id in the same file. A stored file that is an absolute
|
|
285
604
|
path outside the project root (a baseline written on another machine)
|
|
286
605
|
matches on the old id alone. The run tells you to
|
|
287
|
-
|
|
288
|
-
|
|
289
|
-
|
|
606
|
+
generate a new baseline with `--format json`. `migrate` does not rewrite a
|
|
607
|
+
baseline. Generate the new file only after every gate
|
|
608
|
+
that reads it runs 1.3 or later. In the Action, the `version:` input sets
|
|
609
|
+
the version, or your `Gemfile.lock` does when you set `use-bundler: true`.
|
|
610
|
+
In your own CI job, your `Gemfile.lock` sets it. An older version treats every new-format survivor as new.
|
|
290
611
|
|
|
291
612
|
The JSON report's `summary.id_format` is `2` for the new format.
|
|
292
613
|
`summary.legacy_id_matches.ignore` counts the old-format ignore entries a run
|
|
@@ -435,6 +756,10 @@ structured exit codes, and diff-scoped runs. See:
|
|
|
435
756
|
[source](https://davidteren.github.io/mutineer/json-schema.md)
|
|
436
757
|
- **Ruby API (YARD)** — class reference for the shipped gem:
|
|
437
758
|
[https://davidteren.github.io/mutineer/api/](https://davidteren.github.io/mutineer/api/)
|
|
759
|
+
- **Agent skill** ([`skills/mutineer/SKILL.md`](https://github.com/davidteren/mutineer/blob/main/skills/mutineer/SKILL.md)): a short
|
|
760
|
+
card for coding agents with install, the agent loop, and exit codes. Install it
|
|
761
|
+
with `gh skill install davidteren/mutineer mutineer` or
|
|
762
|
+
`npx skills add davidteren/mutineer --skill mutineer`.
|
|
438
763
|
|
|
439
764
|
## Configuration
|
|
440
765
|
|
|
@@ -465,6 +790,7 @@ config file accepts these keys:
|
|
|
465
790
|
| `capture_timeout` | A positive integer; the coverage-capture time limit in seconds (default 120) |
|
|
466
791
|
| `cache_dir` | The coverage cache directory (default `.mutineer`) |
|
|
467
792
|
| `matrix` | `true` or `false`; runs every covering test for each mutant and adds the kill matrix to the report |
|
|
793
|
+
| `allow_empty` | `true` or `false`; a run with no mutants is expected, so it does not warn |
|
|
468
794
|
|
|
469
795
|
Invalid values for known scalar keys, and a blank `operators` list, exit 2
|
|
470
796
|
with a message naming the file and key. An unknown operator name warns and
|
|
@@ -474,9 +800,9 @@ is skipped. If none of the names are known, the run exits 2. An empty
|
|
|
474
800
|
`threshold` and the CLI-only `--baseline-epsilon` use plain decimals such as `90`
|
|
475
801
|
or `0.5`, not `+2`, `1e2`, or `1_0`. String options such as `only` and `baseline`
|
|
476
802
|
cannot be null or boolean. A blank `since` is invalid; use `since: false` to turn
|
|
477
|
-
scoping off. Unknown keys warn and are ignored. Unknown operator names warn and are skipped, and the run exits 2 when none remain. `--operators` replaces a blank or unknown file list.
|
|
803
|
+
scoping off. Unknown keys warn and are ignored. Unknown operator names warn and are skipped, and the run exits 2 when none remain. Both warnings suggest the closest valid name, and both become errors in Mutineer 2.0. `--operators` replaces a blank or unknown file list.
|
|
478
804
|
|
|
479
|
-
`format`, `strategy`, `output`, `baseline_epsilon`, and `
|
|
805
|
+
`format`, `strategy`, `output`, `baseline_epsilon`, `dry_run`, and `coverage_from` are CLI-only.
|
|
480
806
|
For JSON output, use `--format json`, not a `format:` config key. To select RSpec
|
|
481
807
|
in the file, add `framework: rspec`.
|
|
482
808
|
|
|
@@ -510,7 +836,7 @@ no code with Mutant.
|
|
|
510
836
|
| Test frameworks | Minitest and RSpec, in one gem | RSpec, Minitest and Test::Unit, one integration gem each |
|
|
511
837
|
| What you mutate | Files (`mutineer run lib/foo.rb --test test/foo_test.rb`), narrowed to one method with `--only 'Foo#bar'` | Subjects named by expression (`mutant run 'Foo#bar'`, `'Foo*'`) |
|
|
512
838
|
| Which tests run | The test files whose coverage reaches the mutated line | The tests that declare the subject (an RSpec description or a Minitest `cover`), or the tests that ran it in a per-test coverage recording |
|
|
513
|
-
| Operators |
|
|
839
|
+
| Operators | 21 (5 by default, 16 opt-in) | A larger set; the default `light` set applies almost all of it (`full` adds `#==` to `#eql?`) |
|
|
514
840
|
| CI gating | `--threshold`, `--baseline` deltas, `--since`, and a GitHub Action | Incremental mode (`--since`) and a recorded session history |
|
|
515
841
|
| Machine-readable output | Versioned JSON report schema | Session JSON schema |
|
|
516
842
|
|
|
@@ -0,0 +1,53 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Mutineer
|
|
4
|
+
# Lets a test helper start Coverage in a forked mutant child (#248).
|
|
5
|
+
#
|
|
6
|
+
# Once the coverage map is built, the parent suspends Coverage so mutant
|
|
7
|
+
# children do not pay for it (#228). A suspended Coverage reports
|
|
8
|
+
# `running? == false`, so a test helper that starts it when it is off
|
|
9
|
+
# (SimpleCov's `SimpleCov.start`) calls `Coverage.start`, which raises
|
|
10
|
+
# "coverage measurement is already setup" while Coverage is suspended. Every
|
|
11
|
+
# mutant then errors before a test runs.
|
|
12
|
+
#
|
|
13
|
+
# In the child, `Coverage.start` and `Coverage.setup` first stop the
|
|
14
|
+
# suspended Coverage, then run as usual. Stopping builds and drops the whole
|
|
15
|
+
# result, which walks the heap, so only a child whose helper starts Coverage
|
|
16
|
+
# pays for it. Only files loaded after that start are measured. The parent's
|
|
17
|
+
# Coverage is untouched.
|
|
18
|
+
#
|
|
19
|
+
# Stdlib-only, so the app-side daemon can load it.
|
|
20
|
+
module ChildCoverage
|
|
21
|
+
# Prepended to Coverage's singleton class in the child: stops a suspended
|
|
22
|
+
# Coverage before a start or setup.
|
|
23
|
+
module StopBeforeStart
|
|
24
|
+
# Stops a suspended Coverage, then sets it up.
|
|
25
|
+
#
|
|
26
|
+
# @return [nil]
|
|
27
|
+
def setup(...)
|
|
28
|
+
result(stop: true, clear: true) if state == :suspended
|
|
29
|
+
super
|
|
30
|
+
end
|
|
31
|
+
|
|
32
|
+
# Stops a suspended Coverage, then starts it.
|
|
33
|
+
#
|
|
34
|
+
# @return [nil]
|
|
35
|
+
def start(...)
|
|
36
|
+
result(stop: true, clear: true) if state == :suspended
|
|
37
|
+
super
|
|
38
|
+
end
|
|
39
|
+
end
|
|
40
|
+
|
|
41
|
+
# Makes a later `Coverage.start` or `Coverage.setup` work when Coverage is
|
|
42
|
+
# suspended. A running or idle Coverage is left alone.
|
|
43
|
+
#
|
|
44
|
+
# Call it only in a forked child, before any test loads.
|
|
45
|
+
#
|
|
46
|
+
# @return [void]
|
|
47
|
+
def self.allow_start
|
|
48
|
+
return unless defined?(::Coverage) && ::Coverage.state == :suspended
|
|
49
|
+
|
|
50
|
+
::Coverage.singleton_class.prepend(StopBeforeStart)
|
|
51
|
+
end
|
|
52
|
+
end
|
|
53
|
+
end
|