mutineer 1.6.0 → 1.7.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +248 -0
- data/README.md +324 -23
- data/lib/mutineer/child_coverage.rb +53 -0
- data/lib/mutineer/cli.rb +182 -23
- data/lib/mutineer/concern_blocks.rb +80 -0
- data/lib/mutineer/config.rb +148 -8
- data/lib/mutineer/coverage_map.rb +175 -12
- data/lib/mutineer/daemon_backend.rb +182 -56
- data/lib/mutineer/daemon_client.rb +30 -0
- data/lib/mutineer/daemon_server.rb +157 -25
- data/lib/mutineer/external_backend.rb +12 -1
- data/lib/mutineer/init.rb +124 -0
- data/lib/mutineer/isolation.rb +14 -2
- data/lib/mutineer/job_plan.rb +129 -23
- data/lib/mutineer/migrate.rb +705 -0
- data/lib/mutineer/mutator_registry.rb +6 -2
- data/lib/mutineer/mutators/base.rb +14 -7
- data/lib/mutineer/mutators/call_to_receiver.rb +135 -0
- data/lib/mutineer/mutators/chain_link.rb +1 -1
- data/lib/mutineer/orphan_guard.rb +72 -0
- data/lib/mutineer/project.rb +80 -6
- data/lib/mutineer/rails_run_env.rb +73 -0
- data/lib/mutineer/rails_worker_db.rb +1408 -53
- data/lib/mutineer/redaction.rb +98 -0
- data/lib/mutineer/reporter.rb +66 -12
- data/lib/mutineer/result.rb +22 -13
- data/lib/mutineer/runner.rb +24 -12
- data/lib/mutineer/statement_lines.rb +15 -8
- data/lib/mutineer/subject.rb +22 -4
- data/lib/mutineer/test_runners/rspec.rb +15 -3
- data/lib/mutineer/test_runners.rb +3 -2
- data/lib/mutineer/triage.rb +111 -0
- data/lib/mutineer/version.rb +1 -1
- data/lib/mutineer.rb +4 -0
- metadata +9 -1
data/README.md
CHANGED
|
@@ -1,6 +1,8 @@
|
|
|
1
1
|
# Mutineer
|
|
2
2
|
|
|
3
3
|
[](https://rubygems.org/gems/mutineer)
|
|
4
|
+
[](https://github.com/davidteren/mutineer/actions/workflows/ci.yml)
|
|
5
|
+
[](https://github.com/davidteren/mutineer/actions/workflows/ci.yml)
|
|
4
6
|
[](https://github.com/marketplace/actions/mutineer-ruby)
|
|
5
7
|
[](https://socket.dev/rubygems/package/mutineer)
|
|
6
8
|
|
|
@@ -25,7 +27,202 @@ for how the two tools differ.
|
|
|
25
27
|
first, then the rest from fastest to slowest in doubling steps (under 1 s,
|
|
26
28
|
1 to 3 s, 3 to 7 s, and so on); files in one step run in path order.
|
|
27
29
|
|
|
28
|
-
📖 **[
|
|
30
|
+
📖 **[davidteren.github.io/mutineer →](https://davidteren.github.io/mutineer/)** — overview, operators, and usage.
|
|
31
|
+
|
|
32
|
+
New to mutation testing? Read
|
|
33
|
+
[What is mutation testing?](https://github.com/davidteren/mutineer#what-is-mutation-testing)
|
|
34
|
+
first. It takes five minutes and uses one small example.
|
|
35
|
+
|
|
36
|
+
Try it on one file and its test:
|
|
37
|
+
|
|
38
|
+
```sh
|
|
39
|
+
gem install mutineer
|
|
40
|
+
mutineer run lib/calculator.rb --test test/calculator_test.rb
|
|
41
|
+
```
|
|
42
|
+
|
|
43
|
+
## Is it production-ready?
|
|
44
|
+
|
|
45
|
+
Read this section before you gate an app on Mutineer.
|
|
46
|
+
Use the Status column to decide.
|
|
47
|
+
Adopt a row whose Status says supported or serial only.
|
|
48
|
+
Not supported in the Parallelism column means `--daemon` cannot run that row.
|
|
49
|
+
|
|
50
|
+
Mutineer itself needs Ruby 3.4 or later.
|
|
51
|
+
In-process runs fork, so use Linux or macOS.
|
|
52
|
+
A plain Ruby project with no database is supported.
|
|
53
|
+
It runs in parallel on those systems.
|
|
54
|
+
|
|
55
|
+
<!-- contract:support-matrix -->
|
|
56
|
+
| Ruby | Database | Framework | Parallelism | Status |
|
|
57
|
+
| --- | --- | --- | --- | --- |
|
|
58
|
+
| 3.4 or later | SQLite file | Minitest | supported with `--daemon` | supported |
|
|
59
|
+
| 3.4 or later | SQLite file | RSpec | not supported | serial only |
|
|
60
|
+
| 3.4 or later | PostgreSQL | Minitest | supported with `--daemon` | supported |
|
|
61
|
+
| 3.4 or later | PostgreSQL | RSpec | not supported | serial only |
|
|
62
|
+
| 3.4 or later | MySQL 8 | Minitest | supported with `--daemon` | supported |
|
|
63
|
+
| 3.4 or later | MySQL 8 | RSpec | not supported | serial only |
|
|
64
|
+
| 3.4 or later | MariaDB | Minitest | not tested | not tested |
|
|
65
|
+
| 3.4 or later | SQLite in memory | Minitest | not supported | serial only |
|
|
66
|
+
| 3.4 or later | SQLite in memory | RSpec | not supported | serial only |
|
|
67
|
+
| 3.4 or later | more than one database | Minitest | one worker | serial only |
|
|
68
|
+
| 3.4 or later | more than one database | RSpec | not supported | serial only |
|
|
69
|
+
|
|
70
|
+
This matrix is for a Rails app on Ruby 3.4 or later.
|
|
71
|
+
`supported` means you can run that cell in parallel.
|
|
72
|
+
`serial only` means the safe run uses one worker.
|
|
73
|
+
`not supported` means `--daemon` cannot run that cell.
|
|
74
|
+
`not tested` means no CI job runs that cell.
|
|
75
|
+
SQLite file means a database file, not `:memory:`.
|
|
76
|
+
Minitest on that file can use `--daemon`.
|
|
77
|
+
RSpec on that file is serial only.
|
|
78
|
+
PostgreSQL with Minitest can use `--daemon`.
|
|
79
|
+
PostgreSQL with RSpec is serial only.
|
|
80
|
+
MySQL 8 with Minitest can use `--daemon`.
|
|
81
|
+
MariaDB is not tested.
|
|
82
|
+
MySQL 5.7 is not supported.
|
|
83
|
+
A `:memory:` database is serial only.
|
|
84
|
+
An app with more than one database runs one worker.
|
|
85
|
+
`--daemon` isolates only the primary connection.
|
|
86
|
+
`--rails` without `--daemon` is serial on every row.
|
|
87
|
+
A Postgres worker database is copied once, at the start of the run.
|
|
88
|
+
A MySQL 8 worker database is created once, at the start of the run, and the rows are copied then.
|
|
89
|
+
A test that commits rows outside a transaction leaves them in that worker database.
|
|
90
|
+
The next mutant on that worker sees those rows.
|
|
91
|
+
Rails parallel tests have the same ceiling.
|
|
92
|
+
|
|
93
|
+
Mutineer sets `PARALLEL_WORKERS=1`, so a `parallelize` call in `test_helper.rb` starts no Rails workers during a mutant run.
|
|
94
|
+
See [Rails parallel tests, Spring, Bootsnap, and Redis](https://github.com/davidteren/mutineer#rails-parallel-tests-spring-bootsnap-and-redis).
|
|
95
|
+
|
|
96
|
+
<!-- /contract:support-matrix -->
|
|
97
|
+
|
|
98
|
+
An app on an older Ruby uses `--test-command`.
|
|
99
|
+
Mutineer still runs on Ruby 3.4 or later.
|
|
100
|
+
That path is serial.
|
|
101
|
+
It does not narrow tests by coverage.
|
|
102
|
+
A crash in the command can count as a kill.
|
|
103
|
+
|
|
104
|
+
What a release will not break is in
|
|
105
|
+
[STABILITY.md](https://github.com/davidteren/mutineer/blob/main/STABILITY.md).
|
|
106
|
+
|
|
107
|
+
## What is mutation testing?
|
|
108
|
+
|
|
109
|
+
Your tests pass. That shows the code does what the tests check. It does not
|
|
110
|
+
show that the tests check enough. Mutation testing finds out by changing your
|
|
111
|
+
code on purpose and seeing whether any test notices.
|
|
112
|
+
|
|
113
|
+
Each change is small, such as `>=` to `>` or `&&` to `||`. One changed copy of
|
|
114
|
+
your code is a **mutant**. Mutineer runs your tests against each mutant:
|
|
115
|
+
|
|
116
|
+
- When a test fails, the tests **killed** the mutant. Good: they noticed.
|
|
117
|
+
- When every test still passes, the mutant **survived**. Your tests ran that
|
|
118
|
+
code but did not check what it does.
|
|
119
|
+
|
|
120
|
+
### A small example
|
|
121
|
+
|
|
122
|
+
```ruby
|
|
123
|
+
# lib/discount.rb
|
|
124
|
+
class Discount
|
|
125
|
+
def rate(qty)
|
|
126
|
+
return 0.1 if qty >= 10
|
|
127
|
+
0
|
|
128
|
+
end
|
|
129
|
+
end
|
|
130
|
+
```
|
|
131
|
+
|
|
132
|
+
```ruby
|
|
133
|
+
# test/discount_test.rb
|
|
134
|
+
require "minitest/autorun"
|
|
135
|
+
require_relative "../lib/discount"
|
|
136
|
+
|
|
137
|
+
class DiscountTest < Minitest::Test
|
|
138
|
+
def test_bulk_order_gets_ten_percent
|
|
139
|
+
assert_equal 0.1, Discount.new.rate(50)
|
|
140
|
+
end
|
|
141
|
+
|
|
142
|
+
def test_small_order_gets_nothing
|
|
143
|
+
assert_equal 0, Discount.new.rate(1)
|
|
144
|
+
end
|
|
145
|
+
end
|
|
146
|
+
```
|
|
147
|
+
|
|
148
|
+
Both tests pass, and every line runs, so line coverage is 100%. Run Mutineer:
|
|
149
|
+
|
|
150
|
+
```sh
|
|
151
|
+
mutineer run lib/discount.rb --test test/discount_test.rb
|
|
152
|
+
```
|
|
153
|
+
|
|
154
|
+
Part of the report:
|
|
155
|
+
|
|
156
|
+
```text
|
|
157
|
+
Mutation score: 50.0% (killed / (killed + survived); ...)
|
|
158
|
+
|
|
159
|
+
Surviving Mutants
|
|
160
|
+
-----------------
|
|
161
|
+
|
|
162
|
+
lib/discount.rb
|
|
163
|
+
Discount#rate (discount.rb:3)
|
|
164
|
+
Operator: comparison (>= -> >)
|
|
165
|
+
- return 0.1 if qty >= 10
|
|
166
|
+
+ return 0.1 if qty > 10
|
|
167
|
+
```
|
|
168
|
+
|
|
169
|
+
Mutineer changed `>=` to `>`, and both tests still passed. No test checks an
|
|
170
|
+
order of exactly 10, so a bug at that boundary would ship. Add that test:
|
|
171
|
+
|
|
172
|
+
```ruby
|
|
173
|
+
def test_ten_is_the_first_bulk_quantity
|
|
174
|
+
assert_equal 0.1, Discount.new.rate(10)
|
|
175
|
+
end
|
|
176
|
+
```
|
|
177
|
+
|
|
178
|
+
Run Mutineer again. Both mutants are killed, and the score is 100%.
|
|
179
|
+
|
|
180
|
+
### Coverage and mutation testing
|
|
181
|
+
|
|
182
|
+
Line coverage tells you which lines ran during the tests. Mutation testing tells
|
|
183
|
+
you whether the tests would fail if those lines were wrong. A test with no
|
|
184
|
+
assertion still adds coverage, but it kills a mutant only when the change makes
|
|
185
|
+
the code raise an error. That gap is common in
|
|
186
|
+
tests written in a hurry, and in tests written by AI coding agents.
|
|
187
|
+
|
|
188
|
+
### What each result means
|
|
189
|
+
|
|
190
|
+
| Report row | What happened | In the score? |
|
|
191
|
+
| --- | --- | --- |
|
|
192
|
+
| Killed | A test failed against the mutant. | Yes |
|
|
193
|
+
| Survived | Every test passed against the mutant. Look at it. | Yes |
|
|
194
|
+
| No coverage | No test runs that line. Write a test for it first. | No |
|
|
195
|
+
| Timeout | The tests ran past `--timeout`. | No |
|
|
196
|
+
| Errored | The test run crashed, so there is no verdict. | No |
|
|
197
|
+
| Ignored | You marked the mutant as equivalent (see below). | No |
|
|
198
|
+
| Skipped | The change made invalid Ruby, so it did not run. | No |
|
|
199
|
+
|
|
200
|
+
Uncapturable, Unplaceable, and Ran at load are rarer. The
|
|
201
|
+
[JSON schema](https://davidteren.github.io/mutineer/json-schema.html#summary)
|
|
202
|
+
explains each one.
|
|
203
|
+
|
|
204
|
+
The **mutation score** is `killed / (killed + survived)`. The other rows do not
|
|
205
|
+
change the score. When nothing can be scored, the score is empty (`null` in
|
|
206
|
+
JSON), not 0. Under `--threshold`, too many Timeout or Errored mutants still
|
|
207
|
+
fail the run; see [Timeouts and the score](https://github.com/davidteren/mutineer#timeouts-and-the-score).
|
|
208
|
+
|
|
209
|
+
### What to do with a survivor
|
|
210
|
+
|
|
211
|
+
1. Read the diff. Ask what behaviour the change breaks.
|
|
212
|
+
2. Write or strengthen a test that fails under the change. Run again.
|
|
213
|
+
3. Sometimes the change breaks nothing. For example, when `x` is a number,
|
|
214
|
+
swapping `x * 1` for `x / 1` gives the same result. That mutant is **equivalent**, and no test can
|
|
215
|
+
kill it. Suppress it with a reason (see
|
|
216
|
+
[Suppressing equivalent mutants](https://github.com/davidteren/mutineer#suppressing-equivalent-mutants)).
|
|
217
|
+
|
|
218
|
+
### Where to start
|
|
219
|
+
|
|
220
|
+
Each mutant runs part of your suite again, so a run takes longer than one test
|
|
221
|
+
run. Start with one file and its test, as above. Use `--dry-run` to list the
|
|
222
|
+
mutants without running tests. When the report makes sense, widen the run to a
|
|
223
|
+
folder, then gate pull requests on the lines they change with `--since`. On a
|
|
224
|
+
Rails app, the [Rails rollout guide](https://davidteren.github.io/mutineer/rails.html)
|
|
225
|
+
walks through each stage.
|
|
29
226
|
|
|
30
227
|
## Install
|
|
31
228
|
|
|
@@ -43,8 +240,17 @@ gem "mutineer", group: :test
|
|
|
43
240
|
|
|
44
241
|
```sh
|
|
45
242
|
mutineer run <source...> --test <test> [--test <test>...] [options]
|
|
243
|
+
mutineer init [--rails] [--force]
|
|
244
|
+
mutineer triage REPORT.json --reason TEXT (--id ID ... | --all)
|
|
245
|
+
mutineer migrate <source...> [--dry-run] [--operators LIST]
|
|
46
246
|
```
|
|
47
247
|
|
|
248
|
+
- `run` mutates the sources, runs the tests, and prints the report.
|
|
249
|
+
- `init` writes a commented `.mutineer.yml` and prints a first run command.
|
|
250
|
+
- `triage` prints `ignore:` entries for survivors in a JSON report. It does not
|
|
251
|
+
edit files. See [Suppressing equivalent mutants](https://github.com/davidteren/mutineer#suppressing-equivalent-mutants).
|
|
252
|
+
- `migrate` rewrites old ignore ids. See [Migrate old ignore ids](https://github.com/davidteren/mutineer#migrate-old-ignore-ids).
|
|
253
|
+
|
|
48
254
|
Mutate `lib/calculator.rb`, checking it against its test, and fail CI if the
|
|
49
255
|
mutation score drops below 90%:
|
|
50
256
|
|
|
@@ -52,6 +258,26 @@ mutation score drops below 90%:
|
|
|
52
258
|
mutineer run lib/calculator.rb --test test/calculator_test.rb --threshold 90
|
|
53
259
|
```
|
|
54
260
|
|
|
261
|
+
### Migrate old ignore ids
|
|
262
|
+
|
|
263
|
+
```sh
|
|
264
|
+
mutineer migrate <source...> [--dry-run] [--operators LIST]
|
|
265
|
+
```
|
|
266
|
+
|
|
267
|
+
`migrate` rewrites old-format ids in `.mutineer.yml` `ignore:` to the current
|
|
268
|
+
ids. It keeps comments and every other line. When one old id matches several
|
|
269
|
+
mutants, it writes every new id. That keeps the 1.x match. An id that matches
|
|
270
|
+
nothing stays, and the command exits 1. In an `{ id:, reason: }` entry, the
|
|
271
|
+
id changes and the reason stays. One entry holds one id, so an old id there
|
|
272
|
+
that now matches several mutants stays as written: the command names the new
|
|
273
|
+
ids, exits 1, and you split the entry by hand. A block scalar under `ignore:`
|
|
274
|
+
is not supported: the command exits 2 and does not change the file.
|
|
275
|
+
`--dry-run` prints each change and writes nothing.
|
|
276
|
+
|
|
277
|
+
Pass the same sources and `--operators` you use for `run`. The command runs
|
|
278
|
+
no tests. It does not rewrite a baseline. Generate a new baseline with
|
|
279
|
+
`--format json`.
|
|
280
|
+
|
|
55
281
|
### Options
|
|
56
282
|
|
|
57
283
|
| Flag | Meaning |
|
|
@@ -61,7 +287,7 @@ mutineer run lib/calculator.rb --test test/calculator_test.rb --threshold 90
|
|
|
61
287
|
| `--threshold FLOAT` | Exit 1 when the score is below FLOAT, or when nothing could be scored and something broke, or more than one mutant produced no verdict and they exceed 10% of those attempted (default: 0 = off) |
|
|
62
288
|
| `--only NAME` | Restrict to one fully-qualified subject, e.g. `Calculator#add` |
|
|
63
289
|
| `--framework NAME` | `minitest` (default) or `rspec`; auto-detected as rspec when most `--test` files end in `_spec.rb` |
|
|
64
|
-
| `--since REF` | Only mutate lines changed since git `REF` (e.g. `origin/main`). Untracked files are scored in full unless Git ignores them. |
|
|
290
|
+
| `--since REF` | Only mutate lines changed since git `REF` (e.g. `origin/main`). Untracked files are scored in full unless Git ignores them. The report lists only suppressed mutants on changed lines, too. |
|
|
65
291
|
| `--no-since` | Disable diff scoping; a typed no beats a `.mutineer.yml` `since:` key |
|
|
66
292
|
| `--baseline FILE` | Compare against a prior `--format json` run; exit 1 on new survivors / score drop (score drop is skipped under `--since`, whose score covers a different denominator; see [CI](https://github.com/davidteren/mutineer#ci-gating)) |
|
|
67
293
|
| `--baseline-epsilon FLOAT` | Score-drop tolerance for `--baseline` (default: 0) |
|
|
@@ -73,12 +299,14 @@ mutineer run lib/calculator.rb --test test/calculator_test.rb --threshold 90
|
|
|
73
299
|
| `--timeout SECONDS` | Per-mutant time limit for in-process runs, in whole seconds (default: 10). A mutant whose tests run longer is a `timeout`; see [Timeouts and the score](https://github.com/davidteren/mutineer#timeouts-and-the-score). `--daemon` and `--test-command` keep their own limits |
|
|
74
300
|
| `--capture-timeout SECONDS` | Time limit for each in-process coverage-capture subprocess and for the clean run of the unmutated tests, in whole seconds (default: 120). A suite slower than this stops the run as not green. `--daemon` uses it for its coverage capture; `--test-command` ignores it and says so |
|
|
75
301
|
| `--cache-dir DIR` | Directory for the coverage cache (default: `.mutineer`). Give two runs from the same project root different directories so they do not share `coverage.json`. `--test-command` builds no cache, so it ignores it and says so |
|
|
302
|
+
| `--coverage-from FILE` | Reuse the `coverage.json` of an earlier `--rails`/`--boot` run with the same tests over these sources or more, so parallel shards capture coverage once. The run skips the capture. It runs each test that covers one of its sources alone, because a mutant runs only those tests, then all tests once together, and writes its own cache to `--cache-dir`. A test that covers none of its sources does not run alone. A capture that does not fit (changed tests or sources, an owner file added or removed, a source it did not cover, a failed capture, another `PARALLEL_WORKERS`, a cache from before this flag) stops the run (exit 2). Under `--strategy redefine`, eager-load in both runs. It has no effect, and says so, without `--rails`/`--boot` or with `--daemon`, `--test-command` or `--dry-run` |
|
|
76
303
|
| `--test-command CMD` | Run the suite as a subprocess in the app's own runtime (for apps on Ruby < 3.4); `CMD` must contain `%{files}`. See [Apps on Ruby < 3.4](https://github.com/davidteren/mutineer#apps-on-ruby--34) |
|
|
77
|
-
| `--daemon` | Boot the app once in a persistent daemon and fork per mutant, with per-worker DB isolation so `--jobs N` is safe under Rails (needs `--rails`/`--boot`; not with `--test-command`). See [the daemon backend](https://github.com/davidteren/mutineer#faster-parallel-safe-rails-the---daemon-backend) |
|
|
304
|
+
| `--daemon` | Boot the app once in a persistent daemon and fork per mutant, with per-worker DB isolation so `--jobs N` is safe under Rails (needs `--rails`/`--boot`; not with `--test-command`). See [the daemon backend](https://github.com/davidteren/mutineer#faster-parallel-safe-rails-the---daemon-backend). `--no-daemon` beats a `.mutineer.yml` `daemon:` key. It keeps `--rails` serial, which is the 1.x default, and in 2.0 it opts out of the parallel default |
|
|
78
305
|
| `--format human\|json\|html` | Report format (default: human; `html` is a self-contained file) |
|
|
79
306
|
| `--output FILE` | Write the report to FILE instead of stdout |
|
|
80
307
|
| `--dry-run` | List candidate mutations without executing (honors suppression) |
|
|
81
308
|
| `--fail-fast` | Stop at the first surviving mutant (`--no-fail-fast` beats a `.mutineer.yml` `fail_fast:` key) |
|
|
309
|
+
| `--allow-empty` | A run with no mutants is expected. Without it, an empty run warns that Mutineer 2.0 will fail it. A `--since` run whose changes hold no mutants does not warn: it reports that the changes hold nothing to test. A `--since` run with no mutants even before scoping still warns. `--no-allow-empty` beats a `.mutineer.yml` `allow_empty:` key |
|
|
82
310
|
| `--matrix` | Run every covering test for each mutant and report the blind and redundant tests; the JSON report also lists each mutant's killers. In-process only; exits 2 with `--daemon`, `--test-command`, `--fail-fast` or `--dry-run`. RSpec needs 3.3 or later. `--no-matrix` beats a `.mutineer.yml` `matrix:` key. See [Kill matrix](https://github.com/davidteren/mutineer#kill-matrix) |
|
|
83
311
|
| `--list-operators` | List available operators (default vs optional) and exit |
|
|
84
312
|
| `--version`, `--help` | Print version / usage and exit |
|
|
@@ -100,7 +328,7 @@ Run `mutineer --list-operators` to see them. Default (Tier 1): `arithmetic`,
|
|
|
100
328
|
Available but off by default (Tier 2, enable via `--operators`): `return_nil`,
|
|
101
329
|
`literal_mutation`, `condition_negation`, `string_literal`, `regex`,
|
|
102
330
|
`collection_method`, `safe_navigation`, `range`, `negation_removal`, `chain_link`,
|
|
103
|
-
`operand_removal`, `array_literal`, `condition_true`,
|
|
331
|
+
`call_to_receiver`, `operand_removal`, `array_literal`, `condition_true`,
|
|
104
332
|
`condition_false`, `operator_assignment`.
|
|
105
333
|
|
|
106
334
|
## Rails apps
|
|
@@ -139,6 +367,50 @@ Add Mutineer to your Gemfile's test group:
|
|
|
139
367
|
gem "mutineer", group: :test, require: false
|
|
140
368
|
```
|
|
141
369
|
|
|
370
|
+
Start from one command. It writes `.mutineer.yml` and prints the first run.
|
|
371
|
+
It does not replace a file you already wrote. Pass `--force` to replace one.
|
|
372
|
+
|
|
373
|
+
```sh
|
|
374
|
+
RAILS_ENV=test bundle exec mutineer init --rails
|
|
375
|
+
```
|
|
376
|
+
|
|
377
|
+
The path from that first report to a CI gate is the
|
|
378
|
+
[Rails rollout guide](https://davidteren.github.io/mutineer/rails.html).
|
|
379
|
+
|
|
380
|
+
### Rails parallel tests, Spring, Bootsnap, and Redis
|
|
381
|
+
|
|
382
|
+
Mutineer sets `PARALLEL_WORKERS` to `1` for every run.
|
|
383
|
+
Rails then runs each mutant's tests in one process.
|
|
384
|
+
Your `parallelize(workers:)` call stays in `test_helper.rb`.
|
|
385
|
+
It does not start Rails workers during a mutant run.
|
|
386
|
+
If you already set `PARALLEL_WORKERS` to another value, Mutineer replaces it and prints one line.
|
|
387
|
+
With `--test-command`, a script your command runs can set `PARALLEL_WORKERS` again.
|
|
388
|
+
Its value wins, and the Rails workers start again.
|
|
389
|
+
Remove that line from the script, or write `PARALLEL_WORKERS=${PARALLEL_WORKERS:-4}` so it keeps the 1 Mutineer passes.
|
|
390
|
+
|
|
391
|
+
Mutineer does not run your `parallelize_setup` hooks.
|
|
392
|
+
Rails registers its own database hook in that same list.
|
|
393
|
+
Running the list would point the tests at `<base>-<n>` instead of Mutineer's database.
|
|
394
|
+
|
|
395
|
+
`--test-command` also sets `DISABLE_SPRING=1`.
|
|
396
|
+
Spring does not start for that command.
|
|
397
|
+
The `--daemon` path loads `config/environment` itself, so Spring does not start there either.
|
|
398
|
+
A wrapper that sets `DISABLE_SPRING` again wins, same as `PARALLEL_WORKERS`.
|
|
399
|
+
|
|
400
|
+
Bootsnap checks its compile cache by file path, size, and modification time in whole seconds.
|
|
401
|
+
When the size and the time match, it uses the cached code without reading the file content.
|
|
402
|
+
The in-process and `--daemon` paths load each mutant from a new file, so the cache cannot mix them up.
|
|
403
|
+
`--test-command` rewrites the source file in place.
|
|
404
|
+
Two mutants of the same size, written in the same second, can then load the first one's cached code.
|
|
405
|
+
To rule that out, set `DISABLE_BOOTSNAP_COMPILE_CACHE=1` on the Mutineer command.
|
|
406
|
+
Bootsnap keeps its cache in `tmp/cache/bootsnap`, or under `$BOOTSNAP_CACHE_DIR` when that is set.
|
|
407
|
+
If a run uses code that no longer matches your files, delete that folder and run again.
|
|
408
|
+
|
|
409
|
+
`redis-client` (which `redis` 5 uses) opens a new connection after a fork on its own.
|
|
410
|
+
You do not set anything in Mutineer for this.
|
|
411
|
+
`redis` 4.x raises `Redis::InheritedError` instead.
|
|
412
|
+
Open that connection after boot, or reconnect it in an after-fork hook.
|
|
413
|
+
|
|
142
414
|
### Faster, parallel-safe Rails (the `--daemon` backend)
|
|
143
415
|
|
|
144
416
|
`--rails` boots your app once but runs mutants **serially** — parallel `--jobs`
|
|
@@ -169,10 +441,26 @@ RAILS_ENV=test bundle exec mutineer run \
|
|
|
169
441
|
- **One backend at a time** — `--daemon` can't be combined with `--test-command`
|
|
170
442
|
(choose one), and it needs an app to boot (`--rails` or `--boot`).
|
|
171
443
|
|
|
172
|
-
Status: **SQLite**
|
|
173
|
-
|
|
174
|
-
|
|
175
|
-
|
|
444
|
+
Status: **SQLite**, **PostgreSQL**, and **MySQL 8** (Minitest). Each Postgres worker uses a
|
|
445
|
+
database named `<base>-mutineer-<N>` (shortened when that would pass 63 bytes).
|
|
446
|
+
Mutineer copies the test database into those names before the workers start,
|
|
447
|
+
and leaves them in place when the run ends. Drop one with
|
|
448
|
+
`DROP DATABASE "myapp_test-mutineer-0";` (use your test database name and slot).
|
|
449
|
+
Each MySQL 8 worker uses a database named `<base>-mutineer-<N>` (shortened when that would pass 64 bytes).
|
|
450
|
+
Mutineer creates that database, loads `schema.rb` or `structure.sql`, and copies rows before the workers start.
|
|
451
|
+
It does not copy the database again after an error or a timeout.
|
|
452
|
+
The worker databases stay in place when the run ends. The next run drops and rebuilds them.
|
|
453
|
+
Triggers in the schema do not fire on the copied rows: in each worker database, Mutineer drops them for the copy and creates them again.
|
|
454
|
+
The database user needs CREATE, DROP, TRIGGER, and the table and row privileges a schema load and a row copy use (such as ALTER, INDEX, REFERENCES, and INSERT) on the worker databases, plus SELECT on the test database.
|
|
455
|
+
A trigger whose DEFINER is another account also needs SET_USER_ID.
|
|
456
|
+
An app on `structure.sql` also needs the `mysql` client on `PATH`.
|
|
457
|
+
Drop one with ``DROP DATABASE `myapp_test-mutineer-0`;`` (MySQL quotes the name with backticks).
|
|
458
|
+
A second run on the same test database stops with `another Mutineer run is using <base>` until the first run ends.
|
|
459
|
+
MySQL runs at most 256 workers.
|
|
460
|
+
`mysql2` and `trilogy` are supported.
|
|
461
|
+
MariaDB is not tested.
|
|
462
|
+
MySQL 5.7 is not supported.
|
|
463
|
+
A `:memory:` database is not supported on `--daemon`.
|
|
176
464
|
|
|
177
465
|
### Apps on Ruby < 3.4
|
|
178
466
|
|
|
@@ -201,7 +489,9 @@ RAILS_ENV=test mutineer run app/models/order.rb \
|
|
|
201
489
|
prepends rbenv/asdf shims when a pin was scrubbed. Do not rely on
|
|
202
490
|
`RBENV_VERSION=…` on the Mutineer command for the suite; use `.ruby-version`
|
|
203
491
|
or a wrapper. Don't put `KEY=val` prefixes *inside* `--test-command` (no shell;
|
|
204
|
-
that would be treated as the program name).
|
|
492
|
+
that would be treated as the program name). Mutineer also sets
|
|
493
|
+
`PARALLEL_WORKERS=1` and `DISABLE_SPRING=1` in the child and replaces a value
|
|
494
|
+
you set (see [Rails parallel tests, Spring, Bootsnap, and Redis](https://github.com/davidteren/mutineer#rails-parallel-tests-spring-bootsnap-and-redis)).
|
|
205
495
|
|
|
206
496
|
#### Under a version manager (rbenv / asdf / chruby)
|
|
207
497
|
|
|
@@ -235,10 +525,13 @@ Mutineer also surfaces a targeted smoke-check message when Bundler prints
|
|
|
235
525
|
Tradeoffs — this path is correct but not free:
|
|
236
526
|
|
|
237
527
|
- **Slower:** your app re-boots for every mutant (no shared boot yet).
|
|
238
|
-
- **No coverage narrowing:** every mutant runs the
|
|
239
|
-
|
|
240
|
-
|
|
241
|
-
|
|
528
|
+
- **No coverage narrowing:** every mutant runs the full `--test` set.
|
|
529
|
+
The run is serial.
|
|
530
|
+
A crash in the command can count as a kill.
|
|
531
|
+
Uncovered mutants count as survivors.
|
|
532
|
+
Do not compare a `--test-command` score with an in-process score.
|
|
533
|
+
Exit codes 126 and 127 (the command could not start) are scored `error`.
|
|
534
|
+
Mutineer prints this caveat on every run and aborts up
|
|
242
535
|
front (a "smoke check") if your unmutated suite isn't green.
|
|
243
536
|
- **Reload strategy only** (`--strategy redefine` is rejected on this path) and
|
|
244
537
|
**serial** (`--jobs` is forced to 1). For apps on Ruby ≥ 3.4, `--daemon` gives
|
|
@@ -265,9 +558,12 @@ and once they pass 10% of the attempted mutants the run fails.
|
|
|
265
558
|
Some mutants are equivalent (behaviour-identical) and survive forever — keeping a
|
|
266
559
|
file off 100%. Suppress them so the score and `--threshold` gate stay meaningful:
|
|
267
560
|
|
|
268
|
-
- **Inline:** `some_line # mutineer:disable-line` (or scope it: `# mutineer:disable-line comparison`). Put a reason after `--`: `# mutineer:disable-line comparison -- the test checks only 20`.
|
|
269
|
-
- **Config:** a `.mutineer.yml` `ignore:` list
|
|
270
|
-
|
|
561
|
+
- **Inline:** `some_line # mutineer:disable-line` (or scope it: `# mutineer:disable-line comparison`). Put a reason after `--`: `# mutineer:disable-line comparison -- the test checks only 20`. The reason is stored on the ignored mutant and shown in the JSON report and the HTML report. A marker with no reason stores none.
|
|
562
|
+
- **Config:** a `.mutineer.yml` `ignore:` list. A bare mutant id still works. A mapping is `id` plus an optional `reason`. Mutineer 1.7 is the first version that reads a mapping. An older version matches no mutant to a mapping and does not warn, so the mutant it names is not ignored. Add a mapping only after every gate that reads the file runs 1.7 or later. In the Action, the `version:` input sets the version, or your `Gemfile.lock` does when you set `use-bundler: true`. In your own CI job, your `Gemfile.lock` sets it. The reason shows in the same reports as an inline reason. Each survivor's `id` is printed in the JSON report.
|
|
563
|
+
|
|
564
|
+
`mutineer triage REPORT.json --reason TEXT --all` prints one mapping entry per survivor. `--id ID` prints one entry. Repeat `--id` for several. The command does not edit files. Paste the lines under `ignore:`. They are mappings, so every gate that reads the file must run 1.7 or later (see **Config** above).
|
|
565
|
+
|
|
566
|
+
Built-in skips are listed on the [equivalent skips](https://davidteren.github.io/mutineer/equivalent-skips.html) page. There is no switch to turn them off.
|
|
271
567
|
|
|
272
568
|
Suppressed mutants are excluded from the score (so 100% becomes reachable).
|
|
273
569
|
|
|
@@ -300,14 +596,18 @@ working until 2.0, with a warning:
|
|
|
300
596
|
hid mutants you did not mean to ignore. The warning says so. Keep only the ids for the mutant you meant to
|
|
301
597
|
ignore, not all of them. The list covers only the sources and operators in
|
|
302
598
|
that run, so run over every source with every operator set you use (for
|
|
303
|
-
example your Tier-2 `--operators`) for the full list.
|
|
599
|
+
example your Tier-2 `--operators`) for the full list. Or run
|
|
600
|
+
`mutineer migrate` on those sources. It writes every new id. Delete any id
|
|
601
|
+
you did not mean to ignore. See [Migrate old ignore ids](https://github.com/davidteren/mutineer#migrate-old-ignore-ids).
|
|
304
602
|
- **`--baseline`** An old baseline still matches: a survivor matches a stored
|
|
305
603
|
one with the same old id in the same file. A stored file that is an absolute
|
|
306
604
|
path outside the project root (a baseline written on another machine)
|
|
307
605
|
matches on the old id alone. The run tells you to
|
|
308
|
-
|
|
309
|
-
|
|
310
|
-
|
|
606
|
+
generate a new baseline with `--format json`. `migrate` does not rewrite a
|
|
607
|
+
baseline. Generate the new file only after every gate
|
|
608
|
+
that reads it runs 1.3 or later. In the Action, the `version:` input sets
|
|
609
|
+
the version, or your `Gemfile.lock` does when you set `use-bundler: true`.
|
|
610
|
+
In your own CI job, your `Gemfile.lock` sets it. An older version treats every new-format survivor as new.
|
|
311
611
|
|
|
312
612
|
The JSON report's `summary.id_format` is `2` for the new format.
|
|
313
613
|
`summary.legacy_id_matches.ignore` counts the old-format ignore entries a run
|
|
@@ -490,6 +790,7 @@ config file accepts these keys:
|
|
|
490
790
|
| `capture_timeout` | A positive integer; the coverage-capture time limit in seconds (default 120) |
|
|
491
791
|
| `cache_dir` | The coverage cache directory (default `.mutineer`) |
|
|
492
792
|
| `matrix` | `true` or `false`; runs every covering test for each mutant and adds the kill matrix to the report |
|
|
793
|
+
| `allow_empty` | `true` or `false`; a run with no mutants is expected, so it does not warn |
|
|
493
794
|
|
|
494
795
|
Invalid values for known scalar keys, and a blank `operators` list, exit 2
|
|
495
796
|
with a message naming the file and key. An unknown operator name warns and
|
|
@@ -499,9 +800,9 @@ is skipped. If none of the names are known, the run exits 2. An empty
|
|
|
499
800
|
`threshold` and the CLI-only `--baseline-epsilon` use plain decimals such as `90`
|
|
500
801
|
or `0.5`, not `+2`, `1e2`, or `1_0`. String options such as `only` and `baseline`
|
|
501
802
|
cannot be null or boolean. A blank `since` is invalid; use `since: false` to turn
|
|
502
|
-
scoping off. Unknown keys warn and are ignored. Unknown operator names warn and are skipped, and the run exits 2 when none remain. `--operators` replaces a blank or unknown file list.
|
|
803
|
+
scoping off. Unknown keys warn and are ignored. Unknown operator names warn and are skipped, and the run exits 2 when none remain. Both warnings suggest the closest valid name, and both become errors in Mutineer 2.0. `--operators` replaces a blank or unknown file list.
|
|
503
804
|
|
|
504
|
-
`format`, `strategy`, `output`, `baseline_epsilon`, and `
|
|
805
|
+
`format`, `strategy`, `output`, `baseline_epsilon`, `dry_run`, and `coverage_from` are CLI-only.
|
|
505
806
|
For JSON output, use `--format json`, not a `format:` config key. To select RSpec
|
|
506
807
|
in the file, add `framework: rspec`.
|
|
507
808
|
|
|
@@ -535,7 +836,7 @@ no code with Mutant.
|
|
|
535
836
|
| Test frameworks | Minitest and RSpec, in one gem | RSpec, Minitest and Test::Unit, one integration gem each |
|
|
536
837
|
| What you mutate | Files (`mutineer run lib/foo.rb --test test/foo_test.rb`), narrowed to one method with `--only 'Foo#bar'` | Subjects named by expression (`mutant run 'Foo#bar'`, `'Foo*'`) |
|
|
537
838
|
| Which tests run | The test files whose coverage reaches the mutated line | The tests that declare the subject (an RSpec description or a Minitest `cover`), or the tests that ran it in a per-test coverage recording |
|
|
538
|
-
| Operators |
|
|
839
|
+
| Operators | 21 (5 by default, 16 opt-in) | A larger set; the default `light` set applies almost all of it (`full` adds `#==` to `#eql?`) |
|
|
539
840
|
| CI gating | `--threshold`, `--baseline` deltas, `--since`, and a GitHub Action | Incremental mode (`--since`) and a recorded session history |
|
|
540
841
|
| Machine-readable output | Versioned JSON report schema | Session JSON schema |
|
|
541
842
|
|
|
@@ -0,0 +1,53 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Mutineer
|
|
4
|
+
# Lets a test helper start Coverage in a forked mutant child (#248).
|
|
5
|
+
#
|
|
6
|
+
# Once the coverage map is built, the parent suspends Coverage so mutant
|
|
7
|
+
# children do not pay for it (#228). A suspended Coverage reports
|
|
8
|
+
# `running? == false`, so a test helper that starts it when it is off
|
|
9
|
+
# (SimpleCov's `SimpleCov.start`) calls `Coverage.start`, which raises
|
|
10
|
+
# "coverage measurement is already setup" while Coverage is suspended. Every
|
|
11
|
+
# mutant then errors before a test runs.
|
|
12
|
+
#
|
|
13
|
+
# In the child, `Coverage.start` and `Coverage.setup` first stop the
|
|
14
|
+
# suspended Coverage, then run as usual. Stopping builds and drops the whole
|
|
15
|
+
# result, which walks the heap, so only a child whose helper starts Coverage
|
|
16
|
+
# pays for it. Only files loaded after that start are measured. The parent's
|
|
17
|
+
# Coverage is untouched.
|
|
18
|
+
#
|
|
19
|
+
# Stdlib-only, so the app-side daemon can load it.
|
|
20
|
+
module ChildCoverage
|
|
21
|
+
# Prepended to Coverage's singleton class in the child: stops a suspended
|
|
22
|
+
# Coverage before a start or setup.
|
|
23
|
+
module StopBeforeStart
|
|
24
|
+
# Stops a suspended Coverage, then sets it up.
|
|
25
|
+
#
|
|
26
|
+
# @return [nil]
|
|
27
|
+
def setup(...)
|
|
28
|
+
result(stop: true, clear: true) if state == :suspended
|
|
29
|
+
super
|
|
30
|
+
end
|
|
31
|
+
|
|
32
|
+
# Stops a suspended Coverage, then starts it.
|
|
33
|
+
#
|
|
34
|
+
# @return [nil]
|
|
35
|
+
def start(...)
|
|
36
|
+
result(stop: true, clear: true) if state == :suspended
|
|
37
|
+
super
|
|
38
|
+
end
|
|
39
|
+
end
|
|
40
|
+
|
|
41
|
+
# Makes a later `Coverage.start` or `Coverage.setup` work when Coverage is
|
|
42
|
+
# suspended. A running or idle Coverage is left alone.
|
|
43
|
+
#
|
|
44
|
+
# Call it only in a forked child, before any test loads.
|
|
45
|
+
#
|
|
46
|
+
# @return [void]
|
|
47
|
+
def self.allow_start
|
|
48
|
+
return unless defined?(::Coverage) && ::Coverage.state == :suspended
|
|
49
|
+
|
|
50
|
+
::Coverage.singleton_class.prepend(StopBeforeStart)
|
|
51
|
+
end
|
|
52
|
+
end
|
|
53
|
+
end
|