mutineer 1.7.0 → 1.8.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (45) hide show
  1. checksums.yaml +4 -4
  2. data/CHANGELOG.md +432 -0
  3. data/README.md +157 -747
  4. data/lib/mutineer/baseline.rb +1 -1
  5. data/lib/mutineer/changed_lines.rb +2 -1
  6. data/lib/mutineer/cli.rb +119 -27
  7. data/lib/mutineer/concern_blocks.rb +168 -5
  8. data/lib/mutineer/config.rb +6 -0
  9. data/lib/mutineer/coverage_map.rb +92 -17
  10. data/lib/mutineer/daemon_backend.rb +75 -32
  11. data/lib/mutineer/daemon_client.rb +50 -17
  12. data/lib/mutineer/daemon_server.rb +92 -40
  13. data/lib/mutineer/external_backend.rb +150 -17
  14. data/lib/mutineer/isolation.rb +67 -25
  15. data/lib/mutineer/job_plan.rb +117 -33
  16. data/lib/mutineer/kill_channel.rb +24 -14
  17. data/lib/mutineer/migrate.rb +10 -8
  18. data/lib/mutineer/minitest_integration/in_test_database_error.rb +184 -0
  19. data/lib/mutineer/minitest_integration/kill_recorder.rb +25 -17
  20. data/lib/mutineer/minitest_integration.rb +44 -14
  21. data/lib/mutineer/mutators/array_literal.rb +4 -2
  22. data/lib/mutineer/mutators/base.rb +22 -0
  23. data/lib/mutineer/mutators/call_to_receiver.rb +20 -7
  24. data/lib/mutineer/mutators/chain_link.rb +6 -6
  25. data/lib/mutineer/mutators/condition_forcing.rb +6 -3
  26. data/lib/mutineer/mutators/operand_removal.rb +4 -2
  27. data/lib/mutineer/mutators/range_literal.rb +105 -4
  28. data/lib/mutineer/mutators/return_nil.rb +1 -0
  29. data/lib/mutineer/orphan_guard.rb +18 -6
  30. data/lib/mutineer/pairing.rb +62 -1
  31. data/lib/mutineer/project.rb +226 -12
  32. data/lib/mutineer/rails_worker_db.rb +200 -16
  33. data/lib/mutineer/redaction.rb +112 -11
  34. data/lib/mutineer/reporter.rb +109 -21
  35. data/lib/mutineer/result.rb +33 -5
  36. data/lib/mutineer/runner.rb +126 -18
  37. data/lib/mutineer/statement_lines.rb +14 -8
  38. data/lib/mutineer/subject.rb +13 -5
  39. data/lib/mutineer/test_runners/minitest.rb +2 -1
  40. data/lib/mutineer/test_runners.rb +4 -2
  41. data/lib/mutineer/triage.rb +1 -1
  42. data/lib/mutineer/version.rb +1 -1
  43. data/lib/mutineer/worker_pool.rb +2 -1
  44. data/skills/mutineer/SKILL.md +84 -0
  45. metadata +5 -1
data/README.md CHANGED
@@ -6,51 +6,25 @@
6
6
  [![GitHub Marketplace](https://img.shields.io/badge/Marketplace-Mutineer%20Ruby-2da44e?logo=githubactions&logoColor=white)](https://github.com/marketplace/actions/mutineer-ruby)
7
7
  [![Socket](https://img.shields.io/badge/Socket-security%20report-0a66c2?logo=socket&logoColor=white)](https://socket.dev/rubygems/package/mutineer)
8
8
 
9
- A clean-room mutation-testing tool for Ruby. Mutineer mutates your source one
10
- change at a time, runs your test suite (Minitest or RSpec) against each mutant, and reports the
11
- ones your tests failed to catch — the gaps where your suite isn't actually
12
- testing anything.
13
-
14
- "Clean-room" means Mutineer was written from scratch. It contains no code from
15
- [Mutant](https://github.com/mbj/mutant) or any other mutation-testing tool.
16
- Mutineer's own code is licensed under MIT. See
17
- [Mutineer and Mutant](https://github.com/davidteren/mutineer#mutineer-and-mutant)
18
- for how the two tools differ.
19
-
20
- - **Prism + stdlib only** — zero runtime dependencies (Ruby ≥ 3.4).
21
- - **One mutation per mutant**, validity-checked by re-parsing.
22
- - **Fork-isolated**, parallel execution (Linux + macOS).
23
- - **Coverage-guided** — each mutant runs only the test files that cover its line.
24
- - **Stops at the first failing test** — in-process runs (not `--daemon` or
25
- `--test-command`) stop a mutant's test run at the first failure, unless
26
- `--matrix` asks for every covering test. The source's paired test files run
27
- first, then the rest from fastest to slowest in doubling steps (under 1 s,
28
- 1 to 3 s, 3 to 7 s, and so on); files in one step run in path order.
29
-
30
- 📖 **[davidteren.github.io/mutineer →](https://davidteren.github.io/mutineer/)** — overview, operators, and usage.
31
-
32
- New to mutation testing? Read
33
- [What is mutation testing?](https://github.com/davidteren/mutineer#what-is-mutation-testing)
34
- first. It takes five minutes and uses one small example.
35
-
36
- Try it on one file and its test:
9
+ **Find bugs your tests would miss.** Mutineer changes your Ruby code and runs your Minitest or RSpec tests.
10
+ Changes that still pass reveal possible test gaps. This is **mutation testing**.
37
11
 
38
- ```sh
39
- gem install mutineer
40
- mutineer run lib/calculator.rb --test test/calculator_test.rb
41
- ```
12
+ **Features**
42
13
 
43
- ## Is it production-ready?
14
+ - **Minitest and RSpec.** Detected from test filenames; override with `--framework`.
15
+ - **Rails support.** `--rails` boots once. With `--daemon`, each Minitest worker gets its own SQLite file, PostgreSQL, or MySQL 8 test database.
16
+ - **Focused tests.** Runs tests that reach each changed line, likely and fast tests first. Stops at the first failure by default.
17
+ - **PR checks.** `--since origin/main` checks changed lines; `--baseline` gates new survivors, and score drops on full runs. When tests change, run a full scan (`--no-since`).
18
+ - **CI and agents.** GitHub Action, versioned JSON, stable mutant ids, clear exit codes, and zero runtime gem dependencies.
19
+
20
+ 📖 **[davidteren.github.io/mutineer →](https://davidteren.github.io/mutineer/)**: overview, operators, and usage.
44
21
 
45
- Read this section before you gate an app on Mutineer.
46
- Use the Status column to decide.
47
- Adopt a row whose Status says supported or serial only.
48
- Not supported in the Parallelism column means `--daemon` cannot run that row.
22
+ ## Is it production-ready?
49
23
 
50
- Mutineer itself needs Ruby 3.4 or later.
51
- In-process runs fork, so use Linux or macOS.
52
- A plain Ruby project with no database is supported.
53
- It runs in parallel on those systems.
24
+ Check your database and framework before adding a CI gate.
25
+ Mutineer requires Ruby 3.4 or later. In-process runs require Linux or macOS for forks.
26
+ Plain Ruby projects without databases support parallel runs.
27
+ In the Status column, `supported` means parallel runs work and `serial only` runs one mutant at a time.
54
28
 
55
29
  <!-- contract:support-matrix -->
56
30
  | Ruby | Database | Framework | Parallelism | Status |
@@ -62,60 +36,58 @@ It runs in parallel on those systems.
62
36
  | 3.4 or later | MySQL 8 | Minitest | supported with `--daemon` | supported |
63
37
  | 3.4 or later | MySQL 8 | RSpec | not supported | serial only |
64
38
  | 3.4 or later | MariaDB | Minitest | not tested | not tested |
39
+ | 3.4 or later | MySQL 9 or later | Minitest | not tested | not tested |
65
40
  | 3.4 or later | SQLite in memory | Minitest | not supported | serial only |
66
41
  | 3.4 or later | SQLite in memory | RSpec | not supported | serial only |
67
42
  | 3.4 or later | more than one database | Minitest | one worker | serial only |
43
+ | 3.4 or later | more than one database, Rails 7.1 or later, the extra ones in `shared_databases` | Minitest | supported with `--daemon` | supported, unless the test environment registers a pool for a shared database |
68
44
  | 3.4 or later | more than one database | RSpec | not supported | serial only |
69
45
 
70
- This matrix is for a Rails app on Ruby 3.4 or later.
71
- `supported` means you can run that cell in parallel.
72
- `serial only` means the safe run uses one worker.
73
- `not supported` means `--daemon` cannot run that cell.
74
- `not tested` means no CI job runs that cell.
75
- SQLite file means a database file, not `:memory:`.
76
- Minitest on that file can use `--daemon`.
77
- RSpec on that file is serial only.
78
- PostgreSQL with Minitest can use `--daemon`.
79
- PostgreSQL with RSpec is serial only.
80
- MySQL 8 with Minitest can use `--daemon`.
81
- MariaDB is not tested.
82
- MySQL 5.7 is not supported.
83
- A `:memory:` database is serial only.
84
- An app with more than one database runs one worker.
85
- `--daemon` isolates only the primary connection.
86
- `--rails` without `--daemon` is serial on every row.
87
- A Postgres worker database is copied once, at the start of the run.
88
- A MySQL 8 worker database is created once, at the start of the run, and the rows are copied then.
89
- A test that commits rows outside a transaction leaves them in that worker database.
90
- The next mutant on that worker sees those rows.
91
- Rails parallel tests have the same ceiling.
46
+ This matrix is for a Rails app on Ruby 3.4 or later. A plain Ruby project
47
+ with no database runs in parallel with either framework.
48
+ `not supported` means `--daemon` cannot run that cell; `not tested` means no CI
49
+ job runs it. SQLite file means a database file, not `:memory:`.
50
+
51
+ - `--rails` without `--daemon` is serial on every row. `--daemon` is Minitest only.
52
+ - MySQL older than 8.0 stops `--daemon`. MariaDB and MySQL 9 or later warn and run.
53
+ - An app with more than one database runs one worker: `--daemon` isolates only the primary connection.
54
+ On Rails 7.1 or later, list the databases your tests never use in `shared_databases`, and the workers
55
+ run in parallel. CI tests that row on SQLite. See [More than one database](https://github.com/davidteren/mutineer/blob/main/docs/rails-setup.md#more-than-one-database).
56
+ - Postgres worker databases are copied once at the start; MySQL 8 worker databases are created then and the rows copied.
57
+ - A test that commits rows outside a transaction leaves them in that worker's
58
+ database, and the next mutant on that worker sees them. Rails parallel tests have the same limit.
59
+ - RSpec: an example that fails with a database connection error scores
60
+ `killed`, not `errored`, so the score can read too high
61
+ ([#347](https://github.com/davidteren/mutineer/issues/347)).
92
62
 
93
63
  Mutineer sets `PARALLEL_WORKERS=1`, so a `parallelize` call in `test_helper.rb` starts no Rails workers during a mutant run.
94
- See [Rails parallel tests, Spring, Bootsnap, and Redis](https://github.com/davidteren/mutineer#rails-parallel-tests-spring-bootsnap-and-redis).
64
+ See [Rails parallel tests, Spring, Bootsnap, and Redis](https://github.com/davidteren/mutineer/blob/main/docs/rails-setup.md#rails-parallel-tests-spring-bootsnap-and-redis).
95
65
 
96
66
  <!-- /contract:support-matrix -->
97
67
 
98
- An app on an older Ruby uses `--test-command`.
99
- Mutineer still runs on Ruby 3.4 or later.
100
- That path is serial.
101
- It does not narrow tests by coverage.
102
- A crash in the command can count as a kill.
68
+ Older Ruby apps use `--test-command`; Mutineer still requires Ruby 3.4 or later.
69
+ This path uses one Mutineer worker and no coverage-based test selection by default; a wrapper can re-enable Rails test workers by setting `PARALLEL_WORKERS`.
103
70
 
104
- What a release will not break is in
105
- [STABILITY.md](https://github.com/davidteren/mutineer/blob/main/STABILITY.md).
71
+ | Subprocess result | Score |
72
+ |---|---|
73
+ | Exit 1, normally | `killed` |
74
+ | SimpleCov gate exit with a failing test | `killed` |
75
+ | Other exit code | `error` |
76
+ | Crash before a test fails | Can score `error` |
77
+ | Only a SimpleCov gate fails, with its gate message and a passing summary | `survived` |
106
78
 
107
- ## What is mutation testing?
79
+ See [Apps on Ruby < 3.4](https://github.com/davidteren/mutineer/blob/main/docs/rails-setup.md#apps-on-ruby--34)
80
+ and release guarantees in [STABILITY.md](https://github.com/davidteren/mutineer/blob/main/STABILITY.md).
108
81
 
109
- Your tests pass. That shows the code does what the tests check. It does not
110
- show that the tests check enough. Mutation testing finds out by changing your
111
- code on purpose and seeing whether any test notices.
82
+ ## What is mutation testing?
112
83
 
113
- Each change is small, such as `>=` to `>` or `&&` to `||`. One changed copy of
114
- your code is a **mutant**. Mutineer runs your tests against each mutant:
84
+ Mutation testing changes code to check whether your tests notice.
85
+ Each changed copy is a **mutant**, such as replacing `>=` with `>` or `&&` with `||`.
115
86
 
116
- - When a test fails, the tests **killed** the mutant. Good: they noticed.
117
- - When every test still passes, the mutant **survived**. Your tests ran that
118
- code but did not check what it does.
87
+ | Result | Meaning |
88
+ |---|---|
89
+ | **Killed** | A test failed and caught the change. |
90
+ | **Survived** | All tests passed. Inspect the change for a test gap. |
119
91
 
120
92
  ### A small example
121
93
 
@@ -129,36 +101,9 @@ class Discount
129
101
  end
130
102
  ```
131
103
 
132
- ```ruby
133
- # test/discount_test.rb
134
- require "minitest/autorun"
135
- require_relative "../lib/discount"
136
-
137
- class DiscountTest < Minitest::Test
138
- def test_bulk_order_gets_ten_percent
139
- assert_equal 0.1, Discount.new.rate(50)
140
- end
141
-
142
- def test_small_order_gets_nothing
143
- assert_equal 0, Discount.new.rate(1)
144
- end
145
- end
146
- ```
147
-
148
- Both tests pass, and every line runs, so line coverage is 100%. Run Mutineer:
149
-
150
- ```sh
151
- mutineer run lib/discount.rb --test test/discount_test.rb
152
- ```
153
-
154
- Part of the report:
104
+ Tests check quantities 1 and 50: 100% line coverage. Mutineer changes `>=` to `>`; both tests pass:
155
105
 
156
106
  ```text
157
- Mutation score: 50.0% (killed / (killed + survived); ...)
158
-
159
- Surviving Mutants
160
- -----------------
161
-
162
107
  lib/discount.rb
163
108
  Discount#rate (discount.rb:3)
164
109
  Operator: comparison (>= -> >)
@@ -166,150 +111,95 @@ lib/discount.rb
166
111
  + return 0.1 if qty > 10
167
112
  ```
168
113
 
169
- Mutineer changed `>=` to `>`, and both tests still passed. No test checks an
170
- order of exactly 10, so a bug at that boundary would ship. Add that test:
171
-
172
- ```ruby
173
- def test_ten_is_the_first_bulk_quantity
174
- assert_equal 0.1, Discount.new.rate(10)
175
- end
176
- ```
177
-
178
- Run Mutineer again. Both mutants are killed, and the score is 100%.
114
+ No test checks exactly 10. Testing 10 kills the mutant. Run the example, tests and report in [Mutation testing, explained](https://davidteren.github.io/mutineer/learn.html).
179
115
 
180
116
  ### Coverage and mutation testing
181
117
 
182
- Line coverage tells you which lines ran during the tests. Mutation testing tells
183
- you whether the tests would fail if those lines were wrong. A test with no
184
- assertion still adds coverage, but it kills a mutant only when the change makes
185
- the code raise an error. That gap is common in
186
- tests written in a hurry, and in tests written by AI coding agents.
187
-
188
- ### What each result means
189
-
190
- | Report row | What happened | In the score? |
191
- | --- | --- | --- |
192
- | Killed | A test failed against the mutant. | Yes |
193
- | Survived | Every test passed against the mutant. Look at it. | Yes |
194
- | No coverage | No test runs that line. Write a test for it first. | No |
195
- | Timeout | The tests ran past `--timeout`. | No |
196
- | Errored | The test run crashed, so there is no verdict. | No |
197
- | Ignored | You marked the mutant as equivalent (see below). | No |
198
- | Skipped | The change made invalid Ruby, so it did not run. | No |
199
-
200
- Uncapturable, Unplaceable, and Ran at load are rarer. The
201
- [JSON schema](https://davidteren.github.io/mutineer/json-schema.html#summary)
202
- explains each one.
203
-
204
- The **mutation score** is `killed / (killed + survived)`. The other rows do not
205
- change the score. When nothing can be scored, the score is empty (`null` in
206
- JSON), not 0. Under `--threshold`, too many Timeout or Errored mutants still
207
- fail the run; see [Timeouts and the score](https://github.com/davidteren/mutineer#timeouts-and-the-score).
208
-
209
- ### What to do with a survivor
210
-
211
- 1. Read the diff. Ask what behaviour the change breaks.
212
- 2. Write or strengthen a test that fails under the change. Run again.
213
- 3. Sometimes the change breaks nothing. For example, when `x` is a number,
214
- swapping `x * 1` for `x / 1` gives the same result. That mutant is **equivalent**, and no test can
215
- kill it. Suppress it with a reason (see
216
- [Suppressing equivalent mutants](https://github.com/davidteren/mutineer#suppressing-equivalent-mutants)).
217
-
218
- ### Where to start
219
-
220
- Each mutant runs part of your suite again, so a run takes longer than one test
221
- run. Start with one file and its test, as above. Use `--dry-run` to list the
222
- mutants without running tests. When the report makes sense, widen the run to a
223
- folder, then gate pull requests on the lines they change with `--since`. On a
224
- Rails app, the [Rails rollout guide](https://davidteren.github.io/mutineer/rails.html)
225
- walks through each stage.
226
-
227
- ## Install
118
+ | Check | What it shows |
119
+ |---|---|
120
+ | Line coverage | Which lines ran |
121
+ | Mutation testing | Whether tests detect changes to those lines |
122
+
123
+ Tests without assertions add coverage but kill mutants only when changes cause errors.
124
+ This gap is common in rushed tests and tests written by AI agents.
125
+
126
+ ## How it works
127
+
128
+ 1. **Parse.** [Prism](https://github.com/ruby/prism), included in Ruby 3.4, parses source files. Mutineer uses only Prism and the standard library.
129
+ 2. **Mutate.** Operators make one change per mutant. Mutineer parses each result again and skips invalid Ruby.
130
+ 3. **Map coverage and check unchanged tests.** Each test file runs with coverage to map lines to tests.
131
+ `.mutineer/coverage.json` caches the map; source changes rebuild it. Tests must pass alone and
132
+ together, including with a cached map. A failure stops the run.
133
+ 4. **Run each mutant in a fork.** The parent loads code once, including the app with `--rails`.
134
+ Children run in parallel on Linux and macOS.
135
+ Each loads the changed file (`reload`) or replaces the changed method (`redefine`).
136
+ Only covering test files run: paired tests first, then fastest to slowest. In-process runs stop at the first failure unless `--matrix` requests every covering test.
137
+ 5. **Score.** A failing test kills the mutant; all tests passing means it survived.
138
+ Score: `killed / (killed + survived)`. In coverage-guided runs, timeouts, errors, and uncovered lines count as neither; `--test-command` has no coverage, so a mutant no test reaches survives.
139
+
140
+ | Alternative backend | Changes to step 4 |
141
+ |---|---|
142
+ | `--daemon` | Boots Rails once in a persistent process; isolates each worker's test database |
143
+ | `--test-command` | Runs tests in another Ruby subprocess; writes mutants to the real file, then restores it |
144
+
145
+ | At a glance | |
146
+ |---|---|
147
+ | Parser | Prism, with no runtime gem dependencies |
148
+ | Ruby | 3.4 or later; older apps use `--test-command` |
149
+ | Operators | 21 (5 by default, 16 opt-in) |
150
+ | Isolation | One fork per mutant on Linux and macOS |
151
+ | Report formats | human, JSON (versioned schema), HTML |
152
+
153
+ ## Quick start
154
+
155
+ Plain Ruby:
228
156
 
229
157
  ```sh
230
158
  gem install mutineer
159
+ mutineer run lib/discount.rb --test test/discount_test.rb
231
160
  ```
232
161
 
233
- Or in a Gemfile:
162
+ Rails: add to your `Gemfile` test group:
234
163
 
235
164
  ```ruby
236
- gem "mutineer", group: :test
165
+ gem "mutineer", group: :test, require: false
237
166
  ```
238
167
 
239
- ## Usage
168
+ Install, create `.mutineer.yml`, and check one model:
240
169
 
241
170
  ```sh
242
- mutineer run <source...> --test <test> [--test <test>...] [options]
243
- mutineer init [--rails] [--force]
244
- mutineer triage REPORT.json --reason TEXT (--id ID ... | --all)
245
- mutineer migrate <source...> [--dry-run] [--operators LIST]
171
+ bundle install
172
+ RAILS_ENV=test bundle exec mutineer init --rails
173
+ RAILS_ENV=test bundle exec mutineer run app/models/order.rb --rails
246
174
  ```
247
175
 
248
- - `run` mutates the sources, runs the tests, and prints the report.
249
- - `init` writes a commented `.mutineer.yml` and prints a first run command.
250
- - `triage` prints `ignore:` entries for survivors in a JSON report. It does not
251
- edit files. See [Suppressing equivalent mutants](https://github.com/davidteren/mutineer#suppressing-equivalent-mutants).
252
- - `migrate` rewrites old ignore ids. See [Migrate old ignore ids](https://github.com/davidteren/mutineer#migrate-old-ignore-ids).
253
-
254
- Mutate `lib/calculator.rb`, checking it against its test, and fail CI if the
255
- mutation score drops below 90%:
256
-
257
- ```sh
258
- mutineer run lib/calculator.rb --test test/calculator_test.rb --threshold 90
259
- ```
176
+ Without `--test`, `app/models/order.rb` pairs with `test/models/order_test.rb` or `spec/models/order_spec.rb`.
177
+ See [Quick start](https://github.com/davidteren/mutineer/blob/main/docs/quick-start.md).
260
178
 
261
- ### Migrate old ignore ids
179
+ ## Usage
262
180
 
263
181
  ```sh
182
+ mutineer run <source...> [--test <test>...] [options]
183
+ mutineer init [--rails] [--force]
184
+ mutineer triage REPORT.json --reason TEXT (--id ID ... | --all)
264
185
  mutineer migrate <source...> [--dry-run] [--operators LIST]
265
186
  ```
266
187
 
267
- `migrate` rewrites old-format ids in `.mutineer.yml` `ignore:` to the current
268
- ids. It keeps comments and every other line. When one old id matches several
269
- mutants, it writes every new id. That keeps the 1.x match. An id that matches
270
- nothing stays, and the command exits 1. In an `{ id:, reason: }` entry, the
271
- id changes and the reason stays. One entry holds one id, so an old id there
272
- that now matches several mutants stays as written: the command names the new
273
- ids, exits 1, and you split the entry by hand. A block scalar under `ignore:`
274
- is not supported: the command exits 2 and does not change the file.
275
- `--dry-run` prints each change and writes nothing.
276
-
277
- Pass the same sources and `--operators` you use for `run`. The command runs
278
- no tests. It does not rewrite a baseline. Generate a new baseline with
279
- `--format json`.
280
-
281
188
  ### Options
282
189
 
190
+ Full flags, defaults, and `.mutineer.yml` keys: [CLI reference](https://github.com/davidteren/mutineer/blob/main/docs/cli.md).
191
+
283
192
  | Flag | Meaning |
284
193
  |------|---------|
285
- | `--test FILE` | Test file covering the sources; one file per flag, so repeat it for each (`--test a_test.rb --test b_test.rb`) |
286
- | `--operators LIST` | Comma-separated operator names (default: the Tier-1 set) |
194
+ | `--test FILE` | Repeat per test file; omit to pair tests by convention |
287
195
  | `--threshold FLOAT` | Exit 1 when the score is below FLOAT, or when nothing could be scored and something broke, or more than one mutant produced no verdict and they exceed 10% of those attempted (default: 0 = off) |
288
- | `--only NAME` | Restrict to one fully-qualified subject, e.g. `Calculator#add` |
289
- | `--framework NAME` | `minitest` (default) or `rspec`; auto-detected as rspec when most `--test` files end in `_spec.rb` |
290
- | `--since REF` | Only mutate lines changed since git `REF` (e.g. `origin/main`). Untracked files are scored in full unless Git ignores them. The report lists only suppressed mutants on changed lines, too. |
291
- | `--no-since` | Disable diff scoping; a typed no beats a `.mutineer.yml` `since:` key |
292
- | `--baseline FILE` | Compare against a prior `--format json` run; exit 1 on new survivors / score drop (score drop is skipped under `--since`, whose score covers a different denominator; see [CI](https://github.com/davidteren/mutineer#ci-gating)) |
293
- | `--baseline-epsilon FLOAT` | Score-drop tolerance for `--baseline` (default: 0) |
294
- | `--jobs N` | Parallel worker count (default: processor count); forced to `1` by `--test-command`, `--fail-fast`, or `--rails` without `--daemon` |
295
- | `--boot FILE` | Require an app entry point once before forking; select at least one test file |
296
- | `--rails` | Boot `config/environment` and reconnect ActiveRecord per fork; without `--daemon`, defaults to `redefine` and runs serially |
297
- | `--verbose` | Surface the real error when a fork capture fails (alias `--debug`) |
298
- | `--strategy NAME` | Mutation application: `reload` whole-file (default) or `redefine` surgical (`7a`/`7b` accepted as deprecated aliases) |
299
- | `--timeout SECONDS` | Per-mutant time limit for in-process runs, in whole seconds (default: 10). A mutant whose tests run longer is a `timeout`; see [Timeouts and the score](https://github.com/davidteren/mutineer#timeouts-and-the-score). `--daemon` and `--test-command` keep their own limits |
300
- | `--capture-timeout SECONDS` | Time limit for each in-process coverage-capture subprocess and for the clean run of the unmutated tests, in whole seconds (default: 120). A suite slower than this stops the run as not green. `--daemon` uses it for its coverage capture; `--test-command` ignores it and says so |
301
- | `--cache-dir DIR` | Directory for the coverage cache (default: `.mutineer`). Give two runs from the same project root different directories so they do not share `coverage.json`. `--test-command` builds no cache, so it ignores it and says so |
302
- | `--coverage-from FILE` | Reuse the `coverage.json` of an earlier `--rails`/`--boot` run with the same tests over these sources or more, so parallel shards capture coverage once. The run skips the capture. It runs each test that covers one of its sources alone, because a mutant runs only those tests, then all tests once together, and writes its own cache to `--cache-dir`. A test that covers none of its sources does not run alone. A capture that does not fit (changed tests or sources, an owner file added or removed, a source it did not cover, a failed capture, another `PARALLEL_WORKERS`, a cache from before this flag) stops the run (exit 2). Under `--strategy redefine`, eager-load in both runs. It has no effect, and says so, without `--rails`/`--boot` or with `--daemon`, `--test-command` or `--dry-run` |
303
- | `--test-command CMD` | Run the suite as a subprocess in the app's own runtime (for apps on Ruby < 3.4); `CMD` must contain `%{files}`. See [Apps on Ruby < 3.4](https://github.com/davidteren/mutineer#apps-on-ruby--34) |
304
- | `--daemon` | Boot the app once in a persistent daemon and fork per mutant, with per-worker DB isolation so `--jobs N` is safe under Rails (needs `--rails`/`--boot`; not with `--test-command`). See [the daemon backend](https://github.com/davidteren/mutineer#faster-parallel-safe-rails-the---daemon-backend). `--no-daemon` beats a `.mutineer.yml` `daemon:` key. It keeps `--rails` serial, which is the 1.x default, and in 2.0 it opts out of the parallel default |
305
- | `--format human\|json\|html` | Report format (default: human; `html` is a self-contained file) |
306
- | `--output FILE` | Write the report to FILE instead of stdout |
307
- | `--dry-run` | List candidate mutations without executing (honors suppression) |
308
- | `--fail-fast` | Stop at the first surviving mutant (`--no-fail-fast` beats a `.mutineer.yml` `fail_fast:` key) |
309
- | `--allow-empty` | A run with no mutants is expected. Without it, an empty run warns that Mutineer 2.0 will fail it. A `--since` run whose changes hold no mutants does not warn: it reports that the changes hold nothing to test. A `--since` run with no mutants even before scoping still warns. `--no-allow-empty` beats a `.mutineer.yml` `allow_empty:` key |
310
- | `--matrix` | Run every covering test for each mutant and report the blind and redundant tests; the JSON report also lists each mutant's killers. In-process only; exits 2 with `--daemon`, `--test-command`, `--fail-fast` or `--dry-run`. RSpec needs 3.3 or later. `--no-matrix` beats a `.mutineer.yml` `matrix:` key. See [Kill matrix](https://github.com/davidteren/mutineer#kill-matrix) |
311
- | `--list-operators` | List available operators (default vs optional) and exit |
312
- | `--version`, `--help` | Print version / usage and exit |
196
+ | `--since REF` | Mutate only lines changed since git `REF`, such as `origin/main` |
197
+ | `--baseline FILE` | Exit 1 for new survivors, or score drops on full runs, against a prior JSON report from a full run |
198
+ | `--rails` | Boot `config/environment` once; serial without `--daemon` |
199
+ | `--[no-]daemon` | Enable or disable daemon; isolated worker databases, Minitest only |
200
+ | `--jobs N` | Worker count; default: one per CPU |
201
+ | `--format human\|json\|html` | Report format; default: human |
202
+ | `--dry-run` | List mutants without running tests |
313
203
 
314
204
  ### Exit codes
315
205
 
@@ -317,539 +207,59 @@ no tests. It does not rewrite a baseline. Generate a new baseline with
317
207
  | Code | Meaning |
318
208
  |------|---------|
319
209
  | `0` | Score ≥ threshold (or no gate) **and** no baseline regression. |
320
- | `1` | Score below `--threshold`, OR nothing could be scored and something broke, or more than one mutant produced no verdict and they exceed 10% of those attempted, OR a `--baseline` regression, OR a runtime error. |
321
- | `2` | Usage / invalid-flag error (mistyped flag, bad path, unreadable baseline). |
210
+ | `1` | Score below `--threshold`, OR nothing could be scored and something broke, or more than one mutant produced no verdict and they exceed 10% of those attempted, OR a `--baseline` regression, OR a runtime error, including a boot, source or `require:` file that raises while loading. |
211
+ | `2` | Usage / invalid-flag error, including a mistyped flag, a bad path (such as a missing `--boot` or `require:` file), or an unreadable baseline. |
322
212
  <!-- /contract:exit-codes -->
323
213
 
324
214
  ### Operators
325
215
 
326
- Run `mutineer --list-operators` to see them. Default (Tier 1): `arithmetic`,
327
- `comparison`, `boolean_connector`, `boolean_literal`, `statement_removal`.
328
- Available but off by default (Tier 2, enable via `--operators`): `return_nil`,
329
- `literal_mutation`, `condition_negation`, `string_literal`, `regex`,
330
- `collection_method`, `safe_navigation`, `range`, `negation_removal`, `chain_link`,
331
- `call_to_receiver`, `operand_removal`, `array_literal`, `condition_true`,
332
- `condition_false`, `operator_assignment`.
333
-
334
- ## Rails apps
335
-
336
- Rails code needs its environment booted before the suite runs, so point Mutineer
337
- at your app with `--rails` and run it inside the project's bundle:
338
-
339
- ```sh
340
- RAILS_ENV=test bundle exec mutineer run \
341
- app/models/order.rb --test test/models/order_test.rb --rails
342
- ```
343
-
344
- `--rails` boots `config/environment` once in the parent process (every mutant
345
- then forks and inherits it), defaults `--strategy` to `redefine` without `--daemon`
346
- (surgical — it avoids reloading files into the app tree; `--daemon` keeps `reload`), and reconnects ActiveRecord in each
347
- fork so the database connection is fork-safe. Use `--boot FILE` to boot a
348
- different entry point. Boot mode requires at least one `--test` file and is
349
- coverage-guided — each mutant runs only the test files that exercise its line
350
- (coverage is captured by forking the booted app, then cached).
351
-
352
- Some code runs while the app boots or a class loads, for example a class body
353
- that calls a method to build a constant, a `to_prepare` initializer, or a
354
- `--require` file that calls a source method. That run
355
- happens before the mutant is applied, and the forked test does not repeat it.
356
- A mutant on such a line is `ran_at_load`, not `survived` or `no_coverage`. It is
357
- left out of the score and does not fail `--threshold`; a kill still counts.
358
- This holds under both strategies: `reload` runs the mutated file's class body
359
- again, but not an initializer or another file's code, so a survivor on such a
360
- line is not trusted there either.
361
- Run those mutants with `--test-command`, which boots a fresh process per mutant,
362
- to get a verdict.
363
-
364
- Add Mutineer to your Gemfile's test group:
365
-
366
- ```ruby
367
- gem "mutineer", group: :test, require: false
368
- ```
369
-
370
- Start from one command. It writes `.mutineer.yml` and prints the first run.
371
- It does not replace a file you already wrote. Pass `--force` to replace one.
372
-
373
- ```sh
374
- RAILS_ENV=test bundle exec mutineer init --rails
375
- ```
376
-
377
- The path from that first report to a CI gate is the
378
- [Rails rollout guide](https://davidteren.github.io/mutineer/rails.html).
379
-
380
- ### Rails parallel tests, Spring, Bootsnap, and Redis
381
-
382
- Mutineer sets `PARALLEL_WORKERS` to `1` for every run.
383
- Rails then runs each mutant's tests in one process.
384
- Your `parallelize(workers:)` call stays in `test_helper.rb`.
385
- It does not start Rails workers during a mutant run.
386
- If you already set `PARALLEL_WORKERS` to another value, Mutineer replaces it and prints one line.
387
- With `--test-command`, a script your command runs can set `PARALLEL_WORKERS` again.
388
- Its value wins, and the Rails workers start again.
389
- Remove that line from the script, or write `PARALLEL_WORKERS=${PARALLEL_WORKERS:-4}` so it keeps the 1 Mutineer passes.
390
-
391
- Mutineer does not run your `parallelize_setup` hooks.
392
- Rails registers its own database hook in that same list.
393
- Running the list would point the tests at `<base>-<n>` instead of Mutineer's database.
394
-
395
- `--test-command` also sets `DISABLE_SPRING=1`.
396
- Spring does not start for that command.
397
- The `--daemon` path loads `config/environment` itself, so Spring does not start there either.
398
- A wrapper that sets `DISABLE_SPRING` again wins, same as `PARALLEL_WORKERS`.
399
-
400
- Bootsnap checks its compile cache by file path, size, and modification time in whole seconds.
401
- When the size and the time match, it uses the cached code without reading the file content.
402
- The in-process and `--daemon` paths load each mutant from a new file, so the cache cannot mix them up.
403
- `--test-command` rewrites the source file in place.
404
- Two mutants of the same size, written in the same second, can then load the first one's cached code.
405
- To rule that out, set `DISABLE_BOOTSNAP_COMPILE_CACHE=1` on the Mutineer command.
406
- Bootsnap keeps its cache in `tmp/cache/bootsnap`, or under `$BOOTSNAP_CACHE_DIR` when that is set.
407
- If a run uses code that no longer matches your files, delete that folder and run again.
408
-
409
- `redis-client` (which `redis` 5 uses) opens a new connection after a fork on its own.
410
- You do not set anything in Mutineer for this.
411
- `redis` 4.x raises `Redis::InheritedError` instead.
412
- Open that connection after boot, or reconnect it in an after-fork hook.
413
-
414
- ### Faster, parallel-safe Rails (the `--daemon` backend)
415
-
416
- `--rails` boots your app once but runs mutants **serially** — parallel `--jobs`
417
- under Rails is unsafe, because every worker shares one test database and clobbers
418
- the others' fixtures. `--daemon` fixes both: it boots the app once in a persistent
419
- helper and forks per mutant, and gives **each parallel worker its own database**,
420
- so `--jobs N` is safe and its verdicts are proven identical to a serial run.
421
-
422
- ```sh
423
- RAILS_ENV=test bundle exec mutineer run \
424
- app/models/order.rb --test test/models/order_test.rb \
425
- --rails --daemon --jobs 4
426
- ```
427
-
428
- - **One boot, forked per mutant** — restores the shared-boot speed.
429
- - **Coverage-guided** — each mutant runs only its covering tests (like `--rails`);
430
- a mutant on an uncovered line is `no_coverage`, so the score stays comparable to
431
- the in-process `--rails` score.
432
- - **Safe `--jobs N`** — each worker routes to its own copy of the test database, so
433
- parallel verdicts equal serial (no fixture cross-talk). On first use, a
434
- worker's database is a copy of the test database after the app boots, so rows
435
- written by initializers or `--require` files are there, as in-process. When
436
- the copy's schema differs from `db/schema.rb` (another schema version or
437
- `schema_sha1` checksum, or no stored checksum, as in an empty or out-of-date
438
- test database, or one set up with a plain `load "db/schema.rb"`), the
439
- worker loads `db/schema.rb` over it, which drops the copied rows of the
440
- tables it defines.
441
- - **One backend at a time** — `--daemon` can't be combined with `--test-command`
442
- (choose one), and it needs an app to boot (`--rails` or `--boot`).
443
-
444
- Status: **SQLite**, **PostgreSQL**, and **MySQL 8** (Minitest). Each Postgres worker uses a
445
- database named `<base>-mutineer-<N>` (shortened when that would pass 63 bytes).
446
- Mutineer copies the test database into those names before the workers start,
447
- and leaves them in place when the run ends. Drop one with
448
- `DROP DATABASE "myapp_test-mutineer-0";` (use your test database name and slot).
449
- Each MySQL 8 worker uses a database named `<base>-mutineer-<N>` (shortened when that would pass 64 bytes).
450
- Mutineer creates that database, loads `schema.rb` or `structure.sql`, and copies rows before the workers start.
451
- It does not copy the database again after an error or a timeout.
452
- The worker databases stay in place when the run ends. The next run drops and rebuilds them.
453
- Triggers in the schema do not fire on the copied rows: in each worker database, Mutineer drops them for the copy and creates them again.
454
- The database user needs CREATE, DROP, TRIGGER, and the table and row privileges a schema load and a row copy use (such as ALTER, INDEX, REFERENCES, and INSERT) on the worker databases, plus SELECT on the test database.
455
- A trigger whose DEFINER is another account also needs SET_USER_ID.
456
- An app on `structure.sql` also needs the `mysql` client on `PATH`.
457
- Drop one with ``DROP DATABASE `myapp_test-mutineer-0`;`` (MySQL quotes the name with backticks).
458
- A second run on the same test database stops with `another Mutineer run is using <base>` until the first run ends.
459
- MySQL runs at most 256 workers.
460
- `mysql2` and `trilogy` are supported.
461
- MariaDB is not tested.
462
- MySQL 5.7 is not supported.
463
- A `:memory:` database is not supported on `--daemon`.
464
-
465
- ### Apps on Ruby < 3.4
466
-
467
- Mutineer's own process needs Ruby ≥ 3.4 (it parses with stdlib Prism), and the
468
- `--rails` path above boots your app *inside Mutineer's process* — so it can't run
469
- against an app pinned to an older Ruby (`ruby "3.1.6"` in the Gemfile), where the
470
- bundle rejects 3.4.
471
-
472
- `--test-command` decouples the two: Mutineer stays on ≥ 3.4, but your suite runs
473
- as a **subprocess in your app's own runtime** (whatever Ruby its bundle resolves
474
- to). Run Mutineer with a 3.4+ Ruby and hand it the command that runs your tests:
475
-
476
- ```sh
477
- RAILS_ENV=test mutineer run app/models/order.rb \
478
- --test test/models/order_test.rb \
479
- --test-command "bundle exec rails test %{files}"
480
- ```
481
-
482
- - **`%{files}`** is required; it expands to the `--test` paths as separate
483
- arguments (a path with a space stays one argument — there is no shell).
484
- - **Environment:** vars like `RAILS_ENV` / `DATABASE_URL` set on the Mutineer
485
- command are inherited. Mutineer **unsets** `BUNDLE_*`, `GEM_*`, `RUBY*`,
486
- `RBENV_VERSION`, `ASDF_RUBY_VERSION`, and `RBENV_DIR` in the child (so
487
- Mutineer's own Ruby cannot pin the suite), and drops version-manager
488
- **version bins** (e.g. `~/.rbenv/versions/3.4.x/bin`) from `PATH`, then
489
- prepends rbenv/asdf shims when a pin was scrubbed. Do not rely on
490
- `RBENV_VERSION=…` on the Mutineer command for the suite; use `.ruby-version`
491
- or a wrapper. Don't put `KEY=val` prefixes *inside* `--test-command` (no shell;
492
- that would be treated as the program name). Mutineer also sets
493
- `PARALLEL_WORKERS=1` and `DISABLE_SPRING=1` in the child and replaces a value
494
- you set (see [Rails parallel tests, Spring, Bootsnap, and Redis](https://github.com/davidteren/mutineer#rails-parallel-tests-spring-bootsnap-and-redis)).
495
-
496
- #### Under a version manager (rbenv / asdf / chruby)
497
-
498
- Automatic scrub targets **rbenv** and **asdf** (shims + version bins). **chruby**
499
- has no shims: Mutineer still strips `…/rubies/…/bin` so it cannot leave Mutineer's
500
- Ruby pinned, but you need a wrapper that sources chruby and selects the app
501
- version. If the smoke check still reports a **Ruby version mismatch**, wrap the
502
- suite. Example rbenv wrapper (`bin/mutineer-test` in the app):
503
-
504
- ```sh
505
- #!/usr/bin/env bash
506
- set -euo pipefail
507
- cd "$(dirname "$0")/.."
508
- unset GEM_HOME GEM_PATH RUBYLIB RUBYOPT BUNDLE_GEMFILE BUNDLE_BIN_PATH BUNDLER_VERSION
509
- export RBENV_VERSION="$(cat .ruby-version 2>/dev/null || true)"
510
- export RAILS_ENV="${RAILS_ENV:-test}"
511
- export PATH="${HOME}/.rbenv/shims:${PATH}"
512
- exec bundle exec rails test "$@"
513
- ```
514
-
515
- ```sh
516
- # Mutineer on 3.4+; suite on the app's Ruby via the wrapper
517
- mutineer run app/models/order.rb \
518
- --test test/models/order_test.rb \
519
- --test-command "bin/mutineer-test %{files}"
520
- ```
521
-
522
- Mutineer also surfaces a targeted smoke-check message when Bundler prints
523
- `RubyVersionMismatch`, instead of only blaming DB/migrations.
524
-
525
- Tradeoffs — this path is correct but not free:
526
-
527
- - **Slower:** your app re-boots for every mutant (no shared boot yet).
528
- - **No coverage narrowing:** every mutant runs the full `--test` set.
529
- The run is serial.
530
- A crash in the command can count as a kill.
531
- Uncovered mutants count as survivors.
532
- Do not compare a `--test-command` score with an in-process score.
533
- Exit codes 126 and 127 (the command could not start) are scored `error`.
534
- Mutineer prints this caveat on every run and aborts up
535
- front (a "smoke check") if your unmutated suite isn't green.
536
- - **Reload strategy only** (`--strategy redefine` is rejected on this path) and
537
- **serial** (`--jobs` is forced to 1). For apps on Ruby ≥ 3.4, `--daemon` gives
538
- safe parallelism instead (see [the daemon backend](https://github.com/davidteren/mutineer#faster-parallel-safe-rails-the---daemon-backend)).
539
-
540
- ## Timeouts and the score
541
-
542
- The mutation score is `killed / (killed + survived)`. A mutant whose tests run
543
- past `--timeout` is a `timeout`. It is neither killed nor survived, so it is
544
- left out of the score, like `no_coverage`, `uncapturable`, `unplaceable`, `ran_at_load`, `errored`, skipped
545
- and ignored mutants. A timeout is not counted as a kill, because a hang the
546
- mutant caused and a suite that is just slow look the same. The human report
547
- shows the count in its `Timeout:` row, and the JSON report in `summary.timeout`.
548
-
549
- Under `--threshold`, a timeout is a mutant with no verdict. The run exits 1 when
550
- more than one mutant produced no verdict and they exceed 10% of those attempted,
551
- or when nothing could be scored and something broke. If a slow suite times out
552
- mutants that its tests would catch, raise `--timeout`. A limit set below what
553
- the suite needs does not raise the score: it turns those mutants into timeouts,
554
- and once they pass 10% of the attempted mutants the run fails.
555
-
556
- ## Suppressing equivalent mutants
557
-
558
- Some mutants are equivalent (behaviour-identical) and survive forever — keeping a
559
- file off 100%. Suppress them so the score and `--threshold` gate stay meaningful:
560
-
561
- - **Inline:** `some_line # mutineer:disable-line` (or scope it: `# mutineer:disable-line comparison`). Put a reason after `--`: `# mutineer:disable-line comparison -- the test checks only 20`. The reason is stored on the ignored mutant and shown in the JSON report and the HTML report. A marker with no reason stores none.
562
- - **Config:** a `.mutineer.yml` `ignore:` list. A bare mutant id still works. A mapping is `id` plus an optional `reason`. Mutineer 1.7 is the first version that reads a mapping. An older version matches no mutant to a mapping and does not warn, so the mutant it names is not ignored. Add a mapping only after every gate that reads the file runs 1.7 or later. In the Action, the `version:` input sets the version, or your `Gemfile.lock` does when you set `use-bundler: true`. In your own CI job, your `Gemfile.lock` sets it. The reason shows in the same reports as an inline reason. Each survivor's `id` is printed in the JSON report.
563
-
564
- `mutineer triage REPORT.json --reason TEXT --all` prints one mapping entry per survivor. `--id ID` prints one entry. Repeat `--id` for several. The command does not edit files. Paste the lines under `ignore:`. They are mappings, so every gate that reads the file must run 1.7 or later (see **Config** above).
565
-
566
- Built-in skips are listed on the [equivalent skips](https://davidteren.github.io/mutineer/equivalent-skips.html) page. There is no switch to turn them off.
567
-
568
- Suppressed mutants are excluded from the score (so 100% becomes reachable).
569
-
570
- ## Mutant ids
571
-
572
- A mutant id is 12 hex characters. It hashes the file path (relative to the
573
- project root), the method's qualified name, the operator, the mutated code, and
574
- the mutant's position among identical mutants in that method. When one file has
575
- two methods with the same qualified name (for example two top-level `def index`
576
- in two DSL blocks), the second and later ones also hash their position among
577
- those methods, so their ids differ. The first one's id does not change. An edit
578
- outside the method does not change the id. Moving or renaming the file,
579
- renaming the method or its class, or adding an identical mutant earlier in the
580
- method does. Adding a method with the same name earlier in the same file also
581
- does.
582
-
583
- - The project root is the directory mutineer runs from (in the Action, the
584
- `working-directory`). Run from the same root to get the same ids.
585
- - A source outside the project root uses its absolute path, so its ids differ
586
- between machines.
587
-
588
- **Migrating from ids without the file path.** Before 1.3, ids did not include
589
- the file path, so two files could share an id (#126). Old-format ids keep
590
- working until 2.0, with a warning:
591
-
592
- - **`ignore:`** An old entry still suppresses its mutants. The run prints the
593
- new ids for each old entry, each with its file and method. When it names one
594
- mutant, replace the entry with that id. When it names several (in different
595
- files, or same-named methods in one file), the old entry over-matched: it also
596
- hid mutants you did not mean to ignore. The warning says so. Keep only the ids for the mutant you meant to
597
- ignore, not all of them. The list covers only the sources and operators in
598
- that run, so run over every source with every operator set you use (for
599
- example your Tier-2 `--operators`) for the full list. Or run
600
- `mutineer migrate` on those sources. It writes every new id. Delete any id
601
- you did not mean to ignore. See [Migrate old ignore ids](https://github.com/davidteren/mutineer#migrate-old-ignore-ids).
602
- - **`--baseline`** An old baseline still matches: a survivor matches a stored
603
- one with the same old id in the same file. A stored file that is an absolute
604
- path outside the project root (a baseline written on another machine)
605
- matches on the old id alone. The run tells you to
606
- generate a new baseline with `--format json`. `migrate` does not rewrite a
607
- baseline. Generate the new file only after every gate
608
- that reads it runs 1.3 or later. In the Action, the `version:` input sets
609
- the version, or your `Gemfile.lock` does when you set `use-bundler: true`.
610
- In your own CI job, your `Gemfile.lock` sets it. An older version treats every new-format survivor as new.
611
-
612
- The JSON report's `summary.id_format` is `2` for the new format.
613
- `summary.legacy_id_matches.ignore` counts the old-format ignore entries a run
614
- matched, and `summary.legacy_id_matches.baseline` counts the survivors matched
615
- only through an old baseline id.
616
-
617
- Ids are relative to the directory you run mutineer from. mutineer finds
618
- `.mutineer.yml` by walking up. When the file it loads is in a parent directory
619
- (other than your home directory), it warns that the ignore ids will not match
620
- and tells you which directory to run from.
621
-
622
- ## Kill matrix
623
-
624
- `--matrix` reports which tests kill each mutant, and from that, which tests
625
- add nothing to the suite.
626
-
627
- ```sh
628
- mutineer run lib/calculator.rb --test test/calculator_test.rb --matrix
629
- ```
630
-
631
- For each mutant, Mutineer runs every test in its covering files instead of
632
- stopping at the first failure, and records which tests failed. Coverage is
633
- recorded per test file, so that set also holds tests that never reach the
634
- mutated line; they pass and count as having run. A mutant's row is complete
635
- when its whole covering set ran and the suite returned normally (under
636
- Minitest, an `Interrupt` that cut the run short does not count as returning
637
- normally). The report
638
- names two kinds of test:
639
-
640
- - **Blind:** the test ran in at least one complete row and killed no mutant
641
- in any row. It would still pass with this run's mutations of the code, so
642
- it guards none of them. It can still check behavior outside this run's
643
- mutants (code in other files, or edits no operator makes), so look at what
644
- it asserts before you delete it, or give it an assertion that can fail.
645
- - **Redundant:** every mutant the test kills, another test kills too. That
646
- makes it a candidate for deletion, one test at a time: two redundant tests
647
- can be the only killers of one mutant, so re-run after each deletion.
648
-
649
- The answers cover this run's mutants only. A test of code outside the sources
650
- you passed kills nothing here, so mutate the code a test exercises before you
651
- call the test blind.
652
-
653
- The matrix changes no verdict, score or exit code: each mutant gets the
654
- verdict a run without `--matrix` gives. That run skips every later test,
655
- test class and example group after the first failure, so when a later one
656
- exits the process, crashes or runs into the time limit, the mutant is still
657
- `killed`. The run without `--matrix` still runs the code around the failing
658
- test (the rest of its Minitest class `run` wrapper, the `after(:all)` hooks of
659
- its RSpec groups) and the suite's own cleanup (RSpec's `after(:suite)`), so an
660
- end there keeps the exit status. A failure in a Minitest `parallelize_me!`
661
- class also keeps it: those run after every serial class and cannot be stopped
662
- once queued, while a failure in a serial class still stops the run.
663
-
664
- Under RSpec, `--matrix` needs RSpec 3.3 or later, whose example ids tell
665
- apart examples on one line. With an older RSpec every row is incomplete.
666
-
667
- The human report lists up to 20 blind and 20 redundant tests, and the HTML
668
- report lists them all. `--format json` adds a `matrix` block with every test
669
- and each mutant's killers (see the
670
- [JSON schema](https://davidteren.github.io/mutineer/json-schema.html#matrix)).
671
- A test is its file and its id. The name is `CalculatorTest#test_add` under
672
- Minitest, or the example's full description under RSpec, where the example id
673
- (`./spec/calc_spec.rb[1:2]`) tells apart examples that share a description
674
- and keeps an example whose generated description changes with the mutant as
675
- one test.
676
-
677
- Each mutant runs its whole covering set, so every mutant costs what a survivor
678
- costs. In 1.4, stopping at the first failure cut a full run of rack's
679
- `lib/rack/utils.rb` from 86 to 89 seconds down to 35 to 41, so expect a matrix
680
- run to take 2.1 to 2.5 times as long as a normal one. A mutant that reaches the
681
- per-mutant time limit (`--timeout`, 10 seconds by default) after a test already failed stays `killed`
682
- with its row marked incomplete. The report names each incomplete row and warns
683
- that a blind test may have killed one of those mutants, so do not delete a
684
- blind test until its rows are complete. Raise `--timeout` (or `timeout:` in
685
- `.mutineer.yml`) to get complete rows.
686
-
687
- `--matrix` runs on the in-process backend only. It exits 2 with `--daemon`,
688
- `--test-command`, `--fail-fast` or `--dry-run`, and the message says whether
689
- each setting came from the command line or `.mutineer.yml`. To run one of
690
- those with `matrix: true` in the file, pass `--no-matrix`; to run `--matrix`
691
- with `fail_fast: true` in the file, pass `--no-fail-fast`.
692
-
693
- ## CI gating
694
-
695
- Store a JSON run as a baseline, then fail the build only when a PR makes things
696
- worse:
697
-
698
- ```sh
699
- mutineer run app/ --baseline .mutineer/baseline.json # exit 1 on NEW survivors or a score drop
700
- ```
216
+ List with `mutineer --list-operators`. Enable specific operators with `--operators`.
217
+
218
+ | Group | Operators |
219
+ |---|---|
220
+ | Tier 1 (default) | `arithmetic`, `comparison`, `boolean_connector`, `boolean_literal`, `statement_removal` |
221
+ | Tier 2 (opt-in) | `return_nil`, `literal_mutation`, `condition_negation`, `string_literal`, `regex`, `collection_method`, `safe_navigation`, `range`, `negation_removal`, `chain_link`, `call_to_receiver`, `operand_removal`, `array_literal`, `condition_true`, `condition_false`, `operator_assignment` |
222
+
223
+ ## Documentation
224
+
225
+ | Guide | What it covers |
226
+ |-------|----------------|
227
+ | [Quick start](https://github.com/davidteren/mutineer/blob/main/docs/quick-start.md) | Install, run, fix a survivor |
228
+ | [First run to steady state](https://github.com/davidteren/mutineer/blob/main/docs/getting-started.md) | One file through changed-code CI |
229
+ | [Reading results](https://github.com/davidteren/mutineer/blob/main/docs/results.md) | Reports, survivors, suppression, mutant ids, kill matrix |
230
+ | [Rails setup](https://github.com/davidteren/mutineer/blob/main/docs/rails-setup.md) | Rails, daemon, databases, Spring, Bootsnap, Ruby < 3.4 |
231
+ | [Rails rollout](https://davidteren.github.io/mutineer/rails.html) | Four stages from model to CI gate |
232
+ | [CI: changed code and baselines](https://github.com/davidteren/mutineer/blob/main/docs/ci.md) | `--since`, `--baseline`, Action inputs, timeouts |
233
+ | [CLI reference](https://github.com/davidteren/mutineer/blob/main/docs/cli.md) | Flags, defaults, `.mutineer.yml` |
234
+ | [Comparison](https://github.com/davidteren/mutineer/blob/main/docs/comparison.md) | Mutineer, Mutant, other Ruby tools, fit-check prompt |
235
+ | [AI agents and CI](https://davidteren.github.io/mutineer/agentic-coding.html) | Agent loop and CI recipes |
236
+ | [JSON schema](https://davidteren.github.io/mutineer/json-schema.html) | Versioned report contract |
237
+ | [STABILITY.md](https://github.com/davidteren/mutineer/blob/main/STABILITY.md) | Release guarantees |
701
238
 
702
- `--baseline` reports which survivors are new (by [mutant id](https://github.com/davidteren/mutineer#mutant-ids)) and any score drop. It
703
- combines with `--threshold` (the worse of the two sets the exit code). Pass a
704
- directory (or several sources) to audit a whole layer in one boot — tests are
705
- auto-paired by convention and the report breaks down per source. A source
706
- `app/foo/bar.rb` pairs with `test/foo/bar_test.rb` and with unclaimed
707
- `test/foo/bar_*_test.rb` files in that directory, such as `bar_upsert_test.rb`.
708
- It does not take `user_session_test.rb` when `user_session.rb` exists in the same directory.
709
- It does not take `bar_upsert_guards_test.rb` when `bar_upsert.rb` exists.
710
- A spec that already pairs is left as that one file.
711
-
712
- ### GitHub Action
713
-
714
- This repo ships a composite action (`action.yml`) that wraps the CLI for CI:
715
-
716
- ```yaml
717
- - uses: actions/checkout@v4
718
- - uses: ruby/setup-ruby@v1
719
- with: { ruby-version: "3.4", bundler-cache: true }
720
- - uses: davidteren/mutineer@v1
721
- with:
722
- sources: app/
723
- baseline: .mutineer/baseline.json
724
- threshold: "90"
725
- ```
239
+ ## For AI agents & pipelines
726
240
 
727
- **Default change:** on `pull_request` events (not `pull_request_target`) the
728
- action scopes the run to the PR's changed lines, diffing against the PR's exact
729
- base commit (fetched by the action itself when the checkout is shallow; falls
730
- back to the base branch tip). Pass `since: none` for a full scan, or an
731
- explicit `since:` ref (which needs `fetch-depth: 0` on checkout).
241
+ Use versioned JSON, structured exit codes, and runs scoped to changed lines.
242
+ [Mutant ids](https://github.com/davidteren/mutineer/blob/main/docs/results.md#mutant-ids) survive unrelated edits.
732
243
 
733
- With the default JSON format the action also:
244
+ <a id="mutant-ids"></a><a id="suppressing-equivalent-mutants"></a><a id="apps-on-ruby--34"></a>
245
+ Mutant ids and equivalent-mutant suppression moved to [Reading results](https://github.com/davidteren/mutineer/blob/main/docs/results.md). Apps on Ruby < 3.4 moved to [Rails setup](https://github.com/davidteren/mutineer/blob/main/docs/rails-setup.md#apps-on-ruby--34).
734
246
 
735
- - writes a score summary to the job's step summary;
736
- - annotates surviving mutants on the PR diff, up to 50 (`error` level when the
737
- gate failed, `warning` when it passed);
738
- - exposes the report path via the `report` output for later steps (with
739
- `format: human`/`html` this needs the `output` input).
247
+ | Resource | Links and use |
248
+ |---|---|
249
+ | AI agents & CI recipes | [rendered](https://davidteren.github.io/mutineer/agentic-coding.html) · [source](https://davidteren.github.io/mutineer/agentic-coding.md): agent loops, CI gates, avoiding equivalent-mutant loops |
250
+ | JSON schema reference | [rendered](https://davidteren.github.io/mutineer/json-schema.html) · [source](https://davidteren.github.io/mutineer/json-schema.md): `--format json` schema and versioning |
251
+ | Ruby API (YARD) | [https://davidteren.github.io/mutineer/api/](https://davidteren.github.io/mutineer/api/): shipped classes |
252
+ | Agent skill | [`skills/mutineer/SKILL.md`](https://github.com/davidteren/mutineer/blob/main/skills/mutineer/SKILL.md): installation, agent loop, exit codes |
740
253
 
741
- The CLI prints a progress line to the log at every 10% of the run, whatever
742
- the format.
254
+ Install the skill with either command:
743
255
 
744
- ## For AI agents & pipelines
745
-
746
- Mutineer is built for programmatic use — versioned JSON, [mutant ids](https://github.com/davidteren/mutineer#mutant-ids) that survive unrelated edits,
747
- structured exit codes, and diff-scoped runs. See:
748
-
749
- - **AI agents & CI recipes** — the agent inner-loop and CI-gate recipes (and how
750
- to avoid infinite loops on equivalent mutants):
751
- [rendered](https://davidteren.github.io/mutineer/agentic-coding.html) ·
752
- [source](https://davidteren.github.io/mutineer/agentic-coding.md)
753
- - **JSON schema reference** — the `--format json` schema and its versioning
754
- contract:
755
- [rendered](https://davidteren.github.io/mutineer/json-schema.html) ·
756
- [source](https://davidteren.github.io/mutineer/json-schema.md)
757
- - **Ruby API (YARD)** — class reference for the shipped gem:
758
- [https://davidteren.github.io/mutineer/api/](https://davidteren.github.io/mutineer/api/)
759
- - **Agent skill** ([`skills/mutineer/SKILL.md`](https://github.com/davidteren/mutineer/blob/main/skills/mutineer/SKILL.md)): a short
760
- card for coding agents with install, the agent loop, and exit codes. Install it
761
- with `gh skill install davidteren/mutineer mutineer` or
762
- `npx skills add davidteren/mutineer --skill mutineer`.
763
-
764
- ## Configuration
765
-
766
- Mutineer reads an optional `.mutineer.yml` from the project root (nearest one,
767
- walking up). CLI flags override config; config overrides defaults.
768
-
769
- Sources are positional CLI arguments and test files come from `--test`. The
770
- config file accepts these keys:
771
-
772
- | Key | Value and purpose |
773
- |-----|-------------------|
774
- | `operators` | An operator name or list of names; defaults to the Tier-1 set |
775
- | `threshold` | A number from 0 to 100; 0 turns the score gate off |
776
- | `jobs` | A positive integer; the default is the processor count. `test_command`, `fail_fast`, or `--rails` without `--daemon` forces 1. |
777
- | `only` | A fully-qualified subject name, such as `Calculator#add` |
778
- | `require` | A path or list of extra files to load before mutating |
779
- | `boot` | The app entry point to require once before forking |
780
- | `rails` | `true` or `false`; enables the Rails boot defaults |
781
- | `since` | A nonblank git ref, or `false` to disable diff scoping |
782
- | `framework` | `minitest` or `rspec`; an explicit value is kept during test pairing |
783
- | `verbose` | `true` or `false`; shows capture diagnostics |
784
- | `ignore` | A mutant id or list of ids to suppress |
785
- | `baseline` | The path to a prior JSON report |
786
- | `fail_fast` | `true` or `false`; stops scheduling after the first survivor |
787
- | `test_command` | The external-runtime suite command, including `%{files}`; see [Apps on Ruby < 3.4](https://github.com/davidteren/mutineer#apps-on-ruby--34) |
788
- | `daemon` | `true` or `false`; uses the persistent app daemon with worker DB isolation |
789
- | `timeout` | A positive integer; the per-mutant time limit in seconds (default 10) |
790
- | `capture_timeout` | A positive integer; the coverage-capture time limit in seconds (default 120) |
791
- | `cache_dir` | The coverage cache directory (default `.mutineer`) |
792
- | `matrix` | `true` or `false`; runs every covering test for each mutant and adds the kill matrix to the report |
793
- | `allow_empty` | `true` or `false`; a run with no mutants is expected, so it does not warn |
794
-
795
- Invalid values for known scalar keys, and a blank `operators` list, exit 2
796
- with a message naming the file and key. An unknown operator name warns and
797
- is skipped. If none of the names are known, the run exits 2. An empty
798
- `require` or `ignore` list is valid. Boolean keys take `true` or `false` (quoted forms also work), not `"yes"`.
799
- `jobs` must be positive; a string value contains digits only. String values for
800
- `threshold` and the CLI-only `--baseline-epsilon` use plain decimals such as `90`
801
- or `0.5`, not `+2`, `1e2`, or `1_0`. String options such as `only` and `baseline`
802
- cannot be null or boolean. A blank `since` is invalid; use `since: false` to turn
803
- scoping off. Unknown keys warn and are ignored. Unknown operator names warn and are skipped, and the run exits 2 when none remain. Both warnings suggest the closest valid name, and both become errors in Mutineer 2.0. `--operators` replaces a blank or unknown file list.
804
-
805
- `format`, `strategy`, `output`, `baseline_epsilon`, `dry_run`, and `coverage_from` are CLI-only.
806
- For JSON output, use `--format json`, not a `format:` config key. To select RSpec
807
- in the file, add `framework: rspec`.
808
-
809
- ```yaml
810
- # .mutineer.yml
811
- operators: [arithmetic, comparison, boolean_connector, boolean_literal, statement_removal]
812
- threshold: 90
813
- jobs: 4
814
- require:
815
- - config/environment
816
- ```
256
+ - `gh skill install davidteren/mutineer mutineer`
257
+ - `npx skills add davidteren/mutineer --skill mutineer`
817
258
 
818
- Coverage results are cached in `.mutineer/coverage.json` (digest-keyed; rebuilt
819
- automatically when sources change). Add `.mutineer/` to your `.gitignore`, and
820
- the directory you set with `--cache-dir` if it is inside the project. Keep
821
- `.mutineer/` ignored even then: `--test-command` writes lock files to
822
- `.mutineer/` beside each source file.
823
-
824
- ## Mutineer and Mutant
825
-
826
- [Mutant](https://github.com/mbj/mutant), by Markus Schirp, is the established
827
- mutation-testing tool for Ruby. It has been in development since 2012 and is
828
- the subject of published research. Mutineer is a separate tool, and it shares
829
- no code with Mutant.
830
-
831
- | Aspect | Mutineer | Mutant |
832
- |---|---|---|
833
- | License | MIT, for every use | Free for open source (`--usage opensource`); commercial use needs a paid subscription |
834
- | Runtime dependencies | None (Prism + stdlib) | `parser`, `unparser`, `regexp_parser`, `sorbet-runtime` and others |
835
- | Ruby | 3.4 and later to run Mutineer; an app on an older Ruby works through `--test-command` (see [Apps on Ruby < 3.4](https://github.com/davidteren/mutineer#apps-on-ruby--34)) | 3.3 and later |
836
- | Test frameworks | Minitest and RSpec, in one gem | RSpec, Minitest and Test::Unit, one integration gem each |
837
- | What you mutate | Files (`mutineer run lib/foo.rb --test test/foo_test.rb`), narrowed to one method with `--only 'Foo#bar'` | Subjects named by expression (`mutant run 'Foo#bar'`, `'Foo*'`) |
838
- | Which tests run | The test files whose coverage reaches the mutated line | The tests that declare the subject (an RSpec description or a Minitest `cover`), or the tests that ran it in a per-test coverage recording |
839
- | Operators | 21 (5 by default, 16 opt-in) | A larger set; the default `light` set applies almost all of it (`full` adds `#==` to `#eql?`) |
840
- | CI gating | `--threshold`, `--baseline` deltas, `--since`, and a GitHub Action | Incremental mode (`--since`) and a recorded session history |
841
- | Machine-readable output | Versioned JSON report schema | Session JSON schema |
842
-
843
- Choose Mutineer when you want an MIT-licensed tool with no extra gems in your
844
- bundle, file-based runs, and a JSON contract built for CI gates and AI agents.
845
- Choose Mutant when you need its deeper operator set, Test::Unit support, or
846
- subject expressions that select whole namespaces. The two tools do not
847
- conflict, so you can run both on one project.
848
-
849
- This comparison describes Mutant 0.17 (checked on 2026-10-06). Check
850
- [Mutant's README](https://github.com/mbj/mutant#readme) for its current
851
- license terms and features.
259
+ **Mutineer 1.8.0 and later** bundle the skill and offer
260
+ [ Rails Hyperdrive](https://github.com/rails-hyperdrive/rails-hyperdrive) companion installation.
261
+ For Rails apps bundling Mutineer, `bin/rails hyperdrive:init` installs it to `.claude/skills/mutineer/SKILL.md`.
852
262
 
853
263
  ## License
854
264
 
855
- MIT — see [LICENSE](https://github.com/davidteren/mutineer/blob/main/LICENSE).
265
+ MIT. See [LICENSE](https://github.com/davidteren/mutineer/blob/main/LICENSE).