mutineer 1.6.0 → 1.8.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +680 -0
- data/README.md +210 -499
- data/lib/mutineer/baseline.rb +1 -1
- data/lib/mutineer/changed_lines.rb +2 -1
- data/lib/mutineer/child_coverage.rb +53 -0
- data/lib/mutineer/cli.rb +293 -42
- data/lib/mutineer/concern_blocks.rb +243 -0
- data/lib/mutineer/config.rb +154 -8
- data/lib/mutineer/coverage_map.rb +265 -27
- data/lib/mutineer/daemon_backend.rb +244 -75
- data/lib/mutineer/daemon_client.rb +73 -10
- data/lib/mutineer/daemon_server.rb +226 -42
- data/lib/mutineer/external_backend.rb +159 -15
- data/lib/mutineer/init.rb +124 -0
- data/lib/mutineer/isolation.rb +78 -24
- data/lib/mutineer/job_plan.rb +234 -44
- data/lib/mutineer/kill_channel.rb +24 -14
- data/lib/mutineer/migrate.rb +707 -0
- data/lib/mutineer/minitest_integration/in_test_database_error.rb +184 -0
- data/lib/mutineer/minitest_integration/kill_recorder.rb +25 -17
- data/lib/mutineer/minitest_integration.rb +44 -14
- data/lib/mutineer/mutator_registry.rb +6 -2
- data/lib/mutineer/mutators/array_literal.rb +4 -2
- data/lib/mutineer/mutators/base.rb +36 -7
- data/lib/mutineer/mutators/call_to_receiver.rb +148 -0
- data/lib/mutineer/mutators/chain_link.rb +6 -6
- data/lib/mutineer/mutators/condition_forcing.rb +6 -3
- data/lib/mutineer/mutators/operand_removal.rb +4 -2
- data/lib/mutineer/mutators/range_literal.rb +105 -4
- data/lib/mutineer/mutators/return_nil.rb +1 -0
- data/lib/mutineer/orphan_guard.rb +84 -0
- data/lib/mutineer/pairing.rb +62 -1
- data/lib/mutineer/project.rb +302 -14
- data/lib/mutineer/rails_run_env.rb +73 -0
- data/lib/mutineer/rails_worker_db.rb +1596 -57
- data/lib/mutineer/redaction.rb +199 -0
- data/lib/mutineer/reporter.rb +173 -31
- data/lib/mutineer/result.rb +52 -15
- data/lib/mutineer/runner.rb +148 -28
- data/lib/mutineer/statement_lines.rb +24 -11
- data/lib/mutineer/subject.rb +30 -4
- data/lib/mutineer/test_runners/minitest.rb +2 -1
- data/lib/mutineer/test_runners/rspec.rb +15 -3
- data/lib/mutineer/test_runners.rb +5 -2
- data/lib/mutineer/triage.rb +111 -0
- data/lib/mutineer/version.rb +1 -1
- data/lib/mutineer/worker_pool.rb +2 -1
- data/lib/mutineer.rb +4 -0
- data/skills/mutineer/SKILL.md +84 -0
- metadata +13 -1
data/README.md
CHANGED
|
@@ -1,554 +1,265 @@
|
|
|
1
1
|
# Mutineer
|
|
2
2
|
|
|
3
3
|
[](https://rubygems.org/gems/mutineer)
|
|
4
|
+
[](https://github.com/davidteren/mutineer/actions/workflows/ci.yml)
|
|
5
|
+
[](https://github.com/davidteren/mutineer/actions/workflows/ci.yml)
|
|
4
6
|
[](https://github.com/marketplace/actions/mutineer-ruby)
|
|
5
7
|
[](https://socket.dev/rubygems/package/mutineer)
|
|
6
8
|
|
|
7
|
-
|
|
8
|
-
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
9
|
+
**Find bugs your tests would miss.** Mutineer changes your Ruby code and runs your Minitest or RSpec tests.
|
|
10
|
+
Changes that still pass reveal possible test gaps. This is **mutation testing**.
|
|
11
|
+
|
|
12
|
+
**Features**
|
|
13
|
+
|
|
14
|
+
- **Minitest and RSpec.** Detected from test filenames; override with `--framework`.
|
|
15
|
+
- **Rails support.** `--rails` boots once. With `--daemon`, each Minitest worker gets its own SQLite file, PostgreSQL, or MySQL 8 test database.
|
|
16
|
+
- **Focused tests.** Runs tests that reach each changed line, likely and fast tests first. Stops at the first failure by default.
|
|
17
|
+
- **PR checks.** `--since origin/main` checks changed lines; `--baseline` gates new survivors, and score drops on full runs. When tests change, run a full scan (`--no-since`).
|
|
18
|
+
- **CI and agents.** GitHub Action, versioned JSON, stable mutant ids, clear exit codes, and zero runtime gem dependencies.
|
|
19
|
+
|
|
20
|
+
📖 **[davidteren.github.io/mutineer →](https://davidteren.github.io/mutineer/)**: overview, operators, and usage.
|
|
21
|
+
|
|
22
|
+
## Is it production-ready?
|
|
23
|
+
|
|
24
|
+
Check your database and framework before adding a CI gate.
|
|
25
|
+
Mutineer requires Ruby 3.4 or later. In-process runs require Linux or macOS for forks.
|
|
26
|
+
Plain Ruby projects without databases support parallel runs.
|
|
27
|
+
In the Status column, `supported` means parallel runs work and `serial only` runs one mutant at a time.
|
|
28
|
+
|
|
29
|
+
<!-- contract:support-matrix -->
|
|
30
|
+
| Ruby | Database | Framework | Parallelism | Status |
|
|
31
|
+
| --- | --- | --- | --- | --- |
|
|
32
|
+
| 3.4 or later | SQLite file | Minitest | supported with `--daemon` | supported |
|
|
33
|
+
| 3.4 or later | SQLite file | RSpec | not supported | serial only |
|
|
34
|
+
| 3.4 or later | PostgreSQL | Minitest | supported with `--daemon` | supported |
|
|
35
|
+
| 3.4 or later | PostgreSQL | RSpec | not supported | serial only |
|
|
36
|
+
| 3.4 or later | MySQL 8 | Minitest | supported with `--daemon` | supported |
|
|
37
|
+
| 3.4 or later | MySQL 8 | RSpec | not supported | serial only |
|
|
38
|
+
| 3.4 or later | MariaDB | Minitest | not tested | not tested |
|
|
39
|
+
| 3.4 or later | MySQL 9 or later | Minitest | not tested | not tested |
|
|
40
|
+
| 3.4 or later | SQLite in memory | Minitest | not supported | serial only |
|
|
41
|
+
| 3.4 or later | SQLite in memory | RSpec | not supported | serial only |
|
|
42
|
+
| 3.4 or later | more than one database | Minitest | one worker | serial only |
|
|
43
|
+
| 3.4 or later | more than one database, Rails 7.1 or later, the extra ones in `shared_databases` | Minitest | supported with `--daemon` | supported, unless the test environment registers a pool for a shared database |
|
|
44
|
+
| 3.4 or later | more than one database | RSpec | not supported | serial only |
|
|
45
|
+
|
|
46
|
+
This matrix is for a Rails app on Ruby 3.4 or later. A plain Ruby project
|
|
47
|
+
with no database runs in parallel with either framework.
|
|
48
|
+
`not supported` means `--daemon` cannot run that cell; `not tested` means no CI
|
|
49
|
+
job runs it. SQLite file means a database file, not `:memory:`.
|
|
50
|
+
|
|
51
|
+
- `--rails` without `--daemon` is serial on every row. `--daemon` is Minitest only.
|
|
52
|
+
- MySQL older than 8.0 stops `--daemon`. MariaDB and MySQL 9 or later warn and run.
|
|
53
|
+
- An app with more than one database runs one worker: `--daemon` isolates only the primary connection.
|
|
54
|
+
On Rails 7.1 or later, list the databases your tests never use in `shared_databases`, and the workers
|
|
55
|
+
run in parallel. CI tests that row on SQLite. See [More than one database](https://github.com/davidteren/mutineer/blob/main/docs/rails-setup.md#more-than-one-database).
|
|
56
|
+
- Postgres worker databases are copied once at the start; MySQL 8 worker databases are created then and the rows copied.
|
|
57
|
+
- A test that commits rows outside a transaction leaves them in that worker's
|
|
58
|
+
database, and the next mutant on that worker sees them. Rails parallel tests have the same limit.
|
|
59
|
+
- RSpec: an example that fails with a database connection error scores
|
|
60
|
+
`killed`, not `errored`, so the score can read too high
|
|
61
|
+
([#347](https://github.com/davidteren/mutineer/issues/347)).
|
|
62
|
+
|
|
63
|
+
Mutineer sets `PARALLEL_WORKERS=1`, so a `parallelize` call in `test_helper.rb` starts no Rails workers during a mutant run.
|
|
64
|
+
See [Rails parallel tests, Spring, Bootsnap, and Redis](https://github.com/davidteren/mutineer/blob/main/docs/rails-setup.md#rails-parallel-tests-spring-bootsnap-and-redis).
|
|
65
|
+
|
|
66
|
+
<!-- /contract:support-matrix -->
|
|
67
|
+
|
|
68
|
+
Older Ruby apps use `--test-command`; Mutineer still requires Ruby 3.4 or later.
|
|
69
|
+
This path uses one Mutineer worker and no coverage-based test selection by default; a wrapper can re-enable Rails test workers by setting `PARALLEL_WORKERS`.
|
|
70
|
+
|
|
71
|
+
| Subprocess result | Score |
|
|
72
|
+
|---|---|
|
|
73
|
+
| Exit 1, normally | `killed` |
|
|
74
|
+
| SimpleCov gate exit with a failing test | `killed` |
|
|
75
|
+
| Other exit code | `error` |
|
|
76
|
+
| Crash before a test fails | Can score `error` |
|
|
77
|
+
| Only a SimpleCov gate fails, with its gate message and a passing summary | `survived` |
|
|
78
|
+
|
|
79
|
+
See [Apps on Ruby < 3.4](https://github.com/davidteren/mutineer/blob/main/docs/rails-setup.md#apps-on-ruby--34)
|
|
80
|
+
and release guarantees in [STABILITY.md](https://github.com/davidteren/mutineer/blob/main/STABILITY.md).
|
|
81
|
+
|
|
82
|
+
## What is mutation testing?
|
|
83
|
+
|
|
84
|
+
Mutation testing changes code to check whether your tests notice.
|
|
85
|
+
Each changed copy is a **mutant**, such as replacing `>=` with `>` or `&&` with `||`.
|
|
86
|
+
|
|
87
|
+
| Result | Meaning |
|
|
88
|
+
|---|---|
|
|
89
|
+
| **Killed** | A test failed and caught the change. |
|
|
90
|
+
| **Survived** | All tests passed. Inspect the change for a test gap. |
|
|
91
|
+
|
|
92
|
+
### A small example
|
|
37
93
|
|
|
38
94
|
```ruby
|
|
39
|
-
|
|
95
|
+
# lib/discount.rb
|
|
96
|
+
class Discount
|
|
97
|
+
def rate(qty)
|
|
98
|
+
return 0.1 if qty >= 10
|
|
99
|
+
0
|
|
100
|
+
end
|
|
101
|
+
end
|
|
40
102
|
```
|
|
41
103
|
|
|
42
|
-
|
|
104
|
+
Tests check quantities 1 and 50: 100% line coverage. Mutineer changes `>=` to `>`; both tests pass:
|
|
43
105
|
|
|
44
|
-
```
|
|
45
|
-
|
|
106
|
+
```text
|
|
107
|
+
lib/discount.rb
|
|
108
|
+
Discount#rate (discount.rb:3)
|
|
109
|
+
Operator: comparison (>= -> >)
|
|
110
|
+
- return 0.1 if qty >= 10
|
|
111
|
+
+ return 0.1 if qty > 10
|
|
46
112
|
```
|
|
47
113
|
|
|
48
|
-
|
|
49
|
-
mutation score drops below 90%:
|
|
114
|
+
No test checks exactly 10. Testing 10 kills the mutant. Run the example, tests and report in [Mutation testing, explained](https://davidteren.github.io/mutineer/learn.html).
|
|
50
115
|
|
|
51
|
-
|
|
52
|
-
mutineer run lib/calculator.rb --test test/calculator_test.rb --threshold 90
|
|
53
|
-
```
|
|
116
|
+
### Coverage and mutation testing
|
|
54
117
|
|
|
55
|
-
|
|
118
|
+
| Check | What it shows |
|
|
119
|
+
|---|---|
|
|
120
|
+
| Line coverage | Which lines ran |
|
|
121
|
+
| Mutation testing | Whether tests detect changes to those lines |
|
|
56
122
|
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
| `--test FILE` | Test file covering the sources; one file per flag, so repeat it for each (`--test a_test.rb --test b_test.rb`) |
|
|
60
|
-
| `--operators LIST` | Comma-separated operator names (default: the Tier-1 set) |
|
|
61
|
-
| `--threshold FLOAT` | Exit 1 when the score is below FLOAT, or when nothing could be scored and something broke, or more than one mutant produced no verdict and they exceed 10% of those attempted (default: 0 = off) |
|
|
62
|
-
| `--only NAME` | Restrict to one fully-qualified subject, e.g. `Calculator#add` |
|
|
63
|
-
| `--framework NAME` | `minitest` (default) or `rspec`; auto-detected as rspec when most `--test` files end in `_spec.rb` |
|
|
64
|
-
| `--since REF` | Only mutate lines changed since git `REF` (e.g. `origin/main`). Untracked files are scored in full unless Git ignores them. |
|
|
65
|
-
| `--no-since` | Disable diff scoping; a typed no beats a `.mutineer.yml` `since:` key |
|
|
66
|
-
| `--baseline FILE` | Compare against a prior `--format json` run; exit 1 on new survivors / score drop (score drop is skipped under `--since`, whose score covers a different denominator; see [CI](https://github.com/davidteren/mutineer#ci-gating)) |
|
|
67
|
-
| `--baseline-epsilon FLOAT` | Score-drop tolerance for `--baseline` (default: 0) |
|
|
68
|
-
| `--jobs N` | Parallel worker count (default: processor count); forced to `1` by `--test-command`, `--fail-fast`, or `--rails` without `--daemon` |
|
|
69
|
-
| `--boot FILE` | Require an app entry point once before forking; select at least one test file |
|
|
70
|
-
| `--rails` | Boot `config/environment` and reconnect ActiveRecord per fork; without `--daemon`, defaults to `redefine` and runs serially |
|
|
71
|
-
| `--verbose` | Surface the real error when a fork capture fails (alias `--debug`) |
|
|
72
|
-
| `--strategy NAME` | Mutation application: `reload` whole-file (default) or `redefine` surgical (`7a`/`7b` accepted as deprecated aliases) |
|
|
73
|
-
| `--timeout SECONDS` | Per-mutant time limit for in-process runs, in whole seconds (default: 10). A mutant whose tests run longer is a `timeout`; see [Timeouts and the score](https://github.com/davidteren/mutineer#timeouts-and-the-score). `--daemon` and `--test-command` keep their own limits |
|
|
74
|
-
| `--capture-timeout SECONDS` | Time limit for each in-process coverage-capture subprocess and for the clean run of the unmutated tests, in whole seconds (default: 120). A suite slower than this stops the run as not green. `--daemon` uses it for its coverage capture; `--test-command` ignores it and says so |
|
|
75
|
-
| `--cache-dir DIR` | Directory for the coverage cache (default: `.mutineer`). Give two runs from the same project root different directories so they do not share `coverage.json`. `--test-command` builds no cache, so it ignores it and says so |
|
|
76
|
-
| `--test-command CMD` | Run the suite as a subprocess in the app's own runtime (for apps on Ruby < 3.4); `CMD` must contain `%{files}`. See [Apps on Ruby < 3.4](https://github.com/davidteren/mutineer#apps-on-ruby--34) |
|
|
77
|
-
| `--daemon` | Boot the app once in a persistent daemon and fork per mutant, with per-worker DB isolation so `--jobs N` is safe under Rails (needs `--rails`/`--boot`; not with `--test-command`). See [the daemon backend](https://github.com/davidteren/mutineer#faster-parallel-safe-rails-the---daemon-backend) |
|
|
78
|
-
| `--format human\|json\|html` | Report format (default: human; `html` is a self-contained file) |
|
|
79
|
-
| `--output FILE` | Write the report to FILE instead of stdout |
|
|
80
|
-
| `--dry-run` | List candidate mutations without executing (honors suppression) |
|
|
81
|
-
| `--fail-fast` | Stop at the first surviving mutant (`--no-fail-fast` beats a `.mutineer.yml` `fail_fast:` key) |
|
|
82
|
-
| `--matrix` | Run every covering test for each mutant and report the blind and redundant tests; the JSON report also lists each mutant's killers. In-process only; exits 2 with `--daemon`, `--test-command`, `--fail-fast` or `--dry-run`. RSpec needs 3.3 or later. `--no-matrix` beats a `.mutineer.yml` `matrix:` key. See [Kill matrix](https://github.com/davidteren/mutineer#kill-matrix) |
|
|
83
|
-
| `--list-operators` | List available operators (default vs optional) and exit |
|
|
84
|
-
| `--version`, `--help` | Print version / usage and exit |
|
|
123
|
+
Tests without assertions add coverage but kill mutants only when changes cause errors.
|
|
124
|
+
This gap is common in rushed tests and tests written by AI agents.
|
|
85
125
|
|
|
86
|
-
|
|
126
|
+
## How it works
|
|
87
127
|
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
|
|
94
|
-
|
|
128
|
+
1. **Parse.** [Prism](https://github.com/ruby/prism), included in Ruby 3.4, parses source files. Mutineer uses only Prism and the standard library.
|
|
129
|
+
2. **Mutate.** Operators make one change per mutant. Mutineer parses each result again and skips invalid Ruby.
|
|
130
|
+
3. **Map coverage and check unchanged tests.** Each test file runs with coverage to map lines to tests.
|
|
131
|
+
`.mutineer/coverage.json` caches the map; source changes rebuild it. Tests must pass alone and
|
|
132
|
+
together, including with a cached map. A failure stops the run.
|
|
133
|
+
4. **Run each mutant in a fork.** The parent loads code once, including the app with `--rails`.
|
|
134
|
+
Children run in parallel on Linux and macOS.
|
|
135
|
+
Each loads the changed file (`reload`) or replaces the changed method (`redefine`).
|
|
136
|
+
Only covering test files run: paired tests first, then fastest to slowest. In-process runs stop at the first failure unless `--matrix` requests every covering test.
|
|
137
|
+
5. **Score.** A failing test kills the mutant; all tests passing means it survived.
|
|
138
|
+
Score: `killed / (killed + survived)`. In coverage-guided runs, timeouts, errors, and uncovered lines count as neither; `--test-command` has no coverage, so a mutant no test reaches survives.
|
|
95
139
|
|
|
96
|
-
|
|
140
|
+
| Alternative backend | Changes to step 4 |
|
|
141
|
+
|---|---|
|
|
142
|
+
| `--daemon` | Boots Rails once in a persistent process; isolates each worker's test database |
|
|
143
|
+
| `--test-command` | Runs tests in another Ruby subprocess; writes mutants to the real file, then restores it |
|
|
97
144
|
|
|
98
|
-
|
|
99
|
-
|
|
100
|
-
|
|
101
|
-
|
|
102
|
-
|
|
103
|
-
|
|
104
|
-
|
|
145
|
+
| At a glance | |
|
|
146
|
+
|---|---|
|
|
147
|
+
| Parser | Prism, with no runtime gem dependencies |
|
|
148
|
+
| Ruby | 3.4 or later; older apps use `--test-command` |
|
|
149
|
+
| Operators | 21 (5 by default, 16 opt-in) |
|
|
150
|
+
| Isolation | One fork per mutant on Linux and macOS |
|
|
151
|
+
| Report formats | human, JSON (versioned schema), HTML |
|
|
105
152
|
|
|
106
|
-
##
|
|
153
|
+
## Quick start
|
|
107
154
|
|
|
108
|
-
|
|
109
|
-
at your app with `--rails` and run it inside the project's bundle:
|
|
155
|
+
Plain Ruby:
|
|
110
156
|
|
|
111
157
|
```sh
|
|
112
|
-
|
|
113
|
-
|
|
158
|
+
gem install mutineer
|
|
159
|
+
mutineer run lib/discount.rb --test test/discount_test.rb
|
|
114
160
|
```
|
|
115
161
|
|
|
116
|
-
|
|
117
|
-
then forks and inherits it), defaults `--strategy` to `redefine` without `--daemon`
|
|
118
|
-
(surgical — it avoids reloading files into the app tree; `--daemon` keeps `reload`), and reconnects ActiveRecord in each
|
|
119
|
-
fork so the database connection is fork-safe. Use `--boot FILE` to boot a
|
|
120
|
-
different entry point. Boot mode requires at least one `--test` file and is
|
|
121
|
-
coverage-guided — each mutant runs only the test files that exercise its line
|
|
122
|
-
(coverage is captured by forking the booted app, then cached).
|
|
123
|
-
|
|
124
|
-
Some code runs while the app boots or a class loads, for example a class body
|
|
125
|
-
that calls a method to build a constant, a `to_prepare` initializer, or a
|
|
126
|
-
`--require` file that calls a source method. That run
|
|
127
|
-
happens before the mutant is applied, and the forked test does not repeat it.
|
|
128
|
-
A mutant on such a line is `ran_at_load`, not `survived` or `no_coverage`. It is
|
|
129
|
-
left out of the score and does not fail `--threshold`; a kill still counts.
|
|
130
|
-
This holds under both strategies: `reload` runs the mutated file's class body
|
|
131
|
-
again, but not an initializer or another file's code, so a survivor on such a
|
|
132
|
-
line is not trusted there either.
|
|
133
|
-
Run those mutants with `--test-command`, which boots a fresh process per mutant,
|
|
134
|
-
to get a verdict.
|
|
135
|
-
|
|
136
|
-
Add Mutineer to your Gemfile's test group:
|
|
162
|
+
Rails: add to your `Gemfile` test group:
|
|
137
163
|
|
|
138
164
|
```ruby
|
|
139
165
|
gem "mutineer", group: :test, require: false
|
|
140
166
|
```
|
|
141
167
|
|
|
142
|
-
|
|
143
|
-
|
|
144
|
-
`--rails` boots your app once but runs mutants **serially** — parallel `--jobs`
|
|
145
|
-
under Rails is unsafe, because every worker shares one test database and clobbers
|
|
146
|
-
the others' fixtures. `--daemon` fixes both: it boots the app once in a persistent
|
|
147
|
-
helper and forks per mutant, and gives **each parallel worker its own database**,
|
|
148
|
-
so `--jobs N` is safe and its verdicts are proven identical to a serial run.
|
|
168
|
+
Install, create `.mutineer.yml`, and check one model:
|
|
149
169
|
|
|
150
170
|
```sh
|
|
151
|
-
|
|
152
|
-
|
|
153
|
-
|
|
171
|
+
bundle install
|
|
172
|
+
RAILS_ENV=test bundle exec mutineer init --rails
|
|
173
|
+
RAILS_ENV=test bundle exec mutineer run app/models/order.rb --rails
|
|
154
174
|
```
|
|
155
175
|
|
|
156
|
-
|
|
157
|
-
|
|
158
|
-
a mutant on an uncovered line is `no_coverage`, so the score stays comparable to
|
|
159
|
-
the in-process `--rails` score.
|
|
160
|
-
- **Safe `--jobs N`** — each worker routes to its own copy of the test database, so
|
|
161
|
-
parallel verdicts equal serial (no fixture cross-talk). On first use, a
|
|
162
|
-
worker's database is a copy of the test database after the app boots, so rows
|
|
163
|
-
written by initializers or `--require` files are there, as in-process. When
|
|
164
|
-
the copy's schema differs from `db/schema.rb` (another schema version or
|
|
165
|
-
`schema_sha1` checksum, or no stored checksum, as in an empty or out-of-date
|
|
166
|
-
test database, or one set up with a plain `load "db/schema.rb"`), the
|
|
167
|
-
worker loads `db/schema.rb` over it, which drops the copied rows of the
|
|
168
|
-
tables it defines.
|
|
169
|
-
- **One backend at a time** — `--daemon` can't be combined with `--test-command`
|
|
170
|
-
(choose one), and it needs an app to boot (`--rails` or `--boot`).
|
|
171
|
-
|
|
172
|
-
Status: **SQLite** only (hermetic, CI-proven). Per-worker provisioning for
|
|
173
|
-
**Postgres** and other adapters is not supported: on those, `--daemon` scores
|
|
174
|
-
every mutant as `error`. Use `--daemon` with a SQLite test database, or drop
|
|
175
|
-
`--daemon` to run serially on other adapters.
|
|
176
|
-
|
|
177
|
-
### Apps on Ruby < 3.4
|
|
178
|
-
|
|
179
|
-
Mutineer's own process needs Ruby ≥ 3.4 (it parses with stdlib Prism), and the
|
|
180
|
-
`--rails` path above boots your app *inside Mutineer's process* — so it can't run
|
|
181
|
-
against an app pinned to an older Ruby (`ruby "3.1.6"` in the Gemfile), where the
|
|
182
|
-
bundle rejects 3.4.
|
|
183
|
-
|
|
184
|
-
`--test-command` decouples the two: Mutineer stays on ≥ 3.4, but your suite runs
|
|
185
|
-
as a **subprocess in your app's own runtime** (whatever Ruby its bundle resolves
|
|
186
|
-
to). Run Mutineer with a 3.4+ Ruby and hand it the command that runs your tests:
|
|
176
|
+
Without `--test`, `app/models/order.rb` pairs with `test/models/order_test.rb` or `spec/models/order_spec.rb`.
|
|
177
|
+
See [Quick start](https://github.com/davidteren/mutineer/blob/main/docs/quick-start.md).
|
|
187
178
|
|
|
188
|
-
|
|
189
|
-
RAILS_ENV=test mutineer run app/models/order.rb \
|
|
190
|
-
--test test/models/order_test.rb \
|
|
191
|
-
--test-command "bundle exec rails test %{files}"
|
|
192
|
-
```
|
|
193
|
-
|
|
194
|
-
- **`%{files}`** is required; it expands to the `--test` paths as separate
|
|
195
|
-
arguments (a path with a space stays one argument — there is no shell).
|
|
196
|
-
- **Environment:** vars like `RAILS_ENV` / `DATABASE_URL` set on the Mutineer
|
|
197
|
-
command are inherited. Mutineer **unsets** `BUNDLE_*`, `GEM_*`, `RUBY*`,
|
|
198
|
-
`RBENV_VERSION`, `ASDF_RUBY_VERSION`, and `RBENV_DIR` in the child (so
|
|
199
|
-
Mutineer's own Ruby cannot pin the suite), and drops version-manager
|
|
200
|
-
**version bins** (e.g. `~/.rbenv/versions/3.4.x/bin`) from `PATH`, then
|
|
201
|
-
prepends rbenv/asdf shims when a pin was scrubbed. Do not rely on
|
|
202
|
-
`RBENV_VERSION=…` on the Mutineer command for the suite; use `.ruby-version`
|
|
203
|
-
or a wrapper. Don't put `KEY=val` prefixes *inside* `--test-command` (no shell;
|
|
204
|
-
that would be treated as the program name).
|
|
205
|
-
|
|
206
|
-
#### Under a version manager (rbenv / asdf / chruby)
|
|
207
|
-
|
|
208
|
-
Automatic scrub targets **rbenv** and **asdf** (shims + version bins). **chruby**
|
|
209
|
-
has no shims: Mutineer still strips `…/rubies/…/bin` so it cannot leave Mutineer's
|
|
210
|
-
Ruby pinned, but you need a wrapper that sources chruby and selects the app
|
|
211
|
-
version. If the smoke check still reports a **Ruby version mismatch**, wrap the
|
|
212
|
-
suite. Example rbenv wrapper (`bin/mutineer-test` in the app):
|
|
179
|
+
## Usage
|
|
213
180
|
|
|
214
181
|
```sh
|
|
215
|
-
|
|
216
|
-
|
|
217
|
-
|
|
218
|
-
|
|
219
|
-
export RBENV_VERSION="$(cat .ruby-version 2>/dev/null || true)"
|
|
220
|
-
export RAILS_ENV="${RAILS_ENV:-test}"
|
|
221
|
-
export PATH="${HOME}/.rbenv/shims:${PATH}"
|
|
222
|
-
exec bundle exec rails test "$@"
|
|
182
|
+
mutineer run <source...> [--test <test>...] [options]
|
|
183
|
+
mutineer init [--rails] [--force]
|
|
184
|
+
mutineer triage REPORT.json --reason TEXT (--id ID ... | --all)
|
|
185
|
+
mutineer migrate <source...> [--dry-run] [--operators LIST]
|
|
223
186
|
```
|
|
224
187
|
|
|
225
|
-
|
|
226
|
-
# Mutineer on 3.4+; suite on the app's Ruby via the wrapper
|
|
227
|
-
mutineer run app/models/order.rb \
|
|
228
|
-
--test test/models/order_test.rb \
|
|
229
|
-
--test-command "bin/mutineer-test %{files}"
|
|
230
|
-
```
|
|
188
|
+
### Options
|
|
231
189
|
|
|
232
|
-
|
|
233
|
-
`RubyVersionMismatch`, instead of only blaming DB/migrations.
|
|
234
|
-
|
|
235
|
-
Tradeoffs — this path is correct but not free:
|
|
236
|
-
|
|
237
|
-
- **Slower:** your app re-boots for every mutant (no shared boot yet).
|
|
238
|
-
- **No coverage narrowing:** every mutant runs the *full* `--test` set, so the
|
|
239
|
-
score is an **upper bound and not comparable to an in-process (`--rails`)
|
|
240
|
-
score** — uncovered mutants count as survivors, and an infrastructure failure
|
|
241
|
-
is scored as a kill. Mutineer prints this caveat on every run and aborts up
|
|
242
|
-
front (a "smoke check") if your unmutated suite isn't green.
|
|
243
|
-
- **Reload strategy only** (`--strategy redefine` is rejected on this path) and
|
|
244
|
-
**serial** (`--jobs` is forced to 1). For apps on Ruby ≥ 3.4, `--daemon` gives
|
|
245
|
-
safe parallelism instead (see [the daemon backend](https://github.com/davidteren/mutineer#faster-parallel-safe-rails-the---daemon-backend)).
|
|
246
|
-
|
|
247
|
-
## Timeouts and the score
|
|
248
|
-
|
|
249
|
-
The mutation score is `killed / (killed + survived)`. A mutant whose tests run
|
|
250
|
-
past `--timeout` is a `timeout`. It is neither killed nor survived, so it is
|
|
251
|
-
left out of the score, like `no_coverage`, `uncapturable`, `unplaceable`, `ran_at_load`, `errored`, skipped
|
|
252
|
-
and ignored mutants. A timeout is not counted as a kill, because a hang the
|
|
253
|
-
mutant caused and a suite that is just slow look the same. The human report
|
|
254
|
-
shows the count in its `Timeout:` row, and the JSON report in `summary.timeout`.
|
|
255
|
-
|
|
256
|
-
Under `--threshold`, a timeout is a mutant with no verdict. The run exits 1 when
|
|
257
|
-
more than one mutant produced no verdict and they exceed 10% of those attempted,
|
|
258
|
-
or when nothing could be scored and something broke. If a slow suite times out
|
|
259
|
-
mutants that its tests would catch, raise `--timeout`. A limit set below what
|
|
260
|
-
the suite needs does not raise the score: it turns those mutants into timeouts,
|
|
261
|
-
and once they pass 10% of the attempted mutants the run fails.
|
|
262
|
-
|
|
263
|
-
## Suppressing equivalent mutants
|
|
264
|
-
|
|
265
|
-
Some mutants are equivalent (behaviour-identical) and survive forever — keeping a
|
|
266
|
-
file off 100%. Suppress them so the score and `--threshold` gate stay meaningful:
|
|
267
|
-
|
|
268
|
-
- **Inline:** `some_line # mutineer:disable-line` (or scope it: `# mutineer:disable-line comparison`). Put a reason after `--`: `# mutineer:disable-line comparison -- the test checks only 20`.
|
|
269
|
-
- **Config:** a `.mutineer.yml` `ignore:` list of mutant ids. Each survivor's
|
|
270
|
-
`id` is printed in the JSON report, so copy it straight into `ignore:`.
|
|
271
|
-
|
|
272
|
-
Suppressed mutants are excluded from the score (so 100% becomes reachable).
|
|
273
|
-
|
|
274
|
-
## Mutant ids
|
|
275
|
-
|
|
276
|
-
A mutant id is 12 hex characters. It hashes the file path (relative to the
|
|
277
|
-
project root), the method's qualified name, the operator, the mutated code, and
|
|
278
|
-
the mutant's position among identical mutants in that method. When one file has
|
|
279
|
-
two methods with the same qualified name (for example two top-level `def index`
|
|
280
|
-
in two DSL blocks), the second and later ones also hash their position among
|
|
281
|
-
those methods, so their ids differ. The first one's id does not change. An edit
|
|
282
|
-
outside the method does not change the id. Moving or renaming the file,
|
|
283
|
-
renaming the method or its class, or adding an identical mutant earlier in the
|
|
284
|
-
method does. Adding a method with the same name earlier in the same file also
|
|
285
|
-
does.
|
|
286
|
-
|
|
287
|
-
- The project root is the directory mutineer runs from (in the Action, the
|
|
288
|
-
`working-directory`). Run from the same root to get the same ids.
|
|
289
|
-
- A source outside the project root uses its absolute path, so its ids differ
|
|
290
|
-
between machines.
|
|
291
|
-
|
|
292
|
-
**Migrating from ids without the file path.** Before 1.3, ids did not include
|
|
293
|
-
the file path, so two files could share an id (#126). Old-format ids keep
|
|
294
|
-
working until 2.0, with a warning:
|
|
295
|
-
|
|
296
|
-
- **`ignore:`** An old entry still suppresses its mutants. The run prints the
|
|
297
|
-
new ids for each old entry, each with its file and method. When it names one
|
|
298
|
-
mutant, replace the entry with that id. When it names several (in different
|
|
299
|
-
files, or same-named methods in one file), the old entry over-matched: it also
|
|
300
|
-
hid mutants you did not mean to ignore. The warning says so. Keep only the ids for the mutant you meant to
|
|
301
|
-
ignore, not all of them. The list covers only the sources and operators in
|
|
302
|
-
that run, so run over every source with every operator set you use (for
|
|
303
|
-
example your Tier-2 `--operators`) for the full list.
|
|
304
|
-
- **`--baseline`** An old baseline still matches: a survivor matches a stored
|
|
305
|
-
one with the same old id in the same file. A stored file that is an absolute
|
|
306
|
-
path outside the project root (a baseline written on another machine)
|
|
307
|
-
matches on the old id alone. The run tells you to
|
|
308
|
-
regenerate it. Regenerate it with `--format json`, but only after every gate
|
|
309
|
-
that reads it runs 1.3 or later (the Action's `version:` pin, your CI
|
|
310
|
-
`Gemfile.lock`). An older version treats every new-format survivor as new.
|
|
311
|
-
|
|
312
|
-
The JSON report's `summary.id_format` is `2` for the new format.
|
|
313
|
-
`summary.legacy_id_matches.ignore` counts the old-format ignore entries a run
|
|
314
|
-
matched, and `summary.legacy_id_matches.baseline` counts the survivors matched
|
|
315
|
-
only through an old baseline id.
|
|
316
|
-
|
|
317
|
-
Ids are relative to the directory you run mutineer from. mutineer finds
|
|
318
|
-
`.mutineer.yml` by walking up. When the file it loads is in a parent directory
|
|
319
|
-
(other than your home directory), it warns that the ignore ids will not match
|
|
320
|
-
and tells you which directory to run from.
|
|
321
|
-
|
|
322
|
-
## Kill matrix
|
|
323
|
-
|
|
324
|
-
`--matrix` reports which tests kill each mutant, and from that, which tests
|
|
325
|
-
add nothing to the suite.
|
|
190
|
+
Full flags, defaults, and `.mutineer.yml` keys: [CLI reference](https://github.com/davidteren/mutineer/blob/main/docs/cli.md).
|
|
326
191
|
|
|
327
|
-
|
|
328
|
-
|
|
329
|
-
|
|
192
|
+
| Flag | Meaning |
|
|
193
|
+
|------|---------|
|
|
194
|
+
| `--test FILE` | Repeat per test file; omit to pair tests by convention |
|
|
195
|
+
| `--threshold FLOAT` | Exit 1 when the score is below FLOAT, or when nothing could be scored and something broke, or more than one mutant produced no verdict and they exceed 10% of those attempted (default: 0 = off) |
|
|
196
|
+
| `--since REF` | Mutate only lines changed since git `REF`, such as `origin/main` |
|
|
197
|
+
| `--baseline FILE` | Exit 1 for new survivors, or score drops on full runs, against a prior JSON report from a full run |
|
|
198
|
+
| `--rails` | Boot `config/environment` once; serial without `--daemon` |
|
|
199
|
+
| `--[no-]daemon` | Enable or disable daemon; isolated worker databases, Minitest only |
|
|
200
|
+
| `--jobs N` | Worker count; default: one per CPU |
|
|
201
|
+
| `--format human\|json\|html` | Report format; default: human |
|
|
202
|
+
| `--dry-run` | List mutants without running tests |
|
|
203
|
+
|
|
204
|
+
### Exit codes
|
|
330
205
|
|
|
331
|
-
|
|
332
|
-
|
|
333
|
-
|
|
334
|
-
|
|
335
|
-
|
|
336
|
-
|
|
337
|
-
|
|
338
|
-
names two kinds of test:
|
|
339
|
-
|
|
340
|
-
- **Blind:** the test ran in at least one complete row and killed no mutant
|
|
341
|
-
in any row. It would still pass with this run's mutations of the code, so
|
|
342
|
-
it guards none of them. It can still check behavior outside this run's
|
|
343
|
-
mutants (code in other files, or edits no operator makes), so look at what
|
|
344
|
-
it asserts before you delete it, or give it an assertion that can fail.
|
|
345
|
-
- **Redundant:** every mutant the test kills, another test kills too. That
|
|
346
|
-
makes it a candidate for deletion, one test at a time: two redundant tests
|
|
347
|
-
can be the only killers of one mutant, so re-run after each deletion.
|
|
348
|
-
|
|
349
|
-
The answers cover this run's mutants only. A test of code outside the sources
|
|
350
|
-
you passed kills nothing here, so mutate the code a test exercises before you
|
|
351
|
-
call the test blind.
|
|
352
|
-
|
|
353
|
-
The matrix changes no verdict, score or exit code: each mutant gets the
|
|
354
|
-
verdict a run without `--matrix` gives. That run skips every later test,
|
|
355
|
-
test class and example group after the first failure, so when a later one
|
|
356
|
-
exits the process, crashes or runs into the time limit, the mutant is still
|
|
357
|
-
`killed`. The run without `--matrix` still runs the code around the failing
|
|
358
|
-
test (the rest of its Minitest class `run` wrapper, the `after(:all)` hooks of
|
|
359
|
-
its RSpec groups) and the suite's own cleanup (RSpec's `after(:suite)`), so an
|
|
360
|
-
end there keeps the exit status. A failure in a Minitest `parallelize_me!`
|
|
361
|
-
class also keeps it: those run after every serial class and cannot be stopped
|
|
362
|
-
once queued, while a failure in a serial class still stops the run.
|
|
363
|
-
|
|
364
|
-
Under RSpec, `--matrix` needs RSpec 3.3 or later, whose example ids tell
|
|
365
|
-
apart examples on one line. With an older RSpec every row is incomplete.
|
|
366
|
-
|
|
367
|
-
The human report lists up to 20 blind and 20 redundant tests, and the HTML
|
|
368
|
-
report lists them all. `--format json` adds a `matrix` block with every test
|
|
369
|
-
and each mutant's killers (see the
|
|
370
|
-
[JSON schema](https://davidteren.github.io/mutineer/json-schema.html#matrix)).
|
|
371
|
-
A test is its file and its id. The name is `CalculatorTest#test_add` under
|
|
372
|
-
Minitest, or the example's full description under RSpec, where the example id
|
|
373
|
-
(`./spec/calc_spec.rb[1:2]`) tells apart examples that share a description
|
|
374
|
-
and keeps an example whose generated description changes with the mutant as
|
|
375
|
-
one test.
|
|
376
|
-
|
|
377
|
-
Each mutant runs its whole covering set, so every mutant costs what a survivor
|
|
378
|
-
costs. In 1.4, stopping at the first failure cut a full run of rack's
|
|
379
|
-
`lib/rack/utils.rb` from 86 to 89 seconds down to 35 to 41, so expect a matrix
|
|
380
|
-
run to take 2.1 to 2.5 times as long as a normal one. A mutant that reaches the
|
|
381
|
-
per-mutant time limit (`--timeout`, 10 seconds by default) after a test already failed stays `killed`
|
|
382
|
-
with its row marked incomplete. The report names each incomplete row and warns
|
|
383
|
-
that a blind test may have killed one of those mutants, so do not delete a
|
|
384
|
-
blind test until its rows are complete. Raise `--timeout` (or `timeout:` in
|
|
385
|
-
`.mutineer.yml`) to get complete rows.
|
|
386
|
-
|
|
387
|
-
`--matrix` runs on the in-process backend only. It exits 2 with `--daemon`,
|
|
388
|
-
`--test-command`, `--fail-fast` or `--dry-run`, and the message says whether
|
|
389
|
-
each setting came from the command line or `.mutineer.yml`. To run one of
|
|
390
|
-
those with `matrix: true` in the file, pass `--no-matrix`; to run `--matrix`
|
|
391
|
-
with `fail_fast: true` in the file, pass `--no-fail-fast`.
|
|
392
|
-
|
|
393
|
-
## CI gating
|
|
394
|
-
|
|
395
|
-
Store a JSON run as a baseline, then fail the build only when a PR makes things
|
|
396
|
-
worse:
|
|
206
|
+
<!-- contract:exit-codes -->
|
|
207
|
+
| Code | Meaning |
|
|
208
|
+
|------|---------|
|
|
209
|
+
| `0` | Score ≥ threshold (or no gate) **and** no baseline regression. |
|
|
210
|
+
| `1` | Score below `--threshold`, OR nothing could be scored and something broke, or more than one mutant produced no verdict and they exceed 10% of those attempted, OR a `--baseline` regression, OR a runtime error, including a boot, source or `require:` file that raises while loading. |
|
|
211
|
+
| `2` | Usage / invalid-flag error, including a mistyped flag, a bad path (such as a missing `--boot` or `require:` file), or an unreadable baseline. |
|
|
212
|
+
<!-- /contract:exit-codes -->
|
|
397
213
|
|
|
398
|
-
|
|
399
|
-
mutineer run app/ --baseline .mutineer/baseline.json # exit 1 on NEW survivors or a score drop
|
|
400
|
-
```
|
|
214
|
+
### Operators
|
|
401
215
|
|
|
402
|
-
|
|
403
|
-
|
|
404
|
-
|
|
405
|
-
|
|
406
|
-
`
|
|
407
|
-
`
|
|
408
|
-
|
|
409
|
-
|
|
410
|
-
|
|
411
|
-
|
|
412
|
-
|
|
413
|
-
|
|
414
|
-
|
|
415
|
-
|
|
416
|
-
|
|
417
|
-
|
|
418
|
-
|
|
419
|
-
|
|
420
|
-
|
|
421
|
-
|
|
422
|
-
|
|
423
|
-
|
|
424
|
-
threshold: "90"
|
|
425
|
-
```
|
|
216
|
+
List with `mutineer --list-operators`. Enable specific operators with `--operators`.
|
|
217
|
+
|
|
218
|
+
| Group | Operators |
|
|
219
|
+
|---|---|
|
|
220
|
+
| Tier 1 (default) | `arithmetic`, `comparison`, `boolean_connector`, `boolean_literal`, `statement_removal` |
|
|
221
|
+
| Tier 2 (opt-in) | `return_nil`, `literal_mutation`, `condition_negation`, `string_literal`, `regex`, `collection_method`, `safe_navigation`, `range`, `negation_removal`, `chain_link`, `call_to_receiver`, `operand_removal`, `array_literal`, `condition_true`, `condition_false`, `operator_assignment` |
|
|
222
|
+
|
|
223
|
+
## Documentation
|
|
224
|
+
|
|
225
|
+
| Guide | What it covers |
|
|
226
|
+
|-------|----------------|
|
|
227
|
+
| [Quick start](https://github.com/davidteren/mutineer/blob/main/docs/quick-start.md) | Install, run, fix a survivor |
|
|
228
|
+
| [First run to steady state](https://github.com/davidteren/mutineer/blob/main/docs/getting-started.md) | One file through changed-code CI |
|
|
229
|
+
| [Reading results](https://github.com/davidteren/mutineer/blob/main/docs/results.md) | Reports, survivors, suppression, mutant ids, kill matrix |
|
|
230
|
+
| [Rails setup](https://github.com/davidteren/mutineer/blob/main/docs/rails-setup.md) | Rails, daemon, databases, Spring, Bootsnap, Ruby < 3.4 |
|
|
231
|
+
| [Rails rollout](https://davidteren.github.io/mutineer/rails.html) | Four stages from model to CI gate |
|
|
232
|
+
| [CI: changed code and baselines](https://github.com/davidteren/mutineer/blob/main/docs/ci.md) | `--since`, `--baseline`, Action inputs, timeouts |
|
|
233
|
+
| [CLI reference](https://github.com/davidteren/mutineer/blob/main/docs/cli.md) | Flags, defaults, `.mutineer.yml` |
|
|
234
|
+
| [Comparison](https://github.com/davidteren/mutineer/blob/main/docs/comparison.md) | Mutineer, Mutant, other Ruby tools, fit-check prompt |
|
|
235
|
+
| [AI agents and CI](https://davidteren.github.io/mutineer/agentic-coding.html) | Agent loop and CI recipes |
|
|
236
|
+
| [JSON schema](https://davidteren.github.io/mutineer/json-schema.html) | Versioned report contract |
|
|
237
|
+
| [STABILITY.md](https://github.com/davidteren/mutineer/blob/main/STABILITY.md) | Release guarantees |
|
|
426
238
|
|
|
427
|
-
|
|
428
|
-
action scopes the run to the PR's changed lines, diffing against the PR's exact
|
|
429
|
-
base commit (fetched by the action itself when the checkout is shallow; falls
|
|
430
|
-
back to the base branch tip). Pass `since: none` for a full scan, or an
|
|
431
|
-
explicit `since:` ref (which needs `fetch-depth: 0` on checkout).
|
|
239
|
+
## For AI agents & pipelines
|
|
432
240
|
|
|
433
|
-
|
|
241
|
+
Use versioned JSON, structured exit codes, and runs scoped to changed lines.
|
|
242
|
+
[Mutant ids](https://github.com/davidteren/mutineer/blob/main/docs/results.md#mutant-ids) survive unrelated edits.
|
|
434
243
|
|
|
435
|
-
-
|
|
436
|
-
-
|
|
437
|
-
gate failed, `warning` when it passed);
|
|
438
|
-
- exposes the report path via the `report` output for later steps (with
|
|
439
|
-
`format: human`/`html` this needs the `output` input).
|
|
244
|
+
<a id="mutant-ids"></a><a id="suppressing-equivalent-mutants"></a><a id="apps-on-ruby--34"></a>
|
|
245
|
+
Mutant ids and equivalent-mutant suppression moved to [Reading results](https://github.com/davidteren/mutineer/blob/main/docs/results.md). Apps on Ruby < 3.4 moved to [Rails setup](https://github.com/davidteren/mutineer/blob/main/docs/rails-setup.md#apps-on-ruby--34).
|
|
440
246
|
|
|
441
|
-
|
|
442
|
-
|
|
247
|
+
| Resource | Links and use |
|
|
248
|
+
|---|---|
|
|
249
|
+
| AI agents & CI recipes | [rendered](https://davidteren.github.io/mutineer/agentic-coding.html) · [source](https://davidteren.github.io/mutineer/agentic-coding.md): agent loops, CI gates, avoiding equivalent-mutant loops |
|
|
250
|
+
| JSON schema reference | [rendered](https://davidteren.github.io/mutineer/json-schema.html) · [source](https://davidteren.github.io/mutineer/json-schema.md): `--format json` schema and versioning |
|
|
251
|
+
| Ruby API (YARD) | [https://davidteren.github.io/mutineer/api/](https://davidteren.github.io/mutineer/api/): shipped classes |
|
|
252
|
+
| Agent skill | [`skills/mutineer/SKILL.md`](https://github.com/davidteren/mutineer/blob/main/skills/mutineer/SKILL.md): installation, agent loop, exit codes |
|
|
443
253
|
|
|
444
|
-
|
|
254
|
+
Install the skill with either command:
|
|
445
255
|
|
|
446
|
-
|
|
447
|
-
|
|
448
|
-
|
|
449
|
-
- **AI agents & CI recipes** — the agent inner-loop and CI-gate recipes (and how
|
|
450
|
-
to avoid infinite loops on equivalent mutants):
|
|
451
|
-
[rendered](https://davidteren.github.io/mutineer/agentic-coding.html) ·
|
|
452
|
-
[source](https://davidteren.github.io/mutineer/agentic-coding.md)
|
|
453
|
-
- **JSON schema reference** — the `--format json` schema and its versioning
|
|
454
|
-
contract:
|
|
455
|
-
[rendered](https://davidteren.github.io/mutineer/json-schema.html) ·
|
|
456
|
-
[source](https://davidteren.github.io/mutineer/json-schema.md)
|
|
457
|
-
- **Ruby API (YARD)** — class reference for the shipped gem:
|
|
458
|
-
[https://davidteren.github.io/mutineer/api/](https://davidteren.github.io/mutineer/api/)
|
|
459
|
-
- **Agent skill** ([`skills/mutineer/SKILL.md`](https://github.com/davidteren/mutineer/blob/main/skills/mutineer/SKILL.md)): a short
|
|
460
|
-
card for coding agents with install, the agent loop, and exit codes. Install it
|
|
461
|
-
with `gh skill install davidteren/mutineer mutineer` or
|
|
462
|
-
`npx skills add davidteren/mutineer --skill mutineer`.
|
|
463
|
-
|
|
464
|
-
## Configuration
|
|
465
|
-
|
|
466
|
-
Mutineer reads an optional `.mutineer.yml` from the project root (nearest one,
|
|
467
|
-
walking up). CLI flags override config; config overrides defaults.
|
|
468
|
-
|
|
469
|
-
Sources are positional CLI arguments and test files come from `--test`. The
|
|
470
|
-
config file accepts these keys:
|
|
471
|
-
|
|
472
|
-
| Key | Value and purpose |
|
|
473
|
-
|-----|-------------------|
|
|
474
|
-
| `operators` | An operator name or list of names; defaults to the Tier-1 set |
|
|
475
|
-
| `threshold` | A number from 0 to 100; 0 turns the score gate off |
|
|
476
|
-
| `jobs` | A positive integer; the default is the processor count. `test_command`, `fail_fast`, or `--rails` without `--daemon` forces 1. |
|
|
477
|
-
| `only` | A fully-qualified subject name, such as `Calculator#add` |
|
|
478
|
-
| `require` | A path or list of extra files to load before mutating |
|
|
479
|
-
| `boot` | The app entry point to require once before forking |
|
|
480
|
-
| `rails` | `true` or `false`; enables the Rails boot defaults |
|
|
481
|
-
| `since` | A nonblank git ref, or `false` to disable diff scoping |
|
|
482
|
-
| `framework` | `minitest` or `rspec`; an explicit value is kept during test pairing |
|
|
483
|
-
| `verbose` | `true` or `false`; shows capture diagnostics |
|
|
484
|
-
| `ignore` | A mutant id or list of ids to suppress |
|
|
485
|
-
| `baseline` | The path to a prior JSON report |
|
|
486
|
-
| `fail_fast` | `true` or `false`; stops scheduling after the first survivor |
|
|
487
|
-
| `test_command` | The external-runtime suite command, including `%{files}`; see [Apps on Ruby < 3.4](https://github.com/davidteren/mutineer#apps-on-ruby--34) |
|
|
488
|
-
| `daemon` | `true` or `false`; uses the persistent app daemon with worker DB isolation |
|
|
489
|
-
| `timeout` | A positive integer; the per-mutant time limit in seconds (default 10) |
|
|
490
|
-
| `capture_timeout` | A positive integer; the coverage-capture time limit in seconds (default 120) |
|
|
491
|
-
| `cache_dir` | The coverage cache directory (default `.mutineer`) |
|
|
492
|
-
| `matrix` | `true` or `false`; runs every covering test for each mutant and adds the kill matrix to the report |
|
|
493
|
-
|
|
494
|
-
Invalid values for known scalar keys, and a blank `operators` list, exit 2
|
|
495
|
-
with a message naming the file and key. An unknown operator name warns and
|
|
496
|
-
is skipped. If none of the names are known, the run exits 2. An empty
|
|
497
|
-
`require` or `ignore` list is valid. Boolean keys take `true` or `false` (quoted forms also work), not `"yes"`.
|
|
498
|
-
`jobs` must be positive; a string value contains digits only. String values for
|
|
499
|
-
`threshold` and the CLI-only `--baseline-epsilon` use plain decimals such as `90`
|
|
500
|
-
or `0.5`, not `+2`, `1e2`, or `1_0`. String options such as `only` and `baseline`
|
|
501
|
-
cannot be null or boolean. A blank `since` is invalid; use `since: false` to turn
|
|
502
|
-
scoping off. Unknown keys warn and are ignored. Unknown operator names warn and are skipped, and the run exits 2 when none remain. `--operators` replaces a blank or unknown file list.
|
|
503
|
-
|
|
504
|
-
`format`, `strategy`, `output`, `baseline_epsilon`, and `dry_run` are CLI-only.
|
|
505
|
-
For JSON output, use `--format json`, not a `format:` config key. To select RSpec
|
|
506
|
-
in the file, add `framework: rspec`.
|
|
507
|
-
|
|
508
|
-
```yaml
|
|
509
|
-
# .mutineer.yml
|
|
510
|
-
operators: [arithmetic, comparison, boolean_connector, boolean_literal, statement_removal]
|
|
511
|
-
threshold: 90
|
|
512
|
-
jobs: 4
|
|
513
|
-
require:
|
|
514
|
-
- config/environment
|
|
515
|
-
```
|
|
256
|
+
- `gh skill install davidteren/mutineer mutineer`
|
|
257
|
+
- `npx skills add davidteren/mutineer --skill mutineer`
|
|
516
258
|
|
|
517
|
-
|
|
518
|
-
|
|
519
|
-
|
|
520
|
-
`.mutineer/` ignored even then: `--test-command` writes lock files to
|
|
521
|
-
`.mutineer/` beside each source file.
|
|
522
|
-
|
|
523
|
-
## Mutineer and Mutant
|
|
524
|
-
|
|
525
|
-
[Mutant](https://github.com/mbj/mutant), by Markus Schirp, is the established
|
|
526
|
-
mutation-testing tool for Ruby. It has been in development since 2012 and is
|
|
527
|
-
the subject of published research. Mutineer is a separate tool, and it shares
|
|
528
|
-
no code with Mutant.
|
|
529
|
-
|
|
530
|
-
| Aspect | Mutineer | Mutant |
|
|
531
|
-
|---|---|---|
|
|
532
|
-
| License | MIT, for every use | Free for open source (`--usage opensource`); commercial use needs a paid subscription |
|
|
533
|
-
| Runtime dependencies | None (Prism + stdlib) | `parser`, `unparser`, `regexp_parser`, `sorbet-runtime` and others |
|
|
534
|
-
| Ruby | 3.4 and later to run Mutineer; an app on an older Ruby works through `--test-command` (see [Apps on Ruby < 3.4](https://github.com/davidteren/mutineer#apps-on-ruby--34)) | 3.3 and later |
|
|
535
|
-
| Test frameworks | Minitest and RSpec, in one gem | RSpec, Minitest and Test::Unit, one integration gem each |
|
|
536
|
-
| What you mutate | Files (`mutineer run lib/foo.rb --test test/foo_test.rb`), narrowed to one method with `--only 'Foo#bar'` | Subjects named by expression (`mutant run 'Foo#bar'`, `'Foo*'`) |
|
|
537
|
-
| Which tests run | The test files whose coverage reaches the mutated line | The tests that declare the subject (an RSpec description or a Minitest `cover`), or the tests that ran it in a per-test coverage recording |
|
|
538
|
-
| Operators | 20 (5 by default, 15 opt-in) | A larger set; the default `light` set applies almost all of it (`full` adds `#==` to `#eql?`) |
|
|
539
|
-
| CI gating | `--threshold`, `--baseline` deltas, `--since`, and a GitHub Action | Incremental mode (`--since`) and a recorded session history |
|
|
540
|
-
| Machine-readable output | Versioned JSON report schema | Session JSON schema |
|
|
541
|
-
|
|
542
|
-
Choose Mutineer when you want an MIT-licensed tool with no extra gems in your
|
|
543
|
-
bundle, file-based runs, and a JSON contract built for CI gates and AI agents.
|
|
544
|
-
Choose Mutant when you need its deeper operator set, Test::Unit support, or
|
|
545
|
-
subject expressions that select whole namespaces. The two tools do not
|
|
546
|
-
conflict, so you can run both on one project.
|
|
547
|
-
|
|
548
|
-
This comparison describes Mutant 0.17 (checked on 2026-10-06). Check
|
|
549
|
-
[Mutant's README](https://github.com/mbj/mutant#readme) for its current
|
|
550
|
-
license terms and features.
|
|
259
|
+
**Mutineer 1.8.0 and later** bundle the skill and offer
|
|
260
|
+
[ Rails Hyperdrive](https://github.com/rails-hyperdrive/rails-hyperdrive) companion installation.
|
|
261
|
+
For Rails apps bundling Mutineer, `bin/rails hyperdrive:init` installs it to `.claude/skills/mutineer/SKILL.md`.
|
|
551
262
|
|
|
552
263
|
## License
|
|
553
264
|
|
|
554
|
-
MIT
|
|
265
|
+
MIT. See [LICENSE](https://github.com/davidteren/mutineer/blob/main/LICENSE).
|