active_mutator 0.1.0 → 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +160 -63
- data/lib/active_mutator/accepted_ledger.rb +23 -8
- data/lib/active_mutator/atomic_file.rb +1 -1
- data/lib/active_mutator/baseline.rb +13 -4
- data/lib/active_mutator/baseline_delta.rb +67 -1
- data/lib/active_mutator/baseline_hooks.rb +2 -2
- data/lib/active_mutator/cli.rb +16 -4
- data/lib/active_mutator/config.rb +2 -1
- data/lib/active_mutator/config_file.rb +89 -0
- data/lib/active_mutator/coverage_map.rb +1 -1
- data/lib/active_mutator/defined_constants.rb +48 -0
- data/lib/active_mutator/edit.rb +8 -2
- data/lib/active_mutator/engine.rb +15 -2
- data/lib/active_mutator/fingerprint.rb +1 -1
- data/lib/active_mutator/inserter.rb +6 -3
- data/lib/active_mutator/operators/base.rb +2 -1
- data/lib/active_mutator/operators/call_swap.rb +16 -0
- data/lib/active_mutator/operators/literal.rb +14 -2
- data/lib/active_mutator/reporter/github.rb +36 -0
- data/lib/active_mutator/reporter/json.rb +1 -0
- data/lib/active_mutator/reporter/operator_stats.rb +20 -0
- data/lib/active_mutator/reporter/stryker_json.rb +117 -0
- data/lib/active_mutator/reporter/terminal.rb +11 -0
- data/lib/active_mutator/runner.rb +121 -17
- data/lib/active_mutator/scheduler.rb +64 -7
- data/lib/active_mutator/source_location.rb +21 -0
- data/lib/active_mutator/subject.rb +7 -1
- data/lib/active_mutator/subject_finder.rb +46 -7
- data/lib/active_mutator/subject_matcher.rb +23 -0
- data/lib/active_mutator/timeout_calibrator.rb +75 -0
- data/lib/active_mutator/version.rb +1 -1
- data/lib/active_mutator/work_item.rb +8 -1
- data/lib/active_mutator/worker.rb +5 -2
- data/lib/active_mutator.rb +8 -0
- metadata +13 -5
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 5729b9385c24e575708a5227a3838ae28e09dd2ddf3e40e1883e9e74f7399275
|
|
4
|
+
data.tar.gz: 69b41b3304c8c008f54f6c20547ff9283d26a278eea706dc7cf923466dde1b5c
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 3e0dce87a35314cb96e68e7051e3dfa8bbc329b58a3c41beef1201c7c5e8a3205d58a28593bf99382dfd1f9a9fd52859b44def8e2ee5d57dd0879e9dfcadec37
|
|
7
|
+
data.tar.gz: 5ec63925341ef8356801ee171e83af5b6323feb0f15838b1430421c9e5aaf73017d0e5ebaafb653039917a97144933e3572c2d182ba0544de27fbe60a0d7d6ca
|
data/README.md
CHANGED
|
@@ -1,13 +1,16 @@
|
|
|
1
1
|
# active_mutator
|
|
2
2
|
|
|
3
|
+
[](https://rubygems.org/gems/active_mutator)
|
|
4
|
+
|
|
3
5
|
Mutation testing for Ruby, built on [Prism](https://github.com/ruby/prism).
|
|
4
|
-
Open source, RSpec-integrated, Rails-first.
|
|
6
|
+
Open source, RSpec-integrated, Rails-first. Available on
|
|
7
|
+
[RubyGems](https://rubygems.org/gems/active_mutator).
|
|
5
8
|
|
|
6
|
-
active_mutator mutates your code one small change at a time (`>`
|
|
7
|
-
`&&`
|
|
8
|
-
examples that cover the mutated line, and reports
|
|
9
|
-
fails to kill. A surviving mutant is a behavior
|
|
10
|
-
a precise, machine-verified test gap.
|
|
9
|
+
active_mutator mutates your code one small change at a time (`>` becomes `>=`,
|
|
10
|
+
`&&` becomes `||`, a statement gets deleted, a condition gets forced, and so
|
|
11
|
+
on). It runs exactly the examples that cover the mutated line, and reports
|
|
12
|
+
every mutant your suite fails to kill. A surviving mutant is a behavior
|
|
13
|
+
change no test notices: a precise, machine-verified test gap.
|
|
11
14
|
|
|
12
15
|
## A surviving mutant, in one example
|
|
13
16
|
|
|
@@ -35,31 +38,31 @@ Surviving mutants:
|
|
|
35
38
|
+ total <= 100
|
|
36
39
|
```
|
|
37
40
|
|
|
38
|
-
Nothing in the test suite calls `discount(100)
|
|
39
|
-
and `<=` disagree. The tests pass, coverage is green
|
|
40
|
-
still unverified. That gap is invisible to coverage and obvious to
|
|
41
|
+
Nothing in the test suite calls `discount(100)`, the one input where `<`
|
|
42
|
+
and `<=` disagree. The tests pass, and coverage is green. But the boundary
|
|
43
|
+
is still unverified. That gap is invisible to coverage and obvious to
|
|
41
44
|
mutation testing. Add `it { expect(calc.discount(100)).to eq(0) }` and the
|
|
42
45
|
mutant is killed.
|
|
43
46
|
|
|
44
47
|
## What is mutation testing?
|
|
45
48
|
|
|
46
49
|
Coverage answers "did a test run this line?" Mutation testing answers "would
|
|
47
|
-
a test *notice* if this line were wrong?"
|
|
48
|
-
|
|
50
|
+
a test *notice* if this line were wrong?" That is a different, and usually
|
|
51
|
+
more useful, question.
|
|
49
52
|
|
|
50
53
|
active_mutator applies one small, syntactically valid change to your code
|
|
51
54
|
(a "mutant") and re-runs only the examples that cover it. If a test fails,
|
|
52
55
|
the mutant is **killed**: your tests correctly reject that wrong behavior.
|
|
53
56
|
If every covering test still passes, the mutant **survived**: something
|
|
54
|
-
changed and nothing noticed. A survivor is not a hypothetical
|
|
55
|
-
exact line, the exact before
|
|
57
|
+
changed and nothing noticed. A survivor is not a hypothetical. It is the
|
|
58
|
+
exact line, the exact before and after diff, and proof that no assertion
|
|
56
59
|
depends on the difference.
|
|
57
60
|
|
|
58
61
|
Mutation score is `(killed + timeout) / (killed + timeout + survived)`.
|
|
59
|
-
100% is usually not the right target
|
|
60
|
-
*equivalent* to the original and can never be killed by any test
|
|
61
|
-
why active_mutator has a committed acceptance ledger
|
|
62
|
-
out with a stated reason instead of chasing an unreachable score.
|
|
62
|
+
100% is usually not the right target. Some mutants are behaviorally
|
|
63
|
+
*equivalent* to the original and can never be killed by any test. That is
|
|
64
|
+
why active_mutator has a committed acceptance ledger. It lets you close
|
|
65
|
+
survivors out with a stated reason instead of chasing an unreachable score.
|
|
63
66
|
|
|
64
67
|
Full primer, including the origin of the technique and further reading:
|
|
65
68
|
**[`docs/guides/what-is-mutation-testing.md`](docs/guides/what-is-mutation-testing.md)**.
|
|
@@ -73,7 +76,7 @@ group :development, :test do
|
|
|
73
76
|
end
|
|
74
77
|
```
|
|
75
78
|
|
|
76
|
-
Requires Ruby
|
|
79
|
+
Requires Ruby 3.2 or later, RSpec, and a green suite. Linux/macOS (MRI fork).
|
|
77
80
|
|
|
78
81
|
## Quick start
|
|
79
82
|
|
|
@@ -82,9 +85,10 @@ bundle install
|
|
|
82
85
|
bundle exec active_mutator app/models/calculator.rb
|
|
83
86
|
```
|
|
84
87
|
|
|
85
|
-
|
|
86
|
-
coverage map
|
|
87
|
-
|
|
88
|
+
The first run performs an instrumented baseline of your suite to build the
|
|
89
|
+
coverage map. The map is cached in `.active_mutator/` and refreshed
|
|
90
|
+
incrementally after that (see
|
|
91
|
+
[`docs/guides/how-it-works.md`](docs/guides/how-it-works.md)).
|
|
88
92
|
Then each mutant runs in its own fork against only its covering examples.
|
|
89
93
|
|
|
90
94
|
### Reading the output
|
|
@@ -115,52 +119,84 @@ Each character on the progress line is one mutant, printed as it finishes:
|
|
|
115
119
|
|
|
116
120
|
| Char | Status | Meaning |
|
|
117
121
|
|---|---|---|
|
|
118
|
-
| `.` | `killed` | a covering test failed
|
|
119
|
-
| `S` | `survived` | every covering test passed
|
|
120
|
-
| `T` | `timeout` | ran past its time budget
|
|
122
|
+
| `.` | `killed` | a covering test failed. Good, the mutant is dead |
|
|
123
|
+
| `S` | `survived` | every covering test passed. This is a test gap |
|
|
124
|
+
| `T` | `timeout` | ran past its time budget. Counted as detected (likely an infinite loop) |
|
|
121
125
|
| `E` | `error` | the worker crashed, or the mutated code raised outside a test assertion |
|
|
122
|
-
| `U` | `uncovered` | no test executes the mutated line at all
|
|
123
|
-
| `A` | `accepted` | matches a known-equivalent entry in the acceptance ledger
|
|
126
|
+
| `U` | `uncovered` | no test executes the mutated line at all. This is coverage debt, worse than a survivor |
|
|
127
|
+
| `A` | `accepted` | matches a known-equivalent entry in the acceptance ledger. Excluded from the score |
|
|
124
128
|
|
|
125
129
|
`invalid` mutants (edits that don't even re-parse as valid Ruby) are
|
|
126
130
|
discarded before scheduling and reported as a count only. Exit code is `1`
|
|
127
|
-
|
|
128
|
-
only `uncovered
|
|
131
|
+
if unaccepted survivors exist (or, with `--fail-at`, if the score is below
|
|
132
|
+
the threshold), `0` otherwise, including when there are only `uncovered`,
|
|
133
|
+
`accepted`, or `error` results. The JSON report's `exit_reason` field
|
|
134
|
+
reflects survivor presence, independent of the `--fail-at` gate.
|
|
135
|
+
|
|
136
|
+
When survivors exist, the summary also prints a per-operator table showing
|
|
137
|
+
how often each operator's mutants survive, to help spot likely-equivalent
|
|
138
|
+
mutant patterns.
|
|
129
139
|
|
|
130
140
|
## How it works, compactly
|
|
131
141
|
|
|
132
|
-
1. **Subject discovery
|
|
142
|
+
1. **Subject discovery**: a Prism visitor finds every method (`def`) in
|
|
133
143
|
your target files.
|
|
134
|
-
2. **Source-span edits
|
|
135
|
-
against the original file, not a rewritten AST
|
|
144
|
+
2. **Source-span edits**: each operator emits byte-range text edits
|
|
145
|
+
against the original file, not a rewritten AST. Every mutant is
|
|
136
146
|
re-parsed with Prism and discarded (`invalid`) if the edit produced
|
|
137
147
|
something that doesn't parse. No unparser is ever built or maintained.
|
|
138
|
-
3. **Coverage-mapped test selection
|
|
139
|
-
every source line to the examples that cover it
|
|
148
|
+
3. **Coverage-mapped test selection**: one instrumented baseline run maps
|
|
149
|
+
every source line to the examples that cover it. Incremental runs
|
|
140
150
|
refresh only what changed instead of re-running the whole suite.
|
|
141
|
-
4. **Fork-per-mutant kill runs
|
|
142
|
-
helper once
|
|
151
|
+
4. **Fork-per-mutant kill runs**: the parent preloads your app and spec
|
|
152
|
+
helper once. Each mutant is inserted and exercised in its own fork
|
|
143
153
|
against just its covering examples, so results can't bleed state
|
|
144
154
|
between mutants.
|
|
145
155
|
|
|
146
156
|
Full architecture, including the coverage-cache format, the fork pipeline,
|
|
147
|
-
the serial lane for browser specs, timeout budgets, and every status
|
|
157
|
+
the serial lane for browser specs, timeout budgets, and every status, is in
|
|
148
158
|
**[`docs/guides/how-it-works.md`](docs/guides/how-it-works.md)**.
|
|
149
159
|
|
|
150
160
|
## Usage
|
|
151
161
|
|
|
152
162
|
```bash
|
|
153
163
|
active_mutator # mutate app/ and lib/, full run
|
|
154
|
-
active_mutator app/models # scope by path
|
|
164
|
+
active_mutator app/models # scope by path (directory)
|
|
165
|
+
active_mutator app/models/document.rb # scope to a single file
|
|
155
166
|
active_mutator --changed # uncommitted work only (dev loop)
|
|
156
167
|
active_mutator --since origin/main # PR scope (CI)
|
|
157
168
|
active_mutator --subject 'Foo::Bar#baz' # one method
|
|
169
|
+
active_mutator --exclude 'lib/generated' # skip a subtree (repeatable)
|
|
170
|
+
```
|
|
171
|
+
|
|
172
|
+
`--subject` also takes broader expressions: `Foo::Bar` (all methods of
|
|
173
|
+
that constant), `Foo::Bar*` (raw name prefix — matches `Foo::Bar::Qux`
|
|
174
|
+
and also `Foo::Barn`), `Foo::Bar#*` (instance
|
|
175
|
+
methods only), `Foo::Bar.*` (singleton methods only).
|
|
176
|
+
|
|
177
|
+
`--exclude PAT` is a glob relative to the project root, applied during
|
|
178
|
+
subject discovery, and gitignore-like: `lib/generated`, `lib/generated/`,
|
|
179
|
+
and `lib/generated/**` all exclude the whole subtree. File globs like
|
|
180
|
+
`**/legacy/*` work too.
|
|
181
|
+
|
|
182
|
+
Skip a single method by putting `# active_mutator:skip` on the line above
|
|
183
|
+
its `def`:
|
|
184
|
+
|
|
185
|
+
```ruby
|
|
186
|
+
# active_mutator:skip
|
|
187
|
+
def legacy_delegator
|
|
188
|
+
target.call
|
|
189
|
+
end
|
|
158
190
|
```
|
|
159
191
|
|
|
160
|
-
Statuses: `killed` (test failed
|
|
161
|
-
(counts as detected), `uncovered` (no covering example
|
|
162
|
-
`accepted` (known-equivalent, see ledger), `error`,
|
|
163
|
-
|
|
192
|
+
Statuses: `killed` (test failed, this is good), `survived` (test gap),
|
|
193
|
+
`timeout` (counts as detected), `uncovered` (no covering example, this is
|
|
194
|
+
coverage debt), `accepted` (known-equivalent, see ledger), `error`,
|
|
195
|
+
`invalid` (discarded).
|
|
196
|
+
Exit code is 1 if unaccepted survivors exist (or, with `--fail-at`, if the
|
|
197
|
+
score is below the threshold). Mistyped positional paths (a file that
|
|
198
|
+
doesn't exist, or a non-`.rb` file) are an error (exit 2) instead of a
|
|
199
|
+
vacuous green run.
|
|
164
200
|
|
|
165
201
|
Score = (killed + timeout) / (killed + timeout + survived).
|
|
166
202
|
|
|
@@ -179,65 +215,126 @@ bundle exec active_mutator --changed --accept-survivors # records to ledger
|
|
|
179
215
|
git add .active_mutator_accepted.json # committed state
|
|
180
216
|
```
|
|
181
217
|
|
|
182
|
-
Acceptance takes effect on the
|
|
218
|
+
Acceptance takes effect on the next run. The accepting run still exits 1.
|
|
219
|
+
Scoped accepting runs (`--changed`, `--subject`, path args) are safe: the
|
|
220
|
+
ledger only prunes entries in files fully scanned by non-narrowed runs, so
|
|
221
|
+
out-of-scope acceptances are never dropped.
|
|
183
222
|
Agent workflow: see [`docs/skills/mutation-check.md`](docs/skills/mutation-check.md).
|
|
184
223
|
|
|
224
|
+
## Reports
|
|
225
|
+
|
|
226
|
+
`--format stryker-json` writes `.active_mutator/mutation-report.json` in the
|
|
227
|
+
Stryker [mutation-testing-report-schema](https://github.com/stryker-mutator/mutation-testing-elements)
|
|
228
|
+
v2 format. Open it in the
|
|
229
|
+
[Stryker report viewer](https://microsoft.github.io/mutation-testing-elements/)
|
|
230
|
+
for per-file mutant maps with inline diffs, filterable by status.
|
|
231
|
+
|
|
232
|
+
`--format github` prints one `::warning` annotation per surviving mutant, so
|
|
233
|
+
survivors show inline on the PR diff. Pairs with the CI recipe:
|
|
234
|
+
|
|
235
|
+
bundle exec active_mutator --since origin/main --format github
|
|
236
|
+
|
|
185
237
|
## CI recipe
|
|
186
238
|
|
|
187
|
-
- Per-PR: `active_mutator --since origin/main` (minutes
|
|
239
|
+
- Per-PR: `active_mutator --since origin/main --format github` (minutes;
|
|
240
|
+
survivors annotate the PR diff)
|
|
188
241
|
- Nightly: `active_mutator --force-baseline` (full run; also recovers the
|
|
189
|
-
|
|
242
|
+
residual blind spot — constant-reference detection handles the common
|
|
243
|
+
newly-covering-example case since 0.2)
|
|
190
244
|
|
|
191
245
|
## Flags
|
|
192
246
|
|
|
193
247
|
| Flag | Default | Meaning |
|
|
194
248
|
|---|---|---|
|
|
195
249
|
| `--jobs N` | half the cores | fork-pool width |
|
|
196
|
-
| `--changed` |
|
|
197
|
-
| `--since REF` |
|
|
198
|
-
| `--subject
|
|
199
|
-
| `--
|
|
250
|
+
| `--changed` | none | mutate uncommitted + untracked work |
|
|
251
|
+
| `--since REF` | none | mutate methods changed since REF |
|
|
252
|
+
| `--subject EXPR` | none | subject expression, e.g. `Foo#bar`, `Foo::Bar`, `Foo::Bar*`, `Foo#*`, `Foo.*` |
|
|
253
|
+
| `--exclude PAT` | none | skip files matching glob during subject discovery (repeatable, gitignore-like) |
|
|
254
|
+
| `--max-mutants N` | none | deterministic sample of the first N mutants (quick smoke run on huge scopes; accepted/uncovered mutants count against N) |
|
|
255
|
+
| `--debug-plan` | off | print planned mutants as JSON and exit without running |
|
|
256
|
+
| `--format terminal\|json\|stryker-json\|github` | terminal | report format |
|
|
200
257
|
| `--accept-survivors` | off | record survivors to the acceptance ledger |
|
|
201
258
|
| `--force-baseline` | off | ignore cached coverage map |
|
|
202
259
|
| `--preload-helper FILE` / `--no-preload-helper` | auto-detect | parent spec-helper preload |
|
|
203
260
|
| `--serial-pattern PAT` | `spec/system/`, `spec/features/` | covering-path prefixes forced serial |
|
|
204
261
|
| `--browser-boot-seconds S` | 15 | serial-lane timeout bump |
|
|
205
262
|
| `--timeout-factor F` / `--timeout-floor S` | 8 / 10 | mutation timeout budget |
|
|
206
|
-
| `--
|
|
263
|
+
| `--[no-]adaptive-timeout` | on | scale timeout budgets from observed worker wall times (median utilization, grow-only, clamped 1x–4x; `--timeout-factor`/`--timeout-floor` set the starting budget) |
|
|
264
|
+
| `--require FILE` | none | preload files (repeatable) |
|
|
265
|
+
| `--operator FILE` | none | load a custom operator file before analysis (repeatable) |
|
|
266
|
+
| `--fail-at SCORE` | none (strict) | exit 0 if score >= SCORE even with survivors (opt-in relaxation for gradual adoption; 0 = report-only) |
|
|
267
|
+
|
|
268
|
+
`--debug-plan` prints the planned mutant list as one JSON document
|
|
269
|
+
(`{"planned": [...], "pre_resolved": {...}}`) and exits without running
|
|
270
|
+
anything. A coverage baseline is still built or loaded, since timeouts
|
|
271
|
+
and covering examples come from it.
|
|
207
272
|
|
|
208
|
-
Every active_mutator process sets `ENV["ACTIVE_MUTATOR"] = "1"
|
|
273
|
+
Every active_mutator process sets `ENV["ACTIVE_MUTATOR"] = "1"`. Use it to
|
|
209
274
|
guard SimpleCov or other tooling in your spec helper:
|
|
210
275
|
|
|
211
276
|
```ruby
|
|
212
277
|
SimpleCov.start "rails" unless ENV["ACTIVE_MUTATOR"]
|
|
213
278
|
```
|
|
214
279
|
|
|
280
|
+
## Configuration file
|
|
281
|
+
|
|
282
|
+
Put team-wide settings in `.active_mutator.yml` at the project root; CLI
|
|
283
|
+
flags override file values (`--require` and `--exclude` add to the file's
|
|
284
|
+
lists; the first `--serial-pattern` replaces them). Recognized keys:
|
|
285
|
+
`jobs`, `format`, `timeout_factor`, `timeout_floor`,
|
|
286
|
+
`browser_boot_seconds`, `fail_at`, `exclude`, `serial_patterns`,
|
|
287
|
+
`requires`, `operators` (custom operator files, loaded before analysis; see
|
|
288
|
+
[Custom operators](docs/guides/custom-operators.md)),
|
|
289
|
+
`preload_helper` (a path, or `false` to skip preload),
|
|
290
|
+
`adaptive_timeout` (`true`/`false`).
|
|
291
|
+
Unknown keys and wrong types are errors, not silent no-ops.
|
|
292
|
+
|
|
293
|
+
```yaml
|
|
294
|
+
# .active_mutator.yml
|
|
295
|
+
jobs: 4
|
|
296
|
+
exclude:
|
|
297
|
+
- lib/generated
|
|
298
|
+
serial_patterns:
|
|
299
|
+
- spec/system/
|
|
300
|
+
fail_at: 90 # legacy suite: gate on score instead of zero-survivors
|
|
301
|
+
```
|
|
302
|
+
|
|
215
303
|
## Known limits (v1.1)
|
|
216
304
|
|
|
217
|
-
Method bodies only (no class-macro/constant mutation)
|
|
218
|
-
heredoc
|
|
219
|
-
skipped
|
|
220
|
-
|
|
305
|
+
Method bodies only (no class-macro/constant mutation). RSpec only.
|
|
306
|
+
Plain heredoc bodies ARE mutated (emptied); interpolated heredocs are
|
|
307
|
+
skipped. `class << self` bodies are mutated as singleton subjects
|
|
308
|
+
(`class << obj` and top-level `class << self` are skipped). Nested defs
|
|
309
|
+
mutate as part of the enclosing method's body — they get no subject of
|
|
310
|
+
their own (a directly-inserted mutant would be reverted whenever the
|
|
311
|
+
outer method re-runs the `def`). The incremental baseline recovers the residual blind spot —
|
|
312
|
+
constant-reference detection handles the common case since 0.2, and a few
|
|
313
|
+
residual cases (pure indirection, partially-covering files, leaf-only or
|
|
314
|
+
wrapper-only references, `class ::Foo`, `Data.define`/`Struct.new` value
|
|
315
|
+
objects) are caught by nightly `--force-baseline`.
|
|
221
316
|
|
|
222
317
|
## Guides
|
|
223
318
|
|
|
224
|
-
- [What is mutation testing?](docs/guides/what-is-mutation-testing.md)
|
|
225
|
-
the concepts
|
|
226
|
-
- [How it works](docs/guides/how-it-works.md)
|
|
319
|
+
- [What is mutation testing?](docs/guides/what-is-mutation-testing.md):
|
|
320
|
+
the concepts. Kill/survive, score, equivalent mutants, further reading.
|
|
321
|
+
- [How it works](docs/guides/how-it-works.md): architecture. Subject
|
|
227
322
|
discovery, source-span edits, the coverage map, the fork pipeline, and
|
|
228
323
|
honest limits.
|
|
229
|
-
- [Operator reference](docs/guides/operators.md)
|
|
324
|
+
- [Operator reference](docs/guides/operators.md): every mutation
|
|
230
325
|
active_mutator can generate, with before/after examples and what a
|
|
231
326
|
survivor of each one means.
|
|
232
|
-
- [
|
|
233
|
-
|
|
327
|
+
- [Custom operators](docs/guides/custom-operators.md): write and load your
|
|
328
|
+
own mutation operators with `--operator` / the `operators:` config key.
|
|
329
|
+
- [Mutation-check skill](docs/skills/mutation-check.md): the agent-facing
|
|
330
|
+
workflow. Run, read survivors, strengthen tests, or accept with a reason.
|
|
234
331
|
|
|
235
332
|
## Contributing
|
|
236
333
|
|
|
237
334
|
Issues and pull requests welcome. Run `bundle exec rspec` before sending a
|
|
238
|
-
change
|
|
239
|
-
sending a change that touches `lib
|
|
240
|
-
you'd want it run on any other codebase.
|
|
335
|
+
change. Also run `bundle exec active_mutator --changed` on your own diff
|
|
336
|
+
before sending a change that touches `lib/`. This is a good idea for the
|
|
337
|
+
same reason you'd want it run on any other codebase.
|
|
241
338
|
|
|
242
339
|
## License
|
|
243
340
|
|
|
@@ -3,8 +3,12 @@ require "set"
|
|
|
3
3
|
|
|
4
4
|
module ActiveMutator
|
|
5
5
|
# Committed, repo-root ledger of accepted (equivalent) survivors.
|
|
6
|
-
# Deliberately NOT inside .active_mutator
|
|
6
|
+
# Deliberately NOT inside .active_mutator/: that dir is gitignored and
|
|
7
7
|
# disposable, while acceptance decisions are durable team/CI state.
|
|
8
|
+
#
|
|
9
|
+
# Entries whose file no longer exists are kept, not pruned: the file may
|
|
10
|
+
# still exist on another branch, so deletion is the user's call. The runner
|
|
11
|
+
# warns about them on every run instead (see #missing_file_entries).
|
|
8
12
|
class AcceptedLedger
|
|
9
13
|
FILENAME = ".active_mutator_accepted.json"
|
|
10
14
|
|
|
@@ -28,16 +32,27 @@ module ActiveMutator
|
|
|
28
32
|
|
|
29
33
|
def accepted?(fingerprint) = @entries.include?(fingerprint)
|
|
30
34
|
|
|
31
|
-
|
|
35
|
+
# Entries outside the scanned files can't be judged by this run, so they
|
|
36
|
+
# are never stale here. scanned_files: nil means "no file was fully
|
|
37
|
+
# scanned" (subject-level filtering active) — union only, prune nothing.
|
|
38
|
+
# See #24: a scoped accept run once deleted every out-of-scope entry.
|
|
39
|
+
def stale_entries(all_current_fingerprints, scanned_files:)
|
|
40
|
+
return [] if scanned_files.nil?
|
|
41
|
+
|
|
32
42
|
current = all_current_fingerprints.to_set
|
|
33
|
-
|
|
43
|
+
scanned = scanned_files.to_set
|
|
44
|
+
@entries.reject { |e| current.include?(e) || !scanned.include?(e.file) }
|
|
34
45
|
end
|
|
35
46
|
|
|
36
|
-
#
|
|
37
|
-
#
|
|
38
|
-
def
|
|
39
|
-
|
|
40
|
-
|
|
47
|
+
# Missing is objective regardless of run scope: such entries can never
|
|
48
|
+
# appear in scanned_files, so without this they'd be immortal AND silent.
|
|
49
|
+
def missing_file_entries(root)
|
|
50
|
+
@entries.reject { |e| File.exist?(File.join(root, e.file)) }
|
|
51
|
+
end
|
|
52
|
+
|
|
53
|
+
def accept!(new_fingerprints, all_current_fingerprints, scanned_files:)
|
|
54
|
+
stale = stale_entries(all_current_fingerprints, scanned_files: scanned_files).to_set
|
|
55
|
+
@entries = (@entries + new_fingerprints).uniq.reject { |e| stale.include?(e) }
|
|
41
56
|
AtomicFile.write(@path, JSON.pretty_generate(@entries.map(&:to_h)))
|
|
42
57
|
nil
|
|
43
58
|
end
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
module ActiveMutator
|
|
2
2
|
# flock-guarded write-to-temp + rename. Concurrent runs in one repo (an
|
|
3
|
-
# agent plus a human
|
|
3
|
+
# agent plus a human, the dev-loop case) must not corrupt cache or ledger.
|
|
4
4
|
module AtomicFile
|
|
5
5
|
def self.write(path, content)
|
|
6
6
|
File.open("#{path}.lock", File::CREAT | File::RDWR, 0o644) do |lock|
|
|
@@ -42,13 +42,22 @@ module ActiveMutator
|
|
|
42
42
|
|
|
43
43
|
private
|
|
44
44
|
|
|
45
|
-
|
|
45
|
+
# The cache is disposable and must never be committed. Host projects
|
|
46
|
+
# rarely gitignore it themselves, so the directory ignores its own
|
|
47
|
+
# contents (the node_modules trick).
|
|
48
|
+
def prepare_cache_dir
|
|
46
49
|
FileUtils.mkdir_p(@cache_dir)
|
|
50
|
+
ignore = File.join(@cache_dir, ".gitignore")
|
|
51
|
+
File.write(ignore, "*\n") unless File.exist?(ignore)
|
|
52
|
+
end
|
|
53
|
+
|
|
54
|
+
def run_baseline!
|
|
55
|
+
prepare_cache_dir
|
|
47
56
|
env = baseline_env(@out_path)
|
|
48
|
-
# out: :err
|
|
57
|
+
# out: :err: the subprocess suite's progress output must not pollute
|
|
49
58
|
# our stdout (breaks `--format json` consumers).
|
|
50
59
|
ok = system(env, "bundle", "exec", "rspec", chdir: @root, out: :err)
|
|
51
|
-
raise BaselineFailed, "baseline suite failed
|
|
60
|
+
raise BaselineFailed, "baseline suite failed, fix the suite before mutating" unless ok
|
|
52
61
|
raise BaselineFailed, "baseline produced no coverage output" unless File.exist?(@out_path)
|
|
53
62
|
end
|
|
54
63
|
|
|
@@ -80,7 +89,7 @@ module ActiveMutator
|
|
|
80
89
|
if targets.any?
|
|
81
90
|
env = baseline_env(partial_out)
|
|
82
91
|
ok = system(env, "bundle", "exec", "rspec", *targets, chdir: @root, out: :err)
|
|
83
|
-
raise BaselineFailed, "partial baseline run failed
|
|
92
|
+
raise BaselineFailed, "partial baseline run failed, fix the suite before mutating" unless ok
|
|
84
93
|
raise BaselineFailed, "partial baseline produced no output" unless File.exist?(partial_out)
|
|
85
94
|
end
|
|
86
95
|
merge_partial!(partial_out, delta)
|
|
@@ -11,6 +11,10 @@ module ActiveMutator
|
|
|
11
11
|
FULL = Delta.new(full: true, rerun_spec_files: [], rerun_example_ids: [],
|
|
12
12
|
drop_example_ids: [], drop_source_files: [])
|
|
13
13
|
|
|
14
|
+
# If a changed constant is referenced by more than this share of all spec
|
|
15
|
+
# files, a full re-run is cheaper and simpler than a giant partial one.
|
|
16
|
+
REFERENCE_FULL_RATIO = 0.5
|
|
17
|
+
|
|
14
18
|
def self.compute(old_digests:, new_digests:, coverage_map:, root:)
|
|
15
19
|
changed = (old_digests.keys | new_digests.keys)
|
|
16
20
|
.reject { |k| old_digests[k] == new_digests[k] }
|
|
@@ -21,6 +25,13 @@ module ActiveMutator
|
|
|
21
25
|
drop_example_ids = []
|
|
22
26
|
drop_source_files = []
|
|
23
27
|
|
|
28
|
+
# Read the spec-file list and their contents once per compute call, not
|
|
29
|
+
# once per changed source file: newly_covering_candidates scans every
|
|
30
|
+
# spec file, so re-globbing and re-reading inside the loop was
|
|
31
|
+
# O(changed_files x spec_files) IO. Built lazily so a delta with no
|
|
32
|
+
# scannable source change pays nothing.
|
|
33
|
+
spec_contents = nil
|
|
34
|
+
|
|
24
35
|
changed.each do |rel|
|
|
25
36
|
added = !old_digests.key?(rel)
|
|
26
37
|
deleted = !new_digests.key?(rel)
|
|
@@ -39,7 +50,17 @@ module ActiveMutator
|
|
|
39
50
|
else
|
|
40
51
|
abs = File.join(root, rel)
|
|
41
52
|
drop_source_files << abs if deleted
|
|
42
|
-
|
|
53
|
+
unless added
|
|
54
|
+
rerun_example_ids.concat(coverage_map.examples_covering_file(abs))
|
|
55
|
+
end
|
|
56
|
+
unless deleted
|
|
57
|
+
spec_contents ||= Dir[File.join(root, "spec/**/*_spec.rb")].to_h { |f| [f, File.read(f)] }
|
|
58
|
+
candidates = newly_covering_candidates(root: root, rel: rel, coverage_map: coverage_map,
|
|
59
|
+
spec_contents: spec_contents)
|
|
60
|
+
return FULL if candidates == :full
|
|
61
|
+
|
|
62
|
+
rerun_spec_files.concat(candidates)
|
|
63
|
+
end
|
|
43
64
|
end
|
|
44
65
|
end
|
|
45
66
|
|
|
@@ -53,5 +74,50 @@ module ActiveMutator
|
|
|
53
74
|
def self.full_trigger?(rel)
|
|
54
75
|
rel.start_with?("spec/support/") || !rel.end_with?(".rb")
|
|
55
76
|
end
|
|
77
|
+
|
|
78
|
+
# #11: an unchanged spec file can START covering a changed source file
|
|
79
|
+
# because of the edit itself. Cheap static detection: spec files that
|
|
80
|
+
# textually reference a constant the changed file defines, but currently
|
|
81
|
+
# contribute zero coverage to it, get re-run. Files already covering it
|
|
82
|
+
# are handled example-by-example via rerun_example_ids.
|
|
83
|
+
def self.newly_covering_candidates(root:, rel:, coverage_map:, spec_contents:)
|
|
84
|
+
abs = File.join(root, rel)
|
|
85
|
+
return [] unless File.exist?(abs)
|
|
86
|
+
|
|
87
|
+
constants = DefinedConstants.in_source(File.read(abs))
|
|
88
|
+
return [] if constants.empty?
|
|
89
|
+
|
|
90
|
+
all_specs = spec_contents.keys
|
|
91
|
+
|
|
92
|
+
covering_specs = coverage_map.examples_covering_file(abs)
|
|
93
|
+
.map { |id| spec_file_of(id) }.to_a.uniq
|
|
94
|
+
# Escaping is required: dynamic-namespace class definitions (e.g.
|
|
95
|
+
# `class (a)::Baz`, `class foo.bar::Baz`) make constant_path.slice carry
|
|
96
|
+
# regex metachars. Unescaped, "(a)::Baz" would match the literal text
|
|
97
|
+
# "a::Baz" — a false candidate.
|
|
98
|
+
# TODO(#11, Task 10 residual gap): a top-level `class ::Foo` yields the
|
|
99
|
+
# slice "::Foo", and /\b::Foo\b/ can never match (no word boundary
|
|
100
|
+
# before ":"), so such files are silently unscanned.
|
|
101
|
+
pattern = /\b(?:#{constants.map { |c| Regexp.escape(c) }.join("|")})\b/
|
|
102
|
+
candidates = all_specs.filter_map do |spec_abs|
|
|
103
|
+
spec_rel = spec_abs.delete_prefix(root).delete_prefix("/")
|
|
104
|
+
next if covering_specs.include?(spec_rel)
|
|
105
|
+
|
|
106
|
+
spec_rel if spec_contents.fetch(spec_abs).match?(pattern)
|
|
107
|
+
end
|
|
108
|
+
if candidates.size > 1 && candidates.size > all_specs.size * REFERENCE_FULL_RATIO
|
|
109
|
+
# Never silently degrade: a full baseline where the user expected an
|
|
110
|
+
# incremental refresh must be explained, or it looks like a hang.
|
|
111
|
+
warn "active_mutator: constant-reference scan matched #{candidates.size} of " \
|
|
112
|
+
"#{all_specs.size} spec files for #{rel}; falling back to full baseline"
|
|
113
|
+
return :full
|
|
114
|
+
end
|
|
115
|
+
|
|
116
|
+
candidates
|
|
117
|
+
end
|
|
118
|
+
|
|
119
|
+
def self.spec_file_of(example_id)
|
|
120
|
+
example_id.sub(%r{\A\./}, "").sub(/\[.*\]\z/, "")
|
|
121
|
+
end
|
|
56
122
|
end
|
|
57
123
|
end
|
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
# Loaded standalone via RUBYOPT=-ractive_mutator/baseline_hooks in the host
|
|
2
|
-
# project's suite
|
|
2
|
+
# project's suite, before rspec boots, so Coverage instruments everything
|
|
3
3
|
# the suite loads (including code loaded by spec_helper). Records per-example
|
|
4
4
|
# coverage diffs and writes the inverted map to ACTIVE_MUTATOR_BASELINE_OUT.
|
|
5
5
|
require "json"
|
|
@@ -53,7 +53,7 @@ if ENV["ACTIVE_MUTATOR_BASELINE_OUT"]
|
|
|
53
53
|
root = ENV.fetch("ACTIVE_MUTATOR_ROOT")
|
|
54
54
|
ActiveMutator::BaselineHooks::RECORDS[example.id] =
|
|
55
55
|
ActiveMutator::BaselineHooks.diff_coverage(before, after, root)
|
|
56
|
-
# NOT example.execution_result.run_time
|
|
56
|
+
# NOT example.execution_result.run_time: that is nil until after
|
|
57
57
|
# around hooks complete.
|
|
58
58
|
ActiveMutator::BaselineHooks::TIMES[example.id] = elapsed
|
|
59
59
|
end
|
data/lib/active_mutator/cli.rb
CHANGED
|
@@ -22,16 +22,20 @@ module ActiveMutator
|
|
|
22
22
|
# boot cost (RSpec setup + spec file loading).
|
|
23
23
|
requires: [], timeout_factor: 8.0, timeout_floor: 10.0, force_baseline: false,
|
|
24
24
|
preload_helper: nil, serial_patterns: ["spec/system/", "spec/features/"],
|
|
25
|
-
browser_boot_seconds: 15.0, accept_survivors: false
|
|
25
|
+
browser_boot_seconds: 15.0, accept_survivors: false, exclude: [],
|
|
26
|
+
max_mutants: nil, debug_plan: false, fail_at: nil, adaptive_timeout: true,
|
|
27
|
+
operator_paths: []
|
|
26
28
|
}
|
|
29
|
+
options.merge!(ConfigFile.load(Dir.pwd))
|
|
27
30
|
paths = OptionParser.new do |o|
|
|
28
31
|
o.banner = "Usage: active_mutator [paths] [options]"
|
|
29
32
|
o.on("--since REF", "Mutate only methods changed since git REF") { |v| options[:since] = v }
|
|
30
33
|
o.on("--changed", "Mutate uncommitted work (alias for --since HEAD, plus untracked files)") { options[:since] = "HEAD" }
|
|
31
|
-
o.on("--subject NAME", "Mutate
|
|
34
|
+
o.on("--subject NAME", "Mutate matching subjects: Foo::Bar#baz, Foo::Bar, Foo::Bar*, Foo::Bar#*") { |v| options[:subject_filter] = v }
|
|
32
35
|
o.on("--jobs N", Integer, "Concurrent workers (default: half the CPU count)") { |v| options[:jobs] = v }
|
|
33
|
-
o.on("--format FMT",
|
|
34
|
-
o.on("--require FILE", "File to require before mutating (repeatable)") { |v| options[:requires] << v }
|
|
36
|
+
o.on("--format FMT", ConfigFile::FORMATS, "Output format") { |v| options[:format] = v.tr("-", "_").to_sym }
|
|
37
|
+
o.on("--require FILE", "File to require before mutating (repeatable; adds to config-file requires)") { |v| options[:requires] << v }
|
|
38
|
+
o.on("--operator FILE", "Ruby file defining a custom operator, loaded before analysis (repeatable)") { |v| options[:operator_paths] << v }
|
|
35
39
|
o.on("--force-baseline", "Ignore cached coverage map") { options[:force_baseline] = true }
|
|
36
40
|
o.on("--timeout-factor F", Float, "Timeout = baseline time * F + floor") { |v| options[:timeout_factor] = v }
|
|
37
41
|
o.on("--timeout-floor S", Float, "Minimum timeout seconds") { |v| options[:timeout_floor] = v }
|
|
@@ -43,7 +47,15 @@ module ActiveMutator
|
|
|
43
47
|
options[:serial_patterns] << v
|
|
44
48
|
end
|
|
45
49
|
o.on("--browser-boot-seconds S", Float, "Extra timeout budget for serial-lane mutants") { |v| options[:browser_boot_seconds] = v }
|
|
50
|
+
o.on("--[no-]adaptive-timeout", "Scale timeout budgets from observed worker wall times (default: on)") { |v| options[:adaptive_timeout] = v }
|
|
46
51
|
o.on("--accept-survivors", "Record surviving mutants into the acceptance ledger") { options[:accept_survivors] = true }
|
|
52
|
+
o.on("--exclude PAT", "Skip files matching glob, relative to root (repeatable)") { |v| options[:exclude] << v }
|
|
53
|
+
o.on("--max-mutants N", Integer, "Deterministically sample the first N mutants") { |v| options[:max_mutants] = v }
|
|
54
|
+
o.on("--debug-plan", "Print the planned mutant list as JSON and exit") { options[:debug_plan] = true }
|
|
55
|
+
o.on("--fail-at SCORE", Float, "Exit 0 if mutation score >= SCORE even with survivors (default: any survivor fails)") do |v|
|
|
56
|
+
raise OptionParser::InvalidArgument, "--fail-at must be within 0..100" unless (0..100).cover?(v)
|
|
57
|
+
options[:fail_at] = v
|
|
58
|
+
end
|
|
47
59
|
end.parse(argv)
|
|
48
60
|
options.delete(:serial_patterns_replaced)
|
|
49
61
|
|
|
@@ -4,5 +4,6 @@ module ActiveMutator
|
|
|
4
4
|
Config = Data.define(:paths, :since, :subject_filter, :jobs, :format, :requires,
|
|
5
5
|
:timeout_factor, :timeout_floor, :force_baseline, :root,
|
|
6
6
|
:preload_helper, :serial_patterns, :browser_boot_seconds,
|
|
7
|
-
:accept_survivors
|
|
7
|
+
:accept_survivors, :exclude, :max_mutants, :debug_plan,
|
|
8
|
+
:fail_at, :adaptive_timeout, :operator_paths)
|
|
8
9
|
end
|