mutineer 1.1.0 → 1.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +115 -0
- data/README.md +54 -5
- data/lib/mutineer/baseline.rb +74 -11
- data/lib/mutineer/cli.rb +71 -4
- data/lib/mutineer/coverage_map.rb +37 -40
- data/lib/mutineer/daemon_backend.rb +5 -4
- data/lib/mutineer/isolation.rb +9 -4
- data/lib/mutineer/mutant_id.rb +72 -13
- data/lib/mutineer/mutator_registry.rb +9 -3
- data/lib/mutineer/mutators/array_literal.rb +51 -0
- data/lib/mutineer/mutators/base.rb +15 -0
- data/lib/mutineer/mutators/operand_removal.rb +65 -0
- data/lib/mutineer/pairing.rb +10 -5
- data/lib/mutineer/project.rb +47 -18
- data/lib/mutineer/project_path.rb +57 -0
- data/lib/mutineer/reporter.rb +18 -5
- data/lib/mutineer/result.rb +2 -2
- data/lib/mutineer/runner.rb +90 -26
- data/lib/mutineer/subject.rb +13 -2
- data/lib/mutineer/version.rb +1 -1
- data/lib/mutineer.rb +2 -0
- metadata +4 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: e7b8a623ac7db818d33df11a7ca32e26a67db7f156061027f018debeb17ced2f
|
|
4
|
+
data.tar.gz: 5957fb6e57cb5a5149822818f773f3e8d3da271bce7bb0cfc4d0a8640764aa40
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 1de2a131774b7b587abbc8ca91a57da333688cb2fdb937defa002d661426137decf8b86fbde1a2d176a5525f8cb7d9a8b6a2bebc353960729704b3e8d2473213
|
|
7
|
+
data.tar.gz: '094d062c4c8458fda7061a575681f13022131b2b9078e5731bcbbfa300ac67b86e53691e4d9e104cc06e50832b66bd3dd32433c1bdf420e74b75a7b7748b4916'
|
data/CHANGELOG.md
CHANGED
|
@@ -6,6 +6,119 @@ All notable changes to this project are documented here. The format is based on
|
|
|
6
6
|
|
|
7
7
|
## [Unreleased]
|
|
8
8
|
|
|
9
|
+
## [1.3.0] - 2026-09-29
|
|
10
|
+
|
|
11
|
+
### Changed
|
|
12
|
+
- **Mutant ids now include the project-relative file path** (#126). Before, two
|
|
13
|
+
mutants with the same method name, operator and token in different files got
|
|
14
|
+
the same id. That happened with a top-level `def` or block in two files, and
|
|
15
|
+
with a class reopened in another file. One `ignore:` entry then suppressed
|
|
16
|
+
both mutants, and `--baseline` could hide a new survivor behind an old one.
|
|
17
|
+
- Every survivor id changes once in this release. This affects any external
|
|
18
|
+
tool that tracks survivors by id.
|
|
19
|
+
- Moving or renaming a file now changes its ids.
|
|
20
|
+
- Ids follow the project root: the directory mutineer runs from, or the
|
|
21
|
+
Action's `working-directory`. A run from a different root gives different ids.
|
|
22
|
+
mutineer finds `.mutineer.yml` by walking up, so a run from a subdirectory
|
|
23
|
+
prints one `[mutineer]` warning that the loaded ignore ids will not match.
|
|
24
|
+
- A source outside the project root uses its absolute path, so its ids differ
|
|
25
|
+
between machines.
|
|
26
|
+
- Two methods with the same qualified name in one file (for example two
|
|
27
|
+
top-level `def index` in two DSL blocks) now get different ids. The second
|
|
28
|
+
and later ones hash their position among those methods; the first keeps its
|
|
29
|
+
id.
|
|
30
|
+
- **The JSON report marks its id format** (schema `1.4`, additive).
|
|
31
|
+
`summary.id_format` is `2` for ids that include the file path.
|
|
32
|
+
`summary.legacy_id_matches` counts old-format `ignore:` entries (`ignore`) and
|
|
33
|
+
survivors matched only through an old-format baseline id (`baseline`). The
|
|
34
|
+
GitHub Action shows one warning annotation when either count is not zero.
|
|
35
|
+
|
|
36
|
+
### Deprecated
|
|
37
|
+
- **Old-format ids in `ignore:` and in baselines.** Matching on them is removed
|
|
38
|
+
in 2.0. Replace each old `ignore:` entry with the new ids from the warning
|
|
39
|
+
(only the intended ones when it over-matched several mutants).
|
|
40
|
+
Regenerate a baseline (`--format json`) only after every gate that reads it
|
|
41
|
+
runs this version or later (the Action's `version:` pin, your CI
|
|
42
|
+
`Gemfile.lock`). An older version treats every new-format survivor as new.
|
|
43
|
+
|
|
44
|
+
### Fixed
|
|
45
|
+
- **`module_function :name` and `module_function def name` promote only
|
|
46
|
+
their own module's methods** — a class or module in the same file with a
|
|
47
|
+
method of the same name kept an instance method in Ruby, but mutineer
|
|
48
|
+
named it as a class method. `--strategy redefine` then mutated a method the
|
|
49
|
+
tests never call, so a killable mutant falsely survived, and
|
|
50
|
+
`--only Class#name` selected nothing (#98). The affected methods now get
|
|
51
|
+
their correct names, so their mutant ids change: regenerate any ignore
|
|
52
|
+
entries or baseline survivors that pointed at them.
|
|
53
|
+
- **A root-anchored reopening names the top-level constant** — a
|
|
54
|
+
`module ::Root` or `class ::Solo` written inside another module now gives
|
|
55
|
+
`Root` and `Solo`, not `Outer::Root` and `Outer::Solo`. This applies to every
|
|
56
|
+
method in such a body, with or without `module_function`, so their mutant
|
|
57
|
+
ids change too. `--strategy redefine` rebuilds such a body's scope as
|
|
58
|
+
written (`module Outer` then `class ::Solo`), so a constant from `Outer`
|
|
59
|
+
still resolves in the mutated method, as it does under `reload`. Before,
|
|
60
|
+
the method raised NameError in the test, which counted as a false kill
|
|
61
|
+
(#145).
|
|
62
|
+
- **Old-format ids keep working, with a warning** (#126). An old-format
|
|
63
|
+
`ignore:` entry still suppresses the mutants it matched before. The run prints
|
|
64
|
+
one `[mutineer]` warning per entry with each new id, its file and its method.
|
|
65
|
+
When the entry matched several mutants (in different files, or same-named
|
|
66
|
+
methods in one file), the warning says it
|
|
67
|
+
over-matched and to keep only the ids for the mutant you meant to ignore. A
|
|
68
|
+
`--baseline` file without `summary.id_format` matches on new ids, or on old
|
|
69
|
+
ids from the same file, so no survivor reads as new or fixed only because its
|
|
70
|
+
id changed. A stored file outside the project root (a baseline written on
|
|
71
|
+
another machine) matches on the old id alone. The run prints one
|
|
72
|
+
`[mutineer]` warning to regenerate the baseline.
|
|
73
|
+
|
|
74
|
+
## [1.2.0] - 2026-09-28
|
|
75
|
+
|
|
76
|
+
### Added
|
|
77
|
+
- **Operand-removal operator** (Tier-2, opt-in via `--operators`):
|
|
78
|
+
`operand_removal` replaces `a && b` with `(a)` and with `(b)`, and does the
|
|
79
|
+
same for `||`, `and` and `or`. The mutant survives when no test needs the
|
|
80
|
+
operand that the mutant removes. The operator never keeps a jump operand
|
|
81
|
+
(`return`, `break`, `next`, `redo`, `retry`) alone, because a jump does not
|
|
82
|
+
parse in a value context. It never removes an operand that holds a heredoc,
|
|
83
|
+
because the heredoc body stays behind as code. It skips nested method
|
|
84
|
+
definitions, because mutineer mutates each one as its own method.
|
|
85
|
+
- **Array-literal operator** (Tier-2, opt-in via `--operators`):
|
|
86
|
+
`array_literal` replaces a non-empty array literal, such as `[a, b]` or
|
|
87
|
+
`%i[a b]`, with `[]`. The mutant survives when no test checks the contents
|
|
88
|
+
of the array. The operator skips an implicit array (`x = 1, 2`), an array
|
|
89
|
+
that holds a heredoc, and nested method definitions.
|
|
90
|
+
- **Sources also pair with Minitest's `test/**/test_*.rb` files** — after the
|
|
91
|
+
`_test.rb` forms, so existing projects pair as before. `lib/helper.rb` does
|
|
92
|
+
not pair with the `test/test_helper.rb` support file. A failed capture of a
|
|
93
|
+
`test_<name>.rb` file now marks `<name>.rb` uncapturable, as `<name>_test.rb`
|
|
94
|
+
does (#120). Without `framework:` set, a source with `spec/<name>_spec.rb`
|
|
95
|
+
and `test/test_<name>.rb` but no `<name>_test.rb` now pairs with the
|
|
96
|
+
Minitest file, as the "Minitest first" order says.
|
|
97
|
+
|
|
98
|
+
### Fixed
|
|
99
|
+
- **`reload` loads the mutant by an absolute path** — a relative source path
|
|
100
|
+
gave the mutant relative backtrace paths, so code that checks its own frames
|
|
101
|
+
by absolute path failed for every mutant, a false kill (#123).
|
|
102
|
+
- **`require "test_helper"` works without `RUBYOPT`** — a standalone run puts
|
|
103
|
+
`lib`, then each test file's `test_helper.rb` directory, on the load path,
|
|
104
|
+
as boot mode and `rake test` do. A run where no test records coverage because
|
|
105
|
+
captures failed now exits 1 instead of reporting N/A (#119).
|
|
106
|
+
- **A disable-line marker warns about an operator it does not know** — a
|
|
107
|
+
reason written without `--` became part of the operator name, so the marker
|
|
108
|
+
suppressed nothing and said nothing (#124). A marker followed only by spaces
|
|
109
|
+
or commas, such as `disable-line -- why`, now disables the whole line.
|
|
110
|
+
- **Coverage capture and the clean check run each source once** — they read
|
|
111
|
+
sources with `load`, so a test's own `require` ran them again: a `Struct`
|
|
112
|
+
superclass raised `superclass mismatch`, and load-time code ran twice (#122).
|
|
113
|
+
A mutant of such a class still errors under `--strategy reload`, which loads
|
|
114
|
+
the mutated file again; `--strategy redefine` runs it.
|
|
115
|
+
Code that guards itself to run once (`unless defined?(X)`) can now show as
|
|
116
|
+
covered, so its mutants run where they were `no_coverage` before.
|
|
117
|
+
- **A red unmutated suite now shows why it failed** — in a standalone run, the
|
|
118
|
+
Minitest summary or RSpec output of the failing test, with its failure
|
|
119
|
+
message, goes to stderr before the "not green" error. A passing run prints
|
|
120
|
+
nothing extra. Boot mode (`--rails`, `--boot`) is unchanged (#121).
|
|
121
|
+
|
|
9
122
|
## [1.1.0] - 2026-09-28
|
|
10
123
|
|
|
11
124
|
### Added
|
|
@@ -498,6 +611,8 @@ Rails hardening + CI batch (issues #8–#13), all verified Rails-free.
|
|
|
498
611
|
- `.mutineer.yml` configuration (CLI > config > default precedence).
|
|
499
612
|
- Byte-correct source handling for multibyte (UTF-8) sources.
|
|
500
613
|
|
|
614
|
+
[1.3.0]: https://github.com/davidteren/mutineer/releases/tag/v1.3.0
|
|
615
|
+
[1.2.0]: https://github.com/davidteren/mutineer/releases/tag/v1.2.0
|
|
501
616
|
[1.1.0]: https://github.com/davidteren/mutineer/releases/tag/v1.1.0
|
|
502
617
|
[1.0.2]: https://github.com/davidteren/mutineer/releases/tag/v1.0.2
|
|
503
618
|
[1.0.1]: https://github.com/davidteren/mutineer/releases/tag/v1.0.1
|
data/README.md
CHANGED
|
@@ -84,7 +84,8 @@ Run `mutineer --list-operators` to see them. Default (Tier 1): `arithmetic`,
|
|
|
84
84
|
`comparison`, `boolean_connector`, `boolean_literal`, `statement_removal`.
|
|
85
85
|
Available but off by default (Tier 2, enable via `--operators`): `return_nil`,
|
|
86
86
|
`literal_mutation`, `condition_negation`, `string_literal`, `regex`,
|
|
87
|
-
`collection_method`, `safe_navigation`, `range`, `negation_removal`, `chain_link
|
|
87
|
+
`collection_method`, `safe_navigation`, `range`, `negation_removal`, `chain_link`,
|
|
88
|
+
`operand_removal`, `array_literal`.
|
|
88
89
|
|
|
89
90
|
## Rails apps
|
|
90
91
|
|
|
@@ -212,12 +213,60 @@ Tradeoffs — this path is correct but not free:
|
|
|
212
213
|
Some mutants are equivalent (behaviour-identical) and survive forever — keeping a
|
|
213
214
|
file off 100%. Suppress them so the score and `--threshold` gate stay meaningful:
|
|
214
215
|
|
|
215
|
-
- **Inline:** `some_line # mutineer:disable-line` (or scope it: `# mutineer:disable-line comparison`).
|
|
216
|
-
- **Config:** a `.mutineer.yml` `ignore:` list of
|
|
216
|
+
- **Inline:** `some_line # mutineer:disable-line` (or scope it: `# mutineer:disable-line comparison`). Put a reason after `--`: `# mutineer:disable-line comparison -- the test checks only 20`.
|
|
217
|
+
- **Config:** a `.mutineer.yml` `ignore:` list of mutant ids. Each survivor's
|
|
217
218
|
`id` is printed in the JSON report, so copy it straight into `ignore:`.
|
|
218
219
|
|
|
219
220
|
Suppressed mutants are excluded from the score (so 100% becomes reachable).
|
|
220
221
|
|
|
222
|
+
## Mutant ids
|
|
223
|
+
|
|
224
|
+
A mutant id is 12 hex characters. It hashes the file path (relative to the
|
|
225
|
+
project root), the method's qualified name, the operator, the mutated code, and
|
|
226
|
+
the mutant's position among identical mutants in that method. When one file has
|
|
227
|
+
two methods with the same qualified name (for example two top-level `def index`
|
|
228
|
+
in two DSL blocks), the second and later ones also hash their position among
|
|
229
|
+
those methods, so their ids differ. The first one's id does not change. An edit
|
|
230
|
+
outside the method does not change the id. Moving or renaming the file,
|
|
231
|
+
renaming the method or its class, or adding an identical mutant earlier in the
|
|
232
|
+
method does. Adding a method with the same name earlier in the same file also
|
|
233
|
+
does.
|
|
234
|
+
|
|
235
|
+
- The project root is the directory mutineer runs from (in the Action, the
|
|
236
|
+
`working-directory`). Run from the same root to get the same ids.
|
|
237
|
+
- A source outside the project root uses its absolute path, so its ids differ
|
|
238
|
+
between machines.
|
|
239
|
+
|
|
240
|
+
**Migrating from ids without the file path.** Before 1.3, ids did not include
|
|
241
|
+
the file path, so two files could share an id (#126). Old-format ids keep
|
|
242
|
+
working until 2.0, with a warning:
|
|
243
|
+
|
|
244
|
+
- **`ignore:`** An old entry still suppresses its mutants. The run prints the
|
|
245
|
+
new ids for each old entry, each with its file and method. When it names one
|
|
246
|
+
mutant, replace the entry with that id. When it names several (in different
|
|
247
|
+
files, or same-named methods in one file), the old entry over-matched: it also
|
|
248
|
+
hid mutants you did not mean to ignore. The warning says so. Keep only the ids for the mutant you meant to
|
|
249
|
+
ignore, not all of them. The list covers only the sources and operators in
|
|
250
|
+
that run, so run over every source with every operator set you use (for
|
|
251
|
+
example your Tier-2 `--operators`) for the full list.
|
|
252
|
+
- **`--baseline`** An old baseline still matches: a survivor matches a stored
|
|
253
|
+
one with the same old id in the same file. A stored file that is an absolute
|
|
254
|
+
path outside the project root (a baseline written on another machine)
|
|
255
|
+
matches on the old id alone. The run tells you to
|
|
256
|
+
regenerate it. Regenerate it with `--format json`, but only after every gate
|
|
257
|
+
that reads it runs 1.3 or later (the Action's `version:` pin, your CI
|
|
258
|
+
`Gemfile.lock`). An older version treats every new-format survivor as new.
|
|
259
|
+
|
|
260
|
+
The JSON report's `summary.id_format` is `2` for the new format.
|
|
261
|
+
`summary.legacy_id_matches.ignore` counts the old-format ignore entries a run
|
|
262
|
+
matched, and `summary.legacy_id_matches.baseline` counts the survivors matched
|
|
263
|
+
only through an old baseline id.
|
|
264
|
+
|
|
265
|
+
Ids are relative to the directory you run mutineer from. mutineer finds
|
|
266
|
+
`.mutineer.yml` by walking up. When the file it loads is in a parent directory
|
|
267
|
+
(other than your home directory), it warns that the ignore ids will not match
|
|
268
|
+
and tells you which directory to run from.
|
|
269
|
+
|
|
221
270
|
## CI gating
|
|
222
271
|
|
|
223
272
|
Store a JSON run as a baseline, then fail the build only when a PR makes things
|
|
@@ -227,7 +276,7 @@ worse:
|
|
|
227
276
|
mutineer run app/ --baseline .mutineer/baseline.json # exit 1 on NEW survivors or a score drop
|
|
228
277
|
```
|
|
229
278
|
|
|
230
|
-
`--baseline` reports which survivors are new (by
|
|
279
|
+
`--baseline` reports which survivors are new (by [mutant id](#mutant-ids)) and any score drop. It
|
|
231
280
|
combines with `--threshold` (the worse of the two sets the exit code). Pass a
|
|
232
281
|
directory (or several sources) to audit a whole layer in one boot — tests are
|
|
233
282
|
auto-paired by convention and the report breaks down per source.
|
|
@@ -266,7 +315,7 @@ the format.
|
|
|
266
315
|
|
|
267
316
|
## For AI agents & pipelines
|
|
268
317
|
|
|
269
|
-
Mutineer is built for programmatic use — versioned JSON,
|
|
318
|
+
Mutineer is built for programmatic use — versioned JSON, [mutant ids](#mutant-ids) that survive unrelated edits,
|
|
270
319
|
structured exit codes, and diff-scoped runs. See:
|
|
271
320
|
|
|
272
321
|
- **AI agents & CI recipes** — the agent inner-loop and CI-gate recipes (and how
|
data/lib/mutineer/baseline.rb
CHANGED
|
@@ -1,18 +1,20 @@
|
|
|
1
1
|
# frozen_string_literal: true
|
|
2
2
|
|
|
3
3
|
require "json"
|
|
4
|
+
require "set"
|
|
4
5
|
require_relative "config" # for Mutineer::ConfigError
|
|
6
|
+
require_relative "project_path"
|
|
5
7
|
|
|
6
8
|
module Mutineer
|
|
7
9
|
# CI baseline/delta gating. A baseline is a prior
|
|
8
10
|
# `mutineer run --format json` document (no bespoke format to version).
|
|
9
|
-
# Diff the current run against it by
|
|
11
|
+
# Diff the current run against it by survivor id: a NEW survivor
|
|
10
12
|
# (id present now, absent in the baseline) OR a score drop is a regression the
|
|
11
13
|
# CLI turns into exit 1. Pure data, stdlib `json` only, no fork, no Rails, so
|
|
12
14
|
# it is testable in isolation from a canned JSON + a hand-built AggregateResult.
|
|
13
15
|
class Baseline
|
|
14
16
|
# The verdict of diffing a current run against the baseline.
|
|
15
|
-
# new_survivors - current Result objects whose
|
|
17
|
+
# new_survivors - current Result objects whose id is absent from
|
|
16
18
|
# the baseline (the regressions to name).
|
|
17
19
|
# fixed_survivors - baseline survivor hashes absent from the current run
|
|
18
20
|
# (informational, never gates). Empty when either side
|
|
@@ -26,8 +28,18 @@ module Mutineer
|
|
|
26
28
|
# render them side by side. False means the score-drop
|
|
27
29
|
# check was skipped, not that it passed.
|
|
28
30
|
# regressed - any new survivors OR a score drop.
|
|
31
|
+
# legacy_matches - current survivors found in an old-format baseline
|
|
32
|
+
# (no `summary.id_format`) only through their old-format
|
|
33
|
+
# id (#126). Non-zero means the baseline should be
|
|
34
|
+
# regenerated; the CLI warns. Always 0 for a new-format
|
|
35
|
+
# baseline.
|
|
29
36
|
Delta = Data.define(:new_survivors, :fixed_survivors,
|
|
30
|
-
:score_before, :score_after, :score_drop, :score_comparable, :regressed
|
|
37
|
+
:score_before, :score_after, :score_drop, :score_comparable, :regressed,
|
|
38
|
+
:legacy_matches) do
|
|
39
|
+
# @param legacy_matches [Integer] survivors matched only through an old-format id.
|
|
40
|
+
# @return [void]
|
|
41
|
+
def initialize(legacy_matches: 0, **) = super
|
|
42
|
+
end
|
|
31
43
|
|
|
32
44
|
# Load a prior --format json run. Raises ConfigError (NOT exit: a data class
|
|
33
45
|
# must never kill the host) on a missing/unreadable file, unparseable JSON,
|
|
@@ -73,35 +85,74 @@ module Mutineer
|
|
|
73
85
|
# Strict literal true only: a malformed value (say the STRING "false" in a
|
|
74
86
|
# hand-edited baseline) must not silently disable the score-drop gate.
|
|
75
87
|
@scoped = doc.dig("summary", "scoped") == true
|
|
88
|
+
# nil for a report written before ids included the file path (#126).
|
|
89
|
+
@id_format = doc.dig("summary", "id_format")
|
|
76
90
|
end
|
|
77
91
|
|
|
78
|
-
# Diff a current AggregateResult against this baseline by
|
|
92
|
+
# Diff a current AggregateResult against this baseline by survivor id.
|
|
79
93
|
# `epsilon` tolerates float jitter on the score (default 0.0 = any drop
|
|
80
94
|
# gates).
|
|
81
95
|
#
|
|
82
96
|
# `scoped: true` marks the current run as diff-scoped (`--since`): its score
|
|
83
97
|
# is computed over only the changed-line mutants, a different denominator
|
|
84
98
|
# from a full-run baseline, so comparing the two scores manufactures false
|
|
85
|
-
# regressions. A scoped diff keeps the new-survivor gate (
|
|
99
|
+
# regressions. A scoped diff keeps the new-survivor gate (ids compare
|
|
86
100
|
# fine across scopes) and still reports both scores, but never sets
|
|
87
101
|
# score_drop.
|
|
88
102
|
#
|
|
103
|
+
# `id_map` maps each current new-format id to its old-format id (#126). A
|
|
104
|
+
# baseline without `summary.id_format` stores old-format ids, so a current
|
|
105
|
+
# survivor matches if its new OR old id is stored, and a stored id seen under
|
|
106
|
+
# either form is not fixed. Matches made only through the old id are counted
|
|
107
|
+
# on Delta#legacy_matches. The old id has no file path, so an old-id match
|
|
108
|
+
# must also come from the same file (the stored survivor's `file`, normalized
|
|
109
|
+
# against `project_root`): an equal old id from another file is a different
|
|
110
|
+
# mutant and stays new. A stored `file` that is absolute and outside
|
|
111
|
+
# `project_root` (a baseline written on another machine) can never equal a
|
|
112
|
+
# current file, so that survivor matches on its old id alone, as before #126.
|
|
113
|
+
#
|
|
89
114
|
# @param aggregate [Mutineer::AggregateResult] current results.
|
|
90
115
|
# @param epsilon [Float] score-drop tolerance.
|
|
91
116
|
# @param scoped [Boolean] current run was diff-scoped (`--since`).
|
|
117
|
+
# @param id_map [Hash{String => String}] current new id => old-format id.
|
|
118
|
+
# @param project_root [String] root that survivor `file` paths resolve against.
|
|
92
119
|
# @return [Mutineer::Baseline::Delta] delta summary.
|
|
93
|
-
def diff(aggregate, epsilon: 0.0, scoped: false)
|
|
120
|
+
def diff(aggregate, epsilon: 0.0, scoped: false, id_map: {}, project_root: Dir.pwd)
|
|
94
121
|
current = aggregate.surviving_mutants
|
|
95
|
-
|
|
96
|
-
|
|
122
|
+
baseline_ids = @survivors.map { |h| h["id"] }.to_set
|
|
123
|
+
# A new-format baseline never matches on old ids.
|
|
124
|
+
legacy = @id_format.nil? ? id_map : {}
|
|
125
|
+
file_key = ->(path) { path && ProjectPath.relative(path, project_root) }
|
|
126
|
+
# [old id, file] for each stored survivor: an old id alone is ambiguous.
|
|
127
|
+
baseline_pairs = @survivors.map { |h| [h["id"], file_key.call(h["file"])] }.to_set
|
|
128
|
+
# Old ids stored with a file from another machine: matched on the id alone.
|
|
129
|
+
foreign = @survivors.select { |h| foreign_file?(h["file"], file_key) }.to_set
|
|
130
|
+
foreign_ids = foreign.map { |h| h["id"] }.to_set
|
|
131
|
+
legacy_pair = ->(r) { [legacy[r.id], file_key.call(r.subject&.file)] }
|
|
97
132
|
|
|
98
|
-
new_survivors =
|
|
133
|
+
new_survivors = []
|
|
134
|
+
legacy_matches = 0
|
|
135
|
+
current.each do |r|
|
|
136
|
+
next if baseline_ids.include?(r.id)
|
|
137
|
+
|
|
138
|
+
if legacy[r.id] && (baseline_pairs.include?(legacy_pair.call(r)) || foreign_ids.include?(legacy[r.id]))
|
|
139
|
+
legacy_matches += 1
|
|
140
|
+
else
|
|
141
|
+
new_survivors << r
|
|
142
|
+
end
|
|
143
|
+
end
|
|
144
|
+
current_ids = current.map(&:id).to_set
|
|
145
|
+
current_pairs = current.select { |r| legacy[r.id] }.map { |r| legacy_pair.call(r) }.to_set
|
|
146
|
+
current_legacy_ids = current.filter_map { |r| legacy[r.id] }.to_set
|
|
99
147
|
# Under a diff-scoped side an out-of-scope baseline survivor was never
|
|
100
148
|
# re-tested, so reporting it "fixed" would be false: empty is honest.
|
|
101
149
|
fixed = if scoped || @scoped
|
|
102
150
|
[]
|
|
103
151
|
else
|
|
104
|
-
@survivors.reject
|
|
152
|
+
@survivors.reject do |h|
|
|
153
|
+
current_ids.include?(h["id"]) || current_pairs.include?([h["id"], file_key.call(h["file"])]) ||
|
|
154
|
+
(foreign.include?(h) && current_legacy_ids.include?(h["id"]))
|
|
155
|
+
end
|
|
105
156
|
end
|
|
106
157
|
|
|
107
158
|
current_score = aggregate.mutation_score
|
|
@@ -116,7 +167,19 @@ module Mutineer
|
|
|
116
167
|
Delta.new(new_survivors: new_survivors, fixed_survivors: fixed,
|
|
117
168
|
score_before: @score, score_after: current_score,
|
|
118
169
|
score_drop: score_drop, score_comparable: comparable,
|
|
119
|
-
regressed: !new_survivors.empty? || score_drop)
|
|
170
|
+
regressed: !new_survivors.empty? || score_drop, legacy_matches: legacy_matches)
|
|
171
|
+
end
|
|
172
|
+
|
|
173
|
+
private
|
|
174
|
+
|
|
175
|
+
# True when a stored survivor's `file` is absolute and still absolute after
|
|
176
|
+
# normalizing against the project root, so it lies outside this checkout.
|
|
177
|
+
#
|
|
178
|
+
# @param file [String, nil] the stored survivor's `file`.
|
|
179
|
+
# @param file_key [Proc] normalizes a path against the project root.
|
|
180
|
+
# @return [Boolean] whether the file can never equal a current file.
|
|
181
|
+
def foreign_file?(file, file_key)
|
|
182
|
+
!file.nil? && File.absolute_path?(file) && File.absolute_path?(file_key.call(file))
|
|
120
183
|
end
|
|
121
184
|
end
|
|
122
185
|
end
|
data/lib/mutineer/cli.rb
CHANGED
|
@@ -162,6 +162,7 @@ module Mutineer
|
|
|
162
162
|
|
|
163
163
|
case argv.first
|
|
164
164
|
when "run"
|
|
165
|
+
warn_config_root_mismatch(file_path, config.project_root) if file_path
|
|
165
166
|
# A directory source expands to its **/*.rb files; literal files pass
|
|
166
167
|
# through. Test inference (when --test is omitted) happens in validate!.
|
|
167
168
|
config.sources = Pairing.expand_sources(argv[1..], project_root: config.project_root)
|
|
@@ -469,7 +470,8 @@ module Mutineer
|
|
|
469
470
|
exit 2
|
|
470
471
|
end
|
|
471
472
|
|
|
472
|
-
aggregate, source_map = Runner.execute(config)
|
|
473
|
+
aggregate, source_map, extras = Runner.execute(config)
|
|
474
|
+
warn_legacy_ignore_matches(extras[:legacy_ignore_matches])
|
|
473
475
|
reporter = Reporter.new(aggregate, source_map)
|
|
474
476
|
|
|
475
477
|
# Diff the current run against the baseline (preflighted above) by the
|
|
@@ -479,12 +481,18 @@ module Mutineer
|
|
|
479
481
|
# so only the new-survivor half of the gate applies (see Baseline#diff).
|
|
480
482
|
delta = if config.baseline
|
|
481
483
|
Baseline.load(config.baseline).diff(aggregate, epsilon: config.baseline_epsilon,
|
|
482
|
-
scoped: !config.since.nil
|
|
484
|
+
scoped: !config.since.nil?,
|
|
485
|
+
id_map: extras[:id_map],
|
|
486
|
+
project_root: config.project_root)
|
|
483
487
|
end
|
|
488
|
+
warn_legacy_baseline if delta&.legacy_matches&.positive?
|
|
484
489
|
|
|
490
|
+
# ignore counts old-format entries (one warning each), not the ids they matched.
|
|
491
|
+
legacy_id_matches = { ignore: extras[:legacy_ignore_matches].size,
|
|
492
|
+
baseline: delta ? delta.legacy_matches : 0 }
|
|
485
493
|
reporter.report(out: $stdout, err: $stderr, threshold: config.threshold,
|
|
486
494
|
format: config.format, output: config.output, baseline: delta,
|
|
487
|
-
scoped: !config.since.nil
|
|
495
|
+
scoped: !config.since.nil?, legacy_id_matches: legacy_id_matches)
|
|
488
496
|
|
|
489
497
|
# Warn (stderr, so it never pollutes json/html) that an external run's score
|
|
490
498
|
# is not comparable to an in-process run: no coverage narrowing (uncovered
|
|
@@ -523,6 +531,64 @@ module Mutineer
|
|
|
523
531
|
"enable with --operators <list>."
|
|
524
532
|
end
|
|
525
533
|
|
|
534
|
+
# Warns once when the loaded .mutineer.yml sits outside the run directory
|
|
535
|
+
# (#126). Mutant ids hash each file's path relative to the run directory,
|
|
536
|
+
# but the config is found by walking up, so a run from a subdirectory loads
|
|
537
|
+
# the same ignore list while its ids no longer match. A config in the home
|
|
538
|
+
# directory is a personal default, not a project root, so it never warns.
|
|
539
|
+
#
|
|
540
|
+
# @param file_path [String] the .mutineer.yml that was loaded.
|
|
541
|
+
# @param project_root [String] the run directory ids are relative to.
|
|
542
|
+
# @return [void]
|
|
543
|
+
def self.warn_config_root_mismatch(file_path, project_root)
|
|
544
|
+
config_dir = ProjectPath.root_real(File.dirname(file_path))
|
|
545
|
+
return if config_dir == ProjectPath.root_real(project_root)
|
|
546
|
+
return if config_dir == ProjectPath.root_real(Dir.home)
|
|
547
|
+
|
|
548
|
+
warn "[mutineer] loaded #{file_path}, but mutant ids are relative to the run directory " \
|
|
549
|
+
"#{project_root}, not to #{config_dir}. Ignore ids and baselines written from " \
|
|
550
|
+
"#{config_dir} will not match this run. Run mutineer from #{config_dir}."
|
|
551
|
+
end
|
|
552
|
+
|
|
553
|
+
# Warns once per old-format `ignore:` entry (#126), naming each new id it
|
|
554
|
+
# matched with that mutant's file and subject. mutineer cannot tell a full
|
|
555
|
+
# run from a narrowed one, so the text always says the list covers only this
|
|
556
|
+
# run's mutants. An entry that matched more than one distinct mutant (in
|
|
557
|
+
# other files, or same-named methods in one file) over-matched: the old id
|
|
558
|
+
# could not tell them apart, so replacing it with every new id would keep
|
|
559
|
+
# suppressing the mutants it hid by accident.
|
|
560
|
+
#
|
|
561
|
+
# @param matches [Hash{String => Array<Hash{Symbol => String}>}] old-format
|
|
562
|
+
# entry => one `{id:, file:, subject:}` hash per matched mutant.
|
|
563
|
+
# @return [void]
|
|
564
|
+
def self.warn_legacy_ignore_matches(matches)
|
|
565
|
+
matches.each do |old, hits|
|
|
566
|
+
listed = hits.map { |h| "#{h[:id]} (#{h[:file]}, #{h[:subject]})" }.join(", ")
|
|
567
|
+
advice = if hits.map { |h| h[:id] }.uniq.size > 1
|
|
568
|
+
"#{old} over-matched: the old format could not tell these mutants apart. " \
|
|
569
|
+
"Replace #{old} and keep only the ids for the mutant you meant to ignore, not all of them."
|
|
570
|
+
else
|
|
571
|
+
"Replace #{old} with the new ids in your ignore list."
|
|
572
|
+
end
|
|
573
|
+
warn "[mutineer] ignore entry #{old} uses the old id format, which did not include the " \
|
|
574
|
+
"file path. It matched these new ids: #{listed}. This list covers only mutants in " \
|
|
575
|
+
"this run's sources and operators; a run over every source gives the complete " \
|
|
576
|
+
"replacement. #{advice}"
|
|
577
|
+
end
|
|
578
|
+
end
|
|
579
|
+
|
|
580
|
+
# Warns once that the --baseline file stores old-format ids (#126), so the
|
|
581
|
+
# diff fell back to matching on them. Called only when a survivor matched
|
|
582
|
+
# through an old id alone.
|
|
583
|
+
#
|
|
584
|
+
# @return [void]
|
|
585
|
+
def self.warn_legacy_baseline
|
|
586
|
+
warn "[mutineer] the baseline uses the old id format, which did not include the file " \
|
|
587
|
+
"path, so survivors were matched on their old ids and files. Regenerate the baseline " \
|
|
588
|
+
"(run with --format json and save the output), but only after every gate that reads " \
|
|
589
|
+
"it runs this mutineer version or later."
|
|
590
|
+
end
|
|
591
|
+
|
|
526
592
|
# Runs dry-run mode. Reuses Runner.collect_jobs (+ filter_since) so the
|
|
527
593
|
# candidate list cannot drift from a real run's job selection.
|
|
528
594
|
#
|
|
@@ -530,7 +596,8 @@ module Mutineer
|
|
|
530
596
|
# @return [void]
|
|
531
597
|
def self.dry_run(config)
|
|
532
598
|
operator_classes = MutatorRegistry.resolve(config.operators || MutatorRegistry::DEFAULT_NAMES)
|
|
533
|
-
jobs, ignored_results, source_map = Runner.collect_jobs(config, operator_classes)
|
|
599
|
+
jobs, ignored_results, source_map, extras = Runner.collect_jobs(config, operator_classes)
|
|
600
|
+
warn_legacy_ignore_matches(extras[:legacy_ignore_matches])
|
|
534
601
|
# Narrow jobs and ignored the same way so the summary matches the printed list.
|
|
535
602
|
if config.since
|
|
536
603
|
jobs = Runner.filter_since(jobs, source_map, config)
|
|
@@ -9,6 +9,7 @@ require "set"
|
|
|
9
9
|
require_relative "minitest_integration"
|
|
10
10
|
require_relative "test_runners"
|
|
11
11
|
require_relative "child_stdout"
|
|
12
|
+
require_relative "project_path"
|
|
12
13
|
|
|
13
14
|
module Mutineer
|
|
14
15
|
# Maps `(source_file, line) -> [test_files]` so each mutant runs only against
|
|
@@ -96,7 +97,7 @@ module Mutineer
|
|
|
96
97
|
# Is this source file's empty coverage the result of an *errored* capture
|
|
97
98
|
# rather than a genuine coverage gap? True iff some capture failed this run
|
|
98
99
|
# AND this file got zero coverage from any successful capture AND a failed
|
|
99
|
-
# test file maps to it by the
|
|
100
|
+
# test file maps to it by the _test/_spec/test_ naming convention. Derived
|
|
100
101
|
# purely from already-persisted state (@map keys + @failed_test_files); no
|
|
101
102
|
# rerun, no new cached field, no digest change.
|
|
102
103
|
#
|
|
@@ -142,10 +143,17 @@ module Mutineer
|
|
|
142
143
|
@map.keys.map { |k| k.rpartition(":").first }.to_set
|
|
143
144
|
end
|
|
144
145
|
|
|
145
|
-
# Basenames of failed test files with
|
|
146
|
-
#
|
|
146
|
+
# Basenames of the sources that failed test files pair with by convention:
|
|
147
|
+
# a trailing _test/_spec is stripped first, as pairing tries that form first.
|
|
147
148
|
def failed_test_targets
|
|
148
|
-
@failed_test_files.map
|
|
149
|
+
@failed_test_files.map do |t|
|
|
150
|
+
name = File.basename(t, ".rb")
|
|
151
|
+
case name
|
|
152
|
+
when /_(test|spec)\z/ then name.sub(/_(test|spec)\z/, "")
|
|
153
|
+
when "test_helper" then name # Minitest's support file pairs with no source
|
|
154
|
+
else name.delete_prefix("test_")
|
|
155
|
+
end
|
|
156
|
+
end.to_set
|
|
149
157
|
end
|
|
150
158
|
|
|
151
159
|
# Shared cache dance for both build paths: hit the digest-keyed cache, else
|
|
@@ -577,11 +585,17 @@ module Mutineer
|
|
|
577
585
|
loads = Array(test_paths).map { |t| "load #{absolute(t).inspect}" }.join("\n")
|
|
578
586
|
<<~RUBY
|
|
579
587
|
require "minitest"
|
|
588
|
+
require "stringio"
|
|
580
589
|
def Minitest.autorun; end
|
|
590
|
+
_report = StringIO.new
|
|
591
|
+
Minitest.define_singleton_method(:plugin_mutineer_report_init) { |options| reporter << Minitest::SummaryReporter.new(_report, options) }
|
|
592
|
+
Minitest.extensions << "mutineer_report"
|
|
581
593
|
$LOAD_PATH.unshift(*#{abs_load_paths.inspect})
|
|
582
|
-
#{abs_source_paths.inspect}.each { |f|
|
|
594
|
+
#{abs_source_paths.inspect}.each { |f| require f }
|
|
583
595
|
#{loads}
|
|
584
|
-
|
|
596
|
+
_passed = Minitest.run([])
|
|
597
|
+
$stderr.write(_report.string) unless _passed
|
|
598
|
+
exit(_passed ? 0 : 1)
|
|
585
599
|
RUBY
|
|
586
600
|
end
|
|
587
601
|
|
|
@@ -601,9 +615,10 @@ module Mutineer
|
|
|
601
615
|
end
|
|
602
616
|
RSpec::Core::Runner.disable_autorun!
|
|
603
617
|
$LOAD_PATH.unshift(*#{abs_load_paths.inspect})
|
|
604
|
-
#{abs_source_paths.inspect}.each { |f|
|
|
618
|
+
#{abs_source_paths.inspect}.each { |f| require f }
|
|
605
619
|
_sink = StringIO.new
|
|
606
620
|
status = RSpec::Core::Runner.run(["--no-color", #{specs}], _sink, _sink)
|
|
621
|
+
$stderr.write(_sink.string) unless status.zero?
|
|
607
622
|
exit(status.zero? ? 0 : 1)
|
|
608
623
|
RUBY
|
|
609
624
|
end
|
|
@@ -628,12 +643,17 @@ module Mutineer
|
|
|
628
643
|
require "coverage"
|
|
629
644
|
require "json"
|
|
630
645
|
require "minitest"
|
|
646
|
+
require "stringio"
|
|
631
647
|
def Minitest.autorun; end
|
|
648
|
+
_report = StringIO.new
|
|
649
|
+
Minitest.define_singleton_method(:plugin_mutineer_report_init) { |options| reporter << Minitest::SummaryReporter.new(_report, options) }
|
|
650
|
+
Minitest.extensions << "mutineer_report"
|
|
632
651
|
Coverage.start(lines: true)
|
|
633
652
|
$LOAD_PATH.unshift(*#{abs_load_paths.inspect})
|
|
634
|
-
#{abs_source_paths.inspect}.each { |f|
|
|
653
|
+
#{abs_source_paths.inspect}.each { |f| require f }
|
|
635
654
|
load #{absolute(test_path).inspect}
|
|
636
655
|
_passed = Minitest.run([])
|
|
656
|
+
$stderr.write(_report.string) unless _passed
|
|
637
657
|
_result.puts JSON.generate("passed" => _passed == true, "coverage" => Coverage.result,
|
|
638
658
|
"loaded_files" => #{loaded_files_expression})
|
|
639
659
|
_result.close
|
|
@@ -641,7 +661,7 @@ module Mutineer
|
|
|
641
661
|
end
|
|
642
662
|
|
|
643
663
|
# Same coverage-JSON contract as the minitest path, but driven by RSpec:
|
|
644
|
-
# require rspec/core lazily,
|
|
664
|
+
# require rspec/core lazily, require the sources under Coverage, then run the
|
|
645
665
|
# one spec via RSpec::Core::Runner. The JSON goes to the result channel (see
|
|
646
666
|
# {#spawn_script}), so spec output cannot corrupt it. A missing rspec makes
|
|
647
667
|
# the script exit non-zero -> capture() records a skipped (incomplete-map)
|
|
@@ -661,9 +681,10 @@ module Mutineer
|
|
|
661
681
|
RSpec::Core::Runner.disable_autorun!
|
|
662
682
|
Coverage.start(lines: true)
|
|
663
683
|
$LOAD_PATH.unshift(*#{abs_load_paths.inspect})
|
|
664
|
-
#{abs_source_paths.inspect}.each { |f|
|
|
684
|
+
#{abs_source_paths.inspect}.each { |f| require f }
|
|
665
685
|
_sink = StringIO.new
|
|
666
686
|
_status = RSpec::Core::Runner.run(["--no-color", #{absolute(test_path).inspect}], _sink, _sink)
|
|
687
|
+
$stderr.write(_sink.string) unless _status.zero?
|
|
667
688
|
_result.puts JSON.generate("passed" => _status.zero?, "coverage" => Coverage.result,
|
|
668
689
|
"loaded_files" => #{loaded_files_expression})
|
|
669
690
|
_result.close
|
|
@@ -702,11 +723,7 @@ module Mutineer
|
|
|
702
723
|
#
|
|
703
724
|
# @api private
|
|
704
725
|
# @return [String] realpath of the project root when it exists.
|
|
705
|
-
def project_root_real
|
|
706
|
-
File.realpath(File.expand_path(@project_root))
|
|
707
|
-
rescue Errno::ENOENT
|
|
708
|
-
File.expand_path(@project_root)
|
|
709
|
-
end
|
|
726
|
+
def project_root_real = ProjectPath.root_real(@project_root)
|
|
710
727
|
|
|
711
728
|
# Project-local `.rb` files loaded in this process at capture time.
|
|
712
729
|
#
|
|
@@ -887,38 +904,18 @@ module Mutineer
|
|
|
887
904
|
# @return [Array<String>] absolute load paths.
|
|
888
905
|
def abs_load_paths = @load_paths.map { |p| absolute(p) }
|
|
889
906
|
|
|
890
|
-
# Relativizes a path against the project root.
|
|
907
|
+
# Relativizes a path against the project root (see {ProjectPath.relative}).
|
|
891
908
|
#
|
|
892
909
|
# @api private
|
|
893
910
|
# @param path [String] path to relativize.
|
|
894
|
-
# @return [String] relative path.
|
|
895
|
-
def relativize(path)
|
|
896
|
-
abs = path.start_with?("/") ? path : absolute(path)
|
|
897
|
-
abs = realpath_if_exists(abs)
|
|
898
|
-
root = project_root_real
|
|
899
|
-
prefix = root.end_with?("/") ? root : "#{root}/"
|
|
900
|
-
return abs unless abs.start_with?(prefix)
|
|
911
|
+
# @return [String] relative path, or an absolute path when outside the root.
|
|
912
|
+
def relativize(path) = ProjectPath.relative(path, @project_root)
|
|
901
913
|
|
|
902
|
-
|
|
903
|
-
end
|
|
904
|
-
|
|
905
|
-
# Expands a path relative to the project root.
|
|
914
|
+
# Expands a path relative to the project root (see {ProjectPath.absolute}).
|
|
906
915
|
#
|
|
907
916
|
# @api private
|
|
908
917
|
# @param path [String] path to expand.
|
|
909
918
|
# @return [String] absolute path.
|
|
910
|
-
def absolute(path)
|
|
911
|
-
raw = File.absolute_path?(path) ? path : File.expand_path(path, @project_root)
|
|
912
|
-
realpath_if_exists(raw)
|
|
913
|
-
end
|
|
914
|
-
|
|
915
|
-
# Real path when the file exists, otherwise `path` unchanged.
|
|
916
|
-
#
|
|
917
|
-
# @api private
|
|
918
|
-
# @param path [String] absolute or relative path.
|
|
919
|
-
# @return [String]
|
|
920
|
-
def realpath_if_exists(path)
|
|
921
|
-
File.exist?(path) ? File.realpath(path) : path
|
|
922
|
-
end
|
|
919
|
+
def absolute(path) = ProjectPath.absolute(path, @project_root)
|
|
923
920
|
end
|
|
924
921
|
end
|