mutineer 0.11.4 → 1.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +64 -0
- data/README.md +20 -4
- data/lib/mutineer/baseline.rb +48 -10
- data/lib/mutineer/changed_lines.rb +11 -3
- data/lib/mutineer/cli.rb +13 -3
- data/lib/mutineer/config.rb +5 -0
- data/lib/mutineer/daemon_backend.rb +7 -2
- data/lib/mutineer/progress.rb +46 -0
- data/lib/mutineer/reporter.rb +22 -6
- data/lib/mutineer/runner.rb +6 -1
- data/lib/mutineer/version.rb +1 -1
- data/lib/mutineer/worker_pool.rb +7 -1
- data/lib/mutineer.rb +1 -0
- metadata +2 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 9104fa94a852cbd1257f9dce0af285865d98da1100849f5d1b8c5af12d6513fa
|
|
4
|
+
data.tar.gz: 4ca6421fdd7ef6ef862813cbd74481ec6babf176991c8d34301368861ea19272
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 66a7b0eae5bb41331d267b4c26a0b72e9023177d89d9d21cc679e712978836c4884b8b0347fd8263f55041897c2581e7117d5ae522965efbdb9af0daf5b47fee
|
|
7
|
+
data.tar.gz: cf1ed699c65b733b717ce5ed6ee95f9f50387fb2963b0ecd5516fa43d6a18bd21e0c354f2c85bde2df56cec305446cd988697f8eb5ae5b080bdfd9d2edc0fa06
|
data/CHANGELOG.md
CHANGED
|
@@ -6,6 +6,69 @@ All notable changes to this project are documented here. The format is based on
|
|
|
6
6
|
|
|
7
7
|
## [Unreleased]
|
|
8
8
|
|
|
9
|
+
## [1.0.0] - 2026-09-08
|
|
10
|
+
|
|
11
|
+
The GitHub Action's PR default changes in this release, which is why it is a
|
|
12
|
+
new major: workflows pinned to `davidteren/mutineer@v0` keep the old full-scan
|
|
13
|
+
behavior; upgrading to `@v1` opts into diff-scoped PR runs (details under
|
|
14
|
+
Changed).
|
|
15
|
+
|
|
16
|
+
### Added
|
|
17
|
+
- **The Action reports where CI readers look**: with the default JSON format it
|
|
18
|
+
writes a score/pass-fail table (plus the baseline delta, when `baseline` is
|
|
19
|
+
set) to the job step summary, and emits one `file=…,line=…` annotation per
|
|
20
|
+
surviving mutant (up to 50; the summary lists the first 20, the full set
|
|
21
|
+
stays in the JSON report) so results land on the PR diff instead of only in a
|
|
22
|
+
collapsed log group — `error` level when the gate failed, `warning` when it
|
|
23
|
+
passed. When `output` is unset the JSON report is routed to a temp file and
|
|
24
|
+
still printed to the log, and the `report` output now exposes the report
|
|
25
|
+
path in both cases so a later step can consume the JSON without scraping
|
|
26
|
+
the log (#86).
|
|
27
|
+
- **Progress during the run**: every backend prints `[mutineer] N/M mutants
|
|
28
|
+
(P%)` to stderr at each 10% step, so a long run is never silent between
|
|
29
|
+
config resolution and the report. Stdout stays byte-exact for `--format json`
|
|
30
|
+
and `--output`. `WorkerPool#run` gains an optional `on_result:` callback for
|
|
31
|
+
this (called in the parent per reaped result) (#86).
|
|
32
|
+
|
|
33
|
+
### Changed
|
|
34
|
+
- **PR runs scope themselves in the GitHub Action**: on `pull_request` events
|
|
35
|
+
the `since` input now defaults to the PR base, so the action grades just the
|
|
36
|
+
diff out of the box. **Migration note (the reason for the major bump)**: a
|
|
37
|
+
PR gate that previously full-scanned now scores only the PR's changed lines,
|
|
38
|
+
so `threshold` applies to fewer mutants. Stay on `@v0` to keep the old
|
|
39
|
+
default, or pass `since: none` on `@v1` for full scans. Workflows that
|
|
40
|
+
already pass a non-empty `since` are unchanged (an explicit empty string is
|
|
41
|
+
indistinguishable from unset and picks up the new default). A PR with no
|
|
42
|
+
changed source lines (docs- or test-only) scores zero mutants and passes the
|
|
43
|
+
scoped gate vacuously; keep a full-scan baseline refresh on main as the
|
|
44
|
+
backstop. With `use-bundler: true` the caller's Gemfile picks the gem, and
|
|
45
|
+
the new default needs mutineer >= 1.0.0 (the action enforces the floor with
|
|
46
|
+
a clear error). The action scopes to the PR's exact base commit from the
|
|
47
|
+
event payload (immune to the
|
|
48
|
+
base branch advancing mid-job), falling back to a fresh fetch of the base
|
|
49
|
+
branch tip; when neither can be resolved it warns and runs without an
|
|
50
|
+
action-provided `--since` (a `.mutineer.yml` `since:` key, if any, still
|
|
51
|
+
applies). The default deliberately does NOT fire on `pull_request_target`:
|
|
52
|
+
checkout there defaults to the base branch, so auto-scoping would diff the
|
|
53
|
+
base against itself and green the gate on an empty run (#86).
|
|
54
|
+
- **`--baseline` on a diff-scoped run gates on new survivors only**: a
|
|
55
|
+
`--since` run's score covers only the changed-line mutants, a different
|
|
56
|
+
denominator from a full-run baseline, so comparing the two scores
|
|
57
|
+
manufactured false regressions (a 3/4-mutant PR at 75% "dropped" from a
|
|
58
|
+
92% whole-repo baseline with zero new survivors). With `--since`, the
|
|
59
|
+
score-drop half of the baseline gate is skipped; new-survivor detection by
|
|
60
|
+
stable id (and the reported before/after scores) are unchanged. The JSON
|
|
61
|
+
report records the scope in a new additive `summary.scoped` key
|
|
62
|
+
(`schema_version` 1.3), and the `baseline` block records `score_comparable`
|
|
63
|
+
so a consumer knows when not to render the two scores as a comparison
|
|
64
|
+
(`fixed_survivors` is likewise empty under a scoped side: an out-of-scope
|
|
65
|
+
survivor was never re-tested, so absence does not mean fixed). The reverse
|
|
66
|
+
direction is a hard guard: a scoped report is refused as a baseline (exit 2
|
|
67
|
+
with a regenerate hint), because survivors outside its diff would all read
|
|
68
|
+
as new regressions. A new `--no-since` flag disables diff scoping explicitly: a
|
|
69
|
+
typed no beats a `.mutineer.yml` `since:` key, and the action's
|
|
70
|
+
`since: none` passes it through (#86).
|
|
71
|
+
|
|
9
72
|
## [0.11.4] - 2026-07-29
|
|
10
73
|
|
|
11
74
|
### Fixed
|
|
@@ -318,6 +381,7 @@ Rails hardening + CI batch (issues #8–#13), all verified Rails-free.
|
|
|
318
381
|
- `.mutineer.yml` configuration (CLI > config > default precedence).
|
|
319
382
|
- Byte-correct source handling for multibyte (UTF-8) sources.
|
|
320
383
|
|
|
384
|
+
[1.0.0]: https://github.com/davidteren/mutineer/releases/tag/v1.0.0
|
|
321
385
|
[0.11.4]: https://github.com/davidteren/mutineer/releases/tag/v0.11.4
|
|
322
386
|
[0.11.3]: https://github.com/davidteren/mutineer/releases/tag/v0.11.3
|
|
323
387
|
[0.11.2]: https://github.com/davidteren/mutineer/releases/tag/v0.11.2
|
data/README.md
CHANGED
|
@@ -51,7 +51,8 @@ mutineer run lib/calculator.rb --test test/calculator_test.rb --threshold 90
|
|
|
51
51
|
| `--only NAME` | Restrict to one fully-qualified subject, e.g. `Calculator#add` |
|
|
52
52
|
| `--framework NAME` | `minitest` (default) or `rspec`; auto-detected as rspec when most `--test` files end in `_spec.rb` |
|
|
53
53
|
| `--since REF` | Only mutate lines changed since git `REF` (e.g. `origin/main`) — ideal for PR CI |
|
|
54
|
-
| `--
|
|
54
|
+
| `--no-since` | Disable diff scoping; a typed no beats a `.mutineer.yml` `since:` key |
|
|
55
|
+
| `--baseline FILE` | Compare against a prior `--format json` run; exit 1 on new survivors / score drop (score drop is skipped under `--since`, whose score covers a different denominator; see [CI](#ci-gating)) |
|
|
55
56
|
| `--baseline-epsilon FLOAT` | Score-drop tolerance for `--baseline` (default: 0) |
|
|
56
57
|
| `--jobs N` | Parallel worker count (default: processor count; `1` under `--rails`) |
|
|
57
58
|
| `--verbose` | Surface the real error when a fork capture fails (alias `--debug`) |
|
|
@@ -233,17 +234,32 @@ This repo ships a composite action (`action.yml`) that wraps the CLI for CI:
|
|
|
233
234
|
|
|
234
235
|
```yaml
|
|
235
236
|
- uses: actions/checkout@v4
|
|
236
|
-
with: { fetch-depth: 0 } # --since needs full history
|
|
237
237
|
- uses: ruby/setup-ruby@v1
|
|
238
238
|
with: { ruby-version: "3.4", bundler-cache: true }
|
|
239
|
-
- uses: davidteren/mutineer@
|
|
239
|
+
- uses: davidteren/mutineer@v1
|
|
240
240
|
with:
|
|
241
241
|
sources: app/
|
|
242
|
-
since: origin/${{ github.base_ref }}
|
|
243
242
|
baseline: .mutineer/baseline.json
|
|
244
243
|
threshold: "90"
|
|
245
244
|
```
|
|
246
245
|
|
|
246
|
+
**Default change:** on `pull_request` events (not `pull_request_target`) the
|
|
247
|
+
action scopes the run to the PR's changed lines, diffing against the PR's exact
|
|
248
|
+
base commit (fetched by the action itself when the checkout is shallow; falls
|
|
249
|
+
back to the base branch tip). Pass `since: none` for a full scan, or an
|
|
250
|
+
explicit `since:` ref (which needs `fetch-depth: 0` on checkout).
|
|
251
|
+
|
|
252
|
+
With the default JSON format the action also:
|
|
253
|
+
|
|
254
|
+
- writes a score summary to the job's step summary;
|
|
255
|
+
- annotates surviving mutants on the PR diff, up to 50 (`error` level when the
|
|
256
|
+
gate failed, `warning` when it passed);
|
|
257
|
+
- exposes the report path via the `report` output for later steps (with
|
|
258
|
+
`format: human`/`html` this needs the `output` input).
|
|
259
|
+
|
|
260
|
+
The CLI prints a progress line to the log at every 10% of the run, whatever
|
|
261
|
+
the format.
|
|
262
|
+
|
|
247
263
|
## For AI agents & pipelines
|
|
248
264
|
|
|
249
265
|
Mutineer is built for programmatic use — versioned JSON, stable mutant ids,
|
data/lib/mutineer/baseline.rb
CHANGED
|
@@ -15,12 +15,19 @@ module Mutineer
|
|
|
15
15
|
# new_survivors - current Result objects whose stable id is absent from
|
|
16
16
|
# the baseline (the regressions to name).
|
|
17
17
|
# fixed_survivors - baseline survivor hashes absent from the current run
|
|
18
|
-
# (informational, never gates).
|
|
18
|
+
# (informational, never gates). Empty when either side
|
|
19
|
+
# is diff-scoped: an out-of-scope baseline survivor was
|
|
20
|
+
# never re-tested, so absence does not mean fixed.
|
|
19
21
|
# score_drop - current score < baseline score - epsilon. nil on
|
|
20
|
-
# either side skips the check
|
|
22
|
+
# either side skips the check, and a diff-scoped run
|
|
23
|
+
# (`scoped: true`) never sets it (see #diff).
|
|
24
|
+
# score_comparable - the two scores share a denominator (neither side was
|
|
25
|
+
# diff-scoped and both are non-nil), so a consumer may
|
|
26
|
+
# render them side by side. False means the score-drop
|
|
27
|
+
# check was skipped, not that it passed.
|
|
21
28
|
# regressed - any new survivors OR a score drop.
|
|
22
29
|
Delta = Data.define(:new_survivors, :fixed_survivors,
|
|
23
|
-
:score_before, :score_after, :score_drop, :regressed)
|
|
30
|
+
:score_before, :score_after, :score_drop, :score_comparable, :regressed)
|
|
24
31
|
|
|
25
32
|
# Load a prior --format json run. Raises ConfigError (NOT exit: a data class
|
|
26
33
|
# must never kill the host) on a missing/unreadable file, unparseable JSON,
|
|
@@ -35,6 +42,15 @@ module Mutineer
|
|
|
35
42
|
unless doc.is_a?(Hash) && doc["schema_version"] && doc["survivors"].is_a?(Array)
|
|
36
43
|
raise ConfigError, "not a Mutineer JSON report: #{path}"
|
|
37
44
|
end
|
|
45
|
+
# A diff-scoped report covers only that diff's mutants: used as a
|
|
46
|
+
# baseline, every survivor outside the original diff would read as a NEW
|
|
47
|
+
# regression, and its score shares no denominator with any other run.
|
|
48
|
+
# Refuse loudly (exit 2 via the CLI) rather than gate unreliably.
|
|
49
|
+
if doc.dig("summary", "scoped") == true
|
|
50
|
+
raise ConfigError, "#{path} was written by a --since run and covers only that diff's " \
|
|
51
|
+
"mutants; regenerate the baseline from a full run " \
|
|
52
|
+
"(use --no-since if .mutineer.yml sets since:)"
|
|
53
|
+
end
|
|
38
54
|
|
|
39
55
|
new(doc)
|
|
40
56
|
rescue JSON::ParserError => e
|
|
@@ -45,39 +61,61 @@ module Mutineer
|
|
|
45
61
|
|
|
46
62
|
# Builds a baseline from a JSON document.
|
|
47
63
|
#
|
|
48
|
-
# The baseline retains the survivor document
|
|
64
|
+
# The baseline retains the survivor document, score, and scope marker from
|
|
65
|
+
# the JSON report. A report whose `summary.scoped` is true came from a
|
|
66
|
+
# `--since` run: its score covers only changed-line mutants, so later diffs
|
|
67
|
+
# must not compare a full-run score against it.
|
|
49
68
|
#
|
|
50
69
|
# @param doc [Hash] parsed JSON document.
|
|
51
70
|
def initialize(doc)
|
|
52
71
|
@survivors = doc["survivors"] || []
|
|
53
72
|
@score = doc.dig("summary", "score")
|
|
73
|
+
# Strict literal true only: a malformed value (say the STRING "false" in a
|
|
74
|
+
# hand-edited baseline) must not silently disable the score-drop gate.
|
|
75
|
+
@scoped = doc.dig("summary", "scoped") == true
|
|
54
76
|
end
|
|
55
77
|
|
|
56
78
|
# Diff a current AggregateResult against this baseline by stable survivor id.
|
|
57
79
|
# `epsilon` tolerates float jitter on the score (default 0.0 = any drop
|
|
58
80
|
# gates).
|
|
59
81
|
#
|
|
82
|
+
# `scoped: true` marks the current run as diff-scoped (`--since`): its score
|
|
83
|
+
# is computed over only the changed-line mutants, a different denominator
|
|
84
|
+
# from a full-run baseline, so comparing the two scores manufactures false
|
|
85
|
+
# regressions. A scoped diff keeps the new-survivor gate (stable ids compare
|
|
86
|
+
# fine across scopes) and still reports both scores, but never sets
|
|
87
|
+
# score_drop.
|
|
88
|
+
#
|
|
60
89
|
# @param aggregate [Mutineer::AggregateResult] current results.
|
|
61
90
|
# @param epsilon [Float] score-drop tolerance.
|
|
91
|
+
# @param scoped [Boolean] current run was diff-scoped (`--since`).
|
|
62
92
|
# @return [Mutineer::Baseline::Delta] delta summary.
|
|
63
|
-
def diff(aggregate, epsilon: 0.0)
|
|
93
|
+
def diff(aggregate, epsilon: 0.0, scoped: false)
|
|
64
94
|
current = aggregate.surviving_mutants
|
|
65
95
|
current_ids = current.map(&:id)
|
|
66
96
|
baseline_ids = @survivors.map { |h| h["id"] }
|
|
67
97
|
|
|
68
98
|
new_survivors = current.reject { |r| baseline_ids.include?(r.id) }
|
|
69
|
-
|
|
99
|
+
# Under a diff-scoped side an out-of-scope baseline survivor was never
|
|
100
|
+
# re-tested, so reporting it "fixed" would be false: empty is honest.
|
|
101
|
+
fixed = if scoped || @scoped
|
|
102
|
+
[]
|
|
103
|
+
else
|
|
104
|
+
@survivors.reject { |h| current_ids.include?(h["id"]) }
|
|
105
|
+
end
|
|
70
106
|
|
|
71
107
|
current_score = aggregate.mutation_score
|
|
72
108
|
# nil-score discipline (mirrors Reporter#exit_code): a score absent on
|
|
73
109
|
# either side cannot be compared. Skip the drop check, keep the new-
|
|
74
|
-
# survivor check.
|
|
75
|
-
|
|
76
|
-
|
|
110
|
+
# survivor check. Same when EITHER side is diff-scoped (the current run
|
|
111
|
+
# via `scoped:`, or the stored baseline via its `summary.scoped` marker):
|
|
112
|
+
# the denominators differ, so the scores are not comparable.
|
|
113
|
+
comparable = !scoped && !@scoped && !@score.nil? && !current_score.nil?
|
|
114
|
+
score_drop = comparable && current_score < @score - epsilon
|
|
77
115
|
|
|
78
116
|
Delta.new(new_survivors: new_survivors, fixed_survivors: fixed,
|
|
79
117
|
score_before: @score, score_after: current_score,
|
|
80
|
-
score_drop: score_drop,
|
|
118
|
+
score_drop: score_drop, score_comparable: comparable,
|
|
81
119
|
regressed: !new_survivors.empty? || score_drop)
|
|
82
120
|
end
|
|
83
121
|
end
|
|
@@ -57,16 +57,24 @@ module Mutineer
|
|
|
57
57
|
|
|
58
58
|
# Returns the stdout of `git -C <root> diff --unified=0 <ref> -- <file>`.
|
|
59
59
|
#
|
|
60
|
+
# A failure is warned, never silent: an empty result means "no changed
|
|
61
|
+
# lines", which under `--since` removes every mutant for the file — a green
|
|
62
|
+
# gate must not be manufactured by a broken diff without a trace.
|
|
63
|
+
#
|
|
60
64
|
# @param ref [String] git ref to diff against.
|
|
61
65
|
# @param abs_file [String] absolute path of the file being diffed.
|
|
62
66
|
# @param project_root [String] repository root for `git -C`.
|
|
63
|
-
# @return [String] diff text, or `""` on failure.
|
|
67
|
+
# @return [String] diff text, or `""` on failure (after a stderr warning).
|
|
64
68
|
def git_diff(ref, abs_file, project_root)
|
|
65
69
|
out, _err, status = Open3.capture3(
|
|
66
70
|
"git", "-C", project_root, "diff", "--unified=0", ref, "--", abs_file
|
|
67
71
|
)
|
|
68
|
-
status.success?
|
|
69
|
-
|
|
72
|
+
return out if status.success?
|
|
73
|
+
|
|
74
|
+
warn "[mutineer] git diff failed for #{abs_file}; its lines will not be mutated (--since)"
|
|
75
|
+
""
|
|
76
|
+
rescue StandardError => e
|
|
77
|
+
warn "[mutineer] git diff failed for #{abs_file} (#{e.class}); its lines will not be mutated (--since)"
|
|
70
78
|
""
|
|
71
79
|
end
|
|
72
80
|
end
|
data/lib/mutineer/cli.rb
CHANGED
|
@@ -41,6 +41,7 @@ module Mutineer
|
|
|
41
41
|
--baseline-epsilon FLOAT Score-drop tolerance for --baseline (default: 0)
|
|
42
42
|
--only NAME Restrict to one fully-qualified subject
|
|
43
43
|
--since REF Only mutate lines changed since git REF (e.g. origin/main)
|
|
44
|
+
--no-since Disable diff scoping (a typed no beats a .mutineer.yml since:)
|
|
44
45
|
--jobs N Parallel worker count (default: processor count)
|
|
45
46
|
--strategy NAME reload (whole-file) or redefine (surgical); default: reload
|
|
46
47
|
--framework NAME minitest or rspec (default: auto-detect from --test names)
|
|
@@ -96,6 +97,9 @@ module Mutineer
|
|
|
96
97
|
o.on("--fail-fast") { opts[:fail_fast] = true; explicit << :fail_fast }
|
|
97
98
|
o.on("--only NAME") { |v| opts[:only] = v; explicit << :only }
|
|
98
99
|
o.on("--since REF") { |v| opts[:since] = v; explicit << :since }
|
|
100
|
+
# A typed "no" must beat a .mutineer.yml `since:` key (CLI-over-config
|
|
101
|
+
# precedence): marking :since explicit with a nil value blocks the fill.
|
|
102
|
+
o.on("--no-since") { opts[:since] = nil; explicit << :since }
|
|
99
103
|
o.on("--test FILE") { |v| (opts[:tests] ||= []) << v }
|
|
100
104
|
o.on("--operators LIST") { |v| opts[:operators] = v.split(",").map(&:strip); explicit << :operators }
|
|
101
105
|
o.on("--threshold FLOAT") do |v|
|
|
@@ -465,11 +469,17 @@ module Mutineer
|
|
|
465
469
|
|
|
466
470
|
# Diff the current run against the baseline (preflighted above) by the
|
|
467
471
|
# stable survivor id. The delta is rendered inline (human section / additive
|
|
468
|
-
# json block) and gates exit independently of --threshold.
|
|
469
|
-
|
|
472
|
+
# json block) and gates exit independently of --threshold. A --since run is
|
|
473
|
+
# scoped: its score covers a different denominator than a full-run baseline,
|
|
474
|
+
# so only the new-survivor half of the gate applies (see Baseline#diff).
|
|
475
|
+
delta = if config.baseline
|
|
476
|
+
Baseline.load(config.baseline).diff(aggregate, epsilon: config.baseline_epsilon,
|
|
477
|
+
scoped: !config.since.nil?)
|
|
478
|
+
end
|
|
470
479
|
|
|
471
480
|
reporter.report(out: $stdout, err: $stderr, threshold: config.threshold,
|
|
472
|
-
format: config.format, output: config.output, baseline: delta
|
|
481
|
+
format: config.format, output: config.output, baseline: delta,
|
|
482
|
+
scoped: !config.since.nil?)
|
|
473
483
|
|
|
474
484
|
# Warn (stderr, so it never pollutes json/html) that an external run's score
|
|
475
485
|
# is not comparable to an in-process run: no coverage narrowing (uncovered
|
data/lib/mutineer/config.rb
CHANGED
|
@@ -182,6 +182,11 @@ module Mutineer
|
|
|
182
182
|
when "ignore" then Array(value).map(&:to_s)
|
|
183
183
|
when "baseline" then value.to_s
|
|
184
184
|
when "test_command" then value.to_s
|
|
185
|
+
when "since"
|
|
186
|
+
# false / empty normalize to nil ("no scoping"), so every consumer's
|
|
187
|
+
# nil-check (runner scoping, the report's scoped marker) agrees. A
|
|
188
|
+
# false left raw would skip scoping but still mark the report scoped.
|
|
189
|
+
value == false || value.to_s.empty? ? nil : value.to_s
|
|
185
190
|
else value
|
|
186
191
|
end
|
|
187
192
|
end
|
|
@@ -4,6 +4,7 @@ require_relative "parser"
|
|
|
4
4
|
require_relative "result"
|
|
5
5
|
require_relative "coverage_map"
|
|
6
6
|
require_relative "daemon_client"
|
|
7
|
+
require_relative "progress"
|
|
7
8
|
# No require_relative "runner" on purpose: runner.rb requires this file, and the
|
|
8
9
|
# reverse edge makes Ruby warn "circular require considered harmful" on every -w
|
|
9
10
|
# load. Runner is loaded first on every real path; requiring this file alone leaves
|
|
@@ -137,10 +138,12 @@ module Mutineer
|
|
|
137
138
|
client = DaemonClient.new(boot: boot_config(config, abs_tests),
|
|
138
139
|
app_root: config.project_root).start
|
|
139
140
|
results = []
|
|
141
|
+
progress = Progress.new(jobs.size)
|
|
140
142
|
begin
|
|
141
143
|
jobs.each_with_index do |job, i|
|
|
142
144
|
r = job_result(job, i, client, 0, config, coverage_map, abs_tests, source_map)
|
|
143
145
|
results << r
|
|
146
|
+
progress.tick
|
|
144
147
|
break if config.fail_fast && r.survived?
|
|
145
148
|
end
|
|
146
149
|
ensure
|
|
@@ -159,8 +162,9 @@ module Mutineer
|
|
|
159
162
|
# @api private
|
|
160
163
|
# @return [Array<Mutineer::Result>] one result per input job, in input order.
|
|
161
164
|
def self.run_parallel(jobs, worker_count, config, abs_tests, coverage_map, source_map)
|
|
162
|
-
results
|
|
163
|
-
|
|
165
|
+
results = Array.new(jobs.size)
|
|
166
|
+
progress = Progress.new(jobs.size)
|
|
167
|
+
queue = Queue.new
|
|
164
168
|
jobs.each_index { |i| queue << i }
|
|
165
169
|
|
|
166
170
|
# Built one at a time so a refused spawn part-way (EMFILE under a high --jobs)
|
|
@@ -189,6 +193,7 @@ module Mutineer
|
|
|
189
193
|
break
|
|
190
194
|
end
|
|
191
195
|
results[i] = job_result(jobs[i], i, client, worker, config, coverage_map, abs_tests, source_map)
|
|
196
|
+
progress.tick
|
|
192
197
|
end
|
|
193
198
|
rescue DaemonBootError
|
|
194
199
|
# The daemon gave up for good. Stop feeding the other workers rather
|
|
@@ -0,0 +1,46 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Mutineer
|
|
4
|
+
# Coarse run progress on stderr: one line per 10% step of completed mutants
|
|
5
|
+
# (which means every completion on a run smaller than ten). Shared by every
|
|
6
|
+
# backend so a long run is never silent between config resolution and the
|
|
7
|
+
# report. Thread-safe so the daemon's parallel workers can share one instance.
|
|
8
|
+
# Writes only to stderr, keeping `--format json` stdout and `--output` files
|
|
9
|
+
# byte-exact per the Reporter's stream contract.
|
|
10
|
+
class Progress
|
|
11
|
+
# Builds a progress counter for a run of `total` mutants.
|
|
12
|
+
#
|
|
13
|
+
# @param total [Integer] number of mutants that will run.
|
|
14
|
+
# @param io [IO] destination stream (defaults to stderr; injectable for tests).
|
|
15
|
+
def initialize(total, io: $stderr)
|
|
16
|
+
@total = total
|
|
17
|
+
@done = 0
|
|
18
|
+
@last_step = 0
|
|
19
|
+
@mutex = Mutex.new
|
|
20
|
+
@io = io
|
|
21
|
+
end
|
|
22
|
+
|
|
23
|
+
# Records one completed mutant and prints when a 10% boundary is crossed.
|
|
24
|
+
# A zero-total counter never prints (guards the division, and an empty run
|
|
25
|
+
# has nothing to report).
|
|
26
|
+
#
|
|
27
|
+
# @return [void]
|
|
28
|
+
def tick
|
|
29
|
+
@mutex.synchronize do
|
|
30
|
+
return if @total.zero?
|
|
31
|
+
|
|
32
|
+
@done += 1
|
|
33
|
+
step = (@done * 10) / @total
|
|
34
|
+
return if step <= @last_step
|
|
35
|
+
|
|
36
|
+
@last_step = step
|
|
37
|
+
@io.puts "[mutineer] #{@done}/#{@total} mutants (#{(@done * 100) / @total}%)"
|
|
38
|
+
end
|
|
39
|
+
rescue IOError, Errno::EPIPE
|
|
40
|
+
# Progress is best-effort: a closed stderr (piped consumer went away) must
|
|
41
|
+
# never kill the run — in the daemon's worker threads a raise here would
|
|
42
|
+
# propagate through Thread#join and abort scoring.
|
|
43
|
+
nil
|
|
44
|
+
end
|
|
45
|
+
end
|
|
46
|
+
end
|
data/lib/mutineer/reporter.rb
CHANGED
|
@@ -32,11 +32,14 @@ module Mutineer
|
|
|
32
32
|
|
|
33
33
|
# Single entry point. Branches on `format` ("human" | "json" | "html") and
|
|
34
34
|
# routes the rendered report to `output` (a file, with a stderr confirmation)
|
|
35
|
-
# or to `out`. Diagnostics always go to `err`.
|
|
36
|
-
|
|
35
|
+
# or to `out`. Diagnostics always go to `err`. `scoped` marks a diff-scoped
|
|
36
|
+
# (`--since`) run; the JSON report records it so a consumer (or a later
|
|
37
|
+
# `--baseline` load) knows the score covers only the changed-line mutants.
|
|
38
|
+
def report(out: $stdout, err: $stderr, threshold: 0.0, format: "human", output: nil,
|
|
39
|
+
baseline: nil, scoped: false)
|
|
37
40
|
rendered =
|
|
38
41
|
if format == "json"
|
|
39
|
-
json_report(baseline)
|
|
42
|
+
json_report(baseline, scoped: scoped)
|
|
40
43
|
elsif format == "html"
|
|
41
44
|
html_report
|
|
42
45
|
else
|
|
@@ -126,8 +129,10 @@ module Mutineer
|
|
|
126
129
|
#
|
|
127
130
|
# @api private
|
|
128
131
|
# @param baseline [Mutineer::Baseline::Delta, nil] baseline delta.
|
|
132
|
+
# @param scoped [Boolean] the run was diff-scoped (`--since`), so its score
|
|
133
|
+
# covers only the changed-line mutants (additive `summary.scoped` key).
|
|
129
134
|
# @return [String] JSON text.
|
|
130
|
-
def json_report(baseline = nil)
|
|
135
|
+
def json_report(baseline = nil, scoped: false)
|
|
131
136
|
killed = @agg.killed_count
|
|
132
137
|
survived = @agg.survived_count
|
|
133
138
|
# null (not 0.0) on an empty denominator, matching the nil-vs-0.0
|
|
@@ -136,7 +141,7 @@ module Mutineer
|
|
|
136
141
|
score = @agg.mutation_score
|
|
137
142
|
|
|
138
143
|
doc = {
|
|
139
|
-
schema_version: "1.
|
|
144
|
+
schema_version: "1.3",
|
|
140
145
|
summary: {
|
|
141
146
|
total: @agg.total, killed: killed, survived: survived,
|
|
142
147
|
no_coverage: @agg.no_coverage_count,
|
|
@@ -146,7 +151,11 @@ module Mutineer
|
|
|
146
151
|
ignored: @agg.ignored_count,
|
|
147
152
|
# The gate is computed from these two, so a consumer never re-derives them.
|
|
148
153
|
attempted: attempted_count, no_verdict: no_verdict_count,
|
|
149
|
-
score: score
|
|
154
|
+
score: score,
|
|
155
|
+
# Additive: true when the run was diff-scoped (--since). The score then
|
|
156
|
+
# covers only the changed-line mutants, so it is not comparable to a
|
|
157
|
+
# full-run score; Baseline#diff reads this to skip the score-drop gate.
|
|
158
|
+
scoped: scoped
|
|
150
159
|
},
|
|
151
160
|
survivors: @agg.surviving_mutants.map { |r| survivor_json(r) }
|
|
152
161
|
.sort_by { |h| [h[:file], h[:line], h[:operator]] },
|
|
@@ -306,6 +315,10 @@ module Mutineer
|
|
|
306
315
|
score_before: delta.score_before,
|
|
307
316
|
score_after: delta.score_after,
|
|
308
317
|
score_dropped: delta.score_drop,
|
|
318
|
+
# Additive: false when the score-drop check was skipped (a nil score or
|
|
319
|
+
# a diff-scoped side), so a consumer knows not to render the two scores
|
|
320
|
+
# as a comparison.
|
|
321
|
+
score_comparable: delta.score_comparable,
|
|
309
322
|
new_survivors: delta.new_survivors.map { |r| ignored_json(r) }
|
|
310
323
|
.sort_by { |h| [h[:file], h[:line], h[:operator]] },
|
|
311
324
|
fixed_survivors: delta.fixed_survivors.map do |h|
|
|
@@ -577,6 +590,9 @@ module Mutineer
|
|
|
577
590
|
out.puts " + #{r.subject.qualified_name} (#{file}:#{line}) #{r.mutation.operator}"
|
|
578
591
|
end
|
|
579
592
|
out.puts "score dropped #{delta.score_before}% -> #{delta.score_after}%" if delta.score_drop
|
|
593
|
+
# An OK verdict must not imply a check that never ran: say when the score
|
|
594
|
+
# comparison was skipped (a diff-scoped side or a nil score).
|
|
595
|
+
out.puts "score-drop check skipped (scores not comparable)" unless delta.score_comparable
|
|
580
596
|
out.puts(delta.regressed ? "REGRESSION vs baseline" : "OK: no regression vs baseline")
|
|
581
597
|
end
|
|
582
598
|
|
data/lib/mutineer/runner.rb
CHANGED
|
@@ -10,6 +10,7 @@ require_relative "coverage_map"
|
|
|
10
10
|
require_relative "changed_lines"
|
|
11
11
|
require_relative "mutator_registry"
|
|
12
12
|
require_relative "worker_pool"
|
|
13
|
+
require_relative "progress"
|
|
13
14
|
require_relative "mutant_id"
|
|
14
15
|
require_relative "file_swap"
|
|
15
16
|
require_relative "external_backend"
|
|
@@ -121,7 +122,9 @@ module Mutineer
|
|
|
121
122
|
begin
|
|
122
123
|
framework = config.framework
|
|
123
124
|
stop_when = config.fail_fast ? ->(r) { r.survived? } : nil
|
|
124
|
-
|
|
125
|
+
progress = Progress.new(jobs.size)
|
|
126
|
+
bare = WorkerPool.new(jobs_n).run(jobs, stop_when: stop_when,
|
|
127
|
+
on_result: ->(_r) { progress.tick }) do |subject, mutation|
|
|
125
128
|
run(mutation, source_file: subject.file, coverage_map: coverage_map,
|
|
126
129
|
subject: subject, strategy: strategy, rails: config.rails, framework: framework)
|
|
127
130
|
end
|
|
@@ -205,11 +208,13 @@ module Mutineer
|
|
|
205
208
|
timeout = [[smoke_elapsed * 3, 30].max, 300].min.ceil
|
|
206
209
|
|
|
207
210
|
results = []
|
|
211
|
+
progress = Progress.new(jobs.size)
|
|
208
212
|
begin
|
|
209
213
|
jobs.each do |subject, mutation, id|
|
|
210
214
|
r = run_external(subject, mutation, config.test_command, abs_tests,
|
|
211
215
|
timeout: timeout, verbose: config.verbose)
|
|
212
216
|
results << r.with(subject: subject, mutation: mutation, id: id)
|
|
217
|
+
progress.tick
|
|
213
218
|
break if config.fail_fast && r.survived? # stop at the first survivor
|
|
214
219
|
end
|
|
215
220
|
ensure
|
data/lib/mutineer/version.rb
CHANGED
data/lib/mutineer/worker_pool.rb
CHANGED
|
@@ -28,10 +28,15 @@ module Mutineer
|
|
|
28
28
|
# @param stop_when [Proc, nil] called with each collected Result; when it
|
|
29
29
|
# returns truthy, no further items are scheduled and the run drains and
|
|
30
30
|
# returns early (--fail-fast). Unscheduled slots stay nil.
|
|
31
|
+
# @param on_result [Proc, nil] called in the parent with each collected
|
|
32
|
+
# Result as it is reaped, in finish order (progress reporting); its
|
|
33
|
+
# return value is ignored. Must be fast and non-blocking: it runs on the
|
|
34
|
+
# pool's single reap thread, so a slow callback stalls draining the other
|
|
35
|
+
# in-flight children's pipes (the #4 deadlock discipline).
|
|
31
36
|
# @yieldparam item [Array] one work item.
|
|
32
37
|
# @return [Array<Mutineer::Result>] results in input order (nil for any item
|
|
33
38
|
# left unscheduled by an early stop).
|
|
34
|
-
def run(items, stop_when: nil)
|
|
39
|
+
def run(items, stop_when: nil, on_result: nil)
|
|
35
40
|
results = Array.new(items.size)
|
|
36
41
|
queue = (0...items.size).to_a
|
|
37
42
|
running = {} # pid => [index, read_io, buffer]
|
|
@@ -40,6 +45,7 @@ module Mutineer
|
|
|
40
45
|
until queue.empty? && running.empty?
|
|
41
46
|
fill(items, queue, running) { |*args| yield(*args) } unless stopping
|
|
42
47
|
result = reap(results, running)
|
|
48
|
+
on_result&.call(result) if result
|
|
43
49
|
if !stopping && stop_when && result && stop_when.call(result)
|
|
44
50
|
stopping = true
|
|
45
51
|
queue.clear # schedule no more; let in-flight workers drain
|
data/lib/mutineer.rb
CHANGED
|
@@ -27,6 +27,7 @@ require_relative "mutineer/mutators/regex_literal"
|
|
|
27
27
|
require_relative "mutineer/mutators/collection_method"
|
|
28
28
|
require_relative "mutineer/mutator_registry"
|
|
29
29
|
require_relative "mutineer/worker_pool"
|
|
30
|
+
require_relative "mutineer/progress"
|
|
30
31
|
require_relative "mutineer/runner"
|
|
31
32
|
require_relative "mutineer/reporter"
|
|
32
33
|
require_relative "mutineer/baseline"
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: mutineer
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.
|
|
4
|
+
version: 1.0.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- David Teren
|
|
@@ -95,6 +95,7 @@ files:
|
|
|
95
95
|
- lib/mutineer/mutators/string_literal.rb
|
|
96
96
|
- lib/mutineer/pairing.rb
|
|
97
97
|
- lib/mutineer/parser.rb
|
|
98
|
+
- lib/mutineer/progress.rb
|
|
98
99
|
- lib/mutineer/project.rb
|
|
99
100
|
- lib/mutineer/rails_worker_db.rb
|
|
100
101
|
- lib/mutineer/reporter.rb
|