active_mutator 0.6.0 → 0.7.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +45 -13
- data/lib/active_mutator/abort_flag.rb +53 -0
- data/lib/active_mutator/baseline.rb +132 -58
- data/lib/active_mutator/baseline_hooks.rb +24 -2
- data/lib/active_mutator/cli.rb +18 -3
- data/lib/active_mutator/config.rb +8 -1
- data/lib/active_mutator/config_file.rb +17 -1
- data/lib/active_mutator/coverage_map.rb +4 -1
- data/lib/active_mutator/diagnostics/ndjson.rb +25 -0
- data/lib/active_mutator/diagnostics/text.rb +110 -0
- data/lib/active_mutator/events.rb +55 -0
- data/lib/active_mutator/memory_ceiling.rb +59 -0
- data/lib/active_mutator/memory_probe.rb +82 -0
- data/lib/active_mutator/reporter/github.rb +13 -2
- data/lib/active_mutator/reporter/json.rb +19 -5
- data/lib/active_mutator/reporter/stryker_json.rb +7 -1
- data/lib/active_mutator/reporter/terminal.rb +31 -2
- data/lib/active_mutator/result.rb +7 -1
- data/lib/active_mutator/runner.rb +209 -65
- data/lib/active_mutator/sampler.rb +69 -0
- data/lib/active_mutator/scheduler.rb +105 -61
- data/lib/active_mutator/since_filter.rb +68 -7
- data/lib/active_mutator/version.rb +1 -1
- data/lib/active_mutator.rb +7 -0
- metadata +8 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: cf0b7bed1f6b35d6ab688affd01cb2ccc362944019ae5c18bff2e1205cccb53e
|
|
4
|
+
data.tar.gz: 9b4d7cac2fc3f8e3a6d737db4aa7f3b8f3f46c00baa275669fecbc5752e50910
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: f2ed986856e22c4881936ae20f66bac4ecdc57b31239b5786942be6c2128516db64542be9185da5f9e8bc06ad1acd893d9e231602e1cf553ad7c74d60af9ef61
|
|
7
|
+
data.tar.gz: 8f8bd75ed8572b8e1eee883644b412bc9e718eeb0b85f35e22d8f27b6f937113c505bccadca869fd33546895bbf68126a30b9931fc755c3df8d71e33b8168bb0
|
data/README.md
CHANGED
|
@@ -127,19 +127,33 @@ Each character on the progress line is one mutant, printed as it finishes:
|
|
|
127
127
|
| `A` | `accepted` | matches a known-equivalent entry in the acceptance ledger. Excluded from the score |
|
|
128
128
|
|
|
129
129
|
`invalid` mutants (edits that don't even re-parse as valid Ruby) are
|
|
130
|
-
discarded before scheduling and reported as a count only.
|
|
131
|
-
|
|
132
|
-
|
|
133
|
-
|
|
134
|
-
|
|
135
|
-
|
|
130
|
+
discarded before scheduling and reported as a count only.
|
|
131
|
+
|
|
132
|
+
| Exit | When |
|
|
133
|
+
|---|---|
|
|
134
|
+
| `0` | no unaccepted survivors or errors (only `uncovered` or `accepted` results count as clean), or the score meets `--fail-at` |
|
|
135
|
+
| `1` | unaccepted survivors or errors (with `--fail-at`, only when the score is below it), or an empty plan (below) |
|
|
136
|
+
| `2` | a usage or setup problem: a bad flag or config file, or a red baseline suite |
|
|
137
|
+
| `3` | the run reached `--max-rss` and stopped |
|
|
138
|
+
| `130` | the run was stopped by SIGINT (Ctrl-C) |
|
|
139
|
+
| `143` | the run was stopped by SIGTERM (130 before 0.7.0) |
|
|
140
|
+
|
|
141
|
+
A stopped run (`3`, `130`, `143`) kills its child processes, prints the
|
|
142
|
+
summary for the mutants that finished with a `Partial mutation score:`
|
|
143
|
+
line in place of `Mutation score:`, and never passes, whatever `--fail-at`
|
|
144
|
+
says. See [Run diagnostics](docs/guides/diagnostics.md#aborted-runs).
|
|
145
|
+
|
|
146
|
+
The JSON report's `exit_reason` field (`unaccepted_survivors`,
|
|
147
|
+
`worker_errors`, `clean`, `empty_plan`, and for a stopped run `interrupted`
|
|
148
|
+
or `memory_ceiling`) is independent of the `--fail-at` gate. `complete` is
|
|
149
|
+
`false` only for a stopped run. A `--since` or `--subject` run that
|
|
136
150
|
plans zero mutants skips the baseline, prints the usual count block with
|
|
137
151
|
every status at `0` and no `Mutation score:` line (JSON: `score` is `null`,
|
|
138
152
|
`exit_reason` is `empty_plan`), then warns with the cause and exits `1`
|
|
139
153
|
unless `--allow-empty` is given. The count block is printed on every run,
|
|
140
154
|
whatever the exit code, so log-scraping never has to handle a missing block.
|
|
141
|
-
`--allow-empty` forgives the empty plan only when the `--since` diff
|
|
142
|
-
no candidate source file; see [Empty plans in CI](#empty-plans-in-ci).
|
|
155
|
+
`--allow-empty` forgives the empty plan only when the `--since` diff changed
|
|
156
|
+
no code in a candidate source file; see [Empty plans in CI](#empty-plans-in-ci).
|
|
143
157
|
|
|
144
158
|
When survivors exist, the summary also prints a per-operator table showing
|
|
145
159
|
how often each operator's mutants survive, to help spot likely-equivalent
|
|
@@ -263,9 +277,16 @@ itself whether that emptiness is fine:
|
|
|
263
277
|
- **Exit 0** with `--no-class-level` when every changed line in the
|
|
264
278
|
candidate files falls inside class-body code that flag dropped. The
|
|
265
279
|
warning names `--no-class-level` as the reason.
|
|
266
|
-
- **Exit
|
|
267
|
-
|
|
268
|
-
counts as a
|
|
280
|
+
- **Exit 0** when the only changes in the candidate files are comments or
|
|
281
|
+
blank lines, added or deleted. A magic comment such as
|
|
282
|
+
`# frozen_string_literal: true` counts as code, and so does a file that
|
|
283
|
+
fails to parse.
|
|
284
|
+
- **Exit 0** when a file only lost code and no method spans the spot, e.g.
|
|
285
|
+
a whole method was removed. A deletion inside a method makes `--since`
|
|
286
|
+
mutate that method, like any other edit to it, so it never reaches this
|
|
287
|
+
rule unless the method has nothing left to mutate.
|
|
288
|
+
- **Exit 1** otherwise. A candidate file's code changed but produced no
|
|
289
|
+
mutants, and the warning lists the file(s).
|
|
269
290
|
- With `--subject` and no `--since` there is no diff to judge, so
|
|
270
291
|
`--allow-empty` exits `0` unconditionally.
|
|
271
292
|
|
|
@@ -284,7 +305,7 @@ projects used to tell a docs-only PR from a broken `--since` range.
|
|
|
284
305
|
| `--exclude PAT` | none | skip files matching glob during subject discovery (repeatable, gitignore-like) |
|
|
285
306
|
| `--max-mutants N` | none | deterministic sample of the first N mutants (quick smoke run on huge scopes; accepted/uncovered mutants count against N) |
|
|
286
307
|
| `--debug-plan` | off | print planned mutants as JSON and exit without running |
|
|
287
|
-
| `--allow-empty` | off | forgive an empty `--since`/`--subject` plan, but only when the `--since` diff
|
|
308
|
+
| `--allow-empty` | off | forgive an empty `--since`/`--subject` plan, but only when the `--since` diff changed no code in a candidate source file (default: warn and exit 1) |
|
|
288
309
|
| `--format terminal\|json\|stryker-json\|github` | terminal | report format |
|
|
289
310
|
| `--accept-survivors` | off | record survivors to the acceptance ledger |
|
|
290
311
|
| `--force-baseline` | off | ignore cached coverage map |
|
|
@@ -297,6 +318,10 @@ projects used to tell a docs-only PR from a broken `--since` range.
|
|
|
297
318
|
| `--require FILE` | none | preload files (repeatable) |
|
|
298
319
|
| `--operator FILE` | none | load a custom operator file before analysis (repeatable) |
|
|
299
320
|
| `--[no-]class-level` | on | mutate class-level code (macros, constants, DSL/scope lambdas) via class-body subjects |
|
|
321
|
+
| `--diagnostics` | off | print timestamped phase, mutant, and memory lines to stderr, for finding out where a CI run died (see [Run diagnostics](docs/guides/diagnostics.md)) |
|
|
322
|
+
| `--events FILE` | none | write the same events as NDJSON, one object per line, flushed as they happen |
|
|
323
|
+
| `--sample-interval S` | 5 | seconds between memory samples with `--diagnostics`, `--events`, or `--max-rss` |
|
|
324
|
+
| `--max-rss SIZE` | none | stop the run and exit 3 when the gem's total memory reaches SIZE (`6G`, `6144M`, or plain MB); warns once at 90% |
|
|
300
325
|
| `--fail-at SCORE` | none (strict) | exit 0 if score >= SCORE even with survivors (opt-in relaxation for gradual adoption; 0 = report-only) |
|
|
301
326
|
|
|
302
327
|
`--spec-path` tells active_mutator where spec files live (coverage
|
|
@@ -334,7 +359,10 @@ replaces the default `spec`),
|
|
|
334
359
|
`adaptive_timeout` (`true`/`false`),
|
|
335
360
|
`class_level` (`true`/`false`, default `true` — mutate class-level code),
|
|
336
361
|
`class_level_closure_cap` (integer, default `10` — max constants a
|
|
337
|
-
class-body mutant may reload before it is `skipped`)
|
|
362
|
+
class-body mutant may reload before it is `skipped`),
|
|
363
|
+
`diagnostics` (`true`/`false`, default `false`),
|
|
364
|
+
`events_file` (a path), `sample_interval` (seconds, default `5`),
|
|
365
|
+
`max_rss` (a size like `6G`, `6144M`, or plain MB).
|
|
338
366
|
Unknown keys and wrong types are errors, not silent no-ops.
|
|
339
367
|
|
|
340
368
|
```yaml
|
|
@@ -421,6 +449,10 @@ remaining limits are:
|
|
|
421
449
|
- [Operator reference](docs/guides/operators.md): every mutation
|
|
422
450
|
active_mutator can generate, with before/after examples and what a
|
|
423
451
|
survivor of each one means.
|
|
452
|
+
- [Run diagnostics](docs/guides/diagnostics.md): `--diagnostics`,
|
|
453
|
+
`--events`, and `--max-rss`. Find the phase and mutants a dying CI run
|
|
454
|
+
was in, stop a run before it runs out of memory, and the `--events`
|
|
455
|
+
NDJSON schema.
|
|
424
456
|
- [Custom operators](docs/guides/custom-operators.md): write and load your
|
|
425
457
|
own mutation operators with `--operator` / the `operators:` config key.
|
|
426
458
|
- [Mutation-check skill](docs/skills/mutation-check.md): the agent-facing
|
|
@@ -0,0 +1,53 @@
|
|
|
1
|
+
module ActiveMutator
|
|
2
|
+
# A run stopped early by SIGINT, SIGTERM, or the memory ceiling. Carries
|
|
3
|
+
# what finished (`results`) and what was still running (`in_flight`).
|
|
4
|
+
class Aborted < StandardError
|
|
5
|
+
attr_reader :reason, :results, :in_flight
|
|
6
|
+
|
|
7
|
+
def initialize(reason, results: [], in_flight: [])
|
|
8
|
+
super("run aborted: #{reason}")
|
|
9
|
+
@reason = reason
|
|
10
|
+
@results = results
|
|
11
|
+
@in_flight = in_flight
|
|
12
|
+
end
|
|
13
|
+
|
|
14
|
+
# The same abort, with the finished results a caller up the stack knows.
|
|
15
|
+
def with_results(results) = self.class.new(reason, results: results, in_flight: in_flight)
|
|
16
|
+
end
|
|
17
|
+
|
|
18
|
+
# Where a signal or a memory breach lands. `trip!` is safe in a trap
|
|
19
|
+
# handler: no I/O and no locks (a Mutex in a trap raises ThreadError).
|
|
20
|
+
#
|
|
21
|
+
# Outside a `deferred` block, a trip raises Aborted on the main thread at
|
|
22
|
+
# once: boot and planning own no child processes, so there is nothing to
|
|
23
|
+
# clean up. Inside one (the baseline child's wait, the fork pool) it only
|
|
24
|
+
# records the reason; the owner's poll loop kills its children and then
|
|
25
|
+
# raises with what it knows. The first reason wins, and later trips are
|
|
26
|
+
# ignored while the abort is under way.
|
|
27
|
+
class AbortFlag
|
|
28
|
+
attr_reader :reason
|
|
29
|
+
|
|
30
|
+
def trip!(reason)
|
|
31
|
+
return if @reason
|
|
32
|
+
|
|
33
|
+
@reason = reason
|
|
34
|
+
Thread.main.raise(Aborted.new(reason)) unless @deferred
|
|
35
|
+
end
|
|
36
|
+
|
|
37
|
+
# Not an endless def: its one line runs at load, so coverage could
|
|
38
|
+
# never tie it to the specs that call it.
|
|
39
|
+
def tripped?
|
|
40
|
+
!@reason.nil?
|
|
41
|
+
end
|
|
42
|
+
|
|
43
|
+
def deferred
|
|
44
|
+
outer = @deferred
|
|
45
|
+
@deferred = true
|
|
46
|
+
raise Aborted, @reason if @reason
|
|
47
|
+
|
|
48
|
+
yield
|
|
49
|
+
ensure
|
|
50
|
+
@deferred = outer
|
|
51
|
+
end
|
|
52
|
+
end
|
|
53
|
+
end
|
|
@@ -7,48 +7,69 @@ module ActiveMutator
|
|
|
7
7
|
# caches the CoverageMap. Invalidation is coarse: any digest change in
|
|
8
8
|
# {app,lib}/**/*.rb or the configured spec paths triggers a full re-run.
|
|
9
9
|
class Baseline
|
|
10
|
-
def initialize(root:, spec_paths: ["spec"], cache_dir: File.join(root, ".active_mutator")
|
|
10
|
+
def initialize(root:, spec_paths: ["spec"], cache_dir: File.join(root, ".active_mutator"), events: Events.new,
|
|
11
|
+
abort: AbortFlag.new)
|
|
12
|
+
@abort = abort
|
|
11
13
|
@root = root
|
|
12
14
|
@spec_paths = spec_paths
|
|
13
15
|
@cache_dir = cache_dir
|
|
14
16
|
@out_path = File.join(cache_dir, "coverage.json")
|
|
17
|
+
@events = events
|
|
15
18
|
end
|
|
16
19
|
|
|
17
|
-
|
|
20
|
+
POLL_SECONDS = 0.05
|
|
21
|
+
|
|
22
|
+
# child_pid: the running baseline child, nil between runs. Read by the
|
|
23
|
+
# memory sampler from its own thread.
|
|
24
|
+
attr_reader :last_refresh, :child_pid
|
|
18
25
|
|
|
19
26
|
def coverage_map(force: false)
|
|
20
27
|
digests = current_digests
|
|
21
|
-
|
|
22
|
-
map =
|
|
23
|
-
|
|
24
|
-
# under a removed spec path just vanish from the digest scan, so
|
|
25
|
-
# BaselineDelta treats their stale example records as untouched
|
|
26
|
-
# source coverage instead of dropping them. Force a full rebuild
|
|
27
|
-
# whenever the configured spec_paths differ from what the cache was
|
|
28
|
-
# stamped with.
|
|
29
|
-
if stored_spec_paths(map) == @spec_paths && map.fresh?(digests)
|
|
30
|
-
@last_refresh = :cached
|
|
31
|
-
return map
|
|
32
|
-
end
|
|
33
|
-
if stored_spec_paths(map) == @spec_paths && map.version == 2
|
|
34
|
-
delta = BaselineDelta.compute(old_digests: stored_digests(map), new_digests: digests,
|
|
35
|
-
coverage_map: map, root: @root, spec_paths: @spec_paths)
|
|
36
|
-
unless delta.full?
|
|
37
|
-
run_partial!(delta)
|
|
38
|
-
stamp_digests(digests)
|
|
39
|
-
@last_refresh = :partial
|
|
40
|
-
return CoverageMap.load(@out_path)
|
|
41
|
-
end
|
|
42
|
-
end
|
|
28
|
+
unless force
|
|
29
|
+
map = reuse_cache(digests)
|
|
30
|
+
return map if map
|
|
43
31
|
end
|
|
44
|
-
run_baseline!
|
|
45
|
-
stamp_digests(digests)
|
|
32
|
+
data = run_baseline!
|
|
46
33
|
@last_refresh = :full
|
|
47
|
-
|
|
34
|
+
stamp(data, digests)
|
|
48
35
|
end
|
|
49
36
|
|
|
50
37
|
private
|
|
51
38
|
|
|
39
|
+
# The cached map, refreshed in place when a delta allows it; nil when only
|
|
40
|
+
# a full rebuild will do. The old map (records plus its inverted index)
|
|
41
|
+
# lives only in this method, so it is garbage before the full rebuild's
|
|
42
|
+
# child starts.
|
|
43
|
+
def reuse_cache(digests)
|
|
44
|
+
return unless File.exist?(@out_path)
|
|
45
|
+
|
|
46
|
+
data = load_payload
|
|
47
|
+
map = CoverageMap.new(data)
|
|
48
|
+
# A spec_paths change silently degrades the delta classifier: files
|
|
49
|
+
# under a removed spec path just vanish from the digest scan, so
|
|
50
|
+
# BaselineDelta treats their stale example records as untouched
|
|
51
|
+
# source coverage instead of dropping them. Force a full rebuild
|
|
52
|
+
# whenever the configured spec_paths differ from what the cache was
|
|
53
|
+
# stamped with.
|
|
54
|
+
return unless map.spec_paths == @spec_paths
|
|
55
|
+
|
|
56
|
+
if map.fresh?(digests)
|
|
57
|
+
@last_refresh = :cached
|
|
58
|
+
return map
|
|
59
|
+
end
|
|
60
|
+
return unless map.version == 2
|
|
61
|
+
|
|
62
|
+
delta = BaselineDelta.compute(old_digests: map.digests, new_digests: digests,
|
|
63
|
+
coverage_map: map, root: @root, spec_paths: @spec_paths)
|
|
64
|
+
return if delta.full?
|
|
65
|
+
|
|
66
|
+
# The map shares data["records"], and the merge edits it in place: the
|
|
67
|
+
# old map must not be read past this point.
|
|
68
|
+
run_partial!(delta, data)
|
|
69
|
+
@last_refresh = :partial
|
|
70
|
+
stamp(data, digests)
|
|
71
|
+
end
|
|
72
|
+
|
|
52
73
|
# The cache is disposable and must never be committed. Host projects
|
|
53
74
|
# rarely gitignore it themselves, so the directory ignores its own
|
|
54
75
|
# contents (the node_modules trick).
|
|
@@ -60,22 +81,31 @@ module ActiveMutator
|
|
|
60
81
|
|
|
61
82
|
def run_baseline!
|
|
62
83
|
prepare_cache_dir
|
|
63
|
-
|
|
64
|
-
# out: :err: the subprocess suite's progress output must not pollute
|
|
65
|
-
# our stdout (breaks `--format json` consumers).
|
|
66
|
-
ok = system(env, "bundle", "exec", "rspec", chdir: @root, out: :err)
|
|
84
|
+
ok = run_rspec(@out_path)
|
|
67
85
|
raise BaselineFailed, "baseline suite failed, fix the suite before mutating" unless ok
|
|
68
86
|
raise BaselineFailed, "baseline produced no coverage output" unless File.exist?(@out_path)
|
|
69
87
|
|
|
70
|
-
|
|
88
|
+
data = load_payload
|
|
89
|
+
verify_complete!(data)
|
|
90
|
+
data
|
|
91
|
+
end
|
|
92
|
+
|
|
93
|
+
# Its own phase, apart from the child's run: 0.6.0 died here, in the
|
|
94
|
+
# parent reading a huge file back. The size goes out BEFORE the parse, so
|
|
95
|
+
# the log names the cause even if nothing runs after it, and --max-rss
|
|
96
|
+
# can stop the run before the parse.
|
|
97
|
+
def load_payload(path = @out_path)
|
|
98
|
+
@events.emit(:phase_start, phase: :coverage_load, bytes: File.size(path))
|
|
99
|
+
data = JSON.parse(File.read(path))
|
|
100
|
+
@events.emit(:phase_end, phase: :coverage_load, examples: data.fetch("records", {}).size)
|
|
101
|
+
data
|
|
71
102
|
end
|
|
72
103
|
|
|
73
104
|
# An aborted subprocess can still exit 0 with a partial map (RSpec
|
|
74
105
|
# rescues Errno::EPIPE and runs after(:suite)); stamping that as fresh
|
|
75
106
|
# silently reports every mutant uncovered. Payloads without the count
|
|
76
107
|
# predate this check and are accepted as-is.
|
|
77
|
-
def verify_complete!(
|
|
78
|
-
payload = JSON.parse(File.read(out_path))
|
|
108
|
+
def verify_complete!(payload)
|
|
79
109
|
expected = payload["expected_examples"]
|
|
80
110
|
return unless expected
|
|
81
111
|
|
|
@@ -87,6 +117,60 @@ module ActiveMutator
|
|
|
87
117
|
"re-run without interrupting the suite"
|
|
88
118
|
end
|
|
89
119
|
|
|
120
|
+
# Spawned and polled, not `system`, so the parent knows the child's pid
|
|
121
|
+
# and keeps control while it runs. out: :err: the subprocess suite's
|
|
122
|
+
# progress output must not pollute our stdout (breaks `--format json`
|
|
123
|
+
# consumers).
|
|
124
|
+
#
|
|
125
|
+
# The phase starts once the child exists, so it carries the pid the
|
|
126
|
+
# memory sampler follows. The child gets its own process group so an
|
|
127
|
+
# abort can kill the whole suite (browsers, app servers) in one signal;
|
|
128
|
+
# a Ctrl-C reaches it through the Runner's trap instead of the terminal.
|
|
129
|
+
# So does nothing else: a hangup from a closed terminal never reaches it,
|
|
130
|
+
# so anything that ends the wait early (an abort, SIGHUP, SIGQUIT) kills
|
|
131
|
+
# the group on the way out.
|
|
132
|
+
def run_rspec(out_path, targets = [])
|
|
133
|
+
status = nil
|
|
134
|
+
@abort.deferred do
|
|
135
|
+
@child_pid = Process.spawn(baseline_env(out_path), *rspec_command(targets), chdir: @root, out: :err,
|
|
136
|
+
pgroup: true)
|
|
137
|
+
status = @events.phase(:baseline, refresh: targets.empty? ? :full : :partial, pid: @child_pid) do
|
|
138
|
+
wait_child(@child_pid)
|
|
139
|
+
end
|
|
140
|
+
# The memory sample taken as the phase ends can trip the ceiling.
|
|
141
|
+
# Stop here, before the coverage parse: that's the spike it guards.
|
|
142
|
+
raise Aborted, @abort.reason if @abort.tripped?
|
|
143
|
+
|
|
144
|
+
status.success?
|
|
145
|
+
end
|
|
146
|
+
rescue SystemCallError # `bundle` missing: `system` returned nil here
|
|
147
|
+
false
|
|
148
|
+
ensure
|
|
149
|
+
kill_group(@child_pid) if @child_pid && status.nil?
|
|
150
|
+
@child_pid = nil
|
|
151
|
+
end
|
|
152
|
+
|
|
153
|
+
def rspec_command(targets) = ["bundle", "exec", "rspec", *targets]
|
|
154
|
+
|
|
155
|
+
# The flag is checked after the wait, so a trip that lands as the child
|
|
156
|
+
# exits still stops the run here, not after the coverage parse.
|
|
157
|
+
# run_rspec kills the child's group on the way out.
|
|
158
|
+
def wait_child(pid)
|
|
159
|
+
loop do
|
|
160
|
+
_, status = Process.waitpid2(pid, Process::WNOHANG)
|
|
161
|
+
raise Aborted, @abort.reason if @abort.tripped?
|
|
162
|
+
return status if status
|
|
163
|
+
|
|
164
|
+
sleep POLL_SECONDS
|
|
165
|
+
end
|
|
166
|
+
end
|
|
167
|
+
|
|
168
|
+
def kill_group(pid)
|
|
169
|
+
Process.kill("KILL", -pid)
|
|
170
|
+
rescue Errno::ESRCH, Errno::EPERM
|
|
171
|
+
nil # already gone
|
|
172
|
+
end
|
|
173
|
+
|
|
90
174
|
def baseline_env(out_path)
|
|
91
175
|
{
|
|
92
176
|
"ACTIVE_MUTATOR" => "1",
|
|
@@ -106,36 +190,25 @@ module ActiveMutator
|
|
|
106
190
|
}
|
|
107
191
|
end
|
|
108
192
|
|
|
109
|
-
def
|
|
110
|
-
JSON.parse(File.read(@out_path)).fetch("digests", {})
|
|
111
|
-
end
|
|
112
|
-
|
|
113
|
-
# A pre-0.4.0 cache predates spec_paths and has no key; treat that as the
|
|
114
|
-
# old implicit default so existing default-config caches stay valid.
|
|
115
|
-
def stored_spec_paths(map)
|
|
116
|
-
JSON.parse(File.read(@out_path)).fetch("spec_paths", ["spec"])
|
|
117
|
-
end
|
|
118
|
-
|
|
119
|
-
def run_partial!(delta)
|
|
120
|
-
targets = delta.rerun_spec_files + delta.rerun_example_ids
|
|
193
|
+
def run_partial!(delta, cache)
|
|
121
194
|
partial_out = File.join(@cache_dir, "partial.json")
|
|
195
|
+
targets = delta.rerun_spec_files + delta.rerun_example_ids
|
|
196
|
+
part = {}
|
|
122
197
|
if targets.any?
|
|
123
|
-
|
|
124
|
-
ok = system(env, "bundle", "exec", "rspec", *targets, chdir: @root, out: :err)
|
|
198
|
+
ok = run_rspec(partial_out, targets)
|
|
125
199
|
raise BaselineFailed, "partial baseline run failed, fix the suite before mutating" unless ok
|
|
126
200
|
raise BaselineFailed, "partial baseline produced no output" unless File.exist?(partial_out)
|
|
127
201
|
|
|
128
|
-
|
|
202
|
+
part = load_payload(partial_out)
|
|
203
|
+
verify_complete!(part)
|
|
129
204
|
end
|
|
130
|
-
merge_partial!(
|
|
205
|
+
merge_partial!(cache, part, delta)
|
|
131
206
|
ensure
|
|
132
|
-
FileUtils.rm_f(partial_out)
|
|
207
|
+
FileUtils.rm_f(partial_out)
|
|
133
208
|
end
|
|
134
209
|
|
|
135
|
-
|
|
136
|
-
|
|
137
|
-
part = File.exist?(partial_out) ? JSON.parse(File.read(partial_out)) : { "records" => {}, "times" => {} }
|
|
138
|
-
|
|
210
|
+
# Edits `cache` in place; the caller stamps and writes it.
|
|
211
|
+
def merge_partial!(cache, part, delta)
|
|
139
212
|
rerun_prefixes = delta.rerun_spec_files.map { |rel| "#{rel}[" }
|
|
140
213
|
obsolete = lambda do |example_id|
|
|
141
214
|
bare = example_id.sub(%r{\A\./}, "")
|
|
@@ -151,14 +224,15 @@ module ActiveMutator
|
|
|
151
224
|
end
|
|
152
225
|
cache["records"].merge!(part.fetch("records", {}))
|
|
153
226
|
cache["times"].merge!(part.fetch("times", {}))
|
|
154
|
-
AtomicFile.write(@out_path, JSON.generate(cache))
|
|
155
227
|
end
|
|
156
228
|
|
|
157
|
-
|
|
158
|
-
|
|
229
|
+
# Writes the stamped payload once and builds the map from the hash in
|
|
230
|
+
# hand, so the file is never parsed back.
|
|
231
|
+
def stamp(data, digests)
|
|
159
232
|
data["digests"] = digests
|
|
160
233
|
data["spec_paths"] = @spec_paths
|
|
161
234
|
AtomicFile.write(@out_path, JSON.generate(data))
|
|
235
|
+
CoverageMap.new(data)
|
|
162
236
|
end
|
|
163
237
|
|
|
164
238
|
def current_digests
|
|
@@ -9,11 +9,16 @@ module ActiveMutator
|
|
|
9
9
|
RECORDS = {}
|
|
10
10
|
TIMES = {}
|
|
11
11
|
|
|
12
|
-
def self.diff_coverage(before, after, root, spec_paths: ["spec"])
|
|
12
|
+
def self.diff_coverage(before, after, root, spec_paths: ["spec"], gem_dirs: [])
|
|
13
13
|
prefixes = spec_paths.map { |sp| "/#{sp}/" }
|
|
14
|
+
gem_prefixes = gem_dirs.map { |dir| File.join(dir, "") }
|
|
14
15
|
hits = []
|
|
15
16
|
after.each do |path, data|
|
|
16
17
|
next unless path.start_with?(root)
|
|
18
|
+
# Gems installed under the root (e.g. `bundle config path
|
|
19
|
+
# vendor/bundle` on CI) would otherwise be recorded per example and
|
|
20
|
+
# blow up the map's memory.
|
|
21
|
+
next if gem_prefixes.any? { |p| path.start_with?(p) }
|
|
17
22
|
|
|
18
23
|
# Relative to root, not a global substring check: `root` itself may
|
|
19
24
|
# contain "/spec/" (e.g. a fixture nested under this gem's own
|
|
@@ -33,6 +38,20 @@ module ActiveMutator
|
|
|
33
38
|
hits
|
|
34
39
|
end
|
|
35
40
|
|
|
41
|
+
# Computed per example, not at load: this file loads via RUBYOPT before
|
|
42
|
+
# Bundler sets up its install path.
|
|
43
|
+
def self.gem_dirs
|
|
44
|
+
dirs = Gem.path.dup
|
|
45
|
+
if defined?(Bundler)
|
|
46
|
+
begin
|
|
47
|
+
dirs << Bundler.bundle_path.to_s
|
|
48
|
+
rescue Bundler::BundlerError
|
|
49
|
+
# No Gemfile, so Gem.path covers it.
|
|
50
|
+
end
|
|
51
|
+
end
|
|
52
|
+
dirs.uniq
|
|
53
|
+
end
|
|
54
|
+
|
|
36
55
|
def self.build_payload(records, times, expected_examples: nil)
|
|
37
56
|
payload = { "version" => 2, "records" => records, "times" => times }
|
|
38
57
|
payload["expected_examples"] = expected_examples if expected_examples
|
|
@@ -56,7 +75,10 @@ if ENV["ACTIVE_MUTATOR_BASELINE_OUT"]
|
|
|
56
75
|
root = ENV.fetch("ACTIVE_MUTATOR_ROOT")
|
|
57
76
|
spec_paths = ENV.fetch("ACTIVE_MUTATOR_SPEC_PATHS", "spec").split(":")
|
|
58
77
|
ActiveMutator::BaselineHooks::RECORDS[example.id] =
|
|
59
|
-
ActiveMutator::BaselineHooks.diff_coverage(
|
|
78
|
+
ActiveMutator::BaselineHooks.diff_coverage(
|
|
79
|
+
before, after, root,
|
|
80
|
+
spec_paths: spec_paths, gem_dirs: ActiveMutator::BaselineHooks.gem_dirs
|
|
81
|
+
)
|
|
60
82
|
# NOT example.execution_result.run_time: that is nil until after
|
|
61
83
|
# around hooks complete.
|
|
62
84
|
ActiveMutator::BaselineHooks::TIMES[example.id] = elapsed
|
data/lib/active_mutator/cli.rb
CHANGED
|
@@ -32,7 +32,8 @@ module ActiveMutator
|
|
|
32
32
|
spec_paths: ["spec"],
|
|
33
33
|
browser_boot_seconds: 15.0, accept_survivors: false, exclude: [],
|
|
34
34
|
max_mutants: nil, debug_plan: false, fail_at: nil, adaptive_timeout: true,
|
|
35
|
-
operators: [], class_level: true, class_level_closure_cap: 10, allow_empty: false
|
|
35
|
+
operators: [], class_level: true, class_level_closure_cap: 10, allow_empty: false,
|
|
36
|
+
diagnostics: false, events_file: nil, sample_interval: 5.0, max_rss: nil
|
|
36
37
|
}
|
|
37
38
|
options.merge!(ConfigFile.load(Dir.pwd))
|
|
38
39
|
paths = OptionParser.new do |o|
|
|
@@ -67,8 +68,22 @@ module ActiveMutator
|
|
|
67
68
|
o.on("--max-mutants N", Integer, "Deterministically sample the first N mutants") { |v| options[:max_mutants] = v }
|
|
68
69
|
o.on("--debug-plan", "Print the planned mutant list as JSON and exit") { options[:debug_plan] = true }
|
|
69
70
|
o.on("--allow-empty",
|
|
70
|
-
"Exit 0 when --since/--subject plan no mutants and the --since diff
|
|
71
|
-
"source file (default: exit 1)") { options[:allow_empty] = true }
|
|
71
|
+
"Exit 0 when --since/--subject plan no mutants and the --since diff changed no code " \
|
|
72
|
+
"in a mutable source file (default: exit 1)") { options[:allow_empty] = true }
|
|
73
|
+
o.on("--diagnostics", "Print phase, mutant, and memory lines to stderr") { options[:diagnostics] = true }
|
|
74
|
+
o.on("--events FILE", "Write run events to FILE as NDJSON, one object per line") { |v| options[:events_file] = v }
|
|
75
|
+
o.on("--sample-interval S", Float,
|
|
76
|
+
"Seconds between memory samples with --diagnostics, --events, or --max-rss (default: 5)") do |v|
|
|
77
|
+
raise OptionParser::InvalidArgument, "--sample-interval must be > 0" unless v.positive?
|
|
78
|
+
options[:sample_interval] = v
|
|
79
|
+
end
|
|
80
|
+
o.on("--max-rss SIZE", "Stop the run (exit 3) when the gem's total memory reaches SIZE: 6G, 6144M, " \
|
|
81
|
+
"or MB; warns at 90%") do |v|
|
|
82
|
+
kb = MemoryCeiling.parse_kb(v)
|
|
83
|
+
raise OptionParser::InvalidArgument, "--max-rss takes a size like 6G, 6144M, or 6144 (MB)" unless kb
|
|
84
|
+
|
|
85
|
+
options[:max_rss] = kb
|
|
86
|
+
end
|
|
72
87
|
o.on("--fail-at SCORE", Float, "Exit 0 if mutation score >= SCORE even with survivors (default: any survivor fails)") do |v|
|
|
73
88
|
raise OptionParser::InvalidArgument, "--fail-at must be within 0..100" unless (0..100).cover?(v)
|
|
74
89
|
options[:fail_at] = v
|
|
@@ -7,5 +7,12 @@ module ActiveMutator
|
|
|
7
7
|
:browser_boot_seconds,
|
|
8
8
|
:accept_survivors, :exclude, :max_mutants, :debug_plan,
|
|
9
9
|
:fail_at, :adaptive_timeout, :operators,
|
|
10
|
-
:class_level, :class_level_closure_cap, :allow_empty
|
|
10
|
+
:class_level, :class_level_closure_cap, :allow_empty,
|
|
11
|
+
:diagnostics, :events_file, :sample_interval, :max_rss) do
|
|
12
|
+
# Defaults for the 0.7.0 diagnostics fields, so a Config built by hand
|
|
13
|
+
# (specs, embedding hosts) doesn't have to name them. `max_rss` is in kB.
|
|
14
|
+
def initialize(diagnostics: false, events_file: nil, sample_interval: 5.0, max_rss: nil, **fields)
|
|
15
|
+
super
|
|
16
|
+
end
|
|
17
|
+
end
|
|
11
18
|
end
|
|
@@ -26,7 +26,11 @@ module ActiveMutator
|
|
|
26
26
|
"adaptive_timeout" => :boolean,
|
|
27
27
|
"class_level" => :boolean,
|
|
28
28
|
"class_level_closure_cap" => :positive_integer,
|
|
29
|
-
"allow_empty" => :boolean
|
|
29
|
+
"allow_empty" => :boolean,
|
|
30
|
+
"diagnostics" => :boolean,
|
|
31
|
+
"events_file" => :string,
|
|
32
|
+
"sample_interval" => :positive_number,
|
|
33
|
+
"max_rss" => :size
|
|
30
34
|
}.freeze
|
|
31
35
|
|
|
32
36
|
def self.load(root)
|
|
@@ -63,6 +67,10 @@ module ActiveMutator
|
|
|
63
67
|
when :number
|
|
64
68
|
raise Error, "#{FILENAME}: #{key} must be a number" unless value.is_a?(Numeric)
|
|
65
69
|
value.to_f
|
|
70
|
+
when :positive_number
|
|
71
|
+
raise Error, "#{FILENAME}: #{key} must be a number" unless value.is_a?(Numeric)
|
|
72
|
+
raise Error, "#{FILENAME}: #{key} must be > 0" unless value.positive?
|
|
73
|
+
value.to_f
|
|
66
74
|
when :score
|
|
67
75
|
raise Error, "#{FILENAME}: #{key} must be a number" unless value.is_a?(Numeric)
|
|
68
76
|
raise Error, "#{FILENAME}: #{key} must be within 0..100" unless (0..100).cover?(value)
|
|
@@ -72,6 +80,9 @@ module ActiveMutator
|
|
|
72
80
|
raise Error, "#{FILENAME}: format must be one of #{FORMATS.join(", ")}"
|
|
73
81
|
end
|
|
74
82
|
value.tr("-", "_").to_sym
|
|
83
|
+
when :string
|
|
84
|
+
raise Error, "#{FILENAME}: #{key} must be a string" unless value.is_a?(String)
|
|
85
|
+
value
|
|
75
86
|
when :string_list
|
|
76
87
|
unless value.is_a?(Array) && value.all?(String)
|
|
77
88
|
raise Error, "#{FILENAME}: #{key} must be a list of strings"
|
|
@@ -88,6 +99,11 @@ module ActiveMutator
|
|
|
88
99
|
raise Error, "#{FILENAME}: #{key} must be true or false"
|
|
89
100
|
end
|
|
90
101
|
value
|
|
102
|
+
when :size
|
|
103
|
+
kb = MemoryCeiling.parse_kb(value)
|
|
104
|
+
raise Error, "#{FILENAME}: #{key} must be a size like 6G, 6144M, or 6144 (MB)" unless kb
|
|
105
|
+
|
|
106
|
+
kb
|
|
91
107
|
when :preload_helper
|
|
92
108
|
return :none if value == false
|
|
93
109
|
raise Error, "#{FILENAME}: preload_helper must be a path or false" unless value.is_a?(String)
|
|
@@ -8,13 +8,16 @@ module ActiveMutator
|
|
|
8
8
|
class CoverageMap
|
|
9
9
|
def self.load(path) = new(JSON.parse(File.read(path)))
|
|
10
10
|
|
|
11
|
-
attr_reader :version, :records
|
|
11
|
+
attr_reader :version, :records, :digests, :spec_paths
|
|
12
12
|
|
|
13
13
|
def initialize(data)
|
|
14
14
|
@version = data["version"]
|
|
15
15
|
@records = data.fetch("records", {})
|
|
16
16
|
@times = data.fetch("times", {})
|
|
17
17
|
@digests = data.fetch("digests", {})
|
|
18
|
+
# A pre-0.4.0 cache predates spec_paths and has no key; treat that as the
|
|
19
|
+
# old implicit default so existing default-config caches stay valid.
|
|
20
|
+
@spec_paths = data.fetch("spec_paths", ["spec"])
|
|
18
21
|
@map = build_map
|
|
19
22
|
end
|
|
20
23
|
|
|
@@ -0,0 +1,25 @@
|
|
|
1
|
+
require "json"
|
|
2
|
+
|
|
3
|
+
module ActiveMutator
|
|
4
|
+
module Diagnostics
|
|
5
|
+
# --events FILE: every event as one JSON object per line. The public,
|
|
6
|
+
# versioned contract (docs/guides/diagnostics.md): new fields may appear
|
|
7
|
+
# under v1; a rename or removal bumps VERSION. The IO is sync, so each
|
|
8
|
+
# line reaches the OS as it's written and a SIGKILLed run still leaves
|
|
9
|
+
# everything up to the kill.
|
|
10
|
+
class Ndjson
|
|
11
|
+
VERSION = 1
|
|
12
|
+
|
|
13
|
+
def initialize(io)
|
|
14
|
+
@io = io
|
|
15
|
+
@io.sync = true
|
|
16
|
+
end
|
|
17
|
+
|
|
18
|
+
def call(event)
|
|
19
|
+
line = { "v" => VERSION, "event" => event.type.to_s,
|
|
20
|
+
"t" => event.at.utc.strftime("%Y-%m-%dT%H:%M:%S.%LZ"), "elapsed" => event.elapsed.round(3) }
|
|
21
|
+
@io.write("#{JSON.generate(line.merge(event.fields.transform_keys(&:to_s)))}\n")
|
|
22
|
+
end
|
|
23
|
+
end
|
|
24
|
+
end
|
|
25
|
+
end
|