evilution 0.33.0 → 0.35.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.beads/interactions.jsonl +16 -0
- data/.rubocop_todo.yml +2 -1
- data/CHANGELOG.md +26 -0
- data/README.md +39 -10
- data/assets/logo/evilution-mark-1024.png +0 -0
- data/assets/logo/evilution-mark-64.png +0 -0
- data/assets/logo/evilution-mark-transparent-1024.png +0 -0
- data/assets/logo/evilution-mark-transparent.svg +53 -0
- data/assets/logo/evilution-mark.svg +70 -0
- data/docs/isolation.md +32 -2
- data/docs/migration-from-mutant.md +226 -0
- data/lib/evilution/cli/parser/options_builder.rb +17 -0
- data/lib/evilution/config/builders/spec_resolver.rb +3 -1
- data/lib/evilution/config/validators/example_targeting_strategy.rb +22 -0
- data/lib/evilution/config.rb +16 -2
- data/lib/evilution/coverage/digest.rb +16 -0
- data/lib/evilution/coverage/map.rb +64 -0
- data/lib/evilution/coverage/map_builder.rb +82 -0
- data/lib/evilution/coverage/map_store.rb +87 -0
- data/lib/evilution/coverage/recorder.rb +85 -0
- data/lib/evilution/coverage.rb +8 -0
- data/lib/evilution/coverage_example_filter.rb +41 -0
- data/lib/evilution/gem_detector.rb +7 -0
- data/lib/evilution/integration/minitest.rb +14 -3
- data/lib/evilution/integration/rspec/unresolved_spec_warner.rb +17 -3
- data/lib/evilution/integration/test_unit/test_file_resolver.rb +14 -3
- data/lib/evilution/isolation/fork.rb +38 -76
- data/lib/evilution/parallel/work_queue/dispatcher/deadline_tracker.rb +63 -0
- data/lib/evilution/parallel/work_queue/dispatcher.rb +7 -34
- data/lib/evilution/parallel/work_queue/worker.rb +41 -51
- data/lib/evilution/process_supervisor.rb +259 -0
- data/lib/evilution/runner/baseline_runner.rb +52 -0
- data/lib/evilution/runner/canary.rb +31 -4
- data/lib/evilution/runner/isolation_resolver.rb +120 -12
- data/lib/evilution/runner.rb +3 -2
- data/lib/evilution/spec_resolver.rb +93 -1
- data/lib/evilution/spec_selector.rb +14 -4
- data/lib/evilution/version.rb +1 -1
- data/lib/evilution.rb +1 -0
- data/scripts/canary_manifest.yml +47 -0
- data/scripts/compare_targeting +277 -0
- data/scripts/compare_targeting.example.yml +24 -0
- metadata +21 -3
- data/lib/evilution/parallel/work_queue/worker_registry.rb +0 -47
|
@@ -0,0 +1,226 @@
|
|
|
1
|
+
# Migrating from `mutant` to Evilution
|
|
2
|
+
|
|
3
|
+
This guide is for teams moving off [`mutant`](https://github.com/mbj/mutant) — whose
|
|
4
|
+
runtime is under a commercial license — to Evilution, which is MIT-licensed with no
|
|
5
|
+
commercial restrictions. It maps the commands, config, and output you already know
|
|
6
|
+
onto their Evilution equivalents, and is honest about where the two tools differ.
|
|
7
|
+
|
|
8
|
+
Evilution is not a drop-in fork of mutant. It produces its own mutations with its own
|
|
9
|
+
operator set and selects work by **file path** rather than by subject expression. The
|
|
10
|
+
result is the same kind of signal — surviving mutations expose weak tests — but the
|
|
11
|
+
numbers and the workflow are not identical. The `compare` command (below) exists
|
|
12
|
+
precisely so you can line the two tools up and see the difference for yourself.
|
|
13
|
+
|
|
14
|
+
## TL;DR
|
|
15
|
+
|
|
16
|
+
| mutant | Evilution |
|
|
17
|
+
| --- | --- |
|
|
18
|
+
| `mutant run --use rspec -- 'MyApp::Foo*'` | `evilution run lib/my_app/ --target 'MyApp::Foo*'` |
|
|
19
|
+
| `mutant.yml` | `.evilution.yml` (run `evilution init` to scaffold) |
|
|
20
|
+
| `mutant-rspec` / `mutant-minitest` gems | built in: `--integration rspec\|minitest\|test-unit` |
|
|
21
|
+
| "mutation coverage" %, "alive" | "mutation score" 0.0–1.0, "survived" |
|
|
22
|
+
| commercial license for the runtime | MIT |
|
|
23
|
+
|
|
24
|
+
## 1. Install
|
|
25
|
+
|
|
26
|
+
Drop the mutant gems and add Evilution:
|
|
27
|
+
|
|
28
|
+
```ruby
|
|
29
|
+
# Gemfile — remove: gem "mutant", "mutant-rspec"
|
|
30
|
+
gem "evilution", group: :test
|
|
31
|
+
```
|
|
32
|
+
|
|
33
|
+
```sh
|
|
34
|
+
bundle install
|
|
35
|
+
bundle exec evilution init # writes a commented .evilution.yml
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
There is no separate integration gem to install — RSpec, Minitest, and Test::Unit
|
|
39
|
+
support ships in the core gem.
|
|
40
|
+
|
|
41
|
+
## 2. The core workflow shift: files, not subjects
|
|
42
|
+
|
|
43
|
+
mutant is **subject-driven**: you pass a match expression and it finds the code.
|
|
44
|
+
Evilution is **file-driven**: you pass file paths, and optionally narrow with
|
|
45
|
+
`--target`.
|
|
46
|
+
|
|
47
|
+
```sh
|
|
48
|
+
# mutant
|
|
49
|
+
bundle exec mutant run --use rspec -- 'MyApp::Calculator#add'
|
|
50
|
+
|
|
51
|
+
# evilution — point at the file, optionally filter to the method
|
|
52
|
+
bundle exec evilution run lib/my_app/calculator.rb --target 'MyApp::Calculator#add'
|
|
53
|
+
|
|
54
|
+
# whole directory, then filter by expression
|
|
55
|
+
bundle exec evilution run lib/ --target 'MyApp::Calculator*'
|
|
56
|
+
```
|
|
57
|
+
|
|
58
|
+
`--target` accepts the expression shapes you are used to, plus a source glob:
|
|
59
|
+
|
|
60
|
+
| Intent | mutant match | Evilution `--target` |
|
|
61
|
+
| --- | --- | --- |
|
|
62
|
+
| One method | `MyApp::Foo#bar` | `MyApp::Foo#bar` |
|
|
63
|
+
| One class | `MyApp::Foo` | `MyApp::Foo` |
|
|
64
|
+
| Namespace (recursive) | `MyApp::Foo*` | `MyApp::Foo*` |
|
|
65
|
+
| All instance methods | `MyApp::Foo#` | `MyApp::Foo#` |
|
|
66
|
+
| All singleton methods | `MyApp::Foo.` | `MyApp::Foo.` |
|
|
67
|
+
| Subclasses | _(n/a)_ | `descendants:MyApp::Foo` |
|
|
68
|
+
| By file glob | _(use file args)_ | `source:lib/**/*.rb` |
|
|
69
|
+
|
|
70
|
+
## 3. Command mapping
|
|
71
|
+
|
|
72
|
+
| Task | mutant | Evilution |
|
|
73
|
+
| --- | --- | --- |
|
|
74
|
+
| Run mutation testing | `mutant run -- 'Foo'` | `evilution run lib/foo.rb` (alias `mutate`; binary alias `evil`) |
|
|
75
|
+
| Choose framework | `--use rspec` / `mutant-minitest` | `--integration rspec\|minitest\|test-unit` |
|
|
76
|
+
| Parallel workers | `-j N` / `--jobs N` | `-j N` / `--jobs N` |
|
|
77
|
+
| Fail fast | `--fail-fast` | `--fail-fast [N]` (off by default; N defaults to 1 when the flag is given without a value) |
|
|
78
|
+
| Coverage gate | `--score` (default `1.0` / 100%) | `--min-score` (default `0.0`) |
|
|
79
|
+
| Preview mutations | `mutant util mutation` / `mutant environment subject list` | `evilution util mutation -e 'a + b'` / `evilution subjects lib/foo.rb` |
|
|
80
|
+
| Set load path / requires | `--include lib --require my_app` | handled by `--preload` (auto-detects `spec/spec_helper.rb` etc.) + Bundler |
|
|
81
|
+
| JSON report | session JSON (always written) | `--format json` (`--save-session` to persist) |
|
|
82
|
+
| Inline opt-out | `# mutant:disable` | `# evilution:disable` |
|
|
83
|
+
| Aggressive operators | _(always on)_ | `--profile strict` (or `--strict`) |
|
|
84
|
+
|
|
85
|
+
### Things mutant has that Evilution does not (yet)
|
|
86
|
+
|
|
87
|
+
- **`--since <git-ref>`** — mutant can restrict subjects to lines changed since a git
|
|
88
|
+
ref. Evilution has no diff-aware selection. The closest tools are `--incremental`
|
|
89
|
+
(caches `killed`/`timeout` results and skips unchanged mutations on re-runs — a
|
|
90
|
+
different model) and `--target source:<glob>` / explicit file args to scope a run.
|
|
91
|
+
|
|
92
|
+
### Things Evilution adds
|
|
93
|
+
|
|
94
|
+
- `--format html` interactive report, and `evilution compare` (see §6).
|
|
95
|
+
- Per-mutation **example targeting** (runs only the examples that exercise the mutated
|
|
96
|
+
code) — on by default, tune with `example_targeting*` keys.
|
|
97
|
+
- `--suggest-tests` emits concrete test code for survivors.
|
|
98
|
+
- Explicit `:unresolved` status when no spec maps to a source file (a coverage-gap
|
|
99
|
+
signal rather than a silent skip).
|
|
100
|
+
|
|
101
|
+
## 4. Config translation (`mutant.yml` → `.evilution.yml`)
|
|
102
|
+
|
|
103
|
+
```yaml
|
|
104
|
+
# mutant.yml
|
|
105
|
+
integration:
|
|
106
|
+
name: rspec
|
|
107
|
+
jobs: 8
|
|
108
|
+
includes:
|
|
109
|
+
- lib
|
|
110
|
+
requires:
|
|
111
|
+
- my_app
|
|
112
|
+
matcher:
|
|
113
|
+
subjects:
|
|
114
|
+
- MyApp::Billing*
|
|
115
|
+
ignore:
|
|
116
|
+
- MyApp::Billing::Legacy#deprecated
|
|
117
|
+
```
|
|
118
|
+
|
|
119
|
+
```yaml
|
|
120
|
+
# .evilution.yml
|
|
121
|
+
integration: rspec
|
|
122
|
+
jobs: 8
|
|
123
|
+
# `includes` / `requires` are usually unnecessary — Evilution preloads the test
|
|
124
|
+
# helper (and the gem entry) automatically. Override only if needed:
|
|
125
|
+
preload: spec/spec_helper.rb
|
|
126
|
+
# `matcher.subjects` -> run file args + --target; expression below is illustrative.
|
|
127
|
+
target: "MyApp::Billing*"
|
|
128
|
+
# `matcher.ignore` -> AST ignore patterns (see docs/ast_pattern_syntax.md) or an
|
|
129
|
+
# inline `# evilution:disable` comment on the method.
|
|
130
|
+
ignore_patterns: []
|
|
131
|
+
```
|
|
132
|
+
|
|
133
|
+
Key-by-key:
|
|
134
|
+
|
|
135
|
+
| `mutant.yml` | `.evilution.yml` | Notes |
|
|
136
|
+
| --- | --- | --- |
|
|
137
|
+
| `integration.name: rspec` | `integration: rspec` | string, not a mapping |
|
|
138
|
+
| `jobs` | `jobs` | same |
|
|
139
|
+
| `includes` | _(usually omit)_ | Bundler + `preload` cover the load path |
|
|
140
|
+
| `requires` | `preload` | a single preload file (autodetected by default) |
|
|
141
|
+
| `matcher.subjects` | `target` + file args | one expression; combine with positional paths |
|
|
142
|
+
| `matcher.ignore` | `ignore_patterns` / `# evilution:disable` | AST patterns or inline comments |
|
|
143
|
+
| `fail_fast` | `fail_fast` | integer N or `null` |
|
|
144
|
+
| `--score` gate (CLI; default `1.0`) | `min_score` | evilution defaults to `0.0` — set your own gate |
|
|
145
|
+
| _(n/a)_ | `profile: strict` | opt into aggressive truthiness mutators |
|
|
146
|
+
|
|
147
|
+
Run `evilution init` for a fully commented template, and point your editor at
|
|
148
|
+
`schema/evilution.config.schema.json` for autocomplete/validation.
|
|
149
|
+
|
|
150
|
+
> **Note:** the `integration` enum in that JSON schema currently lists only `rspec`
|
|
151
|
+
> and `minitest`. `test-unit` (normalized internally to `test_unit`) is fully
|
|
152
|
+
> supported by the runtime, but a schema-aware editor may flag it as invalid until
|
|
153
|
+
> the schema catches up — it will still run.
|
|
154
|
+
|
|
155
|
+
## 5. Output differences
|
|
156
|
+
|
|
157
|
+
The vocabulary and the math differ — read this before comparing dashboards.
|
|
158
|
+
|
|
159
|
+
| Concept | mutant | Evilution |
|
|
160
|
+
| --- | --- | --- |
|
|
161
|
+
| Detected mutation | `killed` | `killed` |
|
|
162
|
+
| Undetected mutation | `alive` | `survived` |
|
|
163
|
+
| Headline metric | "mutation coverage" (%) | "mutation score" (0.0–1.0) |
|
|
164
|
+
| Pre-existing test failure | `neutral` / `noop` | `neutral` |
|
|
165
|
+
| Operator name in report | **not emitted** | emitted, e.g. `Arithmetic::Swap` |
|
|
166
|
+
| Per-mutation diff | unified diff with `@@` header + context | `- old` / `+ new` pair (and a full `unified_diff` for survivors) |
|
|
167
|
+
| Report shape | nested: session → subject → coverage results | flat per-status buckets |
|
|
168
|
+
|
|
169
|
+
**Score formula.** mutant's coverage is roughly `killed / (total - neutral)` and the
|
|
170
|
+
culture is to gate at 100%. Evilution's score is:
|
|
171
|
+
|
|
172
|
+
```
|
|
173
|
+
score = killed / (total - errors - neutral - equivalent - unresolved - unparseable)
|
|
174
|
+
```
|
|
175
|
+
|
|
176
|
+
i.e. only `killed / (killed + survived + timed_out)`. It defaults to a `min_score` of
|
|
177
|
+
`0.0` (no gate) — set your own threshold in CI with `--min-score`.
|
|
178
|
+
|
|
179
|
+
**Extra statuses** Evilution reports that mutant has no equivalent for:
|
|
180
|
+
|
|
181
|
+
- `unresolved` — no spec file mapped to the mutated source (coverage gap, excluded from score)
|
|
182
|
+
- `unparseable` — the mutated source did not parse, so it never ran (excluded)
|
|
183
|
+
- `equivalent` — proven behaviorally identical to the original (excluded)
|
|
184
|
+
- `timeout` / `error` — surfaced explicitly per mutation
|
|
185
|
+
|
|
186
|
+
**Exit codes:** `0` pass · `1` score below `--min-score` (survivors to address) · `2` error.
|
|
187
|
+
|
|
188
|
+
## 6. Verify the migration with `compare`
|
|
189
|
+
|
|
190
|
+
`evilution compare` ingests **both** a mutant session JSON and an Evilution JSON
|
|
191
|
+
report, normalizes them to a common shape, and buckets the mutations so you can see
|
|
192
|
+
whether the same code is being killed:
|
|
193
|
+
|
|
194
|
+
```sh
|
|
195
|
+
# produce an evilution report
|
|
196
|
+
bundle exec evilution run lib/ --format json --save-session
|
|
197
|
+
|
|
198
|
+
# line it up against your last mutant session JSON
|
|
199
|
+
bundle exec evilution compare \
|
|
200
|
+
--against .mutant/results/<session-id>.json \
|
|
201
|
+
--current .evilution/results/<timestamp>.json \
|
|
202
|
+
--format text
|
|
203
|
+
```
|
|
204
|
+
|
|
205
|
+
The tool auto-detects which file came from which tool. Because the two tools use
|
|
206
|
+
different operator sets, expect the mutation **counts** to differ — the useful signal
|
|
207
|
+
is which subjects lose or gain coverage, not an exact 1:1 match. Operator names are
|
|
208
|
+
absent from mutant's JSON, so comparison keys on file + line + normalized diff, not on
|
|
209
|
+
operator.
|
|
210
|
+
|
|
211
|
+
## 7. Known gaps & parity notes
|
|
212
|
+
|
|
213
|
+
- **Different mutations.** Evilution ships its own operator registry (`--profile
|
|
214
|
+
default` / `strict`); it is not mutant's AST-handler set. Counts and specific
|
|
215
|
+
survivors will differ. Use `compare` to quantify.
|
|
216
|
+
- **No diff-based selection** (`--since <ref>`). Use file args, `--target source:`,
|
|
217
|
+
or `--incremental`.
|
|
218
|
+
- **File-driven, not subject-driven.** Always pass paths; narrow with `--target`.
|
|
219
|
+
- **100%-coverage culture.** Evilution does not assume a 100% gate; choose `min_score`
|
|
220
|
+
deliberately, and note that `:unresolved` mutations are excluded from the score
|
|
221
|
+
rather than counted as failures.
|
|
222
|
+
- **Operator names** appear in Evilution output but not mutant's, so cross-tool diffs
|
|
223
|
+
match on location + change, not operator identity.
|
|
224
|
+
|
|
225
|
+
If something you relied on in mutant has no clear equivalent here, please open an issue
|
|
226
|
+
— parity feedback directly shapes the 1.0 roadmap.
|
|
@@ -66,6 +66,11 @@ class Evilution::CLI::Parser::OptionsBuilder
|
|
|
66
66
|
"Disable per-mutation example targeting (run all examples in resolved spec files)") do
|
|
67
67
|
@options[:example_targeting] = false
|
|
68
68
|
end
|
|
69
|
+
opts.on("--example-targeting MODE", %w[lexical coverage full_file],
|
|
70
|
+
"Per-mutation targeting: lexical (default, name-grep), coverage (run only the",
|
|
71
|
+
"examples that execute the mutated line), or full_file (run all resolved examples)") do |m|
|
|
72
|
+
apply_example_targeting_mode(m)
|
|
73
|
+
end
|
|
69
74
|
opts.on("--example-targeting-fallback MODE", %w[full_file unresolved],
|
|
70
75
|
"Fallback when example targeting finds no match: full_file (default) or unresolved") do |m|
|
|
71
76
|
@options[:example_targeting_fallback] = m
|
|
@@ -77,6 +82,18 @@ class Evilution::CLI::Parser::OptionsBuilder
|
|
|
77
82
|
end
|
|
78
83
|
end
|
|
79
84
|
|
|
85
|
+
# Map the single --example-targeting flag onto the two underlying config axes:
|
|
86
|
+
# full_file is "targeting off" (run all resolved examples); lexical/coverage
|
|
87
|
+
# turn targeting on and select the strategy.
|
|
88
|
+
def apply_example_targeting_mode(mode)
|
|
89
|
+
if mode == "full_file"
|
|
90
|
+
@options[:example_targeting] = false
|
|
91
|
+
else
|
|
92
|
+
@options[:example_targeting] = true
|
|
93
|
+
@options[:example_targeting_strategy] = mode.to_sym
|
|
94
|
+
end
|
|
95
|
+
end
|
|
96
|
+
|
|
80
97
|
def add_flag_options(opts)
|
|
81
98
|
opts.on("--fail-fast", "Stop after N surviving mutants " \
|
|
82
99
|
"(default: disabled; if provided without N, uses 1; use --fail-fast=N)") { @options[:fail_fast] ||= 1 }
|
|
@@ -6,7 +6,9 @@ require_relative "../../spec_resolver"
|
|
|
6
6
|
class Evilution::Config::Builders::SpecResolver
|
|
7
7
|
def self.call(integration:)
|
|
8
8
|
case integration
|
|
9
|
-
when :minitest
|
|
9
|
+
when :minitest, :test_unit
|
|
10
|
+
# test-unit gems mirror minitest's layout (test/ root, _test.rb suffix);
|
|
11
|
+
# without this they default to spec/_spec.rb and resolve to nothing.
|
|
10
12
|
Evilution::SpecResolver.new(test_dir: "test", test_suffix: "_test.rb", request_dir: "integration")
|
|
11
13
|
else
|
|
12
14
|
Evilution::SpecResolver.new
|
|
@@ -0,0 +1,22 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require_relative "base"
|
|
4
|
+
|
|
5
|
+
class Evilution::Config::Validators::ExampleTargetingStrategy < Evilution::Config::Validators::Base
|
|
6
|
+
STRATEGIES = %i[lexical coverage].freeze
|
|
7
|
+
|
|
8
|
+
def self.call(value)
|
|
9
|
+
unless value.is_a?(String) || value.is_a?(Symbol)
|
|
10
|
+
raise Evilution::ConfigError,
|
|
11
|
+
"example_targeting_strategy must be lexical or coverage, got #{value.inspect}"
|
|
12
|
+
end
|
|
13
|
+
|
|
14
|
+
sym = value.to_sym
|
|
15
|
+
unless STRATEGIES.include?(sym)
|
|
16
|
+
raise Evilution::ConfigError,
|
|
17
|
+
"example_targeting_strategy must be lexical or coverage, got #{sym.inspect}"
|
|
18
|
+
end
|
|
19
|
+
|
|
20
|
+
sym
|
|
21
|
+
end
|
|
22
|
+
end
|
data/lib/evilution/config.rb
CHANGED
|
@@ -17,7 +17,7 @@ class Evilution::Config
|
|
|
17
17
|
show_disabled: false, baseline_session: nil, skip_heredoc_literals: false,
|
|
18
18
|
related_specs_heuristic: false, fallback_to_full_suite: false, preload: nil,
|
|
19
19
|
spec_mappings: {}, spec_pattern: nil, example_targeting: true,
|
|
20
|
-
example_targeting_fallback: :full_file,
|
|
20
|
+
example_targeting_fallback: :full_file, example_targeting_strategy: :lexical,
|
|
21
21
|
example_targeting_cache: { max_files: 50, max_blocks: 10_000 },
|
|
22
22
|
quiet_children: false, quiet_children_dir: "tmp/evilution_children",
|
|
23
23
|
profile: :default, canary: true
|
|
@@ -30,7 +30,8 @@ class Evilution::Config
|
|
|
30
30
|
:ignore_patterns, :show_disabled, :baseline_session,
|
|
31
31
|
:skip_heredoc_literals, :related_specs_heuristic,
|
|
32
32
|
:fallback_to_full_suite, :preload, :spec_mappings, :spec_pattern,
|
|
33
|
-
:example_targeting, :example_targeting_fallback, :
|
|
33
|
+
:example_targeting, :example_targeting_fallback, :example_targeting_strategy,
|
|
34
|
+
:example_targeting_cache,
|
|
34
35
|
:spec_selector, :quiet_children, :quiet_children_dir, :profile, :canary
|
|
35
36
|
|
|
36
37
|
def initialize(**options)
|
|
@@ -104,6 +105,10 @@ class Evilution::Config
|
|
|
104
105
|
example_targeting
|
|
105
106
|
end
|
|
106
107
|
|
|
108
|
+
def coverage_targeting?
|
|
109
|
+
example_targeting && example_targeting_strategy == :coverage
|
|
110
|
+
end
|
|
111
|
+
|
|
107
112
|
def fallback_to_full_suite?
|
|
108
113
|
fallback_to_full_suite
|
|
109
114
|
end
|
|
@@ -187,6 +192,13 @@ class Evilution::Config
|
|
|
187
192
|
# without editing the file by exporting EV_DISABLE_EXAMPLE_TARGETING=1.
|
|
188
193
|
# example_targeting: true
|
|
189
194
|
|
|
195
|
+
# How targeting picks examples (default: lexical).
|
|
196
|
+
# lexical - text-grep resolved spec files for the mutated method/class name
|
|
197
|
+
# coverage - run exactly the examples that EXECUTE the mutated line, from a
|
|
198
|
+
# cached full-suite line-coverage map (falls back to lexical for
|
|
199
|
+
# any file the map has not fully built)
|
|
200
|
+
# example_targeting_strategy: lexical
|
|
201
|
+
|
|
190
202
|
# Behavior when targeting finds no matching example (default: full_file).
|
|
191
203
|
# full_file - run every example in the resolved spec files
|
|
192
204
|
# unresolved - mark the mutation :unresolved and skip
|
|
@@ -275,6 +287,7 @@ class Evilution::Config
|
|
|
275
287
|
def assign_example_targeting(merged)
|
|
276
288
|
@example_targeting = merged[:example_targeting] ? true : false
|
|
277
289
|
@example_targeting_fallback = Validators::ExampleTargetingFallback.call(merged[:example_targeting_fallback])
|
|
290
|
+
@example_targeting_strategy = Validators::ExampleTargetingStrategy.call(merged[:example_targeting_strategy])
|
|
278
291
|
@example_targeting_cache = Validators::ExampleTargetingCache.call(merged[:example_targeting_cache])
|
|
279
292
|
end
|
|
280
293
|
end
|
|
@@ -294,6 +307,7 @@ require_relative "config/validators/ignore_patterns"
|
|
|
294
307
|
require_relative "config/validators/spec_pattern"
|
|
295
308
|
require_relative "config/validators/spec_mappings"
|
|
296
309
|
require_relative "config/validators/example_targeting_fallback"
|
|
310
|
+
require_relative "config/validators/example_targeting_strategy"
|
|
297
311
|
require_relative "config/validators/example_targeting_cache"
|
|
298
312
|
require_relative "config/validators/profile"
|
|
299
313
|
require_relative "config/builders"
|
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require "digest"
|
|
4
|
+
require_relative "../coverage"
|
|
5
|
+
|
|
6
|
+
# Stable, content-addressed digest of a source file, used by MapStore to detect
|
|
7
|
+
# when a cached coverage entry has gone stale. Path-independent: the digest
|
|
8
|
+
# depends only on the bytes, so moving a file does not invalidate it. Returns
|
|
9
|
+
# nil for a missing file so callers treat it as "not fresh".
|
|
10
|
+
class Evilution::Coverage::Digest
|
|
11
|
+
def for_file(path)
|
|
12
|
+
return nil unless File.file?(path)
|
|
13
|
+
|
|
14
|
+
::Digest::SHA256.hexdigest(File.binread(path))
|
|
15
|
+
end
|
|
16
|
+
end
|
|
@@ -0,0 +1,64 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require_relative "../coverage"
|
|
4
|
+
|
|
5
|
+
# Immutable query over a per-example line-coverage index:
|
|
6
|
+
# source file -> line -> [example locations ("spec.rb:line")].
|
|
7
|
+
# built_files records the source files for which the build completed, so
|
|
8
|
+
# callers can distinguish "line genuinely uncovered" (file built, no entry)
|
|
9
|
+
# from "we never built this file" (must fall back, not assert a gap).
|
|
10
|
+
class Evilution::Coverage::Map
|
|
11
|
+
def self.from_h(hash)
|
|
12
|
+
index = (hash["index"] || {}).transform_values do |lines|
|
|
13
|
+
lines.transform_keys(&:to_i)
|
|
14
|
+
end
|
|
15
|
+
executed = (hash["executed_lines"] || {}).transform_values { |lines| lines.map(&:to_i) }
|
|
16
|
+
new(index: index, built_files: hash["built_files"] || [], executed_lines: executed)
|
|
17
|
+
end
|
|
18
|
+
|
|
19
|
+
# executed_lines records, per file, the lines that ran at all during the build
|
|
20
|
+
# (including lines covered only at load, e.g. a `def` line, which are
|
|
21
|
+
# attributed to no single example). It lets a caller tell a TRUE coverage gap
|
|
22
|
+
# (line never executed) from a load-covered line an example may still exercise
|
|
23
|
+
# indirectly -- so the latter falls back instead of being mis-skipped.
|
|
24
|
+
def initialize(index:, built_files:, executed_lines: {})
|
|
25
|
+
@index = deep_freeze_index(index)
|
|
26
|
+
@built_files = built_files.to_a.freeze
|
|
27
|
+
@executed_lines = deep_freeze_executed(executed_lines)
|
|
28
|
+
freeze
|
|
29
|
+
end
|
|
30
|
+
|
|
31
|
+
def examples_for(file, line)
|
|
32
|
+
@index.dig(file, line) || []
|
|
33
|
+
end
|
|
34
|
+
|
|
35
|
+
def built?(file)
|
|
36
|
+
@built_files.include?(file)
|
|
37
|
+
end
|
|
38
|
+
|
|
39
|
+
def executed?(file, line)
|
|
40
|
+
lines = @executed_lines[file]
|
|
41
|
+
!lines.nil? && lines.include?(line)
|
|
42
|
+
end
|
|
43
|
+
|
|
44
|
+
def to_h
|
|
45
|
+
{ "index" => @index, "built_files" => @built_files, "executed_lines" => @executed_lines }
|
|
46
|
+
end
|
|
47
|
+
|
|
48
|
+
private
|
|
49
|
+
|
|
50
|
+
def deep_freeze_index(index)
|
|
51
|
+
index.each_with_object({}) do |(file, lines), out|
|
|
52
|
+
frozen_lines = lines.each_with_object({}) do |(line, locs), inner|
|
|
53
|
+
inner[line] = locs.uniq.sort.freeze
|
|
54
|
+
end
|
|
55
|
+
out[file] = frozen_lines.freeze
|
|
56
|
+
end.freeze
|
|
57
|
+
end
|
|
58
|
+
|
|
59
|
+
def deep_freeze_executed(executed)
|
|
60
|
+
executed.each_with_object({}) do |(file, lines), out|
|
|
61
|
+
out[file] = lines.map(&:to_i).uniq.sort.freeze
|
|
62
|
+
end.freeze
|
|
63
|
+
end
|
|
64
|
+
end
|
|
@@ -0,0 +1,82 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require "coverage"
|
|
4
|
+
require "stringio"
|
|
5
|
+
require_relative "../coverage"
|
|
6
|
+
require_relative "map"
|
|
7
|
+
require_relative "recorder"
|
|
8
|
+
|
|
9
|
+
# Builds a per-example coverage Map by running the given spec files under
|
|
10
|
+
# ::Coverage (lines mode) with the Recorder installed as an around(:each)
|
|
11
|
+
# hook. The run happens in a FORKED CHILD so the global ::RSpec.configure
|
|
12
|
+
# hook, ::RSpec.world mutation, and ::Coverage state never leak into the
|
|
13
|
+
# calling process; the parent receives only the serialized map over a pipe.
|
|
14
|
+
class Evilution::Coverage::MapBuilder
|
|
15
|
+
LOCATION_LINE_SUFFIX = /\A(.+?)(:\d+(?::\d+)*)\z/
|
|
16
|
+
|
|
17
|
+
# RSpec reports example locations relative to its run dir as "./spec/x.rb:5".
|
|
18
|
+
# Store them ABSOLUTE (path expanded against the project root, line suffix
|
|
19
|
+
# preserved) so they replay regardless of the per-mutation run's CWD --
|
|
20
|
+
# Integration::RSpec#resolve_target passes absolute targets through unchanged.
|
|
21
|
+
def self.absolute_location(raw, root)
|
|
22
|
+
match = LOCATION_LINE_SUFFIX.match(raw)
|
|
23
|
+
path, suffix = match ? [match[1], match[2]] : [raw, ""]
|
|
24
|
+
"#{File.expand_path(path, root)}#{suffix}"
|
|
25
|
+
end
|
|
26
|
+
|
|
27
|
+
def initialize(spec_files:, target_files:, project_root: Evilution::PROJECT_ROOT)
|
|
28
|
+
@spec_files = spec_files
|
|
29
|
+
@target_files = target_files
|
|
30
|
+
@project_root = project_root.to_s
|
|
31
|
+
end
|
|
32
|
+
|
|
33
|
+
def call
|
|
34
|
+
read_io, write_io = IO.pipe
|
|
35
|
+
read_io.binmode
|
|
36
|
+
write_io.binmode
|
|
37
|
+
pid = fork do
|
|
38
|
+
read_io.close
|
|
39
|
+
Marshal.dump(build_in_child.to_h, write_io)
|
|
40
|
+
write_io.close
|
|
41
|
+
exit!(0)
|
|
42
|
+
end
|
|
43
|
+
write_io.close
|
|
44
|
+
payload = read_io.read
|
|
45
|
+
read_io.close
|
|
46
|
+
Process.wait(pid)
|
|
47
|
+
# Trust boundary: payload is a Marshal dump this process's own forked child
|
|
48
|
+
# produced over a private pipe -- same rationale as Channel::Frame.
|
|
49
|
+
Evilution::Coverage::Map.from_h(Marshal.load(payload))
|
|
50
|
+
end
|
|
51
|
+
|
|
52
|
+
private
|
|
53
|
+
|
|
54
|
+
# Runs entirely inside the forked child.
|
|
55
|
+
def build_in_child
|
|
56
|
+
recorder = Evilution::Coverage::Recorder.new(target_files: @target_files)
|
|
57
|
+
::Coverage.start(lines: true) unless ::Coverage.running?
|
|
58
|
+
# The child inherited the parent's fully-loaded RSpec.world; wipe it so the
|
|
59
|
+
# nested runner executes ONLY @spec_files and never re-enters the host suite
|
|
60
|
+
# (which would recursively fork this builder).
|
|
61
|
+
reset_world
|
|
62
|
+
install_hook(recorder)
|
|
63
|
+
::RSpec::Core::Runner.run(@spec_files, StringIO.new, StringIO.new)
|
|
64
|
+
recorder.to_map(built_files: @target_files)
|
|
65
|
+
end
|
|
66
|
+
|
|
67
|
+
def reset_world
|
|
68
|
+
::RSpec.respond_to?(:clear_examples) ? ::RSpec.clear_examples : ::RSpec.reset
|
|
69
|
+
end
|
|
70
|
+
|
|
71
|
+
def install_hook(recorder)
|
|
72
|
+
root = @project_root
|
|
73
|
+
::RSpec.configure do |config|
|
|
74
|
+
config.around(:each) do |example|
|
|
75
|
+
location = Evilution::Coverage::MapBuilder.absolute_location(
|
|
76
|
+
example.metadata[:location].to_s, root
|
|
77
|
+
)
|
|
78
|
+
recorder.around_example(location) { example.run }
|
|
79
|
+
end
|
|
80
|
+
end
|
|
81
|
+
end
|
|
82
|
+
end
|
|
@@ -0,0 +1,87 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require "json"
|
|
4
|
+
require "fileutils"
|
|
5
|
+
require_relative "../coverage"
|
|
6
|
+
require_relative "map"
|
|
7
|
+
require_relative "digest"
|
|
8
|
+
|
|
9
|
+
# Disk cache for a per-example coverage Map under .evilution/coverage/, keyed by
|
|
10
|
+
# a per-file content digest so a map survives across runs and invalidates one
|
|
11
|
+
# file at a time.
|
|
12
|
+
#
|
|
13
|
+
# load is partial: it returns a Map pruned to the files whose on-disk content
|
|
14
|
+
# still matches the cached digest. A stale or deleted file is dropped (so its
|
|
15
|
+
# `built?` is false and the caller falls back to lexical targeting) while every
|
|
16
|
+
# fresh file stays queryable. A missing or corrupt cache returns nil, signalling
|
|
17
|
+
# the caller to rebuild from scratch.
|
|
18
|
+
class Evilution::Coverage::MapStore
|
|
19
|
+
DEFAULT_ROOT = ".evilution/coverage"
|
|
20
|
+
CACHE_FILE = "map.json"
|
|
21
|
+
|
|
22
|
+
def initialize(root: DEFAULT_ROOT, digest: Evilution::Coverage::Digest.new)
|
|
23
|
+
@root = root
|
|
24
|
+
@digest = digest
|
|
25
|
+
end
|
|
26
|
+
|
|
27
|
+
def save(map, source_files)
|
|
28
|
+
payload = { "digests" => digests_for(source_files), "map" => map.to_h }
|
|
29
|
+
FileUtils.mkdir_p(@root)
|
|
30
|
+
File.write(cache_path, JSON.generate(payload))
|
|
31
|
+
end
|
|
32
|
+
|
|
33
|
+
def load(source_files)
|
|
34
|
+
payload = read_payload
|
|
35
|
+
return nil unless payload
|
|
36
|
+
|
|
37
|
+
cached_digests = payload["digests"] || {}
|
|
38
|
+
fresh = source_files.select { |file| fresh?(file, cached_digests) }
|
|
39
|
+
pruned_map(payload["map"] || {}, fresh)
|
|
40
|
+
end
|
|
41
|
+
|
|
42
|
+
# Source files whose on-disk content no longer matches the cache (changed,
|
|
43
|
+
# deleted, or never cached) -- the caller rebuilds these. Every file is stale
|
|
44
|
+
# when there is no cache at all.
|
|
45
|
+
def stale_files(source_files)
|
|
46
|
+
payload = read_payload
|
|
47
|
+
return source_files.dup unless payload
|
|
48
|
+
|
|
49
|
+
cached_digests = payload["digests"] || {}
|
|
50
|
+
source_files.reject { |file| fresh?(file, cached_digests) }
|
|
51
|
+
end
|
|
52
|
+
|
|
53
|
+
private
|
|
54
|
+
|
|
55
|
+
def fresh?(file, cached_digests)
|
|
56
|
+
cached = cached_digests[file]
|
|
57
|
+
!cached.nil? && cached == @digest.for_file(file)
|
|
58
|
+
end
|
|
59
|
+
|
|
60
|
+
def pruned_map(raw_map, fresh_files)
|
|
61
|
+
index = (raw_map["index"] || {}).slice(*fresh_files)
|
|
62
|
+
built = (raw_map["built_files"] || []) & fresh_files
|
|
63
|
+
executed = (raw_map["executed_lines"] || {}).slice(*fresh_files)
|
|
64
|
+
Evilution::Coverage::Map.from_h(
|
|
65
|
+
"index" => index, "built_files" => built, "executed_lines" => executed
|
|
66
|
+
)
|
|
67
|
+
end
|
|
68
|
+
|
|
69
|
+
def digests_for(source_files)
|
|
70
|
+
source_files.each_with_object({}) do |file, out|
|
|
71
|
+
digest = @digest.for_file(file)
|
|
72
|
+
out[file] = digest unless digest.nil?
|
|
73
|
+
end
|
|
74
|
+
end
|
|
75
|
+
|
|
76
|
+
def read_payload
|
|
77
|
+
return nil unless File.file?(cache_path)
|
|
78
|
+
|
|
79
|
+
JSON.parse(File.read(cache_path))
|
|
80
|
+
rescue JSON::ParserError, Errno::ENOENT
|
|
81
|
+
nil
|
|
82
|
+
end
|
|
83
|
+
|
|
84
|
+
def cache_path
|
|
85
|
+
File.join(@root, CACHE_FILE)
|
|
86
|
+
end
|
|
87
|
+
end
|
|
@@ -0,0 +1,85 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require_relative "../coverage"
|
|
4
|
+
require_relative "map"
|
|
5
|
+
|
|
6
|
+
# Wraps each example with a before/after coverage diff and attributes the
|
|
7
|
+
# newly-executed lines (in target files only) to that example's location.
|
|
8
|
+
# coverage_source is injected for testability; in production it is
|
|
9
|
+
# -> { ::Coverage.peek_result }.
|
|
10
|
+
class Evilution::Coverage::Recorder
|
|
11
|
+
def initialize(target_files:, coverage_source: -> { ::Coverage.peek_result })
|
|
12
|
+
@target_files = target_files.to_a
|
|
13
|
+
@coverage_source = coverage_source
|
|
14
|
+
@index = Hash.new { |h, file| h[file] = Hash.new { |g, line| g[line] = [] } }
|
|
15
|
+
@executed = Hash.new { |h, file| h[file] = [] }
|
|
16
|
+
end
|
|
17
|
+
|
|
18
|
+
def around_example(example_location)
|
|
19
|
+
before = snapshot
|
|
20
|
+
result = yield
|
|
21
|
+
after = snapshot
|
|
22
|
+
attribute(before, after, example_location)
|
|
23
|
+
result
|
|
24
|
+
end
|
|
25
|
+
|
|
26
|
+
def to_map(built_files:)
|
|
27
|
+
Evilution::Coverage::Map.new(
|
|
28
|
+
index: materialize(@index),
|
|
29
|
+
built_files: built_files,
|
|
30
|
+
executed_lines: @executed.transform_values(&:uniq)
|
|
31
|
+
)
|
|
32
|
+
end
|
|
33
|
+
|
|
34
|
+
private
|
|
35
|
+
|
|
36
|
+
def snapshot
|
|
37
|
+
@coverage_source.call || {}
|
|
38
|
+
end
|
|
39
|
+
|
|
40
|
+
def attribute(before, after, example_location)
|
|
41
|
+
@target_files.each do |file|
|
|
42
|
+
after_counts = line_counts(after[file])
|
|
43
|
+
next unless after_counts
|
|
44
|
+
|
|
45
|
+
record_executed(file, after_counts)
|
|
46
|
+
record_increases(file, line_counts(before[file]) || [], after_counts, example_location)
|
|
47
|
+
end
|
|
48
|
+
end
|
|
49
|
+
|
|
50
|
+
# Every line with a non-zero count in the after-snapshot has run at least once
|
|
51
|
+
# by now -- including lines covered only at load (a `def` line is already > 0
|
|
52
|
+
# in the first example's after-snapshot). Recording them lets the Map tell a
|
|
53
|
+
# load-covered line from a line that never ran.
|
|
54
|
+
def record_executed(file, after_counts)
|
|
55
|
+
after_counts.each_with_index do |count, idx|
|
|
56
|
+
next if count.nil? || count.zero?
|
|
57
|
+
|
|
58
|
+
@executed[file] << (idx + 1)
|
|
59
|
+
end
|
|
60
|
+
end
|
|
61
|
+
|
|
62
|
+
# Credit example_location with every line whose execution count rose between
|
|
63
|
+
# the before/after snapshots (a newly-executed, executable line).
|
|
64
|
+
def record_increases(file, before_counts, after_counts, example_location)
|
|
65
|
+
after_counts.each_with_index do |count, idx|
|
|
66
|
+
next if count.nil? || count.zero?
|
|
67
|
+
next unless count > (before_counts[idx] || 0)
|
|
68
|
+
|
|
69
|
+
@index[file][idx + 1] << example_location
|
|
70
|
+
end
|
|
71
|
+
end
|
|
72
|
+
|
|
73
|
+
# Coverage.peek_result yields per-file line counts either as a bare array
|
|
74
|
+
# (legacy Coverage.start) or as a { lines: [...] } hash (Coverage.start with
|
|
75
|
+
# lines:/branches:/methods: modes). Normalize to the bare counts array.
|
|
76
|
+
def line_counts(entry)
|
|
77
|
+
entry.is_a?(Hash) ? entry[:lines] : entry
|
|
78
|
+
end
|
|
79
|
+
|
|
80
|
+
def materialize(index)
|
|
81
|
+
index.each_with_object({}) do |(file, lines), out|
|
|
82
|
+
out[file] = lines.each_with_object({}) { |(line, locs), inner| inner[line] = locs }
|
|
83
|
+
end
|
|
84
|
+
end
|
|
85
|
+
end
|