evilution 1.5.0 → 1.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.beads/interactions.jsonl +1 -0
- data/.rubocop_todo.yml +30 -0
- data/CHANGELOG.md +59 -0
- data/README.md +285 -45
- data/docs/architecture.md +110 -12
- data/docs/ast_pattern_syntax.md +35 -7
- data/docs/combined_runs_spike.md +407 -0
- data/docs/integrations.md +1 -1
- data/docs/isolation.md +63 -2
- data/docs/migration-from-mutant.md +16 -11
- data/docs/mutant_schemata_spike.md +387 -0
- data/docs/operator_yield_study.md +576 -0
- data/docs/probe_first_spike.md +264 -0
- data/lib/evilution/ast/ancestor_lookup.rb +36 -0
- data/lib/evilution/ast/callback_declaration.rb +36 -3
- data/lib/evilution/ast/class_body.rb +62 -16
- data/lib/evilution/ast/def_region.rb +130 -0
- data/lib/evilution/ast/local_declaration.rb +37 -0
- data/lib/evilution/ast/parent_index.rb +50 -0
- data/lib/evilution/ast/parse_cache.rb +27 -0
- data/lib/evilution/ast/parser.rb +8 -3
- data/lib/evilution/ast/pattern/filter.rb +53 -12
- data/lib/evilution/ast/skip_filter.rb +83 -0
- data/lib/evilution/ast/skip_sites/debug_output.rb +67 -0
- data/lib/evilution/ast/skip_sites/error_messages.rb +74 -0
- data/lib/evilution/ast/skip_sites/logging.rb +93 -0
- data/lib/evilution/ast/skip_sites/message.rb +44 -0
- data/lib/evilution/ast/skip_sites/nodes.rb +105 -0
- data/lib/evilution/ast/skip_sites/presets.rb +31 -0
- data/lib/evilution/ast/skip_sites/purity.rb +69 -0
- data/lib/evilution/ast/skip_sites/requires.rb +40 -0
- data/lib/evilution/ast/skip_sites.rb +9 -0
- data/lib/evilution/ast/source_surgeon.rb +20 -2
- data/lib/evilution/ast/syntax_check.rb +10 -8
- data/lib/evilution/baseline/report.rb +11 -2
- data/lib/evilution/baseline.rb +31 -6
- data/lib/evilution/cache/keys.rb +140 -0
- data/lib/evilution/cache/run_fingerprint.rb +51 -0
- data/lib/evilution/cache/skeleton.rb +57 -0
- data/lib/evilution/cache/spec_digest.rb +39 -0
- data/lib/evilution/cache.rb +145 -42
- data/lib/evilution/class_tree.rb +38 -0
- data/lib/evilution/cli/command.rb +12 -4
- data/lib/evilution/cli/commands/run.rb +12 -3
- data/lib/evilution/cli/commands/subjects.rb +5 -2
- data/lib/evilution/cli/parser/options_builder.rb +67 -3
- data/lib/evilution/cli/printers/environment.rb +34 -4
- data/lib/evilution/cli/printers/session_detail.rb +20 -3
- data/lib/evilution/compare/categorizer.rb +2 -1
- data/lib/evilution/compare/normalizer.rb +3 -2
- data/lib/evilution/config/sources.rb +3 -0
- data/lib/evilution/config/validators/base.rb +9 -0
- data/lib/evilution/config/validators/since.rb +22 -0
- data/lib/evilution/config/validators/skip_sites.rb +24 -0
- data/lib/evilution/config/validators/timeout_ceiling.rb +10 -0
- data/lib/evilution/config/validators/timeout_factor.rb +10 -0
- data/lib/evilution/config/validators/timeout_floor.rb +10 -0
- data/lib/evilution/config.rb +122 -6
- data/lib/evilution/coverage/load_tracker.rb +50 -0
- data/lib/evilution/coverage/map.rb +42 -5
- data/lib/evilution/coverage/map_builder.rb +123 -22
- data/lib/evilution/coverage/map_store.rb +15 -5
- data/lib/evilution/coverage/recorder.rb +42 -3
- data/lib/evilution/coverage_example_filter.rb +35 -0
- data/lib/evilution/equivalent/heuristic/comment_marking.rb +14 -1
- data/lib/evilution/example_filter.rb +20 -0
- data/lib/evilution/git/changed_lines.rb +103 -0
- data/lib/evilution/git/since_scope.rb +77 -0
- data/lib/evilution/git/unified_diff.rb +68 -0
- data/lib/evilution/infra_crash.rb +26 -0
- data/lib/evilution/integration/base.rb +97 -6
- data/lib/evilution/integration/crash_detector.rb +46 -6
- data/lib/evilution/integration/known_failures.rb +4 -0
- data/lib/evilution/integration/loading/body_call_neutralizer.rb +49 -11
- data/lib/evilution/integration/loading/callback_equivalence.rb +62 -0
- data/lib/evilution/integration/loading/callback_redeclaration.rb +151 -36
- data/lib/evilution/integration/loading/concern_redeclaration.rb +16 -12
- data/lib/evilution/integration/loading/concern_registry.rb +87 -0
- data/lib/evilution/integration/loading/concern_state_cleaner.rb +19 -8
- data/lib/evilution/integration/loading/created_modules.rb +56 -0
- data/lib/evilution/integration/loading/frozen_namespaces.rb +80 -0
- data/lib/evilution/integration/loading/mutation_applier.rb +42 -11
- data/lib/evilution/integration/loading/neutralization.rb +41 -0
- data/lib/evilution/integration/loading/redefinition_recovery.rb +5 -3
- data/lib/evilution/integration/loading/superseded_modules.rb +22 -19
- data/lib/evilution/integration/minitest/stop_hook.rb +57 -0
- data/lib/evilution/integration/minitest.rb +22 -2
- data/lib/evilution/integration/minitest_crash_detector.rb +19 -0
- data/lib/evilution/integration/rspec/baseline_runner.rb +3 -2
- data/lib/evilution/integration/rspec/crash_detector_lifecycle.rb +6 -1
- data/lib/evilution/integration/rspec/example_filter_applier.rb +16 -0
- data/lib/evilution/integration/rspec/example_ids.rb +23 -0
- data/lib/evilution/integration/rspec/example_order.rb +86 -0
- data/lib/evilution/integration/rspec/killer_first.rb +72 -0
- data/lib/evilution/integration/rspec/result_builder.rb +7 -0
- data/lib/evilution/integration/rspec/state_guard/example_group_descendants.rb +44 -0
- data/lib/evilution/integration/rspec/state_guard.rb +2 -2
- data/lib/evilution/integration/rspec/target_resolver.rb +42 -0
- data/lib/evilution/integration/rspec/test_file_resolver.rb +4 -2
- data/lib/evilution/integration/rspec.rb +98 -47
- data/lib/evilution/integration/stop_policy.rb +49 -0
- data/lib/evilution/integration/test_unit/dispatcher.rb +20 -1
- data/lib/evilution/integration/test_unit/subject_class_registry.rb +6 -1
- data/lib/evilution/integration/test_unit.rb +28 -2
- data/lib/evilution/isolation/fork.rb +58 -13
- data/lib/evilution/isolation/in_process.rb +12 -1
- data/lib/evilution/line_set.rb +72 -0
- data/lib/evilution/mcp/info_tool/actions/environment.rb +12 -16
- data/lib/evilution/mcp/info_tool/actions/subjects.rb +3 -5
- data/lib/evilution/mcp/info_tool/status_glossary.rb +15 -2
- data/lib/evilution/mcp/info_tool.rb +24 -17
- data/lib/evilution/mcp/mutate_tool/config_builder.rb +13 -1
- data/lib/evilution/mcp/mutate_tool/option_parser.rb +2 -2
- data/lib/evilution/mcp/mutate_tool/progress_streamer.rb +1 -1
- data/lib/evilution/mcp/mutate_tool/report_trimmer.rb +28 -5
- data/lib/evilution/mcp/mutate_tool/survived_enricher.rb +12 -7
- data/lib/evilution/mcp/mutate_tool.rb +32 -7
- data/lib/evilution/mcp/session_tool.rb +1 -2
- data/lib/evilution/mcp.rb +4 -0
- data/lib/evilution/mutation/line_diff.rb +111 -0
- data/lib/evilution/mutation.rb +17 -13
- data/lib/evilution/mutator/base.rb +121 -98
- data/lib/evilution/mutator/constant_value_subjects.rb +29 -1
- data/lib/evilution/mutator/line_range_filter.rb +27 -0
- data/lib/evilution/mutator/load_sources.rb +88 -0
- data/lib/evilution/mutator/operator/alias_removal.rb +2 -5
- data/lib/evilution/mutator/operator/argument_method_call_replacement.rb +3 -0
- data/lib/evilution/mutator/operator/attr_accessor_swap.rb +2 -5
- data/lib/evilution/mutator/operator/constant_namespace_strip.rb +3 -3
- data/lib/evilution/mutator/operator/freeze_removal.rb +29 -11
- data/lib/evilution/mutator/operator/method_call_removal.rb +3 -1
- data/lib/evilution/mutator/operator/mixin_removal.rb +2 -5
- data/lib/evilution/mutator/operator/superclass_removal.rb +2 -5
- data/lib/evilution/mutator/operator/symbol_literal.rb +1 -1
- data/lib/evilution/mutator/operator/visibility_removal.rb +2 -5
- data/lib/evilution/mutator/registry.rb +11 -5
- data/lib/evilution/mutator/restore_source_cache.rb +41 -0
- data/lib/evilution/parallel/pool.rb +11 -4
- data/lib/evilution/parallel/work_queue/dispatcher/result_stream.rb +40 -0
- data/lib/evilution/parallel/work_queue/dispatcher.rb +20 -7
- data/lib/evilution/parallel/work_queue/worker/loop.rb +7 -2
- data/lib/evilution/parallel/work_queue/worker.rb +10 -3
- data/lib/evilution/parallel/work_queue/worker_stat.rb +7 -0
- data/lib/evilution/parallel/work_queue.rb +69 -12
- data/lib/evilution/parallel_db_warning.rb +39 -12
- data/lib/evilution/process_supervisor.rb +22 -2
- data/lib/evilution/reporter/cli/item_formatters/coverage_gap.rb +13 -2
- data/lib/evilution/reporter/cli/item_formatters/subject_score.rb +5 -1
- data/lib/evilution/reporter/cli/line_formatters/mutations.rb +9 -2
- data/lib/evilution/reporter/cli/line_formatters/result_line.rb +9 -1
- data/lib/evilution/reporter/cli/line_formatters/skipped_sites.rb +14 -0
- data/lib/evilution/reporter/cli/line_formatters/uncovered_code.rb +17 -0
- data/lib/evilution/reporter/cli/metrics_block.rb +4 -0
- data/lib/evilution/reporter/cli/trailer.rb +4 -4
- data/lib/evilution/reporter/cli.rb +9 -2
- data/lib/evilution/reporter/html/assets/style.css +8 -0
- data/lib/evilution/reporter/html/sections/file_section.rb +9 -0
- data/lib/evilution/reporter/html/sections/uncovered_details.rb +27 -0
- data/lib/evilution/reporter/html/templates/file_section.html.erb +2 -1
- data/lib/evilution/reporter/html/templates/summary_cards.html.erb +3 -0
- data/lib/evilution/reporter/html/templates/uncovered_details.html.erb +11 -0
- data/lib/evilution/reporter/json/mutation_detail.rb +77 -0
- data/lib/evilution/reporter/json/subjects.rb +1 -0
- data/lib/evilution/reporter/json/timings.rb +42 -0
- data/lib/evilution/reporter/json.rb +13 -50
- data/lib/evilution/result/coverage_gap.rb +5 -0
- data/lib/evilution/result/mutation_result.rb +55 -3
- data/lib/evilution/result/mutation_timings.rb +54 -0
- data/lib/evilution/result/run_timings.rb +20 -0
- data/lib/evilution/result/subject_score.rb +8 -1
- data/lib/evilution/result/subject_scorer.rb +5 -3
- data/lib/evilution/result/summary.rb +49 -12
- data/lib/evilution/runner/baseline_runner.rb +47 -5
- data/lib/evilution/runner/diagnostics.rb +0 -13
- data/lib/evilution/runner/isolation_resolver.rb +16 -4
- data/lib/evilution/runner/mutation_executor/isolated_run.rb +34 -0
- data/lib/evilution/runner/mutation_executor/kill_history.rb +66 -0
- data/lib/evilution/runner/mutation_executor/mutation_runner.rb +31 -12
- data/lib/evilution/runner/mutation_executor/neutralizer/baseline_failed.rb +1 -0
- data/lib/evilution/runner/mutation_executor/neutralizer/infra_error.rb +6 -20
- data/lib/evilution/runner/mutation_executor/probe_first.rb +99 -0
- data/lib/evilution/runner/mutation_executor/probe_first_run.rb +59 -0
- data/lib/evilution/runner/mutation_executor/result_cache.rb +45 -14
- data/lib/evilution/runner/mutation_executor/result_notifier.rb +1 -1
- data/lib/evilution/runner/mutation_executor/result_packer.rb +54 -24
- data/lib/evilution/runner/mutation_executor/runless_verdict.rb +39 -0
- data/lib/evilution/runner/mutation_executor/strategy/parallel/hinted_items.rb +27 -0
- data/lib/evilution/runner/mutation_executor/strategy/parallel/ordered_delivery.rb +97 -0
- data/lib/evilution/runner/mutation_executor/strategy/parallel.rb +102 -83
- data/lib/evilution/runner/mutation_executor/timed_run.rb +33 -0
- data/lib/evilution/runner/mutation_executor/timeout_policy/adaptive.rb +67 -0
- data/lib/evilution/runner/mutation_executor/timeout_policy/base.rb +35 -0
- data/lib/evilution/runner/mutation_executor/timeout_policy/fixed.rb +17 -0
- data/lib/evilution/runner/mutation_executor/timeout_policy.rb +56 -0
- data/lib/evilution/runner/mutation_executor.rb +75 -10
- data/lib/evilution/runner/mutation_planner.rb +29 -21
- data/lib/evilution/runner/phase_clock.rb +22 -0
- data/lib/evilution/runner/report_publisher.rb +1 -1
- data/lib/evilution/runner/subject_pipeline.rb +18 -8
- data/lib/evilution/runner.rb +87 -26
- data/lib/evilution/session/diff.rb +9 -2
- data/lib/evilution/session/store.rb +17 -2
- data/lib/evilution/source_ast_cache.rb +10 -0
- data/lib/evilution/spec_ast_cache.rb +15 -0
- data/lib/evilution/spec_resolver/constant_references.rb +66 -0
- data/lib/evilution/spec_resolver/project_files.rb +50 -0
- data/lib/evilution/spec_resolver/sibling_spec_ranker.rb +111 -0
- data/lib/evilution/spec_resolver/spec_pattern.rb +19 -0
- data/lib/evilution/spec_resolver.rb +76 -65
- data/lib/evilution/spec_selector.rb +17 -13
- data/lib/evilution/subject.rb +7 -2
- data/lib/evilution/version.rb +1 -1
- data/lib/evilution/worker_database/copies.rb +73 -0
- data/lib/evilution/worker_database/schema.rb +39 -0
- data/lib/evilution/worker_database/sqlite_copy.rb +65 -0
- data/lib/evilution/worker_database.rb +214 -0
- data/schema/evilution.config.schema.json +74 -2
- metadata +84 -13
- data/lib/evilution/integration/rspec/state_guard/object_space_example_groups.rb +0 -42
- data/scripts/benchmark_density +0 -262
- data/scripts/benchmark_density.yml +0 -19
- data/scripts/canary_manifest.yml +0 -47
- data/scripts/compare_mutations +0 -421
- data/scripts/compare_mutations.yml +0 -24
- data/scripts/compare_targeting +0 -279
- data/scripts/compare_targeting.example.yml +0 -24
- data/scripts/mutant_json_adapter +0 -227
|
@@ -0,0 +1,407 @@
|
|
|
1
|
+
# Combined runs: spike report
|
|
2
|
+
|
|
3
|
+
Status: **no-go**. No prototype in `lib/`. The prototype was a harness
|
|
4
|
+
loaded from outside the gem, kept in `scripts/combined_runs_spike/` (see
|
|
5
|
+
[Reproducing](#reproducing)). The corpus
|
|
6
|
+
gate was met, but a constructed example shows the method can report a
|
|
7
|
+
survivor as killed, and the coverage map cannot rule that out. See
|
|
8
|
+
[Decision](#decision) for what would have to change first.
|
|
9
|
+
|
|
10
|
+
## Idea
|
|
11
|
+
|
|
12
|
+
Running several mutants in one process only pays if one run gives a verdict
|
|
13
|
+
for each of them. Naive grouping (apply k mutants, run their tests once) does
|
|
14
|
+
not: with 85–95% of mutants killed, a group almost always fails, and nothing
|
|
15
|
+
says which member caused it.
|
|
16
|
+
|
|
17
|
+
The variant tested here uses the per-example coverage map
|
|
18
|
+
(`--example-targeting coverage`, RSpec only). Mutants are combined only when
|
|
19
|
+
the sets of examples that execute their changed lines are disjoint. The union
|
|
20
|
+
of their targeted examples runs once. Each failing example executes the
|
|
21
|
+
changed lines of exactly one member in the original code, so its failure is
|
|
22
|
+
attributed to that member. A member with no failing example is not called a
|
|
23
|
+
survivor on that basis. It falls back to a single run.
|
|
24
|
+
|
|
25
|
+
## Prototype
|
|
26
|
+
|
|
27
|
+
A harness (`RUBYOPT=-r<harness>`) that prepends `MutationExecutor#run_batch`.
|
|
28
|
+
|
|
29
|
+
**Eligibility.** A mutant can join a group when all of these hold:
|
|
30
|
+
|
|
31
|
+
- it gets no verdict without a run (not unparseable, not `uncovered`, not
|
|
32
|
+
implied by a probe sentinel);
|
|
33
|
+
- its file is built in the map;
|
|
34
|
+
- the examples covering its reported line are known, so its single run would
|
|
35
|
+
be coverage-targeted and not lexical;
|
|
36
|
+
- every changed line that ran at all ran inside some example. A line that
|
|
37
|
+
only ran while the file loaded (a constant, a class-body call, a `def`
|
|
38
|
+
signature) can reach any example, so its mutant is not eligible.
|
|
39
|
+
|
|
40
|
+
Its *affecting set* is the union of the examples covering its changed lines.
|
|
41
|
+
|
|
42
|
+
**Grouping.** Greedy first fit, in the run's own order. A mutant joins the
|
|
43
|
+
first group where it overlaps no member's affecting set and, within the same
|
|
44
|
+
file, no member's changed byte range (a touching range counts as
|
|
45
|
+
overlapping). An optional cap limits members per group. An optional rule
|
|
46
|
+
puts at most one mutant of each subject in a group.
|
|
47
|
+
|
|
48
|
+
**Group run.** One forked process per group, `--jobs` of them at a time, with
|
|
49
|
+
a sandbox directory and the isolated-worker flags as in `Isolation::Fork`.
|
|
50
|
+
In that process:
|
|
51
|
+
|
|
52
|
+
- All members are applied. Members in the same file are merged: their
|
|
53
|
+
changed regions of the neutralized source are spliced into one source,
|
|
54
|
+
which `MutationApplier` evaluates once.
|
|
55
|
+
- The union of the members' targeted examples runs once, ordered
|
|
56
|
+
killer-first as a single run would be. The parent records each group's
|
|
57
|
+
kills in the kill history as the group finishes, so later groups and the
|
|
58
|
+
remaining single runs start with those killers.
|
|
59
|
+
- **Per-member stop on kill.** An `around(:each)` hook skips the remaining
|
|
60
|
+
examples of a member that has already been killed. When every member is
|
|
61
|
+
killed, the run stops.
|
|
62
|
+
- The time limit is the sum of the members' limits.
|
|
63
|
+
|
|
64
|
+
**Fallback.** These cases go back to a normal single run (targeted run,
|
|
65
|
+
then confirmation):
|
|
66
|
+
|
|
67
|
+
- a group that failed to apply, crashed, timed out, or failed outside any
|
|
68
|
+
example (a spec file that did not load, a context hook);
|
|
69
|
+
- a failure whose example belongs to no member;
|
|
70
|
+
- a member whose only failures were infrastructure crashes.
|
|
71
|
+
|
|
72
|
+
A member that passed its targeted examples in a clean group goes straight to
|
|
73
|
+
the confirmation run over its whole resolved spec file ("direct
|
|
74
|
+
confirmation"). Its targeted examples already passed once.
|
|
75
|
+
|
|
76
|
+
Killed members are reported `killed`, with the killing example and their
|
|
77
|
+
share of the group's duration.
|
|
78
|
+
|
|
79
|
+
## How many mutants combine
|
|
80
|
+
|
|
81
|
+
Measured from the plan on each target. Coverage targeting, `--isolation fork`
|
|
82
|
+
and `--jobs 2` throughout. "Processes" counts the forked processes that
|
|
83
|
+
decide a verdict. *Ideal* assumes every group runs cleanly and only the
|
|
84
|
+
members a single run did not kill fall back.
|
|
85
|
+
|
|
86
|
+
| Target | Mutants (run) | Eligible | Main reasons not eligible | Groups (no cap) | Members grouped | Mean size | Processes: single → ideal |
|
|
87
|
+
|---|---|---|---|---|---|---|---|
|
|
88
|
+
| fixture (3 files) | 440 (440) | 369 | reported line not attributed: 71 | 83 | 343 | 4.1 | 440 → 226 (**−49%**) |
|
|
89
|
+
| dry-monads `maybe.rb` | 610 (561) | 287 | line not attributed: 274; uncovered: 49 | 46 | 267 | 5.8 | 561 → 377 (**−33%**) |
|
|
90
|
+
| httparty `httparty.rb` | 1412 (1203) | 920 | line not attributed: 283; uncovered: 209 | 157 | 818 | 5.2 | 1203 → 656 (**−45%**) |
|
|
91
|
+
| grape `declared_params_handler.rb` | 691 (691) | 618 | line not attributed: 73 | 17 | 34 | 2.0 | 691 → 677 (**−2%**) |
|
|
92
|
+
|
|
93
|
+
"Mutants" leaves out equivalent mutants (1 on dry-monads, 2 on grape), which
|
|
94
|
+
are never run. The verdict table below counts them, hence 611 and 693 there.
|
|
95
|
+
|
|
96
|
+
A cap barely changes these numbers above 8 members. With the ideal process
|
|
97
|
+
count, cap 2 / 4 / 8 / none give −19 / −27 / −31 / −33% on dry-monads and
|
|
98
|
+
−26 / −42 / −45 / −45% on httparty. Grape gains almost nothing: every
|
|
99
|
+
endpoint example of `declared_spec.rb` runs through most of the file, so
|
|
100
|
+
almost no two subjects have disjoint example sets. How much can be combined
|
|
101
|
+
depends on how narrowly the target's examples exercise it, not on the kill
|
|
102
|
+
rate.
|
|
103
|
+
|
|
104
|
+
"Line not attributed" covers mutants reported at a line that no single
|
|
105
|
+
example is credited with: method-body mutants reported at their `def`, and
|
|
106
|
+
endless `def`s. Their single run uses lexical targeting, so they cannot join
|
|
107
|
+
a group.
|
|
108
|
+
|
|
109
|
+
## Results
|
|
110
|
+
|
|
111
|
+
### Verdicts
|
|
112
|
+
|
|
113
|
+
Each mutant was matched by file, line, operator and diff against a single-run
|
|
114
|
+
control on the same tree with the same flags. The control was run 2–4 times
|
|
115
|
+
per target.
|
|
116
|
+
|
|
117
|
+
| Target | Combined runs | Divergent verdicts | Control vs control |
|
|
118
|
+
|---|---|---|---|
|
|
119
|
+
| fixture | 5 (one without direct confirmation) | **0 / 440** each | 0 |
|
|
120
|
+
| dry-monads `maybe.rb` | 3 (one without direct confirmation) | **0 / 611** each | 0 |
|
|
121
|
+
| httparty | 6 (also cap 4, distinct subjects, no direct confirmation) | **0 / 1412** each | 0 |
|
|
122
|
+
| grape | 2 | **0 / 693** each | 0 |
|
|
123
|
+
|
|
124
|
+
That is 16 combined runs and 0 divergences on about 13,900 mutant verdicts,
|
|
125
|
+
so **the gate as stated is met**.
|
|
126
|
+
|
|
127
|
+
The first dry-monads run had 16 divergences. They were not caused by
|
|
128
|
+
combining. They came from a bug in coverage targeting, which also produces
|
|
129
|
+
wrong verdicts in the control (see [Known issues found along the way](#known-issues-found-along-the-way)).
|
|
130
|
+
The harness now treats a failure outside any example as a group failure, so
|
|
131
|
+
those groups fall back, and later runs reproduce the control exactly.
|
|
132
|
+
|
|
133
|
+
### Process count and time
|
|
134
|
+
|
|
135
|
+
Durations are `summary.duration` in seconds. The two modes were alternated
|
|
136
|
+
on the same tree.
|
|
137
|
+
|
|
138
|
+
| Target | Single runs | Combined | Change | Processes (measured) |
|
|
139
|
+
|---|---|---|---|---|
|
|
140
|
+
| fixture | 53.6, 53.4, 54.8 | 50.3, 50.6, 49.9 | **−7%** | 440 → 226 |
|
|
141
|
+
| fixture, without direct confirmation | (same) | 57.2 | +6% | 440 → 226 |
|
|
142
|
+
| dry-monads `maybe.rb` | 45.3, 46.3, 46.2 | 39.4, 39.2 (40.0 without direct confirmation) | **−14%** | 561 → 532 |
|
|
143
|
+
| httparty | 83.8, 85.4, 86.3 | 55.2, 56.0, 56.4 | **−34%** | 1203 → 656 |
|
|
144
|
+
| httparty, cap 4 / distinct subjects / no direct confirmation | (same) | 59.6 / 56.8 / 56.7 | −30% / −33% / −33% | |
|
|
145
|
+
| grape | 54.6, 54.4 | 52.6, 53.4 | −3% (noise) | 691 → 677 |
|
|
146
|
+
|
|
147
|
+
Where the time goes:
|
|
148
|
+
|
|
149
|
+
- **httparty.** A killed mutant costs 128 ms on average in a single run. Of
|
|
150
|
+
that, 107 ms is `test_run`, and only 1.75 examples run per kill. The cost is
|
|
151
|
+
almost all fixed per-process work (fork, sandbox, apply, RSpec setup, reap),
|
|
152
|
+
not examples. Single runs spent 91.3 s of killtime on the 704 mutants that
|
|
153
|
+
groups killed. The 157 group runs that replaced them took 33.3 s, even
|
|
154
|
+
though they went through more examples (5841, skipped ones included, vs 1095). Grouping pays when the fixed
|
|
155
|
+
cost per process outweighs the examples it adds.
|
|
156
|
+
- **fixture.** Each example sleeps 50 ms, so examples dominate. The 83 groups
|
|
157
|
+
cost 44.9–46.0 s, against 39.2–41.0 s for the same 297 kills in single
|
|
158
|
+
runs. Process count halved, examples gone through rose from 490–514 to
|
|
159
|
+
1846–1858 (skipped ones included), and the groups cost about 14% *more*
|
|
160
|
+
than the single runs they replaced. The whole −7% comes from direct
|
|
161
|
+
confirmation: 46 group survivors skipped their targeted run (49.9–50.0 s
|
|
162
|
+
→ 37.2 s). Without it the fixture is 6% *slower*.
|
|
163
|
+
- **dry-monads.** 22 of 46 groups failed outside any example, because of the
|
|
164
|
+
shared-example targeting bug below. 9 more failed to apply: dry-monads turns
|
|
165
|
+
Ruby warnings into errors, and merged sources raised "assigned but unused
|
|
166
|
+
variable" warnings. Only 75 members were killed in groups, saving about
|
|
167
|
+
6.5 s of killtime. The rest of the measured gain was in mutants that still
|
|
168
|
+
ran singly: in all three combined runs they ran fewer examples (1006–1067
|
|
169
|
+
vs 1092–1128 for the singly killed ones), consistent with groups filling the
|
|
170
|
+
kill history first. The ideal −33% process count is out of reach until that
|
|
171
|
+
bug is fixed.
|
|
172
|
+
- **Ordering.** The kill history matters to groups. With an empty history
|
|
173
|
+
(the first prototype recorded kills only after all groups had run), the
|
|
174
|
+
fixture's group phase took 26.9 s. Recording kills as each group finishes
|
|
175
|
+
brought it down to 22.8 s.
|
|
176
|
+
|
|
177
|
+
## Failure modes
|
|
178
|
+
|
|
179
|
+
**Shared or global state between subjects (false kill, demonstrated).**
|
|
180
|
+
Disjoint example sets say nothing about state that outlives an example. In a
|
|
181
|
+
constructed project, `Writer#with_entry` deletes its key in `ensure`, and
|
|
182
|
+
`Reader#read` has a survivor (`return :none if false`). In a single run,
|
|
183
|
+
`Writer`'s mutants `STORE.delete!(key)` and `STORE.delete` are killed, but
|
|
184
|
+
they leave the key in the store. Grouped with the `Reader` survivor, the
|
|
185
|
+
leaked key makes `Reader`'s example fail. The failure is attributed to the
|
|
186
|
+
`Reader` mutant, which is reported **killed**. A single run reports it
|
|
187
|
+
survived. Two of the 109 mutants in that project diverged this way.
|
|
188
|
+
|
|
189
|
+
**Control flow into another member's code (false kill, demonstrated).** The
|
|
190
|
+
map is built from the original code. A mutant can send its example into code
|
|
191
|
+
that the example never reached before, including another member's mutated
|
|
192
|
+
code. In the same project, `Router#route` is mutated from `n > 0` to
|
|
193
|
+
`n == 0`, which routes `route(5)` into `Helper#label`. On its own the mutant
|
|
194
|
+
survives, because the original `label(5)` returns `:positive`. Grouped with a
|
|
195
|
+
`Helper#label` mutant that returns `nil`, `Router`'s example fails, and the
|
|
196
|
+
`Router` mutant is reported killed. One more divergence. The
|
|
197
|
+
distinct-subjects rule does not prevent either case: the members here are in
|
|
198
|
+
different classes.
|
|
199
|
+
|
|
200
|
+
Cheaper static guards do not make the method sound either. Requiring the
|
|
201
|
+
members to be in different files excludes neither case: move `STORE`,
|
|
202
|
+
`Writer` and `Reader` into three files, or `Router` and `Helper` into two,
|
|
203
|
+
and the mechanism is the same. Requiring that no two members' bodies name a
|
|
204
|
+
common constant or each other's classes does exclude both constructed cases
|
|
205
|
+
(`Writer` and `Reader` both name `STORE`, `Router` names `Helper`), but one
|
|
206
|
+
step of indirection gets past it: `Writer` and `Reader` reaching the store
|
|
207
|
+
through two different helper classes, or `Router` calling a `Service` that
|
|
208
|
+
calls `Helper`. (These variants were not run; the failing example and its
|
|
209
|
+
attribution are the same as in the originals.) A static rule cannot follow indirect call chains, dynamic dispatch,
|
|
210
|
+
instance variables on shared objects, globals, `ENV`, the filesystem or a
|
|
211
|
+
database.
|
|
212
|
+
|
|
213
|
+
Both cases produce a false kill, which is the worst error for a mutation
|
|
214
|
+
tool to make: it hides a gap in the tests, and no reader can reproduce it.
|
|
215
|
+
The corpus has none, but nothing in the method rules them out. A check that
|
|
216
|
+
each failing example executed only its own member's mutated lines would
|
|
217
|
+
catch the second case. It would not catch the first: the reader's example
|
|
218
|
+
runs only reader code.
|
|
219
|
+
|
|
220
|
+
**Load order and constants redefined, several mutants at once.** Each file
|
|
221
|
+
of a group is evaluated once, with its members merged, through the normal
|
|
222
|
+
`MutationApplier` (constant pinning, concern cleanup, redefinition
|
|
223
|
+
recovery). Groups spanning the fixture's three files gave identical verdicts.
|
|
224
|
+
The risk shows up in projects that turn warnings into errors. A merged source
|
|
225
|
+
can raise a warning that none of its members raises alone (dry-monads: 9 of
|
|
226
|
+
46 groups). The harness falls back on any apply error, so the cost is time,
|
|
227
|
+
not correctness. A single mutant whose load crashes is reported killed
|
|
228
|
+
(`load_crashed`). Inside a group the same crash only causes a fallback.
|
|
229
|
+
|
|
230
|
+
**Crashes, timeouts and infinite loops.** A group limited by the sum of its
|
|
231
|
+
members' limits loses all of that time when one member loops, and then every
|
|
232
|
+
member runs again on its own. In the constructed project, a `Looper` mutant
|
|
233
|
+
times out after 12 s on its own. Five groups containing one timed out after
|
|
234
|
+
30–50 s each, before their members re-ran. The run took **128 s instead of
|
|
235
|
+
39 s**. A tighter group limit (the largest member's limit plus a margin)
|
|
236
|
+
would cut the waste but risks timing out a slow but healthy group. The
|
|
237
|
+
corpus had no timeouts.
|
|
238
|
+
|
|
239
|
+
**Stop on kill.** A single run stops at its first decisive failure. A group
|
|
240
|
+
cannot stop at the first failure without losing the other members' verdicts,
|
|
241
|
+
so the prototype stops per member: it skips the remaining examples of a
|
|
242
|
+
killed member, and stops the run once all members are killed. A survivor in
|
|
243
|
+
a group still runs all of its targeted examples, and with that group's
|
|
244
|
+
duration, then pays its confirmation run as well. On the fixture, groups went
|
|
245
|
+
through 3.7 times as many examples (skipped ones included) as single runs did
|
|
246
|
+
for the same kills.
|
|
247
|
+
|
|
248
|
+
**Killer-first ordering.** Groups need a filled kill history as much as
|
|
249
|
+
single runs do (see the fixture numbers above). The history fills while
|
|
250
|
+
groups run, so the order in which groups are dispatched matters too.
|
|
251
|
+
|
|
252
|
+
**`uncovered` and probe-first.** Mutants given a verdict without a run are
|
|
253
|
+
never grouped. Probe-first was not measured. It calls `run_batch` once per
|
|
254
|
+
batch (sentinels first), and grouping would work within each batch.
|
|
255
|
+
|
|
256
|
+
**Incremental cache.** The prototype ran with the cache off. A real
|
|
257
|
+
implementation would keep cached mutants out of groups, and would store each
|
|
258
|
+
group kill per member, with its killing example. That kill depends on which
|
|
259
|
+
mutants ran beside it, which the cache key does not record.
|
|
260
|
+
|
|
261
|
+
**The map itself.** Attribution is only as sound as the map. A stale map
|
|
262
|
+
must never be used: `MapStore` invalidates by file digest, and the run
|
|
263
|
+
fingerprint covers the helpers. Coverage from code the tests run in a
|
|
264
|
+
forked subprocess is lost, so such examples are missing from affecting sets.
|
|
265
|
+
Shared examples are recorded under the shared file (below).
|
|
266
|
+
|
|
267
|
+
## Known issues found along the way
|
|
268
|
+
|
|
269
|
+
Both are bugs in coverage targeting as it ships, independent of combined
|
|
270
|
+
runs, and both are filed.
|
|
271
|
+
|
|
272
|
+
- **Coverage targeting runs shared examples by their definition file**
|
|
273
|
+
([#1973](https://github.com/marinazzio/evilution/issues/1973)). The
|
|
274
|
+
map credits examples defined in a shared group to that group's own
|
|
275
|
+
location (dry-monads: `spec/shared/applicative.rb:5`). A targeted run
|
|
276
|
+
passes that location to RSpec, which loads the shared file on its own,
|
|
277
|
+
redefines the shared group and runs nothing. On dry-monads, where warnings
|
|
278
|
+
are errors, this reports **30 mutants of `maybe.rb` killed with 0 examples
|
|
279
|
+
run**, and 11 more as errors, in the plain single-run control. Lexical
|
|
280
|
+
targeting reports 10 of the line-92 mutants as survivors. In a project
|
|
281
|
+
where the shared file loads cleanly on its own, the targeted run runs
|
|
282
|
+
nothing and the confirmation run over the whole spec file still decides,
|
|
283
|
+
at the cost of a wasted run.
|
|
284
|
+
- **Coverage targeting without `--preload` falls back silently**
|
|
285
|
+
([#1974](https://github.com/marinazzio/evilution/issues/1974)). On the
|
|
286
|
+
fixture, the map builder child raised `NameError: uninitialized constant
|
|
287
|
+
RSpec` (in `MapBuilder#reset_world`). The parent printed the backtrace,
|
|
288
|
+
wrote no map, and continued with lexical targeting. Passing
|
|
289
|
+
`--preload spec/spec_helper.rb` avoids it.
|
|
290
|
+
|
|
291
|
+
## Decision
|
|
292
|
+
|
|
293
|
+
**No-go** for now, neither as a default nor as an opt-in, for four reasons:
|
|
294
|
+
|
|
295
|
+
1. **The method is not sound, and the gate does not show that.** Zero
|
|
296
|
+
divergences on the corpus is the stated gate, and it was met in 16 runs.
|
|
297
|
+
But two mechanisms that turn a survivor into a kill (shared state between
|
|
298
|
+
examples, control flow into another member's code) were shown on a
|
|
299
|
+
constructed project. The coverage map cannot see either one, and the
|
|
300
|
+
result is a false kill, which no reader can reproduce.
|
|
301
|
+
2. **The gain depends on the target.** −34% on httparty, where a kill costs
|
|
302
|
+
more in process overhead than in examples. −7% on the fixture, all of it
|
|
303
|
+
from skipping survivors' targeted runs; the groups themselves cost more
|
|
304
|
+
than the single runs they replaced. −3% (noise) on grape, whose
|
|
305
|
+
examples cover almost the whole file. −14% on dry-monads, where most
|
|
306
|
+
groups fell back.
|
|
307
|
+
3. **Failures are expensive.** A looping member costs its group the sum of
|
|
308
|
+
the members' limits, and then everyone runs again (39 s → 128 s on the
|
|
309
|
+
constructed project). Merged sources raise warnings that no single
|
|
310
|
+
mutant raises, and in a project that turns warnings into errors the whole
|
|
311
|
+
group falls back.
|
|
312
|
+
4. **It is RSpec and coverage-targeting only,** and the shared-example bug
|
|
313
|
+
above currently makes coverage targeting give wrong verdicts on one of
|
|
314
|
+
the corpus targets.
|
|
315
|
+
|
|
316
|
+
What would make it worth revisiting:
|
|
317
|
+
|
|
318
|
+
- the shared-example targeting bug fixed;
|
|
319
|
+
- an attribution check: kill a member only if every failing example
|
|
320
|
+
executed that member's mutated lines and no other member's, which closes
|
|
321
|
+
the control-flow case;
|
|
322
|
+
- groups limited to members whose examples share no mutable state. Nothing
|
|
323
|
+
measured here can tell that; marking specs that run in random order and
|
|
324
|
+
isolation is the closest available signal;
|
|
325
|
+
- a group time limit tighter than the sum of the members' limits.
|
|
326
|
+
|
|
327
|
+
Even then, a group kill would be the only verdict evilution gives that a
|
|
328
|
+
single run cannot reproduce. Re-running each group kill alone to check it
|
|
329
|
+
would cost more than grouping saves: on httparty, 704 re-runs against 547
|
|
330
|
+
processes saved (1203 → 656), each re-run mostly the fixed per-process cost
|
|
331
|
+
(128 ms per kill, 1.75 examples, on average), so the combined run would be
|
|
332
|
+
slower than single runs.
|
|
333
|
+
|
|
334
|
+
## Reproducing
|
|
335
|
+
|
|
336
|
+
The harness, the comparison scripts and the constructed project are in
|
|
337
|
+
[`scripts/combined_runs_spike/`](../scripts/combined_runs_spike/), whose
|
|
338
|
+
README covers setup and running. They are measurement code, not loaded by
|
|
339
|
+
the gem.
|
|
340
|
+
|
|
341
|
+
Corpus: local copies of the fixture (`spec/support/fixtures/benchmark_project`),
|
|
342
|
+
of the constructed project (`scripts/combined_runs_spike/fm`), and of the
|
|
343
|
+
dry-monads, httparty and grape targets at the commits pinned in
|
|
344
|
+
`scripts/benchmark_perf.yml`. Each copy has a Gemfile adding
|
|
345
|
+
`gem "evilution", path: <tree>`. In httparty, the cwd-relative fixture paths
|
|
346
|
+
are made absolute (`spec/spec_helper.rb` `file_fixture`, and the
|
|
347
|
+
`tiny.gif` path in `spec/httparty_spec.rb`), because mutation processes run
|
|
348
|
+
in a sandbox directory.
|
|
349
|
+
|
|
350
|
+
Harness (`combined.rb`), loaded with `RUBYOPT=-r<file>` (it requires
|
|
351
|
+
`bundler/setup` and `evilution` first):
|
|
352
|
+
|
|
353
|
+
- A module prepended to `Evilution::Runner::MutationExecutor` whose private
|
|
354
|
+
`run_batch(notifier, pipeline, mutations, baseline_result, integration)`
|
|
355
|
+
plans the groups, as described under [Prototype](#prototype). It reads the
|
|
356
|
+
map from the integration's `CoverageExampleFilter`, and it decides
|
|
357
|
+
eligibility from `Coverage::Map#examples_for` / `#executed?` over
|
|
358
|
+
`mutation.changed_lines`. Group runs are forked by the harness, `--jobs`
|
|
359
|
+
at a time. Then `super` handles the rest, and the two `ExecutionResult`s
|
|
360
|
+
are merged.
|
|
361
|
+
- Merging within a file: each member's change is the common prefix and
|
|
362
|
+
suffix of `restore_source` against `eval_source` (and of
|
|
363
|
+
`original_source` against `mutated_source`). Non-touching ranges are
|
|
364
|
+
spliced into one `Evilution::Mutation`, which the integration's
|
|
365
|
+
`MutationApplier` evaluates.
|
|
366
|
+
- A module prepended to `Evilution::Integration::RSpec` whose `run_tests`
|
|
367
|
+
goes straight to `confirm_run` for members that passed in a clean group.
|
|
368
|
+
- `EV_COMBINED=1` runs groups (otherwise plan only). `EV_COMBINED_CAP`,
|
|
369
|
+
`EV_COMBINED_DISTINCT=1` and `EV_COMBINED_DIRECT_CONFIRM=0` select the
|
|
370
|
+
variants. `EV_COMBINED_OUT` writes the plan, the per-cap groupings and the
|
|
371
|
+
per-group outcomes as JSON.
|
|
372
|
+
- Verdict equivalence (`div.rb`): match the two runs' JSON entries by file,
|
|
373
|
+
line, operator and diff, and compare statuses. `analyze.rb` gives the
|
|
374
|
+
group-phase costs under [Process count and time](#process-count-and-time),
|
|
375
|
+
`ideal.rb` the ideal process counts, and `rest.rb` the cost of the mutants
|
|
376
|
+
left to single runs.
|
|
377
|
+
|
|
378
|
+
Commands (in the target dir; the harness via `RUBYOPT`). `run.sh` wraps them:
|
|
379
|
+
|
|
380
|
+
```
|
|
381
|
+
# fixture
|
|
382
|
+
bundle exec evilution run lib/pricing.rb lib/inventory.rb lib/grade.rb --preload spec/spec_helper.rb \
|
|
383
|
+
--example-targeting coverage --isolation fork --jobs 2 --format json
|
|
384
|
+
# dry-monads
|
|
385
|
+
bundle exec evilution run lib/dry/monads/maybe.rb --spec spec/unit/maybe_spec.rb --integration rspec \
|
|
386
|
+
--preload spec/spec_helper.rb --example-targeting coverage --isolation fork --jobs 2 --format json
|
|
387
|
+
# httparty
|
|
388
|
+
bundle exec evilution run lib/httparty.rb --spec spec/httparty_spec.rb --integration rspec \
|
|
389
|
+
--preload spec/spec_helper.rb --example-targeting coverage --isolation fork --jobs 2 --format json
|
|
390
|
+
# grape
|
|
391
|
+
bundle exec evilution run lib/grape/declared_params_handler.rb --spec spec/grape/endpoint/declared_spec.rb \
|
|
392
|
+
--integration rspec --preload spec/spec_helper.rb --example-targeting coverage --isolation fork \
|
|
393
|
+
--jobs 2 --format json
|
|
394
|
+
```
|
|
395
|
+
|
|
396
|
+
The constructed project behind the failure modes
|
|
397
|
+
(`scripts/combined_runs_spike/fm`) is one file:
|
|
398
|
+
|
|
399
|
+
- a `STORE` hash;
|
|
400
|
+
- `Writer#with_entry` (set the key, yield, delete it in `ensure`);
|
|
401
|
+
- `Reader#read` (with an untested `nil` guard);
|
|
402
|
+
- `Router#route` (`n > 0 ? :positive : Helper.new.label(n)`);
|
|
403
|
+
- `Helper#label` (`n.positive? ? :positive : :other`);
|
|
404
|
+
- `Looper#count` (`i += 1 while i < n`).
|
|
405
|
+
|
|
406
|
+
It has one example per subject, and two for `Helper`. i18n and Redmine were
|
|
407
|
+
left out: they are Minitest, and the coverage map is RSpec only.
|
data/docs/integrations.md
CHANGED
|
@@ -129,7 +129,7 @@ The test-unit collaborators are:
|
|
|
129
129
|
|
|
130
130
|
- `framework_loader.rb` — `require "test-unit"` + disables the at_exit
|
|
131
131
|
auto-run handler so it doesn't fire on evilution exit.
|
|
132
|
-
- `subject_class_registry.rb` —
|
|
132
|
+
- `subject_class_registry.rb` — class-tree tracking of newly-loaded
|
|
133
133
|
`Test::Unit::TestCase` subclasses (test-unit has no public
|
|
134
134
|
registry-clear analog to `Minitest::Runnable.runnables`).
|
|
135
135
|
- `dispatcher.rb` — assembles a `Test::Unit::TestSuite` from the new
|
data/docs/isolation.md
CHANGED
|
@@ -144,6 +144,27 @@ creates and removes their methods when the file is evaluated next, so that
|
|
|
144
144
|
method lookup passes through them. Only modules are emptied, not anonymous
|
|
145
145
|
classes, and only the methods written in that file.
|
|
146
146
|
|
|
147
|
+
To see those modules being created without walking the heap (which touches
|
|
148
|
+
every object in the process, twice per evaluation), an `in_process` run
|
|
149
|
+
prepends a hook to `Module#initialize` and `Module#initialize_copy` (for
|
|
150
|
+
copies made with `dup` or `clone`) the first time it evaluates a file. It
|
|
151
|
+
records nothing outside an evaluation. Likewise, a run that loads
|
|
152
|
+
`ActiveSupport::Concern` modules prepends a hook to the concern module's
|
|
153
|
+
`extended`, so the concerns it must clear before each evaluation are listed
|
|
154
|
+
as they are defined. The heap is walked once per process for the ones that
|
|
155
|
+
already existed: in the parent when ActiveSupport is already loaded there
|
|
156
|
+
(by a preload, for instance), otherwise once in each mutation process.
|
|
157
|
+
|
|
158
|
+
A mutant can also freeze a class or module its file declares (`EMPTY =
|
|
159
|
+
Bag.freeze` where the original builds `Bag.new([].freeze).freeze`), and Ruby
|
|
160
|
+
cannot thaw it. Evilution still reports that mutant: it checks the original
|
|
161
|
+
against a fresh copy of the class, then puts the frozen class back, since
|
|
162
|
+
other files may reopen the class or include the module and keep referring to
|
|
163
|
+
the frozen one. Under `in_process` the class therefore stays frozen for the
|
|
164
|
+
rest of the run, and the later mutants of that file are reported as errors.
|
|
165
|
+
A `--isolation fork` run judges them, since every mutant there starts from a
|
|
166
|
+
process the freeze never touched.
|
|
167
|
+
|
|
147
168
|
Other things a file can add to at load time are not undone: an `at_exit`
|
|
148
169
|
hook, a `Signal.trap`, a `TracePoint`, a callback pushed into a registry of
|
|
149
170
|
some library. If the file you are mutating does one of those at the top
|
|
@@ -293,14 +314,54 @@ mutation subprocess and any grandchildren it spawned rather than orphaning them,
|
|
|
293
314
|
and a terminal interrupt (Ctrl-C) is forwarded to every worker group — an
|
|
294
315
|
aborted parallel run leaves no stray worker or mutation processes behind.
|
|
295
316
|
|
|
317
|
+
### One database per worker
|
|
318
|
+
|
|
319
|
+
A preloaded Rails app is booted, and its `database.yml` resolved, in the
|
|
320
|
+
parent, so every worker inherits a connection to the same test database. When
|
|
321
|
+
that is a SQLite file, workers would load fixtures into and write to one
|
|
322
|
+
database at once, and a test could fail on another worker's data: a kill that
|
|
323
|
+
depends on timing, not on the mutation. Right after the fork, before its first
|
|
324
|
+
mutation, each worker therefore runs `Evilution::WorkerDatabase`:
|
|
325
|
+
|
|
326
|
+
1. it re-renders `database.yml` with its own `TEST_ENV_NUMBER` and, when that
|
|
327
|
+
names a different, prepared database, reconnects `ActiveRecord::Base` to
|
|
328
|
+
it (the `parallel_tests` convention);
|
|
329
|
+
2. otherwise, with `copy_worker_databases` on (the default), it copies the
|
|
330
|
+
preloaded SQLite database through SQLite's backup API into a directory of
|
|
331
|
+
its own (`WorkerDatabase::Copies`) and reconnects to the copy.
|
|
332
|
+
|
|
333
|
+
The per-mutation children forked by the worker inherit the worker's
|
|
334
|
+
connection settings, so all of a worker's mutations run against its database.
|
|
335
|
+
The parent picks the copies directory under the system temp dir before the
|
|
336
|
+
workers fork and removes it when `WorkQueue#map` returns or raises (an
|
|
337
|
+
interrupt reaches the parent as an exception, so it is covered). A worker
|
|
338
|
+
that is recycled, times out or dies is reaped first, and its subdirectory is
|
|
339
|
+
deleted before its replacement forks, so frequent replacement never holds
|
|
340
|
+
more than one copy per worker slot on disk. The
|
|
341
|
+
directory is deliberately not registered with `TempDirTracker`: every forked
|
|
342
|
+
process inherits that tracker's `at_exit` cleanup, and a mutation child dying
|
|
343
|
+
through it would delete the databases of workers still running. See the
|
|
344
|
+
README's "Parallel Runs with SQLite" for what is and is not covered.
|
|
345
|
+
|
|
296
346
|
## Related flags
|
|
297
347
|
|
|
298
348
|
- `--timeout N` sets the per-mutation time limit. Under `fork`, this drives
|
|
299
349
|
SIGKILL. Under `in_process`, this drives `Timeout.timeout` and is subject
|
|
300
|
-
to the interrupt-mask hazard described above.
|
|
350
|
+
to the interrupt-mask hazard described above. Without it each mutation's
|
|
351
|
+
limit is derived from the baseline time of its spec files
|
|
352
|
+
(`--timeout-factor`, `--timeout-floor`, `--timeout-ceiling`; see the README's
|
|
353
|
+
"Per-mutation timeout"). A mutation that uses up such a limit while it is
|
|
354
|
+
shorter than the fixed one is run again under the fixed limit before a
|
|
355
|
+
timeout is reported, and the parallel pool's per-worker deadline is twice
|
|
356
|
+
the longest time a mutation of the run may take, both runs together.
|
|
301
357
|
- `--jobs N` runs N workers in parallel. The parallel pool respects the
|
|
302
358
|
configured isolation strategy, so `--jobs 4 --isolation fork` uses fork
|
|
303
|
-
isolation per-mutation inside each worker
|
|
359
|
+
isolation per-mutation inside each worker; those workers live for the
|
|
360
|
+
whole run, each taking the next mutation as soon as it is done with the
|
|
361
|
+
last. Under `--isolation in_process` the mutation runs in the worker
|
|
362
|
+
itself, so each worker runs one mutation and is replaced by a fresh one
|
|
363
|
+
forked from the clean parent: nothing one mutation leaves behind reaches
|
|
364
|
+
the next.
|
|
304
365
|
- `--[no-]canary` toggles the proof-of-life pre-flight check (see above).
|
|
305
366
|
|
|
306
367
|
[rails-handle-interrupt]: https://github.com/rails/rails/blob/main/activesupport/lib/active_support/concurrency/thread_monitor.rb
|
|
@@ -15,7 +15,7 @@ precisely so you can line the two tools up and see the difference for yourself.
|
|
|
15
15
|
|
|
16
16
|
| mutant | Evilution |
|
|
17
17
|
| --- | --- |
|
|
18
|
-
| `mutant run --use rspec -- 'MyApp::Foo*'` | `evilution run lib/my_app
|
|
18
|
+
| `mutant run --use rspec -- 'MyApp::Foo*'` | `evilution run lib/my_app/**/*.rb --target 'MyApp::Foo*'` |
|
|
19
19
|
| `mutant.yml` | `.evilution.yml` (run `evilution init` to scaffold) |
|
|
20
20
|
| `mutant-rspec` / `mutant-minitest` gems | built in: `--integration rspec\|minitest\|test-unit` |
|
|
21
21
|
| "mutation coverage" %, "alive" | "mutation score" 0.0–1.0, "survived" |
|
|
@@ -51,8 +51,9 @@ bundle exec mutant run --use rspec -- 'MyApp::Calculator#add'
|
|
|
51
51
|
# evilution — point at the file, optionally filter to the method
|
|
52
52
|
bundle exec evilution run lib/my_app/calculator.rb --target 'MyApp::Calculator#add'
|
|
53
53
|
|
|
54
|
-
#
|
|
55
|
-
|
|
54
|
+
# every file of a directory (file arguments must be files, so let the shell
|
|
55
|
+
# expand the glob; bash needs `shopt -s globstar`), then filter by expression
|
|
56
|
+
bundle exec evilution run lib/**/*.rb --target 'MyApp::Calculator*'
|
|
56
57
|
```
|
|
57
58
|
|
|
58
59
|
`--target` accepts the expression shapes you are used to, plus a source glob:
|
|
@@ -81,13 +82,17 @@ bundle exec evilution run lib/ --target 'MyApp::Calculator*'
|
|
|
81
82
|
| JSON report | session JSON (always written) | `--format json` (`--save-session` to persist) |
|
|
82
83
|
| Inline opt-out | `# mutant:disable` | `# evilution:disable` |
|
|
83
84
|
| Aggressive operators | _(always on)_ | `--profile strict` (or `--strict`) |
|
|
85
|
+
| Changed code only | `--since <git-ref>` | `--since <git-ref>` (lines changed since the ref's merge base with `HEAD`, uncommitted changes included) |
|
|
84
86
|
|
|
85
|
-
###
|
|
87
|
+
### `--since` differences
|
|
86
88
|
|
|
87
|
-
-
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
|
|
89
|
+
- mutant selects the **subjects** whose source changed and runs all their
|
|
90
|
+
mutations; Evilution runs only the mutations that change a changed line (a
|
|
91
|
+
subject is picked when any of its lines changed, then its unchanged lines are left
|
|
92
|
+
alone). Expect fewer mutations for the same diff.
|
|
93
|
+
- Evilution diffs to the working tree, so uncommitted and untracked files count.
|
|
94
|
+
- Without file arguments Evilution takes the changed `.rb` files under `lib/` and
|
|
95
|
+
`app/`; `--target source:<glob>` cannot be combined with `--since`.
|
|
91
96
|
|
|
92
97
|
### Things Evilution adds
|
|
93
98
|
|
|
@@ -193,7 +198,7 @@ whether the same code is being killed:
|
|
|
193
198
|
|
|
194
199
|
```sh
|
|
195
200
|
# produce an evilution report
|
|
196
|
-
bundle exec evilution run lib
|
|
201
|
+
bundle exec evilution run --target 'source:lib/**/*.rb' --format json --save-session
|
|
197
202
|
|
|
198
203
|
# line it up against your last mutant session JSON
|
|
199
204
|
bundle exec evilution compare \
|
|
@@ -213,8 +218,8 @@ operator.
|
|
|
213
218
|
- **Different mutations.** Evilution ships its own operator registry (`--profile
|
|
214
219
|
default` / `strict`); it is not mutant's AST-handler set. Counts and specific
|
|
215
220
|
survivors will differ. Use `compare` to quantify.
|
|
216
|
-
- **
|
|
217
|
-
|
|
221
|
+
- **Diff-based selection is line-based** (`--since <ref>`): see the `--since`
|
|
222
|
+
differences above.
|
|
218
223
|
- **File-driven, not subject-driven.** Always pass paths; narrow with `--target`.
|
|
219
224
|
- **100%-coverage culture.** Evilution does not assume a 100% gate; choose `min_score`
|
|
220
225
|
deliberately, and note that `:unresolved` mutations are excluded from the score
|