evilution 1.5.0 → 1.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.beads/interactions.jsonl +1 -0
- data/.rubocop_todo.yml +30 -0
- data/CHANGELOG.md +59 -0
- data/README.md +285 -45
- data/docs/architecture.md +110 -12
- data/docs/ast_pattern_syntax.md +35 -7
- data/docs/combined_runs_spike.md +407 -0
- data/docs/integrations.md +1 -1
- data/docs/isolation.md +63 -2
- data/docs/migration-from-mutant.md +16 -11
- data/docs/mutant_schemata_spike.md +387 -0
- data/docs/operator_yield_study.md +576 -0
- data/docs/probe_first_spike.md +264 -0
- data/lib/evilution/ast/ancestor_lookup.rb +36 -0
- data/lib/evilution/ast/callback_declaration.rb +36 -3
- data/lib/evilution/ast/class_body.rb +62 -16
- data/lib/evilution/ast/def_region.rb +130 -0
- data/lib/evilution/ast/local_declaration.rb +37 -0
- data/lib/evilution/ast/parent_index.rb +50 -0
- data/lib/evilution/ast/parse_cache.rb +27 -0
- data/lib/evilution/ast/parser.rb +8 -3
- data/lib/evilution/ast/pattern/filter.rb +53 -12
- data/lib/evilution/ast/skip_filter.rb +83 -0
- data/lib/evilution/ast/skip_sites/debug_output.rb +67 -0
- data/lib/evilution/ast/skip_sites/error_messages.rb +74 -0
- data/lib/evilution/ast/skip_sites/logging.rb +93 -0
- data/lib/evilution/ast/skip_sites/message.rb +44 -0
- data/lib/evilution/ast/skip_sites/nodes.rb +105 -0
- data/lib/evilution/ast/skip_sites/presets.rb +31 -0
- data/lib/evilution/ast/skip_sites/purity.rb +69 -0
- data/lib/evilution/ast/skip_sites/requires.rb +40 -0
- data/lib/evilution/ast/skip_sites.rb +9 -0
- data/lib/evilution/ast/source_surgeon.rb +20 -2
- data/lib/evilution/ast/syntax_check.rb +10 -8
- data/lib/evilution/baseline/report.rb +11 -2
- data/lib/evilution/baseline.rb +31 -6
- data/lib/evilution/cache/keys.rb +140 -0
- data/lib/evilution/cache/run_fingerprint.rb +51 -0
- data/lib/evilution/cache/skeleton.rb +57 -0
- data/lib/evilution/cache/spec_digest.rb +39 -0
- data/lib/evilution/cache.rb +145 -42
- data/lib/evilution/class_tree.rb +38 -0
- data/lib/evilution/cli/command.rb +12 -4
- data/lib/evilution/cli/commands/run.rb +12 -3
- data/lib/evilution/cli/commands/subjects.rb +5 -2
- data/lib/evilution/cli/parser/options_builder.rb +67 -3
- data/lib/evilution/cli/printers/environment.rb +34 -4
- data/lib/evilution/cli/printers/session_detail.rb +20 -3
- data/lib/evilution/compare/categorizer.rb +2 -1
- data/lib/evilution/compare/normalizer.rb +3 -2
- data/lib/evilution/config/sources.rb +3 -0
- data/lib/evilution/config/validators/base.rb +9 -0
- data/lib/evilution/config/validators/since.rb +22 -0
- data/lib/evilution/config/validators/skip_sites.rb +24 -0
- data/lib/evilution/config/validators/timeout_ceiling.rb +10 -0
- data/lib/evilution/config/validators/timeout_factor.rb +10 -0
- data/lib/evilution/config/validators/timeout_floor.rb +10 -0
- data/lib/evilution/config.rb +122 -6
- data/lib/evilution/coverage/load_tracker.rb +50 -0
- data/lib/evilution/coverage/map.rb +42 -5
- data/lib/evilution/coverage/map_builder.rb +123 -22
- data/lib/evilution/coverage/map_store.rb +15 -5
- data/lib/evilution/coverage/recorder.rb +42 -3
- data/lib/evilution/coverage_example_filter.rb +35 -0
- data/lib/evilution/equivalent/heuristic/comment_marking.rb +14 -1
- data/lib/evilution/example_filter.rb +20 -0
- data/lib/evilution/git/changed_lines.rb +103 -0
- data/lib/evilution/git/since_scope.rb +77 -0
- data/lib/evilution/git/unified_diff.rb +68 -0
- data/lib/evilution/infra_crash.rb +26 -0
- data/lib/evilution/integration/base.rb +97 -6
- data/lib/evilution/integration/crash_detector.rb +46 -6
- data/lib/evilution/integration/known_failures.rb +4 -0
- data/lib/evilution/integration/loading/body_call_neutralizer.rb +49 -11
- data/lib/evilution/integration/loading/callback_equivalence.rb +62 -0
- data/lib/evilution/integration/loading/callback_redeclaration.rb +151 -36
- data/lib/evilution/integration/loading/concern_redeclaration.rb +16 -12
- data/lib/evilution/integration/loading/concern_registry.rb +87 -0
- data/lib/evilution/integration/loading/concern_state_cleaner.rb +19 -8
- data/lib/evilution/integration/loading/created_modules.rb +56 -0
- data/lib/evilution/integration/loading/frozen_namespaces.rb +80 -0
- data/lib/evilution/integration/loading/mutation_applier.rb +42 -11
- data/lib/evilution/integration/loading/neutralization.rb +41 -0
- data/lib/evilution/integration/loading/redefinition_recovery.rb +5 -3
- data/lib/evilution/integration/loading/superseded_modules.rb +22 -19
- data/lib/evilution/integration/minitest/stop_hook.rb +57 -0
- data/lib/evilution/integration/minitest.rb +22 -2
- data/lib/evilution/integration/minitest_crash_detector.rb +19 -0
- data/lib/evilution/integration/rspec/baseline_runner.rb +3 -2
- data/lib/evilution/integration/rspec/crash_detector_lifecycle.rb +6 -1
- data/lib/evilution/integration/rspec/example_filter_applier.rb +16 -0
- data/lib/evilution/integration/rspec/example_ids.rb +23 -0
- data/lib/evilution/integration/rspec/example_order.rb +86 -0
- data/lib/evilution/integration/rspec/killer_first.rb +72 -0
- data/lib/evilution/integration/rspec/result_builder.rb +7 -0
- data/lib/evilution/integration/rspec/state_guard/example_group_descendants.rb +44 -0
- data/lib/evilution/integration/rspec/state_guard.rb +2 -2
- data/lib/evilution/integration/rspec/target_resolver.rb +42 -0
- data/lib/evilution/integration/rspec/test_file_resolver.rb +4 -2
- data/lib/evilution/integration/rspec.rb +98 -47
- data/lib/evilution/integration/stop_policy.rb +49 -0
- data/lib/evilution/integration/test_unit/dispatcher.rb +20 -1
- data/lib/evilution/integration/test_unit/subject_class_registry.rb +6 -1
- data/lib/evilution/integration/test_unit.rb +28 -2
- data/lib/evilution/isolation/fork.rb +58 -13
- data/lib/evilution/isolation/in_process.rb +12 -1
- data/lib/evilution/line_set.rb +72 -0
- data/lib/evilution/mcp/info_tool/actions/environment.rb +12 -16
- data/lib/evilution/mcp/info_tool/actions/subjects.rb +3 -5
- data/lib/evilution/mcp/info_tool/status_glossary.rb +15 -2
- data/lib/evilution/mcp/info_tool.rb +24 -17
- data/lib/evilution/mcp/mutate_tool/config_builder.rb +13 -1
- data/lib/evilution/mcp/mutate_tool/option_parser.rb +2 -2
- data/lib/evilution/mcp/mutate_tool/progress_streamer.rb +1 -1
- data/lib/evilution/mcp/mutate_tool/report_trimmer.rb +28 -5
- data/lib/evilution/mcp/mutate_tool/survived_enricher.rb +12 -7
- data/lib/evilution/mcp/mutate_tool.rb +32 -7
- data/lib/evilution/mcp/session_tool.rb +1 -2
- data/lib/evilution/mcp.rb +4 -0
- data/lib/evilution/mutation/line_diff.rb +111 -0
- data/lib/evilution/mutation.rb +17 -13
- data/lib/evilution/mutator/base.rb +121 -98
- data/lib/evilution/mutator/constant_value_subjects.rb +29 -1
- data/lib/evilution/mutator/line_range_filter.rb +27 -0
- data/lib/evilution/mutator/load_sources.rb +88 -0
- data/lib/evilution/mutator/operator/alias_removal.rb +2 -5
- data/lib/evilution/mutator/operator/argument_method_call_replacement.rb +3 -0
- data/lib/evilution/mutator/operator/attr_accessor_swap.rb +2 -5
- data/lib/evilution/mutator/operator/constant_namespace_strip.rb +3 -3
- data/lib/evilution/mutator/operator/freeze_removal.rb +29 -11
- data/lib/evilution/mutator/operator/method_call_removal.rb +3 -1
- data/lib/evilution/mutator/operator/mixin_removal.rb +2 -5
- data/lib/evilution/mutator/operator/superclass_removal.rb +2 -5
- data/lib/evilution/mutator/operator/symbol_literal.rb +1 -1
- data/lib/evilution/mutator/operator/visibility_removal.rb +2 -5
- data/lib/evilution/mutator/registry.rb +11 -5
- data/lib/evilution/mutator/restore_source_cache.rb +41 -0
- data/lib/evilution/parallel/pool.rb +11 -4
- data/lib/evilution/parallel/work_queue/dispatcher/result_stream.rb +40 -0
- data/lib/evilution/parallel/work_queue/dispatcher.rb +20 -7
- data/lib/evilution/parallel/work_queue/worker/loop.rb +7 -2
- data/lib/evilution/parallel/work_queue/worker.rb +10 -3
- data/lib/evilution/parallel/work_queue/worker_stat.rb +7 -0
- data/lib/evilution/parallel/work_queue.rb +69 -12
- data/lib/evilution/parallel_db_warning.rb +39 -12
- data/lib/evilution/process_supervisor.rb +22 -2
- data/lib/evilution/reporter/cli/item_formatters/coverage_gap.rb +13 -2
- data/lib/evilution/reporter/cli/item_formatters/subject_score.rb +5 -1
- data/lib/evilution/reporter/cli/line_formatters/mutations.rb +9 -2
- data/lib/evilution/reporter/cli/line_formatters/result_line.rb +9 -1
- data/lib/evilution/reporter/cli/line_formatters/skipped_sites.rb +14 -0
- data/lib/evilution/reporter/cli/line_formatters/uncovered_code.rb +17 -0
- data/lib/evilution/reporter/cli/metrics_block.rb +4 -0
- data/lib/evilution/reporter/cli/trailer.rb +4 -4
- data/lib/evilution/reporter/cli.rb +9 -2
- data/lib/evilution/reporter/html/assets/style.css +8 -0
- data/lib/evilution/reporter/html/sections/file_section.rb +9 -0
- data/lib/evilution/reporter/html/sections/uncovered_details.rb +27 -0
- data/lib/evilution/reporter/html/templates/file_section.html.erb +2 -1
- data/lib/evilution/reporter/html/templates/summary_cards.html.erb +3 -0
- data/lib/evilution/reporter/html/templates/uncovered_details.html.erb +11 -0
- data/lib/evilution/reporter/json/mutation_detail.rb +77 -0
- data/lib/evilution/reporter/json/subjects.rb +1 -0
- data/lib/evilution/reporter/json/timings.rb +42 -0
- data/lib/evilution/reporter/json.rb +13 -50
- data/lib/evilution/result/coverage_gap.rb +5 -0
- data/lib/evilution/result/mutation_result.rb +55 -3
- data/lib/evilution/result/mutation_timings.rb +54 -0
- data/lib/evilution/result/run_timings.rb +20 -0
- data/lib/evilution/result/subject_score.rb +8 -1
- data/lib/evilution/result/subject_scorer.rb +5 -3
- data/lib/evilution/result/summary.rb +49 -12
- data/lib/evilution/runner/baseline_runner.rb +47 -5
- data/lib/evilution/runner/diagnostics.rb +0 -13
- data/lib/evilution/runner/isolation_resolver.rb +16 -4
- data/lib/evilution/runner/mutation_executor/isolated_run.rb +34 -0
- data/lib/evilution/runner/mutation_executor/kill_history.rb +66 -0
- data/lib/evilution/runner/mutation_executor/mutation_runner.rb +31 -12
- data/lib/evilution/runner/mutation_executor/neutralizer/baseline_failed.rb +1 -0
- data/lib/evilution/runner/mutation_executor/neutralizer/infra_error.rb +6 -20
- data/lib/evilution/runner/mutation_executor/probe_first.rb +99 -0
- data/lib/evilution/runner/mutation_executor/probe_first_run.rb +59 -0
- data/lib/evilution/runner/mutation_executor/result_cache.rb +45 -14
- data/lib/evilution/runner/mutation_executor/result_notifier.rb +1 -1
- data/lib/evilution/runner/mutation_executor/result_packer.rb +54 -24
- data/lib/evilution/runner/mutation_executor/runless_verdict.rb +39 -0
- data/lib/evilution/runner/mutation_executor/strategy/parallel/hinted_items.rb +27 -0
- data/lib/evilution/runner/mutation_executor/strategy/parallel/ordered_delivery.rb +97 -0
- data/lib/evilution/runner/mutation_executor/strategy/parallel.rb +102 -83
- data/lib/evilution/runner/mutation_executor/timed_run.rb +33 -0
- data/lib/evilution/runner/mutation_executor/timeout_policy/adaptive.rb +67 -0
- data/lib/evilution/runner/mutation_executor/timeout_policy/base.rb +35 -0
- data/lib/evilution/runner/mutation_executor/timeout_policy/fixed.rb +17 -0
- data/lib/evilution/runner/mutation_executor/timeout_policy.rb +56 -0
- data/lib/evilution/runner/mutation_executor.rb +75 -10
- data/lib/evilution/runner/mutation_planner.rb +29 -21
- data/lib/evilution/runner/phase_clock.rb +22 -0
- data/lib/evilution/runner/report_publisher.rb +1 -1
- data/lib/evilution/runner/subject_pipeline.rb +18 -8
- data/lib/evilution/runner.rb +87 -26
- data/lib/evilution/session/diff.rb +9 -2
- data/lib/evilution/session/store.rb +17 -2
- data/lib/evilution/source_ast_cache.rb +10 -0
- data/lib/evilution/spec_ast_cache.rb +15 -0
- data/lib/evilution/spec_resolver/constant_references.rb +66 -0
- data/lib/evilution/spec_resolver/project_files.rb +50 -0
- data/lib/evilution/spec_resolver/sibling_spec_ranker.rb +111 -0
- data/lib/evilution/spec_resolver/spec_pattern.rb +19 -0
- data/lib/evilution/spec_resolver.rb +76 -65
- data/lib/evilution/spec_selector.rb +17 -13
- data/lib/evilution/subject.rb +7 -2
- data/lib/evilution/version.rb +1 -1
- data/lib/evilution/worker_database/copies.rb +73 -0
- data/lib/evilution/worker_database/schema.rb +39 -0
- data/lib/evilution/worker_database/sqlite_copy.rb +65 -0
- data/lib/evilution/worker_database.rb +214 -0
- data/schema/evilution.config.schema.json +74 -2
- metadata +84 -13
- data/lib/evilution/integration/rspec/state_guard/object_space_example_groups.rb +0 -42
- data/scripts/benchmark_density +0 -262
- data/scripts/benchmark_density.yml +0 -19
- data/scripts/canary_manifest.yml +0 -47
- data/scripts/compare_mutations +0 -421
- data/scripts/compare_mutations.yml +0 -24
- data/scripts/compare_targeting +0 -279
- data/scripts/compare_targeting.example.yml +0 -24
- data/scripts/mutant_json_adapter +0 -227
data/README.md
CHANGED
|
@@ -107,9 +107,13 @@ Every command, subcommand, and flag listed in this section is part of evilution'
|
|
|
107
107
|
|
|
108
108
|
| Flag | Type | Default | Description |
|
|
109
109
|
|------------------------------|---------|--------------|---------------------------------------------------|
|
|
110
|
-
| `-t`, `--timeout N` | Integer |
|
|
110
|
+
| `-t`, `--timeout N` | Integer | _(adaptive)_ | Per-mutation timeout in seconds, the same for every mutation. Without it the limit adapts to the baseline, see [Per-mutation timeout](#per-mutation-timeout). |
|
|
111
|
+
| `--timeout-factor K` | Float | 10 | Without `--timeout`: limit each mutation to K × the baseline time of its spec files. |
|
|
112
|
+
| `--timeout-floor N` | Float | 10 | Without `--timeout`: the shortest such limit, in seconds. |
|
|
113
|
+
| `--timeout-ceiling N` | Float | 60 | Without `--timeout`: the longest such limit, in seconds. |
|
|
111
114
|
| `-f`, `--format FORMAT` | String | `text` | Output format: `text`, `json`, or `html`. |
|
|
112
115
|
| `--target EXPR` | String | _(none)_ | Only mutate matching subjects. Supports method name (`Foo::Bar#calculate`), class (`Foo`, which also matches a value-object constant named `Foo`), namespace wildcards (`Foo::Bar*`), method-type selectors (`Foo#`, `Foo.`), descendants (`descendants:Foo`), and source globs (`source:lib/**/*.rb`). |
|
|
116
|
+
| `--since REF` | String | _(none)_ | Only mutate the lines changed since git `REF` (a branch, tag or commit), diffed from its merge base with `HEAD`, uncommitted and untracked changes included. Without file arguments: every changed `.rb` file under `lib/` and `app/`. With them: only those that changed, line ranges narrowed to their changed lines. Also accepted by `subjects`. See [Mutating only what changed](#mutating-only-what-changed---since). |
|
|
113
117
|
| `--output FILE` | String | _(stdout)_ | Write the report to FILE instead of stdout. Useful when a preloaded spec helper writes to stdout on exit. |
|
|
114
118
|
| `--min-score FLOAT` | Float | 0.0 | Minimum mutation score (0.0–1.0) to pass. |
|
|
115
119
|
| `--spec FILES` | Array | _(none)_ | Spec files to run (comma-separated). Defaults to auto-detection via `SpecResolver`, which also resolves non-mirrored (`spec/unit`, `test/unit`), dir-grouped (`test/unit/<class>/*_test.rb`), and flat `test_`-prefixed (`test/test_connection_pool_timed_stack.rb`) layouts. |
|
|
@@ -118,16 +122,20 @@ Every command, subcommand, and flag listed in this section is part of evilution'
|
|
|
118
122
|
| `--no-example-targeting` | Boolean | _(enabled)_ | Disable per-mutation example targeting (always run every example in the resolved spec file). Example targeting scans each example body for symbols from the mutated method and runs only the matching subset. |
|
|
119
123
|
| `--example-targeting MODE` | String | `lexical` | How targeting picks examples: `lexical` (name-grep example bodies for the mutated method/class), `coverage` (run only the examples that actually execute the mutated line, from a cached line-coverage map), or `full_file` (run all resolved examples). |
|
|
120
124
|
| `--example-targeting-fallback MODE` | String | `full_file` | Behavior when no example matches: `full_file` (run the whole spec file) or `unresolved` (skip the mutation as `:unresolved`). |
|
|
125
|
+
| `--[no-]skip-uncovered` | Boolean | _(enabled)_ | With `--example-targeting coverage`: report a mutation whose changed lines no example executes as `uncovered`, without running it. See [Uncovered mutations](#uncovered-mutations). |
|
|
126
|
+
| `--[no-]probe-first` | Boolean | _(disabled)_ | Experimental. Run each method's body-to-raise mutant first; when it survives, report the method's other mutants as survivors implied by it, without running them. See [Probe-first](#probe-first-experimental). |
|
|
121
127
|
| `-j`, `--jobs N` | Integer | 1 | Number of parallel workers. Uses demand-driven work distribution with pipe-based IPC. |
|
|
122
128
|
| `--no-baseline` | Boolean | _(enabled)_ | Skip baseline test suite check. By default, a baseline run detects pre-existing failures, so that a mutation whose tests fail only on those is recorded `neutral` rather than killed. |
|
|
123
129
|
| `--[no-]canary` | Boolean | _(enabled)_ | Run a proof-of-life synthetic mutation at session start; abort the run if the pipeline misreports it. Catches misconfigured isolation, broken autoload, and reporter-plugin eviction before any real score is produced. Pass `--no-canary` to skip (e.g. CI speed, or when the canary itself is the thing under test). |
|
|
130
|
+
| `--[no-]stop-on-kill` | Boolean | _(enabled)_ | Stop each mutation's test run at the first failure that decides the verdict. Disable to run every selected example. |
|
|
131
|
+
| `--[no-]killer-first` | Boolean | _(disabled)_ | RSpec, with stop-on-kill: run first the examples that killed nearby mutations, then the cheapest. Only for suites whose examples pass in any order. See [Killer-first order](#killer-first-order-killer_first). |
|
|
124
132
|
| `--fail-fast [N]` | Integer | _(none)_ | Stop after N surviving mutants (default 1 if no value given). |
|
|
125
133
|
| `-v`, `--verbose` | Boolean | false | Verbose output with RSS memory and GC stats per phase and per mutation; also prints error class, message, and first 5 backtrace lines for errored mutations. |
|
|
126
134
|
| `--suggest-tests` | Boolean | false | Generate concrete test code in suggestions (RSpec or Minitest, based on `--integration`; falls back to generic phrasing for `test-unit`). |
|
|
127
135
|
| `-q`, `--quiet` | Boolean | false | Suppress output. |
|
|
128
136
|
| `--stdin` | Boolean | false | Read target file paths from stdin (one per line). |
|
|
129
137
|
| `--integration NAME` | String | `rspec` | Test framework integration: `rspec`, `minitest`, or `test-unit`. See [docs/integrations.md](docs/integrations.md). |
|
|
130
|
-
| `--[no-]incremental` | Boolean | false | Cache killed
|
|
138
|
+
| `--[no-]incremental` | Boolean | false | Cache each mutation's verdict (killed, survived, timeout) and skip it on re-runs until its subject, its spec files or the run's settings change — see [Incremental cache](#incremental-cache). Pass `--no-incremental` to override `incremental: true` from the config file for one invocation (e.g. cold-cache debugging). Last flag wins when both are given. |
|
|
131
139
|
| `--save-session` | Boolean | false | Persist results as timestamped JSON under `.evilution/results/`. |
|
|
132
140
|
| `--no-progress` | Boolean | _(enabled)_ | Disable the TTY progress bar. |
|
|
133
141
|
| `--quiet-children` | Boolean | false | Redirect each forked worker's stdout/stderr to per-pid files under `tmp/evilution_children/<pid>.{out,err}` so noisy app initializers (Datadog, Bullet, etc.) don't merge with parent output. Trade-off: live worker errors only appear in the side files, not the terminal — `tail -f tmp/evilution_children/*.err` to watch them. |
|
|
@@ -136,7 +144,9 @@ Every command, subcommand, and flag listed in this section is part of evilution'
|
|
|
136
144
|
| `--preload FILE` | String | _(auto)_ | File to require in parent before forking workers. Auto-detect chain for Rails projects: `spec/rails_helper.rb` → `spec/spec_helper.rb` → `test/test_helper.rb`. For non-Rails gems: `spec/spec_helper.rb` → `test/test_helper.rb` → `test/helper.rb`, falling back to the gem entry `lib/<gem>.rb` (with a warning naming the searched paths). Pass `--no-preload` to opt out. |
|
|
137
145
|
| `--no-preload` | Boolean | _(enabled)_ | Disable parent-process preload. |
|
|
138
146
|
| `--warmup NAME` | String | `none` | After preload, warm lazily-initialised framework state once in the parent: `none` or `rails`. See [Warming up Rails before forking](#warming-up-rails-before-forking). |
|
|
147
|
+
| `--[no-]copy-worker-databases` | Boolean | _(enabled)_ | With `--jobs` above 1 and a preloaded Rails app, give each worker its own copy of a SQLite test database the workers would otherwise share. See [Per-worker database copies](#per-worker-database-copies). |
|
|
139
148
|
| `--skip-heredoc-literals` | Boolean | false | Skip all string literal mutations inside heredocs. |
|
|
149
|
+
| `--skip-sites PRESETS` | List | _(none)_ | Leave low-value code unmutated, comma-separated: `logging`, `debug_output`, `error_messages`, `requires`. `--no-skip-sites` overrides `skip_sites` from the config file. See [Skipping Low-Value Code](#skipping-low-value-code). |
|
|
140
150
|
| `--show-disabled` | Boolean | false | Report mutations skipped by `# evilution:disable` comments. |
|
|
141
151
|
| `--fallback-full-suite` | Boolean | false | When no matching spec/test resolves for a mutation, run the whole test suite instead of marking it `:unresolved` and skipping. |
|
|
142
152
|
| `--related-specs-heuristic` | Boolean | false | When a mutation removes an `includes(...)` call, also run matching specs from `spec/{requests,integration,features,system}` (Rails-style domain match on the source file's basename). Trades extra spec runs for higher kill rate on ORM mutations. |
|
|
@@ -151,7 +161,7 @@ Every command, subcommand, and flag listed in this section is part of evilution'
|
|
|
151
161
|
|---------------------|---------|----------------------|----------------------------------------------------------------------------------------------|
|
|
152
162
|
| `--results-dir DIR` | String | `.evilution/results` | Directory containing session result JSON files. Honored by `session list` and `session gc`. |
|
|
153
163
|
| `--limit N` | Integer | _(none)_ | (`session list`) Show only the N most recent sessions. |
|
|
154
|
-
| `--since DATE` | String | _(none)_ | (`session list`) Show only sessions created on or after `DATE` (`YYYY-MM-DD`).
|
|
164
|
+
| `--since DATE` | String | _(none)_ | (`session list`) Show only sessions created on or after `DATE` (`YYYY-MM-DD`). For `run` and `subjects`, `--since` takes a git ref instead. |
|
|
155
165
|
| `--older-than D` | String | _(required)_ | (`session gc`) Delete sessions older than `D` (e.g. `30d`, `24h`, `1w`). |
|
|
156
166
|
|
|
157
167
|
### Options (for `compare` command)
|
|
@@ -181,7 +191,7 @@ Set via `--profile=strict`, the `--strict` shortcut, or `profile: strict` in `.e
|
|
|
181
191
|
|
|
182
192
|
| Code | Meaning | Agent action |
|
|
183
193
|
|------|-----------------------------------------------|---------------------------------------|
|
|
184
|
-
| 0 | Mutation score meets or exceeds `--min-score` | Success. No action needed. |
|
|
194
|
+
| 0 | Mutation score meets or exceeds `--min-score`, or a `--since` run found nothing to mutate | Success. No action needed. |
|
|
185
195
|
| 1 | Mutation score below `--min-score`, or a target file resolved to no spec | Parse output, fix surviving mutants. |
|
|
186
196
|
| 2 | Tool error (bad config, parse failure, a run under `--isolation in_process` that a mutation took down, etc.) | Check stderr, fix invocation. |
|
|
187
197
|
|
|
@@ -201,6 +211,36 @@ Evilution owns the exit status: `--preload` loads the project's own spec helper
|
|
|
201
211
|
|
|
202
212
|
The printed verdict and the exit code always use the same threshold. Previously the line was printed against a hard-coded 80% that the exit code did not share, so a failing-looking run still exited 0 (GH #1604).
|
|
203
213
|
|
|
214
|
+
### Mutating only what changed (`--since`)
|
|
215
|
+
|
|
216
|
+
`--since REF` limits a run to the lines changed since a git ref, which is what a pull request needs checked:
|
|
217
|
+
|
|
218
|
+
```bash
|
|
219
|
+
bundle exec evilution run --since origin/main # every changed .rb file under lib/ and app/
|
|
220
|
+
bundle exec evilution run lib/billing.rb --since origin/main # only this file, if it changed
|
|
221
|
+
bundle exec evilution subjects --since origin/main # preview the subjects and mutation counts
|
|
222
|
+
```
|
|
223
|
+
|
|
224
|
+
What counts as changed:
|
|
225
|
+
|
|
226
|
+
- **The branch's own changes.** The diff runs from the merge base of `REF` and `HEAD` (what `git diff REF...HEAD` compares), so commits `REF` gained after the branch left it do not count, to the working tree, so uncommitted changes, staged or not, do.
|
|
227
|
+
- **Added and modified lines.** A mutation is run when any line it changes is one of them; a subject (method, constant) is mutated when any of its lines is. Lines that were only deleted leave no line behind: the lines either side of the gap count instead, so a method whose only change is a removed line is still mutated there.
|
|
228
|
+
- **New files** count whole, untracked ones included (those `.gitignore` leaves out do not). **Renamed files** count only their edited lines, under the new name. Deleted files, binary files and files outside the current directory do not count.
|
|
229
|
+
- **Which files.** Without file arguments, the changed `.rb` files under `lib/` and `app/`, as a run without arguments takes. File arguments narrow that: a file that did not change drops out, and `lib/foo.rb:15-30` keeps only the changed lines from 15 to 30. A file argument must be an existing file: a missing path or a directory is an error (exit 2), as without `--since`. `--target source:GLOB` cannot be combined with `--since` (pass the files instead).
|
|
230
|
+
- **A method `--target`** is looked for in the changed files only. When its file changed but the method did not, there is nothing to mutate and the run passes. When its file did not change, the target matches nothing and the run fails with exit 2 (`no subject matched '...' in the files changed since REF`), as a misspelled target does: evilution cannot tell the two apart without reading the unchanged files. A CI job that pins a `--target` should run only when the target's file changed.
|
|
231
|
+
- **Nothing to mutate passes.** A run whose changed lines hold nothing to mutate (a docs-only pull request) reports zero mutations and exits 0, whatever `--min-score` says.
|
|
232
|
+
|
|
233
|
+
`REF` must be known to the clone, and the history back to the merge base must be there: a shallow CI checkout fails with a message saying so. In GitHub Actions:
|
|
234
|
+
|
|
235
|
+
```yaml
|
|
236
|
+
- uses: actions/checkout@v4
|
|
237
|
+
with:
|
|
238
|
+
fetch-depth: 0 # the merge base with the target branch must be fetched
|
|
239
|
+
- run: bundle exec evilution run --since origin/${{ github.base_ref }} --format json --min-score 0.8 --output evilution.json
|
|
240
|
+
```
|
|
241
|
+
|
|
242
|
+
`--since` is not a config file key: a ref in `.evilution.yml` would scope every run, explicit file lists included. The MCP `evilution-mutate` tool takes the same as its `since` parameter.
|
|
243
|
+
|
|
204
244
|
## Configuration
|
|
205
245
|
|
|
206
246
|
Generate default config: `bundle exec evilution init`
|
|
@@ -209,7 +249,10 @@ Creates `.evilution.yml`:
|
|
|
209
249
|
|
|
210
250
|
```yaml
|
|
211
251
|
schema_version: 1 # opts into strict validation (rejects unknown keys, refuses future versions)
|
|
212
|
-
# timeout: 30 # seconds per mutation
|
|
252
|
+
# timeout: 30 # seconds per mutation; unset = adaptive (see Per-mutation timeout)
|
|
253
|
+
# timeout_factor: 10 # adaptive limit = factor x baseline time of the mutation's spec files...
|
|
254
|
+
# timeout_floor: 10 # ...but at least this many seconds...
|
|
255
|
+
# timeout_ceiling: 60 # ...and at most this many
|
|
213
256
|
# format: text # text | json | html
|
|
214
257
|
# min_score: 0.0 # 0.0–1.0
|
|
215
258
|
# integration: rspec # test framework: rspec, minitest, test_unit
|
|
@@ -217,12 +260,18 @@ schema_version: 1 # opts into strict validation (rejects unknown keys
|
|
|
217
260
|
# save_session: false # persist results under .evilution/results/
|
|
218
261
|
# isolation: auto # auto | fork | in_process (auto selects fork for Rails + gems, in_process otherwise; in_process cannot contain a mutation that takes the Ruby process down, see docs/isolation.md)
|
|
219
262
|
# canary: true # proof-of-life synthetic mutation at session start (false to skip)
|
|
263
|
+
# stop_on_kill: true # stop each mutation's test run at the first deciding failure (false runs every selected example)
|
|
264
|
+
# killer_first: false # RSpec + stop_on_kill: run likely killers and cheap examples first (only for order-independent suites)
|
|
265
|
+
# skip_uncovered: true # with coverage targeting, report mutations no example executes as uncovered without running them
|
|
266
|
+
# probe_first: false # experimental: when a method's body-to-raise mutant survives, report its other mutants as implied survivors without running them
|
|
220
267
|
# preload: null # path to preload before forking; false to disable; auto-detects for Rails + gems
|
|
221
268
|
# warmup: none # rails: warm I18n, routes, assets and templates once after preload (fork isolation)
|
|
269
|
+
# copy_worker_databases: true # with jobs > 1 and a preloaded Rails app, give each worker its own copy of a shared SQLite test database
|
|
222
270
|
# skip_heredoc_literals: false # skip string literal mutations inside heredocs (recommended for Rails: heredoc SQL/templates rarely have test coverage)
|
|
223
271
|
# show_disabled: false # report mutations skipped by disable comments
|
|
224
272
|
# baseline_session: null # path to session file for HTML comparison
|
|
225
273
|
# ignore_patterns: [] # AST patterns to exclude (see docs/ast_pattern_syntax.md)
|
|
274
|
+
# skip_sites: [] # presets of low-value code to leave unmutated: logging, debug_output, error_messages, requires
|
|
226
275
|
# progress: true # TTY progress bar
|
|
227
276
|
# example_targeting: true # per-mutation example targeting via body-token scan
|
|
228
277
|
# example_targeting_fallback: full_file # full_file | unresolved
|
|
@@ -253,7 +302,10 @@ All keys recognised under `schema_version: 1`:
|
|
|
253
302
|
| Key | Type | Default | Description |
|
|
254
303
|
|------------------------------|-------------------------------|----------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------|
|
|
255
304
|
| `schema_version` | Integer | `1` | Config schema version. Declaring it enables strict validation; omit for lenient mode. |
|
|
256
|
-
| `timeout` | Integer |
|
|
305
|
+
| `timeout` | Integer | _(adaptive)_ | Per-mutation timeout in seconds, the same for every mutation. Unset, the limit adapts to the baseline (see [Per-mutation timeout](#per-mutation-timeout)). |
|
|
306
|
+
| `timeout_factor` | Number | `10` | Adaptive limit: this many times the baseline time of the mutation's spec files. Ignored when `timeout` is set. |
|
|
307
|
+
| `timeout_floor` | Number | `10` | Shortest adaptive limit, in seconds. Must not exceed `timeout_ceiling`. |
|
|
308
|
+
| `timeout_ceiling` | Number | `60` | Longest adaptive limit, in seconds. |
|
|
257
309
|
| `format` | String | `text` | Output format: `text`, `json`, `html`. |
|
|
258
310
|
| `target` | String / null | `null` | Filter expression: method (`Foo#bar`), class (`Foo`), namespace (`Foo*`), descendants (`descendants:Foo`), source glob (`source:**/*.rb`). |
|
|
259
311
|
| `output` | String | _(stdout)_ | Write the report to this file instead of stdout. |
|
|
@@ -264,14 +316,17 @@ All keys recognised under `schema_version: 1`:
|
|
|
264
316
|
| `jobs` | Integer | `1` | Number of parallel workers. |
|
|
265
317
|
| `fail_fast` | Integer / null | `null` | Stop after N surviving mutants. `null` = disabled. |
|
|
266
318
|
| `baseline` | Boolean | `true` | Run baseline test suite to detect pre-existing failures (a mutation failing only on those is `:neutral`). |
|
|
319
|
+
| `stop_on_kill` | Boolean | `true` | Stop each mutation's test run at the first failure that decides the verdict. Set `false` to run every selected example. |
|
|
320
|
+
| `killer_first` | Boolean | `false` | With RSpec and `stop_on_kill`, run first the examples that killed nearby mutations, then the cheapest. Only for suites whose examples pass in any order; see [Killer-first order](#killer-first-order-killer_first). |
|
|
267
321
|
| `isolation` | String | `auto` | Isolation strategy: `auto`, `fork`, `in_process`. `auto` selects `fork` for Rails projects and packaged gems (`*.gemspec`), `in_process` otherwise. Under `in_process` a mutation that takes the Ruby process down ends the run: see [What `in_process` cannot contain](docs/isolation.md#what-in_process-cannot-contain). |
|
|
268
|
-
| `incremental` | Boolean | `false` | Cache
|
|
322
|
+
| `incremental` | Boolean | `false` | Cache verdicts across runs; see [Incremental cache](#incremental-cache). |
|
|
269
323
|
| `suggest_tests` | Boolean | `false` | Generate concrete test code in survivor suggestions (matches `integration`). |
|
|
270
324
|
| `progress` | Boolean | `true` | TTY progress bar. |
|
|
271
325
|
| `save_session` | Boolean | `false` | Save session JSON under `.evilution/results/`. |
|
|
272
326
|
| `line_ranges` | Hash | `{}` | Per-file line-range constraints. Typically set via CLI; rare in YAML. |
|
|
273
327
|
| `spec_files` | Array<String> | `[]` | Explicit spec files to run. Bypasses auto-detection when non-empty. |
|
|
274
328
|
| `ignore_patterns` | Array<String> | `[]` | AST patterns to skip during mutation generation. See [docs/ast_pattern_syntax.md](docs/ast_pattern_syntax.md). |
|
|
329
|
+
| `skip_sites` | Array<String> | `[]` | Presets of low-value code to leave unmutated: `logging`, `debug_output`, `error_messages`, `requires`. See [Skipping Low-Value Code](#skipping-low-value-code). |
|
|
275
330
|
| `show_disabled` | Boolean | `false` | Report mutations skipped by `# evilution:disable` comments. |
|
|
276
331
|
| `baseline_session` | String / null | `null` | Saved session file path for HTML report comparison. |
|
|
277
332
|
| `skip_heredoc_literals` | Boolean | `false` | Skip string literal mutations inside heredocs. |
|
|
@@ -279,17 +334,58 @@ All keys recognised under `schema_version: 1`:
|
|
|
279
334
|
| `fallback_to_full_suite` | Boolean | `false` | When no matching spec resolves, run the entire suite instead of marking the mutation `:unresolved`. |
|
|
280
335
|
| `preload` | String / Boolean / null | `null` | File to preload in parent before forking. `false` to disable. `null` to auto-detect for Rails projects and packaged gems. |
|
|
281
336
|
| `warmup` | String | `none` | `rails` warms lazily-initialised Rails state once after preload so forks do not each pay it. `none` / `false` to skip. |
|
|
337
|
+
| `copy_worker_databases` | Boolean | `true` | With `jobs` above 1 and a preloaded Rails app, give each worker its own copy of a SQLite test database the workers would otherwise share. See [Per-worker database copies](#per-worker-database-copies). |
|
|
282
338
|
| `spec_mappings` | Hash<String, String/Array> | `{}` | Custom mapping from source path to spec path(s). |
|
|
283
339
|
| `spec_pattern` | String / null | `null` | Glob restricting resolved spec candidates. |
|
|
284
340
|
| `example_targeting` | Boolean | `true` | Per-mutation example-level targeting. |
|
|
285
341
|
| `example_targeting_strategy` | String | `lexical` | How targeting picks examples: `lexical` (name-grep) or `coverage` (examples that execute the mutated line, from a cached coverage map). |
|
|
286
342
|
| `example_targeting_fallback` | String | `full_file` | When targeting finds no example: `full_file` or `unresolved`. |
|
|
287
343
|
| `example_targeting_cache` | Hash | `{ max_files: 50, max_blocks: 10000 }` | LRU cache bounds for the example-targeting AST parser. |
|
|
344
|
+
| `skip_uncovered` | Boolean | `true` | With `example_targeting_strategy: coverage`, report a mutation no example executes as `uncovered` without running it. No effect otherwise. |
|
|
345
|
+
| `probe_first` | Boolean | `false` | Experimental. Run each method's `method_body_to_raise` mutant first; when it survives, report the method's other mutants as survivors implied by it, without running them. See [Probe-first](#probe-first-experimental). |
|
|
288
346
|
| `quiet_children` | Boolean | `false` | Redirect each worker's stdout/stderr to per-pid files under `quiet_children_dir`. |
|
|
289
347
|
| `quiet_children_dir` | String | `tmp/evilution_children` | Directory for `--quiet-children` per-pid log files. |
|
|
290
348
|
| `profile` | String | `default` | Operator profile: `default` or `strict`. |
|
|
291
349
|
| `hooks` | Hash<String, String> | `{}` | Lifecycle hooks: event name → path to a Ruby file returning a `Proc`. |
|
|
292
350
|
|
|
351
|
+
#### Stopping at the first kill (`stop_on_kill`)
|
|
352
|
+
|
|
353
|
+
By default each mutation's test run stops at the first failing example that decides the verdict, so the remaining selected examples are not run. Failures that were already failing in the baseline run do not stop it (the mutation is still recorded `neutral` if only those fail), and neither do infrastructure crashes (the mutation is still recorded `neutral` if only those occur). A mutant whose first failure is followed by a hang or a dead test process is now reported `killed`; a `--no-stop-on-kill` run may report the same mutant as `timeout` or `error`, so its score can be slightly lower. Pass `--no-stop-on-kill` (or `stop_on_kill: false`) to run every selected example. Minitest classes that call `parallelize_me!` are not stopped early: their tests all run, as they would with `--no-stop-on-kill`.
|
|
354
|
+
|
|
355
|
+
#### Killer-first order (`killer_first`)
|
|
356
|
+
|
|
357
|
+
With RSpec (3.13 or later), `--killer-first` (or `killer_first: true`) lets a run that stops at its first kill choose the order of its examples, since everything before the kill is its cost: first the examples that killed other mutations on the same line, then elsewhere in the same subject, earlier in this run (the latest killer first); then the rest, cheapest first by the time each took in the baseline; examples with no baseline time last. Examples stay inside their example group, and a group's `before(:context)` hooks still run once around its examples; a group goes first when it holds a killer, otherwise by the time its timed examples take in all (a group with no timed example goes last). Only the order changes, never which examples run. A survivor runs every selected example either way, and its [confirmation](#survivor-confirmation) run against the whole file keeps the defined order, as does the command reported in `test_command`. It has no effect with `--no-stop-on-kill`. In a parallel run, the mutations already handed to a worker when a kill comes back go without that killer. Minitest and test-unit runs keep their own order.
|
|
358
|
+
|
|
359
|
+
It is off by default because it is safe only for a suite whose examples pass in any order. An example that relies on one defined before it (a global, a class variable or a record that an earlier example sets up) fails when it runs first, whatever the mutant, so the mutation is reported `killed` even though a run in the defined order, including the reported `test_command`, would let it survive. Nothing in the report shows this, and it only raises the score. If your suite runs in random order (`config.order = :random`) and passes, it is a fair candidate. A run that times out in the killer-first order is run again in the defined order, which decides the verdict: a mutant that hangs the cheapest example would otherwise be reported `timeout` where the defined order reports `killed`. Such a mutation can take up to twice its time limit, and a parallel run gives each mutation twice the usual time before it gives a worker up. Under an [adaptive limit](#per-mutation-timeout) this happens under each limit: a reordered run that uses up the adaptive limit runs again in the defined order under it, and only when that times out too does the mutation run again under the fixed limit, reordered and then, if it times out once more, in the defined order, which decides.
|
|
360
|
+
|
|
361
|
+
The JSON report lists, per mutation, the example that killed it (`killing_test`) and how many examples ran (`examples_run`), whatever the order.
|
|
362
|
+
|
|
363
|
+
#### Uncovered mutations
|
|
364
|
+
|
|
365
|
+
With coverage targeting (`--example-targeting coverage`), evilution records which lines each example of the resolved spec files executes, and which executable lines nothing executed at all. A mutation whose changed lines nothing executes could only survive: no test reaches the code. Running it would cost the most of any verdict — the whole resolved spec file, then the [survivor confirmation](#survivor-confirmation) run — and say nothing new. Such a mutation is reported `uncovered` instead, without a test run.
|
|
366
|
+
|
|
367
|
+
**Score**: an uncovered mutation counts as a survivor. It stays in the score's denominator, so the score, `--min-score` / `min_score` and the exit status are the same as running it would have given; the report lists it in its own `uncovered` bucket (text, JSON, HTML, MCP and saved sessions) rather than with the survivors that ran, and `--fail-fast` counts it as a survivor.
|
|
368
|
+
|
|
369
|
+
A mutation is reported `uncovered` only when the coverage map can vouch for it, and runs as before otherwise:
|
|
370
|
+
|
|
371
|
+
- Only the lines the mutation changes count. A line that runs only while files load — a class body, a constant, `attr_reader`, a `def` line — counts as executed, so a mutation of it runs. A mutation that replaces a method body is reported at the `def` line but changes only the body, so it is uncovered when no example calls the method.
|
|
372
|
+
- At least one changed line is executable and never ran; none of them ran.
|
|
373
|
+
- The map was built from every spec file the mutation would run. A mutation that runs a spec file the build did not (the related-specs heuristic, the full-suite fallback) runs.
|
|
374
|
+
- Every example of those spec files ran in the build. A spec file that failed to load, or a build cut short (`fail_fast` in `.rspec`, an interrupt), makes the map vouch for no line.
|
|
375
|
+
- The file was loaded while lines were counted. When coverage targeting is on and the tests run in forked processes (`isolation: fork`, the default for Rails apps and gems), evilution counts from before the `preload` runs, so a preloaded file is covered too; a file loaded while counting was off is never vouched for. Each forked test process stops that counting before its tests run, so a spec helper that starts line coverage itself (SimpleCov) works as it does without coverage targeting. Under `in_process` isolation the preload is not counted.
|
|
376
|
+
- No example forked a process. A forked process counts its lines apart, where the build cannot see them, while a mutation applied before the fork reaches it; when any example forks, the map vouches for no line.
|
|
377
|
+
- The mutation neither adds nor removes a local variable assignment. Ruby settles while parsing whether a name is a local, so such a change reaches lines that do run (a later read of the name becomes a method call), even when its own line never runs.
|
|
378
|
+
|
|
379
|
+
The coverage map is cached under `.evilution/coverage/` and rebuilt when any file it depends on changes: a target file, a resolved spec file, one of the helpers every spec file loads (`.rspec`, `spec/spec_helper.rb`, `spec/support/**/*.rb`, ... — the same list as the [incremental cache](#incremental-cache)), any file under the directory the spec files sit in (`spec/`, `test/`: shared examples, factories, fixture data), and any file of the project the build loaded (another file under `lib/` that calls the code, for instance). It is RSpec only, like coverage targeting. Pass `--no-skip-uncovered` (or `skip_uncovered: false`) to run these mutations anyway.
|
|
380
|
+
|
|
381
|
+
`summary.uncovered` counts these mutations; it is unrelated to `summary.uncovered_code`, which lists targeted lines that hold no subject, so no mutation was generated for them.
|
|
382
|
+
|
|
383
|
+
#### Probe-first (experimental)
|
|
384
|
+
|
|
385
|
+
`--probe-first` (or `probe_first: true`) runs one sentinel mutant per method first: the method's whole body replaced by a bare `raise` (`method_body_to_raise`). When even that survives, no selected test calls the method on a path it checks, and the method's other mutants are reported `survived` without running them; their JSON entries carry `implied_by: "method_body_to_raise"`. A sentinel reported `uncovered` counts the same. A method whose sentinel was killed, timed out or errored, and a subject that has no sentinel (an empty body, a body that already only raises, a constant or a declaration), runs all its mutants as before. So does a method with a `rescue` or `ensure` clause of its own (`def safe_div(a, b); a / b; rescue StandardError; -1; end`): its body-to-raise mutant keeps the clause, which can catch the `raise` or return over it, so that mutant surviving says nothing about the tests and is not used as a sentinel.
|
|
386
|
+
|
|
387
|
+
The run goes in two batches: all sentinels, then the rest, so under `--jobs` the second batch waits for the slowest sentinel, and each batch has its own worker pool: the run's `timings.workers` lists the workers of both (up to twice `--jobs`), the first pool's with a few sentinels each. An implied survivor is scored like any survivor, gets no [survivor confirmation](#survivor-confirmation) run and is not cached; the text report marks it `(implied by method_body_to_raise, not run)` and counts the implied survivors next to `survived`. It can be wrong: a test that swallows the sentinel's exception (a `rescue` around the call, a matcher that expects some error) lets the sentinel survive while another mutant of the method is killed. It is off by default; see `docs/probe_first_spike.md` for what it saves and how often it is wrong on real projects.
|
|
388
|
+
|
|
293
389
|
## Disable Comments
|
|
294
390
|
|
|
295
391
|
Suppress mutations on specific code with inline comments:
|
|
@@ -313,6 +409,29 @@ configure_metrics
|
|
|
313
409
|
|
|
314
410
|
Use `--show-disabled` to see which mutations were skipped.
|
|
315
411
|
|
|
412
|
+
## Skipping Low-Value Code
|
|
413
|
+
|
|
414
|
+
Some mutations cost a full test run and rarely say anything: a changed log message, a reworded error message, a dropped `require` that the preloaded process has already run. `skip_sites` (or `--skip-sites`) names presets of such code to leave unmutated:
|
|
415
|
+
|
|
416
|
+
```yaml
|
|
417
|
+
skip_sites: [logging, error_messages]
|
|
418
|
+
```
|
|
419
|
+
|
|
420
|
+
Each preset is a structural rule over the parsed source, like [`ignore_patterns`](docs/ast_pattern_syntax.md), but a preset only skips what changes what the code *reports*. The raise, the class raised, the condition guarding a raise or a log line, code that may change state, a call whose value is used, and any mutation that puts a `raise` in are always mutated.
|
|
421
|
+
|
|
422
|
+
| Preset | Sites | Skipped inside a site |
|
|
423
|
+
|---|---|---|
|
|
424
|
+
| `logging` | A log level (`debug info warn error fatal unknown add log`) sent to a receiver named `logger` or `LOGGER` (`logger`, `@logger`, `Rails.logger`, `config.logger`, `App::LOGGER`); `warn` / `deprecation_warning` sent to a deprecator (`ActiveSupport::Deprecation`, `ActiveSupport.deprecator`, `Rails.deprecator`, `deprecator`); `warn(..., category: :deprecated)`. Not `logger.debug?` or `logger.level = ...`. | Literal text of the message; anything inside an argument or block that only reads. The call itself (deleting the line) and its receiver, when every argument only reads and the call's value goes unused: a statement followed by another one, not the last statement of a method or block (whose value it returns), nor an operand or an assigned value (`logger.info(...) && save`). |
|
|
425
|
+
| `debug_output` | `puts`, `print`, `printf`, `warn` sent to no receiver or to `Kernel`; `puts` / `print` / `printf` / `write` sent to `$stdout`, `$stderr`, `STDOUT`, `STDERR`; `p` / `pp` and a stream's `write` only where their value goes unused, since they return what they print or the bytes written (`p x` as the last statement returns `x`). | As for `logging`. Kept apart from `logging` because console output is a CLI's actual job. |
|
|
426
|
+
| `error_messages` | `raise` / `fail` sent to no receiver or to `Kernel`. The message is the second positional argument (`raise Error, "msg"`), the only one when it is a string (`raise "msg"`), or the string arguments of an error built in place on a constant (`raise Error.new("msg")`, `.exception`). Keywords (`cause:`), the backtrace argument and any argument after a splat are not messages. | Literal text of the message; anything inside a message that only reads. Never the raise, its class, or a non-string argument of `Error.new(...)`, which the error may keep as data. |
|
|
427
|
+
| `requires` | `require` / `require_relative` sent to no receiver or to `Kernel` that run while the file loads: at the top of the file or in a class or module body. Not inside a method, block or lambda (lazy loading), nor a require whose value is used (`HAS_OJ = require "oj"`). | Everything, the call included. Evilution mutates load-time code only in constant assignments and class-body declarations (`include`, `private`, `alias`), none of which is a plain `require` line, so today this preset skips nothing; it guards such lines should a subject ever reach them. |
|
|
428
|
+
|
|
429
|
+
"Only reads" means: literals, variable and constant reads, interpolation, `&&` / `||` / ternaries, `[]`, operators, `format`, `sprintf`, `join`, `to_s`, and, without arguments or a block, predicates (`empty?`) and a fixed list of common readers (`class`, `name`, `message`, `inspect`, `id`, `size`, `length`, `first`, `last`, `to_s`, `to_i`, `to_sym`, `path`, ...). Any other call, even one without arguments, may change state (`queue.pop`, `io.read`, `enum.next`), so a message such as `"left: #{queue.pop}"` keeps the call mutated (only its literal text is skipped), and a log line that calls one is never deleted. Mutations whose node is the whole `raise` call (removing or `nil`-ing its message argument) are not skipped, nor is replacing the message of `raise "msg"` as a whole: `raise nil` raises `TypeError` instead of `RuntimeError`.
|
|
430
|
+
|
|
431
|
+
A skipped mutation is not run and not scored. It is counted in `summary.skipped`, and per preset in `summary.skipped_sites` (JSON) and on a `Skipped sites:` line in the text report. A mutation an `ignore_patterns` pattern also matches is counted under the pattern. `skip_sites` changes no verdict, so it does not invalidate the [incremental cache](#incremental-cache).
|
|
432
|
+
|
|
433
|
+
**Default: none.** Under the 1.x [compatibility policy](#schema-versioning) a new key's default preserves prior behaviour, and turning a preset on removes mutants, and with them points from the score, that existing users see today. Measured over the whole `lib/` of dry-monads, httparty, grape and i18n and Redmine's `app/` and `lib/redmine`, all four presets together skip 0.3-0.7% of the mutants (in Redmine mostly `logging`, elsewhere mostly `error_messages`), and some of the skipped error-message mutants are killed by specs that assert the message, so the speed-up a default would buy is small next to a changed mutant set and score. Turn on the presets that fit your code base.
|
|
434
|
+
|
|
316
435
|
## JSON Output Schema
|
|
317
436
|
|
|
318
437
|
Use `--format json` for machine-readable output. The same shape is used for both stdout reports (`--format json`) and saved session files (`--save-session` → `.evilution/results/*.json`); session files add a small set of extra top-level fields described under [Session JSON files](#session-json-files) below.
|
|
@@ -339,31 +458,47 @@ Schema:
|
|
|
339
458
|
"baseline_neutralized": "integer (optional) — kills not counted because only examples already failing in the baseline failed; those mutations are recorded neutral; present only when non-zero",
|
|
340
459
|
"baseline_failures": "array (optional) — one entry per spec file that was red in the baseline: { spec_file, error: string|null — what stopped the file outside any example (load error, runner exception, timeout, a baseline process that died), failing_examples: integer — every failing example, examples: [{ id, description, message }] — the first few of them }; present only when non-empty",
|
|
341
460
|
"unparseable": "integer — mutations whose mutated source did not parse (short-circuited, never executed)",
|
|
342
|
-
"
|
|
343
|
-
"
|
|
344
|
-
"
|
|
461
|
+
"uncovered": "integer — mutations whose changed lines no example executes, reported without a test run (coverage targeting only); scored as survivors",
|
|
462
|
+
"score": "float — killed / (total - errors - neutral - equivalent - unresolved - unparseable), range 0.0-1.0, rounded to 4 decimals; uncovered mutations stay in the denominator, as survivors",
|
|
463
|
+
"duration": "float — total wall-clock seconds of the run, from subject discovery through the audit of target files for unresolved specs (the phases in timings.phases add up to it), rounded to 4 decimals",
|
|
464
|
+
"peak_memory_mb": "float (optional) — peak RSS across all mutation child processes, in MB",
|
|
465
|
+
"skipped": "integer (optional) — mutations left out during generation (ignore_patterns, skip_sites, # evilution:disable, Sorbet sigs), never run or scored; present only when non-zero",
|
|
466
|
+
"skipped_sites": "object (optional) — the share of each skip_sites preset in skipped, e.g. { logging: 3, error_messages: 12 }; present only when a preset skipped something"
|
|
345
467
|
},
|
|
346
468
|
"survived": [
|
|
347
469
|
{
|
|
348
470
|
"operator": "string — mutation operator name (see Operators table)",
|
|
349
471
|
"file": "string — relative path to mutated file",
|
|
350
472
|
"line": "integer — line number of the mutation",
|
|
351
|
-
"status": "string — result status: 'survived', 'killed', 'timeout', 'error', 'neutral', 'equivalent', 'unresolved', or '
|
|
473
|
+
"status": "string — result status: 'survived', 'killed', 'timeout', 'error', 'neutral', 'equivalent', 'unresolved', 'unparseable', or 'uncovered'",
|
|
352
474
|
"duration": "float — seconds this mutation took, rounded to 4 decimals",
|
|
353
475
|
"diff": "string — legacy +/- diff snippet",
|
|
354
|
-
"unified_diff": "string (optional, survived only) — git-style unified diff with `--- a/file`, `+++ b/file`, `@@` hunk header and sdiff body; omitted when source slices are unavailable",
|
|
355
|
-
"suggestion": "string — actionable hint for surviving mutants (survived only)",
|
|
356
|
-
"
|
|
476
|
+
"unified_diff": "string (optional, survived and uncovered only) — git-style unified diff with `--- a/file`, `+++ b/file`, `@@` hunk header and sdiff body; omitted when source slices are unavailable",
|
|
477
|
+
"suggestion": "string — actionable hint for surviving mutants (survived and uncovered only)",
|
|
478
|
+
"test_command": "string (optional) — the command that reproduces the run behind this verdict, shell-escaped so it can be pasted as is (a targeted RSpec run names example ids such as `spec/x_spec.rb[1:2]`); for a survivor confirmed against its whole spec files, that confirming run",
|
|
479
|
+
"killing_test": "string (optional, RSpec only) — the example whose failure killed the mutation, as `/absolute/path/spec/file_spec.rb[1:2]` (spec file and RSpec scoped id); absent when no example decided the kill",
|
|
480
|
+
"examples_run": "integer (optional, RSpec only) — how many examples ran against the mutation, the survivor confirmation run included; absent for results that did not run (cached, uncovered, unparseable)",
|
|
481
|
+
"implied_by": "string (optional, survived only, --probe-first) — the sentinel operator ('method_body_to_raise') whose survival implied this verdict; the mutation itself did not run",
|
|
482
|
+
"time_limit": "float (optional, timeout only) — the seconds this mutation's test run was allowed, rounded to 4 decimals; for a timeout confirmed by a second run (see Per-mutation timeout) the limit of that second run; for a timeout read from the --incremental cache, the limit it was recorded under",
|
|
483
|
+
"neutral_reason": "object (optional, neutral only) — { kind: 'baseline_failure' | 'infra_error', detail: string|null — the failing spec file or the crash class; null when the run was given explicit --spec files and no single spec can be named }",
|
|
484
|
+
"timings": "object (optional) — where this mutation's duration went, in seconds rounded to 4 decimals; only the phases that happened are listed: { apply — parsing and evaluating the mutated source, test_run — the first test run, confirmation — the re-run of a survivor against its whole spec files (RSpec, targeted runs only), queue_wait — parallel runs only: from the dispatcher sending it to a worker to that worker starting it; a worker is sent its next mutation as soon as it finishes one, so this is the cost of the hand-over, and for a worker's first mutation also of the worker starting up, adaptive_timeout — a first run that used up an adaptive limit shorter than the fixed one, after which the mutation ran again under the fixed limit and the other phases describe that second run (see Per-mutation timeout) }; what duration holds beyond these is the isolation around the run (fork, sandbox, reap); a timed-out mutation reports nothing itself, so its whole duration is counted as test_run; absent for results read from the --incremental cache"
|
|
357
485
|
}
|
|
358
486
|
],
|
|
487
|
+
"timings": {
|
|
488
|
+
"phases": "object — wall-clock seconds per phase of the run, in the order they ran: subject_discovery, preload, canary, baseline, planning (mutation generation and equivalence detection), execution (every mutation, including the serial re-run of infrastructure crashes and building the coverage map when coverage targeting is on), finalize; they add up to about summary.duration",
|
|
489
|
+
"baseline": "array — one entry per spec file the baseline ran, in order: { spec_file, duration }; empty when the baseline did not run",
|
|
490
|
+
"mutations": "object — each per-mutation timing phase summed over all mutations: { apply, test_run, confirmation, queue_wait, adaptive_timeout }; process seconds, so in a parallel run they add up across workers and can exceed the execution phase",
|
|
491
|
+
"workers": "array — one entry per worker process of a parallel run: { pid, items, busy, wall, utilization — busy / wall }; workers live for the whole run, so there are --jobs entries (fewer when there are fewer mutations to run; with --probe-first the sentinels and the other mutations run in two pools one after the other, so up to twice as many) plus one for each worker that replaced one killed after a timeout, a crash or an unreadable reply; under --isolation in_process each mutation gets a fresh worker, so there is one entry per mutation run; busy and wall are 0.0 for a worker that died before reporting them; empty for --jobs 1"
|
|
492
|
+
},
|
|
359
493
|
"subjects": [
|
|
360
494
|
{
|
|
361
495
|
"name": "string — subject name (e.g. 'Foo#bar')",
|
|
362
496
|
"file": "string — relative path to source file",
|
|
363
497
|
"total": "integer — mutations generated for this subject",
|
|
364
498
|
"killed": "integer — mutations detected",
|
|
365
|
-
"verified": "integer — mutations that got a verdict (killed + survived + timed out)",
|
|
499
|
+
"verified": "integer — mutations that got a verdict (killed + survived + timed out + uncovered)",
|
|
366
500
|
"survived": "integer — mutations that went undetected",
|
|
501
|
+
"uncovered": "integer — mutations no example executes, reported without a test run; counted in verified",
|
|
367
502
|
"score": "float — killed / verified, 0.0 when nothing was verified, rounded to 4 decimals",
|
|
368
503
|
"reached": "boolean — false when no mutation of this subject got a verdict at all"
|
|
369
504
|
}
|
|
@@ -374,8 +509,8 @@ Schema:
|
|
|
374
509
|
"subject": "string — method name (e.g. 'Foo#bar')",
|
|
375
510
|
"line": "integer — line number",
|
|
376
511
|
"operators": ["string — operator names involved"],
|
|
377
|
-
"count": "integer — number of survived mutations in this gap",
|
|
378
|
-
"mutations": ["... same shape as survived entries ..."]
|
|
512
|
+
"count": "integer — number of survived and uncovered mutations in this gap",
|
|
513
|
+
"mutations": ["... same shape as survived entries; `status` tells a survivor from an uncovered mutation ..."]
|
|
379
514
|
}
|
|
380
515
|
],
|
|
381
516
|
"killed": ["... same shape as survived entries ..."],
|
|
@@ -383,6 +518,7 @@ Schema:
|
|
|
383
518
|
"equivalent": ["... same shape as survived entries ..."],
|
|
384
519
|
"unresolved": ["... same shape as survived entries — coverage gap: no spec file resolved for these mutations"],
|
|
385
520
|
"unparseable": ["... same shape as survived entries — mutated source failed to parse and was never executed"],
|
|
521
|
+
"uncovered": ["... same shape as survived entries — no example executes the lines the mutation changes; never executed, scored as survived"],
|
|
386
522
|
"timed_out": ["... same shape as survived entries ..."],
|
|
387
523
|
"errors": [
|
|
388
524
|
{
|
|
@@ -409,9 +545,10 @@ Sessions saved by `--save-session` (under `.evilution/results/*.json`) and consu
|
|
|
409
545
|
| `error_count` | Integer | Mirrors `summary.errors`. |
|
|
410
546
|
| `neutral_count` | Integer | Mirrors `summary.neutral`. |
|
|
411
547
|
| `equivalent_count` | Integer | Mirrors `summary.equivalent`. |
|
|
412
|
-
| `
|
|
548
|
+
| `uncovered_count` | Integer | Mirrors `summary.uncovered`. |
|
|
549
|
+
| `skipped_count` | Integer | Mutations left out during generation: `ignore_patterns`, `skip_sites`, `# evilution:disable`, Sorbet sigs (mirrors `summary.skipped`, omitted from `summary` unless positive; `summary.skipped_sites` carries the per-preset share). |
|
|
413
550
|
|
|
414
|
-
Saved sessions also omit the per-status arrays (`killed`, `neutral`, `equivalent`, `unresolved`, `unparseable`, `timed_out`, `errors`) — only `survived` and `coverage_gaps` are persisted. The score, totals, and timestamps are stable for diff/compare consumers.
|
|
551
|
+
Saved sessions also omit the per-status arrays (`killed`, `neutral`, `equivalent`, `unresolved`, `unparseable`, `timed_out`, `errors`) — only `survived`, `uncovered` and `coverage_gaps` are persisted. `evilution session diff` treats an `uncovered` mutation as a survivor, so one that survived before is not reported fixed. The score, totals, and timestamps are stable for diff/compare consumers.
|
|
415
552
|
|
|
416
553
|
#### stdout in JSON mode
|
|
417
554
|
|
|
@@ -437,12 +574,13 @@ Compatibility policy for the `1.x` gem line:
|
|
|
437
574
|
|--------------|-----------------------------------------------------------------------|-------------------|
|
|
438
575
|
| `killed` | A test failed when the mutation was applied — test suite caught it. Also a mutation that made its own file raise while loading, before any test ran, when the original file still loads (`error_message` starts with `load failed:`). Also a mutation whose test process died on an exception no handler saw, after the mutated source loaded and while the tests ran, as a crashing test would: under fork isolation `error_message` starts with `child died` and names the exception, under in-process isolation it starts with `test run ended by` | numerator + denominator |
|
|
439
576
|
| `survived` | No test failed — gap in coverage | denominator only |
|
|
440
|
-
| `timeout` | Test run exceeded `--timeout
|
|
577
|
+
| `timeout` | Test run exceeded its time limit (`--timeout`, or the [adaptive limit](#per-mutation-timeout)) — treated like survived for scoring | denominator only |
|
|
441
578
|
| `error` | Mutation caused an unexpected error (syntax error, boot failure, a load failure the original file shares, etc.). Also a test process that died on an unhandled exception before the tests ran: under fork isolation `error_message` starts with `child died`, says `before the tests ran` and `error_class` is the exception's own class. Also a result that never arrived or could not be read, which is no verdict on the mutation (see [Diagnosing errored mutations](#7-diagnosing-errored-mutations)) | excluded from denominator |
|
|
442
579
|
| `neutral` | No verdict: the tests failed only on examples that were already failing in the baseline, or the test process crashed on infrastructure (DB lock, statement timeout) rather than on the mutation. Every neutral records which of the two, and the report groups them by it | excluded |
|
|
443
580
|
| `equivalent` | Mutation is provably identical to the original (e.g. no-op replacement) | excluded |
|
|
444
581
|
| `unresolved` | No spec file resolved for the mutated source — **coverage gap, not a failure**. Use `--fallback-full-suite` to run the full suite instead. | excluded |
|
|
445
582
|
| `unparseable` | Mutated source failed to parse (e.g. dangling heredoc opener after `method_body_replacement`). Short-circuited — never executed. | excluded |
|
|
583
|
+
| `uncovered` | No example executes the lines the mutation changes (coverage targeting only, see [Uncovered mutations](#uncovered-mutations)). Never executed — it could only have survived. | denominator only, as survived |
|
|
446
584
|
|
|
447
585
|
Unresolved mutations indicate a missing test mapping — the file has no corresponding test file that the resolver could find (for example, an RSpec `_spec.rb` file or a Minitest `_test.rb` file, depending on configuration). The resolver searches the `lib/`-mirrored path, common non-mirrored buckets (`spec/unit`, `spec/lib`, `test/unit`, `test/lib`), and the flat `test_`-prefixed Minitest/Test::Unit convention (`test/test_connection_pool_timed_stack.rb`), so a high unresolved rate usually means a genuinely missing or unconventionally-placed test; a run that leaves many mutations unresolved prints an unresolved-rate warning with a best-guess spec path per source file. They are reported separately so you can act on them (add a test, adjust test naming, pass `--spec`, or opt in to the full-suite fallback) without inflating the error count.
|
|
448
586
|
|
|
@@ -469,6 +607,47 @@ Before any survivor is reported, it is therefore re-run against the whole resolv
|
|
|
469
607
|
|
|
470
608
|
On evilution's own `lib/evilution/reporter/json/subjects.rb` this moved the reported score from 74.19% with 8 survivors to 93.55% with 2 — the six that disappeared were killed by an example in the same file all along, and the run now agrees with `--no-example-targeting` instead of contradicting it.
|
|
471
609
|
|
|
610
|
+
### Per-mutation timeout
|
|
611
|
+
|
|
612
|
+
A mutation that makes the code loop runs until its time limit, so the limit is most of what such a mutant costs. A fixed limit is either far too long for a spec file that passes in a fraction of a second or too short for a slow one.
|
|
613
|
+
|
|
614
|
+
So when no timeout is given, each mutation's limit follows the time its own spec files took in the baseline:
|
|
615
|
+
|
|
616
|
+
```
|
|
617
|
+
limit = clamp(timeout_factor × baseline seconds of the mutation's spec files, timeout_floor, timeout_ceiling)
|
|
618
|
+
```
|
|
619
|
+
|
|
620
|
+
With the defaults (factor 10, floor 10s, ceiling 60s) a mutant on a spec file that passes in 0.3s is first given 10s instead of 30s, and one whose spec files take 4s gets 40s. The baseline runs each spec file in a forked process of its own, so its time holds the fork and loading the file; the factor leaves room for the re-run of a survivor against its whole spec files ([Survivor Confirmation](#survivor-confirmation)) and for workers competing for the CPU under `--jobs`, and the floor covers spec files so fast that scheduling noise outweighs them.
|
|
621
|
+
|
|
622
|
+
A mutant that slows the code down can still fail its test well after its baseline suggests, and a timeout reported there would hide a kill. So a mutation that uses up a limit shorter than the fixed 30s is run again under 30s, and the verdict of that second run is the one reported: a mutation is only reported as a timeout where the fixed limit would have reported one too. The first run is part of what the mutation cost: it is in its `duration` and in the JSON report's `timings` as `adaptive_timeout`. A mutant that loops therefore costs the shorter limit plus 30s; the time saved is on kills and survivors, and on spec files slow enough to get a longer limit. Raise `timeout_floor` if many mutations are run twice.
|
|
623
|
+
|
|
624
|
+
A mutation keeps the fixed 30s limit when the baseline cannot vouch for its spec files: no baseline ran (`--no-baseline`), no spec file resolves for its source (it falls back to the whole suite, or is reported unresolved), one of its spec files failed in the baseline, or `related_specs_heuristic` adds spec files the baseline never ran. Every mutation keeps it when a `worker_process_start`, `mutation_insert_pre` or `mutation_insert_post` hook is configured: those run inside the time a mutation is allowed, and the baseline runs none of them.
|
|
625
|
+
|
|
626
|
+
`--timeout N` (or `timeout:` in the config file) gives every mutation the same N seconds, as before; pass `--timeout 30` for the previous behaviour. The limit a timed-out mutation ran under is in the JSON report as `time_limit`. Under `--jobs` the pool gives up on a worker that has not answered within twice the longest time a mutation of the run may take, both of its runs together.
|
|
627
|
+
|
|
628
|
+
With `--incremental`, a cached timeout is reused only while the mutation would get no more time than the limit it was recorded under; raising `--timeout`, `timeout_floor` or the factor runs it again. Timeouts cached before the limit was recorded are run again once.
|
|
629
|
+
|
|
630
|
+
### Incremental cache
|
|
631
|
+
|
|
632
|
+
`--incremental` (or `incremental: true`) keeps each mutation's verdict — killed, survived or timeout — in `tmp/evilution_cache/` and reuses it on the next run instead of running the mutation again. A reused verdict is reported and scored like a fresh one; it carries no `timings` and no memory figures, since nothing ran. A cached timeout is reused only while the mutation would get no more time than the limit it was recorded under (see [Per-mutation timeout](#per-mutation-timeout)).
|
|
633
|
+
|
|
634
|
+
Verdicts are kept per subject (a method, a constant, a declaration). A cached verdict is reused only while all of these are unchanged:
|
|
635
|
+
|
|
636
|
+
- the subject's own source;
|
|
637
|
+
- the rest of its file outside every method definition: class and module lines, `include`s, `attr_*`s, other class-body code, every constant of the file, and every class-body declaration (callbacks, validations, scopes, AASM machines), since these decide whether and when the file's methods run;
|
|
638
|
+
- the contents of the spec files that resolve for the file (`--spec`, `spec_mappings`, or the resolver), and of the files the related-specs heuristic adds for the mutation when it is on;
|
|
639
|
+
- the conventional helpers every spec file loads: `.rspec`, `spec/spec_helper.rb`, `spec/rails_helper.rb`, `spec/support/**/*.rb`, `test/test_helper.rb`, `test/support/**/*.rb`, the `preload` file, and `Gemfile.lock` / `gems.locked`;
|
|
640
|
+
- the evilution and Ruby versions;
|
|
641
|
+
- the settings that decide how a mutation is run and judged: `integration`, `isolation`, `profile`, `timeout` and the adaptive timeout's `timeout_factor` / `timeout_floor` / `timeout_ceiling`, `baseline`, the example-targeting settings, `related_specs_heuristic`, `fallback_to_full_suite`, `preload`, `stop_on_kill`, `killer_first`, `warmup`, whether uncovered mutations are skipped (`skip_uncovered` with coverage targeting), `probe_first` and `copy_worker_databases`.
|
|
642
|
+
|
|
643
|
+
So editing one method re-runs that method's mutations only, and a method that only moved because a method above it grew or shrank keeps its verdicts. Editing a constant, a callback, validation or scope declaration, any other class-body line (comments between methods included) or a spec file re-runs every subject of the file. A method defined twice in one file (say, in both branches of an `if`) keeps a verdict per definition.
|
|
644
|
+
|
|
645
|
+
What the cache does **not** see: other methods the subject calls (in the same file or another), other source files, and files a spec loads beyond the helpers above (factories, fixtures, shared examples outside `spec/support`), as well as `hooks`. A change there keeps the cached verdicts, so a survivor can stay a survivor after a test it depends on indirectly was fixed, or the other way around. Run without `--incremental` (or delete `tmp/evilution_cache/`) for a result that owes nothing to earlier runs, e.g. in CI.
|
|
646
|
+
|
|
647
|
+
Not cached: `uncovered` mutations, which are worked out again from the coverage map on every run (it has a cache of its own); survivors implied by a `--probe-first` sentinel; mutations whose source has no spec file that resolves (with or without `--fallback-full-suite`), errors, neutrals, kills from an infrastructure crash, and kills in which only examples already failing in the baseline failed.
|
|
648
|
+
|
|
649
|
+
The cache holds one JSON file per source file under `tmp/evilution_cache/v2/`, read once per run and written once per source file, so its cost grows with the number of mutations, not with their square. A file keeps one version of each subject: writing it drops the verdicts of a subject's earlier source and of subjects the file no longer has. Cache files of the earlier format (directly in `tmp/evilution_cache/`, keyed by the whole file's content) are not read, and are deleted the first time the new cache is written.
|
|
650
|
+
|
|
472
651
|
### Neutral Mutations
|
|
473
652
|
|
|
474
653
|
Neutral means the run has no verdict on a mutation. It covers two unrelated situations that want opposite responses, and each neutral records which; the report groups by it, naming the spec or the error class (GH #1606):
|
|
@@ -586,7 +765,7 @@ Each operator name is stable and appears in JSON output under `survived[].operat
|
|
|
586
765
|
| `forwarded_argument_drop` | Stop forwarding one part of the arguments a method passes through unnamed; a `...` signature is spelled out so a single part can be left out (skips a lone anonymous argument and named splats) | `g(*, **, &)` -> `g(**, &)`, `def f(...) = g(...)` -> `def f(*, **, &) = g(*, **)` |
|
|
587
766
|
| `pattern_wildcard_widening` | Widen a pattern so it accepts shapes it rejects: append a rest to an array pattern, wildcard or drop a hash-pattern pair, remove `**nil` (skips pairs that bind a variable and patterns that already have a rest) | `in [a, b]` -> `in [a, b, *]`, `in { age: Integer }` -> `in { age: _ }` |
|
|
588
767
|
| `no_matching_pattern_else` | Add an empty `else` to a `case/in` that has none, so an unmatched value yields `nil` instead of raising (skips a `case/in` with a bare capture clause, which matches everything) | `case v; in Integer then 1; end` -> `case v; in Integer then 1; else; end` |
|
|
589
|
-
| `argument_order_permutation` | Swap neighbouring positional arguments of a call, `super` or `yield` (keeps keywords, block and splats in place; skips identical neighbours, `raise` / `fail`, `format` / `sprintf` / `printf`, index writes, `start_with?` / `end_with?` and `Set[...]`). Calls that accept arguments in any order, such as `OptionParser#on`, can be silenced with `ignore_patterns`, e.g. `call{name=on, receiver=local_variable_read{name=opts}}` | `compute(a, b)` -> `compute(b, a)` |
|
|
768
|
+
| `argument_order_permutation` | Swap neighbouring positional arguments of a call, `super` or `yield` (keeps keywords, block and splats in place; skips identical neighbours, `raise` / `fail`, `format` / `sprintf` / `printf`, index writes, `start_with?` / `end_with?` and `Set[...]`). Calls that accept arguments in any order, such as `OptionParser#on`, can be silenced with `ignore_patterns`, e.g. `call{name=on, receiver=local_variable_read{name=opts}}`; the pattern ignores every mutation inside those calls, their handler blocks included. To keep a handler's logic mutated, write its body on its own lines (`do \|n\|` … `end`) and end the `opts.on` line with `# evilution:disable` instead | `compute(a, b)` -> `compute(b, a)` |
|
|
590
769
|
| `keyword_value_swap` | Swap the values of neighbouring keyword arguments, keeping the keys (keeps positional arguments, `**splat` and block in place; skips identical values and braced hash arguments; spells out shorthand keys) | `compute(x: a, y: b)` -> `compute(x: b, y: a)` |
|
|
591
770
|
| `comparison_operand_swap` | Swap the operands of a spaceship comparison, reversing a sort block or a custom ordering (skips identical operands and safe navigation; `between?` / `clamp` bounds are swapped by `argument_order_permutation`) | `x.age <=> y.age` -> `y.age <=> x.age` |
|
|
592
771
|
| `integer_division_to_fdiv` | Turn a division into `fdiv`, so integer division keeps its remainder (groups a compound dividend; skips float literals, `/=` and the explicit `./(...)` form) | `a / b` -> `a.fdiv(b)` |
|
|
@@ -594,7 +773,7 @@ Each operator name is stable and appears in JSON output under `survived[].operat
|
|
|
594
773
|
| `off_by_one_boundary` | Lower a count held in a variable by one: the receiver of `times`, the argument of `upto` / `downto` / `take` / `first` / `last` / `drop` / `each_slice` / `each_cons` (skips literal counts, which `integer_literal` shifts, and safe navigation) | `n.times` -> `(n - 1).times`, `items.first(n)` -> `items.first(n - 1)` |
|
|
595
774
|
| `format_specifier_swap` | Loosen one conversion specifier of a literal format string in `format` / `sprintf` / `printf` / `String#%`: drop flags, width and precision, or render a non-integer number as `%s` (skips `%%`, `%{name}`, `*` widths, `%c`, `%p` and interpolated or heredoc strings) | `format("%05d", n)` -> `format("%d", n)`, `"%.2f"` -> `"%f"`, `"%s"` |
|
|
596
775
|
| `alias_removal` | Drop an `alias` or `alias_method` declaration from a class, module or `class << self` body, attributed to the first method of that scope (skips aliases inside methods, `alias_method` sent to another receiver and global-variable aliases) | `alias length size` -> _(removed)_ |
|
|
597
|
-
| `freeze_removal` | Drop `.freeze` from a constant defined in a class or module body, attributed to the first method of that scope (skips values that are frozen anyway: symbols, numbers, `nil`, booleans, ranges, regexps, and string literals under `# frozen_string_literal: true`; a `freeze` inside a method is left to `method_call_removal`) | `LIST = %w[a b].freeze` -> `LIST = %w[a b]` |
|
|
776
|
+
| `freeze_removal` | Drop `.freeze` from a constant defined in a class or module body, attributed to the first method of that scope, or to the constant itself when the scope has no method of its own (skips values that are frozen anyway: symbols, numbers, `nil`, booleans, ranges, regexps, and string literals under `# frozen_string_literal: true`; a `freeze` inside a method is left to `method_call_removal`) | `LIST = %w[a b].freeze` -> `LIST = %w[a b]` |
|
|
598
777
|
| `visibility_removal` | Drop a `private`, `protected` or `module_function` declaration from a class, module or `class << self` body, attributed to the first method of that scope; a definition it wraps is kept (skips `public`, `private_constant`, `private_class_method`, and declarations guarded by a modifier) | `private` -> _(removed)_, `private def a; end` -> `def a; end` |
|
|
599
778
|
| `attr_accessor_swap` | Widen an `attr_reader` or `attr_writer` to `attr_accessor`, and remove an `attr_reader` / `attr_writer` / `attr_accessor` declaration, in a class, module or `class << self` body, attributed to the first method of that scope (skips a declaration wrapped in `private` / `protected`, or guarded by a modifier) | `attr_reader :name` -> `attr_accessor :name`, _(removed)_ |
|
|
600
779
|
| `sleep_to_zero` | Take the wait out of a `sleep` / `Kernel.sleep` whose duration is a variable or an expression (a literal duration is left to `integer_literal` / `float_literal`, which already emit `sleep 0`; skips a `sleep` sent to another receiver) | `sleep delay` -> `sleep 0` |
|
|
@@ -714,21 +893,21 @@ The `evilution-mutate` tool accepts a `verbosity` parameter to control response
|
|
|
714
893
|
|
|
715
894
|
| Level | Default | What's included |
|
|
716
895
|
|-------------|---------|--------------------------------------------------------------|
|
|
717
|
-
| `summary` | Yes | `summary` + `survived` + `timed_out` + `errors` + `unresolved` (
|
|
718
|
-
| `full` | | All entries (killed/neutral/equivalent diffs stripped)
|
|
719
|
-
| `minimal` | | `summary` + `survived` (plus a trimmed sample of up to 3 errored entries when `errors > 0`) |
|
|
896
|
+
| `summary` | Yes | `summary` + `survived` + `uncovered` + `timed_out` + `errors` + `unresolved` (entries other than `survived` and `uncovered` shed `diff` and `error_backtrace`; the run's `timings` block and every entry's `timings` are dropped, to bound payload size) |
|
|
897
|
+
| `full` | | All entries (killed/neutral/equivalent diffs stripped), including `timings` |
|
|
898
|
+
| `minimal` | | `summary` + `survived` + `uncovered` (plus a trimmed sample of up to 3 errored entries when `errors > 0`) |
|
|
720
899
|
|
|
721
|
-
Use `minimal` when context window budget is tight and you only need to see what survived. The trimmed `errors` sample (each entry: `error_message`, `error_class`, location, plus the first 5 backtrace lines) is added so a partly-broken run is still self-diagnosable without escalating verbosity. Use `full` when you need to inspect killed/neutral/equivalent entries for debugging.
|
|
900
|
+
Use `minimal` when context window budget is tight and you only need to see what survived (an `uncovered` mutation is a survivor no test executes, see [Uncovered mutations](#uncovered-mutations)). The trimmed `errors` sample (each entry: `error_message`, `error_class`, location, plus the first 5 backtrace lines) is added so a partly-broken run is still self-diagnosable without escalating verbosity. Use `full` when you need to inspect killed/neutral/equivalent entries for debugging.
|
|
722
901
|
|
|
723
902
|
What survives trimming matters when you are deciding whether to trust a score:
|
|
724
903
|
|
|
725
904
|
- `neutral` entries — and with them each `neutral_reason` — are dropped at `summary` and `minimal`. Use `full` to see why mutations were neutralised.
|
|
726
|
-
- `subjects` is kept at `full` and `summary`, and dropped at `minimal`, which keeps only `summary` and `
|
|
905
|
+
- `subjects` is kept at `full` and `summary`, and dropped at `minimal`, which keeps only `summary`, `survived` and `uncovered`.
|
|
727
906
|
- Everything inside `summary` survives at every level, including `unresolved_target_files`, `uncovered_code`, `infra_retried`, `baseline_neutralized`, `baseline_failures` and the `neutral` count — so even a `minimal` response still says whether a target file went untested and how much of the run the score covers.
|
|
728
907
|
|
|
729
908
|
### Enriched Survived Entries
|
|
730
909
|
|
|
731
|
-
Unlike `evilution --format json`, every survived entry returned by `evilution-mutate` carries extra fields so the agent can act without a second round-trip:
|
|
910
|
+
Unlike `evilution --format json`, every survived and uncovered entry returned by `evilution-mutate` carries extra fields so the agent can act without a second round-trip:
|
|
732
911
|
|
|
733
912
|
| Field | What it gives you |
|
|
734
913
|
|---|---|
|
|
@@ -756,11 +935,14 @@ Pass `skip_config: true` to ignore the project config file. This skips loading `
|
|
|
756
935
|
|
|
757
936
|
| Parameter | Purpose |
|
|
758
937
|
|---|---|
|
|
759
|
-
| `incremental` | Cache killed
|
|
938
|
+
| `incremental` | Cache verdicts (killed, survived, timeout) across runs — set `true` when iterating on the same files; editing one method re-runs only that method's mutations |
|
|
760
939
|
| `integration` | `rspec`, `minitest`, or `test_unit` |
|
|
761
940
|
| `isolation` | `auto`, `fork`, or `in_process` |
|
|
762
941
|
| `baseline` | `false` to skip the baseline suite check when you already know it's green |
|
|
763
942
|
| `save_session` | Persist results to `.evilution/results/` for inspection via `evilution-session` |
|
|
943
|
+
| `stop_on_kill` | `false` to run every selected example instead of stopping at the first deciding failure (default `true`) |
|
|
944
|
+
| `skip_sites` | Presets of low-value code to leave unmutated: `logging`, `debug_output`, `error_messages`, `requires` (see [Skipping Low-Value Code](#skipping-low-value-code)); skips are counted per preset in `summary.skipped_sites` |
|
|
945
|
+
| `since` | A git ref: mutate only the lines changed since it, as `--since REF` does (see [Mutating only what changed](#mutating-only-what-changed---since)) |
|
|
764
946
|
|
|
765
947
|
> **Note**: `.mcp.json` is gitignored by default since it is a local editor/agent configuration file.
|
|
766
948
|
|
|
@@ -813,8 +995,8 @@ Per-tool placement:
|
|
|
813
995
|
- **`evilution-session` `diff`** — `{ "schema_version": Integer, "summary": { base_score, head_score, score_delta, base_survived, head_survived, base_total, head_total, base_killed, head_killed }, "fixed": Array, "new_survivors": Array, "persistent": Array }`. The mutation arrays carry the same per-mutation fields the session `survived` list uses (`operator`, `file`, `line`, `subject`, `diff`).
|
|
814
996
|
- **`evilution-info` `subjects`** — `{ "schema_version": Integer, "subjects": Array<{ name, file, line, mutations }>, "total_subjects": Integer, "total_mutations": Integer }`. Discovery only: this lists what *can* be mutated. The `subjects` array in a run's report is a different shape, carrying what each subject scored.
|
|
815
997
|
- **`evilution-info` `tests`** — `{ "schema_version": Integer, "specs": Array<{ source, spec }>, "unresolved": Array<String>, "total_sources": Integer, "total_specs": Integer }`.
|
|
816
|
-
- **`evilution-info` `environment`** — `{ "schema_version": Integer, "version": String, "ruby": String, "config_file": String|null,
|
|
817
|
-
- **`evilution-info` `statuses`** — `{ "schema_version": Integer, "statuses": Array<{
|
|
998
|
+
- **`evilution-info` `environment`** — `{ "schema_version": Integer, "version": String, "ruby": String, "config_file": String|null, "settings": Object }`. `settings` holds the effective value of each run setting under its config key, plus `adaptive_timeout`: when it is `true`, `timeout` is only the fallback and each mutation's limit follows the baseline (`timeout_factor`, `timeout_floor`, `timeout_ceiling`).
|
|
999
|
+
- **`evilution-info` `statuses`** — `{ "schema_version": Integer, "statuses": Array<{ status, meaning, counted_in_score }> }`.
|
|
818
1000
|
- **`evilution-info` `feedback`** — `{ "schema_version": Integer, "discussion_url": String, "consent": String, "privacy": String }`.
|
|
819
1001
|
|
|
820
1002
|
#### Deprecation cycle
|
|
@@ -839,7 +1021,7 @@ When a parameter, action, or output field on the public MCP contract is deprecat
|
|
|
839
1021
|
### 1. Full project scan
|
|
840
1022
|
|
|
841
1023
|
```bash
|
|
842
|
-
bundle exec evilution run lib
|
|
1024
|
+
bundle exec evilution run --target 'source:lib/**/*.rb' --format json --min-score 0.8
|
|
843
1025
|
```
|
|
844
1026
|
|
|
845
1027
|
Parse JSON output. Exit code 0 = pass, 1 = fail — either the score is below `--min-score`, or a target file resolved to no spec and was never tested (`summary.unresolved_target_files` names them). Without `--min-score` no score gate is armed; the run still fails on an untested target file.
|
|
@@ -852,6 +1034,12 @@ bundle exec evilution run lib/foo.rb:15-30 lib/bar.rb:5-20 --format json --min-s
|
|
|
852
1034
|
|
|
853
1035
|
Target the exact lines you changed for fast, focused mutation testing. See line-range syntax below.
|
|
854
1036
|
|
|
1037
|
+
Or let git find the lines: `--since REF` mutates only what changed since `REF` (see [Mutating only what changed](#mutating-only-what-changed---since)):
|
|
1038
|
+
|
|
1039
|
+
```bash
|
|
1040
|
+
bundle exec evilution run --since origin/main --format json --min-score 0.9
|
|
1041
|
+
```
|
|
1042
|
+
|
|
855
1043
|
### 3. Line-range targeted scan (fastest)
|
|
856
1044
|
|
|
857
1045
|
```bash
|
|
@@ -873,14 +1061,16 @@ Methods whose body overlaps the requested range are included. Mix targeted and w
|
|
|
873
1061
|
evilution run lib/foo.rb:15-30 lib/bar.rb --format json
|
|
874
1062
|
```
|
|
875
1063
|
|
|
876
|
-
A range is matched against subjects, and not every line belongs to one. Methods are subjects, and so are scopes, AASM guards and callbacks, callback and validation declarations, `Data.define` / `Struct.new` definitions and constant assignments (see step 2 under [Internals](#internals-for-context-not-for-direct-use)). Other class-body code — a
|
|
1064
|
+
A range is matched against subjects, and not every line belongs to one. Methods are subjects, and so are scopes, AASM guards and callbacks, callback and validation declarations given options or an inline block or lambda (`validates :title, presence: true`, `validate :limit, if: -> { paid? }`), `Data.define` / `Struct.new` definitions and constant assignments (see step 2 under [Internals](#internals-for-context-not-for-direct-use)). Other class-body code — an association, a declaration naming only a method (`validate :within_window`), a DSL call evilution does not know — is not. When the range holds such code, the run says so on stderr, in the text report and in `summary.uncovered_code`:
|
|
877
1065
|
|
|
878
1066
|
```
|
|
879
|
-
[evilution] app/models/order.rb:12-14 holds code outside every subject
|
|
1067
|
+
[evilution] app/models/order.rb:12-14 holds code outside every subject; no mutations target those lines. Of the DSL calls in a class body, only scopes, aasm machines, and callback or validation declarations with options or an inline block or lambda are mutated.
|
|
880
1068
|
```
|
|
881
1069
|
|
|
882
1070
|
Read `0 mutations` next to that message as "not measured", not as "nothing to test".
|
|
883
1071
|
|
|
1072
|
+
Callback and validation declarations with options add to every model, controller and mailer that has them: about 10–20 mutations for each `validates` line. Some of those make the declaration invalid (`length: {}`, `inclusion: { in: nil }`, `validates nil, ...`), so the class raises while it loads, and they count as killed like any other mutation that stops the file loading. Each such mutation also runs the declaration once more as written, so option values computed in the class body (`in: Status.pluck(:name)`) are computed again, and a third-party `validates_*` macro that keeps its own list of attributes may list it once more. To leave a declaration out, put `# evilution:disable` on its line or around it (see [Disable Comments](#disable-comments)). Declarations of one class that read the same — the conditions and the options of one declaration, or two `validates :email` lines — share one subject name, so `subjects[]` reports them together under one score.
|
|
1073
|
+
|
|
884
1074
|
### 4. Method-name targeted scan
|
|
885
1075
|
|
|
886
1076
|
```bash
|
|
@@ -913,7 +1103,7 @@ A score describes only the mutations that got a verdict. Five fields say what it
|
|
|
913
1103
|
|---|---|---|
|
|
914
1104
|
| `summary.unresolved_target_files` | A file you named resolved to no spec and was never tested; the run fails on this alone | Write a spec, pass `--spec`, or map it in `spec_mappings` — do not trust the score until this is empty |
|
|
915
1105
|
| `subjects[].reached == false` | Mutations were generated for that method but none got a verdict | The method is untested even where its file scores well; start here rather than with `survived[]` |
|
|
916
|
-
| `summary.uncovered_code` | Lines you targeted hold code outside every subject — class-body code such as DSL calls evilution does not treat as a subject — so no mutation was generated for them; the same is printed to stderr | A run that reports `0 mutations` for those lines has not shown they are tested. Target the methods that code calls, or cover it with ordinary tests |
|
|
1106
|
+
| `summary.uncovered_code` | Lines you targeted hold code outside every subject — class-body code such as DSL calls evilution does not treat as a subject — so no mutation was generated for them; the same is printed to stderr and listed in the text report | A run that reports `0 mutations` for those lines has not shown they are tested. Target the methods that code calls, or cover it with ordinary tests |
|
|
917
1107
|
| `neutral[].neutral_reason.kind == "baseline_failure"` | The mutation's tests failed only on examples that were already failing before any mutation ran; `detail` names the red spec file | Read `summary.baseline_failures` for why it was red, and fix that first — these mutations have no verdict until it is green |
|
|
918
1108
|
| `neutral[].neutral_reason.kind == "infra_error"` | The test process crashed on infrastructure (DB lock, timeout); `detail` names the class | Not a coverage gap. Give parallel workers their own database, or run `-j 1` |
|
|
919
1109
|
|
|
@@ -930,6 +1120,25 @@ For each entry in `survived[]`:
|
|
|
930
1120
|
4. Write a test that would fail if the mutation were applied
|
|
931
1121
|
5. Re-run evilution on just that file to verify the mutant is now killed
|
|
932
1122
|
|
|
1123
|
+
#### A test exists but the mutant still survives: check the matcher
|
|
1124
|
+
|
|
1125
|
+
Inside `have_attributes`, `match` and nested values of `include(key: [...])`, RSpec compares an expected array against an `Enumerable` actual (other than a Hash or Struct) through its `to_a`. `ActiveModel::Errors#to_a` is `full_messages`, so `have_attributes(errors: ["Name can't be blank"])` passes for *any* errors object with that message, not just the one the code should return. A mutant that hands back a different errors object (another record's, or one built with a message instead of an error kind) survives that test.
|
|
1126
|
+
|
|
1127
|
+
When the errors object itself matters, assert on the error kinds, or on identity:
|
|
1128
|
+
|
|
1129
|
+
```ruby
|
|
1130
|
+
# Before: passes for any errors object whose full messages match
|
|
1131
|
+
expect(result).to have_attributes(errors: ["Name can't be blank"])
|
|
1132
|
+
|
|
1133
|
+
# After: the error kind, not just its rendered message
|
|
1134
|
+
expect(result.errors.details).to eq(name: [{ error: :blank }])
|
|
1135
|
+
expect(result.errors).to be_of_kind(:name, :blank)
|
|
1136
|
+
# or, when it must be the record's own errors object
|
|
1137
|
+
expect(result.errors).to be(participation.errors)
|
|
1138
|
+
```
|
|
1139
|
+
|
|
1140
|
+
`details` and `of_kind?` still accept another record's errors with the same kinds; only the identity check (`be`) rules that out. The same `to_a` comparison applies to other `Enumerable` values (except Hash and Struct) checked against an array inside these matchers.
|
|
1141
|
+
|
|
933
1142
|
### 7. Diagnosing errored mutations
|
|
934
1143
|
|
|
935
1144
|
Entries in the JSON `errors[]` array represent mutations that raised an exception (syntax error, load failure, or runtime crash) rather than producing a test outcome. A mutation that makes its own file raise while loading is not among them when the original file still loads right after it: the mutation is what broke the load, so it is reported `killed` with an `error_message` that starts with `load failed:`. A load failure stays an error when the original fails to load as well, since that says nothing about the mutation. Each entry includes `error_class`, `error_message`, and the first 5 `error_backtrace` lines. Use these fields to decide whether the error is a bug in the mutation operator (file an issue), a load-time problem in the mutated source (often `NoMethodError: super called outside of method` or constant-redefinition issues), or a genuine crash that the original tests should have caught. Run with `--verbose` to stream the same error details to stderr during the run.
|
|
@@ -999,7 +1208,7 @@ RUBYOPT="-Itest" bundle exec evilution mutate lib/<file>.rb \
|
|
|
999
1208
|
### 8. CI gate
|
|
1000
1209
|
|
|
1001
1210
|
```bash
|
|
1002
|
-
bundle exec evilution run lib
|
|
1211
|
+
bundle exec evilution run --target 'source:lib/**/*.rb' --format json --min-score 0.8 --quiet
|
|
1003
1212
|
# Exit code 0 = pass, 1 = fail, 2 = error
|
|
1004
1213
|
```
|
|
1005
1214
|
|
|
@@ -1010,9 +1219,9 @@ Note: `--quiet` suppresses all stdout output (including JSON). Use it in CI only
|
|
|
1010
1219
|
### 9. Regression tracking across runs (`compare`)
|
|
1011
1220
|
|
|
1012
1221
|
```bash
|
|
1013
|
-
bundle exec evilution run lib
|
|
1222
|
+
bundle exec evilution run --target 'source:lib/**/*.rb' --format json --save-session
|
|
1014
1223
|
# later, after edits:
|
|
1015
|
-
bundle exec evilution run lib
|
|
1224
|
+
bundle exec evilution run --target 'source:lib/**/*.rb' --format json --save-session
|
|
1016
1225
|
|
|
1017
1226
|
bundle exec evilution compare \
|
|
1018
1227
|
--against .evilution/results/<earlier>.json \
|
|
@@ -1049,17 +1258,38 @@ Do not warm up by sending a real request from your preload (`Rails.application.c
|
|
|
1049
1258
|
|
|
1050
1259
|
## Parallel Runs with SQLite
|
|
1051
1260
|
|
|
1052
|
-
Running with `-j N` forks worker processes. If your Rails app uses SQLite
|
|
1261
|
+
Running with `-j N` forks worker processes. If your Rails app uses SQLite and every worker opens the same `db/test.sqlite3` file, concurrent writers collide on the database-level lock (`ActiveRecord::StatementTimeout`, `SQLite3::BusyException`, slow runs), and, worse, one worker's fixture loads and writes land in the middle of another worker's tests. Verdicts then depend on which mutations happen to run at the same time: on a Redmine model, a shared database turned 47 of 500 survivors into kills at `-j 4` that a `-j 1` run does not reproduce.
|
|
1262
|
+
|
|
1263
|
+
### Per-worker database copies
|
|
1264
|
+
|
|
1265
|
+
By default each worker gets a database of its own, with no setup. When the app is preloaded in the parent (the default for a Rails app under fork isolation, see `--preload`) and `--jobs` is above 1, a worker that would share the preloaded SQLite test database makes a copy of it when it starts and reconnects `ActiveRecord::Base` to the copy. The copy goes through SQLite's backup API, so it holds what the write-ahead log holds and is no larger than the data; it is taken after the preload and the baseline, from a database no worker writes to, so every worker starts from the same schema and data. A worker started in place of a dead or recycled one makes a fresh copy. The `ActiveRecord::Base.configurations` entry is pointed at the copy too, so `establish_connection(:test)` in a test stays on it.
|
|
1266
|
+
|
|
1267
|
+
The copies live in a directory under the system temp dir (`evilution-worker-db-<pid>-<random>/worker-<worker pid>/`) and are deleted when the parallel pass ends, also when it ends on an error or Ctrl-C. A worker's copy is deleted as soon as the worker is gone and before its replacement starts, so the disk never holds more than one copy per job; only a run killed with `SIGKILL` leaves the directory behind. The database in `database.yml` is not written by the workers.
|
|
1268
|
+
|
|
1269
|
+
A worker copies only when all of these hold:
|
|
1053
1270
|
|
|
1054
|
-
|
|
1271
|
+
- the app was loaded by the preload, so the worker inherits an `ActiveRecord::Base` connection;
|
|
1272
|
+
- that connection still matches `database.yml` (a connection your test helper set up by other means is left alone);
|
|
1273
|
+
- the adapter is SQLite and the database is a file path, not `:memory:` (never shared anyway) or a `file:` URI;
|
|
1274
|
+
- the worker would otherwise use the preloaded database: `database.yml` names one database for all workers, or the worker's own `TEST_ENV_NUMBER` database (below) is not prepared.
|
|
1275
|
+
|
|
1276
|
+
Not covered yet: a run with `--no-preload` (the app boots in each mutation's process and reads `database.yml` there, so use `TEST_ENV_NUMBER` below), connections other than `ActiveRecord::Base`'s (other databases set up with `connects_to`, replicas keep using the shared databases, and each worker says so in a warning), and PostgreSQL or MySQL, where a per-worker database would come from a template database; for those, set up per-worker databases with `TEST_ENV_NUMBER`. If a copy fails (a full disk, a locked database), the worker warns and shares the preloaded database as before.
|
|
1277
|
+
|
|
1278
|
+
Turn it off with `--no-copy-worker-databases` or `copy_worker_databases: false`. The setting is part of the [incremental cache](#incremental-cache) fingerprint, since it can change verdicts.
|
|
1279
|
+
|
|
1280
|
+
### Infrastructure crashes
|
|
1281
|
+
|
|
1282
|
+
Contention that still happens (copies off, or a run the copies do not cover) is handled after the fact. Evilution classifies these crashes as `:neutral` (see [GH #814](https://github.com/marinazzio/evilution/issues/814)) so the mutation score is not polluted, and because the contention only exists while several workers are running, those mutations are re-run one at a time once the pool is done, and the verdict from the quiet re-run is the one reported. Without it the neutral bucket moved with `-j` on identical input — the same files scoring 24 killed / 0 neutral at `-j 1` and 6 killed / 18 neutral at `-j 4` (GH #1607). The run says how much it had to redo:
|
|
1055
1283
|
|
|
1056
1284
|
```
|
|
1057
1285
|
! 18 mutations hit infrastructure errors under parallel workers; re-ran them serially.
|
|
1058
1286
|
```
|
|
1059
1287
|
|
|
1060
|
-
The count is in JSON output as `summary.infra_retried`. Such a crash is also never written to the `--incremental` cache: the cache keeps no error class, so a cached `:killed` would be indistinguishable from a real one on the next run and would short-circuit both the demotion and the retry. A neutral from a failing baseline is never re-run — that is a real statement about the spec, not a missed verdict. The retry costs wall-clock time in proportion to the contention,
|
|
1288
|
+
The count is in JSON output as `summary.infra_retried`. Such a crash is also never written to the `--incremental` cache: the cache keeps no error class, so a cached `:killed` would be indistinguishable from a real one on the next run and would short-circuit both the demotion and the retry. A neutral from a failing baseline is never re-run — that is a real statement about the spec, not a missed verdict. The retry costs wall-clock time in proportion to the contention, and it does not undo a wrong kill: a test that failed on another worker's data rather than on a lock is not an infrastructure crash. Give each worker its own database instead.
|
|
1289
|
+
|
|
1290
|
+
### Your own per-worker databases (TEST_ENV_NUMBER)
|
|
1061
1291
|
|
|
1062
|
-
Evilution follows the [`parallel_tests`](https://github.com/grosser/parallel_tests) convention: each worker receives a `TEST_ENV_NUMBER` environment variable (`""` for worker 1, `"2"` for worker 2, `"3"` for worker 3, …). Interpolate it into `config/database.yml` so each worker gets its own
|
|
1292
|
+
Evilution follows the [`parallel_tests`](https://github.com/grosser/parallel_tests) convention: each worker receives a `TEST_ENV_NUMBER` environment variable (`""` for worker 1, `"2"` for worker 2, `"3"` for worker 3, …). Interpolate it into `config/database.yml` so each worker gets its own database:
|
|
1063
1293
|
|
|
1064
1294
|
```yaml
|
|
1065
1295
|
test:
|
|
@@ -1069,9 +1299,19 @@ test:
|
|
|
1069
1299
|
timeout: 5000
|
|
1070
1300
|
```
|
|
1071
1301
|
|
|
1072
|
-
|
|
1302
|
+
Each worker needs its own prepared database (`db/test.sqlite3`, `db/test2.sqlite3`, …, one per job). Create and load them before running Evilution:
|
|
1303
|
+
|
|
1304
|
+
```bash
|
|
1305
|
+
bin/rails db:test:prepare
|
|
1306
|
+
TEST_ENV_NUMBER=2 bin/rails db:test:prepare
|
|
1307
|
+
TEST_ENV_NUMBER=3 bin/rails db:test:prepare # … up to the number of jobs
|
|
1308
|
+
```
|
|
1309
|
+
|
|
1310
|
+
Without `--preload`, a test helper that loads the schema when it boots (for example `ActiveRecord::Migration.maintain_test_schema!` in `rails_helper.rb`) can do this instead, because it runs inside each worker.
|
|
1311
|
+
|
|
1312
|
+
Per-worker databases also work with `--preload`, which boots the app (and resolves `database.yml`) once in the parent before the workers fork: each worker re-renders `database.yml` with its own `TEST_ENV_NUMBER` and, when the result differs, re-establishes `ActiveRecord::Base`'s connection to it. Only the primary connection is switched, and only when it still matches `database.yml`. Under `--preload` the preload's own schema setup runs once, in the parent, against the parent's database only, so prepare the worker databases with the commands above. A worker whose database is missing or lacks a table or migration of the parent's database goes back to the parent's database and prints which database to prepare; with [per-worker copies](#per-worker-database-copies) on, it then uses a copy of the parent's SQLite database, otherwise its mutations share the parent's database as they would without the switch. Worker 1's `TEST_ENV_NUMBER` is empty, so its database is the parent's; with copies on it uses a copy of it, like any worker left on the parent's database.
|
|
1073
1313
|
|
|
1074
|
-
When Evilution detects a parallel run against a SQLite-
|
|
1314
|
+
When Evilution detects a parallel run against a SQLite test database whose path does not depend on `TEST_ENV_NUMBER` and the per-worker copies do not cover it (they are off, the app was not preloaded, or the preloaded connection is not one they copy: a `file:` URI, or a connection that no longer matches `database.yml`), it prints a one-time startup warning pointing to this section. An in-memory database (`:memory:`) is never shared between workers and does not trigger it.
|
|
1075
1315
|
|
|
1076
1316
|
## Development
|
|
1077
1317
|
|
|
@@ -1100,7 +1340,7 @@ For the full contributor architecture — module map, data flow, and extension
|
|
|
1100
1340
|
points — see [docs/architecture.md](docs/architecture.md).
|
|
1101
1341
|
|
|
1102
1342
|
1. **Parse** — Prism parses Ruby files into ASTs with exact byte offsets
|
|
1103
|
-
2. **Extract** — Methods are identified as mutation subjects, and so are value-object definitions outside any method (`Point = Data.define(:x, :y)`, `class Coord < Struct.new(:lat, :lng)`), which only the operators that apply to them mutate, every other constant assignment outside a method (`LIMIT = 10`, `LIMIT += 1`), named after the constant (`Config::LIMIT`); `constant_write_to_nil` mutates a plain assignment, `compound_assignment` an operator write, and the literal operators the value of either, and ActiveRecord scopes declared with a literal body in a class body (`scope :recent, -> { where(recent: true) }`), named after the class method they define (`Order.recent`) and mutated like a method body — also when declared in a concern's `included do ... end` or `prepended do ... end` block, where they are named after the concern (`Publishable.published`). A method defined in a concern's `included do ... end` or `prepended do ... end` block is mutated on the classes that took the concern before the run as well as on those that take it later; likewise the guards and callbacks written out as lambdas or blocks inside an AASM `event` or `state` declaration (`event :ship, guard: -> { address? }`), named after the method that declaration defines (`Order#ship`, `Order#paid?`), or after the concern when the machine is declared in its `included do ... end` or `prepended do ... end` block (`Shippable#ship`); and the conditions and bodies written out in callback and validation declarations of a class body or of a concern's `included do ... end` or `prepended do ... end` block (`validate :credit_limit, if: -> { paid? }`, `before_save { ... }`), named the way the declaration reads (`Order.validate(:credit_limit)`, `Order.before_save`, `Publishable.validates(:title)`)
|
|
1343
|
+
2. **Extract** — Methods are identified as mutation subjects, and so are value-object definitions outside any method (`Point = Data.define(:x, :y)`, `class Coord < Struct.new(:lat, :lng)`), which only the operators that apply to them mutate, every other constant assignment outside a method (`LIMIT = 10`, `LIMIT += 1`), named after the constant (`Config::LIMIT`); `constant_write_to_nil` mutates a plain assignment, `compound_assignment` an operator write, and the literal operators the value of either, and ActiveRecord scopes declared with a literal body in a class body (`scope :recent, -> { where(recent: true) }`), named after the class method they define (`Order.recent`) and mutated like a method body — also when declared in a concern's `included do ... end` or `prepended do ... end` block, where they are named after the concern (`Publishable.published`). A method defined in a concern's `included do ... end` or `prepended do ... end` block is mutated on the classes that took the concern before the run as well as on those that take it later; likewise the guards and callbacks written out as lambdas or blocks inside an AASM `event` or `state` declaration (`event :ship, guard: -> { address? }`), named after the method that declaration defines (`Order#ship`, `Order#paid?`), or after the concern when the machine is declared in its `included do ... end` or `prepended do ... end` block (`Shippable#ship`); and the conditions and bodies written out in callback and validation declarations of a class body or of a concern's `included do ... end` or `prepended do ... end` block (`validate :credit_limit, if: -> { paid? }`, `before_save { ... }`), named the way the declaration reads (`Order.validate(:credit_limit)`, `Order.before_save`, `Publishable.validates(:title)`); and the arguments of such a declaration given options (`validates :for_lift, inclusion: { in: [true] }, if: :lift_round?`), under the same name — its literals, option values and conditions are mutated and options dropped one at a time. Such a mutation is applied by running the declaration again: what it registered when the file loaded, found by running the declaration as written once more, gives its place to what the mutated declaration registers, or to nothing when it registers none (`presence: false`)
|
|
1104
1344
|
3. **Filter** — Disable comments, Sorbet `sig` blocks, and AST ignore patterns exclude mutations before execution
|
|
1105
1345
|
4. **Mutate** — 155 operators produce text replacements at precise byte offsets (source-level surgery, no AST unparsing); heredoc literal text is skipped by default. Identical byte-mutations from different operators are deduplicated by `(file_path, mutated_source)` so the count is not inflated by overlap
|
|
1106
1346
|
5. **Isolate** — Mutations are applied to temporary file copies (never modifying originals); load-path redirection ensures `require` resolves the mutated copy. Default isolation is in-process for plain Ruby projects (no gemspec) and fork for Rails projects and packaged gems (auto-detected); `--isolation fork` forces forked child processes. Both sequential and parallel (`--jobs N`) modes respect the configured isolation strategy
|