evilution 1.5.1 → 1.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.rubocop_todo.yml +30 -0
- data/CHANGELOG.md +49 -0
- data/README.md +285 -45
- data/docs/architecture.md +110 -12
- data/docs/ast_pattern_syntax.md +35 -7
- data/docs/combined_runs_spike.md +407 -0
- data/docs/integrations.md +1 -1
- data/docs/isolation.md +63 -2
- data/docs/migration-from-mutant.md +16 -11
- data/docs/mutant_schemata_spike.md +387 -0
- data/docs/operator_yield_study.md +576 -0
- data/docs/probe_first_spike.md +264 -0
- data/lib/evilution/ast/ancestor_lookup.rb +36 -0
- data/lib/evilution/ast/callback_declaration.rb +36 -3
- data/lib/evilution/ast/class_body.rb +62 -16
- data/lib/evilution/ast/def_region.rb +130 -0
- data/lib/evilution/ast/local_declaration.rb +37 -0
- data/lib/evilution/ast/parent_index.rb +50 -0
- data/lib/evilution/ast/parse_cache.rb +27 -0
- data/lib/evilution/ast/parser.rb +8 -3
- data/lib/evilution/ast/pattern/filter.rb +53 -12
- data/lib/evilution/ast/skip_filter.rb +83 -0
- data/lib/evilution/ast/skip_sites/debug_output.rb +67 -0
- data/lib/evilution/ast/skip_sites/error_messages.rb +74 -0
- data/lib/evilution/ast/skip_sites/logging.rb +93 -0
- data/lib/evilution/ast/skip_sites/message.rb +44 -0
- data/lib/evilution/ast/skip_sites/nodes.rb +105 -0
- data/lib/evilution/ast/skip_sites/presets.rb +31 -0
- data/lib/evilution/ast/skip_sites/purity.rb +69 -0
- data/lib/evilution/ast/skip_sites/requires.rb +40 -0
- data/lib/evilution/ast/skip_sites.rb +9 -0
- data/lib/evilution/ast/source_surgeon.rb +20 -2
- data/lib/evilution/ast/syntax_check.rb +10 -8
- data/lib/evilution/baseline/report.rb +11 -2
- data/lib/evilution/baseline.rb +31 -6
- data/lib/evilution/cache/keys.rb +140 -0
- data/lib/evilution/cache/run_fingerprint.rb +51 -0
- data/lib/evilution/cache/skeleton.rb +57 -0
- data/lib/evilution/cache/spec_digest.rb +39 -0
- data/lib/evilution/cache.rb +143 -50
- data/lib/evilution/class_tree.rb +38 -0
- data/lib/evilution/cli/command.rb +12 -4
- data/lib/evilution/cli/commands/run.rb +12 -3
- data/lib/evilution/cli/commands/subjects.rb +5 -2
- data/lib/evilution/cli/parser/options_builder.rb +67 -3
- data/lib/evilution/cli/printers/environment.rb +34 -4
- data/lib/evilution/cli/printers/session_detail.rb +20 -3
- data/lib/evilution/compare/categorizer.rb +2 -1
- data/lib/evilution/compare/normalizer.rb +3 -2
- data/lib/evilution/config/sources.rb +3 -0
- data/lib/evilution/config/validators/base.rb +9 -0
- data/lib/evilution/config/validators/since.rb +22 -0
- data/lib/evilution/config/validators/skip_sites.rb +24 -0
- data/lib/evilution/config/validators/timeout_ceiling.rb +10 -0
- data/lib/evilution/config/validators/timeout_factor.rb +10 -0
- data/lib/evilution/config/validators/timeout_floor.rb +10 -0
- data/lib/evilution/config.rb +122 -6
- data/lib/evilution/coverage/load_tracker.rb +50 -0
- data/lib/evilution/coverage/map.rb +42 -5
- data/lib/evilution/coverage/map_builder.rb +123 -22
- data/lib/evilution/coverage/map_store.rb +15 -5
- data/lib/evilution/coverage/recorder.rb +42 -3
- data/lib/evilution/coverage_example_filter.rb +35 -0
- data/lib/evilution/equivalent/heuristic/comment_marking.rb +14 -1
- data/lib/evilution/example_filter.rb +20 -0
- data/lib/evilution/git/changed_lines.rb +103 -0
- data/lib/evilution/git/since_scope.rb +77 -0
- data/lib/evilution/git/unified_diff.rb +68 -0
- data/lib/evilution/infra_crash.rb +26 -0
- data/lib/evilution/integration/base.rb +97 -6
- data/lib/evilution/integration/crash_detector.rb +46 -6
- data/lib/evilution/integration/known_failures.rb +4 -0
- data/lib/evilution/integration/loading/body_call_neutralizer.rb +49 -11
- data/lib/evilution/integration/loading/callback_equivalence.rb +62 -0
- data/lib/evilution/integration/loading/callback_redeclaration.rb +151 -36
- data/lib/evilution/integration/loading/concern_redeclaration.rb +16 -12
- data/lib/evilution/integration/loading/concern_registry.rb +87 -0
- data/lib/evilution/integration/loading/concern_state_cleaner.rb +19 -8
- data/lib/evilution/integration/loading/created_modules.rb +56 -0
- data/lib/evilution/integration/loading/frozen_namespaces.rb +80 -0
- data/lib/evilution/integration/loading/mutation_applier.rb +42 -11
- data/lib/evilution/integration/loading/neutralization.rb +41 -0
- data/lib/evilution/integration/loading/redefinition_recovery.rb +5 -3
- data/lib/evilution/integration/loading/superseded_modules.rb +22 -19
- data/lib/evilution/integration/minitest/stop_hook.rb +57 -0
- data/lib/evilution/integration/minitest.rb +22 -2
- data/lib/evilution/integration/minitest_crash_detector.rb +19 -0
- data/lib/evilution/integration/rspec/baseline_runner.rb +3 -2
- data/lib/evilution/integration/rspec/crash_detector_lifecycle.rb +6 -1
- data/lib/evilution/integration/rspec/example_filter_applier.rb +16 -0
- data/lib/evilution/integration/rspec/example_ids.rb +23 -0
- data/lib/evilution/integration/rspec/example_order.rb +86 -0
- data/lib/evilution/integration/rspec/killer_first.rb +72 -0
- data/lib/evilution/integration/rspec/result_builder.rb +7 -0
- data/lib/evilution/integration/rspec/state_guard/example_group_descendants.rb +44 -0
- data/lib/evilution/integration/rspec/state_guard.rb +2 -2
- data/lib/evilution/integration/rspec/target_resolver.rb +42 -0
- data/lib/evilution/integration/rspec/test_file_resolver.rb +4 -2
- data/lib/evilution/integration/rspec.rb +98 -47
- data/lib/evilution/integration/stop_policy.rb +49 -0
- data/lib/evilution/integration/test_unit/dispatcher.rb +20 -1
- data/lib/evilution/integration/test_unit/subject_class_registry.rb +6 -1
- data/lib/evilution/integration/test_unit.rb +28 -2
- data/lib/evilution/isolation/fork.rb +58 -13
- data/lib/evilution/isolation/in_process.rb +12 -1
- data/lib/evilution/line_set.rb +72 -0
- data/lib/evilution/mcp/info_tool/actions/environment.rb +12 -16
- data/lib/evilution/mcp/info_tool/actions/subjects.rb +3 -5
- data/lib/evilution/mcp/info_tool/status_glossary.rb +15 -2
- data/lib/evilution/mcp/info_tool.rb +24 -17
- data/lib/evilution/mcp/mutate_tool/config_builder.rb +13 -1
- data/lib/evilution/mcp/mutate_tool/option_parser.rb +2 -2
- data/lib/evilution/mcp/mutate_tool/progress_streamer.rb +1 -1
- data/lib/evilution/mcp/mutate_tool/report_trimmer.rb +28 -5
- data/lib/evilution/mcp/mutate_tool/survived_enricher.rb +12 -7
- data/lib/evilution/mcp/mutate_tool.rb +32 -7
- data/lib/evilution/mcp/session_tool.rb +1 -2
- data/lib/evilution/mcp.rb +4 -0
- data/lib/evilution/mutation/line_diff.rb +111 -0
- data/lib/evilution/mutation.rb +17 -13
- data/lib/evilution/mutator/base.rb +121 -98
- data/lib/evilution/mutator/constant_value_subjects.rb +29 -1
- data/lib/evilution/mutator/line_range_filter.rb +27 -0
- data/lib/evilution/mutator/load_sources.rb +88 -0
- data/lib/evilution/mutator/operator/alias_removal.rb +2 -5
- data/lib/evilution/mutator/operator/argument_method_call_replacement.rb +3 -0
- data/lib/evilution/mutator/operator/attr_accessor_swap.rb +2 -5
- data/lib/evilution/mutator/operator/constant_namespace_strip.rb +3 -3
- data/lib/evilution/mutator/operator/freeze_removal.rb +29 -11
- data/lib/evilution/mutator/operator/method_call_removal.rb +3 -1
- data/lib/evilution/mutator/operator/mixin_removal.rb +2 -5
- data/lib/evilution/mutator/operator/superclass_removal.rb +2 -5
- data/lib/evilution/mutator/operator/symbol_literal.rb +1 -1
- data/lib/evilution/mutator/operator/visibility_removal.rb +2 -5
- data/lib/evilution/mutator/registry.rb +11 -5
- data/lib/evilution/mutator/restore_source_cache.rb +41 -0
- data/lib/evilution/parallel/pool.rb +11 -4
- data/lib/evilution/parallel/work_queue/dispatcher/result_stream.rb +40 -0
- data/lib/evilution/parallel/work_queue/dispatcher.rb +20 -7
- data/lib/evilution/parallel/work_queue/worker/loop.rb +7 -2
- data/lib/evilution/parallel/work_queue/worker.rb +10 -3
- data/lib/evilution/parallel/work_queue/worker_stat.rb +7 -0
- data/lib/evilution/parallel/work_queue.rb +69 -12
- data/lib/evilution/parallel_db_warning.rb +39 -12
- data/lib/evilution/process_supervisor.rb +22 -2
- data/lib/evilution/reporter/cli/item_formatters/coverage_gap.rb +13 -2
- data/lib/evilution/reporter/cli/item_formatters/subject_score.rb +5 -1
- data/lib/evilution/reporter/cli/line_formatters/mutations.rb +9 -2
- data/lib/evilution/reporter/cli/line_formatters/result_line.rb +9 -1
- data/lib/evilution/reporter/cli/line_formatters/skipped_sites.rb +14 -0
- data/lib/evilution/reporter/cli/line_formatters/uncovered_code.rb +17 -0
- data/lib/evilution/reporter/cli/metrics_block.rb +4 -0
- data/lib/evilution/reporter/cli/trailer.rb +4 -4
- data/lib/evilution/reporter/cli.rb +9 -2
- data/lib/evilution/reporter/html/assets/style.css +8 -0
- data/lib/evilution/reporter/html/sections/file_section.rb +9 -0
- data/lib/evilution/reporter/html/sections/uncovered_details.rb +27 -0
- data/lib/evilution/reporter/html/templates/file_section.html.erb +2 -1
- data/lib/evilution/reporter/html/templates/summary_cards.html.erb +3 -0
- data/lib/evilution/reporter/html/templates/uncovered_details.html.erb +11 -0
- data/lib/evilution/reporter/json/mutation_detail.rb +77 -0
- data/lib/evilution/reporter/json/subjects.rb +1 -0
- data/lib/evilution/reporter/json/timings.rb +42 -0
- data/lib/evilution/reporter/json.rb +13 -50
- data/lib/evilution/result/coverage_gap.rb +5 -0
- data/lib/evilution/result/mutation_result.rb +55 -3
- data/lib/evilution/result/mutation_timings.rb +54 -0
- data/lib/evilution/result/run_timings.rb +20 -0
- data/lib/evilution/result/subject_score.rb +8 -1
- data/lib/evilution/result/subject_scorer.rb +5 -3
- data/lib/evilution/result/summary.rb +49 -12
- data/lib/evilution/runner/baseline_runner.rb +47 -5
- data/lib/evilution/runner/diagnostics.rb +0 -13
- data/lib/evilution/runner/isolation_resolver.rb +16 -4
- data/lib/evilution/runner/mutation_executor/isolated_run.rb +34 -0
- data/lib/evilution/runner/mutation_executor/kill_history.rb +66 -0
- data/lib/evilution/runner/mutation_executor/mutation_runner.rb +31 -12
- data/lib/evilution/runner/mutation_executor/neutralizer/baseline_failed.rb +1 -0
- data/lib/evilution/runner/mutation_executor/neutralizer/infra_error.rb +6 -20
- data/lib/evilution/runner/mutation_executor/probe_first.rb +99 -0
- data/lib/evilution/runner/mutation_executor/probe_first_run.rb +59 -0
- data/lib/evilution/runner/mutation_executor/result_cache.rb +45 -14
- data/lib/evilution/runner/mutation_executor/result_notifier.rb +1 -1
- data/lib/evilution/runner/mutation_executor/result_packer.rb +54 -24
- data/lib/evilution/runner/mutation_executor/runless_verdict.rb +39 -0
- data/lib/evilution/runner/mutation_executor/strategy/parallel/hinted_items.rb +27 -0
- data/lib/evilution/runner/mutation_executor/strategy/parallel/ordered_delivery.rb +97 -0
- data/lib/evilution/runner/mutation_executor/strategy/parallel.rb +102 -83
- data/lib/evilution/runner/mutation_executor/timed_run.rb +33 -0
- data/lib/evilution/runner/mutation_executor/timeout_policy/adaptive.rb +67 -0
- data/lib/evilution/runner/mutation_executor/timeout_policy/base.rb +35 -0
- data/lib/evilution/runner/mutation_executor/timeout_policy/fixed.rb +17 -0
- data/lib/evilution/runner/mutation_executor/timeout_policy.rb +56 -0
- data/lib/evilution/runner/mutation_executor.rb +75 -10
- data/lib/evilution/runner/mutation_planner.rb +29 -21
- data/lib/evilution/runner/phase_clock.rb +22 -0
- data/lib/evilution/runner/report_publisher.rb +1 -1
- data/lib/evilution/runner/subject_pipeline.rb +18 -8
- data/lib/evilution/runner.rb +87 -26
- data/lib/evilution/session/diff.rb +9 -2
- data/lib/evilution/session/store.rb +17 -2
- data/lib/evilution/source_ast_cache.rb +10 -0
- data/lib/evilution/spec_ast_cache.rb +15 -0
- data/lib/evilution/spec_resolver/constant_references.rb +66 -0
- data/lib/evilution/spec_resolver/project_files.rb +50 -0
- data/lib/evilution/spec_resolver/sibling_spec_ranker.rb +111 -0
- data/lib/evilution/spec_resolver/spec_pattern.rb +19 -0
- data/lib/evilution/spec_resolver.rb +76 -65
- data/lib/evilution/spec_selector.rb +17 -13
- data/lib/evilution/subject.rb +7 -2
- data/lib/evilution/version.rb +1 -1
- data/lib/evilution/worker_database/copies.rb +73 -0
- data/lib/evilution/worker_database/schema.rb +39 -0
- data/lib/evilution/worker_database/sqlite_copy.rb +65 -0
- data/lib/evilution/worker_database.rb +214 -0
- data/schema/evilution.config.schema.json +74 -2
- metadata +84 -13
- data/lib/evilution/integration/rspec/state_guard/object_space_example_groups.rb +0 -42
- data/scripts/benchmark_density +0 -262
- data/scripts/benchmark_density.yml +0 -19
- data/scripts/canary_manifest.yml +0 -47
- data/scripts/compare_mutations +0 -421
- data/scripts/compare_mutations.yml +0 -24
- data/scripts/compare_targeting +0 -279
- data/scripts/compare_targeting.example.yml +0 -24
- data/scripts/mutant_json_adapter +0 -227
data/docs/architecture.md
CHANGED
|
@@ -60,6 +60,7 @@ Everything lives under `lib/evilution/`.
|
|
|
60
60
|
| `Isolation::{Fork,InProcess}`, `ProcessSupervisor` | Run one mutation's tests in isolation; process-group lifecycle, sandboxing, TERM/KILL ladder. | `isolation/fork.rb`, `process_supervisor.rb` |
|
|
61
61
|
| `Integration::{RSpec,Minitest,TestUnit}` | Apply a mutation and run the configured test framework; report the raw outcome. | `integration/base.rb`, `integration/rspec.rb` |
|
|
62
62
|
| `Parallel::{Pool,WorkQueue}` | Fan mutations across worker processes for `jobs > 1`. | `parallel/pool.rb`, `parallel/work_queue.rb` |
|
|
63
|
+
| `WorkerDatabase` | In each parallel worker of a preloaded Rails app, move `ActiveRecord::Base` off the shared test database: to the worker's `TEST_ENV_NUMBER` database, or to a copy of a SQLite one (`WorkerDatabase::Copies`). | `worker_database.rb`, `worker_database/` |
|
|
63
64
|
| `Result::{MutationResult,Summary}` | Per-mutation result + aggregated, scored summary. | `result/mutation_result.rb`, `result/summary.rb` |
|
|
64
65
|
| `Result::{SubjectScore,SubjectScorer,NeutralReason}` | Score each subject on its own, and record why a neutral is neutral — what the file-level score does not speak for. | `result/subject_scorer.rb`, `result/neutral_reason.rb` |
|
|
65
66
|
| `Reporter::{CLI,JSON,HTML,Suggestion}` | Render a `Summary` to text / JSON / HTML. | `reporter/*` |
|
|
@@ -107,9 +108,15 @@ class that owns it.
|
|
|
107
108
|
|
|
108
109
|
1. **Config** — `Config#initialize` merges YAML + CLI + env (`Config::Sources.merge`),
|
|
109
110
|
applies `DEFAULTS`, freezes. `Runner#initialize` builds the `AST::Parser`,
|
|
110
|
-
`Mutator::Registry.for_profile(config.profile)`, and (if `incremental`) a `Cache
|
|
111
|
+
`Mutator::Registry.for_profile(config.profile)`, and (if `incremental`) a `Cache`
|
|
112
|
+
keyed per subject by `Cache::Keys` (see README, "Incremental cache"). Once the
|
|
113
|
+
subjects are found, every parsed subject is handed to `Cache#index`, so the cache
|
|
114
|
+
can key a subject apart from the rest of its file.
|
|
111
115
|
2. **Subjects** — `Runner::SubjectPipeline#call` resolves target files (explicit,
|
|
112
|
-
`source:<glob>`, or `Git::ChangedFiles
|
|
116
|
+
`source:<glob>`, or `Git::ChangedFiles`; with `--since REF` the command layer has
|
|
117
|
+
already replaced the file arguments and line ranges by `Git::SinceScope`, which
|
|
118
|
+
reads `Git::ChangedLines` into one `LineSet` of changed lines per file, and the
|
|
119
|
+
same line-range filtering then applies), then `AST::Parser#call` runs
|
|
113
120
|
`Prism.parse` and `AST::SubjectFinder` (a `Prism::Visitor`) emits one
|
|
114
121
|
`Evilution::Subject` per `def` node (kind `:method`), plus one per
|
|
115
122
|
`Data.define` / `Struct.new` assigned to a constant or used as a superclass
|
|
@@ -140,14 +147,30 @@ class that owns it.
|
|
|
140
147
|
that already fail *before* any mutation, so their mutations aren't miscounted.
|
|
141
148
|
An optional `Runner::Canary` proves the pipeline can observe a known mutation.
|
|
142
149
|
4. **Mutations** — `Runner::MutationPlanner#call` flat-maps subjects through
|
|
143
|
-
`Mutator::Registry#mutations_for`. The
|
|
150
|
+
`Mutator::Registry#mutations_for`. The planner reads each file once and hands
|
|
151
|
+
that one source to every operator of every subject in it. The registry
|
|
152
|
+
instantiates each operator whose
|
|
144
153
|
`subject_kinds` include the subject's kind (`:method`, `:scope`, `:aasm` and `:callback` by default) and
|
|
145
|
-
runs `operator.call(subject, filter:)`;
|
|
154
|
+
runs `operator.call(subject, filter:, file_source:)`; the filter
|
|
155
|
+
(`AST::SkipFilter`) skips a mutation an `ignore_patterns` pattern matches
|
|
156
|
+
(`AST::Pattern::Filter`) or that lies in a site of a `skip_sites` preset
|
|
157
|
+
(`AST::SkipSites::Presets`), counting skips per preset; each operator subclasses
|
|
146
158
|
`Mutator::Base` and calls `add_mutation`, which runs `AST::SourceSurgeon` and
|
|
147
159
|
builds an immutable `Evilution::Mutation`. Whether the mutated source parses
|
|
148
160
|
is `AST::SyntaxCheck`'s call: Prism first and, on an interpreter that
|
|
149
161
|
compiles with parse.y (Ruby 3.3, or `--parser=parse.y`), parse.y as well,
|
|
150
|
-
since the two disagree on a few constructs.
|
|
162
|
+
since the two disagree on a few constructs. A mutation inside a method
|
|
163
|
+
subject's `def ... end` is checked on the def alone (`AST::DefRegion`) when
|
|
164
|
+
that settles it, and its eval source is then the file's cached
|
|
165
|
+
neutralization with the mutation spliced in (`Mutator::LoadSources`,
|
|
166
|
+
`Integration::Loading::Neutralization#carry`); anything the def alone
|
|
167
|
+
cannot settle -- the original or the mutated def does not parse alone, an
|
|
168
|
+
endless def, an encoding comment, a class-body call holding the def --
|
|
169
|
+
parses and neutralizes the whole mutated file as before, so no verdict
|
|
170
|
+
depends on the shortcut. A
|
|
171
|
+
mutation's `diff` hands Diff::LCS only the lines around the change
|
|
172
|
+
(`Mutation::LineDiff`), with the same output as over the whole file. The
|
|
173
|
+
planner then **deduplicates**
|
|
151
174
|
(by `file_path` + mutated source), and filters **disabled** (`DisableComment`),
|
|
152
175
|
**Sorbet `sig`** (`AST::SorbetSigDetector`), and **equivalent**
|
|
153
176
|
(`Equivalent::Detector`) mutations. Equivalent ones become `:equivalent`
|
|
@@ -158,16 +181,86 @@ class that owns it.
|
|
|
158
181
|
mutated token (`ExampleFilter` / `CoverageExampleFilter`). Separately,
|
|
159
182
|
`Runner::TargetSpecAudit` asks the same selector once per target file in the
|
|
160
183
|
parent, so a file that resolves to no spec at all is a fact the summary
|
|
161
|
-
carries rather than a handful of `unresolved` mutations.
|
|
184
|
+
carries rather than a handful of `unresolved` mutations. Before any mutation
|
|
185
|
+
runs, `MutationExecutor` calls `Integration::Base#warm_targeting`: the RSpec
|
|
186
|
+
integration parses each source file and its resolved spec files once in the
|
|
187
|
+
parent (`ExampleFilter#warm` fills `SourceAstCache` / `SpecAstCache`), so
|
|
188
|
+
every forked child inherits the parses instead of repeating them and losing
|
|
189
|
+
them on exit. Warming stops where the bounded caches are full rather than
|
|
190
|
+
evicting what it warmed first; later files are parsed by their children.
|
|
162
191
|
6. **Execute** — `Runner::MutationExecutor#call` picks a strategy by `config.jobs`:
|
|
163
192
|
`Strategy::Sequential` for `jobs == 1`, `Strategy::Parallel` (via
|
|
164
193
|
`Parallel::Pool` / `WorkQueue`) for `jobs > 1`. Either way each mutation reaches
|
|
165
|
-
`@isolator.call(mutation:, test_command:, timeout:)
|
|
194
|
+
`@isolator.call(mutation:, test_command:, timeout:)`, the timeout chosen per
|
|
195
|
+
mutation by `MutationExecutor::TimeoutPolicy` (the given `--timeout`, or a
|
|
196
|
+
multiple of the baseline time of the mutation's spec files). The parallel pool's
|
|
197
|
+
workers live for the whole run and take the next mutation as soon as they
|
|
198
|
+
finish one, so a slow mutation holds up only its own worker; results stream
|
|
199
|
+
back through `Strategy::Parallel::OrderedDelivery`, which puts them in
|
|
200
|
+
mutation order before the notifier (progress, `--fail-fast`) and the reports
|
|
201
|
+
see them. On fail-fast the pool sends nothing more, lets the mutations in
|
|
202
|
+
flight finish and shuts its workers down; their results are not reported
|
|
203
|
+
but still go to the `--incremental` cache. Under in-process isolation each
|
|
204
|
+
worker runs a single mutation and is replaced, since the mutation runs in
|
|
205
|
+
the worker itself.
|
|
206
|
+
Before either strategy runs a mutation, `MutationExecutor::RunlessVerdict`
|
|
207
|
+
gives the verdicts that need no run: `:unparseable` for a mutated source that
|
|
208
|
+
does not parse, and — with coverage targeting and `skip_uncovered` —
|
|
209
|
+
`:uncovered` for a mutation whose changed lines no example executes
|
|
210
|
+
(`Integration::RSpec#uncovered?` → `CoverageExampleFilter#uncovered?` →
|
|
211
|
+
`Coverage::Map#uncovered?`). The map knows the executable lines nothing ran
|
|
212
|
+
(`missed_lines`) only for files the build vouches for: every example of the
|
|
213
|
+
spec files ran, none forked (a forked process counts apart), and the file
|
|
214
|
+
loaded while lines were counted — under fork isolation the runner counts
|
|
215
|
+
from before the preload (`Coverage::LoadTracker`) and `MapBuilder` hands the
|
|
216
|
+
parent's counts to its forked child, whose own start at zero. Every other
|
|
217
|
+
child forked to run tests (`ProcessSupervisor#spawn`, the baseline) stops
|
|
218
|
+
that counting first (`LoadTracker.release_inherited`), so a spec helper can
|
|
219
|
+
start coverage itself. A mutation that adds or removes a local variable
|
|
220
|
+
assignment runs regardless (`AST::LocalDeclaration`). The cached map
|
|
221
|
+
(`Coverage::MapStore`) is keyed on every project file the build loaded and
|
|
222
|
+
every file under the spec directories as well as on the targets, specs and
|
|
223
|
+
helpers. These results are never cached and are scored as survivors.
|
|
224
|
+
With `probe_first`, `MutationExecutor::ProbeFirstRun` runs the batch in two:
|
|
225
|
+
first each subject's sentinel (`method_body_to_raise` at the subject's
|
|
226
|
+
line, see `MutationExecutor::ProbeFirst`; none for a method with its own
|
|
227
|
+
`rescue` or `ensure`, which can swallow the raise), then the rest, each
|
|
228
|
+
batch through the same strategy and one notifier. A sentinel that survived (or is
|
|
229
|
+
uncovered) makes `RunlessVerdict` report the subject's other mutations
|
|
230
|
+
`:survived` with `implied_by` set, without a run; they are not cached either.
|
|
231
|
+
Each mutation also goes to the integration with the tests that killed its
|
|
232
|
+
neighbours earlier in the run (`MutationExecutor::KillHistory`: same line,
|
|
233
|
+
then same subject, latest first), through `Integration::Base#prioritize`
|
|
234
|
+
before the isolator runs, so a forked child inherits it. A parallel run
|
|
235
|
+
looks them up when the pool sends the mutation (`Strategy::Parallel::
|
|
236
|
+
HintedItems`) and records each kill the moment its result is read
|
|
237
|
+
(`WorkQueue#map(on_arrival:)`), ahead of the ordered delivery. The RSpec
|
|
238
|
+
integration (`Integration::RSpec::KillerFirst` / `ExampleOrder`) runs those
|
|
239
|
+
killers first, then the cheapest examples by their baseline time
|
|
240
|
+
(`Baseline::Result#example_durations`), when the run stops at its first kill
|
|
241
|
+
and `killer_first` is set (off by default: a suite whose examples depend on
|
|
242
|
+
their defined order would report kills that order does not produce); it
|
|
243
|
+
sends back the killing example's id (`killing_test`) and the number of
|
|
244
|
+
examples that ran (`examples_run`) either way. A reordered run that times
|
|
245
|
+
out is run again in the defined order (`MutationExecutor::IsolatedRun`, via
|
|
246
|
+
`Integration::Base#reorders?` / `#in_defined_order`), and that run decides.
|
|
247
|
+
`MutationExecutor::TimedRun` runs an `IsolatedRun` under each limit: under
|
|
248
|
+
the adaptive limit first, then, when that times out in the defined order
|
|
249
|
+
too, under the fixed one, so a mutation that times out every way is judged
|
|
250
|
+
by a run in the defined order under the fixed limit.
|
|
166
251
|
7. **Isolate + test** — `Isolation::Fork#call` (default for Rails/gems) spawns a
|
|
167
252
|
sandboxed worker through `ProcessSupervisor#spawn` (own process group, TERM →
|
|
168
253
|
grace → KILL ladder, sandbox reap), applies the mutation
|
|
169
254
|
(`Integration::Loading::MutationApplier`), and runs `Integration::Base#call` →
|
|
170
255
|
`run_tests`. The child's result is marshaled back over a length-prefixed pipe.
|
|
256
|
+
After reading it the parent waits on that pipe for the child to close it,
|
|
257
|
+
then polls for the child's exit with a short backoff (1 ms doubling up to
|
|
258
|
+
50 ms), all bounded by `REAP_DEADLINE`. A reaped child's pid is never
|
|
259
|
+
signalled or waited on again (it may be reused), but its process group is
|
|
260
|
+
still sent one TERM, so grandchildren the tests left behind do not outlive
|
|
261
|
+
the mutation.
|
|
262
|
+
A fork child does not evaluate the original file again after its tests
|
|
263
|
+
(`Evilution.single_mutation_process?`): it exits right after reporting.
|
|
171
264
|
`Isolation::InProcess` is the lighter path for plain-Ruby projects.
|
|
172
265
|
8. **Status** — outcome flags come from the integration
|
|
173
266
|
(`Integration::RSpec::ResultBuilder`: passed / test_crashed / examples-loaded
|
|
@@ -177,7 +270,11 @@ class that owns it.
|
|
|
177
270
|
A crash is a test that raised, or a mutated source that raised while it was
|
|
178
271
|
loaded and whose original still loads (`MutationApplier` marks it
|
|
179
272
|
`load_crashed` after evaluating the original); a load failure the original
|
|
180
|
-
shares stays an `:error`.
|
|
273
|
+
shares stays an `:error`. A mutant that froze a class its file declares
|
|
274
|
+
would fail the original too, on that class; `Loading::FrozenNamespaces`
|
|
275
|
+
sets such a class aside (removes its constant) so the original defines it
|
|
276
|
+
afresh for that check, then puts the frozen class back, unless the class
|
|
277
|
+
was frozen before the mutant was applied.
|
|
181
278
|
A `NeutralizationPipeline` can reclassify results into `:neutral` — either
|
|
182
279
|
because the tests failed on nothing but examples that were already failing
|
|
183
280
|
at baseline (`Neutralizer::BaselineFailed`) or because the test process
|
|
@@ -198,7 +295,7 @@ class that owns it.
|
|
|
198
295
|
mutations serially, once the contention that caused them is gone.
|
|
199
296
|
9. **Aggregate + report** — `Result::Summary` counts each status and computes
|
|
200
297
|
`score = killed / score_denominator` (total minus error/neutral/equivalent/
|
|
201
|
-
unresolved/unparseable). `Runner::ReportPublisher#publish` selects a reporter by
|
|
298
|
+
unresolved/unparseable; `uncovered` stays in, as a survivor). `Runner::ReportPublisher#publish` selects a reporter by
|
|
202
299
|
`config.format` and writes it; `Session::Store` optionally persists the run.
|
|
203
300
|
`Summary#success?` is also false when a target file resolved to no spec, whatever
|
|
204
301
|
the score. `Commands::Run` maps `summary.success?(min_score:)` to exit code
|
|
@@ -264,9 +361,10 @@ A mutator is a `Prism::Visitor` subclass that emits byte-range edits.
|
|
|
264
361
|
evaluates after the tests: the original file, class-body calls
|
|
265
362
|
neutralized. An in-process run would otherwise leave the last mutation
|
|
266
363
|
of a file applied while the next file is mutated, and that file's tests
|
|
267
|
-
would fail on its behalf.
|
|
364
|
+
would fail on its behalf. A fork child skips it: it exits right after
|
|
365
|
+
reporting, so nothing would run against the restored file. For a subject with no declaration to keep it
|
|
268
366
|
is one string per file, built once and shared
|
|
269
|
-
(`Mutator::
|
|
367
|
+
(`Mutator::RestoreSourceCache`).
|
|
270
368
|
- A scope in a concern's `included` block has two audiences. The kept
|
|
271
369
|
`included` call registers the mutated block for classes that include the
|
|
272
370
|
concern later (`ConcernStateCleaner` clears the old one first, on apply
|
|
@@ -356,7 +454,7 @@ A reporter is `.new(**opts)` + `#call(summary) -> String`. It consumes a frozen
|
|
|
356
454
|
|
|
357
455
|
The `Summary` API a reporter reads includes: counts (`total`, `killed`,
|
|
358
456
|
`survived`, `errors`, `neutral`, `equivalent`, `unresolved`, `unparseable`,
|
|
359
|
-
`timed_out`), metrics (`score`, `score_denominator`, `success?(min_score:)`,
|
|
457
|
+
`uncovered`, `timed_out`), metrics (`score`, `score_denominator`, `success?(min_score:)`,
|
|
360
458
|
`efficiency`, `peak_memory_mb`), the raw `results`, and filtered lists
|
|
361
459
|
(`survived_results`, `killed_results`, …). Each result exposes `mutation`
|
|
362
460
|
(`operator_name`, `file_path`, `line`, `diff`), `status`, `duration`, and
|
data/docs/ast_pattern_syntax.md
CHANGED
|
@@ -8,6 +8,12 @@ The AST pattern language allows precise, semantic exclusion of mutations. Instea
|
|
|
8
8
|
file/line targeting, patterns match Prism AST node structures so that mutations on
|
|
9
9
|
logging, debugging, or infrastructure code can be suppressed declaratively.
|
|
10
10
|
|
|
11
|
+
For the common cases -- log lines, debug output, error messages, load-time
|
|
12
|
+
requires -- the `skip_sites` presets (see README, "Skipping Low-Value Code")
|
|
13
|
+
are more precise than a pattern can be: they skip only what changes what the
|
|
14
|
+
code reports, and keep the raise, the class raised and any code that may
|
|
15
|
+
change state mutated. Patterns are for everything else.
|
|
16
|
+
|
|
11
17
|
Patterns are specified in `.evilution.yml` under the `ignore_patterns` key:
|
|
12
18
|
|
|
13
19
|
```yaml
|
|
@@ -140,10 +146,17 @@ call{receiver=**, name=log}
|
|
|
140
146
|
Prefix `!` negates a pattern or value:
|
|
141
147
|
|
|
142
148
|
```
|
|
143
|
-
call{name=!
|
|
144
|
-
call{receiver=!call{name=logger}}
|
|
149
|
+
call{receiver=call{name=logger}, name=!error} # logger calls except logger.error
|
|
150
|
+
call{name=info, receiver=!call{name=logger}} # info calls not sent to logger
|
|
145
151
|
```
|
|
146
152
|
|
|
153
|
+
Keep a negation inside a pattern that is narrow without it. A pattern ignores
|
|
154
|
+
every mutation inside the nodes it matches (see
|
|
155
|
+
[Pattern Matching Semantics](#pattern-matching-semantics)), so a negation that
|
|
156
|
+
leaves the pattern broad ignores almost everything: `call{name=!log}` matches
|
|
157
|
+
nearly every call, and with it every mutation of their receivers, arguments and
|
|
158
|
+
blocks.
|
|
159
|
+
|
|
147
160
|
## YAML Configuration
|
|
148
161
|
|
|
149
162
|
```yaml
|
|
@@ -173,10 +186,25 @@ ignore_patterns:
|
|
|
173
186
|
|
|
174
187
|
## Pattern Matching Semantics
|
|
175
188
|
|
|
176
|
-
1. Each pattern is tested against the **mutation's AST node** and its **ancestors
|
|
177
|
-
|
|
178
|
-
|
|
179
|
-
|
|
189
|
+
1. Each pattern is tested against the **mutation's AST node** and its **ancestors**,
|
|
190
|
+
up to the top of the file. A pattern naming a node therefore ignores every
|
|
191
|
+
mutation inside it: `call{name=validates}` ignores the mutations of the
|
|
192
|
+
literals in `validates :rush, inclusion: { in: [false, true] }, if: :paid`,
|
|
193
|
+
and `keyword_hash` ignores those of the options but not of `:rush`. A `call`
|
|
194
|
+
pattern covers the whole call — its receiver, its arguments and its block — so
|
|
195
|
+
`call{name=on}` on `opts.on("-j", Integer) { |n| @jobs = n > 0 ? n : 1 }`
|
|
196
|
+
ignores the mutations of the handler block as well. There is no form that
|
|
197
|
+
matches the mutated node alone; to leave a call's block mutated, put
|
|
198
|
+
`# evilution:disable` at the end of the call's line and start the block's
|
|
199
|
+
body on the next line (`do |n|` … `end`).
|
|
200
|
+
2. A mutation is ignored if **any** pattern in `ignore_patterns` matches. Only
|
|
201
|
+
mutations the run would otherwise make are counted as skipped: those in the
|
|
202
|
+
requested line ranges, each counted once however many patterns, nodes or
|
|
203
|
+
operators produce it, and not one that the run makes anyway from a node no
|
|
204
|
+
pattern matches.
|
|
205
|
+
3. `def` patterns follow from rule 1: the method a mutation sits in is one of its
|
|
206
|
+
ancestors, so `def{name=to_s}` ignores all mutations within `to_s`. A `class`
|
|
207
|
+
or `module` pattern ignores all mutations within that body the same way.
|
|
180
208
|
4. Attribute matching is **exact** (after `.to_s` coercion) unless alternatives (`|`)
|
|
181
209
|
or wildcards (`*`, `_`, `**`) are used.
|
|
182
210
|
5. Unspecified attributes are unconstrained (implicit wildcard).
|
|
@@ -221,7 +249,7 @@ names are always plain identifiers. A quoted name is at least one character long
|
|
|
221
249
|
| `call{receiver=_}` | `x.foo()`, `obj.bar()` | `foo()` (no receiver) |
|
|
222
250
|
| `call{receiver=**}` | `foo()`, `x.foo()`, `a.b.foo()` | *(matches all)* |
|
|
223
251
|
| `def{name=to_s}` | `def to_s; ... end` | `def to_str; ... end` |
|
|
224
|
-
| `call{name=!
|
|
252
|
+
| `call{receiver=call{name=logger}, name=!error}` | `logger.info()`, `logger.debug()` | `logger.error()`, `info()` |
|
|
225
253
|
| `call{name=valid?\|save!}` | `x.valid?`, `save!` | `valid`, `save` |
|
|
226
254
|
| `call{name=<=>}` | `a <=> b` | `a <= b` |
|
|
227
255
|
| `call{name='\|'}` | `a \| b` | `a & b` |
|