evilution 1.5.0 → 1.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.beads/interactions.jsonl +1 -0
- data/.rubocop_todo.yml +30 -0
- data/CHANGELOG.md +59 -0
- data/README.md +285 -45
- data/docs/architecture.md +110 -12
- data/docs/ast_pattern_syntax.md +35 -7
- data/docs/combined_runs_spike.md +407 -0
- data/docs/integrations.md +1 -1
- data/docs/isolation.md +63 -2
- data/docs/migration-from-mutant.md +16 -11
- data/docs/mutant_schemata_spike.md +387 -0
- data/docs/operator_yield_study.md +576 -0
- data/docs/probe_first_spike.md +264 -0
- data/lib/evilution/ast/ancestor_lookup.rb +36 -0
- data/lib/evilution/ast/callback_declaration.rb +36 -3
- data/lib/evilution/ast/class_body.rb +62 -16
- data/lib/evilution/ast/def_region.rb +130 -0
- data/lib/evilution/ast/local_declaration.rb +37 -0
- data/lib/evilution/ast/parent_index.rb +50 -0
- data/lib/evilution/ast/parse_cache.rb +27 -0
- data/lib/evilution/ast/parser.rb +8 -3
- data/lib/evilution/ast/pattern/filter.rb +53 -12
- data/lib/evilution/ast/skip_filter.rb +83 -0
- data/lib/evilution/ast/skip_sites/debug_output.rb +67 -0
- data/lib/evilution/ast/skip_sites/error_messages.rb +74 -0
- data/lib/evilution/ast/skip_sites/logging.rb +93 -0
- data/lib/evilution/ast/skip_sites/message.rb +44 -0
- data/lib/evilution/ast/skip_sites/nodes.rb +105 -0
- data/lib/evilution/ast/skip_sites/presets.rb +31 -0
- data/lib/evilution/ast/skip_sites/purity.rb +69 -0
- data/lib/evilution/ast/skip_sites/requires.rb +40 -0
- data/lib/evilution/ast/skip_sites.rb +9 -0
- data/lib/evilution/ast/source_surgeon.rb +20 -2
- data/lib/evilution/ast/syntax_check.rb +10 -8
- data/lib/evilution/baseline/report.rb +11 -2
- data/lib/evilution/baseline.rb +31 -6
- data/lib/evilution/cache/keys.rb +140 -0
- data/lib/evilution/cache/run_fingerprint.rb +51 -0
- data/lib/evilution/cache/skeleton.rb +57 -0
- data/lib/evilution/cache/spec_digest.rb +39 -0
- data/lib/evilution/cache.rb +145 -42
- data/lib/evilution/class_tree.rb +38 -0
- data/lib/evilution/cli/command.rb +12 -4
- data/lib/evilution/cli/commands/run.rb +12 -3
- data/lib/evilution/cli/commands/subjects.rb +5 -2
- data/lib/evilution/cli/parser/options_builder.rb +67 -3
- data/lib/evilution/cli/printers/environment.rb +34 -4
- data/lib/evilution/cli/printers/session_detail.rb +20 -3
- data/lib/evilution/compare/categorizer.rb +2 -1
- data/lib/evilution/compare/normalizer.rb +3 -2
- data/lib/evilution/config/sources.rb +3 -0
- data/lib/evilution/config/validators/base.rb +9 -0
- data/lib/evilution/config/validators/since.rb +22 -0
- data/lib/evilution/config/validators/skip_sites.rb +24 -0
- data/lib/evilution/config/validators/timeout_ceiling.rb +10 -0
- data/lib/evilution/config/validators/timeout_factor.rb +10 -0
- data/lib/evilution/config/validators/timeout_floor.rb +10 -0
- data/lib/evilution/config.rb +122 -6
- data/lib/evilution/coverage/load_tracker.rb +50 -0
- data/lib/evilution/coverage/map.rb +42 -5
- data/lib/evilution/coverage/map_builder.rb +123 -22
- data/lib/evilution/coverage/map_store.rb +15 -5
- data/lib/evilution/coverage/recorder.rb +42 -3
- data/lib/evilution/coverage_example_filter.rb +35 -0
- data/lib/evilution/equivalent/heuristic/comment_marking.rb +14 -1
- data/lib/evilution/example_filter.rb +20 -0
- data/lib/evilution/git/changed_lines.rb +103 -0
- data/lib/evilution/git/since_scope.rb +77 -0
- data/lib/evilution/git/unified_diff.rb +68 -0
- data/lib/evilution/infra_crash.rb +26 -0
- data/lib/evilution/integration/base.rb +97 -6
- data/lib/evilution/integration/crash_detector.rb +46 -6
- data/lib/evilution/integration/known_failures.rb +4 -0
- data/lib/evilution/integration/loading/body_call_neutralizer.rb +49 -11
- data/lib/evilution/integration/loading/callback_equivalence.rb +62 -0
- data/lib/evilution/integration/loading/callback_redeclaration.rb +151 -36
- data/lib/evilution/integration/loading/concern_redeclaration.rb +16 -12
- data/lib/evilution/integration/loading/concern_registry.rb +87 -0
- data/lib/evilution/integration/loading/concern_state_cleaner.rb +19 -8
- data/lib/evilution/integration/loading/created_modules.rb +56 -0
- data/lib/evilution/integration/loading/frozen_namespaces.rb +80 -0
- data/lib/evilution/integration/loading/mutation_applier.rb +42 -11
- data/lib/evilution/integration/loading/neutralization.rb +41 -0
- data/lib/evilution/integration/loading/redefinition_recovery.rb +5 -3
- data/lib/evilution/integration/loading/superseded_modules.rb +22 -19
- data/lib/evilution/integration/minitest/stop_hook.rb +57 -0
- data/lib/evilution/integration/minitest.rb +22 -2
- data/lib/evilution/integration/minitest_crash_detector.rb +19 -0
- data/lib/evilution/integration/rspec/baseline_runner.rb +3 -2
- data/lib/evilution/integration/rspec/crash_detector_lifecycle.rb +6 -1
- data/lib/evilution/integration/rspec/example_filter_applier.rb +16 -0
- data/lib/evilution/integration/rspec/example_ids.rb +23 -0
- data/lib/evilution/integration/rspec/example_order.rb +86 -0
- data/lib/evilution/integration/rspec/killer_first.rb +72 -0
- data/lib/evilution/integration/rspec/result_builder.rb +7 -0
- data/lib/evilution/integration/rspec/state_guard/example_group_descendants.rb +44 -0
- data/lib/evilution/integration/rspec/state_guard.rb +2 -2
- data/lib/evilution/integration/rspec/target_resolver.rb +42 -0
- data/lib/evilution/integration/rspec/test_file_resolver.rb +4 -2
- data/lib/evilution/integration/rspec.rb +98 -47
- data/lib/evilution/integration/stop_policy.rb +49 -0
- data/lib/evilution/integration/test_unit/dispatcher.rb +20 -1
- data/lib/evilution/integration/test_unit/subject_class_registry.rb +6 -1
- data/lib/evilution/integration/test_unit.rb +28 -2
- data/lib/evilution/isolation/fork.rb +58 -13
- data/lib/evilution/isolation/in_process.rb +12 -1
- data/lib/evilution/line_set.rb +72 -0
- data/lib/evilution/mcp/info_tool/actions/environment.rb +12 -16
- data/lib/evilution/mcp/info_tool/actions/subjects.rb +3 -5
- data/lib/evilution/mcp/info_tool/status_glossary.rb +15 -2
- data/lib/evilution/mcp/info_tool.rb +24 -17
- data/lib/evilution/mcp/mutate_tool/config_builder.rb +13 -1
- data/lib/evilution/mcp/mutate_tool/option_parser.rb +2 -2
- data/lib/evilution/mcp/mutate_tool/progress_streamer.rb +1 -1
- data/lib/evilution/mcp/mutate_tool/report_trimmer.rb +28 -5
- data/lib/evilution/mcp/mutate_tool/survived_enricher.rb +12 -7
- data/lib/evilution/mcp/mutate_tool.rb +32 -7
- data/lib/evilution/mcp/session_tool.rb +1 -2
- data/lib/evilution/mcp.rb +4 -0
- data/lib/evilution/mutation/line_diff.rb +111 -0
- data/lib/evilution/mutation.rb +17 -13
- data/lib/evilution/mutator/base.rb +121 -98
- data/lib/evilution/mutator/constant_value_subjects.rb +29 -1
- data/lib/evilution/mutator/line_range_filter.rb +27 -0
- data/lib/evilution/mutator/load_sources.rb +88 -0
- data/lib/evilution/mutator/operator/alias_removal.rb +2 -5
- data/lib/evilution/mutator/operator/argument_method_call_replacement.rb +3 -0
- data/lib/evilution/mutator/operator/attr_accessor_swap.rb +2 -5
- data/lib/evilution/mutator/operator/constant_namespace_strip.rb +3 -3
- data/lib/evilution/mutator/operator/freeze_removal.rb +29 -11
- data/lib/evilution/mutator/operator/method_call_removal.rb +3 -1
- data/lib/evilution/mutator/operator/mixin_removal.rb +2 -5
- data/lib/evilution/mutator/operator/superclass_removal.rb +2 -5
- data/lib/evilution/mutator/operator/symbol_literal.rb +1 -1
- data/lib/evilution/mutator/operator/visibility_removal.rb +2 -5
- data/lib/evilution/mutator/registry.rb +11 -5
- data/lib/evilution/mutator/restore_source_cache.rb +41 -0
- data/lib/evilution/parallel/pool.rb +11 -4
- data/lib/evilution/parallel/work_queue/dispatcher/result_stream.rb +40 -0
- data/lib/evilution/parallel/work_queue/dispatcher.rb +20 -7
- data/lib/evilution/parallel/work_queue/worker/loop.rb +7 -2
- data/lib/evilution/parallel/work_queue/worker.rb +10 -3
- data/lib/evilution/parallel/work_queue/worker_stat.rb +7 -0
- data/lib/evilution/parallel/work_queue.rb +69 -12
- data/lib/evilution/parallel_db_warning.rb +39 -12
- data/lib/evilution/process_supervisor.rb +22 -2
- data/lib/evilution/reporter/cli/item_formatters/coverage_gap.rb +13 -2
- data/lib/evilution/reporter/cli/item_formatters/subject_score.rb +5 -1
- data/lib/evilution/reporter/cli/line_formatters/mutations.rb +9 -2
- data/lib/evilution/reporter/cli/line_formatters/result_line.rb +9 -1
- data/lib/evilution/reporter/cli/line_formatters/skipped_sites.rb +14 -0
- data/lib/evilution/reporter/cli/line_formatters/uncovered_code.rb +17 -0
- data/lib/evilution/reporter/cli/metrics_block.rb +4 -0
- data/lib/evilution/reporter/cli/trailer.rb +4 -4
- data/lib/evilution/reporter/cli.rb +9 -2
- data/lib/evilution/reporter/html/assets/style.css +8 -0
- data/lib/evilution/reporter/html/sections/file_section.rb +9 -0
- data/lib/evilution/reporter/html/sections/uncovered_details.rb +27 -0
- data/lib/evilution/reporter/html/templates/file_section.html.erb +2 -1
- data/lib/evilution/reporter/html/templates/summary_cards.html.erb +3 -0
- data/lib/evilution/reporter/html/templates/uncovered_details.html.erb +11 -0
- data/lib/evilution/reporter/json/mutation_detail.rb +77 -0
- data/lib/evilution/reporter/json/subjects.rb +1 -0
- data/lib/evilution/reporter/json/timings.rb +42 -0
- data/lib/evilution/reporter/json.rb +13 -50
- data/lib/evilution/result/coverage_gap.rb +5 -0
- data/lib/evilution/result/mutation_result.rb +55 -3
- data/lib/evilution/result/mutation_timings.rb +54 -0
- data/lib/evilution/result/run_timings.rb +20 -0
- data/lib/evilution/result/subject_score.rb +8 -1
- data/lib/evilution/result/subject_scorer.rb +5 -3
- data/lib/evilution/result/summary.rb +49 -12
- data/lib/evilution/runner/baseline_runner.rb +47 -5
- data/lib/evilution/runner/diagnostics.rb +0 -13
- data/lib/evilution/runner/isolation_resolver.rb +16 -4
- data/lib/evilution/runner/mutation_executor/isolated_run.rb +34 -0
- data/lib/evilution/runner/mutation_executor/kill_history.rb +66 -0
- data/lib/evilution/runner/mutation_executor/mutation_runner.rb +31 -12
- data/lib/evilution/runner/mutation_executor/neutralizer/baseline_failed.rb +1 -0
- data/lib/evilution/runner/mutation_executor/neutralizer/infra_error.rb +6 -20
- data/lib/evilution/runner/mutation_executor/probe_first.rb +99 -0
- data/lib/evilution/runner/mutation_executor/probe_first_run.rb +59 -0
- data/lib/evilution/runner/mutation_executor/result_cache.rb +45 -14
- data/lib/evilution/runner/mutation_executor/result_notifier.rb +1 -1
- data/lib/evilution/runner/mutation_executor/result_packer.rb +54 -24
- data/lib/evilution/runner/mutation_executor/runless_verdict.rb +39 -0
- data/lib/evilution/runner/mutation_executor/strategy/parallel/hinted_items.rb +27 -0
- data/lib/evilution/runner/mutation_executor/strategy/parallel/ordered_delivery.rb +97 -0
- data/lib/evilution/runner/mutation_executor/strategy/parallel.rb +102 -83
- data/lib/evilution/runner/mutation_executor/timed_run.rb +33 -0
- data/lib/evilution/runner/mutation_executor/timeout_policy/adaptive.rb +67 -0
- data/lib/evilution/runner/mutation_executor/timeout_policy/base.rb +35 -0
- data/lib/evilution/runner/mutation_executor/timeout_policy/fixed.rb +17 -0
- data/lib/evilution/runner/mutation_executor/timeout_policy.rb +56 -0
- data/lib/evilution/runner/mutation_executor.rb +75 -10
- data/lib/evilution/runner/mutation_planner.rb +29 -21
- data/lib/evilution/runner/phase_clock.rb +22 -0
- data/lib/evilution/runner/report_publisher.rb +1 -1
- data/lib/evilution/runner/subject_pipeline.rb +18 -8
- data/lib/evilution/runner.rb +87 -26
- data/lib/evilution/session/diff.rb +9 -2
- data/lib/evilution/session/store.rb +17 -2
- data/lib/evilution/source_ast_cache.rb +10 -0
- data/lib/evilution/spec_ast_cache.rb +15 -0
- data/lib/evilution/spec_resolver/constant_references.rb +66 -0
- data/lib/evilution/spec_resolver/project_files.rb +50 -0
- data/lib/evilution/spec_resolver/sibling_spec_ranker.rb +111 -0
- data/lib/evilution/spec_resolver/spec_pattern.rb +19 -0
- data/lib/evilution/spec_resolver.rb +76 -65
- data/lib/evilution/spec_selector.rb +17 -13
- data/lib/evilution/subject.rb +7 -2
- data/lib/evilution/version.rb +1 -1
- data/lib/evilution/worker_database/copies.rb +73 -0
- data/lib/evilution/worker_database/schema.rb +39 -0
- data/lib/evilution/worker_database/sqlite_copy.rb +65 -0
- data/lib/evilution/worker_database.rb +214 -0
- data/schema/evilution.config.schema.json +74 -2
- metadata +84 -13
- data/lib/evilution/integration/rspec/state_guard/object_space_example_groups.rb +0 -42
- data/scripts/benchmark_density +0 -262
- data/scripts/benchmark_density.yml +0 -19
- data/scripts/canary_manifest.yml +0 -47
- data/scripts/compare_mutations +0 -421
- data/scripts/compare_mutations.yml +0 -24
- data/scripts/compare_targeting +0 -279
- data/scripts/compare_targeting.example.yml +0 -24
- data/scripts/mutant_json_adapter +0 -227
|
@@ -0,0 +1,387 @@
|
|
|
1
|
+
# Mutant schemata: spike report
|
|
2
|
+
|
|
3
|
+
Status: **no-go**. No prototype in `lib/`. The prototype was a harness
|
|
4
|
+
loaded from outside the gem, kept in `scripts/schemata_spike/` (see
|
|
5
|
+
[Reproducing](#reproducing)). Two cheaper
|
|
6
|
+
changes, found while measuring, are recommended instead
|
|
7
|
+
([below](#what-to-do-instead)).
|
|
8
|
+
|
|
9
|
+
## Idea
|
|
10
|
+
|
|
11
|
+
Mutation switching ("mutant schemata", Untch; Stryker's mutant switching):
|
|
12
|
+
build one meta-version of a file with every method-body mutant behind a
|
|
13
|
+
runtime switch, evaluate it once in the parent before any mutation process
|
|
14
|
+
forks, and have each mutation process only set the active mutant's id. That
|
|
15
|
+
removes the per-mutant parse, evaluate and restore of the whole file. Class
|
|
16
|
+
bodies, constants, visibility and load-time mutants keep the eval path.
|
|
17
|
+
|
|
18
|
+
## How much there is to save
|
|
19
|
+
|
|
20
|
+
The fixed cost schemata can remove is the `apply` phase of each mutation
|
|
21
|
+
(syntax check, constant pinning, concern cleanup, eval of the file; the
|
|
22
|
+
in-process restore too). It is not the isolation around it (fork, sandbox,
|
|
23
|
+
reap) or the test run. That phase was measured first, from the per-mutation
|
|
24
|
+
`timings` in the JSON report, on the current tree (everything up to the
|
|
25
|
+
operator yield study, so parse-once generation and warm parent caches are
|
|
26
|
+
already in). Each target ran at `--jobs 2`.
|
|
27
|
+
|
|
28
|
+
| Target | Isolation | Mutants | `apply` per mutant (mean / p50) | Isolation overhead per mutant | `test_run` per mutant | `apply` share of killtime |
|
|
29
|
+
|---|---|---|---|---|---|---|
|
|
30
|
+
| fixture (`spec/support/fixtures/benchmark_project`) | in_process | 440 | 0.40 / 0.40 ms | 0.8 ms | 32.8 ms (+ confirmation) | **0.1%** |
|
|
31
|
+
| dry-monads `lib/dry/monads/maybe.rb` | fork | 611 | 5.2 / 5.1 ms | 10.5 ms | 117 ms | **3.6%** |
|
|
32
|
+
| i18n `lib/i18n/backend/base.rb` (Minitest) | fork | 1837 | 8.3 / 7.9 ms | 9.5 ms | 22.5 ms | **20.5%** |
|
|
33
|
+
|
|
34
|
+
So the most schemata can win is what `apply` costs today, and that is only
|
|
35
|
+
large where the tests are fast (i18n: 22 ms per test run).
|
|
36
|
+
|
|
37
|
+
Where that `apply` time goes was measured by wrapping each step in a real
|
|
38
|
+
i18n run (mean per mutant, fork child):
|
|
39
|
+
|
|
40
|
+
| Step | ms |
|
|
41
|
+
|---|---|
|
|
42
|
+
| `ConstantPinner` (a full `Prism.parse` of the **original** file, to list its constants) | **6.08** |
|
|
43
|
+
| `SourceEvaluator` (the eval itself) | 0.96 |
|
|
44
|
+
| `SyntaxValidator` (`Prism.parse_success?`, no tree) | 0.37 |
|
|
45
|
+
| `mark_feature_loaded` | 0.12 |
|
|
46
|
+
| `ConcernStateCleaner` (no ActiveSupport here) | 0.01 |
|
|
47
|
+
| **Total `apply`** | **8.05** |
|
|
48
|
+
|
|
49
|
+
Three quarters of `apply` is the constant pinner building a full syntax tree
|
|
50
|
+
of the unchanged original file, in every mutation process, and getting the
|
|
51
|
+
same names each time. The same parse takes 0.5 ms in a warm parent; in a
|
|
52
|
+
freshly forked child it costs about 6 ms on almost every call (p50 6.0 ms).
|
|
53
|
+
GC is not the reason: only 26 of the 1831 calls ran a GC. The cost is per
|
|
54
|
+
call, in allocating the tree on the child's fresh heap and the copy-on-write
|
|
55
|
+
page faults of those first writes. The eval that schemata would remove costs
|
|
56
|
+
about 1 ms.
|
|
57
|
+
|
|
58
|
+
## Prototype
|
|
59
|
+
|
|
60
|
+
A harness (`RUBYOPT=-r<harness>`), not code in `lib/`. It prepends two hooks:
|
|
61
|
+
|
|
62
|
+
- **Parent**, after `MutationExecutor#warm`, before workers or mutation
|
|
63
|
+
processes fork: per file, it takes each mutation's change as the common
|
|
64
|
+
prefix and suffix of the neutralized original (`restore_source`) and its
|
|
65
|
+
`eval_source`. A mutation is eligible when the change lies inside the body
|
|
66
|
+
of a `def` that is nested only in `class`, `module` and `class << self`
|
|
67
|
+
bodies (or wrapped in `private def ...`). Each such method is rebuilt as
|
|
68
|
+
|
|
69
|
+
```ruby
|
|
70
|
+
def name(params)
|
|
71
|
+
case $__evilution_schema when nil then begin <original body, on its own lines>
|
|
72
|
+
end when 17 then begin
|
|
73
|
+
<body with mutation 17>
|
|
74
|
+
end when 18 then begin
|
|
75
|
+
...
|
|
76
|
+
end else begin
|
|
77
|
+
<original body>
|
|
78
|
+
end end
|
|
79
|
+
end
|
|
80
|
+
```
|
|
81
|
+
|
|
82
|
+
and evaluated **on its own**, wrapped in the file's nesting written on the
|
|
83
|
+
`def`'s line (`module I18n; module Backend; module Base; def ...`) and
|
|
84
|
+
with the file's path and the `def`'s line number. Nothing else in the file
|
|
85
|
+
runs again: no class body, no neutralizer, no `alias` line. Every line
|
|
86
|
+
outside the variants keeps its number, and so does the original branch.
|
|
87
|
+
After the redefinition it re-applies the method's visibility, its
|
|
88
|
+
`module_function` copy and the file's `alias`/`alias_method`s of it.
|
|
89
|
+
The redefinitions and re-aliases are wrapped in `ReevalWarningFilter`, since
|
|
90
|
+
dry-monads' spec helper turns "method redefined" warnings into errors.
|
|
91
|
+
- **Mutation process**: `MutationApplier#call` sets `$__evilution_schema` to
|
|
92
|
+
the mutation's id and returns. `#restore` clears it. Under `in_process`, a
|
|
93
|
+
restore after a mutation of the same file that took the eval path puts the
|
|
94
|
+
meta-methods back.
|
|
95
|
+
|
|
96
|
+
Not eligible: changes outside a static `def` body (constants, class-body
|
|
97
|
+
calls, signatures and default arguments, `def`s inside blocks such as
|
|
98
|
+
`included do`, `Module.new do`, `Class.new do`), endless `def`s (the
|
|
99
|
+
prototype did not rewrite them), scope/aasm/callback/concern-hook subjects
|
|
100
|
+
(their restore source keeps a declaration of their own), variants containing
|
|
101
|
+
`__LINE__`, `binding` or `local_variables`, and the local-variable conflict
|
|
102
|
+
described below.
|
|
103
|
+
|
|
104
|
+
## Correctness questions
|
|
105
|
+
|
|
106
|
+
**Which mutants.** Method bodies only, of `def`s reachable through
|
|
107
|
+
class/module/singleton-class bodies, including `def self.x` and
|
|
108
|
+
`class << self`. Blocks and lambdas inside a body are part of the body and
|
|
109
|
+
switch with it. Not covered: methods built by `define_method` (their body is
|
|
110
|
+
a block evaluated at load time; switching it means re-running the class-body
|
|
111
|
+
call), `def`s inside blocks (concern `included`/`class_methods`, `Module.new`,
|
|
112
|
+
`Class.new`, `Struct.new do`), and endless `def`s. Endless `def`s can be
|
|
113
|
+
rewritten to the long form (an extra 150 of 610 on `maybe.rb`).
|
|
114
|
+
|
|
115
|
+
**Methods redefined later in the file.** Each `def` gets its own switch, and
|
|
116
|
+
the later definition wins at load time, as it does when the file is
|
|
117
|
+
evaluated. Equivalent.
|
|
118
|
+
|
|
119
|
+
**Aliases.** `alias new old` copies the method entry at the moment it runs.
|
|
120
|
+
Evaluating the whole file re-runs the alias, so the alias gets the mutant.
|
|
121
|
+
Redefining one method does not. The prototype re-aliases the file's own
|
|
122
|
+
aliases of the method after redefining it. An alias made in another file
|
|
123
|
+
keeps the original under both paths. A load-time capture other than an alias
|
|
124
|
+
(`HANDLERS = { a: method(:foo) }`, `define_method(:b, instance_method(:a))`,
|
|
125
|
+
a memoizer that wraps by copying the method) keeps the original under
|
|
126
|
+
schemata but gets the mutant under the eval path, unless the neutralizer
|
|
127
|
+
blanked the capturing call. The prototype does not detect these: none
|
|
128
|
+
occurred on the targets, so it can only be counted as an open risk.
|
|
129
|
+
|
|
130
|
+
**Visibility and `module_function`.** A `def` in a reopened class is public.
|
|
131
|
+
The original visibility has to be read before the redefinition and set again
|
|
132
|
+
after it, and a `module_function` method's singleton copy has to be made
|
|
133
|
+
again. The prototype does both.
|
|
134
|
+
|
|
135
|
+
**Local variables across branches.** Ruby decides at parse time whether a
|
|
136
|
+
bare `x` is a local or a method call, by whether an assignment to `x` comes
|
|
137
|
+
before it in the method. In the meta-method every branch shares one local
|
|
138
|
+
table, so a variant that drops `x = compute` and keeps a later `x` turns
|
|
139
|
+
that `x` from a `NoMethodError`/`NameError` (eval path) into `nil`
|
|
140
|
+
(schemata). The prototype parses each variant and drops it when a bare call
|
|
141
|
+
in it names a local assigned in an earlier branch. That dropped 6 of 440
|
|
142
|
+
(fixture), 30 of 1830 (i18n) and 6 of 610 (`maybe.rb`).
|
|
143
|
+
|
|
144
|
+
**Source locations, `__method__`, `super`, `return`, `binding`.** `return`,
|
|
145
|
+
`yield`, `block_given?`, `super` (implicit and explicit), `__method__` and
|
|
146
|
+
`retry` in a body-level `rescue` behave the same inside `case ... begin ...
|
|
147
|
+
end`, because the branch is still the method's own frame. The original
|
|
148
|
+
branch keeps its lines. A variant's lines do not: they come after the
|
|
149
|
+
original body, so a backtrace from inside the active mutant, `__LINE__`, or
|
|
150
|
+
an exception message carrying a line differ from the eval path. Variants
|
|
151
|
+
using `__LINE__` were dropped. `binding` and `local_variables` see the
|
|
152
|
+
locals of every branch (as `nil`), so variants using them were dropped too.
|
|
153
|
+
`Method#source_location` points at the `def` line in both paths.
|
|
154
|
+
|
|
155
|
+
**Runtime cost of the switch.** One global read plus `opt_case_dispatch`
|
|
156
|
+
per call: a method doing `x + 1` went from 17 ns to 36 ns per call (+19 ns).
|
|
157
|
+
A frozen per-file array read instead of the global cost +10 ns. That is
|
|
158
|
+
negligible next to a method that does real work. It is paid on every call
|
|
159
|
+
in every mutation process, including calls of mutants of other methods.
|
|
160
|
+
|
|
161
|
+
**Isolation and workers.** Under `fork` (and the persistent workers, which
|
|
162
|
+
fork a process per mutation), the meta-version is built once in the parent
|
|
163
|
+
before the workers fork, and the mutation process only sets the global. Under
|
|
164
|
+
`in_process`, a schemata mutant needs no restore. An eval-path mutant of the
|
|
165
|
+
same file replaces the meta-methods with its own and must be followed by
|
|
166
|
+
the meta-methods, not just the original (the prototype does that). The
|
|
167
|
+
fixture ran in-process at `--jobs 2` with identical verdicts.
|
|
168
|
+
|
|
169
|
+
**Neutralizer and preload.** Redefining single methods sidesteps the
|
|
170
|
+
neutralizer entirely: no class body runs again. That is the main correctness
|
|
171
|
+
gain over evaluating a whole meta-file, which would re-run class bodies once
|
|
172
|
+
in the parent like any eval. The file must be loaded in the parent first
|
|
173
|
+
(preload does that for gems and Rails; the fixture's files were `require`d
|
|
174
|
+
by the harness). For Rails, a meta-method defined before workers fork is
|
|
175
|
+
inherited like any preloaded code. The prototype was not run on Redmine.
|
|
176
|
+
|
|
177
|
+
**Memory.** The parent's RSS grew by 0.8 MB (fixture), 0.25 MB (`maybe.rb`)
|
|
178
|
+
and **16.5 MB** (i18n `base.rb`: 1751 variants, most of them copies of a few
|
|
179
|
+
long methods). Building and evaluating them took 0.17 s, 0.31 s and **2.3 s**,
|
|
180
|
+
serially in the parent before the first mutation.
|
|
181
|
+
|
|
182
|
+
## Results
|
|
183
|
+
|
|
184
|
+
Eligible share (prototype rules):
|
|
185
|
+
|
|
186
|
+
| Target | Eligible | Outside a static `def` body | Endless `def` | Local conflict | `__LINE__`/`binding` |
|
|
187
|
+
|---|---|---|---|---|---|
|
|
188
|
+
| fixture | 409 / 440 (93%) | 25 | 0 | 6 | 0 |
|
|
189
|
+
| i18n `base.rb` | 1751 / 1830 (96%) | 26 | 0 | 30 | 23 |
|
|
190
|
+
| dry-monads `maybe.rb` | 395 / 610 (65%) | 59 | 150 | 6 | 0 |
|
|
191
|
+
| **Total** | **2555 / 2880 (89%)** | 110 | 150 | 42 | 23 |
|
|
192
|
+
|
|
193
|
+
(Mutants reported without a run, here the equivalent ones, never reach
|
|
194
|
+
`apply` and are not counted.)
|
|
195
|
+
|
|
196
|
+
**Verdict equivalence:** identical on every run. Each mutant was matched by
|
|
197
|
+
file, line, operator and diff against the eval-path run: fixture 440/440,
|
|
198
|
+
i18n 1837/1837 and dry-monads `maybe.rb` 611/611 (one schemata run for the
|
|
199
|
+
fixture, three for dry-monads, and every schemata run on i18n, probes
|
|
200
|
+
included).
|
|
201
|
+
|
|
202
|
+
**`apply` per mutant**, eval path vs schemata (mean, all mutants):
|
|
203
|
+
|
|
204
|
+
| Target | Eval path | Schemata | Saved per mutant |
|
|
205
|
+
|---|---|---|---|
|
|
206
|
+
| fixture | 0.40 ms | 0.11 ms | 0.3 ms |
|
|
207
|
+
| dry-monads `maybe.rb` | 5.2 ms | 2.5 ms (eligible: 0.1 ms) | 2.7 ms |
|
|
208
|
+
| i18n `base.rb` | 8.3 ms | 0.5 ms | 7.8 ms |
|
|
209
|
+
|
|
210
|
+
**Wall time and killtime**, same tree, schemata off and on alternated
|
|
211
|
+
(seconds):
|
|
212
|
+
|
|
213
|
+
| Target | Off: duration / killtime | On: duration / killtime | Change in duration |
|
|
214
|
+
|---|---|---|---|
|
|
215
|
+
| i18n, `--jobs 2` | 39.7 / 76.7, 40.2 / 77.7 | 42.8 / 80.0, 43.6 / 81.5 | **+8% (slower)** |
|
|
216
|
+
| dry-monads `maybe.rb`, `--jobs 2` | 45.3 / 88.8, 43.5 / 85.3, 43.8 / 85.8 | 44.9 / 87.3, 50.2 / 97.6, 43.3 / 83.9 | none measurable (`apply` saves 2.6 s of ~86 s killtime; one run on was a 15% outlier) |
|
|
217
|
+
| fixture, `--jobs 2` (one pair) | 77.7 / 149.7 | 83.9 / 160.8 | no gain possible: `apply` is 0.17 s of the run; the difference is in test and confirmation time |
|
|
218
|
+
|
|
219
|
+
On i18n, schemata removes 16 s of `apply` and adds 19 s of `test_run`
|
|
220
|
+
(42.2 / 42.8 s → 61.5 / 62.0 s). The tests themselves got slower. Four
|
|
221
|
+
measurements, all with verdicts identical to the eval path, narrow down
|
|
222
|
+
why:
|
|
223
|
+
|
|
224
|
+
- **GC per mutation process** went from 10.6 to 13.4 ms, and major GCs from
|
|
225
|
+
0.115 to 0.258 per mutant, with the parent's heap 16.5 MB larger (one run
|
|
226
|
+
each with GC counters).
|
|
227
|
+
- **Full GC in the parent before forking** (`Process.warmup` after the
|
|
228
|
+
meta-methods are built: full GC, compaction, heap trim), three rounds
|
|
229
|
+
alternated with schemata alone, the eval path alone and the eval path
|
|
230
|
+
with the same warm-up. The machine was busier than in the runs above, so
|
|
231
|
+
compare within this table only (means of three runs, seconds):
|
|
232
|
+
|
|
233
|
+
| i18n, `--jobs 2` | duration | killtime | `test_run` | GC per mutant | major GCs per mutant |
|
|
234
|
+
|---|---|---|---|---|---|
|
|
235
|
+
| eval path | 48.0 | 92.6 | 51.5 | 12.2 ms | 0.14 |
|
|
236
|
+
| schemata | 55.1 | 103.5 | 79.9 | 16.5 ms | 0.31 |
|
|
237
|
+
| schemata + parent warm-up | 44.3 | 81.1 | 58.5 | 7.4 ms | 0.14 |
|
|
238
|
+
| eval path + parent warm-up | 42.9 | 82.2 | 41.7 | 6.9 ms | 0.17 |
|
|
239
|
+
|
|
240
|
+
The warm-up removes the GC penalty of the bigger heap: with it, the child
|
|
241
|
+
spends about as long in GC under schemata as under the eval path. It does
|
|
242
|
+
not remove the rest: `test_run` is still 16.7 s (9 ms per mutant) higher
|
|
243
|
+
than the eval path with the same warm-up.
|
|
244
|
+
- **Re-evaluating the meta-methods inside each mutation process** (fresh
|
|
245
|
+
copies compiled in the child, `apply` 252 s) brought `test_run` back to
|
|
246
|
+
the eval path's 42 s. So the switch and the size of the methods do not
|
|
247
|
+
slow the tests by themselves.
|
|
248
|
+
- **`GC.start` inside each mutation process**, after setting the switch and
|
|
249
|
+
before the tests (`apply` 65 s), brought `test_run` down to 29.1 s, below
|
|
250
|
+
the eval path's 42 s, because the tests then ran with no GC debt at all.
|
|
251
|
+
A full GC in the child also touches the whole inherited heap, so this
|
|
252
|
+
probe moves page faults out of the tests as well as GC, and does not
|
|
253
|
+
separate the two.
|
|
254
|
+
|
|
255
|
+
So of the 28 s of extra `test_run` in those rounds, about 12 s (40%) was GC
|
|
256
|
+
from the larger parent heap, which a pre-fork warm-up removes. The
|
|
257
|
+
remaining 17 s (60%) is tied to running methods compiled in the parent
|
|
258
|
+
rather than in the child. First-call writes
|
|
259
|
+
(inline caches) into the parent's instruction sequences, each copying a
|
|
260
|
+
shared page, would explain it, but that mechanism was not measured.
|
|
261
|
+
|
|
262
|
+
On top of that the parent spends 2.3 s building and evaluating the
|
|
263
|
+
meta-methods before the first mutation, serially. That time was not broken
|
|
264
|
+
down; it covers two extra `Prism.parse` calls per variant in the
|
|
265
|
+
local-conflict guard and compiling methods that hold a full copy of the
|
|
266
|
+
body per variant (1751 copies of a few long methods on i18n).
|
|
267
|
+
|
|
268
|
+
## Decision
|
|
269
|
+
|
|
270
|
+
**No-go**, and it still holds with the pre-fork warm-up measured above.
|
|
271
|
+
Four reasons:
|
|
272
|
+
|
|
273
|
+
1. **The ceiling is low.** What schemata removes is `apply`: 0.1% of
|
|
274
|
+
killtime on the fixture, 3.6% on dry-monads `maybe.rb`, 20% on i18n.
|
|
275
|
+
Three quarters of that 20% is one parse (below) that needs no schemata
|
|
276
|
+
to remove.
|
|
277
|
+
2. **On the one target where the ceiling is worth having, it did not win.**
|
|
278
|
+
As built (whole-body variants, no pre-fork GC) i18n ran 8% slower. A
|
|
279
|
+
`Process.warmup` in the parent removes the GC part of that loss, but the
|
|
280
|
+
same warm-up speeds up the eval path too, and against that schemata
|
|
281
|
+
ties (killtime -1%, wall +3%): the tests still run 9 ms per
|
|
282
|
+
mutant slower on parent-compiled methods, which eats the saved `apply`.
|
|
283
|
+
dry-monads came out even. This is measured for this whole-body design
|
|
284
|
+
only; switching at the mutated expression (Untch, Stryker) would cut
|
|
285
|
+
the heap growth and build time, and was not built.
|
|
286
|
+
3. **The correctness surface is large for what is left.** Variant line
|
|
287
|
+
numbers shift. Local variables leak between branches and need a parse per
|
|
288
|
+
variant to guard. Visibility, `module_function` and aliases have to be
|
|
289
|
+
re-applied by hand. Load-time captures of a method (`method(:x)`,
|
|
290
|
+
`instance_method`, memoizers) diverge from the eval path silently.
|
|
291
|
+
`define_method`, `def`s in blocks and endless `def`s are excluded or need
|
|
292
|
+
rewriting. Verdicts were identical on 2888 mutants across three targets,
|
|
293
|
+
but each of these is a way for a later target to disagree.
|
|
294
|
+
4. **It does not compose with what the eval path already has.** Neutralized
|
|
295
|
+
class bodies, redeclared subjects (scope, aasm, callbacks, concern hooks)
|
|
296
|
+
and `load_crashed` detection stay on the eval path. Schemata would be a
|
|
297
|
+
second way to apply a mutation, to be kept equivalent with the first.
|
|
298
|
+
|
|
299
|
+
Reason 1 holds on its own: even a version that cost nothing could remove at
|
|
300
|
+
most a fifth of i18n's killtime, and three quarters of that a parse memo
|
|
301
|
+
removes without schemata. Reasons 3 and 4 hold for any version.
|
|
302
|
+
Expression-level switching was not built: it saves the same `apply`, keeps
|
|
303
|
+
reasons 1, 3 and 4, and whether it also avoids the slower tests of reason 2
|
|
304
|
+
is open.
|
|
305
|
+
|
|
306
|
+
## What to do instead
|
|
307
|
+
|
|
308
|
+
**Parse the original's constant names once, in the parent.**
|
|
309
|
+
`ConstantPinner#call(mutation.original_source)` runs `Prism.parse` on the
|
|
310
|
+
same unchanged source in every mutation process. Measured with the names
|
|
311
|
+
memoized per source and filled in `MutationExecutor#warm` (the same harness,
|
|
312
|
+
`EV_PIN_MEMO=1`), i18n alternated off/on, two runs each:
|
|
313
|
+
|
|
314
|
+
| | `apply` total | killtime | duration |
|
|
315
|
+
|---|---|---|---|
|
|
316
|
+
| off | 17.97 s, 17.58 s | 81.6 s, 79.4 s | 42.1 s, 41.1 s |
|
|
317
|
+
| on | 8.52 s, 8.77 s | 72.6 s, 75.6 s | 37.7 s, 39.2 s |
|
|
318
|
+
|
|
319
|
+
That is about **-8% killtime and -8% wall on i18n** (killtime 80.5 → 74.1 s,
|
|
320
|
+
wall 41.6 → 38.4 s, means of two runs), with verdicts identical
|
|
321
|
+
(1837/1837), using a cache filled in the parent the way the concern
|
|
322
|
+
registry already is. Of the 9.1 s of `apply` saved, about 6.4 s reaches
|
|
323
|
+
killtime: `test_run` rose with the memo on (45.4 / 43.9 → 46.2 / 48.6 s),
|
|
324
|
+
probably the same shift of GC work into the tests that schemata shows
|
|
325
|
+
(not measured for the memo). Two runs each is thin: unmodified i18n runs in
|
|
326
|
+
the same session ranged from 38.2 to 42.1 s, about the size of the gain. It
|
|
327
|
+
is a small change with no new way to apply a mutation. Worth filing as its
|
|
328
|
+
own issue, and the gain should be confirmed there with
|
|
329
|
+
`scripts/benchmark_perf` (interleaved repeats), which this spike did not
|
|
330
|
+
use. `RedefinitionRecovery` parses the original too, but only on a
|
|
331
|
+
conflict.
|
|
332
|
+
|
|
333
|
+
**Collect and compact the parent's heap once before forking.** In the
|
|
334
|
+
warm-up rounds above, `Process.warmup` in the parent after
|
|
335
|
+
`MutationExecutor#warm` cut the eval path's GC per mutation process from
|
|
336
|
+
12.2 to 6.9 ms and its `test_run` from 51.5 to 41.7 s: **-11% killtime and
|
|
337
|
+
-11% wall on i18n** (92.6 → 82.2 s, 48.0 → 42.9 s, means of three
|
|
338
|
+
alternated runs, verdicts identical). It is one call with no new way to
|
|
339
|
+
apply a mutation. Worth its own issue, measured there with
|
|
340
|
+
`scripts/benchmark_perf` on more than one canary (it compacts the heap,
|
|
341
|
+
which a C extension that does not support compaction could mind).
|
|
342
|
+
|
|
343
|
+
The rest of the per-mutant fixed cost is the isolation itself (fork, sandbox,
|
|
344
|
+
reap: 9.5–10.5 ms per mutant under `fork`, more than `apply` on every
|
|
345
|
+
target) and per-mutation framework reset inside `test_run`. Schemata touches
|
|
346
|
+
neither.
|
|
347
|
+
|
|
348
|
+
## Reproducing
|
|
349
|
+
|
|
350
|
+
The harness, the timing wrapper and the aggregation scripts are in
|
|
351
|
+
[`scripts/schemata_spike/`](../scripts/schemata_spike/); its README lists
|
|
352
|
+
every environment switch. Corpus: copies of the dry-monads and i18n canaries
|
|
353
|
+
(pinned commits in `scripts/benchmark_perf.yml`) and of the fixture, each
|
|
354
|
+
with a Gemfile adding `gem "evilution", path: <tree>`.
|
|
355
|
+
|
|
356
|
+
- Baseline: the per-mutation `timings` (`apply`, `test_run`,
|
|
357
|
+
`confirmation`) of a normal JSON run. Isolation overhead per mutant is
|
|
358
|
+
`duration` minus those.
|
|
359
|
+
- Step breakdown (`apply_prof.rb`): prepends a timing wrapper to `call` of
|
|
360
|
+
`SyntaxValidator`, `ConstantPinner`, `ConcernStateCleaner`,
|
|
361
|
+
`SourceEvaluator`, `RedefinitionRecovery` and
|
|
362
|
+
`MutationApplier#mark_feature_loaded`, loaded with `RUBYOPT=-r<file>` (it
|
|
363
|
+
must `require "bundler/setup"` first).
|
|
364
|
+
- Schemata (`schemata.rb`): the harness described under
|
|
365
|
+
[Prototype](#prototype): a module
|
|
366
|
+
prepended to `MutationExecutor` whose private `warm(integration,
|
|
367
|
+
mutations)` calls `super` and then builds and evaluates the meta-methods,
|
|
368
|
+
and one prepended to `MutationApplier` whose `call`/`restore`
|
|
369
|
+
set and clear the global for the mutations it switched (keyed by file and
|
|
370
|
+
`mutated_source.hash`, stable across `fork`).
|
|
371
|
+
- Probes and the memo: switches of the same harness (`EV_SCHEMATA_REEVAL`,
|
|
372
|
+
`EV_SCHEMATA_GC`, `EV_PARENT_WARMUP`, `EV_GCSTAT`, `EV_PIN_MEMO`).
|
|
373
|
+
- Verdict equivalence (`compare_verdicts.rb`): match the two runs' JSON
|
|
374
|
+
entries by file, line, operator and diff, and compare statuses.
|
|
375
|
+
|
|
376
|
+
Commands (in the target dir, `--jobs 2`):
|
|
377
|
+
|
|
378
|
+
```
|
|
379
|
+
# fixture
|
|
380
|
+
bundle exec evilution run lib/pricing.rb lib/inventory.rb lib/grade.rb --jobs 2 --format json
|
|
381
|
+
# dry-monads
|
|
382
|
+
bundle exec evilution run lib/dry/monads/maybe.rb --spec spec/unit/maybe_spec.rb \
|
|
383
|
+
--integration rspec --preload spec/spec_helper.rb --jobs 2 --format json
|
|
384
|
+
# i18n
|
|
385
|
+
RUBYOPT=-Itest bundle exec evilution run lib/i18n/backend/base.rb --spec test/i18n_test.rb \
|
|
386
|
+
--integration minitest --preload test/test_helper.rb --jobs 2 --format json
|
|
387
|
+
```
|