evilution 1.5.1 → 1.6.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (227) hide show
  1. checksums.yaml +4 -4
  2. data/.rubocop_todo.yml +30 -0
  3. data/CHANGELOG.md +49 -0
  4. data/README.md +285 -45
  5. data/docs/architecture.md +110 -12
  6. data/docs/ast_pattern_syntax.md +35 -7
  7. data/docs/combined_runs_spike.md +407 -0
  8. data/docs/integrations.md +1 -1
  9. data/docs/isolation.md +63 -2
  10. data/docs/migration-from-mutant.md +16 -11
  11. data/docs/mutant_schemata_spike.md +387 -0
  12. data/docs/operator_yield_study.md +576 -0
  13. data/docs/probe_first_spike.md +264 -0
  14. data/lib/evilution/ast/ancestor_lookup.rb +36 -0
  15. data/lib/evilution/ast/callback_declaration.rb +36 -3
  16. data/lib/evilution/ast/class_body.rb +62 -16
  17. data/lib/evilution/ast/def_region.rb +130 -0
  18. data/lib/evilution/ast/local_declaration.rb +37 -0
  19. data/lib/evilution/ast/parent_index.rb +50 -0
  20. data/lib/evilution/ast/parse_cache.rb +27 -0
  21. data/lib/evilution/ast/parser.rb +8 -3
  22. data/lib/evilution/ast/pattern/filter.rb +53 -12
  23. data/lib/evilution/ast/skip_filter.rb +83 -0
  24. data/lib/evilution/ast/skip_sites/debug_output.rb +67 -0
  25. data/lib/evilution/ast/skip_sites/error_messages.rb +74 -0
  26. data/lib/evilution/ast/skip_sites/logging.rb +93 -0
  27. data/lib/evilution/ast/skip_sites/message.rb +44 -0
  28. data/lib/evilution/ast/skip_sites/nodes.rb +105 -0
  29. data/lib/evilution/ast/skip_sites/presets.rb +31 -0
  30. data/lib/evilution/ast/skip_sites/purity.rb +69 -0
  31. data/lib/evilution/ast/skip_sites/requires.rb +40 -0
  32. data/lib/evilution/ast/skip_sites.rb +9 -0
  33. data/lib/evilution/ast/source_surgeon.rb +20 -2
  34. data/lib/evilution/ast/syntax_check.rb +10 -8
  35. data/lib/evilution/baseline/report.rb +11 -2
  36. data/lib/evilution/baseline.rb +31 -6
  37. data/lib/evilution/cache/keys.rb +140 -0
  38. data/lib/evilution/cache/run_fingerprint.rb +51 -0
  39. data/lib/evilution/cache/skeleton.rb +57 -0
  40. data/lib/evilution/cache/spec_digest.rb +39 -0
  41. data/lib/evilution/cache.rb +143 -50
  42. data/lib/evilution/class_tree.rb +38 -0
  43. data/lib/evilution/cli/command.rb +12 -4
  44. data/lib/evilution/cli/commands/run.rb +12 -3
  45. data/lib/evilution/cli/commands/subjects.rb +5 -2
  46. data/lib/evilution/cli/parser/options_builder.rb +67 -3
  47. data/lib/evilution/cli/printers/environment.rb +34 -4
  48. data/lib/evilution/cli/printers/session_detail.rb +20 -3
  49. data/lib/evilution/compare/categorizer.rb +2 -1
  50. data/lib/evilution/compare/normalizer.rb +3 -2
  51. data/lib/evilution/config/sources.rb +3 -0
  52. data/lib/evilution/config/validators/base.rb +9 -0
  53. data/lib/evilution/config/validators/since.rb +22 -0
  54. data/lib/evilution/config/validators/skip_sites.rb +24 -0
  55. data/lib/evilution/config/validators/timeout_ceiling.rb +10 -0
  56. data/lib/evilution/config/validators/timeout_factor.rb +10 -0
  57. data/lib/evilution/config/validators/timeout_floor.rb +10 -0
  58. data/lib/evilution/config.rb +122 -6
  59. data/lib/evilution/coverage/load_tracker.rb +50 -0
  60. data/lib/evilution/coverage/map.rb +42 -5
  61. data/lib/evilution/coverage/map_builder.rb +123 -22
  62. data/lib/evilution/coverage/map_store.rb +15 -5
  63. data/lib/evilution/coverage/recorder.rb +42 -3
  64. data/lib/evilution/coverage_example_filter.rb +35 -0
  65. data/lib/evilution/equivalent/heuristic/comment_marking.rb +14 -1
  66. data/lib/evilution/example_filter.rb +20 -0
  67. data/lib/evilution/git/changed_lines.rb +103 -0
  68. data/lib/evilution/git/since_scope.rb +77 -0
  69. data/lib/evilution/git/unified_diff.rb +68 -0
  70. data/lib/evilution/infra_crash.rb +26 -0
  71. data/lib/evilution/integration/base.rb +97 -6
  72. data/lib/evilution/integration/crash_detector.rb +46 -6
  73. data/lib/evilution/integration/known_failures.rb +4 -0
  74. data/lib/evilution/integration/loading/body_call_neutralizer.rb +49 -11
  75. data/lib/evilution/integration/loading/callback_equivalence.rb +62 -0
  76. data/lib/evilution/integration/loading/callback_redeclaration.rb +151 -36
  77. data/lib/evilution/integration/loading/concern_redeclaration.rb +16 -12
  78. data/lib/evilution/integration/loading/concern_registry.rb +87 -0
  79. data/lib/evilution/integration/loading/concern_state_cleaner.rb +19 -8
  80. data/lib/evilution/integration/loading/created_modules.rb +56 -0
  81. data/lib/evilution/integration/loading/frozen_namespaces.rb +80 -0
  82. data/lib/evilution/integration/loading/mutation_applier.rb +42 -11
  83. data/lib/evilution/integration/loading/neutralization.rb +41 -0
  84. data/lib/evilution/integration/loading/redefinition_recovery.rb +5 -3
  85. data/lib/evilution/integration/loading/superseded_modules.rb +22 -19
  86. data/lib/evilution/integration/minitest/stop_hook.rb +57 -0
  87. data/lib/evilution/integration/minitest.rb +22 -2
  88. data/lib/evilution/integration/minitest_crash_detector.rb +19 -0
  89. data/lib/evilution/integration/rspec/baseline_runner.rb +3 -2
  90. data/lib/evilution/integration/rspec/crash_detector_lifecycle.rb +6 -1
  91. data/lib/evilution/integration/rspec/example_filter_applier.rb +16 -0
  92. data/lib/evilution/integration/rspec/example_ids.rb +23 -0
  93. data/lib/evilution/integration/rspec/example_order.rb +86 -0
  94. data/lib/evilution/integration/rspec/killer_first.rb +72 -0
  95. data/lib/evilution/integration/rspec/result_builder.rb +7 -0
  96. data/lib/evilution/integration/rspec/state_guard/example_group_descendants.rb +44 -0
  97. data/lib/evilution/integration/rspec/state_guard.rb +2 -2
  98. data/lib/evilution/integration/rspec/target_resolver.rb +42 -0
  99. data/lib/evilution/integration/rspec/test_file_resolver.rb +4 -2
  100. data/lib/evilution/integration/rspec.rb +98 -47
  101. data/lib/evilution/integration/stop_policy.rb +49 -0
  102. data/lib/evilution/integration/test_unit/dispatcher.rb +20 -1
  103. data/lib/evilution/integration/test_unit/subject_class_registry.rb +6 -1
  104. data/lib/evilution/integration/test_unit.rb +28 -2
  105. data/lib/evilution/isolation/fork.rb +58 -13
  106. data/lib/evilution/isolation/in_process.rb +12 -1
  107. data/lib/evilution/line_set.rb +72 -0
  108. data/lib/evilution/mcp/info_tool/actions/environment.rb +12 -16
  109. data/lib/evilution/mcp/info_tool/actions/subjects.rb +3 -5
  110. data/lib/evilution/mcp/info_tool/status_glossary.rb +15 -2
  111. data/lib/evilution/mcp/info_tool.rb +24 -17
  112. data/lib/evilution/mcp/mutate_tool/config_builder.rb +13 -1
  113. data/lib/evilution/mcp/mutate_tool/option_parser.rb +2 -2
  114. data/lib/evilution/mcp/mutate_tool/progress_streamer.rb +1 -1
  115. data/lib/evilution/mcp/mutate_tool/report_trimmer.rb +28 -5
  116. data/lib/evilution/mcp/mutate_tool/survived_enricher.rb +12 -7
  117. data/lib/evilution/mcp/mutate_tool.rb +32 -7
  118. data/lib/evilution/mcp/session_tool.rb +1 -2
  119. data/lib/evilution/mcp.rb +4 -0
  120. data/lib/evilution/mutation/line_diff.rb +111 -0
  121. data/lib/evilution/mutation.rb +17 -13
  122. data/lib/evilution/mutator/base.rb +121 -98
  123. data/lib/evilution/mutator/constant_value_subjects.rb +29 -1
  124. data/lib/evilution/mutator/line_range_filter.rb +27 -0
  125. data/lib/evilution/mutator/load_sources.rb +88 -0
  126. data/lib/evilution/mutator/operator/alias_removal.rb +2 -5
  127. data/lib/evilution/mutator/operator/argument_method_call_replacement.rb +3 -0
  128. data/lib/evilution/mutator/operator/attr_accessor_swap.rb +2 -5
  129. data/lib/evilution/mutator/operator/constant_namespace_strip.rb +3 -3
  130. data/lib/evilution/mutator/operator/freeze_removal.rb +29 -11
  131. data/lib/evilution/mutator/operator/method_call_removal.rb +3 -1
  132. data/lib/evilution/mutator/operator/mixin_removal.rb +2 -5
  133. data/lib/evilution/mutator/operator/superclass_removal.rb +2 -5
  134. data/lib/evilution/mutator/operator/symbol_literal.rb +1 -1
  135. data/lib/evilution/mutator/operator/visibility_removal.rb +2 -5
  136. data/lib/evilution/mutator/registry.rb +11 -5
  137. data/lib/evilution/mutator/restore_source_cache.rb +41 -0
  138. data/lib/evilution/parallel/pool.rb +11 -4
  139. data/lib/evilution/parallel/work_queue/dispatcher/result_stream.rb +40 -0
  140. data/lib/evilution/parallel/work_queue/dispatcher.rb +20 -7
  141. data/lib/evilution/parallel/work_queue/worker/loop.rb +7 -2
  142. data/lib/evilution/parallel/work_queue/worker.rb +10 -3
  143. data/lib/evilution/parallel/work_queue/worker_stat.rb +7 -0
  144. data/lib/evilution/parallel/work_queue.rb +69 -12
  145. data/lib/evilution/parallel_db_warning.rb +39 -12
  146. data/lib/evilution/process_supervisor.rb +22 -2
  147. data/lib/evilution/reporter/cli/item_formatters/coverage_gap.rb +13 -2
  148. data/lib/evilution/reporter/cli/item_formatters/subject_score.rb +5 -1
  149. data/lib/evilution/reporter/cli/line_formatters/mutations.rb +9 -2
  150. data/lib/evilution/reporter/cli/line_formatters/result_line.rb +9 -1
  151. data/lib/evilution/reporter/cli/line_formatters/skipped_sites.rb +14 -0
  152. data/lib/evilution/reporter/cli/line_formatters/uncovered_code.rb +17 -0
  153. data/lib/evilution/reporter/cli/metrics_block.rb +4 -0
  154. data/lib/evilution/reporter/cli/trailer.rb +4 -4
  155. data/lib/evilution/reporter/cli.rb +9 -2
  156. data/lib/evilution/reporter/html/assets/style.css +8 -0
  157. data/lib/evilution/reporter/html/sections/file_section.rb +9 -0
  158. data/lib/evilution/reporter/html/sections/uncovered_details.rb +27 -0
  159. data/lib/evilution/reporter/html/templates/file_section.html.erb +2 -1
  160. data/lib/evilution/reporter/html/templates/summary_cards.html.erb +3 -0
  161. data/lib/evilution/reporter/html/templates/uncovered_details.html.erb +11 -0
  162. data/lib/evilution/reporter/json/mutation_detail.rb +77 -0
  163. data/lib/evilution/reporter/json/subjects.rb +1 -0
  164. data/lib/evilution/reporter/json/timings.rb +42 -0
  165. data/lib/evilution/reporter/json.rb +13 -50
  166. data/lib/evilution/result/coverage_gap.rb +5 -0
  167. data/lib/evilution/result/mutation_result.rb +55 -3
  168. data/lib/evilution/result/mutation_timings.rb +54 -0
  169. data/lib/evilution/result/run_timings.rb +20 -0
  170. data/lib/evilution/result/subject_score.rb +8 -1
  171. data/lib/evilution/result/subject_scorer.rb +5 -3
  172. data/lib/evilution/result/summary.rb +49 -12
  173. data/lib/evilution/runner/baseline_runner.rb +47 -5
  174. data/lib/evilution/runner/diagnostics.rb +0 -13
  175. data/lib/evilution/runner/isolation_resolver.rb +16 -4
  176. data/lib/evilution/runner/mutation_executor/isolated_run.rb +34 -0
  177. data/lib/evilution/runner/mutation_executor/kill_history.rb +66 -0
  178. data/lib/evilution/runner/mutation_executor/mutation_runner.rb +31 -12
  179. data/lib/evilution/runner/mutation_executor/neutralizer/baseline_failed.rb +1 -0
  180. data/lib/evilution/runner/mutation_executor/neutralizer/infra_error.rb +6 -20
  181. data/lib/evilution/runner/mutation_executor/probe_first.rb +99 -0
  182. data/lib/evilution/runner/mutation_executor/probe_first_run.rb +59 -0
  183. data/lib/evilution/runner/mutation_executor/result_cache.rb +45 -14
  184. data/lib/evilution/runner/mutation_executor/result_notifier.rb +1 -1
  185. data/lib/evilution/runner/mutation_executor/result_packer.rb +54 -24
  186. data/lib/evilution/runner/mutation_executor/runless_verdict.rb +39 -0
  187. data/lib/evilution/runner/mutation_executor/strategy/parallel/hinted_items.rb +27 -0
  188. data/lib/evilution/runner/mutation_executor/strategy/parallel/ordered_delivery.rb +97 -0
  189. data/lib/evilution/runner/mutation_executor/strategy/parallel.rb +102 -83
  190. data/lib/evilution/runner/mutation_executor/timed_run.rb +33 -0
  191. data/lib/evilution/runner/mutation_executor/timeout_policy/adaptive.rb +67 -0
  192. data/lib/evilution/runner/mutation_executor/timeout_policy/base.rb +35 -0
  193. data/lib/evilution/runner/mutation_executor/timeout_policy/fixed.rb +17 -0
  194. data/lib/evilution/runner/mutation_executor/timeout_policy.rb +56 -0
  195. data/lib/evilution/runner/mutation_executor.rb +75 -10
  196. data/lib/evilution/runner/mutation_planner.rb +29 -21
  197. data/lib/evilution/runner/phase_clock.rb +22 -0
  198. data/lib/evilution/runner/report_publisher.rb +1 -1
  199. data/lib/evilution/runner/subject_pipeline.rb +18 -8
  200. data/lib/evilution/runner.rb +87 -26
  201. data/lib/evilution/session/diff.rb +9 -2
  202. data/lib/evilution/session/store.rb +17 -2
  203. data/lib/evilution/source_ast_cache.rb +10 -0
  204. data/lib/evilution/spec_ast_cache.rb +15 -0
  205. data/lib/evilution/spec_resolver/constant_references.rb +66 -0
  206. data/lib/evilution/spec_resolver/project_files.rb +50 -0
  207. data/lib/evilution/spec_resolver/sibling_spec_ranker.rb +111 -0
  208. data/lib/evilution/spec_resolver/spec_pattern.rb +19 -0
  209. data/lib/evilution/spec_resolver.rb +76 -65
  210. data/lib/evilution/spec_selector.rb +17 -13
  211. data/lib/evilution/subject.rb +7 -2
  212. data/lib/evilution/version.rb +1 -1
  213. data/lib/evilution/worker_database/copies.rb +73 -0
  214. data/lib/evilution/worker_database/schema.rb +39 -0
  215. data/lib/evilution/worker_database/sqlite_copy.rb +65 -0
  216. data/lib/evilution/worker_database.rb +214 -0
  217. data/schema/evilution.config.schema.json +74 -2
  218. metadata +84 -13
  219. data/lib/evilution/integration/rspec/state_guard/object_space_example_groups.rb +0 -42
  220. data/scripts/benchmark_density +0 -262
  221. data/scripts/benchmark_density.yml +0 -19
  222. data/scripts/canary_manifest.yml +0 -47
  223. data/scripts/compare_mutations +0 -421
  224. data/scripts/compare_mutations.yml +0 -24
  225. data/scripts/compare_targeting +0 -279
  226. data/scripts/compare_targeting.example.yml +0 -24
  227. data/scripts/mutant_json_adapter +0 -227
@@ -0,0 +1,387 @@
1
+ # Mutant schemata: spike report
2
+
3
+ Status: **no-go**. No prototype in `lib/`. The prototype was a harness
4
+ loaded from outside the gem, kept in `scripts/schemata_spike/` (see
5
+ [Reproducing](#reproducing)). Two cheaper
6
+ changes, found while measuring, are recommended instead
7
+ ([below](#what-to-do-instead)).
8
+
9
+ ## Idea
10
+
11
+ Mutation switching ("mutant schemata", Untch; Stryker's mutant switching):
12
+ build one meta-version of a file with every method-body mutant behind a
13
+ runtime switch, evaluate it once in the parent before any mutation process
14
+ forks, and have each mutation process only set the active mutant's id. That
15
+ removes the per-mutant parse, evaluate and restore of the whole file. Class
16
+ bodies, constants, visibility and load-time mutants keep the eval path.
17
+
18
+ ## How much there is to save
19
+
20
+ The fixed cost schemata can remove is the `apply` phase of each mutation
21
+ (syntax check, constant pinning, concern cleanup, eval of the file; the
22
+ in-process restore too). It is not the isolation around it (fork, sandbox,
23
+ reap) or the test run. That phase was measured first, from the per-mutation
24
+ `timings` in the JSON report, on the current tree (everything up to the
25
+ operator yield study, so parse-once generation and warm parent caches are
26
+ already in). Each target ran at `--jobs 2`.
27
+
28
+ | Target | Isolation | Mutants | `apply` per mutant (mean / p50) | Isolation overhead per mutant | `test_run` per mutant | `apply` share of killtime |
29
+ |---|---|---|---|---|---|---|
30
+ | fixture (`spec/support/fixtures/benchmark_project`) | in_process | 440 | 0.40 / 0.40 ms | 0.8 ms | 32.8 ms (+ confirmation) | **0.1%** |
31
+ | dry-monads `lib/dry/monads/maybe.rb` | fork | 611 | 5.2 / 5.1 ms | 10.5 ms | 117 ms | **3.6%** |
32
+ | i18n `lib/i18n/backend/base.rb` (Minitest) | fork | 1837 | 8.3 / 7.9 ms | 9.5 ms | 22.5 ms | **20.5%** |
33
+
34
+ So the most schemata can win is what `apply` costs today, and that is only
35
+ large where the tests are fast (i18n: 22 ms per test run).
36
+
37
+ Where that `apply` time goes was measured by wrapping each step in a real
38
+ i18n run (mean per mutant, fork child):
39
+
40
+ | Step | ms |
41
+ |---|---|
42
+ | `ConstantPinner` (a full `Prism.parse` of the **original** file, to list its constants) | **6.08** |
43
+ | `SourceEvaluator` (the eval itself) | 0.96 |
44
+ | `SyntaxValidator` (`Prism.parse_success?`, no tree) | 0.37 |
45
+ | `mark_feature_loaded` | 0.12 |
46
+ | `ConcernStateCleaner` (no ActiveSupport here) | 0.01 |
47
+ | **Total `apply`** | **8.05** |
48
+
49
+ Three quarters of `apply` is the constant pinner building a full syntax tree
50
+ of the unchanged original file, in every mutation process, and getting the
51
+ same names each time. The same parse takes 0.5 ms in a warm parent; in a
52
+ freshly forked child it costs about 6 ms on almost every call (p50 6.0 ms).
53
+ GC is not the reason: only 26 of the 1831 calls ran a GC. The cost is per
54
+ call, in allocating the tree on the child's fresh heap and the copy-on-write
55
+ page faults of those first writes. The eval that schemata would remove costs
56
+ about 1 ms.
57
+
58
+ ## Prototype
59
+
60
+ A harness (`RUBYOPT=-r<harness>`), not code in `lib/`. It prepends two hooks:
61
+
62
+ - **Parent**, after `MutationExecutor#warm`, before workers or mutation
63
+ processes fork: per file, it takes each mutation's change as the common
64
+ prefix and suffix of the neutralized original (`restore_source`) and its
65
+ `eval_source`. A mutation is eligible when the change lies inside the body
66
+ of a `def` that is nested only in `class`, `module` and `class << self`
67
+ bodies (or wrapped in `private def ...`). Each such method is rebuilt as
68
+
69
+ ```ruby
70
+ def name(params)
71
+ case $__evilution_schema when nil then begin <original body, on its own lines>
72
+ end when 17 then begin
73
+ <body with mutation 17>
74
+ end when 18 then begin
75
+ ...
76
+ end else begin
77
+ <original body>
78
+ end end
79
+ end
80
+ ```
81
+
82
+ and evaluated **on its own**, wrapped in the file's nesting written on the
83
+ `def`'s line (`module I18n; module Backend; module Base; def ...`) and
84
+ with the file's path and the `def`'s line number. Nothing else in the file
85
+ runs again: no class body, no neutralizer, no `alias` line. Every line
86
+ outside the variants keeps its number, and so does the original branch.
87
+ After the redefinition it re-applies the method's visibility, its
88
+ `module_function` copy and the file's `alias`/`alias_method`s of it.
89
+ The redefinitions and re-aliases are wrapped in `ReevalWarningFilter`, since
90
+ dry-monads' spec helper turns "method redefined" warnings into errors.
91
+ - **Mutation process**: `MutationApplier#call` sets `$__evilution_schema` to
92
+ the mutation's id and returns. `#restore` clears it. Under `in_process`, a
93
+ restore after a mutation of the same file that took the eval path puts the
94
+ meta-methods back.
95
+
96
+ Not eligible: changes outside a static `def` body (constants, class-body
97
+ calls, signatures and default arguments, `def`s inside blocks such as
98
+ `included do`, `Module.new do`, `Class.new do`), endless `def`s (the
99
+ prototype did not rewrite them), scope/aasm/callback/concern-hook subjects
100
+ (their restore source keeps a declaration of their own), variants containing
101
+ `__LINE__`, `binding` or `local_variables`, and the local-variable conflict
102
+ described below.
103
+
104
+ ## Correctness questions
105
+
106
+ **Which mutants.** Method bodies only, of `def`s reachable through
107
+ class/module/singleton-class bodies, including `def self.x` and
108
+ `class << self`. Blocks and lambdas inside a body are part of the body and
109
+ switch with it. Not covered: methods built by `define_method` (their body is
110
+ a block evaluated at load time; switching it means re-running the class-body
111
+ call), `def`s inside blocks (concern `included`/`class_methods`, `Module.new`,
112
+ `Class.new`, `Struct.new do`), and endless `def`s. Endless `def`s can be
113
+ rewritten to the long form (an extra 150 of 610 on `maybe.rb`).
114
+
115
+ **Methods redefined later in the file.** Each `def` gets its own switch, and
116
+ the later definition wins at load time, as it does when the file is
117
+ evaluated. Equivalent.
118
+
119
+ **Aliases.** `alias new old` copies the method entry at the moment it runs.
120
+ Evaluating the whole file re-runs the alias, so the alias gets the mutant.
121
+ Redefining one method does not. The prototype re-aliases the file's own
122
+ aliases of the method after redefining it. An alias made in another file
123
+ keeps the original under both paths. A load-time capture other than an alias
124
+ (`HANDLERS = { a: method(:foo) }`, `define_method(:b, instance_method(:a))`,
125
+ a memoizer that wraps by copying the method) keeps the original under
126
+ schemata but gets the mutant under the eval path, unless the neutralizer
127
+ blanked the capturing call. The prototype does not detect these: none
128
+ occurred on the targets, so it can only be counted as an open risk.
129
+
130
+ **Visibility and `module_function`.** A `def` in a reopened class is public.
131
+ The original visibility has to be read before the redefinition and set again
132
+ after it, and a `module_function` method's singleton copy has to be made
133
+ again. The prototype does both.
134
+
135
+ **Local variables across branches.** Ruby decides at parse time whether a
136
+ bare `x` is a local or a method call, by whether an assignment to `x` comes
137
+ before it in the method. In the meta-method every branch shares one local
138
+ table, so a variant that drops `x = compute` and keeps a later `x` turns
139
+ that `x` from a `NoMethodError`/`NameError` (eval path) into `nil`
140
+ (schemata). The prototype parses each variant and drops it when a bare call
141
+ in it names a local assigned in an earlier branch. That dropped 6 of 440
142
+ (fixture), 30 of 1830 (i18n) and 6 of 610 (`maybe.rb`).
143
+
144
+ **Source locations, `__method__`, `super`, `return`, `binding`.** `return`,
145
+ `yield`, `block_given?`, `super` (implicit and explicit), `__method__` and
146
+ `retry` in a body-level `rescue` behave the same inside `case ... begin ...
147
+ end`, because the branch is still the method's own frame. The original
148
+ branch keeps its lines. A variant's lines do not: they come after the
149
+ original body, so a backtrace from inside the active mutant, `__LINE__`, or
150
+ an exception message carrying a line differ from the eval path. Variants
151
+ using `__LINE__` were dropped. `binding` and `local_variables` see the
152
+ locals of every branch (as `nil`), so variants using them were dropped too.
153
+ `Method#source_location` points at the `def` line in both paths.
154
+
155
+ **Runtime cost of the switch.** One global read plus `opt_case_dispatch`
156
+ per call: a method doing `x + 1` went from 17 ns to 36 ns per call (+19 ns).
157
+ A frozen per-file array read instead of the global cost +10 ns. That is
158
+ negligible next to a method that does real work. It is paid on every call
159
+ in every mutation process, including calls of mutants of other methods.
160
+
161
+ **Isolation and workers.** Under `fork` (and the persistent workers, which
162
+ fork a process per mutation), the meta-version is built once in the parent
163
+ before the workers fork, and the mutation process only sets the global. Under
164
+ `in_process`, a schemata mutant needs no restore. An eval-path mutant of the
165
+ same file replaces the meta-methods with its own and must be followed by
166
+ the meta-methods, not just the original (the prototype does that). The
167
+ fixture ran in-process at `--jobs 2` with identical verdicts.
168
+
169
+ **Neutralizer and preload.** Redefining single methods sidesteps the
170
+ neutralizer entirely: no class body runs again. That is the main correctness
171
+ gain over evaluating a whole meta-file, which would re-run class bodies once
172
+ in the parent like any eval. The file must be loaded in the parent first
173
+ (preload does that for gems and Rails; the fixture's files were `require`d
174
+ by the harness). For Rails, a meta-method defined before workers fork is
175
+ inherited like any preloaded code. The prototype was not run on Redmine.
176
+
177
+ **Memory.** The parent's RSS grew by 0.8 MB (fixture), 0.25 MB (`maybe.rb`)
178
+ and **16.5 MB** (i18n `base.rb`: 1751 variants, most of them copies of a few
179
+ long methods). Building and evaluating them took 0.17 s, 0.31 s and **2.3 s**,
180
+ serially in the parent before the first mutation.
181
+
182
+ ## Results
183
+
184
+ Eligible share (prototype rules):
185
+
186
+ | Target | Eligible | Outside a static `def` body | Endless `def` | Local conflict | `__LINE__`/`binding` |
187
+ |---|---|---|---|---|---|
188
+ | fixture | 409 / 440 (93%) | 25 | 0 | 6 | 0 |
189
+ | i18n `base.rb` | 1751 / 1830 (96%) | 26 | 0 | 30 | 23 |
190
+ | dry-monads `maybe.rb` | 395 / 610 (65%) | 59 | 150 | 6 | 0 |
191
+ | **Total** | **2555 / 2880 (89%)** | 110 | 150 | 42 | 23 |
192
+
193
+ (Mutants reported without a run, here the equivalent ones, never reach
194
+ `apply` and are not counted.)
195
+
196
+ **Verdict equivalence:** identical on every run. Each mutant was matched by
197
+ file, line, operator and diff against the eval-path run: fixture 440/440,
198
+ i18n 1837/1837 and dry-monads `maybe.rb` 611/611 (one schemata run for the
199
+ fixture, three for dry-monads, and every schemata run on i18n, probes
200
+ included).
201
+
202
+ **`apply` per mutant**, eval path vs schemata (mean, all mutants):
203
+
204
+ | Target | Eval path | Schemata | Saved per mutant |
205
+ |---|---|---|---|
206
+ | fixture | 0.40 ms | 0.11 ms | 0.3 ms |
207
+ | dry-monads `maybe.rb` | 5.2 ms | 2.5 ms (eligible: 0.1 ms) | 2.7 ms |
208
+ | i18n `base.rb` | 8.3 ms | 0.5 ms | 7.8 ms |
209
+
210
+ **Wall time and killtime**, same tree, schemata off and on alternated
211
+ (seconds):
212
+
213
+ | Target | Off: duration / killtime | On: duration / killtime | Change in duration |
214
+ |---|---|---|---|
215
+ | i18n, `--jobs 2` | 39.7 / 76.7, 40.2 / 77.7 | 42.8 / 80.0, 43.6 / 81.5 | **+8% (slower)** |
216
+ | dry-monads `maybe.rb`, `--jobs 2` | 45.3 / 88.8, 43.5 / 85.3, 43.8 / 85.8 | 44.9 / 87.3, 50.2 / 97.6, 43.3 / 83.9 | none measurable (`apply` saves 2.6 s of ~86 s killtime; one run on was a 15% outlier) |
217
+ | fixture, `--jobs 2` (one pair) | 77.7 / 149.7 | 83.9 / 160.8 | no gain possible: `apply` is 0.17 s of the run; the difference is in test and confirmation time |
218
+
219
+ On i18n, schemata removes 16 s of `apply` and adds 19 s of `test_run`
220
+ (42.2 / 42.8 s → 61.5 / 62.0 s). The tests themselves got slower. Four
221
+ measurements, all with verdicts identical to the eval path, narrow down
222
+ why:
223
+
224
+ - **GC per mutation process** went from 10.6 to 13.4 ms, and major GCs from
225
+ 0.115 to 0.258 per mutant, with the parent's heap 16.5 MB larger (one run
226
+ each with GC counters).
227
+ - **Full GC in the parent before forking** (`Process.warmup` after the
228
+ meta-methods are built: full GC, compaction, heap trim), three rounds
229
+ alternated with schemata alone, the eval path alone and the eval path
230
+ with the same warm-up. The machine was busier than in the runs above, so
231
+ compare within this table only (means of three runs, seconds):
232
+
233
+ | i18n, `--jobs 2` | duration | killtime | `test_run` | GC per mutant | major GCs per mutant |
234
+ |---|---|---|---|---|---|
235
+ | eval path | 48.0 | 92.6 | 51.5 | 12.2 ms | 0.14 |
236
+ | schemata | 55.1 | 103.5 | 79.9 | 16.5 ms | 0.31 |
237
+ | schemata + parent warm-up | 44.3 | 81.1 | 58.5 | 7.4 ms | 0.14 |
238
+ | eval path + parent warm-up | 42.9 | 82.2 | 41.7 | 6.9 ms | 0.17 |
239
+
240
+ The warm-up removes the GC penalty of the bigger heap: with it, the child
241
+ spends about as long in GC under schemata as under the eval path. It does
242
+ not remove the rest: `test_run` is still 16.7 s (9 ms per mutant) higher
243
+ than the eval path with the same warm-up.
244
+ - **Re-evaluating the meta-methods inside each mutation process** (fresh
245
+ copies compiled in the child, `apply` 252 s) brought `test_run` back to
246
+ the eval path's 42 s. So the switch and the size of the methods do not
247
+ slow the tests by themselves.
248
+ - **`GC.start` inside each mutation process**, after setting the switch and
249
+ before the tests (`apply` 65 s), brought `test_run` down to 29.1 s, below
250
+ the eval path's 42 s, because the tests then ran with no GC debt at all.
251
+ A full GC in the child also touches the whole inherited heap, so this
252
+ probe moves page faults out of the tests as well as GC, and does not
253
+ separate the two.
254
+
255
+ So of the 28 s of extra `test_run` in those rounds, about 12 s (40%) was GC
256
+ from the larger parent heap, which a pre-fork warm-up removes. The
257
+ remaining 17 s (60%) is tied to running methods compiled in the parent
258
+ rather than in the child. First-call writes
259
+ (inline caches) into the parent's instruction sequences, each copying a
260
+ shared page, would explain it, but that mechanism was not measured.
261
+
262
+ On top of that the parent spends 2.3 s building and evaluating the
263
+ meta-methods before the first mutation, serially. That time was not broken
264
+ down; it covers two extra `Prism.parse` calls per variant in the
265
+ local-conflict guard and compiling methods that hold a full copy of the
266
+ body per variant (1751 copies of a few long methods on i18n).
267
+
268
+ ## Decision
269
+
270
+ **No-go**, and it still holds with the pre-fork warm-up measured above.
271
+ Four reasons:
272
+
273
+ 1. **The ceiling is low.** What schemata removes is `apply`: 0.1% of
274
+ killtime on the fixture, 3.6% on dry-monads `maybe.rb`, 20% on i18n.
275
+ Three quarters of that 20% is one parse (below) that needs no schemata
276
+ to remove.
277
+ 2. **On the one target where the ceiling is worth having, it did not win.**
278
+ As built (whole-body variants, no pre-fork GC) i18n ran 8% slower. A
279
+ `Process.warmup` in the parent removes the GC part of that loss, but the
280
+ same warm-up speeds up the eval path too, and against that schemata
281
+ ties (killtime -1%, wall +3%): the tests still run 9 ms per
282
+ mutant slower on parent-compiled methods, which eats the saved `apply`.
283
+ dry-monads came out even. This is measured for this whole-body design
284
+ only; switching at the mutated expression (Untch, Stryker) would cut
285
+ the heap growth and build time, and was not built.
286
+ 3. **The correctness surface is large for what is left.** Variant line
287
+ numbers shift. Local variables leak between branches and need a parse per
288
+ variant to guard. Visibility, `module_function` and aliases have to be
289
+ re-applied by hand. Load-time captures of a method (`method(:x)`,
290
+ `instance_method`, memoizers) diverge from the eval path silently.
291
+ `define_method`, `def`s in blocks and endless `def`s are excluded or need
292
+ rewriting. Verdicts were identical on 2888 mutants across three targets,
293
+ but each of these is a way for a later target to disagree.
294
+ 4. **It does not compose with what the eval path already has.** Neutralized
295
+ class bodies, redeclared subjects (scope, aasm, callbacks, concern hooks)
296
+ and `load_crashed` detection stay on the eval path. Schemata would be a
297
+ second way to apply a mutation, to be kept equivalent with the first.
298
+
299
+ Reason 1 holds on its own: even a version that cost nothing could remove at
300
+ most a fifth of i18n's killtime, and three quarters of that a parse memo
301
+ removes without schemata. Reasons 3 and 4 hold for any version.
302
+ Expression-level switching was not built: it saves the same `apply`, keeps
303
+ reasons 1, 3 and 4, and whether it also avoids the slower tests of reason 2
304
+ is open.
305
+
306
+ ## What to do instead
307
+
308
+ **Parse the original's constant names once, in the parent.**
309
+ `ConstantPinner#call(mutation.original_source)` runs `Prism.parse` on the
310
+ same unchanged source in every mutation process. Measured with the names
311
+ memoized per source and filled in `MutationExecutor#warm` (the same harness,
312
+ `EV_PIN_MEMO=1`), i18n alternated off/on, two runs each:
313
+
314
+ | | `apply` total | killtime | duration |
315
+ |---|---|---|---|
316
+ | off | 17.97 s, 17.58 s | 81.6 s, 79.4 s | 42.1 s, 41.1 s |
317
+ | on | 8.52 s, 8.77 s | 72.6 s, 75.6 s | 37.7 s, 39.2 s |
318
+
319
+ That is about **-8% killtime and -8% wall on i18n** (killtime 80.5 → 74.1 s,
320
+ wall 41.6 → 38.4 s, means of two runs), with verdicts identical
321
+ (1837/1837), using a cache filled in the parent the way the concern
322
+ registry already is. Of the 9.1 s of `apply` saved, about 6.4 s reaches
323
+ killtime: `test_run` rose with the memo on (45.4 / 43.9 → 46.2 / 48.6 s),
324
+ probably the same shift of GC work into the tests that schemata shows
325
+ (not measured for the memo). Two runs each is thin: unmodified i18n runs in
326
+ the same session ranged from 38.2 to 42.1 s, about the size of the gain. It
327
+ is a small change with no new way to apply a mutation. Worth filing as its
328
+ own issue, and the gain should be confirmed there with
329
+ `scripts/benchmark_perf` (interleaved repeats), which this spike did not
330
+ use. `RedefinitionRecovery` parses the original too, but only on a
331
+ conflict.
332
+
333
+ **Collect and compact the parent's heap once before forking.** In the
334
+ warm-up rounds above, `Process.warmup` in the parent after
335
+ `MutationExecutor#warm` cut the eval path's GC per mutation process from
336
+ 12.2 to 6.9 ms and its `test_run` from 51.5 to 41.7 s: **-11% killtime and
337
+ -11% wall on i18n** (92.6 → 82.2 s, 48.0 → 42.9 s, means of three
338
+ alternated runs, verdicts identical). It is one call with no new way to
339
+ apply a mutation. Worth its own issue, measured there with
340
+ `scripts/benchmark_perf` on more than one canary (it compacts the heap,
341
+ which a C extension that does not support compaction could mind).
342
+
343
+ The rest of the per-mutant fixed cost is the isolation itself (fork, sandbox,
344
+ reap: 9.5–10.5 ms per mutant under `fork`, more than `apply` on every
345
+ target) and per-mutation framework reset inside `test_run`. Schemata touches
346
+ neither.
347
+
348
+ ## Reproducing
349
+
350
+ The harness, the timing wrapper and the aggregation scripts are in
351
+ [`scripts/schemata_spike/`](../scripts/schemata_spike/); its README lists
352
+ every environment switch. Corpus: copies of the dry-monads and i18n canaries
353
+ (pinned commits in `scripts/benchmark_perf.yml`) and of the fixture, each
354
+ with a Gemfile adding `gem "evilution", path: <tree>`.
355
+
356
+ - Baseline: the per-mutation `timings` (`apply`, `test_run`,
357
+ `confirmation`) of a normal JSON run. Isolation overhead per mutant is
358
+ `duration` minus those.
359
+ - Step breakdown (`apply_prof.rb`): prepends a timing wrapper to `call` of
360
+ `SyntaxValidator`, `ConstantPinner`, `ConcernStateCleaner`,
361
+ `SourceEvaluator`, `RedefinitionRecovery` and
362
+ `MutationApplier#mark_feature_loaded`, loaded with `RUBYOPT=-r<file>` (it
363
+ must `require "bundler/setup"` first).
364
+ - Schemata (`schemata.rb`): the harness described under
365
+ [Prototype](#prototype): a module
366
+ prepended to `MutationExecutor` whose private `warm(integration,
367
+ mutations)` calls `super` and then builds and evaluates the meta-methods,
368
+ and one prepended to `MutationApplier` whose `call`/`restore`
369
+ set and clear the global for the mutations it switched (keyed by file and
370
+ `mutated_source.hash`, stable across `fork`).
371
+ - Probes and the memo: switches of the same harness (`EV_SCHEMATA_REEVAL`,
372
+ `EV_SCHEMATA_GC`, `EV_PARENT_WARMUP`, `EV_GCSTAT`, `EV_PIN_MEMO`).
373
+ - Verdict equivalence (`compare_verdicts.rb`): match the two runs' JSON
374
+ entries by file, line, operator and diff, and compare statuses.
375
+
376
+ Commands (in the target dir, `--jobs 2`):
377
+
378
+ ```
379
+ # fixture
380
+ bundle exec evilution run lib/pricing.rb lib/inventory.rb lib/grade.rb --jobs 2 --format json
381
+ # dry-monads
382
+ bundle exec evilution run lib/dry/monads/maybe.rb --spec spec/unit/maybe_spec.rb \
383
+ --integration rspec --preload spec/spec_helper.rb --jobs 2 --format json
384
+ # i18n
385
+ RUBYOPT=-Itest bundle exec evilution run lib/i18n/backend/base.rb --spec test/i18n_test.rb \
386
+ --integration minitest --preload test/test_helper.rb --jobs 2 --format json
387
+ ```