mutineer 1.0.2 → 1.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +126 -0
- data/README.md +5 -2
- data/lib/mutineer/child_stdout.rb +35 -0
- data/lib/mutineer/coverage_map.rb +115 -56
- data/lib/mutineer/daemon_server.rb +2 -1
- data/lib/mutineer/isolation.rb +9 -3
- data/lib/mutineer/minitest_integration/stop_at_first_failure.rb +199 -0
- data/lib/mutineer/minitest_integration.rb +22 -8
- data/lib/mutineer/mutator_registry.rb +24 -4
- data/lib/mutineer/mutators/array_literal.rb +51 -0
- data/lib/mutineer/mutators/base.rb +15 -0
- data/lib/mutineer/mutators/chain_link.rb +139 -0
- data/lib/mutineer/mutators/negation_removal.rb +44 -0
- data/lib/mutineer/mutators/operand_removal.rb +65 -0
- data/lib/mutineer/mutators/range_literal.rb +54 -0
- data/lib/mutineer/mutators/safe_navigation.rb +76 -0
- data/lib/mutineer/pairing.rb +10 -5
- data/lib/mutineer/runner.rb +28 -7
- data/lib/mutineer/test_runners/minitest.rb +5 -1
- data/lib/mutineer/test_runners/rspec.rb +9 -11
- data/lib/mutineer/test_runners.rb +4 -2
- data/lib/mutineer/version.rb +1 -1
- data/lib/mutineer.rb +6 -0
- metadata +9 -15
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 2b48ca68bd77d82bd83917312fe4bc2377954403e0a63c99702c9059b9dde07b
|
|
4
|
+
data.tar.gz: 0c11eaa74d07db731902b2fe70313f71b17eded8a59f10e5652b86985c11641c
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 23505720f25f34b80bde51fb0a900c16a40295966ec8bd14713d22eca03ea0058a69d83b3f701672adee6a20b10f342f09d8b561d974f2a8f23d12a82ba22e46
|
|
7
|
+
data.tar.gz: cb2b659574bf1454ef265e3723b2c906485589696316cb63d9fb81d7696e04867e9fec73ee36b8d67063c7d941f4165fd35c922cc47cbc19466ec839ff8cbc2c
|
data/CHANGELOG.md
CHANGED
|
@@ -6,6 +6,130 @@ All notable changes to this project are documented here. The format is based on
|
|
|
6
6
|
|
|
7
7
|
## [Unreleased]
|
|
8
8
|
|
|
9
|
+
## [1.2.0] - 2026-09-28
|
|
10
|
+
|
|
11
|
+
### Added
|
|
12
|
+
- **Operand-removal operator** (Tier-2, opt-in via `--operators`):
|
|
13
|
+
`operand_removal` replaces `a && b` with `(a)` and with `(b)`, and does the
|
|
14
|
+
same for `||`, `and` and `or`. The mutant survives when no test needs the
|
|
15
|
+
operand that the mutant removes. The operator never keeps a jump operand
|
|
16
|
+
(`return`, `break`, `next`, `redo`, `retry`) alone, because a jump does not
|
|
17
|
+
parse in a value context. It never removes an operand that holds a heredoc,
|
|
18
|
+
because the heredoc body stays behind as code. It skips nested method
|
|
19
|
+
definitions, because mutineer mutates each one as its own method.
|
|
20
|
+
- **Array-literal operator** (Tier-2, opt-in via `--operators`):
|
|
21
|
+
`array_literal` replaces a non-empty array literal, such as `[a, b]` or
|
|
22
|
+
`%i[a b]`, with `[]`. The mutant survives when no test checks the contents
|
|
23
|
+
of the array. The operator skips an implicit array (`x = 1, 2`), an array
|
|
24
|
+
that holds a heredoc, and nested method definitions.
|
|
25
|
+
- **Sources also pair with Minitest's `test/**/test_*.rb` files** — after the
|
|
26
|
+
`_test.rb` forms, so existing projects pair as before. `lib/helper.rb` does
|
|
27
|
+
not pair with the `test/test_helper.rb` support file. A failed capture of a
|
|
28
|
+
`test_<name>.rb` file now marks `<name>.rb` uncapturable, as `<name>_test.rb`
|
|
29
|
+
does (#120). Without `framework:` set, a source with `spec/<name>_spec.rb`
|
|
30
|
+
and `test/test_<name>.rb` but no `<name>_test.rb` now pairs with the
|
|
31
|
+
Minitest file, as the "Minitest first" order says.
|
|
32
|
+
|
|
33
|
+
### Fixed
|
|
34
|
+
- **`reload` loads the mutant by an absolute path** — a relative source path
|
|
35
|
+
gave the mutant relative backtrace paths, so code that checks its own frames
|
|
36
|
+
by absolute path failed for every mutant, a false kill (#123).
|
|
37
|
+
- **`require "test_helper"` works without `RUBYOPT`** — a standalone run puts
|
|
38
|
+
`lib`, then each test file's `test_helper.rb` directory, on the load path,
|
|
39
|
+
as boot mode and `rake test` do. A run where no test records coverage because
|
|
40
|
+
captures failed now exits 1 instead of reporting N/A (#119).
|
|
41
|
+
- **A disable-line marker warns about an operator it does not know** — a
|
|
42
|
+
reason written without `--` became part of the operator name, so the marker
|
|
43
|
+
suppressed nothing and said nothing (#124). A marker followed only by spaces
|
|
44
|
+
or commas, such as `disable-line -- why`, now disables the whole line.
|
|
45
|
+
- **Coverage capture and the clean check run each source once** — they read
|
|
46
|
+
sources with `load`, so a test's own `require` ran them again: a `Struct`
|
|
47
|
+
superclass raised `superclass mismatch`, and load-time code ran twice (#122).
|
|
48
|
+
A mutant of such a class still errors under `--strategy reload`, which loads
|
|
49
|
+
the mutated file again; `--strategy redefine` runs it.
|
|
50
|
+
Code that guards itself to run once (`unless defined?(X)`) can now show as
|
|
51
|
+
covered, so its mutants run where they were `no_coverage` before.
|
|
52
|
+
- **A red unmutated suite now shows why it failed** — in a standalone run, the
|
|
53
|
+
Minitest summary or RSpec output of the failing test, with its failure
|
|
54
|
+
message, goes to stderr before the "not green" error. A passing run prints
|
|
55
|
+
nothing extra. Boot mode (`--rails`, `--boot`) is unchanged (#121).
|
|
56
|
+
|
|
57
|
+
## [1.1.0] - 2026-09-28
|
|
58
|
+
|
|
59
|
+
### Added
|
|
60
|
+
- **Safe-navigation operator** (Tier-2, opt-in via `--operators`):
|
|
61
|
+
`safe_navigation` replaces `&.` with `.`. The mutant survives when no test
|
|
62
|
+
passes `nil` to the call.
|
|
63
|
+
- **Range operator** (Tier-2, opt-in via `--operators`): `range` replaces
|
|
64
|
+
`..` with `...` and `...` with `..`. The `..` -> `...` mutant survives when
|
|
65
|
+
no test checks the last element of the range. Endless ranges (`1..`) are
|
|
66
|
+
skipped, because `(1..)` and `(1...)` give the same result for slicing,
|
|
67
|
+
`include?`, `===` and pattern matching.
|
|
68
|
+
- **Negation-removal operator** (Tier-2, opt-in via `--operators`):
|
|
69
|
+
`negation_removal` removes the `!` from `!x` and the `not` from `not x`.
|
|
70
|
+
The mutant survives when no test depends on the negated value. The
|
|
71
|
+
explicit form `x.!` is skipped, because `x.` does not parse.
|
|
72
|
+
|
|
73
|
+
### Changed
|
|
74
|
+
- **Stderr of tests and specs is visible** in the in-process and `--daemon`
|
|
75
|
+
runs. Mutineer silences stdout once per child process and no longer hides
|
|
76
|
+
stderr, so its own child diagnostics always reach you. `--test-command`
|
|
77
|
+
runs still capture stderr with stdout and show it under `--verbose`.
|
|
78
|
+
- **A mutant's test run stops at the first failing test** — one failure
|
|
79
|
+
already kills the mutant, so the forked child does not run the tests that
|
|
80
|
+
remain. Killed mutants cost less time, and survived mutants cost the same.
|
|
81
|
+
Under Minitest, when the outer reporter of the run records a failure or an
|
|
82
|
+
error (a skip does not count), each remaining test and each remaining test
|
|
83
|
+
class returns before it starts. A skipped class does not start its
|
|
84
|
+
class-level hooks. The run does not unwind: a class that is running
|
|
85
|
+
finishes normally, so its `after_all` hooks and a class-level
|
|
86
|
+
`transaction { super; raise ActiveRecord::Rollback }` still run. RSpec runs
|
|
87
|
+
with `--fail-fast`. This applies to the in-process backend only: coverage
|
|
88
|
+
capture and the clean checks still run every test, and the `--daemon` and
|
|
89
|
+
`--test-command` backends do not change. The CLI `--fail-fast` flag keeps
|
|
90
|
+
its meaning. On rack's `lib/rack/utils.rb` (`--jobs 1`), a full run takes
|
|
91
|
+
about 35–41 s instead of about 86–89 s. With the same coverage map, the
|
|
92
|
+
verdicts are the same.
|
|
93
|
+
- **The mutant run uses a fixed Minitest seed** — with the stop, the test
|
|
94
|
+
order can decide the verdict, so the child runs Minitest with seed `1`
|
|
95
|
+
unless the environment sets `SEED`. The same code then gives the same
|
|
96
|
+
verdict on each run. Coverage capture and the clean checks keep the random
|
|
97
|
+
seed, so the clean check runs the tests in a random order while each mutant
|
|
98
|
+
run uses the fixed order. RSpec keeps its configured order: a suite configured with
|
|
99
|
+
`config.order = :random` can still get a different verdict on each run for
|
|
100
|
+
the case below. In an order-dependent Minitest suite, the fixed seed makes
|
|
101
|
+
a false `killed` happen on every run or on no run, not on some runs.
|
|
102
|
+
- **A mutant whose failing test runs before a hanging test is now `killed`,
|
|
103
|
+
not `timeout`** — the run stops at the failure, before the hang. The tests
|
|
104
|
+
did detect the mutation, so `killed` is the correct verdict. If the hanging
|
|
105
|
+
test runs first in the fixed order, the verdict stays `timeout`. Compared
|
|
106
|
+
with a baseline from an earlier version, the score usually goes up. In an
|
|
107
|
+
order-dependent suite it can also go down: a mutant that a random order
|
|
108
|
+
killed on some runs can survive on every run in the fixed order. A change
|
|
109
|
+
to the tests can change the fixed order, so a later run can move such a
|
|
110
|
+
mutant from `killed` to `timeout`, and a `--baseline` gate then reports a
|
|
111
|
+
score drop.
|
|
112
|
+
- **Some runs still run most tests** — Minitest `parallelize_me!`, and Rails
|
|
113
|
+
`parallelize` above its threshold (by default more than 50 tests in the
|
|
114
|
+
child, or at any test count when `PARALLEL_WORKERS` is 2 or more in the
|
|
115
|
+
environment), queue their tests before the first result comes back, so the
|
|
116
|
+
queued tests still run. The verdict is the same as before. Below the
|
|
117
|
+
Rails threshold, the tests run one after the other in the child, and the
|
|
118
|
+
stop works.
|
|
119
|
+
|
|
120
|
+
### Fixed
|
|
121
|
+
- **Tests that reopen `$stdout`** (Minitest's `capture_subprocess_io`,
|
|
122
|
+
RSpec's `to_stdout_from_any_process`) no longer make a green suite
|
|
123
|
+
"not green" or count as false kills.
|
|
124
|
+
- **Test or source files that print while they load** no longer make coverage
|
|
125
|
+
capture fail with `invalid coverage output`. The capture subprocess now
|
|
126
|
+
sends its result over a separate pipe, not over stdout.
|
|
127
|
+
- **Chain-link operator** (Tier-2, opt-in via `--operators`):
|
|
128
|
+
`chain_link` drops one call from a chain, with its arguments and block
|
|
129
|
+
(`user.account.name` -> `user.name`). The mutant survives when no test tells
|
|
130
|
+
the chain apart from the same chain without that step. Conversions and copies
|
|
131
|
+
(`to_s`, `to_a`, `dup`, `freeze`, ...) and `new` are never dropped.
|
|
132
|
+
|
|
9
133
|
## [1.0.2] - 2026-09-21
|
|
10
134
|
|
|
11
135
|
### Added
|
|
@@ -422,6 +546,8 @@ Rails hardening + CI batch (issues #8–#13), all verified Rails-free.
|
|
|
422
546
|
- `.mutineer.yml` configuration (CLI > config > default precedence).
|
|
423
547
|
- Byte-correct source handling for multibyte (UTF-8) sources.
|
|
424
548
|
|
|
549
|
+
[1.2.0]: https://github.com/davidteren/mutineer/releases/tag/v1.2.0
|
|
550
|
+
[1.1.0]: https://github.com/davidteren/mutineer/releases/tag/v1.1.0
|
|
425
551
|
[1.0.2]: https://github.com/davidteren/mutineer/releases/tag/v1.0.2
|
|
426
552
|
[1.0.1]: https://github.com/davidteren/mutineer/releases/tag/v1.0.1
|
|
427
553
|
[1.0.0]: https://github.com/davidteren/mutineer/releases/tag/v1.0.0
|
data/README.md
CHANGED
|
@@ -13,6 +13,8 @@ testing anything.
|
|
|
13
13
|
- **One mutation per mutant**, validity-checked by re-parsing.
|
|
14
14
|
- **Fork-isolated**, parallel execution (Linux + macOS).
|
|
15
15
|
- **Coverage-guided** — each mutant runs only the test files that cover its line.
|
|
16
|
+
- **Stops at the first failing test** — in-process runs (not `--daemon` or
|
|
17
|
+
`--test-command`) stop a mutant's test run at the first failure.
|
|
16
18
|
|
|
17
19
|
📖 **[mutineer.github.io →](https://davidteren.github.io/mutineer/)** — overview, operators, and usage.
|
|
18
20
|
|
|
@@ -82,7 +84,8 @@ Run `mutineer --list-operators` to see them. Default (Tier 1): `arithmetic`,
|
|
|
82
84
|
`comparison`, `boolean_connector`, `boolean_literal`, `statement_removal`.
|
|
83
85
|
Available but off by default (Tier 2, enable via `--operators`): `return_nil`,
|
|
84
86
|
`literal_mutation`, `condition_negation`, `string_literal`, `regex`,
|
|
85
|
-
`collection_method
|
|
87
|
+
`collection_method`, `safe_navigation`, `range`, `negation_removal`, `chain_link`,
|
|
88
|
+
`operand_removal`, `array_literal`.
|
|
86
89
|
|
|
87
90
|
## Rails apps
|
|
88
91
|
|
|
@@ -210,7 +213,7 @@ Tradeoffs — this path is correct but not free:
|
|
|
210
213
|
Some mutants are equivalent (behaviour-identical) and survive forever — keeping a
|
|
211
214
|
file off 100%. Suppress them so the score and `--threshold` gate stay meaningful:
|
|
212
215
|
|
|
213
|
-
- **Inline:** `some_line # mutineer:disable-line` (or scope it: `# mutineer:disable-line comparison`).
|
|
216
|
+
- **Inline:** `some_line # mutineer:disable-line` (or scope it: `# mutineer:disable-line comparison`). Put a reason after `--`: `# mutineer:disable-line comparison -- the test checks only 20`.
|
|
214
217
|
- **Config:** a `.mutineer.yml` `ignore:` list of stable mutant ids. Each survivor's
|
|
215
218
|
`id` is printed in the JSON report, so copy it straight into `ignore:`.
|
|
216
219
|
|
|
@@ -0,0 +1,35 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Mutineer
|
|
4
|
+
# Silences stdout in a forked child that runs tests. Every fork boundary
|
|
5
|
+
# calls {.silence} once, right after `fork`, so the test runners themselves
|
|
6
|
+
# do not touch stdout. The parent reads each child's result from a separate
|
|
7
|
+
# pipe or from the exit status, never from stdout.
|
|
8
|
+
#
|
|
9
|
+
# Stderr stays open: it carries mutineer's own diagnostics from the child.
|
|
10
|
+
#
|
|
11
|
+
# Stdlib-only, so the app-side daemon can load it.
|
|
12
|
+
module ChildStdout
|
|
13
|
+
# Points fd 1 at File::NULL and makes `$stdout` the real STDOUT again.
|
|
14
|
+
#
|
|
15
|
+
# The reopen goes through STDOUT, not `$stdout`: the parent may have left
|
|
16
|
+
# `$stdout` as a StringIO, which cannot reopen. A test that calls
|
|
17
|
+
# `$stdout.reopen` (Minitest's `capture_subprocess_io`, RSpec's
|
|
18
|
+
# `to_stdout_from_any_process`) then gets a real IO. Child processes of the
|
|
19
|
+
# test inherit the silenced fd 1 too.
|
|
20
|
+
#
|
|
21
|
+
# The reopen takes an open IO, not a path. A path reopen checks the access
|
|
22
|
+
# mode of STDOUT and raises ArgumentError when a parent left STDOUT on a
|
|
23
|
+
# file in another mode (for example the "w+x" Tempfile of Minitest's
|
|
24
|
+
# `capture_subprocess_io`).
|
|
25
|
+
#
|
|
26
|
+
# Call it only in a child that exits after the tests run. Nothing restores
|
|
27
|
+
# the previous stdout.
|
|
28
|
+
#
|
|
29
|
+
# @return [void]
|
|
30
|
+
def self.silence
|
|
31
|
+
File.open(File::NULL, "w") { |null| STDOUT.reopen(null) }
|
|
32
|
+
$stdout = STDOUT
|
|
33
|
+
end
|
|
34
|
+
end
|
|
35
|
+
end
|
|
@@ -1,6 +1,5 @@
|
|
|
1
1
|
# frozen_string_literal: true
|
|
2
2
|
|
|
3
|
-
require "open3"
|
|
4
3
|
require "json"
|
|
5
4
|
require "digest"
|
|
6
5
|
require "fileutils"
|
|
@@ -9,6 +8,7 @@ require "coverage"
|
|
|
9
8
|
require "set"
|
|
10
9
|
require_relative "minitest_integration"
|
|
11
10
|
require_relative "test_runners"
|
|
11
|
+
require_relative "child_stdout"
|
|
12
12
|
|
|
13
13
|
module Mutineer
|
|
14
14
|
# Maps `(source_file, line) -> [test_files]` so each mutant runs only against
|
|
@@ -22,6 +22,10 @@ module Mutineer
|
|
|
22
22
|
# Seconds per coverage subprocess before the parent kills it.
|
|
23
23
|
DEFAULT_CAPTURE_TIMEOUT = 120
|
|
24
24
|
|
|
25
|
+
# File descriptor in a capture subprocess that carries the JSON result to
|
|
26
|
+
# the parent. Stdout stays free for test output, which goes to File::NULL.
|
|
27
|
+
RESULT_FD = 3
|
|
28
|
+
|
|
25
29
|
attr_reader :project_root, :failed_test_files, :failed_clean_tests, :phase_a_ran, :map
|
|
26
30
|
|
|
27
31
|
# Build a QUERY-ONLY map from data captured elsewhere (the daemon builds the
|
|
@@ -92,7 +96,7 @@ module Mutineer
|
|
|
92
96
|
# Is this source file's empty coverage the result of an *errored* capture
|
|
93
97
|
# rather than a genuine coverage gap? True iff some capture failed this run
|
|
94
98
|
# AND this file got zero coverage from any successful capture AND a failed
|
|
95
|
-
# test file maps to it by the
|
|
99
|
+
# test file maps to it by the _test/_spec/test_ naming convention. Derived
|
|
96
100
|
# purely from already-persisted state (@map keys + @failed_test_files); no
|
|
97
101
|
# rerun, no new cached field, no digest change.
|
|
98
102
|
#
|
|
@@ -138,10 +142,17 @@ module Mutineer
|
|
|
138
142
|
@map.keys.map { |k| k.rpartition(":").first }.to_set
|
|
139
143
|
end
|
|
140
144
|
|
|
141
|
-
# Basenames of failed test files with
|
|
142
|
-
#
|
|
145
|
+
# Basenames of the sources that failed test files pair with by convention:
|
|
146
|
+
# a trailing _test/_spec is stripped first, as pairing tries that form first.
|
|
143
147
|
def failed_test_targets
|
|
144
|
-
@failed_test_files.map
|
|
148
|
+
@failed_test_files.map do |t|
|
|
149
|
+
name = File.basename(t, ".rb")
|
|
150
|
+
case name
|
|
151
|
+
when /_(test|spec)\z/ then name.sub(/_(test|spec)\z/, "")
|
|
152
|
+
when "test_helper" then name # Minitest's support file pairs with no source
|
|
153
|
+
else name.delete_prefix("test_")
|
|
154
|
+
end
|
|
155
|
+
end.to_set
|
|
145
156
|
end
|
|
146
157
|
|
|
147
158
|
# Shared cache dance for both build paths: hit the digest-keyed cache, else
|
|
@@ -232,6 +243,7 @@ module Mutineer
|
|
|
232
243
|
rd.close
|
|
233
244
|
payload =
|
|
234
245
|
begin
|
|
246
|
+
ChildStdout.silence
|
|
235
247
|
# Fork-safety hook: the in-process path reconnects AR; the daemon
|
|
236
248
|
# routes to its worker DB. Nil (non-Rails) = no-op. Injected so this
|
|
237
249
|
# file needs neither Runner (Prism) nor Rails.
|
|
@@ -293,22 +305,8 @@ module Mutineer
|
|
|
293
305
|
# before any source is loaded. Returns the wrapped capture payload
|
|
294
306
|
# (`passed` + `coverage`), or nil when the subprocess failed (logged + skipped).
|
|
295
307
|
def capture(test_path)
|
|
296
|
-
out =
|
|
297
|
-
|
|
298
|
-
Open3.popen2(RbConfig.ruby, "-") do |stdin, stdout, wait_thr|
|
|
299
|
-
stdin.write(subprocess_script(test_path))
|
|
300
|
-
stdin.close
|
|
301
|
-
reader = Thread.new { out << stdout.read }
|
|
302
|
-
# Bound the subprocess with a wall clock: a hanging test file must not
|
|
303
|
-
# wedge the whole run before any per-mutant timeout.
|
|
304
|
-
unless wait_thr.join(@capture_timeout)
|
|
305
|
-
Process.kill(:KILL, wait_thr.pid) rescue nil # rubocop:disable Style/RescueModifier
|
|
306
|
-
reader.kill
|
|
307
|
-
return fail_test(test_path, "timed out after #{@capture_timeout}s")
|
|
308
|
-
end
|
|
309
|
-
reader.join
|
|
310
|
-
status = wait_thr.value
|
|
311
|
-
end
|
|
308
|
+
status, out = spawn_script(subprocess_script(test_path))
|
|
309
|
+
return fail_test(test_path, "timed out after #{@capture_timeout}s") unless status
|
|
312
310
|
return fail_test(test_path, "subprocess exited #{status.exitstatus}") unless status.success?
|
|
313
311
|
|
|
314
312
|
parsed = JSON.parse(out)
|
|
@@ -442,20 +440,72 @@ module Mutineer
|
|
|
442
440
|
# @param test_paths [Array<String>] test file paths.
|
|
443
441
|
# @return [Boolean]
|
|
444
442
|
def subprocess_clean_pass?(test_paths)
|
|
445
|
-
status =
|
|
446
|
-
|
|
447
|
-
|
|
448
|
-
|
|
449
|
-
|
|
450
|
-
|
|
451
|
-
|
|
452
|
-
|
|
453
|
-
|
|
454
|
-
|
|
455
|
-
|
|
456
|
-
|
|
443
|
+
status, = spawn_script(clean_check_script(test_paths), result: false)
|
|
444
|
+
status&.success? || false
|
|
445
|
+
end
|
|
446
|
+
|
|
447
|
+
# Runs `script` in a fresh `ruby -` that reads the script from stdin. The
|
|
448
|
+
# child's stdout goes to File::NULL, so test output never reaches the user
|
|
449
|
+
# or the result. With `result: true`, the child writes its result as one
|
|
450
|
+
# line to fd {RESULT_FD}, a pipe that only the script uses. The child's
|
|
451
|
+
# stderr is the parent's stderr, so warnings from the script reach the
|
|
452
|
+
# user. A wall clock of `@capture_timeout` bounds the whole call, so a hung
|
|
453
|
+
# test cannot wedge the run.
|
|
454
|
+
#
|
|
455
|
+
# The parent reads one line, not until EOF: a process that a test leaves
|
|
456
|
+
# running can inherit fd {RESULT_FD} (a `fork` without `exec` keeps it
|
|
457
|
+
# despite close-on-exec) and hold the pipe open long after the child exits.
|
|
458
|
+
# A clean check reports only through its exit status, so it gets no pipe.
|
|
459
|
+
#
|
|
460
|
+
# @api private
|
|
461
|
+
# @param script [String] Ruby script text.
|
|
462
|
+
# @param result [Boolean] whether to open the result pipe on fd {RESULT_FD}.
|
|
463
|
+
# @return [Array(Process::Status, String)] the exit status and the line the
|
|
464
|
+
# child wrote to fd {RESULT_FD} (`""` without one); `[nil, ""]` after a
|
|
465
|
+
# timeout.
|
|
466
|
+
def spawn_script(script, result: true)
|
|
467
|
+
deadline = Process.clock_gettime(Process::CLOCK_MONOTONIC) + @capture_timeout
|
|
468
|
+
script_rd, script_wr = IO.pipe
|
|
469
|
+
result_rd, result_wr = IO.pipe if result
|
|
470
|
+
options = { in: script_rd, out: File::NULL }
|
|
471
|
+
options[RESULT_FD] = result_wr if result
|
|
472
|
+
pid = Process.spawn(RbConfig.ruby, "-", **options)
|
|
473
|
+
waiter = Process.detach(pid)
|
|
474
|
+
script_rd.close
|
|
475
|
+
result_wr&.close
|
|
476
|
+
reader = Thread.new { result_rd.gets.to_s } if result
|
|
477
|
+
script_wr.write(script)
|
|
478
|
+
script_wr.close
|
|
479
|
+
unless waiter.join(remaining(deadline))
|
|
480
|
+
Process.kill(:KILL, pid) rescue nil # rubocop:disable Style/RescueModifier
|
|
481
|
+
waiter.join
|
|
482
|
+
reader&.kill
|
|
483
|
+
return [nil, ""]
|
|
457
484
|
end
|
|
458
|
-
|
|
485
|
+
[waiter.value, reader&.join(remaining(deadline))&.value.to_s]
|
|
486
|
+
ensure
|
|
487
|
+
reader&.kill
|
|
488
|
+
[script_rd, script_wr, result_rd, result_wr].compact.each { |io| io.close unless io.closed? }
|
|
489
|
+
end
|
|
490
|
+
|
|
491
|
+
# Seconds left before `deadline`, never negative.
|
|
492
|
+
#
|
|
493
|
+
# @api private
|
|
494
|
+
# @param deadline [Float] a CLOCK_MONOTONIC time.
|
|
495
|
+
# @return [Float]
|
|
496
|
+
def remaining(deadline)
|
|
497
|
+
[deadline - Process.clock_gettime(Process::CLOCK_MONOTONIC), 0].max
|
|
498
|
+
end
|
|
499
|
+
|
|
500
|
+
# Ruby source that opens the result channel in a {#spawn_script} child. The
|
|
501
|
+
# script runs it first, so no file that a test opens can take fd
|
|
502
|
+
# {RESULT_FD}. Close-on-exec keeps the fd out of the test's own
|
|
503
|
+
# subprocesses.
|
|
504
|
+
#
|
|
505
|
+
# @api private
|
|
506
|
+
# @return [String] Ruby script text.
|
|
507
|
+
def result_channel_expression
|
|
508
|
+
"_result = IO.new(#{RESULT_FD}, \"w\"); _result.close_on_exec = true"
|
|
459
509
|
end
|
|
460
510
|
|
|
461
511
|
# Runs test files in a fork of the booted parent and returns whether they passed.
|
|
@@ -473,6 +523,7 @@ module Mutineer
|
|
|
473
523
|
rd.close
|
|
474
524
|
Process.setpgid(0, 0) rescue nil # rubocop:disable Style/RescueModifier
|
|
475
525
|
begin
|
|
526
|
+
ChildStdout.silence
|
|
476
527
|
after_fork&.call
|
|
477
528
|
Coverage.result(clear: true, stop: false) if Coverage.running?
|
|
478
529
|
wr.write(Marshal.dump(TestRunners.for(@framework).run(abs_tests).zero?))
|
|
@@ -535,11 +586,15 @@ module Mutineer
|
|
|
535
586
|
require "minitest"
|
|
536
587
|
require "stringio"
|
|
537
588
|
def Minitest.autorun; end
|
|
589
|
+
_report = StringIO.new
|
|
590
|
+
Minitest.define_singleton_method(:plugin_mutineer_report_init) { |options| reporter << Minitest::SummaryReporter.new(_report, options) }
|
|
591
|
+
Minitest.extensions << "mutineer_report"
|
|
538
592
|
$LOAD_PATH.unshift(*#{abs_load_paths.inspect})
|
|
539
|
-
#{abs_source_paths.inspect}.each { |f|
|
|
593
|
+
#{abs_source_paths.inspect}.each { |f| require f }
|
|
540
594
|
#{loads}
|
|
541
|
-
|
|
542
|
-
|
|
595
|
+
_passed = Minitest.run([])
|
|
596
|
+
$stderr.write(_report.string) unless _passed
|
|
597
|
+
exit(_passed ? 0 : 1)
|
|
543
598
|
RUBY
|
|
544
599
|
end
|
|
545
600
|
|
|
@@ -559,10 +614,10 @@ module Mutineer
|
|
|
559
614
|
end
|
|
560
615
|
RSpec::Core::Runner.disable_autorun!
|
|
561
616
|
$LOAD_PATH.unshift(*#{abs_load_paths.inspect})
|
|
562
|
-
#{abs_source_paths.inspect}.each { |f|
|
|
617
|
+
#{abs_source_paths.inspect}.each { |f| require f }
|
|
563
618
|
_sink = StringIO.new
|
|
564
|
-
$stdout = _sink
|
|
565
619
|
status = RSpec::Core::Runner.run(["--no-color", #{specs}], _sink, _sink)
|
|
620
|
+
$stderr.write(_sink.string) unless status.zero?
|
|
566
621
|
exit(status.zero? ? 0 : 1)
|
|
567
622
|
RUBY
|
|
568
623
|
end
|
|
@@ -583,31 +638,36 @@ module Mutineer
|
|
|
583
638
|
# @return [String] Ruby script text.
|
|
584
639
|
def minitest_subprocess_script(test_path)
|
|
585
640
|
<<~RUBY
|
|
641
|
+
#{result_channel_expression}
|
|
586
642
|
require "coverage"
|
|
587
643
|
require "json"
|
|
588
|
-
require "stringio"
|
|
589
644
|
require "minitest"
|
|
645
|
+
require "stringio"
|
|
590
646
|
def Minitest.autorun; end
|
|
647
|
+
_report = StringIO.new
|
|
648
|
+
Minitest.define_singleton_method(:plugin_mutineer_report_init) { |options| reporter << Minitest::SummaryReporter.new(_report, options) }
|
|
649
|
+
Minitest.extensions << "mutineer_report"
|
|
591
650
|
Coverage.start(lines: true)
|
|
592
651
|
$LOAD_PATH.unshift(*#{abs_load_paths.inspect})
|
|
593
|
-
#{abs_source_paths.inspect}.each { |f|
|
|
652
|
+
#{abs_source_paths.inspect}.each { |f| require f }
|
|
594
653
|
load #{absolute(test_path).inspect}
|
|
595
|
-
_orig = $stdout
|
|
596
|
-
$stdout = StringIO.new
|
|
597
654
|
_passed = Minitest.run([])
|
|
598
|
-
$
|
|
599
|
-
puts JSON.generate("passed" => _passed == true, "coverage" => Coverage.result,
|
|
600
|
-
|
|
655
|
+
$stderr.write(_report.string) unless _passed
|
|
656
|
+
_result.puts JSON.generate("passed" => _passed == true, "coverage" => Coverage.result,
|
|
657
|
+
"loaded_files" => #{loaded_files_expression})
|
|
658
|
+
_result.close
|
|
601
659
|
RUBY
|
|
602
660
|
end
|
|
603
661
|
|
|
604
662
|
# Same coverage-JSON contract as the minitest path, but driven by RSpec:
|
|
605
|
-
# require rspec/core lazily,
|
|
606
|
-
# one spec via RSpec::Core::Runner
|
|
607
|
-
#
|
|
608
|
-
# non-zero -> capture() records a skipped (incomplete-map)
|
|
663
|
+
# require rspec/core lazily, require the sources under Coverage, then run the
|
|
664
|
+
# one spec via RSpec::Core::Runner. The JSON goes to the result channel (see
|
|
665
|
+
# {#spawn_script}), so spec output cannot corrupt it. A missing rspec makes
|
|
666
|
+
# the script exit non-zero -> capture() records a skipped (incomplete-map)
|
|
667
|
+
# test, with a hint.
|
|
609
668
|
def rspec_subprocess_script(test_path)
|
|
610
669
|
<<~RUBY
|
|
670
|
+
#{result_channel_expression}
|
|
611
671
|
require "coverage"
|
|
612
672
|
require "json"
|
|
613
673
|
require "stringio"
|
|
@@ -620,14 +680,13 @@ module Mutineer
|
|
|
620
680
|
RSpec::Core::Runner.disable_autorun!
|
|
621
681
|
Coverage.start(lines: true)
|
|
622
682
|
$LOAD_PATH.unshift(*#{abs_load_paths.inspect})
|
|
623
|
-
#{abs_source_paths.inspect}.each { |f|
|
|
624
|
-
_orig = $stdout
|
|
683
|
+
#{abs_source_paths.inspect}.each { |f| require f }
|
|
625
684
|
_sink = StringIO.new
|
|
626
|
-
$stdout = _sink
|
|
627
685
|
_status = RSpec::Core::Runner.run(["--no-color", #{absolute(test_path).inspect}], _sink, _sink)
|
|
628
|
-
$
|
|
629
|
-
puts JSON.generate("passed" => _status.zero?, "coverage" => Coverage.result,
|
|
630
|
-
|
|
686
|
+
$stderr.write(_sink.string) unless _status.zero?
|
|
687
|
+
_result.puts JSON.generate("passed" => _status.zero?, "coverage" => Coverage.result,
|
|
688
|
+
"loaded_files" => #{loaded_files_expression})
|
|
689
|
+
_result.close
|
|
631
690
|
RUBY
|
|
632
691
|
end
|
|
633
692
|
|
|
@@ -2,6 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
require "json"
|
|
4
4
|
require "tempfile"
|
|
5
|
+
require_relative "child_stdout"
|
|
5
6
|
|
|
6
7
|
module Mutineer
|
|
7
8
|
# App-side daemon (persistent worker).
|
|
@@ -177,9 +178,9 @@ module Mutineer
|
|
|
177
178
|
# New process group so a per-fork timeout can SIGKILL the whole subtree,
|
|
178
179
|
# and silence the child's stdout so test output never corrupts the IPC pipe.
|
|
179
180
|
Process.setpgid(0, 0) rescue nil # rubocop:disable Style/RescueModifier
|
|
180
|
-
$stdout.reopen(File::NULL, "w")
|
|
181
181
|
code =
|
|
182
182
|
begin
|
|
183
|
+
ChildStdout.silence
|
|
183
184
|
# Route THIS fork at its own worker database before any test loads.
|
|
184
185
|
# A routing failure raises here and is scored `error`, never a false verdict.
|
|
185
186
|
@worker_db&.after_fork(worker, schema_for_fork)
|
data/lib/mutineer/isolation.rb
CHANGED
|
@@ -3,6 +3,7 @@
|
|
|
3
3
|
require "tempfile"
|
|
4
4
|
require_relative "result"
|
|
5
5
|
require_relative "parser"
|
|
6
|
+
require_relative "child_stdout"
|
|
6
7
|
|
|
7
8
|
module Mutineer
|
|
8
9
|
# Fork-based isolation for running one mutant. The block runs in a child
|
|
@@ -26,6 +27,9 @@ module Mutineer
|
|
|
26
27
|
# exit code) or any explicit `exit` is honoured; an unhandled exception
|
|
27
28
|
# becomes exit 2 with the cause written to STDERR.
|
|
28
29
|
#
|
|
30
|
+
# The child silences its stdout (see {ChildStdout.silence}) before the
|
|
31
|
+
# block runs, so test output never reaches the user. Stderr stays open.
|
|
32
|
+
#
|
|
29
33
|
# @param timeout [Integer] timeout in seconds.
|
|
30
34
|
# @yieldreturn [Integer] child exit status.
|
|
31
35
|
# @return [Mutineer::Result] result from the child process.
|
|
@@ -36,15 +40,17 @@ module Mutineer
|
|
|
36
40
|
Process.setpgid(0, 0) rescue nil # rubocop:disable Style/RescueModifier
|
|
37
41
|
code = 0
|
|
38
42
|
begin
|
|
43
|
+
ChildStdout.silence
|
|
39
44
|
result = yield
|
|
40
45
|
code = result.is_a?(Integer) ? result : 0
|
|
41
46
|
rescue SystemExit => e
|
|
42
47
|
code = e.status
|
|
43
48
|
rescue Exception => e # rubocop:disable Lint/RescueException
|
|
44
|
-
warn
|
|
49
|
+
# STDERR, not `warn`: a test may have left `$stderr` as a StringIO.
|
|
50
|
+
STDERR.puts "[mutineer-child] #{e.class}: #{e.message}"
|
|
45
51
|
code = 2
|
|
46
52
|
end
|
|
47
|
-
|
|
53
|
+
STDERR.flush
|
|
48
54
|
# exit! skips at_exit handlers — critical, since a child forked from
|
|
49
55
|
# inside our own Minitest suite would otherwise re-run the parent's
|
|
50
56
|
# at_exit autorun hook on the way out.
|
|
@@ -93,7 +99,7 @@ module Mutineer
|
|
|
93
99
|
# @param source_file [String] original source file path.
|
|
94
100
|
# @return [Object] whatever `load` returns.
|
|
95
101
|
def self.apply_whole_file(mutated, source_file)
|
|
96
|
-
Tempfile.create(["mutineer_mutant", ".rb"], File.dirname(source_file)) do |f|
|
|
102
|
+
Tempfile.create(["mutineer_mutant", ".rb"], File.dirname(File.expand_path(source_file))) do |f|
|
|
97
103
|
f.write(mutated)
|
|
98
104
|
f.flush
|
|
99
105
|
load f.path
|