mutineer 1.0.2 → 1.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +77 -0
- data/README.md +3 -1
- data/lib/mutineer/child_stdout.rb +35 -0
- data/lib/mutineer/coverage_map.rb +86 -47
- data/lib/mutineer/daemon_server.rb +2 -1
- data/lib/mutineer/isolation.rb +8 -2
- data/lib/mutineer/minitest_integration/stop_at_first_failure.rb +199 -0
- data/lib/mutineer/minitest_integration.rb +22 -8
- data/lib/mutineer/mutator_registry.rb +18 -4
- data/lib/mutineer/mutators/chain_link.rb +139 -0
- data/lib/mutineer/mutators/negation_removal.rb +44 -0
- data/lib/mutineer/mutators/range_literal.rb +54 -0
- data/lib/mutineer/mutators/safe_navigation.rb +76 -0
- data/lib/mutineer/runner.rb +2 -1
- data/lib/mutineer/test_runners/minitest.rb +5 -1
- data/lib/mutineer/test_runners/rspec.rb +9 -11
- data/lib/mutineer/test_runners.rb +4 -2
- data/lib/mutineer/version.rb +1 -1
- data/lib/mutineer.rb +4 -0
- metadata +7 -15
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 1ca70f2affdf60ddda53d3d38e23ddff4ba6cae27cd789501c7fe3060975f146
|
|
4
|
+
data.tar.gz: 94a9968fe182655e6d566e9edfc43f1f2e15b050c7690f8905e7be9f7d710b61
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 8da96d70db7af241cda4ca44ecbe84522b0c0b68427a46c05e3178723c2eada7d957677c8b2cc7d2853d9593cba546c194f342862058dffdba55d03e8ef27021
|
|
7
|
+
data.tar.gz: 50481e9ca34d07530878f6ff0f25f192ab63c3200376a1a1ea5665341955a00c6bf2f9e7a4429d6cd8fd8d2119c4316c4c993741f219897f57ab493ca6c428b1
|
data/CHANGELOG.md
CHANGED
|
@@ -6,6 +6,82 @@ All notable changes to this project are documented here. The format is based on
|
|
|
6
6
|
|
|
7
7
|
## [Unreleased]
|
|
8
8
|
|
|
9
|
+
## [1.1.0] - 2026-09-28
|
|
10
|
+
|
|
11
|
+
### Added
|
|
12
|
+
- **Safe-navigation operator** (Tier-2, opt-in via `--operators`):
|
|
13
|
+
`safe_navigation` replaces `&.` with `.`. The mutant survives when no test
|
|
14
|
+
passes `nil` to the call.
|
|
15
|
+
- **Range operator** (Tier-2, opt-in via `--operators`): `range` replaces
|
|
16
|
+
`..` with `...` and `...` with `..`. The `..` -> `...` mutant survives when
|
|
17
|
+
no test checks the last element of the range. Endless ranges (`1..`) are
|
|
18
|
+
skipped, because `(1..)` and `(1...)` give the same result for slicing,
|
|
19
|
+
`include?`, `===` and pattern matching.
|
|
20
|
+
- **Negation-removal operator** (Tier-2, opt-in via `--operators`):
|
|
21
|
+
`negation_removal` removes the `!` from `!x` and the `not` from `not x`.
|
|
22
|
+
The mutant survives when no test depends on the negated value. The
|
|
23
|
+
explicit form `x.!` is skipped, because `x.` does not parse.
|
|
24
|
+
|
|
25
|
+
### Changed
|
|
26
|
+
- **Stderr of tests and specs is visible** in the in-process and `--daemon`
|
|
27
|
+
runs. Mutineer silences stdout once per child process and no longer hides
|
|
28
|
+
stderr, so its own child diagnostics always reach you. `--test-command`
|
|
29
|
+
runs still capture stderr with stdout and show it under `--verbose`.
|
|
30
|
+
- **A mutant's test run stops at the first failing test** — one failure
|
|
31
|
+
already kills the mutant, so the forked child does not run the tests that
|
|
32
|
+
remain. Killed mutants cost less time, and survived mutants cost the same.
|
|
33
|
+
Under Minitest, when the outer reporter of the run records a failure or an
|
|
34
|
+
error (a skip does not count), each remaining test and each remaining test
|
|
35
|
+
class returns before it starts. A skipped class does not start its
|
|
36
|
+
class-level hooks. The run does not unwind: a class that is running
|
|
37
|
+
finishes normally, so its `after_all` hooks and a class-level
|
|
38
|
+
`transaction { super; raise ActiveRecord::Rollback }` still run. RSpec runs
|
|
39
|
+
with `--fail-fast`. This applies to the in-process backend only: coverage
|
|
40
|
+
capture and the clean checks still run every test, and the `--daemon` and
|
|
41
|
+
`--test-command` backends do not change. The CLI `--fail-fast` flag keeps
|
|
42
|
+
its meaning. On rack's `lib/rack/utils.rb` (`--jobs 1`), a full run takes
|
|
43
|
+
about 35–41 s instead of about 86–89 s. With the same coverage map, the
|
|
44
|
+
verdicts are the same.
|
|
45
|
+
- **The mutant run uses a fixed Minitest seed** — with the stop, the test
|
|
46
|
+
order can decide the verdict, so the child runs Minitest with seed `1`
|
|
47
|
+
unless the environment sets `SEED`. The same code then gives the same
|
|
48
|
+
verdict on each run. Coverage capture and the clean checks keep the random
|
|
49
|
+
seed, so the clean check runs the tests in a random order while each mutant
|
|
50
|
+
run uses the fixed order. RSpec keeps its configured order: a suite configured with
|
|
51
|
+
`config.order = :random` can still get a different verdict on each run for
|
|
52
|
+
the case below. In an order-dependent Minitest suite, the fixed seed makes
|
|
53
|
+
a false `killed` happen on every run or on no run, not on some runs.
|
|
54
|
+
- **A mutant whose failing test runs before a hanging test is now `killed`,
|
|
55
|
+
not `timeout`** — the run stops at the failure, before the hang. The tests
|
|
56
|
+
did detect the mutation, so `killed` is the correct verdict. If the hanging
|
|
57
|
+
test runs first in the fixed order, the verdict stays `timeout`. Compared
|
|
58
|
+
with a baseline from an earlier version, the score usually goes up. In an
|
|
59
|
+
order-dependent suite it can also go down: a mutant that a random order
|
|
60
|
+
killed on some runs can survive on every run in the fixed order. A change
|
|
61
|
+
to the tests can change the fixed order, so a later run can move such a
|
|
62
|
+
mutant from `killed` to `timeout`, and a `--baseline` gate then reports a
|
|
63
|
+
score drop.
|
|
64
|
+
- **Some runs still run most tests** — Minitest `parallelize_me!`, and Rails
|
|
65
|
+
`parallelize` above its threshold (by default more than 50 tests in the
|
|
66
|
+
child, or at any test count when `PARALLEL_WORKERS` is 2 or more in the
|
|
67
|
+
environment), queue their tests before the first result comes back, so the
|
|
68
|
+
queued tests still run. The verdict is the same as before. Below the
|
|
69
|
+
Rails threshold, the tests run one after the other in the child, and the
|
|
70
|
+
stop works.
|
|
71
|
+
|
|
72
|
+
### Fixed
|
|
73
|
+
- **Tests that reopen `$stdout`** (Minitest's `capture_subprocess_io`,
|
|
74
|
+
RSpec's `to_stdout_from_any_process`) no longer make a green suite
|
|
75
|
+
"not green" or count as false kills.
|
|
76
|
+
- **Test or source files that print while they load** no longer make coverage
|
|
77
|
+
capture fail with `invalid coverage output`. The capture subprocess now
|
|
78
|
+
sends its result over a separate pipe, not over stdout.
|
|
79
|
+
- **Chain-link operator** (Tier-2, opt-in via `--operators`):
|
|
80
|
+
`chain_link` drops one call from a chain, with its arguments and block
|
|
81
|
+
(`user.account.name` -> `user.name`). The mutant survives when no test tells
|
|
82
|
+
the chain apart from the same chain without that step. Conversions and copies
|
|
83
|
+
(`to_s`, `to_a`, `dup`, `freeze`, ...) and `new` are never dropped.
|
|
84
|
+
|
|
9
85
|
## [1.0.2] - 2026-09-21
|
|
10
86
|
|
|
11
87
|
### Added
|
|
@@ -422,6 +498,7 @@ Rails hardening + CI batch (issues #8–#13), all verified Rails-free.
|
|
|
422
498
|
- `.mutineer.yml` configuration (CLI > config > default precedence).
|
|
423
499
|
- Byte-correct source handling for multibyte (UTF-8) sources.
|
|
424
500
|
|
|
501
|
+
[1.1.0]: https://github.com/davidteren/mutineer/releases/tag/v1.1.0
|
|
425
502
|
[1.0.2]: https://github.com/davidteren/mutineer/releases/tag/v1.0.2
|
|
426
503
|
[1.0.1]: https://github.com/davidteren/mutineer/releases/tag/v1.0.1
|
|
427
504
|
[1.0.0]: https://github.com/davidteren/mutineer/releases/tag/v1.0.0
|
data/README.md
CHANGED
|
@@ -13,6 +13,8 @@ testing anything.
|
|
|
13
13
|
- **One mutation per mutant**, validity-checked by re-parsing.
|
|
14
14
|
- **Fork-isolated**, parallel execution (Linux + macOS).
|
|
15
15
|
- **Coverage-guided** — each mutant runs only the test files that cover its line.
|
|
16
|
+
- **Stops at the first failing test** — in-process runs (not `--daemon` or
|
|
17
|
+
`--test-command`) stop a mutant's test run at the first failure.
|
|
16
18
|
|
|
17
19
|
📖 **[mutineer.github.io →](https://davidteren.github.io/mutineer/)** — overview, operators, and usage.
|
|
18
20
|
|
|
@@ -82,7 +84,7 @@ Run `mutineer --list-operators` to see them. Default (Tier 1): `arithmetic`,
|
|
|
82
84
|
`comparison`, `boolean_connector`, `boolean_literal`, `statement_removal`.
|
|
83
85
|
Available but off by default (Tier 2, enable via `--operators`): `return_nil`,
|
|
84
86
|
`literal_mutation`, `condition_negation`, `string_literal`, `regex`,
|
|
85
|
-
`collection_method`.
|
|
87
|
+
`collection_method`, `safe_navigation`, `range`, `negation_removal`, `chain_link`.
|
|
86
88
|
|
|
87
89
|
## Rails apps
|
|
88
90
|
|
|
@@ -0,0 +1,35 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Mutineer
|
|
4
|
+
# Silences stdout in a forked child that runs tests. Every fork boundary
|
|
5
|
+
# calls {.silence} once, right after `fork`, so the test runners themselves
|
|
6
|
+
# do not touch stdout. The parent reads each child's result from a separate
|
|
7
|
+
# pipe or from the exit status, never from stdout.
|
|
8
|
+
#
|
|
9
|
+
# Stderr stays open: it carries mutineer's own diagnostics from the child.
|
|
10
|
+
#
|
|
11
|
+
# Stdlib-only, so the app-side daemon can load it.
|
|
12
|
+
module ChildStdout
|
|
13
|
+
# Points fd 1 at File::NULL and makes `$stdout` the real STDOUT again.
|
|
14
|
+
#
|
|
15
|
+
# The reopen goes through STDOUT, not `$stdout`: the parent may have left
|
|
16
|
+
# `$stdout` as a StringIO, which cannot reopen. A test that calls
|
|
17
|
+
# `$stdout.reopen` (Minitest's `capture_subprocess_io`, RSpec's
|
|
18
|
+
# `to_stdout_from_any_process`) then gets a real IO. Child processes of the
|
|
19
|
+
# test inherit the silenced fd 1 too.
|
|
20
|
+
#
|
|
21
|
+
# The reopen takes an open IO, not a path. A path reopen checks the access
|
|
22
|
+
# mode of STDOUT and raises ArgumentError when a parent left STDOUT on a
|
|
23
|
+
# file in another mode (for example the "w+x" Tempfile of Minitest's
|
|
24
|
+
# `capture_subprocess_io`).
|
|
25
|
+
#
|
|
26
|
+
# Call it only in a child that exits after the tests run. Nothing restores
|
|
27
|
+
# the previous stdout.
|
|
28
|
+
#
|
|
29
|
+
# @return [void]
|
|
30
|
+
def self.silence
|
|
31
|
+
File.open(File::NULL, "w") { |null| STDOUT.reopen(null) }
|
|
32
|
+
$stdout = STDOUT
|
|
33
|
+
end
|
|
34
|
+
end
|
|
35
|
+
end
|
|
@@ -1,6 +1,5 @@
|
|
|
1
1
|
# frozen_string_literal: true
|
|
2
2
|
|
|
3
|
-
require "open3"
|
|
4
3
|
require "json"
|
|
5
4
|
require "digest"
|
|
6
5
|
require "fileutils"
|
|
@@ -9,6 +8,7 @@ require "coverage"
|
|
|
9
8
|
require "set"
|
|
10
9
|
require_relative "minitest_integration"
|
|
11
10
|
require_relative "test_runners"
|
|
11
|
+
require_relative "child_stdout"
|
|
12
12
|
|
|
13
13
|
module Mutineer
|
|
14
14
|
# Maps `(source_file, line) -> [test_files]` so each mutant runs only against
|
|
@@ -22,6 +22,10 @@ module Mutineer
|
|
|
22
22
|
# Seconds per coverage subprocess before the parent kills it.
|
|
23
23
|
DEFAULT_CAPTURE_TIMEOUT = 120
|
|
24
24
|
|
|
25
|
+
# File descriptor in a capture subprocess that carries the JSON result to
|
|
26
|
+
# the parent. Stdout stays free for test output, which goes to File::NULL.
|
|
27
|
+
RESULT_FD = 3
|
|
28
|
+
|
|
25
29
|
attr_reader :project_root, :failed_test_files, :failed_clean_tests, :phase_a_ran, :map
|
|
26
30
|
|
|
27
31
|
# Build a QUERY-ONLY map from data captured elsewhere (the daemon builds the
|
|
@@ -232,6 +236,7 @@ module Mutineer
|
|
|
232
236
|
rd.close
|
|
233
237
|
payload =
|
|
234
238
|
begin
|
|
239
|
+
ChildStdout.silence
|
|
235
240
|
# Fork-safety hook: the in-process path reconnects AR; the daemon
|
|
236
241
|
# routes to its worker DB. Nil (non-Rails) = no-op. Injected so this
|
|
237
242
|
# file needs neither Runner (Prism) nor Rails.
|
|
@@ -293,22 +298,8 @@ module Mutineer
|
|
|
293
298
|
# before any source is loaded. Returns the wrapped capture payload
|
|
294
299
|
# (`passed` + `coverage`), or nil when the subprocess failed (logged + skipped).
|
|
295
300
|
def capture(test_path)
|
|
296
|
-
out =
|
|
297
|
-
|
|
298
|
-
Open3.popen2(RbConfig.ruby, "-") do |stdin, stdout, wait_thr|
|
|
299
|
-
stdin.write(subprocess_script(test_path))
|
|
300
|
-
stdin.close
|
|
301
|
-
reader = Thread.new { out << stdout.read }
|
|
302
|
-
# Bound the subprocess with a wall clock: a hanging test file must not
|
|
303
|
-
# wedge the whole run before any per-mutant timeout.
|
|
304
|
-
unless wait_thr.join(@capture_timeout)
|
|
305
|
-
Process.kill(:KILL, wait_thr.pid) rescue nil # rubocop:disable Style/RescueModifier
|
|
306
|
-
reader.kill
|
|
307
|
-
return fail_test(test_path, "timed out after #{@capture_timeout}s")
|
|
308
|
-
end
|
|
309
|
-
reader.join
|
|
310
|
-
status = wait_thr.value
|
|
311
|
-
end
|
|
301
|
+
status, out = spawn_script(subprocess_script(test_path))
|
|
302
|
+
return fail_test(test_path, "timed out after #{@capture_timeout}s") unless status
|
|
312
303
|
return fail_test(test_path, "subprocess exited #{status.exitstatus}") unless status.success?
|
|
313
304
|
|
|
314
305
|
parsed = JSON.parse(out)
|
|
@@ -442,20 +433,72 @@ module Mutineer
|
|
|
442
433
|
# @param test_paths [Array<String>] test file paths.
|
|
443
434
|
# @return [Boolean]
|
|
444
435
|
def subprocess_clean_pass?(test_paths)
|
|
445
|
-
status =
|
|
446
|
-
|
|
447
|
-
|
|
448
|
-
|
|
449
|
-
|
|
450
|
-
|
|
451
|
-
|
|
452
|
-
|
|
453
|
-
|
|
454
|
-
|
|
455
|
-
|
|
456
|
-
|
|
436
|
+
status, = spawn_script(clean_check_script(test_paths), result: false)
|
|
437
|
+
status&.success? || false
|
|
438
|
+
end
|
|
439
|
+
|
|
440
|
+
# Runs `script` in a fresh `ruby -` that reads the script from stdin. The
|
|
441
|
+
# child's stdout goes to File::NULL, so test output never reaches the user
|
|
442
|
+
# or the result. With `result: true`, the child writes its result as one
|
|
443
|
+
# line to fd {RESULT_FD}, a pipe that only the script uses. The child's
|
|
444
|
+
# stderr is the parent's stderr, so warnings from the script reach the
|
|
445
|
+
# user. A wall clock of `@capture_timeout` bounds the whole call, so a hung
|
|
446
|
+
# test cannot wedge the run.
|
|
447
|
+
#
|
|
448
|
+
# The parent reads one line, not until EOF: a process that a test leaves
|
|
449
|
+
# running can inherit fd {RESULT_FD} (a `fork` without `exec` keeps it
|
|
450
|
+
# despite close-on-exec) and hold the pipe open long after the child exits.
|
|
451
|
+
# A clean check reports only through its exit status, so it gets no pipe.
|
|
452
|
+
#
|
|
453
|
+
# @api private
|
|
454
|
+
# @param script [String] Ruby script text.
|
|
455
|
+
# @param result [Boolean] whether to open the result pipe on fd {RESULT_FD}.
|
|
456
|
+
# @return [Array(Process::Status, String)] the exit status and the line the
|
|
457
|
+
# child wrote to fd {RESULT_FD} (`""` without one); `[nil, ""]` after a
|
|
458
|
+
# timeout.
|
|
459
|
+
def spawn_script(script, result: true)
|
|
460
|
+
deadline = Process.clock_gettime(Process::CLOCK_MONOTONIC) + @capture_timeout
|
|
461
|
+
script_rd, script_wr = IO.pipe
|
|
462
|
+
result_rd, result_wr = IO.pipe if result
|
|
463
|
+
options = { in: script_rd, out: File::NULL }
|
|
464
|
+
options[RESULT_FD] = result_wr if result
|
|
465
|
+
pid = Process.spawn(RbConfig.ruby, "-", **options)
|
|
466
|
+
waiter = Process.detach(pid)
|
|
467
|
+
script_rd.close
|
|
468
|
+
result_wr&.close
|
|
469
|
+
reader = Thread.new { result_rd.gets.to_s } if result
|
|
470
|
+
script_wr.write(script)
|
|
471
|
+
script_wr.close
|
|
472
|
+
unless waiter.join(remaining(deadline))
|
|
473
|
+
Process.kill(:KILL, pid) rescue nil # rubocop:disable Style/RescueModifier
|
|
474
|
+
waiter.join
|
|
475
|
+
reader&.kill
|
|
476
|
+
return [nil, ""]
|
|
457
477
|
end
|
|
458
|
-
|
|
478
|
+
[waiter.value, reader&.join(remaining(deadline))&.value.to_s]
|
|
479
|
+
ensure
|
|
480
|
+
reader&.kill
|
|
481
|
+
[script_rd, script_wr, result_rd, result_wr].compact.each { |io| io.close unless io.closed? }
|
|
482
|
+
end
|
|
483
|
+
|
|
484
|
+
# Seconds left before `deadline`, never negative.
|
|
485
|
+
#
|
|
486
|
+
# @api private
|
|
487
|
+
# @param deadline [Float] a CLOCK_MONOTONIC time.
|
|
488
|
+
# @return [Float]
|
|
489
|
+
def remaining(deadline)
|
|
490
|
+
[deadline - Process.clock_gettime(Process::CLOCK_MONOTONIC), 0].max
|
|
491
|
+
end
|
|
492
|
+
|
|
493
|
+
# Ruby source that opens the result channel in a {#spawn_script} child. The
|
|
494
|
+
# script runs it first, so no file that a test opens can take fd
|
|
495
|
+
# {RESULT_FD}. Close-on-exec keeps the fd out of the test's own
|
|
496
|
+
# subprocesses.
|
|
497
|
+
#
|
|
498
|
+
# @api private
|
|
499
|
+
# @return [String] Ruby script text.
|
|
500
|
+
def result_channel_expression
|
|
501
|
+
"_result = IO.new(#{RESULT_FD}, \"w\"); _result.close_on_exec = true"
|
|
459
502
|
end
|
|
460
503
|
|
|
461
504
|
# Runs test files in a fork of the booted parent and returns whether they passed.
|
|
@@ -473,6 +516,7 @@ module Mutineer
|
|
|
473
516
|
rd.close
|
|
474
517
|
Process.setpgid(0, 0) rescue nil # rubocop:disable Style/RescueModifier
|
|
475
518
|
begin
|
|
519
|
+
ChildStdout.silence
|
|
476
520
|
after_fork&.call
|
|
477
521
|
Coverage.result(clear: true, stop: false) if Coverage.running?
|
|
478
522
|
wr.write(Marshal.dump(TestRunners.for(@framework).run(abs_tests).zero?))
|
|
@@ -533,12 +577,10 @@ module Mutineer
|
|
|
533
577
|
loads = Array(test_paths).map { |t| "load #{absolute(t).inspect}" }.join("\n")
|
|
534
578
|
<<~RUBY
|
|
535
579
|
require "minitest"
|
|
536
|
-
require "stringio"
|
|
537
580
|
def Minitest.autorun; end
|
|
538
581
|
$LOAD_PATH.unshift(*#{abs_load_paths.inspect})
|
|
539
582
|
#{abs_source_paths.inspect}.each { |f| load f }
|
|
540
583
|
#{loads}
|
|
541
|
-
$stdout = StringIO.new
|
|
542
584
|
exit(Minitest.run([]) ? 0 : 1)
|
|
543
585
|
RUBY
|
|
544
586
|
end
|
|
@@ -561,7 +603,6 @@ module Mutineer
|
|
|
561
603
|
$LOAD_PATH.unshift(*#{abs_load_paths.inspect})
|
|
562
604
|
#{abs_source_paths.inspect}.each { |f| load f }
|
|
563
605
|
_sink = StringIO.new
|
|
564
|
-
$stdout = _sink
|
|
565
606
|
status = RSpec::Core::Runner.run(["--no-color", #{specs}], _sink, _sink)
|
|
566
607
|
exit(status.zero? ? 0 : 1)
|
|
567
608
|
RUBY
|
|
@@ -583,31 +624,31 @@ module Mutineer
|
|
|
583
624
|
# @return [String] Ruby script text.
|
|
584
625
|
def minitest_subprocess_script(test_path)
|
|
585
626
|
<<~RUBY
|
|
627
|
+
#{result_channel_expression}
|
|
586
628
|
require "coverage"
|
|
587
629
|
require "json"
|
|
588
|
-
require "stringio"
|
|
589
630
|
require "minitest"
|
|
590
631
|
def Minitest.autorun; end
|
|
591
632
|
Coverage.start(lines: true)
|
|
592
633
|
$LOAD_PATH.unshift(*#{abs_load_paths.inspect})
|
|
593
634
|
#{abs_source_paths.inspect}.each { |f| load f }
|
|
594
635
|
load #{absolute(test_path).inspect}
|
|
595
|
-
_orig = $stdout
|
|
596
|
-
$stdout = StringIO.new
|
|
597
636
|
_passed = Minitest.run([])
|
|
598
|
-
|
|
599
|
-
|
|
600
|
-
|
|
637
|
+
_result.puts JSON.generate("passed" => _passed == true, "coverage" => Coverage.result,
|
|
638
|
+
"loaded_files" => #{loaded_files_expression})
|
|
639
|
+
_result.close
|
|
601
640
|
RUBY
|
|
602
641
|
end
|
|
603
642
|
|
|
604
643
|
# Same coverage-JSON contract as the minitest path, but driven by RSpec:
|
|
605
644
|
# require rspec/core lazily, load the sources under Coverage, then run the
|
|
606
|
-
# one spec via RSpec::Core::Runner
|
|
607
|
-
#
|
|
608
|
-
# non-zero -> capture() records a skipped (incomplete-map)
|
|
645
|
+
# one spec via RSpec::Core::Runner. The JSON goes to the result channel (see
|
|
646
|
+
# {#spawn_script}), so spec output cannot corrupt it. A missing rspec makes
|
|
647
|
+
# the script exit non-zero -> capture() records a skipped (incomplete-map)
|
|
648
|
+
# test, with a hint.
|
|
609
649
|
def rspec_subprocess_script(test_path)
|
|
610
650
|
<<~RUBY
|
|
651
|
+
#{result_channel_expression}
|
|
611
652
|
require "coverage"
|
|
612
653
|
require "json"
|
|
613
654
|
require "stringio"
|
|
@@ -621,13 +662,11 @@ module Mutineer
|
|
|
621
662
|
Coverage.start(lines: true)
|
|
622
663
|
$LOAD_PATH.unshift(*#{abs_load_paths.inspect})
|
|
623
664
|
#{abs_source_paths.inspect}.each { |f| load f }
|
|
624
|
-
_orig = $stdout
|
|
625
665
|
_sink = StringIO.new
|
|
626
|
-
$stdout = _sink
|
|
627
666
|
_status = RSpec::Core::Runner.run(["--no-color", #{absolute(test_path).inspect}], _sink, _sink)
|
|
628
|
-
|
|
629
|
-
|
|
630
|
-
|
|
667
|
+
_result.puts JSON.generate("passed" => _status.zero?, "coverage" => Coverage.result,
|
|
668
|
+
"loaded_files" => #{loaded_files_expression})
|
|
669
|
+
_result.close
|
|
631
670
|
RUBY
|
|
632
671
|
end
|
|
633
672
|
|
|
@@ -2,6 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
require "json"
|
|
4
4
|
require "tempfile"
|
|
5
|
+
require_relative "child_stdout"
|
|
5
6
|
|
|
6
7
|
module Mutineer
|
|
7
8
|
# App-side daemon (persistent worker).
|
|
@@ -177,9 +178,9 @@ module Mutineer
|
|
|
177
178
|
# New process group so a per-fork timeout can SIGKILL the whole subtree,
|
|
178
179
|
# and silence the child's stdout so test output never corrupts the IPC pipe.
|
|
179
180
|
Process.setpgid(0, 0) rescue nil # rubocop:disable Style/RescueModifier
|
|
180
|
-
$stdout.reopen(File::NULL, "w")
|
|
181
181
|
code =
|
|
182
182
|
begin
|
|
183
|
+
ChildStdout.silence
|
|
183
184
|
# Route THIS fork at its own worker database before any test loads.
|
|
184
185
|
# A routing failure raises here and is scored `error`, never a false verdict.
|
|
185
186
|
@worker_db&.after_fork(worker, schema_for_fork)
|
data/lib/mutineer/isolation.rb
CHANGED
|
@@ -3,6 +3,7 @@
|
|
|
3
3
|
require "tempfile"
|
|
4
4
|
require_relative "result"
|
|
5
5
|
require_relative "parser"
|
|
6
|
+
require_relative "child_stdout"
|
|
6
7
|
|
|
7
8
|
module Mutineer
|
|
8
9
|
# Fork-based isolation for running one mutant. The block runs in a child
|
|
@@ -26,6 +27,9 @@ module Mutineer
|
|
|
26
27
|
# exit code) or any explicit `exit` is honoured; an unhandled exception
|
|
27
28
|
# becomes exit 2 with the cause written to STDERR.
|
|
28
29
|
#
|
|
30
|
+
# The child silences its stdout (see {ChildStdout.silence}) before the
|
|
31
|
+
# block runs, so test output never reaches the user. Stderr stays open.
|
|
32
|
+
#
|
|
29
33
|
# @param timeout [Integer] timeout in seconds.
|
|
30
34
|
# @yieldreturn [Integer] child exit status.
|
|
31
35
|
# @return [Mutineer::Result] result from the child process.
|
|
@@ -36,15 +40,17 @@ module Mutineer
|
|
|
36
40
|
Process.setpgid(0, 0) rescue nil # rubocop:disable Style/RescueModifier
|
|
37
41
|
code = 0
|
|
38
42
|
begin
|
|
43
|
+
ChildStdout.silence
|
|
39
44
|
result = yield
|
|
40
45
|
code = result.is_a?(Integer) ? result : 0
|
|
41
46
|
rescue SystemExit => e
|
|
42
47
|
code = e.status
|
|
43
48
|
rescue Exception => e # rubocop:disable Lint/RescueException
|
|
44
|
-
warn
|
|
49
|
+
# STDERR, not `warn`: a test may have left `$stderr` as a StringIO.
|
|
50
|
+
STDERR.puts "[mutineer-child] #{e.class}: #{e.message}"
|
|
45
51
|
code = 2
|
|
46
52
|
end
|
|
47
|
-
|
|
53
|
+
STDERR.flush
|
|
48
54
|
# exit! skips at_exit handlers — critical, since a child forked from
|
|
49
55
|
# inside our own Minitest suite would otherwise re-run the parent's
|
|
50
56
|
# at_exit autorun hook on the way out.
|
|
@@ -0,0 +1,199 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Mutineer
|
|
4
|
+
class MinitestIntegration
|
|
5
|
+
# Stops a Minitest run at the first failing test: one failure already
|
|
6
|
+
# kills the mutant. Child-process only, like MinitestIntegration.
|
|
7
|
+
#
|
|
8
|
+
# It sets a flag instead of unwinding the stack, so that class-level
|
|
9
|
+
# wrappers (`after_all`, a block-form transaction) still finish. The
|
|
10
|
+
# remaining tests and classes then return before they start.
|
|
11
|
+
#
|
|
12
|
+
# Minitest 5 and 6 need different hook points. An unknown shape installs
|
|
13
|
+
# no hook, and the run is a full run.
|
|
14
|
+
module StopAtFirstFailure
|
|
15
|
+
# Prepended on `Minitest::CompositeReporter`.
|
|
16
|
+
module RecordFailure
|
|
17
|
+
# Records the result, then sets the stop flag on a failure or an
|
|
18
|
+
# error. Only the outer reporter counts: a test can build and record
|
|
19
|
+
# on its own reporter.
|
|
20
|
+
#
|
|
21
|
+
# @param result [Minitest::Result] the result of one test.
|
|
22
|
+
# @return [void]
|
|
23
|
+
def record(result)
|
|
24
|
+
super
|
|
25
|
+
return if result.passed? || result.skipped?
|
|
26
|
+
return unless equal?(StopAtFirstFailure.armed_reporter)
|
|
27
|
+
return unless StopAtFirstFailure.armed_here?
|
|
28
|
+
|
|
29
|
+
StopAtFirstFailure.stopped = true
|
|
30
|
+
end
|
|
31
|
+
end
|
|
32
|
+
|
|
33
|
+
# Prepended on the `Minitest` singleton class for Minitest 6.
|
|
34
|
+
module OuterReporter6
|
|
35
|
+
# Keeps the outer reporter. Only the first call counts, because a test
|
|
36
|
+
# can call this method with its own reporter.
|
|
37
|
+
#
|
|
38
|
+
# @param reporter [Minitest::CompositeReporter] the outer reporter.
|
|
39
|
+
# @param options [Hash] the Minitest options.
|
|
40
|
+
# @return [Object] whatever Minitest returns.
|
|
41
|
+
def run_all_suites(reporter, options)
|
|
42
|
+
StopAtFirstFailure.armed_reporter ||= reporter if StopAtFirstFailure.armed_here?
|
|
43
|
+
super
|
|
44
|
+
end
|
|
45
|
+
end
|
|
46
|
+
|
|
47
|
+
# Prepended on the `Minitest` singleton class for Minitest 5.
|
|
48
|
+
module OuterReporter5
|
|
49
|
+
# The Minitest 5 form of OuterReporter6#run_all_suites.
|
|
50
|
+
#
|
|
51
|
+
# @param reporter [Minitest::CompositeReporter] the outer reporter.
|
|
52
|
+
# @param options [Hash] the Minitest options.
|
|
53
|
+
# @return [Object] whatever Minitest returns.
|
|
54
|
+
def __run(reporter, options)
|
|
55
|
+
StopAtFirstFailure.armed_reporter ||= reporter if StopAtFirstFailure.armed_here?
|
|
56
|
+
super
|
|
57
|
+
end
|
|
58
|
+
end
|
|
59
|
+
|
|
60
|
+
# Prepended on the singleton class of each test class for Minitest 6.
|
|
61
|
+
module SkipAfterStop6
|
|
62
|
+
# Skips the test class after a stop.
|
|
63
|
+
#
|
|
64
|
+
# @param reporter [Minitest::CompositeReporter] the reporter.
|
|
65
|
+
# @param options [Hash] the Minitest options.
|
|
66
|
+
# @return [Object, nil] whatever Minitest returns, or nil when skipped.
|
|
67
|
+
def run_suite(reporter, options = {})
|
|
68
|
+
return if StopAtFirstFailure.stopped_here?
|
|
69
|
+
|
|
70
|
+
super
|
|
71
|
+
end
|
|
72
|
+
|
|
73
|
+
# Skips one test after a stop, without a record.
|
|
74
|
+
#
|
|
75
|
+
# @param klass [Class] the test class.
|
|
76
|
+
# @param method_name [String] the test method.
|
|
77
|
+
# @param reporter [Minitest::CompositeReporter] the reporter.
|
|
78
|
+
# @return [Object, nil] whatever Minitest returns, or nil when skipped.
|
|
79
|
+
def run(klass, method_name, reporter)
|
|
80
|
+
return if StopAtFirstFailure.stopped_here?
|
|
81
|
+
|
|
82
|
+
super
|
|
83
|
+
end
|
|
84
|
+
end
|
|
85
|
+
|
|
86
|
+
# Prepended on the singleton class of each test class for Minitest 5.
|
|
87
|
+
module SkipAfterStop5
|
|
88
|
+
# Skips the test class after a stop.
|
|
89
|
+
#
|
|
90
|
+
# @param reporter [Minitest::CompositeReporter] the reporter.
|
|
91
|
+
# @param options [Hash] the Minitest options.
|
|
92
|
+
# @return [Object, nil] whatever Minitest returns, or nil when skipped.
|
|
93
|
+
def run(reporter, options = {})
|
|
94
|
+
return if StopAtFirstFailure.stopped_here?
|
|
95
|
+
|
|
96
|
+
super
|
|
97
|
+
end
|
|
98
|
+
|
|
99
|
+
# Skips one test after a stop, without a record.
|
|
100
|
+
#
|
|
101
|
+
# @param klass [Class] the test class.
|
|
102
|
+
# @param method_name [String] the test method.
|
|
103
|
+
# @param reporter [Minitest::CompositeReporter] the reporter.
|
|
104
|
+
# @return [Object, nil] whatever Minitest returns, or nil when skipped.
|
|
105
|
+
def run_one_method(klass, method_name, reporter)
|
|
106
|
+
return if StopAtFirstFailure.stopped_here?
|
|
107
|
+
|
|
108
|
+
super
|
|
109
|
+
end
|
|
110
|
+
end
|
|
111
|
+
|
|
112
|
+
class << self
|
|
113
|
+
# The pid of the armed process. Forked workers inherit the other
|
|
114
|
+
# state, and the pid keeps them from acting on it.
|
|
115
|
+
#
|
|
116
|
+
# @return [Integer, nil]
|
|
117
|
+
attr_accessor :armed_pid
|
|
118
|
+
|
|
119
|
+
# The outer reporter of the armed run.
|
|
120
|
+
#
|
|
121
|
+
# @return [Minitest::CompositeReporter, nil]
|
|
122
|
+
attr_accessor :armed_reporter
|
|
123
|
+
|
|
124
|
+
# True after the outer reporter records a failure or an error.
|
|
125
|
+
#
|
|
126
|
+
# @return [Boolean, nil]
|
|
127
|
+
attr_accessor :stopped
|
|
128
|
+
|
|
129
|
+
# Installs the hooks and arms the stop for this process. Call it after
|
|
130
|
+
# the test files load, so that every test class gets its prepend.
|
|
131
|
+
#
|
|
132
|
+
# @param runnables [Array<Class>] the loaded test classes.
|
|
133
|
+
# @return [Boolean] false when the Minitest shape is unknown.
|
|
134
|
+
def arm!(runnables)
|
|
135
|
+
outer, skip = hooks_for_loaded_minitest
|
|
136
|
+
return false unless outer
|
|
137
|
+
|
|
138
|
+
prepend_once(::Minitest::CompositeReporter, RecordFailure)
|
|
139
|
+
prepend_once(::Minitest.singleton_class, outer)
|
|
140
|
+
runnables.each { |klass| prepend_once(klass.singleton_class, skip) }
|
|
141
|
+
self.armed_pid = Process.pid
|
|
142
|
+
self.armed_reporter = nil
|
|
143
|
+
self.stopped = false
|
|
144
|
+
true
|
|
145
|
+
end
|
|
146
|
+
|
|
147
|
+
# Clears the armed state.
|
|
148
|
+
#
|
|
149
|
+
# @return [void]
|
|
150
|
+
def disarm!
|
|
151
|
+
self.armed_pid = nil
|
|
152
|
+
self.armed_reporter = nil
|
|
153
|
+
self.stopped = false
|
|
154
|
+
end
|
|
155
|
+
|
|
156
|
+
# True when the run is armed in this process.
|
|
157
|
+
#
|
|
158
|
+
# @return [Boolean]
|
|
159
|
+
def armed_here?
|
|
160
|
+
armed_pid == Process.pid
|
|
161
|
+
end
|
|
162
|
+
|
|
163
|
+
# True when the armed run in this process has stopped.
|
|
164
|
+
#
|
|
165
|
+
# @return [Boolean]
|
|
166
|
+
def stopped_here?
|
|
167
|
+
stopped == true && armed_here?
|
|
168
|
+
end
|
|
169
|
+
|
|
170
|
+
private
|
|
171
|
+
|
|
172
|
+
# Picks the hook modules for the loaded Minitest.
|
|
173
|
+
#
|
|
174
|
+
# @return [Array(Module, Module), nil] the outer reporter hook and the
|
|
175
|
+
# skip hook, or nil for an unknown shape.
|
|
176
|
+
def hooks_for_loaded_minitest
|
|
177
|
+
runnable = ::Minitest::Runnable
|
|
178
|
+
if ::Minitest.respond_to?(:run_all_suites) && runnable.respond_to?(:run_suite)
|
|
179
|
+
[OuterReporter6, SkipAfterStop6]
|
|
180
|
+
elsif ::Minitest.respond_to?(:__run) && runnable.respond_to?(:run_one_method)
|
|
181
|
+
[OuterReporter5, SkipAfterStop5]
|
|
182
|
+
end
|
|
183
|
+
end
|
|
184
|
+
|
|
185
|
+
# Prepends `mod` on `target` once. A copy on a superclass does not
|
|
186
|
+
# count, because it comes after the own methods of `target`.
|
|
187
|
+
#
|
|
188
|
+
# @param target [Module] the class or singleton class.
|
|
189
|
+
# @param mod [Module] the module to prepend.
|
|
190
|
+
# @return [void]
|
|
191
|
+
def prepend_once(target, mod)
|
|
192
|
+
return if target.ancestors.take_while { |a| !a.equal?(target) }.include?(mod)
|
|
193
|
+
|
|
194
|
+
target.prepend(mod)
|
|
195
|
+
end
|
|
196
|
+
end
|
|
197
|
+
end
|
|
198
|
+
end
|
|
199
|
+
end
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# frozen_string_literal: true
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
require_relative "minitest_integration/stop_at_first_failure"
|
|
4
4
|
|
|
5
5
|
module Mutineer
|
|
6
6
|
# Child-process-only: loads a test file in the current process and runs it
|
|
@@ -16,6 +16,11 @@ module Mutineer
|
|
|
16
16
|
# boundary for unexpected errors. Swallowing those here would create a
|
|
17
17
|
# second exit-2 path and break this method's 0/1 return contract.
|
|
18
18
|
class MinitestIntegration
|
|
19
|
+
# The seed of a run that stops at the first failure, unless `SEED` is set.
|
|
20
|
+
# With the stop, the test order can decide `killed` vs `timeout`; a fixed
|
|
21
|
+
# order keeps the verdict stable for a `--baseline` gate.
|
|
22
|
+
STOP_AT_FIRST_FAILURE_SEED = 1
|
|
23
|
+
|
|
19
24
|
# Tested via runner_test.rb, not in isolation — a direct unit test
|
|
20
25
|
# would require forking and duplicate isolation_test's coverage.
|
|
21
26
|
#
|
|
@@ -24,8 +29,10 @@ module Mutineer
|
|
|
24
29
|
# Minitest.run.
|
|
25
30
|
#
|
|
26
31
|
# @param test_files [String, Array<String>] one file or many files.
|
|
32
|
+
# @param stop_at_first_failure [Boolean] end the run at the first failing
|
|
33
|
+
# test. Only the mutant path passes true.
|
|
27
34
|
# @return [Integer] 0 on success, 1 on failure.
|
|
28
|
-
def self.run(test_files)
|
|
35
|
+
def self.run(test_files, stop_at_first_failure: false)
|
|
29
36
|
begin
|
|
30
37
|
require "minitest"
|
|
31
38
|
rescue LoadError
|
|
@@ -47,13 +54,20 @@ module Mutineer
|
|
|
47
54
|
Minitest::Runnable.reset
|
|
48
55
|
Array(test_files).each { |f| load f }
|
|
49
56
|
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
|
|
57
|
+
args = []
|
|
58
|
+
if stop_at_first_failure && StopAtFirstFailure.arm!(Minitest::Runnable.runnables)
|
|
59
|
+
# Pin the seed only when the stop is armed; an unknown Minitest shape
|
|
60
|
+
# gets the normal full, randomly ordered run.
|
|
61
|
+
args = ["--seed", STOP_AT_FIRST_FAILURE_SEED.to_s] unless ENV["SEED"]
|
|
62
|
+
end
|
|
63
|
+
# No silencing here: the fork boundary that calls this method has already
|
|
64
|
+
# pointed stdout at File::NULL (see ChildStdout).
|
|
65
|
+
passed = Minitest.run(args)
|
|
55
66
|
|
|
56
|
-
|
|
67
|
+
# A plugin can replace the summary reporter, so a stop decides by itself.
|
|
68
|
+
passed && !StopAtFirstFailure.stopped_here? ? 0 : 1
|
|
69
|
+
ensure
|
|
70
|
+
StopAtFirstFailure.disarm!
|
|
57
71
|
end
|
|
58
72
|
end
|
|
59
73
|
end
|
|
@@ -11,12 +11,16 @@ require_relative "mutators/condition_negation"
|
|
|
11
11
|
require_relative "mutators/string_literal"
|
|
12
12
|
require_relative "mutators/regex_literal"
|
|
13
13
|
require_relative "mutators/collection_method"
|
|
14
|
+
require_relative "mutators/safe_navigation"
|
|
15
|
+
require_relative "mutators/range_literal"
|
|
16
|
+
require_relative "mutators/negation_removal"
|
|
17
|
+
require_relative "mutators/chain_link"
|
|
14
18
|
|
|
15
19
|
module Mutineer
|
|
16
20
|
# Maps operator names to operator classes.
|
|
17
21
|
#
|
|
18
22
|
# DEFAULT_NAMES is the v1 default set (Tier-1 plus statement-removal).
|
|
19
|
-
# The
|
|
23
|
+
# The Tier-2 operators live in ALL but are OFF by default — they only
|
|
20
24
|
# run when named via `--operators` or `operators:` in `.mutineer.yml`.
|
|
21
25
|
# Keeping DEFAULT_NAMES an explicit subset (not ALL.keys) is what keeps
|
|
22
26
|
# the default survivor set unchanged.
|
|
@@ -33,13 +37,19 @@ module Mutineer
|
|
|
33
37
|
"condition_negation" => Mutators::ConditionNegation,
|
|
34
38
|
"string_literal" => Mutators::StringLiteral,
|
|
35
39
|
"regex" => Mutators::RegexLiteral,
|
|
36
|
-
"collection_method" => Mutators::CollectionMethod
|
|
40
|
+
"collection_method" => Mutators::CollectionMethod,
|
|
41
|
+
"safe_navigation" => Mutators::SafeNavigation,
|
|
42
|
+
"range" => Mutators::RangeLiteral,
|
|
43
|
+
"negation_removal" => Mutators::NegationRemoval,
|
|
44
|
+
"chain_link" => Mutators::ChainLink
|
|
37
45
|
}.freeze
|
|
38
46
|
|
|
39
47
|
# The default Tier-1 operator set.
|
|
40
48
|
DEFAULT_NAMES = %w[arithmetic comparison boolean_connector boolean_literal statement_removal].freeze
|
|
41
49
|
# Tier-2 operators that remain opt-in.
|
|
42
|
-
TIER2_NAMES = %w[return_nil literal_mutation condition_negation string_literal regex collection_method
|
|
50
|
+
TIER2_NAMES = %w[return_nil literal_mutation condition_negation string_literal regex collection_method
|
|
51
|
+
safe_navigation range negation_removal
|
|
52
|
+
chain_link].freeze
|
|
43
53
|
|
|
44
54
|
# Short human-readable descriptions for each operator.
|
|
45
55
|
DESCRIPTIONS = {
|
|
@@ -53,7 +63,11 @@ module Mutineer
|
|
|
53
63
|
"condition_negation" => "wrap if/unless/ternary condition in !( ... )",
|
|
54
64
|
"string_literal" => "non-empty string -> \"\", empty string -> \"mutineer\"",
|
|
55
65
|
"regex" => "drop leading ^ / trailing $, swap + <-> *",
|
|
56
|
-
"collection_method" => "map<->each, all?<->any?, first<->last, min<->max, select<->reject"
|
|
66
|
+
"collection_method" => "map<->each, all?<->any?, first<->last, min<->max, select<->reject",
|
|
67
|
+
"safe_navigation" => "&. -> .",
|
|
68
|
+
"range" => ".. <-> ...",
|
|
69
|
+
"negation_removal" => "!x, not x -> x",
|
|
70
|
+
"chain_link" => "drop one call from a chain: a.b.c -> a.c"
|
|
57
71
|
}.freeze
|
|
58
72
|
|
|
59
73
|
# Resolves operator names to classes.
|
|
@@ -0,0 +1,139 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require_relative "base"
|
|
4
|
+
|
|
5
|
+
module Mutineer
|
|
6
|
+
module Mutators
|
|
7
|
+
# Chain-link mutator (Tier-2).
|
|
8
|
+
#
|
|
9
|
+
# Drops one dotted call from a chain of calls, with its arguments and block:
|
|
10
|
+
# `user.account.owner.name` becomes `user.owner.name` and `user.account.name`.
|
|
11
|
+
# The chain's receiver and its final call stay, so a chain of n dotted calls
|
|
12
|
+
# gives at most n - 1 mutations. A chain begins at a receiver that is not a
|
|
13
|
+
# dotted call: a local, a constant, `self`, `list[i]`, `(a + b)`. A call in
|
|
14
|
+
# an argument or a block starts a chain of its own.
|
|
15
|
+
#
|
|
16
|
+
# The mutant survives when no test tells the chain apart from the same chain
|
|
17
|
+
# without that step: a scope, a filter or a lookup the tests never see.
|
|
18
|
+
#
|
|
19
|
+
# Links in {SKIPPED} are never dropped. On a value that already has the
|
|
20
|
+
# target type or needs no copy, a conversion (`name.to_s.strip`) or a copy
|
|
21
|
+
# (`list.dup.sort`) is a no-op, so dropping it most often makes an
|
|
22
|
+
# equivalent mutant. Dropping `new` sends the next call to the class, which
|
|
23
|
+
# raises (killed by any test that runs the line) or reaches a class method
|
|
24
|
+
# that builds the instance itself (equivalent); neither says anything about
|
|
25
|
+
# the tests.
|
|
26
|
+
class ChainLink < Base
|
|
27
|
+
# Method names whose link is never dropped: core conversions and copies,
|
|
28
|
+
# plus `new`.
|
|
29
|
+
SKIPPED = %i[
|
|
30
|
+
to_s to_str to_sym to_i to_int to_f to_r to_c to_a to_ary to_h to_hash to_proc to_set
|
|
31
|
+
dup clone freeze itself
|
|
32
|
+
new
|
|
33
|
+
].freeze
|
|
34
|
+
|
|
35
|
+
# Resets the per-subject record of calls already placed in a chain.
|
|
36
|
+
#
|
|
37
|
+
# @param subject [Mutineer::Subject] subject whose body is visited.
|
|
38
|
+
# @param source [String] full source text for byte-based slicing.
|
|
39
|
+
# @return [Array<Mutineer::Mutation>] collected mutations.
|
|
40
|
+
def mutations_for(subject, source)
|
|
41
|
+
@links = {}.compare_by_identity
|
|
42
|
+
super
|
|
43
|
+
end
|
|
44
|
+
|
|
45
|
+
# Visits a call. The outermost dotted call of a chain is its final call;
|
|
46
|
+
# the dotted calls below it in the receiver are its links.
|
|
47
|
+
#
|
|
48
|
+
# @param node [Prism::CallNode] call node to inspect.
|
|
49
|
+
# @return [void]
|
|
50
|
+
def visit_call_node(node)
|
|
51
|
+
chain(node) if dotted?(node) && !@links.key?(node)
|
|
52
|
+
super
|
|
53
|
+
end
|
|
54
|
+
|
|
55
|
+
# Visits `a.b.c += 1`, whose final call Prism parses as its own node.
|
|
56
|
+
#
|
|
57
|
+
# @param node [Prism::CallOperatorWriteNode] node to inspect.
|
|
58
|
+
# @return [void]
|
|
59
|
+
def visit_call_operator_write_node(node)
|
|
60
|
+
chain(node)
|
|
61
|
+
super
|
|
62
|
+
end
|
|
63
|
+
|
|
64
|
+
# Visits `a.b.c ||= 1`.
|
|
65
|
+
#
|
|
66
|
+
# @param node [Prism::CallOrWriteNode] node to inspect.
|
|
67
|
+
# @return [void]
|
|
68
|
+
def visit_call_or_write_node(node)
|
|
69
|
+
chain(node)
|
|
70
|
+
super
|
|
71
|
+
end
|
|
72
|
+
|
|
73
|
+
# Visits `a.b.c &&= 1`.
|
|
74
|
+
#
|
|
75
|
+
# @param node [Prism::CallAndWriteNode] node to inspect.
|
|
76
|
+
# @return [void]
|
|
77
|
+
def visit_call_and_write_node(node)
|
|
78
|
+
chain(node)
|
|
79
|
+
super
|
|
80
|
+
end
|
|
81
|
+
|
|
82
|
+
# Visits a call used as an assignment target, as in `a.b.c, d = 1, 2`.
|
|
83
|
+
#
|
|
84
|
+
# @param node [Prism::CallTargetNode] node to inspect.
|
|
85
|
+
# @return [void]
|
|
86
|
+
def visit_call_target_node(node)
|
|
87
|
+
chain(node)
|
|
88
|
+
super
|
|
89
|
+
end
|
|
90
|
+
|
|
91
|
+
# Nested method definitions are discovered as their own subjects; do not
|
|
92
|
+
# recurse into them (prevents double-counting their chains).
|
|
93
|
+
#
|
|
94
|
+
# @param node [Prism::DefNode] nested definition node.
|
|
95
|
+
# @return [void]
|
|
96
|
+
def visit_def_node(node); end
|
|
97
|
+
|
|
98
|
+
private
|
|
99
|
+
|
|
100
|
+
# Walks a final call's receiver chain, recording each link so it is not
|
|
101
|
+
# taken for the final call of a chain of its own, and dropping each link
|
|
102
|
+
# not in {SKIPPED}.
|
|
103
|
+
#
|
|
104
|
+
# @param top [Prism::Node] the chain's final call; responds to `receiver`.
|
|
105
|
+
# @return [void]
|
|
106
|
+
def chain(top)
|
|
107
|
+
link = top.receiver
|
|
108
|
+
while dotted?(link)
|
|
109
|
+
@links[link] = true
|
|
110
|
+
drop(link) unless SKIPPED.include?(link.name)
|
|
111
|
+
link = link.receiver
|
|
112
|
+
end
|
|
113
|
+
end
|
|
114
|
+
|
|
115
|
+
# Emits the removal of one link: from the end of its receiver to the end
|
|
116
|
+
# of its arguments and block, so a chain split across lines keeps the
|
|
117
|
+
# layout of the lines that remain.
|
|
118
|
+
#
|
|
119
|
+
# @param link [Prism::CallNode] the dotted call to drop.
|
|
120
|
+
# @return [void]
|
|
121
|
+
def drop(link)
|
|
122
|
+
@mutations << Mutation.new(
|
|
123
|
+
start_offset: link.receiver.location.end_offset,
|
|
124
|
+
end_offset: link.location.end_offset,
|
|
125
|
+
replacement: "",
|
|
126
|
+
operator: :chain_link
|
|
127
|
+
)
|
|
128
|
+
end
|
|
129
|
+
|
|
130
|
+
# Returns whether a node is a call through `.`, `&.` or `::`.
|
|
131
|
+
#
|
|
132
|
+
# @param node [Prism::Node, nil] node to inspect.
|
|
133
|
+
# @return [Boolean] true for a call node with a call operator.
|
|
134
|
+
def dotted?(node)
|
|
135
|
+
node.is_a?(Prism::CallNode) && !node.call_operator_loc.nil?
|
|
136
|
+
end
|
|
137
|
+
end
|
|
138
|
+
end
|
|
139
|
+
end
|
|
@@ -0,0 +1,44 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require_relative "base"
|
|
4
|
+
|
|
5
|
+
module Mutineer
|
|
6
|
+
module Mutators
|
|
7
|
+
# Negation-removal mutator (Tier-2).
|
|
8
|
+
#
|
|
9
|
+
# Removes the `!` or `not` of a negation, one mutation per occurrence:
|
|
10
|
+
# `!x` becomes `x` and `not x` becomes ` x`. The mutant survives when no
|
|
11
|
+
# test depends on the negated value.
|
|
12
|
+
class NegationRemoval < Base
|
|
13
|
+
# Visits call nodes and emits negation-removal mutations.
|
|
14
|
+
#
|
|
15
|
+
# @param node [Prism::CallNode] call node to inspect.
|
|
16
|
+
# @return [void]
|
|
17
|
+
def visit_call_node(node)
|
|
18
|
+
emit(node)
|
|
19
|
+
super # nested negations (!!x) each get their own mutation
|
|
20
|
+
end
|
|
21
|
+
|
|
22
|
+
private
|
|
23
|
+
|
|
24
|
+
# Emits a mutation when the node is a prefix `!` or `not`.
|
|
25
|
+
#
|
|
26
|
+
# The explicit form `x.!` is skipped: without its message, `x.` does
|
|
27
|
+
# not parse.
|
|
28
|
+
#
|
|
29
|
+
# @param node [Prism::CallNode] call node to inspect.
|
|
30
|
+
# @return [void]
|
|
31
|
+
def emit(node)
|
|
32
|
+
return unless node.name == :! && node.call_operator_loc.nil?
|
|
33
|
+
|
|
34
|
+
loc = node.message_loc
|
|
35
|
+
@mutations << Mutation.new(
|
|
36
|
+
start_offset: loc.start_offset,
|
|
37
|
+
end_offset: loc.end_offset,
|
|
38
|
+
replacement: "",
|
|
39
|
+
operator: :negation_removal
|
|
40
|
+
)
|
|
41
|
+
end
|
|
42
|
+
end
|
|
43
|
+
end
|
|
44
|
+
end
|
|
@@ -0,0 +1,54 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require_relative "base"
|
|
4
|
+
|
|
5
|
+
module Mutineer
|
|
6
|
+
module Mutators
|
|
7
|
+
# Range mutator (Tier-2).
|
|
8
|
+
#
|
|
9
|
+
# Swaps `..` with `...` and `...` with `..`, one mutation per range. The
|
|
10
|
+
# `..` -> `...` mutant survives when no test checks the last element.
|
|
11
|
+
# Flip-flops are FlipFlopNode, not RangeNode, so this mutator skips them.
|
|
12
|
+
#
|
|
13
|
+
# Endless ranges (`1..`, `1..nil`) are skipped. `(1..)` and `(1...)` give
|
|
14
|
+
# the same result for slicing, `include?`, `===`, `size` and patterns, so
|
|
15
|
+
# the mutant is equivalent and no normal test can kill it.
|
|
16
|
+
class RangeLiteral < Base
|
|
17
|
+
# Maps each range operator to its opposite.
|
|
18
|
+
SWAPS = { ".." => "...", "..." => ".." }.freeze
|
|
19
|
+
|
|
20
|
+
# Visits range nodes and emits range mutations.
|
|
21
|
+
#
|
|
22
|
+
# @param node [Prism::RangeNode] range node to inspect.
|
|
23
|
+
# @return [void]
|
|
24
|
+
def visit_range_node(node)
|
|
25
|
+
emit(node) unless endless?(node)
|
|
26
|
+
super # nested ranges ((1..2)...(3..4)) each get their own mutation
|
|
27
|
+
end
|
|
28
|
+
|
|
29
|
+
private
|
|
30
|
+
|
|
31
|
+
# Whether the range has no end: `1..` or `1..nil`.
|
|
32
|
+
#
|
|
33
|
+
# @param node [Prism::RangeNode] range node to inspect.
|
|
34
|
+
# @return [Boolean]
|
|
35
|
+
def endless?(node)
|
|
36
|
+
node.right.nil? || node.right.is_a?(Prism::NilNode)
|
|
37
|
+
end
|
|
38
|
+
|
|
39
|
+
# Emits the mutation that swaps the range operator.
|
|
40
|
+
#
|
|
41
|
+
# @param node [Prism::RangeNode] range node to mutate.
|
|
42
|
+
# @return [void]
|
|
43
|
+
def emit(node)
|
|
44
|
+
loc = node.operator_loc
|
|
45
|
+
@mutations << Mutation.new(
|
|
46
|
+
start_offset: loc.start_offset,
|
|
47
|
+
end_offset: loc.end_offset,
|
|
48
|
+
replacement: SWAPS.fetch(loc.slice),
|
|
49
|
+
operator: :range
|
|
50
|
+
)
|
|
51
|
+
end
|
|
52
|
+
end
|
|
53
|
+
end
|
|
54
|
+
end
|
|
@@ -0,0 +1,76 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require_relative "base"
|
|
4
|
+
|
|
5
|
+
module Mutineer
|
|
6
|
+
module Mutators
|
|
7
|
+
# Safe-navigation mutator (Tier-2).
|
|
8
|
+
#
|
|
9
|
+
# Replaces `&.` with `.`, one mutation per occurrence. The mutant raises
|
|
10
|
+
# NoMethodError on a nil receiver, so it survives when no test passes nil.
|
|
11
|
+
class SafeNavigation < Base
|
|
12
|
+
# Visits call nodes and emits safe-navigation mutations.
|
|
13
|
+
#
|
|
14
|
+
# @param node [Prism::CallNode] call node to inspect.
|
|
15
|
+
# @return [void]
|
|
16
|
+
def visit_call_node(node)
|
|
17
|
+
emit(node)
|
|
18
|
+
super # chained calls (a&.b&.c) each get their own mutation
|
|
19
|
+
end
|
|
20
|
+
|
|
21
|
+
# Visits `a&.b += 1`, which Prism parses as its own node, not a CallNode.
|
|
22
|
+
#
|
|
23
|
+
# @param node [Prism::CallOperatorWriteNode] node to inspect.
|
|
24
|
+
# @return [void]
|
|
25
|
+
def visit_call_operator_write_node(node)
|
|
26
|
+
emit(node)
|
|
27
|
+
super
|
|
28
|
+
end
|
|
29
|
+
|
|
30
|
+
# Visits `a&.b ||= 1`.
|
|
31
|
+
#
|
|
32
|
+
# @param node [Prism::CallOrWriteNode] node to inspect.
|
|
33
|
+
# @return [void]
|
|
34
|
+
def visit_call_or_write_node(node)
|
|
35
|
+
emit(node)
|
|
36
|
+
super
|
|
37
|
+
end
|
|
38
|
+
|
|
39
|
+
# Visits `a&.b &&= 1`.
|
|
40
|
+
#
|
|
41
|
+
# @param node [Prism::CallAndWriteNode] node to inspect.
|
|
42
|
+
# @return [void]
|
|
43
|
+
def visit_call_and_write_node(node)
|
|
44
|
+
emit(node)
|
|
45
|
+
super
|
|
46
|
+
end
|
|
47
|
+
|
|
48
|
+
# Visits a call used as an assignment target, as in `for a&.b in list`.
|
|
49
|
+
#
|
|
50
|
+
# @param node [Prism::CallTargetNode] node to inspect.
|
|
51
|
+
# @return [void]
|
|
52
|
+
def visit_call_target_node(node)
|
|
53
|
+
emit(node)
|
|
54
|
+
super
|
|
55
|
+
end
|
|
56
|
+
|
|
57
|
+
private
|
|
58
|
+
|
|
59
|
+
# Emits a mutation when the node's call operator is `&.`.
|
|
60
|
+
#
|
|
61
|
+
# @param node [Prism::Node] a node with `safe_navigation?` and `call_operator_loc`.
|
|
62
|
+
# @return [void]
|
|
63
|
+
def emit(node)
|
|
64
|
+
return unless node.safe_navigation?
|
|
65
|
+
|
|
66
|
+
loc = node.call_operator_loc
|
|
67
|
+
@mutations << Mutation.new(
|
|
68
|
+
start_offset: loc.start_offset,
|
|
69
|
+
end_offset: loc.end_offset,
|
|
70
|
+
replacement: ".",
|
|
71
|
+
operator: :safe_navigation
|
|
72
|
+
)
|
|
73
|
+
end
|
|
74
|
+
end
|
|
75
|
+
end
|
|
76
|
+
end
|
data/lib/mutineer/runner.rb
CHANGED
|
@@ -434,7 +434,8 @@ module Mutineer
|
|
|
434
434
|
else
|
|
435
435
|
Isolation.apply_whole_file(mutated, source_file)
|
|
436
436
|
end
|
|
437
|
-
|
|
437
|
+
# One failing test already kills the mutant, so the child stops there.
|
|
438
|
+
TestRunners.for(framework).run(abs_tests, stop_at_first_failure: true)
|
|
438
439
|
end
|
|
439
440
|
end
|
|
440
441
|
|
|
@@ -9,8 +9,12 @@ module Mutineer
|
|
|
9
9
|
# Runs the given Minitest files.
|
|
10
10
|
#
|
|
11
11
|
# @param test_files [String, Array<String>] one file or many files.
|
|
12
|
+
# @param stop_at_first_failure [Boolean] when true, the run ends at the
|
|
13
|
+
# first failing test.
|
|
12
14
|
# @return [Integer] 0 on success, 1 on failure.
|
|
13
|
-
def self.run(test_files
|
|
15
|
+
def self.run(test_files, stop_at_first_failure: false)
|
|
16
|
+
MinitestIntegration.run(test_files, stop_at_first_failure: stop_at_first_failure)
|
|
17
|
+
end
|
|
14
18
|
end
|
|
15
19
|
end
|
|
16
20
|
end
|
|
@@ -16,24 +16,22 @@ module Mutineer
|
|
|
16
16
|
# Runs the given RSpec files.
|
|
17
17
|
#
|
|
18
18
|
# @param spec_files [String, Array<String>] one file or many files.
|
|
19
|
+
# @param stop_at_first_failure [Boolean] when true, the run ends at the
|
|
20
|
+
# first failing example (RSpec `--fail-fast`).
|
|
19
21
|
# @return [Integer] 0 on success, 1 on failure.
|
|
20
|
-
def self.run(spec_files)
|
|
22
|
+
def self.run(spec_files, stop_at_first_failure: false)
|
|
21
23
|
require_rspec!
|
|
22
24
|
|
|
23
25
|
::RSpec::Core::Runner.disable_autorun!
|
|
24
26
|
::RSpec.reset
|
|
25
27
|
|
|
28
|
+
# The sink takes RSpec's own formatter output. Spec output is not
|
|
29
|
+
# silenced here: the fork boundary that calls this method has already
|
|
30
|
+
# pointed stdout at File::NULL (see ChildStdout).
|
|
26
31
|
sink = StringIO.new
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
$stderr = sink
|
|
31
|
-
begin
|
|
32
|
-
status = ::RSpec::Core::Runner.run(["--no-color", *Array(spec_files)], sink, sink)
|
|
33
|
-
ensure
|
|
34
|
-
$stdout = orig_out
|
|
35
|
-
$stderr = orig_err
|
|
36
|
-
end
|
|
32
|
+
args = ["--no-color"]
|
|
33
|
+
args << "--fail-fast" if stop_at_first_failure
|
|
34
|
+
status = ::RSpec::Core::Runner.run([*args, *Array(spec_files)], sink, sink)
|
|
37
35
|
|
|
38
36
|
status.zero? ? 0 : 1
|
|
39
37
|
end
|
|
@@ -6,8 +6,10 @@ require_relative "test_runners/rspec"
|
|
|
6
6
|
module Mutineer
|
|
7
7
|
# Picks the test-framework runner.
|
|
8
8
|
#
|
|
9
|
-
# Each runner responds to `.run(files) -> 0/1`
|
|
10
|
-
# failure) and is called only inside a forked child.
|
|
9
|
+
# Each runner responds to `.run(files, stop_at_first_failure: false) -> 0/1`
|
|
10
|
+
# (0 = all passed, 1 = any failure) and is called only inside a forked child.
|
|
11
|
+
# Only the mutant path passes `stop_at_first_failure: true`. The fork boundary
|
|
12
|
+
# silences stdout (see ChildStdout); the runners do not.
|
|
11
13
|
module TestRunners
|
|
12
14
|
# Returns the runner module for a framework name.
|
|
13
15
|
#
|
data/lib/mutineer/version.rb
CHANGED
data/lib/mutineer.rb
CHANGED
|
@@ -25,6 +25,10 @@ require_relative "mutineer/mutators/condition_negation"
|
|
|
25
25
|
require_relative "mutineer/mutators/string_literal"
|
|
26
26
|
require_relative "mutineer/mutators/regex_literal"
|
|
27
27
|
require_relative "mutineer/mutators/collection_method"
|
|
28
|
+
require_relative "mutineer/mutators/safe_navigation"
|
|
29
|
+
require_relative "mutineer/mutators/range_literal"
|
|
30
|
+
require_relative "mutineer/mutators/negation_removal"
|
|
31
|
+
require_relative "mutineer/mutators/chain_link"
|
|
28
32
|
require_relative "mutineer/mutator_registry"
|
|
29
33
|
require_relative "mutineer/worker_pool"
|
|
30
34
|
require_relative "mutineer/progress"
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: mutineer
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 1.0
|
|
4
|
+
version: 1.1.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- David Teren
|
|
@@ -9,20 +9,6 @@ bindir: bin
|
|
|
9
9
|
cert_chain: []
|
|
10
10
|
date: 1980-01-02 00:00:00.000000000 Z
|
|
11
11
|
dependencies:
|
|
12
|
-
- !ruby/object:Gem::Dependency
|
|
13
|
-
name: minitest
|
|
14
|
-
requirement: !ruby/object:Gem::Requirement
|
|
15
|
-
requirements:
|
|
16
|
-
- - "~>"
|
|
17
|
-
- !ruby/object:Gem::Version
|
|
18
|
-
version: '5.0'
|
|
19
|
-
type: :development
|
|
20
|
-
prerelease: false
|
|
21
|
-
version_requirements: !ruby/object:Gem::Requirement
|
|
22
|
-
requirements:
|
|
23
|
-
- - "~>"
|
|
24
|
-
- !ruby/object:Gem::Version
|
|
25
|
-
version: '5.0'
|
|
26
12
|
- !ruby/object:Gem::Dependency
|
|
27
13
|
name: rake
|
|
28
14
|
requirement: !ruby/object:Gem::Requirement
|
|
@@ -68,6 +54,7 @@ files:
|
|
|
68
54
|
- lib/mutineer.rb
|
|
69
55
|
- lib/mutineer/baseline.rb
|
|
70
56
|
- lib/mutineer/changed_lines.rb
|
|
57
|
+
- lib/mutineer/child_stdout.rb
|
|
71
58
|
- lib/mutineer/cli.rb
|
|
72
59
|
- lib/mutineer/config.rb
|
|
73
60
|
- lib/mutineer/coverage_map.rb
|
|
@@ -78,6 +65,7 @@ files:
|
|
|
78
65
|
- lib/mutineer/file_swap.rb
|
|
79
66
|
- lib/mutineer/isolation.rb
|
|
80
67
|
- lib/mutineer/minitest_integration.rb
|
|
68
|
+
- lib/mutineer/minitest_integration/stop_at_first_failure.rb
|
|
81
69
|
- lib/mutineer/mutant_id.rb
|
|
82
70
|
- lib/mutineer/mutation.rb
|
|
83
71
|
- lib/mutineer/mutator_registry.rb
|
|
@@ -85,12 +73,16 @@ files:
|
|
|
85
73
|
- lib/mutineer/mutators/base.rb
|
|
86
74
|
- lib/mutineer/mutators/boolean_connector.rb
|
|
87
75
|
- lib/mutineer/mutators/boolean_literal.rb
|
|
76
|
+
- lib/mutineer/mutators/chain_link.rb
|
|
88
77
|
- lib/mutineer/mutators/collection_method.rb
|
|
89
78
|
- lib/mutineer/mutators/comparison.rb
|
|
90
79
|
- lib/mutineer/mutators/condition_negation.rb
|
|
91
80
|
- lib/mutineer/mutators/literal_mutation.rb
|
|
81
|
+
- lib/mutineer/mutators/negation_removal.rb
|
|
82
|
+
- lib/mutineer/mutators/range_literal.rb
|
|
92
83
|
- lib/mutineer/mutators/regex_literal.rb
|
|
93
84
|
- lib/mutineer/mutators/return_nil.rb
|
|
85
|
+
- lib/mutineer/mutators/safe_navigation.rb
|
|
94
86
|
- lib/mutineer/mutators/statement_removal.rb
|
|
95
87
|
- lib/mutineer/mutators/string_literal.rb
|
|
96
88
|
- lib/mutineer/pairing.rb
|