saboteur 0.2.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 6007efb7a83e5669822a1622876457e9e654187acc923981e641835258c7637d
+  data.tar.gz: 373680f8e52d7cbd24f1eb3dde2322101eccf565c2752e10a174bd5ff9e6d8e2
+SHA512:
+  metadata.gz: c0d37ea099ca68c5dc3608e2229446cb4e4494c623e63bb8033eb85a8a10bd7953e2cc84031678a15adda5da0b0da1f6bb1177badde04fa06dbd472f2d674534
+  data.tar.gz: 7484ed59697969071030c78ab81b8a716da93b09d4e95900b69b110f087809a6e1293153375104732457205af0e0cfb94f64eb34810242febd1e63be613fb97a

data/CHANGELOG.md

@@ -0,0 +1,47 @@
+# Changelog
+
+All notable changes are documented here. The format follows
+[Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and the project uses
+[Semantic Versioning](https://semver.org/).
+
+## [Unreleased]
+
+## [0.2.0] - 2026-10-10
+
+### Added
+- `--seed N` (and `seed:` in `.saboteur.yml`, default 1): every test run seeds
+  Ruby's random numbers with it, so runs repeat; the report prints the seed.
+
+### Fixed
+- Linux: killed orphans adopted by the supervisor are reaped reliably, so no
+  zombies are left behind (first run on Linux CI).
+- The README status line describes what has actually been tested.
+
+## [0.1.0] - 2026-10-10
+
+First release that does mutation testing. MIT licensed.
+
+### Added
+- `saboteur run` with `--since`, `--subject`, `--timeout-factor`, `--jobs`,
+  `--reeval`, `--json`, `--no-cache`, `--incremental`, `--config`.
+- Subjects found with Prism; mutations are byte-range edits: whole-method
+  `nil`/`true`/`false`/`raise` first, then comparison, boolean, arithmetic,
+  numeric literal, statement removal, condition, negation, safe navigation
+  and `||=`.
+- Per-example test selection from a per-test coverage map, cached by content
+  hash; timeouts scaled from each test's recorded duration.
+- Verdict check: every kill is re-run on the unchanged method, so a test that
+  fails anyway cannot count as a kill.
+- Supervised children: whole process tree tracked and killed, deadline,
+  memory and process caps, CPU-limit backstop, recursion guard.
+- Parallel workers (`--jobs N`), including Rails apps on Postgres and SQLite
+  with a database copy per worker.
+- RSpec adapter and a Rails profile. Minitest is out of scope.
+- Terminal report with unified diffs, Stryker JSON report, ignore list with
+  mandatory reasons, incremental result cache.
+- Mutation-source extension point and lifecycle hooks.
+
+## [0.0.1] - unreleased
+
+### Added
+- Gem skeleton to reserve the name. No mutation testing yet.

data/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Vamsi Pavan Mahesh Gunturu
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,145 @@
+# saboteur
+
+Mutation testing for Ruby 3.3+.
+
+saboteur makes small deliberate changes to your methods (mutants), runs only
+the tests that execute each change, and reports the changes no test notices
+(survivors). Each survivor points at a missing test or at code that does not
+matter.
+
+Status: an early release (0.2.0). Tested on the RSpec suites of plain gems
+(dry-inflector, tty-color, hashie, addressable) and of two Rails 8.1 apps:
+human-essentials on Postgres (4,080 mutants, 6 workers) and lobsters on
+SQLite, both on macOS. Not yet tried on MySQL or older Rails versions. RSpec
+only, by design.
+
+## What makes it different
+
+- **Tests picked per mutant by line coverage.** Each mutant runs only the
+  tests that executed the changed statement. A change no test reaches is
+  reported as "no coverage", not as a survivor.
+- **Likeliest killers first.** Tests that killed earlier mutants of the same
+  method run first, then the fastest; the first failure stops the run.
+- **Coarse first.** Each method's body is first replaced with `nil` and with
+  `raise`. If a test suite does not notice that, saboteur reports it and
+  skips the fine-grained mutants for that method.
+- **Diff mode.** `--since origin/main` mutates only methods changed on a branch.
+- **Safe to run.** Every mutant runs in a supervised child; anything it
+  starts is tracked and killed, with deadline, memory and process caps.
+- **Open report format.** JSON in the Stryker mutation-testing-report schema.
+- **Pluggable mutation sources.** Add your own mutants by file and byte range.
+
+## Usage
+
+```sh
+bundle exec saboteur run                          # whole project
+bundle exec saboteur run --since origin/main      # methods changed on this branch
+bundle exec saboteur run --subject 'Billing::Invoice#total'
+bundle exec saboteur run --subject Billing        # a namespace
+bundle exec saboteur run --incremental            # reuse verdicts whose inputs did not change
+bundle exec saboteur run --seed 1234              # repeat a run that used seed 1234
+```
+
+Every test run starts from the same random seed (1 by default), so tests
+that use `rand` see the same values in each mutant's run and in the check
+that confirms a kill. The report prints the seed; pass it to `--seed` to
+repeat a run.
+
+`--incremental` reuses an earlier verdict only while the method, the mutant,
+its covering tests and their files, the lockfile and the config are all
+unchanged. It cannot see a change in a method this one calls, so run without
+it now and then (and in CI on the main branch).
+
+Output: counts per status, the score, timing, and a unified diff for every
+survivor. The JSON report goes to `.saboteur/report.json`.
+
+Exit codes: 0 no survivors (or all on the ignore list), 1 survivors,
+2 bad configuration, 3 the tool failed.
+
+## Configuration
+
+`.saboteur.yml` in the project root; CLI flags override it.
+
+```yaml
+include: [lib/**/*.rb]
+exclude: [lib/my_gem/version.rb]
+require: []                 # boot files, e.g. ./config/environment
+reeval: nesting             # or class_eval; see docs/design.md 6.1
+timeout_factor: 3           # per-mutant deadline = factor x its tests' recorded time
+timeout_min: 5
+max_memory_mb: 2048         # memory growth cap per mutant process tree
+max_processes: 64
+ignore_file: .saboteur-ignore.yml
+seed: 1                     # Ruby's random seed in every test run
+```
+
+### Ignoring a survivor
+
+Every survivor is printed with a digest. Accept it with a reason:
+
+```yaml
+# .saboteur-ignore.yml
+- subject: "Billing::Invoice#total"
+  mutation: "1a2b3c4d5e6f7a8b"
+  reason: "equivalent: amounts are always whole cents"
+```
+
+An entry without a reason is rejected.
+
+### SimpleCov
+
+saboteur needs Ruby's `Coverage` for itself. Disable SimpleCov for the run
+(most projects already guard it with an environment variable).
+
+## Extending
+
+Public API, stable across minor versions:
+
+- `Saboteur::MutationSource::Base#mutations_for(subject)` returning
+  `Saboteur::MutationSource::Mutation` objects made of `Edit`s (file, byte
+  range, replacement). Register with `sources: [builtin, path.rb:ClassName]`.
+- `Saboteur::Integration::Base` for other test frameworks (RSpec is the
+  only one that ships).
+- `Saboteur.hook(:before_boot | :after_boot | :before_fork | :after_fork |
+  :around_mutant)` in files listed under `hooks:`.
+
+## How it works
+
+See [docs/design.md](docs/design.md). Validation results are in
+[docs/validation](docs/validation), research in [docs/research](docs/research),
+and the project checklist in [docs/status.md](docs/status.md).
+
+## Benchmark
+
+A small, non-exhaustive comparison with [mutineer](https://github.com/davidteren/mutineer)
+1.6.0 on four RSpec projects (two libraries, two Rails apps). Each configuration
+ran once on sampled files, so treat the numbers as indicative.
+
+| Metric | saboteur | mutineer |
+|---|---|---|
+| Fake kills (lobsters) | 0 of 20 checked | 8 of 8 checked |
+| Rails app with 6 workers | yes, same verdicts as 1 worker | 1 worker only |
+| Runs on the Postgres app | yes | no |
+| Time per mutant, hashie | 0.008 s | 0.092 s |
+| Peak memory, hashie | 595 MB | 471 MB |
+| Choice of operators, CI features | fixed set, fewer | 20 operators, baseline, HTML, GitHub Action |
+
+Full results, method and limitations:
+[benchmark page](https://vamsipavanmahesh.github.io/saboteur/benchmark/).
+Scripts and raw data: [bench/vs-mutineer](bench/vs-mutineer).
+
+## Requirements
+
+Ruby 3.3 or newer on Linux or macOS (it needs `fork`).
+
+## Contributing
+
+See [CONTRIBUTING.md](CONTRIBUTING.md). Commits need a DCO `Signed-off-by` line.
+
+## Acknowledgements
+
+Thanks to [Glomo](https://glomo.one) for providing the AI tokens used to build saboteur.
+
+## License
+
+MIT. Copyright (c) 2026 Vamsi Pavan Mahesh Gunturu. See [LICENSE](LICENSE).

data/exe/saboteur

@@ -0,0 +1,12 @@
+#!/usr/bin/env ruby
+# SPDX-License-Identifier: MIT
+# frozen_string_literal: true
+
+# Line coverage must be set up before any target code loads, so it happens
+# first. It stays suspended (no counting) except while the coverage map is built.
+require "coverage"
+Coverage.setup(lines: true) if Coverage.state == :idle
+
+require "saboteur/cli"
+
+exit Saboteur::CLI.new(ARGV).call

data/lib/saboteur/cli.rb

@@ -0,0 +1,112 @@
+# SPDX-License-Identifier: MIT
+# frozen_string_literal: true
+
+require "optparse"
+require_relative "../saboteur"
+
+module Saboteur
+  class CLI
+    USAGE = "Usage: saboteur run [--since REF] [--subject 'Klass#meth'] [--timeout-factor N] [--jobs N] [--seed N]"
+
+    EXIT_OK = 0
+    EXIT_SURVIVORS = 1
+    EXIT_USAGE = 2
+    EXIT_TOOL = 3
+
+    class UsageError < ConfigError; end
+
+    attr_reader :options
+
+    def initialize(argv, out: $stdout, err: $stderr)
+      @argv = argv.dup
+      @out = out
+      @err = err
+      @options = {}
+    end
+
+    def call
+      case @argv.shift
+      when "run" then run
+      when "version", "--version" then @out.puts(VERSION) || EXIT_OK
+      else usage_error
+      end
+    rescue OptionParser::ParseError => e
+      usage_error(e.message)
+    end
+
+    def config
+      @config ||= Config.load(root: Dir.pwd, path: options[:config], overrides: overrides)
+    end
+
+    private
+
+    def usage_error(message = nil)
+      @err.puts(message) if message
+      @err.puts(USAGE)
+      EXIT_USAGE
+    end
+
+    def run
+      parser.parse!(@argv)
+      raise UsageError, "unexpected arguments: #{@argv.join(' ')}" if @argv.any?
+
+      report = Engine.new(config, err: @err).call
+      @err.puts
+      Reporter::Terminal.new(@out).call(report)
+      write_json(report)
+      exit_code(report)
+    rescue ConfigError => e
+      @err.puts("saboteur: #{e.message}")
+      EXIT_USAGE
+    rescue Error => e
+      @err.puts("saboteur: #{e.message}")
+      EXIT_TOOL
+    end
+
+    def write_json(report)
+      path = config.path("json_report")
+      return unless path
+
+      Reporter::Stryker.new(path).call(report)
+      @out.puts "\nJSON report: #{path}"
+    end
+
+    def exit_code(report)
+      executed = report.executed
+      return EXIT_TOOL if executed.any? && executed.all? { |result| result.status == :error }
+
+      report.unignored_survivors.any? ? EXIT_SURVIVORS : EXIT_OK
+    end
+
+    def overrides
+      options.except(:config)
+    end
+
+    def parser
+      OptionParser.new do |opts|
+        opts.banner = USAGE
+        opts.on("--since REF", "Only mutate subjects changed since git REF") { |v| options[:since] = v }
+        opts.on("--subject EXPR", "Klass#meth, Klass.meth or a namespace; repeatable") { |v| add_subject(v) }
+        opts.on("--timeout-factor N", Float, "Multiple of baseline runtime (default 3)") { |v| set(:timeout_factor, v) }
+        opts.on("--jobs N", Integer, "Concurrent workers (default 1)") { |v| set(:jobs, v) }
+        opts.on("--seed N", Integer, "Seed for Ruby's random numbers in every test run (default 1)") do |v|
+          set(:seed, v)
+        end
+        opts.on("--config PATH", "Config file (default .saboteur.yml)") { |v| set(:config, v) }
+        opts.on("--integration NAME", "Test adapter (rspec)") { |v| set(:integration, v) }
+        opts.on("--reeval STRATEGY", Execution::Reevaluator::STRATEGIES, "class_eval or nesting") { |v| set(:reeval, v) }
+        opts.on("--json PATH", "Stryker JSON report path") { |v| set(:json_report, v) }
+        opts.on("--no-cache", "Rebuild the coverage map") { set(:cache, false) }
+        opts.on("--incremental", "Reuse earlier verdicts whose inputs did not change") { set(:incremental, true) }
+      end
+    end
+
+    def add_subject(expr)
+      (options[:subjects] ||= []) << expr
+    end
+
+    def set(key, value)
+      options[key] = value
+    end
+  end
+end

data/lib/saboteur/config.rb

@@ -0,0 +1,133 @@
+# SPDX-License-Identifier: MIT
+# frozen_string_literal: true
+
+require "yaml"
+require_relative "mutation_source/builtin"
+
+module Saboteur
+  # Settings from .saboteur.yml, overridden by CLI flags.
+  class Config
+    FILE = ".saboteur.yml"
+
+    DEFAULTS = {
+      "integration" => "rspec",
+      "require" => [],
+      "include" => ["lib/**/*.rb"],
+      "exclude" => [],
+      "arid" => MutationSource::Builtin::DEFAULT_ARID,
+      "timeout_factor" => 3.0,
+      "timeout_min" => 5.0,
+      "baseline_timeout" => 300.0,
+      "coverage_timeout" => 3600.0,
+      "max_memory_mb" => 2048,
+      "max_processes" => 64,
+      "jobs" => 1,
+      "ignore_file" => ".saboteur-ignore.yml",
+      "sources" => ["builtin"],
+      "hooks" => [],
+      "rails" => false,
+      "reeval" => "nesting",
+      "cache_dir" => ".saboteur",
+      "json_report" => ".saboteur/report.json",
+      "rspec_args" => [],
+      "subjects" => [],
+      "since" => nil,
+      "cache" => true,
+      "incremental" => false,
+      "seed" => 1
+    }.freeze
+
+    INTEGER_KEYS = %w[jobs max_memory_mb max_processes seed].freeze
+    FLOAT_KEYS = %w[timeout_factor timeout_min baseline_timeout coverage_timeout].freeze
+    LIST_KEYS = %w[require include exclude arid sources hooks rspec_args subjects].freeze
+
+    attr_reader :root
+
+    def self.load(root: Dir.pwd, path: nil, overrides: {})
+      path ||= File.join(root, FILE)
+      from_file = File.file?(path) ? parse(path) : {}
+      new(from_file.merge(stringify(overrides)), root: root)
+    end
+
+    def self.parse(path)
+      data = YAML.safe_load_file(path) || {}
+      raise ConfigError, "#{path}: expected a mapping at the top level" unless data.is_a?(Hash)
+
+      data
+    rescue Psych::SyntaxError => e
+      raise ConfigError, "#{path}: #{e.message}"
+    end
+
+    def self.stringify(hash)
+      hash.to_h { |key, value| [key.to_s, value] }.compact
+    end
+
+    def initialize(values = {}, root: Dir.pwd)
+      @root = File.expand_path(root)
+      unknown = values.keys - DEFAULTS.keys
+      raise ConfigError, "unknown config key(s): #{unknown.join(', ')}" if unknown.any?
+
+      @values = DEFAULTS.merge(values)
+      coerce!
+      validate!
+    end
+
+    def [](key)
+      @values.fetch(key.to_s)
+    end
+
+    def fetch(key, default = nil)
+      @values.fetch(key.to_s, default)
+    end
+
+    def to_h
+      @values.dup
+    end
+
+    # Ruby files to mutate, as absolute real paths.
+    def included_files
+      matched = self["include"].flat_map { |pattern| Dir.glob(pattern, base: root) }
+      excluded = self["exclude"].flat_map { |pattern| Dir.glob(pattern, base: root) }
+      (matched - excluded).select { |file| file.end_with?(".rb") }
+                          .map { |file| File.realpath(File.join(root, file)) }.uniq.sort
+    end
+
+    # Patterns that match no file. A pattern goes stale when files move;
+    # an exclude that silently stops excluding is a hazard, so say so.
+    def unmatched_patterns
+      %w[include exclude].flat_map do |key|
+        self[key].select { |pattern| Dir.glob(pattern, base: root).empty? }.map { |pattern| "#{key}: #{pattern}" }
+      end
+    end
+
+    def path(key)
+      value = self[key]
+      value && File.expand_path(value, root)
+    end
+
+    private
+
+    def coerce!
+      INTEGER_KEYS.each { |key| @values[key] = Integer(@values[key]) }
+      FLOAT_KEYS.each { |key| @values[key] = Float(@values[key]) }
+      LIST_KEYS.each { |key| @values[key] = Array(@values[key]).map(&:to_s) }
+    rescue ArgumentError, TypeError => e
+      raise ConfigError, "invalid config value: #{e.message}"
+    end
+
+    def validate!
+      raise ConfigError, "jobs must be >= 1" if self["jobs"] < 1
+      raise ConfigError, "seed must be >= 0" if self["seed"].negative?
+      raise ConfigError, "max_memory_mb must be > 0" unless self["max_memory_mb"].positive?
+      raise ConfigError, "max_processes must be > 0" unless self["max_processes"].positive?
+      raise ConfigError, "timeout_factor must be > 0" unless self["timeout_factor"].positive?
+      raise ConfigError, "timeout_min must be > 0" unless self["timeout_min"].positive?
+      raise ConfigError, "reeval must be one of: #{Execution::Reevaluator::STRATEGIES.join(', ')}" unless
+        Execution::Reevaluator::STRATEGIES.include?(self["reeval"].to_s)
+
+      self["arid"].each { |pattern| Regexp.new(pattern) }
+    rescue RegexpError => e
+      raise ConfigError, "invalid arid pattern: #{e.message}"
+    end
+  end
+end

data/lib/saboteur/engine/boot.rb

@@ -0,0 +1,83 @@
+# SPDX-License-Identifier: MIT
+# frozen_string_literal: true
+
+module Saboteur
+  class Engine
+    # Loads the target once in the parent: hook files, the optional Rails
+    # profile, configured boot files, then the test adapter (which loads the
+    # test files without running any test). Returns the ready adapter.
+    class Boot
+      # A test suite that (perhaps only under a mutant) starts a saboteur run
+      # would otherwise run saboteur inside a saboteur child, recursively.
+      def self.refuse_recursion!
+        return unless ENV[Execution::Supervisor::CHILD_ENV]
+
+        raise Error, "refusing to start: this process is a saboteur child " \
+                     "(#{Execution::Supervisor::CHILD_ENV} is set). A test is running saboteur itself; stub it."
+      end
+
+      # Prepended to SimpleCov's singleton: saboteur needs Ruby's Coverage for
+      # its own map, and Coverage can only be set up once per process.
+      module SimpleCovOff
+        def start(*, **)
+          nil
+        end
+      end
+
+      # Watches for the app's SimpleCov and turns its start into a no-op the
+      # moment the constant is defined, before any line of the app's
+      # SimpleCov setup runs. saboteur does not load SimpleCov itself:
+      # loading it runs the app's .simplecov at once.
+      module SimpleCovWatch
+        def const_added(name)
+          super
+          Boot.silence_simplecov if name == :SimpleCov && equal?(Object)
+        end
+      end
+
+      def self.watch_for_simplecov
+        Object.singleton_class.prepend(SimpleCovWatch) unless Object.singleton_class.include?(SimpleCovWatch)
+        silence_simplecov if defined?(::SimpleCov)
+      end
+
+      def self.silence_simplecov
+        return if ::SimpleCov.singleton_class.include?(SimpleCovOff)
+
+        ::SimpleCov.singleton_class.prepend(SimpleCovOff)
+        warn "saboteur: SimpleCov.start ignored for this run (saboteur needs Ruby's Coverage itself)"
+      end
+
+      # The Rails profile when rails: true, else nil (set by load_target).
+      attr_reader :rails
+
+      def initialize(config, hooks)
+        @config = config
+        @hooks = hooks
+        @root = File.realpath(config.root)
+      end
+
+      def load_target
+        self.class.watch_for_simplecov
+        load_hooks
+        @rails = rails = Frameworks::Rails.new(@config, @hooks) if @config["rails"]
+        rails&.before_boot
+        @hooks.run(:before_boot)
+        @config["require"].each { |file| require File.expand_path(file, @root) }
+        integration = Integration.fetch(@config["integration"]).new(@config)
+        integration.setup
+        rails&.after_boot
+        @hooks.run(:after_boot)
+        integration
+      end
+
+      private
+
+      def load_hooks
+        Saboteur.current_hooks = @hooks
+        @config["hooks"].each { |file| load File.expand_path(file, @root) }
+      ensure
+        Saboteur.current_hooks = nil
+      end
+    end
+  end
+end

data/lib/saboteur/engine/candidates.rb

@@ -0,0 +1,63 @@
+# SPDX-License-Identifier: MIT
+# frozen_string_literal: true
+
+require "prism"
+
+module Saboteur
+  class Engine
+    # Builds the mutation sources from config and turns their proposals for
+    # a subject into runnable candidates: applied, re-parsed, de-duplicated,
+    # coarse first. Returns [[mutation, mutated_text]].
+    class Candidates
+      def self.build_sources(config)
+        root = File.realpath(config.root)
+        config["sources"].map do |entry|
+          next MutationSource::Builtin.new(arid: config["arid"]) if entry == "builtin"
+
+          file, const = entry.split(":", 2)
+          raise ConfigError, "mutation source #{entry.inspect} must be builtin or path.rb:ClassName" unless const
+
+          require File.expand_path(file, root)
+          Object.const_get(const).new
+        end
+      end
+
+      def initialize(sources, diagnostics)
+        @sources = sources
+        @diagnostics = diagnostics
+      end
+
+      def for(subject)
+        return [] unless parses?(subject.source)
+
+        seen = Set[subject.source]
+        list = @sources.flat_map { |source| source.mutations_for(subject) }.filter_map do |mutation|
+          candidate(subject, mutation, seen)
+        end
+        list.sort_by.with_index { |(mutation, _), index| [mutation.coarse ? 0 : 1, index] }
+      end
+
+      private
+
+      def candidate(subject, mutation, seen)
+        text = subject.apply(mutation.edits)
+        return if seen.include?(text)
+
+        seen << text
+        unless parses?(text)
+          @diagnostics.count_invalid_mutation
+          return
+        end
+
+        [mutation, text]
+      rescue ArgumentError => e
+        @diagnostics.warn("#{subject.id}: dropped #{mutation.source} mutation: #{e.message}")
+        nil
+      end
+
+      def parses?(text)
+        Prism.parse(text).success?
+      end
+    end
+  end
+end