mutato 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 2aafb54276c4ae2139a14b9422b16ffad91c363f501aa079f1a6579e74031509
+  data.tar.gz: 30bf198b8033421c66222b25653c8645d6a84fc6bc46cc02f7fb4929111af55e
+SHA512:
+  metadata.gz: d01b47493ef62b0df820899c0f92de46d4f93eaee537dcdb232d21db995954b985e98099a3437bc2a0ea45387f7f126c88768a19dd60ff64392c0444e4e47f6a
+  data.tar.gz: fb3270a1f902dc9752670d5ecfe6a04c4879186ea0466c9e03cfdcdf2f767baeae4d85a7dcc72a97dee276baa78455083771618e6e5b3b5169b02c7a83b38dba

data/CHANGELOG.md

@@ -0,0 +1,13 @@
+# Changelog
+
+Notable changes for users of mutato. The format follows
+[Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and versions follow
+[Semantic Versioning](https://semver.org/).
+
+## [Unreleased]
+
+## [0.1.0] - 2026-10-03
+
+### Added
+
+- First release: mutation testing for RSpec, on Ruby 3.2 and later.

data/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Mathias Løkke Madsen
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,218 @@
+<p align="center"><img src=".github/mutato.svg" width="150" alt=""></p>
+
+# mutato
+
+[![CI](https://github.com/LQkkeN/mutato/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/LQkkeN/mutato/actions/workflows/ci.yml)
+
+Mutation testing for Ruby and RSpec. mutato changes your code one small step
+at a time and runs the tests that cover each change. A change that no test
+notices points at a missing test.
+
+## Install
+
+```ruby
+# Gemfile
+group :test do
+  gem 'mutato'
+end
+```
+
+Ruby 3.2 or later.
+
+## Find what your tests miss
+
+Take this class, with specs for VAT on 100 and for a discount of 20:
+
+```ruby
+# lib/shop/price.rb
+module Shop
+  class Price
+    def initialize(amount, vat: 0.25)
+      @amount = amount
+      @vat = vat
+    end
+
+    def total(discount: 0)
+      net = [@amount - discount, 0].max
+      (net * (1 + @vat)).round(2)
+    end
+  end
+end
+```
+
+```
+$ bundle exec mutato run lib
+14 mutants over 1 files, 2 methods
+...
+    7/14 caught      0.02s da2574057ce7  lib/shop/price.rb:9:14: [element] drop `@amount - discount`
+    8/14 missed      0.01s 4c1d0b45c1fc  lib/shop/price.rb:9:34: [element] drop `0`
+...
+results: caught 11, missed 3
+
+MISSED:
+  4c1d0b45c1fc  lib/shop/price.rb:9:34: [element] drop `0`
+  5315afcd609d  lib/shop/price.rb:10:26: [swap-method] replace `.round` with `.floor`
+  efc60b287f00  lib/shop/price.rb:10:26: [swap-method] replace `.round` with `.ceil`
+```
+
+Each missed mutant is a change no test noticed: no spec gives a discount
+larger than the amount, and none checks rounding up and down. Add those two
+specs, then rerun just these mutants by id:
+
+```
+$ bundle exec mutato run lib --only 4c1d0b45c1fc,5315afcd609d,efc60b287f00
+...
+     1/3 caught      0.03s 4c1d0b45c1fc  lib/shop/price.rb:9:34: [element] drop `0`
+     2/3 caught      0.03s 5315afcd609d  lib/shop/price.rb:10:26: [swap-method] replace `.round` with `.floor`
+     3/3 caught      0.02s efc60b287f00  lib/shop/price.rb:10:26: [swap-method] replace `.round` with `.ceil`
+results: caught 3
+```
+
+The exit code is 2 while anything is missed. An id stays the same while its
+method is unchanged and the path is written the same way (`lib` and `./lib`
+differ).
+
+## See what would be tried
+
+```
+$ bundle exec mutato list lib
+27f4e8ead8ca  lib/shop/price.rb:4:7: [value] replace body of Shop::Price#initialize with nil  (Shop::Price#initialize)
+f7bfadab08e0  lib/shop/price.rb:4:7: [statement] delete statement `@amount = amount`  (Shop::Price#initialize)
+...
+14 mutants over 1 files, 2 methods
+```
+
+## Only what a branch changed
+
+```sh
+git diff -U0 origin/main | bundle exec mutato run lib --diff -
+hg diff -U0 -r default | bundle exec mutato run lib --diff -
+jj diff --git | bundle exec mutato run lib --diff -
+bundle exec mutato run lib --diff changes.diff
+```
+
+Any unified diff works; mutato never calls a version control system itself.
+It reads the diff's paths from the current directory, so from a subdirectory
+of the repository use `git diff --relative`.
+
+## On a GitHub pull request
+
+```yaml
+- uses: actions/checkout@v7
+  with:
+    fetch-depth: 0     # git diff needs the base commit
+- run: git diff -U0 ${{ github.event.pull_request.base.sha }} |
+       bundle exec mutato run lib --diff - --format github
+  shell: bash          # pipefail: a failing git diff fails the step
+```
+
+Survivors show as warnings on the changed lines, timeouts as notices, and a
+table of survivors goes into the step summary. Elsewhere, `--format plain`
+prints lines that editors and CI systems parse:
+
+```
+lib/shop/price.rb:9:34: survived: drop `0`
+```
+
+## Choose the specs
+
+```sh
+bundle exec mutato run lib --spec spec/unit                        # default: spec
+bundle exec mutato run app lib --spec 'spec --tag ~slow'           # any rspec arguments
+bundle exec mutato run lib --spec "test --pattern '**/*_test.rb'"  # files not named _spec.rb
+```
+
+## Start small on a big codebase
+
+```sh
+bundle exec mutato run lib/billing                        # one directory or file
+bundle exec mutato run lib --sample 50                    # 50 at random, the same 50 every time
+bundle exec mutato run lib --limit 20                     # the first 20 in file order
+bundle exec mutato run lib --genre arithmetic,relational  # only some kinds of change
+```
+
+## Check the setup
+
+```
+$ bundle exec mutato control lib --sample 40
+...
+control OK: 2 of 2 reinstalled methods passed their tests
+```
+
+`control` puts methods back unchanged and runs their tests. All must pass: a
+method whose tests fail even then gives meaningless results in a real run.
+
+## Project hooks
+
+`.mutato.rb` in the project root, or `--config FILE`:
+
+```ruby
+# Tables cleaned in after hooks: a killed child leaves rows behind.
+Mutato.before_mutant { truncate_tables }
+
+# Rails loads lazily: load everything once, or untested files come out unloaded.
+Mutato.after_boot { Rails.application.eager_load! }
+
+# A socket must not be shared with the forked children.
+Mutato.before_fork { SomeClient.disconnect }
+Mutato.after_fork { SomeClient.connect }
+
+# Accept a known survivor, with the reason on record; `mutato list` shows it.
+Mutato.skip 'Accounts#obtain', 'lock re-check for a race no test reproduces'
+
+# The project's own output calls, never worth mutating.
+Mutato.arid :say, 'UI'
+```
+
+Sequel and ActiveRecord connections are closed before every fork already.
+Other options: `--out DIR` (default `mutato.out`), `--timeout-min SECONDS`
+(default 10), `--version`, `--help`.
+
+## Reading the results
+
+`mutato.out/missed.txt` is the list to act on. Every other outcome has a list
+too, `outcomes.json` holds all of them, and `logs/` has one log per mutant
+with whatever the tests printed, capped at 1 MB.
+
+| outcome | meaning |
+|---|---|
+| caught | a test failed, and passes without the mutant |
+| missed | no test failed: a test is missing |
+| timeout | the tests ran past five times their usual time, often an infinite loop |
+| unjudged | only flaky tests cover it |
+| equivalent | provably no change: a removed `super` into an `initialize` that does nothing |
+| uncovered | no test runs the line |
+| load-time | the code runs only while the suite loads |
+| unloaded | the suite never loads the file |
+| overwritten | the tests redefined the method, as a code reloader does |
+| unviable | the mutant did not load |
+
+Exit codes: 2 when a mutant is missed or unjudged, 3 when there is nothing to
+mutate (0 with `--diff`), 1 on errors, otherwise 0. `control` exits 1 when a
+method fails and 3 when no test ran any.
+
+## What it changes
+
+| genre | for example |
+|---|---|
+| statement | delete `@vat = vat` |
+| condition | `if paid?` -> `if true`, `a && b` -> `a \|\| b`, `!x` -> `x` |
+| relational | `a < b` -> `a <= b`, `a != b`, `false` |
+| arithmetic | `a + b` -> `a - b`, `x += 1` -> `x -= 1`, `a - b` -> `b - a` |
+| swap-method | `.round` -> `.floor`, `.max` -> `.min`, `.any?` -> `.all?` |
+| value | a method's body -> `nil`, `[]`, `0`, `true` or `false` |
+| element | `[net, 0]` -> `[net]` |
+
+It leaves alone what no test could tell apart or should check: logging and
+output, variables nothing reads, `return @x if @x`, and code outside methods.
+`mutato list` names each method it skips, with the reason.
+
+## Limits
+
+RSpec only, and one worker: a run takes as long as the tests it runs. An
+endless `def x = ...` comes out uncovered, and native extensions are
+invisible. A mutant can reach code the tests never did, such as the network.
+
+## Licence
+
+MIT.

data/exe/mutato

@@ -0,0 +1,21 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+# Before mutato's own files load, so that it can run on itself.
+require "coverage"
+Coverage.start(lines: true) unless Coverage.running?
+require "bundler/setup" if File.exist?("Gemfile")
+begin
+  require_relative "../lib/mutato"
+rescue LoadError => error
+  hint = "Add mutato to the project's Gemfile; its dependency brings the prism it needs."
+  abort "mutato: #{error.message}. #{hint}"
+end
+if Gem::Version.new(Prism::VERSION) < Gem::Version.new("1.8")
+  abort "mutato: prism #{Prism::VERSION} is too old, 1.8 is needed"
+end
+require_relative "../lib/mutato/cli"
+
+code = Mutato::CLI.call(ARGV)
+# exit!, not exit: the project's at_exit handlers must not run.
+exit! code

data/lib/mutato/abort.rb

@@ -0,0 +1,12 @@
+# frozen_string_literal: true
+
+module Mutato
+  class Abort < StandardError
+    attr_reader :code
+
+    def initialize(message, code:)
+      super(message)
+      @code = code
+    end
+  end
+end

data/lib/mutato/annotations.rb

@@ -0,0 +1,65 @@
+# frozen_string_literal: true
+
+module Mutato
+  class Annotations
+    FORMATS = { "github" => :github, "plain" => :plain }.freeze
+    SUMMARY = <<~MARKDOWN
+      ## mutato: %<count>d survivors
+
+      | where | mutant |
+      |---|---|
+      %<rows>s
+    MARKDOWN
+    private_constant :FORMATS, :SUMMARY
+
+    def self.command(level, verdict, outcome)
+      file, line, col, description = outcome.fetch_values(:file, :line, :col, :description)
+      "::#{level} file=#{file},line=#{line},col=#{col},title=mutato::#{verdict}: #{description}"
+    end
+
+    def self.line(outcome)
+      file, line, col, description = outcome.fetch_values(:file, :line, :col, :description)
+      "#{file}:#{line}:#{col}: survived: #{description}"
+    end
+
+    def self.step_summary(missed)
+      path = ENV.fetch("GITHUB_STEP_SUMMARY", nil)
+      return unless path && missed.any?
+
+      rows = missed.map { |outcome| row(*outcome.fetch_values(:file, :line, :description)) }
+      File.write(path, format(SUMMARY, count: missed.size, rows: rows.join("\n")), mode: "a")
+    end
+
+    # GitHub splits table cells on every pipe, inside backticks too, unless escaped.
+    def self.row(file, line, description)
+      "| `#{file}:#{line}` | #{description.gsub("|", "\\|")} |"
+    end
+
+    def initialize(console)
+      @console = console
+    end
+
+    def show(outcomes, format)
+      style = FORMATS[format]
+      public_send(style, outcomes) if style
+    end
+
+    def github(outcomes)
+      missed = Mode.missed(outcomes)
+      missed.each { |outcome| @console.out(Annotations.command("warning", "survived", outcome)) }
+      notices(outcomes)
+      Annotations.step_summary(missed)
+    end
+
+    def plain(outcomes)
+      Mode.missed(outcomes).each { |outcome| @console.out(Annotations.line(outcome)) }
+    end
+
+    private
+
+    def notices(outcomes)
+      timeouts = outcomes.select { |outcome| outcome.fetch(:outcome) == :timeout }
+      timeouts.each { |outcome| @console.out(Annotations.command("notice", "timeout", outcome)) }
+    end
+  end
+end

data/lib/mutato/arid.rb

@@ -0,0 +1,104 @@
+# frozen_string_literal: true
+
+require "prism"
+
+module Mutato
+  # Code no test asserts on (output, logging): mutating it only makes survivors.
+  module Arid
+    METHODS = %i[puts warn p pp print require require_relative].freeze
+    LOGGING = /\Alog(?:ger)?\z/i
+    # `log_error(e)`, `logger_info`: a level after the underscore, so `log_in` is not one.
+    HELPER = /\Alog(?:ger)?(?:_(?:debug|info|warn|warning|error|fatal|exception|event|message))?\z/i
+    VARIABLES = [
+      Prism::InstanceVariableReadNode,
+      Prism::ClassVariableReadNode,
+      Prism::GlobalVariableReadNode,
+      Prism::LocalVariableReadNode
+    ].freeze
+    ITERATORS = %i[
+      each
+      each_with_index
+      each_with_object
+      each_pair
+      each_key
+      each_value
+      reverse_each
+      times
+      upto
+      downto
+      step
+    ].freeze
+    private_constant :METHODS, :LOGGING, :HELPER, :VARIABLES, :ITERATORS
+
+    module_function
+
+    def helper_name?(name)
+      name.to_s.match?(HELPER)
+    end
+
+    # A compound of arid parts is arid, and so is one with none: `if x; end`.
+    def arid_node?(node)
+      case node
+      when Prism::CallNode then arid_with_block?(node) || arid_loop?(node)
+      when Prism::IfNode, Prism::UnlessNode then all_arid?(branches_of(node))
+      when Prism::StatementsNode, Prism::ElseNode then all_arid?(node.compact_child_nodes)
+      else false
+      end
+    end
+
+    def all_arid?(parts)
+      parts.all? { |part| arid_node?(part) }
+    end
+
+    # A logging call with a block is arid only if its block is: `logger.tagged { work }` works.
+    def arid_with_block?(call)
+      block = call.block
+      body = block.body if block.is_a?(Prism::BlockNode)
+      arid_call?(call) && all_arid?([body].compact)
+    end
+
+    def arid_loop?(call)
+      block = call.block
+      ITERATORS.include?(call.name) && block.is_a?(Prism::BlockNode) && arid_node?(block.body)
+    end
+
+    def branches_of(conditional)
+      conditional.compact_child_nodes - [conditional.predicate]
+    end
+
+    def arid_call?(call)
+      name = call.name
+      receiver = call.receiver
+      return true if METHODS.include?(name) && console?(receiver)
+
+      Mutato.config.arid?(name) || (receiver.nil? && helper_name?(name)) || logging?(receiver)
+    end
+
+    # `puts` to the console, not to an IO the code was handed.
+    def console?(receiver)
+      receiver.nil? || receiver.slice.match?(/\A(?:\$std(?:out|err)|STD(?:OUT|ERR))\z/)
+    end
+
+    def logging?(receiver)
+      case receiver
+      when Prism::CallNode then logging_call?(receiver)
+      when Prism::ConstantReadNode, Prism::ConstantPathNode then logging_constant?(receiver.slice)
+      when *VARIABLES then receiver.name.to_s.sub(/\A(?:@@?|\$)/, "").match?(LOGGING)
+      else false
+      end
+    end
+
+    # `App.logger`, a reader without arguments (not `Math.log(x)`), or `opts[:logger]`.
+    def logging_call?(call)
+      arguments = Nodes.arguments_of(call)
+      name = call.name
+      return name.to_s.match?(LOGGING) && arguments.empty? unless name == :[]
+
+      (arguments in [key]) && key.slice.match?(/\A[:"']?log(?:ger)?["']?\z/i)
+    end
+
+    def logging_constant?(name)
+      name.split("::").last.match?(LOGGING) || Mutato.config.arid?(name)
+    end
+  end
+end

data/lib/mutato/baseline.rb

@@ -0,0 +1,51 @@
+# frozen_string_literal: true
+
+module Mutato
+  # hooks: the seconds the suite and group hooks took, which every run repeats.
+  Baseline = Data.define(
+    :line_map,
+    :durations,
+    :order,
+    :hooks,
+    :locations,
+    :boot_lines,
+    :loaded_files
+  ) do
+    # `lib/calc/helpers.rb` is tested by `helpers_spec.rb` or `calc_helpers_spec.rb`.
+    def self.stems_of(file)
+      stems = [File.basename(file, ".rb"), file.delete_suffix(".rb").split("/").drop(1).join("_")]
+      stems.reject(&:empty?)
+    end
+
+    def covering(path, lines)
+      ids = lines.flat_map { |line| line_map.fetch("#{path}:#{line}", []) }
+      ids.uniq
+    end
+
+    def budget(ids, minimum)
+      [minimum, ids.sum { |id| durations.fetch(id) } * 5].max + hooks
+    end
+
+    def loaded?(path)
+      loaded_files.include?(path)
+    end
+
+    def booted?(path, lines)
+      lines.any? { |line| boot_lines.include?("#{path}:#{line}") }
+    end
+
+    # Named tests kill soonest; baseline order keeps order-dependent suites working.
+    def prioritize(ids, file)
+      stems = Baseline.stems_of(file)
+      ids.sort_by { |id| [named_after?(id, stems) ? 0 : 1, order.fetch(id, Float::INFINITY)] }
+    end
+
+    private
+
+    def named_after?(id, stems)
+      base = File.basename(locations.fetch(id))
+      stems.any? { |stem| base.start_with?("#{stem}_", "test_#{stem}") }
+    end
+  end
+  public_constant :Baseline
+end

data/lib/mutato/boot.rb

@@ -0,0 +1,99 @@
+# frozen_string_literal: true
+
+require "coverage"
+
+module Mutato
+  class Boot
+    def initialize(selection, console)
+      @selection = selection
+      @console = console
+      @output = nil
+    end
+
+    def suite
+      adapter = booted
+      # What booting executed, before any test ran.
+      executed = Coverage.peek_result
+      started = Clock.now
+      measured = describe(measure(adapter), started)
+      Suite.new(adapter:, baseline: baseline(measured, executed), options:, output:)
+    end
+
+    private
+
+    def options
+      @selection.options
+    end
+
+    def output
+      @output ||= Output.new(options.out)
+    end
+
+    def booted
+      started = Clock.now
+      RSpecAdapter.boot(options.spec_args)
+      Mutato.config.run_hooks(:after_boot)
+      report_boot(started)
+    end
+
+    def report_boot(started)
+      count = RSpecAdapter.locations.size
+      @console.say(
+        format(
+          "boot: %<seconds>.2fs, %<count>d examples loaded",
+          seconds: Clock.since(started),
+          count:
+        )
+      )
+      @console.die("spec files failed to load, see the error above") if RSpecAdapter.load_failed?
+      @console.die(no_tests) if count.zero?
+      RSpecAdapter
+    end
+
+    def no_tests
+      return "the suite is Minitest; mutato is RSpec-only" if defined?(Minitest::Test)
+
+      "no tests loaded from #{options.spec_args.join(" ")}"
+    end
+
+    def measure(adapter)
+      log = output.log_path("baseline")
+      Mutato.config.run_hooks(:before_mutant)
+      check(Child.run(log:) { adapter.baseline(@selection.prefixes) }, log)
+    end
+
+    def check(measured, log)
+      return measured if measured.is_a?(Measure)
+
+      @console.die("baseline crashed: #{measured.inspect}, see #{log}")
+    end
+
+    def describe(measured, started)
+      @console.say(measured.summary(Clock.since(started)))
+      failed = measured.failed
+      if failed.any?
+        @console.say("baseline: #{failed.size} failing tests excluded: #{failed.first(5)}")
+      end
+      @console.die("baseline: nothing passed") if measured.all_failed?
+      measured
+    end
+
+    def baseline(measured, executed)
+      # With autoloading, a file may load first in a test.
+      loaded = executed.keys.to_set.merge(measured.seen)
+      measured.baseline(
+        locations: RSpecAdapter.locations,
+        boot_lines: boot_lines(executed),
+        loaded_files: loaded
+      )
+    end
+
+    def boot_lines(executed)
+      prefixes = @selection.prefixes
+      hits = executed.flat_map do |path, data|
+        path.start_with?(*prefixes) ? Hits.keys(path, data) : []
+      end
+      hits.to_set
+    end
+  end
+end

data/lib/mutato/campaign.rb

@@ -0,0 +1,52 @@
+# frozen_string_literal: true
+
+module Mutato
+  # Outcomes are kept as they come, so an interrupt can still report them.
+  class Campaign
+    attr_reader :suite, :experience
+
+    def initialize(suite, console)
+      @suite = suite
+      @console = console
+      @experience = Experience.fresh
+      @outcomes = []
+    end
+
+    def try(todo, &)
+      watch(&)
+      @console.say("#{todo.size} mutants to try")
+      todo.each_with_index { |mutation, index| @outcomes << judge(mutation, index, todo) }
+      @outcomes
+    end
+
+    private
+
+    # Named, not anonymous: Ruby 3.3.0 rejects `&` forwarded from inside a block.
+    def watch(&report)
+      %w[INT TERM].each { |signal| trap(signal) { interrupted(signal, &report) } }
+    end
+
+    def interrupted(signal)
+      yield(@outcomes)
+      @console.die("interrupted by #{signal}")
+    end
+
+    def judge(mutation, index, todo)
+      outcome = Trial.new(mutation, @suite, @experience).judge
+      @suite.output.append(outcome)
+      @console.say(progress("#{index + 1}/#{todo.size}", outcome, mutation))
+      outcome
+    end
+
+    def progress(position, outcome, mutation)
+      format(
+        "%<position>8s %<outcome>-11s %<seconds>.2fs %<id>s  %<label>s",
+        position:,
+        outcome: outcome.fetch(:outcome),
+        seconds: outcome.fetch(:seconds),
+        id: mutation.id,
+        label: mutation.label
+      )
+    end
+  end
+end