mutant 0.8.19 → 0.8.20

data/config/flay.yml

@@ -1,3 +1,3 @@
 ---
 threshold: 16
-total_score: 1364
+total_score: 1395

data/docs/concurrency.md

@@ -0,0 +1,39 @@
+Concurrency
+===========
+
+By default, mutant will test mutations in parallel by running up
+to one process for each core on your system. You can control the
+number of processes created using the `-j/--jobs` argument.
+
+Mutant forks a new process for each mutation to be tested to prevent side
+affects in your specs and the lack of thread safety in integrations from
+impacting the results.
+
+Database
+--------
+
+If the code under test relies on a database, you may experience problems
+when running mutant because of conflicting data in the database. For
+example, if you have a test like this:
+
+```ruby
+resource = MyModel.create!(...)
+expect(MyModel.first.name).to eql(resource.name)
+```
+
+It might fail if some other test wrote a record to the `MyModel` table
+at the same time as this test was executed. (It would find the MyModel
+record created by the other test.) Most of these issues can be fixed
+by writing more specific tests. Here is a concurrent safe version of
+the same test:
+
+```
+resource = MyModel.create!(...)
+expect(MyModel.find_by_id(m.id).name).to eql(resource.name)
+```
+
+An alternative is to try wrapping each test into an enclosing transaction.
+
+Note that some databases, SQLite in particular, are not designed for
+concurrent access and will fail if used in this manner. If you are
+using SQLite, you should set the `--jobs` to 1.

data/docs/known-problems.md

@@ -0,0 +1,44 @@
+Known Problems
+==============
+
+Mutations with Infinite Runtimes
+---------------------------------
+
+Occasionally mutant will produce a mutation with an infinite runtime. When this happens
+mutant will look like it is running indefinitely without killing a remaining mutation. To
+avoid mutations like this, consider adding a timeout around your tests. For example, in
+RSpec you can add the following to your `spec_helper`:
+```ruby
+config.around(:each) do |example|
+  Timeout.timeout(5, &example)
+end
+```
+which will fail specs which run for longer than 5 seconds.
+
+The Crash / Stuck Problem (MRI)
+-------------------------------
+
+Mutations generated by mutant can cause MRI to enter VM states its not prepared for.
+All MRI versions > 1.9 and < 2.2.1 are affected by this depending on your compiler flags,
+compiler version, and OS scheduling behavior.
+
+This can have the following unintended effects:
+
+* MRI crashes with a segfault. Mutant kills each mutation in a dedicated fork to isolate
+  the mutations side effects when this fork terminates abnormally (segfault) mutant
+  counts the mutation as killed.
+
+* MRI crashes with a segfault and gets stuck when handling the segfault.
+  Depending on the number of active kill jobs mutant might appear to continue normally until
+  all workers are stuck into this state when it begins to hang.
+  Currently mutant must assume that your test suite simply not terminated yet as from the outside
+  (parent process) the difference between a long running test and a stuck MRI is not observable.
+  Its planned to implement a timeout enforced from the parent process, but ideally MRI simply gets fixed.
+
+References:
+
+* [MRI fix](https://github.com/ruby/ruby/commit/8fe95fea9d238a6deb70c8953ceb3a28a67f4636)
+* [MRI backport to 2.2.1](https://github.com/ruby/ruby/commit/8fe95fea9d238a6deb70c8953ceb3a28a67f4636)
+* [Mutant issue](https://github.com/mbj/mutant/issues/265)
+* [Upstream bug redmine](https://bugs.ruby-lang.org/issues/10460)
+* [Upstream bug github](https://github.com/ruby/ruby/pull/822)

data/docs/limitations.md

@@ -0,0 +1,50 @@
+Limitations
+===========
+
+Subject
+-------
+
+Mutant cannot emit mutations for some subjects.
+
+* methods defined within a closure.  For example, methods defined using `module_eval`, `class_eval`,
+  `define_method`, or `define_singleton_method`:
+
+    ```ruby
+    class Example
+      class_eval do
+        def example1
+        end
+      end
+
+      module_eval do
+        def example2
+        end
+      end
+
+      define_method(:example3) do
+      end
+
+      define_singleton_method(:example4) do
+      end
+    end
+    ```
+
+* singleton methods not defined on a constant or `self`
+
+    ```ruby
+    class Foo
+      def self.bar; end   # ok
+      def Foo.baz; end    # ok
+
+      myself = self
+      def myself.qux; end # cannot mutate
+    end
+    ```
+
+* methods defined with eval:
+
+    ```ruby
+    class Foo
+      class_eval('def bar; end') # cannot mutate
+    end
+    ```

data/docs/mutant-minitest.md

@@ -0,0 +1,147 @@
+mutant-minitest
+===============
+
+Before starting with mutant its recommended to understand the
+[nomenclature](/docs/nomenclature.md).
+
+## Setup
+
+To add mutant to your minitest code base you need to:
+
+1. Add `mutant-minitest` as development dependency to your `Gemfile` or `.gemspec`
+
+   This may look like:
+
+   ```ruby
+   # A gemfile
+   gem 'mutant-minitest'
+   ```
+
+2. Add `require 'mutant/minitest/coverage'` to your test environment (example to your `test/test_helper.rb`)
+
+   Example:
+
+   ```ruby
+   require 'minitest/autorun'
+   require 'mutant/minitest/coverage'
+
+   class YourTestBaseClass < MiniTest::Test
+     # ...
+   ```
+
+3. Add `.cover` call sides to your test suite to mark them as eligible for killing mutations in subjects.
+
+   Example:
+
+   ```ruby
+   class YourLibrarySomeClassTest < YourTestBaseClass
+     cover 'YourLibrary::SomeClass*' # tells mutant which subjects this tests should cover
+     # ...
+   ```
+
+4. Run mutant against the minitest integration
+
+   ```sh
+   bundle exec mutant --include lib --require 'your_library.rb' --use minitest -- 'YourLibrary*'
+   ```
+
+## Run through example
+
+This uses [mbj/auom](https://github.com/mbj/auom) a small library that
+has 100% mutation coverage. Its tests execute very fast and do not have any IO
+so its a good playground example to interact with.
+
+All the setup described above is already done.
+
+```sh
+git clone https://github.com/mbj/auom
+bundle install # gemfile references mutant-minitest already
+bundle exec mutant --include lib --require auom --use minitest -- 'AUOM*'
+```
+
+This prints a report like:
+
+```sh
+Mutant configuration:
+Matcher:         #<Mutant::Matcher::Config match_expressions: [AUOM*]>
+Integration:     Mutant::Integration::Minitest
+Jobs:            8
+Includes:        ["lib"]
+Requires:        ["auom"]
+Subjects:        23
+Mutations:       1003
+Results:         1003
+Kills:           1003
+Alive:           0
+Runtime:         9.68s
+Killtime:        3.80s
+Overhead:        154.30%
+Mutations/s:     103.67
+Coverage:        100.00%
+```
+
+Now lets try adding some redundant (or unspecified) code:
+
+```sh
+patch -p1 <<'PATCH'
+--- a/lib/auom/unit.rb
++++ b/lib/auom/unit.rb
+@@ -170,7 +170,7 @@ module AUOM
+     # TODO: Move defaults coercions etc to .build method
+     #
+     def self.new(scalar, numerators = nil, denominators = nil)
+-      scalar = rational(scalar)
++      scalar = rational(scalar) if true
+
+       scalar, numerators   = resolve([*numerators], scalar, :*)
+       scalar, denominators = resolve([*denominators], scalar, :/)
+PATCH
+```
+
+Running mutant again prints the following:
+
+```
+AUOM::Unit.new:/home/mrh-dev/auom/lib/auom/unit.rb:172
+- minitest:AUOMTest::ClassMethods::New#test_reduced_unit
+- minitest:AUOMTest::ClassMethods::New#test_normalized_denominator_scalar
+- minitest:AUOMTest::ClassMethods::New#test_normalized_numerator_unit
+- minitest:AUOMTest::ClassMethods::New#test_incompatible_scalar
+- minitest:AUOMTest::ClassMethods::New#test_integer
+- minitest:AUOMTest::ClassMethods::New#test_sorted_numerator
+- minitest:AUOMTest::ClassMethods::New#test_unknown_unit
+- minitest:AUOMTest::ClassMethods::New#test_rational
+- minitest:AUOMTest::ClassMethods::New#test_normalized_numerator_scalar
+- minitest:AUOMTest::ClassMethods::New#test_sorted_denominator
+- minitest:AUOMTest::ClassMethods::New#test_normalized_denominator_unit
+evil:AUOM::Unit.new:/home/mrh-dev/auom/lib/auom/unit.rb:172:cd9ee
+@@ -1,9 +1,7 @@
+ def self.new(scalar, numerators = nil, denominators = nil)
+-  if true
+-    scalar = rational(scalar)
+-  end
++  scalar = rational(scalar)
+   scalar, numerators = resolve([*numerators], scalar, :*)
+   scalar, denominators = resolve([*denominators], scalar, :/)
+   super(scalar, *[numerators, denominators].map(&:sort)).freeze
+ end
+-----------------------
+Mutant configuration:
+Matcher:         #<Mutant::Matcher::Config match_expressions: [AUOM*]>
+Integration:     Mutant::Integration::Minitest
+Jobs:            8
+Includes:        ["lib"]
+Requires:        ["auom"]
+Subjects:        23
+Mutations:       1009
+Results:         1009
+Kills:           1008
+Alive:           1
+Runtime:         9.38s
+Killtime:        3.47s
+Overhead:        170.06%
+Mutations/s:     107.60
+Coverage:        99.90%
+```
+
+This shows mutant detected the redundant alive conditional.
+Feel free to also remove some tests. Or do other modifications to either tests or code.

data/docs/mutant-rspec.md

@@ -0,0 +1,62 @@
+mutant-rspec
+============
+
+The integration into rspec.
+
+Install `mutant-rspec` and use the `--use rspec` switch in your mutant command line.
+
+```sh
+bundle exec mutant --include lib --require 'your_code' --use rspec -- 'YourCode*'
+```
+
+Examples
+--------
+
+```
+cd virtus
+# Run mutant on virtus namespace
+bundle exec mutant --include lib --require virtus --use rspec Virtus*
+# Run mutant on specific virtus class
+bundle exec mutant --include lib --require virtus --use rspec Virtus::Attribute
+# Run mutant on specific virtus class method
+bundle exec mutant --include lib --require virtus --use rspec Virtus::Attribute.build
+# Run mutant on specific virtus instance method
+bundle exec mutant --include lib --require virtus --use rspec Virtus::Attribute#type
+```
+
+Test-Selection
+--------------
+
+Mutation testing is slow. The key to making it fast is selecting the correct
+set of tests to run.  Mutant currently supports the following built-in
+strategy for selecting tests/specs:
+
+Mutant uses the "longest rspec example group descriptions prefix match" to
+select the tests to run.
+
+Example for a subject like `Foo::Bar#baz` it will run all example groups with
+description prefixes in `Foo::Bar#baz`, `Foo::Bar` and `Foo`. The order is
+important, so if mutant finds example groups in the current prefix level,
+these example groups *must* kill the mutation.
+
+Rails
+-------
+
+To mutation test Rails models with rspec, comment out `require 'rspec/autorun'`
+from your `spec_helper.rb` file. Having done so you should be able to use
+commands like the following:
+
+```sh
+RAILS_ENV=test bundle exec mutant -r ./config/environment --use rspec User
+```
+
+Passing in RSpec Options
+------------------------
+
+**NOTE: Experimental**
+
+You can control some aspects of RSpec using the `SPEC_OPTS` environment variable as usual. If you want mutant to only pay attention to specs in a certain directory, you can run
+
+```sh
+SPEC_OPTS="--pattern spec/subdir_only/**/*_spec.rb" bundle exec mutant --use rspec SomeClass
+```

data/docs/nomenclature.md

@@ -0,0 +1,82 @@
+Nomenclature
+============
+
+The following explains several nouns you may experience in mutant's documentation.
+It's a good idea to familiarize yourself before moving on.
+
+## AST
+
+Acronym for [Abstract Syntax Tree](https://en.wikipedia.org/wiki/Abstract_syntax_tree)
+and the level of abstraction mutant operates on.
+
+## Subject
+
+An addressable piece of code to be targeted for mutation testing.
+
+Mutant currently supports the following subjects:
+
+* Instance methods
+* Singleton (class) methods
+
+Other subjects (constants, class bodies for DSLs, ...) are possible but aren't
+implemented in the OSS version.
+
+## Mutation operator
+
+A transformation applied to the AST of a subject. Mutant knows the following high level operator
+classes:
+
+* Semantic Reduction
+* Orthogonal Replacement
+* [Noop](#neutral-noop-tests)
+
+An exhaustive list can be found in the [mutant-meta](https://github.com/mbj/mutant/tree/master/meta)
+subdirectory of the source.
+
+## Mutation
+
+The result of applying a mutation operator to the AST of a subject. A mutation represents a
+hypothesis that ideally gets falsified by the tests.
+
+## Insertion
+
+The process of inserting a mutation into the runtime environment.
+Mutant currently supports insertion via dynamically created monkeypatches.
+
+Other insertion strategies (such as "boot time") are possible but aren't implemented
+in the OSS version.
+
+## Isolation
+
+The attempt to isolate the (side) effects of killing a mutation via an integration
+to prevent a mutation leaking into adjacent concurrent, or future mutations.
+
+Examples of sources for leaks are
+
+* Global variable writes
+* Thread local writes
+* DB State
+* File system
+
+Natively, mutant offers fork isolation. This works for any state within the executing
+Ruby process. For all state reachable via IO, it's the test author's responsibility to
+provide proper isolation.
+
+## Integration
+
+The method used to determine if a specific inserted mutation is covered by tests.
+
+Currently mutant supports integrations for:
+
+* [mutant-rspec](/docs/mutant-rspec.md) for [rspec](https://rspec.info)
+* [mutant-minitest](/docs/mutant-minitest.md) for [minitest](https://github.com/seattlerb/minitest)
+
+## Report
+
+Mutant currently provides two different reporters:
+
+* Progress (printed during mutation testing).
+* Summary (printed at the end of a finished analysis run)
+
+A reporter producing a machine readable report does not exist in the OSS version
+at the time of writing this documentation.

data/docs/reading-reports.md

@@ -0,0 +1,74 @@
+Reading Reports
+===============
+
+Mutation output is grouped by selection groups. Each group contains three sections:
+
+1. An identifier for the current group.
+
+   **Format**:
+
+   ```text
+   [SUBJECT EXPRESSION]:[SOURCE LOCATION]:[LINENO]
+   ```
+
+   **Example**:
+
+   ```text
+   Book#add_page:Book#add_page:/home/dev/mutant-examples/lib/book.rb:18
+   ```
+
+2. A list of specs that mutant ran to try to kill mutations for the current group.
+
+   **Format**:
+
+   ```text
+   - [INTEGRATION]:0:[SPEC LOCATION]:[SPEC DESCRIPTION]
+   - [INTEGRATION]:1:[SPEC LOCATION]:[SPEC DESCRIPTION]
+   ```
+
+   **Example**:
+
+   ```text
+   - rspec:0:./spec/unit/book_spec.rb:9/Book#add_page should return self
+   - rspec:1:./spec/unit/book_spec.rb:13/Book#add_page should add page to book
+   ```
+
+3. A list of unkilled mutations diffed against the original unparsed source
+
+   **Format**:
+
+   ```text
+   [MUTATION TYPE]:[SUBJECT EXPRESSION]:[SOURCE LOCATION]:[SOURCE LINENO]:[IDENTIFIER]
+   [DIFF]
+   -----------------------
+   ```
+
+   - `[MUTATION TYPE]` will be one of the following:
+      - `evil` - a mutation of your source was not killed by your tests
+      - `neutral` your original source was injected and one or more tests failed
+   - `[IDENTIFIER]` - Unique identifier for this mutation
+
+   **Example**:
+
+   ```diff
+   evil:Book#add_page:Book#add_page:/home/dev/mutant-examples/lib/book.rb:18:01f69
+   @@ -1,6 +1,6 @@
+    def add_page(page)
+   -  @pages << page
+   +  @pages
+      @index[page.number] = page
+      self
+    end
+   -----------------------
+   evil:Book#add_page:Book#add_page:/home/dev/mutant-examples/lib/book.rb:18:b1ff2
+   @@ -1,6 +1,6 @@
+    def add_page(page)
+   -  @pages << page
+   +  self
+      @index[page.number] = page
+      self
+    end
+   -----------------------
+   ```
+
+At this time no machine readable output exists in the opensourced versions of mutant.