suture 0.5.0 → 1.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 9ca2d3a0376378b6ead8f55b0f92e1ef20b73991
-  data.tar.gz: b506cf09899bfd1e372c088aaba8203a0f0c20aa
+  metadata.gz: 65850d2b933384f85c0cd13ec67fe92efcb08320
+  data.tar.gz: ba005e2f0e87b693f89948102559dfccfba7720b
 SHA512:
-  metadata.gz: 30813362842d2a559181b248b08904187d96f78d6feac8b3662ef4a2e7dfa35a2cdda38f5b589184e531266574f4c1e7b23fb7fd68b749bd90d36f73f3a9d652
-  data.tar.gz: 94f1c2b749cadc2a1dc80ac962bc7bfee7d735cea0eae1940a49c09543466f167e25faa809915ae8077e70f34d93590cd1e9bee1d32278be0cd0c77c47ee1122
+  metadata.gz: 7132a9cfa0499d89c786c7ccea1c7ed3da896ec786d3249b1aa3d00433eb0387d3be8917e3bb589f917f888624ab4bab71dc3c2f751a599a100422c09b717f2f
+  data.tar.gz: cd7f1fea872d8887a51f8b65d03cd5bff5189d9d4032141c0862c02265aec1c2acd4552e24e77e611fbb10e57ebd62796d2a956edf754c6dcde57df43438f0ff

data/.travis.yml

@@ -1,7 +1,8 @@
 language: ruby
-cache: bundler
 sudo: false
-before_install: gem install bundler
+before_install:
+  - gem install bundler
+  - cd example/rails_app && ./script/setup.sh && cd ../..
 rvm:
   - 1.8.7
   - 1.9

data/CHANGELOG.md

@@ -2,7 +2,28 @@
 
 ## [Unreleased](https://github.com/testdouble/suture/tree/HEAD)
 
-[Full Changelog](https://github.com/testdouble/suture/compare/v0.3.3...HEAD)
+[Full Changelog](https://github.com/testdouble/suture/compare/v0.4.0...HEAD)
+
+**Implemented enhancements:**
+
+- API to delete a seam's recordings from the database [\#32](https://github.com/testdouble/suture/issues/32)
+
+**Closed issues:**
+
+- Justification for building a custom Docker image instead of using the canonical Ruby one [\#57](https://github.com/testdouble/suture/issues/57)
+- staging/prod: dup arguments [\#56](https://github.com/testdouble/suture/issues/56)
+- Add progress bar [\#54](https://github.com/testdouble/suture/issues/54)
+- warn if no calls are found for the provided seam name [\#51](https://github.com/testdouble/suture/issues/51)
+- Log out an error every time a suture raises [\#50](https://github.com/testdouble/suture/issues/50)
+- Validate test plans [\#28](https://github.com/testdouble/suture/issues/28)
+- Support permitted error types [\#24](https://github.com/testdouble/suture/issues/24)
+
+**Merged pull requests:**
+
+- Remove an unnecessary space \[ci skip\] [\#52](https://github.com/testdouble/suture/pull/52) ([JuanitoFatas](https://github.com/JuanitoFatas))
+
+## [v0.4.0](https://github.com/testdouble/suture/tree/v0.4.0) (2016-08-29)
+[Full Changelog](https://github.com/testdouble/suture/compare/v0.3.3...v0.4.0)
 
 **Closed issues:**
 

data/README.md

@@ -262,7 +262,7 @@ unit tests around new units that shook out from the activity. Having good tests
 for well-factored code is the best guard against seeing it slip once again into
 poorly-understood "legacy" code.
 
-## staging
+## Staging
 
 Once you've changed the code, you still may not be confident enough to delete it
 entirely. It's possible (even likely) that your local exploratory testing didn't
@@ -292,7 +292,7 @@ implementations, and will raise an error if they don't return the same value.
 Obviously, this setting is only helpful if the paths don't trigger major or
 destructive side effects.
 
-## production
+## Production
 
 You're _almost_ ready to delete the old code path and switch production over to
 the new one, but fear lingers: maybe there's an edge case your testing to this
@@ -408,11 +408,155 @@ where you call Suture, like `Suture.create(:foo, { :comparator => my_thing })`
 
 #### Suture.create
 
-TODO
+`Suture.create(name, [options hash])`
+
+* _name_ (Required) - a unique name for the seam, by which any recordings will be
+identified. This should match the name used for any calls to `Suture.verify` by
+your automated tests
+
+* _old_ - (Required) - something that responds to `call` for the provided `args`
+of the seam and either is the legacy code path (e.g.
+`OldCode.new.method(:old_path)`) or invokes it (inside an anonymous Proc or
+lambda)
+
+* _args_ - (Required) - an array of arguments to be passed to the `old` or `new`
+
+* _new_ - like old, but either references or invokes the code path designed to
+replace the `old` legacy code path. When set, Suture will default to invoking
+the `new` path at the exclusion of the `old` path (unless a mode flag like
+`record_calls`, `call_both`, or `fallback_on_error` suggests differently)
+
+* _database_path_ - (Default: `"db/suture.sqlite3"`) - a path relative to the
+current working directory to the Sqlite3 database Suture uses to record and
+playback calls
+
+* _record_calls_ - (Default: false) - when set to true, the `old` path is called
+(regardless of whether `new` is set) and its arguments and result (be it a return
+value or an expected raised error) is recorded into the Suture database for the
+purpose of more coverage for calls to `Suture.verify`. [Read
+more](#3-record-the-current-behavior)
+
+* _call_both_ - (Default: false) - when set to true, the `new` path is invoked,
+then the `old` path is invoked, each with the seam's `args`. The return value
+from each is compared with the `comparator`, and if they are not equivalent, then
+a `Suture::Error::ResultMismatch` is raised. Intended after the `new` path is
+initially developed and to be run in pre-production environments. [Read
+more](#staging)
+
+* _fallback_on_error_ - (Default: false) - designed to be run in production after
+the initial development of the new code path, when set to true, Suture will
+invoke the `new` code path. If `new` raises an error that isn't an
+`expected_error_type`, then Suture will invoke the `old` path with the same args
+in an attempt to recover a working state for the user. [Read more](#production)
+
+* _raise_on_result_mismatch_ - (Default: true) - when set to true, the
+`call_both` mode will merely log incidents of result mismatches, as opposed to
+raising `Suture::Error::ResultMismatch`.
+
+* _comparator_ - (Default: `Suture::Comparator.new`) - determines how return
+values from the Suture are compared when invoking `Suture.verify` or when
+`call_both` mode is activated. By default, results will be considered equivalent
+if `==` returns true or if they `Marshal.dump` to the same string. If this
+default isn't appropriate for the return value of your seam, [read
+on](#creating-a-custom-comparator)
+
+* _expected_error_types_ - (Default: `[]`) - if the seam is expected to raise
+certain types of errors, don't consider them to be exceptional cases. For
+example, if your `:widget` seam is known to raise `WidgetError` objects in
+certain cases, setting `:expected_error_types => [WidgetError]` will result in:
+  * `Suture.create` will record expected errors when `record_calls` is enabled
+  * `Suture.verify` will compare recorded and actual raised errors that are
+    `kind_of?` any recorded error type (regardless of whether `Suture.verify` is
+    passed a redundant list of `expected_error_types`)
+  * `Suture.create`, when `fallback_on_error` is enabled, will allow expected
+    errors raised by the `new` path to propogate, as opposed to logging &
+    rescuing them before calling the `old` path as a fallback
+  * Additionally, `Suture.verify` can be passed `expected_error_types` to squelch
+    warning logs that result from unexpectedly raised errors
+
+* _disable_ - (Default: false) - when enabled, Suture will attempt to revert to
+the original behavior of the `old` path and take no special action. Useful in
+cases where a bug is discovered in a deployed environment and you simply want
+to hit the brakes on any new code path experiments by setting
+`SUTURE_DISABLE=true` globally
+
+* _dup_args_ - (Default: false) - when enabled, Suture will call `dup` on each
+of the args passed to the `old` and/or `new` code paths. Useful when the code
+path(s) mutate the arguments in such a way as to prevent `call_both` or
+`fallback_on_error` from being effective
+
+* _after_new_ - a `call`-able hook that runs after `new` is invoked. If `new`
+raises an error, it is not invoked
+
+* _after_old_ - a `call`-able hook that runs after `old` is invoked. If `old`
+raises an error, it is not invoked
+
+* _on_new_error_ - a `call`-able hook that is invoked after `new` raises an
+unexpected error (see `expected_error_types`).
+
+* _on_old_error_ - a `call`-able hook that is invoked after `old` raises an
+unexpected error (see `expected_error_types`).
 
 #### Suture.verify
 
-TODO
+Many of the settings for `Suture.verify` are analogous to the same settings in
+`Suture.create` and are generally expected to be configured in the same way, as
+if symmetrically with the `Suture.create` call of the seam under test:
+
+* _name_ - (Required) - should be the same name as a seam for which some number
+of recorded calls exist
+
+* _subject_ - (Required) - a `call`-able that will be invoked with each recorded
+set of `args` and have its result compared to that of each recording. This is
+used in lieu of `old` or `new`, since the subject of a `Suture.verify` test might
+be either (or neither!)
+
+* _verify_only_ - (Default: nil) - when set to an ID, Suture.verify` will only
+run against recorded calls for the matching ID. This option is meant to be used
+to focus work on resolving a single verification failure
+
+* _fail_fast_ - (Default: false) - `Suture.verify` will, by default, run against
+every single recording, aggregating and reporting on all errors (just like, say,
+RSpec or Minitest would). However, if the seam is slow to invoke or if you
+confidently expect all of the recordings to pass verification, `fail_fast` is an
+appropriate option to set.
+
+* _call_limit_ - (Default: nil) - when set to a number, Suture will only verify
+up to the set number of recorded calls. Because Suture randomizes the order of
+verifications by default, you can see this as setting Suture.verify to sample a
+random smattering of `call_limit` recordings as a smell test. Potentially useful
+when a seam is very slow
+
+* _time_limit_ - (Default: nil) - when set to a number (in seconds), Suture will
+stop running verifications against recordings once `time_limit` seconds has
+elapsed. Useful when a seam is very slow to invoke
+
+* _error_message_limit_ - (Default: nil) - when set to a number, Suture will only
+print up to `error_message_limit` failure messages. That way, if you currently
+have hundreds of verifications failing, your console isn't overwhelmed by them on
+each run of `Suture.verify`
+
+* _random_seed_ - (Default: it's random!) - a randomized seed used to shuffle
+the recordings before verifying them against the `subject` code path. If set to
+`nil`, the recordings will be invoked in insertion-order. If set to a specific
+number, that number will be used as the random seed (useful when re-running a
+particular verification failure that can't be reproduced otherwise)
+
+* _comparator_ - (Default: `Suture::Comparator`) - If a custom comparator is used
+by the seam in `Suture.create`, then the same comparator should probably be
+used by `Suture.verify` to ensure the results are comparable. [Read
+more](#creating-a-custom-comparator) on creating custom comparators
+)
+
+* _database_path_ - (Default: `"db/suture.sqlite3"`) - as with `Suture.create`, a
+custom database path can be set for almost any invocation of Suture, and
+`Suture.verify is no exception`
+
+* _after_subject_ - a `call`-able hook that runs after `subject` is invoked. If
+`subject` raises an error, it is not invoked
+
+* _on_new_subject_ - a `call`-able hook that is invoked after `subject` raises an
+unexpected error (see `expected_error_types`)
 
 ### Creating a custom comparator
 
@@ -472,6 +616,50 @@ Suture.verify(:my_type, {
 })
 ```
 
+#### Comparing two ActiveRecord objects
+
+Let's face it, a massive proportion of legacy Ruby code in the wild involves
+ActiveRecord objects to some extent, and it's important that Suture be equipped
+to compare them gracefully. If Suture's default comparator (`Suture::Comparator`)
+detects two ActiveRecord model instances being compared, it will behave
+differently, by this logic:
+
+1. Instead of comparing the objects with `==` (which returns true so long as the
+`id` attribute matdhes), Suture will compare the objects' `attributes` hashes
+instead
+2. The built-in `updated_at` and `created_at` will typically differ when code
+is executed at different times and are usually not meaningful to application
+logic, Suture will ignore these attributes by default
+
+Other attributes may or may not matter (for instance, other timestamp fields,
+or the `id` of the object), in those cases, you can instantiate the comparator
+yourself and tell it which attributes to exclude, like so:
+
+``` ruby
+Suture.verify :thing,
+  :subject => Thing.new.method(:stuff),
+  :comparator => Suture::Comparator.new(
+    :active_record_excluded_attributes => [
+      :id,
+      :quality,
+      :created_at,
+      :updated_at
+    ]
+  )
+```
+
+If `Thing#stuff` returns an instance of an ActiveRecord model, the four
+attributes listed above will be ignored when comparing with recorded results.
+
+In all of the above cases, `:comparator` can be set on both `Suture.create` and
+`Suture.verify` and typically ought to be symmetrical for most seams.
+
+## Examples
+
+This repository contains these examples available for your perusal:
+
+* [A Rails app of the Gilded Rose kata](example/rails_app)
+
 ## Troubleshooting
 
 Some ideas if you can't get a particular verification to work or if you keep

data/Rakefile

@@ -26,6 +26,18 @@ Rake::TestTask.new(:test) do |t|
   ]
 end
 
+task :example do
+  Dir.chdir("example/rails_app") do
+    passed = system <<-SH
+      BUNDLE_GEMFILE="$PWD/Gemfile" bundle install --quiet
+      BUNDLE_GEMFILE="$PWD/Gemfile" bundle exec rake suture
+    SH
+    if !passed
+      raise StandardError.new("Rails example failed!")
+    end
+  end
+end
+
 if Gem.ruby_version >= Gem::Version.new("2.2.2")
   require 'github_changelog_generator/task'
   GitHubChangelogGenerator::RakeTask.new :changelog
@@ -39,4 +51,4 @@ if Gem.ruby_version >= Gem::Version.new("2.2.2")
 end
 
 
-task :default => :test
+task :default => [:test, :example]

data/lib/suture/adapter/log.rb

@@ -4,7 +4,7 @@ require "suture/util/env"
 module Suture::Adapter
   module Log
     def self.logger
-      if !@setup
+      if !defined?(@setup) || !@setup
         @logger = Suture::Wrap::Logger.init(Suture.config.merge(Suture::Util::Env.to_map))
         @setup = true
       end

data/lib/suture/comparator.rb

@@ -1,7 +1,53 @@
 module Suture
   class Comparator
+    DEFAULT_ACTIVE_RECORD_EXCLUDED_ATTRIBUTES = [:updated_at, :created_at]
+
+    def initialize(options = {})
+      @options = {
+        :active_record_excluded_attributes => (
+          options[:active_record_excluded_attributes] ||
+            DEFAULT_ACTIVE_RECORD_EXCLUDED_ATTRIBUTES
+        ).map(&:to_s)
+      }
+    end
+
     def call(recorded, actual)
-      recorded == actual || Marshal.dump(recorded) == Marshal.dump(actual)
+      is_equalivalent?(recorded, actual) ||
+        Marshal.dump(recorded) == Marshal.dump(actual)
+    end
+
+    def inspect
+      "#{self.class}.new(#{@options.inspect})"
+    end
+
+    protected
+
+    def compare_active_record(recorded, actual)
+      actual.kind_of?(recorded.class) &&
+        without_excluded_attrs(recorded.attributes) ==
+          without_excluded_attrs(actual.attributes)
+    end
+
+    private
+
+    def without_excluded_attrs(hash)
+      hash.reject do |k, v|
+        @options[:active_record_excluded_attributes].include?(k.to_s)
+      end
+    end
+
+    def is_equalivalent?(recorded, actual)
+      if is_active_record?(recorded, actual)
+        compare_active_record(recorded, actual)
+      else
+        recorded == actual
+      end
+    end
+
+    def is_active_record?(recorded, actual)
+      defined?(ActiveRecord::Base) &&
+        recorded.kind_of?(ActiveRecord::Base) &&
+        actual.kind_of?(ActiveRecord::Base)
     end
   end
 end

data/lib/suture/error/invalid_test_plan.rb

@@ -0,0 +1,7 @@
+module Suture::Error
+  class InvalidTestPlan < StandardError
+    def message
+      "Suture.verify requires a `:subject` that responds to `:call`."
+    end
+  end
+end

data/lib/suture/error/result_mismatch.rb

@@ -7,7 +7,7 @@ module Suture::Error
     end
 
     def message
-      expected_message = <<-MSG.gsub(/^ {8}/,'')
+      <<-MSG.gsub(/^ {8}/,'')
         The results from the old & new code paths did not match for the seam
         #{@plan.name.inspect} and Suture is raising this error because the `:call_both`
         option is enabled, because both code paths are expected to return the
@@ -52,7 +52,6 @@ module Suture::Error
         too disruptive and logging is sufficient for monitoring results, you may
         disable this error by setting `:raise_on_result_mismatch` to false.
       MSG
-
     end
   end
 end

data/lib/suture/error/verification_failed.rb

@@ -1,4 +1,5 @@
 require "suture/adapter/progress_bar"
+require "suture/util/numbers"
 
 module Suture::Error
   class VerificationFailed < StandardError
@@ -47,6 +48,7 @@ module Suture::Error
         results.passed.size,
         results.all.size
       )
+      percent = Suture::Util::Numbers.percent(results.passed.size, results.all.size)
       <<-MSG.gsub(/^ {8}/,'')
         ## Progress
 
@@ -54,7 +56,7 @@ module Suture::Error
 
         #{bar}
 
-        Of #{results.all.size} recorded interactions, #{results.passed.size} are currently passing.
+        Of #{results.all.size} recorded interactions, #{results.passed.size} are currently passing. That's #{percent}%!
       MSG
     end
 
@@ -66,13 +68,13 @@ module Suture::Error
 
         ```
         {
-          :comparator => #{describe_comparator(plan.comparator)}
           :database_path => #{plan.database_path.inspect},
           :fail_fast => #{plan.fail_fast},
           :call_limit => #{plan.call_limit.inspect},#{" # (no limit)" if plan.call_limit.nil?}
           :time_limit => #{plan.time_limit.inspect},#{plan.time_limit.nil? ? " # (no limit)" : " # (in seconds)"}
           :error_message_limit => #{plan.error_message_limit.inspect},#{" # (no limit)" if plan.error_message_limit.nil?}
-          :random_seed => #{plan.random_seed.inspect}#{" # (insertion order)" if plan.random_seed.nil?}
+          :random_seed => #{plan.random_seed.inspect},#{" # (insertion order)" if plan.random_seed.nil?}
+          :comparator => #{describe_comparator(plan.comparator)}
         }
         ```
       MSG
@@ -80,9 +82,9 @@ module Suture::Error
 
     def describe_comparator(comparator)
       if comparator.kind_of?(Proc)
-        "Proc, # (in: `#{describe_source_location(*comparator.source_location)}`)"
+        "Proc # (in: `#{describe_source_location(*comparator.source_location)}`)"
       elsif comparator.respond_to?(:method) && comparator.method(:call)
-        "#{comparator.class}.new, # (in: `#{describe_source_location(*comparator.method(:call).source_location)}`)"
+        "#{comparator.inspect}.new, # (in: `#{describe_source_location(*comparator.method(:call).source_location)}`)"
       end
     end
 

data/lib/suture/util/numbers.rb

@@ -0,0 +1,7 @@
+module Suture::Util
+  module Numbers
+    def self.percent(x, out_of_y)
+      ((x.to_f / out_of_y.to_f) * 100).round
+    end
+  end
+end

data/lib/suture/value/result.rb

@@ -22,6 +22,8 @@ module Suture::Value
       other.kind_of?(self.class) && other.state == state
     end
 
+    alias_method :eql?, :==
+
     def hash
       state.hash
     end

data/lib/suture/verify/tests_patient.rb

@@ -4,6 +4,7 @@ require "suture/adapter/dictaphone"
 require "suture/value/test_results"
 require "suture/util/shuffle"
 require "suture/util/timer"
+require "suture/error/invalid_test_plan"
 require "backports/1.9.2/random"
 
 module Suture
@@ -15,6 +16,7 @@ module Suture
     end
 
     def test(test_plan)
+      validate_test_plan!(test_plan)
       experienced_failure_in_life = false
       timer = Suture::Util::Timer.new(test_plan.time_limit) unless test_plan.time_limit.nil?
       test_cases = build_test_cases(test_plan)
@@ -35,6 +37,12 @@ module Suture
 
     private
 
+    def validate_test_plan!(test_plan)
+      if !test_plan.subject || !test_plan.subject.respond_to?(:call)
+        raise Suture::Error::InvalidTestPlan.new
+      end
+    end
+
     def should_skip?(test_plan, failed_fast, call_count, timer)
       (test_plan.fail_fast && failed_fast) ||
         (test_plan.call_limit && call_count >= test_plan.call_limit) ||

data/lib/suture/version.rb

@@ -1,3 +1,3 @@
 module Suture
-  VERSION = "0.5.0"
+  VERSION = "1.0.0"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: suture
 version: !ruby/object:Gem::Version
-  version: 0.5.0
+  version: 1.0.0
 platform: ruby
 authors:
 - Justin Searls
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2016-09-05 00:00:00.000000000 Z
+date: 2016-09-07 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: sqlite3
@@ -197,6 +197,7 @@ files:
 - lib/suture/create/validates_plan.rb
 - lib/suture/delete.rb
 - lib/suture/error/invalid_plan.rb
+- lib/suture/error/invalid_test_plan.rb
 - lib/suture/error/observation_conflict.rb
 - lib/suture/error/result_mismatch.rb
 - lib/suture/error/schema_version.rb
@@ -208,6 +209,7 @@ files:
 - lib/suture/surgeon/remediator.rb
 - lib/suture/util/compares_results.rb
 - lib/suture/util/env.rb
+- lib/suture/util/numbers.rb
 - lib/suture/util/scalpel.rb
 - lib/suture/util/shuffle.rb
 - lib/suture/util/timer.rb