suture 0.2.0 → 0.3.0

data/.travis.yml

@@ -1,4 +1,5 @@
 language: ruby
+cache: bundler
 sudo: false
 before_install: gem install bundler
 rvm:
@@ -8,3 +9,6 @@ rvm:
   - 2.1
   - 2.2
   - 2.3.1
+addons:
+  code_climate:
+    repo_token: 05e3e31164d59aa626b730b92eb9b7418326dbf23420a4b87eab2555840b39ef

data/README.md

@@ -1,6 +1,6 @@
 # Suture
 
-[![Build Status](https://travis-ci.org/testdouble/suture.svg?branch=master)](https://travis-ci.org/testdouble/suture)
+[![Build Status](https://travis-ci.org/testdouble/suture.svg?branch=master)](https://travis-ci.org/testdouble/suture) [![Code Climate](https://codeclimate.com/github/testdouble/suture/badges/gpa.svg)](https://codeclimate.com/github/testdouble/suture) [![Test Coverage](https://codeclimate.com/github/testdouble/suture/badges/coverage.svg)](https://codeclimate.com/github/testdouble/suture/coverage)
 
 A refactoring tool for Ruby, designed to make it safe to change code you don't
 confidently understand. In fact, changing untrustworthy code is so fraught,
@@ -97,7 +97,7 @@ implementations behave the same way.
 First, we tell Suture to start recording calls by setting the environment
 variable `SUTURE_RECORD_CALLS` to something truthy (e.g.
 `SUTURE_RECORD_CALLS=true bundle exec rails s`). So long as this variable is set,
-any calls to our suture will record the arguments passed to the legacy code path
+any calls to our seam will record the arguments passed to the legacy code path
 and the return value.
 
 As you use the application (whether it's a queue system, a web app, or a CLI),
@@ -304,8 +304,29 @@ you're yet unsure that your refactor or reimplementation is complete.
 
 While your application's logs aren't affected by Suture, it may be helpful for
 Suture to maintain a separate log file for any errors that are raised by the
-refactored code path. Setting the key `:log` to a path will prompt Suture to
-append any errors to a log at that location.
+refactored code path.
+
+Suture has a handful of process-wide logging settings that can be set at any
+point as your app starts up (if you're using Rails, then your
+environment-specific (e.g. `config/environments/production.rb`) config file
+is a good choice).
+
+``` ruby
+Suture.config({
+  :log_level => "WARN", #<-- defaults to "INFO"
+  :log_stdout => false, #<-- defaults to true
+  :log_file => "log/suture.log" #<-- defaults to nil
+})
+```
+
+When your new code path raises an error with the above settings, it will
+propogate and log the error to the specified file.
+
+### Custom error handlers
+
+Additionally, you may have some idea of what you want to do (i.e. phone home to
+a reporting service) in the event that your new code path fails. To add custom
+error handling before, set the `:on_error` option to a callable.
 
 ``` ruby
 class MyWorker
@@ -314,7 +335,7 @@ class MyWorker
       old: LegacyWorker.new,
       new: NewWorker.new,
       args: [id],
-      log: 'log/my_worker_seam.log'
+      on_error: -> (name, args) { PhonesHome.new.phone(name, args) }
     }))
   end
 end
@@ -323,9 +344,9 @@ end
 ### Retrying failures
 
 Since the legacy code path hasn't been deleted yet, there's no reason to leave
-users hanging if the new code path explodes. By setting the `:retry` entry to
-`true`, Suture will rescue any errors raised from the new code path and attempt
-to invoke the legacy code path instead.
+users hanging if the new code path explodes. By setting the `:fallback_to_old`
+entry to `true`, Suture will rescue any errors raised from the new code path and
+attempt to invoke the legacy code path instead.
 
 ``` ruby
 class MyWorker
@@ -334,7 +355,7 @@ class MyWorker
       old: LegacyWorker.new,
       new: NewWorker.new,
       args: [id],
-      retry: true
+      fallback_to_old: true
     }))
   end
 end
@@ -344,3 +365,116 @@ Since this approach rescues errors, it's possible that errors in the new code
 path will go unnoticed, so it's best used in conjunction with Suture's logging
 feature. Before ultimately deciding to finally delete the legacy code path,
 double-check that the logs aren't full of rescued errors!
+
+## Configuration
+
+Legacy code is, necessarily, complex and hard-to-wrangle. That's why Suture comes
+with a bunch of configuration options to modify its behavior, particularly for
+hard-to-compare objects.
+
+### Setting configuration options
+
+In general, most configuration options can be set in several places:
+
+* Globally, via an environment variable. The flag `record_calls` will translate
+to an expected ENV var named `SUTURE_RECORD_CALLS` and can be set from the
+command line like so: `SUTURE_RECORD_CALLS=true bundle exec rails server`, to
+tell Suture to record all your interactions with your seams without touching the
+source code.
+
+* Globally, via the top-level `Suture.config` method. Most variables can be set
+via this top-level configuration, like
+`Suture.config(:database_path => 'my.db')`. Once set, this will apply to all your
+interactions with Suture for the life of the process until you call
+`Suture.reset!`.
+
+* At a `Suture.create` or `Suture.verify` call-site as part of its options hash.
+If you have several seams, you'll probably want to set most options locally
+where you call Suture, like `Suture.create(:foo, { :comparator => my_thing })`
+
+### Supported options
+
+#### Suture.create
+
+TODO
+
+#### Suture.verify
+
+TODO
+
+### Creating a custom comparator
+
+Out-of-the-box, Suture will do its best to compare your recorded & actual results
+to ensure that things are equivalent to one another, but reality is often less
+tidy than a gem can predict up-front. When the built-in equivalency comparator
+fails you, you can define a custom one—globally or at each `Suture.create` or
+`Suture.verify` call-site.
+
+#### Extending the built-in comparator class
+
+If you have a bunch of value types that require special equivalency checks, it
+makes sense to invest the time to extend built-in one:
+
+``` ruby
+class MyComparator < Suture::Comparator
+  def call(recorded, actual)
+    if recorded.kind_of?(MyType)
+      recorded.data_stuff == actual.data_stuff
+    else
+      super
+    end
+  end
+end
+```
+
+So long as you return `super` for non-special cases, it should be safe to set an
+instance of your custom comparator globally for the life of the process with:
+
+``` ruby
+Suture.config({
+  :comparator => MyComparator.new
+})
+```
+
+#### Creating a one-off comparator
+
+If a particular seam requires a custom comparator and will always return
+sufficiently homogeneous types, it may be good enough to set a custom comparator
+inline at the `Suture.create` or `Suture.verify` call-site, like so:
+
+``` ruby
+Suture.create(:my_type, {
+  :old => method(:old_method),
+  :args => [42],
+  :comparator => ->(recorded, actual){ recorded.data_thing == actual.data_thing }
+})
+```
+
+Just be sure to set it the same way if you want `Suture.verify` to be able to
+test your recorded values!
+
+``` ruby
+Suture.verify(:my_type, {
+  :subject => method(:old_method),
+  :comparator => ->(recorded, actual){ recorded.data_thing == actual.data_thing }
+})
+```
+
+## Troubleshooting
+
+Some ideas if you can't get a particular verification to work or if you keep
+seeing false negatives:
+
+  * There may be a side effect in your code that you haven't found, extracted,
+    replicated, or controlled for. Consider contributing to [this
+    milestone](https://github.com/testdouble/suture/milestone/3), which specifies
+    a side-effect detector to be paired with Suture to make it easier to see
+    when observable database, network, and in-memory changes are made during a
+    Suture operation
+  * Consider writing a [custom comparator](#creating-a-custom-comparator) with
+    a relaxed conception of equivalence between the recorded and observed results
+  * If a recording was made in error, you can always delete it, either by
+    dropping Suture's database (which is, by default, stored in
+    `db/suture.sqlite3`) or by observing the ID of the recording from an error
+    message and invoking `Suture.delete(42)`
+

data/Rakefile

@@ -13,4 +13,25 @@ Rake::TestTask.new(:safe) do |t|
   t.test_files = FileList['safe/helper.rb', 'safe/**/*_test.rb']
 end
 
-task :default => [:test, :safe]
+Rake::TestTask.new(:everything) do |t|
+  t.libs << "test"
+  t.libs << "safe"
+  t.libs << "lib"
+  t.test_files = FileList[
+    'safe/support/code_climate',
+    'test/helper.rb',
+    'test/**/*_test.rb',
+    'safe/helper.rb',
+    'safe/**/*_test.rb'
+  ]
+end
+
+require 'github_changelog_generator/task'
+
+GitHubChangelogGenerator::RakeTask.new :changelog do |config|
+  config.since_tag = '0.1.14'
+  config.future_release = '0.2.0'
+end
+task :release => :changelog
+
+task :default => :everything

data/lib/suture/adapter/dictaphone.rb

@@ -1,50 +1,68 @@
 require "suture/wrap/sqlite"
+require "suture/adapter/log"
 require "suture/value/observation"
 require "suture/error/observation_conflict"
 
 module Suture::Adapter
   class Dictaphone
+    include Suture::Adapter::Log
+
     def initialize(plan)
       @db = Suture::Wrap::Sqlite.init(plan.database_path)
       @name = plan.name
-      @args = plan.args
+      @comparator = plan.comparator
+      if plan.respond_to?(:args) # does not apply to TestPlan objects
+        @args_inspect = plan.args.inspect
+        @args_dump = Marshal.dump(plan.args)
+      end
     end
 
     def record(result)
       Suture::Wrap::Sqlite.insert(@db, :observations, [:name, :args, :result],
-                                  [@name.to_s, Marshal.dump(@args), Marshal.dump(result)])
-
-    rescue SQLite3::ConstraintException => e
-      old_result = result_for(@name, @args)
-      if old_result != result # TODO - use comparator
-        raise Suture::Error::ObservationConflict.new(@name, @args, result, old_result)
+                                  [@name.to_s, @args_dump, Marshal.dump(result)])
+      log_info("recorded call for seam #{@name.inspect} with args `#{@args_inspect}` and result `#{result.inspect}`")
+    rescue SQLite3::ConstraintException
+      old_observation = known_observation
+      if @comparator.call(old_observation.result, result)
+        log_debug("skipped recording of duplicate call for seam #{@name.inspect} with args `#{@args_inspect}` and result `#{result.inspect}`")
       else
-        # We're all good here, it was just a duplicative observation. No harm.
+        raise Suture::Error::ObservationConflict.new(@name, @args_inspect, result, old_observation)
       end
     end
 
-    def play
-      rows = Suture::Wrap::Sqlite.select(@db, :observations, "where name = ?", [@name.to_s])
-      rows.map do |row|
-        Suture::Value::Observation.new(
-          row[0],
-          row[1].to_sym,
-          Marshal.load(row[2]),
-          Marshal.load(row[3])
-        )
-      end
+    def play(only_id = nil)
+      rows = Suture::Wrap::Sqlite.select(
+        @db, :observations,
+        "where name = ? #{"and id = ?" if only_id}",
+        [@name.to_s, only_id].compact
+      )
+      log_debug("found #{rows.size} recorded calls for seam #{@name.inspect}.")
+      rows.map { |row| row_to_observation(row) }
+    end
+
+    def delete(id)
+      log_info("deleting call with ID: #{id}")
+      Suture::Wrap::Sqlite.delete(@db, :observations, "where id = ?", [id])
     end
 
   private
 
-    def result_for(name, args)
-      rows = Suture::Wrap::Sqlite.select(
+    def row_to_observation(row)
+      Suture::Value::Observation.new(
+        row[0],
+        row[1].to_sym,
+        Marshal.load(row[2]),
+        Marshal.load(row[3])
+      )
+    end
+
+    def known_observation
+      row_to_observation(Suture::Wrap::Sqlite.select(
         @db,
         :observations,
         "where name = ? and args = ?",
-        [name.to_s, Marshal.dump(args)]
-      )
-      Marshal.load(rows.first[3])
+        [@name.to_s, @args_dump]
+      ).first)
     end
   end
 end

data/lib/suture/adapter/log.rb

@@ -0,0 +1,31 @@
+require "suture/wrap/logger"
+require "suture/util/env"
+
+module Suture::Adapter
+  module Log
+    def self.logger
+      if !@setup
+        @logger = Suture::Wrap::Logger.init(Suture.config.merge(Suture::Util::Env.to_map))
+        @setup = true
+      end
+      @logger
+    end
+
+    def self.reset!
+      @setup = nil
+      @logger = nil
+    end
+
+    def log_debug(*args, &blk)
+      Log.logger.debug(*args, &blk)
+    end
+
+    def log_info(*args, &blk)
+      Log.logger.info(*args, &blk)
+    end
+
+    def log_warn(*args, &blk)
+      Log.logger.warn(*args, &blk)
+    end
+  end
+end

data/lib/suture/builds_plan.rb

@@ -3,11 +3,11 @@ require "suture/util/env"
 
 module Suture
   class BuildsPlan
-    UN_ENV_IABLE_OPTIONS = [:name, :old, :new, :args]
+    UN_ENV_IABLE_OPTIONS = [:name, :old, :new, :args, :comparator]
 
     def build(name, options = {})
       Value::Plan.new(
-        DEFAULT_OPTIONS.
+        Suture.config.
           merge(options).
           merge(:name => name).
           merge(Suture::Util::Env.to_map(UN_ENV_IABLE_OPTIONS))

data/lib/suture/chooses_surgeon.rb

@@ -1,10 +1,20 @@
+require "suture/adapter/log"
 require "suture/surgeon/observer"
 require "suture/surgeon/no_op"
 
 module Suture
   class ChoosesSurgeon
+    include Suture::Adapter::Log
+
     def choose(plan)
       if plan.record_calls
+        if plan.new
+          log_warn <<-MSG.gsub(/^ {12}/,'')
+            Seam #{plan.name.inspect} has a :new code path defined, but because
+            it is set to :record_calls, we will invoke the :old code path
+            instead. If this is not what you intend, set :record_calls to false.
+          MSG
+        end
         Surgeon::Observer.new
       else
         Surgeon::NoOp.new

data/lib/suture/comparator.rb

@@ -0,0 +1,7 @@
+module Suture
+  class Comparator
+    def call(recorded, actual)
+      recorded == actual || Marshal.dump(recorded) == Marshal.dump(actual)
+    end
+  end
+end

data/lib/suture/error/observation_conflict.rb

@@ -1,16 +1,28 @@
 module Suture::Error
   class ObservationConflict < StandardError
-    def initialize(name, args, new_result, old_result)
+    def initialize(name, args_inspect, new_result, old_observation)
       @name = name
-      @args = args
+      @args_inspect = args_inspect
       @new_result = new_result
-      @old_result = old_result
+      @old_id = old_observation.id
+      @old_result = old_observation.result
     end
 
     def message
-      <<-MSG.gsub(/^\s{8}/,'')
-        At suture #{@name.inspect} with inputs `#{@args.inspect}`, the newly-observed return value `#{@new_result.inspect}`
-        conflicts with previously recorded return value `#{@old_result.inspect}`.
+      <<-MSG.gsub(/^ {8}/,'')
+        At seam #{@name.inspect}, we just recorded a duplicate call, but the same arguments
+        resulted in a different output. Read on for details:
+
+        Arguments: ```
+          #{@args_inspect}
+        ```
+        Previously-observed return value: ```
+          #{@old_result.inspect}
+        ```
+
+        Newly-observed return value: ```
+          #{@new_result.inspect}
+        ```
 
         That's not good! Here are a few ideas of what may have happened:
 
@@ -22,21 +34,22 @@ module Suture::Error
            different timestamp) or side effects (e.g. saving to a database
            resulting in a different GUID value) mean that Suture is detecting two
            different results for the same inputs. This can be worked around by
-           providing a custom comparator for the two values nearest common
-           ancestor type. Comparator support is tracked here:
-             https://github.com/testdouble/suture/issues/14
+           providing a custom comparator to Suture. For more info, see the README:
+
+             https://github.com/testdouble/suture#creating-a-custom-comparator
 
         3. If neither of the above are true, it's possible that the old code path
            was changed while still in the early stage of recording characterization
            calls (presumably by mistake). If such a change may have occurred in
            error, check your git history. Otherwise, perhaps you `record_calls` is
-           accidentally still enabled and should be turned off for this suture
+           accidentally still enabled and should be turned off for this seam
            (either with SUTURE_RECORD_CALLS=false or :record_calls => false).
 
-        4. If the old recording was made in error, then you may want to delete it
-           Deletion support via the Suture API is tracked here:
-             https://github.com/testdouble/suture/issues/10
+        4. If, after exhausting the possibilities above, you're pretty sure the
+           recorded result is in error, you can delete it from Suture's database
+           with:
 
+             Suture.delete(#{@old_id})
       MSG
     end
   end

data/lib/suture/error/verification_failed.rb

@@ -1,12 +1,167 @@
 module Suture::Error
   class VerificationFailed < StandardError
-    def initialize(test_results)
-      super
-      @test_results = test_results
+    def initialize(plan, results)
+      @plan = plan
+      @results = results
     end
 
-    # def message
-    # end
+    def message
+      [
+        intro,
+        describe_failures(@results.failed, @plan),
+        configuration(@plan),
+        summarize(@results)
+      ].join("\n")
+    end
+
+    private
+
+    def intro
+      <<-MSG.gsub(/^ {8}/,'')
+
+        # Verification of your seam failed!
+
+        Descriptions of each unsuccessful verification follows:
+      MSG
+    end
+
+    def summarize(results)
+      <<-MSG.gsub(/^ {8}/,'')
+        # Result Summary
+
+          - Passed........#{results.passed_count}
+          - Failed........#{results.failed_count}
+            - with error..#{results.errored_count}
+          - Skipped.......#{results.skipped_count}
+          - Total calls...#{results.total_count}
+      MSG
+    end
+
+    def configuration(plan)
+      <<-MSG.gsub(/^ {8}/,'')
+        # Configuration
+
+        This is the configuration used by this test run:
+
+        ```
+        {
+          :comparator => #{describe_comparator(plan.comparator)}
+          :database_path => #{plan.database_path.inspect},
+          :fail_fast => #{plan.fail_fast},
+          :call_limit => #{plan.call_limit.inspect},#{" # (no limit)" if plan.call_limit.nil?}
+          :time_limit => #{plan.time_limit.inspect},#{plan.time_limit.nil? ? " # (no limit)" : " # (in seconds)"}
+          :error_message_limit => #{plan.error_message_limit.inspect},#{" # (no limit)" if plan.error_message_limit.nil?}
+          :random_seed => #{plan.random_seed.inspect}#{" # (insertion order)" if plan.random_seed.nil?}
+        }
+        ```
+      MSG
+    end
+
+    def describe_comparator(comparator)
+      if comparator.kind_of?(Proc)
+        "Proc, # (in: `#{describe_source_location(*comparator.source_location)}`)"
+      elsif comparator.respond_to?(:method) && comparator.method(:call)
+        "#{comparator.class}.new, # (in: `#{describe_source_location(*comparator.method(:call).source_location)}`)"
+      end
+    end
+
+    def describe_source_location(file, line)
+      root = File.join(Dir.getwd, "/")
+      path = file.start_with?(root) ? file.gsub(root, '') : file
+      "#{path}:#{line}"
+    end
+
+    def describe_failures(failures, plan)
+      return if failures.empty?
+      [
+        "## Failures\n",
+        failures.each_with_index.map { |failure, index|
+          describe_failure(failure, index) if plan.error_message_limit.nil? || index < plan.error_message_limit
+        },
+        describe_squelched_failure_messages(failures.size, plan.error_message_limit),
+        describe_general_failure_advice(plan)
+      ].flatten.compact.join("\n")
+    end
+
+    def describe_failure(failure, index)
+      expected = failure[:observation]
+      return <<-MSG.gsub(/^ {8}/,'')
+        #{index + 1}.) Recorded call for seam #{expected.name.inspect} (ID: #{expected.id}) ran and #{failure[:error] ? "raised an error" : "failed comparison"}.
+
+           Arguments: ```
+             #{expected.args.inspect}
+           ```
+           Expected result: ```
+             #{expected.result.inspect}
+           ```
+           #{failure[:error] ? "Error raised" : "Actual result"}: ```
+             #{if failure[:error]
+                 stringify_error(failure[:error])
+               else
+                 failure[:new_result].inspect
+               end
+             }
+           ```
+
+           Ideas to fix this:
+             * Focus on this test by setting ENV var `SUTURE_VERIFY_ONLY=#{expected.id}`
+             * Is the recording wrong? Delete it! `Suture.delete(#{expected.id})`
+      MSG
+    end
+
+    def describe_squelched_failure_messages(failure_count, error_message_limit)
+      return if error_message_limit.nil? || error_message_limit >= failure_count
+      "(#{failure_count - error_message_limit} more failure messages were hidden because :error_message_limit was set to #{error_message_limit}.)"
+    end
+
+    def describe_general_failure_advice(plan)
+      <<-MSG.gsub(/^ {8}/,'')
+        ### Fixing these failures
+
+        #### Custom comparator
 
+        If any comparison is failing and you believe the results are
+        equivalent, we suggest you look into creating a custom comparator.
+        See more details here:
+
+          https://github.com/testdouble/suture#creating-a-custom-comparator
+
+        #### Random seed
+
+        Suture runs all verifications in random order by default. If you're
+        seeing an erratic failure, it's possibly due to order-dependent
+        behavior somewhere in your subject's code.
+
+        #{if !plan.random_seed.nil?
+            <<-MOAR.gsub(/^ {14}/,'')
+              To re-run the tests with the same random seed as was used in this run,
+              set the env var `SUTURE_RANDOM_SEED=#{plan.random_seed}` or the config entry
+              `:random_seed => #{plan.random_seed}`.
+
+              To re-run the tests without added shuffling (that is, in the order the
+              calls were recorded in), then set the random seed explicitly to nil
+              with env var `SUTURE_RANDOM_SEED=nil` or the config entry
+              `:random_seed => nil`.
+            MOAR
+          else
+            <<-MOAR.gsub(/^ {14}/,'')
+              This test was run in insertion order (by the primary key of the table
+              that stores calls, ascending). This is sometimes necessary when the
+              code has an order-dependent side effect, but shouldn't be set unless it's
+              clearly necessary, so as not to incidentally encourage _porting over_
+              that temporal side effect to the new code path. To restore random
+              ordering, unset the env var `SUTURE_RANDOM_SEED` and/or the config entry
+              `:random_seed`.
+            MOAR
+          end.chomp}
+      MSG
+    end
+
+
+    def stringify_error(error)
+      s = error.inspect
+      s += "\n" + error.backtrace.join("\n") if error.backtrace
+      s
+    end
   end
 end

data/lib/suture/interprets_results.rb

@@ -2,9 +2,9 @@ require "suture/error/verification_failed"
 
 module Suture
   class InterpretsResults
-    def interpret(test_results)
+    def interpret(test_plan, test_results)
       return unless test_results.failed?
-      raise Suture::Error::VerificationFailed.new(test_results)
+      raise Suture::Error::VerificationFailed.new(test_plan, test_results)
     end
   end
 end

data/lib/suture/prescribes_test_plan.rb

@@ -3,14 +3,14 @@ require "suture/util/env"
 
 module Suture
   class PrescribesTestPlan
-    UN_ENV_IABLE_OPTIONS = [:name, :subject, :args]
+    UN_ENV_IABLE_OPTIONS = [:name, :subject, :comparator]
     DEFAULT_TEST_OPTIONS = {
-      :fail_fast => true
+      :fail_fast => false
     }
 
     def prescribe(name, options = {})
-      Value::TestPlan.new(DEFAULT_OPTIONS.
-                          merge(DEFAULT_TEST_OPTIONS).
+      Value::TestPlan.new(DEFAULT_TEST_OPTIONS.
+                          merge(Suture.config).
                           merge(options).
                           merge(:name => name).
                           merge(Suture::Util::Env.to_map(UN_ENV_IABLE_OPTIONS)))