lab_tech 0.1.0 → 0.1.5

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 2f5e6f3f09101904d4cf6fbc209768dba3f58bdb3ca9bbe157473e7d939fe124
-  data.tar.gz: 88dceb210ec735257a5f2cf329fe132085062b39dd2ad207dae2d0ff16392993
+  metadata.gz: 9ae9dc5a34d835d29b76b516dca2dcd6db786c44be5207f38d007104b1dda180
+  data.tar.gz: e47a125456c58867e841f88301a998e05c5750c7df0c74e6fb1c92907126949a
 SHA512:
-  metadata.gz: b52697112c48077ac53e85af052b0e20e5425f448fba77a8139344afad7bb39c4bf497dcf5987d794d5826e5a6dfd920975a0d509d44fb16779d92342d21ffbd
-  data.tar.gz: d6178294c5066e7117435237b7ba2a93a1bbf15bba7152ead2c6f4393a808c0af1d271478126e50a5948ee133c855fa28ca2589da140a305cc5698271ce0fd81
+  metadata.gz: 3941c70268e050eff64b8673401c66e7457a931c6f6bc906cac0218756068fdbaf8f3e1d1f50460164032f63843aefee8b9483d655514a8b47b6121936bc3592
+  data.tar.gz: ce7f941d62618a9de15face19a82d9fdd9bbea27a742602be37cf95efe9629e55a588fe0f974bfb15817d7f24bceb45036ebaee070d621fa9a6857f22aaba2b1

data/README.md

@@ -17,6 +17,33 @@ for accuracy and performance.  (Please feel free to send those back to us in a
 pull request; we simply haven't needed them for ourselves, so they don't
 exist yet.)
 
+## Why Scientist?
+
+Scientist is a great tool for trying out changes where:
+- comprehensive test coverage is impractical
+- your test suite doesn't give you sufficient confidence in your changes
+- you want detailed performance data on your proposed alternative(s)
+
+## Why LabTech?
+
+Scientist is amazing at **generating** data, but it assumes you'll develop your
+own tools for **recording** and **analyzing** it.  Scientist's README examples
+show interactions with StatsD and Redis, but if you're working in a Rails app,
+odds are *pretty darn good* that:
+
+1. you already have access to a RDBMS and ActiveRecord, and
+2. your throughput isn't so huge that some extra database writes will bring
+   said RDBMS to its knees.
+
+If both of those assumptions are true for your application, LabTech might be a
+good fit for you -- it records experimental results to the database so they're
+easy to query later using ActiveRecord.
+
+(If you're legitimately worried about the I/O load on your RDBMS, you can
+always ramp up your LabTech experiments a percentage point or two at a time,
+keeping an eye on your performance monitoring tools and scaling back as
+needed.)
+
 ## Usage
 
 Once you've installed the gem and run its migrations (as described in
@@ -257,6 +284,8 @@ mismatches by running:
 LabTech.compare_mismatches "spiffy-search", limit: 3
 ```
 
+(To view all mismatches, just leave off the `limit: 3`.)
+
 You have the ability to customize the output of this by passing a block that
 takes a "control" parameter followed by a "candidate" parameter; the return
 value of that block will be printed to the console.  How you do this will
@@ -264,6 +293,34 @@ largely depend on the kind of data you're collecting to validate your
 experiments.  There are several examples in the `lib/lab_tech.rb` file; I
 encourage you to check them out.
 
+If you have errors to inspect as well, you can view these with:
+
+```ruby
+LabTech.summarize_errors "spiffy-search"
+```
+
+Note that the `summarize_errors` method also takes an optional `:limit` keyword
+argument.
+
+### Storing Diffs
+
+If you're working with complex data, you might not want to recompute the diffs
+from console.  As such, LabTech adds a `.diff` method to the experiment.  If
+you call this with a block, that block will be passed the control and candidate
+for each candidate, and its result will be stored on the `LabTech::Observation`
+record.  (See the "diff-generating behavior" spec in
+`spec/models/lab_tech/experiment_spec.rb` for examples.)
+
+### A Note About Experimental Design
+
+Scientist supports experiments with more than one candidate at a time, and
+therefore so does LabTech -- it will record as many candidates as you throw at
+it.  However, if you have multiple candidates, we don't have a good way to
+generate performance charts to compare all of the alternatives, so LabTech just
+doesn't bother printing them.  **If you try this, you're on your own.**  (But
+do let us know how it goes, and feel free to submit a PR if you find a good
+solution!)
+
 ## Installation
 
 **NOTE: As this gem is a Rails engine, we assume you have a Rails application to
@@ -311,11 +368,18 @@ Once that's done, you should be good to go!  See the "Usage" section, above.
 
 ## Contributing
 
-This gem was extracted just before its primary author left Real Geeks, so it's
-not quite clear who's going to take responsibility for the gem.  It's probably
-a good idea to open a GitHub issue to start a conversation before undertaking
-any great amount of work -- though, of course, you're perfectly welcome to fork
-the gem and use your modified version at any time.
+Bug reports and pull requests are welcome on GitHub at
+https://github.com/RealGeeks/lab_tech.
+
+It's probably a good idea to open a GitHub issue to start a conversation before
+undertaking any significant amount of work -- though, as always with F/OSS
+code, you're perfectly welcome to fork the gem and use your modified version at
+any time.
+
+This project is intended to be a safe, welcoming space for collaboration.
+While we have not yet formally adopted a code of conduct, it's probably a Very
+Good Idea to act in accordance with the <a
+href="https://www.contributor-covenant.org/">Contributor Covenant</a>.
 
 ## License
 

data/app/models/lab_tech/experiment.rb

@@ -60,29 +60,19 @@ module LabTech
       @_scientist_comparator
     end
 
-    # TODO: DRY up the io.puts structure between this and summarize_errors
-    def compare_mismatches(limit: nil, io: $stdout, &block)
+    def compare_mismatches(limit: nil, width: 100, io: $stdout, &block)
       mismatches = results.mismatched.includes(:observations)
       return if mismatches.empty?
       mismatches = mismatches.limit(limit) if limit
 
-      io.puts
-      io.puts "=" * 100
-      io.puts "Comparing results for #{name}:"
-      io.puts
-
-      mismatches.each do |result|
-        io.puts
-        io.puts "-" * 100
+      display_results mismatches, label: "Comparing results for #{name}:", io: io do |result|
         io.puts "Result ##{result.id}"
         result.compare_observations( io: io, &block )
-        io.puts "-" * 100
       end
+    end
 
-      io.puts
-      io.puts "=" * 100
-      io.puts
-      nil
+    def diff(&block)
+      @diff_with = block
     end
 
     def disable
@@ -106,7 +96,7 @@ module LabTech
 
     def publish(scientist_result)
       return if Rails.env.test? && !LabTech.publish_results_in_test_mode?
-      LabTech::Result.record_a_science( self, scientist_result )
+      LabTech::Result.record_a_science( self, scientist_result, diff_with: @diff_with )
     end
 
     # I don't encourage the willy-nilly destruction of experimental results...
@@ -125,7 +115,7 @@ module LabTech
       n = delete_and_count.call( LabTech::Observation.where(result_id: self.result_ids) )
       m = delete_and_count.call( self.results )
 
-      update_attributes(
+      update(
         equivalent_count:  0,
         timed_out_count:   0,
         other_error_count: 0,
@@ -140,31 +130,17 @@ module LabTech
       super
     end
 
-    # TODO: DRY up the io.puts structure between this and compare_mismatches
-    def summarize_errors(limit: nil, io: $stdout)
+    def summarize_errors(limit: nil, width: 100, io: $stdout)
       errors = results.other_error
       return if errors.empty?
       errors = errors.limit(limit) if limit
 
-      io.puts
-      io.puts "=" * 100
-      io.puts "Comparing results for #{name}:"
-      io.puts
-
-      errors.each do |result|
-        io.puts
-        io.puts "-" * 100
+      display_results errors, label: "Summarizing errors for #{name}:", io: io do |result|
         io.puts "Result ##{result.id}"
         result.candidates.each do |observation|
-          puts "  * " + observation.exception_class + ":  " + observation.exception_message
+          io.puts "  * " + observation.exception_class + ":  " + observation.exception_message
         end
-        io.puts "-" * 100
       end
-
-      io.puts
-      io.puts "=" * 100
-      io.puts
-      nil
     end
 
     def summarize_results
@@ -186,5 +162,27 @@ module LabTech
       return if cleaner.present?
       clean { |value| LabTech::DefaultCleaner.call(value) }
     end
+
+    def display_results(results, label: nil, width: 100, io: $stdout)
+      return if results.empty?
+
+      io.puts
+      io.puts "=" * width
+      io.puts label if label
+      io.puts
+
+      results.each do |result|
+        io.puts
+        io.puts "-" * width
+        yield result
+        io.puts "-" * width
+      end
+
+      io.puts
+      io.puts "=" * width
+      io.puts
+
+      return nil
+    end
   end
 end

data/app/models/lab_tech/observation.rb

@@ -4,6 +4,9 @@ module LabTech
 
     belongs_to :result, class_name: "LabTech::Result", foreign_key: :result_id, optional: true
 
+    scope :timed_out,   -> {     where(exception_class: 'Timeout::Error') }
+    scope :other_error, -> { where.not(exception_class: 'Timeout::Error') }
+
     serialize :value
 
     def raised_error?

data/app/models/lab_tech/result.rb

@@ -4,8 +4,8 @@ module LabTech
 
     belongs_to :experiment, class_name: "LabTech::Experiment"
     has_many :observations, class_name: "LabTech::Observation", dependent: :destroy
-    has_one :control,     ->() { where("name  = 'control'") }, class_name: "LabTech::Observation"
-    has_many :candidates, ->() { where("name != 'control'") }, class_name: "LabTech::Observation"
+    has_one :control,     ->() {     where(name: 'control') }, class_name: "LabTech::Observation"
+    has_many :candidates, ->() { where.not(name: 'control') }, class_name: "LabTech::Observation"
     serialize :context
 
     # NOTE: I don't think this accounts for the possibility that both the
@@ -14,23 +14,17 @@ module LabTech
     scope :correct,     -> { where( equivalent: true,  raised_error: false ) }
     scope :mismatched,  -> { where( equivalent: false, raised_error: false ) }
     scope :errored,     -> { where( equivalent: false, raised_error: true ) }
-    is_timeout = ->(is_or_is_not) {
-      col      = LabTech::Observation.table_name + ".exception_class"
-      operator = is_or_is_not ? "=" : "!="
-      value    = '"Timeout::Error"'
-      [ col, operator, value ].join(" ")
-    }
-    scope :timed_out,   -> { errored.joins(:candidates).where( is_timeout.(true)  ) }
-    scope :other_error, -> { errored.joins(:candidates).where( is_timeout.(false) ) }
+    scope :timed_out,   -> { errored.joins(:candidates).merge(Observation.timed_out) }
+    scope :other_error, -> { errored.joins(:candidates).merge(Observation.other_error) }
 
     after_create :increment_experiment_counters
 
 
     ##### CLASS METHODS #####
 
-    def self.record_a_science( experiment, scientist_result )
+    def self.record_a_science( experiment, scientist_result, **kwargs )
       self.create!(experiment: experiment) do |result|
-        result.record_a_science scientist_result
+        result.record_a_science scientist_result, **kwargs
       end
     end
 
@@ -56,7 +50,7 @@ module LabTech
       return nil
     end
 
-    def record_a_science(scientist_result)
+    def record_a_science(scientist_result, diff_with: nil)
       unless scientist_result.kind_of?( Scientist::Result )
         raise ArgumentError, "expected a Scientist::Result but got #{scientist_result.class}"
       end
@@ -65,7 +59,8 @@ module LabTech
 
       record_observation scientist_result.control
       scientist_result.candidates.each do |candidate|
-        record_observation candidate
+        diff = diff_with&.call(scientist_result.control, candidate)
+        record_observation candidate, diff: diff
       end
 
       record_simple_stats scientist_result
@@ -99,8 +94,9 @@ module LabTech
       end
     end
 
-    def record_observation(scientist_observation)
+    def record_observation(scientist_observation, attrs = {})
       self.observations.build do |observation|
+        observation.assign_attributes attrs if attrs.present?
         observation.record_a_science scientist_observation
       end
     end

data/app/models/lab_tech/summary.rb

@@ -2,8 +2,6 @@ module LabTech
   class Summary
     TAB  = " " * 4
     LINE = "-" * 80
-    VAL = "█"
-    DOT = "·"
 
     def initialize(experiment)
       @experiment = experiment
@@ -46,10 +44,9 @@ module LabTech
     def add_speedup_chart_to(s)
       s.puts
       s.puts "Speedups (by percentiles):"
-      speedup_magnitude = @speedup_factors.minmax.map(&:to_i).map(&:abs).max.ceil
-      speedup_magnitude = 25 if speedup_magnitude.zero?
       (0..100).step(5) do |n|
-        s.puts TAB + speedup_summary_line(n, speedup_magnitude)
+        line = SpeedupLine.new(n, @speedup_factors)
+        s.puts TAB + line.to_s
       end
     end
 
@@ -100,83 +97,12 @@ module LabTech
       end
     end
 
-    def highlight_bar(bar)
-      left, right = bar.split(VAL)
-
-      left  = left         .gsub("  ", " #{DOT}")
-      right = right.reverse.gsub("  ", " #{DOT}").reverse
-
-      left + VAL + right
-    end
-
-    def humanize(n)
-      width = number_helper.number_with_delimiter( @counts[:results] ).length
-      "%#{width}s" % number_helper.number_with_delimiter( n )
-    end
-
-    def pad_left(s, width)
-      n = [ ( width - s.length ), 0 ].max
-      [ " " * n , s ].join
-    end
-
-    def normalized_bar(x, magnitude, bar_scale: 25, highlight: false)
-      neg, pos = " " * bar_scale, " " * bar_scale
-      normalized = ( bar_scale * ( x.abs / magnitude ) ).floor
-
-      # Select an index that's as close to `normalized` as possible without generating IndexErrors
-      # (TODO: actually understand the math involved so I don't have to chop the ends off like an infidel)
-      index = [ 0, normalized ].max
-      index = [ index, bar_scale - 1 ].min
-
-      case
-      when x == 0 ; mid = VAL
-      when x <  0 ; mid = DOT ; neg[ index ] = VAL ; neg = neg.reverse
-      when x  > 0 ; mid = DOT ; pos[ index ] = VAL
-      end
-
-      bar = "[%s%s%s]" % [ neg, mid, pos ]
-      bar = highlight_bar(bar) if highlight
-      bar
-    end
-
-    def number_helper
-      @_number_helper ||= Object.new.tap {|o| o.send :extend, ActionView::Helpers::NumberHelper }
-    end
-
-    def rate(n)
-      "%2.2f%%" % ( 100.0 * n / @counts[:results] )
-    end
-
-    def speedup_summary_line(n, speedup_magnitude)
-      highlight = n == 50
-      label = "%3d%%" % n
-
-      speedup_factor = LabTech::Percentile.call(n, @speedup_factors)
-      rel_speedup    = "%+.1fx" % speedup_factor
-      bar            = normalized_bar( speedup_factor, speedup_magnitude, highlight: highlight)
-
-      speedup_cue    = pad_left( rel_speedup, speedup_width )
-      speedup_cue += " faster" if speedup_factor > 0
-
-      "#{label}  #{bar}  #{speedup_cue}"
-    end
-
-    def speedup_width
-      @_speedup_width ||= [
-        1, # sign
-        4, # digits
-        1, # decimal point
-        1, # digit after decimal point
-      ].sum
-    end
-
     def summarize_count(s, count_name, label = nil)
-      count = @counts[count_name]
-      return if count.zero?
-
+      n     = @counts[count_name]
       total = @counts[:results]
-      label ||= count_name.to_s
-      s.puts "%s of %s (%s) %s" % [ humanize( count ), humanize( total ), rate( count ), label ]
+      count = Count.new(count_name, n, total, label)
+      return if count.zero?
+      s.puts count.to_s
     end
 
   end

data/app/models/lab_tech/summary/count.rb

@@ -0,0 +1,44 @@
+module LabTech
+class Summary
+
+  class Count
+    attr_reader :name, :n, :total, :label
+
+    def initialize(name, n, total, label = nil)
+      @name  = name
+      @n     = n
+      @total = total
+      @label = label || name.to_s
+    end
+
+    def zero?
+      n.zero?
+    end
+
+    def to_s
+      "%s of %s (%s) %s" % [
+        humanize( n ),
+        humanize( total ),
+        rate( n ),
+        label
+      ]
+    end
+
+    private
+
+    def humanize(n)
+      width = number_helper.number_with_delimiter( n ).length
+      "%#{width}s" % number_helper.number_with_delimiter( n )
+    end
+
+    def number_helper
+      @_number_helper ||= Object.new.tap {|o| o.send :extend, ActionView::Helpers::NumberHelper }
+    end
+
+    def rate(n)
+      "%2.2f%%" % ( 100.0 * n / total )
+    end
+  end
+
+end
+end