dat-science 0.0.0 → 1.0.0

data/README.md

@@ -1,26 +1,33 @@
-# Dat Science!
+# Science!
 
-A Ruby library for carefully refactoring critical paths. Science isn't
-a feature flipper or an A/B testing tool, it's a pattern that helps
-measure and validate large code changes without altering behavior.
+A Ruby library for carefully refactoring critical paths. Science isn't a feature
+flipper or an A/B testing tool, it's a pattern that helps measure and validate
+large code changes without altering behavior.
 
 ## How do I do science?
 
+Let's pretend you're changing the way you handle permissions in a large web app.
+Tests can help guide your refactoring, but you really want to compare the
+current and new behaviors live, under load.
+
 ```ruby
 require "dat/science"
 
-include Dat::Science
+class MyApp::Widget
+  def allows?(user)
+    experiment = Dat::Science::Experiment.new "widget-permissions" do |e|
+      e.control   { model.check_user(user).valid? } # old way
+      e.candidate { user.can? :read, model } # new way
+    end
 
-science "user-permissions" do |experiment|
-  experiment.control   { model.check_user(user).valid? }
-  experiment.candidate { user.can? :read, model }
+    experiment.run
+  end
 end
 ```
 
-Wrap a `control` block around the code's original behavior, and wrap
-`candidate` around the new behavior. The `science` block will return
-whatever the `control` block returns, but it does a bunch of stuff
-behind the scenes:
+Wrap a `control` block around the code's original behavior, and wrap `candidate`
+around the new behavior. `experiment.run` will always return whatever the
+`control` block returns, but it does a bunch of stuff behind the scenes:
 
 * Decides whether or not to run `candidate`,
 * Runs `candidate` before `control` 50% of the time,
@@ -29,32 +36,60 @@ behind the scenes:
 * Swallows any exceptions raised by the candidate behavior, and
 * Publishes all this information for tracking and reporting.
 
-## Making Science Useful
+If you'd like a bit less verbosity, the `Dat::Science#science` helper
+instantiates an experiment and calls `run`:
+
+```ruby
+require "dat/science"
 
-(Talk about subclassing `Dat::Science::Experiment` and setting
-`Dat::Science.experiment`)
+class MyApp::Widget
+  include Dat::Science
+
+  def allows?(user)
+    science "widget-permissions" do |e|
+      e.control   { model.check_user(user).valid? } # old way
+      e.candidate { user.can? :read, model } # new way
+    end
+  end
+end
+```
+
+## Making science useful
+
+The examples above will run, but they're not particularly helpful. The
+`candidate` block runs every time, and none of the results get
+published. Let's fix that by creating an app-specific sublass of
+`Dat::Science::Experiment`. This makes it easy to add custom behavior
+for enabling/disabling/throttling experiments and publishing results.
 
 ```ruby
 require "dat/science"
 
-module FooCorp
+module MyApp
   class Experiment < Dat::Science::Experiment
     def enabled?
-      # See "Ramping up Experiments" below.
+      # See "Ramping up experiments" below.
     end
 
     def publish(name, payload)
-      # See "Publishing Results" below.
+      # See "Publishing results" below.
     end
   end
 end
 ```
 
+After creating a subclass, tell `Dat::Science` to instantiate it any time the
+`science` helper is called:
+
 ```ruby
-Dat::Science.experiment = FooCorp::Experiment
+Dat::Science.experiment = MyApp::Experiment
 ```
 
-### Ramping up Experiments
+### Ramping up experiments
+
+By default the `candidate` block of an experiment will run 100% of the time.
+This is often a really bad idea when testing live. `Experiment#enabled?` can be
+overridden to run all candidates, say, 10% of the time:
 
 ```ruby
 def enabled?
@@ -62,22 +97,39 @@ def enabled?
 end
 ```
 
+Or, even better, use a feature flag library like [Flipper][]. Delegating the
+decision makes it easy to define different rules for each experiment, and can
+help keep all your entropy concerns in one place.
+
+[Flipper]: https://github.com/jnunemaker/flipper
+
 ```ruby
 def enabled?
-  Flipper[name].enabled?
+  MyApp.flipper[name].enabled?
 end
 ```
 
-### Publishing Results
+### Publishing results
+
+By default the results of an experiment are discarded. This isn't very useful.
+`Experiment#publish` can be overridden to publish results via any
+instrumentation mechansim, which makes it easy to graph durations or
+matches/mismatches and store results. The only two events published by an
+experiment are `match` when the result of the control and candidate behaviors
+are the same, and `mismatch` when they aren't.
 
 ```ruby
-def publish(name, payload)
-  FooCorp.instrument "science.#{name}", payload
+def publish(event, payload)
+  MyApp.instrument "science.#{event}", payload
 end
 ```
 
+The published `payload` is a Symbol-keyed Hash:
+
 ```ruby
 {
+  :experiment => "widget-permissions",
+
   :candidate => {
     :duration  => 2.5,
     :exception => nil,
@@ -94,13 +146,35 @@ end
 }
 ```
 
-#### Adding Context
+The `:candidate` and `:control` Hashes have the same keys:
+
+* `:duration` is the execution in ms, expressed as a float.
+* `:exception` is a reference to any raised exception or `nil`.
+* `:value` is the result of the block.
+
+`:first` is either `:candidate` or `:control`, depending on which block was run
+first during the experiment. `:experiment` is the name of the experiment.
+
+#### Adding context
+
+It's often useful to add more information to your results, and
+`Experiment#context` makes it easy:
+
+```ruby
+science "widget-permissions" do |experiment|
+  experiment.context :user => user
+
+  experiment.control   { model.check_user(user).valid? } # old way
+  experiment.candidate { user.can? :read, model } # new way
+end
+```
 
-(using `e.context`)
+`context` takes a Symbol-keyed Hash of additional information to publish and
+merges it with the default payload.
 
-## Hacking on Science
+## Hacking on science
 
-Make sure a modern Bundler is available.`script/test` runs the unit
-tests. All development dependencies will be installed automatically if
-they're not available. Dat science happens primarily on Ruby 1.9.3 and
-1.8.7, but science should be universal.
+Be on a Unixy box. Make sure a modern Bundler is available. `script/test` runs
+the unit tests. All development dependencies will be installed automatically if
+they're not available. Dat science happens primarily on Ruby 1.9.3 and 1.8.7,
+but science should be universal.

data/dat-science.gemspec

@@ -1,6 +1,6 @@
 Gem::Specification.new do |gem|
   gem.name          = "dat-science"
-  gem.version       = "0.0.0"
+  gem.version       = "1.0.0"
   gem.authors       = ["John Barnette", "Rick Bradley"]
   gem.email         = ["jbarnette@github.com"]
   gem.description   = "Gradually test, measure, and track refactored code."

data/lib/dat/science/experiment.rb

@@ -9,8 +9,8 @@ module Dat
       # Public: The name of this experiment.
       attr_reader :name
 
-      # Public: Create a new experiment instance. `self` is yielded to
-      # an optional `block` if it's provided.
+      # Public: Create a new experiment instance. `self` is yielded to an
+      # optional `block` if it's provided.
       def initialize(name, &block)
         @candidate  = nil
         @context    = { :experiment => name }
@@ -20,37 +20,37 @@ module Dat
         yield self if block_given?
       end
 
-      # Public: Add a Hash of `payload` data to be included when events
-      # are published.
-      def context(payload)
-        @context.merge! payload
+      # Public: Declare the candidate behavior `block` for this experiment.
+      # Returns `block`.
+      def candidate(&block)
+        @candidate = block if block
+        @candidate
       end
 
-      # Public: Declare the control behavior `block` for this
-      # experiment. Returns `block`.
-      def control(&block)
-        @control = block
+      # Public: Add a Hash of `payload` data to be included when events are
+      # published or returns the current context if `payload` is `nil`.
+      def context(payload = nil)
+        @context.merge! payload if payload
+        @context
       end
 
-      # Public: Declare the candidate behavior `block` for this
-      # experiment. Returns `block`.
-      def candidate(&block)
-        @candidate = block
+      # Public: Declare the control behavior `block` for this experiment.
+      # Returns `block`.
+      def control(&block)
+        @control = block if block
+        @control
       end
 
       # Public: Run the control and candidate behaviors, timing each and
-      # comparing the results. The run order is randomized. Returns the
-      # control behavior's result.
+      # comparing the results. The run order is randomized. Returns the control
+      # behavior's result.
       #
-      # If the experiment is disabled or candidate behavior isn't
-      # provided the control behavior's result will be returned
-      # immediately.
+      # If the experiment is disabled or candidate behavior isn't provided the
+      # control behavior's result will be returned immediately.
       def run
         return run_control unless candidate? && enabled?
 
-        control_goes_first = rand(2) == 0
-
-        if control_goes_first
+        if control_runs_first?
           control   = observe_control
           candidate = observe_candidate
         else
@@ -61,10 +61,10 @@ module Dat
         payload = {
           :candidate => candidate.payload,
           :control   => control.payload,
-          :first     => control_goes_first ? :control : :candidate
+          :first     => control_runs_first? ? :control : :candidate
         }
 
-        kind = control == candidate ? "match" : "mismatch"
+        kind = control == candidate ? :match : :mismatch
         publish_with_context kind, payload
 
         raise control.exception if control.raised?
@@ -76,7 +76,13 @@ module Dat
 
       # Internal: Does this experiment have candidate behavior?
       def candidate?
-        !!@candidate
+        !!candidate
+      end
+
+      # Internal: Should the control behavior run first?
+      def control_runs_first?
+        return @control_runs_first if defined? @control_runs_first
+        @control_runs_first = rand(2) == 0
       end
 
       # Internal: Is this experiment enabled? More specifically, should
@@ -112,23 +118,24 @@ module Dat
         observe { run_control }
       end
 
-      # Internal: Broadcast an event `name` and `payload` Hash. The
+      # Internal: Broadcast an `event` String and `payload` Hash. The
       # default implementation is a no-op. Returns nothing.
-      def publish(name, payload)
+      def publish(event, payload)
       end
 
       # Internal: Call `publish`, merging the `payload` with `context`.
-      def publish_with_context(name, payload)
-        publish name, @context.merge(payload)
+      def publish_with_context(event, payload)
+        publish event, context.merge(payload)
       end
+
       # Internal: Run the candidate behavior and return its result.
       def run_candidate
-        @candidate.call
+        candidate.call
       end
 
       # Internal: Run the control behavior and return its result.
       def run_control
-        @control.call
+        control.call
       end
     end
   end

data/lib/dat/science/result.rb

@@ -1,7 +1,7 @@
 module Dat
-
-  # Internal. The value of a watched behavior.
   module Science
+
+    # Internal. The output of running of an observed behavior.
     class Result
       attr_reader :duration
       attr_reader :exception
@@ -14,7 +14,7 @@ module Dat
       end
 
       def ==(other)
-        return false unless other.is_a? Science::Result
+        return false unless other.is_a? Dat::Science::Result
 
         values_are_equal = other.value == value
         both_raised      = other.raised? && raised?

data/script/release

@@ -0,0 +1,38 @@
+#!/bin/sh
+# Tag and push a release.
+
+set -e
+
+# Make sure we're in the project root.
+
+cd $(dirname "$0")/..
+
+# Build a new gem archive.
+
+rm -rf dat-science-*.gem
+gem build -q dat-science.gemspec
+
+# Make sure we're on the master branch.
+
+(git branch | grep -q '* master') || {
+  echo "Only release from the master branch."
+  exit 1
+}
+
+# Figure out what version we're releasing.
+
+tag=v`ls dat-science-*.gem | sed 's/^dat-science-\(.*\)\.gem$/\1/'`
+
+# Make sure we haven't released this version before.
+
+git fetch -t origin
+
+(git tag -l | grep -q "$tag") && {
+  echo "Whoops, there's already a '${tag}' tag."
+  exit 1
+}
+
+# Tag it and bag it.
+
+gem push dat-science-*.gem && git tag "$tag" &&
+  git push origin master && git push origin "$tag"

data/test/dat_science_experiment_test.rb

@@ -2,7 +2,222 @@ require "minitest/autorun"
 require "dat/science/experiment"
 
 class DatScienceExperimentTest < MiniTest::Unit::TestCase
-  def test_sanity
-    assert Dat::Science::Experiment
+  class Experiment < Dat::Science::Experiment
+    def self.published
+      @published ||= []
+    end
+
+    def publish(name, payload)
+      Experiment.published << [name, payload]
+    end
+  end
+
+  def setup
+    Experiment.published.clear
+  end
+
+  def test_initialize
+    in_block   = nil
+    experiment = Experiment.new("foo") { |e| in_block = e }
+
+    assert_equal "foo", experiment.name
+    assert_equal experiment, in_block
+  end
+
+  def test_candidate_default
+    assert_nil Experiment.new("foo").candidate
+  end
+
+  def test_candidate
+    e = Experiment.new "foo"
+    b = lambda {}
+
+    e.candidate &b
+    assert_same b, e.candidate
+  end
+
+  def test_context_default
+    e = Experiment.new "foo"
+
+    expected = { :experiment => "foo" }
+    assert_equal expected, e.context
+  end
+
+  def test_context
+    e = Experiment.new "foo"
+    e.context :bar => :baz
+
+    assert_equal :baz, e.context[:bar]
+  end
+
+  def test_control_default
+    assert_nil Experiment.new("foo").control
+  end
+
+  def test_control
+    e = Experiment.new "foo"
+    b = lambda {}
+
+    e.control &b
+    assert_same b, e.control
+  end
+
+  def test_run_with_no_candidate
+    e = Experiment.new "foo"
+    e.control { :foo }
+
+    assert_equal :foo, e.run
+    assert Experiment.published.empty?
+  end
+
+  def test_run_disabled
+    e = Experiment.new "foo"
+    e.control { :foo }
+    e.candidate { :bar }
+
+    def e.enabled?
+      false
+    end
+
+    assert_equal :foo, e.run
+    assert Experiment.published.empty?
+  end
+
+  def test_run
+    e = Experiment.new "foo"
+    e.control { :foo }
+
+    candidate_run = false
+    e.candidate { candidate_run = true; :bar }
+
+    def e.control_runs_first?
+      true
+    end
+
+    assert_equal :foo, e.run
+    assert candidate_run
+
+    event, payload = Experiment.published.first
+    refute_nil event
+    refute_nil payload
+
+    assert_equal :mismatch, event
+
+    assert_equal "foo", payload[:experiment]
+    assert_equal :control, payload[:first]
+
+    assert payload[:control][:duration]
+    assert_nil payload[:control][:exception]
+    assert_equal :foo, payload[:control][:value]
+
+    assert payload[:candidate][:duration]
+    assert_nil payload[:candidate][:exception]
+    assert_equal :bar, payload[:candidate][:value]
+  end
+
+  def test_run_candidate_first
+    e = Experiment.new "foo"
+    e.control { :foo }
+    e.candidate { :bar }
+
+    def e.control_runs_first?
+      false
+    end
+
+    assert_equal :foo, e.run
+
+    event, payload = Experiment.published.first
+    refute_nil event
+    refute_nil payload
+
+    assert_equal :mismatch, event
+    assert_equal :candidate, payload[:first]
+  end
+
+  def test_run_match
+    e = Experiment.new "foo"
+    e.control { :foo }
+    e.candidate { :foo }
+
+    assert_equal :foo, e.run
+
+    event, payload = Experiment.published.first
+    refute_nil event
+    refute_nil payload
+
+    assert_equal :match, event
+  end
+
+  def test_run_passes_control_exceptions_through
+    e = Experiment.new "foo"
+    e.control { raise "bar" }
+
+    candidate_run = false
+    e.candidate { candidate_run = true }
+
+    ex = assert_raises RuntimeError do
+      e.run
+    end
+
+    assert candidate_run
+    assert_equal "bar", ex.message
+
+    event, payload = Experiment.published.first
+    refute_nil event
+    refute_nil payload
+
+    assert_equal :mismatch, event
+    refute_nil payload[:control][:exception]
+  end
+
+  def test_run_swallows_candidate_exceptions
+    e = Experiment.new "foo"
+    e.control { :foo }
+    e.candidate { raise "bar" }
+
+    assert_equal :foo, e.run
+
+    event, payload = Experiment.published.first
+    refute_nil event
+    refute_nil payload
+
+    assert_equal :mismatch, event
+    refute_nil payload[:candidate][:exception]
+  end
+
+  def test_run_similar_exceptions_are_a_match
+    e = Experiment.new "foo"
+    e.control { raise "foo" }
+    e.candidate { raise "foo" }
+
+    assert_raises RuntimeError do
+      e.run
+    end
+
+    event, payload = Experiment.published.first
+    refute_nil event
+    refute_nil payload
+
+    assert_equal :match, event
+    refute_nil payload[:control][:exception]
+    refute_nil payload[:candidate][:exception]
+  end
+
+  def test_run_dissimilar_exceptions_are_a_mismatch
+    e = Experiment.new "foo"
+    e.control { raise "foo" }
+    e.candidate { raise "bar" }
+
+    assert_raises RuntimeError do
+      e.run
+    end
+
+    event, payload = Experiment.published.first
+    refute_nil event
+    refute_nil payload
+
+    assert_equal :mismatch, event
+    refute_nil payload[:control][:exception]
+    refute_nil payload[:candidate][:exception]
   end
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: dat-science
 version: !ruby/object:Gem::Version
-  version: 0.0.0
+  version: 1.0.0
   prerelease: 
 platform: ruby
 authors:
@@ -60,6 +60,7 @@ files:
 - lib/dat/science/experiment.rb
 - lib/dat/science/result.rb
 - script/bootstrap
+- script/release
 - script/test
 - test/dat_science_experiment_test.rb
 - test/dat_science_test.rb