verdict 0.2.0 → 0.2.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: bc058be7df27d778fc5c3d4333c1570c64edccc3
-  data.tar.gz: 7095d5c3a2e6408f29018ea2490f978debe05ed9
+  metadata.gz: 969beb7809f021e2f73e70128dd7565081824240
+  data.tar.gz: 1abbcd8a1c3b5d0776d3649049d8b34cada22cf9
 SHA512:
-  metadata.gz: 0a74b192fd5d4a03a6524e9c8f88d761b8eb2e83e6eb33957f46d429d526d9506eaf9787ae6d3665a80424d077f32d1dbb72498b1c7a81bb6a5380d581662155
-  data.tar.gz: c73434af18dcaf65c05689daf3686ddc42a9bd6334018be0d7369fd577742c92ad6630ca4b82d539ba7ddb3aaa6a379a0a0204aa35cb9375dac3e09bd14b7229
+  metadata.gz: ae0034fd9af434c7fec7be4dd0aabc427dca17c7990e150941251209323cbad98522d3057dd26abc6e1fffcb739e881d5cacc54c636b798cda322c1cdc0b056a
+  data.tar.gz: f1e8ef2ca4f3ddc3995b72b33b3fa3dd2259c4520c8e691362a33e1ccfab9ce42aff6bc336b7723c1505db6a86d3ce4bcc90df07376201fbea208486ab3ef4f0

data/.travis.yml

@@ -3,6 +3,7 @@ script: bundle exec rake
 rvm:
   - 1.9.3
   - 2.0.0
+  - 2.1.0
   - ruby-head
   - rbx
   - jruby-19mode

data/Gemfile

@@ -4,3 +4,7 @@ source 'https://rubygems.org'
 gemspec
 gem "rubysl", platform: :rbx
 gem "json", platform: :rbx
+
+group :development do
+  gem "simplecov", platforms: [:mri, :jruby]
+end

data/README.md

@@ -1,12 +1,14 @@
-# Verdict [![Build Status](https://travis-ci.org/Shopify/verdict.png)](https://travis-ci.org/Shopify/verdict)
+# Verdict
+
+[![Build Status](https://travis-ci.org/Shopify/verdict.png)](https://travis-ci.org/Shopify/verdict)
+[![Code Climate](https://codeclimate.com/github/Shopify/verdict.png)](https://codeclimate.com/github/Shopify/verdict)
 
 This library allows you to define and use experiments in your application.
 
-- This library can be used in any Ruby application, and comes with a `Railtie` to
-  make integrating it with a Rails app easy.
-- This library only handles consistently assigning subjects to experiment groups, 
-  and storing/logging these assignments for analysis. It doesn't do any analysis
-  of results. That should happen elsewhere, e.g. in a data warehouse environment.
+- It can be used in any Ruby application, and comes with a `Railtie` to make integrating it with a Rails app easy.
+- It handles consistently assigning subjects to experiment groups, and storing/logging these assignments for analysis.
+
+__*This library doesn't do any analysis of results. That should happen elsewhere, e.g. in a data warehouse environment.*__
 
 
 ## Installation
@@ -15,12 +17,13 @@ Add this line to your application's Gemfile, and run `bundle install`:
 
     gem 'verdict'
 
+If you're using Rails, the Railtie will handle setting the logger to `Rails.logger` and the experiments directory to `app/experiments`. It will also load the rake tasks for you (run `bundle exec rake -T | grep experiments:` for options).
+
 ## Usage
 
-This gem contains the `Verdict::Experiment` model used create the experiment instance,
-in order consistently modify application behaviour based on an object's unique key. 
+The `Verdict::Experiment` class is used to create an experiment, define control and experiment groups, and to qualify subjects.
 
-Define an experiment like so:
+You define an experiment like so:
 
 ``` ruby
 Verdict::Experiment.define :my_experiment do
@@ -39,46 +42,66 @@ Verdict::Experiment.define :my_experiment do
 end
 ```
 
-Usually you want to place this in a file called **my_experiment.rb** in the 
-**/app/experiments** folder. Also, usually you want to subclass `Verdict::Experiment` 
-to set some default options for your app's environment, and call `define` on that class
-instead.
+Usually you'll want to place this in a file called **my_experiment.rb** in the
+**/app/experiments** folder.
+
+_We recommend that you subclass `Verdict::Experiment` to set some default options for your app's environment, and call `define` on that class instead._
 
-Refer to the experiment elsewhere in your codebase like this:
+### Determining a Subject's Group
+
+At the relevant point in your application, you can check the group that a particular subject belongs to using the `switch` method.
+
+You'll need to pass along the subject (think User, Product or any other Model class) as well as any context to be used for qualifying the subject.
 
 ``` ruby
-context = { ... } # anything you want to pass along to the qualify block. 
+context = { ... } # anything you want to pass along to the qualify block.
 case Verdict['my experiment'].switch(shop, context)
 when :test
   # Handle test group
 when :control
   # Handle control group
-else 
-  # Handle unqualified subjects. 
+else
+  # Handle unqualified subjects.
 end
 ```
 
-## Storage & logging
+## Storage
+
+Verdict uses a very simple interface for storing experiment assignments. Out of the box, Verdict ships with storage providers for:
+
+* Memory
+* Redis
+
+You can set up storage for your experiment by calling the `storage` method with
+an object that responds to the following methods:
+
+* `store_assignment(assignment)`
+* `retrieve_assignment(experiment, subject_identifier)`
+* `remove_assignment(experiment, subject_identifier)`
+* `clear_experiment(experiment)`
+* `retrieve_start_timestamp(experiment)`
+* `store_start_timestamp(experiment, timestamp)`
+
+Regarding the method signatures above, `experiment` is the Experiment instance, `subject_identifier` is a string that uniquely identifies the subject, and `assignment` is a `Verdict::Assignment` instance.
+
+By default it will use `subject.id.to_s` as `subject_identifier`, but you can change that by overriding `def subject_identifier(subject)` on the experiment.
+
+Storage providers simply store subject assignments and require quick lookups of subject identifiers. They allow for complex (high CPU) assignments, and for assignments that might not always put the same subject in the same group by storing the assignment for later use.
+
+Storage providers are intended for operational use and should not be used for data analysis. For data analysis, you should use the logger.
+
+For more details about these methods, check out the source code for [Verdict::Storage::MockStorage](lib/verdict/storage/mock.rb)
+
+## Logging
 
-The library uses a basic interface to store experiment assignments. Except for
-a development-only memory store, it doesn't include any storage models.
+Every assignment will be logged to `Verdict.logger`. For rails apps, this logger will be automatically set to `Rails.logger` so experiment assignments will show up in your Rails log.
 
-You can set up storage by calling the `storage` method of your experiment, with
-an object that responds to the following tho methods:
+You can override the logging by overriding the `def log_assignment(assignment)` method on the experiment.
 
-- `def retrieve_assignment(experiment, subject_identifier)`
-- `def store_assignment(assignment)`
+Logging (as opposed to storage) should be used for data analysis. The logger requires a write-only / forward-only stream to write to, e.g. a log file, Kafka, or an insert-only database table.
 
-In which `experiment` is the Experiment instance, `subject_identifier` is a  
-string that uniquely identifies the subject, and `assignment` is an
-`Verdict::Assignment` instance. By default it will use `subject.id.to_s` as
-`subject_identifier`, but you can change that by overriding the 
-`def subject_identifier(subject)` method on the experiment.
+It's possible to run an experiment without defining any storage, though this comes with several drawbacks. Logging on the other hand is required in order to analyze the results.
 
-The library will also log every assignment to `Verdict.logger`. The Railtie
-sets `Verdict.logger` to `Rails.logger`, so experiment assignments will show
-up in your Rails log. You can override the logging by overriding the 
-`def log_assignment(assignment)` method on the experiment.
 
 ## Contributing
 

data/lib/verdict.rb

@@ -1,4 +1,5 @@
 require 'logger'
+require 'digest/md5'
 
 module Verdict
   extend self
@@ -46,7 +47,7 @@ require "verdict/experiment"
 require "verdict/group"
 require "verdict/assignment"
 require "verdict/conversion"
-require "verdict/segmenter"
+require "verdict/segmenters"
 require "verdict/storage"
 require "verdict/event_logger"
 

data/lib/verdict/experiment.rb

@@ -16,7 +16,7 @@ class Verdict::Experiment
     options = default_options.merge(options)
     @qualifier                   = options[:qualifier]
     @event_logger                = options[:event_logger] || Verdict::EventLogger.new(Verdict.default_logger)
-    @subject_storage             = options[:storage] || Verdict::Storage::MemoryStorage.new
+    @subject_storage             = storage(options[:storage] || :memory)
     @store_unqualified           = options[:store_unqualified]
     @segmenter                   = options[:segmenter]
     @subject_type                = options[:subject_type]
@@ -25,6 +25,7 @@ class Verdict::Experiment
     instance_eval(&block) if block_given?
   end
 
+
   def subject_type(type = nil)
     return @subject_type if type.nil?
     @subject_type = type
@@ -38,7 +39,7 @@ class Verdict::Experiment
     segmenter.groups[handle.to_s]
   end
 
-  def groups(segmenter_class = Verdict::FixedPercentageSegmenter, &block)
+  def groups(segmenter_class = Verdict::Segmenters::FixedPercentageSegmenter, &block)
     return segmenter.groups unless block_given?
     @segmenter ||= segmenter_class.new(self)
     @segmenter.instance_eval(&block)
@@ -47,7 +48,7 @@ class Verdict::Experiment
   end
 
   def rollout_percentage(percentage, rollout_group_name = :enabled)
-    groups(Verdict::RolloutSegmenter) do
+    groups(Verdict::Segmenters::RolloutSegmenter) do
       group rollout_group_name, percentage
     end
   end
@@ -56,9 +57,16 @@ class Verdict::Experiment
     @qualifier = block
   end
 
-  def storage(subject_storage, options = {})
+  def storage(subject_storage = nil, options = {})
+    return @subject_storage if subject_storage.nil?
+    
     @store_unqualified = options[:store_unqualified] if options.has_key?(:store_unqualified)
-    @subject_storage = subject_storage
+    @subject_storage = case subject_storage
+      when :memory; Verdict::Storage::MemoryStorage.new
+      when :none;   Verdict::Storage::MockStorage.new
+      when Class;   subject_storage.new
+      else          subject_storage
+    end
   end
 
   def segmenter
@@ -184,7 +192,7 @@ class Verdict::Experiment
 
   def to_json(options = {})
     as_json(options).to_json
-  end 
+  end
 
   def fetch_subject(subject_identifier)
     raise NotImplementedError, "Fetching subjects based in identifier is not implemented for eperiment @{handle.inspect}."
@@ -221,7 +229,7 @@ class Verdict::Experiment
       return previous_assignment unless previous_assignment.nil?
       group = segmenter.assign(subject_identifier, subject, context)
       subject_assignment(subject_identifier, group, nil, group.nil?)
-    else 
+    else
       subject_assignment(subject_identifier, nil, nil)
     end
   end

data/lib/verdict/segmenters.rb

@@ -0,0 +1,4 @@
+require 'verdict/segmenters/base_segmenter'
+require 'verdict/segmenters/static_segmenter'
+require 'verdict/segmenters/fixed_percentage_segmenter'
+require 'verdict/segmenters/rollout_segmenter'

data/lib/verdict/segmenters/base_segmenter.rb

@@ -0,0 +1,84 @@
+module Verdict
+  module Segmenters
+    # Base class of all segmenters.
+    #
+    # The segmenter is responsible for assigning subjects to groups. You can
+    # implement any assignment strategy you like by subclassing this class and
+    # using it in your experiment.
+    #
+    # - You should implement the register_group method for the experiment definition DSL
+    #   to make the system aware of the groups that the segmenter could return.
+    # - The verify! method is called after all the groups have been defined, so it can
+    #   detect internal inconsistencies in the group definitions.
+    # - The assign method is where your assignment magic lives.
+    class BaseSegmenter
+      # The experiment to which this segmenter is associated
+      attr_reader :experiment
+
+      # A hash of the groups that are defined in this experiment, indexed by their
+      # handle. The assign method should return one of the groups in this hash
+      attr_reader :groups
+
+      def initialize(experiment)
+        @experiment = experiment
+        @groups = {}
+      end
+
+      # DSL method to register a group. It calls the register_group method of the
+      # segmenter implementation
+      def group(handle, *args, &block)
+        group = register_group(handle, *args)
+        @groups[group.handle] = group
+        group.instance_eval(&block) if block_given?
+      end
+
+      # The group method is called from the experiment definition DSL.
+      # It should register a new group to the segmenter, with the given handle.
+      #
+      # - The handle parameter is a symbol that uniquely identifies the group within
+      #   this experiment.
+      # - The return value of this method should be a Verdict::Group instance.
+      def register_group(handle, *args)
+        raise NotImplementedError
+      end
+
+      # The verify! method is called after all the groups have been defined in the
+      # experiment definition DSL. You can run any consistency checks in this method,
+      # and if anything is off, you can raise a Verdict::SegmentationError to
+      # signify the problem.
+      def verify!
+        # noop by default
+      end
+
+      # The assign method is called to assign a subject to one of the groups that have been defined
+      # in the segmenter implementation.
+      #
+      # - The identifier parameter is a string that uniquely identifies the subject.
+      # - The subject paramater is the  subject instance that was passed to the framework,
+      #   when the application code calls Experiment#assign or Experiment#switch.
+      # - The context parameter is an object that was passed to the framework, you can use this
+      #   object any way you like in your segmenting logic.
+      #
+      # This method should return the Verdict::Group instance to which the subject should be assigned.
+      # This instance should be one of the group instance that was registered in the definition DSL.
+      def assign(identifier, subject, context)
+        raise NotImplementedError
+      end
+
+
+      # This method is called whenever a subjects converts to a goal, i.e., when Experiment#convert
+      # is called. You can use this to implement a feedback loop in your segmenter.
+      #
+      # - The identifier parameter is a string that uniquely identifies the subject.
+      # - The subject paramater is the  subject instance that was passed to the framework,
+      #   when the application code calls Experiment#assign or Experiment#switch.
+      # - The conversion parameter is a Verdict::Conversion instance that describes what
+      #   goal the subject converted to.
+      #
+      # The return value of this method is not used.
+      def conversion_feedback(identifier, subject, conversion)
+        # noop by default
+      end
+    end
+  end
+end

data/lib/verdict/segmenters/fixed_percentage_segmenter.rb

@@ -0,0 +1,56 @@
+module Verdict
+  module Segmenters
+    class FixedPercentageSegmenter < BaseSegmenter
+
+      def initialize(experiment)
+        super
+        @total_percentage_segmented = 0
+      end
+
+      def verify!
+        raise Verdict::SegmentationError, "Should segment exactly 100% of the cases, but segments add up to #{@total_percentage_segmented}%." if @total_percentage_segmented != 100
+      end
+
+      def register_group(handle, size)
+        percentage = size.kind_of?(Hash) && size[:percentage] ? size[:percentage] : size
+        n = case percentage
+          when :rest; 100 - @total_percentage_segmented
+          when :half; 50
+          when Integer; percentage
+          else Integer(percentage)
+        end
+
+        group = Group.new(experiment, handle, @total_percentage_segmented ... (@total_percentage_segmented + n))
+        @total_percentage_segmented += n
+        return group
+      end
+
+      def assign(identifier, subject, context)
+        percentile = Digest::MD5.hexdigest("#{@experiment.handle}#{identifier}").to_i(16) % 100
+        groups.values.find { |group| group.percentile_range.include?(percentile) }
+      end
+
+      class Group < Verdict::Group
+
+        attr_reader :percentile_range
+
+        def initialize(experiment, handle, percentile_range)
+          super(experiment, handle)
+          @percentile_range = percentile_range
+        end
+
+        def percentage_size
+          percentile_range.end - percentile_range.begin
+        end
+
+        def to_s
+          "#{handle} (#{percentage_size}%)"
+        end
+
+        def as_json(options = {})
+          super(options).merge(percentage: percentage_size)
+        end
+      end
+    end
+  end
+end

data/lib/verdict/segmenters/rollout_segmenter.rb

@@ -0,0 +1,55 @@
+module Verdict
+  module Segmenters
+    class RolloutSegmenter < BaseSegmenter
+
+      def initialize(experiment)
+        super
+        @total_percentage_segmented = 0
+      end
+
+      def verify!
+        raise Verdict::SegmentationError, "Should segment less than 100% of the cases, but segments add up to #{@total_percentage_segmented}%." if @total_percentage_segmented >= 100
+      end
+
+      def register_group(handle, size)
+        percentage = size.kind_of?(Hash) && size[:percentage] ? size[:percentage] : size
+        n = case percentage
+          when :rest; 100 - @total_percentage_segmented
+          when :half; 50
+          when Integer; percentage
+          else Integer(percentage)
+        end
+
+        group = Group.new(experiment, handle, @total_percentage_segmented ... (@total_percentage_segmented + n))
+        @total_percentage_segmented += n
+        return group
+      end
+
+      def assign(identifier, subject, context)
+        percentile = Digest::MD5.hexdigest("#{@experiment.handle}#{identifier}").to_i(16) % 100
+        groups.values.find { |group| group.percentile_range.include?(percentile) }
+      end
+
+      class Group < Verdict::Group
+        attr_reader :percentile_range
+
+        def initialize(experiment, handle, percentile_range)
+          super(experiment, handle)
+          @percentile_range = percentile_range
+        end
+
+        def percentage_size
+          percentile_range.end - percentile_range.begin
+        end
+
+        def to_s
+          "#{handle} (#{percentage_size}%)"
+        end
+
+        def as_json(options = {})
+          super(options).merge(percentage: percentage_size)
+        end
+      end
+    end
+  end
+end

data/lib/verdict/segmenters/static_segmenter.rb

@@ -0,0 +1,27 @@
+module Verdict
+  module Segmenters
+    class StaticSegmenter < BaseSegmenter
+
+      def register_group(handle, subject_identifiers)
+        Group.new(experiment, handle, subject_identifiers)
+      end
+
+      def assign(identifier, subject, context)
+        groups.values.find { |group| group.subject_identifiers.include?(identifier) }
+      end
+
+      class Group < Verdict::Group
+        attr_reader :subject_identifiers
+
+        def initialize(experiment, handle, subject_identifiers)
+          super(experiment, handle)
+          @subject_identifiers = subject_identifiers
+        end
+
+        def as_json(options = {})
+          super(options).merge(subject_identifiers: subject_identifiers)
+        end
+      end
+    end
+  end
+end