verdict 0.12.0 → 0.15.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 7e99d3cb09e2dbfb3cafeb6f05171978721cd5a79fd4570f8b6bd6c6e4975e17
-  data.tar.gz: 4a885cb9d8a991046e3a22e428a35982bf7565b03aaa49fa0da0b9963e34bc56
+  metadata.gz: 50c44fd8fcc0cf5605c9d21fd5eefcd4dddeffeb03c48e7ec046639d7154715b
+  data.tar.gz: d55759fff1145aa793f9ae42232518130ba93efcb5fb62a2f7cbfa24d0322a0f
 SHA512:
-  metadata.gz: 89c2d20c1892c630942099ac0b0ea1d3db012a6aaa9031a7bbf8d834fad295adfaf4cc0194d09930608c4416bbeb8efba89852446e02b091c237d851d8266704
-  data.tar.gz: cdb0271d66903d1f39316c2665c68bf8deae733c105e3aecc8bb665adaef4981129bb0f7cce6baf001eb7f246af50e40d85115bf3105525a44b2111fb2c41fbc
+  metadata.gz: 8ed63a2aaf52ebffe454da41e3022e87df834b80130cb0ab14902466ea6d630ce9d046e1b202e32fc6491a2dc9974f4f825f50aaa14aa862103888f31d5c3df6
+  data.tar.gz: 12c379615c6593caf46749157ebca884d91fb17a97d81b7fc60c53cd3531785a0a0c078e79e4b387cd21676f26d55962e0a2ab1ba0caeacb5d99208c3f1e8f8d

data/.gitignore

@@ -16,3 +16,4 @@ test/tmp
 test/version_tmp
 tmp
 dump.rdb
+log

data/.travis.yml

@@ -3,13 +3,14 @@ cache: bundler
 before_install: gem update bundler
 script: bundle exec rake
 rvm:
-  - 2.0
-  - 2.1
-  - 2.3.3
+  - 2.5
+  - 2.6
+  - 2.7
   - ruby-head
   - jruby
 matrix:
   allow_failures:
     - rvm: ruby-head
+    - rvm: jruby
 services:
   - redis-server

data/CHANGELOG.md

@@ -1,8 +1,26 @@
+## v0.15.2
+* Fix edge case where inheriting from `Verdict::Experiment` and overriding `subject_qualifies?` resulted in an error.
+
+## v0.15.1
+* Make the `dynamic_qualifies` parameter in `Verdict::Experiment#subject_qualifies?` optional. This fixes a bug where users that were previously calling this method directly experienced issues after v0.15.0
+
+## v0.15.0
+* Add optional `qualifiers` parameter to the `Verdict::Experiment#switch` method. This parameter accepts an array of procs and is used as additional qualifiers. The purpose of this parameter is to allow users to define qualification logic outside of the experiment definition.
+
+## v0.14.0
+* Add optional experiment definition method `schedule_stop_new_assignment_timestamp` to support limiting experiment's assignment lifetime with another pre-determined time interval. It allows users to have an assignment cooldown period for stable analysis of the experiment results. Experiment's lifetime now becomes: start experiment -> stop new assignments -> end experiment. 
+
+## v0.13.0
+
+* Add optional experiment definition methods `schedule_start_timestamp` and `schedule_end_timestamp` to support limiting experiment's lifetime in a pre-determined time interval.
+* Support eager loading from within a Rails app using Zeitwerk.
+* Add `CookieStorage` storage backend. This backend is a distributed store for Verdict and does not support experiment timestamps. It is designed to be used with Rails applications and requires that `.cookies` be set to the `CookieJar` instance before use.
+
 ## v0.12.0
 
 * Allow options to be passed to `Experiment#cleanup` so they can be forwarded to storage.
 
-* Changed `Experiment#cleanup` to accept an argument of type `Verdict::Experiment`.       
+* Changed `Experiment#cleanup` to accept an argument of type `Verdict::Experiment`.
   Passing a `String`/`Symbol` argument is still supported, but will log a deprecation warning.
 
 ## v0.11.0

data/CONTRIBUTING.md

@@ -0,0 +1,7 @@
+## Contributing
+
+1. Fork it
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes, including tests (`git commit -am 'Added some feature'`)
+4. Push to the branch (`git push origin my-new-feature`)
+5. Create new Pull Request

data/README.md

@@ -19,6 +19,8 @@ Add this line to your application's Gemfile, and run `bundle install`:
 
 If you're using Rails, the Railtie will handle setting the logger to `Rails.logger` and the experiments directory to `app/experiments`. It will also load the rake tasks for you (run `bundle exec rake -T | grep experiments:` for options).
 
+You may find the [Concepts](docs/concepts.md) documentation a good place to familiarise yourself with Verdict's nomenclature.
+
 ## Usage
 
 The `Verdict::Experiment` class is used to create an experiment, define control and experiment groups, and to qualify subjects.
@@ -42,8 +44,7 @@ Verdict::Experiment.define :my_experiment do
 end
 ```
 
-Usually you'll want to place this in a file called **my_experiment.rb** in the
-**/app/experiments** folder.
+Usually you'll want to place this in a file called **my_experiment.rb** in the **/app/experiments** folder.
 
 _We recommend that you subclass `Verdict::Experiment` to set some default options for your app's environment, and call `define` on that class instead._
 
@@ -71,9 +72,9 @@ Verdict uses a very simple interface for storing experiment assignments. Out of
 
 * Memory
 * Redis
+* Cookies
 
-You can set up storage for your experiment by calling the `storage` method with
-an object that responds to the following methods:
+You can set up storage for your experiment by calling the `storage` method with an object that responds to the following methods:
 
 * `store_assignment(assignment)`
 * `retrieve_assignment(experiment, subject)`
@@ -83,14 +84,15 @@ an object that responds to the following methods:
 
 Regarding the method signatures above, `experiment` is the Experiment instance, `subject` is the Subject instance, and `assignment` is a `Verdict::Assignment` instance.
 
-The `subject` instance will be identified internally by its `subject_identifier`
-By default it will use `subject.id.to_s` as `subject_identifier`, but you can change that by overriding `def subject_identifier(subject)` on the experiment.
+The `subject` instance will be identified internally by its `subject_identifier`. By default it will use `subject.id.to_s` as `subject_identifier`, but you can change that by overriding `def subject_identifier(subject)` on the experiment.
 
 Storage providers simply store subject assignments and require quick lookups of subject identifiers. They allow for complex (high CPU) assignments, and for assignments that might not always put the same subject in the same group by storing the assignment for later use.
 
-Storage providers are intended for operational use and should not be used for data analysis. For data analysis, you should use the logger.
+Storage providers are intended for operational use and should not be used for data analysis. For data analysis, you should use [the logger](#logging).
+
+When removing old experiments you might want to clean up corresponding experiment assignments, to reduce the amount of data stored and loaded. By using the logger, this data removal doesn't impact historic data or data analysis.
 
-For more details about these methods, check out the source code for [Verdict::Storage::MockStorage](lib/verdict/storage/mock_storage.rb)
+For more details about these methods, check out the source code for [`Verdict::Storage::MockStorage`](lib/verdict/storage/mock_storage.rb)
 
 ## Logging
 
@@ -101,12 +103,3 @@ You can override the logging by overriding the `def log_assignment(assignment)`
 Logging (as opposed to storage) should be used for data analysis. The logger requires a write-only / forward-only stream to write to, e.g. a log file, Kafka, or an insert-only database table.
 
 It's possible to run an experiment without defining any storage, though this comes with several drawbacks. Logging on the other hand is required in order to analyze the results.
-
-
-## Contributing
-
-1. Fork it
-2. Create your feature branch (`git checkout -b my-new-feature`)
-3. Commit your changes, including tests (`git commit -am 'Added some feature'`)
-4. Push to the branch (`git push origin my-new-feature`)
-5. Create new Pull Request, and mention @wvanbergen.

data/docs/concepts.md

@@ -0,0 +1,35 @@
+# Verdict Concepts
+
+Understanding the following Verdict concepts can help you implement experiments and discuss them with peers.
+
+- [Experiment](#experiment)
+- [Qualifiers](#qualifiers)
+- [Subject](#subject)
+- [Group](#group)
+- [Assignment](#assignment)
+
+### Experiment
+
+`Experiment`s are defined in `app/experiments` and encapsulate the details of the test including metadata such as the name, description or owners, as well as any [qualifiers](#qualifiers) or the [groups](#groups) available in the test.
+
+### Qualifiers
+
+Qualifiers are methods that take two parameters, `subject` and `context`. They return `true` if the `subject` can take this experiment, or `false` if the `subject` is not eligible. Experiments can have multiple qualifiers which are evaluated in succession. If a qualifier returns `false`, no further qualifiers are checked.
+
+Examples of qualifiers might be subjects that:
+
+- Originate from certain browsers or devices
+- Have a specific `Accept-Language` header
+- Have added items to their cart.
+
+### Subject
+
+A subject is the thing that will be [qualified](#qualifiers), and if successfull, [assigned](#assignment) to a particular [group](#group). For example, a subject might represent a user.
+
+### Group
+
+Each [experiment](#experiment) will have two or more groups, usually a control group and a test group. Qualified [subjects](#subject) are assigned to one of the groups in the experiment.
+
+### Assignment
+
+An assignment occurs when a qualified [subject](#subject) is placed into a [group](#group). Assignments are persisted so repeat visits by the subject skip [qualification](#qualifiers) and re-assignment and simply returns the existing assignment.

data/lib/verdict/experiment.rb

@@ -52,6 +52,29 @@ class Verdict::Experiment
     return self
   end
 
+  # Optional: Together with the "end_timestamp" and "stop_new_assignment_timestamp", limits the experiment run timeline within
+  # the given time interval.
+  #
+  # Timestamps definitions:
+  # start_timestamp: Experiment's start time. No assignments are made i.e. switch will return nil before this timestamp.
+  # stop_new_assignment_timestamp: Experiment's new assignment stop time. No new assignments are made
+  # i.e. switch returns nil for new assignments but the existing assignments are preserved.
+  # end_timestamp: Experiment's end time. No assignments are made i.e. switch returns nil after this timestamp.
+  #
+  # Experiment run timeline:
+  # start_timestamp -> (new assignments occur) -> stop_new_assignment_timestamp -> (no new assignments occur) -> end_timestamp
+  def schedule_start_timestamp(timestamp)
+    @schedule_start_timestamp = timestamp
+  end
+
+  def schedule_end_timestamp(timestamp)
+    @schedule_end_timestamp = timestamp
+  end
+
+  def schedule_stop_new_assignment_timestamp(timestamp)
+    @schedule_stop_new_assignment_timestamp = timestamp
+  end
+
   def rollout_percentage(percentage, rollout_group_name = :enabled)
     groups(Verdict::Segmenters::RolloutSegmenter) do
       group rollout_group_name, percentage
@@ -119,18 +142,18 @@ class Verdict::Experiment
     raise unless disqualify_empty_identifier?
   end
 
-  def assign(subject, context = nil)
+  def assign(subject, context = nil, dynamic_qualifiers: [])
     previous_assignment = lookup(subject)
 
     subject_identifier = retrieve_subject_identifier(subject)
     assignment = if previous_assignment
-                   previous_assignment
-                 elsif subject_qualifies?(subject, context)
-                   group = segmenter.assign(subject_identifier, subject, context)
-                   subject_assignment(subject, group, nil, group.nil?)
-                 else
-                   nil_assignment(subject)
-                 end
+      previous_assignment
+    elsif dynamic_subject_qualifies?(subject, dynamic_qualifiers, context) && is_make_new_assignments?
+      group = segmenter.assign(subject_identifier, subject, context)
+      subject_assignment(subject, group, nil, group.nil?)
+    else
+      nil_assignment(subject)
+    end
 
     store_assignment(assignment)
   rescue Verdict::StorageError
@@ -171,8 +194,11 @@ class Verdict::Experiment
     @storage.remove_assignment(self, subject)
   end
 
-  def switch(subject, context = nil)
-    assign(subject, context).to_sym
+  # The qualifiers param accepts an array of procs.
+  # This is intended for qualification logic that cannot be defined in the experiment definition
+  def switch(subject, context = nil, qualifiers: [])
+    return unless is_scheduled?
+    assign(subject, context, dynamic_qualifiers: qualifiers).to_sym
   end
 
   def lookup(subject)
@@ -217,8 +243,9 @@ class Verdict::Experiment
     @disqualify_empty_identifier
   end
 
-  def subject_qualifies?(subject, context = nil)
+  def subject_qualifies?(subject, context = nil, dynamic_qualifiers: [])
     ensure_experiment_has_started
+    return false unless dynamic_qualifiers.all? { |qualifier| qualifier.call(subject) }
     everybody_qualifies? || @qualifiers.all? { |qualifier| qualifier.call(subject, context) }
   end
 
@@ -239,10 +266,12 @@ class Verdict::Experiment
   def set_start_timestamp
     @storage.store_start_timestamp(self, started_now = Time.now.utc)
     started_now
+  rescue NotImplementedError
+    nil
   end
 
   def ensure_experiment_has_started
-    @started_at ||= @storage.retrieve_start_timestamp(self) || set_start_timestamp
+    @started_at ||= started_at || set_start_timestamp
   rescue Verdict::StorageError
     @started_at ||= Time.now.utc
   end
@@ -250,4 +279,31 @@ class Verdict::Experiment
   def nil_assignment(subject)
     subject_assignment(subject, nil, nil)
   end
+
+  private
+
+  def is_scheduled?
+    if @schedule_start_timestamp && @schedule_start_timestamp > Time.now
+      return false
+    end
+    if @schedule_end_timestamp && @schedule_end_timestamp <= Time.now
+      return false
+    end
+    return true
+  end
+
+  def is_make_new_assignments?
+    return !(@schedule_stop_new_assignment_timestamp && @schedule_stop_new_assignment_timestamp <= Time.now)
+  end
+
+  # Used when a Experiment class has overridden the subject_qualifies? method prior to v0.15.0
+  # The previous version of subject_qualifies did not accept dynamic qualifiers, thus this is used to
+  # determine how many parameters to pass
+  def dynamic_subject_qualifies?(subject, dynamic_qualifiers, context)
+    if method(:subject_qualifies?).parameters.include?([:key, :dynamic_qualifiers])
+      subject_qualifies?(subject, context, dynamic_qualifiers: dynamic_qualifiers)
+    else
+      subject_qualifies?(subject, context)
+    end
+  end
 end

data/lib/verdict/railtie.rb

@@ -5,10 +5,15 @@ class Verdict::Railtie < Rails::Railtie
     Verdict.default_logger = Rails.logger
 
     Verdict.directory ||= Rails.root.join('app', 'experiments')
-    app.config.eager_load_paths -= Dir[Verdict.directory.to_s]
 
-    # Re-freeze eager load paths to ensure they blow up if modified at runtime, as Rails does
-    app.config.eager_load_paths.freeze
+    if Rails.gem_version >= Gem::Version.new('6.0.0') && Rails.autoloaders.zeitwerk_enabled?
+      Rails.autoloaders.main.ignore(Verdict.directory)
+    else
+      app.config.eager_load_paths -= Dir[Verdict.directory.to_s]
+
+      # Re-freeze eager load paths to ensure they blow up if modified at runtime, as Rails does
+      app.config.eager_load_paths.freeze
+    end
   end
 
   config.to_prepare do

data/lib/verdict/storage.rb

@@ -1,4 +1,5 @@
 require 'verdict/storage/base_storage'
+require 'verdict/storage/cookie_storage'
 require 'verdict/storage/mock_storage'
 require 'verdict/storage/memory_storage'
 require 'verdict/storage/redis_storage'

data/lib/verdict/storage/cookie_storage.rb

@@ -0,0 +1,105 @@
+# frozen_string_literal: true
+
+module Verdict
+  module Storage
+    # CookieStorage, unlike other Verdict storage classes, is distributed and stored on the client.
+    # Because cookies are opaque to users, we obsfucate information about the test such as the
+    # human readable names for the experiment or assignment group. This means this class assumes
+    # that `name` will be an obsfucated value, or one that we comfortable being "public".
+    class CookieStorage < BaseStorage
+      DEFAULT_COOKIE_LIFESPAN_SECONDS = 15778476 # 6 months
+      KEY = :assignment
+
+      attr_accessor :cookies
+      attr_reader :cookie_lifespan
+
+      def initialize(cookie_lifespan: DEFAULT_COOKIE_LIFESPAN_SECONDS)
+        @cookies = nil
+        @cookie_lifespan = cookie_lifespan
+      end
+
+      def store_assignment(assignment)
+        data = {
+          group: digest_of(assignment.group.to_s),
+          created_at: assignment.created_at.strftime('%FT%TZ')
+        }
+
+        set(assignment.experiment.handle.to_s, KEY, JSON.dump(data))
+      end
+
+      def retrieve_assignment(experiment, subject)
+        if (value = get(experiment.handle.to_s, KEY))
+          data = parse_cookie_value(value)
+          group = find_group_by_digest(experiment, data['group'])
+
+          if group.nil?
+            experiment.remove_subject_assignment(subject)
+            return nil
+          end
+
+          experiment.subject_assignment(subject, group, Time.xmlschema(data['created_at']))
+        end
+      end
+
+      def remove_assignment(experiment, _subject)
+        remove(experiment.handle.to_s, KEY)
+      end
+
+      def retrieve_start_timestamp(_experiment)
+        nil
+      end
+
+      def store_start_timestamp(_experiment, _timestamp)
+        raise NotImplementedError
+      end
+
+      protected
+
+      def get(scope, key)
+        ensure_cookies_is_set
+        cookies[scope_key(scope, key)]
+      end
+
+      def set(scope, key, value)
+        ensure_cookies_is_set
+        cookies[scope_key(scope, key)] = {
+          value: value,
+          expires: Time.now.utc.advance(seconds: cookie_lifespan),
+        }
+      end
+
+      def remove(scope, key)
+        ensure_cookies_is_set
+        cookies.delete(scope_key(scope, key))
+      end
+
+      private
+
+      def ensure_cookies_is_set
+        raise Verdict::StorageError, 'cookies must be an instance of ActionDispatch::Cookies::CookieJar' if cookies.nil?
+      end
+
+      def digest_of(value)
+        Digest::MD5.hexdigest(value)
+      end
+
+      def find_group_by_digest(experiment, digest)
+        experiment.groups.values.find do |group|
+          digest_of(group.to_s) == digest
+        end
+      end
+
+      def parse_cookie_value(value)
+        value = value[:value] if value.is_a?(Hash)
+
+        JSON.parse(value)
+      rescue JSON::ParserError, TypeError
+        {}
+      end
+
+      def scope_key(scope, key)
+        "#{digest_of(scope)}_#{key}"
+      end
+    end
+  end
+end

data/lib/verdict/version.rb

@@ -1,3 +1,3 @@
 module Verdict
-  VERSION = "0.12.0"
+  VERSION = "0.15.2"
 end

data/test/experiment_test.rb

@@ -414,6 +414,16 @@ class ExperimentTest < Minitest::Test
     assert e.started?, "The experiment should have started after the first assignment"
   end
 
+  def test_experiment_set_start_timestamp_handles_storage_that_does_not_implement_timestamps
+    e = Verdict::Experiment.new('starting_test') do
+      groups { group :all, 100 }
+    end
+
+    e.storage.expects(:store_start_timestamp).raises(NotImplementedError)
+
+    assert_nil e.send(:set_start_timestamp)
+  end
+
   def test_no_storage
     e = Verdict::Experiment.new('starting_test') do
       groups { group :all, 100 }
@@ -460,9 +470,229 @@ class ExperimentTest < Minitest::Test
     end
   end
 
+  def test_is_scheduled
+    e = Verdict::Experiment.new(:json) do
+      groups do
+        group :a, :half
+        group :b, :rest
+      end
+      schedule_start_timestamp Time.new(2020, 1, 1)
+      schedule_end_timestamp Time.new(2020, 1, 3)
+    end
+
+    # Within the interval
+    Timecop.freeze(Time.new(2020, 1, 2)) do
+      assert e.send(:is_scheduled?)
+    end
+    # Too early
+    Timecop.freeze(Time.new(2019, 12, 30)) do
+      assert !e.send(:is_scheduled?)
+    end
+    # Too late
+    Timecop.freeze(Time.new(2020, 1, 4)) do
+      assert !e.send(:is_scheduled?)
+    end
+  end
+
+  def test_is_scheduled_no_end_timestamp
+    e = Verdict::Experiment.new(:json) do
+      groups do
+        group :a, :half
+        group :b, :rest
+      end
+      schedule_start_timestamp Time.new(2020, 1, 1)
+    end
+
+    # Within the interval because there is no end date
+    Timecop.freeze(Time.new(2030, 1, 1)) do
+      assert e.send(:is_scheduled?)
+    end
+    # Too early
+    Timecop.freeze(Time.new(2019, 12, 30)) do
+      assert !e.send(:is_scheduled?)
+    end
+  end
+
+  def test_is_scheduled_no_start_timestamp
+    e = Verdict::Experiment.new(:json) do
+      groups do
+        group :a, :half
+        group :b, :rest
+      end
+      schedule_end_timestamp Time.new(2020, 1, 3)
+    end
+
+    # Within the interval because there is no start date
+    Timecop.freeze(Time.new(2019, 12, 30)) do
+      assert e.send(:is_scheduled?)
+    end
+    # Too late
+    Timecop.freeze(Time.new(2020, 1, 4)) do
+      assert !e.send(:is_scheduled?)
+    end
+  end
+
+  def test_switch_respects_time_schedule
+    e = Verdict::Experiment.new('test') do
+      groups do
+        group :a, :half
+        group :b, :rest
+      end
+      schedule_start_timestamp Time.new(2020, 1, 1)
+      schedule_end_timestamp Time.new(2020, 1, 2)
+    end
+
+    Timecop.freeze(Time.new(2020, 1, 3)) do
+      assert_nil e.switch(1)
+    end
+  end
+
+  def test_switch_respects_time_schedule_even_after_assignment
+    e = Verdict::Experiment.new('test') do
+      groups do
+        group :a, :half
+        group :b, :rest
+      end
+    end
+
+    assert_equal :a, e.switch(1)
+
+    e.schedule_start_timestamp Time.new(2020, 1, 1)
+    e.schedule_end_timestamp Time.new(2020, 1, 2)
+
+    Timecop.freeze(Time.new(2020, 1, 3)) do
+      assert_nil e.switch(1)
+    end
+  end
+
+  def test_is_stop_new_assignments
+    e = Verdict::Experiment.new('test') do
+      groups do
+        group :a, :half
+        group :b, :half
+      end
+      schedule_stop_new_assignment_timestamp Time.new(2020, 1, 15)
+    end
+
+    # new assignments stopped after the stop timestamp
+    Timecop.freeze(Time.new(2020, 1, 16)) do
+      assert !e.send(:is_make_new_assignments?)
+      assert_nil e.switch(1)
+    end
+    # new assignments didn't stop before the stop timestamp
+    Timecop.freeze(Time.new(2020, 1, 3)) do
+      assert e.send(:is_make_new_assignments?)
+      assert :a, e.switch(2)
+    end
+  end
+
+  def test_switch_preserves_old_assignments_after_stop_new_assignments_timestamp
+    e = Verdict::Experiment.new('test') do
+      groups do
+        group :a, :half
+        group :b, :half
+      end
+    end
+
+    assert_equal :a, e.switch(1)
+
+    e.schedule_stop_new_assignment_timestamp Time.new(2020, 4, 15)
+
+    # switch respects to stop new assignment timestamp, old assignment preserves, new assignment returns nil
+    Timecop.freeze(Time.new(2020, 4, 16)) do
+      assert !e.send(:is_make_new_assignments?)
+      # old assignment stay the same
+      assert_equal :a, e.switch(1)
+      # new assignment returns nil
+      assert_nil e.switch(2)
+    end
+  end
+
+  def test_schedule_start_timestamp_and_stop_new_assignemnt_timestamp_are_inclusive_but_end_timestamp_is_exclusive
+    e = Verdict::Experiment.new('test') do
+      groups do
+        group :a, :half
+        group :b, :half
+      end
+
+      schedule_start_timestamp Time.new(2020, 1, 1)
+      schedule_stop_new_assignment_timestamp Time.new(2020, 1, 15)
+      schedule_end_timestamp Time.new(2020, 1, 31)
+    end
+
+    # start_timestamp is included
+    Timecop.freeze(Time.new(2020, 1, 1)) do
+      assert e.send(:is_scheduled?)
+      assert_equal :a, e.switch(1)
+    end
+
+    # stop_new_assignment_timestamp is included
+    Timecop.freeze(Time.new(2020, 1, 15)) do
+      assert !e.send(:is_make_new_assignments?)
+      # old assignment preserved
+      assert_equal :a, e.switch(1)
+      # new assignment returns nil
+      assert_nil e.switch(2)
+    end
+
+    # end_timestamp is excluded
+    Timecop.freeze(Time.new(2020, 1, 31)) do
+      assert !e.send(:is_scheduled?)
+      assert_nil e.switch(1)
+    end
+  end
+
+  def test_custom_qualifiers_success
+    e = Verdict::Experiment.new('test') do
+      groups do
+        group :all, 100
+      end
+    end
+
+    subject = 2
+    custom_qualifier_a = Proc.new { |subject| subject.even? }
+    custom_qualifier_b = Proc.new { |subject| subject > 0 }
+
+    group = e.switch(subject, qualifiers: [custom_qualifier_a, custom_qualifier_b])
+    assert_equal e.group(:all).to_sym, group
+  end
+
+  def test_custom_qualifiers_failure
+    e = Verdict::Experiment.new('test') do
+      groups do
+        group :all, 100
+      end
+    end
+
+    subject = 3
+    custom_qualifier_a = Proc.new { |subject| subject.even? }
+    custom_qualifier_b = Proc.new { |subject| subject > 0 }
+
+    group = e.switch(subject, qualifiers: [custom_qualifier_a, custom_qualifier_b])
+    assert_nil group
+  end
+
+  def test_dynamic_subject_qualifies_call_overridden_method
+    e = MyExperiment.new('test') do
+      groups do
+        group :all, 100
+      end
+    end
+
+    group = e.switch(4)
+    assert_nil group
+  end
+
   private
 
   def redis
     @redis ||= ::Redis.new(host: REDIS_HOST, port: REDIS_PORT)
   end
 end
+
+class MyExperiment < Verdict::Experiment
+  def subject_qualifies?(subject, context = nil)
+    return false if subject.even?
+    super
+  end
+end

data/test/storage/cookie_storage_test.rb

@@ -0,0 +1,81 @@
+# frozen_string_literal: true
+
+require 'test_helper'
+require 'fake_app'
+
+class CookieStorageTest < Minitest::Test
+  def setup
+    @storage = Verdict::Storage::CookieStorage.new.tap do |s|
+      s.cookies = ActionDispatch::Cookies::CookieJar.new(nil)
+    end
+    @experiment = Verdict::Experiment.new(:cookie_storage_test) do
+      groups { group :all, 100 }
+      storage @storage, store_unqualified: true
+    end
+    @subject = stub(id: 'bob')
+    @assignment = Verdict::Assignment.new(@experiment, @subject, @experiment.group(:all), nil)
+  end
+
+  def test_cookie_lifespan_has_a_default
+    cookie_lifespan = Verdict::Storage::CookieStorage.new.cookie_lifespan
+
+    assert_equal Verdict::Storage::CookieStorage::DEFAULT_COOKIE_LIFESPAN_SECONDS, cookie_lifespan
+  end
+
+  def test_cookie_lifespan_can_be_configured
+    storage = Verdict::Storage::CookieStorage.new(cookie_lifespan: 60)
+
+    assert_equal 60, storage.cookie_lifespan
+  end
+
+  def test_raises_storage_error_when_cookies_is_nil
+    storage = Verdict::Storage::CookieStorage.new
+
+    assert_raises(Verdict::StorageError) { storage.store_assignment(@assignment) }
+    assert_raises(Verdict::StorageError) { storage.retrieve_assignment(@experiment, @subject) }
+    assert_raises(Verdict::StorageError) { storage.remove_assignment(@experiment, @subject) }
+  end
+
+  def test_store_assignment_returns_true_when_an_assignment_is_stored
+    assert @storage.store_assignment(@assignment)
+    refute_nil @storage.retrieve_assignment(@experiment, @subject)
+  end
+
+  def test_retrieve_assignment_returns_an_assignment
+    @storage.store_assignment(@assignment)
+    assignment = @storage.retrieve_assignment(@experiment, @subject)
+
+    assert assignment.returning?
+    assert_equal :all, assignment.handle.to_sym
+    assert_equal @experiment, assignment.experiment
+    assert_equal @subject, assignment.subject
+  end
+
+  def test_retrieve_assignment_returns_nil_when_an_assignment_does_not_exist
+    assert_nil @storage.retrieve_assignment(@experiment, @subject)
+  end
+
+  def test_retrieve_assignment_returns_nil_when_the_assignment_group_is_invalid
+    invalid_group = Verdict::Group.new(@experiment, :invalid)
+    assignment = Verdict::Assignment.new(@experiment, @subject, invalid_group, nil)
+
+    @storage.store_assignment(assignment)
+
+    assert_nil @storage.retrieve_assignment(@experiment, @subject)
+  end
+
+  def test_remove_assignment_returns_true_when_removing_an_assignment
+    @storage.store_assignment(@assignment)
+
+    assert @storage.remove_assignment(@experiment, nil)
+    assert_nil @storage.retrieve_assignment(@experiment, @subject)
+  end
+
+  def test_retrieve_start_timestamp_always_returns_nil
+    assert_nil @storage.retrieve_start_timestamp(nil)
+  end
+
+  def test_store_start_timestamp_raises_not_implemented_error
+    assert_raises(NotImplementedError) { @storage.store_start_timestamp(nil, nil) }
+  end
+end

data/verdict.gemspec

@@ -17,6 +17,8 @@ Gem::Specification.new do |gem|
   gem.test_files    = gem.files.grep(%r{^(test|spec|features)/})
   gem.require_paths = ["lib"]
 
+  gem.metadata["allowed_push_host"] = "https://rubygems.org"
+
   gem.add_development_dependency("minitest", '~> 5.2')
   gem.add_development_dependency("rake")
   gem.add_development_dependency("mocha")

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: verdict
 version: !ruby/object:Gem::Version
-  version: 0.12.0
+  version: 0.15.2
 platform: ruby
 authors:
 - Shopify
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2019-09-11 00:00:00.000000000 Z
+date: 2020-07-15 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: minitest
@@ -106,10 +106,12 @@ files:
 - ".gitignore"
 - ".travis.yml"
 - CHANGELOG.md
+- CONTRIBUTING.md
 - Gemfile
 - LICENSE
 - README.md
 - Rakefile
+- docs/concepts.md
 - lib/verdict.rb
 - lib/verdict/assignment.rb
 - lib/verdict/conversion.rb
@@ -126,6 +128,7 @@ files:
 - lib/verdict/segmenters/static_segmenter.rb
 - lib/verdict/storage.rb
 - lib/verdict/storage/base_storage.rb
+- lib/verdict/storage/cookie_storage.rb
 - lib/verdict/storage/legacy_redis_storage.rb
 - lib/verdict/storage/memory_storage.rb
 - lib/verdict/storage/mock_storage.rb
@@ -148,6 +151,7 @@ files:
 - test/segmenters/rollout_segmenter_test.rb
 - test/segmenters/static_segmenter_test.rb
 - test/storage/base_storage_test.rb
+- test/storage/cookie_storage_test.rb
 - test/storage/legacy_redis_storage_test.rb
 - test/storage/memory_storage_test.rb
 - test/storage/redis_storage_test.rb
@@ -156,7 +160,8 @@ files:
 - verdict.gemspec
 homepage: http://github.com/Shopify/verdict
 licenses: []
-metadata: {}
+metadata:
+  allowed_push_host: https://rubygems.org
 post_install_message: 
 rdoc_options: []
 require_paths:
@@ -193,6 +198,7 @@ test_files:
 - test/segmenters/rollout_segmenter_test.rb
 - test/segmenters/static_segmenter_test.rb
 - test/storage/base_storage_test.rb
+- test/storage/cookie_storage_test.rb
 - test/storage/legacy_redis_storage_test.rb
 - test/storage/memory_storage_test.rb
 - test/storage/redis_storage_test.rb