verdict 0.10.0 → 0.15.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 23482b6ebd56ecd7bdb03d318c4a507e8d9ca6b3a70ace6c63a52349855f1666
-  data.tar.gz: fe4b75a65e3daca4c40cfead80d1c9ff266e40faa4af99b912c8f5116b7f5cae
+  metadata.gz: 3a261f90b676053938c42f341b2383b79bb4441ab5a0d2493515f1f90402b5f5
+  data.tar.gz: 1e9759c2062f9671d12bb53dc591f41615e2dd2d43910892fa28bc44042871c7
 SHA512:
-  metadata.gz: a403a445b44731fccaddb181fcda9acbd2b138adad07fadde5d8c3b0f97fd97c02a3873256672b1dc2d6756b7471b6575345b9e525efee27c8c25c7d89d3bf3c
-  data.tar.gz: 3d52f3c2517c91a132c327ba57a695f008f611b82ff17d6bee86cd23628102c4b6d9f70ad62768929eb986d46ce566c215c6f676e4c0883029c24d422ad32c7d
+  metadata.gz: a80328ac1da8ad852d0578df8d16ab435277c39249876b0ed3288d41be8f24fe88155588efd848d7431f053b0d1bcfaa76331451fce7aed8327b81ac59f24484
+  data.tar.gz: 27559293e140391710c4d5cf011bf8cc6418c52fd028943badf21a9e8ce802f39fc0608f479b64d343f9d3ea6c1759f9a53644750dcbf27cde28ef2ee15d470f

data/.gitignore

@@ -16,3 +16,4 @@ test/tmp
 test/version_tmp
 tmp
 dump.rdb
+log

data/.travis.yml

@@ -11,5 +11,6 @@ rvm:
 matrix:
   allow_failures:
     - rvm: ruby-head
+    - rvm: jruby
 services:
   - redis-server

data/CHANGELOG.md

@@ -1,3 +1,26 @@
+## v0.15.0
+* Add optional `qualifiers` parameter to the `Verdict::Experiment#switch` method. This parameter accepts an array of procs and is used as additional qualifiers. The purpose of this parameter is to allow users to define qualification logic outside of the experiment definition.
+
+## v0.14.0
+* Add optional experiment definition method `schedule_stop_new_assignment_timestamp` to support limiting experiment's assignment lifetime with another pre-determined time interval. It allows users to have an assignment cooldown period for stable analysis of the experiment results. Experiment's lifetime now becomes: start experiment -> stop new assignments -> end experiment. 
+
+## v0.13.0
+
+* Add optional experiment definition methods `schedule_start_timestamp` and `schedule_end_timestamp` to support limiting experiment's lifetime in a pre-determined time interval.
+* Support eager loading from within a Rails app using Zeitwerk.
+* Add `CookieStorage` storage backend. This backend is a distributed store for Verdict and does not support experiment timestamps. It is designed to be used with Rails applications and requires that `.cookies` be set to the `CookieJar` instance before use.
+
+## v0.12.0
+
+* Allow options to be passed to `Experiment#cleanup` so they can be forwarded to storage.
+
+* Changed `Experiment#cleanup` to accept an argument of type `Verdict::Experiment`.
+  Passing a `String`/`Symbol` argument is still supported, but will log a deprecation warning.
+
+## v0.11.0
+
+* Automatic eager loading when inside a Rails app.
+
 ## v0.10.0
 
 * Add `Experiment#cleanup` to remove stored redis hashes.

data/CONTRIBUTING.md

@@ -0,0 +1,7 @@
+## Contributing
+
+1. Fork it
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes, including tests (`git commit -am 'Added some feature'`)
+4. Push to the branch (`git push origin my-new-feature`)
+5. Create new Pull Request

data/README.md

@@ -19,6 +19,8 @@ Add this line to your application's Gemfile, and run `bundle install`:
 
 If you're using Rails, the Railtie will handle setting the logger to `Rails.logger` and the experiments directory to `app/experiments`. It will also load the rake tasks for you (run `bundle exec rake -T | grep experiments:` for options).
 
+You may find the [Concepts](docs/concepts.md) documentation a good place to familiarise yourself with Verdict's nomenclature.
+
 ## Usage
 
 The `Verdict::Experiment` class is used to create an experiment, define control and experiment groups, and to qualify subjects.
@@ -42,8 +44,7 @@ Verdict::Experiment.define :my_experiment do
 end
 ```
 
-Usually you'll want to place this in a file called **my_experiment.rb** in the
-**/app/experiments** folder.
+Usually you'll want to place this in a file called **my_experiment.rb** in the **/app/experiments** folder.
 
 _We recommend that you subclass `Verdict::Experiment` to set some default options for your app's environment, and call `define` on that class instead._
 
@@ -71,9 +72,9 @@ Verdict uses a very simple interface for storing experiment assignments. Out of
 
 * Memory
 * Redis
+* Cookies
 
-You can set up storage for your experiment by calling the `storage` method with
-an object that responds to the following methods:
+You can set up storage for your experiment by calling the `storage` method with an object that responds to the following methods:
 
 * `store_assignment(assignment)`
 * `retrieve_assignment(experiment, subject)`
@@ -83,14 +84,15 @@ an object that responds to the following methods:
 
 Regarding the method signatures above, `experiment` is the Experiment instance, `subject` is the Subject instance, and `assignment` is a `Verdict::Assignment` instance.
 
-The `subject` instance will be identified internally by its `subject_identifier`
-By default it will use `subject.id.to_s` as `subject_identifier`, but you can change that by overriding `def subject_identifier(subject)` on the experiment.
+The `subject` instance will be identified internally by its `subject_identifier`. By default it will use `subject.id.to_s` as `subject_identifier`, but you can change that by overriding `def subject_identifier(subject)` on the experiment.
 
 Storage providers simply store subject assignments and require quick lookups of subject identifiers. They allow for complex (high CPU) assignments, and for assignments that might not always put the same subject in the same group by storing the assignment for later use.
 
-Storage providers are intended for operational use and should not be used for data analysis. For data analysis, you should use the logger.
+Storage providers are intended for operational use and should not be used for data analysis. For data analysis, you should use [the logger](#logging).
+
+When removing old experiments you might want to clean up corresponding experiment assignments, to reduce the amount of data stored and loaded. By using the logger, this data removal doesn't impact historic data or data analysis.
 
-For more details about these methods, check out the source code for [Verdict::Storage::MockStorage](lib/verdict/storage/mock_storage.rb)
+For more details about these methods, check out the source code for [`Verdict::Storage::MockStorage`](lib/verdict/storage/mock_storage.rb)
 
 ## Logging
 
@@ -101,12 +103,3 @@ You can override the logging by overriding the `def log_assignment(assignment)`
 Logging (as opposed to storage) should be used for data analysis. The logger requires a write-only / forward-only stream to write to, e.g. a log file, Kafka, or an insert-only database table.
 
 It's possible to run an experiment without defining any storage, though this comes with several drawbacks. Logging on the other hand is required in order to analyze the results.
-
-
-## Contributing
-
-1. Fork it
-2. Create your feature branch (`git checkout -b my-new-feature`)
-3. Commit your changes, including tests (`git commit -am 'Added some feature'`)
-4. Push to the branch (`git push origin my-new-feature`)
-5. Create new Pull Request, and mention @wvanbergen.

data/docs/concepts.md

@@ -0,0 +1,35 @@
+# Verdict Concepts
+
+Understanding the following Verdict concepts can help you implement experiments and discuss them with peers.
+
+- [Experiment](#experiment)
+- [Qualifiers](#qualifiers)
+- [Subject](#subject)
+- [Group](#group)
+- [Assignment](#assignment)
+
+### Experiment
+
+`Experiment`s are defined in `app/experiments` and encapsulate the details of the test including metadata such as the name, description or owners, as well as any [qualifiers](#qualifiers) or the [groups](#groups) available in the test.
+
+### Qualifiers
+
+Qualifiers are methods that take two parameters, `subject` and `context`. They return `true` if the `subject` can take this experiment, or `false` if the `subject` is not eligible. Experiments can have multiple qualifiers which are evaluated in succession. If a qualifier returns `false`, no further qualifiers are checked.
+
+Examples of qualifiers might be subjects that:
+
+- Originate from certain browsers or devices
+- Have a specific `Accept-Language` header
+- Have added items to their cart.
+
+### Subject
+
+A subject is the thing that will be [qualified](#qualifiers), and if successfull, [assigned](#assignment) to a particular [group](#group). For example, a subject might represent a user.
+
+### Group
+
+Each [experiment](#experiment) will have two or more groups, usually a control group and a test group. Qualified [subjects](#subject) are assigned to one of the groups in the experiment.
+
+### Assignment
+
+An assignment occurs when a qualified [subject](#subject) is placed into a [group](#group). Assignments are persisted so repeat visits by the subject skip [qualification](#qualifiers) and re-assignment and simply returns the existing assignment.

data/lib/verdict.rb

@@ -15,6 +15,10 @@ module Verdict
     @repository
   end
 
+  def eager_load!
+    discovery
+  end
+
   def discovery
     @repository = {}
     Dir[File.join(Verdict.directory, '**', '*.rb')].each { |f| load f } if @directory
@@ -54,3 +58,4 @@ require "verdict/event_logger"
 
 Verdict.default_logger ||= Logger.new("/dev/null")
 Verdict.directory = nil
+Verdict.clear_repository_cache

data/lib/verdict/experiment.rb

@@ -2,7 +2,7 @@ class Verdict::Experiment
 
   include Verdict::Metadata
 
-  attr_reader :handle, :qualifiers, :storage, :event_logger
+  attr_reader :handle, :qualifiers, :event_logger
 
   def self.define(handle, *args, &block)
     experiment = self.new(handle, *args, &block)
@@ -11,6 +11,7 @@ class Verdict::Experiment
   end
 
   def initialize(handle, options = {}, &block)
+    @started_at = nil
     @handle = handle.to_s
 
     options = default_options.merge(options)
@@ -51,6 +52,29 @@ class Verdict::Experiment
     return self
   end
 
+  # Optional: Together with the "end_timestamp" and "stop_new_assignment_timestamp", limits the experiment run timeline within
+  # the given time interval.
+  #
+  # Timestamps definitions:
+  # start_timestamp: Experiment's start time. No assignments are made i.e. switch will return nil before this timestamp.
+  # stop_new_assignment_timestamp: Experiment's new assignment stop time. No new assignments are made
+  # i.e. switch returns nil for new assignments but the existing assignments are preserved.
+  # end_timestamp: Experiment's end time. No assignments are made i.e. switch returns nil after this timestamp.
+  #
+  # Experiment run timeline:
+  # start_timestamp -> (new assignments occur) -> stop_new_assignment_timestamp -> (no new assignments occur) -> end_timestamp
+  def schedule_start_timestamp(timestamp)
+    @schedule_start_timestamp = timestamp
+  end
+
+  def schedule_end_timestamp(timestamp)
+    @schedule_end_timestamp = timestamp
+  end
+
+  def schedule_stop_new_assignment_timestamp(timestamp)
+    @schedule_stop_new_assignment_timestamp = timestamp
+  end
+
   def rollout_percentage(percentage, rollout_group_name = :enabled)
     groups(Verdict::Segmenters::RolloutSegmenter) do
       group rollout_group_name, percentage
@@ -88,7 +112,7 @@ class Verdict::Experiment
 
   def started_at
     @started_at ||= @storage.retrieve_start_timestamp(self)
-  rescue Verdict::StorageError => e
+  rescue Verdict::StorageError
     nil
   end
 
@@ -118,18 +142,18 @@ class Verdict::Experiment
     raise unless disqualify_empty_identifier?
   end
 
-  def assign(subject, context = nil)
+  def assign(subject, context = nil, dynamic_qualifiers: [])
     previous_assignment = lookup(subject)
 
     subject_identifier = retrieve_subject_identifier(subject)
     assignment = if previous_assignment
-                   previous_assignment
-                 elsif subject_qualifies?(subject, context)
-                   group = segmenter.assign(subject_identifier, subject, context)
-                   subject_assignment(subject, group, nil, group.nil?)
-                 else
-                   nil_assignment(subject)
-                 end
+      previous_assignment
+    elsif subject_qualifies?(subject, dynamic_qualifiers, context) && is_make_new_assignments?
+      group = segmenter.assign(subject_identifier, subject, context)
+      subject_assignment(subject, group, nil, group.nil?)
+    else
+      nil_assignment(subject)
+    end
 
     store_assignment(assignment)
   rescue Verdict::StorageError
@@ -162,16 +186,19 @@ class Verdict::Experiment
     assignment
   end
 
-  def cleanup
-    @storage.cleanup(handle.to_s)
+  def cleanup(options = {})
+    @storage.cleanup(self, options)
   end
 
   def remove_subject_assignment(subject)
     @storage.remove_assignment(self, subject)
   end
 
-  def switch(subject, context = nil)
-    assign(subject, context).to_sym
+  # The qualifiers param accepts an array of procs.
+  # This is intended for qualification logic that cannot be defined in the experiment definition
+  def switch(subject, context = nil, qualifiers: [])
+    return unless is_scheduled?
+    assign(subject, context, dynamic_qualifiers: qualifiers).to_sym
   end
 
   def lookup(subject)
@@ -193,15 +220,13 @@ class Verdict::Experiment
   end
 
   def as_json(options = {})
-    data = {
+    {
       handle: handle,
       has_qualifier: has_qualifier?,
-      groups: segmenter.groups.values.map { |g| g.as_json(options) },
+      groups: segmenter.groups.values.map { |group| group.as_json(options) },
       metadata: metadata,
       started_at: started_at.nil? ? nil : started_at.utc.strftime('%FT%TZ')
-    }
-
-    data.tap do |data|
+    }.tap do |data|
       data[:subject_type] = subject_type.to_s unless subject_type.nil?
     end
   end
@@ -218,8 +243,9 @@ class Verdict::Experiment
     @disqualify_empty_identifier
   end
 
-  def subject_qualifies?(subject, context = nil)
+  def subject_qualifies?(subject, dynamic_qualifiers, context = nil)
     ensure_experiment_has_started
+    return false unless dynamic_qualifiers.all? { |qualifier| qualifier.call(subject) }
     everybody_qualifies? || @qualifiers.all? { |qualifier| qualifier.call(subject, context) }
   end
 
@@ -240,10 +266,12 @@ class Verdict::Experiment
   def set_start_timestamp
     @storage.store_start_timestamp(self, started_now = Time.now.utc)
     started_now
+  rescue NotImplementedError
+    nil
   end
 
   def ensure_experiment_has_started
-    @started_at ||= @storage.retrieve_start_timestamp(self) || set_start_timestamp
+    @started_at ||= started_at || set_start_timestamp
   rescue Verdict::StorageError
     @started_at ||= Time.now.utc
   end
@@ -251,4 +279,20 @@ class Verdict::Experiment
   def nil_assignment(subject)
     subject_assignment(subject, nil, nil)
   end
+
+  private
+
+  def is_scheduled?
+    if @schedule_start_timestamp && @schedule_start_timestamp > Time.now
+      return false
+    end
+    if @schedule_end_timestamp && @schedule_end_timestamp <= Time.now
+      return false
+    end
+    return true
+  end
+
+  def is_make_new_assignments?
+    return !(@schedule_stop_new_assignment_timestamp && @schedule_stop_new_assignment_timestamp <= Time.now)
+  end
 end

data/lib/verdict/railtie.rb

@@ -1,12 +1,19 @@
 class Verdict::Railtie < Rails::Railtie
   initializer "experiments.configure_rails_initialization" do |app|
+    app.config.eager_load_namespaces << Verdict
+
     Verdict.default_logger = Rails.logger
 
     Verdict.directory ||= Rails.root.join('app', 'experiments')
-    app.config.eager_load_paths -= Dir[Verdict.directory.to_s]
 
-    # Re-freeze eager load paths to ensure they blow up if modified at runtime, as Rails does
-    app.config.eager_load_paths.freeze
+    if Rails.gem_version >= Gem::Version.new('6.0.0') && Rails.autoloaders.zeitwerk_enabled?
+      Rails.autoloaders.main.ignore(Verdict.directory)
+    else
+      app.config.eager_load_paths -= Dir[Verdict.directory.to_s]
+
+      # Re-freeze eager load paths to ensure they blow up if modified at runtime, as Rails does
+      app.config.eager_load_paths.freeze
+    end
   end
 
   config.to_prepare do

data/lib/verdict/storage.rb

@@ -1,4 +1,5 @@
 require 'verdict/storage/base_storage'
+require 'verdict/storage/cookie_storage'
 require 'verdict/storage/mock_storage'
 require 'verdict/storage/memory_storage'
 require 'verdict/storage/redis_storage'

data/lib/verdict/storage/base_storage.rb

@@ -45,11 +45,21 @@ module Verdict
         set(experiment.handle.to_s, 'started_at', timestamp.utc.strftime('%FT%TZ'))
       end
 
-      def cleanup(_scope)
-        raise NotImplementedError
+      # Deletes all assignments (and any other stored data) for the given experiment
+      def cleanup(experiment_or_scope, options = {})
+        if experiment_or_scope.is_a?(Symbol) || experiment_or_scope.is_a?(String)
+          Verdict.default_logger.warn(
+            "Passing a scope string/symbol to #{self.class}#cleanup is deprecated, " \
+            'pass a Verdict::Experiment instance instead.'
+          )
+          clear(experiment_or_scope, options)
+        else
+          clear(experiment_or_scope.handle.to_s, options)
+        end
       end
 
       protected
+
       # Retrieves a key in a given scope from storage.
       # - The scope and key are both provided as string.
       # - Should return a string value if the key is found in the scope, nil otherwise.
@@ -58,7 +68,7 @@ module Verdict
         raise NotImplementedError
       end
 
-      # Retrieves a key in a given scope from storage.
+      # Sets the value of a key in a given scope from storage.
       # - The scope, key, and value are all provided as string.
       # - Should return true if the item was successfully stored.
       # - Should raise Verdict::StorageError if anything goes wrong.
@@ -66,13 +76,21 @@ module Verdict
         raise NotImplementedError
       end
 
-      # Retrieves a key in a given scope from storage.
+      # Removes a key in a given scope from storage.
       # - The scope and key are both provided as string.
       # - Should return true if the item was successfully removed from storage.
       # - Should raise Verdict::StorageError if anything goes wrong.
       def remove(scope, key)
         raise NotImplementedError
       end
+
+      # Removes all keys in a given scope from storage.
+      # - The scope is provided as string.
+      # - Should return true if all items were successfully removed from storage.
+      # - Should raise Verdict::StorageError if anything goes wrong.
+      def clear(scope, options)
+        raise NotImplementedError
+      end
     end
   end
 end

data/lib/verdict/storage/cookie_storage.rb

@@ -0,0 +1,105 @@
+# frozen_string_literal: true
+
+module Verdict
+  module Storage
+    # CookieStorage, unlike other Verdict storage classes, is distributed and stored on the client.
+    # Because cookies are opaque to users, we obsfucate information about the test such as the
+    # human readable names for the experiment or assignment group. This means this class assumes
+    # that `name` will be an obsfucated value, or one that we comfortable being "public".
+    class CookieStorage < BaseStorage
+      DEFAULT_COOKIE_LIFESPAN_SECONDS = 15778476 # 6 months
+      KEY = :assignment
+
+      attr_accessor :cookies
+      attr_reader :cookie_lifespan
+
+      def initialize(cookie_lifespan: DEFAULT_COOKIE_LIFESPAN_SECONDS)
+        @cookies = nil
+        @cookie_lifespan = cookie_lifespan
+      end
+
+      def store_assignment(assignment)
+        data = {
+          group: digest_of(assignment.group.to_s),
+          created_at: assignment.created_at.strftime('%FT%TZ')
+        }
+
+        set(assignment.experiment.handle.to_s, KEY, JSON.dump(data))
+      end
+
+      def retrieve_assignment(experiment, subject)
+        if (value = get(experiment.handle.to_s, KEY))
+          data = parse_cookie_value(value)
+          group = find_group_by_digest(experiment, data['group'])
+
+          if group.nil?
+            experiment.remove_subject_assignment(subject)
+            return nil
+          end
+
+          experiment.subject_assignment(subject, group, Time.xmlschema(data['created_at']))
+        end
+      end
+
+      def remove_assignment(experiment, _subject)
+        remove(experiment.handle.to_s, KEY)
+      end
+
+      def retrieve_start_timestamp(_experiment)
+        nil
+      end
+
+      def store_start_timestamp(_experiment, _timestamp)
+        raise NotImplementedError
+      end
+
+      protected
+
+      def get(scope, key)
+        ensure_cookies_is_set
+        cookies[scope_key(scope, key)]
+      end
+
+      def set(scope, key, value)
+        ensure_cookies_is_set
+        cookies[scope_key(scope, key)] = {
+          value: value,
+          expires: Time.now.utc.advance(seconds: cookie_lifespan),
+        }
+      end
+
+      def remove(scope, key)
+        ensure_cookies_is_set
+        cookies.delete(scope_key(scope, key))
+      end
+
+      private
+
+      def ensure_cookies_is_set
+        raise Verdict::StorageError, 'cookies must be an instance of ActionDispatch::Cookies::CookieJar' if cookies.nil?
+      end
+
+      def digest_of(value)
+        Digest::MD5.hexdigest(value)
+      end
+
+      def find_group_by_digest(experiment, digest)
+        experiment.groups.values.find do |group|
+          digest_of(group.to_s) == digest
+        end
+      end
+
+      def parse_cookie_value(value)
+        value = value[:value] if value.is_a?(Hash)
+
+        JSON.parse(value)
+      rescue JSON::ParserError, TypeError
+        {}
+      end
+
+      def scope_key(scope, key)
+        "#{digest_of(scope)}_#{key}"
+      end
+    end
+  end
+end

data/lib/verdict/storage/memory_storage.rb

@@ -7,6 +7,8 @@ module Verdict
         @storage = {}
       end
 
+      protected
+
       def get(scope, key)
         @storage[scope] ||= {}
         @storage[scope][key]

data/lib/verdict/storage/mock_storage.rb

@@ -1,14 +1,16 @@
 module Verdict
   module Storage
     class MockStorage < BaseStorage
-      def set(scope, key, value)
-        false
-      end
+      protected
 
       def get(scope, key)
         nil
       end
 
+      def set(scope, key, value)
+        false
+      end
+
       def remove(scope, key)
       end
     end

data/lib/verdict/storage/redis_storage.rb

@@ -10,6 +10,8 @@ module Verdict
         @key_prefix = options[:key_prefix] || 'experiments/'
       end
 
+      protected
+
       def get(scope, key)
         redis.hget(scope_key(scope), key)
       rescue ::Redis::BaseError => e
@@ -28,8 +30,8 @@ module Verdict
         raise Verdict::StorageError, "Redis error: #{e.message}"
       end
 
-      def cleanup(scope)
-        clear(scope)
+      def clear(scope, options)
+        scrub(scope)
         redis.del(scope_key(scope))
       rescue ::Redis::BaseError => e
         raise Verdict::StorageError, "Redis error: #{e.message}"
@@ -41,12 +43,12 @@ module Verdict
         "#{@key_prefix}#{scope}"
       end
 
-      def clear(scope, cursor: 0)
+      def scrub(scope, cursor: 0)
         cursor, results = redis.hscan(scope_key(scope), cursor, count: PAGE_SIZE)
         results.map(&:first).each do |key|
           remove(scope, key)
         end
-        clear(scope, cursor: cursor) unless cursor.to_i.zero?
+        scrub(scope, cursor: cursor) unless cursor.to_i.zero?
       end
     end
   end

data/lib/verdict/version.rb

@@ -1,3 +1,3 @@
 module Verdict
-  VERSION = "0.10.0"
+  VERSION = "0.15.0"
 end

data/test/experiment_test.rb

@@ -414,6 +414,16 @@ class ExperimentTest < Minitest::Test
     assert e.started?, "The experiment should have started after the first assignment"
   end
 
+  def test_experiment_set_start_timestamp_handles_storage_that_does_not_implement_timestamps
+    e = Verdict::Experiment.new('starting_test') do
+      groups { group :all, 100 }
+    end
+
+    e.storage.expects(:store_start_timestamp).raises(NotImplementedError)
+
+    assert_nil e.send(:set_start_timestamp)
+  end
+
   def test_no_storage
     e = Verdict::Experiment.new('starting_test') do
       groups { group :all, 100 }
@@ -424,7 +434,6 @@ class ExperimentTest < Minitest::Test
   end
 
   def test_cleanup
-    redis = ::Redis.new(host: REDIS_HOST, port: REDIS_PORT)
     storage = Verdict::Storage::RedisStorage.new(redis)
     experiment = Verdict::Experiment.new(:cleanup_test) do
       groups { group :all, 100 }
@@ -440,6 +449,16 @@ class ExperimentTest < Minitest::Test
     redis.del("experiments/cleanup_test")
   end
 
+  def test_cleanup_options
+    experiment = Verdict::Experiment.new(:cleanup_test) do
+      groups { group :all, 100 }
+    end
+
+    experiment.storage.expects(:clear).with(experiment.handle, some: :thing)
+    experiment.assign("something")
+    experiment.cleanup(some: :thing)
+  end
+
   def test_cleanup_without_redis
     experiment = Verdict::Experiment.new(:cleanup_test) do
       groups { group :all, 100 }
@@ -450,4 +469,214 @@ class ExperimentTest < Minitest::Test
       experiment.cleanup
     end
   end
+
+  def test_is_scheduled
+    e = Verdict::Experiment.new(:json) do
+      groups do
+        group :a, :half
+        group :b, :rest
+      end
+      schedule_start_timestamp Time.new(2020, 1, 1)
+      schedule_end_timestamp Time.new(2020, 1, 3)
+    end
+
+    # Within the interval
+    Timecop.freeze(Time.new(2020, 1, 2)) do
+      assert e.send(:is_scheduled?)
+    end
+    # Too early
+    Timecop.freeze(Time.new(2019, 12, 30)) do
+      assert !e.send(:is_scheduled?)
+    end
+    # Too late
+    Timecop.freeze(Time.new(2020, 1, 4)) do
+      assert !e.send(:is_scheduled?)
+    end
+  end
+
+  def test_is_scheduled_no_end_timestamp
+    e = Verdict::Experiment.new(:json) do
+      groups do
+        group :a, :half
+        group :b, :rest
+      end
+      schedule_start_timestamp Time.new(2020, 1, 1)
+    end
+
+    # Within the interval because there is no end date
+    Timecop.freeze(Time.new(2030, 1, 1)) do
+      assert e.send(:is_scheduled?)
+    end
+    # Too early
+    Timecop.freeze(Time.new(2019, 12, 30)) do
+      assert !e.send(:is_scheduled?)
+    end
+  end
+
+  def test_is_scheduled_no_start_timestamp
+    e = Verdict::Experiment.new(:json) do
+      groups do
+        group :a, :half
+        group :b, :rest
+      end
+      schedule_end_timestamp Time.new(2020, 1, 3)
+    end
+
+    # Within the interval because there is no start date
+    Timecop.freeze(Time.new(2019, 12, 30)) do
+      assert e.send(:is_scheduled?)
+    end
+    # Too late
+    Timecop.freeze(Time.new(2020, 1, 4)) do
+      assert !e.send(:is_scheduled?)
+    end
+  end
+
+  def test_switch_respects_time_schedule
+    e = Verdict::Experiment.new('test') do
+      groups do
+        group :a, :half
+        group :b, :rest
+      end
+      schedule_start_timestamp Time.new(2020, 1, 1)
+      schedule_end_timestamp Time.new(2020, 1, 2)
+    end
+
+    Timecop.freeze(Time.new(2020, 1, 3)) do
+      assert_nil e.switch(1)
+    end
+  end
+
+  def test_switch_respects_time_schedule_even_after_assignment
+    e = Verdict::Experiment.new('test') do
+      groups do
+        group :a, :half
+        group :b, :rest
+      end
+    end
+
+    assert_equal :a, e.switch(1)
+
+    e.schedule_start_timestamp Time.new(2020, 1, 1)
+    e.schedule_end_timestamp Time.new(2020, 1, 2)
+
+    Timecop.freeze(Time.new(2020, 1, 3)) do
+      assert_nil e.switch(1)
+    end
+  end
+
+  def test_is_stop_new_assignments
+    e = Verdict::Experiment.new('test') do
+      groups do
+        group :a, :half
+        group :b, :half
+      end
+      schedule_stop_new_assignment_timestamp Time.new(2020, 1, 15)
+    end
+
+    # new assignments stopped after the stop timestamp
+    Timecop.freeze(Time.new(2020, 1, 16)) do
+      assert !e.send(:is_make_new_assignments?)
+      assert_nil e.switch(1)
+    end
+    # new assignments didn't stop before the stop timestamp
+    Timecop.freeze(Time.new(2020, 1, 3)) do
+      assert e.send(:is_make_new_assignments?)
+      assert :a, e.switch(2)
+    end
+  end
+
+  def test_switch_preserves_old_assignments_after_stop_new_assignments_timestamp
+    e = Verdict::Experiment.new('test') do
+      groups do
+        group :a, :half
+        group :b, :half
+      end
+    end
+
+    assert_equal :a, e.switch(1)
+
+    e.schedule_stop_new_assignment_timestamp Time.new(2020, 4, 15)
+
+    # switch respects to stop new assignment timestamp, old assignment preserves, new assignment returns nil
+    Timecop.freeze(Time.new(2020, 4, 16)) do
+      assert !e.send(:is_make_new_assignments?)
+      # old assignment stay the same
+      assert_equal :a, e.switch(1)
+      # new assignment returns nil
+      assert_nil e.switch(2)
+    end
+  end
+
+  def test_schedule_start_timestamp_and_stop_new_assignemnt_timestamp_are_inclusive_but_end_timestamp_is_exclusive
+    e = Verdict::Experiment.new('test') do
+      groups do
+        group :a, :half
+        group :b, :half
+      end
+
+      schedule_start_timestamp Time.new(2020, 1, 1)
+      schedule_stop_new_assignment_timestamp Time.new(2020, 1, 15)
+      schedule_end_timestamp Time.new(2020, 1, 31)
+    end
+
+    # start_timestamp is included
+    Timecop.freeze(Time.new(2020, 1, 1)) do
+      assert e.send(:is_scheduled?)
+      assert_equal :a, e.switch(1)
+    end
+
+    # stop_new_assignment_timestamp is included
+    Timecop.freeze(Time.new(2020, 1, 15)) do
+      assert !e.send(:is_make_new_assignments?)
+      # old assignment preserved
+      assert_equal :a, e.switch(1)
+      # new assignment returns nil
+      assert_nil e.switch(2)
+    end
+
+    # end_timestamp is excluded
+    Timecop.freeze(Time.new(2020, 1, 31)) do
+      assert !e.send(:is_scheduled?)
+      assert_nil e.switch(1)
+    end
+  end
+
+  def test_custom_qualifiers_success
+    e = Verdict::Experiment.new('test') do
+      groups do
+        group :all, 100
+      end
+    end
+
+    subject = 2
+    custom_qualifier_a = Proc.new { |subject| subject.even? }
+    custom_qualifier_b = Proc.new { |subject| subject > 0 }
+
+    group = e.switch(subject, qualifiers: [custom_qualifier_a, custom_qualifier_b])
+    assert_equal e.group(:all).to_sym, group
+  end
+
+  def test_custom_qualifiers_failure
+    e = Verdict::Experiment.new('test') do
+      groups do
+        group :all, 100
+      end
+    end
+
+    subject = 3
+    custom_qualifier_a = Proc.new { |subject| subject.even? }
+    custom_qualifier_b = Proc.new { |subject| subject > 0 }
+
+    e.switch(subject, qualifiers: [custom_qualifier_a, custom_qualifier_b])
+
+    group = e.switch(subject, qualifiers: [custom_qualifier_a, custom_qualifier_b])
+    assert_nil group
+  end
+
+  private
+
+  def redis
+    @redis ||= ::Redis.new(host: REDIS_HOST, port: REDIS_PORT)
+  end
 end

data/test/experiments_repository_test.rb

@@ -28,5 +28,6 @@ class ExperimentTest < Minitest::Test
     json = JSON.parse(Verdict.repository.to_json)
     assert_equal ['test_1', 'test_2'], json.keys
     assert_equal json['test_1'], JSON.parse(e1.to_json)
+    assert_equal json['test_2'], JSON.parse(e2.to_json)
   end
 end

data/test/fake_app.rb

@@ -1,4 +1,4 @@
-require 'rails'
+require "rails"
 require "active_model/railtie"
 require "action_controller/railtie"
 require "action_view/railtie"

data/test/storage/cookie_storage_test.rb

@@ -0,0 +1,81 @@
+# frozen_string_literal: true
+
+require 'test_helper'
+require 'fake_app'
+
+class CookieStorageTest < Minitest::Test
+  def setup
+    @storage = Verdict::Storage::CookieStorage.new.tap do |s|
+      s.cookies = ActionDispatch::Cookies::CookieJar.new(nil)
+    end
+    @experiment = Verdict::Experiment.new(:cookie_storage_test) do
+      groups { group :all, 100 }
+      storage @storage, store_unqualified: true
+    end
+    @subject = stub(id: 'bob')
+    @assignment = Verdict::Assignment.new(@experiment, @subject, @experiment.group(:all), nil)
+  end
+
+  def test_cookie_lifespan_has_a_default
+    cookie_lifespan = Verdict::Storage::CookieStorage.new.cookie_lifespan
+
+    assert_equal Verdict::Storage::CookieStorage::DEFAULT_COOKIE_LIFESPAN_SECONDS, cookie_lifespan
+  end
+
+  def test_cookie_lifespan_can_be_configured
+    storage = Verdict::Storage::CookieStorage.new(cookie_lifespan: 60)
+
+    assert_equal 60, storage.cookie_lifespan
+  end
+
+  def test_raises_storage_error_when_cookies_is_nil
+    storage = Verdict::Storage::CookieStorage.new
+
+    assert_raises(Verdict::StorageError) { storage.store_assignment(@assignment) }
+    assert_raises(Verdict::StorageError) { storage.retrieve_assignment(@experiment, @subject) }
+    assert_raises(Verdict::StorageError) { storage.remove_assignment(@experiment, @subject) }
+  end
+
+  def test_store_assignment_returns_true_when_an_assignment_is_stored
+    assert @storage.store_assignment(@assignment)
+    refute_nil @storage.retrieve_assignment(@experiment, @subject)
+  end
+
+  def test_retrieve_assignment_returns_an_assignment
+    @storage.store_assignment(@assignment)
+    assignment = @storage.retrieve_assignment(@experiment, @subject)
+
+    assert assignment.returning?
+    assert_equal :all, assignment.handle.to_sym
+    assert_equal @experiment, assignment.experiment
+    assert_equal @subject, assignment.subject
+  end
+
+  def test_retrieve_assignment_returns_nil_when_an_assignment_does_not_exist
+    assert_nil @storage.retrieve_assignment(@experiment, @subject)
+  end
+
+  def test_retrieve_assignment_returns_nil_when_the_assignment_group_is_invalid
+    invalid_group = Verdict::Group.new(@experiment, :invalid)
+    assignment = Verdict::Assignment.new(@experiment, @subject, invalid_group, nil)
+
+    @storage.store_assignment(assignment)
+
+    assert_nil @storage.retrieve_assignment(@experiment, @subject)
+  end
+
+  def test_remove_assignment_returns_true_when_removing_an_assignment
+    @storage.store_assignment(@assignment)
+
+    assert @storage.remove_assignment(@experiment, nil)
+    assert_nil @storage.retrieve_assignment(@experiment, @subject)
+  end
+
+  def test_retrieve_start_timestamp_always_returns_nil
+    assert_nil @storage.retrieve_start_timestamp(nil)
+  end
+
+  def test_store_start_timestamp_raises_not_implemented_error
+    assert_raises(NotImplementedError) { @storage.store_start_timestamp(nil, nil) }
+  end
+end

data/test/storage/memory_storage_test.rb

@@ -35,9 +35,9 @@ class MemoryStorageTest < Minitest::Test
   end
 
   def test_started_at
-    assert @storage.get(@experiment.handle.to_s, 'started_at').nil?
+    assert @storage.send(:get, @experiment.handle.to_s, 'started_at').nil?
     @experiment.send(:ensure_experiment_has_started)
-    refute @storage.get(@experiment.handle.to_s, 'started_at').nil?
+    refute @storage.send(:get, @experiment.handle.to_s, 'started_at').nil?
     assert_instance_of Time, @experiment.started_at
   end
 end

data/test/storage/redis_storage_test.rb

@@ -71,17 +71,29 @@ class RedisStorageTest < Minitest::Test
     assert_equal a, @experiment.started_at
   end
 
-  def test_cleanup
+  def test_cleanup_with_scope_argument
     1000.times do |n|
       @experiment.assign("something_#{n}")
     end
 
     assert_operator @redis, :exists, experiment_key
 
+    Verdict.default_logger.expects(:warn).with(regexp_matches(/deprecated/))
     @storage.cleanup(:redis_storage)
     refute_operator @redis, :exists, experiment_key
   end
 
+  def test_cleanup
+    1000.times do |n|
+      @experiment.assign("something_#{n}")
+    end
+
+    assert_operator @redis, :exists, experiment_key
+
+    @storage.cleanup(@experiment)
+    refute_operator @redis, :exists, experiment_key
+  end
+
   private
 
   def experiment_key

data/test/verdict_rails_test.rb

@@ -5,9 +5,16 @@ class VerdictRailsTest < Minitest::Test
   def setup
     Verdict.clear_repository_cache
     new_rails_app = Dummy::Application.new
+    new_rails_app.config.eager_load = false
     new_rails_app.initialize!
   end
 
+  def teardown
+    Verdict.default_logger = Logger.new("/dev/null")
+    Verdict.directory = nil
+    Verdict.clear_repository_cache
+  end
+
   def test_verdict_railtie_should_find_directory_path
     assert_equal Verdict.directory, Rails.root.join('app', 'experiments')
   end
@@ -16,4 +23,4 @@ class VerdictRailsTest < Minitest::Test
     expected_experiment = Verdict.instance_variable_get('@repository')
     assert expected_experiment.include?("test_rails_app_experiment")
   end
-end
+end

data/verdict.gemspec

@@ -17,6 +17,8 @@ Gem::Specification.new do |gem|
   gem.test_files    = gem.files.grep(%r{^(test|spec|features)/})
   gem.require_paths = ["lib"]
 
+  gem.metadata["allowed_push_host"] = "https://rubygems.org"
+
   gem.add_development_dependency("minitest", '~> 5.2')
   gem.add_development_dependency("rake")
   gem.add_development_dependency("mocha")

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: verdict
 version: !ruby/object:Gem::Version
-  version: 0.10.0
+  version: 0.15.0
 platform: ruby
 authors:
 - Shopify
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2019-08-22 00:00:00.000000000 Z
+date: 2020-06-30 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: minitest
@@ -106,10 +106,12 @@ files:
 - ".gitignore"
 - ".travis.yml"
 - CHANGELOG.md
+- CONTRIBUTING.md
 - Gemfile
 - LICENSE
 - README.md
 - Rakefile
+- docs/concepts.md
 - lib/verdict.rb
 - lib/verdict/assignment.rb
 - lib/verdict/conversion.rb
@@ -126,6 +128,7 @@ files:
 - lib/verdict/segmenters/static_segmenter.rb
 - lib/verdict/storage.rb
 - lib/verdict/storage/base_storage.rb
+- lib/verdict/storage/cookie_storage.rb
 - lib/verdict/storage/legacy_redis_storage.rb
 - lib/verdict/storage/memory_storage.rb
 - lib/verdict/storage/mock_storage.rb
@@ -148,6 +151,7 @@ files:
 - test/segmenters/rollout_segmenter_test.rb
 - test/segmenters/static_segmenter_test.rb
 - test/storage/base_storage_test.rb
+- test/storage/cookie_storage_test.rb
 - test/storage/legacy_redis_storage_test.rb
 - test/storage/memory_storage_test.rb
 - test/storage/redis_storage_test.rb
@@ -156,7 +160,8 @@ files:
 - verdict.gemspec
 homepage: http://github.com/Shopify/verdict
 licenses: []
-metadata: {}
+metadata:
+  allowed_push_host: https://rubygems.org
 post_install_message: 
 rdoc_options: []
 require_paths:
@@ -193,6 +198,7 @@ test_files:
 - test/segmenters/rollout_segmenter_test.rb
 - test/segmenters/static_segmenter_test.rb
 - test/storage/base_storage_test.rb
+- test/storage/cookie_storage_test.rb
 - test/storage/legacy_redis_storage_test.rb
 - test/storage/memory_storage_test.rb
 - test/storage/redis_storage_test.rb