verdict 0.12.0 → 0.13.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 7e99d3cb09e2dbfb3cafeb6f05171978721cd5a79fd4570f8b6bd6c6e4975e17
-  data.tar.gz: 4a885cb9d8a991046e3a22e428a35982bf7565b03aaa49fa0da0b9963e34bc56
+  metadata.gz: ec6df6bdd38dc947ff1072e1dce844ab4d3ddd522701c8de578a35b4c9c51be8
+  data.tar.gz: e6b62b2c579a363f37526079389bea4cb16eb28ed7573dec2d3fa8f8c74ee4ed
 SHA512:
-  metadata.gz: 89c2d20c1892c630942099ac0b0ea1d3db012a6aaa9031a7bbf8d834fad295adfaf4cc0194d09930608c4416bbeb8efba89852446e02b091c237d851d8266704
-  data.tar.gz: cdb0271d66903d1f39316c2665c68bf8deae733c105e3aecc8bb665adaef4981129bb0f7cce6baf001eb7f246af50e40d85115bf3105525a44b2111fb2c41fbc
+  metadata.gz: 1399fa37c4575888ffb83aea54469784db11ceef1ebc99f5c1fa5de75b936f460b1f34dc717c28f4f19a6e969c3a85a3faad022119113cd94f754ad61b4fdb85
+  data.tar.gz: 7f1836b8d23ca79a4338f0a4f4ed244ebe7bdbd8b9f03529f2cb00e110c280b1033e112928da900d805112eb0036e0f74278ea021514016541accf01d8c3308c

data/.gitignore

@@ -16,3 +16,4 @@ test/tmp
 test/version_tmp
 tmp
 dump.rdb
+log

data/.travis.yml

@@ -11,5 +11,6 @@ rvm:
 matrix:
   allow_failures:
     - rvm: ruby-head
+    - rvm: jruby
 services:
   - redis-server

data/CHANGELOG.md

@@ -1,8 +1,14 @@
+## v0.13.0
+
+* Add optional experiment definition methods `schedule_start_timestamp` and `schedule_end_timestamp` to support limiting experiment's lifetime in a pre-determined time interval.
+* Support eager loading from within a Rails app using Zeitwerk.
+* Add `CookieStorage` storage backend. This backend is a distributed store for Verdict and does not support experiment timestamps. It is designed to be used with Rails applications and requires that `.cookies` be set to the `CookieJar` instance before use.
+
 ## v0.12.0
 
 * Allow options to be passed to `Experiment#cleanup` so they can be forwarded to storage.
 
-* Changed `Experiment#cleanup` to accept an argument of type `Verdict::Experiment`.       
+* Changed `Experiment#cleanup` to accept an argument of type `Verdict::Experiment`.
   Passing a `String`/`Symbol` argument is still supported, but will log a deprecation warning.
 
 ## v0.11.0

data/CONTRIBUTING.md

@@ -0,0 +1,7 @@
+## Contributing
+
+1. Fork it
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes, including tests (`git commit -am 'Added some feature'`)
+4. Push to the branch (`git push origin my-new-feature`)
+5. Create new Pull Request

data/README.md

@@ -19,6 +19,8 @@ Add this line to your application's Gemfile, and run `bundle install`:
 
 If you're using Rails, the Railtie will handle setting the logger to `Rails.logger` and the experiments directory to `app/experiments`. It will also load the rake tasks for you (run `bundle exec rake -T | grep experiments:` for options).
 
+You may find the [Concepts](docs/concepts.md) documentation a good place to familiarise yourself with Verdict's nomenclature.
+
 ## Usage
 
 The `Verdict::Experiment` class is used to create an experiment, define control and experiment groups, and to qualify subjects.
@@ -42,8 +44,7 @@ Verdict::Experiment.define :my_experiment do
 end
 ```
 
-Usually you'll want to place this in a file called **my_experiment.rb** in the
-**/app/experiments** folder.
+Usually you'll want to place this in a file called **my_experiment.rb** in the **/app/experiments** folder.
 
 _We recommend that you subclass `Verdict::Experiment` to set some default options for your app's environment, and call `define` on that class instead._
 
@@ -71,9 +72,9 @@ Verdict uses a very simple interface for storing experiment assignments. Out of
 
 * Memory
 * Redis
+* Cookies
 
-You can set up storage for your experiment by calling the `storage` method with
-an object that responds to the following methods:
+You can set up storage for your experiment by calling the `storage` method with an object that responds to the following methods:
 
 * `store_assignment(assignment)`
 * `retrieve_assignment(experiment, subject)`
@@ -83,14 +84,15 @@ an object that responds to the following methods:
 
 Regarding the method signatures above, `experiment` is the Experiment instance, `subject` is the Subject instance, and `assignment` is a `Verdict::Assignment` instance.
 
-The `subject` instance will be identified internally by its `subject_identifier`
-By default it will use `subject.id.to_s` as `subject_identifier`, but you can change that by overriding `def subject_identifier(subject)` on the experiment.
+The `subject` instance will be identified internally by its `subject_identifier`. By default it will use `subject.id.to_s` as `subject_identifier`, but you can change that by overriding `def subject_identifier(subject)` on the experiment.
 
 Storage providers simply store subject assignments and require quick lookups of subject identifiers. They allow for complex (high CPU) assignments, and for assignments that might not always put the same subject in the same group by storing the assignment for later use.
 
-Storage providers are intended for operational use and should not be used for data analysis. For data analysis, you should use the logger.
+Storage providers are intended for operational use and should not be used for data analysis. For data analysis, you should use [the logger](#logging).
+
+When removing old experiments you might want to clean up corresponding experiment assignments, to reduce the amount of data stored and loaded. By using the logger, this data removal doesn't impact historic data or data analysis.
 
-For more details about these methods, check out the source code for [Verdict::Storage::MockStorage](lib/verdict/storage/mock_storage.rb)
+For more details about these methods, check out the source code for [`Verdict::Storage::MockStorage`](lib/verdict/storage/mock_storage.rb)
 
 ## Logging
 
@@ -101,12 +103,3 @@ You can override the logging by overriding the `def log_assignment(assignment)`
 Logging (as opposed to storage) should be used for data analysis. The logger requires a write-only / forward-only stream to write to, e.g. a log file, Kafka, or an insert-only database table.
 
 It's possible to run an experiment without defining any storage, though this comes with several drawbacks. Logging on the other hand is required in order to analyze the results.
-
-
-## Contributing
-
-1. Fork it
-2. Create your feature branch (`git checkout -b my-new-feature`)
-3. Commit your changes, including tests (`git commit -am 'Added some feature'`)
-4. Push to the branch (`git push origin my-new-feature`)
-5. Create new Pull Request, and mention @wvanbergen.

data/docs/concepts.md

@@ -0,0 +1,35 @@
+# Verdict Concepts
+
+Understanding the following Verdict concepts can help you implement experiments and discuss them with peers.
+
+- [Experiment](#experiment)
+- [Qualifiers](#qualifiers)
+- [Subject](#subject)
+- [Group](#group)
+- [Assignment](#assignment)
+
+### Experiment
+
+`Experiment`s are defined in `app/experiments` and encapsulate the details of the test including metadata such as the name, description or owners, as well as any [qualifiers](#qualifiers) or the [groups](#groups) available in the test.
+
+### Qualifiers
+
+Qualifiers are methods that take two parameters, `subject` and `context`. They return `true` if the `subject` can take this experiment, or `false` if the `subject` is not eligible. Experiments can have multiple qualifiers which are evaluated in succession. If a qualifier returns `false`, no further qualifiers are checked.
+
+Examples of qualifiers might be subjects that:
+
+- Originate from certain browsers or devices
+- Have a specific `Accept-Language` header
+- Have added items to their cart.
+
+### Subject
+
+A subject is the thing that will be [qualified](#qualifiers), and if successfull, [assigned](#assignment) to a particular [group](#group). For example, a subject might represent a user.
+
+### Group
+
+Each [experiment](#experiment) will have two or more groups, usually a control group and a test group. Qualified [subjects](#subject) are assigned to one of the groups in the experiment.
+
+### Assignment
+
+An assignment occurs when a qualified [subject](#subject) is placed into a [group](#group). Assignments are persisted so repeat visits by the subject skip [qualification](#qualifiers) and re-assignment and simply returns the existing assignment.

data/lib/verdict/experiment.rb

@@ -52,6 +52,17 @@ class Verdict::Experiment
     return self
   end
 
+  # Optional: Together with the "end timestamp", limits the experiment run timeline within
+  # the given time interval. When experiment is not scheduled, subject switch returns nil.
+  # This is useful when the experimenter requires the experiment to run in a strict timeline.
+  def schedule_start_timestamp(timestamp)
+    @schedule_start_timestamp = timestamp
+  end
+
+  def schedule_end_timestamp(timestamp)
+    @schedule_end_timestamp = timestamp
+  end
+
   def rollout_percentage(percentage, rollout_group_name = :enabled)
     groups(Verdict::Segmenters::RolloutSegmenter) do
       group rollout_group_name, percentage
@@ -172,6 +183,7 @@ class Verdict::Experiment
   end
 
   def switch(subject, context = nil)
+    return unless is_scheduled?
     assign(subject, context).to_sym
   end
 
@@ -239,10 +251,12 @@ class Verdict::Experiment
   def set_start_timestamp
     @storage.store_start_timestamp(self, started_now = Time.now.utc)
     started_now
+  rescue NotImplementedError
+    nil
   end
 
   def ensure_experiment_has_started
-    @started_at ||= @storage.retrieve_start_timestamp(self) || set_start_timestamp
+    @started_at ||= started_at || set_start_timestamp
   rescue Verdict::StorageError
     @started_at ||= Time.now.utc
   end
@@ -250,4 +264,16 @@ class Verdict::Experiment
   def nil_assignment(subject)
     subject_assignment(subject, nil, nil)
   end
+
+  private
+
+  def is_scheduled?
+    if @schedule_start_timestamp and @schedule_start_timestamp > Time.now
+      return false
+    end
+    if @schedule_end_timestamp and @schedule_end_timestamp < Time.now
+      return false
+    end
+    return true
+  end
 end

data/lib/verdict/railtie.rb

@@ -5,10 +5,15 @@ class Verdict::Railtie < Rails::Railtie
     Verdict.default_logger = Rails.logger
 
     Verdict.directory ||= Rails.root.join('app', 'experiments')
-    app.config.eager_load_paths -= Dir[Verdict.directory.to_s]
 
-    # Re-freeze eager load paths to ensure they blow up if modified at runtime, as Rails does
-    app.config.eager_load_paths.freeze
+    if Rails.gem_version >= Gem::Version.new('6.0.0') && Rails.autoloaders.zeitwerk_enabled?
+      Rails.autoloaders.main.ignore(Verdict.directory)
+    else
+      app.config.eager_load_paths -= Dir[Verdict.directory.to_s]
+
+      # Re-freeze eager load paths to ensure they blow up if modified at runtime, as Rails does
+      app.config.eager_load_paths.freeze
+    end
   end
 
   config.to_prepare do

data/lib/verdict/storage.rb

@@ -1,4 +1,5 @@
 require 'verdict/storage/base_storage'
+require 'verdict/storage/cookie_storage'
 require 'verdict/storage/mock_storage'
 require 'verdict/storage/memory_storage'
 require 'verdict/storage/redis_storage'

data/lib/verdict/storage/cookie_storage.rb

@@ -0,0 +1,105 @@
+# frozen_string_literal: true
+
+module Verdict
+  module Storage
+    # CookieStorage, unlike other Verdict storage classes, is distributed and stored on the client.
+    # Because cookies are opaque to users, we obsfucate information about the test such as the
+    # human readable names for the experiment or assignment group. This means this class assumes
+    # that `name` will be an obsfucated value, or one that we comfortable being "public".
+    class CookieStorage < BaseStorage
+      DEFAULT_COOKIE_LIFESPAN_SECONDS = 15778476 # 6 months
+      KEY = :assignment
+
+      attr_accessor :cookies
+      attr_reader :cookie_lifespan
+
+      def initialize(cookie_lifespan: DEFAULT_COOKIE_LIFESPAN_SECONDS)
+        @cookies = nil
+        @cookie_lifespan = cookie_lifespan
+      end
+
+      def store_assignment(assignment)
+        data = {
+          group: digest_of(assignment.group.to_s),
+          created_at: assignment.created_at.strftime('%FT%TZ')
+        }
+
+        set(assignment.experiment.handle.to_s, KEY, JSON.dump(data))
+      end
+
+      def retrieve_assignment(experiment, subject)
+        if (value = get(experiment.handle.to_s, KEY))
+          data = parse_cookie_value(value)
+          group = find_group_by_digest(experiment, data['group'])
+
+          if group.nil?
+            experiment.remove_subject_assignment(subject)
+            return nil
+          end
+
+          experiment.subject_assignment(subject, group, Time.xmlschema(data['created_at']))
+        end
+      end
+
+      def remove_assignment(experiment, _subject)
+        remove(experiment.handle.to_s, KEY)
+      end
+
+      def retrieve_start_timestamp(_experiment)
+        nil
+      end
+
+      def store_start_timestamp(_experiment, _timestamp)
+        raise NotImplementedError
+      end
+
+      protected
+
+      def get(scope, key)
+        ensure_cookies_is_set
+        cookies[scope_key(scope, key)]
+      end
+
+      def set(scope, key, value)
+        ensure_cookies_is_set
+        cookies[scope_key(scope, key)] = {
+          value: value,
+          expires: Time.now.utc.advance(seconds: cookie_lifespan),
+        }
+      end
+
+      def remove(scope, key)
+        ensure_cookies_is_set
+        cookies.delete(scope_key(scope, key))
+      end
+
+      private
+
+      def ensure_cookies_is_set
+        raise Verdict::StorageError, 'cookies must be an instance of ActionDispatch::Cookies::CookieJar' if cookies.nil?
+      end
+
+      def digest_of(value)
+        Digest::MD5.hexdigest(value)
+      end
+
+      def find_group_by_digest(experiment, digest)
+        experiment.groups.values.find do |group|
+          digest_of(group.to_s) == digest
+        end
+      end
+
+      def parse_cookie_value(value)
+        value = value[:value] if value.is_a?(Hash)
+
+        JSON.parse(value)
+      rescue JSON::ParserError, TypeError
+        {}
+      end
+
+      def scope_key(scope, key)
+        "#{digest_of(scope)}_#{key}"
+      end
+    end
+  end
+end

data/lib/verdict/version.rb

@@ -1,3 +1,3 @@
 module Verdict
-  VERSION = "0.12.0"
+  VERSION = "0.13.0"
 end

data/test/experiment_test.rb

@@ -414,6 +414,16 @@ class ExperimentTest < Minitest::Test
     assert e.started?, "The experiment should have started after the first assignment"
   end
 
+  def test_experiment_set_start_timestamp_handles_storage_that_does_not_implement_timestamps
+    e = Verdict::Experiment.new('starting_test') do
+      groups { group :all, 100 }
+    end
+
+    e.storage.expects(:store_start_timestamp).raises(NotImplementedError)
+
+    assert_nil e.send(:set_start_timestamp)
+  end
+
   def test_no_storage
     e = Verdict::Experiment.new('starting_test') do
       groups { group :all, 100 }
@@ -460,6 +470,101 @@ class ExperimentTest < Minitest::Test
     end
   end
 
+  def test_is_scheduled
+    e = Verdict::Experiment.new(:json) do
+      groups do
+        group :a, :half
+        group :b, :rest
+      end
+      schedule_start_timestamp Time.new(2020, 1, 1)
+      schedule_end_timestamp Time.new(2020, 1, 3)
+    end
+
+    # Within the interval
+    Timecop.freeze(Time.new(2020, 1, 2)) do
+      assert e.send(:is_scheduled?)
+    end
+    # Too early
+    Timecop.freeze(Time.new(2019, 12, 30)) do
+      assert !e.send(:is_scheduled?)
+    end
+    # Too late
+    Timecop.freeze(Time.new(2020, 1, 4)) do
+      assert !e.send(:is_scheduled?)
+    end
+  end
+
+  def test_is_scheduled_no_end_timestamp
+    e = Verdict::Experiment.new(:json) do
+      groups do
+        group :a, :half
+        group :b, :rest
+      end
+      schedule_start_timestamp Time.new(2020, 1, 1)
+    end
+
+    # Within the interval because there is no end date
+    Timecop.freeze(Time.new(2030, 1, 1)) do
+      assert e.send(:is_scheduled?)
+    end
+    # Too early
+    Timecop.freeze(Time.new(2019, 12, 30)) do
+      assert !e.send(:is_scheduled?)
+    end
+  end
+
+  def test_is_scheduled_no_start_timestamp
+    e = Verdict::Experiment.new(:json) do
+      groups do
+        group :a, :half
+        group :b, :rest
+      end
+      schedule_end_timestamp Time.new(2020, 1, 3)
+    end
+
+    # Within the interval because there is no start date
+    Timecop.freeze(Time.new(2019, 12, 30)) do
+      assert e.send(:is_scheduled?)
+    end
+    # Too late
+    Timecop.freeze(Time.new(2020, 1, 4)) do
+      assert !e.send(:is_scheduled?)
+    end
+  end
+
+  def test_switch_respects_time_schedule
+    e = Verdict::Experiment.new('test') do
+      groups do
+        group :a, :half
+        group :b, :rest
+      end
+      schedule_start_timestamp Time.new(2020, 1, 1)
+      schedule_end_timestamp Time.new(2020, 1, 2)
+    end
+
+    Timecop.freeze(Time.new(2020, 1, 3)) do
+      assert_nil e.switch(1)
+    end
+  end
+
+  def test_switch_respects_time_schedule_even_after_assignment
+    e = Verdict::Experiment.new('test') do
+      groups do
+        group :a, :half
+        group :b, :rest
+      end
+    end
+
+    assert_equal :a, e.switch(1)
+
+    e.schedule_start_timestamp Time.new(2020, 1, 1)
+    e.schedule_end_timestamp Time.new(2020, 1, 2)
+
+    Timecop.freeze(Time.new(2020, 1, 3)) do
+      assert_nil e.switch(1)
+    end
+  end
+
   private
 
   def redis

data/test/storage/cookie_storage_test.rb

@@ -0,0 +1,81 @@
+# frozen_string_literal: true
+
+require 'test_helper'
+require 'fake_app'
+
+class CookieStorageTest < Minitest::Test
+  def setup
+    @storage = Verdict::Storage::CookieStorage.new.tap do |s|
+      s.cookies = ActionDispatch::Cookies::CookieJar.new(nil)
+    end
+    @experiment = Verdict::Experiment.new(:cookie_storage_test) do
+      groups { group :all, 100 }
+      storage @storage, store_unqualified: true
+    end
+    @subject = stub(id: 'bob')
+    @assignment = Verdict::Assignment.new(@experiment, @subject, @experiment.group(:all), nil)
+  end
+
+  def test_cookie_lifespan_has_a_default
+    cookie_lifespan = Verdict::Storage::CookieStorage.new.cookie_lifespan
+
+    assert_equal Verdict::Storage::CookieStorage::DEFAULT_COOKIE_LIFESPAN_SECONDS, cookie_lifespan
+  end
+
+  def test_cookie_lifespan_can_be_configured
+    storage = Verdict::Storage::CookieStorage.new(cookie_lifespan: 60)
+
+    assert_equal 60, storage.cookie_lifespan
+  end
+
+  def test_raises_storage_error_when_cookies_is_nil
+    storage = Verdict::Storage::CookieStorage.new
+
+    assert_raises(Verdict::StorageError) { storage.store_assignment(@assignment) }
+    assert_raises(Verdict::StorageError) { storage.retrieve_assignment(@experiment, @subject) }
+    assert_raises(Verdict::StorageError) { storage.remove_assignment(@experiment, @subject) }
+  end
+
+  def test_store_assignment_returns_true_when_an_assignment_is_stored
+    assert @storage.store_assignment(@assignment)
+    refute_nil @storage.retrieve_assignment(@experiment, @subject)
+  end
+
+  def test_retrieve_assignment_returns_an_assignment
+    @storage.store_assignment(@assignment)
+    assignment = @storage.retrieve_assignment(@experiment, @subject)
+
+    assert assignment.returning?
+    assert_equal :all, assignment.handle.to_sym
+    assert_equal @experiment, assignment.experiment
+    assert_equal @subject, assignment.subject
+  end
+
+  def test_retrieve_assignment_returns_nil_when_an_assignment_does_not_exist
+    assert_nil @storage.retrieve_assignment(@experiment, @subject)
+  end
+
+  def test_retrieve_assignment_returns_nil_when_the_assignment_group_is_invalid
+    invalid_group = Verdict::Group.new(@experiment, :invalid)
+    assignment = Verdict::Assignment.new(@experiment, @subject, invalid_group, nil)
+
+    @storage.store_assignment(assignment)
+
+    assert_nil @storage.retrieve_assignment(@experiment, @subject)
+  end
+
+  def test_remove_assignment_returns_true_when_removing_an_assignment
+    @storage.store_assignment(@assignment)
+
+    assert @storage.remove_assignment(@experiment, nil)
+    assert_nil @storage.retrieve_assignment(@experiment, @subject)
+  end
+
+  def test_retrieve_start_timestamp_always_returns_nil
+    assert_nil @storage.retrieve_start_timestamp(nil)
+  end
+
+  def test_store_start_timestamp_raises_not_implemented_error
+    assert_raises(NotImplementedError) { @storage.store_start_timestamp(nil, nil) }
+  end
+end

data/verdict.gemspec

@@ -17,6 +17,8 @@ Gem::Specification.new do |gem|
   gem.test_files    = gem.files.grep(%r{^(test|spec|features)/})
   gem.require_paths = ["lib"]
 
+  gem.metadata["allowed_push_host"] = "https://rubygems.org"
+
   gem.add_development_dependency("minitest", '~> 5.2')
   gem.add_development_dependency("rake")
   gem.add_development_dependency("mocha")

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: verdict
 version: !ruby/object:Gem::Version
-  version: 0.12.0
+  version: 0.13.0
 platform: ruby
 authors:
 - Shopify
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2019-09-11 00:00:00.000000000 Z
+date: 2020-04-03 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: minitest
@@ -106,10 +106,12 @@ files:
 - ".gitignore"
 - ".travis.yml"
 - CHANGELOG.md
+- CONTRIBUTING.md
 - Gemfile
 - LICENSE
 - README.md
 - Rakefile
+- docs/concepts.md
 - lib/verdict.rb
 - lib/verdict/assignment.rb
 - lib/verdict/conversion.rb
@@ -126,6 +128,7 @@ files:
 - lib/verdict/segmenters/static_segmenter.rb
 - lib/verdict/storage.rb
 - lib/verdict/storage/base_storage.rb
+- lib/verdict/storage/cookie_storage.rb
 - lib/verdict/storage/legacy_redis_storage.rb
 - lib/verdict/storage/memory_storage.rb
 - lib/verdict/storage/mock_storage.rb
@@ -148,6 +151,7 @@ files:
 - test/segmenters/rollout_segmenter_test.rb
 - test/segmenters/static_segmenter_test.rb
 - test/storage/base_storage_test.rb
+- test/storage/cookie_storage_test.rb
 - test/storage/legacy_redis_storage_test.rb
 - test/storage/memory_storage_test.rb
 - test/storage/redis_storage_test.rb
@@ -156,7 +160,8 @@ files:
 - verdict.gemspec
 homepage: http://github.com/Shopify/verdict
 licenses: []
-metadata: {}
+metadata:
+  allowed_push_host: https://rubygems.org
 post_install_message: 
 rdoc_options: []
 require_paths:
@@ -193,6 +198,7 @@ test_files:
 - test/segmenters/rollout_segmenter_test.rb
 - test/segmenters/static_segmenter_test.rb
 - test/storage/base_storage_test.rb
+- test/storage/cookie_storage_test.rb
 - test/storage/legacy_redis_storage_test.rb
 - test/storage/memory_storage_test.rb
 - test/storage/redis_storage_test.rb