verdict 0.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,15 @@
1
+ ---
2
+ !binary "U0hBMQ==":
3
+ metadata.gz: !binary |-
4
+ MDEzN2VlY2MyNGUyZmQ0MzY2YWViYTQ5NmRjMmVhZjUzMmFjZDNlYg==
5
+ data.tar.gz: !binary |-
6
+ YmVlMzgxMDAxZDRjOGI2NmI0MzY2OTNkYzkxOGY2Y2VmNjFkM2EyMg==
7
+ SHA512:
8
+ metadata.gz: !binary |-
9
+ MzJhMDlmYTRiMjI4NTRiZDI5ZGM4YWY2MTNkNTQxNzFiYjZkZDdkNGE1MmJm
10
+ NDdjZTEzMWU1NTBmNDllNDJhODRhYTcxYTNiYTgwOTI2NjA5ZmIyYTVlN2E1
11
+ ZjZjNzVkZGQyNTBlNGFiNDYzYmI2YTFjOWU4MDRmZWQzMjk2MjE=
12
+ data.tar.gz: !binary |-
13
+ MGZmYTI3ODA4ZGMzMzhlMTM4ZThlYWZkMGU3NWE3YWYzZmY4NzY2ZDk0OGQ2
14
+ ZDEyYTUzN2VhYmU4MTdlNjRlY2M4OWI1MmJlOWMwNmZhMDRkMzg0OWQ2NGE1
15
+ ZGEwMmY3ODFjNDUwNjdlNGZmMDM4NmUyOWRjMmRlZGFiMjM4NTQ=
@@ -0,0 +1,17 @@
1
+ *.gem
2
+ *.rbc
3
+ .bundle
4
+ .config
5
+ .yardoc
6
+ Gemfile.lock
7
+ InstalledFiles
8
+ _yardoc
9
+ coverage
10
+ doc/
11
+ lib/bundler/man
12
+ pkg
13
+ rdoc
14
+ spec/reports
15
+ test/tmp
16
+ test/version_tmp
17
+ tmp
@@ -0,0 +1,13 @@
1
+ language: ruby
2
+ script: bundle exec rake
3
+ rvm:
4
+ - 1.9.3
5
+ - 2.0.0
6
+ - ruby-head
7
+ - rbx
8
+ - jruby-19mode
9
+ matrix:
10
+ allow_failures:
11
+ - rvm: ruby-head
12
+ services:
13
+ - redis-server
data/Gemfile ADDED
@@ -0,0 +1,6 @@
1
+ source 'https://rubygems.org'
2
+
3
+ # Specify your gem's dependencies in experiments.gemspec
4
+ gemspec
5
+ gem "rubysl", platform: :rbx
6
+ gem "json", platform: :rbx
data/LICENSE ADDED
@@ -0,0 +1,20 @@
1
+ Copyright (c) 2012-2013 Shopify Inc.
2
+
3
+ Permission is hereby granted, free of charge, to any person obtaining
4
+ a copy of this software and associated documentation files (the
5
+ "Software"), to deal in the Software without restriction, including
6
+ without limitation the rights to use, copy, modify, merge, publish,
7
+ distribute, sublicense, and/or sell copies of the Software, and to
8
+ permit persons to whom the Software is furnished to do so, subject to
9
+ the following conditions:
10
+
11
+ The above copyright notice and this permission notice shall be
12
+ included in all copies or substantial portions of the Software.
13
+
14
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
15
+ EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
16
+ MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
17
+ NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
18
+ LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
19
+ OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
20
+ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
@@ -0,0 +1,89 @@
1
+ # Verdict
2
+
3
+ This library allows you to define and use experiments in your application.
4
+
5
+ - This library can be used in any Ruby application, and comes with a `Railtie` to
6
+ make integrating it with a Rails app easy.
7
+ - This library only handles consistently assigning subjects to experiment groups,
8
+ and storing/logging these assignments for analysis. It doesn't do any analysis
9
+ of results. That should happen elsewhere, e.g. in a data warehouse environment.
10
+
11
+
12
+ ## Installation
13
+
14
+ Add this line to your application's Gemfile, and run `bundle install`:
15
+
16
+ gem 'verdict'
17
+
18
+ ## Usage
19
+
20
+ This gem contains the `Experiments::Experiment` model used create the experiment instance,
21
+ in order consistently modify application behaviour based on an object's unique key.
22
+
23
+ Define an experiment like so:
24
+
25
+ ``` ruby
26
+ Verdict::Experiment.define :my_experiment do
27
+
28
+ # This block should return true if the subject is qualified to participate
29
+ qualify { |subject, context| ... }
30
+
31
+ # Specify the groups and the percentages
32
+ groups do
33
+ group :test, :half
34
+ group :control, :rest
35
+ end
36
+
37
+ # Specify how assignments will be stored.
38
+ storage Verdict::Storage::MemoryStorage.new
39
+ end
40
+ ```
41
+
42
+ Usually you want to place this in a file called **my_experiment.rb** in the
43
+ **/app/experiments** folder. Also, usually you want to subclass `Experiments::Experiment`
44
+ to set some default options for your app's environment, and call `define` on that class
45
+ instead.
46
+
47
+ Refer to the experiment elsewhere in your codebase like this:
48
+
49
+ ``` ruby
50
+ context = { ... } # anything you want to pass along to the qualify block.
51
+ case Experiments['my experiment'].switch(shop, context)
52
+ when :test
53
+ # Handle test group
54
+ when :control
55
+ # Handle control group
56
+ else
57
+ # Handle unqualified subjects.
58
+ end
59
+ ```
60
+
61
+ ## Storage & logging
62
+
63
+ The library uses a basic interface to store experiment assignments. Except for
64
+ a development-only memory store, it doesn't include any storage models.
65
+
66
+ You can set up storage by calling the `storage` method of your experiment, with
67
+ an object that responds to the following tho methods:
68
+
69
+ - `def retrieve_assignment(experiment, subject_identifier)`
70
+ - `def store_assignment(assignment)`
71
+
72
+ In which `experiment` is the Experiment instance, `subject_identifier` is a
73
+ string that uniquely identifies the subject, and `assignment` is an
74
+ `Experiment::Assignment` instance. By default it will use `subject.id.to_s` as
75
+ `subject_identifier`, but you can change that by overriding the
76
+ `def subject_identifier(subject)` method on the experiment.
77
+
78
+ The library will also log every assignment to `Experiments.logger`. The Railtie
79
+ sets `Experiment.logger` to `Rails.logger`, so experiment assignments will show
80
+ up in your Rails log. You can override the logging by overriding the
81
+ `def log_assignment(assignment)` method on the experiment.
82
+
83
+ ## Contributing
84
+
85
+ 1. Fork it
86
+ 2. Create your feature branch (`git checkout -b my-new-feature`)
87
+ 3. Commit your changes, including tests (`git commit -am 'Added some feature'`)
88
+ 4. Push to the branch (`git push origin my-new-feature`)
89
+ 5. Create new Pull Request, and mention @wvanbergen.
@@ -0,0 +1,11 @@
1
+ require "bundler/gem_tasks"
2
+
3
+ require 'rake/testtask'
4
+
5
+ Rake::TestTask.new(:test) do |t|
6
+ t.test_files = Dir['test/**/*_test.rb']
7
+ t.libs << 'test'
8
+ t.verbose = true
9
+ end
10
+
11
+ task default: :test
@@ -0,0 +1,54 @@
1
+ require 'logger'
2
+
3
+ module Verdict
4
+ extend self
5
+
6
+ attr_accessor :default_logger, :directory
7
+
8
+ def [](handle)
9
+ Verdict.repository[handle.to_s]
10
+ end
11
+
12
+ def repository
13
+ if @repository.nil?
14
+ @repository = {}
15
+ discovery
16
+ end
17
+
18
+ @repository
19
+ end
20
+
21
+ def discovery
22
+ Dir[File.join(Verdict.directory, '**', '*.rb')].each { |f| require f } if @directory
23
+ end
24
+
25
+ class Error < StandardError; end
26
+ class SegmentationError < Verdict::Error; end
27
+ class InvalidSubject < Verdict::Error; end
28
+ class EmptySubjectIdentifier < Verdict::Error; end
29
+ class StorageError < Verdict::Error; end
30
+
31
+ class ExperimentHandleNotUnique < Verdict::Error
32
+ attr_reader :handle
33
+
34
+ def initialize(handle)
35
+ @handle = handle
36
+ super("Another experiment with handle #{handle.inspect} is already defined!")
37
+ end
38
+ end
39
+ end
40
+
41
+ require "verdict/version"
42
+ require "verdict/railtie" if defined?(Rails)
43
+
44
+ require "verdict/metadata"
45
+ require "verdict/experiment"
46
+ require "verdict/group"
47
+ require "verdict/assignment"
48
+ require "verdict/conversion"
49
+ require "verdict/segmenter"
50
+ require "verdict/storage"
51
+ require "verdict/event_logger"
52
+
53
+ Verdict.default_logger ||= Logger.new("/dev/null")
54
+ Verdict.directory = nil
@@ -0,0 +1,61 @@
1
+ class Verdict::Assignment
2
+
3
+ attr_reader :experiment, :subject_identifier, :group, :created_at
4
+
5
+ def initialize(experiment, subject_identifier, group, originally_created_at)
6
+ @experiment = experiment
7
+ @subject_identifier = subject_identifier
8
+ @group = group
9
+ @returning = !originally_created_at.nil?
10
+ @created_at = originally_created_at || Time.now.utc
11
+ end
12
+
13
+ def subject
14
+ @subject ||= experiment.fetch_subject(subject_identifier)
15
+ end
16
+
17
+ def qualified?
18
+ !group.nil?
19
+ end
20
+
21
+ def returning
22
+ self.class.new(@experiment, @subject_identifier, @group, @created_at)
23
+ end
24
+
25
+ def returning?
26
+ @returning
27
+ end
28
+
29
+ def handle
30
+ qualified? ? group.handle : nil
31
+ end
32
+
33
+ def to_sym
34
+ qualified? ? group.to_sym : nil
35
+ end
36
+
37
+ def as_json(options = {})
38
+ {
39
+ experiment: experiment.handle,
40
+ subject: subject_identifier,
41
+ qualified: qualified?,
42
+ returning: returning?,
43
+ group: qualified? ? group.handle : nil,
44
+ created_at: created_at.utc.strftime('%FT%TZ')
45
+ }
46
+ end
47
+
48
+ def to_json(options = {})
49
+ as_json(options).to_json
50
+ end
51
+
52
+ def ===(other)
53
+ case other
54
+ when nil; !qualified?
55
+ when Verdict::Assignment; other.group === group
56
+ when Verdict::Group; other === group
57
+ when Symbol, String; qualified? ? group.handle == other.to_s : false
58
+ else false
59
+ end
60
+ end
61
+ end
@@ -0,0 +1,32 @@
1
+ class Verdict::Conversion
2
+
3
+ attr_reader :experiment, :subject_identifier, :goal, :created_at
4
+
5
+ def initialize(experiment, subject_identifier, goal, created_at = Time.now.utc)
6
+ @experiment = experiment
7
+ @subject_identifier = subject_identifier
8
+ @goal = goal
9
+ @created_at = created_at
10
+ end
11
+
12
+ def subject
13
+ experiment.fetch_subject(subject_identifier)
14
+ end
15
+
16
+ def assignment
17
+ experiment.fetch_assignment(subject_identifier)
18
+ end
19
+
20
+ def as_json(options = {})
21
+ {
22
+ experiment: experiment.handle,
23
+ subject: subject_identifier,
24
+ goal: goal,
25
+ created_at: created_at.utc.strftime('%FT%TZ')
26
+ }
27
+ end
28
+
29
+ def to_json(options = {})
30
+ as_json(options).to_json
31
+ end
32
+ end
@@ -0,0 +1,20 @@
1
+ class Verdict::EventLogger
2
+ attr_reader :logger, :level
3
+
4
+ def initialize(logger, level = :info)
5
+ @logger, @level = logger, level
6
+ end
7
+
8
+ def log_assignment(assignment)
9
+ status = assignment.returning? ? 'returning' : 'new'
10
+ if assignment.qualified?
11
+ logger.send(level, "[Verdict::Assignment] experiment=#{assignment.experiment.handle} subject=#{assignment.subject_identifier} status=#{status} qualified=true group=#{assignment.group.handle}")
12
+ else
13
+ logger.send(level, "[Verdict::Assignment] experiment=#{assignment.experiment.handle} subject=#{assignment.subject_identifier} status=#{status} qualified=false")
14
+ end
15
+ end
16
+
17
+ def log_conversion(conversion)
18
+ logger.send(level, "[Verdict::Conversion] experiment=#{conversion.experiment.handle} subject=#{conversion.subject_identifier} goal=#{conversion.goal}")
19
+ end
20
+ end
@@ -0,0 +1,220 @@
1
+ class Verdict::Experiment
2
+
3
+ include Verdict::Metadata
4
+
5
+ attr_reader :handle, :qualifier, :subject_storage, :event_logger
6
+
7
+ def self.define(handle, *args, &block)
8
+ experiment = self.new(handle, *args, &block)
9
+ raise Verdict::ExperimentHandleNotUnique.new(experiment.handle) if Verdict.repository.has_key?(experiment.handle)
10
+ Verdict.repository[experiment.handle] = experiment
11
+ end
12
+
13
+ def initialize(handle, options = {}, &block)
14
+ @handle = handle.to_s
15
+
16
+ options = default_options.merge(options)
17
+ @qualifier = options[:qualifier]
18
+ @event_logger = options[:event_logger] || Verdict::EventLogger.new(Verdict.default_logger)
19
+ @subject_storage = options[:storage] || Verdict::Storage::MemoryStorage.new
20
+ @store_unqualified = options[:store_unqualified]
21
+ @segmenter = options[:segmenter]
22
+ @subject_type = options[:subject_type]
23
+ instance_eval(&block) if block_given?
24
+ end
25
+
26
+ def subject_type(type = nil)
27
+ return @subject_type if type.nil?
28
+ @subject_type = type
29
+ end
30
+
31
+ def store_unqualified?
32
+ @store_unqualified
33
+ end
34
+
35
+ def group(handle)
36
+ segmenter.groups[handle.to_s]
37
+ end
38
+
39
+ def groups(segmenter_class = Verdict::Segmenter::StaticPercentage, &block)
40
+ return @segmenter.groups unless block_given?
41
+ @segmenter ||= segmenter_class.new(self)
42
+ @segmenter.instance_eval(&block)
43
+ @segmenter.verify!
44
+ return self
45
+ end
46
+
47
+ def qualify(&block)
48
+ @qualifier = block
49
+ end
50
+
51
+ def storage(subject_storage, options = {})
52
+ @store_unqualified = options[:store_unqualified] if options.has_key?(:store_unqualified)
53
+ @subject_storage = subject_storage
54
+ end
55
+
56
+ def segmenter
57
+ raise Verdict::Error, "No groups defined for experiment #{@handle.inspect}." if @segmenter.nil?
58
+ @segmenter
59
+ end
60
+
61
+ def started_at
62
+ @started_at ||= @subject_storage.retrieve_start_timestamp(self)
63
+ end
64
+
65
+ def started?
66
+ !@started_at.nil?
67
+ end
68
+
69
+ def group_handles
70
+ segmenter.groups.keys
71
+ end
72
+
73
+ def subject_assignment(subject_identifier, group, originally_created_at = nil)
74
+ Verdict::Assignment.new(self, subject_identifier, group, originally_created_at)
75
+ end
76
+
77
+ def subject_conversion(subject_identifier, goal, created_at = Time.now.utc)
78
+ Verdict::Conversion.new(self, subject_identifier, goal, created_at)
79
+ end
80
+
81
+ def convert(subject, goal)
82
+ identifier = retrieve_subject_identifier(subject)
83
+ conversion = subject_conversion(identifier, goal)
84
+ event_logger.log_conversion(conversion)
85
+ conversion
86
+ end
87
+
88
+ def assign(subject, context = nil)
89
+ identifier = retrieve_subject_identifier(subject)
90
+ assignment = if store_unqualified?
91
+ assignment_with_unqualified_persistence(identifier, subject, context)
92
+ else
93
+ assignment_without_unqualified_persistence(identifier, subject, context)
94
+ end
95
+
96
+ store_assignment(assignment)
97
+ rescue Verdict::StorageError
98
+ subject_assignment(identifier, nil, nil)
99
+ end
100
+
101
+ def disqualify(subject)
102
+ identifier = retrieve_subject_identifier(subject)
103
+ store_assignment(subject_assignment(identifier, nil, nil))
104
+ end
105
+
106
+ def store_assignment(assignment)
107
+ @subject_storage.store_assignment(assignment) if should_store_assignment?(assignment)
108
+ event_logger.log_assignment(assignment)
109
+ assignment
110
+ end
111
+
112
+ def remove_subject(subject)
113
+ remove_subject_identifier(retrieve_subject_identifier(subject))
114
+ end
115
+
116
+ def remove_subject_identifier(subject_identifier)
117
+ @subject_storage.remove_assignment(self, subject_identifier)
118
+ end
119
+
120
+ def switch(subject, context = nil)
121
+ assign(subject, context).to_sym
122
+ end
123
+
124
+ def lookup(subject)
125
+ lookup_assignment_for_identifier(retrieve_subject_identifier(subject))
126
+ end
127
+
128
+ def lookup_assignment_for_identifier(subject_identifier)
129
+ fetch_assignment(subject_identifier)
130
+ end
131
+
132
+ def wrapup
133
+ @subject_storage.clear_experiment(self)
134
+ end
135
+
136
+ def retrieve_subject_identifier(subject)
137
+ identifier = subject_identifier(subject).to_s
138
+ raise Verdict::EmptySubjectIdentifier, "Subject resolved to an empty identifier!" if identifier.empty?
139
+ identifier
140
+ end
141
+
142
+ def has_qualifier?
143
+ !@qualifier.nil?
144
+ end
145
+
146
+ def everybody_qualifies?
147
+ !has_qualifier?
148
+ end
149
+
150
+ def as_json(options = {})
151
+ data = {
152
+ handle: handle,
153
+ has_qualifier: has_qualifier?,
154
+ groups: segmenter.groups.values.map { |g| g.as_json(options) },
155
+ metadata: metadata,
156
+ started_at: started_at.nil? ? nil : started_at.utc.strftime('%FT%TZ')
157
+ }
158
+
159
+ data.tap do |data|
160
+ data[:subject_type] = subject_type.to_s unless subject_type.nil?
161
+ end
162
+ end
163
+
164
+ def to_json(options = {})
165
+ as_json(options).to_json
166
+ end
167
+
168
+ def fetch_subject(subject_identifier)
169
+ raise NotImplementedError, "Fetching subjects based in identifier is not implemented for eperiment @{handle.inspect}."
170
+ end
171
+
172
+ def fetch_assignment(subject_identifier)
173
+ @subject_storage.retrieve_assignment(self, subject_identifier)
174
+ end
175
+
176
+ protected
177
+
178
+ def default_options
179
+ {}
180
+ end
181
+
182
+ def should_store_assignment?(assignment)
183
+ !assignment.returning? && (store_unqualified? || assignment.qualified?)
184
+ end
185
+
186
+ def assignment_with_unqualified_persistence(subject_identifier, subject, context)
187
+ fetch_assignment(subject_identifier) || (
188
+ subject_qualifies?(subject, context) ?
189
+ subject_assignment(subject_identifier, @segmenter.assign(subject_identifier, subject, context), nil) :
190
+ subject_assignment(subject_identifier, nil, nil)
191
+ )
192
+ end
193
+
194
+ def assignment_without_unqualified_persistence(subject_identifier, subject, context)
195
+ if subject_qualifies?(subject, context)
196
+ fetch_assignment(subject_identifier) ||
197
+ subject_assignment(subject_identifier, @segmenter.assign(subject_identifier, subject, context), nil)
198
+ else
199
+ subject_assignment(subject_identifier, nil, nil)
200
+ end
201
+ end
202
+
203
+ def subject_identifier(subject)
204
+ subject.respond_to?(:id) ? subject.id : subject.to_s
205
+ end
206
+
207
+ def subject_qualifies?(subject, context = nil)
208
+ ensure_experiment_has_started
209
+ everybody_qualifies? || @qualifier.call(subject, context)
210
+ end
211
+
212
+ def set_start_timestamp
213
+ @subject_storage.store_start_timestamp(self, started_now = Time.now.utc)
214
+ started_now
215
+ end
216
+
217
+ def ensure_experiment_has_started
218
+ @started_at ||= @subject_storage.retrieve_start_timestamp(self) || set_start_timestamp
219
+ end
220
+ end