gitlab-experiment 0.4.12 → 0.5.4
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/README.md +37 -30
- data/lib/generators/gitlab/experiment/install/templates/initializer.rb.tt +13 -10
- data/lib/gitlab/experiment.rb +56 -90
- data/lib/gitlab/experiment/base_interface.rb +88 -0
- data/lib/gitlab/experiment/cache.rb +2 -0
- data/lib/gitlab/experiment/callbacks.rb +7 -16
- data/lib/gitlab/experiment/configuration.rb +31 -9
- data/lib/gitlab/experiment/context.rb +2 -0
- data/lib/gitlab/experiment/dsl.rb +10 -1
- data/lib/gitlab/experiment/rollout.rb +35 -0
- data/lib/gitlab/experiment/rollout/random.rb +15 -0
- data/lib/gitlab/experiment/rollout/round_robin.rb +23 -0
- data/lib/gitlab/experiment/rspec.rb +93 -43
- data/lib/gitlab/experiment/version.rb +1 -1
- metadata +25 -7
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 9d7bc7a1946d06b5188ccfe761342209779f75cf635d894bb8743519ff2830dc
|
4
|
+
data.tar.gz: 7a5c347f8b58e5caa2fe814ce013dbf1ca71f6e017148528a0ff2ca74bd562de
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: '01927d9db97ad4b087d445db642aa39615faa68322e2bd0e0b82ae99441478bae3f1939b94956218121fe52d01f432836ba272d0b1ff052537c80f0d0ff107b3'
|
7
|
+
data.tar.gz: f53702ea5bb97fe6c73427850534b9e156744336a37029dce2b96926b70b7a5c4959708ac626f6ac0435125ad4ffe62f2ba9fd528f756af811c6ff66abc47846
|
data/README.md
CHANGED
@@ -3,7 +3,7 @@ GitLab Experiment
|
|
3
3
|
|
4
4
|
<img alt="experiment" src="/uploads/60990b2dbf4c0406bbf8b7f998de2dea/experiment.png" align="right" width="40%">
|
5
5
|
|
6
|
-
Here at GitLab, we run experiments as A/B/n tests and review the data the experiment generates. From that data, we determine the best performing variant and promote it as the new default code path. Or revert back to the control if no variant outperformed it.
|
6
|
+
Here at GitLab, we run experiments as A/B/n tests and review the data the experiment generates. From that data, we determine the best performing variant and promote it as the new default code path. Or revert back to the control if no variant outperformed it. You can read our [Experiment Guide](https://docs.gitlab.com/ee/development/experiment_guide/) docs if you're curious about how we use this gem internally.
|
7
7
|
|
8
8
|
This library provides a clean and elegant DSL (domain specific language) to define, run, and track your GitLab experiment.
|
9
9
|
|
@@ -64,7 +64,7 @@ end
|
|
64
64
|
|
65
65
|
You can define the experiment using simple control/candidate paths, or provide named variants.
|
66
66
|
|
67
|
-
Handling
|
67
|
+
Handling [multivariate](https://en.wikipedia.org/wiki/Multivariate_statistics) experiments is up to the configuration you provide around resolving variants. But in our example we may want to try with and without the confirmation. We can run any number of variations in our experiments this way.
|
68
68
|
|
69
69
|
```ruby
|
70
70
|
experiment(:notification_toggle, actor: user) do |e|
|
@@ -149,37 +149,13 @@ experiment(:notification_toggle, actor: user) do |e|
|
|
149
149
|
end
|
150
150
|
```
|
151
151
|
|
152
|
-
<details>
|
153
|
-
<summary>You can also use the lower level class interface...</summary>
|
154
|
-
|
155
|
-
### Using the `.run` approach
|
156
|
-
|
157
|
-
This is useful if you haven't included the DSL and so don't have access to the `experiment` method, but still want to execute an experiment. This is ultimately what the `experiment` method calls through to, and the method signatures are the same.
|
158
|
-
|
159
|
-
```ruby
|
160
|
-
exp = Gitlab::Experiment.run(:notification_toggle, actor: user) do |e|
|
161
|
-
# Context may be passed in the block, but must be finalized before calling
|
162
|
-
# run or track.
|
163
|
-
e.context(project: project) # add the project to the context
|
164
|
-
|
165
|
-
# Define the control and candidate variant.
|
166
|
-
e.use { render_toggle } # control
|
167
|
-
e.try { render_button } # candidate
|
168
|
-
end
|
169
|
-
|
170
|
-
# Track an event on the experiment we've defined.
|
171
|
-
exp.track(:clicked_button)
|
172
|
-
```
|
173
|
-
|
174
|
-
</details>
|
175
|
-
|
176
152
|
<details>
|
177
153
|
<summary>You can also specify the variant to use for segmentation...</summary>
|
178
154
|
|
179
|
-
### Specifying variant
|
180
|
-
|
181
155
|
Generally, defining segmentation rules is a better way to approach routing into specific variants, but it's possible to explicitly specify the variant when running an experiment. It's important to know what this might do to your data during rollout, so use this with careful consideration.
|
182
156
|
|
157
|
+
Any time a specific variant is provided (including `:control`) it will be cached for that context, if caching is enabled.
|
158
|
+
|
183
159
|
```ruby
|
184
160
|
experiment(:notification_toggle, :no_interface, actor: user) do |e|
|
185
161
|
e.use { render_toggle } # control
|
@@ -360,6 +336,37 @@ experiment(:example, actor: user, project: project)
|
|
360
336
|
|
361
337
|
For edge cases, you can pass the cookie through by assigning it yourself -- e.g. `actor: request.cookie_jar.signed['example_actor']`. The cookie name is the full experiment name (including any configured prefix) with `_actor` appended -- e.g. `gitlab_notification_toggle_actor` for the `:notification_toggle` experiment key with a configured prefix of `gitlab`.
|
362
338
|
|
339
|
+
## How it works
|
340
|
+
|
341
|
+
The way the gem works is best described using the following decision tree illustration. When an experiment is run, the following logic is executed to resolve what experience should be provided, given how the experiment is defined, and the context provided.
|
342
|
+
|
343
|
+
```mermaid
|
344
|
+
graph TD
|
345
|
+
GP[General Pool/Population] --> Enabled?
|
346
|
+
Enabled? -->|Yes| Cached?[Cached? / Pre-segmented?]
|
347
|
+
Enabled? -->|No| Excluded[Control / No Tracking]
|
348
|
+
Cached? -->|No| Excluded?
|
349
|
+
Cached? -->|Yes| Cached[Cached Value]
|
350
|
+
Excluded? -->|Yes / Cached| Excluded
|
351
|
+
Excluded? -->|No| Segmented?
|
352
|
+
Segmented? -->|Yes / Cached| VariantA
|
353
|
+
Segmented? -->|No| Included?[Experiment Group?]
|
354
|
+
Included? -->|Yes| Rollout
|
355
|
+
Included? -->|No| Control
|
356
|
+
Rollout -->|Cached| VariantA
|
357
|
+
Rollout -->|Cached| VariantB
|
358
|
+
Rollout -->|Cached| VariantC
|
359
|
+
|
360
|
+
classDef included fill:#380d75,color:#ffffff,stroke:none
|
361
|
+
classDef excluded fill:#fca121,stroke:none
|
362
|
+
classDef cached fill:#2e2e2e,color:#ffffff,stroke:none
|
363
|
+
classDef default fill:#fff,stroke:#6e49cb
|
364
|
+
|
365
|
+
class VariantA,VariantB,VariantC included
|
366
|
+
class Control,Excluded excluded
|
367
|
+
class Cached cached
|
368
|
+
```
|
369
|
+
|
363
370
|
## Configuration
|
364
371
|
|
365
372
|
This gem needs to be configured before being used in a meaningful way.
|
@@ -534,7 +541,7 @@ https://gitlab.com/gitlab-org/gitlab-experiment. This project is intended to be
|
|
534
541
|
safe, welcoming space for collaboration, and contributors are expected to adhere
|
535
542
|
to the [Contributor Covenant](http://contributor-covenant.org) code of conduct.
|
536
543
|
|
537
|
-
## Release
|
544
|
+
## Release process
|
538
545
|
|
539
546
|
Please refer to the [Release Process](docs/release_process.md).
|
540
547
|
|
@@ -543,7 +550,7 @@ Please refer to the [Release Process](docs/release_process.md).
|
|
543
550
|
The gem is available as open source under the terms of the
|
544
551
|
[MIT License](http://opensource.org/licenses/MIT).
|
545
552
|
|
546
|
-
## Code of
|
553
|
+
## Code of conduct
|
547
554
|
|
548
555
|
Everyone interacting in the `Gitlab::Experiment` project’s codebases, issue trackers,
|
549
556
|
chat rooms and mailing lists is expected to follow the
|
@@ -22,20 +22,23 @@ Gitlab::Experiment.configure do |config|
|
|
22
22
|
# `['www.gitlab.com', '.gitlab.com']`.
|
23
23
|
config.cookie_domain = :all
|
24
24
|
|
25
|
-
#
|
25
|
+
# The default rollout strategy that works for single and multi-variants.
|
26
26
|
#
|
27
|
-
#
|
28
|
-
#
|
27
|
+
# You can provide your own rollout strategies and override them per
|
28
|
+
# experiment.
|
29
|
+
#
|
30
|
+
# Examples include:
|
31
|
+
# Rollout::First, Rollout::Random, Rollout::RoundRobin
|
32
|
+
config.default_rollout = Gitlab::Experiment::Rollout::First
|
33
|
+
|
34
|
+
# Logic this project uses to determine inclusion in a given experiment.
|
35
|
+
#
|
36
|
+
# Expected to return a boolean value.
|
29
37
|
#
|
30
38
|
# This block is executed within the scope of the experiment and so can access
|
31
39
|
# experiment methods, like `name`, `context`, and `signature`.
|
32
|
-
config.
|
33
|
-
|
34
|
-
requested_variant
|
35
|
-
|
36
|
-
# Run the candidate, unless a variant was requested, with a fallback:
|
37
|
-
#
|
38
|
-
# requested_variant || variant_names.first || nil
|
40
|
+
config.inclusion_resolver = lambda do |requested_variant|
|
41
|
+
false
|
39
42
|
end
|
40
43
|
|
41
44
|
# Tracking behavior can be implemented to link an event to an experiment.
|
data/lib/gitlab/experiment.rb
CHANGED
@@ -1,13 +1,18 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
2
|
|
3
3
|
require 'scientist'
|
4
|
+
require 'request_store'
|
4
5
|
require 'active_support/callbacks'
|
5
6
|
require 'active_support/cache'
|
7
|
+
require 'active_support/concern'
|
6
8
|
require 'active_support/core_ext/object/blank'
|
7
9
|
require 'active_support/core_ext/string/inflections'
|
10
|
+
require 'active_support/core_ext/module/delegation'
|
8
11
|
|
12
|
+
require 'gitlab/experiment/base_interface'
|
9
13
|
require 'gitlab/experiment/cache'
|
10
14
|
require 'gitlab/experiment/callbacks'
|
15
|
+
require 'gitlab/experiment/rollout'
|
11
16
|
require 'gitlab/experiment/configuration'
|
12
17
|
require 'gitlab/experiment/cookies'
|
13
18
|
require 'gitlab/experiment/context'
|
@@ -18,58 +23,48 @@ require 'gitlab/experiment/engine' if defined?(Rails::Engine)
|
|
18
23
|
|
19
24
|
module Gitlab
|
20
25
|
class Experiment
|
21
|
-
include
|
26
|
+
include BaseInterface
|
22
27
|
include Cache
|
23
28
|
include Callbacks
|
24
29
|
|
25
30
|
class << self
|
26
|
-
def
|
27
|
-
|
28
|
-
end
|
29
|
-
|
30
|
-
def run(name = nil, variant_name = nil, **context, &block)
|
31
|
-
raise ArgumentError, 'name is required' if name.nil? && base?
|
31
|
+
def default_rollout(rollout = nil, options = {})
|
32
|
+
return @rollout ||= Configuration.default_rollout if rollout.blank?
|
32
33
|
|
33
|
-
|
34
|
-
return instance unless block
|
35
|
-
|
36
|
-
instance.context.frozen? ? instance.run : instance.tap(&:run)
|
34
|
+
@rollout = Rollout.resolve(rollout).new(options)
|
37
35
|
end
|
38
36
|
|
39
|
-
def
|
40
|
-
|
41
|
-
|
42
|
-
|
37
|
+
def exclude(*filter_list, **options, &block)
|
38
|
+
build_callback(:exclusion_check, filter_list.unshift(block), **options) do |target, callback|
|
39
|
+
throw(:abort) if target.instance_variable_get(:@excluded) || callback.call(target, nil) == true
|
40
|
+
end
|
43
41
|
end
|
44
42
|
|
45
|
-
def
|
46
|
-
|
43
|
+
def segment(*filter_list, variant:, **options, &block)
|
44
|
+
build_callback(:segmentation_check, filter_list.unshift(block), **options) do |target, callback|
|
45
|
+
target.variant(variant) if target.instance_variable_get(:@variant_name).nil? && callback.call(target, nil)
|
46
|
+
end
|
47
47
|
end
|
48
48
|
|
49
|
-
|
50
|
-
|
51
|
-
def constantize(name = nil)
|
52
|
-
return self if name.nil?
|
53
|
-
|
54
|
-
experiment_name(name).classify.safe_constantize || Configuration.base_class.constantize
|
49
|
+
def published_experiments
|
50
|
+
RequestStore.store[:published_gitlab_experiments] || {}
|
55
51
|
end
|
56
52
|
end
|
57
53
|
|
58
|
-
def
|
59
|
-
|
60
|
-
|
61
|
-
@name = self.class.experiment_name(name, suffix: false)
|
62
|
-
@context = Context.new(self, **context)
|
63
|
-
@variant_name = cache_variant(variant_name) { nil } if variant_name.present?
|
64
|
-
|
65
|
-
compare { false }
|
54
|
+
def name
|
55
|
+
[Configuration.name_prefix, @name].compact.join('_')
|
56
|
+
end
|
66
57
|
|
67
|
-
|
58
|
+
def control(&block)
|
59
|
+
candidate(:control, &block)
|
68
60
|
end
|
61
|
+
alias_method :use, :control
|
69
62
|
|
70
|
-
def
|
71
|
-
|
63
|
+
def candidate(name = nil, &block)
|
64
|
+
name = (name || :candidate).to_s
|
65
|
+
behaviors[name] = block
|
72
66
|
end
|
67
|
+
alias_method :try, :candidate
|
73
68
|
|
74
69
|
def context(value = nil)
|
75
70
|
return @context if value.blank?
|
@@ -83,12 +78,8 @@ module Gitlab
|
|
83
78
|
return Variant.new(name: (@variant_name || :unresolved).to_s) if @variant_name || @resolving_variant
|
84
79
|
|
85
80
|
if enabled?
|
86
|
-
@variant_name ||= :control if excluded?
|
87
|
-
|
88
81
|
@resolving_variant = true
|
89
|
-
|
90
|
-
@variant_name = result.to_sym
|
91
|
-
end
|
82
|
+
@variant_name = cached_variant_resolver(@variant_name)
|
92
83
|
end
|
93
84
|
|
94
85
|
run_callbacks(segmentation_callback_chain) do
|
@@ -99,12 +90,24 @@ module Gitlab
|
|
99
90
|
@resolving_variant = false
|
100
91
|
end
|
101
92
|
|
93
|
+
def rollout(rollout = nil, options = {})
|
94
|
+
return @rollout ||= self.class.default_rollout(nil, options) if rollout.blank?
|
95
|
+
|
96
|
+
@rollout = Rollout.resolve(rollout).new(options)
|
97
|
+
end
|
98
|
+
|
99
|
+
def exclude!
|
100
|
+
@excluded = true
|
101
|
+
end
|
102
|
+
|
102
103
|
def run(variant_name = nil)
|
103
104
|
@result ||= super(variant(variant_name).name)
|
104
105
|
end
|
105
106
|
|
106
107
|
def publish(result)
|
107
108
|
instance_exec(result, &Configuration.publishing_behavior)
|
109
|
+
|
110
|
+
(RequestStore.store[:published_gitlab_experiments] ||= {})[name] = signature.merge(excluded: excluded?)
|
108
111
|
end
|
109
112
|
|
110
113
|
def track(action, **event_args)
|
@@ -113,41 +116,6 @@ module Gitlab
|
|
113
116
|
instance_exec(action, event_args, &Configuration.tracking_behavior)
|
114
117
|
end
|
115
118
|
|
116
|
-
def name
|
117
|
-
[Configuration.name_prefix, @name].compact.join('_')
|
118
|
-
end
|
119
|
-
|
120
|
-
def variant_names
|
121
|
-
@variant_names ||= behaviors.keys.map(&:to_sym) - [:control]
|
122
|
-
end
|
123
|
-
|
124
|
-
def behaviors
|
125
|
-
@behaviors ||= public_methods.each_with_object(super) do |name, behaviors|
|
126
|
-
next unless name.end_with?('_behavior')
|
127
|
-
|
128
|
-
behavior_name = name.to_s.sub(/_behavior$/, '')
|
129
|
-
behaviors[behavior_name] ||= -> { send(name) } # rubocop:disable GitlabSecurity/PublicSend
|
130
|
-
end
|
131
|
-
end
|
132
|
-
|
133
|
-
def try(name = nil, &block)
|
134
|
-
name = (name || 'candidate').to_s
|
135
|
-
behaviors[name] = block
|
136
|
-
end
|
137
|
-
|
138
|
-
def signature
|
139
|
-
{ variant: variant.name, experiment: name }.merge(context.signature)
|
140
|
-
end
|
141
|
-
|
142
|
-
def id
|
143
|
-
"#{name}:#{key_for(context.value)}"
|
144
|
-
end
|
145
|
-
alias_method :session_id, :id
|
146
|
-
|
147
|
-
def flipper_id
|
148
|
-
"Experiment;#{id}"
|
149
|
-
end
|
150
|
-
|
151
119
|
def enabled?
|
152
120
|
true
|
153
121
|
end
|
@@ -155,37 +123,35 @@ module Gitlab
|
|
155
123
|
def excluded?
|
156
124
|
return @excluded if defined?(@excluded)
|
157
125
|
|
158
|
-
@excluded =
|
159
|
-
|
126
|
+
@excluded = !run_callbacks(:exclusion_check) { :not_excluded }
|
127
|
+
end
|
128
|
+
|
129
|
+
def experiment_group?
|
130
|
+
instance_exec(@variant_name, &Configuration.inclusion_resolver)
|
160
131
|
end
|
161
132
|
|
162
133
|
def should_track?
|
163
|
-
enabled? && !excluded?
|
134
|
+
enabled? && @context.trackable? && !excluded?
|
164
135
|
end
|
165
136
|
|
166
|
-
def
|
167
|
-
|
137
|
+
def signature
|
138
|
+
{ variant: variant.name, experiment: name }.merge(context.signature)
|
139
|
+
end
|
140
|
+
|
141
|
+
def key_for(source, seed = name)
|
142
|
+
instance_exec(source, seed, &Configuration.context_hash_strategy)
|
168
143
|
end
|
169
144
|
|
170
145
|
protected
|
171
146
|
|
172
147
|
def segmentation_callback_chain
|
173
|
-
return :segmentation_check if
|
148
|
+
return :segmentation_check if @variant_name.nil? && enabled? && !excluded?
|
174
149
|
|
175
150
|
:unsegmented
|
176
151
|
end
|
177
152
|
|
178
|
-
def variant_assigned?
|
179
|
-
!@variant_name.nil?
|
180
|
-
end
|
181
|
-
|
182
153
|
def resolve_variant_name
|
183
|
-
|
184
|
-
end
|
185
|
-
|
186
|
-
def generate_result(variant_name)
|
187
|
-
observation = Scientist::Observation.new(variant_name, self, &behaviors[variant_name])
|
188
|
-
Scientist::Result.new(self, [observation], observation)
|
154
|
+
rollout.rollout_for(self) if experiment_group?
|
189
155
|
end
|
190
156
|
end
|
191
157
|
end
|
@@ -0,0 +1,88 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
module Gitlab
|
4
|
+
class Experiment
|
5
|
+
module BaseInterface
|
6
|
+
extend ActiveSupport::Concern
|
7
|
+
include Scientist::Experiment
|
8
|
+
|
9
|
+
class_methods do
|
10
|
+
def configure
|
11
|
+
yield Configuration
|
12
|
+
end
|
13
|
+
|
14
|
+
def experiment_name(name = nil, suffix: true, suffix_word: 'experiment')
|
15
|
+
name = (name.presence || self.name).to_s.underscore.sub(%r{(?<char>[_/]|)#{suffix_word}$}, '')
|
16
|
+
name = "#{name}#{Regexp.last_match(:char) || '_'}#{suffix_word}"
|
17
|
+
suffix ? name : name.sub(/_#{suffix_word}$/, '')
|
18
|
+
end
|
19
|
+
|
20
|
+
def base?
|
21
|
+
self == Gitlab::Experiment || name == Configuration.base_class
|
22
|
+
end
|
23
|
+
|
24
|
+
def constantize(name = nil)
|
25
|
+
return self if name.nil?
|
26
|
+
|
27
|
+
experiment_name(name).classify.safe_constantize || Configuration.base_class.constantize
|
28
|
+
end
|
29
|
+
end
|
30
|
+
|
31
|
+
def initialize(name = nil, variant_name = nil, **context)
|
32
|
+
raise ArgumentError, 'name is required' if name.blank? && self.class.base?
|
33
|
+
|
34
|
+
@name = self.class.experiment_name(name, suffix: false)
|
35
|
+
@context = Context.new(self, **context)
|
36
|
+
@variant_name = cache_variant(variant_name) { nil } if variant_name.present?
|
37
|
+
|
38
|
+
compare { false }
|
39
|
+
|
40
|
+
yield self if block_given?
|
41
|
+
end
|
42
|
+
|
43
|
+
def inspect
|
44
|
+
"#<#{self.class.name || 'AnonymousClass'}:#{format('0x%016X', __id__)} @name=#{name} @context=#{context.value}>"
|
45
|
+
end
|
46
|
+
|
47
|
+
def id
|
48
|
+
"#{name}:#{key_for(context.value)}"
|
49
|
+
end
|
50
|
+
alias_method :session_id, :id
|
51
|
+
|
52
|
+
def flipper_id
|
53
|
+
"Experiment;#{id}"
|
54
|
+
end
|
55
|
+
|
56
|
+
def variant_names
|
57
|
+
@variant_names ||= behaviors.keys.map(&:to_sym) - [:control]
|
58
|
+
end
|
59
|
+
|
60
|
+
def behaviors
|
61
|
+
@behaviors ||= public_methods.each_with_object(super) do |name, behaviors|
|
62
|
+
next unless name.end_with?('_behavior')
|
63
|
+
|
64
|
+
behavior_name = name.to_s.sub(/_behavior$/, '')
|
65
|
+
behaviors[behavior_name] ||= -> { send(name) } # rubocop:disable GitlabSecurity/PublicSend
|
66
|
+
end
|
67
|
+
end
|
68
|
+
|
69
|
+
protected
|
70
|
+
|
71
|
+
def raise_on_mismatches?
|
72
|
+
false
|
73
|
+
end
|
74
|
+
|
75
|
+
def cached_variant_resolver(provided_variant)
|
76
|
+
return :control if excluded?
|
77
|
+
|
78
|
+
result = cache_variant(provided_variant) { resolve_variant_name }
|
79
|
+
result.to_sym if result.present?
|
80
|
+
end
|
81
|
+
|
82
|
+
def generate_result(variant_name)
|
83
|
+
observation = Scientist::Observation.new(variant_name, self, &behaviors[variant_name])
|
84
|
+
Scientist::Result.new(self, [observation], observation)
|
85
|
+
end
|
86
|
+
end
|
87
|
+
end
|
88
|
+
end
|
@@ -1,5 +1,7 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
2
|
|
3
|
+
require 'active_support/callbacks'
|
4
|
+
|
3
5
|
module Gitlab
|
4
6
|
class Experiment
|
5
7
|
module Callbacks
|
@@ -13,28 +15,17 @@ module Gitlab
|
|
13
15
|
end
|
14
16
|
|
15
17
|
class_methods do
|
16
|
-
|
17
|
-
filters = filter_list.unshift(block).compact.map do |filter|
|
18
|
-
result_lambda = ActiveSupport::Callbacks::CallTemplate.build(filter, self).make_lambda
|
19
|
-
lambda do |target|
|
20
|
-
throw(:abort) if target.instance_variable_get(:'@excluded') || result_lambda.call(target, nil) == true
|
21
|
-
end
|
22
|
-
end
|
23
|
-
|
24
|
-
raise ArgumentError, 'no filters provided' if filters.empty?
|
25
|
-
|
26
|
-
set_callback(:exclusion_check, :before, *filters, options)
|
27
|
-
end
|
18
|
+
private
|
28
19
|
|
29
|
-
def
|
30
|
-
filters =
|
20
|
+
def build_callback(chain, filters, **options)
|
21
|
+
filters = filters.compact.map do |filter|
|
31
22
|
result_lambda = ActiveSupport::Callbacks::CallTemplate.build(filter, self).make_lambda
|
32
|
-
->(target) {
|
23
|
+
->(target) { yield(target, result_lambda) }
|
33
24
|
end
|
34
25
|
|
35
26
|
raise ArgumentError, 'no filters provided' if filters.empty?
|
36
27
|
|
37
|
-
set_callback(
|
28
|
+
set_callback(chain, *filters, **options)
|
38
29
|
end
|
39
30
|
end
|
40
31
|
end
|
@@ -4,6 +4,8 @@ require 'singleton'
|
|
4
4
|
require 'logger'
|
5
5
|
require 'digest'
|
6
6
|
|
7
|
+
require 'active_support/deprecation'
|
8
|
+
|
7
9
|
module Gitlab
|
8
10
|
class Experiment
|
9
11
|
class Configuration
|
@@ -24,10 +26,15 @@ module Gitlab
|
|
24
26
|
# The domain to use on cookies.
|
25
27
|
@cookie_domain = :all
|
26
28
|
|
27
|
-
#
|
28
|
-
#
|
29
|
-
|
30
|
-
|
29
|
+
# The default rollout strategy only works for single variant experiments.
|
30
|
+
# It's expected that you use a more advanced rollout for multiple variant
|
31
|
+
# experiments.
|
32
|
+
@default_rollout = Rollout::Base.new
|
33
|
+
|
34
|
+
# Logic this project uses to determine inclusion in a given experiment.
|
35
|
+
# Expected to return a boolean value.
|
36
|
+
@inclusion_resolver = lambda do |requested_variant|
|
37
|
+
false
|
31
38
|
end
|
32
39
|
|
33
40
|
# Tracking behavior can be implemented to link an event to an experiment.
|
@@ -40,20 +47,35 @@ module Gitlab
|
|
40
47
|
track(:assignment)
|
41
48
|
end
|
42
49
|
|
43
|
-
# Algorithm that consistently generates a hash key for a given
|
44
|
-
@context_hash_strategy = lambda do |
|
45
|
-
|
46
|
-
|
50
|
+
# Algorithm that consistently generates a hash key for a given source.
|
51
|
+
@context_hash_strategy = lambda do |source, seed|
|
52
|
+
source = source.keys + source.values if source.is_a?(Hash)
|
53
|
+
data = Array(source).map { |v| (v.respond_to?(:to_global_id) ? v.to_global_id : v).to_s }
|
54
|
+
Digest::MD5.hexdigest(data.unshift(seed).join('|'))
|
47
55
|
end
|
48
56
|
|
49
57
|
class << self
|
58
|
+
# TODO: Added deprecation in release 0.5.0
|
59
|
+
def variant_resolver
|
60
|
+
ActiveSupport::Deprecation.warn('variant_resolver is deprecated, instead use `inclusion_resolver` with a' \
|
61
|
+
'block that returns a boolean.')
|
62
|
+
@inclusion_resolver
|
63
|
+
end
|
64
|
+
|
65
|
+
def variant_resolver=(block)
|
66
|
+
ActiveSupport::Deprecation.warn('variant_resolver is deprecated, instead use `inclusion_resolver` with a' \
|
67
|
+
'block that returns a boolean.')
|
68
|
+
@inclusion_resolver = block
|
69
|
+
end
|
70
|
+
|
50
71
|
attr_accessor(
|
51
72
|
:name_prefix,
|
52
73
|
:logger,
|
53
74
|
:base_class,
|
54
75
|
:cache,
|
55
76
|
:cookie_domain,
|
56
|
-
:
|
77
|
+
:default_rollout,
|
78
|
+
:inclusion_resolver,
|
57
79
|
:tracking_behavior,
|
58
80
|
:publishing_behavior,
|
59
81
|
:context_hash_strategy
|
@@ -4,8 +4,17 @@ module Gitlab
|
|
4
4
|
class Experiment
|
5
5
|
module Dsl
|
6
6
|
def experiment(name, variant_name = nil, **context, &block)
|
7
|
+
raise ArgumentError, 'name is required' if name.nil?
|
8
|
+
|
7
9
|
context[:request] ||= request if respond_to?(:request)
|
8
|
-
|
10
|
+
|
11
|
+
base = Configuration.base_class.constantize
|
12
|
+
klass = base.constantize(name) || base
|
13
|
+
|
14
|
+
instance = klass.new(name, variant_name, **context, &block)
|
15
|
+
return instance unless block
|
16
|
+
|
17
|
+
instance.context.frozen? ? instance.run : instance.tap(&:run)
|
9
18
|
end
|
10
19
|
end
|
11
20
|
end
|
@@ -0,0 +1,35 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
module Gitlab
|
4
|
+
class Experiment
|
5
|
+
module Rollout
|
6
|
+
autoload :Random, 'gitlab/experiment/rollout/random.rb'
|
7
|
+
autoload :RoundRobin, 'gitlab/experiment/rollout/round_robin.rb'
|
8
|
+
|
9
|
+
def self.resolve(klass)
|
10
|
+
return "#{name}::#{klass.to_s.classify}".constantize if klass.is_a?(Symbol) || klass.is_a?(String)
|
11
|
+
|
12
|
+
klass
|
13
|
+
end
|
14
|
+
|
15
|
+
class Base
|
16
|
+
attr_reader :experiment
|
17
|
+
|
18
|
+
delegate :variant_names, :cache, to: :experiment
|
19
|
+
|
20
|
+
def initialize(options = {})
|
21
|
+
@options = options
|
22
|
+
end
|
23
|
+
|
24
|
+
def rollout_for(experiment)
|
25
|
+
@experiment = experiment
|
26
|
+
execute
|
27
|
+
end
|
28
|
+
|
29
|
+
def execute
|
30
|
+
variant_names.first
|
31
|
+
end
|
32
|
+
end
|
33
|
+
end
|
34
|
+
end
|
35
|
+
end
|
@@ -0,0 +1,15 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
module Gitlab
|
4
|
+
class Experiment
|
5
|
+
module Rollout
|
6
|
+
class Random < Base
|
7
|
+
# Pick a random variant if we're in the experiment group. It doesn't
|
8
|
+
# take into account small sample sizes but is useful and performant.
|
9
|
+
def execute
|
10
|
+
variant_names.sample
|
11
|
+
end
|
12
|
+
end
|
13
|
+
end
|
14
|
+
end
|
15
|
+
end
|
@@ -0,0 +1,23 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
module Gitlab
|
4
|
+
class Experiment
|
5
|
+
module Rollout
|
6
|
+
class RoundRobin < Base
|
7
|
+
KEY_NAME = :last_round_robin_variant
|
8
|
+
|
9
|
+
# Requires a cache to be configured.
|
10
|
+
#
|
11
|
+
# Keeps track of the number of assignments into the experiment group,
|
12
|
+
# and uses this to rotate "round robin" style through the variants
|
13
|
+
# that are defined.
|
14
|
+
#
|
15
|
+
# Relatively performant, but requires a cache, and is dependent on the
|
16
|
+
# performance of that cache store.
|
17
|
+
def execute
|
18
|
+
variant_names[(cache.attr_inc(KEY_NAME) - 1) % variant_names.size]
|
19
|
+
end
|
20
|
+
end
|
21
|
+
end
|
22
|
+
end
|
23
|
+
end
|
@@ -3,39 +3,78 @@
|
|
3
3
|
module Gitlab
|
4
4
|
class Experiment
|
5
5
|
module RSpecHelpers
|
6
|
-
def stub_experiments(experiments)
|
7
|
-
experiments.each
|
8
|
-
|
9
|
-
raise ArgumentError, 'variant must be a symbol or false' unless variant.is_a?(Symbol)
|
6
|
+
def stub_experiments(experiments, times = nil)
|
7
|
+
experiments.each { |experiment| wrapped_experiment(experiment, times) }
|
8
|
+
end
|
10
9
|
|
11
|
-
|
10
|
+
def wrapped_experiment(experiment, times = nil, expected = false, &block)
|
11
|
+
klass, experiment_name, variant_name = *experiment_details(experiment)
|
12
|
+
base_klass = Configuration.base_class.constantize
|
12
13
|
|
13
|
-
|
14
|
-
|
15
|
-
|
16
|
-
|
14
|
+
# Set expectations on experiment classes so we can and_wrap_original with more specific args
|
15
|
+
experiment_klasses = base_klass.descendants.reject { |k| k == klass }
|
16
|
+
experiment_klasses.push(base_klass).each do |k|
|
17
|
+
allow(k).to receive(:new).and_call_original
|
17
18
|
end
|
18
|
-
end
|
19
19
|
|
20
|
-
|
21
|
-
|
22
|
-
|
23
|
-
|
20
|
+
receiver = receive(:new)
|
21
|
+
|
22
|
+
# Be specific for BaseClass calls
|
23
|
+
receiver = receiver.with(experiment_name, any_args) if experiment_name && klass == base_klass
|
24
|
+
|
25
|
+
receiver.exactly(times).times if times
|
26
|
+
|
27
|
+
# Set expectations on experiment class of interest
|
28
|
+
allow_or_expect_klass = expected ? expect(klass) : allow(klass)
|
29
|
+
allow_or_expect_klass.to receiver.and_wrap_original do |method, *original_args, &original_block|
|
30
|
+
method.call(*original_args).tap do |e|
|
31
|
+
# Stub internal methods before calling the original_block
|
32
|
+
allow(e).to receive(:enabled?).and_return(true)
|
33
|
+
|
34
|
+
if variant_name == true # passing true allows the rollout to do its job
|
35
|
+
allow(e).to receive(:experiment_group?).and_return(true)
|
36
|
+
else
|
37
|
+
allow(e).to receive(:resolve_variant_name).and_return(variant_name.to_s)
|
38
|
+
end
|
39
|
+
|
40
|
+
# Stub/set expectations before calling the original_block
|
41
|
+
yield e if block
|
42
|
+
|
43
|
+
original_block.call(e) if original_block.present?
|
44
|
+
end
|
24
45
|
end
|
46
|
+
end
|
25
47
|
|
26
|
-
|
27
|
-
|
28
|
-
|
29
|
-
|
30
|
-
|
48
|
+
private
|
49
|
+
|
50
|
+
def experiment_details(experiment)
|
51
|
+
if experiment.is_a?(Symbol)
|
52
|
+
experiment_name = experiment
|
53
|
+
variant_name = nil
|
31
54
|
end
|
32
55
|
|
33
|
-
|
34
|
-
|
35
|
-
|
56
|
+
experiment_name, variant_name = *experiment if experiment.is_a?(Array)
|
57
|
+
|
58
|
+
base_klass = Configuration.base_class.constantize
|
59
|
+
variant_name = experiment.variant.name if experiment.is_a?(base_klass)
|
60
|
+
|
61
|
+
if experiment.class.name.nil? # Anonymous class instance
|
62
|
+
klass = experiment.class
|
63
|
+
elsif experiment.instance_of?(Class) # Class level stubbing, eg. "MyExperiment"
|
64
|
+
klass = experiment
|
36
65
|
else
|
37
|
-
|
66
|
+
experiment_name ||= experiment.instance_variable_get(:@name)
|
67
|
+
klass = base_klass.constantize(experiment_name)
|
38
68
|
end
|
69
|
+
|
70
|
+
if experiment_name && klass == base_klass
|
71
|
+
experiment_name = experiment_name.to_sym
|
72
|
+
|
73
|
+
# For experiment names like: "group/experiment-name"
|
74
|
+
experiment_name = experiment_name.to_s if experiment_name.inspect.include?('"')
|
75
|
+
end
|
76
|
+
|
77
|
+
[klass, experiment_name, variant_name]
|
39
78
|
end
|
40
79
|
end
|
41
80
|
|
@@ -130,33 +169,42 @@ module Gitlab
|
|
130
169
|
end
|
131
170
|
|
132
171
|
chain(:with_context) { |expected_context| @expected_context = expected_context }
|
133
|
-
|
172
|
+
|
173
|
+
chain(:on_next_instance) { @on_next_instance = true }
|
134
174
|
|
135
175
|
def expect_tracking_on(experiment, negated, event, *event_args)
|
136
|
-
|
137
|
-
|
138
|
-
|
139
|
-
|
140
|
-
|
176
|
+
klass = experiment.instance_of?(Class) ? experiment : experiment.class
|
177
|
+
unless klass <= Gitlab::Experiment
|
178
|
+
raise(
|
179
|
+
ArgumentError,
|
180
|
+
"track matcher is limited to experiment instances and classes"
|
181
|
+
)
|
182
|
+
end
|
183
|
+
|
184
|
+
expectations = proc do |e|
|
185
|
+
@experiment = e
|
186
|
+
allow(e).to receive(:track).and_call_original
|
141
187
|
|
142
188
|
if negated
|
143
|
-
expect(
|
189
|
+
expect(e).not_to receive(:track).with(*[event, *event_args])
|
144
190
|
else
|
145
|
-
|
146
|
-
|
147
|
-
|
148
|
-
end
|
191
|
+
if @expected_variant
|
192
|
+
expect(@experiment.variant.name).to eq(@expected_variant), failure_message(:variant, event)
|
193
|
+
end
|
149
194
|
|
150
|
-
|
151
|
-
|
152
|
-
|
153
|
-
expect(@experiment.variant.name).to eq(@expected_variant), failure_message(:variant, event)
|
154
|
-
end
|
195
|
+
if @expected_context
|
196
|
+
expect(@experiment.context.value).to include(@expected_context), failure_message(:context, event)
|
197
|
+
end
|
155
198
|
|
156
|
-
|
157
|
-
expect(@experiment.context.value).to include(@expected_context), failure_message(:context, event)
|
199
|
+
expect(e).to receive(:track).with(*[event, *event_args]).and_call_original
|
158
200
|
end
|
159
201
|
end
|
202
|
+
|
203
|
+
if experiment.instance_of?(Class) || @on_next_instance
|
204
|
+
wrapped_experiment(experiment, nil, true) { |e| expectations.call(e) }
|
205
|
+
else
|
206
|
+
expectations.call(experiment)
|
207
|
+
end
|
160
208
|
end
|
161
209
|
|
162
210
|
def failure_message(failure_type, event)
|
@@ -173,8 +221,6 @@ module Gitlab
|
|
173
221
|
expected context: #{@expected_context}
|
174
222
|
actual context: #{@experiment.context.value}
|
175
223
|
MESSAGE
|
176
|
-
when :no_new
|
177
|
-
%(expected #{@experiment.inspect} to have tracked #{event.inspect}, but no new instances were created)
|
178
224
|
end
|
179
225
|
end
|
180
226
|
end
|
@@ -186,6 +232,10 @@ RSpec.configure do |config|
|
|
186
232
|
config.include Gitlab::Experiment::RSpecHelpers
|
187
233
|
config.include Gitlab::Experiment::Dsl
|
188
234
|
|
235
|
+
config.before(:each, :experiment) do
|
236
|
+
RequestStore.clear!
|
237
|
+
end
|
238
|
+
|
189
239
|
config.include Gitlab::Experiment::RSpecMatchers, :experiment
|
190
240
|
config.define_derived_metadata(file_path: Regexp.new('/spec/experiments/')) do |metadata|
|
191
241
|
metadata[:type] = :experiment
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: gitlab-experiment
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.4
|
4
|
+
version: 0.5.4
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- GitLab
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2021-
|
11
|
+
date: 2021-05-05 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: activesupport
|
@@ -24,26 +24,40 @@ dependencies:
|
|
24
24
|
- - ">="
|
25
25
|
- !ruby/object:Gem::Version
|
26
26
|
version: '3.0'
|
27
|
+
- !ruby/object:Gem::Dependency
|
28
|
+
name: request_store
|
29
|
+
requirement: !ruby/object:Gem::Requirement
|
30
|
+
requirements:
|
31
|
+
- - ">="
|
32
|
+
- !ruby/object:Gem::Version
|
33
|
+
version: '1.0'
|
34
|
+
type: :runtime
|
35
|
+
prerelease: false
|
36
|
+
version_requirements: !ruby/object:Gem::Requirement
|
37
|
+
requirements:
|
38
|
+
- - ">="
|
39
|
+
- !ruby/object:Gem::Version
|
40
|
+
version: '1.0'
|
27
41
|
- !ruby/object:Gem::Dependency
|
28
42
|
name: scientist
|
29
43
|
requirement: !ruby/object:Gem::Requirement
|
30
44
|
requirements:
|
31
45
|
- - "~>"
|
32
46
|
- !ruby/object:Gem::Version
|
33
|
-
version: '1.
|
47
|
+
version: '1.6'
|
34
48
|
- - ">="
|
35
49
|
- !ruby/object:Gem::Version
|
36
|
-
version: 1.
|
50
|
+
version: 1.6.0
|
37
51
|
type: :runtime
|
38
52
|
prerelease: false
|
39
53
|
version_requirements: !ruby/object:Gem::Requirement
|
40
54
|
requirements:
|
41
55
|
- - "~>"
|
42
56
|
- !ruby/object:Gem::Version
|
43
|
-
version: '1.
|
57
|
+
version: '1.6'
|
44
58
|
- - ">="
|
45
59
|
- !ruby/object:Gem::Version
|
46
|
-
version: 1.
|
60
|
+
version: 1.6.0
|
47
61
|
description:
|
48
62
|
email:
|
49
63
|
- gitlab_rubygems@gitlab.com
|
@@ -65,6 +79,7 @@ files:
|
|
65
79
|
- lib/generators/test_unit/experiment/experiment_generator.rb
|
66
80
|
- lib/generators/test_unit/experiment/templates/experiment_test.rb.tt
|
67
81
|
- lib/gitlab/experiment.rb
|
82
|
+
- lib/gitlab/experiment/base_interface.rb
|
68
83
|
- lib/gitlab/experiment/cache.rb
|
69
84
|
- lib/gitlab/experiment/cache/redis_hash_store.rb
|
70
85
|
- lib/gitlab/experiment/callbacks.rb
|
@@ -73,6 +88,9 @@ files:
|
|
73
88
|
- lib/gitlab/experiment/cookies.rb
|
74
89
|
- lib/gitlab/experiment/dsl.rb
|
75
90
|
- lib/gitlab/experiment/engine.rb
|
91
|
+
- lib/gitlab/experiment/rollout.rb
|
92
|
+
- lib/gitlab/experiment/rollout/random.rb
|
93
|
+
- lib/gitlab/experiment/rollout/round_robin.rb
|
76
94
|
- lib/gitlab/experiment/rspec.rb
|
77
95
|
- lib/gitlab/experiment/variant.rb
|
78
96
|
- lib/gitlab/experiment/version.rb
|
@@ -95,7 +113,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
95
113
|
- !ruby/object:Gem::Version
|
96
114
|
version: '0'
|
97
115
|
requirements: []
|
98
|
-
rubygems_version: 3.
|
116
|
+
rubygems_version: 3.2.17
|
99
117
|
signing_key:
|
100
118
|
specification_version: 4
|
101
119
|
summary: GitLab experiment library built on top of scientist.
|