gitlab-experiment 0.4.4 → 0.4.9

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 5b1a2f81e8680cffda06d116655200c055dbb74a42a52a7fa8adcb84c720fb01
4
- data.tar.gz: 715ea6dd8494457c169ef3147ca2edf2fc6e4c63571801e1f5b22814182ebb56
3
+ metadata.gz: c838adc97d97a29963030a4a74617fc48d6facf937e41add14b318322ad9e80d
4
+ data.tar.gz: ea8cda6c8d4848b7be8f4437c3d2c5b605414bd9e053fe768175e248e85c6f48
5
5
  SHA512:
6
- metadata.gz: 7eec246157542ff0897f9af9f3135272d4d7fea31049443519fb550d5ca9cb3ad2d073ae8594104256d353b8042132f7ae5cba399bb44c38d7d5332154d5b849
7
- data.tar.gz: ff4145fbe06fad2de4ef6295f4a0715c71c9ef4165308452d9dacd0c02a0df94d2152d138a6945c583c68d6c5966e1bb1e200fd79a7d9668382ee7c1d68861ea
6
+ metadata.gz: 310ce3f0f80041fa52866082985b744b6853ade992e36bb95dd7e9c1ebf433c91cd949548a362336caa15d96854c9c6050b8462b7b907583ff2f316482896d9b
7
+ data.tar.gz: e2e47f252f92c74dfbed63b3e92aceccb3cdc8522ef1d64e0e6fda9050b4d955846c20b8c49af9dc288fbc6fe80090a6d36ee1b1d5cabcdd88a17ac01c226ece
data/README.md CHANGED
@@ -15,7 +15,7 @@ When we discuss the behavior of this gem, we'll use terms like experiment, conte
15
15
  - `candidate` defines that there's one experimental code path.
16
16
  - `variant(s)` is used when more than one experimental code path exists.
17
17
 
18
- Candidate and variant are the same concept, but simplify how we speak about experimental paths.<br clear="all">
18
+ Candidate and variant are the same concept, but simplify how we speak about experimental paths -- this concept is also widely referred to as the "experiment group".<br clear="all">
19
19
 
20
20
  [[_TOC_]]
21
21
 
@@ -100,6 +100,9 @@ Here are some examples of what you can introduce once you have a custom experime
100
100
 
101
101
  ```ruby
102
102
  class NotificationToggleExperiment < ApplicationExperiment
103
+ # Exclude any users that aren't me.
104
+ exclude :all_but_me
105
+
103
106
  # Segment any account less than 2 weeks old into the candidate, without
104
107
  # asking the variant resolver to decide which variant to provide.
105
108
  segment :account_age, variant: :candidate
@@ -107,17 +110,21 @@ class NotificationToggleExperiment < ApplicationExperiment
107
110
  # Define the default control behavior, which can be overridden at
108
111
  # experiment time.
109
112
  def control_behavior
110
- render_toggle
113
+ # render_toggle
111
114
  end
112
115
 
113
116
  # Define the default candidate behavior, which can be overridden
114
117
  # at experiment time.
115
118
  def candidate_behavior
116
- render_button
119
+ # render_button
117
120
  end
118
121
 
119
122
  private
120
123
 
124
+ def all_but_me
125
+ context.actor&.username == 'jejacks0n'
126
+ end
127
+
121
128
  def account_age
122
129
  context.actor && context.actor.created_at < 2.weeks.ago
123
130
  end
@@ -177,7 +184,7 @@ Generally, defining segmentation rules is a better way to approach routing into
177
184
  experiment(:notification_toggle, :no_interface, actor: user) do |e|
178
185
  e.use { render_toggle } # control
179
186
  e.try { render_button } # candidate
180
- e.try(:no_interface) { no_interface! } # variant
187
+ e.try(:no_interface) { no_interface! } # no_interface variant
181
188
  end
182
189
  ```
183
190
 
@@ -222,6 +229,27 @@ When an experiment is run, the segmentation rules are executed in the order they
222
229
 
223
230
  This means that any user with the name `'jejacks0n'`, regardless of account age, will always be provided the experience as defined in "variant_one".
224
231
 
232
+ ### Exclusion rules
233
+
234
+ Exclusion rules are similar to segmentation rules, but are intended to determine if a context should even be considered as something we should track events towards. Exclusion means we don't care about the events in relation to the given context.
235
+
236
+ ```ruby
237
+ class NotificationToggleExperiment < ApplicationExperiment
238
+ exclude { context.actor.username != 'jejacks0n' }
239
+ exclude { context.actor.created_at < 2.weeks.ago }
240
+ end
241
+ ```
242
+
243
+ The previous examples will force all users with a username not matching `'jejacks0n'` and all newish users into the control. No events would be tracked for those.
244
+
245
+ You may need to check exclusion in custom tracking logic, by checking `excluded?` or wrapping your code in a callback block:
246
+
247
+ ```ruby
248
+ run_callbacks(:exclusion_check) do
249
+ track(:my_event, value: expensive_method_call)
250
+ end
251
+ ```
252
+
225
253
  ### Return value
226
254
 
227
255
  By default the return value is a `Gitlab::Experiment` instance. In simple cases you may want only the results of the experiment though. You can call `run` within the block to get the return value of the assigned variant.
@@ -341,7 +369,7 @@ Gitlab::Experiment.configure do |config|
341
369
  end
342
370
  ```
343
371
 
344
- More examples for configuration are available in the provided [rails initializer](lib/generators/gitlab/experiment/install/templates/initializer.rb).
372
+ More examples for configuration are available in the provided [rails initializer](lib/generators/gitlab/experiment/install/templates/initializer.rb.tt).
345
373
 
346
374
  ### Client layer / JavaScript
347
375
 
@@ -30,7 +30,7 @@ module Gitlab
30
30
  raise ArgumentError, 'name is required' if name.nil? && base?
31
31
 
32
32
  instance = constantize(name).new(name, variant_name, **context, &block)
33
- return instance unless block_given?
33
+ return instance unless block
34
34
 
35
35
  instance.context.frozen? ? instance.run : instance.tap(&:run)
36
36
  end
@@ -58,16 +58,18 @@ module Gitlab
58
58
  raise ArgumentError, 'name is required' if name.blank? && self.class.base?
59
59
 
60
60
  @name = self.class.experiment_name(name, suffix: false)
61
- @variant_name = variant_name
62
- @excluded = []
63
61
  @context = Context.new(self, **context)
62
+ @variant_name = cache_variant(variant_name) { nil } if variant_name.present?
64
63
 
65
- exclude { !@context.trackable? }
66
64
  compare { false }
67
65
 
68
66
  yield self if block_given?
69
67
  end
70
68
 
69
+ def inspect
70
+ "#<#{self.class.name || 'AnonymousClass'}:#{format('0x%016X', __id__)} @name=#{name} @signature=#{signature}>"
71
+ end
72
+
71
73
  def context(value = nil)
72
74
  return @context if value.blank?
73
75
 
@@ -81,32 +83,26 @@ module Gitlab
81
83
  end
82
84
 
83
85
  @variant_name = value unless value.blank?
84
- @variant_name ||= :control if excluded?
85
86
 
86
- @resolving_variant = true
87
- resolved = :control
88
- if (result = cache_variant(@variant_name) { resolve_variant_name }).present?
89
- @variant_name = resolved = result.to_sym
87
+ if enabled?
88
+ @variant_name ||= :control if excluded?
89
+
90
+ @resolving_variant = true
91
+ if (result = cache_variant(@variant_name) { resolve_variant_name }).present?
92
+ @variant_name = result.to_sym
93
+ end
90
94
  end
91
95
 
92
- Variant.new(name: resolved.to_s)
96
+ Variant.new(name: (@variant_name || :control).to_s)
93
97
  ensure
94
98
  @resolving_variant = false
95
99
  end
96
100
 
97
- def exclude(&block)
98
- @excluded << block
99
- end
100
-
101
101
  def run(variant_name = nil)
102
102
  @result ||= begin
103
103
  variant_name = variant(variant_name).name
104
- run_callbacks(variant_assigned? ? :unsegmented_run : :segmented_run) do
105
- if respond_to?((behavior_name = "#{variant_name}_behavior"))
106
- behaviors[variant_name] ||= -> { send(behavior_name) } # rubocop:disable GitlabSecurity/PublicSend
107
- end
108
-
109
- super(@variant_name = variant_name)
104
+ run_callbacks(run_with_segmenting? ? :segmented_run : :unsegmented_run) do
105
+ super(@variant_name ||= variant_name)
110
106
  end
111
107
  end
112
108
  end
@@ -116,7 +112,7 @@ module Gitlab
116
112
  end
117
113
 
118
114
  def track(action, **event_args)
119
- return if excluded?
115
+ return unless should_track?
120
116
 
121
117
  instance_exec(action, event_args, &Configuration.tracking_behavior)
122
118
  end
@@ -129,20 +125,22 @@ module Gitlab
129
125
  @variant_names ||= behaviors.keys.map(&:to_sym) - [:control]
130
126
  end
131
127
 
132
- def signature
133
- { variant: variant.name, experiment: name }.merge(context.signature)
134
- end
128
+ def behaviors
129
+ @behaviors ||= public_methods.each_with_object(super) do |name, behaviors|
130
+ next unless name.end_with?('_behavior')
135
131
 
136
- def enabled?
137
- true
132
+ behavior_name = name.to_s.sub(/_behavior$/, '')
133
+ behaviors[behavior_name] ||= -> { send(name) } # rubocop:disable GitlabSecurity/PublicSend
134
+ end
138
135
  end
139
136
 
140
- def excluded?
141
- @excluded.any? { |exclude| exclude.call(self) }
137
+ def try(name = nil, &block)
138
+ name = (name || 'candidate').to_s
139
+ behaviors[name] = block
142
140
  end
143
141
 
144
- def variant_assigned?
145
- !@variant_name.nil?
142
+ def signature
143
+ { variant: variant.name, experiment: name }.merge(context.signature)
146
144
  end
147
145
 
148
146
  def id
@@ -154,12 +152,33 @@ module Gitlab
154
152
  "Experiment;#{id}"
155
153
  end
156
154
 
155
+ def enabled?
156
+ true
157
+ end
158
+
159
+ def excluded?
160
+ @excluded ||= !@context.trackable? || # adhere to DNT headers
161
+ !run_callbacks(:exclusion_check) { :not_excluded } # didn't pass exclusion check
162
+ end
163
+
164
+ def should_track?
165
+ enabled? && !excluded?
166
+ end
167
+
157
168
  def key_for(hash)
158
169
  instance_exec(hash, &Configuration.context_hash_strategy)
159
170
  end
160
171
 
161
172
  protected
162
173
 
174
+ def run_with_segmenting?
175
+ !variant_assigned? && enabled? && !excluded?
176
+ end
177
+
178
+ def variant_assigned?
179
+ !@variant_name.nil?
180
+ end
181
+
163
182
  def resolve_variant_name
164
183
  instance_exec(@variant_name, &Configuration.variant_resolver)
165
184
  end
@@ -7,31 +7,34 @@ module Gitlab
7
7
  include ActiveSupport::Callbacks
8
8
 
9
9
  included do
10
- define_callbacks(
11
- :unsegmented_run,
12
- skip_after_callbacks_if_terminated: true
13
- )
14
-
15
- define_callbacks(
16
- :segmented_run,
17
- skip_after_callbacks_if_terminated: false,
18
- terminator: lambda do |target, result_lambda|
19
- result_lambda.call
20
- target.variant_assigned?
21
- end
22
- )
10
+ define_callbacks(:unsegmented_run)
11
+ define_callbacks(:segmented_run)
12
+ define_callbacks(:exclusion_check, skip_after_callbacks_if_terminated: true)
23
13
  end
24
14
 
25
15
  class_methods do
16
+ def exclude(*filter_list, **options, &block)
17
+ filters = filter_list.unshift(block).compact.map do |filter|
18
+ result_lambda = ActiveSupport::Callbacks::CallTemplate.build(filter, self).make_lambda
19
+ lambda do |target|
20
+ throw(:abort) if target.instance_variable_get(:'@excluded') || result_lambda.call(target, nil) == true
21
+ end
22
+ end
23
+
24
+ raise ArgumentError, 'no filters provided' if filters.empty?
25
+
26
+ set_callback(:exclusion_check, :before, *filters, options)
27
+ end
28
+
26
29
  def segment(*filter_list, variant:, **options, &block)
27
30
  filters = filter_list.unshift(block).compact.map do |filter|
28
31
  result_lambda = ActiveSupport::Callbacks::CallTemplate.build(filter, self).make_lambda
29
- ->(target) { target.variant(variant) if result_lambda.call(target, nil) }
32
+ ->(target) { target.variant(variant) if !target.variant_assigned? && result_lambda.call(target, nil) }
30
33
  end
31
34
 
32
35
  raise ArgumentError, 'no filters provided' if filters.empty?
33
36
 
34
- set_callback(:segmented_run, :before, *filters, options, &block)
37
+ set_callback(:segmented_run, :before, *filters, options)
35
38
  end
36
39
  end
37
40
  end
@@ -42,6 +42,14 @@ module Gitlab
42
42
  @signature ||= { key: @experiment.key_for(@value), migration_keys: migration_keys }.compact
43
43
  end
44
44
 
45
+ def method_missing(method_name, *)
46
+ @value.include?(method_name.to_sym) ? @value[method_name.to_sym] : super
47
+ end
48
+
49
+ def respond_to_missing?(method_name, *)
50
+ @value.include?(method_name.to_sym) ? true : super
51
+ end
52
+
45
53
  private
46
54
 
47
55
  def process_migrations(value)
@@ -0,0 +1,193 @@
1
+ # frozen_string_literal: true
2
+
3
+ module Gitlab
4
+ class Experiment
5
+ module RSpecHelpers
6
+ def stub_experiments(experiments)
7
+ experiments.each do |name, variant|
8
+ variant = :control if variant == false
9
+ raise ArgumentError, 'variant must be a symbol or false' unless variant.is_a?(Symbol)
10
+
11
+ klass = Gitlab::Experiment.send(:constantize, name) # rubocop:disable GitlabSecurity/PublicSend
12
+
13
+ # We have to use this high level any_instance behavior as there's
14
+ # not an alternative that allows multiple wrappings of `new`.
15
+ allow_any_instance_of(klass).to receive(:enabled?).and_return(true)
16
+ allow_any_instance_of(klass).to receive(:resolve_variant_name).and_return(variant.to_s)
17
+ end
18
+ end
19
+
20
+ def wrapped_experiment(experiment, shallow: false, failure: nil, &block)
21
+ if shallow
22
+ yield experiment if block.present?
23
+ return experiment
24
+ end
25
+
26
+ receive_wrapped_new = receive(:new).and_wrap_original do |new, *new_args, &new_block|
27
+ instance = new.call(*new_args)
28
+ instance.tap(&block) if block.present?
29
+ instance.tap(&new_block) if new_block.present?
30
+ instance
31
+ end
32
+
33
+ klass = experiment.class == Class ? experiment : experiment.class
34
+ if failure
35
+ expect(klass).to receive_wrapped_new, failure
36
+ else
37
+ allow(klass).to receive_wrapped_new
38
+ end
39
+ end
40
+ end
41
+
42
+ module RSpecMatchers
43
+ extend RSpec::Matchers::DSL
44
+
45
+ def require_experiment(experiment, matcher_name, classes: false)
46
+ klass = experiment.class == Class ? experiment : experiment.class
47
+ unless klass <= Gitlab::Experiment
48
+ raise(
49
+ ArgumentError,
50
+ "#{matcher_name} matcher is limited to experiment instances#{classes ? ' and classes' : ''}"
51
+ )
52
+ end
53
+
54
+ if experiment == klass && !classes
55
+ raise ArgumentError, "#{matcher_name} matcher requires an instance of an experiment"
56
+ end
57
+
58
+ experiment != klass
59
+ end
60
+
61
+ matcher :exclude do |context|
62
+ ivar = :'@excluded'
63
+
64
+ match do |experiment|
65
+ require_experiment(experiment, 'exclude')
66
+ experiment.context(context)
67
+
68
+ experiment.instance_variable_set(ivar, nil)
69
+ !experiment.run_callbacks(:exclusion_check) { :not_excluded }
70
+ end
71
+
72
+ failure_message do
73
+ %(expected #{context} to be excluded)
74
+ end
75
+
76
+ failure_message_when_negated do
77
+ %(expected #{context} not to be excluded)
78
+ end
79
+ end
80
+
81
+ matcher :segment do |context|
82
+ ivar = :'@variant_name'
83
+
84
+ match do |experiment|
85
+ require_experiment(experiment, 'segment')
86
+ experiment.context(context)
87
+
88
+ experiment.instance_variable_set(ivar, nil)
89
+ experiment.run_callbacks(:segmented_run)
90
+
91
+ @actual = experiment.instance_variable_get(ivar)
92
+ @expected ? @actual.to_s == @expected.to_s : @actual.present?
93
+ end
94
+
95
+ chain :into do |expected|
96
+ raise ArgumentError, 'variant name must be provided' if expected.blank?
97
+
98
+ @expected = expected.to_s
99
+ end
100
+
101
+ failure_message do
102
+ %(expected #{context} to be segmented#{message_details})
103
+ end
104
+
105
+ failure_message_when_negated do
106
+ %(expected #{context} not to be segmented#{message_details})
107
+ end
108
+
109
+ def message_details
110
+ message = ''
111
+ message += %( into variant\n expected variant: #{@expected}) if @expected
112
+ message += %(\n actual variant: #{@actual}) if @actual
113
+ message
114
+ end
115
+ end
116
+
117
+ matcher :track do |event, *event_args|
118
+ match do |experiment|
119
+ expect_tracking_on(experiment, false, event, *event_args)
120
+ end
121
+
122
+ match_when_negated do |experiment|
123
+ expect_tracking_on(experiment, true, event, *event_args)
124
+ end
125
+
126
+ chain :for do |expected_variant|
127
+ raise ArgumentError, 'variant name must be provided' if expected.blank?
128
+
129
+ @expected_variant = expected_variant.to_s
130
+ end
131
+
132
+ chain(:with_context) { |expected_context| @expected_context = expected_context }
133
+ chain(:on_any_instance) { @on_self = false }
134
+
135
+ def expect_tracking_on(experiment, negated, event, *event_args)
136
+ @experiment = experiment
137
+ @on_self = true if require_experiment(experiment, 'track', classes: !@on_self) && @on_self.nil?
138
+ wrapped_experiment(experiment, shallow: @on_self, failure: failure_message(:no_new, event)) do |instance|
139
+ @experiment = instance
140
+ allow(@experiment).to receive(:track)
141
+
142
+ if negated
143
+ expect(@experiment).not_to receive_tracking_call_for(event, *event_args)
144
+ else
145
+ expect(@experiment).to receive_tracking_call_for(event, *event_args)
146
+ end
147
+ end
148
+ end
149
+
150
+ def receive_tracking_call_for(event, *event_args)
151
+ receive(:track).with(*[event, *event_args]) do # rubocop:disable CodeReuse/ActiveRecord
152
+ if @expected_variant
153
+ expect(@experiment.variant.name).to eq(@expected_variant), failure_message(:variant, event)
154
+ end
155
+
156
+ if @expected_context
157
+ expect(@experiment.context.value).to include(@expected_context), failure_message(:context, event)
158
+ end
159
+ end
160
+ end
161
+
162
+ def failure_message(failure_type, event)
163
+ case failure_type
164
+ when :variant
165
+ <<~MESSAGE.strip
166
+ expected #{@experiment.inspect} to have tracked #{event.inspect} for variant
167
+ expected variant: #{@expected_variant}
168
+ actual variant: #{@experiment.variant.name}
169
+ MESSAGE
170
+ when :context
171
+ <<~MESSAGE.strip
172
+ expected #{@experiment.inspect} to have tracked #{event.inspect} with context
173
+ expected context: #{@expected_context}
174
+ actual context: #{@experiment.context.value}
175
+ MESSAGE
176
+ when :no_new
177
+ %(expected #{@experiment.inspect} to have tracked #{event.inspect}, but no new instances were created)
178
+ end
179
+ end
180
+ end
181
+ end
182
+ end
183
+ end
184
+
185
+ RSpec.configure do |config|
186
+ config.include Gitlab::Experiment::RSpecHelpers
187
+ config.include Gitlab::Experiment::Dsl
188
+
189
+ config.include Gitlab::Experiment::RSpecMatchers, :experiment
190
+ config.define_derived_metadata(file_path: Regexp.new('/spec/experiments/')) do |metadata|
191
+ metadata[:type] = :experiment
192
+ end
193
+ end
@@ -2,6 +2,6 @@
2
2
 
3
3
  module Gitlab
4
4
  class Experiment
5
- VERSION = '0.4.4'
5
+ VERSION = '0.4.9'
6
6
  end
7
7
  end
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: gitlab-experiment
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.4.4
4
+ version: 0.4.9
5
5
  platform: ruby
6
6
  authors:
7
7
  - GitLab
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2020-12-15 00:00:00.000000000 Z
11
+ date: 2021-01-25 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: activesupport
@@ -72,6 +72,7 @@ files:
72
72
  - lib/gitlab/experiment/cookies.rb
73
73
  - lib/gitlab/experiment/dsl.rb
74
74
  - lib/gitlab/experiment/engine.rb
75
+ - lib/gitlab/experiment/rspec.rb
75
76
  - lib/gitlab/experiment/variant.rb
76
77
  - lib/gitlab/experiment/version.rb
77
78
  homepage: https://gitlab.com/gitlab-org/gitlab-experiment