gitlab-experiment 0.4.7 → 0.4.12
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/README.md +134 -26
- data/lib/gitlab/experiment.rb +21 -17
- data/lib/gitlab/experiment/cache.rb +76 -0
- data/lib/gitlab/experiment/cache/redis_hash_store.rb +68 -0
- data/lib/gitlab/experiment/callbacks.rb +3 -3
- data/lib/gitlab/experiment/engine.rb +10 -5
- data/lib/gitlab/experiment/rspec.rb +100 -18
- data/lib/gitlab/experiment/version.rb +1 -1
- metadata +4 -3
- data/lib/gitlab/experiment/caching.rb +0 -39
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 231c8403f2957e01a3b94cb0b1d28463ec56a609afd0116d2761057ecd99dcaf
|
4
|
+
data.tar.gz: 836395307a137302746e0dabed9cda973cc47250c7a4df233c2ddcfe4db3db33
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: e4eab7b1929389e75ec99569e81e9cf2c6f91407b3bef711080db796597093b5106e999be5a005c401d4e125d7bc244911acc30099670d5439329b56663bc196
|
7
|
+
data.tar.gz: f4bb5c44f9511ef2a6e58a5887daa1d387619099884262da63ccdfdb259dbf24c95cf4170afa67f0169ae4d13676166659494c608f7fc61b9cfc6f6e0dbed20b
|
data/README.md
CHANGED
@@ -101,11 +101,11 @@ Here are some examples of what you can introduce once you have a custom experime
|
|
101
101
|
```ruby
|
102
102
|
class NotificationToggleExperiment < ApplicationExperiment
|
103
103
|
# Exclude any users that aren't me.
|
104
|
-
exclude :
|
104
|
+
exclude :users_named_richard
|
105
105
|
|
106
|
-
# Segment any account
|
106
|
+
# Segment any account older than 2 weeks into the candidate, without
|
107
107
|
# asking the variant resolver to decide which variant to provide.
|
108
|
-
segment :
|
108
|
+
segment :old_account?, variant: :candidate
|
109
109
|
|
110
110
|
# Define the default control behavior, which can be overridden at
|
111
111
|
# experiment time.
|
@@ -121,12 +121,12 @@ class NotificationToggleExperiment < ApplicationExperiment
|
|
121
121
|
|
122
122
|
private
|
123
123
|
|
124
|
-
def
|
125
|
-
context.actor
|
124
|
+
def users_named_richard
|
125
|
+
context.actor.first_name == 'Richard'
|
126
126
|
end
|
127
127
|
|
128
|
-
def
|
129
|
-
context.actor
|
128
|
+
def old_account?
|
129
|
+
context.actor.created_at < 2.weeks.ago
|
130
130
|
end
|
131
131
|
end
|
132
132
|
|
@@ -217,39 +217,54 @@ This library comes with the capability to segment contexts into a specific varia
|
|
217
217
|
Segmentation can be achieved by using a custom experiment class and specifying the segmentation rules at a class level.
|
218
218
|
|
219
219
|
```ruby
|
220
|
-
class
|
221
|
-
segment(variant: :variant_one) { context.actor.
|
222
|
-
segment
|
220
|
+
class ExampleExperiment < ApplicationExperiment
|
221
|
+
segment(variant: :variant_one) { context.actor.first_name == 'Richard' }
|
222
|
+
segment :old_account?, variant: :variant_two
|
223
|
+
|
224
|
+
private
|
225
|
+
|
226
|
+
def old_account?
|
227
|
+
context.actor.created_at < 2.weeks.ago
|
228
|
+
end
|
223
229
|
end
|
224
230
|
```
|
225
231
|
|
226
|
-
In the previous examples, any user
|
232
|
+
In the previous examples, any user named `'Richard'` would always receive the experience defined in "variant_one". As well, any account older than 2 weeks old would get the alternate experience defined in "variant_two".
|
227
233
|
|
228
234
|
When an experiment is run, the segmentation rules are executed in the order they're defined. The first segmentation rule to produce a truthy result is the one which gets used to assign the variant. The remaining segmentation rules are skipped.
|
229
235
|
|
230
|
-
This means that any user
|
236
|
+
This means that any user named `'Richard'`, regardless of account age, will always be provided the experience as defined in "variant_one". If you wanted the opposite logic, you can flip the order.
|
231
237
|
|
232
238
|
### Exclusion rules
|
233
239
|
|
234
|
-
Exclusion rules are similar to segmentation rules, but are intended to determine if a context should even be considered as something we should track events towards. Exclusion means we don't care about the events in relation to the given context.
|
240
|
+
Exclusion rules are similar to segmentation rules, but are intended to determine if a context should even be considered as something we should track events towards. Exclusion means we don't care about the events in relation to the given context.
|
235
241
|
|
236
242
|
```ruby
|
237
|
-
class
|
238
|
-
exclude { context.actor.
|
239
|
-
|
243
|
+
class ExampleExperiment < ApplicationExperiment
|
244
|
+
exclude :old_account?, ->{ context.actor.first_name == 'Richard' }
|
245
|
+
|
246
|
+
private
|
247
|
+
|
248
|
+
def old_account?
|
249
|
+
context.actor.created_at < 2.weeks.ago
|
250
|
+
end
|
240
251
|
end
|
241
252
|
```
|
242
253
|
|
243
|
-
The previous examples will
|
254
|
+
The previous examples will exclude all users named `'Richard'` as well as any account older than 2 weeks old. Not only will they be given the control behavior, but no events will be tracked in these cases as well.
|
244
255
|
|
245
|
-
You may need to check exclusion in custom tracking logic
|
256
|
+
You may need to check exclusion in custom tracking logic by calling `should_track?`:
|
246
257
|
|
247
258
|
```ruby
|
248
|
-
|
259
|
+
def expensive_tracking_logic
|
260
|
+
return unless should_track?
|
261
|
+
|
249
262
|
track(:my_event, value: expensive_method_call)
|
250
263
|
end
|
251
264
|
```
|
252
265
|
|
266
|
+
Note: Exclusion rules aren't the best way to determine if an experiment is enabled. There's an `enabled?` method that can be overridden to have a high-level way of determining if an experiment should be running and tracking at all. This `enabled?` check should be as efficient as possible because it's the first early opt out path an experiment can implement.
|
267
|
+
|
253
268
|
### Return value
|
254
269
|
|
255
270
|
By default the return value is a `Gitlab::Experiment` instance. In simple cases you may want only the results of the experiment though. You can call `run` within the block to get the return value of the assigned variant.
|
@@ -297,7 +312,7 @@ In providing the context migration data, we can resolve an experience and its ev
|
|
297
312
|
|
298
313
|
```ruby
|
299
314
|
# Migrate just the `:version` portion of the previous context, `{ actor: project, version: 1 }`:
|
300
|
-
experiment(:
|
315
|
+
experiment(:example, actor: project, version: 2, migrated_with: { version: 1 })
|
301
316
|
```
|
302
317
|
|
303
318
|
You can add or remove context by providing a `migrated_from` option. This approach expects a full context replacement -- i.e. what it was before you added or removed the new context key.
|
@@ -306,7 +321,7 @@ If you wanted to introduce a `version` to your context, provide the full previou
|
|
306
321
|
|
307
322
|
```ruby
|
308
323
|
# Migrate the full context from `{ actor: project }` to `{ actor: project, version: 1 }`:
|
309
|
-
experiment(:
|
324
|
+
experiment(:example, actor: project, version: 1, migrated_from: { actor: project })
|
310
325
|
```
|
311
326
|
|
312
327
|
This can impact an experience if you:
|
@@ -327,23 +342,23 @@ To read and write cookies, we provide the `request` from within the controller a
|
|
327
342
|
You'll need to provide the `request` as an option to the experiment if it's outside of the controller and views.
|
328
343
|
|
329
344
|
```ruby
|
330
|
-
experiment(:
|
345
|
+
experiment(:example, actor: user, request: request)
|
331
346
|
```
|
332
347
|
|
333
348
|
The cookie isn't set if the `actor` key isn't present at all in the context. Meaning that when no `actor` key is provided, the cookie will not be set.
|
334
349
|
|
335
350
|
```ruby
|
336
351
|
# actor is not present, so no cookie is set
|
337
|
-
experiment(:
|
352
|
+
experiment(:example, project: project)
|
338
353
|
|
339
354
|
# actor is present and is nil, so the cookie is set and used
|
340
|
-
experiment(:
|
355
|
+
experiment(:example, actor: nil, project: project)
|
341
356
|
|
342
357
|
# actor is present and set to a value, so no cookie is set
|
343
|
-
experiment(:
|
358
|
+
experiment(:example, actor: user, project: project)
|
344
359
|
```
|
345
360
|
|
346
|
-
For edge cases, you can pass the cookie through by assigning it yourself -- e.g. `actor: request.cookie_jar.signed['
|
361
|
+
For edge cases, you can pass the cookie through by assigning it yourself -- e.g. `actor: request.cookie_jar.signed['example_actor']`. The cookie name is the full experiment name (including any configured prefix) with `_actor` appended -- e.g. `gitlab_notification_toggle_actor` for the `:notification_toggle` experiment key with a configured prefix of `gitlab`.
|
347
362
|
|
348
363
|
## Configuration
|
349
364
|
|
@@ -401,6 +416,99 @@ Gitlab::Experiment.configure do |config|
|
|
401
416
|
end
|
402
417
|
```
|
403
418
|
|
419
|
+
## Testing (rspec support)
|
420
|
+
|
421
|
+
This gem comes with some rspec helpers and custom matchers. These are in flux at the time of writing.
|
422
|
+
|
423
|
+
First, require the rspec support file:
|
424
|
+
|
425
|
+
```ruby
|
426
|
+
require 'gitlab/experiment/rspec'
|
427
|
+
```
|
428
|
+
|
429
|
+
This mixes in some of the basics, but the matchers and other aspects need to be included. This happens automatically for files in `spec/experiments`, but for other files and specs you want to include it in, you can specify the `:experiment` type:
|
430
|
+
|
431
|
+
```ruby
|
432
|
+
it "tests", :experiment do
|
433
|
+
end
|
434
|
+
```
|
435
|
+
|
436
|
+
### Stub helpers
|
437
|
+
|
438
|
+
You can stub experiments using `stub_experiments`. Pass it a hash using experiment names as the keys and the variants you want each to resolve to as the values:
|
439
|
+
|
440
|
+
```ruby
|
441
|
+
# Ensures the experiments named `:example` & `:example2` are both
|
442
|
+
# "enabled" and that each will resolve to the given variant
|
443
|
+
# (`:my_variant` & `:control` respectively).
|
444
|
+
stub_experiments(example: :my_variant, example2: :control)
|
445
|
+
|
446
|
+
experiment(:example) do |e|
|
447
|
+
e.enabled? # => true
|
448
|
+
e.variant.name # => 'my_variant'
|
449
|
+
end
|
450
|
+
|
451
|
+
experiment(:example2) do |e|
|
452
|
+
e.enabled? # => true
|
453
|
+
e.variant.name # => 'control'
|
454
|
+
end
|
455
|
+
```
|
456
|
+
|
457
|
+
### Exclusion and segmentation matchers
|
458
|
+
|
459
|
+
You can also easily test the exclusion and segmentation matchers.
|
460
|
+
|
461
|
+
```ruby
|
462
|
+
class ExampleExperiment < ApplicationExperiment
|
463
|
+
exclude { context.actor.first_name == 'Richard' }
|
464
|
+
segment(variant: :candidate) { context.actor.username == 'jejacks0n' }
|
465
|
+
end
|
466
|
+
|
467
|
+
excluded = double(username: 'rdiggitty', first_name: 'Richard')
|
468
|
+
segmented = double(username: 'jejacks0n', first_name: 'Jeremy')
|
469
|
+
|
470
|
+
# exclude matcher
|
471
|
+
expect(experiment(:example)).to exclude(actor: excluded)
|
472
|
+
expect(experiment(:example)).not_to exclude(actor: segmented)
|
473
|
+
|
474
|
+
# segment matcher
|
475
|
+
expect(experiment(:example)).to segment(actor: segmented).into(:candidate)
|
476
|
+
expect(experiment(:example)).not_to segment(actor: excluded)
|
477
|
+
```
|
478
|
+
|
479
|
+
### Tracking matcher
|
480
|
+
|
481
|
+
Tracking events is a major aspect of experimentation, and because of this we try to provide a flexible way to ensure your tracking calls are covered.
|
482
|
+
|
483
|
+
You can do this on the instance level or at an "any instance" level. At an instance level this is pretty straight forward:
|
484
|
+
|
485
|
+
```ruby
|
486
|
+
subject = experiment(:example)
|
487
|
+
|
488
|
+
expect(subject).to track(:my_event)
|
489
|
+
|
490
|
+
subject.track(:my_event)
|
491
|
+
```
|
492
|
+
|
493
|
+
You can use the `on_any_instance` chain method to specify that it could happen on any instance of the experiment. This can be useful if you're calling `experiment(:example).track` downstream:
|
494
|
+
|
495
|
+
```ruby
|
496
|
+
expect(experiment(:example)).to track(:my_event).on_any_instance
|
497
|
+
|
498
|
+
experiment(:example).track(:my_event)
|
499
|
+
```
|
500
|
+
|
501
|
+
And here's a full example of the methods that can be chained onto the `track` matcher:
|
502
|
+
|
503
|
+
```ruby
|
504
|
+
expect(experiment(:example)).to track(:my_event, value: 1, property: '_property_')
|
505
|
+
.on_any_instance
|
506
|
+
.with_context(foo: :bar)
|
507
|
+
.for(:variant_name)
|
508
|
+
|
509
|
+
experiment(:example, :variant_name, foo: :bar).track(:my_event, value: 1, property: '_property_')
|
510
|
+
```
|
511
|
+
|
404
512
|
## Tracking, anonymity and GDPR
|
405
513
|
|
406
514
|
We generally try not to track things like user identifying values in our experimentation. What we can and do track is the "experiment experience" (a.k.a. the context key).
|
data/lib/gitlab/experiment.rb
CHANGED
@@ -2,10 +2,11 @@
|
|
2
2
|
|
3
3
|
require 'scientist'
|
4
4
|
require 'active_support/callbacks'
|
5
|
+
require 'active_support/cache'
|
5
6
|
require 'active_support/core_ext/object/blank'
|
6
7
|
require 'active_support/core_ext/string/inflections'
|
7
8
|
|
8
|
-
require 'gitlab/experiment/
|
9
|
+
require 'gitlab/experiment/cache'
|
9
10
|
require 'gitlab/experiment/callbacks'
|
10
11
|
require 'gitlab/experiment/configuration'
|
11
12
|
require 'gitlab/experiment/cookies'
|
@@ -18,7 +19,7 @@ require 'gitlab/experiment/engine' if defined?(Rails::Engine)
|
|
18
19
|
module Gitlab
|
19
20
|
class Experiment
|
20
21
|
include Scientist::Experiment
|
21
|
-
include
|
22
|
+
include Cache
|
22
23
|
include Callbacks
|
23
24
|
|
24
25
|
class << self
|
@@ -66,6 +67,10 @@ module Gitlab
|
|
66
67
|
yield self if block_given?
|
67
68
|
end
|
68
69
|
|
70
|
+
def inspect
|
71
|
+
"#<#{self.class.name || 'AnonymousClass'}:#{format('0x%016X', __id__)} @name=#{name} @context=#{context.value}>"
|
72
|
+
end
|
73
|
+
|
69
74
|
def context(value = nil)
|
70
75
|
return @context if value.blank?
|
71
76
|
|
@@ -74,11 +79,8 @@ module Gitlab
|
|
74
79
|
end
|
75
80
|
|
76
81
|
def variant(value = nil)
|
77
|
-
if value.
|
78
|
-
|
79
|
-
end
|
80
|
-
|
81
|
-
@variant_name = value unless value.blank?
|
82
|
+
@variant_name = cache_variant(value) if value.present?
|
83
|
+
return Variant.new(name: (@variant_name || :unresolved).to_s) if @variant_name || @resolving_variant
|
82
84
|
|
83
85
|
if enabled?
|
84
86
|
@variant_name ||= :control if excluded?
|
@@ -89,18 +91,16 @@ module Gitlab
|
|
89
91
|
end
|
90
92
|
end
|
91
93
|
|
92
|
-
|
94
|
+
run_callbacks(segmentation_callback_chain) do
|
95
|
+
@variant_name ||= :control
|
96
|
+
Variant.new(name: @variant_name.to_s)
|
97
|
+
end
|
93
98
|
ensure
|
94
99
|
@resolving_variant = false
|
95
100
|
end
|
96
101
|
|
97
102
|
def run(variant_name = nil)
|
98
|
-
@result ||=
|
99
|
-
variant_name = variant(variant_name).name
|
100
|
-
run_callbacks(run_with_segmenting? ? :segmented_run : :unsegmented_run) do
|
101
|
-
super(@variant_name ||= variant_name)
|
102
|
-
end
|
103
|
-
end
|
103
|
+
@result ||= super(variant(variant_name).name)
|
104
104
|
end
|
105
105
|
|
106
106
|
def publish(result)
|
@@ -153,7 +153,9 @@ module Gitlab
|
|
153
153
|
end
|
154
154
|
|
155
155
|
def excluded?
|
156
|
-
@excluded
|
156
|
+
return @excluded if defined?(@excluded)
|
157
|
+
|
158
|
+
@excluded = !@context.trackable? || # adhere to DNT headers
|
157
159
|
!run_callbacks(:exclusion_check) { :not_excluded } # didn't pass exclusion check
|
158
160
|
end
|
159
161
|
|
@@ -167,8 +169,10 @@ module Gitlab
|
|
167
169
|
|
168
170
|
protected
|
169
171
|
|
170
|
-
def
|
171
|
-
!variant_assigned? && enabled? && !excluded?
|
172
|
+
def segmentation_callback_chain
|
173
|
+
return :segmentation_check if !variant_assigned? && enabled? && !excluded?
|
174
|
+
|
175
|
+
:unsegmented
|
172
176
|
end
|
173
177
|
|
174
178
|
def variant_assigned?
|
@@ -0,0 +1,76 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
module Gitlab
|
4
|
+
class Experiment
|
5
|
+
module Cache
|
6
|
+
autoload :RedisHashStore, 'gitlab/experiment/cache/redis_hash_store.rb'
|
7
|
+
|
8
|
+
class Interface
|
9
|
+
attr_reader :store, :key
|
10
|
+
|
11
|
+
def initialize(experiment, store)
|
12
|
+
@experiment = experiment
|
13
|
+
@store = store
|
14
|
+
@key = experiment.cache_key
|
15
|
+
end
|
16
|
+
|
17
|
+
def read
|
18
|
+
store.read(key)
|
19
|
+
end
|
20
|
+
|
21
|
+
def write(value = nil)
|
22
|
+
store.write(key, value || @experiment.variant.name)
|
23
|
+
end
|
24
|
+
|
25
|
+
def delete
|
26
|
+
store.delete(key)
|
27
|
+
end
|
28
|
+
|
29
|
+
def attr_get(name)
|
30
|
+
store.read(@experiment.cache_key(name, suffix: :attrs))
|
31
|
+
end
|
32
|
+
|
33
|
+
def attr_set(name, value)
|
34
|
+
store.write(@experiment.cache_key(name, suffix: :attrs), value)
|
35
|
+
end
|
36
|
+
|
37
|
+
def attr_inc(name, amount = 1)
|
38
|
+
store.increment(@experiment.cache_key(name, suffix: :attrs), amount)
|
39
|
+
end
|
40
|
+
end
|
41
|
+
|
42
|
+
def cache
|
43
|
+
@cache ||= Interface.new(self, Configuration.cache)
|
44
|
+
end
|
45
|
+
|
46
|
+
def cache_variant(specified = nil, &block)
|
47
|
+
return (specified.presence || yield) unless cache.store
|
48
|
+
|
49
|
+
result = migrated_cache_fetch(cache.store, &block)
|
50
|
+
return result unless specified.present?
|
51
|
+
|
52
|
+
cache.write(specified) if result != specified
|
53
|
+
specified
|
54
|
+
end
|
55
|
+
|
56
|
+
def cache_key(key = nil, suffix: nil)
|
57
|
+
"#{[name, suffix].compact.join('_')}:#{key || context.signature[:key]}"
|
58
|
+
end
|
59
|
+
|
60
|
+
private
|
61
|
+
|
62
|
+
def migrated_cache_fetch(store, &block)
|
63
|
+
migrations = context.signature[:migration_keys]&.map { |key| cache_key(key) } || []
|
64
|
+
migrations.find do |old_key|
|
65
|
+
next unless (value = store.read(old_key))
|
66
|
+
|
67
|
+
store.write(cache_key, value)
|
68
|
+
store.delete(old_key)
|
69
|
+
return value
|
70
|
+
end
|
71
|
+
|
72
|
+
store.fetch(cache_key, &block)
|
73
|
+
end
|
74
|
+
end
|
75
|
+
end
|
76
|
+
end
|
@@ -0,0 +1,68 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
require 'active_support/notifications'
|
4
|
+
|
5
|
+
# This cache strategy is an implementation on top of the redis hash data type,
|
6
|
+
# that also adheres to the ActiveSupport::Cache::Store interface. It's a good
|
7
|
+
# example of how to build a custom caching strategy for Gitlab::Experiment, and
|
8
|
+
# is intended to be a long lived cache -- until the experiment is cleaned up.
|
9
|
+
#
|
10
|
+
# The data structure:
|
11
|
+
# key: experiment.name
|
12
|
+
# fields: context key => variant name
|
13
|
+
#
|
14
|
+
# Gitlab::Experiment::Configuration.cache = Gitlab::Experiment::Cache::RedisHashStore.new(
|
15
|
+
# pool: -> { Gitlab::Redis::SharedState.with { |redis| yield redis } }
|
16
|
+
# )
|
17
|
+
module Gitlab
|
18
|
+
class Experiment
|
19
|
+
module Cache
|
20
|
+
class RedisHashStore < ActiveSupport::Cache::Store
|
21
|
+
# Clears the entire cache for a given experiment. Be careful with this
|
22
|
+
# since it would reset all resolved variants for the entire experiment.
|
23
|
+
def clear(key:)
|
24
|
+
key = hkey(key)[0] # extract only the first part of the key
|
25
|
+
pool do |redis|
|
26
|
+
case redis.type(key)
|
27
|
+
when 'hash', 'none'
|
28
|
+
redis.del(key) # delete the primary experiment key
|
29
|
+
redis.del("#{key}_attrs") # delete the experiment attributes key
|
30
|
+
else raise ArgumentError, 'invalid call to clear a non-hash cache key'
|
31
|
+
end
|
32
|
+
end
|
33
|
+
end
|
34
|
+
|
35
|
+
def increment(key, amount = 1)
|
36
|
+
pool { |redis| redis.hincrby(*hkey(key), amount) }
|
37
|
+
end
|
38
|
+
|
39
|
+
private
|
40
|
+
|
41
|
+
def pool(&block)
|
42
|
+
raise ArgumentError, 'missing block' unless block.present?
|
43
|
+
|
44
|
+
@options[:pool].call(&block)
|
45
|
+
end
|
46
|
+
|
47
|
+
def hkey(key)
|
48
|
+
key.to_s.split(':') # this assumes the default strategy in gitlab-experiment
|
49
|
+
end
|
50
|
+
|
51
|
+
def read_entry(key, **options)
|
52
|
+
value = pool { |redis| redis.hget(*hkey(key)) }
|
53
|
+
value.nil? ? nil : ActiveSupport::Cache::Entry.new(value)
|
54
|
+
end
|
55
|
+
|
56
|
+
def write_entry(key, entry, **options)
|
57
|
+
return false if entry.value.blank? # don't cache any empty values
|
58
|
+
|
59
|
+
pool { |redis| redis.hset(*hkey(key), entry.value) }
|
60
|
+
end
|
61
|
+
|
62
|
+
def delete_entry(key, **options)
|
63
|
+
pool { |redis| redis.hdel(*hkey(key)) }
|
64
|
+
end
|
65
|
+
end
|
66
|
+
end
|
67
|
+
end
|
68
|
+
end
|
@@ -7,8 +7,8 @@ module Gitlab
|
|
7
7
|
include ActiveSupport::Callbacks
|
8
8
|
|
9
9
|
included do
|
10
|
-
define_callbacks(:
|
11
|
-
define_callbacks(:
|
10
|
+
define_callbacks(:unsegmented)
|
11
|
+
define_callbacks(:segmentation_check)
|
12
12
|
define_callbacks(:exclusion_check, skip_after_callbacks_if_terminated: true)
|
13
13
|
end
|
14
14
|
|
@@ -34,7 +34,7 @@ module Gitlab
|
|
34
34
|
|
35
35
|
raise ArgumentError, 'no filters provided' if filters.empty?
|
36
36
|
|
37
|
-
set_callback(:
|
37
|
+
set_callback(:segmentation_check, :before, *filters, options)
|
38
38
|
end
|
39
39
|
end
|
40
40
|
end
|
@@ -4,13 +4,18 @@ module Gitlab
|
|
4
4
|
class Experiment
|
5
5
|
class Engine < ::Rails::Engine
|
6
6
|
def self.include_dsl
|
7
|
-
ActionController
|
8
|
-
|
9
|
-
|
7
|
+
if defined?(ActionController)
|
8
|
+
ActionController::Base.include(Dsl)
|
9
|
+
ActionController::Base.helper_method(:experiment)
|
10
|
+
end
|
11
|
+
|
12
|
+
return unless defined?(ActionMailer)
|
10
13
|
|
11
|
-
|
12
|
-
|
14
|
+
ActionMailer::Base.include(Dsl)
|
15
|
+
ActionMailer::Base.helper_method(:experiment)
|
13
16
|
end
|
17
|
+
|
18
|
+
config.after_initialize { include_dsl }
|
14
19
|
end
|
15
20
|
end
|
16
21
|
end
|
@@ -9,14 +9,32 @@ module Gitlab
|
|
9
9
|
raise ArgumentError, 'variant must be a symbol or false' unless variant.is_a?(Symbol)
|
10
10
|
|
11
11
|
klass = Gitlab::Experiment.send(:constantize, name) # rubocop:disable GitlabSecurity/PublicSend
|
12
|
-
allow(klass).to receive(:new).and_wrap_original do |new, *args, &block|
|
13
|
-
new.call(*args).tap do |instance|
|
14
|
-
allow(instance).to receive(:enabled?).and_return(true)
|
15
|
-
allow(instance).to receive(:resolve_variant_name).and_return(variant.to_s)
|
16
12
|
|
17
|
-
|
18
|
-
|
19
|
-
|
13
|
+
# We have to use this high level any_instance behavior as there's
|
14
|
+
# not an alternative that allows multiple wrappings of `new`.
|
15
|
+
allow_any_instance_of(klass).to receive(:enabled?).and_return(true)
|
16
|
+
allow_any_instance_of(klass).to receive(:resolve_variant_name).and_return(variant.to_s)
|
17
|
+
end
|
18
|
+
end
|
19
|
+
|
20
|
+
def wrapped_experiment(experiment, shallow: false, failure: nil, &block)
|
21
|
+
if shallow
|
22
|
+
yield experiment if block.present?
|
23
|
+
return experiment
|
24
|
+
end
|
25
|
+
|
26
|
+
receive_wrapped_new = receive(:new).and_wrap_original do |new, *new_args, &new_block|
|
27
|
+
instance = new.call(*new_args)
|
28
|
+
instance.tap(&block) if block.present?
|
29
|
+
instance.tap(&new_block) if new_block.present?
|
30
|
+
instance
|
31
|
+
end
|
32
|
+
|
33
|
+
klass = experiment.class == Class ? experiment : experiment.class
|
34
|
+
if failure
|
35
|
+
expect(klass).to receive_wrapped_new, failure
|
36
|
+
else
|
37
|
+
allow(klass).to receive_wrapped_new
|
20
38
|
end
|
21
39
|
end
|
22
40
|
end
|
@@ -24,10 +42,27 @@ module Gitlab
|
|
24
42
|
module RSpecMatchers
|
25
43
|
extend RSpec::Matchers::DSL
|
26
44
|
|
45
|
+
def require_experiment(experiment, matcher_name, classes: false)
|
46
|
+
klass = experiment.class == Class ? experiment : experiment.class
|
47
|
+
unless klass <= Gitlab::Experiment
|
48
|
+
raise(
|
49
|
+
ArgumentError,
|
50
|
+
"#{matcher_name} matcher is limited to experiment instances#{classes ? ' and classes' : ''}"
|
51
|
+
)
|
52
|
+
end
|
53
|
+
|
54
|
+
if experiment == klass && !classes
|
55
|
+
raise ArgumentError, "#{matcher_name} matcher requires an instance of an experiment"
|
56
|
+
end
|
57
|
+
|
58
|
+
experiment != klass
|
59
|
+
end
|
60
|
+
|
27
61
|
matcher :exclude do |context|
|
28
62
|
ivar = :'@excluded'
|
29
63
|
|
30
64
|
match do |experiment|
|
65
|
+
require_experiment(experiment, 'exclude')
|
31
66
|
experiment.context(context)
|
32
67
|
|
33
68
|
experiment.instance_variable_set(ivar, nil)
|
@@ -47,10 +82,11 @@ module Gitlab
|
|
47
82
|
ivar = :'@variant_name'
|
48
83
|
|
49
84
|
match do |experiment|
|
85
|
+
require_experiment(experiment, 'segment')
|
50
86
|
experiment.context(context)
|
51
87
|
|
52
88
|
experiment.instance_variable_set(ivar, nil)
|
53
|
-
experiment.run_callbacks(:
|
89
|
+
experiment.run_callbacks(:segmentation_check)
|
54
90
|
|
55
91
|
@actual = experiment.instance_variable_get(ivar)
|
56
92
|
@expected ? @actual.to_s == @expected.to_s : @actual.present?
|
@@ -72,29 +108,74 @@ module Gitlab
|
|
72
108
|
|
73
109
|
def message_details
|
74
110
|
message = ''
|
75
|
-
message += %(\n
|
76
|
-
message += %(\n
|
111
|
+
message += %( into variant\n expected variant: #{@expected}) if @expected
|
112
|
+
message += %(\n actual variant: #{@actual}) if @actual
|
77
113
|
message
|
78
114
|
end
|
79
115
|
end
|
80
116
|
|
81
117
|
matcher :track do |event, *event_args|
|
82
118
|
match do |experiment|
|
83
|
-
|
119
|
+
expect_tracking_on(experiment, false, event, *event_args)
|
84
120
|
end
|
85
121
|
|
86
122
|
match_when_negated do |experiment|
|
87
|
-
|
123
|
+
expect_tracking_on(experiment, true, event, *event_args)
|
88
124
|
end
|
89
125
|
|
90
|
-
|
91
|
-
|
92
|
-
|
126
|
+
chain :for do |expected_variant|
|
127
|
+
raise ArgumentError, 'variant name must be provided' if expected.blank?
|
128
|
+
|
129
|
+
@expected_variant = expected_variant.to_s
|
130
|
+
end
|
131
|
+
|
132
|
+
chain(:with_context) { |expected_context| @expected_context = expected_context }
|
133
|
+
chain(:on_any_instance) { @on_self = false }
|
134
|
+
|
135
|
+
def expect_tracking_on(experiment, negated, event, *event_args)
|
136
|
+
@experiment = experiment
|
137
|
+
@on_self = true if require_experiment(experiment, 'track', classes: !@on_self) && @on_self.nil?
|
138
|
+
wrapped_experiment(experiment, shallow: @on_self, failure: failure_message(:no_new, event)) do |instance|
|
139
|
+
@experiment = instance
|
140
|
+
allow(@experiment).to receive(:track)
|
141
|
+
|
142
|
+
if negated
|
143
|
+
expect(@experiment).not_to receive_tracking_call_for(event, *event_args)
|
144
|
+
else
|
145
|
+
expect(@experiment).to receive_tracking_call_for(event, *event_args)
|
146
|
+
end
|
93
147
|
end
|
94
148
|
end
|
95
149
|
|
96
|
-
def
|
97
|
-
receive(:track).with(*
|
150
|
+
def receive_tracking_call_for(event, *event_args)
|
151
|
+
receive(:track).with(*[event, *event_args]) do # rubocop:disable CodeReuse/ActiveRecord
|
152
|
+
if @expected_variant
|
153
|
+
expect(@experiment.variant.name).to eq(@expected_variant), failure_message(:variant, event)
|
154
|
+
end
|
155
|
+
|
156
|
+
if @expected_context
|
157
|
+
expect(@experiment.context.value).to include(@expected_context), failure_message(:context, event)
|
158
|
+
end
|
159
|
+
end
|
160
|
+
end
|
161
|
+
|
162
|
+
def failure_message(failure_type, event)
|
163
|
+
case failure_type
|
164
|
+
when :variant
|
165
|
+
<<~MESSAGE.strip
|
166
|
+
expected #{@experiment.inspect} to have tracked #{event.inspect} for variant
|
167
|
+
expected variant: #{@expected_variant}
|
168
|
+
actual variant: #{@experiment.variant.name}
|
169
|
+
MESSAGE
|
170
|
+
when :context
|
171
|
+
<<~MESSAGE.strip
|
172
|
+
expected #{@experiment.inspect} to have tracked #{event.inspect} with context
|
173
|
+
expected context: #{@expected_context}
|
174
|
+
actual context: #{@experiment.context.value}
|
175
|
+
MESSAGE
|
176
|
+
when :no_new
|
177
|
+
%(expected #{@experiment.inspect} to have tracked #{event.inspect}, but no new instances were created)
|
178
|
+
end
|
98
179
|
end
|
99
180
|
end
|
100
181
|
end
|
@@ -103,9 +184,10 @@ end
|
|
103
184
|
|
104
185
|
RSpec.configure do |config|
|
105
186
|
config.include Gitlab::Experiment::RSpecHelpers
|
187
|
+
config.include Gitlab::Experiment::Dsl
|
106
188
|
|
107
189
|
config.include Gitlab::Experiment::RSpecMatchers, :experiment
|
108
|
-
config.define_derived_metadata(file_path:
|
190
|
+
config.define_derived_metadata(file_path: Regexp.new('/spec/experiments/')) do |metadata|
|
109
191
|
metadata[:type] = :experiment
|
110
192
|
end
|
111
193
|
end
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: gitlab-experiment
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.4.
|
4
|
+
version: 0.4.12
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- GitLab
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2021-
|
11
|
+
date: 2021-02-17 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: activesupport
|
@@ -65,7 +65,8 @@ files:
|
|
65
65
|
- lib/generators/test_unit/experiment/experiment_generator.rb
|
66
66
|
- lib/generators/test_unit/experiment/templates/experiment_test.rb.tt
|
67
67
|
- lib/gitlab/experiment.rb
|
68
|
-
- lib/gitlab/experiment/
|
68
|
+
- lib/gitlab/experiment/cache.rb
|
69
|
+
- lib/gitlab/experiment/cache/redis_hash_store.rb
|
69
70
|
- lib/gitlab/experiment/callbacks.rb
|
70
71
|
- lib/gitlab/experiment/configuration.rb
|
71
72
|
- lib/gitlab/experiment/context.rb
|
@@ -1,39 +0,0 @@
|
|
1
|
-
# frozen_string_literal: true
|
2
|
-
|
3
|
-
module Gitlab
|
4
|
-
class Experiment
|
5
|
-
module Caching
|
6
|
-
def cache_variant(specified = nil, &block)
|
7
|
-
cache = Configuration.cache
|
8
|
-
return (specified.presence || yield) unless cache
|
9
|
-
|
10
|
-
key, migrations = cache_strategy
|
11
|
-
result = migrated_cache(cache, migrations || [], key) || cache.fetch(key, &block)
|
12
|
-
return result unless specified.present?
|
13
|
-
|
14
|
-
cache.write(cache_key, specified) if result != specified
|
15
|
-
specified
|
16
|
-
end
|
17
|
-
|
18
|
-
def cache_key(key = nil)
|
19
|
-
"#{name}:#{key || context.signature[:key]}"
|
20
|
-
end
|
21
|
-
|
22
|
-
private
|
23
|
-
|
24
|
-
def cache_strategy
|
25
|
-
[cache_key, context.signature[:migration_keys]&.map { |key| cache_key(key) }]
|
26
|
-
end
|
27
|
-
|
28
|
-
def migrated_cache(cache, migrations, new_key)
|
29
|
-
migrations.find do |old_key|
|
30
|
-
next unless (value = cache.read(old_key))
|
31
|
-
|
32
|
-
cache.write(new_key, value)
|
33
|
-
cache.delete(old_key)
|
34
|
-
break value
|
35
|
-
end
|
36
|
-
end
|
37
|
-
end
|
38
|
-
end
|
39
|
-
end
|