gitlab-experiment 0.4.9 → 0.5.1
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/README.md +171 -56
- data/lib/generators/gitlab/experiment/install/templates/initializer.rb.tt +13 -10
- data/lib/gitlab/experiment.rb +63 -102
- data/lib/gitlab/experiment/base_interface.rb +81 -0
- data/lib/gitlab/experiment/cache.rb +78 -0
- data/lib/gitlab/experiment/cache/redis_hash_store.rb +68 -0
- data/lib/gitlab/experiment/callbacks.rb +9 -18
- data/lib/gitlab/experiment/configuration.rb +24 -5
- data/lib/gitlab/experiment/context.rb +2 -0
- data/lib/gitlab/experiment/dsl.rb +10 -1
- data/lib/gitlab/experiment/engine.rb +10 -5
- data/lib/gitlab/experiment/rollout.rb +27 -0
- data/lib/gitlab/experiment/rollout/first.rb +16 -0
- data/lib/gitlab/experiment/rollout/random.rb +15 -0
- data/lib/gitlab/experiment/rollout/round_robin.rb +23 -0
- data/lib/gitlab/experiment/rspec.rb +15 -8
- data/lib/gitlab/experiment/version.rb +1 -1
- metadata +13 -7
- data/lib/gitlab/experiment/caching.rb +0 -39
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 13f170ab47c88f393041b3f4b8b69c168d8a159fcfcdf2fe7458ceca75a9b9db
|
4
|
+
data.tar.gz: 5bc7fe4a5c8a493ce14413c109ac4424bdd61d8b0c3513530963e52586bfaf5f
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: bc52008aea6015b274fb7200c83b14145b355be637f1d286a8c9590df40c5d5e2c1d6da9e6f41f7f1970c576dfb13af69f43610e162eebff642d3945455e1b24
|
7
|
+
data.tar.gz: 9f16a5b7bdd2ae9ea9b624946d5055878c67d67e3f755900bccdac153c330fab77ed98d1d88b0f1a869e3bbd1b1f347366252db98207513e789c387c6fd509d0
|
data/README.md
CHANGED
@@ -3,7 +3,7 @@ GitLab Experiment
|
|
3
3
|
|
4
4
|
<img alt="experiment" src="/uploads/60990b2dbf4c0406bbf8b7f998de2dea/experiment.png" align="right" width="40%">
|
5
5
|
|
6
|
-
Here at GitLab, we run experiments as A/B/n tests and review the data the experiment generates. From that data, we determine the best performing variant and promote it as the new default code path. Or revert back to the control if no variant outperformed it.
|
6
|
+
Here at GitLab, we run experiments as A/B/n tests and review the data the experiment generates. From that data, we determine the best performing variant and promote it as the new default code path. Or revert back to the control if no variant outperformed it. You can read our [Experiment Guide](https://docs.gitlab.com/ee/development/experiment_guide/) docs if you're curious about how we use this gem internally.
|
7
7
|
|
8
8
|
This library provides a clean and elegant DSL (domain specific language) to define, run, and track your GitLab experiment.
|
9
9
|
|
@@ -64,7 +64,7 @@ end
|
|
64
64
|
|
65
65
|
You can define the experiment using simple control/candidate paths, or provide named variants.
|
66
66
|
|
67
|
-
Handling
|
67
|
+
Handling [multivariate](https://en.wikipedia.org/wiki/Multivariate_statistics) experiments is up to the configuration you provide around resolving variants. But in our example we may want to try with and without the confirmation. We can run any number of variations in our experiments this way.
|
68
68
|
|
69
69
|
```ruby
|
70
70
|
experiment(:notification_toggle, actor: user) do |e|
|
@@ -101,11 +101,11 @@ Here are some examples of what you can introduce once you have a custom experime
|
|
101
101
|
```ruby
|
102
102
|
class NotificationToggleExperiment < ApplicationExperiment
|
103
103
|
# Exclude any users that aren't me.
|
104
|
-
exclude :
|
104
|
+
exclude :users_named_richard
|
105
105
|
|
106
|
-
# Segment any account
|
106
|
+
# Segment any account older than 2 weeks into the candidate, without
|
107
107
|
# asking the variant resolver to decide which variant to provide.
|
108
|
-
segment :
|
108
|
+
segment :old_account?, variant: :candidate
|
109
109
|
|
110
110
|
# Define the default control behavior, which can be overridden at
|
111
111
|
# experiment time.
|
@@ -121,12 +121,12 @@ class NotificationToggleExperiment < ApplicationExperiment
|
|
121
121
|
|
122
122
|
private
|
123
123
|
|
124
|
-
def
|
125
|
-
context.actor
|
124
|
+
def users_named_richard
|
125
|
+
context.actor.first_name == 'Richard'
|
126
126
|
end
|
127
127
|
|
128
|
-
def
|
129
|
-
context.actor
|
128
|
+
def old_account?
|
129
|
+
context.actor.created_at < 2.weeks.ago
|
130
130
|
end
|
131
131
|
end
|
132
132
|
|
@@ -149,37 +149,13 @@ experiment(:notification_toggle, actor: user) do |e|
|
|
149
149
|
end
|
150
150
|
```
|
151
151
|
|
152
|
-
<details>
|
153
|
-
<summary>You can also use the lower level class interface...</summary>
|
154
|
-
|
155
|
-
### Using the `.run` approach
|
156
|
-
|
157
|
-
This is useful if you haven't included the DSL and so don't have access to the `experiment` method, but still want to execute an experiment. This is ultimately what the `experiment` method calls through to, and the method signatures are the same.
|
158
|
-
|
159
|
-
```ruby
|
160
|
-
exp = Gitlab::Experiment.run(:notification_toggle, actor: user) do |e|
|
161
|
-
# Context may be passed in the block, but must be finalized before calling
|
162
|
-
# run or track.
|
163
|
-
e.context(project: project) # add the project to the context
|
164
|
-
|
165
|
-
# Define the control and candidate variant.
|
166
|
-
e.use { render_toggle } # control
|
167
|
-
e.try { render_button } # candidate
|
168
|
-
end
|
169
|
-
|
170
|
-
# Track an event on the experiment we've defined.
|
171
|
-
exp.track(:clicked_button)
|
172
|
-
```
|
173
|
-
|
174
|
-
</details>
|
175
|
-
|
176
152
|
<details>
|
177
153
|
<summary>You can also specify the variant to use for segmentation...</summary>
|
178
154
|
|
179
|
-
### Specifying variant
|
180
|
-
|
181
155
|
Generally, defining segmentation rules is a better way to approach routing into specific variants, but it's possible to explicitly specify the variant when running an experiment. It's important to know what this might do to your data during rollout, so use this with careful consideration.
|
182
156
|
|
157
|
+
Any time a specific variant is provided (including `:control`) it will be cached for that context, if caching is enabled.
|
158
|
+
|
183
159
|
```ruby
|
184
160
|
experiment(:notification_toggle, :no_interface, actor: user) do |e|
|
185
161
|
e.use { render_toggle } # control
|
@@ -217,39 +193,54 @@ This library comes with the capability to segment contexts into a specific varia
|
|
217
193
|
Segmentation can be achieved by using a custom experiment class and specifying the segmentation rules at a class level.
|
218
194
|
|
219
195
|
```ruby
|
220
|
-
class
|
221
|
-
segment(variant: :variant_one) { context.actor.
|
222
|
-
segment
|
196
|
+
class ExampleExperiment < ApplicationExperiment
|
197
|
+
segment(variant: :variant_one) { context.actor.first_name == 'Richard' }
|
198
|
+
segment :old_account?, variant: :variant_two
|
199
|
+
|
200
|
+
private
|
201
|
+
|
202
|
+
def old_account?
|
203
|
+
context.actor.created_at < 2.weeks.ago
|
204
|
+
end
|
223
205
|
end
|
224
206
|
```
|
225
207
|
|
226
|
-
In the previous examples, any user
|
208
|
+
In the previous examples, any user named `'Richard'` would always receive the experience defined in "variant_one". As well, any account older than 2 weeks old would get the alternate experience defined in "variant_two".
|
227
209
|
|
228
210
|
When an experiment is run, the segmentation rules are executed in the order they're defined. The first segmentation rule to produce a truthy result is the one which gets used to assign the variant. The remaining segmentation rules are skipped.
|
229
211
|
|
230
|
-
This means that any user
|
212
|
+
This means that any user named `'Richard'`, regardless of account age, will always be provided the experience as defined in "variant_one". If you wanted the opposite logic, you can flip the order.
|
231
213
|
|
232
214
|
### Exclusion rules
|
233
215
|
|
234
|
-
Exclusion rules are similar to segmentation rules, but are intended to determine if a context should even be considered as something we should track events towards. Exclusion means we don't care about the events in relation to the given context.
|
216
|
+
Exclusion rules are similar to segmentation rules, but are intended to determine if a context should even be considered as something we should track events towards. Exclusion means we don't care about the events in relation to the given context.
|
235
217
|
|
236
218
|
```ruby
|
237
|
-
class
|
238
|
-
exclude { context.actor.
|
239
|
-
|
219
|
+
class ExampleExperiment < ApplicationExperiment
|
220
|
+
exclude :old_account?, ->{ context.actor.first_name == 'Richard' }
|
221
|
+
|
222
|
+
private
|
223
|
+
|
224
|
+
def old_account?
|
225
|
+
context.actor.created_at < 2.weeks.ago
|
226
|
+
end
|
240
227
|
end
|
241
228
|
```
|
242
229
|
|
243
|
-
The previous examples will
|
230
|
+
The previous examples will exclude all users named `'Richard'` as well as any account older than 2 weeks old. Not only will they be given the control behavior, but no events will be tracked in these cases as well.
|
244
231
|
|
245
|
-
You may need to check exclusion in custom tracking logic
|
232
|
+
You may need to check exclusion in custom tracking logic by calling `should_track?`:
|
246
233
|
|
247
234
|
```ruby
|
248
|
-
|
235
|
+
def expensive_tracking_logic
|
236
|
+
return unless should_track?
|
237
|
+
|
249
238
|
track(:my_event, value: expensive_method_call)
|
250
239
|
end
|
251
240
|
```
|
252
241
|
|
242
|
+
Note: Exclusion rules aren't the best way to determine if an experiment is enabled. There's an `enabled?` method that can be overridden to have a high-level way of determining if an experiment should be running and tracking at all. This `enabled?` check should be as efficient as possible because it's the first early opt out path an experiment can implement.
|
243
|
+
|
253
244
|
### Return value
|
254
245
|
|
255
246
|
By default the return value is a `Gitlab::Experiment` instance. In simple cases you may want only the results of the experiment though. You can call `run` within the block to get the return value of the assigned variant.
|
@@ -297,7 +288,7 @@ In providing the context migration data, we can resolve an experience and its ev
|
|
297
288
|
|
298
289
|
```ruby
|
299
290
|
# Migrate just the `:version` portion of the previous context, `{ actor: project, version: 1 }`:
|
300
|
-
experiment(:
|
291
|
+
experiment(:example, actor: project, version: 2, migrated_with: { version: 1 })
|
301
292
|
```
|
302
293
|
|
303
294
|
You can add or remove context by providing a `migrated_from` option. This approach expects a full context replacement -- i.e. what it was before you added or removed the new context key.
|
@@ -306,7 +297,7 @@ If you wanted to introduce a `version` to your context, provide the full previou
|
|
306
297
|
|
307
298
|
```ruby
|
308
299
|
# Migrate the full context from `{ actor: project }` to `{ actor: project, version: 1 }`:
|
309
|
-
experiment(:
|
300
|
+
experiment(:example, actor: project, version: 1, migrated_from: { actor: project })
|
310
301
|
```
|
311
302
|
|
312
303
|
This can impact an experience if you:
|
@@ -327,23 +318,54 @@ To read and write cookies, we provide the `request` from within the controller a
|
|
327
318
|
You'll need to provide the `request` as an option to the experiment if it's outside of the controller and views.
|
328
319
|
|
329
320
|
```ruby
|
330
|
-
experiment(:
|
321
|
+
experiment(:example, actor: user, request: request)
|
331
322
|
```
|
332
323
|
|
333
324
|
The cookie isn't set if the `actor` key isn't present at all in the context. Meaning that when no `actor` key is provided, the cookie will not be set.
|
334
325
|
|
335
326
|
```ruby
|
336
327
|
# actor is not present, so no cookie is set
|
337
|
-
experiment(:
|
328
|
+
experiment(:example, project: project)
|
338
329
|
|
339
330
|
# actor is present and is nil, so the cookie is set and used
|
340
|
-
experiment(:
|
331
|
+
experiment(:example, actor: nil, project: project)
|
341
332
|
|
342
333
|
# actor is present and set to a value, so no cookie is set
|
343
|
-
experiment(:
|
334
|
+
experiment(:example, actor: user, project: project)
|
344
335
|
```
|
345
336
|
|
346
|
-
For edge cases, you can pass the cookie through by assigning it yourself -- e.g. `actor: request.cookie_jar.signed['
|
337
|
+
For edge cases, you can pass the cookie through by assigning it yourself -- e.g. `actor: request.cookie_jar.signed['example_actor']`. The cookie name is the full experiment name (including any configured prefix) with `_actor` appended -- e.g. `gitlab_notification_toggle_actor` for the `:notification_toggle` experiment key with a configured prefix of `gitlab`.
|
338
|
+
|
339
|
+
## How it works
|
340
|
+
|
341
|
+
The way the gem works is best described using the following decision tree illustration. When an experiment is run, the following logic is executed to resolve what experience should be provided, given how the experiment is defined, and the context provided.
|
342
|
+
|
343
|
+
```mermaid
|
344
|
+
graph TD
|
345
|
+
GP[General Pool/Population] --> Enabled?
|
346
|
+
Enabled? -->|Yes| Cached?[Cached? / Pre-segmented?]
|
347
|
+
Enabled? -->|No| Excluded[Control / No Tracking]
|
348
|
+
Cached? -->|No| Excluded?
|
349
|
+
Cached? -->|Yes| Cached[Cached Value]
|
350
|
+
Excluded? -->|Yes / Cached| Excluded
|
351
|
+
Excluded? -->|No| Segmented?
|
352
|
+
Segmented? -->|Yes / Cached| VariantA
|
353
|
+
Segmented? -->|No| Included?[Experiment Group?]
|
354
|
+
Included? -->|Yes| Rollout
|
355
|
+
Included? -->|No| Control
|
356
|
+
Rollout -->|Cached| VariantA
|
357
|
+
Rollout -->|Cached| VariantB
|
358
|
+
Rollout -->|Cached| VariantC
|
359
|
+
|
360
|
+
classDef included fill:#380d75,color:#ffffff,stroke:none
|
361
|
+
classDef excluded fill:#fca121,stroke:none
|
362
|
+
classDef cached fill:#2e2e2e,color:#ffffff,stroke:none
|
363
|
+
classDef default fill:#fff,stroke:#6e49cb
|
364
|
+
|
365
|
+
class VariantA,VariantB,VariantC included
|
366
|
+
class Control,Excluded excluded
|
367
|
+
class Cached cached
|
368
|
+
```
|
347
369
|
|
348
370
|
## Configuration
|
349
371
|
|
@@ -401,6 +423,99 @@ Gitlab::Experiment.configure do |config|
|
|
401
423
|
end
|
402
424
|
```
|
403
425
|
|
426
|
+
## Testing (rspec support)
|
427
|
+
|
428
|
+
This gem comes with some rspec helpers and custom matchers. These are in flux at the time of writing.
|
429
|
+
|
430
|
+
First, require the rspec support file:
|
431
|
+
|
432
|
+
```ruby
|
433
|
+
require 'gitlab/experiment/rspec'
|
434
|
+
```
|
435
|
+
|
436
|
+
This mixes in some of the basics, but the matchers and other aspects need to be included. This happens automatically for files in `spec/experiments`, but for other files and specs you want to include it in, you can specify the `:experiment` type:
|
437
|
+
|
438
|
+
```ruby
|
439
|
+
it "tests", :experiment do
|
440
|
+
end
|
441
|
+
```
|
442
|
+
|
443
|
+
### Stub helpers
|
444
|
+
|
445
|
+
You can stub experiments using `stub_experiments`. Pass it a hash using experiment names as the keys and the variants you want each to resolve to as the values:
|
446
|
+
|
447
|
+
```ruby
|
448
|
+
# Ensures the experiments named `:example` & `:example2` are both
|
449
|
+
# "enabled" and that each will resolve to the given variant
|
450
|
+
# (`:my_variant` & `:control` respectively).
|
451
|
+
stub_experiments(example: :my_variant, example2: :control)
|
452
|
+
|
453
|
+
experiment(:example) do |e|
|
454
|
+
e.enabled? # => true
|
455
|
+
e.variant.name # => 'my_variant'
|
456
|
+
end
|
457
|
+
|
458
|
+
experiment(:example2) do |e|
|
459
|
+
e.enabled? # => true
|
460
|
+
e.variant.name # => 'control'
|
461
|
+
end
|
462
|
+
```
|
463
|
+
|
464
|
+
### Exclusion and segmentation matchers
|
465
|
+
|
466
|
+
You can also easily test the exclusion and segmentation matchers.
|
467
|
+
|
468
|
+
```ruby
|
469
|
+
class ExampleExperiment < ApplicationExperiment
|
470
|
+
exclude { context.actor.first_name == 'Richard' }
|
471
|
+
segment(variant: :candidate) { context.actor.username == 'jejacks0n' }
|
472
|
+
end
|
473
|
+
|
474
|
+
excluded = double(username: 'rdiggitty', first_name: 'Richard')
|
475
|
+
segmented = double(username: 'jejacks0n', first_name: 'Jeremy')
|
476
|
+
|
477
|
+
# exclude matcher
|
478
|
+
expect(experiment(:example)).to exclude(actor: excluded)
|
479
|
+
expect(experiment(:example)).not_to exclude(actor: segmented)
|
480
|
+
|
481
|
+
# segment matcher
|
482
|
+
expect(experiment(:example)).to segment(actor: segmented).into(:candidate)
|
483
|
+
expect(experiment(:example)).not_to segment(actor: excluded)
|
484
|
+
```
|
485
|
+
|
486
|
+
### Tracking matcher
|
487
|
+
|
488
|
+
Tracking events is a major aspect of experimentation, and because of this we try to provide a flexible way to ensure your tracking calls are covered.
|
489
|
+
|
490
|
+
You can do this on the instance level or at an "any instance" level. At an instance level this is pretty straight forward:
|
491
|
+
|
492
|
+
```ruby
|
493
|
+
subject = experiment(:example)
|
494
|
+
|
495
|
+
expect(subject).to track(:my_event)
|
496
|
+
|
497
|
+
subject.track(:my_event)
|
498
|
+
```
|
499
|
+
|
500
|
+
You can use the `on_any_instance` chain method to specify that it could happen on any instance of the experiment. This can be useful if you're calling `experiment(:example).track` downstream:
|
501
|
+
|
502
|
+
```ruby
|
503
|
+
expect(experiment(:example)).to track(:my_event).on_any_instance
|
504
|
+
|
505
|
+
experiment(:example).track(:my_event)
|
506
|
+
```
|
507
|
+
|
508
|
+
And here's a full example of the methods that can be chained onto the `track` matcher:
|
509
|
+
|
510
|
+
```ruby
|
511
|
+
expect(experiment(:example)).to track(:my_event, value: 1, property: '_property_')
|
512
|
+
.on_any_instance
|
513
|
+
.with_context(foo: :bar)
|
514
|
+
.for(:variant_name)
|
515
|
+
|
516
|
+
experiment(:example, :variant_name, foo: :bar).track(:my_event, value: 1, property: '_property_')
|
517
|
+
```
|
518
|
+
|
404
519
|
## Tracking, anonymity and GDPR
|
405
520
|
|
406
521
|
We generally try not to track things like user identifying values in our experimentation. What we can and do track is the "experiment experience" (a.k.a. the context key).
|
@@ -426,7 +541,7 @@ https://gitlab.com/gitlab-org/gitlab-experiment. This project is intended to be
|
|
426
541
|
safe, welcoming space for collaboration, and contributors are expected to adhere
|
427
542
|
to the [Contributor Covenant](http://contributor-covenant.org) code of conduct.
|
428
543
|
|
429
|
-
## Release
|
544
|
+
## Release process
|
430
545
|
|
431
546
|
Please refer to the [Release Process](docs/release_process.md).
|
432
547
|
|
@@ -435,7 +550,7 @@ Please refer to the [Release Process](docs/release_process.md).
|
|
435
550
|
The gem is available as open source under the terms of the
|
436
551
|
[MIT License](http://opensource.org/licenses/MIT).
|
437
552
|
|
438
|
-
## Code of
|
553
|
+
## Code of conduct
|
439
554
|
|
440
555
|
Everyone interacting in the `Gitlab::Experiment` project’s codebases, issue trackers,
|
441
556
|
chat rooms and mailing lists is expected to follow the
|
@@ -22,20 +22,23 @@ Gitlab::Experiment.configure do |config|
|
|
22
22
|
# `['www.gitlab.com', '.gitlab.com']`.
|
23
23
|
config.cookie_domain = :all
|
24
24
|
|
25
|
-
#
|
25
|
+
# The default rollout strategy that works for single and multi-variants.
|
26
26
|
#
|
27
|
-
#
|
28
|
-
#
|
27
|
+
# You can provide your own rollout strategies and override them per
|
28
|
+
# experiment.
|
29
|
+
#
|
30
|
+
# Examples include:
|
31
|
+
# Rollout::First, Rollout::Random, Rollout::RoundRobin
|
32
|
+
config.default_rollout = Gitlab::Experiment::Rollout::First
|
33
|
+
|
34
|
+
# Logic this project uses to determine inclusion in a given experiment.
|
35
|
+
#
|
36
|
+
# Expected to return a boolean value.
|
29
37
|
#
|
30
38
|
# This block is executed within the scope of the experiment and so can access
|
31
39
|
# experiment methods, like `name`, `context`, and `signature`.
|
32
|
-
config.
|
33
|
-
|
34
|
-
requested_variant
|
35
|
-
|
36
|
-
# Run the candidate, unless a variant was requested, with a fallback:
|
37
|
-
#
|
38
|
-
# requested_variant || variant_names.first || nil
|
40
|
+
config.inclusion_resolver = lambda do |requested_variant|
|
41
|
+
false
|
39
42
|
end
|
40
43
|
|
41
44
|
# Tracking behavior can be implemented to link an event to an experiment.
|
data/lib/gitlab/experiment.rb
CHANGED
@@ -2,11 +2,16 @@
|
|
2
2
|
|
3
3
|
require 'scientist'
|
4
4
|
require 'active_support/callbacks'
|
5
|
+
require 'active_support/cache'
|
6
|
+
require 'active_support/concern'
|
5
7
|
require 'active_support/core_ext/object/blank'
|
6
8
|
require 'active_support/core_ext/string/inflections'
|
9
|
+
require 'active_support/core_ext/module/delegation'
|
7
10
|
|
8
|
-
require 'gitlab/experiment/
|
11
|
+
require 'gitlab/experiment/base_interface'
|
12
|
+
require 'gitlab/experiment/cache'
|
9
13
|
require 'gitlab/experiment/callbacks'
|
14
|
+
require 'gitlab/experiment/rollout'
|
10
15
|
require 'gitlab/experiment/configuration'
|
11
16
|
require 'gitlab/experiment/cookies'
|
12
17
|
require 'gitlab/experiment/context'
|
@@ -17,58 +22,44 @@ require 'gitlab/experiment/engine' if defined?(Rails::Engine)
|
|
17
22
|
|
18
23
|
module Gitlab
|
19
24
|
class Experiment
|
20
|
-
include
|
21
|
-
include
|
25
|
+
include BaseInterface
|
26
|
+
include Cache
|
22
27
|
include Callbacks
|
23
28
|
|
24
29
|
class << self
|
25
|
-
def
|
26
|
-
|
27
|
-
end
|
28
|
-
|
29
|
-
def run(name = nil, variant_name = nil, **context, &block)
|
30
|
-
raise ArgumentError, 'name is required' if name.nil? && base?
|
30
|
+
def default_rollout(rollout = nil)
|
31
|
+
return @rollout ||= Configuration.default_rollout if rollout.blank?
|
31
32
|
|
32
|
-
|
33
|
-
return instance unless block
|
34
|
-
|
35
|
-
instance.context.frozen? ? instance.run : instance.tap(&:run)
|
33
|
+
@rollout = Rollout.resolve(rollout)
|
36
34
|
end
|
37
35
|
|
38
|
-
def
|
39
|
-
|
40
|
-
|
41
|
-
|
42
|
-
end
|
43
|
-
|
44
|
-
def base?
|
45
|
-
self == Gitlab::Experiment || name == Configuration.base_class
|
36
|
+
def exclude(*filter_list, **options, &block)
|
37
|
+
build_callback(:exclusion_check, filter_list.unshift(block), **options) do |target, callback|
|
38
|
+
throw(:abort) if target.instance_variable_get(:@excluded) || callback.call(target, nil) == true
|
39
|
+
end
|
46
40
|
end
|
47
41
|
|
48
|
-
|
49
|
-
|
50
|
-
|
51
|
-
|
52
|
-
|
53
|
-
experiment_name(name).classify.safe_constantize || Configuration.base_class.constantize
|
42
|
+
def segment(*filter_list, variant:, **options, &block)
|
43
|
+
build_callback(:segmentation_check, filter_list.unshift(block), **options) do |target, callback|
|
44
|
+
target.variant(variant) if target.instance_variable_get(:@variant_name).nil? && callback.call(target, nil)
|
45
|
+
end
|
54
46
|
end
|
55
47
|
end
|
56
48
|
|
57
|
-
def
|
58
|
-
|
59
|
-
|
60
|
-
@name = self.class.experiment_name(name, suffix: false)
|
61
|
-
@context = Context.new(self, **context)
|
62
|
-
@variant_name = cache_variant(variant_name) { nil } if variant_name.present?
|
63
|
-
|
64
|
-
compare { false }
|
49
|
+
def name
|
50
|
+
[Configuration.name_prefix, @name].compact.join('_')
|
51
|
+
end
|
65
52
|
|
66
|
-
|
53
|
+
def control(&block)
|
54
|
+
candidate(:control, &block)
|
67
55
|
end
|
56
|
+
alias_method :use, :control
|
68
57
|
|
69
|
-
def
|
70
|
-
|
58
|
+
def candidate(name = nil, &block)
|
59
|
+
name = (name || :candidate).to_s
|
60
|
+
behaviors[name] = block
|
71
61
|
end
|
62
|
+
alias_method :try, :candidate
|
72
63
|
|
73
64
|
def context(value = nil)
|
74
65
|
return @context if value.blank?
|
@@ -78,33 +69,36 @@ module Gitlab
|
|
78
69
|
end
|
79
70
|
|
80
71
|
def variant(value = nil)
|
81
|
-
if value.
|
82
|
-
|
83
|
-
end
|
84
|
-
|
85
|
-
@variant_name = value unless value.blank?
|
72
|
+
@variant_name = cache_variant(value) if value.present?
|
73
|
+
return Variant.new(name: (@variant_name || :unresolved).to_s) if @variant_name || @resolving_variant
|
86
74
|
|
87
75
|
if enabled?
|
88
|
-
@variant_name ||= :control if excluded?
|
89
|
-
|
90
76
|
@resolving_variant = true
|
91
|
-
|
92
|
-
|
93
|
-
|
77
|
+
@variant_name ||= :control if excluded?
|
78
|
+
result = cache_variant(@variant_name) { resolve_variant_name }
|
79
|
+
@variant_name = result.to_sym if result.present?
|
94
80
|
end
|
95
81
|
|
96
|
-
|
82
|
+
run_callbacks(segmentation_callback_chain) do
|
83
|
+
@variant_name ||= :control
|
84
|
+
Variant.new(name: @variant_name.to_s)
|
85
|
+
end
|
97
86
|
ensure
|
98
87
|
@resolving_variant = false
|
99
88
|
end
|
100
89
|
|
90
|
+
def rollout(rollout = nil)
|
91
|
+
return @rollout ||= self.class.default_rollout if rollout.blank?
|
92
|
+
|
93
|
+
@rollout = Rollout.resolve(rollout)
|
94
|
+
end
|
95
|
+
|
96
|
+
def exclude!
|
97
|
+
@excluded = true
|
98
|
+
end
|
99
|
+
|
101
100
|
def run(variant_name = nil)
|
102
|
-
@result ||=
|
103
|
-
variant_name = variant(variant_name).name
|
104
|
-
run_callbacks(run_with_segmenting? ? :segmented_run : :unsegmented_run) do
|
105
|
-
super(@variant_name ||= variant_name)
|
106
|
-
end
|
107
|
-
end
|
101
|
+
@result ||= super(variant(variant_name).name)
|
108
102
|
end
|
109
103
|
|
110
104
|
def publish(result)
|
@@ -117,75 +111,42 @@ module Gitlab
|
|
117
111
|
instance_exec(action, event_args, &Configuration.tracking_behavior)
|
118
112
|
end
|
119
113
|
|
120
|
-
def
|
121
|
-
|
114
|
+
def enabled?
|
115
|
+
true
|
122
116
|
end
|
123
117
|
|
124
|
-
def
|
125
|
-
@
|
126
|
-
end
|
118
|
+
def excluded?
|
119
|
+
return @excluded if defined?(@excluded)
|
127
120
|
|
128
|
-
|
129
|
-
|
130
|
-
next unless name.end_with?('_behavior')
|
121
|
+
@excluded = !run_callbacks(:exclusion_check) { :not_excluded }
|
122
|
+
end
|
131
123
|
|
132
|
-
|
133
|
-
|
134
|
-
end
|
124
|
+
def experiment_group?
|
125
|
+
instance_exec(@variant_name, &Configuration.inclusion_resolver)
|
135
126
|
end
|
136
127
|
|
137
|
-
def
|
138
|
-
|
139
|
-
behaviors[name] = block
|
128
|
+
def should_track?
|
129
|
+
enabled? && @context.trackable? && !excluded?
|
140
130
|
end
|
141
131
|
|
142
132
|
def signature
|
143
133
|
{ variant: variant.name, experiment: name }.merge(context.signature)
|
144
134
|
end
|
145
135
|
|
146
|
-
def id
|
147
|
-
"#{name}:#{key_for(context.value)}"
|
148
|
-
end
|
149
|
-
alias_method :session_id, :id
|
150
|
-
|
151
|
-
def flipper_id
|
152
|
-
"Experiment;#{id}"
|
153
|
-
end
|
154
|
-
|
155
|
-
def enabled?
|
156
|
-
true
|
157
|
-
end
|
158
|
-
|
159
|
-
def excluded?
|
160
|
-
@excluded ||= !@context.trackable? || # adhere to DNT headers
|
161
|
-
!run_callbacks(:exclusion_check) { :not_excluded } # didn't pass exclusion check
|
162
|
-
end
|
163
|
-
|
164
|
-
def should_track?
|
165
|
-
enabled? && !excluded?
|
166
|
-
end
|
167
|
-
|
168
136
|
def key_for(hash)
|
169
137
|
instance_exec(hash, &Configuration.context_hash_strategy)
|
170
138
|
end
|
171
139
|
|
172
140
|
protected
|
173
141
|
|
174
|
-
def
|
175
|
-
|
176
|
-
end
|
142
|
+
def segmentation_callback_chain
|
143
|
+
return :segmentation_check if @variant_name.nil? && enabled? && !excluded?
|
177
144
|
|
178
|
-
|
179
|
-
!@variant_name.nil?
|
145
|
+
:unsegmented
|
180
146
|
end
|
181
147
|
|
182
148
|
def resolve_variant_name
|
183
|
-
|
184
|
-
end
|
185
|
-
|
186
|
-
def generate_result(variant_name)
|
187
|
-
observation = Scientist::Observation.new(variant_name, self, &behaviors[variant_name])
|
188
|
-
Scientist::Result.new(self, [observation], observation)
|
149
|
+
rollout.new(self).execute if experiment_group?
|
189
150
|
end
|
190
151
|
end
|
191
152
|
end
|
@@ -0,0 +1,81 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
module Gitlab
|
4
|
+
class Experiment
|
5
|
+
module BaseInterface
|
6
|
+
extend ActiveSupport::Concern
|
7
|
+
include Scientist::Experiment
|
8
|
+
|
9
|
+
class_methods do
|
10
|
+
def configure
|
11
|
+
yield Configuration
|
12
|
+
end
|
13
|
+
|
14
|
+
def experiment_name(name = nil, suffix: true, suffix_word: 'experiment')
|
15
|
+
name = (name.presence || self.name).to_s.underscore.sub(%r{(?<char>[_/]|)#{suffix_word}$}, '')
|
16
|
+
name = "#{name}#{Regexp.last_match(:char) || '_'}#{suffix_word}"
|
17
|
+
suffix ? name : name.sub(/_#{suffix_word}$/, '')
|
18
|
+
end
|
19
|
+
|
20
|
+
def base?
|
21
|
+
self == Gitlab::Experiment || name == Configuration.base_class
|
22
|
+
end
|
23
|
+
|
24
|
+
def constantize(name = nil)
|
25
|
+
return self if name.nil?
|
26
|
+
|
27
|
+
experiment_name(name).classify.safe_constantize || Configuration.base_class.constantize
|
28
|
+
end
|
29
|
+
end
|
30
|
+
|
31
|
+
def initialize(name = nil, variant_name = nil, **context)
|
32
|
+
raise ArgumentError, 'name is required' if name.blank? && self.class.base?
|
33
|
+
|
34
|
+
@name = self.class.experiment_name(name, suffix: false)
|
35
|
+
@context = Context.new(self, **context)
|
36
|
+
@variant_name = cache_variant(variant_name) { nil } if variant_name.present?
|
37
|
+
|
38
|
+
compare { false }
|
39
|
+
|
40
|
+
yield self if block_given?
|
41
|
+
end
|
42
|
+
|
43
|
+
def inspect
|
44
|
+
"#<#{self.class.name || 'AnonymousClass'}:#{format('0x%016X', __id__)} @name=#{name} @context=#{context.value}>"
|
45
|
+
end
|
46
|
+
|
47
|
+
def id
|
48
|
+
"#{name}:#{key_for(context.value)}"
|
49
|
+
end
|
50
|
+
alias_method :session_id, :id
|
51
|
+
|
52
|
+
def flipper_id
|
53
|
+
"Experiment;#{id}"
|
54
|
+
end
|
55
|
+
|
56
|
+
def variant_names
|
57
|
+
@variant_names ||= behaviors.keys.map(&:to_sym) - [:control]
|
58
|
+
end
|
59
|
+
|
60
|
+
def behaviors
|
61
|
+
@behaviors ||= public_methods.each_with_object(super) do |name, behaviors|
|
62
|
+
next unless name.end_with?('_behavior')
|
63
|
+
|
64
|
+
behavior_name = name.to_s.sub(/_behavior$/, '')
|
65
|
+
behaviors[behavior_name] ||= -> { send(name) } # rubocop:disable GitlabSecurity/PublicSend
|
66
|
+
end
|
67
|
+
end
|
68
|
+
|
69
|
+
protected
|
70
|
+
|
71
|
+
def raise_on_mismatches?
|
72
|
+
false
|
73
|
+
end
|
74
|
+
|
75
|
+
def generate_result(variant_name)
|
76
|
+
observation = Scientist::Observation.new(variant_name, self, &behaviors[variant_name])
|
77
|
+
Scientist::Result.new(self, [observation], observation)
|
78
|
+
end
|
79
|
+
end
|
80
|
+
end
|
81
|
+
end
|
@@ -0,0 +1,78 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
require 'active_support/cache'
|
4
|
+
|
5
|
+
module Gitlab
|
6
|
+
class Experiment
|
7
|
+
module Cache
|
8
|
+
autoload :RedisHashStore, 'gitlab/experiment/cache/redis_hash_store.rb'
|
9
|
+
|
10
|
+
class Interface
|
11
|
+
attr_reader :store, :key
|
12
|
+
|
13
|
+
def initialize(experiment, store)
|
14
|
+
@experiment = experiment
|
15
|
+
@store = store
|
16
|
+
@key = experiment.cache_key
|
17
|
+
end
|
18
|
+
|
19
|
+
def read
|
20
|
+
store.read(key)
|
21
|
+
end
|
22
|
+
|
23
|
+
def write(value = nil)
|
24
|
+
store.write(key, value || @experiment.variant.name)
|
25
|
+
end
|
26
|
+
|
27
|
+
def delete
|
28
|
+
store.delete(key)
|
29
|
+
end
|
30
|
+
|
31
|
+
def attr_get(name)
|
32
|
+
store.read(@experiment.cache_key(name, suffix: :attrs))
|
33
|
+
end
|
34
|
+
|
35
|
+
def attr_set(name, value)
|
36
|
+
store.write(@experiment.cache_key(name, suffix: :attrs), value)
|
37
|
+
end
|
38
|
+
|
39
|
+
def attr_inc(name, amount = 1)
|
40
|
+
store.increment(@experiment.cache_key(name, suffix: :attrs), amount)
|
41
|
+
end
|
42
|
+
end
|
43
|
+
|
44
|
+
def cache
|
45
|
+
@cache ||= Interface.new(self, Configuration.cache)
|
46
|
+
end
|
47
|
+
|
48
|
+
def cache_variant(specified = nil, &block)
|
49
|
+
return (specified.presence || yield) unless cache.store
|
50
|
+
|
51
|
+
result = migrated_cache_fetch(cache.store, &block)
|
52
|
+
return result unless specified.present?
|
53
|
+
|
54
|
+
cache.write(specified) if result != specified
|
55
|
+
specified
|
56
|
+
end
|
57
|
+
|
58
|
+
def cache_key(key = nil, suffix: nil)
|
59
|
+
"#{[name, suffix].compact.join('_')}:#{key || context.signature[:key]}"
|
60
|
+
end
|
61
|
+
|
62
|
+
private
|
63
|
+
|
64
|
+
def migrated_cache_fetch(store, &block)
|
65
|
+
migrations = context.signature[:migration_keys]&.map { |key| cache_key(key) } || []
|
66
|
+
migrations.find do |old_key|
|
67
|
+
next unless (value = store.read(old_key))
|
68
|
+
|
69
|
+
store.write(cache_key, value)
|
70
|
+
store.delete(old_key)
|
71
|
+
return value
|
72
|
+
end
|
73
|
+
|
74
|
+
store.fetch(cache_key, &block)
|
75
|
+
end
|
76
|
+
end
|
77
|
+
end
|
78
|
+
end
|
@@ -0,0 +1,68 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
require 'active_support/notifications'
|
4
|
+
|
5
|
+
# This cache strategy is an implementation on top of the redis hash data type,
|
6
|
+
# that also adheres to the ActiveSupport::Cache::Store interface. It's a good
|
7
|
+
# example of how to build a custom caching strategy for Gitlab::Experiment, and
|
8
|
+
# is intended to be a long lived cache -- until the experiment is cleaned up.
|
9
|
+
#
|
10
|
+
# The data structure:
|
11
|
+
# key: experiment.name
|
12
|
+
# fields: context key => variant name
|
13
|
+
#
|
14
|
+
# Gitlab::Experiment::Configuration.cache = Gitlab::Experiment::Cache::RedisHashStore.new(
|
15
|
+
# pool: -> { Gitlab::Redis::SharedState.with { |redis| yield redis } }
|
16
|
+
# )
|
17
|
+
module Gitlab
|
18
|
+
class Experiment
|
19
|
+
module Cache
|
20
|
+
class RedisHashStore < ActiveSupport::Cache::Store
|
21
|
+
# Clears the entire cache for a given experiment. Be careful with this
|
22
|
+
# since it would reset all resolved variants for the entire experiment.
|
23
|
+
def clear(key:)
|
24
|
+
key = hkey(key)[0] # extract only the first part of the key
|
25
|
+
pool do |redis|
|
26
|
+
case redis.type(key)
|
27
|
+
when 'hash', 'none'
|
28
|
+
redis.del(key) # delete the primary experiment key
|
29
|
+
redis.del("#{key}_attrs") # delete the experiment attributes key
|
30
|
+
else raise ArgumentError, 'invalid call to clear a non-hash cache key'
|
31
|
+
end
|
32
|
+
end
|
33
|
+
end
|
34
|
+
|
35
|
+
def increment(key, amount = 1)
|
36
|
+
pool { |redis| redis.hincrby(*hkey(key), amount) }
|
37
|
+
end
|
38
|
+
|
39
|
+
private
|
40
|
+
|
41
|
+
def pool(&block)
|
42
|
+
raise ArgumentError, 'missing block' unless block.present?
|
43
|
+
|
44
|
+
@options[:pool].call(&block)
|
45
|
+
end
|
46
|
+
|
47
|
+
def hkey(key)
|
48
|
+
key.to_s.split(':') # this assumes the default strategy in gitlab-experiment
|
49
|
+
end
|
50
|
+
|
51
|
+
def read_entry(key, **options)
|
52
|
+
value = pool { |redis| redis.hget(*hkey(key)) }
|
53
|
+
value.nil? ? nil : ActiveSupport::Cache::Entry.new(value)
|
54
|
+
end
|
55
|
+
|
56
|
+
def write_entry(key, entry, **options)
|
57
|
+
return false if entry.value.blank? # don't cache any empty values
|
58
|
+
|
59
|
+
pool { |redis| redis.hset(*hkey(key), entry.value) }
|
60
|
+
end
|
61
|
+
|
62
|
+
def delete_entry(key, **options)
|
63
|
+
pool { |redis| redis.hdel(*hkey(key)) }
|
64
|
+
end
|
65
|
+
end
|
66
|
+
end
|
67
|
+
end
|
68
|
+
end
|
@@ -1,5 +1,7 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
2
|
|
3
|
+
require 'active_support/callbacks'
|
4
|
+
|
3
5
|
module Gitlab
|
4
6
|
class Experiment
|
5
7
|
module Callbacks
|
@@ -7,34 +9,23 @@ module Gitlab
|
|
7
9
|
include ActiveSupport::Callbacks
|
8
10
|
|
9
11
|
included do
|
10
|
-
define_callbacks(:
|
11
|
-
define_callbacks(:
|
12
|
+
define_callbacks(:unsegmented)
|
13
|
+
define_callbacks(:segmentation_check)
|
12
14
|
define_callbacks(:exclusion_check, skip_after_callbacks_if_terminated: true)
|
13
15
|
end
|
14
16
|
|
15
17
|
class_methods do
|
16
|
-
|
17
|
-
filters = filter_list.unshift(block).compact.map do |filter|
|
18
|
-
result_lambda = ActiveSupport::Callbacks::CallTemplate.build(filter, self).make_lambda
|
19
|
-
lambda do |target|
|
20
|
-
throw(:abort) if target.instance_variable_get(:'@excluded') || result_lambda.call(target, nil) == true
|
21
|
-
end
|
22
|
-
end
|
23
|
-
|
24
|
-
raise ArgumentError, 'no filters provided' if filters.empty?
|
25
|
-
|
26
|
-
set_callback(:exclusion_check, :before, *filters, options)
|
27
|
-
end
|
18
|
+
private
|
28
19
|
|
29
|
-
def
|
30
|
-
filters =
|
20
|
+
def build_callback(chain, filters, **options)
|
21
|
+
filters = filters.compact.map do |filter|
|
31
22
|
result_lambda = ActiveSupport::Callbacks::CallTemplate.build(filter, self).make_lambda
|
32
|
-
->(target) {
|
23
|
+
->(target) { yield(target, result_lambda) }
|
33
24
|
end
|
34
25
|
|
35
26
|
raise ArgumentError, 'no filters provided' if filters.empty?
|
36
27
|
|
37
|
-
set_callback(
|
28
|
+
set_callback(chain, *filters, **options)
|
38
29
|
end
|
39
30
|
end
|
40
31
|
end
|
@@ -4,6 +4,8 @@ require 'singleton'
|
|
4
4
|
require 'logger'
|
5
5
|
require 'digest'
|
6
6
|
|
7
|
+
require 'active_support/deprecation'
|
8
|
+
|
7
9
|
module Gitlab
|
8
10
|
class Experiment
|
9
11
|
class Configuration
|
@@ -24,10 +26,13 @@ module Gitlab
|
|
24
26
|
# The domain to use on cookies.
|
25
27
|
@cookie_domain = :all
|
26
28
|
|
27
|
-
#
|
28
|
-
|
29
|
-
|
30
|
-
|
29
|
+
# The default rollout strategy that works for single and multi-variants.
|
30
|
+
@default_rollout = Rollout::First
|
31
|
+
|
32
|
+
# Logic this project uses to determine inclusion in a given experiment.
|
33
|
+
# Expected to return a boolean value.
|
34
|
+
@inclusion_resolver = lambda do |requested_variant|
|
35
|
+
false
|
31
36
|
end
|
32
37
|
|
33
38
|
# Tracking behavior can be implemented to link an event to an experiment.
|
@@ -47,13 +52,27 @@ module Gitlab
|
|
47
52
|
end
|
48
53
|
|
49
54
|
class << self
|
55
|
+
# TODO: Added deprecation in release 0.5.0
|
56
|
+
def variant_resolver
|
57
|
+
ActiveSupport::Deprecation.warn('variant_resolver is deprecated, instead use `inclusion_resolver` with a' \
|
58
|
+
'block that returns a boolean.')
|
59
|
+
@inclusion_resolver
|
60
|
+
end
|
61
|
+
|
62
|
+
def variant_resolver=(block)
|
63
|
+
ActiveSupport::Deprecation.warn('variant_resolver is deprecated, instead use `inclusion_resolver` with a' \
|
64
|
+
'block that returns a boolean.')
|
65
|
+
@inclusion_resolver = block
|
66
|
+
end
|
67
|
+
|
50
68
|
attr_accessor(
|
51
69
|
:name_prefix,
|
52
70
|
:logger,
|
53
71
|
:base_class,
|
54
72
|
:cache,
|
55
73
|
:cookie_domain,
|
56
|
-
:
|
74
|
+
:default_rollout,
|
75
|
+
:inclusion_resolver,
|
57
76
|
:tracking_behavior,
|
58
77
|
:publishing_behavior,
|
59
78
|
:context_hash_strategy
|
@@ -4,8 +4,17 @@ module Gitlab
|
|
4
4
|
class Experiment
|
5
5
|
module Dsl
|
6
6
|
def experiment(name, variant_name = nil, **context, &block)
|
7
|
+
raise ArgumentError, 'name is required' if name.nil?
|
8
|
+
|
7
9
|
context[:request] ||= request if respond_to?(:request)
|
8
|
-
|
10
|
+
|
11
|
+
base = Configuration.base_class.constantize
|
12
|
+
klass = base.constantize(name) || base
|
13
|
+
|
14
|
+
instance = klass.new(name, variant_name, **context, &block)
|
15
|
+
return instance unless block
|
16
|
+
|
17
|
+
instance.context.frozen? ? instance.run : instance.tap(&:run)
|
9
18
|
end
|
10
19
|
end
|
11
20
|
end
|
@@ -4,13 +4,18 @@ module Gitlab
|
|
4
4
|
class Experiment
|
5
5
|
class Engine < ::Rails::Engine
|
6
6
|
def self.include_dsl
|
7
|
-
ActionController
|
8
|
-
|
9
|
-
|
7
|
+
if defined?(ActionController)
|
8
|
+
ActionController::Base.include(Dsl)
|
9
|
+
ActionController::Base.helper_method(:experiment)
|
10
|
+
end
|
11
|
+
|
12
|
+
return unless defined?(ActionMailer)
|
10
13
|
|
11
|
-
|
12
|
-
|
14
|
+
ActionMailer::Base.include(Dsl)
|
15
|
+
ActionMailer::Base.helper_method(:experiment)
|
13
16
|
end
|
17
|
+
|
18
|
+
config.after_initialize { include_dsl }
|
14
19
|
end
|
15
20
|
end
|
16
21
|
end
|
@@ -0,0 +1,27 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
module Gitlab
|
4
|
+
class Experiment
|
5
|
+
module Rollout
|
6
|
+
autoload :First, 'gitlab/experiment/rollout/first.rb' # default strategy
|
7
|
+
autoload :Random, 'gitlab/experiment/rollout/random.rb'
|
8
|
+
autoload :RoundRobin, 'gitlab/experiment/rollout/round_robin.rb'
|
9
|
+
|
10
|
+
def self.resolve(klass)
|
11
|
+
return "#{name}::#{klass.to_s.classify}".constantize if klass.is_a?(Symbol) || klass.is_a?(String)
|
12
|
+
|
13
|
+
klass
|
14
|
+
end
|
15
|
+
|
16
|
+
class Base
|
17
|
+
attr_reader :experiment
|
18
|
+
|
19
|
+
delegate :variant_names, :cache, to: :experiment
|
20
|
+
|
21
|
+
def initialize(experiment)
|
22
|
+
@experiment = experiment
|
23
|
+
end
|
24
|
+
end
|
25
|
+
end
|
26
|
+
end
|
27
|
+
end
|
@@ -0,0 +1,16 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
module Gitlab
|
4
|
+
class Experiment
|
5
|
+
module Rollout
|
6
|
+
class First < Base
|
7
|
+
# This rollout strategy just picks the first variant name. It's the
|
8
|
+
# default resolver as it assumes a single variant. You should consider
|
9
|
+
# using a more advanced rollout if you have multiple variants.
|
10
|
+
def execute
|
11
|
+
variant_names.first
|
12
|
+
end
|
13
|
+
end
|
14
|
+
end
|
15
|
+
end
|
16
|
+
end
|
@@ -0,0 +1,15 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
module Gitlab
|
4
|
+
class Experiment
|
5
|
+
module Rollout
|
6
|
+
class Random < Base
|
7
|
+
# Pick a random variant if we're in the experiment group. It doesn't
|
8
|
+
# take into account small sample sizes but is useful and performant.
|
9
|
+
def execute
|
10
|
+
variant_names.sample
|
11
|
+
end
|
12
|
+
end
|
13
|
+
end
|
14
|
+
end
|
15
|
+
end
|
@@ -0,0 +1,23 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
module Gitlab
|
4
|
+
class Experiment
|
5
|
+
module Rollout
|
6
|
+
class RoundRobin < Base
|
7
|
+
KEY_NAME = :last_round_robin_variant
|
8
|
+
|
9
|
+
# Requires a cache to be configured.
|
10
|
+
#
|
11
|
+
# Keeps track of the number of assignments into the experiment group,
|
12
|
+
# and uses this to rotate "round robin" style through the variants
|
13
|
+
# that are defined.
|
14
|
+
#
|
15
|
+
# Relatively performant, but requires a cache, and is dependent on the
|
16
|
+
# performance of that cache store.
|
17
|
+
def execute
|
18
|
+
variant_names[(cache.attr_inc(KEY_NAME) - 1) % variant_names.size]
|
19
|
+
end
|
20
|
+
end
|
21
|
+
end
|
22
|
+
end
|
23
|
+
end
|
@@ -6,14 +6,19 @@ module Gitlab
|
|
6
6
|
def stub_experiments(experiments)
|
7
7
|
experiments.each do |name, variant|
|
8
8
|
variant = :control if variant == false
|
9
|
-
raise ArgumentError, 'variant must be a symbol or false' unless variant.is_a?(Symbol)
|
10
9
|
|
11
|
-
|
10
|
+
base = Configuration.base_class.constantize
|
11
|
+
klass = base.constantize(name) || base
|
12
12
|
|
13
13
|
# We have to use this high level any_instance behavior as there's
|
14
14
|
# not an alternative that allows multiple wrappings of `new`.
|
15
15
|
allow_any_instance_of(klass).to receive(:enabled?).and_return(true)
|
16
|
-
|
16
|
+
|
17
|
+
if variant == true # passing true allows the rollout to do its job
|
18
|
+
allow_any_instance_of(klass).to receive(:experiment_group?).and_return(true)
|
19
|
+
else
|
20
|
+
allow_any_instance_of(klass).to receive(:resolve_variant_name).and_return(variant.to_s)
|
21
|
+
end
|
17
22
|
end
|
18
23
|
end
|
19
24
|
|
@@ -86,7 +91,7 @@ module Gitlab
|
|
86
91
|
experiment.context(context)
|
87
92
|
|
88
93
|
experiment.instance_variable_set(ivar, nil)
|
89
|
-
experiment.run_callbacks(:
|
94
|
+
experiment.run_callbacks(:segmentation_check)
|
90
95
|
|
91
96
|
@actual = experiment.instance_variable_get(ivar)
|
92
97
|
@expected ? @actual.to_s == @expected.to_s : @actual.present?
|
@@ -147,14 +152,16 @@ module Gitlab
|
|
147
152
|
end
|
148
153
|
end
|
149
154
|
|
150
|
-
def receive_tracking_call_for(
|
151
|
-
receive(:track).with(*[
|
155
|
+
def receive_tracking_call_for(expected_event, *expected_event_args)
|
156
|
+
receive(:track).with(*[expected_event, *expected_event_args]).and_wrap_original do |track, event, *event_args|
|
157
|
+
track.call(event, *event_args) # call the original
|
158
|
+
|
152
159
|
if @expected_variant
|
153
|
-
expect(@experiment.variant.name).to eq(@expected_variant), failure_message(:variant,
|
160
|
+
expect(@experiment.variant.name).to eq(@expected_variant), failure_message(:variant, expected_event)
|
154
161
|
end
|
155
162
|
|
156
163
|
if @expected_context
|
157
|
-
expect(@experiment.context.value).to include(@expected_context), failure_message(:context,
|
164
|
+
expect(@experiment.context.value).to include(@expected_context), failure_message(:context, expected_event)
|
158
165
|
end
|
159
166
|
end
|
160
167
|
end
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: gitlab-experiment
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.
|
4
|
+
version: 0.5.1
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- GitLab
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2021-
|
11
|
+
date: 2021-03-17 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: activesupport
|
@@ -30,20 +30,20 @@ dependencies:
|
|
30
30
|
requirements:
|
31
31
|
- - "~>"
|
32
32
|
- !ruby/object:Gem::Version
|
33
|
-
version: '1.
|
33
|
+
version: '1.6'
|
34
34
|
- - ">="
|
35
35
|
- !ruby/object:Gem::Version
|
36
|
-
version: 1.
|
36
|
+
version: 1.6.0
|
37
37
|
type: :runtime
|
38
38
|
prerelease: false
|
39
39
|
version_requirements: !ruby/object:Gem::Requirement
|
40
40
|
requirements:
|
41
41
|
- - "~>"
|
42
42
|
- !ruby/object:Gem::Version
|
43
|
-
version: '1.
|
43
|
+
version: '1.6'
|
44
44
|
- - ">="
|
45
45
|
- !ruby/object:Gem::Version
|
46
|
-
version: 1.
|
46
|
+
version: 1.6.0
|
47
47
|
description:
|
48
48
|
email:
|
49
49
|
- gitlab_rubygems@gitlab.com
|
@@ -65,13 +65,19 @@ files:
|
|
65
65
|
- lib/generators/test_unit/experiment/experiment_generator.rb
|
66
66
|
- lib/generators/test_unit/experiment/templates/experiment_test.rb.tt
|
67
67
|
- lib/gitlab/experiment.rb
|
68
|
-
- lib/gitlab/experiment/
|
68
|
+
- lib/gitlab/experiment/base_interface.rb
|
69
|
+
- lib/gitlab/experiment/cache.rb
|
70
|
+
- lib/gitlab/experiment/cache/redis_hash_store.rb
|
69
71
|
- lib/gitlab/experiment/callbacks.rb
|
70
72
|
- lib/gitlab/experiment/configuration.rb
|
71
73
|
- lib/gitlab/experiment/context.rb
|
72
74
|
- lib/gitlab/experiment/cookies.rb
|
73
75
|
- lib/gitlab/experiment/dsl.rb
|
74
76
|
- lib/gitlab/experiment/engine.rb
|
77
|
+
- lib/gitlab/experiment/rollout.rb
|
78
|
+
- lib/gitlab/experiment/rollout/first.rb
|
79
|
+
- lib/gitlab/experiment/rollout/random.rb
|
80
|
+
- lib/gitlab/experiment/rollout/round_robin.rb
|
75
81
|
- lib/gitlab/experiment/rspec.rb
|
76
82
|
- lib/gitlab/experiment/variant.rb
|
77
83
|
- lib/gitlab/experiment/version.rb
|
@@ -1,39 +0,0 @@
|
|
1
|
-
# frozen_string_literal: true
|
2
|
-
|
3
|
-
module Gitlab
|
4
|
-
class Experiment
|
5
|
-
module Caching
|
6
|
-
def cache_variant(specified = nil, &block)
|
7
|
-
cache = Configuration.cache
|
8
|
-
return (specified.presence || yield) unless cache
|
9
|
-
|
10
|
-
key, migrations = cache_strategy
|
11
|
-
result = migrated_cache(cache, migrations || [], key) || cache.fetch(key, &block)
|
12
|
-
return result unless specified.present?
|
13
|
-
|
14
|
-
cache.write(cache_key, specified) if result != specified
|
15
|
-
specified
|
16
|
-
end
|
17
|
-
|
18
|
-
def cache_key(key = nil)
|
19
|
-
"#{name}:#{key || context.signature[:key]}"
|
20
|
-
end
|
21
|
-
|
22
|
-
private
|
23
|
-
|
24
|
-
def cache_strategy
|
25
|
-
[cache_key, context.signature[:migration_keys]&.map { |key| cache_key(key) }]
|
26
|
-
end
|
27
|
-
|
28
|
-
def migrated_cache(cache, migrations, new_key)
|
29
|
-
migrations.find do |old_key|
|
30
|
-
next unless (value = cache.read(old_key))
|
31
|
-
|
32
|
-
cache.write(new_key, value)
|
33
|
-
cache.delete(old_key)
|
34
|
-
break value
|
35
|
-
end
|
36
|
-
end
|
37
|
-
end
|
38
|
-
end
|
39
|
-
end
|