gitlab-experiment 1.2.0 → 1.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +1058 -282
- data/lib/gitlab/experiment/configuration.rb +10 -0
- data/lib/gitlab/experiment/force_assignment.rb +28 -0
- data/lib/gitlab/experiment/version.rb +1 -1
- data/lib/gitlab/experiment.rb +4 -0
- metadata +3 -2
data/README.md
CHANGED
|
@@ -1,31 +1,165 @@
|
|
|
1
|
-
GitLab Experiment
|
|
1
|
+
GitLab Experiment Platform
|
|
2
2
|
=================
|
|
3
3
|
|
|
4
4
|
<img alt="experiment" src="/uploads/60990b2dbf4c0406bbf8b7f998de2dea/experiment.png" align="right" width="40%">
|
|
5
5
|
|
|
6
|
-
|
|
6
|
+
**A comprehensive experimentation platform for building data-driven organizations**
|
|
7
7
|
|
|
8
|
-
|
|
8
|
+
GitLab Experiment is an enterprise-grade experimentation framework that enables teams to validate hypotheses, optimize
|
|
9
|
+
user experiences, and make evidence-based product decisions at scale.
|
|
10
|
+
Built on years of production experience at GitLab, this platform provides the foundation for a mature experimentation
|
|
11
|
+
culture across your entire organization.
|
|
9
12
|
|
|
10
|
-
|
|
13
|
+
At GitLab, we run experiments as A/B/n tests and review the data they generate.
|
|
14
|
+
From that data, we determine the best performing code path and promote it as the new default, or revert back to the
|
|
15
|
+
original code path.
|
|
16
|
+
You can read our [Experiment Guide](https://docs.gitlab.com/ee/development/experiment_guide/) to learn how we use this
|
|
17
|
+
gem internally at GitLab.
|
|
11
18
|
|
|
12
|
-
|
|
19
|
+
[[_TOC_]]
|
|
20
|
+
|
|
21
|
+
## Why GitLab Experiment?
|
|
22
|
+
|
|
23
|
+
### Built for Scale and Reliability
|
|
24
|
+
- **Production-tested** at GitLab scale with millions of users
|
|
25
|
+
- **Type-safe and testable** with comprehensive RSpec support
|
|
26
|
+
- **Framework agnostic** - works with Rails, Sinatra, or standalone Ruby applications
|
|
27
|
+
- **Redis-backed caching** for consistent user experiences across sessions
|
|
28
|
+
- **GDPR-compliant** with anonymous tracking and built-in DNT (Do Not Track) support
|
|
29
|
+
|
|
30
|
+
### Designed for Teams
|
|
31
|
+
- **Developer-friendly DSL** that reads like natural language
|
|
32
|
+
- **Organized experiment classes** that live alongside your application code
|
|
33
|
+
- **Built-in generators** for rapid experiment creation
|
|
34
|
+
- **Comprehensive testing support** with custom RSpec matchers
|
|
35
|
+
- **Rails integration** with automatic middleware mounting and view helpers
|
|
36
|
+
|
|
37
|
+
### Enterprise-Ready Features
|
|
38
|
+
- **Flexible rollout strategies** (percent-based, random, round-robin, or custom)
|
|
39
|
+
- **Advanced segmentation** to target specific user populations
|
|
40
|
+
- **Multi-variant testing** (A/B/n) with unlimited experimental paths
|
|
41
|
+
- **Progressive rollouts** with the `only_assigned` feature
|
|
42
|
+
- **Context migrations** for evolving experiments without losing data
|
|
43
|
+
- **Integration-ready** with existing feature flag systems (Flipper, Unleash, etc.)
|
|
44
|
+
|
|
45
|
+
<br clear="all">
|
|
46
|
+
|
|
47
|
+
## Use Cases Across Your Organization
|
|
48
|
+
|
|
49
|
+
### Product Teams: Optimize User Experiences
|
|
50
|
+
- **Onboarding flows**: Test different signup sequences to maximize activation
|
|
51
|
+
- **UI/UX changes**: Validate design decisions with real user behavior data
|
|
52
|
+
- **Feature rollouts**: Gradually release features to measure impact before full deployment
|
|
53
|
+
- **Pricing experiments**: Test different pricing strategies and messaging
|
|
54
|
+
|
|
55
|
+
### Growth Teams: Drive Conversion
|
|
56
|
+
- **Call-to-action optimization**: Test button colors, copy, and placement
|
|
57
|
+
- **Landing page variations**: Experiment with different value propositions
|
|
58
|
+
- **Email campaigns**: A/B test subject lines and content
|
|
59
|
+
- **Trial conversion**: Optimize paths from trial to paid subscriptions
|
|
60
|
+
|
|
61
|
+
### Engineering Teams: Safe Deployments
|
|
62
|
+
- **Performance optimizations**: Compare algorithm implementations under real load
|
|
63
|
+
- **Architecture changes**: Validate new code paths before full migration
|
|
64
|
+
- **API versions**: Run multiple API implementations side-by-side
|
|
65
|
+
- **Infrastructure experiments**: Test different caching or database strategies
|
|
66
|
+
|
|
67
|
+
### Data Science Teams: Recommendation Systems
|
|
68
|
+
- **Algorithm tuning**: Compare ML model variations in production
|
|
69
|
+
- **Personalization**: Test different recommendation strategies
|
|
70
|
+
- **Search ranking**: Optimize search results based on user engagement
|
|
71
|
+
- **Content discovery**: Experiment with different content surfaces
|
|
72
|
+
|
|
73
|
+
## Platform Capabilities
|
|
74
|
+
|
|
75
|
+
### Core Experimentation Features
|
|
76
|
+
|
|
77
|
+
**Multi-variant Testing (A/B/n)**
|
|
78
|
+
Run experiments with any number of variants, not just A/B tests. Perfect for testing multiple approaches simultaneously.
|
|
13
79
|
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
80
|
+
**Smart Segmentation**
|
|
81
|
+
Route specific user populations to predetermined variants based on business rules, ensuring consistent experiences for
|
|
82
|
+
targeted groups.
|
|
83
|
+
|
|
84
|
+
**Progressive Rollouts**
|
|
85
|
+
Use the `only_assigned` feature to show experimental features only to users already in the experiment, enabling
|
|
86
|
+
controlled expansion.
|
|
87
|
+
|
|
88
|
+
**Context Flexibility**
|
|
89
|
+
Experiments can be sticky to users, projects, organizations, or any combination - enabling complex scenarios beyond
|
|
90
|
+
user-centric testing.
|
|
91
|
+
|
|
92
|
+
**Anonymous Tracking**
|
|
93
|
+
Built-in privacy protection with anonymous context keys, automatic cookie migration, and GDPR compliance.
|
|
94
|
+
|
|
95
|
+
**Automatic Assignment Tracking**
|
|
96
|
+
Every experiment automatically tracks an `:assignment` event when it runs - zero configuration required. Combined with
|
|
97
|
+
the anonymous context key, this gives your data team a complete picture of variant distribution and funnel entry without
|
|
98
|
+
any additional instrumentation.
|
|
99
|
+
|
|
100
|
+
**Client-Side Integration**
|
|
101
|
+
Seamlessly extend experiments to the frontend with JavaScript integration, enabling full-stack experimentation.
|
|
102
|
+
|
|
103
|
+
**Inline and Class-Based APIs**
|
|
104
|
+
Define experiments inline with blocks for quick iterations, or use dedicated experiment classes for complex logic - or
|
|
105
|
+
combine both. Class-based experiments define default behaviors that can be overridden inline at any call site, giving
|
|
106
|
+
teams the flexibility to start simple and evolve without rewriting:
|
|
107
|
+
|
|
108
|
+
```ruby
|
|
109
|
+
experiment(:pill_color, actor: current_user) do |e|
|
|
110
|
+
e.control { '<strong>control</strong>' }
|
|
111
|
+
end
|
|
112
|
+
```
|
|
20
113
|
|
|
21
|
-
|
|
114
|
+
**Context as a Design Framework**
|
|
115
|
+
Context is the most important design decision in any experiment. It determines stickiness, cache behavior, and how
|
|
116
|
+
events are correlated. Choose per-user context for personalization experiments, per-project for infrastructure tests,
|
|
117
|
+
per-group for organizational rollouts, or combine dimensions for precision targeting. This flexibility enables
|
|
118
|
+
experimentation strategies that go far beyond simple user-centric A/B tests.
|
|
119
|
+
|
|
120
|
+
**Decoupled Assignment with Publish**
|
|
121
|
+
Surface experiment assignments to the client layer without executing server-side behavior using `publish`. This enables
|
|
122
|
+
frontend-only experiments, pre-assignment in `before_action` hooks, and scenarios where variant data needs to be
|
|
123
|
+
available across the stack without triggering server-side code paths:
|
|
124
|
+
|
|
125
|
+
```ruby
|
|
126
|
+
before_action -> { experiment(:pill_color, actor: current_user).publish }, only: [:show]
|
|
127
|
+
```
|
|
128
|
+
|
|
129
|
+
### Integration Ecosystem
|
|
130
|
+
|
|
131
|
+
**Feature Flag Integration**
|
|
132
|
+
Connect with existing feature flag systems like Flipper or Unleash through custom rollout strategies.
|
|
133
|
+
|
|
134
|
+
**Analytics Integration**
|
|
135
|
+
Flexible tracking callbacks integrate with any analytics platform - Snowplow, Amplitude, Mixpanel, or your data
|
|
136
|
+
warehouse.
|
|
137
|
+
|
|
138
|
+
**Monitoring and Observability**
|
|
139
|
+
Built-in logging and callbacks for integration with APM tools and monitoring systems.
|
|
140
|
+
|
|
141
|
+
**Email and Markdown**
|
|
142
|
+
Special middleware for tracking experiments in email links and static content.
|
|
143
|
+
|
|
144
|
+
### Terminology
|
|
145
|
+
|
|
146
|
+
When we discuss the platform, we use specific terms that are worth understanding:
|
|
147
|
+
|
|
148
|
+
- **experiment** - Any deviation of code paths we want to test
|
|
149
|
+
- **context** - Identifies a consistent experience (user, project, session, etc.)
|
|
150
|
+
- **control** - The default or "original" code path
|
|
151
|
+
- **candidate** - One experimental code path (used in A/B tests)
|
|
152
|
+
- **variant(s)** - Multiple experimental paths (used in A/B/n tests)
|
|
153
|
+
- **behaviors** - All possible code paths (control + all variants)
|
|
154
|
+
- **rollout strategy** - Logic determining if an experiment is enabled and how variants are assigned
|
|
155
|
+
- **segmentation** - Rules for routing specific contexts to predetermined variants
|
|
156
|
+
- **exclusion** - Rules for keeping contexts out of experiments entirely
|
|
22
157
|
|
|
23
|
-
Behaviors is a general term for all code paths -- if that's control and candidate, or control and variants. It's all of them.
|
|
24
158
|
<br clear="all">
|
|
25
159
|
|
|
26
|
-
|
|
160
|
+
## Quick Start: From Zero to Experiment in 5 Minutes
|
|
27
161
|
|
|
28
|
-
|
|
162
|
+
### Installation
|
|
29
163
|
|
|
30
164
|
Add the gem to your Gemfile and then `bundle install`.
|
|
31
165
|
|
|
@@ -33,204 +167,390 @@ Add the gem to your Gemfile and then `bundle install`.
|
|
|
33
167
|
gem 'gitlab-experiment'
|
|
34
168
|
```
|
|
35
169
|
|
|
36
|
-
If you're using Rails,
|
|
170
|
+
If you're using Rails, install the initializer which provides basic configuration, documentation, and the base
|
|
171
|
+
experiment class:
|
|
37
172
|
|
|
38
173
|
```shell
|
|
39
174
|
$ rails generate gitlab:experiment:install
|
|
40
175
|
```
|
|
41
176
|
|
|
42
|
-
|
|
177
|
+
### Your First Experiment
|
|
43
178
|
|
|
44
|
-
|
|
179
|
+
Let's create a real-world experiment to optimize a call-to-action button.
|
|
180
|
+
This example demonstrates the power of the platform while remaining practical.
|
|
45
181
|
|
|
46
|
-
|
|
182
|
+
#### Step 1: Generate the experiment
|
|
47
183
|
|
|
48
|
-
|
|
184
|
+
**Hypothesis**: A more prominent call-to-action button will increase conversion rates
|
|
49
185
|
|
|
50
186
|
```shell
|
|
51
|
-
$ rails generate gitlab:experiment
|
|
52
|
-
```
|
|
187
|
+
$ rails generate gitlab:experiment signup_cta
|
|
188
|
+
```
|
|
53
189
|
|
|
54
|
-
This
|
|
190
|
+
This creates `app/experiments/signup_cta_experiment.rb` with helpful inline documentation.
|
|
55
191
|
|
|
56
|
-
|
|
192
|
+
#### Step 2: Define your experiment class
|
|
57
193
|
|
|
58
194
|
```ruby
|
|
59
|
-
class
|
|
60
|
-
|
|
61
|
-
|
|
195
|
+
class SignupCtaExperiment < ApplicationExperiment
|
|
196
|
+
# Define the control (current experience)
|
|
197
|
+
control { 'btn-default' }
|
|
198
|
+
|
|
199
|
+
# Define the candidate (new experience to test)
|
|
200
|
+
candidate { 'btn-primary btn-lg' }
|
|
201
|
+
|
|
202
|
+
# Optional: Exclude certain users
|
|
203
|
+
exclude :existing_customers
|
|
204
|
+
|
|
205
|
+
# Optional: Track when the experiment runs
|
|
206
|
+
after_run :log_experiment_assignment
|
|
207
|
+
|
|
208
|
+
private
|
|
209
|
+
|
|
210
|
+
def existing_customers
|
|
211
|
+
context.actor&.subscribed?
|
|
212
|
+
end
|
|
213
|
+
|
|
214
|
+
def log_experiment_assignment
|
|
215
|
+
Rails.logger.info("User assigned to #{assigned.name} variant")
|
|
216
|
+
end
|
|
62
217
|
end
|
|
63
218
|
```
|
|
64
219
|
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
For our experiment we're going to make it "sticky" to the current user -- and if there isn't a current user, we want to assign and use a cookie value instead. This happens automatically if you use the [`actor` keyword](#cookies-and-the-actor-keyword) in your context. We use that in a lot of our examples, but it's by no means how everything should be done.
|
|
220
|
+
#### Step 3: Use the experiment in your view
|
|
68
221
|
|
|
69
222
|
```haml
|
|
70
|
-
|
|
223
|
+
-# The experiment is sticky to the current user
|
|
224
|
+
-# Anonymous users get a cookie-based assignment
|
|
225
|
+
%button{ class: experiment(:signup_cta, actor: current_user).run }
|
|
226
|
+
Start Free Trial
|
|
71
227
|
```
|
|
72
228
|
|
|
73
|
-
|
|
74
|
-
|
|
75
|
-
You can also provide behavior overrides when you run the experiment. Let's use a view helper for this example because it'll be cleaner than putting our override blocks into the view.
|
|
229
|
+
#### Step 4: Track engagement
|
|
76
230
|
|
|
77
231
|
```ruby
|
|
78
|
-
|
|
79
|
-
|
|
80
|
-
|
|
81
|
-
|
|
232
|
+
# In your controller, track when users click the button
|
|
233
|
+
def create_trial
|
|
234
|
+
experiment(:signup_cta, actor: current_user).track(:signup_completed)
|
|
235
|
+
|
|
236
|
+
# ... rest of your trial creation logic
|
|
82
237
|
end
|
|
83
238
|
```
|
|
84
239
|
|
|
85
|
-
|
|
240
|
+
**That's it!** Your experiment is now running, collecting data, and providing consistent experiences to your users.
|
|
86
241
|
|
|
87
|
-
|
|
88
|
-
|
|
89
|
-
|
|
242
|
+
## Real-World Examples
|
|
243
|
+
|
|
244
|
+
### Example 1: Onboarding Flow Optimization
|
|
90
245
|
|
|
91
|
-
|
|
246
|
+
**Business Context**: Product team wants to increase new user activation by testing different onboarding sequences.
|
|
92
247
|
|
|
93
248
|
```ruby
|
|
94
|
-
|
|
95
|
-
|
|
249
|
+
class OnboardingFlowExperiment < ApplicationExperiment
|
|
250
|
+
# Three different onboarding approaches
|
|
251
|
+
control { :standard_tour } # Current 5-step tour
|
|
252
|
+
variant(:quick) { :quick_start } # Streamlined 2-step flow
|
|
253
|
+
variant(:video) { :video_guide } # Video-based walkthrough
|
|
254
|
+
|
|
255
|
+
# Only show to new users who haven't completed onboarding
|
|
256
|
+
exclude :has_completed_onboarding
|
|
257
|
+
|
|
258
|
+
# Segment enterprise trial users to the standard tour
|
|
259
|
+
segment :enterprise_trial?, variant: :control
|
|
260
|
+
|
|
261
|
+
private
|
|
262
|
+
|
|
263
|
+
def has_completed_onboarding
|
|
264
|
+
context.actor&.onboarding_completed_at.present?
|
|
265
|
+
end
|
|
266
|
+
|
|
267
|
+
def enterprise_trial?
|
|
268
|
+
context.actor&.trial_type == 'enterprise'
|
|
269
|
+
end
|
|
270
|
+
end
|
|
96
271
|
|
|
97
|
-
|
|
272
|
+
# In your onboarding controller
|
|
273
|
+
def show
|
|
274
|
+
flow = experiment(:onboarding_flow, actor: current_user).run
|
|
275
|
+
render_onboarding_flow(flow)
|
|
276
|
+
end
|
|
98
277
|
|
|
99
|
-
|
|
278
|
+
# Track completion
|
|
279
|
+
def complete
|
|
280
|
+
experiment(:onboarding_flow, actor: current_user).track(:completed)
|
|
281
|
+
# ... mark user as onboarded
|
|
282
|
+
end
|
|
283
|
+
```
|
|
100
284
|
|
|
101
|
-
|
|
285
|
+
### Example 2: Pricing Page Experiment
|
|
286
|
+
|
|
287
|
+
**Business Context**: Growth team wants to test whether showing annual savings increases annual plan selection.
|
|
102
288
|
|
|
103
289
|
```ruby
|
|
104
|
-
class
|
|
105
|
-
|
|
106
|
-
|
|
107
|
-
|
|
108
|
-
|
|
290
|
+
class PricingDisplayExperiment < ApplicationExperiment
|
|
291
|
+
control { :monthly_default }
|
|
292
|
+
candidate { :annual_default_with_savings }
|
|
293
|
+
|
|
294
|
+
# Only run for unauthenticated visitors
|
|
295
|
+
exclude :authenticated_user
|
|
296
|
+
|
|
297
|
+
private
|
|
298
|
+
|
|
299
|
+
def authenticated_user
|
|
300
|
+
context.actor.present?
|
|
301
|
+
end
|
|
302
|
+
end
|
|
109
303
|
|
|
110
|
-
|
|
111
|
-
|
|
304
|
+
# In your pricing view
|
|
305
|
+
- pricing_variant = experiment(:pricing_display, actor: current_user).run
|
|
306
|
+
= render "pricing/#{pricing_variant}"
|
|
112
307
|
|
|
113
|
-
|
|
114
|
-
|
|
115
|
-
|
|
116
|
-
|
|
117
|
-
|
|
118
|
-
|
|
308
|
+
# Track plan selections
|
|
309
|
+
def select_plan
|
|
310
|
+
experiment(:pricing_display, actor: current_user).track(:plan_selected,
|
|
311
|
+
value: params[:plan_type] == 'annual' ? 1 : 0
|
|
312
|
+
)
|
|
313
|
+
end
|
|
314
|
+
```
|
|
315
|
+
|
|
316
|
+
### Example 3: Algorithm Performance Test
|
|
119
317
|
|
|
318
|
+
**Business Context**: Engineering team wants to compare a new search algorithm's performance before full rollout.
|
|
319
|
+
|
|
320
|
+
```ruby
|
|
321
|
+
class SearchAlgorithmExperiment < ApplicationExperiment
|
|
322
|
+
control { SearchEngine::Legacy }
|
|
323
|
+
candidate { SearchEngine::Neural }
|
|
324
|
+
|
|
325
|
+
# Only run for 25% of searches
|
|
326
|
+
default_rollout :percent, distribution: { control: 75, candidate: 25 }
|
|
327
|
+
|
|
328
|
+
# Exclude searches from API (higher SLA requirements)
|
|
329
|
+
exclude :api_request
|
|
330
|
+
|
|
331
|
+
# Track performance metrics
|
|
332
|
+
after_run :record_search_timing
|
|
333
|
+
|
|
120
334
|
private
|
|
121
335
|
|
|
122
|
-
|
|
123
|
-
|
|
124
|
-
'grey'
|
|
336
|
+
def api_request
|
|
337
|
+
context.request&.path&.start_with?('/api/')
|
|
125
338
|
end
|
|
126
|
-
|
|
127
|
-
|
|
128
|
-
|
|
129
|
-
'red'
|
|
339
|
+
|
|
340
|
+
def record_search_timing
|
|
341
|
+
# Custom metrics tracking
|
|
130
342
|
end
|
|
343
|
+
end
|
|
131
344
|
|
|
132
|
-
|
|
133
|
-
|
|
134
|
-
|
|
135
|
-
|
|
345
|
+
# In your search service
|
|
346
|
+
def search(query)
|
|
347
|
+
algorithm = experiment(:search_algorithm,
|
|
348
|
+
actor: current_user,
|
|
349
|
+
project: current_project
|
|
350
|
+
).run
|
|
351
|
+
|
|
352
|
+
results = algorithm.search(query)
|
|
353
|
+
|
|
354
|
+
experiment(:search_algorithm,
|
|
355
|
+
actor: current_user,
|
|
356
|
+
project: current_project
|
|
357
|
+
).track(:search_completed, value: results.count)
|
|
358
|
+
|
|
359
|
+
results
|
|
360
|
+
end
|
|
361
|
+
```
|
|
136
362
|
|
|
137
|
-
|
|
138
|
-
|
|
139
|
-
|
|
140
|
-
end
|
|
363
|
+
### Example 4: Progressive Feature Rollout
|
|
364
|
+
|
|
365
|
+
**Business Context**: Launching a new AI-assisted code review feature, want to expand gradually to manage load and gather feedback.
|
|
141
366
|
|
|
142
|
-
|
|
143
|
-
|
|
144
|
-
|
|
367
|
+
```ruby
|
|
368
|
+
class AiCodeReviewExperiment < ApplicationExperiment
|
|
369
|
+
control { false } # Feature disabled
|
|
370
|
+
candidate { true } # Feature enabled
|
|
371
|
+
|
|
372
|
+
# Start with 5% rollout
|
|
373
|
+
default_rollout :percent, distribution: { control: 95, candidate: 5 }
|
|
374
|
+
|
|
375
|
+
# Segment beta program users to always get the feature
|
|
376
|
+
segment :beta_user?, variant: :candidate
|
|
377
|
+
|
|
378
|
+
# Exclude free tier (computational cost consideration)
|
|
379
|
+
exclude :free_tier_user
|
|
380
|
+
|
|
381
|
+
private
|
|
382
|
+
|
|
383
|
+
def beta_user?
|
|
384
|
+
context.actor&.beta_features_enabled?
|
|
145
385
|
end
|
|
146
|
-
|
|
147
|
-
|
|
148
|
-
|
|
149
|
-
# ...hypothetical implementation
|
|
386
|
+
|
|
387
|
+
def free_tier_user
|
|
388
|
+
context.actor&.subscription_tier == 'free'
|
|
150
389
|
end
|
|
151
390
|
end
|
|
391
|
+
|
|
392
|
+
# In your merge request view
|
|
393
|
+
- if experiment(:ai_code_review, actor: current_user, project: @project).run
|
|
394
|
+
.ai-code-review-panel
|
|
395
|
+
= render 'ai_suggestions'
|
|
396
|
+
|
|
397
|
+
# Track usage
|
|
398
|
+
def apply_ai_suggestion
|
|
399
|
+
experiment(:ai_code_review, actor: current_user, project: @project)
|
|
400
|
+
.track(:suggestion_applied)
|
|
401
|
+
end
|
|
152
402
|
```
|
|
153
403
|
|
|
154
|
-
|
|
404
|
+
## Platform Integration Patterns
|
|
155
405
|
|
|
156
|
-
|
|
157
|
-
include Gitlab::Experiment::Dsl
|
|
406
|
+
### Integration with Feature Flags (Flipper)
|
|
158
407
|
|
|
159
|
-
|
|
160
|
-
ex = experiment(:pill_color, actor: User.first) # => #<PillColorExperiment:0x...>
|
|
408
|
+
Many organizations already use feature flag systems. GitLab Experiment integrates seamlessly:
|
|
161
409
|
|
|
162
|
-
|
|
163
|
-
|
|
410
|
+
```ruby
|
|
411
|
+
module Gitlab::Experiment::Rollout
|
|
412
|
+
class Flipper < Percent
|
|
413
|
+
def enabled?
|
|
414
|
+
::Flipper.enabled?(experiment.name, experiment_actor)
|
|
415
|
+
end
|
|
416
|
+
|
|
417
|
+
def experiment_actor
|
|
418
|
+
Struct.new(:flipper_id).new("Experiment;#{id}")
|
|
419
|
+
end
|
|
420
|
+
end
|
|
421
|
+
end
|
|
164
422
|
|
|
165
|
-
#
|
|
166
|
-
|
|
167
|
-
|
|
423
|
+
# Configure globally
|
|
424
|
+
Gitlab::Experiment.configure do |config|
|
|
425
|
+
config.default_rollout = Gitlab::Experiment::Rollout::Flipper.new
|
|
426
|
+
end
|
|
168
427
|
|
|
169
|
-
#
|
|
170
|
-
|
|
171
|
-
ex.publish # => {:variant=>"control", :experiment=>"pill_color", :key=>"45f595...", :excluded=>false}
|
|
428
|
+
# Now Flipper controls your experiments
|
|
429
|
+
Flipper.enable_percentage_of_actors(:signup_cta, 50)
|
|
172
430
|
```
|
|
173
431
|
|
|
174
|
-
|
|
175
|
-
<summary>You can also specify the variant manually...</summary>
|
|
176
|
-
|
|
177
|
-
Generally, defining segmentation rules is a better way to approach routing into specific variants, but it's possible to explicitly specify the variant when running an experiment.
|
|
432
|
+
### Integration with Analytics Platforms
|
|
178
433
|
|
|
179
|
-
|
|
434
|
+
Connect experiments to your analytics stack:
|
|
180
435
|
|
|
181
436
|
```ruby
|
|
182
|
-
|
|
437
|
+
Gitlab::Experiment.configure do |config|
|
|
438
|
+
config.tracking_behavior = lambda do |event_name, **data|
|
|
439
|
+
# Snowplow
|
|
440
|
+
SnowplowTracker.track_struct_event(
|
|
441
|
+
category: 'experiment',
|
|
442
|
+
action: event_name,
|
|
443
|
+
property: data[:experiment],
|
|
444
|
+
context: [{ schema: 'experiment_context', data: data }]
|
|
445
|
+
)
|
|
446
|
+
|
|
447
|
+
# Amplitude (example)
|
|
448
|
+
Amplitude.track(
|
|
449
|
+
user_id: data[:key], # Anonymous experiment key
|
|
450
|
+
event_type: "experiment_#{event_name}",
|
|
451
|
+
event_properties: data
|
|
452
|
+
)
|
|
453
|
+
|
|
454
|
+
# Custom data warehouse
|
|
455
|
+
DataWarehouse.log_experiment_event(event_name, data)
|
|
456
|
+
end
|
|
457
|
+
end
|
|
458
|
+
```
|
|
183
459
|
|
|
184
|
-
|
|
185
|
-
ex = experiment(:pill_color, :red, actor: User.first) # => #<PillColorExperiment:0x..>
|
|
460
|
+
### Multi-Application Consistency
|
|
186
461
|
|
|
187
|
-
|
|
188
|
-
ex.run # => "red"
|
|
462
|
+
Share experiment assignments across multiple applications:
|
|
189
463
|
|
|
190
|
-
|
|
191
|
-
|
|
464
|
+
```ruby
|
|
465
|
+
# Shared Redis cache
|
|
466
|
+
Gitlab::Experiment.configure do |config|
|
|
467
|
+
config.cache = Gitlab::Experiment::Cache::RedisHashStore.new(
|
|
468
|
+
Redis.new(url: ENV['REDIS_URL']),
|
|
469
|
+
expires_in: 30.days
|
|
470
|
+
)
|
|
471
|
+
end
|
|
472
|
+
|
|
473
|
+
# Now experiments stay consistent across your web app, API, and background jobs
|
|
192
474
|
```
|
|
193
475
|
|
|
194
|
-
|
|
476
|
+
## Advanced Features
|
|
195
477
|
|
|
196
|
-
###
|
|
478
|
+
### Multi-Variant (A/B/n) Testing
|
|
197
479
|
|
|
198
|
-
|
|
199
|
-
we're excluding something, it means that we don't want to run the experiment in that case. This can be useful if you
|
|
200
|
-
only want to run experiments on new users for instance.
|
|
480
|
+
Test multiple variations simultaneously to find the optimal solution:
|
|
201
481
|
|
|
202
482
|
```ruby
|
|
203
|
-
class
|
|
204
|
-
#
|
|
205
|
-
|
|
206
|
-
|
|
483
|
+
class NotificationStyleExperiment < ApplicationExperiment
|
|
484
|
+
# Test three different notification approaches
|
|
485
|
+
control { :banner } # Current: banner at top
|
|
486
|
+
variant(:toast) { :toast } # Toast notification
|
|
487
|
+
variant(:modal) { :modal } # Modal dialog
|
|
488
|
+
|
|
489
|
+
# Distribute traffic evenly across all three
|
|
490
|
+
default_rollout :percent,
|
|
491
|
+
distribution: { control: 34, toast: 33, modal: 33 }
|
|
492
|
+
|
|
493
|
+
# Exclude mobile users (different UI constraints)
|
|
494
|
+
exclude :mobile_user
|
|
495
|
+
|
|
496
|
+
# Segment power users to toast (less intrusive)
|
|
497
|
+
segment :power_user?, variant: :toast
|
|
498
|
+
|
|
499
|
+
private
|
|
500
|
+
|
|
501
|
+
def mobile_user
|
|
502
|
+
context.request&.user_agent&.match?(/Mobile/)
|
|
503
|
+
end
|
|
504
|
+
|
|
505
|
+
def power_user?
|
|
506
|
+
context.actor&.actions_count > 1000
|
|
507
|
+
end
|
|
207
508
|
end
|
|
208
509
|
```
|
|
209
510
|
|
|
210
|
-
|
|
211
|
-
only will they be immediately given the control behavior, but no events will be tracked in these cases either.
|
|
511
|
+
### Exclusion Rules
|
|
212
512
|
|
|
213
|
-
|
|
214
|
-
execution of further exclusion checks.
|
|
513
|
+
Keep contexts out of experiments entirely based on business rules:
|
|
215
514
|
|
|
216
|
-
|
|
515
|
+
```ruby
|
|
516
|
+
class FeatureExperiment < ApplicationExperiment
|
|
517
|
+
# Exclude existing customers (only test on prospects)
|
|
518
|
+
exclude :existing_customer
|
|
519
|
+
|
|
520
|
+
# Exclude during maintenance windows
|
|
521
|
+
exclude -> { context.project&.under_maintenance? }
|
|
522
|
+
|
|
523
|
+
# Exclude if feature is explicitly disabled
|
|
524
|
+
exclude :feature_disabled
|
|
525
|
+
|
|
526
|
+
private
|
|
527
|
+
|
|
528
|
+
def existing_customer
|
|
529
|
+
context.actor&.subscribed?
|
|
530
|
+
end
|
|
531
|
+
|
|
532
|
+
def feature_disabled
|
|
533
|
+
!FeatureFlag.enabled?(:allow_experiment, context.actor)
|
|
534
|
+
end
|
|
535
|
+
end
|
|
536
|
+
```
|
|
217
537
|
|
|
218
|
-
|
|
219
|
-
|
|
538
|
+
**Key behaviors:**
|
|
539
|
+
- Excluded users always receive the control experience
|
|
540
|
+
- No tracking events are recorded for excluded users
|
|
541
|
+
- Exclusion rules are evaluated in order, first match wins
|
|
542
|
+
- Exclusions improve performance by exiting early
|
|
543
|
+
|
|
544
|
+
**Inline exclusion** is also supported:
|
|
220
545
|
|
|
221
546
|
```ruby
|
|
222
|
-
experiment(:
|
|
547
|
+
experiment(:feature, actor: current_user) do |e|
|
|
223
548
|
e.exclude! unless can?(current_user, :manage, project)
|
|
224
|
-
|
|
225
|
-
e.
|
|
226
|
-
e.candidate { 'red' }
|
|
549
|
+
e.control { 'standard' }
|
|
550
|
+
e.candidate { 'enhanced' }
|
|
227
551
|
end
|
|
228
552
|
```
|
|
229
553
|
|
|
230
|
-
This approach keeps the experiment logic wrapped nicely within the experiment block, rather than requiring you to wrap
|
|
231
|
-
the entire experiment call in conditional logic. When `exclude!` is called, the experiment will be excluded and return
|
|
232
|
-
the control behavior without tracking any events.
|
|
233
|
-
|
|
234
554
|
Note: Although tracking calls will be ignored on all exclusions, you may want to check exclusion yourself in expensive
|
|
235
555
|
custom logic by calling the `should_track?` or `excluded?` methods.
|
|
236
556
|
|
|
@@ -240,146 +560,319 @@ future experiment run performance but can be a gotcha around caching.
|
|
|
240
560
|
Note: Exclusion rules aren't the best way to determine if an experiment is enabled. There's an `enabled?` method that
|
|
241
561
|
can be overridden to have a high-level way of determining if an experiment should be running and tracking at all. This
|
|
242
562
|
`enabled?` check should be as efficient as possible because it's the first early opt out path an experiment can
|
|
243
|
-
implement. This can be seen in [How
|
|
563
|
+
implement. This can be seen in [How Experiments Work (Technical)](#how-experiments-work-technical).
|
|
244
564
|
|
|
245
|
-
### Segmentation
|
|
565
|
+
### Segmentation Rules
|
|
246
566
|
|
|
247
|
-
|
|
567
|
+
Route specific populations to predetermined variants:
|
|
248
568
|
|
|
249
569
|
```ruby
|
|
250
|
-
class
|
|
251
|
-
#
|
|
252
|
-
|
|
253
|
-
|
|
254
|
-
|
|
570
|
+
class NewFeatureExperiment < ApplicationExperiment
|
|
571
|
+
# Route VIP customers to the new feature
|
|
572
|
+
segment :vip_customer?, variant: :candidate
|
|
573
|
+
|
|
574
|
+
# Route enterprise trial users to the enhanced experience
|
|
575
|
+
segment :enterprise_trial?, variant: :candidate
|
|
576
|
+
|
|
577
|
+
# Route users from specific campaigns to specific variants
|
|
578
|
+
segment(variant: :candidate) { context.campaign == 'product_launch_2024' }
|
|
579
|
+
|
|
580
|
+
private
|
|
581
|
+
|
|
582
|
+
def vip_customer?
|
|
583
|
+
context.actor&.account_value > 100_000
|
|
584
|
+
end
|
|
585
|
+
|
|
586
|
+
def enterprise_trial?
|
|
587
|
+
context.actor&.trial_tier == 'enterprise'
|
|
588
|
+
end
|
|
255
589
|
end
|
|
256
590
|
```
|
|
257
591
|
|
|
258
|
-
|
|
259
|
-
|
|
260
|
-
|
|
592
|
+
**Key behaviors:**
|
|
593
|
+
- Segmentation rules are evaluated in order, first match wins
|
|
594
|
+
- Segmented assignments are cached for consistency
|
|
595
|
+
- Perfect for gradually expanding successful experiments
|
|
596
|
+
- Enables sophisticated population targeting
|
|
261
597
|
|
|
262
|
-
###
|
|
598
|
+
### Lifecycle Callbacks
|
|
263
599
|
|
|
264
|
-
|
|
600
|
+
Execute custom logic at different stages of experiment execution:
|
|
265
601
|
|
|
266
602
|
```ruby
|
|
267
|
-
class
|
|
268
|
-
#
|
|
603
|
+
class PerformanceExperiment < ApplicationExperiment
|
|
604
|
+
# Run before the variant is determined
|
|
605
|
+
before_run :log_experiment_start
|
|
606
|
+
|
|
607
|
+
# Run after the variant is executed
|
|
608
|
+
after_run :record_timing_metrics, :notify_analytics_team
|
|
609
|
+
|
|
610
|
+
# Wrap the entire execution
|
|
611
|
+
around_run do |experiment, block|
|
|
612
|
+
start_time = Time.current
|
|
613
|
+
result = block.call
|
|
614
|
+
duration = Time.current - start_time
|
|
615
|
+
|
|
616
|
+
Metrics.record("experiment.#{experiment.name}.duration", duration)
|
|
617
|
+
result
|
|
618
|
+
end
|
|
269
619
|
|
|
270
|
-
|
|
620
|
+
private
|
|
621
|
+
|
|
622
|
+
def log_experiment_start
|
|
623
|
+
Rails.logger.info("Starting experiment: #{name}")
|
|
624
|
+
end
|
|
625
|
+
|
|
626
|
+
def record_timing_metrics
|
|
627
|
+
# Custom timing logic
|
|
628
|
+
end
|
|
629
|
+
|
|
630
|
+
def notify_analytics_team
|
|
631
|
+
# Send to analytics platform
|
|
632
|
+
end
|
|
271
633
|
end
|
|
272
634
|
```
|
|
273
635
|
|
|
274
|
-
|
|
636
|
+
**Use cases for callbacks:**
|
|
637
|
+
- Performance monitoring and APM integration
|
|
638
|
+
- Custom analytics and data warehouse updates
|
|
639
|
+
- Experiment-specific logging and debugging
|
|
640
|
+
- Integration with external systems
|
|
641
|
+
|
|
642
|
+
### Progressive Rollout with `only_assigned`
|
|
643
|
+
|
|
644
|
+
Control experiment expansion by only showing features to users already assigned to the experiment.
|
|
645
|
+
This is critical for managing blast radius and controlled rollouts:
|
|
646
|
+
|
|
647
|
+
**The Challenge**: You launch an experiment to 10% of new signups. Later, you want to show experimental features on other pages, but only to users already in the experiment - not expand to 10% of all users across the platform.
|
|
648
|
+
|
|
649
|
+
**The Solution**: Use `only_assigned: true`
|
|
275
650
|
|
|
276
651
|
```ruby
|
|
277
|
-
|
|
278
|
-
|
|
652
|
+
# Step 1: Assign users during signup (10% of new signups)
|
|
653
|
+
class RegistrationsController < ApplicationController
|
|
654
|
+
def create
|
|
655
|
+
user = User.create!(user_params)
|
|
656
|
+
|
|
657
|
+
# This assigns 10% to candidate, 90% to control
|
|
658
|
+
experiment(:onboarding_v2, actor: user).publish
|
|
659
|
+
|
|
660
|
+
redirect_to dashboard_path
|
|
661
|
+
end
|
|
662
|
+
end
|
|
279
663
|
|
|
280
|
-
|
|
281
|
-
|
|
282
|
-
|
|
283
|
-
|
|
664
|
+
# Step 2: Later, show features only to those already assigned
|
|
665
|
+
class DashboardController < ApplicationController
|
|
666
|
+
def show
|
|
667
|
+
# This will NOT expand the experiment to 10% of all users
|
|
668
|
+
# Only users assigned in Step 1 will see the experimental UI
|
|
669
|
+
@show_new_features = experiment(:onboarding_v2,
|
|
670
|
+
actor: current_user,
|
|
671
|
+
only_assigned: true
|
|
672
|
+
).assigned.name == 'candidate'
|
|
284
673
|
end
|
|
285
674
|
end
|
|
675
|
+
|
|
676
|
+
# Step 3: Show UI conditionally across the app
|
|
677
|
+
- if experiment(:onboarding_v2, actor: current_user, only_assigned: true).run
|
|
678
|
+
.new-onboarding-features
|
|
679
|
+
= render 'enhanced_dashboard'
|
|
286
680
|
```
|
|
287
681
|
|
|
288
|
-
|
|
682
|
+
**Behavior with `only_assigned: true`:**
|
|
683
|
+
- ✅ If user already assigned → returns their cached variant
|
|
684
|
+
- ✅ If user not assigned → returns control, no tracking
|
|
685
|
+
- ✅ Experiment reach stays controlled
|
|
686
|
+
- ✅ Perfect for multi-page experimental experiences
|
|
687
|
+
|
|
688
|
+
**Real-world use cases:**
|
|
689
|
+
- **Post-signup experiences**: Assign at signup, show features throughout the app
|
|
690
|
+
- **Gradual feature expansion**: Roll out to 5%, then add more touchpoints without expanding population
|
|
691
|
+
- **Cleanup phases**: Maintain experience for existing participants while preventing new assignments
|
|
692
|
+
- **A/B testing with multiple surfaces**: Test a hypothesis across multiple pages without assignment leakage
|
|
289
693
|
|
|
290
|
-
|
|
291
|
-
they're not. This is useful when you want to show experiment-specific content only to users who are already
|
|
292
|
-
participating in the experiment, without expanding the experiment's reach.
|
|
694
|
+
### Custom Rollout Strategies
|
|
293
695
|
|
|
294
|
-
|
|
696
|
+
The platform supports multiple rollout strategies out of the box, and you can create custom strategies for your specific
|
|
697
|
+
needs.
|
|
295
698
|
|
|
296
|
-
|
|
297
|
-
|
|
298
|
-
- experiment
|
|
299
|
-
|
|
300
|
-
|
|
301
|
-
|
|
302
|
-
|
|
699
|
+
**Built-in strategies:**
|
|
700
|
+
- [`Percent`](lib/gitlab/experiment/rollout/percent.rb) - Consistent percentage-based assignment (default, recommended)
|
|
701
|
+
- [`Random`](lib/gitlab/experiment/rollout/random.rb) - True random assignment (useful for load testing)
|
|
702
|
+
- [`RoundRobin`](lib/gitlab/experiment/rollout/round_robin.rb) - Cycle through variants (requires caching)
|
|
703
|
+
- [`Base`](lib/gitlab/experiment/rollout.rb) - Useful for building custom rollout strategies
|
|
704
|
+
|
|
705
|
+
```ruby
|
|
706
|
+
class LoadTestExperiment < ApplicationExperiment
|
|
707
|
+
# Randomly test two different caching strategies
|
|
708
|
+
default_rollout :random
|
|
709
|
+
|
|
710
|
+
control { CacheStrategy::Redis }
|
|
711
|
+
candidate { CacheStrategy::Memcached }
|
|
712
|
+
end
|
|
713
|
+
|
|
714
|
+
class GradualRolloutExperiment < ApplicationExperiment
|
|
715
|
+
# Start with 5% in the new experience
|
|
716
|
+
default_rollout :percent,
|
|
717
|
+
distribution: { control: 95, candidate: 5 }
|
|
718
|
+
end
|
|
303
719
|
```
|
|
304
720
|
|
|
305
|
-
|
|
721
|
+
See the [Advanced: Custom Rollout Strategies](#advanced-custom-rollout-strategies) section for building your own
|
|
722
|
+
integration with feature flag systems.
|
|
306
723
|
|
|
307
|
-
|
|
308
|
-
- If the user has no cached variant assignment, the experiment is excluded and returns the control behavior
|
|
309
|
-
- No new variant assignments are made
|
|
310
|
-
- Tracking is disabled for excluded cases
|
|
311
|
-
- Publishing still works to record the exclusion
|
|
724
|
+
## Organizational Best Practices
|
|
312
725
|
|
|
313
|
-
|
|
726
|
+
### Experiment Lifecycle Management
|
|
314
727
|
|
|
315
|
-
|
|
316
|
-
|
|
317
|
-
|
|
318
|
-
|
|
319
|
-
|
|
320
|
-
|
|
728
|
+
**1. Hypothesis Formation**
|
|
729
|
+
```ruby
|
|
730
|
+
# Document your hypothesis in the experiment class
|
|
731
|
+
class CheckoutFlowExperiment < ApplicationExperiment
|
|
732
|
+
# Hypothesis: Reducing checkout steps from 3 to 2 will increase completion rate
|
|
733
|
+
# Success metric: 5% increase in checkout completion
|
|
734
|
+
# Target: All free trial users
|
|
735
|
+
# Duration: 2 weeks
|
|
736
|
+
# Owner: @growth-team
|
|
737
|
+
|
|
738
|
+
control { :three_step_checkout }
|
|
739
|
+
candidate { :two_step_checkout }
|
|
740
|
+
|
|
741
|
+
exclude :existing_customer
|
|
742
|
+
end
|
|
743
|
+
```
|
|
321
744
|
|
|
745
|
+
**2. Gradual Rollout**
|
|
322
746
|
```ruby
|
|
323
|
-
#
|
|
324
|
-
|
|
325
|
-
|
|
326
|
-
|
|
327
|
-
|
|
328
|
-
|
|
329
|
-
|
|
747
|
+
# Week 1: 5% rollout
|
|
748
|
+
default_rollout :percent, distribution: { control: 95, candidate: 5 }
|
|
749
|
+
|
|
750
|
+
# Week 2: Increase to 25% if metrics look good
|
|
751
|
+
default_rollout :percent, distribution: { control: 75, candidate: 25 }
|
|
752
|
+
|
|
753
|
+
# Week 3: Full rollout if successful
|
|
754
|
+
default_rollout :percent, distribution: { control: 0, candidate: 100 }
|
|
755
|
+
```
|
|
756
|
+
|
|
757
|
+
**3. Monitoring and Alerting**
|
|
758
|
+
```ruby
|
|
759
|
+
class CriticalPathExperiment < ApplicationExperiment
|
|
760
|
+
after_run :monitor_performance
|
|
761
|
+
after_run :alert_on_errors
|
|
762
|
+
|
|
763
|
+
private
|
|
764
|
+
|
|
765
|
+
def monitor_performance
|
|
766
|
+
Metrics.increment("experiment.#{name}.#{assigned.name}")
|
|
767
|
+
end
|
|
768
|
+
|
|
769
|
+
def alert_on_errors
|
|
770
|
+
if context.error_rate > threshold
|
|
771
|
+
PagerDuty.alert("High error rate in #{name}")
|
|
772
|
+
end
|
|
330
773
|
end
|
|
331
774
|
end
|
|
332
775
|
```
|
|
333
776
|
|
|
334
|
-
|
|
335
|
-
|
|
336
|
-
|
|
337
|
-
|
|
338
|
-
|
|
339
|
-
|
|
777
|
+
**4. Experiment Cleanup**
|
|
778
|
+
```ruby
|
|
779
|
+
# When experiment is conclusive, clean up:
|
|
780
|
+
# 1. Remove the experiment code
|
|
781
|
+
# 2. Promote winner to production
|
|
782
|
+
# 3. Document learnings
|
|
783
|
+
|
|
784
|
+
# Before cleanup, archive results:
|
|
785
|
+
experiment(:checkout_flow).publish
|
|
786
|
+
# Export data for historical analysis
|
|
340
787
|
```
|
|
341
788
|
|
|
342
|
-
|
|
789
|
+
### Team Collaboration Patterns
|
|
343
790
|
|
|
791
|
+
**Product + Engineering + Data Science**
|
|
344
792
|
```ruby
|
|
345
|
-
|
|
346
|
-
|
|
347
|
-
|
|
348
|
-
|
|
349
|
-
|
|
793
|
+
class CollaborativeExperiment < ApplicationExperiment
|
|
794
|
+
# Product defines the hypothesis and variants
|
|
795
|
+
control { :current_flow }
|
|
796
|
+
candidate { :new_flow }
|
|
797
|
+
|
|
798
|
+
# Engineering defines segmentation and rollout
|
|
799
|
+
segment :beta_users, variant: :candidate
|
|
800
|
+
default_rollout :percent, distribution: { control: 90, candidate: 10 }
|
|
801
|
+
|
|
802
|
+
# Data science defines tracking and metrics
|
|
803
|
+
after_run :track_funnel_step
|
|
804
|
+
|
|
805
|
+
def track_funnel_step
|
|
806
|
+
Analytics.track_experiment_step(
|
|
807
|
+
experiment: name,
|
|
808
|
+
variant: assigned.name,
|
|
809
|
+
funnel_position: context.step,
|
|
810
|
+
user_segment: context.actor&.segment
|
|
811
|
+
)
|
|
812
|
+
end
|
|
813
|
+
end
|
|
350
814
|
```
|
|
351
815
|
|
|
352
|
-
|
|
353
|
-
checking for cached variant assignments.
|
|
354
|
-
|
|
355
|
-
### Rollout strategies
|
|
816
|
+
### Testing Strategy
|
|
356
817
|
|
|
357
|
-
|
|
818
|
+
Write tests for your experiments using the included RSpec matchers:
|
|
358
819
|
|
|
359
820
|
```ruby
|
|
360
|
-
|
|
361
|
-
|
|
362
|
-
|
|
363
|
-
|
|
821
|
+
RSpec.describe CheckoutFlowExperiment do
|
|
822
|
+
describe 'segmentation' do
|
|
823
|
+
it 'routes existing customers to control' do
|
|
824
|
+
customer = create(:user, :with_subscription)
|
|
825
|
+
expect(experiment(:checkout_flow)).to exclude(actor: customer)
|
|
826
|
+
end
|
|
827
|
+
|
|
828
|
+
it 'routes enterprise trials to candidate' do
|
|
829
|
+
trial = create(:user, :enterprise_trial)
|
|
830
|
+
expect(experiment(:checkout_flow))
|
|
831
|
+
.to segment(actor: trial).into(:candidate)
|
|
832
|
+
end
|
|
833
|
+
end
|
|
834
|
+
|
|
835
|
+
describe 'tracking' do
|
|
836
|
+
it 'tracks checkout completion' do
|
|
837
|
+
expect(experiment(:checkout_flow)).to track(:completed)
|
|
838
|
+
.on_next_instance
|
|
839
|
+
|
|
840
|
+
CheckoutService.complete(user: user)
|
|
841
|
+
end
|
|
842
|
+
end
|
|
364
843
|
end
|
|
365
844
|
```
|
|
366
845
|
|
|
367
|
-
|
|
846
|
+
### Naming Conventions
|
|
847
|
+
|
|
848
|
+
Establish clear naming conventions for your organization:
|
|
849
|
+
|
|
850
|
+
```ruby
|
|
851
|
+
# Good: Descriptive experiment names
|
|
852
|
+
class OnboardingFlowV2Experiment < ApplicationExperiment; end
|
|
853
|
+
class PricingPageAnnualFocusExperiment < ApplicationExperiment; end
|
|
854
|
+
class SearchAlgorithmNeuralExperiment < ApplicationExperiment; end
|
|
855
|
+
|
|
856
|
+
# Avoid: Vague names
|
|
857
|
+
class TestExperiment < ApplicationExperiment; end # What are we testing?
|
|
858
|
+
class ExperimentOne < ApplicationExperiment; end # No context
|
|
859
|
+
```
|
|
368
860
|
|
|
369
|
-
##
|
|
861
|
+
## Technical Reference
|
|
370
862
|
|
|
371
|
-
|
|
372
|
-
following logic is executed to resolve what experience should be provided, given how the experiment is defined, and
|
|
373
|
-
using the context passed to the experiment call.
|
|
863
|
+
### How Experiments Work (Technical)
|
|
374
864
|
|
|
375
|
-
|
|
376
|
-
|
|
865
|
+
Understanding the experiment resolution flow helps you design better experiments and debug issues:
|
|
866
|
+
|
|
867
|
+
**Decision tree for variant assignment:**
|
|
377
868
|
|
|
378
869
|
```mermaid
|
|
379
870
|
graph TD
|
|
380
871
|
GP[General Pool/Population] --> Running?[Rollout Enabled?]
|
|
381
|
-
Running? -->|Yes|
|
|
872
|
+
Running? -->|Yes| Forced?[Forced Assignment?]
|
|
382
873
|
Running? -->|No| Excluded[Control / No Tracking]
|
|
874
|
+
Forced? -->|Yes / Cached| ForcedVariant[Forced Variant]
|
|
875
|
+
Forced? -->|No| Cached?[Cached? / Pre-segmented?]
|
|
383
876
|
Cached? -->|No| Excluded?
|
|
384
877
|
Cached? -->|Yes| Cached[Cached Value]
|
|
385
878
|
Excluded? -->|Yes / Cached| Excluded
|
|
@@ -391,50 +884,198 @@ graph TD
|
|
|
391
884
|
Rollout -->|Cached| VariantB
|
|
392
885
|
Rollout -->|Cached| VariantN
|
|
393
886
|
|
|
394
|
-
class VariantA,VariantB,VariantN included
|
|
887
|
+
class ForcedVariant,VariantA,VariantB,VariantN included
|
|
395
888
|
class Control,Excluded excluded
|
|
396
889
|
class Cached cached
|
|
397
890
|
```
|
|
398
891
|
|
|
399
|
-
|
|
892
|
+
**Key points:**
|
|
893
|
+
1. Rollout must be enabled for any variant assignment (including forced assignment)
|
|
894
|
+
2. Forced assignment takes priority over cache/exclusion/segmentation (via `glex_force` query parameter)
|
|
895
|
+
3. Cache provides consistency across calls
|
|
896
|
+
4. Segmentation takes priority over rollout
|
|
897
|
+
5. `only_assigned: true` exits early if no cache hit
|
|
898
|
+
|
|
899
|
+
### Experiment Context and Stickiness
|
|
900
|
+
|
|
901
|
+
Internally, experiments have what's referred to as the context "key" that represents the unique and anonymous id of a
|
|
902
|
+
given context. This allows us to assign the same variant between different calls to the experiment, is used in caching
|
|
903
|
+
and can be used in event data downstream. This context "key" is how an experiment remains "sticky" to a given context,
|
|
904
|
+
and is an important aspect to understand.
|
|
905
|
+
|
|
906
|
+
**Context defines stickiness** - experiments remain consistent by generating an anonymous key from the context:
|
|
907
|
+
|
|
908
|
+
```ruby
|
|
909
|
+
# Sticky to user - same user gets same variant everywhere
|
|
910
|
+
experiment(:feature, actor: current_user)
|
|
911
|
+
|
|
912
|
+
# Sticky to project - all users on a project get the same experience
|
|
913
|
+
experiment(:feature, project: project)
|
|
914
|
+
|
|
915
|
+
# Sticky to user+project - same user gets same variant per project
|
|
916
|
+
experiment(:feature, actor: current_user, project: project)
|
|
400
917
|
|
|
401
|
-
|
|
918
|
+
# Custom stickiness - explicitly define what creates consistency
|
|
919
|
+
experiment(:feature, actor: current_user, project: project, sticky_to: project)
|
|
920
|
+
```
|
|
921
|
+
|
|
922
|
+
**The `actor` keyword has special behavior:**
|
|
923
|
+
- Anonymous users → temporary cookie-based assignment
|
|
924
|
+
- Upon sign-in → cookie migrates to user ID automatically
|
|
925
|
+
- Enables consistent experience across anonymous → authenticated journey
|
|
402
926
|
|
|
403
|
-
###
|
|
927
|
+
### Using Experiments Beyond Views
|
|
404
928
|
|
|
405
|
-
By default, `Gitlab::Experiment` injects itself into the controller, view, and mailer layers. This exposes the
|
|
929
|
+
By default, `Gitlab::Experiment` injects itself into the controller, view, and mailer layers. This exposes the
|
|
930
|
+
`experiment` method application wide in those layers. Some experiments may extend outside of those layers however, so
|
|
931
|
+
you may want to include it elsewhere. For instance in an irb session or the rails console, or in all your service
|
|
932
|
+
objects, background jobs, or similar:
|
|
406
933
|
|
|
407
934
|
```ruby
|
|
935
|
+
# In all background jobs
|
|
408
936
|
class ApplicationJob < ActiveJob::Base
|
|
409
|
-
include Gitlab::Experiment::Dsl
|
|
937
|
+
include Gitlab::Experiment::Dsl
|
|
938
|
+
end
|
|
939
|
+
|
|
940
|
+
# In service objects
|
|
941
|
+
class ApplicationService
|
|
942
|
+
include Gitlab::Experiment::Dsl
|
|
410
943
|
end
|
|
944
|
+
|
|
945
|
+
# In a console session
|
|
946
|
+
include Gitlab::Experiment::Dsl
|
|
947
|
+
experiment(:feature, actor: User.first).run
|
|
411
948
|
```
|
|
412
949
|
|
|
413
|
-
###
|
|
950
|
+
### Manual Variant Assignment
|
|
414
951
|
|
|
415
|
-
|
|
952
|
+
<details>
|
|
953
|
+
<summary>You can also specify the variant manually...</summary>
|
|
416
954
|
|
|
417
|
-
|
|
955
|
+
Generally, defining segmentation rules is a better way to approach routing into specific variants, but it's possible to
|
|
956
|
+
explicitly specify the variant when running an experiment.
|
|
418
957
|
|
|
419
|
-
|
|
958
|
+
Caching: It's important to understand what this might do to your data during rollout, so use this with careful
|
|
959
|
+
consideration. Any time a specific variant is assigned manually, or through segmentation (including `:control`) it will
|
|
960
|
+
be cached for that context. That means that if you manually assign `:control`, that context will never be moved out of
|
|
961
|
+
the control unless you do it programmatically elsewhere.
|
|
420
962
|
|
|
421
963
|
```ruby
|
|
422
|
-
|
|
964
|
+
include Gitlab::Experiment::Dsl
|
|
965
|
+
|
|
966
|
+
# Assign the candidate manually.
|
|
967
|
+
ex = experiment(:pill_color, :red, actor: User.first) # => #<PillColorExperiment:0x..>
|
|
968
|
+
|
|
969
|
+
# Run the experiment -- returning the result.
|
|
970
|
+
ex.run # => "red"
|
|
971
|
+
|
|
972
|
+
# If caching is enabled this will remain sticky between calls.
|
|
973
|
+
experiment(:pill_color, actor: User.first).run # => "red"
|
|
423
974
|
```
|
|
424
975
|
|
|
425
|
-
|
|
976
|
+
</details>
|
|
977
|
+
|
|
978
|
+
### Forced Variant Assignment (QA/UAT)
|
|
426
979
|
|
|
427
|
-
|
|
980
|
+
For testing and validation purposes, you can force a specific variant assignment via a URL query parameter. This is
|
|
981
|
+
useful for QA testing in staging or production environments where you need to verify a specific variant's behavior.
|
|
982
|
+
|
|
983
|
+
**Configuration:**
|
|
984
|
+
|
|
985
|
+
Forced assignment is disabled by default. Enable it in your initializer:
|
|
986
|
+
|
|
987
|
+
```ruby
|
|
988
|
+
Gitlab::Experiment.configure do |config|
|
|
989
|
+
config.allow_forced_assignment = true
|
|
990
|
+
end
|
|
991
|
+
```
|
|
992
|
+
|
|
993
|
+
**Usage:**
|
|
994
|
+
|
|
995
|
+
Append the `glex_force` query parameter to any URL with the format `experiment_name:variant_name`:
|
|
996
|
+
|
|
997
|
+
```
|
|
998
|
+
https://your-app.com/signup?glex_force=myapp_signup_cta:candidate
|
|
999
|
+
```
|
|
1000
|
+
|
|
1001
|
+
The forced variant is written to the cache (Redis) on the same request, making it permanent for that context. The
|
|
1002
|
+
query parameter only needs to be provided once -- after that, the variant is persisted in the cache like any normal
|
|
1003
|
+
assignment.
|
|
1004
|
+
|
|
1005
|
+
#### Anonymous user (nil actor) -- initial assignment
|
|
1006
|
+
|
|
1007
|
+
This is the primary use case for QA testing signup flows and landing pages. The user is not signed in, so the actor is
|
|
1008
|
+
nil and the experiment uses a cookie-based context key.
|
|
1009
|
+
|
|
1010
|
+
1. Anonymous user visits `https://your-app.com/signup?glex_force=signup_cta:candidate`
|
|
1011
|
+
2. The forced variant `:candidate` is written to Redis under the cookie-based context key
|
|
1012
|
+
3. The user signs in -- the standard cookie migration carries the forced variant to their real identity
|
|
1013
|
+
4. All future requests use `:candidate` from Redis, permanently
|
|
1014
|
+
|
|
1015
|
+
This means a QA tester can force a variant before signup and have it follow the user through the entire
|
|
1016
|
+
anonymous-to-authenticated journey.
|
|
1017
|
+
|
|
1018
|
+
#### Signed-in user -- initial assignment
|
|
1019
|
+
|
|
1020
|
+
When a signed-in user hasn't been assigned a variant yet, the force param assigns and caches it immediately:
|
|
1021
|
+
|
|
1022
|
+
```
|
|
1023
|
+
https://your-app.com/dashboard?glex_force=new_feature:candidate
|
|
1024
|
+
```
|
|
1025
|
+
|
|
1026
|
+
The variant is cached under the user's context key on this request. No further query parameter is needed.
|
|
1027
|
+
|
|
1028
|
+
#### Signed-in user -- re-assignment (overwriting an existing variant)
|
|
1029
|
+
|
|
1030
|
+
If a user was previously assigned `:control` (by the rollout strategy or a prior force), the force param overwrites
|
|
1031
|
+
the cached value:
|
|
1032
|
+
|
|
1033
|
+
```
|
|
1034
|
+
https://your-app.com/dashboard?glex_force=new_feature:candidate
|
|
1035
|
+
```
|
|
1036
|
+
|
|
1037
|
+
The existing `:control` assignment in Redis is replaced with `:candidate`. This is useful when QA needs to switch a
|
|
1038
|
+
user between variants without clearing the cache manually.
|
|
1039
|
+
|
|
1040
|
+
#### Disabled experiments and feature flags
|
|
1041
|
+
|
|
1042
|
+
Forced assignment requires the experiment to be enabled. If the experiment is disabled (as determined by the rollout
|
|
1043
|
+
strategy's `enabled?` method), the `glex_force` parameter is ignored and normal resolution applies (which will assign
|
|
1044
|
+
control).
|
|
1045
|
+
|
|
1046
|
+
This is intentional -- a disabled experiment may be disabled for valid reasons (incomplete implementation, known issues,
|
|
1047
|
+
compliance constraints, etc.) and force assignment should not provide a way to bypass that decision. To use forced
|
|
1048
|
+
assignment, ensure the experiment is enabled first through your rollout strategy.
|
|
1049
|
+
|
|
1050
|
+
**Important notes:**
|
|
1051
|
+
- The experiment name in the parameter must match the full experiment name (including any configured `name_prefix`).
|
|
1052
|
+
- If the variant name doesn't match a registered behavior, the forced assignment is ignored and normal variant resolution
|
|
1053
|
+
proceeds (typically resulting in the control variant).
|
|
1054
|
+
- Forced assignment does not override a variant that was already set via the constructor or an explicit `assigned()`
|
|
1055
|
+
call within the same request.
|
|
1056
|
+
- This feature requires a `request` object with `params` to be available in the experiment context.
|
|
1057
|
+
|
|
1058
|
+
> [!NOTE]
|
|
1059
|
+
> Because forcing the variant ignores the exclusion/segmentation process it will cover up those types of errors so if your experiment relies on these types of logic this testing method should be avoided.
|
|
1060
|
+
|
|
1061
|
+
### Experiment Signature
|
|
1062
|
+
|
|
1063
|
+
The best way to understand the details of an experiment is through its signature. An example signature can be retrieved
|
|
1064
|
+
by calling the `signature` method, and looks like the following:
|
|
428
1065
|
|
|
429
1066
|
```ruby
|
|
430
1067
|
experiment(:example).signature # => {:variant=>"control", :experiment=>"example", :key=>"4d7aee..."}
|
|
431
1068
|
```
|
|
432
1069
|
|
|
433
|
-
An experiment signature is useful when tracking events and when using experiments on the client layer. The signature can
|
|
1070
|
+
An experiment signature is useful when tracking events and when using experiments on the client layer. The signature can
|
|
1071
|
+
also contain the optional `migration_keys`, and `excluded` properties.
|
|
434
1072
|
|
|
435
|
-
### Return
|
|
1073
|
+
### Return Value
|
|
436
1074
|
|
|
437
|
-
By default the return value of calling `experiment` is a `Gitlab::Experiment` instance, or whatever class the
|
|
1075
|
+
By default the return value of calling `experiment` is a `Gitlab::Experiment` instance, or whatever class the
|
|
1076
|
+
experiment is resolved to, which likely inherits from `Gitlab::Experiment`. In simple cases you may want only the
|
|
1077
|
+
results of running the experiment though. You can call `run` within the block to get the return value of the assigned
|
|
1078
|
+
variant.
|
|
438
1079
|
|
|
439
1080
|
```ruby
|
|
440
1081
|
# Normally an experiment instance.
|
|
@@ -453,11 +1094,14 @@ end # => 'A'
|
|
|
453
1094
|
|
|
454
1095
|
### Context migrations
|
|
455
1096
|
|
|
456
|
-
There are times when we need to change context while an experiment is running.
|
|
1097
|
+
There are times when we need to change context while an experiment is running.
|
|
1098
|
+
We make this possible by passing the migration data to the experiment.
|
|
457
1099
|
|
|
458
|
-
Take for instance, that you might be using `version: 1` in your context currently.
|
|
1100
|
+
Take for instance, that you might be using `version: 1` in your context currently.
|
|
1101
|
+
To migrate this to `version: 2`, provide the portion of the context you wish to change using a `migrated_with` option.
|
|
459
1102
|
|
|
460
|
-
In providing the context migration data, we can resolve an experience and its events all the way back.
|
|
1103
|
+
In providing the context migration data, we can resolve an experience and its events all the way back.
|
|
1104
|
+
This can also help in keeping our cache relevant.
|
|
461
1105
|
|
|
462
1106
|
```ruby
|
|
463
1107
|
# First implementation.
|
|
@@ -467,7 +1111,8 @@ experiment(:example, actor: current_user, version: 1)
|
|
|
467
1111
|
experiment(:example, actor: current_user, version: 2, migrated_with: { version: 1 })
|
|
468
1112
|
```
|
|
469
1113
|
|
|
470
|
-
You can add or remove context by providing a `migrated_from` option.
|
|
1114
|
+
You can add or remove context by providing a `migrated_from` option.
|
|
1115
|
+
This approach expects a full context replacement -- i.e. what it was before you added or removed the new context key.
|
|
471
1116
|
|
|
472
1117
|
If you wanted to introduce a `version` to your context, provide the full previous context.
|
|
473
1118
|
|
|
@@ -479,7 +1124,9 @@ experiment(:example, actor: current_user)
|
|
|
479
1124
|
experiment(:example, actor: current_user, version: 1, migrated_from: { actor: current_user })
|
|
480
1125
|
```
|
|
481
1126
|
|
|
482
|
-
When you migrate context, this information is included in the signature of the experiment.
|
|
1127
|
+
When you migrate context, this information is included in the signature of the experiment.
|
|
1128
|
+
This can be used downstream in event handling and reporting to resolve a series of events back to a single experience,
|
|
1129
|
+
while also keeping everything anonymous.
|
|
483
1130
|
|
|
484
1131
|
An example of our experiment signature when we migrate would include the `migration_keys` property:
|
|
485
1132
|
|
|
@@ -493,11 +1140,16 @@ ex.signature # => {:key=>"9e9d93...", :migration_keys=>["20d69a..."], ...}
|
|
|
493
1140
|
|
|
494
1141
|
### Cookies and the actor keyword
|
|
495
1142
|
|
|
496
|
-
We use cookies to auto migrate an unknown value into a known value, often in the case of the current user.
|
|
1143
|
+
We use cookies to auto migrate an unknown value into a known value, often in the case of the current user.
|
|
1144
|
+
The implementation of this uses the same concept outlined above with context migrations, but will happen automatically
|
|
1145
|
+
for you if you use the `actor` keyword.
|
|
497
1146
|
|
|
498
|
-
When you use the `actor: current_user` pattern in your context, the nil case is handled by setting a special cookie for
|
|
1147
|
+
When you use the `actor: current_user` pattern in your context, the nil case is handled by setting a special cookie for
|
|
1148
|
+
the experiment and then deleting the cookie, and migrating the context key to the one generated from the user when
|
|
1149
|
+
they've signed in.
|
|
499
1150
|
|
|
500
|
-
This cookie is a temporary, randomized uuid and isn't associated with a user.
|
|
1151
|
+
This cookie is a temporary, randomized uuid and isn't associated with a user.
|
|
1152
|
+
When we can finally provide an actor, the context is auto migrated from the cookie to that actor.
|
|
501
1153
|
|
|
502
1154
|
```ruby
|
|
503
1155
|
# The actor key is not present, so no cookie is set.
|
|
@@ -511,65 +1163,166 @@ experiment(:example, actor: nil, project: project)
|
|
|
511
1163
|
experiment(:example, actor: current_user, project: project)
|
|
512
1164
|
```
|
|
513
1165
|
|
|
514
|
-
Note: The cookie is deleted when resolved, but can be assigned again if the `actor` is ever nil again.
|
|
1166
|
+
Note: The cookie is deleted when resolved, but can be assigned again if the `actor` is ever nil again.
|
|
1167
|
+
A good example of this scenario would be on a sign in page.
|
|
1168
|
+
When a potential user arrives, they would never be known, so a cookie would be set for them, and then resolved/removed
|
|
1169
|
+
as soon as they signed in.
|
|
1170
|
+
This process would repeat each time they arrived while not being signed in and can complicate reporting unless it's
|
|
1171
|
+
handled well in the data layers.
|
|
515
1172
|
|
|
516
|
-
Note: To read and write cookies, we provide the `request` from within the controller and views.
|
|
1173
|
+
Note: To read and write cookies, we provide the `request` from within the controller and views.
|
|
1174
|
+
The cookie migration will happen automatically if the experiment is within those layers.
|
|
1175
|
+
You'll need to provide the `request` as an option to the experiment if it's outside of the controller and views.
|
|
517
1176
|
|
|
518
1177
|
```ruby
|
|
519
1178
|
experiment(:example, actor: current_user, request: request)
|
|
520
1179
|
```
|
|
521
1180
|
|
|
522
|
-
Note: For edge cases, you can pass the cookie through by assigning it yourself -- e.g. `actor:
|
|
1181
|
+
Note: For edge cases, you can pass the cookie through by assigning it yourself -- e.g. `actor:
|
|
1182
|
+
request.cookie_jar.signed['example_id']`.
|
|
1183
|
+
The cookie name is the full experiment name (including any configured prefix) with `_id` appended -- e.g.
|
|
1184
|
+
`pill_color_id` for the `PillColorExperiment`.
|
|
523
1185
|
|
|
524
1186
|
### Client layer
|
|
525
1187
|
|
|
526
|
-
Experiments that have been run (or published) during the request lifecycle can be pushed into to the client layer by
|
|
1188
|
+
Experiments that have been run (or published) during the request lifecycle can be pushed into to the client layer by
|
|
1189
|
+
injecting the published experiments into javascript in a layout or view using something like:
|
|
527
1190
|
|
|
528
1191
|
```haml
|
|
529
1192
|
= javascript_tag(nonce: content_security_policy_nonce) do
|
|
530
1193
|
window.experiments = #{raw ApplicationExperiment.published_experiments.to_json};
|
|
531
1194
|
```
|
|
532
1195
|
|
|
533
|
-
The `window.experiments` object can then be used in your client implementation to determine experimental behavior at
|
|
1196
|
+
The `window.experiments` object can then be used in your client implementation to determine experimental behavior at
|
|
1197
|
+
that layer as well.
|
|
1198
|
+
For instance, we can now access the `window.experiments.pill_color` object to get the variant that was assigned, if the
|
|
1199
|
+
context was excluded, and to use the context key in our client side events.
|
|
1200
|
+
|
|
1201
|
+
## Adoption Guide for Organizations
|
|
1202
|
+
|
|
1203
|
+
### Phase 1: Foundation (Week 1-2)
|
|
1204
|
+
1. **Install and configure** the gem
|
|
1205
|
+
2. **Set up analytics integration** in the initializer
|
|
1206
|
+
3. **Create a base experiment class** for your organization
|
|
1207
|
+
4. **Run your first small experiment** (low-risk, high-visibility)
|
|
1208
|
+
|
|
1209
|
+
### Phase 2: Team Enablement (Week 3-4)
|
|
1210
|
+
1. **Document your organization's patterns** (naming, testing, rollout)
|
|
1211
|
+
2. **Train teams** on experiment lifecycle
|
|
1212
|
+
3. **Establish experiment review process** (hypothesis → implementation → analysis)
|
|
1213
|
+
4. **Run 2-3 experiments** across different teams
|
|
534
1214
|
|
|
535
|
-
|
|
1215
|
+
### Phase 3: Scale (Month 2+)
|
|
1216
|
+
1. **Integrate with feature flag system** (if applicable)
|
|
1217
|
+
2. **Build dashboards** for experiment monitoring
|
|
1218
|
+
3. **Establish data review cadence** (weekly experiment reviews)
|
|
1219
|
+
4. **Scale to 5-10 concurrent experiments**
|
|
1220
|
+
|
|
1221
|
+
### Common Pitfalls to Avoid
|
|
1222
|
+
|
|
1223
|
+
**❌ Don't: Run experiments without clear success metrics**
|
|
1224
|
+
```ruby
|
|
1225
|
+
class VagueExperiment < ApplicationExperiment
|
|
1226
|
+
# What are we trying to learn?
|
|
1227
|
+
control { :old_way }
|
|
1228
|
+
candidate { :new_way }
|
|
1229
|
+
end
|
|
1230
|
+
```
|
|
536
1231
|
|
|
537
|
-
|
|
1232
|
+
**✅ Do: Document hypothesis and success criteria**
|
|
1233
|
+
```ruby
|
|
1234
|
+
class CheckoutOptimizationExperiment < ApplicationExperiment
|
|
1235
|
+
# Hypothesis: Showing trust badges increases checkout completion
|
|
1236
|
+
# Success Metric: 5% increase in completion rate
|
|
1237
|
+
# Target: Free trial users
|
|
1238
|
+
# Duration: 2 weeks
|
|
1239
|
+
|
|
1240
|
+
control { :without_badges }
|
|
1241
|
+
candidate { :with_trust_badges }
|
|
1242
|
+
end
|
|
1243
|
+
```
|
|
538
1244
|
|
|
539
|
-
|
|
1245
|
+
**❌ Don't: Let experiments run indefinitely**
|
|
1246
|
+
- Set time bounds for every experiment
|
|
1247
|
+
- Review results at planned intervals
|
|
1248
|
+
- Make a decision: promote winner, revert, or iterate
|
|
540
1249
|
|
|
541
|
-
|
|
1250
|
+
**✅ Do: Build experiment cleanup into your process**
|
|
1251
|
+
- Schedule experiment review meetings
|
|
1252
|
+
- Archive experiment results
|
|
1253
|
+
- Clean up experiment code after conclusion
|
|
542
1254
|
|
|
543
|
-
|
|
1255
|
+
## Platform Configuration
|
|
544
1256
|
|
|
545
|
-
|
|
1257
|
+
The platform requires initial configuration to integrate with your analytics and infrastructure.
|
|
546
1258
|
|
|
547
|
-
|
|
1259
|
+
**Basic configuration** (in `config/initializers/gitlab_experiment.rb`):
|
|
548
1260
|
|
|
549
1261
|
```ruby
|
|
550
1262
|
Gitlab::Experiment.configure do |config|
|
|
551
|
-
|
|
1263
|
+
# How experiment events are tracked
|
|
1264
|
+
config.tracking_behavior = lambda do |event_name, **data|
|
|
1265
|
+
YourAnalytics.track(
|
|
1266
|
+
user_id: data[:key], # Anonymous experiment key
|
|
1267
|
+
event: "experiment_#{event_name}",
|
|
1268
|
+
properties: data
|
|
1269
|
+
)
|
|
1270
|
+
end
|
|
1271
|
+
|
|
1272
|
+
# How experiments are cached (recommended: Redis)
|
|
1273
|
+
config.cache = Gitlab::Experiment::Cache::RedisHashStore.new(
|
|
1274
|
+
Redis.new(url: ENV['REDIS_URL']),
|
|
1275
|
+
expires_in: 30.days
|
|
1276
|
+
)
|
|
1277
|
+
|
|
1278
|
+
# Optional: Prefix all experiment names
|
|
1279
|
+
config.name_prefix = 'mycompany'
|
|
1280
|
+
|
|
1281
|
+
# Optional: Default rollout strategy
|
|
1282
|
+
config.default_rollout = Gitlab::Experiment::Rollout::Percent.new
|
|
552
1283
|
end
|
|
553
1284
|
```
|
|
554
1285
|
|
|
555
|
-
|
|
1286
|
+
See the [complete initializer template](lib/generators/gitlab/experiment/install/templates/initializer.rb.tt) for all
|
|
1287
|
+
configuration options.
|
|
1288
|
+
|
|
1289
|
+
### Advanced: Caching Configuration
|
|
556
1290
|
|
|
557
|
-
|
|
1291
|
+
**Why caching matters:**
|
|
1292
|
+
- Ensures consistent user experience across sessions
|
|
1293
|
+
- Improves performance (skip rollout logic after first assignment)
|
|
1294
|
+
- Required for `only_assigned` functionality
|
|
1295
|
+
- Enables context migrations
|
|
558
1296
|
|
|
559
|
-
|
|
1297
|
+
**Cache options:**
|
|
1298
|
+
```ruby
|
|
1299
|
+
# Option 1: Use Rails cache (simple)
|
|
1300
|
+
Gitlab::Experiment.configure do |config|
|
|
1301
|
+
config.cache = Rails.cache
|
|
1302
|
+
end
|
|
560
1303
|
|
|
561
|
-
|
|
1304
|
+
# Option 2: Use Redis directly (recommended for scale)
|
|
1305
|
+
Gitlab::Experiment.configure do |config|
|
|
1306
|
+
config.cache = Gitlab::Experiment::Cache::RedisHashStore.new(
|
|
1307
|
+
Redis.new(url: ENV['REDIS_URL']),
|
|
1308
|
+
expires_in: 30.days
|
|
1309
|
+
)
|
|
1310
|
+
end
|
|
1311
|
+
|
|
1312
|
+
# Option 3: No caching (deterministic rollout strategies only)
|
|
1313
|
+
config.cache = nil
|
|
1314
|
+
```
|
|
562
1315
|
|
|
563
|
-
|
|
564
|
-
|
|
565
|
-
- [`Random`](lib/gitlab/experiment/rollout/random.rb): Random assignment can be useful on some experimentation
|
|
566
|
-
- [`RoundRobin`](lib/gitlab/experiment/rollout/round_robin.rb): Cycles through assignment using the cache to keep track of what was last assigned
|
|
1316
|
+
The gem includes the [`RedisHashStore`](lib/gitlab/experiment/cache/redis_hash_store.rb) cache store, which is
|
|
1317
|
+
documented in its implementation.
|
|
567
1318
|
|
|
568
|
-
|
|
1319
|
+
**Important:** Caching changes how rollout strategies behave. Once cached, subsequent calls return the cached value regardless of rollout strategy changes.
|
|
569
1320
|
|
|
570
|
-
|
|
1321
|
+
### Advanced: Custom Rollout Strategies
|
|
571
1322
|
|
|
572
|
-
|
|
1323
|
+
Build custom integrations with your existing infrastructure:
|
|
1324
|
+
|
|
1325
|
+
**Example: Flipper Integration**
|
|
573
1326
|
|
|
574
1327
|
```ruby
|
|
575
1328
|
# We put it in this module namespace so we can get easy resolution when
|
|
@@ -587,7 +1340,10 @@ module Gitlab::Experiment::Rollout
|
|
|
587
1340
|
end
|
|
588
1341
|
```
|
|
589
1342
|
|
|
590
|
-
So, Flipper needs something that responds to `flipper_id`, and since our experiment "id" (which is also our context key)
|
|
1343
|
+
So, Flipper needs something that responds to `flipper_id`, and since our experiment "id" (which is also our context key)
|
|
1344
|
+
is unique and consistent, we're going to give that to Flipper to manage things like percentage of actors etc.
|
|
1345
|
+
You might want to consider something more complex here if you're using things that can be flipper actors in your
|
|
1346
|
+
experiment context.
|
|
591
1347
|
|
|
592
1348
|
Anyway, now you can use your custom `Flipper` rollout strategy by instantiating it in configuration:
|
|
593
1349
|
|
|
@@ -608,7 +1364,9 @@ class PillColorExperiment < Gitlab::Experiment # OR ApplicationExperiment
|
|
|
608
1364
|
end
|
|
609
1365
|
```
|
|
610
1366
|
|
|
611
|
-
Now, enabling or disabling the Flipper feature flag will control if the experiment is enabled or not.
|
|
1367
|
+
Now, enabling or disabling the Flipper feature flag will control if the experiment is enabled or not.
|
|
1368
|
+
If the experiment is enabled, as determined by our custom rollout strategy, the standard resolution logic will be
|
|
1369
|
+
executed, and a variant (or control) will be assigned.
|
|
612
1370
|
|
|
613
1371
|
```ruby
|
|
614
1372
|
experiment(:pill_color).enabled? # => false
|
|
@@ -623,9 +1381,13 @@ experiment(:pill_color).assigned.name # => "red"
|
|
|
623
1381
|
|
|
624
1382
|
### Middleware
|
|
625
1383
|
|
|
626
|
-
There are times when you'll need to do link tracking in email templates, or markdown content -- or other places you
|
|
1384
|
+
There are times when you'll need to do link tracking in email templates, or markdown content -- or other places you
|
|
1385
|
+
won't be able to implement tracking.
|
|
1386
|
+
For these cases a middleware layer that can redirect to a given URL while also tracking that the URL was visited has
|
|
1387
|
+
been provided.
|
|
627
1388
|
|
|
628
|
-
In Rails this middleware is mounted automatically, with a base path of what's been configured for `mount_at`.
|
|
1389
|
+
In Rails this middleware is mounted automatically, with a base path of what's been configured for `mount_at`.
|
|
1390
|
+
If this path is nil, the middleware won't be mounted at all.
|
|
629
1391
|
|
|
630
1392
|
```ruby
|
|
631
1393
|
Gitlab::Experiment.configure do |config|
|
|
@@ -651,13 +1413,15 @@ experiment_redirect_url(ex, url: 'https//gitlab.com/docs') # => "https://gitlab.
|
|
|
651
1413
|
|
|
652
1414
|
## Testing (rspec support)
|
|
653
1415
|
|
|
654
|
-
This gem comes with some rspec helpers and custom matchers.
|
|
1416
|
+
This gem comes with some rspec helpers and custom matchers.
|
|
1417
|
+
To get the experiment specific rspec support, require the rspec support file:
|
|
655
1418
|
|
|
656
1419
|
```ruby
|
|
657
1420
|
require 'gitlab/experiment/rspec'
|
|
658
1421
|
```
|
|
659
1422
|
|
|
660
|
-
Any file in `spec/experiments` path will automatically get the experiment specific support, but it can also be included
|
|
1423
|
+
Any file in `spec/experiments` path will automatically get the experiment specific support, but it can also be included
|
|
1424
|
+
in other specs by adding the `:experiment` label:
|
|
661
1425
|
|
|
662
1426
|
```ruby
|
|
663
1427
|
describe MyExampleController do
|
|
@@ -759,7 +1523,8 @@ populate the cache naturally by running the experiment first to assign and cache
|
|
|
759
1523
|
|
|
760
1524
|
### Registered behaviors matcher
|
|
761
1525
|
|
|
762
|
-
It's useful to test our registered behaviors, as well as their return values when we implement anything complex in them.
|
|
1526
|
+
It's useful to test our registered behaviors, as well as their return values when we implement anything complex in them.
|
|
1527
|
+
The `register_behavior` matcher is useful for this.
|
|
763
1528
|
|
|
764
1529
|
```ruby
|
|
765
1530
|
it "tests our registered behaviors" do
|
|
@@ -792,7 +1557,8 @@ end
|
|
|
792
1557
|
|
|
793
1558
|
### Tracking matcher
|
|
794
1559
|
|
|
795
|
-
Tracking events is a major aspect of experimentation, and because of this we try to provide a flexible way to ensure
|
|
1560
|
+
Tracking events is a major aspect of experimentation, and because of this we try to provide a flexible way to ensure
|
|
1561
|
+
your tracking calls are covered.
|
|
796
1562
|
|
|
797
1563
|
```ruby
|
|
798
1564
|
before do
|
|
@@ -806,7 +1572,10 @@ it "tests that we track an event on a specific instance" do
|
|
|
806
1572
|
end
|
|
807
1573
|
```
|
|
808
1574
|
|
|
809
|
-
You can use the `on_next_instance` chain method to specify that the tracking call could happen on the next instance of
|
|
1575
|
+
You can use the `on_next_instance` chain method to specify that the tracking call could happen on the next instance of
|
|
1576
|
+
the experiment.
|
|
1577
|
+
This can be useful if you're calling `experiment(:example).track` downstream and don't have access to that instance.
|
|
1578
|
+
Here's a full example of the methods that can be chained onto the `track` matcher:
|
|
810
1579
|
|
|
811
1580
|
```ruby
|
|
812
1581
|
it "tests that we track an event with specific details" do
|
|
@@ -821,13 +1590,16 @@ end
|
|
|
821
1590
|
|
|
822
1591
|
## Tracking, anonymity and GDPR
|
|
823
1592
|
|
|
824
|
-
We generally try not to track things like user identifying values in our experimentation.
|
|
1593
|
+
We generally try not to track things like user identifying values in our experimentation.
|
|
1594
|
+
What we can and do track is the "experiment experience" (a.k.a. the context key).
|
|
825
1595
|
|
|
826
|
-
We generate this key from the context passed to the experiment.
|
|
1596
|
+
We generate this key from the context passed to the experiment.
|
|
1597
|
+
This allows creating funnels without exposing any user information.
|
|
827
1598
|
|
|
828
1599
|
This library attempts to be non-user-centric, in that a context can contain things like a user or a project.
|
|
829
1600
|
|
|
830
|
-
If you only include a user, that user would get the same experience across every project they view.
|
|
1601
|
+
If you only include a user, that user would get the same experience across every project they view.
|
|
1602
|
+
If you only include the project, every user who views that project would get the same experience.
|
|
831
1603
|
|
|
832
1604
|
Each of these approaches could be desirable given the objectives of your experiment.
|
|
833
1605
|
|
|
@@ -837,7 +1609,9 @@ After cloning the repo, run `bundle install` to install dependencies.
|
|
|
837
1609
|
|
|
838
1610
|
## Running tests
|
|
839
1611
|
|
|
840
|
-
The test suite requires Redis to be running.
|
|
1612
|
+
The test suite requires Redis to be running.
|
|
1613
|
+
[Install](https://redis.io/docs/latest/operate/oss_and_stack/install/archive/install-redis/) and start Redis
|
|
1614
|
+
(`redis-server`) before running tests.
|
|
841
1615
|
|
|
842
1616
|
Once Redis is running, execute the tests:
|
|
843
1617
|
`bundle exec rake`
|
|
@@ -846,9 +1620,12 @@ You can also run `bundle exec pry` for an interactive prompt that will allow you
|
|
|
846
1620
|
|
|
847
1621
|
## Contributing
|
|
848
1622
|
|
|
849
|
-
Bug reports and merge requests are welcome on GitLab at https://gitlab.com/gitlab-org/ruby/gems/gitlab-experiment.
|
|
1623
|
+
Bug reports and merge requests are welcome on GitLab at https://gitlab.com/gitlab-org/ruby/gems/gitlab-experiment.
|
|
1624
|
+
This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the
|
|
1625
|
+
[Contributor Covenant](http://contributor-covenant.org) code of conduct.
|
|
850
1626
|
|
|
851
|
-
Make sure to include a changelog entry in your commit message and read the [changelog entries
|
|
1627
|
+
Make sure to include a changelog entry in your commit message and read the [changelog entries
|
|
1628
|
+
section](https://docs.gitlab.com/ee/development/changelog.html).
|
|
852
1629
|
|
|
853
1630
|
## Release process
|
|
854
1631
|
|
|
@@ -860,6 +1637,5 @@ The gem is available as open source under the terms of the [MIT License](http://
|
|
|
860
1637
|
|
|
861
1638
|
## Code of conduct
|
|
862
1639
|
|
|
863
|
-
Everyone interacting in the `Gitlab::Experiment` project’s codebases, issue trackers, chat rooms and mailing lists is
|
|
864
|
-
|
|
865
|
-
***Make code not war***
|
|
1640
|
+
Everyone interacting in the `Gitlab::Experiment` project’s codebases, issue trackers, chat rooms and mailing lists is
|
|
1641
|
+
expected to follow the [code of conduct](CODE_OF_CONDUCT.md).
|