gitlab-experiment 1.2.0 → 1.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
data/README.md CHANGED
@@ -1,31 +1,165 @@
1
- GitLab Experiment
1
+ GitLab Experiment Platform
2
2
  =================
3
3
 
4
4
  <img alt="experiment" src="/uploads/60990b2dbf4c0406bbf8b7f998de2dea/experiment.png" align="right" width="40%">
5
5
 
6
- This README represents the current main branch and may not be applicable to the release you're using in your project. Please refer to the correct release branch if you'd like to review documentation relevant for that release.
6
+ **A comprehensive experimentation platform for building data-driven organizations**
7
7
 
8
- Here at GitLab we run experiments as A/B/n tests and review the data the experiment generates. From that data, we determine the best performing code path and promote it as the new default code path, or revert back to the original code path. You can read our [Experiment Guide](https://docs.gitlab.com/ee/development/experiment_guide/) documentation if you're curious about how we use this gem internally at GitLab.
8
+ GitLab Experiment is an enterprise-grade experimentation framework that enables teams to validate hypotheses, optimize
9
+ user experiences, and make evidence-based product decisions at scale.
10
+ Built on years of production experience at GitLab, this platform provides the foundation for a mature experimentation
11
+ culture across your entire organization.
9
12
 
10
- This library provides a clean and elegant DSL (domain specific language) to define, run, and track your experiments.
13
+ At GitLab, we run experiments as A/B/n tests and review the data they generate.
14
+ From that data, we determine the best performing code path and promote it as the new default, or revert back to the
15
+ original code path.
16
+ You can read our [Experiment Guide](https://docs.gitlab.com/ee/development/experiment_guide/) to learn how we use this
17
+ gem internally at GitLab.
11
18
 
12
- When we discuss the behavior of this gem, we'll use terms like experiment, context, control, candidate, and variant. It's worth defining these terms so they're more understood.
19
+ [[_TOC_]]
20
+
21
+ ## Why GitLab Experiment?
22
+
23
+ ### Built for Scale and Reliability
24
+ - **Production-tested** at GitLab scale with millions of users
25
+ - **Type-safe and testable** with comprehensive RSpec support
26
+ - **Framework agnostic** - works with Rails, Sinatra, or standalone Ruby applications
27
+ - **Redis-backed caching** for consistent user experiences across sessions
28
+ - **GDPR-compliant** with anonymous tracking and built-in DNT (Do Not Track) support
29
+
30
+ ### Designed for Teams
31
+ - **Developer-friendly DSL** that reads like natural language
32
+ - **Organized experiment classes** that live alongside your application code
33
+ - **Built-in generators** for rapid experiment creation
34
+ - **Comprehensive testing support** with custom RSpec matchers
35
+ - **Rails integration** with automatic middleware mounting and view helpers
36
+
37
+ ### Enterprise-Ready Features
38
+ - **Flexible rollout strategies** (percent-based, random, round-robin, or custom)
39
+ - **Advanced segmentation** to target specific user populations
40
+ - **Multi-variant testing** (A/B/n) with unlimited experimental paths
41
+ - **Progressive rollouts** with the `only_assigned` feature
42
+ - **Context migrations** for evolving experiments without losing data
43
+ - **Integration-ready** with existing feature flag systems (Flipper, Unleash, etc.)
44
+
45
+ <br clear="all">
46
+
47
+ ## Use Cases Across Your Organization
48
+
49
+ ### Product Teams: Optimize User Experiences
50
+ - **Onboarding flows**: Test different signup sequences to maximize activation
51
+ - **UI/UX changes**: Validate design decisions with real user behavior data
52
+ - **Feature rollouts**: Gradually release features to measure impact before full deployment
53
+ - **Pricing experiments**: Test different pricing strategies and messaging
54
+
55
+ ### Growth Teams: Drive Conversion
56
+ - **Call-to-action optimization**: Test button colors, copy, and placement
57
+ - **Landing page variations**: Experiment with different value propositions
58
+ - **Email campaigns**: A/B test subject lines and content
59
+ - **Trial conversion**: Optimize paths from trial to paid subscriptions
60
+
61
+ ### Engineering Teams: Safe Deployments
62
+ - **Performance optimizations**: Compare algorithm implementations under real load
63
+ - **Architecture changes**: Validate new code paths before full migration
64
+ - **API versions**: Run multiple API implementations side-by-side
65
+ - **Infrastructure experiments**: Test different caching or database strategies
66
+
67
+ ### Data Science Teams: Recommendation Systems
68
+ - **Algorithm tuning**: Compare ML model variations in production
69
+ - **Personalization**: Test different recommendation strategies
70
+ - **Search ranking**: Optimize search results based on user engagement
71
+ - **Content discovery**: Experiment with different content surfaces
72
+
73
+ ## Platform Capabilities
74
+
75
+ ### Core Experimentation Features
76
+
77
+ **Multi-variant Testing (A/B/n)**
78
+ Run experiments with any number of variants, not just A/B tests. Perfect for testing multiple approaches simultaneously.
13
79
 
14
- - `experiment` is any deviation of code paths we want to run sometimes and not others.
15
- - `context` is used to identify a consistent experience we'll provide in an experiment.
16
- - `control` is the default, or "original" code path.
17
- - `candidate` defines that there's one experimental code path.
18
- - `variant(s)` is used when more than one experimental code path exists.
19
- - `behaviors` is used to reference all possible code paths of an experiment.
80
+ **Smart Segmentation**
81
+ Route specific user populations to predetermined variants based on business rules, ensuring consistent experiences for
82
+ targeted groups.
83
+
84
+ **Progressive Rollouts**
85
+ Use the `only_assigned` feature to show experimental features only to users already in the experiment, enabling
86
+ controlled expansion.
87
+
88
+ **Context Flexibility**
89
+ Experiments can be sticky to users, projects, organizations, or any combination - enabling complex scenarios beyond
90
+ user-centric testing.
91
+
92
+ **Anonymous Tracking**
93
+ Built-in privacy protection with anonymous context keys, automatic cookie migration, and GDPR compliance.
94
+
95
+ **Automatic Assignment Tracking**
96
+ Every experiment automatically tracks an `:assignment` event when it runs - zero configuration required. Combined with
97
+ the anonymous context key, this gives your data team a complete picture of variant distribution and funnel entry without
98
+ any additional instrumentation.
99
+
100
+ **Client-Side Integration**
101
+ Seamlessly extend experiments to the frontend with JavaScript integration, enabling full-stack experimentation.
102
+
103
+ **Inline and Class-Based APIs**
104
+ Define experiments inline with blocks for quick iterations, or use dedicated experiment classes for complex logic - or
105
+ combine both. Class-based experiments define default behaviors that can be overridden inline at any call site, giving
106
+ teams the flexibility to start simple and evolve without rewriting:
107
+
108
+ ```ruby
109
+ experiment(:pill_color, actor: current_user) do |e|
110
+ e.control { '<strong>control</strong>' }
111
+ end
112
+ ```
20
113
 
21
- Candidate and variant are the same concept, but simplify how we speak about experimental paths -- both are widely referred to as the "experiment group". If you use "control and candidate," the assumption is an A/B test, and if you say "variants" the assumption is a multiple variant experiment.
114
+ **Context as a Design Framework**
115
+ Context is the most important design decision in any experiment. It determines stickiness, cache behavior, and how
116
+ events are correlated. Choose per-user context for personalization experiments, per-project for infrastructure tests,
117
+ per-group for organizational rollouts, or combine dimensions for precision targeting. This flexibility enables
118
+ experimentation strategies that go far beyond simple user-centric A/B tests.
119
+
120
+ **Decoupled Assignment with Publish**
121
+ Surface experiment assignments to the client layer without executing server-side behavior using `publish`. This enables
122
+ frontend-only experiments, pre-assignment in `before_action` hooks, and scenarios where variant data needs to be
123
+ available across the stack without triggering server-side code paths:
124
+
125
+ ```ruby
126
+ before_action -> { experiment(:pill_color, actor: current_user).publish }, only: [:show]
127
+ ```
128
+
129
+ ### Integration Ecosystem
130
+
131
+ **Feature Flag Integration**
132
+ Connect with existing feature flag systems like Flipper or Unleash through custom rollout strategies.
133
+
134
+ **Analytics Integration**
135
+ Flexible tracking callbacks integrate with any analytics platform - Snowplow, Amplitude, Mixpanel, or your data
136
+ warehouse.
137
+
138
+ **Monitoring and Observability**
139
+ Built-in logging and callbacks for integration with APM tools and monitoring systems.
140
+
141
+ **Email and Markdown**
142
+ Special middleware for tracking experiments in email links and static content.
143
+
144
+ ### Terminology
145
+
146
+ When we discuss the platform, we use specific terms that are worth understanding:
147
+
148
+ - **experiment** - Any deviation of code paths we want to test
149
+ - **context** - Identifies a consistent experience (user, project, session, etc.)
150
+ - **control** - The default or "original" code path
151
+ - **candidate** - One experimental code path (used in A/B tests)
152
+ - **variant(s)** - Multiple experimental paths (used in A/B/n tests)
153
+ - **behaviors** - All possible code paths (control + all variants)
154
+ - **rollout strategy** - Logic determining if an experiment is enabled and how variants are assigned
155
+ - **segmentation** - Rules for routing specific contexts to predetermined variants
156
+ - **exclusion** - Rules for keeping contexts out of experiments entirely
22
157
 
23
- Behaviors is a general term for all code paths -- if that's control and candidate, or control and variants. It's all of them.
24
158
  <br clear="all">
25
159
 
26
- [[_TOC_]]
160
+ ## Quick Start: From Zero to Experiment in 5 Minutes
27
161
 
28
- ## Installation
162
+ ### Installation
29
163
 
30
164
  Add the gem to your Gemfile and then `bundle install`.
31
165
 
@@ -33,204 +167,390 @@ Add the gem to your Gemfile and then `bundle install`.
33
167
  gem 'gitlab-experiment'
34
168
  ```
35
169
 
36
- If you're using Rails, you can install the initializer which provides basic configuration, documentation, and the base experiment class that all your experiments can inherit from.
170
+ If you're using Rails, install the initializer which provides basic configuration, documentation, and the base
171
+ experiment class:
37
172
 
38
173
  ```shell
39
174
  $ rails generate gitlab:experiment:install
40
175
  ```
41
176
 
42
- ## Implementing an experiment
177
+ ### Your First Experiment
43
178
 
44
- For the sake of having a simple example let's define an experiment around a button color, even if it's not how we'd likely implement a real experiment it makes for a clean example.
179
+ Let's create a real-world experiment to optimize a call-to-action button.
180
+ This example demonstrates the power of the platform while remaining practical.
45
181
 
46
- Currently our button is blue, but we want to test out a red variant. Our hypothesis is that a red button will make it more visible but also might appear like a warning button -- this is why we're testing our hypothesis.
182
+ #### Step 1: Generate the experiment
47
183
 
48
- So let's name our experiment `pill_color`, and use the generator to get some files:
184
+ **Hypothesis**: A more prominent call-to-action button will increase conversion rates
49
185
 
50
186
  ```shell
51
- $ rails generate gitlab:experiment pill_color
52
- ```
187
+ $ rails generate gitlab:experiment signup_cta
188
+ ```
53
189
 
54
- This generator will give us an `app/experiments/pill_color_experiment.rb` file, which we can use to define our experiment class and register our default variant behaviors. The generator also provides a bunch of useful comments in this file unless you `--skip-comments`, so feel free to play around with that.
190
+ This creates `app/experiments/signup_cta_experiment.rb` with helpful inline documentation.
55
191
 
56
- Let's fill out our control and candidate behaviors with some default values to start. For our example it'll just be some made up class names where we've defined our colors.
192
+ #### Step 2: Define your experiment class
57
193
 
58
194
  ```ruby
59
- class PillColorExperiment < ApplicationExperiment
60
- control { 'blue' } # register and define our default control value
61
- candidate { 'red' } # register and define our experimental candidate value
195
+ class SignupCtaExperiment < ApplicationExperiment
196
+ # Define the control (current experience)
197
+ control { 'btn-default' }
198
+
199
+ # Define the candidate (new experience to test)
200
+ candidate { 'btn-primary btn-lg' }
201
+
202
+ # Optional: Exclude certain users
203
+ exclude :existing_customers
204
+
205
+ # Optional: Track when the experiment runs
206
+ after_run :log_experiment_assignment
207
+
208
+ private
209
+
210
+ def existing_customers
211
+ context.actor&.subscribed?
212
+ end
213
+
214
+ def log_experiment_assignment
215
+ Rails.logger.info("User assigned to #{assigned.name} variant")
216
+ end
62
217
  end
63
218
  ```
64
219
 
65
- Now that we've defined our default behaviors, we can utilize our experiment elsewhere by calling the `experiment` method. When you run (or publish) an experiment you'll need to provide a name and a context. The name is how our `PillColorExperiment` class is resolved and will also show up in event data downstream. The context is a hash that's used to determine the variant assigned and should be consistent between calls.
66
-
67
- For our experiment we're going to make it "sticky" to the current user -- and if there isn't a current user, we want to assign and use a cookie value instead. This happens automatically if you use the [`actor` keyword](#cookies-and-the-actor-keyword) in your context. We use that in a lot of our examples, but it's by no means how everything should be done.
220
+ #### Step 3: Use the experiment in your view
68
221
 
69
222
  ```haml
70
- %button{ class: experiment(:pill_color, actor: current_user).run } Click Me!
223
+ -# The experiment is sticky to the current user
224
+ -# Anonymous users get a cookie-based assignment
225
+ %button{ class: experiment(:signup_cta, actor: current_user).run }
226
+ Start Free Trial
71
227
  ```
72
228
 
73
- Now when our view is rendered, the class attribute of the button will be pulled from the experiment and will be sticky to the user that's viewing it.
74
-
75
- You can also provide behavior overrides when you run the experiment. Let's use a view helper for this example because it'll be cleaner than putting our override blocks into the view.
229
+ #### Step 4: Track engagement
76
230
 
77
231
  ```ruby
78
- def pill_color_experiment
79
- experiment(:pill_color, actor: current_user) do |e|
80
- e.candidate { 'purple' } # override the candidate default of 'red'
81
- end
232
+ # In your controller, track when users click the button
233
+ def create_trial
234
+ experiment(:signup_cta, actor: current_user).track(:signup_completed)
235
+
236
+ # ... rest of your trial creation logic
82
237
  end
83
238
  ```
84
239
 
85
- Now we can run the experiment using that helper method. Instead of a red button, users who are assigned the candidate will be given a purple button. Users in the control group will still see the blue button. Using experiments in this way permits a default experiment to be defined while also allowing the experiment to be customized in the places where its run, using the scope, variables and helper methods that are available to us where we want to run the experiment.
240
+ **That's it!** Your experiment is now running, collecting data, and providing consistent experiences to your users.
86
241
 
87
- ```haml
88
- %button{ class: pill_color_experiment.run } Click Me!
89
- ```
242
+ ## Real-World Examples
243
+
244
+ ### Example 1: Onboarding Flow Optimization
90
245
 
91
- Understanding how an experiment can change user behavior or engagement is important in evaluating its performance. To this end, you'll probably want to track events that are important elsewhere in code. By using the same context you can provide a consistent experience and have the ability to anonymously track events to that experience.
246
+ **Business Context**: Product team wants to increase new user activation by testing different onboarding sequences.
92
247
 
93
248
  ```ruby
94
- experiment(:pill_color, actor: current_user).track(:clicked)
95
- ```
249
+ class OnboardingFlowExperiment < ApplicationExperiment
250
+ # Three different onboarding approaches
251
+ control { :standard_tour } # Current 5-step tour
252
+ variant(:quick) { :quick_start } # Streamlined 2-step flow
253
+ variant(:video) { :video_guide } # Video-based walkthrough
254
+
255
+ # Only show to new users who haven't completed onboarding
256
+ exclude :has_completed_onboarding
257
+
258
+ # Segment enterprise trial users to the standard tour
259
+ segment :enterprise_trial?, variant: :control
260
+
261
+ private
262
+
263
+ def has_completed_onboarding
264
+ context.actor&.onboarding_completed_at.present?
265
+ end
266
+
267
+ def enterprise_trial?
268
+ context.actor&.trial_type == 'enterprise'
269
+ end
270
+ end
96
271
 
97
- ## Advanced experimentation
272
+ # In your onboarding controller
273
+ def show
274
+ flow = experiment(:onboarding_flow, actor: current_user).run
275
+ render_onboarding_flow(flow)
276
+ end
98
277
 
99
- Now let's create a more complex experiment with multiple variants. This time we'll start with a grey button, and have red and blue variants.
278
+ # Track completion
279
+ def complete
280
+ experiment(:onboarding_flow, actor: current_user).track(:completed)
281
+ # ... mark user as onboarded
282
+ end
283
+ ```
100
284
 
101
- We'll also do more advanced segmentation when the experiment is run, exclude certain cases, and register callbacks that will be executed after our experiment is run. Here are some examples of what we can start to do in our experiment classes:
285
+ ### Example 2: Pricing Page Experiment
286
+
287
+ **Business Context**: Growth team wants to test whether showing annual savings increases annual plan selection.
102
288
 
103
289
  ```ruby
104
- class PillColorExperiment < Gitlab::Experiment # OR ApplicationExperiment
105
- # Register our behaviors.
106
- control # register our control, which will by default call #control_behavior
107
- variant(:red) # register the red variant that will call #red_behavior
108
- variant(:blue) # register the blue variant that will call #blue_behavior
290
+ class PricingDisplayExperiment < ApplicationExperiment
291
+ control { :monthly_default }
292
+ candidate { :annual_default_with_savings }
293
+
294
+ # Only run for unauthenticated visitors
295
+ exclude :authenticated_user
296
+
297
+ private
298
+
299
+ def authenticated_user
300
+ context.actor.present?
301
+ end
302
+ end
109
303
 
110
- # Exclude any users that are named "Richard".
111
- exclude :users_named_richard
304
+ # In your pricing view
305
+ - pricing_variant = experiment(:pricing_display, actor: current_user).run
306
+ = render "pricing/#{pricing_variant}"
112
307
 
113
- # Segment any account older than 2 weeks into the red variant without asking
114
- # the rollout strategy to assign a variant.
115
- segment :old_account?, variant: :red
116
-
117
- # After the experiment has been run, we want to log some performance metrics.
118
- after_run { log_performance_metrics }
308
+ # Track plan selections
309
+ def select_plan
310
+ experiment(:pricing_display, actor: current_user).track(:plan_selected,
311
+ value: params[:plan_type] == 'annual' ? 1 : 0
312
+ )
313
+ end
314
+ ```
315
+
316
+ ### Example 3: Algorithm Performance Test
119
317
 
318
+ **Business Context**: Engineering team wants to compare a new search algorithm's performance before full rollout.
319
+
320
+ ```ruby
321
+ class SearchAlgorithmExperiment < ApplicationExperiment
322
+ control { SearchEngine::Legacy }
323
+ candidate { SearchEngine::Neural }
324
+
325
+ # Only run for 25% of searches
326
+ default_rollout :percent, distribution: { control: 75, candidate: 25 }
327
+
328
+ # Exclude searches from API (higher SLA requirements)
329
+ exclude :api_request
330
+
331
+ # Track performance metrics
332
+ after_run :record_search_timing
333
+
120
334
  private
121
335
 
122
- # Define the default control behavior, which can be overridden on runs.
123
- def control_behavior
124
- 'grey'
336
+ def api_request
337
+ context.request&.path&.start_with?('/api/')
125
338
  end
126
-
127
- # Define the default red behavior, which can be overridden on runs.
128
- def red_behavior
129
- 'red'
339
+
340
+ def record_search_timing
341
+ # Custom metrics tracking
130
342
  end
343
+ end
131
344
 
132
- # Define the default blue behavior, which can be overridden on runs.
133
- def blue_behavior
134
- 'blue'
135
- end
345
+ # In your search service
346
+ def search(query)
347
+ algorithm = experiment(:search_algorithm,
348
+ actor: current_user,
349
+ project: current_project
350
+ ).run
351
+
352
+ results = algorithm.search(query)
353
+
354
+ experiment(:search_algorithm,
355
+ actor: current_user,
356
+ project: current_project
357
+ ).track(:search_completed, value: results.count)
358
+
359
+ results
360
+ end
361
+ ```
136
362
 
137
- # Define our special exclusion logic.
138
- def users_named_richard
139
- context.try(:actor)&.first_name == 'Richard' # use try for nil actors
140
- end
363
+ ### Example 4: Progressive Feature Rollout
364
+
365
+ **Business Context**: Launching a new AI-assisted code review feature, want to expand gradually to manage load and gather feedback.
141
366
 
142
- # Define our segmentation logic.
143
- def old_account?
144
- context.try(:actor) && context.actor.created_at < 2.weeks.ago
367
+ ```ruby
368
+ class AiCodeReviewExperiment < ApplicationExperiment
369
+ control { false } # Feature disabled
370
+ candidate { true } # Feature enabled
371
+
372
+ # Start with 5% rollout
373
+ default_rollout :percent, distribution: { control: 95, candidate: 5 }
374
+
375
+ # Segment beta program users to always get the feature
376
+ segment :beta_user?, variant: :candidate
377
+
378
+ # Exclude free tier (computational cost consideration)
379
+ exclude :free_tier_user
380
+
381
+ private
382
+
383
+ def beta_user?
384
+ context.actor&.beta_features_enabled?
145
385
  end
146
-
147
- # Let's assume that we've tracked this and want to push it into some system.
148
- def log_performance_metrics
149
- # ...hypothetical implementation
386
+
387
+ def free_tier_user
388
+ context.actor&.subscription_tier == 'free'
150
389
  end
151
390
  end
391
+
392
+ # In your merge request view
393
+ - if experiment(:ai_code_review, actor: current_user, project: @project).run
394
+ .ai-code-review-panel
395
+ = render 'ai_suggestions'
396
+
397
+ # Track usage
398
+ def apply_ai_suggestion
399
+ experiment(:ai_code_review, actor: current_user, project: @project)
400
+ .track(:suggestion_applied)
401
+ end
152
402
  ```
153
403
 
154
- You can play around with our new `PillColorExperiment` using a console or irb session. In our example we've used an actual `User` model with a first name and timestamps. Feel free to use something more appropriate for your project in your exploration, or just not pass in `actor` at all.
404
+ ## Platform Integration Patterns
155
405
 
156
- ```ruby
157
- include Gitlab::Experiment::Dsl
406
+ ### Integration with Feature Flags (Flipper)
158
407
 
159
- # The class will be looked up based on the experiment name provided.
160
- ex = experiment(:pill_color, actor: User.first) # => #<PillColorExperiment:0x...>
408
+ Many organizations already use feature flag systems. GitLab Experiment integrates seamlessly:
161
409
 
162
- # Run the experiment -- returning the result.
163
- ex.run # => "grey" (the value defined in our control)
410
+ ```ruby
411
+ module Gitlab::Experiment::Rollout
412
+ class Flipper < Percent
413
+ def enabled?
414
+ ::Flipper.enabled?(experiment.name, experiment_actor)
415
+ end
416
+
417
+ def experiment_actor
418
+ Struct.new(:flipper_id).new("Experiment;#{id}")
419
+ end
420
+ end
421
+ end
164
422
 
165
- # Track an event on the experiment we've defined, using the logic we've defined
166
- # in configuration.
167
- ex.track(:clicked) # => true
423
+ # Configure globally
424
+ Gitlab::Experiment.configure do |config|
425
+ config.default_rollout = Gitlab::Experiment::Rollout::Flipper.new
426
+ end
168
427
 
169
- # Publish the experiment without running it, using the logic we've defined in
170
- # configuration.
171
- ex.publish # => {:variant=>"control", :experiment=>"pill_color", :key=>"45f595...", :excluded=>false}
428
+ # Now Flipper controls your experiments
429
+ Flipper.enable_percentage_of_actors(:signup_cta, 50)
172
430
  ```
173
431
 
174
- <details>
175
- <summary>You can also specify the variant manually...</summary>
176
-
177
- Generally, defining segmentation rules is a better way to approach routing into specific variants, but it's possible to explicitly specify the variant when running an experiment.
432
+ ### Integration with Analytics Platforms
178
433
 
179
- Caching: It's important to understand what this might do to your data during rollout, so use this with careful consideration. Any time a specific variant is assigned manually, or through segmentation (including `:control`) it will be cached for that context. That means that if you manually assign `:control`, that context will never be moved out of the control unless you do it programmatically elsewhere.
434
+ Connect experiments to your analytics stack:
180
435
 
181
436
  ```ruby
182
- include Gitlab::Experiment::Dsl
437
+ Gitlab::Experiment.configure do |config|
438
+ config.tracking_behavior = lambda do |event_name, **data|
439
+ # Snowplow
440
+ SnowplowTracker.track_struct_event(
441
+ category: 'experiment',
442
+ action: event_name,
443
+ property: data[:experiment],
444
+ context: [{ schema: 'experiment_context', data: data }]
445
+ )
446
+
447
+ # Amplitude (example)
448
+ Amplitude.track(
449
+ user_id: data[:key], # Anonymous experiment key
450
+ event_type: "experiment_#{event_name}",
451
+ event_properties: data
452
+ )
453
+
454
+ # Custom data warehouse
455
+ DataWarehouse.log_experiment_event(event_name, data)
456
+ end
457
+ end
458
+ ```
183
459
 
184
- # Assign the candidate manually.
185
- ex = experiment(:pill_color, :red, actor: User.first) # => #<PillColorExperiment:0x..>
460
+ ### Multi-Application Consistency
186
461
 
187
- # Run the experiment -- returning the result.
188
- ex.run # => "red"
462
+ Share experiment assignments across multiple applications:
189
463
 
190
- # If caching is enabled this will remain sticky between calls.
191
- experiment(:pill_color, actor: User.first).run # => "red"
464
+ ```ruby
465
+ # Shared Redis cache
466
+ Gitlab::Experiment.configure do |config|
467
+ config.cache = Gitlab::Experiment::Cache::RedisHashStore.new(
468
+ Redis.new(url: ENV['REDIS_URL']),
469
+ expires_in: 30.days
470
+ )
471
+ end
472
+
473
+ # Now experiments stay consistent across your web app, API, and background jobs
192
474
  ```
193
475
 
194
- </details>
476
+ ## Advanced Features
195
477
 
196
- ### Exclusion rules
478
+ ### Multi-Variant (A/B/n) Testing
197
479
 
198
- Exclusion rules let us determine if a context should even be considered as something to include in an experiment. If
199
- we're excluding something, it means that we don't want to run the experiment in that case. This can be useful if you
200
- only want to run experiments on new users for instance.
480
+ Test multiple variations simultaneously to find the optimal solution:
201
481
 
202
482
  ```ruby
203
- class PillColorExperiment < Gitlab::Experiment # OR ApplicationExperiment
204
- # ...registered behaviors
205
-
206
- exclude :old_account?, ->{ context.actor.first_name == 'Richard' }
483
+ class NotificationStyleExperiment < ApplicationExperiment
484
+ # Test three different notification approaches
485
+ control { :banner } # Current: banner at top
486
+ variant(:toast) { :toast } # Toast notification
487
+ variant(:modal) { :modal } # Modal dialog
488
+
489
+ # Distribute traffic evenly across all three
490
+ default_rollout :percent,
491
+ distribution: { control: 34, toast: 33, modal: 33 }
492
+
493
+ # Exclude mobile users (different UI constraints)
494
+ exclude :mobile_user
495
+
496
+ # Segment power users to toast (less intrusive)
497
+ segment :power_user?, variant: :toast
498
+
499
+ private
500
+
501
+ def mobile_user
502
+ context.request&.user_agent&.match?(/Mobile/)
503
+ end
504
+
505
+ def power_user?
506
+ context.actor&.actions_count > 1000
507
+ end
207
508
  end
208
509
  ```
209
510
 
210
- In the previous example, we'll exclude all users named `'Richard'` as well as any account older than 2 weeks old. Not
211
- only will they be immediately given the control behavior, but no events will be tracked in these cases either.
511
+ ### Exclusion Rules
212
512
 
213
- Exclusion rules are executed in the order they're defined. The first exclusion rule to produce a truthy result will halt
214
- execution of further exclusion checks.
513
+ Keep contexts out of experiments entirely based on business rules:
215
514
 
216
- #### Excluding from within the experiment block
515
+ ```ruby
516
+ class FeatureExperiment < ApplicationExperiment
517
+ # Exclude existing customers (only test on prospects)
518
+ exclude :existing_customer
519
+
520
+ # Exclude during maintenance windows
521
+ exclude -> { context.project&.under_maintenance? }
522
+
523
+ # Exclude if feature is explicitly disabled
524
+ exclude :feature_disabled
525
+
526
+ private
527
+
528
+ def existing_customer
529
+ context.actor&.subscribed?
530
+ end
531
+
532
+ def feature_disabled
533
+ !FeatureFlag.enabled?(:allow_experiment, context.actor)
534
+ end
535
+ end
536
+ ```
217
537
 
218
- You can also exclude contexts dynamically from within the experiment block using the `exclude!` method. This provides a
219
- convenient way to include exclusion logic directly within the experiment call:
538
+ **Key behaviors:**
539
+ - Excluded users always receive the control experience
540
+ - No tracking events are recorded for excluded users
541
+ - Exclusion rules are evaluated in order, first match wins
542
+ - Exclusions improve performance by exiting early
543
+
544
+ **Inline exclusion** is also supported:
220
545
 
221
546
  ```ruby
222
- experiment(:pill_color, actor: current_user) do |e|
547
+ experiment(:feature, actor: current_user) do |e|
223
548
  e.exclude! unless can?(current_user, :manage, project)
224
-
225
- e.control { 'blue' }
226
- e.candidate { 'red' }
549
+ e.control { 'standard' }
550
+ e.candidate { 'enhanced' }
227
551
  end
228
552
  ```
229
553
 
230
- This approach keeps the experiment logic wrapped nicely within the experiment block, rather than requiring you to wrap
231
- the entire experiment call in conditional logic. When `exclude!` is called, the experiment will be excluded and return
232
- the control behavior without tracking any events.
233
-
234
554
  Note: Although tracking calls will be ignored on all exclusions, you may want to check exclusion yourself in expensive
235
555
  custom logic by calling the `should_track?` or `excluded?` methods.
236
556
 
@@ -240,146 +560,319 @@ future experiment run performance but can be a gotcha around caching.
240
560
  Note: Exclusion rules aren't the best way to determine if an experiment is enabled. There's an `enabled?` method that
241
561
  can be overridden to have a high-level way of determining if an experiment should be running and tracking at all. This
242
562
  `enabled?` check should be as efficient as possible because it's the first early opt out path an experiment can
243
- implement. This can be seen in [How it works](#how-it-works).
563
+ implement. This can be seen in [How Experiments Work (Technical)](#how-experiments-work-technical).
244
564
 
245
- ### Segmentation rules
565
+ ### Segmentation Rules
246
566
 
247
- Segmentation, or assigning certain variants in certain cases, is important to running experiments. This can be useful if you want to push a given population into a specific variant because you've already determined that variant is successful for that population.
567
+ Route specific populations to predetermined variants:
248
568
 
249
569
  ```ruby
250
- class PillColorExperiment < Gitlab::Experiment # OR ApplicationExperiment
251
- # ...registered behaviors
252
-
253
- segment(variant: :red) { context.actor.first_name == 'Richard' }
254
- segment :old_account?, variant: :blue
570
+ class NewFeatureExperiment < ApplicationExperiment
571
+ # Route VIP customers to the new feature
572
+ segment :vip_customer?, variant: :candidate
573
+
574
+ # Route enterprise trial users to the enhanced experience
575
+ segment :enterprise_trial?, variant: :candidate
576
+
577
+ # Route users from specific campaigns to specific variants
578
+ segment(variant: :candidate) { context.campaign == 'product_launch_2024' }
579
+
580
+ private
581
+
582
+ def vip_customer?
583
+ context.actor&.account_value > 100_000
584
+ end
585
+
586
+ def enterprise_trial?
587
+ context.actor&.trial_tier == 'enterprise'
588
+ end
255
589
  end
256
590
  ```
257
591
 
258
- In the previous example, any user named `'Richard'` would always receive the experience defined in the red variant. As well, any account older than 2 weeks would get the alternate experience defined in the blue variant.
259
-
260
- Segmentation rules are executed in the order they're defined. The first segmentation rule to produce a truthy result is the one which gets used to assign the variant. The remaining segmentation rules are skipped. This means that for our example, any user named `'Richard'` regardless of account age, will always be provided the experience as defined in the red variant.
592
+ **Key behaviors:**
593
+ - Segmentation rules are evaluated in order, first match wins
594
+ - Segmented assignments are cached for consistency
595
+ - Perfect for gradually expanding successful experiments
596
+ - Enables sophisticated population targeting
261
597
 
262
- ### Run callbacks
598
+ ### Lifecycle Callbacks
263
599
 
264
- Callbacks can be registered for when you want to execute logic before, after, or around when an experiment is run. These callbacks won't be called unless the experiment is actually run, meaning that exclusion rules take precedence.
600
+ Execute custom logic at different stages of experiment execution:
265
601
 
266
602
  ```ruby
267
- class PillColorExperiment < Gitlab::Experiment # OR ApplicationExperiment
268
- # ...registered behaviors
603
+ class PerformanceExperiment < ApplicationExperiment
604
+ # Run before the variant is determined
605
+ before_run :log_experiment_start
606
+
607
+ # Run after the variant is executed
608
+ after_run :record_timing_metrics, :notify_analytics_team
609
+
610
+ # Wrap the entire execution
611
+ around_run do |experiment, block|
612
+ start_time = Time.current
613
+ result = block.call
614
+ duration = Time.current - start_time
615
+
616
+ Metrics.record("experiment.#{experiment.name}.duration", duration)
617
+ result
618
+ end
269
619
 
270
- after_run :log_performance_metrics, -> { publish_to_database }
620
+ private
621
+
622
+ def log_experiment_start
623
+ Rails.logger.info("Starting experiment: #{name}")
624
+ end
625
+
626
+ def record_timing_metrics
627
+ # Custom timing logic
628
+ end
629
+
630
+ def notify_analytics_team
631
+ # Send to analytics platform
632
+ end
271
633
  end
272
634
  ```
273
635
 
274
- In the previous example, we're going to call the `log_performance_method`, and do a hypothetical publish to the database. If you want to do an `around_run`, you just need to call the block:
636
+ **Use cases for callbacks:**
637
+ - Performance monitoring and APM integration
638
+ - Custom analytics and data warehouse updates
639
+ - Experiment-specific logging and debugging
640
+ - Integration with external systems
641
+
642
+ ### Progressive Rollout with `only_assigned`
643
+
644
+ Control experiment expansion by only showing features to users already assigned to the experiment.
645
+ This is critical for managing blast radius and controlled rollouts:
646
+
647
+ **The Challenge**: You launch an experiment to 10% of new signups. Later, you want to show experimental features on other pages, but only to users already in the experiment - not expand to 10% of all users across the platform.
648
+
649
+ **The Solution**: Use `only_assigned: true`
275
650
 
276
651
  ```ruby
277
- class PillColorExperiment < Gitlab::Experiment # OR ApplicationExperiment
278
- # ...registered behaviors
652
+ # Step 1: Assign users during signup (10% of new signups)
653
+ class RegistrationsController < ApplicationController
654
+ def create
655
+ user = User.create!(user_params)
656
+
657
+ # This assigns 10% to candidate, 90% to control
658
+ experiment(:onboarding_v2, actor: user).publish
659
+
660
+ redirect_to dashboard_path
661
+ end
662
+ end
279
663
 
280
- around_run do |experiment, block|
281
- puts "- before #{experiment.name} run"
282
- block.call
283
- puts "- after #{experiment.name} run"
664
+ # Step 2: Later, show features only to those already assigned
665
+ class DashboardController < ApplicationController
666
+ def show
667
+ # This will NOT expand the experiment to 10% of all users
668
+ # Only users assigned in Step 1 will see the experimental UI
669
+ @show_new_features = experiment(:onboarding_v2,
670
+ actor: current_user,
671
+ only_assigned: true
672
+ ).assigned.name == 'candidate'
284
673
  end
285
674
  end
675
+
676
+ # Step 3: Show UI conditionally across the app
677
+ - if experiment(:onboarding_v2, actor: current_user, only_assigned: true).run
678
+ .new-onboarding-features
679
+ = render 'enhanced_dashboard'
286
680
  ```
287
681
 
288
- ### Checking assignment without assigning
682
+ **Behavior with `only_assigned: true`:**
683
+ - ✅ If user already assigned → returns their cached variant
684
+ - ✅ If user not assigned → returns control, no tracking
685
+ - ✅ Experiment reach stays controlled
686
+ - ✅ Perfect for multi-page experimental experiences
687
+
688
+ **Real-world use cases:**
689
+ - **Post-signup experiences**: Assign at signup, show features throughout the app
690
+ - **Gradual feature expansion**: Roll out to 5%, then add more touchpoints without expanding population
691
+ - **Cleanup phases**: Maintain experience for existing participants while preventing new assignments
692
+ - **A/B testing with multiple surfaces**: Test a hypothesis across multiple pages without assignment leakage
289
693
 
290
- Sometimes you may want to check if a user is already assigned to an experiment without assigning them to a variant if
291
- they're not. This is useful when you want to show experiment-specific content only to users who are already
292
- participating in the experiment, without expanding the experiment's reach.
694
+ ### Custom Rollout Strategies
293
695
 
294
- You can use the `only_assigned` option to achieve this behavior:
696
+ The platform supports multiple rollout strategies out of the box, and you can create custom strategies for your specific
697
+ needs.
295
698
 
296
- ```haml
297
- -# Only show experimental features to users already participating in the experiment
298
- - experiment(:advanced_search, actor: current_user, only_assigned: true) do |e|
299
- - e.control do
300
- = render 'search_filters'
301
- - e.candidate do
302
- = render 'advanced_search_filters'
699
+ **Built-in strategies:**
700
+ - [`Percent`](lib/gitlab/experiment/rollout/percent.rb) - Consistent percentage-based assignment (default, recommended)
701
+ - [`Random`](lib/gitlab/experiment/rollout/random.rb) - True random assignment (useful for load testing)
702
+ - [`RoundRobin`](lib/gitlab/experiment/rollout/round_robin.rb) - Cycle through variants (requires caching)
703
+ - [`Base`](lib/gitlab/experiment/rollout.rb) - Useful for building custom rollout strategies
704
+
705
+ ```ruby
706
+ class LoadTestExperiment < ApplicationExperiment
707
+ # Randomly test two different caching strategies
708
+ default_rollout :random
709
+
710
+ control { CacheStrategy::Redis }
711
+ candidate { CacheStrategy::Memcached }
712
+ end
713
+
714
+ class GradualRolloutExperiment < ApplicationExperiment
715
+ # Start with 5% in the new experience
716
+ default_rollout :percent,
717
+ distribution: { control: 95, candidate: 5 }
718
+ end
303
719
  ```
304
720
 
305
- When `only_assigned: true` is used:
721
+ See the [Advanced: Custom Rollout Strategies](#advanced-custom-rollout-strategies) section for building your own
722
+ integration with feature flag systems.
306
723
 
307
- - If the user has a cached variant assignment, the experiment runs normally and returns that variant
308
- - If the user has no cached variant assignment, the experiment is excluded and returns the control behavior
309
- - No new variant assignments are made
310
- - Tracking is disabled for excluded cases
311
- - Publishing still works to record the exclusion
724
+ ## Organizational Best Practices
312
725
 
313
- This is particularly useful for:
726
+ ### Experiment Lifecycle Management
314
727
 
315
- - **Progressive rollouts**: Show experimental features only to users already in the experiment
316
- - **Conditional UI**: Display experiment-specific UI elements only for assigned users
317
- - **Feature gates**: Check experiment participation without expanding the participant pool
318
- - **Cleanup phases**: Maintain experience for existing participants while preventing new assignments
319
- - **Post-registration experiences**: When users are assigned to experiments during registration, you can later show
320
- experimental features throughout their journey without expanding to existing users who weren't initially assigned
728
+ **1. Hypothesis Formation**
729
+ ```ruby
730
+ # Document your hypothesis in the experiment class
731
+ class CheckoutFlowExperiment < ApplicationExperiment
732
+ # Hypothesis: Reducing checkout steps from 3 to 2 will increase completion rate
733
+ # Success metric: 5% increase in checkout completion
734
+ # Target: All free trial users
735
+ # Duration: 2 weeks
736
+ # Owner: @growth-team
737
+
738
+ control { :three_step_checkout }
739
+ candidate { :two_step_checkout }
740
+
741
+ exclude :existing_customer
742
+ end
743
+ ```
321
744
 
745
+ **2. Gradual Rollout**
322
746
  ```ruby
323
- # During user registration - assign new users to experiment variants
324
- class RegistrationsController < ApplicationController
325
- def create
326
- # ... user creation logic
327
-
328
- # Assign new users to the experiment
329
- experiment(:pill_color, actor: @user)
747
+ # Week 1: 5% rollout
748
+ default_rollout :percent, distribution: { control: 95, candidate: 5 }
749
+
750
+ # Week 2: Increase to 25% if metrics look good
751
+ default_rollout :percent, distribution: { control: 75, candidate: 25 }
752
+
753
+ # Week 3: Full rollout if successful
754
+ default_rollout :percent, distribution: { control: 0, candidate: 100 }
755
+ ```
756
+
757
+ **3. Monitoring and Alerting**
758
+ ```ruby
759
+ class CriticalPathExperiment < ApplicationExperiment
760
+ after_run :monitor_performance
761
+ after_run :alert_on_errors
762
+
763
+ private
764
+
765
+ def monitor_performance
766
+ Metrics.increment("experiment.#{name}.#{assigned.name}")
767
+ end
768
+
769
+ def alert_on_errors
770
+ if context.error_rate > threshold
771
+ PagerDuty.alert("High error rate in #{name}")
772
+ end
330
773
  end
331
774
  end
332
775
  ```
333
776
 
334
- ```haml
335
- -# Later throughout the app - only show experimental features to assigned users
336
- - experiment(:pill_color, actor: current_user, only_assigned: true) do |e|
337
- - e.control do
338
- - e.candidate do
339
- = render 'quick_start_guide'
777
+ **4. Experiment Cleanup**
778
+ ```ruby
779
+ # When experiment is conclusive, clean up:
780
+ # 1. Remove the experiment code
781
+ # 2. Promote winner to production
782
+ # 3. Document learnings
783
+
784
+ # Before cleanup, archive results:
785
+ experiment(:checkout_flow).publish
786
+ # Export data for historical analysis
340
787
  ```
341
788
 
342
- You can also assign the result of the experiment to a variable:
789
+ ### Team Collaboration Patterns
343
790
 
791
+ **Product + Engineering + Data Science**
344
792
  ```ruby
345
- # In a view helper or directly in the view
346
- button_class = experiment(:pill_color, actor: current_user, only_assigned: true) do |e|
347
- e.control { 'btn-default' }
348
- e.candidate { 'btn-primary' }
349
- end.run
793
+ class CollaborativeExperiment < ApplicationExperiment
794
+ # Product defines the hypothesis and variants
795
+ control { :current_flow }
796
+ candidate { :new_flow }
797
+
798
+ # Engineering defines segmentation and rollout
799
+ segment :beta_users, variant: :candidate
800
+ default_rollout :percent, distribution: { control: 90, candidate: 10 }
801
+
802
+ # Data science defines tracking and metrics
803
+ after_run :track_funnel_step
804
+
805
+ def track_funnel_step
806
+ Analytics.track_experiment_step(
807
+ experiment: name,
808
+ variant: assigned.name,
809
+ funnel_position: context.step,
810
+ user_segment: context.actor&.segment
811
+ )
812
+ end
813
+ end
350
814
  ```
351
815
 
352
- Note: The `only_assigned` option requires caching to be enabled in your experiment configuration, as it relies on
353
- checking for cached variant assignments.
354
-
355
- ### Rollout strategies
816
+ ### Testing Strategy
356
817
 
357
- While a default rollout strategy can be defined in configuration, it's useful to be able to override this per experiment if needed. You can do this by specifying a specific `default_rollout` override in your experiment class.
818
+ Write tests for your experiments using the included RSpec matchers:
358
819
 
359
820
  ```ruby
360
- class PillColorExperiment < Gitlab::Experiment # OR ApplicationExperiment
361
- # ...registered behaviors
362
-
363
- default_rollout :random # randomly assign one of the registered behaviors
821
+ RSpec.describe CheckoutFlowExperiment do
822
+ describe 'segmentation' do
823
+ it 'routes existing customers to control' do
824
+ customer = create(:user, :with_subscription)
825
+ expect(experiment(:checkout_flow)).to exclude(actor: customer)
826
+ end
827
+
828
+ it 'routes enterprise trials to candidate' do
829
+ trial = create(:user, :enterprise_trial)
830
+ expect(experiment(:checkout_flow))
831
+ .to segment(actor: trial).into(:candidate)
832
+ end
833
+ end
834
+
835
+ describe 'tracking' do
836
+ it 'tracks checkout completion' do
837
+ expect(experiment(:checkout_flow)).to track(:completed)
838
+ .on_next_instance
839
+
840
+ CheckoutService.complete(user: user)
841
+ end
842
+ end
364
843
  end
365
844
  ```
366
845
 
367
- Obviously random assignment might not be the best rollout strategy for you, but you can define your own rollout strategies, or use one of the ones provided in the gem. You can read more about configuring the default rollout and how to write your own rollout strategies in the [configuration documentation](#rollout-strategies-1) for it.
846
+ ### Naming Conventions
847
+
848
+ Establish clear naming conventions for your organization:
849
+
850
+ ```ruby
851
+ # Good: Descriptive experiment names
852
+ class OnboardingFlowV2Experiment < ApplicationExperiment; end
853
+ class PricingPageAnnualFocusExperiment < ApplicationExperiment; end
854
+ class SearchAlgorithmNeuralExperiment < ApplicationExperiment; end
855
+
856
+ # Avoid: Vague names
857
+ class TestExperiment < ApplicationExperiment; end # What are we testing?
858
+ class ExperimentOne < ApplicationExperiment; end # No context
859
+ ```
368
860
 
369
- ## How it works
861
+ ## Technical Reference
370
862
 
371
- The way experiments work is best described using the following decision tree diagram. When an experiment is run, the
372
- following logic is executed to resolve what experience should be provided, given how the experiment is defined, and
373
- using the context passed to the experiment call.
863
+ ### How Experiments Work (Technical)
374
864
 
375
- Note: When using the `only_assigned` option, experiments that have no cached variant will be excluded, preventing new
376
- variant assignments while maintaining existing ones.
865
+ Understanding the experiment resolution flow helps you design better experiments and debug issues:
866
+
867
+ **Decision tree for variant assignment:**
377
868
 
378
869
  ```mermaid
379
870
  graph TD
380
871
  GP[General Pool/Population] --> Running?[Rollout Enabled?]
381
- Running? -->|Yes| Cached?[Cached? / Pre-segmented?]
872
+ Running? -->|Yes| Forced?[Forced Assignment?]
382
873
  Running? -->|No| Excluded[Control / No Tracking]
874
+ Forced? -->|Yes / Cached| ForcedVariant[Forced Variant]
875
+ Forced? -->|No| Cached?[Cached? / Pre-segmented?]
383
876
  Cached? -->|No| Excluded?
384
877
  Cached? -->|Yes| Cached[Cached Value]
385
878
  Excluded? -->|Yes / Cached| Excluded
@@ -391,50 +884,198 @@ graph TD
391
884
  Rollout -->|Cached| VariantB
392
885
  Rollout -->|Cached| VariantN
393
886
 
394
- class VariantA,VariantB,VariantN included
887
+ class ForcedVariant,VariantA,VariantB,VariantN included
395
888
  class Control,Excluded excluded
396
889
  class Cached cached
397
890
  ```
398
891
 
399
- ## Technical details
892
+ **Key points:**
893
+ 1. Rollout must be enabled for any variant assignment (including forced assignment)
894
+ 2. Forced assignment takes priority over cache/exclusion/segmentation (via `glex_force` query parameter)
895
+ 3. Cache provides consistency across calls
896
+ 4. Segmentation takes priority over rollout
897
+ 5. `only_assigned: true` exits early if no cache hit
898
+
899
+ ### Experiment Context and Stickiness
900
+
901
+ Internally, experiments have what's referred to as the context "key" that represents the unique and anonymous id of a
902
+ given context. This allows us to assign the same variant between different calls to the experiment, is used in caching
903
+ and can be used in event data downstream. This context "key" is how an experiment remains "sticky" to a given context,
904
+ and is an important aspect to understand.
905
+
906
+ **Context defines stickiness** - experiments remain consistent by generating an anonymous key from the context:
907
+
908
+ ```ruby
909
+ # Sticky to user - same user gets same variant everywhere
910
+ experiment(:feature, actor: current_user)
911
+
912
+ # Sticky to project - all users on a project get the same experience
913
+ experiment(:feature, project: project)
914
+
915
+ # Sticky to user+project - same user gets same variant per project
916
+ experiment(:feature, actor: current_user, project: project)
400
917
 
401
- This library is intended to be powerful and easy to use, which can lead to some complex underpinnings in the implementation. Some of those implementation details are important to understand at a technical level when considering how you want to design your experiments.
918
+ # Custom stickiness - explicitly define what creates consistency
919
+ experiment(:feature, actor: current_user, project: project, sticky_to: project)
920
+ ```
921
+
922
+ **The `actor` keyword has special behavior:**
923
+ - Anonymous users → temporary cookie-based assignment
924
+ - Upon sign-in → cookie migrates to user ID automatically
925
+ - Enables consistent experience across anonymous → authenticated journey
402
926
 
403
- ### Including the DSL
927
+ ### Using Experiments Beyond Views
404
928
 
405
- By default, `Gitlab::Experiment` injects itself into the controller, view, and mailer layers. This exposes the `experiment` method application wide in those layers. Some experiments may extend outside of those layers however, so you may want to include it elsewhere. For instance in an irb session or the rails console, or in all your service objects, background jobs, or similar.
929
+ By default, `Gitlab::Experiment` injects itself into the controller, view, and mailer layers. This exposes the
930
+ `experiment` method application wide in those layers. Some experiments may extend outside of those layers however, so
931
+ you may want to include it elsewhere. For instance in an irb session or the rails console, or in all your service
932
+ objects, background jobs, or similar:
406
933
 
407
934
  ```ruby
935
+ # In all background jobs
408
936
  class ApplicationJob < ActiveJob::Base
409
- include Gitlab::Experiment::Dsl # include the `experiment` method for all jobs
937
+ include Gitlab::Experiment::Dsl
938
+ end
939
+
940
+ # In service objects
941
+ class ApplicationService
942
+ include Gitlab::Experiment::Dsl
410
943
  end
944
+
945
+ # In a console session
946
+ include Gitlab::Experiment::Dsl
947
+ experiment(:feature, actor: User.first).run
411
948
  ```
412
949
 
413
- ### Experiment stickiness
950
+ ### Manual Variant Assignment
414
951
 
415
- Internally, experiments have what's referred to as the context "key" that represents the unique and anonymous id of a given context. This allows us to assign the same variant between different calls to the experiment, is used in caching and can be used in event data downstream. This context "key" is how an experiment remains "sticky" to a given context, and is an important aspect to understand.
952
+ <details>
953
+ <summary>You can also specify the variant manually...</summary>
416
954
 
417
- You can specify what the experiment should be sticky to by providing the `:sticky_to` option. By default this will be the entire context, but this can be overridden manually if needed.
955
+ Generally, defining segmentation rules is a better way to approach routing into specific variants, but it's possible to
956
+ explicitly specify the variant when running an experiment.
418
957
 
419
- In a fabricated example, we might want to provide the user and a project to an experiment, but we want it to remain sticky to the project that the user is viewing and not the user viewing that project. This is a very powerful concept that you're encouraged to play around with to understand what it means, and how you can run complex experiments that aren't user centric.
958
+ Caching: It's important to understand what this might do to your data during rollout, so use this with careful
959
+ consideration. Any time a specific variant is assigned manually, or through segmentation (including `:control`) it will
960
+ be cached for that context. That means that if you manually assign `:control`, that context will never be moved out of
961
+ the control unless you do it programmatically elsewhere.
420
962
 
421
963
  ```ruby
422
- experiment(:example, actor: current_user, project: project, sticky_to: project)
964
+ include Gitlab::Experiment::Dsl
965
+
966
+ # Assign the candidate manually.
967
+ ex = experiment(:pill_color, :red, actor: User.first) # => #<PillColorExperiment:0x..>
968
+
969
+ # Run the experiment -- returning the result.
970
+ ex.run # => "red"
971
+
972
+ # If caching is enabled this will remain sticky between calls.
973
+ experiment(:pill_color, actor: User.first).run # => "red"
423
974
  ```
424
975
 
425
- ### Experiment signature
976
+ </details>
977
+
978
+ ### Forced Variant Assignment (QA/UAT)
426
979
 
427
- The best way to understand the details of an experiment is through its signature. An example signature can be retrieved by calling the `signature` method, and looks like the following:
980
+ For testing and validation purposes, you can force a specific variant assignment via a URL query parameter. This is
981
+ useful for QA testing in staging or production environments where you need to verify a specific variant's behavior.
982
+
983
+ **Configuration:**
984
+
985
+ Forced assignment is disabled by default. Enable it in your initializer:
986
+
987
+ ```ruby
988
+ Gitlab::Experiment.configure do |config|
989
+ config.allow_forced_assignment = true
990
+ end
991
+ ```
992
+
993
+ **Usage:**
994
+
995
+ Append the `glex_force` query parameter to any URL with the format `experiment_name:variant_name`:
996
+
997
+ ```
998
+ https://your-app.com/signup?glex_force=myapp_signup_cta:candidate
999
+ ```
1000
+
1001
+ The forced variant is written to the cache (Redis) on the same request, making it permanent for that context. The
1002
+ query parameter only needs to be provided once -- after that, the variant is persisted in the cache like any normal
1003
+ assignment.
1004
+
1005
+ #### Anonymous user (nil actor) -- initial assignment
1006
+
1007
+ This is the primary use case for QA testing signup flows and landing pages. The user is not signed in, so the actor is
1008
+ nil and the experiment uses a cookie-based context key.
1009
+
1010
+ 1. Anonymous user visits `https://your-app.com/signup?glex_force=signup_cta:candidate`
1011
+ 2. The forced variant `:candidate` is written to Redis under the cookie-based context key
1012
+ 3. The user signs in -- the standard cookie migration carries the forced variant to their real identity
1013
+ 4. All future requests use `:candidate` from Redis, permanently
1014
+
1015
+ This means a QA tester can force a variant before signup and have it follow the user through the entire
1016
+ anonymous-to-authenticated journey.
1017
+
1018
+ #### Signed-in user -- initial assignment
1019
+
1020
+ When a signed-in user hasn't been assigned a variant yet, the force param assigns and caches it immediately:
1021
+
1022
+ ```
1023
+ https://your-app.com/dashboard?glex_force=new_feature:candidate
1024
+ ```
1025
+
1026
+ The variant is cached under the user's context key on this request. No further query parameter is needed.
1027
+
1028
+ #### Signed-in user -- re-assignment (overwriting an existing variant)
1029
+
1030
+ If a user was previously assigned `:control` (by the rollout strategy or a prior force), the force param overwrites
1031
+ the cached value:
1032
+
1033
+ ```
1034
+ https://your-app.com/dashboard?glex_force=new_feature:candidate
1035
+ ```
1036
+
1037
+ The existing `:control` assignment in Redis is replaced with `:candidate`. This is useful when QA needs to switch a
1038
+ user between variants without clearing the cache manually.
1039
+
1040
+ #### Disabled experiments and feature flags
1041
+
1042
+ Forced assignment requires the experiment to be enabled. If the experiment is disabled (as determined by the rollout
1043
+ strategy's `enabled?` method), the `glex_force` parameter is ignored and normal resolution applies (which will assign
1044
+ control).
1045
+
1046
+ This is intentional -- a disabled experiment may be disabled for valid reasons (incomplete implementation, known issues,
1047
+ compliance constraints, etc.) and force assignment should not provide a way to bypass that decision. To use forced
1048
+ assignment, ensure the experiment is enabled first through your rollout strategy.
1049
+
1050
+ **Important notes:**
1051
+ - The experiment name in the parameter must match the full experiment name (including any configured `name_prefix`).
1052
+ - If the variant name doesn't match a registered behavior, the forced assignment is ignored and normal variant resolution
1053
+ proceeds (typically resulting in the control variant).
1054
+ - Forced assignment does not override a variant that was already set via the constructor or an explicit `assigned()`
1055
+ call within the same request.
1056
+ - This feature requires a `request` object with `params` to be available in the experiment context.
1057
+
1058
+ > [!NOTE]
1059
+ > Because forcing the variant ignores the exclusion/segmentation process it will cover up those types of errors so if your experiment relies on these types of logic this testing method should be avoided.
1060
+
1061
+ ### Experiment Signature
1062
+
1063
+ The best way to understand the details of an experiment is through its signature. An example signature can be retrieved
1064
+ by calling the `signature` method, and looks like the following:
428
1065
 
429
1066
  ```ruby
430
1067
  experiment(:example).signature # => {:variant=>"control", :experiment=>"example", :key=>"4d7aee..."}
431
1068
  ```
432
1069
 
433
- An experiment signature is useful when tracking events and when using experiments on the client layer. The signature can also contain the optional `migration_keys`, and `excluded` properties.
1070
+ An experiment signature is useful when tracking events and when using experiments on the client layer. The signature can
1071
+ also contain the optional `migration_keys`, and `excluded` properties.
434
1072
 
435
- ### Return value
1073
+ ### Return Value
436
1074
 
437
- By default the return value of calling `experiment` is a `Gitlab::Experiment` instance, or whatever class the experiment is resolved to, which likely inherits from `Gitlab::Experiment`. In simple cases you may want only the results of running the experiment though. You can call `run` within the block to get the return value of the assigned variant.
1075
+ By default the return value of calling `experiment` is a `Gitlab::Experiment` instance, or whatever class the
1076
+ experiment is resolved to, which likely inherits from `Gitlab::Experiment`. In simple cases you may want only the
1077
+ results of running the experiment though. You can call `run` within the block to get the return value of the assigned
1078
+ variant.
438
1079
 
439
1080
  ```ruby
440
1081
  # Normally an experiment instance.
@@ -453,11 +1094,14 @@ end # => 'A'
453
1094
 
454
1095
  ### Context migrations
455
1096
 
456
- There are times when we need to change context while an experiment is running. We make this possible by passing the migration data to the experiment.
1097
+ There are times when we need to change context while an experiment is running.
1098
+ We make this possible by passing the migration data to the experiment.
457
1099
 
458
- Take for instance, that you might be using `version: 1` in your context currently. To migrate this to `version: 2`, provide the portion of the context you wish to change using a `migrated_with` option.
1100
+ Take for instance, that you might be using `version: 1` in your context currently.
1101
+ To migrate this to `version: 2`, provide the portion of the context you wish to change using a `migrated_with` option.
459
1102
 
460
- In providing the context migration data, we can resolve an experience and its events all the way back. This can also help in keeping our cache relevant.
1103
+ In providing the context migration data, we can resolve an experience and its events all the way back.
1104
+ This can also help in keeping our cache relevant.
461
1105
 
462
1106
  ```ruby
463
1107
  # First implementation.
@@ -467,7 +1111,8 @@ experiment(:example, actor: current_user, version: 1)
467
1111
  experiment(:example, actor: current_user, version: 2, migrated_with: { version: 1 })
468
1112
  ```
469
1113
 
470
- You can add or remove context by providing a `migrated_from` option. This approach expects a full context replacement -- i.e. what it was before you added or removed the new context key.
1114
+ You can add or remove context by providing a `migrated_from` option.
1115
+ This approach expects a full context replacement -- i.e. what it was before you added or removed the new context key.
471
1116
 
472
1117
  If you wanted to introduce a `version` to your context, provide the full previous context.
473
1118
 
@@ -479,7 +1124,9 @@ experiment(:example, actor: current_user)
479
1124
  experiment(:example, actor: current_user, version: 1, migrated_from: { actor: current_user })
480
1125
  ```
481
1126
 
482
- When you migrate context, this information is included in the signature of the experiment. This can be used downstream in event handling and reporting to resolve a series of events back to a single experience, while also keeping everything anonymous.
1127
+ When you migrate context, this information is included in the signature of the experiment.
1128
+ This can be used downstream in event handling and reporting to resolve a series of events back to a single experience,
1129
+ while also keeping everything anonymous.
483
1130
 
484
1131
  An example of our experiment signature when we migrate would include the `migration_keys` property:
485
1132
 
@@ -493,11 +1140,16 @@ ex.signature # => {:key=>"9e9d93...", :migration_keys=>["20d69a..."], ...}
493
1140
 
494
1141
  ### Cookies and the actor keyword
495
1142
 
496
- We use cookies to auto migrate an unknown value into a known value, often in the case of the current user. The implementation of this uses the same concept outlined above with context migrations, but will happen automatically for you if you use the `actor` keyword.
1143
+ We use cookies to auto migrate an unknown value into a known value, often in the case of the current user.
1144
+ The implementation of this uses the same concept outlined above with context migrations, but will happen automatically
1145
+ for you if you use the `actor` keyword.
497
1146
 
498
- When you use the `actor: current_user` pattern in your context, the nil case is handled by setting a special cookie for the experiment and then deleting the cookie, and migrating the context key to the one generated from the user when they've signed in.
1147
+ When you use the `actor: current_user` pattern in your context, the nil case is handled by setting a special cookie for
1148
+ the experiment and then deleting the cookie, and migrating the context key to the one generated from the user when
1149
+ they've signed in.
499
1150
 
500
- This cookie is a temporary, randomized uuid and isn't associated with a user. When we can finally provide an actor, the context is auto migrated from the cookie to that actor.
1151
+ This cookie is a temporary, randomized uuid and isn't associated with a user.
1152
+ When we can finally provide an actor, the context is auto migrated from the cookie to that actor.
501
1153
 
502
1154
  ```ruby
503
1155
  # The actor key is not present, so no cookie is set.
@@ -511,65 +1163,166 @@ experiment(:example, actor: nil, project: project)
511
1163
  experiment(:example, actor: current_user, project: project)
512
1164
  ```
513
1165
 
514
- Note: The cookie is deleted when resolved, but can be assigned again if the `actor` is ever nil again. A good example of this scenario would be on a sign in page. When a potential user arrives, they would never be known, so a cookie would be set for them, and then resolved/removed as soon as they signed in. This process would repeat each time they arrived while not being signed in and can complicate reporting unless it's handled well in the data layers.
1166
+ Note: The cookie is deleted when resolved, but can be assigned again if the `actor` is ever nil again.
1167
+ A good example of this scenario would be on a sign in page.
1168
+ When a potential user arrives, they would never be known, so a cookie would be set for them, and then resolved/removed
1169
+ as soon as they signed in.
1170
+ This process would repeat each time they arrived while not being signed in and can complicate reporting unless it's
1171
+ handled well in the data layers.
515
1172
 
516
- Note: To read and write cookies, we provide the `request` from within the controller and views. The cookie migration will happen automatically if the experiment is within those layers. You'll need to provide the `request` as an option to the experiment if it's outside of the controller and views.
1173
+ Note: To read and write cookies, we provide the `request` from within the controller and views.
1174
+ The cookie migration will happen automatically if the experiment is within those layers.
1175
+ You'll need to provide the `request` as an option to the experiment if it's outside of the controller and views.
517
1176
 
518
1177
  ```ruby
519
1178
  experiment(:example, actor: current_user, request: request)
520
1179
  ```
521
1180
 
522
- Note: For edge cases, you can pass the cookie through by assigning it yourself -- e.g. `actor: request.cookie_jar.signed['example_id']`. The cookie name is the full experiment name (including any configured prefix) with `_id` appended -- e.g. `pill_color_id` for the `PillColorExperiment`.
1181
+ Note: For edge cases, you can pass the cookie through by assigning it yourself -- e.g. `actor:
1182
+ request.cookie_jar.signed['example_id']`.
1183
+ The cookie name is the full experiment name (including any configured prefix) with `_id` appended -- e.g.
1184
+ `pill_color_id` for the `PillColorExperiment`.
523
1185
 
524
1186
  ### Client layer
525
1187
 
526
- Experiments that have been run (or published) during the request lifecycle can be pushed into to the client layer by injecting the published experiments into javascript in a layout or view using something like:
1188
+ Experiments that have been run (or published) during the request lifecycle can be pushed into to the client layer by
1189
+ injecting the published experiments into javascript in a layout or view using something like:
527
1190
 
528
1191
  ```haml
529
1192
  = javascript_tag(nonce: content_security_policy_nonce) do
530
1193
  window.experiments = #{raw ApplicationExperiment.published_experiments.to_json};
531
1194
  ```
532
1195
 
533
- The `window.experiments` object can then be used in your client implementation to determine experimental behavior at that layer as well. For instance, we can now access the `window.experiments.pill_color` object to get the variant that was assigned, if the context was excluded, and to use the context key in our client side events.
1196
+ The `window.experiments` object can then be used in your client implementation to determine experimental behavior at
1197
+ that layer as well.
1198
+ For instance, we can now access the `window.experiments.pill_color` object to get the variant that was assigned, if the
1199
+ context was excluded, and to use the context key in our client side events.
1200
+
1201
+ ## Adoption Guide for Organizations
1202
+
1203
+ ### Phase 1: Foundation (Week 1-2)
1204
+ 1. **Install and configure** the gem
1205
+ 2. **Set up analytics integration** in the initializer
1206
+ 3. **Create a base experiment class** for your organization
1207
+ 4. **Run your first small experiment** (low-risk, high-visibility)
1208
+
1209
+ ### Phase 2: Team Enablement (Week 3-4)
1210
+ 1. **Document your organization's patterns** (naming, testing, rollout)
1211
+ 2. **Train teams** on experiment lifecycle
1212
+ 3. **Establish experiment review process** (hypothesis → implementation → analysis)
1213
+ 4. **Run 2-3 experiments** across different teams
534
1214
 
535
- ## Configuration
1215
+ ### Phase 3: Scale (Month 2+)
1216
+ 1. **Integrate with feature flag system** (if applicable)
1217
+ 2. **Build dashboards** for experiment monitoring
1218
+ 3. **Establish data review cadence** (weekly experiment reviews)
1219
+ 4. **Scale to 5-10 concurrent experiments**
1220
+
1221
+ ### Common Pitfalls to Avoid
1222
+
1223
+ **❌ Don't: Run experiments without clear success metrics**
1224
+ ```ruby
1225
+ class VagueExperiment < ApplicationExperiment
1226
+ # What are we trying to learn?
1227
+ control { :old_way }
1228
+ candidate { :new_way }
1229
+ end
1230
+ ```
536
1231
 
537
- The gem is meant to be configured when installed and is ambiguous about how it should track events, and what to do when publishing experiments. Some of the more important aspects are left up to you to implement the logic that's right for your project.
1232
+ **✅ Do: Document hypothesis and success criteria**
1233
+ ```ruby
1234
+ class CheckoutOptimizationExperiment < ApplicationExperiment
1235
+ # Hypothesis: Showing trust badges increases checkout completion
1236
+ # Success Metric: 5% increase in completion rate
1237
+ # Target: Free trial users
1238
+ # Duration: 2 weeks
1239
+
1240
+ control { :without_badges }
1241
+ candidate { :with_trust_badges }
1242
+ end
1243
+ ```
538
1244
 
539
- Simple documentation can be found in the provided [initializer](lib/generators/gitlab/experiment/install/templates/initializer.rb.tt).
1245
+ **❌ Don't: Let experiments run indefinitely**
1246
+ - Set time bounds for every experiment
1247
+ - Review results at planned intervals
1248
+ - Make a decision: promote winner, revert, or iterate
540
1249
 
541
- Read on for comprehensive documentation on some of the more complex configuration options.
1250
+ **✅ Do: Build experiment cleanup into your process**
1251
+ - Schedule experiment review meetings
1252
+ - Archive experiment results
1253
+ - Clean up experiment code after conclusion
542
1254
 
543
- ### Caching
1255
+ ## Platform Configuration
544
1256
 
545
- Caching can be enabled in configuration and is implemented towards the `Rails.cache` / `ActiveSupport::Cache::Store` interface. When you enable caching, any variant resolution will be cached when something other than nil. Migrating the cache through context migrations is handled automatically, and this helps ensure an experiment experience remains consistent.
1257
+ The platform requires initial configuration to integrate with your analytics and infrastructure.
546
1258
 
547
- It's important to understand that using caching can drastically change or negate your specific rollout strategy logic.
1259
+ **Basic configuration** (in `config/initializers/gitlab_experiment.rb`):
548
1260
 
549
1261
  ```ruby
550
1262
  Gitlab::Experiment.configure do |config|
551
- config.cache = Rails.cache
1263
+ # How experiment events are tracked
1264
+ config.tracking_behavior = lambda do |event_name, **data|
1265
+ YourAnalytics.track(
1266
+ user_id: data[:key], # Anonymous experiment key
1267
+ event: "experiment_#{event_name}",
1268
+ properties: data
1269
+ )
1270
+ end
1271
+
1272
+ # How experiments are cached (recommended: Redis)
1273
+ config.cache = Gitlab::Experiment::Cache::RedisHashStore.new(
1274
+ Redis.new(url: ENV['REDIS_URL']),
1275
+ expires_in: 30.days
1276
+ )
1277
+
1278
+ # Optional: Prefix all experiment names
1279
+ config.name_prefix = 'mycompany'
1280
+
1281
+ # Optional: Default rollout strategy
1282
+ config.default_rollout = Gitlab::Experiment::Rollout::Percent.new
552
1283
  end
553
1284
  ```
554
1285
 
555
- The gem includes the following cache stores, which are documented in the implementation:
1286
+ See the [complete initializer template](lib/generators/gitlab/experiment/install/templates/initializer.rb.tt) for all
1287
+ configuration options.
1288
+
1289
+ ### Advanced: Caching Configuration
556
1290
 
557
- - [`RedisHashStore`](lib/gitlab/experiment/cache/redis_hash_store.rb): Useful if using redis
1291
+ **Why caching matters:**
1292
+ - Ensures consistent user experience across sessions
1293
+ - Improves performance (skip rollout logic after first assignment)
1294
+ - Required for `only_assigned` functionality
1295
+ - Enables context migrations
558
1296
 
559
- ### Rollout strategies
1297
+ **Cache options:**
1298
+ ```ruby
1299
+ # Option 1: Use Rails cache (simple)
1300
+ Gitlab::Experiment.configure do |config|
1301
+ config.cache = Rails.cache
1302
+ end
560
1303
 
561
- There are some basic rollout strategies that come with the gem, and you can use these directly, or you can use them to help build your own custom rollout strategies. Each is documented more thoroughly in its implementation file.
1304
+ # Option 2: Use Redis directly (recommended for scale)
1305
+ Gitlab::Experiment.configure do |config|
1306
+ config.cache = Gitlab::Experiment::Cache::RedisHashStore.new(
1307
+ Redis.new(url: ENV['REDIS_URL']),
1308
+ expires_in: 30.days
1309
+ )
1310
+ end
1311
+
1312
+ # Option 3: No caching (deterministic rollout strategies only)
1313
+ config.cache = nil
1314
+ ```
562
1315
 
563
- - [`Base`](lib/gitlab/experiment/rollout.rb): Useful for building custom rollout strategies, not super useful by itself
564
- - [`Percent`](lib/gitlab/experiment/rollout/percent.rb): A comprehensive percent based strategy, it's configured as the default
565
- - [`Random`](lib/gitlab/experiment/rollout/random.rb): Random assignment can be useful on some experimentation
566
- - [`RoundRobin`](lib/gitlab/experiment/rollout/round_robin.rb): Cycles through assignment using the cache to keep track of what was last assigned
1316
+ The gem includes the [`RedisHashStore`](lib/gitlab/experiment/cache/redis_hash_store.rb) cache store, which is
1317
+ documented in its implementation.
567
1318
 
568
- The included rollout strategies are great, but you might want to write your own for your own project, which might already have nice tooling for toggling feature flags.
1319
+ **Important:** Caching changes how rollout strategies behave. Once cached, subsequent calls return the cached value regardless of rollout strategy changes.
569
1320
 
570
- It's an important aspect of this library to be flexible with the approach you choose when determining if an experiment is enabled, and what variant should be assigned.
1321
+ ### Advanced: Custom Rollout Strategies
571
1322
 
572
- To that end, let's go ahead through an example of how to write up a custom rollout strategy that uses the [Flipper](https://github.com/jnunemaker/flipper) gem to manage rollout. For simpliticy in our example, we're going to start by inheriting from `Gitlab::Experiment::Rollout::Percent` because it's already useful.
1323
+ Build custom integrations with your existing infrastructure:
1324
+
1325
+ **Example: Flipper Integration**
573
1326
 
574
1327
  ```ruby
575
1328
  # We put it in this module namespace so we can get easy resolution when
@@ -587,7 +1340,10 @@ module Gitlab::Experiment::Rollout
587
1340
  end
588
1341
  ```
589
1342
 
590
- So, Flipper needs something that responds to `flipper_id`, and since our experiment "id" (which is also our context key) is unique and consistent, we're going to give that to Flipper to manage things like percentage of actors etc. You might want to consider something more complex here if you're using things that can be flipper actors in your experiment context.
1343
+ So, Flipper needs something that responds to `flipper_id`, and since our experiment "id" (which is also our context key)
1344
+ is unique and consistent, we're going to give that to Flipper to manage things like percentage of actors etc.
1345
+ You might want to consider something more complex here if you're using things that can be flipper actors in your
1346
+ experiment context.
591
1347
 
592
1348
  Anyway, now you can use your custom `Flipper` rollout strategy by instantiating it in configuration:
593
1349
 
@@ -608,7 +1364,9 @@ class PillColorExperiment < Gitlab::Experiment # OR ApplicationExperiment
608
1364
  end
609
1365
  ```
610
1366
 
611
- Now, enabling or disabling the Flipper feature flag will control if the experiment is enabled or not. If the experiment is enabled, as determined by our custom rollout strategy, the standard resolution logic will be executed, and a variant (or control) will be assigned.
1367
+ Now, enabling or disabling the Flipper feature flag will control if the experiment is enabled or not.
1368
+ If the experiment is enabled, as determined by our custom rollout strategy, the standard resolution logic will be
1369
+ executed, and a variant (or control) will be assigned.
612
1370
 
613
1371
  ```ruby
614
1372
  experiment(:pill_color).enabled? # => false
@@ -623,9 +1381,13 @@ experiment(:pill_color).assigned.name # => "red"
623
1381
 
624
1382
  ### Middleware
625
1383
 
626
- There are times when you'll need to do link tracking in email templates, or markdown content -- or other places you won't be able to implement tracking. For these cases a middleware layer that can redirect to a given URL while also tracking that the URL was visited has been provided.
1384
+ There are times when you'll need to do link tracking in email templates, or markdown content -- or other places you
1385
+ won't be able to implement tracking.
1386
+ For these cases a middleware layer that can redirect to a given URL while also tracking that the URL was visited has
1387
+ been provided.
627
1388
 
628
- In Rails this middleware is mounted automatically, with a base path of what's been configured for `mount_at`. If this path is nil, the middleware won't be mounted at all.
1389
+ In Rails this middleware is mounted automatically, with a base path of what's been configured for `mount_at`.
1390
+ If this path is nil, the middleware won't be mounted at all.
629
1391
 
630
1392
  ```ruby
631
1393
  Gitlab::Experiment.configure do |config|
@@ -651,13 +1413,15 @@ experiment_redirect_url(ex, url: 'https//gitlab.com/docs') # => "https://gitlab.
651
1413
 
652
1414
  ## Testing (rspec support)
653
1415
 
654
- This gem comes with some rspec helpers and custom matchers. To get the experiment specific rspec support, require the rspec support file:
1416
+ This gem comes with some rspec helpers and custom matchers.
1417
+ To get the experiment specific rspec support, require the rspec support file:
655
1418
 
656
1419
  ```ruby
657
1420
  require 'gitlab/experiment/rspec'
658
1421
  ```
659
1422
 
660
- Any file in `spec/experiments` path will automatically get the experiment specific support, but it can also be included in other specs by adding the `:experiment` label:
1423
+ Any file in `spec/experiments` path will automatically get the experiment specific support, but it can also be included
1424
+ in other specs by adding the `:experiment` label:
661
1425
 
662
1426
  ```ruby
663
1427
  describe MyExampleController do
@@ -759,7 +1523,8 @@ populate the cache naturally by running the experiment first to assign and cache
759
1523
 
760
1524
  ### Registered behaviors matcher
761
1525
 
762
- It's useful to test our registered behaviors, as well as their return values when we implement anything complex in them. The `register_behavior` matcher is useful for this.
1526
+ It's useful to test our registered behaviors, as well as their return values when we implement anything complex in them.
1527
+ The `register_behavior` matcher is useful for this.
763
1528
 
764
1529
  ```ruby
765
1530
  it "tests our registered behaviors" do
@@ -792,7 +1557,8 @@ end
792
1557
 
793
1558
  ### Tracking matcher
794
1559
 
795
- Tracking events is a major aspect of experimentation, and because of this we try to provide a flexible way to ensure your tracking calls are covered.
1560
+ Tracking events is a major aspect of experimentation, and because of this we try to provide a flexible way to ensure
1561
+ your tracking calls are covered.
796
1562
 
797
1563
  ```ruby
798
1564
  before do
@@ -806,7 +1572,10 @@ it "tests that we track an event on a specific instance" do
806
1572
  end
807
1573
  ```
808
1574
 
809
- You can use the `on_next_instance` chain method to specify that the tracking call could happen on the next instance of the experiment. This can be useful if you're calling `experiment(:example).track` downstream and don't have access to that instance. Here's a full example of the methods that can be chained onto the `track` matcher:
1575
+ You can use the `on_next_instance` chain method to specify that the tracking call could happen on the next instance of
1576
+ the experiment.
1577
+ This can be useful if you're calling `experiment(:example).track` downstream and don't have access to that instance.
1578
+ Here's a full example of the methods that can be chained onto the `track` matcher:
810
1579
 
811
1580
  ```ruby
812
1581
  it "tests that we track an event with specific details" do
@@ -821,13 +1590,16 @@ end
821
1590
 
822
1591
  ## Tracking, anonymity and GDPR
823
1592
 
824
- We generally try not to track things like user identifying values in our experimentation. What we can and do track is the "experiment experience" (a.k.a. the context key).
1593
+ We generally try not to track things like user identifying values in our experimentation.
1594
+ What we can and do track is the "experiment experience" (a.k.a. the context key).
825
1595
 
826
- We generate this key from the context passed to the experiment. This allows creating funnels without exposing any user information.
1596
+ We generate this key from the context passed to the experiment.
1597
+ This allows creating funnels without exposing any user information.
827
1598
 
828
1599
  This library attempts to be non-user-centric, in that a context can contain things like a user or a project.
829
1600
 
830
- If you only include a user, that user would get the same experience across every project they view. If you only include the project, every user who views that project would get the same experience.
1601
+ If you only include a user, that user would get the same experience across every project they view.
1602
+ If you only include the project, every user who views that project would get the same experience.
831
1603
 
832
1604
  Each of these approaches could be desirable given the objectives of your experiment.
833
1605
 
@@ -837,7 +1609,9 @@ After cloning the repo, run `bundle install` to install dependencies.
837
1609
 
838
1610
  ## Running tests
839
1611
 
840
- The test suite requires Redis to be running. [Install](https://redis.io/docs/latest/operate/oss_and_stack/install/archive/install-redis/) and start Redis (`redis-server`) before running tests.
1612
+ The test suite requires Redis to be running.
1613
+ [Install](https://redis.io/docs/latest/operate/oss_and_stack/install/archive/install-redis/) and start Redis
1614
+ (`redis-server`) before running tests.
841
1615
 
842
1616
  Once Redis is running, execute the tests:
843
1617
  `bundle exec rake`
@@ -846,9 +1620,12 @@ You can also run `bundle exec pry` for an interactive prompt that will allow you
846
1620
 
847
1621
  ## Contributing
848
1622
 
849
- Bug reports and merge requests are welcome on GitLab at https://gitlab.com/gitlab-org/ruby/gems/gitlab-experiment. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [Contributor Covenant](http://contributor-covenant.org) code of conduct.
1623
+ Bug reports and merge requests are welcome on GitLab at https://gitlab.com/gitlab-org/ruby/gems/gitlab-experiment.
1624
+ This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the
1625
+ [Contributor Covenant](http://contributor-covenant.org) code of conduct.
850
1626
 
851
- Make sure to include a changelog entry in your commit message and read the [changelog entries section](https://docs.gitlab.com/ee/development/changelog.html).
1627
+ Make sure to include a changelog entry in your commit message and read the [changelog entries
1628
+ section](https://docs.gitlab.com/ee/development/changelog.html).
852
1629
 
853
1630
  ## Release process
854
1631
 
@@ -860,6 +1637,5 @@ The gem is available as open source under the terms of the [MIT License](http://
860
1637
 
861
1638
  ## Code of conduct
862
1639
 
863
- Everyone interacting in the `Gitlab::Experiment` project’s codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct](CODE_OF_CONDUCT.md).
864
-
865
- ***Make code not war***
1640
+ Everyone interacting in the `Gitlab::Experiment` project’s codebases, issue trackers, chat rooms and mailing lists is
1641
+ expected to follow the [code of conduct](CODE_OF_CONDUCT.md).