gitlab-experiment 0.6.5 → 0.7.0

data/README.md

@@ -3,9 +3,11 @@ GitLab Experiment
 
 <img alt="experiment" src="/uploads/60990b2dbf4c0406bbf8b7f998de2dea/experiment.png" align="right" width="40%">
 
-Here at GitLab, we run experiments as A/B/n tests and review the data the experiment generates. From that data, we determine the best performing variant and promote it as the new default code path. Or revert back to the control if no variant outperformed it. You can read our [Experiment Guide](https://docs.gitlab.com/ee/development/experiment_guide/) docs if you're curious about how we use this gem internally.
+This README represents the current main branch and may not be applicable to the release you're using in your project. Please refer to the correct release branch if you'd like to review documentation relevant for that release.
 
-This library provides a clean and elegant DSL (domain specific language) to define, run, and track your GitLab experiment.
+Here at GitLab we run experiments as A/B/n tests and review the data the experiment generates. From that data, we determine the best performing code path and promote it as the new default code path, or revert back to the original code path. You can read our [Experiment Guide](https://docs.gitlab.com/ee/development/experiment_guide/) documentation if you're curious about how we use this gem internally at GitLab.
+
+This library provides a clean and elegant DSL (domain specific language) to define, run, and track your experiments.
 
 When we discuss the behavior of this gem, we'll use terms like experiment, context, control, candidate, and variant. It's worth defining these terms so they're more understood.
 
@@ -14,8 +16,12 @@ When we discuss the behavior of this gem, we'll use terms like experiment, conte
 - `control` is the default, or "original" code path.
 - `candidate` defines that there's one experimental code path.
 - `variant(s)` is used when more than one experimental code path exists.
+- `behaviors` is used to reference all possible code paths of an experiment.
+
+Candidate and variant are the same concept, but simplify how we speak about experimental paths -- both are widely referred to as the "experiment group". If you use "control and candidate," the assumption is an A/B test, and if you say "variants" the assumption is a multiple variant experiment.
 
-Candidate and variant are the same concept, but simplify how we speak about experimental paths -- this concept is also widely referred to as the "experiment group".<br clear="all">
+Behaviors is a general term for all code paths -- if that's control and candidate, or control and variants. It's all of them.
+<br clear="all">
 
 [[_TOC_]]
 
@@ -35,253 +41,320 @@ $ rails generate gitlab:experiment:install
 
 ## Implementing an experiment
 
-For the sake of an example let's make one up. Let's run an experiment on what we render for disabling desktop notifications.
-
-In our control (current world) we show a simple toggle interface labeled, "Notifications." In our experiment we want a "Turn on/off desktop notifications" button with a confirmation.
-
-The behavior will be the same, but the interface will be different and may involve more or fewer steps.
+For the sake of having a simple example let's define an experiment around a button color, even if it's not how we'd likely implement a real experiment it makes for a clean example.
 
-Our hypothesis is that this will make the action more clear and will help in making a choice about if that's what the user really wants to do.
+Currently our button is blue, but we want to test out a red variant. Our hypothesis is that a red button will make it more visible but also might appear like a warning button -- this is why we're testing our hypothesis.
 
-We'll name our experiment `notification_toggle`. This name is prefixed based on configuration. If you've set `config.name_prefix = 'gitlab'`, the experiment name would be `gitlab_notification_toggle` elsewhere.
+So let's name our experiment `pill_color`, and use the generator to get some files:
 
-When you implement an experiment you'll need to provide a name, and a context. The name can show up in tracking calls, and potentially other aspects. The context determines the variant assigned, and should be consistent between calls. We'll discuss migrating context in later examples.
+```shell
+$ rails generate gitlab:experiment pill_color
+``` 
 
-A context "key" represents the unique id of a context. It allows us to give the same experience between different calls to the experiment and can be used in caching. This is how an experiment remains "sticky" to a given context.
+This generator will give us an `app/experiments/pill_color_experiment.rb` file, which we can use to define our experiment class and register our default variant behaviors. The generator also provides a bunch of useful comments in this file unless you `--skip-comments`, so feel free to play around with that.
 
-Now in our experiment we're going to render one of two views: the control will be our current view, and the candidate will be the new toggle button with a confirmation flow.
+Let's fill out our control and candidate behaviors with some default values to start. For our example it'll just be some made up class names where we've defined our colors.
 
 ```ruby
-class SubscriptionsController < ApplicationController
-  def show
-    experiment(:notification_toggle, actor: user) do |e|
-      e.use { render_toggle } # control
-      e.try { render_button } # candidate
-    end
-  end
+class PillColorExperiment < ApplicationExperiment
+  control { 'blue' } # register and define our default control value
+  candidate { 'red' } # register and define our experimental candidate value
 end
 ```
 
-You can define the experiment using simple control/candidate paths, or provide named variants.
-
-Handling [multivariate](https://en.wikipedia.org/wiki/Multivariate_statistics) experiments is up to the configuration you provide around resolving variants. But in our example we may want to try with and without the confirmation. We can run any number of variations in our experiments this way.
-
-```ruby
-experiment(:notification_toggle, actor: user) do |e|
-  e.use { render_toggle } # control
-  e.try(:variant_one) { render_button(confirmation: true) }
-  e.try(:variant_two) { render_button(confirmation: false) }
-end
-```
+Now that we've defined our default behaviors, we can utilize our experiment elsewhere by calling the `experiment` method. When you run (or publish) an experiment you'll need to provide a name and a context. The name is how our `PillColorExperiment` class is resolved and will also show up in event data downstream. The context is a hash that's used to determine the variant assigned and should be consistent between calls.
 
-You can specify what the experiment should be "sticky" to by providing a `:sticky_to` option. By default this will be the entire context provided, but this can be overridden manually if needed.
+For our experiment we're going to make it "sticky" to the current user -- and if there isn't a current user, we want to assign and use a cookie value instead. This happens automatically if you use the [`actor` keyword](#cookies-and-the-actor-keyword) in your context. We use that in a lot of our examples, but it's by no means how everything should be done.
 
-```ruby
-experiment(:notification_toggle, actor: user, project: project, sticky_to: project) #...
+```haml
+%button{ class: experiment(:pill_color, actor: current_user).run } Click Me!
 ```
 
-Understanding how an experiment can change behavior is important in evaluating its performance.
+Now when our view is rendered, the class attribute of the button will be pulled from the experiment and will be sticky to the user that's viewing it.
 
-To this end, we track events that are important by calling the same experiment elsewhere in code. By using the same context, you'll have consistent behavior and the ability to track events to it.
+You can also provide behavior overrides when you run the experiment. Let's use a view helper for this example because it'll be cleaner than putting our override blocks into the view.
 
 ```ruby
-experiment(:notification_toggle, actor: user).track(:clicked_button)
+def pill_color_experiment
+  experiment(:pill_color, actor: current_user) do |e|
+    e.candidate { 'purple' } # override the candidate default of 'red'
+  end
+end
 ```
 
-### Custom experiments
+Now we can run the experiment using that helper method. Instead of a red button, users who are assigned the candidate will be given a purple button. Users in the control group will still see the blue button. Using experiments in this way permits a default experiment to be defined while also allowing the experiment to be customized in the places where its run, using the scope, variables and helper methods that are available to us where we want to run the experiment.
 
-You can craft more advanced behaviors by defining custom experiments at a higher level. To do this you can define a class that inherits from `ApplicationExperiment` (or `Gitlab::Experiment`).
+```haml
+%button{ class: pill_color_experiment.run } Click Me!
+```
 
-Let's say you want to do more advanced segmentation, or provide default behavior for the variants on the experiment we've already outlined above -- that way if the variants aren't defined in the block at the time the experiment is run, these methods will be used. 
+Understanding how an experiment can change user behavior or engagement is important in evaluating its performance. To this end, you'll probably want to track events that are important elsewhere in code. By using the same context you can provide a consistent experience and have the ability to anonymously track events to that experience.
 
-You can generate a custom experiment by running:
+```ruby
+experiment(:pill_color, actor: current_user).track(:clicked)
+```
 
-```shell
-$ rails generate gitlab:experiment NotificationToggle control candidate
-``` 
+## Advanced experimentation
 
-This will generate a file in `app/experiments/notification_toggle_experiment.rb`, as well as a test file for you to further expand on.
+Now let's create a more complex experiment with multiple variants. This time we'll start with a grey button, and have red and blue variants.
 
-Here are some examples of what you can introduce once you have a custom experiment defined.
+We'll also do more advanced segmentation when the experiment is run, exclude certain cases, and register callbacks that will be executed after our experiment is run. Here are some examples of what we can start to do in our experiment classes:
 
 ```ruby
-class NotificationToggleExperiment < ApplicationExperiment
-  # Exclude any users that aren't me.
+class PillColorExperiment < Gitlab::Experiment # OR ApplicationExperiment
+  # Register our behaviors.  
+  control # register our control, which will by default call #control_behavior
+  variant(:red) # register the red variant that will call #red_behavior
+  variant(:blue) # register the blue variant that will call #blue_behavior
+
+  # Exclude any users that are named "Richard".
   exclude :users_named_richard
 
-  # Segment any account older than 2 weeks into the candidate, without
-  # asking the variant resolver to decide which variant to provide.
-  segment :old_account?, variant: :candidate
+  # Segment any account older than 2 weeks into the red variant without asking
+  # the rollout strategy to assign a variant.
+  segment :old_account?, variant: :red
+  
+  # After the experiment has been run, we want to log some performance metrics.
+  after_run { log_performance_metrics }
 
-  # Define the default control behavior, which can be overridden at
-  # experiment time.
+  private
+  
+  # Define the default control behavior, which can be overridden on runs.
   def control_behavior
-    # render_toggle
+    'grey'
   end
 
-  # Define the default candidate behavior, which can be overridden
-  # at experiment time.
-  def candidate_behavior
-    # render_button
+  # Define the default red behavior, which can be overridden on runs.
+  def red_behavior
+    'red'
   end
 
-  private
+  # Define the default blue behavior, which can be overridden on runs.
+  def blue_behavior
+    'blue'
+  end
 
+  # Define our special exclusion logic.
   def users_named_richard
-    context.actor.first_name == 'Richard'
+    context.try(:actor)&.first_name == 'Richard' # use try for nil actors
   end
 
+  # Define our segmentation logic.
   def old_account?
-    context.actor.created_at < 2.weeks.ago
+    context.try(:actor) && context.actor.created_at < 2.weeks.ago
+  end
+    
+  # Let's assume that we've tracked this and want to push it into some system.
+  def log_performance_metrics
+    # ...hypothetical implementation 
   end
 end
+```
+
+You can play around with our new `PillColorExperiment` using a console or irb session. In our example we've used an actual `User` model with a first name and timestamps. Feel free to use something more appropriate for your project in your exploration, or just not pass in `actor` at all.
+
+```ruby
+include Gitlab::Experiment::Dsl
 
 # The class will be looked up based on the experiment name provided. 
-exp = experiment(:notification_toggle, actor: user)
-exp # => instance of NotificationToggleExperiment
+ex = experiment(:pill_color, actor: User.first) # => #<PillColorExperiment:0x...>
 
 # Run the experiment -- returning the result.
-exp.run
+ex.run # => "grey" (the value defined in our control)
 
-# Track an event on the experiment we've defined.
-exp.track(:clicked_button)
-```
+# Track an event on the experiment we've defined, using the logic we've defined
+# in configuration.
+ex.track(:clicked) # => true
 
-You can now also do things very similar to the simple examples and override the default variant behaviors defined in the custom experiment -- keeping in mind that this should be carefully considered within the scope of your experiment.
-
-```ruby
-experiment(:notification_toggle, actor: user) do |e|
-  e.use { render_special_toggle } # override default control behavior
-end
+# Publish the experiment without running it, using the logic we've defined in
+# configuration.
+ex.publish # => {:variant=>"control", :experiment=>"pill_color", :key=>"45f595...", :excluded=>false}
 ```
 
 <details>
-  <summary>You can also specify the variant to use for segmentation...</summary>
+  <summary>You can also specify the variant manually...</summary>
 
-Generally, defining segmentation rules is a better way to approach routing into specific variants, but it's possible to explicitly specify the variant when running an experiment. It's important to know what this might do to your data during rollout, so use this with careful consideration.
+Generally, defining segmentation rules is a better way to approach routing into specific variants, but it's possible to explicitly specify the variant when running an experiment.
 
-Any time a specific variant is provided (including `:control`) it will be cached for that context, if caching is enabled. 
+Caching: It's important to understand what this might do to your data during rollout, so use this with careful consideration. Any time a specific variant is assigned manually, or through segmentation (including `:control`) it will be cached for that context. That means that if you manually assign `:control`, that context will never be moved out of the control unless you do it programmatically elsewhere.
 
 ```ruby
-experiment(:notification_toggle, :no_interface, actor: user) do |e|
-  e.use { render_toggle } # control
-  e.try { render_button } # candidate
-  e.try(:no_interface) { no_interface! } # no_interface variant
-end
+include Gitlab::Experiment::Dsl
+
+# Assign the candidate manually.
+ex = experiment(:pill_color, :red, actor: User.first) # => #<PillColorExperiment:0x..>
+
+# Run the experiment -- returning the result.
+ex.run # => "red"
+
+# If caching is enabled this will remain sticky between calls.
+experiment(:pill_color, actor: User.first).run # => "red"
 ```
 
-Or you can set the variant within the block. This allows using unique segmentation logic or variant resolution if you need it.
+</details>
+
+### Exclusion rules
+
+Exclusion rules let us determine if a context should even be considered as something to include in an experiment. If we're excluding something, it means that we don't want to run the experiment in that case. This can be useful if you only want to run experiments on new users for instance.
 
 ```ruby
-experiment(:notification_toggle, actor: user) do |e|
-  # Variant selection must be done before calling run or track.
-  e.variant(:no_interface) # set the variant
-  # ...
+class PillColorExperiment < Gitlab::Experiment # OR ApplicationExperiment
+  # ...registered behaviors
+
+  exclude :old_account?, ->{ context.actor.first_name == 'Richard' }
 end
 ```
 
-Or it can be specified in the call to run if you call it from within the block.
+In the previous example, we'll exclude all users named `'Richard'` as well as any account older than 2 weeks old. Not only will they be immediately given the control behavior, but no events will be tracked in these cases either.
+
+Exclusion rules are executed in the order they're defined. The first exclusion rule to produce a truthy result will halt execution of further exclusion checks.
+
+Note: Although tracking calls will be ignored on all exclusions, you may want to check exclusion yourself in expensive custom logic by calling the `should_track?` or `excluded?` methods.
+
+Note: When using exclusion rules it's important to understand that the control assignment is cached, which improves future experiment run performance but can be a gotcha around caching.
+
+Note: Exclusion rules aren't the best way to determine if an experiment is enabled. There's an `enabled?` method that can be overridden to have a high-level way of determining if an experiment should be running and tracking at all. This `enabled?` check should be as efficient as possible because it's the first early opt out path an experiment can implement. This can be seen in [How it works](#how-it-works).
+
+### Segmentation rules
+
+Segmentation, or assigning certain variants in certain cases, is important to running experiments. This can be useful if you want to push a given population into a specific variant because you've already determined that variant is successful for that population.
 
 ```ruby
-experiment(:notification_toggle, actor: user) do |e|
-  # ...
-  # Variant selection can be specified when calling run.
-  e.run(:no_interface)
+class PillColorExperiment < Gitlab::Experiment # OR ApplicationExperiment
+  # ...registered behaviors
+    
+  segment(variant: :red) { context.actor.first_name == 'Richard' }
+  segment :old_account?, variant: :blue
 end
 ```
 
-</details>
+In the previous example, any user named `'Richard'` would always receive the experience defined in the red variant. As well, any account older than 2 weeks would get the alternate experience defined in the blue variant.
 
-### Segmentation rules
+Segmentation rules are executed in the order they're defined. The first segmentation rule to produce a truthy result is the one which gets used to assign the variant. The remaining segmentation rules are skipped. This means that for our example, any user named `'Richard'` regardless of account age, will always be provided the experience as defined in the red variant.
 
-This library comes with the capability to segment contexts into a specific variant, before asking the variant resolver which variant to provide.
+### Run callbacks
 
-Segmentation can be achieved by using a custom experiment class and specifying the segmentation rules at a class level.
+Callbacks can be registered for when you want to execute logic before, after, or around when an experiment is run. These callbacks won't be called unless the experiment is actually run, meaning that exclusion rules take precedence.
 
 ```ruby
-class ExampleExperiment < ApplicationExperiment
-  segment(variant: :variant_one) { context.actor.first_name == 'Richard' }
-  segment :old_account?, variant: :variant_two
+class PillColorExperiment < Gitlab::Experiment # OR ApplicationExperiment
+  # ...registered behaviors
   
-  private
-
-  def old_account?
-    context.actor.created_at < 2.weeks.ago
-  end
+  after_run :log_performance_metrics, -> { publish_to_database }
 end
 ```
 
-In the previous examples, any user named `'Richard'` would always receive the experience defined in "variant_one". As well, any account older than 2 weeks old would get the alternate experience defined in "variant_two".
+In the previous example, we're going to call the `log_performance_method`, and do a hypothetical publish to the database. If you want to do an `around_run`, you just need to call the block:
 
-When an experiment is run, the segmentation rules are executed in the order they're defined. The first segmentation rule to produce a truthy result is the one which gets used to assign the variant. The remaining segmentation rules are skipped.
+```ruby
+class PillColorExperiment < Gitlab::Experiment # OR ApplicationExperiment
+  # ...registered behaviors
 
-This means that any user named `'Richard'`, regardless of account age, will always be provided the experience as defined in "variant_one". If you wanted the opposite logic, you can flip the order.
+  around_run do |experiment, block|
+    puts "- before #{experiment.name} run"
+    block.call
+    puts "- after #{experiment.name} run"
+  end
+end
+```
 
-### Exclusion rules
+### Rollout strategies
 
-Exclusion rules are similar to segmentation rules, but are intended to determine if a context should even be considered as something we should track events towards. Exclusion means we don't care about the events in relation to the given context.
+While a default rollout strategy can be defined in configuration, it's useful to be able to override this per experiment if needed. You can do this by specifying a specific `default_rollout` override in your experiment class.
 
 ```ruby
-class ExampleExperiment < ApplicationExperiment
-  exclude :old_account?, ->{ context.actor.first_name == 'Richard' }
-  
-  private
+class PillColorExperiment < Gitlab::Experiment # OR ApplicationExperiment
+  # ...registered behaviors
 
-  def old_account? 
-    context.actor.created_at < 2.weeks.ago
-  end
+  default_rollout :random # randomly assign one of the registered behaviors
 end
 ```
 
-The previous examples will exclude all users named `'Richard'` as well as any account older than 2 weeks old. Not only will they be given the control behavior, but no events will be tracked in these cases as well. 
+Obviously random assignment might not be the best rollout strategy for you, but you can define your own rollout strategies, or use one of the ones provided in the gem. You can read more about configuring the default rollout and how to write your own rollout strategies in the [configuration documentation](#rollout-strategies-1) for it.
 
-You may need to check exclusion in custom tracking logic by calling `should_track?`:
+## How it works
+
+The way experiments work is best described using the following decision tree diagram. When an experiment is run, the following logic is executed to resolve what experience should be provided, given how the experiment is defined, and using the context passed to the experiment call.
+
+```mermaid
+graph TD
+    GP[General Pool/Population] --> Running?[Rollout Enabled?]
+    Running? -->|Yes| Cached?[Cached? / Pre-segmented?]
+    Running? -->|No| Excluded[Control / No Tracking]
+    Cached? -->|No| Excluded?
+    Cached? -->|Yes| Cached[Cached Value]
+    Excluded? -->|Yes / Cached| Excluded
+    Excluded? -->|No| Segmented?
+    Segmented? -->|Yes / Cached| VariantA
+    Segmented? -->|No| Rollout[Rollout Resolve]
+    Rollout --> Control
+    Rollout -->|Cached| VariantA
+    Rollout -->|Cached| VariantB
+    Rollout -->|Cached| VariantN
+
+classDef included fill:#380d75,color:#ffffff,stroke:none
+classDef excluded fill:#fca121,stroke:none
+classDef cached fill:#2e2e2e,color:#ffffff,stroke:none
+classDef default fill:#fff,stroke:#6e49cb
+
+class VariantA,VariantB,VariantN included
+class Control,Excluded excluded
+class Cached cached
+```
+
+## Technical details
+
+This library is intended to be powerful and easy to use, which can lead to some complex underpinnings in the implementation. Some of those implementation details are important to understand at a technical level when considering how you want to design your experiments.
+
+### Including the DSL
+
+By default, `Gitlab::Experiment` injects itself into the controller, view, and mailer layers. This exposes the `experiment` method application wide in those layers. Some experiments may extend outside of those layers however, so you may want to include it elsewhere. For instance in an irb session or the rails console, or in all your service objects, background jobs, or similar.
 
 ```ruby
-def expensive_tracking_logic
-  return unless should_track?
-  
-  track(:my_event, value: expensive_method_call)
+class ApplicationJob < ActiveJob::Base
+  include Gitlab::Experiment::Dsl # include the `experiment` method for all jobs
 end
 ```
 
-Note: Exclusion rules aren't the best way to determine if an experiment is enabled. There's an `enabled?` method that can be overridden to have a high-level way of determining if an experiment should be running and tracking at all. This `enabled?` check should be as efficient as possible because it's the first early opt out path an experiment can implement.
+### Experiment stickiness
 
-### Return value
+Internally, experiments have what's referred to as the context "key" that represents the unique and anonymous id of a given context. This allows us to assign the same variant between different calls to the experiment, is used in caching and can be used in event data downstream. This context "key" is how an experiment remains "sticky" to a given context, and is an important aspect to understand.
 
-By default the return value is a `Gitlab::Experiment` instance. In simple cases you may want only the results of the experiment though. You can call `run` within the block to get the return value of the assigned variant.
+You can specify what the experiment should be sticky to by providing the `:sticky_to` option. By default this will be the entire context, but this can be overridden manually if needed.
+
+In a fabricated example, we might want to provide the user and a project to an experiment, but we want it to remain sticky to the project that the user is viewing and not the user viewing that project. This is a very powerful concept that you're encouraged to play around with to understand what it means, and how you can run complex experiments that aren't user centric.
 
 ```ruby
-experiment(:notification_toggle) do |e|
-  e.use { 'A' }
-  e.try { 'B' }
-  e.run
-end # => 'A'
+experiment(:example, actor: current_user, project: project, sticky_to: project)
 ```
 
-### Including the DSL
+### Experiment signature
 
-By default, `Gitlab::Experiment` injects itself into the controller and view layers. This exposes the `experiment` method application wide in those layers.
+The best way to understand the details of an experiment is through its signature. An example signature can be retrieved by calling the `signature` method, and looks like the following:
 
-Some experiments may extend outside of those layers, so you may want to include it elsewhere. For instance in a mailer, service object, background job, or similar.
+```ruby
+experiment(:example).signature # => {:variant=>"control", :experiment=>"example", :key=>"4d7aee..."}
+```
 
-Note: In a lot of these contexts you may not have a reference to the request (unless you pass it in, or provide access to it) which may be needed if you want to enable cookie behaviors and track that through to user conversion.
+An experiment signature is useful when tracking events and when using experiments on the client layer. The signature can also contain the optional `migration_keys`, and `excluded` properties.
 
-```ruby
-class WelcomeMailer < ApplicationMailer
-  include Gitlab::Experiment::Dsl # include the `experiment` method
+### Return value
 
-  def welcome
-    @user = params[:user]
+By default the return value of calling `experiment` is a `Gitlab::Experiment` instance, or whatever class the experiment is resolved to, which likely inherits from `Gitlab::Experiment`. In simple cases you may want only the results of running the experiment though. You can call `run` within the block to get the return value of the assigned variant.
 
-    ex = experiment(:project_suggestions, actor: @user) do |e|
-      e.use { 'welcome' }
-      e.try { 'welcome_with_project_suggestions' }
-    end
+```ruby
+# Normally an experiment instance.
+experiment(:example) do |e|
+  e.control { 'A' }
+  e.candidate { 'B' }
+end # => #<Gitlab::Experiment:0x...>
 
-    mail(to: @user.email, subject: 'Welcome!', template: ex.run)
-  end
-end
+# But calling `run` causes the return value to be the result.
+experiment(:example) do |e|
+  e.control { 'A' }
+  e.candidate { 'B' }
+  e.run
+end # => 'A'
 ```
 
 ### Context migrations
@@ -293,8 +366,11 @@ Take for instance, that you might be using `version: 1` in your context currentl
 In providing the context migration data, we can resolve an experience and its events all the way back. This can also help in keeping our cache relevant.
 
 ```ruby
-# Migrate just the `:version` portion of the previous context, `{ actor: project, version: 1 }`:
-experiment(:example, actor: project, version: 2, migrated_with: { version: 1 })
+# First implementation.
+experiment(:example, actor: current_user, version: 1)
+
+# Migrate just the `:version` portion.
+experiment(:example, actor: current_user, version: 2, migrated_with: { version: 1 })
 ```
 
 You can add or remove context by providing a `migrated_from` option. This approach expects a full context replacement -- i.e. what it was before you added or removed the new context key.
@@ -302,248 +378,292 @@ You can add or remove context by providing a `migrated_from` option. This approa
 If you wanted to introduce a `version` to your context, provide the full previous context.
 
 ```ruby
-# Migrate the full context from `{ actor: project }` to `{ actor: project, version: 1 }`:
-experiment(:example, actor: project, version: 1, migrated_from: { actor: project })
-```
-
-This can impact an experience if you:
+# First implementation.
+experiment(:example, actor: current_user)
 
-1. haven't implemented the concept of migrations in your variant resolver
-1. haven't enabled a reasonable caching mechanism
+# Migrate the full context of `{ actor: current_user }` to `{ actor: current_user, version: 1 }`.
+experiment(:example, actor: current_user, version: 1, migrated_from: { actor: current_user })
+```
 
-### When there isn't an actor (cookie fallback)
+When you migrate context, this information is included in the signature of the experiment. This can be used downstream in event handling and reporting to resolve a series of events back to a single experience, while also keeping everything anonymous.
 
-When there isn't an identifying key in the context (this is `actor` by default), we fall back to cookies to provide a consistent experience for the client viewing them.
+An example of our experiment signature when we migrate would include the `migration_keys` property:
 
-Once we assign a certain variant to a context, we need to always provide the same experience. We achieve this by setting a cookie for the experiment in question, but only when needed.
+```ruby
+ex = experiment(:example, version: 1)
+ex.signature # => {:key=>"20d69a...", ...}
 
-This cookie is a temporary, randomized uuid and isn't associated with a user. When we can finally provide an actor, the context is auto migrated from the cookie to that actor.
+ex = experiment(:example, version: 2, migrated_from: { version: 1 })
+ex.signature # => {:key=>"9e9d93...", :migration_keys=>["20d69a..."], ...}
+```
 
-To read and write cookies, we provide the `request` from within the controller and views. The cookie migration will happen automatically if the experiment is within those layers.
+### Cookies and the actor keyword
 
-You'll need to provide the `request` as an option to the experiment if it's outside of the controller and views.
+We use cookies to auto migrate an unknown value into a known value, often in the case of the current user. The implementation of this uses the same concept outlined above with context migrations, but will happen automatically for you if you use the `actor` keyword.
 
-```ruby
-experiment(:example, actor: user, request: request)
-```
+When you use the `actor: current_user` pattern in your context, the nil case is handled by setting a special cookie for the experiment and then deleting the cookie, and migrating the context key to the one generated from the user when they've signed in.
 
-The cookie isn't set if the `actor` key isn't present at all in the context. Meaning that when no `actor` key is provided, the cookie will not be set.
+This cookie is a temporary, randomized uuid and isn't associated with a user. When we can finally provide an actor, the context is auto migrated from the cookie to that actor.
 
 ```ruby
-# actor is not present, so no cookie is set
+# The actor key is not present, so no cookie is set.
 experiment(:example, project: project)
 
-# actor is present and is nil, so the cookie is set and used
+# The actor key is present but nil, so the cookie is set and used.
 experiment(:example, actor: nil, project: project)
 
-# actor is present and set to a value, so no cookie is set
-experiment(:example, actor: user, project: project)
+# The actor key is present and isn't nil, so the cookie value (if found) is
+# migrated forward and the cookie is deleted.
+experiment(:example, actor: current_user, project: project)
 ```
 
-For edge cases, you can pass the cookie through by assigning it yourself -- e.g. `actor: request.cookie_jar.signed['example_actor']`. The cookie name is the full experiment name (including any configured prefix) with `_actor` appended -- e.g. `gitlab_notification_toggle_actor` for the `:notification_toggle` experiment key with a configured prefix of `gitlab`.
+Note: The cookie is deleted when resolved, but can be assigned again if the `actor` is ever nil again. A good example of this scenario would be on a sign in page. When a potential user arrives, they would never be known, so a cookie would be set for them, and then resolved/removed as soon as they signed in. This process would repeat each time they arrived while not being signed in and can complicate reporting unless it's handled well in the data layers.
 
-## How it works
+Note: To read and write cookies, we provide the `request` from within the controller and views. The cookie migration will happen automatically if the experiment is within those layers. You'll need to provide the `request` as an option to the experiment if it's outside of the controller and views.
 
-The way the gem works is best described using the following decision tree illustration. When an experiment is run, the following logic is executed to resolve what experience should be provided, given how the experiment is defined, and the context provided.
- 
-```mermaid
-graph TD
-    GP[General Pool/Population] --> Enabled?
-    Enabled? -->|Yes| Cached?[Cached? / Pre-segmented?]
-    Enabled? -->|No| Excluded[Control / No Tracking]
-    Cached? -->|No| Excluded?
-    Cached? -->|Yes| Cached[Cached Value]
-    Excluded? -->|Yes / Cached| Excluded
-    Excluded? -->|No| Segmented?
-    Segmented? -->|Yes / Cached| VariantA
-    Segmented? -->|No| Included?[Experiment Group?]
-    Included? -->|Yes| Rollout
-    Included? -->|No| Control
-    Rollout -->|Cached| VariantA
-    Rollout -->|Cached| VariantB
-    Rollout -->|Cached| VariantC
+```ruby
+experiment(:example, actor: current_user, request: request)
+```
 
-classDef included fill:#380d75,color:#ffffff,stroke:none
-classDef excluded fill:#fca121,stroke:none
-classDef cached fill:#2e2e2e,color:#ffffff,stroke:none
-classDef default fill:#fff,stroke:#6e49cb
+Note: For edge cases, you can pass the cookie through by assigning it yourself -- e.g. `actor: request.cookie_jar.signed['example_id']`. The cookie name is the full experiment name (including any configured prefix) with `_id` appended -- e.g. `pill_color_id` for the `PillColorExperiment`.
 
-class VariantA,VariantB,VariantC included
-class Control,Excluded excluded
-class Cached cached
+### Client layer
+
+Experiments that have been run (or published) during the request lifecycle can be pushed into to the client layer by injecting the published experiments into javascript in a layout or view using something like:
+
+```haml        
+= javascript_tag(nonce: content_security_policy_nonce) do
+  window.experiments = #{raw ApplicationExperiment.published_experiments.to_json};
 ```
 
+The `window.experiments` object can then be used in your client implementation to determine experimental behavior at that layer as well. For instance, we can now access the `window.experiments.pill_color` object to get the variant that was assigned, if the context was excluded, and to use the context key in our client side events.
+
 ## Configuration
 
-This gem needs to be configured before being used in a meaningful way.
+The gem is meant to be configured when installed and is ambiguous about how it should track events, and what to do when publishing experiments. Some of the more important aspects are left up to you to implement the logic that's right for your project.
 
-The default configuration will always render the control, so it's important to configure your own logic for resolving variants.
+Simple documentation can be found in the provided [initializer](lib/generators/gitlab/experiment/install/templates/initializer.rb.tt).
 
-Yes, the most important aspect of the gem -- that of determining which variant to render and when -- is up to you. Consider using [Unleash](https://github.com/Unleash/unleash-client-ruby) or [Flipper](https://github.com/jnunemaker/flipper) for this.
+Read on for comprehensive documentation on some of the more complex configuration options.
+
+### Caching
+
+Caching can be enabled in configuration and is implemented towards the `Rails.cache` / `ActiveSupport::Cache::Store` interface. When you enable caching, any variant resolution will be cached when something other than nil. Migrating the cache through context migrations is handled automatically, and this helps ensure an experiment experience remains consistent.
+
+It's important to understand that using caching can drastically change or negate your specific rollout strategy logic.
 
 ```ruby
 Gitlab::Experiment.configure do |config|
-  # The block here is evaluated within the scope of the experiment instance,
-  # which is why we are able to access things like name and context.
-  config.variant_resolver = lambda do |requested_variant|
-    # Return the requested variant if a specific one has been provided in code.
-    return requested_variant unless requested_variant.nil?
-
-    # Ask Unleash to determine the variant, given the context we've built,
-    # using the control as the fallback.
-    fallback = Unleash::Variant.new(name: 'control', enabled: true)
-    UNLEASH.get_variant(name, context.value, fallback)
-  end
+  config.cache = Rails.cache
 end
 ```
 
-More examples for configuration are available in the provided [rails initializer](lib/generators/gitlab/experiment/install/templates/initializer.rb.tt).
+The gem includes the following cache stores, which are documented in the implementation:
+
+- [`RedisHashStore`](lib/gitlab/experiment/cache/redis_hash_store.rb): Useful if using redis
+
+### Rollout strategies
 
-### Client layer / JavaScript
+There are some basic rollout strategies that come with the gem, and you can use these directly, or you can use them to help build your own custom rollout strategies. Each is documented more thoroughly in its implementation file.
 
-This library doesn't attempt to provide any logic for the client layer.
+- [`Base`](lib/gitlab/experiment/rollout.rb): Useful for building custom rollout strategies, not super useful by itself
+- [`Percent`](lib/gitlab/experiment/rollout/percent.rb): A comprehensive percent based strategy, it's configured as the default
+- [`Random`](lib/gitlab/experiment/rollout/random.rb): Random assignment can be useful on some experimentation
+- [`RoundRobin`](lib/gitlab/experiment/rollout/round_robin.rb): Cycles through assignment using the cache to keep track of what was last assigned
 
-Instead it allows you to do this yourself in configuration. Using [Gon](https://github.com/gazay/gon) to publish your experiment information to the client layer is pretty simple.
+The included rollout strategies are great, but you might want to write your own for your own project, which might already have nice tooling for toggling feature flags.
+
+It's an important aspect of this library to be flexible with the approach you choose when determining if an experiment is enabled, and what variant should be assigned.
+
+To that end, let's go ahead through an example of how to write up a custom rollout strategy that uses the [Flipper](https://github.com/jnunemaker/flipper) gem to manage rollout. For simpliticy in our example, we're going to start by inheriting from `Gitlab::Experiment::Rollout::Percent` because it's already useful.
 
 ```ruby
-Gitlab::Experiment.configure do |config|
-  config.publishing_behavior = lambda do |_result|
-    # Push the experiment knowledge into the front end. The signature contains
-    # the context key, and the variant that has been determined.
-    Gon.push({ experiment: { name => signature } }, true)
+# We put it in this module namespace so we can get easy resolution when
+# using `default_rollout :flipper` in our usage later. 
+module Gitlab::Experiment::Rollout
+  class Flipper < Percent
+    def enabled?
+      ::Flipper.enabled?(experiment.name, self)
+    end
+
+    def flipper_id
+      "Experiment;#{id}"
+    end
   end
 end
 ```
 
-In the client you can now access `window.gon.experiment.notificationToggle`.
+So, Flipper needs something that responds to `flipper_id`, and since our experiment "id" (which is also our context key) is unique and consistent, we're going to give that to Flipper to manage things like percentage of actors etc. You might want to consider something more complex here if you're using things that can be flipper actors in your experiment context.
 
-### Caching
+Anyway, now you can use your custom `Flipper` rollout strategy by instantiating it in configuration:
 
-Caching can be enabled in configuration, and is implemented towards the `Rails.cache` / `ActiveSupport::Cache::Store` interface. When you enable caching, any variant resolution will be cached. Migrating the cache through context migrations is handled automatically, and this helps ensure an experiment experience remains consistent.
+```ruby
+Gitlab::Experiment.configure do |config|
+  config.default_rollout = Gitlab::Experiment::Rollout::Flipper.new(
+    include_control: true # specify to include control, which we want to do
+  )
+end
+```
 
-It's important to understand that using caching can drastically change or override your rollout strategy logic.
+Or if you don't want to make that change globally, you can use it in specific experiment classes:
 
 ```ruby
-Gitlab::Experiment.configure do |config|
-  config.cache = Rails.cache
+class PillColorExperiment < Gitlab::Experiment # OR ApplicationExperiment
+  # ...registered behaviors
+
+  default_rollout :flipper,
+    include_control: true, # optionally specify to include control
+    distribution: { control: 26, red: 37, blue: 37 } # optionally specify distribution
 end
 ```
 
-### Middleware
+Now, enabling or disabling the Flipper feature flag will control if the experiment is enabled or not. If the experiment is enabled, as determined by our custom rollout strategy, the standard resolutuon logic will be executed, and a variant (or control) will be assigned.
+
+```ruby
+experiment(:pill_color).enabled? # => false
+experiment(:pill_color).assigned.name # => "control"
+
+# Now we can enable the feature flag to enable the experiment.
+Flipper.enable(:pill_color) # => true
 
-There are times when you'll need to do link tracking in email templates, or markdown content -- or other places you won't be able to implement tracking. For these cases, gitlab-experiment comes with middleware that will redirect to a given URL while also tracking that the URL was visited.
+experiment(:pill_color).enabled? # => true
+experiment(:pill_color).assigned.name # => "red"
+```
 
-In Rails this middleware is mounted automatically, with a base path of what's been configured for `mount_at`. If this path is empty the middleware won't be mounted at all.
+### Middleware
 
-Once mounted, the redirect URLs can be generated using the Rails route helpers. If not using Rails, mount the middleware and generate these URLs yourself.
+There are times when you'll need to do link tracking in email templates, or markdown content -- or other places you won't be able to implement tracking. For these cases a middleware layer that can redirect to a given URL while also tracking that the URL was visited has been provided.
+
+In Rails this middleware is mounted automatically, with a base path of what's been configured for `mount_at`. If this path is nil, the middleware won't be mounted at all.
 
 ```ruby
 Gitlab::Experiment.configure do |config|
   config.mount_at = '/experiment'
+
+  # Only redirect on permitted domains.
+  config.redirect_url_validator = ->(url) { (url = URI.parse(url)) && url.host == 'gitlab.com' }    
 end
+```
 
-ex = experiment(:example, foo: :bar)
+Once configured to be mounted, the experiment tracking redirect URLs can be generated using the Rails route helpers.
 
-# using rails path/url helpers 
-experiment_redirect_path(ex, 'https//docs.gitlab.com/') # => /experiment/example:[context_key]?https//docs.gitlab.com/
+```ruby
+ex = experiment(:example)
 
-# manually
-"#{Gitlab::Experiment.configure.mount_at}/#{ex.to_param}?https//docs.gitlab.com/"
-```
+# Generating the path/url using the path and url helper.
+experiment_redirect_path(ex, url: 'https//gitlab.com/docs') # => "/experiment/example:20d69a...?https//gitlab.com/docs"
+experiment_redirect_url(ex, url: 'https//gitlab.com/docs') # => "https://gitlab.com/experiment/example:20d69a...?https//gitlab.com/docs"
 
-URLS that match the base path will be handled by the middleware and will redirect to the provided redirect path.
+# Manually generating a url is a bit less clean, but is possible.
+"#{Gitlab::Experiment::Configuration.mount_at}/#{ex.to_param}?https//docs.gitlab.com/"
+```
 
 ## Testing (rspec support)
 
-This gem comes with some rspec helpers and custom matchers. These are in flux at the time of writing.
-
-First, require the rspec support file:
+This gem comes with some rspec helpers and custom matchers. To get the experiment specific rspec support, require the rspec support file:
 
 ```ruby
 require 'gitlab/experiment/rspec'
 ```
 
-This mixes in some of the basics, but the matchers and other aspects need to be included. This happens automatically for files in `spec/experiments`, but for other files and specs you want to include it in, you can specify the `:experiment` type:
+Any file in `spec/experiments` path will automatically get the experiment specific support, but it can also be included in other specs by adding the `:experiment` label:
 
 ```ruby
-it "tests", :experiment do
+describe MyExampleController do
+  context "with my experiment", :experiment do
+    # experiment helpers and matchers will be available here.
+  end
 end
 ```
 
 ### Stub helpers
 
-You can stub experiments using `stub_experiments`. Pass it a hash using experiment names as the keys and the variants you want each to resolve to as the values:
+You can stub experiment variant resolution using the `stub_experiments` helper. Pass a hash of experiment names and the variant each should resolve to.
 
 ```ruby
-# Ensures the experiments named `:example` & `:example2` are both
-# "enabled" and that each will resolve to the given variant
-# (`:my_variant` & `:control` respectively).
-stub_experiments(example: :my_variant, example2: :control)
+it "stubs experiments to resolve to a specific variant" do
+  stub_experiments(pill_color: :red)
 
-experiment(:example) do |e|
-  e.enabled? # => true
-  e.variant.name # => 'my_variant'
-end
-
-experiment(:example2) do |e|
-  e.enabled? # => true
-  e.variant.name # => 'control'
-end
+  experiment(:pill_color) do |e|
+    expect(e).to be_enabled
+    expect(e.assigned.name).to eq('red')
+  end
+end 
 ```
 
-### Exclusion and segmentation matchers
-
-You can also easily test the exclusion and segmentation matchers.
+In special cases you can use a boolean `true` instead of a variant name. This allows the rollout strategy to resolve the variant however it wants to, but is otherwise just making sure the experiment is considered enabled.
 
 ```ruby
-class ExampleExperiment < ApplicationExperiment
-  exclude { context.actor.first_name == 'Richard' }
-  segment(variant: :candidate) { context.actor.username == 'jejacks0n' }
+it "stubs experiments while allowing the rollout strategy to assign the variant" do
+  stub_experiments(pill_color: true) # only stubs enabled?
+
+  experiment(:pill_color) do |e|
+    expect(e).to be_enabled
+    # expect(e.assigned.name).to eq([whatever the rollout strategy assigns])
+  end
 end
+```
 
-excluded = double(username: 'rdiggitty', first_name: 'Richard')
-segmented = double(username: 'jejacks0n', first_name: 'Jeremy')
+### Registered behaviors matcher
 
-# exclude matcher
-expect(experiment(:example)).to exclude(actor: excluded)
-expect(experiment(:example)).not_to exclude(actor: segmented)
+It's useful to test our registered behaviors, as well as their return values when we implement anything complex in them. The `register_behavior` matcher is useful for this. 
 
-# segment matcher
-expect(experiment(:example)).to segment(actor: segmented).into(:candidate)
-expect(experiment(:example)).not_to segment(actor: excluded)
+```ruby
+it "tests our registered behaviors" do
+  expect(experiment(:pill_color)).to register_behavior(:control)
+    .with('grey') # with a default return value of "grey"
+  expect(experiment(:pill_color)).to register_behavior(:red)
+  expect(experiment(:pill_color)).to register_behavior(:blue)
+end
 ```
 
-### Tracking matcher
-
-Tracking events is a major aspect of experimentation, and because of this we try to provide a flexible way to ensure your tracking calls are covered.
+### Exclusion and segmentation matchers
 
-You can do this on the instance level or at an "any instance" level. At an instance level this is pretty straight forward:
+You can also easily test your experiment classes using the `exclude`, `segment` metchers.
 
 ```ruby
-subject = experiment(:example)
+let(:excluded) { double(first_name: 'Richard', created_at: Time.current) }
+let(:segmented) { double(first_name: 'Jeremy', created_at: 3.weeks.ago) }
 
-expect(subject).to track(:my_event)
+it "tests the exclusion rules" do
+  expect(experiment(:pill_color)).to exclude(actor: excluded)
+  expect(experiment(:pill_color)).not_to exclude(actor: segmented)
+end
 
-subject.track(:my_event)
+it "tests the segmentation rules" do
+  expect(experiment(:pill_color)).to segment(actor: segmented)
+    .into(:red) # into a specific variant
+  expect(experiment(:pill_color)).not_to segment(actor: excluded)
+end
 ```
 
-You can use the `on_any_instance` chain method to specify that it could happen on any instance of the experiment. This can be useful if you're calling `experiment(:example).track` downstream:
+### Tracking matcher
+
+Tracking events is a major aspect of experimentation, and because of this we try to provide a flexible way to ensure your tracking calls are covered.
 
 ```ruby
-expect(experiment(:example)).to track(:my_event).on_any_instance
+before do
+  stub_experiments(pill_color: true) # stub the experiment so tracking is permitted    
+end
+
+it "tests that we track an event on a specific instance" do
+  expect(subject = experiment(:pill_color)).to track(:clicked)
 
-experiment(:example).track(:my_event)
+  subject.track(:clicked)
+end
 ```
 
-And here's a full example of the methods that can be chained onto the `track` matcher:
+You can use the `on_next_instance` chain method to specify that the tracking call could happen on the next instance of the experiment. This can be useful if you're calling `experiment(:example).track` downstream and don't have access to that instance. Here's a full example of the methods that can be chained onto the `track` matcher:
 
 ```ruby
-expect(experiment(:example)).to track(:my_event, value: 1, property: '_property_')
-  .on_any_instance
-  .with_context(foo: :bar)
-  .for(:variant_name)
+it "tests that we track an event with specific details" do
+  expect(experiment(:pill_color)).to track(:clicked, value: 1, property: '_property_') 
+    .on_next_instance # any time in the future
+    .with_context(foo: :bar) # with the expected context
+    .for(:red) # and assigned the correct variant
 
-experiment(:example, :variant_name, foo: :bar).track(:my_event, value: 1, property: '_property_')
+  experiment(:pill_color, :red, foo: :bar).track(:clicked, value: 1, property: '_property_')
+end
 ```
 
 ## Tracking, anonymity and GDPR
@@ -560,16 +680,13 @@ Each of these approaches could be desirable given the objectives of your experim
 
 ## Development
 
-After checking out the repo, run `bundle install` to install dependencies.
-Then, run `bundle exec rake` to run the tests. You can also run `bundle exec pry` for an
-interactive prompt that will allow you to experiment.
+After cloning the repo, run `bundle install` to install dependencies.
+
+Run `bundle exec rake` to run the tests. You can also run `bundle exec pry` for an interactive prompt that will allow you to experiment.
 
 ## Contributing
 
-Bug reports and merge requests are welcome on GitLab at
-https://gitlab.com/gitlab-org/gitlab-experiment. This project is intended to be a
-safe, welcoming space for collaboration, and contributors are expected to adhere
-to the [Contributor Covenant](http://contributor-covenant.org) code of conduct.
+Bug reports and merge requests are welcome on GitLab at https://gitlab.com/gitlab-org/ruby/gems/gitlab-experiment. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [Contributor Covenant](http://contributor-covenant.org) code of conduct.
 
 ## Release process
 
@@ -577,13 +694,10 @@ Please refer to the [Release Process](docs/release_process.md).
 
 ## License
 
-The gem is available as open source under the terms of the
-[MIT License](http://opensource.org/licenses/MIT).
+The gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).
 
 ## Code of conduct
 
-Everyone interacting in the `Gitlab::Experiment` project’s codebases, issue trackers,
-chat rooms and mailing lists is expected to follow the
-[code of conduct](CODE_OF_CONDUCT.md).
+Everyone interacting in the `Gitlab::Experiment` project’s codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct](CODE_OF_CONDUCT.md).
 
 ***Make code not war***