gitlab-experiment 0.2.2 → 0.2.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: '08b63f640a90ed463a4b2bf58c2e0b1ef041bbd8f80c2d43bf88960ba9aac2a9'
-  data.tar.gz: 01a8a227c467d097b9dbb9dc9644a1d70012fb3989f2e189e3e63a41e7f39cfd
+  metadata.gz: 1a35b23202fa542fa548093c65ec4580e764fdbda2549291a54bfa3ed98cec96
+  data.tar.gz: 05f4c1ed8c5761ab03ed1e9d763d6e612f661cf9ee8736e46fb0e48b947ed52d
 SHA512:
-  metadata.gz: b161004e60bbbdb724aeb685013ce4b787d0ced9526bdd241e52be327adbc3461ea60eb3a279f25b4f89d46ee37825ae1d571501d0196a8505a9ff4eb8b3d214
-  data.tar.gz: 8730e755e09ec85fa88b2721af42ec0919d6534449a91d82bc0d2f6ae0055f70e4ad783d045d20dc09c44496c63465656abfe33d6759e3956a153d793f57a2ab
+  metadata.gz: 9505898d8870749dbccab8fa98974948895a9898838aecc622e9f3243624a889076b248f6c0698ad8d44c63726447cc1b46638158614bb008015a82ce42a8d19
+  data.tar.gz: 3484817693dbaf9254d16083c6fbb745fbe47adc7868bf924bd418d8f67735fd26ac00f7f486aecc4f9ec84647f58e7bfcb0ff15199370b2e782d68cf2a0b2a2

data/README.md

@@ -1,26 +1,47 @@
 GitLab Experiment
 =================
 
-Here at GitLab, experiments are run as A/B/n tests and are evaluated by the data the experiment generates. The team reviews the data, determines the variant that performed most effectively, and promotes that variant as the new default code path (or reverts back to the control).
+Here at GitLab, we run experiments as A/B/n tests and review the data the experiment generates. From that data, we determine the best performing variant and promote it as the new default code path. Or revert back to the control if no variant outperformed it.
 
-When we discuss the behavior of this gem, we'll use terms like experiment, control, candidate, and variant. It's worth defining these terms so they're more understood.
+This library provides a clean and elegant DSL to define, run, and track your GitLab experiment.
+
+When we discuss the behavior of this gem, we'll use terms like experiment, context, control, candidate, and variant. It's worth defining these terms so they're more understood.
 
 - `experiment` is any deviation of code paths we want to run sometimes and not others.
+- `context` is used to identify a consistent experience we'll provide in an experiment.
 - `control` is the default, or "original" code path.
-- `candidate` is used if there's one experimental code path.
+- `candidate` defines that there's one experimental code path.
 - `variant(s)` is used when more than one experimental code path exists.
 
-Candidate and variant are referencing the same concept, but can simplify how we speak about the available code paths of a given experiment.
+Candidate and variant are the same concept, but simplify how we speak about experimental paths.
+
+## Opinionated elevator pitch
+
+The last time I travelled I went through a process at the airport when I arrived. My passport was checked, I was asked a series of questions about my visit, and was ushered to a digital kiosk where I was prompted with a few more questions on a touch screen. 
+
+It was Iceland, and for the record, it's a beautiful place and you should visit if you haven't yet.
+
+Anyway, at various stages of this process travellers are presented with a physical emoji button interface, and are encouraged to rate their satisfaction within the various stage.
+
+Running an experiment could be to change some aspect of the process for only some travelers, let's say by routing them down a different hallway. At various stages of both hallways we still have the emoji interfaces that can be happily tapped or angrily jabbed in their passing.
+
+After a while we can compare the results and can evaluate which hallway had the better overall experience, based on the ratings provided at the various stages.
+
+This library is about keeping track of which passports we send down which hallway, so we can consistently route them down the same hallway, and to know which hallway they're rating when they do.
+
+In this model we don’t need to know anything about the passport holder unless we decide to "ask", as we determine which hallway to send them down initially.
+
+This library doesn't provide a system of linking passports back to their passport holders, but it doesn't explicitly make doing so impossible. Doing so is often not a relevant detail on well defined and well executed experiments.
 
 ## Installation
 
-Add the gem to your Gemfile and then bundle install.
+Add the gem to your Gemfile and then `bundle install`.
 
 ```ruby
 gem 'gitlab-experiment'
 ```
 
-If you're using Rails, you can install an initializer that provides basic configuration by running the install generator.  
+If you're using Rails, you can install the initializer. It provides basic configuration and documentation.
 
 ```shell
 $ rails generate gitlab-experiment:install
@@ -28,46 +49,51 @@ $ rails generate gitlab-experiment:install
 
 ## Implementing an experiment
 
-For the sake of our example, let's say we want to run an experiment for the interface we render for subscription cancellation.
+For the sake of an example let's make one up. Let's run an experiment on what we render for disabling desktop notifications.
+
+In our control (current world) we show a simple toggle interface that reads "Notifications". In our experiment we want a "Turn on/off desktop notifications" button with a confirmation.
 
-In our control (current world) we show a simple toggle interface that reads "Auto-renew", and in our experiment candidate we want to show a "Cancel subscription" button with a confirmation dialog. Ultimately the behavior will be the same, but the interface will be considerably different and may involve more or less steps.
+The behavior will be the same, but the interface will be different and may involve more or less steps.
  
-We hypothesize that making it more clear what the action is will help people in making a choice about taking that action or not. 
+This makes the action more clear and will help the user in making a choice about if that's what they want to do. Or that's what we're going to try to find out.
 
-We'll name our experiment `subscription_cancellation`. It's important to understand this name may be prefixed based on your configuration, so if you've set `config.name_prefix = "gitlab"` it would be `gitlab_subscription_cancellation`.   
+We'll name our experiment `notification_toggle`. This name is prefixed based on configuration. If you've set `config.name_prefix = 'gitlab'`, the experiment name would be `gitlab_notification_toggle` elsewhere.
 
-When you implement and run an experiment you'll need to provide the name you've given it, and a context hash. The context hash is used to determine which variant to provide, and is expected to be consistent between calls to your experiment, so a context hash key can be generated and cached -- which is used to consistently render the same experience given the same context.
+When you implement an experiment you'll need to provide a name, and a context. The name can show up in tracking calls, and potentially other aspects. The context determines the variant assigned, and should be consistent between calls. We'll discuss migrating context in later examples.
 
-In our experiment we're going to render one of two views. Control will be our current one, and candidate will be the new view with a cancel button and javascript confirm call.
+A context "key" represents the unique id of a context. It allows us to give the same experience between different calls to the experiment and can be used in caching.
+
+Now in our experiment we're going to render one of two views. Control will be our current view, and candidate will be the new toggle button with a confirmation flow.
  
 ```ruby
 class SubscriptionsController < ApplicationController
-  include Gitlab::Experiment::Dsl
-
   def show
-    experiment(:subscription_cancellation, user_id: user.id) do |e|
-      e.use { render_toggle_button } # control
-      e.try { render_cancel_button } # candidate
+    experiment(:notification_toggle, user_id: user.id) do |e|
+      e.use { render_toggle } # control
+      e.try { render_button } # candidate
     end
   end
 end
 ```
 
-You can also provide different variant names if you want to. In our case we might want to include the confirmation dialog step or not, to see which one behaves better within our wider experiment.  
+You can define the experiment using simple control/candidate paths, or provide named variants.
+
+Handling multi-variant experiments is up to the configuration you provide around resolving variants. But in our example we may want to try with and without the confirmation. We can run any number of variations in our experiments this way.
 
 ```ruby
-experiment(:subscription_cancellation, user_id: user.id) do |e|
-  e.use { render_toggle_button } # control
-  e.try(:variant_one) { render_cancel_button(confirmation: true) }
-  e.try(:variant_two) { render_cancel_button(confirmation: false) }
+experiment(:notification_toggle, user_id: user.id) do |e|
+  e.use { render_toggle } # control
+  e.try(:variant_one) { render_button(confirmation: true) }
+  e.try(:variant_two) { render_button(confirmation: false) }
 end
 ```
 
-Later, and elsewhere in code you can use the same `experiment` call to track events on the experiment. You can use this consistent event concept to build out funnels from the data that's being tracked. The important detail here is to use the same context between your calls to `experiment`. If the context is the same, we're able to consistently track the event in a way that associates it to which variant is being presented.
+Understanding how an experiment can change behavior is important in evaluating its performance.
+
+To this end, we track events that are important by calling the same experiment elsewhere in code. By using the same context, you'll have consistent behavior and the ability to track events to it.
 
 ```ruby
-exp = experiment(:subscription_cancellation, user_id: user.id)
-exp.track('clicked_button')
+experiment(:notification_toggle, user_id: user.id).track(:clicked_button)
 ```
 
 <details>
@@ -76,14 +102,14 @@ exp.track('clicked_button')
 ### Class level interface using `.run`
 
 ```ruby
-exp = Gitlab::Experiment.run(:subscription_cancellation, user_id: user.id) do |e|
-  # Context may be passed in the block, but must be finalized before calling 
+exp = Gitlab::Experiment.run(:notification_toggle, user_id: user.id) do |e|
+  # Context may be passed in the block, but must be finalized before calling
   # run or track.
   e.context(project_id: project.id) # add the project id to the context
-  e.variant(:candidate) # always run the candidate
+
   # Define the control and candidate variant.
-  e.use { toggle_button_interface } # control
-  e.try { cancel_button_interface } # candidate
+  e.use { render_toggle } # control
+  e.try { render_button } # candidate
 end
 
 # Track an event on the experiment we've defined.
@@ -93,14 +119,14 @@ exp.track(:clicked_button)
 ### Instance level interface
 
 ```ruby
-exp = Gitlab::Experiment.new(:subscription_cancellation, user_id: user.id)
-# Context may be passed in the block, but must be finalized before calling 
+exp = Gitlab::Experiment.new(:notification_toggle, user_id: user.id)
+# Context may be passed in the block, but must be finalized before calling
 # run or track.
 exp.context(project_id: project.id) # add the project id to the context
 
 # Define the control and candidate variant.
-exp.use { toggle_button_interface } # control
-exp.try { cancel_button_interface } # candidate
+exp.use { render_toggle } # control
+exp.try { render_button } # candidate
 
 # Run the experiment -- returning the result.
 exp.run
@@ -112,23 +138,23 @@ exp.track(:clicked_button)
 </details>
 
 <details>
-  <summary>You can define use custom classes...</summary>
+  <summary>You can define and use custom classes...</summary>
 
 ### Custom class
 
 ```ruby
-class CancellationExperiment < Gitlab::Experiment
+class NotificationExperiment < Gitlab::Experiment
   def initialize(variant_name = nil, **context, &block)
-    super(:subscription_cancellation, variant_name, **context, &block)
+    super(:notification_toggle, variant_name, **context, &block)
 
     # Define the control and candidate variant.
-    use { toggle_button_interface } # control
-    try { cancel_button_interface } # candidate
+    use { render_toggle } # control
+    try { render_button } # candidate
   end
 end
 
-exp = CancellationExperiment.new(user_id: user.id) do |e|
-  # Context may be passed in the block, but must be finalized before calling 
+exp = NotificationExperiment.new(user_id: user.id) do |e|
+  # Context may be passed in the block, but must be finalized before calling
   # run or track.
   e.context(project_id: project.id) # add the project id to the context
 end
@@ -143,24 +169,24 @@ exp.track(:clicked_button)
 </details>
 
 <details>
-  <summary>You can also hard specify the variant to use...</summary>
+  <summary>You can also specify the variant to use...</summary>
 
-### Specifying which variant to use
+### Specifying variant
 
-This should generally be discouraged, as it can change the experience users have during rollout, and may confuse generating reports from the tracking calls. It is possible however, and may be useful if you understand the implications.
+You can hardcode the variant if you want. It's important to know what this might do to your data during rollout, so use this with consideration.
 
 ```ruby
-experiment(:subscription_cancellation, :no_interface, user_id: user.id) do |e|
-  e.use { toggle_button_interface } # control
-  e.try { cancel_button_interface } # candidate
+experiment(:notification_toggle, :no_interface, user_id: user.id) do |e|
+  e.use { render_toggle } # control
+  e.try { render_button } # candidate
   e.try(:no_interface) { no_interface! } # variant
 end
 ```
 
-Or you can set the variant within the block -- potentially using some unique or different segmentation strategy that you've written specifically for the experiment at hand.
+Or you can set the variant within the block. This allows using unique segmentation logic or variant resolution if you need it.
 
 ```ruby
-experiment(:subscription_cancellation, user_id: user.id) do |e|
+experiment(:notification_toggle, user_id: user.id) do |e|
   e.variant(:no_interface) # set the variant
   # ...
 end
@@ -168,51 +194,150 @@ end
 
 </details>
 
-The `experiment` method, and the underlying `Gitlab::Experiment` instance is an implementation on top of [Scientist](https://github.com/github/scientist). Generally speaking you can use the DSL that Scientist defines, but for experiments we use `experiment` instead of `science`, and specify the variant on initialization (or via `#variant` and not in the call to `#run`. The interface is otherwise the same, even though not every aspect of Scientist makes sense for experiments.
+### Return value
+
+By default the return value is a `Gitlab::Experiment` instance. In simple cases you may want only the results of the experiment though. You can call `run` within the block to get the return value of the assigned variant.
+
+```ruby
+experiment(:notification_toggle) do |e|
+  e.use { 'A' }
+  e.try { 'B' }
+  e.run
+end # => 'A'
+```
+
+### Including the DSL
+
+By default, `Gitlab::Experiment` injects itself into the controller and view layers. This exposes the `experiment` method application wide in those layers.
+
+Some experiments may extend outside of those layers, so you may want to include it elsewhere. For instance in a mailer, service object, background job, or similar.
+
+Note: In a lot of these contexts you may not have a reference to the request (unless you pass it in, or provide access to it) which may be needed if you want to enable cookie behaviors and track that through to user conversion.
+
+```ruby
+class UserMailer < ApplicationMailer
+  include Gitlab::Experiment::Dsl # include the `experiment` method
+
+  def welcome
+    @user = params[:user]
+
+    ex = experiment(:project_suggestions, user_id: @user.id) do |e|
+      e.use { 'welcome' }
+      e.try { 'welcome_with_project_suggestions' }
+    end
+
+    mail(to: @user.email, subject: 'Welcome!', template: ex.run)
+  end
+end
+```
 
 ### Context migrations
 
-There are times when we may need to change something that we're providing in the context while an experiment is running, or even add or remove contexts. We make this possible by passing the migration data to the experiment.
+There are times when we need to change context while an experiment is running. We make this possible by passing the migration data to the experiment.
+
+Take for instance, that you might be using `version: 1` in your context currently. To migrate this `version: 2`, provide the context to change using a `migrated_with` option.
 
-Take for instance, that you might be using `version: 1` in your context currently. If you want to migrate this to `version: 2`, you just need to provide the context that you want to change using a `migrated_with` option. In doing this, a given experience (variant rendered/events tracked) can be resolved back through any number of migrations, and can be cached/resolved by using the context key value that was already already generated.
+In providing the context migration data, we can resolve an experience and its events all the way back. This can also help in keeping our cache relevant.
 
 ```ruby
 experiment(:my_experiment, user_id: 42, version: 2, migrated_with: { version: 1 })
 ```
 
-If you're adding or removing a new a value from the context, you'll need to use `migrated_from`, which expects a full context replacement -- e.g. what it was before you added or removed the new context key. For instance, if you wanted to introduce the `version: 1` concept to your context, you would need to use something like the following. 
+You can add or remove context by providing a `migrated_from` option. This approach expects a full context replacement -- e.g. what it was before you added or removed the new context key.
+
+If you wanted to introduce a `version` to your context, provide the full previous context.
  
 ```ruby
 experiment(:my_experiment, user_id: 42, version: 1, migrated_from: { user_id: 42 })
 ```
 
-It's important to understand that this can bucket a user in a new experience (depending on the logic in determining in the variant), so you should investigate how this might impact your experiment before using it and that your experiment is implemented in a way that supports migrations.
+This can impact an experience if you haven't:
+
+1. implemented the concept of migrations in your variant resolver
+1. haven't enabled a reasonable caching mechanism
 
 ### When there isn't a user (cookies)
 
-When there isn't a user we typically have to fall back to another concept to provide a consistent experience. What this means, is that once we assign someone a certain variant, we want to always give them the same experience, and we do this by setting a cookie for the experiment we're currently checking on. This cookie value is a random uuid, which we auto migrate to a user_id when our experiment context is finally provided that information. This means that you really only need to provide the request as an option if you're wanting to have an experiment that flows from not-signed-in (or not registered) users to eventually signed-in users.
+When there isn't an identifying key in the context (this is `user_id` by default), we fall back to cookies to provide a consistent experience.
+
+Once we assign a certain variant to a context, we need to always provide the same experience. We achieve this by setting a cookie for the experiment in question.
 
-This is considered a temporary cookie value, and isn't used for tracking purposes other than to give a given "user" (in this case it's actually the browser), a consistent experience after we've assigned one.
+This cookie is a randomized uuid and isn't associated with the user. When we can finally provide an identifying key, the context is auto migrated from the cookie to that identifying key. The cookie is a temporary value, and isn't used for tracking.
 
-To read and write cookies, we allow for passing the `request` as an option. This allows us to read, write, and even clean up a cookie when appropriate. We've provided this by default in the ActionController interface though, so this is primarily useful if you're trying to have an experiment span controller and model layers. 
+To read and write cookies, we provide the `request` from within the controller and views. The cookie migration will happen automatically if the experiment is within those layers.
+
+You'll need to provide the `request` as an option to the experiment if it's outside of the controller and views.
 
 ```ruby
-experiment(:subscription_cancellation, user_id: user.id, request: request) do |e|
-  e.use { render_toggle_button } # control
-  e.try { render_cancel_button } # candidate
-end
-``` 
+experiment(:my_experiment, user_id: user&.id, request: request)
+```
+
+The cookie isn't set if the identifying key isn't present in the context. In this case, if there was no `user_id` key provided, the cookie wouldn't be set.
 
-If needed, for edge cases (like in a background job), you can manually pass in the cookie value. Passing it in as `user_id: request.cookie_jar.signed['subscription_cancellation_id']`. The cookie name is the experiment name (prefixed if configured to be) with `_id` appended.
+For edge cases, you can pass the cookie through by assigning it yourself -- e.g. `user_id: request.cookie_jar.signed['my_experiment_id']`. The cookie name is the experiment name (prefixed if configured) with `_id` appended.
 
 ## Configuration
 
-The gem is meant to be configured before being used. The default configuration will always render the control behavior, so it's important to implement your own logic for this or you will always get the control of an experiment.
+This gem needs to be configured before being used in a meaningful way.
 
-The most important aspect of the gem, determining which variant to render and when, is up to you, and you may want to consider using [Unleash](https://github.com/Unleash/unleash-client-ruby) (which has the concept of multi-variants built in), or [Flipper](https://github.com/jnunemaker/flipper) in helping with this.
+The default configuration will always render the control. So it's important to configure your own logic for resolving variants.
 
-Examples for configuration are available in the provided install generator, or in the source code configuration.rb file itself.
+Yes, the most important aspect of the gem, determining which variant to render and when, is up to you. Consider using [Unleash](https://github.com/Unleash/unleash-client-ruby) or [Flipper](https://github.com/jnunemaker/flipper) for this.
+
+```ruby
+Gitlab::Experiment.configure do |config|
+  config.variant_resolver = lambda do |requested_variant|
+    # Return the requested variant if a specific one has been provided in code.
+    return requested_variant unless requested_variant.nil?
+
+    # Ask Unleash to determine the variant, given the context we've built,
+    # using the control as the fallback.
+    fallback = Unleash::Variant.new(name: 'control', enabled: true)
+    UNLEASH.get_variant(name, context.value, fallback)
+  end
+end
+```
+
+More examples for configuration are available in the provided [rails initializer](lib/generators/gitlab_experiment/install/templates/initializer.rb).
+
+### Client layer / JavaScript
+
+This library doesn't attempt to provide any logic for the client layer.
+
+Instead it allows you to do this yourself in configuration. Using Gon to publish your experiment information to the client layer is pretty simple.
+
+```ruby
+Gitlab::Experiment.configure do |config|
+  config.publishing_behavior = lambda do |_result|
+    # Push the experiment knowledge into the front end. The signature contains
+    # the context key, and the variant that has been determined.
+    Gon.push(experiment: { name => signature })
+  end
+end
+```
+
+In the client you can now access `window.gon.experiment.notificationToggle`.
+
+### Caching
+
+Caching can be enabled in configuration, and is implemented towards the `Rails.cache` / `ActiveSupport::Cache::Store` interface. When you enable caching, any variant resolution will be cached. Migrating the cache through context migrations is handled automatically, and this helps ensure an experiment experience remains consistent.
+
+```ruby
+Gitlab::Experiment.configure do |config|
+  config.cache = Rails.cache
+end
+```
 
 ## Tracking, anonymity and GDPR
 
-We intentionally don't, and shouldn't, track things like user ids on our experiments. What we can and do track is what we consider an "experiment experience" key. This key is generated from the context we pass to the experiment and has the concept of migrating through different versions of context. If we consistently pass the same context to an experiment, we're able to consistently track events generated in that experience. A context can contain things like user, or project -- so, if you only included a user in the context that user would get the same experience across all projects they view, but if you include the currently viewed project in the context the user would potentially have a different experience on each of their projects. Each can be desirable given the objectives of the experiment.
+We generally try not to track things like user identifying values in our experimentation. What we can and do track is the "experiment experience" (a.k.a the context key).
+
+We generate this key from the context passed to the experiment. This allows creating funnels without exposing any user information.
+
+This library attempts to be non user centric, in that a context can contain things like a user, or a project.
+
+If you only include a user, that user would get the same experience across every project they view. If you only include the project, every user who views that project would get the same experience.
+
+Each of these approaches could be desirable given the objectives of your experiment.
+
+### Make code not war

data/lib/generators/gitlab_experiment/install/templates/initializer.rb

@@ -7,6 +7,9 @@ Gitlab::Experiment.configure do |config|
   # The logger is used to log various details of the experiments.
   config.logger = Logger.new(STDOUT)
 
+  # The caching layer is expected to respond to fetch, like Rails.cache.
+  config.cache = nil
+
   # Logic this project uses to resolve a variant for a given experiment.
   #
   # This can return an instance of any object that responds to `name`, or can
@@ -16,53 +19,62 @@ Gitlab::Experiment.configure do |config|
   # This block will be executed within the scope of the experiment instance,
   # so can easily access experiment methods, like getting the name or context.
   config.variant_resolver = lambda do |requested_variant|
-    # An example of running the control, unless a variant was requested:
+    # Run the control, unless a variant was requested in code:
     requested_variant || 'control'
 
-    # Always run candidate, unless a variant was requested, with fallback:
-    # variant_name || variant_names.first || 'control'
+    # Run the candidate, unless a variant was requested, with a fallback:
+    #
+    # requested_variant || variant_names.first || 'control'
 
     # Using unleash to determine the variant:
-    # TODO: this isn't entirely accurate.
+    #
     # fallback = Unleash::Variant.new(name: requested_variant || 'control', enabled: true)
-    # UNLEASH.get_variant(name, context.value, fallback)
+    # Unleash.get_variant(name, context.value, fallback)
 
     # Using Flipper to determine the variant:
+    #
     # TODO: provide example.
-    # Variant.new(name: resolved)
+    # Variant.new(name: requested_variant || 'control')
   end
 
   # Tracking behavior can be implemented to link an event to an experiment.
   #
-  # Similar to the variant resolver, this is called within the scope of the
+  # Similar to the variant_resolver, this is called within the scope of the
   # experiment instance and so can access any methods on the experiment.
-  config.tracking_behavior = lambda do |action, event_args|
+  #
+  #
+  config.tracking_behavior = lambda do |event, args|
     # An example of using a generic logger to track events:
-    (event_args[:context] ||= []) << context.signature.merge(group: variant.name)
-    config.logger.info "Gitlab::Experiment[#{name}] #{action}: #{event_args}"
+    config.logger.info "Gitlab::Experiment[#{name}] #{event}: #{args.merge(signature: signature)}"
 
     # Using something like snowplow to track events:
-    # (event_args[:context] ||= []) << SnowplowTracker::SelfDescribingJson.new(
-    #   'iglu:com.gitlab/gitlab_experiment/jsonschema/1-0-0',
-    #   experiment: context.signature.merge(group: variant.name)
-    # )
     #
-    # Tracking.event(name, action, **event_args)
+    # Gitlab::Tracking.event(name, event, **args.merge(
+    #   context: (args[:context] || []) << SnowplowTracker::SelfDescribingJson.new(
+    #     'iglu:com.gitlab/gitlab_experiment/jsonschema/0-2-0',
+    #     signature: signature
+    #   )
+    # ))
   end
 
   # Called at the end of every experiment run, with the result.
   #
   # You may want to track that you've assigned a variant to a given context,
   # or push the experiment into the client or publish results elsewhere, like
-  # into redis.
-  config.publishing_behavior = lambda do |_result|
-    track(:assignment) # this will call the config.tracking_behavior
-
-    # Log the results, so we can inspect them if we wanted to later.
-    # LogRage.log(result: _result)
+  # into redis. Also called within the scope of the experiment instance.
+  config.publishing_behavior = lambda do |result|
+    # Track the event using our own configured tracking logic.
+    track(:assignment)
 
-    # Push the experiment knowledge into the client using Gon.
+    # Push the experiment knowledge into the front end. The signature contains
+    # the context key, and the variant that has been determined.
+    #
     # Gon.push(experiment: { name => signature })
+
+    # Log using our logging system, so the result (which can be large) can be
+    # reviewed later if we want to.
+    #
+    # Lograge::Event.log(experiment: name, result: result, signature: signature)
   end
 
   # Algorithm that consistently generates a hash key for a given hash map.

data/lib/gitlab/experiment.rb

@@ -2,6 +2,7 @@
 
 require 'scientist'
 
+require 'gitlab/experiment/caching'
 require 'gitlab/experiment/configuration'
 require 'gitlab/experiment/context'
 require 'gitlab/experiment/dsl'
@@ -12,6 +13,7 @@ require 'gitlab/experiment/engine' if defined?(Rails::Engine)
 module Gitlab
   class Experiment
     include Scientist::Experiment
+    include Caching
 
     class << self
       def configure
@@ -26,7 +28,7 @@ module Gitlab
       end
     end
 
-    delegate :signature, to: :context
+    delegate :signature, :cache_strategy, to: :context
 
     def initialize(name, variant_name = nil, **context)
       @name = name
@@ -49,13 +51,14 @@ module Gitlab
     end
 
     def variant(value = nil)
-      @variant_name = value unless value.nil?
+      return @variant_name = value unless value.nil?
+
       result = instance_exec(@variant_name, &Configuration.variant_resolver)
       result.respond_to?(:name) ? result : Variant.new(name: result.to_s)
     end
 
     def run
-      @result ||= super(variant.name)
+      @result ||= super(cache { variant.name })
     end
 
     def publish(result)
@@ -71,13 +74,21 @@ module Gitlab
     end
 
     def variant_names
-      @variant_names = behaviors.keys.tap { |keys| keys.delete('control') }.map(&:to_sym)
+      @variant_names ||= behaviors.keys.tap { |keys| keys.delete('control') }.map(&:to_sym)
     end
 
     def enabled?
       true
     end
 
+    def identifying_key
+      :user_id
+    end
+
+    def cache_key_for(key, migration: false)
+      "#{name}:#{key}"
+    end
+
     protected
 
     def generate_result(variant_name)

data/lib/gitlab/experiment/caching.rb

@@ -0,0 +1,24 @@
+# frozen_string_literal: true
+
+module Gitlab
+  class Experiment
+    module Caching
+      def cache(&block)
+        return yield unless (cache = Configuration.cache)
+
+        key, migrations = cache_strategy
+        migrated_cache(cache, migrations || [], key) or cache.fetch(key, &block)
+      end
+
+      def migrated_cache(cache, migrations, new_key)
+        migrations.find do |old_key|
+          next unless (value = cache.read(old_key))
+
+          cache.write(new_key, value)
+          cache.delete(old_key)
+          break value
+        end
+      end
+    end
+  end
+end

data/lib/gitlab/experiment/configuration.rb

@@ -15,20 +15,23 @@ module Gitlab
       # The logger is used to log various details of the experiments.
       @logger = Logger.new(STDOUT)
 
+      # Cache layer. Expected to respond to fetch, like Rails.cache.
+      @cache = nil
+
       # Logic this project uses to resolve a variant for a given experiment.
       @variant_resolver = lambda do |requested_variant|
         requested_variant || 'control'
       end
 
       # Tracking behavior can be implemented to link an event to an experiment.
-      @tracking_behavior = lambda do |action, event_args|
-        Configuration.logger.info "Gitlab::Experiment[#{name}] #{action}: #{event_args.merge(signature: signature)}"
+      @tracking_behavior = lambda do |event, args|
+        Configuration.logger.info "Gitlab::Experiment[#{name}] #{event}: #{args.merge(signature: signature)}"
       end
 
       # Called at the end of every experiment run, with the results. You may
       # want to push the experiment into the client or push results elsewhere.
       @publishing_behavior = lambda do |_result|
-        track(:assignment) # this will call the config.tracking_behavior
+        track(:assignment)
       end
 
       # Algorithm that consistently generates a hash key for a given hash map.
@@ -38,7 +41,7 @@ module Gitlab
       end
 
       class << self
-        attr_accessor :name_prefix, :logger
+        attr_accessor :name_prefix, :logger, :cache
         attr_accessor :variant_resolver, :tracking_behavior, :publishing_behavior, :context_hash_strategy
       end
     end

data/lib/gitlab/experiment/context.rb

@@ -16,18 +16,27 @@ module Gitlab
         return @value if value.nil?
 
         value = value.dup # dup so we don't mutate
-        @signature = nil # clear memoized signature
+        @signature = @cache_strategy = nil # clear memoization
 
         @migrations_from << value.delete(:migrated_from) if value[:migrated_from]
         @migrations_with << value.delete(:migrated_with) if value[:migrated_with]
-        @value.merge!(migrate_cookie_to_user_id(value))
+        @value.merge!(auto_migrate_cookie(value, value.delete(:request)))
       end
 
       def freeze
-        signature # ensure we're done before being frozen
+        cache_strategy # ensure we memoize before freezing
         super
       end
 
+      def cache_strategy
+        @cache_strategy ||= [
+          @experiment.cache_key_for(signature[:key]),
+          signature[:migration_keys]&.map do |key|
+            @experiment.cache_key_for(key, migration: true)
+          end
+        ]
+      end
+
       def signature
         @signature ||= {
           key: key_for(@value),
@@ -38,16 +47,18 @@ module Gitlab
 
       private
 
-      def migrate_cookie_to_user_id(hash)
-        return hash unless (request = hash.delete(:request))
-        return hash unless request.respond_to?(:headers) && request.respond_to?(:cookie_jar)
+      def auto_migrate_cookie(hash, request)
+        return hash unless request&.respond_to?(:headers) && request&.respond_to?(:cookie_jar)
         return hash if request.headers['DNT'].to_s.match?(DNT_REGEXP)
 
-        jar = request.cookie_jar
-        resolver = [jar, hash, :user_id, jar.signed[cookie_name]].compact
+        resolver = cookie_resolver(request.cookie_jar, hash)
         resolve_cookie(*resolver) or generate_cookie(*resolver)
       end
 
+      def cookie_resolver(jar, hash)
+        [jar, hash, @experiment.identifying_key, jar.signed[cookie_name]].compact
+      end
+
       def cookie_name
         @cookie_name ||= [@experiment.name, 'id'].join('_')
       end
@@ -56,13 +67,15 @@ module Gitlab
         return if cookie.blank? && hash[key].blank?
         return hash.merge(key => cookie) if hash[key].blank?
 
-        @migrations_with << { user_id: cookie }
+        @migrations_with << { key => cookie }
         jar.delete(cookie_name, domain: :all)
 
         hash
       end
 
       def generate_cookie(jar, hash, key, cookie = SecureRandom.uuid)
+        return hash unless hash.key?(key)
+
         jar.permanent.signed[cookie_name] = {
           value: cookie, secure: true, domain: :all, httponly: true
         }

data/lib/gitlab/experiment/version.rb

@@ -2,6 +2,6 @@
 
 module Gitlab
   class Experiment
-    VERSION = '0.2.2'
+    VERSION = '0.2.3'
   end
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: gitlab-experiment
 version: !ruby/object:Gem::Version
-  version: 0.2.2
+  version: 0.2.3
 platform: ruby
 authors:
 - GitLab
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2020-09-24 00:00:00.000000000 Z
+date: 2020-09-27 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: scientist
@@ -43,6 +43,7 @@ files:
 - lib/generators/gitlab_experiment/install/install_generator.rb
 - lib/generators/gitlab_experiment/install/templates/initializer.rb
 - lib/gitlab/experiment.rb
+- lib/gitlab/experiment/caching.rb
 - lib/gitlab/experiment/configuration.rb
 - lib/gitlab/experiment/context.rb
 - lib/gitlab/experiment/dsl.rb