gitlab-experiment 0.2.0 → 0.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: fae9519317810c50decdff810249899d5c1855881544a20a31a2219cf9bda715
-  data.tar.gz: 19987d9c24486e99424aa6cfcade34d3123dbe1fc4a6afb8dc6c4c6ad4f5a99b
+  metadata.gz: 6134f703a49eb7411ff59e1fa7778421890498a537419cd765a99c5e07ff1523
+  data.tar.gz: ea3736188b46c9527818109dc87839c33937e0c9086bae4464992046f425874a
 SHA512:
-  metadata.gz: afe30984c7b7f96bb4ad1122c5a4f5ff48a45f8c7cfca5b1d2d04368ba3eef25c84776e3768588cde6cd717ee918b25e8158746ea9ea3e9bb20fe9ef39d20377
-  data.tar.gz: 2492d43cbc76cfa20c1b785033fdc635ca75874d1052270c950a86acff17d04011c406bf1879bc5f21b06fb604c5ed259c149aecbea33efe2adae59ab12f6164
+  metadata.gz: ec048634699257e018d7e67f290dfa2dd5e2712d58b1ef25eeb1c9353c5a23ad2f7a8ec0484493e6bccaabc57801aa22ceb8632a5a8c355652ae4a5541a43286
+  data.tar.gz: 51f63fb0256c49f393887451723483233927a0a983c51ca80f98494634e156dab28d0f87f21319a4df750392bb8329923e94ab23091929b3b26904edb25ce5b3

data/README.md

@@ -1,182 +1,354 @@
-# Experiment Guide
+# GitLab Experiment
 
-Experiments can be conducted by any GitLab team, but are most often conducted by the teams from the [Growth Sub-department](https://about.gitlab.com/handbook/engineering/development/growth/).
+<img alt="experiment" src="/uploads/60990b2dbf4c0406bbf8b7f998de2dea/experiment.png" align="right" width="40%">
 
-Experiments are run as A/B/n tests and are evaluated by the data the experiment generates. The team reviews the data, determines the variant that performed most effectively, and promotes that variant as the new default code path (or reverts back to the control). In all instances, an experiment will be cleaned up after it has generated enough data to determine performance and be considered resolved.
+Here at GitLab, we run experiments as A/B/n tests and review the data the experiment generates. From that data, we determine the best performing variant and promote it as the new default code path. Or revert back to the control if no variant outperformed it.
 
-## Process and tracking issue
+This library provides a clean and elegant DSL to define, run, and track your GitLab experiment.
 
-Each experiment should have a related [Experiment tracking issue](https://gitlab.com/gitlab-org/gitlab/-/issues/new?issuable_template=Experiment%20tracking) created, which is intended to track the experiment from deployment and rollout through to resolution and cleanup.
+When we discuss the behavior of this gem, we'll use terms like experiment, context, control, candidate, and variant. It's worth defining these terms so they're more understood.
 
-At the time an experiment is rolled out, the due date of the tracking issue should be specified. The timeline depends on the experiment and can be up to several weeks in the future.
+- `experiment` is any deviation of code paths we want to run sometimes and not others.
+- `context` is used to identify a consistent experience we'll provide in an experiment.
+- `control` is the default, or "original" code path.
+- `candidate` defines that there's one experimental code path.
+- `variant(s)` is used when more than one experimental code path exists.
 
-With experiment resolution, the outcome of the experiment should be posted to the tracking issue with the reasoning for the decision. Any and all non-relevant experiment code should be removed and review concerns should be addressed during cleanup. After cleanup, the tracking issue can be closed.
+Candidate and variant are the same concept, but simplify how we speak about experimental paths.<br clear="all">
+
+## Installation
+
+Add the gem to your Gemfile and then `bundle install`.
+
+```ruby
+gem 'gitlab-experiment'
+```
+
+If you're using Rails, you can install the initializer. It provides basic configuration and documentation.
+
+```shell
+$ rails generate gitlab-experiment:install
+```
 
 ## Implementing an experiment
 
-For the sake of our example, let's say we want to run an experiment for how to cancel a subscription. In our control (current world) we show a toggle that reads "Auto-renew", and in our experiment candidate we want to show a "Cancel subscription" button with a confirmation. Ultimately the behavior is the same, but the interface will be considerably different.
+For the sake of an example let's make one up. Let's run an experiment on what we render for disabling desktop notifications.
+
+In our control (current world) we show a simple toggle interface labeled, "Notifications." In our experiment we want a "Turn on/off desktop notifications" button with a confirmation.
 
-### Defining the feature flag
+The behavior will be the same, but the interface will be different and may involve more or fewer steps.
 
-Let's name our experiment `subscription_cancellation`. It's important to understand how this name is prefixed as `growth_experiment_subscription_cancellation` in our Unleash feature detection and in our Snowplow tracking calls. We prefix our experiments so we can consistently identify them as experiments and clean them up over time.
+Our hypothesis is that this will make the action more clear and will help in making a choice about if that's what the user really wants to do.
 
-This means that you'll need to go to the [Feature Flags interface](https://gitlab.com/gitlab-org/gitlab/-/feature_flags) interface (for the project you're working on) and add a `growth_experiment_subscription_cancellation` feature flag and define which environment(s) it should be rolled out on for initial deployment, for instance, `gitlab-staging` or `customers-staging`.
+We'll name our experiment `notification_toggle`. This name is prefixed based on configuration. If you've set `config.name_prefix = 'gitlab'`, the experiment name would be `gitlab_notification_toggle` elsewhere.
 
-You can include yourself and others in a list and assign that list using the `User List` rollout strategy, or you can add your user id to the experiment manually. You can use this to include or exclude yourself from an experiment for verification before rolling it out to others.
+When you implement an experiment you'll need to provide a name, and a context. The name can show up in tracking calls, and potentially other aspects. The context determines the variant assigned, and should be consistent between calls. We'll discuss migrating context in later examples.
 
-### Implementation
+A context "key" represents the unique id of a context. It allows us to give the same experience between different calls to the experiment and can be used in caching.
 
-When you implement an experiment in code you'll need to provide the name that you've given it in the feature flags interface, and a context -- which usually will include something like a user / user id, but may also include several other aspects.
+Now in our experiment we're going to render one of two views: the control will be our current view, and the candidate will be the new toggle button with a confirmation flow.
 
 ```ruby
 class SubscriptionsController < ApplicationController
-  include Gitlab::GrowthExperiment::Interface
-
   def show
-    experiment(:subscription_cancellation, user_id: user.id) do |e|
-      e.use { render_toggle_button } # control
-      e.try { render_cancel_button } # candidate
+    experiment(:notification_toggle, actor: user) do |e|
+      e.use { render_toggle } # control
+      e.try { render_button } # candidate
     end
   end
 end
 ```
 
-You can also provide different variants for the experience if you've defined variants in the Feature Flag interface (not yet available).
+You can define the experiment using simple control/candidate paths, or provide named variants.
+
+Handling multi-variant experiments is up to the configuration you provide around resolving variants. But in our example we may want to try with and without the confirmation. We can run any number of variations in our experiments this way.
 
 ```ruby
-experiment(:subscription_cancellation, user_id: user.id) do |e|
-  e.use { render_toggle_button } # control
-  e.try(:variant_one) { render_cancel_button(confirmation: true) }
-  e.try(:variant_two) { render_cancel_button(confirmation: false) }
+experiment(:notification_toggle, actor: user) do |e|
+  e.use { render_toggle } # control
+  e.try(:variant_one) { render_button(confirmation: true) }
+  e.try(:variant_two) { render_button(confirmation: false) }
 end
 ```
 
-Later, and elsewhere in code you can use the same `experiment` call to track events on the experiment. The important detail here is to use the same context between your calls to `experiment`. If the context is the same, we're able to consistently track the event in a way that associates it to which variant is being presented -- which may be based on the user we're presenting it to.
+Understanding how an experiment can change behavior is important in evaluating its performance.
+
+To this end, we track events that are important by calling the same experiment elsewhere in code. By using the same context, you'll have consistent behavior and the ability to track events to it.
 
 ```ruby
-exp = experiment(:subscription_cancellation, user_id: user.id)
-exp.track('clicked_button')
+experiment(:notification_toggle, actor: user).track(:clicked_button)
 ```
 
 <details>
-  <summary>You can use the more low level class or instance interfaces...</summary>
+  <summary>You can also use the more low level class or instance interfaces...</summary>
 
 ### Class level interface using `.run`
 
 ```ruby
-exp = Gitlab::GrowthExperiment.run(:subscription_cancellation, user_id: user.id) do |e|
-  # context can be passed to `experiment`, `.run`, `new`, or after the fact like here.
-  # context must be added before `#run` or `#track` calls.
-  e.context(project_id: project.id)
-
-  e.use { toggle_button_interface } # control
-  e.try { cancel_button_interface } # candidate
+exp = Gitlab::Experiment.run(:notification_toggle, actor: user) do |e|
+  # Context may be passed in the block, but must be finalized before calling
+  # run or track.
+  e.context(project: project) # add the project to the context
+
+  # Define the control and candidate variant.
+  e.use { render_toggle } # control
+  e.try { render_button } # candidate
 end
 
-# track an event on the experiment we've defined.
+# Track an event on the experiment we've defined.
 exp.track(:clicked_button)
 ```
 
-While `Gitlab::GrowthExperiment.run` is what we document, you can also use `Gitlab::GrowthExperiment.experiment`.
-
 ### Instance level interface
 
 ```ruby
-exp = Gitlab::GrowthExperiment.new(:subscription_cancellation, user_id: user.id)
-# context can be passed to `.new`, or after the fact like here.
-# context must be added before `#run` or `#track` calls.
-exp.context(project_id: project.id)
-exp.use { toggle_button_interface } # control
-exp.try { cancel_button_interface } # candidate
+exp = Gitlab::Experiment.new(:notification_toggle, actor: user)
+# Additional context may be provided to the instance (exp) but must be
+# finalized before calling run or track.
+exp.context(project: project) # add the project id to the context
+
+# Define the control and candidate variant.
+exp.use { render_toggle } # control
+exp.try { render_button } # candidate
+
+# Run the experiment, returning the result.
 exp.run
 
-# track an event on the experiment we've defined.
+# Track an event on the experiment we've defined.
 exp.track(:clicked_button)
 ```
 
 </details>
 
 <details>
-  <summary>You can define use custom classes...</summary>
+  <summary>You can define and use custom classes...</summary>
 
 ### Custom class
 
 ```ruby
-class CancellationExperiment < Gitlab::GrowthExperiment
+class NotificationExperiment < Gitlab::Experiment
   def initialize(variant_name = nil, **context, &block)
-    super(:subscription_cancellation, variant_name, **context, &block)
+    super(:notification_toggle, variant_name, **context, &block)
+
+    # Define the control and candidate variant.
+    use { render_toggle } # control
+    try { render_button } # candidate
   end
 end
 
-exp = CancellationExperiment.run(user_id: user.id) do |e|
-  # context can be passed to `.run`, or after the fact like here.
-  # context must be added before `#run` or `#track` calls.
-  e.context(project_id: project.id)
-
-  e.use { toggle_button_interface } # control
-  e.try { cancel_button_interface } # candidate
+exp = NotificationExperiment.new(actor: user) do |e|
+  # Context may be provided within the block or to the instance (exp) but must
+  # be finalized before calling run or track.
+  e.context(project: project) # add the project id to the context
 end
 
-# track an event on the experiment we've defined.
+# Run the experiment -- returning the result.
+exp.run
+
+# Track an event on the experiment we've defined.
 exp.track(:clicked_button)
 ```
 
 </details>
 
 <details>
-  <summary>You can hard specify the variant to use...</summary>
+  <summary>You can also specify the variant to use...</summary>
 
-### Specifying which variant to use
+### Specifying variant
 
-This should generally be discouraged, as it can change the experience users have during rollout, and may confuse generating reports from the tracking calls. It is possible however, and may be useful if you understand the implications.
+You can hardcode the variant if you want. It's important to know what this might do to your data during rollout, so use this with consideration.
 
 ```ruby
-experiment(:subscription_cancellation, :no_interface, user_id: user.id) do |e|
-  e.use { toggle_button_interface } # control
-  e.try { cancel_button_interface } # candidate
+experiment(:notification_toggle, :no_interface, actor: user) do |e|
+  e.use { render_toggle } # control
+  e.try { render_button } # candidate
   e.try(:no_interface) { no_interface! } # variant
 end
 ```
 
-Or you can set the variant within the block.
+Or you can set the variant within the block. This allows using unique segmentation logic or variant resolution if you need it.
 
 ```ruby
-experiment(:subscription_cancellation, user_id: user.id) do |e|
-  e.variant(:variant) # set the variant
+experiment(:notification_toggle, actor: user) do |e|
+  # Variant selection must be done before calling run or track.
+  e.variant(:no_interface) # set the variant
   # ...
 end
 ```
 
+Or it can be specified in the call to run if you call it from within the block.
+
+```ruby
+experiment(:notification_toggle, actor: user) do |e|
+  # ...
+  # Variant selection can be specified when calling run.
+  e.run(:no_interface)
+end
+```
+
 </details>
 
-The `experiment` method, and the underlying `Gitlab::GrowthExperiment` is an implementation on top of [Scientist](https://github.com/github/scientist). Generally speaking you can use the DSL that Scientist defines, but for experiments we use `experiment` instead of `science`, and specify the variant on initialization (or via `#variant` and not in the call to `#run`. The interface is otherwise the same, even though not every aspect of Scientist makes sense for experiments.
+### Return value
+
+By default the return value is a `Gitlab::Experiment` instance. In simple cases you may want only the results of the experiment though. You can call `run` within the block to get the return value of the assigned variant.
+
+```ruby
+experiment(:notification_toggle) do |e|
+  e.use { 'A' }
+  e.try { 'B' }
+  e.run
+end # => 'A'
+```
+
+### Including the DSL
+
+By default, `Gitlab::Experiment` injects itself into the controller and view layers. This exposes the `experiment` method application wide in those layers.
+
+Some experiments may extend outside of those layers, so you may want to include it elsewhere. For instance in a mailer, service object, background job, or similar.
+
+Note: In a lot of these contexts you may not have a reference to the request (unless you pass it in, or provide access to it) which may be needed if you want to enable cookie behaviors and track that through to user conversion.
+
+```ruby
+class WelcomeMailer < ApplicationMailer
+  include Gitlab::Experiment::Dsl # include the `experiment` method
+
+  def welcome
+    @user = params[:user]
+
+    ex = experiment(:project_suggestions, actor: @user) do |e|
+      e.use { 'welcome' }
+      e.try { 'welcome_with_project_suggestions' }
+    end
+
+    mail(to: @user.email, subject: 'Welcome!', template: ex.run)
+  end
+end
+```
 
 ### Context migrations
 
-There are times when we may need to add new values or change something that we're providing in context while an experiment is running. We make this possible by passing the `migrated_from` context key.
+There are times when we need to change context while an experiment is running. We make this possible by passing the migration data to the experiment.
+
+Take for instance, that you might be using `version: 1` in your context currently. To migrate this to `version: 2`, provide the portion of the context you wish to change using a `migrated_with` option.
+
+In providing the context migration data, we can resolve an experience and its events all the way back. This can also help in keeping our cache relevant.
 
-Take for instance, that you might be using `version: 1` in your context. If you want to migrate this to `version: 2`, you just need to provide the context that you provided prior, in a `migrated_from` context key. In doing this, a given experiment experience can be resolved back through any number of migrations.
+```ruby
+# Migrate just the `:version` portion of the previous context, `{ actor: project, version: 1 }`:
+experiment(:my_experiment, actor: project, version: 2, migrated_with: { version: 1 })
+```
+
+You can add or remove context by providing a `migrated_from` option. This approach expects a full context replacement -- i.e. what it was before you added or removed the new context key.
+
+If you wanted to introduce a `version` to your context, provide the full previous context.
 
 ```ruby
-experiment(:my_experiment, version: 2, migrated_from: { version: 1 })
+# Migrate the full context from `{ actor: project }` to `{ actor: project, version: 1 }`:
+experiment(:my_experiment, actor: project, version: 1, migrated_from: { actor: project })
 ```
 
-It's important to understand that this can bucket a user in a new experience (depending on the rollout strategy being used and what is changing in the context), so you should investigate how this might impact your experiment before using it.
+This can impact an experience if you:
+
+1. haven't implemented the concept of migrations in your variant resolver
+1. haven't enabled a reasonable caching mechanism
+
+### When there isn't an actor (cookie fallback)
 
-### When there isn't a user
+When there isn't an identifying key in the context (this is `actor` by default), we fall back to cookies to provide a consistent experience for the client viewing them.
 
-When there isn't a user, we typically have to fall back to another concept to provide a consistent experiment experience. What this means, is that once we bucket someone in a certain bucket, we always bucket them in that bucket.
+Once we assign a certain variant to a context, we need to always provide the same experience. We achieve this by setting a cookie for the experiment in question, but only when needed.
+
+This cookie is a temporary, randomized uuid and isn't associated with a user. When we can finally provide an actor, the context is auto migrated from the cookie to that actor.
+
+To read and write cookies, we provide the `request` from within the controller and views. The cookie migration will happen automatically if the experiment is within those layers.
+
+You'll need to provide the `request` as an option to the experiment if it's outside of the controller and views.
+
+```ruby
+experiment(:my_experiment, actor: user, request: request)
+```
 
-We do this by using cookies.... [document more]
+The cookie isn't set if the `actor` key isn't present at all in the context. Meaning that when no `actor` key is provided, the cookie will not be set.
+
+```ruby
+# actor is not present, so no cookie is set
+experiment(:my_experiment, project: project)
+
+# actor is present and is nil, so the cookie is set and used
+experiment(:my_experiment, actor: nil, project: project)
+
+# actor is present and set to a value, so no cookie is set
+experiment(:my_experiment, actor: user, project: project)
+```
+
+For edge cases, you can pass the cookie through by assigning it yourself -- e.g. `actor: request.cookie_jar.signed['my_experiment_actor']`. The cookie name is the full experiment name (including any configured prefix) with `_actor` appended -- e.g. `gitlab_notification_toggle_actor` for the `:notification_toggle` experiment key with a configured prefix of `gitlab`.
+
+## Configuration
+
+This gem needs to be configured before being used in a meaningful way.
+
+The default configuration will always render the control, so it's important to configure your own logic for resolving variants.
+
+Yes, the most important aspect of the gem -- that of determining which variant to render and when -- is up to you. Consider using [Unleash](https://github.com/Unleash/unleash-client-ruby) or [Flipper](https://github.com/jnunemaker/flipper) for this.
+
+```ruby
+Gitlab::Experiment.configure do |config|
+  # The block here is evaluated within the scope of the experiment instance,
+  # which is why we are able to access things like name and context.
+  config.variant_resolver = lambda do |requested_variant|
+    # Return the requested variant if a specific one has been provided in code.
+    return requested_variant unless requested_variant.nil?
+
+    # Ask Unleash to determine the variant, given the context we've built,
+    # using the control as the fallback.
+    fallback = Unleash::Variant.new(name: 'control', enabled: true)
+    UNLEASH.get_variant(name, context.value, fallback)
+  end
+end
+```
+
+More examples for configuration are available in the provided [rails initializer](lib/generators/gitlab_experiment/install/templates/initializer.rb).
+
+### Client layer / JavaScript
+
+This library doesn't attempt to provide any logic for the client layer.
+
+Instead it allows you to do this yourself in configuration. Using [Gon](https://github.com/gazay/gon) to publish your experiment information to the client layer is pretty simple.
+
+```ruby
+Gitlab::Experiment.configure do |config|
+  config.publishing_behavior = lambda do |_result|
+    # Push the experiment knowledge into the front end. The signature contains
+    # the context key, and the variant that has been determined.
+    Gon.push({ experiment: { name => signature } }, true)
+  end
+end
+```
+
+In the client you can now access `window.gon.experiment.notificationToggle`.
+
+### Caching
+
+Caching can be enabled in configuration, and is implemented towards the `Rails.cache` / `ActiveSupport::Cache::Store` interface. When you enable caching, any variant resolution will be cached. Migrating the cache through context migrations is handled automatically, and this helps ensure an experiment experience remains consistent.
+
+It's important to understand that using caching can drastically change or override your rollout strategy logic.
+
+```ruby
+Gitlab::Experiment.configure do |config|
+  config.cache = Rails.cache
+end
+```
 
 ## Tracking, anonymity and GDPR
 
-We intentionally don't, and shouldn't, track things like user ids. What we can and do track is what we consider an "experiment experience" key. This key is generated by the context we pass to the experiment implementation. If we consistently pass the same context to an experiment, we're able to consistently track events generated in that experience. A context can contain things like user, or project -- so, if you only included a user in the context that user would get the same experience across all projects they view, but if you include the currently viewed project in the context the user would potentially have a different experience on each of their projects. Each can be desirable given the objectives of the experiment.
+We generally try not to track things like user identifying values in our experimentation. What we can and do track is the "experiment experience" (a.k.a. the context key).
+
+We generate this key from the context passed to the experiment. This allows creating funnels without exposing any user information.
 
-## Code quality expectations
+This library attempts to be non-user-centric, in that a context can contain things like a user or a project.
 
-Since experimental code is inherently short lived, an intentionally stated goal is to iterate quickly to generate and evaluate performance data.
+If you only include a user, that user would get the same experience across every project they view. If you only include the project, every user who views that project would get the same experience.
 
-This goal prioritizes iteration and resolution over code quality, which means that experiment code may not always meet our code standards guidelines, but should also not negatively impact the availability of GitLab nor contribute to bad data. Even though experiments will be deployed to a minority of users, we still expect a flawless experience for those users, therefore, good test coverage is still required.
+Each of these approaches could be desirable given the objectives of your experiment.
 
-Reviewers and maintainers are encouraged to note when code doesn't meet our code standards guidelines. Please mention your concerns and include or link to them on the experiment tracking issue. The experiment author(s) are responsible for addressing these concerns when the experiment is resolved.
+### Make code not war

data/lib/generators/gitlab_experiment/install/install_generator.rb

@@ -0,0 +1,21 @@
+# frozen_string_literal: true
+
+require 'rails/generators'
+
+module GitlabExperiment
+  module Generators
+    class InstallGenerator < Rails::Generators::Base
+      source_root File.expand_path(__dir__)
+
+      desc 'Installs the Gitlab Experiment initializer into your application.'
+
+      def copy_initializers
+        copy_file 'templates/initializer.rb', 'config/initializers/gitlab_experiment.rb'
+      end
+
+      def display_post_install
+        readme 'POST_INSTALL' if behavior == :invoke
+      end
+    end
+  end
+end

data/lib/generators/gitlab_experiment/install/templates/initializer.rb

@@ -0,0 +1,87 @@
+# frozen_string_literal: true
+
+Gitlab::Experiment.configure do |config|
+  # Prefix all experiment names with a given value. Use `nil` for none.
+  config.name_prefix = nil
+
+  # The logger is used to log various details of the experiments.
+  config.logger = Logger.new(STDOUT)
+
+  # The caching layer is expected to respond to fetch, like Rails.cache.
+  config.cache = nil
+
+  # Logic this project uses to resolve a variant for a given experiment.
+  #
+  # This can return an instance of any object that responds to `name`, or can
+  # return a variant name as a string, in which case the build in variant
+  # class will be used.
+  #
+  # This block will be executed within the scope of the experiment instance,
+  # so can easily access experiment methods, like getting the name or context.
+  config.variant_resolver = lambda do |requested_variant|
+    # Run the control, unless a variant was requested in code:
+    requested_variant || 'control'
+
+    # Run the candidate, unless a variant was requested, with a fallback:
+    #
+    # requested_variant || variant_names.first || 'control'
+
+    # Using Unleash to determine the variant:
+    #
+    # fallback = Unleash::Variant.new(name: requested_variant || 'control', enabled: true)
+    # Unleash.get_variant(name, context.value, fallback)
+
+    # Using Flipper to determine the variant:
+    #
+    # TODO: provide example.
+    # Variant.new(name: requested_variant || 'control')
+  end
+
+  # Tracking behavior can be implemented to link an event to an experiment.
+  #
+  # Similar to the variant_resolver, this is called within the scope of the
+  # experiment instance and so can access any methods on the experiment,
+  # such as name and signature.
+  config.tracking_behavior = lambda do |event, args|
+    # An example of using a generic logger to track events:
+    config.logger.info "Gitlab::Experiment[#{name}] #{event}: #{args.merge(signature: signature)}"
+
+    # Using something like snowplow to track events (in gitlab):
+    #
+    # Gitlab::Tracking.event(name, event, **args.merge(
+    #   context: (args[:context] || []) << SnowplowTracker::SelfDescribingJson.new(
+    #     'iglu:com.gitlab/gitlab_experiment/jsonschema/0-2-0', signature
+    #   )
+    # ))
+  end
+
+  # Called at the end of every experiment run, with the result.
+  #
+  # You may want to track that you've assigned a variant to a given context,
+  # or push the experiment into the client or publish results elsewhere, like
+  # into redis. Also called within the scope of the experiment instance.
+  config.publishing_behavior = lambda do |result|
+    # Track the event using our own configured tracking logic.
+    track(:assignment)
+
+    # Push the experiment knowledge into the front end. The signature contains
+    # the context key, and the variant that has been determined.
+    #
+    # Gon.push({ experiment: { name => signature } }, true)
+
+    # Log using our logging system, so the result (which can be large) can be
+    # reviewed later if we want to.
+    #
+    # Lograge::Event.log(experiment: name, result: result, signature: signature)
+  end
+
+  # Algorithm that consistently generates a hash key for a given hash map.
+  #
+  # Given a specific context hash map, we need to generate a consistent hash
+  # key. The logic in here will be used for generating cache keys, and may also
+  # be used when determining which variant may be presented.
+  config.context_hash_strategy = lambda do |context|
+    values = context.values.map { |v| (v.respond_to?(:to_global_id) ? v.to_global_id : v).to_s }
+    Digest::MD5.hexdigest((context.keys + values).join('|'))
+  end
+end

data/lib/gitlab/experiment.rb

@@ -2,15 +2,19 @@
 
 require 'scientist'
 
+require 'gitlab/experiment/caching'
 require 'gitlab/experiment/configuration'
+require 'gitlab/experiment/cookies'
 require 'gitlab/experiment/context'
 require 'gitlab/experiment/dsl'
 require 'gitlab/experiment/variant'
 require 'gitlab/experiment/version'
+require 'gitlab/experiment/engine' if defined?(Rails::Engine)
 
 module Gitlab
   class Experiment
     include Scientist::Experiment
+    include Caching
 
     class << self
       def configure
@@ -28,11 +32,10 @@ module Gitlab
     def initialize(name, variant_name = nil, **context)
       @name = name
       @variant_name = variant_name
-      @context = Context.new(self)
+      @excluded = []
+      @context = Context.new(self, context)
 
-      context(context)
-
-      ignore { true }
+      exclude { !@context.trackable? }
       compare { false }
 
       yield self if block_given?
@@ -46,19 +49,32 @@ module Gitlab
     end
 
     def variant(value = nil)
-      @variant_name = value unless value.nil?
-      instance_exec(@variant_name, &Configuration.variant_resolver)
+      return @variant_name = value unless value.nil?
+
+      result = instance_exec(@variant_name, &Configuration.variant_resolver)
+      result.respond_to?(:name) ? result : Variant.new(name: result.to_s)
+    end
+
+    def exclude(&block)
+      @excluded << block
     end
 
-    def run
-      @result ||= super(variant.name)
+    def run(variant_name = nil)
+      @result ||= begin
+        @variant_name = variant_name unless variant_name.nil?
+        @variant_name ||= :control if excluded?
+
+        super(cache { variant.name })
+      end
     end
 
-    def publish(_result)
-      track(:assignment)
+    def publish(result)
+      instance_exec(result, &Configuration.publishing_behavior)
     end
 
     def track(action, **event_args)
+      return if excluded?
+
       instance_exec(action, event_args, &Configuration.tracking_behavior)
     end
 
@@ -67,13 +83,21 @@ module Gitlab
     end
 
     def variant_names
-      @variant_names = behaviors.keys.tap { |keys| keys.delete('control') }.map(&:to_sym)
+      @variant_names ||= behaviors.keys.map(&:to_sym) - [:control]
+    end
+
+    def signature
+      { variant: variant.name, experiment: name }.merge(context.signature)
     end
 
     def enabled?
       true
     end
 
+    def excluded?
+      @excluded.any? { |exclude| exclude.call(self) }
+    end
+
     protected
 
     def generate_result(variant_name)

data/lib/gitlab/experiment/caching.rb

@@ -0,0 +1,33 @@
+# frozen_string_literal: true
+
+module Gitlab
+  class Experiment
+    module Caching
+      def cache(&block)
+        return yield unless (cache = Configuration.cache)
+
+        key, migrations = cache_strategy
+        migrated_cache(cache, migrations || [], key) or cache.fetch(key, &block)
+      end
+
+      private
+
+      def cache_strategy
+        [
+          "#{name}:#{signature[:key]}",
+          signature[:migration_keys]&.map { |key| "#{name}:#{key}" }
+        ]
+      end
+
+      def migrated_cache(cache, migrations, new_key)
+        migrations.find do |old_key|
+          next unless (value = cache.read(old_key))
+
+          cache.write(new_key, value)
+          cache.delete(old_key)
+          break value
+        end
+      end
+    end
+  end
+end

data/lib/gitlab/experiment/configuration.rb

@@ -10,41 +10,28 @@ module Gitlab
       include Singleton
 
       # Prefix all experiment names with a given value. Use `nil` for none.
-      @name_prefix = 'gitlab_experiment'
+      @name_prefix = nil
 
       # The logger is used to log various details of the experiments.
       @logger = Logger.new(STDOUT)
 
+      # Cache layer. Expected to respond to fetch, like Rails.cache.
+      @cache = nil
+
       # Logic this project uses to resolve a variant for a given experiment.
       @variant_resolver = lambda do |requested_variant|
-        # An example of running the control, unless a variant was requested:
-        Variant.new(name: requested_variant || 'control')
-
-        # Always run candidate, unless a variant was requested, with fallback:
-        # Variant.new(name: variant_name || variant_names.first || 'control')
-
-        # Using unleash to determine the variant:
-        # fallback = Unleash::Variant.new(name: 'control', enabled: true)
-        # Unleash.get_variant(name, context.value, fallback)
-
-        # Using Flipper to determine the variant:
-        # TODO: provide example
-        # Variant.new(name: resolved)
+        requested_variant || 'control'
       end
 
       # Tracking behavior can be implemented to link an event to an experiment.
-      @tracking_behavior = lambda do |action, event_args|
-        # An example of using a generic logger to track events:
-        (event_args[:context] ||= []) << context.signature.merge(group: variant.name)
-        Configuration.logger.info "EXPERIMENT[#{name}] #{action}: #{event_args}"
-
-        # Using snowplow to track events:
-        # (event_args[:context] ||= []) << SnowplowTracker::SelfDescribingJson.new(
-        #   'iglu:com.gitlab/gitlab_experiment/jsonschema/1-0-0',
-        #   experiment: context.signature.merge(group: variant.name)
-        # )
-        #
-        # Tracking.event(name, action, **event_args)
+      @tracking_behavior = lambda do |event, args|
+        Configuration.logger.info "Gitlab::Experiment[#{name}] #{event}: #{args.merge(signature: signature)}"
+      end
+
+      # Called at the end of every experiment run, with the results. You may
+      # want to push the experiment into the client or push results elsewhere.
+      @publishing_behavior = lambda do |_result|
+        track(:assignment)
       end
 
       # Algorithm that consistently generates a hash key for a given hash map.
@@ -54,8 +41,8 @@ module Gitlab
       end
 
       class << self
-        attr_accessor :name_prefix, :logger
-        attr_accessor :variant_resolver, :tracking_behavior, :context_hash_strategy
+        attr_accessor :name_prefix, :logger, :cache
+        attr_accessor :variant_resolver, :tracking_behavior, :publishing_behavior, :context_hash_strategy
       end
     end
   end

data/lib/gitlab/experiment/context.rb

@@ -3,26 +3,38 @@
 module Gitlab
   class Experiment
     class Context
-      def initialize(experiment)
+      include Cookies
+
+      DNT_REGEXP = /^(true|t|yes|y|1|on)$/i.freeze
+
+      def initialize(experiment, **initial_value)
         @experiment = experiment
         @value = {}
-        @migrations_from = []
-        @migrations_with = []
+        @migrations = { merged: [], unmerged: [] }
+
+        value(initial_value)
+      end
+
+      def reinitialize(request)
+        @signature = nil # clear memoization
+        @request = request if request.respond_to?(:headers) && request.respond_to?(:cookie_jar)
       end
 
       def value(value = nil)
         return @value if value.nil?
 
         value = value.dup # dup so we don't mutate
-        @signature = nil # clear memoized signature
+        reinitialize(value.delete(:request))
 
-        @migrations_from << value.delete(:migrated_from) if value[:migrated_from]
-        @migrations_with << value.delete(:migrated_with) if value[:migrated_with]
-        @value.merge!(migrate_cookie_to_user_id(value))
+        @value.merge!(process_migrations(value))
+      end
+
+      def trackable?
+        !(@request && @request.headers['DNT'].to_s.match?(DNT_REGEXP))
       end
 
       def freeze
-        signature # ensure we're done before being frozen
+        signature # finalize before freezing
         super
       end
 
@@ -32,34 +44,24 @@ module Gitlab
 
       private
 
-      def migrate_cookie_to_user_id(value)
-        return value unless (request = value.delete(:request))
-        return value if request.headers['DNT'].to_s.match?(/^(true|t|yes|y|1|on)$/i)
-
-        cookie_name = [@experiment.name, 'id'].join('_')
-        cookie_value = request.cookie_jar.signed[cookie_name]
-
-        if value[:user_id].blank? && cookie_value.blank?
-          request.cookie_jar.permanent.signed[cookie_name] = {
-            value: value[:user_id] = SecureRandom.uuid, secure: true, domain: :all, httponly: true
-          }
-        elsif cookie_value # we know via the cookie
-          if value[:user_id].blank?
-            value[:user_id] = cookie_value
-          else
-            @migrations_with << { user_id: cookie_value }
-            request.cookie_jar.delete(cookie_name, domain: :all)
-          end
-        end
-
-        value
+      def process_migrations(value)
+        add_migration(value.delete(:migrated_from))
+        add_migration(value.delete(:migrated_with), merge: true)
+
+        migrate_cookie(value, "#{@experiment.name}_id")
+      end
+
+      def add_migration(value, merge: false)
+        return unless value.is_a?(Hash)
+
+        @migrations[merge ? :merged : :unmerged] << value
       end
 
       def migration_keys
-        return nil if @migrations_from.empty? && @migrations_with.empty?
+        return nil if @migrations[:unmerged].empty? && @migrations[:merged].empty?
 
-        @migrations_from.map { |m| key_for(m) } +
-          @migrations_with.map { |m| key_for(@value.merge(m)) }
+        @migrations[:unmerged].map { |m| key_for(m) } +
+          @migrations[:merged].map { |m| key_for(@value.merge(m)) }
       end
 
       def key_for(context)

data/lib/gitlab/experiment/cookies.rb

@@ -0,0 +1,44 @@
+# frozen_string_literal: true
+
+require 'securerandom'
+
+module Gitlab
+  class Experiment
+    module Cookies
+      private
+
+      def migrate_cookie(hash, cookie_name)
+        return hash if cookie_jar.nil?
+
+        resolver = [hash, :actor, cookie_name, cookie_jar.signed[cookie_name]]
+        resolve_cookie(*resolver) or generate_cookie(*resolver)
+      end
+
+      def cookie_jar
+        @request&.cookie_jar
+      end
+
+      def resolve_cookie(hash, key, cookie_name, cookie)
+        return if cookie.to_s.empty? && hash[key].nil?
+        return hash if cookie.to_s.empty?
+        return hash.merge(key => cookie) if hash[key].nil?
+
+        add_migration(key => cookie)
+        cookie_jar.delete(cookie_name, domain: :all)
+
+        hash
+      end
+
+      def generate_cookie(hash, key, cookie_name, cookie)
+        return hash unless hash.key?(key)
+
+        cookie ||= SecureRandom.uuid
+        cookie_jar.permanent.signed[cookie_name] = {
+          value: cookie, secure: true, domain: :all, httponly: true
+        }
+
+        hash.merge(key => cookie)
+      end
+    end
+  end
+end

data/lib/gitlab/experiment/dsl.rb

@@ -4,6 +4,7 @@ module Gitlab
   class Experiment
     module Dsl
       def experiment(name, variant_name = nil, **context, &block)
+        context[:request] ||= request if respond_to?(:request)
         Experiment.run(name, variant_name, **context, &block)
       end
     end

data/lib/gitlab/experiment/engine.rb

@@ -0,0 +1,16 @@
+# frozen_string_literal: true
+
+module Gitlab
+  class Experiment
+    class Engine < ::Rails::Engine
+      def self.include_dsl
+        ActionController::Base.include(Dsl)
+        ActionController::Base.helper_method(:experiment)
+      end
+
+      config.after_initialize do
+        include_dsl if defined?(ActionController)
+      end
+    end
+  end
+end

data/lib/gitlab/experiment/version.rb

@@ -2,6 +2,6 @@
 
 module Gitlab
   class Experiment
-    VERSION = '0.2.0'
+    VERSION = '0.3.0'
   end
 end

metadata

@@ -1,35 +1,35 @@
 --- !ruby/object:Gem::Specification
 name: gitlab-experiment
 version: !ruby/object:Gem::Version
-  version: 0.2.0
+  version: 0.3.0
 platform: ruby
 authors:
 - GitLab
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2020-09-20 00:00:00.000000000 Z
+date: 2020-10-21 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: scientist
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: 1.5.0
     - - "~>"
       - !ruby/object:Gem::Version
         version: '1.5'
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 1.5.0
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: 1.5.0
     - - "~>"
       - !ruby/object:Gem::Version
         version: '1.5'
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 1.5.0
 description: 
 email:
 - gitlab_rubygems@gitlab.com
@@ -39,10 +39,16 @@ extra_rdoc_files: []
 files:
 - LICENSE.txt
 - README.md
+- lib/generators/gitlab_experiment/install/POST_INSTALL
+- lib/generators/gitlab_experiment/install/install_generator.rb
+- lib/generators/gitlab_experiment/install/templates/initializer.rb
 - lib/gitlab/experiment.rb
+- lib/gitlab/experiment/caching.rb
 - lib/gitlab/experiment/configuration.rb
 - lib/gitlab/experiment/context.rb
+- lib/gitlab/experiment/cookies.rb
 - lib/gitlab/experiment/dsl.rb
+- lib/gitlab/experiment/engine.rb
 - lib/gitlab/experiment/variant.rb
 - lib/gitlab/experiment/version.rb
 homepage: https://gitlab.com/gitlab-org/gitlab-experiment
@@ -64,7 +70,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.0.3
+rubygems_version: 3.1.4
 signing_key: 
 specification_version: 4
 summary: GitLab experiment library built on top of scientist.