gitlab-experiment 0.2.3 → 0.4.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 1a35b23202fa542fa548093c65ec4580e764fdbda2549291a54bfa3ed98cec96
-  data.tar.gz: 05f4c1ed8c5761ab03ed1e9d763d6e612f661cf9ee8736e46fb0e48b947ed52d
+  metadata.gz: 740552e72dc8655bde106bbca76a2b5be2776196ad957c780cf06d2bc3324a11
+  data.tar.gz: 19f8fbc4588b8efcff729464b5f3480a9d41a0429f8ea8153d097a0c9c948aae
 SHA512:
-  metadata.gz: 9505898d8870749dbccab8fa98974948895a9898838aecc622e9f3243624a889076b248f6c0698ad8d44c63726447cc1b46638158614bb008015a82ce42a8d19
-  data.tar.gz: 3484817693dbaf9254d16083c6fbb745fbe47adc7868bf924bd418d8f67735fd26ac00f7f486aecc4f9ec84647f58e7bfcb0ff15199370b2e782d68cf2a0b2a2
+  metadata.gz: b4f4b3f0a7c56087a0bccaf916d568a82ad02cd7784ba94e86254f3c11fabc106cf5da171d889d95818fb4229b34ab9712ddddf597c2dc6075d724c3c08af80b
+  data.tar.gz: aa6c559ca3e62e810ffa8efd7b1846da0beefd6ad91c10e425d7ed1477eb3073cb792b909d39199cb5847ce1b25d9bf19189793190f6e5a519c43468a341fe02

data/README.md

@@ -1,9 +1,10 @@
-GitLab Experiment
-=================
+# GitLab Experiment
+
+<img alt="experiment" src="/uploads/60990b2dbf4c0406bbf8b7f998de2dea/experiment.png" align="right" width="40%">
 
 Here at GitLab, we run experiments as A/B/n tests and review the data the experiment generates. From that data, we determine the best performing variant and promote it as the new default code path. Or revert back to the control if no variant outperformed it.
 
-This library provides a clean and elegant DSL to define, run, and track your GitLab experiment.
+This library provides a clean and elegant DSL (domain specific language) to define, run, and track your GitLab experiment.
 
 When we discuss the behavior of this gem, we'll use terms like experiment, context, control, candidate, and variant. It's worth defining these terms so they're more understood.
 
@@ -13,25 +14,7 @@ When we discuss the behavior of this gem, we'll use terms like experiment, conte
 - `candidate` defines that there's one experimental code path.
 - `variant(s)` is used when more than one experimental code path exists.
 
-Candidate and variant are the same concept, but simplify how we speak about experimental paths.
-
-## Opinionated elevator pitch
-
-The last time I travelled I went through a process at the airport when I arrived. My passport was checked, I was asked a series of questions about my visit, and was ushered to a digital kiosk where I was prompted with a few more questions on a touch screen. 
-
-It was Iceland, and for the record, it's a beautiful place and you should visit if you haven't yet.
-
-Anyway, at various stages of this process travellers are presented with a physical emoji button interface, and are encouraged to rate their satisfaction within the various stage.
-
-Running an experiment could be to change some aspect of the process for only some travelers, let's say by routing them down a different hallway. At various stages of both hallways we still have the emoji interfaces that can be happily tapped or angrily jabbed in their passing.
-
-After a while we can compare the results and can evaluate which hallway had the better overall experience, based on the ratings provided at the various stages.
-
-This library is about keeping track of which passports we send down which hallway, so we can consistently route them down the same hallway, and to know which hallway they're rating when they do.
-
-In this model we don’t need to know anything about the passport holder unless we decide to "ask", as we determine which hallway to send them down initially.
-
-This library doesn't provide a system of linking passports back to their passport holders, but it doesn't explicitly make doing so impossible. Doing so is often not a relevant detail on well defined and well executed experiments.
+Candidate and variant are the same concept, but simplify how we speak about experimental paths.<br clear="all">
 
 ## Installation
 
@@ -44,18 +27,18 @@ gem 'gitlab-experiment'
 If you're using Rails, you can install the initializer. It provides basic configuration and documentation.
 
 ```shell
-$ rails generate gitlab-experiment:install
+$ rails generate gitlab:experiment:install
 ```
 
 ## Implementing an experiment
 
 For the sake of an example let's make one up. Let's run an experiment on what we render for disabling desktop notifications.
 
-In our control (current world) we show a simple toggle interface that reads "Notifications". In our experiment we want a "Turn on/off desktop notifications" button with a confirmation.
+In our control (current world) we show a simple toggle interface labeled, "Notifications." In our experiment we want a "Turn on/off desktop notifications" button with a confirmation.
+
+The behavior will be the same, but the interface will be different and may involve more or fewer steps.
 
-The behavior will be the same, but the interface will be different and may involve more or less steps.
- 
-This makes the action more clear and will help the user in making a choice about if that's what they want to do. Or that's what we're going to try to find out.
+Our hypothesis is that this will make the action more clear and will help in making a choice about if that's what the user really wants to do.
 
 We'll name our experiment `notification_toggle`. This name is prefixed based on configuration. If you've set `config.name_prefix = 'gitlab'`, the experiment name would be `gitlab_notification_toggle` elsewhere.
 
@@ -63,12 +46,12 @@ When you implement an experiment you'll need to provide a name, and a context. T
 
 A context "key" represents the unique id of a context. It allows us to give the same experience between different calls to the experiment and can be used in caching.
 
-Now in our experiment we're going to render one of two views. Control will be our current view, and candidate will be the new toggle button with a confirmation flow.
- 
+Now in our experiment we're going to render one of two views: the control will be our current view, and the candidate will be the new toggle button with a confirmation flow.
+
 ```ruby
 class SubscriptionsController < ApplicationController
   def show
-    experiment(:notification_toggle, user_id: user.id) do |e|
+    experiment(:notification_toggle, actor: user) do |e|
       e.use { render_toggle } # control
       e.try { render_button } # candidate
     end
@@ -81,7 +64,7 @@ You can define the experiment using simple control/candidate paths, or provide n
 Handling multi-variant experiments is up to the configuration you provide around resolving variants. But in our example we may want to try with and without the confirmation. We can run any number of variations in our experiments this way.
 
 ```ruby
-experiment(:notification_toggle, user_id: user.id) do |e|
+experiment(:notification_toggle, actor: user) do |e|
   e.use { render_toggle } # control
   e.try(:variant_one) { render_button(confirmation: true) }
   e.try(:variant_two) { render_button(confirmation: false) }
@@ -93,7 +76,7 @@ Understanding how an experiment can change behavior is important in evaluating i
 To this end, we track events that are important by calling the same experiment elsewhere in code. By using the same context, you'll have consistent behavior and the ability to track events to it.
 
 ```ruby
-experiment(:notification_toggle, user_id: user.id).track(:clicked_button)
+experiment(:notification_toggle, actor: user).track(:clicked_button)
 ```
 
 <details>
@@ -102,10 +85,10 @@ experiment(:notification_toggle, user_id: user.id).track(:clicked_button)
 ### Class level interface using `.run`
 
 ```ruby
-exp = Gitlab::Experiment.run(:notification_toggle, user_id: user.id) do |e|
+exp = Gitlab::Experiment.run(:notification_toggle, actor: user) do |e|
   # Context may be passed in the block, but must be finalized before calling
   # run or track.
-  e.context(project_id: project.id) # add the project id to the context
+  e.context(project: project) # add the project to the context
 
   # Define the control and candidate variant.
   e.use { render_toggle } # control
@@ -119,16 +102,16 @@ exp.track(:clicked_button)
 ### Instance level interface
 
 ```ruby
-exp = Gitlab::Experiment.new(:notification_toggle, user_id: user.id)
-# Context may be passed in the block, but must be finalized before calling
-# run or track.
-exp.context(project_id: project.id) # add the project id to the context
+exp = Gitlab::Experiment.new(:notification_toggle, actor: user)
+# Additional context may be provided to the instance (exp) but must be
+# finalized before calling run or track.
+exp.context(project: project) # add the project id to the context
 
 # Define the control and candidate variant.
 exp.use { render_toggle } # control
 exp.try { render_button } # candidate
 
-# Run the experiment -- returning the result.
+# Run the experiment, returning the result.
 exp.run
 
 # Track an event on the experiment we've defined.
@@ -153,10 +136,10 @@ class NotificationExperiment < Gitlab::Experiment
   end
 end
 
-exp = NotificationExperiment.new(user_id: user.id) do |e|
-  # Context may be passed in the block, but must be finalized before calling
-  # run or track.
-  e.context(project_id: project.id) # add the project id to the context
+exp = NotificationExperiment.new(actor: user) do |e|
+  # Context may be provided within the block or to the instance (exp) but must
+  # be finalized before calling run or track.
+  e.context(project: project) # add the project id to the context
 end
 
 # Run the experiment -- returning the result.
@@ -176,7 +159,7 @@ exp.track(:clicked_button)
 You can hardcode the variant if you want. It's important to know what this might do to your data during rollout, so use this with consideration.
 
 ```ruby
-experiment(:notification_toggle, :no_interface, user_id: user.id) do |e|
+experiment(:notification_toggle, :no_interface, actor: user) do |e|
   e.use { render_toggle } # control
   e.try { render_button } # candidate
   e.try(:no_interface) { no_interface! } # variant
@@ -186,12 +169,23 @@ end
 Or you can set the variant within the block. This allows using unique segmentation logic or variant resolution if you need it.
 
 ```ruby
-experiment(:notification_toggle, user_id: user.id) do |e|
+experiment(:notification_toggle, actor: user) do |e|
+  # Variant selection must be done before calling run or track.
   e.variant(:no_interface) # set the variant
   # ...
 end
 ```
 
+Or it can be specified in the call to run if you call it from within the block.
+
+```ruby
+experiment(:notification_toggle, actor: user) do |e|
+  # ...
+  # Variant selection can be specified when calling run.
+  e.run(:no_interface)
+end
+```
+
 </details>
 
 ### Return value
@@ -215,13 +209,13 @@ Some experiments may extend outside of those layers, so you may want to include
 Note: In a lot of these contexts you may not have a reference to the request (unless you pass it in, or provide access to it) which may be needed if you want to enable cookie behaviors and track that through to user conversion.
 
 ```ruby
-class UserMailer < ApplicationMailer
+class WelcomeMailer < ApplicationMailer
   include Gitlab::Experiment::Dsl # include the `experiment` method
 
   def welcome
     @user = params[:user]
 
-    ex = experiment(:project_suggestions, user_id: @user.id) do |e|
+    ex = experiment(:project_suggestions, actor: @user) do |e|
       e.use { 'welcome' }
       e.try { 'welcome_with_project_suggestions' }
     end
@@ -235,57 +229,72 @@ end
 
 There are times when we need to change context while an experiment is running. We make this possible by passing the migration data to the experiment.
 
-Take for instance, that you might be using `version: 1` in your context currently. To migrate this `version: 2`, provide the context to change using a `migrated_with` option.
+Take for instance, that you might be using `version: 1` in your context currently. To migrate this to `version: 2`, provide the portion of the context you wish to change using a `migrated_with` option.
 
 In providing the context migration data, we can resolve an experience and its events all the way back. This can also help in keeping our cache relevant.
 
 ```ruby
-experiment(:my_experiment, user_id: 42, version: 2, migrated_with: { version: 1 })
+# Migrate just the `:version` portion of the previous context, `{ actor: project, version: 1 }`:
+experiment(:my_experiment, actor: project, version: 2, migrated_with: { version: 1 })
 ```
 
-You can add or remove context by providing a `migrated_from` option. This approach expects a full context replacement -- e.g. what it was before you added or removed the new context key.
+You can add or remove context by providing a `migrated_from` option. This approach expects a full context replacement -- i.e. what it was before you added or removed the new context key.
 
 If you wanted to introduce a `version` to your context, provide the full previous context.
- 
+
 ```ruby
-experiment(:my_experiment, user_id: 42, version: 1, migrated_from: { user_id: 42 })
+# Migrate the full context from `{ actor: project }` to `{ actor: project, version: 1 }`:
+experiment(:my_experiment, actor: project, version: 1, migrated_from: { actor: project })
 ```
 
-This can impact an experience if you haven't:
+This can impact an experience if you:
 
-1. implemented the concept of migrations in your variant resolver
+1. haven't implemented the concept of migrations in your variant resolver
 1. haven't enabled a reasonable caching mechanism
 
-### When there isn't a user (cookies)
+### When there isn't an actor (cookie fallback)
 
-When there isn't an identifying key in the context (this is `user_id` by default), we fall back to cookies to provide a consistent experience.
+When there isn't an identifying key in the context (this is `actor` by default), we fall back to cookies to provide a consistent experience for the client viewing them.
 
-Once we assign a certain variant to a context, we need to always provide the same experience. We achieve this by setting a cookie for the experiment in question.
+Once we assign a certain variant to a context, we need to always provide the same experience. We achieve this by setting a cookie for the experiment in question, but only when needed.
 
-This cookie is a randomized uuid and isn't associated with the user. When we can finally provide an identifying key, the context is auto migrated from the cookie to that identifying key. The cookie is a temporary value, and isn't used for tracking.
+This cookie is a temporary, randomized uuid and isn't associated with a user. When we can finally provide an actor, the context is auto migrated from the cookie to that actor.
 
 To read and write cookies, we provide the `request` from within the controller and views. The cookie migration will happen automatically if the experiment is within those layers.
 
 You'll need to provide the `request` as an option to the experiment if it's outside of the controller and views.
 
 ```ruby
-experiment(:my_experiment, user_id: user&.id, request: request)
+experiment(:my_experiment, actor: user, request: request)
 ```
 
-The cookie isn't set if the identifying key isn't present in the context. In this case, if there was no `user_id` key provided, the cookie wouldn't be set.
+The cookie isn't set if the `actor` key isn't present at all in the context. Meaning that when no `actor` key is provided, the cookie will not be set.
+
+```ruby
+# actor is not present, so no cookie is set
+experiment(:my_experiment, project: project)
 
-For edge cases, you can pass the cookie through by assigning it yourself -- e.g. `user_id: request.cookie_jar.signed['my_experiment_id']`. The cookie name is the experiment name (prefixed if configured) with `_id` appended.
+# actor is present and is nil, so the cookie is set and used
+experiment(:my_experiment, actor: nil, project: project)
+
+# actor is present and set to a value, so no cookie is set
+experiment(:my_experiment, actor: user, project: project)
+```
+
+For edge cases, you can pass the cookie through by assigning it yourself -- e.g. `actor: request.cookie_jar.signed['my_experiment_actor']`. The cookie name is the full experiment name (including any configured prefix) with `_actor` appended -- e.g. `gitlab_notification_toggle_actor` for the `:notification_toggle` experiment key with a configured prefix of `gitlab`.
 
 ## Configuration
 
 This gem needs to be configured before being used in a meaningful way.
 
-The default configuration will always render the control. So it's important to configure your own logic for resolving variants.
+The default configuration will always render the control, so it's important to configure your own logic for resolving variants.
 
-Yes, the most important aspect of the gem, determining which variant to render and when, is up to you. Consider using [Unleash](https://github.com/Unleash/unleash-client-ruby) or [Flipper](https://github.com/jnunemaker/flipper) for this.
+Yes, the most important aspect of the gem -- that of determining which variant to render and when -- is up to you. Consider using [Unleash](https://github.com/Unleash/unleash-client-ruby) or [Flipper](https://github.com/jnunemaker/flipper) for this.
 
 ```ruby
 Gitlab::Experiment.configure do |config|
+  # The block here is evaluated within the scope of the experiment instance,
+  # which is why we are able to access things like name and context.
   config.variant_resolver = lambda do |requested_variant|
     # Return the requested variant if a specific one has been provided in code.
     return requested_variant unless requested_variant.nil?
@@ -298,20 +307,20 @@ Gitlab::Experiment.configure do |config|
 end
 ```
 
-More examples for configuration are available in the provided [rails initializer](lib/generators/gitlab_experiment/install/templates/initializer.rb).
+More examples for configuration are available in the provided [rails initializer](lib/generators/gitlab/experiment/install/templates/initializer.rb).
 
 ### Client layer / JavaScript
 
 This library doesn't attempt to provide any logic for the client layer.
 
-Instead it allows you to do this yourself in configuration. Using Gon to publish your experiment information to the client layer is pretty simple.
+Instead it allows you to do this yourself in configuration. Using [Gon](https://github.com/gazay/gon) to publish your experiment information to the client layer is pretty simple.
 
 ```ruby
 Gitlab::Experiment.configure do |config|
   config.publishing_behavior = lambda do |_result|
     # Push the experiment knowledge into the front end. The signature contains
     # the context key, and the variant that has been determined.
-    Gon.push(experiment: { name => signature })
+    Gon.push({ experiment: { name => signature } }, true)
   end
 end
 ```
@@ -322,6 +331,8 @@ In the client you can now access `window.gon.experiment.notificationToggle`.
 
 Caching can be enabled in configuration, and is implemented towards the `Rails.cache` / `ActiveSupport::Cache::Store` interface. When you enable caching, any variant resolution will be cached. Migrating the cache through context migrations is handled automatically, and this helps ensure an experiment experience remains consistent.
 
+It's important to understand that using caching can drastically change or override your rollout strategy logic.
+
 ```ruby
 Gitlab::Experiment.configure do |config|
   config.cache = Rails.cache
@@ -330,11 +341,11 @@ end
 
 ## Tracking, anonymity and GDPR
 
-We generally try not to track things like user identifying values in our experimentation. What we can and do track is the "experiment experience" (a.k.a the context key).
+We generally try not to track things like user identifying values in our experimentation. What we can and do track is the "experiment experience" (a.k.a. the context key).
 
 We generate this key from the context passed to the experiment. This allows creating funnels without exposing any user information.
 
-This library attempts to be non user centric, in that a context can contain things like a user, or a project.
+This library attempts to be non-user-centric, in that a context can contain things like a user or a project.
 
 If you only include a user, that user would get the same experience across every project they view. If you only include the project, every user who views that project would get the same experience.
 

data/lib/generators/gitlab/experiment/USAGE

@@ -0,0 +1,17 @@
+Description:
+    Stubs out a new experiment and its variants. Pass the experiment name,
+    either CamelCased or under_scored, and a list of variants as arguments.
+
+    To create an experiment within a module, specify the experiment name as a
+    path like 'parent_module/experiment_name'.
+
+    This generates an experiment class in app/experiments and invokes feature
+    flag, and test framework generators.
+
+Example:
+    `rails generate gitlab:experiment NullHypothesis control candidate alt_variant`
+
+    NullHypothesis experiment with default variants.
+        Experiment:   app/experiments/null_hypothesis_experiment.rb
+        Feature Flag: config/feature_flags/experiment/null_hypothesis.yaml
+        Test:         test/experiments/null_hypothesis_experiment_test.rb

data/lib/generators/gitlab/experiment/experiment_generator.rb

@@ -0,0 +1,33 @@
+# frozen_string_literal: true
+
+require 'rails/generators'
+
+module Gitlab
+  module Generators
+    class ExperimentGenerator < Rails::Generators::NamedBase
+      source_root File.expand_path('templates/', __dir__)
+      check_class_collision suffix: 'Experiment'
+
+      argument :variants,
+               type: :array,
+               default: %w[control candidate],
+               banner: 'variant variant'
+
+      def create_experiment
+        template 'experiment.rb', File.join('app/experiments', class_path, "#{file_name}_experiment.rb")
+      end
+
+      hook_for :test_framework
+
+      private
+
+      def file_name
+        @_file_name ||= remove_possible_suffix(super)
+      end
+
+      def remove_possible_suffix(name)
+        name.sub(/_?exp[ei]riment$/i, "") # be somewhat forgiving with spelling
+      end
+    end
+  end
+end

data/lib/generators/gitlab/experiment/install/install_generator.rb

@@ -0,0 +1,41 @@
+# frozen_string_literal: true
+
+require 'rails/generators'
+
+module Gitlab
+  module Generators
+    module Experiment
+      class InstallGenerator < Rails::Generators::Base
+        source_root File.expand_path('templates', __dir__)
+
+        desc 'Installs the Gitlab::Experiment initializer and optional ApplicationExperiment into your application.'
+
+        class_option :skip_initializer,
+                     type: :boolean,
+                     default: false,
+                     desc: 'Skip the initializer with default configuration'
+
+        class_option :skip_baseclass,
+                     type: :boolean,
+                     default: false,
+                     desc: 'Skip the ApplicationExperiment base class'
+
+        def create_initializer
+          return if options[:skip_initializer]
+
+          template 'initializer.rb', 'config/initializers/gitlab_experiment.rb'
+        end
+
+        def create_baseclass
+          return if options[:skip_baseclass]
+
+          template 'application_experiment.rb', 'app/experiments/application_experiment.rb'
+        end
+
+        def display_post_install
+          readme 'POST_INSTALL' if behavior == :invoke
+        end
+      end
+    end
+  end
+end

data/lib/generators/gitlab/experiment/install/templates/POST_INSTALL

@@ -0,0 +1,2 @@
+Gitlab::Experiment has been installed. You may want to adjust the configuration
+that's been provided in the Rails initializer.

data/lib/generators/gitlab/experiment/install/templates/application_experiment.rb.tt

@@ -0,0 +1,4 @@
+# frozen_string_literal: true
+
+class ApplicationExperiment < Gitlab::Experiment
+end

data/lib/generators/{gitlab_experiment/install/templates/initializer.rb → gitlab/experiment/install/templates/initializer.rb.tt}

@@ -5,7 +5,10 @@ Gitlab::Experiment.configure do |config|
   config.name_prefix = nil
 
   # The logger is used to log various details of the experiments.
-  config.logger = Logger.new(STDOUT)
+  config.logger = Logger.new($stdout)
+
+  # The base class that should be instantiated for basic experiments.
+  config.base_class = 'ApplicationExperiment'
 
   # The caching layer is expected to respond to fetch, like Rails.cache.
   config.cache = nil
@@ -26,7 +29,7 @@ Gitlab::Experiment.configure do |config|
     #
     # requested_variant || variant_names.first || 'control'
 
-    # Using unleash to determine the variant:
+    # Using Unleash to determine the variant:
     #
     # fallback = Unleash::Variant.new(name: requested_variant || 'control', enabled: true)
     # Unleash.get_variant(name, context.value, fallback)
@@ -40,19 +43,17 @@ Gitlab::Experiment.configure do |config|
   # Tracking behavior can be implemented to link an event to an experiment.
   #
   # Similar to the variant_resolver, this is called within the scope of the
-  # experiment instance and so can access any methods on the experiment.
-  #
-  #
+  # experiment instance and so can access any methods on the experiment,
+  # such as name and signature.
   config.tracking_behavior = lambda do |event, args|
     # An example of using a generic logger to track events:
     config.logger.info "Gitlab::Experiment[#{name}] #{event}: #{args.merge(signature: signature)}"
 
-    # Using something like snowplow to track events:
+    # Using something like snowplow to track events (in gitlab):
     #
     # Gitlab::Tracking.event(name, event, **args.merge(
     #   context: (args[:context] || []) << SnowplowTracker::SelfDescribingJson.new(
-    #     'iglu:com.gitlab/gitlab_experiment/jsonschema/0-2-0',
-    #     signature: signature
+    #     'iglu:com.gitlab/gitlab_experiment/jsonschema/0-2-0', signature
     #   )
     # ))
   end
@@ -69,7 +70,7 @@ Gitlab::Experiment.configure do |config|
     # Push the experiment knowledge into the front end. The signature contains
     # the context key, and the variant that has been determined.
     #
-    # Gon.push(experiment: { name => signature })
+    # Gon.push({ experiment: { name => signature } }, true)
 
     # Log using our logging system, so the result (which can be large) can be
     # reviewed later if we want to.
@@ -86,4 +87,10 @@ Gitlab::Experiment.configure do |config|
     values = context.values.map { |v| (v.respond_to?(:to_global_id) ? v.to_global_id : v).to_s }
     Digest::MD5.hexdigest((context.keys + values).join('|'))
   end
+
+  # The domain for which this cookie applies so you can restrict to the domain level.
+  #
+  # When not set, it uses the current host. If you want to provide specific hosts, you can
+  # provide them either via an array like `['www.gitlab.com', .gitlab.com']`, or set it to `:all`.
+  config.cookie_domain = :all
 end

data/lib/generators/gitlab/experiment/templates/experiment.rb.tt

@@ -0,0 +1,15 @@
+# frozen_string_literal: true
+
+<% if namespaced? -%>
+require_dependency "<%= namespaced_path %>/application_experiment"
+
+<% end -%>
+<% module_namespacing do -%>
+class <%= class_name %>Experiment < ApplicationExperiment
+<% variants.each do |variant| -%>
+  def <%= variant %>_behavior
+  end
+<%= "\n" unless variant == variants.last -%>
+<% end -%>
+end
+<% end -%>

data/lib/generators/rspec/experiment/experiment_generator.rb

@@ -0,0 +1,15 @@
+# frozen_string_literal: true
+
+require 'generators/rspec'
+
+module Rspec
+  module Generators
+    class ExperimentGenerator < Rspec::Generators::Base
+      source_root File.expand_path('templates/', __dir__)
+
+      def create_experiment_spec
+        template 'experiment_spec.rb', File.join('spec/experiments', class_path, "#{file_name}_experiment_spec.rb")
+      end
+    end
+  end
+end

data/lib/generators/rspec/experiment/templates/experiment_spec.rb.tt

@@ -0,0 +1,9 @@
+# frozen_string_literal: true
+
+require 'rails_helper'
+
+<% module_namespacing do -%>
+RSpec.describe <%= class_name %>Experiment do
+  pending "add some examples to (or delete) #{__FILE__}"
+end
+<% end -%>

data/lib/generators/test_unit/experiment/experiment_generator.rb

@@ -0,0 +1,17 @@
+# frozen_string_literal: true
+
+require 'rails/generators/test_unit'
+
+module TestUnit # :nodoc:
+  module Generators # :nodoc:
+    class ExperimentGenerator < TestUnit::Generators::Base # :nodoc:
+      source_root File.expand_path('templates/', __dir__)
+
+      check_class_collision suffix: 'Test'
+
+      def create_test_file
+        template 'experiment_test.rb', File.join('test/experiments', class_path, "#{file_name}_experiment_test.rb")
+      end
+    end
+  end
+end

data/lib/generators/test_unit/experiment/templates/experiment_test.rb.tt

@@ -0,0 +1,11 @@
+# frozen_string_literal: true
+
+require 'test_helper'
+
+<% module_namespacing do -%>
+class <%= class_name %>ExperimentTest < ActiveSupport::TestCase
+  # test "the truth" do
+  #   assert true
+  # end
+end
+<% end -%>

data/lib/gitlab/experiment.rb

@@ -1,9 +1,14 @@
 # frozen_string_literal: true
 
 require 'scientist'
+require 'active_support/callbacks'
+require 'active_support/core_ext/object/blank'
+require 'active_support/core_ext/string/inflections'
 
 require 'gitlab/experiment/caching'
+require 'gitlab/experiment/callbacks'
 require 'gitlab/experiment/configuration'
+require 'gitlab/experiment/cookies'
 require 'gitlab/experiment/context'
 require 'gitlab/experiment/dsl'
 require 'gitlab/experiment/variant'
@@ -14,37 +19,57 @@ module Gitlab
   class Experiment
     include Scientist::Experiment
     include Caching
+    include Callbacks
 
     class << self
       def configure
         yield Configuration
       end
 
-      def run(name, variant_name = nil, **context, &block)
-        instance = new(name, variant_name, **context, &block)
+      def run(name = nil, variant_name = nil, **context, &block)
+        raise ArgumentError, 'name is required' if name.nil? && base?
+
+        instance = constantize(name).new(name, variant_name, **context, &block)
         return instance unless block_given?
 
         instance.context.frozen? ? instance.run : instance.tap(&:run)
       end
+
+      def experiment_name(name = nil, suffix: true, suffix_word: 'experiment')
+        name = (name.presence || self.name).to_s.underscore.sub(%r{(?<char>[_/]|)#{suffix_word}$}, '')
+        name = "#{name}#{Regexp.last_match(:char) || '_'}#{suffix_word}"
+        suffix ? name : name.sub(/_#{suffix_word}$/, '')
+      end
+
+      def base?
+        self == Gitlab::Experiment || name == Configuration.base_class
+      end
+
+      private
+
+      def constantize(name = nil)
+        return self if name.nil?
+
+        experiment_name(name).classify.safe_constantize || Configuration.base_class.constantize
+      end
     end
 
-    delegate :signature, :cache_strategy, to: :context
+    def initialize(name = nil, variant_name = nil, **context)
+      raise ArgumentError, 'name is required' if name.blank? && self.class.base?
 
-    def initialize(name, variant_name = nil, **context)
-      @name = name
+      @name = self.class.experiment_name(name, suffix: false)
       @variant_name = variant_name
-      @context = Context.new(self)
-
-      context(context)
+      @excluded = []
+      @context = Context.new(self, context)
 
-      ignore { true }
+      exclude { !@context.trackable? }
       compare { false }
 
       yield self if block_given?
     end
 
     def context(value = nil)
-      return @context if value.nil?
+      return @context if value.blank?
 
       @context.value(value)
       @context
@@ -54,11 +79,30 @@ module Gitlab
       return @variant_name = value unless value.nil?
 
       result = instance_exec(@variant_name, &Configuration.variant_resolver)
-      result.respond_to?(:name) ? result : Variant.new(name: result.to_s)
+      result.respond_to?(:name) ? result : Variant.new(name: (result.presence || :control).to_s)
     end
 
-    def run
-      @result ||= super(cache { variant.name })
+    def exclude(&block)
+      @excluded << block
+    end
+
+    def run(variant_name = nil)
+      @result ||= begin
+        @variant_name = variant_name unless variant_name.nil?
+        @variant_name ||= :control if excluded?
+
+        chain = variant_assigned? ? :unsegmented_run : :segmented_run
+        run_callbacks(chain) do
+          variant_name = cache { variant.name }
+
+          method_name = "#{variant_name}_behavior"
+          if respond_to?(method_name)
+            behaviors[variant_name] ||= -> { send(method_name) } # rubocop:disable GitlabSecurity/PublicSend
+          end
+
+          super(variant_name)
+        end
+      end
     end
 
     def publish(result)
@@ -66,6 +110,8 @@ module Gitlab
     end
 
     def track(action, **event_args)
+      return if excluded?
+
       instance_exec(action, event_args, &Configuration.tracking_behavior)
     end
 
@@ -74,19 +120,36 @@ module Gitlab
     end
 
     def variant_names
-      @variant_names ||= behaviors.keys.tap { |keys| keys.delete('control') }.map(&:to_sym)
+      @variant_names ||= behaviors.keys.map(&:to_sym) - [:control]
+    end
+
+    def signature
+      { variant: variant.name, experiment: name }.merge(context.signature)
     end
 
     def enabled?
       true
     end
 
-    def identifying_key
-      :user_id
+    def excluded?
+      @excluded.any? { |exclude| exclude.call(self) }
+    end
+
+    def variant_assigned?
+      !@variant_name.nil?
+    end
+
+    def id
+      "#{name}:#{signature[:key]}"
+    end
+    alias_method :session_id, :id
+
+    def flipper_id
+      "Experiment;#{id}"
     end
 
-    def cache_key_for(key, migration: false)
-      "#{name}:#{key}"
+    def key_for(hash)
+      instance_exec(hash, &Configuration.context_hash_strategy)
     end
 
     protected

data/lib/gitlab/experiment/caching.rb

@@ -10,6 +10,15 @@ module Gitlab
         migrated_cache(cache, migrations || [], key) or cache.fetch(key, &block)
       end
 
+      private
+
+      def cache_strategy
+        [
+          "#{name}:#{signature[:key]}",
+          signature[:migration_keys]&.map { |key| "#{name}:#{key}" }
+        ]
+      end
+
       def migrated_cache(cache, migrations, new_key)
         migrations.find do |old_key|
           next unless (value = cache.read(old_key))

data/lib/gitlab/experiment/callbacks.rb

@@ -0,0 +1,39 @@
+# frozen_string_literal: true
+
+module Gitlab
+  class Experiment
+    module Callbacks
+      extend ActiveSupport::Concern
+      include ActiveSupport::Callbacks
+
+      included do
+        define_callbacks(
+          :unsegmented_run,
+          skip_after_callbacks_if_terminated: true
+        )
+
+        define_callbacks(
+          :segmented_run,
+          skip_after_callbacks_if_terminated: false,
+          terminator: lambda do |target, result_lambda|
+            result_lambda.call
+            target.variant_assigned?
+          end
+        )
+      end
+
+      class_methods do
+        def segment(*filter_list, variant:, **options, &block)
+          filters = filter_list.unshift(block).compact.map do |filter|
+            result_lambda = ActiveSupport::Callbacks::CallTemplate.build(filter, self).make_lambda
+            ->(target) { target.variant(variant) if result_lambda.call(target, nil) }
+          end
+
+          raise ArgumentError, 'no filters provided' if filters.empty?
+
+          set_callback(:segmented_run, :before, *filters, options, &block)
+        end
+      end
+    end
+  end
+end

data/lib/gitlab/experiment/configuration.rb

@@ -13,7 +13,10 @@ module Gitlab
       @name_prefix = nil
 
       # The logger is used to log various details of the experiments.
-      @logger = Logger.new(STDOUT)
+      @logger = Logger.new($stdout)
+
+      # The base class that should be instantiated for basic experiments.
+      @base_class = 'Gitlab::Experiment'
 
       # Cache layer. Expected to respond to fetch, like Rails.cache.
       @cache = nil
@@ -35,14 +38,28 @@ module Gitlab
       end
 
       # Algorithm that consistently generates a hash key for a given hash map.
-      @context_hash_strategy = lambda do |context|
-        values = context.values.map { |v| (v.respond_to?(:to_global_id) ? v.to_global_id : v).to_s }
-        Digest::MD5.hexdigest((context.keys + values).join('|'))
+      @context_hash_strategy = lambda do |hash_map|
+        values = hash_map.values.map { |v| (v.respond_to?(:to_global_id) ? v.to_global_id : v).to_s }
+        Digest::MD5.hexdigest(([name] + hash_map.keys + values).join('|'))
       end
 
+      # The domain for which this cookie applies so you can restrict to the domain level.
+      # When not set, it uses the current host. If you want to provide specific hosts, you can
+      # provide them either via an array like `['www.gitlab.com', .gitlab.com']`, or set it to `:all`.
+      @cookie_domain = :all
+
       class << self
-        attr_accessor :name_prefix, :logger, :cache
-        attr_accessor :variant_resolver, :tracking_behavior, :publishing_behavior, :context_hash_strategy
+        attr_accessor(
+          :name_prefix,
+          :logger,
+          :base_class,
+          :cache,
+          :variant_resolver,
+          :tracking_behavior,
+          :publishing_behavior,
+          :context_hash_strategy,
+          :cookie_domain
+        )
       end
     end
   end

data/lib/gitlab/experiment/context.rb

@@ -3,95 +3,65 @@
 module Gitlab
   class Experiment
     class Context
+      include Cookies
+
       DNT_REGEXP = /^(true|t|yes|y|1|on)$/i.freeze
 
-      def initialize(experiment)
+      def initialize(experiment, **initial_value)
         @experiment = experiment
         @value = {}
-        @migrations_from = []
-        @migrations_with = []
+        @migrations = { merged: [], unmerged: [] }
+
+        value(initial_value)
+      end
+
+      def reinitialize(request)
+        @signature = nil # clear memoization
+        @request = request if request.respond_to?(:headers) && request.respond_to?(:cookie_jar)
       end
 
       def value(value = nil)
         return @value if value.nil?
 
         value = value.dup # dup so we don't mutate
-        @signature = @cache_strategy = nil # clear memoization
+        reinitialize(value.delete(:request))
 
-        @migrations_from << value.delete(:migrated_from) if value[:migrated_from]
-        @migrations_with << value.delete(:migrated_with) if value[:migrated_with]
-        @value.merge!(auto_migrate_cookie(value, value.delete(:request)))
+        @value.merge!(process_migrations(value))
       end
 
-      def freeze
-        cache_strategy # ensure we memoize before freezing
-        super
+      def trackable?
+        !(@request && @request.headers['DNT'].to_s.match?(DNT_REGEXP))
       end
 
-      def cache_strategy
-        @cache_strategy ||= [
-          @experiment.cache_key_for(signature[:key]),
-          signature[:migration_keys]&.map do |key|
-            @experiment.cache_key_for(key, migration: true)
-          end
-        ]
+      def freeze
+        signature # finalize before freezing
+        super
       end
 
       def signature
-        @signature ||= {
-          key: key_for(@value),
-          migration_keys: migration_keys,
-          variant: @experiment.variant.name
-        }.compact
+        @signature ||= { key: @experiment.key_for(@value), migration_keys: migration_keys }.compact
       end
 
       private
 
-      def auto_migrate_cookie(hash, request)
-        return hash unless request&.respond_to?(:headers) && request&.respond_to?(:cookie_jar)
-        return hash if request.headers['DNT'].to_s.match?(DNT_REGEXP)
+      def process_migrations(value)
+        add_migration(value.delete(:migrated_from))
+        add_migration(value.delete(:migrated_with), merge: true)
 
-        resolver = cookie_resolver(request.cookie_jar, hash)
-        resolve_cookie(*resolver) or generate_cookie(*resolver)
+        migrate_cookie(value, "#{@experiment.name}_id")
       end
 
-      def cookie_resolver(jar, hash)
-        [jar, hash, @experiment.identifying_key, jar.signed[cookie_name]].compact
-      end
+      def add_migration(value, merge: false)
+        return unless value.is_a?(Hash)
 
-      def cookie_name
-        @cookie_name ||= [@experiment.name, 'id'].join('_')
-      end
-
-      def resolve_cookie(jar, hash, key, cookie = nil)
-        return if cookie.blank? && hash[key].blank?
-        return hash.merge(key => cookie) if hash[key].blank?
-
-        @migrations_with << { key => cookie }
-        jar.delete(cookie_name, domain: :all)
-
-        hash
-      end
-
-      def generate_cookie(jar, hash, key, cookie = SecureRandom.uuid)
-        return hash unless hash.key?(key)
-
-        jar.permanent.signed[cookie_name] = {
-          value: cookie, secure: true, domain: :all, httponly: true
-        }
-
-        hash.merge(key => cookie)
+        @migrations[merge ? :merged : :unmerged] << value
       end
 
       def migration_keys
-        return nil if @migrations_from.empty? && @migrations_with.empty?
-
-        @migrations_from.map { |m| key_for(m) } +
-          @migrations_with.map { |m| key_for(@value.merge(m)) }
-      end
+        return nil if @migrations[:unmerged].empty? && @migrations[:merged].empty?
 
-      def key_for(context)
-        Configuration.context_hash_strategy.call(context)
+        @migrations[:unmerged].map { |m| @experiment.key_for(m) } +
+          @migrations[:merged].map { |m| @experiment.key_for(@value.merge(m)) }
       end
     end
   end

data/lib/gitlab/experiment/cookies.rb

@@ -0,0 +1,48 @@
+# frozen_string_literal: true
+
+require 'securerandom'
+
+module Gitlab
+  class Experiment
+    module Cookies
+      private
+
+      def migrate_cookie(hash, cookie_name)
+        return hash if cookie_jar.nil?
+
+        resolver = [hash, :actor, cookie_name, cookie_jar.signed[cookie_name]]
+        resolve_cookie(*resolver) or generate_cookie(*resolver)
+      end
+
+      def cookie_jar
+        @request&.cookie_jar
+      end
+
+      def resolve_cookie(hash, key, cookie_name, cookie)
+        return if cookie.to_s.empty? && hash[key].nil?
+        return hash if cookie.to_s.empty?
+        return hash.merge(key => cookie) if hash[key].nil?
+
+        add_migration(key => cookie)
+        cookie_jar.delete(cookie_name, domain: domain)
+
+        hash
+      end
+
+      def generate_cookie(hash, key, cookie_name, cookie)
+        return hash unless hash.key?(key)
+
+        cookie ||= SecureRandom.uuid
+        cookie_jar.permanent.signed[cookie_name] = {
+          value: cookie, secure: true, domain: domain, httponly: true
+        }
+
+        hash.merge(key => cookie)
+      end
+
+      def domain
+        Configuration.cookie_domain
+      end
+    end
+  end
+end

data/lib/gitlab/experiment/version.rb

@@ -2,6 +2,6 @@
 
 module Gitlab
   class Experiment
-    VERSION = '0.2.3'
+    VERSION = '0.4.1'
   end
 end

metadata

@@ -1,35 +1,49 @@
 --- !ruby/object:Gem::Specification
 name: gitlab-experiment
 version: !ruby/object:Gem::Version
-  version: 0.2.3
+  version: 0.4.1
 platform: ruby
 authors:
 - GitLab
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2020-09-27 00:00:00.000000000 Z
+date: 2020-11-17 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
-  name: scientist
+  name: activesupport
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: 1.5.0
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: '1.5'
+        version: '3.0'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '3.0'
+- !ruby/object:Gem::Dependency
+  name: scientist
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.5'
     - - ">="
       - !ruby/object:Gem::Version
         version: 1.5.0
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
     - - "~>"
       - !ruby/object:Gem::Version
         version: '1.5'
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 1.5.0
 description: 
 email:
 - gitlab_rubygems@gitlab.com
@@ -39,13 +53,23 @@ extra_rdoc_files: []
 files:
 - LICENSE.txt
 - README.md
-- lib/generators/gitlab_experiment/install/POST_INSTALL
-- lib/generators/gitlab_experiment/install/install_generator.rb
-- lib/generators/gitlab_experiment/install/templates/initializer.rb
+- lib/generators/gitlab/experiment/USAGE
+- lib/generators/gitlab/experiment/experiment_generator.rb
+- lib/generators/gitlab/experiment/install/install_generator.rb
+- lib/generators/gitlab/experiment/install/templates/POST_INSTALL
+- lib/generators/gitlab/experiment/install/templates/application_experiment.rb.tt
+- lib/generators/gitlab/experiment/install/templates/initializer.rb.tt
+- lib/generators/gitlab/experiment/templates/experiment.rb.tt
+- lib/generators/rspec/experiment/experiment_generator.rb
+- lib/generators/rspec/experiment/templates/experiment_spec.rb.tt
+- lib/generators/test_unit/experiment/experiment_generator.rb
+- lib/generators/test_unit/experiment/templates/experiment_test.rb.tt
 - lib/gitlab/experiment.rb
 - lib/gitlab/experiment/caching.rb
+- lib/gitlab/experiment/callbacks.rb
 - lib/gitlab/experiment/configuration.rb
 - lib/gitlab/experiment/context.rb
+- lib/gitlab/experiment/cookies.rb
 - lib/gitlab/experiment/dsl.rb
 - lib/gitlab/experiment/engine.rb
 - lib/gitlab/experiment/variant.rb
@@ -69,7 +93,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.0.3
+rubygems_version: 3.1.4
 signing_key: 
 specification_version: 4
 summary: GitLab experiment library built on top of scientist.

data/lib/generators/gitlab_experiment/install/install_generator.rb

@@ -1,21 +0,0 @@
-# frozen_string_literal: true
-
-require 'rails/generators'
-
-module GitlabExperiment
-  module Generators
-    class InstallGenerator < Rails::Generators::Base
-      source_root File.expand_path(__dir__)
-
-      desc 'Installs the Gitlab Experiment initializer into your application.'
-
-      def copy_initializers
-        copy_file 'templates/initializer.rb', 'config/initializers/gitlab_experiment.rb'
-      end
-
-      def display_post_install
-        readme 'POST_INSTALL' if behavior == :invoke
-      end
-    end
-  end
-end