gitlab-experiment 0.2.4 → 0.4.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: a6284349854da93da979695a6493df95c0247616e10d5adc047d56d5dbdbf324
-  data.tar.gz: 1e4456945313fce9fbf03250299e3a0eb1b406035e4cce0d0559eee1fb3cfe53
+  metadata.gz: 865d51c081670824fc7ffb48d4bc764fd5370606d3aa63b2196421b5570506e7
+  data.tar.gz: a007f872a3d56ee81c9925117f603e4930d1b7d689078edb074a2ba5567833f2
 SHA512:
-  metadata.gz: '08d5c04e817c2679d115a845df588a4b81358e1b7ac7ddc8282a4cdaa2a878f6a4111b318fa3155671fdd05c0cb950ab076210518142e5f8a04db89e7b164104'
-  data.tar.gz: 118de1ae6b9995edee4058a1b756030ee6659199bf19a29cf2c1446ddb87a1a8ff29f479cf9ff20f7af47ff7706bbc97256929ed800f803fc381450bcedbf82c
+  metadata.gz: 03d401a71b952b74519a21fa851a87ee9e104ccd19011af37ac41c36bec5fd63db3e68c62c05beb34ab296d354e00e15f013dd34a547c3b4c5c94474b89096cc
+  data.tar.gz: 2bb0260e40d4689446c1317273d85140fae9f56394821fda96c7bbc5d1acbe441f3e4024bf00ec05812f64f5588897137fc8061e790d871c5719742f6cedf516

data/README.md

@@ -1,10 +1,11 @@
-# GitLab Experiment
+GitLab Experiment
+=================
 
 <img alt="experiment" src="/uploads/60990b2dbf4c0406bbf8b7f998de2dea/experiment.png" align="right" width="40%">
 
 Here at GitLab, we run experiments as A/B/n tests and review the data the experiment generates. From that data, we determine the best performing variant and promote it as the new default code path. Or revert back to the control if no variant outperformed it.
 
-This library provides a clean and elegant DSL to define, run, and track your GitLab experiment.
+This library provides a clean and elegant DSL (domain specific language) to define, run, and track your GitLab experiment.
 
 When we discuss the behavior of this gem, we'll use terms like experiment, context, control, candidate, and variant. It's worth defining these terms so they're more understood.
 
@@ -16,6 +17,8 @@ When we discuss the behavior of this gem, we'll use terms like experiment, conte
 
 Candidate and variant are the same concept, but simplify how we speak about experimental paths.<br clear="all">
 
+[[_TOC_]]
+
 ## Installation
 
 Add the gem to your Gemfile and then `bundle install`.
@@ -24,10 +27,10 @@ Add the gem to your Gemfile and then `bundle install`.
 gem 'gitlab-experiment'
 ```
 
-If you're using Rails, you can install the initializer. It provides basic configuration and documentation.
+If you're using Rails, you can install the initializer which provides basic configuration, documentation, and the base experiment class that all your experiments can inherit from.
 
 ```shell
-$ rails generate gitlab-experiment:install
+$ rails generate gitlab:experiment:install
 ```
 
 ## Implementing an experiment
@@ -38,7 +41,7 @@ In our control (current world) we show a simple toggle interface labeled, "Notif
 
 The behavior will be the same, but the interface will be different and may involve more or fewer steps.
 
-Our hypothesis is that this will make the action more clear and will help the user in making a choice about if that's what they really want to do.
+Our hypothesis is that this will make the action more clear and will help in making a choice about if that's what the user really wants to do.
 
 We'll name our experiment `notification_toggle`. This name is prefixed based on configuration. If you've set `config.name_prefix = 'gitlab'`, the experiment name would be `gitlab_notification_toggle` elsewhere.
 
@@ -51,7 +54,7 @@ Now in our experiment we're going to render one of two views: the control will b
 ```ruby
 class SubscriptionsController < ApplicationController
   def show
-    experiment(:notification_toggle, user_id: user.id) do |e|
+    experiment(:notification_toggle, actor: user) do |e|
       e.use { render_toggle } # control
       e.try { render_button } # candidate
     end
@@ -64,7 +67,7 @@ You can define the experiment using simple control/candidate paths, or provide n
 Handling multi-variant experiments is up to the configuration you provide around resolving variants. But in our example we may want to try with and without the confirmation. We can run any number of variations in our experiments this way.
 
 ```ruby
-experiment(:notification_toggle, user_id: user.id) do |e|
+experiment(:notification_toggle, actor: user) do |e|
   e.use { render_toggle } # control
   e.try(:variant_one) { render_button(confirmation: true) }
   e.try(:variant_two) { render_button(confirmation: false) }
@@ -76,75 +79,87 @@ Understanding how an experiment can change behavior is important in evaluating i
 To this end, we track events that are important by calling the same experiment elsewhere in code. By using the same context, you'll have consistent behavior and the ability to track events to it.
 
 ```ruby
-experiment(:notification_toggle, user_id: user.id).track(:clicked_button)
+experiment(:notification_toggle, actor: user).track(:clicked_button)
 ```
 
-<details>
-  <summary>You can also use the more low level class or instance interfaces...</summary>
+### Custom experiments
 
-### Class level interface using `.run`
+You can craft more advanced behaviors by defining custom experiments at a higher level. To do this you can define a class that inherits from `ApplicationExperiment` (or `Gitlab::Experiment`).
 
-```ruby
-exp = Gitlab::Experiment.run(:notification_toggle, user_id: user.id) do |e|
-  # Context may be passed in the block, but must be finalized before calling
-  # run or track.
-  e.context(project_id: project.id) # add the project id to the context
+Let's say you want to do more advanced segmentation, or provide default behavior for the variants on the experiment we've already outlined above -- that way if the variants aren't defined in the block at the time the experiment is run, these methods will be used. 
 
-  # Define the control and candidate variant.
-  e.use { render_toggle } # control
-  e.try { render_button } # candidate
-end
+You can generate a custom experiment by running:
 
-# Track an event on the experiment we've defined.
-exp.track(:clicked_button)
-```
+```shell
+$ rails generate gitlab:experiment NotificationToggle control candidate
+``` 
+
+This will generate a file in `app/experiments/notification_toggle_experiment.rb`, as well as a test file for you to further expand on.
 
-### Instance level interface
+Here are some examples of what you can introduce once you have a custom experiment defined.
 
 ```ruby
-exp = Gitlab::Experiment.new(:notification_toggle, user_id: user.id)
-# Additional context may be provided to the instance (exp) but must be
-# finalized before calling run or track.
-exp.context(project_id: project.id) # add the project id to the context
+class NotificationToggleExperiment < ApplicationExperiment
+  # Segment any account less than 2 weeks old into the candidate, without
+  # asking the variant resolver to decide which variant to provide.
+  segment :account_age, variant: :candidate
+
+  # Define the default control behavior, which can be overridden at
+  # experiment time.
+  def control_behavior
+    render_toggle
+  end
 
-# Define the control and candidate variant.
-exp.use { render_toggle } # control
-exp.try { render_button } # candidate
+  # Define the default candidate behavior, which can be overridden
+  # at experiment time.
+  def candidate_behavior
+    render_button
+  end
+
+  private
+
+  def account_age
+    context.actor && context.actor.created_at < 2.weeks.ago
+  end
+end
+
+# The class will be looked up based on the experiment name provided. 
+exp = experiment(:notification_toggle, actor: user)
+exp # => instance of NotificationToggleExperiment
 
-# Run the experiment, returning the result.
+# Run the experiment -- returning the result.
 exp.run
 
 # Track an event on the experiment we've defined.
 exp.track(:clicked_button)
 ```
 
-</details>
+You can now also do things very similar to the simple examples and override the default variant behaviors defined in the custom experiment -- keeping in mind that this should be carefully considered within the scope of your experiment.
+
+```ruby
+experiment(:notification_toggle, actor: user) do |e|
+  e.use { render_special_toggle } # override default control behavior
+end
+```
 
 <details>
-  <summary>You can define and use custom classes...</summary>
+  <summary>You can also use the lower level class interface...</summary>
 
-### Custom class
+### Using the `.run` approach
 
-```ruby
-class NotificationExperiment < Gitlab::Experiment
-  def initialize(variant_name = nil, **context, &block)
-    super(:notification_toggle, variant_name, **context, &block)
+This is useful if you haven't included the DSL and so don't have access to the `experiment` method, but still want to execute an experiment. This is ultimately what the `experiment` method calls through to, and the method signatures are the same.
 
-    # Define the control and candidate variant.
-    use { render_toggle } # control
-    try { render_button } # candidate
-  end
-end
+```ruby
+exp = Gitlab::Experiment.run(:notification_toggle, actor: user) do |e|
+  # Context may be passed in the block, but must be finalized before calling
+  # run or track.
+  e.context(project: project) # add the project to the context
 
-exp = NotificationExperiment.new(user_id: user.id) do |e|
-  # Context may be provided within the block or to the instance (exp) but must
-  # be finalized before calling run or track.
-  e.context(project_id: project.id) # add the project id to the context
+  # Define the control and candidate variant.
+  e.use { render_toggle } # control
+  e.try { render_button } # candidate
 end
 
-# Run the experiment -- returning the result.
-exp.run
-
 # Track an event on the experiment we've defined.
 exp.track(:clicked_button)
 ```
@@ -152,14 +167,14 @@ exp.track(:clicked_button)
 </details>
 
 <details>
-  <summary>You can also specify the variant to use...</summary>
+  <summary>You can also specify the variant to use for segmentation...</summary>
 
 ### Specifying variant
 
-You can hardcode the variant if you want. It's important to know what this might do to your data during rollout, so use this with consideration.
+Generally, defining segmentation rules is a better way to approach routing into specific variants, but it's possible to explicitly specify the variant when running an experiment. It's important to know what this might do to your data during rollout, so use this with careful consideration.
 
 ```ruby
-experiment(:notification_toggle, :no_interface, user_id: user.id) do |e|
+experiment(:notification_toggle, :no_interface, actor: user) do |e|
   e.use { render_toggle } # control
   e.try { render_button } # candidate
   e.try(:no_interface) { no_interface! } # variant
@@ -169,7 +184,7 @@ end
 Or you can set the variant within the block. This allows using unique segmentation logic or variant resolution if you need it.
 
 ```ruby
-experiment(:notification_toggle, user_id: user.id) do |e|
+experiment(:notification_toggle, actor: user) do |e|
   # Variant selection must be done before calling run or track.
   e.variant(:no_interface) # set the variant
   # ...
@@ -179,7 +194,7 @@ end
 Or it can be specified in the call to run if you call it from within the block.
 
 ```ruby
-experiment(:notification_toggle, user_id: user.id) do |e|
+experiment(:notification_toggle, actor: user) do |e|
   # ...
   # Variant selection can be specified when calling run.
   e.run(:no_interface)
@@ -188,6 +203,25 @@ end
 
 </details>
 
+### Segmentation rules
+
+This library comes with the capability to segment contexts into a specific variant, before asking the variant resolver which variant to provide.
+
+Segmentation can be achieved by using a custom experiment class and specifying the segmentation rules at a class level.
+
+```ruby
+class NotificationToggleExperiment < ApplicationExperiment
+  segment(variant: :variant_one) { context.actor.username == 'jejacks0n' }
+  segment(variant: :variant_two) { context.actor.created_at < 2.weeks.ago }
+end
+```
+
+In the previous examples, any user with the username `'jejacks0n'` would always receive the experience defined in "variant_one". As well, any account less than 2 weeks old would get the alternate experience defined in "variant_two".
+
+When an experiment is run, the segmentation rules are executed in the order they're defined. The first segmentation rule to produce a truthy result is the one which gets used to assign the variant. The remaining segmentation rules are skipped.
+
+This means that any user with the name `'jejacks0n'`, regardless of account age, will always be provided the experience as defined in "variant_one". 
+
 ### Return value
 
 By default the return value is a `Gitlab::Experiment` instance. In simple cases you may want only the results of the experiment though. You can call `run` within the block to get the return value of the assigned variant.
@@ -209,13 +243,13 @@ Some experiments may extend outside of those layers, so you may want to include
 Note: In a lot of these contexts you may not have a reference to the request (unless you pass it in, or provide access to it) which may be needed if you want to enable cookie behaviors and track that through to user conversion.
 
 ```ruby
-class UserMailer < ApplicationMailer
+class WelcomeMailer < ApplicationMailer
   include Gitlab::Experiment::Dsl # include the `experiment` method
 
   def welcome
     @user = params[:user]
 
-    ex = experiment(:project_suggestions, user_id: @user.id) do |e|
+    ex = experiment(:project_suggestions, actor: @user) do |e|
       e.use { 'welcome' }
       e.try { 'welcome_with_project_suggestions' }
     end
@@ -234,8 +268,8 @@ Take for instance, that you might be using `version: 1` in your context currentl
 In providing the context migration data, we can resolve an experience and its events all the way back. This can also help in keeping our cache relevant.
 
 ```ruby
-# Migrate just the `:version` portion of the previous context, `{ user_id: 42, version: 1 }`:
-experiment(:my_experiment, user_id: 42, version: 2, migrated_with: { version: 1 })
+# Migrate just the `:version` portion of the previous context, `{ actor: project, version: 1 }`:
+experiment(:my_experiment, actor: project, version: 2, migrated_with: { version: 1 })
 ```
 
 You can add or remove context by providing a `migrated_from` option. This approach expects a full context replacement -- i.e. what it was before you added or removed the new context key.
@@ -243,8 +277,8 @@ You can add or remove context by providing a `migrated_from` option. This approa
 If you wanted to introduce a `version` to your context, provide the full previous context.
 
 ```ruby
-# Migrate the full context from `{ user_id: 42 }` to `{ user_id: 42, version: 1 }`:
-experiment(:my_experiment, user_id: 42, version: 1, migrated_from: { user_id: 42 })
+# Migrate the full context from `{ actor: project }` to `{ actor: project, version: 1 }`:
+experiment(:my_experiment, actor: project, version: 1, migrated_from: { actor: project })
 ```
 
 This can impact an experience if you:
@@ -252,36 +286,36 @@ This can impact an experience if you:
 1. haven't implemented the concept of migrations in your variant resolver
 1. haven't enabled a reasonable caching mechanism
 
-### When there isn't a user (cookies)
+### When there isn't an actor (cookie fallback)
 
-When there isn't an identifying key in the context (this is `user_id` by default), we fall back to cookies to provide a consistent experience.
+When there isn't an identifying key in the context (this is `actor` by default), we fall back to cookies to provide a consistent experience for the client viewing them.
 
-Once we assign a certain variant to a context, we need to always provide the same experience. We achieve this by setting a cookie for the experiment in question.
+Once we assign a certain variant to a context, we need to always provide the same experience. We achieve this by setting a cookie for the experiment in question, but only when needed.
 
-This cookie is a randomized uuid and isn't associated with the user. When we can finally provide an identifying key, the context is auto migrated from the cookie to that identifying key. The cookie is a temporary value, and isn't used for tracking.
+This cookie is a temporary, randomized uuid and isn't associated with a user. When we can finally provide an actor, the context is auto migrated from the cookie to that actor.
 
 To read and write cookies, we provide the `request` from within the controller and views. The cookie migration will happen automatically if the experiment is within those layers.
 
 You'll need to provide the `request` as an option to the experiment if it's outside of the controller and views.
 
 ```ruby
-experiment(:my_experiment, user_id: user&.id, request: request)
+experiment(:my_experiment, actor: user, request: request)
 ```
 
-The cookie isn't set if the identifying key isn't present at all in the context. Using the default identifying key, when there is no `user_id` key provided, the cookie will not be set.
+The cookie isn't set if the `actor` key isn't present at all in the context. Meaning that when no `actor` key is provided, the cookie will not be set.
 
 ```ruby
-# no user_id context key is present, so no cookie is set
-experiment(:my_experiment, project_id: @project.id)
+# actor is not present, so no cookie is set
+experiment(:my_experiment, project: project)
 
-# user_id context key is present, but is set to nil, so the cookie is set & used instead
-experiment(:my_experiment, user_id: nil, project_id: @project.id)
+# actor is present and is nil, so the cookie is set and used
+experiment(:my_experiment, actor: nil, project: project)
 
-# user_id context key is present and set to a value, so no cookie is set
-experiment(:my_experiment, user_id: @user.id, project_id: @project.id)
+# actor is present and set to a value, so no cookie is set
+experiment(:my_experiment, actor: user, project: project)
 ```
 
-For edge cases, you can pass the cookie through by assigning it yourself -- e.g. `user_id: request.cookie_jar.signed['my_experiment_id']`. The cookie name is the full experiment name (including any configured prefix) with `_id` appended -- e.g. `gitlab_notification_toggle_id` for the `:notification_toggle` experiment key with a configured prefix of `gitlab`.
+For edge cases, you can pass the cookie through by assigning it yourself -- e.g. `actor: request.cookie_jar.signed['my_experiment_actor']`. The cookie name is the full experiment name (including any configured prefix) with `_actor` appended -- e.g. `gitlab_notification_toggle_actor` for the `:notification_toggle` experiment key with a configured prefix of `gitlab`.
 
 ## Configuration
 
@@ -307,7 +341,7 @@ Gitlab::Experiment.configure do |config|
 end
 ```
 
-More examples for configuration are available in the provided [rails initializer](lib/generators/gitlab_experiment/install/templates/initializer.rb).
+More examples for configuration are available in the provided [rails initializer](lib/generators/gitlab/experiment/install/templates/initializer.rb).
 
 ### Client layer / JavaScript
 
@@ -351,4 +385,32 @@ If you only include a user, that user would get the same experience across every
 
 Each of these approaches could be desirable given the objectives of your experiment.
 
-### Make code not war
+## Development
+
+After checking out the repo, run `bundle install` to install dependencies.
+Then, run `bundle exec rspec` to run the tests. You can also run `bundle exec pry` for an
+interactive prompt that will allow you to experiment.
+
+## Contributing
+
+Bug reports and merge requests are welcome on GitLab at
+https://gitlab.com/gitlab-org/gitlab-experiment. This project is intended to be a
+safe, welcoming space for collaboration, and contributors are expected to adhere
+to the [Contributor Covenant](http://contributor-covenant.org) code of conduct.
+
+## Release Process
+
+Please refer to the [Release Process](docs/release_process.md).
+
+## License
+
+The gem is available as open source under the terms of the
+[MIT License](http://opensource.org/licenses/MIT).
+
+## Code of Conduct
+
+Everyone interacting in the `Gitlab::Experiment` project’s codebases, issue trackers,
+chat rooms and mailing lists is expected to follow the
+[code of conduct](CODE_OF_CONDUCT.md).
+
+***Make code not war***

data/lib/generators/gitlab/experiment/USAGE

@@ -0,0 +1,17 @@
+Description:
+    Stubs out a new experiment and its variants. Pass the experiment name,
+    either CamelCased or under_scored, and a list of variants as arguments.
+
+    To create an experiment within a module, specify the experiment name as a
+    path like 'parent_module/experiment_name'.
+
+    This generates an experiment class in app/experiments and invokes feature
+    flag, and test framework generators.
+
+Example:
+    `rails generate gitlab:experiment NullHypothesis control candidate alt_variant`
+
+    NullHypothesis experiment with default variants.
+        Experiment:   app/experiments/null_hypothesis_experiment.rb
+        Feature Flag: config/feature_flags/experiment/null_hypothesis.yaml
+        Test:         test/experiments/null_hypothesis_experiment_test.rb

data/lib/generators/gitlab/experiment/experiment_generator.rb

@@ -0,0 +1,33 @@
+# frozen_string_literal: true
+
+require 'rails/generators'
+
+module Gitlab
+  module Generators
+    class ExperimentGenerator < Rails::Generators::NamedBase
+      source_root File.expand_path('templates/', __dir__)
+      check_class_collision suffix: 'Experiment'
+
+      argument :variants,
+               type: :array,
+               default: %w[control candidate],
+               banner: 'variant variant'
+
+      def create_experiment
+        template 'experiment.rb', File.join('app/experiments', class_path, "#{file_name}_experiment.rb")
+      end
+
+      hook_for :test_framework
+
+      private
+
+      def file_name
+        @_file_name ||= remove_possible_suffix(super)
+      end
+
+      def remove_possible_suffix(name)
+        name.sub(/_?exp[ei]riment$/i, "") # be somewhat forgiving with spelling
+      end
+    end
+  end
+end

data/lib/generators/gitlab/experiment/install/install_generator.rb

@@ -0,0 +1,41 @@
+# frozen_string_literal: true
+
+require 'rails/generators'
+
+module Gitlab
+  module Generators
+    module Experiment
+      class InstallGenerator < Rails::Generators::Base
+        source_root File.expand_path('templates', __dir__)
+
+        desc 'Installs the Gitlab::Experiment initializer and optional ApplicationExperiment into your application.'
+
+        class_option :skip_initializer,
+                     type: :boolean,
+                     default: false,
+                     desc: 'Skip the initializer with default configuration'
+
+        class_option :skip_baseclass,
+                     type: :boolean,
+                     default: false,
+                     desc: 'Skip the ApplicationExperiment base class'
+
+        def create_initializer
+          return if options[:skip_initializer]
+
+          template 'initializer.rb', 'config/initializers/gitlab_experiment.rb'
+        end
+
+        def create_baseclass
+          return if options[:skip_baseclass]
+
+          template 'application_experiment.rb', 'app/experiments/application_experiment.rb'
+        end
+
+        def display_post_install
+          readme 'POST_INSTALL' if behavior == :invoke
+        end
+      end
+    end
+  end
+end

data/lib/generators/gitlab/experiment/install/templates/POST_INSTALL

@@ -0,0 +1,2 @@
+Gitlab::Experiment has been installed. You may want to adjust the configuration
+that's been provided in the Rails initializer.

data/lib/generators/gitlab/experiment/install/templates/application_experiment.rb.tt

@@ -0,0 +1,4 @@
+# frozen_string_literal: true
+
+class ApplicationExperiment < Gitlab::Experiment
+end

data/lib/generators/{gitlab_experiment/install/templates/initializer.rb → gitlab/experiment/install/templates/initializer.rb.tt}

@@ -5,19 +5,30 @@ Gitlab::Experiment.configure do |config|
   config.name_prefix = nil
 
   # The logger is used to log various details of the experiments.
-  config.logger = Logger.new(STDOUT)
+  config.logger = Logger.new($stdout)
 
-  # The caching layer is expected to respond to fetch, like Rails.cache.
+  # The base class that should be instantiated for basic experiments. It should
+  # be a string, so we can constantize it later.
+  config.base_class = 'ApplicationExperiment'
+
+  # The caching layer is expected to respond to fetch, like Rails.cache for
+  # instance -- or anything that adheres to ActiveSupport::Cache::Store.
   config.cache = nil
 
+  # The domain to use on cookies.
+  #
+  # When not set, it uses the current host. If you want to provide specific
+  # hosts, you use `:all`, or provide an array like
+  # `['www.gitlab.com', '.gitlab.com']`.
+  config.cookie_domain = :all
+
   # Logic this project uses to resolve a variant for a given experiment.
   #
-  # This can return an instance of any object that responds to `name`, or can
-  # return a variant name as a string, in which case the build in variant
-  # class will be used.
+  # Should return a symbol or string that represents the variant that should
+  # be assigned.
   #
-  # This block will be executed within the scope of the experiment instance,
-  # so can easily access experiment methods, like getting the name or context.
+  # This block is executed within the scope of the experiment and so can access
+  # experiment methods, like `name`, `context`, and `signature`.
   config.variant_resolver = lambda do |requested_variant|
     # Run the control, unless a variant was requested in code:
     requested_variant || 'control'
@@ -25,23 +36,12 @@ Gitlab::Experiment.configure do |config|
     # Run the candidate, unless a variant was requested, with a fallback:
     #
     # requested_variant || variant_names.first || 'control'
-
-    # Using Unleash to determine the variant:
-    #
-    # fallback = Unleash::Variant.new(name: requested_variant || 'control', enabled: true)
-    # Unleash.get_variant(name, context.value, fallback)
-
-    # Using Flipper to determine the variant:
-    #
-    # TODO: provide example.
-    # Variant.new(name: requested_variant || 'control')
   end
 
   # Tracking behavior can be implemented to link an event to an experiment.
   #
-  # Similar to the variant_resolver, this is called within the scope of the
-  # experiment instance and so can access any methods on the experiment,
-  # such as name and signature.
+  # This block is executed within the scope of the experiment and so can access
+  # experiment methods, like `name`, `context`, and `signature`.
   config.tracking_behavior = lambda do |event, args|
     # An example of using a generic logger to track events:
     config.logger.info "Gitlab::Experiment[#{name}] #{event}: #{args.merge(signature: signature)}"
@@ -58,8 +58,11 @@ Gitlab::Experiment.configure do |config|
   # Called at the end of every experiment run, with the result.
   #
   # You may want to track that you've assigned a variant to a given context,
-  # or push the experiment into the client or publish results elsewhere, like
-  # into redis. Also called within the scope of the experiment instance.
+  # or push the experiment into the client or publish results elsewhere like
+  # into redis.
+  #
+  # This block is executed within the scope of the experiment and so can access
+  # experiment methods, like `name`, `context`, and `signature`.
   config.publishing_behavior = lambda do |result|
     # Track the event using our own configured tracking logic.
     track(:assignment)
@@ -80,6 +83,9 @@ Gitlab::Experiment.configure do |config|
   # Given a specific context hash map, we need to generate a consistent hash
   # key. The logic in here will be used for generating cache keys, and may also
   # be used when determining which variant may be presented.
+  #
+  # This block is executed within the scope of the experiment and so can access
+  # experiment methods, like `name`, `context`, and `signature`.
   config.context_hash_strategy = lambda do |context|
     values = context.values.map { |v| (v.respond_to?(:to_global_id) ? v.to_global_id : v).to_s }
     Digest::MD5.hexdigest((context.keys + values).join('|'))

data/lib/generators/gitlab/experiment/templates/experiment.rb.tt

@@ -0,0 +1,15 @@
+# frozen_string_literal: true
+
+<% if namespaced? -%>
+require_dependency "<%= namespaced_path %>/application_experiment"
+
+<% end -%>
+<% module_namespacing do -%>
+class <%= class_name %>Experiment < ApplicationExperiment
+<% variants.each do |variant| -%>
+  def <%= variant %>_behavior
+  end
+<%= "\n" unless variant == variants.last -%>
+<% end -%>
+end
+<% end -%>

data/lib/generators/rspec/experiment/experiment_generator.rb

@@ -0,0 +1,15 @@
+# frozen_string_literal: true
+
+require 'generators/rspec'
+
+module Rspec
+  module Generators
+    class ExperimentGenerator < Rspec::Generators::Base
+      source_root File.expand_path('templates/', __dir__)
+
+      def create_experiment_spec
+        template 'experiment_spec.rb', File.join('spec/experiments', class_path, "#{file_name}_experiment_spec.rb")
+      end
+    end
+  end
+end

data/lib/generators/rspec/experiment/templates/experiment_spec.rb.tt

@@ -0,0 +1,9 @@
+# frozen_string_literal: true
+
+require 'rails_helper'
+
+<% module_namespacing do -%>
+RSpec.describe <%= class_name %>Experiment do
+  pending "add some examples to (or delete) #{__FILE__}"
+end
+<% end -%>

data/lib/generators/test_unit/experiment/experiment_generator.rb

@@ -0,0 +1,17 @@
+# frozen_string_literal: true
+
+require 'rails/generators/test_unit'
+
+module TestUnit # :nodoc:
+  module Generators # :nodoc:
+    class ExperimentGenerator < TestUnit::Generators::Base # :nodoc:
+      source_root File.expand_path('templates/', __dir__)
+
+      check_class_collision suffix: 'Test'
+
+      def create_test_file
+        template 'experiment_test.rb', File.join('test/experiments', class_path, "#{file_name}_experiment_test.rb")
+      end
+    end
+  end
+end

data/lib/generators/test_unit/experiment/templates/experiment_test.rb.tt

@@ -0,0 +1,11 @@
+# frozen_string_literal: true
+
+require 'test_helper'
+
+<% module_namespacing do -%>
+class <%= class_name %>ExperimentTest < ActiveSupport::TestCase
+  # test "the truth" do
+  #   assert true
+  # end
+end
+<% end -%>

data/lib/gitlab/experiment.rb

@@ -1,9 +1,14 @@
 # frozen_string_literal: true
 
 require 'scientist'
+require 'active_support/callbacks'
+require 'active_support/core_ext/object/blank'
+require 'active_support/core_ext/string/inflections'
 
 require 'gitlab/experiment/caching'
+require 'gitlab/experiment/callbacks'
 require 'gitlab/experiment/configuration'
+require 'gitlab/experiment/cookies'
 require 'gitlab/experiment/context'
 require 'gitlab/experiment/dsl'
 require 'gitlab/experiment/variant'
@@ -14,53 +19,85 @@ module Gitlab
   class Experiment
     include Scientist::Experiment
     include Caching
+    include Callbacks
 
     class << self
       def configure
         yield Configuration
       end
 
-      def run(name, variant_name = nil, **context, &block)
-        instance = new(name, variant_name, **context, &block)
+      def run(name = nil, variant_name = nil, **context, &block)
+        raise ArgumentError, 'name is required' if name.nil? && base?
+
+        instance = constantize(name).new(name, variant_name, **context, &block)
         return instance unless block_given?
 
         instance.context.frozen? ? instance.run : instance.tap(&:run)
       end
+
+      def experiment_name(name = nil, suffix: true, suffix_word: 'experiment')
+        name = (name.presence || self.name).to_s.underscore.sub(%r{(?<char>[_/]|)#{suffix_word}$}, '')
+        name = "#{name}#{Regexp.last_match(:char) || '_'}#{suffix_word}"
+        suffix ? name : name.sub(/_#{suffix_word}$/, '')
+      end
+
+      def base?
+        self == Gitlab::Experiment || name == Configuration.base_class
+      end
+
+      private
+
+      def constantize(name = nil)
+        return self if name.nil?
+
+        experiment_name(name).classify.safe_constantize || Configuration.base_class.constantize
+      end
     end
 
-    delegate :signature, :cache_strategy, to: :context
+    def initialize(name = nil, variant_name = nil, **context)
+      raise ArgumentError, 'name is required' if name.blank? && self.class.base?
 
-    def initialize(name, variant_name = nil, **context)
-      @name = name
+      @name = self.class.experiment_name(name, suffix: false)
       @variant_name = variant_name
-      @context = Context.new(self)
+      @excluded = []
+      @context = Context.new(self, context)
 
-      context(context)
-
-      ignore { true }
+      exclude { !@context.trackable? }
       compare { false }
 
       yield self if block_given?
     end
 
     def context(value = nil)
-      return @context if value.nil?
+      return @context if value.blank?
 
       @context.value(value)
       @context
     end
 
     def variant(value = nil)
-      return @variant_name = value unless value.nil?
+      @variant_name = value unless value.blank?
+      @variant_name ||= :control if excluded?
 
-      result = instance_exec(@variant_name, &Configuration.variant_resolver)
-      result.respond_to?(:name) ? result : Variant.new(name: result.to_s)
+      resolved = cache { resolve_variant_name }
+      Variant.new(name: (resolved.presence || :control).to_s)
     end
 
-    def run(variant_name = nil)
-      @variant_name = variant_name unless variant_name.nil?
+    def exclude(&block)
+      @excluded << block
+    end
 
-      @result ||= super(cache { variant.name })
+    def run(variant_name = nil)
+      @result ||= begin
+        variant_name = variant(variant_name).name
+        run_callbacks(variant_assigned? ? :unsegmented_run : :segmented_run) do
+          if respond_to?((behavior_name = "#{variant_name}_behavior"))
+            behaviors[variant_name] ||= -> { send(behavior_name) } # rubocop:disable GitlabSecurity/PublicSend
+          end
+
+          super(@variant_name = variant_name)
+        end
+      end
     end
 
     def publish(result)
@@ -68,6 +105,8 @@ module Gitlab
     end
 
     def track(action, **event_args)
+      return if excluded?
+
       instance_exec(action, event_args, &Configuration.tracking_behavior)
     end
 
@@ -76,23 +115,49 @@ module Gitlab
     end
 
     def variant_names
-      @variant_names ||= behaviors.keys.tap { |keys| keys.delete('control') }.map(&:to_sym)
+      @variant_names ||= behaviors.keys.map(&:to_sym) - [:control]
+    end
+
+    def signature
+      { variant: variant.name, experiment: name }.merge(context.signature)
     end
 
     def enabled?
       true
     end
 
-    def identifying_key
-      :user_id
+    def excluded?
+      @excluded.any? { |exclude| exclude.call(self) }
+    end
+
+    def variant_assigned?
+      !@variant_name.nil?
+    end
+
+    def id
+      "#{name}:#{key_for(context.value)}"
+    end
+    alias_method :session_id, :id
+
+    def flipper_id
+      "Experiment;#{id}"
     end
 
-    def cache_key_for(key, migration: false)
-      "#{name}:#{key}"
+    def key_for(hash)
+      instance_exec(hash, &Configuration.context_hash_strategy)
     end
 
     protected
 
+    def resolve_variant_name
+      return :unresolved if @resolving
+
+      @resolving = true
+      result = instance_exec(@variant_name, &Configuration.variant_resolver)
+      @resolving = false
+      result
+    end
+
     def generate_result(variant_name)
       observation = Scientist::Observation.new(variant_name, self, &behaviors[variant_name])
       Scientist::Result.new(self, [observation], observation)

data/lib/gitlab/experiment/caching.rb

@@ -10,6 +10,15 @@ module Gitlab
         migrated_cache(cache, migrations || [], key) or cache.fetch(key, &block)
       end
 
+      private
+
+      def cache_strategy
+        [
+          "#{name}:#{context.signature[:key]}",
+          context.signature[:migration_keys]&.map { |key| "#{name}:#{key}" }
+        ]
+      end
+
       def migrated_cache(cache, migrations, new_key)
         migrations.find do |old_key|
           next unless (value = cache.read(old_key))

data/lib/gitlab/experiment/callbacks.rb

@@ -0,0 +1,39 @@
+# frozen_string_literal: true
+
+module Gitlab
+  class Experiment
+    module Callbacks
+      extend ActiveSupport::Concern
+      include ActiveSupport::Callbacks
+
+      included do
+        define_callbacks(
+          :unsegmented_run,
+          skip_after_callbacks_if_terminated: true
+        )
+
+        define_callbacks(
+          :segmented_run,
+          skip_after_callbacks_if_terminated: false,
+          terminator: lambda do |target, result_lambda|
+            result_lambda.call
+            target.variant_assigned?
+          end
+        )
+      end
+
+      class_methods do
+        def segment(*filter_list, variant:, **options, &block)
+          filters = filter_list.unshift(block).compact.map do |filter|
+            result_lambda = ActiveSupport::Callbacks::CallTemplate.build(filter, self).make_lambda
+            ->(target) { target.variant(variant) if result_lambda.call(target, nil) }
+          end
+
+          raise ArgumentError, 'no filters provided' if filters.empty?
+
+          set_callback(:segmented_run, :before, *filters, options, &block)
+        end
+      end
+    end
+  end
+end

data/lib/gitlab/experiment/configuration.rb

@@ -13,14 +13,20 @@ module Gitlab
       @name_prefix = nil
 
       # The logger is used to log various details of the experiments.
-      @logger = Logger.new(STDOUT)
+      @logger = Logger.new($stdout)
 
-      # Cache layer. Expected to respond to fetch, like Rails.cache.
+      # The base class that should be instantiated for basic experiments.
+      @base_class = 'Gitlab::Experiment'
+
+      # The caching layer is expected to respond to fetch, like Rails.cache.
       @cache = nil
 
+      # The domain to use on cookies.
+      @cookie_domain = :all
+
       # Logic this project uses to resolve a variant for a given experiment.
       @variant_resolver = lambda do |requested_variant|
-        requested_variant || 'control'
+        requested_variant || :control
       end
 
       # Tracking behavior can be implemented to link an event to an experiment.
@@ -28,21 +34,29 @@ module Gitlab
         Configuration.logger.info "Gitlab::Experiment[#{name}] #{event}: #{args.merge(signature: signature)}"
       end
 
-      # Called at the end of every experiment run, with the results. You may
-      # want to push the experiment into the client or push results elsewhere.
+      # Called at the end of every experiment run, with the result.
       @publishing_behavior = lambda do |_result|
         track(:assignment)
       end
 
       # Algorithm that consistently generates a hash key for a given hash map.
-      @context_hash_strategy = lambda do |context|
-        values = context.values.map { |v| (v.respond_to?(:to_global_id) ? v.to_global_id : v).to_s }
-        Digest::MD5.hexdigest((context.keys + values).join('|'))
+      @context_hash_strategy = lambda do |hash_map|
+        values = hash_map.values.map { |v| (v.respond_to?(:to_global_id) ? v.to_global_id : v).to_s }
+        Digest::MD5.hexdigest(([name] + hash_map.keys + values).join('|'))
       end
 
       class << self
-        attr_accessor :name_prefix, :logger, :cache
-        attr_accessor :variant_resolver, :tracking_behavior, :publishing_behavior, :context_hash_strategy
+        attr_accessor(
+          :name_prefix,
+          :logger,
+          :base_class,
+          :cache,
+          :cookie_domain,
+          :variant_resolver,
+          :tracking_behavior,
+          :publishing_behavior,
+          :context_hash_strategy
+        )
       end
     end
   end

data/lib/gitlab/experiment/context.rb

@@ -3,96 +3,65 @@
 module Gitlab
   class Experiment
     class Context
+      include Cookies
+
       DNT_REGEXP = /^(true|t|yes|y|1|on)$/i.freeze
 
-      def initialize(experiment)
+      def initialize(experiment, **initial_value)
         @experiment = experiment
         @value = {}
-        @migrations_from = []
-        @migrations_with = []
+        @migrations = { merged: [], unmerged: [] }
+
+        value(initial_value)
+      end
+
+      def reinitialize(request)
+        @signature = nil # clear memoization
+        @request = request if request.respond_to?(:headers) && request.respond_to?(:cookie_jar)
       end
 
       def value(value = nil)
         return @value if value.nil?
 
         value = value.dup # dup so we don't mutate
-        @signature = @cache_strategy = nil # clear memoization
+        reinitialize(value.delete(:request))
 
-        @migrations_from << value.delete(:migrated_from) if value[:migrated_from]
-        @migrations_with << value.delete(:migrated_with) if value[:migrated_with]
-        @value.merge!(auto_migrate_cookie(value, value.delete(:request)))
+        @value.merge!(process_migrations(value))
       end
 
-      def freeze
-        cache_strategy # ensure we memoize before freezing
-        super
+      def trackable?
+        !(@request && @request.headers['DNT'].to_s.match?(DNT_REGEXP))
       end
 
-      def cache_strategy
-        @cache_strategy ||= [
-          @experiment.cache_key_for(signature[:key]),
-          signature[:migration_keys]&.map do |key|
-            @experiment.cache_key_for(key, migration: true)
-          end
-        ]
+      def freeze
+        signature # finalize before freezing
+        super
       end
 
       def signature
-        @signature ||= {
-          key: key_for(@value),
-          migration_keys: migration_keys,
-          variant: @experiment.variant.name
-        }.compact
+        @signature ||= { key: @experiment.key_for(@value), migration_keys: migration_keys }.compact
       end
 
       private
 
-      def auto_migrate_cookie(hash, request)
-        return hash unless request&.respond_to?(:headers) && request&.respond_to?(:cookie_jar)
-        return hash if request.headers['DNT'].to_s.match?(DNT_REGEXP)
+      def process_migrations(value)
+        add_migration(value.delete(:migrated_from))
+        add_migration(value.delete(:migrated_with), merge: true)
 
-        resolver = cookie_resolver(request.cookie_jar, hash)
-        resolve_cookie(*resolver) or generate_cookie(*resolver)
+        migrate_cookie(value, "#{@experiment.name}_id")
       end
 
-      def cookie_resolver(jar, hash)
-        [jar, hash, @experiment.identifying_key, jar.signed[cookie_name]].compact
-      end
+      def add_migration(value, merge: false)
+        return unless value.is_a?(Hash)
 
-      def cookie_name
-        @cookie_name ||= [@experiment.name, 'id'].join('_')
-      end
-
-      def resolve_cookie(jar, hash, key, cookie = nil)
-        return if cookie.blank? && hash[key].blank?
-        return hash if cookie.blank?
-        return hash.merge(key => cookie) if hash[key].blank?
-
-        @migrations_with << { key => cookie }
-        jar.delete(cookie_name, domain: :all)
-
-        hash
-      end
-
-      def generate_cookie(jar, hash, key, cookie = SecureRandom.uuid)
-        return hash unless hash.key?(key)
-
-        jar.permanent.signed[cookie_name] = {
-          value: cookie, secure: true, domain: :all, httponly: true
-        }
-
-        hash.merge(key => cookie)
+        @migrations[merge ? :merged : :unmerged] << value
       end
 
       def migration_keys
-        return nil if @migrations_from.empty? && @migrations_with.empty?
-
-        @migrations_from.map { |m| key_for(m) } +
-          @migrations_with.map { |m| key_for(@value.merge(m)) }
-      end
+        return nil if @migrations[:unmerged].empty? && @migrations[:merged].empty?
 
-      def key_for(context)
-        Configuration.context_hash_strategy.call(context)
+        @migrations[:unmerged].map { |m| @experiment.key_for(m) } +
+          @migrations[:merged].map { |m| @experiment.key_for(@value.merge(m)) }
       end
     end
   end

data/lib/gitlab/experiment/cookies.rb

@@ -0,0 +1,48 @@
+# frozen_string_literal: true
+
+require 'securerandom'
+
+module Gitlab
+  class Experiment
+    module Cookies
+      private
+
+      def migrate_cookie(hash, cookie_name)
+        return hash if cookie_jar.nil?
+
+        resolver = [hash, :actor, cookie_name, cookie_jar.signed[cookie_name]]
+        resolve_cookie(*resolver) or generate_cookie(*resolver)
+      end
+
+      def cookie_jar
+        @request&.cookie_jar
+      end
+
+      def resolve_cookie(hash, key, cookie_name, cookie)
+        return if cookie.to_s.empty? && hash[key].nil?
+        return hash if cookie.to_s.empty?
+        return hash.merge(key => cookie) if hash[key].nil?
+
+        add_migration(key => cookie)
+        cookie_jar.delete(cookie_name, domain: domain)
+
+        hash
+      end
+
+      def generate_cookie(hash, key, cookie_name, cookie)
+        return hash unless hash.key?(key)
+
+        cookie ||= SecureRandom.uuid
+        cookie_jar.permanent.signed[cookie_name] = {
+          value: cookie, secure: true, domain: domain, httponly: true
+        }
+
+        hash.merge(key => cookie)
+      end
+
+      def domain
+        Configuration.cookie_domain
+      end
+    end
+  end
+end

data/lib/gitlab/experiment/version.rb

@@ -2,6 +2,6 @@
 
 module Gitlab
   class Experiment
-    VERSION = '0.2.4'
+    VERSION = '0.4.2'
   end
 end

metadata

@@ -1,15 +1,29 @@
 --- !ruby/object:Gem::Specification
 name: gitlab-experiment
 version: !ruby/object:Gem::Version
-  version: 0.2.4
+  version: 0.4.2
 platform: ruby
 authors:
 - GitLab
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2020-10-06 00:00:00.000000000 Z
+date: 2020-11-19 00:00:00.000000000 Z
 dependencies:
+- !ruby/object:Gem::Dependency
+  name: activesupport
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '3.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '3.0'
 - !ruby/object:Gem::Dependency
   name: scientist
   requirement: !ruby/object:Gem::Requirement
@@ -39,13 +53,23 @@ extra_rdoc_files: []
 files:
 - LICENSE.txt
 - README.md
-- lib/generators/gitlab_experiment/install/POST_INSTALL
-- lib/generators/gitlab_experiment/install/install_generator.rb
-- lib/generators/gitlab_experiment/install/templates/initializer.rb
+- lib/generators/gitlab/experiment/USAGE
+- lib/generators/gitlab/experiment/experiment_generator.rb
+- lib/generators/gitlab/experiment/install/install_generator.rb
+- lib/generators/gitlab/experiment/install/templates/POST_INSTALL
+- lib/generators/gitlab/experiment/install/templates/application_experiment.rb.tt
+- lib/generators/gitlab/experiment/install/templates/initializer.rb.tt
+- lib/generators/gitlab/experiment/templates/experiment.rb.tt
+- lib/generators/rspec/experiment/experiment_generator.rb
+- lib/generators/rspec/experiment/templates/experiment_spec.rb.tt
+- lib/generators/test_unit/experiment/experiment_generator.rb
+- lib/generators/test_unit/experiment/templates/experiment_test.rb.tt
 - lib/gitlab/experiment.rb
 - lib/gitlab/experiment/caching.rb
+- lib/gitlab/experiment/callbacks.rb
 - lib/gitlab/experiment/configuration.rb
 - lib/gitlab/experiment/context.rb
+- lib/gitlab/experiment/cookies.rb
 - lib/gitlab/experiment/dsl.rb
 - lib/gitlab/experiment/engine.rb
 - lib/gitlab/experiment/variant.rb

data/lib/generators/gitlab_experiment/install/install_generator.rb

@@ -1,21 +0,0 @@
-# frozen_string_literal: true
-
-require 'rails/generators'
-
-module GitlabExperiment
-  module Generators
-    class InstallGenerator < Rails::Generators::Base
-      source_root File.expand_path(__dir__)
-
-      desc 'Installs the Gitlab Experiment initializer into your application.'
-
-      def copy_initializers
-        copy_file 'templates/initializer.rb', 'config/initializers/gitlab_experiment.rb'
-      end
-
-      def display_post_install
-        readme 'POST_INSTALL' if behavior == :invoke
-      end
-    end
-  end
-end