gitlab-experiment 0.2.2 → 0.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: '08b63f640a90ed463a4b2bf58c2e0b1ef041bbd8f80c2d43bf88960ba9aac2a9'
-  data.tar.gz: 01a8a227c467d097b9dbb9dc9644a1d70012fb3989f2e189e3e63a41e7f39cfd
+  metadata.gz: 8524dff7f908e481923e3131f3dd67c34f426e31ed10171896ea10a209e0b74a
+  data.tar.gz: 151f2e7f7bbf9692350c02f46f0c4c8cfa6f9c7ec05fdcb0fda50e34324c06fb
 SHA512:
-  metadata.gz: b161004e60bbbdb724aeb685013ce4b787d0ced9526bdd241e52be327adbc3461ea60eb3a279f25b4f89d46ee37825ae1d571501d0196a8505a9ff4eb8b3d214
-  data.tar.gz: 8730e755e09ec85fa88b2721af42ec0919d6534449a91d82bc0d2f6ae0055f70e4ad783d045d20dc09c44496c63465656abfe33d6759e3956a153d793f57a2ab
+  metadata.gz: 3f23698aabf3968c77f49cdb924125879663670c12d22c390a47f7c791687f87fb1211adf432777f66012e3be1db2e90d5c33a1dc0a156332bb8926938c4e587
+  data.tar.gz: 51973494136edef02672c9086970fec2ecb1fb3a6716edb10d1cfe758172f8d256ecd6ef1b5cefdd50133898637e022af527fc7e20df2212b1f7210db2dd087e

data/README.md

@@ -1,73 +1,82 @@
-GitLab Experiment
-=================
+# GitLab Experiment
 
-Here at GitLab, experiments are run as A/B/n tests and are evaluated by the data the experiment generates. The team reviews the data, determines the variant that performed most effectively, and promotes that variant as the new default code path (or reverts back to the control).
+<img alt="experiment" src="/uploads/60990b2dbf4c0406bbf8b7f998de2dea/experiment.png" align="right" width="40%">
 
-When we discuss the behavior of this gem, we'll use terms like experiment, control, candidate, and variant. It's worth defining these terms so they're more understood.
+Here at GitLab, we run experiments as A/B/n tests and review the data the experiment generates. From that data, we determine the best performing variant and promote it as the new default code path. Or revert back to the control if no variant outperformed it.
+
+This library provides a clean and elegant DSL (domain specific language) to define, run, and track your GitLab experiment.
+
+When we discuss the behavior of this gem, we'll use terms like experiment, context, control, candidate, and variant. It's worth defining these terms so they're more understood.
 
 - `experiment` is any deviation of code paths we want to run sometimes and not others.
+- `context` is used to identify a consistent experience we'll provide in an experiment.
 - `control` is the default, or "original" code path.
-- `candidate` is used if there's one experimental code path.
+- `candidate` defines that there's one experimental code path.
 - `variant(s)` is used when more than one experimental code path exists.
 
-Candidate and variant are referencing the same concept, but can simplify how we speak about the available code paths of a given experiment.
+Candidate and variant are the same concept, but simplify how we speak about experimental paths.<br clear="all">
 
 ## Installation
 
-Add the gem to your Gemfile and then bundle install.
+Add the gem to your Gemfile and then `bundle install`.
 
 ```ruby
 gem 'gitlab-experiment'
 ```
 
-If you're using Rails, you can install an initializer that provides basic configuration by running the install generator.  
+If you're using Rails, you can install the initializer. It provides basic configuration and documentation.
 
 ```shell
-$ rails generate gitlab-experiment:install
+$ rails generate gitlab:experiment:install
 ```
 
 ## Implementing an experiment
 
-For the sake of our example, let's say we want to run an experiment for the interface we render for subscription cancellation.
+For the sake of an example let's make one up. Let's run an experiment on what we render for disabling desktop notifications.
+
+In our control (current world) we show a simple toggle interface labeled, "Notifications." In our experiment we want a "Turn on/off desktop notifications" button with a confirmation.
+
+The behavior will be the same, but the interface will be different and may involve more or fewer steps.
+
+Our hypothesis is that this will make the action more clear and will help in making a choice about if that's what the user really wants to do.
 
-In our control (current world) we show a simple toggle interface that reads "Auto-renew", and in our experiment candidate we want to show a "Cancel subscription" button with a confirmation dialog. Ultimately the behavior will be the same, but the interface will be considerably different and may involve more or less steps.
- 
-We hypothesize that making it more clear what the action is will help people in making a choice about taking that action or not. 
+We'll name our experiment `notification_toggle`. This name is prefixed based on configuration. If you've set `config.name_prefix = 'gitlab'`, the experiment name would be `gitlab_notification_toggle` elsewhere.
 
-We'll name our experiment `subscription_cancellation`. It's important to understand this name may be prefixed based on your configuration, so if you've set `config.name_prefix = "gitlab"` it would be `gitlab_subscription_cancellation`.   
+When you implement an experiment you'll need to provide a name, and a context. The name can show up in tracking calls, and potentially other aspects. The context determines the variant assigned, and should be consistent between calls. We'll discuss migrating context in later examples.
 
-When you implement and run an experiment you'll need to provide the name you've given it, and a context hash. The context hash is used to determine which variant to provide, and is expected to be consistent between calls to your experiment, so a context hash key can be generated and cached -- which is used to consistently render the same experience given the same context.
+A context "key" represents the unique id of a context. It allows us to give the same experience between different calls to the experiment and can be used in caching.
+
+Now in our experiment we're going to render one of two views: the control will be our current view, and the candidate will be the new toggle button with a confirmation flow.
 
-In our experiment we're going to render one of two views. Control will be our current one, and candidate will be the new view with a cancel button and javascript confirm call.
- 
 ```ruby
 class SubscriptionsController < ApplicationController
-  include Gitlab::Experiment::Dsl
-
   def show
-    experiment(:subscription_cancellation, user_id: user.id) do |e|
-      e.use { render_toggle_button } # control
-      e.try { render_cancel_button } # candidate
+    experiment(:notification_toggle, actor: user) do |e|
+      e.use { render_toggle } # control
+      e.try { render_button } # candidate
     end
   end
 end
 ```
 
-You can also provide different variant names if you want to. In our case we might want to include the confirmation dialog step or not, to see which one behaves better within our wider experiment.  
+You can define the experiment using simple control/candidate paths, or provide named variants.
+
+Handling multi-variant experiments is up to the configuration you provide around resolving variants. But in our example we may want to try with and without the confirmation. We can run any number of variations in our experiments this way.
 
 ```ruby
-experiment(:subscription_cancellation, user_id: user.id) do |e|
-  e.use { render_toggle_button } # control
-  e.try(:variant_one) { render_cancel_button(confirmation: true) }
-  e.try(:variant_two) { render_cancel_button(confirmation: false) }
+experiment(:notification_toggle, actor: user) do |e|
+  e.use { render_toggle } # control
+  e.try(:variant_one) { render_button(confirmation: true) }
+  e.try(:variant_two) { render_button(confirmation: false) }
 end
 ```
 
-Later, and elsewhere in code you can use the same `experiment` call to track events on the experiment. You can use this consistent event concept to build out funnels from the data that's being tracked. The important detail here is to use the same context between your calls to `experiment`. If the context is the same, we're able to consistently track the event in a way that associates it to which variant is being presented.
+Understanding how an experiment can change behavior is important in evaluating its performance.
+
+To this end, we track events that are important by calling the same experiment elsewhere in code. By using the same context, you'll have consistent behavior and the ability to track events to it.
 
 ```ruby
-exp = experiment(:subscription_cancellation, user_id: user.id)
-exp.track('clicked_button')
+experiment(:notification_toggle, actor: user).track(:clicked_button)
 ```
 
 <details>
@@ -76,14 +85,14 @@ exp.track('clicked_button')
 ### Class level interface using `.run`
 
 ```ruby
-exp = Gitlab::Experiment.run(:subscription_cancellation, user_id: user.id) do |e|
-  # Context may be passed in the block, but must be finalized before calling 
+exp = Gitlab::Experiment.run(:notification_toggle, actor: user) do |e|
+  # Context may be passed in the block, but must be finalized before calling
   # run or track.
-  e.context(project_id: project.id) # add the project id to the context
-  e.variant(:candidate) # always run the candidate
+  e.context(project: project) # add the project to the context
+
   # Define the control and candidate variant.
-  e.use { toggle_button_interface } # control
-  e.try { cancel_button_interface } # candidate
+  e.use { render_toggle } # control
+  e.try { render_button } # candidate
 end
 
 # Track an event on the experiment we've defined.
@@ -93,16 +102,16 @@ exp.track(:clicked_button)
 ### Instance level interface
 
 ```ruby
-exp = Gitlab::Experiment.new(:subscription_cancellation, user_id: user.id)
-# Context may be passed in the block, but must be finalized before calling 
-# run or track.
-exp.context(project_id: project.id) # add the project id to the context
+exp = Gitlab::Experiment.new(:notification_toggle, actor: user)
+# Additional context may be provided to the instance (exp) but must be
+# finalized before calling run or track.
+exp.context(project: project) # add the project id to the context
 
 # Define the control and candidate variant.
-exp.use { toggle_button_interface } # control
-exp.try { cancel_button_interface } # candidate
+exp.use { render_toggle } # control
+exp.try { render_button } # candidate
 
-# Run the experiment -- returning the result.
+# Run the experiment, returning the result.
 exp.run
 
 # Track an event on the experiment we've defined.
@@ -112,25 +121,25 @@ exp.track(:clicked_button)
 </details>
 
 <details>
-  <summary>You can define use custom classes...</summary>
+  <summary>You can define and use custom classes...</summary>
 
 ### Custom class
 
 ```ruby
-class CancellationExperiment < Gitlab::Experiment
+class NotificationExperiment < Gitlab::Experiment
   def initialize(variant_name = nil, **context, &block)
-    super(:subscription_cancellation, variant_name, **context, &block)
+    super(:notification_toggle, variant_name, **context, &block)
 
     # Define the control and candidate variant.
-    use { toggle_button_interface } # control
-    try { cancel_button_interface } # candidate
+    use { render_toggle } # control
+    try { render_button } # candidate
   end
 end
 
-exp = CancellationExperiment.new(user_id: user.id) do |e|
-  # Context may be passed in the block, but must be finalized before calling 
-  # run or track.
-  e.context(project_id: project.id) # add the project id to the context
+exp = NotificationExperiment.new(actor: user) do |e|
+  # Context may be provided within the block or to the instance (exp) but must
+  # be finalized before calling run or track.
+  e.context(project: project) # add the project id to the context
 end
 
 # Run the experiment -- returning the result.
@@ -143,76 +152,203 @@ exp.track(:clicked_button)
 </details>
 
 <details>
-  <summary>You can also hard specify the variant to use...</summary>
+  <summary>You can also specify the variant to use...</summary>
 
-### Specifying which variant to use
+### Specifying variant
 
-This should generally be discouraged, as it can change the experience users have during rollout, and may confuse generating reports from the tracking calls. It is possible however, and may be useful if you understand the implications.
+You can hardcode the variant if you want. It's important to know what this might do to your data during rollout, so use this with consideration.
 
 ```ruby
-experiment(:subscription_cancellation, :no_interface, user_id: user.id) do |e|
-  e.use { toggle_button_interface } # control
-  e.try { cancel_button_interface } # candidate
+experiment(:notification_toggle, :no_interface, actor: user) do |e|
+  e.use { render_toggle } # control
+  e.try { render_button } # candidate
   e.try(:no_interface) { no_interface! } # variant
 end
 ```
 
-Or you can set the variant within the block -- potentially using some unique or different segmentation strategy that you've written specifically for the experiment at hand.
+Or you can set the variant within the block. This allows using unique segmentation logic or variant resolution if you need it.
 
 ```ruby
-experiment(:subscription_cancellation, user_id: user.id) do |e|
+experiment(:notification_toggle, actor: user) do |e|
+  # Variant selection must be done before calling run or track.
   e.variant(:no_interface) # set the variant
   # ...
 end
 ```
 
+Or it can be specified in the call to run if you call it from within the block.
+
+```ruby
+experiment(:notification_toggle, actor: user) do |e|
+  # ...
+  # Variant selection can be specified when calling run.
+  e.run(:no_interface)
+end
+```
+
 </details>
 
-The `experiment` method, and the underlying `Gitlab::Experiment` instance is an implementation on top of [Scientist](https://github.com/github/scientist). Generally speaking you can use the DSL that Scientist defines, but for experiments we use `experiment` instead of `science`, and specify the variant on initialization (or via `#variant` and not in the call to `#run`. The interface is otherwise the same, even though not every aspect of Scientist makes sense for experiments.
+### Return value
+
+By default the return value is a `Gitlab::Experiment` instance. In simple cases you may want only the results of the experiment though. You can call `run` within the block to get the return value of the assigned variant.
+
+```ruby
+experiment(:notification_toggle) do |e|
+  e.use { 'A' }
+  e.try { 'B' }
+  e.run
+end # => 'A'
+```
+
+### Including the DSL
+
+By default, `Gitlab::Experiment` injects itself into the controller and view layers. This exposes the `experiment` method application wide in those layers.
+
+Some experiments may extend outside of those layers, so you may want to include it elsewhere. For instance in a mailer, service object, background job, or similar.
+
+Note: In a lot of these contexts you may not have a reference to the request (unless you pass it in, or provide access to it) which may be needed if you want to enable cookie behaviors and track that through to user conversion.
+
+```ruby
+class WelcomeMailer < ApplicationMailer
+  include Gitlab::Experiment::Dsl # include the `experiment` method
+
+  def welcome
+    @user = params[:user]
+
+    ex = experiment(:project_suggestions, actor: @user) do |e|
+      e.use { 'welcome' }
+      e.try { 'welcome_with_project_suggestions' }
+    end
+
+    mail(to: @user.email, subject: 'Welcome!', template: ex.run)
+  end
+end
+```
 
 ### Context migrations
 
-There are times when we may need to change something that we're providing in the context while an experiment is running, or even add or remove contexts. We make this possible by passing the migration data to the experiment.
+There are times when we need to change context while an experiment is running. We make this possible by passing the migration data to the experiment.
+
+Take for instance, that you might be using `version: 1` in your context currently. To migrate this to `version: 2`, provide the portion of the context you wish to change using a `migrated_with` option.
+
+In providing the context migration data, we can resolve an experience and its events all the way back. This can also help in keeping our cache relevant.
+
+```ruby
+# Migrate just the `:version` portion of the previous context, `{ actor: project, version: 1 }`:
+experiment(:my_experiment, actor: project, version: 2, migrated_with: { version: 1 })
+```
+
+You can add or remove context by providing a `migrated_from` option. This approach expects a full context replacement -- i.e. what it was before you added or removed the new context key.
 
-Take for instance, that you might be using `version: 1` in your context currently. If you want to migrate this to `version: 2`, you just need to provide the context that you want to change using a `migrated_with` option. In doing this, a given experience (variant rendered/events tracked) can be resolved back through any number of migrations, and can be cached/resolved by using the context key value that was already already generated.
+If you wanted to introduce a `version` to your context, provide the full previous context.
 
 ```ruby
-experiment(:my_experiment, user_id: 42, version: 2, migrated_with: { version: 1 })
+# Migrate the full context from `{ actor: project }` to `{ actor: project, version: 1 }`:
+experiment(:my_experiment, actor: project, version: 1, migrated_from: { actor: project })
 ```
 
-If you're adding or removing a new a value from the context, you'll need to use `migrated_from`, which expects a full context replacement -- e.g. what it was before you added or removed the new context key. For instance, if you wanted to introduce the `version: 1` concept to your context, you would need to use something like the following. 
- 
+This can impact an experience if you:
+
+1. haven't implemented the concept of migrations in your variant resolver
+1. haven't enabled a reasonable caching mechanism
+
+### When there isn't an actor (cookie fallback)
+
+When there isn't an identifying key in the context (this is `actor` by default), we fall back to cookies to provide a consistent experience for the client viewing them.
+
+Once we assign a certain variant to a context, we need to always provide the same experience. We achieve this by setting a cookie for the experiment in question, but only when needed.
+
+This cookie is a temporary, randomized uuid and isn't associated with a user. When we can finally provide an actor, the context is auto migrated from the cookie to that actor.
+
+To read and write cookies, we provide the `request` from within the controller and views. The cookie migration will happen automatically if the experiment is within those layers.
+
+You'll need to provide the `request` as an option to the experiment if it's outside of the controller and views.
+
 ```ruby
-experiment(:my_experiment, user_id: 42, version: 1, migrated_from: { user_id: 42 })
+experiment(:my_experiment, actor: user, request: request)
 ```
 
-It's important to understand that this can bucket a user in a new experience (depending on the logic in determining in the variant), so you should investigate how this might impact your experiment before using it and that your experiment is implemented in a way that supports migrations.
+The cookie isn't set if the `actor` key isn't present at all in the context. Meaning that when no `actor` key is provided, the cookie will not be set.
+
+```ruby
+# actor is not present, so no cookie is set
+experiment(:my_experiment, project: project)
+
+# actor is present and is nil, so the cookie is set and used
+experiment(:my_experiment, actor: nil, project: project)
+
+# actor is present and set to a value, so no cookie is set
+experiment(:my_experiment, actor: user, project: project)
+```
+
+For edge cases, you can pass the cookie through by assigning it yourself -- e.g. `actor: request.cookie_jar.signed['my_experiment_actor']`. The cookie name is the full experiment name (including any configured prefix) with `_actor` appended -- e.g. `gitlab_notification_toggle_actor` for the `:notification_toggle` experiment key with a configured prefix of `gitlab`.
+
+## Configuration
+
+This gem needs to be configured before being used in a meaningful way.
+
+The default configuration will always render the control, so it's important to configure your own logic for resolving variants.
+
+Yes, the most important aspect of the gem -- that of determining which variant to render and when -- is up to you. Consider using [Unleash](https://github.com/Unleash/unleash-client-ruby) or [Flipper](https://github.com/jnunemaker/flipper) for this.
+
+```ruby
+Gitlab::Experiment.configure do |config|
+  # The block here is evaluated within the scope of the experiment instance,
+  # which is why we are able to access things like name and context.
+  config.variant_resolver = lambda do |requested_variant|
+    # Return the requested variant if a specific one has been provided in code.
+    return requested_variant unless requested_variant.nil?
+
+    # Ask Unleash to determine the variant, given the context we've built,
+    # using the control as the fallback.
+    fallback = Unleash::Variant.new(name: 'control', enabled: true)
+    UNLEASH.get_variant(name, context.value, fallback)
+  end
+end
+```
 
-### When there isn't a user (cookies)
+More examples for configuration are available in the provided [rails initializer](lib/generators/gitlab/experiment/install/templates/initializer.rb).
 
-When there isn't a user we typically have to fall back to another concept to provide a consistent experience. What this means, is that once we assign someone a certain variant, we want to always give them the same experience, and we do this by setting a cookie for the experiment we're currently checking on. This cookie value is a random uuid, which we auto migrate to a user_id when our experiment context is finally provided that information. This means that you really only need to provide the request as an option if you're wanting to have an experiment that flows from not-signed-in (or not registered) users to eventually signed-in users.
+### Client layer / JavaScript
 
-This is considered a temporary cookie value, and isn't used for tracking purposes other than to give a given "user" (in this case it's actually the browser), a consistent experience after we've assigned one.
+This library doesn't attempt to provide any logic for the client layer.
 
-To read and write cookies, we allow for passing the `request` as an option. This allows us to read, write, and even clean up a cookie when appropriate. We've provided this by default in the ActionController interface though, so this is primarily useful if you're trying to have an experiment span controller and model layers. 
+Instead it allows you to do this yourself in configuration. Using [Gon](https://github.com/gazay/gon) to publish your experiment information to the client layer is pretty simple.
 
 ```ruby
-experiment(:subscription_cancellation, user_id: user.id, request: request) do |e|
-  e.use { render_toggle_button } # control
-  e.try { render_cancel_button } # candidate
+Gitlab::Experiment.configure do |config|
+  config.publishing_behavior = lambda do |_result|
+    # Push the experiment knowledge into the front end. The signature contains
+    # the context key, and the variant that has been determined.
+    Gon.push({ experiment: { name => signature } }, true)
+  end
 end
-``` 
+```
 
-If needed, for edge cases (like in a background job), you can manually pass in the cookie value. Passing it in as `user_id: request.cookie_jar.signed['subscription_cancellation_id']`. The cookie name is the experiment name (prefixed if configured to be) with `_id` appended.
+In the client you can now access `window.gon.experiment.notificationToggle`.
 
-## Configuration
+### Caching
 
-The gem is meant to be configured before being used. The default configuration will always render the control behavior, so it's important to implement your own logic for this or you will always get the control of an experiment.
+Caching can be enabled in configuration, and is implemented towards the `Rails.cache` / `ActiveSupport::Cache::Store` interface. When you enable caching, any variant resolution will be cached. Migrating the cache through context migrations is handled automatically, and this helps ensure an experiment experience remains consistent.
 
-The most important aspect of the gem, determining which variant to render and when, is up to you, and you may want to consider using [Unleash](https://github.com/Unleash/unleash-client-ruby) (which has the concept of multi-variants built in), or [Flipper](https://github.com/jnunemaker/flipper) in helping with this.
+It's important to understand that using caching can drastically change or override your rollout strategy logic.
 
-Examples for configuration are available in the provided install generator, or in the source code configuration.rb file itself.
+```ruby
+Gitlab::Experiment.configure do |config|
+  config.cache = Rails.cache
+end
+```
 
 ## Tracking, anonymity and GDPR
 
-We intentionally don't, and shouldn't, track things like user ids on our experiments. What we can and do track is what we consider an "experiment experience" key. This key is generated from the context we pass to the experiment and has the concept of migrating through different versions of context. If we consistently pass the same context to an experiment, we're able to consistently track events generated in that experience. A context can contain things like user, or project -- so, if you only included a user in the context that user would get the same experience across all projects they view, but if you include the currently viewed project in the context the user would potentially have a different experience on each of their projects. Each can be desirable given the objectives of the experiment.
+We generally try not to track things like user identifying values in our experimentation. What we can and do track is the "experiment experience" (a.k.a. the context key).
+
+We generate this key from the context passed to the experiment. This allows creating funnels without exposing any user information.
+
+This library attempts to be non-user-centric, in that a context can contain things like a user or a project.
+
+If you only include a user, that user would get the same experience across every project they view. If you only include the project, every user who views that project would get the same experience.
+
+Each of these approaches could be desirable given the objectives of your experiment.
+
+### Make code not war

data/lib/generators/gitlab/experiment/USAGE

@@ -0,0 +1,17 @@
+Description:
+    Stubs out a new experiment and its variants. Pass the experiment name,
+    either CamelCased or under_scored, and a list of variants as arguments.
+
+    To create an experiment within a module, specify the experiment name as a
+    path like 'parent_module/experiment_name'.
+
+    This generates an experiment class in app/experiments and invokes feature
+    flag, and test framework generators.
+
+Example:
+    `rails generate gitlab:experiment NullHypothesis control candidate alt_variant`
+
+    NullHypothesis experiment with default variants.
+        Experiment:   app/experiments/null_hypothesis_experiment.rb
+        Feature Flag: config/feature_flags/experiment/null_hypothesis.yaml
+        Test:         test/experiments/null_hypothesis_experiment_test.rb