gitlab-experiment 0.2.3 → 0.2.4

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 1a35b23202fa542fa548093c65ec4580e764fdbda2549291a54bfa3ed98cec96
-  data.tar.gz: 05f4c1ed8c5761ab03ed1e9d763d6e612f661cf9ee8736e46fb0e48b947ed52d
+  metadata.gz: a6284349854da93da979695a6493df95c0247616e10d5adc047d56d5dbdbf324
+  data.tar.gz: 1e4456945313fce9fbf03250299e3a0eb1b406035e4cce0d0559eee1fb3cfe53
 SHA512:
-  metadata.gz: 9505898d8870749dbccab8fa98974948895a9898838aecc622e9f3243624a889076b248f6c0698ad8d44c63726447cc1b46638158614bb008015a82ce42a8d19
-  data.tar.gz: 3484817693dbaf9254d16083c6fbb745fbe47adc7868bf924bd418d8f67735fd26ac00f7f486aecc4f9ec84647f58e7bfcb0ff15199370b2e782d68cf2a0b2a2
+  metadata.gz: '08d5c04e817c2679d115a845df588a4b81358e1b7ac7ddc8282a4cdaa2a878f6a4111b318fa3155671fdd05c0cb950ab076210518142e5f8a04db89e7b164104'
+  data.tar.gz: 118de1ae6b9995edee4058a1b756030ee6659199bf19a29cf2c1446ddb87a1a8ff29f479cf9ff20f7af47ff7706bbc97256929ed800f803fc381450bcedbf82c

data/README.md

@@ -1,5 +1,6 @@
-GitLab Experiment
-=================
+# GitLab Experiment
+
+<img alt="experiment" src="/uploads/60990b2dbf4c0406bbf8b7f998de2dea/experiment.png" align="right" width="40%">
 
 Here at GitLab, we run experiments as A/B/n tests and review the data the experiment generates. From that data, we determine the best performing variant and promote it as the new default code path. Or revert back to the control if no variant outperformed it.
 
@@ -13,25 +14,7 @@ When we discuss the behavior of this gem, we'll use terms like experiment, conte
 - `candidate` defines that there's one experimental code path.
 - `variant(s)` is used when more than one experimental code path exists.
 
-Candidate and variant are the same concept, but simplify how we speak about experimental paths.
-
-## Opinionated elevator pitch
-
-The last time I travelled I went through a process at the airport when I arrived. My passport was checked, I was asked a series of questions about my visit, and was ushered to a digital kiosk where I was prompted with a few more questions on a touch screen. 
-
-It was Iceland, and for the record, it's a beautiful place and you should visit if you haven't yet.
-
-Anyway, at various stages of this process travellers are presented with a physical emoji button interface, and are encouraged to rate their satisfaction within the various stage.
-
-Running an experiment could be to change some aspect of the process for only some travelers, let's say by routing them down a different hallway. At various stages of both hallways we still have the emoji interfaces that can be happily tapped or angrily jabbed in their passing.
-
-After a while we can compare the results and can evaluate which hallway had the better overall experience, based on the ratings provided at the various stages.
-
-This library is about keeping track of which passports we send down which hallway, so we can consistently route them down the same hallway, and to know which hallway they're rating when they do.
-
-In this model we don’t need to know anything about the passport holder unless we decide to "ask", as we determine which hallway to send them down initially.
-
-This library doesn't provide a system of linking passports back to their passport holders, but it doesn't explicitly make doing so impossible. Doing so is often not a relevant detail on well defined and well executed experiments.
+Candidate and variant are the same concept, but simplify how we speak about experimental paths.<br clear="all">
 
 ## Installation
 
@@ -51,11 +34,11 @@ $ rails generate gitlab-experiment:install
 
 For the sake of an example let's make one up. Let's run an experiment on what we render for disabling desktop notifications.
 
-In our control (current world) we show a simple toggle interface that reads "Notifications". In our experiment we want a "Turn on/off desktop notifications" button with a confirmation.
+In our control (current world) we show a simple toggle interface labeled, "Notifications." In our experiment we want a "Turn on/off desktop notifications" button with a confirmation.
 
-The behavior will be the same, but the interface will be different and may involve more or less steps.
- 
-This makes the action more clear and will help the user in making a choice about if that's what they want to do. Or that's what we're going to try to find out.
+The behavior will be the same, but the interface will be different and may involve more or fewer steps.
+
+Our hypothesis is that this will make the action more clear and will help the user in making a choice about if that's what they really want to do.
 
 We'll name our experiment `notification_toggle`. This name is prefixed based on configuration. If you've set `config.name_prefix = 'gitlab'`, the experiment name would be `gitlab_notification_toggle` elsewhere.
 
@@ -63,8 +46,8 @@ When you implement an experiment you'll need to provide a name, and a context. T
 
 A context "key" represents the unique id of a context. It allows us to give the same experience between different calls to the experiment and can be used in caching.
 
-Now in our experiment we're going to render one of two views. Control will be our current view, and candidate will be the new toggle button with a confirmation flow.
- 
+Now in our experiment we're going to render one of two views: the control will be our current view, and the candidate will be the new toggle button with a confirmation flow.
+
 ```ruby
 class SubscriptionsController < ApplicationController
   def show
@@ -120,15 +103,15 @@ exp.track(:clicked_button)
 
 ```ruby
 exp = Gitlab::Experiment.new(:notification_toggle, user_id: user.id)
-# Context may be passed in the block, but must be finalized before calling
-# run or track.
+# Additional context may be provided to the instance (exp) but must be
+# finalized before calling run or track.
 exp.context(project_id: project.id) # add the project id to the context
 
 # Define the control and candidate variant.
 exp.use { render_toggle } # control
 exp.try { render_button } # candidate
 
-# Run the experiment -- returning the result.
+# Run the experiment, returning the result.
 exp.run
 
 # Track an event on the experiment we've defined.
@@ -154,8 +137,8 @@ class NotificationExperiment < Gitlab::Experiment
 end
 
 exp = NotificationExperiment.new(user_id: user.id) do |e|
-  # Context may be passed in the block, but must be finalized before calling
-  # run or track.
+  # Context may be provided within the block or to the instance (exp) but must
+  # be finalized before calling run or track.
   e.context(project_id: project.id) # add the project id to the context
 end
 
@@ -187,11 +170,22 @@ Or you can set the variant within the block. This allows using unique segmentati
 
 ```ruby
 experiment(:notification_toggle, user_id: user.id) do |e|
+  # Variant selection must be done before calling run or track.
   e.variant(:no_interface) # set the variant
   # ...
 end
 ```
 
+Or it can be specified in the call to run if you call it from within the block.
+
+```ruby
+experiment(:notification_toggle, user_id: user.id) do |e|
+  # ...
+  # Variant selection can be specified when calling run.
+  e.run(:no_interface)
+end
+```
+
 </details>
 
 ### Return value
@@ -235,25 +229,27 @@ end
 
 There are times when we need to change context while an experiment is running. We make this possible by passing the migration data to the experiment.
 
-Take for instance, that you might be using `version: 1` in your context currently. To migrate this `version: 2`, provide the context to change using a `migrated_with` option.
+Take for instance, that you might be using `version: 1` in your context currently. To migrate this to `version: 2`, provide the portion of the context you wish to change using a `migrated_with` option.
 
 In providing the context migration data, we can resolve an experience and its events all the way back. This can also help in keeping our cache relevant.
 
 ```ruby
+# Migrate just the `:version` portion of the previous context, `{ user_id: 42, version: 1 }`:
 experiment(:my_experiment, user_id: 42, version: 2, migrated_with: { version: 1 })
 ```
 
-You can add or remove context by providing a `migrated_from` option. This approach expects a full context replacement -- e.g. what it was before you added or removed the new context key.
+You can add or remove context by providing a `migrated_from` option. This approach expects a full context replacement -- i.e. what it was before you added or removed the new context key.
 
 If you wanted to introduce a `version` to your context, provide the full previous context.
- 
+
 ```ruby
+# Migrate the full context from `{ user_id: 42 }` to `{ user_id: 42, version: 1 }`:
 experiment(:my_experiment, user_id: 42, version: 1, migrated_from: { user_id: 42 })
 ```
 
-This can impact an experience if you haven't:
+This can impact an experience if you:
 
-1. implemented the concept of migrations in your variant resolver
+1. haven't implemented the concept of migrations in your variant resolver
 1. haven't enabled a reasonable caching mechanism
 
 ### When there isn't a user (cookies)
@@ -272,20 +268,33 @@ You'll need to provide the `request` as an option to the experiment if it's outs
 experiment(:my_experiment, user_id: user&.id, request: request)
 ```
 
-The cookie isn't set if the identifying key isn't present in the context. In this case, if there was no `user_id` key provided, the cookie wouldn't be set.
+The cookie isn't set if the identifying key isn't present at all in the context. Using the default identifying key, when there is no `user_id` key provided, the cookie will not be set.
+
+```ruby
+# no user_id context key is present, so no cookie is set
+experiment(:my_experiment, project_id: @project.id)
 
-For edge cases, you can pass the cookie through by assigning it yourself -- e.g. `user_id: request.cookie_jar.signed['my_experiment_id']`. The cookie name is the experiment name (prefixed if configured) with `_id` appended.
+# user_id context key is present, but is set to nil, so the cookie is set & used instead
+experiment(:my_experiment, user_id: nil, project_id: @project.id)
+
+# user_id context key is present and set to a value, so no cookie is set
+experiment(:my_experiment, user_id: @user.id, project_id: @project.id)
+```
+
+For edge cases, you can pass the cookie through by assigning it yourself -- e.g. `user_id: request.cookie_jar.signed['my_experiment_id']`. The cookie name is the full experiment name (including any configured prefix) with `_id` appended -- e.g. `gitlab_notification_toggle_id` for the `:notification_toggle` experiment key with a configured prefix of `gitlab`.
 
 ## Configuration
 
 This gem needs to be configured before being used in a meaningful way.
 
-The default configuration will always render the control. So it's important to configure your own logic for resolving variants.
+The default configuration will always render the control, so it's important to configure your own logic for resolving variants.
 
-Yes, the most important aspect of the gem, determining which variant to render and when, is up to you. Consider using [Unleash](https://github.com/Unleash/unleash-client-ruby) or [Flipper](https://github.com/jnunemaker/flipper) for this.
+Yes, the most important aspect of the gem -- that of determining which variant to render and when -- is up to you. Consider using [Unleash](https://github.com/Unleash/unleash-client-ruby) or [Flipper](https://github.com/jnunemaker/flipper) for this.
 
 ```ruby
 Gitlab::Experiment.configure do |config|
+  # The block here is evaluated within the scope of the experiment instance,
+  # which is why we are able to access things like name and context.
   config.variant_resolver = lambda do |requested_variant|
     # Return the requested variant if a specific one has been provided in code.
     return requested_variant unless requested_variant.nil?
@@ -304,14 +313,14 @@ More examples for configuration are available in the provided [rails initializer
 
 This library doesn't attempt to provide any logic for the client layer.
 
-Instead it allows you to do this yourself in configuration. Using Gon to publish your experiment information to the client layer is pretty simple.
+Instead it allows you to do this yourself in configuration. Using [Gon](https://github.com/gazay/gon) to publish your experiment information to the client layer is pretty simple.
 
 ```ruby
 Gitlab::Experiment.configure do |config|
   config.publishing_behavior = lambda do |_result|
     # Push the experiment knowledge into the front end. The signature contains
     # the context key, and the variant that has been determined.
-    Gon.push(experiment: { name => signature })
+    Gon.push({ experiment: { name => signature } }, true)
   end
 end
 ```
@@ -322,6 +331,8 @@ In the client you can now access `window.gon.experiment.notificationToggle`.
 
 Caching can be enabled in configuration, and is implemented towards the `Rails.cache` / `ActiveSupport::Cache::Store` interface. When you enable caching, any variant resolution will be cached. Migrating the cache through context migrations is handled automatically, and this helps ensure an experiment experience remains consistent.
 
+It's important to understand that using caching can drastically change or override your rollout strategy logic.
+
 ```ruby
 Gitlab::Experiment.configure do |config|
   config.cache = Rails.cache
@@ -330,11 +341,11 @@ end
 
 ## Tracking, anonymity and GDPR
 
-We generally try not to track things like user identifying values in our experimentation. What we can and do track is the "experiment experience" (a.k.a the context key).
+We generally try not to track things like user identifying values in our experimentation. What we can and do track is the "experiment experience" (a.k.a. the context key).
 
 We generate this key from the context passed to the experiment. This allows creating funnels without exposing any user information.
 
-This library attempts to be non user centric, in that a context can contain things like a user, or a project.
+This library attempts to be non-user-centric, in that a context can contain things like a user or a project.
 
 If you only include a user, that user would get the same experience across every project they view. If you only include the project, every user who views that project would get the same experience.
 

data/lib/generators/gitlab_experiment/install/templates/initializer.rb

@@ -26,7 +26,7 @@ Gitlab::Experiment.configure do |config|
     #
     # requested_variant || variant_names.first || 'control'
 
-    # Using unleash to determine the variant:
+    # Using Unleash to determine the variant:
     #
     # fallback = Unleash::Variant.new(name: requested_variant || 'control', enabled: true)
     # Unleash.get_variant(name, context.value, fallback)
@@ -40,19 +40,17 @@ Gitlab::Experiment.configure do |config|
   # Tracking behavior can be implemented to link an event to an experiment.
   #
   # Similar to the variant_resolver, this is called within the scope of the
-  # experiment instance and so can access any methods on the experiment.
-  #
-  #
+  # experiment instance and so can access any methods on the experiment,
+  # such as name and signature.
   config.tracking_behavior = lambda do |event, args|
     # An example of using a generic logger to track events:
     config.logger.info "Gitlab::Experiment[#{name}] #{event}: #{args.merge(signature: signature)}"
 
-    # Using something like snowplow to track events:
+    # Using something like snowplow to track events (in gitlab):
     #
     # Gitlab::Tracking.event(name, event, **args.merge(
     #   context: (args[:context] || []) << SnowplowTracker::SelfDescribingJson.new(
-    #     'iglu:com.gitlab/gitlab_experiment/jsonschema/0-2-0',
-    #     signature: signature
+    #     'iglu:com.gitlab/gitlab_experiment/jsonschema/0-2-0', signature
     #   )
     # ))
   end
@@ -69,7 +67,7 @@ Gitlab::Experiment.configure do |config|
     # Push the experiment knowledge into the front end. The signature contains
     # the context key, and the variant that has been determined.
     #
-    # Gon.push(experiment: { name => signature })
+    # Gon.push({ experiment: { name => signature } }, true)
 
     # Log using our logging system, so the result (which can be large) can be
     # reviewed later if we want to.

data/lib/gitlab/experiment.rb

@@ -57,7 +57,9 @@ module Gitlab
       result.respond_to?(:name) ? result : Variant.new(name: result.to_s)
     end
 
-    def run
+    def run(variant_name = nil)
+      @variant_name = variant_name unless variant_name.nil?
+
       @result ||= super(cache { variant.name })
     end
 

data/lib/gitlab/experiment/context.rb

@@ -65,6 +65,7 @@ module Gitlab
 
       def resolve_cookie(jar, hash, key, cookie = nil)
         return if cookie.blank? && hash[key].blank?
+        return hash if cookie.blank?
         return hash.merge(key => cookie) if hash[key].blank?
 
         @migrations_with << { key => cookie }

data/lib/gitlab/experiment/version.rb

@@ -2,6 +2,6 @@
 
 module Gitlab
   class Experiment
-    VERSION = '0.2.3'
+    VERSION = '0.2.4'
   end
 end

metadata

@@ -1,35 +1,35 @@
 --- !ruby/object:Gem::Specification
 name: gitlab-experiment
 version: !ruby/object:Gem::Version
-  version: 0.2.3
+  version: 0.2.4
 platform: ruby
 authors:
 - GitLab
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2020-09-27 00:00:00.000000000 Z
+date: 2020-10-06 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: scientist
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: 1.5.0
     - - "~>"
       - !ruby/object:Gem::Version
         version: '1.5'
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 1.5.0
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: 1.5.0
     - - "~>"
       - !ruby/object:Gem::Version
         version: '1.5'
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 1.5.0
 description: 
 email:
 - gitlab_rubygems@gitlab.com
@@ -69,7 +69,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.0.3
+rubygems_version: 3.1.4
 signing_key: 
 specification_version: 4
 summary: GitLab experiment library built on top of scientist.