gitlab-experiment 0.2.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: fae9519317810c50decdff810249899d5c1855881544a20a31a2219cf9bda715
+  data.tar.gz: 19987d9c24486e99424aa6cfcade34d3123dbe1fc4a6afb8dc6c4c6ad4f5a99b
+SHA512:
+  metadata.gz: afe30984c7b7f96bb4ad1122c5a4f5ff48a45f8c7cfca5b1d2d04368ba3eef25c84776e3768588cde6cd717ee918b25e8158746ea9ea3e9bb20fe9ef39d20377
+  data.tar.gz: 2492d43cbc76cfa20c1b785033fdc635ca75874d1052270c950a86acff17d04011c406bf1879bc5f21b06fb604c5ed259c149aecbea33efe2adae59ab12f6164

data/LICENSE.txt

@@ -0,0 +1,25 @@
+Copyright (c) 2020-2022 GitLab B.V.
+
+With regard to the GitLab Software:
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.
+
+For all third party components incorporated into the GitLab Software, those
+components are licensed under the original license provided by the owner of the
+applicable component.

data/README.md

@@ -0,0 +1,182 @@
+# Experiment Guide
+
+Experiments can be conducted by any GitLab team, but are most often conducted by the teams from the [Growth Sub-department](https://about.gitlab.com/handbook/engineering/development/growth/).
+
+Experiments are run as A/B/n tests and are evaluated by the data the experiment generates. The team reviews the data, determines the variant that performed most effectively, and promotes that variant as the new default code path (or reverts back to the control). In all instances, an experiment will be cleaned up after it has generated enough data to determine performance and be considered resolved.
+
+## Process and tracking issue
+
+Each experiment should have a related [Experiment tracking issue](https://gitlab.com/gitlab-org/gitlab/-/issues/new?issuable_template=Experiment%20tracking) created, which is intended to track the experiment from deployment and rollout through to resolution and cleanup.
+
+At the time an experiment is rolled out, the due date of the tracking issue should be specified. The timeline depends on the experiment and can be up to several weeks in the future.
+
+With experiment resolution, the outcome of the experiment should be posted to the tracking issue with the reasoning for the decision. Any and all non-relevant experiment code should be removed and review concerns should be addressed during cleanup. After cleanup, the tracking issue can be closed.
+
+## Implementing an experiment
+
+For the sake of our example, let's say we want to run an experiment for how to cancel a subscription. In our control (current world) we show a toggle that reads "Auto-renew", and in our experiment candidate we want to show a "Cancel subscription" button with a confirmation. Ultimately the behavior is the same, but the interface will be considerably different.
+
+### Defining the feature flag
+
+Let's name our experiment `subscription_cancellation`. It's important to understand how this name is prefixed as `growth_experiment_subscription_cancellation` in our Unleash feature detection and in our Snowplow tracking calls. We prefix our experiments so we can consistently identify them as experiments and clean them up over time.
+
+This means that you'll need to go to the [Feature Flags interface](https://gitlab.com/gitlab-org/gitlab/-/feature_flags) interface (for the project you're working on) and add a `growth_experiment_subscription_cancellation` feature flag and define which environment(s) it should be rolled out on for initial deployment, for instance, `gitlab-staging` or `customers-staging`.
+
+You can include yourself and others in a list and assign that list using the `User List` rollout strategy, or you can add your user id to the experiment manually. You can use this to include or exclude yourself from an experiment for verification before rolling it out to others.
+
+### Implementation
+
+When you implement an experiment in code you'll need to provide the name that you've given it in the feature flags interface, and a context -- which usually will include something like a user / user id, but may also include several other aspects.
+
+```ruby
+class SubscriptionsController < ApplicationController
+  include Gitlab::GrowthExperiment::Interface
+
+  def show
+    experiment(:subscription_cancellation, user_id: user.id) do |e|
+      e.use { render_toggle_button } # control
+      e.try { render_cancel_button } # candidate
+    end
+  end
+end
+```
+
+You can also provide different variants for the experience if you've defined variants in the Feature Flag interface (not yet available).
+
+```ruby
+experiment(:subscription_cancellation, user_id: user.id) do |e|
+  e.use { render_toggle_button } # control
+  e.try(:variant_one) { render_cancel_button(confirmation: true) }
+  e.try(:variant_two) { render_cancel_button(confirmation: false) }
+end
+```
+
+Later, and elsewhere in code you can use the same `experiment` call to track events on the experiment. The important detail here is to use the same context between your calls to `experiment`. If the context is the same, we're able to consistently track the event in a way that associates it to which variant is being presented -- which may be based on the user we're presenting it to.
+
+```ruby
+exp = experiment(:subscription_cancellation, user_id: user.id)
+exp.track('clicked_button')
+```
+
+<details>
+  <summary>You can use the more low level class or instance interfaces...</summary>
+
+### Class level interface using `.run`
+
+```ruby
+exp = Gitlab::GrowthExperiment.run(:subscription_cancellation, user_id: user.id) do |e|
+  # context can be passed to `experiment`, `.run`, `new`, or after the fact like here.
+  # context must be added before `#run` or `#track` calls.
+  e.context(project_id: project.id)
+
+  e.use { toggle_button_interface } # control
+  e.try { cancel_button_interface } # candidate
+end
+
+# track an event on the experiment we've defined.
+exp.track(:clicked_button)
+```
+
+While `Gitlab::GrowthExperiment.run` is what we document, you can also use `Gitlab::GrowthExperiment.experiment`.
+
+### Instance level interface
+
+```ruby
+exp = Gitlab::GrowthExperiment.new(:subscription_cancellation, user_id: user.id)
+# context can be passed to `.new`, or after the fact like here.
+# context must be added before `#run` or `#track` calls.
+exp.context(project_id: project.id)
+exp.use { toggle_button_interface } # control
+exp.try { cancel_button_interface } # candidate
+exp.run
+
+# track an event on the experiment we've defined.
+exp.track(:clicked_button)
+```
+
+</details>
+
+<details>
+  <summary>You can define use custom classes...</summary>
+
+### Custom class
+
+```ruby
+class CancellationExperiment < Gitlab::GrowthExperiment
+  def initialize(variant_name = nil, **context, &block)
+    super(:subscription_cancellation, variant_name, **context, &block)
+  end
+end
+
+exp = CancellationExperiment.run(user_id: user.id) do |e|
+  # context can be passed to `.run`, or after the fact like here.
+  # context must be added before `#run` or `#track` calls.
+  e.context(project_id: project.id)
+
+  e.use { toggle_button_interface } # control
+  e.try { cancel_button_interface } # candidate
+end
+
+# track an event on the experiment we've defined.
+exp.track(:clicked_button)
+```
+
+</details>
+
+<details>
+  <summary>You can hard specify the variant to use...</summary>
+
+### Specifying which variant to use
+
+This should generally be discouraged, as it can change the experience users have during rollout, and may confuse generating reports from the tracking calls. It is possible however, and may be useful if you understand the implications.
+
+```ruby
+experiment(:subscription_cancellation, :no_interface, user_id: user.id) do |e|
+  e.use { toggle_button_interface } # control
+  e.try { cancel_button_interface } # candidate
+  e.try(:no_interface) { no_interface! } # variant
+end
+```
+
+Or you can set the variant within the block.
+
+```ruby
+experiment(:subscription_cancellation, user_id: user.id) do |e|
+  e.variant(:variant) # set the variant
+  # ...
+end
+```
+
+</details>
+
+The `experiment` method, and the underlying `Gitlab::GrowthExperiment` is an implementation on top of [Scientist](https://github.com/github/scientist). Generally speaking you can use the DSL that Scientist defines, but for experiments we use `experiment` instead of `science`, and specify the variant on initialization (or via `#variant` and not in the call to `#run`. The interface is otherwise the same, even though not every aspect of Scientist makes sense for experiments.
+
+### Context migrations
+
+There are times when we may need to add new values or change something that we're providing in context while an experiment is running. We make this possible by passing the `migrated_from` context key.
+
+Take for instance, that you might be using `version: 1` in your context. If you want to migrate this to `version: 2`, you just need to provide the context that you provided prior, in a `migrated_from` context key. In doing this, a given experiment experience can be resolved back through any number of migrations.
+
+```ruby
+experiment(:my_experiment, version: 2, migrated_from: { version: 1 })
+```
+
+It's important to understand that this can bucket a user in a new experience (depending on the rollout strategy being used and what is changing in the context), so you should investigate how this might impact your experiment before using it.
+
+### When there isn't a user
+
+When there isn't a user, we typically have to fall back to another concept to provide a consistent experiment experience. What this means, is that once we bucket someone in a certain bucket, we always bucket them in that bucket.
+
+We do this by using cookies.... [document more]
+
+## Tracking, anonymity and GDPR
+
+We intentionally don't, and shouldn't, track things like user ids. What we can and do track is what we consider an "experiment experience" key. This key is generated by the context we pass to the experiment implementation. If we consistently pass the same context to an experiment, we're able to consistently track events generated in that experience. A context can contain things like user, or project -- so, if you only included a user in the context that user would get the same experience across all projects they view, but if you include the currently viewed project in the context the user would potentially have a different experience on each of their projects. Each can be desirable given the objectives of the experiment.
+
+## Code quality expectations
+
+Since experimental code is inherently short lived, an intentionally stated goal is to iterate quickly to generate and evaluate performance data.
+
+This goal prioritizes iteration and resolution over code quality, which means that experiment code may not always meet our code standards guidelines, but should also not negatively impact the availability of GitLab nor contribute to bad data. Even though experiments will be deployed to a minority of users, we still expect a flawless experience for those users, therefore, good test coverage is still required.
+
+Reviewers and maintainers are encouraged to note when code doesn't meet our code standards guidelines. Please mention your concerns and include or link to them on the experiment tracking issue. The experiment author(s) are responsible for addressing these concerns when the experiment is resolved.

data/lib/gitlab/experiment.rb

@@ -0,0 +1,84 @@
+# frozen_string_literal: true
+
+require 'scientist'
+
+require 'gitlab/experiment/configuration'
+require 'gitlab/experiment/context'
+require 'gitlab/experiment/dsl'
+require 'gitlab/experiment/variant'
+require 'gitlab/experiment/version'
+
+module Gitlab
+  class Experiment
+    include Scientist::Experiment
+
+    class << self
+      def configure
+        yield Configuration
+      end
+
+      def run(name, variant_name = nil, **context, &block)
+        instance = new(name, variant_name, **context, &block)
+        return instance unless block_given?
+
+        instance.context.frozen? ? instance.run : instance.tap(&:run)
+      end
+    end
+
+    def initialize(name, variant_name = nil, **context)
+      @name = name
+      @variant_name = variant_name
+      @context = Context.new(self)
+
+      context(context)
+
+      ignore { true }
+      compare { false }
+
+      yield self if block_given?
+    end
+
+    def context(value = nil)
+      return @context if value.nil?
+
+      @context.value(value)
+      @context
+    end
+
+    def variant(value = nil)
+      @variant_name = value unless value.nil?
+      instance_exec(@variant_name, &Configuration.variant_resolver)
+    end
+
+    def run
+      @result ||= super(variant.name)
+    end
+
+    def publish(_result)
+      track(:assignment)
+    end
+
+    def track(action, **event_args)
+      instance_exec(action, event_args, &Configuration.tracking_behavior)
+    end
+
+    def name
+      [Configuration.name_prefix, @name].compact.join('_')
+    end
+
+    def variant_names
+      @variant_names = behaviors.keys.tap { |keys| keys.delete('control') }.map(&:to_sym)
+    end
+
+    def enabled?
+      true
+    end
+
+    protected
+
+    def generate_result(variant_name)
+      observation = Scientist::Observation.new(variant_name, self, &behaviors[variant_name])
+      Scientist::Result.new(self, [observation], observation)
+    end
+  end
+end

data/lib/gitlab/experiment/configuration.rb

@@ -0,0 +1,62 @@
+# frozen_string_literal: true
+
+require 'singleton'
+require 'logger'
+require 'digest'
+
+module Gitlab
+  class Experiment
+    class Configuration
+      include Singleton
+
+      # Prefix all experiment names with a given value. Use `nil` for none.
+      @name_prefix = 'gitlab_experiment'
+
+      # The logger is used to log various details of the experiments.
+      @logger = Logger.new(STDOUT)
+
+      # Logic this project uses to resolve a variant for a given experiment.
+      @variant_resolver = lambda do |requested_variant|
+        # An example of running the control, unless a variant was requested:
+        Variant.new(name: requested_variant || 'control')
+
+        # Always run candidate, unless a variant was requested, with fallback:
+        # Variant.new(name: variant_name || variant_names.first || 'control')
+
+        # Using unleash to determine the variant:
+        # fallback = Unleash::Variant.new(name: 'control', enabled: true)
+        # Unleash.get_variant(name, context.value, fallback)
+
+        # Using Flipper to determine the variant:
+        # TODO: provide example
+        # Variant.new(name: resolved)
+      end
+
+      # Tracking behavior can be implemented to link an event to an experiment.
+      @tracking_behavior = lambda do |action, event_args|
+        # An example of using a generic logger to track events:
+        (event_args[:context] ||= []) << context.signature.merge(group: variant.name)
+        Configuration.logger.info "EXPERIMENT[#{name}] #{action}: #{event_args}"
+
+        # Using snowplow to track events:
+        # (event_args[:context] ||= []) << SnowplowTracker::SelfDescribingJson.new(
+        #   'iglu:com.gitlab/gitlab_experiment/jsonschema/1-0-0',
+        #   experiment: context.signature.merge(group: variant.name)
+        # )
+        #
+        # Tracking.event(name, action, **event_args)
+      end
+
+      # Algorithm that consistently generates a hash key for a given hash map.
+      @context_hash_strategy = lambda do |context|
+        values = context.values.map { |v| (v.respond_to?(:to_global_id) ? v.to_global_id : v).to_s }
+        Digest::MD5.hexdigest((context.keys + values).join('|'))
+      end
+
+      class << self
+        attr_accessor :name_prefix, :logger
+        attr_accessor :variant_resolver, :tracking_behavior, :context_hash_strategy
+      end
+    end
+  end
+end

data/lib/gitlab/experiment/context.rb

@@ -0,0 +1,70 @@
+# frozen_string_literal: true
+
+module Gitlab
+  class Experiment
+    class Context
+      def initialize(experiment)
+        @experiment = experiment
+        @value = {}
+        @migrations_from = []
+        @migrations_with = []
+      end
+
+      def value(value = nil)
+        return @value if value.nil?
+
+        value = value.dup # dup so we don't mutate
+        @signature = nil # clear memoized signature
+
+        @migrations_from << value.delete(:migrated_from) if value[:migrated_from]
+        @migrations_with << value.delete(:migrated_with) if value[:migrated_with]
+        @value.merge!(migrate_cookie_to_user_id(value))
+      end
+
+      def freeze
+        signature # ensure we're done before being frozen
+        super
+      end
+
+      def signature
+        @signature ||= { key: key_for(@value), migration_keys: migration_keys }.compact
+      end
+
+      private
+
+      def migrate_cookie_to_user_id(value)
+        return value unless (request = value.delete(:request))
+        return value if request.headers['DNT'].to_s.match?(/^(true|t|yes|y|1|on)$/i)
+
+        cookie_name = [@experiment.name, 'id'].join('_')
+        cookie_value = request.cookie_jar.signed[cookie_name]
+
+        if value[:user_id].blank? && cookie_value.blank?
+          request.cookie_jar.permanent.signed[cookie_name] = {
+            value: value[:user_id] = SecureRandom.uuid, secure: true, domain: :all, httponly: true
+          }
+        elsif cookie_value # we know via the cookie
+          if value[:user_id].blank?
+            value[:user_id] = cookie_value
+          else
+            @migrations_with << { user_id: cookie_value }
+            request.cookie_jar.delete(cookie_name, domain: :all)
+          end
+        end
+
+        value
+      end
+
+      def migration_keys
+        return nil if @migrations_from.empty? && @migrations_with.empty?
+
+        @migrations_from.map { |m| key_for(m) } +
+          @migrations_with.map { |m| key_for(@value.merge(m)) }
+      end
+
+      def key_for(context)
+        Configuration.context_hash_strategy.call(context)
+      end
+    end
+  end
+end

data/lib/gitlab/experiment/dsl.rb

@@ -0,0 +1,11 @@
+# frozen_string_literal: true
+
+module Gitlab
+  class Experiment
+    module Dsl
+      def experiment(name, variant_name = nil, **context, &block)
+        Experiment.run(name, variant_name, **context, &block)
+      end
+    end
+  end
+end

data/lib/gitlab/experiment/variant.rb

@@ -0,0 +1,7 @@
+# frozen_string_literal: true
+
+module Gitlab
+  class Experiment
+    Variant = Struct.new(:name, :payload, keyword_init: true)
+  end
+end

data/lib/gitlab/experiment/version.rb

@@ -0,0 +1,7 @@
+# frozen_string_literal: true
+
+module Gitlab
+  class Experiment
+    VERSION = '0.2.0'
+  end
+end

metadata

@@ -0,0 +1,71 @@
+--- !ruby/object:Gem::Specification
+name: gitlab-experiment
+version: !ruby/object:Gem::Version
+  version: 0.2.0
+platform: ruby
+authors:
+- GitLab
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2020-09-20 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: scientist
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 1.5.0
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.5'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 1.5.0
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.5'
+description: 
+email:
+- gitlab_rubygems@gitlab.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- LICENSE.txt
+- README.md
+- lib/gitlab/experiment.rb
+- lib/gitlab/experiment/configuration.rb
+- lib/gitlab/experiment/context.rb
+- lib/gitlab/experiment/dsl.rb
+- lib/gitlab/experiment/variant.rb
+- lib/gitlab/experiment/version.rb
+homepage: https://gitlab.com/gitlab-org/gitlab-experiment
+licenses:
+- MIT
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '2.6'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 3.0.3
+signing_key: 
+specification_version: 4
+summary: GitLab experiment library built on top of scientist.
+test_files: []