gitlab-experiment 0.4.11 → 0.5.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 207b046a8c0b1c98ad958442716be6a8232c431bfcc702a42817d8eb1aa3f465
-  data.tar.gz: c654346312abfd1684af031fe1784cb1db61ac583c0c52c3ef50b08d0337182b
+  metadata.gz: 1c4e4a4b19b67700aab0629cd615617c4452d8b4b145ef8ff85d12443994362b
+  data.tar.gz: 802c31143890f2431a80ed3dcb247cbf46247ce276053e8e27a4d190822650df
 SHA512:
-  metadata.gz: 3f3b9d02b8a268e99bb9cda60e7c1a76d55962f9d04754209558deebd58624c5407bf883a6405c3d248b13f3a246a96a9d4a370bfca052e512f125a57aa89e8f
-  data.tar.gz: b43611c028598b362972079b6b785dd62e7c4b35dffa2abc6bd9822a6a2b488385101a5aea975b0d6c429638a8e8661ff6e8aca9e0a0e184c9510a6e5c591d48
+  metadata.gz: 86f86cd9dd01006be9668a2ceb4d60351e943487576d236e12b40eb9405e4255fdbbffb0c5811b17a4f1c8d2784b011eb20bc783e622f4f4f05fc6fb166f0df3
+  data.tar.gz: bca224adeaade21b59077effd1faa0d6ca95607fe02f6c7735b8c9b79b7d5a08f76675549ddbc874ce63adbe39895d7b3f6f1fb71e908ae22a76c817a1d0f88d

data/README.md

@@ -3,7 +3,7 @@ GitLab Experiment
 
 <img alt="experiment" src="/uploads/60990b2dbf4c0406bbf8b7f998de2dea/experiment.png" align="right" width="40%">
 
-Here at GitLab, we run experiments as A/B/n tests and review the data the experiment generates. From that data, we determine the best performing variant and promote it as the new default code path. Or revert back to the control if no variant outperformed it.
+Here at GitLab, we run experiments as A/B/n tests and review the data the experiment generates. From that data, we determine the best performing variant and promote it as the new default code path. Or revert back to the control if no variant outperformed it. You can read our [Experiment Guide](https://docs.gitlab.com/ee/development/experiment_guide/) docs if you're curious about how we use this gem internally.
 
 This library provides a clean and elegant DSL (domain specific language) to define, run, and track your GitLab experiment.
 
@@ -64,7 +64,7 @@ end
 
 You can define the experiment using simple control/candidate paths, or provide named variants.
 
-Handling multi-variant experiments is up to the configuration you provide around resolving variants. But in our example we may want to try with and without the confirmation. We can run any number of variations in our experiments this way.
+Handling [multivariate](https://en.wikipedia.org/wiki/Multivariate_statistics) experiments is up to the configuration you provide around resolving variants. But in our example we may want to try with and without the confirmation. We can run any number of variations in our experiments this way.
 
 ```ruby
 experiment(:notification_toggle, actor: user) do |e|
@@ -149,37 +149,13 @@ experiment(:notification_toggle, actor: user) do |e|
 end
 ```
 
-<details>
-  <summary>You can also use the lower level class interface...</summary>
-
-### Using the `.run` approach
-
-This is useful if you haven't included the DSL and so don't have access to the `experiment` method, but still want to execute an experiment. This is ultimately what the `experiment` method calls through to, and the method signatures are the same.
-
-```ruby
-exp = Gitlab::Experiment.run(:notification_toggle, actor: user) do |e|
-  # Context may be passed in the block, but must be finalized before calling
-  # run or track.
-  e.context(project: project) # add the project to the context
-
-  # Define the control and candidate variant.
-  e.use { render_toggle } # control
-  e.try { render_button } # candidate
-end
-
-# Track an event on the experiment we've defined.
-exp.track(:clicked_button)
-```
-
-</details>
-
 <details>
   <summary>You can also specify the variant to use for segmentation...</summary>
 
-### Specifying variant
-
 Generally, defining segmentation rules is a better way to approach routing into specific variants, but it's possible to explicitly specify the variant when running an experiment. It's important to know what this might do to your data during rollout, so use this with careful consideration.
 
+Any time a specific variant is provided (including `:control`) it will be cached for that context, if caching is enabled. 
+
 ```ruby
 experiment(:notification_toggle, :no_interface, actor: user) do |e|
   e.use { render_toggle } # control
@@ -360,6 +336,37 @@ experiment(:example, actor: user, project: project)
 
 For edge cases, you can pass the cookie through by assigning it yourself -- e.g. `actor: request.cookie_jar.signed['example_actor']`. The cookie name is the full experiment name (including any configured prefix) with `_actor` appended -- e.g. `gitlab_notification_toggle_actor` for the `:notification_toggle` experiment key with a configured prefix of `gitlab`.
 
+## How it works
+
+The way the gem works is best described using the following decision tree illustration. When an experiment is run, the following logic is executed to resolve what experience should be provided, given how the experiment is defined, and the context provided.
+ 
+```mermaid
+graph TD
+    GP[General Pool/Population] --> Enabled?
+    Enabled? -->|Yes| Cached?[Cached? / Pre-segmented?]
+    Enabled? -->|No| Excluded[Control / No Tracking]
+    Cached? -->|No| Excluded?
+    Cached? -->|Yes| Cached[Cached Value]
+    Excluded? -->|Yes / Cached| Excluded
+    Excluded? -->|No| Segmented?
+    Segmented? -->|Yes / Cached| VariantA
+    Segmented? -->|No| Included?[Experiment Group?]
+    Included? -->|Yes| Rollout
+    Included? -->|No| Control
+    Rollout -->|Cached| VariantA
+    Rollout -->|Cached| VariantB
+    Rollout -->|Cached| VariantC
+
+classDef included fill:#380d75,color:#ffffff,stroke:none
+classDef excluded fill:#fca121,stroke:none
+classDef cached fill:#2e2e2e,color:#ffffff,stroke:none
+classDef default fill:#fff,stroke:#6e49cb
+
+class VariantA,VariantB,VariantC included
+class Control,Excluded excluded
+class Cached cached
+```
+
 ## Configuration
 
 This gem needs to be configured before being used in a meaningful way.
@@ -534,7 +541,7 @@ https://gitlab.com/gitlab-org/gitlab-experiment. This project is intended to be
 safe, welcoming space for collaboration, and contributors are expected to adhere
 to the [Contributor Covenant](http://contributor-covenant.org) code of conduct.
 
-## Release Process
+## Release process
 
 Please refer to the [Release Process](docs/release_process.md).
 
@@ -543,7 +550,7 @@ Please refer to the [Release Process](docs/release_process.md).
 The gem is available as open source under the terms of the
 [MIT License](http://opensource.org/licenses/MIT).
 
-## Code of Conduct
+## Code of conduct
 
 Everyone interacting in the `Gitlab::Experiment` project’s codebases, issue trackers,
 chat rooms and mailing lists is expected to follow the

data/lib/generators/gitlab/experiment/install/templates/initializer.rb.tt

@@ -22,20 +22,23 @@ Gitlab::Experiment.configure do |config|
   # `['www.gitlab.com', '.gitlab.com']`.
   config.cookie_domain = :all
 
-  # Logic this project uses to resolve a variant for a given experiment.
+  # The default rollout strategy that works for single and multi-variants.
   #
-  # Should return a symbol or string that represents the variant that should
-  # be assigned. Blank or nil values will be defaulted to the control.
+  # You can provide your own rollout strategies and override them per
+  # experiment.
+  #
+  # Examples include:
+  #   Rollout::First, Rollout::Random, Rollout::RoundRobin
+  config.default_rollout = Gitlab::Experiment::Rollout::First
+
+  # Logic this project uses to determine inclusion in a given experiment.
+  #
+  # Expected to return a boolean value.
   #
   # This block is executed within the scope of the experiment and so can access
   # experiment methods, like `name`, `context`, and `signature`.
-  config.variant_resolver = lambda do |requested_variant|
-    # Run the control, unless a variant was requested in code:
-    requested_variant
-
-    # Run the candidate, unless a variant was requested, with a fallback:
-    #
-    # requested_variant || variant_names.first || nil
+  config.inclusion_resolver = lambda do |requested_variant|
+    false
   end
 
   # Tracking behavior can be implemented to link an event to an experiment.

data/lib/gitlab/experiment.rb

@@ -3,11 +3,15 @@
 require 'scientist'
 require 'active_support/callbacks'
 require 'active_support/cache'
+require 'active_support/concern'
 require 'active_support/core_ext/object/blank'
 require 'active_support/core_ext/string/inflections'
+require 'active_support/core_ext/module/delegation'
 
+require 'gitlab/experiment/base_interface'
 require 'gitlab/experiment/cache'
 require 'gitlab/experiment/callbacks'
+require 'gitlab/experiment/rollout'
 require 'gitlab/experiment/configuration'
 require 'gitlab/experiment/cookies'
 require 'gitlab/experiment/context'
@@ -18,58 +22,44 @@ require 'gitlab/experiment/engine' if defined?(Rails::Engine)
 
 module Gitlab
   class Experiment
-    include Scientist::Experiment
+    include BaseInterface
     include Cache
     include Callbacks
 
     class << self
-      def configure
-        yield Configuration
-      end
-
-      def run(name = nil, variant_name = nil, **context, &block)
-        raise ArgumentError, 'name is required' if name.nil? && base?
+      def default_rollout(rollout = nil)
+        return @rollout ||= Configuration.default_rollout if rollout.blank?
 
-        instance = constantize(name).new(name, variant_name, **context, &block)
-        return instance unless block
-
-        instance.context.frozen? ? instance.run : instance.tap(&:run)
+        @rollout = Rollout.resolve(rollout)
       end
 
-      def experiment_name(name = nil, suffix: true, suffix_word: 'experiment')
-        name = (name.presence || self.name).to_s.underscore.sub(%r{(?<char>[_/]|)#{suffix_word}$}, '')
-        name = "#{name}#{Regexp.last_match(:char) || '_'}#{suffix_word}"
-        suffix ? name : name.sub(/_#{suffix_word}$/, '')
-      end
-
-      def base?
-        self == Gitlab::Experiment || name == Configuration.base_class
+      def exclude(*filter_list, **options, &block)
+        build_callback(:exclusion_check, filter_list.unshift(block), **options) do |target, callback|
+          throw(:abort) if target.instance_variable_get(:@excluded) || callback.call(target, nil) == true
+        end
       end
 
-      private
-
-      def constantize(name = nil)
-        return self if name.nil?
-
-        experiment_name(name).classify.safe_constantize || Configuration.base_class.constantize
+      def segment(*filter_list, variant:, **options, &block)
+        build_callback(:segmentation_check, filter_list.unshift(block), **options) do |target, callback|
+          target.variant(variant) if target.instance_variable_get(:@variant_name).nil? && callback.call(target, nil)
+        end
       end
     end
 
-    def initialize(name = nil, variant_name = nil, **context)
-      raise ArgumentError, 'name is required' if name.blank? && self.class.base?
-
-      @name = self.class.experiment_name(name, suffix: false)
-      @context = Context.new(self, **context)
-      @variant_name = cache_variant(variant_name) { nil } if variant_name.present?
-
-      compare { false }
+    def name
+      [Configuration.name_prefix, @name].compact.join('_')
+    end
 
-      yield self if block_given?
+    def control(&block)
+      candidate(:control, &block)
     end
+    alias_method :use, :control
 
-    def inspect
-      "#<#{self.class.name || 'AnonymousClass'}:#{format('0x%016X', __id__)} @name=#{name} @context=#{context.value}>"
+    def candidate(name = nil, &block)
+      name = (name || :candidate).to_s
+      behaviors[name] = block
     end
+    alias_method :try, :candidate
 
     def context(value = nil)
       return @context if value.blank?
@@ -79,16 +69,12 @@ module Gitlab
     end
 
     def variant(value = nil)
-      @variant_name = value unless value.blank?
+      @variant_name = cache_variant(value) if value.present?
       return Variant.new(name: (@variant_name || :unresolved).to_s) if @variant_name || @resolving_variant
 
       if enabled?
-        @variant_name ||= :control if excluded?
-
         @resolving_variant = true
-        if (result = cache_variant(@variant_name) { resolve_variant_name }).present?
-          @variant_name = result.to_sym
-        end
+        @variant_name = cached_variant_resolver(@variant_name)
       end
 
       run_callbacks(segmentation_callback_chain) do
@@ -99,6 +85,16 @@ module Gitlab
       @resolving_variant = false
     end
 
+    def rollout(rollout = nil)
+      return @rollout ||= self.class.default_rollout if rollout.blank?
+
+      @rollout = Rollout.resolve(rollout)
+    end
+
+    def exclude!
+      @excluded = true
+    end
+
     def run(variant_name = nil)
       @result ||= super(variant(variant_name).name)
     end
@@ -113,41 +109,6 @@ module Gitlab
       instance_exec(action, event_args, &Configuration.tracking_behavior)
     end
 
-    def name
-      [Configuration.name_prefix, @name].compact.join('_')
-    end
-
-    def variant_names
-      @variant_names ||= behaviors.keys.map(&:to_sym) - [:control]
-    end
-
-    def behaviors
-      @behaviors ||= public_methods.each_with_object(super) do |name, behaviors|
-        next unless name.end_with?('_behavior')
-
-        behavior_name = name.to_s.sub(/_behavior$/, '')
-        behaviors[behavior_name] ||= -> { send(name) } # rubocop:disable GitlabSecurity/PublicSend
-      end
-    end
-
-    def try(name = nil, &block)
-      name = (name || 'candidate').to_s
-      behaviors[name] = block
-    end
-
-    def signature
-      { variant: variant.name, experiment: name }.merge(context.signature)
-    end
-
-    def id
-      "#{name}:#{key_for(context.value)}"
-    end
-    alias_method :session_id, :id
-
-    def flipper_id
-      "Experiment;#{id}"
-    end
-
     def enabled?
       true
     end
@@ -155,12 +116,19 @@ module Gitlab
     def excluded?
       return @excluded if defined?(@excluded)
 
-      @excluded = !@context.trackable? || # adhere to DNT headers
-        !run_callbacks(:exclusion_check) { :not_excluded } # didn't pass exclusion check
+      @excluded = !run_callbacks(:exclusion_check) { :not_excluded }
+    end
+
+    def experiment_group?
+      instance_exec(@variant_name, &Configuration.inclusion_resolver)
     end
 
     def should_track?
-      enabled? && !excluded?
+      enabled? && @context.trackable? && !excluded?
+    end
+
+    def signature
+      { variant: variant.name, experiment: name }.merge(context.signature)
     end
 
     def key_for(hash)
@@ -170,22 +138,13 @@ module Gitlab
     protected
 
     def segmentation_callback_chain
-      return :segmentation_check if !variant_assigned? && enabled? && !excluded?
+      return :segmentation_check if @variant_name.nil? && enabled? && !excluded?
 
       :unsegmented
     end
 
-    def variant_assigned?
-      !@variant_name.nil?
-    end
-
     def resolve_variant_name
-      instance_exec(@variant_name, &Configuration.variant_resolver)
-    end
-
-    def generate_result(variant_name)
-      observation = Scientist::Observation.new(variant_name, self, &behaviors[variant_name])
-      Scientist::Result.new(self, [observation], observation)
+      rollout.new(self).execute if experiment_group?
     end
   end
 end

data/lib/gitlab/experiment/base_interface.rb

@@ -0,0 +1,88 @@
+# frozen_string_literal: true
+
+module Gitlab
+  class Experiment
+    module BaseInterface
+      extend ActiveSupport::Concern
+      include Scientist::Experiment
+
+      class_methods do
+        def configure
+          yield Configuration
+        end
+
+        def experiment_name(name = nil, suffix: true, suffix_word: 'experiment')
+          name = (name.presence || self.name).to_s.underscore.sub(%r{(?<char>[_/]|)#{suffix_word}$}, '')
+          name = "#{name}#{Regexp.last_match(:char) || '_'}#{suffix_word}"
+          suffix ? name : name.sub(/_#{suffix_word}$/, '')
+        end
+
+        def base?
+          self == Gitlab::Experiment || name == Configuration.base_class
+        end
+
+        def constantize(name = nil)
+          return self if name.nil?
+
+          experiment_name(name).classify.safe_constantize || Configuration.base_class.constantize
+        end
+      end
+
+      def initialize(name = nil, variant_name = nil, **context)
+        raise ArgumentError, 'name is required' if name.blank? && self.class.base?
+
+        @name = self.class.experiment_name(name, suffix: false)
+        @context = Context.new(self, **context)
+        @variant_name = cache_variant(variant_name) { nil } if variant_name.present?
+
+        compare { false }
+
+        yield self if block_given?
+      end
+
+      def inspect
+        "#<#{self.class.name || 'AnonymousClass'}:#{format('0x%016X', __id__)} @name=#{name} @context=#{context.value}>"
+      end
+
+      def id
+        "#{name}:#{key_for(context.value)}"
+      end
+      alias_method :session_id, :id
+
+      def flipper_id
+        "Experiment;#{id}"
+      end
+
+      def variant_names
+        @variant_names ||= behaviors.keys.map(&:to_sym) - [:control]
+      end
+
+      def behaviors
+        @behaviors ||= public_methods.each_with_object(super) do |name, behaviors|
+          next unless name.end_with?('_behavior')
+
+          behavior_name = name.to_s.sub(/_behavior$/, '')
+          behaviors[behavior_name] ||= -> { send(name) } # rubocop:disable GitlabSecurity/PublicSend
+        end
+      end
+
+      protected
+
+      def raise_on_mismatches?
+        false
+      end
+
+      def cached_variant_resolver(provided_variant)
+        return :control if excluded?
+
+        result = cache_variant(provided_variant) { resolve_variant_name }
+        result.to_sym if result.present?
+      end
+
+      def generate_result(variant_name)
+        observation = Scientist::Observation.new(variant_name, self, &behaviors[variant_name])
+        Scientist::Result.new(self, [observation], observation)
+      end
+    end
+  end
+end

data/lib/gitlab/experiment/cache.rb

@@ -1,5 +1,7 @@
 # frozen_string_literal: true
 
+require 'active_support/cache'
+
 module Gitlab
   class Experiment
     module Cache

data/lib/gitlab/experiment/callbacks.rb

@@ -1,5 +1,7 @@
 # frozen_string_literal: true
 
+require 'active_support/callbacks'
+
 module Gitlab
   class Experiment
     module Callbacks
@@ -13,28 +15,17 @@ module Gitlab
       end
 
       class_methods do
-        def exclude(*filter_list, **options, &block)
-          filters = filter_list.unshift(block).compact.map do |filter|
-            result_lambda = ActiveSupport::Callbacks::CallTemplate.build(filter, self).make_lambda
-            lambda do |target|
-              throw(:abort) if target.instance_variable_get(:'@excluded') || result_lambda.call(target, nil) == true
-            end
-          end
-
-          raise ArgumentError, 'no filters provided' if filters.empty?
-
-          set_callback(:exclusion_check, :before, *filters, options)
-        end
+        private
 
-        def segment(*filter_list, variant:, **options, &block)
-          filters = filter_list.unshift(block).compact.map do |filter|
+        def build_callback(chain, filters, **options)
+          filters = filters.compact.map do |filter|
             result_lambda = ActiveSupport::Callbacks::CallTemplate.build(filter, self).make_lambda
-            ->(target) { target.variant(variant) if !target.variant_assigned? && result_lambda.call(target, nil) }
+            ->(target) { yield(target, result_lambda) }
           end
 
           raise ArgumentError, 'no filters provided' if filters.empty?
 
-          set_callback(:segmentation_check, :before, *filters, options)
+          set_callback(chain, *filters, **options)
         end
       end
     end

data/lib/gitlab/experiment/configuration.rb

@@ -4,6 +4,8 @@ require 'singleton'
 require 'logger'
 require 'digest'
 
+require 'active_support/deprecation'
+
 module Gitlab
   class Experiment
     class Configuration
@@ -24,10 +26,13 @@ module Gitlab
       # The domain to use on cookies.
       @cookie_domain = :all
 
-      # Logic this project uses to resolve a variant for a given experiment.
-      # If no variant is determined, the control will be used.
-      @variant_resolver = lambda do |requested_variant|
-        requested_variant
+      # The default rollout strategy that works for single and multi-variants.
+      @default_rollout = Rollout::First
+
+      # Logic this project uses to determine inclusion in a given experiment.
+      # Expected to return a boolean value.
+      @inclusion_resolver = lambda do |requested_variant|
+        false
       end
 
       # Tracking behavior can be implemented to link an event to an experiment.
@@ -47,13 +52,27 @@ module Gitlab
       end
 
       class << self
+        # TODO: Added deprecation in release 0.5.0
+        def variant_resolver
+          ActiveSupport::Deprecation.warn('variant_resolver is deprecated, instead use `inclusion_resolver` with a' \
+            'block that returns a boolean.')
+          @inclusion_resolver
+        end
+
+        def variant_resolver=(block)
+          ActiveSupport::Deprecation.warn('variant_resolver is deprecated, instead use `inclusion_resolver` with a' \
+            'block that returns a boolean.')
+          @inclusion_resolver = block
+        end
+
         attr_accessor(
           :name_prefix,
           :logger,
           :base_class,
           :cache,
           :cookie_domain,
-          :variant_resolver,
+          :default_rollout,
+          :inclusion_resolver,
           :tracking_behavior,
           :publishing_behavior,
           :context_hash_strategy

data/lib/gitlab/experiment/context.rb

@@ -1,5 +1,7 @@
 # frozen_string_literal: true
 
+require 'gitlab/experiment/cookies'
+
 module Gitlab
   class Experiment
     class Context

data/lib/gitlab/experiment/dsl.rb

@@ -4,8 +4,17 @@ module Gitlab
   class Experiment
     module Dsl
       def experiment(name, variant_name = nil, **context, &block)
+        raise ArgumentError, 'name is required' if name.nil?
+
         context[:request] ||= request if respond_to?(:request)
-        Experiment.run(name, variant_name, **context, &block)
+
+        base = Configuration.base_class.constantize
+        klass = base.constantize(name) || base
+
+        instance = klass.new(name, variant_name, **context, &block)
+        return instance unless block
+
+        instance.context.frozen? ? instance.run : instance.tap(&:run)
       end
     end
   end

data/lib/gitlab/experiment/rollout.rb

@@ -0,0 +1,27 @@
+# frozen_string_literal: true
+
+module Gitlab
+  class Experiment
+    module Rollout
+      autoload :First, 'gitlab/experiment/rollout/first.rb' # default strategy
+      autoload :Random, 'gitlab/experiment/rollout/random.rb'
+      autoload :RoundRobin, 'gitlab/experiment/rollout/round_robin.rb'
+
+      def self.resolve(klass)
+        return "#{name}::#{klass.to_s.classify}".constantize if klass.is_a?(Symbol) || klass.is_a?(String)
+
+        klass
+      end
+
+      class Base
+        attr_reader :experiment
+
+        delegate :variant_names, :cache, to: :experiment
+
+        def initialize(experiment)
+          @experiment = experiment
+        end
+      end
+    end
+  end
+end

data/lib/gitlab/experiment/rollout/first.rb

@@ -0,0 +1,16 @@
+# frozen_string_literal: true
+
+module Gitlab
+  class Experiment
+    module Rollout
+      class First < Base
+        # This rollout strategy just picks the first variant name. It's the
+        # default resolver as it assumes a single variant. You should consider
+        # using a more advanced rollout if you have multiple variants.
+        def execute
+          variant_names.first
+        end
+      end
+    end
+  end
+end

data/lib/gitlab/experiment/rollout/random.rb

@@ -0,0 +1,15 @@
+# frozen_string_literal: true
+
+module Gitlab
+  class Experiment
+    module Rollout
+      class Random < Base
+        # Pick a random variant if we're in the experiment group. It doesn't
+        # take into account small sample sizes but is useful and performant.
+        def execute
+          variant_names.sample
+        end
+      end
+    end
+  end
+end

data/lib/gitlab/experiment/rollout/round_robin.rb

@@ -0,0 +1,23 @@
+# frozen_string_literal: true
+
+module Gitlab
+  class Experiment
+    module Rollout
+      class RoundRobin < Base
+        KEY_NAME = :last_round_robin_variant
+
+        # Requires a cache to be configured.
+        #
+        # Keeps track of the number of assignments into the experiment group,
+        # and uses this to rotate "round robin" style through the variants
+        # that are defined.
+        #
+        # Relatively performant, but requires a cache, and is dependent on the
+        # performance of that cache store.
+        def execute
+          variant_names[(cache.attr_inc(KEY_NAME) - 1) % variant_names.size]
+        end
+      end
+    end
+  end
+end

data/lib/gitlab/experiment/rspec.rb

@@ -3,39 +3,78 @@
 module Gitlab
   class Experiment
     module RSpecHelpers
-      def stub_experiments(experiments)
-        experiments.each do |name, variant|
-          variant = :control if variant == false
-          raise ArgumentError, 'variant must be a symbol or false' unless variant.is_a?(Symbol)
+      def stub_experiments(experiments, times = nil)
+        experiments.each { |experiment| wrapped_experiment(experiment, times) }
+      end
 
-          klass = Gitlab::Experiment.send(:constantize, name) # rubocop:disable GitlabSecurity/PublicSend
+      def wrapped_experiment(experiment, times = nil, expected = false, &block)
+        klass, experiment_name, variant_name = *experiment_details(experiment)
+        base_klass = Configuration.base_class.constantize
 
-          # We have to use this high level any_instance behavior as there's
-          # not an alternative that allows multiple wrappings of `new`.
-          allow_any_instance_of(klass).to receive(:enabled?).and_return(true)
-          allow_any_instance_of(klass).to receive(:resolve_variant_name).and_return(variant.to_s)
+        # Set expectations on experiment classes so we can and_wrap_original with more specific args
+        experiment_klasses = base_klass.descendants.reject { |k| k == klass }
+        experiment_klasses.push(base_klass).each do |k|
+          allow(k).to receive(:new).and_call_original
         end
-      end
 
-      def wrapped_experiment(experiment, shallow: false, failure: nil, &block)
-        if shallow
-          yield experiment if block.present?
-          return experiment
+        receiver = receive(:new)
+
+        # Be specific for BaseClass calls
+        receiver = receiver.with(experiment_name, any_args) if experiment_name && klass == base_klass
+
+        receiver.exactly(times).times if times
+
+        # Set expectations on experiment class of interest
+        allow_or_expect_klass = expected ? expect(klass) : allow(klass)
+        allow_or_expect_klass.to receiver.and_wrap_original do |method, *original_args, &original_block|
+          method.call(*original_args).tap do |e|
+            # Stub internal methods before calling the original_block
+            allow(e).to receive(:enabled?).and_return(true)
+
+            if variant_name == true # passing true allows the rollout to do its job
+              allow(e).to receive(:experiment_group?).and_return(true)
+            else
+              allow(e).to receive(:resolve_variant_name).and_return(variant_name.to_s)
+            end
+
+            # Stub/set expectations before calling the original_block
+            yield e if block
+
+            original_block.call(e) if original_block.present?
+          end
         end
+      end
 
-        receive_wrapped_new = receive(:new).and_wrap_original do |new, *new_args, &new_block|
-          instance = new.call(*new_args)
-          instance.tap(&block) if block.present?
-          instance.tap(&new_block) if new_block.present?
-          instance
+      private
+
+      def experiment_details(experiment)
+        if experiment.is_a?(Symbol)
+          experiment_name = experiment
+          variant_name = nil
         end
 
-        klass = experiment.class == Class ? experiment : experiment.class
-        if failure
-          expect(klass).to receive_wrapped_new, failure
+        experiment_name, variant_name = *experiment if experiment.is_a?(Array)
+
+        base_klass = Configuration.base_class.constantize
+        variant_name = experiment.variant.name if experiment.is_a?(base_klass)
+
+        if experiment.class.name.nil? # Anonymous class instance
+          klass = experiment.class
+        elsif experiment.instance_of?(Class) # Class level stubbing, eg. "MyExperiment"
+          klass = experiment
         else
-          allow(klass).to receive_wrapped_new
+          experiment_name ||= experiment.instance_variable_get(:@name)
+          klass = base_klass.constantize(experiment_name)
+        end
+
+        if experiment_name && klass == base_klass
+          experiment_name = experiment_name.to_sym
+
+          # For experiment names like: "group/experiment-name"
+          experiment_name = experiment_name.to_s if experiment_name.inspect.include?('"')
         end
+
+        [klass, experiment_name, variant_name]
       end
     end
 
@@ -130,33 +169,42 @@ module Gitlab
         end
 
         chain(:with_context) { |expected_context| @expected_context = expected_context }
-        chain(:on_any_instance) { @on_self = false }
+
+        chain(:on_next_instance) { @on_next_instance = true }
 
         def expect_tracking_on(experiment, negated, event, *event_args)
-          @experiment = experiment
-          @on_self = true if require_experiment(experiment, 'track', classes: !@on_self) && @on_self.nil?
-          wrapped_experiment(experiment, shallow: @on_self, failure: failure_message(:no_new, event)) do |instance|
-            @experiment = instance
-            allow(@experiment).to receive(:track)
+          klass = experiment.instance_of?(Class) ? experiment : experiment.class
+          unless klass <= Gitlab::Experiment
+            raise(
+              ArgumentError,
+              "track matcher is limited to experiment instances and classes"
+            )
+          end
+
+          expectations = proc do |e|
+            @experiment = e
+            allow(e).to receive(:track).and_call_original
 
             if negated
-              expect(@experiment).not_to receive_tracking_call_for(event, *event_args)
+              expect(e).not_to receive(:track).with(*[event, *event_args])
             else
-              expect(@experiment).to receive_tracking_call_for(event, *event_args)
-            end
-          end
-        end
+              if @expected_variant
+                expect(@experiment.variant.name).to eq(@expected_variant), failure_message(:variant, event)
+              end
 
-        def receive_tracking_call_for(event, *event_args)
-          receive(:track).with(*[event, *event_args]) do # rubocop:disable CodeReuse/ActiveRecord
-            if @expected_variant
-              expect(@experiment.variant.name).to eq(@expected_variant), failure_message(:variant, event)
-            end
+              if @expected_context
+                expect(@experiment.context.value).to include(@expected_context), failure_message(:context, event)
+              end
 
-            if @expected_context
-              expect(@experiment.context.value).to include(@expected_context), failure_message(:context, event)
+              expect(e).to receive(:track).with(*[event, *event_args]).and_call_original
             end
           end
+
+          if experiment.instance_of?(Class) || @on_next_instance
+            wrapped_experiment(experiment, nil, true) { |e| expectations.call(e) }
+          else
+            expectations.call(experiment)
+          end
         end
 
         def failure_message(failure_type, event)
@@ -173,8 +221,6 @@ module Gitlab
                   expected context: #{@expected_context}
                     actual context: #{@experiment.context.value}
             MESSAGE
-          when :no_new
-            %(expected #{@experiment.inspect} to have tracked #{event.inspect}, but no new instances were created)
           end
         end
       end

data/lib/gitlab/experiment/version.rb

@@ -2,6 +2,6 @@
 
 module Gitlab
   class Experiment
-    VERSION = '0.4.11'
+    VERSION = '0.5.3'
   end
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: gitlab-experiment
 version: !ruby/object:Gem::Version
-  version: 0.4.11
+  version: 0.5.3
 platform: ruby
 authors:
 - GitLab
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2021-02-17 00:00:00.000000000 Z
+date: 2021-04-06 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activesupport
@@ -30,20 +30,20 @@ dependencies:
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '1.5'
+        version: '1.6'
     - - ">="
       - !ruby/object:Gem::Version
-        version: 1.5.0
+        version: 1.6.0
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '1.5'
+        version: '1.6'
     - - ">="
       - !ruby/object:Gem::Version
-        version: 1.5.0
+        version: 1.6.0
 description: 
 email:
 - gitlab_rubygems@gitlab.com
@@ -65,6 +65,7 @@ files:
 - lib/generators/test_unit/experiment/experiment_generator.rb
 - lib/generators/test_unit/experiment/templates/experiment_test.rb.tt
 - lib/gitlab/experiment.rb
+- lib/gitlab/experiment/base_interface.rb
 - lib/gitlab/experiment/cache.rb
 - lib/gitlab/experiment/cache/redis_hash_store.rb
 - lib/gitlab/experiment/callbacks.rb
@@ -73,6 +74,10 @@ files:
 - lib/gitlab/experiment/cookies.rb
 - lib/gitlab/experiment/dsl.rb
 - lib/gitlab/experiment/engine.rb
+- lib/gitlab/experiment/rollout.rb
+- lib/gitlab/experiment/rollout/first.rb
+- lib/gitlab/experiment/rollout/random.rb
+- lib/gitlab/experiment/rollout/round_robin.rb
 - lib/gitlab/experiment/rspec.rb
 - lib/gitlab/experiment/variant.rb
 - lib/gitlab/experiment/version.rb