activeexperiment 0.1.0.alpha

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 5809078002c2de223c3c7f1017304defd4a1c7aaa3f9ac53fc7446980657ef14
+  data.tar.gz: 0c954cec3f8bd0af0c399ec4e3af1f2e63a4e7841df961a2df37f761bedaa682
+SHA512:
+  metadata.gz: b05783407e7c4bda3a6833154949250a654ccfcaec30a25330bf9407d89cd613986fbe80660913c704b266329a756626ba09b5d065826cc7d3968f594b0a32e6
+  data.tar.gz: 649646685dcf34cc67fc8c7de2a02dd9067778382e3266ff87f25c69a4f0256eca210d0c3d1f039b5491fbab07122070338f9de89bcf5cd70b6b55e4d4ecde95

data/MIT-LICENSE

@@ -0,0 +1,20 @@
+Copyright 2022 Jeremy Jackson
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,373 @@
+# Active Experiment – Decide what to do next
+
+Active Experiment is a framework for defining and running experiments. It supports using a variety of rollout and reporting strategies and/or services.
+
+Experiments can be everything from determining which query has the best performance, to which feature gets the most engagement, to rolling out a canary version of a new api service.
+
+Experimentation is complex. There are a lot of different ways to run experiments, and even more ways to report on them. Active Experiment is designed to be flexible enough to support a variety of use cases, but also to be consistent and easy to use.
+
+## Usage
+
+Start by defining an experiment class and adding some variants to it:
+
+```ruby
+class MyExperiment < ApplicationExperiment
+  variant(:red) { "red" }
+  variant(:blue) { "blue" }
+end
+```
+
+This experiment can be generated using the Rails generator:
+
+```bash
+rails generate experiment my_experiment red blue
+```
+
+Run the experiment anywhere in the application by providing it a context:
+
+```ruby
+MyExperiment.run(current_user) # => "red" or "blue"
+```
+
+The experiment can also be run using local scope and helpers to override the default variants:
+
+```ruby
+MyExperiment.run(current_user) do |experiment|
+  experiment.on(:red) { redirect_to red_path }
+  experiment.on(:blue) { redirect_to blue_path }
+end
+```
+
+That's it! When this experiment is encountered by different users, half<sup>&#8224;</sup> will get the red variant, half will get the blue variant, and each will always get the same.
+
+<small>&#8224; roughly half, for the statistically pedantic.</small> 
+
+## Download and Installation
+
+Add this line to your Gemfile:
+
+```ruby
+gem "activeexperiment"
+```
+
+Or install the latest version with RubyGems:
+
+```bash
+gem install activeexperiment
+```
+
+Source code can be downloaded as part of the project on GitHub:
+
+* https://github.com/jejacks0n/activeexperiment
+
+## Advanced Experimentation
+
+This area provides a high level overview of the tools that more complex experiments can benefit from.
+
+For example, some experiments need to define a default variant (also known as a _control_) that will be assigned if the experiment is skipped:
+
+```ruby
+class MyExperiment < ApplicationExperiment
+  variant(:red) { "red" }
+  variant(:blue) { "blue" }
+
+  # The term control is simply a convention that means the default variant, and
+  # any variant can be set as the default with +use_default_variant(:red)+
+  control { "default" }
+end
+```
+
+Callbacks can be used to hook into the lifecycle when experiments are run, and can be targeted to when a specific variant has been assigned:
+
+```ruby
+class MyExperiment < ApplicationExperiment
+  control { "default" }
+  variant(:red) { "red" }
+  variant(:blue) { "blue" }
+
+  # Skipping an experiment will always assign the default variant, which could
+  # be nothing, but since there's a control defined, it will be used.
+  before_run { skip if context.admin? }
+  
+  # Only invoked when the red variant has been assigned.
+  before_variant(:red) { puts "running the red variant" }
+  
+  # Maybe there's cleanup or logging to do afterwards?
+  after_run { puts "run complete with the #{variant} variant" unless skipped? }
+end
+```
+
+Segment rules can be used to assign specific variants for certain cases:
+
+```ruby
+class MyExperiment < ApplicationExperiment
+  control { "default" }
+  variant(:red) { "red" }
+  variant(:blue) { "blue" }
+
+  segment :admins, into: :red
+  segment :old_accounts, into: :control
+  
+  private
+  
+  def admins
+    context.admin?
+  end
+
+  def old_accounts
+    context.created_at < 1.year.ago
+  end
+end
+```
+
+## Rollouts
+
+Rollouts are a core concept in Active Experiment. They allow specifying how an experiment should be rolled out, and even if it should be skipped or not. For example, the default rollout in Active Experiment is percentage based and accepts distribution rules -- if no rules are provided, even distribution is used.
+
+A rollout can implement any number of different strategies, interact with services, and can be used on a per-experiment basis.
+
+Here's an example of using the default percent rollout with custom distribution rules:
+
+```ruby
+class MyExperiment < ApplicationExperiment
+  variant(:red) { "red" }
+  variant(:blue) { "blue" }
+  variant(:green) { "green" }
+
+  # Will assign the green variant 80% of the time, red and blue 10% each.
+  use_rollout :percent, rules: { red: 10, blue: 10, green: 80 }
+end
+```
+
+### Defining Custom Rollouts
+
+Project specific rollouts can be defined and registered too. To illustrate, here's a custom rollout that inherits from the base rollout, uses a fictional feature flag library, and assigns a random variant.
+
+```ruby
+class FeatureFlagRollout < ActiveExperiment::Rollouts::BaseRollout
+  def skipped_for(experiment)
+    !Feature.enabled?(@rollout_options[:flag_name] || experiment.name)
+  end
+
+  def variant_for(experiment)
+    experiment.variant_names.sample
+  end
+end
+
+ActiveExperiment::Rollouts.register(:feature_flag, FeatureRollout)
+```
+
+This rollout can now be used the same way the built-in rollouts are:
+
+```ruby
+class MyExperiment < ActiveExperiment::Base
+  variant(:red) { "red" }
+  variant(:blue) { "blue" }
+
+  # Using a custom rollout with options.
+  rollout :feature_flag, flag_name: "my_feature_flag"
+end
+```
+
+Custom rollouts can be registered to autoload as well, so they're only loaded when needed:
+
+```ruby
+ActiveExperiment::Rollouts.register(
+  :feature_flag, 
+  "lib/feature_flag_rollout.rb"
+)
+```
+
+There's a world of flexibility with custom rollouts. One creative and simple rollout concept is to use the experiment itself:
+
+```ruby
+class MyExperiment < ActiveExperiment::Base
+  variant(:red) { "red" }
+  variant(:blue) { "blue" }
+
+  def self.skipped_for(*)
+    false
+  end
+
+  def self.variant_for(*)
+    variant_names.sample
+  end
+  
+  use_rollout self
+end
+```
+
+## Reporting
+
+Reporting is a core concept in Active Experiment. It allows for collecting data about experiments and variants, and can be used to track performance metrics, analyze results, and more.
+
+Some simple reporting strategies might simply be added to `after_run` callbacks, but more complex reporting strategies can be implemented using a subscriber.
+
+A subscriber can be used to listen for experiment events and report them to a service. For example, here's a subscriber that reports to a fictional analytics service:
+
+```ruby
+class MyAnalyticsSubscriber
+  def process_run(event)
+    experiment = event.payload[:experiment]
+    return if experiment.skipped?
+
+    Analytics.report(
+      experiment.serialize,
+      error: event.payload[:exception_object]
+    )
+  end
+end
+
+MyAnalyticsSubscriber.attach_to(:active_experiment)
+```
+
+The following Active Experiment events are available for subscribers:
+
+- `start_experiment` - The experiment has begun.
+- `process_segment_callbacks` - The experiment has processed all segment rules. A variant may have been resolved through this step.
+- `process_variant_steps` - An experiment variant has been run.
+- `process_variant_callbacks` - The experiment has processed variant callbacks.
+- `process_run_callbacks` - The experiment has processed run callbacks.
+- `process_run` - The experiment has completed and can be reported on.
+
+In each of these events, the experiment instance is available in the `event.payload` hash.
+
+## Experiments in Views
+
+Experiments can be used in views, just like in any other part of your application. Sometimes though, you might want to render markup inside your run block too, and to do this, you'll need to "capture" the experiment.
+
+To accomplish this, you can ask the experiment to capture itself by providing the view scope. The following examples (HAML or ERB) help illustrate how to avoid duplicating markup within each variant block by putting it (the container div for instance) in the run block.
+
+Remember to include the `ActiveExperiment::Capturable` module in your experiment class:
+
+```ruby
+class MyExperiment < ActiveExperiment::Base
+  include ActiveExperiment::Capturable
+  
+  variant(:red) { "red" }
+  variant(:blue) { "blue" }
+end
+```
+
+<details>
+<summary>Expand HAML example</summary>
+
+```haml
+!= MyExperiment.set(capture: self).run(current_user) do |experiment|
+  %div.container
+    = experiment.on(:red) do
+      %button.red-pill Red
+    = experiment.on(:blue) do
+      %button.blue-pill Blue
+```
+</details>
+
+<details>
+<summary>Expand ERB example</summary>
+
+```erb
+<%== MyExperiment.set(capture: self).run do |experiment| %>
+  <div class="container">
+    <%= experiment.on(:red) do %>
+      <button class="red-pill">Red</button>
+    <% end %>
+    <%= experiment.on(:blue) do %>
+      <button class="blue-pill">Blue</button>
+    <% end %>
+  </div>
+<% end %>
+```
+</details>
+
+## Client Side Experimentation
+
+While Active Experiment doesn't include any specific tooling for client side experimentation at this time, it does provide the ability to surface experiments in the client layer.
+
+Whenever an experiment is run in the request lifecycle, it's stored so it can be provided to the client. This means that if an experiment is run in controller, a view, a helper, etc. it will be available to the client.
+
+In the layout, the experiment data can be rendered as JSON for instance:
+
+```erb
+<title>My App</title>
+<script>
+  window.experiments = <%== ActiveExperiment::Executed.to_json %>
+</script>
+```
+
+Or each experiment can be iterated over and rendered individually:
+
+```erb
+<% ActiveExperiment::Executed.experiments.each do |experiment| %>
+  <meta name="<%= experiment.name %>" content="<%== experiment.serialize.to_json %>">
+<% end %>
+```
+
+## Testing
+
+Active Experiment provides a test helper that can be used to stub experiments and assert that the expected experiments have been run.
+
+To use the test helper, include it in your test case:
+
+```ruby
+class MyTestCase < ActiveSupport::TestCase
+  include ActiveExperiment::TestHelper
+end
+```
+
+Now you can stub experiments in your tests:
+
+```ruby
+test "stubbing experiments" do
+  stub_experiment(MyExperiment, :red) do
+    # Now all MyExperiment experiments will assign the :red variant.
+  end
+
+  stub_experiment(MyExperiment, skip: true) do
+    # Now all MyExperiment experiments be skipped.
+  end
+end
+```
+
+Assertion helpers are also available:
+
+```ruby
+test "asserting experiments" do
+  # no experiments has been run
+  assert_no_experiments
+
+  MyExperiment.run(id: 1)
+
+  # 1 experiment has been run
+  assert_experiments 1
+
+  # 2 experiments expected within the block.
+  assert_experiments 2 do
+    MyExperiment.run(id: 2)
+    MyExperiment.run(id: 3)
+  end
+  
+  # assert an experiment has been run with context.
+  assert_experiment_with(MyExperiment, context: { id: 1 })
+  
+  # experiment with context, and a variant assigned expected within the block.
+  assert_experiment_with(MyExperiment, variant: :red, context: { id: 4 }) do
+    MyExperiment.set(variant: :red).run(id: 4)
+  end
+end
+```
+
+RSpec support can be added by requiring `active_experiment/rspec` in the appropriate spec helper.
+
+## GlobalID support
+
+Active Experiment supports [GlobalID serialization](https://github.com/rails/globalid/) for experiment contexts. This is part of what makes it possible to utilize Active Record objects as context to consistently assign the same variant across multiple runs.
+
+## License
+
+Active Experiment is released under the MIT license:
+
+* https://opensource.org/licenses/MIT
+
+Copyright 2022 [jejacks0n](https://github.com/jejacks0n)
+
+## Make Code Not War

data/lib/active_experiment/base.rb

@@ -0,0 +1,76 @@
+# frozen_string_literal: true
+
+require "active_support/core_ext/module/attribute_accessors"
+require "active_experiment"
+require "active_experiment/log_subscriber"
+require "active_experiment/core"
+require "active_experiment/caching"
+require "active_experiment/callbacks"
+require "active_experiment/execution"
+require "active_experiment/instrumentation"
+require "active_experiment/logging"
+require "active_experiment/rollout"
+require "active_experiment/run_key"
+require "active_experiment/segments"
+require "active_experiment/variants"
+
+module ActiveExperiment
+  # = Active Experiment
+  #
+  # Active Experiment is a library that helps with defining, running, and
+  # reporting on experiments.
+  #
+  # In general terms, defining an experiment is done by subclassing
+  # +ActiveExperiment::Base+ and using the +control+ and +variant+ methods to
+  # register the variants of the experiment. Variants are the names of the code
+  # paths that will deviate from one another.
+  #
+  # An example of an experiment definition might look like:
+  #
+  #   class MyExperiment < ActiveExperiment::Base
+  #     control { }
+  #     variant(:red) { "red" }
+  #     variant(:blue) { "blue" }
+  #   end
+  #
+  # The term "control" is used to refer to the default variant, or when no
+  # other variant is assigned. Calling it the control is largely a convention
+  # and is not enforced by the library.
+  #
+  # Once an experiment has been defined, it can be run in various areas of an
+  # application. When running an experiment, variants can be overridden, which
+  # allows utilizing the scope and helpers available where the experiment is
+  # being run.
+  #
+  # For instance, within a view it can be useful to render different partials:
+  #
+  #   MyExperiment.run(current_user) do |experiment|
+  #     experiment.on(:red) { render partial: "red" }
+  #     experiment.on(:blue) { render partial: "blue" }
+  #   end
+  #
+  # Or within a controller, it can be useful to redirect to different paths:
+  #
+  #   MyExperiment.run(current_user) do |experiment|
+  #     experiment.on(:red) { redirect_to red_path }
+  #     experiment.on(:blue) { redirect_to blue_path }
+  #   end
+  #
+  # This approach allows for the same experiment to be run in different areas
+  # of an application, with consistent variant assignment, even if the
+  # experimental behavior is different in different parts of the application.
+  class Base
+    include Core
+    include Caching
+    include Callbacks
+    include Execution
+    include Instrumentation
+    include Logging
+    include Rollout
+    include RunKey
+    include Segments
+    include Variants
+
+    ActiveSupport.run_load_hooks(:active_experiment, self)
+  end
+end

data/lib/active_experiment/cache/active_record_cache_store.rb

@@ -0,0 +1,116 @@
+# frozen_string_literal: true
+
+require "monitor"
+require "active_record"
+
+module ActiveExperiment
+  module Cache
+    # == Active Experiment Active Record Cache Store
+    #
+    # This cache store is an implementation on top of ActiveRecord and expects
+    # that the cache will live until the experiment is cleaned up and removed.
+    #
+    # This is a useful but not particularly performant cache store. It's useful
+    # because a lot of Rails projects already have a usable ActiveRecord
+    # connection, and because it's likely to be a long lived datastore.
+    #
+    # This cache store doesn't use a model class directly, and instead executes
+    # raw sql to minimize memory usage and allocations.
+    #
+    # The data structure:
+    #   key: experiment name : run key
+    #   value: cache entry
+    #
+    # To use this cache in an experiment the table needs to be created. All
+    # experiments will use the same table by default for their cache store, and
+    # can be distinguishable by the experiment name that's part of the cache
+    # key.
+    #
+    #   create_table :active_experiment_cache_entries, id: false do |t|
+    #     t.string :key, null: false
+    #     t.string :value, null: false
+    #   end
+    #
+    #   add_index :active_experiment_cache_entries, :key, unique: true
+    #
+    # Once a table is created, the cache store can be used in an experiment:
+    #
+    #   class MyExperiment < ActiveExperiment::Base
+    #     variant(:red) { "red" }
+    #     variant(:blue) { "blue" }
+    #
+    #     use_cache_store :active_record
+    #   end
+    class ActiveRecordCacheStore < ActiveSupport::Cache::Store
+      DEFAULT_TABLE_NAME = "active_experiment_cache_entries"
+
+      def initialize(options = nil)
+        super
+        @connection = ActiveRecord::Base.connection
+      end
+
+      def length(options = nil)
+        options = merged_options(options)
+        execute(<<~SQL).first["COUNT(key)"]
+          SELECT COUNT(key) FROM #{table_name(options)}
+        SQL
+      end
+
+      def clear(options = nil)
+        options = merged_options(options)
+        execute(<<~SQL)
+          DELETE FROM #{table_name(options)}
+        SQL
+      end
+
+      def delete_matched(matcher, options = nil)
+        options = merged_options(options)
+        execute(<<~SQL, key_matcher(matcher, options))
+          DELETE FROM #{table_name(options)} WHERE key LIKE $1
+        SQL
+      end
+
+      private
+        def read_entry(key, **options)
+          results = execute(<<~SQL, key)&.first.try(:[], "value")
+            SELECT value FROM #{table_name(options)} WHERE key = $1
+          SQL
+
+          deserialize_entry(results)
+        end
+
+        def write_entry(key, entry, **options)
+          return false if options[:unless_exist] && exist?(key, options)
+
+          execute(<<~SQL, key, serialize_entry(entry, **options))
+            INSERT INTO #{table_name(options)} (key, value) VALUES ($1, $2)
+          SQL
+
+          true
+        end
+
+        def delete_entry(key, **options)
+          execute(<<~SQL, key)
+            DELETE FROM #{table_name(options)} WHERE key = $1
+          SQL
+
+          true
+        end
+
+        def table_name(options)
+          options[:table_name] || DEFAULT_TABLE_NAME
+        end
+
+        def key_matcher(source, options)
+          source = "#{source}%"
+
+          return source unless options[:namespace]
+          namespace_key(source, options)
+        end
+
+        def execute(sql, *args, prepare: true, **kws, &block)
+          @connection.exec_query(sql, "SQL", args, prepare: prepare, **kws, &block)
+        end
+    end
+  end
+end

data/lib/active_experiment/cache/redis_hash_cache_store.rb

@@ -0,0 +1,68 @@
+# frozen_string_literal: true
+
+require "active_support/cache/redis_cache_store"
+
+module ActiveExperiment
+  module Cache
+    # == Active Experiment Redis Hash Cache Store
+    #
+    # This cache store is an implementation on top of the redis hash data type
+    # (https://redis.io/docs/data-types/hashes/) and expects that the cache
+    # will live until the experiment is cleaned up and removed.
+    #
+    # This is a good cache store to use with Active Experiment because of the
+    # optimized way that redis stores hashes.
+    #
+    # The data structure:
+    #   key: experiment name
+    #   fields: key => entry
+    #
+    # To use this cache in an experiment:
+    #
+    #   class MyExperiment < ActiveExperiment::Base
+    #     variant(:red) { "red" }
+    #     variant(:blue) { "blue" }
+    #
+    #     use_cache_store :redis_hash
+    #   end
+    class RedisHashCacheStore < ActiveSupport::Cache::RedisCacheStore
+      def length(hkey = nil)
+        if hkey
+          failsafe :read_hlen do
+            redis.then { |c| c.hlen(hkey) }
+          end
+        else
+          failsafe :read_dbsize do
+            redis.then { |c| c.dbsize }
+          end
+        end
+      end
+
+      private
+        def hkey(key)
+          parts = key.to_s.split(":")
+          run_key = parts.pop
+          [Array(parts).join(":"), run_key]
+        end
+
+        def read_serialized_entry(key, raw: false, **options)
+          failsafe :read_entry do
+            redis.then { |c| c.hget(*hkey(key)) }
+          end
+        end
+
+        def write_serialized_entry(key, payload, raw: false, unless_exist: false, expires_in: nil, race_condition_ttl: nil, pipeline: nil, **options)
+          # TODO: Support pipeline?
+          failsafe :write_entry, returning: false do
+            redis.then { |c| c.hset(*hkey(key), payload) }
+          end
+        end
+
+        def delete_entry(key, **options)
+          failsafe :delete_entry, returning: false do
+            redis.then { |c| c.hdel(*hkey(key)) }
+          end
+        end
+    end
+  end
+end

data/lib/active_experiment/cache.rb

@@ -0,0 +1,28 @@
+# frozen_string_literal: true
+
+module ActiveExperiment
+  # == Cache Stores
+  #
+  # TODO: finish documenting.
+  module Cache
+    extend ActiveSupport::Autoload
+
+    autoload :ActiveRecordCacheStore
+    autoload :RedisHashCacheStore
+
+    CACHE_STORE_SUFFIX = "CacheStore"
+    private_constant :CACHE_STORE_SUFFIX
+
+    # Allows looking up a cache store by name.
+    #
+    # Raises an +ArgumentError+ if the cache store isn't found.
+    def self.lookup(name, *args, **options)
+      const_get("#{name.to_s.camelize}#{CACHE_STORE_SUFFIX}").new(*args, **options)
+    rescue NameError
+      store = ActiveSupport::Cache.lookup_store(name, *args, **options)
+      raise ArgumentError, "No cache store found for #{name.inspect}" unless store
+
+      store
+    end
+  end
+end