activeexperiment 0.1.0.alpha → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 5809078002c2de223c3c7f1017304defd4a1c7aaa3f9ac53fc7446980657ef14
-  data.tar.gz: 0c954cec3f8bd0af0c399ec4e3af1f2e63a4e7841df961a2df37f761bedaa682
+  metadata.gz: af34351515f9c77859089c5ac937f4fd7a7f241ee6fef23dcd59cc42150e90fb
+  data.tar.gz: e02cd7f96c2b8a758313956de1e7ebe49218d6fed28e8eaafe543e9af1f8f6fb
 SHA512:
-  metadata.gz: b05783407e7c4bda3a6833154949250a654ccfcaec30a25330bf9407d89cd613986fbe80660913c704b266329a756626ba09b5d065826cc7d3968f594b0a32e6
-  data.tar.gz: 649646685dcf34cc67fc8c7de2a02dd9067778382e3266ff87f25c69a4f0256eca210d0c3d1f039b5491fbab07122070338f9de89bcf5cd70b6b55e4d4ecde95
+  metadata.gz: 33bcf6e7d09affba6e26a9048f22b951721e69dc5a98038debf923985be45b0c3a6edd4c70933f98b551348a6a33301aeb21e4d8b741a079e343eeb5b654cc97
+  data.tar.gz: 90843e649a0457470fd74c603b1936dca0bb06419fdca402bde3ead122b4d3a371a2088dee1fcdd4564b9f89bac61f9091d3c2498a1aa39b5bc20f92f15b33fc

data/CHANGELOG.md

@@ -0,0 +1,61 @@
+# Changelog
+
+## 0.2.0 (unreleased)
+
+### Breaking changes
+
+**Variant assignment changes for every existing context.**
+
+Run keys are computed differently in this release, and the percent rollout's
+boundaries have shifted. Any context that has already been assigned a variant
+may be assigned a different one after upgrading, and existing cache entries no
+longer correspond to the keys that will be looked up.
+
+If you have experiments in flight, treat this as the end of those experiments.
+
+Collect your results, clear the experiment cache (`MyExperiment.clear_cache`),
+and start new experiments fresh.
+
+* Run keys are now computed from a context hash in a stable order. Previously
+  `{a: 1, b: 2}` and `{b: 2, a: 1}` produced different digests, so the same
+  context written two different ways would split a subject across variants and
+  across cache entries.
+
+* Contexts containing objects that aren't `GlobalID::Identification` now raise
+  an `ArgumentError`. Such objects fell through to `inspect`, embedding a memory
+  address that changes every process — which silently destroyed both consistent
+  assignment and cache stability.
+  Set `ActiveExperiment::Base.unsafe_context_digest = true` to restore the
+  previous behavior if you don't want exceptions raised on this / want the
+  existing behavior.
+
+* The percent rollout no longer over-allocates the first bucket and
+  under-allocates the last. Declared rules of `{control: 25, red: 30, blue: 45}`
+  previously distributed as `26/30/44`.
+
+* Run key digests now include a version marker, so future changes to the key
+  algorithm are detectable rather than silent.
+
+* `ActiveExperiment::Rollouts.lookup` no longer resolves constants it wasn't
+  given. Lookup fell through to the top level, so an unregistered `FooRollout`
+  defined anywhere in an application would answer `lookup(:foo)` and could be
+  used by `use_rollout :foo`. Rollouts now have to be registered. Registering by
+  path still works, including when the file defines the class at the top level.
+
+### Fixed
+
+* The Railtie no longer calls `Rails.application.secrets`, which was removed in
+  Rails 7.2 and raised `NoMethodError` during initialization on Rails 7.2+.
+
+* Active Experiment doesn't consider a missing `secret_key_base` an issue and 
+  instead falls back to an unsalted digest rather than keeping the application
+  from booting over an optional feature.
+
+* Compatibility with Ruby 3.4+ and Rails 8.1.
+
+### Changed
+
+* Minimum supported versions are now Ruby 3.2 and Active Support 7.1.
+
+* Integration tests now run as part of `rake test`. They previously existed but
+  were never executed.

data/README.md

@@ -1,5 +1,12 @@
 # Active Experiment – Decide what to do next
 
+<img alt="Active Experiment" width="150" height="183" src="https://user-images.githubusercontent.com/13765/208318101-b48c9493-15ed-4a99-b42f-b20720dd7c77.png" align="right" hspace="20">
+
+[![Gem Version](https://badge.fury.io/rb/activeexperiment.svg)](https://badge.fury.io/rb/activeexperiment)
+[![License](https://img.shields.io/badge/license-MIT-brightgreen.svg)](https://opensource.org/licenses/MIT)
+[![CI](https://github.com/jejacks0n/activeexperiment/actions/workflows/ci.yml/badge.svg)](https://github.com/jejacks0n/activeexperiment/actions/workflows/ci.yml)
+[![codecov](https://codecov.io/gh/jejacks0n/activeexperiment/branch/main/graph/badge.svg)](https://codecov.io/gh/jejacks0n/activeexperiment)
+
 Active Experiment is a framework for defining and running experiments. It supports using a variety of rollout and reporting strategies and/or services.
 
 Experiments can be everything from determining which query has the best performance, to which feature gets the most engagement, to rolling out a canary version of a new api service.
@@ -8,10 +15,10 @@ Experimentation is complex. There are a lot of different ways to run experiments
 
 ## Usage
 
-Start by defining an experiment class and adding some variants to it:
+Define your experiments using easily testable classes:
 
 ```ruby
-class MyExperiment < ApplicationExperiment
+class MyExperiment < ActiveExperiment::Base
   variant(:red) { "red" }
   variant(:blue) { "blue" }
 end
@@ -23,13 +30,13 @@ This experiment can be generated using the Rails generator:
 rails generate experiment my_experiment red blue
 ```
 
-Run the experiment anywhere in the application by providing it a context:
+Run the experiment with a context, like the current user, or the post being rendered:
 
 ```ruby
 MyExperiment.run(current_user) # => "red" or "blue"
 ```
 
-The experiment can also be run using local scope and helpers to override the default variants:
+Optionally override the defaults using local scope and helpers:
 
 ```ruby
 MyExperiment.run(current_user) do |experiment|
@@ -60,14 +67,18 @@ Source code can be downloaded as part of the project on GitHub:
 
 * https://github.com/jejacks0n/activeexperiment
 
-## Advanced Experimentation
+Adapters can be added to integrate with various services:
+
+- [Unleash adapter](https://github.com/jejacks0n/activeexperiment-unleash) 
+
+## Advanced experimentation
 
 This area provides a high level overview of the tools that more complex experiments can benefit from.
 
 For example, some experiments need to define a default variant (also known as a _control_) that will be assigned if the experiment is skipped:
 
 ```ruby
-class MyExperiment < ApplicationExperiment
+class MyExperiment < ActiveExperiment::Base
   variant(:red) { "red" }
   variant(:blue) { "blue" }
 
@@ -80,7 +91,7 @@ end
 Callbacks can be used to hook into the lifecycle when experiments are run, and can be targeted to when a specific variant has been assigned:
 
 ```ruby
-class MyExperiment < ApplicationExperiment
+class MyExperiment < ActiveExperiment::Base
   control { "default" }
   variant(:red) { "red" }
   variant(:blue) { "blue" }
@@ -100,7 +111,7 @@ end
 Segment rules can be used to assign specific variants for certain cases:
 
 ```ruby
-class MyExperiment < ApplicationExperiment
+class MyExperiment < ActiveExperiment::Base
   control { "default" }
   variant(:red) { "red" }
   variant(:blue) { "blue" }
@@ -129,7 +140,7 @@ A rollout can implement any number of different strategies, interact with servic
 Here's an example of using the default percent rollout with custom distribution rules:
 
 ```ruby
-class MyExperiment < ApplicationExperiment
+class MyExperiment < ActiveExperiment::Base
   variant(:red) { "red" }
   variant(:blue) { "blue" }
   variant(:green) { "green" }
@@ -139,12 +150,14 @@ class MyExperiment < ApplicationExperiment
 end
 ```
 
-### Defining Custom Rollouts
+### Defining custom rollouts
 
 Project specific rollouts can be defined and registered too. To illustrate, here's a custom rollout that inherits from the base rollout, uses a fictional feature flag library, and assigns a random variant.
 
 ```ruby
 class FeatureFlagRollout < ActiveExperiment::Rollouts::BaseRollout
+  register_as :feature_flag
+
   def skipped_for(experiment)
     !Feature.enabled?(@rollout_options[:flag_name] || experiment.name)
   end
@@ -153,8 +166,6 @@ class FeatureFlagRollout < ActiveExperiment::Rollouts::BaseRollout
     experiment.variant_names.sample
   end
 end
-
-ActiveExperiment::Rollouts.register(:feature_flag, FeatureRollout)
 ```
 
 This rollout can now be used the same way the built-in rollouts are:
@@ -165,7 +176,7 @@ class MyExperiment < ActiveExperiment::Base
   variant(:blue) { "blue" }
 
   # Using a custom rollout with options.
-  rollout :feature_flag, flag_name: "my_feature_flag"
+  use_rollout :feature_flag, flag_name: "my_feature_flag"
 end
 ```
 
@@ -206,7 +217,9 @@ Some simple reporting strategies might simply be added to `after_run` callbacks,
 A subscriber can be used to listen for experiment events and report them to a service. For example, here's a subscriber that reports to a fictional analytics service:
 
 ```ruby
-class MyAnalyticsSubscriber
+class MyAnalyticsSubscriber < ActiveSupport::Subscriber
+  attach_to :active_experiment
+
   def process_run(event)
     experiment = event.payload[:experiment]
     return if experiment.skipped?
@@ -217,8 +230,6 @@ class MyAnalyticsSubscriber
     )
   end
 end
-
-MyAnalyticsSubscriber.attach_to(:active_experiment)
 ```
 
 The following Active Experiment events are available for subscribers:
@@ -232,7 +243,7 @@ The following Active Experiment events are available for subscribers:
 
 In each of these events, the experiment instance is available in the `event.payload` hash.
 
-## Experiments in Views
+## Experiments in views
 
 Experiments can be used in views, just like in any other part of your application. Sometimes though, you might want to render markup inside your run block too, and to do this, you'll need to "capture" the experiment.
 
@@ -266,7 +277,7 @@ end
 <summary>Expand ERB example</summary>
 
 ```erb
-<%== MyExperiment.set(capture: self).run do |experiment| %>
+<%== MyExperiment.set(capture: self).run(current_user) do |experiment| %>
   <div class="container">
     <%= experiment.on(:red) do %>
       <button class="red-pill">Red</button>
@@ -279,7 +290,20 @@ end
 ```
 </details>
 
-## Client Side Experimentation
+If you don't need to capture the experiment, simply run like you would anywhere else:
+
+```erb
+<% MyExperiment.run(current_user) do |experiment| %>
+  <% experiment.on(:red) do %>
+    <button class="red-pill">Red</button>
+  <% end %>
+  <% experiment.on(:blue) do %>
+    <button class="blue-pill">Blue</button>
+  <% end %>
+<% end %>
+```
+
+## Client side experimentation
 
 While Active Experiment doesn't include any specific tooling for client side experimentation at this time, it does provide the ability to surface experiments in the client layer.
 
@@ -297,7 +321,7 @@ In the layout, the experiment data can be rendered as JSON for instance:
 Or each experiment can be iterated over and rendered individually:
 
 ```erb
-<% ActiveExperiment::Executed.experiments.each do |experiment| %>
+<% ActiveExperiment::Executed.as_array.each do |experiment| %>
   <meta name="<%= experiment.name %>" content="<%== experiment.serialize.to_json %>">
 <% end %>
 ```
@@ -323,7 +347,7 @@ test "stubbing experiments" do
   end
 
   stub_experiment(MyExperiment, skip: true) do
-    # Now all MyExperiment experiments be skipped.
+    # Now all MyExperiment experiments will be skipped.
   end
 end
 ```
@@ -332,24 +356,24 @@ Assertion helpers are also available:
 
 ```ruby
 test "asserting experiments" do
-  # no experiments has been run
+  # Assert that no experiments have been run.
   assert_no_experiments
 
   MyExperiment.run(id: 1)
 
-  # 1 experiment has been run
+  # Assert that 1 experiment has been run.
   assert_experiments 1
 
-  # 2 experiments expected within the block.
+  # Assert that within the block, 2 experiments will be run.
   assert_experiments 2 do
     MyExperiment.run(id: 2)
     MyExperiment.run(id: 3)
   end
   
-  # assert an experiment has been run with context.
+  # Assert an experiment has been run with a given context.
   assert_experiment_with(MyExperiment, context: { id: 1 })
   
-  # experiment with context, and a variant assigned expected within the block.
+  # Assert that within the block, a matching experiment will be run.
   assert_experiment_with(MyExperiment, variant: :red, context: { id: 4 }) do
     MyExperiment.set(variant: :red).run(id: 4)
   end
@@ -362,12 +386,20 @@ RSpec support can be added by requiring `active_experiment/rspec` in the appropr
 
 Active Experiment supports [GlobalID serialization](https://github.com/rails/globalid/) for experiment contexts. This is part of what makes it possible to utilize Active Record objects as context to consistently assign the same variant across multiple runs.
 
+## Similar and noteworthy projects
+
+- [Vanity](https://vanity.labnotes.org/) - Experiment Driven Development framework for Rails.
+- [Scientist](https://github.com/github/scientist) - A Ruby library for carefully refactoring critical paths.
+- [Gitlab::Experiment](https://gitlab.com/gitlab-org/ruby/gems/gitlab-experiment) - A framework for running experiments, by GitLab.
+- [Split](https://github.com/splitrb/split) - The Rack Based A/B testing framework.
+
 ## License
 
 Active Experiment is released under the MIT license:
 
 * https://opensource.org/licenses/MIT
 
-Copyright 2022 [jejacks0n](https://github.com/jejacks0n)
+Copyright 2022-2026 &copy; [jejacks0n](https://github.com/jejacks0n)
+
+## Make Code Not War ♥
 
-## Make Code Not War

data/lib/active_experiment/cache/active_record_cache_store.rb

@@ -1,6 +1,5 @@
 # frozen_string_literal: true
 
-require "monitor"
 require "active_record"
 
 module ActiveExperiment
@@ -28,11 +27,30 @@ module ActiveExperiment
     #
     #   create_table :active_experiment_cache_entries, id: false do |t|
     #     t.string :key, null: false
-    #     t.string :value, null: false
+    #     t.binary :value, null: false
     #   end
     #
     #   add_index :active_experiment_cache_entries, :key, unique: true
     #
+    # == Database Support
+    #
+    # The value column is binary because that's what entries serialize to -- a
+    # payload that leads with a null byte and isn't necessarily valid in the
+    # connection's encoding. PostgreSQL rejects that outright in a text or
+    # varchar column, and MySQL will too unless the column is binary.
+    #
+    # A +t.string :value+ column still works on SQLite, which doesn't enforce
+    # column types, so tables created before this was documented don't need
+    # migrating. Anything else wants +t.binary+.
+    #
+    # The statements are written with +?+ placeholders and quoted through the
+    # connection, so they don't depend on a particular adapter's bind syntax.
+    #
+    # SQLite and PostgreSQL are both covered by the test suite, the latter
+    # against a real server in CI. MySQL should work on the same basis -- it
+    # takes the same +x'..'+ binary literal SQLite does -- but nothing verifies
+    # that, so treat it as untested.
+    #
     # Once a table is created, the cache store can be used in an experiment:
     #
     #   class MyExperiment < ActiveExperiment::Base
@@ -44,15 +62,10 @@ module ActiveExperiment
     class ActiveRecordCacheStore < ActiveSupport::Cache::Store
       DEFAULT_TABLE_NAME = "active_experiment_cache_entries"
 
-      def initialize(options = nil)
-        super
-        @connection = ActiveRecord::Base.connection
-      end
-
       def length(options = nil)
         options = merged_options(options)
-        execute(<<~SQL).first["COUNT(key)"]
-          SELECT COUNT(key) FROM #{table_name(options)}
+        execute(<<~SQL).first["count"]
+          SELECT COUNT(key) AS count FROM #{table_name(options)}
         SQL
       end
 
@@ -66,24 +79,24 @@ module ActiveExperiment
       def delete_matched(matcher, options = nil)
         options = merged_options(options)
         execute(<<~SQL, key_matcher(matcher, options))
-          DELETE FROM #{table_name(options)} WHERE key LIKE $1
+          DELETE FROM #{table_name(options)} WHERE key LIKE ?
         SQL
       end
 
       private
         def read_entry(key, **options)
-          results = execute(<<~SQL, key)&.first.try(:[], "value")
-            SELECT value FROM #{table_name(options)} WHERE key = $1
+          result = execute(<<~SQL, key)
+            SELECT value FROM #{table_name(options)} WHERE key = ?
           SQL
 
-          deserialize_entry(results)
+          deserialize_entry(column_value(result, "value"))
         end
 
         def write_entry(key, entry, **options)
           return false if options[:unless_exist] && exist?(key, options)
 
-          execute(<<~SQL, key, serialize_entry(entry, **options))
-            INSERT INTO #{table_name(options)} (key, value) VALUES ($1, $2)
+          execute(<<~SQL, key, binary(serialize_entry(entry, **options)))
+            INSERT INTO #{table_name(options)} (key, value) VALUES (?, ?)
           SQL
 
           true
@@ -91,7 +104,7 @@ module ActiveExperiment
 
         def delete_entry(key, **options)
           execute(<<~SQL, key)
-            DELETE FROM #{table_name(options)} WHERE key = $1
+            DELETE FROM #{table_name(options)} WHERE key = ?
           SQL
 
           true
@@ -108,8 +121,24 @@ module ActiveExperiment
           namespace_key(source, options)
         end
 
-        def execute(sql, *args, prepare: true, **kws, &block)
-          @connection.exec_query(sql, "SQL", args, prepare: prepare, **kws, &block)
+        def column_value(result, column)
+          row = result&.first
+          return if row.nil?
+
+          type = result.column_types[column]
+          type ? type.deserialize(row[column]) : row[column]
+        end
+
+        def binary(payload)
+          ActiveRecord::Type::Binary::Data.new(payload)
+        end
+
+        def execute(sql, *binds)
+          sql = ActiveRecord::Base.sanitize_sql_array([sql, *binds]) if binds.any?
+
+          ActiveRecord::Base.connection_pool.with_connection do |connection|
+            connection.exec_query(sql, "ActiveExperiment::Cache")
+          end
         end
     end
   end

data/lib/active_experiment/callbacks.rb

@@ -60,7 +60,7 @@ module ActiveExperiment
 
     included do
       define_callbacks :run, skip_after_callbacks_if_terminated: true
-      private :__callbacks, :__callbacks?, :run_callbacks, :_run_callbacks, :_run_run_callbacks
+      private :__callbacks, :run_callbacks, :_run_callbacks, :_run_run_callbacks
     end
 
     # These methods will be included into any Active Experiment object, adding

data/lib/active_experiment/capturable.rb

@@ -82,20 +82,30 @@ module ActiveExperiment
     def on(*variant_names, &block)
       super
 
-      "{{#{variant_names.join("}}{{")}}}"
+      variant_names.map { |variant| capture_placeholder(variant) }.join
     end
 
     def run(&block)
       super
 
       if capturable?
-        @results = @capture.to_s.gsub(/{{([\w]+)}}/) { $1 == variant.to_s ? @results : "" }
+        @results = @capture.to_s.gsub(capture_placeholder_pattern) do
+          $1 == variant.to_s ? @results : ""
+        end
       else
         @results
       end
     end
 
     private
+      def capture_placeholder(variant)
+        "{{#{run_id}:#{variant}}}"
+      end
+
+      def capture_placeholder_pattern
+        /\{\{#{Regexp.escape(run_id)}:(\w+)\}\}/
+      end
+
       def resolve_results
         @results = capture { super }
       end

data/lib/active_experiment/core.rb

@@ -156,15 +156,15 @@ module ActiveExperiment
     # experiment.
     #
     # Calling this before the variant has been assigned or resolved will result
-    # in the variant being empty. Generally, this should be called after the
+    # in the variant being nil. Generally, this should be called after the
     # experiment has been run.
     def serialize
       {
-        "experiment" => name,
-        "run_id" => run_id,
-        "run_key" => run_key,
-        "variant" => variant.to_s,
-        "skipped" => skipped?
+        experiment: name,
+        run_id: run_id,
+        run_key: run_key,
+        variant: variant,
+        skipped: skipped?
       }
     end
 

data/lib/active_experiment/execution.rb

@@ -106,9 +106,14 @@ module ActiveExperiment
     #     experiment.on(:treatment) { "treatment" }
     #   end
     #
+    # Running an experiment that's already been run returns the result of the
+    # first run instead of running it again, and the block isn't called. That
+    # makes +run+ safe to call more than once -- repeatedly in a view, or by
+    # accident from within a run or variant block -- without resolving a second
+    # variant or recording a second execution.
+    #
     # Raises an ActiveExperiment::ExecutionError if there are no variants
-    # registered, or if the experiment is already running, in the case of
-    # accidentally calling run again within a run or variant block.
+    # registered.
     def run(&block)
       return @results if defined?(@results)
       raise ExecutionError, "No variants registered" if variant_names.empty?

data/lib/active_experiment/gem_version.rb

@@ -9,9 +9,9 @@ module ActiveExperiment
 
   module VERSION
     MAJOR = 0
-    MINOR = 1
+    MINOR = 2
     TINY  = 0
-    PRE   = "alpha"
+    PRE   = nil
 
     STRING = [MAJOR, MINOR, TINY, PRE].compact.join(".")
   end

data/lib/active_experiment/railtie.rb

@@ -5,6 +5,12 @@ require "active_experiment"
 
 module ActiveExperiment
   class Railtie < Rails::Railtie # :nodoc:
+    def self.default_digest_secret_key(app)
+      app.secret_key_base
+    rescue ArgumentError
+      nil
+    end
+
     config.active_experiment = ActiveSupport::OrderedOptions.new
     config.active_experiment.custom_rollouts = {}
     config.active_experiment.log_query_tags_around_run = true
@@ -24,23 +30,16 @@ module ActiveExperiment
     initializer "active_experiment.set_configs" do |app|
       options = app.config.active_experiment
       config.after_initialize do
-        options.digest_secret_key ||= app.secrets.secret_key_base
-
-        options.each do |k, v|
-          k = "#{k}="
-          if ActiveExperiment.respond_to?(k)
-            ActiveExperiment.send(k, v)
-          end
-        end
-      end
+        options.digest_secret_key ||= Railtie.default_digest_secret_key(app)
 
-      ActiveSupport.on_load(:active_experiment) do
-        options.each do |k, v|
-          k = "#{k}="
-          if ActiveExperiment.respond_to?(k)
-            ActiveExperiment.send(k, v)
-          elsif respond_to?(k)
-            send(k, v)
+        ActiveSupport.on_load(:active_experiment) do
+          options.each do |k, v|
+            setter = "#{k}="
+            if ActiveExperiment.respond_to?(setter)
+              ActiveExperiment.send(setter, v)
+            elsif respond_to?(setter)
+              send(setter, v)
+            end
           end
         end
       end

data/lib/active_experiment/rollout.rb

@@ -20,7 +20,7 @@ module ActiveExperiment
   #     variant(:red) { "red" }
   #     variant(:blue) { "blue" }
   #
-  #     rollout :percent, rules: { blue: 60, red: 40 }
+  #     use_rollout :percent, rules: { blue: 60, red: 40 }
   #   end
   #
   # An experiment might even be configured to use itself as a rollout. As

data/lib/active_experiment/rollouts/percent_rollout.rb

@@ -21,13 +21,13 @@ module ActiveExperiment
     #     variant(:blue) { }
     #
     #     # Assign even distribution to all variants.
-    #     rollout :percent
+    #     use_rollout :percent
     #
     #     # Assign 25% to control, 30% to red, and 45% to blue.
-    #     rollout :percent, rules: {control: 25, red: 30, blue: 45}
+    #     use_rollout :percent, rules: {control: 25, red: 30, blue: 45}
     #
     #     # Same as above, but using an array.
-    #     rollout :percent, rules: [25, 30, 45]
+    #     use_rollout :percent, rules: [25, 30, 45]
     #   end
     #
     # To use as the default, configure it to +:percent+.
@@ -47,8 +47,8 @@ module ActiveExperiment
         total = 0
 
         case rules
-        when Array then variants[rules.find_index { |percent| crc % 100 <= total += percent }]
-        when Hash then rules.find { |_, percent| crc % 100 <= total += percent }.first
+        when Array then variants[rules.find_index { |percent| crc % 100 < total += percent }]
+        when Hash then rules.find { |_, percent| crc % 100 < total += percent }.first
         else variants[crc % variants.length]
         end
       end

data/lib/active_experiment/rollouts/random_rollout.rb

@@ -22,10 +22,10 @@ module ActiveExperiment
     #     variant(:blue) { }
     #
     #     # Randomize between all variants, every run.
-    #     rollout :random
+    #     use_rollout :random
     #
     #     # Random, but once assigned, cache the assignment.
-    #     rollout :random, cache: true
+    #     use_rollout :random, cache: true
     #   end
     #
     # To use as the default, configure it to +:random+.

data/lib/active_experiment/rollouts.rb

@@ -83,6 +83,7 @@ module ActiveExperiment
       const_name = "#{name.to_s.camelize}#{ROLLOUT_SUFFIX}"
       case rollout
       when String, Pathname
+        registered_by_path << const_name
         autoload(const_name, rollout)
       when Class
         const_set(const_name, rollout)
@@ -95,17 +96,37 @@ module ActiveExperiment
     #
     # Raises an +ArgumentError+ if the rollout hasn't been registered.
     def self.lookup(name)
-      const_get("#{name.to_s.camelize}#{ROLLOUT_SUFFIX}")
+      const_name = "#{name.to_s.camelize}#{ROLLOUT_SUFFIX}"
+
+      # Scoped to constants on this module, so an unrelated top level
+      # FooRollout can't answer lookup(:foo) without having been registered.
+      # Rollouts registered by path are the exception -- the file they point at
+      # usually defines the class at the top level, so the constant isn't here
+      # once it loads, and the autoload entry that was is gone.
+      unless const_defined?(const_name, false) || registered_by_path.include?(const_name)
+        raise ArgumentError, "No rollout registered for #{name.inspect}"
+      end
+
+      const_get(const_name)
     rescue NameError
       raise ArgumentError, "No rollout registered for #{name.inspect}"
     end
 
+    def self.registered_by_path # :nodoc:
+      @registered_by_path ||= []
+    end
+
     # Base class for the included rollouts. Useful for custom rollouts.
     #
     # Any rollout that inherits from this class will be valid, not skipped, and
     # will assign the first defined variant unless the provided methods are
     # overridden.
     class BaseRollout
+      # Convenience method to register the rollout with Active Experiment.
+      def self.register_as(name)
+        Rollouts.register(name, self)
+      end
+
       def initialize(experiment_class, *args, **options, &block) # :nodoc:
         @experiment_class = experiment_class
         @rollout_args = args

data/lib/active_experiment/run_key.rb

@@ -1,5 +1,6 @@
 # frozen_string_literal: true
 
+require "date"
 require "digest/sha2"
 
 module ActiveExperiment
@@ -9,12 +10,31 @@ module ActiveExperiment
   # is generally referred to as the run key and can be used as the cache key
   # and for variant assignment.
   #
+  # For this to be meaningful, the digest has to be stable, meaning that the
+  # same context must produce the same run key in every process, forever. Two
+  # rules follow from that:
+  #
+  # - Hashes are digested in a sorted order, so the order the keys happen to be
+  #   written in at a given call site can't change the result, so these would
+  #   produce the same run key:
+  #
+  #   MyExperiment.run(account: account, user: user)
+  #   MyExperiment.run(user: user, account: account)
+  #
+  # - And objects that can't be identified stably are rejected rather than
+  #   silently digested by their +inspect+ output, which embeds a memory
+  #   address that changes every process. Contexts should be built from
+  #   +GlobalID::Identification+ objects (any Active Record model) and
+  #   primitives. You can disable this by configuring:
+  #
+  #   ActiveExperiment::Base.unsafe_context_digest = true
+  #
   # You can configure the details used in generating the digest by specifying a
   # secret key and a bit length. The secret key is used to salt the digest, and
   # the bit length is used to determine the length of the digest.
   #
-  # The secret key will default to +Rails.application.secrets.secret_key_base+
-  # when possible, and can be configured by:
+  # The secret key will default to +Rails.application.secret_key_base+ when
+  # possible, and can be configured by:
   #
   #   ActiveExperiment::Base.digest_secret_key = ENV["AE_SECRET_KEY"]
   #
@@ -25,31 +45,71 @@ module ActiveExperiment
   module RunKey
     extend ActiveSupport::Concern
 
+    # Included in every digest, and bumped whenever the way run keys are
+    # generated changes. Without it a change to the algorithm would silently
+    # reassign variants with no way to tell old keys from new ones.
+    DIGEST_VERSION = "2"
+    private_constant :DIGEST_VERSION
+
+    # Types that inspect to the same string in every process, and so can be
+    # digested directly.
+    STABLE_TYPES = [
+      NilClass, TrueClass, FalseClass, String, Symbol, Numeric, Time, Date
+    ].freeze
+    private_constant :STABLE_TYPES
+
     included do
       class_attribute :digest_secret_key, instance_writer: false, instance_predicate: false
       class_attribute :digest_bit_length, instance_writer: false, instance_predicate: false, default: 256
-      private :digest_secret_key, :digest_bit_length
+      class_attribute :unsafe_context_digest, instance_writer: false, instance_predicate: false, default: false
+      private :digest_secret_key, :digest_bit_length, :unsafe_context_digest
     end
 
     private
       def run_key_hexdigest(source)
-        source = source.keys + source.values if source.is_a?(Hash)
-        ingredients = Array(source).map { |value| identify_object(value).inspect }
-        ingredients.unshift(name, digest_secret_key)
+        ingredients = [DIGEST_VERSION, name, digest_secret_key, digest_ingredient(source)]
 
         ::Digest::SHA2.new(digest_bit_length).hexdigest(ingredients.join("|"))
       end
 
-      def identify_object(arg)
-        case arg
+      # Renders a value as a string that's identical for equivalent contexts.
+      def digest_ingredient(value)
+        case value
+        when Hash
+          # Sorted, so key order at the call site can't change the digest.
+          pairs = value.map { |k, v| "#{digest_ingredient(k)}=>#{digest_ingredient(v)}" }
+          "{#{pairs.sort.join(",")}}"
+        when Array
+          # Not sorted: order is meaningful in an array.
+          "[#{value.map { |v| digest_ingredient(v) }.join(",")}]"
         when GlobalID::Identification
-          arg.to_global_id.to_s rescue arg
+          global_id_ingredient(value)
+        when *STABLE_TYPES
+          value.inspect
         else
-          # TODO: maybe we should strip things out that might cause issues?
-          #   e.g. `#<User:0x00007f9b0a0b0e60>` is going to change every run,
-          #   and we don't want that to happen by accident.
-          arg
+          unstable_ingredient(value)
         end
       end
+
+      def global_id_ingredient(value)
+        value.to_global_id.to_s
+      rescue StandardError => error
+        # Most often a record that hasn't been saved yet, and so has no id to
+        # identify it by. Whatever the reason, there's no stable identity here.
+        unstable_ingredient(value, because: "#{error.class}: #{error.message}")
+      end
+
+      def unstable_ingredient(value, because: nil)
+        return value.inspect if unsafe_context_digest
+
+        raise ArgumentError, <<~MESSAGE.squish
+          Unable to generate a stable run key from the #{value.class} in the
+          #{name} context#{because ? " (#{because})" : ""}. Experiment contexts
+          should be built from GlobalID::Identification objects and primitives.
+          Set `ActiveExperiment::Base.unsafe_context_digest = true` to fall back
+          to the object's inspect output instead, accepting that the run key
+          can change every process and variant assignment won't be consistent.
+        MESSAGE
+      end
   end
 end

data/lib/active_experiment.rb

@@ -1,5 +1,13 @@
 # frozen_string_literal: true
 
+# GlobalID's URI parsing calls CGI.unescape without requiring it. Ruby 4.0
+# removed the CGI library, leaving cgi/escape as the only place that method
+# lives; before 4.0 cgi/escape can't stand on its own, so cgi/util is needed.
+if RUBY_VERSION >= "4.0"
+  require "cgi/escape"
+else
+  require "cgi/util"
+end
 require "global_id"
 require "active_support"
 require "active_support/rails"

data/lib/rails/generators/test_unit/experiment/templates/experiment_test.rb.tt

@@ -1,7 +1,7 @@
 require "test_helper"
 
 <% module_namespacing do -%>
-class <%= class_name %>ExperimentTest < ActiveJob::TestCase
+class <%= class_name %>ExperimentTest < ActiveExperiment::TestCase
   # test "the truth" do
   #   assert true
   # end

metadata

@@ -1,14 +1,13 @@
 --- !ruby/object:Gem::Specification
 name: activeexperiment
 version: !ruby/object:Gem::Version
-  version: 0.1.0.alpha
+  version: 0.2.0
 platform: ruby
 authors:
 - Jeremy Jackson
-autorequire:
 bindir: bin
 cert_chain: []
-date: 2022-12-10 00:00:00.000000000 Z
+date: 1980-01-02 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activesupport
@@ -16,14 +15,14 @@ dependencies:
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: 7.0.4
+        version: '7.1'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: 7.0.4
+        version: '7.1'
 - !ruby/object:Gem::Dependency
   name: globalid
   requirement: !ruby/object:Gem::Requirement
@@ -86,17 +85,16 @@ files:
 - lib/rails/generators/rspec/experiment/templates/experiment_spec.rb.tt
 - lib/rails/generators/test_unit/experiment/experiment_generator.rb
 - lib/rails/generators/test_unit/experiment/templates/experiment_test.rb.tt
-homepage: https://github.com/jejacks0n/active_experiment
+homepage: https://github.com/jejacks0n/activeexperiment
 licenses:
 - MIT
 metadata:
-  homepage_uri: https://github.com/jejacks0n/active_experiment
-  source_code_uri: https://github.com/jejacks0n/active_experiment
-  bug_tracker_uri: https://github.com/jejacks0n/active_experiment/issues
-  changelog_uri: https://github.com/jejacks0n/active_experiment/CHANGELOG.md
-  documentation_uri: https://github.com/jejacks0n/active_experiment/README.md
+  homepage_uri: https://github.com/jejacks0n/activeexperiment
+  source_code_uri: https://github.com/jejacks0n/activeexperiment
+  bug_tracker_uri: https://github.com/jejacks0n/activeexperiment/issues
+  changelog_uri: https://github.com/jejacks0n/activeexperiment/CHANGELOG.md
+  documentation_uri: https://github.com/jejacks0n/activeexperiment/README.md
   rubygems_mfa_required: 'true'
-post_install_message:
 rdoc_options: []
 require_paths:
 - lib
@@ -104,15 +102,14 @@ required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
-      version: 2.7.0
+      version: 3.2.0
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
-  - - ">"
+  - - ">="
     - !ruby/object:Gem::Version
-      version: 1.3.1
+      version: '0'
 requirements: []
-rubygems_version: 3.3.7
-signing_key:
+rubygems_version: 4.0.19
 specification_version: 4
 summary: Experiment framework with pluggable rollouts and instrumentation.
 test_files: []