lab_coat 0.1.4 → 0.1.5

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 0c1c5580cec0ec677137fcbf23a90291fab80564c955a1d8b10fb48903f1f43b
-  data.tar.gz: 15540d166b6a0d6098c07979efc8b780ecebb26d5059ac3db1217489e2448e06
+  metadata.gz: 19e5e50fffea78d509d6c9e9aa7454460250a80917689f0651003a62ba43c7d9
+  data.tar.gz: e0f1a47e5cc78a8a3779dc17f53574b407b3191d870aadb28854ac58db567166
 SHA512:
-  metadata.gz: c84bde55abd4974f068a802f181552111d847c3eeca07e28046fe3492919ba25a12fdefb10ac547bc22fbce90c1785fa01d73084db769f51ebe55993d52b2cc9
-  data.tar.gz: 128f1a14f2a2ec4f370d40d5c5b147d35b26abd75f558d5f19755fb71e82cfe8aad4de8673835a82c9b12f21088642ea346aa3bba4d29cdcc7d334a15b4622db
+  metadata.gz: da1dcbed4d24ae2ec3e68ab82aa7eb97d6314fff05aa7746d6e7075548c70faae9f11e804af9ef69bd43185ca7d9a9fbe7bf7748472cae629db1e9ea27893cd7
+  data.tar.gz: 2f466dc529e11ae7967e93e72ff0b58d718e9b428891e913efe83b074ccb1e1d95ab50d1203e5e49e4a4c6ff8102145c630f422e742e4f736999f0dd6d495891

data/CHANGELOG.md

@@ -1,4 +1,7 @@
-## [0.1.4] - Unreleased
+## [0.1.5] - 2024-05-20
+- Adds `select_observation` to allow users to control which observation value is returned by the experiment. This helps with controlled rollout.
+
+## [0.1.4] - 2024-04-19
 - Remove the arity check, it's not very intuitive
 - Adds a `@context` that gets set at runtime and reset after each run. This is a much simpler way for methods to access a shared runtime context that can be set per `run!`.
 

data/README.md

@@ -9,6 +9,7 @@ This library is heavily inspired by [Scientist](https://github.com/github/scient
 - There is no app wide default experiment that gets magically set.
 - The `Result` only supports one comparison at a time, i.e. only 1 `candidate` is allowed per run.
 - The `duration` is measured using Ruby's `Benchmark`.
+- The final return value of the `Experiment` run can be selected dynamically.
 
 ## Installation
 
@@ -54,7 +55,7 @@ See the [`Experiment`](lib/lab_coat/experiment.rb) class for more details.
 |`candidate`|The new behavior you want to test.|
 |`control`|The existing or default behavior. This will always be returned from `#run!`.|
 |`enabled?`|Returns a `Boolean` that controls whether or not the experiment runs.|
-|`publish!`|This is not _technically_ required, but `Experiments` are not useful unless you can analyze the results. Override this method to record the `Result` however you wish.|
+|`publish!`|This is _technically_ not required, but `Experiments` are not useful unless you can analyze the results. Override this method to record the `Result` however you wish.|
 
 > [!IMPORTANT]
 > The `#run!` method accepts arbitrary key word arguments and stores them in an instance variable called `@context` in case you need to provide data at runtime. You can access the runtime context via `@context` or `context`. The runtime context **is reset** after each run.
@@ -63,9 +64,11 @@ See the [`Experiment`](lib/lab_coat/experiment.rb) class for more details.
 
 |Method|Description|
 |---|---|
-|`compare`|Whether or not the result is a match. This is how you can run complex/custom comparisons.|
-|`ignore?`|Whether or not the result should be ignored. Ignored `Results` are still passed to `#publish!`|
+|`compare`|Whether or not the result is a match. This is how you can run complex/custom comparisons. Defaults to `control.value == candidate.value`.|
+|`ignore?`|Whether or not the result should be ignored. Ignored `Results` are still passed to `#publish!`. Defaults to `false`, i.e. nothing is ignored.|
+|`publishable_value`|The data to publish for a given `Observation`. This value is only for publishing and **is not** returned by `run!`. Defaults to `Observation#value`.|
 |`raised`|Callback method that's called when an `Observation` raises.|
+|`select_observation`|Override this method to select which observation's `value` should be returned by the `Experiment`. Defaults to the control `Observation`.|
 
 > [!TIP]
 > You should create a shared base class(es) to maintain consistency across experiments within your app.
@@ -90,7 +93,7 @@ class ApplicationExperiment < LabCoat::Experiment
   def publish!(result)
     payload = result.to_h.merge(
       user_id: @user.id, # e.g. something from the `Experiment` state
-      build_number: context.version # e.g. something from the runtime context
+      build_number: context[:version] # e.g. something from the runtime context
     )
     YourO11yService.track_experiment_result(payload)
   end
@@ -122,6 +125,21 @@ class ApplicationExperiment < LabCoat::Experiment
 end
 ```
 
+You might want to rollout the new code path in certain cases.
+
+```ruby
+# application_experiment.rb
+class ApplicationExperiment < LabCoat::Experiment
+   def select_observation(result)
+    if result.matched? || YourFeatureFlagService.flag_enabled?(@user.id, @context[:rollout_flag_name])
+      candidate
+    else
+      super
+    end
+  end
+end
+```
+
 ### Make some `Observations` via `run!`
 
 You don't have to create an `Observation` yourself; that happens automatically when you call `Experiment#run!`. The control and candidate `Observations` are packaged into a `Result` and [passed to `Experiment#publish!`](#publish-the-result).
@@ -151,8 +169,10 @@ def compare(control, candidate)
 end
 
 def ignore?(control, candidate)
+  # You might ignore runs that throw errors and handle them separately via `raised`.
   return true if control.raised? || candidate.raised?
-  return true if candidate.value.some_guard?
+  # You might ignore runs where the candidate meets some condition.
+  return true if candidate.value.some_condition?
 
   false
 end
@@ -185,19 +205,19 @@ A `Result` represents a single run of an `Experiment`.
 |`matched?`|Whether or not the `control` and `candidate` match, as defined by `Experiment#compare`|
 |`to_h`|A hash representation of the `Result`. Useful for publishing and/or reporting.|
 
-The `Result` is passed to your implementation of `#publish!` when an `Experiment` is finished running. The `to_h` method on a Result is a good place to start and might be sufficient for most experiments. You might want to `merge` additional data such as the runtime `context` or other state if you find that relevant for analysis.
+The `Result` is passed to your implementation of `#publish!` when an `Experiment` is finished running. The `to_h` method on a Result is a good place to start and might be sufficient for most experiments. You might want to include additional data such as the runtime `context` or other state if you find that relevant for analysis.
 
 ```ruby
 # your_experiment.rb
 def publish!(result)
   return if result.ignored?
 
-  puts result.to_h.merge(run_context: context)
+  puts result.to_h.merge(context:)
 end
 ```
 
 > [!NOTE]
-> All `Results` are passed to `publish!`, **including ignored ones**. It is your responsibility to call the `ignored?` method and handle those as you wish.
+> All `Results` are passed to `publish!`, **including ignored ones**. It is your responsibility to check the `ignored?` method and handle those as you wish.
 
 You can always access all of the attributes of the `Result` and its `Observations` directly to fully customize what your experiment publishing looks like.
 

data/lib/lab_coat/experiment.rb

@@ -12,7 +12,7 @@ module LabCoat
 
     # Override this method to control whether or not the experiment runs.
     # @return [TrueClass, FalseClass]
-    def enabled?(...)
+    def enabled?
       raise InvalidExperimentError, "`#enabled?` must be implemented in your Experiment class."
     end
 
@@ -38,12 +38,16 @@ module LabCoat
     end
 
     # Override this method to define which results are ignored. Must return a boolean.
+    # @param control [LabCoat::Observation] The control `Observation`.
+    # @param candidate [LabCoat::Observation] The candidate `Observation`.
+    # @return [TrueClass, FalseClass]
     def ignore?(_control, _candidate)
       false
     end
 
     # Called when the control and/or candidate observations raise an error.
     # @param observation [LabCoat::Observation]
+    # @return [void]
     def raised(observation); end
 
     # Override this method to transform the value for publishing. This could mean turning the value into something
@@ -56,11 +60,23 @@ module LabCoat
     # Override this method to publish the `Result`. It's recommended to override this once in an application wide base
     # class.
     # @param result [LabCoat::Result] The result of this experiment.
+    # @return [void]
     def publish!(result); end
 
+    # Override this method to select which observation's `value` should be returned by the `Experiment`. Defaults to
+    # the control `Observation`. This method is only called if the `Experiment` is enabled. This is useful for rolling
+    # out new behavior in a controlled way.
+    # @param result [LabCoat::Result] The result of the experiment.
+    # @return [LabCoat::Observation] Either the control or candidate `Observation` from the given `Result`.
+    def select_observation(result)
+      result.control
+    end
+
     # Runs the control and candidate and publishes the result. Always returns the result of `control`.
+    # It's not recommended to override this method.
     # @param context [Hash] Any data needed at runtime.
-    def run!(**context)
+    # @return [Object] An `Observation` value.
+    def run!(**context) # rubocop:disable Metrics/MethodLength
       # Set the context for this run.
       @context = context
 
@@ -69,6 +85,7 @@ module LabCoat
       raised(control_obs) if control_obs.raised?
       return control_obs.value unless enabled?
 
+      # Run the candidate.
       candidate_obs = Observation.new("candidate", self) { candidate }
       raised(candidate_obs) if candidate_obs.raised?
 
@@ -76,23 +93,11 @@ module LabCoat
       result = Result.new(self, control_obs, candidate_obs)
       publish!(result)
 
-      # Reset the context for this run.
-      @context = {}
-
       # Always return the control.
-      control_obs.value
-    end
-
-    private
-
-    # Because `run!` forwards arbitrary args to `#enabled?`, `control`, and `candidate`, the methods must have the same
-    # arity. Otherwise
-    def enforce_arity!
-      return if %i[enabled? control candidate].map { |m| method(m).arity }.uniq.size == 1
-
-      raise InvalidExperimentError,
-            "The `#enabled?`, `#control` and `#candidate` methods must have the same arity. All runtime args passed " \
-            "to `#run!` are forwarded to these methods."
+      select_observation(result).value.tap do
+        # Reset the context for this run. Done here so that `select_observation` has access to the runtime context.
+        @context = {}
+      end
     end
   end
 end

data/lib/lab_coat/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module LabCoat
-  VERSION = "0.1.4"
+  VERSION = "0.1.5"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: lab_coat
 version: !ruby/object:Gem::Version
-  version: 0.1.4
+  version: 0.1.5
 platform: ruby
 authors:
 - Omkar Moghe
 autorequire:
 bindir: exe
 cert_chain: []
-date: 2024-04-19 00:00:00.000000000 Z
+date: 2024-05-21 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: minitest