chaotic_job 0.2.0 → 0.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 8241f1235f37ed46c1da18d53e18c6fce43fb5758273bb51dba8db332dddb5b6
-  data.tar.gz: fc86ee73cf18b16d6142f7111e5d1cfa446b46bfea2e1959eb7aa222a5bf42dd
+  metadata.gz: 98b01482666de5069e40dbe4eeb828cb527e4dd31004b3900a134ceaec5cb42a
+  data.tar.gz: f3df1be0fa3192aa842b74fa56754e56ed071f48d3c2fd027ca88d05da3f5e90
 SHA512:
-  metadata.gz: 7d32f1ab9748ea544d13c15692934634a6c3bcc01d602903e5e554abd229b7623a6a9aff111357031f474ec701fcd96131994604a121f9e55672eaed2616107b
-  data.tar.gz: c8ef73d8d07a091f59ebd1347ecc3f11fbd6ba6abc3eee218e251600ff156f748ec2a911d8108834cad96f9ce7ef59d1b3989606a27a827f1bb7bd93bfd9b062
+  metadata.gz: aff436f38402fddae8e62a7dbb441dc8949169abd1aa992fdf5e59f33e007f8dc3aba29787a3cea4bf3d421698ed1ac76c171388c26a3d2ab884e7880296ba84
+  data.tar.gz: 72986fa7274aade0e1fe7f9a5b71e4911467f18bf4929b90960ba62b03721611e46da8958c0536a92f92e98f4b4a2c7315e674db73bbcc6866cc177070a6f92f

data/CHANGELOG.md

@@ -1,5 +1,14 @@
 ## [Unreleased]
 
+## [0.4.0] - 2025-05-27
+
+- Allow a Glitch to be defined for a method call or method return [#4](https://github.com/fractaledmind/chaotic_job/pull/4)
+
+## [0.3.0] - 2024-12-17
+
+- Ensure that assertion failure messages raised within a simulation contain the scenario description
+- Add a `ChaoticJob.journal_entries` top-level method
+
 ## [0.2.0] - 2024-11-06
 
 - Update the `perform_all` helper method to `perform_all_jobs`

data/README.md

@@ -42,6 +42,10 @@ end
 
 The `ChaoticJob::Helpers` module provides 6 methods, 4 of which simply allow you to perform a job with retries in the proper way while the other 2 allow you to simulate failures and glitches.
 
+### Glitches
+
+A central concept in `ChaoticJob` is the _glitch_. A glitch is an error injected into the job execution flow via a [`TracePoint`](https://docs.ruby-lang.org/en/master/TracePoint.html). Glitches are transient errors, which means they occur _once_ and **only once**, making them perfect for testing a job's resilience to unpredictable failures that can occur while running jobs, like network issues, upstream API outages, rate limits, or  infrastructure failure. By default, `ChaoticJob` raises a custom error defined by the gem (`ChaoticJob::RetryableError`), which the internals of the gem ensure that the job under test is configured to retry on; you can, however, raise specific errors as needed when setting up your [scenarios](#simulating-failures). By forcing a retry via the error handling mechanisms of Active Job, glitches are a simple but effective way to test that your job is resilient to any kind of transient error that the job is configured to retry on.
+
 ### Performing Jobs
 
 When testing job resilience, you will necessarily be testing how a job behaves when it retries. Unfortunately, the helpers provided by `ActiveJob::TestHelper` are tailored to testing the job's behavior on the first attempt.
@@ -103,7 +107,7 @@ test "scenario of a simple job" do
     def step_3; ChaoticJob::Journal.log; end
   end
 
-  run_scenario(Job.new, glitch: ["before", "#{__FILE__}:6"])
+  run_scenario(Job.new, glitch: [:before_call, "Job#step_3"])
 
   assert_equal 5, ChaoticJob::Journal.total
 end
@@ -120,7 +124,16 @@ end
 > | `Journal.entries` | get all of the logged values under the default scope |
 > | `Journal.entries(scope: :special)` | get all of the logged values under a particular scope |
 
-In this example, the job being tested is defined within the test case. You can, of course, also test jobs defined in your application. The key detail is the `glitch` keyword argument. A "glitch" is simply a tuple that describes precisely where you would like the failure to occur. The first element of the tuple is the location of the glitch, which can be either *before* or *after*. The second element is the location of the code that will be affected by the glitch, defined by its file path and line number. What this example scenario does is inject a glitch before the `step_3` method is called, here:
+In this example, the job being tested is defined within the test case. You can, of course, also test jobs defined in your application. The key detail is the `glitch` keyword argument. A "glitch" is simply a tuple that describes precisely where you would like the failure to occur. The first element of the tuple is the _kind_ of glitch, which can be either `:before_line`, `:before_call`, or `:before_return`. These refer to the three kinds of `TracePoint` events that the gem hooks into. The second element is the _key_ for the code that will be affected by the glitch. This _key_ is a specially formatted string that defines the specific bit of code that the glitch should be inserted before. The different kinds of glitches are identified by different kinds of keys:
+|kind|key format|key example|
+|---|---|---|
+|`:before_line`|`"#{file_path}:#{line_number}"`|`"/Users/you/path/to/file.rb:123"`|
+|`:before_call`|`"#{YourClass.name}(.|#)#{method_name}"`|`"YourClass.some_class_method"`|
+|`:before_return`|`"#{YourClass.name}(.|#)#{method_name}"`|`"YourClass#some_instance_method"`|
+
+As you can see, the `:before_call` and `:before_return` keys are formatted the same, and can identify any instance (`#`) or class (`.`) method.
+
+What the example scenario above does is inject a glitch before the `step_3` method is called, here:
 
 ```ruby
 def perform
@@ -131,14 +144,33 @@ def perform
 end
 ```
 
-This glitch is a transient error, which are the only kind of errors that matter when testing resilience, as permanent errors mean your job will simply end up in the dead set. So, the glitch failure will occur once and only once, this forces a retry but does not prevent the job from completing.
+If we wanted to inject a glitch right before the `step_3` method finishes, we could define the glitch as a `:before_return`, like this:
+
+```ruby
+run_scenario(Job.new, glitch: [:before_return, "Job#step_3"])
+```
+
+and it would inject the transient error right here:
+
+```ruby
+def step_3
+  ChaoticJob::Journal.log
+  # <-- HERE
+end
+```
+
+Finally, if you need to inject a glitch right before a particular line of code is executed that is neither a method call nor a method return, you can use the `:before_line` key, like this:
+
+```ruby
+run_scenario(Job.new, glitch: [:before_line, "#{__FILE__}:6"])
+```
 
 If you want to simulate multiple glitches affecting a job run, you can use the plural `glitches` keyword argument instead and pass an array of tuples:
 
 ```ruby
 run_scenario(Job.new, glitches: [
-  ["before", "#{__FILE__}:6"],
-  ["before", "#{__FILE__}:7"]
+  [:before_call, "Job#step_1"],
+  [:before_return, "Job#step_1"]
 ])
 ```
 
@@ -164,28 +196,26 @@ test "simulation of a simple job" do
 end
 ```
 
-In this example, the simulation will run 12 scenarios:
+More specifically, it will create a scenario injecting a glitch before every line of code executed in your job. So, in this example, the simulation will run 12 scenarios:
 
 ```ruby
 [
-  [[:after, "test_chaotic_job.rb:69"]],
-  [[:before, "test_chaotic_job.rb:75"]],
-  [[:after, "test_chaotic_job.rb:74"]],
-  [[:before, "test_chaotic_job.rb:74"]],
-  [[:after, "test_chaotic_job.rb:68"]],
-  [[:after, "test_chaotic_job.rb:70"]],
-  [[:before, "test_chaotic_job.rb:68"]],
-  [[:after, "test_chaotic_job.rb:73"]],
-  [[:after, "test_chaotic_job.rb:75"]],
-  [[:before, "test_chaotic_job.rb:69"]],
-  [[:before, "test_chaotic_job.rb:70"]],
-  [[:before, "test_chaotic_job.rb:73"]]
+  [[:before_line, "test_chaotic_job.rb:69"]],
+  [[:before_line, "test_chaotic_job.rb:75"]],
+  [[:before_line, "test_chaotic_job.rb:74"]],
+  [[:before_line, "test_chaotic_job.rb:74"]],
+  [[:before_line, "test_chaotic_job.rb:68"]],
+  [[:before_line, "test_chaotic_job.rb:70"]],
+  [[:before_line, "test_chaotic_job.rb:68"]],
+  [[:before_line, "test_chaotic_job.rb:73"]],
+  [[:before_line, "test_chaotic_job.rb:75"]],
+  [[:before_line, "test_chaotic_job.rb:69"]],
+  [[:before_line, "test_chaotic_job.rb:70"]],
+  [[:before_line, "test_chaotic_job.rb:73"]]
 ]
 ```
 
-It generates all possible glitch scenarios by performing your job once with a [`TracePoint`](https://docs.ruby-lang.org/en/master/TracePoint.html) that captures each line executed in your job. It then computes all possible glitch locations to produce a set of scenarios that will be run.[^1] The block that you pass to `run_simulation` will be called for each scenario, allowing you to make assertions about the behavior of your job under all scenarios.
-
-[^1]: The logic to determine all possible glitch locations essentially produces two locations, before and after, for each executed line. It then dedupes the functionally equivalent locations of `[:after, "file:1"]` and `[:before, "file:2"]`.
+It generates all possible glitch scenarios by performing your job once with a [`TracePoint`](https://docs.ruby-lang.org/en/master/TracePoint.html) that captures each line executed in your job. It then computes all possible glitch locations to produce a set of scenarios that will be run. The block that you pass to `run_simulation` will be called for each scenario, allowing you to make assertions about the behavior of your job under all scenarios.
 
 In your application tests, you will want to make assertions about the side-effects that your job performs, asserting that they are correctly idempotent (only occur once) and result in the correct state.
 

data/lib/chaotic_job/glitch.rb

@@ -1,46 +1,47 @@
 # frozen_string_literal: true
 
-# Glitch.new.before("job_crucible.rb:10") { do_anything }
-# Glitch.new.after("job_crucible.rb:11") { do_anything }
+# Glitch.new.before_line("job_crucible.rb:10") { do_anything }
+# Glitch.new.before_call("Model#method", String, name: "Joel") { do_anything }
+# Glitch.new.before_return("Model#method", String, name: "Joel") { do_anything }
 # Glitch.new.inject! { execute code to glitch }
 
 module ChaoticJob
   class Glitch
     def initialize
       @breakpoints = {}
-      @file_contents = {}
     end
 
-    def before(path_with_line, &block)
-      set_breakpoint(path_with_line, :before, &block)
+    def before_line(key, &block)
+      set_breakpoint(key, :line, &block)
+      self
     end
 
-    def after(path_with_line, &block)
-      set_breakpoint(path_with_line, :after, &block)
+    def before_call(key, ...)
+      set_breakpoint(key, :call, ...)
+      self
     end
 
-    def inject!
-      prev_key = nil
-      trace = TracePoint.new(:line) do |tp|
-        key = "#{tp.path}:#{tp.lineno}"
-        # content = @file_contents[tp.path]
-        # line = content[tp.lineno - 1]
-        # next unless line.match? key
+    def before_return(key, return_type = nil, &block)
+      set_breakpoint(key, :return, retval: return_type, &block)
+      self
+    end
 
-        begin
-          execute_block(@breakpoints[prev_key][:after]) if prev_key && @breakpoints.key?(prev_key)
+    def inject!(&block)
+      breakpoints = @breakpoints
 
-          execute_block(@breakpoints[key][:before]) if @breakpoints.key?(key)
-        ensure
-          prev_key = key
-        end
+      trace = TracePoint.new(:line, :call, :return) do |tp|
+        # :nocov: SimpleCov cannot track code executed _within_ a TracePoint
+        key = derive_key(tp)
+        matchers = derive_matchers(tp)
+
+        next unless (defn = breakpoints.dig(key, tp.event))
+        next unless matches?(defn, matchers)
+
+        execute_block(defn)
+        # :nocov:
       end
 
-      trace.enable
-      yield if block_given?
-    ensure
-      trace.disable
-      execute_block(@breakpoints[prev_key][:after]) if prev_key && @breakpoints.key?(prev_key)
+      trace.enable(&block)
     end
 
     def all_executed?
@@ -57,11 +58,51 @@ module ChaoticJob
 
     private
 
-    def set_breakpoint(path_with_line, position, &block)
-      @breakpoints[path_with_line] ||= {}
-      # contents = File.read(file_path).split("\n") unless @file_contents.key?(path_with_line)
-      # @file_contents << contents
-      @breakpoints[path_with_line][position] = {block: block, executed: false}
+    def set_breakpoint(key, event, *args, retval: nil, **kwargs, &block)
+      @breakpoints[key] ||= {}
+      @breakpoints[key][event] = {args: args, kwargs: kwargs, retval: retval, block: block, executed: false}
+    end
+
+    # :nocov: SimpleCov cannot track code executed _within_ a TracePoint
+    def matches?(defn, matchers)
+      return true if defn.nil?
+      return true if matchers.nil?
+      return true if defn[:args].empty? && defn[:kwargs].empty? && defn[:retval].nil?
+
+      args = []
+      kwargs = {}
+      retval = nil
+
+      matchers.each do |kind, name, value|
+        case kind
+        when :req
+          args << value
+        when :opt
+          args << value if value
+        when :rest
+          args.concat(value) if value
+        when :keyreq
+          kwargs[name] = value
+        when :key
+          kwargs[name] = value
+        when :keyrest
+          kwargs.merge!(value) if value
+        when :retval
+          retval = value
+        end
+      end
+
+      defn[:args].each_with_index do |type, index|
+        return false unless type === args[index]
+      end
+
+      defn[:kwargs].each do |key, type|
+        return false unless type === kwargs[key]
+      end
+
+      return false unless defn[:retval] === retval
+
+      true
     end
 
     def execute_block(handler)
@@ -71,5 +112,33 @@ module ChaoticJob
       handler[:executed] = true
       handler[:block].call
     end
+
+    def derive_key(trace)
+      case trace.event
+      when :line
+        "#{trace.path}:#{trace.lineno}"
+      when :call, :return
+        if Module === trace.self
+          "#{trace.self}.#{trace.method_id}"
+        else
+          "#{trace.defined_class}##{trace.method_id}"
+        end
+      end
+    end
+
+    def derive_matchers(trace)
+      case trace.event
+      when :line
+        nil
+      when :call
+        trace.parameters.map do |type, name|
+          value = trace.binding.local_variable_get(name) rescue nil # standard:disable Style/RescueModifier
+          [type, name, value]
+        end
+      when :return
+        [[:retval, nil, trace.return_value]]
+      end
+    end
+    # :nocov:
   end
 end

data/lib/chaotic_job/journal.rb

@@ -18,6 +18,7 @@ module ChaoticJob
       @logs ||= {}
       @logs[scope] ||= []
       @logs[scope] << item
+      item
     end
 
     def size(scope: :default)
@@ -29,7 +30,7 @@ module ChaoticJob
     end
 
     def top(scope: :default)
-      entries&.first
+      entries(scope: scope)&.first
     end
   end
 end

data/lib/chaotic_job/scenario.rb

@@ -10,7 +10,7 @@ module ChaoticJob
     def initialize(job, glitches:, raise: RetryableError, capture: /active_job/)
       @job = job
       @glitches = glitches
-      @raise = raise
+      @raise = binding.local_variable_get(:raise)
       @capture = capture
       @glitch = nil
       @events = []
@@ -45,8 +45,8 @@ module ChaoticJob
 
     def glitch
       @glitch ||= Glitch.new.tap do |glitch|
-        @glitches.each do |position, location, _description|
-          glitch.public_send(position, location) { raise @raise }
+        @glitches.each do |kind, location, _description|
+          glitch.public_send(kind, location) { raise @raise }
         end
       end
     end

data/lib/chaotic_job/simulation.rb

@@ -27,20 +27,10 @@ module ChaoticJob
     end
 
     def permutations
-      callstack = capture_callstack.to_a
-      error_locations = callstack.each_cons(2).flat_map do |left, right|
-        lpath, lno = left
-        rpath, rno = right
-        key = "#{lpath}:#{lno}"
-        # inject an error before and after each non-adjacent line
-        if lpath == rpath && rno == lno + 1
-          [[:before, key]]
-        else
-          [[:before, key], [:after, key]]
-        end
+      callstack = capture_callstack.map { |path, line| "#{path}:#{line}" }
+      error_locations = callstack.map do |path, lineno|
+        [:before_line, "#{path}:#{lineno}"]
       end
-      final_key = callstack.last.join(":")
-      error_locations.push [:before, final_key], [:after, final_key]
       error_locations.permutation(@depth)
     end
 
@@ -84,9 +74,12 @@ module ChaoticJob
     def run_scenario(scenario, &callback)
       debug "👾 Running simulation with scenario: #{scenario}"
       @test.before_setup
+      @test.simulation_scenario = scenario.to_s
       scenario.run
       @test.after_teardown
       callback.call(scenario)
+    ensure
+      @test.simulation_scenario = nil
     end
 
     def clone_job_template

data/lib/chaotic_job/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module ChaoticJob
-  VERSION = "0.2.0"
+  VERSION = "0.4.0"
 end

data/lib/chaotic_job.rb

@@ -23,6 +23,14 @@ module ChaoticJob
     end
   end
 
+  def self.journal_entries(scope: nil)
+    if scope
+      Journal.entries(scope: scope)
+    else
+      Journal.entries
+    end
+  end
+
   def self.journal_size(scope: nil)
     if scope
       Journal.size(scope: scope)
@@ -40,6 +48,8 @@ module ChaoticJob
   end
 
   module Helpers
+    attr_accessor :simulation_scenario
+
     def perform_all_jobs
       Performer.perform_all
     end
@@ -58,12 +68,13 @@ module ChaoticJob
       kwargs = {test: self, seed: seed}
       kwargs[:depth] = depth if depth
       kwargs[:variations] = variations if variations
+      self.simulation_scenario = nil
       Simulation.new(job, **kwargs).run(&block)
     end
 
     def run_scenario(job, glitch: nil, glitches: nil, raise: nil, capture: nil, &block)
       kwargs = {glitches: glitches || [glitch]}
-      kwargs[:raise] = raise if raise
+      kwargs[:raise] = binding.local_variable_get(:raise) if binding.local_variable_get(:raise)
       kwargs[:capture] = capture if capture
       if block
         Scenario.new(job, **kwargs).run(&block)
@@ -71,5 +82,19 @@ module ChaoticJob
         Scenario.new(job, **kwargs).run
       end
     end
+
+    def assert(test, msg = nil)
+      return super unless @simulation_scenario
+
+      contextual_msg = lambda do
+        # copied from the original `assert` method in Minitest::Assertions
+        default_msg = "Expected #{mu_pp test} to be truthy."
+        custom_msg = msg.is_a?(Proc) ? msg.call : msg
+        full_msg = custom_msg || default_msg
+        "  #{@simulation_scenario}\n#{full_msg}"
+      end
+
+      super(test, contextual_msg)
+    end
   end
 end

metadata

@@ -1,14 +1,13 @@
 --- !ruby/object:Gem::Specification
 name: chaotic_job
 version: !ruby/object:Gem::Version
-  version: 0.2.0
+  version: 0.4.0
 platform: ruby
 authors:
 - Stephen Margheim
-autorequire:
 bindir: exe
 cert_chain: []
-date: 2024-11-07 00:00:00.000000000 Z
+date: 2025-05-27 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activejob
@@ -24,7 +23,6 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '7.0'
-description:
 email:
 - stephen.margheim@gmail.com
 executables: []
@@ -52,7 +50,6 @@ metadata:
   homepage_uri: https://github.com/fractaledmind/chaotic_job
   source_code_uri: https://github.com/fractaledmind/chaotic_job
   changelog_uri: https://github.com/fractaledmind/chaotic_job/CHANGELOG.md
-post_install_message:
 rdoc_options: []
 require_paths:
 - lib
@@ -67,8 +64,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.5.21
-signing_key:
+rubygems_version: 3.6.3
 specification_version: 4
 summary: Test ActiveJobs for reliability and resilience.
 test_files: []