chaotic_job 0.3.0 → 0.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: c84ddd59307328e8986c00f5c03da961e4d3910c4e42eb81f847e2703f2fe128
-  data.tar.gz: 9d65a1efddcb025569d5f97c8f87d3e5a920fee0c40f3b6d2bedf3f8c907b857
+  metadata.gz: 0c4da24d7fc4aa39030976d4f7249a14c87995ef5d5b05c523ad5e1e83724f37
+  data.tar.gz: 0a7be90ecb5381a5c9e67c6673bb41bc3ad2997c904ca42bfedbb4c701ef6bf0
 SHA512:
-  metadata.gz: e1fe18d93d38e9e15ecb011fc578b94543b2fd8c83c017d7df3f69d0ad243d7f676e8892562c346b07e0b86631891d16465f358c5c5999b639191cad34796543
-  data.tar.gz: 6837ab188e03fc32cb66feb689f561bc3a7e37dd98bbdff412504876f713d6d1ac4721b4cd4d0bd217a0423cfe938064b441203cd70b60f8c2bf1fe66b687502
+  metadata.gz: 11a054f050a3e95aa40c392e281b69bf2c43dbba3b91bce3d17473d589f4e2057d26a18af5071d7a5f4cb2b055775bbdd293af6c712792fbe1d042c4c99cca85
+  data.tar.gz: d6750f62adecc71334dd42cfb15e35e33fb36680be62a6c9d0d5f4e2ed29eb598f9f92fac6a09ec1a4636816789f13996bc0241faa5a1ef9eb4acac0577fc06f

data/CHANGELOG.md

@@ -1,5 +1,13 @@
 ## [Unreleased]
 
+## [0.5.0] - 2025-06-04
+
+- Add a Tracer class [#3](https://github.com/fractaledmind/chaotic_job/pull/3)
+
+## [0.4.0] - 2025-05-27
+
+- Allow a Glitch to be defined for a method call or method return [#4](https://github.com/fractaledmind/chaotic_job/pull/4)
+
 ## [0.3.0] - 2024-12-17
 
 - Ensure that assertion failure messages raised within a simulation contain the scenario description

data/README.md

@@ -44,7 +44,7 @@ The `ChaoticJob::Helpers` module provides 6 methods, 4 of which simply allow you
 
 ### Glitches
 
-A central concept in `ChaoticJob` is the _glitch_. A glitch is an error injected into the job execution flow via a [`TracePoint`](https://docs.ruby-lang.org/en/master/TracePoint.html). Glitches are transient errors, which means they occur once and only once, making them perfect for testing a job's resilience to unpredictable failures that can occur while running jobs, like network issues, upstream API outages, rate limits, or  infrastructure failure. By default, `ChaoticJob` raises a custom error defined by the gem (`ChaoticJob::RetryableError`), which the internals of the gem ensure that the job under test is configured to retry on; you can, however, raise specific errors as needed when setting up your [scenarios](#simulating-failures). By forcing a retry via the error handling mechanisms of Active Job, glitches are a simple but effective way to test that your job is resilient to any kind of transient error that the job is configured to retry on.
+A central concept in `ChaoticJob` is the _glitch_. A glitch is an error injected into the job execution flow via a [`TracePoint`](https://docs.ruby-lang.org/en/master/TracePoint.html). Glitches are transient errors, which means they occur _once_ and **only once**, making them perfect for testing a job's resilience to unpredictable failures that can occur while running jobs, like network issues, upstream API outages, rate limits, or  infrastructure failure. By default, `ChaoticJob` raises a custom error defined by the gem (`ChaoticJob::RetryableError`), which the internals of the gem ensure that the job under test is configured to retry on; you can, however, raise specific errors as needed when setting up your [scenarios](#simulating-failures). By forcing a retry via the error handling mechanisms of Active Job, glitches are a simple but effective way to test that your job is resilient to any kind of transient error that the job is configured to retry on.
 
 ### Performing Jobs
 
@@ -107,7 +107,7 @@ test "scenario of a simple job" do
     def step_3; ChaoticJob::Journal.log; end
   end
 
-  run_scenario(Job.new, glitch: ["before", "#{__FILE__}:6"])
+  run_scenario(Job.new, glitch: [:before_call, "Job#step_3"])
 
   assert_equal 5, ChaoticJob::Journal.total
 end
@@ -124,7 +124,16 @@ end
 > | `Journal.entries` | get all of the logged values under the default scope |
 > | `Journal.entries(scope: :special)` | get all of the logged values under a particular scope |
 
-In this example, the job being tested is defined within the test case. You can, of course, also test jobs defined in your application. The key detail is the `glitch` keyword argument. A "glitch" is simply a tuple that describes precisely where you would like the failure to occur. The first element of the tuple is the location of the glitch, which can be either *before* or *after* a line of code. The second element is the location of the code that will be affected by the glitch, defined by its file path and line number. What this example scenario does is inject a glitch before the `step_3` method is called, here:
+In this example, the job being tested is defined within the test case. You can, of course, also test jobs defined in your application. The key detail is the `glitch` keyword argument. A "glitch" is simply a tuple that describes precisely where you would like the failure to occur. The first element of the tuple is the _kind_ of glitch, which can be either `:before_line`, `:before_call`, or `:before_return`. These refer to the three kinds of `TracePoint` events that the gem hooks into. The second element is the _key_ for the code that will be affected by the glitch. This _key_ is a specially formatted string that defines the specific bit of code that the glitch should be inserted before. The different kinds of glitches are identified by different kinds of keys:
+|kind|key format|key example|
+|---|---|---|
+|`:before_line`|`"#{file_path}:#{line_number}"`|`"/Users/you/path/to/file.rb:123"`|
+|`:before_call`|`"#{YourClass.name}(.|#)#{method_name}"`|`"YourClass.some_class_method"`|
+|`:before_return`|`"#{YourClass.name}(.|#)#{method_name}"`|`"YourClass#some_instance_method"`|
+
+As you can see, the `:before_call` and `:before_return` keys are formatted the same, and can identify any instance (`#`) or class (`.`) method.
+
+What the example scenario above does is inject a glitch before the `step_3` method is called, here:
 
 ```ruby
 def perform
@@ -135,12 +144,33 @@ def perform
 end
 ```
 
+If we wanted to inject a glitch right before the `step_3` method finishes, we could define the glitch as a `:before_return`, like this:
+
+```ruby
+run_scenario(Job.new, glitch: [:before_return, "Job#step_3"])
+```
+
+and it would inject the transient error right here:
+
+```ruby
+def step_3
+  ChaoticJob::Journal.log
+  # <-- HERE
+end
+```
+
+Finally, if you need to inject a glitch right before a particular line of code is executed that is neither a method call nor a method return, you can use the `:before_line` key, like this:
+
+```ruby
+run_scenario(Job.new, glitch: [:before_line, "#{__FILE__}:6"])
+```
+
 If you want to simulate multiple glitches affecting a job run, you can use the plural `glitches` keyword argument instead and pass an array of tuples:
 
 ```ruby
 run_scenario(Job.new, glitches: [
-  ["before", "#{__FILE__}:6"],
-  ["before", "#{__FILE__}:7"]
+  [:before_call, "Job#step_1"],
+  [:before_return, "Job#step_1"]
 ])
 ```
 
@@ -166,30 +196,45 @@ test "simulation of a simple job" do
 end
 ```
 
-In this example, the simulation will run 12 scenarios:
+More specifically, it will create a scenario injecting a glitch before every line of code executed in your job. So, in this example, the simulation will run 12 scenarios:
 
 ```ruby
 [
-  [[:after, "test_chaotic_job.rb:69"]],
-  [[:before, "test_chaotic_job.rb:75"]],
-  [[:after, "test_chaotic_job.rb:74"]],
-  [[:before, "test_chaotic_job.rb:74"]],
-  [[:after, "test_chaotic_job.rb:68"]],
-  [[:after, "test_chaotic_job.rb:70"]],
-  [[:before, "test_chaotic_job.rb:68"]],
-  [[:after, "test_chaotic_job.rb:73"]],
-  [[:after, "test_chaotic_job.rb:75"]],
-  [[:before, "test_chaotic_job.rb:69"]],
-  [[:before, "test_chaotic_job.rb:70"]],
-  [[:before, "test_chaotic_job.rb:73"]]
+  [[:before_line, "test_chaotic_job.rb:69"]],
+  [[:before_line, "test_chaotic_job.rb:75"]],
+  [[:before_line, "test_chaotic_job.rb:74"]],
+  [[:before_line, "test_chaotic_job.rb:74"]],
+  [[:before_line, "test_chaotic_job.rb:68"]],
+  [[:before_line, "test_chaotic_job.rb:70"]],
+  [[:before_line, "test_chaotic_job.rb:68"]],
+  [[:before_line, "test_chaotic_job.rb:73"]],
+  [[:before_line, "test_chaotic_job.rb:75"]],
+  [[:before_line, "test_chaotic_job.rb:69"]],
+  [[:before_line, "test_chaotic_job.rb:70"]],
+  [[:before_line, "test_chaotic_job.rb:73"]]
 ]
 ```
 
-It generates all possible glitch scenarios by performing your job once with a [`TracePoint`](https://docs.ruby-lang.org/en/master/TracePoint.html) that captures each line executed in your job. It then computes all possible glitch locations to produce a set of scenarios that will be run.[^1] The block that you pass to `run_simulation` will be called for each scenario, allowing you to make assertions about the behavior of your job under all scenarios.
+It generates all possible glitch scenarios by performing your job once with a [`TracePoint`](https://docs.ruby-lang.org/en/master/TracePoint.html) that captures each line executed in your job. It then computes all possible glitch locations to produce a set of scenarios that will be run. The block that you pass to `run_simulation` will be called for each scenario, allowing you to make assertions about the behavior of your job under all scenarios.
+
+If you want to have the simulation run against a larger collection of scenarios, you can capture a custom callstack using the `ChaoticJob::Tracer` class and pass it to the `run_simulation` method as the `callstack` parameter. A `Tracer` is initialized with a block that determines which `TracePoint` events to collect. You then call `capture` with a block that defines the code to be traced. The default `Simulation` tracer collects all events for the passed job and then traces the job execution, essentially like this:
+
+```ruby
+job_file_path = YourJob.instance_method(:perform).source_location&.first
+tracer = Tracer.new { |tp| tp.path == job_file_path || tp.defined_class == YourJob }
+tracer.capture { YourJob.perform_now }
+```
+
+To capture, for example, a custom callstack that includes all events within your application, you can use the `ChaoticJob::Tracer` class as follows:
+
+```ruby
+tracer = ChaoticJob::Tracer.new { |tp| tp.path.start_with?(Rails.root.to_s) }
+tracer.capture { YourJob.perform_now }
+```
 
-[^1]: The logic to determine all possible glitch locations essentially produces two locations, before and after, for each executed line. It then dedupes the functionally equivalent locations of `[:after, "file:1"]` and `[:before, "file:2"]`.
+If you passed this callstack to your simulation, it would test what happens to your job whenever a transient glitch is injected anywhere in your application code called as a part of executing the job under test.
 
-In your application tests, you will want to make assertions about the side-effects that your job performs, asserting that they are correctly idempotent (only occur once) and result in the correct state.
+Remember, in your application tests, you will want to make assertions about the side-effects that your job performs, asserting that they are correctly idempotent (only occur once) and result in the correct state.
 
 ## Development
 

data/lib/chaotic_job/glitch.rb

@@ -1,46 +1,47 @@
 # frozen_string_literal: true
 
-# Glitch.new.before("job_crucible.rb:10") { do_anything }
-# Glitch.new.after("job_crucible.rb:11") { do_anything }
+# Glitch.new.before_line("job_crucible.rb:10") { do_anything }
+# Glitch.new.before_call("Model#method", String, name: "Joel") { do_anything }
+# Glitch.new.before_return("Model#method", String, name: "Joel") { do_anything }
 # Glitch.new.inject! { execute code to glitch }
 
 module ChaoticJob
   class Glitch
     def initialize
       @breakpoints = {}
-      @file_contents = {}
     end
 
-    def before(path_with_line, &block)
-      set_breakpoint(path_with_line, :before, &block)
+    def before_line(key, &block)
+      set_breakpoint(key, :line, &block)
+      self
     end
 
-    def after(path_with_line, &block)
-      set_breakpoint(path_with_line, :after, &block)
+    def before_call(key, ...)
+      set_breakpoint(key, :call, ...)
+      self
     end
 
-    def inject!
-      prev_key = nil
-      trace = TracePoint.new(:line) do |tp|
-        key = "#{tp.path}:#{tp.lineno}"
-        # content = @file_contents[tp.path]
-        # line = content[tp.lineno - 1]
-        # next unless line.match? key
+    def before_return(key, return_type = nil, &block)
+      set_breakpoint(key, :return, retval: return_type, &block)
+      self
+    end
 
-        begin
-          execute_block(@breakpoints[prev_key][:after]) if prev_key && @breakpoints.key?(prev_key)
+    def inject!(&block)
+      breakpoints = @breakpoints
 
-          execute_block(@breakpoints[key][:before]) if @breakpoints.key?(key)
-        ensure
-          prev_key = key
-        end
+      trace = TracePoint.new(:line, :call, :return) do |tp|
+        # :nocov: SimpleCov cannot track code executed _within_ a TracePoint
+        key = derive_key(tp)
+        matchers = derive_matchers(tp)
+
+        next unless (defn = breakpoints.dig(key, tp.event))
+        next unless matches?(defn, matchers)
+
+        execute_block(defn)
+        # :nocov:
       end
 
-      trace.enable
-      yield if block_given?
-    ensure
-      trace.disable
-      execute_block(@breakpoints[prev_key][:after]) if prev_key && @breakpoints.key?(prev_key)
+      trace.enable(&block)
     end
 
     def all_executed?
@@ -57,11 +58,51 @@ module ChaoticJob
 
     private
 
-    def set_breakpoint(path_with_line, position, &block)
-      @breakpoints[path_with_line] ||= {}
-      # contents = File.read(file_path).split("\n") unless @file_contents.key?(path_with_line)
-      # @file_contents << contents
-      @breakpoints[path_with_line][position] = {block: block, executed: false}
+    def set_breakpoint(key, event, *args, retval: nil, **kwargs, &block)
+      @breakpoints[key] ||= {}
+      @breakpoints[key][event] = {args: args, kwargs: kwargs, retval: retval, block: block, executed: false}
+    end
+
+    # :nocov: SimpleCov cannot track code executed _within_ a TracePoint
+    def matches?(defn, matchers)
+      return true if defn.nil?
+      return true if matchers.nil?
+      return true if defn[:args].empty? && defn[:kwargs].empty? && defn[:retval].nil?
+
+      args = []
+      kwargs = {}
+      retval = nil
+
+      matchers.each do |kind, name, value|
+        case kind
+        when :req
+          args << value
+        when :opt
+          args << value if value
+        when :rest
+          args.concat(value) if value
+        when :keyreq
+          kwargs[name] = value
+        when :key
+          kwargs[name] = value
+        when :keyrest
+          kwargs.merge!(value) if value
+        when :retval
+          retval = value
+        end
+      end
+
+      defn[:args].each_with_index do |type, index|
+        return false unless type === args[index]
+      end
+
+      defn[:kwargs].each do |key, type|
+        return false unless type === kwargs[key]
+      end
+
+      return false unless defn[:retval] === retval
+
+      true
     end
 
     def execute_block(handler)
@@ -71,5 +112,33 @@ module ChaoticJob
       handler[:executed] = true
       handler[:block].call
     end
+
+    def derive_key(trace)
+      case trace.event
+      when :line
+        "#{trace.path}:#{trace.lineno}"
+      when :call, :return
+        if Module === trace.self
+          "#{trace.self}.#{trace.method_id}"
+        else
+          "#{trace.defined_class}##{trace.method_id}"
+        end
+      end
+    end
+
+    def derive_matchers(trace)
+      case trace.event
+      when :line
+        nil
+      when :call
+        trace.parameters.map do |type, name|
+          value = trace.binding.local_variable_get(name) rescue nil # standard:disable Style/RescueModifier
+          [type, name, value]
+        end
+      when :return
+        [[:retval, nil, trace.return_value]]
+      end
+    end
+    # :nocov:
   end
 end

data/lib/chaotic_job/journal.rb

@@ -18,6 +18,7 @@ module ChaoticJob
       @logs ||= {}
       @logs[scope] ||= []
       @logs[scope] << item
+      item
     end
 
     def size(scope: :default)
@@ -29,7 +30,7 @@ module ChaoticJob
     end
 
     def top(scope: :default)
-      entries&.first
+      entries(scope: scope)&.first
     end
   end
 end

data/lib/chaotic_job/scenario.rb

@@ -10,7 +10,7 @@ module ChaoticJob
     def initialize(job, glitches:, raise: RetryableError, capture: /active_job/)
       @job = job
       @glitches = glitches
-      @raise = raise
+      @raise = binding.local_variable_get(:raise)
       @capture = capture
       @glitch = nil
       @events = []
@@ -45,8 +45,8 @@ module ChaoticJob
 
     def glitch
       @glitch ||= Glitch.new.tap do |glitch|
-        @glitches.each do |position, location, _description|
-          glitch.public_send(position, location) { raise @raise }
+        @glitches.each do |kind, location, _description|
+          glitch.public_send(kind, location) { raise @raise }
         end
       end
     end

data/lib/chaotic_job/simulation.rb

@@ -6,13 +6,16 @@
 # Simulation.new(job).scenarios
 module ChaoticJob
   class Simulation
-    def initialize(job, depth: 1, variations: 100, test: nil, seed: nil)
+    def initialize(job, callstack: nil, depth: 1, variations: 100, test: nil, seed: nil)
       @template = job
+      @callstack = callstack || capture_callstack
       @depth = depth
       @variations = variations
       @test = test
       @seed = seed || Random.new_seed
       @random = Random.new(@seed)
+
+      raise Error.new("callstack must be a generated via the ChaoticJob::Tracer") unless @callstack.is_a?(Stack)
     end
 
     def run(&callback)
@@ -27,20 +30,9 @@ module ChaoticJob
     end
 
     def permutations
-      callstack = capture_callstack.to_a
-      error_locations = callstack.each_cons(2).flat_map do |left, right|
-        lpath, lno = left
-        rpath, rno = right
-        key = "#{lpath}:#{lno}"
-        # inject an error before and after each non-adjacent line
-        if lpath == rpath && rno == lno + 1
-          [[:before, key]]
-        else
-          [[:before, key], [:after, key]]
-        end
+      error_locations = @callstack.map do |event, key|
+        ["before_#{event}", key]
       end
-      final_key = callstack.last.join(":")
-      error_locations.push [:before, final_key], [:after, final_key]
       error_locations.permutation(@depth)
     end
 
@@ -62,23 +54,13 @@ module ChaoticJob
     private
 
     def capture_callstack
-      return @callstack if defined?(@callstack)
-
-      @callstack = Set.new
       job_class = @template.class
       job_file_path = job_class.instance_method(:perform).source_location&.first
+      tracer = Tracer.new { |tp| tp.path == job_file_path || tp.defined_class == job_class }
+      callstack = tracer.capture { @template.dup.perform_now }
 
-      trace = TracePoint.new(:line) do |tp|
-        next if tp.defined_class == self.class
-        next unless tp.path == job_file_path ||
-          tp.defined_class == job_class
-
-        @callstack << [tp.path, tp.lineno]
-      end
-
-      trace.enable { @template.dup.perform_now }
       @template.class.queue_adapter.enqueued_jobs = []
-      @callstack
+      callstack
     end
 
     def run_scenario(scenario, &callback)

data/lib/chaotic_job/tracer.rb

@@ -0,0 +1,50 @@
+# frozen_string_literal: true
+
+# Tracer.new { |tp| tp.path.start_with? "foo" }
+# Tracer.new.capture { code_execution_to_trace_callstack }
+
+module ChaoticJob
+  class Tracer
+    def initialize(&constraint)
+      @constraint = constraint
+      @callstack = Stack.new
+    end
+
+    def capture(&block)
+      trace = TracePoint.new(:line, :call, :return) do |tp|
+        # :nocov: SimpleCov cannot track code executed _within_ a TracePoint
+        next if tp.defined_class == self.class
+        next unless @constraint.call(tp)
+
+        case tp.event
+        when :line
+          key = line_key(tp)
+        when :call, :return
+          key = call_key(tp)
+        end
+
+        @callstack << [tp.event, key]
+        # :nocov:
+      end
+
+      trace.enable(&block)
+      @callstack
+    end
+
+    private
+
+    # :nocov: SimpleCov cannot track code executed _within_ a TracePoint
+    def line_key(event)
+      "#{event.path}:#{event.lineno}"
+    end
+
+    def call_key(event)
+      if Module === event.self
+        "#{event.self}.#{event.method_id}"
+      else
+        "#{event.defined_class}##{event.method_id}"
+      end
+    end
+    # :nocov:
+  end
+end

data/lib/chaotic_job/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module ChaoticJob
-  VERSION = "0.3.0"
+  VERSION = "0.5.0"
 end

data/lib/chaotic_job.rb

@@ -3,13 +3,16 @@
 require_relative "chaotic_job/version"
 require_relative "chaotic_job/journal"
 require_relative "chaotic_job/performer"
+require_relative "chaotic_job/tracer"
 require_relative "chaotic_job/glitch"
 require_relative "chaotic_job/scenario"
 require_relative "chaotic_job/simulation"
+require "set"
 
 module ChaoticJob
-  class RetryableError < StandardError
-  end
+  Error = Class.new(StandardError)
+  RetryableError = Class.new(Error)
+  Stack = Set
 
   def self.log_to_journal!(item = nil, scope: nil)
     if item && scope
@@ -63,18 +66,19 @@ module ChaoticJob
       Performer.perform_all_after(time)
     end
 
-    def run_simulation(job, depth: nil, variations: nil, &block)
+    def run_simulation(job, depth: nil, variations: nil, callstack: nil, &block)
       seed = defined?(RSpec) ? RSpec.configuration.seed : Minitest.seed
       kwargs = {test: self, seed: seed}
       kwargs[:depth] = depth if depth
       kwargs[:variations] = variations if variations
+      kwargs[:callstack] = callstack if callstack
       self.simulation_scenario = nil
       Simulation.new(job, **kwargs).run(&block)
     end
 
     def run_scenario(job, glitch: nil, glitches: nil, raise: nil, capture: nil, &block)
       kwargs = {glitches: glitches || [glitch]}
-      kwargs[:raise] = raise if raise
+      kwargs[:raise] = binding.local_variable_get(:raise) if binding.local_variable_get(:raise)
       kwargs[:capture] = capture if capture
       if block
         Scenario.new(job, **kwargs).run(&block)
@@ -89,7 +93,7 @@ module ChaoticJob
       contextual_msg = lambda do
         # copied from the original `assert` method in Minitest::Assertions
         default_msg = "Expected #{mu_pp test} to be truthy."
-        custom_msg = original_msg.is_a?(Proc) ? original_msg.call : original_msg
+        custom_msg = msg.is_a?(Proc) ? msg.call : msg
         full_msg = custom_msg || default_msg
         "  #{@simulation_scenario}\n#{full_msg}"
       end

metadata

@@ -1,14 +1,13 @@
 --- !ruby/object:Gem::Specification
 name: chaotic_job
 version: !ruby/object:Gem::Version
-  version: 0.3.0
+  version: 0.5.0
 platform: ruby
-original_platform: ''
 authors:
 - Stephen Margheim
 bindir: exe
 cert_chain: []
-date: 2024-12-17 00:00:00.000000000 Z
+date: 2025-06-03 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activejob
@@ -42,6 +41,7 @@ files:
 - lib/chaotic_job/performer.rb
 - lib/chaotic_job/scenario.rb
 - lib/chaotic_job/simulation.rb
+- lib/chaotic_job/tracer.rb
 - lib/chaotic_job/version.rb
 - sig/chaotic_job.rbs
 homepage: https://github.com/fractaledmind/chaotic_job
@@ -65,7 +65,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.6.0
+rubygems_version: 3.6.3
 specification_version: 4
 summary: Test ActiveJobs for reliability and resilience.
 test_files: []