chaotic_job 0.0.1 → 0.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 970f6e59807e48fc6bc71581fe2a12cd0c890429879b079dc8205f73aa74f2d5
-  data.tar.gz: 3378067f6be89a0f2a715ea60a18ada92af2d430d555d96756c6d3024802b5be
+  metadata.gz: d7a3e97903b960aad5ff163a8ea80d3d1c94d8427e86d0d06cb6c44ceffd7ad4
+  data.tar.gz: b4c2200a5436d1575f0f66d8647834514bd6563ac3fbf62a74014557d198ed90
 SHA512:
-  metadata.gz: 37a614599cc3b971ec6fb93b19d145be3cce82198bd631c945d996c1f1b30f0d0ac0f0bf643649548d780ffeecbb1a237fb5a5051043c394e087c7cbd04f94f8
-  data.tar.gz: 042ae03b7c80a1fc4570fb0617645f1d3801e331fa73dbd80b8a6bc11529136db4f99d8313bc2f7efb7f5808c66c792a419d8197d81734622cc174bf83bfe76c
+  metadata.gz: fa8f9e58bf1fa3eab4e4b6441e91824322e02043daea6fccb7f4d21844c739b34d376880ef03924fed43a7f7d49cf72fa536400698216ca58247cb287c49abd6
+  data.tar.gz: f8e83e868e13ed287f1ae9ff3b2560257b5c6fe1e6ba715bf400a93d2148c5a36d9372ac70aaadb5a498db198b3a0f0d9cd4bb3c00d138aa0ac4a8961c1e463a

data/.standard.yml

@@ -1,3 +1,7 @@
 # For available configuration options, see:
 #   https://github.com/standardrb/standard
 ruby_version: 3.0
+
+ignore:
+  - 'test/**/*':
+    - Lint/ConstantDefinitionInBlock

data/CHANGELOG.md

@@ -2,4 +2,13 @@
 
 ## [0.1.0] - 2024-11-06
 
+- Added `Journal` to log activity for tests
+- Added `Performer` to correctly perform jobs with retries
+- Added `Glitch` to inject transient failures into code execution
+- Added `Scenario` to define a glitch for a specific job
+- Added `Simulation` to run all possible error scenarios for a job
+- Added `Helpers` module to provide easy to use methods for testing
+
+## [0.0.1] - 2024-11-06
+
 - Initial release

data/README.md

@@ -3,13 +3,16 @@
 [![Gem Version](https://badge.fury.io/rb/chaotic_job.svg)](https://rubygems.org/gems/chaotic_job)
 [![Gem Downloads](https://img.shields.io/gem/dt/chaotic_job)](https://rubygems.org/gems/chaotic_job)
 ![Tests](https://github.com/fractaledmind/chaotic_job/actions/workflows/main.yml/badge.svg)
-![Coverage](https://img.shields.io/badge/code%20coverage-98%25-success)
+![Coverage](https://img.shields.io/badge/code%20coverage-92%25-success)
 [![Sponsors](https://img.shields.io/github/sponsors/fractaledmind?color=eb4aaa&logo=GitHub%20Sponsors)](https://github.com/sponsors/fractaledmind)
 [![Twitter Follow](https://img.shields.io/twitter/url?label=%40fractaledmind&style=social&url=https%3A%2F%2Ftwitter.com%2Ffractaledmind)](https://twitter.com/fractaledmind)
 
-## Installation
+> [!TIP]
+> This gem helps you test that your Active Jobs are reliable and resilient to failures. If you want to more easily *build* reliable and resilient Active Jobs, check out the companion [Acidic Job](https://github.com/fractaledmind/acidic_job/tree/alpha-1.0) gem.
+
+`ChaoticJob` provides a set of tools to help you test the reliability and resilience of your Active Jobs. It does this by allowing you to simulate various types of failures and glitches that can occur in a production environment.
 
-TODO: Replace `chaotic_job` with your gem name right after releasing it to RubyGems.org. Please do not do it earlier due to security reasons. Alternatively, replace this section with instructions to install your gem from git if you don't plan to release to RubyGems.org.
+## Installation
 
 Install the gem and add to the application's Gemfile by executing:
 
@@ -25,7 +28,166 @@ gem install chaotic_job
 
 ## Usage
 
-TODO: Write usage instructions here
+`ChaoticJob` should be used primarily by including its helpers into your Active Job tests:
+
+```ruby
+class TestYourJob < ActiveJob::TestCase
+  include ChaoticJob::Helpers
+
+  test "job is reliable" do
+    # ...
+  end
+end
+```
+
+The `ChaoticJob::Helpers` module provides 6 methods, 4 of which simply allow you to perform a job with retries in the proper way while the other 2 allow you to simulate failures and glitches.
+
+### Performing Jobs
+
+When testing job resilience, you will necessarily be testing how a job behaves when it retries. Unfortunately, the helpers provided by `ActiveJob::TestHelper` are tailored to testing the job's behavior on the first attempt.
+
+Specifically, when you want to perform a job and all of its retries, you would naturally reach for the [`perform_enqueued_jobs`](https://api.rubyonrails.org/classes/ActiveJob/TestHelper.html#method-i-perform_enqueued_jobs) method.
+
+> [!WARNING]
+> Do not use `perform_enqueued_jobs` to test job retries.
+
+```ruby
+perform_enqueued_jobs do
+  Job.perform_later
+end
+```
+
+But, this method does not behave as you would expect. Functionally, it overwrites the `enqueue` method to immediately perform the job, which means that instead of your job being performed in waves, the retry is performed _within_ the execution of the original job. This both confuses the logs and means the behavior in your tests are not representative of the behavior in production.
+
+In order to properly test job retries, you should use the `perform_all` method provided by `ChaoticJob::Helpers`:
+
+```ruby
+Job.perform_later
+perform_all
+```
+
+This helper will perform the job and all of its retries in the proper way, in waves, just like it would in production.
+
+If you need more control over which batches of jobs are performed, you can use the `perform_all_before` and `perform_all_after` methods. These are particularly useful if you need to test the behavior of a job that schedules another job. You can use these methods to perform only the original job and its retries, assert the state of the system, and then perform the scheduled job and its retries.
+
+```ruby
+JobThatSchedules.perform_later
+perform_all_before(4.seconds)
+assert_equal 1, enqueued_jobs.size
+assert_equal 2, performed_jobs.size
+
+perform_all_after(1.day)
+assert_equal 0, enqueued_jobs.size
+assert_equal 3, performed_jobs.size
+```
+
+You can pass either a `Time` object or an `ActiveSupport::Duration` object to these methods. And, to make the code as readable as possible, the `perform_all_before` is also aliased as the `perform_all_within` method. This allows you to write the example above as `perform_all_within(4.seconds)`.
+
+### Simulating Failures
+
+The helper methods for correctly performing jobs and their retries are useful, but they are not the primary reason for using `ChaoticJob`. The real power of this gem comes from its ability to simulate failures and glitches.
+
+The first helper you can use is the `run_scenario` method. A scenario is simply a set of glitches that will be injected into the specified code once. Here is an example:
+
+```ruby
+test "scenario of a simple job" do
+  class Job < ActiveJob::Base
+    def perform
+      step_1
+      step_2
+      step_3
+    end
+
+    def step_1; ChaoticJob::Journal.log; end
+    def step_2; ChaoticJob::Journal.log; end
+    def step_3; ChaoticJob::Journal.log; end
+  end
+
+  run_scenario(Job.new, glitch: ["before", "#{__FILE__}:6"])
+
+  assert_equal 5, ChaoticJob::Journal.total
+end
+```
+
+> [!NOTE]
+> The `ChaoticJob::Journal` class is a simple class that you can use to log things happening. It is used here to track the behavior of the job. It's has a lean, but highly useful, interface:
+> |method|description|
+> |---|---|
+> | `Journal.log` | log simply that something happened within the default scope |
+> | `Journal.log(thing, scope: :special)` | log a particular value within a particular scope |
+> | `Journal.total` | get the total number of logs under the default scope |
+> | `Journal.total(scope: :special)` | get the total number of logs under a particular scope |
+> | `Journal.all` | get all of the logged values under the default scope |
+> | `Journal.all(scope: :special)` | get all of the logged values under a particular scope |
+
+In this example, the job being tested is defined within the test case. You can, of course, also test jobs defined in your application. The key detail is the `glitch` keyword argument. A "glitch" is simply a tuple that describes precisely where you would like the failure to occur. The first element of the tuple is the location of the glitch, which can be either *before* or *after*. The second element is the location of the code that will be affected by the glitch, defined by its file path and line number. What this example scenario does is inject a glitch before the `step_3` method is called, here:
+
+```ruby
+def perform
+  step_1
+  step_2
+  # <-- HERE
+  step_3
+end
+```
+
+This glitch is a transient error, which are the only kind of errors that matter when testing resilience, as permanent errors mean your job will simply end up in the dead set. So, the glitch failure will occur once and only once, this forces a retry but does not prevent the job from completing.
+
+If you want to simulate multiple glitches affecting a job run, you can use the plural `glitches` keyword argument instead and pass an array of tuples:
+
+```ruby
+run_scenario(Job.new, glitches: [
+  ["before", "#{__FILE__}:6"],
+  ["before", "#{__FILE__}:7"]
+])
+```
+
+Scenario testing is useful to test the behavior of a job under a specific set of conditions. But, if you want to test the behavior of a job under a variety of conditions, you can use the `run_simulation` method. Instead of running a single scenario, a simulation will run the full set of possible error scenarios for your job.
+
+```ruby
+test "simulation of a simple job" do
+  class Job < ActiveJob::Base
+    def perform
+      step_1
+      step_2
+      step_3
+    end
+
+    def step_1 = ChaoticJob::Journal.log
+    def step_2 = ChaoticJob::Journal.log
+    def step_3 = ChaoticJob::Journal.log
+  end
+
+  run_simulation(Job.new) do |scenario|
+    assert_operator ChaoticJob::Journal.total, :>=, 3
+  end
+end
+```
+
+In this example, the simulation will run 12 scenarios:
+
+```ruby
+[
+  [[:after, "test_chaotic_job.rb:69"]],
+  [[:before, "test_chaotic_job.rb:75"]],
+  [[:after, "test_chaotic_job.rb:74"]],
+  [[:before, "test_chaotic_job.rb:74"]],
+  [[:after, "test_chaotic_job.rb:68"]],
+  [[:after, "test_chaotic_job.rb:70"]],
+  [[:before, "test_chaotic_job.rb:68"]],
+  [[:after, "test_chaotic_job.rb:73"]],
+  [[:after, "test_chaotic_job.rb:75"]],
+  [[:before, "test_chaotic_job.rb:69"]],
+  [[:before, "test_chaotic_job.rb:70"]],
+  [[:before, "test_chaotic_job.rb:73"]]
+]
+```
+
+It generates all possible glitch scenarios by performing your job once with a [`TracePoint`](https://docs.ruby-lang.org/en/master/TracePoint.html) that captures each line executed in your job. It then computes all possible glitch locations to produce a set of scenarios that will be run.[^1] The block that you pass to `run_simulation` will be called for each scenario, allowing you to make assertions about the behavior of your job under all scenarios.
+
+[^1]: The logic to determine all possible glitch locations essentially produces two locations, before and after, for each executed line. It then dedupes the functionally equivalent locations of `[:after, "file:1"]` and `[:before, "file:2"]`.
+
+In your application tests, you will want to make assertions about the side-effects that your job performs, asserting that they are correctly idempotent (only occur once) and result in the correct state.
 
 ## Development
 
@@ -35,7 +197,7 @@ To install this gem onto your local machine, run `bundle exec rake install`. To
 
 ## Contributing
 
-Bug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/chaotic_job. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [code of conduct](https://github.com/[USERNAME]/chaotic_job/blob/main/CODE_OF_CONDUCT.md).
+Bug reports and pull requests are welcome on GitHub at https://github.com/fractaledmind/chaotic_job. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [code of conduct](https://github.com/fractaledmind/chaotic_job/blob/main/CODE_OF_CONDUCT.md).
 
 ## License
 
@@ -43,4 +205,4 @@ The gem is available as open source under the terms of the [MIT License](https:/
 
 ## Code of Conduct
 
-Everyone interacting in the ChaoticJob project's codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct](https://github.com/[USERNAME]/chaotic_job/blob/main/CODE_OF_CONDUCT.md).
+Everyone interacting in the ChaoticJob project's codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct](https://github.com/fractaledmind/chaotic_job/blob/main/CODE_OF_CONDUCT.md).

data/Rakefile

@@ -3,7 +3,9 @@
 require "bundler/gem_tasks"
 require "minitest/test_task"
 
-Minitest::TestTask.create
+Minitest::TestTask.create :test do |t|
+  t.framework = nil
+end
 
 require "standard/rake"
 

data/lib/chaotic_job/glitch.rb

@@ -61,7 +61,7 @@ module ChaoticJob
       @breakpoints[path_with_line] ||= {}
       # contents = File.read(file_path).split("\n") unless @file_contents.key?(path_with_line)
       # @file_contents << contents
-      @breakpoints[path_with_line][position] = { block: block, executed: false }
+      @breakpoints[path_with_line][position] = {block: block, executed: false}
     end
 
     def execute_block(handler)

data/lib/chaotic_job/performer.rb

@@ -1,40 +1,84 @@
 # frozen_string_literal: true
 
-# Performer.new(Job1).perform_all
-# Performer.new(Job1).perform_all_within(time)
-# Performer.new(Job1).perform_all_after(time)
+require "active_job"
 
 module ChaoticJob
-  class Performer
-    include ActiveJob::TestHelper
+  module Performer
+    extend ActiveJob::TestHelper
+    extend self
 
-    def initialize(job, retry_window: 4)
-      @job = job
-      @retry_window = retry_window
+    def perform_all
+      until (jobs = enqueued_jobs_where).empty?
+        perform(jobs)
+      end
     end
 
-    def perform_all
-      @job.enqueue
-      enqueued_jobs_with.sort_by(&:scheduled_at, nil: :first).each do |job|
-        perform_job(job)
+    def perform_all_before(cutoff)
+      time = resolve_cutoff(cutoff)
+
+      until (jobs = enqueued_jobs_where(before: time)).empty?
+        perform(jobs)
       end
     end
+    alias_method :perform_all_within, :perform_all_before
+
+    def perform_all_after(cutoff)
+      time = resolve_cutoff(cutoff)
 
-    def perform_all_after(2.seconds.from_now)
+      until (jobs = enqueued_jobs_where(after: time)).empty?
+        perform(jobs)
+      end
     end
 
-    def perform_all_within(time)
+    def perform(jobs)
+      jobs.each do |payload|
+        queue_adapter.enqueued_jobs.delete(payload)
+        queue_adapter.performed_jobs << payload
+        instantiate_job(payload, skip_deserialize_arguments: true).perform_now
+      end.count
     end
 
-    private
+    def enqueued_jobs_where(before: nil, after: nil)
+      enqueued_jobs
+        .sort_by { |job| job["scheduled_at"] }
+        .select do |job|
+          scheduled_at = job[:at]
 
-    def perform_enqueued_jobs_only_with_retries
-      retry_window = Time.now + @retry_window
-      flush_enqueued_jobs(at: retry_window) until enqueued_jobs_with(at: retry_window).empty?
+          next true if scheduled_at.nil?
+
+          # Skip if the job is scheduled after the cutoff time
+          if before
+            next false if scheduled_at > before.to_f
+          end
+
+          # Skip if the job is scheduled before the cutoff time
+          if after
+            next false if scheduled_at < after.to_f
+          end
+
+          true
+        end
     end
 
-    def perform_any_enqueued_jobs_including_future_scheduled_ones
-      flush_enqueued_jobs until enqueued_jobs_with.empty?
+    def resolve_cutoff(cutoff)
+      time = case cutoff
+      in ActiveSupport::Duration
+        cutoff.from_now
+      in Time
+        cutoff
+      end
+      delta = (Time.now - time).abs
+      changeset = case delta
+      when 0..59                    # seconds
+        {usec: 0}
+      when 60..3599                 # minutes
+        {sec: 0, usec: 0}
+      when 3600..86_399             # hours
+        {min: 0, sec: 0, usec: 0}
+      when 86_400..Float::INFINITY  # days+
+        {hour: 0, min: 0, sec: 0, usec: 0}
+      end
+      time.change(**changeset)
     end
   end
 end

data/lib/chaotic_job/scenario.rb

@@ -21,7 +21,12 @@ module ChaoticJob
 
       ActiveSupport::Notifications.subscribed(->(event) { @events << event.dup }, @capture) do
         glitch.inject! do
-          block_given? ? yield : Performance.rehearse(@job)
+          if block_given?
+            yield
+          else
+            @job.enqueue
+            Performer.perform_all
+          end
         end
       end
 

data/lib/chaotic_job/simulation.rb

@@ -6,22 +6,23 @@
 # Simulation.new(job).scenarios
 module ChaoticJob
   class Simulation
-    def initialize(job, test: nil, variations: 100, seed: nil, depth: 3)
+    def initialize(job, depth: 1, variations: 100, test: nil, seed: nil)
       @template = job
-      @test = test
+      @depth = depth
       @variations = variations
+      @test = test
       @seed = seed || Random.new_seed
       @random = Random.new(@seed)
-      @depth = depth
     end
 
     def run(&callback)
       @template.class.retry_on RetryableError, attempts: @depth + 2, wait: 1, jitter: 0
 
-      debug "Running #{variants.size} simulations of the total #{permutations.size} possibilities..."
+      debug "👾 Running #{variants.size} simulations of the total #{permutations.size} possibilities..."
 
       scenarios.map do |scenario|
         run_scenario(scenario, &callback)
+        print "·"
       end
     end
 
@@ -70,7 +71,7 @@ module ChaoticJob
       trace = TracePoint.new(:line) do |tp|
         next if tp.defined_class == self.class
         next unless tp.path == job_file_path ||
-                    tp.defined_class == job_class
+          tp.defined_class == job_class
 
         @callstack << [tp.path, tp.lineno]
       end
@@ -81,9 +82,9 @@ module ChaoticJob
     end
 
     def run_scenario(scenario, &callback)
-      debug "Running simulation with scenario: #{scenario}"
+      debug "👾 Running simulation with scenario: #{scenario}"
       @test.before_setup
-      scenario.enact!
+      scenario.run
       @test.after_teardown
       callback.call(scenario)
     end

data/lib/chaotic_job/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module ChaoticJob
-  VERSION = "0.0.1"
+  VERSION = "0.1.0"
 end

data/lib/chaotic_job.rb

@@ -1,17 +1,46 @@
 # frozen_string_literal: true
 
 require_relative "chaotic_job/version"
+require_relative "chaotic_job/journal"
+require_relative "chaotic_job/performer"
+require_relative "chaotic_job/glitch"
+require_relative "chaotic_job/scenario"
+require_relative "chaotic_job/simulation"
 
 module ChaoticJob
-  class RetryableError < StandardError; end
+  class RetryableError < StandardError
+  end
 
   module Helpers
-    def run_simulation(job, ..., &block)
-      Simulation.new(job, test: self, ...).run(&block)
+    def perform_all
+      Performer.perform_all
+    end
+
+    def perform_all_within(time)
+      Performer.perform_all_within(time)
+    end
+
+    def perform_all_before(time)
+      Performer.perform_all_before(time)
+    end
+
+    def perform_all_after(time)
+      Performer.perform_all_after(time)
+    end
+
+    def run_simulation(job, depth: nil, variations: nil, &block)
+      seed = defined?(RSpec) ? RSpec.configuration.seed : Minitest.seed
+      kwargs = {test: self, seed: seed}
+      kwargs[:depth] = depth if depth
+      kwargs[:variations] = variations if variations
+      Simulation.new(job, **kwargs).run(&block)
     end
 
-    def run_scenario(job)
-      Scenario.new(job).run
+    def run_scenario(job, glitch: nil, glitches: nil, raise: nil, capture: nil)
+      kwargs = {glitches: glitches || [glitch]}
+      kwargs[:raise] = raise if raise
+      kwargs[:capture] = capture if capture
+      Scenario.new(job, **kwargs).run
     end
   end
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: chaotic_job
 version: !ruby/object:Gem::Version
-  version: 0.0.1
+  version: 0.1.0
 platform: ruby
 authors:
 - Stephen Margheim
@@ -9,7 +9,21 @@ autorequire:
 bindir: exe
 cert_chain: []
 date: 2024-11-07 00:00:00.000000000 Z
-dependencies: []
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: activejob
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '7.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '7.0'
 description:
 email:
 - stephen.margheim@gmail.com