acidic_job 1.0.0.pre29 → 1.0.0.rc1

data/README.md

@@ -1,106 +1,92 @@
-# AcidicJob
+# 🧪 Acidic Job
 
-[![Gem Version](https://badge.fury.io/rb/acidic_job.svg)](https://badge.fury.io/rb/acidic_job)
-![main workflow](https://github.com/fractaledmind/acidic_job/actions/workflows/main.yml/badge.svg)
+[![Gem Version](https://badge.fury.io/rb/acidic_job.svg)](https://rubygems.org/gems/acidic_job)
+[![Gem Downloads](https://img.shields.io/gem/dt/acidic_job)](https://rubygems.org/gems/acidic_job)
+![Tests](https://github.com/fractaledmind/acidic_job/actions/workflows/main.yml/badge.svg)
+![Coverage](https://img.shields.io/badge/code%20coverage-98%25-success)
+[![Codacy Badge](https://app.codacy.com/project/badge/Grade/e0df63f7a6f141d4aecc3c477314fdb2)](https://www.codacy.com/gh/fractaledmind/acidic_job/dashboard?utm_source=github.com&utm_medium=referral&utm_content=fractaledmind/acidic_job&utm_campaign=Badge_Grade)
+[![Sponsors](https://img.shields.io/github/sponsors/fractaledmind?color=eb4aaa&logo=GitHub%20Sponsors)](https://github.com/sponsors/fractaledmind)
+[![Twitter Follow](https://img.shields.io/twitter/url?label=%40fractaledmind&style=social&url=https%3A%2F%2Ftwitter.com%2Ffractaledmind)](https://twitter.com/fractaledmind)
 
-### Idempotent operations for Rails apps (for ActiveJob or Sidekiq)
+> [!WARNING]
+> This is the README for the _new_ release candidate of v1, which is a major refactor from the [previous release candidate of v1](https://github.com/fractaledmind/acidic_job/tree/v1.0.0.pre29). If you are looking for the stable release, please refer to the [v0.9.0 README](https://github.com/fractaledmind/acidic_job/tree/v0.9.0).
 
-At the conceptual heart of basically any software are "operations"—the discrete actions the software performs. Rails provides a powerful abstraction layer for building operations in the form of `ActiveJob`, or we Rubyists can use the tried and true power of pure `Sidekiq`. With either we can easily trigger from other Ruby code throughout our Rails application (controller actions, model methods, model callbacks, etc.); we can run operations both synchronously (blocking execution and then returning its response to the caller) and asychronously (non-blocking and the caller doesn't know its response); and we can also retry a specific operation if needed seamlessly.
 
-However, in order to ensure that our operational jobs are _robust_, we need to ensure that they are properly [idempotent and transactional](https://github.com/mperham/sidekiq/wiki/Best-Practices#2-make-your-job-idempotent-and-transactional). As stated in the [GitLab Sidekiq Style Guide](https://docs.gitlab.com/ee/development/sidekiq_style_guide.html#idempotent-jobs):
+## Durable execution workflows for Active Job
 
->As a general rule, a worker can be considered idempotent if:
->  * It can safely run multiple times with the same arguments.
->  * Application side-effects are expected to happen only once (or side-effects of a second run do not have an effect).
+Rails applications today frequently need to coordinate complex multi-step operations across external services, databases, and systems. While Active Job provides eventual consistency guarantees, it doesn't address the challenges of managing stateful, long-running operations that must be resilient to failures, timeouts, and partial completions. `AcidicJob` enhances Active Job with durable execution workflows that automatically track state and resiliently handle retries, while providing you the tools to ensure your operations are truly idempotent through careful state management and IO awareness.
 
-This is, of course, far easier said than done. Thus, `AcidicJob`.
+With AcidicJob, you can write reliable and repeatable multi-step distributed operations that are Atomic ⚛️, Consistent 🤖, Isolated 🕴🏼, and Durable ⛰️.
 
-`AcidicJob` provides a framework to help you make your operational jobs atomic ⚛️, consistent 🤖, isolated 🕴🏼, and durable ⛰️. Its conceptual framework is directly inspired by a truly wonderful loosely collected series of articles written by Brandur Leach, which together lay out core techniques and principles required to make an HTTP API properly ACIDic:
-
-1. https://brandur.org/acid
-2. https://brandur.org/http-transactions
-3. https://brandur.org/job-drain
-4. https://brandur.org/idempotency-keys
-
-`AcidicJob` brings these techniques and principles into the world of a standard Rails application.
 
 ## Installation
 
-Add this line to your application's Gemfile:
+Install the gem and add to the application's Gemfile by executing:
 
-```ruby
-gem 'acidic_job'
+```sh
+bundle add acidic_job
 ```
 
-And then execute:
-
-    $ bundle install
+If `bundler` is not being used to manage dependencies, install the gem by executing:
 
-Or simply execute to install the gem yourself:
-
-    $ bundle add acidic_job
+```sh
+gem install acidic_job
+```
 
-Then, use the following command to copy over the `AcidicJob::Run` migration file.
+After installing the gem, run the installer:
 
-```
+```sh
 rails generate acidic_job:install
 ```
 
+The installer will create a migration file at `db/migrate` to setup the tables that the gem requires.
+
+
 ## Usage
 
-`AcidicJob` is a concern that you `include` into your base `ApplicationJob`.
+`AcidicJob` provides a simple DSL to define linear workflows within your job. In order to define and execute a workflow within a particular job, simply `include AcidicJob::Workflow`. This will provide the `execute_workflow` method to the job, which takes a `unique_by` keyword argument and a block where you define the steps of the workflow:
 
 ```ruby
-class ApplicationJob < ActiveJob::Base
-  include AcidicJob
-end
-```
+class Job < ActiveJob::Base
+  include AcidicJob::Workflow
 
-This is useful because the module needs to be mixed into any and all jobs that you want to either make acidic or enqueue acidicly.
+  def perform(arg)
+    @arg = arg
 
-It provides a suite of functionality that empowers you to create complex, robust, and _acidic_ jobs.
+    execute_workflow(unique_by: @arg) do |w|
+      w.step :step_1, transactional: true
+      w.step :step_2
+      w.step :step_3
+    end
+  end
 
-### TL;DR
+  # ...
+end
+```
 
-#### Key Features
 
-* **Transactional Steps**  
-    break your job into a series of steps, each of which will be run within an acidic database transaction, allowing retries to jump back to the last "recovery point".
-* **Steps that Await Jobs**  
-    have workflow steps await other jobs, which will be enqueued and processed independently, and only when they all have finished will the parent job be re-enqueued to continue the workflow
-* **Iterable Steps**  
-    define steps that iterate over some collection fully until moving on to the next step
-* **Persisted Attributes**  
-    when retrying jobs at later steps, we need to ensure that data created in previous steps is still available to later steps on retry.
-* **Transactionally Staged Jobs**  
-    enqueue additional jobs within the acidic transaction safely
-* **Custom Idempotency Keys**  
-    use something other than the job ID for the idempotency key of the job run
-* **Sidekiq Callbacks**  
-    bring ActiveJob-like callbacks into your pure Sidekiq Workers
-* **Run Finished Callbacks**  
-    set callbacks for when a job run finishes fully
+## Key Features
 
 
-### Transactional Steps
+### Workflow Steps
 
-The first and foundational feature `acidic_job` provides is the `with_acidity` method, which takes a block of transactional step methods (defined via the `step`) method:
+The foundational feature `AcidicJob` provides is the `execute_workflow` method, which takes a block where you define your workflow's step methods:
 
 ```ruby
-class RideCreateJob < ActiveJob::Base
-  include AcidicJob
-
+class RideCreateJob < AcidicJob::Base
   def perform(user_id, ride_params)
     @user = User.find(user_id)
     @params = ride_params
 
-    with_acidity providing: { ride: nil } do
-      step :create_ride_and_audit_record
-      step :create_stripe_charge
-      step :send_receipt
+    execute_workflow(unique_by: [@user, @params]) do |workflow|
+      workflow.step :create_ride_and_audit_record, transactional: true
+      workflow.step :create_stripe_charge
+      workflow.step :send_receipt
     end
   end
 
+  private
+
   def create_ride_and_audit_record
     # ...
   end
@@ -115,256 +101,158 @@ class RideCreateJob < ActiveJob::Base
 end
 ```
 
-`with_acidity` takes only the `providing:` named parameter and a block where you define the steps of this operation. `step` simply takes the name of a method available in the job. That's all!
-
-Now, each execution of this job will find or create an `AcidicJob::Run` record, which we leverage to wrap every step in a database transaction. Moreover, this database record allows `acidic_job` to ensure that if your job fails on step 3, when it retries, it will simply jump right back to trying to execute the method defined for the 3rd step, and won't even execute the first two step methods. This means your step methods only need to be idempotent on failure, not on success, since they will never be run again if they succeed.
-
-
-### Steps that Await Jobs
-
-By simply adding the `awaits` option to your step declarations, you can attach any number of additional, asynchronous jobs to your step. This is profoundly powerful, as it means that you can define a workflow where step 2 is started _if and only if_ step 1 succeeds, but step 1 can have 3 different jobs enqueued on 3 different queues, each running in parallel. Once all 3 jobs succeed, `acidic_job` will re-enqueue the parent job and it will move on to step 2. That's right, you can have workers that are _executed in parallel_, **on separate queues**, and _asynchronously_, but are still **blocking**—as a group—the next step in your workflow! This unlocks incredible power and flexibility for defining and structuring complex workflows and operations.
-
-```ruby
-class RideCreateJob < ActiveJob::Base
-  include AcidicJob
-
-  def perform(user_id, ride_params)
-    @user = User.find(user_id)
-    @params = ride_params
-
-    with_acidity providing: { ride: nil } do
-      step :create_ride_and_audit_record, awaits: [SomeJob, AnotherJob]
-      step :create_stripe_charge
-      step :send_receipt
-    end
-  end
-end
-```
+The `unique_by` keyword argument is used to define the unique identifier for a particular execution of the workflow. This helps to ensure that the workflow is idempotent, as retries of the job will correctly resume the pre-existing workflow execution. The `unique_by` argument can be anything that `JSON.dump` can handle.
 
-If you need to await a job that takes arguments, you can prepare that job along with its arguments using the `with` class method that `acidic_job` will add to your jobs:
+The block passed to `execute_workflow` is where you define the steps of the workflow. Each step is defined by calling the `step` method on the yielded workflow builder object. The `step` method takes the name of a method in the job that will be executed as part of the workflow. The `transactional` keyword argument can be used to ensure that the step is executed within a database transaction.
 
-```ruby
-class RideCreateJob < ActiveJob::Base
-  include AcidicJob
+The `step` method is the only method available on the yielded workflow builder object, and it simply takes the name of a method available in the job.
 
-  def perform(user_id, ride_params)
-    @user = User.find(user_id)
-    @params = ride_params
+> [!IMPORTANT]
+> In order to craft resilient workflows, you need to ensure that each step method wraps a single unit of IO-bound work. You **must not** have a step method that performs multiple IO-bound operations, like writing to your database and calling an external API. Steps should be as granular and self-contained as possible. This allows your own logic to be more durable in case of failures in third-party APIs, network errors, and so on. So, the rule of thumb is to have only one _state mutation_ per step. And this rule of thumb graduates to a hard and fast rule for _foreign state mutations_. You **must** only have **one** foreign state mutation per step, where a foreign state mutation is any operation that writes to a system beyond your own boundaries. This might be creating a charge on Stripe, adding a DNS record, or sending an email.[^1]
 
-    with_acidity providing: { ride: nil } do
-      step :create_ride_and_audit_record, awaits: awaits: [SomeJob.with('argument_1', keyword: 'value'), AnotherJob.with(1, 2, 3, some: 'thing')]
-      step :create_stripe_charge
-      step :send_receipt
-    end
-  end
-end
-```
+[^1]: I first learned this rule from [Brandur Leach](https://twitter.com/brandur) reminds in his post on [Implementing Stripe-like Idempotency Keys in Postgres](https://brandur.org/idempotency-keys).
 
-If your step awaits multiple jobs (e.g. `awaits: [SomeJob, AnotherJob.with('argument_1', keyword: 'value')]`), your top level workflow job will only continue to the next step once **all** of the jobs in your `awaits` array have finished.
+When your job calls `execute_workflow`, you initiate a durable execution workflow. The execution is made durable via the `AcidicJob::Execution` record that is created. This record is used to track the state of the workflow, and to ensure that if a step fails, the job can be retried and the workflow will pick up where it left off. This is a powerful feature that allows you to build resilient workflows that can handle failures gracefully, because if your job fails on step 3, when it retries, it will simply jump right back to trying to execute the method defined for the 3rd step, _**and won't even execute the first two step methods**_. This means your step methods only need to be idempotent on failure, not on success, since they will never be run again if they succeed.
 
-In some cases, you may need to _dynamically_ determine the collection of jobs that the step should wait for; in these cases, you can pass the name of a method to the `awaits` option:
+By default, each step is executed and upon completion, the `AcidicJob::Execution` record is updated to reflect the completion of that step. This default makes sense for _foreign state mutations_, but for _local state mutations_, i.e. writes to your application's primary database, it makes sense to wrap the both the step execution and the record update in a single transaction. This is done by passing the `transactional` option to the `step` method:
 
 ```ruby
-class RideCreateJob < ActiveJob::Base
-  include AcidicJob
-  set_callback :finish, :after, :delete_run_record
-
-  def perform(user_id, ride_params)
-    @user = User.find(user_id)
-    @params = ride_params
-
-    with_acidity providing: { ride: nil } do
-      step :create_ride_and_audit_record, awaits: :dynamic_awaits
-      step :create_stripe_charge
-      step :send_receipt
-    end
-  end
-  
-  def dynamic_awaits
-    if @params["key"].present?
-      [SomeJob.with('argument_1', keyword: 'value')]
-    else
-      [AnotherJob.with(1, 2, 3, some: 'thing')]
-    end
-  end
+execute_workflow(unique_by: [@user, @params]) do |workflow|
+  workflow.step :create_ride_and_audit_record, transactional: true
+  workflow.step :create_stripe_charge
+  workflow.step :send_receipt
 end
 ```
 
 
-### Iterable Steps
-
-Sometimes our workflows have steps that need to iterate over a collection and perform an action for each item in the collection before moving on to the next step in the workflow. In these cases, we can use the `for_each` option when defining our step to specific the collection, and `acidic_job` will pass each item into your step method for processing, keeping the same transactional guarantees as for any step. This means that if your step encounters an error in processing any item in the collection, when your job is retried, the job will jump right back to that step and right back to that item in the collection to try again.
-
-```ruby
-class ExampleJob < ActiveJob::Base
-  include AcidicJob
-
-  def perform(record:)
-    with_acidity providing: { collection: [1, 2, 3, 4, 5] } do
-      step :process_item, for_each: :collection
-      step :next_step
-    end
-  end
-  
-  def process_item(item)
-    # do whatever work needs to be done with this individual item
-  end
-end
-```
-
-**Note:** This feature relies on the "Persisted Attributes" feature detailed below. This means that you can only iterate over collections that ActiveRecord can serialize.
-
-
 ### Persisted Attributes
 
-Any objects passed to the `providing` option on the `with_acidity` method are made available across retries. This means that you can set an attribute in step 1, access it in step 2, have step 2 fail, have the job retry, jump directly back to step 2 on retry, and have that object still accessible. This is done by serializing all objects to a field on the `AcidicJob::Run` and manually providing getters and setters that sync with the database record.
+In addition to the workflow steps, `AcidicJob` also provides you with an isolated context where you can persist data that is needed across steps and  across retries. This means that you can set an attribute in step 1, access it in step 2, have step 2 fail, have the job retry, jump directly back to step 2 on retry, and have that object still accessible. This is available via the `ctx` object, which is an instance of `AcidicJob::Context`, in all of your step methods:
 
 ```ruby
-class RideCreateJob < ActiveJob::Base
-  include AcidicJob
-
+class RideCreateJob < AcidicJob::Base
   def perform(user_id, ride_params)
     @user = User.find(user_id)
     @params = ride_params
 
-    with_acidity providing: { ride: nil } do
-      step :create_ride_and_audit_record
-      step :create_stripe_charge
-      step :send_receipt
+    execute_workflow(unique_by: [@user, @params]) do |workflow|
+      workflow.step :create_ride_and_audit_record, transactional: true
+      workflow.step :create_stripe_charge
+      workflow.step :send_receipt
     end
   end
 
   def create_ride_and_audit_record
-    self.ride = Ride.create!
+    ctx[:ride] = @user.rides.create(@params)
   end
 
   def create_stripe_charge
-    Stripe::Charge.create(amount: 20_00, customer: @ride.user)
+    Stripe::Charge.create(amount: 20_00, customer: ctx[:ride].user)
   end
 
   # ...
 end
 ```
 
-**Note:** This does mean that you are restricted to objects that can be serialized by **`ActiveJob`** (for more info, see [here](https://edgeguides.rubyonrails.org/active_job_basics.html#supported-types-for-arguments)). This means you can persist ActiveRecord models, and any simple Ruby data types, but you can't persist things like Procs or custom class instances, for example.
+As you see, you access the `ctx` object as if it were a hash, though it is a custom `AcidicJob::Context` object that persists the data to `AcidicJob::Value` records associated with the workflow's `AcidicJob::Execution` record.
 
-**Note:** You will note the use of `self.ride = ...` in the code sample above. In order to call the attribute setter method that will sync with the database record, you _must_ use this style. `@ride = ...` and/or `ride = ...` will both fail to sync the value with the database record.
+> [!NOTE]
+> This does mean that you are restricted to objects that can be serialized by **_`ActiveJob`_** (for more info, see [the Rails Guide on `ActiveJob`](https://edgeguides.rubyonrails.org/active_job_basics.html#supported-types-for-arguments)). This means you can persist Active Record models, and any simple Ruby data types, but you can't persist things like Procs or custom class instances, for example. `AcidicJob` does, though, extend the standard set of supported types to include Active Job instances themselves, unpersisted Active Record instances, and Ruby exceptions.
 
-The default pattern you should follow when defining your `perform` method is to make any values that your `step` methods need access to, but are present at the start of the `perform` method simply instance variables. You only need to `provide` attributes that will be set _during a step_. This means, the initial value will almost always be `nil`.
+As the code sample also suggests, you should always use standard instance variables defined in your `perform` method when you have any values that your `step` methods need access to, but are present at the start of the `perform` method. You only need to persist attributes that will be set _during a step_ via `ctx`.
 
 
-### Transactionally Staged Jobs
+### Custom Workflow Uniqueness
 
-A standard problem when inside of database transactions is enqueuing other jobs. On the one hand, you could enqueue a job inside of a transaction that then rollbacks, which would leave that job to fail and retry and fail. On the other hand, you could enqueue a job that is picked up before the transaction commits, which would mean the records are not yet available to this job.
+Resilient workflows must, necessarily, be idempotent.[^2] Idempotency is a fancy word that simply means your jobs need to be able to be run multiple times while any side effects only happen once. In order for your workflow executions to be idempotent, `AcidicJob` needs to know what constitutes a unique execution of your job. You can define what makes your job unique by passing the `unique_by` argument when executing the workflow:
 
-In order to mitigate against such issues without forcing you to use a database-backed job queue, `acidic_job` provides `perform_acidicly` and `deliver_acidicly` methods to "transactionally stage" enqueuing other jobs from within a step (whether another ActiveJob or a Sidekiq::Worker or an ActionMailer delivery). These methods will create a new `AcidicJob::Run` record, but inside of the database transaction of the `step`. Upon commit of that transaction, a model callback pushes the job to your actual job queue.  Once the job has been successfully performed, the `AcidicJob::Run` record is deleted so that this table doesn't grow unbounded and unnecessarily.
+[^2]: This is echoed both by [Mike Perham](https://www.mikeperham.com), the creator of Sidekiq, in the Sidekiq [docs on best practices](https://github.com/mperham/sidekiq/wiki/Best-Practices#2-make-your-job-idempotent-and-transactional) by the GitLab team in their [Sidekiq Style Guide](https://docs.gitlab.com/ee/development/sidekiq/idempotent_jobs.html).
 
 ```ruby
-class RideCreateJob < ActiveJob::Base
-  include AcidicJob
+class Job < ActiveJob::Base
+  include AcidicJob::Workflow
 
-  def perform(user_id, ride_params)
-    @user = User.find(user_id)
-    @params = ride_params
-    
-    with_acidity providing: { ride: nil } do
-      step :create_ride_and_audit_record
-      step :create_stripe_charge
-      step :send_receipt
+  def perform(record:)
+    execute_workflow(unique_by: [record.id, record.status]) do |w|
+      w.step :step_1
+      w.step :step_2
+      w.step :step_3
     end
   end
-
-  # ...
-
-  def send_receipt
-    RideMailer.with(user: @user, ride: @ride).confirm_charge.delivery_acidicly
-  end
-end
 ```
 
+> [!TIP]
+> You should think carefully about what constitutes a unique execution of a workflow. Imagine you had a workflow job for balance transers. Jill transfers $10 to John. Your system **must** be able to differentiate between retries of this transfer and new independent transfers. If you were only to use the `sender`, `recipient`, and `amount` as your `unique_by` values, then if Jill tries to transfer another $10 to John at some point in the future, that work will be considered a retry of the first transfer and not a new transfer.
 
-### Custom Idempotency Keys
-
-By default, `AcidicJob` uses the job identifier provided by the queueing system (ActiveJob or Sidekiq) as the idempotency key for the job run. The idempotency key is what is used to guarantee that no two runs of the same job occur. However, sometimes we need particular jobs to be idempotent based on some other criteria. In these cases, `AcidicJob` provides a collection of tools to allow you to ensure the idempotency of your jobs.
 
-Firstly, you can configure your job class to explicitly use either the job identifier or the job arguments as the foundation for the idempotency key. A job class that calls the `acidic_by_job_id` class method (which is the default behavior) will simply make the job run's idempotency key the job's identifier:
+### Orchestrating steps
 
-```ruby
-class ExampleJob < ActiveJob::Base
-  include AcidicJob
-  acidic_by_job_id
-
-  def perform
-  end
-end
-```
+In addition to the workflow definition setup, `AcidicJob` also provides a couple of methods to precisely control the workflow step execution. From within any step method, you can call either `repeat_step!` or `halt_step!`.
 
-Conversely, a job class can use the `acidic_by_job_args` method to configure that job class to use the arguments passed to the job as the foundation for the job run's idempotency key:
+`repeat_step!` will cause the current step to be re-executed on the next iteration of the workflow. This is useful when you need to traverse a collection of items and perform the same operation on each item. For example, if you need to send an email to each user in a collection, you could do something like this:
 
 ```ruby
-class ExampleJob < ActiveJob::Base
-  include AcidicJob
-  acidic_by_job_args
+class Job < ActiveJob::Base
+  include AcidicJob::Workflow
 
-  def perform(arg_1, arg_2)
-    # the idempotency key will be based on whatever the values of `arg_1` and `arg_2` are
+  def perform(users)
+    @users = users
+    execute_workflow(unique_by: @users) do |w|
+      w.step :notify_users
+    end
   end
-end
-```
 
-These options cover the two common situations, but sometimes our systems need finer-grained control. For example, our job might take some record as the job argument, but we need to use a combination of the record identifier and record status as the foundation for the idempotency key. In these cases you can pass a `Proc` to an `acidic_by` class method:
+  def notify_users
+    cursor = ctx[:cursor] || 0
+    user = @users[cursor]
+    return if user.nil?
 
-```ruby
-class ExampleJob < ActiveJob::Base
-  include AcidicJob
-  acidic_by ->(record:) { [record.id, record.status] }
+    UserMailer.with(user: user).welcome_email.deliver_later
 
-  def perform(record:)
-    # the idempotency key will be based on whatever the values of `record.id` and `record.status` are
+    ctx[:cursor] = cursor + 1
+    repeat_step!
   end
 end
 ```
 
-> **Note:** The signature of the `acidic_by` proc _needs to match the signature_ of the job's `perform` method.
-
-
-### Sidekiq Callbacks
-
-In order to ensure that `AcidicJob::Staged` records are only destroyed once the related job has been successfully performed, whether it is an ActiveJob or a Sidekiq Worker, `acidic_job` also extends Sidekiq to support the [ActiveJob callback interface](https://edgeguides.rubyonrails.org/active_job_basics.html#callbacks).
-
-This allows `acidic_job` to use an `after_perform` callback to delete the `AcidicJob::Staged` record, whether you are using the gem with ActiveJob or pure Sidekiq Workers. Of course, this means that you can add your own callbacks to any jobs or workers that include the `AcidicJob` module as well.
-
+This example demonstrates how you can leverage the basic building blocks provided by `AcidicJob` to orchestrate complex workflows. In this case, the `notify_users` step sends an email to each user in the collection, one at a time, and resiliently handles errors by storing a cursor in the `ctx` object to keep track of the current user being processed. If any error occurs while traversing the `@users` collection, the job will be retried, and the `notify_users` step will be re-executed from the last successful cursor position.
 
-### Run Finished Callbacks
-
-When working with workflow jobs that make use of the `awaits` feature for a step, it is important to remember that the `after_perform` callback will be called _as soon as the first `awaits` step has enqueued job_, and **not** when the entire job run has finished. `acidic_job` allows the `perform` method to finish so that the queue for the workflow job is cleared to pick up new work while the `awaits` jobs are running. `acidic_job` will automatically re-enqueue the workflow job and progress to the next step when all of the `awaits` jobs have successfully finished. However, this means that `after_perform` **is not necessarily** the same as `after_finish`. In order to provide the opportunity for you to execute callback logic _if and only if_ a job run has finished, we provide callback hooks for the `finish` event.
-
-For example, you could use this hook to immediately clean up the `AcidicJob::Run` database record whenever the workflow job finishes successfully like so:
+The `halt_step!` method, on the other hand, stops not just the execution of the current step but the job as a whole. This is useful when you either need to conditionally stop the workflow based on some criteria or need to delay the job for some amount of time before being restarted. For example, if you need to send a follow-up email to a user 14 days after they sign up, you could do something like this:
 
 ```ruby
-class RideCreateJob < ActiveJob::Base
-  include AcidicJob
-  set_callback :finish, :after, :delete_run_record
-
-  def perform(user_id, ride_params)
-    @user = User.find(user_id)
-    @params = ride_params
-
-    with_acidity providing: { ride: nil } do
-      step :create_ride_and_audit_record, awaits: [SomeJob.with('argument_1', keyword: 'value')]
-      step :create_stripe_charge, args: [1, 2, 3], kwargs: { some: 'thing' }
-      step :send_receipt
+class Job < ActiveJob::Base
+  include AcidicJob::Workflow
+
+  def perform(user)
+    @user = user
+    execute_workflow(unique_by: @user) do |w|
+      w.step :delay
+      w.step :send_welcome_email
     end
   end
-  
-  def delete_run_record
-    return unless acidic_job_run.succeeded?
 
-    acidic_job_run.destroy!
+  def delay
+    enqueue(wait: 14.days)
+    ctx[:halt] = true
+  end
+
+  def send_welcome_email
+    if ctx[:halt]
+      ctx[:halt] = false
+      halt_step!
+    end
+    UserMailer.with(user: @user).welcome_email.deliver_later
   end
 end
 ```
 
+In this example, the `delay` step creates a new instance of the job and enqueues it to run 14 days in the future. It then sets a flag in the `ctx` object to halt the job. We want to halt the job in the following step and only halt it once. This ensures that when the job is re-enqueued and performed, it jumps to the `send_welcome_email` step and that step send the email only on this second run of the job. By checking for this flag and, if it is set, clears the flag and halting the job, the `send_welcome_email` step can free the worker queue from doing work, let the system waits 2 weeks, and then pick right back up where it paused originally.
+
+
+### Overview
+
+`AcidicJob` is a library that provides a small yet powerful set of tools to build cohesive and resilient workflows in your Active Jobs. All of the tools are made available by `include`ing the `AcidicJob::Workflow` module. The primary and most important tool is the `execute_workflow` method, which you call within your `perform` method. Then, if you need to store any contextual data, you use the `ctx` objects setters and getters. Finally, within any step methods, you can call `repeat_step!` or `halt_step!` to control the execution of the workflow. If you need, you can also access the `execution` Active Record object to get information about the current execution of the workflow. With these lightweight tools, you can build complex workflows that are resilient to failures and can handle a wide range of use cases.
+
 
 ## Testing
 
@@ -374,13 +262,13 @@ When testing acidic jobs, you are likely to run into `ActiveRecord::TransactionI
 ActiveRecord::TransactionIsolationError: cannot set transaction isolation in a nested transaction
 ```
 
-This error is thrown because by default RSpec and most MiniTest test suites use database transactions to keep the test database clean between tests. The database transaction that is wrapping all of the code executed in your test is run at the standard isolation level, but acidic jobs then try to create another transaction run at a more conservative isolation level. You cannot have a nested transaction that runs at a different isolation level, thus, this error. 
+This error is thrown because by default RSpec and most MiniTest test suites use database transactions to keep the test database clean between tests. The database transaction that is wrapping all of the code executed in your test is run at the standard isolation level, but `AcidicJob` then tries to create another transaction at a more conservative isolation level. You cannot have a nested transaction that runs at a different isolation level, thus, this error.
 
 In order to avoid this error, you need to ensure firstly that your tests that run your acidic jobs are not using a database transaction and secondly that they use some different strategy to keep your test database clean. The [DatabaseCleaner](https://github.com/DatabaseCleaner/database_cleaner) gem is a commonly used tool to manage different strategies for keeping your test database clean. As for which strategy to use, `truncation` and `deletion` are both safe, but their speed varies based on our app's table structure (see https://github.com/DatabaseCleaner/database_cleaner#what-strategy-is-fastest). Either is fine; use whichever is faster for your app.
 
-In order to make this test setup simpler, `AcidicJob` provides a `TestCase` class that your MiniTest jobs tests can inherit from. It is simple; it inherits from `ActiveJob::TestCase`, sets `use_transactional_tests` to `false`, and ensures `DatabaseCleaner` is run for each of your tests. Moreover, it ensures that the system's original DatabaseCleaner configuration is maintained, options included, except that any `transaction` strategies for any ORMs are replaced with a `deletion` strategy. It does so by storing whatever the system DatabaseCleaner configuration is at the start of `before_setup` phase in an instance variable and then restores that configuration at the end of `after_teardown` phase. In between, it runs the configuration thru a pipeline that selectively replaces any `transaction` strategies with a corresponding `deletion` strategy, leaving any other configured strategies untouched.
+In order to make this test setup simpler, `AcidicJob` provides a `Testing` module that your job tests can include. It is simple; it sets `use_transactional_tests` to `false` (if the test is an `ActiveJob::TestCase`), and ensures a transaction-safe `DatabaseCleaner` strategy is run for each of your tests. Moreover, it ensures that the system's original DatabaseCleaner configuration is maintained, options included, except that any `transaction` strategies for any ORMs are replaced with a `deletion` strategy. It does so by storing whatever the system DatabaseCleaner configuration is at the start of `before_setup` phase in an instance variable and then restores that configuration at the end of `after_teardown` phase. In between, it runs the configuration thru a pipeline that selectively replaces any `transaction` strategies with a corresponding `deletion` strategy, leaving any other configured strategies untouched.
 
-For those of you using RSpec, you can require the `acidic_job/rspec_configuration` file, which will configure RSpec in the exact same way I have used in my RSpec projects to allow me to test acidic jobs with either the `deletion` strategy but still have all of my other tests use the fast `transaction` strategy:
+For those of you using RSpec, use this as a baseline to configure RSpec in the exact same way I have used in my RSpec projects to allow me to test `AcidicJob` with the `deletion` strategy but still have all of my other tests use the fast `transaction` strategy:
 
 ```ruby
 require "database_cleaner/active_record"
@@ -413,12 +301,41 @@ RSpec.configure do |config|
 end
 ```
 
+
 ## Development
 
 After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake test` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
 
 To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and the created tag, and push the `.gem` file to [rubygems.org](https://rubygems.org).
 
+You can run a specific Rails version using one of the Gemfiles defined in the `/gemfiles` directory via the `BUNDLE_GEMFILE` ENV variable, e.g.:
+
+```sh
+BUNDLE_GEMFILE=gemfiles/rails_7.0.gemfile bundle exec rake test
+```
+
+You can likewise test only one particular test file using the `TEST` ENV variable, e.g.:
+
+```sh
+TEST=test/acidic_job/basics_test.rb
+```
+
+Finally, if you need to only run one particular test case itself, use the `TESTOPTS` ENV variable with the `--name` option, e.g.:
+
+```sh
+TESTOPTS="--name=test_workflow_with_each_step_succeeding"
+```
+
+You may also need to run the test suite with a particular Ruby version. If you are using the ASDF version manager, you can set the Ruby version with the `ASDF_RUBY_VERSION` ENV variable, e.g.:
+
+```sh
+ASDF_RUBY_VERSION=2.7.7 bundle exec rake test
+```
+
+If you are using `rbenv` to manage your Ruby versions, you can use the `RBENV_VERSION` ENV variable instead.
+
+These options can of course be combined to help narrow down your debugging when you find a failing test in CI.
+
 ## Contributing
 
-Bug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/acidic_job.
+Bug reports and pull requests are welcome on GitHub at https://github.com/fractaledmind/acidic_job.