chrono_forge 0.9.1 → 0.11.0

data/README.md

@@ -7,20 +7,109 @@
 
 > A robust framework for building durable, distributed workflows in Ruby on Rails applications
 
-ChronoForge provides a powerful solution for handling long-running processes, managing state, and recovering from failures in your Rails applications. Built on top of ActiveJob, it ensures your critical business processes remain resilient and traceable.
+ChronoForge handles long-running processes, manages state, and recovers from failures in your Rails applications. Built on ActiveJob, it keeps critical business processes resilient and traceable.
+
+Workflows are **plain Ruby**. Ordinary `if`/`else`, loops, and early returns drive the flow. There's no declarative DSL to learn and no extra service to run, which makes ChronoForge a good fit for business processes whose shape depends on runtime state: conditional branches, iteration over data, and built-in periodic tasks (`durably_repeat`).
+
+> [!NOTE]
+> **In production** at **achieve by Petra**, an investment platform in the Petra Group — where it has executed over 3.6 million workflows and 32 million durable steps across scheduled payments, investment rollovers, and membership lifecycle management.
+
+## 🧭 Why ChronoForge
+
+Most Rails workflow tools ask you to declare your steps up front in a DSL:
+
+```ruby
+step :send_welcome_email
+step :remind_of_tasks, wait: 2.days
+step :complete_onboarding, wait: 15.days
+```
+
+That reads cleanly for a fixed, linear sequence. But many business processes branch, loop, and react to data that only exists at runtime, and a declarative schema gets awkward there. ChronoForge takes the opposite approach: **the workflow is plain Ruby, and each step is just a method.** Conditionals, iteration, early returns, and helper methods all work the way they normally do; you drive durable steps, waits, and retries inline with a few primitives like `durably_execute` and `wait_until`:
+
+```ruby
+class OnboardingWorkflow < ApplicationJob
+  prepend ChronoForge::Executor
+
+  def perform(user_id:)
+    @user_id = user_id
+
+    durably_execute :send_welcome_email
+    wait 2.days, :remind_delay
+    durably_execute :remind_of_tasks
+    wait 15.days, :onboarding_delay
+    durably_execute :complete_onboarding
+  end
+
+  private
+
+  def send_welcome_email = UserMailer.welcome(@user_id).deliver_now
+  def remind_of_tasks = UserMailer.task_reminder(@user_id).deliver_now
+  def complete_onboarding = User.find(@user_id).complete_onboarding!
+end
+```
+
+There is a trade-off, though a smaller one than it used to be. A declarative engine can show the steps a run *hasn't* reached yet; plain imperative code can't. The [definition graph](#-dashboard) closes most of that gap. It reads your `perform` with Prism (without running it) and draws the steps the run will take, with live status on each. The view is best-effort: a computed step name or a data-dependent loop collapses to one `dynamic` node. For a simple, fixed sequence ("send email, wait 2 days, send another") a declarative DSL can still read more cleanly, and that's a fine reason to use one.
+
+### How it compares
+
+|                              | ChronoForge          | AJ Continuations           | GenevaDrive        | AcidicJob       | Temporal        |
+| ---------------------------- | -------------------- | -------------------------- | ------------------ | --------------- | --------------- |
+| Programming model            | procedural (plain Ruby) | procedural (`step` blocks) | declarative DSL | declarative DSL | procedural (via SDK) |
+| Built-in periodic tasks      | ✓ `durably_repeat`   | ✗                          | ✗                  | ✗               | ✓               |
+| Parallel sub-workflows       | ✓ `branch` / `spawn` | ✗                          | ✗                  | ✗               | ✓               |
+| Pending-step visibility      | ✗ (procedural)       | ✗ (procedural)             | ✓                  | ✓               | ✗ (procedural)  |
+| Web dashboard                | ✓ (free gem)         | job-level (Mission Control)| paid only          | ✗               | ✓               |
+| Extra infrastructure         | none (DB + ActiveJob)| none (built into Rails)    | none               | none            | server required |
+| Rails support                | 7.1+                 | 8.1+                       | 7.2+               | 7.1+            | any (Ruby SDK)  |
+| License                      | MIT                  | MIT                        | LGPL / commercial  | MIT             | MIT             |
+
+<sub>Comparison reflects each project's documented features as of mid-2026, to the best of our knowledge; corrections welcome via PR.</sub>
+
+A few deliberate choices behind that table:
+
+- **Periodic tasks are built in.** `durably_repeat` runs a step on a schedule until a condition holds, with automatic catch-up for missed runs, so a workflow can be its own recurring job and cron-style monitor, right alongside the rest of its logic. Without built-in support, periodic behavior usually lives in a separate scheduler that you reconcile with workflow state by hand.
+- **No extra infrastructure.** ChronoForge is a gem over your existing database and ActiveJob backend. There's no separate server or daemon to operate, unlike Temporal.
+- **Large-scale fan-out is built in.** `branch` with `spawn`/`spawn_each` fans a workflow out into child workflows that run concurrently and join when their results are needed, streaming ActiveRecord relations in constant memory for large sets. Among the Ruby-native engines here, only ChronoForge offers this without a separate orchestration server (Temporal does, server-side).
+- **Recovery is built into the model.** Steps are append-only history, so a crashed step leaves the workflow `stalled`, recoverable directly with `retry_later`.
+- **A real dashboard, free.** ChronoForge's [mountable dashboard](#-dashboard) — workflow list, step-replay timeline, a per-run **definition graph** (the steps a run will take, read from `perform`, with live status on each), context inspector, retry/unlock — ships as a separate MIT gem.
+- **MIT licensed.** Permissive and dependency-policy-friendly.
+- **ActiveJob Continuations solve a narrower problem.** Rails 8.1's built-in [continuations](https://api.rubyonrails.org/classes/ActiveJob/Continuation.html) make a *single* long job survive interruptions: you wrap work in `step` blocks and track a `cursor`, and at each checkpoint the job asks the queue adapter whether it's `stopping?`, re-enqueuing to resume from the last completed step/cursor — no gem, no tables. They deliberately stop short of being a workflow engine: there's no durable waiting on time, conditions, or external events; no periodic steps; no parallel fan-out; and no persisted, queryable history. Reach for continuations to make one big job restart-safe; reach for ChronoForge when a process spans steps that wait, recur, fan out, and need recovery and visibility. They also compose — a ChronoForge workflow *is* ActiveJob work.
 
 ## 🌟 Features
 
+- **Plain-Ruby control flow**: Branching, loops, and iteration over runtime data, without a DSL or step registry
 - **Durable Execution**: Automatically tracks and recovers from failures during workflow execution
+- **Periodic tasks built in**: `durably_repeat` runs a step on an interval until a condition is met, with catch-up for missed runs. Acts as a recurring task and a cron-style monitor in one
+- **Wait States**: Time-based waits and condition-based waiting (`wait_until`) that survive restarts
+- **Parallel sub-workflows**: `branch` with `spawn`/`spawn_each` fans out into concurrent child workflows and joins them (`automerge` or `merge_branches`); large sets stream in constant memory. See [Branches](#-branches-parallel-sub-workflows)
 - **State Management**: Built-in workflow state tracking with persistent context storage
 - **Concurrency Control**: Advanced locking mechanisms to prevent parallel execution of the same workflow
-- **Error Handling**: Comprehensive error tracking with configurable retry strategies
+- **Error Handling**: Error tracking with a unified, configurable [`RetryPolicy`](#-retry-policies) (including per-error-type policies)
 - **Execution Logging**: Detailed logging of workflow steps and errors for visibility
-- **Wait States**: Support for time-based waits and condition-based waiting
-- **Database-Backed**: All workflow state is persisted to ensure durability
+- **Database-Backed**: All workflow state is persisted to ensure durability, with no extra services to run
 - **ActiveJob Integration**: Compatible with all ActiveJob backends, though database-backed processors (like Solid Queue) provide the most reliable experience for long-running workflows
 - **Retention & Cleanup**: A schedulable job to prune finished workflows and the unbounded logs that periodic tasks accumulate (see [Cleanup & Retention](#-cleanup--retention))
 
+## 🖥️ Dashboard
+
+ChronoForge has a free, mountable dashboard for visibility and recovery: workflow list, step replay timeline, a per-run **definition graph** (the durable steps a workflow will run, statically parsed from `perform`, with live run status overlaid), context inspector, periodic-task health, wait-state age, and retry/unlock actions. It ships as a separate gem, `chrono_forge-dashboard`, so the core stays lean.
+
+[![ChronoForge dashboard](chrono_forge-dashboard/docs/screenshots/workflows.png)](chrono_forge-dashboard/README.md#screenshots)
+
+The per-run **definition graph**: the steps a workflow will run, read from `perform`, with the run's status shown on each node.
+
+[![Definition graph](chrono_forge-dashboard/docs/screenshots/definition-graph-scheduled-payment.png)](chrono_forge-dashboard/README.md#screenshots)
+
+```ruby
+# Gemfile
+gem "chrono_forge-dashboard"
+
+# config/routes.rb
+mount ChronoForge::Dashboard::Engine, at: "/chrono_forge"
+```
+
+See [`chrono_forge-dashboard`](chrono_forge-dashboard/README.md) for setup, authentication, and [more screenshots](chrono_forge-dashboard/README.md#screenshots).
+
 ## 📦 Installation
 
 Add to your application's Gemfile:
@@ -59,9 +148,9 @@ $ rails generate chrono_forge:upgrade
 $ rails db:migrate
 ```
 
-The upgrade migration is idempotent (`if_not_exists`), so it is safe to run even
-if your schema already has the index. Fresh installs get the index from the
-install migration and do **not** need to run the upgrade.
+Re-running the generator is safe — it skips any migrations you already have.
+Fresh installs get everything from `chrono_forge:install` and don't need the
+upgrade.
 
 ## 📋 Usage
 
@@ -136,6 +225,54 @@ class OrderProcessingWorkflow < ApplicationJob
 end
 ```
 
+### A workflow you can't flatten into a step list
+
+The example above is linear, but most real processes aren't. Because a ChronoForge workflow is plain Ruby, branching and dynamic iteration are just… branching and iteration:
+
+```ruby
+class OrderProcessingWorkflow < ApplicationJob
+  prepend ChronoForge::Executor
+
+  def perform(order_id:)
+    @order_id = order_id
+
+    wait_until :payment_confirmed?
+    durably_execute :validate_order
+
+    # Runtime branching: the path depends on data known only at execution time
+    if context["requires_compliance_check"]
+      durably_execute :run_compliance_review
+      wait_until :compliance_approved?, timeout: 48.hours
+    end
+
+    # Iterate over runtime data: one durable, idempotent step per item
+    context["line_item_ids"].each do |item_id|
+      context["current_item_id"] = item_id
+      durably_execute :fulfill_item, name: "fulfill_#{item_id}"
+    end
+
+    # Recurring notification: nudge the customer until they confirm delivery
+    durably_repeat :send_delivery_reminder, every: 3.days, till: :delivery_confirmed?
+
+    durably_execute :complete_order
+  end
+
+  private
+
+  def fulfill_item
+    FulfillmentService.fulfill(@order_id, context["current_item_id"])
+  end
+
+  def send_delivery_reminder
+    OrderMailer.delivery_reminder(@order_id).deliver_later
+  end
+
+  # ... other condition and step methods ...
+end
+```
+
+Each `durably_execute` is checkpointed by its step name, so on resume the completed branches and items are skipped and the workflow continues where it left off. A fixed, declared list of steps can't easily express runtime branches, a loop over a runtime-sized collection, and an open-ended recurring notification.
+
 ### Core Workflow Features
 
 #### 🚀 Executing Workflows
@@ -162,14 +299,15 @@ OrderProcessingWorkflow.perform_later(
 
 #### ⚡ Durable Execution
 
-The `durably_execute` method ensures operations are executed exactly once with automatic retry logic and fault tolerance:
+The `durably_execute` method runs an operation with automatic retries, and skips it on replay once it has completed:
 
 ```ruby
 # Basic execution
 durably_execute :send_welcome_email
 
-# With custom retry attempts
-durably_execute :critical_payment_processing, max_attempts: 5
+# With a custom retry policy
+durably_execute :critical_payment_processing,
+  retry_policy: RetryPolicy.new(max_attempts: 5)
 
 # With custom name for tracking multiple calls to same method
 durably_execute :upload_file, name: "profile_image_upload"
@@ -182,10 +320,10 @@ class FileProcessingWorkflow < ApplicationJob
     @file_id = file_id
     
     # This might fail due to network issues, rate limits, etc.
-    durably_execute :upload_to_s3, max_attempts: 5
+    durably_execute :upload_to_s3, retry_policy: RetryPolicy.new(max_attempts: 5)
     
     # Process file after successful upload
-    durably_execute :generate_thumbnails, max_attempts: 3
+    durably_execute :generate_thumbnails, retry_policy: RetryPolicy.new(max_attempts: 3)
   end
 
   private
@@ -204,9 +342,77 @@ end
 
 **Key Features:**
 - **Idempotent**: Same operation won't be executed twice during replays
-- **Automatic Retries**: Failed executions retry with exponential backoff (2^attempt seconds, capped at 32s)
+- **Automatic Retries**: Failed executions retry per a unified `RetryPolicy` (exponential backoff with jitter; the step default caps at 30s over 3 attempts)
 - **Error Tracking**: All failures are logged with detailed error information
-- **Configurable**: Customize retry attempts and step naming
+- **Configurable**: Pass a `retry_policy:` per call, or set a class-wide default with the `retry_policy` DSL (see [Retry Policies](#retry-policies))
+
+#### 🔁 Retry Policies
+
+All retrying in ChronoForge goes through a single `RetryPolicy` (`ChronoForge::Executor::RetryPolicy`). It answers two questions: *should this failure be retried?* and *how long until the next attempt?*
+
+```ruby
+RetryPolicy.new(
+  max_attempts: 3,        # cap on total attempts; nil = no count cap (bounded elsewhere)
+  base: 1,                # seconds; delay of the first retry
+  cap: 30,                # seconds; ceiling for a single delay
+  jitter: true,           # spread retries with equal jitter
+  retry_on: nil           # nil = retry any StandardError; [Classes] = only those; [] = none
+)
+```
+
+Backoff is exponential with equal jitter, computed once at re-enqueue time (never replayed, so it stays deterministic where it matters).
+
+**Resolution order:**
+
+- **`durably_execute`, `durably_repeat`, workflow-level errors**: per-call `retry_policy:` → class-level `retry_policy` default → built-in default.
+- **`wait_until`**: per-call `retry_policy:` → built-in default. It deliberately does **not** inherit the class default, so a class-wide "retry everything" can't silently turn condition-evaluation bugs into retried errors.
+
+**Built-in defaults:**
+
+| Site | Default | Why |
+|------|---------|-----|
+| Steps (`durably_execute`/`durably_repeat`) | 3 attempts, cap 30s, retry any error | flaky calls fail fast |
+| Workflow-level (uncaught errors) | 10 attempts, cap 600s, retry any error | tolerant window up to ~8.5 min (≈4 min typical w/ jitter) for transient infra errors; each retry replays the whole workflow from the top |
+| `wait_until` condition errors | retry nothing | a raised condition is usually a bug, not transient |
+
+**Class-wide default via the `retry_policy` DSL:**
+
+```ruby
+class ChargeWorkflow < ApplicationJob
+  prepend ChronoForge::Executor
+  retry_policy max_attempts: 5, base: 2, cap: 60   # applies to steps + workflow-level
+
+  def perform
+    durably_execute :charge,
+      retry_policy: RetryPolicy.new(max_attempts: 8, retry_on: [Net::OpenTimeout])
+    wait_until :settled?,
+      retry_policy: RetryPolicy.new(retry_on: [BankApiError])
+  end
+end
+```
+
+**Composite policies (per-error budgets):**
+
+Pass an **array** of policies to handle different error types differently. On a failure, the **first** policy whose `retry_on` matches the raised error applies, and each error type gets its **own attempt budget and backoff**:
+
+```ruby
+durably_execute :charge_card, retry_policy: [
+  RetryPolicy.new(retry_on: [NetworkError],         max_attempts: 5),            # transient: retry hard
+  RetryPolicy.new(retry_on: [RateLimitError],       max_attempts: 10, base: 5),  # back off longer
+  RetryPolicy.new(retry_on: [PaymentDeclinedError], max_attempts: 1),            # fail fast, never retry
+  RetryPolicy.new(retry_on: nil)                                                 # catch-all (optional), keep last
+]
+```
+
+- **Order matters**: the first matching policy wins, so list specific errors first and a catch-all (`retry_on: nil`) last. An error matched by no policy is **not retried** (fails fast).
+- A subclass of a listed error routes to that policy and draws from its budget.
+- Per-error counts are tracked by the policy's declared errors, so the budgets are stable even if you reorder the list.
+- The class-level DSL accepts the same form as positional arguments (applies to steps **and** workflow-level errors):
+
+  ```ruby
+  retry_policy RetryPolicy.new(retry_on: [NetworkError], max_attempts: 5),
+               RetryPolicy.new(retry_on: nil, max_attempts: 2)
+  ```
 
 #### ⏱️ Wait States
 
@@ -243,11 +449,11 @@ wait_until :external_api_ready?,
   timeout: 30.minutes, 
   check_interval: 1.minute
 
-# Wait with retry on specific errors
+# Wait with retry on specific errors raised while evaluating the condition
 wait_until :database_migration_complete?,
   timeout: 2.hours,
   check_interval: 30.seconds,
-  retry_on: [ActiveRecord::ConnectionNotEstablished, Net::TimeoutError]
+  retry_policy: RetryPolicy.new(retry_on: [ActiveRecord::ConnectionNotEstablished, Net::TimeoutError])
 
 # Complex condition example
 def third_party_service_ready?
@@ -258,7 +464,7 @@ end
 wait_until :third_party_service_ready?,
   timeout: 1.hour,
   check_interval: 2.minutes,
-  retry_on: [Net::TimeoutError, Net::HTTPClientException]
+  retry_policy: RetryPolicy.new(retry_on: [Net::TimeoutError, Net::HTTPClientException])
 ```
 
 **3. Event-driven Waits (`continue_if`)**
@@ -328,7 +534,7 @@ PaymentWorkflow.perform_later("order-#{order_id}", order_id: order_id)
 
 #### 🔄 Periodic Tasks
 
-The `durably_repeat` method enables robust periodic task execution within workflows. Tasks are scheduled at regular intervals until a specified condition is met, with automatic catch-up for missed executions and configurable error handling.
+`durably_repeat` runs periodic tasks inside a workflow. A task is scheduled at a regular interval until a condition is met, with automatic catch-up for missed executions and configurable error handling.
 
 ```ruby
 class NotificationWorkflow < ApplicationJob
@@ -379,7 +585,7 @@ end
 
 - **Idempotent Execution**: Each repetition gets a unique execution log, preventing duplicates during replays
 - **Automatic Catch-up**: Missed executions due to downtime are automatically skipped using timeout-based fast-forwarding
-- **Flexible Timing**: Support for custom start times and precise interval scheduling
+- **Custom Timing**: Custom start times and precise interval scheduling
 - **Error Resilience**: Individual execution failures don't break the periodic schedule
 - **Configurable Error Handling**: Choose between continuing despite failures or failing the entire workflow
 
@@ -390,7 +596,7 @@ durably_repeat :generate_daily_report,
   every: 1.day,                          # Execution interval
   till: :reports_complete?,              # Stop condition
   start_at: Date.tomorrow.beginning_of_day, # Custom start time (optional)
-  max_attempts: 5,                       # Retries per execution (default: 3)
+  retry_policy: RetryPolicy.new(max_attempts: 5), # Retry policy per execution (default: step_default)
   timeout: 2.hours,                      # Catch-up timeout (default: 1.hour)
   on_error: :fail_workflow,              # Error handling (:continue or :fail_workflow)
   name: "daily_reports"                  # Custom task name (optional)
@@ -418,6 +624,21 @@ def cleanup_files(scheduled_time)
 end
 ```
 
+#### 🌿 Branches
+
+When a workflow needs to fan out — process every pending order, reconcile each region — `branch` spawns child workflows that run concurrently and join when their results are needed:
+
+```ruby
+branch :reconcile, automerge: true do
+  spawn :eu, ReconcileWorkflow, region: "EU"
+  spawn_each :orders, Order.pending do |order|
+    [OrderWorkflow, { order_id: order.id }]
+  end
+end
+```
+
+`spawn_each` streams ActiveRecord relations by primary key in constant memory, so a branch can fan out over very large sets. Join inline with `automerge: true`, or open branches without it and `merge_branches` them later after doing other work. See [Branches: parallel sub-workflows](#-branches-parallel-sub-workflows) for the full model, a worked example, and the crash-recovery caveats.
+
 #### 🔄 Workflow Context
 
 ChronoForge provides a persistent context that survives job restarts. The context behaves like a Hash but with additional capabilities:
@@ -439,6 +660,12 @@ context.set(:total_amount, 99.99)
 # Set a value only if the key doesn't already exist
 context.set_once(:created_at, Time.current.iso8601)
 
+# Set several values at once (alias: set_multiple)
+context.merge(status: "processing", total_amount: 99.99, attempts: 0)
+
+# Set several values at once, but only keys that don't already exist (alias: set_multiple_once)
+context.merge_once(created_at: Time.current.iso8601, attempts: 0)
+
 # Check if a key exists
 if context.key?(:user_id)
   # Do something with the user ID
@@ -447,7 +674,7 @@ end
 
 The context supports serializable Ruby objects (Hash, Array, String, Integer, Float, Boolean, and nil) and validates types automatically.
 
-Hash and Array values are stored as JSON, which has no symbols — so **symbol keys inside a stored hash come back as strings**:
+Hash and Array values are stored as JSON, which has no symbols, so **symbol keys inside a stored hash come back as strings**:
 
 ```ruby
 context[:totals] = { paid: 5, pending: 2 }
@@ -455,33 +682,138 @@ context[:totals]          # => { "paid" => 5, "pending" => 2 }
 context[:totals]["paid"]  # => 5   (not context[:totals][:paid])
 ```
 
-(The top-level context key itself is interchangeable — `context[:totals]` and `context["totals"]` refer to the same entry.)
+(The top-level context key itself is interchangeable: `context[:totals]` and `context["totals"]` refer to the same entry.)
 
-Context is meant for **small working state** — ids, flags, timestamps, and small structures used to coordinate steps. Each value is capped at **16 KB** (a `ChronoForge::Executor::Context::ValidationError` is raised above that). Store large payloads (documents, uploads, API responses) in their own storage and keep just a reference (an id or key) in the context.
+Context is meant for **small working state**: ids, flags, timestamps, and small structures used to coordinate steps. Each value is capped at **16 KB** (a `ChronoForge::Executor::Context::ValidationError` is raised above that). Store large payloads (documents, uploads, API responses) in their own storage and keep just a reference (an id or key) in the context.
 
 ### 🛡️ Error Handling
 
-ChronoForge automatically tracks errors and provides configurable retry capabilities:
+ChronoForge automatically tracks errors and routes all retrying through a single [`RetryPolicy`](#-retry-policies). Configure it per call with `retry_policy:`, or set a class-wide default with the `retry_policy` DSL:
 
 ```ruby
 class MyWorkflow < ApplicationJob
   prepend ChronoForge::Executor
 
-  private
+  # Class-wide default for workflow-level errors and steps without an override
+  retry_policy max_attempts: 5, base: 2, cap: 60
+
+  def perform
+    # Retry only network errors, up to 5 times, for this step
+    durably_execute :call_external_api,
+      retry_policy: RetryPolicy.new(max_attempts: 5, retry_on: [NetworkError])
+  end
+end
+```
+
+To make an error non-retryable, leave it out of `retry_on:` (an empty `retry_on: []` retries nothing).
+
+## 🌿 Branches: parallel sub-workflows
+
+`branch` / `spawn` / `spawn_each` / `merge_branches` let a workflow fan out into
+child workflows that run concurrently, then join them when their results are
+needed.
+
+### Model
 
-  def should_retry?(error, attempt_count)
-    case error
-    when NetworkError
-      attempt_count < 5  # Retry network errors up to 5 times
-    when ValidationError
-      false  # Don't retry validation errors
-    else
-      attempt_count < 3  # Default retry policy
+- **`branch :name do … end`** opens a named branch (a durable step). Inside the
+  block, `spawn` and `spawn_each` create and immediately enqueue child workflows —
+  children start running as soon as the branch block is entered.
+- **`spawn :name, WorkflowClass, **kwargs`** — enqueues one child workflow.
+- **`spawn_each :name, source do |item| [WorkflowClass, kwargs] end`** — enqueues
+  one child per item. The block returns the class and kwargs, so one branch can
+  fan out into mixed workflow types. Sources are iterated in constant memory;
+  ActiveRecord relations are streamed by primary key — pass them **without** an
+  explicit `.order`.
+- **`automerge: true`** — joins the branch **inline at the block's close**.
+  Execution does not continue past the `branch` call until every child has
+  completed. Use it for "dispatch this group and wait right here."
+- **`merge_branches :a, :b`** (or the singular alias `merge_branch :a`) — the
+  separate join point. Open branches without `automerge`, do other work while the
+  children run, then join when you need their results. `merge_branches` blocks
+  until all named branches are complete.
+
+### Worked example
+
+```ruby
+class FulfillmentWorkflow < ApplicationJob
+  prepend ChronoForge::Executor
+
+  def perform(cycle_id:)
+    # automerge: the branch is joined inline, right where the block closes —
+    # `perform` does not continue past it until every child has completed.
+    branch :reconcile, automerge: true do
+      spawn :eu, ReconcileWorkflow, region: "EU"
+      spawn_each :orders, Order.pending do |order|
+        order.priority? ? [PriorityOrderWorkflow, { order_id: order.id }]
+                        : [OrderWorkflow, { order_id: order.id }]
+      end
+    end
+
+    # For branches you want to run concurrently and join later, omit automerge
+    # and use merge_branches:
+    branch :invoices do
+      spawn_each :unpaid, Invoice.unpaid do |inv|
+        [InvoiceWorkflow, { invoice_id: inv.id }]
+      end
+    end
+    branch :shipments do
+      spawn_each :ready, Shipment.ready do |s|
+        [ShipmentWorkflow, { shipment_id: s.id }]
+      end
     end
+    do_other_work                        # runs while :invoices and :shipments dispatch/run
+    merge_branches :invoices, :shipments # join both here
+
+    durably_execute :finalize
   end
 end
 ```
 
+### Caveats
+
+> **Every branch must be joined.** A branch opened and never joined raises
+> `ChronoForge::Executor::UnmergedBranchError` when the workflow tries to
+> complete — fail-fast, no silently-orphaned children. Use either
+> `automerge: true` or a matching `merge_branches` call.
+
+> **The parent isn't replayed while waiting.** A lightweight
+> `ChronoForge::BranchMergeJob` polls for child completion; the parent workflow
+> only runs again once the branch is fully done. Polling cadence tracks the
+> **estimated time-to-drain** (measured from the children's completion rate), so
+> the parent is woken within ~`min_interval` of the last child finishing rather
+> than up to `max_interval` late; a branch that can only wait or is blocked on a
+> failure backs off to `max_interval` (those need a wait to elapse or operator
+> recovery, not faster polling).
+>
+> **Queue placement matters.** The poller is enqueued *after* the branch's
+> children, so on a queue those children saturate it starves behind the backlog
+> (the parent then converges up to `max_interval` late). Point it at a queue that
+> isn't saturated by the fan-out:
+>
+> ```ruby
+> ChronoForge.configure { |c| c.branch_merge_queue = :chrono_forge_pollers } # default: :default
+> ```
+
+> **`spawn_each` sources must re-enumerate deterministically across replays.**
+> ActiveRecord relations are streamed by primary key (children are keyed by
+> record id, so crash-resume is idempotent); a relation carrying an explicit
+> `.order(...)` raises. For non-AR enumerables, items are keyed by position, so
+> inserting or removing items mid-dispatch would shift keys and break idempotency.
+
+> **`spawn_each` AR sources must have stable membership.** Dispatch streams by
+> ascending primary key and resumes from the last key on crash-recovery, so a row
+> that enters the relation *below* the cursor after it has passed (e.g. a
+> `where(state: …)` scope whose rows mutate mid-dispatch) will never get a child.
+> Point `spawn_each` at a set that is fixed for the branch's lifetime — a frozen id
+> range, an append-only table, or `where(id: [...])` over a snapshot.
+
+> **`branch` blocks cannot be lexically nested within one workflow.** Opening a
+> `branch` inside another `branch` block raises `ArgumentError`; spawns belong to
+> exactly one branch. (A *spawned child workflow* may open its own branches — it
+> runs in its own executor — so cross-workflow nesting is fine.)
+
+Verified correct at 500,000 children on a single Postgres instance. A follow-up commit-consolidation change halved per-child execution time. See the [scale test](docs/fanout-scale-test.md).
+
 ## 🧪 Testing
 
 ChronoForge is designed to be easily testable using [ChaoticJob](https://github.com/fractaledmind/chaotic_job), a testing framework that makes it simple to test complex job workflows:
@@ -550,7 +882,7 @@ ChronoForge is ideal for:
 
 ## 🧠 Advanced State Management
 
-ChronoForge workflows follow a sophisticated state machine model to ensure durability and fault tolerance. Understanding these states and transitions is essential for troubleshooting and recovery.
+ChronoForge workflows move through a state machine. Understanding these states and transitions helps with troubleshooting and recovery.
 
 ### Workflow State Diagram
 
@@ -609,8 +941,7 @@ stateDiagram-v2
 
 #### Recovering Stalled/Failed Workflows
 
-Re-execute a failed or stalled workflow directly from its record — no need to
-constantize the job class or re-pass the key. Execution resumes via replay, so
+Re-execute a failed or stalled workflow directly from its record. Execution resumes via replay, so
 completed steps are skipped and it picks up at the step that failed:
 
 ```ruby
@@ -621,7 +952,7 @@ workflow.retry_now     # re-run inline (console/debugging)
 ```
 
 Only `stalled` or `failed` workflows are retryable. `retryable?` lets you check
-first, and both methods **validate up front** — calling `retry_later`
+first, and both methods **validate up front**: calling `retry_later`
 on a non-retryable workflow raises `ChronoForge::Executor::WorkflowNotRetryableError`
 immediately rather than enqueuing a job that would fail in the worker:
 
@@ -660,14 +991,14 @@ ChronoForge keeps every workflow and execution-log row indefinitely so that
 replays remain idempotent. Over time two things grow without bound:
 
 1. **Terminal workflows** (`completed` / `failed`) that are no longer needed.
-2. **`durably_repeat` repetition logs** — one row per scheduled execution. A
+2. **`durably_repeat` repetition logs**: one row per scheduled execution. A
    long-lived periodic workflow never reaches a terminal state, so its
    repetition logs accumulate indefinitely. Past repetitions (those behind the
    task's current frontier) are never read again, since each resume recomputes
-   the next execution from the coordination log — so they are safe to prune (see
+   the next execution from the coordination log, so they are safe to prune (see
    the safety note below).
 
-`ChronoForge::Cleanup` reclaims both. It is **not** run automatically — schedule
+`ChronoForge::Cleanup` reclaims both. It is **not** run automatically; schedule
 it from your own scheduler so you stay in control of retention:
 
 ```ruby
@@ -692,14 +1023,14 @@ Notes:
   that are both older than the window **and** scheduled strictly before the
   periodic task's current frontier (the coordination log's `last_execution_at`).
   Anything at or after the frontier is kept so `durably_repeat`'s catch-up
-  mechanism is never disrupted — so the window is purely a retention preference
+  mechanism is never disrupted, so the window is purely a retention preference
   and is safe even for yearly schedules.
 - Workflow retention is measured from when a workflow became terminal, not when
-  it was created — a long-running workflow that only just finished is kept for
+  it was created. A long-running workflow that only just finished is kept for
   the full window. Completed workflows use `completed_at` (immutable); failed
   workflows use `updated_at` (they have no `completed_at`).
 - The composite `[state, completed_at]` index added in this version keeps these
-  scans efficient — run `chrono_forge:upgrade` if you installed an earlier
+  scans efficient; run `chrono_forge:upgrade` if you installed an earlier
   version.
 
 A ready-made job is bundled so you can schedule it with any recurring-job
@@ -762,11 +1093,22 @@ This gem is available as open source under the terms of the [MIT License](https:
 
 | Method | Purpose | Key Parameters |
 |--------|---------|----------------|
-| `durably_execute` | Execute method with retry logic | `method`, `max_attempts: 3`, `name: nil` |
+| `durably_execute` | Execute method with retry logic | `method`, `retry_policy: nil`, `name: nil` |
 | `wait` | Time-based pause | `duration`, `name` |
-| `wait_until` | Condition-based waiting | `condition`, `timeout: 1.hour`, `check_interval: 15.minutes`, `retry_on: []` |
+| `wait_until` | Condition-based waiting | `condition`, `timeout: 1.hour`, `check_interval: 15.minutes`, `retry_policy: nil` |
 | `continue_if` | Manual continuation wait | `condition`, `name: nil` |
-| `durably_repeat` | Periodic task execution | `method`, `every:`, `till:`, `start_at: nil`, `max_attempts: 3`, `timeout: 1.hour`, `on_error: :continue` |
+| `durably_repeat` | Periodic task execution | `method`, `every:`, `till:`, `start_at: nil`, `retry_policy: nil`, `timeout: 1.hour`, `on_error: :continue` |
+
+### Branch Methods
+
+Fan a workflow out into parallel child sub-workflows (see [Branches](#-branches-parallel-sub-workflows)).
+
+| Method | Purpose | Key Parameters |
+|--------|---------|----------------|
+| `branch` | Open a named branch (takes a block) to dispatch children | `name`, `automerge: false` |
+| `spawn` | Enqueue one child workflow inside a branch | `name`, `workflow_class`, `**kwargs` |
+| `spawn_each` | Enqueue one child per item, streamed (block returns `[WorkflowClass, kwargs]`) | `name`, `source`, `of: 1000` |
+| `merge_branches` | Join named branches; blocks until all complete (alias `merge_branch`) | `*names`, `min_interval: 5.seconds`, `max_interval: 5.minutes` |
 
 ### Context Methods
 
@@ -776,6 +1118,8 @@ This gem is available as open source under the terms of the [MIT License](https:
 | `context[:key]` | Get context value | `user_id = context[:user_id]` |
 | `context.set(key, value)` | Set context value (alias) | `context.set(:status, "active")` |
 | `context.set_once(key, value)` | Set only if key doesn't exist | `context.set_once(:created_at, Time.current)` |
+| `context.merge(hash)` | Set multiple values atomically (alias: `set_multiple`) | `context.merge(status: "active", count: 0)` |
+| `context.merge_once(hash)` | Set multiple values, skipping existing keys (alias: `set_multiple_once`) | `context.merge_once(created_at: Time.current, count: 0)` |
 | `context.fetch(key, default)` | Get with default value | `context.fetch(:count, 0)` |
 | `context.key?(key)` | Check if key exists | `context.key?(:user_id)` |
 

data/Rakefile

@@ -12,3 +12,7 @@ Rake::TestTask.new do |t|
   t.test_files = FileList["test/**/*_test.rb"]
   t.verbose = true
 end
+
+# Release tasks (release:core:*, release:dashboard:*). Loaded after
+# bundler/gem_tasks so it can neutralize the bare `rake release` footgun.
+Dir.glob(File.expand_path("lib/tasks/*.rake", __dir__)).sort.each { |f| load f }