geneva_drive 0.5.0 → 0.7.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 45df42e923b2d50be98ff8572ae6c2391faa03c9d604372aabd3bb9ac603507e
-  data.tar.gz: 84e99503be26c28fef613972a4b8a6335d96cc87e775076def279857513825bf
+  metadata.gz: 1738e58c5f40199781a27c2345b20657e63ccc0c5a8b28010c2ac820774c2c2f
+  data.tar.gz: f729ef68e929f39fa26b09108f446a9b53ec13013d22d89f22d788b5ac0988e3
 SHA512:
-  metadata.gz: d8b2efa46d1e7d43da57afae65a3564e3826801d29a9f142b9217c3bc462f0f48122993c01a8710adbf968733db5c05e35fad081901e824f3b9ff87ca577bca0
-  data.tar.gz: fbfd86cec20556569803023f9a45483cc73cd7ded12d775dd2684f6693a883b1bff6d69ab61c40d006cd40715d6b696840f894f988001e7c61d8a07080283a44
+  metadata.gz: 87a29981a0278755e146cebf6e5e8d7075832665335b54511d5c24734a44040c62e9b2f9ae8fcd2558941d5f8cfe9eefc588efef65db09dc131d6602e2089d40
+  data.tar.gz: 9dcbb2ca84072315438dcacfd78d5463563b0ef6f5a940a5ccd12afcef7d2f10fb40cd710ce4bd5ba72142941242ec709f92e4d33927d15ce146538eed93834f

data/CHANGELOG.md

@@ -1,6 +1,19 @@
 # Changelog
 
-## [Unreleased]
+## [0.7.0]
+
+- Add signals: external events delivered to a workflow instance with `workflow.signal!(:payment_confirmed, payload: {...}, idempotency_key: event["id"])`, and a step primitive that parks until one arrives with `step :capture, wait_for: :payment_confirmed`. Signals are rows in a new `geneva_drive_signals` table, persisted before any processing, so arrival order does not matter: a signal that lands before its waiting step exists is buffered and claimed when the step runs, and one that lands after the step parked wakes it immediately. Both sides of the rendezvous run under the workflow row lock, so no signal can slip through the gap, and delivery is one transaction: the insert, the waking of parked executions and every counter bump land together or not at all, so a crashed `signal!` leaves nothing half-delivered (the job enqueue stays after-commit, with the existing stuck-scheduled housekeeping sweep as its backstop). Waiting is a first-class step execution state (`waiting`) that holds no queue slot and no timer — nothing polls. Matching is by name - `wait_for:` also takes a `GenevaDrive::SignalMatcher`, whose optional block narrows by payload and is evaluated on the workflow the signal was sent to (so `hero` is in scope), or any other object responding to `#matches?(signal)`. Matchers are plain reusable objects, so a rule can live in a constant and be shared across steps. Signals move `pending` → `claimed` → `consumed`, consumed only once every attached execution chain has finished cleanly — where skipping counts as finishing, so any flavor of skip (`skip_if:` at wake, `skip!` in a step, an exception policy, or an operator `skip!` on a ready or paused workflow) settles the signal that step was delivered, while skips of steps that never attached leave buffered signals alone — and in the same transaction as the step's own completion, so a reattempt or a `resume!` re-reads the same payload and the audit trail never claims an event was handled when it was not. Two counters on the row, `claimed` and `consumed`, count attachments and clean resolutions (with `claimed?` / `consumed?` reading those counters rather than the lifecycle state). Redelivery is deduplicated by a unique index on `(workflow_id, name, idempotency_key)` — the duplicate call returns the original row flagged `duplicate_delivery?` without dispatching. Payloads go through ActiveJob serializers, come back with indifferent access, and are bounded by `GenevaDrive.max_signal_payload_size` (128 KB, `nil` disables). `signal!` on a finished or canceled workflow raises `GenevaDrive::WorkflowNotOngoing` unless it is a redelivery of a known event; on a paused workflow the row is buffered and delivered by `resume!`. Timeouts are deliberately out of scope for now; waiting is instead made loud through `waiting_since` and two housekeeping gauges (`geneva_drive.waiting_step_executions`, `geneva_drive.waiting_overdue` past `GenevaDrive.waiting_visibility_threshold`, 7 days by default), with `skip!` and `cancel!` as operator escape hatches. Needs one new installer migration (the signals table plus two columns on step executions); until it runs everything else is unaffected and executing a `wait_for:` step raises a configuration error pointing at the missing migration. Test helpers: `assert_waiting_for_signal`, `assert_signal_state`, and the step-driving helpers now raise instead of spinning on a parked execution. Mixing up `wait:` and `wait_for:` now raises `StepConfigurationError` at class load in both directions, which also tightens `wait:` to reject Strings (previously `wait: "later"` silently meant zero seconds).
+
+## [0.6.0]
+
+- Add resumable steps for cursor-based iteration over large collections. `resumable_step :name do |iter| ... end` passes an `IterableStep` (API-compatible with Rails 8.1's `ActiveJob::Continuation::Step`) whose cursor is checkpointed to the database, so the step can be interrupted — by `max_iterations:`, `max_runtime:`, queue shutdown, external pause/cancel, or explicit `suspend!`/`skip_to!` — and continued in a later job execution from where it left off. Interruption completes the current step execution with outcome `continued` and creates a successor execution linked via `continues_from_id`, keeping a full audit chain instead of introducing a suspended state. Resumable steps run through the same `Executor` as regular steps: exception policies (step- and class-level, `max_reattempts`, `terminal_action:`, `report:`), tagged/injected loggers, `Rails.error` context, and merged job options all apply, with `reattempt!` and `resume!` continuing from the last checkpoint (use `reattempt!(rewind: true)` to start over). `HousekeepingJob` recovery is cursor-aware. The feature needs two new columns (`cursor`, `continues_from_id`) added by the installer migration; until it runs, everything else degrades gracefully (regular steps, pause/resume and housekeeping are unaffected) and executing an actual `resumable_step` raises a configuration error pointing at the missing migration. Cursors are limited to 128 KB of serialized JSON by default — the cursor is a position marker, not a payload — configurable via `GenevaDrive.max_cursor_size` (`nil` disables the check). Test helpers: `speedrun_current_step` follows the execution chain, `run_iterations(workflow, count:)` exercises partial progress, plus `assert_cursor` and `assert_step_has_successor`.
+- Validate `job_options` uniformly across `set_step_job_options`, per-step `job_options:`, and per-instance `workflow.step_job_options=`. Only the keys Active Job's `set` actually consumes (`:queue`, `:priority`, `:wait`, `:wait_until`) are accepted, and values are type-checked. Typos like `piority:` now raise `StepConfigurationError` at configuration time instead of being silently ignored.
+- Add partial index on `geneva_drive_step_executions(started_at) WHERE state = 'in_progress'` and order `HousekeepingJob` recovery batches by the filter column (`started_at` for stuck-in-progress, `scheduled_for` for stuck-scheduled). Fixes `HousekeepingJob` timing out on large in-progress sets where most rows are fresh — the planner previously walked the whole `state = 'in_progress'` set looking for old rows and tripped the caller's `statement_timeout`. Fixes AppSignal incident b9a7d1d1.
+- Add a normalized `geneva_drive.paused_ratio` gauge emitted by `HousekeepingJob`. For each workflow class (STI `type`) it reports a float in `0.0..1.0`, tagged `workflow: <ClassName>`, equal to that class's `paused` count divided by its total population (all states). Unlike absolute paused counts, this is bounded and easy to alert on (50 paused of 50 is a fire; 50 of 500,000 is noise). Derived from the existing single grouped query — no extra database load.
+- Add optional `metadata` JSON column to workflows (mirrors the existing step executions pattern). Exposes `step_job_options` / `step_job_options=` for per-instance job option overrides (queue, priority, etc.) that merge with class-level `set_step_job_options` and persist across step boundaries. Dedupe is unaffected since metadata is not part of the uniqueness constraint.
+- Add GenevaDrive workflow and step metadata to `Rails.error` execution context when steps execute, so Rails error reports include workflow id/class, step execution id/name, and hero identifiers.
+- Add per-step Active Job option overrides with `step :name, job_options: { queue: :critical, priority: 0 }`. Step options take precedence over workflow-level `set_step_job_options` and per-instance options, and are preserved across reattempts and resume re-enqueueing.
+- Add `report:` option to exception policies and class-level `on_exception` for controlling when exceptions are reported to `Rails.error.report`. Accepts `:always` (default — preserves existing behavior), `:never` (suppress reporting for expected exceptions like rate limits), or `:terminal_only` (suppress during reattempts, report only when `terminal_action` fires). The executor now defers error reporting until after policy resolution.
 
 ## [0.5.0]
 

data/MANUAL.md

@@ -274,22 +274,30 @@ The loop bodies execute once when Ruby loads the class. Each iteration adds a ne
 
 ### Instance Methods as Steps
 
-For complex steps, you can define instance methods and reference them with `step def`:
+The step definition can be a block, but if you give the step the same name as an instance method of your workflow that method will be called instead:
 
 ```ruby
 class DataExportWorkflow < GenevaDrive::Workflow
-  step def gather_records
-    hero.update!(export_data: hero.exportable_records.to_json)
+  step :anonymize_records
+  
+  def anonymize_records
+    Anonymizer.process_arel(hero.records)
   end
+end
+```
+
+Since `def` in modern Rubies returns the name of the method you define as a `Symbol` you can use the `step def` shorthand as well. Together with "endless methods" this can give you a very compact description:
+
+```ruby
+class DataExportWorkflow < GenevaDrive::Workflow
+  step def gather_records = hero.update!(export_data: hero.exportable_records.to_json)
 
   step def write_to_storage
     Storage.write(hero.export_path, hero.export_data)
     hero.update!(exported_at: Time.current)
   end
 
-  step def notify_user
-    ExportMailer.complete(hero).deliver_later
-  end
+  step def notify_user = ExportMailer.complete(hero).deliver_later
 end
 ```
 
@@ -538,6 +546,312 @@ class OrderFulfillmentWorkflow < GenevaDrive::Workflow
 end
 ```
 
+## Resumable Steps
+
+A regular step must finish within a single job execution. When a step has to churn through a large collection — sending a campaign to 200 000 subscribers, syncing a paginated API, backfilling a table — that single execution becomes a liability: a deploy, a worker restart, or a queue timeout loses all progress. Resumable steps solve this with **cursor-based iteration**: the step periodically checkpoints its position into the database, and can be interrupted and continued in a later job execution from exactly where it left off.
+
+Resumable steps store their state in two extra columns on `geneva_drive_step_executions` (`cursor` and `continues_from_id`), added by the installer migrations — re-run `bin/rails generate geneva_drive:install` on an existing installation to pick them up. Until the migration runs, everything else keeps working: regular steps, pause/resume and housekeeping are unaffected, and executing an actual `resumable_step` fails with a configuration error pointing at the missing migration.
+
+Define one with `resumable_step`. The block receives an `IterableStep` object (API-compatible with Rails 8.1's `ActiveJob::Continuation::Step`):
+
+```ruby
+class CampaignWorkflow < GenevaDrive::Workflow
+  step :prepare do
+    hero.update!(status: "sending")
+  end
+
+  resumable_step :send_notifications do |iter|
+    iter.iterate_over_records(hero.subscribers) do |subscriber|
+      CampaignMailer.notify(hero, subscriber).deliver_later
+    end
+  end
+
+  step :finalize do
+    hero.update!(status: "sent")
+  end
+end
+```
+
+### The Cursor
+
+The cursor is a value persisted on the step execution after every checkpoint. It is whatever your iteration needs to pick up where it stopped: a record ID, a page number, an opaque API token, a date. Cursors are serialized with ActiveJob serializers, so anything ActiveJob can serialize works — including `Date` and `Time` — without manual conversion.
+
+The cursor is a position marker, not a place to store the data being processed: it is rewritten on every checkpoint and copied to every successor execution. To keep that write path cheap, the serialized JSON is limited to 128 KB by default — exceeding it raises `GenevaDrive::CursorTooLargeError`. The limit is configurable in the initializer via `GenevaDrive.max_cursor_size` (`nil` disables the check).
+
+```ruby
+resumable_step :process_records do |iter|
+  hero.records.where("id > ?", iter.cursor || 0).find_each do |record|
+    process(record)
+    iter.set!(record.id)   # persist cursor, check for interruption
+  end
+end
+```
+
+The `IterableStep` API:
+
+| Method | Effect |
+|--------|--------|
+| `iter.cursor` | Current cursor value (`nil` on first run) |
+| `iter.set!(value)` | Set the cursor, persist it, check for interruption |
+| `iter.advance!` | Increment an integer cursor by 1 (integers only) |
+| `iter.checkpoint!` | Persist cursor and check for interruption |
+| `iter.resumed?` | `true` when continuing from a previous execution |
+| `iter.skip_to!(value, wait: nil)` | Set the cursor and suspend immediately; `wait:` delays the continuation |
+
+Helpers for common iteration shapes:
+
+```ruby
+# ActiveRecord relation, one record at a time (find_each under the hood)
+iter.iterate_over_records(hero.subscribers) { |subscriber| ... }
+
+# ActiveRecord relation in batches, yielding relations for bulk operations
+iter.iterate_over_subrelations(hero.subscribers, batch_size: 500) do |batch|
+  batch.update_all(notified_at: Time.current)
+end
+
+# Stable in-memory arrays, index used as cursor
+iter.iterate_over(items) { |item| ... }
+```
+
+### Chained Executions
+
+When a resumable step is interrupted, the current step execution **completes** (with outcome `continued`) and a successor execution is created, linked to its predecessor via `continues_from_id` and carrying the cursor forward. There is no special "suspended" state — each execution is a normal record with a clear start and end, so the full history of a long iteration is visible as a chain:
+
+```ruby
+workflow.step_executions.where(step_name: "send_notifications").order(:created_at)
+# => chunk 1 (completed/continued), chunk 2 (completed/continued), ..., chunk N (completed/success)
+```
+
+A step is interrupted when any of these happen:
+
+- `max_iterations:` is reached (`resumable_step :import, max_iterations: 10_000`)
+- `max_runtime:` is exceeded (`resumable_step :import, max_runtime: 5.minutes`)
+- The job queue signals shutdown (e.g. Sidekiq stopping)
+- The workflow is paused or canceled externally
+- The step calls `skip_to!` or `suspend!` explicitly
+
+```ruby
+# Suspend explicitly, e.g. to respect a rate limit
+resumable_step :sync_api do |iter|
+  page = iter.cursor || 1
+  loop do
+    response = ExternalApi.fetch(page: page)
+    suspend!(wait: response.retry_after) if response.rate_limited?
+    break if response.empty?
+    response.items.each { |item| process(item) }
+    page += 1
+    iter.set!(page)
+  end
+end
+```
+
+### Flow Control and Errors in Resumable Steps
+
+All flow control works inside resumable steps, with cursor-aware semantics:
+
+- `pause!` completes the current execution keeping the cursor; `resume!` continues from it.
+- `reattempt!` continues from the cursor by default; `reattempt!(rewind: true)` clears the cursor and starts the iteration over.
+- `cancel!`, `skip!` and `finished!` behave as in regular steps.
+
+Exception policies (`on_exception:` on the step, class-level `on_exception`, `max_reattempts:`, `terminal_action:`, `report:`) apply exactly as for regular steps. A `:reattempt!` policy continues from the last checkpoint, so a transient failure halfway through a large collection does not redo the completed portion. When an unhandled exception pauses the workflow, `resume!` also retries the failed step from its last checkpoint.
+
+Housekeeping recovery is cursor-aware too: a resumable execution stuck `in_progress` (dead worker) is recovered by continuing from its persisted cursor, not by restarting the iteration.
+
+### Writing Restart-Safe Iterations
+
+The cursor marks the last *checkpointed* position, and one item may be re-processed if execution stops between doing the work and checkpointing. Make each iteration idempotent (e.g. guard with a uniqueness constraint or a state flag on the processed record) rather than assuming exactly-once delivery.
+
+### Testing Resumable Steps
+
+`speedrun_workflow` and `speedrun_current_step` run resumable steps to completion with interruption checks disabled, following the execution chain across explicit suspensions. To exercise partial progress, use `run_iterations`:
+
+```ruby
+test "keeps its place across interruptions" do
+  workflow = CampaignWorkflow.create!(hero: campaign)
+  perform_next_step(workflow)             # :prepare
+
+  run_iterations(workflow, count: 3)      # three iterations, then interrupt
+  assert_cursor(workflow, 3)
+  assert_step_has_successor(workflow, :send_notifications)
+
+  speedrun_workflow(workflow)             # run the rest
+  assert workflow.finished?
+end
+```
+
+## Waiting for Signals
+
+Some steps cannot finish on their own. A payment needs the provider's webhook, a contract needs a countersignature, a shipment needs the carrier to scan the parcel. The usual Rails answer is to poll — schedule the step, check whether the thing happened, `reattempt!(wait: 5.minutes)` if it did not — and that works, but it burns a worker slot on every check and it puts a floor under how fast the workflow can react.
+
+A **signal** inverts that. It is an event record addressed to one workflow instance:
+
+```ruby
+# app/controllers/payment_webhooks_controller.rb
+workflow = OrderFulfillmentWorkflow.ongoing.for_hero(order).first
+workflow.signal!(:payment_confirmed,
+  payload: {amount_cents: event["amount"]},
+  idempotency_key: event["id"])
+```
+
+And a step declares that it waits for one with `wait_for:`:
+
+```ruby
+class OrderFulfillmentWorkflow < GenevaDrive::Workflow
+  step :reserve_stock do
+    hero.reserve_stock!
+  end
+
+  step :capture_payment, wait_for: :payment_confirmed do
+    hero.capture!(received_signal.payload[:amount_cents])
+  end
+
+  step :ship do
+    ShippingLabel.create!(order: hero)
+  end
+end
+```
+
+When `:capture_payment` runs and no matching signal has arrived, its execution parks in the `waiting` state and the job ends. Nothing is enqueued, nothing polls, and the parked row costs nothing until `signal!` wakes it. When the webhook lands, `signal!` wakes the execution and enqueues its job again.
+
+Signals need one migration — it creates the `geneva_drive_signals` table and adds two columns to `geneva_drive_step_executions` — so re-run `bin/rails generate geneva_drive:install` on an existing installation to pick it up. Until then everything else keeps working, and executing a step that declares `wait_for:` fails with a configuration error pointing at the missing migration.
+
+### Arrival Order Does Not Matter
+
+The signal row is written before anything is dispatched, which is what makes the timing irrelevant. There are exactly two places where sender and receiver meet, and both run under the workflow row lock:
+
+- When the waiting step's job runs, it looks for a matching unconsumed signal. Found: attach and run. Not found: park.
+- When `signal!` is called, it looks for parked executions. Found: attach, reschedule, enqueue. Not found: the row simply sits there.
+
+So a webhook that arrives while the workflow is still three steps away from the waiter is not lost and not early — it is buffered, and claimed when the waiting step finally runs. A webhook that arrives a week after the step parked wakes it immediately.
+
+Delivery is one database transaction: the signal row, the waking of the parked execution and the bookkeeping around it either all land or none of them do. If `signal!` raises, nothing was written and the sender can simply try again.
+
+> [!IMPORTANT]
+> `signal!` is an instance method: you find the workflow the way you find anything in Rails, usually `SomeWorkflow.ongoing.for_hero(record).first`. Delivering a brand-new event to a workflow that has already finished or been canceled raises `GenevaDrive::WorkflowNotOngoing` rather than silently doing nothing.
+
+### Deduplicating Redelivery
+
+Webhook providers redeliver. Pass an `idempotency_key:` — the provider's event id is the natural choice — and the second delivery becomes a database-level no-op that returns the row the first one wrote:
+
+```ruby
+signal = workflow.signal!(:payment_confirmed, idempotency_key: event["id"])
+Rails.logger.info("Ignoring redelivered #{event["id"]}") if signal.duplicate_delivery?
+```
+
+Redelivering the very event that finished the workflow is also a no-op, not an error — that specific case is what the deduplication is for. Signals sent without an idempotency key never deduplicate: two calls make two rows, the older one is delivered first, and the newer one stays in the table as an auditable record of what arrived.
+
+### Narrowing What Counts as a Match
+
+`wait_for: :document_signed` is shorthand for `wait_for: GenevaDrive::SignalMatcher.new(:document_signed)`, which matches on name alone. Give the matcher a block and it also has to like the payload:
+
+```ruby
+class ContractWorkflow < GenevaDrive::Workflow
+  # The block receives the payload and runs on the workflow the signal was
+  # sent to, so `hero` and the workflow's own methods are in scope.
+  step :await_countersignature,
+    wait_for: GenevaDrive::SignalMatcher.new(:document_signed) { |payload|
+      payload[:document_id] == hero.contract_id
+    } do
+    hero.mark_countersigned!
+  end
+end
+```
+
+A matcher is a plain object with no ties to the step that uses it, so a rule worth repeating can live in a constant and be shared across steps and workflows:
+
+```ruby
+SETTLED = GenevaDrive::SignalMatcher.new(:payment_confirmed) { |payload| payload[:amount_cents].to_i > 0 }
+
+step :capture_payment, wait_for: SETTLED do
+  hero.capture!
+end
+```
+
+Anything responding to `#matches?(signal)` works too, and owns the whole predicate — which is how a single step waits on either of two names:
+
+```ruby
+class PaymentSettled
+  def initialize(min_cents:) = @min_cents = min_cents
+
+  def matches?(signal)
+    %w[payment_confirmed payment_captured].include?(signal.name) &&
+      signal.payload[:amount_cents].to_i >= @min_cents
+  end
+end
+
+step :capture_payment, wait_for: PaymentSettled.new(min_cents: 100) do
+  hero.capture!
+end
+```
+
+Payloads are serialized with ActiveJob serializers, so `Date`, `Time` and ActiveRecord objects survive the round trip. Hashes come back with indifferent access, because a webhook sender produces string keys and a Ruby caller produces symbols and a matcher should not have to care. The serialized payload is limited to 128 KB (`GenevaDrive.max_signal_payload_size`, `nil` disables) — a payload describes the event; the data your workflow works on belongs on the hero.
+
+> [!WARNING]
+> Matchers run at the gate *and* inside `signal!`, in the sender's process. Keep them cheap and free of side effects, exactly like `skip_if:`. A matcher that raises during `signal!` raises to whoever sent the signal — and because `signal!` is a single transaction, the delivery rolls back whole: no signal row is recorded, nothing is attached or woken, no counter moves. The sender (or the webhook provider's retry) re-delivers once the matcher is fixed.
+
+### How Waiting Composes
+
+- `wait: 2.days, wait_for: :payment_confirmed` means "no earlier than two days from now, and only once signaled". A signal arriving during the delay is buffered and claimed when the step finally runs.
+- `skip_if:` is evaluated before waiting, so "wait for the signature — unless the contract was pre-signed" skips immediately instead of parking forever. Both `skip_if:` and `cancel_if` are re-evaluated when a signal wakes the step; a parked workflow does not re-check them while it sleeps. A step skipped at wake consumes the signal that woke it, so the next waiter does not inherit a stale event.
+- `resumable_step` accepts `wait_for:`. The wait happens once, at the start of the chain, and `received_signal` stays the same across every successor execution.
+- A reattempt — yours or an exception policy's — re-reads the *same* signal. Reattempting means "process this event again", not "wait for another event".
+
+The signal itself moves through `pending` → `claimed` → `consumed`. It is claimed when an execution attaches to it, and consumed once every attached execution has finished cleanly, in the same transaction as the step's own completion. A step that fails, pauses, or gets canceled leaves its signal claimed, so the retry picks up the same payload and the audit trail does not claim an event was handled when it was not.
+
+Skipping counts as finishing: skipping a step that was delivered a signal consumes that signal, whether the skip came from `skip_if:`, from `skip!` inside the step, from an exception policy, or from an operator calling `workflow.skip!` on a ready or paused workflow. Moving past a step deliberately settles the event it was given. Skipping a step that never attached — one still parked, or one whose `skip_if:` fired before it ever looked for a signal — leaves buffered signals exactly as they were.
+
+Two counters on the row make that legible: `signal.claimed` counts attachments (a step that failed and was retried attaches twice), and `signal.consumed` counts cleanly resolved ones. In a linear workflow they end at one apiece. The predicates `signal.claimed?` and `signal.consumed?` read those counters, so they answer "was this event ever picked up" and "was it ever handled" — which is not the same question as `signal.state`, and is usually the one worth asking.
+
+> [!NOTE]
+> Two sequential steps waiting on `:payment_confirmed` need two signals. The first step consumes the first signal, so the second step parks until another one arrives.
+
+### Getting Unstuck
+
+There is no `timeout:` option. Waiting forever is a legitimate state — a contract may genuinely sit unsigned for months — so instead of a timer, waiting is made loud: a distinct `waiting` state, a `waiting_since` timestamp, and two housekeeping gauges, `geneva_drive.waiting_step_executions` and `geneva_drive.waiting_overdue` (parked longer than `GenevaDrive.waiting_visibility_threshold`, seven days by default). Alert on the overdue gauge.
+
+When something does stall, three escape hatches already exist:
+
+- `workflow.skip!` abandons the wait and moves to the next step.
+- `workflow.cancel!` gives up on the workflow.
+- The queue is already a clock: schedule a job that signals a deadline, and let the step decide.
+
+```ruby
+# Send the workflow its own deadline; the step reads it off the payload
+class ContractDeadlineJob < ApplicationJob
+  def perform(contract)
+    workflow = ContractWorkflow.ongoing.for_hero(contract).first
+    workflow&.signal!(:document_signed,
+      payload: {timed_out: true},
+      idempotency_key: "deadline-#{workflow.id}")
+  end
+end
+
+step :await_countersignature, wait_for: :document_signed do
+  cancel! if received_signal.payload[:timed_out]
+  hero.mark_countersigned!
+end
+```
+
+### Testing Waiting Steps
+
+The step-driving helpers refuse to spin on a parked execution — they raise and tell you which signal is missing, because a test that hangs on a rendezvous is far worse than one that fails:
+
+```ruby
+test "captures once the payment is confirmed" do
+  workflow = OrderFulfillmentWorkflow.create!(hero: order)
+  perform_next_step(workflow)                    # :reserve_stock
+  perform_next_step(workflow)                    # :capture_payment parks
+
+  assert_waiting_for_signal(workflow, :payment_confirmed)
+
+  workflow.signal!(:payment_confirmed, payload: {amount_cents: 12_500})
+  speedrun_workflow(workflow)
+
+  assert_signal_state(workflow, :payment_confirmed, :consumed)
+  assert workflow.finished?
+end
+```
+
 ## Exception Handling
 
 ### Default Behavior
@@ -644,7 +958,7 @@ step :flaky_api, on_exception: :reattempt!, max_reattempts: 10, terminal_action:
 end
 ```
 
-`terminal_action:` accepts `:pause!` (default) or `:cancel!`.
+`terminal_action:` accepts `:pause!` (default), `:cancel!`, or `:skip!`.
 
 ### Reusable Exception Policies
 
@@ -808,6 +1122,63 @@ If the block returns without calling a flow control method, the workflow pauses.
 
 Imperative handlers cannot be combined with `max_reattempts:`, `wait:`, or `terminal_action:` — manage that logic inside the block.
 
+### Controlling Error Reporting
+
+When a step raises an exception, GenevaDrive reports it to your error tracker via `Rails.error.report`. This is the right default — you want visibility into failures. But some exceptions are expected and handled: rate limits, transient timeouts, throttling responses. Reporting these on every reattempt floods your error tracker with noise and obscures the errors that actually need attention.
+
+The `report:` option controls when `Rails.error.report` is called. It accepts three values:
+
+- `:always` — (default) Report every exception, regardless of what the policy does with it. This is the safest choice and preserves the behavior you're used to.
+- `:never` — Never report the exception. Use this for errors that are fully expected and handled — rate limits, throttles, circuit breaker trips. The exception still triggers the policy action (reattempt, skip, etc.), but your error tracker stays clean.
+- `:terminal_only` — Suppress reports while the step is being reattempted, but report when reattempts are exhausted and the `terminal_action` fires. This is the sweet spot for transient errors: you don't care about individual retries, but you want to know when the retries give up.
+
+Use `report:` on both class-level `on_exception` and on `ExceptionPolicy` objects:
+
+```ruby
+class CalendarSyncWorkflow < GenevaDrive::Workflow
+  # Rate limits are expected — never report them
+  on_exception Pecorino::Throttle::Throttled, report: :never do |error|
+    reattempt!(wait: error.retry_after.clamp(10, 600).seconds)
+  end
+
+  # Transient timeouts: only report when we give up
+  on_exception Net::OpenTimeout,
+    action: :reattempt!,
+    wait: 10.seconds,
+    max_reattempts: 5,
+    terminal_action: :cancel!,
+    report: :terminal_only
+
+  step :sync_events do
+    GoogleCalendar.sync(hero)
+  end
+end
+```
+
+The same option works on `ExceptionPolicy` objects for reusable policies and composable arrays:
+
+```ruby
+RATE_LIMIT_POLICY = GenevaDrive::ExceptionPolicy.new(
+  :reattempt!,
+  matching: RateLimitError,
+  wait: 30.seconds,
+  max_reattempts: 10,
+  terminal_action: :pause!,
+  report: :terminal_only
+)
+
+step :call_api, on_exception: [
+  RATE_LIMIT_POLICY,
+  GenevaDrive::ExceptionPolicy.new(:reattempt!, matching: Timeout::Error, report: :never, max_reattempts: 3),
+  GenevaDrive::ExceptionPolicy.new(:pause!)  # blanket fallback, reports by default
+] do
+  ExternalApi.call(hero)
+end
+```
+
+> [!IMPORTANT]
+> The `report:` option only controls `Rails.error.report`. The exception is still re-raised after the executor commits its state transitions — your background job framework (Sidekiq, SolidQueue, etc.) will see it. If your error tracker also hooks into the job framework's error handler, you may need to configure that separately.
+
 ### Manual Exception Handling
 
 For the most granular control, handle exceptions directly within the step using standard Ruby `rescue`:
@@ -882,6 +1253,7 @@ Step executions have their own state machine:
 | State | Meaning |
 |-------|---------|
 | `scheduled` | Waiting to run |
+| `waiting` | Parked until a matching signal arrives |
 | `in_progress` | Currently executing |
 | `completed` | Finished successfully |
 | `failed` | Exception occurred |
@@ -1170,19 +1542,65 @@ The method is thread-safe and won't affect other concurrent requests.
 
 ### Custom Job Options
 
-Override the queue or priority for all steps in a workflow:
+You can override the Active Job settings used to enqueue a workflow's step jobs at three levels: for the entire workflow class, for an individual step, or for a specific workflow instance. All three accept the same keys that Active Job's `set` method understands: `:queue`, `:priority`, `:wait`, and `:wait_until`. Unknown keys are rejected at configuration time so that typos surface immediately instead of silently falling back to defaults.
 
 ```ruby
 class HighPriorityWorkflow < GenevaDrive::Workflow
   set_step_job_options queue: :critical, priority: 0
 
-  step :urgent_action do
+  step :urgent_action, job_options: {queue: :urgent, priority: -1} do
     UrgentService.process!(hero)
   end
 end
 ```
 
-The options are passed directly to ActiveJob's `set` method.
+Options merge from lowest to highest precedence: **class defaults** (`set_step_job_options`) are overridden by **per-instance options** (`workflow.step_job_options = {...}`), which are in turn overridden by **per-step options** (`step ..., job_options: {...}`). The merged hash is passed straight to `PerformStepJob.set(...)` for every enqueue of that step, including reattempts and resume re-enqueueing.
+
+Per-instance options are stored in the workflow's `metadata` column, so they survive across step boundaries and are excluded from the dedupe uniqueness check — two workflows for the same hero cannot coexist just because their job options differ.
+
+#### When per-step job options matter: protecting in-flight work from a fresh backlog
+
+Consider a workflow that downloads, processes, and cleans up a large artifact per hero:
+
+```ruby
+class RubygemsWorkflow < GenevaDrive::Workflow
+  step :extract do
+    # Download a large tarball to local disk
+  end
+
+  step :process do
+    # Do work on the extracted data
+  end
+
+  step :cleanup do
+    # Delete the on-disk artifact
+  end
+end
+```
+
+If you enqueue 50,000 of these at once, all 50,000 `:extract` jobs land on the queue at the default priority — ahead of the `:process` and `:cleanup` steps of workflows that have already downloaded something. On queue adapters where lower numbers run first, the workers happily drain `:extract` jobs first, filling the disk with unprocessed artifacts until the volume runs out and everything starts failing.
+
+Bumping the later steps to a higher priority (lower number) fixes this:
+
+```ruby
+class RubygemsWorkflow < GenevaDrive::Workflow
+  step :extract, job_options: {priority: 10} do
+    # Download a large tarball to local disk
+  end
+
+  step :process do  # runs at default priority — ahead of :extract
+    # Do work on the extracted data
+  end
+
+  step :cleanup do  # also runs ahead of :extract
+    # Delete the on-disk artifact
+  end
+end
+```
+
+Workflows that have already extracted their artifact can now drain through `:process` and `:cleanup` — releasing disk — while new `:extract` jobs wait their turn. The fleet reaches steady state instead of collapsing under its own backlog.
+
+The same pattern is useful whenever a later step releases a scarce resource (disk, external quota, a database lock) that earlier steps consume: give the release-work a higher priority than the acquire-work.
 
 ## Housekeeping
 
@@ -1351,6 +1769,18 @@ ActiveSupport::Notifications.subscribe("step.geneva_drive") do |event|
 end
 ```
 
+### Metric gauges
+
+`GenevaDrive::HousekeepingJob` reports gauges via [Measurometer](https://rubygems.org/gems/measurometer) on every run:
+
+| Gauge | Tags | Value |
+|-------|------|-------|
+| `geneva_drive.<state>` | _(none)_ | Absolute count of workflows in `<state>` (e.g. `ready`, `paused`, `finished`), summed across all classes |
+| `geneva_drive.<state>` | `workflow: <ClassName>` | Absolute count of workflows in `<state>` for a single workflow class |
+| `geneva_drive.paused_ratio` | `workflow: <ClassName>` | Float `0.0..1.0` — that class's `paused` count divided by its total population (all states) |
+
+`paused_ratio` is normalized on purpose: absolute paused counts are hard to alert on (50 paused of 50 is a fire; 50 of 500,000 is noise), whereas the ratio is bounded and comparable across classes. The denominator is the whole population (paused ÷ total), so it stays within `0..1` even when nothing is ongoing.
+
 ---
 
 # Part VI — Appendix
@@ -1704,9 +2134,10 @@ end
 |--------|--------|
 | `cancel!` | Stop workflow, mark canceled |
 | `pause!` | Stop workflow, await manual resume |
-| `reattempt!(wait:)` | Retry current step, optionally after delay |
+| `reattempt!(wait:, rewind:)` | Retry current step; `rewind: true` clears cursor |
 | `skip!` | Skip current step, proceed to next |
 | `finished!` | Complete workflow early |
+| `suspend!(wait:)` | Interrupt resumable step, continue via successor after delay |
 
 ### Workflow States
 
@@ -1723,19 +2154,26 @@ end
 | State | Meaning |
 |-------|---------|
 | `scheduled` | Waiting to run |
+| `waiting` | Parked until a matching signal arrives |
 | `in_progress` | Currently executing |
 | `completed` | Finished successfully |
 | `failed` | Exception occurred |
 | `canceled` | Canceled before execution |
 | `skipped` | Skipped via `skip_if` or `skip!` |
 
+A resumable step interrupted mid-iteration completes its execution with outcome `continued` and schedules a successor execution — there is no separate state for it.
+
 ### Step Options
 
 | Option | Type | Description |
 |--------|------|-------------|
 | `wait:` | Duration | Delay before step executes |
+| `wait_for:` | Symbol, String, `SignalMatcher`, matcher object | Park until a matching signal arrives |
+| `job_options:` | Hash | Options passed to Active Job's `set` method for this step |
 | `skip_if:` | Proc, Symbol, Boolean | Condition to skip step |
 | `on_exception:` | Symbol | Exception handler (`:pause!`, `:cancel!`, `:reattempt!`, `:skip!`) |
 | `max_reattempts:` | Integer, nil | Max consecutive reattempts before pausing (default: 100, `nil` = unlimited) |
 | `before_step:` | Symbol | Insert before this step |
 | `after_step:` | Symbol | Insert after this step |
+| `max_iterations:` | Integer | (resumable_step) Interrupt after N iterations, continue via successor |
+| `max_runtime:` | Duration | (resumable_step) Interrupt after duration elapsed, continue via successor |

data/README.md

@@ -8,6 +8,8 @@ GenevaDrive provides a clean DSL for defining multi-step workflows that execute
 
 - **Hero-oriented**: Workflows are associated with a polymorphic "hero" (the subject of the workflow)
 - **Step-based execution**: Define workflows as a series of steps with optional wait times
+- **Resumable steps**: Iterate over large collections with a database-checkpointed cursor, surviving interruptions
+- **Signals**: Steps can park until an external event arrives (`wait_for:`), woken by `workflow.signal!` from a webhook or controller
 - **Durable**: Steps are persisted to the database, surviving process restarts
 - **Idempotent**: Database constraints ensure each step runs exactly once
 - **Flow control**: Methods like `cancel!`, `pause!`, `skip!`, `reattempt!`, and `finished!`
@@ -164,12 +166,14 @@ class PremiumWorkflow < BaseWorkflow
   cancel_if { hero.subscription_expired? }
   set_step_job_options queue: :premium, priority: 10
 
-  step :premium_feature do
+  step :premium_feature, job_options: {priority: -1} do
     # Inherits cancel_if from parent
   end
 end
 ```
 
+Step job options override workflow defaults when the step is initially enqueued, reattempted, or resumed. Options are passed directly to Active Job's `set` method.
+
 ### Hero Deletion Handling
 
 By default, workflows cancel if their hero is deleted:

data/lib/generators/geneva_drive/install/install_generator.rb

@@ -46,10 +46,30 @@ module GenevaDrive
           "db/migrate/add_metadata_to_geneva_drive_step_executions.rb"
         )
 
+        migration_template(
+          "add_metadata_to_workflows.rb",
+          "db/migrate/add_metadata_to_geneva_drive_workflows.rb"
+        )
+
         migration_template(
           "allow_null_hero_on_workflows.rb",
           "db/migrate/allow_null_hero_on_geneva_drive_workflows.rb"
         )
+
+        migration_template(
+          "add_started_at_index_to_step_executions.rb",
+          "db/migrate/add_started_at_index_to_geneva_drive_step_executions.rb"
+        )
+
+        migration_template(
+          "add_resumable_step_support.rb",
+          "db/migrate/add_resumable_step_support_to_geneva_drive_step_executions.rb"
+        )
+
+        migration_template(
+          "add_signals_support.rb",
+          "db/migrate/add_signals_support_to_geneva_drive.rb"
+        )
       end
 
       # Creates the initializer file.