geneva_drive 0.5.0 → 0.6.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 45df42e923b2d50be98ff8572ae6c2391faa03c9d604372aabd3bb9ac603507e
-  data.tar.gz: 84e99503be26c28fef613972a4b8a6335d96cc87e775076def279857513825bf
+  metadata.gz: a4a73216fcf8355550c3b5e26e76eee14468eb301e8eb315a979bc8f34bdb670
+  data.tar.gz: a9b9cc0d8998f6c479806158dd12725774431e7dba8486c5781ed68f238175b5
 SHA512:
-  metadata.gz: d8b2efa46d1e7d43da57afae65a3564e3826801d29a9f142b9217c3bc462f0f48122993c01a8710adbf968733db5c05e35fad081901e824f3b9ff87ca577bca0
-  data.tar.gz: fbfd86cec20556569803023f9a45483cc73cd7ded12d775dd2684f6693a883b1bff6d69ab61c40d006cd40715d6b696840f894f988001e7c61d8a07080283a44
+  metadata.gz: b77f44d451d256d081af6d7b30d62bcaa46bfd41b0606cdcd1d07529e18db7ff5e2d1753d52f6bd24f600f92b7e14587146072a53e46e1052728c760cf6ded78
+  data.tar.gz: 9bdd9f474cd8370ebcc77b4126617833a249b6e1237ed8021463b66d082db63e17bc024fa8718ec3aee1d031731c72b3a9c90903d2ce40ced2f01eb0b1c421ef

data/CHANGELOG.md

@@ -2,6 +2,17 @@
 
 ## [Unreleased]
 
+## [0.6.0]
+
+- Add resumable steps for cursor-based iteration over large collections. `resumable_step :name do |iter| ... end` passes an `IterableStep` (API-compatible with Rails 8.1's `ActiveJob::Continuation::Step`) whose cursor is checkpointed to the database, so the step can be interrupted — by `max_iterations:`, `max_runtime:`, queue shutdown, external pause/cancel, or explicit `suspend!`/`skip_to!` — and continued in a later job execution from where it left off. Interruption completes the current step execution with outcome `continued` and creates a successor execution linked via `continues_from_id`, keeping a full audit chain instead of introducing a suspended state. Resumable steps run through the same `Executor` as regular steps: exception policies (step- and class-level, `max_reattempts`, `terminal_action:`, `report:`), tagged/injected loggers, `Rails.error` context, and merged job options all apply, with `reattempt!` and `resume!` continuing from the last checkpoint (use `reattempt!(rewind: true)` to start over). `HousekeepingJob` recovery is cursor-aware. The feature needs two new columns (`cursor`, `continues_from_id`) added by the installer migration; until it runs, everything else degrades gracefully (regular steps, pause/resume and housekeeping are unaffected) and executing an actual `resumable_step` raises a configuration error pointing at the missing migration. Cursors are limited to 128 KB of serialized JSON by default — the cursor is a position marker, not a payload — configurable via `GenevaDrive.max_cursor_size` (`nil` disables the check). Test helpers: `speedrun_current_step` follows the execution chain, `run_iterations(workflow, count:)` exercises partial progress, plus `assert_cursor` and `assert_step_has_successor`.
+- Validate `job_options` uniformly across `set_step_job_options`, per-step `job_options:`, and per-instance `workflow.step_job_options=`. Only the keys Active Job's `set` actually consumes (`:queue`, `:priority`, `:wait`, `:wait_until`) are accepted, and values are type-checked. Typos like `piority:` now raise `StepConfigurationError` at configuration time instead of being silently ignored.
+- Add partial index on `geneva_drive_step_executions(started_at) WHERE state = 'in_progress'` and order `HousekeepingJob` recovery batches by the filter column (`started_at` for stuck-in-progress, `scheduled_for` for stuck-scheduled). Fixes `HousekeepingJob` timing out on large in-progress sets where most rows are fresh — the planner previously walked the whole `state = 'in_progress'` set looking for old rows and tripped the caller's `statement_timeout`. Fixes AppSignal incident b9a7d1d1.
+- Add a normalized `geneva_drive.paused_ratio` gauge emitted by `HousekeepingJob`. For each workflow class (STI `type`) it reports a float in `0.0..1.0`, tagged `workflow: <ClassName>`, equal to that class's `paused` count divided by its total population (all states). Unlike absolute paused counts, this is bounded and easy to alert on (50 paused of 50 is a fire; 50 of 500,000 is noise). Derived from the existing single grouped query — no extra database load.
+- Add optional `metadata` JSON column to workflows (mirrors the existing step executions pattern). Exposes `step_job_options` / `step_job_options=` for per-instance job option overrides (queue, priority, etc.) that merge with class-level `set_step_job_options` and persist across step boundaries. Dedupe is unaffected since metadata is not part of the uniqueness constraint.
+- Add GenevaDrive workflow and step metadata to `Rails.error` execution context when steps execute, so Rails error reports include workflow id/class, step execution id/name, and hero identifiers.
+- Add per-step Active Job option overrides with `step :name, job_options: { queue: :critical, priority: 0 }`. Step options take precedence over workflow-level `set_step_job_options` and per-instance options, and are preserved across reattempts and resume re-enqueueing.
+- Add `report:` option to exception policies and class-level `on_exception` for controlling when exceptions are reported to `Rails.error.report`. Accepts `:always` (default — preserves existing behavior), `:never` (suppress reporting for expected exceptions like rate limits), or `:terminal_only` (suppress during reattempts, report only when `terminal_action` fires). The executor now defers error reporting until after policy resolution.
+
 ## [0.5.0]
 
 - Add `ongoing?` predicate and model-level uniqueness validation for ongoing workflows. Mirrors the database unique index on `(type, hero_type, hero_id)` so that `Workflow.create` (without bang) returns validation errors instead of raising a database constraint violation.

data/MANUAL.md

@@ -274,22 +274,30 @@ The loop bodies execute once when Ruby loads the class. Each iteration adds a ne
 
 ### Instance Methods as Steps
 
-For complex steps, you can define instance methods and reference them with `step def`:
+The step definition can be a block, but if you give the step the same name as an instance method of your workflow that method will be called instead:
 
 ```ruby
 class DataExportWorkflow < GenevaDrive::Workflow
-  step def gather_records
-    hero.update!(export_data: hero.exportable_records.to_json)
+  step :anonymize_records
+  
+  def anonymize_records
+    Anonymizer.process_arel(hero.records)
   end
+end
+```
+
+Since `def` in modern Rubies returns the name of the method you define as a `Symbol` you can use the `step def` shorthand as well. Together with "endless methods" this can give you a very compact description:
+
+```ruby
+class DataExportWorkflow < GenevaDrive::Workflow
+  step def gather_records = hero.update!(export_data: hero.exportable_records.to_json)
 
   step def write_to_storage
     Storage.write(hero.export_path, hero.export_data)
     hero.update!(exported_at: Time.current)
   end
 
-  step def notify_user
-    ExportMailer.complete(hero).deliver_later
-  end
+  step def notify_user = ExportMailer.complete(hero).deliver_later
 end
 ```
 
@@ -538,6 +546,139 @@ class OrderFulfillmentWorkflow < GenevaDrive::Workflow
 end
 ```
 
+## Resumable Steps
+
+A regular step must finish within a single job execution. When a step has to churn through a large collection — sending a campaign to 200 000 subscribers, syncing a paginated API, backfilling a table — that single execution becomes a liability: a deploy, a worker restart, or a queue timeout loses all progress. Resumable steps solve this with **cursor-based iteration**: the step periodically checkpoints its position into the database, and can be interrupted and continued in a later job execution from exactly where it left off.
+
+Resumable steps store their state in two extra columns on `geneva_drive_step_executions` (`cursor` and `continues_from_id`), added by the installer migrations — re-run `bin/rails generate geneva_drive:install` on an existing installation to pick them up. Until the migration runs, everything else keeps working: regular steps, pause/resume and housekeeping are unaffected, and executing an actual `resumable_step` fails with a configuration error pointing at the missing migration.
+
+Define one with `resumable_step`. The block receives an `IterableStep` object (API-compatible with Rails 8.1's `ActiveJob::Continuation::Step`):
+
+```ruby
+class CampaignWorkflow < GenevaDrive::Workflow
+  step :prepare do
+    hero.update!(status: "sending")
+  end
+
+  resumable_step :send_notifications do |iter|
+    iter.iterate_over_records(hero.subscribers) do |subscriber|
+      CampaignMailer.notify(hero, subscriber).deliver_later
+    end
+  end
+
+  step :finalize do
+    hero.update!(status: "sent")
+  end
+end
+```
+
+### The Cursor
+
+The cursor is a value persisted on the step execution after every checkpoint. It is whatever your iteration needs to pick up where it stopped: a record ID, a page number, an opaque API token, a date. Cursors are serialized with ActiveJob serializers, so anything ActiveJob can serialize works — including `Date` and `Time` — without manual conversion.
+
+The cursor is a position marker, not a place to store the data being processed: it is rewritten on every checkpoint and copied to every successor execution. To keep that write path cheap, the serialized JSON is limited to 128 KB by default — exceeding it raises `GenevaDrive::CursorTooLargeError`. The limit is configurable in the initializer via `GenevaDrive.max_cursor_size` (`nil` disables the check).
+
+```ruby
+resumable_step :process_records do |iter|
+  hero.records.where("id > ?", iter.cursor || 0).find_each do |record|
+    process(record)
+    iter.set!(record.id)   # persist cursor, check for interruption
+  end
+end
+```
+
+The `IterableStep` API:
+
+| Method | Effect |
+|--------|--------|
+| `iter.cursor` | Current cursor value (`nil` on first run) |
+| `iter.set!(value)` | Set the cursor, persist it, check for interruption |
+| `iter.advance!` | Increment an integer cursor by 1 (integers only) |
+| `iter.checkpoint!` | Persist cursor and check for interruption |
+| `iter.resumed?` | `true` when continuing from a previous execution |
+| `iter.skip_to!(value, wait: nil)` | Set the cursor and suspend immediately; `wait:` delays the continuation |
+
+Helpers for common iteration shapes:
+
+```ruby
+# ActiveRecord relation, one record at a time (find_each under the hood)
+iter.iterate_over_records(hero.subscribers) { |subscriber| ... }
+
+# ActiveRecord relation in batches, yielding relations for bulk operations
+iter.iterate_over_subrelations(hero.subscribers, batch_size: 500) do |batch|
+  batch.update_all(notified_at: Time.current)
+end
+
+# Stable in-memory arrays, index used as cursor
+iter.iterate_over(items) { |item| ... }
+```
+
+### Chained Executions
+
+When a resumable step is interrupted, the current step execution **completes** (with outcome `continued`) and a successor execution is created, linked to its predecessor via `continues_from_id` and carrying the cursor forward. There is no special "suspended" state — each execution is a normal record with a clear start and end, so the full history of a long iteration is visible as a chain:
+
+```ruby
+workflow.step_executions.where(step_name: "send_notifications").order(:created_at)
+# => chunk 1 (completed/continued), chunk 2 (completed/continued), ..., chunk N (completed/success)
+```
+
+A step is interrupted when any of these happen:
+
+- `max_iterations:` is reached (`resumable_step :import, max_iterations: 10_000`)
+- `max_runtime:` is exceeded (`resumable_step :import, max_runtime: 5.minutes`)
+- The job queue signals shutdown (e.g. Sidekiq stopping)
+- The workflow is paused or canceled externally
+- The step calls `skip_to!` or `suspend!` explicitly
+
+```ruby
+# Suspend explicitly, e.g. to respect a rate limit
+resumable_step :sync_api do |iter|
+  page = iter.cursor || 1
+  loop do
+    response = ExternalApi.fetch(page: page)
+    suspend!(wait: response.retry_after) if response.rate_limited?
+    break if response.empty?
+    response.items.each { |item| process(item) }
+    page += 1
+    iter.set!(page)
+  end
+end
+```
+
+### Flow Control and Errors in Resumable Steps
+
+All flow control works inside resumable steps, with cursor-aware semantics:
+
+- `pause!` completes the current execution keeping the cursor; `resume!` continues from it.
+- `reattempt!` continues from the cursor by default; `reattempt!(rewind: true)` clears the cursor and starts the iteration over.
+- `cancel!`, `skip!` and `finished!` behave as in regular steps.
+
+Exception policies (`on_exception:` on the step, class-level `on_exception`, `max_reattempts:`, `terminal_action:`, `report:`) apply exactly as for regular steps. A `:reattempt!` policy continues from the last checkpoint, so a transient failure halfway through a large collection does not redo the completed portion. When an unhandled exception pauses the workflow, `resume!` also retries the failed step from its last checkpoint.
+
+Housekeeping recovery is cursor-aware too: a resumable execution stuck `in_progress` (dead worker) is recovered by continuing from its persisted cursor, not by restarting the iteration.
+
+### Writing Restart-Safe Iterations
+
+The cursor marks the last *checkpointed* position, and one item may be re-processed if execution stops between doing the work and checkpointing. Make each iteration idempotent (e.g. guard with a uniqueness constraint or a state flag on the processed record) rather than assuming exactly-once delivery.
+
+### Testing Resumable Steps
+
+`speedrun_workflow` and `speedrun_current_step` run resumable steps to completion with interruption checks disabled, following the execution chain across explicit suspensions. To exercise partial progress, use `run_iterations`:
+
+```ruby
+test "keeps its place across interruptions" do
+  workflow = CampaignWorkflow.create!(hero: campaign)
+  perform_next_step(workflow)             # :prepare
+
+  run_iterations(workflow, count: 3)      # three iterations, then interrupt
+  assert_cursor(workflow, 3)
+  assert_step_has_successor(workflow, :send_notifications)
+
+  speedrun_workflow(workflow)             # run the rest
+  assert workflow.finished?
+end
+```
+
 ## Exception Handling
 
 ### Default Behavior
@@ -644,7 +785,7 @@ step :flaky_api, on_exception: :reattempt!, max_reattempts: 10, terminal_action:
 end
 ```
 
-`terminal_action:` accepts `:pause!` (default) or `:cancel!`.
+`terminal_action:` accepts `:pause!` (default), `:cancel!`, or `:skip!`.
 
 ### Reusable Exception Policies
 
@@ -808,6 +949,63 @@ If the block returns without calling a flow control method, the workflow pauses.
 
 Imperative handlers cannot be combined with `max_reattempts:`, `wait:`, or `terminal_action:` — manage that logic inside the block.
 
+### Controlling Error Reporting
+
+When a step raises an exception, GenevaDrive reports it to your error tracker via `Rails.error.report`. This is the right default — you want visibility into failures. But some exceptions are expected and handled: rate limits, transient timeouts, throttling responses. Reporting these on every reattempt floods your error tracker with noise and obscures the errors that actually need attention.
+
+The `report:` option controls when `Rails.error.report` is called. It accepts three values:
+
+- `:always` — (default) Report every exception, regardless of what the policy does with it. This is the safest choice and preserves the behavior you're used to.
+- `:never` — Never report the exception. Use this for errors that are fully expected and handled — rate limits, throttles, circuit breaker trips. The exception still triggers the policy action (reattempt, skip, etc.), but your error tracker stays clean.
+- `:terminal_only` — Suppress reports while the step is being reattempted, but report when reattempts are exhausted and the `terminal_action` fires. This is the sweet spot for transient errors: you don't care about individual retries, but you want to know when the retries give up.
+
+Use `report:` on both class-level `on_exception` and on `ExceptionPolicy` objects:
+
+```ruby
+class CalendarSyncWorkflow < GenevaDrive::Workflow
+  # Rate limits are expected — never report them
+  on_exception Pecorino::Throttle::Throttled, report: :never do |error|
+    reattempt!(wait: error.retry_after.clamp(10, 600).seconds)
+  end
+
+  # Transient timeouts: only report when we give up
+  on_exception Net::OpenTimeout,
+    action: :reattempt!,
+    wait: 10.seconds,
+    max_reattempts: 5,
+    terminal_action: :cancel!,
+    report: :terminal_only
+
+  step :sync_events do
+    GoogleCalendar.sync(hero)
+  end
+end
+```
+
+The same option works on `ExceptionPolicy` objects for reusable policies and composable arrays:
+
+```ruby
+RATE_LIMIT_POLICY = GenevaDrive::ExceptionPolicy.new(
+  :reattempt!,
+  matching: RateLimitError,
+  wait: 30.seconds,
+  max_reattempts: 10,
+  terminal_action: :pause!,
+  report: :terminal_only
+)
+
+step :call_api, on_exception: [
+  RATE_LIMIT_POLICY,
+  GenevaDrive::ExceptionPolicy.new(:reattempt!, matching: Timeout::Error, report: :never, max_reattempts: 3),
+  GenevaDrive::ExceptionPolicy.new(:pause!)  # blanket fallback, reports by default
+] do
+  ExternalApi.call(hero)
+end
+```
+
+> [!IMPORTANT]
+> The `report:` option only controls `Rails.error.report`. The exception is still re-raised after the executor commits its state transitions — your background job framework (Sidekiq, SolidQueue, etc.) will see it. If your error tracker also hooks into the job framework's error handler, you may need to configure that separately.
+
 ### Manual Exception Handling
 
 For the most granular control, handle exceptions directly within the step using standard Ruby `rescue`:
@@ -1170,19 +1368,65 @@ The method is thread-safe and won't affect other concurrent requests.
 
 ### Custom Job Options
 
-Override the queue or priority for all steps in a workflow:
+You can override the Active Job settings used to enqueue a workflow's step jobs at three levels: for the entire workflow class, for an individual step, or for a specific workflow instance. All three accept the same keys that Active Job's `set` method understands: `:queue`, `:priority`, `:wait`, and `:wait_until`. Unknown keys are rejected at configuration time so that typos surface immediately instead of silently falling back to defaults.
 
 ```ruby
 class HighPriorityWorkflow < GenevaDrive::Workflow
   set_step_job_options queue: :critical, priority: 0
 
-  step :urgent_action do
+  step :urgent_action, job_options: {queue: :urgent, priority: -1} do
     UrgentService.process!(hero)
   end
 end
 ```
 
-The options are passed directly to ActiveJob's `set` method.
+Options merge from lowest to highest precedence: **class defaults** (`set_step_job_options`) are overridden by **per-instance options** (`workflow.step_job_options = {...}`), which are in turn overridden by **per-step options** (`step ..., job_options: {...}`). The merged hash is passed straight to `PerformStepJob.set(...)` for every enqueue of that step, including reattempts and resume re-enqueueing.
+
+Per-instance options are stored in the workflow's `metadata` column, so they survive across step boundaries and are excluded from the dedupe uniqueness check — two workflows for the same hero cannot coexist just because their job options differ.
+
+#### When per-step job options matter: protecting in-flight work from a fresh backlog
+
+Consider a workflow that downloads, processes, and cleans up a large artifact per hero:
+
+```ruby
+class RubygemsWorkflow < GenevaDrive::Workflow
+  step :extract do
+    # Download a large tarball to local disk
+  end
+
+  step :process do
+    # Do work on the extracted data
+  end
+
+  step :cleanup do
+    # Delete the on-disk artifact
+  end
+end
+```
+
+If you enqueue 50,000 of these at once, all 50,000 `:extract` jobs land on the queue at the default priority — ahead of the `:process` and `:cleanup` steps of workflows that have already downloaded something. On queue adapters where lower numbers run first, the workers happily drain `:extract` jobs first, filling the disk with unprocessed artifacts until the volume runs out and everything starts failing.
+
+Bumping the later steps to a higher priority (lower number) fixes this:
+
+```ruby
+class RubygemsWorkflow < GenevaDrive::Workflow
+  step :extract, job_options: {priority: 10} do
+    # Download a large tarball to local disk
+  end
+
+  step :process do  # runs at default priority — ahead of :extract
+    # Do work on the extracted data
+  end
+
+  step :cleanup do  # also runs ahead of :extract
+    # Delete the on-disk artifact
+  end
+end
+```
+
+Workflows that have already extracted their artifact can now drain through `:process` and `:cleanup` — releasing disk — while new `:extract` jobs wait their turn. The fleet reaches steady state instead of collapsing under its own backlog.
+
+The same pattern is useful whenever a later step releases a scarce resource (disk, external quota, a database lock) that earlier steps consume: give the release-work a higher priority than the acquire-work.
 
 ## Housekeeping
 
@@ -1351,6 +1595,18 @@ ActiveSupport::Notifications.subscribe("step.geneva_drive") do |event|
 end
 ```
 
+### Metric gauges
+
+`GenevaDrive::HousekeepingJob` reports gauges via [Measurometer](https://rubygems.org/gems/measurometer) on every run:
+
+| Gauge | Tags | Value |
+|-------|------|-------|
+| `geneva_drive.<state>` | _(none)_ | Absolute count of workflows in `<state>` (e.g. `ready`, `paused`, `finished`), summed across all classes |
+| `geneva_drive.<state>` | `workflow: <ClassName>` | Absolute count of workflows in `<state>` for a single workflow class |
+| `geneva_drive.paused_ratio` | `workflow: <ClassName>` | Float `0.0..1.0` — that class's `paused` count divided by its total population (all states) |
+
+`paused_ratio` is normalized on purpose: absolute paused counts are hard to alert on (50 paused of 50 is a fire; 50 of 500,000 is noise), whereas the ratio is bounded and comparable across classes. The denominator is the whole population (paused ÷ total), so it stays within `0..1` even when nothing is ongoing.
+
 ---
 
 # Part VI — Appendix
@@ -1704,9 +1960,10 @@ end
 |--------|--------|
 | `cancel!` | Stop workflow, mark canceled |
 | `pause!` | Stop workflow, await manual resume |
-| `reattempt!(wait:)` | Retry current step, optionally after delay |
+| `reattempt!(wait:, rewind:)` | Retry current step; `rewind: true` clears cursor |
 | `skip!` | Skip current step, proceed to next |
 | `finished!` | Complete workflow early |
+| `suspend!(wait:)` | Interrupt resumable step, continue via successor after delay |
 
 ### Workflow States
 
@@ -1729,13 +1986,18 @@ end
 | `canceled` | Canceled before execution |
 | `skipped` | Skipped via `skip_if` or `skip!` |
 
+A resumable step interrupted mid-iteration completes its execution with outcome `continued` and schedules a successor execution — there is no separate state for it.
+
 ### Step Options
 
 | Option | Type | Description |
 |--------|------|-------------|
 | `wait:` | Duration | Delay before step executes |
+| `job_options:` | Hash | Options passed to Active Job's `set` method for this step |
 | `skip_if:` | Proc, Symbol, Boolean | Condition to skip step |
 | `on_exception:` | Symbol | Exception handler (`:pause!`, `:cancel!`, `:reattempt!`, `:skip!`) |
 | `max_reattempts:` | Integer, nil | Max consecutive reattempts before pausing (default: 100, `nil` = unlimited) |
 | `before_step:` | Symbol | Insert before this step |
 | `after_step:` | Symbol | Insert after this step |
+| `max_iterations:` | Integer | (resumable_step) Interrupt after N iterations, continue via successor |
+| `max_runtime:` | Duration | (resumable_step) Interrupt after duration elapsed, continue via successor |

data/README.md

@@ -164,12 +164,14 @@ class PremiumWorkflow < BaseWorkflow
   cancel_if { hero.subscription_expired? }
   set_step_job_options queue: :premium, priority: 10
 
-  step :premium_feature do
+  step :premium_feature, job_options: {priority: -1} do
     # Inherits cancel_if from parent
   end
 end
 ```
 
+Step job options override workflow defaults when the step is initially enqueued, reattempted, or resumed. Options are passed directly to Active Job's `set` method.
+
 ### Hero Deletion Handling
 
 By default, workflows cancel if their hero is deleted:

data/lib/generators/geneva_drive/install/install_generator.rb

@@ -46,10 +46,25 @@ module GenevaDrive
           "db/migrate/add_metadata_to_geneva_drive_step_executions.rb"
         )
 
+        migration_template(
+          "add_metadata_to_workflows.rb",
+          "db/migrate/add_metadata_to_geneva_drive_workflows.rb"
+        )
+
         migration_template(
           "allow_null_hero_on_workflows.rb",
           "db/migrate/allow_null_hero_on_geneva_drive_workflows.rb"
         )
+
+        migration_template(
+          "add_started_at_index_to_step_executions.rb",
+          "db/migrate/add_started_at_index_to_geneva_drive_step_executions.rb"
+        )
+
+        migration_template(
+          "add_resumable_step_support.rb",
+          "db/migrate/add_resumable_step_support_to_geneva_drive_step_executions.rb"
+        )
       end
 
       # Creates the initializer file.

data/lib/generators/geneva_drive/install/templates/add_metadata_to_workflows.rb

@@ -0,0 +1,17 @@
+# frozen_string_literal: true
+
+class AddMetadataToGenevaDriveWorkflows < ActiveRecord::Migration[7.2]
+  def change
+    return if column_exists?(:geneva_drive_workflows, :metadata)
+
+    adapter = connection.adapter_name.downcase
+
+    if adapter.include?("postgresql")
+      add_column :geneva_drive_workflows, :metadata, :jsonb
+    elsif adapter.include?("mysql")
+      add_column :geneva_drive_workflows, :metadata, :text, limit: 4_294_967_295
+    else
+      add_column :geneva_drive_workflows, :metadata, :text
+    end
+  end
+end

data/lib/generators/geneva_drive/install/templates/add_resumable_step_support.rb

@@ -0,0 +1,29 @@
+# frozen_string_literal: true
+
+class AddResumableStepSupportToGenevaDriveStepExecutions < ActiveRecord::Migration[7.2]
+  include GenevaDrive::MigrationHelpers
+
+  def change
+    unless column_exists?(:geneva_drive_step_executions, :cursor)
+      # Cursor for resumable steps. Use database-native JSON type:
+      # - PostgreSQL: jsonb (indexed, efficient, supports containment queries)
+      # - MySQL 5.7+: json (native validation and storage)
+      # - SQLite: json (Rails handles as TEXT with serialization)
+      if connection.adapter_name.downcase.include?("postgresql")
+        add_column :geneva_drive_step_executions, :cursor, :jsonb
+      else
+        add_column :geneva_drive_step_executions, :cursor, :json
+      end
+    end
+
+    unless column_exists?(:geneva_drive_step_executions, :continues_from_id)
+      # Link successor executions to their predecessor, chaining the execution
+      # records of a resumable step. Match the primary key type (bigint or uuid)
+      # of the step_executions table.
+      # No foreign key constraint - SQLite rewrites the table on add_foreign_key,
+      # which can destroy data.
+      add_column :geneva_drive_step_executions, :continues_from_id, geneva_drive_key_type
+      add_index :geneva_drive_step_executions, :continues_from_id
+    end
+  end
+end

data/lib/generators/geneva_drive/install/templates/add_started_at_index_to_step_executions.rb

@@ -0,0 +1,48 @@
+# frozen_string_literal: true
+
+class AddStartedAtIndexToGenevaDriveStepExecutions < ActiveRecord::Migration[7.2]
+  # Partial index scoped to state = 'in_progress'. HousekeepingJob's
+  # recover_stuck_in_progress! filters `state = 'in_progress' AND
+  # started_at < cutoff LIMIT N`; without this the planner walks the whole
+  # in-progress set looking for old rows and trips the caller's
+  # statement_timeout on large / mostly-fresh workloads.
+  INDEX_NAME = :index_geneva_drive_step_executions_in_progress_started_at
+
+  disable_ddl_transaction!
+
+  def up
+    return if index_name_exists?(:geneva_drive_step_executions, INDEX_NAME)
+
+    adapter = connection.adapter_name.downcase
+
+    if adapter.include?("postgresql")
+      add_index :geneva_drive_step_executions, :started_at,
+        name: INDEX_NAME,
+        where: "state = 'in_progress'",
+        algorithm: :concurrently
+    elsif adapter.include?("sqlite")
+      # SQLite supports partial indexes but has no CONCURRENTLY.
+      add_index :geneva_drive_step_executions, :started_at,
+        name: INDEX_NAME,
+        where: "state = 'in_progress'"
+    else
+      # MySQL does not support partial indexes; fall back to a plain index.
+      # Composite (state, started_at) so the planner can seek by state first.
+      add_index :geneva_drive_step_executions, [:state, :started_at],
+        name: INDEX_NAME
+    end
+  end
+
+  def down
+    return unless index_name_exists?(:geneva_drive_step_executions, INDEX_NAME)
+
+    adapter = connection.adapter_name.downcase
+    if adapter.include?("postgresql")
+      remove_index :geneva_drive_step_executions,
+        name: INDEX_NAME,
+        algorithm: :concurrently
+    else
+      remove_index :geneva_drive_step_executions, name: INDEX_NAME
+    end
+  end
+end

data/lib/generators/geneva_drive/install/templates/create_workflows_migration.rb

@@ -4,6 +4,8 @@ class CreateGenevaDriveWorkflows < ActiveRecord::Migration[7.2]
   include GenevaDrive::MigrationHelpers
 
   def change
+    adapter = connection.adapter_name.downcase
+
     create_table :geneva_drive_workflows, **geneva_drive_table_options do |t|
       # Core identification (STI)
       t.string :type, null: false, index: true
@@ -22,6 +24,15 @@ class CreateGenevaDriveWorkflows < ActiveRecord::Migration[7.2]
       # Multiple workflows of same type for same hero
       t.boolean :allow_multiple, default: false, null: false
 
+      # Freeform JSON metadata
+      if adapter.include?("postgresql")
+        t.jsonb :metadata
+      elsif adapter.include?("mysql")
+        t.text :metadata, limit: 4_294_967_295
+      else
+        t.text :metadata
+      end
+
       # Timestamps
       t.datetime :started_at
       t.datetime :transitioned_at

data/lib/generators/geneva_drive/install/templates/initializer.rb.tt

@@ -31,6 +31,14 @@
 #
 # GenevaDrive.housekeeping_batch_size = 1000
 
+# Maximum size in bytes of a resumable step cursor once serialized to JSON.
+# The cursor is a position marker (an id, page number, or pagination token),
+# not a place to store the data being processed - it is rewritten on every
+# checkpoint and copied to every successor execution. Exceeding the limit
+# raises GenevaDrive::CursorTooLargeError. Set to nil to disable the check.
+#
+# GenevaDrive.max_cursor_size = 128.kilobytes
+
 # Whether to defer job enqueueing until after the database transaction commits.
 # In production this ensures step execution records are visible to job workers
 # before the job runs. In test environments (the default when Rails.env.test?)

data/lib/geneva_drive/combined_exception_policy.rb

@@ -88,7 +88,7 @@ class GenevaDrive::CombinedExceptionPolicy
     cap = max_reattempts
     if cap && result[:action] == :reattempt && reattempt_count >= cap
       terminal = child.terminal_action.to_s.chomp("!").to_sym
-      return {action: terminal, error: error}
+      return {action: terminal, error: error, report: result[:report], terminal: true}
     end
 
     result