geneva_drive 0.7.0 → 0.9.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 1738e58c5f40199781a27c2345b20657e63ccc0c5a8b28010c2ac820774c2c2f
-  data.tar.gz: f729ef68e929f39fa26b09108f446a9b53ec13013d22d89f22d788b5ac0988e3
+  metadata.gz: b415480aa40bc610a79bdc6802a636d70ac54dd6adddbd3d8b44d8afbb4a1e44
+  data.tar.gz: a7e729983cf52ad726d12ea83661b17da8d34413b03a8f349919e342dbfdac52
 SHA512:
-  metadata.gz: 87a29981a0278755e146cebf6e5e8d7075832665335b54511d5c24734a44040c62e9b2f9ae8fcd2558941d5f8cfe9eefc588efef65db09dc131d6602e2089d40
-  data.tar.gz: 9dcbb2ca84072315438dcacfd78d5463563b0ef6f5a940a5ccd12afcef7d2f10fb40cd710ce4bd5ba72142941242ec709f92e4d33927d15ce146538eed93834f
+  metadata.gz: 1c8c27663ca3f19cb671b0c5c16c62c96487059f823ce5a9dfc35a4366a31d01582ba9ced0fc4140f6fba542e0ede4046b0b5f92518555b1fac575875440fb71
+  data.tar.gz: b5d56f17405a2e032b3ef4f977bd38dc7dc071a7d2160811b996dabe0301bf6275ebb9c0b23aaee4c5cfce86403d8e76ce8b438e716bddd022d02248e9b95b5b

data/CHANGELOG.md

@@ -1,5 +1,26 @@
 # Changelog
 
+## Unreleased
+
+## [0.9.0]
+
+- Run on Sidekiq without Active Job. `GenevaDrive.dispatcher = :sidekiq` pushes every step as `GenevaDrive::SidekiqPerformStepJob`, a plain `Sidekiq::Job` carrying the step execution id, through `Sidekiq::Client`. Client middleware applies, the jid lands in `job_id`, and jobs are tagged with the workflow class. Supported are Sidekiq 7.3 and later 7.x, and 8.x; with any other version the dispatcher raises at boot. The default dispatcher is still `:active_job`.
+- Sidekiq retries are off for step jobs, since exception policies already decide reattempts. The step's exception is still raised, so Sidekiq's error handlers report it.
+- The Sidekiq dispatcher pushes synchronously, bypassing `Sidekiq.transactional_push!`, which would defer the push a second time from inside GenevaDrive's own after-commit callback - the double deferral already fixed for Solid Queue.
+- `priority:` in job options raises under the Sidekiq dispatcher, as Sidekiq has queues with weights rather than priorities.
+- Add `GenevaDrive::SidekiqHousekeepingJob` for running housekeeping from Sidekiq Enterprise periodic jobs or sidekiq-cron.
+- The install generator writes `GenevaDrive.dispatcher = :sidekiq` into a new initializer when Active Job's adapter is `:sidekiq`. Nothing is detected at boot, so upgrading never switches an existing app.
+- `resume!` now records the id of the job it pushes for an execution that was paused while scheduled. The row used to keep the previous job's id, which could already have run and done nothing, so the job behind the execution could not be found from it.
+- Fix resumable steps not handing off when Sidekiq shuts down. The executor asked `Sidekiq::CLI.instance.stopping?`, a method the CLI does not have (only its launcher does), so inside a `sidekiq` process every interruption check of a resumable step raised `NoMethodError`. The engine now subscribes to Sidekiq's `:quiet` event, which fires on TSTP (what Enterprise rolling restarts send first) and at the start of every shutdown, and resumable steps checkpoint and continue in a successor execution. This applies with either dispatcher, including Active Job on the Sidekiq adapter.
+- Create `hero_id` as a string column when the application's tables use string primary keys. Key type detection only recognized `uuid`, `char(36)` and `varchar(36)` id columns, but on SQLite an `id: :string` table (for example with `primary_key_type: :string` and UUIDv7 ids) reports a bare `varchar` with no length. Detection fell through to `:bigint`, the workflows migration created an integer `hero_id`, and every string hero id was silently cast to an integer on assignment, pointing workflows at the wrong hero and colliding in the ongoing-uniqueness index. The workflows migration now uses a separate `geneva_drive_hero_key_type` helper that returns `:uuid`, `:string` or `:bigint` and treats `varchar`/`char(n)`/`text` ids of any length as string keys. GenevaDrive's own tables keep their bigint ids. Existing installs affected by this need to change `hero_id` to a string column themselves.
+
+## [0.8.0]
+
+- Add `removed_step :name` for taking a step out of a workflow that has instances in flight. Deleting a `step` outright strands every workflow holding a scheduled execution for it: the executor finds no definition and pauses with `StepNotDefinedError`, which during a rolling deploy can hit every workflow sitting on that step at once. The gem cannot recover from this on its own — it knows the step is gone, but not whether the steps after it were ever meant to run, and a step's position says nothing about reachability (a `capture_payment` that normally ends the workflow by calling `finished!` sits directly before `refund`, so skipping forward would refund a payment that was never captured). `removed_step` is how the author of the removal records that judgement in advance: declared in the position the step used to occupy, it keeps the name resolvable so executions scheduled before the removal are skipped and the workflow continues to the next runnable step, while scheduling never spools an execution for the name again and `before_step:`/`after_step:` references keep working. An execution parked on the step waiting for a signal is released on `resume!` rather than waiting for an event that can no longer match. `Workflow.removed_steps_in_flight` reports which gravestones step execution rows still reference (ignoring finished and canceled workflows), so a `removed_step` line can be deleted once it comes back empty. The default for an undeclared removal is unchanged: the workflow still pauses.
+- Add `workflow.resume_at!(:step_name)` for workflows already stranded on a step that was deleted without a `removed_step` gravestone. Plain `resume!` had nowhere to send them and died with a `NoMethodError` on nil deep inside `create_step_execution`; it now raises `StepNotDefinedError` naming the missing step, the steps still defined, and the way out. Where to land is deliberately the operator's call, for the same reason the executor will not guess it.
+- Support MariaDB for the JSON columns (resumable step `cursor`, signal `payload`). MariaDB has no JSON type: `json` there is an alias for LONGTEXT with an auto-generated CHECK constraint named after the column, and since `cursor` is reserved in MariaDB, loading a schema dump that re-emits that constraint died on a syntax error. Rails also types such a column as text, so a Hash written to it was stored as `#inspect` output, which the same CHECK constraint then rejected. The installer migrations now ask for plain LONGTEXT on MariaDB via the new `geneva_drive_json_column` migration helper, and the models encode and decode JSON themselves whenever the column is text. The check is on the column, not the adapter, so installs that already ran the old migration on MariaDB get the encoding without having to rewrite the column.
+- Test against Rails 7.2, 8.0 and 8.1 on SQLite, PostgreSQL, MySQL and MariaDB. Previously the appraisals stopped at Rails 8.0, and the MySQL CI job could not build `mysql2`.
+
 ## [0.7.0]
 
 - Add signals: external events delivered to a workflow instance with `workflow.signal!(:payment_confirmed, payload: {...}, idempotency_key: event["id"])`, and a step primitive that parks until one arrives with `step :capture, wait_for: :payment_confirmed`. Signals are rows in a new `geneva_drive_signals` table, persisted before any processing, so arrival order does not matter: a signal that lands before its waiting step exists is buffered and claimed when the step runs, and one that lands after the step parked wakes it immediately. Both sides of the rendezvous run under the workflow row lock, so no signal can slip through the gap, and delivery is one transaction: the insert, the waking of parked executions and every counter bump land together or not at all, so a crashed `signal!` leaves nothing half-delivered (the job enqueue stays after-commit, with the existing stuck-scheduled housekeeping sweep as its backstop). Waiting is a first-class step execution state (`waiting`) that holds no queue slot and no timer — nothing polls. Matching is by name - `wait_for:` also takes a `GenevaDrive::SignalMatcher`, whose optional block narrows by payload and is evaluated on the workflow the signal was sent to (so `hero` is in scope), or any other object responding to `#matches?(signal)`. Matchers are plain reusable objects, so a rule can live in a constant and be shared across steps. Signals move `pending` → `claimed` → `consumed`, consumed only once every attached execution chain has finished cleanly — where skipping counts as finishing, so any flavor of skip (`skip_if:` at wake, `skip!` in a step, an exception policy, or an operator `skip!` on a ready or paused workflow) settles the signal that step was delivered, while skips of steps that never attached leave buffered signals alone — and in the same transaction as the step's own completion, so a reattempt or a `resume!` re-reads the same payload and the audit trail never claims an event was handled when it was not. Two counters on the row, `claimed` and `consumed`, count attachments and clean resolutions (with `claimed?` / `consumed?` reading those counters rather than the lifecycle state). Redelivery is deduplicated by a unique index on `(workflow_id, name, idempotency_key)` — the duplicate call returns the original row flagged `duplicate_delivery?` without dispatching. Payloads go through ActiveJob serializers, come back with indifferent access, and are bounded by `GenevaDrive.max_signal_payload_size` (128 KB, `nil` disables). `signal!` on a finished or canceled workflow raises `GenevaDrive::WorkflowNotOngoing` unless it is a redelivery of a known event; on a paused workflow the row is buffered and delivered by `resume!`. Timeouts are deliberately out of scope for now; waiting is instead made loud through `waiting_since` and two housekeeping gauges (`geneva_drive.waiting_step_executions`, `geneva_drive.waiting_overdue` past `GenevaDrive.waiting_visibility_threshold`, 7 days by default), with `skip!` and `cancel!` as operator escape hatches. Needs one new installer migration (the signals table plus two columns on step executions); until it runs everything else is unaffected and executing a `wait_for:` step raises a configuration error pointing at the missing migration. Test helpers: `assert_waiting_for_signal`, `assert_signal_state`, and the step-driving helpers now raise instead of spinning on a parked execution. Mixing up `wait:` and `wait_for:` now raises `StepConfigurationError` at class load in both directions, which also tightens `wait:` to reject Strings (previously `wait: "later"` silently meant zero seconds).

data/MANUAL.md

@@ -85,6 +85,8 @@ If your application uses UUID primary keys, the migrations will detect this and
 
 Note that you don't want to mix integer IDs and UUIDs in the same application. geneva_drive uses a polymorphic relation for the `hero` of the workflow, which has a typed `hero_id` column. You will thus want either all of your models to be used as heroes to have UUID primary keys or bigint primary keys, but not mix the two.
 
+String primary keys (`id: :string`, for example with `primary_key_type: :string` in your generator config) are detected too: `hero_id` is then created as a string column, while the geneva_drive tables themselves keep bigint ids. If detection picks the wrong type for your heroes, edit the `hero_id` column in the generated workflows migration before running it.
+
 > [!WARNING]
 > If you are using UUIDs, we also strongly recommend adopting a lexicographically ordered UUID flavour, at least for the geneva_drive tables. Such flavours include [UUIDv7](https://github.com/seouri/rails-uuid-pk) and [tou](https://github.com/cheddar-me/tou) - you will want your primary keys to be pre-sorted for using the admin effectively, as well as for efficient `find_each` usage with the geneva_drive tables.
 
@@ -325,6 +327,80 @@ end
 
 The referenced step must already be defined — you can only insert before or after steps that appear earlier in the class body.
 
+### Removing a Step with `removed_step`
+
+Deleting a `step` is not safe while workflows are in flight. Step executions are rows scheduled ahead of time, so at the moment a deploy lands there are workflows holding a row that names a step the new code no longer has. When such a row's job runs, the executor finds no definition and pauses the workflow with `StepNotDefinedError`. During a rolling deploy this can happen to every workflow sitting on that step at once.
+
+GenevaDrive cannot recover from this by itself, and the reason is worth being precise about. It knows the step is gone. It does not know whether the work that step did still needs doing, and — more importantly — it does not know which of the remaining steps would have run had it executed. A step's position in the sequence says nothing about reachability: a `capture_payment` that normally ends the workflow by calling `finished!` sits directly before `refund`, and nothing in the database distinguishes "carry on to the next one" from "stop here". Skipping forward on the workflow's behalf would, in exactly that case, issue a refund for a payment that was never captured. So the executor pauses and waits for a human, which is the only correct thing it can do on its own.
+
+`removed_step` is how you tell it the answer in advance. Declare it in the position the step used to occupy:
+
+```ruby
+class PaymentWorkflow < GenevaDrive::Workflow
+  step :authorize do
+    PaymentGateway.authorize(hero)
+  end
+
+  removed_step :capture_payment
+
+  step :send_receipt do
+    ReceiptMailer.receipt(hero).deliver_later
+  end
+end
+```
+
+The declaration is an assertion only you can make: *this step no longer needs to happen, carry on past it.* Having been told, GenevaDrive can act without guessing:
+
+- Executions already scheduled under the old name resolve, are marked `skipped`, and the workflow proceeds to the next runnable step.
+- Scheduling never spools a new execution for the name again — a new `PaymentWorkflow` goes `authorize` → `send_receipt`.
+- `before_step:` / `after_step:` references to the name keep working.
+- An execution parked on the step waiting for a signal is released on `resume!` instead of waiting for an event that can no longer match.
+
+A removed step takes a name and nothing else. There is nothing to configure about a step that does not run.
+
+Because the gravestone is in the source, in order, the decision is visible in the diff that removes the step — which is where a reviewer can still object to it.
+
+### Retiring the Gravestone
+
+A `removed_step` line is temporary. It can go once no step execution row can land on it any more, which is a question you can ask directly:
+
+```ruby
+PaymentWorkflow.removed_steps_in_flight
+# => {"capture_payment" => 3}   # three rows still point at it — leave the line in place
+# => {}                          # drained — the removed_step line can be deleted
+```
+
+Rows belonging to finished and canceled workflows are ignored, since they will never execute again. The full lifecycle is: replace `step` with `removed_step` and deploy; let in-flight workflows drain past it; delete the line in a later deploy once `removed_steps_in_flight` comes back empty.
+
+### Recovering a Workflow Stranded by a Deleted Step
+
+If a step was deleted without leaving a gravestone, the workflows that were sitting on it are paused and pointed at a name that no longer resolves. Plain `resume!` has nowhere to send them and says so:
+
+```ruby
+workflow.resume!
+# GenevaDrive::StepNotDefinedError: Cannot resume PaymentWorkflow #42: step 'capture_payment'
+# is no longer defined in PaymentWorkflow. Pick the step to continue from with
+# resume_at!(:step_name) — the steps still defined are authorize, send_receipt — or cancel!
+# the workflow. ...
+```
+
+`resume_at!` is the escape hatch. You name the step to continue from, because only you know what the deleted step would have done:
+
+```ruby
+workflow.resume_at!(:send_receipt)
+```
+
+It cancels any stray execution, points the workflow at the named step, and continues from there. It refuses names that are not defined, and refuses to land on a `removed_step`. To find the affected workflows:
+
+```ruby
+GenevaDrive::Workflow.paused
+  .joins(:step_executions)
+  .where(geneva_drive_step_executions: {error_class_name: "GenevaDrive::StepNotDefinedError"})
+  .distinct
+```
+
+If the workflow should not continue at all, `cancel!` it.
+
 ## Conditional Execution
 
 ### Skipping Steps with `skip_if:`
@@ -626,7 +702,7 @@ A step is interrupted when any of these happen:
 
 - `max_iterations:` is reached (`resumable_step :import, max_iterations: 10_000`)
 - `max_runtime:` is exceeded (`resumable_step :import, max_runtime: 5.minutes`)
-- The job queue signals shutdown (e.g. Sidekiq stopping)
+- The job queue signals shutdown (Sidekiq going quiet or stopping — see [Shutdowns and rolling restarts](#shutdowns-and-rolling-restarts))
 - The workflow is paused or canceled externally
 - The step calls `skip_to!` or `suspend!` explicitly
 
@@ -1227,8 +1303,13 @@ end
 # Resume a specific workflow
 workflow = GenevaDrive::Workflow.find(id)
 workflow.resume!  # Re-enqueues existing scheduled step or creates new one
+
+# Or continue from a specific step, when the step it was pointed at is gone
+workflow.resume_at!(:send_receipt)
 ```
 
+`resume_at!` exists for workflows paused on a step that no longer exists in the code. See [Recovering a Workflow Stranded by a Deleted Step](#recovering-a-workflow-stranded-by-a-deleted-step).
+
 ## Workflow States
 
 ### State Machine Diagram
@@ -1504,7 +1585,7 @@ We recommend using a queue adapter that co-commits with your database:
 
 Co-committing matters because GenevaDrive relies on transactional guarantees. When a step completes and schedules the next step, both the state change and the job enqueue should be atomic. With co-committing adapters, if the transaction rolls back, the job is never enqueued.
 
-With non-transactional adapters (Sidekiq, Resque), there's a small window where the job is enqueued but the transaction hasn't committed. GenevaDrive handles this gracefully — the job will see the step execution in the wrong state and skip it — but you may see occasional log warnings.
+With a non-transactional adapter such as Resque, or Sidekiq behind Active Job, the job and the database are two systems, and the step execution row is what makes that safe: GenevaDrive enqueues only after the row has committed, and housekeeping re-enqueues any execution whose job got lost on the way. If your app writes Sidekiq jobs rather than Active Job ones, you do not have to go through Active Job at all — see [Running on Sidekiq](#running-on-sidekiq).
 
 ### Inline Enqueueing for Bulk Operations
 
@@ -1542,7 +1623,7 @@ The method is thread-safe and won't affect other concurrent requests.
 
 ### Custom Job Options
 
-You can override the Active Job settings used to enqueue a workflow's step jobs at three levels: for the entire workflow class, for an individual step, or for a specific workflow instance. All three accept the same keys that Active Job's `set` method understands: `:queue`, `:priority`, `:wait`, and `:wait_until`. Unknown keys are rejected at configuration time so that typos surface immediately instead of silently falling back to defaults.
+You can override the Active Job settings used to enqueue a workflow's step jobs at three levels: for the entire workflow class, for an individual step, or for a specific workflow instance. All three accept the same keys that Active Job's `set` method understands: `:queue`, `:priority`, `:wait`, and `:wait_until`. (On the Sidekiq dispatcher `:priority` is rejected, since Sidekiq has none — see [Queues instead of priorities](#queues-instead-of-priorities).) Unknown keys are rejected at configuration time so that typos surface immediately instead of silently falling back to defaults.
 
 ```ruby
 class HighPriorityWorkflow < GenevaDrive::Workflow
@@ -1554,7 +1635,7 @@ class HighPriorityWorkflow < GenevaDrive::Workflow
 end
 ```
 
-Options merge from lowest to highest precedence: **class defaults** (`set_step_job_options`) are overridden by **per-instance options** (`workflow.step_job_options = {...}`), which are in turn overridden by **per-step options** (`step ..., job_options: {...}`). The merged hash is passed straight to `PerformStepJob.set(...)` for every enqueue of that step, including reattempts and resume re-enqueueing.
+Options merge from lowest to highest precedence: **class defaults** (`set_step_job_options`) are overridden by **per-instance options** (`workflow.step_job_options = {...}`), which are in turn overridden by **per-step options** (`step ..., job_options: {...}`). The merged hash is handed to the dispatcher — with the default one, straight to `PerformStepJob.set(...)` — for every enqueue of that step, including reattempts and resume re-enqueueing.
 
 Per-instance options are stored in the workflow's `metadata` column, so they survive across step boundaries and are excluded from the dedupe uniqueness check — two workflows for the same hero cannot coexist just because their job options differ.
 
@@ -1602,6 +1683,197 @@ Workflows that have already extracted their artifact can now drain through `:pro
 
 The same pattern is useful whenever a later step releases a scarce resource (disk, external quota, a database lock) that earlier steps consume: give the release-work a higher priority than the acquire-work.
 
+## Running on Sidekiq
+
+Plenty of Rails apps never adopted Active Job. They have a `app/sidekiq` directory full of `include Sidekiq::Job` classes, a Sidekiq Pro or Enterprise license, a Web UI the on-call engineer knows by heart, and a set of client middleware that encrypts arguments or deduplicates pushes. For those apps, "steps run through Active Job" means a second job system bolted onto the first: another serialization format, another set of retry semantics, and jobs in the Busy page called `Sidekiq::ActiveJob::Wrapper` instead of anything you recognise.
+
+GenevaDrive does not actually need much from a job system. The step execution row is the durable record of what has to happen next; the job is only a wake-up call that carries the row's id. So we ship a dispatcher that pushes that wake-up call to Sidekiq directly:
+
+```ruby
+# config/initializers/geneva_drive.rb
+GenevaDrive.dispatcher = :sidekiq
+```
+
+That is the whole switch. Every step of every workflow is now pushed as a `GenevaDrive::SidekiqPerformStepJob` — a plain `Sidekiq::Job` — through `Sidekiq::Client`, with no Active Job in between. Set it in an initializer, so that it is in place before any workflow class loads: job options are checked against the dispatcher when a workflow class is defined.
+
+We support **Sidekiq 7.3 and later 7.x, and Sidekiq 8.x** — OSS, Pro and Enterprise alike — and test the oldest supported Sidekiq on the oldest supported Rails and the newest on the newest, against a real Redis. Sidekiq 9 is not supported until we have tested it. With any other version loaded, `GenevaDrive.dispatcher = :sidekiq` raises at boot, naming the version it found.
+
+#### What the installer picks
+
+When `bin/rails generate geneva_drive:install` writes `config/initializers/geneva_drive.rb`, it checks Active Job's adapter once. If the adapter is `:sidekiq` and a supported Sidekiq is loaded, the initializer gets an uncommented `GenevaDrive.dispatcher = :sidekiq`. Otherwise the line stays commented out. An unset adapter does not count: the generator runs in development, and an app that leaves development on `:async` may well run Solid Queue in production.
+
+Nothing is detected at boot. Without that line the dispatcher is `:active_job`, so upgrading the gem never switches an app that is already running workflows.
+
+### What the jobs look like
+
+A step job carries one argument, the step execution id, which is a plain integer (or string, with UUID keys) and therefore passes Sidekiq's strict argument checks. It is tagged `geneva_drive` plus the workflow class name, and the Web UI shows those tags next to each job:
+
+```ruby
+{
+  "class" => "GenevaDrive::SidekiqPerformStepJob",
+  "args" => [48213],
+  "queue" => "default",
+  "retry" => false,
+  "tags" => ["geneva_drive", "SubscriptionRenewalWorkflow"]
+}
+```
+
+While a step runs, the workflow id, step name and step execution id are added to `Sidekiq::Context`, so Sidekiq's own log lines for that job carry them. The jid is written to the step execution's `job_id`, the same as the Active Job id is with the default dispatcher, so you can go from a row in the database to the job in the Web UI and back.
+
+A step with `wait:` is pushed with an `at` timestamp and waits in Sidekiq's Scheduled set; a step that is due now goes straight onto its queue.
+
+### Retries belong to the step, not the job
+
+Sidekiq's default is to retry a failing job 25 times over three weeks. The step job switches that off with `retry: false`, and this is deliberate.
+
+GenevaDrive already decides what happens when a step raises: that is what [exception policies](#exception-handling) are for. A policy might reattempt the step in five minutes, pause the workflow for a human, or cancel it. Every one of those outcomes is committed to the step execution row before the job finishes. A reattempt is a *new* step execution with a job of its own. If Sidekiq also retried the original job, the retry would find the execution already settled and do nothing — so all a Sidekiq retry would add is ghost entries in the Retry set, and a Web UI that disagrees with the workflow about what failed.
+
+The exception is still raised out of the job once the executor has committed the step's state, so Sidekiq's error handlers — Sentry, Honeybadger, AppSignal, whatever you have configured — report it exactly as for any other failing job. Because the job is not retried, Sidekiq's death handlers see it too. Treat that as "this job is done", not "this workflow is dead": the step execution row is the ledger, and the workflow's state is the answer to "is this broken?".
+
+### Why we bypass transactional push
+
+If your app calls `Sidekiq.transactional_push!`, every `perform_async` is deferred until the surrounding Active Record transaction commits. That is a good default for application jobs, and the dispatcher deliberately does not use it.
+
+GenevaDrive already defers its push: it hands the step to the dispatcher from an `after_all_transactions_commit` callback, once the step execution row is visible to other connections. That callback runs after the SQL `COMMIT`, but possibly while the transaction is still unwinding on the Ruby side. A second deferral registered from inside it can be dropped on the floor — the step execution then has no job and sits in `scheduled` until housekeeping notices, 15 minutes later by default. We hit exactly this with Solid Queue (it is described at length in `PerformStepJob`), and the remedy is the same here: since we are already past the commit, push synchronously. The dispatcher builds its own `Sidekiq::Client` for that.
+
+Client middleware still runs on that push, so argument encryption, Enterprise unique jobs, and your own middleware all see GenevaDrive's jobs like any others. `Sidekiq::Client.via` sharding is honoured as well.
+
+### Queues instead of priorities
+
+Sidekiq has no job priority. It has queues, and a process works its queues either in strict order or by weight. So under the Sidekiq dispatcher `job_options: {priority: ...}` raises `StepConfigurationError` when the workflow class loads, rather than being silently ignored. `queue:`, `wait:` and `wait_until:` work as usual.
+
+The backlog problem from [Custom Job Options](#custom-job-options) — 50,000 fresh `:extract` steps crowding out the `:process` and `:cleanup` steps that free up disk — is solved with a queue that is worked first:
+
+```ruby
+class RubygemsWorkflow < GenevaDrive::Workflow
+  step :extract do
+    # Download a large tarball to local disk
+  end
+
+  step :process, job_options: {queue: :workflows_drain} do
+    # Do work on the extracted data
+  end
+
+  step :cleanup, job_options: {queue: :workflows_drain} do
+    # Delete the on-disk artifact
+  end
+end
+```
+
+```yaml
+# config/sidekiq.yml
+:queues:
+  - [workflows_drain, 5]
+  - [default, 1]
+```
+
+With weights, the drain queue is picked five times as often as `default`; list the queues without weights and Sidekiq works them strictly in order, so `default` only gets attention once the drain queue is empty.
+
+### Your own job class
+
+To carry options of your own — a dedicated queue, extra tags, Enterprise's `unique_for`, Pro's `expires_in` — subclass the step job and hand the subclass to the dispatcher:
+
+```ruby
+# app/sidekiq/workflow_step_job.rb
+class WorkflowStepJob < GenevaDrive::SidekiqPerformStepJob
+  sidekiq_options queue: "workflows", tags: ["geneva_drive", "billing"]
+end
+
+# config/initializers/geneva_drive.rb
+Rails.application.config.to_prepare do
+  GenevaDrive.dispatcher = GenevaDrive::SidekiqDispatcher.new(job_class: WorkflowStepJob)
+end
+```
+
+The `to_prepare` block is there because `WorkflowStepJob` lives in `app/`, and Rails does not let an initializer autoload it. `to_prepare` runs after the initializers but before eager loading, so workflow classes still see the dispatcher when they are defined. In development it also runs again on every code reload, so the dispatcher picks up the reloaded class. The plain `GenevaDrive.dispatcher = :sidekiq` needs none of this, as it refers to nothing in `app/`.
+
+Keep `retry: false` on the subclass for the reasons above. A queue set through `job_options:` on a workflow or step still wins over the class's queue.
+
+### Shutdowns and rolling restarts
+
+When Sidekiq is told to stop — or to go quiet, which is what Enterprise rolling restarts and most deploy scripts do first — it fires its `:quiet` event. GenevaDrive's engine subscribes to it whichever dispatcher you use, including Active Job on the Sidekiq adapter. From that moment every [resumable step](#resumable-steps) running in the process checkpoints its cursor at the end of the current iteration and hands off to a successor execution, which the next process picks up from where this one stopped.
+
+That only helps if an iteration is short enough to finish before Sidekiq gives up waiting, which is 25 seconds by default (`-t`). So the rule of thumb is: a step must either finish well within the shutdown timeout, or be a `resumable_step` whose single iteration does. A plain step that is still running when the timeout expires is killed; Sidekiq puts its job back on the queue (Sidekiq Pro's `super_fetch` does this even after a hard crash), but the step execution stays `in_progress`. When the re-delivered job arrives, the executor sees an execution that is no longer `scheduled` and skips it as a duplicate. Housekeeping then recovers the execution once it has been `in_progress` for longer than `stuck_in_progress_threshold`.
+
+Sidekiq 7.3 has iterable jobs of its own, and the two are worth comparing if you are deciding where a long iteration should live:
+
+| | `Sidekiq::IterableJob` | GenevaDrive `resumable_step` |
+|---|---|---|
+| Where the cursor lives | Redis, flushed about every 5 seconds and on error | the step execution row, written by `checkpoint!` or on every iteration |
+| After a crash | resumes from the last Redis flush | resumes from the last checkpoint, in the database your work writes to |
+| On shutdown | raises `Interrupted` and re-enqueues the same job | completes the execution as `continued` and schedules a successor |
+| What you can see | the job in the Web UI | every chunk as its own step execution, with the cursor it stopped at |
+
+### Pro and Enterprise features
+
+Several Sidekiq features fit workflows naturally.
+
+**Batches (Pro)** fan work out; [signals](#waiting-for-signals) bring the result back into a workflow. Park a step on a signal, run the batch, and let the batch's success callback deliver it:
+
+```ruby
+class InvoiceRunWorkflow < GenevaDrive::Workflow
+  step :render_invoices do
+    batch = Sidekiq::Batch.new
+    batch.description = "Invoices for #{hero.billing_period}"
+    batch.on(:success, InvoiceRunCallbacks, "workflow_id" => id)
+    batch.jobs do
+      hero.accounts.find_each { |account| RenderInvoiceJob.perform_async(account.id, hero.id) }
+    end
+  end
+
+  step :send_invoices, wait_for: :invoices_rendered do
+    InvoiceMailer.batch_ready(hero).deliver_later
+  end
+end
+
+class InvoiceRunCallbacks
+  def on_success(status, options)
+    workflow = GenevaDrive::Workflow.find(options["workflow_id"])
+    workflow.signal!(:invoices_rendered, idempotency_key: status.bid)
+  end
+end
+```
+
+The batch id makes the signal idempotent, so a callback Sidekiq delivers twice is harmless.
+
+**Rate limiters (Enterprise)** raise `Sidekiq::Limiter::OverLimit` when the limit is hit. Inside a step, turn that into a reattempt instead of a failure:
+
+```ruby
+class CrmSyncWorkflow < GenevaDrive::Workflow
+  CRM_LIMIT = Sidekiq::Limiter.window("crm-api", 100, :minute, wait_timeout: 0)
+
+  on_exception Sidekiq::Limiter::OverLimit, action: :reattempt!, wait: 1.minute, report: :never
+
+  step :push_contact do
+    CRM_LIMIT.within_limit { CrmClient.upsert_contact(hero) }
+  end
+end
+```
+
+**Periodic jobs (Enterprise)** and **sidekiq-cron** can schedule housekeeping through `GenevaDrive::SidekiqHousekeepingJob` — see [Running Housekeeping](#running-housekeeping).
+
+### Testing
+
+The [test helpers](#test-helpers) drive the executor directly, so `speedrun_workflow` and friends work the same with either dispatcher. To exercise the real push path, use Sidekiq's fake mode and drain the step job:
+
+```ruby
+test "renewal runs end to end through Sidekiq" do
+  workflow = SubscriptionRenewalWorkflow.create!(hero: subscriptions(:annual))
+
+  GenevaDrive::SidekiqPerformStepJob.drain
+
+  assert workflow.reload.finished?
+end
+```
+
+The fake queue ignores `at`, so steps with `wait:` run as soon as you drain.
+
+### Switching an existing app over
+
+Upgrading GenevaDrive leaves your dispatcher alone. When you re-run the installer to pick up new migrations, it asks before overwriting the initializer you already have - answer no, or pass `--skip`. To switch, add `GenevaDrive.dispatcher = :sidekiq` yourself. Tests that assert on enqueued `PerformStepJob`s will then need to look for `GenevaDrive::SidekiqPerformStepJob` in Sidekiq's fake mode instead.
+
+Changing the dispatcher only affects jobs pushed from then on. `PerformStepJob`s already queued through Active Job keep running on whatever adapter delivered them, and new steps go out as native Sidekiq jobs. Deploy the change, let the Active Job queue drain, and that is the migration.
+
 ## Housekeeping
 
 ### The Housekeeping Job
@@ -1643,6 +1915,24 @@ GoodJob::Cron.schedule(
 GenevaDrive::HousekeepingJob.perform_later
 ```
 
+On Sidekiq without Active Job, schedule `GenevaDrive::SidekiqHousekeepingJob` instead. It runs the same code in-process:
+
+```ruby
+# Sidekiq Enterprise periodic jobs
+Sidekiq.configure_server do |config|
+  config.periodic do |mgr|
+    mgr.register("*/30 * * * *", "GenevaDrive::SidekiqHousekeepingJob")
+  end
+end
+
+# sidekiq-cron
+Sidekiq::Cron::Job.create(
+  name: "GenevaDrive housekeeping",
+  cron: "*/30 * * * *",
+  class: "GenevaDrive::SidekiqHousekeepingJob"
+)
+```
+
 ## Testing
 
 ### Test Helpers
@@ -1957,7 +2247,11 @@ The workflow will sit in `paused` state — holding the slot, preventing duplica
 
 ### Pause on missing step definitions
 
-There is one more scenario where pause happens automatically. If a workflow references a step name that no longer exists in the class definition — typically after a deploy that removed or renamed a step — the executor pauses the workflow rather than silently skipping or crashing. This gives operators a chance to notice the mismatch and decide whether to skip, cancel, or deploy a fix.
+There is one more scenario where pause happens automatically. If a workflow references a step name that no longer exists in the class definition — typically after a deploy that removed or renamed a step — the executor pauses the workflow rather than silently skipping or crashing.
+
+This is not a limitation to be worked around. The executor knows the step is gone but not whether the steps after it were ever meant to run, and guessing wrong means performing work the missing step would have prevented. Pausing hands that judgement to someone who has it.
+
+Declare `removed_step :name` when you take a step out and the judgement is recorded ahead of time, so nothing pauses; see [Removing a Step with `removed_step`](#removing-a-step-with-removed_step). For workflows already stranded by a step deleted without one, `resume_at!` is the way back — see [Recovering a Workflow Stranded by a Deleted Step](#recovering-a-workflow-stranded-by-a-deleted-step).
 
 ## Complete Example Workflows
 

data/README.md

@@ -41,6 +41,19 @@ geneva_drive_housekeeping:
   class: "GenevaDrive::HousekeepingJob"
 ```
 
+### Sidekiq without Active Job
+
+If your app writes `Sidekiq::Job` classes rather than Active Job ones, steps can be pushed to Sidekiq directly:
+
+```ruby
+# config/initializers/geneva_drive.rb
+GenevaDrive.dispatcher = :sidekiq
+```
+
+The install generator writes this line for you when Active Job is already set up for Sidekiq. Supported: Sidekiq 7.3 and later 7.x, and 8.x, including Pro and Enterprise.
+
+Step jobs then run as `GenevaDrive::SidekiqPerformStepJob`, a plain Sidekiq job with Sidekiq retries switched off (exception policies decide reattempts), and housekeeping can be scheduled as `GenevaDrive::SidekiqHousekeepingJob`. See "Running on Sidekiq" in the manual for the details, including rolling restarts, Batches and rate limiters.
+
 
 
 ## Usage
@@ -172,7 +185,7 @@ class PremiumWorkflow < BaseWorkflow
 end
 ```
 
-Step job options override workflow defaults when the step is initially enqueued, reattempted, or resumed. Options are passed directly to Active Job's `set` method.
+Step job options override workflow defaults when the step is initially enqueued, reattempted, or resumed. With the default dispatcher, options are passed directly to Active Job's `set` method.
 
 ### Hero Deletion Handling
 

data/Rakefile

@@ -1,7 +1,6 @@
 require "bundler/setup"
 require "bundler/gem_tasks"
 
-require "rake/testtask"
 require "standard/rake"
 require "yard"
 
@@ -15,3 +14,31 @@ end
 
 APP_RAKEFILE = File.expand_path("test/dummy/Rakefile", __dir__)
 load "rails/tasks/engine.rake"
+
+# Rails' own plugin test runner rather than a Rake::TestTask: it is what sets up
+# and tears down the per-worker databases that ActiveSupport::TestCase.parallelize
+# creates, and a plain Rake::TestTask leaves those workers hanging at exit.
+desc "Run the test suite"
+task :test do
+  ruby "-Itest", "bin/test"
+end
+
+# The Sidekiq dispatcher's tests, and the installer that recommends it. Sidekiq
+# is only bundled by the sidekiq_* appraisals, so run this under one of them:
+#
+#   BUNDLE_GEMFILE=gemfiles/sidekiq_8.gemfile bundle exec rake test:sidekiq
+#
+# Set GENEVA_DRIVE_SIDEKIQ_REDIS_URL to include the tests against a real Redis.
+namespace :test do
+  desc "Run the Sidekiq tests (needs a sidekiq_* appraisal)"
+  task :sidekiq do
+    abort "Sidekiq is not in this bundle - use BUNDLE_GEMFILE=gemfiles/sidekiq_8.gemfile" unless Gem.loaded_specs["sidekiq"]
+    ruby "-Itest", "bin/test", "test/sidekiq", "test/generators"
+  end
+end
+
+# The root Gemfile resolves to the newest released Rails and the dummy app
+# defaults to SQLite, so a plain `rake` is the "latest Rails on SQLite" run.
+# The appraisal gemfiles cover the older Rails minors, and CI additionally
+# points DATABASE_URL at PostgreSQL and MySQL.
+task default: [:standard, :test]

data/bin/matrix

@@ -0,0 +1,76 @@
+#!/usr/bin/env bash
+# Local CI: run the suite for every Rails appraisal against every database
+# engine the CI matrix covers, so that a one-engine failure can be reproduced
+# without pushing. Brings compose.yaml up first and leaves it running.
+#
+#   bin/matrix                    # the whole matrix
+#   bin/matrix rails_8_1          # one Rails version, all engines
+#   bin/matrix rails_8_1 sqlite   # one cell
+set -uo pipefail
+cd "$(dirname "$0")/.."
+
+# The gemfiles are generated from Appraisals, which makes them the one place
+# the supported Rails versions are written down. CI reads the same list off
+# them, so neither side can drift from the other.
+RAILS_VERSIONS=${1:-$(ls gemfiles/rails_*.gemfile | xargs -n1 basename | sed 's/\.gemfile$//' | tr '\n' ' ')}
+ENGINES=${2:-"sqlite postgres mysql mariadb"}
+
+for engine in $ENGINES; do
+  case "$engine" in
+    sqlite|postgres|mysql|mariadb) ;;
+    *) echo "Unknown engine: $engine (pick from sqlite, postgres, mysql, mariadb)" >&2; exit 1 ;;
+  esac
+done
+
+# Only the engines actually asked for - CI runs one cell per job and should not
+# be pulling a MySQL image to test against PostgreSQL.
+services=""
+for engine in $ENGINES; do
+  case "$engine" in
+    postgres|mysql|mariadb) services="$services $engine" ;;
+  esac
+done
+if [ -n "$services" ]; then
+  docker compose up -d --wait $services || exit 1
+fi
+
+# Compose publishes to an ephemeral host port, so ask it which one it got.
+port_of() {
+  docker compose port "$1" "$2" | sed 's/.*://'
+}
+
+# SQLite gets no DATABASE_URL, exactly as in CI: the dummy app defaults to it
+# and resolves its own path relative to Rails.root.
+url_for() {
+  case "$1" in
+    postgres) echo "postgresql://postgres:postgres@127.0.0.1:$(port_of postgres 5432)/geneva_drive_test" ;;
+    mysql)    echo "mysql2://root:root@127.0.0.1:$(port_of mysql 3306)/geneva_drive_test?encoding=utf8mb4" ;;
+    mariadb)  echo "mysql2://root:root@127.0.0.1:$(port_of mariadb 3306)/geneva_drive_test?encoding=utf8mb4" ;;
+  esac
+}
+
+failed=0
+results=()
+for rails in $RAILS_VERSIONS; do
+  gemfile="gemfiles/$rails.gemfile"
+  [ -f "$gemfile" ] || { echo "No such appraisal: $gemfile" >&2; exit 1; }
+  BUNDLE_GEMFILE=$gemfile bundle check >/dev/null 2>&1 || BUNDLE_GEMFILE=$gemfile bundle install || exit 1
+  for engine in $ENGINES; do
+    echo "==> $rails / $engine"
+    if [ "$engine" = sqlite ]; then
+      env -u DATABASE_URL BUNDLE_GEMFILE=$gemfile RAILS_ENV=test GENEVA_DRIVE_ENGINE=$engine bundle exec rake test
+    else
+      env BUNDLE_GEMFILE=$gemfile RAILS_ENV=test GENEVA_DRIVE_ENGINE=$engine DATABASE_URL="$(url_for "$engine")" bundle exec rake test
+    fi
+    if [ $? -eq 0 ]; then
+      results+=("ok    $rails / $engine")
+    else
+      results+=("FAIL  $rails / $engine")
+      failed=1
+    fi
+  done
+done
+
+printf '\n'
+printf '%s\n' "${results[@]}"
+exit $failed