riverqueue 0.11.0 → 0.12.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (83) hide show
  1. checksums.yaml +4 -4
  2. data/CHANGELOG.md +132 -0
  3. data/LICENSE +374 -0
  4. data/docs/README.md +765 -0
  5. data/docs/migrations.md +121 -0
  6. data/docs/testing.md +173 -0
  7. data/docs/workers.md +126 -0
  8. data/exe/river +7 -0
  9. data/lib/cli.rb +55 -0
  10. data/lib/client.rb +314 -54
  11. data/lib/client_runtime.rb +612 -0
  12. data/lib/config.rb +154 -0
  13. data/lib/driver/job_row_decoder.rb +85 -0
  14. data/lib/driver/postgres_capabilities.rb +41 -0
  15. data/lib/driver/runtime.rb +661 -0
  16. data/lib/driver.rb +13 -6
  17. data/lib/errors.rb +83 -0
  18. data/lib/event.rb +78 -0
  19. data/lib/insert_opts.rb +40 -11
  20. data/lib/job.rb +28 -13
  21. data/lib/job_persisted_logging/plugin.rb +108 -0
  22. data/lib/migration_cli.rb +90 -0
  23. data/lib/migrator.rb +189 -0
  24. data/lib/params.rb +96 -0
  25. data/lib/periodic_cron.rb +24 -0
  26. data/lib/periodic_job.rb +135 -0
  27. data/lib/resumable.rb +148 -0
  28. data/lib/riverqueue/testing/minitest.rb +18 -0
  29. data/lib/riverqueue/testing/rspec.rb +109 -0
  30. data/lib/riverqueue/testing.rb +134 -0
  31. data/lib/riverqueue.rb +16 -0
  32. data/lib/unique_args.rb +97 -0
  33. data/lib/unique_bitmask.rb +14 -12
  34. data/lib/worker.rb +152 -0
  35. data/lib/worker_runner.rb +106 -0
  36. data/migration/LICENSE +374 -0
  37. data/migration/README.md +11 -0
  38. data/migration/manifest.json +38 -0
  39. data/migration/postgresql/main/001_create_river_migration.down.sql +1 -0
  40. data/migration/postgresql/main/001_create_river_migration.up.sql +8 -0
  41. data/migration/postgresql/main/002_initial_schema.down.sql +5 -0
  42. data/migration/postgresql/main/002_initial_schema.up.sql +96 -0
  43. data/migration/postgresql/main/003_river_job_tags_non_null.down.sql +3 -0
  44. data/migration/postgresql/main/003_river_job_tags_non_null.up.sql +3 -0
  45. data/migration/postgresql/main/004_pending_and_more.down.sql +42 -0
  46. data/migration/postgresql/main/004_pending_and_more.up.sql +45 -0
  47. data/migration/postgresql/main/005_migration_unique_client.down.sql +57 -0
  48. data/migration/postgresql/main/005_migration_unique_client.up.sql +79 -0
  49. data/migration/postgresql/main/006_bulk_unique.down.sql +16 -0
  50. data/migration/postgresql/main/006_bulk_unique.up.sql +40 -0
  51. data/migration/postgresql/main/007_notification_outbox_sqlite_jsonb_and_sql_cleanup.down.sql +56 -0
  52. data/migration/postgresql/main/007_notification_outbox_sqlite_jsonb_and_sql_cleanup.up.sql +44 -0
  53. data/migration/postgresql/main/008_job_id_autoincrement.down.sql +3 -0
  54. data/migration/postgresql/main/008_job_id_autoincrement.up.sql +3 -0
  55. data/migration/sqlite/main/001_create_river_migration.down.sql +1 -0
  56. data/migration/sqlite/main/001_create_river_migration.up.sql +8 -0
  57. data/migration/sqlite/main/002_initial_schema.down.sql +8 -0
  58. data/migration/sqlite/main/002_initial_schema.up.sql +19 -0
  59. data/migration/sqlite/main/003_river_job_tags_non_null.down.sql +6 -0
  60. data/migration/sqlite/main/003_river_job_tags_non_null.up.sql +6 -0
  61. data/migration/sqlite/main/004_pending_and_more.down.sql +26 -0
  62. data/migration/sqlite/main/004_pending_and_more.up.sql +33 -0
  63. data/migration/sqlite/main/005_migration_unique_client.down.sql +37 -0
  64. data/migration/sqlite/main/005_migration_unique_client.up.sql +64 -0
  65. data/migration/sqlite/main/006_bulk_unique.down.sql +7 -0
  66. data/migration/sqlite/main/006_bulk_unique.up.sql +63 -0
  67. data/migration/sqlite/main/007_notification_outbox_sqlite_jsonb_and_sql_cleanup.down.sql +255 -0
  68. data/migration/sqlite/main/007_notification_outbox_sqlite_jsonb_and_sql_cleanup.up.sql +261 -0
  69. data/migration/sqlite/main/008_job_id_autoincrement.down.sql +121 -0
  70. data/migration/sqlite/main/008_job_id_autoincrement.up.sql +123 -0
  71. data/sig/client.rbs +95 -0
  72. data/sig/driver.rbs +170 -0
  73. data/sig/insert_opts.rbs +25 -0
  74. data/sig/job.rbs +72 -0
  75. data/sig/job_persisted_logging/plugin.rbs +29 -0
  76. data/sig/migrator.rbs +44 -0
  77. data/sig/riverqueue.rbs +2 -0
  78. data/sig/runtime.rbs +382 -0
  79. data/sig/testing.rbs +95 -0
  80. data/sig/unique_args.rbs +8 -0
  81. data/sig/unique_bitmask.rbs +8 -0
  82. data/sig/worker_runner.rbs +22 -0
  83. metadata +165 -10
data/docs/README.md ADDED
@@ -0,0 +1,765 @@
1
+ # River client for Ruby [![Build Status](https://github.com/riverqueue/riverqueue-ruby/workflows/CI/badge.svg)](https://github.com/riverqueue/riverqueue-ruby/actions) [![Gem Version](https://badge.fury.io/rb/riverqueue.svg)](https://badge.fury.io/rb/riverqueue)
2
+
3
+ A Ruby client for [River](https://github.com/riverqueue/river), packaged in the [`riverqueue` gem](https://rubygems.org/gems/riverqueue). It inserts and works jobs using River's canonical database schema and state machine, so Ruby and Go clients can safely share a River database. Separate queues are recommended when each language recognizes different job kinds.
4
+
5
+ ## Installation
6
+
7
+ Moving an existing application? See [Migrating from Sidekiq](migrating_from_sidekiq.md).
8
+
9
+ Add one River driver and only the database adapter your application uses. The driver brings in `riverqueue`:
10
+
11
+ ```ruby
12
+ # Sequel
13
+ gem "riverqueue-sequel"
14
+ gem "pg" # or: gem "sqlite3"
15
+
16
+ # Active Record
17
+ gem "riverqueue-activerecord"
18
+ gem "pg" # or: gem "sqlite3"
19
+ ```
20
+
21
+ Apply River's canonical migrations before using the client. See [Schema and migrations](#schema-and-migrations).
22
+
23
+ Both SQL drivers also support [YugabyteDB](yugabyte.md), automatically detecting
24
+ its uniqueness and notification capabilities.
25
+
26
+ ## Schema and migrations
27
+
28
+ Run migrations before inserting jobs or starting workers. The bundled `river`
29
+ command uses Go's canonical PostgreSQL and SQLite migrations; no Go installation
30
+ is needed. After installing a River driver and its database adapter, create your
31
+ application's PostgreSQL database and run:
32
+
33
+ ```sh
34
+ export DATABASE_URL=postgres://localhost/my_app
35
+
36
+ bundle exec river migrate-status
37
+ bundle exec river migrate-up --dry-run
38
+ bundle exec river migrate-up
39
+ ```
40
+
41
+ `migrate-up` applies all pending migrations and is safe to rerun: already-applied
42
+ versions are skipped, including migrations applied by Go. `--dry-run` previews
43
+ the plan without changing the database. Pass `--database-url URL` to override
44
+ `DATABASE_URL`.
45
+
46
+ For SQLite, ensure the parent directory exists, then use a SQLite URL:
47
+
48
+ ```sh
49
+ # With riverqueue-sequel:
50
+ bundle exec river migrate-up --database-url sqlite://storage/river.sqlite3
51
+ ```
52
+
53
+ With `riverqueue-activerecord`, use `sqlite3://storage/river.sqlite3` instead.
54
+ The command auto-detects the installed River driver, preferring Sequel when
55
+ both are in your bundle.
56
+
57
+ For River Pro, install `riverqueue-pro`, apply the main migrations above, then
58
+ run `bundle exec river migrate-up --line pro` against the same database.
59
+
60
+ See the [migration guide](migrations.md) for the Ruby API, PostgreSQL schemas,
61
+ target versions, downgrade precautions, and existing SQLite Pro installations.
62
+ Test schema snapshots under `spec/support` are test-only and must not be used
63
+ to provision production databases.
64
+
65
+ ## Basic usage
66
+
67
+ Define JSON-serializable job arguments and a worker with the same `kind`, register the worker on a queue, and start the client:
68
+
69
+ ```ruby
70
+ require "riverqueue-sequel"
71
+
72
+ class SortArgs
73
+ attr_reader :strings
74
+
75
+ def initialize(strings:)
76
+ @strings = strings
77
+ end
78
+
79
+ def kind = "sort"
80
+ def to_json = JSON.generate(strings: strings)
81
+ end
82
+
83
+ class SortWorker
84
+ def self.kind = "sort"
85
+
86
+ def work(job)
87
+ job.output = {strings: job.args.fetch("strings").sort}
88
+ end
89
+ end
90
+
91
+ client = River::Client.new(
92
+ River::Driver::Sequel.new(DB),
93
+ config: River::Config.new(
94
+ queues: {ruby: 10},
95
+ workers: River::Workers.new.add(SortWorker)
96
+ )
97
+ ).start
98
+
99
+ result = client.insert(
100
+ SortArgs.new(strings: %w[whale tiger bear]),
101
+ queue: :ruby
102
+ )
103
+ result.job # River::JobRow
104
+
105
+ client.stop
106
+ ```
107
+
108
+ Job arguments must respond to `#kind` and `#to_json`. They may also return default options from `#insert_opts`; options passed directly to `#insert` take precedence. Workers receive a `River::Job`, which delegates persisted attributes like `id`, `args`, `attempt`, and `metadata` to its `River::JobRow`.
109
+
110
+ Use strings for `#kind` definitions, and symbols for identifiers such as queues,
111
+ states, periodic job IDs, and resumable step names. Both forms are accepted.
112
+ Persisted job attributes and JSON object keys remain strings, preserving
113
+ compatibility with Go clients.
114
+ Examples omit parentheses on simple calls and `do...end` blocks, retaining them
115
+ for nested expressions and `{ ... }` blocks where they make binding clear.
116
+
117
+ ## Core features
118
+
119
+ ### [Accessing the client from workers](https://riverqueue.com/docs/context-client)
120
+
121
+ Every running `River::Job` exposes the client that claimed it. Workers can use
122
+ `job.client` to insert follow-up work or call other client APIs without relying
123
+ on a global:
124
+
125
+ ```ruby
126
+ def work(job)
127
+ result = process(job.args)
128
+ job.client.insert NotifyArgs.new(result_id: result.id)
129
+ end
130
+ ```
131
+
132
+ ### [Job insertion and options](https://riverqueue.com/docs/inserting-and-working-jobs)
133
+
134
+ Insertion keywords control the queue, priority, maximum attempts, schedule,
135
+ tags, metadata, and uniqueness of a job. Priority `1` is highest.
136
+
137
+ ```ruby
138
+ result = client.insert(args,
139
+ max_attempts: 10,
140
+ metadata: {trace_id: trace_id},
141
+ priority: 1,
142
+ queue: :critical,
143
+ tags: %w[billing customer-42]
144
+ )
145
+ ```
146
+
147
+ For reusable options, pass `insert_opts: River::InsertOpts.new(...)` instead.
148
+ Do not mix an options object with keyword options in the same call. Options
149
+ supplied at insertion override argument-level `#insert_opts` defaults; metadata
150
+ is merged, with call-site values winning.
151
+
152
+ For simple jobs, `River::JobArgsHash.new(:kind, hash)` avoids defining an argument class.
153
+
154
+ ### [Transactional enqueueing](https://riverqueue.com/docs/transactional-enqueueing)
155
+
156
+ Insertion hooks and middleware execute inside the insertion transaction. If
157
+ they raise, their database writes and the enqueue roll back together. PostgreSQL
158
+ insert notifications are delivered only when the surrounding transaction commits.
159
+
160
+ Inserts automatically join a transaction opened through the same Active Record connection or Sequel database object. A rollback also rolls back the job:
161
+
162
+ The ActiveRecord driver defaults to `ActiveRecord::Base`. Use
163
+ `River::Driver::ActiveRecord.new(connection_class: ApplicationRecord)` to select
164
+ an abstract connection class. Queries, inserts, transactions, runtime operations,
165
+ and migrations all use its pool. Each driver has an isolated internal model and
166
+ does not inherit application scopes or callbacks. Transactions on a different
167
+ connection do not roll back River inserts.
168
+
169
+ ```ruby
170
+ DB.transaction do
171
+ save_order
172
+ client.insert FulfillOrderArgs.new(order_id: order.id)
173
+ end
174
+ ```
175
+
176
+ The equivalent works inside `ActiveRecord::Base.transaction` with the Active Record driver.
177
+
178
+ ### [Bulk insertion](https://riverqueue.com/docs/inserting-many-jobs)
179
+
180
+ `#insert_many` inserts a batch atomically and returns one `River::JobInsertResult` per input. Use `River::InsertManyParams` when jobs need different options:
181
+
182
+ ```ruby
183
+ results = client.insert_many([
184
+ SortArgs.new(strings: %w[c b a]),
185
+ River::InsertManyParams.new(
186
+ SortArgs.new(strings: %w[z y x]),
187
+ queue: :bulk
188
+ )
189
+ ])
190
+ ```
191
+
192
+ ### [Scheduled jobs](https://riverqueue.com/docs/scheduled-jobs)
193
+
194
+ Set `scheduled_at` to keep a job from becoming available before a future UTC time. River's maintenance leader promotes it when due:
195
+
196
+ ```ruby
197
+ client.insert args, scheduled_at: Time.now.utc + 3600
198
+ ```
199
+
200
+ ### [Unique jobs](https://riverqueue.com/docs/unique-jobs)
201
+
202
+ `River::UniqueOpts` can make a kind unique by all or selected arguments, time period, queue, and state. A conflict returns the existing job with `result.unique_skipped_as_duplicate?` true. The older `unique_skipped_as_duplicated` reader remains available as a compatibility alias.
203
+
204
+ ```ruby
205
+ result = client.insert(args,
206
+ unique_opts: River::UniqueOpts.new(
207
+ by_args: [:account_id],
208
+ by_period: 15 * 60,
209
+ by_queue: true
210
+ )
211
+ )
212
+ ```
213
+
214
+ Custom `by_state` sets must contain `:available`, `:pending`, `:running`, and `:scheduled`. Set `exclude_kind: true` to enforce the same key across multiple job kinds.
215
+ It requires `by_args`, `by_queue`, or a nonzero `by_period`; without another
216
+ key dimension, insertion raises `ArgumentError`. Setting `by_state` alone
217
+ does not satisfy this requirement.
218
+
219
+ Use nested arrays to select nested unique fields, for example
220
+ `by_args: [[:account, :id], :region]`. A string such as `"account.id"` selects a
221
+ literal top-level key. For cross-language deduplication, producers must agree
222
+ on encoded argument values: escaping, number representations, and nested key
223
+ order affect the hash even when the decoded JSON is equivalent.
224
+
225
+ ### [Reliable execution and stuck jobs](https://riverqueue.com/docs/reliable-workers)
226
+
227
+ Claims and state transitions are atomic in the database. If a process disappears while working, the elected maintenance client rescues stale running jobs after an hour, retrying or discarding them according to their attempt count. `attempted_by`, attempt errors, and final state remain in the canonical River row for inspection by Ruby, Go, or River UI.
228
+
229
+ ### [Job retries](https://riverqueue.com/docs/job-retries)
230
+
231
+ An exception normally moves a job to `retryable`; exhausting `max_attempts` moves it to `discarded`. Workers may choose an absolute retry time, or a client-wide policy may calculate it:
232
+
233
+ ```ruby
234
+ class APIWorker
235
+ def self.kind = "api"
236
+ def work(job) = call_api(job.args)
237
+ def next_retry(_job, _error) = Time.now.utc + 30
238
+ end
239
+
240
+ config = River::Config.new(
241
+ retry_policy: MyRetryPolicy.new # responds to next_retry(job, error, now:)
242
+ )
243
+ ```
244
+
245
+ Use `client.job_retry(job_id)` to make a non-running job available immediately.
246
+
247
+ A worker may implement `retry?(job, error)` and return false to discard a
248
+ reported error immediately, without reducing the attempt budget used for crash
249
+ recovery. The Rails integration uses this to let Active Job own application retries.
250
+ If a retry hook or policy raises, River logs the callback error and falls back to
251
+ retrying with the default backoff. The original work error is still recorded.
252
+
253
+ ### [Error handling and timeouts](https://riverqueue.com/docs/error-handling)
254
+
255
+ Errors are recorded on the job with their attempt, message, timestamp, and trace. `error_handler` may return `:cancel` or `true` to cancel instead of retrying. `job_timeout` defaults to 60 seconds; a worker-specific `timeout(job)` may override it, return `nil` to disable it, or return `0` to use the client default.
256
+
257
+ A stored job that cannot be decoded fails with `River::JobRowDecodeError` before
258
+ worker hooks or middleware run. Its attempt uses the client's error handler and
259
+ retry policy; healthy jobs in the same fetch still run. Administrative reads
260
+ raise the same exception, with readable fields available through `error.job`.
261
+ Corrupt values remain stored for diagnosis, except that malformed error history
262
+ is wrapped in an array so the new failure can be appended.
263
+
264
+ ```ruby
265
+ config = River::Config.new(
266
+ error_handler: ->(error, _job) { :cancel if error.is_a?(PermanentError) },
267
+ job_timeout: 30
268
+ )
269
+ ```
270
+
271
+ ### [Cancelling jobs](https://riverqueue.com/docs/cancelling-jobs)
272
+
273
+ Cancel a job externally with `client.job_cancel(id)`. Available jobs finalize immediately; running workers are interrupted after the runtime observes the cancellation marker. A worker can cancel itself by raising the error returned from `River.job_cancel`.
274
+
275
+ ```ruby
276
+ client.job_cancel job_id
277
+
278
+ def work(job)
279
+ raise River.job_cancel("account closed") if account_closed?(job)
280
+ end
281
+ ```
282
+
283
+ `River.job_cancel` also accepts an exception, which is retained as the
284
+ `River::JobCancelError` cause. The error class is public for rescue clauses and
285
+ test assertions.
286
+
287
+ ### [Snoozing jobs](https://riverqueue.com/docs/snoozing-jobs)
288
+
289
+ Raise the error returned by `River.job_snooze` to reschedule without consuming an attempt. Short snoozes become immediately fetchable after their delay; longer ones are promoted by maintenance. `River::JobSnoozeError` remains public for rescue clauses and test assertions.
290
+
291
+ ```ruby
292
+ def work(job)
293
+ raise River.job_snooze(30) unless dependency_ready?(job)
294
+ end
295
+ ```
296
+
297
+ ### [Multiple queues](https://riverqueue.com/docs/multiple-queues)
298
+
299
+ Queues isolate throughput and set independent thread concurrency. Each queue has a producer thread, and claimed jobs run in worker threads up to `max_workers`.
300
+
301
+ ```ruby
302
+ config = River::Config.new(queues: {
303
+ bulk: River::QueueConfig.new(
304
+ fetch_cooldown: 0.2,
305
+ fetch_poll_interval: 1.0,
306
+ max_workers: 4
307
+ ),
308
+ critical: River::QueueConfig.new(max_workers: 20)
309
+ })
310
+ ```
311
+
312
+ Queues may also be added and removed at runtime with `client.queue_add(name, config)` and `client.queue_remove(name)`.
313
+
314
+ ### [Pausing queues](https://riverqueue.com/docs/pausing-queues)
315
+
316
+ Pausing is persisted, so every client sharing the database observes it. Pass `"*"` to affect all queues.
317
+
318
+ ```ruby
319
+ client.queue_pause :bulk
320
+ client.queue_resume :bulk
321
+
322
+ client.queue_pause "*"
323
+ client.queue_resume "*"
324
+ ```
325
+
326
+ Use `queue_get`, `queue_list`, and `queue_update` to inspect queues and attach metadata.
327
+
328
+ ### Rails and Active Job
329
+
330
+ Install the separate `riverqueue-rails` gem for `config.active_job.queue_adapter = :river`,
331
+ Active Job/Action Mailer execution, Rails context handling, and `bin/jobs start`.
332
+ See the [Rails integration guide](../rails/riverqueue-rails/README.md) for setup,
333
+ transactional enqueueing, retry semantics, and supported Rails versions.
334
+
335
+ ### [Periodic jobs](https://riverqueue.com/docs/periodic-jobs)
336
+
337
+ Register a schedule and a factory block that returns job arguments,
338
+ `[arguments, insert_options]`, or `nil` to skip that run. The block runs when
339
+ the job is due, not during registration. A reusable callable can be passed as
340
+ `constructor:` instead; don't supply both. Core periodic schedules live in the
341
+ client process; River Pro adds durable schedules.
342
+
343
+ ```ruby
344
+ cleanup = River::PeriodicJob.new(
345
+ id: :cleanup,
346
+ run_on_start: true,
347
+ schedule: River::PeriodicInterval.new(3600)
348
+ ) do
349
+ [CleanupArgs.new, River::InsertOpts.new(queue: :maintenance)]
350
+ end
351
+
352
+ config = River::Config.new(periodic_jobs: [cleanup])
353
+ handle = client.periodic_jobs.add(another_periodic_job)
354
+ client.periodic_jobs.remove handle
355
+ ```
356
+
357
+ `client.periodic_jobs` returns a `River::PeriodicJobBundle`. Removing a handle
358
+ returns the removed `PeriodicJob`, or `nil` when absent. `clear` removes all
359
+ registrations and returns the bundle.
360
+ `add_many` registers a batch atomically: duplicate IDs or invalid schedules leave
361
+ the registry unchanged.
362
+
363
+ Schedules can be callbacks (`schedule: ->(now) { now + 300 }`) or objects
364
+ implementing `next(time)`. They must return a `Time` strictly after the supplied
365
+ time; they do not execute the job themselves. A schedule that raises at runtime
366
+ is logged and retried on the next maintenance pass without blocking other jobs.
367
+
368
+ For calendar schedules, add `gem "fugit", "~> 1.13"` to your Gemfile:
369
+
370
+ ```ruby
371
+ cleanup = River::PeriodicJob.new(
372
+ id: :weekday_cleanup,
373
+ schedule: River::PeriodicCron.new("0 9 * * 1-5", timezone: "America/New_York")
374
+ ) { CleanupArgs.new }
375
+ ```
376
+
377
+ `PeriodicCron` parses once and loads Fugit only when constructed. Fugit is not
378
+ a runtime dependency of the River gem. The timezone defaults explicitly to UTC;
379
+ provide it through `timezone:`, not inside the expression. Five-field cron,
380
+ optional seconds, and aliases such as `@daily` use Fugit's syntax. Results are
381
+ UTC `Time` objects. Local calendar times follow Fugit's daylight-saving rules;
382
+ nonexistent spring-forward times are skipped. Test ambiguous fall-back times
383
+ for your schedules. Cron does not change core scheduling durability or replay
384
+ missed occurrences after downtime.
385
+
386
+ For a one-time date, insert a job with
387
+ `client.insert(args, scheduled_at: Time.utc(2026, 9, 20, 9))` instead of registering
388
+ a periodic job. This stores the scheduled job immediately in the database.
389
+
390
+ ### [Resumable jobs](https://riverqueue.com/docs/resumable-jobs)
391
+
392
+ Long jobs can checkpoint idempotent steps and cursor progress. On retry, River skips completed steps and resumes a cursor step from its last recorded value using the same metadata format as Go.
393
+
394
+ ```ruby
395
+ def work(job)
396
+ job.resumable_step :download do
397
+ download(job.args)
398
+ end
399
+
400
+ job.resumable_step_cursor :rows, default: 0 do |last_row|
401
+ import_rows(after: last_row) do |row|
402
+ job.resumable_set_cursor row.id
403
+ end
404
+ end
405
+ end
406
+ ```
407
+
408
+ Step exceptions propagate immediately: later code in the worker does not run
409
+ unless it explicitly rescues the error. Middleware and error hooks see the same
410
+ exception as for ordinary work.
411
+ Step names must be unique within an invocation, and steps cannot be nested:
412
+ skipping an outer step on retry would make an inner checkpoint unreachable.
413
+
414
+ `job.resumable_checkpoint(cursor: value)` writes a checkpoint immediately. Omit
415
+ `cursor:` to checkpoint the current step with any cursor already recorded. For
416
+ atomic application writes, put the transaction **inside** the step and let
417
+ rollback errors propagate out of it:
418
+
419
+ ```ruby
420
+ job.resumable_step_cursor :rows, default: 0 do |last_row|
421
+ import_rows(after: last_row) do |row|
422
+ job.client.driver.transaction do
423
+ save_row(row)
424
+ job.resumable_checkpoint cursor: row.id
425
+ end
426
+ end
427
+ end
428
+ ```
429
+
430
+ Use the same database connection for `save_row` and the checkpoint. A rolled-back
431
+ checkpoint is not replayed when the attempt fails; retries use the last committed
432
+ progress. Do not swallow rollback errors or wrap an entire completed step in a
433
+ transaction that may subsequently roll back. As with ordinary jobs, steps must
434
+ remain idempotent.
435
+
436
+ ### [Recorded output and metadata](https://riverqueue.com/docs/recorded-output)
437
+
438
+ Assign `job.output` to store JSON-compatible output under `metadata["output"]`. Use `job.update_metadata` for other metadata that should be committed with the attempt's final transition.
439
+
440
+ Values are validated and copied when assigned. Invalid JSON fails the attempt
441
+ normally without preventing its error from being recorded. `job.metadata` and
442
+ `job.metadata_updates` return snapshots; use `update_metadata` to persist changes.
443
+
444
+ ```ruby
445
+ def work(job)
446
+ job.update_metadata provider_request_id: request_id
447
+ job.output = {imported: 42}
448
+ end
449
+ ```
450
+
451
+ ### Job-persisted logging
452
+
453
+ Add `River::JobPersistedLogging::Plugin` to save worker logs with each job attempt,
454
+ using the same `metadata["river:log"]` format as Go's `riverlog` and River UI. This
455
+ is a core feature; it does not require Pro or additional migrations.
456
+
457
+ ```ruby
458
+ config = River::Config.new(
459
+ plugins: [River::JobPersistedLogging::Plugin.new],
460
+ workers: workers,
461
+ queues: {default: 10}
462
+ )
463
+
464
+ def work(job)
465
+ job.logger.info "Starting import"
466
+ job.logger.warn "Skipped a malformed row"
467
+ end
468
+ ```
469
+
470
+ `job.logger` is a fresh standard Ruby `Logger` at INFO level for each attempt.
471
+ Calling it without the plugin raises a configuration error. To customize
472
+ formatting, severity, or use another logger, supply a factory block:
473
+
474
+ ```ruby
475
+ logging = River::JobPersistedLogging::Plugin.new(
476
+ max_size_bytes: 256 * 1024,
477
+ max_total_bytes: 1024 * 1024
478
+ ) do |writer|
479
+ Logger.new(writer, level: :debug,
480
+ formatter: ->(severity, time, _progname, message) {
481
+ JSON.generate(level: severity, time: time.utc.iso8601, message: message) + "\n"
482
+ })
483
+ end
484
+ ```
485
+
486
+ The factory runs once per attempt and receives a thread-safe, bounded writer
487
+ supporting `write` and `close`. Return a new logger that writes to it. Only this
488
+ logger's output is captured: the plugin does not replace `Config#logger`,
489
+ `Rails.logger`, Active Job's logger, or redirect stdout/stderr. Pass `job.logger`
490
+ to application code that should contribute to the job log. Put the plugin
491
+ before other work middleware if that middleware also needs `job.logger`.
492
+
493
+ Logs are saved with the attempt's final state transition, including failures,
494
+ timeouts, snoozes, cancellations, and graceful interruption. Entries have the
495
+ shape `{"attempt": 1, "log": "..."}` and append across retries. Empty attempts
496
+ add nothing. No separate database write is made for each log line. Logs are not
497
+ live-streamed; a process crash or failed finalization can lose the current
498
+ attempt's buffered logs. Jobs deleted by an ephemeral-job plugin retain no logs.
499
+
500
+ The default capture limit is 2 MiB per attempt; excess bytes are discarded as
501
+ they are written. The default history limit is 8 MiB of serialized JSON, capped
502
+ at 64 MiB. Oldest entries are dropped first, but the newest entry is always
503
+ retained even if it alone exceeds the history limit. Both settings must be
504
+ positive integers. Invalid/incomplete UTF-8 sequences and NUL bytes are removed
505
+ so the log is safe to store. Truncation, dropped history, and malformed existing
506
+ log metadata are reported through `Config#logger`; malformed history is left
507
+ unchanged. Join any child threads before returning from work; writes after
508
+ capture closes are ignored. Avoid logging secrets: logs live in job metadata
509
+ and follow the job's retention policy.
510
+
511
+ ### Plugins
512
+
513
+ Plugins provide one ordered configuration point for lifecycle callbacks and
514
+ wrapping middleware. These are two distinct extension styles even though both
515
+ are registered through `Config#plugins`:
516
+
517
+ - A **hook** runs at one specific lifecycle point and then returns. Hooks are
518
+ appropriate for observing or making a small change at that point.
519
+ - **Middleware** wraps a complete insertion or work operation. It can run code
520
+ before and after the inner operation, and must call `operation.call` to let
521
+ that operation continue.
522
+
523
+ A plugin may implement any combination of these methods:
524
+
525
+ | Style | Method | When it runs |
526
+ | ---------- | -------------------------------- | -------------------------------------------------------------------------------------------- |
527
+ | Hook | `insert_begin(params)` | Before each job is inserted; `params` may be modified. |
528
+ | Hook | `insert_end(result)` | After each job is inserted. |
529
+ | Hook | `work_begin(job)` | After a job is claimed, immediately before its worker runs. |
530
+ | Hook | `work_end(job, error)` | After the worker returns or raises; `error` is `nil` on success. |
531
+ | Hook | `job_finalize(job, state)` | Before successful finalization; returning `:delete` deletes the job instead of retaining it. |
532
+ | Middleware | `insert_many(params, operation)` | Around one insertion call; `params` is an array even for `Client#insert`. |
533
+ | Middleware | `work(job, operation)` | Around the work hooks and worker for one claimed job. |
534
+
535
+ A single plugin can provide both styles. For example, it might use
536
+ `insert_begin` to add metadata and `work` to time the complete work operation.
537
+
538
+ ```ruby
539
+ class TimingPlugin
540
+ def work(job, operation)
541
+ started_at = Process.clock_gettime(Process::CLOCK_MONOTONIC)
542
+ operation.call
543
+ ensure
544
+ Metrics.observe(
545
+ job.kind,
546
+ Process.clock_gettime(Process::CLOCK_MONOTONIC) - started_at
547
+ )
548
+ end
549
+ end
550
+
551
+ config = River::Config.new(plugins: [AuditPlugin.new, TimingPlugin.new])
552
+ ```
553
+
554
+ Plugins earlier in the list are the outermost wrappers. Begin callbacks run in
555
+ configuration order, while `insert_end` callbacks run in reverse order. In
556
+ effect, work execution is nested as: middleware before, `work_begin`, worker,
557
+ `work_end`, middleware after.
558
+
559
+ ### [Subscriptions](https://riverqueue.com/docs/subscriptions)
560
+
561
+ Subscribe to job and queue events for logging or metrics. Subscriptions are
562
+ bounded and drop new events rather than blocking workers when their buffer is
563
+ full. `subscription.close` unregisters it from the client and wakes waiting
564
+ readers; closing more than once is safe. Buffered events remain readable, then
565
+ `each` ends and blocking `pop` calls return `nil`. Non-blocking `pop(true)` raises
566
+ `ThreadError` whenever no event is available, including after closure.
567
+
568
+ ```ruby
569
+ subscription = client.subscribe(
570
+ :job_completed,
571
+ :job_failed,
572
+ buffer_size: 1_000
573
+ )
574
+
575
+ subscription.each { |event| consume(event) }
576
+ subscription.close
577
+ ```
578
+
579
+ Events include completed, failed, cancelled, snoozed, and interrupted jobs, plus paused and resumed queues.
580
+
581
+ ### Job administration
582
+
583
+ The client can fetch, filter, update, cancel, retry, and delete jobs. Lists return
584
+ a `JobListCursor` in `last_cursor`. Pass it as `after`, preserving the same filters
585
+ and ordering, to fetch the next page. Cursors retain both the sort value and ID,
586
+ so timestamp ordering handles ties and continues working if the cursor job is
587
+ deleted. Null timestamps sort last in either direction. For ID ordering only,
588
+ `after_id` is also available as a shortcut with an integer ID.
589
+
590
+ ```ruby
591
+ list_options = {
592
+ limit: 100,
593
+ queues: [:bulk],
594
+ sort_by: :scheduled_at,
595
+ states: [:discarded],
596
+ tags_any: ["billing"]
597
+ }
598
+ page = client.job_list(**list_options)
599
+
600
+ next_page = client.job_list(**list_options, after: page.last_cursor)
601
+ client.job_update job_id, max_attempts: 50
602
+ client.job_delete_many states: [:cancelled]
603
+ ```
604
+
605
+ Reusable `JobListParams` and `JobUpdateParams` objects are also accepted as
606
+ positional arguments, instead of keywords. For updates, omitted fields remain
607
+ unchanged; an explicit `nil` clears a nullable field.
608
+
609
+ Bulk deletion requires at least one filter and never deletes running jobs.
610
+ Metadata filters compare complete JSON values at each supplied top-level key,
611
+ including nested objects and arrays. Numbers, strings, and booleans remain
612
+ distinct; a null value matches a present JSON null, not a missing key.
613
+
614
+ ### [Leader election](https://riverqueue.com/docs/leader-election)
615
+
616
+ Running clients coordinate through the canonical `river_leader` table. Only the
617
+ current leader performs database-wide scheduling, stuck-job rescue, retention,
618
+ and custom maintenance, and another client can take over after its lease
619
+ expires.
620
+
621
+ Set `leader_election_disabled: true` for a worker-only client. It continues
622
+ fetching and executing jobs, including queues added after startup, but never
623
+ runs maintenance. Another client in the same database/schema must remain
624
+ eligible to lead. Periodic jobs cannot be configured or modified on a client
625
+ with leader election disabled.
626
+
627
+ For clients that share a queue but implement different job kinds, set
628
+ `fetch_only_known_kinds: true`. Only registered kinds and aliases are claimed;
629
+ other jobs remain available without consuming attempts. Register workers before
630
+ starting the client. An empty registry fetches nothing. This option affects
631
+ fetching only; combine it with `leader_election_disabled` when another client
632
+ should own database-wide maintenance. Both options default to `false`.
633
+
634
+ ### [Maintenance services and retention](https://riverqueue.com/docs/maintenance-services)
635
+
636
+ The maintenance leader promotes scheduled jobs, rescues stuck work, deletes
637
+ finalized rows, and runs custom services. Retention is configured in seconds;
638
+ use `nil` or `-1` to retain a state indefinitely.
639
+
640
+ ```ruby
641
+ config = River::Config.new(
642
+ cancelled_job_retention_period: 86_400,
643
+ completed_job_retention_period: 86_400,
644
+ discarded_job_retention_period: 7 * 86_400,
645
+ maintenance_services: [MyMaintenanceService.new]
646
+ )
647
+ ```
648
+
649
+ A custom service implements `run(client, driver, now)` and runs only while this client holds leadership.
650
+
651
+ On SQLite, the leader also removes notification outbox entries older than five
652
+ minutes in bounded batches. Cancellation requests write Go-compatible control
653
+ notifications in the same transaction as the job update; Ruby workers continue
654
+ to observe cancellation through polling.
655
+
656
+ ### [Renaming job kinds](https://riverqueue.com/docs/renaming-jobs)
657
+
658
+ Register old names as aliases while producers migrate to a new kind. All aliases resolve to the same worker:
659
+
660
+ ```ruby
661
+ workers = River::Workers.new.add(NewReportWorker, aliases: [:old_report])
662
+ ```
663
+
664
+ Keep aliases registered until no jobs with the old kind remain.
665
+
666
+ Use `workers[:kind]` for a lookup that returns `nil` when missing, or
667
+ `workers.fetch(:kind)` to raise `KeyError`. Like `Hash#fetch`, it also accepts
668
+ an explicit default (`workers.fetch(:kind, nil)`) or a fallback block.
669
+
670
+ ### [Stopping gracefully](https://riverqueue.com/docs/graceful-shutdown)
671
+
672
+ For a dedicated foreground worker process with application boot and signal
673
+ handling, use the [worker command](./workers.md):
674
+
675
+ ```sh
676
+ bundle exec river worker --config config/river.rb --stop-timeout 30
677
+ # Rails, from the application root:
678
+ RAILS_ENV=production bundle exec river worker --rails
679
+ ```
680
+
681
+ The Ruby configuration file must return an unstarted client. The following client
682
+ methods are for applications managing their own runtime lifecycle:
683
+
684
+ `client.stop` stops fetching and waits for active jobs to finish. `client.stop_and_cancel` interrupts active worker threads and returns their jobs to `available` without consuming the interrupted attempt.
685
+
686
+ Use `client.stop(wait: false)` to request stop and return immediately without
687
+ interrupting active workers. Call `client.stop` later to wait for draining and
688
+ finish cleanup. Until that waiting call completes, `started?` remains true and
689
+ `stopped?` remains false. In-flight fetches or maintenance operations may finish.
690
+
691
+ ### [Insert-only clients](https://riverqueue.com/docs/insert-only-clients)
692
+
693
+ A client with no configured queues can insert and administer jobs without starting worker or maintenance threads:
694
+
695
+ ```ruby
696
+ client = River::Client.new(driver, config: River::Config.new(queues: {}))
697
+ client.insert args
698
+ ```
699
+
700
+ ## Threads and Ractors
701
+
702
+ Queue producers, maintenance, and jobs run in threads. River keeps core constants shareable and mutable runtime state per client. Tests exercise insertion, uniqueness, resumable work, events, periodic scheduling, and worker threads inside a non-main Ractor using an in-memory test driver. This is groundwork for Ractor support, not a guarantee that the supplied database drivers work in Ractors.
703
+
704
+ Load River and worker definitions in the main Ractor before spawning others. Each Ractor must create and own its clients, configuration, worker instances, callbacks, and database connections; do not share live clients or pools across Ractors. Custom argument encoders should prefer `JSON.generate` to `JSON.dump`, which depends on mutable global options.
705
+
706
+ Database drivers, Active Record, Sequel, and optional dependencies such as Fugit still need their own Ractor compatibility. Job timeouts also depend on Ruby and the `timeout` gem: the test suite exercises them on Ruby 4 with `timeout` 0.6.1; tests on older Rubies disable job timeouts. `WorkerRunner` handles process-wide signals and must run on the main thread of the main Ractor.
707
+
708
+ Ruby 4.0.2 with `timeout` 0.6.1 can intermittently deadlock during VM shutdown
709
+ while terminating Ractors and their timeout helper threads, even after work
710
+ finishes successfully. This reproduces without River. Tests bypass that shutdown
711
+ path only in their disposable subprocesses, after checking both Ractors' results.
712
+ Keep production workers on the thread-based runtime until the Ruby and driver
713
+ limitations are resolved.
714
+
715
+ ## RBS and type checking
716
+
717
+ The gem bundles [RBS files](https://github.com/riverqueue/riverqueue-ruby/tree/master/sig) for tools such as [Steep](https://github.com/soutaro/steep) and other RBS-compatible type checkers.
718
+
719
+ ## Drivers
720
+
721
+ ### Active Record
722
+
723
+ ```ruby
724
+ require "riverqueue-activerecord"
725
+
726
+ ActiveRecord::Base.establish_connection("postgres://...")
727
+ client = River::Client.new(River::Driver::ActiveRecord.new)
728
+ ```
729
+
730
+ ### Sequel
731
+
732
+ ```ruby
733
+ require "riverqueue-sequel"
734
+
735
+ DB = Sequel.connect("postgres://...")
736
+ client = River::Client.new(River::Driver::Sequel.new(DB))
737
+ ```
738
+
739
+ Neither driver installs `pg` or `sqlite3`; the application chooses its adapter.
740
+
741
+ ## Testing
742
+
743
+ For database-backed insertion assertions and synchronous worker tests, see
744
+ [Testing River jobs](./testing.md). Helpers ship in the core gem; RSpec and
745
+ Minitest integrations are optional and explicitly loaded.
746
+
747
+ ## River Pro
748
+
749
+ River Pro is kept in the separate, privately distributed `riverqueue-pro` gem, so possession of that package is the access boundary. It is not included in the MPL-2.0 core gem. Its implementation and documentation live in the private `riverqueue-ruby-pro` repository; see that repository's README for configuration and feature examples.
750
+
751
+ ## Current differences from the Go client
752
+
753
+ The shared PostgreSQL insert-only conformance profile is exercised through both
754
+ SQL drivers. Full runtime, SQLite, and multi-engine conformance remain in
755
+ progress; see the [conformance status and differences](../conformance/README.md).
756
+
757
+ The Ruby client does not currently provide dedicated OpenTelemetry/metrics
758
+ integrations or
759
+ transactional job completion alongside application writes. Plugins and
760
+ subscriptions provide integration points for telemetry. Job execution uses Ruby
761
+ worker objects rather than Go's work-function API.
762
+
763
+ ## Development
764
+
765
+ See [development](./development.md).