riverqueue 0.12.0 → 0.14.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +80 -4
- data/README.md +91 -0
- data/docs/README.md +213 -36
- data/docs/conformance.md +72 -0
- data/docs/migrations.md +20 -23
- data/docs/testing.md +1 -1
- data/lib/client.rb +103 -15
- data/lib/client_runtime.rb +434 -87
- data/lib/config.rb +61 -9
- data/lib/driver/job_row_decoder.rb +2 -2
- data/lib/driver/notification_listener.rb +61 -0
- data/lib/driver/postgres_capabilities.rb +1 -1
- data/lib/driver/runtime.rb +419 -146
- data/lib/driver.rb +1 -0
- data/lib/event.rb +8 -1
- data/lib/insert_opts.rb +8 -3
- data/lib/job.rb +5 -5
- data/lib/job_persisted_logging/plugin.rb +3 -2
- data/lib/migrator.rb +1 -1
- data/lib/periodic_cron.rb +42 -6
- data/lib/periodic_job.rb +23 -4
- data/lib/resumable.rb +24 -1
- data/lib/worker.rb +32 -2
- data/migration/README.md +4 -2
- data/migration/manifest.json +6 -7
- data/migration/postgresql/main/007_notification_outbox_sqlite_jsonb_and_sql_cleanup.down.sql +1 -1
- data/migration/postgresql/main/007_notification_outbox_sqlite_jsonb_and_sql_cleanup.up.sql +1 -1
- data/migration/postgresql/main/008_job_id_autoincrement.down.sql +1 -1
- data/migration/postgresql/main/008_job_id_autoincrement.up.sql +1 -1
- data/migration/sqlite/main/008_job_id_autoincrement.up.sql +1 -1
- data/sig/client.rbs +9 -3
- data/sig/driver.rbs +46 -9
- data/sig/insert_opts.rbs +2 -2
- data/sig/job.rbs +2 -0
- data/sig/runtime.rbs +76 -8
- metadata +9 -6
data/docs/README.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
|
-
# River client for Ruby [](https://github.com/riverqueue/river/actions/workflows/ruby.yaml) [](https://badge.fury.io/rb/riverqueue)
|
|
2
2
|
|
|
3
|
-
A Ruby client for [River](https://github.com/riverqueue/river), packaged in the [`riverqueue` gem](https://rubygems.org/gems/riverqueue). It inserts and works jobs using River's canonical database schema and state machine, so Ruby and
|
|
3
|
+
A Ruby client for [River](https://github.com/riverqueue/river), packaged in the [`riverqueue` gem](https://rubygems.org/gems/riverqueue). It inserts and works jobs using River's canonical database schema and state machine, so Ruby and River clients in other languages can safely share a River database. Separate queues are recommended when each language recognizes different job kinds.
|
|
4
4
|
|
|
5
5
|
## Installation
|
|
6
6
|
|
|
@@ -26,9 +26,9 @@ its uniqueness and notification capabilities.
|
|
|
26
26
|
## Schema and migrations
|
|
27
27
|
|
|
28
28
|
Run migrations before inserting jobs or starting workers. The bundled `river`
|
|
29
|
-
command uses
|
|
30
|
-
|
|
31
|
-
application's
|
|
29
|
+
command uses River's canonical Postgres and SQLite migrations, bundled with the
|
|
30
|
+
gem. After installing a River driver and its database adapter, create your
|
|
31
|
+
application's Postgres database and run:
|
|
32
32
|
|
|
33
33
|
```sh
|
|
34
34
|
export DATABASE_URL=postgres://localhost/my_app
|
|
@@ -39,8 +39,8 @@ bundle exec river migrate-up
|
|
|
39
39
|
```
|
|
40
40
|
|
|
41
41
|
`migrate-up` applies all pending migrations and is safe to rerun: already-applied
|
|
42
|
-
versions are skipped, including migrations applied by
|
|
43
|
-
the plan without changing the database. Pass `--database-url URL` to override
|
|
42
|
+
versions are skipped, including migrations applied by other River clients.
|
|
43
|
+
`--dry-run` previews the plan without changing the database. Pass `--database-url URL` to override
|
|
44
44
|
`DATABASE_URL`.
|
|
45
45
|
|
|
46
46
|
For SQLite, ensure the parent directory exists, then use a SQLite URL:
|
|
@@ -57,7 +57,7 @@ both are in your bundle.
|
|
|
57
57
|
For River Pro, install `riverqueue-pro`, apply the main migrations above, then
|
|
58
58
|
run `bundle exec river migrate-up --line pro` against the same database.
|
|
59
59
|
|
|
60
|
-
See the [migration guide](migrations.md) for the Ruby API,
|
|
60
|
+
See the [migration guide](migrations.md) for the Ruby API, Postgres schemas,
|
|
61
61
|
target versions, downgrade precautions, and existing SQLite Pro installations.
|
|
62
62
|
Test schema snapshots under `spec/support` are test-only and must not be used
|
|
63
63
|
to provision production databases.
|
|
@@ -110,12 +110,37 @@ Job arguments must respond to `#kind` and `#to_json`. They may also return defau
|
|
|
110
110
|
Use strings for `#kind` definitions, and symbols for identifiers such as queues,
|
|
111
111
|
states, periodic job IDs, and resumable step names. Both forms are accepted.
|
|
112
112
|
Persisted job attributes and JSON object keys remain strings, preserving
|
|
113
|
-
compatibility with
|
|
113
|
+
compatibility with River clients in other languages.
|
|
114
114
|
Examples omit parentheses on simple calls and `do...end` blocks, retaining them
|
|
115
115
|
for nested expressions and `{ ... }` blocks where they make binding clear.
|
|
116
116
|
|
|
117
117
|
## Core features
|
|
118
118
|
|
|
119
|
+
### Block workers
|
|
120
|
+
|
|
121
|
+
Pass a kind and a block to `Workers#add` to define a worker inline:
|
|
122
|
+
|
|
123
|
+
```ruby
|
|
124
|
+
workers = River::Workers.new
|
|
125
|
+
workers.add(:message) do |job|
|
|
126
|
+
puts "Message: #{job.args.fetch("message")}"
|
|
127
|
+
end
|
|
128
|
+
|
|
129
|
+
# With client configured to use this registry and the ruby queue:
|
|
130
|
+
client.insert River::JobArgsHash.new(:message, message: "Hello!"), queue: :ruby
|
|
131
|
+
```
|
|
132
|
+
|
|
133
|
+
Pass this registry as `workers:` in `River::Config`. The kind accepts a string
|
|
134
|
+
or symbol and must match the inserted job's kind. The block receives a
|
|
135
|
+
`River::Job` and uses the client's retry and timeout policies. Returning normally
|
|
136
|
+
completes the job; the block's return value is ignored. Raise exceptions to fail,
|
|
137
|
+
cancel, or snooze work as with a worker class. Optionally use `job.output =` to
|
|
138
|
+
persist a result. Aliases are supported through `aliases:` as for other workers.
|
|
139
|
+
Supply a worker or a block, not both. Existing procs or methods can be passed as
|
|
140
|
+
the block with `&`.
|
|
141
|
+
The same block may run concurrently for multiple jobs, so any captured mutable
|
|
142
|
+
state must be safe to share across threads.
|
|
143
|
+
|
|
119
144
|
### [Accessing the client from workers](https://riverqueue.com/docs/context-client)
|
|
120
145
|
|
|
121
146
|
Every running `River::Job` exposes the client that claimed it. Workers can use
|
|
@@ -133,6 +158,12 @@ end
|
|
|
133
158
|
|
|
134
159
|
Insertion keywords control the queue, priority, maximum attempts, schedule,
|
|
135
160
|
tags, metadata, and uniqueness of a job. Priority `1` is highest.
|
|
161
|
+
`max_attempts` must be between 1 and 32,767 on every database.
|
|
162
|
+
An explicit `state` must be `:available`, `:pending`, or `:scheduled` (strings
|
|
163
|
+
are also accepted). Other states raise `ArgumentError` before any jobs are
|
|
164
|
+
inserted, including when an invalid state appears in a batch. Without an
|
|
165
|
+
explicit state, jobs are available immediately or scheduled when `scheduled_at`
|
|
166
|
+
is in the future.
|
|
136
167
|
|
|
137
168
|
```ruby
|
|
138
169
|
result = client.insert(args,
|
|
@@ -153,8 +184,8 @@ For simple jobs, `River::JobArgsHash.new(:kind, hash)` avoids defining an argume
|
|
|
153
184
|
|
|
154
185
|
### [Transactional enqueueing](https://riverqueue.com/docs/transactional-enqueueing)
|
|
155
186
|
|
|
156
|
-
Insertion hooks and middleware execute inside the insertion transaction
|
|
157
|
-
|
|
187
|
+
Insertion hooks and middleware execute inside the insertion transaction, so
|
|
188
|
+
their database writes and the enqueue commit or roll back together. Postgres
|
|
158
189
|
insert notifications are delivered only when the surrounding transaction commits.
|
|
159
190
|
|
|
160
191
|
Inserts automatically join a transaction opened through the same Active Record connection or Sequel database object. A rollback also rolls back the job:
|
|
@@ -175,6 +206,62 @@ end
|
|
|
175
206
|
|
|
176
207
|
The equivalent works inside `ActiveRecord::Base.transaction` with the Active Record driver.
|
|
177
208
|
|
|
209
|
+
River calls inside an open transaction run in that transaction and open no
|
|
210
|
+
savepoint of their own. If a River call raises, it may leave partial changes,
|
|
211
|
+
so roll back the transaction. To recover from the error and continue the
|
|
212
|
+
transaction, wrap the call in your own savepoint, such as
|
|
213
|
+
`DB.transaction(savepoint: true)` or
|
|
214
|
+
`ApplicationRecord.transaction(requires_new: true)`.
|
|
215
|
+
|
|
216
|
+
### Transactional completion
|
|
217
|
+
|
|
218
|
+
Call `job.client.job_complete_tx(job)` inside a transaction to commit application
|
|
219
|
+
writes and job completion together (the equivalent of Go's `JobCompleteTx`):
|
|
220
|
+
|
|
221
|
+
```ruby
|
|
222
|
+
class FulfillOrderWorker
|
|
223
|
+
def work(job)
|
|
224
|
+
DB.transaction do
|
|
225
|
+
DB[:orders].where(id: job.args.fetch("order_id")).update(fulfilled: true)
|
|
226
|
+
job.output = {fulfilled: true}
|
|
227
|
+
job.client.job_complete_tx(job)
|
|
228
|
+
end
|
|
229
|
+
end
|
|
230
|
+
end
|
|
231
|
+
```
|
|
232
|
+
|
|
233
|
+
With Active Record 7.2 or later, use it inside an `ApplicationRecord.transaction` block. As
|
|
234
|
+
with insertion, the transaction must use the driver's connection on the same
|
|
235
|
+
thread. No explicit transaction argument is needed. Calling `job_complete_tx`
|
|
236
|
+
without an active transaction on that connection raises `River::Error` without
|
|
237
|
+
changing the job.
|
|
238
|
+
|
|
239
|
+
Sequel 5.20 or later is required so transaction callbacks respect savepoints.
|
|
240
|
+
|
|
241
|
+
`job_complete_tx` accepts the running `River::Job` passed to the worker and returns
|
|
242
|
+
an updated `River::JobRow`, leaving the original job unchanged. Set output and
|
|
243
|
+
metadata before calling it so they're included in the transaction. A missing
|
|
244
|
+
job raises `River::NotFoundError`; a pending cancellation raises
|
|
245
|
+
`River::JobCancelError`, allowing the transaction to roll back.
|
|
246
|
+
The supplied job must still belong to the current attempt; a stale worker cannot
|
|
247
|
+
complete a replacement attempt.
|
|
248
|
+
Metadata added afterward, including persisted logs, is saved separately when
|
|
249
|
+
the worker finishes, provided the completed attempt has not been retried.
|
|
250
|
+
|
|
251
|
+
Make completion the last operation in the transaction and let the transaction
|
|
252
|
+
commit before returning from `work`. Propagate errors that roll back the
|
|
253
|
+
transaction so River retries the job. If you swallow a rollback and return
|
|
254
|
+
successfully, River completes the job normally. Once completion commits, later
|
|
255
|
+
worker errors cannot retry or cancel the job. River emits `:job_completed` when
|
|
256
|
+
the worker finishes, and skips its usual finalization hooks for the already
|
|
257
|
+
completed job.
|
|
258
|
+
If the completed row is deleted before the worker finishes, River emits the
|
|
259
|
+
completion event using the snapshot retained when the transaction committed.
|
|
260
|
+
|
|
261
|
+
If the job is manually retried before the original worker finishes, that worker
|
|
262
|
+
leaves the replacement attempt untouched and does not emit a completion event
|
|
263
|
+
for it.
|
|
264
|
+
|
|
178
265
|
### [Bulk insertion](https://riverqueue.com/docs/inserting-many-jobs)
|
|
179
266
|
|
|
180
267
|
`#insert_many` inserts a batch atomically and returns one `River::JobInsertResult` per input. Use `River::InsertManyParams` when jobs need different options:
|
|
@@ -224,7 +311,7 @@ order affect the hash even when the decoded JSON is equivalent.
|
|
|
224
311
|
|
|
225
312
|
### [Reliable execution and stuck jobs](https://riverqueue.com/docs/reliable-workers)
|
|
226
313
|
|
|
227
|
-
Claims and state transitions are atomic in the database. If a process disappears while working, the elected maintenance client
|
|
314
|
+
Claims and state transitions are atomic in the database. If a process disappears while working, the elected maintenance client checks for stale running jobs after an hour, retrying or discarding them according to their attempt count. Rescue respects longer client and worker timeouts; disabling a job's timeout also disables automatic timeout-based rescue for it. Cancellation requests still take precedence. Keep worker registrations and timeout configuration consistent across maintenance clients. `attempted_by`, attempt errors, and final state remain in the canonical River row for inspection by any River client or River UI.
|
|
228
315
|
|
|
229
316
|
### [Job retries](https://riverqueue.com/docs/job-retries)
|
|
230
317
|
|
|
@@ -243,12 +330,18 @@ config = River::Config.new(
|
|
|
243
330
|
```
|
|
244
331
|
|
|
245
332
|
Use `client.job_retry(job_id)` to make a non-running job available immediately.
|
|
333
|
+
This clears its previous cancellation request. Once a job has reached 32,767
|
|
334
|
+
attempts, requesting another retry raises `ArgumentError` without changing it.
|
|
246
335
|
|
|
247
336
|
A worker may implement `retry?(job, error)` and return false to discard a
|
|
248
337
|
reported error immediately, without reducing the attempt budget used for crash
|
|
249
338
|
recovery. The Rails integration uses this to let Active Job own application retries.
|
|
339
|
+
Jobs claimed and finished by batch extensions honor the registered worker's
|
|
340
|
+
`retry?` and `next_retry` hooks too, including workers registered as classes.
|
|
250
341
|
If a retry hook or policy raises, River logs the callback error and falls back to
|
|
251
342
|
retrying with the default backoff. The original work error is still recorded.
|
|
343
|
+
Stuck-job rescue also falls back per job when a retry policy raises or returns
|
|
344
|
+
an invalid time, allowing other rescues and cleanup to continue.
|
|
252
345
|
|
|
253
346
|
### [Error handling and timeouts](https://riverqueue.com/docs/error-handling)
|
|
254
347
|
|
|
@@ -310,6 +403,9 @@ config = River::Config.new(queues: {
|
|
|
310
403
|
```
|
|
311
404
|
|
|
312
405
|
Queues may also be added and removed at runtime with `client.queue_add(name, config)` and `client.queue_remove(name)`.
|
|
406
|
+
Queue removal stops fetching and keeps observing cancellation requests until its
|
|
407
|
+
active jobs finish. Removing a queue from one of its own workers raises
|
|
408
|
+
`ThreadError`, because that worker cannot wait for itself to finish.
|
|
313
409
|
|
|
314
410
|
### [Pausing queues](https://riverqueue.com/docs/pausing-queues)
|
|
315
411
|
|
|
@@ -324,6 +420,9 @@ client.queue_resume "*"
|
|
|
324
420
|
```
|
|
325
421
|
|
|
326
422
|
Use `queue_get`, `queue_list`, and `queue_update` to inspect queues and attach metadata.
|
|
423
|
+
Pause and resume events contain a snapshot of each affected queue. Wildcard
|
|
424
|
+
operations publish events for every affected queue; size subscription buffers
|
|
425
|
+
to accommodate the number of events you expect to receive.
|
|
327
426
|
|
|
328
427
|
### Rails and Active Job
|
|
329
428
|
|
|
@@ -340,6 +439,11 @@ the job is due, not during registration. A reusable callable can be passed as
|
|
|
340
439
|
`constructor:` instead; don't supply both. Core periodic schedules live in the
|
|
341
440
|
client process; River Pro adds durable schedules.
|
|
342
441
|
|
|
442
|
+
A started client can produce periodic jobs with `queues: {}` while other clients
|
|
443
|
+
consume them. Jobs added through `client.periodic_jobs` also start maintenance
|
|
444
|
+
when needed. Periodic registration requires leader election to be enabled,
|
|
445
|
+
including jobs registered after startup.
|
|
446
|
+
|
|
343
447
|
```ruby
|
|
344
448
|
cleanup = River::PeriodicJob.new(
|
|
345
449
|
id: :cleanup,
|
|
@@ -360,6 +464,10 @@ registrations and returns the bundle.
|
|
|
360
464
|
`add_many` registers a batch atomically: duplicate IDs or invalid schedules leave
|
|
361
465
|
the registry unchanged.
|
|
362
466
|
|
|
467
|
+
Inserted jobs carry `periodic: true` in metadata and, when the registration has
|
|
468
|
+
an ID, `river:periodic_job_id`. Constructor options and application metadata are
|
|
469
|
+
preserved.
|
|
470
|
+
|
|
363
471
|
Schedules can be callbacks (`schedule: ->(now) { now + 300 }`) or objects
|
|
364
472
|
implementing `next(time)`. They must return a `Time` strictly after the supplied
|
|
365
473
|
time; they do not execute the job themselves. A schedule that raises at runtime
|
|
@@ -376,12 +484,15 @@ cleanup = River::PeriodicJob.new(
|
|
|
376
484
|
|
|
377
485
|
`PeriodicCron` parses once and loads Fugit only when constructed. Fugit is not
|
|
378
486
|
a runtime dependency of the River gem. The timezone defaults explicitly to UTC;
|
|
379
|
-
|
|
380
|
-
optional seconds, and aliases such as `@daily` use Fugit's
|
|
381
|
-
|
|
382
|
-
|
|
383
|
-
|
|
384
|
-
|
|
487
|
+
set `timezone:` or use a `CRON_TZ=` / `TZ=` prefix, which takes precedence.
|
|
488
|
+
Five-field cron, optional seconds, and aliases such as `@daily` use Fugit's
|
|
489
|
+
syntax; `?` is also accepted as a wildcard. `@every` accepts durations such
|
|
490
|
+
as `1h30m`, built from `ns`, `us`, `ms`, `s`, `m`, and `h` units, rounded down
|
|
491
|
+
to whole seconds with a one-second minimum. Unlike `PeriodicInterval`, these intervals align to whole-second boundaries.
|
|
492
|
+
Results are UTC `Time` objects. Nonexistent spring-forward times are skipped;
|
|
493
|
+
both occurrences of a repeated fall-back time are scheduled. See
|
|
494
|
+
[conformance differences](conformance.md) before sharing schedules across
|
|
495
|
+
languages. Cron does not replay missed occurrences after downtime.
|
|
385
496
|
|
|
386
497
|
For a one-time date, insert a job with
|
|
387
498
|
`client.insert(args, scheduled_at: Time.utc(2026, 9, 20, 9))` instead of registering
|
|
@@ -389,7 +500,7 @@ a periodic job. This stores the scheduled job immediately in the database.
|
|
|
389
500
|
|
|
390
501
|
### [Resumable jobs](https://riverqueue.com/docs/resumable-jobs)
|
|
391
502
|
|
|
392
|
-
Long jobs can checkpoint idempotent steps and cursor progress. On retry, River skips completed steps and resumes a cursor step from its last recorded value using
|
|
503
|
+
Long jobs can checkpoint idempotent steps and cursor progress. On retry, River skips completed steps and resumes a cursor step from its last recorded value using River's shared metadata format.
|
|
393
504
|
|
|
394
505
|
```ruby
|
|
395
506
|
def work(job)
|
|
@@ -451,7 +562,7 @@ end
|
|
|
451
562
|
### Job-persisted logging
|
|
452
563
|
|
|
453
564
|
Add `River::JobPersistedLogging::Plugin` to save worker logs with each job attempt,
|
|
454
|
-
using the
|
|
565
|
+
using the shared `metadata["river:log"]` format that River UI displays. This
|
|
455
566
|
is a core feature; it does not require Pro or additional migrations.
|
|
456
567
|
|
|
457
568
|
```ruby
|
|
@@ -527,13 +638,16 @@ A plugin may implement any combination of these methods:
|
|
|
527
638
|
| Hook | `insert_begin(params)` | Before each job is inserted; `params` may be modified. |
|
|
528
639
|
| Hook | `insert_end(result)` | After each job is inserted. |
|
|
529
640
|
| Hook | `work_begin(job)` | After a job is claimed, immediately before its worker runs. |
|
|
530
|
-
| Hook | `work_end(job, error)` | After the worker returns or raises
|
|
641
|
+
| Hook | `work_end(job, error)` | After the worker returns or raises a `StandardError`; returns the attempt's error or `nil`. |
|
|
531
642
|
| Hook | `job_finalize(job, state)` | Before successful finalization; returning `:delete` deletes the job instead of retaining it. |
|
|
532
643
|
| Middleware | `insert_many(params, operation)` | Around one insertion call; `params` is an array even for `Client#insert`. |
|
|
533
644
|
| Middleware | `work(job, operation)` | Around the work hooks and worker for one claimed job. |
|
|
534
645
|
|
|
535
646
|
A single plugin can provide both styles. For example, it might use
|
|
536
647
|
`insert_begin` to add metadata and `work` to time the complete work operation.
|
|
648
|
+
After insertion hooks run, each job must still have an initial state of
|
|
649
|
+
`available`, `pending`, or `scheduled`. Invalid states roll back the batch
|
|
650
|
+
and any database writes made by its hooks.
|
|
537
651
|
|
|
538
652
|
```ruby
|
|
539
653
|
class TimingPlugin
|
|
@@ -551,10 +665,33 @@ end
|
|
|
551
665
|
config = River::Config.new(plugins: [AuditPlugin.new, TimingPlugin.new])
|
|
552
666
|
```
|
|
553
667
|
|
|
554
|
-
Plugins earlier in the list are the outermost wrappers. Begin callbacks
|
|
555
|
-
configuration order, while `insert_end` callbacks
|
|
556
|
-
effect, work execution is nested as: middleware before,
|
|
557
|
-
`work_end`, middleware after.
|
|
668
|
+
Plugins earlier in the list are the outermost wrappers. Begin callbacks and
|
|
669
|
+
`work_end` callbacks run in configuration order, while `insert_end` callbacks
|
|
670
|
+
run in reverse order. In effect, work execution is nested as: middleware before,
|
|
671
|
+
`work_begin`, worker, `work_end`, middleware after.
|
|
672
|
+
|
|
673
|
+
Each `work_end(job, error)` receives `nil` on success or the worker's exception
|
|
674
|
+
on failure. Return `error` to preserve that outcome, a different exception to
|
|
675
|
+
replace it, or `nil` to clear a failure. Each subsequent end hook receives the
|
|
676
|
+
previous hook's return value. After all end hooks run, River raises the final
|
|
677
|
+
error through the middleware, or completes the operation if it is `nil`.
|
|
678
|
+
Returning `River.job_cancel(...)` or `River.job_snooze(...)` selects the
|
|
679
|
+
corresponding cancellation or snooze behavior.
|
|
680
|
+
|
|
681
|
+
An observing hook must explicitly return its incoming error:
|
|
682
|
+
|
|
683
|
+
```ruby
|
|
684
|
+
def work_end(job, error)
|
|
685
|
+
Metrics.record_attempt(job.kind, error: error)
|
|
686
|
+
error
|
|
687
|
+
end
|
|
688
|
+
```
|
|
689
|
+
|
|
690
|
+
If a `work_begin` hook raises, River skips the worker and all end hooks. If an
|
|
691
|
+
end hook raises instead of returning an error, it stops the end-hook chain and
|
|
692
|
+
propagates that exception through the middleware. Fatal exceptions outside
|
|
693
|
+
`StandardError` also bypass end hooks. Use middleware with `ensure` for cleanup
|
|
694
|
+
that must run when hooks or work raise.
|
|
558
695
|
|
|
559
696
|
### [Subscriptions](https://riverqueue.com/docs/subscriptions)
|
|
560
697
|
|
|
@@ -604,7 +741,11 @@ client.job_delete_many states: [:cancelled]
|
|
|
604
741
|
|
|
605
742
|
Reusable `JobListParams` and `JobUpdateParams` objects are also accepted as
|
|
606
743
|
positional arguments, instead of keywords. For updates, omitted fields remain
|
|
607
|
-
unchanged; an explicit `nil` clears a nullable field.
|
|
744
|
+
unchanged; an explicit `nil` clears a nullable field. Metadata must be a Hash;
|
|
745
|
+
use `metadata: {}` to clear it. Values inside the Hash may be `nil`.
|
|
746
|
+
Updated `max_attempts` must be between 1 and 32,767; `attempt` must be between
|
|
747
|
+
0 and 32,767. Claims cap the attempt counter at 32,767 so an administratively
|
|
748
|
+
requeued job cannot prevent other jobs from being claimed.
|
|
608
749
|
|
|
609
750
|
Bulk deletion requires at least one filter and never deletes running jobs.
|
|
610
751
|
Metadata filters compare complete JSON values at each supplied top-level key,
|
|
@@ -647,11 +788,35 @@ config = River::Config.new(
|
|
|
647
788
|
```
|
|
648
789
|
|
|
649
790
|
A custom service implements `run(client, driver, now)` and runs only while this client holds leadership.
|
|
791
|
+
Service errors are logged without interrupting other services, stuck-job rescue,
|
|
792
|
+
or cleanup. A failed service is retried on the next scheduled maintenance pass.
|
|
793
|
+
|
|
794
|
+
Call `client.request_resign` to ask the current maintenance leader to resign.
|
|
795
|
+
The leader releases its lease and briefly delays reelection so another client
|
|
796
|
+
can take over. The request joins the caller's transaction and is delivered only
|
|
797
|
+
on commit.
|
|
650
798
|
|
|
651
799
|
On SQLite, the leader also removes notification outbox entries older than five
|
|
652
|
-
minutes in bounded batches.
|
|
653
|
-
|
|
654
|
-
|
|
800
|
+
minutes in bounded batches.
|
|
801
|
+
|
|
802
|
+
### Notifications
|
|
803
|
+
|
|
804
|
+
Started clients receive job insertion, queue control, cancellation, and
|
|
805
|
+
leadership notifications. Postgres uses one dedicated LISTEN connection per
|
|
806
|
+
started client, separate from the application's connection pool. SQLite polls
|
|
807
|
+
the notification outbox with an independent cursor for each client. Inserts,
|
|
808
|
+
scheduled jobs becoming available, and manual retries wake workers promptly.
|
|
809
|
+
|
|
810
|
+
The receiver starts before workers and stays active during graceful shutdown.
|
|
811
|
+
Startup fails if the initial subscription fails; an established receiver retries
|
|
812
|
+
after connection errors and refreshes persisted state on reconnect. Normal job
|
|
813
|
+
and queue polling also remains active.
|
|
814
|
+
|
|
815
|
+
Set `poll_only: true` in `River::Config` to disable receiving notifications, for
|
|
816
|
+
example when connecting through a proxy that cannot support LISTEN. Job, queue,
|
|
817
|
+
and cancellation changes are then observed at the polling interval, and
|
|
818
|
+
`request_resign` is not received. Backends without native notification support,
|
|
819
|
+
such as Yugabyte with LISTEN/NOTIFY disabled, use polling automatically.
|
|
655
820
|
|
|
656
821
|
### [Renaming job kinds](https://riverqueue.com/docs/renaming-jobs)
|
|
657
822
|
|
|
@@ -682,6 +847,13 @@ The Ruby configuration file must return an unstarted client. The following clien
|
|
|
682
847
|
methods are for applications managing their own runtime lifecycle:
|
|
683
848
|
|
|
684
849
|
`client.stop` stops fetching and waits for active jobs to finish. `client.stop_and_cancel` interrupts active worker threads and returns their jobs to `available` without consuming the interrupted attempt.
|
|
850
|
+
While draining, the client continues receiving and polling for cancellation requests from
|
|
851
|
+
other clients, including for workers with no timeout. Transient polling errors
|
|
852
|
+
are retried until the active attempts finish.
|
|
853
|
+
|
|
854
|
+
Blocking stop calls from the client's own workers or other runtime threads
|
|
855
|
+
raise `ThreadError`. A worker can request shutdown with `client.stop(wait: false)`
|
|
856
|
+
and then finish its work.
|
|
685
857
|
|
|
686
858
|
Use `client.stop(wait: false)` to request stop and return immediately without
|
|
687
859
|
interrupting active workers. Call `client.stop` later to wait for draining and
|
|
@@ -705,6 +877,11 @@ Load River and worker definitions in the main Ractor before spawning others. Eac
|
|
|
705
877
|
|
|
706
878
|
Database drivers, Active Record, Sequel, and optional dependencies such as Fugit still need their own Ractor compatibility. Job timeouts also depend on Ruby and the `timeout` gem: the test suite exercises them on Ruby 4 with `timeout` 0.6.1; tests on older Rubies disable job timeouts. `WorkerRunner` handles process-wide signals and must run on the main thread of the main Ractor.
|
|
707
879
|
|
|
880
|
+
Ruby 3.3 and 3.4 can deadlock while scheduling threads inside Ractors, even
|
|
881
|
+
without River ([Ruby bug #20905](https://bugs.ruby-lang.org/issues/20905), fixed
|
|
882
|
+
in Ruby 4). The test suite skips the producer/maintenance lifecycle test on
|
|
883
|
+
those versions; insertion and synchronous worker checks still run.
|
|
884
|
+
|
|
708
885
|
Ruby 4.0.2 with `timeout` 0.6.1 can intermittently deadlock during VM shutdown
|
|
709
886
|
while terminating Ractors and their timeout helper threads, even after work
|
|
710
887
|
finishes successfully. This reproduces without River. Tests bypass that shutdown
|
|
@@ -714,7 +891,7 @@ limitations are resolved.
|
|
|
714
891
|
|
|
715
892
|
## RBS and type checking
|
|
716
893
|
|
|
717
|
-
The gem bundles [RBS files](https://github.com/riverqueue/
|
|
894
|
+
The gem bundles [RBS files](https://github.com/riverqueue/river/tree/master/ruby/sig) for tools such as [Steep](https://github.com/soutaro/steep) and other RBS-compatible type checkers.
|
|
718
895
|
|
|
719
896
|
## Drivers
|
|
720
897
|
|
|
@@ -748,17 +925,17 @@ Minitest integrations are optional and explicitly loaded.
|
|
|
748
925
|
|
|
749
926
|
River Pro is kept in the separate, privately distributed `riverqueue-pro` gem, so possession of that package is the access boundary. It is not included in the MPL-2.0 core gem. Its implementation and documentation live in the private `riverqueue-ruby-pro` repository; see that repository's README for configuration and feature examples.
|
|
750
927
|
|
|
751
|
-
## Current
|
|
928
|
+
## Current limitations
|
|
752
929
|
|
|
753
|
-
|
|
754
|
-
|
|
755
|
-
|
|
930
|
+
Ruby checks shared fixtures for unique keys, cron, snooze counters, and
|
|
931
|
+
protocol values.
|
|
932
|
+
See [conformance coverage and differences](conformance.md); this is not yet a
|
|
933
|
+
full mixed-language runtime suite.
|
|
756
934
|
|
|
757
935
|
The Ruby client does not currently provide dedicated OpenTelemetry/metrics
|
|
758
936
|
integrations or
|
|
759
937
|
transactional job completion alongside application writes. Plugins and
|
|
760
|
-
subscriptions provide integration points for telemetry.
|
|
761
|
-
worker objects rather than Go's work-function API.
|
|
938
|
+
subscriptions provide integration points for telemetry.
|
|
762
939
|
|
|
763
940
|
## Development
|
|
764
941
|
|
data/docs/conformance.md
ADDED
|
@@ -0,0 +1,72 @@
|
|
|
1
|
+
# Ruby conformance
|
|
2
|
+
|
|
3
|
+
Ruby reads fixtures generated from this checkout's Go implementation, alongside
|
|
4
|
+
Rust and JavaScript. From the repository root:
|
|
5
|
+
|
|
6
|
+
```sh
|
|
7
|
+
make -C ruby install
|
|
8
|
+
make test/ruby/conformance
|
|
9
|
+
```
|
|
10
|
+
|
|
11
|
+
This generates `conformance/testdata/` and runs `ruby/spec/conformance_spec.rb`.
|
|
12
|
+
No Postgres server, pinned upstream checkout, protocol adapter, or separately
|
|
13
|
+
maintained golden files are needed. Missing fixtures fail with instructions to
|
|
14
|
+
regenerate them. The regular Ruby suite includes these checks too.
|
|
15
|
+
|
|
16
|
+
The fixture checks cover:
|
|
17
|
+
|
|
18
|
+
- All unique-key cases, including typed-only cases, raw JSON tokens, selected
|
|
19
|
+
fields, periods, queues, and state masks.
|
|
20
|
+
- Job state names and unique bits, attempt-error encoding/decoding, retry bounds,
|
|
21
|
+
and all six shared metadata keys, including actual output, rescue, periodic,
|
|
22
|
+
unique insertion, and resumable-step persistence.
|
|
23
|
+
- All snooze-counter fixtures through real worker attempts and both SQL drivers.
|
|
24
|
+
- Cron occurrences, named zones, daylight-saving transitions, interval schedules,
|
|
25
|
+
and invalid expressions, with the Ruby API differences below checked explicitly.
|
|
26
|
+
- Insertion, cancellation, queue pause/resume/metadata, and leadership resignation/request
|
|
27
|
+
notification encoding through both SQL drivers, using in-memory SQLite and
|
|
28
|
+
the bundled canonical migrations.
|
|
29
|
+
|
|
30
|
+
The Ruby workflow runs on Ruby changes. The shared Conformance workflow also
|
|
31
|
+
runs these tests when Go code changes. The ordinary driver suites retain
|
|
32
|
+
Postgres notification delivery/commit-ordering tests and shared Postgres /
|
|
33
|
+
SQLite insertion, transaction, uniqueness, and worker tests.
|
|
34
|
+
|
|
35
|
+
This replaces the old `insert-only-v1` adapter and its pinned external harness.
|
|
36
|
+
It checks shared data formats, not live mixed-language worker execution. Adapter
|
|
37
|
+
handshake and request-schema tests were specific to that removed harness; their
|
|
38
|
+
protocol is not part of the Ruby API.
|
|
39
|
+
|
|
40
|
+
## Coverage gaps and API differences
|
|
41
|
+
|
|
42
|
+
Ruby consumes Postgres LISTEN events and the SQLite notification outbox. The
|
|
43
|
+
ordinary driver suites exercise insert wakeups, cancellation while draining,
|
|
44
|
+
queue pause/resume, and leadership handoff on `request_resign` through both
|
|
45
|
+
backends and both SQL drivers. They also check commit ordering, rollback,
|
|
46
|
+
independent listeners, startup failures, and recovery after listener failures.
|
|
47
|
+
The fixture target checks wire formats; live mixed-language worker execution
|
|
48
|
+
remains outside its scope.
|
|
49
|
+
|
|
50
|
+
Shared snooze-counter fixtures cover non-negative integers and absent counters.
|
|
51
|
+
Recovery from other JSON values is implementation-specific. Ruby's shared driver
|
|
52
|
+
tests cover its lenient conversion rules separately on Postgres and SQLite.
|
|
53
|
+
|
|
54
|
+
Cron retains Ruby's existing Fugit extensions: six fields with seconds, Sunday
|
|
55
|
+
as 7, hour 24, and descending ranges. These four expressions are accepted in
|
|
56
|
+
Ruby and rejected in Go; the fixture suite explicitly asserts the difference.
|
|
57
|
+
Fugit rejects impossible calendar dates at construction instead of returning
|
|
58
|
+
Go's zero time. Ruby's default calendar zone remains UTC; tests explicitly pass
|
|
59
|
+
the reference time's fixed offset to model Go's time-location API. `CRON_TZ=` /
|
|
60
|
+
`TZ=` prefixes, `?`, and `@every` duration rounding now follow Go for the shared
|
|
61
|
+
fixtures. No new dependencies were added.
|
|
62
|
+
|
|
63
|
+
Selected unique fields use `by_args: [:field, [:nested, :field]]`, rather than Go
|
|
64
|
+
struct tags. Dotted string keys are literal. Cross-language unique keys require
|
|
65
|
+
the same encoded argument values, including escaping, numeric tokens, and
|
|
66
|
+
nested ordering; Ruby does not override the application's JSON serialization.
|
|
67
|
+
|
|
68
|
+
Transactions are connection-scoped blocks. Workers use exceptions and thread
|
|
69
|
+
interruption. List cursors encode the selected field explicitly. Historical
|
|
70
|
+
error timestamps accept Ruby's `Time.parse` formats, and metadata must decode
|
|
71
|
+
to an object. Live interoperability across these boundaries is not covered by
|
|
72
|
+
the fixture target.
|
data/docs/migrations.md
CHANGED
|
@@ -1,8 +1,8 @@
|
|
|
1
1
|
# Schema migrations
|
|
2
2
|
|
|
3
|
-
River Ruby includes a migration API and the `river` command.
|
|
4
|
-
|
|
5
|
-
|
|
3
|
+
River Ruby includes a migration API and the `river` command. The gems bundle
|
|
4
|
+
byte-for-byte copies of River's canonical Postgres and SQLite migration SQL,
|
|
5
|
+
pinned to release v0.48.0 through migration 008;
|
|
6
6
|
only the upstream schema template placeholders are substituted when executing
|
|
7
7
|
them.
|
|
8
8
|
|
|
@@ -24,7 +24,7 @@ bundle exec river migrate-up --database-url sqlite://storage/river.sqlite3
|
|
|
24
24
|
|
|
25
25
|
`DATABASE_URL` supplies the URL when `--database-url` is omitted. ActiveRecord
|
|
26
26
|
uses `sqlite3:` URLs instead of Sequel's `sqlite:` URLs. To target an existing
|
|
27
|
-
|
|
27
|
+
Postgres schema, pass `--schema jobs`; otherwise the connection's current
|
|
28
28
|
schema is used. Schema names must be simple SQL identifiers. SQLite always
|
|
29
29
|
uses its main schema.
|
|
30
30
|
|
|
@@ -52,22 +52,22 @@ migrator.migrate(dry_run: true, target: 8)
|
|
|
52
52
|
migrator.migrate(direction: :down) # One version; no interactive confirmation.
|
|
53
53
|
```
|
|
54
54
|
|
|
55
|
-
The API accepts either driver and optionally `schema:` for
|
|
55
|
+
The API accepts either driver and optionally `schema:` for Postgres. `migrate`
|
|
56
56
|
returns the migration records applied (or planned in dry-run mode), with their
|
|
57
57
|
version, name, and original up/down SQL. Each version's SQL and history update
|
|
58
58
|
commit in their own transaction. A failing version rolls back while earlier
|
|
59
59
|
versions remain committed, so rerunning resumes safely. Do not wrap the migrator
|
|
60
|
-
in an application transaction: some
|
|
60
|
+
in an application transaction: some Postgres changes require a real commit
|
|
61
61
|
before the next version can run.
|
|
62
62
|
|
|
63
63
|
An up migration targeting an already-applied version does nothing, even when
|
|
64
64
|
newer migrations are pending.
|
|
65
65
|
|
|
66
|
-
##
|
|
66
|
+
## Compatibility and Pro
|
|
67
67
|
|
|
68
|
-
The migrator reads and writes
|
|
69
|
-
legacy pre-version-5 format.
|
|
70
|
-
applied
|
|
68
|
+
The migrator reads and writes the `river_migration` history shared by all River
|
|
69
|
+
clients, including the legacy pre-version-5 format. Any River client can
|
|
70
|
+
continue from versions applied by another. Unknown newer versions and gaps in history cause an error
|
|
71
71
|
rather than guessing what SQL to run. It does not baseline an existing schema
|
|
72
72
|
that has no River migration history.
|
|
73
73
|
|
|
@@ -75,7 +75,7 @@ Pro SQL is distributed only inside `riverqueue-pro`, not the public core gem.
|
|
|
75
75
|
SQLite main migration 008 rebuilds `river_job` with `AUTOINCREMENT` so deleted
|
|
76
76
|
job IDs are not reused. Its upgrade and downgrade refuse to run while Pro's
|
|
77
77
|
workflow/sequence schema is installed; coordinate that upgrade with Pro before
|
|
78
|
-
running it on an existing Pro database.
|
|
78
|
+
running it on an existing Pro database. Postgres migration 008 is a no-op.
|
|
79
79
|
|
|
80
80
|
Migrate main first, then Pro:
|
|
81
81
|
|
|
@@ -92,30 +92,27 @@ River::Pro::Migrator.new(driver).migrate
|
|
|
92
92
|
```
|
|
93
93
|
|
|
94
94
|
Remove other migration lines before downgrading main. Run only one migration
|
|
95
|
-
process at a time across languages. Ruby migrators take a
|
|
95
|
+
process at a time across languages. Ruby migrators take a Postgres advisory
|
|
96
96
|
lock per schema; SQLite takes a write lock per version and checks for concurrent
|
|
97
|
-
history changes. These are not shared
|
|
97
|
+
history changes. These locks are not shared with other River clients' migration runners.
|
|
98
98
|
|
|
99
99
|
## Updating the bundled SQL
|
|
100
100
|
|
|
101
|
-
`migration/manifest.json` records
|
|
102
|
-
|
|
101
|
+
`migration/manifest.json` records each canonical file's SHA-256. From `ruby/`,
|
|
102
|
+
synchronize or verify against the canonical migrations in this same checkout:
|
|
103
103
|
|
|
104
104
|
```sh
|
|
105
|
-
ruby scripts/sync_migrations.rb
|
|
105
|
+
ruby scripts/sync_migrations.rb
|
|
106
106
|
make verify
|
|
107
|
-
make verify RIVER_PATH=/path/to/river
|
|
108
107
|
```
|
|
109
108
|
|
|
110
|
-
|
|
111
|
-
|
|
112
|
-
River repository at that recorded revision and runs the same check; it does not
|
|
113
|
-
require a sibling checkout or access to the private Pro repository.
|
|
109
|
+
Verification checks SQL contents, filenames, license, and manifest checksums.
|
|
110
|
+
CI uses the same working-tree comparison; no separate checkout is needed.
|
|
114
111
|
|
|
115
112
|
Pro migrations have their own manifest and sync script in the private
|
|
116
113
|
`riverqueue-ruby-pro` repository. Run that repository's `make verify` against
|
|
117
|
-
the
|
|
114
|
+
the River Pro checkout; the public sync script handles only the main migrations.
|
|
118
115
|
|
|
119
116
|
Review the upstream changes and run the complete test matrix before publishing.
|
|
120
117
|
Do not edit the copied SQL independently. Upstream changes are bundled in gem
|
|
121
|
-
releases; users do not need the
|
|
118
|
+
releases; users do not need the upstream repositories or network access at runtime.
|
data/docs/testing.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Testing River jobs
|
|
2
2
|
|
|
3
|
-
Testing helpers ship in `riverqueue`. They use your real
|
|
3
|
+
Testing helpers ship in `riverqueue`. They use your real Postgres or SQLite
|
|
4
4
|
database through either driver; there is no fake queue, global testing mode,
|
|
5
5
|
implicit migration, or automatic cleanup. Require them explicitly:
|
|
6
6
|
|