sidekiq-batch-jobs 0.3.1 → 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +72 -0
- data/README.md +117 -19
- data/app/models/sidekiq_batch/announcement.rb +3 -1
- data/app/models/sidekiq_batch/callback_target.rb +111 -0
- data/app/models/sidekiq_batch.rb +7 -3
- data/app/models/sidekiq_batch_job.rb +2 -3
- data/lib/sidekiq/batch/jobs/rspec.rb +46 -0
- data/lib/sidekiq/batch/jobs/version.rb +1 -1
- data/lib/sidekiq/batch/jobs.rb +0 -1
- metadata +8 -7
- data/lib/sidekiq/batch/jobs/enum_compat.rb +0 -29
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: c2fb395443c88a5e18c67c1689b97710f137447909fcb7892ece8f178c235ce8
|
|
4
|
+
data.tar.gz: 8554f4ca87f6785a6d61cb8170af63621782c5aeb9d3e92ae2cfbc9e231ea43b
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: d2e5630d1411ec8f692a8e1f1cb386b4c1e0d3ba8f3d04b13414f06eeeaf28fb8128bea6dc81d40286afcc7c0ac6a1d4ac0bc6a70441f9ad4b44025dcc61788f
|
|
7
|
+
data.tar.gz: 3fa57b6c628c226b258580f2b60a3d68c53088153a9efc7410c74b0c90619470359f0526c7101f5ea1384efa6421f76bf3c0ad0949c53a9453f429b84c785220
|
data/CHANGELOG.md
CHANGED
|
@@ -5,6 +5,78 @@ All notable changes to this project are documented here.
|
|
|
5
5
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
|
6
6
|
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
7
7
|
|
|
8
|
+
## [0.4.0] - 2026-09-22
|
|
9
|
+
|
|
10
|
+
A support-window release. No behaviour changes, no schema change, and no API change beyond one
|
|
11
|
+
shim that the new floor makes dead.
|
|
12
|
+
|
|
13
|
+
### Breaking
|
|
14
|
+
|
|
15
|
+
- **Rails 6.1, 7.0 and 7.1 are no longer supported.** The gemspec now requires
|
|
16
|
+
`activerecord >= 7.2, < 9`. Nothing in the gem stopped working on those versions, they are
|
|
17
|
+
simply no longer tested against, and with 8.1 shipped they are all past Rails' own security
|
|
18
|
+
support. Stay on 0.3.1 if you need them; it is unaffected by this release.
|
|
19
|
+
- **Ruby 3.0 is out, and the floor is 3.1.** Not our choice to make: Rails 7.2 requires
|
|
20
|
+
Ruby >= 3.1, so supporting one sets the other.
|
|
21
|
+
- **`Sidekiq::Batch::Jobs::EnumCompat` is gone.** It existed only to bridge Rails 6.1's
|
|
22
|
+
hash-form `enum` against the positional form 7.0 introduced, and with 7.2 as the floor every
|
|
23
|
+
supported Rails takes the same branch. The models now call `enum` directly. Nothing
|
|
24
|
+
downstream changes, because the generated surface is identical: `.statuses`, the
|
|
25
|
+
`pending_status?` predicates and the `pending_status` scopes are all still there. Only a host
|
|
26
|
+
that referenced the module by name is affected, which it had no reason to.
|
|
27
|
+
|
|
28
|
+
### Added
|
|
29
|
+
|
|
30
|
+
- A `rails-8.1` lane, so the newest Rails series is actually covered rather than merely
|
|
31
|
+
permitted by the `< 9` ceiling.
|
|
32
|
+
- **Callbacks may now be ActiveJob classes.** `batch.on(:complete, SomeApplicationJob)` works
|
|
33
|
+
alongside the Sidekiq workers that were previously the only option. In an application that is
|
|
34
|
+
otherwise all ActiveJob, reaching for `ApplicationJob` was the natural move and the one thing
|
|
35
|
+
that did not work, silently: the announcement raised `NoMethodError` on `perform_async`, which
|
|
36
|
+
was swallowed into an alert that then repeated on every reaper run until grooming.
|
|
37
|
+
|
|
38
|
+
The payload is built and pushed directly rather than by calling `perform_later`, because
|
|
39
|
+
Sidekiq's ActiveJob adapter declares `enqueue_after_transaction_commit?` and would defer the
|
|
40
|
+
push until after the claim had committed. That would leave a claimed event with nothing sent
|
|
41
|
+
and nothing to retry it, since a spent claim is what stops the orphaned-callback reaper
|
|
42
|
+
looking again. The cost is that ActiveJob's enqueue callbacks do not run for batch callbacks.
|
|
43
|
+
`perform`, `around_perform` and `retry_on` are unaffected.
|
|
44
|
+
- `require "sidekiq/batch/jobs/rspec"`, which is the test-suite wiring every host was writing by
|
|
45
|
+
hand. It sets the enrolment transaction baseline that transactional fixtures otherwise trip,
|
|
46
|
+
and registers the server middleware that `Sidekiq::Testing` never gets from `install!`. The
|
|
47
|
+
second is the one worth shipping: without it inline jobs run, nothing marks their rows
|
|
48
|
+
complete, and batches never finish, with nothing raised to say why. The gem's own suite now
|
|
49
|
+
uses this file rather than its own copy, so it cannot rot unnoticed.
|
|
50
|
+
- `SidekiqBatch#on` rejects a class that can be neither `perform_async`ed nor `perform_later`ed,
|
|
51
|
+
naming both interfaces. Registration is where that is cheap to catch.
|
|
52
|
+
|
|
53
|
+
### Changed
|
|
54
|
+
|
|
55
|
+
- The lanes are now `rails-7.2` (Ruby 3.1.7, Sidekiq 7.3), `rails-8.0` and `rails-8.1` (both
|
|
56
|
+
Ruby 3.4.6, Sidekiq 8). `rails-7.2` carries the most weight: it is the only lane on Ruby 3.1
|
|
57
|
+
and the only one on Sidekiq 7, so it alone proves the bottom of both declared ranges.
|
|
58
|
+
- The development image moved from Ruby 3.0.7 to 3.1.7, which also moves its base from Debian
|
|
59
|
+
bullseye to bookworm. Worth knowing because the bullseye apt mirror had begun 404ing on
|
|
60
|
+
`git`, which left `bin/setup` and `bin/test` unable to build an image at all. Both work
|
|
61
|
+
again.
|
|
62
|
+
- `rake audit` now hard-fails on both Rails 8 lanes, where before only one lane was held to
|
|
63
|
+
that. `rails-7.2` still soft-fails, for a reason that has nothing to do with Rails: it runs
|
|
64
|
+
Ruby 3.1, which caps nokogiri at 1.18.10 because 1.19 requires 3.2, and those advisories have
|
|
65
|
+
no version to move to. nokogiri is test-only, reaching the lockfile through actionview, so
|
|
66
|
+
the gem declares no runtime dependency on it and nothing reaches a host application.
|
|
67
|
+
- Development dependencies now cap `json` below 3.0. The September 2026 release dropped the
|
|
68
|
+
`quirks_mode` keyword from `generate` and changed `parse`'s arity, which breaks every Rails
|
|
69
|
+
series this gem supports. Deliberately in the Gemfile rather than the gemspec: it is Rails'
|
|
70
|
+
incompatibility to resolve, and capping a host application's json for them would be
|
|
71
|
+
overreach.
|
|
72
|
+
|
|
73
|
+
### Upgrading from 0.3.1
|
|
74
|
+
|
|
75
|
+
Nothing to run. On Rails 7.2+ and Ruby 3.1+, `bundle update sidekiq-batch-jobs` is the whole
|
|
76
|
+
upgrade: no migration, no configuration change, no API change.
|
|
77
|
+
|
|
78
|
+
On Rails 7.1 or older, Bundler will simply decline the upgrade and leave you on 0.3.1.
|
|
79
|
+
|
|
8
80
|
## [0.3.1] - 2026-09-08
|
|
9
81
|
|
|
10
82
|
### Fixed
|
data/README.md
CHANGED
|
@@ -4,7 +4,7 @@ Batch tracking and completion callbacks for Sidekiq, backed by ActiveRecord (Pos
|
|
|
4
4
|
|
|
5
5
|
A hand-rolled alternative to Sidekiq Pro batches. Group a set of `perform_async` calls into a batch, persist their state in the database, and fire callback workers when the batch finishes. You decide what counts as a failure: one bad job, all of them, or some tolerance in between.
|
|
6
6
|
|
|
7
|
-
Tested against Ruby 3.
|
|
7
|
+
Tested against Ruby 3.1–3.4, Rails 7.2–8.1, Sidekiq 7–8.
|
|
8
8
|
|
|
9
9
|
## Contents
|
|
10
10
|
|
|
@@ -215,11 +215,61 @@ calls. Usually that is merely surprising. `retry_on` is worse: ActiveJob catches
|
|
|
215
215
|
itself and queues a *new* job, so Sidekiq only ever sees the original succeed. The batch marks
|
|
216
216
|
that row complete and can report success while the work is still failing and retrying.
|
|
217
217
|
|
|
218
|
+
If you do enrol an ActiveJob, control its retries with `sidekiq_options` rather than
|
|
219
|
+
`retry_on`, and the batch reads them correctly:
|
|
220
|
+
|
|
221
|
+
```ruby
|
|
222
|
+
class RescoreJob < ApplicationJob
|
|
223
|
+
# Read by Sidekiq, not by ActiveJob.
|
|
224
|
+
sidekiq_options retry: 5
|
|
225
|
+
|
|
226
|
+
def perform(entry_id)
|
|
227
|
+
Entry.find(entry_id).rescore!
|
|
228
|
+
end
|
|
229
|
+
end
|
|
230
|
+
|
|
231
|
+
batch.jobs { RescoreJob.perform_later(entry.id) }
|
|
232
|
+
```
|
|
233
|
+
|
|
234
|
+
The middleware decides whether a failure is a job's last attempt from `job["retry"]` in the
|
|
235
|
+
Sidekiq payload, and Sidekiq's client merges the wrapped ActiveJob class's `sidekiq_options`
|
|
236
|
+
into that payload as it is pushed. So `retry: 5` arrives as the Integer `5`, exactly as it
|
|
237
|
+
would for a plain worker, and the row is only marked failed on the final attempt. `retry: false`
|
|
238
|
+
arrives as `false` and means "no retries, so this attempt is the last one". Omitting
|
|
239
|
+
`sidekiq_options` entirely gives `true`, which falls back to Sidekiq's configured `max_retries`
|
|
240
|
+
(25 by default). All three are handled, and this holds across every Sidekiq version the gem
|
|
241
|
+
supports.
|
|
242
|
+
|
|
218
243
|
Beyond that you don't have to think about jid tracking, race conditions, or middleware
|
|
219
244
|
ordering. You just write the block.
|
|
220
245
|
|
|
221
246
|
### What a callback worker receives
|
|
222
247
|
|
|
248
|
+
**A callback can be a Sidekiq worker or an ActiveJob class.** Either is fine, and an
|
|
249
|
+
`ApplicationJob` subclass works exactly as you would expect:
|
|
250
|
+
|
|
251
|
+
```ruby
|
|
252
|
+
batch.on(:complete, RebuildLeaderboardCacheWorker) # include Sidekiq::Job
|
|
253
|
+
batch.on(:failure, AlertOpsJob) # < ApplicationJob
|
|
254
|
+
```
|
|
255
|
+
|
|
256
|
+
Anything else is refused where you register it, rather than failing later:
|
|
257
|
+
|
|
258
|
+
```ruby
|
|
259
|
+
batch.on(:complete, SomePlainClass)
|
|
260
|
+
# => ArgumentError: SomePlainClass cannot be a batch callback: it answers to neither
|
|
261
|
+
# `perform_async` (from `Sidekiq::Job`) nor `perform_later` (from `ActiveJob::Base`) ...
|
|
262
|
+
```
|
|
263
|
+
|
|
264
|
+
One caveat on ActiveJob callbacks. The gem builds the Sidekiq payload and pushes it itself
|
|
265
|
+
rather than calling `perform_later`, because Sidekiq's ActiveJob adapter defers a
|
|
266
|
+
`perform_later` until after the surrounding transaction commits, and the announcement has to
|
|
267
|
+
enqueue inside the transaction that claims the event. Deferring it would let the claim commit
|
|
268
|
+
with nothing sent, which nothing retries, because a spent claim is exactly what stops the
|
|
269
|
+
reaper looking again. The consequence for you is that ActiveJob's **enqueue** callbacks
|
|
270
|
+
(`before_enqueue` and friends) do not run for a batch callback. Everything after that point,
|
|
271
|
+
including `perform`, `around_perform` and `retry_on`, is untouched.
|
|
272
|
+
|
|
223
273
|
Every callback gets one argument: the batch id. From there it can inspect the batch's full state:
|
|
224
274
|
|
|
225
275
|
```ruby
|
|
@@ -387,6 +437,50 @@ different from the event sitting right beside it.
|
|
|
387
437
|
Batches do not finish by themselves in a test suite, and it is worth knowing why before you go
|
|
388
438
|
looking for the bug.
|
|
389
439
|
|
|
440
|
+
**Start with the shipped wiring.** One require covers both of the things below that every
|
|
441
|
+
suite needs:
|
|
442
|
+
|
|
443
|
+
```ruby
|
|
444
|
+
# spec/rails_helper.rb
|
|
445
|
+
require "sidekiq/batch/jobs/rspec"
|
|
446
|
+
```
|
|
447
|
+
|
|
448
|
+
That sets the enrolment transaction baseline around each example, and registers the server
|
|
449
|
+
middleware that `Sidekiq::Testing` does not otherwise get. Both are explained below, because
|
|
450
|
+
they are worth understanding and doing by hand is perfectly fine.
|
|
451
|
+
|
|
452
|
+
**Transactional fixtures look exactly like a caller-opened transaction.** Rails turns
|
|
453
|
+
`use_transactional_fixtures` on by default, which wraps every example in a transaction that is
|
|
454
|
+
never committed. `jobs {}` refuses to run inside a transaction the caller opened, so in a
|
|
455
|
+
standard suite every single call raises `TransactionError` before it enrols anything.
|
|
456
|
+
|
|
457
|
+
Tell the guard which depth counts as "no transaction" rather than turning it off:
|
|
458
|
+
|
|
459
|
+
```ruby
|
|
460
|
+
# spec/rails_helper.rb
|
|
461
|
+
RSpec.configure do |config|
|
|
462
|
+
config.before do
|
|
463
|
+
Thread.current[SidekiqBatch::BatchEnrollmentContext::TXN_BASELINE] =
|
|
464
|
+
ActiveRecord::Base.connection.open_transactions
|
|
465
|
+
end
|
|
466
|
+
|
|
467
|
+
config.after do
|
|
468
|
+
Thread.current[SidekiqBatch::BatchEnrollmentContext::TXN_BASELINE] = nil
|
|
469
|
+
end
|
|
470
|
+
end
|
|
471
|
+
```
|
|
472
|
+
|
|
473
|
+
Recording the depth keeps the guard doing its job: a transaction your test opens *on top of*
|
|
474
|
+
the fixture one still raises, which is the case actually worth catching. Clearing it afterwards
|
|
475
|
+
stops a stale baseline from hiding a real transaction in a later example. This gem's own suite
|
|
476
|
+
does exactly this, in `spec/support/sidekiq_batch.rb`. Minitest wraps examples the same way, so
|
|
477
|
+
the same pair works from `setup` and `teardown`.
|
|
478
|
+
|
|
479
|
+
The guard is not arbitrary: enrolment rows have to be committed before their jobs reach Redis,
|
|
480
|
+
or a worker can start before its row is visible, and a rollback can discard the row for a job
|
|
481
|
+
that is already running. Disabling the check instead of baselining it gives up that guarantee
|
|
482
|
+
in the suite that is meant to be proving it.
|
|
483
|
+
|
|
390
484
|
**In fake mode** (Sidekiq's default) pushed jobs go into an array instead of running, so the
|
|
391
485
|
enrollment rows stay `pending` and the batch stays `running`. That is usually what you want:
|
|
392
486
|
assert on `batch.total_jobs` and the rows, then drive the outcome yourself.
|
|
@@ -402,12 +496,17 @@ batch.attempt_completion! # now it is `succeeded` and the callbacks a
|
|
|
402
496
|
```
|
|
403
497
|
|
|
404
498
|
**In inline mode** jobs run at push time, but Sidekiq's inline path uses its own middleware
|
|
405
|
-
chain, which is empty by default.
|
|
499
|
+
chain, which is empty by default. `install!` will not fill it for you, because it only adds
|
|
500
|
+
the server middleware when `Sidekiq.server?` is true, which never happens under RSpec. Add it
|
|
501
|
+
yourself, or take the require above, which does this:
|
|
406
502
|
|
|
407
503
|
```ruby
|
|
408
504
|
Sidekiq::Testing.server_middleware { |chain| chain.add(SidekiqBatch::Middleware) }
|
|
409
505
|
```
|
|
410
506
|
|
|
507
|
+
Without it nothing raises. Inline jobs run, no row is ever marked complete, and batches simply
|
|
508
|
+
never finish, so callbacks never fire and the suite quietly proves nothing.
|
|
509
|
+
|
|
411
510
|
Even then, an inline job runs *while the block is still open*, so anything it enqueues joins
|
|
412
511
|
the batch. Fake mode avoids that entirely, which is why this gem's own suite uses it.
|
|
413
512
|
|
|
@@ -545,14 +644,14 @@ Run `bin/setup` again whenever the `Dockerfile`, the `Gemfile` or `Appraisals` c
|
|
|
545
644
|
### Running things
|
|
546
645
|
|
|
547
646
|
```bash
|
|
548
|
-
bin/test # rails-
|
|
647
|
+
bin/test # rails-7.2, the default lane
|
|
549
648
|
bin/test spec/models # arguments pass through to rspec
|
|
550
649
|
bin/test spec/models/sidekiq_batch_spec.rb:42 # including a single example
|
|
551
|
-
bin/test --lane rails-8.
|
|
650
|
+
bin/test --lane rails-8.1 # one specific lane
|
|
552
651
|
bin/test --all # every lane, in order
|
|
553
652
|
|
|
554
653
|
bin/shell # interactive shell in the default lane
|
|
555
|
-
bin/shell rails-8.
|
|
654
|
+
bin/shell rails-8.1 # or in another one
|
|
556
655
|
```
|
|
557
656
|
|
|
558
657
|
Rake tasks run inside a lane rather than on your host, so reach them through `bin/shell`:
|
|
@@ -565,28 +664,27 @@ bundle exec rake audit # this lane's lockfile against the ruby-advisory-db
|
|
|
565
664
|
|
|
566
665
|
`audit` sits outside the default task because it needs network access.
|
|
567
666
|
|
|
568
|
-
> **Expect advisories on
|
|
569
|
-
>
|
|
570
|
-
>
|
|
571
|
-
>
|
|
572
|
-
>
|
|
667
|
+
> **Expect nokogiri advisories on `rails-7.2`.** That lane runs Ruby 3.1, and nokogiri 1.19
|
|
668
|
+
> requires 3.2, so it is held at 1.18.10 with advisories that have no version to move to.
|
|
669
|
+
> nokogiri arrives through actionview and is test-only: the gem declares no runtime dependency
|
|
670
|
+
> on it, so none of this reaches a host application. CI audits every lane but only *fails* on
|
|
671
|
+
> `rails-8.0` and `rails-8.1`, both of which are clean.
|
|
573
672
|
|
|
574
673
|
### The three lanes
|
|
575
674
|
|
|
576
675
|
| Lane | Ruby | Rails | Sidekiq | Why it exists |
|
|
577
676
|
| --- | --- | --- | --- | --- |
|
|
578
|
-
| `rails-
|
|
579
|
-
| `rails-
|
|
580
|
-
| `rails-8.
|
|
677
|
+
| `rails-7.2` | 3.1.7 | 7.2 | 7.3 | The floor: oldest supported Rails, Ruby and Sidekiq |
|
|
678
|
+
| `rails-8.0` | 3.4.6 | 8.0 | 8.0 | ActiveRecord 8 removed the hash form of `enum` |
|
|
679
|
+
| `rails-8.1` | 3.4.6 | 8.1 | 8.1 | The newest supported pairing, and the `< 9` ceiling |
|
|
581
680
|
|
|
582
|
-
`rails-
|
|
583
|
-
|
|
584
|
-
positional one.
|
|
681
|
+
`rails-7.2` is the one carrying the most weight. It is the only lane running Ruby 3.1 and the
|
|
682
|
+
only lane running Sidekiq 7, so it alone proves the bottom of both declared ranges.
|
|
585
683
|
|
|
586
684
|
**Two images, and Ruby is why.** Appraisal varies gem versions; it cannot vary Ruby. Rails 8
|
|
587
|
-
and Sidekiq 8 both require Ruby >= 3.2, so
|
|
588
|
-
|
|
589
|
-
|
|
685
|
+
and Sidekiq 8 both require Ruby >= 3.2, so neither 8 lane can run on the Ruby 3.1.7 image that
|
|
686
|
+
`rails-7.2` needs. `docker-compose.yml` pairs each lane with a Ruby that can run it, and CI
|
|
687
|
+
mirrors that pairing.
|
|
590
688
|
|
|
591
689
|
Postgres and Redis come up through compose and are gated on healthchecks, so there is no wait
|
|
592
690
|
loop to care about. Postgres data is on tmpfs, so `docker compose down` any time. The suite
|
|
@@ -78,7 +78,9 @@ class SidekiqBatch
|
|
|
78
78
|
SidekiqBatch.transaction do
|
|
79
79
|
next unless claim!(event)
|
|
80
80
|
|
|
81
|
-
|
|
81
|
+
# Inside the transaction on purpose: see CallbackTarget for why an
|
|
82
|
+
# ActiveJob callback cannot be sent with `perform_later`.
|
|
83
|
+
CallbackTarget.enqueue(worker, batch.id)
|
|
82
84
|
|
|
83
85
|
fired = Fired.new(event: event, job_class: job_class_name)
|
|
84
86
|
end
|
|
@@ -0,0 +1,111 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
class SidekiqBatch
|
|
4
|
+
# Turns a registered callback class into an enqueued job.
|
|
5
|
+
#
|
|
6
|
+
# Two shapes are accepted, told apart by the enqueue method each answers to:
|
|
7
|
+
# including `Sidekiq::Job` defines `perform_async` and never `perform_later`,
|
|
8
|
+
# while `ActiveJob::Base` defines exactly the reverse. Nothing here inspects
|
|
9
|
+
# ancestors, so a class that provides either interface some other way works
|
|
10
|
+
# too.
|
|
11
|
+
module CallbackTarget
|
|
12
|
+
class << self
|
|
13
|
+
# Whether {.enqueue} could send this class. Used by `SidekiqBatch#on` so a
|
|
14
|
+
# class that can never be delivered is rejected where it is registered.
|
|
15
|
+
def supported?(job_class)
|
|
16
|
+
sidekiq_job?(job_class) || active_job?(job_class)
|
|
17
|
+
end
|
|
18
|
+
|
|
19
|
+
# Runs inside the transaction that claims the event, which is the whole
|
|
20
|
+
# constraint on this method: neither branch may defer its push past that
|
|
21
|
+
# commit. See {.push_active_job}.
|
|
22
|
+
def enqueue(job_class, batch_id)
|
|
23
|
+
return job_class.perform_async(batch_id) if sidekiq_job?(job_class)
|
|
24
|
+
|
|
25
|
+
push_active_job(job_class, batch_id)
|
|
26
|
+
end
|
|
27
|
+
|
|
28
|
+
# Registration is where an undeliverable callback is cheap to catch. Left
|
|
29
|
+
# to the announcement it costs a great deal more: the push raises, the
|
|
30
|
+
# claim rolls back, `fire` swallows it into an alert, and the unclaimed
|
|
31
|
+
# event keeps the batch in the orphaned-callback scope, alerting again on
|
|
32
|
+
# every reaper run until grooming deletes it.
|
|
33
|
+
def validate!(job_class)
|
|
34
|
+
klass = job_class.is_a?(Module) ? job_class : job_class.to_s.safe_constantize
|
|
35
|
+
|
|
36
|
+
# A name that does not resolve here is left alone rather than guessed
|
|
37
|
+
# at: in a Rails application it may simply not be autoloaded yet, and
|
|
38
|
+
# the announcement resolves it again when it fires.
|
|
39
|
+
return if klass.nil?
|
|
40
|
+
return if supported?(klass)
|
|
41
|
+
|
|
42
|
+
raise ArgumentError, unsupported_message(klass)
|
|
43
|
+
end
|
|
44
|
+
|
|
45
|
+
def unsupported_message(job_class)
|
|
46
|
+
if job_class.respond_to?(:perform_later)
|
|
47
|
+
"#{job_class} is an ActiveJob class, but Sidekiq's ActiveJob adapter is not loaded, " \
|
|
48
|
+
"so there is no wrapper to enqueue it through. Set " \
|
|
49
|
+
"`config.active_job.queue_adapter = :sidekiq`, or register a class that includes " \
|
|
50
|
+
"`Sidekiq::Job` instead."
|
|
51
|
+
else
|
|
52
|
+
"#{job_class} cannot be a batch callback: it answers to neither `perform_async` " \
|
|
53
|
+
"(from `Sidekiq::Job`) nor `perform_later` (from `ActiveJob::Base`), so there is no " \
|
|
54
|
+
"way to enqueue it. Callbacks are enqueued by this gem, not called inline."
|
|
55
|
+
end
|
|
56
|
+
end
|
|
57
|
+
|
|
58
|
+
private
|
|
59
|
+
|
|
60
|
+
def sidekiq_job?(job_class)
|
|
61
|
+
job_class.respond_to?(:perform_async)
|
|
62
|
+
end
|
|
63
|
+
|
|
64
|
+
# The adapter is what supplies the wrapper the payload is pushed as, so
|
|
65
|
+
# without it an ActiveJob cannot be delivered through Sidekiq at all.
|
|
66
|
+
def active_job?(job_class)
|
|
67
|
+
job_class.respond_to?(:perform_later) && !active_job_wrapper.nil?
|
|
68
|
+
end
|
|
69
|
+
|
|
70
|
+
# The wrapper the host's own adapter would use, rather than a second one
|
|
71
|
+
# that merely happens to work. Rails ships a SidekiqAdapter of its own,
|
|
72
|
+
# and Sidekiq replaces it from 8.0, aliasing `JobWrapper` to its version.
|
|
73
|
+
# On Rails 7.2 with Sidekiq 7.3 the two are different classes, so reading
|
|
74
|
+
# it through the adapter is what keeps the payload identical to the one
|
|
75
|
+
# `perform_later` would have pushed on whichever pairing the host runs.
|
|
76
|
+
def active_job_wrapper
|
|
77
|
+
if defined?(::ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper)
|
|
78
|
+
::ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper
|
|
79
|
+
elsif defined?(::Sidekiq::ActiveJob::Wrapper)
|
|
80
|
+
::Sidekiq::ActiveJob::Wrapper
|
|
81
|
+
end
|
|
82
|
+
end
|
|
83
|
+
|
|
84
|
+
# Deliberately not `perform_later`.
|
|
85
|
+
#
|
|
86
|
+
# Sidekiq's ActiveJob adapter declares `enqueue_after_transaction_commit?`,
|
|
87
|
+
# so `perform_later` hands the push to an after-commit hook rather than
|
|
88
|
+
# running it here. The claim would then commit first, and a crash in that
|
|
89
|
+
# window leaves the event claimed with nothing sent. Nothing recovers
|
|
90
|
+
# that: a spent claim is exactly what stops the orphaned-callback reaper
|
|
91
|
+
# looking at the batch again. Pushing the wrapper payload ourselves keeps
|
|
92
|
+
# the enqueue inside the claim transaction, where the existing guarantee
|
|
93
|
+
# holds, which is that a rollback un-claims the event and the reaper
|
|
94
|
+
# retries it.
|
|
95
|
+
#
|
|
96
|
+
# The payload is the one the adapter builds. The cost of not going through
|
|
97
|
+
# ActiveJob is that its enqueue callbacks (`before_enqueue` and friends)
|
|
98
|
+
# do not run.
|
|
99
|
+
def push_active_job(job_class, batch_id)
|
|
100
|
+
job = job_class.new(batch_id)
|
|
101
|
+
|
|
102
|
+
::Sidekiq::Client.push(
|
|
103
|
+
"class" => active_job_wrapper,
|
|
104
|
+
"wrapped" => job_class,
|
|
105
|
+
"queue" => job.queue_name,
|
|
106
|
+
"args" => [job.serialize]
|
|
107
|
+
)
|
|
108
|
+
end
|
|
109
|
+
end
|
|
110
|
+
end
|
|
111
|
+
end
|
data/app/models/sidekiq_batch.rb
CHANGED
|
@@ -22,8 +22,6 @@
|
|
|
22
22
|
# updated_at :datetime not null
|
|
23
23
|
#
|
|
24
24
|
class SidekiqBatch < ::Sidekiq::Batch::Jobs.base_class
|
|
25
|
-
extend ::Sidekiq::Batch::Jobs::EnumCompat
|
|
26
|
-
|
|
27
25
|
# `complete` fires whatever the outcome; `success` and `failure` name it.
|
|
28
26
|
# Three rather than two, so "always do X, and separately alert on failure"
|
|
29
27
|
# needs no duplicate registration. Same vocabulary as Sidekiq Pro.
|
|
@@ -34,7 +32,11 @@ class SidekiqBatch < ::Sidekiq::Batch::Jobs.base_class
|
|
|
34
32
|
# `succeeded`, not `complete`: `:complete` is the event that fires whatever
|
|
35
33
|
# the outcome, so a status of that name would mean the opposite of the event
|
|
36
34
|
# firing beside it.
|
|
37
|
-
|
|
35
|
+
#
|
|
36
|
+
# Positional, with `suffix:`, because ActiveRecord 8 accepts no other form.
|
|
37
|
+
# The suffix is what keeps `failed_status?` clear of the `failure` callback
|
|
38
|
+
# vocabulary sitting beside it.
|
|
39
|
+
enum :status, { pending: 0, running: 1, succeeded: 2, failed: 3 }, suffix: :status
|
|
38
40
|
|
|
39
41
|
has_many :sidekiq_batch_jobs, dependent: :delete_all
|
|
40
42
|
|
|
@@ -90,6 +92,8 @@ class SidekiqBatch < ::Sidekiq::Batch::Jobs.base_class
|
|
|
90
92
|
"cannot register a #{event_str} callback on a batch that has already finished (#{status})"
|
|
91
93
|
end
|
|
92
94
|
|
|
95
|
+
CallbackTarget.validate!(job_class)
|
|
96
|
+
|
|
93
97
|
self.callbacks = callbacks.merge(event_str => job_class.to_s)
|
|
94
98
|
|
|
95
99
|
save!
|
|
@@ -16,9 +16,8 @@
|
|
|
16
16
|
# sidekiq_batch_id :bigint not null
|
|
17
17
|
#
|
|
18
18
|
class SidekiqBatchJob < ::Sidekiq::Batch::Jobs.base_class
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
status_enum(pending: 0, complete: 1, failed: 2)
|
|
19
|
+
# Positional, with `suffix:`, because ActiveRecord 8 accepts no other form.
|
|
20
|
+
enum :status, { pending: 0, complete: 1, failed: 2 }, suffix: :status
|
|
22
21
|
|
|
23
22
|
belongs_to :sidekiq_batch
|
|
24
23
|
|
|
@@ -0,0 +1,46 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require "rspec/core"
|
|
4
|
+
|
|
5
|
+
# Test-suite wiring, in one require:
|
|
6
|
+
#
|
|
7
|
+
# # spec/rails_helper.rb
|
|
8
|
+
# require "sidekiq/batch/jobs/rspec"
|
|
9
|
+
#
|
|
10
|
+
# Both halves are mechanical, identical in every host, and awkward to work out
|
|
11
|
+
# from the symptom. Requiring this is optional; doing it by hand is fine.
|
|
12
|
+
RSpec.configure do |config|
|
|
13
|
+
# Transactional fixtures wrap each example in a transaction that is never
|
|
14
|
+
# committed, which is indistinguishable from one the caller opened, so
|
|
15
|
+
# without a baseline every `jobs {}` call in the suite raises
|
|
16
|
+
# TransactionError before enrolling anything.
|
|
17
|
+
#
|
|
18
|
+
# Recording the harness's depth rather than disabling the check keeps the
|
|
19
|
+
# guard live: a transaction the example opens on top of the fixture one still
|
|
20
|
+
# raises, which is the case worth catching. Read from the gem's own model, so
|
|
21
|
+
# it is the same connection the guard measures.
|
|
22
|
+
config.before do
|
|
23
|
+
Thread.current[::SidekiqBatch::BatchEnrollmentContext::TXN_BASELINE] =
|
|
24
|
+
::SidekiqBatchJob.connection.open_transactions
|
|
25
|
+
end
|
|
26
|
+
|
|
27
|
+
# Cleared so a stale baseline cannot hide a real transaction in a later
|
|
28
|
+
# example that runs outside this hook.
|
|
29
|
+
config.after do
|
|
30
|
+
Thread.current[::SidekiqBatch::BatchEnrollmentContext::TXN_BASELINE] = nil
|
|
31
|
+
end
|
|
32
|
+
|
|
33
|
+
# `install!` only adds the server middleware when `Sidekiq.server?` is true,
|
|
34
|
+
# which it never is under RSpec. Without this, inline jobs run and nothing
|
|
35
|
+
# marks their rows complete, so batches never finish and callbacks never
|
|
36
|
+
# fire, with nothing raised to say why. That silence is the reason this is
|
|
37
|
+
# worth shipping rather than documenting.
|
|
38
|
+
#
|
|
39
|
+
# In `before(:suite)` so the host can require its Sidekiq testing API either
|
|
40
|
+
# side of this file. Harmless under fake mode, where no job runs.
|
|
41
|
+
config.before(:suite) do
|
|
42
|
+
next unless defined?(::Sidekiq::Testing)
|
|
43
|
+
|
|
44
|
+
::Sidekiq::Testing.server_middleware { |chain| chain.add(::SidekiqBatch::Middleware) }
|
|
45
|
+
end
|
|
46
|
+
end
|
data/lib/sidekiq/batch/jobs.rb
CHANGED
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: sidekiq-batch-jobs
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.
|
|
4
|
+
version: 0.4.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Douglas Greyling
|
|
@@ -15,7 +15,7 @@ dependencies:
|
|
|
15
15
|
requirements:
|
|
16
16
|
- - ">="
|
|
17
17
|
- !ruby/object:Gem::Version
|
|
18
|
-
version: '
|
|
18
|
+
version: '7.2'
|
|
19
19
|
- - "<"
|
|
20
20
|
- !ruby/object:Gem::Version
|
|
21
21
|
version: '9'
|
|
@@ -25,7 +25,7 @@ dependencies:
|
|
|
25
25
|
requirements:
|
|
26
26
|
- - ">="
|
|
27
27
|
- !ruby/object:Gem::Version
|
|
28
|
-
version: '
|
|
28
|
+
version: '7.2'
|
|
29
29
|
- - "<"
|
|
30
30
|
- !ruby/object:Gem::Version
|
|
31
31
|
version: '9'
|
|
@@ -35,7 +35,7 @@ dependencies:
|
|
|
35
35
|
requirements:
|
|
36
36
|
- - ">="
|
|
37
37
|
- !ruby/object:Gem::Version
|
|
38
|
-
version: '
|
|
38
|
+
version: '7.2'
|
|
39
39
|
- - "<"
|
|
40
40
|
- !ruby/object:Gem::Version
|
|
41
41
|
version: '9'
|
|
@@ -45,7 +45,7 @@ dependencies:
|
|
|
45
45
|
requirements:
|
|
46
46
|
- - ">="
|
|
47
47
|
- !ruby/object:Gem::Version
|
|
48
|
-
version: '
|
|
48
|
+
version: '7.2'
|
|
49
49
|
- - "<"
|
|
50
50
|
- !ruby/object:Gem::Version
|
|
51
51
|
version: '9'
|
|
@@ -88,6 +88,7 @@ files:
|
|
|
88
88
|
- app/models/sidekiq_batch/abandoned_enrollment_reaper.rb
|
|
89
89
|
- app/models/sidekiq_batch/announcement.rb
|
|
90
90
|
- app/models/sidekiq_batch/batch_enrollment_context.rb
|
|
91
|
+
- app/models/sidekiq_batch/callback_target.rb
|
|
91
92
|
- app/models/sidekiq_batch/client_middleware.rb
|
|
92
93
|
- app/models/sidekiq_batch/completion_query.rb
|
|
93
94
|
- app/models/sidekiq_batch/groomer_worker.rb
|
|
@@ -106,8 +107,8 @@ files:
|
|
|
106
107
|
- lib/sidekiq/batch/jobs.rb
|
|
107
108
|
- lib/sidekiq/batch/jobs/configuration.rb
|
|
108
109
|
- lib/sidekiq/batch/jobs/engine.rb
|
|
109
|
-
- lib/sidekiq/batch/jobs/enum_compat.rb
|
|
110
110
|
- lib/sidekiq/batch/jobs/failure_policy.rb
|
|
111
|
+
- lib/sidekiq/batch/jobs/rspec.rb
|
|
111
112
|
- lib/sidekiq/batch/jobs/schema.rb
|
|
112
113
|
- lib/sidekiq/batch/jobs/version.rb
|
|
113
114
|
homepage: https://github.com/douglasgreyling/sidekiq-batch-jobs
|
|
@@ -124,7 +125,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
|
|
|
124
125
|
requirements:
|
|
125
126
|
- - ">="
|
|
126
127
|
- !ruby/object:Gem::Version
|
|
127
|
-
version: 3.
|
|
128
|
+
version: 3.1.0
|
|
128
129
|
required_rubygems_version: !ruby/object:Gem::Requirement
|
|
129
130
|
requirements:
|
|
130
131
|
- - ">="
|
|
@@ -1,29 +0,0 @@
|
|
|
1
|
-
# frozen_string_literal: true
|
|
2
|
-
|
|
3
|
-
module Sidekiq
|
|
4
|
-
module Batch
|
|
5
|
-
module Jobs
|
|
6
|
-
# ActiveRecord's `enum` signature moved twice inside the range this gem
|
|
7
|
-
# supports, and no single call works across all of it:
|
|
8
|
-
#
|
|
9
|
-
# 6.1 def enum(definitions) hash only, `_suffix:`
|
|
10
|
-
# 7.0-7.2 def enum(name = nil, values = nil, **opts) either form
|
|
11
|
-
# 8.0 def enum(name, values = nil, **opts) positional, `suffix:`
|
|
12
|
-
#
|
|
13
|
-
# So 6.1 rejects the modern call and 8.0 rejects the legacy one. Both forms
|
|
14
|
-
# generate an identical surface (`.statuses`, the `pending_status?`
|
|
15
|
-
# predicates, the `pending_status` scopes), so which branch a given Rails
|
|
16
|
-
# takes is invisible to everything downstream.
|
|
17
|
-
module EnumCompat
|
|
18
|
-
# @param values [Hash{Symbol => Integer}] status name => column value
|
|
19
|
-
def status_enum(values)
|
|
20
|
-
if ::ActiveRecord::VERSION::MAJOR >= 7
|
|
21
|
-
enum :status, values, suffix: :status
|
|
22
|
-
else
|
|
23
|
-
enum status: values, _suffix: :status
|
|
24
|
-
end
|
|
25
|
-
end
|
|
26
|
-
end
|
|
27
|
-
end
|
|
28
|
-
end
|
|
29
|
-
end
|