sidekiq-unique-jobs 8.0.10 → 8.0.12
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +54 -6
- data/README.md +62 -49
- data/lib/sidekiq_unique_jobs/cli.rb +2 -2
- data/lib/sidekiq_unique_jobs/config.rb +65 -33
- data/lib/sidekiq_unique_jobs/digests.rb +1 -1
- data/lib/sidekiq_unique_jobs/exceptions.rb +2 -2
- data/lib/sidekiq_unique_jobs/job.rb +1 -1
- data/lib/sidekiq_unique_jobs/lock/base_lock.rb +8 -4
- data/lib/sidekiq_unique_jobs/lock/until_and_while_executing.rb +7 -4
- data/lib/sidekiq_unique_jobs/lock/until_executing.rb +1 -1
- data/lib/sidekiq_unique_jobs/lock/while_executing.rb +1 -1
- data/lib/sidekiq_unique_jobs/lock.rb +1 -1
- data/lib/sidekiq_unique_jobs/lock_args.rb +3 -3
- data/lib/sidekiq_unique_jobs/lock_digest.rb +6 -1
- data/lib/sidekiq_unique_jobs/lock_ttl.rb +34 -8
- data/lib/sidekiq_unique_jobs/locksmith.rb +25 -7
- data/lib/sidekiq_unique_jobs/logging.rb +2 -2
- data/lib/sidekiq_unique_jobs/lua/shared/_find_digest_in_process_set.lua +8 -3
- data/lib/sidekiq_unique_jobs/lua/shared/_find_digest_in_queues.lua +11 -0
- data/lib/sidekiq_unique_jobs/lua/shared/_find_digest_in_sorted_set.lua +5 -1
- data/lib/sidekiq_unique_jobs/lua/unlock.lua +20 -12
- data/lib/sidekiq_unique_jobs/on_conflict/reject.rb +10 -1
- data/lib/sidekiq_unique_jobs/on_conflict/replace.rb +3 -3
- data/lib/sidekiq_unique_jobs/on_conflict/reschedule.rb +1 -1
- data/lib/sidekiq_unique_jobs/on_conflict.rb +2 -2
- data/lib/sidekiq_unique_jobs/orphans/manager.rb +3 -3
- data/lib/sidekiq_unique_jobs/orphans/ruby_reaper.rb +36 -9
- data/lib/sidekiq_unique_jobs/reflections.rb +3 -3
- data/lib/sidekiq_unique_jobs/rspec/matchers/have_valid_sidekiq_options.rb +3 -1
- data/lib/sidekiq_unique_jobs/script/client.rb +11 -3
- data/lib/sidekiq_unique_jobs/script/lua_error.rb +2 -0
- data/lib/sidekiq_unique_jobs/script/scripts.rb +42 -46
- data/lib/sidekiq_unique_jobs/sidekiq_unique_ext.rb +2 -2
- data/lib/sidekiq_unique_jobs/sidekiq_unique_jobs.rb +4 -4
- data/lib/sidekiq_unique_jobs/sidekiq_worker_methods.rb +1 -1
- data/lib/sidekiq_unique_jobs/testing.rb +2 -2
- data/lib/sidekiq_unique_jobs/version.rb +1 -1
- data/lib/sidekiq_unique_jobs/web/helpers.rb +29 -1
- data/lib/sidekiq_unique_jobs/web.rb +38 -30
- metadata +5 -8
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: ac54a1a32a5e8e0d11d10907799a1cc495da0439868857db6f7516e2334e8a2e
|
|
4
|
+
data.tar.gz: '0998be2173de200c826873e1d7e96aceb72f495b3d0b0d3e316196c52e0844e2'
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: e10ce9fe4a23b720f8ecd701bc139ac5f5b3457c53ca6077996abce19d04003b283943fcbb4eeee22284b9df870f477e33c74852e3ec838d50aff28e19440c38
|
|
7
|
+
data.tar.gz: daeead0b5f5b95dcc22987ad6ec3c2b27118f4166b04c7c478ad790d8a0d10629388866cfcd26b341f2bbfd2e48f7d4b1210583b28ea9287788b23ac3333c659
|
data/CHANGELOG.md
CHANGED
|
@@ -1,12 +1,55 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## [v8.0.11](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v8.0.11) (2025-05-25)
|
|
4
|
+
|
|
5
|
+
[Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v8.0.10...v8.0.11)
|
|
6
|
+
|
|
7
|
+
**Implemented enhancements:**
|
|
8
|
+
|
|
9
|
+
- chore: address recent rubocop changes [\#880](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/880) ([mhenrixon](https://github.com/mhenrixon))
|
|
10
|
+
- feat\(digest\): allow modern algorithm [\#853](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/853) ([mhenrixon](https://github.com/mhenrixon))
|
|
11
|
+
|
|
12
|
+
**Closed issues:**
|
|
13
|
+
|
|
14
|
+
- Your paypal link doesn't work [\#876](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/876)
|
|
15
|
+
- Replace MD5 with SHA256+ [\#848](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/848)
|
|
16
|
+
- NoMethodError: undefined method `\[\]' for true:TrueClass [\#643](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/643)
|
|
17
|
+
|
|
18
|
+
**Merged pull requests:**
|
|
19
|
+
|
|
20
|
+
- Add support for lock\_ttl to be a Proc/class method [\#879](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/879) ([brayden-onesignal](https://github.com/brayden-onesignal))
|
|
21
|
+
- Move from Sidekiq 8 beta to released version [\#872](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/872) ([jukra](https://github.com/jukra))
|
|
22
|
+
- update Reject\#kill\_with\_options? for Ruby 3 kwargs [\#868](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/868) ([stathis-alexander](https://github.com/stathis-alexander))
|
|
23
|
+
- Remove redundant include to locales \(for Sidekiq 8\) [\#867](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/867) ([jukra](https://github.com/jukra))
|
|
24
|
+
- Support for Sidekiq 8 [\#866](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/866) ([jukra](https://github.com/jukra))
|
|
25
|
+
- 📝 Improve README.md [\#860](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/860) ([jaredsmithse](https://github.com/jaredsmithse))
|
|
26
|
+
- mention ttl and timeout unit \(seconds\) in README [\#859](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/859) ([fwolfst](https://github.com/fwolfst))
|
|
27
|
+
- Add a note to README on `schedule_in` option for `reschedule` conflict strategy [\#849](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/849) ([vittorius](https://github.com/vittorius))
|
|
28
|
+
|
|
29
|
+
## [v8.0.10](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v8.0.10) (2024-02-22)
|
|
30
|
+
|
|
31
|
+
[Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v7.1.33...v8.0.10)
|
|
32
|
+
|
|
33
|
+
**Closed issues:**
|
|
34
|
+
|
|
35
|
+
- until\_and\_while\_executing and lock\_ttl: jobs silently dropped [\#788](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/788)
|
|
36
|
+
- Slow evalsha causing timeouts [\#668](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/668)
|
|
37
|
+
|
|
38
|
+
**Merged pull requests:**
|
|
39
|
+
|
|
40
|
+
- tweak changelog for 8.0.9 [\#836](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/836) ([Earlopain](https://github.com/Earlopain))
|
|
41
|
+
- Add digest scores for faster deletes in sorted sets [\#835](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/835) ([ezekg](https://github.com/ezekg))
|
|
42
|
+
|
|
43
|
+
## [v7.1.33](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v7.1.33) (2024-02-12)
|
|
44
|
+
|
|
45
|
+
[Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v8.0.9...v7.1.33)
|
|
46
|
+
|
|
3
47
|
## [v8.0.9](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v8.0.9) (2024-02-12)
|
|
4
48
|
|
|
5
49
|
[Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v8.0.8...v8.0.9)
|
|
6
50
|
|
|
7
51
|
**Fixed bugs:**
|
|
8
52
|
|
|
9
|
-
- note: The RCE vulnerability was a false alarm, `sidekiq-unique-jobs` was not vulnerable to RCE. You can find additional information in the PR linked below.
|
|
10
53
|
- fix\(rce\): prevent remot code execution [\#833](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/833) ([mhenrixon](https://github.com/mhenrixon))
|
|
11
54
|
|
|
12
55
|
## [v8.0.8](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v8.0.8) (2024-02-12)
|
|
@@ -50,7 +93,7 @@
|
|
|
50
93
|
|
|
51
94
|
## [v8.0.6](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v8.0.6) (2024-01-24)
|
|
52
95
|
|
|
53
|
-
[Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/
|
|
96
|
+
[Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v7.1.32...v8.0.6)
|
|
54
97
|
|
|
55
98
|
**Implemented enhancements:**
|
|
56
99
|
|
|
@@ -60,7 +103,6 @@
|
|
|
60
103
|
**Closed issues:**
|
|
61
104
|
|
|
62
105
|
- should respond to `has_valid_sidekiq_options?` [\#822](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/822)
|
|
63
|
-
- Reaper manager registration is subject to race conditions [\#801](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/801)
|
|
64
106
|
- `while_executing` with `on_conflict: :reschedule` Reschedule job after job execution [\#800](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/800)
|
|
65
107
|
- Large retry queue causes reaper to run too slow [\#759](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/759)
|
|
66
108
|
|
|
@@ -68,6 +110,14 @@
|
|
|
68
110
|
|
|
69
111
|
- fix: skip unless reaper was registered [\#820](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/820) ([mhenrixon](https://github.com/mhenrixon))
|
|
70
112
|
|
|
113
|
+
## [v7.1.32](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v7.1.32) (2023-11-11)
|
|
114
|
+
|
|
115
|
+
[Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v8.0.5...v7.1.32)
|
|
116
|
+
|
|
117
|
+
**Closed issues:**
|
|
118
|
+
|
|
119
|
+
- Reaper manager registration is subject to race conditions [\#801](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/801)
|
|
120
|
+
|
|
71
121
|
## [v8.0.5](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v8.0.5) (2023-11-11)
|
|
72
122
|
|
|
73
123
|
[Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v8.0.4...v8.0.5)
|
|
@@ -484,7 +534,7 @@
|
|
|
484
534
|
|
|
485
535
|
**Merged pull requests:**
|
|
486
536
|
|
|
487
|
-
- Update docs [\#644](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/644) ([
|
|
537
|
+
- Update docs [\#644](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/644) ([ruzig](https://github.com/ruzig))
|
|
488
538
|
|
|
489
539
|
## [v7.0.13](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v7.0.13) (2021-09-27)
|
|
490
540
|
|
|
@@ -2138,7 +2188,6 @@
|
|
|
2138
2188
|
- Attempt to constantize String `worker_class` arguments passed to client middleware [\#17](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/17) ([disbelief](https://github.com/disbelief))
|
|
2139
2189
|
- Compatibility with Sidekiq 2.12.1 Scheduled Jobs [\#16](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/16) ([lsimoneau](https://github.com/lsimoneau))
|
|
2140
2190
|
- Allow worker to specify which arguments to include in uniquing hash [\#12](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/12) ([sax](https://github.com/sax))
|
|
2141
|
-
- Add support for unique when using Sidekiq's delay function [\#11](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/11) ([eduardosasso](https://github.com/eduardosasso))
|
|
2142
2191
|
- Adding the unique prefix option [\#8](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/8) ([KensoDev](https://github.com/KensoDev))
|
|
2143
2192
|
- Remove unnecessary log messages [\#7](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/7) ([marclennox](https://github.com/marclennox))
|
|
2144
2193
|
|
|
@@ -2152,7 +2201,6 @@
|
|
|
2152
2201
|
|
|
2153
2202
|
**Merged pull requests:**
|
|
2154
2203
|
|
|
2155
|
-
- Fix multiple bugs, cleaned up dependencies, and added a feature [\#4](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/4) ([kemper-blinq](https://github.com/kemper-blinq))
|
|
2156
2204
|
- Dependency on sidekiq 2.2.0 and up [\#3](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/3) ([philostler](https://github.com/philostler))
|
|
2157
2205
|
|
|
2158
2206
|
## [v2.2.1](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v2.2.1) (2012-08-19)
|
data/README.md
CHANGED
|
@@ -4,7 +4,7 @@
|
|
|
4
4
|
|
|
5
5
|
## Support Me
|
|
6
6
|
|
|
7
|
-
Want to show me some ❤️ for the hard work I do on this gem? You can use the following PayPal link: [https://paypal.me/
|
|
7
|
+
Want to show me some ❤️ for the hard work I do on this gem? You can use the following PayPal link: [https://paypal.me/mhenrixon2](https://paypal.me/mhenrixon2). Any amount is welcome and let me tell you it feels good to be appreciated. Even a dollar makes me super excited about all of this.
|
|
8
8
|
|
|
9
9
|
<!-- MarkdownTOC -->
|
|
10
10
|
|
|
@@ -59,6 +59,7 @@ Want to show me some ❤️ for the hard work I do on this gem? You can use the
|
|
|
59
59
|
- [sidekiq-global_id](#sidekiq-global_id)
|
|
60
60
|
- [sidekiq-status](#sidekiq-status)
|
|
61
61
|
- [Global Configuration](#global-configuration)
|
|
62
|
+
- [digest_algorithm](#digest_algorithm)
|
|
62
63
|
- [debug_lua](#debug_lua)
|
|
63
64
|
- [lock_timeout](#lock_timeout)
|
|
64
65
|
- [lock_ttl](#lock_ttl)
|
|
@@ -88,7 +89,7 @@ Want to show me some ❤️ for the hard work I do on this gem? You can use the
|
|
|
88
89
|
|
|
89
90
|
## Introduction
|
|
90
91
|
|
|
91
|
-
This gem adds unique constraints to
|
|
92
|
+
This gem adds unique constraints to Sidekiq jobs. The uniqueness is achieved by creating a set of keys in redis based off of `queue`, `class`, `args` (in the Sidekiq job hash).
|
|
92
93
|
|
|
93
94
|
By default, only one lock for a given hash can be acquired. What happens when a lock can't be acquired is governed by a chosen [Conflict Strategy](#conflict-strategy) strategy. Unless a conflict strategy is chosen (?)
|
|
94
95
|
|
|
@@ -108,7 +109,7 @@ Here are links to some of the old versions
|
|
|
108
109
|
Add this line to your application's Gemfile:
|
|
109
110
|
|
|
110
111
|
```ruby
|
|
111
|
-
gem
|
|
112
|
+
gem "sidekiq-unique-jobs"
|
|
112
113
|
```
|
|
113
114
|
|
|
114
115
|
And then execute:
|
|
@@ -153,7 +154,7 @@ end
|
|
|
153
154
|
|
|
154
155
|
### Your first worker
|
|
155
156
|
|
|
156
|
-
The lock type most likely to be is `:until_executed`. This type of lock creates a lock from when `UntilExecutedWorker.perform_async` is called until right after `UntilExecutedWorker.new.perform` has been called.
|
|
157
|
+
The lock type most likely to be used is `:until_executed`. This type of lock creates a lock from when `UntilExecutedWorker.perform_async` is called until right after `UntilExecutedWorker.new.perform` has been called.
|
|
157
158
|
|
|
158
159
|
```ruby
|
|
159
160
|
# frozen_string_literal: true
|
|
@@ -184,7 +185,7 @@ You can read more about the worker configuration in [Worker Configuration](#work
|
|
|
184
185
|
- [ActiveJob officially not supported][48]
|
|
185
186
|
- [redis-namespace officially not supported][49]
|
|
186
187
|
|
|
187
|
-
See [Sidekiq requirements][24] for detailed requirements of Sidekiq itself (be sure to check the right sidekiq version).
|
|
188
|
+
See [Sidekiq requirements][24] for detailed requirements of Sidekiq itself (be sure to check the right `sidekiq` version).
|
|
188
189
|
|
|
189
190
|
## Locks
|
|
190
191
|
|
|
@@ -196,7 +197,7 @@ A lock is created when `UntilExecuting.perform_async` is called. Then it is eith
|
|
|
196
197
|
|
|
197
198
|
```ruby
|
|
198
199
|
class UntilExecuting
|
|
199
|
-
include Sidekiq::
|
|
200
|
+
include Sidekiq::Worker
|
|
200
201
|
|
|
201
202
|
sidekiq_options lock: :until_executing
|
|
202
203
|
|
|
@@ -218,7 +219,7 @@ A lock is created when `UntilExecuted.perform_async` is called. Then it is eithe
|
|
|
218
219
|
|
|
219
220
|
```ruby
|
|
220
221
|
class UntilExecuted
|
|
221
|
-
include Sidekiq::
|
|
222
|
+
include Sidekiq::Worker
|
|
222
223
|
|
|
223
224
|
sidekiq_options lock: :until_executed
|
|
224
225
|
|
|
@@ -236,9 +237,9 @@ This lock behaves identically to the [Until Executed](#until-executed) except fo
|
|
|
236
237
|
|
|
237
238
|
```ruby
|
|
238
239
|
class UntilExpired
|
|
239
|
-
include Sidekiq::
|
|
240
|
+
include Sidekiq::Worker
|
|
240
241
|
|
|
241
|
-
sidekiq_options lock: :until_expired, lock_ttl: 1.day
|
|
242
|
+
sidekiq_options lock: :until_expired, lock_ttl: 1.day.to_i
|
|
242
243
|
|
|
243
244
|
def perform
|
|
244
245
|
# Do work
|
|
@@ -254,7 +255,7 @@ This lock is a combination of two locks (`:until_executing` and `:while_executin
|
|
|
254
255
|
|
|
255
256
|
```ruby
|
|
256
257
|
class UntilAndWhileExecutingWorker
|
|
257
|
-
include Sidekiq::
|
|
258
|
+
include Sidekiq::Worker
|
|
258
259
|
|
|
259
260
|
sidekiq_options lock: :until_and_while_executing,
|
|
260
261
|
lock_timeout: 2,
|
|
@@ -276,7 +277,7 @@ These locks are put on a queue without any type of locking mechanism, the lockin
|
|
|
276
277
|
|
|
277
278
|
```ruby
|
|
278
279
|
class WhileExecutingWorker
|
|
279
|
-
include Sidekiq::
|
|
280
|
+
include Sidekiq::Worker
|
|
280
281
|
|
|
281
282
|
sidekiq_options lock: :while_executing,
|
|
282
283
|
lock_timeout: 2,
|
|
@@ -347,7 +348,7 @@ Please not that if you try to override a default lock, an `ArgumentError` will b
|
|
|
347
348
|
|
|
348
349
|
## Conflict Strategy
|
|
349
350
|
|
|
350
|
-
Decides how we handle conflict. We can either `reject` the job to the dead queue or `reschedule` it. Both are useful for jobs that absolutely need to run and have been configured to use the lock `WhileExecuting` that is used only by the
|
|
351
|
+
Decides how we handle conflict. We can either `reject` the job to the dead queue or `reschedule` it. Both are useful for jobs that absolutely need to run and have been configured to use the lock `WhileExecuting` that is used only by the Sidekiq server process.
|
|
351
352
|
|
|
352
353
|
Furthermore, `log` can be be used with the lock `UntilExecuted` and `UntilExpired`. Now we write a log entry saying the job could not be pushed because it is a duplicate of another job with the same arguments.
|
|
353
354
|
|
|
@@ -401,7 +402,7 @@ always scheduled in the future. Currently only attempting to retry one time.
|
|
|
401
402
|
sidekiq_options on_conflict: :reschedule
|
|
402
403
|
```
|
|
403
404
|
|
|
404
|
-
This strategy is intended to be used with `WhileExecuting` and will delay the job to be tried again in 5 seconds. This will mess up the
|
|
405
|
+
This strategy is intended to be used with `WhileExecuting` and will delay the job to be tried again in 5 seconds (this delay can be configured via `sidekiq_options schedule_in: {seconds}`). This will mess up the Sidekiq stats but will prevent exceptions from being logged and confuse your sysadmins.
|
|
405
406
|
|
|
406
407
|
### Custom Strategies
|
|
407
408
|
|
|
@@ -440,12 +441,12 @@ Please not that if you try to override a default lock, an `ArgumentError` will b
|
|
|
440
441
|
|
|
441
442
|
### 3 Cleanup Dead Locks
|
|
442
443
|
|
|
443
|
-
For sidekiq versions < 5.1 a `sidekiq_retries_exhausted` block is required per worker class. This is deprecated in Sidekiq 6.0
|
|
444
|
+
For `sidekiq` versions < 5.1 a `sidekiq_retries_exhausted` block is required per worker class. This is deprecated in Sidekiq 6.0
|
|
444
445
|
|
|
445
446
|
```ruby
|
|
446
447
|
class MyWorker
|
|
447
448
|
sidekiq_retries_exhausted do |msg, _ex|
|
|
448
|
-
digest = msg[
|
|
449
|
+
digest = msg["lock_digest"]
|
|
449
450
|
SidekiqUniqueJobs::Digests.new.delete_by_digest(digest) if digest
|
|
450
451
|
end
|
|
451
452
|
end
|
|
@@ -456,7 +457,7 @@ Starting in v5.1, Sidekiq can also fire a global callback when a job dies: In ve
|
|
|
456
457
|
```ruby
|
|
457
458
|
Sidekiq.configure_server do |config|
|
|
458
459
|
config.death_handlers << ->(job, _ex) do
|
|
459
|
-
digest = job[
|
|
460
|
+
digest = job["lock_digest"]
|
|
460
461
|
SidekiqUniqueJobs::Digests.new.delete_by_digest(digest) if digest
|
|
461
462
|
end
|
|
462
463
|
end
|
|
@@ -472,11 +473,11 @@ To use the web extension you need to require it in your routes.
|
|
|
472
473
|
|
|
473
474
|
```ruby
|
|
474
475
|
#app/config/routes.rb
|
|
475
|
-
require
|
|
476
|
-
mount Sidekiq::Web, at:
|
|
476
|
+
require "sidekiq_unique_jobs/web"
|
|
477
|
+
mount Sidekiq::Web, at: "/sidekiq"
|
|
477
478
|
```
|
|
478
479
|
|
|
479
|
-
There is no need to `require
|
|
480
|
+
There is no need to `require "sidekiq/web"` since `sidekiq_unique_jobs/web`
|
|
480
481
|
already does this.
|
|
481
482
|
|
|
482
483
|
To filter/search for keys we can use the wildcard `*`. If we have a unique digest `'uniquejobs:9e9b5ce5d423d3ea470977004b50ff84` we can search for it by enter `*ff84` and it should return all digests that end with `ff84`.
|
|
@@ -490,10 +491,10 @@ To setup reflections for logging or metrics, use the following API:
|
|
|
490
491
|
```ruby
|
|
491
492
|
|
|
492
493
|
def extract_log_from_job(message, job_hash)
|
|
493
|
-
worker = job_hash[
|
|
494
|
-
args = job_hash[
|
|
495
|
-
lock_args = job_hash[
|
|
496
|
-
queue = job_hash[
|
|
494
|
+
worker = job_hash["class"]
|
|
495
|
+
args = job_hash["args"]
|
|
496
|
+
lock_args = job_hash["lock_args"]
|
|
497
|
+
queue = job_hash["queue"]
|
|
497
498
|
{
|
|
498
499
|
message: message,
|
|
499
500
|
worker: worker,
|
|
@@ -505,7 +506,7 @@ end
|
|
|
505
506
|
|
|
506
507
|
SidekiqUniqueJobs.reflect do |on|
|
|
507
508
|
on.lock_failed do |job_hash|
|
|
508
|
-
message = extract_log_from_job(
|
|
509
|
+
message = extract_log_from_job("Lock Failed", job_hash)
|
|
509
510
|
Sidekiq.logger.warn(message)
|
|
510
511
|
end
|
|
511
512
|
end
|
|
@@ -555,7 +556,7 @@ Also mostly useful for reporting purposes. The job was successfully unlocked.
|
|
|
555
556
|
|
|
556
557
|
#### unknown_sidekiq_worker
|
|
557
558
|
|
|
558
|
-
The reason this happens is that the server couldn't find a valid
|
|
559
|
+
The reason this happens is that the server couldn't find a valid Sidekiq worker class. Most likely, that worker isn't intended to be processed by this Sidekiq server instance.
|
|
559
560
|
|
|
560
561
|
### Show Locks
|
|
561
562
|
|
|
@@ -603,7 +604,7 @@ assert_raise(InvalidWorker){ SidekiqUniqueJobs.validate_worker!(BadWorker.get_si
|
|
|
603
604
|
|
|
604
605
|
### Uniqueness
|
|
605
606
|
|
|
606
|
-
This has been probably the most confusing part of this gem. People get really confused with how unreliable the unique jobs have been. I
|
|
607
|
+
This has been probably the most confusing part of this gem. People get really confused with how unreliable the unique jobs have been. I therefore decided to do what Mike is doing for Sidekiq Enterprise. Read the section about unique jobs: [Enterprise unique jobs][](?)
|
|
607
608
|
|
|
608
609
|
```ruby
|
|
609
610
|
SidekiqUniqueJobs.configure do |config|
|
|
@@ -612,7 +613,7 @@ SidekiqUniqueJobs.configure do |config|
|
|
|
612
613
|
end
|
|
613
614
|
```
|
|
614
615
|
|
|
615
|
-
If you truly wanted to test the sidekiq client push you could do something like below. Note that it will only work for the jobs that lock when the client pushes the job to redis (UntilExecuted, UntilAndWhileExecuting and UntilExpired).
|
|
616
|
+
If you truly wanted to test the `sidekiq` client push you could do something like below. Note that it will only work for the jobs that lock when the client pushes the job to redis (UntilExecuted, UntilAndWhileExecuting and UntilExpired).
|
|
616
617
|
|
|
617
618
|
```ruby
|
|
618
619
|
require "sidekiq_unique_jobs/testing"
|
|
@@ -624,7 +625,7 @@ RSpec.describe Workers::CoolOne do
|
|
|
624
625
|
|
|
625
626
|
# ... your tests that don't test uniqueness
|
|
626
627
|
|
|
627
|
-
context
|
|
628
|
+
context "when Sidekiq::Testing.disabled?" do
|
|
628
629
|
before do
|
|
629
630
|
Sidekiq::Testing.disable!
|
|
630
631
|
Sidekiq.redis(&:flushdb)
|
|
@@ -634,7 +635,7 @@ RSpec.describe Workers::CoolOne do
|
|
|
634
635
|
Sidekiq.redis(&:flushdb)
|
|
635
636
|
end
|
|
636
637
|
|
|
637
|
-
it
|
|
638
|
+
it "prevents duplicate jobs from being scheduled" do
|
|
638
639
|
SidekiqUniqueJobs.use_config(enabled: true) do
|
|
639
640
|
expect(described_class.perform_in(3600, 1)).not_to eq(nil)
|
|
640
641
|
expect(described_class.perform_async(1)).to eq(nil)
|
|
@@ -734,17 +735,29 @@ Configure SidekiqUniqueJobs in an initializer or the sidekiq initializer on appl
|
|
|
734
735
|
|
|
735
736
|
```ruby
|
|
736
737
|
SidekiqUniqueJobs.configure do |config|
|
|
737
|
-
config.logger = Sidekiq.logger
|
|
738
|
-
config.logger_enabled
|
|
739
|
-
config.debug_lua
|
|
740
|
-
config.lock_info
|
|
741
|
-
config.lock_ttl
|
|
742
|
-
config.lock_timeout
|
|
743
|
-
config.max_history
|
|
744
|
-
config.reaper
|
|
745
|
-
config.reaper_count
|
|
746
|
-
config.reaper_interval
|
|
747
|
-
config.reaper_timeout
|
|
738
|
+
config.logger = Sidekiq.logger # default, change at your own discretion
|
|
739
|
+
config.logger_enabled = true # default, disable for test environments
|
|
740
|
+
config.debug_lua = false # Turn on when debugging
|
|
741
|
+
config.lock_info = false # Turn on when debugging
|
|
742
|
+
config.lock_ttl = 600 # Expire locks after 10 minutes
|
|
743
|
+
config.lock_timeout = nil # turn off lock timeout
|
|
744
|
+
config.max_history = 0 # Turn on when debugging
|
|
745
|
+
config.reaper = :ruby # :ruby, :lua or :none/nil
|
|
746
|
+
config.reaper_count = 1000 # Stop reaping after this many keys
|
|
747
|
+
config.reaper_interval = 600 # Reap orphans every 10 minutes
|
|
748
|
+
config.reaper_timeout = 150 # Timeout reaper after 2.5 minutes
|
|
749
|
+
config.digest_algorithm = :modern # Timeout reaper after 2.5 minutes
|
|
750
|
+
end
|
|
751
|
+
```
|
|
752
|
+
#### digest_algorithm
|
|
753
|
+
|
|
754
|
+
For backwards compatibility this one is set to `:legacy` by the default. If you happen to run into issues with FIPS being enabled on your redis server you might want to set this to `:modern`.
|
|
755
|
+
|
|
756
|
+
See: https://github.com/mhenrixon/sidekiq-unique-jobs/issues/848 for explanation
|
|
757
|
+
|
|
758
|
+
```ruby
|
|
759
|
+
SidekiqUniqueJobs.configure do |config|
|
|
760
|
+
config.digest_algorithm = :modern # Timeout reaper after 2.5 minutes
|
|
748
761
|
end
|
|
749
762
|
```
|
|
750
763
|
|
|
@@ -762,7 +775,7 @@ Turning on debug_lua will allow the lua scripts to output debug information abou
|
|
|
762
775
|
SidekiqUniqueJobs.config.lock_timeout #=> 0
|
|
763
776
|
```
|
|
764
777
|
|
|
765
|
-
Set a global lock_timeout to use for all jobs that don't otherwise specify a lock_timeout.
|
|
778
|
+
Set a global lock_timeout (in seconds) to use for all jobs that don't otherwise specify a lock_timeout.
|
|
766
779
|
|
|
767
780
|
Lock timeout decides how long to wait for acquiring the lock. A value of nil means to wait indefinitely for a lock resource to become available.
|
|
768
781
|
|
|
@@ -772,7 +785,7 @@ Lock timeout decides how long to wait for acquiring the lock. A value of nil mea
|
|
|
772
785
|
SidekiqUniqueJobs.config.lock_ttl #=> nil
|
|
773
786
|
```
|
|
774
787
|
|
|
775
|
-
Set a global lock_ttl to use for all jobs that don't otherwise specify a lock_ttl.
|
|
788
|
+
Set a global lock_ttl (in seconds) to use for all jobs that don't otherwise specify a lock_ttl.
|
|
776
789
|
|
|
777
790
|
Lock TTL decides how long to wait at most before considering a lock to be expired and making it possible to reuse that lock.
|
|
778
791
|
|
|
@@ -808,7 +821,7 @@ This is a log that can be accessed by a lock to see what happened for that lock.
|
|
|
808
821
|
SidekiqUniqueJobs.config.reaper #=> :ruby
|
|
809
822
|
```
|
|
810
823
|
|
|
811
|
-
If using the orphans cleanup process it is critical to be aware of the following. The `:ruby` job is much slower but the `:lua` job locks redis while executing. While doing intense processing it is best to avoid locking redis with a lua script.
|
|
824
|
+
If using the orphans cleanup process it is critical to be aware of the following. The `:ruby` job is much slower but the `:lua` job locks redis while executing. While doing intense processing it is best to avoid locking redis with a lua script. Therefore the batch size (controlled by the `reaper_count` setting) needs to be reduced.
|
|
812
825
|
|
|
813
826
|
In my benchmarks deleting 1000 orphaned locks with lua performs around 65% faster than deleting 1000 keys in ruby.
|
|
814
827
|
|
|
@@ -913,13 +926,13 @@ This is mainly intended for `Worker.set(queue: :another).perform_async`.
|
|
|
913
926
|
class Worker
|
|
914
927
|
include Sidekiq::Worker
|
|
915
928
|
|
|
916
|
-
sidekiq_options unique_across_queues: true, queue:
|
|
929
|
+
sidekiq_options unique_across_queues: true, queue: "default"
|
|
917
930
|
|
|
918
931
|
def perform(args); end
|
|
919
932
|
end
|
|
920
933
|
```
|
|
921
934
|
|
|
922
|
-
Now if you push override the queue with `Worker.set(queue:
|
|
935
|
+
Now if you push override the queue with `Worker.set(queue: "another").perform_async(1)` it will still be considered unique when compared to `Worker.perform_async(1)` (that was actually pushed to the queue `default`).
|
|
923
936
|
|
|
924
937
|
#### unique_across_workers
|
|
925
938
|
|
|
@@ -929,7 +942,7 @@ This configuration option is slightly misleading. It doesn't disregard the worke
|
|
|
929
942
|
class WorkerOne
|
|
930
943
|
include Sidekiq::Worker
|
|
931
944
|
|
|
932
|
-
sidekiq_options unique_across_workers: true, queue:
|
|
945
|
+
sidekiq_options unique_across_workers: true, queue: "default"
|
|
933
946
|
|
|
934
947
|
def perform(args); end
|
|
935
948
|
end
|
|
@@ -937,14 +950,14 @@ end
|
|
|
937
950
|
class WorkerTwo
|
|
938
951
|
include Sidekiq::Worker
|
|
939
952
|
|
|
940
|
-
sidekiq_options unique_across_workers: true, queue:
|
|
953
|
+
sidekiq_options unique_across_workers: true, queue: "default"
|
|
941
954
|
|
|
942
955
|
def perform(args); end
|
|
943
956
|
end
|
|
944
957
|
|
|
945
958
|
|
|
946
959
|
WorkerOne.perform_async(1)
|
|
947
|
-
# =>
|
|
960
|
+
# => "the jobs unique id"
|
|
948
961
|
|
|
949
962
|
WorkerTwo.perform_async(1)
|
|
950
963
|
# => nil because WorkerOne just stole the lock
|
|
@@ -1036,7 +1049,7 @@ There is a [
|
|
1039
|
-
1. Commit your changes (`git commit -am
|
|
1052
|
+
1. Commit your changes (`git commit -am "Add some feature"`)
|
|
1040
1053
|
1. Push to the branch (`git push origin my-new-feature`)
|
|
1041
1054
|
1. Create new Pull Request
|
|
1042
1055
|
|
|
@@ -12,7 +12,7 @@ module SidekiqUniqueJobs
|
|
|
12
12
|
# :nodoc:
|
|
13
13
|
# rubocop:disable Style/OptionalBooleanParameter
|
|
14
14
|
def self.banner(command, _namespace = nil, _subcommand = false) # rubocop:disable Style/OptionalBooleanParameter
|
|
15
|
-
"jobs #{@package_name} #{command.usage}" # rubocop:disable ThreadSafety/
|
|
15
|
+
"jobs #{@package_name} #{command.usage}" # rubocop:disable ThreadSafety/ClassInstanceVariable
|
|
16
16
|
end
|
|
17
17
|
# rubocop:enable Style/OptionalBooleanParameter
|
|
18
18
|
|
|
@@ -51,7 +51,7 @@ module SidekiqUniqueJobs
|
|
|
51
51
|
console_class.start
|
|
52
52
|
end
|
|
53
53
|
|
|
54
|
-
no_commands do
|
|
54
|
+
no_commands do
|
|
55
55
|
# :nodoc:
|
|
56
56
|
def digests
|
|
57
57
|
@digests ||= SidekiqUniqueJobs::Digests.new
|
|
@@ -2,33 +2,39 @@
|
|
|
2
2
|
|
|
3
3
|
module SidekiqUniqueJobs
|
|
4
4
|
# ThreadSafe config exists to be able to document the config class without errors
|
|
5
|
-
ThreadSafeConfig = Concurrent::MutableStruct.new(
|
|
6
|
-
|
|
7
|
-
|
|
8
|
-
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
|
|
5
|
+
ThreadSafeConfig = Concurrent::MutableStruct.new(
|
|
6
|
+
"ThreadSafeConfig",
|
|
7
|
+
:lock_timeout,
|
|
8
|
+
:lock_ttl,
|
|
9
|
+
:enabled,
|
|
10
|
+
:lock_prefix,
|
|
11
|
+
:logger,
|
|
12
|
+
:logger_enabled,
|
|
13
|
+
:locks,
|
|
14
|
+
:strategies,
|
|
15
|
+
:debug_lua,
|
|
16
|
+
:max_history,
|
|
17
|
+
:reaper,
|
|
18
|
+
:reaper_count,
|
|
19
|
+
:reaper_interval,
|
|
20
|
+
:reaper_timeout,
|
|
21
|
+
:reaper_resurrector_interval,
|
|
22
|
+
:reaper_resurrector_enabled,
|
|
23
|
+
:lock_info,
|
|
24
|
+
:raise_on_config_error,
|
|
25
|
+
:current_redis_version,
|
|
26
|
+
:digest_algorithm,
|
|
27
|
+
)
|
|
25
28
|
|
|
26
29
|
#
|
|
27
30
|
# Shared class for dealing with gem configuration
|
|
28
31
|
#
|
|
29
32
|
# @author Mauro Berlanda <mauro.berlanda@gmail.com>
|
|
30
|
-
# rubocop:disable Metrics/ClassLength
|
|
31
33
|
class Config < ThreadSafeConfig
|
|
34
|
+
def initialize(*)
|
|
35
|
+
super
|
|
36
|
+
@redis_version_mutex = Mutex.new
|
|
37
|
+
end
|
|
32
38
|
#
|
|
33
39
|
# @return [Hash<Symbol, SidekiqUniqueJobs::Lock::BaseLock] all available queued locks
|
|
34
40
|
LOCKS_WHILE_ENQUEUED = {
|
|
@@ -118,11 +124,9 @@ module SidekiqUniqueJobs
|
|
|
118
124
|
#
|
|
119
125
|
# @return [3600] check if reaper is dead each 3600 seconds
|
|
120
126
|
REAPER_RESURRECTOR_INTERVAL = 3600
|
|
121
|
-
|
|
122
127
|
#
|
|
123
128
|
# @return [false] enable reaper resurrector
|
|
124
129
|
REAPER_RESURRECTOR_ENABLED = false
|
|
125
|
-
|
|
126
130
|
#
|
|
127
131
|
# @return [false] while useful it also adds overhead so disable lock_info by default
|
|
128
132
|
USE_LOCK_INFO = false
|
|
@@ -132,6 +136,9 @@ module SidekiqUniqueJobs
|
|
|
132
136
|
#
|
|
133
137
|
# @return [0.0.0] default redis version is only to avoid NoMethodError on nil
|
|
134
138
|
REDIS_VERSION = "0.0.0"
|
|
139
|
+
#
|
|
140
|
+
# @return [:legacy] default digest algorithm :modern or :legacy
|
|
141
|
+
DIGEST_ALGORITHM = :legacy
|
|
135
142
|
|
|
136
143
|
#
|
|
137
144
|
# Returns a default configuration
|
|
@@ -177,7 +184,7 @@ module SidekiqUniqueJobs
|
|
|
177
184
|
#
|
|
178
185
|
# @return [SidekiqUniqueJobs::Config] a default configuration
|
|
179
186
|
#
|
|
180
|
-
def self.default
|
|
187
|
+
def self.default
|
|
181
188
|
new(
|
|
182
189
|
LOCK_TIMEOUT,
|
|
183
190
|
LOCK_TTL,
|
|
@@ -198,6 +205,7 @@ module SidekiqUniqueJobs
|
|
|
198
205
|
USE_LOCK_INFO,
|
|
199
206
|
RAISE_ON_CONFIG_ERROR,
|
|
200
207
|
REDIS_VERSION,
|
|
208
|
+
DIGEST_ALGORITHM,
|
|
201
209
|
)
|
|
202
210
|
end
|
|
203
211
|
|
|
@@ -210,8 +218,8 @@ module SidekiqUniqueJobs
|
|
|
210
218
|
# @return [<type>] <description>
|
|
211
219
|
#
|
|
212
220
|
def default_lock_ttl=(obj)
|
|
213
|
-
warn "[DEPRECATION] `#{class_name}##{__method__}` is deprecated." \
|
|
214
|
-
"
|
|
221
|
+
warn "[DEPRECATION] `#{class_name}##{__method__}` is deprecated. " \
|
|
222
|
+
"Please use `#{class_name}#lock_ttl=` instead."
|
|
215
223
|
self.lock_ttl = obj
|
|
216
224
|
end
|
|
217
225
|
|
|
@@ -224,8 +232,8 @@ module SidekiqUniqueJobs
|
|
|
224
232
|
# @return [Integer]
|
|
225
233
|
#
|
|
226
234
|
def default_lock_timeout=(obj)
|
|
227
|
-
warn "[DEPRECATION] `#{class_name}##{__method__}` is deprecated." \
|
|
228
|
-
"
|
|
235
|
+
warn "[DEPRECATION] `#{class_name}##{__method__}` is deprecated. " \
|
|
236
|
+
"Please use `#{class_name}#lock_timeout=` instead."
|
|
229
237
|
self.lock_timeout = obj
|
|
230
238
|
end
|
|
231
239
|
|
|
@@ -236,8 +244,8 @@ module SidekiqUniqueJobs
|
|
|
236
244
|
# @return [nil, Integer] configured value or nil
|
|
237
245
|
#
|
|
238
246
|
def default_lock_ttl
|
|
239
|
-
warn "[DEPRECATION] `#{class_name}##{__method__}` is deprecated." \
|
|
240
|
-
"
|
|
247
|
+
warn "[DEPRECATION] `#{class_name}##{__method__}` is deprecated. " \
|
|
248
|
+
"Please use `#{class_name}#lock_ttl` instead."
|
|
241
249
|
lock_ttl
|
|
242
250
|
end
|
|
243
251
|
|
|
@@ -249,8 +257,8 @@ module SidekiqUniqueJobs
|
|
|
249
257
|
# @return [nil, Integer] configured value or nil
|
|
250
258
|
#
|
|
251
259
|
def default_lock_timeout
|
|
252
|
-
warn "[DEPRECATION] `#{class_name}##{__method__}` is deprecated." \
|
|
253
|
-
"
|
|
260
|
+
warn "[DEPRECATION] `#{class_name}##{__method__}` is deprecated. " \
|
|
261
|
+
"Please use `#{class_name}#lock_timeout` instead."
|
|
254
262
|
lock_timeout
|
|
255
263
|
end
|
|
256
264
|
|
|
@@ -304,14 +312,38 @@ module SidekiqUniqueJobs
|
|
|
304
312
|
self.strategies = new_strategies
|
|
305
313
|
end
|
|
306
314
|
|
|
315
|
+
#
|
|
316
|
+
# Sets digest_algorithm to either :modern or :legacy
|
|
317
|
+
#
|
|
318
|
+
# @param [Symbol] value
|
|
319
|
+
#
|
|
320
|
+
# @return [Symbol] the new value
|
|
321
|
+
#
|
|
322
|
+
def digest_algorithm=(value)
|
|
323
|
+
unless [:modern, :legacy].include?(value)
|
|
324
|
+
raise ArgumentError, "Invalid digest algorithm: #{value} (should be :modern or :legacy)"
|
|
325
|
+
end
|
|
326
|
+
|
|
327
|
+
super
|
|
328
|
+
end
|
|
329
|
+
|
|
307
330
|
#
|
|
308
331
|
# The current version of redis
|
|
309
332
|
#
|
|
333
|
+
# Thread-safe: Uses mutex to prevent multiple threads from fetching version simultaneously
|
|
310
334
|
#
|
|
311
335
|
# @return [String] a version string eg. `5.0.1`
|
|
312
336
|
#
|
|
313
337
|
def redis_version
|
|
314
|
-
|
|
338
|
+
# Fast path: if already fetched, return immediately without locking
|
|
339
|
+
return current_redis_version if current_redis_version != REDIS_VERSION
|
|
340
|
+
|
|
341
|
+
# Slow path: fetch version with mutex protection
|
|
342
|
+
@redis_version_mutex.synchronize do
|
|
343
|
+
# Double-check inside mutex in case another thread just fetched it
|
|
344
|
+
self.current_redis_version = SidekiqUniqueJobs.fetch_redis_version if current_redis_version == REDIS_VERSION
|
|
345
|
+
end
|
|
346
|
+
|
|
315
347
|
current_redis_version
|
|
316
348
|
end
|
|
317
349
|
end
|