sidekiq-unique-jobs 7.0.12 → 7.1.3
Sign up to get free protection for your applications and to get access to all the features.
Potentially problematic release.
This version of sidekiq-unique-jobs might be problematic. Click here for more details.
- checksums.yaml +4 -4
- data/CHANGELOG.md +59 -0
- data/README.md +541 -424
- data/lib/sidekiq_unique_jobs.rb +4 -0
- data/lib/sidekiq_unique_jobs/config.rb +12 -0
- data/lib/sidekiq_unique_jobs/constants.rb +0 -1
- data/lib/sidekiq_unique_jobs/deprecation.rb +35 -0
- data/lib/sidekiq_unique_jobs/exceptions.rb +9 -0
- data/lib/sidekiq_unique_jobs/lock/base_lock.rb +56 -51
- data/lib/sidekiq_unique_jobs/lock/until_and_while_executing.rb +31 -9
- data/lib/sidekiq_unique_jobs/lock/until_executed.rb +17 -5
- data/lib/sidekiq_unique_jobs/lock/until_executing.rb +15 -1
- data/lib/sidekiq_unique_jobs/lock/until_expired.rb +21 -0
- data/lib/sidekiq_unique_jobs/lock/while_executing.rb +11 -6
- data/lib/sidekiq_unique_jobs/lock_config.rb +3 -3
- data/lib/sidekiq_unique_jobs/locksmith.rb +84 -82
- data/lib/sidekiq_unique_jobs/middleware/client.rb +8 -10
- data/lib/sidekiq_unique_jobs/middleware/server.rb +2 -0
- data/lib/sidekiq_unique_jobs/on_conflict/reschedule.rb +7 -3
- data/lib/sidekiq_unique_jobs/options_with_fallback.rb +0 -9
- data/lib/sidekiq_unique_jobs/orphans/manager.rb +1 -0
- data/lib/sidekiq_unique_jobs/orphans/reaper_resurrector.rb +170 -0
- data/lib/sidekiq_unique_jobs/redis/sorted_set.rb +1 -1
- data/lib/sidekiq_unique_jobs/reflectable.rb +17 -0
- data/lib/sidekiq_unique_jobs/reflections.rb +68 -0
- data/lib/sidekiq_unique_jobs/script/caller.rb +3 -1
- data/lib/sidekiq_unique_jobs/server.rb +1 -0
- data/lib/sidekiq_unique_jobs/sidekiq_unique_jobs.rb +21 -0
- data/lib/sidekiq_unique_jobs/version.rb +1 -1
- metadata +7 -3
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 57accefc169e11923a88ad34751e2e7c5f2a2c25a8f672084d946d1c536a7a46
|
|
4
|
+
data.tar.gz: e019c6288d6b87fb0b306b18e0435095a81a56ab04046ca25485b531c418f692
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 1989a4f0d6a335be4f7b35e97028a32fabd39212dd3f914471fa78c6654cefacc9e2725654299a10b7df5c55f0778566283f7e5e0ba01542de7900995dcfb734
|
|
7
|
+
data.tar.gz: d9be5add9dbaac2c523dbb6ed8dfe2879026087132c4812a2a03e4e58945cf481047aff00ed49e78942d8450749fc2864b09b44e6646dd4125452568cbc902ec
|
data/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,64 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## [Unreleased](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/HEAD)
|
|
4
|
+
|
|
5
|
+
[Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v7.1.2...HEAD)
|
|
6
|
+
|
|
7
|
+
**Fixed bugs:**
|
|
8
|
+
|
|
9
|
+
- Documentation fixes [\#622](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/622) ([ursm](https://github.com/ursm))
|
|
10
|
+
|
|
11
|
+
**Merged pull requests:**
|
|
12
|
+
|
|
13
|
+
- Improve readme [\#621](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/621) ([mhenrixon](https://github.com/mhenrixon))
|
|
14
|
+
|
|
15
|
+
## [v7.1.2](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v7.1.2) (2021-07-01)
|
|
16
|
+
|
|
17
|
+
[Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v7.1.1...v7.1.2)
|
|
18
|
+
|
|
19
|
+
**Fixed bugs:**
|
|
20
|
+
|
|
21
|
+
- Ensure `limit` and `timeout` are always numbers [\#620](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/620) ([pinkahd](https://github.com/pinkahd))
|
|
22
|
+
|
|
23
|
+
## [v7.1.1](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v7.1.1) (2021-06-30)
|
|
24
|
+
|
|
25
|
+
[Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v7.1.0...v7.1.1)
|
|
26
|
+
|
|
27
|
+
**Fixed bugs:**
|
|
28
|
+
|
|
29
|
+
- Fix handling of lock timeout [\#619](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/619) ([mhenrixon](https://github.com/mhenrixon))
|
|
30
|
+
|
|
31
|
+
**Closed issues:**
|
|
32
|
+
|
|
33
|
+
- Max expiration for locks [\#593](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/593)
|
|
34
|
+
|
|
35
|
+
## [v7.1.0](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v7.1.0) (2021-06-29)
|
|
36
|
+
|
|
37
|
+
[Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v7.0.12...v7.1.0)
|
|
38
|
+
|
|
39
|
+
**Implemented enhancements:**
|
|
40
|
+
|
|
41
|
+
- Reflections [\#611](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/611) ([mhenrixon](https://github.com/mhenrixon))
|
|
42
|
+
- Start new orphan reaper process if orphan reaper is not running [\#604](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/604) ([Assa1121](https://github.com/Assa1121))
|
|
43
|
+
|
|
44
|
+
**Fixed bugs:**
|
|
45
|
+
|
|
46
|
+
- Fix numerous small issues with locking [\#616](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/616) ([mhenrixon](https://github.com/mhenrixon))
|
|
47
|
+
- Allow locksmith delete to work with strings [\#615](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/615) ([pinkahd](https://github.com/pinkahd))
|
|
48
|
+
|
|
49
|
+
## [v7.0.12](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v7.0.12) (2021-06-04)
|
|
50
|
+
|
|
51
|
+
[Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v7.0.11...v7.0.12)
|
|
52
|
+
|
|
53
|
+
**Implemented enhancements:**
|
|
54
|
+
|
|
55
|
+
- Reduce noise of perfectly valid scenario [\#610](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/610) ([mhenrixon](https://github.com/mhenrixon))
|
|
56
|
+
|
|
57
|
+
**Merged pull requests:**
|
|
58
|
+
|
|
59
|
+
- Set correct namespace for custom strategy example [\#609](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/609) ([Wolfer](https://github.com/Wolfer))
|
|
60
|
+
- Clarify the documentation related to lock\_ttl [\#607](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/607) ([donaldpiret](https://github.com/donaldpiret))
|
|
61
|
+
|
|
3
62
|
## [v7.0.11](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v7.0.11) (2021-05-16)
|
|
4
63
|
|
|
5
64
|
[Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v7.0.10...v7.0.11)
|
data/README.md
CHANGED
|
@@ -2,6 +2,10 @@
|
|
|
2
2
|
|
|
3
3
|
[](https://gitter.im/mhenrixon/sidekiq-unique-jobs?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)  [](https://codeclimate.com/github/mhenrixon/sidekiq-unique-jobs) [](https://codeclimate.com/github/mhenrixon/sidekiq-unique-jobs/coverage)
|
|
4
4
|
|
|
5
|
+
## Support Me
|
|
6
|
+
|
|
7
|
+
Want to show me some ❤️ for the hard work I do on this gem? You can use the following PayPal link: [https://paypal.me/mhenrixon1](https://paypal.me/mhenrixon1). Any amount is welcome and let me tell you it feels good to be appreciated. Even a dollar makes me super excited about all of this.
|
|
8
|
+
|
|
5
9
|
<!-- MarkdownTOC -->
|
|
6
10
|
|
|
7
11
|
- [Introduction](#introduction)
|
|
@@ -9,35 +13,18 @@
|
|
|
9
13
|
- [Installation](#installation)
|
|
10
14
|
- [Add the middleware](#add-the-middleware)
|
|
11
15
|
- [Your first worker](#your-first-worker)
|
|
12
|
-
- [Support Me](#support-me)
|
|
13
16
|
- [Requirements](#requirements)
|
|
14
|
-
- [General Information](#general-information)
|
|
15
|
-
- [Global Configuration](#global-configuration)
|
|
16
|
-
- [debug_lua](#debug_lua)
|
|
17
|
-
- [lock_timeout](#lock_timeout)
|
|
18
|
-
- [lock_ttl](#lock_ttl)
|
|
19
|
-
- [enabled](#enabled)
|
|
20
|
-
- [logger](#logger)
|
|
21
|
-
- [max_history](#max_history)
|
|
22
|
-
- [reaper](#reaper)
|
|
23
|
-
- [reaper_count](#reaper_count)
|
|
24
|
-
- [reaper_interval](#reaper_interval)
|
|
25
|
-
- [reaper_timeout](#reaper_timeout)
|
|
26
|
-
- [lock_prefix](#lock_prefix)
|
|
27
|
-
- [lock_info](#lock_info)
|
|
28
|
-
- [Worker Configuration](#worker-configuration)
|
|
29
|
-
- [lock_info](#lock_info-1)
|
|
30
|
-
- [lock_prefix](#lock_prefix-1)
|
|
31
|
-
- [lock_ttl](#lock_ttl-1)
|
|
32
|
-
- [lock_timeout](#lock_timeout-1)
|
|
33
|
-
- [unique_across_queues](#unique_across_queues)
|
|
34
|
-
- [unique_across_workers](#unique_across_workers)
|
|
35
17
|
- [Locks](#locks)
|
|
36
18
|
- [Until Executing](#until-executing)
|
|
19
|
+
- [Example worker](#example-worker)
|
|
37
20
|
- [Until Executed](#until-executed)
|
|
21
|
+
- [Example worker](#example-worker-1)
|
|
38
22
|
- [Until Expired](#until-expired)
|
|
23
|
+
- [Example worker](#example-worker-2)
|
|
39
24
|
- [Until And While Executing](#until-and-while-executing)
|
|
25
|
+
- [Example worker](#example-worker-3)
|
|
40
26
|
- [While Executing](#while-executing)
|
|
27
|
+
- [Example worker](#example-worker-4)
|
|
41
28
|
- [Custom Locks](#custom-locks)
|
|
42
29
|
- [Conflict Strategy](#conflict-strategy)
|
|
43
30
|
- [log](#log)
|
|
@@ -46,23 +33,54 @@
|
|
|
46
33
|
- [replace](#replace)
|
|
47
34
|
- [Reschedule](#reschedule)
|
|
48
35
|
- [Custom Strategies](#custom-strategies)
|
|
49
|
-
- [
|
|
50
|
-
- [Finer Control over Uniqueness](#finer-control-over-uniqueness)
|
|
51
|
-
- [After Unlock Callback](#after-unlock-callback)
|
|
52
|
-
- [Logging](#logging)
|
|
53
|
-
- [Cleanup Dead Locks](#cleanup-dead-locks)
|
|
54
|
-
- [Other Sidekiq gems](#other-sidekiq-gems)
|
|
55
|
-
- [apartment-sidekiq](#apartment-sidekiq)
|
|
56
|
-
- [sidekiq-global_id](#sidekiq-global_id)
|
|
57
|
-
- [sidekiq-status](#sidekiq-status)
|
|
36
|
+
- [3 Cleanup Dead Locks](#3-cleanup-dead-locks)
|
|
58
37
|
- [Debugging](#debugging)
|
|
59
38
|
- [Sidekiq Web](#sidekiq-web)
|
|
39
|
+
- [Reflections \(metrics, logging, etc.\)](#reflections-metrics-logging-etc)
|
|
40
|
+
- [after_unlock_callback_failed](#after_unlock_callback_failed)
|
|
41
|
+
- [error](#error)
|
|
42
|
+
- [execution_failed](#execution_failed)
|
|
43
|
+
- [lock_failed](#lock_failed)
|
|
44
|
+
- [locked](#locked)
|
|
45
|
+
- [reschedule_failed](#reschedule_failed)
|
|
46
|
+
- [rescheduled](#rescheduled)
|
|
47
|
+
- [timeout](#timeout)
|
|
48
|
+
- [unlock_failed](#unlock_failed)
|
|
49
|
+
- [unlocked](#unlocked)
|
|
50
|
+
- [unknown_sidekiq_worker](#unknown_sidekiq_worker)
|
|
60
51
|
- [Show Locks](#show-locks)
|
|
61
52
|
- [Show Lock](#show-lock)
|
|
62
|
-
- [Communication](#communication)
|
|
63
53
|
- [Testing](#testing)
|
|
64
|
-
- [
|
|
54
|
+
- [Validating Worker Configuration](#validating-worker-configuration)
|
|
65
55
|
- [Uniqueness](#uniqueness)
|
|
56
|
+
- [Configuration](#configuration)
|
|
57
|
+
- [Other Sidekiq gems](#other-sidekiq-gems)
|
|
58
|
+
- [apartment-sidekiq](#apartment-sidekiq)
|
|
59
|
+
- [sidekiq-global_id](#sidekiq-global_id)
|
|
60
|
+
- [sidekiq-status](#sidekiq-status)
|
|
61
|
+
- [Global Configuration](#global-configuration)
|
|
62
|
+
- [debug_lua](#debug_lua)
|
|
63
|
+
- [lock_timeout](#lock_timeout)
|
|
64
|
+
- [lock_ttl](#lock_ttl)
|
|
65
|
+
- [enabled](#enabled)
|
|
66
|
+
- [logger](#logger)
|
|
67
|
+
- [max_history](#max_history)
|
|
68
|
+
- [reaper](#reaper)
|
|
69
|
+
- [reaper_count](#reaper_count)
|
|
70
|
+
- [reaper_interval](#reaper_interval)
|
|
71
|
+
- [reaper_timeout](#reaper_timeout)
|
|
72
|
+
- [lock_prefix](#lock_prefix)
|
|
73
|
+
- [lock_info](#lock_info)
|
|
74
|
+
- [Worker Configuration](#worker-configuration)
|
|
75
|
+
- [lock_info](#lock_info-1)
|
|
76
|
+
- [lock_prefix](#lock_prefix-1)
|
|
77
|
+
- [lock_ttl](#lock_ttl-1)
|
|
78
|
+
- [lock_timeout](#lock_timeout-1)
|
|
79
|
+
- [unique_across_queues](#unique_across_queues)
|
|
80
|
+
- [unique_across_workers](#unique_across_workers)
|
|
81
|
+
- [Finer Control over Uniqueness](#finer-control-over-uniqueness)
|
|
82
|
+
- [After Unlock Callback](#after-unlock-callback)
|
|
83
|
+
- [Communication](#communication)
|
|
66
84
|
- [Contributing](#contributing)
|
|
67
85
|
- [Contributors](#contributors)
|
|
68
86
|
|
|
@@ -70,7 +88,9 @@
|
|
|
70
88
|
|
|
71
89
|
## Introduction
|
|
72
90
|
|
|
73
|
-
This gem adds unique constraints to
|
|
91
|
+
This gem adds unique constraints to sidekiq jobs. The uniqueness is achieved by creating a set of keys in redis based off of `queue`, `class`, `args` (in the sidekiq job hash).
|
|
92
|
+
|
|
93
|
+
By default, only one lock for a given hash can be acquired. What happens when a lock can't be acquired is governed by a chosen [Conflict Strategy](#conflict-strategy) strategy. Unless a conflict strategy is chosen
|
|
74
94
|
|
|
75
95
|
This is the documentation for the master branch. You can find the documentation for each release by navigating to its tag.
|
|
76
96
|
|
|
@@ -130,20 +150,16 @@ end
|
|
|
130
150
|
|
|
131
151
|
### Your first worker
|
|
132
152
|
|
|
153
|
+
The most likely to be used worker is `:until_executed`. This type of lock creates a lock from when `UntilExecutedWorker.perform_async` is called until right after `UntilExecutedWorker.new.perform` has been called.
|
|
154
|
+
|
|
133
155
|
```ruby
|
|
134
156
|
# frozen_string_literal: true
|
|
135
157
|
|
|
136
158
|
class UntilExecutedWorker
|
|
137
159
|
include Sidekiq::Worker
|
|
138
160
|
|
|
139
|
-
sidekiq_options queue: :
|
|
140
|
-
|
|
141
|
-
lock: :until_executed,
|
|
142
|
-
lock_info: true,
|
|
143
|
-
lock_timeout: 0,
|
|
144
|
-
lock_prefix: "special",
|
|
145
|
-
lock_ttl: 0,
|
|
146
|
-
lock_limit: 5
|
|
161
|
+
sidekiq_options queue: :until_executed,
|
|
162
|
+
lock: :until_executed
|
|
147
163
|
|
|
148
164
|
def perform
|
|
149
165
|
logger.info("cowboy")
|
|
@@ -151,15 +167,10 @@ class UntilExecutedWorker
|
|
|
151
167
|
logger.info("beebop")
|
|
152
168
|
end
|
|
153
169
|
end
|
|
154
|
-
|
|
155
170
|
```
|
|
156
171
|
|
|
157
172
|
You can read more about the worker configuration in [Worker Configuration](#worker-configuration) below.
|
|
158
173
|
|
|
159
|
-
## Support Me
|
|
160
|
-
|
|
161
|
-
Want to show me some ❤️ for the hard work I do on this gem? You can use the following PayPal link: [https://paypal.me/mhenrixon1](https://paypal.me/mhenrixon1). Any amount is welcome and let me tell you it feels good to be appreciated. Even a dollar makes me super excited about all of this.
|
|
162
|
-
|
|
163
174
|
## Requirements
|
|
164
175
|
|
|
165
176
|
- Sidekiq `>= 5.0` (`>= 5.2` recommended)
|
|
@@ -173,280 +184,110 @@ Want to show me some ❤️ for the hard work I do on this gem? You can use the
|
|
|
173
184
|
|
|
174
185
|
See [Sidekiq requirements][24] for detailed requirements of Sidekiq itself (be sure to check the right sidekiq version).
|
|
175
186
|
|
|
176
|
-
##
|
|
177
|
-
|
|
178
|
-
See [Interaction w/ Sidekiq](https://github.com/mhenrixon/sidekiq-unique-jobs/wiki/How-this-gem-interacts-with-Sidekiq) on how the gem interacts with Sidekiq.
|
|
179
|
-
|
|
180
|
-
See [Locking & Unlocking](https://github.com/mhenrixon/sidekiq-unique-jobs/wiki/Locking-&-Unlocking) for an overview of the differences on when the various lock types are locked and unlocked.
|
|
181
|
-
|
|
182
|
-
## Global Configuration
|
|
183
|
-
|
|
184
|
-
The gem supports a few different configuration options that might be of interest if you run into some weird issues.
|
|
185
|
-
|
|
186
|
-
Configure SidekiqUniqueJobs in an initializer or the sidekiq initializer on application startup.
|
|
187
|
-
|
|
188
|
-
```ruby
|
|
189
|
-
SidekiqUniqueJobs.configure do |config|
|
|
190
|
-
config.logger = Sidekiq.logger # default, change at your own discretion
|
|
191
|
-
config.debug_lua = false # Turn on when debugging
|
|
192
|
-
config.lock_info = false # Turn on when debugging
|
|
193
|
-
config.lock_ttl = 600 # Expire locks after 10 minutes
|
|
194
|
-
config.lock_timeout = nil # turn off lock timeout
|
|
195
|
-
config.max_history = 0 # Turn on when debugging
|
|
196
|
-
config.reaper = :ruby # :ruby, :lua or :none/nil
|
|
197
|
-
config.reaper_count = 1000 # Stop reaping after this many keys
|
|
198
|
-
config.reaper_interval = 600 # Reap orphans every 10 minutes
|
|
199
|
-
config.reaper_timeout = 150 # Timeout reaper after 2.5 minutes
|
|
200
|
-
end
|
|
201
|
-
```
|
|
202
|
-
|
|
203
|
-
### debug_lua
|
|
204
|
-
|
|
205
|
-
```ruby
|
|
206
|
-
SidekiqUniqueJobs.config.debug_lua #=> false
|
|
207
|
-
```
|
|
208
|
-
|
|
209
|
-
Turning on debug_lua will allow the lua scripts to output debug information about what the lua scripts do. It will log all redis commands that are executed and also some helpful messages about what is going on inside the lua script.
|
|
210
|
-
|
|
211
|
-
### lock_timeout
|
|
212
|
-
|
|
213
|
-
```ruby
|
|
214
|
-
SidekiqUniqueJobs.config.lock_timeout #=> 0
|
|
215
|
-
```
|
|
216
|
-
|
|
217
|
-
Set a global lock_timeout to use for all jobs that don't otherwise specify a lock_timeout.
|
|
218
|
-
|
|
219
|
-
Lock timeout decides how long to wait for acquiring the lock. A value of nil means to wait indefinitely for a lock resource to become available.
|
|
220
|
-
|
|
221
|
-
### lock_ttl
|
|
222
|
-
|
|
223
|
-
```ruby
|
|
224
|
-
SidekiqUniqueJobs.config.lock_ttl #=> nil
|
|
225
|
-
```
|
|
226
|
-
|
|
227
|
-
Set a global lock_ttl to use for all jobs that don't otherwise specify a lock_ttl.
|
|
228
|
-
|
|
229
|
-
Lock TTL decides how long to wait at most before considering a lock to be expired and making it possible to reuse that lock.
|
|
230
|
-
|
|
231
|
-
### enabled
|
|
232
|
-
|
|
233
|
-
```ruby
|
|
234
|
-
SidekiqUniqueJobs.config.enabled #=> true
|
|
235
|
-
```
|
|
236
|
-
|
|
237
|
-
Globally turn the locking mechanism on or off.
|
|
238
|
-
|
|
239
|
-
### logger
|
|
240
|
-
|
|
241
|
-
```ruby
|
|
242
|
-
SidekiqUniqueJobs.config.logger #=> #<Sidekiq::Logger:0x00007fdc1f96d180>
|
|
243
|
-
```
|
|
244
|
-
|
|
245
|
-
By default this gem piggybacks on the Sidekiq logger. It is not recommended to change this as the gem uses some features in the Sidekiq logger and you might run into problems. If you need a different logger and you do run into problems then get in touch and we'll see what we can do about it.
|
|
246
|
-
|
|
247
|
-
### max_history
|
|
248
|
-
|
|
249
|
-
```ruby
|
|
250
|
-
SidekiqUniqueJobs.config.max_history #=> 1_000
|
|
251
|
-
```
|
|
252
|
-
|
|
253
|
-
The max_history setting can be used to tweak the number of changelogs generated. It can also be completely turned off if performance suffers or if you are just not interested in using the changelog.
|
|
254
|
-
|
|
255
|
-
This is a log that can be accessed by a lock to see what happened for that lock. Any items after the configured `max_history` will be automatically deleted as new items are added.
|
|
256
|
-
|
|
257
|
-
### reaper
|
|
258
|
-
|
|
259
|
-
```ruby
|
|
260
|
-
SidekiqUniqueJobs.config.reaper #=> :ruby
|
|
261
|
-
```
|
|
262
|
-
|
|
263
|
-
If using the orphans cleanup process it is critical to be aware of the following. The `:ruby` job is much slower but the `:lua` job locks redis while executing. While doing intense processing it is best to avoid locking redis with a lua script. There for the batch size (controlled by the `reaper_count` setting) needs to be reduced.
|
|
264
|
-
|
|
265
|
-
In my benchmarks deleting 1000 orphaned locks with lua performs around 65% faster than deleting 1000 keys in ruby.
|
|
266
|
-
|
|
267
|
-
On the other hand if I increase it to 10 000 orphaned locks per cleanup (`reaper_count: 10_0000`) then redis starts throwing:
|
|
268
|
-
|
|
269
|
-
> BUSY Redis is busy running a script. You can only call SCRIPT KILL or SHUTDOWN NOSAVE. (Redis::CommandError)
|
|
270
|
-
|
|
271
|
-
If you want to disable the reaper set it to `:none`, `nil` or `false`. Actually, any value that isn't `:ruby` or `:lua` will disable the reaping.
|
|
272
|
-
|
|
273
|
-
```ruby
|
|
274
|
-
SidekiqUniqueJobs.config.reaper = :none
|
|
275
|
-
SidekiqUniqueJobs.config.reaper = nil
|
|
276
|
-
SidekiqUniqueJobs.config.reaper = false
|
|
277
|
-
```
|
|
278
|
-
|
|
279
|
-
### reaper_count
|
|
280
|
-
|
|
281
|
-
```ruby
|
|
282
|
-
SidekiqUniqueJobs.config.reaper_count #=> 1_000
|
|
283
|
-
```
|
|
284
|
-
|
|
285
|
-
The reaper_count setting configures how many orphans at a time will be cleaned up by the orphan cleanup job. This might have to be tweaked depending on which orphan job is running.
|
|
286
|
-
|
|
287
|
-
### reaper_interval
|
|
288
|
-
|
|
289
|
-
```ruby
|
|
290
|
-
SidekiqUniqueJobs.config.reaper_interval #=> 600
|
|
291
|
-
```
|
|
292
|
-
|
|
293
|
-
The number of seconds between reaping.
|
|
294
|
-
|
|
295
|
-
### reaper_timeout
|
|
296
|
-
|
|
297
|
-
```ruby
|
|
298
|
-
SidekiqUniqueJobs.config.reaper_timeout #=> 10
|
|
299
|
-
```
|
|
300
|
-
|
|
301
|
-
The number of seconds to wait for the reaper to finish before raising a TimeoutError. This is done to ensure that the next time we reap isn't getting stuck due to the previous process already running.
|
|
302
|
-
|
|
303
|
-
### lock_prefix
|
|
187
|
+
## Locks
|
|
304
188
|
|
|
305
|
-
|
|
306
|
-
SidekiqUniqueJobs.config.lock_prefix #=> "uniquejobs"
|
|
307
|
-
```
|
|
189
|
+
### Until Executing
|
|
308
190
|
|
|
309
|
-
|
|
191
|
+
A lock is created when `UntilExecuting.perform_async` is called. Then it is either unlocked when `lock_ttl` is hit or before Sidekiq calls the `perform` method on your worker.
|
|
310
192
|
|
|
311
|
-
|
|
193
|
+
#### Example worker
|
|
312
194
|
|
|
313
195
|
```ruby
|
|
314
|
-
|
|
315
|
-
|
|
316
|
-
|
|
317
|
-
Using lock info will create an additional key for the lock with a json object containing information about the lock. This will be presented in the web interface and might help track down why some jobs are getting stuck.
|
|
318
|
-
|
|
319
|
-
## Worker Configuration
|
|
196
|
+
class UntilExecuting
|
|
197
|
+
include Sidekiq::Workers
|
|
320
198
|
|
|
321
|
-
|
|
322
|
-
|
|
323
|
-
Lock info gathers information about a specific lock. It collects things like which `lock_args` where used to compute the `lock_digest` that is used for maintaining uniqueness.
|
|
199
|
+
sidekiq_options lock: :until_executing
|
|
324
200
|
|
|
325
|
-
|
|
326
|
-
|
|
201
|
+
def perform(id)
|
|
202
|
+
# Do work
|
|
203
|
+
end
|
|
204
|
+
end
|
|
327
205
|
```
|
|
328
206
|
|
|
329
|
-
|
|
330
|
-
|
|
331
|
-
Use if you want a different key prefix for the keys in redis.
|
|
207
|
+
**NOTE** this is probably not so good for jobs that shouldn't be running simultaneously (aka slow jobs).
|
|
332
208
|
|
|
333
|
-
|
|
334
|
-
sidekiq_options lock_prefix: "uniquejobs" # this is the default value
|
|
335
|
-
```
|
|
209
|
+
The reason this type of lock exists is to fix the following problem: [sidekiq/issues/3471](https://github.com/mperham/sidekiq/issues/3471#issuecomment-300866335)
|
|
336
210
|
|
|
337
|
-
###
|
|
211
|
+
### Until Executed
|
|
338
212
|
|
|
339
|
-
|
|
213
|
+
A lock is created when `UntilExecuted.perform_async` is called. Then it is either unlocked when `lock_ttl` is hit or when Sidekiq has called the `perform` method on your worker.
|
|
340
214
|
|
|
341
|
-
|
|
215
|
+
#### Example worker
|
|
342
216
|
|
|
343
217
|
```ruby
|
|
344
|
-
|
|
345
|
-
|
|
346
|
-
```
|
|
218
|
+
class UntilExecuted
|
|
219
|
+
include Sidekiq::Workers
|
|
347
220
|
|
|
348
|
-
|
|
221
|
+
sidekiq_options lock: :until_executed
|
|
349
222
|
|
|
350
|
-
|
|
351
|
-
|
|
352
|
-
|
|
353
|
-
|
|
354
|
-
sidekiq_options lock_timeout: 5 # wait 5 seconds
|
|
355
|
-
sidekiq_options lock_timeout: nil # lock indefinitely, this process won't continue until it gets a lock. VERY DANGEROUS!!
|
|
223
|
+
def perform(id)
|
|
224
|
+
# Do work
|
|
225
|
+
end
|
|
226
|
+
end
|
|
356
227
|
```
|
|
357
228
|
|
|
358
|
-
###
|
|
229
|
+
### Until Expired
|
|
359
230
|
|
|
360
|
-
This
|
|
231
|
+
This lock behaves identically to the [Until Executed](#until-executed) except for one thing. This job won't be unlocked until the expiration is hit. For jobs that need to run only once per day, this would be the perfect lock. This way, we can't create more jobs until one day after this job was first pushed.
|
|
361
232
|
|
|
362
|
-
|
|
233
|
+
#### Example worker
|
|
363
234
|
|
|
364
235
|
```ruby
|
|
365
|
-
class
|
|
366
|
-
include Sidekiq::
|
|
236
|
+
class UntilExpired
|
|
237
|
+
include Sidekiq::Workers
|
|
367
238
|
|
|
368
|
-
sidekiq_options
|
|
239
|
+
sidekiq_options lock: :until_expired, lock_ttl: 1.day
|
|
369
240
|
|
|
370
|
-
def perform
|
|
241
|
+
def perform
|
|
242
|
+
# Do work
|
|
243
|
+
end
|
|
371
244
|
end
|
|
372
245
|
```
|
|
373
246
|
|
|
374
|
-
|
|
247
|
+
### Until And While Executing
|
|
375
248
|
|
|
376
|
-
|
|
249
|
+
This lock is a combination of two locks (`:until_executing` and `:while_executing`). Please see the configuration for [Until Executing](#until-executing) and [While Executing](#while-executing)
|
|
377
250
|
|
|
378
|
-
|
|
251
|
+
#### Example worker
|
|
379
252
|
|
|
380
253
|
```ruby
|
|
381
|
-
class
|
|
382
|
-
include Sidekiq::
|
|
383
|
-
|
|
384
|
-
sidekiq_options unique_across_workers: true, queue: 'default'
|
|
385
|
-
|
|
386
|
-
def perform(args); end
|
|
387
|
-
end
|
|
388
|
-
|
|
389
|
-
class WorkerTwo
|
|
390
|
-
include Sidekiq::Worker
|
|
391
|
-
|
|
392
|
-
sidekiq_options unique_across_workers: true, queue: 'default'
|
|
254
|
+
class UntilAndWhileExecutingWorker
|
|
255
|
+
include Sidekiq::Workers
|
|
393
256
|
|
|
394
|
-
|
|
257
|
+
sidekiq_options lock: :until_and_while_executing,
|
|
258
|
+
lock_timeout: 2,
|
|
259
|
+
on_conflict: {
|
|
260
|
+
client: :log,
|
|
261
|
+
server: :raise
|
|
262
|
+
}
|
|
263
|
+
def perform(id)
|
|
264
|
+
# Do work
|
|
265
|
+
end
|
|
395
266
|
end
|
|
396
|
-
|
|
397
|
-
|
|
398
|
-
WorkerOne.perform_async(1)
|
|
399
|
-
# => 'the jobs unique id'
|
|
400
|
-
|
|
401
|
-
WorkerTwo.perform_async(1)
|
|
402
|
-
# => nil because WorkerOne just stole the lock
|
|
403
|
-
```
|
|
404
|
-
|
|
405
|
-
## Locks
|
|
406
|
-
|
|
407
|
-
### Until Executing
|
|
408
|
-
|
|
409
|
-
```ruby
|
|
410
|
-
sidekiq_options lock: :until_executing
|
|
411
|
-
```
|
|
412
|
-
|
|
413
|
-
Locks from when the client pushes the job to the queue. Will be unlocked before the server starts processing the job.
|
|
414
|
-
|
|
415
|
-
**NOTE** this is probably not so good for jobs that shouldn't be running simultaneously (aka slow jobs).
|
|
416
|
-
|
|
417
|
-
The reason this type of lock exists is to fix the following problem: [sidekiq/issues/3471](https://github.com/mperham/sidekiq/issues/3471#issuecomment-300866335)
|
|
418
|
-
|
|
419
|
-
### Until Executed
|
|
420
|
-
|
|
421
|
-
```ruby
|
|
422
|
-
sidekiq_options lock: :until_executed
|
|
423
267
|
```
|
|
424
268
|
|
|
425
|
-
|
|
426
|
-
|
|
427
|
-
### Until Expired
|
|
428
|
-
|
|
429
|
-
```ruby
|
|
430
|
-
sidekiq_options lock: :until_expired
|
|
431
|
-
```
|
|
269
|
+
### While Executing
|
|
432
270
|
|
|
433
|
-
|
|
271
|
+
Tese locks are put on a queue without any type of locking mechanism, the locking doesn't happen until Sidekiq pops the job from the queue and starts processing it.
|
|
434
272
|
|
|
435
|
-
|
|
273
|
+
#### Example worker
|
|
436
274
|
|
|
437
275
|
```ruby
|
|
438
|
-
|
|
439
|
-
|
|
440
|
-
|
|
441
|
-
Locks when the client pushes the job to the queue. The queue will be unlocked when the server starts processing the job. The server then goes on to creating a runtime lock for the job to prevent simultaneous jobs from being executed. As soon as the server starts processing a job, the client can push the same job to the queue.
|
|
276
|
+
class WhileExecutingWorker
|
|
277
|
+
include Sidekiq::Workers
|
|
442
278
|
|
|
443
|
-
|
|
444
|
-
|
|
445
|
-
|
|
446
|
-
|
|
279
|
+
sidekiq_options lock: :while_executing,
|
|
280
|
+
lock_timeout: 2,
|
|
281
|
+
on_conflict: {
|
|
282
|
+
server: :raise
|
|
283
|
+
}
|
|
284
|
+
def perform(id)
|
|
285
|
+
# Do work
|
|
286
|
+
end
|
|
287
|
+
end
|
|
447
288
|
```
|
|
448
289
|
|
|
449
|
-
|
|
290
|
+
**NOTE** Unless a conflict strategy of `:raise` is specified, if lock fails, the job will be dropped without notice. When told to raise, the job will be put back and retried. It would also be possible to use `:reschedule` with this lock.
|
|
450
291
|
|
|
451
292
|
**NOTE** Unless this job is configured with a `lock_timeout: nil` or `lock_timeout: > 0` then all jobs that are attempted to be executed will just be dropped without waiting.
|
|
452
293
|
|
|
@@ -595,133 +436,223 @@ sidekiq_options lock: :while_executing, on_conflict: :my_custom_strategy
|
|
|
595
436
|
|
|
596
437
|
Please not that if you try to override a default lock, an `ArgumentError` will be raised.
|
|
597
438
|
|
|
598
|
-
|
|
439
|
+
### 3 Cleanup Dead Locks
|
|
599
440
|
|
|
600
|
-
|
|
441
|
+
For sidekiq versions < 5.1 a `sidekiq_retries_exhausted` block is required per worker class. This is deprecated in Sidekiq 6.0
|
|
601
442
|
|
|
602
443
|
```ruby
|
|
603
|
-
|
|
444
|
+
class MyWorker
|
|
445
|
+
sidekiq_retries_exhausted do |msg, _ex|
|
|
446
|
+
digest = msg['lock_digest']
|
|
447
|
+
SidekiqUniqueJobs::Digests.new.delete_by_digest(digest) if digest
|
|
448
|
+
end
|
|
449
|
+
end
|
|
604
450
|
```
|
|
605
451
|
|
|
606
|
-
|
|
607
|
-
|
|
608
|
-
### Finer Control over Uniqueness
|
|
452
|
+
Starting in v5.1, Sidekiq can also fire a global callback when a job dies: In version 7, this is handled automatically for you. You don't need to add a death handler, if you configure v7 like in [Add the middleware](#add-the-middleware) you don't have to worry about the below.
|
|
609
453
|
|
|
610
|
-
|
|
611
|
-
|
|
612
|
-
|
|
454
|
+
```ruby
|
|
455
|
+
Sidekiq.configure_server do |config|
|
|
456
|
+
config.death_handlers << ->(job, _ex) do
|
|
457
|
+
digest = job['lock_digest']
|
|
458
|
+
SidekiqUniqueJobs::Digests.new.delete_by_digest(digest) if digest
|
|
459
|
+
end
|
|
460
|
+
end
|
|
461
|
+
```
|
|
613
462
|
|
|
614
|
-
|
|
463
|
+
## Debugging
|
|
615
464
|
|
|
616
|
-
The
|
|
465
|
+
There are several ways of removing keys that are stuck. The prefered way is by using the unique extension to `Sidekiq::Web`. The old console and command line versions still work but might be deprecated in the future. It is better to search for the digest itself and delete the keys matching that digest.
|
|
617
466
|
|
|
618
|
-
|
|
619
|
-
class UniqueJobWithFilterMethod
|
|
620
|
-
include Sidekiq::Worker
|
|
621
|
-
sidekiq_options lock: :until_and_while_executing,
|
|
622
|
-
lock_args_method: :lock_args # this is default and will be used if such a method is defined
|
|
467
|
+
### Sidekiq Web
|
|
623
468
|
|
|
624
|
-
|
|
625
|
-
[ args[0], args[2][:type] ]
|
|
626
|
-
end
|
|
469
|
+
To use the web extension you need to require it in your routes.
|
|
627
470
|
|
|
628
|
-
|
|
471
|
+
```ruby
|
|
472
|
+
#app/config/routes.rb
|
|
473
|
+
require 'sidekiq_unique_jobs/web'
|
|
474
|
+
mount Sidekiq::Web, at: '/sidekiq'
|
|
475
|
+
```
|
|
629
476
|
|
|
630
|
-
|
|
477
|
+
There is no need to `require 'sidekiq/web'` since `sidekiq_unique_jobs/web`
|
|
478
|
+
already does this.
|
|
631
479
|
|
|
632
|
-
|
|
633
|
-
include Sidekiq::Worker
|
|
634
|
-
sidekiq_options lock: :until_executed,
|
|
635
|
-
lock_args_method: ->(args) { [ args.first ] }
|
|
480
|
+
To filter/search for keys we can use the wildcard `*`. If we have a unique digest `'uniquejobs:9e9b5ce5d423d3ea470977004b50ff84` we can search for it by enter `*ff84` and it should return all digests that end with `ff84`.
|
|
636
481
|
|
|
637
|
-
|
|
482
|
+
### Reflections (metrics, logging, etc.)
|
|
638
483
|
|
|
639
|
-
|
|
640
|
-
```
|
|
484
|
+
To be able to gather some insights on what is going on inside this gem. I provide a reflection API that can be used.
|
|
641
485
|
|
|
642
|
-
|
|
486
|
+
To setup reflections for logging or metrics, use the following API:
|
|
643
487
|
|
|
644
488
|
```ruby
|
|
645
|
-
class UniqueJobWithFilterMethod
|
|
646
|
-
include Sidekiq::Worker
|
|
647
|
-
sidekiq_options lock: :until_and_while_executing, lock_args_method: :lock_args
|
|
648
489
|
|
|
649
|
-
|
|
650
|
-
|
|
651
|
-
|
|
652
|
-
|
|
653
|
-
|
|
654
|
-
|
|
655
|
-
|
|
656
|
-
|
|
490
|
+
def extract_log_from_job(message, job_hash)
|
|
491
|
+
worker = job_hash['class']
|
|
492
|
+
args = job_hash['args']
|
|
493
|
+
lock_args = job_hash['lock_args']
|
|
494
|
+
queue = job_hash['queue']
|
|
495
|
+
{
|
|
496
|
+
message: message,
|
|
497
|
+
worker: worker,
|
|
498
|
+
args: args,
|
|
499
|
+
lock_args: lock_args,
|
|
500
|
+
queue: queue
|
|
501
|
+
}
|
|
502
|
+
end
|
|
503
|
+
|
|
504
|
+
SidekiqUniqueJobs.reflect do |on|
|
|
505
|
+
on.lock_failed do |job_hash|
|
|
506
|
+
message = extract_log_from_job('Lock Failed', job_hash)
|
|
507
|
+
Sidekiq.logger.warn(message)
|
|
657
508
|
end
|
|
658
509
|
end
|
|
659
510
|
```
|
|
660
511
|
|
|
661
|
-
|
|
512
|
+
#### after_unlock_callback_failed
|
|
662
513
|
|
|
663
|
-
|
|
514
|
+
This is called when you have configured a custom callback for when a lock has been released.
|
|
664
515
|
|
|
665
|
-
|
|
666
|
-
**Exception 2:** UntilExpired expires eventually, no after_unlock hook is called.
|
|
516
|
+
#### error
|
|
667
517
|
|
|
668
|
-
|
|
518
|
+
Not in use yet but will be used deep into the stack to provide a means to catch and report errors inside the gem.
|
|
669
519
|
|
|
670
|
-
|
|
671
|
-
class UniqueJobWithFilterMethod
|
|
672
|
-
include Sidekiq::Worker
|
|
673
|
-
sidekiq_options lock: :while_executing,
|
|
520
|
+
#### execution_failed
|
|
674
521
|
|
|
675
|
-
|
|
676
|
-
# block has yielded and lock is released
|
|
677
|
-
end
|
|
522
|
+
When the sidekiq processor picks the job of the queue for certain jobs but your job raised an error to the middleware. This will be the reflection. It is probably nothing to worry about. When your worker raises an error, we need to handle some edge cases for until and while executing.
|
|
678
523
|
|
|
679
|
-
|
|
680
|
-
|
|
681
|
-
|
|
682
|
-
|
|
683
|
-
|
|
684
|
-
|
|
524
|
+
#### lock_failed
|
|
525
|
+
|
|
526
|
+
If we can't achieve a lock, this will be the reflection. It most likely is nothing to worry about. We just couldn't retrieve a lock in a timely fashion.
|
|
527
|
+
|
|
528
|
+
The biggest reason for this reflection would be to gather metrics on which workers fail the most at the locking step for example.
|
|
529
|
+
|
|
530
|
+
#### locked
|
|
531
|
+
|
|
532
|
+
For when a lock has been successful. Again, mostly useful for metrics I suppose.
|
|
533
|
+
|
|
534
|
+
#### reschedule_failed
|
|
535
|
+
|
|
536
|
+
For when the reschedule strategy failed to reschedule the job.
|
|
537
|
+
|
|
538
|
+
#### rescheduled
|
|
539
|
+
|
|
540
|
+
For when a job was successfully rescheduled
|
|
541
|
+
|
|
542
|
+
#### timeout
|
|
543
|
+
|
|
544
|
+
This is also mostly useful for reporting/metrics purposes. What this reflection does is signal that the job was configured to wait (`lock_timeout` was configured), but we couldn't retrieve a lock even though we waited for some time.
|
|
545
|
+
|
|
546
|
+
### unlock_failed
|
|
547
|
+
|
|
548
|
+
This is not got, this is worth
|
|
549
|
+
|
|
550
|
+
### unlocked
|
|
551
|
+
|
|
552
|
+
Also mostly useful for reporting purposes. The job was successfully unlocked.
|
|
553
|
+
|
|
554
|
+
### unknown_sidekiq_worker
|
|
555
|
+
|
|
556
|
+
The reason this happens is that the server couldn't find a valid sidekiq worker class. Most likely, that worker isn't intended to be processed by this sidekiq server instance.
|
|
557
|
+
|
|
558
|
+
#### Show Locks
|
|
559
|
+
|
|
560
|
+
|
|
561
|
+
|
|
562
|
+
#### Show Lock
|
|
563
|
+
|
|
564
|
+
|
|
685
565
|
|
|
686
|
-
|
|
566
|
+
## Testing
|
|
687
567
|
|
|
688
|
-
|
|
568
|
+
### Validating Worker Configuration
|
|
569
|
+
|
|
570
|
+
Since v7 it is possible to perform some simple validation against your workers sidekiq_options. What it does is scan for some issues that are known to cause problems in production.
|
|
571
|
+
|
|
572
|
+
Let's take a _bad_ worker:
|
|
689
573
|
|
|
690
574
|
```ruby
|
|
691
|
-
|
|
692
|
-
|
|
693
|
-
sidekiq_options lock: :while_executing,
|
|
694
|
-
|
|
575
|
+
#app/workers/bad_worker.rb
|
|
576
|
+
class BadWorker
|
|
577
|
+
sidekiq_options lock: :while_executing, on_conflict: :replace
|
|
578
|
+
end
|
|
695
579
|
|
|
696
|
-
|
|
580
|
+
#spec/workers/bad_worker_spec.rb
|
|
697
581
|
|
|
582
|
+
require "sidekiq_unique_jobs/testing"
|
|
583
|
+
#OR
|
|
584
|
+
require "sidekiq_unique_jobs/rspec/matchers"
|
|
585
|
+
|
|
586
|
+
RSpec.describe BadWorker do
|
|
587
|
+
specify { expect(described_class).to have_valid_sidekiq_options }
|
|
698
588
|
end
|
|
699
589
|
```
|
|
700
590
|
|
|
701
|
-
|
|
591
|
+
This gives us a helpful error message for a wrongly configured worker:
|
|
592
|
+
|
|
593
|
+
```bash
|
|
594
|
+
Expected BadWorker to have valid sidekiq options but found the following problems:
|
|
595
|
+
on_server_conflict: :replace is incompatible with the server process
|
|
596
|
+
```
|
|
597
|
+
|
|
598
|
+
If you are not using RSpec (a lot of people prefer minitest or test unit) you can do something like:
|
|
599
|
+
|
|
600
|
+
```ruby
|
|
601
|
+
assert SidekiqUniqueJobs.validate_worker!(BadWorker.get_sidekiq_options)
|
|
602
|
+
```
|
|
603
|
+
|
|
604
|
+
### Uniqueness
|
|
702
605
|
|
|
703
|
-
|
|
606
|
+
This has been probably the most confusing part of this gem. People get really confused with how unreliable the unique jobs have been. I there for decided to do what Mike is doing for sidekiq enterprise. Read the section about unique jobs: [Enterprise unique jobs][]
|
|
704
607
|
|
|
705
608
|
```ruby
|
|
706
|
-
|
|
707
|
-
|
|
708
|
-
digest = msg['lock_digest']
|
|
709
|
-
SidekiqUniqueJobs::Digests.new.delete_by_digest(digest) if digest
|
|
710
|
-
end
|
|
609
|
+
SidekiqUniqueJobs.configure do |config|
|
|
610
|
+
config.enabled = !Rails.env.test?
|
|
711
611
|
end
|
|
712
612
|
```
|
|
713
613
|
|
|
714
|
-
|
|
614
|
+
If you truly wanted to test the sidekiq client push you could do something like below. Note that it will only work for the jobs that lock when the client pushes the job to redis (UntilExecuted, UntilAndWhileExecuting and UntilExpired).
|
|
715
615
|
|
|
716
616
|
```ruby
|
|
717
|
-
|
|
718
|
-
|
|
719
|
-
|
|
720
|
-
|
|
617
|
+
require "sidekiq_unique_jobs/testing"
|
|
618
|
+
|
|
619
|
+
RSpec.describe Workers::CoolOne do
|
|
620
|
+
before do
|
|
621
|
+
SidekiqUniqueJobs.config.enabled = false
|
|
622
|
+
end
|
|
623
|
+
|
|
624
|
+
# ... your tests that don't test uniqueness
|
|
625
|
+
|
|
626
|
+
context 'when Sidekiq::Testing.disabled?' do
|
|
627
|
+
before do
|
|
628
|
+
Sidekiq::Testing.disable!
|
|
629
|
+
Sidekiq.redis(&:flushdb)
|
|
630
|
+
end
|
|
631
|
+
|
|
632
|
+
after do
|
|
633
|
+
Sidekiq.redis(&:flushdb)
|
|
634
|
+
end
|
|
635
|
+
|
|
636
|
+
it 'prevents duplicate jobs from being scheduled' do
|
|
637
|
+
SidekiqUniqueJobs.use_config(enabled: true) do
|
|
638
|
+
expect(described_class.perform_in(3600, 1)).not_to eq(nil)
|
|
639
|
+
expect(described_class.perform_async(1)).to eq(nil)
|
|
640
|
+
end
|
|
641
|
+
end
|
|
721
642
|
end
|
|
722
643
|
end
|
|
723
644
|
```
|
|
724
645
|
|
|
646
|
+
It is recommended to leave the uniqueness testing to the gem maintainers. If you care about how the gem is integration tested have a look at the following specs:
|
|
647
|
+
|
|
648
|
+
- [spec/sidekiq_unique_jobs/lock/until_and_while_executing_spec.rb](https://github.com/mhenrixon/sidekiq-unique-jobs/blob/master/spec/sidekiq_unique_jobs/lock/until_and_while_executing_spec.rb)
|
|
649
|
+
- [spec/sidekiq_unique_jobs/lock/until_executed_spec.rb](https://github.com/mhenrixon/sidekiq-unique-jobs/blob/master/spec/sidekiq_unique_jobs/lock/until_executed_spec.rb)
|
|
650
|
+
- [spec/sidekiq_unique_jobs/lock/until_expired_spec.rb](https://github.com/mhenrixon/sidekiq-unique-jobs/blob/master/spec/sidekiq_unique_jobs/lock/until_expired_spec.rb)
|
|
651
|
+
- [spec/sidekiq_unique_jobs/lock/while_executing_reject_spec.rb](https://github.com/mhenrixon/sidekiq-unique-jobs/blob/master/spec/sidekiq_unique_jobs/lock/while_executing_reject_spec.rb)
|
|
652
|
+
- [spec/sidekiq_unique_jobs/lock/while_executing_spec.rb](https://github.com/mhenrixon/sidekiq-unique-jobs/blob/master/spec/sidekiq_unique_jobs/lock/while_executing_spec.rb)
|
|
653
|
+
|
|
654
|
+
## Configuration
|
|
655
|
+
|
|
725
656
|
### Other Sidekiq gems
|
|
726
657
|
|
|
727
658
|
#### apartment-sidekiq
|
|
@@ -794,124 +725,310 @@ end
|
|
|
794
725
|
|
|
795
726
|
The reason for this is that if a job is duplicated it shouldn't end up with the status middleware at all. Status is just a monitor so to prevent clashes, leftovers and ensure cleanup. The status middleware should run after uniqueness on client and before on server. This will lead to less surprises.
|
|
796
727
|
|
|
797
|
-
|
|
728
|
+
### Global Configuration
|
|
798
729
|
|
|
799
|
-
|
|
730
|
+
The gem supports a few different configuration options that might be of interest if you run into some weird issues.
|
|
800
731
|
|
|
801
|
-
|
|
732
|
+
Configure SidekiqUniqueJobs in an initializer or the sidekiq initializer on application startup.
|
|
802
733
|
|
|
803
|
-
|
|
734
|
+
```ruby
|
|
735
|
+
SidekiqUniqueJobs.configure do |config|
|
|
736
|
+
config.logger = Sidekiq.logger # default, change at your own discretion
|
|
737
|
+
config.debug_lua = false # Turn on when debugging
|
|
738
|
+
config.lock_info = false # Turn on when debugging
|
|
739
|
+
config.lock_ttl = 600 # Expire locks after 10 minutes
|
|
740
|
+
config.lock_timeout = nil # turn off lock timeout
|
|
741
|
+
config.max_history = 0 # Turn on when debugging
|
|
742
|
+
config.reaper = :ruby # :ruby, :lua or :none/nil
|
|
743
|
+
config.reaper_count = 1000 # Stop reaping after this many keys
|
|
744
|
+
config.reaper_interval = 600 # Reap orphans every 10 minutes
|
|
745
|
+
config.reaper_timeout = 150 # Timeout reaper after 2.5 minutes
|
|
746
|
+
end
|
|
747
|
+
```
|
|
748
|
+
|
|
749
|
+
#### debug_lua
|
|
804
750
|
|
|
805
751
|
```ruby
|
|
806
|
-
|
|
807
|
-
require 'sidekiq_unique_jobs/web'
|
|
808
|
-
mount Sidekiq::Web, at: '/sidekiq'
|
|
752
|
+
SidekiqUniqueJobs.config.debug_lua #=> false
|
|
809
753
|
```
|
|
810
754
|
|
|
811
|
-
|
|
812
|
-
already does this.
|
|
755
|
+
Turning on debug_lua will allow the lua scripts to output debug information about what the lua scripts do. It will log all redis commands that are executed and also some helpful messages about what is going on inside the lua script.
|
|
813
756
|
|
|
814
|
-
|
|
757
|
+
#### lock_timeout
|
|
815
758
|
|
|
816
|
-
|
|
759
|
+
```ruby
|
|
760
|
+
SidekiqUniqueJobs.config.lock_timeout #=> 0
|
|
761
|
+
```
|
|
817
762
|
|
|
818
|
-
|
|
763
|
+
Set a global lock_timeout to use for all jobs that don't otherwise specify a lock_timeout.
|
|
819
764
|
|
|
820
|
-
|
|
765
|
+
Lock timeout decides how long to wait for acquiring the lock. A value of nil means to wait indefinitely for a lock resource to become available.
|
|
821
766
|
|
|
822
|
-
|
|
767
|
+
#### lock_ttl
|
|
823
768
|
|
|
824
|
-
|
|
769
|
+
```ruby
|
|
770
|
+
SidekiqUniqueJobs.config.lock_ttl #=> nil
|
|
771
|
+
```
|
|
825
772
|
|
|
826
|
-
|
|
773
|
+
Set a global lock_ttl to use for all jobs that don't otherwise specify a lock_ttl.
|
|
827
774
|
|
|
828
|
-
|
|
775
|
+
Lock TTL decides how long to wait at most before considering a lock to be expired and making it possible to reuse that lock.
|
|
829
776
|
|
|
830
|
-
|
|
777
|
+
#### enabled
|
|
831
778
|
|
|
832
|
-
|
|
779
|
+
```ruby
|
|
780
|
+
SidekiqUniqueJobs.config.enabled #=> true
|
|
781
|
+
```
|
|
833
782
|
|
|
834
|
-
|
|
783
|
+
Globally turn the locking mechanism on or off.
|
|
784
|
+
|
|
785
|
+
#### logger
|
|
835
786
|
|
|
836
787
|
```ruby
|
|
837
|
-
|
|
838
|
-
|
|
839
|
-
sidekiq_options lock: :while_executing, on_conflict: :replace
|
|
840
|
-
end
|
|
788
|
+
SidekiqUniqueJobs.config.logger #=> #<Sidekiq::Logger:0x00007fdc1f96d180>
|
|
789
|
+
```
|
|
841
790
|
|
|
842
|
-
|
|
791
|
+
By default this gem piggybacks on the Sidekiq logger. It is not recommended to change this as the gem uses some features in the Sidekiq logger and you might run into problems. If you need a different logger and you do run into problems then get in touch and we'll see what we can do about it.
|
|
843
792
|
|
|
844
|
-
|
|
845
|
-
#OR
|
|
846
|
-
require "sidekiq_unique_jobs/rspec/matchers"
|
|
793
|
+
#### max_history
|
|
847
794
|
|
|
848
|
-
|
|
849
|
-
|
|
850
|
-
end
|
|
795
|
+
```ruby
|
|
796
|
+
SidekiqUniqueJobs.config.max_history #=> 1_000
|
|
851
797
|
```
|
|
852
798
|
|
|
853
|
-
|
|
799
|
+
The max_history setting can be used to tweak the number of changelogs generated. It can also be completely turned off if performance suffers or if you are just not interested in using the changelog.
|
|
854
800
|
|
|
855
|
-
|
|
856
|
-
|
|
857
|
-
|
|
801
|
+
This is a log that can be accessed by a lock to see what happened for that lock. Any items after the configured `max_history` will be automatically deleted as new items are added.
|
|
802
|
+
|
|
803
|
+
#### reaper
|
|
804
|
+
|
|
805
|
+
```ruby
|
|
806
|
+
SidekiqUniqueJobs.config.reaper #=> :ruby
|
|
858
807
|
```
|
|
859
808
|
|
|
860
|
-
If
|
|
809
|
+
If using the orphans cleanup process it is critical to be aware of the following. The `:ruby` job is much slower but the `:lua` job locks redis while executing. While doing intense processing it is best to avoid locking redis with a lua script. There for the batch size (controlled by the `reaper_count` setting) needs to be reduced.
|
|
810
|
+
|
|
811
|
+
In my benchmarks deleting 1000 orphaned locks with lua performs around 65% faster than deleting 1000 keys in ruby.
|
|
812
|
+
|
|
813
|
+
On the other hand if I increase it to 10 000 orphaned locks per cleanup (`reaper_count: 10_0000`) then redis starts throwing:
|
|
814
|
+
|
|
815
|
+
> BUSY Redis is busy running a script. You can only call SCRIPT KILL or SHUTDOWN NOSAVE. (Redis::CommandError)
|
|
816
|
+
|
|
817
|
+
If you want to disable the reaper set it to `:none`, `nil` or `false`. Actually, any value that isn't `:ruby` or `:lua` will disable the reaping.
|
|
861
818
|
|
|
862
819
|
```ruby
|
|
863
|
-
|
|
820
|
+
SidekiqUniqueJobs.config.reaper = :none
|
|
821
|
+
SidekiqUniqueJobs.config.reaper = nil
|
|
822
|
+
SidekiqUniqueJobs.config.reaper = false
|
|
864
823
|
```
|
|
865
824
|
|
|
866
|
-
|
|
825
|
+
#### reaper_count
|
|
867
826
|
|
|
868
|
-
|
|
827
|
+
```ruby
|
|
828
|
+
SidekiqUniqueJobs.config.reaper_count #=> 1_000
|
|
829
|
+
```
|
|
830
|
+
|
|
831
|
+
The reaper_count setting configures how many orphans at a time will be cleaned up by the orphan cleanup job. This might have to be tweaked depending on which orphan job is running.
|
|
832
|
+
|
|
833
|
+
#### reaper_interval
|
|
869
834
|
|
|
870
835
|
```ruby
|
|
871
|
-
SidekiqUniqueJobs.
|
|
872
|
-
|
|
836
|
+
SidekiqUniqueJobs.config.reaper_interval #=> 600
|
|
837
|
+
```
|
|
838
|
+
|
|
839
|
+
The number of seconds between reaping.
|
|
840
|
+
|
|
841
|
+
#### reaper_timeout
|
|
842
|
+
|
|
843
|
+
```ruby
|
|
844
|
+
SidekiqUniqueJobs.config.reaper_timeout #=> 10
|
|
845
|
+
```
|
|
846
|
+
|
|
847
|
+
The number of seconds to wait for the reaper to finish before raising a TimeoutError. This is done to ensure that the next time we reap isn't getting stuck due to the previous process already running.
|
|
848
|
+
|
|
849
|
+
#### lock_prefix
|
|
850
|
+
|
|
851
|
+
```ruby
|
|
852
|
+
SidekiqUniqueJobs.config.lock_prefix #=> "uniquejobs"
|
|
853
|
+
```
|
|
854
|
+
|
|
855
|
+
Use if you want a different key prefix for the keys in redis.
|
|
856
|
+
|
|
857
|
+
### lock_info
|
|
858
|
+
|
|
859
|
+
```ruby
|
|
860
|
+
SidekiqUniqueJobs.config.lock_info #=> false
|
|
861
|
+
```
|
|
862
|
+
|
|
863
|
+
Using lock info will create an additional key for the lock with a json object containing information about the lock. This will be presented in the web interface and might help track down why some jobs are getting stuck.
|
|
864
|
+
|
|
865
|
+
### Worker Configuration
|
|
866
|
+
|
|
867
|
+
#### lock_info
|
|
868
|
+
|
|
869
|
+
Lock info gathers information about a specific lock. It collects things like which `lock_args` where used to compute the `lock_digest` that is used for maintaining uniqueness.
|
|
870
|
+
|
|
871
|
+
```ruby
|
|
872
|
+
sidekiq_options lock_info: false # this is the default, set to true to turn on
|
|
873
|
+
```
|
|
874
|
+
|
|
875
|
+
#### lock_prefix
|
|
876
|
+
|
|
877
|
+
Use if you want a different key prefix for the keys in redis.
|
|
878
|
+
|
|
879
|
+
```ruby
|
|
880
|
+
sidekiq_options lock_prefix: "uniquejobs" # this is the default value
|
|
881
|
+
```
|
|
882
|
+
|
|
883
|
+
#### lock_ttl
|
|
884
|
+
|
|
885
|
+
Lock TTL decides how long to wait at most before considering a lock to be expired and making it possible to reuse that lock.
|
|
886
|
+
|
|
887
|
+
Starting from `v7` the expiration will take place when the job is pushed to the queue.
|
|
888
|
+
|
|
889
|
+
```ruby
|
|
890
|
+
sidekiq_options lock_ttl: nil # default - don't expire keys
|
|
891
|
+
sidekiq_options lock_ttl: 20.days.to_i # expire this lock in 20 days
|
|
892
|
+
```
|
|
893
|
+
|
|
894
|
+
#### lock_timeout
|
|
895
|
+
|
|
896
|
+
This is the timeout (how long to wait) when creating the lock. By default we don't use a timeout so we won't wait for the lock to be created. If you want it is possible to set this like below.
|
|
897
|
+
|
|
898
|
+
```ruby
|
|
899
|
+
sidekiq_options lock_timeout: 0 # default - don't wait at all
|
|
900
|
+
sidekiq_options lock_timeout: 5 # wait 5 seconds
|
|
901
|
+
sidekiq_options lock_timeout: nil # lock indefinitely, this process won't continue until it gets a lock. VERY DANGEROUS!!
|
|
902
|
+
```
|
|
903
|
+
|
|
904
|
+
#### unique_across_queues
|
|
905
|
+
|
|
906
|
+
This configuration option is slightly misleading. It doesn't disregard the queue on other jobs. Just on itself, this means that a worker that might schedule jobs into multiple queues will be able to have uniqueness enforced on all queues it is pushed to.
|
|
907
|
+
|
|
908
|
+
This is mainly intended for `Worker.set(queue: :another).perform_async`.
|
|
909
|
+
|
|
910
|
+
```ruby
|
|
911
|
+
class Worker
|
|
912
|
+
include Sidekiq::Worker
|
|
913
|
+
|
|
914
|
+
sidekiq_options unique_across_queues: true, queue: 'default'
|
|
915
|
+
|
|
916
|
+
def perform(args); end
|
|
873
917
|
end
|
|
874
918
|
```
|
|
875
919
|
|
|
876
|
-
|
|
920
|
+
Now if you push override the queue with `Worker.set(queue: 'another').perform_async(1)` it will still be considered unique when compared to `Worker.perform_async(1)` (that was actually pushed to the queue `default`).
|
|
921
|
+
|
|
922
|
+
#### unique_across_workers
|
|
923
|
+
|
|
924
|
+
This configuration option is slightly misleading. It doesn't disregard the worker class on other jobs. Just on itself, this means that the worker class won't be used for generating the unique digest. The only way this option really makes sense is when you want to have uniqueness between two different worker classes.
|
|
877
925
|
|
|
878
926
|
```ruby
|
|
879
|
-
|
|
927
|
+
class WorkerOne
|
|
928
|
+
include Sidekiq::Worker
|
|
880
929
|
|
|
881
|
-
|
|
882
|
-
|
|
883
|
-
|
|
930
|
+
sidekiq_options unique_across_workers: true, queue: 'default'
|
|
931
|
+
|
|
932
|
+
def perform(args); end
|
|
933
|
+
end
|
|
934
|
+
|
|
935
|
+
class WorkerTwo
|
|
936
|
+
include Sidekiq::Worker
|
|
937
|
+
|
|
938
|
+
sidekiq_options unique_across_workers: true, queue: 'default'
|
|
939
|
+
|
|
940
|
+
def perform(args); end
|
|
941
|
+
end
|
|
942
|
+
|
|
943
|
+
|
|
944
|
+
WorkerOne.perform_async(1)
|
|
945
|
+
# => 'the jobs unique id'
|
|
946
|
+
|
|
947
|
+
WorkerTwo.perform_async(1)
|
|
948
|
+
# => nil because WorkerOne just stole the lock
|
|
949
|
+
```
|
|
950
|
+
|
|
951
|
+
### Finer Control over Uniqueness
|
|
952
|
+
|
|
953
|
+
Sometimes it is desired to have a finer control over which arguments are used in determining uniqueness of the job, and others may be _transient_. For this use-case, you need to define either a `lock_args` method, or a ruby proc.
|
|
954
|
+
|
|
955
|
+
*NOTE:* The lock_args method need to return an array of values to use for uniqueness check.
|
|
956
|
+
|
|
957
|
+
*NOTE:* The arguments passed to the proc or the method is always an array. If your method takes a single array as argument the value of args will be `[[...]]`.
|
|
958
|
+
|
|
959
|
+
The method or the proc can return a modified version of args without the transient arguments included, as shown below:
|
|
960
|
+
|
|
961
|
+
```ruby
|
|
962
|
+
class UniqueJobWithFilterMethod
|
|
963
|
+
include Sidekiq::Worker
|
|
964
|
+
sidekiq_options lock: :until_and_while_executing,
|
|
965
|
+
lock_args_method: :lock_args # this is default and will be used if such a method is defined
|
|
966
|
+
|
|
967
|
+
def self.lock_args(args)
|
|
968
|
+
[ args[0], args[2][:type] ]
|
|
884
969
|
end
|
|
885
970
|
|
|
886
|
-
|
|
971
|
+
...
|
|
887
972
|
|
|
888
|
-
|
|
889
|
-
before do
|
|
890
|
-
Sidekiq::Testing.disable!
|
|
891
|
-
Sidekiq.redis(&:flushdb)
|
|
892
|
-
end
|
|
973
|
+
end
|
|
893
974
|
|
|
894
|
-
|
|
895
|
-
|
|
896
|
-
|
|
975
|
+
class UniqueJobWithFilterProc
|
|
976
|
+
include Sidekiq::Worker
|
|
977
|
+
sidekiq_options lock: :until_executed,
|
|
978
|
+
lock_args_method: ->(args) { [ args.first ] }
|
|
897
979
|
|
|
898
|
-
|
|
899
|
-
|
|
900
|
-
|
|
901
|
-
|
|
902
|
-
|
|
980
|
+
...
|
|
981
|
+
|
|
982
|
+
end
|
|
983
|
+
```
|
|
984
|
+
|
|
985
|
+
It is possible to ensure different types of unique args based on context. I can't vouch for the below example but see [#203](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/203) for the discussion.
|
|
986
|
+
|
|
987
|
+
```ruby
|
|
988
|
+
class UniqueJobWithFilterMethod
|
|
989
|
+
include Sidekiq::Worker
|
|
990
|
+
sidekiq_options lock: :until_and_while_executing, lock_args_method: :lock_args
|
|
991
|
+
|
|
992
|
+
def self.lock_args(args)
|
|
993
|
+
if Sidekiq::ProcessSet.new.size > 1
|
|
994
|
+
# sidekiq runtime; uniqueness for the object (first arg)
|
|
995
|
+
args.first
|
|
996
|
+
else
|
|
997
|
+
# queuing from the app; uniqueness for all params
|
|
998
|
+
args
|
|
903
999
|
end
|
|
904
1000
|
end
|
|
905
1001
|
end
|
|
906
1002
|
```
|
|
907
1003
|
|
|
908
|
-
|
|
1004
|
+
### After Unlock Callback
|
|
909
1005
|
|
|
910
|
-
|
|
911
|
-
|
|
912
|
-
|
|
913
|
-
|
|
914
|
-
|
|
1006
|
+
If you need to perform any additional work after the lock has been released you can provide an `#after_unlock` instance method. The method will be called when the lock has been unlocked. Most times this means after yield but there are two exceptions to that.
|
|
1007
|
+
|
|
1008
|
+
**Exception 1:** UntilExecuting unlocks and uses callback before yielding.
|
|
1009
|
+
**Exception 2:** UntilExpired expires eventually, no after_unlock hook is called.
|
|
1010
|
+
|
|
1011
|
+
**NOTE:** _It is also possible to write this code as a class method._
|
|
1012
|
+
|
|
1013
|
+
```ruby
|
|
1014
|
+
class UniqueJobWithFilterMethod
|
|
1015
|
+
include Sidekiq::Worker
|
|
1016
|
+
sidekiq_options lock: :while_executing,
|
|
1017
|
+
|
|
1018
|
+
def self.after_unlock
|
|
1019
|
+
# block has yielded and lock is released
|
|
1020
|
+
end
|
|
1021
|
+
|
|
1022
|
+
def after_unlock
|
|
1023
|
+
# block has yielded and lock is released
|
|
1024
|
+
end
|
|
1025
|
+
...
|
|
1026
|
+
end.
|
|
1027
|
+
```
|
|
1028
|
+
|
|
1029
|
+
## Communication
|
|
1030
|
+
|
|
1031
|
+
There is a [](https://gitter.im/mhenrixon/sidekiq-unique-jobs?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) for praise or scorn. This would be a good place to have lengthy discuss or brilliant suggestions or simply just nudge me if I forget about anything.
|
|
915
1032
|
|
|
916
1033
|
## Contributing
|
|
917
1034
|
|
|
@@ -925,8 +1042,8 @@ It is recommended to leave the uniqueness testing to the gem maintainers. If you
|
|
|
925
1042
|
|
|
926
1043
|
You can find a list of contributors over on [Contributors][]
|
|
927
1044
|
|
|
928
|
-
[v5.0.10]: https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v5.0.10.
|
|
929
|
-
[v4.0.18]: https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v4.0.18
|
|
930
|
-
[Sidekiq requirements]: https://github.com/mperham/sidekiq#requirements
|
|
931
1045
|
[Enterprise unique jobs]: https://www.dailydrip.com/topics/sidekiq/drips/sidekiq-enterprise-unique-jobs
|
|
932
1046
|
[Contributors]: https://github.com/mhenrixon/sidekiq-unique-jobs/graphs/contributors
|
|
1047
|
+
[v4.0.18]: https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v4.0.18
|
|
1048
|
+
[v5.0.10]: https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v5.0.10.
|
|
1049
|
+
[Sidekiq requirements]: https://github.com/mperham/sidekiq#requirements
|