sidekiq-unique-jobs 7.1.2 → 7.1.7
Sign up to get free protection for your applications and to get access to all the features.
Potentially problematic release.
This version of sidekiq-unique-jobs might be problematic. Click here for more details.
- checksums.yaml +4 -4
- data/CHANGELOG.md +55 -0
- data/README.md +592 -541
- data/lib/sidekiq_unique_jobs/config.rb +45 -6
- data/lib/sidekiq_unique_jobs/constants.rb +44 -44
- data/lib/sidekiq_unique_jobs/deprecation.rb +30 -0
- data/lib/sidekiq_unique_jobs/exceptions.rb +1 -0
- data/lib/sidekiq_unique_jobs/json.rb +7 -0
- data/lib/sidekiq_unique_jobs/lock/base_lock.rb +30 -22
- data/lib/sidekiq_unique_jobs/lock/until_and_while_executing.rb +9 -3
- data/lib/sidekiq_unique_jobs/lock/until_executed.rb +9 -3
- data/lib/sidekiq_unique_jobs/lock/until_executing.rb +10 -4
- data/lib/sidekiq_unique_jobs/lock/until_expired.rb +10 -4
- data/lib/sidekiq_unique_jobs/lock/while_executing.rb +6 -4
- data/lib/sidekiq_unique_jobs/lock_ttl.rb +1 -1
- data/lib/sidekiq_unique_jobs/locksmith.rb +4 -1
- data/lib/sidekiq_unique_jobs/logging.rb +9 -0
- data/lib/sidekiq_unique_jobs/middleware/client.rb +1 -1
- data/lib/sidekiq_unique_jobs/options_with_fallback.rb +2 -4
- data/lib/sidekiq_unique_jobs/orphans/manager.rb +35 -0
- data/lib/sidekiq_unique_jobs/orphans/ruby_reaper.rb +3 -1
- data/lib/sidekiq_unique_jobs/reflectable.rb +11 -2
- data/lib/sidekiq_unique_jobs/reflections.rb +12 -1
- data/lib/sidekiq_unique_jobs/server.rb +13 -1
- data/lib/sidekiq_unique_jobs/sidekiq_unique_ext.rb +35 -13
- data/lib/sidekiq_unique_jobs/timing.rb +1 -1
- data/lib/sidekiq_unique_jobs/version.rb +1 -1
- data/lib/tasks/changelog.rake +14 -14
- metadata +8 -8
data/README.md
CHANGED
@@ -2,6 +2,10 @@
|
|
2
2
|
|
3
3
|
[![Join the chat at https://gitter.im/mhenrixon/sidekiq-unique-jobs](https://badges.gitter.im/mhenrixon/sidekiq-unique-jobs.svg)](https://gitter.im/mhenrixon/sidekiq-unique-jobs?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) ![Build Status](https://github.com/mhenrixon/sidekiq-unique-jobs/actions/workflows/rspec.yml/badge.svg?branch=master) [![Code Climate](https://codeclimate.com/github/mhenrixon/sidekiq-unique-jobs.svg)](https://codeclimate.com/github/mhenrixon/sidekiq-unique-jobs) [![Test Coverage](https://codeclimate.com/github/mhenrixon/sidekiq-unique-jobs/badges/coverage.svg)](https://codeclimate.com/github/mhenrixon/sidekiq-unique-jobs/coverage)
|
4
4
|
|
5
|
+
## Support Me
|
6
|
+
|
7
|
+
Want to show me some ❤️ for the hard work I do on this gem? You can use the following PayPal link: [https://paypal.me/mhenrixon1](https://paypal.me/mhenrixon1). Any amount is welcome and let me tell you it feels good to be appreciated. Even a dollar makes me super excited about all of this.
|
8
|
+
|
5
9
|
<!-- MarkdownTOC -->
|
6
10
|
|
7
11
|
- [Introduction](#introduction)
|
@@ -9,47 +13,18 @@
|
|
9
13
|
- [Installation](#installation)
|
10
14
|
- [Add the middleware](#add-the-middleware)
|
11
15
|
- [Your first worker](#your-first-worker)
|
12
|
-
- [Support Me](#support-me)
|
13
16
|
- [Requirements](#requirements)
|
14
|
-
- [General Information](#general-information)
|
15
|
-
- [Reflections \(metrics, logging, etc.\)](#reflections-metrics-logging-etc)
|
16
|
-
- [after_unlock_callback_failed](#after_unlock_callback_failed)
|
17
|
-
- [error](#error)
|
18
|
-
- [execution_failed](#execution_failed)
|
19
|
-
- [lock_failed](#lock_failed)
|
20
|
-
- [locked](#locked)
|
21
|
-
- [reschedule_failed](#reschedule_failed)
|
22
|
-
- [rescheduled](#rescheduled)
|
23
|
-
- [timeout](#timeout)
|
24
|
-
- [unlock_failed](#unlock_failed)
|
25
|
-
- [unlocked](#unlocked)
|
26
|
-
- [unknown_sidekiq_worker](#unknown_sidekiq_worker)
|
27
|
-
- [Global Configuration](#global-configuration)
|
28
|
-
- [debug_lua](#debug_lua)
|
29
|
-
- [lock_timeout](#lock_timeout)
|
30
|
-
- [lock_ttl](#lock_ttl)
|
31
|
-
- [enabled](#enabled)
|
32
|
-
- [logger](#logger)
|
33
|
-
- [max_history](#max_history)
|
34
|
-
- [reaper](#reaper)
|
35
|
-
- [reaper_count](#reaper_count)
|
36
|
-
- [reaper_interval](#reaper_interval)
|
37
|
-
- [reaper_timeout](#reaper_timeout)
|
38
|
-
- [lock_prefix](#lock_prefix)
|
39
|
-
- [lock_info](#lock_info)
|
40
|
-
- [Worker Configuration](#worker-configuration)
|
41
|
-
- [lock_info](#lock_info-1)
|
42
|
-
- [lock_prefix](#lock_prefix-1)
|
43
|
-
- [lock_ttl](#lock_ttl-1)
|
44
|
-
- [lock_timeout](#lock_timeout-1)
|
45
|
-
- [unique_across_queues](#unique_across_queues)
|
46
|
-
- [unique_across_workers](#unique_across_workers)
|
47
17
|
- [Locks](#locks)
|
48
18
|
- [Until Executing](#until-executing)
|
19
|
+
- [Example worker](#example-worker)
|
49
20
|
- [Until Executed](#until-executed)
|
21
|
+
- [Example worker](#example-worker-1)
|
50
22
|
- [Until Expired](#until-expired)
|
23
|
+
- [Example worker](#example-worker-2)
|
51
24
|
- [Until And While Executing](#until-and-while-executing)
|
25
|
+
- [Example worker](#example-worker-3)
|
52
26
|
- [While Executing](#while-executing)
|
27
|
+
- [Example worker](#example-worker-4)
|
53
28
|
- [Custom Locks](#custom-locks)
|
54
29
|
- [Conflict Strategy](#conflict-strategy)
|
55
30
|
- [log](#log)
|
@@ -58,22 +33,54 @@
|
|
58
33
|
- [replace](#replace)
|
59
34
|
- [Reschedule](#reschedule)
|
60
35
|
- [Custom Strategies](#custom-strategies)
|
61
|
-
- [
|
62
|
-
- [Finer Control over Uniqueness](#finer-control-over-uniqueness)
|
63
|
-
- [After Unlock Callback](#after-unlock-callback)
|
64
|
-
- [Cleanup Dead Locks](#cleanup-dead-locks)
|
65
|
-
- [Other Sidekiq gems](#other-sidekiq-gems)
|
66
|
-
- [apartment-sidekiq](#apartment-sidekiq)
|
67
|
-
- [sidekiq-global_id](#sidekiq-global_id)
|
68
|
-
- [sidekiq-status](#sidekiq-status)
|
36
|
+
- [3 Cleanup Dead Locks](#3-cleanup-dead-locks)
|
69
37
|
- [Debugging](#debugging)
|
70
38
|
- [Sidekiq Web](#sidekiq-web)
|
39
|
+
- [Reflections \(metrics, logging, etc.\)](#reflections-metrics-logging-etc)
|
40
|
+
- [after_unlock_callback_failed](#after_unlock_callback_failed)
|
41
|
+
- [error](#error)
|
42
|
+
- [execution_failed](#execution_failed)
|
43
|
+
- [lock_failed](#lock_failed)
|
44
|
+
- [locked](#locked)
|
45
|
+
- [reschedule_failed](#reschedule_failed)
|
46
|
+
- [rescheduled](#rescheduled)
|
47
|
+
- [timeout](#timeout)
|
48
|
+
- [unlock_failed](#unlock_failed)
|
49
|
+
- [unlocked](#unlocked)
|
50
|
+
- [unknown_sidekiq_worker](#unknown_sidekiq_worker)
|
71
51
|
- [Show Locks](#show-locks)
|
72
52
|
- [Show Lock](#show-lock)
|
73
|
-
- [Communication](#communication)
|
74
53
|
- [Testing](#testing)
|
75
|
-
- [
|
54
|
+
- [Validating Worker Configuration](#validating-worker-configuration)
|
76
55
|
- [Uniqueness](#uniqueness)
|
56
|
+
- [Configuration](#configuration)
|
57
|
+
- [Other Sidekiq gems](#other-sidekiq-gems)
|
58
|
+
- [apartment-sidekiq](#apartment-sidekiq)
|
59
|
+
- [sidekiq-global_id](#sidekiq-global_id)
|
60
|
+
- [sidekiq-status](#sidekiq-status)
|
61
|
+
- [Global Configuration](#global-configuration)
|
62
|
+
- [debug_lua](#debug_lua)
|
63
|
+
- [lock_timeout](#lock_timeout)
|
64
|
+
- [lock_ttl](#lock_ttl)
|
65
|
+
- [enabled](#enabled)
|
66
|
+
- [logger](#logger)
|
67
|
+
- [max_history](#max_history)
|
68
|
+
- [reaper](#reaper)
|
69
|
+
- [reaper_count](#reaper_count)
|
70
|
+
- [reaper_interval](#reaper_interval)
|
71
|
+
- [reaper_timeout](#reaper_timeout)
|
72
|
+
- [lock_prefix](#lock_prefix)
|
73
|
+
- [lock_info](#lock_info)
|
74
|
+
- [Worker Configuration](#worker-configuration)
|
75
|
+
- [lock_info](#lock_info-1)
|
76
|
+
- [lock_prefix](#lock_prefix-1)
|
77
|
+
- [lock_ttl](#lock_ttl-1)
|
78
|
+
- [lock_timeout](#lock_timeout-1)
|
79
|
+
- [unique_across_queues](#unique_across_queues)
|
80
|
+
- [unique_across_workers](#unique_across_workers)
|
81
|
+
- [Finer Control over Uniqueness](#finer-control-over-uniqueness)
|
82
|
+
- [After Unlock Callback](#after-unlock-callback)
|
83
|
+
- [Communication](#communication)
|
77
84
|
- [Contributing](#contributing)
|
78
85
|
- [Contributors](#contributors)
|
79
86
|
|
@@ -81,12 +88,15 @@
|
|
81
88
|
|
82
89
|
## Introduction
|
83
90
|
|
84
|
-
This gem adds unique constraints to
|
91
|
+
This gem adds unique constraints to sidekiq jobs. The uniqueness is achieved by creating a set of keys in redis based off of `queue`, `class`, `args` (in the sidekiq job hash).
|
85
92
|
|
86
|
-
|
93
|
+
By default, only one lock for a given hash can be acquired. What happens when a lock can't be acquired is governed by a chosen [Conflict Strategy](#conflict-strategy) strategy. Unless a conflict strategy is chosen
|
94
|
+
|
95
|
+
This is the documentation for the `main` branch. You can find the documentation for each release by navigating to its tag.
|
87
96
|
|
88
97
|
Here are links to some of the old versions
|
89
98
|
|
99
|
+
- [v7.0.12](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v7.0.12)
|
90
100
|
- [v6.0.25](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v6.0.25)
|
91
101
|
- [v5.0.10](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v5.0.10)
|
92
102
|
- [v4.0.18](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v4.0.18)
|
@@ -141,7 +151,7 @@ end
|
|
141
151
|
|
142
152
|
### Your first worker
|
143
153
|
|
144
|
-
The most likely to be used worker is `:until_executed`. This type of lock creates a lock from when `UntilExecutedWorker.perform_async` is called until
|
154
|
+
The most likely to be used worker is `:until_executed`. This type of lock creates a lock from when `UntilExecutedWorker.perform_async` is called until right after `UntilExecutedWorker.new.perform` has been called.
|
145
155
|
|
146
156
|
```ruby
|
147
157
|
# frozen_string_literal: true
|
@@ -162,10 +172,6 @@ end
|
|
162
172
|
|
163
173
|
You can read more about the worker configuration in [Worker Configuration](#worker-configuration) below.
|
164
174
|
|
165
|
-
## Support Me
|
166
|
-
|
167
|
-
Want to show me some ❤️ for the hard work I do on this gem? You can use the following PayPal link: [https://paypal.me/mhenrixon1](https://paypal.me/mhenrixon1). Any amount is welcome and let me tell you it feels good to be appreciated. Even a dollar makes me super excited about all of this.
|
168
|
-
|
169
175
|
## Requirements
|
170
176
|
|
171
177
|
- Sidekiq `>= 5.0` (`>= 5.2` recommended)
|
@@ -179,514 +185,770 @@ Want to show me some ❤️ for the hard work I do on this gem? You can use the
|
|
179
185
|
|
180
186
|
See [Sidekiq requirements][24] for detailed requirements of Sidekiq itself (be sure to check the right sidekiq version).
|
181
187
|
|
182
|
-
##
|
183
|
-
|
184
|
-
See [Interaction w/ Sidekiq](https://github.com/mhenrixon/sidekiq-unique-jobs/wiki/How-this-gem-interacts-with-Sidekiq) on how the gem interacts with Sidekiq.
|
185
|
-
|
186
|
-
See [Locking & Unlocking](https://github.com/mhenrixon/sidekiq-unique-jobs/wiki/Locking-&-Unlocking) for an overview of the differences on when the various lock types are locked and unlocked.
|
188
|
+
## Locks
|
187
189
|
|
188
|
-
|
190
|
+
### Until Executing
|
189
191
|
|
190
|
-
|
192
|
+
A lock is created when `UntilExecuting.perform_async` is called. Then it is either unlocked when `lock_ttl` is hit or before Sidekiq calls the `perform` method on your worker.
|
191
193
|
|
192
|
-
|
194
|
+
#### Example worker
|
193
195
|
|
194
196
|
```ruby
|
197
|
+
class UntilExecuting
|
198
|
+
include Sidekiq::Workers
|
195
199
|
|
196
|
-
|
197
|
-
worker = job_hash['class']
|
198
|
-
args = job_hash['args']
|
199
|
-
lock_args = job_hash['lock_args']
|
200
|
-
queue = job_hash['queue']
|
201
|
-
{
|
202
|
-
message: message,
|
203
|
-
worker: worker,
|
204
|
-
args: args,
|
205
|
-
lock_args: lock_args,
|
206
|
-
queue: queue
|
207
|
-
}
|
208
|
-
end
|
200
|
+
sidekiq_options lock: :until_executing
|
209
201
|
|
210
|
-
|
211
|
-
|
212
|
-
message = extract_log_from_job('Lock Failed', job_hash)
|
213
|
-
Sidekiq.logger.warn(message)
|
202
|
+
def perform(id)
|
203
|
+
# Do work
|
214
204
|
end
|
215
205
|
end
|
216
206
|
```
|
217
207
|
|
218
|
-
|
219
|
-
|
220
|
-
This is called when you have configured a custom callback for when a lock has been released.
|
221
|
-
|
222
|
-
### error
|
223
|
-
|
224
|
-
Not in use yet but will be used deep into the stack to provide a means to catch and report errors inside the gem.
|
225
|
-
|
226
|
-
### execution_failed
|
208
|
+
**NOTE** this is probably not so good for jobs that shouldn't be running simultaneously (aka slow jobs).
|
227
209
|
|
228
|
-
|
210
|
+
The reason this type of lock exists is to fix the following problem: [sidekiq/issues/3471](https://github.com/mperham/sidekiq/issues/3471#issuecomment-300866335)
|
229
211
|
|
230
|
-
###
|
212
|
+
### Until Executed
|
231
213
|
|
232
|
-
|
214
|
+
A lock is created when `UntilExecuted.perform_async` is called. Then it is either unlocked when `lock_ttl` is hit or when Sidekiq has called the `perform` method on your worker.
|
233
215
|
|
234
|
-
|
216
|
+
#### Example worker
|
235
217
|
|
236
|
-
|
218
|
+
```ruby
|
219
|
+
class UntilExecuted
|
220
|
+
include Sidekiq::Workers
|
237
221
|
|
238
|
-
|
222
|
+
sidekiq_options lock: :until_executed
|
239
223
|
|
240
|
-
|
224
|
+
def perform(id)
|
225
|
+
# Do work
|
226
|
+
end
|
227
|
+
end
|
228
|
+
```
|
241
229
|
|
242
|
-
|
230
|
+
### Until Expired
|
243
231
|
|
244
|
-
|
232
|
+
This lock behaves identically to the [Until Executed](#until-executed) except for one thing. This job won't be unlocked until the expiration is hit. For jobs that need to run only once per day, this would be the perfect lock. This way, we can't create more jobs until one day after this job was first pushed.
|
245
233
|
|
246
|
-
|
234
|
+
#### Example worker
|
247
235
|
|
248
|
-
|
236
|
+
```ruby
|
237
|
+
class UntilExpired
|
238
|
+
include Sidekiq::Workers
|
249
239
|
|
250
|
-
|
240
|
+
sidekiq_options lock: :until_expired, lock_ttl: 1.day
|
251
241
|
|
252
|
-
|
242
|
+
def perform
|
243
|
+
# Do work
|
244
|
+
end
|
245
|
+
end
|
246
|
+
```
|
253
247
|
|
254
|
-
|
248
|
+
### Until And While Executing
|
255
249
|
|
256
|
-
|
250
|
+
This lock is a combination of two locks (`:until_executing` and `:while_executing`). Please see the configuration for [Until Executing](#until-executing) and [While Executing](#while-executing)
|
257
251
|
|
258
|
-
|
252
|
+
#### Example worker
|
259
253
|
|
260
|
-
|
254
|
+
```ruby
|
255
|
+
class UntilAndWhileExecutingWorker
|
256
|
+
include Sidekiq::Workers
|
261
257
|
|
262
|
-
|
258
|
+
sidekiq_options lock: :until_and_while_executing,
|
259
|
+
lock_timeout: 2,
|
260
|
+
on_conflict: {
|
261
|
+
client: :log,
|
262
|
+
server: :raise
|
263
|
+
}
|
264
|
+
def perform(id)
|
265
|
+
# Do work
|
266
|
+
end
|
267
|
+
end
|
268
|
+
```
|
263
269
|
|
264
|
-
|
270
|
+
### While Executing
|
265
271
|
|
266
|
-
|
272
|
+
These locks are put on a queue without any type of locking mechanism, the locking doesn't happen until Sidekiq pops the job from the queue and starts processing it.
|
267
273
|
|
268
|
-
|
274
|
+
#### Example worker
|
269
275
|
|
270
276
|
```ruby
|
271
|
-
|
272
|
-
|
273
|
-
|
274
|
-
|
275
|
-
|
276
|
-
|
277
|
-
|
278
|
-
|
279
|
-
|
280
|
-
|
281
|
-
|
277
|
+
class WhileExecutingWorker
|
278
|
+
include Sidekiq::Workers
|
279
|
+
|
280
|
+
sidekiq_options lock: :while_executing,
|
281
|
+
lock_timeout: 2,
|
282
|
+
on_conflict: {
|
283
|
+
server: :raise
|
284
|
+
}
|
285
|
+
def perform(id)
|
286
|
+
# Do work
|
287
|
+
end
|
282
288
|
end
|
283
289
|
```
|
284
290
|
|
285
|
-
|
291
|
+
**NOTE** Unless a conflict strategy of `:raise` is specified, if lock fails, the job will be dropped without notice. When told to raise, the job will be put back and retried. It would also be possible to use `:reschedule` with this lock.
|
286
292
|
|
287
|
-
|
288
|
-
SidekiqUniqueJobs.config.debug_lua #=> false
|
289
|
-
```
|
293
|
+
**NOTE** Unless this job is configured with a `lock_timeout: nil` or `lock_timeout: > 0` then all jobs that are attempted to be executed will just be dropped without waiting.
|
290
294
|
|
291
|
-
|
295
|
+
There is an example of this to try it out in the `myapp` application. Run `foreman start` in the root of the directory and open the url: `localhost:5000/work/duplicate_while_executing`.
|
292
296
|
|
293
|
-
|
297
|
+
In the console you should see something like:
|
294
298
|
|
295
|
-
```
|
296
|
-
|
299
|
+
```bash
|
300
|
+
0:32:24 worker.1 | 2017-04-23T08:32:24.955Z 84404 TID-ougq4thko WhileExecutingWorker JID-400ec51c9523f41cd4a35058 INFO: start
|
301
|
+
10:32:24 worker.1 | 2017-04-23T08:32:24.956Z 84404 TID-ougq8csew WhileExecutingWorker JID-8d6d9168368eedaed7f75763 INFO: start
|
302
|
+
10:32:24 worker.1 | 2017-04-23T08:32:24.957Z 84404 TID-ougq8crt8 WhileExecutingWorker JID-affcd079094c9b26e8b9ba60 INFO: start
|
303
|
+
10:32:24 worker.1 | 2017-04-23T08:32:24.959Z 84404 TID-ougq8cs8s WhileExecutingWorker JID-9e197460c067b22eb1b5d07f INFO: start
|
304
|
+
10:32:24 worker.1 | 2017-04-23T08:32:24.959Z 84404 TID-ougq4thko WhileExecutingWorker JID-400ec51c9523f41cd4a35058 WhileExecutingWorker INFO: perform(1, 2)
|
305
|
+
10:32:34 worker.1 | 2017-04-23T08:32:34.964Z 84404 TID-ougq4thko WhileExecutingWorker JID-400ec51c9523f41cd4a35058 INFO: done: 10.009 sec
|
306
|
+
10:32:34 worker.1 | 2017-04-23T08:32:34.965Z 84404 TID-ougq8csew WhileExecutingWorker JID-8d6d9168368eedaed7f75763 WhileExecutingWorker INFO: perform(1, 2)
|
307
|
+
10:32:44 worker.1 | 2017-04-23T08:32:44.965Z 84404 TID-ougq8crt8 WhileExecutingWorker JID-affcd079094c9b26e8b9ba60 WhileExecutingWorker INFO: perform(1, 2)
|
308
|
+
10:32:44 worker.1 | 2017-04-23T08:32:44.965Z 84404 TID-ougq8csew WhileExecutingWorker JID-8d6d9168368eedaed7f75763 INFO: done: 20.009 sec
|
309
|
+
10:32:54 worker.1 | 2017-04-23T08:32:54.970Z 84404 TID-ougq8cs8s WhileExecutingWorker JID-9e197460c067b22eb1b5d07f WhileExecutingWorker INFO: perform(1, 2)
|
310
|
+
10:32:54 worker.1 | 2017-04-23T08:32:54.969Z 84404 TID-ougq8crt8 WhileExecutingWorker JID-affcd079094c9b26e8b9ba60 INFO: done: 30.012 sec
|
311
|
+
10:33:04 worker.1 | 2017-04-23T08:33:04.973Z 84404 TID-ougq8cs8s WhileExecutingWorker JID-9e197460c067b22eb1b5d07f INFO: done: 40.014 sec
|
297
312
|
```
|
298
313
|
|
299
|
-
|
300
|
-
|
301
|
-
Lock timeout decides how long to wait for acquiring the lock. A value of nil means to wait indefinitely for a lock resource to become available.
|
314
|
+
### Custom Locks
|
302
315
|
|
303
|
-
|
316
|
+
You may need to define some custom lock. You can define it in one project folder:
|
304
317
|
|
305
318
|
```ruby
|
306
|
-
|
319
|
+
# lib/locks/my_custom_lock.rb
|
320
|
+
module Locks
|
321
|
+
class MyCustomLock < SidekiqUniqueJobs::Lock::BaseLock
|
322
|
+
def execute
|
323
|
+
# Do something ...
|
324
|
+
end
|
325
|
+
end
|
326
|
+
end
|
307
327
|
```
|
308
328
|
|
309
|
-
|
329
|
+
You can refer on all the locks defined in `lib/sidekiq_unique_jobs/lock/*.rb`.
|
310
330
|
|
311
|
-
|
331
|
+
In order to make it available, you should call in your project startup:
|
312
332
|
|
313
|
-
|
333
|
+
(For rails application config/initializers/sidekiq_unique_jobs.rb or other projects, wherever you prefer)
|
314
334
|
|
315
335
|
```ruby
|
316
|
-
SidekiqUniqueJobs.
|
336
|
+
SidekiqUniqueJobs.configure do |config|
|
337
|
+
config.add_lock :my_custom_lock, Locks::MyCustomLock
|
338
|
+
end
|
317
339
|
```
|
318
340
|
|
319
|
-
|
320
|
-
|
321
|
-
### logger
|
322
|
-
|
323
|
-
```ruby
|
324
|
-
SidekiqUniqueJobs.config.logger #=> #<Sidekiq::Logger:0x00007fdc1f96d180>
|
325
|
-
```
|
341
|
+
And then you can use it in the jobs definition:
|
326
342
|
|
327
|
-
|
343
|
+
`sidekiq_options lock: :my_custom_lock, on_conflict: :log`
|
328
344
|
|
329
|
-
|
345
|
+
Please not that if you try to override a default lock, an `ArgumentError` will be raised.
|
330
346
|
|
331
|
-
|
332
|
-
SidekiqUniqueJobs.config.max_history #=> 1_000
|
333
|
-
```
|
347
|
+
## Conflict Strategy
|
334
348
|
|
335
|
-
|
349
|
+
Decides how we handle conflict. We can either reject the job to the dead queue or reschedule it. Both are useful for jobs that absolutely need to run and have been configured to use the lock `WhileExecuting` that is used only by the sidekiq server process.
|
336
350
|
|
337
|
-
|
351
|
+
The last one is log which can be be used with the lock `UntilExecuted` and `UntilExpired`. Now we write a log entry saying the job could not be pushed because it is a duplicate of another job with the same arguments.
|
338
352
|
|
339
|
-
|
353
|
+
It is possible for locks to have different conflict strategy for the client and server. This is useful for `:until_and_while_executing`.
|
340
354
|
|
341
355
|
```ruby
|
342
|
-
|
356
|
+
sidekiq_options lock: :until_and_while_executing,
|
357
|
+
on_conflict: { client: :log, server: :reject }
|
343
358
|
```
|
344
359
|
|
345
|
-
|
346
|
-
|
347
|
-
In my benchmarks deleting 1000 orphaned locks with lua performs around 65% faster than deleting 1000 keys in ruby.
|
348
|
-
|
349
|
-
On the other hand if I increase it to 10 000 orphaned locks per cleanup (`reaper_count: 10_0000`) then redis starts throwing:
|
350
|
-
|
351
|
-
> BUSY Redis is busy running a script. You can only call SCRIPT KILL or SHUTDOWN NOSAVE. (Redis::CommandError)
|
352
|
-
|
353
|
-
If you want to disable the reaper set it to `:none`, `nil` or `false`. Actually, any value that isn't `:ruby` or `:lua` will disable the reaping.
|
360
|
+
### log
|
354
361
|
|
355
362
|
```ruby
|
356
|
-
|
357
|
-
SidekiqUniqueJobs.config.reaper = nil
|
358
|
-
SidekiqUniqueJobs.config.reaper = false
|
363
|
+
sidekiq_options on_conflict: :log
|
359
364
|
```
|
360
365
|
|
361
|
-
|
366
|
+
This strategy is intended to be used with `UntilExecuted` and `UntilExpired`. It will log a line about that this is job is a duplicate of another.
|
367
|
+
|
368
|
+
### raise
|
362
369
|
|
363
370
|
```ruby
|
364
|
-
|
371
|
+
sidekiq_options on_conflict: :raise
|
365
372
|
```
|
366
373
|
|
367
|
-
|
374
|
+
This strategy is intended to be used with `WhileExecuting`. Basically it will allow us to let the server process crash with a specific error message and be retried without messing up the Sidekiq stats.
|
368
375
|
|
369
|
-
###
|
376
|
+
### reject
|
370
377
|
|
371
378
|
```ruby
|
372
|
-
|
379
|
+
sidekiq_options on_conflict: :reject
|
373
380
|
```
|
374
381
|
|
375
|
-
|
382
|
+
This strategy is intended to be used with `WhileExecuting` and will push the job to the dead queue on conflict.
|
376
383
|
|
377
|
-
###
|
384
|
+
### replace
|
378
385
|
|
379
386
|
```ruby
|
380
|
-
|
387
|
+
sidekiq_options on_conflict: :replace
|
381
388
|
```
|
382
389
|
|
383
|
-
|
390
|
+
This strategy is intended to be used with client locks like `UntilExecuted`.
|
391
|
+
It will delete any existing job for these arguments from retry, schedule and
|
392
|
+
queue and retry the lock again.
|
393
|
+
|
394
|
+
This is slightly dangerous and should probably only be used for jobs that are
|
395
|
+
always scheduled in the future. Currently only attempting to retry one time.
|
384
396
|
|
385
|
-
###
|
397
|
+
### Reschedule
|
386
398
|
|
387
399
|
```ruby
|
388
|
-
|
400
|
+
sidekiq_options on_conflict: :reschedule
|
389
401
|
```
|
390
402
|
|
391
|
-
|
403
|
+
This strategy is intended to be used with `WhileExecuting` and will delay the job to be tried again in 5 seconds. This will mess up the sidekiq stats but will prevent exceptions from being logged and confuse your sysadmins.
|
392
404
|
|
393
|
-
###
|
405
|
+
### Custom Strategies
|
406
|
+
|
407
|
+
You may need to define some custom strategy. You can define it in one project folder:
|
394
408
|
|
395
409
|
```ruby
|
396
|
-
|
410
|
+
# lib/strategies/my_custom_strategy.rb
|
411
|
+
module Strategies
|
412
|
+
class MyCustomStrategy < SidekiqUniqueJobs::OnConflict::Strategy
|
413
|
+
def call
|
414
|
+
# Do something ...
|
415
|
+
end
|
416
|
+
end
|
417
|
+
end
|
397
418
|
```
|
398
419
|
|
399
|
-
|
400
|
-
|
401
|
-
## Worker Configuration
|
420
|
+
You can refer to all the strategies defined in `lib/sidekiq_unique_jobs/on_conflict`.
|
402
421
|
|
403
|
-
|
422
|
+
In order to make it available, you should call in your project startup:
|
404
423
|
|
405
|
-
|
424
|
+
(For rails application config/initializers/sidekiq_unique_jobs.rb for other projects, wherever you prefer)
|
406
425
|
|
407
426
|
```ruby
|
408
|
-
|
427
|
+
SidekiqUniqueJobs.configure do |config|
|
428
|
+
config.add_strategy :my_custom_strategy, Strategies::MyCustomStrategy
|
429
|
+
end
|
409
430
|
```
|
410
431
|
|
411
|
-
|
412
|
-
|
413
|
-
Use if you want a different key prefix for the keys in redis.
|
432
|
+
And then you can use it in the jobs definition:
|
414
433
|
|
415
434
|
```ruby
|
416
|
-
sidekiq_options
|
435
|
+
sidekiq_options lock: :while_executing, on_conflict: :my_custom_strategy
|
417
436
|
```
|
418
437
|
|
419
|
-
|
438
|
+
Please not that if you try to override a default lock, an `ArgumentError` will be raised.
|
420
439
|
|
421
|
-
|
440
|
+
### 3 Cleanup Dead Locks
|
422
441
|
|
423
|
-
|
442
|
+
For sidekiq versions < 5.1 a `sidekiq_retries_exhausted` block is required per worker class. This is deprecated in Sidekiq 6.0
|
424
443
|
|
425
444
|
```ruby
|
426
|
-
|
427
|
-
|
428
|
-
|
429
|
-
|
430
|
-
|
431
|
-
|
432
|
-
|
433
|
-
|
445
|
+
class MyWorker
|
446
|
+
sidekiq_retries_exhausted do |msg, _ex|
|
447
|
+
digest = msg['lock_digest']
|
448
|
+
SidekiqUniqueJobs::Digests.new.delete_by_digest(digest) if digest
|
449
|
+
end
|
450
|
+
end
|
451
|
+
```
|
452
|
+
|
453
|
+
Starting in v5.1, Sidekiq can also fire a global callback when a job dies: In version 7, this is handled automatically for you. You don't need to add a death handler, if you configure v7 like in [Add the middleware](#add-the-middleware) you don't have to worry about the below.
|
454
|
+
|
434
455
|
```ruby
|
435
|
-
|
436
|
-
|
437
|
-
|
456
|
+
Sidekiq.configure_server do |config|
|
457
|
+
config.death_handlers << ->(job, _ex) do
|
458
|
+
digest = job['lock_digest']
|
459
|
+
SidekiqUniqueJobs::Digests.new.delete_by_digest(digest) if digest
|
460
|
+
end
|
461
|
+
end
|
438
462
|
```
|
439
463
|
|
440
|
-
|
464
|
+
## Debugging
|
441
465
|
|
442
|
-
|
466
|
+
There are several ways of removing keys that are stuck. The prefered way is by using the unique extension to `Sidekiq::Web`. The old console and command line versions still work but might be deprecated in the future. It is better to search for the digest itself and delete the keys matching that digest.
|
443
467
|
|
444
|
-
|
468
|
+
### Sidekiq Web
|
469
|
+
|
470
|
+
To use the web extension you need to require it in your routes.
|
445
471
|
|
446
472
|
```ruby
|
447
|
-
|
448
|
-
|
473
|
+
#app/config/routes.rb
|
474
|
+
require 'sidekiq_unique_jobs/web'
|
475
|
+
mount Sidekiq::Web, at: '/sidekiq'
|
476
|
+
```
|
449
477
|
|
450
|
-
|
478
|
+
There is no need to `require 'sidekiq/web'` since `sidekiq_unique_jobs/web`
|
479
|
+
already does this.
|
451
480
|
|
452
|
-
|
481
|
+
To filter/search for keys we can use the wildcard `*`. If we have a unique digest `'uniquejobs:9e9b5ce5d423d3ea470977004b50ff84` we can search for it by enter `*ff84` and it should return all digests that end with `ff84`.
|
482
|
+
|
483
|
+
### Reflections (metrics, logging, etc.)
|
484
|
+
|
485
|
+
To be able to gather some insights on what is going on inside this gem. I provide a reflection API that can be used.
|
486
|
+
|
487
|
+
To setup reflections for logging or metrics, use the following API:
|
488
|
+
|
489
|
+
```ruby
|
490
|
+
|
491
|
+
def extract_log_from_job(message, job_hash)
|
492
|
+
worker = job_hash['class']
|
493
|
+
args = job_hash['args']
|
494
|
+
lock_args = job_hash['lock_args']
|
495
|
+
queue = job_hash['queue']
|
496
|
+
{
|
497
|
+
message: message,
|
498
|
+
worker: worker,
|
499
|
+
args: args,
|
500
|
+
lock_args: lock_args,
|
501
|
+
queue: queue
|
502
|
+
}
|
503
|
+
end
|
504
|
+
|
505
|
+
SidekiqUniqueJobs.reflect do |on|
|
506
|
+
on.lock_failed do |job_hash|
|
507
|
+
message = extract_log_from_job('Lock Failed', job_hash)
|
508
|
+
Sidekiq.logger.warn(message)
|
509
|
+
end
|
453
510
|
end
|
454
511
|
```
|
455
512
|
|
456
|
-
|
513
|
+
#### after_unlock_callback_failed
|
457
514
|
|
458
|
-
|
515
|
+
This is called when you have configured a custom callback for when a lock has been released.
|
459
516
|
|
460
|
-
|
517
|
+
#### error
|
518
|
+
|
519
|
+
Not in use yet but will be used deep into the stack to provide a means to catch and report errors inside the gem.
|
520
|
+
|
521
|
+
#### execution_failed
|
522
|
+
|
523
|
+
When the sidekiq processor picks the job of the queue for certain jobs but your job raised an error to the middleware. This will be the reflection. It is probably nothing to worry about. When your worker raises an error, we need to handle some edge cases for until and while executing.
|
524
|
+
|
525
|
+
#### lock_failed
|
526
|
+
|
527
|
+
If we can't achieve a lock, this will be the reflection. It most likely is nothing to worry about. We just couldn't retrieve a lock in a timely fashion.
|
528
|
+
|
529
|
+
The biggest reason for this reflection would be to gather metrics on which workers fail the most at the locking step for example.
|
530
|
+
|
531
|
+
#### locked
|
532
|
+
|
533
|
+
For when a lock has been successful. Again, mostly useful for metrics I suppose.
|
534
|
+
|
535
|
+
#### reschedule_failed
|
536
|
+
|
537
|
+
For when the reschedule strategy failed to reschedule the job.
|
538
|
+
|
539
|
+
#### rescheduled
|
540
|
+
|
541
|
+
For when a job was successfully rescheduled
|
542
|
+
|
543
|
+
#### timeout
|
544
|
+
|
545
|
+
This is also mostly useful for reporting/metrics purposes. What this reflection does is signal that the job was configured to wait (`lock_timeout` was configured), but we couldn't retrieve a lock even though we waited for some time.
|
546
|
+
|
547
|
+
### unlock_failed
|
548
|
+
|
549
|
+
This is not got, this is worth
|
550
|
+
|
551
|
+
### unlocked
|
552
|
+
|
553
|
+
Also mostly useful for reporting purposes. The job was successfully unlocked.
|
554
|
+
|
555
|
+
### unknown_sidekiq_worker
|
556
|
+
|
557
|
+
The reason this happens is that the server couldn't find a valid sidekiq worker class. Most likely, that worker isn't intended to be processed by this sidekiq server instance.
|
558
|
+
|
559
|
+
#### Show Locks
|
560
|
+
|
561
|
+
![Locks](assets/unique_digests_1.png)
|
562
|
+
|
563
|
+
#### Show Lock
|
564
|
+
|
565
|
+
![Lock](assets/unique_digests_2.png)
|
566
|
+
|
567
|
+
## Testing
|
568
|
+
|
569
|
+
### Validating Worker Configuration
|
570
|
+
|
571
|
+
Since v7 it is possible to perform some simple validation against your workers sidekiq_options. What it does is scan for some issues that are known to cause problems in production.
|
572
|
+
|
573
|
+
Let's take a _bad_ worker:
|
461
574
|
|
462
575
|
```ruby
|
463
|
-
|
464
|
-
|
576
|
+
#app/workers/bad_worker.rb
|
577
|
+
class BadWorker
|
578
|
+
sidekiq_options lock: :while_executing, on_conflict: :replace
|
579
|
+
end
|
465
580
|
|
466
|
-
|
581
|
+
#spec/workers/bad_worker_spec.rb
|
467
582
|
|
468
|
-
|
583
|
+
require "sidekiq_unique_jobs/testing"
|
584
|
+
#OR
|
585
|
+
require "sidekiq_unique_jobs/rspec/matchers"
|
586
|
+
|
587
|
+
RSpec.describe BadWorker do
|
588
|
+
specify { expect(described_class).to have_valid_sidekiq_options }
|
469
589
|
end
|
590
|
+
```
|
470
591
|
|
471
|
-
|
472
|
-
include Sidekiq::Worker
|
592
|
+
This gives us a helpful error message for a wrongly configured worker:
|
473
593
|
|
474
|
-
|
594
|
+
```bash
|
595
|
+
Expected BadWorker to have valid sidekiq options but found the following problems:
|
596
|
+
on_server_conflict: :replace is incompatible with the server process
|
597
|
+
```
|
475
598
|
|
476
|
-
|
599
|
+
If you are not using RSpec (a lot of people prefer minitest or test unit) you can do something like:
|
600
|
+
|
601
|
+
```ruby
|
602
|
+
assert SidekiqUniqueJobs.validate_worker!(BadWorker.get_sidekiq_options)
|
603
|
+
```
|
604
|
+
|
605
|
+
### Uniqueness
|
606
|
+
|
607
|
+
This has been probably the most confusing part of this gem. People get really confused with how unreliable the unique jobs have been. I there for decided to do what Mike is doing for sidekiq enterprise. Read the section about unique jobs: [Enterprise unique jobs][]
|
608
|
+
|
609
|
+
```ruby
|
610
|
+
SidekiqUniqueJobs.configure do |config|
|
611
|
+
config.enabled = !Rails.env.test?
|
477
612
|
end
|
613
|
+
```
|
478
614
|
|
615
|
+
If you truly wanted to test the sidekiq client push you could do something like below. Note that it will only work for the jobs that lock when the client pushes the job to redis (UntilExecuted, UntilAndWhileExecuting and UntilExpired).
|
479
616
|
|
480
|
-
|
481
|
-
|
617
|
+
```ruby
|
618
|
+
require "sidekiq_unique_jobs/testing"
|
482
619
|
|
483
|
-
|
484
|
-
|
620
|
+
RSpec.describe Workers::CoolOne do
|
621
|
+
before do
|
622
|
+
SidekiqUniqueJobs.config.enabled = false
|
623
|
+
end
|
624
|
+
|
625
|
+
# ... your tests that don't test uniqueness
|
626
|
+
|
627
|
+
context 'when Sidekiq::Testing.disabled?' do
|
628
|
+
before do
|
629
|
+
Sidekiq::Testing.disable!
|
630
|
+
Sidekiq.redis(&:flushdb)
|
631
|
+
end
|
632
|
+
|
633
|
+
after do
|
634
|
+
Sidekiq.redis(&:flushdb)
|
635
|
+
end
|
636
|
+
|
637
|
+
it 'prevents duplicate jobs from being scheduled' do
|
638
|
+
SidekiqUniqueJobs.use_config(enabled: true) do
|
639
|
+
expect(described_class.perform_in(3600, 1)).not_to eq(nil)
|
640
|
+
expect(described_class.perform_async(1)).to eq(nil)
|
641
|
+
end
|
642
|
+
end
|
643
|
+
end
|
644
|
+
end
|
485
645
|
```
|
486
646
|
|
487
|
-
|
647
|
+
It is recommended to leave the uniqueness testing to the gem maintainers. If you care about how the gem is integration tested have a look at the following specs:
|
488
648
|
|
489
|
-
|
649
|
+
- [spec/sidekiq_unique_jobs/lock/until_and_while_executing_spec.rb](https://github.com/mhenrixon/sidekiq-unique-jobs/blob/master/spec/sidekiq_unique_jobs/lock/until_and_while_executing_spec.rb)
|
650
|
+
- [spec/sidekiq_unique_jobs/lock/until_executed_spec.rb](https://github.com/mhenrixon/sidekiq-unique-jobs/blob/master/spec/sidekiq_unique_jobs/lock/until_executed_spec.rb)
|
651
|
+
- [spec/sidekiq_unique_jobs/lock/until_expired_spec.rb](https://github.com/mhenrixon/sidekiq-unique-jobs/blob/master/spec/sidekiq_unique_jobs/lock/until_expired_spec.rb)
|
652
|
+
- [spec/sidekiq_unique_jobs/lock/while_executing_reject_spec.rb](https://github.com/mhenrixon/sidekiq-unique-jobs/blob/master/spec/sidekiq_unique_jobs/lock/while_executing_reject_spec.rb)
|
653
|
+
- [spec/sidekiq_unique_jobs/lock/while_executing_spec.rb](https://github.com/mhenrixon/sidekiq-unique-jobs/blob/master/spec/sidekiq_unique_jobs/lock/while_executing_spec.rb)
|
654
|
+
|
655
|
+
## Configuration
|
656
|
+
|
657
|
+
### Other Sidekiq gems
|
658
|
+
|
659
|
+
#### apartment-sidekiq
|
660
|
+
|
661
|
+
It was reported in [#536](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/536) that the order of the Sidekiq middleware needs to be as follows.
|
662
|
+
|
663
|
+
```ruby
|
664
|
+
Sidekiq.client_middleware do |chain|
|
665
|
+
chain.add Apartment::Sidekiq::Middleware::Client
|
666
|
+
chain.add SidekiqUniqueJobs::Middleware::Client
|
667
|
+
end
|
668
|
+
|
669
|
+
Sidekiq.server_middleware do |chain|
|
670
|
+
chain.add Apartment::Sidekiq::Middleware::Server
|
671
|
+
chain.add SidekiqUniqueJobs::Middleware::Server
|
672
|
+
end
|
673
|
+
```
|
674
|
+
|
675
|
+
The reason being that this gem needs to be configured AFTER the apartment gem or the apartment will not be able to be considered for uniqueness
|
676
|
+
|
677
|
+
#### sidekiq-global_id
|
678
|
+
|
679
|
+
It was reported in [#235](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/235) that the order of the Sidekiq middleware needs to be as follows.
|
680
|
+
|
681
|
+
For a working setup check the following [file](https://github.com/mhenrixon/sidekiq-unique-jobs/blob/master/myapp/config/sidekiq.rb#L12).
|
490
682
|
|
491
683
|
```ruby
|
492
|
-
|
684
|
+
Sidekiq.client_middleware do |chain|
|
685
|
+
chain.add Sidekiq::GlobalId::ClientMiddleware
|
686
|
+
chain.add SidekiqUniqueJobs::Middleware::Client
|
687
|
+
end
|
688
|
+
|
689
|
+
Sidekiq.server_middleware do |chain|
|
690
|
+
chain.add Sidekiq::GlobalId::ServerMiddleware
|
691
|
+
chain.add SidekiqUniqueJobs::Middleware::Server
|
692
|
+
end
|
493
693
|
```
|
494
694
|
|
495
|
-
|
695
|
+
The reason for this is that the global id needs to be set before the unique jobs middleware runs. Otherwise that won't be available for uniqueness.
|
496
696
|
|
497
|
-
|
697
|
+
#### sidekiq-status
|
498
698
|
|
499
|
-
|
699
|
+
It was reported in [#564](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/564) that the order of the middleware needs to be as follows.
|
500
700
|
|
501
|
-
|
701
|
+
```ruby
|
702
|
+
# Thanks to @ArturT for the correction
|
703
|
+
|
704
|
+
Sidekiq.configure_server do |config|
|
705
|
+
config.client_middleware do |chain|
|
706
|
+
chain.add SidekiqUniqueJobs::Middleware::Client
|
707
|
+
chain.add Sidekiq::Status::ClientMiddleware, expiration: 30.minutes
|
708
|
+
end
|
709
|
+
|
710
|
+
config.server_middleware do |chain|
|
711
|
+
chain.add Sidekiq::Status::ServerMiddleware, expiration: 30.minutes
|
712
|
+
chain.add SidekiqUniqueJobs::Middleware::Server
|
713
|
+
end
|
714
|
+
|
715
|
+
SidekiqUniqueJobs::Server.configure(config)
|
716
|
+
end
|
717
|
+
|
718
|
+
|
719
|
+
Sidekiq.configure_client do |config|
|
720
|
+
config.client_middleware do |chain|
|
721
|
+
chain.add SidekiqUniqueJobs::Middleware::Client
|
722
|
+
chain.add Sidekiq::Status::ClientMiddleware, expiration: 30.minutes
|
723
|
+
end
|
724
|
+
end
|
725
|
+
```
|
726
|
+
|
727
|
+
The reason for this is that if a job is duplicated it shouldn't end up with the status middleware at all. Status is just a monitor so to prevent clashes, leftovers and ensure cleanup. The status middleware should run after uniqueness on client and before on server. This will lead to less surprises.
|
728
|
+
|
729
|
+
### Global Configuration
|
730
|
+
|
731
|
+
The gem supports a few different configuration options that might be of interest if you run into some weird issues.
|
732
|
+
|
733
|
+
Configure SidekiqUniqueJobs in an initializer or the sidekiq initializer on application startup.
|
502
734
|
|
503
735
|
```ruby
|
504
|
-
|
736
|
+
SidekiqUniqueJobs.configure do |config|
|
737
|
+
config.logger = Sidekiq.logger # default, change at your own discretion
|
738
|
+
config.debug_lua = false # Turn on when debugging
|
739
|
+
config.lock_info = false # Turn on when debugging
|
740
|
+
config.lock_ttl = 600 # Expire locks after 10 minutes
|
741
|
+
config.lock_timeout = nil # turn off lock timeout
|
742
|
+
config.max_history = 0 # Turn on when debugging
|
743
|
+
config.reaper = :ruby # :ruby, :lua or :none/nil
|
744
|
+
config.reaper_count = 1000 # Stop reaping after this many keys
|
745
|
+
config.reaper_interval = 600 # Reap orphans every 10 minutes
|
746
|
+
config.reaper_timeout = 150 # Timeout reaper after 2.5 minutes
|
747
|
+
end
|
505
748
|
```
|
506
749
|
|
507
|
-
|
750
|
+
#### debug_lua
|
508
751
|
|
509
|
-
|
752
|
+
```ruby
|
753
|
+
SidekiqUniqueJobs.config.debug_lua #=> false
|
754
|
+
```
|
755
|
+
|
756
|
+
Turning on debug_lua will allow the lua scripts to output debug information about what the lua scripts do. It will log all redis commands that are executed and also some helpful messages about what is going on inside the lua script.
|
757
|
+
|
758
|
+
#### lock_timeout
|
510
759
|
|
511
760
|
```ruby
|
512
|
-
|
761
|
+
SidekiqUniqueJobs.config.lock_timeout #=> 0
|
513
762
|
```
|
514
763
|
|
515
|
-
|
764
|
+
Set a global lock_timeout to use for all jobs that don't otherwise specify a lock_timeout.
|
516
765
|
|
517
|
-
|
766
|
+
Lock timeout decides how long to wait for acquiring the lock. A value of nil means to wait indefinitely for a lock resource to become available.
|
767
|
+
|
768
|
+
#### lock_ttl
|
518
769
|
|
519
770
|
```ruby
|
520
|
-
|
771
|
+
SidekiqUniqueJobs.config.lock_ttl #=> nil
|
521
772
|
```
|
522
773
|
|
523
|
-
|
774
|
+
Set a global lock_ttl to use for all jobs that don't otherwise specify a lock_ttl.
|
524
775
|
|
525
|
-
|
776
|
+
Lock TTL decides how long to wait at most before considering a lock to be expired and making it possible to reuse that lock.
|
777
|
+
|
778
|
+
#### enabled
|
526
779
|
|
527
780
|
```ruby
|
528
|
-
|
781
|
+
SidekiqUniqueJobs.config.enabled #=> true
|
529
782
|
```
|
530
783
|
|
531
|
-
|
784
|
+
Globally turn the locking mechanism on or off.
|
532
785
|
|
533
|
-
|
786
|
+
#### logger
|
534
787
|
|
535
|
-
|
788
|
+
```ruby
|
789
|
+
SidekiqUniqueJobs.config.logger #=> #<Sidekiq::Logger:0x00007fdc1f96d180>
|
790
|
+
```
|
536
791
|
|
537
|
-
|
792
|
+
By default this gem piggybacks on the Sidekiq logger. It is not recommended to change this as the gem uses some features in the Sidekiq logger and you might run into problems. If you need a different logger and you do run into problems then get in touch and we'll see what we can do about it.
|
538
793
|
|
539
|
-
|
540
|
-
|
541
|
-
|
542
|
-
|
543
|
-
10:32:24 worker.1 | 2017-04-23T08:32:24.959Z 84404 TID-ougq8cs8s WhileExecutingWorker JID-9e197460c067b22eb1b5d07f INFO: start
|
544
|
-
10:32:24 worker.1 | 2017-04-23T08:32:24.959Z 84404 TID-ougq4thko WhileExecutingWorker JID-400ec51c9523f41cd4a35058 WhileExecutingWorker INFO: perform(1, 2)
|
545
|
-
10:32:34 worker.1 | 2017-04-23T08:32:34.964Z 84404 TID-ougq4thko WhileExecutingWorker JID-400ec51c9523f41cd4a35058 INFO: done: 10.009 sec
|
546
|
-
10:32:34 worker.1 | 2017-04-23T08:32:34.965Z 84404 TID-ougq8csew WhileExecutingWorker JID-8d6d9168368eedaed7f75763 WhileExecutingWorker INFO: perform(1, 2)
|
547
|
-
10:32:44 worker.1 | 2017-04-23T08:32:44.965Z 84404 TID-ougq8crt8 WhileExecutingWorker JID-affcd079094c9b26e8b9ba60 WhileExecutingWorker INFO: perform(1, 2)
|
548
|
-
10:32:44 worker.1 | 2017-04-23T08:32:44.965Z 84404 TID-ougq8csew WhileExecutingWorker JID-8d6d9168368eedaed7f75763 INFO: done: 20.009 sec
|
549
|
-
10:32:54 worker.1 | 2017-04-23T08:32:54.970Z 84404 TID-ougq8cs8s WhileExecutingWorker JID-9e197460c067b22eb1b5d07f WhileExecutingWorker INFO: perform(1, 2)
|
550
|
-
10:32:54 worker.1 | 2017-04-23T08:32:54.969Z 84404 TID-ougq8crt8 WhileExecutingWorker JID-affcd079094c9b26e8b9ba60 INFO: done: 30.012 sec
|
551
|
-
10:33:04 worker.1 | 2017-04-23T08:33:04.973Z 84404 TID-ougq8cs8s WhileExecutingWorker JID-9e197460c067b22eb1b5d07f INFO: done: 40.014 sec
|
794
|
+
#### max_history
|
795
|
+
|
796
|
+
```ruby
|
797
|
+
SidekiqUniqueJobs.config.max_history #=> 1_000
|
552
798
|
```
|
553
799
|
|
554
|
-
|
800
|
+
The max_history setting can be used to tweak the number of changelogs generated. It can also be completely turned off if performance suffers or if you are just not interested in using the changelog.
|
555
801
|
|
556
|
-
|
802
|
+
This is a log that can be accessed by a lock to see what happened for that lock. Any items after the configured `max_history` will be automatically deleted as new items are added.
|
803
|
+
|
804
|
+
#### reaper
|
557
805
|
|
558
806
|
```ruby
|
559
|
-
|
560
|
-
module Locks
|
561
|
-
class MyCustomLock < SidekiqUniqueJobs::Lock::BaseLock
|
562
|
-
def execute
|
563
|
-
# Do something ...
|
564
|
-
end
|
565
|
-
end
|
566
|
-
end
|
807
|
+
SidekiqUniqueJobs.config.reaper #=> :ruby
|
567
808
|
```
|
568
809
|
|
569
|
-
|
810
|
+
If using the orphans cleanup process it is critical to be aware of the following. The `:ruby` job is much slower but the `:lua` job locks redis while executing. While doing intense processing it is best to avoid locking redis with a lua script. There for the batch size (controlled by the `reaper_count` setting) needs to be reduced.
|
570
811
|
|
571
|
-
In
|
812
|
+
In my benchmarks deleting 1000 orphaned locks with lua performs around 65% faster than deleting 1000 keys in ruby.
|
572
813
|
|
573
|
-
|
814
|
+
On the other hand if I increase it to 10 000 orphaned locks per cleanup (`reaper_count: 10_0000`) then redis starts throwing:
|
815
|
+
|
816
|
+
> BUSY Redis is busy running a script. You can only call SCRIPT KILL or SHUTDOWN NOSAVE. (Redis::CommandError)
|
817
|
+
|
818
|
+
If you want to disable the reaper set it to `:none`, `nil` or `false`. Actually, any value that isn't `:ruby` or `:lua` will disable the reaping.
|
574
819
|
|
575
820
|
```ruby
|
576
|
-
SidekiqUniqueJobs.
|
577
|
-
|
578
|
-
|
821
|
+
SidekiqUniqueJobs.config.reaper = :none
|
822
|
+
SidekiqUniqueJobs.config.reaper = nil
|
823
|
+
SidekiqUniqueJobs.config.reaper = false
|
579
824
|
```
|
580
825
|
|
581
|
-
|
826
|
+
#### reaper_count
|
582
827
|
|
583
|
-
|
828
|
+
```ruby
|
829
|
+
SidekiqUniqueJobs.config.reaper_count #=> 1_000
|
830
|
+
```
|
584
831
|
|
585
|
-
|
832
|
+
The reaper_count setting configures how many orphans at a time will be cleaned up by the orphan cleanup job. This might have to be tweaked depending on which orphan job is running.
|
586
833
|
|
587
|
-
|
834
|
+
#### reaper_interval
|
588
835
|
|
589
|
-
|
836
|
+
```ruby
|
837
|
+
SidekiqUniqueJobs.config.reaper_interval #=> 600
|
838
|
+
```
|
590
839
|
|
591
|
-
The
|
840
|
+
The number of seconds between reaping.
|
592
841
|
|
593
|
-
|
842
|
+
#### reaper_timeout
|
594
843
|
|
595
844
|
```ruby
|
596
|
-
|
597
|
-
on_conflict: { client: :log, server: :reject }
|
845
|
+
SidekiqUniqueJobs.config.reaper_timeout #=> 10
|
598
846
|
```
|
599
847
|
|
600
|
-
|
848
|
+
The number of seconds to wait for the reaper to finish before raising a TimeoutError. This is done to ensure that the next time we reap isn't getting stuck due to the previous process already running.
|
849
|
+
|
850
|
+
#### lock_prefix
|
601
851
|
|
602
852
|
```ruby
|
603
|
-
|
853
|
+
SidekiqUniqueJobs.config.lock_prefix #=> "uniquejobs"
|
604
854
|
```
|
605
855
|
|
606
|
-
|
856
|
+
Use if you want a different key prefix for the keys in redis.
|
607
857
|
|
608
|
-
###
|
858
|
+
### lock_info
|
609
859
|
|
610
860
|
```ruby
|
611
|
-
|
861
|
+
SidekiqUniqueJobs.config.lock_info #=> false
|
612
862
|
```
|
613
863
|
|
614
|
-
|
864
|
+
Using lock info will create an additional key for the lock with a json object containing information about the lock. This will be presented in the web interface and might help track down why some jobs are getting stuck.
|
615
865
|
|
616
|
-
###
|
866
|
+
### Worker Configuration
|
867
|
+
|
868
|
+
#### lock_info
|
869
|
+
|
870
|
+
Lock info gathers information about a specific lock. It collects things like which `lock_args` where used to compute the `lock_digest` that is used for maintaining uniqueness.
|
617
871
|
|
618
872
|
```ruby
|
619
|
-
sidekiq_options
|
873
|
+
sidekiq_options lock_info: false # this is the default, set to true to turn on
|
620
874
|
```
|
621
875
|
|
622
|
-
|
876
|
+
#### lock_prefix
|
623
877
|
|
624
|
-
|
878
|
+
Use if you want a different key prefix for the keys in redis.
|
625
879
|
|
626
880
|
```ruby
|
627
|
-
sidekiq_options
|
881
|
+
sidekiq_options lock_prefix: "uniquejobs" # this is the default value
|
628
882
|
```
|
629
883
|
|
630
|
-
|
631
|
-
It will delete any existing job for these arguments from retry, schedule and
|
632
|
-
queue and retry the lock again.
|
884
|
+
#### lock_ttl
|
633
885
|
|
634
|
-
|
635
|
-
always scheduled in the future. Currently only attempting to retry one time.
|
886
|
+
Lock TTL decides how long to wait at most before considering a lock to be expired and making it possible to reuse that lock.
|
636
887
|
|
637
|
-
|
888
|
+
Starting from `v7` the expiration will take place when the job is pushed to the queue.
|
638
889
|
|
639
890
|
```ruby
|
640
|
-
sidekiq_options
|
891
|
+
sidekiq_options lock_ttl: nil # default - don't expire keys
|
892
|
+
sidekiq_options lock_ttl: 20.days.to_i # expire this lock in 20 days
|
641
893
|
```
|
642
894
|
|
643
|
-
|
895
|
+
#### lock_timeout
|
644
896
|
|
645
|
-
|
897
|
+
This is the timeout (how long to wait) when creating the lock. By default we don't use a timeout so we won't wait for the lock to be created. If you want it is possible to set this like below.
|
898
|
+
|
899
|
+
```ruby
|
900
|
+
sidekiq_options lock_timeout: 0 # default - don't wait at all
|
901
|
+
sidekiq_options lock_timeout: 5 # wait 5 seconds
|
902
|
+
sidekiq_options lock_timeout: nil # lock indefinitely, this process won't continue until it gets a lock. VERY DANGEROUS!!
|
903
|
+
```
|
904
|
+
|
905
|
+
#### unique_across_queues
|
906
|
+
|
907
|
+
This configuration option is slightly misleading. It doesn't disregard the queue on other jobs. Just on itself, this means that a worker that might schedule jobs into multiple queues will be able to have uniqueness enforced on all queues it is pushed to.
|
908
|
+
|
909
|
+
This is mainly intended for `Worker.set(queue: :another).perform_async`.
|
910
|
+
|
911
|
+
```ruby
|
912
|
+
class Worker
|
913
|
+
include Sidekiq::Worker
|
646
914
|
|
647
|
-
|
915
|
+
sidekiq_options unique_across_queues: true, queue: 'default'
|
648
916
|
|
649
|
-
|
650
|
-
# lib/strategies/my_custom_strategy.rb
|
651
|
-
module Strategies
|
652
|
-
class MyCustomStrategy < SidekiqUniqueJobs::OnConflict::Strategy
|
653
|
-
def call
|
654
|
-
# Do something ...
|
655
|
-
end
|
656
|
-
end
|
917
|
+
def perform(args); end
|
657
918
|
end
|
658
919
|
```
|
659
920
|
|
660
|
-
|
921
|
+
Now if you push override the queue with `Worker.set(queue: 'another').perform_async(1)` it will still be considered unique when compared to `Worker.perform_async(1)` (that was actually pushed to the queue `default`).
|
661
922
|
|
662
|
-
|
923
|
+
#### unique_across_workers
|
663
924
|
|
664
|
-
|
925
|
+
This configuration option is slightly misleading. It doesn't disregard the worker class on other jobs. Just on itself, this means that the worker class won't be used for generating the unique digest. The only way this option really makes sense is when you want to have uniqueness between two different worker classes.
|
665
926
|
|
666
927
|
```ruby
|
667
|
-
|
668
|
-
|
928
|
+
class WorkerOne
|
929
|
+
include Sidekiq::Worker
|
930
|
+
|
931
|
+
sidekiq_options unique_across_workers: true, queue: 'default'
|
932
|
+
|
933
|
+
def perform(args); end
|
669
934
|
end
|
670
|
-
```
|
671
935
|
|
672
|
-
|
936
|
+
class WorkerTwo
|
937
|
+
include Sidekiq::Worker
|
673
938
|
|
674
|
-
|
675
|
-
sidekiq_options lock: :while_executing, on_conflict: :my_custom_strategy
|
676
|
-
```
|
939
|
+
sidekiq_options unique_across_workers: true, queue: 'default'
|
677
940
|
|
678
|
-
|
941
|
+
def perform(args); end
|
942
|
+
end
|
679
943
|
|
680
|
-
## Usage
|
681
944
|
|
682
|
-
|
945
|
+
WorkerOne.perform_async(1)
|
946
|
+
# => 'the jobs unique id'
|
683
947
|
|
684
|
-
|
685
|
-
|
948
|
+
WorkerTwo.perform_async(1)
|
949
|
+
# => nil because WorkerOne just stole the lock
|
686
950
|
```
|
687
951
|
|
688
|
-
Requiring the gem in your gemfile should be sufficient to enable unique jobs.
|
689
|
-
|
690
952
|
### Finer Control over Uniqueness
|
691
953
|
|
692
954
|
Sometimes it is desired to have a finer control over which arguments are used in determining uniqueness of the job, and others may be _transient_. For this use-case, you need to define either a `lock_args` method, or a ruby proc.
|
@@ -765,221 +1027,10 @@ class UniqueJobWithFilterMethod
|
|
765
1027
|
end.
|
766
1028
|
```
|
767
1029
|
|
768
|
-
### Cleanup Dead Locks
|
769
|
-
|
770
|
-
For sidekiq versions before 5.1 a `sidekiq_retries_exhausted` block is required per worker class. This is deprecated in Sidekiq 6.0
|
771
|
-
|
772
|
-
```ruby
|
773
|
-
class MyWorker
|
774
|
-
sidekiq_retries_exhausted do |msg, _ex|
|
775
|
-
digest = msg['lock_digest']
|
776
|
-
SidekiqUniqueJobs::Digests.new.delete_by_digest(digest) if digest
|
777
|
-
end
|
778
|
-
end
|
779
|
-
```
|
780
|
-
|
781
|
-
Starting in v5.1, Sidekiq can also fire a global callback when a job dies: In version 7, this is handled automatically for you. You don't need to add a death handler, if you configure v7 like in [Add the middleware](#add-the-middleware) you don't have to worry about the below.
|
782
|
-
|
783
|
-
```ruby
|
784
|
-
Sidekiq.configure_server do |config|
|
785
|
-
config.death_handlers << ->(job, _ex) do
|
786
|
-
digest = job['lock_digest']
|
787
|
-
SidekiqUniqueJobs::Digests.new.delete_by_digest(digest) if digest
|
788
|
-
end
|
789
|
-
end
|
790
|
-
```
|
791
|
-
|
792
|
-
### Other Sidekiq gems
|
793
|
-
|
794
|
-
#### apartment-sidekiq
|
795
|
-
|
796
|
-
It was reported in [#536](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/536) that the order of the Sidekiq middleware needs to be as follows.
|
797
|
-
|
798
|
-
```ruby
|
799
|
-
Sidekiq.client_middleware do |chain|
|
800
|
-
chain.add Apartment::Sidekiq::Middleware::Client
|
801
|
-
chain.add SidekiqUniqueJobs::Middleware::Client
|
802
|
-
end
|
803
|
-
|
804
|
-
Sidekiq.server_middleware do |chain|
|
805
|
-
chain.add Apartment::Sidekiq::Middleware::Server
|
806
|
-
chain.add SidekiqUniqueJobs::Middleware::Server
|
807
|
-
end
|
808
|
-
```
|
809
|
-
|
810
|
-
The reason being that this gem needs to be configured AFTER the apartment gem or the apartment will not be able to be considered for uniqueness
|
811
|
-
|
812
|
-
#### sidekiq-global_id
|
813
|
-
|
814
|
-
It was reported in [#235](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/235) that the order of the Sidekiq middleware needs to be as follows.
|
815
|
-
|
816
|
-
For a working setup check the following [file](https://github.com/mhenrixon/sidekiq-unique-jobs/blob/master/myapp/config/sidekiq.rb#L12).
|
817
|
-
|
818
|
-
```ruby
|
819
|
-
Sidekiq.client_middleware do |chain|
|
820
|
-
chain.add Sidekiq::GlobalId::ClientMiddleware
|
821
|
-
chain.add SidekiqUniqueJobs::Middleware::Client
|
822
|
-
end
|
823
|
-
|
824
|
-
Sidekiq.server_middleware do |chain|
|
825
|
-
chain.add Sidekiq::GlobalId::ServerMiddleware
|
826
|
-
chain.add SidekiqUniqueJobs::Middleware::Server
|
827
|
-
end
|
828
|
-
```
|
829
|
-
|
830
|
-
The reason for this is that the global id needs to be set before the unique jobs middleware runs. Otherwise that won't be available for uniqueness.
|
831
|
-
|
832
|
-
#### sidekiq-status
|
833
|
-
|
834
|
-
It was reported in [#564](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/564) that the order of the middleware needs to be as follows.
|
835
|
-
|
836
|
-
```ruby
|
837
|
-
# Thanks to @ArturT for the correction
|
838
|
-
|
839
|
-
Sidekiq.configure_server do |config|
|
840
|
-
config.client_middleware do |chain|
|
841
|
-
chain.add SidekiqUniqueJobs::Middleware::Client
|
842
|
-
chain.add Sidekiq::Status::ClientMiddleware, expiration: 30.minutes
|
843
|
-
end
|
844
|
-
|
845
|
-
config.server_middleware do |chain|
|
846
|
-
chain.add Sidekiq::Status::ServerMiddleware, expiration: 30.minutes
|
847
|
-
chain.add SidekiqUniqueJobs::Middleware::Server
|
848
|
-
end
|
849
|
-
|
850
|
-
SidekiqUniqueJobs::Server.configure(config)
|
851
|
-
end
|
852
|
-
|
853
|
-
|
854
|
-
Sidekiq.configure_client do |config|
|
855
|
-
config.client_middleware do |chain|
|
856
|
-
chain.add SidekiqUniqueJobs::Middleware::Client
|
857
|
-
chain.add Sidekiq::Status::ClientMiddleware, expiration: 30.minutes
|
858
|
-
end
|
859
|
-
end
|
860
|
-
```
|
861
|
-
|
862
|
-
The reason for this is that if a job is duplicated it shouldn't end up with the status middleware at all. Status is just a monitor so to prevent clashes, leftovers and ensure cleanup. The status middleware should run after uniqueness on client and before on server. This will lead to less surprises.
|
863
|
-
|
864
|
-
## Debugging
|
865
|
-
|
866
|
-
There are several ways of removing keys that are stuck. The prefered way is by using the unique extension to `Sidekiq::Web`. The old console and command line versions still work but might be deprecated in the future. It is better to search for the digest itself and delete the keys matching that digest.
|
867
|
-
|
868
|
-
### Sidekiq Web
|
869
|
-
|
870
|
-
To use the web extension you need to require it in your routes.
|
871
|
-
|
872
|
-
```ruby
|
873
|
-
#app/config/routes.rb
|
874
|
-
require 'sidekiq_unique_jobs/web'
|
875
|
-
mount Sidekiq::Web, at: '/sidekiq'
|
876
|
-
```
|
877
|
-
|
878
|
-
There is no need to `require 'sidekiq/web'` since `sidekiq_unique_jobs/web`
|
879
|
-
already does this.
|
880
|
-
|
881
|
-
To filter/search for keys we can use the wildcard `*`. If we have a unique digest `'uniquejobs:9e9b5ce5d423d3ea470977004b50ff84` we can search for it by enter `*ff84` and it should return all digests that end with `ff84`.
|
882
|
-
|
883
|
-
#### Show Locks
|
884
|
-
|
885
|
-
![Locks](assets/unique_digests_1.png)
|
886
|
-
|
887
|
-
#### Show Lock
|
888
|
-
|
889
|
-
![Lock](assets/unique_digests_2.png)
|
890
|
-
|
891
1030
|
## Communication
|
892
1031
|
|
893
1032
|
There is a [![Join the chat at https://gitter.im/mhenrixon/sidekiq-unique-jobs](https://badges.gitter.im/mhenrixon/sidekiq-unique-jobs.svg)](https://gitter.im/mhenrixon/sidekiq-unique-jobs?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) for praise or scorn. This would be a good place to have lengthy discuss or brilliant suggestions or simply just nudge me if I forget about anything.
|
894
1033
|
|
895
|
-
## Testing
|
896
|
-
|
897
|
-
### Unique Sidekiq Configuration
|
898
|
-
|
899
|
-
Since v7 it is possible to perform some simple validation against your workers sidekiq_options. What it does is scan for some issues that are known to cause problems in production.
|
900
|
-
|
901
|
-
Let's take a _bad_ worker:
|
902
|
-
|
903
|
-
```ruby
|
904
|
-
#app/workers/bad_worker.rb
|
905
|
-
class BadWorker
|
906
|
-
sidekiq_options lock: :while_executing, on_conflict: :replace
|
907
|
-
end
|
908
|
-
|
909
|
-
#spec/workers/bad_worker_spec.rb
|
910
|
-
|
911
|
-
require "sidekiq_unique_jobs/testing"
|
912
|
-
#OR
|
913
|
-
require "sidekiq_unique_jobs/rspec/matchers"
|
914
|
-
|
915
|
-
RSpec.describe BadWorker do
|
916
|
-
specify { expect(described_class).to have_valid_sidekiq_options }
|
917
|
-
end
|
918
|
-
```
|
919
|
-
|
920
|
-
This gives us a helpful error message for a wrongly configured worker:
|
921
|
-
|
922
|
-
```bash
|
923
|
-
Expected BadWorker to have valid sidekiq options but found the following problems:
|
924
|
-
on_server_conflict: :replace is incompatible with the server process
|
925
|
-
```
|
926
|
-
|
927
|
-
If you are not using RSpec (a lot of people prefer minitest or test unit) you can do something like:
|
928
|
-
|
929
|
-
```ruby
|
930
|
-
assert SidekiqUniqueJobs.validate_worker!(BadWorker.get_sidekiq_options)
|
931
|
-
```
|
932
|
-
|
933
|
-
### Uniqueness
|
934
|
-
|
935
|
-
This has been probably the most confusing part of this gem. People get really confused with how unreliable the unique jobs have been. I there for decided to do what Mike is doing for sidekiq enterprise. Read the section about unique jobs: [Enterprise unique jobs][]
|
936
|
-
|
937
|
-
```ruby
|
938
|
-
SidekiqUniqueJobs.configure do |config|
|
939
|
-
config.enabled = !Rails.env.test?
|
940
|
-
end
|
941
|
-
```
|
942
|
-
|
943
|
-
If you truly wanted to test the sidekiq client push you could do something like below. Note that it will only work for the jobs that lock when the client pushes the job to redis (UntilExecuted, UntilAndWhileExecuting and UntilExpired).
|
944
|
-
|
945
|
-
```ruby
|
946
|
-
require "sidekiq_unique_jobs/testing"
|
947
|
-
|
948
|
-
RSpec.describe Workers::CoolOne do
|
949
|
-
before do
|
950
|
-
SidekiqUniqueJobs.config.enabled = false
|
951
|
-
end
|
952
|
-
|
953
|
-
# ... your tests that don't test uniqueness
|
954
|
-
|
955
|
-
context 'when Sidekiq::Testing.disabled?' do
|
956
|
-
before do
|
957
|
-
Sidekiq::Testing.disable!
|
958
|
-
Sidekiq.redis(&:flushdb)
|
959
|
-
end
|
960
|
-
|
961
|
-
after do
|
962
|
-
Sidekiq.redis(&:flushdb)
|
963
|
-
end
|
964
|
-
|
965
|
-
it 'prevents duplicate jobs from being scheduled' do
|
966
|
-
SidekiqUniqueJobs.use_config(enabled: true) do
|
967
|
-
expect(described_class.perform_in(3600, 1)).not_to eq(nil)
|
968
|
-
expect(described_class.perform_async(1)).to eq(nil)
|
969
|
-
end
|
970
|
-
end
|
971
|
-
end
|
972
|
-
end
|
973
|
-
```
|
974
|
-
|
975
|
-
It is recommended to leave the uniqueness testing to the gem maintainers. If you care about how the gem is integration tested have a look at the following specs:
|
976
|
-
|
977
|
-
- [spec/sidekiq_unique_jobs/lock/until_and_while_executing_spec.rb](https://github.com/mhenrixon/sidekiq-unique-jobs/blob/master/spec/sidekiq_unique_jobs/lock/until_and_while_executing_spec.rb)
|
978
|
-
- [spec/sidekiq_unique_jobs/lock/until_executed_spec.rb](https://github.com/mhenrixon/sidekiq-unique-jobs/blob/master/spec/sidekiq_unique_jobs/lock/until_executed_spec.rb)
|
979
|
-
- [spec/sidekiq_unique_jobs/lock/until_expired_spec.rb](https://github.com/mhenrixon/sidekiq-unique-jobs/blob/master/spec/sidekiq_unique_jobs/lock/until_expired_spec.rb)
|
980
|
-
- [spec/sidekiq_unique_jobs/lock/while_executing_reject_spec.rb](https://github.com/mhenrixon/sidekiq-unique-jobs/blob/master/spec/sidekiq_unique_jobs/lock/while_executing_reject_spec.rb)
|
981
|
-
- [spec/sidekiq_unique_jobs/lock/while_executing_spec.rb](https://github.com/mhenrixon/sidekiq-unique-jobs/blob/master/spec/sidekiq_unique_jobs/lock/while_executing_spec.rb)
|
982
|
-
|
983
1034
|
## Contributing
|
984
1035
|
|
985
1036
|
1. Fork it
|
@@ -992,8 +1043,8 @@ It is recommended to leave the uniqueness testing to the gem maintainers. If you
|
|
992
1043
|
|
993
1044
|
You can find a list of contributors over on [Contributors][]
|
994
1045
|
|
995
|
-
[v5.0.10]: https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v5.0.10.
|
996
|
-
[v4.0.18]: https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v4.0.18
|
997
|
-
[Sidekiq requirements]: https://github.com/mperham/sidekiq#requirements
|
998
1046
|
[Enterprise unique jobs]: https://www.dailydrip.com/topics/sidekiq/drips/sidekiq-enterprise-unique-jobs
|
999
1047
|
[Contributors]: https://github.com/mhenrixon/sidekiq-unique-jobs/graphs/contributors
|
1048
|
+
[v4.0.18]: https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v4.0.18
|
1049
|
+
[v5.0.10]: https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v5.0.10.
|
1050
|
+
[Sidekiq requirements]: https://github.com/mperham/sidekiq#requirements
|