sidekiq-unique-jobs 7.0.2 → 7.1.12

Sign up to get free protection for your applications and to get access to all the features.

Potentially problematic release.


This version of sidekiq-unique-jobs might be problematic. Click here for more details.

Files changed (50) hide show
  1. checksums.yaml +4 -4
  2. data/CHANGELOG.md +331 -69
  3. data/README.md +546 -426
  4. data/lib/sidekiq_unique_jobs/changelog.rb +2 -2
  5. data/lib/sidekiq_unique_jobs/config.rb +55 -4
  6. data/lib/sidekiq_unique_jobs/constants.rb +44 -45
  7. data/lib/sidekiq_unique_jobs/deprecation.rb +65 -0
  8. data/lib/sidekiq_unique_jobs/digests.rb +5 -8
  9. data/lib/sidekiq_unique_jobs/exceptions.rb +10 -0
  10. data/lib/sidekiq_unique_jobs/json.rb +7 -0
  11. data/lib/sidekiq_unique_jobs/lock/base_lock.rb +64 -51
  12. data/lib/sidekiq_unique_jobs/lock/until_and_while_executing.rb +37 -9
  13. data/lib/sidekiq_unique_jobs/lock/until_executed.rb +23 -5
  14. data/lib/sidekiq_unique_jobs/lock/until_executing.rb +21 -1
  15. data/lib/sidekiq_unique_jobs/lock/until_expired.rb +27 -0
  16. data/lib/sidekiq_unique_jobs/lock/while_executing.rb +15 -8
  17. data/lib/sidekiq_unique_jobs/lock_config.rb +8 -4
  18. data/lib/sidekiq_unique_jobs/lock_ttl.rb +1 -1
  19. data/lib/sidekiq_unique_jobs/locksmith.rb +93 -80
  20. data/lib/sidekiq_unique_jobs/logging.rb +40 -11
  21. data/lib/sidekiq_unique_jobs/lua/lock.lua +3 -3
  22. data/lib/sidekiq_unique_jobs/lua/shared/_find_digest_in_process_set.lua +1 -1
  23. data/lib/sidekiq_unique_jobs/lua/unlock.lua +12 -5
  24. data/lib/sidekiq_unique_jobs/middleware/client.rb +8 -10
  25. data/lib/sidekiq_unique_jobs/middleware/server.rb +2 -0
  26. data/lib/sidekiq_unique_jobs/on_conflict/reschedule.rb +7 -3
  27. data/lib/sidekiq_unique_jobs/options_with_fallback.rb +2 -13
  28. data/lib/sidekiq_unique_jobs/orphans/manager.rb +49 -3
  29. data/lib/sidekiq_unique_jobs/orphans/reaper_resurrector.rb +170 -0
  30. data/lib/sidekiq_unique_jobs/orphans/ruby_reaper.rb +38 -8
  31. data/lib/sidekiq_unique_jobs/redis/entity.rb +7 -1
  32. data/lib/sidekiq_unique_jobs/redis/sorted_set.rb +1 -1
  33. data/lib/sidekiq_unique_jobs/reflectable.rb +26 -0
  34. data/lib/sidekiq_unique_jobs/reflections.rb +79 -0
  35. data/lib/sidekiq_unique_jobs/script/caller.rb +3 -1
  36. data/lib/sidekiq_unique_jobs/server.rb +14 -1
  37. data/lib/sidekiq_unique_jobs/sidekiq_unique_ext.rb +35 -13
  38. data/lib/sidekiq_unique_jobs/sidekiq_unique_jobs.rb +57 -2
  39. data/lib/sidekiq_unique_jobs/sidekiq_worker_methods.rb +1 -11
  40. data/lib/sidekiq_unique_jobs/timer_task.rb +78 -0
  41. data/lib/sidekiq_unique_jobs/timing.rb +1 -1
  42. data/lib/sidekiq_unique_jobs/version.rb +1 -1
  43. data/lib/sidekiq_unique_jobs/web/helpers.rb +5 -5
  44. data/lib/sidekiq_unique_jobs/web/views/_paging.erb +4 -4
  45. data/lib/sidekiq_unique_jobs/web/views/changelogs.erb +1 -1
  46. data/lib/sidekiq_unique_jobs/web/views/locks.erb +17 -15
  47. data/lib/sidekiq_unique_jobs/web.rb +11 -4
  48. data/lib/sidekiq_unique_jobs.rb +7 -1
  49. data/lib/tasks/changelog.rake +15 -15
  50. metadata +17 -16
data/README.md CHANGED
@@ -1,4 +1,10 @@
1
- # SidekiqUniqueJobs [![Join the chat at https://gitter.im/mhenrixon/sidekiq-unique-jobs](https://badges.gitter.im/mhenrixon/sidekiq-unique-jobs.svg)](https://gitter.im/mhenrixon/sidekiq-unique-jobs?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) [![Build Status](https://travis-ci.org/mhenrixon/sidekiq-unique-jobs.svg?branch=master)](https://travis-ci.org/mhenrixon/sidekiq-unique-jobs) [![Code Climate](https://codeclimate.com/github/mhenrixon/sidekiq-unique-jobs.svg)](https://codeclimate.com/github/mhenrixon/sidekiq-unique-jobs) [![Test Coverage](https://codeclimate.com/github/mhenrixon/sidekiq-unique-jobs/badges/coverage.svg)](https://codeclimate.com/github/mhenrixon/sidekiq-unique-jobs/coverage)
1
+ # SidekiqUniqueJobs
2
+
3
+ [![Join the chat at https://gitter.im/mhenrixon/sidekiq-unique-jobs](https://badges.gitter.im/mhenrixon/sidekiq-unique-jobs.svg)](https://gitter.im/mhenrixon/sidekiq-unique-jobs?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) ![Build Status](https://github.com/mhenrixon/sidekiq-unique-jobs/actions/workflows/rspec.yml/badge.svg?branch=master) [![Code Climate](https://codeclimate.com/github/mhenrixon/sidekiq-unique-jobs.svg)](https://codeclimate.com/github/mhenrixon/sidekiq-unique-jobs) [![Test Coverage](https://codeclimate.com/github/mhenrixon/sidekiq-unique-jobs/badges/coverage.svg)](https://codeclimate.com/github/mhenrixon/sidekiq-unique-jobs/coverage)
4
+
5
+ ## Support Me
6
+
7
+ Want to show me some ❤️ for the hard work I do on this gem? You can use the following PayPal link: [https://paypal.me/mhenrixon1](https://paypal.me/mhenrixon1). Any amount is welcome and let me tell you it feels good to be appreciated. Even a dollar makes me super excited about all of this.
2
8
 
3
9
  <!-- MarkdownTOC -->
4
10
 
@@ -7,35 +13,18 @@
7
13
  - [Installation](#installation)
8
14
  - [Add the middleware](#add-the-middleware)
9
15
  - [Your first worker](#your-first-worker)
10
- - [Support Me](#support-me)
11
16
  - [Requirements](#requirements)
12
- - [General Information](#general-information)
13
- - [Global Configuration](#global-configuration)
14
- - [debug_lua](#debug_lua)
15
- - [lock_timeout](#lock_timeout)
16
- - [lock_ttl](#lock_ttl)
17
- - [enabled](#enabled)
18
- - [logger](#logger)
19
- - [max_history](#max_history)
20
- - [reaper](#reaper)
21
- - [reaper_count](#reaper_count)
22
- - [reaper_interval](#reaper_interval)
23
- - [reaper_timeout](#reaper_timeout)
24
- - [lock_prefix](#lock_prefix)
25
- - [lock_info](#lock_info)
26
- - [Worker Configuration](#worker-configuration)
27
- - [lock_info](#lock_info-1)
28
- - [lock_prefix](#lock_prefix-1)
29
- - [lock_ttl](#lock_ttl-1)
30
- - [lock_timeout](#lock_timeout-1)
31
- - [unique_across_queues](#unique_across_queues)
32
- - [unique_across_workers](#unique_across_workers)
33
17
  - [Locks](#locks)
34
18
  - [Until Executing](#until-executing)
19
+ - [Example worker](#example-worker)
35
20
  - [Until Executed](#until-executed)
21
+ - [Example worker](#example-worker-1)
36
22
  - [Until Expired](#until-expired)
23
+ - [Example worker](#example-worker-2)
37
24
  - [Until And While Executing](#until-and-while-executing)
25
+ - [Example worker](#example-worker-3)
38
26
  - [While Executing](#while-executing)
27
+ - [Example worker](#example-worker-4)
39
28
  - [Custom Locks](#custom-locks)
40
29
  - [Conflict Strategy](#conflict-strategy)
41
30
  - [log](#log)
@@ -44,23 +33,54 @@
44
33
  - [replace](#replace)
45
34
  - [Reschedule](#reschedule)
46
35
  - [Custom Strategies](#custom-strategies)
47
- - [Usage](#usage-1)
48
- - [Finer Control over Uniqueness](#finer-control-over-uniqueness)
49
- - [After Unlock Callback](#after-unlock-callback)
50
- - [Logging](#logging)
51
- - [Cleanup Dead Locks](#cleanup-dead-locks)
52
- - [Other Sidekiq gems](#other-sidekiq-gems)
53
- - [apartment-sidekiq](#apartment-sidekiq)
54
- - [sidekiq-global_id](#sidekiq-global_id)
55
- - [sidekiq-status](#sidekiq-status)
36
+ - [3 Cleanup Dead Locks](#3-cleanup-dead-locks)
56
37
  - [Debugging](#debugging)
57
38
  - [Sidekiq Web](#sidekiq-web)
39
+ - [Reflections \(metrics, logging, etc.\)](#reflections-metrics-logging-etc)
40
+ - [after_unlock_callback_failed](#after_unlock_callback_failed)
41
+ - [error](#error)
42
+ - [execution_failed](#execution_failed)
43
+ - [lock_failed](#lock_failed)
44
+ - [locked](#locked)
45
+ - [reschedule_failed](#reschedule_failed)
46
+ - [rescheduled](#rescheduled)
47
+ - [timeout](#timeout)
48
+ - [unlock_failed](#unlock_failed)
49
+ - [unlocked](#unlocked)
50
+ - [unknown_sidekiq_worker](#unknown_sidekiq_worker)
58
51
  - [Show Locks](#show-locks)
59
52
  - [Show Lock](#show-lock)
60
- - [Communication](#communication)
61
53
  - [Testing](#testing)
62
- - [Unique Sidekiq Configuration](#unique-sidekiq-configuration)
54
+ - [Validating Worker Configuration](#validating-worker-configuration)
63
55
  - [Uniqueness](#uniqueness)
56
+ - [Configuration](#configuration)
57
+ - [Other Sidekiq gems](#other-sidekiq-gems)
58
+ - [apartment-sidekiq](#apartment-sidekiq)
59
+ - [sidekiq-global_id](#sidekiq-global_id)
60
+ - [sidekiq-status](#sidekiq-status)
61
+ - [Global Configuration](#global-configuration)
62
+ - [debug_lua](#debug_lua)
63
+ - [lock_timeout](#lock_timeout)
64
+ - [lock_ttl](#lock_ttl)
65
+ - [enabled](#enabled)
66
+ - [logger](#logger)
67
+ - [max_history](#max_history)
68
+ - [reaper](#reaper)
69
+ - [reaper_count](#reaper_count)
70
+ - [reaper_interval](#reaper_interval)
71
+ - [reaper_timeout](#reaper_timeout)
72
+ - [lock_prefix](#lock_prefix)
73
+ - [lock_info](#lock_info)
74
+ - [Worker Configuration](#worker-configuration)
75
+ - [lock_info](#lock_info-1)
76
+ - [lock_prefix](#lock_prefix-1)
77
+ - [lock_ttl](#lock_ttl-1)
78
+ - [lock_timeout](#lock_timeout-1)
79
+ - [unique_across_queues](#unique_across_queues)
80
+ - [unique_across_workers](#unique_across_workers)
81
+ - [Finer Control over Uniqueness](#finer-control-over-uniqueness)
82
+ - [After Unlock Callback](#after-unlock-callback)
83
+ - [Communication](#communication)
64
84
  - [Contributing](#contributing)
65
85
  - [Contributors](#contributors)
66
86
 
@@ -68,12 +88,15 @@
68
88
 
69
89
  ## Introduction
70
90
 
71
- This gem adds unique constraints to the sidekiq queues. The uniqueness is achieved by acquiring locks for a hash of a queue name, a worker class, and job's arguments. By default, only one lock for a given hash can be acquired. What happens when a lock can't be acquired is governed by a chosen `on_conflict`strategy.
91
+ This gem adds unique constraints to sidekiq jobs. The uniqueness is achieved by creating a set of keys in redis based off of `queue`, `class`, `args` (in the sidekiq job hash).
92
+
93
+ By default, only one lock for a given hash can be acquired. What happens when a lock can't be acquired is governed by a chosen [Conflict Strategy](#conflict-strategy) strategy. Unless a conflict strategy is chosen
72
94
 
73
- This is the documentation for the master branch. You can find the documentation for each release by navigating to its tag.
95
+ This is the documentation for the `main` branch. You can find the documentation for each release by navigating to its tag.
74
96
 
75
97
  Here are links to some of the old versions
76
98
 
99
+ - [v7.0.12](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v7.0.12)
77
100
  - [v6.0.25](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v6.0.25)
78
101
  - [v5.0.10](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v5.0.10)
79
102
  - [v4.0.18](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v4.0.18)
@@ -128,20 +151,16 @@ end
128
151
 
129
152
  ### Your first worker
130
153
 
154
+ The most likely to be used worker is `:until_executed`. This type of lock creates a lock from when `UntilExecutedWorker.perform_async` is called until right after `UntilExecutedWorker.new.perform` has been called.
155
+
131
156
  ```ruby
132
157
  # frozen_string_literal: true
133
158
 
134
159
  class UntilExecutedWorker
135
160
  include Sidekiq::Worker
136
161
 
137
- sidekiq_options queue: :special,
138
- retry: false,
139
- lock: :until_executed,
140
- lock_info: true,
141
- lock_timeout: 0,
142
- lock_prefix: "special",
143
- lock_ttl: 0,
144
- lock_limit: 5
162
+ sidekiq_options queue: :until_executed,
163
+ lock: :until_executed
145
164
 
146
165
  def perform
147
166
  logger.info("cowboy")
@@ -149,15 +168,10 @@ class UntilExecutedWorker
149
168
  logger.info("beebop")
150
169
  end
151
170
  end
152
-
153
171
  ```
154
172
 
155
173
  You can read more about the worker configuration in [Worker Configuration](#worker-configuration) below.
156
174
 
157
- ## Support Me
158
-
159
- Want to show me some ❤️ for the hard work I do on this gem? You can use the following PayPal link: [https://paypal.me/mhenrixon1](https://paypal.me/mhenrixon1). Any amount is welcome and let me tell you it feels good to be appreciated. Even a dollar makes me super excited about all of this.
160
-
161
175
  ## Requirements
162
176
 
163
177
  - Sidekiq `>= 5.0` (`>= 5.2` recommended)
@@ -171,280 +185,110 @@ Want to show me some ❤️ for the hard work I do on this gem? You can use the
171
185
 
172
186
  See [Sidekiq requirements][24] for detailed requirements of Sidekiq itself (be sure to check the right sidekiq version).
173
187
 
174
- ## General Information
175
-
176
- See [Interaction w/ Sidekiq](https://github.com/mhenrixon/sidekiq-unique-jobs/wiki/How-this-gem-interacts-with-Sidekiq) on how the gem interacts with Sidekiq.
177
-
178
- See [Locking & Unlocking](https://github.com/mhenrixon/sidekiq-unique-jobs/wiki/Locking-&-Unlocking) for an overview of the differences on when the various lock types are locked and unlocked.
179
-
180
- ## Global Configuration
181
-
182
- The gem supports a few different configuration options that might be of interest if you run into some weird issues.
183
-
184
- Configure SidekiqUniqueJobs in an initializer or the sidekiq initializer on application startup.
185
-
186
- ```ruby
187
- SidekiqUniqueJobs.configure do |config|
188
- config.debug_lua = true
189
- config.lock_info = true
190
- config.lock_ttl = 10.minutes
191
- config.lock_timeout = 10.minutes
192
- config.logger = Sidekiq.logger
193
- config.max_history = 10_000
194
- config.reaper = :lua
195
- config.reaper_count = 100
196
- config.reaper_interval = 10
197
- config.reaper_timeout = 5
198
- end
199
- ```
200
-
201
- ### debug_lua
202
-
203
- ```ruby
204
- SidekiqUniqueJobs.config.debug_lua #=> false
205
- ```
206
-
207
- Turning on debug_lua will allow the lua scripts to output debug information about what the lua scripts do. It will log all redis commands that are executed and also some helpful messages about what is going on inside the lua script.
208
-
209
- ### lock_timeout
210
-
211
- ```ruby
212
- SidekiqUniqueJobs.config.lock_timeout #=> 0
213
- ```
214
-
215
- Set a global lock_timeout to use for all jobs that don't otherwise specify a lock_timeout.
216
-
217
- Lock timeout decides how long to wait for acquiring the lock. A value of nil means to wait indefinitely for a lock resource to become available.
218
-
219
- ### lock_ttl
220
-
221
- ```ruby
222
- SidekiqUniqueJobs.config.lock_ttl #=> nil
223
- ```
224
-
225
- Set a global lock_ttl to use for all jobs that don't otherwise specify a lock_ttl.
226
-
227
- Lock TTL decides how long to wait after the job has been successfully processed before making it possible to reuse that lock.
228
-
229
- ### enabled
230
-
231
- ```ruby
232
- SidekiqUniqueJobs.config.enabled #=> true
233
- ```
234
-
235
- Globally turn the locking mechanism on or off.
236
-
237
- ### logger
238
-
239
- ```ruby
240
- SidekiqUniqueJobs.config.logger #=> #<Sidekiq::Logger:0x00007fdc1f96d180>
241
- ```
242
-
243
- By default this gem piggybacks on the Sidekiq logger. It is not recommended to change this as the gem uses some features in the Sidekiq logger and you might run into problems. If you need a different logger and you do run into problems then get in touch and we'll see what we can do about it.
244
-
245
- ### max_history
246
-
247
- ```ruby
248
- SidekiqUniqueJobs.config.max_history #=> 1_000
249
- ```
250
-
251
- The max_history setting can be used to tweak the number of changelogs generated. It can also be completely turned off if performance suffers or if you are just not interested in using the changelog.
252
-
253
- This is a log that can be accessed by a lock to see what happened for that lock. Any items after the configured `max_history` will be automatically deleted as new items are added.
254
-
255
- ### reaper
256
-
257
- ```ruby
258
- SidekiqUniqueJobs.config.reaper #=> :ruby
259
- ```
260
-
261
- If using the orphans cleanup process it is critical to be aware of the following. The `:ruby` job is much slower but the `:lua` job locks redis while executing. While doing intense processing it is best to avoid locking redis with a lua script. There for the batch size (controlled by the `reaper_count` setting) needs to be reduced.
262
-
263
- In my benchmarks deleting 1000 orphaned locks with lua performs around 65% faster than deleting 1000 keys in ruby.
264
-
265
- On the other hand if I increase it to 10 000 orphaned locks per cleanup (`reaper_count: 10_0000`) then redis starts throwing:
266
-
267
- > BUSY Redis is busy running a script. You can only call SCRIPT KILL or SHUTDOWN NOSAVE. (Redis::CommandError)
268
-
269
- If you want to disable the reaper set it to `:none`, `nil` or `false`. Actually, any value that isn't `:ruby` or `:lua` will disable the reaping.
270
-
271
- ```ruby
272
- SidekiqUniqueJobs.config.reaper = :none
273
- SidekiqUniqueJobs.config.reaper = nil
274
- SidekiqUniqueJobs.config.reaper = false
275
- ```
276
-
277
- ### reaper_count
278
-
279
- ```ruby
280
- SidekiqUniqueJobs.config.reaper_count #=> 1_000
281
- ```
282
-
283
- The reaper_count setting configures how many orphans at a time will be cleaned up by the orphan cleanup job. This might have to be tweaked depending on which orphan job is running.
284
-
285
- ### reaper_interval
286
-
287
- ```ruby
288
- SidekiqUniqueJobs.config.reaper_interval #=> 600
289
- ```
290
-
291
- The number of seconds between reaping.
292
-
293
- ### reaper_timeout
294
-
295
- ```ruby
296
- SidekiqUniqueJobs.config.reaper_timeout #=> 10
297
- ```
298
-
299
- The number of seconds to wait for the reaper to finish before raising a TimeoutError. This is done to ensure that the next time we reap isn't getting stuck due to the previous process already running.
300
-
301
- ### lock_prefix
188
+ ## Locks
302
189
 
303
- ```ruby
304
- SidekiqUniqueJobs.config.lock_prefix #=> "uniquejobs"
305
- ```
190
+ ### Until Executing
306
191
 
307
- Use if you want a different key prefix for the keys in redis.
192
+ A lock is created when `UntilExecuting.perform_async` is called. Then it is either unlocked when `lock_ttl` is hit or before Sidekiq calls the `perform` method on your worker.
308
193
 
309
- ### lock_info
194
+ #### Example worker
310
195
 
311
196
  ```ruby
312
- SidekiqUniqueJobs.config.lock_info #=> false
313
- ```
314
-
315
- Using lock info will create an additional key for the lock with a json object containing information about the lock. This will be presented in the web interface and might help track down why some jobs are getting stuck.
316
-
317
- ## Worker Configuration
318
-
319
- ### lock_info
197
+ class UntilExecuting
198
+ include Sidekiq::Workers
320
199
 
321
- Lock info gathers information about a specific lock. It collects things like which `lock_args` where used to compute the `lock_digest` that is used for maintaining uniqueness.
200
+ sidekiq_options lock: :until_executing
322
201
 
323
- ```ruby
324
- sidekiq_options lock_info: false # this is the default, set to true to turn on
202
+ def perform(id)
203
+ # Do work
204
+ end
205
+ end
325
206
  ```
326
207
 
327
- ### lock_prefix
328
-
329
- Use if you want a different key prefix for the keys in redis.
208
+ **NOTE** this is probably not so good for jobs that shouldn't be running simultaneously (aka slow jobs).
330
209
 
331
- ```ruby
332
- sidekiq_options lock_prefix: "uniquejobs" # this is the default value
333
- ```
210
+ The reason this type of lock exists is to fix the following problem: [sidekiq/issues/3471](https://github.com/mperham/sidekiq/issues/3471#issuecomment-300866335)
334
211
 
335
- ### lock_ttl
212
+ ### Until Executed
336
213
 
337
- Lock TTL decides how long to wait after the job has been successfully processed before making it possible to reuse that lock.
214
+ A lock is created when `UntilExecuted.perform_async` is called. Then it is either unlocked when `lock_ttl` is hit or when Sidekiq has called the `perform` method on your worker.
338
215
 
339
- Starting from `v7` the expiration will take place when the job is pushed to the queue.
216
+ #### Example worker
340
217
 
341
218
  ```ruby
342
- sidekiq_options lock_ttl: nil # default - don't expire keys
343
- sidekiq_options lock_ttl: 20.days.to_i # expire this lock in 20 days
344
- ```
345
-
346
- ### lock_timeout
219
+ class UntilExecuted
220
+ include Sidekiq::Workers
347
221
 
348
- This is the timeout (how long to wait) when creating the lock. By default we don't use a timeout so we won't wait for the lock to be created. If you want it is possible to set this like below.
222
+ sidekiq_options lock: :until_executed
349
223
 
350
- ```ruby
351
- sidekiq_options lock_timeout: 0 # default - don't wait at all
352
- sidekiq_options lock_timeout: 5 # wait 5 seconds
353
- sidekiq_options lock_timeout: nil # lock indefinitely, this process won't continue until it gets a lock. VERY DANGEROUS!!
224
+ def perform(id)
225
+ # Do work
226
+ end
227
+ end
354
228
  ```
355
229
 
356
- ### unique_across_queues
230
+ ### Until Expired
357
231
 
358
- This configuration option is slightly misleading. It doesn't disregard the queue on other jobs. Just on itself, this means that a worker that might schedule jobs into multiple queues will be able to have uniqueness enforced on all queues it is pushed to.
232
+ This lock behaves identically to the [Until Executed](#until-executed) except for one thing. This job won't be unlocked until the expiration is hit. For jobs that need to run only once per day, this would be the perfect lock. This way, we can't create more jobs until one day after this job was first pushed.
359
233
 
360
- This is mainly intended for `Worker.set(queue: :another).perform_async`.
234
+ #### Example worker
361
235
 
362
236
  ```ruby
363
- class Worker
364
- include Sidekiq::Worker
237
+ class UntilExpired
238
+ include Sidekiq::Workers
365
239
 
366
- sidekiq_options unique_across_queues: true, queue: 'default'
240
+ sidekiq_options lock: :until_expired, lock_ttl: 1.day
367
241
 
368
- def perform(args); end
242
+ def perform
243
+ # Do work
244
+ end
369
245
  end
370
246
  ```
371
247
 
372
- Now if you push override the queue with `Worker.set(queue: 'another').perform_async(1)` it will still be considered unique when compared to `Worker.perform_async(1)` (that was actually pushed to the queue `default`).
248
+ ### Until And While Executing
373
249
 
374
- ### unique_across_workers
250
+ This lock is a combination of two locks (`:until_executing` and `:while_executing`). Please see the configuration for [Until Executing](#until-executing) and [While Executing](#while-executing)
375
251
 
376
- This configuration option is slightly misleading. It doesn't disregard the worker class on other jobs. Just on itself, this means that the worker class won't be used for generating the unique digest. The only way this option really makes sense is when you want to have uniqueness between two different worker classes.
252
+ #### Example worker
377
253
 
378
254
  ```ruby
379
- class WorkerOne
380
- include Sidekiq::Worker
381
-
382
- sidekiq_options unique_across_workers: true, queue: 'default'
383
-
384
- def perform(args); end
385
- end
386
-
387
- class WorkerTwo
388
- include Sidekiq::Worker
389
-
390
- sidekiq_options unique_across_workers: true, queue: 'default'
255
+ class UntilAndWhileExecutingWorker
256
+ include Sidekiq::Workers
391
257
 
392
- def perform(args); end
258
+ sidekiq_options lock: :until_and_while_executing,
259
+ lock_timeout: 2,
260
+ on_conflict: {
261
+ client: :log,
262
+ server: :raise
263
+ }
264
+ def perform(id)
265
+ # Do work
266
+ end
393
267
  end
394
-
395
-
396
- WorkerOne.perform_async(1)
397
- # => 'the jobs unique id'
398
-
399
- WorkerTwo.perform_async(1)
400
- # => nil because WorkerOne just stole the lock
401
- ```
402
-
403
- ## Locks
404
-
405
- ### Until Executing
406
-
407
- ```ruby
408
- sidekiq_options lock: :until_executing
409
- ```
410
-
411
- Locks from when the client pushes the job to the queue. Will be unlocked before the server starts processing the job.
412
-
413
- **NOTE** this is probably not so good for jobs that shouldn't be running simultaneously (aka slow jobs).
414
-
415
- The reason this type of lock exists is to fix the following problem: [sidekiq/issues/3471](https://github.com/mperham/sidekiq/issues/3471#issuecomment-300866335)
416
-
417
- ### Until Executed
418
-
419
- ```ruby
420
- sidekiq_options lock: :until_executed
421
268
  ```
422
269
 
423
- Locks from when the client pushes the job to the queue. Will be unlocked when the server has successfully processed the job.
424
-
425
- ### Until Expired
426
-
427
- ```ruby
428
- sidekiq_options lock: :until_expired
429
- ```
270
+ ### While Executing
430
271
 
431
- Locks from when the client pushes the job to the queue. Will be unlocked when the specified timeout has been reached.
272
+ These locks are put on a queue without any type of locking mechanism, the locking doesn't happen until Sidekiq pops the job from the queue and starts processing it.
432
273
 
433
- ### Until And While Executing
274
+ #### Example worker
434
275
 
435
276
  ```ruby
436
- sidekiq_options lock: :until_and_while_executing
437
- ```
277
+ class WhileExecutingWorker
278
+ include Sidekiq::Workers
438
279
 
439
- Locks when the client pushes the job to the queue. The queue will be unlocked when the server starts processing the job. The server then goes on to creating a runtime lock for the job to prevent simultaneous jobs from being executed. As soon as the server starts processing a job, the client can push the same job to the queue.
440
-
441
- ### While Executing
442
-
443
- ```ruby
444
- sidekiq_options lock: :while_executing, lock_timeout: 10
280
+ sidekiq_options lock: :while_executing,
281
+ lock_timeout: 2,
282
+ on_conflict: {
283
+ server: :raise
284
+ }
285
+ def perform(id)
286
+ # Do work
287
+ end
288
+ end
445
289
  ```
446
290
 
447
- With this lock type it is possible to put any number of these jobs on the queue, but as the server pops the job from the queue it will create a lock and then wait until other locks are done processing. It _looks_ like multiple jobs are running at the same time but in fact the second job will only be waiting for the first job to finish.
291
+ **NOTE** Unless a conflict strategy of `:raise` is specified, if lock fails, the job will be dropped without notice. When told to raise, the job will be put back and retried. It would also be possible to use `:reschedule` with this lock.
448
292
 
449
293
  **NOTE** Unless this job is configured with a `lock_timeout: nil` or `lock_timeout: > 0` then all jobs that are attempted to be executed will just be dropped without waiting.
450
294
 
@@ -565,7 +409,7 @@ You may need to define some custom strategy. You can define it in one project fo
565
409
  ```ruby
566
410
  # lib/strategies/my_custom_strategy.rb
567
411
  module Strategies
568
- class MyCustomStrategy < OnConflict::Strategy
412
+ class MyCustomStrategy < SidekiqUniqueJobs::OnConflict::Strategy
569
413
  def call
570
414
  # Do something ...
571
415
  end
@@ -593,133 +437,223 @@ sidekiq_options lock: :while_executing, on_conflict: :my_custom_strategy
593
437
 
594
438
  Please not that if you try to override a default lock, an `ArgumentError` will be raised.
595
439
 
596
- ## Usage
440
+ ### 3 Cleanup Dead Locks
597
441
 
598
- All that is required is that you specifically set the sidekiq option for _unique_ to a valid value like below:
442
+ For sidekiq versions < 5.1 a `sidekiq_retries_exhausted` block is required per worker class. This is deprecated in Sidekiq 6.0
599
443
 
600
444
  ```ruby
601
- sidekiq_options lock: :while_executing
445
+ class MyWorker
446
+ sidekiq_retries_exhausted do |msg, _ex|
447
+ digest = msg['lock_digest']
448
+ SidekiqUniqueJobs::Digests.new.delete_by_digest(digest) if digest
449
+ end
450
+ end
602
451
  ```
603
452
 
604
- Requiring the gem in your gemfile should be sufficient to enable unique jobs.
453
+ Starting in v5.1, Sidekiq can also fire a global callback when a job dies: In version 7, this is handled automatically for you. You don't need to add a death handler, if you configure v7 like in [Add the middleware](#add-the-middleware) you don't have to worry about the below.
605
454
 
606
- ### Finer Control over Uniqueness
455
+ ```ruby
456
+ Sidekiq.configure_server do |config|
457
+ config.death_handlers << ->(job, _ex) do
458
+ digest = job['lock_digest']
459
+ SidekiqUniqueJobs::Digests.new.delete_by_digest(digest) if digest
460
+ end
461
+ end
462
+ ```
607
463
 
608
- Sometimes it is desired to have a finer control over which arguments are used in determining uniqueness of the job, and others may be _transient_. For this use-case, you need to define either a `lock_args` method, or a ruby proc.
464
+ ## Debugging
609
465
 
610
- *NOTE:* The lock_args method need to return an array of values to use for uniqueness check.
466
+ There are several ways of removing keys that are stuck. The prefered way is by using the unique extension to `Sidekiq::Web`. The old console and command line versions still work but might be deprecated in the future. It is better to search for the digest itself and delete the keys matching that digest.
611
467
 
612
- *NOTE:* The arguments passed to the proc or the method is always an array. If your method takes a single array as argument the value of args will be `[[...]]`.
468
+ ### Sidekiq Web
613
469
 
614
- The method or the proc can return a modified version of args without the transient arguments included, as shown below:
470
+ To use the web extension you need to require it in your routes.
615
471
 
616
472
  ```ruby
617
- class UniqueJobWithFilterMethod
618
- include Sidekiq::Worker
619
- sidekiq_options lock: :until_and_while_executing,
620
- lock_args_method: :lock_args # this is default and will be used if such a method is defined
621
-
622
- def self.lock_args(args)
623
- [ args[0], args[2][:type] ]
624
- end
625
-
626
- ...
473
+ #app/config/routes.rb
474
+ require 'sidekiq_unique_jobs/web'
475
+ mount Sidekiq::Web, at: '/sidekiq'
476
+ ```
627
477
 
628
- end
478
+ There is no need to `require 'sidekiq/web'` since `sidekiq_unique_jobs/web`
479
+ already does this.
629
480
 
630
- class UniqueJobWithFilterProc
631
- include Sidekiq::Worker
632
- sidekiq_options lock: :until_executed,
633
- lock_args_method: ->(args) { [ args.first ] }
481
+ To filter/search for keys we can use the wildcard `*`. If we have a unique digest `'uniquejobs:9e9b5ce5d423d3ea470977004b50ff84` we can search for it by enter `*ff84` and it should return all digests that end with `ff84`.
634
482
 
635
- ...
483
+ ### Reflections (metrics, logging, etc.)
636
484
 
637
- end
638
- ```
485
+ To be able to gather some insights on what is going on inside this gem. I provide a reflection API that can be used.
639
486
 
640
- It is possible to ensure different types of unique args based on context. I can't vouch for the below example but see [#203](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/203) for the discussion.
487
+ To setup reflections for logging or metrics, use the following API:
641
488
 
642
489
  ```ruby
643
- class UniqueJobWithFilterMethod
644
- include Sidekiq::Worker
645
- sidekiq_options lock: :until_and_while_executing, lock_args_method: :lock_args
646
490
 
647
- def self.lock_args(args)
648
- if Sidekiq::ProcessSet.new.size > 1
649
- # sidekiq runtime; uniqueness for the object (first arg)
650
- args.first
651
- else
652
- # queuing from the app; uniqueness for all params
653
- args
654
- end
491
+ def extract_log_from_job(message, job_hash)
492
+ worker = job_hash['class']
493
+ args = job_hash['args']
494
+ lock_args = job_hash['lock_args']
495
+ queue = job_hash['queue']
496
+ {
497
+ message: message,
498
+ worker: worker,
499
+ args: args,
500
+ lock_args: lock_args,
501
+ queue: queue
502
+ }
503
+ end
504
+
505
+ SidekiqUniqueJobs.reflect do |on|
506
+ on.lock_failed do |job_hash|
507
+ message = extract_log_from_job('Lock Failed', job_hash)
508
+ Sidekiq.logger.warn(message)
655
509
  end
656
510
  end
657
511
  ```
658
512
 
659
- ### After Unlock Callback
513
+ #### after_unlock_callback_failed
660
514
 
661
- If you need to perform any additional work after the lock has been released you can provide an `#after_unlock` instance method. The method will be called when the lock has been unlocked. Most times this means after yield but there are two exceptions to that.
515
+ This is called when you have configured a custom callback for when a lock has been released.
662
516
 
663
- **Exception 1:** UntilExecuting unlocks and uses callback before yielding.
664
- **Exception 2:** UntilExpired expires eventually, no after_unlock hook is called.
517
+ #### error
665
518
 
666
- **NOTE:** _It is also possible to write this code as a class method._
519
+ Not in use yet but will be used deep into the stack to provide a means to catch and report errors inside the gem.
667
520
 
668
- ```ruby
669
- class UniqueJobWithFilterMethod
670
- include Sidekiq::Worker
671
- sidekiq_options lock: :while_executing,
521
+ #### execution_failed
672
522
 
673
- def self.after_unlock
674
- # block has yielded and lock is released
675
- end
523
+ When the sidekiq processor picks the job of the queue for certain jobs but your job raised an error to the middleware. This will be the reflection. It is probably nothing to worry about. When your worker raises an error, we need to handle some edge cases for until and while executing.
676
524
 
677
- def after_unlock
678
- # block has yielded and lock is released
679
- end
680
- ...
681
- end.
682
- ```
525
+ #### lock_failed
526
+
527
+ If we can't achieve a lock, this will be the reflection. It most likely is nothing to worry about. We just couldn't retrieve a lock in a timely fashion.
528
+
529
+ The biggest reason for this reflection would be to gather metrics on which workers fail the most at the locking step for example.
530
+
531
+ #### locked
532
+
533
+ For when a lock has been successful. Again, mostly useful for metrics I suppose.
534
+
535
+ #### reschedule_failed
536
+
537
+ For when the reschedule strategy failed to reschedule the job.
538
+
539
+ #### rescheduled
540
+
541
+ For when a job was successfully rescheduled
542
+
543
+ #### timeout
544
+
545
+ This is also mostly useful for reporting/metrics purposes. What this reflection does is signal that the job was configured to wait (`lock_timeout` was configured), but we couldn't retrieve a lock even though we waited for some time.
546
+
547
+ ### unlock_failed
548
+
549
+ This is not got, this is worth
550
+
551
+ ### unlocked
552
+
553
+ Also mostly useful for reporting purposes. The job was successfully unlocked.
554
+
555
+ ### unknown_sidekiq_worker
556
+
557
+ The reason this happens is that the server couldn't find a valid sidekiq worker class. Most likely, that worker isn't intended to be processed by this sidekiq server instance.
558
+
559
+ #### Show Locks
560
+
561
+ ![Locks](assets/unique_digests_1.png)
562
+
563
+ #### Show Lock
564
+
565
+ ![Lock](assets/unique_digests_2.png)
566
+
567
+ ## Testing
683
568
 
684
- ### Logging
569
+ ### Validating Worker Configuration
685
570
 
686
- To see logging in sidekiq when duplicate payload has been filtered out you can enable on a per worker basis using the sidekiq options. The default value is false
571
+ Since v7 it is possible to perform some simple validation against your workers sidekiq_options. What it does is scan for some issues that are known to cause problems in production.
572
+
573
+ Let's take a _bad_ worker:
687
574
 
688
575
  ```ruby
689
- class UniqueJobWithFilterMethod
690
- include Sidekiq::Worker
691
- sidekiq_options lock: :while_executing,
692
- log_duplicate: true
576
+ #app/workers/bad_worker.rb
577
+ class BadWorker
578
+ sidekiq_options lock: :while_executing, on_conflict: :replace
579
+ end
693
580
 
694
- ...
581
+ #spec/workers/bad_worker_spec.rb
695
582
 
583
+ require "sidekiq_unique_jobs/testing"
584
+ #OR
585
+ require "sidekiq_unique_jobs/rspec/matchers"
586
+
587
+ RSpec.describe BadWorker do
588
+ specify { expect(described_class).to have_valid_sidekiq_options }
696
589
  end
697
590
  ```
698
591
 
699
- ### Cleanup Dead Locks
592
+ This gives us a helpful error message for a wrongly configured worker:
700
593
 
701
- For sidekiq versions before 5.1 a `sidekiq_retries_exhausted` block is required per worker class. This is deprecated in Sidekiq 6.0
594
+ ```bash
595
+ Expected BadWorker to have valid sidekiq options but found the following problems:
596
+ on_server_conflict: :replace is incompatible with the server process
597
+ ```
598
+
599
+ If you are not using RSpec (a lot of people prefer minitest or test unit) you can do something like:
702
600
 
703
601
  ```ruby
704
- class MyWorker
705
- sidekiq_retries_exhausted do |msg, _ex|
706
- digest = msg['lock_digest']
707
- SidekiqUniqueJobs::Digests.new.delete_by_digest(digest) if digest
708
- end
602
+ assert SidekiqUniqueJobs.validate_worker!(BadWorker.get_sidekiq_options)
603
+ ```
604
+
605
+ ### Uniqueness
606
+
607
+ This has been probably the most confusing part of this gem. People get really confused with how unreliable the unique jobs have been. I there for decided to do what Mike is doing for sidekiq enterprise. Read the section about unique jobs: [Enterprise unique jobs][]
608
+
609
+ ```ruby
610
+ SidekiqUniqueJobs.configure do |config|
611
+ config.enabled = !Rails.env.test?
709
612
  end
710
613
  ```
711
614
 
712
- Starting in v5.1, Sidekiq can also fire a global callback when a job dies: In version 7, this is handled automatically for you. You don't need to add a death handler, if you configure v7 like in [Add the middleware](#add-the-middleware) you don't have to worry about the below.
615
+ If you truly wanted to test the sidekiq client push you could do something like below. Note that it will only work for the jobs that lock when the client pushes the job to redis (UntilExecuted, UntilAndWhileExecuting and UntilExpired).
713
616
 
714
617
  ```ruby
715
- Sidekiq.configure_server do |config|
716
- config.death_handlers << ->(job, _ex) do
717
- digest = job['lock_digest']
718
- SidekiqUniqueJobs::Digests.new.delete_by_digest(digest) if digest
618
+ require "sidekiq_unique_jobs/testing"
619
+
620
+ RSpec.describe Workers::CoolOne do
621
+ before do
622
+ SidekiqUniqueJobs.config.enabled = false
623
+ end
624
+
625
+ # ... your tests that don't test uniqueness
626
+
627
+ context 'when Sidekiq::Testing.disabled?' do
628
+ before do
629
+ Sidekiq::Testing.disable!
630
+ Sidekiq.redis(&:flushdb)
631
+ end
632
+
633
+ after do
634
+ Sidekiq.redis(&:flushdb)
635
+ end
636
+
637
+ it 'prevents duplicate jobs from being scheduled' do
638
+ SidekiqUniqueJobs.use_config(enabled: true) do
639
+ expect(described_class.perform_in(3600, 1)).not_to eq(nil)
640
+ expect(described_class.perform_async(1)).to eq(nil)
641
+ end
642
+ end
719
643
  end
720
644
  end
721
645
  ```
722
646
 
647
+ It is recommended to leave the uniqueness testing to the gem maintainers. If you care about how the gem is integration tested have a look at the following specs:
648
+
649
+ - [spec/sidekiq_unique_jobs/lock/until_and_while_executing_spec.rb](https://github.com/mhenrixon/sidekiq-unique-jobs/blob/master/spec/sidekiq_unique_jobs/lock/until_and_while_executing_spec.rb)
650
+ - [spec/sidekiq_unique_jobs/lock/until_executed_spec.rb](https://github.com/mhenrixon/sidekiq-unique-jobs/blob/master/spec/sidekiq_unique_jobs/lock/until_executed_spec.rb)
651
+ - [spec/sidekiq_unique_jobs/lock/until_expired_spec.rb](https://github.com/mhenrixon/sidekiq-unique-jobs/blob/master/spec/sidekiq_unique_jobs/lock/until_expired_spec.rb)
652
+ - [spec/sidekiq_unique_jobs/lock/while_executing_reject_spec.rb](https://github.com/mhenrixon/sidekiq-unique-jobs/blob/master/spec/sidekiq_unique_jobs/lock/while_executing_reject_spec.rb)
653
+ - [spec/sidekiq_unique_jobs/lock/while_executing_spec.rb](https://github.com/mhenrixon/sidekiq-unique-jobs/blob/master/spec/sidekiq_unique_jobs/lock/while_executing_spec.rb)
654
+
655
+ ## Configuration
656
+
723
657
  ### Other Sidekiq gems
724
658
 
725
659
  #### apartment-sidekiq
@@ -792,124 +726,310 @@ end
792
726
 
793
727
  The reason for this is that if a job is duplicated it shouldn't end up with the status middleware at all. Status is just a monitor so to prevent clashes, leftovers and ensure cleanup. The status middleware should run after uniqueness on client and before on server. This will lead to less surprises.
794
728
 
795
- ## Debugging
729
+ ### Global Configuration
796
730
 
797
- There are several ways of removing keys that are stuck. The prefered way is by using the unique extension to `Sidekiq::Web`. The old console and command line versions still work but might be deprecated in the future. It is better to search for the digest itself and delete the keys matching that digest.
731
+ The gem supports a few different configuration options that might be of interest if you run into some weird issues.
798
732
 
799
- ### Sidekiq Web
733
+ Configure SidekiqUniqueJobs in an initializer or the sidekiq initializer on application startup.
800
734
 
801
- To use the web extension you need to require it in your routes.
735
+ ```ruby
736
+ SidekiqUniqueJobs.configure do |config|
737
+ config.logger = Sidekiq.logger # default, change at your own discretion
738
+ config.debug_lua = false # Turn on when debugging
739
+ config.lock_info = false # Turn on when debugging
740
+ config.lock_ttl = 600 # Expire locks after 10 minutes
741
+ config.lock_timeout = nil # turn off lock timeout
742
+ config.max_history = 0 # Turn on when debugging
743
+ config.reaper = :ruby # :ruby, :lua or :none/nil
744
+ config.reaper_count = 1000 # Stop reaping after this many keys
745
+ config.reaper_interval = 600 # Reap orphans every 10 minutes
746
+ config.reaper_timeout = 150 # Timeout reaper after 2.5 minutes
747
+ end
748
+ ```
749
+
750
+ #### debug_lua
802
751
 
803
752
  ```ruby
804
- #app/config/routes.rb
805
- require 'sidekiq_unique_jobs/web'
806
- mount Sidekiq::Web, at: '/sidekiq'
753
+ SidekiqUniqueJobs.config.debug_lua #=> false
807
754
  ```
808
755
 
809
- There is no need to `require 'sidekiq/web'` since `sidekiq_unique_jobs/web`
810
- already does this.
756
+ Turning on debug_lua will allow the lua scripts to output debug information about what the lua scripts do. It will log all redis commands that are executed and also some helpful messages about what is going on inside the lua script.
811
757
 
812
- To filter/search for keys we can use the wildcard `*`. If we have a unique digest `'uniquejobs:9e9b5ce5d423d3ea470977004b50ff84` we can search for it by enter `*ff84` and it should return all digests that end with `ff84`.
758
+ #### lock_timeout
813
759
 
814
- #### Show Locks
760
+ ```ruby
761
+ SidekiqUniqueJobs.config.lock_timeout #=> 0
762
+ ```
815
763
 
816
- ![Locks](assets/unique_digests_1.png)
764
+ Set a global lock_timeout to use for all jobs that don't otherwise specify a lock_timeout.
817
765
 
818
- #### Show Lock
766
+ Lock timeout decides how long to wait for acquiring the lock. A value of nil means to wait indefinitely for a lock resource to become available.
819
767
 
820
- ![Lock](assets/unique_digests_2.png)
768
+ #### lock_ttl
821
769
 
822
- ## Communication
770
+ ```ruby
771
+ SidekiqUniqueJobs.config.lock_ttl #=> nil
772
+ ```
823
773
 
824
- There is a [![Join the chat at https://gitter.im/mhenrixon/sidekiq-unique-jobs](https://badges.gitter.im/mhenrixon/sidekiq-unique-jobs.svg)](https://gitter.im/mhenrixon/sidekiq-unique-jobs?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) for praise or scorn. This would be a good place to have lengthy discuss or brilliant suggestions or simply just nudge me if I forget about anything.
774
+ Set a global lock_ttl to use for all jobs that don't otherwise specify a lock_ttl.
825
775
 
826
- ## Testing
776
+ Lock TTL decides how long to wait at most before considering a lock to be expired and making it possible to reuse that lock.
827
777
 
828
- ### Unique Sidekiq Configuration
778
+ #### enabled
829
779
 
830
- Since v7 it is possible to perform some simple validation against your workers sidekiq_options. What it does is scan for some issues that are known to cause problems in production.
780
+ ```ruby
781
+ SidekiqUniqueJobs.config.enabled #=> true
782
+ ```
831
783
 
832
- Let's take a _bad_ worker:
784
+ Globally turn the locking mechanism on or off.
785
+
786
+ #### logger
833
787
 
834
788
  ```ruby
835
- #app/workers/bad_worker.rb
836
- class BadWorker
837
- sidekiq_options lock: :while_executing, on_conflict: :replace
838
- end
789
+ SidekiqUniqueJobs.config.logger #=> #<Sidekiq::Logger:0x00007fdc1f96d180>
790
+ ```
839
791
 
840
- #spec/workers/bad_worker_spec.rb
792
+ By default this gem piggybacks on the Sidekiq logger. It is not recommended to change this as the gem uses some features in the Sidekiq logger and you might run into problems. If you need a different logger and you do run into problems then get in touch and we'll see what we can do about it.
841
793
 
842
- require "sidekiq_unique_jobs/testing"
843
- #OR
844
- require "sidekiq_unique_jobs/rspec/matchers"
794
+ #### max_history
845
795
 
846
- RSpec.describe BadWorker do
847
- specify { expect(described_class).to have_valid_sidekiq_options }
848
- end
796
+ ```ruby
797
+ SidekiqUniqueJobs.config.max_history #=> 1_000
849
798
  ```
850
799
 
851
- This gives us a helpful error message for a wrongly configured worker:
800
+ The max_history setting can be used to tweak the number of changelogs generated. It can also be completely turned off if performance suffers or if you are just not interested in using the changelog.
852
801
 
853
- ```bash
854
- Expected BadWorker to have valid sidekiq options but found the following problems:
855
- on_server_conflict: :replace is incompatible with the server process
802
+ This is a log that can be accessed by a lock to see what happened for that lock. Any items after the configured `max_history` will be automatically deleted as new items are added.
803
+
804
+ #### reaper
805
+
806
+ ```ruby
807
+ SidekiqUniqueJobs.config.reaper #=> :ruby
856
808
  ```
857
809
 
858
- If you are not using RSpec (a lot of people prefer minitest or test unit) you can do something like:
810
+ If using the orphans cleanup process it is critical to be aware of the following. The `:ruby` job is much slower but the `:lua` job locks redis while executing. While doing intense processing it is best to avoid locking redis with a lua script. There for the batch size (controlled by the `reaper_count` setting) needs to be reduced.
811
+
812
+ In my benchmarks deleting 1000 orphaned locks with lua performs around 65% faster than deleting 1000 keys in ruby.
813
+
814
+ On the other hand if I increase it to 10 000 orphaned locks per cleanup (`reaper_count: 10_0000`) then redis starts throwing:
815
+
816
+ > BUSY Redis is busy running a script. You can only call SCRIPT KILL or SHUTDOWN NOSAVE. (Redis::CommandError)
817
+
818
+ If you want to disable the reaper set it to `:none`, `nil` or `false`. Actually, any value that isn't `:ruby` or `:lua` will disable the reaping.
859
819
 
860
820
  ```ruby
861
- assert SidekiqUniqueJobs.validate_worker!(BadWorker.get_sidekiq_options)
821
+ SidekiqUniqueJobs.config.reaper = :none
822
+ SidekiqUniqueJobs.config.reaper = nil
823
+ SidekiqUniqueJobs.config.reaper = false
862
824
  ```
863
825
 
864
- ### Uniqueness
826
+ #### reaper_count
865
827
 
866
- This has been probably the most confusing part of this gem. People get really confused with how unreliable the unique jobs have been. I there for decided to do what Mike is doing for sidekiq enterprise. Read the section about unique jobs: [Enterprise unique jobs][]
828
+ ```ruby
829
+ SidekiqUniqueJobs.config.reaper_count #=> 1_000
830
+ ```
831
+
832
+ The reaper_count setting configures how many orphans at a time will be cleaned up by the orphan cleanup job. This might have to be tweaked depending on which orphan job is running.
833
+
834
+ #### reaper_interval
867
835
 
868
836
  ```ruby
869
- SidekiqUniqueJobs.configure do |config|
870
- config.enabled = !Rails.env.test?
837
+ SidekiqUniqueJobs.config.reaper_interval #=> 600
838
+ ```
839
+
840
+ The number of seconds between reaping.
841
+
842
+ #### reaper_timeout
843
+
844
+ ```ruby
845
+ SidekiqUniqueJobs.config.reaper_timeout #=> 10
846
+ ```
847
+
848
+ The number of seconds to wait for the reaper to finish before raising a TimeoutError. This is done to ensure that the next time we reap isn't getting stuck due to the previous process already running.
849
+
850
+ #### lock_prefix
851
+
852
+ ```ruby
853
+ SidekiqUniqueJobs.config.lock_prefix #=> "uniquejobs"
854
+ ```
855
+
856
+ Use if you want a different key prefix for the keys in redis.
857
+
858
+ ### lock_info
859
+
860
+ ```ruby
861
+ SidekiqUniqueJobs.config.lock_info #=> false
862
+ ```
863
+
864
+ Using lock info will create an additional key for the lock with a json object containing information about the lock. This will be presented in the web interface and might help track down why some jobs are getting stuck.
865
+
866
+ ### Worker Configuration
867
+
868
+ #### lock_info
869
+
870
+ Lock info gathers information about a specific lock. It collects things like which `lock_args` where used to compute the `lock_digest` that is used for maintaining uniqueness.
871
+
872
+ ```ruby
873
+ sidekiq_options lock_info: false # this is the default, set to true to turn on
874
+ ```
875
+
876
+ #### lock_prefix
877
+
878
+ Use if you want a different key prefix for the keys in redis.
879
+
880
+ ```ruby
881
+ sidekiq_options lock_prefix: "uniquejobs" # this is the default value
882
+ ```
883
+
884
+ #### lock_ttl
885
+
886
+ Lock TTL decides how long to wait at most before considering a lock to be expired and making it possible to reuse that lock.
887
+
888
+ Starting from `v7` the expiration will take place when the job is pushed to the queue.
889
+
890
+ ```ruby
891
+ sidekiq_options lock_ttl: nil # default - don't expire keys
892
+ sidekiq_options lock_ttl: 20.days.to_i # expire this lock in 20 days
893
+ ```
894
+
895
+ #### lock_timeout
896
+
897
+ This is the timeout (how long to wait) when creating the lock. By default we don't use a timeout so we won't wait for the lock to be created. If you want it is possible to set this like below.
898
+
899
+ ```ruby
900
+ sidekiq_options lock_timeout: 0 # default - don't wait at all
901
+ sidekiq_options lock_timeout: 5 # wait 5 seconds
902
+ sidekiq_options lock_timeout: nil # lock indefinitely, this process won't continue until it gets a lock. VERY DANGEROUS!!
903
+ ```
904
+
905
+ #### unique_across_queues
906
+
907
+ This configuration option is slightly misleading. It doesn't disregard the queue on other jobs. Just on itself, this means that a worker that might schedule jobs into multiple queues will be able to have uniqueness enforced on all queues it is pushed to.
908
+
909
+ This is mainly intended for `Worker.set(queue: :another).perform_async`.
910
+
911
+ ```ruby
912
+ class Worker
913
+ include Sidekiq::Worker
914
+
915
+ sidekiq_options unique_across_queues: true, queue: 'default'
916
+
917
+ def perform(args); end
871
918
  end
872
919
  ```
873
920
 
874
- If you truly wanted to test the sidekiq client push you could do something like below. Note that it will only work for the jobs that lock when the client pushes the job to redis (UntilExecuted, UntilAndWhileExecuting and UntilExpired).
921
+ Now if you push override the queue with `Worker.set(queue: 'another').perform_async(1)` it will still be considered unique when compared to `Worker.perform_async(1)` (that was actually pushed to the queue `default`).
922
+
923
+ #### unique_across_workers
924
+
925
+ This configuration option is slightly misleading. It doesn't disregard the worker class on other jobs. Just on itself, this means that the worker class won't be used for generating the unique digest. The only way this option really makes sense is when you want to have uniqueness between two different worker classes.
875
926
 
876
927
  ```ruby
877
- require "sidekiq_unique_jobs/testing"
928
+ class WorkerOne
929
+ include Sidekiq::Worker
878
930
 
879
- RSpec.describe Workers::CoolOne do
880
- before do
881
- SidekiqUniqueJobs.config.enabled = false
931
+ sidekiq_options unique_across_workers: true, queue: 'default'
932
+
933
+ def perform(args); end
934
+ end
935
+
936
+ class WorkerTwo
937
+ include Sidekiq::Worker
938
+
939
+ sidekiq_options unique_across_workers: true, queue: 'default'
940
+
941
+ def perform(args); end
942
+ end
943
+
944
+
945
+ WorkerOne.perform_async(1)
946
+ # => 'the jobs unique id'
947
+
948
+ WorkerTwo.perform_async(1)
949
+ # => nil because WorkerOne just stole the lock
950
+ ```
951
+
952
+ ### Finer Control over Uniqueness
953
+
954
+ Sometimes it is desired to have a finer control over which arguments are used in determining uniqueness of the job, and others may be _transient_. For this use-case, you need to define either a `lock_args` method, or a ruby proc.
955
+
956
+ *NOTE:* The lock_args method need to return an array of values to use for uniqueness check.
957
+
958
+ *NOTE:* The arguments passed to the proc or the method is always an array. If your method takes a single array as argument the value of args will be `[[...]]`.
959
+
960
+ The method or the proc can return a modified version of args without the transient arguments included, as shown below:
961
+
962
+ ```ruby
963
+ class UniqueJobWithFilterMethod
964
+ include Sidekiq::Worker
965
+ sidekiq_options lock: :until_and_while_executing,
966
+ lock_args_method: :lock_args # this is default and will be used if such a method is defined
967
+
968
+ def self.lock_args(args)
969
+ [ args[0], args[2][:type] ]
882
970
  end
883
971
 
884
- # ... your tests that don't test uniqueness
972
+ ...
885
973
 
886
- context 'when Sidekiq::Testing.disabled?' do
887
- before do
888
- Sidekiq::Testing.disable!
889
- Sidekiq.redis(&:flushdb)
890
- end
974
+ end
891
975
 
892
- after do
893
- Sidekiq.redis(&:flushdb)
894
- end
976
+ class UniqueJobWithFilterProc
977
+ include Sidekiq::Worker
978
+ sidekiq_options lock: :until_executed,
979
+ lock_args_method: ->(args) { [ args.first ] }
895
980
 
896
- it 'prevents duplicate jobs from being scheduled' do
897
- SidekiqUniqueJobs.use_config(enabled: true) do
898
- expect(described_class.perform_in(3600, 1)).not_to eq(nil)
899
- expect(described_class.perform_async(1)).to eq(nil)
900
- end
981
+ ...
982
+
983
+ end
984
+ ```
985
+
986
+ It is possible to ensure different types of unique args based on context. I can't vouch for the below example but see [#203](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/203) for the discussion.
987
+
988
+ ```ruby
989
+ class UniqueJobWithFilterMethod
990
+ include Sidekiq::Worker
991
+ sidekiq_options lock: :until_and_while_executing, lock_args_method: :lock_args
992
+
993
+ def self.lock_args(args)
994
+ if Sidekiq::ProcessSet.new.size > 1
995
+ # sidekiq runtime; uniqueness for the object (first arg)
996
+ args.first
997
+ else
998
+ # queuing from the app; uniqueness for all params
999
+ args
901
1000
  end
902
1001
  end
903
1002
  end
904
1003
  ```
905
1004
 
906
- It is recommened to leave the uniqueness testing to the gem maintainers. If you care about how the gem is integration tested have a look at the following specs:
1005
+ ### After Unlock Callback
907
1006
 
908
- - [spec/integration/sidekiq_unique_jobs/lock/until_and_while_executing_spec.rb](https://github.com/mhenrixon/sidekiq-unique-jobs/blob/master/spec/integration/sidekiq_unique_jobs/lock/until_and_while_executing_spec.rb)
909
- - [spec/integration/sidekiq_unique_jobs/lock/until_executed_spec.rb](https://github.com/mhenrixon/sidekiq-unique-jobs/blob/master/spec/integration/sidekiq_unique_jobs/lock/until_executed_spec.rb)
910
- - [spec/integration/sidekiq_unique_jobs/lock/until_expired_spec.rb](https://github.com/mhenrixon/sidekiq-unique-jobs/blob/master/spec/integration/sidekiq_unique_jobs/lock/until_expired_spec.rb)
911
- - [spec/integration/sidekiq_unique_jobs/lock/while_executing_reject_spec.rb](https://github.com/mhenrixon/sidekiq-unique-jobs/blob/master/spec/integration/sidekiq_unique_jobs/lock/while_executing_reject_spec.rb)
912
- - [spec/integration/sidekiq_unique_jobs/lock/while_executing_spec.rb](https://github.com/mhenrixon/sidekiq-unique-jobs/blob/master/spec/integration/sidekiq_unique_jobs/lock/while_executing_spec.rb)
1007
+ If you need to perform any additional work after the lock has been released you can provide an `#after_unlock` instance method. The method will be called when the lock has been unlocked. Most times this means after yield but there are two exceptions to that.
1008
+
1009
+ **Exception 1:** UntilExecuting unlocks and uses callback before yielding.
1010
+ **Exception 2:** UntilExpired expires eventually, no after_unlock hook is called.
1011
+
1012
+ **NOTE:** _It is also possible to write this code as a class method._
1013
+
1014
+ ```ruby
1015
+ class UniqueJobWithFilterMethod
1016
+ include Sidekiq::Worker
1017
+ sidekiq_options lock: :while_executing,
1018
+
1019
+ def self.after_unlock
1020
+ # block has yielded and lock is released
1021
+ end
1022
+
1023
+ def after_unlock
1024
+ # block has yielded and lock is released
1025
+ end
1026
+ ...
1027
+ end.
1028
+ ```
1029
+
1030
+ ## Communication
1031
+
1032
+ There is a [![Join the chat at https://gitter.im/mhenrixon/sidekiq-unique-jobs](https://badges.gitter.im/mhenrixon/sidekiq-unique-jobs.svg)](https://gitter.im/mhenrixon/sidekiq-unique-jobs?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) for praise or scorn. This would be a good place to have lengthy discuss or brilliant suggestions or simply just nudge me if I forget about anything.
913
1033
 
914
1034
  ## Contributing
915
1035
 
@@ -923,8 +1043,8 @@ It is recommened to leave the uniqueness testing to the gem maintainers. If you
923
1043
 
924
1044
  You can find a list of contributors over on [Contributors][]
925
1045
 
926
- [v5.0.10]: https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v5.0.10.
1046
+ [Enterprise unique jobs]: https://github.com/mperham/sidekiq/wiki/Ent-Unique-Jobs
1047
+ [Contributors]: https://github.com/mhenrixon/sidekiq-unique-jobs/graphs/contributors
927
1048
  [v4.0.18]: https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v4.0.18
1049
+ [v5.0.10]: https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v5.0.10.
928
1050
  [Sidekiq requirements]: https://github.com/mperham/sidekiq#requirements
929
- [Enterprise unique jobs]: https://www.dailydrip.com/topics/sidekiq/drips/sidekiq-enterprise-unique-jobs
930
- [Contributors]: https://github.com/mhenrixon/sidekiq-unique-jobs/graphs/contributors