sidekiq-unique-jobs 3.0.11 → 8.0.10

Sign up to get free protection for your applications and to get access to all the features.
Files changed (158) hide show
  1. checksums.yaml +5 -5
  2. data/CHANGELOG.md +2163 -25
  3. data/LICENSE.txt +21 -0
  4. data/README.md +984 -47
  5. data/bin/uniquejobs +7 -0
  6. data/lib/sidekiq-unique-jobs.rb +2 -36
  7. data/lib/sidekiq_unique_jobs/batch_delete.rb +120 -0
  8. data/lib/sidekiq_unique_jobs/changelog.rb +68 -0
  9. data/lib/sidekiq_unique_jobs/cli.rb +95 -0
  10. data/lib/sidekiq_unique_jobs/config.rb +306 -33
  11. data/lib/sidekiq_unique_jobs/connection.rb +20 -0
  12. data/lib/sidekiq_unique_jobs/constants.rb +55 -0
  13. data/lib/sidekiq_unique_jobs/core_ext.rb +132 -0
  14. data/lib/sidekiq_unique_jobs/deprecation.rb +65 -0
  15. data/lib/sidekiq_unique_jobs/digests.rb +134 -0
  16. data/lib/sidekiq_unique_jobs/exceptions.rb +105 -0
  17. data/lib/sidekiq_unique_jobs/expiring_digests.rb +14 -0
  18. data/lib/sidekiq_unique_jobs/job.rb +63 -0
  19. data/lib/sidekiq_unique_jobs/json.rb +47 -0
  20. data/lib/sidekiq_unique_jobs/key.rb +98 -0
  21. data/lib/sidekiq_unique_jobs/lock/base_lock.rb +165 -0
  22. data/lib/sidekiq_unique_jobs/lock/client_validator.rb +28 -0
  23. data/lib/sidekiq_unique_jobs/lock/server_validator.rb +27 -0
  24. data/lib/sidekiq_unique_jobs/lock/until_and_while_executing.rb +71 -0
  25. data/lib/sidekiq_unique_jobs/lock/until_executed.rb +48 -0
  26. data/lib/sidekiq_unique_jobs/lock/until_executing.rb +43 -0
  27. data/lib/sidekiq_unique_jobs/lock/until_expired.rb +42 -0
  28. data/lib/sidekiq_unique_jobs/lock/validator.rb +96 -0
  29. data/lib/sidekiq_unique_jobs/lock/while_executing.rb +70 -0
  30. data/lib/sidekiq_unique_jobs/lock/while_executing_reject.rb +21 -0
  31. data/lib/sidekiq_unique_jobs/lock.rb +348 -0
  32. data/lib/sidekiq_unique_jobs/lock_args.rb +127 -0
  33. data/lib/sidekiq_unique_jobs/lock_config.rb +132 -0
  34. data/lib/sidekiq_unique_jobs/lock_digest.rb +79 -0
  35. data/lib/sidekiq_unique_jobs/lock_info.rb +68 -0
  36. data/lib/sidekiq_unique_jobs/lock_timeout.rb +62 -0
  37. data/lib/sidekiq_unique_jobs/lock_ttl.rb +77 -0
  38. data/lib/sidekiq_unique_jobs/lock_type.rb +37 -0
  39. data/lib/sidekiq_unique_jobs/locksmith.rb +390 -0
  40. data/lib/sidekiq_unique_jobs/logging/middleware_context.rb +44 -0
  41. data/lib/sidekiq_unique_jobs/logging.rb +236 -0
  42. data/lib/sidekiq_unique_jobs/lua/delete.lua +49 -0
  43. data/lib/sidekiq_unique_jobs/lua/delete_by_digest.lua +39 -0
  44. data/lib/sidekiq_unique_jobs/lua/delete_job_by_digest.lua +38 -0
  45. data/lib/sidekiq_unique_jobs/lua/find_digest_in_queues.lua +26 -0
  46. data/lib/sidekiq_unique_jobs/lua/lock.lua +108 -0
  47. data/lib/sidekiq_unique_jobs/lua/lock_until_expired.lua +92 -0
  48. data/lib/sidekiq_unique_jobs/lua/locked.lua +35 -0
  49. data/lib/sidekiq_unique_jobs/lua/queue.lua +88 -0
  50. data/lib/sidekiq_unique_jobs/lua/reap_orphans.lua +119 -0
  51. data/lib/sidekiq_unique_jobs/lua/shared/_common.lua +35 -0
  52. data/lib/sidekiq_unique_jobs/lua/shared/_current_time.lua +8 -0
  53. data/lib/sidekiq_unique_jobs/lua/shared/_delete_from_queue.lua +22 -0
  54. data/lib/sidekiq_unique_jobs/lua/shared/_delete_from_sorted_set.lua +29 -0
  55. data/lib/sidekiq_unique_jobs/lua/shared/_find_digest_in_process_set.lua +53 -0
  56. data/lib/sidekiq_unique_jobs/lua/shared/_find_digest_in_queues.lua +43 -0
  57. data/lib/sidekiq_unique_jobs/lua/shared/_find_digest_in_sorted_set.lua +24 -0
  58. data/lib/sidekiq_unique_jobs/lua/shared/_hgetall.lua +13 -0
  59. data/lib/sidekiq_unique_jobs/lua/shared/_upgrades.lua +3 -0
  60. data/lib/sidekiq_unique_jobs/lua/unlock.lua +112 -0
  61. data/lib/sidekiq_unique_jobs/lua/update_version.lua +40 -0
  62. data/lib/sidekiq_unique_jobs/lua/upgrade.lua +66 -0
  63. data/lib/sidekiq_unique_jobs/middleware/client.rb +42 -0
  64. data/lib/sidekiq_unique_jobs/middleware/server.rb +31 -0
  65. data/lib/sidekiq_unique_jobs/middleware.rb +41 -15
  66. data/lib/sidekiq_unique_jobs/normalizer.rb +17 -0
  67. data/lib/sidekiq_unique_jobs/on_conflict/log.rb +24 -0
  68. data/lib/sidekiq_unique_jobs/on_conflict/null_strategy.rb +16 -0
  69. data/lib/sidekiq_unique_jobs/on_conflict/raise.rb +17 -0
  70. data/lib/sidekiq_unique_jobs/on_conflict/reject.rb +75 -0
  71. data/lib/sidekiq_unique_jobs/on_conflict/replace.rb +82 -0
  72. data/lib/sidekiq_unique_jobs/on_conflict/reschedule.rb +39 -0
  73. data/lib/sidekiq_unique_jobs/on_conflict/strategy.rb +51 -0
  74. data/lib/sidekiq_unique_jobs/on_conflict.rb +44 -0
  75. data/lib/sidekiq_unique_jobs/options_with_fallback.rb +78 -0
  76. data/lib/sidekiq_unique_jobs/orphans/lua_reaper.rb +29 -0
  77. data/lib/sidekiq_unique_jobs/orphans/manager.rb +242 -0
  78. data/lib/sidekiq_unique_jobs/orphans/null_reaper.rb +24 -0
  79. data/lib/sidekiq_unique_jobs/orphans/observer.rb +42 -0
  80. data/lib/sidekiq_unique_jobs/orphans/reaper.rb +115 -0
  81. data/lib/sidekiq_unique_jobs/orphans/reaper_resurrector.rb +170 -0
  82. data/lib/sidekiq_unique_jobs/orphans/ruby_reaper.rb +313 -0
  83. data/lib/sidekiq_unique_jobs/redis/entity.rb +112 -0
  84. data/lib/sidekiq_unique_jobs/redis/hash.rb +56 -0
  85. data/lib/sidekiq_unique_jobs/redis/list.rb +32 -0
  86. data/lib/sidekiq_unique_jobs/redis/set.rb +32 -0
  87. data/lib/sidekiq_unique_jobs/redis/sorted_set.rb +102 -0
  88. data/lib/sidekiq_unique_jobs/redis/string.rb +51 -0
  89. data/lib/sidekiq_unique_jobs/redis.rb +11 -0
  90. data/lib/sidekiq_unique_jobs/reflectable.rb +26 -0
  91. data/lib/sidekiq_unique_jobs/reflections.rb +79 -0
  92. data/lib/sidekiq_unique_jobs/rspec/matchers/have_valid_sidekiq_options.rb +51 -0
  93. data/lib/sidekiq_unique_jobs/rspec/matchers.rb +26 -0
  94. data/lib/sidekiq_unique_jobs/script/caller.rb +133 -0
  95. data/lib/sidekiq_unique_jobs/script/client.rb +94 -0
  96. data/lib/sidekiq_unique_jobs/script/config.rb +68 -0
  97. data/lib/sidekiq_unique_jobs/script/dsl.rb +60 -0
  98. data/lib/sidekiq_unique_jobs/script/logging.rb +95 -0
  99. data/lib/sidekiq_unique_jobs/script/lua_error.rb +96 -0
  100. data/lib/sidekiq_unique_jobs/script/script.rb +75 -0
  101. data/lib/sidekiq_unique_jobs/script/scripts.rb +123 -0
  102. data/lib/sidekiq_unique_jobs/script/template.rb +41 -0
  103. data/lib/sidekiq_unique_jobs/script/timing.rb +35 -0
  104. data/lib/sidekiq_unique_jobs/script.rb +46 -0
  105. data/lib/sidekiq_unique_jobs/server.rb +62 -0
  106. data/lib/sidekiq_unique_jobs/sidekiq_unique_ext.rb +110 -37
  107. data/lib/sidekiq_unique_jobs/sidekiq_unique_jobs.rb +304 -0
  108. data/lib/sidekiq_unique_jobs/sidekiq_worker_methods.rb +84 -0
  109. data/lib/sidekiq_unique_jobs/testing.rb +132 -9
  110. data/lib/sidekiq_unique_jobs/timer_task.rb +299 -0
  111. data/lib/sidekiq_unique_jobs/timing.rb +58 -0
  112. data/lib/sidekiq_unique_jobs/unlockable.rb +43 -0
  113. data/lib/sidekiq_unique_jobs/update_version.rb +25 -0
  114. data/lib/sidekiq_unique_jobs/upgrade_locks.rb +152 -0
  115. data/lib/sidekiq_unique_jobs/version.rb +5 -1
  116. data/lib/sidekiq_unique_jobs/version_check.rb +114 -0
  117. data/lib/sidekiq_unique_jobs/web/helpers.rb +175 -0
  118. data/lib/sidekiq_unique_jobs/web/views/_paging.erb +10 -0
  119. data/lib/sidekiq_unique_jobs/web/views/changelogs.erb +60 -0
  120. data/lib/sidekiq_unique_jobs/web/views/lock.erb +110 -0
  121. data/lib/sidekiq_unique_jobs/web/views/locks.erb +59 -0
  122. data/lib/sidekiq_unique_jobs/web.rb +109 -0
  123. data/lib/sidekiq_unique_jobs.rb +83 -0
  124. data/lib/tasks/changelog.rake +23 -0
  125. metadata +157 -126
  126. data/.gitignore +0 -10
  127. data/.rspec +0 -3
  128. data/.rubocop.yml +0 -36
  129. data/.travis.yml +0 -25
  130. data/Appraisals +0 -20
  131. data/Gemfile +0 -5
  132. data/LICENSE +0 -22
  133. data/Rakefile +0 -11
  134. data/gemfiles/sidekiq_2.15.gemfile +0 -9
  135. data/gemfiles/sidekiq_2.16.gemfile +0 -9
  136. data/gemfiles/sidekiq_2.17.gemfile +0 -9
  137. data/gemfiles/sidekiq_3.0.gemfile +0 -9
  138. data/gemfiles/sidekiq_develop.gemfile +0 -9
  139. data/lib/sidekiq_unique_jobs/connectors/redis_pool.rb +0 -11
  140. data/lib/sidekiq_unique_jobs/connectors/sidekiq_redis.rb +0 -9
  141. data/lib/sidekiq_unique_jobs/connectors/testing.rb +0 -11
  142. data/lib/sidekiq_unique_jobs/connectors.rb +0 -16
  143. data/lib/sidekiq_unique_jobs/middleware/client/strategies/testing_inline.rb +0 -25
  144. data/lib/sidekiq_unique_jobs/middleware/client/strategies/unique.rb +0 -76
  145. data/lib/sidekiq_unique_jobs/middleware/client/unique_jobs.rb +0 -39
  146. data/lib/sidekiq_unique_jobs/middleware/server/unique_jobs.rb +0 -69
  147. data/lib/sidekiq_unique_jobs/payload_helper.rb +0 -42
  148. data/sidekiq-unique-jobs.gemspec +0 -27
  149. data/spec/lib/.sidekiq_testing_enabled_spec.rb.swp +0 -0
  150. data/spec/lib/client_spec.rb +0 -173
  151. data/spec/lib/middleware/server/unique_jobs_spec.rb +0 -81
  152. data/spec/lib/sidekiq_testing_enabled_spec.rb +0 -123
  153. data/spec/lib/sidekiq_unique_ext_spec.rb +0 -70
  154. data/spec/lib/unlock_order_spec.rb +0 -64
  155. data/spec/spec_helper.rb +0 -37
  156. data/spec/support/my_worker.rb +0 -13
  157. data/spec/support/sidekiq_meta.rb +0 -17
  158. data/spec/support/unique_worker.rb +0 -13
data/README.md CHANGED
@@ -1,64 +1,973 @@
1
- # SidekiqUniqueJobs [![Build Status](https://travis-ci.org/mhenrixon/sidekiq-unique-jobs.png?branch=master)](https://travis-ci.org/mhenrixon/sidekiq-unique-jobs) [![Code Climate](https://codeclimate.com/github/mhenrixon/sidekiq-unique-jobs.png)](https://codeclimate.com/github/mhenrixon/sidekiq-unique-jobs)
1
+ # SidekiqUniqueJobs
2
2
 
3
- The missing unique jobs for sidekiq
3
+ [![Join the chat at https://gitter.im/mhenrixon/sidekiq-unique-jobs](https://badges.gitter.im/mhenrixon/sidekiq-unique-jobs.svg)](https://gitter.im/mhenrixon/sidekiq-unique-jobs?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) ![Build Status](https://github.com/mhenrixon/sidekiq-unique-jobs/actions/workflows/rspec.yml/badge.svg?branch=main) [![Code Climate](https://codeclimate.com/github/mhenrixon/sidekiq-unique-jobs.svg)](https://codeclimate.com/github/mhenrixon/sidekiq-unique-jobs) [![Test Coverage](https://codeclimate.com/github/mhenrixon/sidekiq-unique-jobs/badges/coverage.svg)](https://codeclimate.com/github/mhenrixon/sidekiq-unique-jobs/coverage)
4
4
 
5
- ## Installation
5
+ ## Support Me
6
+
7
+ Want to show me some ❤️ for the hard work I do on this gem? You can use the following PayPal link: [https://paypal.me/mhenrixon1](https://paypal.me/mhenrixon1). Any amount is welcome and let me tell you it feels good to be appreciated. Even a dollar makes me super excited about all of this.
8
+
9
+ <!-- MarkdownTOC -->
10
+
11
+ - [Introduction](#introduction)
12
+ - [Usage](#usage)
13
+ - [Installation](#installation)
14
+ - [Add the middleware](#add-the-middleware)
15
+ - [Your first worker](#your-first-worker)
16
+ - [Requirements](#requirements)
17
+ - [Locks](#locks)
18
+ - [Until Executing](#until-executing)
19
+ - [Example worker](#example-worker)
20
+ - [Until Executed](#until-executed)
21
+ - [Example worker](#example-worker-1)
22
+ - [Until Expired](#until-expired)
23
+ - [Example worker](#example-worker-2)
24
+ - [Until And While Executing](#until-and-while-executing)
25
+ - [Example worker](#example-worker-3)
26
+ - [While Executing](#while-executing)
27
+ - [Example worker](#example-worker-4)
28
+ - [Custom Locks](#custom-locks)
29
+ - [Conflict Strategy](#conflict-strategy)
30
+ - [log](#log)
31
+ - [raise](#raise)
32
+ - [reject](#reject)
33
+ - [replace](#replace)
34
+ - [reschedule](#reschedule)
35
+ - [Custom Strategies](#custom-strategies)
36
+ - [3 Cleanup Dead Locks](#3-cleanup-dead-locks)
37
+ - [Debugging](#debugging)
38
+ - [Sidekiq Web](#sidekiq-web)
39
+ - [Reflections \(metrics, logging, etc.\)](#reflections-metrics-logging-etc)
40
+ - [after_unlock_callback_failed](#after_unlock_callback_failed)
41
+ - [error](#error)
42
+ - [execution_failed](#execution_failed)
43
+ - [lock_failed](#lock_failed)
44
+ - [locked](#locked)
45
+ - [reschedule_failed](#reschedule_failed)
46
+ - [rescheduled](#rescheduled)
47
+ - [timeout](#timeout)
48
+ - [unlock_failed](#unlock_failed)
49
+ - [unlocked](#unlocked)
50
+ - [unknown_sidekiq_worker](#unknown_sidekiq_worker)
51
+ - [Show Locks](#show-locks)
52
+ - [Show Lock](#show-lock)
53
+ - [Testing](#testing)
54
+ - [Validating Worker Configuration](#validating-worker-configuration)
55
+ - [Uniqueness](#uniqueness)
56
+ - [Configuration](#configuration)
57
+ - [Other Sidekiq gems](#other-sidekiq-gems)
58
+ - [apartment-sidekiq](#apartment-sidekiq)
59
+ - [sidekiq-global_id](#sidekiq-global_id)
60
+ - [sidekiq-status](#sidekiq-status)
61
+ - [Global Configuration](#global-configuration)
62
+ - [debug_lua](#debug_lua)
63
+ - [lock_timeout](#lock_timeout)
64
+ - [lock_ttl](#lock_ttl)
65
+ - [enabled](#enabled)
66
+ - [logger](#logger)
67
+ - [max_history](#max_history)
68
+ - [reaper](#reaper)
69
+ - [reaper_count](#reaper_count)
70
+ - [reaper_interval](#reaper_interval)
71
+ - [reaper_timeout](#reaper_timeout)
72
+ - [lock_prefix](#lock_prefix)
73
+ - [lock_info](#lock_info)
74
+ - [Worker Configuration](#worker-configuration)
75
+ - [lock_info](#lock_info-1)
76
+ - [lock_prefix](#lock_prefix-1)
77
+ - [lock_ttl](#lock_ttl-1)
78
+ - [lock_timeout](#lock_timeout-1)
79
+ - [unique_across_queues](#unique_across_queues)
80
+ - [unique_across_workers](#unique_across_workers)
81
+ - [Finer Control over Uniqueness](#finer-control-over-uniqueness)
82
+ - [After Unlock Callback](#after-unlock-callback)
83
+ - [Communication](#communication)
84
+ - [Contributing](#contributing)
85
+ - [Contributors](#contributors)
86
+
87
+ <!-- /MarkdownTOC -->
88
+
89
+ ## Introduction
90
+
91
+ This gem adds unique constraints to sidekiq jobs. The uniqueness is achieved by creating a set of keys in redis based off of `queue`, `class`, `args` (in the sidekiq job hash).
92
+
93
+ By default, only one lock for a given hash can be acquired. What happens when a lock can't be acquired is governed by a chosen [Conflict Strategy](#conflict-strategy) strategy. Unless a conflict strategy is chosen (?)
94
+
95
+ This is the documentation for the `main` branch. You can find the documentation for each release by navigating to its tag.
96
+
97
+ Here are links to some of the old versions
98
+
99
+ - [v7.0.12](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v7.0.12)
100
+ - [v6.0.25](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v6.0.25)
101
+ - [v5.0.10](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v5.0.10)
102
+ - [v4.0.18](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v4.0.18)
103
+
104
+ ## Usage
105
+
106
+ ### Installation
6
107
 
7
108
  Add this line to your application's Gemfile:
8
109
 
9
- gem 'sidekiq-unique-jobs'
110
+ ```ruby
111
+ gem 'sidekiq-unique-jobs'
112
+ ```
10
113
 
11
114
  And then execute:
12
115
 
13
- $ bundle
116
+ ```bash
117
+ bundle
118
+ ```
119
+
120
+ ### Add the middleware
14
121
 
15
- Or install it yourself as:
122
+ Before v7, the middleware was configured automatically. Since some people reported issues with other gems (see [Other Sidekiq Gems](#other-sidekiq-gems)) it was decided to give full control over to the user.
16
123
 
17
- $ gem install sidekiq-unique-jobs
124
+ *NOTE* if you want to use the reaper you also need to configure the server middleware.
18
125
 
19
- ## Usage
126
+ The following shows how to modify your `config/initializers/sidekiq.rb` file to use the middleware. [Here is a full example.](https://github.com/mhenrixon/sidekiq-unique-jobs/blob/master/myapp/config/initializers/sidekiq.rb#L12)
127
+
128
+ ```ruby
129
+ require "sidekiq-unique-jobs"
130
+
131
+ Sidekiq.configure_server do |config|
132
+ config.redis = { url: ENV["REDIS_URL"], driver: :hiredis }
133
+
134
+ config.client_middleware do |chain|
135
+ chain.add SidekiqUniqueJobs::Middleware::Client
136
+ end
137
+
138
+ config.server_middleware do |chain|
139
+ chain.add SidekiqUniqueJobs::Middleware::Server
140
+ end
141
+
142
+ SidekiqUniqueJobs::Server.configure(config)
143
+ end
144
+
145
+ Sidekiq.configure_client do |config|
146
+ config.redis = { url: ENV["REDIS_URL"], driver: :hiredis }
20
147
 
21
- All that is required is that you specifically set the sidekiq option for *unique* to true like below:
148
+ config.client_middleware do |chain|
149
+ chain.add SidekiqUniqueJobs::Middleware::Client
150
+ end
151
+ end
152
+ ```
153
+
154
+ ### Your first worker
155
+
156
+ The lock type most likely to be is `:until_executed`. This type of lock creates a lock from when `UntilExecutedWorker.perform_async` is called until right after `UntilExecutedWorker.new.perform` has been called.
22
157
 
23
158
  ```ruby
24
- sidekiq_options unique: true
159
+ # frozen_string_literal: true
160
+
161
+ class UntilExecutedWorker
162
+ include Sidekiq::Worker
163
+
164
+ sidekiq_options lock: :until_executed
165
+
166
+ def perform
167
+ logger.info("cowboy")
168
+ sleep(1) # hardcore processing
169
+ logger.info("beebop")
170
+ end
171
+ end
25
172
  ```
26
173
 
27
- For jobs scheduled in the future it is possible to set for how long the job
28
- should be unique. The job will be unique for the number of seconds configured (default 30 minutes)
29
- or until the job has been completed. Thus, the job will be unique for the shorter of the two. Note that Sidekiq versions before 3.0 will remove job keys after an hour, which means jobs can remain unique for at most an hour.
174
+ You can read more about the worker configuration in [Worker Configuration](#worker-configuration) below.
175
+
176
+ ## Requirements
177
+
178
+ - Sidekiq `>= 5.0` (`>= 5.2` recommended)
179
+ - Ruby:
180
+ - MRI `>= 2.5` (`>= 2.6` recommended)
181
+ - JRuby `>= 9.0` (`>= 9.2` recommended)
182
+ - Truffleruby
183
+ - Redis Server `>= 3.2` (`>= 5.0` recommended)
184
+ - [ActiveJob officially not supported][48]
185
+ - [redis-namespace officially not supported][49]
186
+
187
+ See [Sidekiq requirements][24] for detailed requirements of Sidekiq itself (be sure to check the right sidekiq version).
30
188
 
31
- *If you want the unique job to stick around even after it has been successfully
32
- processed then just set the unique_unlock_order to anything except `:before_yield` or `:after_yield` (`unique_unlock_order = :never`)
189
+ ## Locks
33
190
 
34
- You can also control the expiration length of the uniqueness check. If you want to enforce uniqueness over a longer period than the default of 30 minutes then you can pass the number of seconds you want to use to the sidekiq options:
191
+ ### Until Executing
192
+
193
+ A lock is created when `UntilExecuting.perform_async` is called. Then it is either unlocked when `lock_ttl` is hit or before Sidekiq calls the `perform` method on your worker.
194
+
195
+ #### Example worker
35
196
 
36
197
  ```ruby
37
- sidekiq_options unique: true, unique_job_expiration: 120 * 60 # 2 hours
198
+ class UntilExecuting
199
+ include Sidekiq::Workers
200
+
201
+ sidekiq_options lock: :until_executing
202
+
203
+ def perform(id)
204
+ # Do work
205
+ end
206
+ end
38
207
  ```
39
208
 
40
- Requiring the gem in your gemfile should be sufficient to enable unique jobs.
209
+ **NOTE** this is probably not so good for jobs that shouldn't be running simultaneously (aka slow jobs).
41
210
 
42
- ### Finer Control over Uniqueness
211
+ The reason this type of lock exists is to fix the following problem: [sidekiq/issues/3471](https://github.com/mperham/sidekiq/issues/3471#issuecomment-300866335)
43
212
 
44
- Sometimes it is desired to have a finer control over which arguments are used in determining uniqueness of the job, and others may be _transient_. For this use-case, you need to set `SidekiqUniqueJobs.config.unique_args_enabled` to true in an initializer, and then defined either `unique_args` method, or a ruby proc.
213
+ ### Until Executed
45
214
 
46
- The unique_args method need to return an array of values to use for uniqueness check.
215
+ A lock is created when `UntilExecuted.perform_async` is called. Then it is either unlocked when `lock_ttl` is hit or when Sidekiq has called the `perform` method on your worker.
216
+
217
+ #### Example worker
47
218
 
48
219
  ```ruby
49
- SidekiqUniqueJobs.config.unique_args_enabled = true
220
+ class UntilExecuted
221
+ include Sidekiq::Workers
222
+
223
+ sidekiq_options lock: :until_executed
224
+
225
+ def perform(id)
226
+ # Do work
227
+ end
228
+ end
50
229
  ```
51
230
 
231
+ ### Until Expired
232
+
233
+ This lock behaves identically to the [Until Executed](#until-executed) except for one thing. This job won't be unlocked until the expiration is hit. For jobs that need to run only once per day, this would be the perfect lock. This way, we can't create more jobs until one day after this job was first pushed.
234
+
235
+ #### Example worker
236
+
237
+ ```ruby
238
+ class UntilExpired
239
+ include Sidekiq::Workers
240
+
241
+ sidekiq_options lock: :until_expired, lock_ttl: 1.day
242
+
243
+ def perform
244
+ # Do work
245
+ end
246
+ end
247
+ ```
248
+
249
+ ### Until And While Executing
250
+
251
+ This lock is a combination of two locks (`:until_executing` and `:while_executing`). Please see the configuration for [Until Executing](#until-executing) and [While Executing](#while-executing)
252
+
253
+ #### Example worker
254
+
255
+ ```ruby
256
+ class UntilAndWhileExecutingWorker
257
+ include Sidekiq::Workers
258
+
259
+ sidekiq_options lock: :until_and_while_executing,
260
+ lock_timeout: 2,
261
+ on_conflict: {
262
+ client: :log,
263
+ server: :raise
264
+ }
265
+ def perform(id)
266
+ # Do work
267
+ end
268
+ end
269
+ ```
270
+
271
+ ### While Executing
272
+
273
+ These locks are put on a queue without any type of locking mechanism, the locking doesn't happen until Sidekiq pops the job from the queue and starts processing it.
274
+
275
+ #### Example worker
276
+
277
+ ```ruby
278
+ class WhileExecutingWorker
279
+ include Sidekiq::Workers
280
+
281
+ sidekiq_options lock: :while_executing,
282
+ lock_timeout: 2,
283
+ on_conflict: {
284
+ server: :raise
285
+ }
286
+ def perform(id)
287
+ # Do work
288
+ end
289
+ end
290
+ ```
291
+
292
+ **NOTE** Unless a conflict strategy of `:raise` is specified, if lock fails, the job will be dropped without notice. When told to raise, the job will be put back and retried. It would also be possible to use `:reschedule` with this lock.
293
+
294
+ **NOTE** Unless this job is configured with a `lock_timeout: nil` or `lock_timeout: > 0` then all jobs that are attempted to be executed will just be dropped without waiting.
295
+
296
+ There is an example of this to try it out in the `myapp` application. Run `foreman start` in the root of the directory and open the url: `localhost:5000/work/duplicate_while_executing`.
297
+
298
+ In the console you should see something like:
299
+
300
+ ```bash
301
+ 0:32:24 worker.1 | 2017-04-23T08:32:24.955Z 84404 TID-ougq4thko WhileExecutingWorker JID-400ec51c9523f41cd4a35058 INFO: start
302
+ 10:32:24 worker.1 | 2017-04-23T08:32:24.956Z 84404 TID-ougq8csew WhileExecutingWorker JID-8d6d9168368eedaed7f75763 INFO: start
303
+ 10:32:24 worker.1 | 2017-04-23T08:32:24.957Z 84404 TID-ougq8crt8 WhileExecutingWorker JID-affcd079094c9b26e8b9ba60 INFO: start
304
+ 10:32:24 worker.1 | 2017-04-23T08:32:24.959Z 84404 TID-ougq8cs8s WhileExecutingWorker JID-9e197460c067b22eb1b5d07f INFO: start
305
+ 10:32:24 worker.1 | 2017-04-23T08:32:24.959Z 84404 TID-ougq4thko WhileExecutingWorker JID-400ec51c9523f41cd4a35058 WhileExecutingWorker INFO: perform(1, 2)
306
+ 10:32:34 worker.1 | 2017-04-23T08:32:34.964Z 84404 TID-ougq4thko WhileExecutingWorker JID-400ec51c9523f41cd4a35058 INFO: done: 10.009 sec
307
+ 10:32:34 worker.1 | 2017-04-23T08:32:34.965Z 84404 TID-ougq8csew WhileExecutingWorker JID-8d6d9168368eedaed7f75763 WhileExecutingWorker INFO: perform(1, 2)
308
+ 10:32:44 worker.1 | 2017-04-23T08:32:44.965Z 84404 TID-ougq8crt8 WhileExecutingWorker JID-affcd079094c9b26e8b9ba60 WhileExecutingWorker INFO: perform(1, 2)
309
+ 10:32:44 worker.1 | 2017-04-23T08:32:44.965Z 84404 TID-ougq8csew WhileExecutingWorker JID-8d6d9168368eedaed7f75763 INFO: done: 20.009 sec
310
+ 10:32:54 worker.1 | 2017-04-23T08:32:54.970Z 84404 TID-ougq8cs8s WhileExecutingWorker JID-9e197460c067b22eb1b5d07f WhileExecutingWorker INFO: perform(1, 2)
311
+ 10:32:54 worker.1 | 2017-04-23T08:32:54.969Z 84404 TID-ougq8crt8 WhileExecutingWorker JID-affcd079094c9b26e8b9ba60 INFO: done: 30.012 sec
312
+ 10:33:04 worker.1 | 2017-04-23T08:33:04.973Z 84404 TID-ougq8cs8s WhileExecutingWorker JID-9e197460c067b22eb1b5d07f INFO: done: 40.014 sec
313
+ ```
314
+
315
+ ### Custom Locks
316
+
317
+ You may need to define some custom lock. You can define it in one project folder:
318
+
319
+ ```ruby
320
+ # lib/locks/my_custom_lock.rb
321
+ module Locks
322
+ class MyCustomLock < SidekiqUniqueJobs::Lock::BaseLock
323
+ def execute
324
+ # Do something ...
325
+ end
326
+ end
327
+ end
328
+ ```
329
+
330
+ You can refer on all the locks defined in `lib/sidekiq_unique_jobs/lock/*.rb`.
331
+
332
+ In order to make it available, you should call in your project startup:
333
+
334
+ (For rails application config/initializers/sidekiq_unique_jobs.rb or other projects, wherever you prefer)
335
+
336
+ ```ruby
337
+ SidekiqUniqueJobs.configure do |config|
338
+ config.add_lock :my_custom_lock, Locks::MyCustomLock
339
+ end
340
+ ```
341
+
342
+ And then you can use it in the jobs definition:
343
+
344
+ `sidekiq_options lock: :my_custom_lock, on_conflict: :log`
345
+
346
+ Please not that if you try to override a default lock, an `ArgumentError` will be raised.
347
+
348
+ ## Conflict Strategy
349
+
350
+ Decides how we handle conflict. We can either `reject` the job to the dead queue or `reschedule` it. Both are useful for jobs that absolutely need to run and have been configured to use the lock `WhileExecuting` that is used only by the sidekiq server process.
351
+
352
+ Furthermore, `log` can be be used with the lock `UntilExecuted` and `UntilExpired`. Now we write a log entry saying the job could not be pushed because it is a duplicate of another job with the same arguments.
353
+
354
+ It is possible for locks to have different conflict strategy for the client and server. This is useful for `:until_and_while_executing`.
355
+
356
+ ```ruby
357
+ sidekiq_options lock: :until_and_while_executing,
358
+ on_conflict: { client: :log, server: :reject }
359
+ ```
360
+
361
+ ### log
362
+
363
+ ```ruby
364
+ sidekiq_options on_conflict: :log
365
+ ```
366
+
367
+ This strategy is intended to be used with `UntilExecuted` and `UntilExpired`. It will log a line that this job is a duplicate of another.
368
+
369
+ ### raise
370
+
371
+ ```ruby
372
+ sidekiq_options on_conflict: :raise
373
+ ```
374
+
375
+ This strategy is intended to be used with `WhileExecuting`. Basically it will allow us to let the server process crash with a specific error message and be retried without messing up the Sidekiq stats.
376
+
377
+ ### reject
378
+
379
+ ```ruby
380
+ sidekiq_options on_conflict: :reject
381
+ ```
382
+
383
+ This strategy is intended to be used with `WhileExecuting` and will push the job to the dead queue on conflict.
384
+
385
+ ### replace
386
+
387
+ ```ruby
388
+ sidekiq_options on_conflict: :replace
389
+ ```
390
+
391
+ This strategy is intended to be used with client locks like `UntilExecuted`.
392
+ It will delete any existing job for these arguments from retry, schedule and
393
+ queue and retry the lock again.
394
+
395
+ This is slightly dangerous and should probably only be used for jobs that are
396
+ always scheduled in the future. Currently only attempting to retry one time.
397
+
398
+ ### reschedule
399
+
400
+ ```ruby
401
+ sidekiq_options on_conflict: :reschedule
402
+ ```
403
+
404
+ This strategy is intended to be used with `WhileExecuting` and will delay the job to be tried again in 5 seconds. This will mess up the sidekiq stats but will prevent exceptions from being logged and confuse your sysadmins.
405
+
406
+ ### Custom Strategies
407
+
408
+ You may need to define some custom strategy. You can define it in one project folder:
409
+
410
+ ```ruby
411
+ # lib/strategies/my_custom_strategy.rb
412
+ module Strategies
413
+ class MyCustomStrategy < SidekiqUniqueJobs::OnConflict::Strategy
414
+ def call
415
+ # Do something ...
416
+ end
417
+ end
418
+ end
419
+ ```
420
+
421
+ You can refer to all the strategies defined in `lib/sidekiq_unique_jobs/on_conflict`.
422
+
423
+ In order to make it available, you should call in your project startup:
424
+
425
+ (For rails application config/initializers/sidekiq_unique_jobs.rb for other projects, wherever you prefer)
426
+
427
+ ```ruby
428
+ SidekiqUniqueJobs.configure do |config|
429
+ config.add_strategy :my_custom_strategy, Strategies::MyCustomStrategy
430
+ end
431
+ ```
432
+
433
+ And then you can use it in the jobs definition:
434
+
435
+ ```ruby
436
+ sidekiq_options lock: :while_executing, on_conflict: :my_custom_strategy
437
+ ```
438
+
439
+ Please not that if you try to override a default lock, an `ArgumentError` will be raised.
440
+
441
+ ### 3 Cleanup Dead Locks
442
+
443
+ For sidekiq versions < 5.1 a `sidekiq_retries_exhausted` block is required per worker class. This is deprecated in Sidekiq 6.0
444
+
445
+ ```ruby
446
+ class MyWorker
447
+ sidekiq_retries_exhausted do |msg, _ex|
448
+ digest = msg['lock_digest']
449
+ SidekiqUniqueJobs::Digests.new.delete_by_digest(digest) if digest
450
+ end
451
+ end
452
+ ```
453
+
454
+ Starting in v5.1, Sidekiq can also fire a global callback when a job dies: In version 7, this is handled automatically for you. You don't need to add a death handler, if you configure v7 like in [Add the middleware](#add-the-middleware) you don't have to worry about the below.
455
+
456
+ ```ruby
457
+ Sidekiq.configure_server do |config|
458
+ config.death_handlers << ->(job, _ex) do
459
+ digest = job['lock_digest']
460
+ SidekiqUniqueJobs::Digests.new.delete_by_digest(digest) if digest
461
+ end
462
+ end
463
+ ```
464
+
465
+ ## Debugging
466
+
467
+ There are several ways of removing keys that are stuck. The prefered way is by using the unique extension to `Sidekiq::Web`. The old console and command line versions still work but might be deprecated in the future. It is better to search for the digest itself and delete the keys matching that digest.
468
+
469
+ ### Sidekiq Web
470
+
471
+ To use the web extension you need to require it in your routes.
472
+
473
+ ```ruby
474
+ #app/config/routes.rb
475
+ require 'sidekiq_unique_jobs/web'
476
+ mount Sidekiq::Web, at: '/sidekiq'
477
+ ```
478
+
479
+ There is no need to `require 'sidekiq/web'` since `sidekiq_unique_jobs/web`
480
+ already does this.
481
+
482
+ To filter/search for keys we can use the wildcard `*`. If we have a unique digest `'uniquejobs:9e9b5ce5d423d3ea470977004b50ff84` we can search for it by enter `*ff84` and it should return all digests that end with `ff84`.
483
+
484
+ ### Reflections (metrics, logging, etc.)
485
+
486
+ To be able to gather some insights on what is going on inside this gem. I provide a reflection API that can be used.
487
+
488
+ To setup reflections for logging or metrics, use the following API:
489
+
490
+ ```ruby
491
+
492
+ def extract_log_from_job(message, job_hash)
493
+ worker = job_hash['class']
494
+ args = job_hash['args']
495
+ lock_args = job_hash['lock_args']
496
+ queue = job_hash['queue']
497
+ {
498
+ message: message,
499
+ worker: worker,
500
+ args: args,
501
+ lock_args: lock_args,
502
+ queue: queue
503
+ }
504
+ end
505
+
506
+ SidekiqUniqueJobs.reflect do |on|
507
+ on.lock_failed do |job_hash|
508
+ message = extract_log_from_job('Lock Failed', job_hash)
509
+ Sidekiq.logger.warn(message)
510
+ end
511
+ end
512
+ ```
513
+
514
+ #### after_unlock_callback_failed
515
+
516
+ This is called when you have configured a custom callback for when a lock has been released.
517
+
518
+ #### error
519
+
520
+ Not in use yet but will be used deep into the stack to provide a means to catch and report errors inside the gem.
521
+
522
+ #### execution_failed
523
+
524
+ When the sidekiq processor picks the job of the queue for certain jobs but your job raised an error to the middleware. This will be the reflection. It is probably nothing to worry about. When your worker raises an error, we need to handle some edge cases for until and while executing.
525
+
526
+ #### lock_failed
527
+
528
+ If we can't achieve a lock, this will be the reflection. It most likely is nothing to worry about. We just couldn't retrieve a lock in a timely fashion.
529
+
530
+ The biggest reason for this reflection would be to gather metrics on which workers fail the most at the locking step for example.
531
+
532
+ #### locked
533
+
534
+ For when a lock has been successful. Again, mostly useful for metrics I suppose.
535
+
536
+ #### reschedule_failed
537
+
538
+ For when the reschedule strategy failed to reschedule the job.
539
+
540
+ #### rescheduled
541
+
542
+ For when a job was successfully rescheduled
543
+
544
+ #### timeout
545
+
546
+ This is also mostly useful for reporting/metrics purposes. What this reflection does is signal that the job was configured to wait (`lock_timeout` was configured), but we couldn't retrieve a lock even though we waited for some time.
547
+
548
+ #### unlock_failed
549
+
550
+ This means that the server middleware could not unlock your job and the lock is kept (potentially preventing subsequent jobs from being pushed or processed).
551
+
552
+ #### unlocked
553
+
554
+ Also mostly useful for reporting purposes. The job was successfully unlocked.
555
+
556
+ #### unknown_sidekiq_worker
557
+
558
+ The reason this happens is that the server couldn't find a valid sidekiq worker class. Most likely, that worker isn't intended to be processed by this sidekiq server instance.
559
+
560
+ ### Show Locks
561
+
562
+ ![Locks](assets/unique_digests_1.png)
563
+
564
+ ### Show Lock
565
+
566
+ ![Lock](assets/unique_digests_2.png)
567
+
568
+ ## Testing
569
+
570
+ ### Validating Worker Configuration
571
+
572
+ Since v7 it is possible to perform some simple validation against your workers `sidekiq_options`. What it does is scan for some issues that are known to cause problems in production.
573
+
574
+ Let's take a _bad_ worker:
575
+
576
+ ```ruby
577
+ #app/workers/bad_worker.rb
578
+ class BadWorker
579
+ sidekiq_options lock: :while_executing, on_conflict: :replace
580
+ end
581
+
582
+ #spec/workers/bad_worker_spec.rb
583
+
584
+ require "sidekiq_unique_jobs/testing"
585
+
586
+ RSpec.describe BadWorker do
587
+ specify { expect(described_class).to have_valid_sidekiq_options }
588
+ end
589
+ ```
590
+
591
+ This gives us a helpful error message for a wrongly configured worker:
592
+
593
+ ```bash
594
+ Expected BadWorker to have valid sidekiq options but found the following problems:
595
+ on_server_conflict: :replace is incompatible with the server process
596
+ ```
597
+
598
+ If you are not using RSpec (a lot of people prefer minitest or test unit) you can do something like:
599
+
600
+ ```ruby
601
+ assert_raise(InvalidWorker){ SidekiqUniqueJobs.validate_worker!(BadWorker.get_sidekiq_options) }
602
+ ```
603
+
604
+ ### Uniqueness
605
+
606
+ This has been probably the most confusing part of this gem. People get really confused with how unreliable the unique jobs have been. I there for decided to do what Mike is doing for sidekiq enterprise. Read the section about unique jobs: [Enterprise unique jobs][](?)
607
+
608
+ ```ruby
609
+ SidekiqUniqueJobs.configure do |config|
610
+ config.enabled = !Rails.env.test?
611
+ config.logger_enabled = !Rails.env.test?
612
+ end
613
+ ```
614
+
615
+ If you truly wanted to test the sidekiq client push you could do something like below. Note that it will only work for the jobs that lock when the client pushes the job to redis (UntilExecuted, UntilAndWhileExecuting and UntilExpired).
616
+
617
+ ```ruby
618
+ require "sidekiq_unique_jobs/testing"
619
+
620
+ RSpec.describe Workers::CoolOne do
621
+ before do
622
+ SidekiqUniqueJobs.config.enabled = false
623
+ end
624
+
625
+ # ... your tests that don't test uniqueness
626
+
627
+ context 'when Sidekiq::Testing.disabled?' do
628
+ before do
629
+ Sidekiq::Testing.disable!
630
+ Sidekiq.redis(&:flushdb)
631
+ end
632
+
633
+ after do
634
+ Sidekiq.redis(&:flushdb)
635
+ end
636
+
637
+ it 'prevents duplicate jobs from being scheduled' do
638
+ SidekiqUniqueJobs.use_config(enabled: true) do
639
+ expect(described_class.perform_in(3600, 1)).not_to eq(nil)
640
+ expect(described_class.perform_async(1)).to eq(nil)
641
+ end
642
+ end
643
+ end
644
+ end
645
+ ```
646
+
647
+ It is recommended to leave the uniqueness testing to the gem maintainers. If you care about how the gem is integration tested have a look at the following specs:
648
+
649
+ - [spec/sidekiq_unique_jobs/lock/until_and_while_executing_spec.rb](https://github.com/mhenrixon/sidekiq-unique-jobs/blob/master/spec/sidekiq_unique_jobs/lock/until_and_while_executing_spec.rb)
650
+ - [spec/sidekiq_unique_jobs/lock/until_executed_spec.rb](https://github.com/mhenrixon/sidekiq-unique-jobs/blob/master/spec/sidekiq_unique_jobs/lock/until_executed_spec.rb)
651
+ - [spec/sidekiq_unique_jobs/lock/until_expired_spec.rb](https://github.com/mhenrixon/sidekiq-unique-jobs/blob/master/spec/sidekiq_unique_jobs/lock/until_expired_spec.rb)
652
+ - [spec/sidekiq_unique_jobs/lock/while_executing_reject_spec.rb](https://github.com/mhenrixon/sidekiq-unique-jobs/blob/master/spec/sidekiq_unique_jobs/lock/while_executing_reject_spec.rb)
653
+ - [spec/sidekiq_unique_jobs/lock/while_executing_spec.rb](https://github.com/mhenrixon/sidekiq-unique-jobs/blob/master/spec/sidekiq_unique_jobs/lock/while_executing_spec.rb)
654
+
655
+ ## Configuration
656
+
657
+ ### Other Sidekiq gems
658
+
659
+ #### apartment-sidekiq
660
+
661
+ It was reported in [#536](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/536) that the order of the Sidekiq middleware needs to be as follows.
662
+
663
+ ```ruby
664
+ Sidekiq.client_middleware do |chain|
665
+ chain.add Apartment::Sidekiq::Middleware::Client
666
+ chain.add SidekiqUniqueJobs::Middleware::Client
667
+ end
668
+
669
+ Sidekiq.server_middleware do |chain|
670
+ chain.add Apartment::Sidekiq::Middleware::Server
671
+ chain.add SidekiqUniqueJobs::Middleware::Server
672
+ end
673
+ ```
674
+
675
+ The reason being that this gem needs to be configured AFTER the apartment gem or the apartment will not be able to be considered for uniqueness
676
+
677
+ #### sidekiq-global_id
678
+
679
+ It was reported in [#235](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/235) that the order of the Sidekiq middleware needs to be as follows.
680
+
681
+ For a working setup check the following [file](https://github.com/mhenrixon/sidekiq-unique-jobs/blob/master/myapp/config/sidekiq.rb#L12).
682
+
683
+ ```ruby
684
+ Sidekiq.client_middleware do |chain|
685
+ chain.add Sidekiq::GlobalId::ClientMiddleware
686
+ chain.add SidekiqUniqueJobs::Middleware::Client
687
+ end
688
+
689
+ Sidekiq.server_middleware do |chain|
690
+ chain.add Sidekiq::GlobalId::ServerMiddleware
691
+ chain.add SidekiqUniqueJobs::Middleware::Server
692
+ end
693
+ ```
694
+
695
+ The reason for this is that the global id needs to be set before the unique jobs middleware runs. Otherwise that won't be available for uniqueness.
696
+
697
+ #### sidekiq-status
698
+
699
+ It was reported in [#564](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/564) that the order of the middleware needs to be as follows.
700
+
701
+ ```ruby
702
+ # Thanks to @ArturT for the correction
703
+
704
+ Sidekiq.configure_server do |config|
705
+ config.client_middleware do |chain|
706
+ chain.add SidekiqUniqueJobs::Middleware::Client
707
+ chain.add Sidekiq::Status::ClientMiddleware, expiration: 30.minutes
708
+ end
709
+
710
+ config.server_middleware do |chain|
711
+ chain.add Sidekiq::Status::ServerMiddleware, expiration: 30.minutes
712
+ chain.add SidekiqUniqueJobs::Middleware::Server
713
+ end
714
+
715
+ SidekiqUniqueJobs::Server.configure(config)
716
+ end
717
+
718
+
719
+ Sidekiq.configure_client do |config|
720
+ config.client_middleware do |chain|
721
+ chain.add SidekiqUniqueJobs::Middleware::Client
722
+ chain.add Sidekiq::Status::ClientMiddleware, expiration: 30.minutes
723
+ end
724
+ end
725
+ ```
726
+
727
+ The reason for this is that if a job is duplicated it shouldn't end up with the status middleware at all. Status is just a monitor so to prevent clashes, leftovers and ensure cleanup. The status middleware should run after uniqueness on client and before on server. This will lead to less surprises.
728
+
729
+ ### Global Configuration
730
+
731
+ The gem supports a few different configuration options that might be of interest if you run into some weird issues.
732
+
733
+ Configure SidekiqUniqueJobs in an initializer or the sidekiq initializer on application startup.
734
+
735
+ ```ruby
736
+ SidekiqUniqueJobs.configure do |config|
737
+ config.logger = Sidekiq.logger # default, change at your own discretion
738
+ config.logger_enabled = true # default, disable for test environments
739
+ config.debug_lua = false # Turn on when debugging
740
+ config.lock_info = false # Turn on when debugging
741
+ config.lock_ttl = 600 # Expire locks after 10 minutes
742
+ config.lock_timeout = nil # turn off lock timeout
743
+ config.max_history = 0 # Turn on when debugging
744
+ config.reaper = :ruby # :ruby, :lua or :none/nil
745
+ config.reaper_count = 1000 # Stop reaping after this many keys
746
+ config.reaper_interval = 600 # Reap orphans every 10 minutes
747
+ config.reaper_timeout = 150 # Timeout reaper after 2.5 minutes
748
+ end
749
+ ```
750
+
751
+ #### debug_lua
752
+
753
+ ```ruby
754
+ SidekiqUniqueJobs.config.debug_lua #=> false
755
+ ```
756
+
757
+ Turning on debug_lua will allow the lua scripts to output debug information about what the lua scripts do. It will log all redis commands that are executed and also some helpful messages about what is going on inside the lua script.
758
+
759
+ #### lock_timeout
760
+
761
+ ```ruby
762
+ SidekiqUniqueJobs.config.lock_timeout #=> 0
763
+ ```
764
+
765
+ Set a global lock_timeout to use for all jobs that don't otherwise specify a lock_timeout.
766
+
767
+ Lock timeout decides how long to wait for acquiring the lock. A value of nil means to wait indefinitely for a lock resource to become available.
768
+
769
+ #### lock_ttl
770
+
771
+ ```ruby
772
+ SidekiqUniqueJobs.config.lock_ttl #=> nil
773
+ ```
774
+
775
+ Set a global lock_ttl to use for all jobs that don't otherwise specify a lock_ttl.
776
+
777
+ Lock TTL decides how long to wait at most before considering a lock to be expired and making it possible to reuse that lock.
778
+
779
+ #### enabled
780
+
781
+ ```ruby
782
+ SidekiqUniqueJobs.config.enabled #=> true
783
+ ```
784
+
785
+ Globally turn the locking mechanism on or off.
786
+
787
+ #### logger
788
+
789
+ ```ruby
790
+ SidekiqUniqueJobs.config.logger #=> #<Sidekiq::Logger:0x00007fdc1f96d180>
791
+ ```
792
+
793
+ By default this gem piggybacks on the Sidekiq logger. It is not recommended to change this as the gem uses some features in the Sidekiq logger and you might run into problems. If you need a different logger and you do run into problems then get in touch and we'll see what we can do about it.
794
+
795
+ #### max_history
796
+
797
+ ```ruby
798
+ SidekiqUniqueJobs.config.max_history #=> 1_000
799
+ ```
800
+
801
+ The max_history setting can be used to tweak the number of changelogs generated. It can also be completely turned off if performance suffers or if you are just not interested in using the changelog.
802
+
803
+ This is a log that can be accessed by a lock to see what happened for that lock. Any items after the configured `max_history` will be automatically deleted as new items are added.
804
+
805
+ #### reaper
806
+
807
+ ```ruby
808
+ SidekiqUniqueJobs.config.reaper #=> :ruby
809
+ ```
810
+
811
+ If using the orphans cleanup process it is critical to be aware of the following. The `:ruby` job is much slower but the `:lua` job locks redis while executing. While doing intense processing it is best to avoid locking redis with a lua script. There for the batch size (controlled by the `reaper_count` setting) needs to be reduced.
812
+
813
+ In my benchmarks deleting 1000 orphaned locks with lua performs around 65% faster than deleting 1000 keys in ruby.
814
+
815
+ On the other hand if I increase it to 10 000 orphaned locks per cleanup (`reaper_count: 10_0000`) then redis starts throwing:
816
+
817
+ > BUSY Redis is busy running a script. You can only call SCRIPT KILL or SHUTDOWN NOSAVE. (RedisClient::CommandError)
818
+
819
+ If you want to disable the reaper set it to `:none`, `nil` or `false`. Actually, any value that isn't `:ruby` or `:lua` will disable the reaping.
820
+
821
+ ```ruby
822
+ SidekiqUniqueJobs.config.reaper = :none
823
+ SidekiqUniqueJobs.config.reaper = nil
824
+ SidekiqUniqueJobs.config.reaper = false
825
+ ```
826
+
827
+ #### reaper_count
828
+
829
+ ```ruby
830
+ SidekiqUniqueJobs.config.reaper_count #=> 1_000
831
+ ```
832
+
833
+ The reaper_count setting configures how many orphans at a time will be cleaned up by the orphan cleanup job. This might have to be tweaked depending on which orphan job is running.
834
+
835
+ #### reaper_interval
836
+
837
+ ```ruby
838
+ SidekiqUniqueJobs.config.reaper_interval #=> 600
839
+ ```
840
+
841
+ The number of seconds between reaping.
842
+
843
+ #### reaper_timeout
844
+
845
+ ```ruby
846
+ SidekiqUniqueJobs.config.reaper_timeout #=> 10
847
+ ```
848
+
849
+ The number of seconds to wait for the reaper to finish before raising a TimeoutError. This is done to ensure that the next time we reap isn't getting stuck due to the previous process already running.
850
+
851
+ #### lock_prefix
852
+
853
+ ```ruby
854
+ SidekiqUniqueJobs.config.lock_prefix #=> "uniquejobs"
855
+ ```
856
+
857
+ Use if you want a different key prefix for the keys in redis.
858
+
859
+ ### lock_info
860
+
861
+ ```ruby
862
+ SidekiqUniqueJobs.config.lock_info #=> false
863
+ ```
864
+
865
+ Using lock info will create an additional key for the lock with a json object containing information about the lock. This will be presented in the web interface and might help track down why some jobs are getting stuck.
866
+
867
+ ### Worker Configuration
868
+
869
+ #### lock_info
870
+
871
+ Lock info gathers information about a specific lock. It collects things like which `lock_args` where used to compute the `lock_digest` that is used for maintaining uniqueness.
872
+
873
+ ```ruby
874
+ sidekiq_options lock_info: false # this is the default, set to true to turn on
875
+ ```
876
+
877
+ #### lock_prefix
878
+
879
+ Use if you want a different key prefix for the keys in redis.
880
+
881
+ ```ruby
882
+ sidekiq_options lock_prefix: "uniquejobs" # this is the default value
883
+ ```
884
+
885
+ #### lock_ttl
886
+
887
+ Lock TTL decides how long to wait at most before considering a lock to be expired and making it possible to reuse that lock.
888
+
889
+ Starting from `v7` the expiration will take place when the job is pushed to the queue.
890
+
891
+ ```ruby
892
+ sidekiq_options lock_ttl: nil # default - don't expire keys
893
+ sidekiq_options lock_ttl: 20.days.to_i # expire this lock in 20 days
894
+ ```
895
+
896
+ #### lock_timeout
897
+
898
+ This is the timeout (how long to wait) when creating the lock. By default we don't use a timeout so we won't wait for the lock to be created. If you want it is possible to set this like below.
899
+
900
+ ```ruby
901
+ sidekiq_options lock_timeout: 0 # default - don't wait at all
902
+ sidekiq_options lock_timeout: 5 # wait 5 seconds
903
+ sidekiq_options lock_timeout: nil # lock indefinitely, this process won't continue until it gets a lock. VERY DANGEROUS!!
904
+ ```
905
+
906
+ #### unique_across_queues
907
+
908
+ This configuration option is slightly misleading. It doesn't disregard the queue on other jobs. Just on itself, this means that a worker that might schedule jobs into multiple queues will be able to have uniqueness enforced on all queues it is pushed to.
909
+
910
+ This is mainly intended for `Worker.set(queue: :another).perform_async`.
911
+
912
+ ```ruby
913
+ class Worker
914
+ include Sidekiq::Worker
915
+
916
+ sidekiq_options unique_across_queues: true, queue: 'default'
917
+
918
+ def perform(args); end
919
+ end
920
+ ```
921
+
922
+ Now if you push override the queue with `Worker.set(queue: 'another').perform_async(1)` it will still be considered unique when compared to `Worker.perform_async(1)` (that was actually pushed to the queue `default`).
923
+
924
+ #### unique_across_workers
925
+
926
+ This configuration option is slightly misleading. It doesn't disregard the worker class on other jobs. Just on itself, this means that the worker class won't be used for generating the unique digest. The only way this option really makes sense is when you want to have uniqueness between two different worker classes.
927
+
928
+ ```ruby
929
+ class WorkerOne
930
+ include Sidekiq::Worker
931
+
932
+ sidekiq_options unique_across_workers: true, queue: 'default'
933
+
934
+ def perform(args); end
935
+ end
936
+
937
+ class WorkerTwo
938
+ include Sidekiq::Worker
939
+
940
+ sidekiq_options unique_across_workers: true, queue: 'default'
941
+
942
+ def perform(args); end
943
+ end
944
+
945
+
946
+ WorkerOne.perform_async(1)
947
+ # => 'the jobs unique id'
948
+
949
+ WorkerTwo.perform_async(1)
950
+ # => nil because WorkerOne just stole the lock
951
+ ```
952
+
953
+ ### Finer Control over Uniqueness
954
+
955
+ Sometimes it is desired to have a finer control over which arguments are used in determining uniqueness of the job, and others may be _transient_. For this use-case, you need to define either a `lock_args` method, or a ruby proc.
956
+
957
+ *NOTE:* The lock_args method need to return an array of values to use for uniqueness check.
958
+
959
+ *NOTE:* The arguments passed to the proc or the method is always an array. If your method takes a single array as argument the value of args will be `[[...]]`.
960
+
52
961
  The method or the proc can return a modified version of args without the transient arguments included, as shown below:
53
962
 
54
963
  ```ruby
55
964
  class UniqueJobWithFilterMethod
56
965
  include Sidekiq::Worker
57
- sidekiq_options unique: true,
58
- unique_args: :unique_args
966
+ sidekiq_options lock: :until_and_while_executing,
967
+ lock_args_method: :lock_args # this is default and will be used if such a method is defined
59
968
 
60
- def self.unique_args(name, id, options)
61
- [ name, options[:type] ]
969
+ def self.lock_args(args)
970
+ [ args[0], args[2][:type] ]
62
971
  end
63
972
 
64
973
  ...
@@ -67,48 +976,76 @@ end
67
976
 
68
977
  class UniqueJobWithFilterProc
69
978
  include Sidekiq::Worker
70
- sidekiq_options unique: true,
71
- unique_args: ->(args) { [ args.first ] }
979
+ sidekiq_options lock: :until_executed,
980
+ lock_args_method: ->(args) { [ args.first ] }
72
981
 
73
982
  ...
74
983
 
75
984
  end
76
985
  ```
77
986
 
78
- Note that objects passed into workers are converted to JSON *after* running through client middleware. In server middleware, the JSON is passed directly to the worker `#perform` method. So, you may run into issues where the arguments are different when enqueuing than they are when performing. Your `unique_args` method may need to account for this.
987
+ It is possible to ensure different types of unique args based on context. I can't vouch for the below example but see [#203](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/203) for the discussion.
79
988
 
80
- ### Unlock Ordering
989
+ ```ruby
990
+ class UniqueJobWithFilterMethod
991
+ include Sidekiq::Worker
992
+ sidekiq_options lock: :until_and_while_executing, lock_args_method: :lock_args
81
993
 
82
- By default the server middleware will release the worker lock after yielding to the next middleware or worker. Alternatively, this can be changed by passing the `unique_unlock_order` option:
994
+ def self.lock_args(args)
995
+ if Sidekiq::ProcessSet.new.size > 1
996
+ # sidekiq runtime; uniqueness for the object (first arg)
997
+ args.first
998
+ else
999
+ # queuing from the app; uniqueness for all params
1000
+ args
1001
+ end
1002
+ end
1003
+ end
1004
+ ```
1005
+
1006
+ ### After Unlock Callback
1007
+
1008
+ If you need to perform any additional work after the lock has been released you can provide an `#after_unlock` instance method. The method will be called when the lock has been unlocked. Most times this means after yield but there are two exceptions to that.
1009
+
1010
+ **Exception 1:** UntilExecuting unlocks and uses callback before yielding.
1011
+ **Exception 2:** UntilExpired expires eventually, no after_unlock hook is called.
1012
+
1013
+ **NOTE:** _It is also possible to write this code as a class method._
83
1014
 
84
1015
  ```ruby
85
1016
  class UniqueJobWithFilterMethod
86
1017
  include Sidekiq::Worker
87
- sidekiq_options unique: true,
88
- unique_unlock_order: :before_yield
1018
+ sidekiq_options lock: :while_executing,
89
1019
 
90
- ...
1020
+ def self.after_unlock
1021
+ # block has yielded and lock is released
1022
+ end
91
1023
 
92
- end
1024
+ def after_unlock
1025
+ # block has yielded and lock is released
1026
+ end
1027
+ ...
1028
+ end.
93
1029
  ```
94
1030
 
95
- ### Testing
1031
+ ## Communication
96
1032
 
97
- SidekiqUniqueJobs uses mock_redis for inline testing. Due to complaints about having that as a runtime dependency it was made a development dependency so if you are relying on inline testing you will have to add `gem 'mock_redis'` to your Gemfile.
1033
+ There is a [![Join the chat at https://gitter.im/mhenrixon/sidekiq-unique-jobs](https://badges.gitter.im/mhenrixon/sidekiq-unique-jobs.svg)](https://gitter.im/mhenrixon/sidekiq-unique-jobs?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) for praise or scorn. This would be a good place to have lengthy discuss or brilliant suggestions or simply just nudge me if I forget about anything.
98
1034
 
99
1035
  ## Contributing
100
1036
 
101
1037
  1. Fork it
102
- 2. Create your feature branch (`git checkout -b my-new-feature`)
103
- 3. Commit your changes (`git commit -am 'Add some feature'`)
104
- 4. Push to the branch (`git push origin my-new-feature`)
105
- 5. Create new Pull Request
1038
+ 1. Create your feature branch (`git checkout -b my-new-feature`)
1039
+ 1. Commit your changes (`git commit -am 'Add some feature'`)
1040
+ 1. Push to the branch (`git push origin my-new-feature`)
1041
+ 1. Create new Pull Request
106
1042
 
107
1043
  ## Contributors
108
1044
 
109
- - https://github.com/salrepe
110
- - https://github.com/rickenharp
111
- - https://github.com/sax
112
- - https://github.com/eduardosasso
113
- - https://github.com/KensoDev
114
- - https://github.com/adstage-david
1045
+ You can find a list of contributors over on [Contributors][]
1046
+
1047
+ [Enterprise unique jobs]: https://github.com/mperham/sidekiq/wiki/Ent-Unique-Jobs
1048
+ [Contributors]: https://github.com/mhenrixon/sidekiq-unique-jobs/graphs/contributors
1049
+ [v4.0.18]: https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v4.0.18
1050
+ [v5.0.10]: https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v5.0.10.
1051
+ [Sidekiq requirements]: https://github.com/mperham/sidekiq#requirements