sidekiq-unique-jobs 6.0.20 → 7.0.10
Sign up to get free protection for your applications and to get access to all the features.
Potentially problematic release.
This version of sidekiq-unique-jobs might be problematic. Click here for more details.
- checksums.yaml +4 -4
- data/CHANGELOG.md +742 -41
- data/README.md +529 -110
- data/lib/sidekiq_unique_jobs.rb +48 -7
- data/lib/sidekiq_unique_jobs/batch_delete.rb +123 -0
- data/lib/sidekiq_unique_jobs/changelog.rb +78 -0
- data/lib/sidekiq_unique_jobs/cli.rb +34 -31
- data/lib/sidekiq_unique_jobs/config.rb +263 -0
- data/lib/sidekiq_unique_jobs/connection.rb +6 -5
- data/lib/sidekiq_unique_jobs/constants.rb +46 -24
- data/lib/sidekiq_unique_jobs/core_ext.rb +80 -0
- data/lib/sidekiq_unique_jobs/digests.rb +71 -100
- data/lib/sidekiq_unique_jobs/exceptions.rb +78 -12
- data/lib/sidekiq_unique_jobs/job.rb +41 -12
- data/lib/sidekiq_unique_jobs/json.rb +40 -0
- data/lib/sidekiq_unique_jobs/key.rb +93 -0
- data/lib/sidekiq_unique_jobs/lock.rb +325 -0
- data/lib/sidekiq_unique_jobs/lock/base_lock.rb +68 -52
- data/lib/sidekiq_unique_jobs/lock/client_validator.rb +28 -0
- data/lib/sidekiq_unique_jobs/lock/server_validator.rb +27 -0
- data/lib/sidekiq_unique_jobs/lock/until_and_while_executing.rb +7 -10
- data/lib/sidekiq_unique_jobs/lock/until_executed.rb +6 -6
- data/lib/sidekiq_unique_jobs/lock/until_executing.rb +1 -1
- data/lib/sidekiq_unique_jobs/lock/until_expired.rb +4 -21
- data/lib/sidekiq_unique_jobs/lock/validator.rb +96 -0
- data/lib/sidekiq_unique_jobs/lock/while_executing.rb +13 -9
- data/lib/sidekiq_unique_jobs/lock/while_executing_reject.rb +3 -3
- data/lib/sidekiq_unique_jobs/lock_args.rb +123 -0
- data/lib/sidekiq_unique_jobs/lock_config.rb +126 -0
- data/lib/sidekiq_unique_jobs/lock_digest.rb +79 -0
- data/lib/sidekiq_unique_jobs/lock_info.rb +68 -0
- data/lib/sidekiq_unique_jobs/lock_timeout.rb +62 -0
- data/lib/sidekiq_unique_jobs/lock_ttl.rb +77 -0
- data/lib/sidekiq_unique_jobs/locksmith.rb +267 -101
- data/lib/sidekiq_unique_jobs/logging.rb +179 -33
- data/lib/sidekiq_unique_jobs/logging/middleware_context.rb +44 -0
- data/lib/sidekiq_unique_jobs/lua/delete.lua +51 -0
- data/lib/sidekiq_unique_jobs/lua/delete_by_digest.lua +42 -0
- data/lib/sidekiq_unique_jobs/lua/delete_job_by_digest.lua +38 -0
- data/lib/sidekiq_unique_jobs/lua/find_digest_in_queues.lua +26 -0
- data/lib/sidekiq_unique_jobs/lua/lock.lua +93 -0
- data/lib/sidekiq_unique_jobs/lua/locked.lua +35 -0
- data/lib/sidekiq_unique_jobs/lua/queue.lua +87 -0
- data/lib/sidekiq_unique_jobs/lua/reap_orphans.lua +94 -0
- data/lib/sidekiq_unique_jobs/lua/shared/_common.lua +40 -0
- data/lib/sidekiq_unique_jobs/lua/shared/_current_time.lua +8 -0
- data/lib/sidekiq_unique_jobs/lua/shared/_delete_from_queue.lua +22 -0
- data/lib/sidekiq_unique_jobs/lua/shared/_delete_from_sorted_set.lua +18 -0
- data/lib/sidekiq_unique_jobs/lua/shared/_find_digest_in_process_set.lua +53 -0
- data/lib/sidekiq_unique_jobs/lua/shared/_find_digest_in_queues.lua +43 -0
- data/lib/sidekiq_unique_jobs/lua/shared/_find_digest_in_sorted_set.lua +24 -0
- data/lib/sidekiq_unique_jobs/lua/shared/_hgetall.lua +13 -0
- data/lib/sidekiq_unique_jobs/lua/shared/_upgrades.lua +3 -0
- data/lib/sidekiq_unique_jobs/lua/unlock.lua +95 -0
- data/lib/sidekiq_unique_jobs/lua/update_version.lua +40 -0
- data/lib/sidekiq_unique_jobs/lua/upgrade.lua +68 -0
- data/lib/sidekiq_unique_jobs/middleware.rb +29 -31
- data/lib/sidekiq_unique_jobs/middleware/client.rb +42 -0
- data/lib/sidekiq_unique_jobs/middleware/server.rb +27 -0
- data/lib/sidekiq_unique_jobs/normalizer.rb +4 -4
- data/lib/sidekiq_unique_jobs/on_conflict.rb +23 -10
- data/lib/sidekiq_unique_jobs/on_conflict/log.rb +9 -5
- data/lib/sidekiq_unique_jobs/on_conflict/null_strategy.rb +1 -1
- data/lib/sidekiq_unique_jobs/on_conflict/raise.rb +1 -1
- data/lib/sidekiq_unique_jobs/on_conflict/reject.rb +61 -15
- data/lib/sidekiq_unique_jobs/on_conflict/replace.rb +54 -14
- data/lib/sidekiq_unique_jobs/on_conflict/reschedule.rb +12 -5
- data/lib/sidekiq_unique_jobs/on_conflict/strategy.rb +25 -6
- data/lib/sidekiq_unique_jobs/options_with_fallback.rb +41 -27
- data/lib/sidekiq_unique_jobs/orphans/lua_reaper.rb +29 -0
- data/lib/sidekiq_unique_jobs/orphans/manager.rb +212 -0
- data/lib/sidekiq_unique_jobs/orphans/null_reaper.rb +24 -0
- data/lib/sidekiq_unique_jobs/orphans/observer.rb +42 -0
- data/lib/sidekiq_unique_jobs/orphans/reaper.rb +114 -0
- data/lib/sidekiq_unique_jobs/orphans/ruby_reaper.rb +213 -0
- data/lib/sidekiq_unique_jobs/redis.rb +11 -0
- data/lib/sidekiq_unique_jobs/redis/entity.rb +112 -0
- data/lib/sidekiq_unique_jobs/redis/hash.rb +56 -0
- data/lib/sidekiq_unique_jobs/redis/list.rb +32 -0
- data/lib/sidekiq_unique_jobs/redis/set.rb +32 -0
- data/lib/sidekiq_unique_jobs/redis/sorted_set.rb +86 -0
- data/lib/sidekiq_unique_jobs/redis/string.rb +49 -0
- data/lib/sidekiq_unique_jobs/rspec/matchers.rb +26 -0
- data/lib/sidekiq_unique_jobs/rspec/matchers/have_valid_sidekiq_options.rb +51 -0
- data/lib/sidekiq_unique_jobs/script.rb +15 -0
- data/lib/sidekiq_unique_jobs/script/caller.rb +125 -0
- data/lib/sidekiq_unique_jobs/server.rb +48 -0
- data/lib/sidekiq_unique_jobs/sidekiq_unique_ext.rb +92 -65
- data/lib/sidekiq_unique_jobs/sidekiq_unique_jobs.rb +204 -34
- data/lib/sidekiq_unique_jobs/sidekiq_worker_methods.rb +18 -9
- data/lib/sidekiq_unique_jobs/testing.rb +62 -21
- data/lib/sidekiq_unique_jobs/timer_task.rb +78 -0
- data/lib/sidekiq_unique_jobs/timing.rb +58 -0
- data/lib/sidekiq_unique_jobs/unlockable.rb +20 -4
- data/lib/sidekiq_unique_jobs/update_version.rb +25 -0
- data/lib/sidekiq_unique_jobs/upgrade_locks.rb +155 -0
- data/lib/sidekiq_unique_jobs/version.rb +3 -1
- data/lib/sidekiq_unique_jobs/version_check.rb +23 -4
- data/lib/sidekiq_unique_jobs/web.rb +57 -27
- data/lib/sidekiq_unique_jobs/web/helpers.rb +128 -13
- data/lib/sidekiq_unique_jobs/web/views/_paging.erb +4 -4
- data/lib/sidekiq_unique_jobs/web/views/changelogs.erb +54 -0
- data/lib/sidekiq_unique_jobs/web/views/lock.erb +108 -0
- data/lib/sidekiq_unique_jobs/web/views/locks.erb +54 -0
- data/lib/tasks/changelog.rake +5 -5
- metadata +122 -176
- data/lib/sidekiq_unique_jobs/client/middleware.rb +0 -56
- data/lib/sidekiq_unique_jobs/scripts.rb +0 -118
- data/lib/sidekiq_unique_jobs/server/middleware.rb +0 -46
- data/lib/sidekiq_unique_jobs/timeout.rb +0 -8
- data/lib/sidekiq_unique_jobs/timeout/calculator.rb +0 -63
- data/lib/sidekiq_unique_jobs/unique_args.rb +0 -149
- data/lib/sidekiq_unique_jobs/util.rb +0 -103
- data/lib/sidekiq_unique_jobs/web/views/unique_digest.erb +0 -28
- data/lib/sidekiq_unique_jobs/web/views/unique_digests.erb +0 -46
- data/redis/acquire_lock.lua +0 -21
- data/redis/convert_legacy_lock.lua +0 -13
- data/redis/delete.lua +0 -14
- data/redis/delete_by_digest.lua +0 -23
- data/redis/delete_job_by_digest.lua +0 -60
- data/redis/lock.lua +0 -62
- data/redis/release_stale_locks.lua +0 -90
- data/redis/unlock.lua +0 -35
data/README.md
CHANGED
@@ -1,45 +1,68 @@
|
|
1
|
-
# SidekiqUniqueJobs
|
1
|
+
# SidekiqUniqueJobs
|
2
|
+
|
3
|
+
[![Join the chat at https://gitter.im/mhenrixon/sidekiq-unique-jobs](https://badges.gitter.im/mhenrixon/sidekiq-unique-jobs.svg)](https://gitter.im/mhenrixon/sidekiq-unique-jobs?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) ![Build Status](https://github.com/mhenrixon/sidekiq-unique-jobs/actions/workflows/rspec.yml/badge.svg?branch=master) [![Code Climate](https://codeclimate.com/github/mhenrixon/sidekiq-unique-jobs.svg)](https://codeclimate.com/github/mhenrixon/sidekiq-unique-jobs) [![Test Coverage](https://codeclimate.com/github/mhenrixon/sidekiq-unique-jobs/badges/coverage.svg)](https://codeclimate.com/github/mhenrixon/sidekiq-unique-jobs/coverage)
|
2
4
|
|
3
5
|
<!-- MarkdownTOC -->
|
4
6
|
|
5
7
|
- [Introduction](#introduction)
|
6
|
-
- [
|
7
|
-
- [
|
8
|
-
- [
|
9
|
-
- [
|
10
|
-
- [Installation](#installation)
|
8
|
+
- [Usage](#usage)
|
9
|
+
- [Installation](#installation)
|
10
|
+
- [Add the middleware](#add-the-middleware)
|
11
|
+
- [Your first worker](#your-first-worker)
|
11
12
|
- [Support Me](#support-me)
|
13
|
+
- [Requirements](#requirements)
|
12
14
|
- [General Information](#general-information)
|
13
|
-
- [
|
14
|
-
- [
|
15
|
-
- [
|
16
|
-
- [
|
17
|
-
- [
|
15
|
+
- [Global Configuration](#global-configuration)
|
16
|
+
- [debug_lua](#debug_lua)
|
17
|
+
- [lock_timeout](#lock_timeout)
|
18
|
+
- [lock_ttl](#lock_ttl)
|
19
|
+
- [enabled](#enabled)
|
20
|
+
- [logger](#logger)
|
21
|
+
- [max_history](#max_history)
|
22
|
+
- [reaper](#reaper)
|
23
|
+
- [reaper_count](#reaper_count)
|
24
|
+
- [reaper_interval](#reaper_interval)
|
25
|
+
- [reaper_timeout](#reaper_timeout)
|
26
|
+
- [lock_prefix](#lock_prefix)
|
27
|
+
- [lock_info](#lock_info)
|
28
|
+
- [Worker Configuration](#worker-configuration)
|
29
|
+
- [lock_info](#lock_info-1)
|
30
|
+
- [lock_prefix](#lock_prefix-1)
|
31
|
+
- [lock_ttl](#lock_ttl-1)
|
32
|
+
- [lock_timeout](#lock_timeout-1)
|
33
|
+
- [unique_across_queues](#unique_across_queues)
|
34
|
+
- [unique_across_workers](#unique_across_workers)
|
18
35
|
- [Locks](#locks)
|
19
36
|
- [Until Executing](#until-executing)
|
20
37
|
- [Until Executed](#until-executed)
|
21
|
-
- [Until
|
22
|
-
- [
|
38
|
+
- [Until Expired](#until-expired)
|
39
|
+
- [Until And While Executing](#until-and-while-executing)
|
23
40
|
- [While Executing](#while-executing)
|
41
|
+
- [Custom Locks](#custom-locks)
|
24
42
|
- [Conflict Strategy](#conflict-strategy)
|
25
|
-
- [
|
26
|
-
- [
|
27
|
-
- [
|
28
|
-
- [
|
43
|
+
- [log](#log)
|
44
|
+
- [raise](#raise)
|
45
|
+
- [reject](#reject)
|
46
|
+
- [replace](#replace)
|
29
47
|
- [Reschedule](#reschedule)
|
30
|
-
- [
|
48
|
+
- [Custom Strategies](#custom-strategies)
|
49
|
+
- [Usage](#usage-1)
|
31
50
|
- [Finer Control over Uniqueness](#finer-control-over-uniqueness)
|
32
51
|
- [After Unlock Callback](#after-unlock-callback)
|
33
52
|
- [Logging](#logging)
|
34
53
|
- [Cleanup Dead Locks](#cleanup-dead-locks)
|
35
54
|
- [Other Sidekiq gems](#other-sidekiq-gems)
|
55
|
+
- [apartment-sidekiq](#apartment-sidekiq)
|
36
56
|
- [sidekiq-global_id](#sidekiq-global_id)
|
57
|
+
- [sidekiq-status](#sidekiq-status)
|
37
58
|
- [Debugging](#debugging)
|
38
59
|
- [Sidekiq Web](#sidekiq-web)
|
39
|
-
- [Show
|
40
|
-
- [Show
|
60
|
+
- [Show Locks](#show-locks)
|
61
|
+
- [Show Lock](#show-lock)
|
41
62
|
- [Communication](#communication)
|
42
63
|
- [Testing](#testing)
|
64
|
+
- [Unique Sidekiq Configuration](#unique-sidekiq-configuration)
|
65
|
+
- [Uniqueness](#uniqueness)
|
43
66
|
- [Contributing](#contributing)
|
44
67
|
- [Contributors](#contributors)
|
45
68
|
|
@@ -47,75 +70,282 @@
|
|
47
70
|
|
48
71
|
## Introduction
|
49
72
|
|
50
|
-
The
|
73
|
+
This gem adds unique constraints to the sidekiq queues. The uniqueness is achieved by acquiring locks for a hash of a queue name, a worker class, and job's arguments. By default, only one lock for a given hash can be acquired. What happens when a lock can't be acquired is governed by a chosen `on_conflict`strategy.
|
74
|
+
|
75
|
+
This is the documentation for the master branch. You can find the documentation for each release by navigating to its tag.
|
76
|
+
|
77
|
+
Here are links to some of the old versions
|
78
|
+
|
79
|
+
- [v6.0.25](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v6.0.25)
|
80
|
+
- [v5.0.10](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v5.0.10)
|
81
|
+
- [v4.0.18](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v4.0.18)
|
82
|
+
|
83
|
+
## Usage
|
84
|
+
|
85
|
+
### Installation
|
86
|
+
|
87
|
+
Add this line to your application's Gemfile:
|
88
|
+
|
89
|
+
```ruby
|
90
|
+
gem 'sidekiq-unique-jobs'
|
91
|
+
```
|
92
|
+
|
93
|
+
And then execute:
|
94
|
+
|
95
|
+
```bash
|
96
|
+
bundle
|
97
|
+
```
|
98
|
+
|
99
|
+
### Add the middleware
|
100
|
+
|
101
|
+
Before v7, the middleware was configured automatically. Since some people reported issues with other gems (see [Other Sidekiq Gems](#other-sidekiq-gems)) it was decided to give full control over to the user.
|
102
|
+
|
103
|
+
*NOTE* if you want to use the reaper you also need to configure the server middleware.
|
104
|
+
|
105
|
+
[A full example](https://github.com/mhenrixon/sidekiq-unique-jobs/blob/master/myapp/config/initializers/sidekiq.rb#L12)
|
106
|
+
|
107
|
+
```ruby
|
108
|
+
Sidekiq.configure_server do |config|
|
109
|
+
config.redis = { url: ENV["REDIS_URL"], driver: :hiredis }
|
110
|
+
|
111
|
+
config.client_middleware do |chain|
|
112
|
+
chain.add SidekiqUniqueJobs::Middleware::Client
|
113
|
+
end
|
114
|
+
|
115
|
+
config.server_middleware do |chain|
|
116
|
+
chain.add SidekiqUniqueJobs::Middleware::Server
|
117
|
+
end
|
118
|
+
|
119
|
+
SidekiqUniqueJobs::Server.configure(config)
|
120
|
+
end
|
51
121
|
|
52
|
-
|
122
|
+
Sidekiq.configure_client do |config|
|
123
|
+
config.redis = { url: ENV["REDIS_URL"], driver: :hiredis }
|
53
124
|
|
54
|
-
|
125
|
+
config.client_middleware do |chain|
|
126
|
+
chain.add SidekiqUniqueJobs::Middleware::Client
|
127
|
+
end
|
128
|
+
end
|
129
|
+
```
|
55
130
|
|
56
|
-
|
131
|
+
### Your first worker
|
132
|
+
|
133
|
+
```ruby
|
134
|
+
# frozen_string_literal: true
|
135
|
+
|
136
|
+
class UntilExecutedWorker
|
137
|
+
include Sidekiq::Worker
|
138
|
+
|
139
|
+
sidekiq_options queue: :special,
|
140
|
+
retry: false,
|
141
|
+
lock: :until_executed,
|
142
|
+
lock_info: true,
|
143
|
+
lock_timeout: 0,
|
144
|
+
lock_prefix: "special",
|
145
|
+
lock_ttl: 0,
|
146
|
+
lock_limit: 5
|
147
|
+
|
148
|
+
def perform
|
149
|
+
logger.info("cowboy")
|
150
|
+
sleep(1) # hardcore processing
|
151
|
+
logger.info("beebop")
|
152
|
+
end
|
153
|
+
end
|
154
|
+
|
155
|
+
```
|
156
|
+
|
157
|
+
You can read more about the worker configuration in [Worker Configuration](#worker-configuration) below.
|
158
|
+
|
159
|
+
## Support Me
|
57
160
|
|
58
|
-
|
59
|
-
- [v4.0.18][]
|
161
|
+
Want to show me some ❤️ for the hard work I do on this gem? You can use the following PayPal link: [https://paypal.me/mhenrixon1](https://paypal.me/mhenrixon1). Any amount is welcome and let me tell you it feels good to be appreciated. Even a dollar makes me super excited about all of this.
|
60
162
|
|
61
163
|
## Requirements
|
62
164
|
|
63
|
-
|
165
|
+
- Sidekiq `>= 5.0` (`>= 5.2` recommended)
|
166
|
+
- Ruby:
|
167
|
+
- MRI `>= 2.5` (`>= 2.6` recommended)
|
168
|
+
- JRuby `>= 9.0` (`>= 9.2` recommended)
|
169
|
+
- Truffleruby
|
170
|
+
- Redis Server `>= 3.2` (`>= 5.0` recommended)
|
171
|
+
- [ActiveJob officially not supported][48]
|
172
|
+
- [redis-namespace officially not supported][49]
|
64
173
|
|
65
|
-
|
174
|
+
See [Sidekiq requirements][24] for detailed requirements of Sidekiq itself (be sure to check the right sidekiq version).
|
66
175
|
|
67
|
-
|
176
|
+
## General Information
|
68
177
|
|
69
|
-
|
178
|
+
See [Interaction w/ Sidekiq](https://github.com/mhenrixon/sidekiq-unique-jobs/wiki/How-this-gem-interacts-with-Sidekiq) on how the gem interacts with Sidekiq.
|
70
179
|
|
71
|
-
|
180
|
+
See [Locking & Unlocking](https://github.com/mhenrixon/sidekiq-unique-jobs/wiki/Locking-&-Unlocking) for an overview of the differences on when the various lock types are locked and unlocked.
|
72
181
|
|
73
|
-
|
182
|
+
## Global Configuration
|
74
183
|
|
75
|
-
|
184
|
+
The gem supports a few different configuration options that might be of interest if you run into some weird issues.
|
76
185
|
|
77
|
-
|
186
|
+
Configure SidekiqUniqueJobs in an initializer or the sidekiq initializer on application startup.
|
78
187
|
|
188
|
+
```ruby
|
189
|
+
SidekiqUniqueJobs.configure do |config|
|
190
|
+
config.logger = Sidekiq.logger # default, change at your own discretion
|
191
|
+
config.debug_lua = false # Turn on when debugging
|
192
|
+
config.lock_info = false # Turn on when debugging
|
193
|
+
config.lock_ttl = 600 # Expire locks after 10 minutes
|
194
|
+
config.lock_timeout = nil # turn off lock timeout
|
195
|
+
config.max_history = 0 # Turn on when debugging
|
196
|
+
config.reaper = :ruby # :ruby, :lua or :none/nil
|
197
|
+
config.reaper_count = 1000 # Stop reaping after this many keys
|
198
|
+
config.reaper_interval = 600 # Reap orphans every 10 minutes
|
199
|
+
config.reaper_timeout = 150 # Timeout reaper after 2.5 minutes
|
200
|
+
end
|
79
201
|
```
|
80
|
-
|
202
|
+
|
203
|
+
### debug_lua
|
204
|
+
|
205
|
+
```ruby
|
206
|
+
SidekiqUniqueJobs.config.debug_lua #=> false
|
81
207
|
```
|
82
208
|
|
83
|
-
|
209
|
+
Turning on debug_lua will allow the lua scripts to output debug information about what the lua scripts do. It will log all redis commands that are executed and also some helpful messages about what is going on inside the lua script.
|
84
210
|
|
211
|
+
### lock_timeout
|
212
|
+
|
213
|
+
```ruby
|
214
|
+
SidekiqUniqueJobs.config.lock_timeout #=> 0
|
85
215
|
```
|
86
|
-
|
216
|
+
|
217
|
+
Set a global lock_timeout to use for all jobs that don't otherwise specify a lock_timeout.
|
218
|
+
|
219
|
+
Lock timeout decides how long to wait for acquiring the lock. A value of nil means to wait indefinitely for a lock resource to become available.
|
220
|
+
|
221
|
+
### lock_ttl
|
222
|
+
|
223
|
+
```ruby
|
224
|
+
SidekiqUniqueJobs.config.lock_ttl #=> nil
|
87
225
|
```
|
88
226
|
|
89
|
-
|
227
|
+
Set a global lock_ttl to use for all jobs that don't otherwise specify a lock_ttl.
|
228
|
+
|
229
|
+
Lock TTL decides how long to wait after the job has been successfully processed before making it possible to reuse that lock.
|
90
230
|
|
231
|
+
### enabled
|
232
|
+
|
233
|
+
```ruby
|
234
|
+
SidekiqUniqueJobs.config.enabled #=> true
|
91
235
|
```
|
92
|
-
|
236
|
+
|
237
|
+
Globally turn the locking mechanism on or off.
|
238
|
+
|
239
|
+
### logger
|
240
|
+
|
241
|
+
```ruby
|
242
|
+
SidekiqUniqueJobs.config.logger #=> #<Sidekiq::Logger:0x00007fdc1f96d180>
|
93
243
|
```
|
94
244
|
|
95
|
-
|
245
|
+
By default this gem piggybacks on the Sidekiq logger. It is not recommended to change this as the gem uses some features in the Sidekiq logger and you might run into problems. If you need a different logger and you do run into problems then get in touch and we'll see what we can do about it.
|
96
246
|
|
97
|
-
|
247
|
+
### max_history
|
98
248
|
|
99
|
-
|
249
|
+
```ruby
|
250
|
+
SidekiqUniqueJobs.config.max_history #=> 1_000
|
251
|
+
```
|
100
252
|
|
101
|
-
|
253
|
+
The max_history setting can be used to tweak the number of changelogs generated. It can also be completely turned off if performance suffers or if you are just not interested in using the changelog.
|
102
254
|
|
103
|
-
|
255
|
+
This is a log that can be accessed by a lock to see what happened for that lock. Any items after the configured `max_history` will be automatically deleted as new items are added.
|
256
|
+
|
257
|
+
### reaper
|
258
|
+
|
259
|
+
```ruby
|
260
|
+
SidekiqUniqueJobs.config.reaper #=> :ruby
|
261
|
+
```
|
262
|
+
|
263
|
+
If using the orphans cleanup process it is critical to be aware of the following. The `:ruby` job is much slower but the `:lua` job locks redis while executing. While doing intense processing it is best to avoid locking redis with a lua script. There for the batch size (controlled by the `reaper_count` setting) needs to be reduced.
|
264
|
+
|
265
|
+
In my benchmarks deleting 1000 orphaned locks with lua performs around 65% faster than deleting 1000 keys in ruby.
|
266
|
+
|
267
|
+
On the other hand if I increase it to 10 000 orphaned locks per cleanup (`reaper_count: 10_0000`) then redis starts throwing:
|
268
|
+
|
269
|
+
> BUSY Redis is busy running a script. You can only call SCRIPT KILL or SHUTDOWN NOSAVE. (Redis::CommandError)
|
270
|
+
|
271
|
+
If you want to disable the reaper set it to `:none`, `nil` or `false`. Actually, any value that isn't `:ruby` or `:lua` will disable the reaping.
|
272
|
+
|
273
|
+
```ruby
|
274
|
+
SidekiqUniqueJobs.config.reaper = :none
|
275
|
+
SidekiqUniqueJobs.config.reaper = nil
|
276
|
+
SidekiqUniqueJobs.config.reaper = false
|
277
|
+
```
|
278
|
+
|
279
|
+
### reaper_count
|
280
|
+
|
281
|
+
```ruby
|
282
|
+
SidekiqUniqueJobs.config.reaper_count #=> 1_000
|
283
|
+
```
|
104
284
|
|
105
|
-
|
285
|
+
The reaper_count setting configures how many orphans at a time will be cleaned up by the orphan cleanup job. This might have to be tweaked depending on which orphan job is running.
|
106
286
|
|
107
|
-
###
|
287
|
+
### reaper_interval
|
108
288
|
|
109
|
-
|
289
|
+
```ruby
|
290
|
+
SidekiqUniqueJobs.config.reaper_interval #=> 600
|
291
|
+
```
|
292
|
+
|
293
|
+
The number of seconds between reaping.
|
110
294
|
|
111
|
-
|
295
|
+
### reaper_timeout
|
112
296
|
|
113
297
|
```ruby
|
114
|
-
|
115
|
-
sidekiq_options lock_expiration: 20.days.to_i # expire this lock in 20 days
|
298
|
+
SidekiqUniqueJobs.config.reaper_timeout #=> 10
|
116
299
|
```
|
117
300
|
|
118
|
-
|
301
|
+
The number of seconds to wait for the reaper to finish before raising a TimeoutError. This is done to ensure that the next time we reap isn't getting stuck due to the previous process already running.
|
302
|
+
|
303
|
+
### lock_prefix
|
304
|
+
|
305
|
+
```ruby
|
306
|
+
SidekiqUniqueJobs.config.lock_prefix #=> "uniquejobs"
|
307
|
+
```
|
308
|
+
|
309
|
+
Use if you want a different key prefix for the keys in redis.
|
310
|
+
|
311
|
+
### lock_info
|
312
|
+
|
313
|
+
```ruby
|
314
|
+
SidekiqUniqueJobs.config.lock_info #=> false
|
315
|
+
```
|
316
|
+
|
317
|
+
Using lock info will create an additional key for the lock with a json object containing information about the lock. This will be presented in the web interface and might help track down why some jobs are getting stuck.
|
318
|
+
|
319
|
+
## Worker Configuration
|
320
|
+
|
321
|
+
### lock_info
|
322
|
+
|
323
|
+
Lock info gathers information about a specific lock. It collects things like which `lock_args` where used to compute the `lock_digest` that is used for maintaining uniqueness.
|
324
|
+
|
325
|
+
```ruby
|
326
|
+
sidekiq_options lock_info: false # this is the default, set to true to turn on
|
327
|
+
```
|
328
|
+
|
329
|
+
### lock_prefix
|
330
|
+
|
331
|
+
Use if you want a different key prefix for the keys in redis.
|
332
|
+
|
333
|
+
```ruby
|
334
|
+
sidekiq_options lock_prefix: "uniquejobs" # this is the default value
|
335
|
+
```
|
336
|
+
|
337
|
+
### lock_ttl
|
338
|
+
|
339
|
+
Lock TTL decides how long to wait after the job has been successfully processed before making it possible to reuse that lock.
|
340
|
+
|
341
|
+
Starting from `v7` the expiration will take place when the job is pushed to the queue.
|
342
|
+
|
343
|
+
```ruby
|
344
|
+
sidekiq_options lock_ttl: nil # default - don't expire keys
|
345
|
+
sidekiq_options lock_ttl: 20.days.to_i # expire this lock in 20 days
|
346
|
+
```
|
347
|
+
|
348
|
+
### lock_timeout
|
119
349
|
|
120
350
|
This is the timeout (how long to wait) when creating the lock. By default we don't use a timeout so we won't wait for the lock to be created. If you want it is possible to set this like below.
|
121
351
|
|
@@ -125,10 +355,12 @@ sidekiq_options lock_timeout: 5 # wait 5 seconds
|
|
125
355
|
sidekiq_options lock_timeout: nil # lock indefinitely, this process won't continue until it gets a lock. VERY DANGEROUS!!
|
126
356
|
```
|
127
357
|
|
128
|
-
###
|
358
|
+
### unique_across_queues
|
129
359
|
|
130
360
|
This configuration option is slightly misleading. It doesn't disregard the queue on other jobs. Just on itself, this means that a worker that might schedule jobs into multiple queues will be able to have uniqueness enforced on all queues it is pushed to.
|
131
361
|
|
362
|
+
This is mainly intended for `Worker.set(queue: :another).perform_async`.
|
363
|
+
|
132
364
|
```ruby
|
133
365
|
class Worker
|
134
366
|
include Sidekiq::Worker
|
@@ -141,9 +373,9 @@ end
|
|
141
373
|
|
142
374
|
Now if you push override the queue with `Worker.set(queue: 'another').perform_async(1)` it will still be considered unique when compared to `Worker.perform_async(1)` (that was actually pushed to the queue `default`).
|
143
375
|
|
144
|
-
###
|
376
|
+
### unique_across_workers
|
145
377
|
|
146
|
-
This configuration option is slightly misleading. It doesn't disregard the worker class on other jobs. Just on itself, this means
|
378
|
+
This configuration option is slightly misleading. It doesn't disregard the worker class on other jobs. Just on itself, this means that the worker class won't be used for generating the unique digest. The only way this option really makes sense is when you want to have uniqueness between two different worker classes.
|
147
379
|
|
148
380
|
```ruby
|
149
381
|
class WorkerOne
|
@@ -174,53 +406,55 @@ WorkerTwo.perform_async(1)
|
|
174
406
|
|
175
407
|
### Until Executing
|
176
408
|
|
409
|
+
```ruby
|
410
|
+
sidekiq_options lock: :until_executing
|
411
|
+
```
|
412
|
+
|
177
413
|
Locks from when the client pushes the job to the queue. Will be unlocked before the server starts processing the job.
|
178
414
|
|
179
415
|
**NOTE** this is probably not so good for jobs that shouldn't be running simultaneously (aka slow jobs).
|
180
416
|
|
181
|
-
|
182
|
-
sidekiq_options lock: :until_executing
|
183
|
-
```
|
417
|
+
The reason this type of lock exists is to fix the following problem: [sidekiq/issues/3471](https://github.com/mperham/sidekiq/issues/3471#issuecomment-300866335)
|
184
418
|
|
185
419
|
### Until Executed
|
186
420
|
|
187
|
-
Locks from when the client pushes the job to the queue. Will be unlocked when the server has successfully processed the job.
|
188
|
-
|
189
421
|
```ruby
|
190
422
|
sidekiq_options lock: :until_executed
|
191
423
|
```
|
192
424
|
|
193
|
-
|
425
|
+
Locks from when the client pushes the job to the queue. Will be unlocked when the server has successfully processed the job.
|
194
426
|
|
195
|
-
|
427
|
+
### Until Expired
|
196
428
|
|
197
429
|
```ruby
|
198
430
|
sidekiq_options lock: :until_expired
|
199
431
|
```
|
200
432
|
|
201
|
-
|
433
|
+
Locks from when the client pushes the job to the queue. Will be unlocked when the specified timeout has been reached.
|
202
434
|
|
203
|
-
|
435
|
+
### Until And While Executing
|
204
436
|
|
205
437
|
```ruby
|
206
438
|
sidekiq_options lock: :until_and_while_executing
|
207
439
|
```
|
208
440
|
|
441
|
+
Locks when the client pushes the job to the queue. The queue will be unlocked when the server starts processing the job. The server then goes on to creating a runtime lock for the job to prevent simultaneous jobs from being executed. As soon as the server starts processing a job, the client can push the same job to the queue.
|
442
|
+
|
209
443
|
### While Executing
|
210
444
|
|
445
|
+
```ruby
|
446
|
+
sidekiq_options lock: :while_executing, lock_timeout: 10
|
447
|
+
```
|
448
|
+
|
211
449
|
With this lock type it is possible to put any number of these jobs on the queue, but as the server pops the job from the queue it will create a lock and then wait until other locks are done processing. It _looks_ like multiple jobs are running at the same time but in fact the second job will only be waiting for the first job to finish.
|
212
450
|
|
213
451
|
**NOTE** Unless this job is configured with a `lock_timeout: nil` or `lock_timeout: > 0` then all jobs that are attempted to be executed will just be dropped without waiting.
|
214
452
|
|
215
|
-
|
216
|
-
sidekiq_options lock: :while_executing, lock_timeout: nil
|
217
|
-
```
|
218
|
-
|
219
|
-
There is an example of this to try it out in the `rails_example` application. Run `foreman start` in the root of the directory and open the url: `localhost:5000/work/duplicate_while_executing`.
|
453
|
+
There is an example of this to try it out in the `myapp` application. Run `foreman start` in the root of the directory and open the url: `localhost:5000/work/duplicate_while_executing`.
|
220
454
|
|
221
455
|
In the console you should see something like:
|
222
456
|
|
223
|
-
```
|
457
|
+
```bash
|
224
458
|
0:32:24 worker.1 | 2017-04-23T08:32:24.955Z 84404 TID-ougq4thko WhileExecutingWorker JID-400ec51c9523f41cd4a35058 INFO: start
|
225
459
|
10:32:24 worker.1 | 2017-04-23T08:32:24.956Z 84404 TID-ougq8csew WhileExecutingWorker JID-8d6d9168368eedaed7f75763 INFO: start
|
226
460
|
10:32:24 worker.1 | 2017-04-23T08:32:24.957Z 84404 TID-ougq8crt8 WhileExecutingWorker JID-affcd079094c9b26e8b9ba60 INFO: start
|
@@ -235,31 +469,81 @@ In the console you should see something like:
|
|
235
469
|
10:33:04 worker.1 | 2017-04-23T08:33:04.973Z 84404 TID-ougq8cs8s WhileExecutingWorker JID-9e197460c067b22eb1b5d07f INFO: done: 40.014 sec
|
236
470
|
```
|
237
471
|
|
472
|
+
### Custom Locks
|
473
|
+
|
474
|
+
You may need to define some custom lock. You can define it in one project folder:
|
475
|
+
|
476
|
+
```ruby
|
477
|
+
# lib/locks/my_custom_lock.rb
|
478
|
+
module Locks
|
479
|
+
class MyCustomLock < SidekiqUniqueJobs::Lock::BaseLock
|
480
|
+
def execute
|
481
|
+
# Do something ...
|
482
|
+
end
|
483
|
+
end
|
484
|
+
end
|
485
|
+
```
|
486
|
+
|
487
|
+
You can refer on all the locks defined in `lib/sidekiq_unique_jobs/lock/*.rb`.
|
488
|
+
|
489
|
+
In order to make it available, you should call in your project startup:
|
490
|
+
|
491
|
+
(For rails application config/initializers/sidekiq_unique_jobs.rb or other projects, wherever you prefer)
|
492
|
+
|
493
|
+
```ruby
|
494
|
+
SidekiqUniqueJobs.configure do |config|
|
495
|
+
config.add_lock :my_custom_lock, Locks::MyCustomLock
|
496
|
+
end
|
497
|
+
```
|
498
|
+
|
499
|
+
And then you can use it in the jobs definition:
|
500
|
+
|
501
|
+
`sidekiq_options lock: :my_custom_lock, on_conflict: :log`
|
502
|
+
|
503
|
+
Please not that if you try to override a default lock, an `ArgumentError` will be raised.
|
504
|
+
|
238
505
|
## Conflict Strategy
|
239
506
|
|
240
507
|
Decides how we handle conflict. We can either reject the job to the dead queue or reschedule it. Both are useful for jobs that absolutely need to run and have been configured to use the lock `WhileExecuting` that is used only by the sidekiq server process.
|
241
508
|
|
242
|
-
The last one is log which can be be used with the lock `UntilExecuted` and `UntilExpired`. Now we write a log entry saying the job could not be pushed because it is a duplicate of another job with the same arguments
|
509
|
+
The last one is log which can be be used with the lock `UntilExecuted` and `UntilExpired`. Now we write a log entry saying the job could not be pushed because it is a duplicate of another job with the same arguments.
|
243
510
|
|
244
|
-
|
511
|
+
It is possible for locks to have different conflict strategy for the client and server. This is useful for `:until_and_while_executing`.
|
512
|
+
|
513
|
+
```ruby
|
514
|
+
sidekiq_options lock: :until_and_while_executing,
|
515
|
+
on_conflict: { client: :log, server: :reject }
|
516
|
+
```
|
517
|
+
|
518
|
+
### log
|
519
|
+
|
520
|
+
```ruby
|
521
|
+
sidekiq_options on_conflict: :log
|
522
|
+
```
|
245
523
|
|
246
524
|
This strategy is intended to be used with `UntilExecuted` and `UntilExpired`. It will log a line about that this is job is a duplicate of another.
|
247
525
|
|
248
|
-
|
526
|
+
### raise
|
249
527
|
|
250
|
-
|
528
|
+
```ruby
|
529
|
+
sidekiq_options on_conflict: :raise
|
530
|
+
```
|
251
531
|
|
252
532
|
This strategy is intended to be used with `WhileExecuting`. Basically it will allow us to let the server process crash with a specific error message and be retried without messing up the Sidekiq stats.
|
253
533
|
|
254
|
-
|
534
|
+
### reject
|
255
535
|
|
256
|
-
|
536
|
+
```ruby
|
537
|
+
sidekiq_options on_conflict: :reject
|
538
|
+
```
|
257
539
|
|
258
540
|
This strategy is intended to be used with `WhileExecuting` and will push the job to the dead queue on conflict.
|
259
541
|
|
260
|
-
|
542
|
+
### replace
|
261
543
|
|
262
|
-
|
544
|
+
```ruby
|
545
|
+
sidekiq_options on_conflict: :replace
|
546
|
+
```
|
263
547
|
|
264
548
|
This strategy is intended to be used with client locks like `UntilExecuted`.
|
265
549
|
It will delete any existing job for these arguments from retry, schedule and
|
@@ -268,13 +552,48 @@ queue and retry the lock again.
|
|
268
552
|
This is slightly dangerous and should probably only be used for jobs that are
|
269
553
|
always scheduled in the future. Currently only attempting to retry one time.
|
270
554
|
|
271
|
-
`sidekiq_options lock: :until_executed, on_conflict: :replace`
|
272
|
-
|
273
555
|
### Reschedule
|
274
556
|
|
557
|
+
```ruby
|
558
|
+
sidekiq_options on_conflict: :reschedule
|
559
|
+
```
|
560
|
+
|
275
561
|
This strategy is intended to be used with `WhileExecuting` and will delay the job to be tried again in 5 seconds. This will mess up the sidekiq stats but will prevent exceptions from being logged and confuse your sysadmins.
|
276
562
|
|
277
|
-
|
563
|
+
### Custom Strategies
|
564
|
+
|
565
|
+
You may need to define some custom strategy. You can define it in one project folder:
|
566
|
+
|
567
|
+
```ruby
|
568
|
+
# lib/strategies/my_custom_strategy.rb
|
569
|
+
module Strategies
|
570
|
+
class MyCustomStrategy < OnConflict::Strategy
|
571
|
+
def call
|
572
|
+
# Do something ...
|
573
|
+
end
|
574
|
+
end
|
575
|
+
end
|
576
|
+
```
|
577
|
+
|
578
|
+
You can refer to all the strategies defined in `lib/sidekiq_unique_jobs/on_conflict`.
|
579
|
+
|
580
|
+
In order to make it available, you should call in your project startup:
|
581
|
+
|
582
|
+
(For rails application config/initializers/sidekiq_unique_jobs.rb for other projects, wherever you prefer)
|
583
|
+
|
584
|
+
```ruby
|
585
|
+
SidekiqUniqueJobs.configure do |config|
|
586
|
+
config.add_strategy :my_custom_strategy, Strategies::MyCustomStrategy
|
587
|
+
end
|
588
|
+
```
|
589
|
+
|
590
|
+
And then you can use it in the jobs definition:
|
591
|
+
|
592
|
+
```ruby
|
593
|
+
sidekiq_options lock: :while_executing, on_conflict: :my_custom_strategy
|
594
|
+
```
|
595
|
+
|
596
|
+
Please not that if you try to override a default lock, an `ArgumentError` will be raised.
|
278
597
|
|
279
598
|
## Usage
|
280
599
|
|
@@ -288,9 +607,11 @@ Requiring the gem in your gemfile should be sufficient to enable unique jobs.
|
|
288
607
|
|
289
608
|
### Finer Control over Uniqueness
|
290
609
|
|
291
|
-
Sometimes it is desired to have a finer control over which arguments are used in determining uniqueness of the job, and others may be _transient_. For this use-case, you need to define either a `
|
610
|
+
Sometimes it is desired to have a finer control over which arguments are used in determining uniqueness of the job, and others may be _transient_. For this use-case, you need to define either a `lock_args` method, or a ruby proc.
|
292
611
|
|
293
|
-
The
|
612
|
+
*NOTE:* The lock_args method need to return an array of values to use for uniqueness check.
|
613
|
+
|
614
|
+
*NOTE:* The arguments passed to the proc or the method is always an array. If your method takes a single array as argument the value of args will be `[[...]]`.
|
294
615
|
|
295
616
|
The method or the proc can return a modified version of args without the transient arguments included, as shown below:
|
296
617
|
|
@@ -298,9 +619,9 @@ The method or the proc can return a modified version of args without the transie
|
|
298
619
|
class UniqueJobWithFilterMethod
|
299
620
|
include Sidekiq::Worker
|
300
621
|
sidekiq_options lock: :until_and_while_executing,
|
301
|
-
|
622
|
+
lock_args_method: :lock_args # this is default and will be used if such a method is defined
|
302
623
|
|
303
|
-
def self.
|
624
|
+
def self.lock_args(args)
|
304
625
|
[ args[0], args[2][:type] ]
|
305
626
|
end
|
306
627
|
|
@@ -311,21 +632,21 @@ end
|
|
311
632
|
class UniqueJobWithFilterProc
|
312
633
|
include Sidekiq::Worker
|
313
634
|
sidekiq_options lock: :until_executed,
|
314
|
-
|
635
|
+
lock_args_method: ->(args) { [ args.first ] }
|
315
636
|
|
316
637
|
...
|
317
638
|
|
318
639
|
end
|
319
640
|
```
|
320
641
|
|
321
|
-
It is
|
642
|
+
It is possible to ensure different types of unique args based on context. I can't vouch for the below example but see [#203](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/203) for the discussion.
|
322
643
|
|
323
644
|
```ruby
|
324
645
|
class UniqueJobWithFilterMethod
|
325
646
|
include Sidekiq::Worker
|
326
|
-
sidekiq_options lock: :until_and_while_executing,
|
647
|
+
sidekiq_options lock: :until_and_while_executing, lock_args_method: :lock_args
|
327
648
|
|
328
|
-
def self.
|
649
|
+
def self.lock_args(args)
|
329
650
|
if Sidekiq::ProcessSet.new.size > 1
|
330
651
|
# sidekiq runtime; uniqueness for the object (first arg)
|
331
652
|
args.first
|
@@ -341,14 +662,20 @@ end
|
|
341
662
|
|
342
663
|
If you need to perform any additional work after the lock has been released you can provide an `#after_unlock` instance method. The method will be called when the lock has been unlocked. Most times this means after yield but there are two exceptions to that.
|
343
664
|
|
344
|
-
**Exception 1:** UntilExecuting unlocks and
|
665
|
+
**Exception 1:** UntilExecuting unlocks and uses callback before yielding.
|
345
666
|
**Exception 2:** UntilExpired expires eventually, no after_unlock hook is called.
|
346
667
|
|
668
|
+
**NOTE:** _It is also possible to write this code as a class method._
|
669
|
+
|
347
670
|
```ruby
|
348
671
|
class UniqueJobWithFilterMethod
|
349
672
|
include Sidekiq::Worker
|
350
673
|
sidekiq_options lock: :while_executing,
|
351
674
|
|
675
|
+
def self.after_unlock
|
676
|
+
# block has yielded and lock is released
|
677
|
+
end
|
678
|
+
|
352
679
|
def after_unlock
|
353
680
|
# block has yielded and lock is released
|
354
681
|
end
|
@@ -364,7 +691,7 @@ To see logging in sidekiq when duplicate payload has been filtered out you can e
|
|
364
691
|
class UniqueJobWithFilterMethod
|
365
692
|
include Sidekiq::Worker
|
366
693
|
sidekiq_options lock: :while_executing,
|
367
|
-
|
694
|
+
log_duplicate: true
|
368
695
|
|
369
696
|
...
|
370
697
|
|
@@ -378,40 +705,95 @@ For sidekiq versions before 5.1 a `sidekiq_retries_exhausted` block is required
|
|
378
705
|
```ruby
|
379
706
|
class MyWorker
|
380
707
|
sidekiq_retries_exhausted do |msg, _ex|
|
381
|
-
|
708
|
+
digest = msg['lock_digest']
|
709
|
+
SidekiqUniqueJobs::Digests.new.delete_by_digest(digest) if digest
|
382
710
|
end
|
383
711
|
end
|
384
712
|
```
|
385
713
|
|
386
|
-
Starting in v5.1, Sidekiq can also fire a global callback when a job dies:
|
714
|
+
Starting in v5.1, Sidekiq can also fire a global callback when a job dies: In version 7, this is handled automatically for you. You don't need to add a death handler, if you configure v7 like in [Add the middleware](#add-the-middleware) you don't have to worry about the below.
|
387
715
|
|
388
716
|
```ruby
|
389
|
-
# this goes in your initializer
|
390
717
|
Sidekiq.configure_server do |config|
|
391
718
|
config.death_handlers << ->(job, _ex) do
|
392
|
-
|
719
|
+
digest = job['lock_digest']
|
720
|
+
SidekiqUniqueJobs::Digests.new.delete_by_digest(digest) if digest
|
393
721
|
end
|
394
722
|
end
|
395
723
|
```
|
396
724
|
|
397
725
|
### Other Sidekiq gems
|
398
726
|
|
727
|
+
#### apartment-sidekiq
|
728
|
+
|
729
|
+
It was reported in [#536](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/536) that the order of the Sidekiq middleware needs to be as follows.
|
730
|
+
|
731
|
+
```ruby
|
732
|
+
Sidekiq.client_middleware do |chain|
|
733
|
+
chain.add Apartment::Sidekiq::Middleware::Client
|
734
|
+
chain.add SidekiqUniqueJobs::Middleware::Client
|
735
|
+
end
|
736
|
+
|
737
|
+
Sidekiq.server_middleware do |chain|
|
738
|
+
chain.add Apartment::Sidekiq::Middleware::Server
|
739
|
+
chain.add SidekiqUniqueJobs::Middleware::Server
|
740
|
+
end
|
741
|
+
```
|
742
|
+
|
743
|
+
The reason being that this gem needs to be configured AFTER the apartment gem or the apartment will not be able to be considered for uniqueness
|
744
|
+
|
399
745
|
#### sidekiq-global_id
|
400
746
|
|
401
747
|
It was reported in [#235](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/235) that the order of the Sidekiq middleware needs to be as follows.
|
402
748
|
|
749
|
+
For a working setup check the following [file](https://github.com/mhenrixon/sidekiq-unique-jobs/blob/master/myapp/config/sidekiq.rb#L12).
|
750
|
+
|
403
751
|
```ruby
|
404
752
|
Sidekiq.client_middleware do |chain|
|
405
753
|
chain.add Sidekiq::GlobalId::ClientMiddleware
|
406
|
-
chain.add SidekiqUniqueJobs::Client
|
754
|
+
chain.add SidekiqUniqueJobs::Middleware::Client
|
407
755
|
end
|
408
756
|
|
409
757
|
Sidekiq.server_middleware do |chain|
|
410
|
-
chain.add SidekiqUniqueJobs::Server::Middleware
|
411
758
|
chain.add Sidekiq::GlobalId::ServerMiddleware
|
759
|
+
chain.add SidekiqUniqueJobs::Middleware::Server
|
412
760
|
end
|
413
761
|
```
|
414
762
|
|
763
|
+
The reason for this is that the global id needs to be set before the unique jobs middleware runs. Otherwise that won't be available for uniqueness.
|
764
|
+
|
765
|
+
#### sidekiq-status
|
766
|
+
|
767
|
+
It was reported in [#564](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/564) that the order of the middleware needs to be as follows.
|
768
|
+
|
769
|
+
```ruby
|
770
|
+
# Thanks to @ArturT for the correction
|
771
|
+
|
772
|
+
Sidekiq.configure_server do |config|
|
773
|
+
config.client_middleware do |chain|
|
774
|
+
chain.add SidekiqUniqueJobs::Middleware::Client
|
775
|
+
chain.add Sidekiq::Status::ClientMiddleware, expiration: 30.minutes
|
776
|
+
end
|
777
|
+
|
778
|
+
config.server_middleware do |chain|
|
779
|
+
chain.add Sidekiq::Status::ServerMiddleware, expiration: 30.minutes
|
780
|
+
chain.add SidekiqUniqueJobs::Middleware::Server
|
781
|
+
end
|
782
|
+
|
783
|
+
SidekiqUniqueJobs::Server.configure(config)
|
784
|
+
end
|
785
|
+
|
786
|
+
|
787
|
+
Sidekiq.configure_client do |config|
|
788
|
+
config.client_middleware do |chain|
|
789
|
+
chain.add SidekiqUniqueJobs::Middleware::Client
|
790
|
+
chain.add Sidekiq::Status::ClientMiddleware, expiration: 30.minutes
|
791
|
+
end
|
792
|
+
end
|
793
|
+
```
|
794
|
+
|
795
|
+
The reason for this is that if a job is duplicated it shouldn't end up with the status middleware at all. Status is just a monitor so to prevent clashes, leftovers and ensure cleanup. The status middleware should run after uniqueness on client and before on server. This will lead to less surprises.
|
796
|
+
|
415
797
|
## Debugging
|
416
798
|
|
417
799
|
There are several ways of removing keys that are stuck. The prefered way is by using the unique extension to `Sidekiq::Web`. The old console and command line versions still work but might be deprecated in the future. It is better to search for the digest itself and delete the keys matching that digest.
|
@@ -421,7 +803,7 @@ There are several ways of removing keys that are stuck. The prefered way is by u
|
|
421
803
|
To use the web extension you need to require it in your routes.
|
422
804
|
|
423
805
|
```ruby
|
424
|
-
#
|
806
|
+
#app/config/routes.rb
|
425
807
|
require 'sidekiq_unique_jobs/web'
|
426
808
|
mount Sidekiq::Web, at: '/sidekiq'
|
427
809
|
```
|
@@ -431,13 +813,13 @@ already does this.
|
|
431
813
|
|
432
814
|
To filter/search for keys we can use the wildcard `*`. If we have a unique digest `'uniquejobs:9e9b5ce5d423d3ea470977004b50ff84` we can search for it by enter `*ff84` and it should return all digests that end with `ff84`.
|
433
815
|
|
434
|
-
#### Show
|
816
|
+
#### Show Locks
|
435
817
|
|
436
|
-
![
|
818
|
+
![Locks](assets/unique_digests_1.png)
|
437
819
|
|
438
|
-
#### Show
|
820
|
+
#### Show Lock
|
439
821
|
|
440
|
-
![
|
822
|
+
![Lock](assets/unique_digests_2.png)
|
441
823
|
|
442
824
|
## Communication
|
443
825
|
|
@@ -445,9 +827,45 @@ There is a [![Join the chat at https://gitter.im/mhenrixon/sidekiq-unique-jobs](
|
|
445
827
|
|
446
828
|
## Testing
|
447
829
|
|
448
|
-
|
830
|
+
### Unique Sidekiq Configuration
|
831
|
+
|
832
|
+
Since v7 it is possible to perform some simple validation against your workers sidekiq_options. What it does is scan for some issues that are known to cause problems in production.
|
833
|
+
|
834
|
+
Let's take a _bad_ worker:
|
835
|
+
|
836
|
+
```ruby
|
837
|
+
#app/workers/bad_worker.rb
|
838
|
+
class BadWorker
|
839
|
+
sidekiq_options lock: :while_executing, on_conflict: :replace
|
840
|
+
end
|
841
|
+
|
842
|
+
#spec/workers/bad_worker_spec.rb
|
843
|
+
|
844
|
+
require "sidekiq_unique_jobs/testing"
|
845
|
+
#OR
|
846
|
+
require "sidekiq_unique_jobs/rspec/matchers"
|
847
|
+
|
848
|
+
RSpec.describe BadWorker do
|
849
|
+
specify { expect(described_class).to have_valid_sidekiq_options }
|
850
|
+
end
|
851
|
+
```
|
852
|
+
|
853
|
+
This gives us a helpful error message for a wrongly configured worker:
|
449
854
|
|
450
|
-
|
855
|
+
```bash
|
856
|
+
Expected BadWorker to have valid sidekiq options but found the following problems:
|
857
|
+
on_server_conflict: :replace is incompatible with the server process
|
858
|
+
```
|
859
|
+
|
860
|
+
If you are not using RSpec (a lot of people prefer minitest or test unit) you can do something like:
|
861
|
+
|
862
|
+
```ruby
|
863
|
+
assert SidekiqUniqueJobs.validate_worker!(BadWorker.get_sidekiq_options)
|
864
|
+
```
|
865
|
+
|
866
|
+
### Uniqueness
|
867
|
+
|
868
|
+
This has been probably the most confusing part of this gem. People get really confused with how unreliable the unique jobs have been. I there for decided to do what Mike is doing for sidekiq enterprise. Read the section about unique jobs: [Enterprise unique jobs][]
|
451
869
|
|
452
870
|
```ruby
|
453
871
|
SidekiqUniqueJobs.configure do |config|
|
@@ -458,6 +876,8 @@ end
|
|
458
876
|
If you truly wanted to test the sidekiq client push you could do something like below. Note that it will only work for the jobs that lock when the client pushes the job to redis (UntilExecuted, UntilAndWhileExecuting and UntilExpired).
|
459
877
|
|
460
878
|
```ruby
|
879
|
+
require "sidekiq_unique_jobs/testing"
|
880
|
+
|
461
881
|
RSpec.describe Workers::CoolOne do
|
462
882
|
before do
|
463
883
|
SidekiqUniqueJobs.config.enabled = false
|
@@ -485,13 +905,13 @@ RSpec.describe Workers::CoolOne do
|
|
485
905
|
end
|
486
906
|
```
|
487
907
|
|
488
|
-
|
908
|
+
It is recommended to leave the uniqueness testing to the gem maintainers. If you care about how the gem is integration tested have a look at the following specs:
|
489
909
|
|
490
|
-
- [spec/
|
491
|
-
- [spec/
|
492
|
-
- [spec/
|
493
|
-
- [spec/
|
494
|
-
- [spec/
|
910
|
+
- [spec/sidekiq_unique_jobs/lock/until_and_while_executing_spec.rb](https://github.com/mhenrixon/sidekiq-unique-jobs/blob/master/spec/sidekiq_unique_jobs/lock/until_and_while_executing_spec.rb)
|
911
|
+
- [spec/sidekiq_unique_jobs/lock/until_executed_spec.rb](https://github.com/mhenrixon/sidekiq-unique-jobs/blob/master/spec/sidekiq_unique_jobs/lock/until_executed_spec.rb)
|
912
|
+
- [spec/sidekiq_unique_jobs/lock/until_expired_spec.rb](https://github.com/mhenrixon/sidekiq-unique-jobs/blob/master/spec/sidekiq_unique_jobs/lock/until_expired_spec.rb)
|
913
|
+
- [spec/sidekiq_unique_jobs/lock/while_executing_reject_spec.rb](https://github.com/mhenrixon/sidekiq-unique-jobs/blob/master/spec/sidekiq_unique_jobs/lock/while_executing_reject_spec.rb)
|
914
|
+
- [spec/sidekiq_unique_jobs/lock/while_executing_spec.rb](https://github.com/mhenrixon/sidekiq-unique-jobs/blob/master/spec/sidekiq_unique_jobs/lock/while_executing_spec.rb)
|
495
915
|
|
496
916
|
## Contributing
|
497
917
|
|
@@ -510,4 +930,3 @@ You can find a list of contributors over on [Contributors][]
|
|
510
930
|
[Sidekiq requirements]: https://github.com/mperham/sidekiq#requirements
|
511
931
|
[Enterprise unique jobs]: https://www.dailydrip.com/topics/sidekiq/drips/sidekiq-enterprise-unique-jobs
|
512
932
|
[Contributors]: https://github.com/mhenrixon/sidekiq-unique-jobs/graphs/contributors
|
513
|
-
[Paypal link https://paypal.me/mhenrixon]: https://paypal.me/mhenrixon
|