sidekiq-unique-jobs 7.0.13 → 7.1.0
Sign up to get free protection for your applications and to get access to all the features.
Potentially problematic release.
This version of sidekiq-unique-jobs might be problematic. Click here for more details.
- checksums.yaml +4 -4
- data/CHANGELOG.md +13 -0
- data/README.md +92 -25
- data/lib/sidekiq_unique_jobs/config.rb +16 -8
- data/lib/sidekiq_unique_jobs/constants.rb +44 -45
- data/lib/sidekiq_unique_jobs/deprecation.rb +35 -0
- data/lib/sidekiq_unique_jobs/exceptions.rb +9 -0
- data/lib/sidekiq_unique_jobs/lock/base_lock.rb +56 -51
- data/lib/sidekiq_unique_jobs/lock/until_and_while_executing.rb +31 -9
- data/lib/sidekiq_unique_jobs/lock/until_executed.rb +17 -5
- data/lib/sidekiq_unique_jobs/lock/until_executing.rb +15 -1
- data/lib/sidekiq_unique_jobs/lock/until_expired.rb +21 -0
- data/lib/sidekiq_unique_jobs/lock/while_executing.rb +12 -7
- data/lib/sidekiq_unique_jobs/lock_config.rb +1 -1
- data/lib/sidekiq_unique_jobs/lock_ttl.rb +1 -1
- data/lib/sidekiq_unique_jobs/locksmith.rb +80 -81
- data/lib/sidekiq_unique_jobs/middleware/client.rb +8 -10
- data/lib/sidekiq_unique_jobs/middleware/server.rb +2 -0
- data/lib/sidekiq_unique_jobs/on_conflict/reschedule.rb +7 -3
- data/lib/sidekiq_unique_jobs/options_with_fallback.rb +4 -11
- data/lib/sidekiq_unique_jobs/orphans/manager.rb +1 -0
- data/lib/sidekiq_unique_jobs/orphans/reaper_resurrector.rb +170 -0
- data/lib/sidekiq_unique_jobs/orphans/ruby_reaper.rb +1 -1
- data/lib/sidekiq_unique_jobs/redis/sorted_set.rb +1 -1
- data/lib/sidekiq_unique_jobs/reflectable.rb +17 -0
- data/lib/sidekiq_unique_jobs/reflections.rb +68 -0
- data/lib/sidekiq_unique_jobs/script/caller.rb +3 -1
- data/lib/sidekiq_unique_jobs/server.rb +2 -1
- data/lib/sidekiq_unique_jobs/sidekiq_unique_ext.rb +13 -35
- data/lib/sidekiq_unique_jobs/sidekiq_unique_jobs.rb +21 -0
- data/lib/sidekiq_unique_jobs/version.rb +1 -1
- data/lib/sidekiq_unique_jobs.rb +4 -0
- data/lib/tasks/changelog.rake +14 -14
- metadata +12 -8
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: e47af152a018a49730a81e9f77ad68debea6a82d1750c6f41e8375be664da2bc
|
4
|
+
data.tar.gz: 3bbb5a4fdb354b6c49e774e26c525b8a5678e83d4a1deb156d544522e76ecf3e
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 0c4d4c979b44940d230d3ad0f3df7487f223b2545aae90eca00eddf0b05fb450fe21beb924929aa40c53bb012fa01cd82e0e611136048370c33baea0375bb184
|
7
|
+
data.tar.gz: 289cbba203cde12c4c070057fd642843d036f12f4623c70ecd3e9c5a777f12f416ea567e971512a86fe8556df4e03316b726f1320096655d0c54ada469758f94
|
data/CHANGELOG.md
CHANGED
@@ -1,5 +1,18 @@
|
|
1
1
|
# Changelog
|
2
2
|
|
3
|
+
## [v7.0.12](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v7.0.12) (2021-06-04)
|
4
|
+
|
5
|
+
[Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v7.0.11...v7.0.12)
|
6
|
+
|
7
|
+
**Implemented enhancements:**
|
8
|
+
|
9
|
+
- Reduce noise of perfectly valid scenario [\#610](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/610) ([mhenrixon](https://github.com/mhenrixon))
|
10
|
+
|
11
|
+
**Merged pull requests:**
|
12
|
+
|
13
|
+
- Set correct namespace for custom strategy example [\#609](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/609) ([Wolfer](https://github.com/Wolfer))
|
14
|
+
- Clarify the documentation related to lock\_ttl [\#607](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/607) ([donaldpiret](https://github.com/donaldpiret))
|
15
|
+
|
3
16
|
## [v7.0.11](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v7.0.11) (2021-05-16)
|
4
17
|
|
5
18
|
[Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v7.0.10...v7.0.11)
|
data/README.md
CHANGED
@@ -12,6 +12,18 @@
|
|
12
12
|
- [Support Me](#support-me)
|
13
13
|
- [Requirements](#requirements)
|
14
14
|
- [General Information](#general-information)
|
15
|
+
- [Reflections \(metrics, logging, etc.\)](#reflections-metrics-logging-etc)
|
16
|
+
- [after_unlock_callback_failed](#after_unlock_callback_failed)
|
17
|
+
- [error](#error)
|
18
|
+
- [execution_failed](#execution_failed)
|
19
|
+
- [lock_failed](#lock_failed)
|
20
|
+
- [locked](#locked)
|
21
|
+
- [reschedule_failed](#reschedule_failed)
|
22
|
+
- [rescheduled](#rescheduled)
|
23
|
+
- [timeout](#timeout)
|
24
|
+
- [unlock_failed](#unlock_failed)
|
25
|
+
- [unlocked](#unlocked)
|
26
|
+
- [unknown_sidekiq_worker](#unknown_sidekiq_worker)
|
15
27
|
- [Global Configuration](#global-configuration)
|
16
28
|
- [debug_lua](#debug_lua)
|
17
29
|
- [lock_timeout](#lock_timeout)
|
@@ -49,7 +61,6 @@
|
|
49
61
|
- [Usage](#usage-1)
|
50
62
|
- [Finer Control over Uniqueness](#finer-control-over-uniqueness)
|
51
63
|
- [After Unlock Callback](#after-unlock-callback)
|
52
|
-
- [Logging](#logging)
|
53
64
|
- [Cleanup Dead Locks](#cleanup-dead-locks)
|
54
65
|
- [Other Sidekiq gems](#other-sidekiq-gems)
|
55
66
|
- [apartment-sidekiq](#apartment-sidekiq)
|
@@ -130,20 +141,16 @@ end
|
|
130
141
|
|
131
142
|
### Your first worker
|
132
143
|
|
144
|
+
The most likely to be used worker is `:until_executed`. This type of lock creates a lock from when `UntilExecutedWorker.perform_async` is called until the the sidekiq processor has processed the job.
|
145
|
+
|
133
146
|
```ruby
|
134
147
|
# frozen_string_literal: true
|
135
148
|
|
136
149
|
class UntilExecutedWorker
|
137
150
|
include Sidekiq::Worker
|
138
151
|
|
139
|
-
sidekiq_options queue: :
|
140
|
-
|
141
|
-
lock: :until_executed,
|
142
|
-
lock_info: true,
|
143
|
-
lock_timeout: 0,
|
144
|
-
lock_prefix: "special",
|
145
|
-
lock_ttl: 0,
|
146
|
-
lock_limit: 5
|
152
|
+
sidekiq_options queue: :until_executed,
|
153
|
+
lock: :until_executed
|
147
154
|
|
148
155
|
def perform
|
149
156
|
logger.info("cowboy")
|
@@ -151,7 +158,6 @@ class UntilExecutedWorker
|
|
151
158
|
logger.info("beebop")
|
152
159
|
end
|
153
160
|
end
|
154
|
-
|
155
161
|
```
|
156
162
|
|
157
163
|
You can read more about the worker configuration in [Worker Configuration](#worker-configuration) below.
|
@@ -179,6 +185,82 @@ See [Interaction w/ Sidekiq](https://github.com/mhenrixon/sidekiq-unique-jobs/wi
|
|
179
185
|
|
180
186
|
See [Locking & Unlocking](https://github.com/mhenrixon/sidekiq-unique-jobs/wiki/Locking-&-Unlocking) for an overview of the differences on when the various lock types are locked and unlocked.
|
181
187
|
|
188
|
+
## Reflections (metrics, logging, etc.)
|
189
|
+
|
190
|
+
To be able to gather some insights on what is going on inside this gem. I provide a reflection API that can be used.
|
191
|
+
|
192
|
+
To setup reflections for logging or metrics, use the following API:
|
193
|
+
|
194
|
+
```ruby
|
195
|
+
|
196
|
+
def extract_log_from_job(message, job_hash)
|
197
|
+
worker = job_hash['class']
|
198
|
+
args = job_hash['args']
|
199
|
+
lock_args = job_hash['lock_args']
|
200
|
+
queue = job_hash['queue']
|
201
|
+
{
|
202
|
+
message: message,
|
203
|
+
worker: worker,
|
204
|
+
args: args,
|
205
|
+
lock_args: lock_args,
|
206
|
+
queue: queue
|
207
|
+
}
|
208
|
+
end
|
209
|
+
|
210
|
+
SidekiqUniqueJobs.reflect do |on|
|
211
|
+
on.lock_failed do |job_hash|
|
212
|
+
message = extract_log_from_job('Lock Failed', job_hash)
|
213
|
+
Sidekiq.logger.warn(message)
|
214
|
+
end
|
215
|
+
end
|
216
|
+
```
|
217
|
+
|
218
|
+
### after_unlock_callback_failed
|
219
|
+
|
220
|
+
This is called when you have configured a custom callback for when a lock has been released.
|
221
|
+
|
222
|
+
### error
|
223
|
+
|
224
|
+
Not in use yet but will be used deep into the stack to provide a means to catch and report errors inside the gem.
|
225
|
+
|
226
|
+
### execution_failed
|
227
|
+
|
228
|
+
When the sidekiq processor picks the job of the queue for certain jobs but your job raised an error to the middleware. This will be the reflection. It is probably nothing to worry about. When your worker raises an error, we need to handle some edge cases for until and while executing.
|
229
|
+
|
230
|
+
### lock_failed
|
231
|
+
|
232
|
+
If we can't achieve a lock, this will be the reflection. It most likely is nothing to worry about. We just couldn't retrieve a lock in a timely fashion.
|
233
|
+
|
234
|
+
The biggest reason for this reflection would be to gather metrics on which workers fail the most at the locking step for example.
|
235
|
+
|
236
|
+
### locked
|
237
|
+
|
238
|
+
For when a lock has been successful. Again, mostly useful for metrics I suppose.
|
239
|
+
|
240
|
+
### reschedule_failed
|
241
|
+
|
242
|
+
For when the reschedule strategy failed to reschedule the job.
|
243
|
+
|
244
|
+
### rescheduled
|
245
|
+
|
246
|
+
For when a job was successfully rescheduled
|
247
|
+
|
248
|
+
### timeout
|
249
|
+
|
250
|
+
This is also mostly useful for reporting/metrics purposes. What this reflection does is signal that the job was configured to wait (`lock_timeout` was configured), but we couldn't retrieve a lock even though we waited for some time.
|
251
|
+
|
252
|
+
### unlock_failed
|
253
|
+
|
254
|
+
This is not got, this is worth
|
255
|
+
|
256
|
+
### unlocked
|
257
|
+
|
258
|
+
Also mostly useful for reporting purposes. The job was successfully unlocked.
|
259
|
+
|
260
|
+
### unknown_sidekiq_worker
|
261
|
+
|
262
|
+
The reason this happens is that the server couldn't find a valid sidekiq worker class. Most likely, that worker isn't intended to be processed by this sidekiq server instance.
|
263
|
+
|
182
264
|
## Global Configuration
|
183
265
|
|
184
266
|
The gem supports a few different configuration options that might be of interest if you run into some weird issues.
|
@@ -683,21 +765,6 @@ class UniqueJobWithFilterMethod
|
|
683
765
|
end.
|
684
766
|
```
|
685
767
|
|
686
|
-
### Logging
|
687
|
-
|
688
|
-
To see logging in sidekiq when duplicate payload has been filtered out you can enable on a per worker basis using the sidekiq options. The default value is false
|
689
|
-
|
690
|
-
```ruby
|
691
|
-
class UniqueJobWithFilterMethod
|
692
|
-
include Sidekiq::Worker
|
693
|
-
sidekiq_options lock: :while_executing,
|
694
|
-
log_duplicate: true
|
695
|
-
|
696
|
-
...
|
697
|
-
|
698
|
-
end
|
699
|
-
```
|
700
|
-
|
701
768
|
### Cleanup Dead Locks
|
702
769
|
|
703
770
|
For sidekiq versions before 5.1 a `sidekiq_retries_exhausted` block is required per worker class. This is deprecated in Sidekiq 6.0
|
@@ -16,6 +16,8 @@ module SidekiqUniqueJobs
|
|
16
16
|
:reaper_count,
|
17
17
|
:reaper_interval,
|
18
18
|
:reaper_timeout,
|
19
|
+
:reaper_resurrector_interval,
|
20
|
+
:reaper_resurrector_enabled,
|
19
21
|
:lock_info,
|
20
22
|
:raise_on_config_error,
|
21
23
|
:current_redis_version)
|
@@ -109,6 +111,14 @@ module SidekiqUniqueJobs
|
|
109
111
|
#
|
110
112
|
# @return [10] stop reaper after 10 seconds
|
111
113
|
REAPER_TIMEOUT = 10
|
114
|
+
#
|
115
|
+
# @return [3600] check if reaper is dead each 3600 seconds
|
116
|
+
REAPER_RESURRECTOR_INTERVAL = 3600
|
117
|
+
|
118
|
+
#
|
119
|
+
# @return [true] enable reaper resurrector
|
120
|
+
REAPER_RESURRECTOR_ENABLED = true
|
121
|
+
|
112
122
|
#
|
113
123
|
# @return [false] while useful it also adds overhead so disable lock_info by default
|
114
124
|
USE_LOCK_INFO = false
|
@@ -178,6 +188,8 @@ module SidekiqUniqueJobs
|
|
178
188
|
REAPER_COUNT,
|
179
189
|
REAPER_INTERVAL,
|
180
190
|
REAPER_TIMEOUT,
|
191
|
+
REAPER_RESURRECTOR_INTERVAL,
|
192
|
+
REAPER_RESURRECTOR_ENABLED,
|
181
193
|
USE_LOCK_INFO,
|
182
194
|
RAISE_ON_CONFIG_ERROR,
|
183
195
|
REDIS_VERSION,
|
@@ -185,26 +197,22 @@ module SidekiqUniqueJobs
|
|
185
197
|
end
|
186
198
|
|
187
199
|
def default_lock_ttl=(obj)
|
188
|
-
warn "[DEPRECATION] `#{class_name}##{__method__}` is deprecated."
|
189
|
-
" Please use `#{class_name}#lock_ttl=` instead."
|
200
|
+
warn "[DEPRECATION] `#{class_name}##{__method__}` is deprecated. Please use `#{class_name}#lock_ttl=` instead."
|
190
201
|
self.lock_ttl = obj
|
191
202
|
end
|
192
203
|
|
193
204
|
def default_lock_timeout=(obj)
|
194
|
-
warn "[DEPRECATION] `#{class_name}##{__method__}` is deprecated."
|
195
|
-
" Please use `#{class_name}#lock_timeout=` instead."
|
205
|
+
warn "[DEPRECATION] `#{class_name}##{__method__}` is deprecated. Please use `#{class_name}#lock_timeout=` instead."
|
196
206
|
self.lock_timeout = obj
|
197
207
|
end
|
198
208
|
|
199
209
|
def default_lock_ttl
|
200
|
-
warn "[DEPRECATION] `#{class_name}##{__method__}` is deprecated."
|
201
|
-
" Please use `#{class_name}#lock_ttl` instead."
|
210
|
+
warn "[DEPRECATION] `#{class_name}##{__method__}` is deprecated. Please use `#{class_name}#lock_ttl` instead."
|
202
211
|
lock_ttl
|
203
212
|
end
|
204
213
|
|
205
214
|
def default_lock_timeout
|
206
|
-
warn "[DEPRECATION] `#{class_name}##{__method__}` is deprecated."
|
207
|
-
" Please use `#{class_name}#lock_timeout` instead."
|
215
|
+
warn "[DEPRECATION] `#{class_name}##{__method__}` is deprecated. Please use `#{class_name}#lock_timeout` instead."
|
208
216
|
lock_timeout
|
209
217
|
end
|
210
218
|
|
@@ -6,49 +6,48 @@
|
|
6
6
|
# @author Mikael Henriksson <mikael@mhenrixon.com>
|
7
7
|
#
|
8
8
|
module SidekiqUniqueJobs
|
9
|
-
ARGS
|
10
|
-
APARTMENT
|
11
|
-
AT
|
12
|
-
CHANGELOGS
|
13
|
-
CLASS
|
14
|
-
CREATED_AT
|
15
|
-
DEAD_VERSION
|
16
|
-
DIGESTS
|
17
|
-
ERRORS
|
18
|
-
JID
|
19
|
-
LIMIT
|
20
|
-
LIVE_VERSION
|
21
|
-
LOCK
|
22
|
-
LOCK_ARGS
|
23
|
-
LOCK_ARGS_METHOD
|
24
|
-
LOCK_DIGEST
|
25
|
-
LOCK_EXPIRATION
|
26
|
-
LOCK_INFO
|
27
|
-
LOCK_LIMIT
|
28
|
-
LOCK_PREFIX
|
29
|
-
LOCK_TIMEOUT
|
30
|
-
LOCK_TTL
|
31
|
-
LOCK_TYPE
|
32
|
-
|
33
|
-
|
34
|
-
|
35
|
-
|
36
|
-
|
37
|
-
|
38
|
-
|
39
|
-
|
40
|
-
|
41
|
-
|
42
|
-
|
43
|
-
|
44
|
-
|
45
|
-
|
46
|
-
|
47
|
-
|
48
|
-
|
49
|
-
|
50
|
-
|
51
|
-
|
52
|
-
|
53
|
-
WORKER = "worker"
|
9
|
+
ARGS ||= "args"
|
10
|
+
APARTMENT ||= "apartment"
|
11
|
+
AT ||= "at"
|
12
|
+
CHANGELOGS ||= "uniquejobs:changelog"
|
13
|
+
CLASS ||= "class"
|
14
|
+
CREATED_AT ||= "created_at"
|
15
|
+
DEAD_VERSION ||= "uniquejobs:dead"
|
16
|
+
DIGESTS ||= "uniquejobs:digests"
|
17
|
+
ERRORS ||= "errors"
|
18
|
+
JID ||= "jid"
|
19
|
+
LIMIT ||= "limit"
|
20
|
+
LIVE_VERSION ||= "uniquejobs:live"
|
21
|
+
LOCK ||= "lock"
|
22
|
+
LOCK_ARGS ||= "lock_args"
|
23
|
+
LOCK_ARGS_METHOD ||= "lock_args_method"
|
24
|
+
LOCK_DIGEST ||= "lock_digest"
|
25
|
+
LOCK_EXPIRATION ||= "lock_expiration"
|
26
|
+
LOCK_INFO ||= "lock_info"
|
27
|
+
LOCK_LIMIT ||= "lock_limit"
|
28
|
+
LOCK_PREFIX ||= "lock_prefix"
|
29
|
+
LOCK_TIMEOUT ||= "lock_timeout"
|
30
|
+
LOCK_TTL ||= "lock_ttl"
|
31
|
+
LOCK_TYPE ||= "lock_type"
|
32
|
+
ON_CLIENT_CONFLICT ||= "on_client_conflict"
|
33
|
+
ON_CONFLICT ||= "on_conflict"
|
34
|
+
ON_SERVER_CONFLICT ||= "on_server_conflict"
|
35
|
+
PAYLOAD ||= "payload"
|
36
|
+
PROCESSES ||= "processes"
|
37
|
+
QUEUE ||= "queue"
|
38
|
+
RETRY ||= "retry"
|
39
|
+
SCHEDULE ||= "schedule"
|
40
|
+
TIME ||= "time"
|
41
|
+
TIMEOUT ||= "timeout"
|
42
|
+
TTL ||= "ttl"
|
43
|
+
TYPE ||= "type"
|
44
|
+
UNIQUE ||= "unique"
|
45
|
+
UNIQUE_ACROSS_QUEUES ||= "unique_across_queues"
|
46
|
+
UNIQUE_ACROSS_WORKERS ||= "unique_across_workers"
|
47
|
+
UNIQUE_ARGS ||= "unique_args"
|
48
|
+
UNIQUE_ARGS_METHOD ||= "unique_args_method"
|
49
|
+
UNIQUE_DIGEST ||= "unique_digest"
|
50
|
+
UNIQUE_PREFIX ||= "unique_prefix"
|
51
|
+
UNIQUE_REAPER ||= "uniquejobs:reaper"
|
52
|
+
WORKER ||= "worker"
|
54
53
|
end
|
@@ -0,0 +1,35 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
module SidekiqUniqueJobs
|
4
|
+
#
|
5
|
+
# Class Deprecation provides logging of deprecations
|
6
|
+
#
|
7
|
+
# @author Mikael Henriksson <mikael@mhenrixon.com>
|
8
|
+
#
|
9
|
+
class Deprecation
|
10
|
+
def self.muted
|
11
|
+
orig_val = Thread.current[:uniquejobs_mute_deprecations]
|
12
|
+
Thread.current[:uniquejobs_mute_deprecations] = true
|
13
|
+
yield
|
14
|
+
ensure
|
15
|
+
Thread.current[:uniquejobs_mute_deprecations] = orig_val
|
16
|
+
end
|
17
|
+
|
18
|
+
def self.muted?
|
19
|
+
Thread.current[:uniquejobs_mute_deprecations] == true
|
20
|
+
end
|
21
|
+
|
22
|
+
def self.warn(msg)
|
23
|
+
return if SidekiqUniqueJobs::Deprecation.muted?
|
24
|
+
|
25
|
+
warn "DEPRECATION WARNING: #{msg}"
|
26
|
+
end
|
27
|
+
|
28
|
+
def self.warn_with_backtrace(msg)
|
29
|
+
return if SidekiqUniqueJobs::Deprecation.muted?
|
30
|
+
|
31
|
+
trace = "\n\nCALLED FROM:\n#{caller.join("\n")}"
|
32
|
+
warn(msg + trace)
|
33
|
+
end
|
34
|
+
end
|
35
|
+
end
|
@@ -18,6 +18,15 @@ module SidekiqUniqueJobs
|
|
18
18
|
end
|
19
19
|
end
|
20
20
|
|
21
|
+
#
|
22
|
+
# Raised when no block was given
|
23
|
+
#
|
24
|
+
class NoBlockGiven < SidekiqUniqueJobs::UniqueJobsError; end
|
25
|
+
#
|
26
|
+
# Raised when a notification has been mistyped
|
27
|
+
#
|
28
|
+
class NoSuchNotificationError < UniqueJobsError; end
|
29
|
+
|
21
30
|
#
|
22
31
|
# Error raised when trying to add a duplicate lock
|
23
32
|
#
|
@@ -7,8 +7,16 @@ module SidekiqUniqueJobs
|
|
7
7
|
# @abstract
|
8
8
|
# @author Mikael Henriksson <mikael@mhenrixon.com>
|
9
9
|
class BaseLock
|
10
|
+
extend Forwardable
|
11
|
+
|
12
|
+
# includes "SidekiqUniqueJobs::Logging"
|
13
|
+
# @!parse include SidekiqUniqueJobs::Logging
|
10
14
|
include SidekiqUniqueJobs::Logging
|
11
15
|
|
16
|
+
# includes "SidekiqUniqueJobs::Reflectable"
|
17
|
+
# @!parse include SidekiqUniqueJobs::Reflectable
|
18
|
+
include SidekiqUniqueJobs::Reflectable
|
19
|
+
|
12
20
|
#
|
13
21
|
# Validates that the sidekiq_options for the worker is valid
|
14
22
|
#
|
@@ -20,6 +28,10 @@ module SidekiqUniqueJobs
|
|
20
28
|
Validator.validate(options)
|
21
29
|
end
|
22
30
|
|
31
|
+
# NOTE: Mainly used for a clean testing API
|
32
|
+
#
|
33
|
+
def_delegators :locksmith, :locked?
|
34
|
+
|
23
35
|
# @param [Hash] item the Sidekiq job hash
|
24
36
|
# @param [Proc] callback the callback to use after unlock
|
25
37
|
# @param [Sidekiq::RedisConnection, ConnectionPool] redis_pool the redis connection
|
@@ -41,10 +53,8 @@ module SidekiqUniqueJobs
|
|
41
53
|
#
|
42
54
|
# @yield to the caller when given a block
|
43
55
|
#
|
44
|
-
def lock
|
45
|
-
|
46
|
-
|
47
|
-
locked_token
|
56
|
+
def lock
|
57
|
+
raise NotImplementedError, "##{__method__} needs to be implemented in #{self.class}"
|
48
58
|
end
|
49
59
|
|
50
60
|
# Execute the job in the Sidekiq server processor
|
@@ -53,31 +63,6 @@ module SidekiqUniqueJobs
|
|
53
63
|
raise NotImplementedError, "##{__method__} needs to be implemented in #{self.class}"
|
54
64
|
end
|
55
65
|
|
56
|
-
# Unlocks the job from redis
|
57
|
-
# @return [String] sidekiq job id when successful
|
58
|
-
# @return [false] when unsuccessful
|
59
|
-
def unlock
|
60
|
-
locksmith.unlock # Only signal to release the lock
|
61
|
-
end
|
62
|
-
|
63
|
-
# Deletes the job from redis if it is locked.
|
64
|
-
def delete
|
65
|
-
locksmith.delete # Soft delete (don't forcefully remove when expiration is set)
|
66
|
-
end
|
67
|
-
|
68
|
-
# Forcefully deletes the job from redis.
|
69
|
-
# This is good for jobs when a previous lock was not unlocked
|
70
|
-
def delete!
|
71
|
-
locksmith.delete! # Force delete the lock
|
72
|
-
end
|
73
|
-
|
74
|
-
# Checks if the item has achieved a lock
|
75
|
-
# @return [true] when this jid has locked the job
|
76
|
-
# @return [false] when this jid has not locked the job
|
77
|
-
def locked?
|
78
|
-
locksmith.locked?
|
79
|
-
end
|
80
|
-
|
81
66
|
#
|
82
67
|
# The lock manager/client
|
83
68
|
#
|
@@ -90,23 +75,6 @@ module SidekiqUniqueJobs
|
|
90
75
|
|
91
76
|
private
|
92
77
|
|
93
|
-
def prepare_item
|
94
|
-
return if item.key?(LOCK_DIGEST)
|
95
|
-
|
96
|
-
# The below should only be done to ease testing
|
97
|
-
# in production this will be done by the middleware
|
98
|
-
SidekiqUniqueJobs::Job.prepare(item)
|
99
|
-
end
|
100
|
-
|
101
|
-
def call_strategy
|
102
|
-
@attempt += 1
|
103
|
-
client_strategy.call { lock if replace? }
|
104
|
-
end
|
105
|
-
|
106
|
-
def replace?
|
107
|
-
client_strategy.replace? && attempt < 2
|
108
|
-
end
|
109
|
-
|
110
78
|
# @!attribute [r] item
|
111
79
|
# @return [Hash<String, Object>] the Sidekiq job hash
|
112
80
|
attr_reader :item
|
@@ -123,18 +91,55 @@ module SidekiqUniqueJobs
|
|
123
91
|
# @return [Integer] the current locking attempt
|
124
92
|
attr_reader :attempt
|
125
93
|
|
126
|
-
def
|
127
|
-
return
|
94
|
+
def prepare_item
|
95
|
+
return if item.key?(LOCK_DIGEST)
|
128
96
|
|
129
|
-
|
130
|
-
|
97
|
+
# The below should only be done to ease testing
|
98
|
+
# in production this will be done by the middleware
|
99
|
+
SidekiqUniqueJobs::Job.prepare(item)
|
100
|
+
end
|
101
|
+
|
102
|
+
#
|
103
|
+
# Handle when lock failed
|
104
|
+
#
|
105
|
+
# @param [Symbol] location: :client or :server
|
106
|
+
#
|
107
|
+
# @return [void]
|
108
|
+
#
|
109
|
+
def lock_failed(origin: :client)
|
110
|
+
reflect(:lock_failed, item)
|
111
|
+
call_strategy(origin: origin)
|
112
|
+
end
|
113
|
+
|
114
|
+
def call_strategy(origin:)
|
115
|
+
@attempt += 1
|
116
|
+
|
117
|
+
case origin
|
118
|
+
when :client
|
119
|
+
client_strategy.call { lock if replace? }
|
120
|
+
when :server
|
121
|
+
server_strategy.call { lock if replace? }
|
122
|
+
else
|
123
|
+
raise SidekiqUniqueJobs::InvalidArgument,
|
124
|
+
"either `for: :server` or `for: :client` needs to be specified"
|
125
|
+
end
|
126
|
+
end
|
127
|
+
|
128
|
+
def replace?
|
129
|
+
client_strategy.replace? && attempt < 2
|
130
|
+
end
|
131
|
+
|
132
|
+
def unlock_and_callback
|
133
|
+
return callback_safely if locksmith.unlock
|
134
|
+
|
135
|
+
reflect(:unlock_failed, item)
|
131
136
|
end
|
132
137
|
|
133
138
|
def callback_safely
|
134
139
|
callback&.call
|
135
140
|
item[JID]
|
136
141
|
rescue StandardError
|
137
|
-
|
142
|
+
reflect(:after_unlock_callback_failed, item)
|
138
143
|
raise
|
139
144
|
end
|
140
145
|
|
@@ -13,30 +13,52 @@ module SidekiqUniqueJobs
|
|
13
13
|
#
|
14
14
|
# @author Mikael Henriksson <mikael@mhenrixon.com>
|
15
15
|
class UntilAndWhileExecuting < BaseLock
|
16
|
+
#
|
17
|
+
# Locks a sidekiq job
|
18
|
+
#
|
19
|
+
# @note Will call a conflict strategy if lock can't be achieved.
|
20
|
+
#
|
21
|
+
# @return [String, nil] the locked jid when properly locked, else nil.
|
22
|
+
#
|
23
|
+
# @yield to the caller when given a block
|
24
|
+
#
|
25
|
+
def lock(origin: :client)
|
26
|
+
return lock_failed(origin: origin) unless (token = locksmith.lock)
|
27
|
+
return yield token if block_given?
|
28
|
+
|
29
|
+
token
|
30
|
+
end
|
31
|
+
|
16
32
|
# Executes in the Sidekiq server process
|
17
33
|
# @yield to the worker class perform method
|
18
34
|
def execute
|
19
|
-
if unlock
|
20
|
-
|
21
|
-
|
22
|
-
end
|
35
|
+
if locksmith.unlock
|
36
|
+
# ensure_relocked do
|
37
|
+
runtime_lock.execute { return yield }
|
38
|
+
# end
|
23
39
|
else
|
24
|
-
|
40
|
+
reflect(:unlock_failed, item)
|
25
41
|
end
|
42
|
+
rescue Exception # rubocop:disable Lint/RescueException
|
43
|
+
reflect(:execution_failed, item)
|
44
|
+
locksmith.lock(wait: 2)
|
45
|
+
|
46
|
+
raise
|
26
47
|
end
|
27
48
|
|
28
49
|
private
|
29
50
|
|
30
|
-
def
|
51
|
+
def ensure_relocked
|
31
52
|
yield
|
32
53
|
rescue Exception # rubocop:disable Lint/RescueException
|
33
|
-
|
34
|
-
lock
|
54
|
+
reflect(:execution_failed, item)
|
55
|
+
locksmith.lock
|
56
|
+
|
35
57
|
raise
|
36
58
|
end
|
37
59
|
|
38
60
|
def runtime_lock
|
39
|
-
@runtime_lock ||= SidekiqUniqueJobs::Lock::WhileExecuting.new(item, callback, redis_pool)
|
61
|
+
@runtime_lock ||= SidekiqUniqueJobs::Lock::WhileExecuting.new(item.dup, callback, redis_pool)
|
40
62
|
end
|
41
63
|
end
|
42
64
|
end
|
@@ -8,16 +8,28 @@ module SidekiqUniqueJobs
|
|
8
8
|
#
|
9
9
|
# @author Mikael Henriksson <mikael@mhenrixon.com>
|
10
10
|
class UntilExecuted < BaseLock
|
11
|
-
|
11
|
+
#
|
12
|
+
# Locks a sidekiq job
|
13
|
+
#
|
14
|
+
# @note Will call a conflict strategy if lock can't be achieved.
|
15
|
+
#
|
16
|
+
# @return [String, nil] the locked jid when properly locked, else nil.
|
17
|
+
#
|
18
|
+
# @yield to the caller when given a block
|
19
|
+
#
|
20
|
+
def lock
|
21
|
+
return lock_failed(origin: :client) unless (token = locksmith.lock)
|
22
|
+
return yield token if block_given?
|
23
|
+
|
24
|
+
token
|
25
|
+
end
|
12
26
|
|
13
27
|
# Executes in the Sidekiq server process
|
14
28
|
# @yield to the worker class perform method
|
15
29
|
def execute
|
16
|
-
|
30
|
+
locksmith.execute do
|
17
31
|
yield
|
18
|
-
|
19
|
-
callback_safely
|
20
|
-
item[JID]
|
32
|
+
unlock_and_callback
|
21
33
|
end
|
22
34
|
end
|
23
35
|
end
|