sidekiq-unique-jobs 7.1.6 → 7.1.11
Sign up to get free protection for your applications and to get access to all the features.
Potentially problematic release.
This version of sidekiq-unique-jobs might be problematic. Click here for more details.
- checksums.yaml +4 -4
- data/CHANGELOG.md +48 -1
- data/README.md +3 -2
- data/lib/sidekiq_unique_jobs/config.rb +43 -4
- data/lib/sidekiq_unique_jobs/constants.rb +44 -44
- data/lib/sidekiq_unique_jobs/deprecation.rb +30 -0
- data/lib/sidekiq_unique_jobs/digests.rb +3 -6
- data/lib/sidekiq_unique_jobs/exceptions.rb +1 -0
- data/lib/sidekiq_unique_jobs/json.rb +7 -0
- data/lib/sidekiq_unique_jobs/lock/base_lock.rb +30 -23
- data/lib/sidekiq_unique_jobs/lock/until_and_while_executing.rb +9 -3
- data/lib/sidekiq_unique_jobs/lock/until_executed.rb +9 -3
- data/lib/sidekiq_unique_jobs/lock/until_executing.rb +10 -4
- data/lib/sidekiq_unique_jobs/lock/until_expired.rb +10 -4
- data/lib/sidekiq_unique_jobs/lock/while_executing.rb +6 -4
- data/lib/sidekiq_unique_jobs/lock_ttl.rb +1 -1
- data/lib/sidekiq_unique_jobs/logging.rb +9 -0
- data/lib/sidekiq_unique_jobs/lua/lock.lua +3 -3
- data/lib/sidekiq_unique_jobs/lua/unlock.lua +12 -5
- data/lib/sidekiq_unique_jobs/middleware/client.rb +1 -1
- data/lib/sidekiq_unique_jobs/options_with_fallback.rb +2 -4
- data/lib/sidekiq_unique_jobs/orphans/manager.rb +35 -0
- data/lib/sidekiq_unique_jobs/orphans/ruby_reaper.rb +4 -2
- data/lib/sidekiq_unique_jobs/reflectable.rb +11 -2
- data/lib/sidekiq_unique_jobs/reflections.rb +12 -1
- data/lib/sidekiq_unique_jobs/server.rb +13 -1
- data/lib/sidekiq_unique_jobs/version.rb +1 -1
- data/lib/tasks/changelog.rake +15 -15
- metadata +10 -14
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: b360fae82f2428f0a6c2d390b6320291bc34da86cb6670e858d902ea111751cc
|
4
|
+
data.tar.gz: c9668d930a39b39635317451a21bd560bb971aa2eecfad6f6aeabbc5559cda1d
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 60540d8412247eae7dc99328eb77a79ad18dccc4fe83345b4abfb1bf4e3aa4e9fe713a49b6c2adaa5b081122471d95e88898cf19f5c97914a84f04fdfa259a11
|
7
|
+
data.tar.gz: c5c5c92b36675ce1567cbbba76839ba9466ac03fdd53000947d611f9f8a3a68df6cafca6909d947c3c367839014cdc61e5fd8a45c7f5c3e5ac114a811aa69828
|
data/CHANGELOG.md
CHANGED
@@ -1,5 +1,53 @@
|
|
1
1
|
# Changelog
|
2
2
|
|
3
|
+
## [v7.1.8](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v7.1.8) (2021-10-08)
|
4
|
+
|
5
|
+
[Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v7.0.13...v7.1.8)
|
6
|
+
|
7
|
+
**Fixed bugs:**
|
8
|
+
|
9
|
+
- undefined method `delete' for class `Sidekiq::Job' [\#634](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/634)
|
10
|
+
- INFO keys not persisted when job is enqueued [\#602](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/602)
|
11
|
+
- lock\_info set to true but no lock info showing up in web ui [\#589](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/589)
|
12
|
+
- Prevent too eager cleanup of lock info [\#645](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/645) ([mhenrixon](https://github.com/mhenrixon))
|
13
|
+
|
14
|
+
**Closed issues:**
|
15
|
+
|
16
|
+
- Compatibility with unreleased Sidekiq 6.3.0 [\#636](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/636)
|
17
|
+
|
18
|
+
**Merged pull requests:**
|
19
|
+
|
20
|
+
- Update docs [\#644](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/644) ([andypple](https://github.com/andypple))
|
21
|
+
|
22
|
+
## [v7.0.13](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v7.0.13) (2021-09-27)
|
23
|
+
|
24
|
+
[Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v7.1.7...v7.0.13)
|
25
|
+
|
26
|
+
## [v7.1.7](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v7.1.7) (2021-09-27)
|
27
|
+
|
28
|
+
[Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v7.1.6...v7.1.7)
|
29
|
+
|
30
|
+
**Implemented enhancements:**
|
31
|
+
|
32
|
+
- Styles [\#642](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/642) ([mhenrixon](https://github.com/mhenrixon))
|
33
|
+
|
34
|
+
**Fixed bugs:**
|
35
|
+
|
36
|
+
- OnConflict::Replace: yield when lock was achieved [\#640](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/640) ([mhenrixon](https://github.com/mhenrixon))
|
37
|
+
|
38
|
+
## [v7.1.6](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v7.1.6) (2021-09-21)
|
39
|
+
|
40
|
+
[Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v7.1.5...v7.1.6)
|
41
|
+
|
42
|
+
**Closed issues:**
|
43
|
+
|
44
|
+
- until\_and\_while\_executing is not running the job at all in sidekiq-unique-jobs 7.1.4 [\#626](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/626)
|
45
|
+
|
46
|
+
**Merged pull requests:**
|
47
|
+
|
48
|
+
- Necessary upgrades for Sidekiq v6.2.2 [\#639](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/639) ([mhenrixon](https://github.com/mhenrixon))
|
49
|
+
- Tese to these in README.md [\#633](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/633) ([carrickr](https://github.com/carrickr))
|
50
|
+
|
3
51
|
## [v7.1.5](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v7.1.5) (2021-07-28)
|
4
52
|
|
5
53
|
[Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v7.1.4...v7.1.5)
|
@@ -112,7 +160,6 @@
|
|
112
160
|
|
113
161
|
**Fixed bugs:**
|
114
162
|
|
115
|
-
- lock\_info set to true but no lock info showing up in web ui [\#589](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/589)
|
116
163
|
- Fix recording lock\_info [\#599](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/599) ([mhenrixon](https://github.com/mhenrixon))
|
117
164
|
|
118
165
|
## [v7.0.8](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v7.0.8) (2021-04-14)
|
data/README.md
CHANGED
@@ -92,10 +92,11 @@ This gem adds unique constraints to sidekiq jobs. The uniqueness is achieved by
|
|
92
92
|
|
93
93
|
By default, only one lock for a given hash can be acquired. What happens when a lock can't be acquired is governed by a chosen [Conflict Strategy](#conflict-strategy) strategy. Unless a conflict strategy is chosen
|
94
94
|
|
95
|
-
This is the documentation for the
|
95
|
+
This is the documentation for the `main` branch. You can find the documentation for each release by navigating to its tag.
|
96
96
|
|
97
97
|
Here are links to some of the old versions
|
98
98
|
|
99
|
+
- [v7.0.12](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v7.0.12)
|
99
100
|
- [v6.0.25](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v6.0.25)
|
100
101
|
- [v5.0.10](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v5.0.10)
|
101
102
|
- [v4.0.18](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v4.0.18)
|
@@ -1042,7 +1043,7 @@ There is a [![Join the chat at https://gitter.im/mhenrixon/sidekiq-unique-jobs](
|
|
1042
1043
|
|
1043
1044
|
You can find a list of contributors over on [Contributors][]
|
1044
1045
|
|
1045
|
-
[Enterprise unique jobs]: https://
|
1046
|
+
[Enterprise unique jobs]: https://github.com/mperham/sidekiq/wiki/Ent-Unique-Jobs
|
1046
1047
|
[Contributors]: https://github.com/mhenrixon/sidekiq-unique-jobs/graphs/contributors
|
1047
1048
|
[v4.0.18]: https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v4.0.18
|
1048
1049
|
[v5.0.10]: https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v5.0.10.
|
@@ -196,26 +196,65 @@ module SidekiqUniqueJobs
|
|
196
196
|
)
|
197
197
|
end
|
198
198
|
|
199
|
+
#
|
200
|
+
# Set the default_lock_ttl
|
201
|
+
# @deprecated
|
202
|
+
#
|
203
|
+
# @param [Integer] obj value to set (seconds)
|
204
|
+
#
|
205
|
+
# @return [<type>] <description>
|
206
|
+
#
|
199
207
|
def default_lock_ttl=(obj)
|
200
|
-
warn "[DEPRECATION] `#{class_name}##{__method__}` is deprecated.
|
208
|
+
warn "[DEPRECATION] `#{class_name}##{__method__}` is deprecated." \
|
209
|
+
" Please use `#{class_name}#lock_ttl=` instead."
|
201
210
|
self.lock_ttl = obj
|
202
211
|
end
|
203
212
|
|
213
|
+
#
|
214
|
+
# Set new value for default_lock_timeout
|
215
|
+
# @deprecated
|
216
|
+
#
|
217
|
+
# @param [Integer] obj value to set (seconds)
|
218
|
+
#
|
219
|
+
# @return [Integer]
|
220
|
+
#
|
204
221
|
def default_lock_timeout=(obj)
|
205
|
-
warn "[DEPRECATION] `#{class_name}##{__method__}` is deprecated.
|
222
|
+
warn "[DEPRECATION] `#{class_name}##{__method__}` is deprecated." \
|
223
|
+
" Please use `#{class_name}#lock_timeout=` instead."
|
206
224
|
self.lock_timeout = obj
|
207
225
|
end
|
208
226
|
|
227
|
+
#
|
228
|
+
# Default lock TTL (Time To Live)
|
229
|
+
# @deprecated
|
230
|
+
#
|
231
|
+
# @return [nil, Integer] configured value or nil
|
232
|
+
#
|
209
233
|
def default_lock_ttl
|
210
|
-
warn "[DEPRECATION] `#{class_name}##{__method__}` is deprecated.
|
234
|
+
warn "[DEPRECATION] `#{class_name}##{__method__}` is deprecated." \
|
235
|
+
" Please use `#{class_name}#lock_ttl` instead."
|
211
236
|
lock_ttl
|
212
237
|
end
|
213
238
|
|
239
|
+
#
|
240
|
+
# Default Lock Timeout
|
241
|
+
# @deprecated
|
242
|
+
#
|
243
|
+
#
|
244
|
+
# @return [nil, Integer] configured value or nil
|
245
|
+
#
|
214
246
|
def default_lock_timeout
|
215
|
-
warn "[DEPRECATION] `#{class_name}##{__method__}` is deprecated.
|
247
|
+
warn "[DEPRECATION] `#{class_name}##{__method__}` is deprecated." \
|
248
|
+
" Please use `#{class_name}#lock_timeout` instead."
|
216
249
|
lock_timeout
|
217
250
|
end
|
218
251
|
|
252
|
+
#
|
253
|
+
# Memoized variable to get the class name
|
254
|
+
#
|
255
|
+
#
|
256
|
+
# @return [String] name of the class
|
257
|
+
#
|
219
258
|
def class_name
|
220
259
|
@class_name ||= self.class.name
|
221
260
|
end
|
@@ -6,48 +6,48 @@
|
|
6
6
|
# @author Mikael Henriksson <mikael@mhenrixon.com>
|
7
7
|
#
|
8
8
|
module SidekiqUniqueJobs
|
9
|
-
ARGS
|
10
|
-
APARTMENT
|
11
|
-
AT
|
12
|
-
CHANGELOGS
|
13
|
-
CLASS
|
14
|
-
CREATED_AT
|
15
|
-
DEAD_VERSION
|
16
|
-
DIGESTS
|
17
|
-
ERRORS
|
18
|
-
JID
|
19
|
-
LIMIT
|
20
|
-
LIVE_VERSION
|
21
|
-
LOCK
|
22
|
-
LOCK_ARGS
|
23
|
-
LOCK_ARGS_METHOD
|
24
|
-
LOCK_DIGEST
|
25
|
-
LOCK_EXPIRATION
|
26
|
-
LOCK_INFO
|
27
|
-
LOCK_LIMIT
|
28
|
-
LOCK_PREFIX
|
29
|
-
LOCK_TIMEOUT
|
30
|
-
LOCK_TTL
|
31
|
-
LOCK_TYPE
|
32
|
-
ON_CLIENT_CONFLICT
|
33
|
-
ON_CONFLICT
|
34
|
-
ON_SERVER_CONFLICT
|
35
|
-
PAYLOAD
|
36
|
-
PROCESSES
|
37
|
-
QUEUE
|
38
|
-
RETRY
|
39
|
-
SCHEDULE
|
40
|
-
TIME
|
41
|
-
TIMEOUT
|
42
|
-
TTL
|
43
|
-
TYPE
|
44
|
-
UNIQUE
|
45
|
-
UNIQUE_ACROSS_QUEUES
|
46
|
-
UNIQUE_ACROSS_WORKERS
|
47
|
-
UNIQUE_ARGS
|
48
|
-
UNIQUE_ARGS_METHOD
|
49
|
-
UNIQUE_DIGEST
|
50
|
-
UNIQUE_PREFIX
|
51
|
-
UNIQUE_REAPER
|
52
|
-
WORKER
|
9
|
+
ARGS = "args"
|
10
|
+
APARTMENT = "apartment"
|
11
|
+
AT = "at"
|
12
|
+
CHANGELOGS = "uniquejobs:changelog"
|
13
|
+
CLASS = "class"
|
14
|
+
CREATED_AT = "created_at"
|
15
|
+
DEAD_VERSION = "uniquejobs:dead"
|
16
|
+
DIGESTS = "uniquejobs:digests"
|
17
|
+
ERRORS = "errors"
|
18
|
+
JID = "jid"
|
19
|
+
LIMIT = "limit"
|
20
|
+
LIVE_VERSION = "uniquejobs:live"
|
21
|
+
LOCK = "lock"
|
22
|
+
LOCK_ARGS = "lock_args"
|
23
|
+
LOCK_ARGS_METHOD = "lock_args_method"
|
24
|
+
LOCK_DIGEST = "lock_digest"
|
25
|
+
LOCK_EXPIRATION = "lock_expiration"
|
26
|
+
LOCK_INFO = "lock_info"
|
27
|
+
LOCK_LIMIT = "lock_limit"
|
28
|
+
LOCK_PREFIX = "lock_prefix"
|
29
|
+
LOCK_TIMEOUT = "lock_timeout"
|
30
|
+
LOCK_TTL = "lock_ttl"
|
31
|
+
LOCK_TYPE = "lock_type"
|
32
|
+
ON_CLIENT_CONFLICT = "on_client_conflict"
|
33
|
+
ON_CONFLICT = "on_conflict"
|
34
|
+
ON_SERVER_CONFLICT = "on_server_conflict"
|
35
|
+
PAYLOAD = "payload"
|
36
|
+
PROCESSES = "processes"
|
37
|
+
QUEUE = "queue"
|
38
|
+
RETRY = "retry"
|
39
|
+
SCHEDULE = "schedule"
|
40
|
+
TIME = "time"
|
41
|
+
TIMEOUT = "timeout"
|
42
|
+
TTL = "ttl"
|
43
|
+
TYPE = "type"
|
44
|
+
UNIQUE = "unique"
|
45
|
+
UNIQUE_ACROSS_QUEUES = "unique_across_queues"
|
46
|
+
UNIQUE_ACROSS_WORKERS = "unique_across_workers"
|
47
|
+
UNIQUE_ARGS = "unique_args"
|
48
|
+
UNIQUE_ARGS_METHOD = "unique_args_method"
|
49
|
+
UNIQUE_DIGEST = "unique_digest"
|
50
|
+
UNIQUE_PREFIX = "unique_prefix"
|
51
|
+
UNIQUE_REAPER = "uniquejobs:reaper"
|
52
|
+
WORKER = "worker"
|
53
53
|
end
|
@@ -7,6 +7,13 @@ module SidekiqUniqueJobs
|
|
7
7
|
# @author Mikael Henriksson <mikael@mhenrixon.com>
|
8
8
|
#
|
9
9
|
class Deprecation
|
10
|
+
#
|
11
|
+
# Mute warnings from this gem in a threaded context
|
12
|
+
#
|
13
|
+
#
|
14
|
+
# @return [void] <description>
|
15
|
+
#
|
16
|
+
# @yieldreturn [void]
|
10
17
|
def self.muted
|
11
18
|
orig_val = Thread.current[:uniquejobs_mute_deprecations]
|
12
19
|
Thread.current[:uniquejobs_mute_deprecations] = true
|
@@ -15,21 +22,44 @@ module SidekiqUniqueJobs
|
|
15
22
|
Thread.current[:uniquejobs_mute_deprecations] = orig_val
|
16
23
|
end
|
17
24
|
|
25
|
+
#
|
26
|
+
# Check if deprecation warnings have been muted
|
27
|
+
#
|
28
|
+
#
|
29
|
+
# @return [true,false]
|
30
|
+
#
|
18
31
|
def self.muted?
|
19
32
|
Thread.current[:uniquejobs_mute_deprecations] == true
|
20
33
|
end
|
21
34
|
|
35
|
+
#
|
36
|
+
# Warn about deprecation
|
37
|
+
#
|
38
|
+
# @param [String] msg a descriptive reason for why the deprecation
|
39
|
+
#
|
40
|
+
# @return [void]
|
41
|
+
#
|
22
42
|
def self.warn(msg)
|
23
43
|
return if SidekiqUniqueJobs::Deprecation.muted?
|
24
44
|
|
25
45
|
warn "DEPRECATION WARNING: #{msg}"
|
46
|
+
nil
|
26
47
|
end
|
27
48
|
|
49
|
+
#
|
50
|
+
# Warn about deprecation and provide a context
|
51
|
+
#
|
52
|
+
# @param [String] msg a descriptive reason for why the deprecation
|
53
|
+
#
|
54
|
+
# @return [void]
|
55
|
+
#
|
28
56
|
def self.warn_with_backtrace(msg)
|
29
57
|
return if SidekiqUniqueJobs::Deprecation.muted?
|
30
58
|
|
31
59
|
trace = "\n\nCALLED FROM:\n#{caller.join("\n")}"
|
32
60
|
warn(msg + trace)
|
61
|
+
|
62
|
+
nil
|
33
63
|
end
|
34
64
|
end
|
35
65
|
end
|
@@ -31,7 +31,8 @@ module SidekiqUniqueJobs
|
|
31
31
|
#
|
32
32
|
# @param [String] pattern a key pattern to match with
|
33
33
|
# @param [Integer] count the maximum number
|
34
|
-
# @return [
|
34
|
+
# @return [Hash<String,Float>] Hash mapping of digest matching the given pattern and score
|
35
|
+
|
35
36
|
def delete_by_pattern(pattern, count: DEFAULT_COUNT)
|
36
37
|
result, elapsed = timed do
|
37
38
|
digests = entries(pattern: pattern, count: count).keys
|
@@ -80,11 +81,7 @@ module SidekiqUniqueJobs
|
|
80
81
|
options[:match] = pattern
|
81
82
|
options[:count] = count
|
82
83
|
|
83
|
-
|
84
|
-
|
85
|
-
result.each_with_object({}) do |entry, hash|
|
86
|
-
hash[entry[0]] = entry[1]
|
87
|
-
end
|
84
|
+
redis { |conn| conn.zscan_each(key, **options).to_a }.to_h
|
88
85
|
end
|
89
86
|
|
90
87
|
#
|
@@ -20,6 +20,13 @@ module SidekiqUniqueJobs
|
|
20
20
|
::JSON.parse(string)
|
21
21
|
end
|
22
22
|
|
23
|
+
#
|
24
|
+
# Prevents trying JSON.load from raising errors given argument is a hash
|
25
|
+
#
|
26
|
+
# @param [String, Hash] string the JSON string to parse
|
27
|
+
#
|
28
|
+
# @return [Hash,Array]
|
29
|
+
#
|
23
30
|
def safe_load_json(string)
|
24
31
|
return string if string.is_a?(Hash)
|
25
32
|
|
@@ -91,6 +91,13 @@ module SidekiqUniqueJobs
|
|
91
91
|
# @return [Integer] the current locking attempt
|
92
92
|
attr_reader :attempt
|
93
93
|
|
94
|
+
#
|
95
|
+
# Eases testing by allowing the lock implementation to add the missing
|
96
|
+
# keys to the job hash.
|
97
|
+
#
|
98
|
+
#
|
99
|
+
# @return [void] the return value should be irrelevant
|
100
|
+
#
|
94
101
|
def prepare_item
|
95
102
|
return if item.key?(LOCK_DIGEST)
|
96
103
|
|
@@ -100,34 +107,22 @@ module SidekiqUniqueJobs
|
|
100
107
|
end
|
101
108
|
|
102
109
|
#
|
103
|
-
#
|
110
|
+
# Call whatever strategry that has been configured
|
104
111
|
#
|
105
|
-
# @param [Symbol]
|
112
|
+
# @param [Symbol] origin: the origin `:client` or `:server`
|
106
113
|
#
|
107
|
-
# @return [void]
|
114
|
+
# @return [void] the return value is irrelevant
|
108
115
|
#
|
109
|
-
|
110
|
-
|
111
|
-
|
112
|
-
nil
|
113
|
-
end
|
114
|
-
|
116
|
+
# @yieldparam [void] if a new job id was set and a block is given
|
117
|
+
# @yieldreturn [void] the yield is irrelevant, it only provides a mechanism in
|
118
|
+
# one specific situation to yield back to the middleware.
|
115
119
|
def call_strategy(origin:)
|
116
|
-
|
120
|
+
new_job_id = nil
|
121
|
+
strategy = strategy_for(origin)
|
122
|
+
@attempt += 1
|
117
123
|
|
118
|
-
|
119
|
-
|
120
|
-
client_strategy.call { lock if replace? }
|
121
|
-
when :server
|
122
|
-
server_strategy.call { lock if replace? }
|
123
|
-
else
|
124
|
-
raise SidekiqUniqueJobs::InvalidArgument,
|
125
|
-
"either `for: :server` or `for: :client` needs to be specified"
|
126
|
-
end
|
127
|
-
end
|
128
|
-
|
129
|
-
def replace?
|
130
|
-
client_strategy.replace? && attempt < 2
|
124
|
+
strategy.call { new_job_id = lock if strategy.replace? && @attempt < 2 }
|
125
|
+
yield if new_job_id && block_given?
|
131
126
|
end
|
132
127
|
|
133
128
|
def unlock_and_callback
|
@@ -144,6 +139,18 @@ module SidekiqUniqueJobs
|
|
144
139
|
raise
|
145
140
|
end
|
146
141
|
|
142
|
+
def strategy_for(origin)
|
143
|
+
case origin
|
144
|
+
when :client
|
145
|
+
client_strategy
|
146
|
+
when :server
|
147
|
+
server_strategy
|
148
|
+
else
|
149
|
+
raise SidekiqUniqueJobs::InvalidArgument,
|
150
|
+
"#origin needs to be either `:server` or `:client`"
|
151
|
+
end
|
152
|
+
end
|
153
|
+
|
147
154
|
def client_strategy
|
148
155
|
@client_strategy ||=
|
149
156
|
OnConflict.find_strategy(lock_config.on_client_conflict).new(item, redis_pool)
|
@@ -22,9 +22,15 @@ module SidekiqUniqueJobs
|
|
22
22
|
#
|
23
23
|
# @yield to the caller when given a block
|
24
24
|
#
|
25
|
-
def lock(origin: :client)
|
26
|
-
|
27
|
-
|
25
|
+
def lock(origin: :client, &block)
|
26
|
+
unless (token = locksmith.lock)
|
27
|
+
reflect(:lock_failed, item)
|
28
|
+
call_strategy(origin: origin, &block)
|
29
|
+
|
30
|
+
return
|
31
|
+
end
|
32
|
+
|
33
|
+
yield if block
|
28
34
|
|
29
35
|
token
|
30
36
|
end
|
@@ -17,9 +17,15 @@ module SidekiqUniqueJobs
|
|
17
17
|
#
|
18
18
|
# @yield to the caller when given a block
|
19
19
|
#
|
20
|
-
def lock
|
21
|
-
|
22
|
-
|
20
|
+
def lock(&block)
|
21
|
+
unless (token = locksmith.lock)
|
22
|
+
reflect(:lock_failed, item)
|
23
|
+
call_strategy(origin: :client, &block)
|
24
|
+
|
25
|
+
return
|
26
|
+
end
|
27
|
+
|
28
|
+
yield if block
|
23
29
|
|
24
30
|
token
|
25
31
|
end
|
@@ -15,11 +15,17 @@ module SidekiqUniqueJobs
|
|
15
15
|
#
|
16
16
|
# @return [String, nil] the locked jid when properly locked, else nil.
|
17
17
|
#
|
18
|
-
def lock
|
19
|
-
|
20
|
-
|
18
|
+
def lock(&block)
|
19
|
+
unless (token = locksmith.lock)
|
20
|
+
reflect(:lock_failed, item)
|
21
|
+
call_strategy(origin: :client, &block)
|
21
22
|
|
22
|
-
|
23
|
+
return
|
24
|
+
end
|
25
|
+
|
26
|
+
yield if block
|
27
|
+
|
28
|
+
token
|
23
29
|
end
|
24
30
|
|
25
31
|
# Executes in the Sidekiq server process
|
@@ -17,11 +17,17 @@ module SidekiqUniqueJobs
|
|
17
17
|
#
|
18
18
|
# @yield to the caller when given a block
|
19
19
|
#
|
20
|
-
def lock
|
21
|
-
|
22
|
-
|
20
|
+
def lock(&block)
|
21
|
+
unless (token = locksmith.lock)
|
22
|
+
reflect(:lock_failed, item)
|
23
|
+
call_strategy(origin: :client, &block)
|
23
24
|
|
24
|
-
|
25
|
+
return
|
26
|
+
end
|
27
|
+
|
28
|
+
yield if block
|
29
|
+
|
30
|
+
token
|
25
31
|
end
|
26
32
|
|
27
33
|
# Executes in the Sidekiq server process
|
@@ -11,7 +11,7 @@ module SidekiqUniqueJobs
|
|
11
11
|
#
|
12
12
|
# @author Mikael Henriksson <mikael@mhenrixon.com>
|
13
13
|
class WhileExecuting < BaseLock
|
14
|
-
RUN_SUFFIX
|
14
|
+
RUN_SUFFIX = ":RUN"
|
15
15
|
|
16
16
|
include SidekiqUniqueJobs::OptionsWithFallback
|
17
17
|
include SidekiqUniqueJobs::Logging::Middleware
|
@@ -30,7 +30,7 @@ module SidekiqUniqueJobs
|
|
30
30
|
# @return [true] always returns true
|
31
31
|
def lock
|
32
32
|
job_id = item[JID]
|
33
|
-
yield
|
33
|
+
yield if block_given?
|
34
34
|
|
35
35
|
job_id
|
36
36
|
end
|
@@ -38,14 +38,16 @@ module SidekiqUniqueJobs
|
|
38
38
|
# Executes in the Sidekiq server process.
|
39
39
|
# These jobs are locked in the server process not from the client
|
40
40
|
# @yield to the worker class perform method
|
41
|
-
def execute
|
41
|
+
def execute(&block)
|
42
42
|
with_logging_context do
|
43
|
-
|
43
|
+
executed = locksmith.execute do
|
44
44
|
yield
|
45
45
|
callback_safely if locksmith.unlock
|
46
46
|
ensure
|
47
47
|
locksmith.unlock
|
48
48
|
end
|
49
|
+
|
50
|
+
call_strategy(origin: :server, &block) unless executed
|
49
51
|
end
|
50
52
|
end
|
51
53
|
|
@@ -71,7 +71,7 @@ module SidekiqUniqueJobs
|
|
71
71
|
ttl ||= item[LOCK_EXPIRATION] # TODO: Deprecate at some point
|
72
72
|
ttl ||= worker_options[LOCK_EXPIRATION] # TODO: Deprecate at some point
|
73
73
|
ttl ||= SidekiqUniqueJobs.config.lock_ttl
|
74
|
-
ttl && ttl.to_i + time_until_scheduled
|
74
|
+
ttl && (ttl.to_i + time_until_scheduled)
|
75
75
|
end
|
76
76
|
end
|
77
77
|
end
|
@@ -92,9 +92,18 @@ module SidekiqUniqueJobs
|
|
92
92
|
def log_fatal(message_or_exception = nil, item = nil, &block)
|
93
93
|
message = build_message(message_or_exception, item)
|
94
94
|
logger.fatal(message, &block)
|
95
|
+
|
95
96
|
nil
|
96
97
|
end
|
97
98
|
|
99
|
+
#
|
100
|
+
# Build a log message
|
101
|
+
#
|
102
|
+
# @param [String, Exception] message_or_exception an entry to log
|
103
|
+
# @param [Hash] item the sidekiq job hash
|
104
|
+
#
|
105
|
+
# @return [String] a complete log entry
|
106
|
+
#
|
98
107
|
def build_message(message_or_exception, item = nil)
|
99
108
|
return nil if message_or_exception.nil?
|
100
109
|
return message_or_exception if item.nil?
|
@@ -76,6 +76,9 @@ if pttl and pttl > 0 then
|
|
76
76
|
|
77
77
|
log_debug("PEXPIRE", locked, pttl)
|
78
78
|
redis.call("PEXPIRE", locked, pttl)
|
79
|
+
|
80
|
+
log_debug("PEXPIRE", info, pttl)
|
81
|
+
redis.call("PEXPIRE", info, pttl)
|
79
82
|
end
|
80
83
|
|
81
84
|
log_debug("PEXPIRE", queued, 1000)
|
@@ -84,9 +87,6 @@ redis.call("PEXPIRE", queued, 1000)
|
|
84
87
|
log_debug("PEXPIRE", primed, 1000)
|
85
88
|
redis.call("PEXPIRE", primed, 1000)
|
86
89
|
|
87
|
-
log_debug("PEXPIRE", info, 1000)
|
88
|
-
redis.call("PEXPIRE", info, 1000)
|
89
|
-
|
90
90
|
log("Locked")
|
91
91
|
log_debug("END lock digest:", digest, "job_id:", job_id)
|
92
92
|
return job_id
|
@@ -65,11 +65,6 @@ redis.call("LREM", queued, -1, job_id)
|
|
65
65
|
log_debug("LREM", primed, -1, job_id)
|
66
66
|
redis.call("LREM", primed, -1, job_id)
|
67
67
|
|
68
|
-
if limit and limit <= 1 and locked_count and locked_count <= 1 then
|
69
|
-
log_debug("ZREM", digests, digest)
|
70
|
-
redis.call("ZREM", digests, digest)
|
71
|
-
end
|
72
|
-
|
73
68
|
local redis_version = toversion(redisversion)
|
74
69
|
local del_cmd = "DEL"
|
75
70
|
|
@@ -83,6 +78,18 @@ if lock_type ~= "until_expired" then
|
|
83
78
|
redis.call("HDEL", locked, job_id)
|
84
79
|
end
|
85
80
|
|
81
|
+
local locked_count = redis.call("HLEN", locked)
|
82
|
+
|
83
|
+
if locked_count and locked_count < 1 then
|
84
|
+
log_debug(del_cmd, locked)
|
85
|
+
redis.call(del_cmd, locked)
|
86
|
+
end
|
87
|
+
|
88
|
+
if limit and limit <= 1 and locked_count and locked_count <= 1 then
|
89
|
+
log_debug("ZREM", digests, digest)
|
90
|
+
redis.call("ZREM", digests, digest)
|
91
|
+
end
|
92
|
+
|
86
93
|
log_debug("LPUSH", queued, "1")
|
87
94
|
redis.call("LPUSH", queued, "1")
|
88
95
|
|
@@ -46,10 +46,8 @@ module SidekiqUniqueJobs
|
|
46
46
|
# @return [Class]
|
47
47
|
#
|
48
48
|
def lock_class
|
49
|
-
@lock_class ||=
|
50
|
-
|
51
|
-
raise UnknownLock, "No implementation for `lock: :#{lock_type}`"
|
52
|
-
end
|
49
|
+
@lock_class ||= locks.fetch(lock_type.to_sym) do
|
50
|
+
raise UnknownLock, "No implementation for `lock: :#{lock_type}`"
|
53
51
|
end
|
54
52
|
end
|
55
53
|
|
@@ -10,10 +10,18 @@ module SidekiqUniqueJobs
|
|
10
10
|
module Manager
|
11
11
|
module_function
|
12
12
|
|
13
|
+
#
|
14
|
+
# @return [Float] the amount to add to the reaper interval
|
13
15
|
DRIFT_FACTOR = 0.02
|
16
|
+
#
|
17
|
+
# @return [Symbol] allowed reapers (:ruby or :lua)
|
14
18
|
REAPERS = [:ruby, :lua].freeze
|
15
19
|
|
20
|
+
# includes "SidekiqUniqueJobs::Connection"
|
21
|
+
# @!parse include SidekiqUniqueJobs::Connection
|
16
22
|
include SidekiqUniqueJobs::Connection
|
23
|
+
# includes "SidekiqUniqueJobs::Logging"
|
24
|
+
# @!parse include SidekiqUniqueJobs::Logging
|
17
25
|
include SidekiqUniqueJobs::Logging
|
18
26
|
|
19
27
|
#
|
@@ -65,6 +73,12 @@ module SidekiqUniqueJobs
|
|
65
73
|
@task ||= default_task
|
66
74
|
end
|
67
75
|
|
76
|
+
#
|
77
|
+
# A properly configured timer task
|
78
|
+
#
|
79
|
+
#
|
80
|
+
# @return [SidekiqUniqueJobs::TimerTask]
|
81
|
+
#
|
68
82
|
def default_task
|
69
83
|
SidekiqUniqueJobs::TimerTask.new(timer_task_options) do
|
70
84
|
with_logging_context do
|
@@ -76,6 +90,13 @@ module SidekiqUniqueJobs
|
|
76
90
|
end
|
77
91
|
end
|
78
92
|
|
93
|
+
#
|
94
|
+
# Store a task to use for scheduled execution
|
95
|
+
#
|
96
|
+
# @param [SidekiqUniqueJobs::TimerTask] task the task to use
|
97
|
+
#
|
98
|
+
# @return [void]
|
99
|
+
#
|
79
100
|
def task=(task)
|
80
101
|
@task = task
|
81
102
|
end
|
@@ -201,10 +222,24 @@ module SidekiqUniqueJobs
|
|
201
222
|
redis { |conn| conn.del(UNIQUE_REAPER) }
|
202
223
|
end
|
203
224
|
|
225
|
+
#
|
226
|
+
# Reaper interval with a little drift
|
227
|
+
# Redis isn't exact enough so to give a little bufffer,
|
228
|
+
# we add a tiny value to the reaper interval.
|
229
|
+
#
|
230
|
+
#
|
231
|
+
# @return [Integer] <description>
|
232
|
+
#
|
204
233
|
def drift_reaper_interval
|
205
234
|
reaper_interval + (reaper_interval * DRIFT_FACTOR).to_i
|
206
235
|
end
|
207
236
|
|
237
|
+
#
|
238
|
+
# Current time (as integer value)
|
239
|
+
#
|
240
|
+
#
|
241
|
+
# @return [Integer]
|
242
|
+
#
|
208
243
|
def current_timestamp
|
209
244
|
Time.now.to_i
|
210
245
|
end
|
@@ -10,6 +10,8 @@ module SidekiqUniqueJobs
|
|
10
10
|
# @author Mikael Henriksson <mikael@mhenrixon.com>
|
11
11
|
#
|
12
12
|
class RubyReaper < Reaper
|
13
|
+
#
|
14
|
+
# @return [String] the suffix for :RUN locks
|
13
15
|
RUN_SUFFIX = ":RUN"
|
14
16
|
#
|
15
17
|
# @!attribute [r] digests
|
@@ -57,7 +59,7 @@ module SidekiqUniqueJobs
|
|
57
59
|
next if belongs_to_job?(digest)
|
58
60
|
|
59
61
|
memo << digest
|
60
|
-
break if memo.size >= reaper_count
|
62
|
+
break memo if memo.size >= reaper_count
|
61
63
|
end
|
62
64
|
end
|
63
65
|
|
@@ -182,7 +184,7 @@ module SidekiqUniqueJobs
|
|
182
184
|
page_size = 50
|
183
185
|
|
184
186
|
loop do
|
185
|
-
range_start = page * page_size - deleted_size
|
187
|
+
range_start = (page * page_size) - deleted_size
|
186
188
|
range_end = range_start + page_size - 1
|
187
189
|
entries = conn.lrange(queue_key, range_start, range_end)
|
188
190
|
page += 1
|
@@ -7,11 +7,20 @@ module SidekiqUniqueJobs
|
|
7
7
|
# @author Mikael Henriksson <mikael@mhenrixon.com>
|
8
8
|
#
|
9
9
|
module Reflectable
|
10
|
-
|
11
|
-
|
10
|
+
#
|
11
|
+
# Reflects on specific event
|
12
|
+
#
|
13
|
+
# @param [Symbol] reflection the reflected event
|
14
|
+
# @param [Array] args arguments to provide to reflector
|
15
|
+
#
|
16
|
+
# @return [void]
|
17
|
+
#
|
18
|
+
def reflect(reflection, *args)
|
19
|
+
SidekiqUniqueJobs.reflections.dispatch(reflection, *args)
|
12
20
|
nil
|
13
21
|
rescue UniqueJobsError => ex
|
14
22
|
SidekiqUniqueJobs.logger.error(ex)
|
23
|
+
nil
|
15
24
|
end
|
16
25
|
end
|
17
26
|
end
|
@@ -42,6 +42,14 @@ module SidekiqUniqueJobs
|
|
42
42
|
@reflections = {}
|
43
43
|
end
|
44
44
|
|
45
|
+
#
|
46
|
+
# Dispatch a reflected event
|
47
|
+
#
|
48
|
+
# @param [reflection] reflection the reflected event
|
49
|
+
# @param [Array] args the arguments to provide to the block
|
50
|
+
#
|
51
|
+
# @return [void] <description>
|
52
|
+
#
|
45
53
|
def dispatch(reflection, *args)
|
46
54
|
if (block = @reflections[reflection])
|
47
55
|
block.call(*args)
|
@@ -49,12 +57,15 @@ module SidekiqUniqueJobs
|
|
49
57
|
if DEPRECATIONS.key?(reflection)
|
50
58
|
replacement, removal_version = DEPRECATIONS[reflection]
|
51
59
|
SidekiqUniqueJobs::Deprecation.warn(
|
52
|
-
"#{reflection} is deprecated and will be removed in version #{removal_version}.
|
60
|
+
"#{reflection} is deprecated and will be removed in version #{removal_version}." \
|
61
|
+
" Use #{replacement} instead.",
|
53
62
|
)
|
54
63
|
end
|
55
64
|
elsif misconfigured?(reflection)
|
56
65
|
raise NoSuchNotificationError, reflection
|
57
66
|
end
|
67
|
+
|
68
|
+
nil
|
58
69
|
end
|
59
70
|
|
60
71
|
def configured?(reflection)
|
@@ -5,7 +5,7 @@ module SidekiqUniqueJobs
|
|
5
5
|
#
|
6
6
|
# @author Mikael Henriksson <mikael@mhenrixon.com>
|
7
7
|
class Server
|
8
|
-
DEATH_HANDLER
|
8
|
+
DEATH_HANDLER = (lambda do |job, _ex|
|
9
9
|
return unless (digest = job["lock_digest"])
|
10
10
|
|
11
11
|
SidekiqUniqueJobs::Digests.new.delete_by_digest(digest)
|
@@ -25,6 +25,12 @@ module SidekiqUniqueJobs
|
|
25
25
|
config.death_handlers << death_handler
|
26
26
|
end
|
27
27
|
|
28
|
+
#
|
29
|
+
# Start the sidekiq unique jobs server process
|
30
|
+
#
|
31
|
+
#
|
32
|
+
# @return [void]
|
33
|
+
#
|
28
34
|
def self.start
|
29
35
|
SidekiqUniqueJobs::UpdateVersion.call
|
30
36
|
SidekiqUniqueJobs::UpgradeLocks.call
|
@@ -32,6 +38,12 @@ module SidekiqUniqueJobs
|
|
32
38
|
SidekiqUniqueJobs::Orphans::ReaperResurrector.start
|
33
39
|
end
|
34
40
|
|
41
|
+
#
|
42
|
+
# Stop the sidekiq unique jobs server process
|
43
|
+
#
|
44
|
+
#
|
45
|
+
# @return [void]
|
46
|
+
#
|
35
47
|
def self.stop
|
36
48
|
SidekiqUniqueJobs::Orphans::Manager.stop
|
37
49
|
end
|
data/lib/tasks/changelog.rake
CHANGED
@@ -1,22 +1,22 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
2
|
|
3
|
+
# rubocop:disable Style/MutableConstant
|
4
|
+
CHANGELOG_CMD = %w[
|
5
|
+
github_changelog_generator
|
6
|
+
--no-verbose
|
7
|
+
--user
|
8
|
+
mhenrixon
|
9
|
+
--project
|
10
|
+
sidekiq-unique-jobs
|
11
|
+
--token
|
12
|
+
]
|
13
|
+
ADD_CHANGELOG_CMD = "git add --all"
|
14
|
+
COMMIT_CHANGELOG_CMD = "git commit -a -m 'Update changelog'"
|
15
|
+
# rubocop:enable Style/MutableConstant
|
16
|
+
|
3
17
|
desc "Generate a Changelog"
|
4
18
|
task :changelog do
|
5
|
-
|
6
|
-
CHANGELOG_CMD ||= %w[
|
7
|
-
github_changelog_generator
|
8
|
-
--no-verbose
|
9
|
-
--user
|
10
|
-
mhenrixon
|
11
|
-
--project
|
12
|
-
sidekiq-unique-jobs
|
13
|
-
--token
|
14
|
-
]
|
15
|
-
ADD_CHANGELOG_CMD ||= "git add --all"
|
16
|
-
COMMIT_CHANGELOG_CMD ||= "git commit -a -m 'Update changelog'"
|
17
|
-
# rubocop:enable Style/MutableConstant
|
18
|
-
|
19
|
-
sh("git checkout master")
|
19
|
+
sh("git checkout main")
|
20
20
|
sh(*CHANGELOG_CMD.push(ENV["CHANGELOG_GITHUB_TOKEN"]))
|
21
21
|
sh(ADD_CHANGELOG_CMD)
|
22
22
|
sh(COMMIT_CHANGELOG_CMD)
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: sidekiq-unique-jobs
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 7.1.
|
4
|
+
version: 7.1.11
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Mikael Henriksson
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2021-
|
11
|
+
date: 2021-11-30 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: brpoplpush-redis_script
|
@@ -59,7 +59,7 @@ dependencies:
|
|
59
59
|
version: '5.0'
|
60
60
|
- - "<"
|
61
61
|
- !ruby/object:Gem::Version
|
62
|
-
version: '
|
62
|
+
version: '8.0'
|
63
63
|
type: :runtime
|
64
64
|
prerelease: false
|
65
65
|
version_requirements: !ruby/object:Gem::Requirement
|
@@ -69,7 +69,7 @@ dependencies:
|
|
69
69
|
version: '5.0'
|
70
70
|
- - "<"
|
71
71
|
- !ruby/object:Gem::Version
|
72
|
-
version: '
|
72
|
+
version: '8.0'
|
73
73
|
- !ruby/object:Gem::Dependency
|
74
74
|
name: thor
|
75
75
|
requirement: !ruby/object:Gem::Requirement
|
@@ -79,7 +79,7 @@ dependencies:
|
|
79
79
|
version: '0.20'
|
80
80
|
- - "<"
|
81
81
|
- !ruby/object:Gem::Version
|
82
|
-
version: '
|
82
|
+
version: '3.0'
|
83
83
|
type: :runtime
|
84
84
|
prerelease: false
|
85
85
|
version_requirements: !ruby/object:Gem::Requirement
|
@@ -89,7 +89,7 @@ dependencies:
|
|
89
89
|
version: '0.20'
|
90
90
|
- - "<"
|
91
91
|
- !ruby/object:Gem::Version
|
92
|
-
version: '
|
92
|
+
version: '3.0'
|
93
93
|
description: |
|
94
94
|
Prevents simultaneous Sidekiq jobs with the same unique arguments to run.
|
95
95
|
Highly configurable to suite your specific needs.
|
@@ -215,15 +215,11 @@ homepage: https://mhenrixon.github.io/sidekiq-unique-jobs
|
|
215
215
|
licenses:
|
216
216
|
- MIT
|
217
217
|
metadata:
|
218
|
-
|
219
|
-
bug_tracker_uri: https://github.com/mhenrixon/sidekiq-unique-jobs/issues
|
220
|
-
documentation_uri: https://mhenrixon.github.io/sidekiq-unique-jobs
|
221
|
-
source_code_uri: https://github.com/mhenrixon/sidekiq-unique-jobs
|
222
|
-
changelog_uri: https://github.com/mhenrixon/sidekiq-unique-jobs/blob/master/CHANGELOG.md
|
218
|
+
rubygems_mfa_required: 'true'
|
223
219
|
post_install_message: |
|
224
220
|
IMPORTANT!
|
225
221
|
|
226
|
-
Automatic configuration of the sidekiq
|
222
|
+
Automatic configuration of the sidekiq middleware is no longer done.
|
227
223
|
Please see: https://github.com/mhenrixon/sidekiq-unique-jobs/blob/master/README.md#add-the-middleware
|
228
224
|
|
229
225
|
This version deprecated the following sidekiq_options
|
@@ -254,14 +250,14 @@ required_ruby_version: !ruby/object:Gem::Requirement
|
|
254
250
|
requirements:
|
255
251
|
- - ">="
|
256
252
|
- !ruby/object:Gem::Version
|
257
|
-
version: 2.5
|
253
|
+
version: '2.5'
|
258
254
|
required_rubygems_version: !ruby/object:Gem::Requirement
|
259
255
|
requirements:
|
260
256
|
- - ">="
|
261
257
|
- !ruby/object:Gem::Version
|
262
258
|
version: '0'
|
263
259
|
requirements: []
|
264
|
-
rubygems_version: 3.2.
|
260
|
+
rubygems_version: 3.2.32
|
265
261
|
signing_key:
|
266
262
|
specification_version: 4
|
267
263
|
summary: Sidekiq middleware that prevents duplicates jobs
|