sidekiq-unique-jobs 8.0.6 → 8.0.8
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +52 -4
- data/README.md +1 -3
- data/lib/sidekiq_unique_jobs/batch_delete.rb +1 -1
- data/lib/sidekiq_unique_jobs/config.rb +4 -4
- data/lib/sidekiq_unique_jobs/constants.rb +1 -0
- data/lib/sidekiq_unique_jobs/job.rb +4 -4
- data/lib/sidekiq_unique_jobs/lock/base_lock.rb +1 -1
- data/lib/sidekiq_unique_jobs/lock/while_executing.rb +2 -0
- data/lib/sidekiq_unique_jobs/lock.rb +13 -10
- data/lib/sidekiq_unique_jobs/locksmith.rb +3 -3
- data/lib/sidekiq_unique_jobs/lua/unlock.lua +16 -9
- data/lib/sidekiq_unique_jobs/middleware/client.rb +1 -1
- data/lib/sidekiq_unique_jobs/middleware/server.rb +1 -1
- data/lib/sidekiq_unique_jobs/middleware.rb +5 -5
- data/lib/sidekiq_unique_jobs/orphans/reaper.rb +2 -1
- data/lib/sidekiq_unique_jobs/orphans/ruby_reaper.rb +26 -7
- data/lib/sidekiq_unique_jobs/redis/sorted_set.rb +8 -0
- data/lib/sidekiq_unique_jobs/script/script.rb +1 -6
- data/lib/sidekiq_unique_jobs/server.rb +2 -0
- data/lib/sidekiq_unique_jobs/sidekiq_unique_ext.rb +13 -35
- data/lib/sidekiq_unique_jobs/sidekiq_unique_jobs.rb +3 -3
- data/lib/sidekiq_unique_jobs/sidekiq_worker_methods.rb +3 -3
- data/lib/sidekiq_unique_jobs/testing.rb +4 -4
- data/lib/sidekiq_unique_jobs/version.rb +1 -1
- data/lib/sidekiq_unique_jobs/web/helpers.rb +3 -3
- data/lib/sidekiq_unique_jobs/web/views/changelogs.erb +1 -1
- data/lib/sidekiq_unique_jobs/web/views/lock.erb +2 -2
- data/lib/sidekiq_unique_jobs/web/views/locks.erb +1 -1
- data/lib/sidekiq_unique_jobs/web.rb +16 -16
- metadata +3 -3
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 3e0587400a796faf2d96ad2882bb2da1d39378d188ce9b257f845f1234acc4bf
|
4
|
+
data.tar.gz: 7bbd398cc60c7b99bd491b352804f1b3d57e6c02318ad82d40855739dcb30582
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 2efce8d2064bd4b572c4a1db8533514eff3ada446f939b68d437d3d72c7244169594a84c3ecd400a5aa90f9e0666ee88ef849590aea37c3471bb9b32631816dc
|
7
|
+
data.tar.gz: 01af8820641cf873f01379da41279db5a81b90d23465dde4a1eff81b21ce60b70688e6e28c59deadfba3088e1ad148d3987c4fbb8f9fdaf11cbb3e4da6331a25
|
data/CHANGELOG.md
CHANGED
@@ -1,5 +1,46 @@
|
|
1
1
|
# Changelog
|
2
2
|
|
3
|
+
## [v8.0.7](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v8.0.7) (2024-02-05)
|
4
|
+
|
5
|
+
[Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v8.0.6...v8.0.7)
|
6
|
+
|
7
|
+
**Implemented enhancements:**
|
8
|
+
|
9
|
+
- chore\(ci\): add test coverage from bropoplpush [\#828](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/828) ([mhenrixon](https://github.com/mhenrixon))
|
10
|
+
|
11
|
+
**Fixed bugs:**
|
12
|
+
|
13
|
+
- fix\(xss\): sanitize parameters [\#829](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/829) ([mhenrixon](https://github.com/mhenrixon))
|
14
|
+
|
15
|
+
**Closed issues:**
|
16
|
+
|
17
|
+
- No 'Changelog' link is being displayed on https://rubygems.org/gems/sidekiq-unique-jobs [\#825](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/825)
|
18
|
+
- Using rspec matcher and getting: `NameError: uninitialized constant SidekiqUniqueJobs::Lock::BaseLock::Validator` [\#741](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/741)
|
19
|
+
|
20
|
+
**Merged pull requests:**
|
21
|
+
|
22
|
+
- Fix Testing Instructions [\#827](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/827) ([jherdman](https://github.com/jherdman))
|
23
|
+
|
24
|
+
## [v8.0.6](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v8.0.6) (2024-01-24)
|
25
|
+
|
26
|
+
[Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v8.0.5...v8.0.6)
|
27
|
+
|
28
|
+
**Implemented enhancements:**
|
29
|
+
|
30
|
+
- Returning same job id [\#814](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/814)
|
31
|
+
- fix: various minor issues [\#826](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/826) ([mhenrixon](https://github.com/mhenrixon))
|
32
|
+
|
33
|
+
**Closed issues:**
|
34
|
+
|
35
|
+
- should respond to `has_valid_sidekiq_options?` [\#822](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/822)
|
36
|
+
- Reaper manager registration is subject to race conditions [\#801](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/801)
|
37
|
+
- `while_executing` with `on_conflict: :reschedule` Reschedule job after job execution [\#800](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/800)
|
38
|
+
- Large retry queue causes reaper to run too slow [\#759](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/759)
|
39
|
+
|
40
|
+
**Merged pull requests:**
|
41
|
+
|
42
|
+
- fix: skip unless reaper was registered [\#820](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/820) ([mhenrixon](https://github.com/mhenrixon))
|
43
|
+
|
3
44
|
## [v8.0.5](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v8.0.5) (2023-11-11)
|
4
45
|
|
5
46
|
[Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v8.0.4...v8.0.5)
|
@@ -15,7 +56,7 @@
|
|
15
56
|
|
16
57
|
## [v8.0.4](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v8.0.4) (2023-11-11)
|
17
58
|
|
18
|
-
[Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v7.1.
|
59
|
+
[Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v7.1.31...v8.0.4)
|
19
60
|
|
20
61
|
**Implemented enhancements:**
|
21
62
|
|
@@ -31,9 +72,6 @@
|
|
31
72
|
**Closed issues:**
|
32
73
|
|
33
74
|
- register\_reaper\_process nx: true crash Sidekiq on startup [\#817](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/817)
|
34
|
-
- Should client middleware also be added to the Sidekiq server config? [\#803](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/803)
|
35
|
-
- 8.0.1 Time on locks & changelog UI is incorrect/wrong [\#761](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/761)
|
36
|
-
- Job executed twice when reaper runs [\#738](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/738)
|
37
75
|
|
38
76
|
**Merged pull requests:**
|
39
77
|
|
@@ -42,6 +80,16 @@
|
|
42
80
|
- Handle strategy fallbacks properly [\#809](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/809) ([matejrisek](https://github.com/matejrisek))
|
43
81
|
- Fix CI status badge [\#802](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/802) ([tagliala](https://github.com/tagliala))
|
44
82
|
|
83
|
+
## [v7.1.31](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v7.1.31) (2023-08-28)
|
84
|
+
|
85
|
+
[Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v7.1.30...v7.1.31)
|
86
|
+
|
87
|
+
**Closed issues:**
|
88
|
+
|
89
|
+
- Should client middleware also be added to the Sidekiq server config? [\#803](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/803)
|
90
|
+
- 8.0.1 Time on locks & changelog UI is incorrect/wrong [\#761](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/761)
|
91
|
+
- Job executed twice when reaper runs [\#738](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/738)
|
92
|
+
|
45
93
|
## [v7.1.30](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v7.1.30) (2023-07-17)
|
46
94
|
|
47
95
|
[Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v8.0.3...v7.1.30)
|
data/README.md
CHANGED
@@ -582,8 +582,6 @@ end
|
|
582
582
|
#spec/workers/bad_worker_spec.rb
|
583
583
|
|
584
584
|
require "sidekiq_unique_jobs/testing"
|
585
|
-
#OR
|
586
|
-
require "sidekiq_unique_jobs/rspec/matchers"
|
587
585
|
|
588
586
|
RSpec.describe BadWorker do
|
589
587
|
specify { expect(described_class).to have_valid_sidekiq_options }
|
@@ -816,7 +814,7 @@ In my benchmarks deleting 1000 orphaned locks with lua performs around 65% faste
|
|
816
814
|
|
817
815
|
On the other hand if I increase it to 10 000 orphaned locks per cleanup (`reaper_count: 10_0000`) then redis starts throwing:
|
818
816
|
|
819
|
-
> BUSY Redis is busy running a script. You can only call SCRIPT KILL or SHUTDOWN NOSAVE. (
|
817
|
+
> BUSY Redis is busy running a script. You can only call SCRIPT KILL or SHUTDOWN NOSAVE. (RedisClient::CommandError)
|
820
818
|
|
821
819
|
If you want to disable the reaper set it to `:none`, `nil` or `false`. Actually, any value that isn't `:ruby` or `:lua` will disable the reaping.
|
822
820
|
|
@@ -67,10 +67,10 @@ module SidekiqUniqueJobs
|
|
67
67
|
# @return [Hash<Symbol, SidekiqUniqueJobs::Lock::BaseLock] all available default locks
|
68
68
|
LOCKS =
|
69
69
|
LOCKS_WHEN_BUSY.dup
|
70
|
-
|
71
|
-
|
72
|
-
|
73
|
-
|
70
|
+
.merge(LOCKS_WHILE_ENQUEUED.dup)
|
71
|
+
.merge(LOCKS_WITHOUT_UNLOCK.dup)
|
72
|
+
.merge(LOCKS_FROM_PUSH_TO_PROCESSED.dup)
|
73
|
+
.freeze
|
74
74
|
|
75
75
|
#
|
76
76
|
# @return [Hash<Symbol, SidekiqUniqueJobs::OnConflict::Strategy] all available default strategies
|
@@ -41,19 +41,19 @@ module SidekiqUniqueJobs
|
|
41
41
|
end
|
42
42
|
|
43
43
|
def add_lock_timeout(item)
|
44
|
-
item[LOCK_TIMEOUT]
|
44
|
+
item[LOCK_TIMEOUT] = SidekiqUniqueJobs::LockTimeout.calculate(item)
|
45
45
|
end
|
46
46
|
|
47
47
|
def add_lock_args(item)
|
48
|
-
item[LOCK_ARGS]
|
48
|
+
item[LOCK_ARGS] = SidekiqUniqueJobs::LockArgs.call(item)
|
49
49
|
end
|
50
50
|
|
51
51
|
def add_lock_digest(item)
|
52
|
-
item[LOCK_DIGEST]
|
52
|
+
item[LOCK_DIGEST] = SidekiqUniqueJobs::LockDigest.call(item)
|
53
53
|
end
|
54
54
|
|
55
55
|
def add_lock_prefix(item)
|
56
|
-
item[LOCK_PREFIX]
|
56
|
+
item[LOCK_PREFIX] = SidekiqUniqueJobs.config.lock_prefix
|
57
57
|
end
|
58
58
|
|
59
59
|
def add_lock_type(item)
|
@@ -109,7 +109,7 @@ module SidekiqUniqueJobs
|
|
109
109
|
#
|
110
110
|
# Call whatever strategry that has been configured
|
111
111
|
#
|
112
|
-
# @param [Symbol] origin
|
112
|
+
# @param [Symbol] origin the origin `:client` or `:server`
|
113
113
|
#
|
114
114
|
# @return [void] the return value is irrelevant
|
115
115
|
#
|
@@ -33,9 +33,9 @@ module SidekiqUniqueJobs
|
|
33
33
|
#
|
34
34
|
# @return [Lock] a newly lock that has been locked
|
35
35
|
#
|
36
|
-
def self.create(digest, job_id, lock_info
|
37
|
-
lock = new(digest, time:
|
38
|
-
lock.lock(job_id, lock_info)
|
36
|
+
def self.create(digest, job_id, lock_info: {}, time: Timing.now_f, score: nil)
|
37
|
+
lock = new(digest, time: time)
|
38
|
+
lock.lock(job_id, lock_info, score)
|
39
39
|
lock
|
40
40
|
end
|
41
41
|
|
@@ -63,15 +63,16 @@ module SidekiqUniqueJobs
|
|
63
63
|
#
|
64
64
|
# @return [void]
|
65
65
|
#
|
66
|
-
def lock(job_id, lock_info = {})
|
66
|
+
def lock(job_id, lock_info = {}, score = nil)
|
67
|
+
score ||= now_f
|
67
68
|
redis do |conn|
|
68
69
|
conn.multi do |pipeline|
|
69
70
|
pipeline.set(key.digest, job_id)
|
70
71
|
pipeline.hset(key.locked, job_id, now_f)
|
71
72
|
info.set(lock_info, pipeline)
|
72
|
-
add_digest_to_set(pipeline, lock_info)
|
73
|
-
pipeline.zadd(key.changelog,
|
74
|
-
pipeline.zadd(key.changelog,
|
73
|
+
add_digest_to_set(pipeline, lock_info, score)
|
74
|
+
pipeline.zadd(key.changelog, score, changelog_json(job_id, "queue.lua", "Queued"))
|
75
|
+
pipeline.zadd(key.changelog, score, changelog_json(job_id, "lock.lua", "Locked"))
|
75
76
|
end
|
76
77
|
end
|
77
78
|
end
|
@@ -333,12 +334,14 @@ module SidekiqUniqueJobs
|
|
333
334
|
#
|
334
335
|
# @return [nil]
|
335
336
|
#
|
336
|
-
def add_digest_to_set(pipeline, lock_info)
|
337
|
+
def add_digest_to_set(pipeline, lock_info, score = nil)
|
338
|
+
score ||= now_f
|
337
339
|
digest_string = key.digest
|
340
|
+
|
338
341
|
if lock_info["lock"] == :until_expired
|
339
|
-
pipeline.zadd(key.expiring_digests,
|
342
|
+
pipeline.zadd(key.expiring_digests, score + lock_info["ttl"], digest_string)
|
340
343
|
else
|
341
|
-
pipeline.zadd(key.digests,
|
344
|
+
pipeline.zadd(key.digests, score, digest_string)
|
342
345
|
end
|
343
346
|
end
|
344
347
|
end
|
@@ -188,7 +188,7 @@ module SidekiqUniqueJobs
|
|
188
188
|
#
|
189
189
|
# @param [Sidekiq::RedisConnection, ConnectionPool] conn the redis connection
|
190
190
|
# @param [Method] primed_method reference to the method to use for getting a primed token
|
191
|
-
# @param [nil, Integer, Float] time to wait before timeout
|
191
|
+
# @param [nil, Integer, Float] wait time to wait before timeout
|
192
192
|
#
|
193
193
|
# @yieldparam [string] job_id the sidekiq JID
|
194
194
|
# @yieldreturn [void] whatever the calling block returns
|
@@ -258,8 +258,8 @@ module SidekiqUniqueJobs
|
|
258
258
|
|
259
259
|
# NOTE: When debugging, change .value to .value!
|
260
260
|
primed_jid = Concurrent::Promises
|
261
|
-
|
262
|
-
|
261
|
+
.future(conn) { |red_con| pop_queued(red_con, timeout) }
|
262
|
+
.value
|
263
263
|
|
264
264
|
handle_primed(primed_jid, &block)
|
265
265
|
end
|
@@ -75,21 +75,28 @@ if lock_type ~= "until_expired" then
|
|
75
75
|
redis.call("HDEL", locked, job_id)
|
76
76
|
end
|
77
77
|
|
78
|
+
if redis.call("LLEN", primed) == 0 then
|
79
|
+
log_debug("UNLINK", primed)
|
80
|
+
redis.call("UNLINK", primed)
|
81
|
+
end
|
82
|
+
|
78
83
|
local locked_count = redis.call("HLEN", locked)
|
79
84
|
|
80
|
-
if locked_count
|
85
|
+
if locked_count < 1 then
|
81
86
|
log_debug("UNLINK", locked)
|
82
87
|
redis.call("UNLINK", locked)
|
83
88
|
end
|
84
89
|
|
85
|
-
if
|
86
|
-
|
87
|
-
|
88
|
-
|
89
|
-
|
90
|
-
|
91
|
-
|
92
|
-
|
90
|
+
if limit then
|
91
|
+
if limit <= 1 and locked_count <= 1 then
|
92
|
+
log_debug("ZREM", digests, digest)
|
93
|
+
redis.call("ZREM", digests, digest)
|
94
|
+
end
|
95
|
+
else
|
96
|
+
if locked_count <= 1 then
|
97
|
+
log_debug("ZREM", digests, digest)
|
98
|
+
redis.call("ZREM", digests, digest)
|
99
|
+
end
|
93
100
|
end
|
94
101
|
|
95
102
|
log_debug("LPUSH", queued, "1")
|
@@ -6,7 +6,7 @@ module SidekiqUniqueJobs
|
|
6
6
|
#
|
7
7
|
# @author Mikael Henriksson <mikael@mhenrixon.com>
|
8
8
|
class Client
|
9
|
-
include Sidekiq::ClientMiddleware
|
9
|
+
include Sidekiq::ClientMiddleware
|
10
10
|
|
11
11
|
# prepend "SidekiqUniqueJobs::Middleware"
|
12
12
|
# @!parse prepends SidekiqUniqueJobs::Middleware
|
@@ -6,7 +6,7 @@ module SidekiqUniqueJobs
|
|
6
6
|
#
|
7
7
|
# @author Mikael Henriksson <mikael@mhenrixon.com>
|
8
8
|
class Server
|
9
|
-
include Sidekiq::ServerMiddleware
|
9
|
+
include Sidekiq::ServerMiddleware
|
10
10
|
|
11
11
|
# prepend "SidekiqUniqueJobs::Middleware"
|
12
12
|
# @!parse prepends SidekiqUniqueJobs::Middleware
|
@@ -19,15 +19,15 @@ module SidekiqUniqueJobs
|
|
19
19
|
# This method runs before (prepended) the actual middleware implementation.
|
20
20
|
# This is done to reduce duplication
|
21
21
|
#
|
22
|
-
# @param [Sidekiq::
|
22
|
+
# @param [Sidekiq::Job] worker_class
|
23
23
|
# @param [Hash] item a sidekiq job hash
|
24
24
|
# @param [String] queue name of the queue
|
25
25
|
# @param [ConnectionPool] redis_pool only used for compatility reasons
|
26
26
|
#
|
27
|
-
# @return [yield<super>]
|
27
|
+
# @return [yield<super>] call the rest of the middleware stack
|
28
28
|
#
|
29
|
-
# @yieldparam [
|
30
|
-
# @yieldreturn [
|
29
|
+
# @yieldparam [void] if uniquejobs is disable
|
30
|
+
# @yieldreturn [void] delegate back to other sidekiq middleware
|
31
31
|
def call(worker_class, item, queue, redis_pool = nil)
|
32
32
|
@item = item
|
33
33
|
@queue = queue
|
@@ -35,7 +35,7 @@ module SidekiqUniqueJobs
|
|
35
35
|
self.job_class = worker_class
|
36
36
|
return yield if unique_disabled?
|
37
37
|
|
38
|
-
SidekiqUniqueJobs::Job.prepare(item)
|
38
|
+
SidekiqUniqueJobs::Job.prepare(item)
|
39
39
|
|
40
40
|
with_logging_context do
|
41
41
|
super
|
@@ -22,11 +22,12 @@ module SidekiqUniqueJobs
|
|
22
22
|
#
|
23
23
|
# @return [Hash<Symbol, SidekiqUniqueJobs::Orphans::Reaper] the current implementation of reapers
|
24
24
|
REAPERS = {
|
25
|
-
lua: SidekiqUniqueJobs::Orphans::
|
25
|
+
lua: SidekiqUniqueJobs::Orphans::RubyReaper,
|
26
26
|
ruby: SidekiqUniqueJobs::Orphans::RubyReaper,
|
27
27
|
none: SidekiqUniqueJobs::Orphans::NullReaper,
|
28
28
|
nil => SidekiqUniqueJobs::Orphans::NullReaper,
|
29
29
|
false => SidekiqUniqueJobs::Orphans::NullReaper,
|
30
|
+
true => SidekiqUniqueJobs::Orphans::RubyReaper,
|
30
31
|
}.freeze
|
31
32
|
|
32
33
|
#
|
@@ -13,6 +13,9 @@ module SidekiqUniqueJobs
|
|
13
13
|
class RubyReaper < Reaper
|
14
14
|
include SidekiqUniqueJobs::Timing
|
15
15
|
|
16
|
+
#
|
17
|
+
# @return [Integer] a best guess of Sidekiq::Launcher::BEAT_PAUSE
|
18
|
+
SIDEKIQ_BEAT_PAUSE = 10
|
16
19
|
#
|
17
20
|
# @return [String] the suffix for :RUN locks
|
18
21
|
RUN_SUFFIX = ":RUN"
|
@@ -74,14 +77,30 @@ module SidekiqUniqueJobs
|
|
74
77
|
|
75
78
|
BatchDelete.call(expired_digests, conn)
|
76
79
|
BatchDelete.call(orphans, conn)
|
80
|
+
|
81
|
+
# orphans.each_slice(500) do |chunk|
|
82
|
+
# conn.pipelined do |pipeline|
|
83
|
+
# chunk.each do |digest|
|
84
|
+
# next if belongs_to_job?(digest)
|
85
|
+
|
86
|
+
# pipeline.zadd(ORPHANED_DIGESTS, now_f, digest)
|
87
|
+
# end
|
88
|
+
# end
|
89
|
+
# end
|
77
90
|
end
|
78
91
|
|
79
92
|
def expired_digests
|
80
|
-
max_score = (start_time - reaper_timeout).to_f
|
81
|
-
|
82
93
|
conn.zrange(EXPIRING_DIGESTS, 0, max_score, "byscore")
|
83
94
|
end
|
84
95
|
|
96
|
+
def orphaned_digests
|
97
|
+
conn.zrange(ORPHANED_DIGESTS, 0, max_score, "byscore")
|
98
|
+
end
|
99
|
+
|
100
|
+
def max_score
|
101
|
+
(start_time - reaper_timeout - SIDEKIQ_BEAT_PAUSE).to_f
|
102
|
+
end
|
103
|
+
|
85
104
|
#
|
86
105
|
# Find orphaned digests
|
87
106
|
#
|
@@ -89,10 +108,10 @@ module SidekiqUniqueJobs
|
|
89
108
|
# @return [Array<String>] an array of orphaned digests
|
90
109
|
#
|
91
110
|
def orphans # rubocop:disable Metrics/MethodLength, Metrics/CyclomaticComplexity
|
92
|
-
page = 0
|
93
|
-
per = reaper_count * 2
|
94
111
|
orphans = []
|
95
|
-
|
112
|
+
page = 0
|
113
|
+
per = reaper_count * 2
|
114
|
+
results = digests.byscore(0, max_score, offset: page * per, count: (page + 1) * per)
|
96
115
|
|
97
116
|
while results.size.positive?
|
98
117
|
results.each do |digest|
|
@@ -107,7 +126,7 @@ module SidekiqUniqueJobs
|
|
107
126
|
break if orphans.size >= reaper_count
|
108
127
|
|
109
128
|
page += 1
|
110
|
-
results =
|
129
|
+
results = digests.byscore(0, max_score, offset: page * per, count: (page + 1) * per)
|
111
130
|
end
|
112
131
|
|
113
132
|
orphans
|
@@ -218,7 +237,7 @@ module SidekiqUniqueJobs
|
|
218
237
|
end
|
219
238
|
|
220
239
|
def considered_active?(time_f)
|
221
|
-
|
240
|
+
max_score < time_f
|
222
241
|
end
|
223
242
|
|
224
243
|
#
|
@@ -48,6 +48,14 @@ module SidekiqUniqueJobs
|
|
48
48
|
end
|
49
49
|
end
|
50
50
|
|
51
|
+
def byscore(min, max, offset: nil, count: nil)
|
52
|
+
redis do |conn|
|
53
|
+
return conn.zrange(key, min, max, "byscore") unless offset && count
|
54
|
+
|
55
|
+
conn.zrange(key, min, max, "byscore", "limit", offset, count)
|
56
|
+
end
|
57
|
+
end
|
58
|
+
|
51
59
|
#
|
52
60
|
# Return the zrak of the member
|
53
61
|
#
|
@@ -5,6 +5,8 @@ module SidekiqUniqueJobs
|
|
5
5
|
#
|
6
6
|
# @author Mikael Henriksson <mikael@mhenrixon.com>
|
7
7
|
class Server
|
8
|
+
#
|
9
|
+
# @return [Proc] returns a default death handler for the gem to cleanup dead locks
|
8
10
|
DEATH_HANDLER = (lambda do |job, _ex|
|
9
11
|
return unless (digest = job["lock_digest"])
|
10
12
|
|
@@ -68,46 +68,24 @@ module Sidekiq
|
|
68
68
|
prepend UniqueExtension
|
69
69
|
end
|
70
70
|
|
71
|
-
|
72
|
-
|
73
|
-
|
74
|
-
|
75
|
-
|
71
|
+
# See Sidekiq::Api
|
72
|
+
class JobRecord
|
73
|
+
#
|
74
|
+
# Provides extensions for unlocking jobs that are removed and deleted
|
75
|
+
#
|
76
|
+
# @author Mikael Henriksson <mikael@mhenrixon.com>
|
77
|
+
#
|
78
|
+
module UniqueExtension
|
76
79
|
#
|
77
|
-
#
|
80
|
+
# Wraps the original method to ensure locks for the job are deleted
|
78
81
|
#
|
79
|
-
|
80
|
-
|
81
|
-
|
82
|
-
#
|
83
|
-
def delete
|
84
|
-
SidekiqUniqueJobs::Unlockable.delete!(item)
|
85
|
-
super
|
86
|
-
end
|
82
|
+
def delete
|
83
|
+
SidekiqUniqueJobs::Unlockable.delete!(item)
|
84
|
+
super
|
87
85
|
end
|
88
|
-
|
89
|
-
prepend UniqueExtension
|
90
86
|
end
|
91
|
-
else
|
92
|
-
# See Sidekiq::Api
|
93
|
-
class Job
|
94
|
-
#
|
95
|
-
# Provides extensions for unlocking jobs that are removed and deleted
|
96
|
-
#
|
97
|
-
# @author Mikael Henriksson <mikael@mhenrixon.com>
|
98
|
-
#
|
99
|
-
module UniqueExtension
|
100
|
-
#
|
101
|
-
# Wraps the original method to ensure locks for the job are deleted
|
102
|
-
#
|
103
|
-
def delete
|
104
|
-
SidekiqUniqueJobs::Unlockable.delete!(item)
|
105
|
-
super
|
106
|
-
end
|
107
|
-
end
|
108
87
|
|
109
|
-
|
110
|
-
end
|
88
|
+
prepend UniqueExtension
|
111
89
|
end
|
112
90
|
|
113
91
|
# See Sidekiq::Api
|
@@ -250,9 +250,9 @@ module SidekiqUniqueJobs # rubocop:disable Metrics/ModuleLength
|
|
250
250
|
# Attempt to constantize a string worker_class argument, always
|
251
251
|
# failing back to the original argument when the constant can't be found
|
252
252
|
#
|
253
|
-
# @return [Sidekiq::
|
253
|
+
# @return [Sidekiq::Job]
|
254
254
|
def constantize(str)
|
255
|
-
return str.class if str.is_a?(Sidekiq::
|
255
|
+
return str.class if str.is_a?(Sidekiq::Job) # sidekiq v6.x
|
256
256
|
return str unless str.is_a?(String)
|
257
257
|
return Object.const_get(str) unless str.include?("::")
|
258
258
|
|
@@ -269,7 +269,7 @@ module SidekiqUniqueJobs # rubocop:disable Metrics/ModuleLength
|
|
269
269
|
# Attempt to constantize a string worker_class argument, always
|
270
270
|
# failing back to the original argument when the constant can't be found
|
271
271
|
#
|
272
|
-
# @return [Sidekiq::
|
272
|
+
# @return [Sidekiq::Job, String]
|
273
273
|
def safe_constantize(str)
|
274
274
|
constantize(str)
|
275
275
|
rescue NameError => ex
|
@@ -1,13 +1,13 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
2
|
|
3
3
|
module SidekiqUniqueJobs
|
4
|
-
# Module with convenience methods for the Sidekiq::
|
4
|
+
# Module with convenience methods for the Sidekiq::Job class
|
5
5
|
#
|
6
6
|
# @author Mikael Henriksson <mikael@mhenrixon.com>
|
7
7
|
module SidekiqWorkerMethods
|
8
8
|
#
|
9
9
|
# @!attribute [r] job_class
|
10
|
-
# @return [Sidekiq::
|
10
|
+
# @return [Sidekiq::Job] The Sidekiq::Job implementation
|
11
11
|
attr_reader :job_class
|
12
12
|
|
13
13
|
# Avoids duplicating worker_class.respond_to? in multiple places
|
@@ -62,7 +62,7 @@ module SidekiqUniqueJobs
|
|
62
62
|
# Attempt to constantize a string worker_class argument, always
|
63
63
|
# failing back to the original argument when the constant can't be found
|
64
64
|
#
|
65
|
-
# @return [Sidekiq::
|
65
|
+
# @return [Sidekiq::Job]
|
66
66
|
def job_class_constantize(klazz = @job_class)
|
67
67
|
SidekiqUniqueJobs.safe_constantize(klazz)
|
68
68
|
end
|
@@ -42,11 +42,11 @@ module Sidekiq
|
|
42
42
|
end
|
43
43
|
|
44
44
|
#
|
45
|
-
# See Sidekiq::
|
45
|
+
# See Sidekiq::Job in Sidekiq gem for more details
|
46
46
|
#
|
47
47
|
module Worker
|
48
48
|
#
|
49
|
-
# Adds class methods to Sidekiq::
|
49
|
+
# Adds class methods to Sidekiq::Job
|
50
50
|
#
|
51
51
|
module ClassMethods
|
52
52
|
#
|
@@ -110,14 +110,14 @@ module Sidekiq
|
|
110
110
|
prepend Overrides
|
111
111
|
|
112
112
|
#
|
113
|
-
# Prepends methods to Sidekiq::
|
113
|
+
# Prepends methods to Sidekiq::Job
|
114
114
|
#
|
115
115
|
module ClassMethods
|
116
116
|
prepend Overrides::ClassMethods
|
117
117
|
end
|
118
118
|
|
119
119
|
#
|
120
|
-
# Prepends singleton methods to Sidekiq::
|
120
|
+
# Prepends singleton methods to Sidekiq::Job
|
121
121
|
#
|
122
122
|
module SignletonOverrides
|
123
123
|
#
|
@@ -133,7 +133,7 @@ module SidekiqUniqueJobs
|
|
133
133
|
#
|
134
134
|
# @return [String] a html safe string with relative time information
|
135
135
|
#
|
136
|
-
def
|
136
|
+
def _relative_time(time)
|
137
137
|
stamp = time.getutc.iso8601
|
138
138
|
%(<time class="ltr" dir="ltr" title="#{stamp}" datetime="#{stamp}">#{time}</time>)
|
139
139
|
end
|
@@ -145,12 +145,12 @@ module SidekiqUniqueJobs
|
|
145
145
|
#
|
146
146
|
# @return [String] a html safe string with relative time information
|
147
147
|
#
|
148
|
-
def
|
148
|
+
def _safe_relative_time(time)
|
149
149
|
return unless time
|
150
150
|
|
151
151
|
time = parse_time(time)
|
152
152
|
|
153
|
-
|
153
|
+
_relative_time(time)
|
154
154
|
end
|
155
155
|
|
156
156
|
#
|
@@ -42,7 +42,7 @@
|
|
42
42
|
<tbody>
|
43
43
|
<% @changelogs.each do |changelog| %>
|
44
44
|
<tr class="changelog-row">
|
45
|
-
<td><%=
|
45
|
+
<td><%= _safe_relative_time(changelog['time']) || "bogus" %></td>
|
46
46
|
<td><%= changelog["digest"] %></td>
|
47
47
|
<td><%= changelog["script"] %></td>
|
48
48
|
<td><%= changelog["job_id"] %></td>
|
@@ -65,7 +65,7 @@
|
|
65
65
|
<% @lock.locked_jids(with_values: true).each do |job_id, time| %>
|
66
66
|
<tr>
|
67
67
|
<td><%= job_id %></td>
|
68
|
-
<td><%=
|
68
|
+
<td><%= _safe_relative_time(time.to_f) %></td>
|
69
69
|
<td>
|
70
70
|
<form action="<%= root_path %>locks/<%= @lock.key %>/jobs/<%= job_id %>/delete" method="get">
|
71
71
|
<%= csrf_tag %>
|
@@ -92,7 +92,7 @@
|
|
92
92
|
<tbody>
|
93
93
|
<% @lock.changelogs.each do |entry| %>
|
94
94
|
<tr>
|
95
|
-
<td scope="row"><%=
|
95
|
+
<td scope="row"><%= _safe_relative_time(entry["time"].to_f) %></td>
|
96
96
|
<td><%= entry["job_id"] %></td>
|
97
97
|
<td><%= entry["message"] %></td>
|
98
98
|
<td><%= entry["script"] %></td>
|
@@ -46,7 +46,7 @@
|
|
46
46
|
<td><a href="<%= root_path %>locks/<%= lock.key %>"><%= lock.key %></a></td>
|
47
47
|
<td><%= lock.info["lock"] %></td>
|
48
48
|
<td><%= lock.locked.count %></td>
|
49
|
-
<td><%=
|
49
|
+
<td><%= _safe_relative_time(lock.created_at) %></td>
|
50
50
|
</tr>
|
51
51
|
</tbody>
|
52
52
|
<% end %>
|
@@ -13,11 +13,11 @@ module SidekiqUniqueJobs
|
|
13
13
|
end
|
14
14
|
|
15
15
|
app.get "/changelogs" do
|
16
|
-
@filter = params[:filter] || "*"
|
16
|
+
@filter = h(params[:filter] || "*")
|
17
17
|
@filter = "*" if @filter == ""
|
18
|
-
@count = (params[:count] || 100).to_i
|
19
|
-
@current_cursor = params[:cursor].to_i
|
20
|
-
@prev_cursor = params[:prev_cursor].to_i
|
18
|
+
@count = h(params[:count] || 100).to_i
|
19
|
+
@current_cursor = h(params[:cursor]).to_i
|
20
|
+
@prev_cursor = h(params[:prev_cursor]).to_i
|
21
21
|
@total_size, @next_cursor, @changelogs = changelog.page(
|
22
22
|
cursor: @current_cursor,
|
23
23
|
pattern: @filter,
|
@@ -33,11 +33,11 @@ module SidekiqUniqueJobs
|
|
33
33
|
end
|
34
34
|
|
35
35
|
app.get "/locks" do
|
36
|
-
@filter = params[:filter] || "*"
|
36
|
+
@filter = h(params[:filter]) || "*"
|
37
37
|
@filter = "*" if @filter == ""
|
38
|
-
@count = (params[:count] || 100).to_i
|
39
|
-
@current_cursor = params[:cursor].to_i
|
40
|
-
@prev_cursor = params[:prev_cursor].to_i
|
38
|
+
@count = h(params[:count] || 100).to_i
|
39
|
+
@current_cursor = h(params[:cursor]).to_i
|
40
|
+
@prev_cursor = h(params[:prev_cursor]).to_i
|
41
41
|
|
42
42
|
@total_size, @next_cursor, @locks = digests.page(
|
43
43
|
cursor: @current_cursor,
|
@@ -49,11 +49,11 @@ module SidekiqUniqueJobs
|
|
49
49
|
end
|
50
50
|
|
51
51
|
app.get "/expiring_locks" do
|
52
|
-
@filter = params[:filter] || "*"
|
52
|
+
@filter = h(params[:filter]) || "*"
|
53
53
|
@filter = "*" if @filter == ""
|
54
|
-
@count = (params[:count] || 100).to_i
|
55
|
-
@current_cursor = params[:cursor].to_i
|
56
|
-
@prev_cursor = params[:prev_cursor].to_i
|
54
|
+
@count = h(params[:count] || 100).to_i
|
55
|
+
@current_cursor = h(params[:cursor]).to_i
|
56
|
+
@prev_cursor = h(params[:prev_cursor]).to_i
|
57
57
|
|
58
58
|
@total_size, @next_cursor, @locks = expiring_digests.page(
|
59
59
|
cursor: @current_cursor,
|
@@ -71,20 +71,20 @@ module SidekiqUniqueJobs
|
|
71
71
|
end
|
72
72
|
|
73
73
|
app.get "/locks/:digest" do
|
74
|
-
@digest = params[:digest]
|
74
|
+
@digest = h(params[:digest])
|
75
75
|
@lock = SidekiqUniqueJobs::Lock.new(@digest)
|
76
76
|
|
77
77
|
erb(unique_template(:lock))
|
78
78
|
end
|
79
79
|
|
80
80
|
app.get "/locks/:digest/delete" do
|
81
|
-
digests.delete_by_digest(params[:digest])
|
82
|
-
expiring_digests.delete_by_digest(params[:digest])
|
81
|
+
digests.delete_by_digest(h(params[:digest]))
|
82
|
+
expiring_digests.delete_by_digest(h(params[:digest]))
|
83
83
|
redirect_to :locks
|
84
84
|
end
|
85
85
|
|
86
86
|
app.get "/locks/:digest/jobs/:job_id/delete" do
|
87
|
-
@digest = params[:digest]
|
87
|
+
@digest = h(params[:digest])
|
88
88
|
@lock = SidekiqUniqueJobs::Lock.new(@digest)
|
89
89
|
@lock.unlock(params[:job_id])
|
90
90
|
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: sidekiq-unique-jobs
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 8.0.
|
4
|
+
version: 8.0.8
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Mikael Henriksson
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2024-
|
11
|
+
date: 2024-02-12 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: concurrent-ruby
|
@@ -223,7 +223,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
223
223
|
- !ruby/object:Gem::Version
|
224
224
|
version: '0'
|
225
225
|
requirements: []
|
226
|
-
rubygems_version: 3.5.
|
226
|
+
rubygems_version: 3.5.6
|
227
227
|
signing_key:
|
228
228
|
specification_version: 4
|
229
229
|
summary: Sidekiq middleware that prevents duplicates jobs
|