sidekiq-unique-jobs 7.1.20 → 7.1.30
Sign up to get free protection for your applications and to get access to all the features.
Potentially problematic release.
This version of sidekiq-unique-jobs might be problematic. Click here for more details.
- checksums.yaml +4 -4
- data/CHANGELOG.md +135 -1
- data/README.md +4 -2
- data/lib/sidekiq_unique_jobs/batch_delete.rb +2 -1
- data/lib/sidekiq_unique_jobs/cli.rb +33 -8
- data/lib/sidekiq_unique_jobs/config.rb +5 -0
- data/lib/sidekiq_unique_jobs/constants.rb +1 -0
- data/lib/sidekiq_unique_jobs/core_ext.rb +1 -1
- data/lib/sidekiq_unique_jobs/digests.rb +2 -2
- data/lib/sidekiq_unique_jobs/exceptions.rb +3 -3
- data/lib/sidekiq_unique_jobs/expiring_digests.rb +14 -0
- data/lib/sidekiq_unique_jobs/job.rb +5 -0
- data/lib/sidekiq_unique_jobs/key.rb +13 -8
- data/lib/sidekiq_unique_jobs/lock/until_executed.rb +1 -0
- data/lib/sidekiq_unique_jobs/lock/until_executing.rb +4 -0
- data/lib/sidekiq_unique_jobs/lock/while_executing.rb +1 -2
- data/lib/sidekiq_unique_jobs/lock.rb +18 -1
- data/lib/sidekiq_unique_jobs/lock_args.rb +18 -14
- data/lib/sidekiq_unique_jobs/lock_config.rb +4 -4
- data/lib/sidekiq_unique_jobs/lock_digest.rb +7 -7
- data/lib/sidekiq_unique_jobs/lock_timeout.rb +4 -4
- data/lib/sidekiq_unique_jobs/lock_ttl.rb +4 -4
- data/lib/sidekiq_unique_jobs/lock_type.rb +37 -0
- data/lib/sidekiq_unique_jobs/locksmith.rb +40 -10
- data/lib/sidekiq_unique_jobs/logging.rb +14 -0
- data/lib/sidekiq_unique_jobs/lua/lock.lua +15 -9
- data/lib/sidekiq_unique_jobs/lua/lock_until_expired.lua +92 -0
- data/lib/sidekiq_unique_jobs/lua/reap_orphans.lua +31 -3
- data/lib/sidekiq_unique_jobs/lua/shared/_find_digest_in_process_set.lua +1 -1
- data/lib/sidekiq_unique_jobs/lua/unlock.lua +5 -0
- data/lib/sidekiq_unique_jobs/middleware/client.rb +2 -0
- data/lib/sidekiq_unique_jobs/middleware/server.rb +2 -0
- data/lib/sidekiq_unique_jobs/middleware.rb +4 -4
- data/lib/sidekiq_unique_jobs/on_conflict/reschedule.rb +3 -3
- data/lib/sidekiq_unique_jobs/options_with_fallback.rb +4 -4
- data/lib/sidekiq_unique_jobs/orphans/lua_reaper.rb +1 -1
- data/lib/sidekiq_unique_jobs/orphans/manager.rb +2 -2
- data/lib/sidekiq_unique_jobs/orphans/ruby_reaper.rb +30 -8
- data/lib/sidekiq_unique_jobs/script/caller.rb +7 -7
- data/lib/sidekiq_unique_jobs/sidekiq_unique_jobs.rb +13 -3
- data/lib/sidekiq_unique_jobs/sidekiq_worker_methods.rb +33 -24
- data/lib/sidekiq_unique_jobs/testing.rb +31 -13
- data/lib/sidekiq_unique_jobs/version.rb +1 -1
- data/lib/sidekiq_unique_jobs/web/helpers.rb +10 -0
- data/lib/sidekiq_unique_jobs/web/views/lock.erb +5 -3
- data/lib/sidekiq_unique_jobs/web.rb +22 -3
- data/lib/sidekiq_unique_jobs.rb +2 -0
- metadata +22 -5
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 8fec809cda45f395d8eed686c8b2ac69d59c332d04dc1a4968d66aa03b521c04
|
4
|
+
data.tar.gz: da85dc0ef4b155d3baec392d11230f7f59955ee617c1459cfaba4455e1cd4a17
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 6df2fccd762fe81f71ce9fa20d3e413ff2b4147bac2d10f2bcfbe479d420b895fe888050f8f3f980bfeee693448c052d12e9c78b5ecd524c96277e29d4277685
|
7
|
+
data.tar.gz: c60f92fdd29a4f37f5903776108f11e0c3ea1e04c5d2462700d5ee01ad57d1cdd5b20fa725ed662f95c873663ce04d932f6c90267b0cea631095285f57324de3
|
data/CHANGELOG.md
CHANGED
@@ -1,5 +1,140 @@
|
|
1
1
|
# Changelog
|
2
2
|
|
3
|
+
## [Unreleased](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/HEAD)
|
4
|
+
|
5
|
+
[Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v7.1.28...HEAD)
|
6
|
+
|
7
|
+
**Fixed bugs:**
|
8
|
+
|
9
|
+
- fix\(digests\): ensure consistent digests [\#743](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/743) ([mhenrixon](https://github.com/mhenrixon))
|
10
|
+
|
11
|
+
**Merged pull requests:**
|
12
|
+
|
13
|
+
- fix\(after\_unlock\): regression from \#707 [\#737](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/737) ([adamcreekroad](https://github.com/adamcreekroad))
|
14
|
+
|
15
|
+
## [v7.1.28](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v7.1.28) (2022-11-28)
|
16
|
+
|
17
|
+
[Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v7.1.27...v7.1.28)
|
18
|
+
|
19
|
+
**Fixed bugs:**
|
20
|
+
|
21
|
+
- Unique Jobs Not Running with Version 7.1.26 [\#730](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/730)
|
22
|
+
|
23
|
+
**Closed issues:**
|
24
|
+
|
25
|
+
- Error "undefined method `redis\_info' for Sidekiq:Module" on upgrade [\#740](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/740)
|
26
|
+
- spammed by `Nothing to delete; exiting` during spec [\#733](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/733)
|
27
|
+
|
28
|
+
**Merged pull requests:**
|
29
|
+
|
30
|
+
- sentence correction [\#744](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/744) ([SupriyaMedankar](https://github.com/SupriyaMedankar))
|
31
|
+
|
32
|
+
## [v7.1.27](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v7.1.27) (2022-07-30)
|
33
|
+
|
34
|
+
[Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v7.1.26...v7.1.27)
|
35
|
+
|
36
|
+
**Implemented enhancements:**
|
37
|
+
|
38
|
+
- Feat\(logging\): Allow disabling logging [\#729](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/729) ([mhenrixon](https://github.com/mhenrixon))
|
39
|
+
|
40
|
+
**Fixed bugs:**
|
41
|
+
|
42
|
+
- Fix\(namespace\): Prevent self-conflict when redis-namespace is present [\#732](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/732) ([mhenrixon](https://github.com/mhenrixon))
|
43
|
+
|
44
|
+
**Closed issues:**
|
45
|
+
|
46
|
+
- Disable logging in Rails testing [\#727](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/727)
|
47
|
+
- Memory bloat / dangling keys / reaper not cleaning orphans [\#637](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/637)
|
48
|
+
|
49
|
+
## [v7.1.26](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v7.1.26) (2022-07-28)
|
50
|
+
|
51
|
+
[Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v7.1.25...v7.1.26)
|
52
|
+
|
53
|
+
**Implemented enhancements:**
|
54
|
+
|
55
|
+
- Fix\(until\_expired\): Fix test and implementation [\#725](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/725) ([mhenrixon](https://github.com/mhenrixon))
|
56
|
+
|
57
|
+
**Fixed bugs:**
|
58
|
+
|
59
|
+
- Fix\(until\_and\_while\_executing\): Improve timeouts slightly [\#728](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/728) ([mhenrixon](https://github.com/mhenrixon))
|
60
|
+
- Fix\(unlock\): Delete primed keys on last entry [\#726](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/726) ([mhenrixon](https://github.com/mhenrixon))
|
61
|
+
|
62
|
+
**Merged pull requests:**
|
63
|
+
|
64
|
+
- Ensure batch delete removes expiring locks [\#724](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/724) ([francesmcmullin](https://github.com/francesmcmullin))
|
65
|
+
- Chore: Update dependencies [\#722](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/722) ([mhenrixon](https://github.com/mhenrixon))
|
66
|
+
- Move until\_expired digests to separate zset [\#721](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/721) ([francesmcmullin](https://github.com/francesmcmullin))
|
67
|
+
- Avoid skipping ranges when looping through queues [\#720](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/720) ([francesmcmullin](https://github.com/francesmcmullin))
|
68
|
+
- Bump actions/checkout from 2 to 3 [\#718](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/718) ([dependabot[bot]](https://github.com/apps/dependabot))
|
69
|
+
- Add Dependabot for GitHub Actions [\#717](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/717) ([petergoldstein](https://github.com/petergoldstein))
|
70
|
+
- Fix Sidekiq::Worker.clear\_all override not being applied [\#714](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/714) ([dsander](https://github.com/dsander))
|
71
|
+
|
72
|
+
## [v7.1.25](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v7.1.25) (2022-06-13)
|
73
|
+
|
74
|
+
[Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v7.1.24...v7.1.25)
|
75
|
+
|
76
|
+
**Fixed bugs:**
|
77
|
+
|
78
|
+
- Fix: Include the correct middleware [\#716](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/716) ([mhenrixon](https://github.com/mhenrixon))
|
79
|
+
|
80
|
+
## [v7.1.24](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v7.1.24) (2022-06-09)
|
81
|
+
|
82
|
+
[Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v7.1.23...v7.1.24)
|
83
|
+
|
84
|
+
**Implemented enhancements:**
|
85
|
+
|
86
|
+
- Chore: Sidekiq 6.5 compatibility [\#715](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/715) ([mhenrixon](https://github.com/mhenrixon))
|
87
|
+
|
88
|
+
**Merged pull requests:**
|
89
|
+
|
90
|
+
- Use sidekiq/testing `Worker.clear` API in sidekiq\_unique\_jobs/testing [\#713](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/713) ([dsander](https://github.com/dsander))
|
91
|
+
|
92
|
+
## [v7.1.23](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v7.1.23) (2022-05-23)
|
93
|
+
|
94
|
+
[Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v7.1.22...v7.1.23)
|
95
|
+
|
96
|
+
**Fixed bugs:**
|
97
|
+
|
98
|
+
- fix: raise on error [\#712](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/712) ([mhenrixon](https://github.com/mhenrixon))
|
99
|
+
|
100
|
+
## [v7.1.22](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v7.1.22) (2022-05-04)
|
101
|
+
|
102
|
+
[Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v7.1.21...v7.1.22)
|
103
|
+
|
104
|
+
**Fixed bugs:**
|
105
|
+
|
106
|
+
- Failed jobs waiting to be retried are not considered when fetching uniqueness [\#394](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/394)
|
107
|
+
- fix\(locksmith\): execute to yield without arguments [\#710](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/710) ([mhenrixon](https://github.com/mhenrixon))
|
108
|
+
- fix: re:lock until\_executing on worker failure [\#709](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/709) ([mhenrixon](https://github.com/mhenrixon))
|
109
|
+
|
110
|
+
**Closed issues:**
|
111
|
+
|
112
|
+
- Reviwing: Failed jobs waiting to be retried are not considered when fetching uniqueness [\#708](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/708)
|
113
|
+
|
114
|
+
## [v7.1.21](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v7.1.21) (2022-04-23)
|
115
|
+
|
116
|
+
[Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v7.1.20...v7.1.21)
|
117
|
+
|
118
|
+
**Implemented enhancements:**
|
119
|
+
|
120
|
+
- Prepare for Sidekiq v7 [\#707](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/707) ([mhenrixon](https://github.com/mhenrixon))
|
121
|
+
|
122
|
+
**Closed issues:**
|
123
|
+
|
124
|
+
- DEPRECATION WARNING: default\_worker\_options is deprecated and will be removed from Sidekiq 7.0 \(use default\_job\_options instead\) \(called from notify\_agents at /Users/hackeron/Development/Tether/timeline/app/models/user.rb:303\) [\#705](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/705)
|
125
|
+
|
126
|
+
## [v7.1.20](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v7.1.20) (2022-04-22)
|
127
|
+
|
128
|
+
[Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v7.1.19...v7.1.20)
|
129
|
+
|
130
|
+
**Implemented enhancements:**
|
131
|
+
|
132
|
+
- Manually handle timeouts [\#706](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/706) ([mhenrixon](https://github.com/mhenrixon))
|
133
|
+
|
134
|
+
**Merged pull requests:**
|
135
|
+
|
136
|
+
- improve README wrt. middleware config [\#704](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/704) ([slhck](https://github.com/slhck))
|
137
|
+
|
3
138
|
## [v7.1.19](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v7.1.19) (2022-04-09)
|
4
139
|
|
5
140
|
[Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v7.1.18...v7.1.19)
|
@@ -383,7 +518,6 @@
|
|
383
518
|
- Tasks run once, and then there is no launch [\#464](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/464)
|
384
519
|
- Jobs executing and immediately returning [\#418](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/418)
|
385
520
|
- until\_and\_while\_executing + sidekiq retry mechanism [\#395](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/395)
|
386
|
-
- Failed jobs waiting to be retried are not considered when fetching uniqueness [\#394](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/394)
|
387
521
|
- Fix that :PRIMED keys are seemingly not removed [\#574](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/574) ([mhenrixon](https://github.com/mhenrixon))
|
388
522
|
|
389
523
|
**Closed issues:**
|
data/README.md
CHANGED
@@ -31,7 +31,7 @@ Want to show me some ❤️ for the hard work I do on this gem? You can use the
|
|
31
31
|
- [raise](#raise)
|
32
32
|
- [reject](#reject)
|
33
33
|
- [replace](#replace)
|
34
|
-
- [
|
34
|
+
- [reschedule](#reschedule)
|
35
35
|
- [Custom Strategies](#custom-strategies)
|
36
36
|
- [3 Cleanup Dead Locks](#3-cleanup-dead-locks)
|
37
37
|
- [Debugging](#debugging)
|
@@ -364,7 +364,7 @@ sidekiq_options lock: :until_and_while_executing,
|
|
364
364
|
sidekiq_options on_conflict: :log
|
365
365
|
```
|
366
366
|
|
367
|
-
This strategy is intended to be used with `UntilExecuted` and `UntilExpired`. It will log a line
|
367
|
+
This strategy is intended to be used with `UntilExecuted` and `UntilExpired`. It will log a line that this job is a duplicate of another.
|
368
368
|
|
369
369
|
### raise
|
370
370
|
|
@@ -610,6 +610,7 @@ This has been probably the most confusing part of this gem. People get really co
|
|
610
610
|
```ruby
|
611
611
|
SidekiqUniqueJobs.configure do |config|
|
612
612
|
config.enabled = !Rails.env.test?
|
613
|
+
config.logger_enabled = !Rails.env.test?
|
613
614
|
end
|
614
615
|
```
|
615
616
|
|
@@ -736,6 +737,7 @@ Configure SidekiqUniqueJobs in an initializer or the sidekiq initializer on appl
|
|
736
737
|
```ruby
|
737
738
|
SidekiqUniqueJobs.configure do |config|
|
738
739
|
config.logger = Sidekiq.logger # default, change at your own discretion
|
740
|
+
config.logger_enabled = true # default, disable for test environments
|
739
741
|
config.debug_lua = false # Turn on when debugging
|
740
742
|
config.lock_info = false # Turn on when debugging
|
741
743
|
config.lock_ttl = 600 # Expire locks after 10 minutes
|
@@ -91,6 +91,7 @@ module SidekiqUniqueJobs
|
|
91
91
|
chunk.each do |digest|
|
92
92
|
del_digest(pipeline, digest)
|
93
93
|
pipeline.zrem(SidekiqUniqueJobs::DIGESTS, digest)
|
94
|
+
pipeline.zrem(SidekiqUniqueJobs::EXPIRING_DIGESTS, digest)
|
94
95
|
@count += 1
|
95
96
|
end
|
96
97
|
end
|
@@ -111,7 +112,7 @@ module SidekiqUniqueJobs
|
|
111
112
|
|
112
113
|
def keys_for_digest(digest)
|
113
114
|
[digest, "#{digest}:RUN"].each_with_object([]) do |key, digest_keys|
|
114
|
-
digest_keys.
|
115
|
+
digest_keys.push(key)
|
115
116
|
digest_keys.concat(SUFFIXES.map { |suffix| "#{key}:#{suffix}" })
|
116
117
|
end
|
117
118
|
end
|
@@ -20,9 +20,11 @@ module SidekiqUniqueJobs
|
|
20
20
|
option :count, aliases: :c, type: :numeric, default: 1000, desc: "The max number of digests to return"
|
21
21
|
# :nodoc:
|
22
22
|
def list(pattern = "*")
|
23
|
-
|
24
|
-
say "
|
25
|
-
|
23
|
+
max_count = options[:count]
|
24
|
+
say "Searching for regular digests"
|
25
|
+
list_entries(digests.entries(pattern: pattern, count: max_count), pattern)
|
26
|
+
say "Searching for expiring digests"
|
27
|
+
list_entries(expiring_digests.entries(pattern: pattern, count: max_count), pattern)
|
26
28
|
end
|
27
29
|
|
28
30
|
desc "del PATTERN", "deletes unique digests from redis by pattern"
|
@@ -32,11 +34,9 @@ module SidekiqUniqueJobs
|
|
32
34
|
def del(pattern)
|
33
35
|
max_count = options[:count]
|
34
36
|
if options[:dry_run]
|
35
|
-
|
36
|
-
say "Would delete #{result.size} digests matching '#{pattern}'"
|
37
|
+
count_entries_for_del(max_count, pattern)
|
37
38
|
else
|
38
|
-
|
39
|
-
say "Deleted #{deleted_count} digests matching '#{pattern}'"
|
39
|
+
del_entries(max_count, pattern)
|
40
40
|
end
|
41
41
|
end
|
42
42
|
|
@@ -51,12 +51,17 @@ module SidekiqUniqueJobs
|
|
51
51
|
console_class.start
|
52
52
|
end
|
53
53
|
|
54
|
-
no_commands do
|
54
|
+
no_commands do # rubocop:disable Metrics/BlockLength
|
55
55
|
# :nodoc:
|
56
56
|
def digests
|
57
57
|
@digests ||= SidekiqUniqueJobs::Digests.new
|
58
58
|
end
|
59
59
|
|
60
|
+
# :nodoc:
|
61
|
+
def expiring_digests
|
62
|
+
@expiring_digests ||= SidekiqUniqueJobs::ExpiringDigests.new
|
63
|
+
end
|
64
|
+
|
60
65
|
# :nodoc:
|
61
66
|
def console_class
|
62
67
|
require "pry"
|
@@ -65,6 +70,26 @@ module SidekiqUniqueJobs
|
|
65
70
|
require "irb"
|
66
71
|
IRB
|
67
72
|
end
|
73
|
+
|
74
|
+
# :nodoc:
|
75
|
+
def list_entries(entries, pattern)
|
76
|
+
say "Found #{entries.size} digests matching '#{pattern}':"
|
77
|
+
print_in_columns(entries.sort) if entries.any?
|
78
|
+
end
|
79
|
+
|
80
|
+
# :nodoc:
|
81
|
+
def count_entries_for_del(max_count, pattern)
|
82
|
+
count = digests.entries(pattern: pattern, count: max_count).size +
|
83
|
+
expiring_digests.entries(pattern: pattern, count: max_count).size
|
84
|
+
say "Would delete #{count} digests matching '#{pattern}'"
|
85
|
+
end
|
86
|
+
|
87
|
+
# :nodoc:
|
88
|
+
def del_entries(max_count, pattern)
|
89
|
+
deleted_count = digests.delete_by_pattern(pattern, count: max_count).to_i +
|
90
|
+
expiring_digests.delete_by_pattern(pattern, count: max_count).to_i
|
91
|
+
say "Deleted #{deleted_count} digests matching '#{pattern}'"
|
92
|
+
end
|
68
93
|
end
|
69
94
|
end
|
70
95
|
end
|
@@ -8,6 +8,7 @@ module SidekiqUniqueJobs
|
|
8
8
|
:enabled,
|
9
9
|
:lock_prefix,
|
10
10
|
:logger,
|
11
|
+
:logger_enabled,
|
11
12
|
:locks,
|
12
13
|
:strategies,
|
13
14
|
:debug_lua,
|
@@ -91,6 +92,9 @@ module SidekiqUniqueJobs
|
|
91
92
|
# @return [nil]
|
92
93
|
LOCK_TTL = nil
|
93
94
|
#
|
95
|
+
# @return [true,false] by default false (don't disable logger)
|
96
|
+
LOGGER_ENABLED = true
|
97
|
+
#
|
94
98
|
# @return [true] by default the gem is enabled
|
95
99
|
ENABLED = true
|
96
100
|
#
|
@@ -180,6 +184,7 @@ module SidekiqUniqueJobs
|
|
180
184
|
ENABLED,
|
181
185
|
PREFIX,
|
182
186
|
Sidekiq.logger,
|
187
|
+
LOGGER_ENABLED,
|
183
188
|
LOCKS,
|
184
189
|
STRATEGIES,
|
185
190
|
DEBUG_LUA,
|
@@ -98,7 +98,7 @@ class Hash
|
|
98
98
|
def _deep_transform_keys_in_object(object, &block)
|
99
99
|
case object
|
100
100
|
when Hash
|
101
|
-
object.each_with_object(
|
101
|
+
object.each_with_object(self.class.new) do |(key, value), result|
|
102
102
|
result[yield(key)] = _deep_transform_keys_in_object(value, &block)
|
103
103
|
end
|
104
104
|
when Array
|
@@ -72,14 +72,14 @@ module SidekiqUniqueJobs
|
|
72
72
|
class InvalidUniqueArguments < UniqueJobsError
|
73
73
|
def initialize(options)
|
74
74
|
given = options[:given]
|
75
|
-
|
75
|
+
job_class = options[:job_class]
|
76
76
|
lock_args_method = options[:lock_args_method]
|
77
|
-
lock_args_meth =
|
77
|
+
lock_args_meth = job_class.method(lock_args_method)
|
78
78
|
num_args = lock_args_meth.arity
|
79
79
|
source_location = lock_args_meth.source_location
|
80
80
|
|
81
81
|
super(
|
82
|
-
"#{
|
82
|
+
"#{job_class}##{lock_args_method} takes #{num_args} arguments, received #{given.inspect}" \
|
83
83
|
"\n\n" \
|
84
84
|
" #{source_location.join(':')}"
|
85
85
|
)
|
@@ -0,0 +1,14 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
module SidekiqUniqueJobs
|
4
|
+
#
|
5
|
+
# Class ExpiringDigests provides access to the expiring digests used by until_expired locks
|
6
|
+
#
|
7
|
+
# @author Mikael Henriksson <mikael@mhenrixon.com>
|
8
|
+
#
|
9
|
+
class ExpiringDigests < Digests
|
10
|
+
def initialize
|
11
|
+
super(EXPIRING_DIGESTS)
|
12
|
+
end
|
13
|
+
end
|
14
|
+
end
|
@@ -11,6 +11,7 @@ module SidekiqUniqueJobs
|
|
11
11
|
# @return [Hash] the job hash
|
12
12
|
def prepare(item)
|
13
13
|
stringify_on_conflict_hash(item)
|
14
|
+
add_lock_type(item)
|
14
15
|
add_lock_timeout(item)
|
15
16
|
add_lock_ttl(item)
|
16
17
|
add_digest(item)
|
@@ -54,5 +55,9 @@ module SidekiqUniqueJobs
|
|
54
55
|
def add_lock_prefix(item)
|
55
56
|
item[LOCK_PREFIX] ||= SidekiqUniqueJobs.config.lock_prefix
|
56
57
|
end
|
58
|
+
|
59
|
+
def add_lock_type(item)
|
60
|
+
item[LOCK] ||= SidekiqUniqueJobs::LockType.call(item)
|
61
|
+
end
|
57
62
|
end
|
58
63
|
end
|
@@ -33,6 +33,10 @@ module SidekiqUniqueJobs
|
|
33
33
|
# @!attribute [r] digests
|
34
34
|
# @return [String] the zset with locked digests
|
35
35
|
attr_reader :digests
|
36
|
+
#
|
37
|
+
# @!attribute [r] expiring_digests
|
38
|
+
# @return [String] the zset with locked expiring_digests
|
39
|
+
attr_reader :expiring_digests
|
36
40
|
|
37
41
|
#
|
38
42
|
# Initialize a new Key
|
@@ -40,13 +44,14 @@ module SidekiqUniqueJobs
|
|
40
44
|
# @param [String] digest the digest to use as key
|
41
45
|
#
|
42
46
|
def initialize(digest)
|
43
|
-
@digest
|
44
|
-
@queued
|
45
|
-
@primed
|
46
|
-
@locked
|
47
|
-
@info
|
48
|
-
@changelog
|
49
|
-
@digests
|
47
|
+
@digest = digest
|
48
|
+
@queued = suffixed_key("QUEUED")
|
49
|
+
@primed = suffixed_key("PRIMED")
|
50
|
+
@locked = suffixed_key("LOCKED")
|
51
|
+
@info = suffixed_key("INFO")
|
52
|
+
@changelog = CHANGELOGS
|
53
|
+
@digests = DIGESTS
|
54
|
+
@expiring_digests = EXPIRING_DIGESTS
|
50
55
|
end
|
51
56
|
|
52
57
|
#
|
@@ -81,7 +86,7 @@ module SidekiqUniqueJobs
|
|
81
86
|
# @return [Array] an ordered array with all keys
|
82
87
|
#
|
83
88
|
def to_a
|
84
|
-
[digest, queued, primed, locked, info, changelog, digests]
|
89
|
+
[digest, queued, primed, locked, info, changelog, digests, expiring_digests]
|
85
90
|
end
|
86
91
|
|
87
92
|
private
|
@@ -66,7 +66,7 @@ module SidekiqUniqueJobs
|
|
66
66
|
pipeline.set(key.digest, job_id)
|
67
67
|
pipeline.hset(key.locked, job_id, now_f)
|
68
68
|
info.set(lock_info, pipeline)
|
69
|
-
pipeline
|
69
|
+
add_digest_to_set(pipeline, lock_info)
|
70
70
|
pipeline.zadd(key.changelog, now_f, changelog_json(job_id, "queue.lua", "Queued"))
|
71
71
|
pipeline.zadd(key.changelog, now_f, changelog_json(job_id, "lock.lua", "Locked"))
|
72
72
|
end
|
@@ -321,5 +321,22 @@ module SidekiqUniqueJobs
|
|
321
321
|
time: now_f,
|
322
322
|
)
|
323
323
|
end
|
324
|
+
|
325
|
+
#
|
326
|
+
# Add the digest to the correct sorted set
|
327
|
+
#
|
328
|
+
# @param [Object] pipeline a redis pipeline object for issue commands
|
329
|
+
# @param [Hash] lock_info the lock info relevant to the digest
|
330
|
+
#
|
331
|
+
# @return [nil]
|
332
|
+
#
|
333
|
+
def add_digest_to_set(pipeline, lock_info)
|
334
|
+
digest_string = key.digest
|
335
|
+
if lock_info["lock"] == :until_expired
|
336
|
+
pipeline.zadd(key.expiring_digests, now_f + lock_info["ttl"], digest_string)
|
337
|
+
else
|
338
|
+
pipeline.zadd(key.digests, now_f, digest_string)
|
339
|
+
end
|
340
|
+
end
|
324
341
|
end
|
325
342
|
end
|
@@ -26,9 +26,9 @@ module SidekiqUniqueJobs
|
|
26
26
|
|
27
27
|
# @param [Hash] item a Sidekiq job hash
|
28
28
|
def initialize(item)
|
29
|
-
@item
|
30
|
-
@
|
31
|
-
|
29
|
+
@item = item
|
30
|
+
@args = item[ARGS]
|
31
|
+
self.job_class = item[CLASS]
|
32
32
|
end
|
33
33
|
|
34
34
|
# The unique arguments to use for creating a lock
|
@@ -83,31 +83,31 @@ module SidekiqUniqueJobs
|
|
83
83
|
|
84
84
|
# Filters unique arguments by method configured in the sidekiq worker
|
85
85
|
# @param [Array] args the arguments passed to the sidekiq worker
|
86
|
-
# @return [Array] unfiltered unless {#
|
86
|
+
# @return [Array] unfiltered unless {#job_method_defined?}
|
87
87
|
# @return [Array] with the filtered arguments
|
88
88
|
def filter_by_symbol(args)
|
89
|
-
return args unless
|
89
|
+
return args unless job_method_defined?(lock_args_method)
|
90
90
|
|
91
|
-
|
91
|
+
job_class.send(lock_args_method, args)
|
92
92
|
rescue ArgumentError
|
93
93
|
raise SidekiqUniqueJobs::InvalidUniqueArguments,
|
94
94
|
given: args,
|
95
|
-
|
95
|
+
job_class: job_class,
|
96
96
|
lock_args_method: lock_args_method
|
97
97
|
end
|
98
98
|
|
99
99
|
# The method to use for filtering unique arguments
|
100
100
|
def lock_args_method
|
101
|
-
@lock_args_method ||=
|
102
|
-
@lock_args_method ||= :lock_args if
|
103
|
-
@lock_args_method ||= :unique_args if
|
101
|
+
@lock_args_method ||= job_options.slice(LOCK_ARGS_METHOD, UNIQUE_ARGS_METHOD).values.first
|
102
|
+
@lock_args_method ||= :lock_args if job_method_defined?(:lock_args)
|
103
|
+
@lock_args_method ||= :unique_args if job_method_defined?(:unique_args)
|
104
104
|
@lock_args_method ||= default_lock_args_method
|
105
105
|
end
|
106
106
|
|
107
107
|
# The global worker options defined in Sidekiq directly
|
108
108
|
def default_lock_args_method
|
109
|
-
|
110
|
-
|
109
|
+
default_job_options[LOCK_ARGS_METHOD] ||
|
110
|
+
default_job_options[UNIQUE_ARGS_METHOD]
|
111
111
|
end
|
112
112
|
|
113
113
|
#
|
@@ -116,8 +116,12 @@ module SidekiqUniqueJobs
|
|
116
116
|
#
|
117
117
|
# @return [Hash<String, Object>]
|
118
118
|
#
|
119
|
-
def
|
120
|
-
@
|
119
|
+
def default_job_options
|
120
|
+
@default_job_options ||= if Sidekiq.respond_to?(:default_job_options)
|
121
|
+
Sidekiq.default_job_options.stringify_keys
|
122
|
+
else
|
123
|
+
Sidekiq.default_worker_options.stringify_keys
|
124
|
+
end
|
121
125
|
end
|
122
126
|
end
|
123
127
|
end
|
@@ -13,9 +13,9 @@ module SidekiqUniqueJobs
|
|
13
13
|
# @return [Symbol] the type of lock
|
14
14
|
attr_reader :type
|
15
15
|
#
|
16
|
-
# @!attribute [r]
|
17
|
-
# @return [Symbol] the
|
18
|
-
attr_reader :
|
16
|
+
# @!attribute [r] job
|
17
|
+
# @return [Symbol] the job class
|
18
|
+
attr_reader :job
|
19
19
|
#
|
20
20
|
# @!attribute [r] limit
|
21
21
|
# @return [Integer] the number of simultaneous locks
|
@@ -58,7 +58,7 @@ module SidekiqUniqueJobs
|
|
58
58
|
|
59
59
|
def initialize(job_hash = {})
|
60
60
|
@type = job_hash[LOCK]&.to_sym
|
61
|
-
@
|
61
|
+
@job = SidekiqUniqueJobs.safe_constantize(job_hash[CLASS])
|
62
62
|
@limit = job_hash.fetch(LOCK_LIMIT, 1)&.to_i
|
63
63
|
@timeout = job_hash.fetch(LOCK_TIMEOUT, 0)&.to_i
|
64
64
|
@ttl = job_hash.fetch(LOCK_TTL) { job_hash.fetch(LOCK_EXPIRATION, nil) }.to_i
|
@@ -36,10 +36,10 @@ module SidekiqUniqueJobs
|
|
36
36
|
|
37
37
|
# @param [Hash] item a Sidekiq job hash
|
38
38
|
def initialize(item)
|
39
|
-
@item
|
40
|
-
@
|
41
|
-
@
|
42
|
-
|
39
|
+
@item = item
|
40
|
+
@lock_args = item[LOCK_ARGS] || item[UNIQUE_ARGS] # TODO: Deprecate UNIQUE_ARGS
|
41
|
+
@lock_prefix = item[LOCK_PREFIX] || item[UNIQUE_PREFIX] # TODO: Deprecate UNIQUE_PREFIX
|
42
|
+
self.job_class = item[CLASS]
|
43
43
|
end
|
44
44
|
|
45
45
|
# Memoized lock_digest
|
@@ -51,7 +51,7 @@ module SidekiqUniqueJobs
|
|
51
51
|
# Creates a namespaced unique digest based on the {#digestable_hash} and the {#lock_prefix}
|
52
52
|
# @return [String] a unique digest
|
53
53
|
def create_digest
|
54
|
-
digest = OpenSSL::Digest::MD5.hexdigest(dump_json(digestable_hash))
|
54
|
+
digest = OpenSSL::Digest::MD5.hexdigest(dump_json(digestable_hash.sort))
|
55
55
|
"#{lock_prefix}:#{digest}"
|
56
56
|
end
|
57
57
|
|
@@ -67,13 +67,13 @@ module SidekiqUniqueJobs
|
|
67
67
|
# Checks if we should disregard the queue when creating the unique digest
|
68
68
|
# @return [true, false]
|
69
69
|
def unique_across_queues?
|
70
|
-
item[UNIQUE_ACROSS_QUEUES] ||
|
70
|
+
item[UNIQUE_ACROSS_QUEUES] || job_options[UNIQUE_ACROSS_QUEUES]
|
71
71
|
end
|
72
72
|
|
73
73
|
# Checks if we should disregard the worker when creating the unique digest
|
74
74
|
# @return [true, false]
|
75
75
|
def unique_across_workers?
|
76
|
-
item[UNIQUE_ACROSS_WORKERS] ||
|
76
|
+
item[UNIQUE_ACROSS_WORKERS] || job_options[UNIQUE_ACROSS_WORKERS]
|
77
77
|
end
|
78
78
|
end
|
79
79
|
end
|
@@ -30,8 +30,8 @@ module SidekiqUniqueJobs
|
|
30
30
|
# @option item [String] :class the class of the sidekiq worker
|
31
31
|
# @option item [Float] :at the unix time the job is scheduled at
|
32
32
|
def initialize(item)
|
33
|
-
@item
|
34
|
-
|
33
|
+
@item = item
|
34
|
+
self.job_class = item[CLASS]
|
35
35
|
end
|
36
36
|
|
37
37
|
#
|
@@ -42,9 +42,9 @@ module SidekiqUniqueJobs
|
|
42
42
|
# @return [Integer, nil]
|
43
43
|
#
|
44
44
|
def calculate
|
45
|
-
timeout =
|
45
|
+
timeout = default_job_options[LOCK_TIMEOUT]
|
46
46
|
timeout = default_lock_timeout if default_lock_timeout
|
47
|
-
timeout =
|
47
|
+
timeout = job_options[LOCK_TIMEOUT] if job_options.key?(LOCK_TIMEOUT)
|
48
48
|
timeout
|
49
49
|
end
|
50
50
|
|