sidekiq-unique-jobs 7.1.20 → 7.1.30

Sign up to get free protection for your applications and to get access to all the features.

Potentially problematic release.


This version of sidekiq-unique-jobs might be problematic. Click here for more details.

Files changed (48) hide show
  1. checksums.yaml +4 -4
  2. data/CHANGELOG.md +135 -1
  3. data/README.md +4 -2
  4. data/lib/sidekiq_unique_jobs/batch_delete.rb +2 -1
  5. data/lib/sidekiq_unique_jobs/cli.rb +33 -8
  6. data/lib/sidekiq_unique_jobs/config.rb +5 -0
  7. data/lib/sidekiq_unique_jobs/constants.rb +1 -0
  8. data/lib/sidekiq_unique_jobs/core_ext.rb +1 -1
  9. data/lib/sidekiq_unique_jobs/digests.rb +2 -2
  10. data/lib/sidekiq_unique_jobs/exceptions.rb +3 -3
  11. data/lib/sidekiq_unique_jobs/expiring_digests.rb +14 -0
  12. data/lib/sidekiq_unique_jobs/job.rb +5 -0
  13. data/lib/sidekiq_unique_jobs/key.rb +13 -8
  14. data/lib/sidekiq_unique_jobs/lock/until_executed.rb +1 -0
  15. data/lib/sidekiq_unique_jobs/lock/until_executing.rb +4 -0
  16. data/lib/sidekiq_unique_jobs/lock/while_executing.rb +1 -2
  17. data/lib/sidekiq_unique_jobs/lock.rb +18 -1
  18. data/lib/sidekiq_unique_jobs/lock_args.rb +18 -14
  19. data/lib/sidekiq_unique_jobs/lock_config.rb +4 -4
  20. data/lib/sidekiq_unique_jobs/lock_digest.rb +7 -7
  21. data/lib/sidekiq_unique_jobs/lock_timeout.rb +4 -4
  22. data/lib/sidekiq_unique_jobs/lock_ttl.rb +4 -4
  23. data/lib/sidekiq_unique_jobs/lock_type.rb +37 -0
  24. data/lib/sidekiq_unique_jobs/locksmith.rb +40 -10
  25. data/lib/sidekiq_unique_jobs/logging.rb +14 -0
  26. data/lib/sidekiq_unique_jobs/lua/lock.lua +15 -9
  27. data/lib/sidekiq_unique_jobs/lua/lock_until_expired.lua +92 -0
  28. data/lib/sidekiq_unique_jobs/lua/reap_orphans.lua +31 -3
  29. data/lib/sidekiq_unique_jobs/lua/shared/_find_digest_in_process_set.lua +1 -1
  30. data/lib/sidekiq_unique_jobs/lua/unlock.lua +5 -0
  31. data/lib/sidekiq_unique_jobs/middleware/client.rb +2 -0
  32. data/lib/sidekiq_unique_jobs/middleware/server.rb +2 -0
  33. data/lib/sidekiq_unique_jobs/middleware.rb +4 -4
  34. data/lib/sidekiq_unique_jobs/on_conflict/reschedule.rb +3 -3
  35. data/lib/sidekiq_unique_jobs/options_with_fallback.rb +4 -4
  36. data/lib/sidekiq_unique_jobs/orphans/lua_reaper.rb +1 -1
  37. data/lib/sidekiq_unique_jobs/orphans/manager.rb +2 -2
  38. data/lib/sidekiq_unique_jobs/orphans/ruby_reaper.rb +30 -8
  39. data/lib/sidekiq_unique_jobs/script/caller.rb +7 -7
  40. data/lib/sidekiq_unique_jobs/sidekiq_unique_jobs.rb +13 -3
  41. data/lib/sidekiq_unique_jobs/sidekiq_worker_methods.rb +33 -24
  42. data/lib/sidekiq_unique_jobs/testing.rb +31 -13
  43. data/lib/sidekiq_unique_jobs/version.rb +1 -1
  44. data/lib/sidekiq_unique_jobs/web/helpers.rb +10 -0
  45. data/lib/sidekiq_unique_jobs/web/views/lock.erb +5 -3
  46. data/lib/sidekiq_unique_jobs/web.rb +22 -3
  47. data/lib/sidekiq_unique_jobs.rb +2 -0
  48. metadata +22 -5
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 2a112ea37ca98a7909abf3b536e5d5f2b147e3a40f7749bf3c7d819c259d069e
4
- data.tar.gz: a3f34c72a78723c0dd441d005ae0300662183ddbe24cde430e7714587feccbd3
3
+ metadata.gz: 8fec809cda45f395d8eed686c8b2ac69d59c332d04dc1a4968d66aa03b521c04
4
+ data.tar.gz: da85dc0ef4b155d3baec392d11230f7f59955ee617c1459cfaba4455e1cd4a17
5
5
  SHA512:
6
- metadata.gz: 0a6d73f2beade75f9b0261a65ff3be37dfe0fe35c6ff3550635d3b8c634f1ad80c646c36be63c304806ef4f4f471eb0bb56ad641714ed48a53a14ee1d9f1a1aa
7
- data.tar.gz: 93f518d9743445c8b844032dbcd8e583a88909b7e3bed9b88807ce3e3e337d2bdf101b8b413e8610705a22aae98747c8fa63549a2b001adc9800903577ad3694
6
+ metadata.gz: 6df2fccd762fe81f71ce9fa20d3e413ff2b4147bac2d10f2bcfbe479d420b895fe888050f8f3f980bfeee693448c052d12e9c78b5ecd524c96277e29d4277685
7
+ data.tar.gz: c60f92fdd29a4f37f5903776108f11e0c3ea1e04c5d2462700d5ee01ad57d1cdd5b20fa725ed662f95c873663ce04d932f6c90267b0cea631095285f57324de3
data/CHANGELOG.md CHANGED
@@ -1,5 +1,140 @@
1
1
  # Changelog
2
2
 
3
+ ## [Unreleased](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/HEAD)
4
+
5
+ [Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v7.1.28...HEAD)
6
+
7
+ **Fixed bugs:**
8
+
9
+ - fix\(digests\): ensure consistent digests [\#743](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/743) ([mhenrixon](https://github.com/mhenrixon))
10
+
11
+ **Merged pull requests:**
12
+
13
+ - fix\(after\_unlock\): regression from \#707 [\#737](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/737) ([adamcreekroad](https://github.com/adamcreekroad))
14
+
15
+ ## [v7.1.28](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v7.1.28) (2022-11-28)
16
+
17
+ [Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v7.1.27...v7.1.28)
18
+
19
+ **Fixed bugs:**
20
+
21
+ - Unique Jobs Not Running with Version 7.1.26 [\#730](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/730)
22
+
23
+ **Closed issues:**
24
+
25
+ - Error "undefined method `redis\_info' for Sidekiq:Module" on upgrade [\#740](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/740)
26
+ - spammed by `Nothing to delete; exiting` during spec [\#733](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/733)
27
+
28
+ **Merged pull requests:**
29
+
30
+ - sentence correction [\#744](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/744) ([SupriyaMedankar](https://github.com/SupriyaMedankar))
31
+
32
+ ## [v7.1.27](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v7.1.27) (2022-07-30)
33
+
34
+ [Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v7.1.26...v7.1.27)
35
+
36
+ **Implemented enhancements:**
37
+
38
+ - Feat\(logging\): Allow disabling logging [\#729](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/729) ([mhenrixon](https://github.com/mhenrixon))
39
+
40
+ **Fixed bugs:**
41
+
42
+ - Fix\(namespace\): Prevent self-conflict when redis-namespace is present [\#732](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/732) ([mhenrixon](https://github.com/mhenrixon))
43
+
44
+ **Closed issues:**
45
+
46
+ - Disable logging in Rails testing [\#727](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/727)
47
+ - Memory bloat / dangling keys / reaper not cleaning orphans [\#637](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/637)
48
+
49
+ ## [v7.1.26](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v7.1.26) (2022-07-28)
50
+
51
+ [Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v7.1.25...v7.1.26)
52
+
53
+ **Implemented enhancements:**
54
+
55
+ - Fix\(until\_expired\): Fix test and implementation [\#725](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/725) ([mhenrixon](https://github.com/mhenrixon))
56
+
57
+ **Fixed bugs:**
58
+
59
+ - Fix\(until\_and\_while\_executing\): Improve timeouts slightly [\#728](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/728) ([mhenrixon](https://github.com/mhenrixon))
60
+ - Fix\(unlock\): Delete primed keys on last entry [\#726](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/726) ([mhenrixon](https://github.com/mhenrixon))
61
+
62
+ **Merged pull requests:**
63
+
64
+ - Ensure batch delete removes expiring locks [\#724](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/724) ([francesmcmullin](https://github.com/francesmcmullin))
65
+ - Chore: Update dependencies [\#722](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/722) ([mhenrixon](https://github.com/mhenrixon))
66
+ - Move until\_expired digests to separate zset [\#721](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/721) ([francesmcmullin](https://github.com/francesmcmullin))
67
+ - Avoid skipping ranges when looping through queues [\#720](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/720) ([francesmcmullin](https://github.com/francesmcmullin))
68
+ - Bump actions/checkout from 2 to 3 [\#718](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/718) ([dependabot[bot]](https://github.com/apps/dependabot))
69
+ - Add Dependabot for GitHub Actions [\#717](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/717) ([petergoldstein](https://github.com/petergoldstein))
70
+ - Fix Sidekiq::Worker.clear\_all override not being applied [\#714](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/714) ([dsander](https://github.com/dsander))
71
+
72
+ ## [v7.1.25](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v7.1.25) (2022-06-13)
73
+
74
+ [Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v7.1.24...v7.1.25)
75
+
76
+ **Fixed bugs:**
77
+
78
+ - Fix: Include the correct middleware [\#716](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/716) ([mhenrixon](https://github.com/mhenrixon))
79
+
80
+ ## [v7.1.24](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v7.1.24) (2022-06-09)
81
+
82
+ [Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v7.1.23...v7.1.24)
83
+
84
+ **Implemented enhancements:**
85
+
86
+ - Chore: Sidekiq 6.5 compatibility [\#715](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/715) ([mhenrixon](https://github.com/mhenrixon))
87
+
88
+ **Merged pull requests:**
89
+
90
+ - Use sidekiq/testing `Worker.clear` API in sidekiq\_unique\_jobs/testing [\#713](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/713) ([dsander](https://github.com/dsander))
91
+
92
+ ## [v7.1.23](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v7.1.23) (2022-05-23)
93
+
94
+ [Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v7.1.22...v7.1.23)
95
+
96
+ **Fixed bugs:**
97
+
98
+ - fix: raise on error [\#712](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/712) ([mhenrixon](https://github.com/mhenrixon))
99
+
100
+ ## [v7.1.22](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v7.1.22) (2022-05-04)
101
+
102
+ [Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v7.1.21...v7.1.22)
103
+
104
+ **Fixed bugs:**
105
+
106
+ - Failed jobs waiting to be retried are not considered when fetching uniqueness [\#394](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/394)
107
+ - fix\(locksmith\): execute to yield without arguments [\#710](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/710) ([mhenrixon](https://github.com/mhenrixon))
108
+ - fix: re:lock until\_executing on worker failure [\#709](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/709) ([mhenrixon](https://github.com/mhenrixon))
109
+
110
+ **Closed issues:**
111
+
112
+ - Reviwing: Failed jobs waiting to be retried are not considered when fetching uniqueness [\#708](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/708)
113
+
114
+ ## [v7.1.21](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v7.1.21) (2022-04-23)
115
+
116
+ [Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v7.1.20...v7.1.21)
117
+
118
+ **Implemented enhancements:**
119
+
120
+ - Prepare for Sidekiq v7 [\#707](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/707) ([mhenrixon](https://github.com/mhenrixon))
121
+
122
+ **Closed issues:**
123
+
124
+ - DEPRECATION WARNING: default\_worker\_options is deprecated and will be removed from Sidekiq 7.0 \(use default\_job\_options instead\) \(called from notify\_agents at /Users/hackeron/Development/Tether/timeline/app/models/user.rb:303\) [\#705](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/705)
125
+
126
+ ## [v7.1.20](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v7.1.20) (2022-04-22)
127
+
128
+ [Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v7.1.19...v7.1.20)
129
+
130
+ **Implemented enhancements:**
131
+
132
+ - Manually handle timeouts [\#706](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/706) ([mhenrixon](https://github.com/mhenrixon))
133
+
134
+ **Merged pull requests:**
135
+
136
+ - improve README wrt. middleware config [\#704](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/704) ([slhck](https://github.com/slhck))
137
+
3
138
  ## [v7.1.19](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v7.1.19) (2022-04-09)
4
139
 
5
140
  [Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v7.1.18...v7.1.19)
@@ -383,7 +518,6 @@
383
518
  - Tasks run once, and then there is no launch [\#464](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/464)
384
519
  - Jobs executing and immediately returning [\#418](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/418)
385
520
  - until\_and\_while\_executing + sidekiq retry mechanism [\#395](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/395)
386
- - Failed jobs waiting to be retried are not considered when fetching uniqueness [\#394](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/394)
387
521
  - Fix that :PRIMED keys are seemingly not removed [\#574](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/574) ([mhenrixon](https://github.com/mhenrixon))
388
522
 
389
523
  **Closed issues:**
data/README.md CHANGED
@@ -31,7 +31,7 @@ Want to show me some ❤️ for the hard work I do on this gem? You can use the
31
31
  - [raise](#raise)
32
32
  - [reject](#reject)
33
33
  - [replace](#replace)
34
- - [Reschedule](#reschedule)
34
+ - [reschedule](#reschedule)
35
35
  - [Custom Strategies](#custom-strategies)
36
36
  - [3 Cleanup Dead Locks](#3-cleanup-dead-locks)
37
37
  - [Debugging](#debugging)
@@ -364,7 +364,7 @@ sidekiq_options lock: :until_and_while_executing,
364
364
  sidekiq_options on_conflict: :log
365
365
  ```
366
366
 
367
- This strategy is intended to be used with `UntilExecuted` and `UntilExpired`. It will log a line about that this is job is a duplicate of another.
367
+ This strategy is intended to be used with `UntilExecuted` and `UntilExpired`. It will log a line that this job is a duplicate of another.
368
368
 
369
369
  ### raise
370
370
 
@@ -610,6 +610,7 @@ This has been probably the most confusing part of this gem. People get really co
610
610
  ```ruby
611
611
  SidekiqUniqueJobs.configure do |config|
612
612
  config.enabled = !Rails.env.test?
613
+ config.logger_enabled = !Rails.env.test?
613
614
  end
614
615
  ```
615
616
 
@@ -736,6 +737,7 @@ Configure SidekiqUniqueJobs in an initializer or the sidekiq initializer on appl
736
737
  ```ruby
737
738
  SidekiqUniqueJobs.configure do |config|
738
739
  config.logger = Sidekiq.logger # default, change at your own discretion
740
+ config.logger_enabled = true # default, disable for test environments
739
741
  config.debug_lua = false # Turn on when debugging
740
742
  config.lock_info = false # Turn on when debugging
741
743
  config.lock_ttl = 600 # Expire locks after 10 minutes
@@ -91,6 +91,7 @@ module SidekiqUniqueJobs
91
91
  chunk.each do |digest|
92
92
  del_digest(pipeline, digest)
93
93
  pipeline.zrem(SidekiqUniqueJobs::DIGESTS, digest)
94
+ pipeline.zrem(SidekiqUniqueJobs::EXPIRING_DIGESTS, digest)
94
95
  @count += 1
95
96
  end
96
97
  end
@@ -111,7 +112,7 @@ module SidekiqUniqueJobs
111
112
 
112
113
  def keys_for_digest(digest)
113
114
  [digest, "#{digest}:RUN"].each_with_object([]) do |key, digest_keys|
114
- digest_keys.concat([key])
115
+ digest_keys.push(key)
115
116
  digest_keys.concat(SUFFIXES.map { |suffix| "#{key}:#{suffix}" })
116
117
  end
117
118
  end
@@ -20,9 +20,11 @@ module SidekiqUniqueJobs
20
20
  option :count, aliases: :c, type: :numeric, default: 1000, desc: "The max number of digests to return"
21
21
  # :nodoc:
22
22
  def list(pattern = "*")
23
- entries = digests.entries(pattern: pattern, count: options[:count])
24
- say "Found #{entries.size} digests matching '#{pattern}':"
25
- print_in_columns(entries.sort) if entries.any?
23
+ max_count = options[:count]
24
+ say "Searching for regular digests"
25
+ list_entries(digests.entries(pattern: pattern, count: max_count), pattern)
26
+ say "Searching for expiring digests"
27
+ list_entries(expiring_digests.entries(pattern: pattern, count: max_count), pattern)
26
28
  end
27
29
 
28
30
  desc "del PATTERN", "deletes unique digests from redis by pattern"
@@ -32,11 +34,9 @@ module SidekiqUniqueJobs
32
34
  def del(pattern)
33
35
  max_count = options[:count]
34
36
  if options[:dry_run]
35
- result = digests.entries(pattern: pattern, count: max_count)
36
- say "Would delete #{result.size} digests matching '#{pattern}'"
37
+ count_entries_for_del(max_count, pattern)
37
38
  else
38
- deleted_count = digests.delete_by_pattern(pattern, count: max_count)
39
- say "Deleted #{deleted_count} digests matching '#{pattern}'"
39
+ del_entries(max_count, pattern)
40
40
  end
41
41
  end
42
42
 
@@ -51,12 +51,17 @@ module SidekiqUniqueJobs
51
51
  console_class.start
52
52
  end
53
53
 
54
- no_commands do
54
+ no_commands do # rubocop:disable Metrics/BlockLength
55
55
  # :nodoc:
56
56
  def digests
57
57
  @digests ||= SidekiqUniqueJobs::Digests.new
58
58
  end
59
59
 
60
+ # :nodoc:
61
+ def expiring_digests
62
+ @expiring_digests ||= SidekiqUniqueJobs::ExpiringDigests.new
63
+ end
64
+
60
65
  # :nodoc:
61
66
  def console_class
62
67
  require "pry"
@@ -65,6 +70,26 @@ module SidekiqUniqueJobs
65
70
  require "irb"
66
71
  IRB
67
72
  end
73
+
74
+ # :nodoc:
75
+ def list_entries(entries, pattern)
76
+ say "Found #{entries.size} digests matching '#{pattern}':"
77
+ print_in_columns(entries.sort) if entries.any?
78
+ end
79
+
80
+ # :nodoc:
81
+ def count_entries_for_del(max_count, pattern)
82
+ count = digests.entries(pattern: pattern, count: max_count).size +
83
+ expiring_digests.entries(pattern: pattern, count: max_count).size
84
+ say "Would delete #{count} digests matching '#{pattern}'"
85
+ end
86
+
87
+ # :nodoc:
88
+ def del_entries(max_count, pattern)
89
+ deleted_count = digests.delete_by_pattern(pattern, count: max_count).to_i +
90
+ expiring_digests.delete_by_pattern(pattern, count: max_count).to_i
91
+ say "Deleted #{deleted_count} digests matching '#{pattern}'"
92
+ end
68
93
  end
69
94
  end
70
95
  end
@@ -8,6 +8,7 @@ module SidekiqUniqueJobs
8
8
  :enabled,
9
9
  :lock_prefix,
10
10
  :logger,
11
+ :logger_enabled,
11
12
  :locks,
12
13
  :strategies,
13
14
  :debug_lua,
@@ -91,6 +92,9 @@ module SidekiqUniqueJobs
91
92
  # @return [nil]
92
93
  LOCK_TTL = nil
93
94
  #
95
+ # @return [true,false] by default false (don't disable logger)
96
+ LOGGER_ENABLED = true
97
+ #
94
98
  # @return [true] by default the gem is enabled
95
99
  ENABLED = true
96
100
  #
@@ -180,6 +184,7 @@ module SidekiqUniqueJobs
180
184
  ENABLED,
181
185
  PREFIX,
182
186
  Sidekiq.logger,
187
+ LOGGER_ENABLED,
183
188
  LOCKS,
184
189
  STRATEGIES,
185
190
  DEBUG_LUA,
@@ -14,6 +14,7 @@ module SidekiqUniqueJobs
14
14
  CREATED_AT = "created_at"
15
15
  DEAD_VERSION = "uniquejobs:dead"
16
16
  DIGESTS = "uniquejobs:digests"
17
+ EXPIRING_DIGESTS = "uniquejobs:expiring_digests"
17
18
  ERRORS = "errors"
18
19
  JID = "jid"
19
20
  LIMIT = "limit"
@@ -98,7 +98,7 @@ class Hash
98
98
  def _deep_transform_keys_in_object(object, &block)
99
99
  case object
100
100
  when Hash
101
- object.each_with_object({}) do |(key, value), result|
101
+ object.each_with_object(self.class.new) do |(key, value), result|
102
102
  result[yield(key)] = _deep_transform_keys_in_object(value, &block)
103
103
  end
104
104
  when Array
@@ -14,8 +14,8 @@ module SidekiqUniqueJobs
14
14
  # @return [String] the default pattern to use for matching
15
15
  SCAN_PATTERN = "*"
16
16
 
17
- def initialize
18
- super(DIGESTS)
17
+ def initialize(digests_key = DIGESTS)
18
+ super(digests_key)
19
19
  end
20
20
 
21
21
  #
@@ -72,14 +72,14 @@ module SidekiqUniqueJobs
72
72
  class InvalidUniqueArguments < UniqueJobsError
73
73
  def initialize(options)
74
74
  given = options[:given]
75
- worker_class = options[:worker_class]
75
+ job_class = options[:job_class]
76
76
  lock_args_method = options[:lock_args_method]
77
- lock_args_meth = worker_class.method(lock_args_method)
77
+ lock_args_meth = job_class.method(lock_args_method)
78
78
  num_args = lock_args_meth.arity
79
79
  source_location = lock_args_meth.source_location
80
80
 
81
81
  super(
82
- "#{worker_class}##{lock_args_method} takes #{num_args} arguments, received #{given.inspect}" \
82
+ "#{job_class}##{lock_args_method} takes #{num_args} arguments, received #{given.inspect}" \
83
83
  "\n\n" \
84
84
  " #{source_location.join(':')}"
85
85
  )
@@ -0,0 +1,14 @@
1
+ # frozen_string_literal: true
2
+
3
+ module SidekiqUniqueJobs
4
+ #
5
+ # Class ExpiringDigests provides access to the expiring digests used by until_expired locks
6
+ #
7
+ # @author Mikael Henriksson <mikael@mhenrixon.com>
8
+ #
9
+ class ExpiringDigests < Digests
10
+ def initialize
11
+ super(EXPIRING_DIGESTS)
12
+ end
13
+ end
14
+ end
@@ -11,6 +11,7 @@ module SidekiqUniqueJobs
11
11
  # @return [Hash] the job hash
12
12
  def prepare(item)
13
13
  stringify_on_conflict_hash(item)
14
+ add_lock_type(item)
14
15
  add_lock_timeout(item)
15
16
  add_lock_ttl(item)
16
17
  add_digest(item)
@@ -54,5 +55,9 @@ module SidekiqUniqueJobs
54
55
  def add_lock_prefix(item)
55
56
  item[LOCK_PREFIX] ||= SidekiqUniqueJobs.config.lock_prefix
56
57
  end
58
+
59
+ def add_lock_type(item)
60
+ item[LOCK] ||= SidekiqUniqueJobs::LockType.call(item)
61
+ end
57
62
  end
58
63
  end
@@ -33,6 +33,10 @@ module SidekiqUniqueJobs
33
33
  # @!attribute [r] digests
34
34
  # @return [String] the zset with locked digests
35
35
  attr_reader :digests
36
+ #
37
+ # @!attribute [r] expiring_digests
38
+ # @return [String] the zset with locked expiring_digests
39
+ attr_reader :expiring_digests
36
40
 
37
41
  #
38
42
  # Initialize a new Key
@@ -40,13 +44,14 @@ module SidekiqUniqueJobs
40
44
  # @param [String] digest the digest to use as key
41
45
  #
42
46
  def initialize(digest)
43
- @digest = digest
44
- @queued = suffixed_key("QUEUED")
45
- @primed = suffixed_key("PRIMED")
46
- @locked = suffixed_key("LOCKED")
47
- @info = suffixed_key("INFO")
48
- @changelog = CHANGELOGS
49
- @digests = DIGESTS
47
+ @digest = digest
48
+ @queued = suffixed_key("QUEUED")
49
+ @primed = suffixed_key("PRIMED")
50
+ @locked = suffixed_key("LOCKED")
51
+ @info = suffixed_key("INFO")
52
+ @changelog = CHANGELOGS
53
+ @digests = DIGESTS
54
+ @expiring_digests = EXPIRING_DIGESTS
50
55
  end
51
56
 
52
57
  #
@@ -81,7 +86,7 @@ module SidekiqUniqueJobs
81
86
  # @return [Array] an ordered array with all keys
82
87
  #
83
88
  def to_a
84
- [digest, queued, primed, locked, info, changelog, digests]
89
+ [digest, queued, primed, locked, info, changelog, digests, expiring_digests]
85
90
  end
86
91
 
87
92
  private
@@ -35,6 +35,7 @@ module SidekiqUniqueJobs
35
35
  def execute
36
36
  executed = locksmith.execute do
37
37
  yield
38
+ ensure
38
39
  unlock_and_callback
39
40
  end
40
41
 
@@ -33,6 +33,10 @@ module SidekiqUniqueJobs
33
33
  def execute
34
34
  callback_safely if locksmith.unlock
35
35
  yield
36
+ rescue StandardError => ex
37
+ reflect(:execution_failed, item, ex)
38
+ locksmith.lock(wait: 1)
39
+ raise
36
40
  end
37
41
  end
38
42
  end
@@ -42,9 +42,8 @@ module SidekiqUniqueJobs
42
42
  with_logging_context do
43
43
  executed = locksmith.execute do
44
44
  yield
45
- callback_safely if locksmith.unlock
46
45
  ensure
47
- locksmith.unlock
46
+ unlock_and_callback
48
47
  end
49
48
 
50
49
  unless executed
@@ -66,7 +66,7 @@ module SidekiqUniqueJobs
66
66
  pipeline.set(key.digest, job_id)
67
67
  pipeline.hset(key.locked, job_id, now_f)
68
68
  info.set(lock_info, pipeline)
69
- pipeline.zadd(key.digests, now_f, key.digest)
69
+ add_digest_to_set(pipeline, lock_info)
70
70
  pipeline.zadd(key.changelog, now_f, changelog_json(job_id, "queue.lua", "Queued"))
71
71
  pipeline.zadd(key.changelog, now_f, changelog_json(job_id, "lock.lua", "Locked"))
72
72
  end
@@ -321,5 +321,22 @@ module SidekiqUniqueJobs
321
321
  time: now_f,
322
322
  )
323
323
  end
324
+
325
+ #
326
+ # Add the digest to the correct sorted set
327
+ #
328
+ # @param [Object] pipeline a redis pipeline object for issue commands
329
+ # @param [Hash] lock_info the lock info relevant to the digest
330
+ #
331
+ # @return [nil]
332
+ #
333
+ def add_digest_to_set(pipeline, lock_info)
334
+ digest_string = key.digest
335
+ if lock_info["lock"] == :until_expired
336
+ pipeline.zadd(key.expiring_digests, now_f + lock_info["ttl"], digest_string)
337
+ else
338
+ pipeline.zadd(key.digests, now_f, digest_string)
339
+ end
340
+ end
324
341
  end
325
342
  end
@@ -26,9 +26,9 @@ module SidekiqUniqueJobs
26
26
 
27
27
  # @param [Hash] item a Sidekiq job hash
28
28
  def initialize(item)
29
- @item = item
30
- @worker_class = item[CLASS]
31
- @args = item[ARGS]
29
+ @item = item
30
+ @args = item[ARGS]
31
+ self.job_class = item[CLASS]
32
32
  end
33
33
 
34
34
  # The unique arguments to use for creating a lock
@@ -83,31 +83,31 @@ module SidekiqUniqueJobs
83
83
 
84
84
  # Filters unique arguments by method configured in the sidekiq worker
85
85
  # @param [Array] args the arguments passed to the sidekiq worker
86
- # @return [Array] unfiltered unless {#worker_method_defined?}
86
+ # @return [Array] unfiltered unless {#job_method_defined?}
87
87
  # @return [Array] with the filtered arguments
88
88
  def filter_by_symbol(args)
89
- return args unless worker_method_defined?(lock_args_method)
89
+ return args unless job_method_defined?(lock_args_method)
90
90
 
91
- worker_class.send(lock_args_method, args)
91
+ job_class.send(lock_args_method, args)
92
92
  rescue ArgumentError
93
93
  raise SidekiqUniqueJobs::InvalidUniqueArguments,
94
94
  given: args,
95
- worker_class: worker_class,
95
+ job_class: job_class,
96
96
  lock_args_method: lock_args_method
97
97
  end
98
98
 
99
99
  # The method to use for filtering unique arguments
100
100
  def lock_args_method
101
- @lock_args_method ||= worker_options.slice(LOCK_ARGS_METHOD, UNIQUE_ARGS_METHOD).values.first
102
- @lock_args_method ||= :lock_args if worker_method_defined?(:lock_args)
103
- @lock_args_method ||= :unique_args if worker_method_defined?(:unique_args)
101
+ @lock_args_method ||= job_options.slice(LOCK_ARGS_METHOD, UNIQUE_ARGS_METHOD).values.first
102
+ @lock_args_method ||= :lock_args if job_method_defined?(:lock_args)
103
+ @lock_args_method ||= :unique_args if job_method_defined?(:unique_args)
104
104
  @lock_args_method ||= default_lock_args_method
105
105
  end
106
106
 
107
107
  # The global worker options defined in Sidekiq directly
108
108
  def default_lock_args_method
109
- default_worker_options[LOCK_ARGS_METHOD] ||
110
- default_worker_options[UNIQUE_ARGS_METHOD]
109
+ default_job_options[LOCK_ARGS_METHOD] ||
110
+ default_job_options[UNIQUE_ARGS_METHOD]
111
111
  end
112
112
 
113
113
  #
@@ -116,8 +116,12 @@ module SidekiqUniqueJobs
116
116
  #
117
117
  # @return [Hash<String, Object>]
118
118
  #
119
- def default_worker_options
120
- @default_worker_options ||= Sidekiq.default_worker_options.stringify_keys
119
+ def default_job_options
120
+ @default_job_options ||= if Sidekiq.respond_to?(:default_job_options)
121
+ Sidekiq.default_job_options.stringify_keys
122
+ else
123
+ Sidekiq.default_worker_options.stringify_keys
124
+ end
121
125
  end
122
126
  end
123
127
  end
@@ -13,9 +13,9 @@ module SidekiqUniqueJobs
13
13
  # @return [Symbol] the type of lock
14
14
  attr_reader :type
15
15
  #
16
- # @!attribute [r] worker
17
- # @return [Symbol] the worker class
18
- attr_reader :worker
16
+ # @!attribute [r] job
17
+ # @return [Symbol] the job class
18
+ attr_reader :job
19
19
  #
20
20
  # @!attribute [r] limit
21
21
  # @return [Integer] the number of simultaneous locks
@@ -58,7 +58,7 @@ module SidekiqUniqueJobs
58
58
 
59
59
  def initialize(job_hash = {})
60
60
  @type = job_hash[LOCK]&.to_sym
61
- @worker = SidekiqUniqueJobs.safe_constantize(job_hash[CLASS])
61
+ @job = SidekiqUniqueJobs.safe_constantize(job_hash[CLASS])
62
62
  @limit = job_hash.fetch(LOCK_LIMIT, 1)&.to_i
63
63
  @timeout = job_hash.fetch(LOCK_TIMEOUT, 0)&.to_i
64
64
  @ttl = job_hash.fetch(LOCK_TTL) { job_hash.fetch(LOCK_EXPIRATION, nil) }.to_i
@@ -36,10 +36,10 @@ module SidekiqUniqueJobs
36
36
 
37
37
  # @param [Hash] item a Sidekiq job hash
38
38
  def initialize(item)
39
- @item = item
40
- @worker_class = item[CLASS]
41
- @lock_args = item[LOCK_ARGS] || item[UNIQUE_ARGS] # TODO: Deprecate UNIQUE_ARGS
42
- @lock_prefix = item[LOCK_PREFIX] || item[UNIQUE_PREFIX] # TODO: Deprecate UNIQUE_PREFIX
39
+ @item = item
40
+ @lock_args = item[LOCK_ARGS] || item[UNIQUE_ARGS] # TODO: Deprecate UNIQUE_ARGS
41
+ @lock_prefix = item[LOCK_PREFIX] || item[UNIQUE_PREFIX] # TODO: Deprecate UNIQUE_PREFIX
42
+ self.job_class = item[CLASS]
43
43
  end
44
44
 
45
45
  # Memoized lock_digest
@@ -51,7 +51,7 @@ module SidekiqUniqueJobs
51
51
  # Creates a namespaced unique digest based on the {#digestable_hash} and the {#lock_prefix}
52
52
  # @return [String] a unique digest
53
53
  def create_digest
54
- digest = OpenSSL::Digest::MD5.hexdigest(dump_json(digestable_hash))
54
+ digest = OpenSSL::Digest::MD5.hexdigest(dump_json(digestable_hash.sort))
55
55
  "#{lock_prefix}:#{digest}"
56
56
  end
57
57
 
@@ -67,13 +67,13 @@ module SidekiqUniqueJobs
67
67
  # Checks if we should disregard the queue when creating the unique digest
68
68
  # @return [true, false]
69
69
  def unique_across_queues?
70
- item[UNIQUE_ACROSS_QUEUES] || worker_options[UNIQUE_ACROSS_QUEUES]
70
+ item[UNIQUE_ACROSS_QUEUES] || job_options[UNIQUE_ACROSS_QUEUES]
71
71
  end
72
72
 
73
73
  # Checks if we should disregard the worker when creating the unique digest
74
74
  # @return [true, false]
75
75
  def unique_across_workers?
76
- item[UNIQUE_ACROSS_WORKERS] || worker_options[UNIQUE_ACROSS_WORKERS]
76
+ item[UNIQUE_ACROSS_WORKERS] || job_options[UNIQUE_ACROSS_WORKERS]
77
77
  end
78
78
  end
79
79
  end
@@ -30,8 +30,8 @@ module SidekiqUniqueJobs
30
30
  # @option item [String] :class the class of the sidekiq worker
31
31
  # @option item [Float] :at the unix time the job is scheduled at
32
32
  def initialize(item)
33
- @item = item
34
- @worker_class = item[CLASS]
33
+ @item = item
34
+ self.job_class = item[CLASS]
35
35
  end
36
36
 
37
37
  #
@@ -42,9 +42,9 @@ module SidekiqUniqueJobs
42
42
  # @return [Integer, nil]
43
43
  #
44
44
  def calculate
45
- timeout = default_worker_options[LOCK_TIMEOUT]
45
+ timeout = default_job_options[LOCK_TIMEOUT]
46
46
  timeout = default_lock_timeout if default_lock_timeout
47
- timeout = worker_options[LOCK_TIMEOUT] if worker_options.key?(LOCK_TIMEOUT)
47
+ timeout = job_options[LOCK_TIMEOUT] if job_options.key?(LOCK_TIMEOUT)
48
48
  timeout
49
49
  end
50
50