sidekiq-unique-jobs 7.1.10 → 7.1.14

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 64a9b84d435ae24be5efae573644c34ba8f16086428ce50c238d54103a39d169
-  data.tar.gz: 671bd927bfbcae9c5cd69eaea7191f7a04e8e1e1180b72e7122f06baefab7750
+  metadata.gz: 282d21cd3a381132e1a4cbe3852e158f3bd9d5b80430cf58b5192d3879282d53
+  data.tar.gz: 396796dfb7757f83fed54835495334993304dd144c6aebad129b736b30b38107
 SHA512:
-  metadata.gz: ffd863678e2f883bee00e8656ba4561b5326f8f60936c5ceb65a1e25886f874173bc4c3f190639294f2e32c106b13515c2c4e37a2845dcf41b1794e47c4f85fd
-  data.tar.gz: 151ec6e893be20ee73048156deb38a7deaf0f4ab030947903eb9501d587348d3ee0dcab3b8f87dae9cc1c35ab397257182722bedbf0dfe4e712e560c17940078
+  metadata.gz: 26ed510d584ce8c4c00db7fdaa8b8267dfa58014d01bb5857e3ccf1ce974256a0e8dc82187dee9cb6bdfad4bdc97ead436ca10d707cae44954df55a3ef3c552a
+  data.tar.gz: 94d354d3f405e23562588d0df93ce54fd3883c45c4a1e40dac40516d41bbf3e0b97e9ab3a57883d957a1d6de2b10f4fa1544e03f8f2e845bf39c6639c04ba1b1

data/CHANGELOG.md

@@ -1,5 +1,66 @@
 # Changelog
 
+## [v7.1.13](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v7.1.13) (2022-02-03)
+
+[Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v7.1.12...v7.1.13)
+
+**Implemented enhancements:**
+
+- Prepare for redis 5.0.0 [\#680](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/680) ([mhenrixon](https://github.com/mhenrixon))
+
+**Fixed bugs:**
+
+- Fix homepage url [\#667](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/667) ([dal-ioki](https://github.com/dal-ioki))
+
+**Closed issues:**
+
+- Job finished, but lock is not cleared [\#677](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/677)
+- sidekiq\_options lock conflicts with sidekiq-lock gem lock option [\#669](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/669)
+- Slow evalsha causing timeouts [\#668](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/668)
+- Inconsistent documentation for config validation [\#647](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/647)
+
+**Merged pull requests:**
+
+- Bump bundler and friends [\#674](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/674) ([mhenrixon](https://github.com/mhenrixon))
+- readme: fix minitest assertion. [\#672](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/672) ([crondaemon](https://github.com/crondaemon))
+- Pass `item` in `after_unlock` callback [\#665](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/665) ([piloos](https://github.com/piloos))
+
+## [v7.1.12](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v7.1.12) (2021-12-01)
+
+[Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v7.1.11...v7.1.12)
+
+**Implemented enhancements:**
+
+- Improve Ruby Reaper performance under heavy load [\#663](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/663)
+- Improve reaper performance under heavy load [\#666](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/666) ([mhenrixon](https://github.com/mhenrixon))
+
+## [v7.1.11](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v7.1.11) (2021-11-30)
+
+[Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v7.1.10...v7.1.11)
+
+**Fixed bugs:**
+
+- Fix ruby reaper edge case [\#661](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/661) ([mhenrixon](https://github.com/mhenrixon))
+
+**Closed issues:**
+
+- Question: Wait instead of cancelling if it is executing? [\#655](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/655)
+- Expired Locks remain in zset of digests \[using "until\_expired" lock\] [\#653](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/653)
+- \[Q&A\] Performance & Dead Locks [\#652](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/652)
+- Sidekiq 6.3.0 includes Job module that clashes with sidekiq\_unique\_ext.rb class Job [\#651](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/651)
+
+## [v7.1.10](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v7.1.10) (2021-10-18)
+
+[Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v7.1.8...v7.1.10)
+
+**Fixed bugs:**
+
+- "IndexError: string not matched" when job is replaced on client [\#635](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/635)
+
+**Merged pull requests:**
+
+- Update URL for Sidekiq's Enterprise unique jobs [\#648](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/648) ([jsantos](https://github.com/jsantos))
+
 ## [v7.1.8](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v7.1.8) (2021-10-08)
 
 [Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v7.0.13...v7.1.8)
@@ -109,7 +170,7 @@
 **Implemented enhancements:**
 
 - Reflections [\#611](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/611) ([mhenrixon](https://github.com/mhenrixon))
-- Start new orphan reaper process if orphan reaper is not running [\#604](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/604) ([Assa1121](https://github.com/Assa1121))
+- Start new orphan reaper process if orphan reaper is not running [\#604](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/604) ([AlexFlint73](https://github.com/AlexFlint73))
 
 **Fixed bugs:**
 

data/README.md

@@ -599,7 +599,7 @@ Expected BadWorker to have valid sidekiq options but found the following problem
 If you are not using RSpec (a lot of people prefer minitest or test unit) you can do something like:
 
 ```ruby
-assert SidekiqUniqueJobs.validate_worker!(BadWorker.get_sidekiq_options)
+assert_raise(InvalidWorker){ SidekiqUniqueJobs.validate_worker!(BadWorker.get_sidekiq_options) }
 ```
 
 ### Uniqueness

data/lib/sidekiq_unique_jobs/batch_delete.rb

@@ -87,10 +87,10 @@ module SidekiqUniqueJobs
     #
     def batch_delete(conn)
       digests.each_slice(BATCH_SIZE) do |chunk|
-        conn.pipelined do
+        conn.pipelined do |pipeline|
           chunk.each do |digest|
-            del_digest(conn, digest)
-            conn.zrem(SidekiqUniqueJobs::DIGESTS, digest)
+            del_digest(pipeline, digest)
+            pipeline.zrem(SidekiqUniqueJobs::DIGESTS, digest)
             @count += 1
           end
         end
@@ -99,13 +99,13 @@ module SidekiqUniqueJobs
       @count
     end
 
-    def del_digest(conn, digest)
+    def del_digest(pipeline, digest)
       removable_keys = keys_for_digest(digest)
 
       if VersionCheck.satisfied?(redis_version, ">= 4.0.0")
-        conn.unlink(*removable_keys)
+        pipeline.unlink(*removable_keys)
       else
-        conn.del(*removable_keys)
+        pipeline.del(*removable_keys)
       end
     end
 

data/lib/sidekiq_unique_jobs/changelog.rb

@@ -62,9 +62,9 @@ module SidekiqUniqueJobs
     #
     def page(cursor: 0, pattern: "*", page_size: 100)
       redis do |conn|
-        total_size, result = conn.multi do
-          conn.zcard(key)
-          conn.zscan(key, cursor, match: pattern, count: page_size)
+        total_size, result = conn.multi do |pipeline|
+          pipeline.zcard(key)
+          pipeline.zscan(key, cursor, match: pattern, count: page_size)
         end
 
         [

data/lib/sidekiq_unique_jobs/digests.rb

@@ -95,9 +95,9 @@ module SidekiqUniqueJobs
     #
     def page(cursor: 0, pattern: SCAN_PATTERN, page_size: 100)
       redis do |conn|
-        total_size, digests = conn.multi do
-          conn.zcard(key)
-          conn.zscan(key, cursor, match: pattern, count: page_size)
+        total_size, digests = conn.multi do |pipeline|
+          pipeline.zcard(key)
+          pipeline.zscan(key, cursor, match: pattern, count: page_size)
         end
 
         [

data/lib/sidekiq_unique_jobs/lock.rb

@@ -62,13 +62,13 @@ module SidekiqUniqueJobs
     #
     def lock(job_id, lock_info = {})
       redis do |conn|
-        conn.multi do
-          conn.set(key.digest, job_id)
-          conn.hset(key.locked, job_id, now_f)
-          info.set(lock_info)
-          conn.zadd(key.digests, now_f, key.digest)
-          conn.zadd(key.changelog, now_f, changelog_json(job_id, "queue.lua", "Queued"))
-          conn.zadd(key.changelog, now_f, changelog_json(job_id, "lock.lua", "Locked"))
+        conn.multi do |pipeline|
+          pipeline.set(key.digest, job_id)
+          pipeline.hset(key.locked, job_id, now_f)
+          info.set(lock_info, pipeline)
+          pipeline.zadd(key.digests, now_f, key.digest)
+          pipeline.zadd(key.changelog, now_f, changelog_json(job_id, "queue.lua", "Queued"))
+          pipeline.zadd(key.changelog, now_f, changelog_json(job_id, "lock.lua", "Locked"))
         end
       end
     end
@@ -123,9 +123,9 @@ module SidekiqUniqueJobs
     #
     def del
       redis do |conn|
-        conn.multi do
-          conn.zrem(DIGESTS, key.digest)
-          conn.del(key.digest, key.queued, key.primed, key.locked, key.info)
+        conn.multi do |pipeline|
+          pipeline.zrem(DIGESTS, key.digest)
+          pipeline.del(key.digest, key.queued, key.primed, key.locked, key.info)
         end
       end
     end

data/lib/sidekiq_unique_jobs/lock_info.rb

@@ -55,13 +55,13 @@ module SidekiqUniqueJobs
     #
     # @return [Hash]
     #
-    def set(obj)
+    def set(obj, pipeline = nil)
       return unless SidekiqUniqueJobs.config.lock_info
       raise InvalidArgument, "argument `obj` (#{obj}) needs to be a hash" unless obj.is_a?(Hash)
 
       json = dump_json(obj)
       @value = load_json(json)
-      super(json)
+      super(json, pipeline)
       value
     end
   end

data/lib/sidekiq_unique_jobs/lua/unlock.lua

@@ -65,18 +65,12 @@ redis.call("LREM", queued, -1, job_id)
 log_debug("LREM", primed, -1, job_id)
 redis.call("LREM", primed, -1, job_id)
 
-if limit and limit <= 1 and locked_count and locked_count <= 1 then
-  log_debug("ZREM", digests, digest)
-  redis.call("ZREM", digests, digest)
-end
-
 local redis_version = toversion(redisversion)
 local del_cmd       = "DEL"
 
 if tonumber(redis_version["major"]) >= 4 then del_cmd =  "UNLINK"; end
 
 if lock_type ~= "until_expired" then
-
   log_debug(del_cmd, digest, info)
   redis.call(del_cmd, digest, info)
 
@@ -86,11 +80,16 @@ end
 
 local locked_count = redis.call("HLEN", locked)
 
-if tonumber(locked_count) < 1 then
+if locked_count and locked_count < 1 then
   log_debug(del_cmd, locked)
   redis.call(del_cmd, locked)
 end
 
+if limit and limit <= 1 and locked_count and locked_count <= 1 then
+  log_debug("ZREM", digests, digest)
+  redis.call("ZREM", digests, digest)
+end
+
 log_debug("LPUSH", queued, "1")
 redis.call("LPUSH", queued, "1")
 

data/lib/sidekiq_unique_jobs/on_conflict/reject.rb

@@ -96,10 +96,10 @@ module SidekiqUniqueJobs
       #
       def push_to_deadset
         redis do |conn|
-          conn.multi do
-            conn.zadd("dead", now_f, payload)
-            conn.zremrangebyscore("dead", "-inf", now_f - Sidekiq::DeadSet.timeout)
-            conn.zremrangebyrank("dead", 0, -Sidekiq::DeadSet.max_jobs)
+          conn.multi do |pipeline|
+            pipeline.zadd("dead", now_f, payload)
+            pipeline.zremrangebyscore("dead", "-inf", now_f - Sidekiq::DeadSet.timeout)
+            pipeline.zremrangebyrank("dead", 0, -Sidekiq::DeadSet.max_jobs)
           end
         end
       end

data/lib/sidekiq_unique_jobs/orphans/ruby_reaper.rb

@@ -9,6 +9,7 @@ module SidekiqUniqueJobs
     #
     # @author Mikael Henriksson <mikael@mhenrixon.com>
     #
+    # rubocop:disable Metrics/ClassLength
     class RubyReaper < Reaper
       #
       # @return [String] the suffix for :RUN locks
@@ -54,13 +55,27 @@ module SidekiqUniqueJobs
       #
       # @return [Array<String>] an array of orphaned digests
       #
-      def orphans
-        conn.zrevrange(digests.key, 0, -1).each_with_object([]) do |digest, memo|
-          next if belongs_to_job?(digest)
+      def orphans # rubocop:disable Metrics/MethodLength
+        page = 0
+        per = reaper_count * 2
+        orphans = []
+        results = conn.zrange(digests.key, page * per, (page + 1) * per)
 
-          memo << digest
-          break if memo.size >= reaper_count
+        while results.size.positive?
+          results.each do |digest|
+            next if belongs_to_job?(digest)
+
+            orphans << digest
+            break if orphans.size >= reaper_count
+          end
+
+          break if orphans.size >= reaper_count
+
+          page += 1
+          results = conn.zrange(digests.key, page * per, (page + 1) * per)
         end
+
+        orphans
       end
 
       #
@@ -126,14 +141,14 @@ module SidekiqUniqueJobs
           return false if procs.empty?
 
           procs.sort.each do |key|
-            valid, workers = conn.pipelined do
+            valid, workers = conn.pipelined do |pipeline|
               # TODO: Remove the if statement in the future
-              if conn.respond_to?(:exists?)
-                conn.exists?(key)
+              if pipeline.respond_to?(:exists?)
+                pipeline.exists?(key)
               else
-                conn.exists(key)
+                pipeline.exists(key)
               end
-              conn.hgetall("#{key}:workers")
+              pipeline.hgetall("#{key}:workers")
             end
 
             next unless valid
@@ -211,5 +226,6 @@ module SidekiqUniqueJobs
         conn.zscan_each(key, match: "*#{digest}*", count: 1).to_a.any?
       end
     end
+    # rubocop:enable Metrics/ClassLength
   end
 end

data/lib/sidekiq_unique_jobs/redis/string.rb

@@ -25,7 +25,9 @@ module SidekiqUniqueJobs
       #
       # @return [true, false]
       #
-      def set(obj)
+      def set(obj, pipeline = nil)
+        return pipeline.set(key, obj) if pipeline
+
         redis { |conn| conn.set(key, obj) }
       end
 

data/lib/sidekiq_unique_jobs/sidekiq_unique_ext.rb

@@ -68,7 +68,7 @@ module Sidekiq
     prepend UniqueExtension
   end
 
-  if Sidekiq.const_defined?("JobRecord")
+  if Sidekiq.const_defined?(:JobRecord)
     # See Sidekiq::Api
     class JobRecord
       #

data/lib/sidekiq_unique_jobs/sidekiq_worker_methods.rb

@@ -34,12 +34,22 @@ module SidekiqUniqueJobs
 
     # The hook to call after a successful unlock
     # @return [Proc]
-    def after_unlock_hook
+    def after_unlock_hook # rubocop:disable Metrics/MethodLength
       lambda do
         if @worker_class.respond_to?(:after_unlock)
-          @worker_class.after_unlock # instance method in sidekiq v6
+          # instance method in sidekiq v6
+          if @worker_class.method(:after_unlock).arity.positive? # arity check to maintain backwards compatibility
+            @worker_class.after_unlock(item)
+          else
+            @worker_class.after_unlock
+          end
         elsif worker_class.respond_to?(:after_unlock)
-          worker_class.after_unlock # class method regardless of sidekiq version
+          # class method regardless of sidekiq version
+          if worker_class.method(:after_unlock).arity.positive? # arity check to maintain backwards compatibility
+            worker_class.after_unlock(item)
+          else
+            worker_class.after_unlock
+          end
         end
       end
     end

data/lib/sidekiq_unique_jobs/timing.rb

@@ -48,7 +48,7 @@ module SidekiqUniqueJobs
     # @return [Float]
     #
     def clock_stamp
-      if Process.const_defined?("CLOCK_MONOTONIC")
+      if Process.const_defined?(:CLOCK_MONOTONIC)
         Process.clock_gettime(Process::CLOCK_MONOTONIC)
       else
         now_f

data/lib/sidekiq_unique_jobs/upgrade_locks.rb

@@ -87,9 +87,9 @@ module SidekiqUniqueJobs
       digest     = grabbed_key.gsub(":GRABBED", "")
       locks      = conn.hgetall(grabbed_key)
 
-      conn.pipelined do
-        conn.hmset(locked_key, *locks.to_a)
-        conn.zadd(DIGESTS, locks.values.first, digest)
+      conn.pipelined do |pipeline|
+        pipeline.hmset(locked_key, *locks.to_a)
+        pipeline.zadd(DIGESTS, locks.values.first, digest)
       end
     end
 
@@ -114,11 +114,11 @@ module SidekiqUniqueJobs
     def batch_delete(*keys)
       return if keys.empty?
 
-      conn.pipelined do
+      conn.pipelined do |pipeline|
         if VersionCheck.satisfied?(redis_version, ">= 4.0.0")
-          conn.unlink(*keys)
+          pipeline.unlink(*keys)
         else
-          conn.del(*keys)
+          pipeline.del(*keys)
         end
       end
     end

data/lib/sidekiq_unique_jobs/version.rb

@@ -3,5 +3,5 @@
 module SidekiqUniqueJobs
   #
   # @return [String] the current SidekiqUniqueJobs version
-  VERSION = "7.1.10"
+  VERSION = "7.1.14"
 end

data/lib/sidekiq_unique_jobs/web/helpers.rb

@@ -27,7 +27,7 @@ module SidekiqUniqueJobs
       # @return [String] the file contents of the template
       #
       def unique_template(name)
-        File.open(unique_filename(name)).read
+        File.read(unique_filename(name))
       end
 
       #

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: sidekiq-unique-jobs
 version: !ruby/object:Gem::Version
-  version: 7.1.10
+  version: 7.1.14
 platform: ruby
 authors:
 - Mikael Henriksson
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2021-11-30 00:00:00.000000000 Z
+date: 2022-02-04 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: brpoplpush-redis_script
@@ -211,15 +211,11 @@ files:
 - lib/sidekiq_unique_jobs/web/views/lock.erb
 - lib/sidekiq_unique_jobs/web/views/locks.erb
 - lib/tasks/changelog.rake
-homepage: https://mhenrixon.github.io/sidekiq-unique-jobs
+homepage: https://github.com/mhenrixon/sidekiq-unique-jobs
 licenses:
 - MIT
 metadata:
-  homepage_uri: https://mhenrixon.github.io/sidekiq-unique-jobs
-  bug_tracker_uri: https://github.com/mhenrixon/sidekiq-unique-jobs/issues
-  documentation_uri: https://mhenrixon.github.io/sidekiq-unique-jobs
-  source_code_uri: https://github.com/mhenrixon/sidekiq-unique-jobs
-  changelog_uri: https://github.com/mhenrixon/sidekiq-unique-jobs/blob/master/CHANGELOG.md
+  rubygems_mfa_required: 'true'
 post_install_message: |
   IMPORTANT!
 
@@ -261,7 +257,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.2.32
+rubygems_version: 3.3.6
 signing_key:
 specification_version: 4
 summary: Sidekiq middleware that prevents duplicates jobs