sidekiq-unique-jobs 7.1.14 → 7.1.17

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 282d21cd3a381132e1a4cbe3852e158f3bd9d5b80430cf58b5192d3879282d53
-  data.tar.gz: 396796dfb7757f83fed54835495334993304dd144c6aebad129b736b30b38107
+  metadata.gz: 787e3a8e85324b12df2f67dc04bee3ad3ad26deadc06ce170c9a5c5be62c1047
+  data.tar.gz: d280fac7bf32f98bbe8071822463c7184e24339ad756953131d113f2b18710a3
 SHA512:
-  metadata.gz: 26ed510d584ce8c4c00db7fdaa8b8267dfa58014d01bb5857e3ccf1ce974256a0e8dc82187dee9cb6bdfad4bdc97ead436ca10d707cae44954df55a3ef3c552a
-  data.tar.gz: 94d354d3f405e23562588d0df93ce54fd3883c45c4a1e40dac40516d41bbf3e0b97e9ab3a57883d957a1d6de2b10f4fa1544e03f8f2e845bf39c6639c04ba1b1
+  metadata.gz: c6549578ab2200965dcbab5e8acbf9e159f5dccce5bd2eb277b6914739e3279195169fd59349d47ae8c2cf9dec2bb7b2116fbceafd17d442b1be6096f9e559fb
+  data.tar.gz: e2469b5c19d39d97fb231db49f291a8e5dcd6763f39c28d794b69b43f1551ffc5275075b39a6d11a5eda82409f87f3d4a0d5d0ce2789b4cf90f84c5940c58c2c

data/CHANGELOG.md

@@ -1,5 +1,51 @@
 # Changelog
 
+## [v7.1.16](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v7.1.16) (2022-04-02)
+
+[Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v7.1.15...v7.1.16)
+
+**Implemented enhancements:**
+
+- Abort Ruby Reaper when sidekiq queues are full [\#690](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/690) ([francesmcmullin](https://github.com/francesmcmullin))
+- Quote '3.0' to ensure CI uses Ruby 3.0.x for the 3.0 entry [\#689](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/689) ([petergoldstein](https://github.com/petergoldstein))
+
+**Fixed bugs:**
+
+- Hotfix: Ensure consistent lock args [\#699](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/699) ([mhenrixon](https://github.com/mhenrixon))
+- Expire older changelog entries first [\#698](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/698) ([mhenrixon](https://github.com/mhenrixon))
+- Fix drift [\#688](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/688) ([mhenrixon](https://github.com/mhenrixon))
+
+**Closed issues:**
+
+- concurrent-ruby has dropped support for TimerTask timeouts [\#697](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/697)
+- Most recent changelogs are removed first [\#696](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/696)
+- Improve README slightly [\#694](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/694)
+- locksmith.rb:327: NoMethodError:  undefined method `+' for nil:NilClass [\#686](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/686)
+- lock\_timeout cannot be nil [\#675](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/675)
+- Skip reaping when queues are too large [\#670](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/670)
+
+**Merged pull requests:**
+
+- Improve readme [\#695](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/695) ([fwolfst](https://github.com/fwolfst))
+- Add funding\_uri to gemspec [\#693](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/693) ([fwolfst](https://github.com/fwolfst))
+- Fix worker validator [\#685](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/685) ([victorfgs](https://github.com/victorfgs))
+
+## [v7.1.15](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v7.1.15) (2022-02-10)
+
+[Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v7.1.14...v7.1.15)
+
+**Merged pull requests:**
+
+- Fixing reschedule when using a non default queue [\#679](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/679) ([bigzed](https://github.com/bigzed))
+
+## [v7.1.14](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v7.1.14) (2022-02-04)
+
+[Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v7.1.13...v7.1.14)
+
+**Implemented enhancements:**
+
+- Fix the remaining deprecation warnings [\#681](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/681) ([mhenrixon](https://github.com/mhenrixon))
+
 ## [v7.1.13](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v7.1.13) (2022-02-03)
 
 [Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v7.1.12...v7.1.13)

data/README.md

@@ -90,7 +90,7 @@ Want to show me some ❤️ for the hard work I do on this gem? You can use the
 
 This gem adds unique constraints to sidekiq jobs. The uniqueness is achieved by creating a set of keys in redis based off of `queue`, `class`, `args` (in the sidekiq job hash).
 
-By default, only one lock for a given hash can be acquired. What happens when a lock can't be acquired is governed by a chosen [Conflict Strategy](#conflict-strategy) strategy. Unless a conflict strategy is chosen
+By default, only one lock for a given hash can be acquired. What happens when a lock can't be acquired is governed by a chosen [Conflict Strategy](#conflict-strategy) strategy. Unless a conflict strategy is chosen (?)
 
 This is the documentation for the `main` branch. You can find the documentation for each release by navigating to its tag.
 
@@ -151,7 +151,7 @@ end
 
 ### Your first worker
 
-The most likely to be used worker is `:until_executed`. This type of lock creates a lock from when `UntilExecutedWorker.perform_async` is called until right after `UntilExecutedWorker.new.perform` has been called.
+The lock type most likely to be is `:until_executed`. This type of lock creates a lock from when `UntilExecutedWorker.perform_async` is called until right after `UntilExecutedWorker.new.perform` has been called.
 
 ```ruby
 # frozen_string_literal: true
@@ -159,8 +159,7 @@ The most likely to be used worker is `:until_executed`. This type of lock create
 class UntilExecutedWorker
   include Sidekiq::Worker
 
-  sidekiq_options queue: :until_executed,
-                  lock: :until_executed
+  sidekiq_options lock: :until_executed
 
   def perform
     logger.info("cowboy")
@@ -346,9 +345,9 @@ Please not that if you try to override a default lock, an `ArgumentError` will b
 
 ## Conflict Strategy
 
-Decides how we handle conflict. We can either reject the job to the dead queue or reschedule it. Both are useful for jobs that absolutely need to run and have been configured to use the lock `WhileExecuting` that is used only by the sidekiq server process.
+Decides how we handle conflict. We can either `reject` the job to the dead queue or `reschedule` it. Both are useful for jobs that absolutely need to run and have been configured to use the lock `WhileExecuting` that is used only by the sidekiq server process.
 
-The last one is log which can be be used with the lock `UntilExecuted` and `UntilExpired`. Now we write a log entry saying the job could not be pushed because it is a duplicate of another job with the same arguments.
+Furthermore, `log` can be be used with the lock `UntilExecuted` and `UntilExpired`. Now we write a log entry saying the job could not be pushed because it is a duplicate of another job with the same arguments.
 
 It is possible for locks to have different conflict strategy for the client and server. This is useful for `:until_and_while_executing`.
 
@@ -394,7 +393,7 @@ queue and retry the lock again.
 This is slightly dangerous and should probably only be used for jobs that are
 always scheduled in the future. Currently only attempting to retry one time.
 
-### Reschedule
+### reschedule
 
 ```ruby
 sidekiq_options on_conflict: :reschedule
@@ -568,7 +567,7 @@ The reason this happens is that the server couldn't find a valid sidekiq worker
 
 ### Validating Worker Configuration
 
-Since v7 it is possible to perform some simple validation against your workers sidekiq_options. What it does is scan for some issues that are known to cause problems in production.
+Since v7 it is possible to perform some simple validation against your workers `sidekiq_options`. What it does is scan for some issues that are known to cause problems in production.
 
 Let's take a _bad_ worker:
 
@@ -604,7 +603,7 @@ assert_raise(InvalidWorker){ SidekiqUniqueJobs.validate_worker!(BadWorker.get_si
 
 ### Uniqueness
 
-This has been probably the most confusing part of this gem. People get really confused with how unreliable the unique jobs have been. I there for decided to do what Mike is doing for sidekiq enterprise. Read the section about unique jobs: [Enterprise unique jobs][]
+This has been probably the most confusing part of this gem. People get really confused with how unreliable the unique jobs have been. I there for decided to do what Mike is doing for sidekiq enterprise. Read the section about unique jobs: [Enterprise unique jobs][](?)
 
 ```ruby
 SidekiqUniqueJobs.configure do |config|

data/lib/sidekiq_unique_jobs/lock/until_executed.rb

@@ -33,10 +33,14 @@ module SidekiqUniqueJobs
       # Executes in the Sidekiq server process
       # @yield to the worker class perform method
       def execute
-        locksmith.execute do
+        executed = locksmith.execute do
           yield
           unlock_and_callback
         end
+
+        reflect(:execution_failed, item) unless executed
+
+        nil
       end
     end
   end

data/lib/sidekiq_unique_jobs/lock/until_expired.rb

@@ -33,7 +33,9 @@ module SidekiqUniqueJobs
       # Executes in the Sidekiq server process
       # @yield to the worker class perform method
       def execute(&block)
-        locksmith.execute(&block)
+        executed = locksmith.execute(&block)
+
+        reflect(:execution_failed, item) unless executed
       end
     end
   end

data/lib/sidekiq_unique_jobs/lock/while_executing.rb

@@ -47,7 +47,10 @@ module SidekiqUniqueJobs
             locksmith.unlock
           end
 
-          call_strategy(origin: :server, &block) unless executed
+          unless executed
+            reflect(:execution_failed, item)
+            call_strategy(origin: :server, &block)
+          end
         end
       end
 

data/lib/sidekiq_unique_jobs/lock_args.rb

@@ -34,7 +34,7 @@ module SidekiqUniqueJobs
     # The unique arguments to use for creating a lock
     # @return [Array] the arguments filters by the {#filtered_args} method if {#lock_args_enabled?}
     def lock_args
-      @lock_args ||= filtered_args
+      @lock_args ||= filtered_args || []
     end
 
     # Checks if the worker class has enabled lock_args

data/lib/sidekiq_unique_jobs/lock_config.rb

@@ -113,13 +113,13 @@ module SidekiqUniqueJobs
 
     # the strategy to use as conflict resolution from sidekiq client
     def on_client_conflict
-      @on_client_conflict ||= on_conflict["client"] if on_conflict.is_a?(Hash)
+      @on_client_conflict ||= on_conflict["client"] || on_conflict[:client] if on_conflict.is_a?(Hash)
       @on_client_conflict ||= on_conflict
     end
 
     # the strategy to use as conflict resolution from sidekiq server
     def on_server_conflict
-      @on_server_conflict ||= on_conflict["server"] if on_conflict.is_a?(Hash)
+      @on_server_conflict ||= on_conflict["server"] || on_conflict[:server] if on_conflict.is_a?(Hash)
       @on_server_conflict ||= on_conflict
     end
   end

data/lib/sidekiq_unique_jobs/lock_digest.rb

@@ -38,8 +38,8 @@ module SidekiqUniqueJobs
     def initialize(item)
       @item         = item
       @worker_class = item[CLASS]
-      @lock_args    = item.slice(LOCK_ARGS, UNIQUE_ARGS).values.first # TODO: Deprecate UNIQUE_ARGS
-      @lock_prefix  = item.slice(LOCK_PREFIX, UNIQUE_PREFIX).values.first # TODO: Deprecate UNIQUE_PREFIX
+      @lock_args    = item[LOCK_ARGS] || item[UNIQUE_ARGS] # TODO: Deprecate UNIQUE_ARGS
+      @lock_prefix  = item[LOCK_PREFIX] || item[UNIQUE_PREFIX] # TODO: Deprecate UNIQUE_PREFIX
     end
 
     # Memoized lock_digest

data/lib/sidekiq_unique_jobs/locksmith.rb

@@ -328,6 +328,7 @@ module SidekiqUniqueJobs
     end
 
     def add_drift(val)
+      val = val.to_f
       val + drift(val)
     end
 

data/lib/sidekiq_unique_jobs/lua/shared/_common.lua

@@ -31,7 +31,7 @@ local function log(message, prev_jid)
   log_debug("ZADD", changelog, current_time, entry);
   redis.call("ZADD", changelog, current_time, entry);
   local total_entries = redis.call("ZCARD", changelog)
-  local removed_entries = redis.call("ZREMRANGEBYRANK", changelog, max_history, -1)
+  local removed_entries = redis.call("ZREMRANGEBYRANK", changelog, 0, -1 * max_history)
   if removed_entries > 0 then
     log_debug("Removing", removed_entries , "entries from changelog (total entries", total_entries, "exceeds max_history:", max_history ..")");
   end

data/lib/sidekiq_unique_jobs/on_conflict/reschedule.rb

@@ -21,7 +21,7 @@ module SidekiqUniqueJobs
       #   This will mess up sidekiq stats because a new job is created
       def call
         if sidekiq_worker_class?
-          if worker_class.perform_in(5, *item[ARGS])
+          if worker_class.set(queue: item["queue"].to_sym).perform_in(5, *item[ARGS])
             reflect(:rescheduled, item)
           else
             reflect(:reschedule_failed, item)

data/lib/sidekiq_unique_jobs/orphans/manager.rb

@@ -1,5 +1,7 @@
 # frozen_string_literal: true
 
+require "concurrent/version"
+
 module SidekiqUniqueJobs
   module Orphans
     #
@@ -108,9 +110,13 @@ module SidekiqUniqueJobs
       # @return [Hash]
       #
       def timer_task_options
-        { run_now: true,
-          execution_interval: reaper_interval,
-          timeout_interval: reaper_timeout }
+        timer_task_options = { run_now: true, execution_interval: reaper_interval }
+
+        if VersionCheck.satisfied?(::Concurrent::VERSION, "< 1.1.10")
+          timer_task_options[:timeout_interval] = reaper_timeout
+        end
+
+        timer_task_options
       end
 
       #

data/lib/sidekiq_unique_jobs/orphans/ruby_reaper.rb

@@ -15,6 +15,9 @@ module SidekiqUniqueJobs
       # @return [String] the suffix for :RUN locks
       RUN_SUFFIX = ":RUN"
       #
+      # @return [Integer] the maximum combined length of sidekiq queues for running the reaper
+      MAX_QUEUE_LENGTH = 1000
+      #
       # @!attribute [r] digests
       #   @return [SidekiqUniqueJobs::Digests] digest collection
       attr_reader :digests
@@ -46,6 +49,8 @@ module SidekiqUniqueJobs
       # @return [Integer] the number of reaped locks
       #
       def call
+        return if queues_very_full?
+
         BatchDelete.call(orphans, conn)
       end
 
@@ -212,6 +217,22 @@ module SidekiqUniqueJobs
         end
       end
 
+      # If sidekiq queues are very full, it becomes highly inefficient for the reaper
+      # because it must check every queued job to verify a digest is safe to delete
+      # The reaper checks queued jobs in batches of 50, adding 2 reads per digest
+      # With a queue length of 1,000 jobs, that's over 20 extra reads per digest.
+      def queues_very_full?
+        total_queue_size = 0
+        Sidekiq.redis do |conn|
+          queues(conn) do |queue|
+            total_queue_size += conn.llen("queue:#{queue}")
+
+            return true if total_queue_size > MAX_QUEUE_LENGTH
+          end
+        end
+        false
+      end
+
       #
       # Checks a sorted set for the existance of this digest
       #

data/lib/sidekiq_unique_jobs/testing.rb

@@ -21,16 +21,24 @@ module Sidekiq
   #
   # @param [Hash<Symbol, Object>] tmp_config the temporary config to use
   #
-  def self.use_options(tmp_config = {})
-    old_options = default_worker_options.dup
+  def self.use_options(tmp_config = {}) # rubocop:disable Metrics/MethodLength
+    if respond_to?(:default_job_options)
+      default_job_options.clear
+      self.default_job_options = tmp_config
+    else
+      default_worker_options.clear
+      self.default_worker_options = tmp_config
+    end
 
-    default_worker_options.clear
-    self.default_worker_options = tmp_config
     yield
   ensure
-    default_worker_options.clear
-    self.default_worker_options = DEFAULT_WORKER_OPTIONS
-    self.default_worker_options = old_options
+    if respond_to?(:default_job_options)
+      default_job_options.clear
+      self.default_job_options = default_job_options
+    else
+      default_worker_options.clear
+      self.default_worker_options = DEFAULT_WORKER_OPTIONS
+    end
   end
 
   #
@@ -54,7 +62,13 @@ module Sidekiq
 
         yield
       ensure
-        self.sidekiq_options_hash = Sidekiq::DEFAULT_WORKER_OPTIONS
+        self.sidekiq_options_hash =
+          if Sidekiq.respond_to?(:default_job_options)
+            Sidekiq.default_job_options
+          else
+            DEFAULT_WORKER_OPTIONS
+          end
+
         sidekiq_options(old_options)
       end
 

data/lib/sidekiq_unique_jobs/timer_task.rb

@@ -1,78 +1,84 @@
 # frozen_string_literal: true
 
+require "concurrent/version"
+require_relative "version_check"
+
 module SidekiqUniqueJobs
   # @see [Concurrent::TimerTask] https://www.rubydoc.info/gems/concurrent-ruby/Concurrent/TimerTask
   #
   class TimerTask < ::Concurrent::TimerTask
-    private
+    if VersionCheck.satisfied?(::Concurrent::VERSION, "< 1.1.10")
 
-    def ns_initialize(opts, &task)
-      set_deref_options(opts)
+      private
 
-      self.execution_interval = opts[:execution] || opts[:execution_interval] || EXECUTION_INTERVAL
-      self.timeout_interval = opts[:timeout] || opts[:timeout_interval] || TIMEOUT_INTERVAL
-      @run_now  = opts[:now] || opts[:run_now]
-      @executor = Concurrent::RubySingleThreadExecutor.new
-      @running  = Concurrent::AtomicBoolean.new(false)
-      @task     = task
-      @value    = nil
+      def ns_initialize(opts, &task)
+        set_deref_options(opts)
 
-      self.observers = Concurrent::Collection::CopyOnNotifyObserverSet.new
-    end
+        self.execution_interval = opts[:execution] || opts[:execution_interval] || EXECUTION_INTERVAL
+        self.timeout_interval = opts[:timeout] || opts[:timeout_interval] || TIMEOUT_INTERVAL
+        @run_now  = opts[:now] || opts[:run_now]
+        @executor = Concurrent::RubySingleThreadExecutor.new
+        @running  = Concurrent::AtomicBoolean.new(false)
+        @task     = task
+        @value    = nil
 
-    def schedule_next_task(interval = execution_interval)
-      exec_task = ->(completion) { execute_task(completion) }
-      Concurrent::ScheduledTask.execute(interval, args: [Concurrent::Event.new], &exec_task)
-      nil
-    end
+        self.observers = Concurrent::Collection::CopyOnNotifyObserverSet.new
+      end
 
-    # @!visibility private
-    def execute_task(completion) # rubocop:disable Metrics/MethodLength
-      return nil unless @running.true?
-
-      timeout_task = -> { timeout_task(completion) }
-
-      Concurrent::ScheduledTask.execute(
-        timeout_interval,
-        args: [completion],
-        &timeout_task
-      )
-      @thread_completed = Concurrent::Event.new
-
-      @value = @reason  = nil
-      @executor.post do
-        @value = @task.call(self)
-      rescue Exception => ex # rubocop:disable Lint/RescueException
-        @reason = ex
-      ensure
-        @thread_completed.set
+      def schedule_next_task(interval = execution_interval)
+        exec_task = ->(completion) { execute_task(completion) }
+        Concurrent::ScheduledTask.execute(interval, args: [Concurrent::Event.new], &exec_task)
+        nil
       end
 
-      @thread_completed.wait
+      # @!visibility private
+      def execute_task(completion) # rubocop:disable Metrics/MethodLength
+        return nil unless @running.true?
 
-      if completion.try?
-        schedule_next_task
-        time = Time.now
-        observers.notify_observers do
-          [time, value, @reason]
+        timeout_task = -> { timeout_task(completion) }
+
+        Concurrent::ScheduledTask.execute(
+          timeout_interval,
+          args: [completion],
+          &timeout_task
+        )
+        @thread_completed = Concurrent::Event.new
+
+        @value = @reason  = nil
+        @executor.post do
+          @value = @task.call(self)
+        rescue Exception => ex # rubocop:disable Lint/RescueException
+          @reason = ex
+        ensure
+          @thread_completed.set
         end
+
+        @thread_completed.wait
+
+        if completion.try?
+          schedule_next_task
+          time = Time.now
+          observers.notify_observers do
+            [time, value, @reason]
+          end
+        end
+        nil
       end
-      nil
-    end
 
-    # @!visibility private
-    def timeout_task(completion)
-      return unless @running.true?
-      return unless completion.try?
+      # @!visibility private
+      def timeout_task(completion)
+        return unless @running.true?
+        return unless completion.try?
 
-      @executor.kill
-      @executor.wait_for_termination
-      @executor = Concurrent::RubySingleThreadExecutor.new
+        @executor.kill
+        @executor.wait_for_termination
+        @executor = Concurrent::RubySingleThreadExecutor.new
 
-      @thread_completed.set
+        @thread_completed.set
 
-      schedule_next_task
-      observers.notify_observers(Time.now, nil, Concurrent::TimeoutError.new)
+        schedule_next_task
+        observers.notify_observers(Time.now, nil, Concurrent::TimeoutError.new)
+      end
     end
   end
 end

data/lib/sidekiq_unique_jobs/version.rb

@@ -3,5 +3,5 @@
 module SidekiqUniqueJobs
   #
   # @return [String] the current SidekiqUniqueJobs version
-  VERSION = "7.1.14"
+  VERSION = "7.1.17"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: sidekiq-unique-jobs
 version: !ruby/object:Gem::Version
-  version: 7.1.14
+  version: 7.1.17
 platform: ruby
 authors:
 - Mikael Henriksson
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2022-02-04 00:00:00.000000000 Z
+date: 2022-04-05 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: brpoplpush-redis_script
@@ -257,7 +257,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.3.6
+rubygems_version: 3.3.7
 signing_key:
 specification_version: 4
 summary: Sidekiq middleware that prevents duplicates jobs