sidekiq-deferred_jobs 1.0.1 → 1.0.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: e4232cb827ec1a6bd2f218f34ac5b7f3a0b175a0a4dd34841633605d669bfb16
-  data.tar.gz: 3db11095242ca521612cae0b91e1b20abf5fb0676b5fa8f75d0220ab9fa8e8c6
+  metadata.gz: f46a91476e205cb7c8ac96647c7b138f5a525ac1e1e1a8a70578835b6d946850
+  data.tar.gz: 4e0b11956ca657075e5dde2de0fb31764c6e9063d88be065528c587fb35e4579
 SHA512:
-  metadata.gz: 2d8f9ebdad90e5641e6d96c8356ce83e9aad680203f51eb3f0ca933a5d2384f5a52c29d55af0b1fb51ee5e67de9e9a51e085408ce2e56be90b834d95ffdf7279
-  data.tar.gz: 6ee5fc3e3a76a7cd4e7fd96d642fab6664a831d5801240fb0614fe03738f43c0ab6108628478191a9f9a3cd231b7d6dffeb4e9c2d25ba76cd917c19b14223090
+  metadata.gz: 7e2e71026882f25f043d43710ddc228c1a8aa74f984621b3a9a62ab5297a13289a93e583958ef5379ed5a9b05c10752041815e21627a3705e52b3997664ee180
+  data.tar.gz: ef95d50cad3d5ab1e21e5beb912eed08c39d774f856df4105c8db41a828957b1258d3a5bdb2ca7d0d0240bf849fc39ad2a37c9da2b73fdd808c4f66b7425af2c

data/CHANGELOG.md

@@ -4,6 +4,23 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## 1.0.2
+
+### Fixed
+
+- Scheduled jobs enqueued through a `Setter` (i.e. `MyWorker.set(queue: "high").perform_in(60)` or `MyWorker.set(wait: 60).perform_async`) are no longer deferred. This matches the existing behavior of `MyWorker.perform_in` since scheduled jobs are not enqueued to run immediately anyway.
+- Added missing `require "set"` which could cause a `NameError` when deferred jobs were enqueued on Ruby < 3.2 in applications that did not already load the set library.
+- Deferred jobs that have not yet been enqueued are no longer silently dropped if an error is raised while enqueuing jobs. The error is still raised to the calling code, but the remaining jobs are retained so `Sidekiq.enqueue_deferred_jobs!` can be safely retried.
+- Duplicate unique jobs are no longer suppressed if they are enqueued to different queues.
+- Runtime options are no longer lost when a job is passed through `Sidekiq::DeferredJobs.defer_worker` outside of a defer block.
+- Fixed a `NameError` on application boot with Sidekiq 5.0 through 6.2 when `sidekiq/api` was loaded before this gem. Those versions define an unrelated `Sidekiq::Job` class in the API which was mistaken for the `Sidekiq::Job` mixin added in Sidekiq 6.3.
+- Jobs enqueued with `set(sync: true)` are no longer deferred; they run inline at the call site as Sidekiq intends.
+- A hash filter passed directly to `Sidekiq::DeferredJobs.defer` is now applied as a `sidekiq_options` filter. Previously it silently matched all jobs.
+
+### Changed
+
+- Filters passed to `Sidekiq.defer_jobs`, `Sidekiq.abort_deferred_jobs!`, and `Sidekiq.enqueue_deferred_jobs!` now raise an `ArgumentError` if they are not classes, modules, or hashes. Previously invalid filter values would silently match all jobs.
+
 ## 1.0.1
 
 ### Changed

data/README.md

@@ -4,13 +4,13 @@
 [![Ruby Style Guide](https://img.shields.io/badge/code_style-standard-brightgreen.svg)](https://github.com/testdouble/standard)
 [![Gem Version](https://badge.fury.io/rb/sidekiq-deferred_jobs.svg)](https://badge.fury.io/rb/sidekiq-deferred_jobs)
 
-This gem provides an enhancement to [Sidekiq](https://github.com/mperham/sidekiq) to defer enqueuing jobs until the end of a block of code. This is useful in situations where you need to better coordinate when jobs are enqueued to guard against race conditions or deduplicate jobs. In most cases, this provides no functional difference to your code; it just delays slightly when jobs are enqueued.
+This gem provides an enhancement to [Sidekiq](https://github.com/sidekiq/sidekiq) to defer enqueuing jobs until the end of a block of code. This is useful in situations where you need to better coordinate when jobs are enqueued to guard against race conditions or deduplicate jobs. In most cases, this provides no functional difference to your code; it just delays slightly when jobs are enqueued.
 
 ## Usage
 
-If you have a complex operation composed of several discrete service objects that each fire off Sidekiq jobs, but you need to coordinate when those jobs are actually run, you could use this code to do so. This might be to avoid race condition where you don't want some jobs running until the entire operation is finished or because some of the code fires off duplicate jobs that you'd like to squash. If you have a worker that automatically fires on data updates to send synchronization messages to another systems, you might want to have only a single job run at the end of the all the updates rather than sending multiple updates within a few milliseconds. This gem is designed to give you control over that situation rather than having to refactor code that may have side effects in other situations.
+If you have a complex operation composed of several discrete service objects that each fire off Sidekiq jobs, but you need to coordinate when those jobs are actually run, you could use this code to do so. This might be to avoid a race condition where you don't want some jobs running until the entire operation is finished, or because some of the code fires off duplicate jobs that you'd like to squash. If you have a worker that automatically fires on data updates to send synchronization messages to other systems, you might want to have only a single job run at the end of all the updates rather than sending multiple updates within a few milliseconds. This gem is designed to give you control over that situation rather than having to refactor code that may have side effects in other situations.
 
-If you are using either [the sidekiq-unique-jobs gem](https://github.com/mhenrixon/sidekiq-unique-jobs) or [Sidekiq Enterprise unique jobs](https://github.com/mperham/sidekiq/wiki/Ent-Unique-Jobs), then unique jobs equeued in the deferred jobs block will be suppressed. This is useful since Sidekiq can be so fast that duplicate jobs can be picked up by worker threads almost instataneously so the system never detects that duplicate jobs were being enqueued.
+If you are using either [the sidekiq-unique-jobs gem](https://github.com/mhenrixon/sidekiq-unique-jobs) or [Sidekiq Enterprise unique jobs](https://github.com/sidekiq/sidekiq/wiki/Ent-Unique-Jobs), then duplicate jobs deferred within the block will be collapsed into a single job when the block exits. A job is considered a duplicate if it has the same class, arguments, and queue as a job already enqueued by the same flush of deferred jobs. The collapse does not read the uniqueness mechanism's own lock configuration (i.e. `unique_args`, `lock_args`, or `unique_across_queues`); jobs that are duplicates only under those settings are all enqueued and the uniqueness mechanism resolves them when they are pushed. When jobs are collapsed, the first job and its runtime options are kept. This is useful since Sidekiq can be so fast that duplicate jobs can be picked up by worker threads almost instantaneously, so the system never detects that duplicate jobs were being enqueued. Jobs that do not declare a uniqueness constraint (`unique_for` for Sidekiq Enterprise, or `lock` for sidekiq-unique-jobs) are never collapsed; neither are jobs locked only `while_executing`.
 
 Using the scheduled jobs mechanism in Sidekiq to accomplish the same thing is less than ideal because the scheduling mechanism in Sidekiq is not designed to be very precise. If you schedule a job to run one second in the future, it might not run for several seconds.
 
@@ -35,18 +35,18 @@ end
 # All the jobs are now enqueued
 ```
 
-The workers are fired in an `ensure` block, so even if an error is raised, any jobs that would have been enqueued prior to the error will still be enqueued.
+The deferred jobs are enqueued from an `ensure` block, so even if an error is raised inside the block, any jobs that would have been enqueued prior to the error will still be enqueued.
 
-You can also pass a filter to `defer_jobs` to filter either by class or by `sidekiq_options`.
+You can also pass a filter to `defer_jobs` so that only some jobs are deferred. A filter can be a class, a module, or a hash of `sidekiq_options`. A job matches a class or module filter if the worker class is that class, a subclass of it, or includes that module. It matches a hash filter if all of the entries in the hash match the worker's `sidekiq_options` (including any options set at runtime with `set`). You can pass multiple filters; a job is deferred if it matches any of them.
 
 ```ruby
 class SomeWorker
-  include Sidekiq::Worker
+  include Sidekiq::Job
   sidekiq_options priority: "high"
 end
 
 class OtherWorker
-  include Sidekiq::Worker
+  include Sidekiq::Job
 end
 
 # Filter by worker class
@@ -70,35 +70,71 @@ end
 # The SomeWorker job will now be enqueued
 ```
 
-You can also pass `false` to `Sidekiq.defer_jobs` turn off deferral entirely within a block.
+Blocks can be nested. Deferred jobs are always enqueued at the end of the outermost block, and the filters from all of the enclosing blocks apply, so a job is deferred if it matches the filter on any block it is inside of.
 
 ```ruby
-Sidekiq.defer_jobs(false) do
+Sidekiq.defer_jobs(SomeWorker) do
+  Sidekiq.defer_jobs(OtherWorker) do
+    SomeWorker.perform_async(1)  # deferred by the outer block's filter
+    OtherWorker.perform_async(2) # deferred by the inner block's filter
+  end
+  # Neither job is enqueued here; the inner block does not flush jobs.
+end
+# Both jobs are now enqueued
+```
+
+You can pass `false` to `Sidekiq.defer_jobs` to turn off deferral entirely within a block. This is mostly useful inside a `defer_jobs` block to opt a section of code back into enqueuing jobs immediately.
+
+```ruby
+Sidekiq.defer_jobs do
   SomeWorker.perform_async(1)
-  # The SomeWorker job will be enqueued
+  # The SomeWorker job is deferred
+
+  Sidekiq.defer_jobs(false) do
+    OtherWorker.perform_async(2)
+    # The OtherWorker job is enqueued immediately
+  end
 end
 ```
 
-You can also manually control over when deferred jobs are enqueued or even remove previously deferred jobs.
+You can also manually control when deferred jobs are enqueued or even remove previously deferred jobs.
 
 ```ruby
 Sidekiq.defer_jobs do
   SomeWorker.perform_async(1)
 
-  # This will cancel SomeWorker.perform(1); it won't be enqueued
+  # This will cancel SomeWorker.perform_async(1); it won't be enqueued
   Sidekiq.abort_deferred_jobs!
 
   SomeWorker.perform_async(2)
-  # SomeWorker.perform(2) is not yet equeued
+  # SomeWorker.perform_async(2) is not yet enqueued
 
   Sidekiq.enqueue_deferred_jobs!
-  # SomeWorker.perform(2) will now be be equeued
+  # SomeWorker.perform_async(2) has now been enqueued
 end
 ```
 
-You can pass filters to the `Sidekiq.abort_deferred_jobs!` and `Sidekiq.enqueue_deferred_jobs!` methods if you want to enqueue or abort just specific jobs. These filters work the same as the fitlers to `Sidekiq.defer_jobs`.
+You can pass filters to the `Sidekiq.abort_deferred_jobs!` and `Sidekiq.enqueue_deferred_jobs!` methods if you want to enqueue or abort just specific jobs. These filters work the same as the class, module, and hash filters for `Sidekiq.defer_jobs` (`false` is not supported here). `Sidekiq.abort_deferred_jobs!` returns the jobs it removed; both methods do nothing when called outside of a `defer_jobs` block.
+
+Scheduled jobs (i.e. `perform_in`, `perform_at`, or setting the `at` option with `set`) are never deferred since they are not enqueued to run immediately anyway.
 
-Note that if you are running with a relational database you may want to use another mechanism to work with transactional data (i.e. the `after_commit` hook in ActiveRecord). However, if you have a single logical operation that contains multiple transactions, this mechanism could be a good fit. For example, if you have a complex business operation that updates multiple rows and calls external services, you may not want a single transaction since it could lock database rows for a long period creating performance problems. This gem could be used to orchestrate transactional logic for Sidekiq workers in systems with native transaction support.
+If an error is raised while deferred jobs are being enqueued (for instance, if Redis is briefly unavailable), the error will be raised to the calling code. Any jobs that had not yet been enqueued when the error occurred are retained in the deferred jobs list rather than being silently dropped. If the error came from an explicit call to `Sidekiq.enqueue_deferred_jobs!`, you can rescue it and call the method again to enqueue the remaining jobs without duplicating the ones that already made it to Redis.
+
+## Limitations
+
+- Only jobs enqueued with `perform_async` are deferred. Jobs enqueued through other mechanisms (`perform_bulk`, `Sidekiq::Client.push`, or ActiveJob with the Sidekiq adapter) are not intercepted and will be enqueued immediately.
+
+- `perform_async` does not return a job id for a deferred job. The job id is not assigned until the job is actually enqueued, so any code that captures the return value of `perform_async` should not be run inside a `defer_jobs` block.
+
+- The list of deferred jobs is stored in fiber-local storage. Jobs enqueued from other threads or fibers spawned inside a `defer_jobs` block (including code using lazy enumerators or async frameworks that switch fibers) will not be deferred.
+
+- If an error is raised by the automatic flush at the end of the outermost `defer_jobs` block, any jobs that had not been enqueued yet are lost; there is no way to retry them since the deferred job list is no longer reachable at that point.
+
+> [!IMPORTANT]
+> Deferred jobs are not persisted to Redis within the `defer_jobs` block. If your application crashes or is force killed before the block completes, any jobs that were deferred will be lost. This is a tradeoff to avoid the performance overhead of persisting jobs to Redis when they are deferred.
+
+> [!NOTE]
+> If you are running with a relational database you may want to use another mechanism to work with transactional data (i.e. the `after_commit` hook in ActiveRecord). However, if you have a single logical operation that contains multiple transactions, this mechanism could be a good fit. For example, if you have a complex business operation that updates multiple rows and calls external services, you may not want a single transaction since it could lock database rows for a long period creating performance problems. This gem could be used to orchestrate transactional logic for Sidekiq workers in systems with native transaction support.
 
 ## Installation
 
@@ -118,6 +154,15 @@ Or install it yourself as:
 $ gem install sidekiq-deferred_jobs
 ```
 
+No further setup is required; requiring the gem patches Sidekiq automatically.
+
+### Requirements
+
+- Ruby 2.5 or later
+- Sidekiq 5.0 or later
+
+The examples in this README use `Sidekiq::Job`, which was added in Sidekiq 6.3. If you are on an older version of Sidekiq, use `Sidekiq::Worker` instead.
+
 ## Contributing
 
 Open a pull request on GitHub.

data/VERSION

@@ -1 +1 @@
-1.0.1
+1.0.2

data/lib/sidekiq/deferred_jobs.rb

@@ -1,26 +1,36 @@
 # frozen_string_literal: true
 
+require "set"
 require "sidekiq"
 
 module Sidekiq
+  # Adds the ability to defer enqueuing Sidekiq jobs until the end of a block of code.
+  # See Sidekiq::DeferredJobs::DeferBlock for the methods that are added to `Sidekiq`
+  # itself (`Sidekiq.defer_jobs`, `Sidekiq.abort_deferred_jobs!`, and
+  # `Sidekiq.enqueue_deferred_jobs!`).
   module DeferredJobs
     class << self
       # Defer enqueuing Sidekiq workers within the block until the end of the block.
       # Any workers that normally would have been enqueued with a `perform_async` call
       # will instead be queued up and run in an ensure clause at the end of the block.
       #
+      # Blocks can be nested. Deferred jobs are only enqueued at the end of the outermost
+      # block and the filters from all of the enclosing blocks apply, so a job is deferred
+      # if it matches the filter on any block it is nested inside of.
+      #
       # @param filter [Array<Module>, Array<Hash>] An array of either classes, modules, or hashes.
       #        If this is provided, only workers that match either a class or module or which have
-      #        sidekiq_options that match a hash will be deferred. All other worker will be enqueued as normal.
-      # @return [void]
+      #        sidekiq_options that match a hash will be deferred. All other workers will be enqueued as normal.
+      # @return [Object] the return value of the block
       def defer(filter, &block)
+        new_filter = Filter.new(filter)
         jobs, filters = Thread.current[:sidekiq_deferred_jobs_jobs]
         unless jobs
           filters = []
           jobs = Jobs.new
           Thread.current[:sidekiq_deferred_jobs_jobs] = [jobs, filters]
         end
-        filters.push(Filter.new(filter))
+        filters.push(new_filter)
         begin
           yield
         ensure
@@ -33,9 +43,10 @@ module Sidekiq
       end
 
       # Disable deferred workers within the block. All workers will be enqueued normally
-      # within the block.
+      # within the block. Any jobs already deferred by an enclosing block are left alone
+      # and will still be enqueued when that block exits.
       #
-      # @return [void]
+      # @return [Object] the return value of the block
       def undeferred(&block)
         save_val = Thread.current[:sidekiq_deferred_jobs_jobs]
         begin
@@ -47,17 +58,20 @@ module Sidekiq
       end
 
       # Return true if the specified class with optional options should be deferred.
+      # Jobs scheduled to run in the future and jobs that run inline are never deferred.
       #
       # @param klass [Class] A Sidekiq worker class
-      # @param opts [Hash, Nil] Optionsl options set at runtime for the worker.
-      # @return Boolean
+      # @param opts [Hash, Nil] Optional options set at runtime for the worker.
+      # @return [Boolean]
       def defer?(klass, opts = nil)
+        return false if scheduled_opts?(opts) || inline_opts?(opts)
         _jobs, filters = Thread.current[:sidekiq_deferred_jobs_jobs]
         return false if filters.nil?
         filters.any? { |filter| filter.match?(klass, opts) }
       end
 
-      # Schedule a worker to be run at the end of the outermost defer block.
+      # Schedule a worker to be enqueued at the end of the outermost defer block.
+      # If there is no defer block in progress, the worker is enqueued immediately.
       #
       # @param klass [Class] Sidekiq worker class
       # @param args [Array] Sidekiq job arguments
@@ -67,12 +81,64 @@ module Sidekiq
         jobs, _filters = Thread.current[:sidekiq_deferred_jobs_jobs]
         if jobs
           jobs.defer(klass, args, opts)
+        else
+          enqueue_job(klass, args, opts)
+        end
+      end
+
+      # Return true if the options indicate a job scheduled to run in the future.
+      # Scheduled jobs are never deferred since they are not enqueued immediately anyway.
+      #
+      # @param opts [Hash, Nil] Runtime options set for a job.
+      # @return [Boolean]
+      # @api private
+      def scheduled_opts?(opts)
+        return false unless opts
+        !!(opts["at"] || opts[:at])
+      end
+
+      # Return true if the options indicate a job that runs inline at the call site
+      # rather than being pushed to a queue. Inline jobs are never deferred.
+      #
+      # @param opts [Hash, Nil] Runtime options set for a job.
+      # @return [Boolean]
+      # @api private
+      def inline_opts?(opts)
+        return false unless opts
+        opts["sync"] == true || opts[:sync] == true
+      end
+
+      # Enqueue a job through the normal Sidekiq mechanism.
+      #
+      # @param klass [Class] Sidekiq worker class
+      # @param args [Array] Sidekiq job arguments
+      # @param opts [Hash, Nil] Optional runtime options for the job
+      # @return [Object] the value returned by `perform_async`
+      # @api private
+      def enqueue_job(klass, args, opts = nil)
+        if opts
+          klass.set(opts).perform_async(*args)
         else
           klass.perform_async(*args)
         end
       end
+
+      # Merge runtime options with the worker class sidekiq_options.
+      #
+      # @param klass [Class] Sidekiq worker class
+      # @param opts [Hash, Nil] Optional runtime options for the job
+      # @return [Hash] the merged options with string keys
+      # @api private
+      def worker_options(klass, opts = nil)
+        if opts
+          klass.sidekiq_options.merge(opts.transform_keys(&:to_s))
+        else
+          klass.sidekiq_options
+        end
+      end
     end
 
+    # Methods added to the `Sidekiq` module itself for controlling deferred jobs.
     module DeferBlock
       # Defer enqueuing Sidekiq workers within the block until the end of the block.
       # Any workers that normally would have been enqueued with a `perform_async` call
@@ -80,10 +146,12 @@ module Sidekiq
       #
       # @param filter [Array<Module>, Hash, false] Optional filter on which workers should be deferred.
       #                If a filter is specified, only matching workers will be deferred. To match the
-      #                filter, the worker must either be the class specfied or include the module or
-      #                have sidekiq_options that match the specified hash. If the filter is `false`
-      #                then job deferral will be disabled entirely within the block.
-      # @return [void]
+      #                filter, the worker must either be the class specified (or a subclass of it),
+      #                include the module, or have sidekiq_options that match the specified hash.
+      #                Multiple filters can be specified and a worker will be deferred if it matches
+      #                any of them. If the filter is `false` then job deferral will be disabled
+      #                entirely within the block.
+      # @return [Object] the return value of the block
       def defer_jobs(*filter, &block)
         if filter.size == 1 && filter.first == false
           Sidekiq::DeferredJobs.undeferred(&block)
@@ -92,10 +160,12 @@ module Sidekiq
         end
       end
 
-      # Abort any already deferred Sidkiq workers in the current `defer_job` block.
+      # Abort any already deferred Sidekiq workers in the current `defer_jobs` block.
       # If a filter is specified, then only matching Sidekiq jobs will be aborted.
+      # Does nothing if called outside of a `defer_jobs` block.
       #
-      # @param filter [Array<Module>, Hash, false] See #defer_job for filter specification.
+      # @param filter [Array<Module>, Hash] See #defer_jobs for filter specification.
+      #                Unlike #defer_jobs, `false` is not a valid filter here.
       # @return [Array<Sidekiq::DeferredJobs::Job>] the jobs that were aborted
       def abort_deferred_jobs!(*filter)
         jobs, _filters = Thread.current[:sidekiq_deferred_jobs_jobs]
@@ -106,10 +176,12 @@ module Sidekiq
         end
       end
 
-      # Immediately enqueue any already deferred Sidkiq workers in the current `defer_job` block.
+      # Immediately enqueue any already deferred Sidekiq workers in the current `defer_jobs` block.
       # If a filter is specified, then only matching Sidekiq jobs will be enqueued.
+      # Does nothing if called outside of a `defer_jobs` block.
       #
-      # @param filter [Array<Module>, Hash, false] See #defer_job for filter specification.
+      # @param filter [Array<Module>, Hash] See #defer_jobs for filter specification.
+      #                Unlike #defer_jobs, `false` is not a valid filter here.
       # @return [void]
       def enqueue_deferred_jobs!(*filter)
         jobs, _filters = Thread.current[:sidekiq_deferred_jobs_jobs]
@@ -122,6 +194,8 @@ module Sidekiq
 
     # Override logic for Sidekiq::Worker.
     module DeferredWorker
+      # @return [Object] the job id if the job was enqueued. If the job was deferred, the
+      #         return value is unspecified since no job id exists until the job is enqueued.
       def perform_async(*args)
         if Sidekiq::DeferredJobs.defer?(self)
           Sidekiq::DeferredJobs.defer_worker(self, args)
@@ -133,6 +207,8 @@ module Sidekiq
 
     # Override logic for Sidekiq::Worker::Setter.
     module DeferredSetter
+      # @return [Object] the job id if the job was enqueued. If the job was deferred, the
+      #         return value is unspecified since no job id exists until the job is enqueued.
       def perform_async(*args)
         if Sidekiq::DeferredJobs.defer?(@klass, @opts)
           Sidekiq::DeferredJobs.defer_worker(@klass, args, @opts)
@@ -145,7 +221,11 @@ module Sidekiq
     # Logic for filtering jobs by worker class and/or sidekiq_options.
     class Filter
       def initialize(filters)
-        @filters = Array(filters).flatten
+        @filters = (filters.is_a?(Hash) ? [filters] : Array(filters).flatten)
+        invalid = @filters.reject { |filter| filter.is_a?(Module) || filter.is_a?(Hash) }
+        unless invalid.empty?
+          raise ArgumentError, "Filters must be classes, modules, or hashes; got #{invalid.collect(&:inspect).join(", ")}"
+        end
       end
 
       # @return [Boolean] true if the job matches the filters.
@@ -154,11 +234,9 @@ module Sidekiq
         @filters.any? do |filter|
           if filter.is_a?(Module)
             klass <= filter
-          elsif filter.is_a?(Hash)
-            worker_options = (opts ? klass.sidekiq_options.merge(opts.transform_keys(&:to_s)) : klass.sidekiq_options)
-            filter.all? { |key, value| worker_options[key.to_s] == value }
           else
-            filter
+            worker_options = Sidekiq::DeferredJobs.worker_options(klass, opts)
+            filter.all? { |key, value| worker_options[key.to_s] == value }
           end
         end
       end
@@ -171,6 +249,7 @@ module Sidekiq
     class Jobs
       def initialize
         @jobs = []
+        @dedup_keys = Set.new
       end
 
       # Add a job to the deferred job list.
@@ -196,39 +275,50 @@ module Sidekiq
 
       # Enqueue any deferred jobs that match the filter.
       #
-      # @param filters [Array<Module>, Array<Hash>] Filter for jobs to clear
+      # If a worker declares a uniqueness constraint (either with Sidekiq Enterprise or the
+      # sidekiq-unique-jobs gem), then duplicate jobs with the same class, arguments, and queue
+      # are collapsed into a single job within this call.
+      #
+      # If enqueuing a job raises an error, the error will be propagated to the caller
+      # and any jobs that have not yet been enqueued will be retained in the deferred
+      # job list so they are not silently lost and can be enqueued again. The keys of
+      # unique jobs that were already enqueued are remembered until a call completes
+      # without an error so that retrying does not enqueue duplicates.
+      #
+      # @param filters [Array<Module>, Array<Hash>] Filter for jobs to enqueue
       # @return [void]
       def enqueue!(filters = nil)
         filter = Filter.new(filters)
-        remaining_jobs = []
+        unmatched_jobs = []
+        pending_jobs = @jobs
         begin
-          duplicates = Set.new
-          @jobs.each do |job|
+          until pending_jobs.empty?
+            job = pending_jobs.first
             if filter.match?(job.klass, job.opts)
-              if unique_job?(job.klass, job.opts)
-                next if duplicates.include?([job.klass, job.args])
-                duplicates << [job.klass, job.args]
-              end
-              if job.opts
-                job.klass.set(job.opts).perform_async(*job.args)
-              else
-                job.klass.perform_async(*job.args)
+              options = Sidekiq::DeferredJobs.worker_options(job.klass, job.opts)
+              dedup_key = nil
+              dedup_key = [job.klass, job.args, options["queue"]] if unique_job?(options)
+              unless dedup_key && @dedup_keys.include?(dedup_key)
+                Sidekiq::DeferredJobs.enqueue_job(job.klass, job.args, job.opts)
               end
+              @dedup_keys << dedup_key if dedup_key
             else
-              remaining_jobs << job
+              unmatched_jobs << job
             end
+            pending_jobs.shift
           end
         ensure
-          @jobs = remaining_jobs
+          @jobs = unmatched_jobs + pending_jobs
+          @dedup_keys.clear if pending_jobs.empty?
         end
       end
 
       private
 
-      # @return [Boolean] true if the worker support a uniqueness constraint
-      def unique_job?(klass, opts)
-        enterprise_option = worker_options(klass, opts)["unique_for"] if defined?(Sidekiq::Enterprise)
-        unique_jobs_option = worker_options(klass, opts)["lock"] if defined?(SidekiqUniqueJobs)
+      # @return [Boolean] true if the options declare a uniqueness constraint
+      def unique_job?(options)
+        enterprise_option = options["unique_for"] if defined?(Sidekiq::Enterprise)
+        unique_jobs_option = options["lock"] if defined?(SidekiqUniqueJobs)
 
         if enterprise_option
           true
@@ -238,15 +328,6 @@ module Sidekiq
           false
         end
       end
-
-      # Merge runtime options with the worker class sidekiq_options.
-      def worker_options(klass, opts)
-        if opts
-          klass.sidekiq_options.merge(opts.transform_keys(&:to_s))
-        else
-          klass.sidekiq_options
-        end
-      end
     end
   end
 end

data/lib/sidekiq/deferred_jobs_ext.rb

@@ -6,7 +6,10 @@
 
 Sidekiq.extend(Sidekiq::DeferredJobs::DeferBlock)
 
-if defined?(Sidekiq::Job)
+# Sidekiq::Job is the job mixin on Sidekiq 6.3+, but older versions define an
+# unrelated Sidekiq::Job class in sidekiq/api.rb, so check for the mixin's
+# ClassMethods constant rather than just the presence of the Sidekiq::Job constant.
+if defined?(Sidekiq::Job) && Sidekiq::Job.const_defined?(:ClassMethods)
   Sidekiq::Job::ClassMethods.prepend(Sidekiq::DeferredJobs::DeferredWorker)
   Sidekiq::Job::Setter.prepend(Sidekiq::DeferredJobs::DeferredSetter)
 else

data/sidekiq-deferred_jobs.gemspec

@@ -35,7 +35,5 @@ Gem::Specification.new do |spec|
 
   spec.add_dependency "sidekiq", ">= 5.0"
 
-  spec.add_development_dependency "bundler"
-
   spec.required_ruby_version = ">= 2.5"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: sidekiq-deferred_jobs
 version: !ruby/object:Gem::Version
-  version: 1.0.1
+  version: 1.0.2
 platform: ruby
 authors:
 - Brian Durand
@@ -23,20 +23,6 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '5.0'
-- !ruby/object:Gem::Dependency
-  name: bundler
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
-  type: :development
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
 email:
 - bbdurand@gmail.com
 executables: []