sidekiq-deferred_jobs 1.0.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 74140b3a7726b81f99deb8eee4f6b12cbe306a89fc47ee7c1f9c1498b6b39588
+  data.tar.gz: aefb420df0176f907074fdbce0a1a6c40f85209b5143ae41342f2cc78ddcf66a
+SHA512:
+  metadata.gz: fd4ef0a5ead779291359d18c8388ddc5fa1ee8a5615bf4e1d11add7f0ed605919979380cba966ed8c8422e35d03e4d5910470c405145f31c1f5d299a7e07747e
+  data.tar.gz: f1ea50c549baea07d8b8388455c948d052060ee7e68effe12e85e261aa1b6c330702819b84c42ecfa4d11b95a4c7af74bf031d154a372f097461342b6c3f002a

data/CHANGELOG.md

@@ -0,0 +1,10 @@
+# Changelog
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## 1.0.0
+
+### Added
+- Add behavior to Sidekiq to allow deferring enqueing jobs until after a block of code finishes.

data/MIT-LICENSE

@@ -0,0 +1,20 @@
+Copyright 2021 Brian Durand
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,130 @@
+[![Continuous Integration](https://github.com/bdurand/sidekiq-deferred_jobs/actions/workflows/continuous_integration.yml/badge.svg)](https://github.com/bdurand/sidekiq-deferred_jobs/actions/workflows/continuous_integration.yml)
+[![Maintainability](https://api.codeclimate.com/v1/badges/3f5fb49ca1d03f698d5b/maintainability)](https://codeclimate.com/github/bdurand/sidekiq-deferred_jobs/maintainability)
+[![Test Coverage](https://api.codeclimate.com/v1/badges/3f5fb49ca1d03f698d5b/test_coverage)](https://codeclimate.com/github/bdurand/sidekiq-deferred_jobs/test_coverage)
+[![Ruby Style Guide](https://img.shields.io/badge/code_style-standard-brightgreen.svg)](https://github.com/testdouble/standard)
+
+# Sidekiq Deferred Jobs
+
+This gem provides an enhancement to [Sidekiq](https://github.com/mperham/sidekiq) to defer enqueuing jobs until the end of a block of code. This is useful in situations where you need to better coordinate when jobs are enqueued to guard against race conditions or deduplicate jobs. In most cases, this provides no functional difference to your code; it just delays slightly when jobs are enqueued.
+
+## Usage
+
+If you have a complex operation composed of several discrete service objects that each fire off Sidekiq jobs, but you need to coordinate when those jobs are actually run, you could use this code to do so. This might be to avoid race condition where you don't want some jobs running until the entire operation is finished or because some of the code fires off duplicate jobs that you'd like to squash. If you have a worker that automatically fires on data updates to send synchronization messages to another systems, you might want to have only a single job run at the end of the all the updates rather than sending multiple updates within a few milliseconds. This gem is designed to give you control over that situation rather than having to refactor code that may have side effects in other situations.
+
+If you are using either [the sidekiq-unique-jobs gem](https://github.com/mhenrixon/sidekiq-unique-jobs) or [Sidekiq Enterprise unique jobs](https://github.com/mperham/sidekiq/wiki/Ent-Unique-Jobs), then unique jobs equeued in the deferred jobs block will be suppressed. This is useful since Sidekiq can be so fast that duplicate jobs can be picked up by worker threads almost instataneously so the system never detects that duplicate jobs were being enqueued.
+
+Using the scheduled jobs mechanism in Sidekiq to accomplish the same thing is less than ideal because the scheduling mechanism in Sidekiq is not designed to be very precise. If you schedule a job to run one second in the future, it might not run for several seconds.
+
+Normally, Sidekiq will enqueue jobs immediately.
+
+```ruby
+ids.each do |id|
+  SomeWorker.perform_async(id)
+  # Each SomeWorker job is enqueued here
+end
+```
+
+Calling `Sidekiq.defer_jobs` with a block prevents any workers from being immediately enqueued within the block.
+
+```ruby
+Sidekiq.defer_jobs do
+  ids.each do |id|
+    SomeWorker.perform_async(id)
+    # SomeWorker jobs will not be enqueued here
+  end
+end
+# All the jobs are now enqueued
+```
+
+The workers are fired in an `ensure` block, so even if an error is raised, any jobs that would have been enqueued prior to the error will still be enqueued.
+
+You can also pass a filter to `defer_jobs` to filter either by class or by `sidekiq_options`.
+
+```ruby
+class SomeWorker
+  include Sidekiq::Worker
+  sidekiq_options priority: "high"
+end
+
+class OtherWorker
+  include Sidekiq::Worker
+end
+
+# Filter by worker class
+Sidekiq.defer_jobs(SomeWorker) do
+  SomeWorker.perform_async(1)
+  # The SomeWorker job will not be enqueued yet
+
+  OtherWorker.perform_async(2)
+  # The OtherWorker job will be enqueued here since it doesn't match the filter
+end
+# The SomeWorker job will now be enqueued
+
+# Filter by sidekiq_options
+Sidekiq.defer_jobs(priority: "high") do
+  SomeWorker.perform_async(3)
+  # The SomeWorker job will not be enqueued yet
+
+  OtherWorker.perform_async(4)
+  # The OtherWorker job will be enqueued here since it doesn't match the filter
+end
+# The SomeWorker job will now be enqueued
+```
+
+You can also pass `false` to `Sidekiq.defer_jobs` turn off deferral entirely within a block.
+
+```ruby
+Sidekiq.defer_jobs(false) do
+  SomeWorker.perform_async(1)
+  # The SomeWorker job will be enqueued
+end
+```
+
+You can also manually control over when deferred jobs are enqueued or even remove previously deferred jobs.
+
+```ruby
+Sidekiq.defer_jobs do
+  SomeWorker.perform_async(1)
+
+  # This will cancel SomeWorker.perform(1); it won't be enqueued
+  Sidekiq.abort_deferred_jobs!
+
+  SomeWorker.perform_async(2)
+  # SomeWorker.perform(2) is not yet equeued
+
+  Sidekiq.enqueue_deferred_jobs!
+  # SomeWorker.perform(2) will now be be equeued
+end
+```
+
+You can pass filters to the `Sidekiq.abort_deferred_jobs!` and `Sidekiq.enqueue_deferred_jobs!` methods if you want to enqueue or abort just specific jobs. These filters work the same as the fitlers to `Sidekiq.defer_jobs`.
+
+Note that if you are running with a relational database you may want to use another mechanism to work with transactional data (i.e. the `after_commit` hook in ActiveRecord). However, if you have a single logical operation that contains multiple transactions, this mechanism could be a good fit. For example, if you have a complex business operation that updates multiple rows and calls external services, you may not want a single transaction since it could lock database rows for a long period creating performance problems. This gem could be used to orchestrate transactional logic for Sidekiq workers in systems with native transaction support.
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'sidekiq-deferred_jobs'
+```
+
+And then execute:
+```bash
+$ bundle
+```
+
+Or install it yourself as:
+```bash
+$ gem install sidekiq-deferred_jobs
+```
+
+## Contributing
+
+Open a pull request on GitHub.
+
+Please use the [standardrb](https://github.com/testdouble/standard) syntax and lint your code with `standardrb --fix` before submitting.
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).

data/VERSION

@@ -0,0 +1 @@
+1.0.0

data/lib/sidekiq/deferred_jobs.rb

@@ -0,0 +1,243 @@
+# frozen_string_literal: true
+
+require "sidekiq"
+
+module Sidekiq
+  module DeferredJobs
+    class << self
+      # Defer enqueuing Sidekiq workers within the block until the end of the block.
+      # Any workers that normally would have been enqueued with a `perform_async` call
+      # will instead be queued up and run in an ensure clause at the end of the block.
+      # @param filter [Array<Module>, Array<Hash>] An array of either classes, modules, or hashes.
+      #        If this is provided, only workers that match either a class or module or which have
+      #        sidekiq_options that match a hash will be deferred. All other worker will be enqueued as normal.
+      # @return [void]
+      def defer(filter, &block)
+        jobs, filters = Thread.current[:sidekiq_deferred_jobs_jobs]
+        unless jobs
+          filters = []
+          jobs = Jobs.new
+          Thread.current[:sidekiq_deferred_jobs_jobs] = [jobs, filters]
+        end
+        filters.push(Filter.new(filter))
+        begin
+          yield
+        ensure
+          filters.pop
+          if filters.empty?
+            Thread.current[:sidekiq_deferred_jobs_jobs] = nil
+            jobs.enqueue!
+          end
+        end
+      end
+
+      # Disable deferred workers within the block. All workers will be enqueued normally
+      # within the block.
+      # @return [void]
+      def undeferred(&block)
+        save_val = Thread.current[:sidekiq_deferred_jobs_jobs]
+        begin
+          Thread.current[:sidekiq_deferred_jobs_jobs] = nil
+          yield
+        ensure
+          Thread.current[:sidekiq_deferred_jobs_jobs] = save_val
+        end
+      end
+
+      # Return true if the specified class with optional options should be deferred.
+      # @param klass [Class] A Sidekiq worker class
+      # @param opts [Hash, Nil] Optionsl options set at runtime for the worker.
+      # @return Boolean
+      def defer?(klass, opts = nil)
+        _jobs, filters = Thread.current[:sidekiq_deferred_jobs_jobs]
+        return false if filters.nil?
+        filters.any? { |filter| filter.match?(klass, opts) }
+      end
+
+      # Schedule a worker to be run at the end of the outermost defer block.
+      # @param klass [Class] Sidekiq worker class
+      # @param args [Array] Sidekiq job arguments
+      # @param opts [Hash, Nil] Optional sidekiq options specified for the job
+      # @return [void]
+      def defer_worker(klass, args, opts = nil)
+        jobs, _filters = Thread.current[:sidekiq_deferred_jobs_jobs]
+        if jobs
+          jobs.defer(klass, args, opts)
+        else
+          klass.perform_async(*args)
+        end
+      end
+    end
+
+    module DeferBlock
+      # Defer enqueuing Sidekiq workers within the block until the end of the block.
+      # Any workers that normally would have been enqueued with a `perform_async` call
+      # will instead be queued up and run in an ensure clause at the end of the block.
+      # @param *filter [Module>, Hash, FalseClass] Optional filter on which workers should be deferred.
+      #                If a filter is specified, only matching workers will be deferred. To match the
+      #                filter, the worker must either be the class specfied or include the module or
+      #                have sidekiq_options that match the specified hash. If the filter is `false`
+      #                then job deferral will be disabled entirely within the block.
+      # @return [void]
+      def defer_jobs(*filter, &block)
+        if filter.size == 1 && filter.first == false
+          Sidekiq::DeferredJobs.undeferred(&block)
+        else
+          Sidekiq::DeferredJobs.defer(filter, &block)
+        end
+      end
+
+      # Abort any already deferred Sidkiq workers in the current `defer_job` block.
+      # If a filter is specified, then only matching Sidekiq jobs will be aborted.
+      # @param *filter See #defer_job for filter specification.
+      # @return [Array<Sidekiq::DeferredJobs::Job>] the jobs that were aborted
+      def abort_deferred_jobs!(*filter)
+        jobs, _filters = Thread.current[:sidekiq_deferred_jobs_jobs]
+        if jobs
+          jobs.clear!(filter)
+        else
+          []
+        end
+      end
+
+      # Immediately enqueue any already deferred Sidkiq workers in the current `defer_job` block.
+      # If a filter is specified, then only matching Sidekiq jobs will be enqueued.
+      # @param *filter See #defer_job for filter specification.
+      # @return [void]
+      def enqueue_deferred_jobs!(*filter)
+        jobs, _filters = Thread.current[:sidekiq_deferred_jobs_jobs]
+        if jobs
+          Sidekiq::DeferredJobs.undeferred { jobs.enqueue!(filter) }
+        end
+        nil
+      end
+    end
+
+    # Override logic for Sidekiq::Worker.
+    module DeferredWorker
+      def perform_async(*args)
+        if Sidekiq::DeferredJobs.defer?(self)
+          Sidekiq::DeferredJobs.defer_worker(self, args)
+        else
+          super
+        end
+      end
+    end
+
+    # Override logic for Sidekiq::Worker::Setter.
+    module DeferredSetter
+      def perform_async(*args)
+        if Sidekiq::DeferredJobs.defer?(@klass, @opts)
+          Sidekiq::DeferredJobs.defer_worker(@klass, args, @opts)
+        else
+          super
+        end
+      end
+    end
+
+    # Logic for filtering jobs by worker class and/or sidekiq_options.
+    class Filter
+      def initialize(filters)
+        @filters = Array(filters).flatten
+      end
+
+      # @return [Boolean] true if the job matches the filters.
+      def match?(klass, opts = nil)
+        return true if @filters.empty?
+        @filters.any? do |filter|
+          if filter.is_a?(Module)
+            klass <= filter
+          elsif filter.is_a?(Hash)
+            worker_options = (opts ? klass.sidekiq_options.merge(opts.transform_keys(&:to_s)) : klass.sidekiq_options)
+            filter.all? { |key, value| worker_options[key.to_s] == value }
+          else
+            filter
+          end
+        end
+      end
+    end
+
+    # Data structure to hold job information.
+    Job = Struct.new(:klass, :args, :opts)
+
+    # Class for holding deferred jobs.
+    class Jobs
+      def initialize
+        @jobs = []
+      end
+
+      # Add a job to the deferred job list.
+      # @param klass [Class] Sidekiq worker class.
+      # @param args [Array] Sidekiq job arguments
+      # @param opts [Hash, Nil] optional runtime jobs options
+      def defer(klass, args, opts = nil)
+        @jobs << Job.new(klass, args&.dup, opts&.dup)
+      end
+
+      # Clear any deferred jobs that match the filter.
+      # @filter [Array<Module>, Array<Hash>] Filter for jobs to clear
+      def clear!(filters = nil)
+        filter = Filter.new(filters)
+        cleared_jobs = @jobs.select { |job| filter.match?(job.klass, job.opts) }
+        @jobs -= cleared_jobs
+        cleared_jobs
+      end
+
+      # Enqueue any deferred jobs that match the filter.
+      # @filter [Array<Module>, Array<Hash>] Filter for jobs to clear
+      def enqueue!(filters = nil)
+        filter = Filter.new(filters)
+        remaining_jobs = []
+        begin
+          duplicates = Set.new
+          @jobs.each do |job|
+            if filter.match?(job.klass, job.opts)
+              if unique_job?(job.klass, job.opts)
+                next if duplicates.include?([job.klass, job.args])
+                duplicates << [job.klass, job.args]
+              end
+              if job.opts
+                job.klass.set(job.opts).perform_async(*job.args)
+              else
+                job.klass.perform_async(*job.args)
+              end
+            else
+              remaining_jobs << job
+            end
+          end
+        ensure
+          @jobs = remaining_jobs
+        end
+      end
+
+      private
+
+      # @return [Boolean] true if the worker support a uniqueness constraint
+      def unique_job?(klass, opts)
+        enterprise_option = worker_options(klass, opts)["unique_for"] if defined?(Sidekiq::Enterprise)
+        unique_jobs_option = worker_options(klass, opts)["lock"] if defined?(SidekiqUniqueJobs)
+
+        if enterprise_option
+          true
+        elsif unique_jobs_option
+          unique_jobs_option.to_s != "while_executing"
+        else
+          false
+        end
+      end
+
+      # Merge runtime options with the worker class sidekiq_options.
+      def worker_options(klass, opts)
+        if opts
+          klass.sidekiq_options.merge(opts.transform_keys(&:to_s))
+        else
+          klass.sidekiq_options
+        end
+      end
+    end
+  end
+end
+
+Sidekiq.extend(Sidekiq::DeferredJobs::DeferBlock)
+Sidekiq::Worker::ClassMethods.prepend(Sidekiq::DeferredJobs::DeferredWorker)
+Sidekiq::Worker::Setter.prepend(Sidekiq::DeferredJobs::DeferredSetter)

data/lib/sidekiq-deferred_jobs.rb

@@ -0,0 +1,3 @@
+# frozen_string_literal: true
+
+require_relative "sidekiq/deferred_jobs"

data/sidekiq-deferred_jobs.gemspec

@@ -0,0 +1,34 @@
+Gem::Specification.new do |spec|
+  spec.name = "sidekiq-deferred_jobs"
+  spec.version = File.read(File.expand_path("../VERSION", __FILE__)).strip
+  spec.authors = ["Brian Durand"]
+  spec.email = ["bbdurand@gmail.com"]
+
+  spec.summary = "Adds ability to defer the enqueuing of Sidekiq workers until the end of a block of code."
+  spec.homepage = "https://github.com/bdurand/sidekiq-deferred_jobs"
+  spec.license = "MIT"
+
+  # Specify which files should be added to the gem when it is released.
+  # The `git ls-files -z` loads the files in the RubyGem that have been added into git.
+  ignore_files = %w[
+    .
+    Appraisals
+    Gemfile
+    Gemfile.lock
+    Rakefile
+    bin/
+    gemfiles/
+    spec/
+  ]
+  spec.files = Dir.chdir(File.expand_path("..", __FILE__)) do
+    `git ls-files -z`.split("\x0").reject { |f| ignore_files.any? { |path| f.start_with?(path) } }
+  end
+
+  spec.require_paths = ["lib"]
+
+  spec.add_dependency "sidekiq", ">= 5.0"
+
+  spec.add_development_dependency "bundler"
+
+  spec.required_ruby_version = ">= 2.5"
+end

metadata

@@ -0,0 +1,79 @@
+--- !ruby/object:Gem::Specification
+name: sidekiq-deferred_jobs
+version: !ruby/object:Gem::Version
+  version: 1.0.0
+platform: ruby
+authors:
+- Brian Durand
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2021-08-30 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: sidekiq
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '5.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '5.0'
+- !ruby/object:Gem::Dependency
+  name: bundler
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+description: 
+email:
+- bbdurand@gmail.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- CHANGELOG.md
+- MIT-LICENSE
+- README.md
+- VERSION
+- lib/sidekiq-deferred_jobs.rb
+- lib/sidekiq/deferred_jobs.rb
+- sidekiq-deferred_jobs.gemspec
+homepage: https://github.com/bdurand/sidekiq-deferred_jobs
+licenses:
+- MIT
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '2.5'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 3.0.3
+signing_key: 
+specification_version: 4
+summary: Adds ability to defer the enqueuing of Sidekiq workers until the end of a
+  block of code.
+test_files: []