sidekiq-disposal 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 8ecd41e2b17b1168b119bf60002c1680c960561f0f4c1d972648a08386ed1423
+  data.tar.gz: 14df2e7593177e8c2fb90d96d1acbe3ed57206987cb59763e06fdf9db4a7ba63
+SHA512:
+  metadata.gz: 9e5e2a2084643fe72cafc7cbe7eb44dfacf500a63744d75585a6a38f63f8bf853090589183de163e487fbbeb975cd3e8e3ee8f11459c5b3e03a0ef890da2770e
+  data.tar.gz: 7bc4001093d88ef742f20102b5b11d98087726e9b076a7e733c8f695c200799a9c11282ef7c43d6b5d7de05fbd6a879ff2c83638e79ab86eaecc90b3e1aec24f

data/.rspec

@@ -0,0 +1,3 @@
+--format progress 
+--color
+--require spec_helper

data/.standard.yml

@@ -0,0 +1,6 @@
+# For available configuration options, see:
+#   https://github.com/standardrb/standard
+ruby_version: 3.0
+parallel: true
+ignore:
+  - tmp/**/*

data/CHANGELOG.md

@@ -0,0 +1,10 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+- A new thing!

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2024 Hazel Bachrach, Steven Harman
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,184 @@
+# Sidekiq::Disposal
+
+A [Sidekiq][sidekiq] extension mark Sidekiq Jobs to be disposed of based on the Job ID, Batch ID, or Job Class.
+Disposal here means to either `:kill` the Job (send to the Dead queue) or `:discard` it (throw it away), at the time the job is picked up and processed by Sidekiq.
+A disposed Job's `#perform` method will _not_ be called.
+
+Disposing of queued Jobs is particularly useful as a mitigation technique during an incident.
+For example, an issue with a 3rd party API that causes Jobs of a certain Class to take longer than expected/normal to run.
+Or a code change/edge case that unexpectedly fans out more than expected, enqueuing a large volume of Jobs which then drown out other Jobs.
+Or… any number of other ways that some Job, Batch, or Job Class has been enqueued, but you [don't want it to actually run][cancel-jobs].
+
+`sidekiq-disposal` has your back!
+
+## Installation
+
+Install the gem and add to the application's Gemfile by executing:
+
+```console
+bundle add UPDATE_WITH_YOUR_GEM_NAME_IMMEDIATELY_AFTER_RELEASE_TO_RUBYGEMS_ORG
+```
+
+If bundler is not being used to manage dependencies, install the gem by executing:
+
+```console
+gem install UPDATE_WITH_YOUR_GEM_NAME_IMMEDIATELY_AFTER_RELEASE_TO_RUBYGEMS_ORG
+```
+
+## Usage
+
+From a console (Rails console, or the like) you need a `Sidekiq::Disposal::Client` instance, which is used to `#mark` a Job, Batch, or Job Class to be disposed.
+
+```ruby
+client = Sidekiq::Disposal::Client.new
+```
+
+### Marking to Kill
+
+A Job marked to be killed means it will be moved to the Dead queue.
+
+```
+# Mark a specific Job, by its ID to be killed
+client.mark(:kill, :jid, some_job_id)
+
+# Mark a Batch of Jobs to be killed, by Batch ID
+client.mark(:kill, :bid, some_batch_id)
+
+# Mark an entire Job Class to be killed
+client.mark(:kill, :bid, "SomeJobClass")
+```
+
+A Job, Batch, or Job Class can also be `#unmarked` for disposal via a corresponding API.
+
+```
+# Un-mark a specific Job from being killed, by Job ID
+client.unmark(:kill, :jid, some_job_id)
+
+# Un-mark a Batch of Jobs from being killed, by Batch ID
+client.unmark(:kill, :bid, some_batch_id)
+
+# Un-mark an entire Job Class from being killed
+client.unmark(:kill, :bid, "SomeJobClass")
+```
+
+### Marking to Drop
+
+Similarly, a Job, Batch, or Job Class can be marked to be dropped.
+Dropped jobs are discarded by Sidekiq - think of them as simply being deleted from the queue, without ever being run.
+
+```
+# Mark a specific Job, by its ID to be dropped
+client.mark(:drop, :jid, some_job_id)
+
+# Mark a Batch of Jobs to be dropped, by Batch ID
+client.mark(:drop, :bid, some_batch_id)
+
+# Mark an entire Job Class to be dropped
+client.mark(:drop, :bid, "SomeJobClass")
+```
+
+And again, there is a corresponding API for un-marking a Job, Batch, or Job Class from being dropped.
+
+```
+# Un-mark a specific Job from being dropped, by Job ID
+client.unmark(:drop, :jid, some_job_id)
+
+# Un-mark a Batch of Jobs from being dropped, by Batch ID
+client.unmark(:drop, :bid, some_batch_id)
+
+# Un-mark an entire Job Class from being dropped
+client.unmark(:drop, :bid, "SomeJobClass")
+```
+
+### Un-marking All
+
+Clearing all `:kill` or `:drop` marks can be done in one fell swoop as well.
+
+```ruby
+client.unmark_all(:kill)
+
+# or …
+
+client.unmark_all(:drop)
+```
+
+## Configuration
+
+With `sidekiq-dispose` installed, [register its Sidekiq server middleware][sidekiq-register-middleware].
+Typically this is done via `config/initializers/sidekiq.rb` in a Rails app.
+
+```ruby
+Sidekiq.configure_server do |config|
+  config.server_middleware do |chain|
+    chain.add Sidekiq::Disposal::ServerMiddleware
+  end
+end
+```
+
+This piece of middleware checks each job, after it's been dequeued, but before it's `#perform` has been called, to see if it should be disposed of.
+If the job is marked for disposal (by Job ID, Batch ID, or Job Class) a corresponding error is raised by the middleware.
+
+A Job marked `:kill` will raise a `Sidekiq::Disposal::JobKilled` error, while one marked `:drop` will raise `Sidekiq::Disposal::JobDropped`.
+Out of the box these errors will cause [Sidekiq's error handling and retry mechanism][sidekiq-retries] to kick in, re-enqueueing the Job.
+And round-and-round it will go until the default error/death handling kicks in.
+
+To avoid this, you need to handle those specific `Sidekiq::Disposal` errors accordingly.
+
+Adjust the base Sidekiq Job class, often called `ApplicationJob` or similar, to overwrite or tweak the `#sidekiq_retry_in` method:
+
+```ruby
+sidekiq_retry_in do |_count, exception, jobhash|
+  case exception
+  when Infrastructure::Sidekiq::Disposal::JobKilled
+    # Optionally log/collect telemetry here too…
+    :kill
+  when Infrastructure::Sidekiq::Disposal::JobDropped
+    # Optionally log/collect telemetry here too…
+    :discard
+  end
+end
+```
+
+_NOTE_: If is not a base job, consider adding one, or you'll need to add this to every job you want to be disposable.
+
+Returning `:kill` from this method will cause Sidekiq to immediately move the Job to the Dead Queue.
+Similarly, returning `:discard` will cause Sidekiq to drop the job on the floor.
+Either way, the Job's `#perform` is never called.
+
+### Non-Disposable Jobs
+
+By default all Jobs are disposable, meaning they _can_ be marked to be `:kill`-ed or `:drop`-ed.
+However, checking if a specific Job should be disposed of is not free; it requires round trip(s) to Redis.
+Therefore, you might want to make some Jobs non-disposable to avoid these extra round trips.
+Or because there are some Jobs that simply should never be disposed of for… _reasons_.
+
+This is done via a Job's `sidekiq_options`.
+
+```ruby
+sidekiq_options disposable: false
+```
+
+With that in place, the server middleware will ignore the Job, and pass it down the middleware Chain.
+No extra Redis calls, no funny business.
+
+## Development
+
+After checking out the repo, run `bin/setup` to install dependencies.
+Then, run `bin/rspec` to run the tests.
+You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+
+To install this gem onto your local machine, run `bin/rake install`.
+To release a new version, update the version number in `version.rb`, and then run `bin/rake release`, which will create a git tag for the version, push git commits and the created tag, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/hibachrach/sidekiq-disposal.
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
+
+[sidekiq]: https://sidekiq.org "Simple, efficient background jobs for Ruby."
+[sidekiq-register-middleware]: https://github.com/sidekiq/sidekiq/wiki/Middleware#registering-middleware "Registering Sidekiq Middleware"
+[sidekiq-retries]: https://github.com/sidekiq/sidekiq/wiki/Error-Handling "Sidekiq Error Handling and Retries"
+[cancel-jobs]: https://github.com/sidekiq/sidekiq/wiki/FAQ#how-do-i-cancel-a-sidekiq-job "How do I cancel a Sidekiq Job?"

data/Rakefile

@@ -0,0 +1,10 @@
+# frozen_string_literal: true
+
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+
+RSpec::Core::RakeTask.new(:spec)
+
+require "standard/rake"
+
+task default: %i[spec standard]

data/lib/sidekiq/disposal/client.rb

@@ -0,0 +1,128 @@
+# frozen_string_literal: true
+
+require "sidekiq"
+
+module Sidekiq
+  module Disposal
+    # A client for marking enqueued jobs for disposal. Disposal can be a job
+    # getting "killed" (sent straight to the dead queue/morgue) or "dropped"
+    # (hard deleted entirely).
+    #
+    # This task is accomplished with "markers": A job can be "marked" for a
+    # type of disposal. This means that a "marker" (a job id/jid, batch id/bid,
+    # or class name) can be formatted and then added to the relevant target
+    # set.
+    #
+    # When a worker picks up the job, the corresponding `ServerMiddleware` will
+    # then ensure that the job is not executed (see that class for more
+    # information).
+    class Client
+      REDIS_KILL_TARGET_SET = "sidekiq-disposal:kill_targets"
+      REDIS_DROP_TARGET_SET = "sidekiq-disposal:drop_targets"
+
+      ALLOWED_MARKER_TYPES = [
+        :jid,
+        :bid,
+        :class
+      ].freeze
+
+      def initialize(sidekiq_api = ::Sidekiq)
+        @sidekiq_api = sidekiq_api
+      end
+
+      # @param disposal_method [:kill, :drop] How to handle job
+      # @param marker_type [:jid, :bid, :class_name]
+      # @param marker [String]
+      def mark(disposal_method, marker_type, marker)
+        redis do |conn|
+          formatted_marker = formatted_marker(marker_type, marker)
+          disposal_target_set = disposal_target_set(disposal_method)
+          conn.sadd(disposal_target_set, formatted_marker)
+        end
+      end
+
+      # @param disposal_method [:kill, :drop] How to handle job
+      # @param marker_type [:jid, :bid, :class_name]
+      # @param marker [String]
+      def unmark(disposal_method, marker_type, marker)
+        redis do |conn|
+          formatted_marker = formatted_marker(marker_type, marker)
+          disposal_target_set = disposal_target_set(disposal_method)
+          conn.srem(disposal_target_set, formatted_marker)
+        end
+      end
+
+      def unmark_all(disposal_method)
+        redis do |conn|
+          conn.del(disposal_target_set(disposal_method))
+        end
+      end
+
+      def markers(disposal_method)
+        redis do |conn|
+          conn.smembers(disposal_target_set(disposal_method))
+        end
+      end
+
+      def kill_target?(job)
+        job_in_target_set?(job, disposal_target_set(:kill))
+      end
+
+      def drop_target?(job)
+        job_in_target_set?(job, disposal_target_set(:drop))
+      end
+
+      private
+
+      def redis(&blk)
+        sidekiq_api.redis(&blk)
+      end
+
+      def disposal_target_set(disposal_method)
+        case disposal_method
+        when :kill
+          REDIS_KILL_TARGET_SET
+        when :drop
+          REDIS_DROP_TARGET_SET
+        else
+          raise ArgumentError, "disposal_method must be either :kill or :drop, instead got: #{disposal_method}"
+        end
+      end
+
+      def job_in_target_set?(job, target_set)
+        redis do |conn|
+          # `SMISEMBERS setname element1 [element2 ...]` asks whether each
+          # element given is in `setname`; redis-client (the low-level redis
+          # api used by Sidekiq) returns an array of integer answers for
+          # each element: 1 if it's a member, and 0 otherwise.
+          conn.smismember(target_set, formatted_markers(job)).any? do |match|
+            match == 1
+          end
+        end
+      end
+
+      # @return [Array] A list of identifying formatted_markers/features which
+      #                 indicates a job is targeted for disposal.
+      def formatted_markers(job)
+        ALLOWED_MARKER_TYPES.map do |marker_type|
+          formatted_marker_for_job(marker_type, job)
+        end.compact
+      end
+
+      # @returns the formatted marker that would be in Redis if this job has
+      # been targeted
+      def formatted_marker_for_job(marker_type, job)
+        formatted_marker(marker_type, job[marker_type.to_s])
+      end
+
+      # @returns the marker as it is stored in Redis
+      def formatted_marker(marker_type, marker)
+        return nil if marker.nil?
+        raise ArgumentError unless ALLOWED_MARKER_TYPES.include?(marker_type.to_sym)
+        "#{marker_type}:#{marker}"
+      end
+
+      attr_reader :sidekiq_api
+    end
+  end
+end

data/lib/sidekiq/disposal/server_middleware.rb

@@ -0,0 +1,32 @@
+# frozen_string_literal: true
+
+require "sidekiq"
+require_relative "../disposal"
+
+module Sidekiq
+  module Disposal
+    class ServerMiddleware
+      include ::Sidekiq::ServerMiddleware
+
+      def initialize(client = Client.new)
+        @client = client
+      end
+
+      def call(job_instance, job, _queue)
+        if job_instance && !job_instance.class.get_sidekiq_options.fetch("disposable", true)
+          yield
+        elsif client.kill_target?(job)
+          raise JobKilled
+        elsif client.drop_target?(job)
+          raise JobDropped
+        else
+          yield
+        end
+      end
+
+      private
+
+      attr_reader :client
+    end
+  end
+end

data/lib/sidekiq/disposal/version.rb

@@ -0,0 +1,7 @@
+# frozen_string_literal: true
+
+module Sidekiq
+  module Disposal
+    VERSION = "0.1.0"
+  end
+end

data/lib/sidekiq/disposal.rb

@@ -0,0 +1,17 @@
+# frozen_string_literal: true
+
+require_relative "disposal/version"
+require_relative "disposal/client"
+require_relative "disposal/server_middleware"
+
+module Sidekiq
+  # Namespace for everything related to job disposal: the process of putting
+  # jobs' markers (i.e. identifying features) on a list so they can be "killed"
+  # (sent immediately to dead set/morgue) or "dropped" (completely discarded
+  # from Sidekiq) when picked up from the queue.
+  module Disposal
+    Error = Class.new(StandardError)
+    JobKilled = Class.new(Error)
+    JobDropped = Class.new(Error)
+  end
+end

data/sig/sidekiq/disposal.rbs

@@ -0,0 +1,6 @@
+module Sidekiq
+  module Disposal
+    VERSION: String
+    # See the writing guide of rbs: https://github.com/ruby/rbs#guides
+  end
+end

metadata

@@ -0,0 +1,77 @@
+--- !ruby/object:Gem::Specification
+name: sidekiq-disposal
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- Hazel Bachrach
+- Steven Harman
+autorequire:
+bindir: exe
+cert_chain: []
+date: 2024-12-13 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: sidekiq
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '7.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '7.0'
+description: |
+  A mechanism to mark Sidekiq Jobs to be disposed of by Job ID, Batch ID, or Job Class.
+  Disposal here means to either `:kill` the Job (send to the Dead queue) or `:discard` it (throw it away), at the time the job is picked up and processed by Sidekiq.
+email:
+- dacheatbot@gmail.com
+- steven@harmanly.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- ".rspec"
+- ".standard.yml"
+- CHANGELOG.md
+- LICENSE.txt
+- README.md
+- Rakefile
+- lib/sidekiq/disposal.rb
+- lib/sidekiq/disposal/client.rb
+- lib/sidekiq/disposal/server_middleware.rb
+- lib/sidekiq/disposal/version.rb
+- sig/sidekiq/disposal.rbs
+homepage: https://github.com/hibachrach/sidekiq-disposal
+licenses:
+- MIT
+metadata:
+  changelog_uri: https://github.com/hibachrach/sidekiq-disposal/blob/main/CHANGELOG.md
+  documentation_uri: https://github.com/hibachrach/sidekiq-disposal
+  homepage_uri: https://github.com/hibachrach/sidekiq-disposal
+  source_code_uri: https://github.com/hibachrach/sidekiq-disposal
+post_install_message:
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: 3.0.0
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 3.5.23
+signing_key:
+specification_version: 4
+summary: A mechanism to dispose of (cancel) queued jobs by Job ID, Batch ID, or Job
+  Class.
+test_files: []