sidekiq-throttled 0.17.0 → 1.1.0

data/README.md

@@ -1,297 +0,0 @@
-# Sidekiq::Throttled
-
-[![CI Status](https://github.com/ixti/sidekiq-throttled/actions/workflows/ci.yml/badge.svg)](https://github.com/ixti/sidekiq-throttled/actions/workflows/ci.yml)
-[![Latest Version](https://badge.fury.io/rb/sidekiq-throttled.svg)](http://rubygems.org/gems/sidekiq-throttled)
-[![API Docs](https://img.shields.io/badge/yard-docs-blue.svg)](http://www.rubydoc.info/gems/sidekiq-throttled)
-
-Concurrency and threshold throttling for [Sidekiq][sidekiq].
-
-## Installation
-
-Add this line to your application's Gemfile:
-
-``` ruby
-gem "sidekiq-throttled"
-```
-
-And then execute:
-
-    $ bundle
-
-Or install it yourself as:
-
-    $ gem install sidekiq-throttled
-
-
-## Usage
-
-Add somewhere in your app's bootstrap (e.g. `config/initializers/sidekiq.rb` if
-you are using Rails):
-
-``` ruby
-require "sidekiq/throttled"
-Sidekiq::Throttled.setup!
-```
-
-Load order can be an issue if you are using other Sidekiq plugins and/or middleware.
-To prevent any problems, add the `.setup!` call to the bottom of your init file.
-
-Once you've done that you can include `Sidekiq::Throttled::Job` to your
-job classes and configure throttling:
-
-``` ruby
-class MyJob
-  include Sidekiq::Job
-  include Sidekiq::Throttled::Job
-
-  sidekiq_options :queue => :my_queue
-
-  sidekiq_throttle(
-    # Allow maximum 10 concurrent jobs of this class at a time.
-    :concurrency => { :limit => 10 },
-    # Allow maximum 1K jobs being processed within one hour window.
-    :threshold => { :limit => 1_000, :period => 1.hour }
-  )
-
-  def perform
-    # ...
-  end
-end
-```
-
-**NOTE:** `Sidekiq::Throttled::Job` is aliased as `Sidekiq::Throttled::Worker`,
-thus if you're using Sidekiq prior 6.3.0 version, or you using `Sidekiq::Worker`
-naming convention, you can use the alias for consistency:
-
-``` ruby
-class MyWorker
-  include Sidekiq::Worker
-  include Sidekiq::Throttled::Worker
-
-  # ...
-end
-```
-
-
-### Observer
-
-You can specify an observer that will be called on throttling. To do so pass an
-`:observer` option with callable object:
-
-``` ruby
-class MyJob
-  include Sidekiq::Job
-  include Sidekiq::Throttled::Job
-
-  MY_OBSERVER = lambda do |strategy, *args|
-    # do something
-  end
-
-  sidekiq_options :queue => :my_queue
-
-  sidekiq_throttle(
-    :concurrency => { :limit => 10 },
-    :threshold   => { :limit => 100, :period => 1.hour }
-    :observer    => MY_OBSERVER
-  )
-
-  def perform(*args)
-    # ...
-  end
-end
-```
-
-Observer will receive `strategy, *args` arguments, where `strategy` is a Symbol
-`:concurrency` or `:threshold`, and `*args` are the arguments that were passed
-to the job.
-
-
-### Dynamic throttling
-
-You can throttle jobs dynamically with `:key_suffix` option:
-
-``` ruby
-class MyJob
-  include Sidekiq::Job
-  include Sidekiq::Throttled::Job
-
-  sidekiq_options :queue => :my_queue
-
-  sidekiq_throttle(
-    # Allow maximum 10 concurrent jobs per user at a time.
-    :concurrency => { :limit => 10, :key_suffix => -> (user_id) { user_id } }
-  )
-
-  def perform(user_id)
-    # ...
-  end
-end
-```
-
-You can also supply dynamic values for limits and periods by supplying a proc
-for these values. The proc will be evaluated at the time the job is fetched
-and will receive the same arguments that are passed to the job.
-
-``` ruby
-class MyJob
-  include Sidekiq::Job
-  include Sidekiq::Throttled::Job
-
-  sidekiq_options :queue => :my_queue
-
-  sidekiq_throttle(
-    # Allow maximum 1000 concurrent jobs of this class at a time for VIPs and 10 for all other users.
-    :concurrency => {
-      :limit      => ->(user_id) { User.vip?(user_id) ? 1_000 : 10 },
-      :key_suffix => ->(user_id) { User.vip?(user_id) ? "vip" : "std" }
-    },
-    # Allow 1000 jobs/hour to be processed for VIPs and 10/day for all others
-    :threshold   => {
-      :limit      => ->(user_id) { User.vip?(user_id) ? 1_000 : 10 },
-      :period     => ->(user_id) { User.vip?(user_id) ? 1.hour : 1.day },
-      :key_suffix => ->(user_id) { User.vip?(user_id) ? "vip" : "std" }
-  )
-
-  def perform(user_id)
-    # ...
-  end
-end
-```
-
-You also can use several different keys to throttle one worker.
-
-``` ruby
-class MyJob
-  include Sidekiq::Job
-  include Sidekiq::Throttled::Job
-
-  sidekiq_options :queue => :my_queue
-
-  sidekiq_throttle(
-    # Allow maximum 10 concurrent jobs per project at a time and maximum 2 jobs per user
-    :concurrency => [
-      { :limit => 10, :key_suffix => -> (project_id, user_id) { project_id } },
-      { :limit => 2, :key_suffix => -> (project_id, user_id) { user_id } }
-    ]
-    # For :threshold it works the same
-  )
-
-  def perform(project_id, user_id)
-    # ...
-  end
-end
-```
-
-**NB** Don't forget to specify `:key_suffix` and make it return different values
-if you are using dynamic limit/period options. Otherwise you risk getting into
-some trouble.
-
-
-### Concurrency throttling fine-tuning
-
-Concurrency throttling is based on distributed locks. Those locks have default
-time to live (TTL) set to 15 minutes. If your job takes more than 15 minutes
-to finish, lock will be released and you might end up with more jobs running
-concurrently than you expect.
-
-This is done to avoid deadlocks - when by any reason (e.g. Sidekiq process was
-OOM-killed) cleanup middleware wasn't executed and locks were not released.
-
-If your job takes more than 15 minutes to complete, you can tune concurrency
-lock TTL to fit your needs:
-
-``` ruby
-# Set concurrency strategy lock TTL to 1 hour.
-sidekiq_throttle(:concurrency => { :limit => 20, :ttl => 1.hour.to_i })
-```
-
-
-## Enhanced Queues list
-
-This gem provides ability to pause/resume queues from processing by workers.
-So you may simply pause particular queue without need to stop and reconfigure
-workers by simply pushing a button on sidekiq web UI.
-
-By default we add *Enhanced Queues* tab with this functionality. But if you
-want you can override default *Queues* tab completely (notice that page will
-still be available using it's URL, but tab will be pointing enhanced version).
-To do so, just call `Sidekiq::Throttled::Web.enhance_queues_tab!` somewhere
-in your initializer/bootstrap code. If you are using rails, you might want to
-add it right into your `config/routes.rb` file:
-
-``` ruby
-# file config/routes.rb
-
-require "sidekiq/web"
-require "sidekiq/throttled/web"
-
-Rails.application.routes.draw do
-  # ...
-
-  # Replace Sidekiq Queues with enhanced version!
-  Sidekiq::Throttled::Web.enhance_queues_tab!
-
-  # Mount Sidekiq Web UI to `/sidekiq` endpoint
-  mount Sidekiq::Web => "/sidekiq"
-
-  # ...
-end
-```
-
-
-## Supported Ruby Versions
-
-This library aims to support and is [tested against][ci] the following Ruby
-versions:
-
-* Ruby 2.7.x
-* Ruby 3.0.x
-* Ruby 3.1.x
-
-If something doesn't work on one of these versions, it's a bug.
-
-This library may inadvertently work (or seem to work) on other Ruby versions,
-however support will only be provided for the versions listed above.
-
-If you would like this library to support another Ruby version or
-implementation, you may volunteer to be a maintainer. Being a maintainer
-entails making sure all tests run and pass on that implementation. When
-something breaks on your implementation, you will be responsible for providing
-patches in a timely fashion. If critical issues for a particular implementation
-exist at the time of a major release, support for that Ruby version may be
-dropped.
-
-
-## Supported Sidekiq Versions
-
-This library aims to support work with following [Sidekiq][sidekiq] versions:
-
-* Sidekiq 6.4.x
-* Sidekiq 6.5.x
-
-
-## Contributing
-
-* Fork sidekiq-throttled on GitHub
-* Make your changes
-* Ensure all tests pass (`bundle exec rake`)
-* Send a pull request
-* If we like them we'll merge them
-* If we've accepted a patch, feel free to ask for commit access!
-
-
-## Development
-
-```
-bundle update
-bundle exec appraisal install   # install dependencies for all gemfiles
-bundle exec appraisal update    # update dependencies for all gemfiles
-bundle exec appraisal rspec     # run rspec against each gemfile
-bundle exec rubocop             # run static code analysis
-```
-
-Don't forget to run `appraisal update` after any changes to `Gemfile`.
-
-
-[ci]: https://github.com/ixti/sidekiq-throttled/actions/workflows/ci.yml
-[sidekiq]: https://github.com/mperham/sidekiq

data/Rakefile

@@ -1,27 +0,0 @@
-# frozen_string_literal: true
-
-require "bundler/gem_tasks"
-
-require "rspec/core/rake_task"
-RSpec::Core::RakeTask.new(:spec)
-
-desc "Run RuboCop"
-task :rubocop do
-  require "rubocop"
-  result = RuboCop::CLI.new.run([])
-  abort("RuboCop failed!") if result.nonzero?
-end
-
-namespace :rubocop do
-  desc "Auto-correct RuboCop offenses"
-  task :autocorrect do
-    require "rubocop"
-    result = RuboCop::CLI.new.run(["--auto-correct"])
-    abort("RuboCop failed!") if result.nonzero?
-  end
-end
-
-default_suite = ENV["CI"] ? :spec : %i[spec rubocop]
-named_suites  = { "rubocop" => :rubocop, "rspec" => :spec }
-
-task :default => named_suites.fetch(ENV.fetch("SUITE", nil), default_suite)

data/gemfiles/sidekiq_6.4.gemfile

@@ -1,33 +0,0 @@
-# This file was generated by Appraisal
-
-source "https://rubygems.org"
-
-gem "appraisal"
-gem "rake"
-gem "rspec"
-gem "sidekiq", "~> 6.4.0"
-
-group :development do
-  gem "byebug"
-  gem "guard", require: false
-  gem "guard-rspec", require: false
-  gem "guard-rubocop", require: false
-end
-
-group :test do
-  gem "apparition"
-  gem "capybara"
-  gem "puma"
-  gem "rack-test"
-  gem "sinatra"
-  gem "timecop"
-end
-
-group :lint do
-  gem "rubocop", require: false
-  gem "rubocop-performance", require: false
-  gem "rubocop-rake", require: false
-  gem "rubocop-rspec", require: false
-end
-
-gemspec path: "../"

data/gemfiles/sidekiq_6.5.gemfile

@@ -1,33 +0,0 @@
-# This file was generated by Appraisal
-
-source "https://rubygems.org"
-
-gem "appraisal"
-gem "rake"
-gem "rspec"
-gem "sidekiq", "~> 6.5.0"
-
-group :development do
-  gem "byebug"
-  gem "guard", require: false
-  gem "guard-rspec", require: false
-  gem "guard-rubocop", require: false
-end
-
-group :test do
-  gem "apparition"
-  gem "capybara"
-  gem "puma"
-  gem "rack-test"
-  gem "sinatra"
-  gem "timecop"
-end
-
-group :lint do
-  gem "rubocop", require: false
-  gem "rubocop-performance", require: false
-  gem "rubocop-rake", require: false
-  gem "rubocop-rspec", require: false
-end
-
-gemspec path: "../"

data/lib/sidekiq/throttled/communicator/callbacks.rb

@@ -1,72 +0,0 @@
-# frozen_string_literal: true
-
-require "fiber"
-
-require "sidekiq/throttled/communicator/exception_handler"
-
-module Sidekiq
-  module Throttled
-    class Communicator
-      # Callbacks registry and runner. Runs registered callbacks in dedicated
-      # Fiber solving issue with ConnectionPool and Redis client in subscriber
-      # mode.
-      #
-      # Once Redis entered subscriber mode `#subscribe` method, it can't be used
-      # for any command but pub/sub or quit, making it impossible to use for
-      # anything else. ConnectionPool binds reserved client to Thread, thus
-      # nested `#with` calls inside same thread result into a same connection.
-      # That makes it impossible to issue any normal Redis commands from
-      # within listener Thread.
-      #
-      # @private
-      class Callbacks
-        include ExceptionHandler
-
-        # Initializes callbacks registry.
-        def initialize
-          @mutex    = Mutex.new
-          @handlers = Hash.new { |h, k| h[k] = [] }
-        end
-
-        # Registers handler of given event.
-        #
-        # @example
-        #
-        #   callbacks.on "and out comes wolves" do |who|
-        #     puts "#{who} let the dogs out?!"
-        #   end
-        #
-        # @param [#to_s] event
-        # @raise [ArgumentError] if no handler block given
-        # @yield [*args] Runs given block upon `event`
-        # @yieldreturn [void]
-        # @return [self]
-        def on(event, &handler)
-          raise ArgumentError, "No block given" unless handler
-
-          @mutex.synchronize { @handlers[event.to_s] << handler }
-          self
-        end
-
-        # Runs event handlers with given args.
-        #
-        # @param [#to_s] event
-        # @param [Object] payload
-        # @return [void]
-        def run(event, payload = nil)
-          @mutex.synchronize do
-            fiber = Fiber.new do
-              @handlers[event.to_s].each do |callback|
-                callback.call(payload)
-              rescue => e
-                handle_exception(e, :context => "sidekiq:throttled")
-              end
-            end
-
-            fiber.resume
-          end
-        end
-      end
-    end
-  end
-end

data/lib/sidekiq/throttled/communicator/exception_handler.rb

@@ -1,25 +0,0 @@
-# frozen_string_literal: true
-
-require "sidekiq"
-require "sidekiq/version"
-
-module Sidekiq
-  module Throttled
-    class Communicator
-      if Sidekiq::VERSION >= "6.5.0"
-        module ExceptionHandler
-          def handle_exception(*args)
-            Sidekiq.handle_exception(*args)
-          end
-        end
-
-        # NOTE: `Sidekiq.default_error_handler` is private API
-        Sidekiq.error_handlers << Sidekiq.method(:default_error_handler)
-      else
-        require "sidekiq/exception_handler"
-
-        ExceptionHandler = ::Sidekiq::ExceptionHandler
-      end
-    end
-  end
-end

data/lib/sidekiq/throttled/communicator/listener.rb

@@ -1,109 +0,0 @@
-# frozen_string_literal: true
-
-require "sidekiq/throttled/communicator/exception_handler"
-
-module Sidekiq
-  module Throttled
-    class Communicator
-      # Redis subscription listener thread.
-      #
-      # @private
-      class Listener < Thread
-        include ExceptionHandler
-
-        # Starts listener thread.
-        #
-        # @param [String] channel Redis pub/sub channel to listen
-        # @param [Callbacks] callbacks Message callbacks registry
-        def initialize(channel, callbacks)
-          @channel    = channel
-          @callbacks  = callbacks
-          @terminated = false
-          @subscribed = false
-
-          super { listen until @terminated }
-        end
-
-        # Whenever underlying redis client subscribed to pub/sup channel.
-        #
-        # @return [Boolean]
-        def ready?
-          @subscribed
-        end
-
-        # Whenever main loop is still running.
-        #
-        # @return [Boolean]
-        def listening?
-          !@terminated
-        end
-
-        # Stops listener.
-        #
-        # @return [void]
-        def stop
-          # Raising exception while client is in subscription mode makes
-          # redis close connection and thus causing ConnectionPool reopen
-          # it (normal mode). Otherwise subscription mode client will be
-          # pushed back to ConnectionPool causing problems.
-          raise Sidekiq::Shutdown
-        end
-
-        private
-
-        # Wraps {#subscribe} with exception handlers:
-        #
-        # - `Sidekiq::Shutdown` exception marks listener as stopped and returns
-        #   making `while` loop of listener thread terminate.
-        #
-        # - `StandardError` got recorded to the log and swallowed,
-        #   making `while` loop of the listener thread restart.
-        #
-        # - `Exception` is recorded to the log and re-raised.
-        #
-        # @return [void]
-        def listen # rubocop:disable Metrics/MethodLength
-          subscribe
-        rescue Sidekiq::Shutdown
-          @terminated = true
-          @subscribed = false
-        rescue StandardError => e # rubocop:disable Style/RescueStandardError
-          @subscribed = false
-          handle_exception(e, { :context => "sidekiq:throttled" })
-          sleep 1
-        rescue Exception => e # rubocop:disable Lint/RescueException
-          @terminated = true
-          @subscribed = false
-          handle_exception(e, { :context => "sidekiq:throttled" })
-          raise
-        end
-
-        # Subscribes to channel and triggers all registerd handlers for
-        # received messages.
-        #
-        # @note Puts thread's Redis connection to subscription mode and
-        #   locks thread.
-        #
-        # @see http://redis.io/topics/pubsub
-        # @see http://redis.io/commands/subscribe
-        # @see Callbacks#run
-        # @return [void]
-        def subscribe # rubocop:disable Metrics/MethodLength
-          Sidekiq.redis do |conn|
-            conn.subscribe @channel do |on|
-              on.subscribe do
-                @subscribed = true
-                @callbacks.run("ready")
-              end
-
-              on.message do |_channel, data|
-                message, payload = Marshal.load(data) # rubocop:disable Security/MarshalLoad:
-                @callbacks.run("message:#{message}", payload)
-              end
-            end
-          end
-        end
-      end
-    end
-  end
-end