sidekiq-limit_fetch 4.0.0 → 4.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 88e19126b173869fae0b5046533e8615d2785fb903f01b979cbcdea65e7824b5
-  data.tar.gz: 212cfdcf179fa2aceb6eb065e7bd384109f3afbf2ff08da27699197a300cf044
+  metadata.gz: f4e35bb2ed507c01ab307ba39254c5d82ba6837d60a771c33e765f3a92ede831
+  data.tar.gz: 3883ea7ccd095c1f9f0f3ae3900d642500e14f146a74b37a9cffd48440e4e6ec
 SHA512:
-  metadata.gz: 0f0c99a5e6d183f9ce41c7bfd85f74926efad1515f29d88d24c7c1076b108121ed9c963ea60d76406c4a006e9b49dcb09239d9fe5b14e59e8f3702b7e4591689
-  data.tar.gz: 888322ef2940592c4d65095b136a4a4cf27cfe7f9bb470cb071fe3a47028cd095fc9b4c30392cb3d25be26536126851f357a61f6b31145dfb31add97f9250bec
+  metadata.gz: b1ae9ec40f0f303c352579478a75c7d85cf1fb77fa5f48f73bd2d44b35dd1d6e9daf1f88251108990408a1efc6077f3b250a571ff3d2806de9a72a2766d033b3
+  data.tar.gz: 9ace46d937a68560dc050fee75ebd4ce663f17bce4ea1be25825d568816282b8ce6059133786e8463fe48d3349baffb4c7a93e04613568367e9114672d23f314

data/CHANGELOG.md

@@ -0,0 +1,22 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [4.1.0] - 2022-03-29
+
+### Changed
+
+- #101 - Fix stuck queues bug on Redis restart from [@907th](https://github.com/907th).
+
+## [4.0.0] - 2022-03-26
+
+This project was taken over by [@deanpcmad](https://github.com/deanpcmad)
+
+### Changed
+
+- #120 - Migrate CI to GitHub Actions from [@petergoldstein](https://github.com/petergoldstein).
+- #124 - Fixed redis v4.6.0 pipelines deprecation warning from [@iurev](https://github.com/iurev).
+- #83  - Processing dynamic queues from [@alexey-yanchenko](https://github.com/alexey-yanchenko).

data/LICENSE

@@ -0,0 +1,22 @@
+MIT License
+
+Copyright (c) 2022 Dean Perry
+Copyright (c) 2013 brainopia
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -1,6 +1,6 @@
 ## Description
 
-*This project is currently being taken over by [@deanpcmad](https://github.com/deanpcmad) and will be updated soon*
+*This project has been taken over by [@deanpcmad](https://github.com/deanpcmad)*
 
 Sidekiq strategy to support a granular queue control – limiting, pausing, blocking, querying.
 
@@ -11,7 +11,11 @@ Sidekiq strategy to support a granular queue control – limiting, pausing, bloc
 
 Add this line to your application's Gemfile:
 
-    gem 'sidekiq-limit_fetch'
+```
+gem 'sidekiq-limit_fetch'
+```
+
+Then `bundle install`.
 
 ### Limitations
 
@@ -31,19 +35,19 @@ To use this Gem in other Ruby projects, just add `require 'sidekiq-limit_fetch'`
 Specify limits which you want to place on queues inside sidekiq.yml:
 
 ```yaml
-  :limits:
-    queue_name1: 5
-    queue_name2: 10
+:limits:
+  queue_name1: 5
+  queue_name2: 10
 ```
 
 Or set it dynamically in your code:
 ```ruby
-  Sidekiq::Queue['queue_name1'].limit = 5
-  Sidekiq::Queue['queue_name2'].limit = 10
+Sidekiq::Queue['queue_name1'].limit = 5
+Sidekiq::Queue['queue_name2'].limit = 10
 ```
 
-In these examples, tasks for the ```queue_name1``` will be run by at most 5
-workers at the same time and the ```queue_name2``` will have no more than 10
+In these examples, tasks for the `queue_name1` will be run by at most 5
+workers at the same time and the `queue_name2` will have no more than 10
 workers simultaneously.
 
 Ability to set limits dynamically allows you to resize worker
@@ -54,14 +58,14 @@ distribution among queues any time you want.
 If you use multiple sidekiq processes then you can specify limits per process:
 
 ```yaml
-  :process_limits:
-    queue_name: 2
+:process_limits:
+  queue_name: 2
 ```
 
 Or set it in your code:
 
 ```ruby
-  Sidekiq::Queue['queue_name'].process_limit = 2
+Sidekiq::Queue['queue_name'].process_limit = 2
 ```
 
 ### Busy workers by queue
@@ -69,7 +73,7 @@ Or set it in your code:
 You can see how many workers currently handling a queue:
 
 ```ruby
-  Sidekiq::Queue['name'].busy # number of busy workers
+Sidekiq::Queue['name'].busy # number of busy workers
 ```
 
 ### Pauses
@@ -78,10 +82,10 @@ You can also pause your queues temporarily. Upon continuing their limits
 will be preserved.
 
 ```ruby
-  Sidekiq::Queue['name'].pause # prevents workers from running tasks from this queue
-  Sidekiq::Queue['name'].paused? # => true
-  Sidekiq::Queue['name'].unpause # allows workers to use the queue
-  Sidekiq::Queue['name'].pause_for_ms(1000) # will pause for a second
+Sidekiq::Queue['name'].pause # prevents workers from running tasks from this queue
+Sidekiq::Queue['name'].paused? # => true
+Sidekiq::Queue['name'].unpause # allows workers to use the queue
+Sidekiq::Queue['name'].pause_for_ms(1000) # will pause for a second
 ```
 
 ### Blocking queue mode
@@ -92,12 +96,12 @@ queue task is executing then no new task from lesser priority queues will
 be ran. Eg,
 
 ```yaml
-  :queues:
-    - a
-    - b
-    - c
-  :blocking:
-    - b
+:queues:
+  - a
+  - b
+  - c
+:blocking:
+  - b
 ```
 
 In this case when a task for `b` queue is ran no new task from `c` queue
@@ -106,9 +110,9 @@ will be started.
 You can also enable and disable blocking mode for queues on the fly:
 
 ```ruby
-  Sidekiq::Queue['name'].block
-  Sidekiq::Queue['name'].blocking? # => true
-  Sidekiq::Queue['name'].unblock
+Sidekiq::Queue['name'].block
+Sidekiq::Queue['name'].blocking? # => true
+Sidekiq::Queue['name'].unblock
 ```
 
 ### Advanced blocking queues
@@ -118,13 +122,13 @@ running only queues higher and queues from their blocking group can
 run. It will be easier to understand with an example:
 
 ```yaml
-  :queues:
-    - a
-    - b
-    - c
-    - d
-  :blocking:
-    - [b, c]
+:queues:
+  - a
+  - b
+  - c
+  - d
+:blocking:
+  - [b, c]
 ```
 
 In this case tasks from `d` will be blocked when a task from queue `b` or `c` is executed.
@@ -132,7 +136,7 @@ In this case tasks from `d` will be blocked when a task from queue `b` or `c` is
 You can dynamically set exceptions for queue blocking:
 
 ```ruby
-  Sidekiq::Queue['queue1'].block_except 'queue2'
+Sidekiq::Queue['queue1'].block_except 'queue2'
 ```
 
 ### Dynamic queues
@@ -143,11 +147,11 @@ that have tasks pushed to them (usually with `Sidekiq::Client.push`)).
 To use this mode you need to specify a following line in sidekiq.yml:
 
 ```yaml
-  :dynamic: true
+:dynamic: true
 ```
 
 Dynamic queues will be ran at the lowest priority.
 
 ### Maintenance
 
-If you use ```flushdb```, restart the sidekiq process to re-populate the dynamic configuration.
+If you use `flushdb`, restart the sidekiq process to re-populate the dynamic configuration.

data/lib/sidekiq/limit_fetch/queues.rb

@@ -20,15 +20,17 @@ module Sidekiq::LimitFetch::Queues
   end
 
   def acquire
-    selector.acquire(ordered_queues, namespace)
-      .tap {|it| save it }
-      .map {|it| "queue:#{it}" }
+    queues = saved
+    queues ||= Sidekiq::LimitFetch.redis_retryable do
+      selector.acquire(ordered_queues, namespace)
+    end
+    save queues
+    queues.map { |it| "queue:#{it}" }
   end
 
   def release_except(full_name)
     queues = restore
     queues.delete full_name[/queue:(.*)/, 1] if full_name
-
     Sidekiq::LimitFetch.redis_retryable do
       selector.release queues, namespace
     end
@@ -141,13 +143,17 @@ module Sidekiq::LimitFetch::Queues
     Sidekiq::LimitFetch::Global::Selector
   end
 
+  def saved
+    Thread.current[THREAD_KEY]
+  end
+
   def save(queues)
     Thread.current[THREAD_KEY] = queues
   end
 
   def restore
-    Thread.current[THREAD_KEY] || []
+    saved || []
   ensure
-    Thread.current[THREAD_KEY] = nil
+    save nil
   end
 end

data/lib/sidekiq/limit_fetch.rb

@@ -14,6 +14,8 @@ module Sidekiq::LimitFetch
   require_relative 'extensions/queue'
   require_relative 'extensions/manager'
 
+  TIMEOUT = Sidekiq::BasicFetch::TIMEOUT
+
   extend self
 
   def new(_)
@@ -39,14 +41,21 @@ module Sidekiq::LimitFetch
   def redis_retryable
     yield
   rescue Redis::BaseConnectionError
-    sleep 1
+    sleep TIMEOUT
     retry
+  rescue Redis::CommandError => error
+    # If Redis was restarted and is still loading its snapshot,
+    # then we should treat this as a temporary connection error too.
+    if error.message =~ /^LOADING/
+      sleep TIMEOUT
+      retry
+    else
+      raise
+    end
   end
 
   private
 
-  TIMEOUT = Sidekiq::BasicFetch::TIMEOUT
-
   def redis_brpop(queues)
     if queues.empty?
       sleep TIMEOUT  # there are no queues to handle, so lets sleep

data/sidekiq-limit_fetch.gemspec

@@ -1,6 +1,6 @@
 Gem::Specification.new do |gem|
   gem.name          = 'sidekiq-limit_fetch'
-  gem.version       = '4.0.0'
+  gem.version       = '4.1.0'
   gem.license       = 'MIT'
   gem.authors       = ['Dean Perry', 'brainopia']
   gem.email         = 'dean@deanpcmad.com'
@@ -8,6 +8,10 @@ Gem::Specification.new do |gem|
   gem.homepage      = 'https://github.com/deanpcmad/sidekiq-limit_fetch'
   gem.description   = "Sidekiq strategy to restrict number of workers which are able to run specified queues simultaneously."
 
+  gem.metadata["homepage_uri"] = gem.homepage
+  gem.metadata["source_code_uri"] = "https://github.com/deanpcmad/sidekiq-limit_fetch"
+  gem.metadata["changelog_uri"] = "https://github.com/deanpcmad/sidekiq-limit_fetch/blob/master/CHANGELOG.md"
+
   gem.files         = `git ls-files`.split($/)
   gem.test_files    = gem.files.grep %r{^spec/}
   gem.require_paths = %w(lib)

data/spec/sidekiq/limit_fetch/queues_spec.rb

@@ -15,61 +15,72 @@ RSpec.describe Sidekiq::LimitFetch::Queues do
 
   before { subject.start options }
 
+  def in_thread(&block)
+    thr = Thread.new(&block)
+    thr.join
+  end
+
   it 'should acquire queues' do
-    subject.acquire
+    in_thread { subject.acquire }
     expect(Sidekiq::Queue['queue1'].probed).to eq 1
     expect(Sidekiq::Queue['queue2'].probed).to eq 1
   end
 
   it 'should acquire dynamically blocking queues' do
-    subject.acquire
+    in_thread { subject.acquire }
     expect(Sidekiq::Queue['queue1'].probed).to eq 1
     expect(Sidekiq::Queue['queue2'].probed).to eq 1
 
     Sidekiq::Queue['queue1'].block
 
-    subject.acquire
+    in_thread { subject.acquire }
     expect(Sidekiq::Queue['queue1'].probed).to eq 2
     expect(Sidekiq::Queue['queue2'].probed).to eq 1
   end
 
   it 'should block except given queues' do
     Sidekiq::Queue['queue1'].block_except 'queue2'
-    subject.acquire
+    in_thread { subject.acquire }
     expect(Sidekiq::Queue['queue1'].probed).to eq 1
     expect(Sidekiq::Queue['queue2'].probed).to eq 1
 
     Sidekiq::Queue['queue1'].block_except 'queue404'
-    subject.acquire
+    in_thread { subject.acquire }
     expect(Sidekiq::Queue['queue1'].probed).to eq 2
     expect(Sidekiq::Queue['queue2'].probed).to eq 1
   end
 
   it 'should release queues' do
-    subject.acquire
-    subject.release_except nil
+    in_thread {
+      subject.acquire
+      subject.release_except nil
+    }
     expect(Sidekiq::Queue['queue1'].probed).to eq 0
     expect(Sidekiq::Queue['queue2'].probed).to eq 0
   end
 
   it 'should release queues except selected' do
-    subject.acquire
-    subject.release_except 'queue:queue1'
+    in_thread {
+      subject.acquire
+      subject.release_except 'queue:queue1'
+    }
     expect(Sidekiq::Queue['queue1'].probed).to eq 1
     expect(Sidekiq::Queue['queue2'].probed).to eq 0
   end
 
   it 'should release when no queues was acquired' do
     queues.each {|name| Sidekiq::Queue[name].pause }
-    subject.acquire
-    expect { subject.release_except nil }.not_to raise_exception
+    in_thread {
+      subject.acquire
+      expect { subject.release_except nil }.not_to raise_exception
+    }
   end
 
   context 'blocking' do
     let(:blocking) { %w(queue1) }
 
     it 'should acquire blocking queues' do
-      3.times { subject.acquire }
+      3.times { in_thread { subject.acquire  } }
       expect(Sidekiq::Queue['queue1'].probed).to eq 3
       expect(Sidekiq::Queue['queue2'].probed).to eq 1
     end

data/spec/sidekiq/limit_fetch_spec.rb

@@ -6,7 +6,7 @@ RSpec.describe Sidekiq::LimitFetch do
   let(:limits) {{ 'queue1' => 1, 'queue2' => 2 }}
 
   before do
-    subject::Queues.start options 
+    subject::Queues.start options
 
     Sidekiq.redis do |it|
       it.del 'queue:queue1'

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: sidekiq-limit_fetch
 version: !ruby/object:Gem::Version
-  version: 4.0.0
+  version: 4.1.0
 platform: ruby
 authors:
 - Dean Perry
@@ -9,7 +9,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2022-03-26 00:00:00.000000000 Z
+date: 2022-03-29 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: sidekiq
@@ -112,8 +112,9 @@ files:
 - ".gitignore"
 - ".rspec"
 - Appraisals
+- CHANGELOG.md
 - Gemfile
-- LICENSE.txt
+- LICENSE
 - README.md
 - Rakefile
 - bench/compare.rb
@@ -167,7 +168,10 @@ files:
 homepage: https://github.com/deanpcmad/sidekiq-limit_fetch
 licenses:
 - MIT
-metadata: {}
+metadata:
+  homepage_uri: https://github.com/deanpcmad/sidekiq-limit_fetch
+  source_code_uri: https://github.com/deanpcmad/sidekiq-limit_fetch
+  changelog_uri: https://github.com/deanpcmad/sidekiq-limit_fetch/blob/master/CHANGELOG.md
 post_install_message: 
 rdoc_options: []
 require_paths:
@@ -183,7 +187,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.1.6
+rubygems_version: 3.2.22
 signing_key: 
 specification_version: 4
 summary: Sidekiq strategy to support queue limits

data/LICENSE.txt

@@ -1,22 +0,0 @@
-Copyright (c) 2013 brainopia
-
-MIT License
-
-Permission is hereby granted, free of charge, to any person obtaining
-a copy of this software and associated documentation files (the
-"Software"), to deal in the Software without restriction, including
-without limitation the rights to use, copy, modify, merge, publish,
-distribute, sublicense, and/or sell copies of the Software, and to
-permit persons to whom the Software is furnished to do so, subject to
-the following conditions:
-
-The above copyright notice and this permission notice shall be
-included in all copies or substantial portions of the Software.
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
-EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
-MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
-NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
-LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
-OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
-WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.