resque-restriction 0.6.0 → 0.7.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
-SHA1:
-  metadata.gz: 135cae91708b3d84ff8c6feafdb715e7043893ec
-  data.tar.gz: c4d04692d1b62a3b0bf8dc797a1c679965e05c69
+SHA256:
+  metadata.gz: 681852b48ac3fb45d6a5d1806af5617d6c4c566ee6483c777d1e7d1e837443ba
+  data.tar.gz: '0706749a9fc0beb80cb532a84e5988157055759b0151e48b1a08d2fa8c515b21'
 SHA512:
-  metadata.gz: e0640880f52054dcfee91e434373a2db2b1cd5f61997fbaaefedce2d59f3647fb10ca0d921afd541c0fb2310ad8b29befe5d7f35c080e55be8d50183db089130
-  data.tar.gz: ecb9750f0198139276220ccb85178d09c44f824eba13bf83c3b11e8f3313c837f83f24064a65ca8a43de3b34f6fffc546ade0f5bc890052a88283e1075924876
+  metadata.gz: 2015e5527ba932161e0cdfbf20efb102214403232f5595f496571df3b8964d5157bef27fa21e3feb2363f89fbf502576c8b13596b9322ef61873861669c71a32
+  data.tar.gz: b7fce8d53acc85f50d6c21626b03e7aed86a5f04d0cd01f3a40301ff2534ae149bf3caf7cd53a5e093765a25d233c06ff6819f733f7fcc502a7586d1ac940ec9

data/.travis.yml

@@ -0,0 +1,5 @@
+sudo: false
+language: ruby
+rvm:
+  - 2.4.3
+before_install: gem install bundler -v 1.16.0

data/CHANGELOG.md

@@ -1,5 +1,10 @@
 ### Unreleased
 
+# 0.7.0 (2021-06-06)
+
+* Support `per_second`
+* Reduce Redis thrashing on large restriction queues
+
 # 0.6.0
 
 * support customized period like `per_minute_and_user_id`

data/README.md

@@ -1,6 +1,9 @@
 resque-restriction
 ===============
 
+[![AwesomeCode Status for flyerhzm/resque-restriction](https://awesomecode.io/projects/ebc1a493-78f5-4d8a-9bb7-097fdaa657ec/status)](https://awesomecode.io/repos/flyerhzm/resque-restriction)
+[![Build Status](https://secure.travis-ci.org/flyerhzm/resque-restriction.png)](http://travis-ci.org/flyerhzm/resque-restriction)
+
 Resque Restriction is a plugin for the [Resque](https://github.com/resque/resque) queueing system. It adds two functions:
 
 1. it will limit the execution number of certain jobs in a period time. For example, it can limit a certain job can be executed 1000 times per day, 100 time per hour and 30 times per 300 seconds.
@@ -52,11 +55,13 @@ end
 
 That means the InvitationJob can not be executed more than 1000 times per day, 100 times per hour and 30 times per 300 seconds.  All restrictions have to be met for the job to execute.
 
-The argument of restrict method is a hash, the key of the hash is a period time, including :concurrent, :per_minute, :per_hour, :per_day, :per_week, :per_month, :per_year, and you can also define any period like :per_300 means per 300 seconds. The value of the hash is the job execution limit number in a period.  The :concurrent option restricts the number of jobs that run simultaneously.
+The argument of restrict method is a hash, the key of the hash is a period time, including :concurrent, :per_second, :per_minute, :per_hour, :per_day, :per_week, :per_month, :per_year, and you can also define any period like :per_300 means per 300 seconds. The value of the hash is the job execution limit number in a period.  The :concurrent option restricts the number of jobs that run simultaneously.
 
 Advance
 -------
 
+### Custom Restrictions
+
 You can also add customized restriction as you like. For example, we have a job to restrict the facebook post numbers 40 times per user per day, we can define as:
 
 
@@ -89,6 +94,17 @@ end
 
 options["user_id"] returns the user's facebook uid, the key point is that the different restriction_identifiers can restrict different job execution numbers.
 
+### Reducing Redis Thrashing
+
+With large restriction queues (1,000+ jobs), Redis is hit with a large number of `Resque.pop` and re-enqueue calls which puts the job immediately back into the restriction queue. That process translates to extraneous Redis requests.
+
+To avoid the extraneous Redis calls, configure `max_queue_peek`. An appropriate `max_queue_peek` value depends on your application, number of workers, job processing time, and peek jobs in the queue. By default `max_queue_peek` is disabled.
+
+```ruby
+Resque::Restriction.configure do |config|
+  config.max_queue_peek = 100 # jobs
+end
+```
 
 Contributing
 ------------

data/Rakefile

@@ -3,5 +3,4 @@ require 'rspec/core/rake_task'
 
 RSpec::Core::RakeTask.new(:spec)
 
-task :default => :spec
-
+task default: :spec

data/lib/resque/plugins/job.rb

@@ -7,8 +7,8 @@ module Resque
         if queue =~ /^#{Plugins::Restriction::RESTRICTION_QUEUE_PREFIX}/
           # If processing the restriction queue, when poping and pushing to end,
           # we can't tell when we reach the original one, so just walk the length
-          # of the queue so we don't run infinitely long
-          Resque.size(queue).times do |i|
+          # of the queue or up to max_queue_peek so we don't run infinitely long
+          [Resque.size(queue), Restriction.config.max_queue_peek(queue)].compact.min.times do
             # For the job at the head of the queue, repush to restricition queue
             # if still restricted, otherwise we have a runnable job, so create it
             # and return
@@ -19,7 +19,7 @@ module Resque
               end
             end
           end
-          return nil
+          nil
         else
           # drop through to original Job::Reserve if not restriction queue
           origin_reserve(queue)

data/lib/resque/plugins/restriction.rb

@@ -2,6 +2,7 @@ module Resque
   module Plugins
     module Restriction
       SECONDS = {
+        :per_second => 1,
         :per_minute => 60,
         :per_hour => 60*60,
         :per_day => 24*60*60,
@@ -67,7 +68,7 @@ module Resque
         period_key, custom_key = period.to_s.split('_and_')
         period_str = case period_key.to_sym
                      when :concurrent then "*"
-                     when :per_minute, :per_hour, :per_day, :per_week then (Time.now.to_i / SECONDS[period_key.to_sym]).to_s
+                     when :per_second, :per_minute, :per_hour, :per_day, :per_week then (Time.now.to_i / SECONDS[period_key.to_sym]).to_s
                      when :per_month then Date.today.strftime("%Y-%m")
                      when :per_year then Date.today.year.to_s
                      else period_key =~ /^per_(\d+)$/ and (Time.now.to_i / $1.to_i).to_s end

data/lib/resque/restriction.rb

@@ -1,3 +1,4 @@
 require 'resque'
+require 'resque/restriction/config'
 require 'resque/plugins/job'
 require 'resque/plugins/restriction'

data/lib/resque/restriction/config.rb

@@ -0,0 +1,49 @@
+module Resque
+  module Restriction
+    def self.configure
+      yield config
+    end
+
+    def self.config
+      @config ||= Config.new
+    end
+
+    class Config
+      def initialize
+        @max_queue_peek = nil
+      end
+
+      def max_queue_peek=(value_or_callable)
+        if value_or_callable.respond_to?(:call)
+          @max_queue_peek = value_or_callable
+        elsif value_or_callable.nil?
+          @max_queue_peek = nil
+        else
+          @max_queue_peek = validated_max_queue_peek(value_or_callable)
+        end
+      end
+
+      def max_queue_peek(queue)
+        @max_queue_peek.respond_to?(:call) ? @max_queue_peek.call(queue) : @max_queue_peek
+      end
+
+      private
+
+      def validated_max_queue_peek(value)
+        peek = nil
+
+        begin
+          peek = Integer(value)
+
+          if peek <= 0
+            raise ArgumentError
+          end
+        rescue ArgumentError
+          raise ArgumentError, "max_queue_peek should be either nil or an Integer greater than 0 but #{value.inspect} was provided"
+        end
+
+        peek
+      end
+    end
+  end
+end

data/lib/resque/restriction/version.rb

@@ -1,5 +1,5 @@
 module Resque
   module Restriction
-    VERSION = '0.6.0'
+    VERSION = '0.7.0'
   end
 end

data/resque-restriction.gemspec

@@ -1,12 +1,12 @@
-lib = File.expand_path("../lib", __FILE__)
+lib = File.expand_path('lib', __dir__)
 $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
-require "resque/restriction/version"
+require 'resque/restriction/version'
 
 Gem::Specification.new do |spec|
-  spec.name          = "resque-restriction"
+  spec.name          = 'resque-restriction'
   spec.version       = Resque::Restriction::VERSION
-  spec.authors       = ["Richard Huang"]
-  spec.email         = ["flyerhzm@gmail.com"]
+  spec.authors       = ['Richard Huang']
+  spec.email         = ['flyerhzm@gmail.com']
 
   spec.summary       = 'resque-restriction is an extension to resque queue system that restricts the execution number of certain jobs in a period time.'
   spec.description   = 'resque-restriction is an extension to resque queue system that restricts the execution number of certain jobs in a period time, the exceeded jobs will be executed at the next period.'
@@ -15,12 +15,12 @@ Gem::Specification.new do |spec|
   spec.files         = `git ls-files -z`.split("\x0").reject do |f|
     f.match(%r{^(test|spec|features)/})
   end
-  spec.bindir        = "exe"
+  spec.bindir        = 'exe'
   spec.executables   = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
-  spec.require_paths = ["lib"]
+  spec.require_paths = ['lib']
 
-  spec.add_development_dependency "bundler", "~> 1.16"
-  spec.add_development_dependency "rake", "~> 10.0"
-  spec.add_development_dependency "rspec", "~> 3.0"
-  spec.add_development_dependency "resque", ">= 1.7.0"
+  spec.add_development_dependency 'bundler'
+  spec.add_development_dependency 'rake'
+  spec.add_development_dependency 'resque', '>= 1.7.0'
+  spec.add_development_dependency 'rspec', '~> 3.0'
 end

metadata

@@ -1,71 +1,71 @@
 --- !ruby/object:Gem::Specification
 name: resque-restriction
 version: !ruby/object:Gem::Version
-  version: 0.6.0
+  version: 0.7.0
 platform: ruby
 authors:
 - Richard Huang
-autorequire: 
+autorequire:
 bindir: exe
 cert_chain: []
-date: 2018-06-14 00:00:00.000000000 Z
+date: 2021-06-06 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - ">="
       - !ruby/object:Gem::Version
-        version: '1.16'
+        version: '0'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - ">="
       - !ruby/object:Gem::Version
-        version: '1.16'
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: rake
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - ">="
       - !ruby/object:Gem::Version
-        version: '10.0'
+        version: '0'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - ">="
       - !ruby/object:Gem::Version
-        version: '10.0'
+        version: '0'
 - !ruby/object:Gem::Dependency
-  name: rspec
+  name: resque
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - ">="
       - !ruby/object:Gem::Version
-        version: '3.0'
+        version: 1.7.0
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - ">="
       - !ruby/object:Gem::Version
-        version: '3.0'
+        version: 1.7.0
 - !ruby/object:Gem::Dependency
-  name: resque
+  name: rspec
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - ">="
+    - - "~>"
       - !ruby/object:Gem::Version
-        version: 1.7.0
+        version: '3.0'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - ">="
+    - - "~>"
       - !ruby/object:Gem::Version
-        version: 1.7.0
+        version: '3.0'
 description: resque-restriction is an extension to resque queue system that restricts
   the execution number of certain jobs in a period time, the exceeded jobs will be
   executed at the next period.
@@ -76,6 +76,7 @@ extensions: []
 extra_rdoc_files: []
 files:
 - ".gitignore"
+- ".travis.yml"
 - CHANGELOG.md
 - Gemfile
 - LICENSE
@@ -84,12 +85,13 @@ files:
 - lib/resque/plugins/job.rb
 - lib/resque/plugins/restriction.rb
 - lib/resque/restriction.rb
+- lib/resque/restriction/config.rb
 - lib/resque/restriction/version.rb
 - resque-restriction.gemspec
 homepage: https://github.com/flyerhzm/resque-restriction
 licenses: []
 metadata: {}
-post_install_message: 
+post_install_message:
 rdoc_options: []
 require_paths:
 - lib
@@ -104,9 +106,8 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubyforge_project: 
-rubygems_version: 2.6.14
-signing_key: 
+rubygems_version: 3.1.6
+signing_key:
 specification_version: 4
 summary: resque-restriction is an extension to resque queue system that restricts
   the execution number of certain jobs in a period time.