sidekiq-rate-limiter 0.0.1 → 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: eed802823e01b40545973fe5e2ddd21c5fa85002
+  data.tar.gz: 33ffa14b502ca8872d1573752c0e3eab5fc9799e
+SHA512:
+  metadata.gz: ae5a1738821693e8a9bde19b9495cdbf4e87fd9ec7d20457460ec42a76c327cac522db118032178e833b2c83ec7328b89d0f75d19bac2f0fd6c34c0d4bbd30a3
+  data.tar.gz: 79e48fc5d5b222bdb3e1bc0cedb0b4a9b32372508ef284b4f246fd8c235c5cbaedf1d80f3d9e9048241bbcf2e3f3ade04756284a8169e6d4f49f508d30706e7a

data/CHANGELOG.md

@@ -0,0 +1,11 @@
+## 0.1.0 (May 30, 2014)
+
+* Support callables for all configuration options
+
+  *Kevin Yank*
+
+## 0.0.1 (December 3, 2013)
+
+* Initial release.
+
+  *Blake Thomas*

data/README.md

@@ -3,8 +3,7 @@ sidekiq-rate-limiter
 
 [![Build Status](https://secure.travis-ci.org/enova/sidekiq-rate-limiter.png)](http://travis-ci.org/enova/sidekiq-rate-limiter)
 
-Adds to sidekiq the ability to rate limit job execution on a per-worker basis in
-a redis-backed fashion.
+Redis-backed, per-worker rate limits for job processing.
 
 ## Compatibility
 
@@ -12,10 +11,10 @@ sidekiq-rate-limiter is actively tested against MRI versions 2.0.0 and 1.9.3.
 
 sidekiq-rate-limiter works by using a custom fetch class, the class responsible
 for pulling work from the queue stored in redis. Consequently you'll want to be
-careful about using other gems that use a same strategy ([sidekiq-priority](https://github.com/socialpandas/sidekiq-priority)
+careful about using other gems that use a same strategy, [sidekiq-priority](https://github.com/socialpandas/sidekiq-priority)
 being one example.
 
-I've attempted to support the same options as used by [sidekiq-throttler](/gevans/sidekiq-throttler). So, if
+I've attempted to support the same options as used by [sidekiq-throttler](https://github.com/gevans/sidekiq-throttler). So, if
 your worker already looks like this example I lifted from the sidekiq-throttler wiki:
 
 ```ruby
@@ -87,15 +86,37 @@ The configuration above would result in any jobs beyond the first 50 in a one
 hour period being delayed. The server will continue to fetch items from redis, &
 will place any items that are beyond the threshold at the back of their queue.
 
+### Dynamic Configuration
+
+The simplest way to set the rate-limiting options (`:name`, `:limit`, and `:period`) is to assign them each a static value (as above). In some cases, you may wish to calculate values for these options for each specific job. You can do this by supplying a `Proc` for any or all of these options.
+
+The `Proc` may receive as its arguments the same values that will be passed to `perform` when the job is finally performed.
+
+```ruby
+class Job
+  include Sidekiq::Worker
+
+  sidekiq_options :queue => "my_queue",
+                  :rate => {
+                    :name   => ->(user_id, rate_limit) { user_id },
+                    :limit  => ->(user_id, rate_limit) { rate_limit },
+                    :period => ->{ Date.today.monday? ? 2.hours : 4.hours }, # can ignore arguments
+                  }
+
+  def perform(user_id, rate_limit)
+    ## do something
+```
+
+**Caveat**: Normally, Sidekiq stores the `sidekiq_options` with the job on your Redis server at the time the job is enqueued, and it is these stored values that are used for rate-limiting. This means that if you deploy a new version of your code with different `sidekiq_options`, the already-queued jobs will continue to behave according to the options that were in place when they were created. When you supply a `Proc` for one or more of your configuration options, your rate-limiting options can no longer be stored in Redis, but must instead be calculated when the job is fetched by your Sidekiq server for potential execution. If your application code changes while a job is in the queue, it may run with different `sidekiq_options` than existed when it was first enqueued.
+
 ## Motivation
 
 Sidekiq::Throttler is great for smaller quantities of jobs, but falls down a bit
-for larger queues (see [issue #8](/gevans/sidekiq-throttler/issues/8). In addition, jobs that are
-limited multiple times are counted as 'processed' each time, so the stats baloon quickly.
+for larger queues (see [issue #8](https://github.com/gevans/sidekiq-throttler/issues/8)). In addition, jobs that are
+limited multiple times are counted as 'processed' each time, so the stats balloon quickly.
 
 ## TODO
 
-* Most or all of the configuration options should support procs
 * While it subclasses instead of monkey patching, setting Sidekiq.options[:fetch]
 is still asking for interaction issues. It would be better for this to be directly
 in sidekiq or to use some other means to accomplish this goal.

data/lib/sidekiq-rate-limiter/fetch.rb

@@ -14,18 +14,20 @@ module Sidekiq::RateLimiter
     def limit(work)
       message = JSON.parse(work.message) rescue {}
 
+      args      = message['args']
       klass     = message['class']
-      rate      = message['rate'] || message['throttle'] || {}
-      limit     = rate['limit']   || rate['threshold']
-      interval  = rate['period']  || rate['interval']
-      name      = rate['name']    || DEFAULT_LIMIT_NAME
+      rate      = Rate.new(message)
 
-      return work unless !!(klass && limit && interval)
+      return work unless !!(klass && rate.valid?)
+
+      limit     = rate.limit
+      interval  = rate.interval
+      name      = rate.name
 
       options = {
-        :limit    => limit,
-        :interval => interval,
-        :name     => name,
+        :limit    => (limit.respond_to?(:call) ? limit.call(*args) : limit).to_i,
+        :interval => (interval.respond_to?(:call) ? interval.call(*args) : interval).to_f,
+        :name     => (name.respond_to?(:call) ? name.call(*args) : name).to_s,
       }
 
       Sidekiq.redis do |conn|
@@ -39,6 +41,56 @@ module Sidekiq::RateLimiter
         end
       end
     end
+
+  end
+
+  class Rate
+    def initialize(message)
+      @message = message
+    end
+
+    def limit
+      rate['limit'] || rate['threshold']
+    end
+
+    def interval
+      rate['interval'] || rate['period']
+    end
+
+    def name
+      rate['name'] || DEFAULT_LIMIT_NAME
+    end
+
+    def valid?
+      !!(limit && interval)
+    end
+
+    private
+
+    def rate
+      use_server_rate? ? server_rate : client_rate
+    end
+
+    def use_server_rate?
+      server_rate['limit'] && server_rate['limit'].respond_to?(:call) ||
+        server_rate['threshold'] && server_rate['threshold'].respond_to?(:call) ||
+        server_rate['period'] && server_rate['period'].respond_to?(:call) ||
+        server_rate['interval'] && server_rate['interval'].respond_to?(:call) ||
+        server_rate['name'] && server_rate['name'].respond_to?(:call)
+    end
+
+    def client_rate
+      @client_rate ||= @message['rate'] || @message['throttle'] || {}
+    end
+
+    def server_rate
+      return @server_rate if @server_rate
+
+      worker_class = @message['class']
+      options = Object.const_get(worker_class).get_sidekiq_options rescue {}
+      server_rate = options['rate'] || options['throttle'] || {}
+      @server_rate = server_rate.stringify_keys
+    end
   end
 
   class Limit < RedisRateLimiter

data/lib/sidekiq-rate-limiter/version.rb

@@ -1,5 +1,5 @@
 module Sidekiq
   module RateLimiter
-    VERSION = "0.0.1"
+    VERSION = "0.1.0"
   end
 end

data/sidekiq-rate-limiter.gemspec

@@ -11,8 +11,8 @@ Gem::Specification.new do |s|
   s.authors     = ["Blake Thomas", "Enova"]
   s.email       = ["bwthomas@gmail.com"]
   s.homepage    = "https://github.com/enova/sidekiq-rate-limiter"
-  s.summary     = %q{Rate-limit Sidekiq fetches by worker class}
-  s.description = %q{Rate-limit Sidekiq fetches by worker class}
+  s.summary     = %q{Redis-backed, per-worker rate limits for job processing}
+  s.description = %q{Redis-backed, per-worker rate limits for job processing}
   s.rubyforge_project = "nowarning"
 
   s.files         = `git ls-files`.split("\n")
@@ -27,6 +27,6 @@ Gem::Specification.new do |s|
   s.add_development_dependency "simplecov-rcov"
 
   s.add_dependency "redis"
-  s.add_dependency "sidekiq"
+  s.add_dependency "sidekiq", ">= 2.0", "< 4.0"
   s.add_dependency "redis_rate_limiter"
 end

data/spec/sidekiq-rate-limiter/fetch_spec.rb

@@ -1,5 +1,6 @@
 require 'spec_helper'
 require 'sidekiq'
+require 'sidekiq/api'
 
 describe Sidekiq::RateLimiter::Fetch do
   before(:all) do
@@ -13,6 +14,17 @@ describe Sidekiq::RateLimiter::Fetch do
                       }
       def perform(*args); end
     end
+    class ProcJob
+      include Sidekiq::Worker
+      sidekiq_options 'queue'    => 'basic',
+                      'retry'    => false,
+                      'rate' => {
+                          'limit'  => ->(arg1, arg2) { arg2 },
+                          'name'   => ->(arg1, arg2) { arg2 },
+                          'period' => ->(arg1, arg2) { arg2 }
+                      }
+      def perform(arg1, arg2); end
+    end
   end
 
   let(:options)       { { queues: [queue, another_queue, another_queue] } }
@@ -20,6 +32,7 @@ describe Sidekiq::RateLimiter::Fetch do
   let(:another_queue) { 'some_other_queue' }
   let(:args)          { ['I am some args'] }
   let(:worker)        { Job }
+  let(:proc_worker)   { ProcJob }
   let(:redis_class)   { Sidekiq.redis { |conn| conn.class } }
 
   it 'should inherit from Sidekiq::BasicFetch' do
@@ -61,4 +74,13 @@ describe Sidekiq::RateLimiter::Fetch do
     q.size.should == 1
   end
 
+  it 'should accept procs for limit, name, and period config keys', queuing: true do
+    proc_worker.perform_async(1,2)
+
+    Sidekiq::RateLimiter::Limit.should_receive(:new).with(anything(), {:limit => 2, :interval => 2, :name => "2"}).and_call_original
+
+    fetch = described_class.new(options)
+    work   = fetch.retrieve_work
+  end
+
 end

metadata

@@ -1,8 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: sidekiq-rate-limiter
 version: !ruby/object:Gem::Version
-  version: 0.0.1
-  prerelease: 
+  version: 0.1.0
 platform: ruby
 authors:
 - Blake Thomas
@@ -10,137 +9,127 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2013-12-03 00:00:00.000000000 Z
+date: 2014-05-30 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: pry
   requirement: !ruby/object:Gem::Requirement
-    none: false
     requirements:
-    - - ! '>='
+    - - '>='
       - !ruby/object:Gem::Version
         version: '0'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
-    none: false
     requirements:
-    - - ! '>='
+    - - '>='
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
   name: rake
   requirement: !ruby/object:Gem::Requirement
-    none: false
     requirements:
-    - - ! '>='
+    - - '>='
       - !ruby/object:Gem::Version
         version: '0'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
-    none: false
     requirements:
-    - - ! '>='
+    - - '>='
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
   name: rspec
   requirement: !ruby/object:Gem::Requirement
-    none: false
     requirements:
-    - - ! '>='
+    - - '>='
       - !ruby/object:Gem::Version
         version: '0'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
-    none: false
     requirements:
-    - - ! '>='
+    - - '>='
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
   name: simplecov
   requirement: !ruby/object:Gem::Requirement
-    none: false
     requirements:
-    - - ! '>='
+    - - '>='
       - !ruby/object:Gem::Version
         version: '0'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
-    none: false
     requirements:
-    - - ! '>='
+    - - '>='
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
   name: simplecov-rcov
   requirement: !ruby/object:Gem::Requirement
-    none: false
     requirements:
-    - - ! '>='
+    - - '>='
       - !ruby/object:Gem::Version
         version: '0'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
-    none: false
     requirements:
-    - - ! '>='
+    - - '>='
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
   name: redis
   requirement: !ruby/object:Gem::Requirement
-    none: false
     requirements:
-    - - ! '>='
+    - - '>='
       - !ruby/object:Gem::Version
         version: '0'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
-    none: false
     requirements:
-    - - ! '>='
+    - - '>='
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
   name: sidekiq
   requirement: !ruby/object:Gem::Requirement
-    none: false
     requirements:
-    - - ! '>='
+    - - '>='
       - !ruby/object:Gem::Version
-        version: '0'
+        version: '2.0'
+    - - <
+      - !ruby/object:Gem::Version
+        version: '4.0'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
-    none: false
     requirements:
-    - - ! '>='
+    - - '>='
       - !ruby/object:Gem::Version
-        version: '0'
+        version: '2.0'
+    - - <
+      - !ruby/object:Gem::Version
+        version: '4.0'
 - !ruby/object:Gem::Dependency
   name: redis_rate_limiter
   requirement: !ruby/object:Gem::Requirement
-    none: false
     requirements:
-    - - ! '>='
+    - - '>='
       - !ruby/object:Gem::Version
         version: '0'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
-    none: false
     requirements:
-    - - ! '>='
+    - - '>='
       - !ruby/object:Gem::Version
         version: '0'
-description: Rate-limit Sidekiq fetches by worker class
+description: Redis-backed, per-worker rate limits for job processing
 email:
 - bwthomas@gmail.com
 executables: []
@@ -151,6 +140,7 @@ files:
 - .rspec
 - .simplecov
 - .travis.yml
+- CHANGELOG.md
 - Gemfile
 - LICENSE
 - README.md
@@ -167,31 +157,29 @@ files:
 homepage: https://github.com/enova/sidekiq-rate-limiter
 licenses:
 - MIT
+metadata: {}
 post_install_message: 
 rdoc_options: []
 require_paths:
 - lib
 required_ruby_version: !ruby/object:Gem::Requirement
-  none: false
   requirements:
-  - - ! '>='
+  - - '>='
     - !ruby/object:Gem::Version
       version: '0'
 required_rubygems_version: !ruby/object:Gem::Requirement
-  none: false
   requirements:
-  - - ! '>='
+  - - '>='
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
 rubyforge_project: nowarning
-rubygems_version: 1.8.23
+rubygems_version: 2.0.14
 signing_key: 
-specification_version: 3
-summary: Rate-limit Sidekiq fetches by worker class
+specification_version: 4
+summary: Redis-backed, per-worker rate limits for job processing
 test_files:
 - spec/sidekiq-rate-limiter/fetch_spec.rb
 - spec/sidekiq-rate-limiter/server_spec.rb
 - spec/spec_helper.rb
 - spec/support/redis/.keep
-has_rdoc: