resque-rate_limited_queue 0.0.30

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: b71e99ba935500ac4f784badf3fb036ac4cc9d78
+  data.tar.gz: 93fd4bc5c271b4b2761b8df4e133684d22671bfa
+SHA512:
+  metadata.gz: ceb160532dbd82f4128b480ed324969a9368193ddecb1c3465a72c088adb798b9fe80ebb2c1ecd163f4cd8a93c308c0dfc482fcd2d4377a5b6360969da807f44
+  data.tar.gz: f71914524346ce7f3c8afd53be98f489d0a05ad29cc11f89130fff68084bdf6495f98b3305cda0089d9d24ad177746482e5158b3c3e1a27839839692525cf0f7

data/.gitignore

@@ -0,0 +1,14 @@
+/.bundle/
+/.yardoc
+/Gemfile.lock
+/_yardoc/
+/coverage/
+/doc/
+/pkg/
+/spec/reports/
+/tmp/
+*.bundle
+*.so
+*.o
+*.a
+mkmf.log

data/.rspec

@@ -0,0 +1,2 @@
+--color
+--format progress

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in resque_rate_limited_queue.gemspec
+gemspec

data/LICENSE.txt

@@ -0,0 +1,22 @@
+Copyright (c) 2014 pavoni
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,157 @@
+# Resque Rate Limited Queue
+
+PLEASE NOTE THIS IS STILL EXPERIMENTAL - I EXPECT TO RELEASE A STABLE VERSION IN EARLY JAN 2015
+
+A Resque plugin which makes handling jobs that use rate limited apis easier
+
+If you have a series of jobs in a queue, this gem will pause the queue when one of the jobs hits a rate limit, and re-start the queue when the rate limit has expired.
+
+There are two ways to use the gem.
+
+If the api you are using has a dedicated queue included in the gem (currently Twitter, Angellist and Evernote) then you just need to make some very minor changes to how you queue jobs, and the gem will do the rest.
+
+If you are using another API, then you need to write a little code that catches the rate limit signal.
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'resque-rate-limited-queue'
+```
+
+And then execute:
+
+    $ bundle
+
+Or install it yourself as:
+
+    $ gem install resque-rate-limited-queue
+
+## Usage
+
+### Configuration
+#### Redis
+The gem uses [redis-mutex](https://github.com/kenn/redis-mutex ) which requires you to register the Redis server: (e.g. in `config/initializers/redis_mutex.rb` for Rails)
+
+```ruby
+RedisClassy.redis = Redis.new
+```
+Note that Redis Mutex uses the `redis-classy` gem internally.
+
+#### Un Pause
+Queues can be unpaused in two ways.
+
+The most elegant is using [resque-scheduler](https://github.com/resque/resque-scheduler), this works well as long as you aren't running on a platform like heroku which requires a dedicated worker to run the resque-scheduler.
+
+To tell the gem to use resque-scheduler you need to include resque-scheduler in your Gemfile - and also let the gem know which queue to use to schedule the unpause job (make sure this isn't a queue that could get paused). Put this in an initializer.
+
+```ruby
+Resque::Plugins::RateLimitedQueue::UnPause.queue = :my_queue
+```
+
+Please see the section below on how to unpause on heroku as an alternative. If you don't install resque-scheduler AND configure the queue, then the gem will not schedule unpause jobs this way.
+
+#### Workers
+Queues are paused by renaming them, so a resque queue called 'twitter\_api' will be renamed 'twitter\_api\_paused' when it hits a rate limit. Of course this will only work if your resque workers are not also taking jobs from the 'twitter\_api\_paused' queue. So your worker commands need to look like:
+
+Either
+```ruby
+bin/resque work --queues=high,low,twitter_api
+```
+or
+```ruby
+env QUEUES=high,low,twitter_api bundle exec rake jobs:work
+```
+
+NOT
+```ruby
+bin/resque work --queues=*
+```
+or NOT
+```ruby
+env QUEUES=* bundle exec rake jobs:work
+```
+
+#### Unpausing on heroku
+The built in schedler on heroku doesn't support dynamic scheduling from an API, so unless you want to provision an extra worker to run resque-scheduler - the best option is just to unpause all your queues on a regular basis. If they aren't paused this is a harmless no-op. If not enough time has elapsed the jobs will just hit the rate_limit and get paused again. We've found that a hourly 'rake unpause' job seems to work well. The rake task would need to call:
+
+```ruby
+Resque::Plugins::RateLimitedQueue.TwitterQueue.un_pause
+Resque::Plugins::RateLimitedQueue.AngellistQueue.un_pause
+MyQueue.un_pause
+MyJob.un_pause
+```
+### A Pausable job using one of the build-in queues (Twitter, Angellist, Evernote)
+If you're using the [twitter gem[ (https://github.com/sferik/twitter), this is really simple. Instead of queuing using Resque.enqueue, you just use Resque::Plugins::RateLimitedQueue:TwitterQueue.enqueue.
+
+Make sure your code in perform doesn't catch the rate_limit exception.
+
+The TwitterQueue will catch the exception and pause the queue (as well as re-scheduling the jobs and scheduling an un pause (if you are using resque-scheduler)). Any jobs you add while the queue is paused will be added to the paused queue
+
+```ruby
+class TwitterJob
+  class << self
+    def refresh(handle)
+      Resque::Plugins::RateLimitedQueue:TwitterQueue.enqueue(TwitterJob, handle)
+    end
+
+    def perform(handle)
+      do_something_with_the_twitter_gem(handle)
+    end
+  end
+end
+```
+
+### A single class of pausable job using a new api
+If you only have one class of job you want to queue using the api, then you can use the PauseQueue module directly
+
+```ruby
+class MyApiJob
+  extend Resque::Plugins::RateLimitedQueue
+  @queue = :my_api
+  WAIT_TIME = 60*60
+
+  def self.perform(*params)
+    do_api_stuff
+  rescue MyApiRateLimit
+    pause_for(Time.now + WAIT_TIME, name)
+    rate_limited_requeue(self, *params)
+  end
+
+  def self.enqueue(*params)
+    rate_limited_enqueue(self, *params)
+  end
+end
+````
+
+### Multiple classes of pausable job using a new api
+If you have more than one class of job you want to queue to the api, then you can need to add another Queue class. This isn't hard
+
+```ruby
+class MyApiQueue < Resque::Plugins::RateLimitedQueue::BaseApiQueue
+  @queue = :my_api
+  WAIT_TIME = 60*60
+
+  def self.perform(klass, *params)
+    super
+  rescue MyApiRateLimit
+    pause_for(Time.now + WAIT_TIME, name)
+    rate_limited_requeue(self, klass, *params)
+  end
+end
+````
+If you do this - please contribute - and I'll add to the gem.
+
+## Contributing
+
+1. Fork it ( https://github.com/[my-github-username]/resque_rate_limited_queue/fork )
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes (`git commit -am 'Add some feature'`)
+4. Push to the branch (`git push origin my-new-feature`)
+5. Create a new Pull Request
+
+## Final thoughts
+Thanks to [Dominic](https://github.com/dominicsayers) for idea of renaming the redis key - and the sample  code that does this.
+
+This is my first gem - so please forgive any errors - and feedback very welcome

data/Rakefile

@@ -0,0 +1,2 @@
+require "bundler/gem_tasks"
+

data/lib/resque-rate_limited_queue/version.rb

@@ -0,0 +1,3 @@
+module RateLimitedQueue
+  VERSION = "0.0.30"
+end

data/lib/resque/plugins/rate_limited/apis/angellist_queue.rb

@@ -0,0 +1,19 @@
+require 'angellist_api'
+
+module Resque
+  module Plugins
+    module RateLimitedQueue
+      class AngellistQueue < BaseApiQueue
+        WAIT_TIME = 60
+        @queue = :angellist_api
+
+        def self.perform(klass, *params)
+          super
+        rescue AngellistApi::Error::TooManyRequests
+          pause_for(Time.now + (60 * 60))
+          rate_limited_requeue(self, klass, *params)
+        end
+      end
+    end
+  end
+end

data/lib/resque/plugins/rate_limited/apis/base_api_queue.rb

@@ -0,0 +1,16 @@
+module Resque
+  module Plugins
+    module RateLimitedQueue
+      class BaseApiQueue
+        extend Resque::Plugins::RateLimitedQueue
+        def self.perform(klass, *params)
+          find_class(klass).perform(*params)
+        end
+
+        def self.enqueue(klass, *params)
+          rate_limited_enqueue(self, klass.to_s, *params)
+        end
+      end
+    end
+  end
+end

data/lib/resque/plugins/rate_limited/apis/evernote_queue.rb

@@ -0,0 +1,18 @@
+require 'evernote-thrift'
+
+module Resque
+  module Plugins
+    module RateLimitedQueue
+      class EvernoteQueue < BaseApiQueue
+        @queue = :evernote_api
+
+        def self.perform(klass, *params)
+          super
+        rescue Evernote::EDAM::Error::EDAMSystemException => e
+          pause_for(Time.now + 60 * e.rateLimitDuration.seconds)
+          rate_limited_requeue(self, klass, *params)
+        end
+      end
+    end
+  end
+end

data/lib/resque/plugins/rate_limited/apis/twitter_queue.rb

@@ -0,0 +1,19 @@
+require 'twitter'
+
+module Resque
+  module Plugins
+    module RateLimitedQueue
+      class TwitterQueue < BaseApiQueue
+        @queue = :twitter_api
+
+        def self.perform(klass, *params)
+          super
+        rescue Twitter::Error::TooManyRequests,
+               Twitter::Error::EnhanceYourCalm => e
+          pause_for(Time.now + e.rate_limit.reset_in)
+          rate_limited_requeue(self, klass, *params)
+        end
+      end
+    end
+  end
+end

data/lib/resque/plugins/rate_limited/rate_limited_queue.rb

@@ -0,0 +1,80 @@
+module Resque
+  module Plugins
+    module RateLimitedQueue
+      RESQUE_PREFIX = 'queue:'
+      MUTEX = 'Resque::Plugins::RateLimitedQueue'
+
+      def around_perform_with_check_and_requeue(*params)
+        paused = false
+        with_lock do
+          paused = paused?
+          Resque.enqueue_to(paused_queue_name, self, *params) if paused
+        end
+        return if paused
+        yield
+      end
+
+      def rate_limited_enqueue(klass, *params)
+        with_lock do
+          if paused?
+            Resque.enqueue_to(paused_queue_name, klass, *params)
+          else
+            Resque.enqueue_to(@queue, klass, *params)
+          end
+        end
+      end
+
+      def rate_limited_requeue(klass, *params)
+        # split from above to make it easy to stub for testing
+        rate_limited_enqueue(klass, *params)
+      end
+
+      def pause_for(timestamp)
+        pause
+        UnPause.enqueue(timestamp, name)
+      end
+
+      def un_pause
+        with_lock do
+          if paused?
+            begin
+              Resque.redis.renamenx(RESQUE_PREFIX + paused_queue_name, RESQUE_PREFIX + @queue.to_s)
+            rescue Redis::CommandError => e
+              raise unless e.message == 'ERR no such key'
+            end
+          end
+        end
+      end
+
+      def pause
+        Resque.redis.renamenx(RESQUE_PREFIX + @queue.to_s, RESQUE_PREFIX + paused_queue_name)
+      rescue Redis::CommandError => e
+        raise unless e.message == 'ERR no such key'
+      end
+
+      def paused?
+        Resque.redis.exists(RESQUE_PREFIX + paused_queue_name)
+      end
+
+      def paused_queue_name
+        @queue.to_s + '_paused'
+      end
+
+      def with_lock
+        if Resque.inline
+          yield
+        else
+          RedisMutex.with_lock(MUTEX, block: 60, expire: 120) { yield }
+        end
+      end
+
+      def find_class(klass)
+        return klass if klass.is_a? Class
+        return Object.const_get(klass) unless klass.include?('::')
+        klass.split('::').reduce(Object) do |mod, class_name|
+          mod.const_get(class_name)
+        end
+      end
+    end
+  end
+end

data/lib/resque/plugins/rate_limited/rate_limited_un_pause.rb

@@ -0,0 +1,37 @@
+module Resque
+  module Plugins
+    module RateLimitedQueue
+      class UnPause
+        @queue = nil
+
+        class << self
+          attr_writer(:queue)
+
+          def use?
+            Resque.respond_to?(:enqueue_at_with_queue) && @queue
+          end
+
+          def enqueue(timestamp, klass)
+            # If Resque scheduler is installed and queue is set - use it to queue a wake up job
+            return unless use?
+            Resque.enqueue_at_with_queue(
+              @queue,
+              timestamp,
+              Resque::Plugins::RateLimitedQueue::UnPause,
+              klass)
+          end
+
+          def perform(klass)
+            class_from_string(klass.to_s).un_pause
+          end
+
+          def class_from_string(str)
+            str.split('::').reduce(Object) do |mod, class_name|
+              mod.const_get(class_name)
+            end
+          end
+        end
+      end
+    end
+  end
+end

data/lib/resque/rate_limited_queue.rb

@@ -0,0 +1,9 @@
+require 'resque'
+require 'redis-mutex'
+require 'resque/version'
+require 'resque/plugins/rate_limited/rate_limited_queue'
+require 'resque/plugins/rate_limited/rate_limited_un_pause'
+require 'resque/plugins/rate_limited/apis/base_api_queue'
+require 'resque/plugins/rate_limited/apis/angellist_queue'
+require 'resque/plugins/rate_limited/apis/evernote_queue'
+require 'resque/plugins/rate_limited/apis/twitter_queue'

data/resque-rate_limited_queue.gemspec

@@ -0,0 +1,40 @@
+# coding: utf-8
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'resque-rate_limited_queue/version'
+
+Gem::Specification.new do |spec|
+  spec.name          = "resque-rate_limited_queue"
+  spec.version       = RateLimitedQueue::VERSION
+  spec.authors       = ["Greg Dowling"]
+  spec.email         = ["mail@greddowling.com"]
+  spec.summary     = %q{A Resque plugin to help manage jobs that use rate limited apis, pausing when you hit the limits and restarting later.}
+  spec.description = %q{A Resque plugin which allows you to create dedicated queues for jobs that use rate limited apis.
+These queues will pause when one of the jobs hits a rate limit, and unpause after a suitable time period.
+The rate_limited_queue can be used directly, and just requires catching the rate limit exception and pausing the
+queue. There are also additional queues provided that already include the pause/rety logic for twitter, angelist
+and evernote; these allow you to support rate limited apis with minimal changes.}
+
+  spec.homepage      = ""
+  spec.license       = "MIT"
+
+  spec.files         = `git ls-files -z`.split("\x0")
+  spec.executables   = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
+  spec.test_files    = spec.files.grep(%r{^(test|spec|features)/})
+  spec.require_paths = ["lib"]
+
+  spec.add_dependency('resque', '~> 1.9', '>= 1.9.10')
+  spec.add_dependency('redis-mutex','~> 4.0', '>= 4.0.0')
+
+  spec.add_dependency("angellist_api", '~> 1.0', '>= 1.0.7')
+  spec.add_dependency("evernote-thrift", '~> 1.25', '>= 1.25.1')
+  spec.add_dependency("twitter", '~> 5.11', '>= 5.11.0')
+
+
+  spec.add_development_dependency("bundler", "~> 1.7")
+  spec.add_development_dependency("rake", "~> 10.0")
+  spec.add_development_dependency("rspec", "~> 2.6")
+  spec.add_development_dependency("simplecov", '~> 0.9.1')
+
+
+end

data/spec/apis/angellist_queue_spec.rb

@@ -0,0 +1,46 @@
+require 'spec_helper'
+require 'resque_rate_limited_queue'
+
+class RateLimitedTestQueueAL
+  def self.perform(succeed)
+    raise(AngellistApi::Error::TooManyRequests, 'error') unless succeed
+  end
+end
+
+describe Resque::Plugins::RateLimitedQueue::AngellistQueue do
+  describe 'enqueue' do
+    it 'enqueues to the correct queue with the correct parameters' do
+      Resque.should_receive(:enqueue_to).with(
+        :angellist_api,
+        Resque::Plugins::RateLimitedQueue::AngellistQueue,
+        RateLimitedTestQueueAL.to_s,
+        true)
+      Resque::Plugins::RateLimitedQueue::AngellistQueue
+        .enqueue(RateLimitedTestQueueAL, true)
+    end
+  end
+
+  describe 'perform' do
+    before do
+      Resque.inline = true
+    end
+    context 'with everything' do
+      it 'calls the class with the right parameters' do
+        RateLimitedTestQueueAL.should_receive(:perform).with('test_param')
+        Resque::Plugins::RateLimitedQueue::AngellistQueue
+          .enqueue(RateLimitedTestQueueAL, 'test_param')
+      end
+    end
+
+    context 'with rate limit exception' do
+      before do
+        Resque::Plugins::RateLimitedQueue::AngellistQueue.stub(:rate_limited_requeue)
+      end
+      it 'pauses queue when request fails' do
+        Resque::Plugins::RateLimitedQueue::AngellistQueue.should_receive(:pause_for)
+        Resque::Plugins::RateLimitedQueue::AngellistQueue
+          .enqueue(RateLimitedTestQueueAL, false)
+      end
+    end
+  end
+end

data/spec/apis/evernote_queue_spec.rb

@@ -0,0 +1,53 @@
+require 'spec_helper'
+require 'resque_rate_limited_queue'
+
+class RateLimitDuration
+  def self.seconds
+    60
+  end
+end
+
+class RateLimitedTestQueueEn
+  def self.perform(succeed)
+    raise(Evernote::EDAM::Error::EDAMSystemException, rateLimitDuration: RateLimitDuration) unless succeed
+  end
+end
+
+describe Resque::Plugins::RateLimitedQueue::EvernoteQueue do
+  describe 'enqueue' do
+    it 'enqueues to the correct queue with the correct parameters' do
+      Resque.should_receive(:enqueue_to).with(
+        :evernote_api,
+        Resque::Plugins::RateLimitedQueue::EvernoteQueue,
+        RateLimitedTestQueueEn.to_s,
+        true)
+      Resque::Plugins::RateLimitedQueue::EvernoteQueue
+        .enqueue(RateLimitedTestQueueEn, true)
+    end
+  end
+
+  describe 'perform' do
+    before do
+      Resque.inline = true
+    end
+    context 'with everything' do
+      it 'calls the class with the right parameters' do
+        RateLimitedTestQueueEn.should_receive(:perform).with('test_param')
+        Resque::Plugins::RateLimitedQueue::EvernoteQueue
+          .enqueue(RateLimitedTestQueueEn, 'test_param')
+      end
+    end
+
+    context 'with rate limit exception' do
+      before do
+        Resque::Plugins::RateLimitedQueue::EvernoteQueue.stub(:rate_limited_requeue)
+      end
+      it 'pauses queue when request fails' do
+        Resque::Plugins::RateLimitedQueue::EvernoteQueue.should_receive(:pause_for)
+        Resque::Plugins::RateLimitedQueue::EvernoteQueue
+          .enqueue(RateLimitedTestQueueEn, false)
+      end
+    end
+  end
+end
+

data/spec/apis/twitter_queue_spec.rb

@@ -0,0 +1,47 @@
+require 'spec_helper'
+require 'resque_rate_limited_queue'
+
+class RateLimitedTestQueueTw
+  def self.perform(succeed)
+    raise(Twitter::Error::TooManyRequests
+      .new('', 'x-rate-limit-reset' => (Time.now + 60).to_i)) unless succeed
+  end
+end
+
+describe Resque::Plugins::RateLimitedQueue::TwitterQueue do
+  describe 'enqueue' do
+    it 'enqueues to the correct queue with the correct parameters' do
+      Resque.should_receive(:enqueue_to).with(
+        :twitter_api,
+        Resque::Plugins::RateLimitedQueue::TwitterQueue,
+        RateLimitedTestQueueTw.to_s,
+        true)
+      Resque::Plugins::RateLimitedQueue::TwitterQueue
+        .enqueue(RateLimitedTestQueueTw, true)
+    end
+  end
+
+  describe 'perform' do
+    before do
+      Resque.inline = true
+    end
+    context 'with everything' do
+      it 'calls the class with the right parameters' do
+        RateLimitedTestQueueTw.should_receive(:perform).with('test_param')
+        Resque::Plugins::RateLimitedQueue::TwitterQueue
+          .enqueue(RateLimitedTestQueueTw, 'test_param')
+      end
+    end
+
+    context 'with rate limit exception' do
+      before do
+        Resque::Plugins::RateLimitedQueue::TwitterQueue.stub(:rate_limited_requeue)
+      end
+      it 'pauses queue when request fails' do
+        Resque::Plugins::RateLimitedQueue::TwitterQueue.should_receive(:pause_for)
+        Resque::Plugins::RateLimitedQueue::TwitterQueue
+          .enqueue(RateLimitedTestQueueTw, false)
+      end
+    end
+  end
+end

data/spec/rate_limited_queue_spec.rb

@@ -0,0 +1,177 @@
+require 'spec_helper'
+require 'resque_rate_limited_queue'
+
+class RateLimitedTestQueue
+  extend Resque::Plugins::RateLimitedQueue
+
+  @queue = :test
+
+  def self.perform(succeed)
+    rate_limited_requeue(self, succeed) unless succeed
+  end
+
+  def self.queue_name
+    @queue.to_s
+  end
+
+  def self.queue
+    @queue
+  end
+end
+
+describe Resque::Plugins::RateLimitedQueue do
+  it 'should be compliance with Resque::Plugin document' do
+    expect { Resque::Plugin.lint(Resque::Plugins::RateLimitedQueue) }.to_not raise_error
+  end
+
+  shared_examples_for 'queue' do |queue_suffix|
+    it 'should queue to the correct queue' do
+      queue_param = queue_suffix.empty? ? RateLimitedTestQueue.queue : "#{RateLimitedTestQueue.queue_name}#{queue_suffix}"
+      Resque.should_receive(:enqueue_to).with(queue_param, nil, nil)
+      RateLimitedTestQueue.rate_limited_enqueue(nil, nil)
+    end
+  end
+
+  context 'when queue is not paused' do
+    before do
+      RateLimitedTestQueue.stub(:paused?).and_return(false)
+    end
+
+    describe 'enqueue' do
+      include_examples 'queue', ''
+    end
+
+    describe 'paused?' do
+      it { RateLimitedTestQueue.paused?.should be false }
+    end
+
+    describe 'perform' do
+      it 'should requeue the job on failure' do
+        Resque.should_receive(:enqueue_to)
+        RateLimitedTestQueue.perform(false)
+      end
+
+      it 'should not requeue the job on success' do
+        Resque.should_not_receive(:enqueue_to)
+        RateLimitedTestQueue.perform(true)
+      end
+
+    end
+
+    describe 'pause' do
+      it 'should rename the queue to paused' do
+        Resque.redis.should_receive(:renamenx).with("queue:#{RateLimitedTestQueue.queue_name}", "queue:#{RateLimitedTestQueue.queue_name}_paused")
+        RateLimitedTestQueue.pause
+      end
+    end
+
+    describe 'un_pause' do
+      it 'should not unpause the queue' do
+        Resque.redis.should_not_receive(:renamenx).with("queue:#{RateLimitedTestQueue.queue_name}", "queue:#{RateLimitedTestQueue.queue_name}_paused")
+        RateLimitedTestQueue.un_pause
+      end
+    end
+
+    describe 'pause_for' do
+      before do
+        Resque.redis.stub(:renamenx).and_return(true)
+      end
+
+      it 'should pause the queue' do
+        RateLimitedTestQueue.should_receive(:pause)
+        RateLimitedTestQueue.pause_for(Time.now + (5 * 60 * 60))
+      end
+
+      it 'should schedule an unpause job' do
+        Resque::Plugins::RateLimitedQueue::UnPause.should_receive(:enqueue)
+          .with(nil, 'RateLimitedTestQueue')
+        RateLimitedTestQueue.pause_for(nil)
+      end
+    end
+  end
+
+  context 'when queue is paused' do
+    before do
+      RateLimitedTestQueue.stub(:paused?).and_return(true)
+    end
+
+    describe 'enqueue' do
+      include_examples 'queue', '_paused'
+    end
+
+    describe 'paused?' do
+      it { RateLimitedTestQueue.paused?.should be true }
+    end
+
+    describe 'perform' do
+      it 'should not execute the block' do
+        Resque.should_receive(:enqueue_to).with("#{RateLimitedTestQueue.queue_name}_paused", RateLimitedTestQueue, true)
+        RateLimitedTestQueue.should_not_receive(:perform)
+        RateLimitedTestQueue.around_perform_with_check_and_requeue(true)
+      end
+    end
+
+    describe 'un_pause' do
+      it 'should rename the queue to live' do
+        Resque.redis.should_receive(:renamenx).with("queue:#{RateLimitedTestQueue.queue_name}_paused", "queue:#{RateLimitedTestQueue.queue_name}")
+        RateLimitedTestQueue.un_pause
+      end
+    end
+  end
+
+  describe 'find_class' do
+    it 'works with symbol' do
+      RateLimitedTestQueue.find_class(RateLimitedTestQueue).should eq RateLimitedTestQueue
+    end
+
+    it 'works with simple string' do
+      RateLimitedTestQueue.find_class('RateLimitedTestQueue').should eq RateLimitedTestQueue
+    end
+
+    it 'works with complex string' do
+      RateLimitedTestQueue.find_class('Resque::Plugins::RateLimitedQueue').should eq Resque::Plugins::RateLimitedQueue
+    end
+  end
+
+  context 'with redis errors' do
+    before do
+      RateLimitedTestQueue.stub(:paused?).and_return(true)
+    end
+    context 'with not found error' do
+      before do
+        Resque.redis.stub(:renamenx).and_raise(Redis::CommandError.new('ERR no such key'))
+      end
+
+      describe 'pause' do
+        it 'should not throw exception' do
+          expect { RateLimitedTestQueue.pause }.to_not raise_error
+        end
+      end
+
+      describe 'un_pause' do
+        it 'should not throw exception' do
+          expect { RateLimitedTestQueue.un_pause }.to_not raise_error
+        end
+      end
+
+    end
+
+    context 'with other errror' do
+      before do
+        Resque.redis.stub(:renamenx).and_raise(Redis::CommandError.new('ERR something else'))
+      end
+
+      describe 'pause' do
+        it 'should throw exception' do
+          expect { RateLimitedTestQueue.pause }.to raise_error(Redis::CommandError)
+        end
+      end
+
+      describe 'un_pause' do
+        it 'should throw exception' do
+          expect { RateLimitedTestQueue.un_pause }.to raise_error(Redis::CommandError)
+        end
+      end
+    end
+  end
+end

data/spec/rate_limited_un_pause_spec.rb

@@ -0,0 +1,38 @@
+require 'spec_helper'
+require 'resque_rate_limited_queue'
+
+class RateLimitedTestQueue
+end
+
+describe Resque::Plugins::RateLimitedQueue::UnPause do
+  describe 'perform' do
+    it 'unpauses the queue' do
+      RateLimitedTestQueue.should_receive(:un_pause)
+      Resque::Plugins::RateLimitedQueue::UnPause.perform(RateLimitedTestQueue)
+    end
+  end
+
+  describe 'enqueue' do
+    before { Resque.stub(:respond_to?).and_return(true) }
+    context 'with no queue defined' do
+      it 'does not queue the job' do
+        Resque.should_not_receive(:enqueue_at_with_queue)
+        Resque::Plugins::RateLimitedQueue::UnPause.enqueue(Time.now, RateLimitedTestQueue)
+      end
+    end
+
+    context 'with queue defined' do
+      before { Resque::Plugins::RateLimitedQueue::UnPause.queue = :queue_name }
+      it 'queues the job' do
+        Resque.should_receive(:enqueue_at_with_queue).with(
+          :queue_name,
+          nil,
+          Resque::Plugins::RateLimitedQueue::UnPause,
+          RateLimitedTestQueue)
+
+        Resque::Plugins::RateLimitedQueue::UnPause.enqueue(nil, RateLimitedTestQueue)
+      end
+    end
+  end
+
+end

data/spec/spec_helper.rb

@@ -0,0 +1,18 @@
+require 'rubygems'
+require 'bundler/setup'
+
+require 'rspec'
+require 'redis-classy'
+require 'redis-mutex'
+
+require 'simplecov'
+
+SimpleCov.start
+
+RSpec.configure do |config|
+  # Use database 15 for testing so we don't accidentally step on your real data.
+  RedisClassy.redis = Redis.new(db: 15)
+  unless RedisClassy.keys.empty?
+    abort '[ERROR]: Redis database 15 not empty! If you are sure, run "rake flushdb" beforehand.'
+  end
+end

metadata

@@ -0,0 +1,233 @@
+--- !ruby/object:Gem::Specification
+name: resque-rate_limited_queue
+version: !ruby/object:Gem::Version
+  version: 0.0.30
+platform: ruby
+authors:
+- Greg Dowling
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2014-12-28 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: resque
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.9'
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 1.9.10
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.9'
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 1.9.10
+- !ruby/object:Gem::Dependency
+  name: redis-mutex
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '4.0'
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 4.0.0
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '4.0'
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 4.0.0
+- !ruby/object:Gem::Dependency
+  name: angellist_api
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.0'
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 1.0.7
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.0'
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 1.0.7
+- !ruby/object:Gem::Dependency
+  name: evernote-thrift
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.25'
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 1.25.1
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.25'
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 1.25.1
+- !ruby/object:Gem::Dependency
+  name: twitter
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '5.11'
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 5.11.0
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '5.11'
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 5.11.0
+- !ruby/object:Gem::Dependency
+  name: bundler
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.7'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.7'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '10.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '10.0'
+- !ruby/object:Gem::Dependency
+  name: rspec
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.6'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.6'
+- !ruby/object:Gem::Dependency
+  name: simplecov
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 0.9.1
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 0.9.1
+description: |-
+  A Resque plugin which allows you to create dedicated queues for jobs that use rate limited apis.
+  These queues will pause when one of the jobs hits a rate limit, and unpause after a suitable time period.
+  The rate_limited_queue can be used directly, and just requires catching the rate limit exception and pausing the
+  queue. There are also additional queues provided that already include the pause/rety logic for twitter, angelist
+  and evernote; these allow you to support rate limited apis with minimal changes.
+email:
+- mail@greddowling.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- ".gitignore"
+- ".rspec"
+- Gemfile
+- LICENSE.txt
+- README.md
+- Rakefile
+- lib/resque-rate_limited_queue/version.rb
+- lib/resque/plugins/rate_limited/apis/angellist_queue.rb
+- lib/resque/plugins/rate_limited/apis/base_api_queue.rb
+- lib/resque/plugins/rate_limited/apis/evernote_queue.rb
+- lib/resque/plugins/rate_limited/apis/twitter_queue.rb
+- lib/resque/plugins/rate_limited/rate_limited_queue.rb
+- lib/resque/plugins/rate_limited/rate_limited_un_pause.rb
+- lib/resque/rate_limited_queue.rb
+- resque-rate_limited_queue.gemspec
+- spec/apis/angellist_queue_spec.rb
+- spec/apis/evernote_queue_spec.rb
+- spec/apis/twitter_queue_spec.rb
+- spec/rate_limited_queue_spec.rb
+- spec/rate_limited_un_pause_spec.rb
+- spec/spec_helper.rb
+homepage: ''
+licenses:
+- MIT
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.4.2
+signing_key: 
+specification_version: 4
+summary: A Resque plugin to help manage jobs that use rate limited apis, pausing when
+  you hit the limits and restarting later.
+test_files:
+- spec/apis/angellist_queue_spec.rb
+- spec/apis/evernote_queue_spec.rb
+- spec/apis/twitter_queue_spec.rb
+- spec/rate_limited_queue_spec.rb
+- spec/rate_limited_un_pause_spec.rb
+- spec/spec_helper.rb