resque-director 2.1.1 → 2.2.0

data/README.rdoc

@@ -4,9 +4,7 @@ Resque Director is a plugin for the Resque queueing system (http://github.com/de
 
 ==About
 
-resque-director is mainly useful for when you are managing a large number of workers and don't want to waste resources keeping all of them waiting when they are not being used. Also useful in queues where the influx of jobs can change dramatically from time to time: enabling more workers during the times when the queue is filling up more quickly, and less in the opposite scenario. Different queues can be given different directions as well.
-
-resque-director does not currently operate on workers that serve multiple queues, and does not include those workers in any scaling calculations.
+resque-director is mainly useful for when you are managing a large number of workers and don't want to waste resources keeping all of them waiting when they are not being used. Also for jobs that are high priority or time sensitive you can have it autoscale workers when the time it took for the job to go through the queue is too long. For this reason resque director is useful in queues where the influx of jobs can change dramatically from time to time: enabling more workers during the times when the queue is filling up more quickly, and less in the opposite scenario. Different queues can be given different directions as well.
 
 == Usage
 
@@ -35,15 +33,28 @@ For Example:
 
 <b>wait_time</b>:: the time in seconds that it will wait after adding or removing a worker before being allowed to add or remove workers again. The default is 60 seconds.
 
-=== Scale Options
+=== Conditions For Starting Workers
 
-<b>no_enqueue_scale</b>:: When a job is enqueued workers will be scaled within the max/min requirements you set, if there is not a single worker running then the minimum number of workers will be scaled up (one worker will be scaled up if the minimum is zero). The worker itself handles the rest of the scaling. If you do not want scaling on enqueue and want to have the workers perform all the scaling then you can set this option to true, however if you do this you must be responsible for making sure that at least one worker is running on the queue. The default is false.
+A worker will be started if the queue length is greater than <b>max_queue</b> or if the time it takes a job to go through the queue is greater than <b>max_time</b>. Also a worker will only be started if the time since the last scaling is greater than <b>wait_time</b>. Workers will not be started if there are already the <b>max_workers</b> number of workers. By default resque-director allows you to have zero jobs running, when a job is enqueued then workers will be scaled within the max/min requirements you set. If there is not a single worker running, then the minimum number of workers will be scaled up (one worker will be scaled up if the minimum is zero).
+
+=== Conditions For Removing Workers
+
+A worker will be removed if the jobs in the queue fall below half of the <b>max_queue</b>, and if the time it takes for a job to be pulled off of a queue falls below half of the <b>max_time</b>. Workers will be scaled down to the minimum if there are no jobs on the queue. Workers will not be stopped if there are only <b>min_workers</b> number of workers. Also a worker will only be stopped if the time since the last scaling is greater than <b>wait_time</b>.
+
+=== Special Cases
+
+* If a max_worker is less than min_worker then the default for max_worker will be used (there will be no maximum). 
+* If a min_workers is set to anything less than 1 then it will be treated as 0.
+
+== Advanced Options
+
+The above options are plenty to handle the basic scaling of workers. The options below allow the user much more control over the way in which workers are scaled, the information tracked, and the times when a worker is scaled.
 
 === Start/Stop Options
 
-<b>start_override</b>:: This option takes a lambda closure that accepts the queue as an argument, the block you pass in will be responsible for starting a SINGLE worker allowing you to fully customize the starting of a worker. If your block returns false then that signifies that a worker was not started and the last scaled time will not be set. By default to start a worker the system command "QUEUE=queue_name rake resque:work &" is run, where queue_name is whatever queue the job is running.
+<b>start_override</b>:: You can set this option if you want to override the way to start a worker. This option takes a lambda closure that accepts the queue as an argument, the block you pass in will be responsible for starting a <em>SINGLE</em> worker allowing you to fully customize the starting of a worker. If your block returns false then that signifies that a worker was not started and the last scaled time will not be set. The default way a worker is started is with the system command: <tt>QUEUE=queue_name rake resque:work &</tt> where "queue_name" is the queue on which the job is running.
 
-<b>stop_override</b>:: This option takes a lambda closure that accepts the queue as an argument, the block you pass in will be responsible for stopping a SINGLE worker allowing you to fully customize the stopping of a worker. If your block returns false then that signifies that a worker was not stopped and the last scaled time will not be set. By default a to stop a worker the worker process on that host is sent a QUIT signal.
+<b>stop_override</b>:: You can set this option if you want to override the way to stop a worker. This option takes a lambda closure that accepts the queue as an argument, the block you pass in will be responsible for stopping a <em>SINGLE</em> worker allowing you to fully customize the stopping of a worker. If your block returns false then that signifies that a worker was not stopped and the last scaled time will not be set. The default way a worker is stopped is that a QUIT signal is sent to a worker process. 
 
 ===	Customized Starting/Stopping Workers Example
 
@@ -58,14 +69,6 @@ For Example:
     #rest of your Job class here
   end
 
-=== Conditions For Starting Workers
-
-A worker will be started if the queue length is greater than <b>max_queue</b> or if the time it takes a job to go through the queue is greater than <b>max_time</b>. Also a worker will only be started if the time since the last scaling is greater than <b>wait_time</b>. Workers will not be started if there are already the <b>max_workers</b> number of workers. When a job is enqueued and if <b>no_enqueue_scale</b> is not set or false, then workers will be scaled within the max/min requirements you set, if there is not a single worker running then the minimum number of workers will be scaled up (one worker will be scaled up if the minimum is zero).
-
-=== Conditions For Removing Workers
-
-A worker will be removed if the jobs in the queue fall below half of the <b>max_queue</b>, and if the time it takes for a job to be pulled off of a queue falls below half of the <b>max_time</b>. Workers will be scaled down to the minimum if there are no jobs on the queue.
-
 === Logging
 
 To add director logging you can pass a logger as an option. It will prepend all log messages with "DIRECTORS LOG:" and will log when scaling up or down a worker. It will also log when a worker wants to scale up or down but is unable to do so due to the fact that the maximum or minimum number of workers has been reached for that queue. 
@@ -88,10 +91,26 @@ To add director logging you can pass a logger as an option. It will prepend all
 
 <b>log_level</b>:: This sets the level to log at as a symbol. The default level is :debug.
 
-=== Special Cases
+=== Scale Options
 
-* If a max_worker is less than min_worker then the default for max_worker will be used (there will be no maximum). 
-* If a min_workers is set to anything less than 1 then it will be treated as 0.
+<b>no_enqueue_scale</b>:: When a job is enqueued workers will be scaled within the max/min requirements you set, if there is not a single worker running then the minimum number of workers will be scaled up (one worker will be scaled up if the minimum is zero). The worker itself handles the rest of the scaling. If you do not want scaling on enqueue and want to have the workers perform all the scaling then you can set this option to true, however if you do this you must be responsible for making sure that at least one worker is running on the queue. The default is false.
+
+=== Multiqueue Options
+
+<b>queue</b>:: The queue name(s) that you will be scaling and using to calculate the number of workers working. The default is the queue name of the Job.
+
+===	Multiqueue Example
+
+  class Job
+    direct :queue = [:test, :test2]
+    @queue = :test
+    
+    #rest of your Job class here
+  end
+
+=== Multiqueue
+
+resque-director will still look at the job's queue to determine when to scale. However if you set the <b>queue</b> option it will scale a worker working on that queue, or a worker working on multiple queues if you pass it an array of queues. 
 
 == Requirements
 

data/VERSION

@@ -1 +1 @@
-2.1.1
+2.2.0

data/lib/resque-director.rb

@@ -5,3 +5,7 @@ require 'resque/plugins/director/worker_tracker'
 require 'resque/plugins/director/config'
 require 'resque/plugins/director/scaler'
 require 'resque/plugins/director/push_pop'
+
+module Resque
+  include Resque::Plugins::Director::PushPop
+end

data/lib/resque/plugins/director.rb

@@ -3,21 +3,23 @@ module Resque
     module Director
       def direct(options={})
         Config.setup(options)
+        Config.queue = options[:queue]
       end
     
       def after_enqueue_scale_workers(*args)
-        Config.queue = @queue.to_s
+        set_queue
         return if Config.no_enqueue_scale
         Scaler.scale_within_requirements
       end
       
       def before_perform_direct_workers(*args)
+        set_queue
         Scaler.scale_within_requirements if Config.no_enqueue_scale
       end
     
       def after_pop_direct_workers(start_time=Time.now.utc)
         return unless scaling_config_set?
-        Config.queue = @queue.to_s
+        set_queue
         
         time_through_queue = Time.now.utc - start_time
         jobs_in_queue = Resque.size(@queue.to_s)
@@ -30,18 +32,22 @@ module Resque
       end
     
       def after_perform_direct_workers(*args)
-        Config.queue = @queue.to_s
+        set_queue
         jobs_in_queue = Resque.size(@queue.to_s)
         Scaler.scale_down_to_minimum if jobs_in_queue == 0
       end
     
       def on_failure_direct_workers(*args)
-        Config.queue = @queue.to_s
+        set_queue
         jobs_in_queue = Resque.size(@queue.to_s)
         Scaler.scale_down_to_minimum if jobs_in_queue == 0
       end
     
       private
+      
+      def set_queue
+        Config.queue ||= @queue.to_s
+      end
     
       def scaling_config_set?
         Config.max_time > 0 || Config.max_queue > 0

data/lib/resque/plugins/director/push_pop.rb

@@ -34,8 +34,4 @@ module Resque
       end
     end
   end
-end
-
-module Resque
-  include Resque::Plugins::Director::PushPop
 end

data/lib/resque/plugins/director/scaler.rb

@@ -14,7 +14,6 @@ module Resque
           def scale_down(number_of_workers=1)
             tracker = WorkerTracker.new
             number_of_workers = tracker.total_to_remove(number_of_workers)
-            
             scaling(number_of_workers) do
               stop(tracker, number_of_workers)
             end
@@ -28,7 +27,6 @@ module Resque
           
           def scale_within_requirements
             number_of_workers = WorkerTracker.new.total_for_requirements
-            
             if number_of_workers > 0
               scale_up(number_of_workers)
             elsif number_of_workers < 0
@@ -54,22 +52,28 @@ module Resque
           
           def start(number_of_workers)
             Config.log("starting #{number_of_workers} workers on queue:#{Config.queue}") if number_of_workers > 0
-            if Config.start_override
-              number_of_workers.times { Config.start_override.call(Config.queue) }
-            else
-              number_of_workers.times { system("QUEUE=#{Config.queue} rake resque:work &") }
-            end
+            return override(number_of_workers, Config.start_override) if Config.start_override
+            start_default(number_of_workers)
           end
           
           def stop(tracker, number_of_workers)
             Config.log("stopping #{number_of_workers} workers on queue:#{Config.queue}") if number_of_workers > 0
-            if Config.stop_override
-              number_of_workers.times {Config.stop_override.call(Config.queue) }
-            else
-              worker_pids = tracker.valid_worker_pids[0...number_of_workers]
-              worker_pids.each do |pid| 
-                Process.kill("QUIT", pid) rescue nil
-              end
+            return override(number_of_workers, Config.stop_override) if Config.stop_override
+            stop_default(tracker, number_of_workers)
+          end
+          
+          def override(number_of_workers, override_block)
+            number_of_workers.times {override_block.call(Config.queue) }
+          end
+          
+          def start_default(number_of_workers)
+            number_of_workers.times { system("QUEUE=#{[Config.queue].flatten.join(",")} rake resque:work &") }
+          end
+
+          def stop_default(tracker, number_of_workers)
+            worker_pids = tracker.valid_worker_pids[0...number_of_workers]
+            worker_pids.each do |pid| 
+              Process.kill("QUIT", pid) rescue nil
             end
           end
         end

data/lib/resque/plugins/director/worker_tracker.rb

@@ -57,7 +57,7 @@ module Resque
         
         def current_workers
           Resque.workers.select do |w|
-            w.queues.map(&:to_s) == [Config.queue.to_s] && !w.shutdown?
+            w.queues.map(&:to_s) == Config.queue.map(&:to_s) && !w.shutdown?
           end
         end
       end

data/resque-director.gemspec

@@ -5,11 +5,11 @@
 
 Gem::Specification.new do |s|
   s.name = %q{resque-director}
-  s.version = "2.1.1"
+  s.version = "2.2.0"
 
   s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
   s.authors = [%q{Nolan Frausto}]
-  s.date = %q{2011-09-02}
+  s.date = %q{2011-09-07}
   s.description = %q{resque plugin for automatically scaling workers based on the amount of time it takes a job to go through the queue and/or the length of the queue }
   s.email = %q{nrfrausto@gmail.com}
   s.extra_rdoc_files = [

data/spec/resque/plugins/director/scaler_spec.rb

@@ -30,6 +30,12 @@ describe Resque::Plugins::Director::Scaler do
       test_block.should_receive(:call).with("test")
       subject.scale_up
     end
+    
+    it "scales up a worker to work on multiple queues" do
+      Resque::Plugins::Director::Config.queue = [:test, :test2]
+      subject.should_receive(:system).with("QUEUE=test,test2 rake resque:work &")
+      subject.scale_up
+    end
   end
   
   describe "#scaling" do
@@ -43,6 +49,13 @@ describe Resque::Plugins::Director::Scaler do
       @times_scaled.should == 1
     end
     
+    it "should not scale workers if last time scaled is too soon for watching multiple queues" do
+      Resque::Plugins::Director::Config.setup(:wait_time => 60)
+      Resque::Plugins::Director::Config.queue = [:test,:test2]
+      2.times { subject.scaling { @times_scaled += 1 } }
+      @times_scaled.should == 1
+    end
+    
     it "should scale workers if wait time has passed" do
       Resque::Plugins::Director::Config.setup(:wait_time => 0)
       2.times { subject.scaling { @times_scaled += 1 } }
@@ -55,6 +68,12 @@ describe Resque::Plugins::Director::Scaler do
         @now = Time.now
         Time.stub(:now => @now)
       end
+      
+      it "should the last scaled for scaling multiple queue workers" do
+        Resque::Plugins::Director::Config.queue = [:test,:test2]
+        subject.scaling { true }
+        Resque.redis.get("last_scaled_testtest2").to_i.should == @now.utc.to_i
+      end
     
       it "should set the last scaled time" do
         subject.scaling { true }

data/spec/resque/plugins/director/worker_tracker_spec.rb

@@ -145,5 +145,18 @@ describe Resque::Plugins::Director::WorkerTracker do
       Resque.should_receive(:workers).and_return [shutdown_worker, @worker, @worker]
       subject.new.workers.should == [@worker, @worker]
     end
+    
+    it "does not set workers that have the queue included with others" do
+      other_worker = Resque::Worker.new(:other, :test)
+      Resque.should_receive(:workers).and_return [@worker, other_worker, @worker]
+      subject.new.workers.should == [@worker, @worker]
+    end
+    
+    it "finds workers working on multiple queues if specified" do
+      other_worker = Resque::Worker.new(:other, :test)
+      Resque::Plugins::Director::Config.queue = [:other,:test]
+      Resque.should_receive(:workers).and_return [@worker, other_worker]
+      subject.new.workers.should == [other_worker]
+    end
   end
 end

data/spec/resque/plugins/director_spec.rb

@@ -21,6 +21,12 @@ describe Resque::Plugins::Director do
       Resque::Plugins::Director::Config.queue.should == "test"
     end
     
+    it "should not set the queue if it was manually set" do
+      TestJob.direct :queue => "real_queue"
+      Resque.enqueue(TestJob)
+      Resque::Plugins::Director::Config.queue.should == "real_queue"
+    end
+    
     it "should not scale if the no_enqueue_scale option is set" do
       Resque::Worker.new(:test).register_worker
       Resque::Plugins::Director::Scaler.should_not_receive(:scale_within_requirements)
@@ -30,6 +36,17 @@ describe Resque::Plugins::Director do
   end
   
   describe "#after_perform_direct_workers" do
+    it "should set the queue" do
+      TestJob.after_perform_direct_workers
+      Resque::Plugins::Director::Config.queue.should == "test"
+    end
+    
+    it "should not set the queue if it is set" do
+      TestJob.direct :queue => "real_queue"
+      TestJob.after_perform_direct_workers
+      Resque::Plugins::Director::Config.queue.should == "real_queue"
+    end
+    
     it "should scale down to the minumum workers if there are no jobs in the queue" do
       Resque::Plugins::Director::Scaler.should_receive(:scale_down_to_minimum)
       TestJob.after_perform_direct_workers
@@ -43,6 +60,17 @@ describe Resque::Plugins::Director do
   end
   
   describe "#on_failure_direct_workers" do
+    it "should set the queue" do
+      TestJob.on_failure_direct_workers
+      Resque::Plugins::Director::Config.queue.should == "test"
+    end
+    
+    it "should not set the queue if it is set" do
+      TestJob.direct :queue => "real_queue"
+      TestJob.on_failure_direct_workers
+      Resque::Plugins::Director::Config.queue.should == "real_queue"
+    end
+    
     it "should scale down to the minumum workers if there are no jobs in the queue" do
       Resque::Plugins::Director::Scaler.should_receive(:scale_down_to_minimum)
       TestJob.on_failure_direct_workers
@@ -67,6 +95,12 @@ describe Resque::Plugins::Director do
         TestJob.after_pop_direct_workers(@start_time)
         Resque::Plugins::Director::Config.queue.should == "test"
       end
+      
+      it "should not set the queue if it is set" do
+        TestJob.direct :max_time => 20, :queue => "real_queue"
+        TestJob.after_pop_direct_workers(@start_time)
+        Resque::Plugins::Director::Config.queue.should == "real_queue"
+      end
     
       it "should not start workers if max_time is not set" do
         Resque::Plugins::Director::Scaler.should_not_receive(:scale_up)
@@ -183,6 +217,17 @@ describe Resque::Plugins::Director do
   end
   
   describe "#before_perform_direct_workers" do
+    it "should set the queue" do
+      TestJob.before_perform_direct_workers
+      Resque::Plugins::Director::Config.queue.should == "test"
+    end
+    
+    it "should not set the queue if it is set" do
+      TestJob.direct :queue => "real_queue"
+      TestJob.before_perform_direct_workers
+      Resque::Plugins::Director::Config.queue.should == "real_queue"
+    end
+    
     it "should scale within requirements if no_enqueue_scale is set" do
       TestJob.direct :no_enqueue_scale => true
       Resque::Plugins::Director::Scaler.should_receive(:scale_within_requirements)

metadata

@@ -1,13 +1,13 @@
 --- !ruby/object:Gem::Specification 
 name: resque-director
 version: !ruby/object:Gem::Version 
-  hash: 9
+  hash: 7
   prerelease: 
   segments: 
   - 2
-  - 1
-  - 1
-  version: 2.1.1
+  - 2
+  - 0
+  version: 2.2.0
 platform: ruby
 authors: 
 - Nolan Frausto
@@ -15,7 +15,7 @@ autorequire:
 bindir: bin
 cert_chain: []
 
-date: 2011-09-02 00:00:00 Z
+date: 2011-09-07 00:00:00 Z
 dependencies: 
 - !ruby/object:Gem::Dependency 
   requirement: &id001 !ruby/object:Gem::Requirement