resque-director 2.0.0 → 2.1.0
Sign up to get free protection for your applications and to get access to all the features.
- data/Gemfile +1 -1
- data/README.rdoc +36 -16
- data/Rakefile +2 -2
- data/VERSION +1 -1
- data/lib/resque/plugins/director/config.rb +10 -9
- data/lib/resque/plugins/director/{extra_hooks.rb → push_pop.rb} +3 -3
- data/lib/resque/plugins/director/scaler.rb +2 -3
- data/lib/resque/plugins/director/worker_tracker.rb +8 -3
- data/lib/resque/plugins/director.rb +7 -0
- data/lib/resque-director.rb +1 -1
- data/resque-director.gemspec +9 -10
- data/spec/resque/plugins/director/config_spec.rb +2 -2
- data/spec/resque/plugins/director/{extra_hooks_spec.rb → push_pop_spec.rb} +2 -2
- data/spec/resque/plugins/director/scaler_spec.rb +13 -11
- data/spec/resque/plugins/director/worker_tracker_spec.rb +12 -0
- data/spec/resque/plugins/director_spec.rb +21 -0
- metadata +11 -13
- data/.rspec +0 -1
data/Gemfile
CHANGED
data/README.rdoc
CHANGED
@@ -6,7 +6,7 @@ Resque Director is a plugin for the Resque queueing system (http://github.com/de
|
|
6
6
|
|
7
7
|
resque-director is mainly useful for when you are managing a large number of workers and don't want to waste resources keeping all of them waiting when they are not being used. Also useful in queues where the influx of jobs can change dramatically from time to time: enabling more workers during the times when the queue is filling up more quickly, and less in the opposite scenario. Different queues can be given different directions as well.
|
8
8
|
|
9
|
-
resque-director does not operate on workers that serve multiple queues, and does not include those workers in any scaling calculations.
|
9
|
+
resque-director does not currently operate on workers that serve multiple queues, and does not include those workers in any scaling calculations.
|
10
10
|
|
11
11
|
== Usage
|
12
12
|
|
@@ -27,59 +27,79 @@ For Example:
|
|
27
27
|
|
28
28
|
<b>min_workers</b>:: specifies the minimum number of workers running at any point in time. If there are no workers running or less than the minimum running it will start as many workers necessary to get it to the minimum. The default is 1.
|
29
29
|
|
30
|
-
<b>max_workers</b>:: specifies the maximum number of workers running at any point in time. It will never start more than the maximum number of workers. If anything less than or equal to zero is specified as the maximum it will be treated as if there is no maximum, and theoretically an infinite number of workers could be
|
30
|
+
<b>max_workers</b>:: specifies the maximum number of workers running at any point in time. It will never start more than the maximum number of workers. If anything less than or equal to zero is specified as the maximum it will be treated as if there is no maximum, and theoretically an infinite number of workers could be started. The default is 0.
|
31
31
|
|
32
|
-
<b>max_time</b>:: the maximum time in seconds that a job takes to get pulled off the queue, if a job takes longer than this time then a worker is
|
32
|
+
<b>max_time</b>:: the maximum time in seconds that a job takes to get pulled off the queue, if a job takes longer than this time then a worker is started. If anything less than or equal to zero is specified as the maximum time, this field will be ignored. The default is 0.
|
33
33
|
|
34
|
-
<b>max_queue</b>:: the maximum jobs that can build up in a queue, if more than this number of jobs build up then a worker is
|
34
|
+
<b>max_queue</b>:: the maximum jobs that can build up in a queue, if more than this number of jobs build up then a worker is started. If anything less than or equal to zero is specified as the maximum queue, this field will be ignored. The default is 0.
|
35
35
|
|
36
|
-
<b>wait_time</b>:: the time that it will wait after adding or removing a worker before being allowed to add or remove workers again. The default is 60 seconds.
|
36
|
+
<b>wait_time</b>:: the time in seconds that it will wait after adding or removing a worker before being allowed to add or remove workers again. The default is 60 seconds.
|
37
37
|
|
38
|
-
===
|
38
|
+
=== Scale Options
|
39
39
|
|
40
|
-
<b>
|
40
|
+
<b>no_enqueue_scale</b>:: When a job is enqueued workers will be scaled within the max/min requirements you set, if there is not a single worker running then the minimum number of workers will be scaled up (one worker will be scaled up if the minimum is zero). The worker itself handles the rest of the scaling. If you do not want scaling on enqueue and want to have the workers perform all the scaling then you can set this option to true, however if you do this you must be responsible for making sure that at least one worker is running on the queue. The default is false.
|
41
41
|
|
42
|
-
|
42
|
+
=== Start/Stop Options
|
43
43
|
|
44
|
-
|
44
|
+
<b>start_override</b>:: This option takes a lambda closure that accepts the queue as an argument, the block you pass in will be responsible for starting a SINGLE worker allowing you to fully customize the starting of a worker. By default to start a worker the system command "QUEUE=queue_name rake resque:work &" is run, where queue_name is whatever queue the job is running.
|
45
|
+
|
46
|
+
<b>stop_override</b>:: This option takes a lambda closure that accepts the queue as an argument, the block you pass in will be responsible for stopping a SINGLE worker allowing you to fully customize the stopping of a worker. By default a to stop a worker the worker process on that host is sent a QUIT signal.
|
47
|
+
|
48
|
+
=== Customized Starting/Stopping Workers Example
|
45
49
|
|
46
50
|
class Job
|
47
51
|
extend Resque::Plugins::Director
|
48
52
|
|
49
|
-
|
50
|
-
|
53
|
+
start_block = lambda{|queue| ... }
|
54
|
+
stop_block = lambda{|queue| ... }
|
51
55
|
direct :start_override => start_block, :stop_override => stop_block
|
52
56
|
@queue = :test
|
53
57
|
|
54
58
|
#rest of your Job class here
|
55
59
|
end
|
56
60
|
|
61
|
+
=== Conditions For Starting Workers
|
62
|
+
|
63
|
+
A worker will be started if the queue length is greater than <b>max_queue</b> or if the time it takes a job to go through the queue is greater than <b>max_time</b>. Also a worker will only be started if the time since the last scaling is greater than <b>wait_time</b>. Workers will not be started if there are already the <b>max_workers</b> number of workers. When a job is enqueued and if <b>no_enqueue_scale</b> is not set or false, then workers will be scaled within the max/min requirements you set, if there is not a single worker running then the minimum number of workers will be scaled up (one worker will be scaled up if the minimum is zero).
|
64
|
+
|
57
65
|
=== Conditions For Removing Workers
|
58
66
|
|
59
|
-
A worker will be removed if the jobs in the queue fall below half of the <b>max_queue</b>,
|
67
|
+
A worker will be removed if the jobs in the queue fall below half of the <b>max_queue</b>, and if the time it takes for a job to be pulled off of a queue falls below half of the <b>max_time</b>. Workers will be scaled down to the minimum if there are no jobs on the queue.
|
60
68
|
|
61
69
|
=== Logging
|
62
70
|
|
63
71
|
To add director logging you can pass a logger as an option. It will prepend all log messages with "DIRECTORS LOG:" and will log when scaling up or down a worker. It will also log when a worker wants to scale up or down but is unable to do so due to the fact that the maximum or minimum number of workers has been reached for that queue.
|
64
72
|
|
73
|
+
=== Logging Example
|
74
|
+
|
75
|
+
class Job
|
76
|
+
logger = Logger.new('logfile.log')
|
77
|
+
extend Resque::Plugins::Director
|
78
|
+
|
79
|
+
direct :logger => logger, :log_level => :info
|
80
|
+
@queue = :test
|
81
|
+
|
82
|
+
#rest of your Job class here
|
83
|
+
end
|
84
|
+
|
65
85
|
=== Logger Options
|
66
86
|
|
67
87
|
<b>logger</b>:: This will set the logger that will be used. It takes a Logger from the Ruby Standard Library. If this is not set the director will not write to any logs.
|
68
88
|
|
69
|
-
<b>log_level</b>:: This sets the level to log at as a symbol. The default level is :
|
89
|
+
<b>log_level</b>:: This sets the level to log at as a symbol. The default level is :debug.
|
70
90
|
|
71
91
|
=== Special Cases
|
72
92
|
|
73
93
|
* If a max_worker is less than min_worker then the default for max_worker will be used (there will be no maximum).
|
74
94
|
* If a min_workers is set to anything less than 1 then it will be treated as 0.
|
75
95
|
|
76
|
-
|
96
|
+
== Requirements
|
77
97
|
|
78
|
-
* resque-director requires resque ~> 1.
|
98
|
+
* resque-director requires resque ~> 1.10.
|
79
99
|
* resque-director may be incompatible with gems that modify resque's push/pop functionality.
|
80
100
|
|
81
101
|
|
82
|
-
== Contributing to resque-
|
102
|
+
== Contributing to resque-director
|
83
103
|
|
84
104
|
* Check out the latest master to make sure the feature hasn't been implemented or the bug hasn't been fixed yet
|
85
105
|
* Check out the issue tracker to make sure someone already hasn't requested it and/or contributed it
|
data/Rakefile
CHANGED
@@ -20,8 +20,8 @@ Jeweler::Tasks.new do |gem|
|
|
20
20
|
gem.name = "resque-director"
|
21
21
|
gem.homepage = "http://github.com/frausto/resque-director"
|
22
22
|
gem.license = "MIT"
|
23
|
-
gem.summary = %Q{resque plugin for
|
24
|
-
gem.description = %Q{resque plugin for
|
23
|
+
gem.summary = %Q{A resque plugin for automatically scaling workers}
|
24
|
+
gem.description = %Q{resque plugin for automatically scaling workers based on the amount of time it takes a job to go through the queue and/or the length of the queue }
|
25
25
|
gem.email = "nrfrausto@gmail.com"
|
26
26
|
gem.authors = ["Nolan Frausto"]
|
27
27
|
# dependencies defined in Gemfile
|
data/VERSION
CHANGED
@@ -1 +1 @@
|
|
1
|
-
2.
|
1
|
+
2.1.0
|
@@ -7,15 +7,16 @@ module Resque
|
|
7
7
|
attr_accessor :queue
|
8
8
|
|
9
9
|
DEFAULT_OPTIONS = {
|
10
|
-
:min_workers
|
11
|
-
:max_workers
|
12
|
-
:max_time
|
13
|
-
:max_queue
|
14
|
-
:wait_time
|
15
|
-
:start_override
|
16
|
-
:stop_override
|
17
|
-
:logger
|
18
|
-
:log_level
|
10
|
+
:min_workers => 1,
|
11
|
+
:max_workers => 0,
|
12
|
+
:max_time => 0,
|
13
|
+
:max_queue => 0,
|
14
|
+
:wait_time => 60,
|
15
|
+
:start_override => nil,
|
16
|
+
:stop_override => nil,
|
17
|
+
:logger => nil,
|
18
|
+
:log_level => :debug,
|
19
|
+
:no_enqueue_scale => false
|
19
20
|
}
|
20
21
|
|
21
22
|
def reset!
|
@@ -1,9 +1,9 @@
|
|
1
1
|
module Resque
|
2
2
|
module Plugins
|
3
3
|
module Director
|
4
|
-
module
|
4
|
+
module PushPop
|
5
5
|
|
6
|
-
def self.included(base)
|
6
|
+
def self.included(base)
|
7
7
|
base.class_eval do
|
8
8
|
alias_method :original_pop, :pop
|
9
9
|
alias_method :original_push, :push
|
@@ -37,5 +37,5 @@ module Resque
|
|
37
37
|
end
|
38
38
|
|
39
39
|
module Resque
|
40
|
-
include Resque::Plugins::Director::
|
40
|
+
include Resque::Plugins::Director::PushPop
|
41
41
|
end
|
@@ -22,7 +22,7 @@ module Resque
|
|
22
22
|
|
23
23
|
def scale_down_to_minimum
|
24
24
|
tracker = WorkerTracker.new
|
25
|
-
number_of_workers = tracker.total_to_go_to_minimum
|
25
|
+
number_of_workers = tracker.total_to_go_to_minimum
|
26
26
|
stop(tracker, number_of_workers)
|
27
27
|
end
|
28
28
|
|
@@ -65,8 +65,7 @@ module Resque
|
|
65
65
|
if Config.stop_override
|
66
66
|
number_of_workers.times {Config.stop_override.call(Config.queue) }
|
67
67
|
else
|
68
|
-
|
69
|
-
worker_pids = valid_workers[0...number_of_workers].map(&:pid)
|
68
|
+
worker_pids = tracker.valid_worker_pids[0...number_of_workers]
|
70
69
|
worker_pids.each do |pid|
|
71
70
|
Process.kill("QUIT", pid) rescue nil
|
72
71
|
end
|
@@ -38,21 +38,26 @@ module Resque
|
|
38
38
|
number_to_stop > scale_limit ? scale_limit : number_to_stop
|
39
39
|
end
|
40
40
|
|
41
|
+
def valid_worker_pids
|
42
|
+
valid_workers = @workers.select{|w| w.hostname == `hostname`.chomp}
|
43
|
+
valid_workers.map{|worker| worker.to_s.split(":")[1].to_i }
|
44
|
+
end
|
45
|
+
|
41
46
|
private
|
42
47
|
|
43
48
|
def workers_to_start
|
44
49
|
min_workers = Config.min_workers <= 0 ? 1 : Config.min_workers
|
45
|
-
|
50
|
+
min_workers - @number_working
|
46
51
|
end
|
47
52
|
|
48
53
|
def workers_to_stop
|
49
54
|
return 0 if Config.max_workers <= 0
|
50
|
-
|
55
|
+
Config.max_workers - @number_working
|
51
56
|
end
|
52
57
|
|
53
58
|
def current_workers
|
54
59
|
Resque.workers.select do |w|
|
55
|
-
w.queues == [Config.queue] && !w.shutdown?
|
60
|
+
w.queues.map(&:to_s) == [Config.queue.to_s] && !w.shutdown?
|
56
61
|
end
|
57
62
|
end
|
58
63
|
end
|
@@ -7,8 +7,13 @@ module Resque
|
|
7
7
|
|
8
8
|
def after_enqueue_scale_workers(*args)
|
9
9
|
Config.queue = @queue.to_s
|
10
|
+
return if Config.no_enqueue_scale
|
10
11
|
Scaler.scale_within_requirements
|
11
12
|
end
|
13
|
+
|
14
|
+
def before_perform_direct_workers(*args)
|
15
|
+
Scaler.scale_within_requirements if Config.no_enqueue_scale
|
16
|
+
end
|
12
17
|
|
13
18
|
def after_pop_direct_workers(start_time=Time.now.utc)
|
14
19
|
return unless scaling_config_set?
|
@@ -25,11 +30,13 @@ module Resque
|
|
25
30
|
end
|
26
31
|
|
27
32
|
def after_perform_direct_workers(*args)
|
33
|
+
Config.queue = @queue.to_s
|
28
34
|
jobs_in_queue = Resque.size(@queue.to_s)
|
29
35
|
Scaler.scale_down_to_minimum if jobs_in_queue == 0
|
30
36
|
end
|
31
37
|
|
32
38
|
def on_failure_direct_workers(*args)
|
39
|
+
Config.queue = @queue.to_s
|
33
40
|
jobs_in_queue = Resque.size(@queue.to_s)
|
34
41
|
Scaler.scale_down_to_minimum if jobs_in_queue == 0
|
35
42
|
end
|
data/lib/resque-director.rb
CHANGED
data/resque-director.gemspec
CHANGED
@@ -5,12 +5,12 @@
|
|
5
5
|
|
6
6
|
Gem::Specification.new do |s|
|
7
7
|
s.name = %q{resque-director}
|
8
|
-
s.version = "2.
|
8
|
+
s.version = "2.1.0"
|
9
9
|
|
10
10
|
s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
|
11
11
|
s.authors = [%q{Nolan Frausto}]
|
12
|
-
s.date = %q{2011-
|
13
|
-
s.description = %q{resque plugin for
|
12
|
+
s.date = %q{2011-09-01}
|
13
|
+
s.description = %q{resque plugin for automatically scaling workers based on the amount of time it takes a job to go through the queue and/or the length of the queue }
|
14
14
|
s.email = %q{nrfrausto@gmail.com}
|
15
15
|
s.extra_rdoc_files = [
|
16
16
|
"LICENSE.txt",
|
@@ -18,7 +18,6 @@ Gem::Specification.new do |s|
|
|
18
18
|
]
|
19
19
|
s.files = [
|
20
20
|
".document",
|
21
|
-
".rspec",
|
22
21
|
"Gemfile",
|
23
22
|
"LICENSE.txt",
|
24
23
|
"README.rdoc",
|
@@ -27,13 +26,13 @@ Gem::Specification.new do |s|
|
|
27
26
|
"lib/resque-director.rb",
|
28
27
|
"lib/resque/plugins/director.rb",
|
29
28
|
"lib/resque/plugins/director/config.rb",
|
30
|
-
"lib/resque/plugins/director/
|
29
|
+
"lib/resque/plugins/director/push_pop.rb",
|
31
30
|
"lib/resque/plugins/director/scaler.rb",
|
32
31
|
"lib/resque/plugins/director/worker_tracker.rb",
|
33
32
|
"resque-director.gemspec",
|
34
33
|
"spec/redis-test.conf",
|
35
34
|
"spec/resque/plugins/director/config_spec.rb",
|
36
|
-
"spec/resque/plugins/director/
|
35
|
+
"spec/resque/plugins/director/push_pop_spec.rb",
|
37
36
|
"spec/resque/plugins/director/scaler_spec.rb",
|
38
37
|
"spec/resque/plugins/director/worker_tracker_spec.rb",
|
39
38
|
"spec/resque/plugins/director_spec.rb",
|
@@ -44,26 +43,26 @@ Gem::Specification.new do |s|
|
|
44
43
|
s.licenses = [%q{MIT}]
|
45
44
|
s.require_paths = [%q{lib}]
|
46
45
|
s.rubygems_version = %q{1.8.8}
|
47
|
-
s.summary = %q{resque plugin for
|
46
|
+
s.summary = %q{A resque plugin for automatically scaling workers}
|
48
47
|
|
49
48
|
if s.respond_to? :specification_version then
|
50
49
|
s.specification_version = 3
|
51
50
|
|
52
51
|
if Gem::Version.new(Gem::VERSION) >= Gem::Version.new('1.2.0') then
|
53
|
-
s.add_runtime_dependency(%q<resque>, ["~> 1.
|
52
|
+
s.add_runtime_dependency(%q<resque>, ["~> 1.10"])
|
54
53
|
s.add_development_dependency(%q<rspec>, ["~> 2.3.0"])
|
55
54
|
s.add_development_dependency(%q<bundler>, ["~> 1.0.0"])
|
56
55
|
s.add_development_dependency(%q<jeweler>, ["~> 1.6.4"])
|
57
56
|
s.add_development_dependency(%q<rcov>, [">= 0"])
|
58
57
|
else
|
59
|
-
s.add_dependency(%q<resque>, ["~> 1.
|
58
|
+
s.add_dependency(%q<resque>, ["~> 1.10"])
|
60
59
|
s.add_dependency(%q<rspec>, ["~> 2.3.0"])
|
61
60
|
s.add_dependency(%q<bundler>, ["~> 1.0.0"])
|
62
61
|
s.add_dependency(%q<jeweler>, ["~> 1.6.4"])
|
63
62
|
s.add_dependency(%q<rcov>, [">= 0"])
|
64
63
|
end
|
65
64
|
else
|
66
|
-
s.add_dependency(%q<resque>, ["~> 1.
|
65
|
+
s.add_dependency(%q<resque>, ["~> 1.10"])
|
67
66
|
s.add_dependency(%q<rspec>, ["~> 2.3.0"])
|
68
67
|
s.add_dependency(%q<bundler>, ["~> 1.0.0"])
|
69
68
|
s.add_dependency(%q<jeweler>, ["~> 1.6.4"])
|
@@ -43,10 +43,10 @@ describe Resque::Plugins::Director::Config do
|
|
43
43
|
subject.log("test message")
|
44
44
|
end
|
45
45
|
|
46
|
-
it "defaults log level to
|
46
|
+
it "defaults log level to debug" do
|
47
47
|
log = mock('Logger')
|
48
48
|
subject.setup(:logger => log)
|
49
|
-
log.should_receive(:
|
49
|
+
log.should_receive(:debug).with("DIRECTORS LOG: test message")
|
50
50
|
subject.log("test message")
|
51
51
|
end
|
52
52
|
end
|
@@ -1,7 +1,7 @@
|
|
1
1
|
require 'spec_helper'
|
2
2
|
|
3
|
-
describe Resque::Plugins::Director::
|
4
|
-
subject { Resque::Plugins::Director::
|
3
|
+
describe Resque::Plugins::Director::PushPop do
|
4
|
+
subject { Resque::Plugins::Director::PushPop }
|
5
5
|
|
6
6
|
before do
|
7
7
|
Resque::Plugins::Director::Config.queue = "test"
|
@@ -92,7 +92,8 @@ describe Resque::Plugins::Director::Scaler do
|
|
92
92
|
|
93
93
|
it "should scale down multiple workers" do
|
94
94
|
Resque.should_receive(:workers).and_return [@worker, @worker, @worker]
|
95
|
-
|
95
|
+
pid = @worker.to_s.split(":")[1].to_i
|
96
|
+
Process.should_receive(:kill).with("QUIT", pid)
|
96
97
|
subject.scale_down(2)
|
97
98
|
end
|
98
99
|
|
@@ -112,12 +113,12 @@ describe Resque::Plugins::Director::Scaler do
|
|
112
113
|
|
113
114
|
it "should not scale down workers on different queues" do
|
114
115
|
worker2 = Resque::Worker.new(:not_test)
|
115
|
-
@worker.stub(:
|
116
|
-
worker2.stub(:
|
116
|
+
@worker.stub(:to_s => "host:1:test")
|
117
|
+
worker2.stub(:to_s => "host:2:test")
|
117
118
|
Resque.should_receive(:workers).and_return [worker2, @worker, @worker, worker2]
|
118
119
|
|
119
|
-
Process.should_not_receive(:kill).with("QUIT",
|
120
|
-
Process.should_receive(:kill).with("QUIT",
|
120
|
+
Process.should_not_receive(:kill).with("QUIT", 2)
|
121
|
+
Process.should_receive(:kill).with("QUIT", 1)
|
121
122
|
subject.scale_down
|
122
123
|
end
|
123
124
|
end
|
@@ -125,6 +126,7 @@ describe Resque::Plugins::Director::Scaler do
|
|
125
126
|
describe "#stop" do
|
126
127
|
before do
|
127
128
|
@worker = Resque::Worker.new(:test)
|
129
|
+
@pid = @worker.to_s.split(":")[1].to_i
|
128
130
|
Resque::Plugins::Director::Config.setup :min_workers => 0
|
129
131
|
end
|
130
132
|
|
@@ -132,7 +134,7 @@ describe Resque::Plugins::Director::Scaler do
|
|
132
134
|
Resque.should_receive(:workers).and_return [@worker]
|
133
135
|
tracker = Resque::Plugins::Director::WorkerTracker.new
|
134
136
|
|
135
|
-
Process.should_receive(:kill).with("QUIT", @
|
137
|
+
Process.should_receive(:kill).with("QUIT", @pid)
|
136
138
|
subject.send(:stop, tracker, 1)
|
137
139
|
end
|
138
140
|
|
@@ -153,7 +155,7 @@ describe Resque::Plugins::Director::Scaler do
|
|
153
155
|
Resque.should_receive(:workers).and_return [@worker]
|
154
156
|
tracker = Resque::Plugins::Director::WorkerTracker.new
|
155
157
|
|
156
|
-
Process.should_not_receive(:kill).with("QUIT", @
|
158
|
+
Process.should_not_receive(:kill).with("QUIT", @pid)
|
157
159
|
subject.send(:stop, tracker, 1)
|
158
160
|
end
|
159
161
|
|
@@ -162,17 +164,17 @@ describe Resque::Plugins::Director::Scaler do
|
|
162
164
|
Resque.should_receive(:workers).and_return [@worker]
|
163
165
|
tracker = Resque::Plugins::Director::WorkerTracker.new
|
164
166
|
|
165
|
-
Process.should_not_receive(:kill).with("QUIT", @
|
167
|
+
Process.should_not_receive(:kill).with("QUIT", @pid)
|
166
168
|
subject.send(:stop, tracker, 1)
|
167
169
|
end
|
168
170
|
|
169
171
|
it "stops workers on the same host if possible" do
|
170
|
-
@worker.stub!(:hostname => "different_machine")
|
171
172
|
worker2 = Resque::Worker.new(:test)
|
172
|
-
|
173
|
+
worker2.stub!(:hostname => "different_machine")
|
174
|
+
Resque.should_receive(:workers).and_return [worker2, @worker]
|
173
175
|
tracker = Resque::Plugins::Director::WorkerTracker.new
|
174
176
|
|
175
|
-
Process.should_receive(:kill).with("QUIT",
|
177
|
+
Process.should_receive(:kill).with("QUIT", @pid)
|
176
178
|
subject.send(:stop, tracker, 1)
|
177
179
|
end
|
178
180
|
|
@@ -115,6 +115,18 @@ describe Resque::Plugins::Director::WorkerTracker do
|
|
115
115
|
end
|
116
116
|
end
|
117
117
|
|
118
|
+
describe "#valid_worker_pids" do
|
119
|
+
it "should return only pids of workers on the same host" do
|
120
|
+
@worker.stub!(:hostname => "different_machine")
|
121
|
+
worker2 = Resque::Worker.new(:test)
|
122
|
+
pid = worker2.to_s.split(":")[1].to_i
|
123
|
+
Resque.should_receive(:workers).and_return [@worker, worker2]
|
124
|
+
tracker = Resque::Plugins::Director::WorkerTracker.new
|
125
|
+
|
126
|
+
tracker.valid_worker_pids.should == [pid]
|
127
|
+
end
|
128
|
+
end
|
129
|
+
|
118
130
|
describe "#initialize" do
|
119
131
|
it "sets the workers from the queue" do
|
120
132
|
Resque.should_receive(:workers).and_return [@worker, @worker, @worker]
|
@@ -20,6 +20,13 @@ describe Resque::Plugins::Director do
|
|
20
20
|
Resque.enqueue(TestJob)
|
21
21
|
Resque::Plugins::Director::Config.queue.should == "test"
|
22
22
|
end
|
23
|
+
|
24
|
+
it "should not scale if the no_enqueue_scale option is set" do
|
25
|
+
Resque::Worker.new(:test).register_worker
|
26
|
+
Resque::Plugins::Director::Scaler.should_not_receive(:scale_within_requirements)
|
27
|
+
TestJob.direct :no_enqueue_scale => true
|
28
|
+
Resque.enqueue(TestJob)
|
29
|
+
end
|
23
30
|
end
|
24
31
|
|
25
32
|
describe "#after_perform_direct_workers" do
|
@@ -174,4 +181,18 @@ describe Resque::Plugins::Director do
|
|
174
181
|
end
|
175
182
|
end
|
176
183
|
end
|
184
|
+
|
185
|
+
describe "#before_perform_direct_workers" do
|
186
|
+
it "should scale within requirements if no_enqueue_scale is set" do
|
187
|
+
TestJob.direct :no_enqueue_scale => true
|
188
|
+
Resque::Plugins::Director::Scaler.should_receive(:scale_within_requirements)
|
189
|
+
TestJob.before_perform_direct_workers
|
190
|
+
end
|
191
|
+
|
192
|
+
it "should not scale within requirements if no_enqueue_scale is not set" do
|
193
|
+
TestJob.direct :no_enqueue_scale => false
|
194
|
+
Resque::Plugins::Director::Scaler.should_not_receive(:scale_within_requirements)
|
195
|
+
TestJob.before_perform_direct_workers
|
196
|
+
end
|
197
|
+
end
|
177
198
|
end
|
metadata
CHANGED
@@ -1,13 +1,13 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: resque-director
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
hash:
|
4
|
+
hash: 11
|
5
5
|
prerelease:
|
6
6
|
segments:
|
7
7
|
- 2
|
8
|
+
- 1
|
8
9
|
- 0
|
9
|
-
|
10
|
-
version: 2.0.0
|
10
|
+
version: 2.1.0
|
11
11
|
platform: ruby
|
12
12
|
authors:
|
13
13
|
- Nolan Frausto
|
@@ -15,7 +15,7 @@ autorequire:
|
|
15
15
|
bindir: bin
|
16
16
|
cert_chain: []
|
17
17
|
|
18
|
-
date: 2011-
|
18
|
+
date: 2011-09-01 00:00:00 Z
|
19
19
|
dependencies:
|
20
20
|
- !ruby/object:Gem::Dependency
|
21
21
|
requirement: &id001 !ruby/object:Gem::Requirement
|
@@ -23,12 +23,11 @@ dependencies:
|
|
23
23
|
requirements:
|
24
24
|
- - ~>
|
25
25
|
- !ruby/object:Gem::Version
|
26
|
-
hash:
|
26
|
+
hash: 27
|
27
27
|
segments:
|
28
28
|
- 1
|
29
|
-
-
|
30
|
-
|
31
|
-
version: 1.14.0
|
29
|
+
- 10
|
30
|
+
version: "1.10"
|
32
31
|
version_requirements: *id001
|
33
32
|
name: resque
|
34
33
|
prerelease: false
|
@@ -95,7 +94,7 @@ dependencies:
|
|
95
94
|
name: rcov
|
96
95
|
prerelease: false
|
97
96
|
type: :development
|
98
|
-
description: resque plugin for
|
97
|
+
description: "resque plugin for automatically scaling workers based on the amount of time it takes a job to go through the queue and/or the length of the queue "
|
99
98
|
email: nrfrausto@gmail.com
|
100
99
|
executables: []
|
101
100
|
|
@@ -106,7 +105,6 @@ extra_rdoc_files:
|
|
106
105
|
- README.rdoc
|
107
106
|
files:
|
108
107
|
- .document
|
109
|
-
- .rspec
|
110
108
|
- Gemfile
|
111
109
|
- LICENSE.txt
|
112
110
|
- README.rdoc
|
@@ -115,13 +113,13 @@ files:
|
|
115
113
|
- lib/resque-director.rb
|
116
114
|
- lib/resque/plugins/director.rb
|
117
115
|
- lib/resque/plugins/director/config.rb
|
118
|
-
- lib/resque/plugins/director/
|
116
|
+
- lib/resque/plugins/director/push_pop.rb
|
119
117
|
- lib/resque/plugins/director/scaler.rb
|
120
118
|
- lib/resque/plugins/director/worker_tracker.rb
|
121
119
|
- resque-director.gemspec
|
122
120
|
- spec/redis-test.conf
|
123
121
|
- spec/resque/plugins/director/config_spec.rb
|
124
|
-
- spec/resque/plugins/director/
|
122
|
+
- spec/resque/plugins/director/push_pop_spec.rb
|
125
123
|
- spec/resque/plugins/director/scaler_spec.rb
|
126
124
|
- spec/resque/plugins/director/worker_tracker_spec.rb
|
127
125
|
- spec/resque/plugins/director_spec.rb
|
@@ -159,6 +157,6 @@ rubyforge_project:
|
|
159
157
|
rubygems_version: 1.8.8
|
160
158
|
signing_key:
|
161
159
|
specification_version: 3
|
162
|
-
summary: resque plugin for
|
160
|
+
summary: A resque plugin for automatically scaling workers
|
163
161
|
test_files: []
|
164
162
|
|
data/.rspec
DELETED
@@ -1 +0,0 @@
|
|
1
|
-
--color
|