resque_stuck_queue 0.0.11 → 0.0.13

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
- SHA512:
3
- metadata.gz: df4e25ba257df91726f7c92cfe1a92f1a04b2afecb0106a4a80d0da12b494c67672fd5c40a981c813b4e18ede7d18b5e2aba3c8a445692e910acc17b9adc360e
4
- data.tar.gz: 22f4307be4f19c7e3b138c4d18013ac8fefc3a7a4d53a4071c6a6f8c1f32ec9dce2f187db9476e12d4f871a332ba6d9842fd6eb6d2ac6bf576dad79522eae783
5
2
  SHA1:
6
- metadata.gz: ff579897e68c118a67fe037962f318c2a265e851
7
- data.tar.gz: a82bad8bbf4ce498aac49d8b1f54c42047ee6b15
3
+ metadata.gz: 5d18f8d4224a9e8a9d30495ae0c7b6f3f0901f63
4
+ data.tar.gz: fd796adf7d09bb92e76c4806df3f725106d04d7a
5
+ SHA512:
6
+ metadata.gz: 476f6beaff11089bcd25c08428bdf6e7355af941ca5500c7e941c70addc5ed542e61a2f9d7d5df09970270a7401a880a0e3e58939139f00d3c5879222bc3d4f9
7
+ data.tar.gz: 6325fd7e73422455a97ea5479c82d36ef131ad5ba05a982db92d106bf088cd4804dd1126391c377bba7c8a538b56059164a303596123ab3f742c3c719ea71b87
data/README.md CHANGED
@@ -20,42 +20,41 @@ It will trigger a pre-defined proc (see below) if the last time the hearbeat job
20
20
 
21
21
  ## Usage
22
22
 
23
- Configure it first:
23
+ Configure it first. Optional settings are below. You'll most likely at the least want to tune `:handler`,`:heartbeat` and `:trigger_timeout` settings.
24
24
 
25
25
  <pre>
26
- # how often to push that 'heartbeat' job to refresh the latest time it worked.
27
- Resque::StuckQueue.config[:heartbeat] = 5.minutes
26
+ handler:
27
+ set to what gets triggered when resque-stuck-queue will detect the latest heartbeat is older than the trigger_timeout time setting.
28
+ Example:
29
+ Resque::StuckQueue.config[:handler] = proc { |queue_name, lagtime| send_email('queue #{queue_name} isnt working, aaah the daemons') }
28
30
 
29
- # since there is an realistic and acceptable lag for job queues, set this to how much you're
30
- # willing to accept between the current time and when the last hearbeat job went through.
31
- #
32
- # take the heartbeat into consideration when setting this (it will fire 10 hours + 5 minutes with above heartbeat).
33
- Resque::StuckQueue.config[:trigger_timeout] = 10.hours
31
+ heartbeat:
32
+ set to how often to push that 'heartbeat' job to refresh the latest time it worked.
33
+ Example:
34
+ Resque::StuckQueue.config[:heartbeat] = 5.minutes
34
35
 
35
- # what gets triggered when resque-stuck-queue will detect the latest heartbeat is older than the trigger_timeout time set above.
36
- #
37
- # triggering will update the key, so you'll have to wait the trigger_timeout again
38
- # in order for it to trigger again even if workers are still stale.
39
- Resque::StuckQueue.config[:handler] = proc { |queue_name| send_email("queue #{queue_name} isnt working, aaah the daemons") }
36
+ trigger_timeout:
37
+ set to how much of a resque work lag you are willing to accept before being notified. note: take the :heartbeat setting into account when setting this timeout.
38
+ Example:
39
+ Resque::StuckQueue.config[:trigger_timeout] = 55.minutes
40
40
 
41
- # optional, in case you want to set your own name for the key that will be used as the last good hearbeat time
42
- # note this will be namespaced under the specific queue it's monitoring, for eg "app:name-the-refresh-key-as-you-please"
43
- Resque::StuckQueue.config[:global_key] = "name-the-refresh-key-as-you-please"
41
+ redis:
42
+ set the Redis instance StuckQueue will use
44
43
 
45
- # optional, monitor specific queues you want to send a heartbeat/monitor
46
- Resque::StuckQueue.config[:queues] = [:app, :high, :my_custom_queue_name]
44
+ heartbeat_key:
45
+ optional, name of keys to keep track of the last good resque heartbeat time
47
46
 
48
- # optional, if you want the resque-stuck-queue threads to explicitly raise, default is false
49
- Resque::StuckQueue.config[:abort_on_exception] = true
47
+ logger:
48
+ optional, pass a Logger. Default a ruby logger will be instantiated. Needs to respond to that interface.
50
49
 
51
- # optional, pass a logger. Default a ruby logger will be instantiated. Needs to respond to that interface.
52
- Resque::StuckQueue.config[:logger] = Logger.new($stdout)
50
+ queues:
51
+ optional, monitor specific queues you want to send a heartbeat/monitor to. default is :app
53
52
 
54
- # optional, pass a redis.
55
- Resque::StuckQueue.config[:redis] = Redis.new
53
+ abort_on_exception:
54
+ optional, if you want the resque-stuck-queue threads to explicitly raise, default is false
56
55
 
57
- # optional, your own custom refresh job. below has an example.
58
- Resque::StuckQueue.config[:refresh_job] = proc { your_own_enqueue_refreshing_global_key }
56
+ refresh_job:
57
+ optional, your own custom refreshing job. if you are using something other than resque
59
58
  </pre>
60
59
 
61
60
  Then start it:
@@ -110,7 +109,7 @@ $ bundle exec rake --trace resque:stuck_queue
110
109
 
111
110
  ## Sidekiq/Other redis-based job queues
112
111
 
113
- If you have trouble with other queues you can use this lib by setting your own custom refresh job (aka, the job that refreshes your queue specific global_key). The one thing you need to take care of is ensure whatever and however you enque your own custom job, it sets the global_key to Time.now:
112
+ If you have trouble with other queues you can use this lib by setting your own custom refresh job (aka, the job that refreshes your queue specific heartbeat_key). The one thing you need to take care of is ensure whatever and however you enque your own custom job, it sets the heartbeat_key to Time.now:
114
113
 
115
114
  <pre>
116
115
 
@@ -118,7 +117,7 @@ class CustomJob
118
117
  include Sidekiq::Worker
119
118
  def perform
120
119
  # ensure you're setting the key in the redis the job queue is using
121
- $redis.set(Resque::StuckQueue.global_key_for(queue_name), Time.now.to_i)
120
+ $redis.set(Resque::StuckQueue.heartbeat_key_for(queue_name), Time.now.to_i)
122
121
  end
123
122
  end
124
123
 
data/THOUGHTS CHANGED
@@ -14,3 +14,10 @@ add a trap{} to force_stop. ok for overwriting process's trap handlers? use conf
14
14
 
15
15
  fix skeleton recipe https://github.com/shaiguitar/resque_stuck_queue/blame/master/README.md#L103
16
16
  raise appname, => :environment, log path
17
+
18
+ set a redis key instead of doing a manual refresh and have a recover handler. also pass the lag time into the handlers/procs (same semantics)
19
+ investigate: why is temple getting triggered? how often? is the enqueing/checking taking too much time?
20
+ also, if one queue is bad, does it trigger other queue's handlers? write some tests, asshole.
21
+ woes of redis namespace, in regards to Awsm.redis != Resque.redis etc. (which is important for setting the key through @redis)
22
+
23
+ with lag time, it will continue to trigger, for every heartbeat time it's supposed to tick, find some way to do that, and then maybe add some resolved handler/proc?
@@ -1,4 +1,5 @@
1
1
  require "resque_stuck_queue/version"
2
+ require "resque_stuck_queue/config"
2
3
 
3
4
  # TODO move this require into a configurable?
4
5
  require 'resque'
@@ -11,35 +12,12 @@ require 'logger'
11
12
  module Resque
12
13
  module StuckQueue
13
14
 
14
- GLOBAL_KEY = "resque-stuck-queue"
15
- HEARTBEAT = 60 * 60 # check/refresh every hour
16
- TRIGGER_TIMEOUT = 5 * 60 * 60 # warn/trigger 5 hours
17
- HANDLER = proc { |queue_name| $stdout.puts("Shit gone bad with them queues...on #{queue_name}.") }
18
-
19
15
  class << self
20
16
 
21
17
  attr_accessor :config
22
18
 
23
- # # how often we refresh the key
24
- # :heartbeat = 5 * 60
25
- #
26
- # # this could just be :heartbeat but it's possible there's an acceptable lag/bottleneck
27
- # # in the queue that we want to allow to be before we think it's bad.
28
- # :trigger_timeout = 10 * 60
29
- #
30
- # # The global key that will be used to check the latest time
31
- # :global_key = "resque-stuck-queue"
32
- #
33
- # # for threads involved here. default is false
34
- # :abort_on_exception
35
- #
36
- # # default handler
37
- # config[:handler] = proc { |queue_name| send_mail }
38
- #
39
- # # explicit redis
40
- # config[:redis] = Redis.new
41
19
  def config
42
- @config ||= {}
20
+ @config ||= Config.new
43
21
  end
44
22
 
45
23
  def logger
@@ -56,12 +34,16 @@ module Resque
56
34
  Resque.redis = @redis
57
35
  end
58
36
 
59
- def global_key_for(under_queue)
60
- "#{under_queue}:#{config[:global_key] || GLOBAL_KEY}"
37
+ def heartbeat_key_for(queue)
38
+ "#{queue}:#{config[:heartbeat_key] || HEARTBEAT_KEY}"
39
+ end
40
+
41
+ def triggered_key_for(queue)
42
+ "#{queue}:#{config[:triggered_key] || TRIGGERED_KEY}"
61
43
  end
62
44
 
63
- def global_keys
64
- queues.map{|q| global_key_for(q) }
45
+ def heartbeat_keys
46
+ queues.map{|q| heartbeat_key_for(q) }
65
47
  end
66
48
 
67
49
  def queues
@@ -130,7 +112,7 @@ module Resque
130
112
  while @running
131
113
  # we want to go through resque jobs, because that's what we're trying to test here:
132
114
  # ensure that jobs get executed and the time is updated!
133
- logger.info("Sending refresh job")
115
+ logger.info("Sending refresh jobs")
134
116
  enqueue_jobs
135
117
  wait_for_it
136
118
  end
@@ -143,7 +125,7 @@ module Resque
143
125
  config[:refresh_job].call
144
126
  else
145
127
  queues.each do |queue_name|
146
- Resque.enqueue_to(queue_name, RefreshLatestTimestamp, [global_key_for(queue_name), redis.client.host, redis.client.port])
128
+ Resque.enqueue_to(queue_name, RefreshLatestTimestamp, [heartbeat_key_for(queue_name), redis.client.host, redis.client.port])
147
129
  end
148
130
  end
149
131
  end
@@ -157,8 +139,11 @@ module Resque
157
139
  if mutex.lock
158
140
  begin
159
141
  queues.each do |queue_name|
160
- if Time.now.to_i - last_time_worked(queue_name) > max_wait_time
161
- logger.info("Triggering handler for #{queue_name} at #{Time.now} (pid: #{Process.pid})")
142
+ logger.info("Checking if queue #{queue_name} is lagging")
143
+ # else TODO recovered?
144
+ if should_trigger?(queue_name)
145
+ logger.info("Triggering handler for #{queue_name} at #{Time.now}.")
146
+ logger.info("Lag time for #{queue_name} is #{lag_time(queue_name)} seconds.")
162
147
  trigger_handler(queue_name)
163
148
  end
164
149
  end
@@ -171,35 +156,77 @@ module Resque
171
156
  end
172
157
  end
173
158
 
174
- def last_time_worked(queue_name)
175
- time_set = read_from_redis(queue_name)
159
+ #def recovered?
160
+ # already triggered once (last_trigger is not nil), but lag time is ok.
161
+ # then over here we'll rm last_triggered
162
+ # and fire a recovered handler. by rm the last_triggered the next time
163
+ # there is a problem, should_trigger? should return true
164
+ #end
165
+
166
+ def last_successful_heartbeat(queue_name)
167
+ time_set = read_from_redis(heartbeat_key_for(queue_name))
176
168
  if time_set
177
169
  time_set
178
170
  else
171
+ # the first time this is being used, key wont be there
172
+ # so just start now.
179
173
  manual_refresh(queue_name)
180
174
  end.to_i
181
175
  end
182
176
 
183
- def manual_refresh(queue_name)
184
- time = Time.now.to_i
185
- redis.set(global_key_for(queue_name), time)
186
- time
177
+ def manual_refresh(queue_name, type = :first_time)
178
+ if type == :triggered
179
+ time = Time.now.to_i
180
+ redis.set(triggered_key_for(queue_name), time)
181
+ time
182
+ elsif type == :first_time
183
+ time = Time.now.to_i
184
+ redis.set(heartbeat_key_for(queue_name), time)
185
+ time
186
+ end
187
+ end
188
+
189
+ def lag_time(queue_name)
190
+ Time.now.to_i - last_successful_heartbeat(queue_name)
191
+ end
192
+
193
+ def last_triggered(queue_name)
194
+ time_set = read_from_redis(triggered_key_for(queue_name))
195
+ if !time_set.nil?
196
+ Time.now.to_i - time_set.to_i
197
+ end
198
+ end
199
+
200
+ def should_trigger?(queue_name)
201
+ if lag_time(queue_name) > max_wait_time
202
+ last_trigger = last_triggered(queue_name)
203
+
204
+ if last_trigger.nil?
205
+ return true
206
+ elsif last_trigger > max_wait_time
207
+ return true
208
+ else
209
+ # if we've already triggered, the next trigger should be on the next iteration of max_wait_time.
210
+ return false
211
+ end
212
+ end
187
213
  end
188
214
 
189
215
  def trigger_handler(queue_name)
190
- (config[:handler] || HANDLER).call(queue_name)
191
- manual_refresh(queue_name)
216
+ (config[:handler] || HANDLER).call(queue_name, lag_time(queue_name))
217
+ manual_refresh(queue_name, :triggered)
192
218
  rescue => e
193
219
  logger.info("handler for #{queue_name} crashed: #{e.inspect}")
220
+ logger.info("\n#{e.backtrace.join("\n")}")
194
221
  force_stop!
195
222
  end
196
223
 
197
- def read_from_redis(queue_name)
198
- redis.get(global_key_for(queue_name))
224
+ def read_from_redis(keyname)
225
+ redis.get(keyname)
199
226
  end
200
227
 
201
228
  def wait_for_it
202
- sleep config[:heartbeat] || HEARTBEAT
229
+ sleep config[:heartbeat] || HEARTBEAT_TIMEOUT
203
230
  end
204
231
 
205
232
  def max_wait_time
@@ -0,0 +1,60 @@
1
+ module Resque
2
+ module StuckQueue
3
+
4
+ # defaults
5
+ HEARTBEAT_KEY = "resque-stuck-queue"
6
+ TRIGGERED_KEY = "resque-stuck-queue-last-triggered"
7
+ HEARTBEAT_TIMEOUT = 20 * 60 # check/refresh every 20 mins.
8
+ TRIGGER_TIMEOUT = 40 * 60 # warn/trigger after an hour (with 20 min heartbeat time).
9
+ HANDLER = proc { |queue_name, lag| $stdout.puts("Shit gone bad with them queues...on #{queue_name}. Lag time is #{lag}") }
10
+
11
+ class Config < Hash
12
+
13
+ OPTIONS_DESCRIPTIONS = {
14
+ :handler => "set to what gets triggered when resque-stuck-queue will detect the latest heartbeat is older than the trigger_timeout time setting.\n\tExample:\n\tResque::StuckQueue.config[:handler] = proc { |queue_name, lagtime| send_email('queue \#{queue_name} isnt working, aaah the daemons') }",
15
+ :heartbeat => "set to how often to push that 'heartbeat' job to refresh the latest time it worked.\n\tExample:\n\tResque::StuckQueue.config[:heartbeat] = 5.minutes",
16
+ :trigger_timeout => "set to how much of a resque work lag you are willing to accept before being notified. note: take the :heartbeat setting into account when setting this timeout.\n\tExample:\n\tResque::StuckQueue.config[:trigger_timeout] = 55.minutes",
17
+ :redis => "set the Redis instance StuckQueue will use",
18
+ :heartbeat_key => "optional, name of keys to keep track of the last good resque heartbeat time",
19
+ :triggered_key => "optional, name of keys to keep track of the last trigger time",
20
+ :logger => "optional, pass a Logger. Default a ruby logger will be instantiated. Needs to respond to that interface.",
21
+ :queues => "optional, monitor specific queues you want to send a heartbeat/monitor to. default is :app",
22
+ :abort_on_exception => "optional, if you want the resque-stuck-queue threads to explicitly raise, default is false",
23
+ :refresh_job => "optional, your own custom refreshing job. if you are using something other than resque",
24
+ }
25
+
26
+ OPTIONS = OPTIONS_DESCRIPTIONS.keys
27
+
28
+ def []=(k,v)
29
+ validate_key_exists!(k)
30
+ super(k,v)
31
+ end
32
+
33
+ def [](k)
34
+ validate_key_exists!(k)
35
+ super(k)
36
+ end
37
+
38
+ class NoConfigError < StandardError; end
39
+
40
+ def validate_key_exists!(k)
41
+ if ! OPTIONS.include?(k)
42
+ raise NoConfigError, "no such config key exists!"
43
+ end
44
+ end
45
+
46
+ def description_for(k)
47
+ OPTIONS_DESCRIPTIONS[k.to_sym]
48
+ end
49
+
50
+ def pretty_descriptions
51
+ out = "\n"
52
+ OPTIONS_DESCRIPTIONS.map{|key,msg|
53
+ out << "#{key}:\n\t#{msg}\n\n"
54
+ }
55
+ out
56
+ end
57
+
58
+ end
59
+ end
60
+ end
@@ -1,5 +1,5 @@
1
1
  module Resque
2
2
  module StuckQueue
3
- VERSION = "0.0.11"
3
+ VERSION = "0.0.13"
4
4
  end
5
5
  end
@@ -1,6 +1,6 @@
1
1
  require File.join(File.expand_path(File.dirname(__FILE__)), "test_helper")
2
2
 
3
- class TestResqueStuckQueue < Minitest::Test
3
+ class TestConfig < Minitest::Test
4
4
 
5
5
  include TestHelper
6
6
 
@@ -14,6 +14,17 @@ class TestResqueStuckQueue < Minitest::Test
14
14
  Resque::StuckQueue.reset!
15
15
  end
16
16
 
17
+ def test_config_has_descriptions
18
+ c = Resque::StuckQueue::Config.new
19
+ assert c.description_for(:logger) =~ /Logger/, "has descriptions"
20
+ end
21
+
22
+ def test_outputs_all_config_options
23
+ c = Resque::StuckQueue::Config.new
24
+ puts c.pretty_descriptions
25
+ assert true
26
+ end
27
+
17
28
  def test_has_logger
18
29
  puts "#{__method__}"
19
30
  begin
data/test/test_helper.rb CHANGED
@@ -11,12 +11,20 @@ require File.join(File.expand_path(File.dirname(__FILE__)), "resque", "refresh_l
11
11
 
12
12
  module TestHelper
13
13
 
14
+ extend self
15
+
14
16
  def run_resque(queue_name = "*")
15
- pid = fork { exec("QUEUE=#{queue_name} bundle exec rake --trace resque:work") }
17
+ pid = fork { exec("export QUEUE=#{queue_name}; bundle exec rake --trace resque:work") }
16
18
  sleep 3 # wait for resque to boot up
17
19
  pid
18
20
  end
19
21
 
22
+ def hax_kill_resque
23
+ # ugly, FIXME how to get pid of forked forked process. run_resque pid is incorrect.
24
+ `ps aux |grep resque |awk '{print $2}' |xargs kill`
25
+ sleep 2 # wait for shutdown
26
+ end
27
+
20
28
  def start_and_stop_loops_after(secs)
21
29
  ops = []
22
30
  ops << Thread.new { Resque::StuckQueue.start }
@@ -12,84 +12,91 @@ require File.join(File.expand_path(File.dirname(__FILE__)), "test_helper")
12
12
  class TestIntegration < Minitest::Test
13
13
 
14
14
  include TestHelper
15
- # TODODS there's a better way to do this.
16
- #
17
- #
18
- # run test with VVERBOSE=1 DEBUG=1 for more output
19
- #
20
- #
21
- # => sleeping suckc
22
- # => resque sleeps 5 between checking enqueed jobs, can be configurable?
23
- #
24
- # cleanup processes correctly?
25
- # ps aux |grep resqu |awk '{print $2}' | xargs kill
15
+
16
+ # UBER HAXING no after(:all) or before(:all)
17
+ # if adding a test here, add
18
+ # self.class.tests_ran += 1
19
+ class << self
20
+ def tests_running?
21
+ test_count = public_instance_methods.select{|m| m.to_s.match(/^test_/)}.size
22
+ true if tests_ran != test_count
23
+ end
24
+
25
+ def tests_done?
26
+ !tests_running?
27
+ end
28
+
29
+ attr_accessor :tests_ran, :resque_pid
30
+ def tests_ran
31
+ @tests_ran ||= 0
32
+ end
33
+
34
+ def run_resque_before_all
35
+ return if @running_resque
36
+ @running_resque = true
37
+
38
+ @resque_pid = TestHelper.run_resque
39
+ end
40
+ end
26
41
 
27
42
  def setup
28
43
  Resque::StuckQueue.redis = Redis.new
29
44
  Resque::StuckQueue.redis.flushall
30
45
  Resque::StuckQueue.config[:abort_on_exception] = true
46
+ self.class.run_resque_before_all
31
47
  end
32
48
 
33
49
  def teardown
34
- `kill -9 #{@resque_pid}` # CONT falls throughs sometimes? hax, rm this and SIGSTOP/SIGCONT
35
- Resque::StuckQueue.force_stop!
36
- Process.waitpid(@resque_pid)
50
+ if self.class.tests_done?
51
+ hax_kill_resque
52
+ Process.waitall
53
+ return
54
+ end
37
55
  end
38
56
 
39
57
  def test_resque_enqueues_a_job_does_not_trigger
40
58
  puts "#{__method__}"
41
- Resque::StuckQueue.config[:trigger_timeout] = 100 # wait a while so we don't trigger
42
- Resque::StuckQueue.config[:heartbeat] = 2
59
+ self.class.tests_ran += 1
60
+
61
+ Resque::StuckQueue.config[:trigger_timeout] = 10
62
+ Resque::StuckQueue.config[:heartbeat] = 1
43
63
  @triggered = false
44
64
  Resque::StuckQueue.config[:handler] = proc { @triggered = true }
45
- Thread.new { Resque::StuckQueue.start }
46
-
47
- # job gets enqueued successfully
48
- @resque_pid = run_resque
65
+ start_and_stop_loops_after(5)
49
66
  Resque::StuckQueue.redis.del(SetRedisKey::NAME)
50
67
  Resque.enqueue_to(:app, SetRedisKey)
51
- sleep 6 # let resque pick up the job
52
- assert_equal Resque::StuckQueue.redis.get(SetRedisKey::NAME), "1" # job ran
53
-
54
- # check handler did not get called
68
+ sleep 3
69
+ assert_equal Resque::StuckQueue.redis.get(SetRedisKey::NAME), "1"
70
+ # job ran successfully, so don't trigger
55
71
  assert_equal @triggered, false
56
72
  end
57
73
 
58
74
  def test_resque_does_not_enqueues_a_job_does_trigger
59
75
  puts "#{__method__}"
60
- Resque::StuckQueue.config[:trigger_timeout] = 2 # won't allow waiting too much and will complain (eg trigger) sooner than later
76
+ self.class.tests_ran += 1
77
+
78
+ Resque::StuckQueue.config[:trigger_timeout] = 0
61
79
  Resque::StuckQueue.config[:heartbeat] = 1
62
80
  @triggered = false
63
81
  Resque::StuckQueue.config[:handler] = proc { @triggered = true }
64
- Thread.new { Resque::StuckQueue.start }
65
-
66
- # job gets enqueued successfully
67
- @resque_pid = run_resque
68
- Resque::StuckQueue.redis.del(SetRedisKey::NAME)
69
- Process.kill("SIGSTOP", @resque_pid) # jic, do not process jobs so we definitely trigger
70
- Resque.enqueue(SetRedisKey)
71
- assert_equal Resque::StuckQueue.redis.get(SetRedisKey::NAME), nil
72
- sleep 2 # allow timeout to trigger
73
-
82
+ start_and_stop_loops_after(2)
74
83
  # check handler did get called
75
84
  assert_equal @triggered, true
76
-
77
- # unstick the process so we can kill it in teardown
78
- Process.kill("SIGCONT", @resque_pid)
79
85
  end
80
86
 
81
87
  def test_has_settable_custom_hearbeat_job
82
88
  puts "#{__method__}"
89
+ self.class.tests_ran += 1
90
+
83
91
  Resque::StuckQueue.config[:trigger_timeout] = 2 # won't allow waiting too much and will complain (eg trigger) sooner than later
84
92
  Resque::StuckQueue.config[:heartbeat] = 1
85
93
 
86
94
  begin
87
- Resque::StuckQueue.config[:refresh_job] = proc { Resque.enqueue(RefreshLatestTimestamp, Resque::StuckQueue.global_key_for(:app)) }
95
+ Resque::StuckQueue.config[:refresh_job] = proc { Resque.enqueue(RefreshLatestTimestamp, Resque::StuckQueue.heartbeat_key_for(:app)) }
88
96
  @triggered = false
89
97
  Resque::StuckQueue.config[:handler] = proc { @triggered = true }
90
- Thread.new { Resque::StuckQueue.start }
98
+ start_and_stop_loops_after(4)
91
99
 
92
- @resque_pid = run_resque
93
100
  sleep 3 # allow trigger
94
101
  assert true, "should not have raised"
95
102
  assert @triggered, "should have triggered"
@@ -98,5 +105,4 @@ class TestIntegration < Minitest::Test
98
105
  end
99
106
  end
100
107
 
101
-
102
108
  end
@@ -0,0 +1,34 @@
1
+ require 'minitest'
2
+ require "minitest/autorun"
3
+ require 'pry'
4
+
5
+ $:.unshift(".")
6
+ require 'resque_stuck_queue'
7
+ require File.join(File.expand_path(File.dirname(__FILE__)), "resque", "set_redis_key")
8
+ require File.join(File.expand_path(File.dirname(__FILE__)), "resque", "refresh_latest_timestamp")
9
+ require File.join(File.expand_path(File.dirname(__FILE__)), "test_helper")
10
+
11
+ class TestLagTime < Minitest::Test
12
+
13
+ include TestHelper
14
+
15
+ def setup
16
+ Resque::StuckQueue.redis = Redis.new
17
+ Resque::StuckQueue.redis.flushall
18
+ Resque::StuckQueue.config[:abort_on_exception] = true
19
+ end
20
+
21
+ def test_triggers_handler_with_lagtime
22
+ Resque::StuckQueue.config[:trigger_timeout] = 2 # won't allow waiting too much and will complain (eg trigger) sooner than later
23
+ Resque::StuckQueue.config[:heartbeat] = 1
24
+ @lagtime = 0
25
+ Resque::StuckQueue.config[:handler] = proc { |queue_name, lagtime| @lagtime = lagtime }
26
+ start_and_stop_loops_after(5)
27
+
28
+ # check handler did get called
29
+ assert @lagtime > 0, "lagtime shoudl be set"
30
+ assert @lagtime < 5, "lagtime shoudl be set"
31
+ end
32
+
33
+
34
+ end
@@ -13,7 +13,7 @@ class TestNamedQueues < Minitest::Test
13
13
  end
14
14
 
15
15
  def teardown
16
- `kill -9 #{@resque_pid}` if @resque_pid
16
+ hax_kill_resque
17
17
  Resque::StuckQueue.force_stop!
18
18
  Process.waitpid(@resque_pid) if @resque_pid
19
19
  end
@@ -22,14 +22,14 @@ class TestNamedQueues < Minitest::Test
22
22
  puts "#{__method__}"
23
23
  Resque::StuckQueue.config[:queues] = nil
24
24
  start_and_stop_loops_after(2)
25
- assert Resque::StuckQueue.global_keys.include?("app:resque-stuck-queue"), 'has global keys'
25
+ assert Resque::StuckQueue.heartbeat_keys.include?("app:resque-stuck-queue"), 'has global keys'
26
26
  end
27
27
 
28
28
  def test_has_custom_queues
29
29
  puts "#{__method__}"
30
30
  Resque::StuckQueue.config[:queues] = [:foo,:bar]
31
31
  start_and_stop_loops_after(2)
32
- assert Resque::StuckQueue.global_keys.include?("foo:resque-stuck-queue"), 'has global keys'
32
+ assert Resque::StuckQueue.heartbeat_keys.include?("foo:resque-stuck-queue"), 'has global keys'
33
33
  end
34
34
 
35
35
  def test_resque_enqueues_a_job_with_resqueue_running_but_on_that_queue_does_trigger
@@ -54,7 +54,7 @@ class TestNamedQueues < Minitest::Test
54
54
  Resque::StuckQueue.config[:trigger_timeout] = 2 # won't allow waiting too much and will complain (eg trigger) sooner than later
55
55
  Resque::StuckQueue.config[:heartbeat] = 1
56
56
  Resque::StuckQueue.config[:queues] = [:custom_queue_name, :diff_one]
57
- assert Resque::StuckQueue.global_keys.include?("custom_queue_name:resque-stuck-queue"), 'has global keys'
57
+ assert Resque::StuckQueue.heartbeat_keys.include?("custom_queue_name:resque-stuck-queue"), 'has global keys'
58
58
  @triggered = false
59
59
  Resque::StuckQueue.config[:handler] = proc { @triggered = true }
60
60
  @resque_pid = run_resque("custom_queue_name")
@@ -19,11 +19,11 @@ class TestResqueStuckQueue < Minitest::Test
19
19
  Resque::StuckQueue.config[:abort_on_exception] = true
20
20
  end
21
21
 
22
- def test_configure_global_key
22
+ def test_configure_heartbeat_key
23
23
  puts "#{__method__}"
24
24
  assert_nil Resque::StuckQueue.redis.get("it-is-configurable"), "global key should not be set"
25
- Resque::StuckQueue.config[:global_key] = "it-is-configurable"
26
- start_and_stop_loops_after(2)
25
+ Resque::StuckQueue.config[:heartbeat_key] = "it-is-configurable"
26
+ start_and_stop_loops_after(3)
27
27
  refute_nil Resque::StuckQueue.redis.get("app:it-is-configurable"), "global key should be set"
28
28
  end
29
29
 
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: resque_stuck_queue
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.0.11
4
+ version: 0.0.13
5
5
  platform: ruby
6
6
  authors:
7
7
  - Shai Rosenfeld
@@ -9,7 +9,7 @@ autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
11
 
12
- date: 2014-01-23 00:00:00 Z
12
+ date: 2014-01-26 00:00:00 Z
13
13
  dependencies:
14
14
  - !ruby/object:Gem::Dependency
15
15
  name: redis-mutex
@@ -58,14 +58,16 @@ files:
58
58
  - THOUGHTS
59
59
  - lib/resque/stuck_queue.rb
60
60
  - lib/resque_stuck_queue.rb
61
+ - lib/resque_stuck_queue/config.rb
61
62
  - lib/resque_stuck_queue/version.rb
62
63
  - resque_stuck_queue.gemspec
63
64
  - test/resque/refresh_latest_timestamp.rb
64
65
  - test/resque/set_redis_key.rb
65
66
  - test/test_collision.rb
67
+ - test/test_config.rb
66
68
  - test/test_helper.rb
67
69
  - test/test_integration.rb
68
- - test/test_logger.rb
70
+ - test/test_lagtime.rb
69
71
  - test/test_named_queues.rb
70
72
  - test/test_resque_2.rb
71
73
  - test/test_resque_stuck_queue.rb
@@ -97,9 +99,10 @@ test_files:
97
99
  - test/resque/refresh_latest_timestamp.rb
98
100
  - test/resque/set_redis_key.rb
99
101
  - test/test_collision.rb
102
+ - test/test_config.rb
100
103
  - test/test_helper.rb
101
104
  - test/test_integration.rb
102
- - test/test_logger.rb
105
+ - test/test_lagtime.rb
103
106
  - test/test_named_queues.rb
104
107
  - test/test_resque_2.rb
105
108
  - test/test_resque_stuck_queue.rb