resque_stuck_queue 0.0.11 → 0.0.13

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
- SHA512:
3
- metadata.gz: df4e25ba257df91726f7c92cfe1a92f1a04b2afecb0106a4a80d0da12b494c67672fd5c40a981c813b4e18ede7d18b5e2aba3c8a445692e910acc17b9adc360e
4
- data.tar.gz: 22f4307be4f19c7e3b138c4d18013ac8fefc3a7a4d53a4071c6a6f8c1f32ec9dce2f187db9476e12d4f871a332ba6d9842fd6eb6d2ac6bf576dad79522eae783
5
2
  SHA1:
6
- metadata.gz: ff579897e68c118a67fe037962f318c2a265e851
7
- data.tar.gz: a82bad8bbf4ce498aac49d8b1f54c42047ee6b15
3
+ metadata.gz: 5d18f8d4224a9e8a9d30495ae0c7b6f3f0901f63
4
+ data.tar.gz: fd796adf7d09bb92e76c4806df3f725106d04d7a
5
+ SHA512:
6
+ metadata.gz: 476f6beaff11089bcd25c08428bdf6e7355af941ca5500c7e941c70addc5ed542e61a2f9d7d5df09970270a7401a880a0e3e58939139f00d3c5879222bc3d4f9
7
+ data.tar.gz: 6325fd7e73422455a97ea5479c82d36ef131ad5ba05a982db92d106bf088cd4804dd1126391c377bba7c8a538b56059164a303596123ab3f742c3c719ea71b87
data/README.md CHANGED
@@ -20,42 +20,41 @@ It will trigger a pre-defined proc (see below) if the last time the hearbeat job
20
20
 
21
21
  ## Usage
22
22
 
23
- Configure it first:
23
+ Configure it first. Optional settings are below. You'll most likely at the least want to tune `:handler`,`:heartbeat` and `:trigger_timeout` settings.
24
24
 
25
25
  <pre>
26
- # how often to push that 'heartbeat' job to refresh the latest time it worked.
27
- Resque::StuckQueue.config[:heartbeat] = 5.minutes
26
+ handler:
27
+ set to what gets triggered when resque-stuck-queue will detect the latest heartbeat is older than the trigger_timeout time setting.
28
+ Example:
29
+ Resque::StuckQueue.config[:handler] = proc { |queue_name, lagtime| send_email('queue #{queue_name} isnt working, aaah the daemons') }
28
30
 
29
- # since there is an realistic and acceptable lag for job queues, set this to how much you're
30
- # willing to accept between the current time and when the last hearbeat job went through.
31
- #
32
- # take the heartbeat into consideration when setting this (it will fire 10 hours + 5 minutes with above heartbeat).
33
- Resque::StuckQueue.config[:trigger_timeout] = 10.hours
31
+ heartbeat:
32
+ set to how often to push that 'heartbeat' job to refresh the latest time it worked.
33
+ Example:
34
+ Resque::StuckQueue.config[:heartbeat] = 5.minutes
34
35
 
35
- # what gets triggered when resque-stuck-queue will detect the latest heartbeat is older than the trigger_timeout time set above.
36
- #
37
- # triggering will update the key, so you'll have to wait the trigger_timeout again
38
- # in order for it to trigger again even if workers are still stale.
39
- Resque::StuckQueue.config[:handler] = proc { |queue_name| send_email("queue #{queue_name} isnt working, aaah the daemons") }
36
+ trigger_timeout:
37
+ set to how much of a resque work lag you are willing to accept before being notified. note: take the :heartbeat setting into account when setting this timeout.
38
+ Example:
39
+ Resque::StuckQueue.config[:trigger_timeout] = 55.minutes
40
40
 
41
- # optional, in case you want to set your own name for the key that will be used as the last good hearbeat time
42
- # note this will be namespaced under the specific queue it's monitoring, for eg "app:name-the-refresh-key-as-you-please"
43
- Resque::StuckQueue.config[:global_key] = "name-the-refresh-key-as-you-please"
41
+ redis:
42
+ set the Redis instance StuckQueue will use
44
43
 
45
- # optional, monitor specific queues you want to send a heartbeat/monitor
46
- Resque::StuckQueue.config[:queues] = [:app, :high, :my_custom_queue_name]
44
+ heartbeat_key:
45
+ optional, name of keys to keep track of the last good resque heartbeat time
47
46
 
48
- # optional, if you want the resque-stuck-queue threads to explicitly raise, default is false
49
- Resque::StuckQueue.config[:abort_on_exception] = true
47
+ logger:
48
+ optional, pass a Logger. Default a ruby logger will be instantiated. Needs to respond to that interface.
50
49
 
51
- # optional, pass a logger. Default a ruby logger will be instantiated. Needs to respond to that interface.
52
- Resque::StuckQueue.config[:logger] = Logger.new($stdout)
50
+ queues:
51
+ optional, monitor specific queues you want to send a heartbeat/monitor to. default is :app
53
52
 
54
- # optional, pass a redis.
55
- Resque::StuckQueue.config[:redis] = Redis.new
53
+ abort_on_exception:
54
+ optional, if you want the resque-stuck-queue threads to explicitly raise, default is false
56
55
 
57
- # optional, your own custom refresh job. below has an example.
58
- Resque::StuckQueue.config[:refresh_job] = proc { your_own_enqueue_refreshing_global_key }
56
+ refresh_job:
57
+ optional, your own custom refreshing job. if you are using something other than resque
59
58
  </pre>
60
59
 
61
60
  Then start it:
@@ -110,7 +109,7 @@ $ bundle exec rake --trace resque:stuck_queue
110
109
 
111
110
  ## Sidekiq/Other redis-based job queues
112
111
 
113
- If you have trouble with other queues you can use this lib by setting your own custom refresh job (aka, the job that refreshes your queue specific global_key). The one thing you need to take care of is ensure whatever and however you enque your own custom job, it sets the global_key to Time.now:
112
+ If you have trouble with other queues you can use this lib by setting your own custom refresh job (aka, the job that refreshes your queue specific heartbeat_key). The one thing you need to take care of is ensure whatever and however you enque your own custom job, it sets the heartbeat_key to Time.now:
114
113
 
115
114
  <pre>
116
115
 
@@ -118,7 +117,7 @@ class CustomJob
118
117
  include Sidekiq::Worker
119
118
  def perform
120
119
  # ensure you're setting the key in the redis the job queue is using
121
- $redis.set(Resque::StuckQueue.global_key_for(queue_name), Time.now.to_i)
120
+ $redis.set(Resque::StuckQueue.heartbeat_key_for(queue_name), Time.now.to_i)
122
121
  end
123
122
  end
124
123
 
data/THOUGHTS CHANGED
@@ -14,3 +14,10 @@ add a trap{} to force_stop. ok for overwriting process's trap handlers? use conf
14
14
 
15
15
  fix skeleton recipe https://github.com/shaiguitar/resque_stuck_queue/blame/master/README.md#L103
16
16
  raise appname, => :environment, log path
17
+
18
+ set a redis key instead of doing a manual refresh and have a recover handler. also pass the lag time into the handlers/procs (same semantics)
19
+ investigate: why is temple getting triggered? how often? is the enqueing/checking taking too much time?
20
+ also, if one queue is bad, does it trigger other queue's handlers? write some tests, asshole.
21
+ woes of redis namespace, in regards to Awsm.redis != Resque.redis etc. (which is important for setting the key through @redis)
22
+
23
+ with lag time, it will continue to trigger, for every heartbeat time it's supposed to tick, find some way to do that, and then maybe add some resolved handler/proc?
@@ -1,4 +1,5 @@
1
1
  require "resque_stuck_queue/version"
2
+ require "resque_stuck_queue/config"
2
3
 
3
4
  # TODO move this require into a configurable?
4
5
  require 'resque'
@@ -11,35 +12,12 @@ require 'logger'
11
12
  module Resque
12
13
  module StuckQueue
13
14
 
14
- GLOBAL_KEY = "resque-stuck-queue"
15
- HEARTBEAT = 60 * 60 # check/refresh every hour
16
- TRIGGER_TIMEOUT = 5 * 60 * 60 # warn/trigger 5 hours
17
- HANDLER = proc { |queue_name| $stdout.puts("Shit gone bad with them queues...on #{queue_name}.") }
18
-
19
15
  class << self
20
16
 
21
17
  attr_accessor :config
22
18
 
23
- # # how often we refresh the key
24
- # :heartbeat = 5 * 60
25
- #
26
- # # this could just be :heartbeat but it's possible there's an acceptable lag/bottleneck
27
- # # in the queue that we want to allow to be before we think it's bad.
28
- # :trigger_timeout = 10 * 60
29
- #
30
- # # The global key that will be used to check the latest time
31
- # :global_key = "resque-stuck-queue"
32
- #
33
- # # for threads involved here. default is false
34
- # :abort_on_exception
35
- #
36
- # # default handler
37
- # config[:handler] = proc { |queue_name| send_mail }
38
- #
39
- # # explicit redis
40
- # config[:redis] = Redis.new
41
19
  def config
42
- @config ||= {}
20
+ @config ||= Config.new
43
21
  end
44
22
 
45
23
  def logger
@@ -56,12 +34,16 @@ module Resque
56
34
  Resque.redis = @redis
57
35
  end
58
36
 
59
- def global_key_for(under_queue)
60
- "#{under_queue}:#{config[:global_key] || GLOBAL_KEY}"
37
+ def heartbeat_key_for(queue)
38
+ "#{queue}:#{config[:heartbeat_key] || HEARTBEAT_KEY}"
39
+ end
40
+
41
+ def triggered_key_for(queue)
42
+ "#{queue}:#{config[:triggered_key] || TRIGGERED_KEY}"
61
43
  end
62
44
 
63
- def global_keys
64
- queues.map{|q| global_key_for(q) }
45
+ def heartbeat_keys
46
+ queues.map{|q| heartbeat_key_for(q) }
65
47
  end
66
48
 
67
49
  def queues
@@ -130,7 +112,7 @@ module Resque
130
112
  while @running
131
113
  # we want to go through resque jobs, because that's what we're trying to test here:
132
114
  # ensure that jobs get executed and the time is updated!
133
- logger.info("Sending refresh job")
115
+ logger.info("Sending refresh jobs")
134
116
  enqueue_jobs
135
117
  wait_for_it
136
118
  end
@@ -143,7 +125,7 @@ module Resque
143
125
  config[:refresh_job].call
144
126
  else
145
127
  queues.each do |queue_name|
146
- Resque.enqueue_to(queue_name, RefreshLatestTimestamp, [global_key_for(queue_name), redis.client.host, redis.client.port])
128
+ Resque.enqueue_to(queue_name, RefreshLatestTimestamp, [heartbeat_key_for(queue_name), redis.client.host, redis.client.port])
147
129
  end
148
130
  end
149
131
  end
@@ -157,8 +139,11 @@ module Resque
157
139
  if mutex.lock
158
140
  begin
159
141
  queues.each do |queue_name|
160
- if Time.now.to_i - last_time_worked(queue_name) > max_wait_time
161
- logger.info("Triggering handler for #{queue_name} at #{Time.now} (pid: #{Process.pid})")
142
+ logger.info("Checking if queue #{queue_name} is lagging")
143
+ # else TODO recovered?
144
+ if should_trigger?(queue_name)
145
+ logger.info("Triggering handler for #{queue_name} at #{Time.now}.")
146
+ logger.info("Lag time for #{queue_name} is #{lag_time(queue_name)} seconds.")
162
147
  trigger_handler(queue_name)
163
148
  end
164
149
  end
@@ -171,35 +156,77 @@ module Resque
171
156
  end
172
157
  end
173
158
 
174
- def last_time_worked(queue_name)
175
- time_set = read_from_redis(queue_name)
159
+ #def recovered?
160
+ # already triggered once (last_trigger is not nil), but lag time is ok.
161
+ # then over here we'll rm last_triggered
162
+ # and fire a recovered handler. by rm the last_triggered the next time
163
+ # there is a problem, should_trigger? should return true
164
+ #end
165
+
166
+ def last_successful_heartbeat(queue_name)
167
+ time_set = read_from_redis(heartbeat_key_for(queue_name))
176
168
  if time_set
177
169
  time_set
178
170
  else
171
+ # the first time this is being used, key wont be there
172
+ # so just start now.
179
173
  manual_refresh(queue_name)
180
174
  end.to_i
181
175
  end
182
176
 
183
- def manual_refresh(queue_name)
184
- time = Time.now.to_i
185
- redis.set(global_key_for(queue_name), time)
186
- time
177
+ def manual_refresh(queue_name, type = :first_time)
178
+ if type == :triggered
179
+ time = Time.now.to_i
180
+ redis.set(triggered_key_for(queue_name), time)
181
+ time
182
+ elsif type == :first_time
183
+ time = Time.now.to_i
184
+ redis.set(heartbeat_key_for(queue_name), time)
185
+ time
186
+ end
187
+ end
188
+
189
+ def lag_time(queue_name)
190
+ Time.now.to_i - last_successful_heartbeat(queue_name)
191
+ end
192
+
193
+ def last_triggered(queue_name)
194
+ time_set = read_from_redis(triggered_key_for(queue_name))
195
+ if !time_set.nil?
196
+ Time.now.to_i - time_set.to_i
197
+ end
198
+ end
199
+
200
+ def should_trigger?(queue_name)
201
+ if lag_time(queue_name) > max_wait_time
202
+ last_trigger = last_triggered(queue_name)
203
+
204
+ if last_trigger.nil?
205
+ return true
206
+ elsif last_trigger > max_wait_time
207
+ return true
208
+ else
209
+ # if we've already triggered, the next trigger should be on the next iteration of max_wait_time.
210
+ return false
211
+ end
212
+ end
187
213
  end
188
214
 
189
215
  def trigger_handler(queue_name)
190
- (config[:handler] || HANDLER).call(queue_name)
191
- manual_refresh(queue_name)
216
+ (config[:handler] || HANDLER).call(queue_name, lag_time(queue_name))
217
+ manual_refresh(queue_name, :triggered)
192
218
  rescue => e
193
219
  logger.info("handler for #{queue_name} crashed: #{e.inspect}")
220
+ logger.info("\n#{e.backtrace.join("\n")}")
194
221
  force_stop!
195
222
  end
196
223
 
197
- def read_from_redis(queue_name)
198
- redis.get(global_key_for(queue_name))
224
+ def read_from_redis(keyname)
225
+ redis.get(keyname)
199
226
  end
200
227
 
201
228
  def wait_for_it
202
- sleep config[:heartbeat] || HEARTBEAT
229
+ sleep config[:heartbeat] || HEARTBEAT_TIMEOUT
203
230
  end
204
231
 
205
232
  def max_wait_time
@@ -0,0 +1,60 @@
1
+ module Resque
2
+ module StuckQueue
3
+
4
+ # defaults
5
+ HEARTBEAT_KEY = "resque-stuck-queue"
6
+ TRIGGERED_KEY = "resque-stuck-queue-last-triggered"
7
+ HEARTBEAT_TIMEOUT = 20 * 60 # check/refresh every 20 mins.
8
+ TRIGGER_TIMEOUT = 40 * 60 # warn/trigger after an hour (with 20 min heartbeat time).
9
+ HANDLER = proc { |queue_name, lag| $stdout.puts("Shit gone bad with them queues...on #{queue_name}. Lag time is #{lag}") }
10
+
11
+ class Config < Hash
12
+
13
+ OPTIONS_DESCRIPTIONS = {
14
+ :handler => "set to what gets triggered when resque-stuck-queue will detect the latest heartbeat is older than the trigger_timeout time setting.\n\tExample:\n\tResque::StuckQueue.config[:handler] = proc { |queue_name, lagtime| send_email('queue \#{queue_name} isnt working, aaah the daemons') }",
15
+ :heartbeat => "set to how often to push that 'heartbeat' job to refresh the latest time it worked.\n\tExample:\n\tResque::StuckQueue.config[:heartbeat] = 5.minutes",
16
+ :trigger_timeout => "set to how much of a resque work lag you are willing to accept before being notified. note: take the :heartbeat setting into account when setting this timeout.\n\tExample:\n\tResque::StuckQueue.config[:trigger_timeout] = 55.minutes",
17
+ :redis => "set the Redis instance StuckQueue will use",
18
+ :heartbeat_key => "optional, name of keys to keep track of the last good resque heartbeat time",
19
+ :triggered_key => "optional, name of keys to keep track of the last trigger time",
20
+ :logger => "optional, pass a Logger. Default a ruby logger will be instantiated. Needs to respond to that interface.",
21
+ :queues => "optional, monitor specific queues you want to send a heartbeat/monitor to. default is :app",
22
+ :abort_on_exception => "optional, if you want the resque-stuck-queue threads to explicitly raise, default is false",
23
+ :refresh_job => "optional, your own custom refreshing job. if you are using something other than resque",
24
+ }
25
+
26
+ OPTIONS = OPTIONS_DESCRIPTIONS.keys
27
+
28
+ def []=(k,v)
29
+ validate_key_exists!(k)
30
+ super(k,v)
31
+ end
32
+
33
+ def [](k)
34
+ validate_key_exists!(k)
35
+ super(k)
36
+ end
37
+
38
+ class NoConfigError < StandardError; end
39
+
40
+ def validate_key_exists!(k)
41
+ if ! OPTIONS.include?(k)
42
+ raise NoConfigError, "no such config key exists!"
43
+ end
44
+ end
45
+
46
+ def description_for(k)
47
+ OPTIONS_DESCRIPTIONS[k.to_sym]
48
+ end
49
+
50
+ def pretty_descriptions
51
+ out = "\n"
52
+ OPTIONS_DESCRIPTIONS.map{|key,msg|
53
+ out << "#{key}:\n\t#{msg}\n\n"
54
+ }
55
+ out
56
+ end
57
+
58
+ end
59
+ end
60
+ end
@@ -1,5 +1,5 @@
1
1
  module Resque
2
2
  module StuckQueue
3
- VERSION = "0.0.11"
3
+ VERSION = "0.0.13"
4
4
  end
5
5
  end
@@ -1,6 +1,6 @@
1
1
  require File.join(File.expand_path(File.dirname(__FILE__)), "test_helper")
2
2
 
3
- class TestResqueStuckQueue < Minitest::Test
3
+ class TestConfig < Minitest::Test
4
4
 
5
5
  include TestHelper
6
6
 
@@ -14,6 +14,17 @@ class TestResqueStuckQueue < Minitest::Test
14
14
  Resque::StuckQueue.reset!
15
15
  end
16
16
 
17
+ def test_config_has_descriptions
18
+ c = Resque::StuckQueue::Config.new
19
+ assert c.description_for(:logger) =~ /Logger/, "has descriptions"
20
+ end
21
+
22
+ def test_outputs_all_config_options
23
+ c = Resque::StuckQueue::Config.new
24
+ puts c.pretty_descriptions
25
+ assert true
26
+ end
27
+
17
28
  def test_has_logger
18
29
  puts "#{__method__}"
19
30
  begin
data/test/test_helper.rb CHANGED
@@ -11,12 +11,20 @@ require File.join(File.expand_path(File.dirname(__FILE__)), "resque", "refresh_l
11
11
 
12
12
  module TestHelper
13
13
 
14
+ extend self
15
+
14
16
  def run_resque(queue_name = "*")
15
- pid = fork { exec("QUEUE=#{queue_name} bundle exec rake --trace resque:work") }
17
+ pid = fork { exec("export QUEUE=#{queue_name}; bundle exec rake --trace resque:work") }
16
18
  sleep 3 # wait for resque to boot up
17
19
  pid
18
20
  end
19
21
 
22
+ def hax_kill_resque
23
+ # ugly, FIXME how to get pid of forked forked process. run_resque pid is incorrect.
24
+ `ps aux |grep resque |awk '{print $2}' |xargs kill`
25
+ sleep 2 # wait for shutdown
26
+ end
27
+
20
28
  def start_and_stop_loops_after(secs)
21
29
  ops = []
22
30
  ops << Thread.new { Resque::StuckQueue.start }
@@ -12,84 +12,91 @@ require File.join(File.expand_path(File.dirname(__FILE__)), "test_helper")
12
12
  class TestIntegration < Minitest::Test
13
13
 
14
14
  include TestHelper
15
- # TODODS there's a better way to do this.
16
- #
17
- #
18
- # run test with VVERBOSE=1 DEBUG=1 for more output
19
- #
20
- #
21
- # => sleeping suckc
22
- # => resque sleeps 5 between checking enqueed jobs, can be configurable?
23
- #
24
- # cleanup processes correctly?
25
- # ps aux |grep resqu |awk '{print $2}' | xargs kill
15
+
16
+ # UBER HAXING no after(:all) or before(:all)
17
+ # if adding a test here, add
18
+ # self.class.tests_ran += 1
19
+ class << self
20
+ def tests_running?
21
+ test_count = public_instance_methods.select{|m| m.to_s.match(/^test_/)}.size
22
+ true if tests_ran != test_count
23
+ end
24
+
25
+ def tests_done?
26
+ !tests_running?
27
+ end
28
+
29
+ attr_accessor :tests_ran, :resque_pid
30
+ def tests_ran
31
+ @tests_ran ||= 0
32
+ end
33
+
34
+ def run_resque_before_all
35
+ return if @running_resque
36
+ @running_resque = true
37
+
38
+ @resque_pid = TestHelper.run_resque
39
+ end
40
+ end
26
41
 
27
42
  def setup
28
43
  Resque::StuckQueue.redis = Redis.new
29
44
  Resque::StuckQueue.redis.flushall
30
45
  Resque::StuckQueue.config[:abort_on_exception] = true
46
+ self.class.run_resque_before_all
31
47
  end
32
48
 
33
49
  def teardown
34
- `kill -9 #{@resque_pid}` # CONT falls throughs sometimes? hax, rm this and SIGSTOP/SIGCONT
35
- Resque::StuckQueue.force_stop!
36
- Process.waitpid(@resque_pid)
50
+ if self.class.tests_done?
51
+ hax_kill_resque
52
+ Process.waitall
53
+ return
54
+ end
37
55
  end
38
56
 
39
57
  def test_resque_enqueues_a_job_does_not_trigger
40
58
  puts "#{__method__}"
41
- Resque::StuckQueue.config[:trigger_timeout] = 100 # wait a while so we don't trigger
42
- Resque::StuckQueue.config[:heartbeat] = 2
59
+ self.class.tests_ran += 1
60
+
61
+ Resque::StuckQueue.config[:trigger_timeout] = 10
62
+ Resque::StuckQueue.config[:heartbeat] = 1
43
63
  @triggered = false
44
64
  Resque::StuckQueue.config[:handler] = proc { @triggered = true }
45
- Thread.new { Resque::StuckQueue.start }
46
-
47
- # job gets enqueued successfully
48
- @resque_pid = run_resque
65
+ start_and_stop_loops_after(5)
49
66
  Resque::StuckQueue.redis.del(SetRedisKey::NAME)
50
67
  Resque.enqueue_to(:app, SetRedisKey)
51
- sleep 6 # let resque pick up the job
52
- assert_equal Resque::StuckQueue.redis.get(SetRedisKey::NAME), "1" # job ran
53
-
54
- # check handler did not get called
68
+ sleep 3
69
+ assert_equal Resque::StuckQueue.redis.get(SetRedisKey::NAME), "1"
70
+ # job ran successfully, so don't trigger
55
71
  assert_equal @triggered, false
56
72
  end
57
73
 
58
74
  def test_resque_does_not_enqueues_a_job_does_trigger
59
75
  puts "#{__method__}"
60
- Resque::StuckQueue.config[:trigger_timeout] = 2 # won't allow waiting too much and will complain (eg trigger) sooner than later
76
+ self.class.tests_ran += 1
77
+
78
+ Resque::StuckQueue.config[:trigger_timeout] = 0
61
79
  Resque::StuckQueue.config[:heartbeat] = 1
62
80
  @triggered = false
63
81
  Resque::StuckQueue.config[:handler] = proc { @triggered = true }
64
- Thread.new { Resque::StuckQueue.start }
65
-
66
- # job gets enqueued successfully
67
- @resque_pid = run_resque
68
- Resque::StuckQueue.redis.del(SetRedisKey::NAME)
69
- Process.kill("SIGSTOP", @resque_pid) # jic, do not process jobs so we definitely trigger
70
- Resque.enqueue(SetRedisKey)
71
- assert_equal Resque::StuckQueue.redis.get(SetRedisKey::NAME), nil
72
- sleep 2 # allow timeout to trigger
73
-
82
+ start_and_stop_loops_after(2)
74
83
  # check handler did get called
75
84
  assert_equal @triggered, true
76
-
77
- # unstick the process so we can kill it in teardown
78
- Process.kill("SIGCONT", @resque_pid)
79
85
  end
80
86
 
81
87
  def test_has_settable_custom_hearbeat_job
82
88
  puts "#{__method__}"
89
+ self.class.tests_ran += 1
90
+
83
91
  Resque::StuckQueue.config[:trigger_timeout] = 2 # won't allow waiting too much and will complain (eg trigger) sooner than later
84
92
  Resque::StuckQueue.config[:heartbeat] = 1
85
93
 
86
94
  begin
87
- Resque::StuckQueue.config[:refresh_job] = proc { Resque.enqueue(RefreshLatestTimestamp, Resque::StuckQueue.global_key_for(:app)) }
95
+ Resque::StuckQueue.config[:refresh_job] = proc { Resque.enqueue(RefreshLatestTimestamp, Resque::StuckQueue.heartbeat_key_for(:app)) }
88
96
  @triggered = false
89
97
  Resque::StuckQueue.config[:handler] = proc { @triggered = true }
90
- Thread.new { Resque::StuckQueue.start }
98
+ start_and_stop_loops_after(4)
91
99
 
92
- @resque_pid = run_resque
93
100
  sleep 3 # allow trigger
94
101
  assert true, "should not have raised"
95
102
  assert @triggered, "should have triggered"
@@ -98,5 +105,4 @@ class TestIntegration < Minitest::Test
98
105
  end
99
106
  end
100
107
 
101
-
102
108
  end
@@ -0,0 +1,34 @@
1
+ require 'minitest'
2
+ require "minitest/autorun"
3
+ require 'pry'
4
+
5
+ $:.unshift(".")
6
+ require 'resque_stuck_queue'
7
+ require File.join(File.expand_path(File.dirname(__FILE__)), "resque", "set_redis_key")
8
+ require File.join(File.expand_path(File.dirname(__FILE__)), "resque", "refresh_latest_timestamp")
9
+ require File.join(File.expand_path(File.dirname(__FILE__)), "test_helper")
10
+
11
+ class TestLagTime < Minitest::Test
12
+
13
+ include TestHelper
14
+
15
+ def setup
16
+ Resque::StuckQueue.redis = Redis.new
17
+ Resque::StuckQueue.redis.flushall
18
+ Resque::StuckQueue.config[:abort_on_exception] = true
19
+ end
20
+
21
+ def test_triggers_handler_with_lagtime
22
+ Resque::StuckQueue.config[:trigger_timeout] = 2 # won't allow waiting too much and will complain (eg trigger) sooner than later
23
+ Resque::StuckQueue.config[:heartbeat] = 1
24
+ @lagtime = 0
25
+ Resque::StuckQueue.config[:handler] = proc { |queue_name, lagtime| @lagtime = lagtime }
26
+ start_and_stop_loops_after(5)
27
+
28
+ # check handler did get called
29
+ assert @lagtime > 0, "lagtime shoudl be set"
30
+ assert @lagtime < 5, "lagtime shoudl be set"
31
+ end
32
+
33
+
34
+ end
@@ -13,7 +13,7 @@ class TestNamedQueues < Minitest::Test
13
13
  end
14
14
 
15
15
  def teardown
16
- `kill -9 #{@resque_pid}` if @resque_pid
16
+ hax_kill_resque
17
17
  Resque::StuckQueue.force_stop!
18
18
  Process.waitpid(@resque_pid) if @resque_pid
19
19
  end
@@ -22,14 +22,14 @@ class TestNamedQueues < Minitest::Test
22
22
  puts "#{__method__}"
23
23
  Resque::StuckQueue.config[:queues] = nil
24
24
  start_and_stop_loops_after(2)
25
- assert Resque::StuckQueue.global_keys.include?("app:resque-stuck-queue"), 'has global keys'
25
+ assert Resque::StuckQueue.heartbeat_keys.include?("app:resque-stuck-queue"), 'has global keys'
26
26
  end
27
27
 
28
28
  def test_has_custom_queues
29
29
  puts "#{__method__}"
30
30
  Resque::StuckQueue.config[:queues] = [:foo,:bar]
31
31
  start_and_stop_loops_after(2)
32
- assert Resque::StuckQueue.global_keys.include?("foo:resque-stuck-queue"), 'has global keys'
32
+ assert Resque::StuckQueue.heartbeat_keys.include?("foo:resque-stuck-queue"), 'has global keys'
33
33
  end
34
34
 
35
35
  def test_resque_enqueues_a_job_with_resqueue_running_but_on_that_queue_does_trigger
@@ -54,7 +54,7 @@ class TestNamedQueues < Minitest::Test
54
54
  Resque::StuckQueue.config[:trigger_timeout] = 2 # won't allow waiting too much and will complain (eg trigger) sooner than later
55
55
  Resque::StuckQueue.config[:heartbeat] = 1
56
56
  Resque::StuckQueue.config[:queues] = [:custom_queue_name, :diff_one]
57
- assert Resque::StuckQueue.global_keys.include?("custom_queue_name:resque-stuck-queue"), 'has global keys'
57
+ assert Resque::StuckQueue.heartbeat_keys.include?("custom_queue_name:resque-stuck-queue"), 'has global keys'
58
58
  @triggered = false
59
59
  Resque::StuckQueue.config[:handler] = proc { @triggered = true }
60
60
  @resque_pid = run_resque("custom_queue_name")
@@ -19,11 +19,11 @@ class TestResqueStuckQueue < Minitest::Test
19
19
  Resque::StuckQueue.config[:abort_on_exception] = true
20
20
  end
21
21
 
22
- def test_configure_global_key
22
+ def test_configure_heartbeat_key
23
23
  puts "#{__method__}"
24
24
  assert_nil Resque::StuckQueue.redis.get("it-is-configurable"), "global key should not be set"
25
- Resque::StuckQueue.config[:global_key] = "it-is-configurable"
26
- start_and_stop_loops_after(2)
25
+ Resque::StuckQueue.config[:heartbeat_key] = "it-is-configurable"
26
+ start_and_stop_loops_after(3)
27
27
  refute_nil Resque::StuckQueue.redis.get("app:it-is-configurable"), "global key should be set"
28
28
  end
29
29
 
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: resque_stuck_queue
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.0.11
4
+ version: 0.0.13
5
5
  platform: ruby
6
6
  authors:
7
7
  - Shai Rosenfeld
@@ -9,7 +9,7 @@ autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
11
 
12
- date: 2014-01-23 00:00:00 Z
12
+ date: 2014-01-26 00:00:00 Z
13
13
  dependencies:
14
14
  - !ruby/object:Gem::Dependency
15
15
  name: redis-mutex
@@ -58,14 +58,16 @@ files:
58
58
  - THOUGHTS
59
59
  - lib/resque/stuck_queue.rb
60
60
  - lib/resque_stuck_queue.rb
61
+ - lib/resque_stuck_queue/config.rb
61
62
  - lib/resque_stuck_queue/version.rb
62
63
  - resque_stuck_queue.gemspec
63
64
  - test/resque/refresh_latest_timestamp.rb
64
65
  - test/resque/set_redis_key.rb
65
66
  - test/test_collision.rb
67
+ - test/test_config.rb
66
68
  - test/test_helper.rb
67
69
  - test/test_integration.rb
68
- - test/test_logger.rb
70
+ - test/test_lagtime.rb
69
71
  - test/test_named_queues.rb
70
72
  - test/test_resque_2.rb
71
73
  - test/test_resque_stuck_queue.rb
@@ -97,9 +99,10 @@ test_files:
97
99
  - test/resque/refresh_latest_timestamp.rb
98
100
  - test/resque/set_redis_key.rb
99
101
  - test/test_collision.rb
102
+ - test/test_config.rb
100
103
  - test/test_helper.rb
101
104
  - test/test_integration.rb
102
- - test/test_logger.rb
105
+ - test/test_lagtime.rb
103
106
  - test/test_named_queues.rb
104
107
  - test/test_resque_2.rb
105
108
  - test/test_resque_stuck_queue.rb