resque-scheduler 4.3.0 → 4.7.0
Sign up to get free protection for your applications and to get access to all the features.
Potentially problematic release.
This version of resque-scheduler might be problematic. Click here for more details.
- checksums.yaml +5 -5
- data/.github/dependabot.yml +12 -0
- data/.github/funding.yml +4 -0
- data/.github/workflows/codeql-analysis.yml +59 -0
- data/.github/workflows/rubocop.yml +27 -0
- data/.github/workflows/ruby.yml +78 -0
- data/AUTHORS.md +9 -0
- data/CHANGELOG.md +510 -0
- data/Gemfile +23 -0
- data/README.md +115 -23
- data/Rakefile +2 -5
- data/lib/resque/scheduler/cli.rb +1 -0
- data/lib/resque/scheduler/configuration.rb +31 -8
- data/lib/resque/scheduler/delaying_extensions.rb +65 -18
- data/lib/resque/scheduler/env.rb +3 -7
- data/lib/resque/scheduler/lock/resilient.rb +19 -12
- data/lib/resque/scheduler/locking.rb +3 -3
- data/lib/resque/scheduler/scheduling_extensions.rb +4 -3
- data/lib/resque/scheduler/server/views/delayed.erb +4 -4
- data/lib/resque/scheduler/server/views/delayed_timestamp.erb +1 -1
- data/lib/resque/scheduler/server/views/scheduler.erb +2 -2
- data/lib/resque/scheduler/server/views/search.erb +0 -3
- data/lib/resque/scheduler/server/views/search_form.erb +1 -5
- data/lib/resque/scheduler/server.rb +1 -1
- data/lib/resque/scheduler/signal_handling.rb +2 -2
- data/lib/resque/scheduler/util.rb +1 -1
- data/lib/resque/scheduler/version.rb +1 -1
- data/lib/resque/scheduler.rb +39 -17
- data/resque-scheduler.gemspec +14 -7
- metadata +39 -27
- data/HISTORY.md +0 -310
data/README.md
CHANGED
@@ -1,10 +1,9 @@
|
|
1
1
|
resque-scheduler
|
2
2
|
================
|
3
3
|
|
4
|
-
|
4
|
+
|
5
5
|
[![Gem Version](https://badge.fury.io/rb/resque-scheduler.svg)](https://badge.fury.io/rb/resque-scheduler)
|
6
|
-
[![
|
7
|
-
[![Windows Build Status](https://ci.appveyor.com/api/projects/status/sxvf2086v5j0absb/branch/master?svg=true)](https://ci.appveyor.com/project/resque/resque-scheduler/branch/master)
|
6
|
+
[![Ruby specs](https://github.com/resque/resque-scheduler/actions/workflows/ruby.yml/badge.svg)](https://github.com/resque/resque-scheduler/actions)
|
8
7
|
[![Code Climate](https://codeclimate.com/github/resque/resque-scheduler/badges/gpa.svg)](https://codeclimate.com/github/resque/resque-scheduler)
|
9
8
|
|
10
9
|
### Description
|
@@ -12,7 +11,7 @@ resque-scheduler
|
|
12
11
|
Resque-scheduler is an extension to [Resque](http://github.com/resque/resque)
|
13
12
|
that adds support for queueing items in the future.
|
14
13
|
|
15
|
-
Job scheduling is supported in two different
|
14
|
+
Job scheduling is supported in two different ways: Recurring (scheduled) and
|
16
15
|
Delayed.
|
17
16
|
|
18
17
|
Scheduled jobs are like cron jobs, recurring on a regular basis. Delayed
|
@@ -159,7 +158,7 @@ following task to wherever tasks are kept, such as
|
|
159
158
|
```ruby
|
160
159
|
task 'resque:pool:setup' do
|
161
160
|
Resque::Pool.after_prefork do |job|
|
162
|
-
Resque.redis.
|
161
|
+
Resque.redis.reconnect
|
163
162
|
end
|
164
163
|
end
|
165
164
|
```
|
@@ -212,7 +211,7 @@ since the jobs are stored in a redis sorted set (zset). I can't imagine this
|
|
212
211
|
being an issue for someone since redis is stupidly fast even at log(n), but full
|
213
212
|
disclosure is always best.
|
214
213
|
|
215
|
-
#### Removing Delayed
|
214
|
+
#### Removing Delayed Jobs
|
216
215
|
|
217
216
|
If you have the need to cancel a delayed job, you can do like so:
|
218
217
|
|
@@ -234,6 +233,17 @@ Resque.remove_delayed_selection { |args| args[0]['account_id'] == current_accoun
|
|
234
233
|
Resque.remove_delayed_selection { |args| args[0]['user_id'] == current_user.id }
|
235
234
|
```
|
236
235
|
|
236
|
+
If you need to cancel a delayed job based on some matching arguments AND by which class the job is, but don't wish to specify each argument from when the job was created, you can do like so:
|
237
|
+
|
238
|
+
``` ruby
|
239
|
+
# after you've enqueued a job like:
|
240
|
+
Resque.enqueue_at(5.days.from_now, SendFollowUpEmail, :account_id => current_account.id, :user_id => current_user.id)
|
241
|
+
# remove jobs matching just the account and that were of the class SendFollowUpEmail:
|
242
|
+
Resque.remove_delayed_selection(SendFollowUpEmail) { |args| args[0]['account_id'] == current_account.id }
|
243
|
+
# or remove jobs matching just the user and that were of the class SendFollowUpEmail:
|
244
|
+
Resque.remove_delayed_selection(SendFollowUpEmail) { |args| args[0]['user_id'] == current_user.id }
|
245
|
+
```
|
246
|
+
|
237
247
|
If you need to enqueue immediately a delayed job based on some matching arguments, but don't wish to specify each argument from when the job was created, you can do like so:
|
238
248
|
|
239
249
|
``` ruby
|
@@ -245,6 +255,49 @@ Resque.enqueue_delayed_selection { |args| args[0]['account_id'] == current_accou
|
|
245
255
|
Resque.enqueue_delayed_selection { |args| args[0]['user_id'] == current_user.id }
|
246
256
|
```
|
247
257
|
|
258
|
+
#### Updating Delayed Jobs
|
259
|
+
|
260
|
+
Previously delayed jobs may be delayed even further into the future like so:
|
261
|
+
|
262
|
+
```ruby
|
263
|
+
# after you've enqueued a job like:
|
264
|
+
Resque.enqueue_at(1.minute.from_now, SendNotifications, :user_id => current_user.id)
|
265
|
+
# delay running the job until two minutes from now
|
266
|
+
Resque.delay_or_enqueue_at(2.minutes.from_now, SendNotifications, :user_id => current_user.id)
|
267
|
+
```
|
268
|
+
|
269
|
+
You don't need to worry if a matching job has already been queued, because if no matching jobs are found a new job is created and enqueued as if you had called `enqueue_at`. This means you don't need any special conditionals to know if a job has already been queued. You simply create the job like so:
|
270
|
+
|
271
|
+
```ruby
|
272
|
+
Resque.delay_or_enqueue_at(1.minute.from_now, SendNotifications, :user_id => current_user.id)
|
273
|
+
```
|
274
|
+
|
275
|
+
If multiple matching jobs are found, all of the matching jobs will be updated to have the same timestamp even if their original timestamps were not the same.
|
276
|
+
|
277
|
+
```ruby
|
278
|
+
# enqueue multiple jobs with different delay timestamps
|
279
|
+
Resque.enqueue_at(1.minute.from_now, SendNotifications, :user_id => current_user.id)
|
280
|
+
Resque.enqueue_at(2.minutes.from_now, SendNotifications, :user_id => current_user.id)
|
281
|
+
|
282
|
+
# delay running the two jobs until 5 minutes from now
|
283
|
+
Resque.delay_or_enqueue_at(5.minutes.from_now, SendNotifications, :user_id => current_user.id)
|
284
|
+
```
|
285
|
+
|
286
|
+
The most useful case for increasing the delay of an already delayed job is to batch together work based on multiple events. For example, if you wanted to send a notification email to a user when an event triggers but didn't want to send 10 emails if many events happened within a short period, you could use this technique to delay the noficication email until no events have triggered for a period of time. This way you could send 1 email containing the 10 notifications once no events have triggered for 2 minutes. You could implement this like so:
|
287
|
+
|
288
|
+
```ruby
|
289
|
+
# Send a notification when an event is created.
|
290
|
+
# app/models/event.rb
|
291
|
+
after_commit on: :create do
|
292
|
+
Resque.delay_or_enqueue_in(2.minutes, SendNotifications, :user_id => user.id)
|
293
|
+
end
|
294
|
+
```
|
295
|
+
|
296
|
+
When the first event is created a job will be scheduled to send unsent notifications to the associated user. If another event is created within the 2 minute window, the timer will be reset to 2 minutes. This will continue as long as new events are created for the specific user before the 2 minute timer expires. Once the timer expires and the job is scheduled any new events that are created will schedule a new job and start the process over. By adjusting the window you can tweak the trade-off between sending notification emails quickly after an event happens and sending fewer emails.
|
297
|
+
|
298
|
+
Read more in the [original PR](https://github.com/resque/resque-scheduler/pull/645)
|
299
|
+
|
300
|
+
|
248
301
|
### Scheduled Jobs (Recurring Jobs)
|
249
302
|
|
250
303
|
Scheduled (or recurring) jobs are logically no different than a standard cron
|
@@ -290,6 +343,18 @@ clear_leaderboards_contributors:
|
|
290
343
|
description: "This job resets the weekly leaderboard for contributions"
|
291
344
|
```
|
292
345
|
|
346
|
+
If you would like to setup a job that is executed manually you can configure like this in your YAML file.
|
347
|
+
|
348
|
+
```yaml
|
349
|
+
ImportOrdersManual:
|
350
|
+
custom_job_class: 'AmazonMws::ImportOrdersJob'
|
351
|
+
never: "* * * * *"
|
352
|
+
queue: high
|
353
|
+
description: "This is a manual job for importing orders."
|
354
|
+
args:
|
355
|
+
days_in_arrears: 7
|
356
|
+
```
|
357
|
+
|
293
358
|
The queue value is optional, but if left unspecified resque-scheduler will
|
294
359
|
attempt to get the queue from the job class, which means it needs to be
|
295
360
|
defined. If you're getting "uninitialized constant" errors, you probably
|
@@ -321,11 +386,43 @@ seconds past the minute).
|
|
321
386
|
A big shout out to [rufus-scheduler](http://github.com/jmettraux/rufus-scheduler)
|
322
387
|
for handling the heavy lifting of the actual scheduling engine.
|
323
388
|
|
389
|
+
##### Queue with parameters
|
390
|
+
|
391
|
+
It's possible to specify parameters, that must be given by the user when they manually queue the job. To enable this feature add `parameters` key to scheduled job definition.
|
392
|
+
|
393
|
+
```yaml
|
394
|
+
queue_documents_for_indexing:
|
395
|
+
cron: "0 0 * * *"
|
396
|
+
class: "QueueDocuments"
|
397
|
+
queue: high
|
398
|
+
args:
|
399
|
+
foo: "bar"
|
400
|
+
a: "b"
|
401
|
+
parameters:
|
402
|
+
foo:
|
403
|
+
description: "value of foo"
|
404
|
+
default: "baz"
|
405
|
+
|
406
|
+
description: "This job queues all content for indexing in solr"
|
407
|
+
```
|
408
|
+
|
409
|
+
One can use following options for each parameter:
|
410
|
+
* description - tooltip to be shown next to the parameter input
|
411
|
+
* default - prefilled value in the parameter input
|
412
|
+
|
413
|
+
**NOTE**: When sheduling the job, parameters are merged into job args. Assuming the example above and default parametr value, the job will be run with the following args:
|
414
|
+
|
415
|
+
```ruby
|
416
|
+
{"foo"=>"baz", "a"=>"b"}
|
417
|
+
```
|
418
|
+
|
419
|
+
**NOTE**: If user leaves the parameter value empty, it'll be sent as empty string.
|
420
|
+
|
324
421
|
#### Dynamic schedules
|
325
422
|
|
326
423
|
Dynamic schedules are programmatically set on a running `resque-scheduler`.
|
327
|
-
|
328
|
-
when setting schedules.
|
424
|
+
Most [rufus-scheduler](http://github.com/jmettraux/rufus-scheduler) options are supported
|
425
|
+
when setting schedules. Specifically the `overlap` option will not work.
|
329
426
|
|
330
427
|
Dynamic schedules are not enabled by default. To be able to dynamically set schedules, you
|
331
428
|
must pass the following to `resque-scheduler` initialization (see *Installation* above for a more complete example):
|
@@ -382,18 +479,17 @@ Resque.set_schedule(name, config)
|
|
382
479
|
|
383
480
|
#### Time zones
|
384
481
|
|
385
|
-
|
386
|
-
rather than the `config.time_zone` specified in Rails.
|
387
|
-
|
482
|
+
If you use the cron syntax, by default it is interpreted in the server time zone.
|
388
483
|
You can explicitly specify the time zone that rufus-scheduler will use:
|
389
|
-
|
390
484
|
```yaml
|
391
485
|
cron: "30 6 * * 1 Europe/Stockholm"
|
392
486
|
```
|
393
487
|
|
394
|
-
|
395
|
-
|
396
|
-
|
488
|
+
##### Rails
|
489
|
+
In Rails, `config.time_zone` will be used to determine the time zone for `resque-scheduler`.
|
490
|
+
|
491
|
+
Note that `config.time_zone` allows for a shorthand (e.g. "Stockholm")
|
492
|
+
that rufus-scheduler does not accept, so make sure it's the right format, e.g. with:
|
397
493
|
|
398
494
|
```ruby
|
399
495
|
ActiveSupport::TimeZone.find_tzinfo(Rails.configuration.time_zone).name
|
@@ -502,7 +598,7 @@ RESQUE_SCHEDULER_MASTER_LOCK_PREFIX=MyApp: rake resque:scheduler
|
|
502
598
|
|
503
599
|
### resque-web Additions
|
504
600
|
|
505
|
-
Resque-scheduler also adds
|
601
|
+
Resque-scheduler also adds two tabs to the resque-web UI. One is for viewing
|
506
602
|
(and manually queueing) the schedule and one is for viewing pending jobs in
|
507
603
|
the delayed queue.
|
508
604
|
|
@@ -536,11 +632,7 @@ require 'resque/scheduler/server'
|
|
536
632
|
|
537
633
|
That should make the scheduler tabs show up in `resque-web`.
|
538
634
|
|
539
|
-
|
540
|
-
|
541
|
-
As of resque-scheduler 2.0.0, it's no longer necessary to have the resque-web
|
542
|
-
process aware of the schedule because it reads it from redis. But prior to
|
543
|
-
2.0, you'll want to make sure you load the schedule in this file as well.
|
635
|
+
You'll want to make sure you load the schedule in this file as well.
|
544
636
|
Something like this:
|
545
637
|
|
546
638
|
```ruby
|
@@ -554,7 +646,7 @@ Now make sure you're passing that file to resque-web like so:
|
|
554
646
|
|
555
647
|
### Running in the background
|
556
648
|
|
557
|
-
|
649
|
+
There are scenarios where it's helpful for
|
558
650
|
the resque worker to run itself in the background (usually in combination with
|
559
651
|
PIDFILE). Use the BACKGROUND option so that rake will return as soon as the
|
560
652
|
worker is started.
|
@@ -620,7 +712,7 @@ that happens on Travis CI and Appveyor:
|
|
620
712
|
bundle install
|
621
713
|
|
622
714
|
# Make sure tests are green before you change stuff
|
623
|
-
bundle exec rake
|
715
|
+
bundle exec rubocop && bundle exec rake
|
624
716
|
# Change stuff
|
625
717
|
# Repeat
|
626
718
|
```
|
data/Rakefile
CHANGED
@@ -1,13 +1,9 @@
|
|
1
1
|
# vim:fileencoding=utf-8
|
2
2
|
require 'bundler/gem_tasks'
|
3
3
|
require 'rake/testtask'
|
4
|
-
require 'rubocop/rake_task'
|
5
4
|
require 'yard'
|
6
5
|
|
7
|
-
task default:
|
8
|
-
task default: [:test] if RUBY_PLATFORM =~ /java/
|
9
|
-
|
10
|
-
RuboCop::RakeTask.new
|
6
|
+
task default: :test
|
11
7
|
|
12
8
|
Rake::TestTask.new do |t|
|
13
9
|
t.libs << 'test'
|
@@ -16,6 +12,7 @@ Rake::TestTask.new do |t|
|
|
16
12
|
o << "--seed #{ENV['SEED']} " if ENV['SEED']
|
17
13
|
o << '--verbose ' if ENV['VERBOSE']
|
18
14
|
end
|
15
|
+
t.warning = false
|
19
16
|
end
|
20
17
|
|
21
18
|
YARD::Rake::YardocTask.new
|
data/lib/resque/scheduler/cli.rb
CHANGED
@@ -8,13 +8,18 @@ module Resque
|
|
8
8
|
yield self
|
9
9
|
end
|
10
10
|
|
11
|
+
attr_writer :environment
|
12
|
+
def environment
|
13
|
+
@environment ||= ENV
|
14
|
+
end
|
15
|
+
|
11
16
|
# Used in `#load_schedule_job`
|
12
17
|
attr_writer :env
|
13
18
|
|
14
19
|
def env
|
15
20
|
return @env if @env
|
16
21
|
@env ||= Rails.env if defined?(Rails) && Rails.respond_to?(:env)
|
17
|
-
@env ||=
|
22
|
+
@env ||= environment['RAILS_ENV']
|
18
23
|
@env
|
19
24
|
end
|
20
25
|
|
@@ -22,42 +27,42 @@ module Resque
|
|
22
27
|
attr_writer :verbose
|
23
28
|
|
24
29
|
def verbose
|
25
|
-
@verbose ||=
|
30
|
+
@verbose ||= to_bool(environment['VERBOSE'])
|
26
31
|
end
|
27
32
|
|
28
33
|
# If set, produces no output
|
29
34
|
attr_writer :quiet
|
30
35
|
|
31
36
|
def quiet
|
32
|
-
@quiet ||=
|
37
|
+
@quiet ||= to_bool(environment['QUIET'])
|
33
38
|
end
|
34
39
|
|
35
40
|
# If set, will write messages to the file
|
36
41
|
attr_writer :logfile
|
37
42
|
|
38
43
|
def logfile
|
39
|
-
@logfile ||=
|
44
|
+
@logfile ||= environment['LOGFILE']
|
40
45
|
end
|
41
46
|
|
42
47
|
# Sets whether to log in 'text' or 'json'
|
43
48
|
attr_writer :logformat
|
44
49
|
|
45
50
|
def logformat
|
46
|
-
@logformat ||=
|
51
|
+
@logformat ||= environment['LOGFORMAT']
|
47
52
|
end
|
48
53
|
|
49
54
|
# If set, will try to update the schedule in the loop
|
50
55
|
attr_writer :dynamic
|
51
56
|
|
52
57
|
def dynamic
|
53
|
-
@dynamic ||=
|
58
|
+
@dynamic ||= to_bool(environment['DYNAMIC_SCHEDULE'])
|
54
59
|
end
|
55
60
|
|
56
61
|
# If set, will append the app name to procline
|
57
62
|
attr_writer :app_name
|
58
63
|
|
59
64
|
def app_name
|
60
|
-
@app_name ||=
|
65
|
+
@app_name ||= environment['APP_NAME']
|
61
66
|
end
|
62
67
|
|
63
68
|
# Amount of time in seconds to sleep between polls of the delayed
|
@@ -66,7 +71,25 @@ module Resque
|
|
66
71
|
|
67
72
|
def poll_sleep_amount
|
68
73
|
@poll_sleep_amount ||=
|
69
|
-
Float(
|
74
|
+
Float(environment.fetch('RESQUE_SCHEDULER_INTERVAL', '5'))
|
75
|
+
end
|
76
|
+
|
77
|
+
private
|
78
|
+
|
79
|
+
# Copied from https://github.com/rails/rails/blob/main/activemodel/lib/active_model/type/boolean.rb#L17
|
80
|
+
TRUE_VALUES = [
|
81
|
+
true, 1,
|
82
|
+
'1', :'1',
|
83
|
+
't', :t,
|
84
|
+
'T', :T,
|
85
|
+
'true', :true,
|
86
|
+
'TRUE', :TRUE,
|
87
|
+
'on', :on,
|
88
|
+
'ON', :ON
|
89
|
+
].to_set.freeze
|
90
|
+
|
91
|
+
def to_bool(value)
|
92
|
+
TRUE_VALUES.include?(value)
|
70
93
|
end
|
71
94
|
end
|
72
95
|
end
|
@@ -24,7 +24,7 @@ module Resque
|
|
24
24
|
def enqueue_at_with_queue(queue, timestamp, klass, *args)
|
25
25
|
return false unless plugin.run_before_schedule_hooks(klass, *args)
|
26
26
|
|
27
|
-
if Resque.inline? || timestamp.to_i
|
27
|
+
if Resque.inline? || timestamp.to_i <= Time.now.to_i
|
28
28
|
# Just create the job and let resque perform it right away with
|
29
29
|
# inline. If the class is a custom job class, call self#scheduled
|
30
30
|
# on it. This allows you to do things like
|
@@ -33,7 +33,7 @@ module Resque
|
|
33
33
|
if klass.respond_to?(:scheduled)
|
34
34
|
klass.scheduled(queue, klass.to_s, *args)
|
35
35
|
else
|
36
|
-
Resque
|
36
|
+
Resque.enqueue_to(queue, klass, *args)
|
37
37
|
end
|
38
38
|
else
|
39
39
|
delayed_push(timestamp, job_to_hash_with_queue(queue, klass, args))
|
@@ -45,6 +45,9 @@ module Resque
|
|
45
45
|
# Identical to enqueue_at but takes number_of_seconds_from_now
|
46
46
|
# instead of a timestamp.
|
47
47
|
def enqueue_in(number_of_seconds_from_now, klass, *args)
|
48
|
+
unless number_of_seconds_from_now.is_a?(Numeric)
|
49
|
+
raise ArgumentError, 'Please supply a numeric number of seconds'
|
50
|
+
end
|
48
51
|
enqueue_at(Time.now + number_of_seconds_from_now, klass, *args)
|
49
52
|
end
|
50
53
|
|
@@ -53,14 +56,35 @@ module Resque
|
|
53
56
|
# number of seconds has passed.
|
54
57
|
def enqueue_in_with_queue(queue, number_of_seconds_from_now,
|
55
58
|
klass, *args)
|
59
|
+
unless number_of_seconds_from_now.is_a?(Numeric)
|
60
|
+
raise ArgumentError, 'Please supply a numeric number of seconds'
|
61
|
+
end
|
56
62
|
enqueue_at_with_queue(queue, Time.now + number_of_seconds_from_now,
|
57
63
|
klass, *args)
|
58
64
|
end
|
59
65
|
|
66
|
+
# Update the delayed timestamp of any matching delayed jobs or enqueue a
|
67
|
+
# new job if no matching jobs are found. Returns the number of delayed or
|
68
|
+
# enqueued jobs.
|
69
|
+
def delay_or_enqueue_at(timestamp, klass, *args)
|
70
|
+
count = remove_delayed(klass, *args)
|
71
|
+
count = 1 if count == 0
|
72
|
+
|
73
|
+
count.times do
|
74
|
+
enqueue_at(timestamp, klass, *args)
|
75
|
+
end
|
76
|
+
end
|
77
|
+
|
78
|
+
# Identical to +delay_or_enqueue_at+, except it takes
|
79
|
+
# number_of_seconds_from_now instead of a timestamp
|
80
|
+
def delay_or_enqueue_in(number_of_seconds_from_now, klass, *args)
|
81
|
+
delay_or_enqueue_at(Time.now + number_of_seconds_from_now, klass, *args)
|
82
|
+
end
|
83
|
+
|
60
84
|
# Used internally to stuff the item into the schedule sorted list.
|
61
|
-
# +timestamp+ can be either in seconds or a datetime object
|
62
|
-
#
|
63
|
-
# that time, else false
|
85
|
+
# +timestamp+ can be either in seconds or a datetime object. The
|
86
|
+
# insertion time complexity is O(log(n)). Returns true if it's
|
87
|
+
# the first job to be scheduled at that time, else false.
|
64
88
|
def delayed_push(timestamp, item)
|
65
89
|
# First add this item to the list for this timestamp
|
66
90
|
redis.rpush("delayed:#{timestamp.to_i}", encode(item))
|
@@ -82,6 +106,7 @@ module Resque
|
|
82
106
|
end
|
83
107
|
|
84
108
|
# Returns the size of the delayed queue schedule
|
109
|
+
# this does not represent the number of items in the queue to be scheduled
|
85
110
|
def delayed_queue_schedule_size
|
86
111
|
redis.zcard :delayed_queue_schedule
|
87
112
|
end
|
@@ -128,10 +153,7 @@ module Resque
|
|
128
153
|
Array(redis.zrange(:delayed_queue_schedule, 0, -1)).each do |item|
|
129
154
|
key = "delayed:#{item}"
|
130
155
|
items = redis.lrange(key, 0, -1)
|
131
|
-
redis.
|
132
|
-
items.each { |ts_item| redis.del("timestamps:#{ts_item}") }
|
133
|
-
end
|
134
|
-
redis.del key
|
156
|
+
redis.del(key, items.map { |ts_item| "timestamps:#{ts_item}" })
|
135
157
|
end
|
136
158
|
|
137
159
|
redis.del :delayed_queue_schedule
|
@@ -143,6 +165,11 @@ module Resque
|
|
143
165
|
remove_delayed_job(search)
|
144
166
|
end
|
145
167
|
|
168
|
+
def remove_delayed_in_queue(klass, queue, *args)
|
169
|
+
search = encode(job_to_hash_with_queue(queue, klass, args))
|
170
|
+
remove_delayed_job(search)
|
171
|
+
end
|
172
|
+
|
146
173
|
# Given an encoded item, enqueue it now
|
147
174
|
def enqueue_delayed(klass, *args)
|
148
175
|
hash = job_to_hash(klass, args)
|
@@ -151,6 +178,13 @@ module Resque
|
|
151
178
|
end
|
152
179
|
end
|
153
180
|
|
181
|
+
def enqueue_delayed_with_queue(klass, queue, *args)
|
182
|
+
hash = job_to_hash_with_queue(queue, klass, args)
|
183
|
+
remove_delayed_in_queue(klass, queue, *args).times do
|
184
|
+
Resque::Scheduler.enqueue_from_config(hash)
|
185
|
+
end
|
186
|
+
end
|
187
|
+
|
154
188
|
# Given a block, remove jobs that return true from a block
|
155
189
|
#
|
156
190
|
# This allows for removal of delayed jobs that have arguments matching
|
@@ -175,7 +209,15 @@ module Resque
|
|
175
209
|
found_jobs.reduce(0) do |sum, encoded_job|
|
176
210
|
decoded_job = decode(encoded_job)
|
177
211
|
klass = Util.constantize(decoded_job['class'])
|
178
|
-
|
212
|
+
queue = decoded_job['queue']
|
213
|
+
|
214
|
+
if queue
|
215
|
+
jobs_queued = enqueue_delayed_with_queue(klass, queue, *decoded_job['args'])
|
216
|
+
else
|
217
|
+
jobs_queued = enqueue_delayed(klass, *decoded_job['args'])
|
218
|
+
end
|
219
|
+
|
220
|
+
jobs_queued + sum
|
179
221
|
end
|
180
222
|
end
|
181
223
|
|
@@ -190,9 +232,9 @@ module Resque
|
|
190
232
|
|
191
233
|
# Beyond 100 there's almost no improvement in speed
|
192
234
|
found = timestamps.each_slice(100).map do |ts_group|
|
193
|
-
jobs = redis.pipelined do |
|
235
|
+
jobs = redis.pipelined do |pipeline|
|
194
236
|
ts_group.each do |ts|
|
195
|
-
|
237
|
+
pipeline.lrange("delayed:#{ts}", 0, -1)
|
196
238
|
end
|
197
239
|
end
|
198
240
|
|
@@ -265,18 +307,23 @@ module Resque
|
|
265
307
|
{ class: klass.to_s, args: args, queue: queue }
|
266
308
|
end
|
267
309
|
|
310
|
+
# Removes a job from the queue, but not modify the timestamp schedule. This method
|
311
|
+
# will not effect the output of `delayed_queue_schedule_size`
|
268
312
|
def remove_delayed_job(encoded_job)
|
269
313
|
return 0 if Resque.inline?
|
270
314
|
|
271
315
|
timestamps = redis.smembers("timestamps:#{encoded_job}")
|
272
316
|
|
273
|
-
replies = redis.pipelined do
|
317
|
+
replies = redis.pipelined do |pipeline|
|
274
318
|
timestamps.each do |key|
|
275
|
-
|
276
|
-
|
319
|
+
pipeline.lrem(key, 0, encoded_job)
|
320
|
+
pipeline.srem("timestamps:#{encoded_job}", key)
|
277
321
|
end
|
278
322
|
end
|
279
323
|
|
324
|
+
# timestamp key is not removed from the schedule, this is done later
|
325
|
+
# by the scheduler loop
|
326
|
+
|
280
327
|
return 0 if replies.nil? || replies.empty?
|
281
328
|
replies.each_slice(2).map(&:first).inject(:+)
|
282
329
|
end
|
@@ -287,9 +334,9 @@ module Resque
|
|
287
334
|
redis.watch(key) do
|
288
335
|
if redis.llen(key).to_i == 0
|
289
336
|
# If the list is empty, remove it.
|
290
|
-
redis.multi do
|
291
|
-
|
292
|
-
|
337
|
+
redis.multi do |transaction|
|
338
|
+
transaction.del(key)
|
339
|
+
transaction.zrem(:delayed_queue_schedule, timestamp.to_i)
|
293
340
|
end
|
294
341
|
else
|
295
342
|
redis.redis.unwatch
|
data/lib/resque/scheduler/env.rb
CHANGED
@@ -38,12 +38,8 @@ module Resque
|
|
38
38
|
true
|
39
39
|
end
|
40
40
|
|
41
|
-
unless Process.respond_to?('daemon')
|
42
|
-
abort 'background option is set, which requires ruby >= 1.9'
|
43
|
-
end
|
44
|
-
|
45
41
|
Process.daemon(true, !Resque::Scheduler.quiet)
|
46
|
-
Resque.redis.
|
42
|
+
Resque.redis.reconnect
|
47
43
|
end
|
48
44
|
|
49
45
|
def setup_pid_file
|
@@ -64,13 +60,13 @@ module Resque
|
|
64
60
|
|
65
61
|
c.dynamic = !!options[:dynamic] if options.key?(:dynamic)
|
66
62
|
|
67
|
-
c.env = options[:env] if options.key(:env)
|
63
|
+
c.env = options[:env] if options.key?(:env)
|
68
64
|
|
69
65
|
c.logfile = options[:logfile] if options.key?(:logfile)
|
70
66
|
|
71
67
|
c.logformat = options[:logformat] if options.key?(:logformat)
|
72
68
|
|
73
|
-
if psleep = options[:poll_sleep_amount] && !psleep.nil?
|
69
|
+
if (psleep = options[:poll_sleep_amount]) && !psleep.nil?
|
74
70
|
c.poll_sleep_amount = Float(psleep)
|
75
71
|
end
|
76
72
|
|
@@ -6,19 +6,11 @@ module Resque
|
|
6
6
|
module Lock
|
7
7
|
class Resilient < Base
|
8
8
|
def acquire!
|
9
|
-
|
10
|
-
acquire_sha,
|
11
|
-
keys: [key],
|
12
|
-
argv: [value]
|
13
|
-
).to_i == 1
|
9
|
+
evalsha(:acquire, [key], [value]).to_i == 1
|
14
10
|
end
|
15
11
|
|
16
12
|
def locked?
|
17
|
-
|
18
|
-
locked_sha,
|
19
|
-
keys: [key],
|
20
|
-
argv: [value]
|
21
|
-
).to_i == 1
|
13
|
+
evalsha(:locked, [key], [value]).to_i == 1
|
22
14
|
end
|
23
15
|
|
24
16
|
def timeout=(seconds)
|
@@ -32,11 +24,26 @@ module Resque
|
|
32
24
|
|
33
25
|
private
|
34
26
|
|
27
|
+
def evalsha(script, keys, argv, refresh: false)
|
28
|
+
sha_method_name = "#{script}_sha"
|
29
|
+
Resque.redis.evalsha(
|
30
|
+
send(sha_method_name, refresh),
|
31
|
+
keys: keys,
|
32
|
+
argv: argv
|
33
|
+
)
|
34
|
+
rescue Redis::CommandError => e
|
35
|
+
if e.message =~ /NOSCRIPT/
|
36
|
+
refresh = true
|
37
|
+
retry
|
38
|
+
end
|
39
|
+
raise
|
40
|
+
end
|
41
|
+
|
35
42
|
def locked_sha(refresh = false)
|
36
43
|
@locked_sha = nil if refresh
|
37
44
|
|
38
45
|
@locked_sha ||=
|
39
|
-
Resque.redis.script(:load, <<-EOF.gsub(/^ {14}/, ''))
|
46
|
+
Resque.data_store.redis.script(:load, <<-EOF.gsub(/^ {14}/, ''))
|
40
47
|
if redis.call('GET', KEYS[1]) == ARGV[1]
|
41
48
|
then
|
42
49
|
redis.call('EXPIRE', KEYS[1], #{timeout})
|
@@ -55,7 +62,7 @@ module Resque
|
|
55
62
|
@acquire_sha = nil if refresh
|
56
63
|
|
57
64
|
@acquire_sha ||=
|
58
|
-
Resque.redis.script(:load, <<-EOF.gsub(/^ {14}/, ''))
|
65
|
+
Resque.data_store.redis.script(:load, <<-EOF.gsub(/^ {14}/, ''))
|
59
66
|
if redis.call('SETNX', KEYS[1], ARGV[1]) == 1
|
60
67
|
then
|
61
68
|
redis.call('EXPIRE', KEYS[1], #{timeout})
|
@@ -2,7 +2,7 @@
|
|
2
2
|
|
3
3
|
# ### Locking the scheduler process
|
4
4
|
#
|
5
|
-
# There are two places in resque-scheduler that need to be
|
5
|
+
# There are two places in resque-scheduler that need to be synchronized in order
|
6
6
|
# to be able to run redundant scheduler processes while ensuring jobs don't get
|
7
7
|
# queued multiple times when the master process changes.
|
8
8
|
#
|
@@ -76,7 +76,7 @@ module Resque
|
|
76
76
|
|
77
77
|
def release_master_lock
|
78
78
|
master_lock.release
|
79
|
-
rescue
|
79
|
+
rescue *INTERMITTENT_ERRORS
|
80
80
|
@master_lock = nil
|
81
81
|
end
|
82
82
|
|
@@ -97,7 +97,7 @@ module Resque
|
|
97
97
|
end
|
98
98
|
|
99
99
|
def redis_master_version
|
100
|
-
Resque.redis.info['redis_version'].to_f
|
100
|
+
Resque.data_store.redis.info['redis_version'].to_f
|
101
101
|
end
|
102
102
|
end
|
103
103
|
end
|
@@ -36,7 +36,7 @@ module Resque
|
|
36
36
|
# :args can be any yaml which will be converted to a ruby literal and
|
37
37
|
# passed in a params. (optional)
|
38
38
|
#
|
39
|
-
# :
|
39
|
+
# :rails_env is the list of envs where the job gets loaded. Envs are
|
40
40
|
# comma separated (optional)
|
41
41
|
#
|
42
42
|
# :description is just that, a description of the job (optional). If
|
@@ -101,12 +101,13 @@ module Resque
|
|
101
101
|
end
|
102
102
|
|
103
103
|
# remove a given schedule by name
|
104
|
-
|
104
|
+
# Preventing a reload is optional and available to batch operations
|
105
|
+
def remove_schedule(name, reload = true)
|
105
106
|
non_persistent_schedules.delete(name)
|
106
107
|
redis.hdel(:persistent_schedules, name)
|
107
108
|
redis.sadd(:schedules_changed, name)
|
108
109
|
|
109
|
-
reload_schedule!
|
110
|
+
reload_schedule! if reload
|
110
111
|
end
|
111
112
|
|
112
113
|
private
|