resque-scheduler 2.2.0 → 4.10.2
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +7 -0
- data/.github/dependabot.yml +12 -0
- data/.github/funding.yml +4 -0
- data/.github/workflows/codeql-analysis.yml +59 -0
- data/.github/workflows/rubocop.yml +27 -0
- data/.github/workflows/ruby.yml +81 -0
- data/AUTHORS.md +31 -0
- data/CHANGELOG.md +539 -0
- data/CODE_OF_CONDUCT.md +74 -0
- data/Gemfile +26 -2
- data/LICENSE +1 -1
- data/README.md +423 -91
- data/Rakefile +12 -35
- data/exe/resque-scheduler +5 -0
- data/lib/resque/scheduler/cli.rb +147 -0
- data/lib/resque/scheduler/configuration.rb +102 -0
- data/lib/resque/scheduler/delaying_extensions.rb +371 -0
- data/lib/resque/scheduler/env.rb +85 -0
- data/lib/resque/scheduler/extension.rb +13 -0
- data/lib/resque/scheduler/failure_handler.rb +11 -0
- data/lib/resque/scheduler/lock/base.rb +12 -3
- data/lib/resque/scheduler/lock/basic.rb +4 -5
- data/lib/resque/scheduler/lock/resilient.rb +52 -43
- data/lib/resque/scheduler/lock.rb +2 -1
- data/lib/resque/scheduler/locking.rb +104 -0
- data/lib/resque/scheduler/logger_builder.rb +83 -0
- data/lib/resque/scheduler/plugin.rb +31 -0
- data/lib/resque/scheduler/scheduling_extensions.rb +142 -0
- data/lib/{resque_scheduler → resque/scheduler}/server/views/delayed.erb +23 -8
- data/lib/resque/scheduler/server/views/delayed_schedules.erb +20 -0
- data/lib/{resque_scheduler → resque/scheduler}/server/views/delayed_timestamp.erb +1 -1
- data/lib/resque/scheduler/server/views/scheduler.erb +58 -0
- data/lib/resque/scheduler/server/views/search.erb +69 -0
- data/lib/resque/scheduler/server/views/search_form.erb +4 -0
- data/lib/resque/scheduler/server.rb +268 -0
- data/lib/resque/scheduler/signal_handling.rb +40 -0
- data/lib/resque/scheduler/tasks.rb +25 -0
- data/lib/resque/scheduler/util.rb +39 -0
- data/lib/resque/scheduler/version.rb +7 -0
- data/lib/resque/scheduler.rb +333 -149
- data/lib/resque-scheduler.rb +4 -0
- data/resque-scheduler.gemspec +56 -20
- metadata +205 -104
- data/.gitignore +0 -8
- data/.rubocop.yml +0 -120
- data/.travis.yml +0 -10
- data/HISTORY.md +0 -180
- data/lib/resque/scheduler_locking.rb +0 -90
- data/lib/resque_scheduler/logger_builder.rb +0 -51
- data/lib/resque_scheduler/plugin.rb +0 -25
- data/lib/resque_scheduler/server/views/scheduler.erb +0 -44
- data/lib/resque_scheduler/server.rb +0 -92
- data/lib/resque_scheduler/tasks.rb +0 -40
- data/lib/resque_scheduler/version.rb +0 -3
- data/lib/resque_scheduler.rb +0 -355
- data/script/migrate_to_timestamps_set.rb +0 -14
- data/tasks/resque_scheduler.rake +0 -2
- data/test/delayed_queue_test.rb +0 -383
- data/test/redis-test.conf +0 -108
- data/test/resque-web_test.rb +0 -116
- data/test/scheduler_args_test.rb +0 -156
- data/test/scheduler_hooks_test.rb +0 -23
- data/test/scheduler_locking_test.rb +0 -180
- data/test/scheduler_setup_test.rb +0 -59
- data/test/scheduler_test.rb +0 -256
- data/test/support/redis_instance.rb +0 -129
- data/test/test_helper.rb +0 -92
- /data/lib/{resque_scheduler → resque/scheduler}/server/views/requeue-params.erb +0 -0
data/README.md
CHANGED
@@ -1,24 +1,17 @@
|
|
1
1
|
resque-scheduler
|
2
2
|
================
|
3
3
|
|
4
|
-
|
5
|
-
[![Gem Version](https://badge.fury.io/rb/resque-scheduler.
|
6
|
-
[![
|
4
|
+
|
5
|
+
[![Gem Version](https://badge.fury.io/rb/resque-scheduler.svg)](https://badge.fury.io/rb/resque-scheduler)
|
6
|
+
[![Ruby specs](https://github.com/resque/resque-scheduler/actions/workflows/ruby.yml/badge.svg)](https://github.com/resque/resque-scheduler/actions)
|
7
|
+
[![Code Climate](https://codeclimate.com/github/resque/resque-scheduler/badges/gpa.svg)](https://codeclimate.com/github/resque/resque-scheduler)
|
7
8
|
|
8
9
|
### Description
|
9
10
|
|
10
11
|
Resque-scheduler is an extension to [Resque](http://github.com/resque/resque)
|
11
12
|
that adds support for queueing items in the future.
|
12
13
|
|
13
|
-
|
14
|
-
|
15
|
-
| resque-scheduler version | required redis version|
|
16
|
-
|:-------------------------|----------------------:|
|
17
|
-
| >= 2.0.0 | >= 2.2.0 |
|
18
|
-
| >= 0.0.1 | >= 1.3 |
|
19
|
-
|
20
|
-
|
21
|
-
Job scheduling is supported in two different way: Recurring (scheduled) and
|
14
|
+
Job scheduling is supported in two different ways: Recurring (scheduled) and
|
22
15
|
Delayed.
|
23
16
|
|
24
17
|
Scheduled jobs are like cron jobs, recurring on a regular basis. Delayed
|
@@ -26,15 +19,16 @@ jobs are resque jobs that you want to run at some point in the future.
|
|
26
19
|
The syntax is pretty explanatory:
|
27
20
|
|
28
21
|
```ruby
|
29
|
-
Resque.enqueue_in(5.days, SendFollowupEmail) #
|
22
|
+
Resque.enqueue_in(5.days, SendFollowupEmail, argument) # runs a job in 5 days, calling SendFollowupEmail.perform(argument)
|
30
23
|
# or
|
31
|
-
Resque.enqueue_at(5.days.from_now, SomeJob) #
|
24
|
+
Resque.enqueue_at(5.days.from_now, SomeJob, argument) # runs a job at a specific time, calling SomeJob.perform(argument)
|
32
25
|
```
|
33
26
|
|
34
27
|
### Documentation
|
35
28
|
|
36
|
-
This README covers what most people need to know. If you're looking
|
37
|
-
details on individual methods, you might want to try the
|
29
|
+
This `README` covers what most people need to know. If you're looking
|
30
|
+
for details on individual methods, you might want to try the
|
31
|
+
[rdoc](http://rdoc.info/github/resque/resque-scheduler/master/frames).
|
38
32
|
|
39
33
|
### Installation
|
40
34
|
|
@@ -42,39 +36,40 @@ To install:
|
|
42
36
|
|
43
37
|
gem install resque-scheduler
|
44
38
|
|
45
|
-
If you use a Gemfile
|
39
|
+
If you use a Gemfile:
|
46
40
|
|
47
41
|
```ruby
|
48
|
-
gem 'resque-scheduler'
|
42
|
+
gem 'resque-scheduler'
|
49
43
|
```
|
50
44
|
|
51
45
|
Adding the resque:scheduler rake task:
|
52
46
|
|
53
47
|
```ruby
|
54
|
-
require '
|
48
|
+
require 'resque/scheduler/tasks'
|
55
49
|
```
|
56
50
|
|
57
|
-
|
58
|
-
|
59
|
-
|
60
|
-
|
61
|
-
|
62
|
-
`resque
|
63
|
-
to know.
|
51
|
+
### Rake integration
|
52
|
+
|
53
|
+
By default, `resque-scheduler` depends on the "resque:setup" rake task.
|
54
|
+
Since you probably already have this task, lets just put our
|
55
|
+
configuration there. `resque-scheduler` pretty much needs to know
|
56
|
+
everything `resque` needs to know.
|
64
57
|
|
65
58
|
```ruby
|
66
59
|
# Resque tasks
|
67
60
|
require 'resque/tasks'
|
68
|
-
require '
|
61
|
+
require 'resque/scheduler/tasks'
|
69
62
|
|
70
63
|
namespace :resque do
|
71
64
|
task :setup do
|
72
65
|
require 'resque'
|
73
|
-
require 'resque_scheduler'
|
74
|
-
require 'resque/scheduler'
|
75
66
|
|
76
67
|
# you probably already have this somewhere
|
77
68
|
Resque.redis = 'localhost:6379'
|
69
|
+
end
|
70
|
+
|
71
|
+
task :setup_schedule => :setup do
|
72
|
+
require 'resque-scheduler'
|
78
73
|
|
79
74
|
# If you want to be able to dynamically change the schedule,
|
80
75
|
# uncomment this line. A dynamic schedule can be updated via the
|
@@ -82,7 +77,7 @@ namespace :resque do
|
|
82
77
|
# When dynamic is set to true, the scheduler process looks for
|
83
78
|
# schedule changes and applies them on the fly.
|
84
79
|
# Note: This feature is only available in >=2.0.0.
|
85
|
-
#Resque::Scheduler.dynamic = true
|
80
|
+
# Resque::Scheduler.dynamic = true
|
86
81
|
|
87
82
|
# The schedule doesn't need to be stored in a YAML, it just needs to
|
88
83
|
# be a hash. YAML is usually the easiest.
|
@@ -95,20 +90,78 @@ namespace :resque do
|
|
95
90
|
# So, something like this:
|
96
91
|
require 'jobs'
|
97
92
|
end
|
93
|
+
|
94
|
+
task :scheduler => :setup_schedule
|
98
95
|
end
|
99
96
|
```
|
100
97
|
|
101
|
-
The scheduler
|
102
|
-
|
103
|
-
|
104
|
-
|
98
|
+
The scheduler rake task is responsible for both queueing items from the
|
99
|
+
schedule and polling the delayed queue for items ready to be pushed on
|
100
|
+
to the work queues. For obvious reasons, this process never exits.
|
101
|
+
|
102
|
+
``` bash
|
103
|
+
rake resque:scheduler
|
104
|
+
```
|
105
|
+
|
106
|
+
or, if you want to load the environment first:
|
105
107
|
|
106
|
-
|
108
|
+
``` bash
|
109
|
+
rake environment resque:scheduler
|
110
|
+
```
|
107
111
|
|
108
|
-
|
109
|
-
|
110
|
-
|
111
|
-
|
112
|
+
|
113
|
+
### Standalone Executable
|
114
|
+
|
115
|
+
The scheduler may also be run via a standalone `resque-scheduler`
|
116
|
+
executable, which will be available once the gem is installed.
|
117
|
+
|
118
|
+
``` bash
|
119
|
+
# Get some help
|
120
|
+
resque-scheduler --help
|
121
|
+
```
|
122
|
+
|
123
|
+
The executable accepts options via option flags as well as via
|
124
|
+
[environment variables](#environment-variables).
|
125
|
+
|
126
|
+
### Environment Variables
|
127
|
+
|
128
|
+
Both the Rake task and standalone executable support the following
|
129
|
+
environment variables:
|
130
|
+
|
131
|
+
* `APP_NAME` - Application name used in procline (`$0`) (default empty)
|
132
|
+
* `BACKGROUND` - [Run in the background](#running-in-the-background) if
|
133
|
+
non-empty (via `Process.daemon`, if supported) (default `false`)
|
134
|
+
* `DYNAMIC_SCHEDULE` - Enables [dynamic scheduling](#dynamic-schedules)
|
135
|
+
if non-empty (default `false`)
|
136
|
+
* `RAILS_ENV` - Environment to use in procline (`$0`) (default empty)
|
137
|
+
* `INITIALIZER_PATH` - Path to a Ruby file that will be loaded *before*
|
138
|
+
requiring `resque` and `resque/scheduler` (default empty).
|
139
|
+
* `RESQUE_SCHEDULER_INTERVAL` - Interval in seconds for checking if a
|
140
|
+
scheduled job must run (coerced with `Kernel#Float()`) (default `5`)
|
141
|
+
* `LOGFILE` - Log file name (default empty, meaning `$stdout`)
|
142
|
+
* `LOGFORMAT` - Log output format to use (either `'text'`, `'json'` or `'logfmt'`,
|
143
|
+
default `'text'`)
|
144
|
+
* `PIDFILE` - If non-empty, write process PID to file (default empty)
|
145
|
+
* `QUIET` - Silence most output if non-empty (equivalent to a level of
|
146
|
+
`MonoLogger::FATAL`, default `false`)
|
147
|
+
* `VERBOSE` - Maximize log verbosity if non-empty (equivalent to a level
|
148
|
+
of `MonoLogger::DEBUG`, default `false`)
|
149
|
+
|
150
|
+
|
151
|
+
### Resque Pool integration
|
152
|
+
|
153
|
+
For normal work with the
|
154
|
+
[resque-pool](https://github.com/nevans/resque-pool) gem, add the
|
155
|
+
following task to wherever tasks are kept, such as
|
156
|
+
`./lib/tasks/resque.rake`:
|
157
|
+
|
158
|
+
```ruby
|
159
|
+
task 'resque:pool:setup' do
|
160
|
+
Resque::Pool.after_prefork do |job|
|
161
|
+
Resque.redis.reconnect
|
162
|
+
end
|
163
|
+
end
|
164
|
+
```
|
112
165
|
|
113
166
|
|
114
167
|
### Delayed jobs
|
@@ -117,7 +170,11 @@ Delayed jobs are one-off jobs that you want to be put into a queue at some point
|
|
117
170
|
in the future. The classic example is sending email:
|
118
171
|
|
119
172
|
```ruby
|
120
|
-
Resque.enqueue_in(
|
173
|
+
Resque.enqueue_in(
|
174
|
+
5.days,
|
175
|
+
SendFollowUpEmail,
|
176
|
+
user_id: current_user.id
|
177
|
+
)
|
121
178
|
```
|
122
179
|
|
123
180
|
This will store the job for 5 days in the resque delayed queue at which time
|
@@ -125,27 +182,36 @@ the scheduler process will pull it from the delayed queue and put it in the
|
|
125
182
|
appropriate work queue for the given job and it will be processed as soon as
|
126
183
|
a worker is available (just like any other resque job).
|
127
184
|
|
128
|
-
NOTE
|
185
|
+
**NOTE**: The job does not fire **exactly** at the time supplied. Rather, once that
|
129
186
|
time is in the past, the job moves from the delayed queue to the actual resque
|
130
|
-
work queue and will be completed as workers
|
187
|
+
work queue and will be completed as workers are free to process it.
|
131
188
|
|
132
189
|
Also supported is `Resque.enqueue_at` which takes a timestamp to queue the
|
133
190
|
job, and `Resque.enqueue_at_with_queue` which takes both a timestamp and a
|
134
|
-
queue name
|
191
|
+
queue name:
|
192
|
+
|
193
|
+
```ruby
|
194
|
+
Resque.enqueue_at_with_queue(
|
195
|
+
'queue_name',
|
196
|
+
5.days.from_now,
|
197
|
+
SendFollowUpEmail,
|
198
|
+
user_id: current_user.id
|
199
|
+
)
|
200
|
+
```
|
135
201
|
|
136
202
|
The delayed queue is stored in redis and is persisted in the same way the
|
137
203
|
standard resque jobs are persisted (redis writing to disk). Delayed jobs differ
|
138
204
|
from scheduled jobs in that if your scheduler process is down or workers are
|
139
205
|
down when a particular job is supposed to be queue, they will simply "catch up"
|
140
206
|
once they are started again. Jobs are guaranteed to run (provided they make it
|
141
|
-
into the delayed queue) after their given queue_at time has passed.
|
207
|
+
into the delayed queue) after their given `queue_at` time has passed.
|
142
208
|
|
143
209
|
One other thing to note is that insertion into the delayed queue is O(log(n))
|
144
210
|
since the jobs are stored in a redis sorted set (zset). I can't imagine this
|
145
211
|
being an issue for someone since redis is stupidly fast even at log(n), but full
|
146
212
|
disclosure is always best.
|
147
213
|
|
148
|
-
#### Removing Delayed
|
214
|
+
#### Removing Delayed Jobs
|
149
215
|
|
150
216
|
If you have the need to cancel a delayed job, you can do like so:
|
151
217
|
|
@@ -156,15 +222,103 @@ Resque.enqueue_at(5.days.from_now, SendFollowUpEmail, :user_id => current_user.i
|
|
156
222
|
Resque.remove_delayed(SendFollowUpEmail, :user_id => current_user.id)
|
157
223
|
```
|
158
224
|
|
225
|
+
If you need to cancel a delayed job based on some matching arguments, but don't wish to specify each argument from when the job was created, you can do like so:
|
226
|
+
|
227
|
+
``` ruby
|
228
|
+
# after you've enqueued a job like:
|
229
|
+
Resque.enqueue_at(5.days.from_now, SendFollowUpEmail, :account_id => current_account.id, :user_id => current_user.id)
|
230
|
+
# remove jobs matching just the account:
|
231
|
+
Resque.remove_delayed_selection { |args| args[0]['account_id'] == current_account.id }
|
232
|
+
# or remove jobs matching just the user:
|
233
|
+
Resque.remove_delayed_selection { |args| args[0]['user_id'] == current_user.id }
|
234
|
+
```
|
235
|
+
|
236
|
+
If you need to cancel a delayed job based on some matching arguments AND by which class the job is, but don't wish to specify each argument from when the job was created, you can do like so:
|
237
|
+
|
238
|
+
``` ruby
|
239
|
+
# after you've enqueued a job like:
|
240
|
+
Resque.enqueue_at(5.days.from_now, SendFollowUpEmail, :account_id => current_account.id, :user_id => current_user.id)
|
241
|
+
# remove jobs matching just the account and that were of the class SendFollowUpEmail:
|
242
|
+
Resque.remove_delayed_selection(SendFollowUpEmail) { |args| args[0]['account_id'] == current_account.id }
|
243
|
+
# or remove jobs matching just the user and that were of the class SendFollowUpEmail:
|
244
|
+
Resque.remove_delayed_selection(SendFollowUpEmail) { |args| args[0]['user_id'] == current_user.id }
|
245
|
+
```
|
246
|
+
|
247
|
+
If you need to enqueue immediately a delayed job based on some matching arguments, but don't wish to specify each argument from when the job was created, you can do like so:
|
248
|
+
|
249
|
+
``` ruby
|
250
|
+
# after you've enqueued a job like:
|
251
|
+
Resque.enqueue_at(5.days.from_now, SendFollowUpEmail, :account_id => current_account.id, :user_id => current_user.id)
|
252
|
+
# enqueue immediately jobs matching just the account:
|
253
|
+
Resque.enqueue_delayed_selection { |args| args[0]['account_id'] == current_account.id }
|
254
|
+
# or enqueue immediately jobs matching just the user:
|
255
|
+
Resque.enqueue_delayed_selection { |args| args[0]['user_id'] == current_user.id }
|
256
|
+
```
|
257
|
+
|
258
|
+
#### Updating Delayed Jobs
|
259
|
+
|
260
|
+
Previously delayed jobs may be delayed even further into the future like so:
|
261
|
+
|
262
|
+
```ruby
|
263
|
+
# after you've enqueued a job like:
|
264
|
+
Resque.enqueue_at(1.minute.from_now, SendNotifications, :user_id => current_user.id)
|
265
|
+
# delay running the job until two minutes from now
|
266
|
+
Resque.delay_or_enqueue_at(2.minutes.from_now, SendNotifications, :user_id => current_user.id)
|
267
|
+
```
|
268
|
+
|
269
|
+
You don't need to worry if a matching job has already been queued, because if no matching jobs are found a new job is created and enqueued as if you had called `enqueue_at`. This means you don't need any special conditionals to know if a job has already been queued. You simply create the job like so:
|
270
|
+
|
271
|
+
```ruby
|
272
|
+
Resque.delay_or_enqueue_at(1.minute.from_now, SendNotifications, :user_id => current_user.id)
|
273
|
+
```
|
274
|
+
|
275
|
+
If multiple matching jobs are found, all of the matching jobs will be updated to have the same timestamp even if their original timestamps were not the same.
|
276
|
+
|
277
|
+
```ruby
|
278
|
+
# enqueue multiple jobs with different delay timestamps
|
279
|
+
Resque.enqueue_at(1.minute.from_now, SendNotifications, :user_id => current_user.id)
|
280
|
+
Resque.enqueue_at(2.minutes.from_now, SendNotifications, :user_id => current_user.id)
|
281
|
+
|
282
|
+
# delay running the two jobs until 5 minutes from now
|
283
|
+
Resque.delay_or_enqueue_at(5.minutes.from_now, SendNotifications, :user_id => current_user.id)
|
284
|
+
```
|
285
|
+
|
286
|
+
The most useful case for increasing the delay of an already delayed job is to batch together work based on multiple events. For example, if you wanted to send a notification email to a user when an event triggers but didn't want to send 10 emails if many events happened within a short period, you could use this technique to delay the noficication email until no events have triggered for a period of time. This way you could send 1 email containing the 10 notifications once no events have triggered for 2 minutes. You could implement this like so:
|
287
|
+
|
288
|
+
```ruby
|
289
|
+
# Send a notification when an event is created.
|
290
|
+
# app/models/event.rb
|
291
|
+
after_commit on: :create do
|
292
|
+
Resque.delay_or_enqueue_in(2.minutes, SendNotifications, :user_id => user.id)
|
293
|
+
end
|
294
|
+
```
|
295
|
+
|
296
|
+
When the first event is created a job will be scheduled to send unsent notifications to the associated user. If another event is created within the 2 minute window, the timer will be reset to 2 minutes. This will continue as long as new events are created for the specific user before the 2 minute timer expires. Once the timer expires and the job is scheduled any new events that are created will schedule a new job and start the process over. By adjusting the window you can tweak the trade-off between sending notification emails quickly after an event happens and sending fewer emails.
|
297
|
+
|
298
|
+
Read more in the [original PR](https://github.com/resque/resque-scheduler/pull/645)
|
299
|
+
|
300
|
+
|
159
301
|
### Scheduled Jobs (Recurring Jobs)
|
160
302
|
|
161
303
|
Scheduled (or recurring) jobs are logically no different than a standard cron
|
162
|
-
job. They are jobs that run based on a
|
163
|
-
|
304
|
+
job. They are jobs that run based on a schedule which can be static or dynamic.
|
305
|
+
|
306
|
+
#### Static schedules
|
164
307
|
|
165
|
-
|
308
|
+
Static schedules are set when `resque-scheduler` starts by passing a schedule file
|
309
|
+
to `resque-scheduler` initialization like this (see *Installation* above for a more complete example):
|
310
|
+
|
311
|
+
```ruby
|
312
|
+
Resque.schedule = YAML.load_file('your_resque_schedule.yml')
|
313
|
+
```
|
314
|
+
|
315
|
+
If a static schedule is not set `resque-scheduler` will issue a "Schedule empty!" warning on
|
316
|
+
startup, but despite that warning setting a static schedule is totally optional. It is possible
|
317
|
+
to use only dynamic schedules (see below).
|
318
|
+
|
319
|
+
The schedule file is a list of Resque job classes with arguments and a
|
166
320
|
schedule frequency (in crontab syntax). The schedule is just a hash, but
|
167
|
-
is
|
321
|
+
is usually stored in a YAML like this:
|
168
322
|
|
169
323
|
```yaml
|
170
324
|
CancelAbandonedOrders:
|
@@ -173,22 +327,34 @@ CancelAbandonedOrders:
|
|
173
327
|
queue_documents_for_indexing:
|
174
328
|
cron: "0 0 * * *"
|
175
329
|
# you can use rufus-scheduler "every" syntax in place of cron if you prefer
|
176
|
-
# every:
|
330
|
+
# every: 1h
|
177
331
|
# By default the job name (hash key) will be taken as worker class name.
|
178
332
|
# If you want to have a different job name and class name, provide the 'class' option
|
179
|
-
class: QueueDocuments
|
333
|
+
class: "QueueDocuments"
|
180
334
|
queue: high
|
181
335
|
args:
|
182
336
|
description: "This job queues all content for indexing in solr"
|
183
337
|
|
184
338
|
clear_leaderboards_contributors:
|
185
339
|
cron: "30 6 * * 1"
|
186
|
-
class: ClearLeaderboards
|
340
|
+
class: "ClearLeaderboards"
|
187
341
|
queue: low
|
188
342
|
args: contributors
|
189
343
|
description: "This job resets the weekly leaderboard for contributions"
|
190
344
|
```
|
191
345
|
|
346
|
+
If you would like to setup a job that is executed manually you can configure like this in your YAML file.
|
347
|
+
|
348
|
+
```yaml
|
349
|
+
ImportOrdersManual:
|
350
|
+
custom_job_class: 'AmazonMws::ImportOrdersJob'
|
351
|
+
never: "* * * * *"
|
352
|
+
queue: high
|
353
|
+
description: "This is a manual job for importing orders."
|
354
|
+
args:
|
355
|
+
days_in_arrears: 7
|
356
|
+
```
|
357
|
+
|
192
358
|
The queue value is optional, but if left unspecified resque-scheduler will
|
193
359
|
attempt to get the queue from the job class, which means it needs to be
|
194
360
|
defined. If you're getting "uninitialized constant" errors, you probably
|
@@ -201,13 +367,18 @@ You can provide options to "every" or "cron" via Array:
|
|
201
367
|
clear_leaderboards_moderator:
|
202
368
|
every:
|
203
369
|
- "30s"
|
204
|
-
- :first_in:
|
205
|
-
class: CheckDaemon
|
370
|
+
- :first_in: '120s'
|
371
|
+
class: "CheckDaemon"
|
206
372
|
queue: daemons
|
207
373
|
description: "This job will check Daemon every 30 seconds after 120 seconds after start"
|
208
374
|
```
|
209
375
|
|
210
|
-
|
376
|
+
IMPORTANT: Rufus `every` syntax will calculate jobs scheduling time starting from the moment of deploy,
|
377
|
+
resulting in resetting schedule time on every deploy, so it's probably a good idea to use it only for
|
378
|
+
frequent jobs (like every 10-30 minutes), otherwise - when you use something like `every 20h` and deploy once-twice per day -
|
379
|
+
it will schedule the job for 20 hours from deploy, resulting in a job to never be run.
|
380
|
+
|
381
|
+
**NOTE**: Six parameter cron's are also supported (as they supported by
|
211
382
|
rufus-scheduler which powers the resque-scheduler process). This allows you
|
212
383
|
to schedule jobs per second (ie: `"30 * * * * *"` would fire a job every 30
|
213
384
|
seconds past the minute).
|
@@ -215,21 +386,110 @@ seconds past the minute).
|
|
215
386
|
A big shout out to [rufus-scheduler](http://github.com/jmettraux/rufus-scheduler)
|
216
387
|
for handling the heavy lifting of the actual scheduling engine.
|
217
388
|
|
389
|
+
##### Queue with parameters
|
218
390
|
|
219
|
-
|
391
|
+
It's possible to specify parameters, that must be given by the user when they manually queue the job. To enable this feature add `parameters` key to scheduled job definition.
|
392
|
+
|
393
|
+
```yaml
|
394
|
+
queue_documents_for_indexing:
|
395
|
+
cron: "0 0 * * *"
|
396
|
+
class: "QueueDocuments"
|
397
|
+
queue: high
|
398
|
+
args:
|
399
|
+
foo: "bar"
|
400
|
+
a: "b"
|
401
|
+
parameters:
|
402
|
+
foo:
|
403
|
+
description: "value of foo"
|
404
|
+
default: "baz"
|
220
405
|
|
221
|
-
|
222
|
-
|
406
|
+
description: "This job queues all content for indexing in solr"
|
407
|
+
```
|
223
408
|
|
224
|
-
|
409
|
+
One can use following options for each parameter:
|
410
|
+
* description - tooltip to be shown next to the parameter input
|
411
|
+
* default - prefilled value in the parameter input
|
412
|
+
|
413
|
+
**NOTE**: When sheduling the job, parameters are merged into job args. Assuming the example above and default parametr value, the job will be run with the following args:
|
414
|
+
|
415
|
+
```ruby
|
416
|
+
{"foo"=>"baz", "a"=>"b"}
|
417
|
+
```
|
225
418
|
|
419
|
+
**NOTE**: If user leaves the parameter value empty, it'll be sent as empty string.
|
420
|
+
|
421
|
+
#### Dynamic schedules
|
422
|
+
|
423
|
+
Dynamic schedules are programmatically set on a running `resque-scheduler`.
|
424
|
+
Most [rufus-scheduler](http://github.com/jmettraux/rufus-scheduler) options are supported
|
425
|
+
when setting schedules. Specifically the `overlap` option will not work.
|
426
|
+
|
427
|
+
Dynamic schedules are not enabled by default. To be able to dynamically set schedules, you
|
428
|
+
must pass the following to `resque-scheduler` initialization (see *Installation* above for a more complete example):
|
429
|
+
|
430
|
+
```ruby
|
431
|
+
Resque::Scheduler.dynamic = true
|
432
|
+
```
|
433
|
+
|
434
|
+
**NOTE**: In order to delete dynamic schedules via `resque-web` in the
|
435
|
+
"Schedule" tab, you must include the `Rack::MethodOverride` middleware (in
|
436
|
+
`config.ru` or equivalent).
|
437
|
+
|
438
|
+
Dynamic schedules allow for greater flexibility than static schedules as they can be set,
|
439
|
+
unset or changed without having to restart `resque-scheduler`. You can specify, if the schedule
|
440
|
+
must survive a resque-scheduler restart or not. This is done by setting the `persist` configuration
|
441
|
+
for the schedule: it is a boolean value, if set the schedule will persist a restart. By default,
|
442
|
+
a schedule will not be persisted.
|
443
|
+
|
444
|
+
The job to be scheduled must be a valid Resque job class.
|
445
|
+
|
446
|
+
For example, suppose you have a SendEmail job which sends emails. The `perform` method of the
|
447
|
+
job receives a string argument with the email subject. To run the SendEmail job every hour
|
448
|
+
starting five minutes from now, you can do:
|
449
|
+
|
450
|
+
```ruby
|
451
|
+
name = 'send_emails'
|
452
|
+
config = {}
|
453
|
+
config[:class] = 'SendEmail'
|
454
|
+
config[:args] = 'POC email subject'
|
455
|
+
config[:every] = ['1h', {first_in: 5.minutes}]
|
456
|
+
config[:persist] = true
|
457
|
+
Resque.set_schedule(name, config)
|
458
|
+
```
|
459
|
+
|
460
|
+
Schedules can later be removed by passing their name to the `remove_schedule` method:
|
461
|
+
|
462
|
+
```ruby
|
463
|
+
name = 'send_emails'
|
464
|
+
Resque.remove_schedule(name)
|
465
|
+
```
|
466
|
+
|
467
|
+
Schedule names are unique; i.e. two dynamic schedules cannot have the same name. If `set_schedule` is
|
468
|
+
passed the name of an existing schedule, that schedule is updated. E.g. if after setting the above schedule
|
469
|
+
we want the job to run every day instead of every hour from now on, we can do:
|
470
|
+
|
471
|
+
```ruby
|
472
|
+
name = 'send_emails'
|
473
|
+
config = {}
|
474
|
+
config[:class] = 'SendEmail'
|
475
|
+
config[:args] = 'POC email subject'
|
476
|
+
config[:every] = '1d'
|
477
|
+
Resque.set_schedule(name, config)
|
478
|
+
```
|
479
|
+
|
480
|
+
#### Time zones
|
481
|
+
|
482
|
+
If you use the cron syntax, by default it is interpreted in the server time zone.
|
483
|
+
You can explicitly specify the time zone that rufus-scheduler will use:
|
226
484
|
```yaml
|
227
485
|
cron: "30 6 * * 1 Europe/Stockholm"
|
228
486
|
```
|
229
487
|
|
230
|
-
|
231
|
-
|
232
|
-
|
488
|
+
##### Rails
|
489
|
+
In Rails, `config.time_zone` will be used to determine the time zone for `resque-scheduler`.
|
490
|
+
|
491
|
+
Note that `config.time_zone` allows for a shorthand (e.g. "Stockholm")
|
492
|
+
that rufus-scheduler does not accept, so make sure it's the right format, e.g. with:
|
233
493
|
|
234
494
|
```ruby
|
235
495
|
ActiveSupport::TimeZone.find_tzinfo(Rails.configuration.time_zone).name
|
@@ -252,6 +512,13 @@ Similar to the `before_enqueue`- and `after_enqueue`-hooks provided in Resque
|
|
252
512
|
removed from the delayed queue, but not yet put on a normal queue. It is
|
253
513
|
called before `before_enqueue`-hooks, and on the same job instance as the
|
254
514
|
`before_enqueue`-hooks will be invoked on. Return values are ignored.
|
515
|
+
* `on_enqueue_failure`: Called with the job args and the exception that was raised
|
516
|
+
while enqueueing a job to resque or external application fails. Return
|
517
|
+
values are ignored. For example:
|
518
|
+
|
519
|
+
```ruby
|
520
|
+
Resque::Scheduler.failure_handler = ExceptionHandlerClass
|
521
|
+
```
|
255
522
|
|
256
523
|
#### Support for resque-status (and other custom jobs)
|
257
524
|
|
@@ -278,7 +545,7 @@ And then a schedule:
|
|
278
545
|
create_fake_leaderboards:
|
279
546
|
cron: "30 6 * * 1"
|
280
547
|
queue: scoring
|
281
|
-
custom_job_class: FakeLeaderboard
|
548
|
+
custom_job_class: "FakeLeaderboard"
|
282
549
|
args:
|
283
550
|
rails_env: demo
|
284
551
|
description: "This job will auto-create leaderboards for our online demo and the status will update as the worker makes progress"
|
@@ -304,13 +571,13 @@ end
|
|
304
571
|
*>= 2.0.1 only. Prior to 2.0.1, it is not recommended to run multiple resque-scheduler processes and will result in duplicate jobs.*
|
305
572
|
|
306
573
|
You may want to have resque-scheduler running on multiple machines for
|
307
|
-
|
574
|
+
redundancy. Electing a master and failover is built in and default. Simply
|
308
575
|
run resque-scheduler on as many machine as you want pointing to the same
|
309
576
|
redis instance and schedule. The scheduler processes will use redis to
|
310
577
|
elect a master process and detect failover when the master dies. Precautions are
|
311
578
|
taken to prevent jobs from potentially being queued twice during failover even
|
312
579
|
when the clocks of the scheduler machines are slightly out of sync (or load affects
|
313
|
-
scheduled job firing time). If you want the gory details, look at Resque::
|
580
|
+
scheduled job firing time). If you want the gory details, look at Resque::Scheduler::Locking.
|
314
581
|
|
315
582
|
If the scheduler process(es) goes down for whatever reason, the delayed items
|
316
583
|
that should have fired during the outage will fire once the scheduler process
|
@@ -325,11 +592,13 @@ run, leading to undesired behaviour. To allow different scheduler configs run at
|
|
325
592
|
on one redis, you can either namespace your redis connections, or supply an environment variable
|
326
593
|
to split the shared lock key resque-scheduler uses thus:
|
327
594
|
|
328
|
-
|
595
|
+
``` bash
|
596
|
+
RESQUE_SCHEDULER_MASTER_LOCK_PREFIX=MyApp: rake resque:scheduler
|
597
|
+
```
|
329
598
|
|
330
599
|
### resque-web Additions
|
331
600
|
|
332
|
-
Resque-scheduler also adds
|
601
|
+
Resque-scheduler also adds two tabs to the resque-web UI. One is for viewing
|
333
602
|
(and manually queueing) the schedule and one is for viewing pending jobs in
|
334
603
|
the delayed queue.
|
335
604
|
|
@@ -357,18 +626,13 @@ Now, you want to add the following:
|
|
357
626
|
|
358
627
|
```ruby
|
359
628
|
# This will make the tabs show up.
|
360
|
-
require '
|
361
|
-
require '
|
629
|
+
require 'resque-scheduler'
|
630
|
+
require 'resque/scheduler/server'
|
362
631
|
```
|
363
632
|
|
364
633
|
That should make the scheduler tabs show up in `resque-web`.
|
365
634
|
|
366
|
-
|
367
|
-
#### Changes as of 2.0.0
|
368
|
-
|
369
|
-
As of resque-scheduler 2.0.0, it's no longer necessary to have the resque-web
|
370
|
-
process aware of the schedule because it reads it from redis. But prior to
|
371
|
-
2.0, you'll want to make sure you load the schedule in this file as well.
|
635
|
+
You'll want to make sure you load the schedule in this file as well.
|
372
636
|
Something like this:
|
373
637
|
|
374
638
|
```ruby
|
@@ -382,7 +646,7 @@ Now make sure you're passing that file to resque-web like so:
|
|
382
646
|
|
383
647
|
### Running in the background
|
384
648
|
|
385
|
-
|
649
|
+
There are scenarios where it's helpful for
|
386
650
|
the resque worker to run itself in the background (usually in combination with
|
387
651
|
PIDFILE). Use the BACKGROUND option so that rake will return as soon as the
|
388
652
|
worker is started.
|
@@ -396,32 +660,100 @@ worker is started.
|
|
396
660
|
There are several options to toggle the way scheduler logs its actions. They
|
397
661
|
are toggled by environment variables:
|
398
662
|
|
399
|
-
- `
|
400
|
-
- `VERBOSE` opposite
|
401
|
-
- `LOGFILE` specifies the file to write logs to.
|
663
|
+
- `QUIET` will stop logging anything. Completely silent.
|
664
|
+
- `VERBOSE` opposite of 'QUIET'; will log even debug information
|
665
|
+
- `LOGFILE` specifies the file to write logs to. (default standard output)
|
666
|
+
- `LOGFORMAT` specifies either "text", "json" or "logfmt" output format
|
667
|
+
(default "text")
|
402
668
|
|
403
|
-
All
|
669
|
+
All of these variables are optional and will be given the following default
|
670
|
+
values:
|
404
671
|
|
405
672
|
```ruby
|
406
|
-
Resque::Scheduler.
|
407
|
-
|
408
|
-
|
673
|
+
Resque::Scheduler.configure do |c|
|
674
|
+
c.quiet = false
|
675
|
+
c.verbose = false
|
676
|
+
c.logfile = nil # meaning all messages go to $stdout
|
677
|
+
c.logformat = 'text'
|
678
|
+
end
|
409
679
|
```
|
410
680
|
|
411
|
-
|
412
681
|
### Polling frequency
|
413
682
|
|
414
|
-
You can pass
|
415
|
-
The default is 5 seconds, but
|
683
|
+
You can pass a `RESQUE_SCHEDULER_INTERVAL` option which is an integer or
|
684
|
+
float representing the polling frequency. The default is 5 seconds, but
|
685
|
+
for a semi-active app you may want to use a smaller value.
|
416
686
|
|
417
|
-
$
|
687
|
+
$ RESQUE_SCHEDULER_INTERVAL=1 rake resque:scheduler
|
688
|
+
|
689
|
+
**NOTE** This value was previously `INTERVAL` but was renamed to
|
690
|
+
`RESQUE_SCHEDULER_INTERVAL` to avoid clashing with the interval Resque
|
691
|
+
uses for its jobs.
|
418
692
|
|
419
693
|
### Plagiarism alert
|
420
694
|
|
421
|
-
This was intended to be an extension to resque and so resulted in a lot
|
422
|
-
code looking very similar to resque, particularly in resque-web
|
423
|
-
wanted it to be similar enough that someone familiar
|
424
|
-
work on resque-scheduler.
|
695
|
+
This was intended to be an extension to resque and so resulted in a lot
|
696
|
+
of the code looking very similar to resque, particularly in resque-web
|
697
|
+
and the views. I wanted it to be similar enough that someone familiar
|
698
|
+
with resque could easily work on resque-scheduler.
|
699
|
+
|
700
|
+
### Development
|
701
|
+
|
702
|
+
Working on resque-scheduler requires the following:
|
703
|
+
|
704
|
+
* A relatively modern Ruby interpreter
|
705
|
+
* bundler
|
706
|
+
|
707
|
+
The development setup looks like this, which is roughly the same thing
|
708
|
+
that happens on Travis CI and Appveyor:
|
709
|
+
|
710
|
+
``` bash
|
711
|
+
# Install everything
|
712
|
+
bundle install
|
713
|
+
|
714
|
+
# Make sure tests are green before you change stuff
|
715
|
+
bundle exec rubocop && bundle exec rake
|
716
|
+
# Change stuff
|
717
|
+
# Repeat
|
718
|
+
```
|
719
|
+
|
720
|
+
If you have [vagrant](http://www.vagrantup.com) installed, there is a
|
721
|
+
development box available that requires no plugins or external
|
722
|
+
provisioners:
|
723
|
+
|
724
|
+
``` bash
|
725
|
+
vagrant up
|
726
|
+
```
|
727
|
+
|
728
|
+
### Deployment Notes
|
729
|
+
|
730
|
+
It is recommended that a production deployment of `resque-scheduler` be hosted
|
731
|
+
on a dedicated Redis database. While making and managing scheduled tasks,
|
732
|
+
`resque-scheduler` currently scans the entire Redis keyspace, which may cause
|
733
|
+
latency and stability issues if `resque-scheduler` is hosted on a Redis instance
|
734
|
+
storing a large number of keys (such as those written by a different system
|
735
|
+
hosted on the same Redis instance).
|
736
|
+
|
737
|
+
#### Compatibility Notes
|
738
|
+
|
739
|
+
Different versions of the `redis` and `rufus-scheduler` gems are needed
|
740
|
+
depending on your version of `resque-scheduler`. This is typically not a
|
741
|
+
problem with `resque-scheduler` itself, but when mixing dependencies with an
|
742
|
+
existing application.
|
743
|
+
|
744
|
+
This table explains the version requirements for redis gem
|
745
|
+
|
746
|
+
| resque-scheduler | redis gem |
|
747
|
+
|:-----------------|-----------:|
|
748
|
+
| `~> 2.0` | `>= 3.0.0` |
|
749
|
+
| `>= 0.0.1` | `~> 1.3` |
|
750
|
+
|
751
|
+
This table explains the version requirements for rufus-scheduler
|
752
|
+
|
753
|
+
| resque-scheduler | rufus-scheduler |
|
754
|
+
|:-----------------|----------------:|
|
755
|
+
| `~> 4.0` | `~> 3.0` |
|
756
|
+
| `< 4.0` | `~> 2.0` |
|
425
757
|
|
426
758
|
|
427
759
|
### Contributing
|