procrastinator 0.6.1 → 1.0.0.pre.rc2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +5 -5
- data/.gitignore +6 -1
- data/.rubocop.yml +26 -0
- data/.ruby-version +1 -1
- data/Gemfile +2 -0
- data/README.md +492 -144
- data/RELEASE_NOTES.md +44 -0
- data/Rakefile +5 -3
- data/lib/procrastinator/config.rb +149 -0
- data/lib/procrastinator/logged_task.rb +50 -0
- data/lib/procrastinator/queue.rb +206 -0
- data/lib/procrastinator/queue_worker.rb +66 -91
- data/lib/procrastinator/rake/daemon_tasks.rb +54 -0
- data/lib/procrastinator/rake/tasks.rb +3 -0
- data/lib/procrastinator/scheduler.rb +393 -0
- data/lib/procrastinator/task.rb +64 -0
- data/lib/procrastinator/task_meta_data.rb +172 -0
- data/lib/procrastinator/task_store/file_transaction.rb +76 -0
- data/lib/procrastinator/task_store/simple_comma_store.rb +161 -0
- data/lib/procrastinator/test/mocks.rb +35 -0
- data/lib/procrastinator/version.rb +3 -1
- data/lib/procrastinator.rb +29 -23
- data/procrastinator.gemspec +17 -11
- metadata +66 -28
- data/lib/procrastinator/environment.rb +0 -148
- data/lib/procrastinator/task_worker.rb +0 -120
data/README.md
CHANGED
@@ -1,9 +1,80 @@
|
|
1
1
|
# Procrastinator
|
2
2
|
|
3
|
-
Procrastinator is a
|
4
|
-
a subprocess for each queue to performs tasks at the designated times. Or maybe later, depending on how busy it is.
|
3
|
+
Procrastinator is a pure Ruby job scheduling gem. Put off tasks until later or for another process to handle.
|
5
4
|
|
6
|
-
|
5
|
+
Tasks are can be rescheduled and retried after failures, and you can use whatever storage mechanism is needed.
|
6
|
+
|
7
|
+
## Big Picture
|
8
|
+
|
9
|
+
If you have tasks like this:
|
10
|
+
|
11
|
+
```ruby
|
12
|
+
# Sends a welcome email
|
13
|
+
class SendWelcomeEmail
|
14
|
+
def run
|
15
|
+
# ... etc
|
16
|
+
end
|
17
|
+
end
|
18
|
+
```
|
19
|
+
|
20
|
+
Setup a procrastination environment like:
|
21
|
+
|
22
|
+
```ruby
|
23
|
+
scheduler = Procrastinator.setup do |env|
|
24
|
+
env.with_store some_email_task_database do
|
25
|
+
env.define_queue :greeting, SendWelcomeEmail
|
26
|
+
env.define_queue :birthday, SendBirthdayEmail, max_attempts: 3
|
27
|
+
end
|
28
|
+
|
29
|
+
env.define_queue :thumbnail, GenerateThumbnail, store: 'imgtasks.csv', timeout: 60
|
30
|
+
end
|
31
|
+
```
|
32
|
+
|
33
|
+
Put jobs off until later:
|
34
|
+
|
35
|
+
```ruby
|
36
|
+
scheduler.delay(:greeting, data: 'bob@example.com')
|
37
|
+
|
38
|
+
scheduler.delay(:thumbnail, data: {file: 'full_image.png', width: 100, height: 100})
|
39
|
+
|
40
|
+
scheduler.delay(:send_birthday_email, run_at: Time.now + 3600, data: {user_id: 5})
|
41
|
+
```
|
42
|
+
|
43
|
+
And tell a process to actually do them:
|
44
|
+
|
45
|
+
```ruby
|
46
|
+
# Starts a daemon with a thread for each queue.
|
47
|
+
# Other options are direct control (for testing) or threaded (for terminals)
|
48
|
+
scheduler.work.daemonized!
|
49
|
+
```
|
50
|
+
|
51
|
+
## Contents
|
52
|
+
|
53
|
+
- [Installation](#installation)
|
54
|
+
- [Setup](#setup)
|
55
|
+
* [Defining Queues](#defining-queues)
|
56
|
+
* [Task Store](#task-store)
|
57
|
+
+ [Data Fields](#data-fields)
|
58
|
+
* [Task Container](#task-container)
|
59
|
+
- [Tasks](#tasks)
|
60
|
+
* [Accessing Task Attributes](#accessing-task-attributes)
|
61
|
+
* [Errors & Logging](#errors-logging)
|
62
|
+
- [Scheduling Tasks](#scheduling-tasks)
|
63
|
+
* [Providing Data](#providing-data)
|
64
|
+
* [Scheduling](#scheduling)
|
65
|
+
* [Rescheduling](#rescheduling)
|
66
|
+
* [Retries](#retries)
|
67
|
+
* [Cancelling](#cancelling)
|
68
|
+
- [Working on Tasks](#working-on-tasks)
|
69
|
+
* [Stepwise Working](#stepwise-working)
|
70
|
+
* [Threaded Working](#threaded-working)
|
71
|
+
* [Daemonized Working](#daemonized-working)
|
72
|
+
+ [PID Files](#pid-files)
|
73
|
+
- [Contributing](#contributing)
|
74
|
+
* [Developers](#developers)
|
75
|
+
- [License](#license)
|
76
|
+
|
77
|
+
<!-- ToC generated with http://ecotrust-canada.github.io/markdown-toc/ -->
|
7
78
|
|
8
79
|
## Installation
|
9
80
|
|
@@ -17,238 +88,515 @@ And then run:
|
|
17
88
|
|
18
89
|
bundle install
|
19
90
|
|
20
|
-
##
|
21
|
-
|
91
|
+
## Setup
|
92
|
+
|
93
|
+
`Procrastinator.setup` allows you to define which queues are available and other settings.
|
22
94
|
|
23
95
|
```ruby
|
24
|
-
procrastinator
|
25
|
-
|
26
|
-
|
96
|
+
require 'procrastinator'
|
97
|
+
|
98
|
+
scheduler = Procrastinator.setup do |config|
|
99
|
+
# ...
|
27
100
|
end
|
28
101
|
```
|
29
102
|
|
30
|
-
|
103
|
+
It then returns a `Scheduler` that your code can use to schedule tasks or tell to start working.
|
104
|
+
|
105
|
+
* See [Scheduling Tasks](#scheduling-tasks)
|
106
|
+
* See [Start Working](#start-working)
|
107
|
+
|
108
|
+
### Defining Queues
|
109
|
+
|
110
|
+
In setup, call `#define_queue` with a symbol name and the class that performs those jobs:
|
111
|
+
|
112
|
+
```ruby
|
113
|
+
# You must provide a queue name and the class that handles those jobs
|
114
|
+
config.define_queue :greeting, SendWelcomeEmail
|
115
|
+
|
116
|
+
# but queues have some optional settings, too
|
117
|
+
config.define_queue :greeting, SendWelcomeEmail, store: 'tasks.csv', timeout: 60, max_attempts: 2, update_period: 1
|
118
|
+
|
119
|
+
# all defaults set explicitly
|
120
|
+
config.define_queue :greeting, SendWelcomeEmail, store: 'procrastinator.csv', timeout: 3600, max_attempts: 20, update_period: 10
|
121
|
+
```
|
122
|
+
|
123
|
+
Description of keyword options:
|
124
|
+
|
125
|
+
| Option | Description |
|
126
|
+
|------------------| ------------- |
|
127
|
+
| `:store` | Storage IO object for tasks. See [Task Store](#task-store) |
|
128
|
+
| `:timeout` | Max duration (seconds) before tasks are failed for taking too long |
|
129
|
+
| `:max_attempts` | Once a task has been attempted `max_attempts` times, it will be permanently failed. |
|
130
|
+
| `:update_period` | Delay (seconds) between reloads of all tasks from the task store |
|
131
|
+
|
132
|
+
### Task Store
|
133
|
+
|
134
|
+
A task store is a [strategy](https://en.wikipedia.org/wiki/Strategy_pattern) pattern object that knows how to read and
|
135
|
+
write tasks in your data storage (eg. database, CSV file, etc).
|
31
136
|
|
32
137
|
```ruby
|
33
|
-
|
34
|
-
|
138
|
+
task_store = ReminderStore.new # eg. some SQL task storage class you wrote
|
139
|
+
|
140
|
+
Procrastinator.setup do |config|
|
141
|
+
config.define_queue(:reminder, ReminderTask, store: task_store)
|
142
|
+
|
143
|
+
# to use the default CSV storage, provide :store with a string or Pathname
|
144
|
+
config.define_queue(:reminder, ReminderTask, store: '/var/myapp/tasks.csv')
|
145
|
+
end
|
35
146
|
```
|
36
147
|
|
37
|
-
|
148
|
+
A task store is required to implement *all* of the following methods or else it will raise a
|
149
|
+
`MalformedPersisterError`:
|
150
|
+
|
151
|
+
1. `#read(attributes)`
|
152
|
+
|
153
|
+
Returns a list of hashes from your datastore that match the given attributes hash. The search attributes will be in
|
154
|
+
their final form (eg. `:data` will already be serialized). Each hash must contain the properties listed
|
155
|
+
in [Task Data](#task-data) below.
|
156
|
+
|
157
|
+
2. `#create(queue:, run_at:, initial_run_at:, expire_at:, data:)`
|
158
|
+
|
159
|
+
Saves a task in your storage. Receives a hash with [Task Data](#task-data) keys:
|
160
|
+
`:queue`, `:run_at`, `:initial_run_at`, `:expire_at`, and `:data`.
|
161
|
+
|
162
|
+
3. `#update(id, new_data)`
|
38
163
|
|
39
|
-
|
40
|
-
The setup phase first defines which queues are available and the persistence strategy to use for reading
|
41
|
-
and writing tasks. It then spins off a sub process for working on each queue within that environment.
|
164
|
+
Saves the provided full [Task Data](#task-data) hash to your datastore.
|
42
165
|
|
166
|
+
4. `#delete(id)`
|
43
167
|
|
44
|
-
|
45
|
-
The persister instance is the last step between Procrastinator and your data storage (eg. database). As core Procrastinator is framework-agnostic, it needs you to provide an object that knows how to read and write task data.
|
168
|
+
Deletes the task with the given identifier from storage
|
46
169
|
|
47
|
-
|
170
|
+
Procrastinator comes with a simple CSV file task store by default, but you are encouraged to build one that suits your
|
171
|
+
situation.
|
48
172
|
|
49
|
-
|
50
|
-
* `#create_task(data)` - Creates a task in your datastore. Receives a hash with keys `:queue`, `:run_at`, `:initial_run_at`, `:expire_at`, and `:task` as described in *Attributes Hash*
|
51
|
-
* `#update_task(attributes)` - Receives the Attributes Hash as the data to be saved
|
52
|
-
* `#delete_task(id)` - Deletes the task with the given id.
|
173
|
+
_Warning_: Task stores shared between queues **must** be thread-safe if using threaded or daemonized work modes.
|
53
174
|
|
54
|
-
|
175
|
+
#### Data Fields
|
55
176
|
|
56
|
-
|
177
|
+
These are the data fields for each individual scheduled task. When using the built-in task store, these are the field
|
178
|
+
names. If you have a database, use this to inform your table schema.
|
57
179
|
|
58
|
-
| Hash Key | Type
|
59
|
-
|
60
|
-
| `:id` | int
|
61
|
-
| `:queue` | symbol
|
62
|
-
| `:run_at` |
|
63
|
-
| `:initial_run_at` |
|
64
|
-
| `:expire_at` |
|
65
|
-
| `:attempts` | int
|
66
|
-
| `:last_fail_at` |
|
67
|
-
| `:last_error` | string
|
68
|
-
| `:
|
180
|
+
| Hash Key | Type | Description |
|
181
|
+
|-------------------|---------| ----------------------------------------------------------------------------------------|
|
182
|
+
| `:id` | int | Unique identifier for this exact task |
|
183
|
+
| `:queue` | symbol | Name of the queue the task is inside |
|
184
|
+
| `:run_at` | iso8601 | Time to attempt running the task next. ¹ |
|
185
|
+
| `:initial_run_at` | iso8601 | Originally requested run_at. Reset when rescheduled. |
|
186
|
+
| `:expire_at` | iso8601 | Time to permanently fail the task because it is too late to be useful |
|
187
|
+
| `:attempts` | int | Number of times the task has tried to run; this should only be > 0 if the task fails |
|
188
|
+
| `:last_fail_at` | iso8601 | Unix timestamp of when the most recent failure happened |
|
189
|
+
| `:last_error` | string | Error message + bracktrace of the most recent failure. May be very long. |
|
190
|
+
| `:data` | string | Serialized data accessible in the task instance. |
|
69
191
|
|
70
|
-
|
192
|
+
> ¹ `nil` indicates that it is permanently failed and will never run, either due to expiry or too many attempts.
|
71
193
|
|
72
|
-
|
73
|
-
|
194
|
+
Data is serialized using `JSON.dump` and `JSON.parse` with **symbolized keys**. It is strongly recommended to only
|
195
|
+
supply simple data types (eg. id numbers) to reduce storage space, eliminate redundancy, and reduce the chance of a
|
196
|
+
serialization error.
|
74
197
|
|
75
|
-
|
76
|
-
* :max_attempts - Maximum number of attempts for tasks in this queue. If attempts is >= max_attempts, the task will be final_failed and marked to never run again
|
77
|
-
* :update_period - Delay, in seconds, between refreshing the task list from the persister
|
78
|
-
* :max_tasks - The maximum number of tasks to run concurrently with multi-threading.
|
198
|
+
Times are all handled as [ISO8601](https://en.wikipedia.org/wiki/ISO_8601) formatted strings.
|
79
199
|
|
80
|
-
|
81
|
-
|
82
|
-
Procrastinator.
|
83
|
-
|
84
|
-
|
85
|
-
|
86
|
-
|
200
|
+
#### Default Task Store
|
201
|
+
|
202
|
+
Specifying no storage will cause Procrastinator to save tasks using the very basic built-in CSV storage. It is not
|
203
|
+
designed for heavy loads, so you should replace it in a production environment.
|
204
|
+
|
205
|
+
The file path is defined in `Procrastinator::Store::SimpleCommaStore::DEFAULT_FILE`.
|
206
|
+
|
207
|
+
```ruby
|
208
|
+
Procrastinator.setup do |config|
|
209
|
+
# this will use the default CSV task store.
|
210
|
+
config.define_queue(:reminder, ReminderTask)
|
87
211
|
end
|
88
212
|
```
|
89
|
-
|
90
|
-
#### Sub-Processes
|
91
|
-
Each queue is worked in a separate process.
|
92
|
-
|
93
|
-
<!-- , and each process multi-threaded to handle more than one task at a time. This should help prevent a single task from clogging up the whole queue, or a single queue clogging up the entire system. -->
|
94
213
|
|
95
|
-
|
214
|
+
#### Shared Task Stores
|
96
215
|
|
97
|
-
|
216
|
+
When there are tasks use the same storage, you can wrap them in a `with_store` block.
|
98
217
|
|
99
218
|
```ruby
|
100
|
-
|
101
|
-
|
219
|
+
email_task_store = EmailTaskStore.new # eg. some SQL task storage class you wrote
|
220
|
+
|
221
|
+
Procrastinator.setup do |config|
|
222
|
+
with_store(email_task_store) do
|
223
|
+
# queues defined inside this block will use the email task store
|
224
|
+
config.define_queue(:welcome, WelcomeTask)
|
225
|
+
config.define_queue(:reminder, ReminderTask)
|
226
|
+
end
|
227
|
+
|
228
|
+
# and this will not use it
|
229
|
+
config.define_queue(:thumbnails, ThumbnailTask)
|
102
230
|
end
|
103
231
|
```
|
104
232
|
|
105
|
-
###
|
106
|
-
|
233
|
+
### Task Container
|
234
|
+
|
235
|
+
Whatever is given to `#provide_container` will available to Tasks through the task attribute `:container`.
|
236
|
+
|
237
|
+
This can be useful for things like app containers, but you can use it for whatever you like.
|
107
238
|
|
108
239
|
```ruby
|
109
|
-
|
110
|
-
env.
|
240
|
+
Procrastinator.setup do |env|
|
241
|
+
env.provide_container lunch: 'Lasagna'
|
242
|
+
|
243
|
+
# .. other setup stuff ...
|
111
244
|
end
|
112
245
|
|
113
|
-
|
246
|
+
# ... and in your task ...
|
247
|
+
class LunchTask
|
248
|
+
attr_accessor :logger, :scheduler, :container
|
249
|
+
|
250
|
+
def run
|
251
|
+
logger.info("Today's Lunch is: #{ container[:lunch] }")
|
252
|
+
end
|
253
|
+
end
|
114
254
|
```
|
115
255
|
|
116
|
-
|
256
|
+
## Tasks
|
257
|
+
|
258
|
+
Your task class is what actually gets run on the task queue. They'll look like this:
|
117
259
|
|
118
260
|
```ruby
|
119
|
-
procrastinator = Procrastinator.setup(task_persister) do |env|
|
120
|
-
env.define_queue(:email)
|
121
|
-
env.define_queue(:cleanup)
|
122
|
-
end
|
123
261
|
|
124
|
-
|
262
|
+
class MyTask
|
263
|
+
# These attributes will be assigned by Procrastinator when the task is run.
|
264
|
+
attr_accessor :logger, :scheduler, :container, :data
|
265
|
+
|
266
|
+
# Performs the core work of the task.
|
267
|
+
def run
|
268
|
+
# ... perform your task ...
|
269
|
+
end
|
270
|
+
|
271
|
+
# ========================================
|
272
|
+
# OPTIONAL HOOKS
|
273
|
+
#
|
274
|
+
# You can always omit any of the methods
|
275
|
+
# below. Only #run is mandatory.
|
276
|
+
#
|
277
|
+
# ========================================
|
278
|
+
|
279
|
+
# Called after the task has completed successfully.
|
280
|
+
# Receives the result of #run.
|
281
|
+
def success(run_result)
|
282
|
+
# ...
|
283
|
+
end
|
284
|
+
|
285
|
+
# Called after #run raises any StandardError (or subclass).
|
286
|
+
# Receives the raised error.
|
287
|
+
def fail(error)
|
288
|
+
# ...
|
289
|
+
end
|
290
|
+
|
291
|
+
# Called after either is true:
|
292
|
+
# 1. the time reported by Time.now is past the task's expire_at time.
|
293
|
+
# 2. the task has failed and the number of attempts is equal to or greater than the queue's `max_attempts`.
|
294
|
+
# In this case, #fail will not be executed, only #final_fail.
|
295
|
+
#
|
296
|
+
# When called, the task will be marked to never be run again.
|
297
|
+
# Receives the raised error.
|
298
|
+
def final_fail(error)
|
299
|
+
# ...
|
300
|
+
end
|
301
|
+
end
|
125
302
|
```
|
126
303
|
|
127
|
-
|
128
|
-
|
304
|
+
### Attribute Accessors
|
305
|
+
|
306
|
+
Tasks must provide accessors for `:logger`, `:container`, and `:scheduler`, while the `:data` accessor is semi-optional
|
307
|
+
(see below). This is to prevent the tasks from referencing unknown variables once they actually get run.
|
308
|
+
|
309
|
+
* `:data` The data you provided in the call to `#delay`. Any task with a `:data` accessor will require data be passed
|
310
|
+
to `#delay`, and vice versa. See [Task Data](#task-data) for more.
|
311
|
+
|
312
|
+
* `:container` The container you've provided in your setup. See [Task Container](#task-container) for more.
|
313
|
+
|
314
|
+
* `:logger` The queue's Logger object. See [Logging](#logging) for more.
|
315
|
+
|
316
|
+
* `:scheduler` A scheduler object that you can use to schedule new tasks (eg. with `#delay`).
|
317
|
+
|
318
|
+
### Errors & Logging
|
319
|
+
|
320
|
+
Errors that trigger `#fail` or `#final_fail` are saved to the task storage under columns `last_error` and
|
321
|
+
`last_error_at`.
|
322
|
+
|
323
|
+
Each queue worker also keeps a logfile log using the Ruby
|
324
|
+
[Logger class](https://ruby-doc.org/stdlib-2.7.1/libdoc/logger/rdoc/Logger.html). Log files are named after the queue (
|
325
|
+
eg. `log/welcome-queue-worker.log`).
|
129
326
|
|
130
327
|
```ruby
|
131
|
-
|
132
|
-
|
328
|
+
scheduler = Procrastinator.setup do |env|
|
329
|
+
# you can set custom log location and level:
|
330
|
+
env.log_with(directory: '/var/log/myapp/', level: Logger::DEBUG)
|
331
|
+
|
332
|
+
# you can also set the log rotation age or size (see Logger docs for details)
|
333
|
+
env.log_with(shift: 1024, age: 5)
|
334
|
+
|
335
|
+
# use a falsey log level to disable logging entirely:
|
336
|
+
env.log_with(level: false)
|
133
337
|
end
|
338
|
+
```
|
134
339
|
|
135
|
-
|
136
|
-
procrastinator.delay(run_at: Time.new(3000, 1, 1), task: EmailGreeting.new('philip_j_fry@example.com'))
|
340
|
+
The logger can be accessed in your tasks by calling `logger` or `@logger`.
|
137
341
|
|
138
|
-
|
139
|
-
|
342
|
+
```ruby
|
343
|
+
|
344
|
+
class MyTask
|
345
|
+
attr_accessor :logger, :scheduler, :container
|
346
|
+
|
347
|
+
def run
|
348
|
+
logger.info('This task got run. Hooray!')
|
349
|
+
end
|
350
|
+
end
|
140
351
|
```
|
141
352
|
|
142
|
-
|
353
|
+
Some events are always logged by default:
|
354
|
+
|
355
|
+
|event |level |
|
356
|
+
|--------------------|-------|
|
357
|
+
|process started | INFO |
|
358
|
+
|#success called | DEBUG |
|
359
|
+
|#fail called | DEBUG |
|
360
|
+
|#final_fail called | DEBUG |
|
361
|
+
|
362
|
+
## Scheduling Tasks
|
363
|
+
|
364
|
+
To schedule tasks, just call `#delay` on the environment returned from `Procrastinator.setup`:
|
143
365
|
|
144
366
|
```ruby
|
145
|
-
|
146
|
-
env.define_queue
|
367
|
+
scheduler = Procrastinator.setup do |env|
|
368
|
+
env.define_queue :reminder, EmailReminder
|
369
|
+
env.define_queue :thumbnail, CreateThumbnail
|
147
370
|
end
|
148
371
|
|
149
|
-
|
372
|
+
# Provide the queue name and any data you want passed in
|
373
|
+
scheduler.delay(:reminder, data: 'bob@example.com')
|
374
|
+
```
|
375
|
+
|
376
|
+
If there is only one queue, you may omit the queue name:
|
377
|
+
|
378
|
+
```ruby
|
379
|
+
scheduler = Procrastinator.setup do |env|
|
380
|
+
env.define_queue :reminder, EmailReminder
|
381
|
+
end
|
150
382
|
|
151
|
-
|
152
|
-
procrastinator.delay(expire_at: nil, task: EmailGreeting.new('bob@example.com'))
|
383
|
+
scheduler.delay(data: 'bob@example.com')
|
153
384
|
```
|
154
385
|
|
155
|
-
|
156
|
-
Like the persister provided to `.setup`, your task is a strategy object that fills in the details of what to do. For this,
|
157
|
-
your task **must provide** a `#run` method:
|
386
|
+
### Providing Data
|
158
387
|
|
159
|
-
|
388
|
+
Most tasks need some additional information to complete their work, like id numbers,
|
160
389
|
|
161
|
-
|
390
|
+
The `:data` parameter is serialized to string as YAML, so it's better to keep it as simple as possible. For example, if
|
391
|
+
you have a database instead of passing in a complex Ruby object, pass in just the primary key and reload it in the
|
392
|
+
task's `#run`. This will require less space in your database and avoids obsolete or duplicated information.
|
162
393
|
|
163
|
-
|
164
|
-
* `#fail(logger, error)` - run after the task has failed due to `#run` producing a `StandardError` or subclass.
|
165
|
-
* `#final_fail(logger, error)` - run after the task has failed for the last time because either:
|
166
|
-
1. the number of attempts is >= the `max_attempts` defined for the queue; or
|
167
|
-
2. the time reported by `Time.now` is past the task's `expire_at` time.
|
394
|
+
### Scheduling
|
168
395
|
|
169
|
-
|
396
|
+
You can set when the particular task is to be run and/or when it should expire. Be aware that the task is not guaranteed
|
397
|
+
to run at a precise time; the only promise is that the task will be attempted *after* `run_at` and before `expire_at`.
|
170
398
|
|
171
|
-
|
399
|
+
```ruby
|
400
|
+
# runs on or after 1 January 3000
|
401
|
+
scheduler.delay(:greeting, run_at: Time.new(3000, 1, 1), data: 'philip_j_fry@example.com')
|
172
402
|
|
173
|
-
|
174
|
-
|
175
|
-
|
176
|
-
n = the number of attempts
|
403
|
+
# run_at defaults to right now:
|
404
|
+
scheduler.delay(:thumbnail, run_at: Time.now, data: 'shut_up_and_take_my_money.gif')
|
405
|
+
```
|
177
406
|
|
178
|
-
|
407
|
+
You can also set an `expire_at` deadline. If the task has not been run before `expire_at` is passed, then it will be
|
408
|
+
final-failed the next time it would be attempted. Setting `expire_at` to `nil` means it will never expire (but may still
|
409
|
+
fail permanently if, say, `max_attempts` is reached).
|
179
410
|
|
180
|
-
|
181
|
-
|
182
|
-
|
411
|
+
```ruby
|
412
|
+
# will not run at or after
|
413
|
+
scheduler.delay(:happy_birthday, expire_at: Time.new(2018, 03, 17, 12, 00, '-06:00'), data: 'contact@tenjin.ca')
|
183
414
|
|
184
|
-
|
185
|
-
|
415
|
+
# expire_at defaults to nil:
|
416
|
+
scheduler.delay(:greeting, expire_at: nil, data: 'bob@example.com')
|
417
|
+
```
|
418
|
+
|
419
|
+
### Rescheduling
|
420
|
+
|
421
|
+
Call `#reschedule` with the queue name and some identifying information, and then calling #to on that to provide the new
|
422
|
+
time.
|
186
423
|
|
187
424
|
```ruby
|
188
|
-
|
189
|
-
|
190
|
-
|
191
|
-
# or you can also enable it directly in the setup
|
192
|
-
env = Procrastinator.setup do |env|
|
193
|
-
env.enable_test_mode
|
194
|
-
|
195
|
-
# other settings...
|
425
|
+
scheduler = Procrastinator.setup do |env|
|
426
|
+
env.define_queue :reminder, EmailReminder
|
196
427
|
end
|
428
|
+
|
429
|
+
scheduler.delay(:reminder, run_at: Time.parse('June 1'), data: 'bob@example.com')
|
430
|
+
|
431
|
+
# we can reschedule the task made above
|
432
|
+
scheduler.reschedule(:reminder, data: 'bob@example.com').to(run_at: Time.parse('June 20 12:00'))
|
433
|
+
|
434
|
+
# we can also change the expiry time
|
435
|
+
scheduler.reschedule(:reminder, data: 'bob@example.com').to(expire_at: Time.parse('June 23 12:00'))
|
436
|
+
|
437
|
+
# or both
|
438
|
+
scheduler.reschedule(:reminder, data: 'bob@example.com').to(run_at: Time.parse('June 20 12:00'),
|
439
|
+
expire_at: Time.parse('June 23 12:00'))
|
197
440
|
```
|
198
441
|
|
199
|
-
|
442
|
+
Rescheduling sets the task's:
|
443
|
+
|
444
|
+
* `:run_at` and `:initial_run_at` to a new value, if provided
|
445
|
+
* `:expire_at` to a new value if provided.
|
446
|
+
* `:attempts` to `0`
|
447
|
+
* `:last_error` and `:last_error_at` to `nil`.
|
448
|
+
|
449
|
+
Rescheduling will not change `:id`, `:queue` or `:data`. A `RuntimeError` is raised if the runtime is after the expiry.
|
450
|
+
|
451
|
+
### Retries
|
452
|
+
|
453
|
+
Failed tasks have their `run_at` rescheduled on an increasing delay (in seconds) according to this formula:
|
454
|
+
|
455
|
+
> 30 + (number_of_attempts)<sup>4</sup>
|
456
|
+
|
457
|
+
Situations that call `#fail` or `#final_fail` will cause the error timestamp and reason to be stored in `:last_fail_at`
|
458
|
+
and `:last_error`.
|
459
|
+
|
460
|
+
### Cancelling
|
200
461
|
|
462
|
+
Call `#cancel` with the queue name and some identifying information to narrow the search to a single task.
|
463
|
+
|
464
|
+
```ruby
|
465
|
+
scheduler = Procrastinator.setup do |env|
|
466
|
+
env.define_queue :reminder, EmailReminder
|
467
|
+
end
|
468
|
+
|
469
|
+
scheduler.delay(:reminder, run_at: Time.parse('June 1'), data: 'bob@example.com')
|
470
|
+
|
471
|
+
# we can cancel the task made above using whatever we know about it, like the saved :data
|
472
|
+
scheduler.reschedule(:reminder, data: 'bob@example.com')
|
473
|
+
|
474
|
+
# or multiple attributes
|
475
|
+
scheduler.reschedule(:reminder, run_at: Time.parse('June 1'), data: 'bob@example.com')
|
476
|
+
|
477
|
+
# you could also use the id number directly, if you have it
|
478
|
+
scheduler.reschedule(:reminder, id: 137)
|
201
479
|
```
|
202
|
-
# works one task on all queues
|
203
|
-
env.act
|
204
480
|
|
205
|
-
|
206
|
-
|
481
|
+
## Working on Tasks
|
482
|
+
|
483
|
+
Use the scheduler object returned by setup to `#work` queues **serially**, **threaded**, or **daemonized**.
|
484
|
+
|
485
|
+
### Serial Working
|
486
|
+
|
487
|
+
Working serially performs a task from each queue directly. There is no multithreading or daemonizing.
|
488
|
+
|
489
|
+
Work serially for TDD tests or other situations you need close direct control.
|
490
|
+
|
491
|
+
```ruby
|
492
|
+
# work just one task, no threading
|
493
|
+
scheduler.work.serially
|
494
|
+
|
495
|
+
# work the first five tasks
|
496
|
+
scheduler.work.serially(steps: 5)
|
497
|
+
|
498
|
+
# only work tasks on greeting and reminder queues
|
499
|
+
scheduler.work(:greeting, :reminders).serially(steps: 2)
|
207
500
|
```
|
208
501
|
|
209
|
-
###
|
210
|
-
|
211
|
-
|
212
|
-
|
213
|
-
|
502
|
+
### Threaded Working
|
503
|
+
|
504
|
+
Threaded working will spawn a worker thread per queue.
|
505
|
+
|
506
|
+
Use threaded working for task queues that should only run while the main application is running. This includes the usual
|
507
|
+
caveats around multithreading, so proceed with caution.
|
214
508
|
|
215
509
|
```ruby
|
216
|
-
|
217
|
-
|
218
|
-
|
510
|
+
# work tasks until the application exits
|
511
|
+
scheduler.work.threaded
|
512
|
+
|
513
|
+
# work tasks for 5 seconds
|
514
|
+
scheduler.work.threaded(timeout: 5)
|
515
|
+
|
516
|
+
# only work tasks on greeting and reminder queues
|
517
|
+
scheduler.work(:greeting, :reminders).threaded
|
219
518
|
```
|
220
|
-
|
221
|
-
Each worker creates its own log named after the queue it is working on (eg. `log/email-queue-worker.log`). The default
|
222
|
-
directory is `./log/`, relative to wherever the application is running. Logging will not occur at all if `log_dir` is
|
223
|
-
assigned a falsey value.
|
224
519
|
|
225
|
-
|
226
|
-
|
520
|
+
### Daemonized Working
|
521
|
+
|
522
|
+
Daemonized working **consumes the current process** and then proceeds with threaded working in the new daemon.
|
227
523
|
|
228
|
-
|
229
|
-
|
524
|
+
Use daemonized working for production environments, especially in conjunction with daemon monitors
|
525
|
+
like [Monit](https://mmonit.com/monit/). Provide a block to daemonized! to get
|
230
526
|
|
231
527
|
```ruby
|
232
|
-
|
233
|
-
|
234
|
-
|
528
|
+
# work tasks forever as a headless daemon process.
|
529
|
+
scheduler.work.daemonized!
|
530
|
+
|
531
|
+
# you can specify the new process name and the directory to save the procrastinator.pid file
|
532
|
+
scheduler.work.daemonized!(name: 'myapp-queue', pid_path: '/var/run')
|
533
|
+
|
534
|
+
# ... or set the pid file name precisely by giving a .pid path
|
535
|
+
scheduler.work.daemonized!(pid_path: '/var/run/myapp.pid')
|
536
|
+
|
537
|
+
# only work tasks in the 'greeting' and 'reminder' queues
|
538
|
+
scheduler.work(:greeting, :reminders).daemonized!
|
539
|
+
|
540
|
+
# supply a block to run code after the daemon subprocess has forked off
|
541
|
+
scheduler.work.daemonized! do
|
542
|
+
# this gets run after the daemon is spawned
|
543
|
+
task_store.reconnect_mysql
|
235
544
|
end
|
236
545
|
```
|
237
546
|
|
547
|
+
Procrastinator endeavours to be thread-safe and support concurrency, but this flexibility allows for many possible
|
548
|
+
combinations.
|
549
|
+
|
550
|
+
Expected use is a single process with one thread per queue. More complex use is possible but Procrastinator can't
|
551
|
+
guarantee concurrency in your Task Store.
|
552
|
+
|
553
|
+
#### PID Files
|
554
|
+
|
555
|
+
Process ID files are a single-line file that saves the daemon's process ID number. It's saved to the directory given
|
556
|
+
by `:pid_dir`. The default location is `pids/` relative to the file that called `#daemonized!`.
|
557
|
+
|
558
|
+
## Similar Tools
|
559
|
+
|
560
|
+
Procrastinator is a library that exists to enable job queues with flexibility in storage mechanism and minimal
|
561
|
+
dependencies. It's neat but it is specifically intended for smaller datasets. Some other approaches include:
|
562
|
+
|
563
|
+
### Linux etc: Cron and At
|
564
|
+
|
565
|
+
Consider [Cron](https://en.wikipedia.org/wiki/Cron) for tasks that run on a regular schedule.
|
566
|
+
|
567
|
+
Consider [At](https://en.wikipedia.org/wiki/At_(command)) for tasks that run once at a particular time.
|
568
|
+
|
569
|
+
While neither tool natively supports retry, they can be great solutions for simple situations.
|
570
|
+
|
571
|
+
### Gem: Resque
|
572
|
+
|
573
|
+
Consider [Resque](https://rubygems.org/gems/resque) for larger datasets (eg. 10,000+ jobs) where performance
|
574
|
+
optimization becomes relevant.
|
575
|
+
|
576
|
+
### Gem: Rails ActiveJob / DelayedJob
|
577
|
+
|
578
|
+
Consider [DelayedJob](https://rubygems.org/gems/delayed_job) for projects that are tightly integrated with Rails and
|
579
|
+
fully commit to live in that ecosystem.
|
580
|
+
|
238
581
|
## Contributing
|
239
|
-
|
582
|
+
|
583
|
+
Bug reports and pull requests are welcome on GitHub at
|
240
584
|
[https://github.com/TenjinInc/procrastinator](https://github.com/TenjinInc/procrastinator).
|
241
|
-
|
242
|
-
This project is intended to be a friendly space for collaboration, and contributors are expected to adhere to the
|
585
|
+
|
586
|
+
This project is intended to be a friendly space for collaboration, and contributors are expected to adhere to the
|
243
587
|
[Contributor Covenant](http://contributor-covenant.org) code of conduct.
|
244
588
|
|
245
|
-
|
246
|
-
|
589
|
+
Play nice.
|
590
|
+
|
591
|
+
### Developers
|
592
|
+
|
593
|
+
After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can
|
247
594
|
also run `bin/console` for an interactive prompt that will allow you to experiment.
|
248
595
|
|
249
|
-
To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the
|
250
|
-
version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version,
|
596
|
+
To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the
|
597
|
+
version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version,
|
251
598
|
push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
|
252
599
|
|
253
600
|
## License
|
601
|
+
|
254
602
|
The gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).
|