delayed 0.1.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +7 -0
- data/LICENSE +20 -0
- data/README.md +560 -0
- data/Rakefile +35 -0
- data/lib/delayed.rb +72 -0
- data/lib/delayed/active_job_adapter.rb +65 -0
- data/lib/delayed/backend/base.rb +166 -0
- data/lib/delayed/backend/job_preparer.rb +43 -0
- data/lib/delayed/exceptions.rb +14 -0
- data/lib/delayed/job.rb +250 -0
- data/lib/delayed/lifecycle.rb +85 -0
- data/lib/delayed/message_sending.rb +65 -0
- data/lib/delayed/monitor.rb +134 -0
- data/lib/delayed/performable_mailer.rb +22 -0
- data/lib/delayed/performable_method.rb +47 -0
- data/lib/delayed/plugin.rb +15 -0
- data/lib/delayed/plugins/connection.rb +13 -0
- data/lib/delayed/plugins/instrumentation.rb +39 -0
- data/lib/delayed/priority.rb +164 -0
- data/lib/delayed/psych_ext.rb +135 -0
- data/lib/delayed/railtie.rb +7 -0
- data/lib/delayed/runnable.rb +46 -0
- data/lib/delayed/serialization/active_record.rb +18 -0
- data/lib/delayed/syck_ext.rb +42 -0
- data/lib/delayed/tasks.rb +40 -0
- data/lib/delayed/worker.rb +233 -0
- data/lib/delayed/yaml_ext.rb +10 -0
- data/lib/delayed_job.rb +1 -0
- data/lib/delayed_job_active_record.rb +1 -0
- data/lib/generators/delayed/generator.rb +7 -0
- data/lib/generators/delayed/migration_generator.rb +28 -0
- data/lib/generators/delayed/next_migration_version.rb +14 -0
- data/lib/generators/delayed/templates/migration.rb +22 -0
- data/spec/autoloaded/clazz.rb +6 -0
- data/spec/autoloaded/instance_clazz.rb +5 -0
- data/spec/autoloaded/instance_struct.rb +6 -0
- data/spec/autoloaded/struct.rb +7 -0
- data/spec/database.yml +25 -0
- data/spec/delayed/active_job_adapter_spec.rb +267 -0
- data/spec/delayed/job_spec.rb +953 -0
- data/spec/delayed/monitor_spec.rb +276 -0
- data/spec/delayed/plugins/instrumentation_spec.rb +49 -0
- data/spec/delayed/priority_spec.rb +154 -0
- data/spec/delayed/serialization/active_record_spec.rb +15 -0
- data/spec/delayed/tasks_spec.rb +116 -0
- data/spec/helper.rb +196 -0
- data/spec/lifecycle_spec.rb +77 -0
- data/spec/message_sending_spec.rb +149 -0
- data/spec/performable_mailer_spec.rb +68 -0
- data/spec/performable_method_spec.rb +123 -0
- data/spec/psych_ext_spec.rb +94 -0
- data/spec/sample_jobs.rb +117 -0
- data/spec/worker_spec.rb +235 -0
- data/spec/yaml_ext_spec.rb +48 -0
- metadata +326 -0
checksums.yaml
ADDED
@@ -0,0 +1,7 @@
|
|
1
|
+
---
|
2
|
+
SHA256:
|
3
|
+
metadata.gz: 9ab35af5acde370d84971fff4e2aad4a6903a790fc42787917c0ee018a227077
|
4
|
+
data.tar.gz: 792234868782d73f6960127f06293020cd11ad32680074b6bcabee2db7a8a383
|
5
|
+
SHA512:
|
6
|
+
metadata.gz: 9af114801499f370343a41cffe87151509882d62973c61a0c7d82ecc4395ffbad35d15b1715309813dfacfb543d14b11e8d8ac829003fbf5892e1639dff7b387
|
7
|
+
data.tar.gz: c4e80bd8917d15eb9b4c67736571d349002945f9d8f621b5e7a2da5db179f6a7cbc18da09fa831af21cccbfacf1beec56bda3914d1d44c133d2fdb4858834c09
|
data/LICENSE
ADDED
@@ -0,0 +1,20 @@
|
|
1
|
+
Copyright (c) 2005 Tobias Lütke
|
2
|
+
|
3
|
+
Permission is hereby granted, free of charge, to any person obtaining
|
4
|
+
a copy of this software and associated documentation files (the
|
5
|
+
"Software"), to deal in the Software without restriction, including
|
6
|
+
without limitation the rights to use, copy, modify, merge, publish,
|
7
|
+
distribute, sublicense, and/or sell copies of the Software, and to
|
8
|
+
permit persons to whom the Software is furnished to do so, subject to
|
9
|
+
the following conditions:
|
10
|
+
|
11
|
+
The above copyright notice and this permission notice shall be
|
12
|
+
included in all copies or substantial portions of the Software.
|
13
|
+
|
14
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
|
15
|
+
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
|
16
|
+
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOa AND
|
17
|
+
NONINFRINGEMENT. IN NO EVENT SaALL THE AUTHORS OR COPYRIGHT HOLDERS BE
|
18
|
+
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
|
19
|
+
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
20
|
+
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
data/README.md
ADDED
@@ -0,0 +1,560 @@
|
|
1
|
+
Delayed
|
2
|
+
=======
|
3
|
+
[![Gem Version](https://badge.fury.io/rb/delayed.svg)](https://rubygems.org/gems/delayed)
|
4
|
+
![CI](https://github.com/Betterment/delayed/workflows/CI/badge.svg)
|
5
|
+
|
6
|
+
**`Delayed` is a multi-threaded, SQL-driven ActiveJob backend used at
|
7
|
+
[Betterment](https://betterment.com) to process millions of background jobs per day.**
|
8
|
+
|
9
|
+
It supports **postgres**, **mysql**, and **sqlite**, and is designed to be:
|
10
|
+
|
11
|
+
- **Reliable**, with co-transactional job enqueues and guaranteed, at-least-once execution
|
12
|
+
- **Scalable**, with an optimized pickup query and concurrent job execution
|
13
|
+
- **Resilient**, with built-in retry mechanisms, exponential back-off, and failed job preservation
|
14
|
+
- **Maintainable**, with robust instrumentation, continuous monitoring, and priority-based alerting
|
15
|
+
|
16
|
+
For an overview of how Betterment uses `delayed` to build resilience into distributed systems, check
|
17
|
+
out the talk ✨[Can I break this?](https://www.youtube.com/watch?v=TuhS13rBoVY)✨ given at RailsConf
|
18
|
+
2021!
|
19
|
+
|
20
|
+
|
21
|
+
### Why `Delayed`?
|
22
|
+
|
23
|
+
The `delayed` gem is a targeted fork of both `delayed_job` and `delayed_job_active_record`,
|
24
|
+
combining them into a single library. It is designed for applications with the kinds of operational
|
25
|
+
needs seen at Betterment, and includes numerous features extracted from Betterment's codebases, such
|
26
|
+
as:
|
27
|
+
|
28
|
+
- Multithreaded job execution via
|
29
|
+
[`concurrent-ruby`](https://github.com/ruby-concurrency/concurrent-ruby)
|
30
|
+
- A highly optimized, `SKIP LOCKED`-based pickup query (on postgres)
|
31
|
+
- Built-in instrumentation and continuous monitoring via a new `monitor` process
|
32
|
+
- Named priority ranges, defaulting to `:interactive`, `:user_visible`, `:eventual`, and `:reporting`
|
33
|
+
- Priority-based alerting threshholds for job age, run time, and attempts
|
34
|
+
- An experimental autoscaling metric, for use by a horizontal autoscaler (we use Kubernetes)
|
35
|
+
- A custom adapter that extends `ActiveJob` with `Delayed`-specific behaviors
|
36
|
+
|
37
|
+
This gem benefits immensely from the **many years** of development, maintenance, and community
|
38
|
+
support that have gone into `delayed_job`, and many of the core DJ APIs (like `.delay`) are still
|
39
|
+
available in `delayed` as undocumented features. Over time, these APIs may be removed as this gem
|
40
|
+
focuses itself around `ActiveJob`-based usage, but the aim will be to provide bidirectional
|
41
|
+
migration paths where possible.
|
42
|
+
|
43
|
+
## Table of Contents
|
44
|
+
|
45
|
+
* [Getting Started](#getting-started)
|
46
|
+
* [Basic Usage](#basic-usage)
|
47
|
+
* [Running a worker process](#running-a-worker-process)
|
48
|
+
* [Enqueuing Jobs](#enqueuing-jobs)
|
49
|
+
* [Operational Considerations](#operational-considerations)
|
50
|
+
* [Monitoring Jobs & Workers](#monitoring-jobs--workers)
|
51
|
+
* [Lifecycle Hooks](#lifecycle-hooks)
|
52
|
+
* [Priority-based Alerting Threshholds](#priority-based-alerting-threshholds)
|
53
|
+
* [Continuous Monitoring](#continuous-monitoring)
|
54
|
+
* [Configuration](#configuration)
|
55
|
+
* [Migrating from other ActiveJob backends](#migrating-from-other-activejob-backends)
|
56
|
+
* [Migrating from DelayedJob](#migrating-from-delayedjob)
|
57
|
+
* [How to Contribute](#how-to-contribute)
|
58
|
+
|
59
|
+
## Getting Started
|
60
|
+
|
61
|
+
This gem is designed to work with Rails 5.2+ and Ruby 2.6+ on postgres 9.5+ or mysql 5.6+
|
62
|
+
|
63
|
+
### Installation
|
64
|
+
|
65
|
+
Add the following to your Gemfile:
|
66
|
+
|
67
|
+
```
|
68
|
+
gem 'delayed'
|
69
|
+
```
|
70
|
+
|
71
|
+
Then run `bundle install`.
|
72
|
+
|
73
|
+
Before you can enqueue and run jobs, you will need a jobs table. You can create this table by
|
74
|
+
running the following command:
|
75
|
+
|
76
|
+
```bash
|
77
|
+
rails generate delayed:migration
|
78
|
+
rails db:migrate
|
79
|
+
```
|
80
|
+
|
81
|
+
Then, to use this background job processor with ActiveJob, add the following to your application config:
|
82
|
+
|
83
|
+
```ruby
|
84
|
+
config.active_job.queue_adapter = :delayed
|
85
|
+
```
|
86
|
+
|
87
|
+
See the [Rails guide](http://guides.rubyonrails.org/active_job_basics.html#setting-the-backend) for
|
88
|
+
more details.
|
89
|
+
|
90
|
+
## Basic Usage
|
91
|
+
|
92
|
+
### Running a worker process
|
93
|
+
|
94
|
+
In order for any jobs to execute, you must first start a worker process, which will work off jobs:
|
95
|
+
|
96
|
+
```
|
97
|
+
rake delayed:work
|
98
|
+
```
|
99
|
+
|
100
|
+
By default, a worker process will pick up 2 jobs at a time (ordered by priority) and run each in a
|
101
|
+
separate thread. To change the number of jobs picked up (and, in turn, increase the size of the
|
102
|
+
thread pool), use the `MAX_CLAIMS` environment variable:
|
103
|
+
|
104
|
+
```bash
|
105
|
+
MAX_CLAIMS=5 rake delayed:work
|
106
|
+
```
|
107
|
+
|
108
|
+
Work off specific queues by setting the `QUEUE` or `QUEUES` environment variable:
|
109
|
+
|
110
|
+
```bash
|
111
|
+
QUEUE=tracking rake delayed:work
|
112
|
+
QUEUES=mailers,tasks rake delayed:work
|
113
|
+
```
|
114
|
+
|
115
|
+
You can stop the worker with `CTRL-C` or by sending a `SIGTERM` signal to the process. The worker
|
116
|
+
will attempt to complete outstanding jobs and gracefully shutdown. Some platforms (like Heroku) will
|
117
|
+
send a `SIGKILL` after a designated timeout, which will immediately terminate the process and may
|
118
|
+
result in long-running jobs remaining locked until `Delayed::Worker.max_run_time` has elapsed. (By
|
119
|
+
default this is 20 minutes.)
|
120
|
+
|
121
|
+
### Enqueuing Jobs
|
122
|
+
|
123
|
+
The recommended usage of this gem is via `ActiveJob`. You can define a job like so:
|
124
|
+
|
125
|
+
```ruby
|
126
|
+
def MyJob < ApplicationJob
|
127
|
+
def perform(any: 'arguments')
|
128
|
+
# do something here
|
129
|
+
end
|
130
|
+
end
|
131
|
+
```
|
132
|
+
|
133
|
+
Then, enqueue the job with `perform_later`:
|
134
|
+
|
135
|
+
```ruby
|
136
|
+
MyJob.perform_later(arguments: 'go here')
|
137
|
+
```
|
138
|
+
|
139
|
+
Jobs will be enqueued to the `delayed_jobs` table, which can be accessed via
|
140
|
+
the `Delayed::Job` ActiveRecord model using standard ActiveRecord query methods
|
141
|
+
(`.find`, `.where`, etc).
|
142
|
+
|
143
|
+
To override specific columns or parameters of the job, use `set`:
|
144
|
+
|
145
|
+
```ruby
|
146
|
+
MyJob.set(priority: 11).perform_later(some_more: 'arguments')
|
147
|
+
MyJob.set(queue: 'video_encoding').perform_later(video)
|
148
|
+
MyJob.set(wait: 3.hours).perform_later
|
149
|
+
MyJob.set(wait_until: 1.day.from_now).perform_later
|
150
|
+
```
|
151
|
+
|
152
|
+
Priority ranges are mapped to configurable shorthand names:
|
153
|
+
|
154
|
+
```ruby
|
155
|
+
MyJob.set(priority: :interactive).perform_later
|
156
|
+
MyJob.set(priority: :user_visible).perform_later
|
157
|
+
MyJob.set(priority: :eventual).perform_later
|
158
|
+
MyJob.set(priority: :reporting).perform_later
|
159
|
+
|
160
|
+
Delayed::Job.last.priority.user_visible? # => false
|
161
|
+
Delayed::Priority.new(99).reporting? # => true
|
162
|
+
Delayed::Priority.new(11).to_i # => 11
|
163
|
+
Delayed::Priority.new(3).to_s # => 'interactive'
|
164
|
+
```
|
165
|
+
|
166
|
+
**To change the default priority names, or to adjust other aspects of job
|
167
|
+
execution, see the [Configuration](#configuration) section below.**
|
168
|
+
|
169
|
+
#### Other ActiveJob Features
|
170
|
+
|
171
|
+
All other ActiveJob features should work out of the box, such as the `queue_as`
|
172
|
+
and `queue_with_priority` class-level directives:
|
173
|
+
|
174
|
+
```ruby
|
175
|
+
class MyJob < ApplicationJob
|
176
|
+
queue_as 'some_other_queue'
|
177
|
+
queue_with_priority 42
|
178
|
+
|
179
|
+
# ...
|
180
|
+
end
|
181
|
+
```
|
182
|
+
|
183
|
+
ActiveJob also supports the following lifecycle hooks:
|
184
|
+
|
185
|
+
- [before_enqueue](https://edgeapi.rubyonrails.org/classes/ActiveJob/Callbacks/ClassMethods.html#method-i-before_enqueue)
|
186
|
+
- [around_enqueue](https://edgeapi.rubyonrails.org/classes/ActiveJob/Callbacks/ClassMethods.html#method-i-around_enqueue)
|
187
|
+
- [after_enqueue](https://edgeapi.rubyonrails.org/classes/ActiveJob/Callbacks/ClassMethods.html#method-i-after_enqueue)
|
188
|
+
- [before_perform](https://edgeapi.rubyonrails.org/classes/ActiveJob/Callbacks/ClassMethods.html#method-i-before_perform)
|
189
|
+
- [around_perform](https://edgeapi.rubyonrails.org/classes/ActiveJob/Callbacks/ClassMethods.html#method-i-around_perform)
|
190
|
+
- [after_perform](https://edgeapi.rubyonrails.org/classes/ActiveJob/Callbacks/ClassMethods.html#method-i-after_perform)
|
191
|
+
|
192
|
+
**Read more about ActiveJob usage on the [Active Job
|
193
|
+
Basics](https://guides.rubyonrails.org/active_job_basics.html) documentation page.**
|
194
|
+
|
195
|
+
|
196
|
+
## Operational Considerations
|
197
|
+
|
198
|
+
`Delayed` has been shaped around Betterment's day-to-day operational needs. In order to benefit from
|
199
|
+
these design decisions, there are a few things you'll want to keep in mind.
|
200
|
+
|
201
|
+
#### Co-transactionality
|
202
|
+
|
203
|
+
The `:delayed` job backend is designed for **co-transactional** job enqueues. This means that you
|
204
|
+
can safely enqueue jobs inside of [ACID](https://en.wikipedia.org/wiki/ACID)-compliant business
|
205
|
+
operations, like so:
|
206
|
+
|
207
|
+
```ruby
|
208
|
+
def save
|
209
|
+
ActiveRecord::Base.transaction do
|
210
|
+
user.lock!
|
211
|
+
|
212
|
+
if user.update(email: new_email)
|
213
|
+
EmailChangeJob.perform_later(user, new_email, old_email)
|
214
|
+
|
215
|
+
true
|
216
|
+
else
|
217
|
+
false
|
218
|
+
end
|
219
|
+
end
|
220
|
+
end
|
221
|
+
```
|
222
|
+
|
223
|
+
If the transaction rolls back, the enqueued job will _also_ roll back, ensuring that the entire
|
224
|
+
operation is all-or-nothing. A job will never become visible to a worker until the transaction
|
225
|
+
commits.
|
226
|
+
|
227
|
+
Important: the above assumes that the connection used by the transaction is the one provided by
|
228
|
+
`ActiveRecord::Base`. (Support for enqueuing jobs via other database connections is possible, but
|
229
|
+
is not yet exposed as a configuration.)
|
230
|
+
|
231
|
+
#### At-Least-Once Delivery
|
232
|
+
|
233
|
+
Each job is guaranteed to run _at least once_, but under certain conditions may run more than once.
|
234
|
+
As such, you'll want to ensure that your jobs are
|
235
|
+
[idempotent](https://en.wikipedia.org/wiki/Idempotence), meaning they can be safely repeated,
|
236
|
+
regardless of the outcome of any prior attempts.
|
237
|
+
|
238
|
+
#### When Jobs Fail
|
239
|
+
|
240
|
+
Unlike other job queue backends, `delayed` will **not** delete failing jobs by default. These are
|
241
|
+
jobs that have reached their `max_attempts` (25 by default), and they will remain in the queue until
|
242
|
+
you manually intervene.
|
243
|
+
|
244
|
+
The general idea is that you should treat these as operational issues (like an error on your
|
245
|
+
bugtracker), and you should aim to resolve the issue by making the job succeed. This might involve
|
246
|
+
shipping a bugfix, making a data change, or updating the job's implementation to handle certain
|
247
|
+
corner cases more gracefully (perhaps by no-opping). When you're ready to re-run, you may clear the
|
248
|
+
`failed_at` column and reset `attempts` to 0:
|
249
|
+
|
250
|
+
```ruby
|
251
|
+
Delayed::Job.find(failing_job_id).update!(failed_at: nil, attempts: 0, run_at: Time.zone.now)
|
252
|
+
```
|
253
|
+
|
254
|
+
## Monitoring Jobs & Workers
|
255
|
+
|
256
|
+
`Delayed` will emit `ActiveSupport::Notification`s at various points during job and worker
|
257
|
+
lifecycles, and can also be configured for continuious monitoring. You are strongly encouraged to
|
258
|
+
tie these up to your preferred application monitoring solution by calling
|
259
|
+
`ActiveSupport::Notification.subscribe` in an initializer.
|
260
|
+
|
261
|
+
### Lifecycle Hooks
|
262
|
+
|
263
|
+
The following events will be emitted automatically by workers as jobs are reserved and performed:
|
264
|
+
|
265
|
+
- **delayed.job.run** - an event measuring the duration of a job's execution
|
266
|
+
- **delayed.job.error** - an event indicating that a job has errored and may be retried (no duration attached)
|
267
|
+
- **delayed.job.failure** - an event indicating that a job has permanently failed (no duration attached)
|
268
|
+
- **delayed.job.enqueue** - an event measuring the time it takes to enqueue a job
|
269
|
+
- **delayed.worker.reserve_jobs** - an event measuring the duration of the job "pickup query"
|
270
|
+
|
271
|
+
The "run", "error", "failure" and "enqueue" events will include a `:job` argument in the event's payload,
|
272
|
+
providing access to the job instance.
|
273
|
+
|
274
|
+
```ruby
|
275
|
+
ActiveSupport::Notifications.subscribe('delayed.job.run') do |*args|
|
276
|
+
event = ActiveSupport::Notifications::Event.new(*args)
|
277
|
+
|
278
|
+
# Emit the event via your preferred metrics/instrumentation provider:
|
279
|
+
tags = event.payload.except(:job).map { |k,v| "#{k.to_s[0..64]}:#{v.to_s[0..255]}" }
|
280
|
+
StatsD.distribution(event.name, event.duration, tags: tags)
|
281
|
+
end
|
282
|
+
|
283
|
+
ActiveSupport::Notifications.subscribe(/delayed\.job\.(error|failure)/) do |*args|
|
284
|
+
# ...
|
285
|
+
Statsd.increment(...)
|
286
|
+
end
|
287
|
+
|
288
|
+
ActiveSupport::Notifications.subscribe('delayed.job.enqueue') do |*args|
|
289
|
+
# ...
|
290
|
+
StatsD.distribution(...)
|
291
|
+
end
|
292
|
+
|
293
|
+
ActiveSupport::Notifications.subscribe('delayed.worker.reserve_jobs') do |*args|
|
294
|
+
# ...
|
295
|
+
StatsD.distribution(...)
|
296
|
+
end
|
297
|
+
```
|
298
|
+
|
299
|
+
### Priority-based Alerting Threshholds
|
300
|
+
|
301
|
+
By default, jobs support "alerting threshholds" that allow them to warn if they
|
302
|
+
come within range of `max_run_time` or `max_attempts` (without exceeding them),
|
303
|
+
or if they spend too long waiting in the queue (i.e. their "age").
|
304
|
+
|
305
|
+
The threshholds are fully configurable, and default to the following values:
|
306
|
+
|
307
|
+
```ruby
|
308
|
+
Delayed::Priority.alerts = {
|
309
|
+
interactive: { age: 1.minute, run_time: 30.seconds, attempts: 3 },
|
310
|
+
user_visible: { age: 3.minutes, run_time: 90.seconds, attempts: 5 },
|
311
|
+
eventual: { age: 1.5.hours, run_time: 5.minutes, attempts: 8 },
|
312
|
+
reporting: { age: 4.hours, run_time: 10.minutes, attempts: 8 },
|
313
|
+
}
|
314
|
+
```
|
315
|
+
|
316
|
+
These may also be configured on a per-job basis:
|
317
|
+
|
318
|
+
```ruby
|
319
|
+
class MyVeryHighThroughputJob < ApplicationJob
|
320
|
+
# ...
|
321
|
+
|
322
|
+
def alert_run_time
|
323
|
+
5.seconds # must execute in under 5 seconds
|
324
|
+
end
|
325
|
+
|
326
|
+
def alert_attempts
|
327
|
+
1 # will begin alerting after 1 attempt
|
328
|
+
end
|
329
|
+
end
|
330
|
+
```
|
331
|
+
|
332
|
+
If a job completes but was uncomfortably close to timing-out, it may make sense
|
333
|
+
to emit an alert:
|
334
|
+
|
335
|
+
```ruby
|
336
|
+
ActiveSupport::Notifications.subscribe('delayed.job.run') do |_name, _start, _finish, _id, payload|
|
337
|
+
job = payload[:job]
|
338
|
+
TeamAlerter.alert!("Job with ID #{job.id} took #{job.run_time} seconds to run") if job.run_time_alert?
|
339
|
+
end
|
340
|
+
```
|
341
|
+
|
342
|
+
Similarly, if a job is erroring repeatedly, you may choose to emit some form of
|
343
|
+
notification before it reaches its full attempt count:
|
344
|
+
|
345
|
+
```ruby
|
346
|
+
ActiveSupport::Notifications.subscribe('delayed.job.error') do |_name, _start, _finish, _id, payload|
|
347
|
+
job = payload[:job]
|
348
|
+
TeamAlerter.alert!("Job with ID #{job.id} has made #{job.attempts} attempts") if job.attempts_alert?
|
349
|
+
end
|
350
|
+
```
|
351
|
+
|
352
|
+
The last threshhold (`job.age_alert?`) refers to the time spent in the queue,
|
353
|
+
and may be best monitored in aggregate (covered in the next section!), as it
|
354
|
+
generally describes the ability of workers to pick up jobs fast enough.
|
355
|
+
|
356
|
+
### Continuous Monitoring
|
357
|
+
|
358
|
+
To continuously monitor the state of your job queues, you may run a single "monitor" process
|
359
|
+
alongside your workers. (Only one instance of this process is needed, as it will emit aggregate
|
360
|
+
metrics.)
|
361
|
+
|
362
|
+
```
|
363
|
+
rake delayed:monitor
|
364
|
+
```
|
365
|
+
|
366
|
+
The monitor process accepts the same queue configurations as the worker process, and can be used to
|
367
|
+
monitor the same sets of queues as the workers:
|
368
|
+
|
369
|
+
```bash
|
370
|
+
QUEUE=tracking rake delayed:monitor
|
371
|
+
QUEUES=mailers,tasks rake delayed:monitor
|
372
|
+
```
|
373
|
+
|
374
|
+
The following events will be emitted, grouped by priority name (e.g. "interactive") and queue name,
|
375
|
+
and the metric's "`:value`" will be available in the event's payload. **This means that there will
|
376
|
+
be one value _per_ unique combination of queue & priority**, and totals must be computed via
|
377
|
+
downstream aggregation (e.g. as a StatsD "gauge" metric).
|
378
|
+
|
379
|
+
- **delayed.job.count** - the total number of jobs
|
380
|
+
- **delayed.job.future_count** - jobs where run_at is in the future
|
381
|
+
- **delayed.job.working_count** - jobs that are currently being worked off (excludes failed jobs)
|
382
|
+
- **delayed.job.workable_count** - jobs that are waiting to be worked off
|
383
|
+
- **delayed.job.erroring_count** - jobs where attempts > 0
|
384
|
+
- **delayed.job.failed_count** - jobs where failed_at is not nil
|
385
|
+
- **delayed.job.max_lock_age** - the age of the oldest locked_at value (excludes failed jobs)
|
386
|
+
- **delayed.job.max_age** - the age of the oldest run_at value (excludes failed jobs)
|
387
|
+
|
388
|
+
An additional _experimental_ metric is available, intended for use with application autoscaling:
|
389
|
+
|
390
|
+
- **delayed.job.alert_age_percent** - the _percent_ to which the oldest job has reached the "age alert" threshold. (See the [Alerting Threshholds](#priority-based-alerting-threshholds) section above.)
|
391
|
+
|
392
|
+
All of these events may be subscribed to via a single regular expression (again, in your application
|
393
|
+
config or in an initializer):
|
394
|
+
|
395
|
+
```ruby
|
396
|
+
ActiveSupport::Notifications.subscribe(/delayed\.job\..*_(count|age|percent)/) do |*args|
|
397
|
+
event = ActiveSupport::Notifications::Event.new(*args)
|
398
|
+
value = event.payload.delete(:value)
|
399
|
+
|
400
|
+
# Emit the event via your preferred metrics/instrumentation provider:
|
401
|
+
tags = event.payload.map { |k,v| "#{k.to_s[0..64]}:#{v.to_s[0..255]}" }
|
402
|
+
StatsD.gauge(event.name, value, sample_rate: 1.0, tags: tags)
|
403
|
+
end
|
404
|
+
```
|
405
|
+
|
406
|
+
Additionally, the monitor process with emit a **delayed.monitor.run** event with a duration
|
407
|
+
attached, so that you can monitor the time it takes to emit these aggregate metrics.
|
408
|
+
|
409
|
+
```ruby
|
410
|
+
ActiveSupport::Notifications.subscribe('delayed.monitor.run') do |*args|
|
411
|
+
# ...
|
412
|
+
StatsD.distribution(...)
|
413
|
+
end
|
414
|
+
```
|
415
|
+
|
416
|
+
## Configuration
|
417
|
+
|
418
|
+
`Delayed` is highly configurable, but ships with opinionated defaults. If you need to change any
|
419
|
+
default behaviors, you can do so in an initializer (e.g. `config/initializers/delayed.rb`).
|
420
|
+
|
421
|
+
By default, workers will claim 5 jobs at a time (run in concurrent threads). If no jobs are found,
|
422
|
+
workers will sleep for 5 seconds.
|
423
|
+
|
424
|
+
```ruby
|
425
|
+
# The max number of jobs a worker may lock at a time (also the size of the thread pool):
|
426
|
+
Delayed::Worker.max_claims = 5
|
427
|
+
|
428
|
+
# The number of jobs to which a worker may "read ahead" when locking jobs (mysql only!):
|
429
|
+
Delayed::Worker.read_ahead = 5
|
430
|
+
|
431
|
+
# If a worker finds no jobs, it will sleep this number of seconds in between attempts:
|
432
|
+
Delayed::Worker.sleep_delay = 5
|
433
|
+
```
|
434
|
+
|
435
|
+
If a job fails, it will be rerun up to 25 times (with an exponential back-off). Jobs will also
|
436
|
+
time-out after 20 minutes.
|
437
|
+
|
438
|
+
```ruby
|
439
|
+
# The max number of attempts jobs are given before they are permanently marked as failed:
|
440
|
+
Delayed::Worker.max_attempts = 25
|
441
|
+
|
442
|
+
# The max amount of time a job is allowed to run before it is stopped:
|
443
|
+
Delayed::Worker.max_run_time = 20.minutes
|
444
|
+
```
|
445
|
+
|
446
|
+
Individual jobs may specify their own `max_attempts` and `max_run_time`:
|
447
|
+
|
448
|
+
```ruby
|
449
|
+
class MyJob < ApplicationJob
|
450
|
+
def perform; end
|
451
|
+
|
452
|
+
def max_run_time
|
453
|
+
15.minutes # must be less than the global `max_run_time` default!
|
454
|
+
end
|
455
|
+
|
456
|
+
def max_attempts
|
457
|
+
1
|
458
|
+
end
|
459
|
+
end
|
460
|
+
```
|
461
|
+
|
462
|
+
By default, workers will work off all queues (including `nil`), and jobs will be enqueued to a
|
463
|
+
`'default'` queue.
|
464
|
+
|
465
|
+
```ruby
|
466
|
+
# A list of queues to which all work is restricted. (e.g. `%w(queue1 queue2 queue3)`)
|
467
|
+
# If no queues are specified, then all queues will be worked off
|
468
|
+
Delayed::Worker.queues = []
|
469
|
+
|
470
|
+
# The default queue that jobs will be enqueued to, when no other queue is specified:
|
471
|
+
Delayed::Worker.default_queue_name = 'default'
|
472
|
+
```
|
473
|
+
|
474
|
+
Priority ranges are given names. These will default to "interactive" for 0-9, "user visible" for
|
475
|
+
10-19, "eventual" for 20-29, and "reporting" for 30+. The default priority for enqueued jobs is
|
476
|
+
"user visible" (10), and workers will work off all priorities, unless otherwise configured.
|
477
|
+
|
478
|
+
```ruby
|
479
|
+
# Default priority names, useful for enqueuing and for instrumentation/metrics.
|
480
|
+
Delayed::Priority.names = { interactive: 0, user_visible: 10, eventual: 20, reporting: 30 }
|
481
|
+
|
482
|
+
# The default priority for enqueued jobs, when no priority is specified.
|
483
|
+
# This aligns with the "user_visible" named priority.
|
484
|
+
Delayed::Worker.default_priority = 10
|
485
|
+
|
486
|
+
# A worker can also be told to work off specific priority ranges,
|
487
|
+
# if, say, you'd like a dedicated worker for high priority jobs:
|
488
|
+
Delayed::Worker.min_priority = nil
|
489
|
+
Delayed::Worker.max_priority = nil
|
490
|
+
```
|
491
|
+
|
492
|
+
Logging verbosity is also configurable. The gem will attempt to default to `Rails.logger` with an
|
493
|
+
"info" log level.
|
494
|
+
|
495
|
+
```ruby
|
496
|
+
# Specify an alternate logger class:
|
497
|
+
Delayed.logger = Rails.logger
|
498
|
+
|
499
|
+
# Specify a default log level for all job lifecycle logging:
|
500
|
+
Delayed.default_log_level = 'info'
|
501
|
+
```
|
502
|
+
|
503
|
+
## Migrating from other ActiveJob backends
|
504
|
+
|
505
|
+
For the most part, standard ActiveJob APIs should be fully compatible. However, when migrating from
|
506
|
+
a Redis-backed queue (or some other queue that is not co-located with your ActiveRecord data), the
|
507
|
+
[Operational Considerations](#operational-considerations) section of this README should be noted.
|
508
|
+
You may wish to change the way that jobs are enqueued and executed in order to benefit from
|
509
|
+
co-transactional / ACID guarantees.
|
510
|
+
|
511
|
+
To assist in migrating, you are encouraged to set `queue_adapter` on a per-job basis, so that you
|
512
|
+
can move and monitor fewer job classes at a time:
|
513
|
+
|
514
|
+
```ruby
|
515
|
+
class NewsletterJob < ApplicationJob
|
516
|
+
self.queue_adapter = :sidekiq
|
517
|
+
end
|
518
|
+
|
519
|
+
class OrderPurchaseJob < ApplicationJob
|
520
|
+
self.queue_adapter = :delayed
|
521
|
+
end
|
522
|
+
```
|
523
|
+
|
524
|
+
#### Migrating from DelayedJob
|
525
|
+
|
526
|
+
If you choose to use `delayed` in an app that was originally written against `delayed_job`, several
|
527
|
+
non-ActiveJob APIs are still available. These include "plugins", lifecycle hooks, and the `.delay`
|
528
|
+
and `.handle_asynchronously` methods. **These APIs are intended to assist in migrating older
|
529
|
+
codebases onto `ActiveJob`**, and may eventually be removed or extracted into an optional gem.
|
530
|
+
|
531
|
+
For comprehensive information on the APIs and features that `delayed` has inherited from
|
532
|
+
`delayed_job` and `delayed_job_active_record`, refer to [DelayedJob's
|
533
|
+
documentation](https://github.com/collectiveidea/delayed_job).
|
534
|
+
|
535
|
+
When migrating from `delayed_job`, you may choose to manually apply its default configurations:
|
536
|
+
|
537
|
+
```ruby
|
538
|
+
Delayed::Worker.max_run_time = 4.hours
|
539
|
+
Delayed::Worker.default_priority = 0
|
540
|
+
Delayed::Worker.default_queue_name = nil
|
541
|
+
Delayed::Worker.destroy_failed_jobs = true # WARNING: This will irreversably delete jobs.
|
542
|
+
```
|
543
|
+
|
544
|
+
Note that some configurations, like `queue_attributes`, `exit_on_complete`, `backend`, and
|
545
|
+
`raise_signal_exceptions` have been removed entirely.
|
546
|
+
|
547
|
+
## How to Contribute
|
548
|
+
|
549
|
+
We would love for you to contribute! Anything that benefits the majority of users—from a
|
550
|
+
documentation fix to an entirely new feature—is encouraged.
|
551
|
+
|
552
|
+
Before diving in, [check our issue tracker](//github.com/Betterment/delayed/issues) and consider
|
553
|
+
creating a new issue to get early feedback on your proposed change.
|
554
|
+
|
555
|
+
### Suggested Workflow
|
556
|
+
|
557
|
+
* Fork the project and create a new branch for your contribution.
|
558
|
+
* Write your contribution (and any applicable test coverage).
|
559
|
+
* Make sure all tests pass (`bundle exec rake`).
|
560
|
+
* Submit a pull request.
|