good_job 1.2.2 β 1.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +111 -2
- data/README.md +324 -160
- data/engine/app/controllers/good_job/active_jobs_controller.rb +8 -0
- data/engine/app/controllers/good_job/base_controller.rb +5 -0
- data/engine/app/controllers/good_job/dashboards_controller.rb +50 -0
- data/engine/app/helpers/good_job/application_helper.rb +4 -0
- data/engine/app/views/assets/_style.css.erb +16 -0
- data/engine/app/views/good_job/active_jobs/show.html.erb +1 -0
- data/engine/app/views/good_job/dashboards/index.html.erb +19 -0
- data/engine/app/views/layouts/good_job/base.html.erb +61 -0
- data/engine/app/views/shared/_chart.erb +51 -0
- data/engine/app/views/shared/_jobs_table.erb +26 -0
- data/engine/app/views/vendor/bootstrap/_bootstrap-native.js.erb +1662 -0
- data/engine/app/views/vendor/bootstrap/_bootstrap.css.erb +10258 -0
- data/engine/app/views/vendor/chartist/_chartist.css.erb +613 -0
- data/engine/app/views/vendor/chartist/_chartist.js.erb +4516 -0
- data/engine/config/routes.rb +4 -0
- data/engine/lib/good_job/engine.rb +5 -0
- data/lib/active_job/queue_adapters/good_job_adapter.rb +3 -2
- data/lib/generators/good_job/install_generator.rb +8 -0
- data/lib/good_job.rb +59 -27
- data/lib/good_job/adapter.rb +38 -0
- data/lib/good_job/cli.rb +66 -13
- data/lib/good_job/configuration.rb +61 -2
- data/lib/good_job/job.rb +126 -36
- data/lib/good_job/lockable.rb +119 -6
- data/lib/good_job/log_subscriber.rb +70 -6
- data/lib/good_job/multi_scheduler.rb +6 -0
- data/lib/good_job/notifier.rb +55 -29
- data/lib/good_job/performer.rb +38 -0
- data/lib/good_job/railtie.rb +1 -0
- data/lib/good_job/scheduler.rb +48 -40
- data/lib/good_job/version.rb +2 -1
- metadata +163 -7
- data/lib/good_job/pg_locks.rb +0 -21
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 17fac9485f08bf55c8d2b2fbbc6e85efde95202d759d6155a4b23bf0755236ce
|
|
4
|
+
data.tar.gz: 562115863243ac98e3fd0cca7c04ad9f57a80f932682d7161e43336b521ea62d
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: a06ee5d2bea077e7b612f6d44aa30666d52517849e5345d8882a9f21653513be2386e577dbbab8ad41897aeb48d14885cbfe8cd51784c96da300aee7233a5190
|
|
7
|
+
data.tar.gz: 7c366b21a4f28d66ce8d59d22fa56a081a63c181ee0b35ddd7965422c7c758f020bc7fba42aa34e61ec8abb17e9dcc5a9d0cb3b15f2d1c03d7153d83d7fbffdd
|
data/CHANGELOG.md
CHANGED
|
@@ -1,6 +1,115 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
-
## [v1.
|
|
3
|
+
## [v1.3.0](https://github.com/bensheldon/good_job/tree/v1.3.0) (2020-10-03)
|
|
4
|
+
|
|
5
|
+
[Full Changelog](https://github.com/bensheldon/good_job/compare/v1.2.6...v1.3.0)
|
|
6
|
+
|
|
7
|
+
**Merged pull requests:**
|
|
8
|
+
|
|
9
|
+
- Lengthen default poll interval from 1 to 5 seconds [\#156](https://github.com/bensheldon/good_job/pull/156) ([bensheldon](https://github.com/bensheldon))
|
|
10
|
+
- Rename reperform\_jobs\_on\_standard\_error to retry\_on\_unhandled\_error [\#154](https://github.com/bensheldon/good_job/pull/154) ([morgoth](https://github.com/morgoth))
|
|
11
|
+
|
|
12
|
+
## [v1.2.6](https://github.com/bensheldon/good_job/tree/v1.2.6) (2020-09-29)
|
|
13
|
+
|
|
14
|
+
[Full Changelog](https://github.com/bensheldon/good_job/compare/v1.2.5...v1.2.6)
|
|
15
|
+
|
|
16
|
+
**Implemented enhancements:**
|
|
17
|
+
|
|
18
|
+
- Preserve only failed jobs [\#136](https://github.com/bensheldon/good_job/issues/136)
|
|
19
|
+
- Add `GoodJob.preserve\_job\_records = :on\_unhandled\_error` option to only preserve jobs that errored [\#145](https://github.com/bensheldon/good_job/pull/145) ([morgoth](https://github.com/morgoth))
|
|
20
|
+
|
|
21
|
+
**Fixed bugs:**
|
|
22
|
+
|
|
23
|
+
- Fix LogSubscriber notifications for finished\_timer\_task and finished\_job\_task [\#148](https://github.com/bensheldon/good_job/pull/148) ([bensheldon](https://github.com/bensheldon))
|
|
24
|
+
|
|
25
|
+
**Closed issues:**
|
|
26
|
+
|
|
27
|
+
- run-once guarantee? [\#151](https://github.com/bensheldon/good_job/issues/151)
|
|
28
|
+
|
|
29
|
+
**Merged pull requests:**
|
|
30
|
+
|
|
31
|
+
- Add info how to setup basic auth for engine [\#153](https://github.com/bensheldon/good_job/pull/153) ([morgoth](https://github.com/morgoth))
|
|
32
|
+
- Add documentation for Dashboard Rails::Engine [\#149](https://github.com/bensheldon/good_job/pull/149) ([bensheldon](https://github.com/bensheldon))
|
|
33
|
+
- Style cleanup to Job error handling [\#147](https://github.com/bensheldon/good_job/pull/147) ([bensheldon](https://github.com/bensheldon))
|
|
34
|
+
- Replace gerund titles in Readme [\#146](https://github.com/bensheldon/good_job/pull/146) ([bensheldon](https://github.com/bensheldon))
|
|
35
|
+
- Only allow Scheduler to be initialized with max\_threads and poll\_interval; remove full access to pool and timer\_task options [\#137](https://github.com/bensheldon/good_job/pull/137) ([bensheldon](https://github.com/bensheldon))
|
|
36
|
+
|
|
37
|
+
## [v1.2.5](https://github.com/bensheldon/good_job/tree/v1.2.5) (2020-09-17)
|
|
38
|
+
|
|
39
|
+
[Full Changelog](https://github.com/bensheldon/good_job/compare/v1.2.4...v1.2.5)
|
|
40
|
+
|
|
41
|
+
**Implemented enhancements:**
|
|
42
|
+
|
|
43
|
+
- Use Zeitwerk for auto-loading [\#87](https://github.com/bensheldon/good_job/issues/87)
|
|
44
|
+
- Spike on data dashboard; pull in full Bootstrap CSS and JS [\#131](https://github.com/bensheldon/good_job/pull/131) ([bensheldon](https://github.com/bensheldon))
|
|
45
|
+
|
|
46
|
+
**Fixed bugs:**
|
|
47
|
+
|
|
48
|
+
- `poll-interval=-1` does not disable polling as intended [\#133](https://github.com/bensheldon/good_job/issues/133)
|
|
49
|
+
- Update Gemspec to reflect that GoodJob is not compatible with Rails 5.1 [\#143](https://github.com/bensheldon/good_job/pull/143) ([bensheldon](https://github.com/bensheldon))
|
|
50
|
+
- Prevent jobs hanging [\#141](https://github.com/bensheldon/good_job/pull/141) ([morgoth](https://github.com/morgoth))
|
|
51
|
+
- Add explicit require\_paths to gemspec for engine [\#134](https://github.com/bensheldon/good_job/pull/134) ([bensheldon](https://github.com/bensheldon))
|
|
52
|
+
- Use `connection.quote\_table\_name` and add spacing for SQL concatenation [\#124](https://github.com/bensheldon/good_job/pull/124) ([bensheldon](https://github.com/bensheldon))
|
|
53
|
+
|
|
54
|
+
**Closed issues:**
|
|
55
|
+
|
|
56
|
+
- Lint - Introduce line character limits [\#122](https://github.com/bensheldon/good_job/issues/122)
|
|
57
|
+
- Jobs are not processed in multi schema setup. Apartment + GoodJob \( post 1.1.2 \) [\#117](https://github.com/bensheldon/good_job/issues/117)
|
|
58
|
+
- Host a documentation sprint [\#48](https://github.com/bensheldon/good_job/issues/48)
|
|
59
|
+
|
|
60
|
+
**Merged pull requests:**
|
|
61
|
+
|
|
62
|
+
- Test GoodJob against Rails HEAD [\#144](https://github.com/bensheldon/good_job/pull/144) ([bensheldon](https://github.com/bensheldon))
|
|
63
|
+
- Drop Ruby 2.4 support [\#142](https://github.com/bensheldon/good_job/pull/142) ([morgoth](https://github.com/morgoth))
|
|
64
|
+
- Remove arguments from perform method [\#140](https://github.com/bensheldon/good_job/pull/140) ([morgoth](https://github.com/morgoth))
|
|
65
|
+
- Extract "execute" method to reduce "perform" method complexity [\#138](https://github.com/bensheldon/good_job/pull/138) ([morgoth](https://github.com/morgoth))
|
|
66
|
+
- Correct example on how to configure multiple queues by command line. [\#135](https://github.com/bensheldon/good_job/pull/135) ([morgoth](https://github.com/morgoth))
|
|
67
|
+
- Update ActionMailer Job class, to match the default [\#130](https://github.com/bensheldon/good_job/pull/130) ([morgoth](https://github.com/morgoth))
|
|
68
|
+
- Add initial Engine scaffold [\#125](https://github.com/bensheldon/good_job/pull/125) ([bensheldon](https://github.com/bensheldon))
|
|
69
|
+
- Zeitwerk Loader Implementation [\#123](https://github.com/bensheldon/good_job/pull/123) ([gadimbaylisahil](https://github.com/gadimbaylisahil))
|
|
70
|
+
- Update code-level documentation [\#111](https://github.com/bensheldon/good_job/pull/111) ([bensheldon](https://github.com/bensheldon))
|
|
71
|
+
|
|
72
|
+
## [v1.2.4](https://github.com/bensheldon/good_job/tree/v1.2.4) (2020-09-01)
|
|
73
|
+
|
|
74
|
+
[Full Changelog](https://github.com/bensheldon/good_job/compare/v1.2.3...v1.2.4)
|
|
75
|
+
|
|
76
|
+
**Implemented enhancements:**
|
|
77
|
+
|
|
78
|
+
- Add environment variable to mirror `cleanup\_preserved\_jobs --before-seconds-ago=SECONDS` [\#110](https://github.com/bensheldon/good_job/issues/110)
|
|
79
|
+
- Allow env variable config for cleanups [\#114](https://github.com/bensheldon/good_job/pull/114) ([gadimbaylisahil](https://github.com/gadimbaylisahil))
|
|
80
|
+
|
|
81
|
+
**Fixed bugs:**
|
|
82
|
+
|
|
83
|
+
- Better table name detection for Job queries [\#119](https://github.com/bensheldon/good_job/pull/119) ([gadimbaylisahil](https://github.com/gadimbaylisahil))
|
|
84
|
+
|
|
85
|
+
**Closed issues:**
|
|
86
|
+
|
|
87
|
+
- Remove unused PgLocks class [\#121](https://github.com/bensheldon/good_job/issues/121)
|
|
88
|
+
- Fix minor issue with CommandLine option links in README.md [\#116](https://github.com/bensheldon/good_job/issues/116)
|
|
89
|
+
- Unused .advisory\_lock\_details in PgLocks [\#105](https://github.com/bensheldon/good_job/issues/105)
|
|
90
|
+
|
|
91
|
+
**Merged pull requests:**
|
|
92
|
+
|
|
93
|
+
- Remove unused PgLocks class [\#120](https://github.com/bensheldon/good_job/pull/120) ([gadimbaylisahil](https://github.com/gadimbaylisahil))
|
|
94
|
+
- Fix readme CommandLine option links [\#115](https://github.com/bensheldon/good_job/pull/115) ([gadimbaylisahil](https://github.com/gadimbaylisahil))
|
|
95
|
+
- Have YARD render markdown files with GFM \(Github Flavored Markdown\) [\#113](https://github.com/bensheldon/good_job/pull/113) ([bensheldon](https://github.com/bensheldon))
|
|
96
|
+
- Add markdownlint to lint readme [\#109](https://github.com/bensheldon/good_job/pull/109) ([bensheldon](https://github.com/bensheldon))
|
|
97
|
+
- Remove unused method in PgLocks [\#107](https://github.com/bensheldon/good_job/pull/107) ([gadimbaylisahil](https://github.com/gadimbaylisahil))
|
|
98
|
+
- Re-organize Readme: frontload configuration, add Table of Contents [\#106](https://github.com/bensheldon/good_job/pull/106) ([bensheldon](https://github.com/bensheldon))
|
|
99
|
+
|
|
100
|
+
## [v1.2.3](https://github.com/bensheldon/good_job/tree/v1.2.3) (2020-08-27)
|
|
101
|
+
|
|
102
|
+
[Full Changelog](https://github.com/bensheldon/good_job/compare/v1.2.2...v1.2.3)
|
|
103
|
+
|
|
104
|
+
**Closed issues:**
|
|
105
|
+
|
|
106
|
+
- requiring more dependencies in then needed [\#103](https://github.com/bensheldon/good_job/issues/103)
|
|
107
|
+
|
|
108
|
+
**Merged pull requests:**
|
|
109
|
+
|
|
110
|
+
- stop depending on all rails libs [\#104](https://github.com/bensheldon/good_job/pull/104) ([thilo](https://github.com/thilo))
|
|
111
|
+
|
|
112
|
+
## [v1.2.2](https://github.com/bensheldon/good_job/tree/v1.2.2) (2020-08-27)
|
|
4
113
|
|
|
5
114
|
[Full Changelog](https://github.com/bensheldon/good_job/compare/v1.2.1...v1.2.2)
|
|
6
115
|
|
|
@@ -77,6 +186,7 @@
|
|
|
77
186
|
|
|
78
187
|
**Merged pull requests:**
|
|
79
188
|
|
|
189
|
+
- Capture errors via instrumentation from retry\_on and discard\_on [\#79](https://github.com/bensheldon/good_job/pull/79) ([bensheldon](https://github.com/bensheldon))
|
|
80
190
|
- Document GoodJob::Scheduler with Yard [\#78](https://github.com/bensheldon/good_job/pull/78) ([bensheldon](https://github.com/bensheldon))
|
|
81
191
|
|
|
82
192
|
## [v1.1.2](https://github.com/bensheldon/good_job/tree/v1.1.2) (2020-08-13)
|
|
@@ -101,7 +211,6 @@
|
|
|
101
211
|
|
|
102
212
|
**Merged pull requests:**
|
|
103
213
|
|
|
104
|
-
- Capture errors via instrumentation from retry\_on and discard\_on [\#79](https://github.com/bensheldon/good_job/pull/79) ([bensheldon](https://github.com/bensheldon))
|
|
105
214
|
- Allow instantiation of multiple schedulers via --queues [\#76](https://github.com/bensheldon/good_job/pull/76) ([bensheldon](https://github.com/bensheldon))
|
|
106
215
|
- Extract options parsing to Configuration object [\#74](https://github.com/bensheldon/good_job/pull/74) ([bensheldon](https://github.com/bensheldon))
|
|
107
216
|
|
data/README.md
CHANGED
|
@@ -1,213 +1,333 @@
|
|
|
1
1
|
# GoodJob
|
|
2
2
|
|
|
3
|
+
[](https://rubygems.org/gems/good_job)
|
|
4
|
+
[](https://github.com/bensheldon/good_job/actions)
|
|
5
|
+
|
|
3
6
|
GoodJob is a multithreaded, Postgres-based, ActiveJob backend for Ruby on Rails.
|
|
4
7
|
|
|
5
8
|
**Inspired by [Delayed::Job](https://github.com/collectiveidea/delayed_job) and [Que](https://github.com/que-rb/que), GoodJob is designed for maximum compatibility with Ruby on Rails, ActiveJob, and Postgres to be simple and performant for most workloads.**
|
|
6
9
|
|
|
7
|
-
- **Designed for ActiveJob.** Complete support for [async, queues, delays, priorities, timeouts, and retries](https://edgeguides.rubyonrails.org/active_job_basics.html) with near-zero configuration.
|
|
8
|
-
- **Built for Rails.** Fully adopts Ruby on Rails [threading and code execution guidelines](https://guides.rubyonrails.org/threading_and_code_execution.html) with [Concurrent::Ruby](https://github.com/ruby-concurrency/concurrent-ruby).
|
|
10
|
+
- **Designed for ActiveJob.** Complete support for [async, queues, delays, priorities, timeouts, and retries](https://edgeguides.rubyonrails.org/active_job_basics.html) with near-zero configuration.
|
|
11
|
+
- **Built for Rails.** Fully adopts Ruby on Rails [threading and code execution guidelines](https://guides.rubyonrails.org/threading_and_code_execution.html) with [Concurrent::Ruby](https://github.com/ruby-concurrency/concurrent-ruby).
|
|
9
12
|
- **Backed by Postgres.** Relies upon Postgres integrity, session-level Advisory Locks to provide run-once safety and stay within the limits of `schema.rb`, and LISTEN/NOTIFY to reduce queuing latency.
|
|
10
13
|
- **For most workloads.** Targets full-stack teams, economy-minded solo developers, and applications that enqueue less than 1-million jobs/day.
|
|
11
14
|
|
|
12
15
|
For more of the story of GoodJob, read the [introductory blog post](https://island94.org/2020/07/introducing-goodjob-1-0).
|
|
13
16
|
|
|
14
|
-
<details>
|
|
17
|
+
<details markdown="1">
|
|
15
18
|
<summary><strong>π Comparison of GoodJob with other job queue backends (click to expand)</strong></summary>
|
|
16
19
|
|
|
17
20
|
| | Queues, priority, retries | Database | Concurrency | Reliability/Integrity | Latency |
|
|
18
21
|
|-----------------|---------------------------|---------------------------------------|-------------------|------------------------|--------------------------|
|
|
19
22
|
| **GoodJob** | β
Yes | β
Postgres | β
Multithreaded | β
ACID, Advisory Locks | β
Postgres LISTEN/NOTIFY |
|
|
20
|
-
| **Que** | β
Yes |
|
|
21
|
-
| **Delayed Job** | β
Yes | β
Postgres |
|
|
22
|
-
| **Sidekiq** | β
Yes |
|
|
23
|
-
| **Sidekiq Pro** | β
Yes |
|
|
23
|
+
| **Que** | β
Yes | πΆοΈ Postgres, requires `structure.sql` | β
Multithreaded | β
ACID, Advisory Locks | β
Postgres LISTEN/NOTIFY |
|
|
24
|
+
| **Delayed Job** | β
Yes | β
Postgres | π΄ Single-threaded | β
ACID, record-based | πΆ Polling |
|
|
25
|
+
| **Sidekiq** | β
Yes | π΄ Redis | β
Multithreaded | π΄ Crashes lose jobs | β
Redis BRPOP |
|
|
26
|
+
| **Sidekiq Pro** | β
Yes | π΄ Redis | β
Multithreaded | β
Redis RPOPLPUSH | β
Redis RPOPLPUSH |
|
|
24
27
|
|
|
25
28
|
</details>
|
|
26
29
|
|
|
27
|
-
##
|
|
30
|
+
## Table of contents
|
|
31
|
+
|
|
32
|
+
- [Set up](#set-up)
|
|
33
|
+
- [Configuration](#configuration)
|
|
34
|
+
- [Command-line options](#command-line-options)
|
|
35
|
+
- [`good_job start`](#good_job-start)
|
|
36
|
+
- [`good_job cleanup_preserved_jobs`](#good_job-cleanup_preserved_jobs)
|
|
37
|
+
- [Adapter options](#adapter-options)
|
|
38
|
+
- [Global options](#global-options)
|
|
39
|
+
- [Dashboard](#dashboard)
|
|
40
|
+
- [Go deeper](#go-deeper)
|
|
41
|
+
- [Exceptions, retries, and reliability](#exceptions-retries-and-reliability)
|
|
42
|
+
- [Exceptions](#exceptions)
|
|
43
|
+
- [Retries](#retries)
|
|
44
|
+
- [ActionMailer retries](#actionmailer-retries)
|
|
45
|
+
- [Timeouts](#timeouts)
|
|
46
|
+
- [Optimize queues, threads, and processes](#optimize-queues-threads-and-processes)
|
|
47
|
+
- [Database connections](#database-connections)
|
|
48
|
+
- [Execute jobs async / in-process](#execute-jobs-async--in-process)
|
|
49
|
+
- [Migrate to GoodJob from a different ActiveJob backend](#migrate-to-goodjob-from-a-different-activejob-backend)
|
|
50
|
+
- [Monitor and preserve worked jobs](#monitor-and-preserve-worked-jobs)
|
|
51
|
+
- [Contribute](#contribute)
|
|
52
|
+
- [Gem development](#gem-development)
|
|
53
|
+
- [Release](#release)
|
|
54
|
+
- [License](#license)
|
|
55
|
+
|
|
56
|
+
## Set up
|
|
57
|
+
|
|
58
|
+
1. Add `good_job` to your application's Gemfile:
|
|
28
59
|
|
|
29
|
-
|
|
60
|
+
```ruby
|
|
61
|
+
gem 'good_job'
|
|
62
|
+
```
|
|
30
63
|
|
|
31
|
-
|
|
32
|
-
gem 'good_job'
|
|
33
|
-
```
|
|
64
|
+
1. Install the gem:
|
|
34
65
|
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
```
|
|
66
|
+
```bash
|
|
67
|
+
$ bundle install
|
|
68
|
+
```
|
|
39
69
|
|
|
40
|
-
|
|
70
|
+
1. Run the GoodJob install generator. This will generate a database migration to create a table for GoodJob's job records:
|
|
41
71
|
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
```bash
|
|
72
|
+
```bash
|
|
45
73
|
$ bin/rails g good_job:install
|
|
46
74
|
```
|
|
47
75
|
|
|
48
76
|
Run the migration:
|
|
49
|
-
|
|
77
|
+
|
|
50
78
|
```bash
|
|
51
79
|
$ bin/rails db:migrate
|
|
52
80
|
```
|
|
53
|
-
|
|
81
|
+
|
|
54
82
|
1. Configure the ActiveJob adapter:
|
|
55
|
-
|
|
56
|
-
|
|
83
|
+
|
|
84
|
+
```ruby
|
|
57
85
|
# config/application.rb
|
|
58
86
|
config.active_job.queue_adapter = :good_job
|
|
59
87
|
```
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
|
|
88
|
+
|
|
89
|
+
1. Inside of your application, queue your job π:
|
|
90
|
+
|
|
63
91
|
```ruby
|
|
64
|
-
|
|
65
|
-
config.active_job.queue_adapter = GoodJob::Adapter.new(execution_mode: :inline)
|
|
66
|
-
|
|
67
|
-
# config/environments/test.rb
|
|
68
|
-
config.active_job.queue_adapter = GoodJob::Adapter.new(execution_mode: :inline)
|
|
69
|
-
|
|
70
|
-
# config/environments/production.rb
|
|
71
|
-
config.active_job.queue_adapter = GoodJob::Adapter.new(execution_mode: :external)
|
|
92
|
+
YourJob.perform_later
|
|
72
93
|
```
|
|
73
94
|
|
|
74
|
-
|
|
75
|
-
|
|
95
|
+
GoodJob supports all ActiveJob features:
|
|
96
|
+
|
|
76
97
|
```ruby
|
|
77
98
|
YourJob.set(queue: :some_queue, wait: 5.minutes, priority: 10).perform_later
|
|
78
99
|
```
|
|
79
100
|
|
|
80
|
-
1. In
|
|
81
|
-
|
|
82
|
-
```bash
|
|
83
|
-
$ bundle exec good_job
|
|
84
|
-
```
|
|
85
|
-
|
|
86
|
-
Configuration options available with `help`:
|
|
101
|
+
1. In development, GoodJob executes jobs immediately. In production, GoodJob provides different options:
|
|
87
102
|
|
|
88
|
-
|
|
89
|
-
$ bundle exec good_job help start
|
|
90
|
-
|
|
91
|
-
Usage:
|
|
92
|
-
good_job start
|
|
93
|
-
|
|
94
|
-
Options:
|
|
95
|
-
[--max-threads=N] # Maximum number of threads to use for working jobs (default: ActiveRecord::Base.connection_pool.size)
|
|
96
|
-
[--queues=queue1,queue2(;queue3,queue4:5;-queue1,queue2)] # Queues to work from. Separate multiple queues with commas; exclude queues with a leading minus; separate isolated execution pools with semicolons and threads with colons (default: *)
|
|
97
|
-
[--poll-interval=N] # Interval between polls for available jobs in seconds (default: 1)
|
|
98
|
-
|
|
99
|
-
Start job worker
|
|
100
|
-
```
|
|
103
|
+
- By default, GoodJob separates job enqueuing from job execution so that jobs can be scaled independently of the web server. Use the GoodJob command-line tool to execute jobs:
|
|
101
104
|
|
|
102
|
-
|
|
105
|
+
```bash
|
|
106
|
+
$ bundle exec good_job start
|
|
107
|
+
```
|
|
103
108
|
|
|
104
|
-
|
|
109
|
+
Ideally the command-line tool should be run on a separate machine or container from the web process. For example, on Heroku:
|
|
105
110
|
|
|
106
|
-
|
|
107
|
-
|
|
108
|
-
|
|
109
|
-
$ bundle exec good_job --queues=transactional_messages:2;batch_processing:1;-transactional_messages,batch_processing:2;* --max-threads=5
|
|
111
|
+
```Procfile
|
|
112
|
+
web: rails server
|
|
113
|
+
worker: bundle exec good_job start
|
|
110
114
|
```
|
|
111
|
-
|
|
112
|
-
|
|
113
|
-
|
|
114
|
-
|
|
115
|
-
|
|
116
|
-
- `-transactional_messages,batch_processing`: execute jobs enqueued on _any_ queue _excluding_ `transactional_messages` or `batch_processing` with up to 2 threads.
|
|
117
|
-
- `*`: execute jobs on any queue on up to 5 threads, as configured by `--max-threads=5`
|
|
118
|
-
|
|
119
|
-
For moderate workloads, multiple isolated thread execution pools offers a good balance between congestion management and economy.
|
|
120
|
-
|
|
121
|
-
Configuration can be injected by environment variables too:
|
|
122
|
-
|
|
123
|
-
```bash
|
|
124
|
-
$ GOOD_JOB_QUEUES="transactional_messages:2;batch_processing:1;-transactional_messages,batch_processing:2;*" GOOD_JOB_MAX_THREADS=5 bundle exec good_job
|
|
115
|
+
|
|
116
|
+
The command-line tool supports a variety of options, see the reference below for command-line configuration.
|
|
117
|
+
|
|
118
|
+
- GoodJob can also be configured to execute jobs within the web server process to save on resources. This is useful for low-workloads when economy is paramount.
|
|
119
|
+
|
|
125
120
|
```
|
|
126
|
-
|
|
127
|
-
- Multiple processes; for example, on Heroku:
|
|
128
|
-
|
|
129
|
-
```procfile
|
|
130
|
-
# Procfile
|
|
131
|
-
|
|
132
|
-
# Separate dyno types
|
|
133
|
-
worker: bundle exec good_job --max-threads=5
|
|
134
|
-
transactional_worker: bundle exec good_job --queues=transactional_messages --max-threads=2
|
|
135
|
-
batch_worker: bundle exec good_job --queues=batch_processing --max-threads=1
|
|
136
|
-
|
|
137
|
-
# Combined multi-process dyno
|
|
138
|
-
combined_worker: bundle exec good_job --max-threads=5 & bundle exec good_job --queues=transactional_messages --max-threads=2 & bundle exec good_job --queues=batch_processing --max-threads=1 & wait -n
|
|
121
|
+
$ GOOD_JOB_EXECUTION_MODE=async rails server
|
|
139
122
|
```
|
|
140
|
-
|
|
141
|
-
Running multiple processes can optimize for CPU performance at the expense of greater memory and system resource usage.
|
|
142
123
|
|
|
143
|
-
|
|
124
|
+
Additional configuration is likely necessary, see the reference below for async configuration.
|
|
125
|
+
|
|
126
|
+
## Configuration
|
|
127
|
+
|
|
128
|
+
### Command-line options
|
|
129
|
+
|
|
130
|
+
There several top-level commands available through the `good_job` command-line tool.
|
|
131
|
+
|
|
132
|
+
Configuration options are available with `help`.
|
|
133
|
+
|
|
134
|
+
#### `good_job start`
|
|
135
|
+
|
|
136
|
+
`good_job start` executes queued jobs.
|
|
137
|
+
|
|
138
|
+
```bash
|
|
139
|
+
$ bundle exec good_job help start
|
|
140
|
+
|
|
141
|
+
Usage:
|
|
142
|
+
good_job start
|
|
143
|
+
|
|
144
|
+
Options:
|
|
145
|
+
[--max-threads=COUNT] # Maximum number of threads to use for working jobs. (env var: GOOD_JOB_MAX_THREADS, default: 5)
|
|
146
|
+
[--queues=QUEUE_LIST] # Queues to work from. (env var: GOOD_JOB_QUEUES, default: *)
|
|
147
|
+
[--poll-interval=SECONDS] # Interval between polls for available jobs in seconds (env var: GOOD_JOB_POLL_INTERVAL, default: 1)
|
|
148
|
+
|
|
149
|
+
Executes queued jobs.
|
|
150
|
+
|
|
151
|
+
All options can be configured with environment variables.
|
|
152
|
+
See option descriptions for the matching environment variable name.
|
|
153
|
+
|
|
154
|
+
== Configuring queues
|
|
155
|
+
Separate multiple queues with commas; exclude queues with a leading minus;
|
|
156
|
+
separate isolated execution pools with semicolons and threads with colons.
|
|
157
|
+
```
|
|
158
|
+
|
|
159
|
+
#### `good_job cleanup_preserved_jobs`
|
|
160
|
+
|
|
161
|
+
`good_job cleanup_preserved_jobs` deletes preserved job records. See [`GoodJob.preserve_job_records` for when this command is useful.
|
|
162
|
+
|
|
163
|
+
```bash
|
|
164
|
+
$ bundle exec good_job help cleanup_preserved_jobs
|
|
165
|
+
|
|
166
|
+
Usage:
|
|
167
|
+
good_job cleanup_preserved_jobs
|
|
168
|
+
|
|
169
|
+
Options:
|
|
170
|
+
[--before-seconds-ago=SECONDS] # Delete records finished more than this many seconds ago (env var: GOOD_JOB_CLEANUP_PRESERVED_JOBS_BEFORE_SECONDS_AGO, default: 86400)
|
|
171
|
+
|
|
172
|
+
Deletes preserved job records.
|
|
173
|
+
|
|
174
|
+
By default, GoodJob deletes job records when the job is performed and this
|
|
175
|
+
command is not necessary.
|
|
176
|
+
|
|
177
|
+
However, when `GoodJob.preserve_job_records = true`, the jobs will be
|
|
178
|
+
preserved in the database. This is useful when wanting to analyze or
|
|
179
|
+
inspect job performance.
|
|
180
|
+
|
|
181
|
+
If you are preserving job records this way, use this command regularly
|
|
182
|
+
to delete old records and preserve space in your database.
|
|
183
|
+
```
|
|
184
|
+
|
|
185
|
+
### Adapter options
|
|
186
|
+
|
|
187
|
+
To use GoodJob, you can set `config.active_job.queue_adapter` to a `:good_job` or to an instance of `GoodJob::Adapter`, which you can configure with several options:
|
|
144
188
|
|
|
145
|
-
|
|
189
|
+
- `execution_mode` (symbol) specifies how and where jobs should be executed. You can also set this with the environment variable `GOOD_JOB_EXECUTION_MODE`. It can be any one of:
|
|
190
|
+
- `:inline` executes jobs immediately in whatever process queued them (usually the web server process). This should only be used in test and development environments.
|
|
191
|
+
- `:external` causes the adapter to enqueue jobs, but not execute them. When using this option (the default for production environments), youβll need to use the command-line tool to actually execute your jobs.
|
|
192
|
+
- `:async` causes the adapter to execute you jobs in separate threads in whatever process queued them (usually the web process). This is akin to running the command-line toolβs code inside your web server. It can be more economical for small workloads (you donβt need a separate machine or environment for running your jobs), but if your web server is under heavy load or your jobs require a lot of resources, you should choose `:external` instead.
|
|
193
|
+
- `max_threads` (integer) sets the maximum number of threads to use when `execution_mode` is set to `:async`. You can also set this with the environment variable `GOOD_JOB_MAX_THREADS`.
|
|
194
|
+
- `queues` (string) determines which queues to execute jobs from when `execution_mode` is set to `:async`. See the description of `good_job start` for more details on the format of this string. You can also set this with the environment variable `GOOD_JOB_QUEUES`.
|
|
195
|
+
- `poll_interval` (integer) sets the number of seconds between polls for jobs when `execution_mode` is set to `:async`. You can also set this with the environment variable `GOOD_JOB_POLL_INTERVAL`.
|
|
146
196
|
|
|
147
|
-
|
|
197
|
+
Using the symbol instead of explicitly configuring the options above (i.e. setting `config.active_job.queue_adapter = :good_job`) is equivalent to:
|
|
148
198
|
|
|
149
|
-
|
|
199
|
+
```ruby
|
|
200
|
+
# config/environments/development.rb
|
|
201
|
+
config.active_job.queue_adapter = GoodJob::Adapter.new(execution_mode: :inline)
|
|
202
|
+
|
|
203
|
+
# config/environments/test.rb
|
|
204
|
+
config.active_job.queue_adapter = GoodJob::Adapter.new(execution_mode: :inline)
|
|
205
|
+
|
|
206
|
+
# config/environments/production.rb
|
|
207
|
+
config.active_job.queue_adapter = GoodJob::Adapter.new(execution_mode: :external)
|
|
208
|
+
```
|
|
209
|
+
|
|
210
|
+
### Global options
|
|
150
211
|
|
|
151
|
-
|
|
212
|
+
Good Jobβs general behavior can also be configured via several attributes directly on the `GoodJob` module:
|
|
213
|
+
|
|
214
|
+
- **`GoodJob.logger`** ([Rails Logger](https://api.rubyonrails.org/classes/ActiveSupport/Logger.html)) lets you set a custom logger for GoodJob. It should be an instance of a Rails `Logger`.
|
|
215
|
+
- **`GoodJob.preserve_job_records`** (boolean) keeps job records in your database even after jobs are completed. (Default: `false`)
|
|
216
|
+
- **`GoodJob.retry_on_unhandled_error`** (boolean) causes jobs to be re-queued and retried if they raise an instance of `StandardError`. Instances of `Exception`, like SIGINT, will *always* be retried, regardless of this attributeβs value. (Default: `true`)
|
|
217
|
+
- **`GoodJob.on_thread_error`** (proc, lambda, or callable) will be called when an Exception. It can be useful for logging errors to bug tracking services, like Sentry or Airbrake.
|
|
218
|
+
|
|
219
|
+
Youβll generally want to configure these in `config/initializers/good_job.rb`, like so:
|
|
220
|
+
|
|
221
|
+
```ruby
|
|
222
|
+
# config/initializers/good_job.rb
|
|
223
|
+
GoodJob.preserve_job_records = true
|
|
224
|
+
GoodJob.retry_on_unhandled_error = false
|
|
225
|
+
GoodJob.on_thread_error = -> (exception) { Raven.capture_exception(exception) }
|
|
226
|
+
```
|
|
227
|
+
|
|
228
|
+
### Dashboard
|
|
229
|
+
|
|
230
|
+
_π§ GoodJob's dashboard is a work in progress. Please contribute ideas and code on [Github](https://github.com/bensheldon/good_job/issues)._
|
|
231
|
+
|
|
232
|
+
GoodJob includes a Dashboard as a mountable `Rails::Engine`.
|
|
233
|
+
|
|
234
|
+
1. Explicitly require the Engine code at the top of your `config/application.rb` file, immediately after Rails is required. This is necessary because the mountable engine is an optional feature of GoodJob.
|
|
235
|
+
|
|
236
|
+
```ruby
|
|
237
|
+
# config/application.rb
|
|
238
|
+
require_relative 'boot'
|
|
239
|
+
|
|
240
|
+
require 'rails/all'
|
|
241
|
+
require 'good_job/engine' # <= Add this line
|
|
242
|
+
# ...
|
|
243
|
+
```
|
|
244
|
+
|
|
245
|
+
1. Mount the engine in your `config/routes.rb` file. The following will mount it at `http://example.com/good_job`.
|
|
246
|
+
|
|
247
|
+
```ruby
|
|
248
|
+
# config/routes.rb
|
|
249
|
+
# ...
|
|
250
|
+
mount GoodJob::Engine => 'good_job'
|
|
251
|
+
```
|
|
252
|
+
|
|
253
|
+
Because jobs can potentially contain sensitive information, you should authorize access. For example, using Devise's `authenticate` helper, that might look like:
|
|
254
|
+
|
|
255
|
+
```ruby
|
|
256
|
+
# config/routes.rb
|
|
257
|
+
# ...
|
|
258
|
+
authenticate :user, ->(user) { user.admin? } do
|
|
259
|
+
mount GoodJob::Engine => 'good_job'
|
|
260
|
+
end
|
|
261
|
+
```
|
|
262
|
+
|
|
263
|
+
Another option is using basic auth like this:
|
|
152
264
|
|
|
153
|
-
- `Exception`-type errors, such as a SIGINT, will always cause a job to be re-performed.
|
|
154
|
-
- `StandardError`-type errors, by default, will cause a job to be re-performed, though this is configurable:
|
|
155
|
-
|
|
156
265
|
```ruby
|
|
157
266
|
# config/initializers/good_job.rb
|
|
158
|
-
GoodJob.
|
|
267
|
+
GoodJob::Engine.middleware.use(Rack::Auth::Basic) do |username, password|
|
|
268
|
+
ActiveSupport::SecurityUtils.secure_compare(Rails.application.credentials.good_job_username, username) &&
|
|
269
|
+
ActiveSupport::SecurityUtils.secure_compare(Rails.application.credentials.good_job_password, password)
|
|
270
|
+
end
|
|
159
271
|
```
|
|
160
272
|
|
|
161
|
-
|
|
273
|
+
## Go deeper
|
|
274
|
+
|
|
275
|
+
### Exceptions, retries, and reliability
|
|
276
|
+
|
|
277
|
+
GoodJob guarantees that a completely-performed job will run once and only once. GoodJob fully supports ActiveJob's built-in functionality for error handling, retries and timeouts.
|
|
278
|
+
|
|
279
|
+
#### Exceptions
|
|
280
|
+
|
|
281
|
+
ActiveJob provides [tools for rescuing and retrying exceptions](https://guides.rubyonrails.org/active_job_basics.html#exceptions), including `retry_on`, `discard_on`, `rescue_from` that will rescue exceptions before they get to GoodJob.
|
|
282
|
+
|
|
283
|
+
If errors do reach GoodJob, you can assign a callable to `GoodJob.on_thread_error` to be notified. For example, to log errors to an exception monitoring service like Sentry (or Bugsnag, Airbrake, Honeybadger, etc.):
|
|
162
284
|
|
|
163
285
|
```ruby
|
|
164
286
|
# config/initializers/good_job.rb
|
|
165
|
-
|
|
166
|
-
# With Sentry (or Bugsnag, Airbrake, Honeybadger, etc.)
|
|
167
287
|
GoodJob.on_thread_error = -> (exception) { Raven.capture_exception(exception) }
|
|
168
288
|
```
|
|
169
289
|
|
|
170
|
-
|
|
290
|
+
#### Retries
|
|
171
291
|
|
|
172
|
-
|
|
292
|
+
By default, GoodJob will automatically and immediately retry a job when an exception is raised to GoodJob.
|
|
293
|
+
|
|
294
|
+
However, ActiveJob can be configured to retry an infinite number of times, with an exponential backoff. Using ActiveJob's `retry_on` prevents exceptions from reaching GoodJob:
|
|
173
295
|
|
|
174
296
|
```ruby
|
|
175
|
-
class ApplicationJob < ActiveJob::Base
|
|
297
|
+
class ApplicationJob < ActiveJob::Base
|
|
176
298
|
retry_on StandardError, wait: :exponentially_longer, attempts: Float::INFINITY
|
|
177
299
|
# ...
|
|
178
300
|
end
|
|
179
301
|
```
|
|
180
302
|
|
|
181
|
-
When
|
|
303
|
+
When using `retry_on` with _a limited number of retries_, the final exception will not be rescued and will raise to GoodJob. GoodJob can be configured to discard un-handled exceptions instead of retrying them:
|
|
182
304
|
|
|
183
305
|
```ruby
|
|
184
|
-
|
|
306
|
+
# config/initializers/good_job.rb
|
|
307
|
+
GoodJob.retry_on_unhandled_error = false
|
|
308
|
+
```
|
|
309
|
+
|
|
310
|
+
Alternatively, pass a block to `retry_on` to handle the final exception instead of raising it to GoodJob:
|
|
311
|
+
|
|
312
|
+
```ruby
|
|
313
|
+
class ApplicationJob < ActiveJob::Base
|
|
185
314
|
retry_on StandardError, attempts: 5 do |_job, _exception|
|
|
186
|
-
# Log error, etc.
|
|
187
|
-
# You must implement this block, otherwise,
|
|
188
|
-
# Active Job will re-raise the error.
|
|
189
|
-
# Do not re-raise the error, otherwise
|
|
190
|
-
# GoodJob will immediately re-perform the job.
|
|
315
|
+
# Log error, do nothing, etc.
|
|
191
316
|
end
|
|
192
317
|
# ...
|
|
193
318
|
end
|
|
194
319
|
```
|
|
195
320
|
|
|
196
|
-
|
|
321
|
+
When using `retry_on` with an infinite number of retries, exceptions will never be raised to GoodJob, which means `GoodJob.on_thread_error` will never be called. To report log or report exceptions to an exception monitoring service (e.g. Sentry, Bugsnag, Airbrake, Honeybadger, etc), create an explicit exception wrapper. For example:
|
|
197
322
|
|
|
198
323
|
```ruby
|
|
199
|
-
|
|
200
|
-
|
|
201
|
-
# Do NOT re-perform a job if a StandardError bubbles up to the GoodJob backend
|
|
202
|
-
GoodJob.reperform_jobs_on_standard_error = false
|
|
203
|
-
```
|
|
324
|
+
class ApplicationJob < ActiveJob::Base
|
|
325
|
+
retry_on StandardError, wait: :exponentially_longer, attempts: Float::INFINITY
|
|
204
326
|
|
|
205
|
-
|
|
327
|
+
retry_on SpecialError, attempts: 5 do |_job, exception|
|
|
328
|
+
Raven.capture_exception(exception)
|
|
329
|
+
end
|
|
206
330
|
|
|
207
|
-
```ruby
|
|
208
|
-
class ApplicationJob < ActiveJob::Base
|
|
209
|
-
retry_on StandardError, wait: :exponentially_longer, attempts: Float::INFINITY
|
|
210
|
-
|
|
211
331
|
around_perform do |_job, block|
|
|
212
332
|
block.call
|
|
213
333
|
rescue StandardError => e
|
|
@@ -218,19 +338,18 @@ class ApplicationJob < ActiveJob::Base
|
|
|
218
338
|
end
|
|
219
339
|
```
|
|
220
340
|
|
|
221
|
-
|
|
222
|
-
ActiveJob's `discard_on` functionality is supported too.
|
|
223
|
-
|
|
224
341
|
#### ActionMailer retries
|
|
225
342
|
|
|
226
|
-
|
|
343
|
+
Any configuration in `ApplicationJob` will have to be duplicated on `ActionMailer::MailDeliveryJob` (`ActionMailer::DeliveryJob` in Rails 5.2 or earlier) because ActionMailer uses a custom class, `ActionMailer::MailDeliveryJob`, which inherits from `ActiveJob::Base`, rather than your applications `ApplicationJob`.
|
|
344
|
+
|
|
345
|
+
You can use an initializer to configure `ActionMailer::MailDeliveryJob`, for example:
|
|
227
346
|
|
|
228
347
|
```ruby
|
|
229
348
|
# config/initializers/good_job.rb
|
|
230
|
-
ActionMailer::
|
|
349
|
+
ActionMailer::MailDeliveryJob.retry_on StandardError, wait: :exponentially_longer, attempts: Float::INFINITY
|
|
231
350
|
|
|
232
351
|
# With Sentry (or Bugsnag, Airbrake, Honeybadger, etc.)
|
|
233
|
-
ActionMailer::
|
|
352
|
+
ActionMailer::MailDeliveryJob.around_perform do |_job, block|
|
|
234
353
|
block.call
|
|
235
354
|
rescue StandardError => e
|
|
236
355
|
Raven.capture_exception(e)
|
|
@@ -238,14 +357,17 @@ rescue StandardError => e
|
|
|
238
357
|
end
|
|
239
358
|
```
|
|
240
359
|
|
|
241
|
-
|
|
360
|
+
Note, that `ActionMailer::MailDeliveryJob` is a default since Rails 6.0. Be sure that your app is using that class, as it
|
|
361
|
+
might also be configured to use (deprecated now) `ActionMailer::DeliveryJob`.
|
|
362
|
+
|
|
363
|
+
### Timeouts
|
|
242
364
|
|
|
243
365
|
Job timeouts can be configured with an `around_perform`:
|
|
244
366
|
|
|
245
367
|
```ruby
|
|
246
|
-
class ApplicationJob < ActiveJob::Base
|
|
368
|
+
class ApplicationJob < ActiveJob::Base
|
|
247
369
|
JobTimeoutError = Class.new(StandardError)
|
|
248
|
-
|
|
370
|
+
|
|
249
371
|
around_perform do |_job, block|
|
|
250
372
|
# Timeout jobs after 10 minutes
|
|
251
373
|
Timeout.timeout(10.minutes, JobTimeoutError) do
|
|
@@ -255,20 +377,61 @@ class ApplicationJob < ActiveJob::Base
|
|
|
255
377
|
end
|
|
256
378
|
```
|
|
257
379
|
|
|
258
|
-
###
|
|
259
|
-
|
|
260
|
-
GoodJob
|
|
380
|
+
### Optimize queues, threads, and processes
|
|
381
|
+
|
|
382
|
+
By default, GoodJob creates a single thread execution pool that will execute jobs from any queue. Depending on your application's workload, job types, and service level objectives, you may wish to optimize execution resources. For example, providing dedicated execution resources for transactional emails so they are not delayed by long-running batch jobs. Some options:
|
|
383
|
+
|
|
384
|
+
- Multiple execution pools within a single process:
|
|
385
|
+
|
|
386
|
+
```bash
|
|
387
|
+
$ bundle exec good_job --queues="transactional_messages:2;batch_processing:1;-transactional_messages,batch_processing:2;*" --max-threads=5
|
|
388
|
+
```
|
|
389
|
+
|
|
390
|
+
This configuration will result in a single process with 4 isolated thread execution pools. Isolated execution pools are separated with a semicolon (`;`) and queue names and thread counts with a colon (`:`)
|
|
261
391
|
|
|
262
|
-
-
|
|
263
|
-
-
|
|
264
|
-
|
|
265
|
-
|
|
266
|
-
3. `$ RAILS_MAX_THREADS=4 bundle exec good_job`
|
|
267
|
-
4. Implicitly via Rails's database connection pool size (`ActiveRecord::Base.connection_pool.size`)
|
|
392
|
+
- `transactional_messages:2`: execute jobs enqueued on `transactional_messages` with up to 2 threads.
|
|
393
|
+
- `batch_processing:1` execute jobs enqueued on `batch_processing` with a single thread.
|
|
394
|
+
- `-transactional_messages,batch_processing`: execute jobs enqueued on _any_ queue _excluding_ `transactional_messages` or `batch_processing` with up to 2 threads.
|
|
395
|
+
- `*`: execute jobs on any queue on up to 5 threads, as configured by `--max-threads=5`
|
|
268
396
|
|
|
269
|
-
|
|
397
|
+
For moderate workloads, multiple isolated thread execution pools offers a good balance between congestion management and economy.
|
|
270
398
|
|
|
271
|
-
|
|
399
|
+
Configuration can be injected by environment variables too:
|
|
400
|
+
|
|
401
|
+
```bash
|
|
402
|
+
$ GOOD_JOB_QUEUES="transactional_messages:2;batch_processing:1;-transactional_messages,batch_processing:2;*" GOOD_JOB_MAX_THREADS=5 bundle exec good_job
|
|
403
|
+
```
|
|
404
|
+
|
|
405
|
+
- Multiple processes; for example, on Heroku:
|
|
406
|
+
|
|
407
|
+
```procfile
|
|
408
|
+
# Procfile
|
|
409
|
+
|
|
410
|
+
# Separate dyno types
|
|
411
|
+
worker: bundle exec good_job --max-threads=5
|
|
412
|
+
transactional_worker: bundle exec good_job --queues="transactional_messages" --max-threads=2
|
|
413
|
+
batch_worker: bundle exec good_job --queues="batch_processing" --max-threads=1
|
|
414
|
+
|
|
415
|
+
# Combined multi-process dyno
|
|
416
|
+
combined_worker: bundle exec good_job --max-threads=5 & bundle exec good_job --queues="transactional_messages" --max-threads=2 & bundle exec good_job --queues="batch_processing" --max-threads=1 & wait -n
|
|
417
|
+
```
|
|
418
|
+
|
|
419
|
+
Running multiple processes can optimize for CPU performance at the expense of greater memory and system resource usage.
|
|
420
|
+
|
|
421
|
+
Keep in mind, queue operations and management is an advanced discipline. This stuff is complex, especially for heavy workloads and unique processing requirements. Good job π
|
|
422
|
+
|
|
423
|
+
### Database connections
|
|
424
|
+
|
|
425
|
+
Each GoodJob execution thread requires its own database connection that is automatically checked out from Railsβs connection pool. _Allowing GoodJob to create more threads than available database connections can lead to timeouts and is not recommended._ For example:
|
|
426
|
+
|
|
427
|
+
```yaml
|
|
428
|
+
# config/database.yml
|
|
429
|
+
pool: <%= [ENV.fetch("RAILS_MAX_THREADS", 5).to_i, ENV.fetch("GOOD_JOB_MAX_THREADS", 4).to_i].max %>
|
|
430
|
+
```
|
|
431
|
+
|
|
432
|
+
### Execute jobs async / in-process
|
|
433
|
+
|
|
434
|
+
GoodJob can execute jobs "async" in the same process as the webserver (e.g. `bin/rail s`). GoodJob's async execution mode offers benefits of economy by not requiring a separate job worker process, but with the tradeoff of increased complexity. Async mode can be configured in two ways:
|
|
272
435
|
|
|
273
436
|
- Directly configure the ActiveJob adapter:
|
|
274
437
|
|
|
@@ -276,12 +439,13 @@ GoodJob is able to run "async" in the same process as the webserver (e.g. `bin/r
|
|
|
276
439
|
# config/environments/production.rb
|
|
277
440
|
config.active_job.queue_adapter = GoodJob::Adapter.new(execution_mode: :async, max_threads: 4, poll_interval: 30)
|
|
278
441
|
```
|
|
442
|
+
|
|
279
443
|
- Or, when using `...queue_adapter = :good_job`, via environment variables:
|
|
280
444
|
|
|
281
445
|
```bash
|
|
282
446
|
$ GOOD_JOB_EXECUTION_MODE=async GOOD_JOB_MAX_THREADS=4 GOOD_JOB_POLL_INTERVAL=30 bin/rails server
|
|
283
447
|
```
|
|
284
|
-
|
|
448
|
+
|
|
285
449
|
Depending on your application configuration, you may need to take additional steps:
|
|
286
450
|
|
|
287
451
|
- Ensure that you have enough database connections for both web and job execution threads:
|
|
@@ -295,28 +459,28 @@ Depending on your application configuration, you may need to take additional ste
|
|
|
295
459
|
|
|
296
460
|
```ruby
|
|
297
461
|
# config/puma.rb
|
|
298
|
-
|
|
462
|
+
|
|
299
463
|
before_fork do
|
|
300
464
|
GoodJob.shutdown
|
|
301
465
|
end
|
|
302
|
-
|
|
466
|
+
|
|
303
467
|
on_worker_boot do
|
|
304
468
|
GoodJob.restart
|
|
305
469
|
end
|
|
306
|
-
|
|
470
|
+
|
|
307
471
|
on_worker_shutdown do
|
|
308
472
|
GoodJob.shutdown
|
|
309
473
|
end
|
|
310
|
-
|
|
474
|
+
|
|
311
475
|
MAIN_PID = Process.pid
|
|
312
476
|
at_exit do
|
|
313
477
|
GoodJob.shutdown if Process.pid == MAIN_PID
|
|
314
478
|
end
|
|
315
479
|
```
|
|
316
|
-
|
|
480
|
+
|
|
317
481
|
GoodJob is compatible with Puma's `preload_app!` method.
|
|
318
|
-
|
|
319
|
-
###
|
|
482
|
+
|
|
483
|
+
### Migrate to GoodJob from a different ActiveJob backend
|
|
320
484
|
|
|
321
485
|
If your application is already using an ActiveJob backend, you will need to install GoodJob to enqueue and perform newly created jobs _and_ finish performing pre-existing jobs on the previous backend.
|
|
322
486
|
|
|
@@ -331,7 +495,7 @@ If your application is already using an ActiveJob backend, you will need to inst
|
|
|
331
495
|
```
|
|
332
496
|
|
|
333
497
|
1. Continue running executors for both backends. For example, on Heroku it's possible to run [two processes](https://help.heroku.com/CTFS2TJK/how-do-i-run-multiple-processes-on-a-dyno) within the same dyno:
|
|
334
|
-
|
|
498
|
+
|
|
335
499
|
```procfile
|
|
336
500
|
# Procfile
|
|
337
501
|
# ...
|
|
@@ -340,11 +504,11 @@ If your application is already using an ActiveJob backend, you will need to inst
|
|
|
340
504
|
|
|
341
505
|
1. Once you are confident that no unperformed jobs remain in the previous ActiveJob backend, code and configuration for that backend can be completely removed.
|
|
342
506
|
|
|
343
|
-
###
|
|
507
|
+
### Monitor and preserve worked jobs
|
|
344
508
|
|
|
345
509
|
GoodJob is fully instrumented with [`ActiveSupport::Notifications`](https://edgeguides.rubyonrails.org/active_support_instrumentation.html#introduction-to-instrumentation).
|
|
346
510
|
|
|
347
|
-
By default, GoodJob will delete job records after they are run, regardless of whether they succeed or not (raising a kind of `StandardError`), unless they are interrupted (raising a kind of `Exception`).
|
|
511
|
+
By default, GoodJob will delete job records after they are run, regardless of whether they succeed or not (raising a kind of `StandardError`), unless they are interrupted (raising a kind of `Exception`).
|
|
348
512
|
|
|
349
513
|
To preserve job records for later inspection, set an initializer:
|
|
350
514
|
|
|
@@ -356,7 +520,7 @@ GoodJob.preserve_job_records = true
|
|
|
356
520
|
It is also necessary to delete these preserved jobs from the database after a certain time period:
|
|
357
521
|
|
|
358
522
|
- For example, in a Rake task:
|
|
359
|
-
|
|
523
|
+
|
|
360
524
|
```ruby
|
|
361
525
|
GoodJob::Job.finished(1.day.ago).delete_all
|
|
362
526
|
```
|
|
@@ -367,7 +531,7 @@ It is also necessary to delete these preserved jobs from the database after a ce
|
|
|
367
531
|
$ bundle exec good_job cleanup_preserved_jobs --before-seconds-ago=86400
|
|
368
532
|
```
|
|
369
533
|
|
|
370
|
-
##
|
|
534
|
+
## Contribute
|
|
371
535
|
|
|
372
536
|
Contributions are welcomed and appreciated π
|
|
373
537
|
|
|
@@ -412,7 +576,7 @@ $ bundle install
|
|
|
412
576
|
# => Using good_job 0.1.0 from https://github.com/bensheldon/good_job.git (at /Users/You/Projects/good_job@dc57fb0)
|
|
413
577
|
```
|
|
414
578
|
|
|
415
|
-
###
|
|
579
|
+
### Release
|
|
416
580
|
|
|
417
581
|
Package maintainers can release this gem by running:
|
|
418
582
|
|
|
@@ -426,7 +590,7 @@ $ gem signin
|
|
|
426
590
|
# Update version number, changelog, and create git commit:
|
|
427
591
|
$ bundle exec rake release[minor] # major,minor,patch
|
|
428
592
|
|
|
429
|
-
# ..and follow subsequent directions.
|
|
593
|
+
# ..and follow subsequent directions.
|
|
430
594
|
```
|
|
431
595
|
|
|
432
596
|
## License
|