good_job 1.2.0 → 1.2.5

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (37) hide show
  1. checksums.yaml +4 -4
  2. data/CHANGELOG.md +118 -2
  3. data/README.md +280 -152
  4. data/engine/app/controllers/good_job/active_jobs_controller.rb +8 -0
  5. data/engine/app/controllers/good_job/base_controller.rb +5 -0
  6. data/engine/app/controllers/good_job/dashboards_controller.rb +50 -0
  7. data/engine/app/helpers/good_job/application_helper.rb +4 -0
  8. data/engine/app/views/assets/_style.css.erb +16 -0
  9. data/engine/app/views/good_job/active_jobs/show.html.erb +1 -0
  10. data/engine/app/views/good_job/dashboards/index.html.erb +19 -0
  11. data/engine/app/views/layouts/good_job/base.html.erb +50 -0
  12. data/engine/app/views/shared/_chart.erb +51 -0
  13. data/engine/app/views/shared/_jobs_table.erb +26 -0
  14. data/engine/app/views/vendor/bootstrap/_bootstrap-native.js.erb +1662 -0
  15. data/engine/app/views/vendor/bootstrap/_bootstrap.css.erb +10258 -0
  16. data/engine/app/views/vendor/chartist/_chartist.css.erb +613 -0
  17. data/engine/app/views/vendor/chartist/_chartist.js.erb +4516 -0
  18. data/engine/config/routes.rb +4 -0
  19. data/engine/lib/good_job/engine.rb +5 -0
  20. data/lib/active_job/queue_adapters/good_job_adapter.rb +3 -2
  21. data/lib/generators/good_job/install_generator.rb +8 -0
  22. data/lib/good_job.rb +40 -25
  23. data/lib/good_job/adapter.rb +44 -4
  24. data/lib/good_job/cli.rb +66 -13
  25. data/lib/good_job/configuration.rb +52 -0
  26. data/lib/good_job/current_execution.rb +2 -0
  27. data/lib/good_job/job.rb +118 -21
  28. data/lib/good_job/lockable.rb +125 -14
  29. data/lib/good_job/log_subscriber.rb +70 -4
  30. data/lib/good_job/multi_scheduler.rb +6 -0
  31. data/lib/good_job/notifier.rb +64 -27
  32. data/lib/good_job/performer.rb +38 -0
  33. data/lib/good_job/railtie.rb +1 -0
  34. data/lib/good_job/scheduler.rb +34 -20
  35. data/lib/good_job/version.rb +2 -1
  36. metadata +163 -7
  37. data/lib/good_job/pg_locks.rb +0 -21
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 5fbc2c6b6b2d464188d5c89aa0d1a270c753bea661cda246ac41a4a86fa9cd77
4
- data.tar.gz: 5fa873aa2732881e5b42cc63bbf2bdfe8fd823f224380fdc5c7e92e665b96584
3
+ metadata.gz: ca2673887424565881a47ad6eaf9300b6281e43c87066f01ceb899b76716d43e
4
+ data.tar.gz: 6a0f749171316300ebfc7257d1d62efc330092391f026d6f50e20ccda5fee208
5
5
  SHA512:
6
- metadata.gz: d7278790575965381edc0c5b9ee478e4156bbfe4dc97e9304dfed5a60f6d89043bf1ff567249680ff24aa85dd6a3b1539fda519e98a19762b02d657b1d943630
7
- data.tar.gz: ce1bc4d1c2d9fa480f1a7544010f58ee7d1b656fe31d77869999e265cf3d0481a9da1e95ee855a351e36f11d992fb40ac85113cf0d0082c22e4988ca32ee149c
6
+ metadata.gz: 4fcef37d707e0d25f44965949abfeba914a4ee9f743ed32498e28df95751f666bdfb942c12049b23d78ff2b52e14465ca1cda2007bfae3541f9f0c53e6f74964
7
+ data.tar.gz: db7d224567bde1427210638226d5005dd279a2c73c97ff9c19eb40edc523ae7863973cccf7d263a08dcf56ec66fd112a492afa269dad4b12cc70b79027baf296
data/CHANGELOG.md CHANGED
@@ -1,6 +1,122 @@
1
1
  # Changelog
2
2
 
3
- ## [v1.2.0](https://github.com/bensheldon/good_job/tree/v1.2.0) (2020-08-19)
3
+ ## [v1.2.5](https://github.com/bensheldon/good_job/tree/v1.2.5) (2020-09-17)
4
+
5
+ [Full Changelog](https://github.com/bensheldon/good_job/compare/v1.2.4...v1.2.5)
6
+
7
+ **Implemented enhancements:**
8
+
9
+ - Use Zeitwerk for auto-loading [\#87](https://github.com/bensheldon/good_job/issues/87)
10
+
11
+ **Fixed bugs:**
12
+
13
+ - `poll-interval=-1` does not disable polling as intended [\#133](https://github.com/bensheldon/good_job/issues/133)
14
+
15
+ **Closed issues:**
16
+
17
+ - Lint - Introduce line character limits [\#122](https://github.com/bensheldon/good_job/issues/122)
18
+ - Jobs are not processed in multi schema setup. Apartment + GoodJob \( post 1.1.2 \) [\#117](https://github.com/bensheldon/good_job/issues/117)
19
+ - Host a documentation sprint [\#48](https://github.com/bensheldon/good_job/issues/48)
20
+
21
+ **Merged pull requests:**
22
+
23
+ - Test GoodJob against Rails HEAD [\#144](https://github.com/bensheldon/good_job/pull/144) ([bensheldon](https://github.com/bensheldon))
24
+ - Update Gemspec to reflect that GoodJob is not compatible with Rails 5.1 [\#143](https://github.com/bensheldon/good_job/pull/143) ([bensheldon](https://github.com/bensheldon))
25
+ - Drop Ruby 2.4 support [\#142](https://github.com/bensheldon/good_job/pull/142) ([morgoth](https://github.com/morgoth))
26
+ - Prevent jobs hanging [\#141](https://github.com/bensheldon/good_job/pull/141) ([morgoth](https://github.com/morgoth))
27
+ - Remove arguments from perform method [\#140](https://github.com/bensheldon/good_job/pull/140) ([morgoth](https://github.com/morgoth))
28
+ - Extract "execute" method to reduce "perform" method complexity [\#138](https://github.com/bensheldon/good_job/pull/138) ([morgoth](https://github.com/morgoth))
29
+ - Correct example on how to configure multiple queues by command line. [\#135](https://github.com/bensheldon/good_job/pull/135) ([morgoth](https://github.com/morgoth))
30
+ - Add explicit require\_paths to gemspec for engine [\#134](https://github.com/bensheldon/good_job/pull/134) ([bensheldon](https://github.com/bensheldon))
31
+ - Spike on data dashboard; pull in full Bootstrap CSS and JS [\#131](https://github.com/bensheldon/good_job/pull/131) ([bensheldon](https://github.com/bensheldon))
32
+ - Update ActionMailer Job class, to match the default [\#130](https://github.com/bensheldon/good_job/pull/130) ([morgoth](https://github.com/morgoth))
33
+ - Add initial Engine scaffold [\#125](https://github.com/bensheldon/good_job/pull/125) ([bensheldon](https://github.com/bensheldon))
34
+ - Use `connection.quote\_table\_name` and add spacing for SQL concatenation [\#124](https://github.com/bensheldon/good_job/pull/124) ([bensheldon](https://github.com/bensheldon))
35
+ - Zeitwerk Loader Implementation [\#123](https://github.com/bensheldon/good_job/pull/123) ([gadimbaylisahil](https://github.com/gadimbaylisahil))
36
+ - Update code-level documentation [\#111](https://github.com/bensheldon/good_job/pull/111) ([bensheldon](https://github.com/bensheldon))
37
+
38
+ ## [v1.2.4](https://github.com/bensheldon/good_job/tree/v1.2.4) (2020-09-01)
39
+
40
+ [Full Changelog](https://github.com/bensheldon/good_job/compare/v1.2.3...v1.2.4)
41
+
42
+ **Implemented enhancements:**
43
+
44
+ - Add environment variable to mirror `cleanup\_preserved\_jobs --before-seconds-ago=SECONDS` [\#110](https://github.com/bensheldon/good_job/issues/110)
45
+
46
+ **Closed issues:**
47
+
48
+ - Remove unused PgLocks class [\#121](https://github.com/bensheldon/good_job/issues/121)
49
+ - Fix minor issue with CommandLine option links in README.md [\#116](https://github.com/bensheldon/good_job/issues/116)
50
+ - Unused .advisory\_lock\_details in PgLocks [\#105](https://github.com/bensheldon/good_job/issues/105)
51
+
52
+ **Merged pull requests:**
53
+
54
+ - Remove unused PgLocks class [\#120](https://github.com/bensheldon/good_job/pull/120) ([gadimbaylisahil](https://github.com/gadimbaylisahil))
55
+ - Better table name detection for Job queries [\#119](https://github.com/bensheldon/good_job/pull/119) ([gadimbaylisahil](https://github.com/gadimbaylisahil))
56
+ - Fix readme CommandLine option links [\#115](https://github.com/bensheldon/good_job/pull/115) ([gadimbaylisahil](https://github.com/gadimbaylisahil))
57
+ - Allow env variable config for cleanups [\#114](https://github.com/bensheldon/good_job/pull/114) ([gadimbaylisahil](https://github.com/gadimbaylisahil))
58
+ - Have YARD render markdown files with GFM \(Github Flavored Markdown\) [\#113](https://github.com/bensheldon/good_job/pull/113) ([bensheldon](https://github.com/bensheldon))
59
+ - Add markdownlint to lint readme [\#109](https://github.com/bensheldon/good_job/pull/109) ([bensheldon](https://github.com/bensheldon))
60
+ - Remove unused method in PgLocks [\#107](https://github.com/bensheldon/good_job/pull/107) ([gadimbaylisahil](https://github.com/gadimbaylisahil))
61
+ - Re-organize Readme: frontload configuration, add Table of Contents [\#106](https://github.com/bensheldon/good_job/pull/106) ([bensheldon](https://github.com/bensheldon))
62
+
63
+ ## [v1.2.3](https://github.com/bensheldon/good_job/tree/v1.2.3) (2020-08-27)
64
+
65
+ [Full Changelog](https://github.com/bensheldon/good_job/compare/v1.2.2...v1.2.3)
66
+
67
+ **Closed issues:**
68
+
69
+ - requiring more dependencies in then needed [\#103](https://github.com/bensheldon/good_job/issues/103)
70
+
71
+ **Merged pull requests:**
72
+
73
+ - stop depending on all rails libs [\#104](https://github.com/bensheldon/good_job/pull/104) ([thilo](https://github.com/thilo))
74
+
75
+ ## [v1.2.2](https://github.com/bensheldon/good_job/tree/v1.2.2) (2020-08-27)
76
+
77
+ [Full Changelog](https://github.com/bensheldon/good_job/compare/v1.2.1...v1.2.2)
78
+
79
+ **Implemented enhancements:**
80
+
81
+ - Run Github Action tests against Ruby 2.5, 2.6, 2.7 [\#100](https://github.com/bensheldon/good_job/issues/100)
82
+
83
+ **Fixed bugs:**
84
+
85
+ - Freezes puma on code change [\#95](https://github.com/bensheldon/good_job/issues/95)
86
+ - Ruby 2.7 keyword arguments warning [\#93](https://github.com/bensheldon/good_job/issues/93)
87
+
88
+ **Closed issues:**
89
+
90
+ - Add test for `rails g good\_job:install` [\#57](https://github.com/bensheldon/good_job/issues/57)
91
+
92
+ **Merged pull requests:**
93
+
94
+ - Use more ActiveRecord in Lockable and not connection.execute [\#102](https://github.com/bensheldon/good_job/pull/102) ([bensheldon](https://github.com/bensheldon))
95
+ - Run CI tests on Ruby 2.5, 2.6, and 2.7 [\#101](https://github.com/bensheldon/good_job/pull/101) ([arku](https://github.com/arku))
96
+ - Return to using executor.wrap around Scheduler execution task [\#99](https://github.com/bensheldon/good_job/pull/99) ([bensheldon](https://github.com/bensheldon))
97
+ - Fix Ruby 2.7 keyword arguments warning [\#98](https://github.com/bensheldon/good_job/pull/98) ([arku](https://github.com/arku))
98
+ - Remove executor/reloader for less interlocking [\#97](https://github.com/bensheldon/good_job/pull/97) ([sj26](https://github.com/sj26))
99
+ - Name the thread pools [\#96](https://github.com/bensheldon/good_job/pull/96) ([sj26](https://github.com/sj26))
100
+ - Add test for `rails g good\_job:install` [\#94](https://github.com/bensheldon/good_job/pull/94) ([arku](https://github.com/arku))
101
+
102
+ ## [v1.2.1](https://github.com/bensheldon/good_job/tree/v1.2.1) (2020-08-21)
103
+
104
+ [Full Changelog](https://github.com/bensheldon/good_job/compare/v1.2.0...v1.2.1)
105
+
106
+ **Fixed bugs:**
107
+
108
+ - undefined method `thread\_mattr\_accessor' when not requiring the Sprockets Railstie [\#85](https://github.com/bensheldon/good_job/issues/85)
109
+
110
+ **Closed issues:**
111
+
112
+ - Document comparison of GoodJob with other backends [\#51](https://github.com/bensheldon/good_job/issues/51)
113
+
114
+ **Merged pull requests:**
115
+
116
+ - Explicitly require thread\_mattr\_accessor from ActiveSupport [\#86](https://github.com/bensheldon/good_job/pull/86) ([bensheldon](https://github.com/bensheldon))
117
+ - Add comparison of other backends to Readme [\#84](https://github.com/bensheldon/good_job/pull/84) ([bensheldon](https://github.com/bensheldon))
118
+
119
+ ## [v1.2.0](https://github.com/bensheldon/good_job/tree/v1.2.0) (2020-08-20)
4
120
 
5
121
  [Full Changelog](https://github.com/bensheldon/good_job/compare/v1.1.4...v1.2.0)
6
122
 
@@ -238,6 +354,7 @@
238
354
  - Add pg gem as explicit dependency [\#13](https://github.com/bensheldon/good_job/pull/13) ([bensheldon](https://github.com/bensheldon))
239
355
  - Bump nokogiri from 1.10.7 to 1.10.9 [\#12](https://github.com/bensheldon/good_job/pull/12) ([dependabot[bot]](https://github.com/apps/dependabot))
240
356
  - Add Appraisal with tests for Rails 5.1, 5.2, 6.0 [\#11](https://github.com/bensheldon/good_job/pull/11) ([bensheldon](https://github.com/bensheldon))
357
+ - Use Rails.logger and ActiveSupport::Notifications for logging instead of puts [\#10](https://github.com/bensheldon/good_job/pull/10) ([bensheldon](https://github.com/bensheldon))
241
358
 
242
359
  ## [v0.2.0](https://github.com/bensheldon/good_job/tree/v0.2.0) (2020-03-06)
243
360
 
@@ -245,7 +362,6 @@
245
362
 
246
363
  **Merged pull requests:**
247
364
 
248
- - Use Rails.logger and ActiveSupport::Notifications for logging instead of puts [\#10](https://github.com/bensheldon/good_job/pull/10) ([bensheldon](https://github.com/bensheldon))
249
365
  - Remove minitest files [\#9](https://github.com/bensheldon/good_job/pull/9) ([bensheldon](https://github.com/bensheldon))
250
366
  - Use scheduled\_at and priority for scheduling [\#8](https://github.com/bensheldon/good_job/pull/8) ([bensheldon](https://github.com/bensheldon))
251
367
  - Create Github Action workflow for PRs and Issues [\#7](https://github.com/bensheldon/good_job/pull/7) ([bensheldon](https://github.com/bensheldon))
data/README.md CHANGED
@@ -4,197 +4,281 @@ GoodJob is a multithreaded, Postgres-based, ActiveJob backend for Ruby on Rails.
4
4
 
5
5
  **Inspired by [Delayed::Job](https://github.com/collectiveidea/delayed_job) and [Que](https://github.com/que-rb/que), GoodJob is designed for maximum compatibility with Ruby on Rails, ActiveJob, and Postgres to be simple and performant for most workloads.**
6
6
 
7
- - **Designed for ActiveJob.** Complete support for [async, queues, delays, priorities, timeouts, and retries](https://edgeguides.rubyonrails.org/active_job_basics.html) with near-zero configuration.
8
- - **Built for Rails.** Fully adopts Ruby on Rails [threading and code execution guidelines](https://guides.rubyonrails.org/threading_and_code_execution.html) with [Concurrent::Ruby](https://github.com/ruby-concurrency/concurrent-ruby).
7
+ - **Designed for ActiveJob.** Complete support for [async, queues, delays, priorities, timeouts, and retries](https://edgeguides.rubyonrails.org/active_job_basics.html) with near-zero configuration.
8
+ - **Built for Rails.** Fully adopts Ruby on Rails [threading and code execution guidelines](https://guides.rubyonrails.org/threading_and_code_execution.html) with [Concurrent::Ruby](https://github.com/ruby-concurrency/concurrent-ruby).
9
9
  - **Backed by Postgres.** Relies upon Postgres integrity, session-level Advisory Locks to provide run-once safety and stay within the limits of `schema.rb`, and LISTEN/NOTIFY to reduce queuing latency.
10
10
  - **For most workloads.** Targets full-stack teams, economy-minded solo developers, and applications that enqueue less than 1-million jobs/day.
11
11
 
12
12
  For more of the story of GoodJob, read the [introductory blog post](https://island94.org/2020/07/introducing-goodjob-1-0).
13
13
 
14
- ## Installation
14
+ <details markdown="1">
15
+ <summary><strong>📊 Comparison of GoodJob with other job queue backends (click to expand)</strong></summary>
16
+
17
+ | | Queues, priority, retries | Database | Concurrency | Reliability/Integrity | Latency |
18
+ |-----------------|---------------------------|---------------------------------------|-------------------|------------------------|--------------------------|
19
+ | **GoodJob** | ✅ Yes | ✅ Postgres | ✅ Multithreaded | ✅ ACID, Advisory Locks | ✅ Postgres LISTEN/NOTIFY |
20
+ | **Que** | ✅ Yes | 🔶️ Postgres, requires `structure.sql` | ✅ Multithreaded | ✅ ACID, Advisory Locks | ✅ Postgres LISTEN/NOTIFY |
21
+ | **Delayed Job** | ✅ Yes | ✅ Postgres | 🔴 Single-threaded | ✅ ACID, record-based | 🔶 Polling |
22
+ | **Sidekiq** | ✅ Yes | 🔴 Redis | ✅ Multithreaded | 🔴 Crashes lose jobs | ✅ Redis BRPOP |
23
+ | **Sidekiq Pro** | ✅ Yes | 🔴 Redis | ✅ Multithreaded | ✅ Redis RPOPLPUSH | ✅ Redis RPOPLPUSH |
24
+
25
+ </details>
26
+
27
+ ## Table of contents
28
+
29
+ - [Set up](#set-up)
30
+ - [Configuration](#configuration)
31
+ - [Command-line options](#command-line-options)
32
+ - [`good_job start`](#good_job-start)
33
+ - [`good_job cleanup_preserved_jobs`](#good_job-cleanup_preserved_jobs)
34
+ - [Adapter options](#adapter-options)
35
+ - [Global options](#global-options)
36
+ - [Going deeper](#going-deeper)
37
+ - [Exceptions, retries, and reliability](#exceptions-retries-and-reliability)
38
+ - [Exceptions](#exceptions)
39
+ - [Retries](#retries)
40
+ - [ActionMailer retries](#actionmailer-retries)
41
+ - [Timeouts](#timeouts)
42
+ - [Optimize queues, threads, and processes](#optimize-queues-threads-and-processes)
43
+ - [Database connections](#database-connections)
44
+ - [Executing jobs async / in-process](#executing-jobs-async--in-process)
45
+ - [Migrating to GoodJob from a different ActiveJob backend](#migrating-to-goodjob-from-a-different-activejob-backend)
46
+ - [Monitoring and preserving worked jobs](#monitoring-and-preserving-worked-jobs)
47
+ - [Contributing](#contributing)
48
+ - [Gem development](#gem-development)
49
+ - [Releasing](#releasing)
50
+ - [License](#license)
51
+
52
+ ## Set up
53
+
54
+ 1. Add `good_job` to your application's Gemfile:
15
55
 
16
- Add this line to your application's Gemfile:
56
+ ```ruby
57
+ gem 'good_job'
58
+ ```
17
59
 
18
- ```ruby
19
- gem 'good_job'
20
- ```
60
+ 1. Install the gem:
21
61
 
22
- And then execute:
23
- ```bash
24
- $ bundle install
25
- ```
62
+ ```bash
63
+ $ bundle install
64
+ ```
26
65
 
27
- ## Usage
66
+ 1. Run the GoodJob install generator. This will generate a database migration to create a table for GoodJob's job records:
28
67
 
29
- 1. Create a database migration:
30
-
31
- ```bash
68
+ ```bash
32
69
  $ bin/rails g good_job:install
33
70
  ```
34
71
 
35
72
  Run the migration:
36
-
73
+
37
74
  ```bash
38
75
  $ bin/rails db:migrate
39
76
  ```
40
-
77
+
41
78
  1. Configure the ActiveJob adapter:
42
-
43
- ```ruby
79
+
80
+ ```ruby
44
81
  # config/application.rb
45
82
  config.active_job.queue_adapter = :good_job
46
83
  ```
47
-
48
- By default, using `:good_job` is equivalent to manually configuring the adapter:
49
-
84
+
85
+ 1. Inside of your application, queue your job 🎉:
86
+
50
87
  ```ruby
51
- # config/environments/development.rb
52
- config.active_job.queue_adapter = GoodJob::Adapter.new(execution_mode: :inline)
53
-
54
- # config/environments/test.rb
55
- config.active_job.queue_adapter = GoodJob::Adapter.new(execution_mode: :inline)
56
-
57
- # config/environments/production.rb
58
- config.active_job.queue_adapter = GoodJob::Adapter.new(execution_mode: :external)
88
+ YourJob.perform_later
59
89
  ```
60
90
 
61
- 1. Queue your job 🎉:
62
-
91
+ GoodJob supports all ActiveJob features:
92
+
63
93
  ```ruby
64
94
  YourJob.set(queue: :some_queue, wait: 5.minutes, priority: 10).perform_later
65
95
  ```
66
96
 
67
- 1. In production, the scheduler is designed to run in its own process:
68
-
69
- ```bash
70
- $ bundle exec good_job
71
- ```
72
-
73
- Configuration options available with `help`:
97
+ 1. In development, GoodJob executes jobs immediately. In production, GoodJob provides different options:
74
98
 
75
- ```bash
76
- $ bundle exec good_job help start
77
-
78
- Usage:
79
- good_job start
80
-
81
- Options:
82
- [--max-threads=N] # Maximum number of threads to use for working jobs (default: ActiveRecord::Base.connection_pool.size)
83
- [--queues=queue1,queue2(;queue3,queue4:5;-queue1,queue2)] # Queues to work from. Separate multiple queues with commas; exclude queues with a leading minus; separate isolated execution pools with semicolons and threads with colons (default: *)
84
- [--poll-interval=N] # Interval between polls for available jobs in seconds (default: 1)
85
-
86
- Start job worker
87
- ```
99
+ - By default, GoodJob separates job enqueuing from job execution so that jobs can be scaled independently of the web server. Use the GoodJob command-line tool to execute jobs:
88
100
 
89
- 1. Optimize execution to reduce congestion and execution latency.
101
+ ```bash
102
+ $ bundle exec good_job start
103
+ ```
90
104
 
91
- By default, GoodJob creates a single thread execution pool that will execute jobs from any queue. Depending on your application's workload, job types, and service level objectives, you may wish to optimize execution resources; for example, providing dedicated execution resources for transactional emails so they are not delayed by long-running batch jobs. Some options:
105
+ Ideally the command-line tool should be run on a separate machine or container from the web process. For example, on Heroku:
92
106
 
93
- - Multiple execution pools within a single process:
94
-
95
- ```bash
96
- $ bundle exec good_job --queues=transactional_messages:2;batch_processing:1;-transactional_messages,batch_processing:2;* --max-threads=5
107
+ ```Procfile
108
+ web: rails server
109
+ worker: bundle exec good_job start
97
110
  ```
98
-
99
- This configuration will result in a single process with 4 isolated thread execution pools. Isolated execution pools are separated with a semicolon (`;`) and queue names and thread counts with a colon (`:`)
100
-
101
- - `transactional_messages:2`: execute jobs enqueued on `transactional_messages` with up to 2 threads.
102
- - `batch_processing:1` execute jobs enqueued on `batch_processing` with a single thread.
103
- - `-transactional_messages,batch_processing`: execute jobs enqueued on _any_ queue _excluding_ `transactional_messages` or `batch_processing` with up to 2 threads.
104
- - `*`: execute jobs on any queue on up to 5 threads, as configured by `--max-threads=5`
105
-
106
- For moderate workloads, multiple isolated thread execution pools offers a good balance between congestion management and economy.
107
-
108
- Configuration can be injected by environment variables too:
109
-
110
- ```bash
111
- $ GOOD_JOB_QUEUES="transactional_messages:2;batch_processing:1;-transactional_messages,batch_processing:2;*" GOOD_JOB_MAX_THREADS=5 bundle exec good_job
111
+
112
+ The command-line tool supports a variety of options, see the reference below for command-line configuration.
113
+
114
+ - GoodJob can also be configured to execute jobs within the web server process to save on resources. This is useful for low-workloads when economy is paramount.
115
+
112
116
  ```
113
-
114
- - Multiple processes; for example, on Heroku:
115
-
116
- ```procfile
117
- # Procfile
118
-
119
- # Separate dyno types
120
- worker: bundle exec good_job --max-threads=5
121
- transactional_worker: bundle exec good_job --queues=transactional_messages --max-threads=2
122
- batch_worker: bundle exec good_job --queues=batch_processing --max-threads=1
123
-
124
- # Combined multi-process dyno
125
- combined_worker: bundle exec good_job --max-threads=5 & bundle exec good_job --queues=transactional_messages --max-threads=2 & bundle exec good_job --queues=batch_processing --max-threads=1 & wait -n
117
+ $ GOOD_JOB_EXECUTION_MODE=async rails server
126
118
  ```
127
-
128
- Running multiple processes can optimize for CPU performance at the expense of greater memory and system resource usage.
129
119
 
130
- _Keep in mind, queue operations and management is an advanced discipline. This stuff is complex, especially for heavy workloads and unique processing requirements. Good job 👍_
120
+ Additional configuration is likely necessary, see the reference below for async configuration.
131
121
 
132
- ### Error handling, retries, and reliability
122
+ ## Configuration
133
123
 
134
- GoodJob guarantees that a completely-performed job will run once and only once. GoodJob fully supports ActiveJob's built-in functionality for error handling, retries and timeouts. Writing reliable, transactional, and idempotent `ActiveJob#perform` methods is outside the scope of GoodJob.
124
+ ### Command-line options
135
125
 
136
- #### Error handling
126
+ There several top-level commands available through the `good_job` command-line tool.
137
127
 
138
- By default, if a job raises an error while it is being performed, _and it bubbles up to the GoodJob backend_, GoodJob will be immediately re-perform the job until it finishes successfully.
128
+ Configuration options are available with `help`.
139
129
 
140
- - `Exception`-type errors, such as a SIGINT, will always cause a job to be re-performed.
141
- - `StandardError`-type errors, by default, will cause a job to be re-performed, though this is configurable:
142
-
143
- ```ruby
144
- # config/initializers/good_job.rb
145
- GoodJob.reperform_jobs_on_standard_error = true # => default
146
- ```
130
+ #### `good_job start`
131
+
132
+ `good_job start` executes queued jobs.
133
+
134
+ ```bash
135
+ $ bundle exec good_job help start
136
+
137
+ Usage:
138
+ good_job start
139
+
140
+ Options:
141
+ [--max-threads=COUNT] # Maximum number of threads to use for working jobs. (env var: GOOD_JOB_MAX_THREADS, default: 5)
142
+ [--queues=QUEUE_LIST] # Queues to work from. (env var: GOOD_JOB_QUEUES, default: *)
143
+ [--poll-interval=SECONDS] # Interval between polls for available jobs in seconds (env var: GOOD_JOB_POLL_INTERVAL, default: 1)
144
+
145
+ Executes queued jobs.
146
+
147
+ All options can be configured with environment variables.
148
+ See option descriptions for the matching environment variable name.
149
+
150
+ == Configuring queues
151
+ Separate multiple queues with commas; exclude queues with a leading minus;
152
+ separate isolated execution pools with semicolons and threads with colons.
153
+ ```
154
+
155
+ #### `good_job cleanup_preserved_jobs`
156
+
157
+ `good_job cleanup_preserved_jobs` deletes preserved job records. See [`GoodJob.preserve_job_records` for when this command is useful.
158
+
159
+ ```bash
160
+ $ bundle exec good_job help cleanup_preserved_jobs
161
+
162
+ Usage:
163
+ good_job cleanup_preserved_jobs
164
+
165
+ Options:
166
+ [--before-seconds-ago=SECONDS] # Delete records finished more than this many seconds ago (env var: GOOD_JOB_CLEANUP_PRESERVED_JOBS_BEFORE_SECONDS_AGO, default: 86400)
167
+
168
+ Deletes preserved job records.
169
+
170
+ By default, GoodJob deletes job records when the job is performed and this
171
+ command is not necessary.
172
+
173
+ However, when `GoodJob.preserve_job_records = true`, the jobs will be
174
+ preserved in the database. This is useful when wanting to analyze or
175
+ inspect job performance.
176
+
177
+ If you are preserving job records this way, use this command regularly
178
+ to delete old records and preserve space in your database.
179
+ ```
180
+
181
+ ### Adapter options
182
+
183
+ To use GoodJob, you can set `config.active_job.queue_adapter` to a `:good_job` or to an instance of `GoodJob::Adapter`, which you can configure with several options:
184
+
185
+ - `execution_mode` (symbol) specifies how and where jobs should be executed. You can also set this with the environment variable `GOOD_JOB_EXECUTION_MODE`. It can be any one of:
186
+ - `:inline` executes jobs immediately in whatever process queued them (usually the web server process). This should only be used in test and development environments.
187
+ - `:external` causes the adapter to enqueue jobs, but not execute them. When using this option (the default for production environments), you’ll need to use the command-line tool to actually execute your jobs.
188
+ - `:async` causes the adapter to execute you jobs in separate threads in whatever process queued them (usually the web process). This is akin to running the command-line tool’s code inside your web server. It can be more economical for small workloads (you don’t need a separate machine or environment for running your jobs), but if your web server is under heavy load or your jobs require a lot of resources, you should choose `:external` instead.
189
+ - `max_threads` (integer) sets the maximum number of threads to use when `execution_mode` is set to `:async`. You can also set this with the environment variable `GOOD_JOB_MAX_THREADS`.
190
+ - `queues` (string) determines which queues to execute jobs from when `execution_mode` is set to `:async`. See the description of `good_job start` for more details on the format of this string. You can also set this with the environment variable `GOOD_JOB_QUEUES`.
191
+ - `poll_interval` (integer) sets the number of seconds between polls for jobs when `execution_mode` is set to `:async`. You can also set this with the environment variable `GOOD_JOB_POLL_INTERVAL`.
192
+
193
+ Using the symbol instead of explicitly configuring the options above (i.e. setting `config.active_job.queue_adapter = :good_job`) is equivalent to:
194
+
195
+ ```ruby
196
+ # config/environments/development.rb
197
+ config.active_job.queue_adapter = GoodJob::Adapter.new(execution_mode: :inline)
198
+
199
+ # config/environments/test.rb
200
+ config.active_job.queue_adapter = GoodJob::Adapter.new(execution_mode: :inline)
201
+
202
+ # config/environments/production.rb
203
+ config.active_job.queue_adapter = GoodJob::Adapter.new(execution_mode: :external)
204
+ ```
205
+
206
+ ### Global options
207
+
208
+ Good Job’s general behavior can also be configured via several attributes directly on the `GoodJob` module:
209
+
210
+ - **`GoodJob.logger`** ([Rails Logger](https://api.rubyonrails.org/classes/ActiveSupport/Logger.html)) lets you set a custom logger for GoodJob. It should be an instance of a Rails `Logger`.
211
+ - **`GoodJob.preserve_job_records`** (boolean) keeps job records in your database even after jobs are completed. (Default: `false`)
212
+ - **`GoodJob.reperform_jobs_on_standard_error`** (boolean) causes jobs to be re-queued and retried if they raise an instance of `StandardError`. Instances of `Exception`, like SIGINT, will *always* be retried, regardless of this attribute’s value. (Default: `true`)
213
+ - **`GoodJob.on_thread_error`** (proc, lambda, or callable) will be called when an Exception. It can be useful for logging errors to bug tracking services, like Sentry or Airbrake.
147
214
 
148
- To report errors that _do_ bubble up to the GoodJob backend, assign a callable to `GoodJob.on_thread_error`. For example:
215
+ You’ll generally want to configure these in `config/initializers/good_job.rb`, like so:
149
216
 
150
217
  ```ruby
151
218
  # config/initializers/good_job.rb
219
+ GoodJob.preserve_job_records = true
220
+ GoodJob.reperform_jobs_on_standard_error = false
221
+ GoodJob.on_thread_error = -> (exception) { Raven.capture_exception(exception) }
222
+ ```
152
223
 
153
- # With Sentry (or Bugsnag, Airbrake, Honeybadger, etc.)
224
+ ## Going deeper
225
+
226
+ ### Exceptions, retries, and reliability
227
+
228
+ GoodJob guarantees that a completely-performed job will run once and only once. GoodJob fully supports ActiveJob's built-in functionality for error handling, retries and timeouts.
229
+
230
+ #### Exceptions
231
+
232
+ ActiveJob provides [tools for rescuing and retrying exceptions](https://guides.rubyonrails.org/active_job_basics.html#exceptions), including `retry_on`, `discard_on`, `rescue_from` that will rescue exceptions before they get to GoodJob.
233
+
234
+ If errors do reach GoodJob, you can assign a callable to `GoodJob.on_thread_error` to be notified. For example, to log errors to an exception monitoring service like Sentry (or Bugsnag, Airbrake, Honeybadger, etc.):
235
+
236
+ ```ruby
237
+ # config/initializers/good_job.rb
154
238
  GoodJob.on_thread_error = -> (exception) { Raven.capture_exception(exception) }
155
239
  ```
156
240
 
157
- ### Retrying jobs
241
+ #### Retries
158
242
 
159
- ActiveJob can be configured to retry an infinite number of times, with an exponential backoff. Using ActiveJob's `retry_on` will ensure that errors do not bubble up to the GoodJob backend:
243
+ By default, GoodJob will automatically and immediately retry a job when an exception is raised to GoodJob.
244
+
245
+ However, ActiveJob can be configured to retry an infinite number of times, with an exponential backoff. Using ActiveJob's `retry_on` prevents exceptions from reaching GoodJob:
160
246
 
161
247
  ```ruby
162
- class ApplicationJob < ActiveJob::Base
248
+ class ApplicationJob < ActiveJob::Base
163
249
  retry_on StandardError, wait: :exponentially_longer, attempts: Float::INFINITY
164
250
  # ...
165
251
  end
166
252
  ```
167
253
 
168
- When specifying a limited number of retries, care must be taken to ensure that an error does not bubble up to the GoodJob backend because that will result in the job being re-performed:
254
+ When using `retry_on` with _a limited number of retries_, the final exception will not be rescued and will raise to GoodJob. GoodJob can be configured to discard un-handled exceptions instead of retrying them:
255
+
256
+ ```ruby
257
+ # config/initializers/good_job.rb
258
+ GoodJob.reperform_jobs_on_standard_error = false
259
+ ```
260
+
261
+ Alternatively, pass a block to `retry_on` to handle the final exception instead of raising it to GoodJob:
169
262
 
170
263
  ```ruby
171
- class ApplicationJob < ActiveJob::Base
264
+ class ApplicationJob < ActiveJob::Base
172
265
  retry_on StandardError, attempts: 5 do |_job, _exception|
173
- # Log error, etc.
174
- # You must implement this block, otherwise,
175
- # Active Job will re-raise the error.
176
- # Do not re-raise the error, otherwise
177
- # GoodJob will immediately re-perform the job.
266
+ # Log error, do nothing, etc.
178
267
  end
179
268
  # ...
180
269
  end
181
270
  ```
182
271
 
183
- GoodJob can be configured to allow omitting `retry_on`'s block argument and implicitly discard un-handled errors:
272
+ When using `retry_on` with an infinite number of retries, exceptions will never be raised to GoodJob, which means `GoodJob.on_thread_error` will never be called. To report log or report exceptions to an exception monitoring service (e.g. Sentry, Bugsnag, Airbrake, Honeybadger, etc), create an explicit exception wrapper. For example:
184
273
 
185
274
  ```ruby
186
- # config/initializers/good_job.rb
187
-
188
- # Do NOT re-perform a job if a StandardError bubbles up to the GoodJob backend
189
- GoodJob.reperform_jobs_on_standard_error = false
190
- ```
275
+ class ApplicationJob < ActiveJob::Base
276
+ retry_on StandardError, wait: :exponentially_longer, attempts: Float::INFINITY
191
277
 
192
- When using an exception monitoring service (e.g. Sentry, Bugsnag, Airbrake, Honeybadger, etc), the use of `rescue_on` may be incompatible with their ActiveJob integration. It's safest to explicitly wrap jobs with an exception reporter. For example:
278
+ retry_on SpecialError, attempts: 5 do |_job, exception|
279
+ Raven.capture_exception(exception)
280
+ end
193
281
 
194
- ```ruby
195
- class ApplicationJob < ActiveJob::Base
196
- retry_on StandardError, wait: :exponentially_longer, attempts: Float::INFINITY
197
-
198
282
  around_perform do |_job, block|
199
283
  block.call
200
284
  rescue StandardError => e
@@ -205,19 +289,18 @@ class ApplicationJob < ActiveJob::Base
205
289
  end
206
290
  ```
207
291
 
208
-
209
- ActiveJob's `discard_on` functionality is supported too.
210
-
211
292
  #### ActionMailer retries
212
293
 
213
- Using a Mailer's `#deliver_later` will enqueue an instance of `ActionMailer::DeliveryJob` which inherits from `ActiveJob::Base` rather than your applications `ApplicationJob`. You can use an initializer to configure retries on `ActionMailer::DeliveryJob`:
294
+ Any configuration in `ApplicationJob` will have to be duplicated on `ActionMailer::MailDeliveryJob` (`ActionMailer::DeliveryJob` in Rails 5.2 or earlier) because ActionMailer uses a custom class, `ActionMailer::MailDeliveryJob`, which inherits from `ActiveJob::Base`, rather than your applications `ApplicationJob`.
295
+
296
+ You can use an initializer to configure `ActionMailer::MailDeliveryJob`, for example:
214
297
 
215
298
  ```ruby
216
299
  # config/initializers/good_job.rb
217
- ActionMailer::DeliveryJob.retry_on StandardError, wait: :exponentially_longer, attempts: Float::INFINITY
300
+ ActionMailer::MailDeliveryJob.retry_on StandardError, wait: :exponentially_longer, attempts: Float::INFINITY
218
301
 
219
302
  # With Sentry (or Bugsnag, Airbrake, Honeybadger, etc.)
220
- ActionMailer::DeliveryJob.around_perform do |_job, block|
303
+ ActionMailer::MailDeliveryJob.around_perform do |_job, block|
221
304
  block.call
222
305
  rescue StandardError => e
223
306
  Raven.capture_exception(e)
@@ -225,14 +308,17 @@ rescue StandardError => e
225
308
  end
226
309
  ```
227
310
 
228
- #### Timeouts
311
+ Note, that `ActionMailer::MailDeliveryJob` is a default since Rails 6.0. Be sure that your app is using that class, as it
312
+ might also be configured to use (deprecated now) `ActionMailer::DeliveryJob`.
313
+
314
+ ### Timeouts
229
315
 
230
316
  Job timeouts can be configured with an `around_perform`:
231
317
 
232
318
  ```ruby
233
- class ApplicationJob < ActiveJob::Base
319
+ class ApplicationJob < ActiveJob::Base
234
320
  JobTimeoutError = Class.new(StandardError)
235
-
321
+
236
322
  around_perform do |_job, block|
237
323
  # Timeout jobs after 10 minutes
238
324
  Timeout.timeout(10.minutes, JobTimeoutError) do
@@ -242,20 +328,61 @@ class ApplicationJob < ActiveJob::Base
242
328
  end
243
329
  ```
244
330
 
245
- ### Configuring job execution threads
246
-
247
- GoodJob executes enqueued jobs using threads. There is a lot than can be said about [multithreaded behavior in Ruby on Rails](https://guides.rubyonrails.org/threading_and_code_execution.html), but briefly:
331
+ ### Optimize queues, threads, and processes
248
332
 
249
- - Each GoodJob execution thread requires its own database connection, which are automatically checked out from Rails’s connection pool. _Allowing GoodJob to schedule more threads than are available in the database connection pool can lead to timeouts and is not recommended._
250
- - The maximum number of GoodJob threads can be configured, in decreasing precedence:
251
- 1. `$ bundle exec good_job --max_threads 4`
252
- 2. `$ GOOD_JOB_MAX_THREADS=4 bundle exec good_job`
253
- 3. `$ RAILS_MAX_THREADS=4 bundle exec good_job`
254
- 4. Implicitly via Rails's database connection pool size (`ActiveRecord::Base.connection_pool.size`)
333
+ By default, GoodJob creates a single thread execution pool that will execute jobs from any queue. Depending on your application's workload, job types, and service level objectives, you may wish to optimize execution resources. For example, providing dedicated execution resources for transactional emails so they are not delayed by long-running batch jobs. Some options:
334
+
335
+ - Multiple execution pools within a single process:
336
+
337
+ ```bash
338
+ $ bundle exec good_job --queues="transactional_messages:2;batch_processing:1;-transactional_messages,batch_processing:2;*" --max-threads=5
339
+ ```
340
+
341
+ This configuration will result in a single process with 4 isolated thread execution pools. Isolated execution pools are separated with a semicolon (`;`) and queue names and thread counts with a colon (`:`)
342
+
343
+ - `transactional_messages:2`: execute jobs enqueued on `transactional_messages` with up to 2 threads.
344
+ - `batch_processing:1` execute jobs enqueued on `batch_processing` with a single thread.
345
+ - `-transactional_messages,batch_processing`: execute jobs enqueued on _any_ queue _excluding_ `transactional_messages` or `batch_processing` with up to 2 threads.
346
+ - `*`: execute jobs on any queue on up to 5 threads, as configured by `--max-threads=5`
347
+
348
+ For moderate workloads, multiple isolated thread execution pools offers a good balance between congestion management and economy.
349
+
350
+ Configuration can be injected by environment variables too:
351
+
352
+ ```bash
353
+ $ GOOD_JOB_QUEUES="transactional_messages:2;batch_processing:1;-transactional_messages,batch_processing:2;*" GOOD_JOB_MAX_THREADS=5 bundle exec good_job
354
+ ```
355
+
356
+ - Multiple processes; for example, on Heroku:
357
+
358
+ ```procfile
359
+ # Procfile
360
+
361
+ # Separate dyno types
362
+ worker: bundle exec good_job --max-threads=5
363
+ transactional_worker: bundle exec good_job --queues="transactional_messages" --max-threads=2
364
+ batch_worker: bundle exec good_job --queues="batch_processing" --max-threads=1
365
+
366
+ # Combined multi-process dyno
367
+ combined_worker: bundle exec good_job --max-threads=5 & bundle exec good_job --queues="transactional_messages" --max-threads=2 & bundle exec good_job --queues="batch_processing" --max-threads=1 & wait -n
368
+ ```
369
+
370
+ Running multiple processes can optimize for CPU performance at the expense of greater memory and system resource usage.
371
+
372
+ Keep in mind, queue operations and management is an advanced discipline. This stuff is complex, especially for heavy workloads and unique processing requirements. Good job 👍
373
+
374
+ ### Database connections
375
+
376
+ Each GoodJob execution thread requires its own database connection that is automatically checked out from Rails’s connection pool. _Allowing GoodJob to create more threads than available database connections can lead to timeouts and is not recommended._ For example:
377
+
378
+ ```yaml
379
+ # config/database.yml
380
+ pool: <%= [ENV.fetch("RAILS_MAX_THREADS", 5).to_i, ENV.fetch("GOOD_JOB_MAX_THREADS", 4).to_i].max %>
381
+ ```
255
382
 
256
383
  ### Executing jobs async / in-process
257
384
 
258
- GoodJob is able to run "async" in the same process as the webserver (e.g. `bin/rail s`). GoodJob's async execution mode offers benefits of economy by not requiring a separate job worker process, but with the tradeoff of increased complexity. Async mode can be configured in two ways:
385
+ GoodJob can execute jobs "async" in the same process as the webserver (e.g. `bin/rail s`). GoodJob's async execution mode offers benefits of economy by not requiring a separate job worker process, but with the tradeoff of increased complexity. Async mode can be configured in two ways:
259
386
 
260
387
  - Directly configure the ActiveJob adapter:
261
388
 
@@ -263,12 +390,13 @@ GoodJob is able to run "async" in the same process as the webserver (e.g. `bin/r
263
390
  # config/environments/production.rb
264
391
  config.active_job.queue_adapter = GoodJob::Adapter.new(execution_mode: :async, max_threads: 4, poll_interval: 30)
265
392
  ```
393
+
266
394
  - Or, when using `...queue_adapter = :good_job`, via environment variables:
267
395
 
268
396
  ```bash
269
397
  $ GOOD_JOB_EXECUTION_MODE=async GOOD_JOB_MAX_THREADS=4 GOOD_JOB_POLL_INTERVAL=30 bin/rails server
270
398
  ```
271
-
399
+
272
400
  Depending on your application configuration, you may need to take additional steps:
273
401
 
274
402
  - Ensure that you have enough database connections for both web and job execution threads:
@@ -282,27 +410,27 @@ Depending on your application configuration, you may need to take additional ste
282
410
 
283
411
  ```ruby
284
412
  # config/puma.rb
285
-
413
+
286
414
  before_fork do
287
415
  GoodJob.shutdown
288
416
  end
289
-
417
+
290
418
  on_worker_boot do
291
419
  GoodJob.restart
292
420
  end
293
-
421
+
294
422
  on_worker_shutdown do
295
423
  GoodJob.shutdown
296
424
  end
297
-
425
+
298
426
  MAIN_PID = Process.pid
299
427
  at_exit do
300
428
  GoodJob.shutdown if Process.pid == MAIN_PID
301
429
  end
302
430
  ```
303
-
431
+
304
432
  GoodJob is compatible with Puma's `preload_app!` method.
305
-
433
+
306
434
  ### Migrating to GoodJob from a different ActiveJob backend
307
435
 
308
436
  If your application is already using an ActiveJob backend, you will need to install GoodJob to enqueue and perform newly created jobs _and_ finish performing pre-existing jobs on the previous backend.
@@ -318,7 +446,7 @@ If your application is already using an ActiveJob backend, you will need to inst
318
446
  ```
319
447
 
320
448
  1. Continue running executors for both backends. For example, on Heroku it's possible to run [two processes](https://help.heroku.com/CTFS2TJK/how-do-i-run-multiple-processes-on-a-dyno) within the same dyno:
321
-
449
+
322
450
  ```procfile
323
451
  # Procfile
324
452
  # ...
@@ -331,7 +459,7 @@ If your application is already using an ActiveJob backend, you will need to inst
331
459
 
332
460
  GoodJob is fully instrumented with [`ActiveSupport::Notifications`](https://edgeguides.rubyonrails.org/active_support_instrumentation.html#introduction-to-instrumentation).
333
461
 
334
- By default, GoodJob will delete job records after they are run, regardless of whether they succeed or not (raising a kind of `StandardError`), unless they are interrupted (raising a kind of `Exception`).
462
+ By default, GoodJob will delete job records after they are run, regardless of whether they succeed or not (raising a kind of `StandardError`), unless they are interrupted (raising a kind of `Exception`).
335
463
 
336
464
  To preserve job records for later inspection, set an initializer:
337
465
 
@@ -343,7 +471,7 @@ GoodJob.preserve_job_records = true
343
471
  It is also necessary to delete these preserved jobs from the database after a certain time period:
344
472
 
345
473
  - For example, in a Rake task:
346
-
474
+
347
475
  ```ruby
348
476
  GoodJob::Job.finished(1.day.ago).delete_all
349
477
  ```
@@ -413,7 +541,7 @@ $ gem signin
413
541
  # Update version number, changelog, and create git commit:
414
542
  $ bundle exec rake release[minor] # major,minor,patch
415
543
 
416
- # ..and follow subsequent directions.
544
+ # ..and follow subsequent directions.
417
545
  ```
418
546
 
419
547
  ## License