good_job 1.13.1 → 2.0.1
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +57 -0
- data/README.md +50 -20
- data/lib/generators/good_job/templates/install/migrations/create_good_jobs.rb.erb +1 -0
- data/lib/generators/good_job/templates/update/migrations/01_create_good_jobs.rb +8 -0
- data/lib/good_job/active_job_extensions/concurrency.rb +6 -2
- data/lib/good_job/adapter.rb +6 -36
- data/lib/good_job/configuration.rb +21 -9
- data/lib/good_job/job.rb +16 -66
- data/lib/good_job/lockable.rb +4 -4
- data/lib/good_job/scheduler.rb +3 -1
- data/lib/good_job/version.rb +1 -1
- data/lib/good_job.rb +1 -29
- metadata +2 -5
- data/lib/generators/good_job/templates/update/migrations/02_add_active_job_id_concurrency_key_cron_key_to_good_jobs.rb +0 -16
- data/lib/generators/good_job/templates/update/migrations/03_add_active_job_id_index_and_concurrency_key_index_to_good_jobs.rb +0 -33
- data/lib/generators/good_job/templates/update/migrations/04_add_retried_good_job_id_to_good_jobs.rb +0 -14
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 917d4630bb8be8ea639b3516e8c4fc5465c5dfd52e0ea361259cf68d58bafe90
|
4
|
+
data.tar.gz: 867861782ccd6cc20934162c3ca7c2c76ed09407d9e9a7d00ee4d33f9de120cf
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: a45d3b4eb52724ef8faf8d86c74f7051500aa33b3a98c0482bfc1568df95b444088e9ead34820606abf8f2389f3dc6b8ca51f84e6c07421b861a5f49dd825e38
|
7
|
+
data.tar.gz: 11790854220ca578a92f8b3e4a37fe5cc2d23205cd41e4cb55acfe3e7db5547b091ea4d542b3c466aa12cc59da35903d1566cbf3911cae9c153b8ec1a547e293
|
data/CHANGELOG.md
CHANGED
@@ -1,5 +1,62 @@
|
|
1
1
|
# Changelog
|
2
2
|
|
3
|
+
## [v2.0.1](https://github.com/bensheldon/good_job/tree/v2.0.1) (2021-08-24)
|
4
|
+
|
5
|
+
[Full Changelog](https://github.com/bensheldon/good_job/compare/v2.0.0...v2.0.1)
|
6
|
+
|
7
|
+
**Closed issues:**
|
8
|
+
|
9
|
+
- Is there any value in seeing a backtrace for ConcurrencyExceededError? [\#347](https://github.com/bensheldon/good_job/issues/347)
|
10
|
+
- Release GoodJob 2.0 [\#307](https://github.com/bensheldon/good_job/issues/307)
|
11
|
+
- Unhandled ActiveJob errors should trigger GoodJob.on\_thread\_error [\#247](https://github.com/bensheldon/good_job/issues/247)
|
12
|
+
|
13
|
+
**Merged pull requests:**
|
14
|
+
|
15
|
+
- Suppress backtrace of ConcurrencyExceededError [\#348](https://github.com/bensheldon/good_job/pull/348) ([reczy](https://github.com/reczy))
|
16
|
+
|
17
|
+
## [v2.0.0](https://github.com/bensheldon/good_job/tree/v2.0.0) (2021-08-24)
|
18
|
+
|
19
|
+
[Full Changelog](https://github.com/bensheldon/good_job/compare/v1.99.0...v2.0.0)
|
20
|
+
|
21
|
+
**Implemented enhancements:**
|
22
|
+
|
23
|
+
- Concurrency's enqueue\_limit should exclude performing jobs from count [\#317](https://github.com/bensheldon/good_job/issues/317)
|
24
|
+
- Rename `:async` to `:async_all`; `:async_server` to `:async` and set as Development environment default; do not poll in async development [\#343](https://github.com/bensheldon/good_job/pull/343) ([bensheldon](https://github.com/bensheldon))
|
25
|
+
- Exclude executing jobs from Concurrency's enqueue\_limit's count [\#342](https://github.com/bensheldon/good_job/pull/342) ([bensheldon](https://github.com/bensheldon))
|
26
|
+
- Unhandled ActiveJob errors should trigger GoodJob.on\_thread\_error [\#312](https://github.com/bensheldon/good_job/pull/312) ([bensheldon](https://github.com/bensheldon))
|
27
|
+
|
28
|
+
**Closed issues:**
|
29
|
+
|
30
|
+
- Swap behavior of `async` with `async_server`; rename `async` execution mode to be `async_all`; default `async` in Development; [\#340](https://github.com/bensheldon/good_job/issues/340)
|
31
|
+
- Add hyphen to lock key. e.g. "\[table\_name\]-\[column\]" instead of "\[table\_name\]\[column\]" [\#335](https://github.com/bensheldon/good_job/issues/335)
|
32
|
+
- Use `async_server` as default execution mode in Development environment [\#139](https://github.com/bensheldon/good_job/issues/139)
|
33
|
+
|
34
|
+
**Merged pull requests:**
|
35
|
+
|
36
|
+
- Remove v1.0 deprecation notices and incremental migrations [\#338](https://github.com/bensheldon/good_job/pull/338) ([bensheldon](https://github.com/bensheldon))
|
37
|
+
- Lock GoodJob::Job on active\_job\_id instead of the row id; adds separator hyphen to lock key [\#337](https://github.com/bensheldon/good_job/pull/337) ([bensheldon](https://github.com/bensheldon))
|
38
|
+
|
39
|
+
## [v1.99.0](https://github.com/bensheldon/good_job/tree/v1.99.0) (2021-08-24)
|
40
|
+
|
41
|
+
[Full Changelog](https://github.com/bensheldon/good_job/compare/v1.13.2...v1.99.0)
|
42
|
+
|
43
|
+
**Closed issues:**
|
44
|
+
|
45
|
+
- Set Advisory Lock on ActiveJob job uuid instead of GoodJob's job uuid [\#272](https://github.com/bensheldon/good_job/issues/272)
|
46
|
+
|
47
|
+
**Merged pull requests:**
|
48
|
+
|
49
|
+
- Add upgrade instructions for v1 to v2 [\#345](https://github.com/bensheldon/good_job/pull/345) ([bensheldon](https://github.com/bensheldon))
|
50
|
+
- Add transitional/temporary additional lock on good\_jobs-\[active\_job\_id\] [\#336](https://github.com/bensheldon/good_job/pull/336) ([bensheldon](https://github.com/bensheldon))
|
51
|
+
|
52
|
+
## [v1.13.2](https://github.com/bensheldon/good_job/tree/v1.13.2) (2021-08-18)
|
53
|
+
|
54
|
+
[Full Changelog](https://github.com/bensheldon/good_job/compare/v1.13.1...v1.13.2)
|
55
|
+
|
56
|
+
**Merged pull requests:**
|
57
|
+
|
58
|
+
- Add deprecation notice that `async` mode will be renamed `async_all` in GoodJob v2.0 [\#339](https://github.com/bensheldon/good_job/pull/339) ([bensheldon](https://github.com/bensheldon))
|
59
|
+
|
3
60
|
## [v1.13.1](https://github.com/bensheldon/good_job/tree/v1.13.1) (2021-08-18)
|
4
61
|
|
5
62
|
[Full Changelog](https://github.com/bensheldon/good_job/compare/v1.13.0...v1.13.1)
|
data/README.md
CHANGED
@@ -41,6 +41,8 @@ For more of the story of GoodJob, read the [introductory blog post](https://isla
|
|
41
41
|
- [ActiveJob concurrency](#activejob-concurrency)
|
42
42
|
- [Cron-style repeating/recurring jobs](#cron-style-repeatingrecurring-jobs)
|
43
43
|
- [Updating](#updating)
|
44
|
+
- [Upgrading minor versions](#upgrading-minor-versions)
|
45
|
+
- [Upgrading v1 to v2](#upgrading-v1-to-v2)
|
44
46
|
- [Go deeper](#go-deeper)
|
45
47
|
- [Exceptions, retries, and reliability](#exceptions-retries-and-reliability)
|
46
48
|
- [Exceptions](#exceptions)
|
@@ -123,7 +125,7 @@ For more of the story of GoodJob, read the [introductory blog post](https://isla
|
|
123
125
|
- GoodJob can also be configured to execute jobs within the web server process to save on resources. This is useful for low-workloads when economy is paramount.
|
124
126
|
|
125
127
|
```
|
126
|
-
$ GOOD_JOB_EXECUTION_MODE=
|
128
|
+
$ GOOD_JOB_EXECUTION_MODE=async rails server
|
127
129
|
```
|
128
130
|
|
129
131
|
Additional configuration is likely necessary, see the reference below for f configuration.
|
@@ -211,7 +213,7 @@ Additional configuration can be provided via `config.good_job.OPTION = ...` for
|
|
211
213
|
config.active_job.queue_adapter = :good_job
|
212
214
|
|
213
215
|
# Configure options individually...
|
214
|
-
config.good_job.execution_mode = :
|
216
|
+
config.good_job.execution_mode = :async
|
215
217
|
config.good_job.max_threads = 5
|
216
218
|
config.good_job.poll_interval = 30 # seconds
|
217
219
|
config.good_job.shutdown_timeout = 25 # seconds
|
@@ -220,7 +222,7 @@ config.good_job.cron = { example: { cron: '0 * * * *', class: 'ExampleJob' } }
|
|
220
222
|
|
221
223
|
# ...or all at once.
|
222
224
|
config.good_job = {
|
223
|
-
execution_mode: :
|
225
|
+
execution_mode: :async,
|
224
226
|
max_threads: 5,
|
225
227
|
poll_interval: 30,
|
226
228
|
shutdown_timeout: 25,
|
@@ -239,11 +241,11 @@ Available configuration options are:
|
|
239
241
|
- `execution_mode` (symbol) specifies how and where jobs should be executed. You can also set this with the environment variable `GOOD_JOB_EXECUTION_MODE`. It can be any one of:
|
240
242
|
- `:inline` executes jobs immediately in whatever process queued them (usually the web server process). This should only be used in test and development environments.
|
241
243
|
- `:external` causes the adapter to enqueue jobs, but not execute them. When using this option (the default for production environments), you’ll need to use the command-line tool to actually execute your jobs.
|
242
|
-
- `:async_server` executes jobs in separate threads within the Rails webserver process (`bundle exec rails server`). It can be more economical for small workloads because you don’t need a separate machine or environment for running your jobs, but if your web server is under heavy load or your jobs require a lot of resources, you should choose `:external` instead. When not in the Rails webserver, jobs will execute in `:external` mode to ensure jobs are not executed within `rails console`, `rails db:migrate`, `rails assets:prepare`, etc.
|
243
|
-
- `:
|
244
|
-
- `max_threads` (integer) sets the maximum number of threads to use when `execution_mode` is set to `:async
|
245
|
-
- `queues` (string) determines which queues to execute jobs from when `execution_mode` is set to `:async
|
246
|
-
- `poll_interval` (integer) sets the number of seconds between polls for jobs when `execution_mode` is set to `:async
|
244
|
+
- `:async` (or `:async_server`) executes jobs in separate threads within the Rails webserver process (`bundle exec rails server`). It can be more economical for small workloads because you don’t need a separate machine or environment for running your jobs, but if your web server is under heavy load or your jobs require a lot of resources, you should choose `:external` instead. When not in the Rails webserver, jobs will execute in `:external` mode to ensure jobs are not executed within `rails console`, `rails db:migrate`, `rails assets:prepare`, etc.
|
245
|
+
- `:async_all` executes jobs in separate threads in _any_ Rails process.
|
246
|
+
- `max_threads` (integer) sets the maximum number of threads to use when `execution_mode` is set to `:async`. You can also set this with the environment variable `GOOD_JOB_MAX_THREADS`.
|
247
|
+
- `queues` (string) determines which queues to execute jobs from when `execution_mode` is set to `:async`. See the description of `good_job start` for more details on the format of this string. You can also set this with the environment variable `GOOD_JOB_QUEUES`.
|
248
|
+
- `poll_interval` (integer) sets the number of seconds between polls for jobs when `execution_mode` is set to `:async`. You can also set this with the environment variable `GOOD_JOB_POLL_INTERVAL`.
|
247
249
|
- `max_cache` (integer) sets the maximum number of scheduled jobs that will be stored in memory to reduce execution latency when also polling for scheduled jobs. Caching 10,000 scheduled jobs uses approximately 20MB of memory. You can also set this with the environment variable `GOOD_JOB_MAX_CACHE`.
|
248
250
|
- `shutdown_timeout` (float) number of seconds to wait for jobs to finish when shutting down before stopping the thread. Defaults to forever: `-1`. You can also set this with the environment variable `GOOD_JOB_SHUTDOWN_TIMEOUT`.
|
249
251
|
- `enable_cron` (boolean) whether to run cron process. Defaults to `false`. You can also set this with the environment variable `GOOD_JOB_ENABLE_CRON`.
|
@@ -255,7 +257,7 @@ By default, GoodJob configures the following execution modes per environment:
|
|
255
257
|
|
256
258
|
# config/environments/development.rb
|
257
259
|
config.active_job.queue_adapter = :good_job
|
258
|
-
config.good_job.execution_mode = :
|
260
|
+
config.good_job.execution_mode = :async
|
259
261
|
|
260
262
|
# config/environments/test.rb
|
261
263
|
config.active_job.queue_adapter = :good_job
|
@@ -405,17 +407,45 @@ config.good_job.cron = {
|
|
405
407
|
|
406
408
|
GoodJob follows semantic versioning, though updates may be encouraged through deprecation warnings in minor versions.
|
407
409
|
|
408
|
-
|
410
|
+
#### Upgrading minor versions
|
409
411
|
|
410
|
-
|
411
|
-
bin/rails g good_job:update
|
412
|
-
```
|
412
|
+
Upgrading between minor versions (e.g. v1.4 to v1.5) should not introduce breaking changes, but can introduce new deprecation warnings and database migration notices.
|
413
413
|
|
414
|
-
|
414
|
+
To perform upgrades to the GoodJob database tables:
|
415
415
|
|
416
|
-
|
417
|
-
|
418
|
-
```
|
416
|
+
1. Generate new database migration files:
|
417
|
+
|
418
|
+
```bash
|
419
|
+
bin/rails g good_job:update
|
420
|
+
```
|
421
|
+
|
422
|
+
1. Run the database migration locally
|
423
|
+
|
424
|
+
```bash
|
425
|
+
bin/rails db:migrate
|
426
|
+
```
|
427
|
+
|
428
|
+
1. Commit the migration files and resulting `db/schema.rb` changes.
|
429
|
+
1. Deploy the code, run the migrations against the production database, and restart server/worker processes.
|
430
|
+
|
431
|
+
#### Upgrading v1 to v2
|
432
|
+
|
433
|
+
GoodJob v2 introduces a new Advisory Lock key format that is different than the v1 advisory lock key format; it's therefore necessary to perform a simple, but staged production upgrade. If you are already using `>= v1.12+` no other changes are necessary.
|
434
|
+
|
435
|
+
1. Upgrade your production environment to `v1.99.x` following the minor version upgrade process, including database migrations. `v1.99` is a transitional release that is safely compatible with both `v1.x` and `v2.0.0` because it uses both `v1`- and `v2`-formatted advisory locks.
|
436
|
+
1. Address any deprecation warnings generated by `v1.99`.
|
437
|
+
1. Upgrade your production environment to `v1.99.x` to `v2.0.x` again following the _minor_ upgrade process.
|
438
|
+
|
439
|
+
Notable changes:
|
440
|
+
|
441
|
+
- Renames `:async_server` execution mode to `:async`; renames prior `:async` execution mode to `:async_all`.
|
442
|
+
- Sets default Development environment's execution mode to `:async` with disabled polling.
|
443
|
+
- Excludes performing jobs from `enqueue_limit`'s count in `GoodJob::ActiveJobExtensions::Concurrency`.
|
444
|
+
- Triggers `GoodJob.on_thread_error` for unhandled ActiveJob exceptions.
|
445
|
+
- Renames `GoodJob.reperform_jobs_on_standard_error` accessor to `GoodJob.retry_on_unhandled_error`.
|
446
|
+
- Renames `GoodJob::Adapter.shutdown(wait:)` argument to `GoodJob::Adapter.shutdown(timeout:)`.
|
447
|
+
- Changes Advisory Lock key format from `good_jobs[ROW_ID]` to `good_jobs-[ACTIVE_JOB_ID]`.
|
448
|
+
- Expects presence of columns `good_jobs.active_job_id`, `good_jobs.concurrency_key`, `good_jobs.concurrency_key`, and `good_jobs.retried_good_job_id`.
|
419
449
|
|
420
450
|
## Go deeper
|
421
451
|
|
@@ -587,11 +617,11 @@ GoodJob can execute jobs "async" in the same process as the webserver (e.g. `bin
|
|
587
617
|
config.active_job.queue_adapter = :good_job
|
588
618
|
|
589
619
|
# To change the execution mode
|
590
|
-
config.good_job.execution_mode = :
|
620
|
+
config.good_job.execution_mode = :async
|
591
621
|
|
592
622
|
# Or with more configuration
|
593
623
|
config.good_job = {
|
594
|
-
execution_mode: :
|
624
|
+
execution_mode: :async,
|
595
625
|
max_threads: 4,
|
596
626
|
poll_interval: 30
|
597
627
|
}
|
@@ -600,7 +630,7 @@ GoodJob can execute jobs "async" in the same process as the webserver (e.g. `bin
|
|
600
630
|
- Or, with environment variables:
|
601
631
|
|
602
632
|
```bash
|
603
|
-
$ GOOD_JOB_EXECUTION_MODE=
|
633
|
+
$ GOOD_JOB_EXECUTION_MODE=async GOOD_JOB_MAX_THREADS=4 GOOD_JOB_POLL_INTERVAL=30 bin/rails server
|
604
634
|
```
|
605
635
|
|
606
636
|
Depending on your application configuration, you may need to take additional steps:
|
@@ -13,9 +13,17 @@ class CreateGoodJobs < ActiveRecord::Migration[5.2]
|
|
13
13
|
t.text :error
|
14
14
|
|
15
15
|
t.timestamps
|
16
|
+
|
17
|
+
t.uuid :active_job_id
|
18
|
+
t.text :concurrency_key
|
19
|
+
t.text :cron_key
|
20
|
+
t.uuid :retried_good_job_id
|
16
21
|
end
|
17
22
|
|
18
23
|
add_index :good_jobs, :scheduled_at, where: "(finished_at IS NULL)", name: "index_good_jobs_on_scheduled_at"
|
19
24
|
add_index :good_jobs, [:queue_name, :scheduled_at], where: "(finished_at IS NULL)", name: :index_good_jobs_on_queue_name_and_scheduled_at
|
25
|
+
add_index :good_jobs, [:active_job_id, :created_at], name: :index_good_jobs_on_active_job_id_and_created_at
|
26
|
+
add_index :good_jobs, :concurrency_key, where: "(finished_at IS NULL)", name: :index_good_jobs_on_concurrency_key_when_unfinished
|
27
|
+
add_index :good_jobs, [:cron_key, :created_at], name: :index_good_jobs_on_cron_key_and_created_at
|
20
28
|
end
|
21
29
|
end
|
@@ -4,7 +4,11 @@ module GoodJob
|
|
4
4
|
module Concurrency
|
5
5
|
extend ActiveSupport::Concern
|
6
6
|
|
7
|
-
ConcurrencyExceededError
|
7
|
+
class ConcurrencyExceededError < StandardError
|
8
|
+
def backtrace
|
9
|
+
[] # suppress backtrace
|
10
|
+
end
|
11
|
+
end
|
8
12
|
|
9
13
|
included do
|
10
14
|
class_attribute :good_job_concurrency_config, instance_accessor: false, default: {}
|
@@ -24,7 +28,7 @@ module GoodJob
|
|
24
28
|
|
25
29
|
GoodJob::Job.new.with_advisory_lock(key: key, function: "pg_advisory_lock") do
|
26
30
|
# TODO: Why is `unscoped` necessary? Nested scope is bleeding into subsequent query?
|
27
|
-
enqueue_concurrency = GoodJob::Job.unscoped.where(concurrency_key: key).unfinished.count
|
31
|
+
enqueue_concurrency = GoodJob::Job.unscoped.where(concurrency_key: key).unfinished.advisory_unlocked.count
|
28
32
|
# The job has not yet been enqueued, so check if adding it will go over the limit
|
29
33
|
block.call unless enqueue_concurrency + 1 > limit
|
30
34
|
end
|
data/lib/good_job/adapter.rb
CHANGED
@@ -4,16 +4,13 @@ module GoodJob
|
|
4
4
|
# ActiveJob Adapter.
|
5
5
|
#
|
6
6
|
class Adapter
|
7
|
-
# Valid execution modes.
|
8
|
-
EXECUTION_MODES = [:async, :async_server, :external, :inline].freeze
|
9
|
-
|
10
7
|
# @param execution_mode [Symbol, nil] specifies how and where jobs should be executed. You can also set this with the environment variable +GOOD_JOB_EXECUTION_MODE+.
|
11
8
|
#
|
12
9
|
# - +:inline+ executes jobs immediately in whatever process queued them (usually the web server process). This should only be used in test and development environments.
|
13
10
|
# - +:external+ causes the adapter to enqueue jobs, but not execute them. When using this option (the default for production environments), you'll need to use the command-line tool to actually execute your jobs.
|
14
|
-
# - +:async_server+ executes jobs in separate threads within the Rails webserver process (`bundle exec rails server`). It can be more economical for small workloads because you don't need a separate machine or environment for running your jobs, but if your web server is under heavy load or your jobs require a lot of resources, you should choose +:external+ instead.
|
11
|
+
# - +:async+ (or +:async_server+) executes jobs in separate threads within the Rails webserver process (`bundle exec rails server`). It can be more economical for small workloads because you don't need a separate machine or environment for running your jobs, but if your web server is under heavy load or your jobs require a lot of resources, you should choose +:external+ instead.
|
15
12
|
# When not in the Rails webserver, jobs will execute in +:external+ mode to ensure jobs are not executed within `rails console`, `rails db:migrate`, `rails assets:prepare`, etc.
|
16
|
-
# - +:
|
13
|
+
# - +:async_all+ executes jobs in any Rails process.
|
17
14
|
#
|
18
15
|
# The default value depends on the Rails environment:
|
19
16
|
#
|
@@ -24,23 +21,6 @@ module GoodJob
|
|
24
21
|
# @param queues [String, nil] determines which queues to execute jobs from when +execution_mode+ is set to +:async+. See {file:README.md#optimize-queues-threads-and-processes} for more details on the format of this string. You can also set this with the environment variable +GOOD_JOB_QUEUES+. Defaults to +"*"+.
|
25
22
|
# @param poll_interval [Integer, nil] sets the number of seconds between polls for jobs when +execution_mode+ is set to +:async+. You can also set this with the environment variable +GOOD_JOB_POLL_INTERVAL+. Defaults to +1+.
|
26
23
|
def initialize(execution_mode: nil, queues: nil, max_threads: nil, poll_interval: nil)
|
27
|
-
if caller[0..4].find { |c| c.include?("/config/application.rb") || c.include?("/config/environments/") }
|
28
|
-
ActiveSupport::Deprecation.warn(<<~DEPRECATION)
|
29
|
-
GoodJob no longer recommends creating a GoodJob::Adapter instance:
|
30
|
-
|
31
|
-
config.active_job.queue_adapter = GoodJob::Adapter.new...
|
32
|
-
|
33
|
-
Instead, configure GoodJob through configuration:
|
34
|
-
|
35
|
-
config.active_job.queue_adapter = :good_job
|
36
|
-
config.good_job.execution_mode = :#{execution_mode}
|
37
|
-
config.good_job.max_threads = #{max_threads}
|
38
|
-
config.good_job.poll_interval = #{poll_interval}
|
39
|
-
# etc...
|
40
|
-
|
41
|
-
DEPRECATION
|
42
|
-
end
|
43
|
-
|
44
24
|
@configuration = GoodJob::Configuration.new(
|
45
25
|
{
|
46
26
|
execution_mode: execution_mode,
|
@@ -105,18 +85,8 @@ module GoodJob
|
|
105
85
|
# * +-1+, the scheduler will wait until the shutdown is complete.
|
106
86
|
# * +0+, the scheduler will immediately shutdown and stop any threads.
|
107
87
|
# * A positive number will wait that many seconds before stopping any remaining active threads.
|
108
|
-
# @param wait [Boolean, nil] Deprecated. Use +timeout:+ instead.
|
109
88
|
# @return [void]
|
110
|
-
def shutdown(timeout: :default
|
111
|
-
timeout = if wait.nil?
|
112
|
-
timeout
|
113
|
-
else
|
114
|
-
ActiveSupport::Deprecation.warn(
|
115
|
-
"Using `GoodJob::Adapter.shutdown` with `wait:` kwarg is deprecated; use `timeout:` kwarg instead e.g. GoodJob::Adapter.shutdown(timeout: #{wait ? '-1' : 'nil'})"
|
116
|
-
)
|
117
|
-
wait ? -1 : nil
|
118
|
-
end
|
119
|
-
|
89
|
+
def shutdown(timeout: :default)
|
120
90
|
timeout = if timeout == :default
|
121
91
|
@configuration.shutdown_timeout
|
122
92
|
else
|
@@ -130,15 +100,15 @@ module GoodJob
|
|
130
100
|
# Whether in +:async+ execution mode.
|
131
101
|
# @return [Boolean]
|
132
102
|
def execute_async?
|
133
|
-
@configuration.execution_mode == :
|
134
|
-
@configuration.execution_mode
|
103
|
+
@configuration.execution_mode == :async_all ||
|
104
|
+
@configuration.execution_mode.in?([:async, :async_server]) && in_server_process?
|
135
105
|
end
|
136
106
|
|
137
107
|
# Whether in +:external+ execution mode.
|
138
108
|
# @return [Boolean]
|
139
109
|
def execute_externally?
|
140
110
|
@configuration.execution_mode == :external ||
|
141
|
-
@configuration.execution_mode
|
111
|
+
@configuration.execution_mode.in?([:async, :async_server]) && !in_server_process?
|
142
112
|
end
|
143
113
|
|
144
114
|
# Whether in +:inline+ execution mode.
|
@@ -7,11 +7,13 @@ module GoodJob
|
|
7
7
|
#
|
8
8
|
class Configuration
|
9
9
|
# Valid execution modes.
|
10
|
-
EXECUTION_MODES = [:async, :async_server, :external, :inline].freeze
|
10
|
+
EXECUTION_MODES = [:async, :async_all, :async_server, :external, :inline].freeze
|
11
11
|
# Default number of threads to use per {Scheduler}
|
12
12
|
DEFAULT_MAX_THREADS = 5
|
13
13
|
# Default number of seconds between polls for jobs
|
14
14
|
DEFAULT_POLL_INTERVAL = 10
|
15
|
+
# Default poll interval for async in development environment
|
16
|
+
DEFAULT_DEVELOPMENT_ASYNC_POLL_INTERVAL = -1
|
15
17
|
# Default number of threads to use per {Scheduler}
|
16
18
|
DEFAULT_MAX_CACHE = 10000
|
17
19
|
# Default number of seconds to preserve jobs for {CLI#cleanup_preserved_jobs}
|
@@ -59,7 +61,9 @@ module GoodJob
|
|
59
61
|
|
60
62
|
if mode
|
61
63
|
mode.to_sym
|
62
|
-
elsif Rails.env.development?
|
64
|
+
elsif Rails.env.development?
|
65
|
+
:async
|
66
|
+
elsif Rails.env.test?
|
63
67
|
:inline
|
64
68
|
else
|
65
69
|
:external
|
@@ -98,12 +102,19 @@ module GoodJob
|
|
98
102
|
# poll (using this interval) for new queued jobs to execute.
|
99
103
|
# @return [Integer]
|
100
104
|
def poll_interval
|
101
|
-
(
|
105
|
+
interval = (
|
102
106
|
options[:poll_interval] ||
|
103
107
|
rails_config[:poll_interval] ||
|
104
|
-
env['GOOD_JOB_POLL_INTERVAL']
|
105
|
-
|
106
|
-
|
108
|
+
env['GOOD_JOB_POLL_INTERVAL']
|
109
|
+
)
|
110
|
+
|
111
|
+
if interval
|
112
|
+
interval.to_i
|
113
|
+
elsif Rails.env.development? && execution_mode.in?([:async, :async_all, :async_server])
|
114
|
+
DEFAULT_DEVELOPMENT_ASYNC_POLL_INTERVAL
|
115
|
+
else
|
116
|
+
DEFAULT_POLL_INTERVAL
|
117
|
+
end
|
107
118
|
end
|
108
119
|
|
109
120
|
# The maximum number of future-scheduled jobs to store in memory.
|
@@ -136,12 +147,13 @@ module GoodJob
|
|
136
147
|
def enable_cron
|
137
148
|
value = ActiveModel::Type::Boolean.new.cast(
|
138
149
|
options[:enable_cron] ||
|
139
|
-
|
140
|
-
|
141
|
-
|
150
|
+
rails_config[:enable_cron] ||
|
151
|
+
env['GOOD_JOB_ENABLE_CRON'] ||
|
152
|
+
false
|
142
153
|
)
|
143
154
|
value && cron.size.positive?
|
144
155
|
end
|
156
|
+
|
145
157
|
alias enable_cron? enable_cron
|
146
158
|
|
147
159
|
def cron
|
data/lib/good_job/job.rb
CHANGED
@@ -16,7 +16,7 @@ module GoodJob
|
|
16
16
|
DEFAULT_PRIORITY = 0
|
17
17
|
|
18
18
|
self.table_name = 'good_jobs'
|
19
|
-
self.advisory_lockable_column = '
|
19
|
+
self.advisory_lockable_column = 'active_job_id'
|
20
20
|
|
21
21
|
attr_readonly :serialized_params
|
22
22
|
|
@@ -52,20 +52,6 @@ module GoodJob
|
|
52
52
|
end
|
53
53
|
end
|
54
54
|
|
55
|
-
def self._migration_pending_warning
|
56
|
-
ActiveSupport::Deprecation.warn(<<~DEPRECATION)
|
57
|
-
GoodJob has pending database migrations. To create the migration files, run:
|
58
|
-
|
59
|
-
rails generate good_job:update
|
60
|
-
|
61
|
-
To apply the migration files, run:
|
62
|
-
|
63
|
-
rails db:migrate
|
64
|
-
|
65
|
-
DEPRECATION
|
66
|
-
nil
|
67
|
-
end
|
68
|
-
|
69
55
|
# Get Jobs with given class name
|
70
56
|
# @!method with_job_class
|
71
57
|
# @!scope class
|
@@ -78,14 +64,7 @@ module GoodJob
|
|
78
64
|
# @!method unfinished
|
79
65
|
# @!scope class
|
80
66
|
# @return [ActiveRecord::Relation]
|
81
|
-
scope :unfinished, (
|
82
|
-
if column_names.include?('finished_at')
|
83
|
-
where(finished_at: nil)
|
84
|
-
else
|
85
|
-
ActiveSupport::Deprecation.warn('GoodJob expects a good_jobs.finished_at column to exist. Please see the GoodJob README.md for migration instructions.')
|
86
|
-
nil
|
87
|
-
end
|
88
|
-
end)
|
67
|
+
scope :unfinished, -> { where(finished_at: nil) }
|
89
68
|
|
90
69
|
# Get Jobs that are not scheduled for a later time than now (i.e. jobs that
|
91
70
|
# are not scheduled or scheduled for earlier than the current time).
|
@@ -189,7 +168,13 @@ module GoodJob
|
|
189
168
|
break if good_job.blank?
|
190
169
|
break :unlocked unless good_job&.executable?
|
191
170
|
|
192
|
-
|
171
|
+
begin
|
172
|
+
good_job.with_advisory_lock(key: "good_jobs-#{good_job.active_job_id}") do
|
173
|
+
good_job.perform
|
174
|
+
end
|
175
|
+
rescue RecordAlreadyAdvisoryLockedError => e
|
176
|
+
ExecutionResult.new(value: nil, handled_error: e)
|
177
|
+
end
|
193
178
|
end
|
194
179
|
end
|
195
180
|
|
@@ -225,6 +210,7 @@ module GoodJob
|
|
225
210
|
def self.enqueue(active_job, scheduled_at: nil, create_with_advisory_lock: false)
|
226
211
|
ActiveSupport::Notifications.instrument("enqueue_job.good_job", { active_job: active_job, scheduled_at: scheduled_at, create_with_advisory_lock: create_with_advisory_lock }) do |instrument_payload|
|
227
212
|
good_job_args = {
|
213
|
+
active_job_id: active_job.job_id,
|
228
214
|
queue_name: active_job.queue_name.presence || DEFAULT_QUEUE_NAME,
|
229
215
|
priority: active_job.priority || DEFAULT_PRIORITY,
|
230
216
|
serialized_params: active_job.serialize,
|
@@ -232,26 +218,12 @@ module GoodJob
|
|
232
218
|
create_with_advisory_lock: create_with_advisory_lock,
|
233
219
|
}
|
234
220
|
|
235
|
-
if
|
236
|
-
good_job_args[:active_job_id] = active_job.job_id
|
237
|
-
else
|
238
|
-
_migration_pending_warning
|
239
|
-
end
|
240
|
-
|
241
|
-
if column_names.include?('concurrency_key')
|
242
|
-
good_job_args[:concurrency_key] = active_job.good_job_concurrency_key if active_job.respond_to?(:good_job_concurrency_key)
|
243
|
-
else
|
244
|
-
_migration_pending_warning
|
245
|
-
end
|
221
|
+
good_job_args[:concurrency_key] = active_job.good_job_concurrency_key if active_job.respond_to?(:good_job_concurrency_key)
|
246
222
|
|
247
|
-
if
|
248
|
-
|
249
|
-
|
250
|
-
|
251
|
-
good_job_args[:cron_key] = CurrentExecution.good_job.cron_key
|
252
|
-
end
|
253
|
-
else
|
254
|
-
_migration_pending_warning
|
223
|
+
if CurrentExecution.cron_key
|
224
|
+
good_job_args[:cron_key] = CurrentExecution.cron_key
|
225
|
+
elsif CurrentExecution.active_job_id == active_job.job_id
|
226
|
+
good_job_args[:cron_key] = CurrentExecution.good_job.cron_key
|
255
227
|
end
|
256
228
|
|
257
229
|
good_job = GoodJob::Job.new(**good_job_args)
|
@@ -261,11 +233,7 @@ module GoodJob
|
|
261
233
|
good_job.save!
|
262
234
|
active_job.provider_job_id = good_job.id
|
263
235
|
|
264
|
-
if
|
265
|
-
CurrentExecution.good_job.retried_good_job_id = good_job.id if CurrentExecution.good_job && CurrentExecution.good_job.active_job_id == active_job.job_id
|
266
|
-
else
|
267
|
-
_migration_pending_warning
|
268
|
-
end
|
236
|
+
CurrentExecution.good_job.retried_good_job_id = good_job.id if CurrentExecution.good_job && CurrentExecution.good_job.active_job_id == active_job.job_id
|
269
237
|
|
270
238
|
good_job
|
271
239
|
end
|
@@ -305,24 +273,6 @@ module GoodJob
|
|
305
273
|
self.class.unscoped.unfinished.owns_advisory_locked.exists?(id: id)
|
306
274
|
end
|
307
275
|
|
308
|
-
def active_job_id
|
309
|
-
if self.class.column_names.include?('active_job_id')
|
310
|
-
super
|
311
|
-
else
|
312
|
-
self.class._migration_pending_warning
|
313
|
-
serialized_params['job_id']
|
314
|
-
end
|
315
|
-
end
|
316
|
-
|
317
|
-
def cron_key
|
318
|
-
if self.class.column_names.include?('cron_key')
|
319
|
-
super
|
320
|
-
else
|
321
|
-
self.class._migration_pending_warning
|
322
|
-
nil
|
323
|
-
end
|
324
|
-
end
|
325
|
-
|
326
276
|
private
|
327
277
|
|
328
278
|
# @return [ExecutionResult]
|
data/lib/good_job/lockable.rb
CHANGED
@@ -51,7 +51,7 @@ module GoodJob
|
|
51
51
|
composed_cte = Arel::Nodes::As.new(cte_table, Arel::Nodes::SqlLiteral.new([cte_type, "(", cte_query.to_sql, ")"].join(' ')))
|
52
52
|
query = cte_table.project(cte_table[:id])
|
53
53
|
.with(composed_cte)
|
54
|
-
.where(Arel.sql(sanitize_sql_for_conditions(["#{function}(('x' || substr(md5(:table_name || #{connection.quote_table_name(cte_table.name)}.#{connection.quote_column_name(column)}::text), 1, 16))::bit(64)::bigint)", { table_name: table_name }])))
|
54
|
+
.where(Arel.sql(sanitize_sql_for_conditions(["#{function}(('x' || substr(md5(:table_name || '-' || #{connection.quote_table_name(cte_table.name)}.#{connection.quote_column_name(column)}::text), 1, 16))::bit(64)::bigint)", { table_name: table_name }])))
|
55
55
|
|
56
56
|
limit = original_query.arel.ast.limit
|
57
57
|
query.limit = limit.value if limit.present?
|
@@ -75,8 +75,8 @@ module GoodJob
|
|
75
75
|
join_sql = <<~SQL.squish
|
76
76
|
LEFT JOIN pg_locks ON pg_locks.locktype = 'advisory'
|
77
77
|
AND pg_locks.objsubid = 1
|
78
|
-
AND pg_locks.classid = ('x' || substr(md5(:table_name || #{quoted_table_name}.#{connection.quote_column_name(column)}::text), 1, 16))::bit(32)::int
|
79
|
-
AND pg_locks.objid = (('x' || substr(md5(:table_name || #{quoted_table_name}.#{connection.quote_column_name(column)}::text), 1, 16))::bit(64) << 32)::bit(32)::int
|
78
|
+
AND pg_locks.classid = ('x' || substr(md5(:table_name || '-' || #{quoted_table_name}.#{connection.quote_column_name(column)}::text), 1, 16))::bit(32)::int
|
79
|
+
AND pg_locks.objid = (('x' || substr(md5(:table_name || '-' || #{quoted_table_name}.#{connection.quote_column_name(column)}::text), 1, 16))::bit(64) << 32)::bit(32)::int
|
80
80
|
SQL
|
81
81
|
|
82
82
|
joins(sanitize_sql_for_conditions([join_sql, { table_name: table_name }]))
|
@@ -315,7 +315,7 @@ module GoodJob
|
|
315
315
|
# Default Advisory Lock key
|
316
316
|
# @return [String]
|
317
317
|
def lockable_key
|
318
|
-
|
318
|
+
"#{self.class.table_name}-#{self[self.class._advisory_lockable_column]}"
|
319
319
|
end
|
320
320
|
|
321
321
|
delegate :pg_or_jdbc_query, to: :class
|
data/lib/good_job/scheduler.rb
CHANGED
@@ -168,7 +168,9 @@ module GoodJob # :nodoc:
|
|
168
168
|
# @!visibility private
|
169
169
|
# @return [void]
|
170
170
|
def task_observer(time, output, thread_error)
|
171
|
-
|
171
|
+
error = thread_error || (output.is_a?(GoodJob::ExecutionResult) ? output.unhandled_error : nil)
|
172
|
+
GoodJob.on_thread_error.call(error) if error && GoodJob.on_thread_error.respond_to?(:call)
|
173
|
+
|
172
174
|
instrument("finished_job_task", { result: output, error: thread_error, time: time })
|
173
175
|
create_task if output
|
174
176
|
end
|
data/lib/good_job/version.rb
CHANGED
data/lib/good_job.rb
CHANGED
@@ -55,25 +55,6 @@ module GoodJob
|
|
55
55
|
# @return [Boolean, nil]
|
56
56
|
mattr_accessor :retry_on_unhandled_error, default: true
|
57
57
|
|
58
|
-
# @deprecated Use {GoodJob#retry_on_unhandled_error} instead.
|
59
|
-
# @return [Boolean, nil]
|
60
|
-
def self.reperform_jobs_on_standard_error
|
61
|
-
ActiveSupport::Deprecation.warn(
|
62
|
-
"Calling 'GoodJob.reperform_jobs_on_standard_error' is deprecated. Please use 'retry_on_unhandled_error'"
|
63
|
-
)
|
64
|
-
retry_on_unhandled_error
|
65
|
-
end
|
66
|
-
|
67
|
-
# @deprecated Use {GoodJob#retry_on_unhandled_error=} instead.
|
68
|
-
# @param value [Boolean]
|
69
|
-
# @return [Boolean]
|
70
|
-
def self.reperform_jobs_on_standard_error=(value)
|
71
|
-
ActiveSupport::Deprecation.warn(
|
72
|
-
"Setting 'GoodJob.reperform_jobs_on_standard_error=' is deprecated. Please use 'retry_on_unhandled_error='"
|
73
|
-
)
|
74
|
-
self.retry_on_unhandled_error = value
|
75
|
-
end
|
76
|
-
|
77
58
|
# @!attribute [rw] on_thread_error
|
78
59
|
# @!scope class
|
79
60
|
# This callable will be called when an exception reaches GoodJob (default: +nil+).
|
@@ -96,16 +77,7 @@ module GoodJob
|
|
96
77
|
# * +1..+, the scheduler will wait that many seconds before stopping any remaining active tasks.
|
97
78
|
# @param wait [Boolean] whether to wait for shutdown
|
98
79
|
# @return [void]
|
99
|
-
def self.shutdown(timeout: -1
|
100
|
-
timeout = if wait.nil?
|
101
|
-
timeout
|
102
|
-
else
|
103
|
-
ActiveSupport::Deprecation.warn(
|
104
|
-
"Using `GoodJob.shutdown` with `wait:` kwarg is deprecated; use `timeout:` kwarg instead e.g. GoodJob.shutdown(timeout: #{wait ? '-1' : 'nil'})"
|
105
|
-
)
|
106
|
-
wait ? -1 : nil
|
107
|
-
end
|
108
|
-
|
80
|
+
def self.shutdown(timeout: -1)
|
109
81
|
_shutdown_all(_executables, timeout: timeout)
|
110
82
|
end
|
111
83
|
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: good_job
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version:
|
4
|
+
version: 2.0.1
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Ben Sheldon
|
8
8
|
autorequire:
|
9
9
|
bindir: exe
|
10
10
|
cert_chain: []
|
11
|
-
date: 2021-08-
|
11
|
+
date: 2021-08-24 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: activejob
|
@@ -372,9 +372,6 @@ files:
|
|
372
372
|
- lib/generators/good_job/install_generator.rb
|
373
373
|
- lib/generators/good_job/templates/install/migrations/create_good_jobs.rb.erb
|
374
374
|
- lib/generators/good_job/templates/update/migrations/01_create_good_jobs.rb
|
375
|
-
- lib/generators/good_job/templates/update/migrations/02_add_active_job_id_concurrency_key_cron_key_to_good_jobs.rb
|
376
|
-
- lib/generators/good_job/templates/update/migrations/03_add_active_job_id_index_and_concurrency_key_index_to_good_jobs.rb
|
377
|
-
- lib/generators/good_job/templates/update/migrations/04_add_retried_good_job_id_to_good_jobs.rb
|
378
375
|
- lib/generators/good_job/update_generator.rb
|
379
376
|
- lib/good_job.rb
|
380
377
|
- lib/good_job/active_job_extensions.rb
|
@@ -1,16 +0,0 @@
|
|
1
|
-
# frozen_string_literal: true
|
2
|
-
class AddActiveJobIdConcurrencyKeyCronKeyToGoodJobs < ActiveRecord::Migration[5.2]
|
3
|
-
def change
|
4
|
-
reversible do |dir|
|
5
|
-
dir.up do
|
6
|
-
# Ensure this incremental update migration is idempotent
|
7
|
-
# with monolithic install migration.
|
8
|
-
return if connection.column_exists?(:good_jobs, :active_job_id)
|
9
|
-
end
|
10
|
-
end
|
11
|
-
|
12
|
-
add_column :good_jobs, :active_job_id, :uuid
|
13
|
-
add_column :good_jobs, :concurrency_key, :text
|
14
|
-
add_column :good_jobs, :cron_key, :text
|
15
|
-
end
|
16
|
-
end
|
@@ -1,33 +0,0 @@
|
|
1
|
-
# frozen_string_literal: true
|
2
|
-
class AddActiveJobIdIndexAndConcurrencyKeyIndexToGoodJobs < ActiveRecord::Migration[5.2]
|
3
|
-
disable_ddl_transaction!
|
4
|
-
|
5
|
-
UPDATE_BATCH_SIZE = 1_000
|
6
|
-
|
7
|
-
class GoodJobJobs < ActiveRecord::Base
|
8
|
-
self.table_name = "good_jobs"
|
9
|
-
end
|
10
|
-
|
11
|
-
def change
|
12
|
-
reversible do |dir|
|
13
|
-
dir.up do
|
14
|
-
# Ensure this incremental update migration is idempotent
|
15
|
-
# with monolithic install migration.
|
16
|
-
return if connection.index_name_exists?(:good_jobs, :index_good_jobs_on_active_job_id_and_created_at)
|
17
|
-
end
|
18
|
-
end
|
19
|
-
|
20
|
-
add_index :good_jobs, [:active_job_id, :created_at], algorithm: :concurrently, name: :index_good_jobs_on_active_job_id_and_created_at
|
21
|
-
add_index :good_jobs, :concurrency_key, where: "(finished_at IS NULL)", algorithm: :concurrently, name: :index_good_jobs_on_concurrency_key_when_unfinished
|
22
|
-
add_index :good_jobs, [:cron_key, :created_at], algorithm: :concurrently, name: :index_good_jobs_on_cron_key_and_created_at
|
23
|
-
|
24
|
-
reversible do |dir|
|
25
|
-
dir.up do
|
26
|
-
start_time = Time.current
|
27
|
-
loop do
|
28
|
-
break if GoodJobJobs.where(active_job_id: nil, finished_at: nil).where("created_at < ?", start_time).limit(UPDATE_BATCH_SIZE).update_all("active_job_id = (serialized_params->>'job_id')::uuid").zero?
|
29
|
-
end
|
30
|
-
end
|
31
|
-
end
|
32
|
-
end
|
33
|
-
end
|
data/lib/generators/good_job/templates/update/migrations/04_add_retried_good_job_id_to_good_jobs.rb
DELETED
@@ -1,14 +0,0 @@
|
|
1
|
-
# frozen_string_literal: true
|
2
|
-
class AddRetriedGoodJobIdToGoodJobs < ActiveRecord::Migration[5.2]
|
3
|
-
def change
|
4
|
-
reversible do |dir|
|
5
|
-
dir.up do
|
6
|
-
# Ensure this incremental update migration is idempotent
|
7
|
-
# with monolithic install migration.
|
8
|
-
return if connection.column_exists?(:good_jobs, :retried_good_job_id)
|
9
|
-
end
|
10
|
-
end
|
11
|
-
|
12
|
-
add_column :good_jobs, :retried_good_job_id, :uuid
|
13
|
-
end
|
14
|
-
end
|