good_job 0.8.2 → 1.0.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +81 -3
- data/README.md +95 -29
- data/lib/generators/good_job/install_generator.rb +24 -0
- data/lib/generators/good_job/templates/migration.rb.erb +20 -0
- data/lib/good_job.rb +4 -1
- data/lib/good_job/cli.rb +14 -2
- data/lib/good_job/job.rb +64 -22
- data/lib/good_job/performer.rb +12 -0
- data/lib/good_job/scheduler.rb +27 -14
- data/lib/good_job/version.rb +1 -1
- metadata +5 -2
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: fc7b3997aaa38583efb4c91e66a2a5f9b8bfd75d4c14c9baaf8f1af7c236a42e
|
4
|
+
data.tar.gz: 285b441a5e5add440142960884370a1b74168266252fa48ce569cbb56e80f240
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: f54e83d6ac19d80da2123a395ccf6ffd7eb2faab8e68bd5d50a572c8f0bf64fbc907125a47b7ff91dccac5497af113d043bba4f62ee6ae10fecaf3c004a1114d
|
7
|
+
data.tar.gz: 0b65a60bde26db0cfcfa55f166e80043bcbccb6e2c80f5de1840d2deae3ee28880c904bb53ffc2478ad834ffef8cfe567cee9a179932e20e4f92fdf8504838e4
|
data/CHANGELOG.md
CHANGED
@@ -1,20 +1,98 @@
|
|
1
1
|
# Changelog
|
2
2
|
|
3
|
+
## [v1.0.3](https://github.com/bensheldon/good_job/tree/v1.0.3) (2020-07-25)
|
4
|
+
|
5
|
+
[Full Changelog](https://github.com/bensheldon/good_job/compare/v1.0.2...v1.0.3)
|
6
|
+
|
7
|
+
**Fixed bugs:**
|
8
|
+
|
9
|
+
- Preserve GoodJob::Jobs when a StandardError is raised [\#60](https://github.com/bensheldon/good_job/issues/60)
|
10
|
+
|
11
|
+
**Closed issues:**
|
12
|
+
|
13
|
+
- Have an initial setup generator [\#6](https://github.com/bensheldon/good_job/issues/6)
|
14
|
+
|
15
|
+
**Merged pull requests:**
|
16
|
+
|
17
|
+
- Re-perform a job if a StandardError bubbles up; better document job reliability [\#62](https://github.com/bensheldon/good_job/pull/62) ([bensheldon](https://github.com/bensheldon))
|
18
|
+
- Update the setup documentation to use correct bin setup command [\#61](https://github.com/bensheldon/good_job/pull/61) ([jm96441n](https://github.com/jm96441n))
|
19
|
+
|
20
|
+
## [v1.0.2](https://github.com/bensheldon/good_job/tree/v1.0.2) (2020-07-25)
|
21
|
+
|
22
|
+
[Full Changelog](https://github.com/bensheldon/good_job/compare/v1.0.1...v1.0.2)
|
23
|
+
|
24
|
+
**Fixed bugs:**
|
25
|
+
|
26
|
+
- Fix counting of available execution threads [\#58](https://github.com/bensheldon/good_job/pull/58) ([bensheldon](https://github.com/bensheldon))
|
27
|
+
|
28
|
+
**Closed issues:**
|
29
|
+
|
30
|
+
- repeating/recurring jobs [\#53](https://github.com/bensheldon/good_job/issues/53)
|
31
|
+
|
32
|
+
**Merged pull requests:**
|
33
|
+
|
34
|
+
- Add migration generator [\#56](https://github.com/bensheldon/good_job/pull/56) ([thedanbob](https://github.com/thedanbob))
|
35
|
+
- Fix migration script in readme [\#55](https://github.com/bensheldon/good_job/pull/55) ([thedanbob](https://github.com/thedanbob))
|
36
|
+
|
37
|
+
## [v1.0.1](https://github.com/bensheldon/good_job/tree/v1.0.1) (2020-07-22)
|
38
|
+
|
39
|
+
[Full Changelog](https://github.com/bensheldon/good_job/compare/v1.0.0...v1.0.1)
|
40
|
+
|
41
|
+
**Merged pull requests:**
|
42
|
+
|
43
|
+
- Change threadpool idletime default to 60 seconds from 0 [\#49](https://github.com/bensheldon/good_job/pull/49) ([bensheldon](https://github.com/bensheldon))
|
44
|
+
|
45
|
+
## [v1.0.0](https://github.com/bensheldon/good_job/tree/v1.0.0) (2020-07-20)
|
46
|
+
|
47
|
+
[Full Changelog](https://github.com/bensheldon/good_job/compare/v0.9.0...v1.0.0)
|
48
|
+
|
49
|
+
## [v0.9.0](https://github.com/bensheldon/good_job/tree/v0.9.0) (2020-07-20)
|
50
|
+
|
51
|
+
[Full Changelog](https://github.com/bensheldon/good_job/compare/v0.8.2...v0.9.0)
|
52
|
+
|
53
|
+
**Merged pull requests:**
|
54
|
+
|
55
|
+
- Allow preservation of finished job records [\#46](https://github.com/bensheldon/good_job/pull/46) ([bensheldon](https://github.com/bensheldon))
|
56
|
+
|
3
57
|
## [v0.8.2](https://github.com/bensheldon/good_job/tree/v0.8.2) (2020-07-18)
|
4
58
|
|
5
|
-
[Full Changelog](https://github.com/bensheldon/good_job/compare/v0.
|
59
|
+
[Full Changelog](https://github.com/bensheldon/good_job/compare/v0.8.1...v0.8.2)
|
6
60
|
|
7
61
|
**Closed issues:**
|
8
62
|
|
9
|
-
- Always store a default priority \(0\) and scheduled\_at\(Time.current\) [\#30](https://github.com/bensheldon/good_job/issues/30)
|
10
63
|
- Add a job timeout configuration to time out jobs that have run too long [\#19](https://github.com/bensheldon/good_job/issues/19)
|
11
64
|
|
12
65
|
**Merged pull requests:**
|
13
66
|
|
14
67
|
- Run Github Action tests on PRs from forks [\#44](https://github.com/bensheldon/good_job/pull/44) ([bensheldon](https://github.com/bensheldon))
|
15
68
|
- Fix Rubygems homepage URL [\#43](https://github.com/bensheldon/good_job/pull/43) ([joshmn](https://github.com/joshmn))
|
69
|
+
|
70
|
+
## [v0.8.1](https://github.com/bensheldon/good_job/tree/v0.8.1) (2020-07-18)
|
71
|
+
|
72
|
+
[Full Changelog](https://github.com/bensheldon/good_job/compare/v0.8.0...v0.8.1)
|
73
|
+
|
74
|
+
**Merged pull requests:**
|
75
|
+
|
16
76
|
- Move where\(scheduled\_at: Time.current\) into dynamic part of GoodJob::Job::Performer [\#42](https://github.com/bensheldon/good_job/pull/42) ([bensheldon](https://github.com/bensheldon))
|
77
|
+
|
78
|
+
## [v0.8.0](https://github.com/bensheldon/good_job/tree/v0.8.0) (2020-07-17)
|
79
|
+
|
80
|
+
[Full Changelog](https://github.com/bensheldon/good_job/compare/v0.7.0...v0.8.0)
|
81
|
+
|
82
|
+
**Merged pull requests:**
|
83
|
+
|
17
84
|
- Replace Adapter inline boolean kwarg with execution\_mode instead [\#41](https://github.com/bensheldon/good_job/pull/41) ([bensheldon](https://github.com/bensheldon))
|
85
|
+
|
86
|
+
## [v0.7.0](https://github.com/bensheldon/good_job/tree/v0.7.0) (2020-07-16)
|
87
|
+
|
88
|
+
[Full Changelog](https://github.com/bensheldon/good_job/compare/v0.6.0...v0.7.0)
|
89
|
+
|
90
|
+
**Closed issues:**
|
91
|
+
|
92
|
+
- Always store a default priority \(0\) and scheduled\_at\(Time.current\) [\#30](https://github.com/bensheldon/good_job/issues/30)
|
93
|
+
|
94
|
+
**Merged pull requests:**
|
95
|
+
|
18
96
|
- Add more examples to Readme [\#39](https://github.com/bensheldon/good_job/pull/39) ([bensheldon](https://github.com/bensheldon))
|
19
97
|
- Add additional Rubocops and lint [\#38](https://github.com/bensheldon/good_job/pull/38) ([bensheldon](https://github.com/bensheldon))
|
20
98
|
- Always store a default queue\_name, priority and scheduled\_at; index by queue\_name and scheduled\_at [\#37](https://github.com/bensheldon/good_job/pull/37) ([bensheldon](https://github.com/bensheldon))
|
@@ -35,6 +113,7 @@
|
|
35
113
|
- Add configuration options to good\_job executable [\#33](https://github.com/bensheldon/good_job/pull/33) ([bensheldon](https://github.com/bensheldon))
|
36
114
|
- Extract Job querying behavior out of Scheduler [\#31](https://github.com/bensheldon/good_job/pull/31) ([bensheldon](https://github.com/bensheldon))
|
37
115
|
- Allow configuration of Rails queue adapter with `:good\_job` [\#28](https://github.com/bensheldon/good_job/pull/28) ([bensheldon](https://github.com/bensheldon))
|
116
|
+
- Update development Ruby to 2.6.5 [\#22](https://github.com/bensheldon/good_job/pull/22) ([bensheldon](https://github.com/bensheldon))
|
38
117
|
|
39
118
|
## [v0.5.0](https://github.com/bensheldon/good_job/tree/v0.5.0) (2020-07-13)
|
40
119
|
|
@@ -59,7 +138,6 @@
|
|
59
138
|
|
60
139
|
**Merged pull requests:**
|
61
140
|
|
62
|
-
- Update development Ruby to 2.6.5 [\#22](https://github.com/bensheldon/good_job/pull/22) ([bensheldon](https://github.com/bensheldon))
|
63
141
|
- Simplify the internal API, removing JobWrapper and InlineScheduler [\#21](https://github.com/bensheldon/good_job/pull/21) ([bensheldon](https://github.com/bensheldon))
|
64
142
|
- Generate a new future for every executed job [\#20](https://github.com/bensheldon/good_job/pull/20) ([bensheldon](https://github.com/bensheldon))
|
65
143
|
- Configuration for maximum number of job execution threads [\#18](https://github.com/bensheldon/good_job/pull/18) ([bensheldon](https://github.com/bensheldon))
|
data/README.md
CHANGED
@@ -9,6 +9,8 @@ GoodJob is a multithreaded, Postgres-based, ActiveJob backend for Ruby on Rails.
|
|
9
9
|
- **Backed by Postgres.** Relies upon Postgres integrity and session-level Advisory Locks to provide run-once safety and stay within the limits of `schema.rb`.
|
10
10
|
- **For most workloads.** Targets full-stack teams, economy-minded solo developers, and applications that enqueue less than 1-million jobs/day.
|
11
11
|
|
12
|
+
For more of the story of GoodJob, read the [introductory blog post](https://island94.org/2020/07/introducing-goodjob-1-0).
|
13
|
+
|
12
14
|
## Installation
|
13
15
|
|
14
16
|
Add this line to your application's Gemfile:
|
@@ -26,31 +28,9 @@ $ bundle install
|
|
26
28
|
|
27
29
|
1. Create a database migration:
|
28
30
|
```bash
|
29
|
-
$ bin/rails g
|
31
|
+
$ bin/rails g good_job:install
|
30
32
|
```
|
31
33
|
|
32
|
-
Add to the newly created migration file:
|
33
|
-
|
34
|
-
```ruby
|
35
|
-
class CreateGoodJobs < ActiveRecord::Migration[6.0]
|
36
|
-
def change
|
37
|
-
enable_extension 'pgcrypto'
|
38
|
-
|
39
|
-
create_table :good_jobs, id: :uuid do |t|
|
40
|
-
t.timestamps
|
41
|
-
|
42
|
-
t.text :queue_name
|
43
|
-
t.integer :priority
|
44
|
-
t.jsonb :serialized_params
|
45
|
-
t.timestamp :scheduled_at
|
46
|
-
|
47
|
-
t.index :scheduled_at
|
48
|
-
t.index [:queue_name, :scheduled_at]
|
49
|
-
end
|
50
|
-
end
|
51
|
-
end
|
52
|
-
```
|
53
|
-
|
54
34
|
Run the migration:
|
55
35
|
|
56
36
|
```bash
|
@@ -99,25 +79,85 @@ $ bundle install
|
|
99
79
|
# [--poll-interval=N] # Interval between polls for available jobs in seconds (default: 1)
|
100
80
|
```
|
101
81
|
|
102
|
-
###
|
82
|
+
### Error handling, retries, and reliability
|
83
|
+
|
84
|
+
GoodJob guarantees _at-least-once_ performance of jobs. GoodJob fully supports ActiveJob's built-in functionality for error handling, retries and timeouts.
|
85
|
+
|
86
|
+
#### Error handling
|
87
|
+
|
88
|
+
By default, if a job raises an error while it is being performed, _and it bubbles up to the GoodJob backend_, GoodJob will be immediately re-perform the job until it finishes successfully.
|
103
89
|
|
104
|
-
|
90
|
+
- `Exception`-type errors, such as a SIGINT, will always cause a job to be re-performed.
|
91
|
+
- `StandardError`-type errors, by default, will cause a job to be re-performed, though this is configurable:
|
92
|
+
|
93
|
+
```ruby
|
94
|
+
# config/initializers/good_job.rb
|
95
|
+
GoodJob.reperform_jobs_on_standard_error = true # => default
|
96
|
+
```
|
97
|
+
|
98
|
+
### Retrying jobs
|
99
|
+
|
100
|
+
ActiveJob can be configured to retry an infinite number of times, with an exponential backoff. Using ActiveJob's `retry_on` will ensure that errors do not bubble up to the GoodJob backend:
|
105
101
|
|
106
102
|
```ruby
|
107
103
|
class ApplicationJob < ActiveJob::Base
|
108
|
-
# Retry errors an infinite number of times with exponential back-off
|
109
104
|
retry_on StandardError, wait: :exponentially_longer, attempts: Float::INFINITY
|
105
|
+
# ...
|
106
|
+
end
|
107
|
+
```
|
108
|
+
|
109
|
+
When specifying a limited number of retries, care must be taken to ensure that an error does not bubble up to the GoodJob backend because that will result in the job being re-performed:
|
110
|
+
|
111
|
+
```ruby
|
112
|
+
class ApplicationJob < ActiveJob::Base
|
113
|
+
retry_on StandardError, attempts: 5 do |_job, _exception|
|
114
|
+
# Log error, etc.
|
115
|
+
# You must implement this block, otherwise,
|
116
|
+
# Active Job will re-raise the error.
|
117
|
+
# Do not re-raise the error, otherwise
|
118
|
+
# GoodJob will immediately re-perform the job.
|
119
|
+
end
|
120
|
+
# ...
|
121
|
+
end
|
122
|
+
```
|
110
123
|
|
111
|
-
|
124
|
+
GoodJob can be configured to allow omitting `retry_on`'s block argument and implicitly discard un-handled errors:
|
125
|
+
|
126
|
+
```ruby
|
127
|
+
# config/initializers/good_job.rb
|
128
|
+
|
129
|
+
# Do NOT re-perform a job if a StandardError bubbles up to the GoodJob backend
|
130
|
+
GoodJob.reperform_jobs_on_standard_error = false
|
131
|
+
```
|
132
|
+
|
133
|
+
ActiveJob's `discard_on` functionality is supported too.
|
134
|
+
|
135
|
+
#### ActionMailer retries
|
136
|
+
|
137
|
+
Using a Mailer's `#deliver_later` will enqueue an instance of `ActionMailer::DeliveryJob` which inherits from `ActiveJob::Base` rather than your applications `ApplicationJob`. You can use an initializer to configure retries on `ActionMailer::DeliveryJob`:
|
138
|
+
|
139
|
+
```ruby
|
140
|
+
# config/initializers/good_job.rb
|
141
|
+
ActionMailer::DeliveryJob.retry_on StandardError, wait: :exponentially_longer, attempts: Float::INFINITY
|
142
|
+
```
|
143
|
+
|
144
|
+
#### Timeouts
|
145
|
+
|
146
|
+
Job timeouts can be configured with an `around_perform`:
|
147
|
+
|
148
|
+
```ruby
|
149
|
+
class ApplicationJob < ActiveJob::Base
|
112
150
|
JobTimeoutError = Class.new(StandardError)
|
151
|
+
|
113
152
|
around_perform do |_job, block|
|
153
|
+
# Timeout jobs after 10 minutes
|
114
154
|
Timeout.timeout(10.minutes, JobTimeoutError) do
|
115
155
|
block.call
|
116
156
|
end
|
117
157
|
end
|
118
158
|
end
|
119
159
|
```
|
120
|
-
|
160
|
+
|
121
161
|
### Configuring Job Execution Threads
|
122
162
|
|
123
163
|
GoodJob executes enqueued jobs using threads. There is a lot than can be said about [multithreaded behavior in Ruby on Rails](https://guides.rubyonrails.org/threading_and_code_execution.html), but briefly:
|
@@ -152,6 +192,32 @@ If your application is already using an ActiveJob backend, you will need to inst
|
|
152
192
|
|
153
193
|
1. Once you are confident that no unperformed jobs remain in the previous ActiveJob backend, code and configuration for that backend can be completely removed.
|
154
194
|
|
195
|
+
### Monitoring and preserving worked jobs
|
196
|
+
|
197
|
+
GoodJob is fully instrumented with [`ActiveSupport::Notifications`](https://edgeguides.rubyonrails.org/active_support_instrumentation.html#introduction-to-instrumentation).
|
198
|
+
|
199
|
+
By default, GoodJob will delete job records after they are run, regardless of whether they succeed or not (raising a kind of `StandardError`), unless they are interrupted (raising a kind of `Exception`).
|
200
|
+
|
201
|
+
To preserve job records for later inspection, set an initializer:
|
202
|
+
|
203
|
+
```ruby
|
204
|
+
# config/initializers/good_job.rb
|
205
|
+
GoodJob.preserve_job_records = true
|
206
|
+
```
|
207
|
+
|
208
|
+
It is also necessary to delete these preserved jobs from the database after a certain time period:
|
209
|
+
|
210
|
+
- For example, in a Rake task:
|
211
|
+
|
212
|
+
```ruby
|
213
|
+
# GoodJob::Job.finished(1.day.ago).delete_all
|
214
|
+
```
|
215
|
+
- For example, using the `good_job` command-line utility:
|
216
|
+
|
217
|
+
```bash
|
218
|
+
$ bundle exec good_job cleanup_preserved_jobs --before-seconds-ago=86400
|
219
|
+
```
|
220
|
+
|
155
221
|
## Development
|
156
222
|
|
157
223
|
To run tests:
|
@@ -161,7 +227,7 @@ To run tests:
|
|
161
227
|
$ git clone git@github.com:bensheldon/good_job.git
|
162
228
|
|
163
229
|
# Set up the local environment
|
164
|
-
$ bin/
|
230
|
+
$ bin/setup
|
165
231
|
|
166
232
|
# Run the tests
|
167
233
|
$ bin/rspec
|
@@ -0,0 +1,24 @@
|
|
1
|
+
require 'rails/generators'
|
2
|
+
require 'rails/generators/active_record'
|
3
|
+
|
4
|
+
module GoodJob
|
5
|
+
class InstallGenerator < Rails::Generators::Base
|
6
|
+
include Rails::Generators::Migration
|
7
|
+
|
8
|
+
class << self
|
9
|
+
delegate :next_migration_number, to: ActiveRecord::Generators::Base
|
10
|
+
end
|
11
|
+
|
12
|
+
source_paths << File.join(File.dirname(__FILE__), "templates")
|
13
|
+
|
14
|
+
def create_migration_file
|
15
|
+
migration_template 'migration.rb.erb', 'db/migrate/create_good_jobs.rb', migration_version: migration_version
|
16
|
+
end
|
17
|
+
|
18
|
+
private
|
19
|
+
|
20
|
+
def migration_version
|
21
|
+
"[#{ActiveRecord::VERSION::MAJOR}.#{ActiveRecord::VERSION::MINOR}]"
|
22
|
+
end
|
23
|
+
end
|
24
|
+
end
|
@@ -0,0 +1,20 @@
|
|
1
|
+
class CreateGoodJobs < ActiveRecord::Migration<%= migration_version %>
|
2
|
+
def change
|
3
|
+
enable_extension 'pgcrypto'
|
4
|
+
|
5
|
+
create_table :good_jobs, id: :uuid do |t|
|
6
|
+
t.text :queue_name
|
7
|
+
t.integer :priority
|
8
|
+
t.jsonb :serialized_params
|
9
|
+
t.timestamp :scheduled_at
|
10
|
+
t.timestamp :performed_at
|
11
|
+
t.timestamp :finished_at
|
12
|
+
t.text :error
|
13
|
+
|
14
|
+
t.timestamps
|
15
|
+
end
|
16
|
+
|
17
|
+
add_index :good_jobs, :scheduled_at, where: "(finished_at IS NULL)"
|
18
|
+
add_index :good_jobs, [:queue_name, :scheduled_at], where: "(finished_at IS NULL)"
|
19
|
+
end
|
20
|
+
end
|
data/lib/good_job.rb
CHANGED
@@ -4,13 +4,16 @@ require 'good_job/railtie'
|
|
4
4
|
require 'good_job/logging'
|
5
5
|
require 'good_job/lockable'
|
6
6
|
require 'good_job/job'
|
7
|
-
require
|
7
|
+
require 'good_job/scheduler'
|
8
8
|
require 'good_job/adapter'
|
9
9
|
require 'good_job/pg_locks'
|
10
|
+
require 'good_job/performer'
|
10
11
|
|
11
12
|
require 'active_job/queue_adapters/good_job_adapter'
|
12
13
|
|
13
14
|
module GoodJob
|
15
|
+
mattr_accessor :preserve_job_records, default: false
|
16
|
+
mattr_accessor :reperform_jobs_on_standard_error, default: true
|
14
17
|
include Logging
|
15
18
|
|
16
19
|
ActiveSupport.run_load_hooks(:good_job, self)
|
data/lib/good_job/cli.rb
CHANGED
@@ -39,8 +39,7 @@ module GoodJob
|
|
39
39
|
job_query = GoodJob::Job.all.priority_ordered
|
40
40
|
queue_names_without_all = queue_names.reject { |q| q == '*' }
|
41
41
|
job_query = job_query.where(queue_name: queue_names_without_all) unless queue_names_without_all.size.zero?
|
42
|
-
|
43
|
-
job_performer = job_query.to_performer
|
42
|
+
job_performer = GoodJob::Performer.new(job_query, :perform_with_advisory_lock)
|
44
43
|
|
45
44
|
$stdout.puts "GoodJob worker starting with max_threads=#{max_threads} on queues=#{queue_names.join(',')}"
|
46
45
|
|
@@ -68,6 +67,19 @@ module GoodJob
|
|
68
67
|
$stdout.puts "GoodJob's jobs finished, exiting..."
|
69
68
|
end
|
70
69
|
|
70
|
+
desc :cleanup_preserved_jobs, "Delete preserved job records"
|
71
|
+
method_option :before_seconds_ago,
|
72
|
+
type: :numeric,
|
73
|
+
default: 24 * 60 * 60,
|
74
|
+
desc: "Delete records finished more than this many seconds ago"
|
75
|
+
def cleanup_preserved_jobs
|
76
|
+
require RAILS_ENVIRONMENT_RB
|
77
|
+
|
78
|
+
timestamp = Time.current - options[:before_seconds_ago]
|
79
|
+
result = GoodJob::Job.finished(timestamp).delete_all
|
80
|
+
$stdout.puts "Deleted #{result} preserved #{'job'.pluralize(result)} finished before #{timestamp}."
|
81
|
+
end
|
82
|
+
|
71
83
|
default_task :start
|
72
84
|
end
|
73
85
|
end
|
data/lib/good_job/job.rb
CHANGED
@@ -2,32 +2,38 @@ module GoodJob
|
|
2
2
|
class Job < ActiveRecord::Base
|
3
3
|
include Lockable
|
4
4
|
|
5
|
+
PreviouslyPerformedError = Class.new(StandardError)
|
6
|
+
|
5
7
|
DEFAULT_QUEUE_NAME = 'default'.freeze
|
6
8
|
DEFAULT_PRIORITY = 0
|
7
9
|
|
8
10
|
self.table_name = 'good_jobs'.freeze
|
9
11
|
|
12
|
+
scope :unfinished, (lambda do
|
13
|
+
if column_names.include?('finished_at')
|
14
|
+
where(finished_at: nil)
|
15
|
+
else
|
16
|
+
ActiveSupport::Deprecation.warn('GoodJob expects a good_jobs.finished_at column to exist. Please see the GoodJob README.md for migration instructions.')
|
17
|
+
nil
|
18
|
+
end
|
19
|
+
end)
|
10
20
|
scope :only_scheduled, -> { where(arel_table['scheduled_at'].lteq(Time.current)).or(where(scheduled_at: nil)) }
|
11
21
|
scope :priority_ordered, -> { order(priority: :desc) }
|
12
|
-
scope :
|
13
|
-
|
14
|
-
class Performer
|
15
|
-
def initialize(query)
|
16
|
-
@query = query
|
17
|
-
end
|
22
|
+
scope :finished, ->(timestamp = nil) { timestamp ? where(arel_table['finished_at'].lteq(timestamp)) : where.not(finished_at: nil) }
|
18
23
|
|
19
|
-
|
20
|
-
|
24
|
+
def self.perform_with_advisory_lock
|
25
|
+
good_job = nil
|
26
|
+
result = nil
|
27
|
+
error = nil
|
21
28
|
|
22
|
-
|
23
|
-
|
24
|
-
|
29
|
+
unfinished.only_scheduled.limit(1).with_advisory_lock do |good_jobs|
|
30
|
+
good_job = good_jobs.first
|
31
|
+
break unless good_job
|
25
32
|
|
26
|
-
|
27
|
-
end
|
28
|
-
|
29
|
-
good_job
|
33
|
+
result, error = good_job.perform
|
30
34
|
end
|
35
|
+
|
36
|
+
[good_job, result, error] if good_job
|
31
37
|
end
|
32
38
|
|
33
39
|
def self.enqueue(active_job, scheduled_at: nil, create_with_advisory_lock: false)
|
@@ -50,16 +56,52 @@ module GoodJob
|
|
50
56
|
good_job
|
51
57
|
end
|
52
58
|
|
53
|
-
def perform
|
59
|
+
def perform(destroy_after: !GoodJob.preserve_job_records, reperform_on_standard_error: GoodJob.reperform_jobs_on_standard_error)
|
60
|
+
raise PreviouslyPerformedError, 'Cannot perform a job that has already been performed' if finished_at
|
61
|
+
|
62
|
+
result = nil
|
63
|
+
rescued_error = nil
|
64
|
+
error = nil
|
65
|
+
|
54
66
|
ActiveSupport::Notifications.instrument("before_perform_job.good_job", { good_job: self })
|
55
|
-
|
56
|
-
|
57
|
-
"provider_job_id" => id
|
58
|
-
)
|
59
|
-
ActiveJob::Base.execute(params)
|
67
|
+
self.performed_at = Time.current
|
68
|
+
save! unless destroy_after
|
60
69
|
|
61
|
-
|
70
|
+
params = serialized_params.merge(
|
71
|
+
"provider_job_id" => id
|
72
|
+
)
|
73
|
+
|
74
|
+
begin
|
75
|
+
ActiveSupport::Notifications.instrument("perform_job.good_job", { good_job: self }) do
|
76
|
+
result = ActiveJob::Base.execute(params)
|
77
|
+
end
|
78
|
+
rescue StandardError => e
|
79
|
+
rescued_error = e
|
80
|
+
end
|
81
|
+
|
82
|
+
if rescued_error
|
83
|
+
error = rescued_error
|
84
|
+
elsif result.is_a?(Exception)
|
85
|
+
error = result
|
86
|
+
result = nil
|
87
|
+
end
|
88
|
+
|
89
|
+
error_message = "#{error.class}: #{error.message}" if error
|
90
|
+
self.error = error_message
|
91
|
+
|
92
|
+
if rescued_error && reperform_on_standard_error
|
93
|
+
save!
|
94
|
+
else
|
95
|
+
self.finished_at = Time.current
|
96
|
+
|
97
|
+
if destroy_after
|
98
|
+
destroy!
|
99
|
+
else
|
100
|
+
save!
|
101
|
+
end
|
62
102
|
end
|
103
|
+
|
104
|
+
[result, error]
|
63
105
|
end
|
64
106
|
end
|
65
107
|
end
|
data/lib/good_job/scheduler.rb
CHANGED
@@ -15,19 +15,18 @@ module GoodJob
|
|
15
15
|
min_threads: 0,
|
16
16
|
max_threads: Concurrent.processor_count,
|
17
17
|
auto_terminate: true,
|
18
|
-
idletime:
|
19
|
-
max_queue:
|
20
|
-
fallback_policy: :
|
18
|
+
idletime: 60,
|
19
|
+
max_queue: -1,
|
20
|
+
fallback_policy: :discard,
|
21
21
|
}.freeze
|
22
22
|
|
23
23
|
def initialize(performer, timer_options: {}, pool_options: {})
|
24
24
|
raise ArgumentError, "Performer argument must implement #next" unless performer.respond_to?(:next)
|
25
25
|
|
26
26
|
@performer = performer
|
27
|
-
@pool =
|
27
|
+
@pool = ThreadPoolExecutor.new(DEFAULT_POOL_OPTIONS.merge(pool_options))
|
28
28
|
@timer = Concurrent::TimerTask.new(DEFAULT_TIMER_OPTIONS.merge(timer_options)) do
|
29
|
-
|
30
|
-
create_thread if idle_threads.positive?
|
29
|
+
create_thread
|
31
30
|
end
|
32
31
|
@timer.add_observer(self, :timer_observer)
|
33
32
|
@timer.execute
|
@@ -58,22 +57,36 @@ module GoodJob
|
|
58
57
|
end
|
59
58
|
|
60
59
|
def create_thread
|
60
|
+
return false unless @pool.ready_worker_count.positive?
|
61
|
+
|
61
62
|
future = Concurrent::Future.new(args: [@performer], executor: @pool) do |performer|
|
62
|
-
|
63
|
-
Rails.application.executor.wrap {
|
64
|
-
|
63
|
+
output = nil
|
64
|
+
Rails.application.executor.wrap { output = performer.next }
|
65
|
+
output
|
65
66
|
end
|
66
67
|
future.add_observer(self, :task_observer)
|
67
68
|
future.execute
|
68
69
|
end
|
69
70
|
|
70
|
-
def timer_observer(time, executed_task,
|
71
|
-
ActiveSupport::Notifications.instrument("finished_timer_task.good_job", { result: executed_task, error:
|
71
|
+
def timer_observer(time, executed_task, thread_error)
|
72
|
+
ActiveSupport::Notifications.instrument("finished_timer_task.good_job", { result: executed_task, error: thread_error, time: time })
|
73
|
+
end
|
74
|
+
|
75
|
+
def task_observer(time, output, thread_error)
|
76
|
+
ActiveSupport::Notifications.instrument("finished_job_task.good_job", { result: output, error: thread_error, time: time })
|
77
|
+
create_thread if output
|
72
78
|
end
|
73
79
|
|
74
|
-
|
75
|
-
|
76
|
-
|
80
|
+
class ThreadPoolExecutor < Concurrent::ThreadPoolExecutor
|
81
|
+
# https://github.com/ruby-concurrency/concurrent-ruby/issues/684#issuecomment-427594437
|
82
|
+
def ready_worker_count
|
83
|
+
synchronize do
|
84
|
+
workers_still_to_be_created = @max_length - @pool.length
|
85
|
+
workers_created_but_waiting = @ready.length
|
86
|
+
|
87
|
+
workers_still_to_be_created + workers_created_but_waiting
|
88
|
+
end
|
89
|
+
end
|
77
90
|
end
|
78
91
|
end
|
79
92
|
end
|
data/lib/good_job/version.rb
CHANGED
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: good_job
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.
|
4
|
+
version: 1.0.3
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Ben Sheldon
|
8
8
|
autorequire:
|
9
9
|
bindir: exe
|
10
10
|
cert_chain: []
|
11
|
-
date: 2020-07-
|
11
|
+
date: 2020-07-26 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: concurrent-ruby
|
@@ -236,12 +236,15 @@ files:
|
|
236
236
|
- README.md
|
237
237
|
- exe/good_job
|
238
238
|
- lib/active_job/queue_adapters/good_job_adapter.rb
|
239
|
+
- lib/generators/good_job/install_generator.rb
|
240
|
+
- lib/generators/good_job/templates/migration.rb.erb
|
239
241
|
- lib/good_job.rb
|
240
242
|
- lib/good_job/adapter.rb
|
241
243
|
- lib/good_job/cli.rb
|
242
244
|
- lib/good_job/job.rb
|
243
245
|
- lib/good_job/lockable.rb
|
244
246
|
- lib/good_job/logging.rb
|
247
|
+
- lib/good_job/performer.rb
|
245
248
|
- lib/good_job/pg_locks.rb
|
246
249
|
- lib/good_job/railtie.rb
|
247
250
|
- lib/good_job/scheduler.rb
|